>
> |
Pipeline Meetings - Pipeline/SCIRun Interaction Meeting (Tele-Conference Call, DATE!)
Participants:
Aims of this joint (LONI/SCI) CCB Project
- To develop a plan for Pipeline and SCIRun integration and tool integration by April 2007.
- 2004 Plans: In the CCB/NCBC Core 2, we had an aim on Data and Algorithm Interaction. This broad aim had two sub-aims that pertained to Pipeline/SCIRun interaction (more details are included in the appendix at the end):
- We will extend the Pipeline to incorporate some SCIRun tools
- We will extend the SCIRun Environment to allow running the Pipeline workflows.
Current Project State
- Both Pipeline and SCIRun have made great leaps forward in terms of extending their scopes, processing and visualization tool libraries and their validation using variety of imaging studies.
- With the exception of brief interaction efforts (Summer 2005, Pipeline/SCIRun Hands-on Meeting, and Summer 2006, CCB AHM), there has been little joint effort to make headway on this interaction between the two infrastructures.
- Pipeline data-modeling language is XML-based and solves the challenges identified by the pipeline user community. Similarly, SCIRun’s perl-based scripting language addresses the needs of the SCIRun users/developers. Both allow batch-mode execution. It’s not clear what the Venn-diagram of these 2 feature-sets looks like (how much commonality is there, what is the extend of the disjoint parts, what it may take to integrate these schemas or is it feasible to write a converter between them).
- Does it make sense to start by externally integrating PowerApps? from SCIRun as Tools/Libs within the Pipeline, and vise-versa?
Action Plan
- TCon early in the week of Oct 09-13, 2006. Suggested days/times:
- Wed 10/11/06, 12 PM (PDT)
- Wed 10/11/06, 2 PM (PDT)
- Thu 10/12/06, 11 AM (PDT)
- David Weinstein (SCIRun) and Arash Payan (LONI) will take charge, suggest TCon times and communicate the aims of this effort to their groups.
- Develop a plan with specific deliverables, milestones and a timeline for the next 6 months – in order to facilitate the progress reporting on this project for the April 2007 (3rd year) CCB progress report.
- Wiki page for this effort is at: http://www.loni.ucla.edu/twiki/bin/view/Pipeline/MeetingMinutesOctober2006PipelineSCIRun (SCI members that have not already registered to edit our Wiki resources, may consider doing so. You can read the Wiki pages, but not edit, unless you registered first).
Pipeline/SCIRun Interaction Specification:
- Summary: Visual programming environments allow users to view the stages of algorithms. By extending the Pipeline to incorporate SCIRun, we will increase the level of granularity at which the user can control a data process. This will also increase the availability of visualization methods within the Pipeline Environment. We will develop intelligent algorithms to automatically launch viewers that are appropriate for a given data type. Similarly, the Pipeline Environment will be available within SCIRun, allowing users access to a broad collection of algorithms. Pipeline processing protocols will be introduced that wrap our current perl and script language commands for gene splicing, cross-species homology computation and sequence alignment into graphical Pipeline and SCIRun modules for automated and robust genetic modeling.
- We will expand the capabilities of the Pipeline Processing Environment so that its users have access to SCIRun modules within the Pipeline environment. This will leverage SCIRun’s visualization abilities with the Pipeline’s integrative stance and massive processing architecture, thereby allowing for complex visualization integrated with the Pipeline. This will allow for client/server processing of SCIRun modules, with client side execution of visualization processing. Modules from SCIRun will integrate seamlessly in the Pipeline workspace receiving processed data and launching visualization and other user interactive windows.
Core 2, Figure 37: This figure illustrates the similarity and the interaction of the LONI Pipeline and the SCIRun environments.
One of the first benefits that this integration will produce is to provide low level processing routines in the Pipeline. Currently, to perform sequences of mathematical tasks, users must either write their own programs or make use of multiple executables within the Pipeline. The Pipeline/SCIRun integration will allow users to utilize the math, geometry, and field libraries of SCIRun to allow efficient low level processing from within the Pipeline. Pipeline will no longer require separate program launches for small tasks, instead it will pass these smaller manipulations to the SCIRun executer. External algorithms and executables will still be integrated with the Pipeline through descriptors without an API.
We will integrate the SCIRun interface into the Pipeline, which will provide access to SCIRun modules whenever SCIRun is detected on the host machine. The SCIRun modules will appear in the tools panel of the environment. Users will be able to drag and drop SCIRun modules into the Pipeline workspace and work with them as they would any other module. SCIRun modules will maintain a look and feel similar to their native SCIRun look and feel. This will allow the user to edit the method as it would be edited within the SCIRun environment.
Currently, the Pipeline and SCIRun both use XML to describe their modules. We will extend the Pipeline to understand the SCIRun XML by integrating these descriptors into the Pipeline XML schema. We will then use the SCIRun XML descriptions of components to build the Pipeline modules. We will provide mechanisms for data translation using the data mediation architecture, described in Section E.4. The LONI Debabeler Engine will translate data in the Pipeline into data types and formats that are understood by SCIRun.
The CCB Pipeline Processing Environment will start a SCIRun executor when a pipeline requires this functionality. The CCB Pipeline will make calls to the executer for specific functions when the dataflow of a Pipeline requires it. The SCIRun executer will provide a multithreaded approach, thus maintaining the parallelism that is dictated by the Pipeline.
Pipelines in SCIRun: SCIRun will be extended to interpret the LONI Pipeline XML descriptors, utilizing them to insert modules and pipelines into the SCIRun environment. SCIRun will make calls to a Pipeline engine, executing specified algorithms and pipelets. Again, the LONI Debabeler Engine will provide the necessary conversion of files and data to be exchanged between SCIRun and the Pipeline.
Core 2, Figure 38: SCIRun2 with different component models: integrating raw data, image processing and high dimensional data visualization.
The knowledge management tools described in Section D.3 provide a mechanism to determine the nature of the data in a given file. This information will be used within the LPPE to automatically select visualization methods that are appropriate for data at any stage of a pipeline. When requested by the user, the LPPE will launch an appropriate viewer from the available SCIRun modules. We will also use these mechanisms to connect the SCIRun environment to data stored within the database environment (Section D.3). SCIRun will select appropriate visualization strategies to provide interactive viewing of stored raw and processed data. We will extend SCIRun to maintain the provenance of data processed within its environment. The CCB provenance tools will be extended to appropriately interpret the SCIRun results.
- Integration of the CCB Pipeline into the SCIRun/BioPSE Problem-solving Environment using the Meta Component Model
The SCI Institute has recently created SCIRun2, which is an update to the SCIRun/BioPSE system specifically built to allow the seamless operation of external applications within SCIRun. SCIRun2 provides a meta component model that allows components to be used together even if they do not share the same underlying component architecture. Currently, SCIRun2 provides support for the DOE Common Component Architecture, a model that is more familiar to programmers who have used the Object Management Group’s (OMG) CORBA (OMG June 1995) or Microsoft’s COM. While SCIRun2 will still support for older SCIRun dataflow components, we are planning support for CORBA, COM, Itk and Vtk and we will extend SCIRun2 to support the CCB Pipeline. As a result, SCIRun2 will allow the native utilization of SCIRun components, CCA components, CCB components and networks, Itk, Vtk, and others in the same simulation. This capability will provide a variety of components available to the application developer, without requiring buy-in to any one particular methodology.
Figure 2-26 demonstrates with a simple example how SCIRun2 handles different component models. Two CCA components, Driver and Integrator, and one CORBA component, Function, are created in the SCIRun2 framework. In this simple example, the Driver is connected to both the Function and Integrator. Inside SCIRun2, two frameworks are hidden: the CCA framework and the CORBA Object Request Broker (ORB). The CCA framework creates the CCA components, Driver and Integrator. The CORBA framework creates the CORBA component, Function. The two CCA components can be connected in a straightforward manner through the CCA component model. However, the components Driver and Function cannot be connected directly, because neither CCA nor CORBA allow a connection from a component of a different model. Instead, a bridge component is created. Bridges belong to a special internal component model that is used to build a connection between components of different component models. In this example, Bridge has two ports: one CCA port and one CORBA port. In this way it can be connected to both CCA component and CORBA component. The CORBA invocation is converted to request to the CCA port inside the bridge component. In the case of the CCB pipeline bridge components will be automatically generated. Thus, the meta component model will provide a plug-in architecture for the CCB pipeline component models. The abstracted components will be manipulated and managed by the SCIRun2 framework, while the concrete CCB pipeline components will perform the actual work. This will allow the CCB components to communicate among each other in the most efficient manner, while facilitating their connection to SCIRun visualization and other SCIRun2 components.
|
