|
The LONI Pipeline v3 has the following features
|
|
- Ontology enabled processing
- Documentation via Wiki, Doxygen, and Javadoc
|
|
<
< |
We have identified features which we would like to implement for future versions of the software. The features are classified under their respective categories, and both are sorted by descending order of expected implementation (not to be confused with order of importance):
| Core execution |
|---|
| feature | status |
|---|
| srb integration | in testing |
| submission of jobs from extenal applications | in progress |
| job completion notification (via events within application, system to external applications, and email to users) | in progress |
| allow server disconnections without pipeline termination | in progress |
| transparent re-establishment of client connection, including message queuing | in progress |
| user groups for execution, file, and configuration permissions | in progress |
| dynamically configured permissions for server modules | identified |
| processing statistics gathering | identified |
| execution estimation | identified |
| control modules (e.g. loops, multiplexors, breakpoints) | identified |
| self updating capabilities | identified |
| nested list processing | identified |
| quality assurance test suite | identified |
| analysis validation | identified |
| provenance | identified |
| Error handling |
|---|
| feature | status |
|---|
| automated file translations | identified |
| ensure disk space availability | identified |
| Logging |
|---|
| feature | status |
|---|
| write ahead logging for crash recovery | identified |
| search capabilities | identified |
| User interface |
|---|
| feature | status |
|---|
| data-centric graph | in progress |
| message of the day | identified |
| advanced data viewer interface | identified |
| tab visibility | identified |
| Documentation |
|---|
| feature | status |
|---|
| graduated difficulty tutorial | in progress |
| Miscellaneous |
|---|
| feature | status |
|---|
| explore computation as utility to be rented | identified |
- network communication between process monitors. there are 3 types of singletonmonitors- vanilla flavored which runs locally (VF), remotelyexecuted (RE), which means it is submitted to a server somewhere on the network, and remotelysubmitted (RS), which means it runs locally, but was submitted by some client on the network. each processmonitor (PM) which a singletonmonitor holds needs to query for the status of the previous processes to know when it can begin. so if RE follows VF, that means the PMs under RE's corresponding RS need to submit queries over the network, the RE needs to query the VF, and then submit a response back to the RS. to handle the case when a VF follows an RE, the VS will push the status of its PMs to the RE.
- Application.getApplicationResource should load the default config file, and then overwite any property specified in the user's directory. currently is exclusive.
- modify build.xml to package the gui as its own jar file
- correctly convert extended tabs (talk to allan about this)
- windows url parsing (the regex in URL needs to be modified)
- junit tests (in tests directory)
- persistence for data/ProvenanceEntry
Got more items on your wish list? Add them here!
|
|
-- MichaelJPan - 24 Sep 2004
|
|
The LONI Pipeline v3 has the following features
|
|
- Runtime list processing
- Bash shell script output
- Transparent module syntax updater framework
|
|
>
> |
- Ontology enabled processing
|
|
- Documentation via Wiki, Doxygen, and Javadoc
We have identified features which we would like to implement for future versions of the software. The features are classified under their respective categories, and both are sorted by descending order of expected implementation (not to be confused with order of importance):
| Core execution |
|---|
| feature | status |
|---|
|
|
<
< |
| srb integration | in progress |
|
>
> |
| srb integration | in testing |
|
|
| submission of jobs from extenal applications | in progress |
| job completion notification (via events within application, system to external applications, and email to users) | in progress |
| allow server disconnections without pipeline termination | in progress |
|
|
<
< |
| transparent re-establishment of client connection, including message queuing | identified |
| user groups for execution, file, and configuration permissions | identified |
|
>
> |
| transparent re-establishment of client connection, including message queuing | in progress |
| user groups for execution, file, and configuration permissions | in progress |
|
|
| dynamically configured permissions for server modules | identified |
| processing statistics gathering | identified |
| execution estimation | identified |
|
|
| self updating capabilities | identified |
| nested list processing | identified |
| quality assurance test suite | identified |
|
|
<
< |
|
>
> |
| analysis validation | identified |
|
|
|
|
| User interface |
|---|
| feature | status |
|---|
|
|
>
> |
| data-centric graph | in progress |
|
|
| message of the day | identified |
|
|
<
< |
| data-centric graph | identified |
|
|
| advanced data viewer interface | identified |
| tab visibility | identified |
| Documentation |
|---|
| feature | status |
|---|
|
|
<
< |
| graduated difficulty tutorial | identified |
| specification of handshaking process in establishing network connection and encryption keys | identified |
|
>
> |
| graduated difficulty tutorial | in progress |
|
|
| Miscellaneous |
|---|
| feature | status |
|---|
| explore computation as utility to be rented | identified |
- network communication between process monitors. there are 3 types of singletonmonitors- vanilla flavored which runs locally (VF), remotelyexecuted (RE), which means it is submitted to a server somewhere on the network, and remotelysubmitted (RS), which means it runs locally, but was submitted by some client on the network. each processmonitor (PM) which a singletonmonitor holds needs to query for the status of the previous processes to know when it can begin. so if RE follows VF, that means the PMs under RE's corresponding RS need to submit queries over the network, the RE needs to query the VF, and then submit a response back to the RS. to handle the case when a VF follows an RE, the VS will push the status of its PMs to the RE.
|
|
<
< |
- getXMLConfigFile in extension manager should load the default config file, and then overwite any property specified in the user's directory. currently is exclusive.
|
>
> |
- Application.getApplicationResource should load the default config file, and then overwite any property specified in the user's directory. currently is exclusive.
|
|
- modify build.xml to package the gui as its own jar file
- correctly convert extended tabs (talk to allan about this)
- windows url parsing (the regex in URL needs to be modified)
|
|
<
< |
- connection to the srb (see SRBConnection)
|
|
- junit tests (in tests directory)
- persistence for data/ProvenanceEntry
|
|
The LONI Pipeline v3 has the following features
|
|
-
- Consolidated messaging
- Node connection highlighting
- Multiple node operators
|
|
>
> |
- Runtime list processing
- Bash shell script output
- Transparent module syntax updater framework
- Documentation via Wiki, Doxygen, and Javadoc
|
|
We have identified features which we would like to implement for future versions of the software. The features are classified under their respective categories, and both are sorted by descending order of expected implementation (not to be confused with order of importance):
|
|
| submission of jobs from extenal applications | in progress |
| job completion notification (via events within application, system to external applications, and email to users) | in progress |
| allow server disconnections without pipeline termination | in progress |
|
|
<
< |
| translation of v1 syntax to v3 syntax | in progress |
| translation of pipeline files to shell scripts | in progress |
|
|
| transparent re-establishment of client connection, including message queuing | identified |
|
|
<
< |
| runtime list processing (where output of one module to the input of another is a list) | identified |
|
|
| user groups for execution, file, and configuration permissions | identified |
| dynamically configured permissions for server modules | identified |
| processing statistics gathering | identified |
|
|
| Documentation |
|---|
| feature | status |
|---|
|
|
<
< |
| documentation | wiki, doxygen established |
| module repository | wiki established |
|
|
| graduated difficulty tutorial | identified |
| specification of handshaking process in establishing network connection and encryption keys | identified |
|
|
>
> |
The LONI Pipeline v3 has the following features
- Modular architecture
- Server/client communications via TCP
- AES encrypted communications
- Ontology enabled data processing using OWL
- Runtime processing of filenames
- Automatic cache management
- Grid engine integration
- File streaming
- URL wrapper and data resource abstraction
- Comprehensive logging
- Process stream capturing
- Integrated viewer
- User friendly GUI
- Tooltips
- Consolidated messaging
- Node connection highlighting
- Multiple node operators
|
|
We have identified features which we would like to implement for future versions of the software. The features are classified under their respective categories, and both are sorted by descending order of expected implementation (not to be confused with order of importance):
| Core execution |
|---|
| feature | status |
|---|
|
|
<
< |
| extension architecture | completed |
| server/client communications via tcp | completed |
| aes encryption | completed |
| ontology enabled data processing using owl | completed |
| runtime processing of temporary filenames | completed |
| automatic cache management | completed |
| url wrapper and data resource abstraction | completed |
|
|
| srb integration | in progress |
|
|
<
< |
| sun grid engine integration | completed |
| file streaming | completed |
|
|
| submission of jobs from extenal applications | in progress |
| job completion notification (via events within application, system to external applications, and email to users) | in progress |
| allow server disconnections without pipeline termination | in progress |
|
|
|
|
<
< |
| capture of stderr streams | completed |
| logging to persistent files | completed |
|
|
| write ahead logging for crash recovery | identified |
| search capabilities | identified |
| User interface |
|---|
| feature | status |
|---|
|
|
<
< |
| allow user comments on modules and module arguments | completed |
| single dialog window which aggregates all error messages | completed |
| selection of a module on the gui should highlight all its connections | completed |
| simple viewer for common filetypes | completed |
| ability to select multiple nodes and apply an operator (e.g. delete) to them | completed |
|
|
| message of the day | identified |
| data-centric graph | identified |
| advanced data viewer interface | identified |
|
|
-- MichaelJPan - 24 Sep 2004
|
|
>
> |
| META TOPICMOVED | MichaelJPan? | date="1109398505" from="Pipeline.FeaturesWishList" to="Pipeline.PipelineFeatures" |
|---|
|
|
We have identified features which we would like to implement for future versions of the software. The features are classified under their respective categories, and both are sorted by descending order of expected implementation (not to be confused with order of importance):
|
|
| url wrapper and data resource abstraction | completed |
| srb integration | in progress |
| sun grid engine integration | completed |
|
|
>
> |
|
|
| submission of jobs from extenal applications | in progress |
| job completion notification (via events within application, system to external applications, and email to users) | in progress |
| allow server disconnections without pipeline termination | in progress |
|
|
>
> |
| translation of v1 syntax to v3 syntax | in progress |
|
|
| translation of pipeline files to shell scripts | in progress |
| transparent re-establishment of client connection, including message queuing | identified |
| runtime list processing (where output of one module to the input of another is a list) | identified |
|
|
| allow user comments on modules and module arguments | completed |
| single dialog window which aggregates all error messages | completed |
| selection of a module on the gui should highlight all its connections | completed |
|
|
<
< |
| simple viewer for common filetypes | identified |
| ability to select multiple nodes and apply an operator (e.g. delete) to them | identified |
|
>
> |
| simple viewer for common filetypes | completed |
| ability to select multiple nodes and apply an operator (e.g. delete) to them | completed |
|
|
| message of the day | identified |
|
|
>
> |
| data-centric graph | identified |
| advanced data viewer interface | identified |
| tab visibility | identified |
|
|
| Documentation |
|---|
| feature | status |
|---|
|
|
<
< |
| documentation | wiki established |
|
>
> |
| documentation | wiki, doxygen established |
|
|
| module repository | wiki established |
| graduated difficulty tutorial | identified |
| specification of handshaking process in establishing network connection and encryption keys | identified |
|
|
| feature | status |
|---|
| explore computation as utility to be rented | identified |
|
|
<
< |
- file streaming. currently, URL.slurp() handles it, but that crashes the jvm with an OutOfMemoryException. so to send files over the network, we need to break it up and reconstruct it at the other end. other files to look at are net.Packet and net.PipelineConnection.
|
|
- network communication between process monitors. there are 3 types of singletonmonitors- vanilla flavored which runs locally (VF), remotelyexecuted (RE), which means it is submitted to a server somewhere on the network, and remotelysubmitted (RS), which means it runs locally, but was submitted by some client on the network. each processmonitor (PM) which a singletonmonitor holds needs to query for the status of the previous processes to know when it can begin. so if RE follows VF, that means the PMs under RE's corresponding RS need to submit queries over the network, the RE needs to query the VF, and then submit a response back to the RS. to handle the case when a VF follows an RE, the VS will push the status of its PMs to the RE.
- getXMLConfigFile in extension manager should load the default config file, and then overwite any property specified in the user's directory. currently is exclusive.
- modify build.xml to package the gui as its own jar file
|
|
We have identified features which we would like to implement for future versions of the software. The features are classified under their respective categories, and both are sorted by descending order of expected implementation (not to be confused with order of importance):
|
|
| feature | status |
|---|
| explore computation as utility to be rented | identified |
|
|
>
> |
- file streaming. currently, URL.slurp() handles it, but that crashes the jvm with an OutOfMemoryException. so to send files over the network, we need to break it up and reconstruct it at the other end. other files to look at are net.Packet and net.PipelineConnection.
- network communication between process monitors. there are 3 types of singletonmonitors- vanilla flavored which runs locally (VF), remotelyexecuted (RE), which means it is submitted to a server somewhere on the network, and remotelysubmitted (RS), which means it runs locally, but was submitted by some client on the network. each processmonitor (PM) which a singletonmonitor holds needs to query for the status of the previous processes to know when it can begin. so if RE follows VF, that means the PMs under RE's corresponding RS need to submit queries over the network, the RE needs to query the VF, and then submit a response back to the RS. to handle the case when a VF follows an RE, the VS will push the status of its PMs to the RE.
- getXMLConfigFile in extension manager should load the default config file, and then overwite any property specified in the user's directory. currently is exclusive.
- modify build.xml to package the gui as its own jar file
- correctly convert extended tabs (talk to allan about this)
- windows url parsing (the regex in URL needs to be modified)
- connection to the srb (see SRBConnection)
- junit tests (in tests directory)
- persistence for data/ProvenanceEntry
|
|
Got more items on your wish list? Add them here!
|
|
We have identified features which we would like to implement for future versions of the software. The features are classified under their respective categories, and both are sorted by descending order of expected implementation (not to be confused with order of importance):
|
|
| automatic cache management | completed |
| url wrapper and data resource abstraction | completed |
| srb integration | in progress |
|
|
<
< |
| sun grid engine integration | in progress |
|
>
> |
| sun grid engine integration | completed |
|
|
| submission of jobs from extenal applications | in progress |
| job completion notification (via events within application, system to external applications, and email to users) | in progress |
| allow server disconnections without pipeline termination | in progress |
|
|
|
|
<
< |
| capture of stderr streams | identified |
| logging to persistent files | identified |
|
>
> |
| capture of stderr streams | completed |
| logging to persistent files | completed |
|
|
| write ahead logging for crash recovery | identified |
| search capabilities | identified |
| User interface |
|---|
| feature | status |
|---|
| allow user comments on modules and module arguments | completed |
|
|
<
< |
| single dialog window which aggregates all error messages | in progress |
|
>
> |
| single dialog window which aggregates all error messages | completed |
|
|
| selection of a module on the gui should highlight all its connections | identified |
| simple viewer for common filetypes | identified |
| ability to select multiple nodes and apply an operator (e.g. delete) to them | identified |
|
|
<
< |
We have identified features which we would like to implement for future versions of the software. Text in red indicates the developmental status of each item.
|
>
> |
We have identified features which we would like to implement for future versions of the software. The features are classified under their respective categories, and both are sorted by descending order of expected implementation (not to be confused with order of importance):
|
|
|
|
<
< |
- ability to select multiple nodes and apply an operator (e.g. delete) to them
- notification of process completion (via system to applications, and email to users) see item 3
- submission of jobs from external (non-pipeline) applications communications via tcp complete. implementation of client library, and documentation of handshaking specification in progress
- single dialog window which aggregates all error messages
- validation
- dynamically configurable permissions for server modules
- allow users some control in expanding allowed server modules
- multiplexor modules
- loop modules
- breakpoints
- better error handling
- better (and more sophisticated) gui for editing modules (e.g. global text substitution)
- allow user comments on modules and module arguments implementation of <rdfs:label/> and <rdfs:comment/> tags complete
- community documentation this wiki
- communal repository of existing modules this wiki
- allow users to disconnect from server but not have their pipelines terminated see item 3
- processing statistics gathering
- execution estimation
- runtime list processing (when output of one module is a list file, which is an input to another)
- different user levels have different capabilities (i.e. editing of certain variables)
- better handling of "+" and "-" string operators
- automated processing of compressed files
- shell script/pipeline translation
- user modifiable configuration files (e.g. background color, window size, or user-specific temporary directory)
- write ahead logging
- output log to a file
- better search abilities (in text of processing log, or in pipeline itself)
- runtime processing of temporary filenames completed
- simple viewer for common file types (allows quick inspection of data)
- failure message should specify which module failed
- capture of stderr to logfile
- abstraction/wrapper for various data sources implementation of URL wrapper in progress
- allow user to specify execution phases (for pre- and post- processing)
- message of the day
- self updating software
- check for disk space
- more informative error messages
- look into computation as utility to be rented
- graduated difficulty tutorial
- nested lists (to specify synchronization of files)
- implicit file conversion
- pipeline syntax translator
- test suite for robustness
- selection a module should highlight its input and output connections
|
>
> |
| Core execution |
|---|
| feature | status |
|---|
| extension architecture | completed |
| server/client communications via tcp | completed |
| aes encryption | completed |
| ontology enabled data processing using owl | completed |
| runtime processing of temporary filenames | completed |
| automatic cache management | completed |
| url wrapper and data resource abstraction | completed |
| srb integration | in progress |
| sun grid engine integration | in progress |
| submission of jobs from extenal applications | in progress |
| job completion notification (via events within application, system to external applications, and email to users) | in progress |
| allow server disconnections without pipeline termination | in progress |
| translation of pipeline files to shell scripts | in progress |
| transparent re-establishment of client connection, including message queuing | identified |
| runtime list processing (where output of one module to the input of another is a list) | identified |
| user groups for execution, file, and configuration permissions | identified |
| dynamically configured permissions for server modules | identified |
| processing statistics gathering | identified |
| execution estimation | identified |
| control modules (e.g. loops, multiplexors, breakpoints) | identified |
| self updating capabilities | identified |
| nested list processing | identified |
| quality assurance test suite | identified |
| validation | identified |
| provenance | identified |
| Error handling |
|---|
| feature | status |
|---|
| automated file translations | identified |
| ensure disk space availability | identified |
| Logging |
|---|
| feature | status |
|---|
| capture of stderr streams | identified |
| logging to persistent files | identified |
| write ahead logging for crash recovery | identified |
| search capabilities | identified |
| User interface |
|---|
| feature | status |
|---|
| allow user comments on modules and module arguments | completed |
| single dialog window which aggregates all error messages | in progress |
| selection of a module on the gui should highlight all its connections | identified |
| simple viewer for common filetypes | identified |
| ability to select multiple nodes and apply an operator (e.g. delete) to them | identified |
| message of the day | identified |
| Documentation |
|---|
| feature | status |
|---|
| documentation | wiki established |
| module repository | wiki established |
| graduated difficulty tutorial | identified |
| specification of handshaking process in establishing network connection and encryption keys | identified |
| Miscellaneous |
|---|
| feature | status |
|---|
| explore computation as utility to be rented | identified |
|
|
Got more items on your wish list? Add them here!
|
|
<
< |
Based upon the design meeting on 24 september 2004, we have identified features which we would like to implement for future versions of the software.
|
>
> |
We have identified features which we would like to implement for future versions of the software. Text in red indicates the developmental status of each item.
|
|
- ability to select multiple nodes and apply an operator (e.g. delete) to them
|
|
<
< |
- notification of process completion (via system to applications, and email to users)
- submission of jobs from external (non-pipeline) applications
|
>
> |
- notification of process completion (via system to applications, and email to users) see item 3
- submission of jobs from external (non-pipeline) applications communications via tcp complete. implementation of client library, and documentation of handshaking specification in progress
|
|
- single dialog window which aggregates all error messages
- validation
- dynamically configurable permissions for server modules
|
|
- breakpoints
- better error handling
- better (and more sophisticated) gui for editing modules (e.g. global text substitution)
|
|
<
< |
- allow user comments on modules and module arguments
- community documentation (this wiki)
- communal repository of existing modules (this wiki)
- allow users to disconnect from server but not have their pipelines terminated
|
>
> |
- allow user comments on modules and module arguments implementation of <rdfs:label/> and <rdfs:comment/> tags complete
- community documentation this wiki
- communal repository of existing modules this wiki
- allow users to disconnect from server but not have their pipelines terminated see item 3
|
|
- processing statistics gathering
- execution estimation
- runtime list processing (when output of one module is a list file, which is an input to another)
|
|
- write ahead logging
- output log to a file
- better search abilities (in text of processing log, or in pipeline itself)
|
|
<
< |
- runtime processing of temporary filenames
|
>
> |
- runtime processing of temporary filenames completed
|
|
- simple viewer for common file types (allows quick inspection of data)
- failure message should specify which module failed
- capture of stderr to logfile
|
|
<
< |
- abstraction/wrapper for various data sources
|
>
> |
- abstraction/wrapper for various data sources implementation of URL wrapper in progress
|
|
- allow user to specify execution phases (for pre- and post- processing)
- message of the day
- self updating software
|
