@@ -35,7 +35,7 @@ A step is a processing stage. It takes an input 0 to virtually unlimited inputs
A pipe is a collection of Steps. In real life situations, it is very often that the same data structure can be having several Steps that add some data to it. For example, imaging data can be obtained, and segmented into tables of neurons fluorescence over time. As the execution goes further in the processing pipeline, some of the processing steps may calculate the responsivness of the neurons to one or another type of stimulation, obtained in another step. It then seems logical to only add this responsiveness information to the tables data, to not overcrowd the disk with duplicated stages with increasingly detailed/further processed data. The pipe serves this purpose. When a given ``Step`` requires the output of another ``Step``, it looks up wich ``Pipe`` (e.g. general data structure) it is attached to, and loads the most advanced one available in that ``Pipe``. As such, it means a ``Step`` output at level 6 of a ``Pipe``, has to also be valid for the data that it holds, to a ``Step`` that requires the level 1 of this ``Pipe``. (meaning : as a rule of thumb, don't delete data from a ``Pipe`` output with increasing steps, but only add new fields/columns to it.)
Below, is the example of a full ``Pipeline`` graph :
On the same column, the dots represents the different Steps of a ``Pipe``. A ``Pipe`` is then a single column on this graph. Links between steps represent the dependancies between them.
...
...
@@ -47,13 +47,16 @@ The most usefull parts that this package allows you do to is :
To implement a Pipeline, you have to define at least a Step, attached inside a Pipe.
Here is a simple example.
You can test this example, and see the result on your own computer. (you require the two csv files located in the ``test/data`` folder of this repository)
You can test this example, and see the result on your own computer. (Two csv files located in the ``test/data`` folder of this repository will be used for demonstration purposes.)