diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..8144b3b1e5fb9363d310b2acab6bf828abe96ffa --- /dev/null +++ b/README.md @@ -0,0 +1,133 @@ +# ISBI SPT challenge performance metrics measurements. + +This repository contains Java code to measure the tracking performance of a tracking algorithm, using the ISBI Challenge on Single-Particle Tracking metrics. + +See the paper: + +> Chenouard, *et al*., (**2014**). *Objective comparison of particle tracking methods*. Nature Methods, 11(3), 281–289. https://doi.org/10.1038/nmeth.2808 + +I adapted and copied code from by N. Chenouard, F. De Chaumont and S. Dallongeville, initially part of the Icy software (http://icy.bioimageanalysis.org/) so that it can run in batch and without Icy. So they should be considered authors of this code as well. The original Icy code is linked and acknowledged in the files of this repository. + +Because running the metrics can take very long when the particles are dense, I made this translation of the existing Icy code so that it can run in batch in a multithreaded manner. + +The main class that serves as a runner is `ISBIScoring` in the `fr.pasteur.iah.trackingperformance` package. + +## Running the performance measurement. + +### Main command. + +If you want to run a performance measurement in batch, you can compile this code with maven, then run the following command from the folder containing the compiled jar: + +```sh +$ java -cp TrackingPerformance-0.0.1-SNAPSHOT.jar fr.pasteur.iah.trackingperformance.ISBIScoring "/path/to/ground/truth/tracks" "/path/to/candidate/tracks" +``` + +If you want to run performance assessment only over a subset of scenarios, specify their names as third argument: + +```sh +$ java -cp TrackingPerformance-0.0.1-SNAPSHOT.jar fr.pasteur.iah.trackingperformance.ISBIScoring "/path/to/ground/truth/tracks" "/path/to/candidate/tracks" "MICROTUBULE" +``` + +Otherwise the program will run on all the scenarios it will find. + +### Organization of the ground-truth folder. + +The `"/path/to/ground/truth/tracks"` folder needs to be arranged as follow with one folder per scenario, spelled as in the paper: + +```sh +$ ls + +MICROTUBULE/ RECEPTOR/ VESICLE/ +``` + +and the ground truth XML files should be in each scenario folder: + +```sh +$ ls MICROTUBULE/ +'MICROTUBULE snr 1 density high.xml' 'MICROTUBULE snr 4 density high.xml' +'MICROTUBULE snr 1 density low.xml' 'MICROTUBULE snr 4 density low.xml' +'MICROTUBULE snr 1 density mid.xml' 'MICROTUBULE snr 4 density mid.xml' +'MICROTUBULE snr 2 density high.xml' 'MICROTUBULE snr 7 density high.xml' +'MICROTUBULE snr 2 density low.xml' 'MICROTUBULE snr 7 density low.xml' +'MICROTUBULE snr 2 density mid.xml' 'MICROTUBULE snr 7 density mid.xml' +``` + +Each of these files contain the ground truth tracks, that can be obtained from the ISB SPT challenge website: + +- [VIRUS scenario](http://bioimageanalysis.org/track/bench/virus.zip) +- [VESICLE scenario](http://bioimageanalysis.org/track/bench/vesicle.zip) +- [RECEPTOR scenario](http://bioimageanalysis.org/track/bench/receptor.zip) +- [MICROTUBULE scenario](http://bioimageanalysis.org/track/bench/microtubule.zip) + +The look like this: + +```xml +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<root> +<TrackContestISBI2012 SNR="7" density="low" generationDateTime="Mon Feb 13 18:47:50 CET 2012" info="http://bioimageanalysis.org/track/" scenario="MICROTUBULE"> +<particle> +<detection t="0" x="244.883" y="38.532" z="0"/> +<detection t="1" x="249.978" y="41.272" z="0"/> +<detection t="2" x="254.58" y="43.64" z="0"/> +<detection t="3" x="259.498" y="47.254" z="0"/> +</particle> + ... +``` + +### Organization of the candidate folder. + +The `"/path/to/candidate/tracks"` needs to be arranged roughly in the same way, except that it allows for having many candidate files for one ground-truth. This allows running parameter sweep looking for optimal parameter set. + +The root of the candidate folder should contain the scenario folder: + +```sh +$ ls +MICROTUBULE/ RECEPTOR/ VESICLE/ +``` + +Inside each scenario folder, the should be **one folder** per matching ground truth file: + +```sh +$ ls MICROTUBULE +'MICROTUBULE snr 4 density high'/ 'MICROTUBULE snr 1 density high'/ +'MICROTUBULE snr 4 density low'/ 'MICROTUBULE snr 1 density low'/ +'MICROTUBULE snr 4 density mid'/ 'MICROTUBULE snr 1 density mid'/ +'MICROTUBULE snr 7 density high'/ 'MICROTUBULE snr 2 density high'/ +'MICROTUBULE snr 7 density low'/ 'MICROTUBULE snr 2 density low'/ +'MICROTUBULE snr 7 density mid'/ 'MICROTUBULE snr 2 density mid'/ +``` + +and in these subfolders, all the candidates you want to test: + +```sh +$ ls MICROTUBULE/MICROTUBULE\ snr\ 7\ density\ low +'MICROTUBULE snr 7 density low-KT-d18-g1.xml' +'MICROTUBULE snr 7 density low-KT-d18-g2.xml' +'MICROTUBULE snr 7 density low-KT-d8-g1.xml' +'MICROTUBULE snr 7 density low-KT-d8-g2.xml' +'MICROTUBULE snr 7 density low-LAP-d18-g1.xml' +'MICROTUBULE snr 7 density low-LAP-d18-g2.xml' +'MICROTUBULE snr 7 density low-LAP-d8-g1.xml' +'MICROTUBULE snr 7 density low-LAP-d8-g2.xml' +``` + +## Output of the program. + +The program will output a CSV file, one per ground truth file, that contains the ISBI metrics. The results are arranged with one line per candidate file: + +```sh +$ ls MICROTUBULE/ +'MICROTUBULE snr 4 density high'/ +'MICROTUBULE snr 4 density high.csv' +'MICROTUBULE snr 1 density high'/ +'MICROTUBULE snr 1 density high.csv' +... +``` + +and in the CSV file: + +| name | alpha | beta | rmse | detectionsJaccard | tracksJaccard | +| ------------------------------------------ | -------- | -------- | -------- | ----------------- | ------------- | +| MICROTUBULE snr 7 density low-KT-d8-g1.xml | 0.855058 | 0.850296 | 0.515457 | 0.937079 | 0.971474 | +| ... | | | | | | +