MaggotUBA backend adapter

Wrapper project to allow the Nyx tagger UI to call MaggotUBA.

This project heavily depends on the TaggingBackends package that drives the development of automatic tagging backends.

Principle

MaggotUBA is an autoencoder trained on randomly sampled 20-time-step time segments drawn from the t5 and t15 databases (from t15 only for current default), with a computational budget of 1,000 training epochs (10,000 for current default). In its original "unsupervised" or self-supervised form, it reconstructs series of spines from a compressed latent representation.

For the automatic tagging, the encoder is combined with a classifier. On the same dataset, the combined encoder-classifier are (re-)trained to predict discrete behaviors.

Taggers

6-behavior classification task

6-behavior refers to run, cast, back, hunch, roll and stop_large.

20220418

As a first prototype, the 20220418 trained model is based on a simple random forest classifier, and only the classifier was trained; the encoder was not retrained. See module maggotuba.models.randomforest.

It was trained on the entire t5+t15 database. No interpolation was performed and the prototype does not properly handle data with different frame rates.

20221005 and 20221005-1

A second tagger called 20221005 involves a classifier with dense layers, and the encoder was fine-tuned while training the combined encoder+classifier. See modules maggotuba.models.trainers and maggotuba.models.modules.

This second tagger was dimensioned following a parametric exploration for the 6-behavior classification task: 2-second time segments, 100-dimension latent space, 3 dense layers.

It was trained on a subset of 5000 files from the t5 and t15 databases. Spines were/are linearly interpolated at 10 Hz in each time segment individually.

The 20221005-1 tagger is identical. It was first thought to implement a post-prediction correction step referred to as ABA -> AAA, but actually did not.

20221228

This tagger combines 25 taggers based on a 1.5-s time window and 25 taggers based on a 3-s time window, and uses a voting approach (or bagging). All simple taggers featured 25 latent dimensions and 1 dense layer (as classifier) only, following another parametric exploration for the 6-behavior classification task.

This complex tagger was not distributed in any latest Docker image. See experimental image 0.7.2-20221228.

It runs 50 times slower and does not solve any of the issues of 20221005.

7-behavior classification task

7-behavior refers to run_large, cast_large, back_large, hunch_large, roll_large, stop_large and small_motion.

20230111

As a stronger default tagger, the small_motion was reintroduced to lower the detection rate of hunches and rolls.

The 20230111 tagger uses a 2-s time window, features 25 latent dimensions and a single dense layer as classifier. It applies a post-prediction rule referred to as ABC -> AAC that consists in correcting all single-step actions with the previous action.

20230129

Previous tagger 20230111 revealed a temporal leakage issue that might have affected all previous taggers.

A similar tagger called 20230129 has been proposed to moderate this issue. This tagger shares the same characteristics as 20230111 and differs in three important aspects:

• the number of training epochs was brought from 1,000 to 10,000 to let the original features be largely forgotten,
• the training stage involved more data: 1,200,235 time segments were used instead of 100,000; these data were unbalanced and training was performed with the newly introduced balancing strategy auto (see larvatagger.jl#92),
• pretraining and training data were drawn from t15 only (as opposed to previous taggers that were pretrained and trained with data from t15 and t5).

Note the last difference was not meant to improve performance, at all. The 20230129 was trained this way to study its performance on t5, and was kept as is after it showed better performance on t5 data, compared to previous taggers trained with t5 data in addition to t15 data.

Usage

For installation, see TaggingBackends' README.

A MaggotUBA-based tagger is typically called using the poetry run tagging-backend command from the root directory of the backend's project.

All the command arguments supported by TaggingBackends are also supported by MaggotUBA-adapter.

Automatic tagging

Using the 20230129 branch, the 20230129 tagger can be called on a supported tracking data file with:

poetry run tagging-backend predict --model-instance 20230129

Note: since TaggingBackends==0.10, the --skip-make-dataset argument is default behavior. Pass --make-dataset instead to enforce the former default.

For the above command to work, the track data file must be placed (e.g. copied) in the data/raw/20230129 directory, to be first created or cleared.

The resulting label file can be found as data/processed/20230129/predicted.label. Like all .label files, this file should be stored as a sibling of the corresponding track data file (in the same directory).

Similarly, with an arbitrary tagger named, say mytagger, in the above explanation all occurences of 20230129 or 20230129 must be replaced by the tagger's name. For example, the input data file would go into data/raw/mytagger.

On HPC clusters

Simultaneously tagging multiple tracking data files result in file conflicts because the same data directories are used internally. Use the larvatagger.jl script of the LarvaTagger.jl project instead, with argument --data-isolation.

Retraining a tagger

A new model instance can be trained on a data repository with:

poetry run tagging-backend train --model-instance <tagger-name>

Similarly to the predict command, for this one to work, the data repository must be made available in the data/raw/<tagger-name> directory (again, to be created or cleared).

The above command will first load a pretrained model (pretrained_models/default in MaggotUBA-adapter) to determine additional parameters, such as whether to interpolate the spines or not and at which frequency, or the window length.

Beware that the default pretrained model may depend on the branch you are on.

The default pretrained model in the 20221005 branch involves linearly interpolating the spines at 10 Hz, and relies on a 20-time-step window (2 seconds). The dimensionality of the latent space is 100.

The default pretrained models in the 20230111 and 20230129 branches similarly interpolate spines at 10 Hz and rely on a 20-time-step window (2 seconds), but feature 25 latent dimensions only.

Alternative pretrained models can be specified using the --pretrained-model-instance option.

The data files are discovered in the repository (more exactly in data/raw/<tagger-name>) and behavior tags are counted. A subset of tags can be specified using the --labels option followed by a list of comma-separated tags.

A two-level balancing rule is followed to randomly select time segments and thus form a training dataset in the shape of a larva_dataset hdf5 file. See also the make_dataset.py script.

Training operates in two steps, first pretraining the dense-layer classifier, second simultaneously fine-tuning the encoder and classifier. See also the train_model.py script.

This generates a new sub-directory in the models directory of the MaggotUBA-adapter project, which makes the trained model discoverable for automatic tagging (predict command).