Development status

Alpha version released

Tagged v0.5, this release is supposed to be suitable for manual tagging, and assisted manual tagging using a prototype automatic tagger.

About LarvaTagger.jl

LarvaTagger.jl is a 2D larva viewer and discrete behavior tag editor.

The graphical user interface is an app that opens up in a javascript-enabled web browser. On starting the program, you will be instructed to access a local web server entering an address in your web browser. On some systems, the program can automatically open a new tab in your default web browser, so that the app is loaded without any extra manipulation after starting the program.

Installation

Check out the home page.

Basically, you have two options:

Using Docker

Recommended if:

you already have Docker installed and set up,
you want to give LarvaTagger.jl a try and easily delete it afterwards, altogether with the required Julia+Python stack,
you are interested in the automatic tagging feature and do not have read access to the MaggotUBA project.

From source

Recommended for:

macOS users?
- It is not clear how valid are the Docker-related usage instructions for macOS. For example, volume mounting seems to act different,
heavy users of the app.
- Julia is a just-in-time compiled language, meaning that a Julia program runs faster over time. These optimizations are persisted in a Docker container, but not in a Docker image; if you stop the container (= running image), you loose the extra binary generated at runtime.

What was tested? What was not?

Tested platforms:

Windows
Windows + Docker (from PowerShell)
WSL + Docker (not recommended because of the flushing behavior of the associated terminal window)
Linux
Linux + Docker

Platforms to be tested:

macOS
macOS + Docker
Windows + Docker (from cmd)
WSL

Manual tagging

A few tips about not-so-obvious features:

The 2D view can be panned pressing the right mouse button.
In the single larva view, a tag can be assigned over a trajectory segment in a single user action by dragging-and-dropping the larva over the desired part of the larva's trajectory.
Drag-and-drop assignments are cancelled if too short in duration, to prevent accidental assignments. The minimum duration for a drag-and-drop to be registered as a tag assignment is currently quite long (1 second).
Dragging-and-dropping a larva from a tag-free time point unassigns the previously assigned tags over the drawn trajectory segment.
In the Tracks panel, unticking the Edited checkbox for a larva undoes the changes made by the user for that larva.

Automatic tagging

LarvaTagger.jl features a generic-enough tagging backend architecture to accommodate virtually all possibilities. However, a single tagging backend is currently available.

This backend is based on the MaggotUBA (Drosophila Larva Unsupervised Behavior Analysis) project, whose code is still access restricted. An article featuring MaggotUBA is in preparation.

As a consequence, a MaggotUBA-based prototype is publicly available in the Docker image only.

In the current state, the backend can only be used to predict behavior (perform automatic tagging). It is not yet ready for retraining on arbitrary data repositories and discrete behaviors.

(cont.)

File formats

File manipulation is implemented in another Julia package.

First of all, the LarvaTagger.jl editor cannot be launched arg-free. The path to a valid data file must be passed as argument. If you need sample data files, you can download and extract those used for the automated tests.

Behavior labels

Behavior tags, be they manually or automatically assigned, are stored in json files with extension .labels. These files are human-readable.

If you want trx.mat files instead, a MatLab script is available to this aim.

.labels files should not be considered standalone files. They do not replicate the data from the original tracking data file. They only give a pointer to the corresponding tracking data file(s) and, if .labels file should be moved or copied, the associated data files should also be copied so that they can be found as siblings in a same directory.

FIMTrack v2 csv files

FIMTrack v2 generates table.csv files that contain too little information about the data to be rendered and analysed. In particular, space is encoded in pixels and time in frame numbers.

To overcome the lack of information about essential units, LarvaTagger.jl currently makes the assumption the camera framerate is 30 fps. The larvatagger.jl script, in addition to its main open command, features an import command that allows adding metadata such as the camera framerate with option --framerate. Metadata are stored in the shape of a .labels file that should be opened as a replacement for the corresponding table.csv file.

The automatic behavior tagging shows poorer performance on these data. The spines may be drawn in a significantly different fashion.

A few screenshots:

Beta version released

Tagged v0.6, this version allows training new MaggotUBA-based taggers.

Training

Although LarvaTagger.jl provides a simplified commandline interface to train new taggers (and use them for automatic tagging), most of the training logic is implemented in Python. As a consequence, the LarvaTagger.jl project can be disregarded in favor of MaggotUBA-adapter to this aim.

Below we summarize the documentation for training taggers.

Training with LarvaTagger.jl

On Unix-like OSes with Docker available, there is no need to install LarvaTagger.jl. Download the larvatagger.sh script, make it executable and run it from a parent location of your data repository:

./larvatagger.sh train <relative/path/to/your/data/repository> <tagger-name>

This script uses a Docker image and wraps the docker command to hide a number of details.

The train command proceeds in several steps:

labelled data files are discovered; the labels are counted and a balanced dataset is generated (this step is implemented in Julia);
a pretrained encoder is combined with a fully-connected layer classifier; the classifier is first pretrained, and then the combined encoder and classifier are simultaneously fine-tuned;
the resulting model is saved in a directory that, in the case of larvatagger.sh, is to be manual managed (a local models directory is created).

To apply the new tagger to a data file:

./larvatagger.sh predict <relative/path/to/your/data/file> --model-instance models/<tagger-name> --skip-make-dataset

EDIT: The above command has been corrected since the announcement.

In principle, trained taggers are stored so that the user only needs to specify a tagger's name, not a path. However, because we are using Docker, we cannot safely store data inside the container. As a consequence, we cannot let LarvaTagger.jl automatically manage the trained taggers, and instead we specify a relative path to the tagger directory (models/<tagger-name>).

A predicted.label file is generated as a sibling of the track data file. Feel free to rename it.

The above steps can be reproduced on Windows, calling docker in a PowerShell console. Again, no installation required.

With a working installation of LarvaTagger.jl, the larvatagger.jl script is to be used instead. Taggers are stored at a discoverable location and the user does not need to know about the models directory.

Training with MaggotUBA-adapter

MaggotUBA-adapter is to be installed using poetry in a local copy of the project (download or use git):

Poetry is used to hide the details of using a virtual environment. See the recommended installation steps.

As a tagging backend, MaggotUBA-adapter features the tagging-backend command from the TaggingBackends project. This command can be called from MaggotUBA-adapter root directory as follows:

poetry run tagging-backend train --model-instance <tagger-name>

Prior to running the above command, the data repository to be used for training must be made available in the data/raw/<tagger-name> directory (to be created).

The resulting model files are stored in models/<tagger-name>.

To apply the new tagger to a data file, clear the data/raw/<tagger-name> directory, put the data file in there, and run:

poetry run tagging-backend predict --model-instance <tagger-name> --skip-make-dataset

The resulting predicted.label file can be found in the data/processed/<tagger-name> directory.

Training using the TaggingBackends API

See the example notebook.

To reproduce the example notebook, appart from getting specific data files, the train_model function should be called with argument subsets=(.8, .2, 0), to split the dataset file into 80% training data, 20% validation data (and no test data, since test data are provided by different dataset files). The default value for this subsets argument has changed since then. (EDIT: notebook updated)

Prediction

First, maggotUBA-based taggers now properly handle time. As a consequence, they perform equally well on FIMTrack data for example, and more generally on data acquired at different frame rates.

The reference 20221005 tagger exhibits 85-86% agreement with JBM's tagger on a balanced t5+t15 dataset. Bends and rolls show the poorest agreement.

We would like interested people to try out the tagger on manually tagged (or retagged) data to determine whether this difference corresponds to poorer overall performance or, on the contrary, to an interesting and possibly better alternative.

The ability to train new taggers may lead to better alternatives, but the performance can only be properly evaluated on manually-tagged data.

There is also room for improvement such as issue #62. Please file issues to discuss other ideas.

locked this issue

0.7 series and the beta3 release

The v0.7 (beta2) and v0.7.1 (beta3) releases bring major improvements to the GUI, especially in terms of speed and reactivity. As a consequence, LarvaTagger.jl now performs well on Chrome and Chromium-based web browsers.

Other improvements are:

the Play function emulates real time;
the middle or right mouse buttons can be used to cancel a drag-n-drop tag assignment.

The larvatagger.jl and larvatagger.sh scripts admit a --version argument that prints LarvaTagger.jl's version number, useful for bug reports. The larvatagger.sh script that wraps the Docker image additionally features a --update argument to pull the latest larvatagger image from Docker Hub.

UI/UX improvements were made following bug reports and suggestions by Ben Jones and Peter Hague. A big thank you to both of them.

The 0.11 series: building up ground-truth data

After the introduction of tagging annotations (e.g. "edited") in the form of secondary labels, with version 0.10, the v0.11.1 release features a merge command to combine label files so that the track steps marked as manually edited in a label file can be copied into another file.

This is expected to be useful for incrementally refining a training dataset, maintaining a ground-truth label file for each track data file (for each assay/run), and updating this file everytime the predicted behavior for some track step is to be corrected.

The intended workflow is as follows:

make a clean repository with the original track data files and associated ground truth label files (no other label files there)
train a tagger on the clean repository (command train)
generate a predicted.label file using the trained tagger (command predict)
open the predicted.label file in the GUI and save your manual modifications if any (command open);
(a) if you don't have a ground truth file yet, turn the resulting label file into that file
(b) otherwise merge the latest label file into the ground truth file with:
larvatagger.sh|jl merge <path/to>/groundtruth.label <latestfile.label> groundtruth.label
once all data files are inspected, loop back to step 1.

Steps 3, 4 and 5 refer to a single track data file (or subdirectory) in the main data repository, and are to be performed on each data file, while steps 1, 2 and 6 are to be done once per loop iteration.

Here, a unique file name is used for the ground truth label files. In the future, the above process may be facilitated with more automation, which most likely may rely on such a naming convention.

mentioned in issue #95 (closed)

mentioned in issue #114 (closed)

With releases 0.15 and 0.15.1, the above workflow can be simplified.

Step 1. can be skipped, passing a file filter as training data, e.g. /path/to/repository/**/groundtruth.label.

Step 5. can also be skipped if step 4. saves to file groundtruth.label. If the file exists, the GUI allows the user to perform the merge operation at 5.(b) with a simple mouse click.

LarvaTagger's paper is out! If you use LarvaTagger with the MaggotUBA-adapter tagging backend, check out the supplementary sections.

LarvaTagger.jl now supports a second official backend: PasteurJanelia-adapter, based on the tagger Jean-Baptiste Masson developped at Janelia and modified at Institut Pasteur!

A Docker image is available with both MaggotUBA-adapter and PasteurJanelia-adapter: flaur/larvatagger:0.18.5-bigfat. Note this first image is not tagged as latest. It is a release candidate for PasteurJanelia-adapter. In addition, the image file is substantially larger than the MaggotUBA-only images.