LarvaTagger

LarvaTagger allows the manual and automatic tagging of actions or postures of behaving Drosophila larvae.

So far, it has been used to study Drosophila melanogaster third instar larvae only.

The LarvaTagger project is divided into several components. Although this README file serves as home page for the entire project, the present code repository hosts one of LarvaTagger's components, called LarvaTagger.jl.

LarvaTagger.jl

LarvaTagger.jl is a graphical user application for the manual and automatic tagging of actions or postures of behaving Drosophila larvae.

This package features a web-based graphical user interface (GUI) for visualizing the tracked larvae and assigning discrete behavior tags, at each time step.

A command-line interface (CLI) is also available for batch processing, including the automatic tagging of track data files, training new taggers from labeled data, etc.

Although LarvaTagger.jl alone comes with no automatic tagger, it is designed to work in combination with MaggotUBA for the identification of larval actions or postures.

The term LarvaTagger refers to the full software with all the currently available tagging capabilities included.

Docker images are available for LarvaTagger. See the dedicated instructions page or the Quick start section below.

Support for various data file formats is implemented by PlanarLarvae.jl. Behavior labels from manual and automatic tagging are saved in JSON files along with the corresponding track data files. These JSON files are text files and can be conveniently inspected with a text editor.

Quick start with Docker

The present section will guide you through the following steps:

• “install” the complete LarvaTagger package,
• open a tracking data file in LarvaTagger.jl's viewer,
• “uninstall” LarvaTagger.

These instructions only require Docker to be available, more specifically the Docker Engine. On Windows and macOS, Docker Engine is provided by the Docker Desktop product. Docker Desktop is free of charge.

Ensure the currently logged-in user has access to Docker, and the Docker engine is running.

The Docker image is obtained and operated using a script:

• larvatagger.bat on Windows,
• larvatagger.sh on macOS and Linux.

Download the script and, optionally, some demo data to be unzipped (note: a gzipped tar file is also available).

Open a terminal window. On Windows, this can be done typing cmd in the finder.

Change directory (cd) to the Downloads directory. In this example, we will assume the larvatagger script and unzipped demo data files are available in that directory.

On macOS and Linux, change the permissions of the script file so that it can be executed:

chmod a+x larvatagger.sh

The demo data can be opened in the web browser for visual inspection, on macOS and Linux with:

./larvatagger.sh open Masson_et_al_2020.label --browser

and on Windows with:

larvatagger.bat open Masson_et_al_2020.label --browser

The viewer should open in a new tab in the default web browser. If no tabs appear, open a new one and go to http://127.0.0.1:9284.

Not all the web browsers are supported. Prefer Firefox or Chrome-like browsers.

Note that, on the first execution, the Docker image will be downloaded. This is done once, and can be considered as the installation step.

Docker stores the image locally. Uninstalling LarvaTagger simply consists in removing the image from the local Docker's store. Docker Desktop users can open Docker Desktop and delete the image listed as flaur/larvatagger:latest.

Main controls

The tracked larvae can be animated pressing the "Play/Pause" button of the player below the 2D view.

From the view displaying multiple larvae, a larva can be selected. As a result, the 2D view displays the selected larva only. From this single-larva view, the top right button allows going back to the multi-larva view.

The 2D view(s) can be panned holding the mouse right button. The mouse wheel allows zooming in and out.

In the single-larva view, action tags can be assigned at each time step selecting the desired tag in the horizontal list below the player. Alternatively, once the desired tag has been selected for the active time step, the same tag can be assigned at all the time steps in a time segment starting from or ending at that initial time step. This is done holding the mouse left button and moving the mouse pointer along the track until reaching the other end of the desired time segment. A drag-and-drop effect is displayed.

Important: Do NOT release the mouse left button unless you are sure! As long as the left button is held, the assignment operation can be cancelled pressing the mouse middle or right button (before releasing the left button).

In the event an assignment operation needs to be undone, the only undo mechanism available for now undoes all the changes made on a larva, and can be accessed in the Tracks panel on the right, unchecking the Edited checkbox (middle column) corresponding to the larva.

Alternatively, tags can be similarly unassigned at individual time steps or over time segments. For example, the tag at the active time step can be unselected in the tag list below the player, and then the no tag condition can be “assigned” with the very same drag-and-drop mechanism as previously described.

Installing

Using the Docker image may come handy, as it ships LarvaTagger with the default MaggotUBA-based tagger as a standalone package and is easier to update.

Using the scripts/install.sh script

On macOS, Linux and WSL, a script is provided to automate the installation process from source. This script installs LarvaTagger.jl and Julia if missing. It also adds the larvatagger command to the user's path variable. The scripts/install.sh script can be run with:

/bin/bash -c "$(curl -sSL "https://gitlab.pasteur.fr/nyx/larvatagger.jl/-/raw/dev/scripts/install.sh?ref_type=heads&inline=false")"

To install the full LarvaTagger suite (Linux and WSL only!), run instead:

curl -sSL "https://gitlab.pasteur.fr/nyx/larvatagger.jl/-/raw/dev/scripts/install.sh?ref_type=heads&inline=false" | /bin/bash -s -- --with-default-backend

In the latter case, the script may install several extra dependencies, but not all of them. In particular, Python is required; either 3.8 with --with-default-backend, or 3.11 with --with-backend --experimental. If pyenv is available, the script will use this tool to install Python. Otherwise, python3.8 and python3.8-venv may have to be manually installed. On WSL, the script will attempt to install pyenv and Python (tested with Ubuntu 20.04).

This script can also uninstall LarvaTagger (if installed with the same script) with: curl -sSL "https://gitlab.pasteur.fr/nyx/larvatagger.jl/-/raw/dev/scripts/install.sh?ref_type=heads&inline=false" | /bin/bash -s -- --uninstall which can be useful for example prior to reinstalling after failure.

Manually from source

To natively run LarvaTagger.jl instead, you will need julia>=1.7.

If you are not familiar with installing Julia, you may appreciate installation helpers such as Juliaup or jill.

git clone https://gitlab.pasteur.fr/nyx/larvatagger.jl LarvaTagger
cd LarvaTagger
julia --project=. -e 'using Pkg; Pkg.instantiate()'

In May 2024, the gitlab.pasteur.fr server began to request an authentication token on cloning public repositories. If the git clone command requests an authentication token you do not have, do instead:

curl -sSL https://gitlab.pasteur.fr/nyx/larvatagger.jl/-/archive/main/larvatagger.jl-main.tar.gz | tar zxv && mv larvatagger.jl-main LarvaTagger

Calling Pkg.instantiate in a copy of the project is preferred over using Pkg.add, because LarvaTagger.jl depends on several unregistered packages.

Alternative procedure

Users who would prefer not to clone the repository or implicitly use the shipped Manifest.toml file can follow the following steps instead:

mkdir LarvaTagger && cd $_ && \
julia --project=. -e 'using Pkg; Pkg.add([ \
  PackageSpec(url="https://gitlab.com/dbc-nyx/ObservationPolicies.jl"), \
  PackageSpec(url="https://gitlab.com/dbc-nyx/TidyObservables.jl"), \
  PackageSpec(url="https://gitlab.pasteur.fr/nyx/PlanarLarvae.jl"), \
  PackageSpec(url="https://gitlab.pasteur.fr/nyx/LarvaTagger.jl")])'

The --project=. option will make the newly created LarvaTagger directory a new Julia project with a dedicated Manifest.toml file.

Upgrading

Using the Docker image, run scripts/larvatagger.sh --update.

Using the scripts/install.sh script, upgrading is done uninstalling and installing LarvaTagger again.

Using a local copy of the git repository, go to that repository, be sure to be on the appropriate branch (in most cases the main branch) and update (git pull) it:

git checkout main
git pull

If the alternative installation procedure was followed instead, go to the LarvaTagger directory and type:

julia --project=. -e 'using Pkg; Pkg.update()'

Launching the graphical user interface

The GUI is provided by a web server and can be accessed using a web browser, preferably Firefox or Chrome-like browsers.

Running LarvaTagger.jl sets the server up. Once the server is ready, the app can be loaded in a web browser at the address indicated, typically http://localhost:9284.

See the Quick start section for information about some of the controls available in the GUI.

Using the larvatagger script

If you cloned the repository, we recommend you run LarvaTagger using the larvatagger script to be found in the scripts directory:

scripts/larvatagger open path/to/data/file --browser

If LarvaTagger was installed using the scripts/install.sh script, the larvatagger script should be in the user's path environment variable and, as a consequence, available from everywhere in the commandline:

larvatagger open path/to/data/file --browser

The script will actually open a Julia interpreter, and give some guidance on how to exit the interpreter.

For now, larvatagger open cannot be run with no input arguments. A track data file is required.

The --browser argument may open a new tab in your web browser, but this feature is known to be ineffective in some situations. In such an event, open a new tab and go to http://localhost:9284.

The first time the application is loaded, it may take a while for a window in your web browser to open, and the data to be plotted.

From the Julia interpreter

As an alternative to the larvatagger script, in the LarvaTagger directory created above, launch the Julia interpreter:

julia --project=.

In the interpreter, to launch the editor, type:

julia> using LarvaTagger; display(larvaeditor("path/to/data/file"))

To exit the interpreter, type exit() or press Ctrl+D.

macOS

On macOS computers, the 2D larva view may show up twice as small as expected. To mitigate this undesired behavior, larvatagger open admits a --view-factor option, and equivalently larvaeditor admits a viewfactor argument. This option/argument is 2 per default on macOS, 1 on the other platforms. Feel free to adjust the value if the 2D view is too small or large.

Automatic tagging

LarvaTagger.jl comes with no automatic tagger per default, unless installed with the scripts/install.sh script and the --with-default-backend option.

To extend the editor with MaggotUBA-based automatic tagging, see the recommended installation steps for TaggingBackends and MaggotUBA.

Strictly speaking, the action identification package (or tagging backend) is called MaggotUBA-adapter. It is based on MaggotUBA, which is an autoencoder for action-agnostic behavior analysis. The terms MaggotUBA and MaggotUBA-adapter are used interchangeably in the present documentation.

If the JULIA_PROJECT environment variable was used to control the location of the TaggingBackends Julia package, this variable should be equally set prior to carrying out any of the below steps.

If the backend directory is created right in the LarvaTagger directory, LarvaTagger.jl will automatically find it.

Otherwise, to let the larvaeditor function know about MaggotUBA or any other backend, in the Julia interpreter, type:

julia> using LarvaTagger; display(larvaeditor("path/to/data/file"; backend_directory="path/to/MaggotUBA/parent/directory"))

Similarly, to let LarvaTagger know about MaggotUBA:

scripts/larvatagger open <path/to/data/file> --backends=<path/to/MaggotUBA/parent/directory> --browser

The larvatagger script can also be used to train a new tagger:

scripts/larvatagger train <path/to/backend> <path/to/data/repository> <tagger-name>

and apply this tagger to a tracking data file:

scripts/larvatagger predict <path/to/backend> <tagger-name> <path/to/data/file>

Among the many optional arguments to the train command, an important one is --iterations. It allows specifying the training budget. In several applications, higher training scores were achieved increasing the value for this argument. The default for MaggotUBA-adapter tagging backend is 1000, which may be too few iterations in many cases.

MaggotUBA-adapter admits either a single value or a comma-separated pair of values. Indeed, MaggotUBA-adapter training is performed in two phases: first the classifier stage is trained, with static weights in the pretrained MaggotUBA encoder; second both the classifier and encoder are fine-tuned. A higher training budget for the second fine-tuning stage may significantly increase the training accuracy.

Note: since TaggingBackends==0.10, argument --skip-make-dataset is default behavior; pass --make-dataset instead to enforce the former default.

To run larvatagger predict in parallel on multiple data files using the same tagger, append the --data-isolation argument to avoid data conflicts. Indeed, without this latter argument, a tagger accesses dedicated locations in the filesystem and these locations would be shared by all the simultaneously running larvatagger instances. --data-isolation makes the tagger use temporary directories instead, so that these directories are specific to the running process, in addition to be specific to the tagger.

Since version 0.16 with TaggingBackends>=0.14, larvatagger train admits a --fine-tune option to further train an already trained tagger. This use case differs from standard training in that it loads a classifier whose output cannot be redesigned. As a consequence, the labels in the training dataset should match those used to train the tagger in the first place.

The train and predict commands admit more arguments. Check them out with:

scripts/larvatagger --help

The trained taggers are stored in the models directory in the backend. MaggotUBA-based taggers are stored in the shape of PT and JSON files. After training a tagger, post-processing rules can be manually added into the clf_config.json file. See the following comment.

Developer documentation

See the release announcements, change logs and project structure.

Known issues

Data from remote stores

In the graphical user interface, viewing data files loaded from a network mount may cause the player to exhibit erratic behavior, jumping to/from past and future time steps, and refusing to stop.

In addition, loading such a data file may take LarvaTagger.jl a lot more time than transferring the file locally and opening the local copy instead.

Ensure the data can be loaded without latencies or slow downs.

CUDA errors

On calling larvatagger predict using a MaggotUBA-based tagger, if CUDA complains with messages such as:

.../torch/cuda/__init__.py:... Can't initialize NVML

you might have upgraded your NVIDIA drivers and not rebooted the OS. Try restarting your computer.