diff --git a/Project.toml b/Project.toml index 1065fe0451fa0df87b736929708f1fe57df28e07..1b5dd3db1655b9b9c4b897d22d63728e669243f6 100644 --- a/Project.toml +++ b/Project.toml @@ -8,7 +8,6 @@ Colors = "5ae59095-9a9b-59fe-a467-6f913c188581" Dates = "ade2ca70-3891-5945-98fb-dc099432e06a" DocOpt = "968ba79b-81e4-546f-ab3a-2eecfa62a9db" Formatting = "59287772-0a20-5a39-b81b-1366585eb4c0" -JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1" JSServe = "824d6782-a2ef-11e9-3a09-e5662e0c26f9" LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" Logging = "56ddb016-857b-54e1-b83d-db4d58db5568" diff --git a/README.md b/README.md index bffde0751334148e9e498286d01485fa2c6dc03e..0703838e7f34f6024b247a034bead3cd014fd3ff 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,11 @@ This package features a GUI for visualizing behaving larvae and editing discrete behavior tags at each time step. -To run `LarvaTagger.jl`, you will need [`julia>=1.6`](https://julialang.org/downloads/) and some input data file, *e.g.* a `trx.mat` file or a pair of `.spine`/`.outline` files, preferably located in a repository with directory structure: `<screen_name>/<genotype@effector>/<protocol>/<date_time>`. +To natively run `LarvaTagger.jl`, you will need [`julia>=1.6`](https://julialang.org/downloads/) and some input data file, *e.g.* a `trx.mat` file or a pair of `.spine`/`.outline` files, preferably located in a repository with directory structure: `<screen_name>/<genotype@effector>/<protocol>/<date_time>`. -If you are not familiar with Julia, you may appreciate installation helpers such as [JILL.py](https://github.com/johnnychen94/jill.py). -An example install procedure can be found in [`scripts/make_demontrator.sh`](https://gitlab.pasteur.fr/nyx/larvatagger.jl/-/blob/main/scripts/make_demonstrator.sh). +If you are not familiar with installing Julia, you may appreciate installation helpers such as [JILL.py](https://github.com/johnnychen94/jill.py). + +Alternatively, if you have Docker installed, take a look at the [dedicated instructions page](https://gitlab.pasteur.fr/nyx/larvatagger.jl/-/tree/docker/recipes). ## Install @@ -17,6 +18,23 @@ cd LarvaTagger julia --project=. -e 'using Pkg; Pkg.instantiate()' ``` +Calling `instantiate` in a copy of the project is preferred to using `Pkg.add`, +because `LarvaTagger.jl` depends on several unregistered packages. + +Users who would prefer not to clone the repository or implicitly use the shipped `Manifest.toml` file +can follow the following steps instead: + +``` +mkdir LarvaTagger +cd LarvaTagger +julia --project=. -q +julia> +(LarvaTagger) pkg> add https://gitlab.pasteur.fr/nyx/PlanarLarvae.jl +(LarvaTagger) pkg> add https://gitlab.com/dbc-nyx/ObservationPolicies.jl +(LarvaTagger) pkg> add https://gitlab.com/dbc-nyx/TidyObservables.jl +(LarvaTagger) pkg> add https://gitlab.pasteur.fr/nyx/LarvaTagger.jl +``` + ## Run ### Using the Julia interpreter @@ -27,23 +45,23 @@ julia --project=. ``` In the interpreter, to launch the editor, type: ``` -julia> using LarvaTagger; larvaeditor("path/to/data/file") +julia> using LarvaTagger; display(larvaeditor("path/to/data/file")) ``` The first time the editor is loaded, it may take a while for a window in your webbrowser to open, and the data to be plotted. -The Firefox web browser showed better performance than Chrome-like browsers. - To exit the interpreter, type `exit()` or press Ctrl+D. ### Using the `larvatagger.jl` script -The `LarvaTagger/scripts` directory contains a `larvatagger.jl` executable file. -To launch the editor, type: +As an alternative to explicitly launching a Julia interpreter and typing some Julia code, if you cloned the repository, you can run the `larvatagger.jl` script instead: + ``` -scripts/larvatagger.jl open path/to/data/file +scripts/larvatagger.jl open path/to/data/file --browser ``` +The script will also launch a Julia interpreter, and give extra guidance on *e.g.* how to exit the interpreter. + ## Automatic tagging To extend the editor with `MaggotUBA` automatic tagging: @@ -53,16 +71,20 @@ cd MaggotUBA poetry install -vvv cd .. ``` -To let *larvaeditor* know about MaggotUBA, in the Julia interpreter, type: + +If the backend directory is created right in the `LarvaTagger` directory, `LarvaTagger.jl` will automatically find it. + +Otherwise, to let *larvaeditor* know about MaggotUBA or any other backend, in the Julia interpreter, type: ``` -julia> using LarvaTagger; larvaeditor("path/to/data/file"; backend_directory="path/to/MaggotUBA's/parent/directory") +julia> using LarvaTagger; display(larvaeditor("path/to/data/file"; backend_directory="path/to/MaggotUBA's/parent/directory")) ``` + Similarly, to let *larvatagger.jl* know about MaggotUBA: ``` -scripts/larvatagger.jl open "path/to/data/file" --backends="path/to/MaggotUBA's/parent/directory" +scripts/larvatagger.jl open "path/to/data/file" --backends="path/to/MaggotUBA's/parent/directory" --browser ``` -Note however that [`MaggotUBA-adapter`](https://gitlab.pasteur.fr/nyx/MaggotUBA-adapter) requires the actual [`MaggotUBA`](https://gitlab.pasteur.fr/les-larves/structured-temporal-convolution/-/tree/dev-branch) code that is not open access yet. +Note however that [`MaggotUBA-adapter`](https://gitlab.pasteur.fr/nyx/MaggotUBA-adapter) requires the actual [`MaggotUBA`](https://gitlab.pasteur.fr/les-larves/structured-temporal-convolution/-/tree/light-stable-for-tagging) code that is not open access yet. As a consequence, it is likely the installation step fails. The `MaggotUBA` tagging core will be fully released in the near future. diff --git a/recipes/Dockerfile b/recipes/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..04b44f00361cba35779325a036d474c059411742 --- /dev/null +++ b/recipes/Dockerfile @@ -0,0 +1,39 @@ +FROM julia:1.7.3-bullseye + +ENV DIRNAME app +ARG BRANCH=main +RUN apt-get update && apt-get install -y \ + git \ + && rm -rf /var/lib/apt/lists/* \ + && git clone --depth 1 --single-branch -b $BRANCH https://gitlab.pasteur.fr/nyx/larvatagger.jl $DIRNAME \ + && julia --project=$DIRNAME -e 'using Pkg; Pkg.instantiate()' \ + && ln -s /$DIRNAME/scripts/larvatagger.jl /bin + +# COPY requires at least one file defined, hence `test/precompile.jl` +COPY test/precompile.jl test/data_sample* $DIRNAME/test/ + +ARG TIMEZONE=UTC +RUN ln -snf /usr/share/zoneinfo/$TIMEZONE /etc/localtime \ + && echo $TIMEZONE > /etc/timezone \ + && $DIRNAME/test/precompile.sh --shallow \ + && mv larvatagger.so /lib/ \ + && rm -f $DIRNAME/test/data_sample* + +COPY external $DIRNAME/ + +ARG BACKEND +RUN if [ "$BACKEND" = "MaggotUBA/20220418" ]; then \ + apt-get update && apt-get install -y \ + python3-pip \ + && python3 -m pip install poetry \ + && cd $DIRNAME \ + && git clone --depth 1 --single-branch -b 20220418 https://gitlab.pasteur.fr/nyx/MaggotUBA-adapter MaggotUBA \ + && cd MaggotUBA \ + && python3 -m poetry remove structured-temporal-convolution \ + && python3 -m poetry add ../structured-temporal-convolution \ + && python3 -m poetry install \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/*; \ + fi + +ENTRYPOINT ["larvatagger.jl", "--sysimage", "/lib/larvatagger.so"] diff --git a/recipes/Dockerfile.local b/recipes/Dockerfile.local new file mode 100644 index 0000000000000000000000000000000000000000..2e44d254ecec6aba6ff3576bf68662400f7c2e2e --- /dev/null +++ b/recipes/Dockerfile.local @@ -0,0 +1,14 @@ +FROM julia:1.7.3-bullseye + +COPY src /app/src/ +COPY scripts /app/scripts/ +COPY Project.toml Manifest.toml /app/ + +RUN apt-get update && apt-get install -y \ + git \ + && rm -rf /var/lib/apt/lists/* \ + && cd /app \ + && julia --project=. -e 'using Pkg; Pkg.instantiate()' \ + ln -s /app/scripts/larvatagger.jl /bin + +ENTRYPOINT ["larvatagger.jl"] diff --git a/recipes/README.md b/recipes/README.md new file mode 100644 index 0000000000000000000000000000000000000000..eab2798d5e99490e51a91ae07aa8bcef696b1d8c --- /dev/null +++ b/recipes/README.md @@ -0,0 +1,55 @@ +# Docker images + +## Getting available images + +Images are published on [Docker Hub](https://hub.docker.com/repository/docker/flaur/larvatagger). + +They ship with `LarvaTagger.jl` and, optionally, with a MaggotUBA-based toy backend for automatic tagging. + +Try: +``` +docker pull flaur/larvatagger:0.5-20220418 +``` + +Beware that images that ship with backends are relatively large files (>6GB). + +## Running an image + +A number of options must be passed to the `docker` command for the container to open a data file: + +``` +docker -iv $(pwd):/data -p 9284:9284 larvatagger open /data/path/to/your/file +``` +with `path/to/your/file` a relative path to your file. + +In the command above, `/data` represents the local directory, made available within the container. +This is required, since no files outside the container can be accessed. + +The port *LarvaTagger.jl* listens to must also be exported with `-p 9284:9284` so that it can be accessed from outside the container. + +Just like the `scripts/larvatagger.jl` script, the docker image admits more commands, including `import`, or options such as `--help`. +For these other commands, neither `-i` nor `-p 9284:9284` are necessary. + +See also standalone script [`scripts/larvatagger.sh`](https://gitlab.pasteur.fr/nyx/larvatagger.jl/-/blob/main/scripts/larvatagger.sh) that wraps `docker` and exposes a simpler interface. For example: + +``` +scripts/larvatagger.sh open path/to/your/file +``` + +## Building an image + +The `scripts/larvatagger.sh` script features a `build` command: + +``` +scripts/larvatagger.sh build +``` + +Optionally, if you have read access to the original MaggotUBA repository, you can fetch include a toy backend with: + +``` +scripts/larvatagger.sh build --get-backend +``` + +In both cases, `larvatagger.sh` will try to run the server and UI using a sample data file expected as file `test/sample_data`. + +Pick a file in the format you expect to most frequently manipulate (either *Choreography* files, FIMTrack v2 csv files, *trx.mat* files as generated by JBM tagging pipeline, etc) and copy it at that location. diff --git a/scripts/larvatagger.jl b/scripts/larvatagger.jl index ec839916cd9f96161c3b0f5a072bac9ca78b6666..609459d89d4c15f5951674920deada1dcfe72f34 100755 --- a/scripts/larvatagger.jl +++ b/scripts/larvatagger.jl @@ -1,97 +1,14 @@ #!/bin/bash #= PROJECT_DIR=$(dirname $(dirname $(realpath "${BASH_SOURCE[0]}"))) -if [ "$1" == "open" ]; then FLAGS="-iq "; fi +FLAGS= +if [ "$1" == "--sysimage" ]; then FLAGS="--sysimage $2 "; shift 2; fi +if [ "$1" == "open" ]; then FLAGS="$FLAGS -iq "; fi exec julia --project="$PROJECT_DIR" --color=yes --startup-file=no $FLAGS\ "${BASH_SOURCE[0]}" "$@" =# -using DocOpt -using PlanarLarvae.Datasets, PlanarLarvae.Formats -using LarvaTagger -using JSServe: JSServe, Server +projectdir = dirname(Base.active_project()) +include(joinpath(projectdir, "src/cli.jl")) # defines `main` -usage = """Larva Tagger. - -Usage: - larvatagger.jl import <input-path> [<output-file>] [--id=<id>] [--framerate=<fps>] - larvatagger.jl open <input-path> [--backends=<path>] [--port=<number>] [--quiet] - larvatagger.jl open <input-path> [--viewer] [--browser] - larvatagger.jl train <backend-path> <data-repository> [--instance=<name>] - larvatagger.jl -h | --help - -Options: - -h --help Show this screen. - -q --quiet Do not show instructions. - --id=<id> Run ID, e.g. `date_time`. - --framerate=<fps> Camera frame rate, in frames per second. - --backends=<path> Path to backend repository. - --port=<number> Port number the server listens to. - --viewer Disable editing capabilities. - --browser Automatically open a browser tab at the served location. - --instance=<name> Name of the trained model. - -""" - -function main() - parsed_args = docopt(usage) - if parsed_args["import"] - infile = parsed_args["<input-path>"] - if !isfile(infile) - @error "File not found" infile - return - end - outfile = parsed_args["<output-file>"] - kwargs = Dict{Symbol, Any}() - framerate = parsed_args["--framerate"] - if !isnothing(framerate) - kwargs[:framerate] = parse(Float64, framerate) - end - file = load(infile; kwargs...) - runid = parsed_args["--id"] - run = Run(isnothing(runid) ? file.run.id : runid, Track[]; - file.run.attributes[:metadata]...) - run.attributes[:labels] = String[] - Datasets.pushdependency!(run, infile) - if isnothing(outfile) - Datasets.write_json(stdout, run) - else - outfile = joinpath(dirname(infile), outfile) - Datasets.to_json_file(outfile, run) - end - elseif parsed_args["open"] - verbose = !parsed_args["--quiet"] - infile = parsed_args["<input-path>"] - if !isfile(infile) - @error "File not found" infile - return - end - if parsed_args["--viewer"] - app = larvaviewer(infile) - else - kwargs = Dict{Symbol, Any}() - backends = parsed_args["--backends"] - if !isnothing(backends) - kwargs[:backend_directory] = backends - end - app = larvaeditor(infile; kwargs...) - end - # - port = parsed_args["--port"] - port = isnothing(port) ? 9284 : parse(Int, port) - server = Server(app, "0.0.0.0", port) - if parsed_args["--browser"] - JSServe.openurl("http://127.0.0.1:$(port)") - #display(JSServe.BrowserDisplay(), app) - elseif verbose - @info "The server is listening to http://127.0.0.1:$(port)" - end - if verbose - @info "Press Ctrl+D to kill the server" - end - elseif parsed_args["train"] - @error "Not implemented yet" - end -end - -main() +main(); diff --git a/scripts/larvatagger.sh b/scripts/larvatagger.sh new file mode 100755 index 0000000000000000000000000000000000000000..e90e3671118345cfc9cbc600fda87e3f0d7aac73 --- /dev/null +++ b/scripts/larvatagger.sh @@ -0,0 +1,72 @@ +#!/bin/bash + +cmd=$1; shift + +if [ -z "$LARVATAGGER_IMAGE" ]; then LARVATAGGER_IMAGE=larvatagger; fi + +case "$cmd" in + + build) + +if ! [ -f recipes/Dockerfile ]; then +echo "the build command can only be run from the project root directory" +exit 1 +fi +mkdir -p ./external +while ! [ -z "$1" ]; do +if [ "$1" == "--dev" -o "$1" == "--stable" ]; then +BUILD=$1; shift +elif [ "$1" == "--get-backend" ]; then +DOCKER_ARGS="--build-arg BACKEND=MaggotUBA/20220418 "; shift +cd ./external +if [ -d structured-temporal-convolution ]; then +cd structured-temporal-convolution; git pull; cd .. +else +git clone --depth 1 --single-branch -b light-stable-for-tagging git@gitlab.pasteur.fr:les-larves/structured-temporal-convolution.git +fi +cd .. +else +echo "argument not supported: $1"; shift +#rm -rf ./external +exit 1 +fi +done +DOCKER_ARGS="--build-arg TIMEZONE=$(cat /etc/timezone) $DOCKER_ARGS" +if [ "$BUILD" == "--dev" ]; then +docker build -t "${LARVATAGGER_IMAGE}:dev" -f recipes/Dockerfile.local ${DOCKER_ARGS}. +elif [ "$BUILD" == "--stable" ]; then +docker build -t "${LARVATAGGER_IMAGE}:stable" -f recipes/Dockerfile ${DOCKER_ARGS}. +else +if [ -z "$LARVATAGGER_DEFAULT_BRANCH" ]; then LARVATAGGER_DEFAULT_BRANCH=dev; fi +docker build -t "${LARVATAGGER_IMAGE}:latest" -f recipes/Dockerfile ${DOCKER_ARGS}--build-arg BRANCH=$LARVATAGGER_DEFAULT_BRANCH . +fi +#rm -rf ./external +;; + + open) + +if [ -z "$LARVATAGGER_PORT" -o "$LARVATAGGER_PORT" == "9284" ]; then +DOCKER_ARGS="-p 9284:9284" +TAGGER_ARGS= +else +DOCKER_ARGS="-p $LARVATAGGER_PORT:$LARVATAGGER_PORT" +TAGGER_ARGS="--port=$LARVATAGGER_PORT" +fi +file=$1; shift +exec docker run -iv $(pwd):/data $DOCKER_ARGS "$LARVATAGGER_IMAGE" open "/data/$file" $TAGGER_ARGS $@ +;; + + import) + +file=$1; shift +docker run -v $(pwd):/data "$LARVATAGGER_IMAGE" import "/data/$file" $@ +;; + + *) + +echo "usage: $0 build [--stable] [--dev] [--get-backend]" +echo " $0 open <filepath>" +echo " $0 import <filepath> [<outputfilename>] [--id=<runid>] [--framerate=<fps>]" +;; + +esac diff --git a/src/cli.jl b/src/cli.jl new file mode 100644 index 0000000000000000000000000000000000000000..288748f3d1f95c0f263026f0144c35d420cb6a28 --- /dev/null +++ b/src/cli.jl @@ -0,0 +1,87 @@ +using DocOpt +using PlanarLarvae.Datasets, PlanarLarvae.Formats +using LarvaTagger +using JSServe: JSServe, Server + +usage = """Larva Tagger. + +Usage: + larvatagger.jl import <input-path> [<output-file>] [--id=<id>] [--framerate=<fps>] + larvatagger.jl open <input-path> [--backends=<path>] [--port=<number>] [--quiet] + larvatagger.jl open <input-path> [--viewer] [--browser] + larvatagger.jl train <backend-path> <data-repository> [--instance=<name>] + larvatagger.jl -h | --help + +Options: + -h --help Show this screen. + -q --quiet Do not show instructions. + --id=<id> Run ID, e.g. `date_time`. + --framerate=<fps> Camera frame rate, in frames per second. + --backends=<path> Path to backend repository. + --port=<number> Port number the server listens to. + --viewer Disable editing capabilities. + --browser Automatically open a browser tab at the served location. + --instance=<name> Name of the trained model. + +""" + +function main(args=ARGS) + parsed_args = docopt(usage, args isa String ? split(args) : args) + if parsed_args["import"] + infile = parsed_args["<input-path>"] + if !isfile(infile) + @error "File not found" infile + return + end + outfile = parsed_args["<output-file>"] + kwargs = Dict{Symbol, Any}() + framerate = parsed_args["--framerate"] + if !isnothing(framerate) + kwargs[:framerate] = parse(Float64, framerate) + end + file = load(infile; kwargs...) + runid = parsed_args["--id"] + run = Run(isnothing(runid) ? file.run.id : runid, Track[]; + file.run.attributes[:metadata]...) + run.attributes[:labels] = String[] + Datasets.pushdependency!(run, infile) + if isnothing(outfile) + Datasets.write_json(stdout, run) + else + outfile = joinpath(dirname(infile), outfile) + Datasets.to_json_file(outfile, run) + end + elseif parsed_args["open"] + verbose = !parsed_args["--quiet"] + infile = parsed_args["<input-path>"] + if !isfile(infile) + @error "File not found" infile + return + end + if parsed_args["--viewer"] + app = larvaviewer(infile) + else + kwargs = Dict{Symbol, Any}() + backends = parsed_args["--backends"] + if !isnothing(backends) + kwargs[:backend_directory] = backends + end + app = larvaeditor(infile; kwargs...) + end + # + port = parsed_args["--port"] + port = isnothing(port) ? 9284 : parse(Int, port) + server = Server(app, "0.0.0.0", port) + if parsed_args["--browser"] + JSServe.openurl("http://127.0.0.1:$(port)") + elseif verbose + @info "The server is listening to http://127.0.0.1:$(port)" + end + if verbose + @info "Press Ctrl+D to stop the server" + end + return server + elseif parsed_args["train"] + @error "Not implemented yet" + end +end diff --git a/src/editor.jl b/src/editor.jl index b1c11cdf8ffe9b09b9a631745731653db9013ff1..9c5216d3234dc4cd1c2a8b151b22df540aa34312 100644 --- a/src/editor.jl +++ b/src/editor.jl @@ -25,8 +25,10 @@ function JSServe.jsrender(session::Session, ev::EditorView) class="flex flex-row")) end +projectdir = dirname(Base.active_project()) + function larvaeditor(path=nothing; allow_multiple_tags::Union{Nothing, Bool}=nothing, - backend_directory::String=".") + backend_directory::String=projectdir) # to (re-)load a file, the app is reloaded with the filepath as sole information # from previous session diff --git a/test/precompile-common.jl b/test/precompile-common.jl new file mode 100644 index 0000000000000000000000000000000000000000..4843e0259907da6f12813418eb27203f840487d2 --- /dev/null +++ b/test/precompile-common.jl @@ -0,0 +1,7 @@ +datafile = joinpath(@__DIR__, "data_sample") +ispath(datafile) || exit() + +projectdir = dirname(Base.active_project()) +include(joinpath(projectdir, "src/cli.jl")) + +srv = main(["open", datafile]) diff --git a/test/precompile-electron.jl b/test/precompile-electron.jl new file mode 100644 index 0000000000000000000000000000000000000000..42a0cc2a6543b84a64b2762c6e2814d3983f0e28 --- /dev/null +++ b/test/precompile-electron.jl @@ -0,0 +1,9 @@ +using Blink + +include(joinpath(@__DIR__, "precompile-common.jl")) # defines `srv` + +win = Window(async=false) +loadurl(win, "http://127.0.0.1:9284") +close(win) + +close(srv) diff --git a/test/precompile.jl b/test/precompile.jl new file mode 100644 index 0000000000000000000000000000000000000000..75f5b7c71eb42106b31a4255ce65f6564f4f83bf --- /dev/null +++ b/test/precompile.jl @@ -0,0 +1,7 @@ +using HTTP + +include(joinpath(@__DIR__, "precompile-common.jl")) # defines `srv` + +HTTP.request(:GET, "http://127.0.0.1:9284") + +close(srv) diff --git a/test/precompile.sh b/test/precompile.sh new file mode 100755 index 0000000000000000000000000000000000000000..970dfa413aab5a0eda86f6151c9202fb7a618d53 --- /dev/null +++ b/test/precompile.sh @@ -0,0 +1,34 @@ +#!/bin/sh +root=$(dirname $(dirname $0)) +if [ "$1" = "--shallow" ]; then + +client=HTTP +julia="julia" +script="$root/test/precompile.jl" +packages="g++" + +else + +client=Blink +julia="xvfb-run julia" +script="$root/test/precompile-electron.jl" +# dependencies on Debian Bullseye and Ubuntu Focal +packages="unzip xvfb libgtk-3-0 libnss3 libxss1 libasound2 g++" + +fi +apt-get update +apt-get install -y $packages + +echo "Project root: $root" +$julia --project=$root -e " +using Pkg +Pkg.add([\"$client\",\"PackageCompiler\"]) +using PackageCompiler +println(\"🢃 🢃 🢃 Includes logs from coverage script. Please ignore 🢃 🢃 🢃\") +create_sysimage(String[], sysimage_path=\"larvatagger.so\", precompile_execution_file=\"$script\") +println(\"🢠🢠🢠Includes logs from coverage script. Please ignore 🢠🢠ðŸ¢\") +#Pkg.rm([\"PackageCompiler\",\"$client\"])" + +apt-get autoremove -y $packages +apt-get clean +rm -rf /var/lib/apt/lists/*