diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 6200194ca74c4792b5e00d9f037b408f3658bf92..15804fd7494c224fca2ed1e87b730cb4b75c97fe 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -72,7 +72,8 @@ pages:
image: python:3.9
stage: deploy
script:
- - pip install sphinx sphinx-argparse sphinxcontrib.bibtex sphinx_rtd_theme
+ - pip install -e .
+ - pip install -r doc/requirements.txt
- cd doc
- make html
- mv build/html/ ../public
diff --git a/chart/HOWTO.md b/chart/HOWTO.md
new file mode 100644
index 0000000000000000000000000000000000000000..55b9c2eb4935f0957b2d99778c7703b40815e310
--- /dev/null
+++ b/chart/HOWTO.md
@@ -0,0 +1,56 @@
+# Configure you local machine
+
+```shell
+K8S_USER=${USER} # MUST BE YOU SHORT LOGIN
+kubectl config set-context jass-dev --cluster=k8sdev-01 --user=${K8S_USER}@k8sdev-01 --namespace=jass-dev
+kubectl config set-context jass-prod --cluster=k8sprod-02 --user=${K8S_USER}@k8sprod-02 --namespace=jass-prod
+```
+
+# Starting a new namespace
+
+As the data ar not that small, Bryan decided to put the content of /data in a shared pvc, and that all JASS instances in
+a namespace use this shared pvc to get the data (initTable.hdf5, ...). When starting a new namespace, prior to any
+JASS deployment, the data must already me there. You have to create the pvc, and then copy the data
+(cf. Updating initTable.hdf5).
+
+### Create/update the pvc and the sidekick
+
+```shell
+kubectl config use-context jass-dev
+kubectl apply -f chart/pvc-shared-data.yaml
+```
+
+# Updating /data, including initTable.hdf5
+
+### Update the pvc, start the sidekick
+
+```shell
+kubectl config use-context jass-dev
+kubectl apply -f chart/sidekick.yaml -f chart/pvc-shared-data.yaml
+```
+
+### Copy the data
+
+```shell
+kubectl config use-context jass-dev
+kubectl cp ./data sidekick:/data
+# all data are not where they should be yet, you have to put them one level higher
+kubectl exec sidekick -- ls -lah /data
+kubectl exec sidekick -- sh -c "mv /data/data/* /data/"
+kubectl exec sidekick -- rmdir /data/data
+kubectl exec sidekick -- ls -lah /data
+```
+
+### Alternative: Copy only initTable.hdf5
+
+```shell
+kubectl config use-context jass-dev
+kubectl cp ./data/initTable.hdf5 sidekick:/data/initTable.hdf5
+kubectl exec sidekick -- ls -lah /data
+```
+
+### shutdown the sidekick
+```shell
+kubectl config use-context jass-dev
+kubectl delete -f chart/sidekick.yaml
+```
\ No newline at end of file
diff --git a/chart/pvc-shared-data.yaml b/chart/pvc-shared-data.yaml
index 24b77916f081974bf6df02bc24abe4ac140cdc0a..612b8a528f2a940d5875939c79c9c2ebc10b9461 100644
--- a/chart/pvc-shared-data.yaml
+++ b/chart/pvc-shared-data.yaml
@@ -7,6 +7,6 @@ spec:
- ReadWriteMany
resources:
requests:
- storage: 15Gi
+ storage: 30Gi
storageClassName: isilon
status: {}
\ No newline at end of file
diff --git a/doc/requirements.txt b/doc/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ad6604bbf81288bb4dcb5a3d4933c741e437976c
--- /dev/null
+++ b/doc/requirements.txt
@@ -0,0 +1,5 @@
+sphinx
+sphinx-argparse
+sphinxcontrib.bibtex
+sphinx_rtd_theme
+sphinx-autobuild
\ No newline at end of file
diff --git a/doc/source/data_import.rst b/doc/source/data_import.rst
index 28e7e98b48abb54e2db22e631f67a0d8538d59c5..8953ca7ecf57e6ef16efcadb6833409cccd8e503 100644
--- a/doc/source/data_import.rst
+++ b/doc/source/data_import.rst
@@ -11,7 +11,10 @@ JASS input data
JASS data, from which all statistics can be computed, are stored in an HDF5 file.
This file can be created with the procedure `create-inittable`. This procedure needs the following input files to complete:
-* **GWAS description** file that must contain the following columns and tab-separated:
+GWAS description
+~~~~~~~~~~~~~~~~
+
+This file that must contain the following columns and tab-separated:
+-------------+----------+-------------------+--------------+--------------------+-----------------+-------------+--------------------+
| Consortium | Outcome | FullName | Type | Reference | ReferenceLink | dataLink | internalDataLink |
@@ -22,7 +25,10 @@ This file can be created with the procedure `create-inittable`. This procedure n
The Consortium and outcome names must correspond to the name of the summary statistic files and covariance columns. The last four columns
can be left blank if the user doesn't want to run JASS on a server.
-* **GWAS results files** in the tabular format by chromosome (tab separated) *all in the same folder* with the following columns with the same header:
+GWAS results files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+GWAS results files in the tabular format by chromosome (tab separated) *all in the same folder* with the following columns with the same header:
+----------+-------+------+-----+--------+
| rsID | pos | A0 | A1 | Z |
@@ -33,7 +39,10 @@ can be left blank if the user doesn't want to run JASS on a server.
The name of file *MUST* follow this pattern : "z_{CONSORTIUM}_{TRAIT}_chr{chromosome number}.txt".
The consortium and the trait must be capitalized and must *NOT* contain _ .
-* OPTIONAL : a **covariance file** that corresponds to the covariance between traits under H0. This file is a tab-separated tabular file.
+Covariance file (OPTIONAL)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A covariance file that corresponds to the covariance between traits under H0. This file is a tab-separated tabular file.
+
We recommend that this covariance file to be computed using the `LDScore regression <https://github.com/bulik/ldsc/wiki/Heritability-and-Genetic-Correlation>`_
However, this step can be fastidious and if not provided by the user, a matrix will be inferred from low signal zscore.
@@ -46,8 +55,12 @@ The traits names (columns and row names of the matrix) must correspond to the su
GABRIEL_ASTHMA 0.085 0.025 0.0382 1.0134 -0.0104
GEFOS_BMD-FOREARM -0.0061 -0.0002 0.0048 -0.0104 1.0123
-* **Region file** of approximately independant LD regions to the BED file. For european ancestry and grch37/hg19, we suggest to use the regions as defined by :cite:`Berisa2015`, which is already available in `the data folder of the package <https://gitlab.pasteur.fr/statistical-genetics/jass/blob/master/data/fourier_ls-all.bed>`_.
-For grch38, we computed these regions for the five superpopulation available in 1000G using Big SNPR :cite:`10.1093/bioinformatics/btab519`. The corresponding files are stored at <https://gitlab.pasteur.fr/statistical-genetics/jass_suite_pipeline/-/tree/pipeline_ancestry/input_files>`_.
+Region file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Region file of approximately independant LD regions to the BED file. For european ancestry and grch37/hg19, we suggest to use the regions as defined by :cite:`Berisa2015`, which is already available in `the data folder of the package <https://gitlab.pasteur.fr/statistical-genetics/jass/blob/master/data/fourier_ls-all.bed>`_.
+
+ For grch38, we computed these regions for the five superpopulation available in 1000G using Big SNPR :cite:`10.1093/bioinformatics/btab519`. The corresponding files are stored at <https://gitlab.pasteur.fr/statistical-genetics/jass_suite_pipeline/-/tree/pipeline_ancestry/input_files>`_.
+----------+--------+--------+
| chr | start | stop |
@@ -55,8 +68,8 @@ For grch38, we computed these regions for the five superpopulation available in
| chr1 | 10583 | 1892607|
+----------+--------+--------+
-For inferring approximately independant LD regions from your own panel we recommend using https://privefl.github.io/bigsnpr/ .
-See :cite:`10.1093/bioinformatics/btab519` on the matter
+ For inferring approximately independant LD regions from your own panel we recommend using https://privefl.github.io/bigsnpr/ .
+ See :cite:`10.1093/bioinformatics/btab519` on the matter.
Init table generation
@@ -75,14 +88,14 @@ How to generate input data for JASS
-----------------------------------
Option 1 nextflow pipeline :
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Preprocessing steps for JASS (data harmonisation and imputation)have been gathered in one nextflow pipeline : `JASS pipeline Suite <https://gitlab.pasteur.fr/statistical-genetics/jass_suite_pipeline>`_.
While this option might have stronger installation requirements, it ensure reproducibility by leveraging docker containers (fixed version of JASS and accompanying packages).
It will also be much more efficient is you a large number of heterogeneous data to handle and a computing cluster available.
Option 2 manually prepare input data:
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To standardize the format of the input GWAS datasets, you can use the `JASS Pre-processing package <https://gitlab.pasteur.fr/statistical-genetics/JASS_Pre-processing>`_. The `JASS Pre-processing documentation <http://statistical-genetics.pages.pasteur.fr/JASS_Pre-processing/>`_ details the use of this tool.
We think that the best way to compute such covariance from summary statistics is to use the LD-score regression (https://github.com/bulik/ldsc/wiki/Heritability-and-Genetic-Correlation). In the output of the LD-score genetic correlation use the intercept (intercept heritability of trait i for variance of trait i and intercept of the genetic covariance for the covariance between the two traits)::
@@ -123,6 +136,7 @@ Once, GWAS summary statistics are harmonized, they are integrated into
one file by the using jass command line (see detail in command line usage)
.. code-block:: console
+
jass create-inittable --input-data-path "harmonized_GWAS_files/*.txt" --init-covariance-path $path1/Covariance_matrix_H0.csv --regions-map-path $path2/Region_file.bed --description-file-path $path3/Data_summary.csv --init-table-path $path4/init_table_EUR_not_imputed.hdf5
.. bibliography:: reference.bib
diff --git a/doc/source/generating_joint_analysis.rst b/doc/source/generating_joint_analysis.rst
index 3369d6d353222733d891f0cd7ed4aac229dc9bce..88b4c1d8d3259f01b76f07e5c27d85db7c62fffe 100644
--- a/doc/source/generating_joint_analysis.rst
+++ b/doc/source/generating_joint_analysis.rst
@@ -17,10 +17,12 @@ Whatever the test used, the command will generate three output:
- 'summaryTable': a double entry table summarizing the number of significant regions by test (univariate vs joint test)
* A **.png Manhattan plot** of the joint test p-values:
+
.. image:: ./_static/manhattan_glycemic_blood_asthma.png
* A **.png Quadrant plot** which is a scatter plot of the minimum p-value by region of the joint test with respect to the minimum p-value by region of the univariate tests.
- This plot provides an easy way to see if your joint analysis detected association not previously reported in the litterature.
+ This plot provides an easy way to see if your joint analysis detected association not previously reported in the litterature.
+
.. image:: ./_static/quadrant_glycemic_blood_asthma.png
The Omnibus tests
@@ -61,4 +63,5 @@ Command Line example
See command line usage for details
.. code-block:: shell
+
jass create-project-data --init-table-path init_table/init_table_EUR_not_imputed.hdf5 --phenotype z_MAGIC_GLUCOSE-TOLERANCE z_MAGIC_FAST-GLUCOSE z_MAGIC_FAST-INSULIN z_MAGIC_HBA1C --worktable-path ./work_glycemic.hdf5 --manhattan-plot-path ./manhattan_glycemic.png --quadrant-plot-path ./quadrant_glycemic.png
diff --git a/doc/source/install.rst b/doc/source/install.rst
index 7f1019c08be4d5f1770070e9b2813d40ce4e1377..23328fc9268658c532e8b258583dba89deed834c 100644
--- a/doc/source/install.rst
+++ b/doc/source/install.rst
@@ -112,7 +112,7 @@ To run locally JASS as a web application, it is recommended to use the docker-co
If using docker-compose is not an option, you need to launch two servers in two different processes, the `celery` task management server and the web server. The web server handles the HTTP requests, and sends all computation requests to the task management server.
Launching with docker compose (recommended)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once docker and docker-compose installed, just run
.. code-block:: shell
diff --git a/jass/__init__.py b/jass/__init__.py
index a88eb7abf651d44a7e45578285661a85262703a3..f467725b00ccfe19f921fb8a6fa41d12cfafa082 100644
--- a/jass/__init__.py
+++ b/jass/__init__.py
@@ -8,13 +8,10 @@ Submodules
:toctree: _autosummary
config
- controllers
- encoder
models
- util
- wsgi
server
+ tasks
+ util
"""
-import os
from jass.tasks import celery