Update README and documentation. Large improvement of the conda install and we...

Update README and documentation. Large improvement of the conda install and we get closer to the publication

.gitignore

@@ -2,8 +2,4 @@
 *.Rhistory
 *.RData
 .Rproj.user
-*.Rproj
-www/masque/doing/*
-www/masque/todo/*
-www/masque/error/*
 www/masque.png

.gitlab-ci.yml

@@ -33,13 +33,15 @@ build_master:
     - docker login -u "$DOCKER_USER" -p "$DOCKER_PASS"
     - docker build -t aghozlane/shaman .
     - docker push aghozlane/shaman
-  only:
-    - master
+  except:
+    - dev
 
 
 build_conda:
   image: continuumio/miniconda3:latest
   stage: build_conda
+  variables:
+    CI_DEBUG_TRACE: "false"
   before_script:
     - conda config --add channels bioconda
     - conda config --add channels conda-forge
@@ -49,7 +51,9 @@ build_conda:
     - conda install conda-verify
     - conda install anaconda-client
     - conda install conda-build
-
   script:
     - anaconda login --username "$DOCKER_USER" --password "$DOCKER_PASS"
-    - conda build conda_inst
\ No newline at end of file
+    - conda build conda_inst
+  except:
+    - dev
+    - master
\ No newline at end of file

LoadPackages.R

@@ -51,13 +51,13 @@ if (!require(circlize)) {
 if(!require(BiocManager)){
   install.packages("BiocManager")
 }
-# if (!require(genefilter)) {
-#   BiocManager::install("genefilter", ask=FALSE)
-# }
-# 
-# if(!require("SummarizedExperiment")){
-#   BiocManager::install("SummarizedExperiment", ask=FALSE)
-# }
+if (!require(genefilter)) {
+  BiocManager::install("genefilter", ask=FALSE)
+}
+
+if(!require("SummarizedExperiment")){
+  BiocManager::install("SummarizedExperiment", ask=FALSE)
+}
 
 if (!require(googleVis)) {
   install.packages('googleVis',  repos="https://cran.univ-paris1.fr/")

README.md

@@ -14,14 +14,19 @@
 ## Introduction
 
 SHAMAN is a shiny application for differential analysis of metagenomic data (16S, 18S, 23S, 28S, ITS and WGS) including bioinformatics treatment of raw reads for targeted metagenomics, statistical analysis and results visualization with a large variety of plots (barplot, boxplot, heatmap, …).
-The statistical analysis performed by SHAMAN is based on DESeq2 R package [Anders and Huber 2010](http://www.ncbi.nlm.nih.gov/pubmed/20979621) ] which robustly identifies the differential abundant features as suggested in  [McMurdie and Holmes 2014](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3974642/) and [Jonsson2016](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4727335/).
-SHAMAN robustly identifies the differential abundant genera with the Generalized Linear Model implemented in DESeq2 [Love 2014](http://www.ncbi.nlm.nih.gov/pubmed/25516281).
-SHAMAN is compatible with standard formats for metagenomic analysis (.csv, .tsv, .biom) and generated figures can be downloaded in several formats. Hereafter is the global workflow of the SHAMAN application:  
+The statistical analysis performed by SHAMAN is based on DESeq2 R package [[Anders and Huber 2010](http://www.ncbi.nlm.nih.gov/pubmed/20979621)] which robustly identifies the differential abundant features as suggested in  [[McMurdie and Holmes 2014](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3974642/)] and [[Jonsson2016](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4727335/)].
+SHAMAN robustly identifies the differential abundant genera with the Generalized Linear Model implemented in DESeq2 [[Love 2014](http://www.ncbi.nlm.nih.gov/pubmed/25516281)].
+SHAMAN is compatible with standard formats for metagenomic analysis (.csv, .tsv, .biom) and figures can be downloaded in several formats. Hereafter is the global workflow of the SHAMAN application:  
 <img src="www/Workflow.png" align="center" />
 
 ## Installation
 
-SHAMAN is available for R 3.6.1. The installation, download and execution can all be performed with a small R script :
+SHAMAN is available for R 3.6.1. Here are different ways to install the application:
+
+### R install
+
+R install will require Gfortran on mac (https://cran.r-project.org/bin/macosx/tools/) and Rtools on windows (https://cran.r-project.org/bin/windows/Rtools/index.html). 
+Of note, raw data submission is not possible with this version.The installation, download and execution can all be performed with a small R script :
 ```
 # Load shiny packages
 if(!require('shiny')){
@@ -34,28 +39,75 @@ runGitHub('aghozlane/shaman')
 ```
 This script can also be executed as following : 
 ```
-chmod +x ./shaman/www/shamanapp.R && Rscript ./shaman/www/shamanapp.R
+chmod +x ./shaman/www/shaman.R && Rscript ./shaman/www/shaman.R
+```
+
+
+### R install with packrat (linux only)
+
+SHAMAN is available for R=3.6.1. Packrat framework allows an easy installation of all the dependencies. Of note, raw data submission is not possible with this version.
+```
+# Download packrat package
+wget ftp://shiny01.hosting.pasteur.fr/pub/shaman_package_201908.tar.gz
+mkdir packrat/shaman
+```
+Now you can run R:
+```
+install.packages("packrat")
+packrat::unbundle("shaman_package_201908.tar.gz", "packrat/shaman")
+packrat::init("packrat/shaman")
+system("Rscript -e 'shiny::runGitHub("pierreLec/KronaRShy",port=5438)'",wait=FALSE),
+shiny::runGitHub('aghozlane/shaman')
 ```
+### Docker install
+
+Docker install is the easiest way to use SHAMAN locally. First, install docker:
+- Install docker on ubuntu (Linux): https://docs.docker.com/engine/installation/linux/ubuntulinux/
+- Install docker on Windows and Mac: https://www.docker.com/
+
+Now in your terminal:
+```
+docker pull aghozlane/shaman && docker run --rm -p 80:80 -p 5438:5438 aghozlane/shaman
+```
+Then connect to http://0.0.0.0/ with your favorite web navigator.
+Failed: port is already allocated ?
+```
+docker run --rm -p 3838:80 -p 5438:5438 aghozlane/shaman
+```
+Then connect to http://0.0.0.0:3838/.
+Docker update after an update of SHAMAN:
+```
+docker pull aghozlane/shaman
+```
+
+### CONDA
+
+Conda installation of SHAMAN is available for linux and mac. It can be performed as follow:
+```
+# Download and install SHAMAN
+conda install -c aghozlane shaman
+# Now run shaman
+shaman_conda.R
+```
+Then connect to http://0.0.0.0/.
 
 ## Citation
 
 If you use SHAMAN for your project, please cite our first application of SHAMAN in Quereda et al. 2016.
 Publication using SHAMAN :  
+- Prediction of the intestinal resistome by a three-dimensional structure-based method, Ruppé E, Ghozlane A, Tap J, et al.;Nature microbiology 2018 [PUBMED](https://www.ncbi.nlm.nih.gov/pubmed/30478291)
+- Combined bacterial and fungal intestinal microbiota analyses: Impact of storage conditions and DNA extraction protocols, Angebault C, Ghozlane A, Volant S, Botterel F, d’Enfert C, Bougnoux ME, PloS one 2018 [PUBMED](https://www.ncbi.nlm.nih.gov/pubmed/30074988)
+ - Clinical Efficacy and Microbiome Changes Following Fecal Microbiota Transplantation in Children With Recurrent Clostridium Difficile Infection, Li X, Gao X, Hu H, Xiao Y, Li D, Yu G, Yu D, Zhang T, Wang Y, Frontiers in Microbiology [PUBMED](https://www.ncbi.nlm.nih.gov/pubmed/26566371)                                                                                       
 - Diverse laboratory colonies of Aedes aegypti harbor the same adult midgut bacterial microbiome. Dickson LB, Ghozlane A, Volant S, Bouchier C, Ma L, Vega-Rúa A, Dusfour I, Jiolle D, Paupy C, Mayanja MN, Kohl A, Lutwama JJ, Duong V, Lambrechts L; Parasit Vectors 2018 [PUBMED](https://www.ncbi.nlm.nih.gov/pubmed/29587819").
-- Characteristics of Fecal Microbiota in Pediatric Crohn’s Disease and Their Dynamic Changes During Infliximab Therapy.", href="https://www.ncbi.nlm.nih.gov/pubmed/29194468"), "Wang Y, Gao X, Ghozlane A, Hu H, Li X, Xiao Y, Li D, Yu G, Zhang T; Journal of Crohn's & colitis  2017",style = "font-family: 'times'; font-si16pt"),
-- Carryover effects of larval exposure to different environmental bacteria drive adult trait variation in a mosquito vector.", href="https://www.ncbi.nlm.nih.gov/pubmed/28835919"), "Dickson LB, Jiolle D, Minard G, Moltini-Conclois I, Volant S, Ghozlane A, Bouchier C, Ayala D, Paupy C, Moro CV, Lambrechts L; Science Advances 2017",style = "font-family: 'times'; font-si16pt"),
-- A bacteriocin from epidemic Listeria strains alters the host intestinal microbiota to favor infection.", href="http://www.ncbi.nlm.nih.gov/pubmed/27140611"), "Quereda JJ, Dussurget O, Nahori MA, Ghozlane A, Volant S, Dillies MA, Regnault B, Kennedy S, Mondot S, Villoing B, Cossart P, Pizarro-Cerda J.; PNAS 2016",style = "font-family: 'times'; font-si16pt"),
-- Reporting bugs, ask for help",style = "font-family: 'times'; font-si18pt; font-style: strong"),
-- If you have any comments, questions or suggestions, or need help to use SHAMAN, please contact us at", a("shaman@pasteur.fr", href="mailto:shaman@pasteur.fr"),"and please provide us with enough information that we can recreate the problem. Useful things to include are:", style = "font-family: 'times'; font-si16pt;"),
-                   
-Input data (or examples, a small test case sufficient to recreate the problem)"), 
-Information about which system your are using: web version, docker or R installation")
-
-No papers about SHAMAN have been published yet, but a manuscript is in preparation.  
-Publication using SHAMAN :  
-  
+- Characteristics of Fecal Microbiota in Pediatric Crohn’s Disease and Their Dynamic Changes During Infliximab Therapy., Wang Y, Gao X, Ghozlane A, Hu H, Li X, Xiao Y, Li D, Yu G, Zhang T; Journal of Crohn's & colitis  2017 [PUBMED](https://www.ncbi.nlm.nih.gov/pubmed/29194468)
+- Carryover effects of larval exposure to different environmental bacteria drive adult trait variation in a mosquito vector, Dickson LB, Jiolle D, Minard G, Moltini-Conclois I, Volant S, Ghozlane A, Bouchier C, Ayala D, Paupy C, Moro CV, Lambrechts L; Science Advances 2017 [PUBMED](https://www.ncbi.nlm.nih.gov/pubmed/28835919) 
 - A bacteriocin from epidemic Listeria strains alters the host intestinal microbiota to favor infection. Quereda JJ, Dussurget O, Nahori MA, Ghozlane A, Volant S, Dillies MA, Regnault B, Kennedy S, Mondot S, Villoing B, Cossart P, Pizarro-Cerda J.; PNAS 2016 [PUBMED](http://www.ncbi.nlm.nih.gov/pubmed/27140611).
 
+If you have any comments, questions or suggestions, or need help to use SHAMAN, contact us at [shaman@pasteur.fr](mailto:shaman@pasteur.fr), and provide us with enough information that we can recreate the problem. Useful things to include are:
+- Input data
+- Error message from SHAMAN
+- Key number
+
 
 ## Contact
 
@@ -64,10 +116,11 @@ If you have any comments, questions or suggestions, or need help to use SHAMAN,
 ## Acknowledgements
 
 The main contributors to SHAMAN:
-Stevenn Volant, (Initiator, coding, testing, documentation, evaluation),
-Amine Ghozlane, (Coding, testing, documentation, evaluation, packaging),
+Stevenn Volant (Initiator, coding, testing, documentation, evaluation),
+Amine Ghozlane (Coding, testing, documentation, evaluation, packaging),
+Perrine Woringer (Coding, testing, documentation, feature suggestions),
 Pierre Lechat (Coding, testing, feature suggestions)  
 
 Thanks to the following people for patches and other suggestions for improvements:  
-Carine Rey, Hugo Varet, Julien Tap, Anna Zhukova
+Carine Rey, Valentin Marcon, Hugo Varet, Julien Tap, Anna Zhukova
 

Rfunctions/Data_Management.R

@@ -524,7 +524,6 @@ GetDataFromBIOM <-function(dataBIOM)
     OTUnames = rownames(taxo)
     ## Modif taxo table (remove p__,... and change the colnames)
     taxo_biom = taxo
-    saveRDS(taxo, file="taxo.RDS")
     taxo = as.data.frame(sapply(taxo,gsub,pattern="^.*__",replacement=""))
     colnames(taxo) = c("Kingdom", "Phylum","Class","Order","Family","Genus","Species")
     rownames(taxo) = OTUnames

conda_inst/meta.yaml

 package:
     name: shaman
-    version: {{ GIT_DESCRIBE_TAG }}
-
-channels:
-    - defaults
-    - conda-forge
-    - bioconda
-    - r
-    - aghozlane
+    version: {{ environ.get('GIT_DESCRIBE_TAG', '') }}
 
 source:
     - git_url: https://gitlab.pasteur.fr/aghozlan/shaman.git
-      git_rev: {{ GIT_DESCRIBE_TAG }}
+      git_rev: {{ environ.get('GIT_DESCRIBE_TAG', '') }}
       folder: shamanapp
     - git_url: https://github.com/aghozlane/KronaRShy.git
       git_rev: 0.9
@@ -39,8 +32,6 @@ requirements:
         - r-vegan
         - r-dendextend
         - r-circlize
-        #- bioconductor-genefilter
-        #- bioconductor-summarizedexperiment
         - r-biocmanager
         - r-googlevis
         - r-shinyjs
@@ -83,6 +74,9 @@ requirements:
         - lockfile
         - python-daemon
         - bioblend
+        - bioconductor-genomicranges=1.32.7
+        #- bioconductor-genefilter
+        - bioconductor-summarizedexperiment=1.10.1
 test:
     commands:
         - 'R -e "libshaman=c(\"shiny\", \"rjson\", \"ape\", \"GUniFrac\", \"psych\", \"ggplot2\", \"vegan\", \"dendextend\", 

shaman.Rproj

+Version: 1.0
+
+RestoreWorkspace: Default
+SaveWorkspace: Default
+AlwaysSaveHistory: Default
+
+EnableCodeIndexing: Yes
+UseSpacesForTab: Yes
+NumSpacesForTab: 2
+Encoding: UTF-8
+
+RnwWeave: Sweave
+LaTeX: pdfLaTeX

ui.R

@@ -70,7 +70,6 @@ function(request) {
                                              p("Publication using SHAMAN:",style = "font-family: 'times'; font-si18pt; font-style: strong"),
                                              p(a("Prediction of the intestinal resistome by a three-dimensional structure-based method.", href="https://www.ncbi.nlm.nih.gov/pubmed/30478291"),"Ruppé E, Ghozlane A, Tap J, et al.;Nature microbiology 2018",style = "font-family: 'times'; font-si16pt"),
                                              
-                                             p(a("Diverse laboratory colonies of Aedes aegypti harbor the same adult midgut bacterial microbiome.", href="https://www.ncbi.nlm.nih.gov/pubmed/29587819"), "Dickson LB, Ghozlane A, Volant S, Bouchier C, Ma L, Vega-Rúa A, Dusfour I, Jiolle D, Paupy C, Mayanja MN, Kohl A, Lutwama JJ, Duong V, Lambrechts L; Parasit Vectors 2018",style = "font-family: 'times'; font-si16pt"),
                                              p(a("Combined bacterial and fungal intestinal microbiota analyses: Impact of storage conditions and DNA extraction protocols", href="https://www.ncbi.nlm.nih.gov/pubmed/30074988"), "Angebault C, Ghozlane A, Volant S, Botterel F, d’Enfert C, Bougnoux ME, PloS one 2018",style = "font-family: 'times'; font-si16pt"),
                                              p(a("Clinical Efficacy and Microbiome Changes Following Fecal Microbiota Transplantation in Children With Recurrent Clostridium Difficile Infection", href="https://www.ncbi.nlm.nih.gov/pubmed/26566371"), "Li X, Gao X, Hu H, Xiao Y, Li D, Yu G, Yu D, Zhang T, Wang Y, Frontiers in Microbiology",style = "font-family: 'times'; font-si16pt"),
                                              p(a("Diverse laboratory colonies of Aedes aegypti harbor the same adult midgut bacterial microbiome", href="https://www.ncbi.nlm.nih.gov/pubmed/29587819"),"Dickson LB, Ghozlane A, Volant S, Bouchier C, Ma L, Vega-Rúa A, Dusfour I, Jiolle D, Paupy C, Mayanja MN, Kohl A, Lutwama JJ, Duong V, Lambrechts L",style = "font-family: 'times'; font-si16pt"),
@@ -215,9 +214,7 @@ function(request) {
                                              p("Docker install is the easiest way to use SHAMAN locally."),
                                              p("- Install docker on ubuntu (Linux):",a("here",href="https://docs.docker.com/engine/installation/linux/ubuntulinux/")),
                                              p("- Install docker on Windows and Mac:"),
-                                             p("Download and install docker from",a("https://www.docker.com/", href="https://www.docker.com/"),
-                                               "Then, you will need to install the", a("Docker toolbox.", href="https://www.docker.com/products/docker-toolbox"),
-                                               "Once installed, run ‘Docker Quickstart Terminal’ application."),
+                                             p("Download and install docker from",a("https://www.docker.com/", href="https://www.docker.com/")),
                                              p("- Running SHAMAN:"),
                                              mainPanel(div(style = 'max-width: 900px',"docker pull aghozlane/shaman && docker run --rm -p 80:80 -p 5438:5438 aghozlane/shaman"),width=8,class="mainwell"),
                                              p("Then connect to http://0.0.0.0/ with your favorite web navigator."),
@@ -227,21 +224,23 @@ function(request) {
                                              p("- Docker update after an update of SHAMAN:"),
                                              mainPanel(div(style = 'max-width: 900px',"docker pull aghozlane/shaman"),width=4,class="mainwell")
                                     ),
-                                    tabPanel("R install with Packrat",
+                                    tabPanel("R install with Packrat (linux only)",
                                              p("SHAMAN is available for R=3.6.1. Packrat framework allows an easy installation of all the dependencies. Of note, raw data submission is not possible with this version.",style = "font-family: 'times'; font-si16pt"),
                                              mainPanel(div(style = 'max-width: 900px; word-wrap: break-word;',
                                                            "# Download packrat package",br(),"wget ftp://shiny01.hosting.pasteur.fr/pub/shaman_package_201908.tar.gz", br(),
                                                            "mkdir packrat/shaman"), width=9,class="mainwell"),
                                              p("Now you can run R:"),
                                              mainPanel(div(style = 'max-width: 900px; word-wrap: break-word;',
-                                                           "install.packages(c(\"packrat\", \"shiny\"))",br(),
+                                                           "install.packages(\"packrat\")",br(),
                                                            "packrat::unbundle(\"shaman_package_201908.tar.gz\", \"packrat/shaman\")", br(),
                                                            "packrat::init(\"packrat/shaman\")",br(),
                                                            "system(\"Rscript -e 'shiny::runGitHub(\\\"pierreLec/KronaRShy\\\",port=5438)'\",wait=FALSE)",
                                                            br(),"shiny::runGitHub('aghozlane/shaman')"),width=9,class="mainwell")),
                                     
                                     tabPanel("R install",
-                                             p("SHAMAN is available for R=3.6.1. Of note, raw data submission is not possible with this version. The installation, download and execution can all be performed with a small R script:",style = "font-family: 'times'; font-si16pt"),
+                                             p("SHAMAN is available for R=3.6.1. Gfortran is required on mac (https://cran.r-project.org/bin/macosx/tools/) and Rtools on windows (https://cran.r-project.org/bin/windows/Rtools/index.html). 
+                                             Of note, raw data submission is not possible with this version. 
+                                               The installation, download and execution can all be performed with a small R script:",style = "font-family: 'times'; font-si16pt"),
                                              mainPanel(div(style = 'max-width: 900px; word-wrap: break-word;',"# Load shiny packages",br(),
                                                            "if(!require('shiny')){",br(),"  install.packages('shiny')",br(),"}",br(),
                                                            "system(\"Rscript -e 'shiny::runGitHub(\\\"pierreLec/KronaRShy\\\",port=5438)'\",wait=FALSE)",
@@ -251,11 +250,11 @@ function(request) {
                                              mainPanel(div(style = 'max-width: 900px; word-wrap: break-word;',"chmod +x ./shaman.R && Rscript ./shaman.R"),width=6,class="mainwell"),br(),
                                              p("Of note, contribution to SHAMAN code are always welcome and can be performed with the", a("github deposit.",href="https://github.com/aghozlane/shaman"))),
                                     tabPanel("Conda install",
-                                             p("Conda installation of SHAMAN is available on linux and soon on mac. It can be performed as follow: ",style = "font-family: 'times'; font-si16pt"),
+                                             p("Conda installation of SHAMAN is available for linux and mac. It can be performed as follow: ",style = "font-family: 'times'; font-si16pt"),
                                              mainPanel(div(style = 'max-width: 900px; word-wrap: break-word;',
                                                            "# Download and install SHAMAN", br(),
                                                            "conda install -c aghozlane shaman", br(),
-                                                           "# Now run shaman", br(), "Rscript shaman_conda.R"), width=6,class="mainwell"))
+                                                           "# Now run shaman", br(), "shaman_conda.R"), width=6,class="mainwell"))
                              )
                          )
                   )

www/masque/doing/purpose

+Json currently in progress

www/masque/done/purpose

+Output the results

www/masque/error/purpose

+Error json storage and message from galaxy.

www/masque/todo/purpose

+Input json from the users

www/shaman_conda.R

-#!/usr/bin/Rscript --vanilla
+#!/usr/bin/env Rscript --vanilla
 prefix=Sys.getenv(x = "CONDA_PREFIX")
 prefix=paste(prefix, "bin", sep=.Platform$file.sep)
 system(paste("python ", prefix, "shaman_bioblend/shaman_bioblend.py -w ", prefix,"shamanapp/www/masque/ -s -d", sep=.Platform$file.sep))
-system(paste("R -e 'library(\"shiny\");runApp(appDir=\"", prefix,"KronaRShy\", port=5438)'", sep=.Platform$file.sep), wait=FALSE)
-system(paste("R -e 'library(\"shiny\");runApp(appDir=\"", prefix,"shamanapp\", port=80, host=\"0.0.0.0\")'", sep=.Platform$file.sep), wait=FALSE)
+kronarshy=paste(tempfile(),".R",sep="")
+write(paste("library(\"shiny\");runApp(appDir=\"", prefix,"KronaRShy\", port=5438)", sep=.Platform$file.sep),kronarshy)
+system(paste("Rscript", kronarshy), wait=FALSE)
+shamanapp=paste(tempfile(),".R",sep="")
+write(paste("library(\"shiny\");runApp(appDir=\"", prefix,"shamanapp\", port=80, host=\"0.0.0.0\")", sep=.Platform$file.sep), shamanapp)
+system(paste("Rscript", shamanapp), wait=FALSE)