README.md 6.49 KB
Newer Older
Julie Sullivan's avatar
Julie Sullivan committed
1
# docker-intermine-gradle
Julie Sullivan's avatar
Julie Sullivan committed
2
3

You can use these docker images to create your own InterMine instance. You can launch our test InterMine (BioTestMine) or you can use your own data to build a custom mine.
Ankur Kumar's avatar
Ankur Kumar committed
4
5

## Requirements
Julie Sullivan's avatar
Julie Sullivan committed
6
7
8

 - [Docker](https://docs.docker.com/install/)
 - [Docker compose](https://docs.docker.com/compose/install/)
Ankur Kumar's avatar
Ankur Kumar committed
9
10

## Quickstart
Julie Sullivan's avatar
Julie Sullivan committed
11
12
13

Run the command to build the images locally and start an example InterMine instance - BioTestMine.

Ankur Kumar's avatar
Ankur Kumar committed
14
```bash
Julie Sullivan's avatar
Julie Sullivan committed
15
# Build images locally and start mine
Ankur Kumar's avatar
Ankur Kumar committed
16
docker-compose -f local.docker-compose.yml up --build --force-recreate
Ankur Kumar's avatar
Ankur Kumar committed
17
18
```

Julie Sullivan's avatar
Julie Sullivan committed
19
20
Alternatively, you can also use our images on docker hub.

Ankur Kumar's avatar
Ankur Kumar committed
21
```bash
Julie Sullivan's avatar
Julie Sullivan committed
22
23
# OR
# use the images on dockerhub
Ankur Kumar's avatar
Ankur Kumar committed
24
25
26
27
28
29
30
31
docker-compose -f dockerhub.docker-compose.yml up 
```

Wait for 10-15 mins for the build to finish. Then visit **`localhost:9999/biotestmine`**
to visit your mine (yes it is that easy now!!)
> to surely determine whether build is finished or not, search for `intermine_builder exited with code 0` message in your docker-compose logs.
## Customizing your mine instance

32
### Using a custom mine
33
34
You can use your custom mine for the builds. 

Ankur Kumar's avatar
Ankur Kumar committed
35
#### Step 1
36

Ankur Kumar's avatar
Ankur Kumar committed
37
- Set the git url of your mine to the env var `MINE_REPO_URL`. (directory structure similar to biotestmine is assumed)
38

Ankur Kumar's avatar
Ankur Kumar committed
39
40
   If you do not have your mine hosted on github/gitlab then you can also mount your   mine files directly to the build container. Add your mine folder inside `./data/mine`  folder created by docker compose and uncomment the following line in the    docker-compose.yml file. 
   ```bash
41
42
43
44
   # - ./data/mine/[PUT_YOUR_MINE_NAME_HERE]:/home/intermine/intermine/ [PUT_YOUR_MINE_NAME_HERE]

   ```
   > Note: build will do a few changes to your mine files when mine files are directly mounted to the build container. So, it is recommended to use a copy of your mine for the build. This is not the case when you are using MINE_REPO_URL to pull mine files.
Ankur Kumar's avatar
Ankur Kumar committed
45

Julie Sullivan's avatar
Julie Sullivan committed
46
To determine whether build is finished or not, search for `intermine_builder exited with code 0` message in your docker-compose logs.
47

Julie Sullivan's avatar
Julie Sullivan committed
48
## Deploy your own InterMine with Docker
49

Julie Sullivan's avatar
Julie Sullivan committed
50
Instead of building our test mine, you can launch your own custom InterMine by following these instructions.
Ankur Kumar's avatar
Ankur Kumar committed
51

Julie Sullivan's avatar
Julie Sullivan committed
52
### Set Environment variables
53

54
55
56
57
#### Step 4 (optional)
- Set the git url of your bio-sources repo to the env var `BIOSOURCES_REPO_URL`.

#### Step 5
Ankur Kumar's avatar
Ankur Kumar committed
58
- Follow [Adding data and project configs](#adding-data-and-project-configs) section to load data and config to your mine.
Julie Sullivan's avatar
Julie Sullivan committed
59
These will tell Docker where your mine is and its name.
60

Julie Sullivan's avatar
Julie Sullivan committed
61
62
| ENV variable  | Notes | Example |
| ------------- | ------------- | ------------- |
Julie Sullivan's avatar
Julie Sullivan committed
63
| MINE_REPO_URL | the git url of your mine | https://github.com/intermine/flymine  |
Julie Sullivan's avatar
Julie Sullivan committed
64
65
| MINE_NAME  | Name of your mine | FlyMine  |
| IM_DATA_DIR | Data directory as used in your project XML file. | /data/flymine (no trailing slash "/") |
66

Julie Sullivan's avatar
Julie Sullivan committed
67
### Update data location
Ankur Kumar's avatar
Ankur Kumar committed
68

Julie Sullivan's avatar
Julie Sullivan committed
69
We now need to tell Docker where the data is located.
Ankur Kumar's avatar
Ankur Kumar committed
70

Julie Sullivan's avatar
Julie Sullivan committed
71
1. Create the directory `./data/mine/data` in your **current working directory**. If you did the quickstart above, it will already have been created for you.
Ankur Kumar's avatar
Ankur Kumar committed
72

Julie Sullivan's avatar
Julie Sullivan committed
73
2. Uncomment the following line in intermine_builder section of your docker-compose file:
Ankur Kumar's avatar
Ankur Kumar committed
74

Ankur Kumar's avatar
Ankur Kumar committed
75
76
77
```bash
# - ./data/mine/data:/home/intermine/intermine/data
```
Ankur Kumar's avatar
Ankur Kumar committed
78

Julie Sullivan's avatar
Julie Sullivan committed
79
3. Update `./data/mine/data` to be the absolute path to your data directory. 
Ankur Kumar's avatar
Ankur Kumar committed
80

Julie Sullivan's avatar
Julie Sullivan committed
81
### Configs
Ankur Kumar's avatar
Ankur Kumar committed
82

Julie Sullivan's avatar
Julie Sullivan committed
83
Add your mine property file (e.g. flymine.properties) to the **`./data/mine/config`** folder. 
Ankur Kumar's avatar
Ankur Kumar committed
84

Julie Sullivan's avatar
Julie Sullivan committed
85
In your properties file, do the following changes:
Ankur Kumar's avatar
Ankur Kumar committed
86

Julie Sullivan's avatar
Julie Sullivan committed
87
```
Ankur Kumar's avatar
Ankur Kumar committed
88
89
90
91
92
93
94
95
96
97
98
99
100
101
      db.production.datasource.serverName=postgres
      db.production.datasource.databaseName=PSQL_DB_NAME
      db.production.datasource.user=PSQL_USER
      db.production.datasource.password=PSQL_PWD
  
      db.common-tgt-items.datasource.serverName=postgres
      db.common-tgt-items.datasource.databaseName=items-PSQL_DB_NAME
      db.common-tgt-items.datasource.user=PSQL_USER
      db.common-tgt-items.datasource.password=PSQL_PWD
      
      db.userprofile-production.datasource.serverName=postgres
      db.userprofile-production.datasource.databaseName=userprofile-PSQL_DB_NAME
      db.userprofile-production.datasource.user=PSQL_USER
      db.userprofile-production.datasource.password=PSQL_PWD
102

Ankur Kumar's avatar
Ankur Kumar committed
103
104
      webapp.manager=TOMCAT_USER
      webapp.password=TOMCAT_PWD
Julie Sullivan's avatar
Julie Sullivan committed
105
```
Ankur Kumar's avatar
Ankur Kumar committed
106

Julie Sullivan's avatar
Julie Sullivan committed
107
108
109
The build will replace these placeholders with the generated usernames and passwords.

### Change default settings (optional)
Ankur Kumar's avatar
Ankur Kumar committed
110

Ankur Kumar's avatar
Ankur Kumar committed
111
112
113
You can configure a lot of options by creating a `.env` file in the current working directory and adding the required key value pairs. These are used as env vars by docker-compose. For example:
```bash
MINE_NAME=humanmine
Julie Sullivan's avatar
Julie Sullivan committed
114
MINE_REPO_URL=https://github.com/intermine/humanmine
Ankur Kumar's avatar
Ankur Kumar committed
115
116
IM_DATA_DIR=/tmp/data
```
Ankur Kumar's avatar
Ankur Kumar committed
117
118
119

Available options are:
 - MINE_NAME
120
121
122
123
124
125
126

   Set your desired mine name here.

 - MINE_REPO_URL
   
   Add you mine github/gitlab url. It will be used to pull your mine inside build container

127
128
129
130
 - BIOSOURCES_REPO_URL

  Add you bio-sources repo github/gitlab url. It will be used to pull your bio sources inside build container    

131
132
133
134
 - IM_DATA_DIR

   Set it to the data dir prefix that you have in your project.xml file. This will be used for search and replace to fix data locations inside the build container.

Ankur Kumar's avatar
Ankur Kumar committed
135
    > Note: DO NOT add the trailing "/"
Julie Sullivan's avatar
Julie Sullivan committed
136
Here are the ENV variables available.
Ankur Kumar's avatar
Ankur Kumar committed
137

Julie Sullivan's avatar
Julie Sullivan committed
138
139
140
141
142
143
| ENV variable  | Notes | 
| ------------- | ------------- |
| MINE_REPO_URL | the git url of your mine | 
| MINE_NAME  | Name of your mine |
| IM_DATA_DIR | Data directory as used in your project XML file. This will be used for search and replace to fix data locations inside the build container.| 
| TOMCAT_HOST_PORT  | the port of your host machine to which tomcat docker container binds to | 
144
145
146
147
148


# load data to pvc

```
Remi  PLANEL's avatar
Remi PLANEL committed
149
150

# Create PVC
Remi  PLANEL's avatar
Remi PLANEL committed
151
Create PVC: kubectl apply -n legiolist-dev -f ci/kubernetes/pvc-data.yaml
Remi  PLANEL's avatar
Remi PLANEL committed
152

153
kubectl apply -f ci/kubernetes/intermine-pod-load-data.yaml --namespace legiolist-dev
154
155
156
157
158
159
160
161
162

kubectl get pod -n legiolist-dev
NAME                          READY   STATUS    RESTARTS   AGE
intermine-load-data           1/1     Running   0          45s
postgresql-5c8975bd8c-6k7bg   1/1     Running   0          9m14s
solr-5fb4768f98-ggcph         1/1     Running   0          9m14s
tomcat-77855d8f44-9wbgf       1/1     Running   0          9m14s

kubectl  --namespace=legiolist-dev cp /mnt/gaia/home/projects/legiolist/data/pre-process-data/ intermine-load-data:/data
Remi  PLANEL's avatar
Remi PLANEL committed
163

164
kubectl  --namespace=legiolist-dev cp /mnt/gaia/home/projects/legiolist/data/so.obo intermine-load-data:/data
Remi  PLANEL's avatar
Remi PLANEL committed
165

166
kubectl  --namespace=legiolist-dev cp /mnt/gaia/home/projects/legiolist/data/data_newLegioList intermine-load-data:/data
Remi  PLANEL's avatar
Remi PLANEL committed
167
168
169

# Delete pod
kubectl delete -n legiolist-dev pod intermine-load-data
170
```