Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
Rayan CHIKHI
serratus-batch-dl
Commits
1bac8ece
Commit
1bac8ece
authored
Jun 16, 2020
by
Rayan CHIKHI
Browse files
one more attempt at making an EBS volume..
parent
3dee182a
Changes
3
Hide whitespace changes
Inline
Side-by-side
src/Dockerfile
View file @
1bac8ece
...
...
@@ -45,6 +45,6 @@ RUN wget --quiet http://opengene.org/fastp/fastp
RUN
chmod
+x fastp
RUN
pwd
RUN
df
-
h
.
RUN
df
-
T
RUN
ls
src/batch_processor.py
View file @
1bac8ece
...
...
@@ -25,11 +25,13 @@ def process_file(accession, region):
# go to /tmp (that's where local storage / nvme is)
# go to /data instead (EBS)
os
.
chdir
(
"/data"
)
os
.
chdir
(
"/
mnt/serratus-
data"
)
os
.
system
(
' '
.
join
([
"pwd"
]))
# check free space
os
.
system
(
' '
.
join
([
"df"
,
"-h"
,
"."
]))
os
.
system
(
' '
.
join
([
"df"
,
"-T"
]))
os
.
system
(
' '
.
join
([
"df"
,
"-h"
,
"."
]))
os
.
system
(
' '
.
join
([
"lsblk"
]))
# download reads from accession
os
.
system
(
'mkdir -p out/'
)
...
...
template/template.yaml
View file @
1bac8ece
...
...
@@ -110,15 +110,15 @@ Resources:
- Ref
:
AWS::Region
- ".amazonaws.com/serratus-dl-batch-job:latest"
Vcpus
:
4
Memory
:
6000
Memory
:
8000
# seems that 4000 wasn't enough for parallel-fastq-dump, 4 threads
MountPoints
:
-
ContainerPath
:
"
/data"
-
ContainerPath
:
"
/
serratus-
data"
ReadOnly
:
false
SourceVolume
:
data
SourceVolume
:
serratus-
data
Volumes
:
-
Name
:
data
-
Name
:
serratus-
data
Host
:
SourcePath
:
"
/data"
SourcePath
:
"
/
serratus-
data"
RetryStrategy
:
Attempts
:
1
RayanSerratusDlBatchProcessingJobQueue
:
...
...
@@ -136,8 +136,8 @@ Resources:
Type
:
MANAGED
ComputeResources
:
Type
:
SPOT
MinvCpus
:
0
DesiredvCpus
:
0
MinvCpus
:
4
DesiredvCpus
:
4
MaxvCpus
:
1000
AllocationStrategy
:
SPOT_CAPACITY_OPTIMIZED
InstanceTypes
:
...
...
@@ -162,11 +162,11 @@ Resources:
LaunchTemplateName
:
"
Special-inc-nvme-assembly"
LaunchTemplateData
:
BlockDeviceMappings
:
-
DeviceName
:
'
/dev/sd
b
'
# Amazon ECS-Optimized Amazon Linux 2 ami accepts EDS vol /dev/sd[b-l]
-
DeviceName
:
'
/dev/sd
h
'
# Amazon ECS-Optimized Amazon Linux 2 ami accepts EDS vol /dev/sd[b-l]
Ebs
:
DeleteOnTermination
:
true
Encrypted
:
tru
e
VolumeSize
:
1000
# high because of multitenancy
Encrypted
:
fals
e
VolumeSize
:
500
VolumeType
:
"
gp2"
UserData
:
"
Fn::Base64"
:
!Sub
|
...
...
@@ -175,21 +175,16 @@ Resources:
--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
mkfs.ext4 /dev/sdb
(
echo n # Add a new partition
echo p # Primary partition
echo 1 # Partition number
echo # First sector (Accept default: 1)
echo # Last sector (Accept default: varies)
echo w # Write changes
) | fdisk /dev/sdb
mkfs.ext4 /dev/xvdb1
mkdir /data
echo " /dev/xvdb1 /data ext4 defaults 0 2" >> /etc/fstab
#!/bin/bash -xe
# https://stackoverflow.com/questions/41073906/how-to-attach-and-mount-volumes-to-an-ec2-instance-using-cloudformation
mkfs.ext4 /dev/xvdh
mkdir /mnt/serratus-data
echo -e "/dev/xvdh\t/mnt/serratus-data\text4\tdefaults\t0\t0" >> /etc/fstab
mount -a
chmod 777 /serratus-data
service docker restart
--==MYBOUNDARY==--
Outputs
:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment