Loading/activating PEPATAC crate_Errors

[Kmrouj]

Hi Nathan,
I'm working on sherlock (Stanford cluster)
I'm trying to load PEPATAC crate using the "bulker load databio/pepatac -b -r" command. During the load, I had this return
'INFO: Creating SIF file...
FATAL: While making image from oci registry: while building SIF from layers: while creating squashfs: create command failed: signal: killed:
mv: cannot stat ‘rpipe’: No such file or directory
------ Error building. Build script used: ------
#!/bin/sh

if [ ! -f "/home/users/karim90/simages/databio/rpipe" ]; then
singularity pull rpipe docker://databio/rpipe
mv rpipe /home/users/karim90/simages/databio/rpipe
fi

Container available at: /home/users/karim90/simages/databio/rpipe
Container available at: /home/users/karim90/simages/quay.io/biocontainers/samtools:1.9--h91753b0_8
Container available at: /home/users/karim90/simages/quay.io/biocontainers/ucsc-wigtobigwig:357--h35c10e6_3
Populating host commands
Loading manifest: 'databio/pepatac:default'. Activate with 'bulker activate databio/pepatac:default'.
Commands available: bedGraphToBigWig, bedToBigBed, bedtools, bigWigCat, bowtie2, fastqc, java, macs2, pigz, preseq, samblaster, skewer, R, Rscript, samtools, wigToBigWig
Host commands available: python3, Perl"

When I tried to activate bulker using the "bulker activate databio/pepatac" command on sherlock, I had this error

[karim90@sh02-ln03 login ]$ bulker activate databio/pepatac
Bulker config: /home/users/karim90/bulker_config.yaml
Activating bulker crate: databio/pepatac
Error for command "pull": unknown shorthand flag: 'n' in -n
"Run 'singularity pull --help' for more detailed usage information.
mv: cannot stat ‘alpine-coreutils’: No such file or directory
FATAL: could not open image /home/users/karim90/simages/databio/alpine-coreutils: failed to retrieve path for /home/users/karim90/simages/databio/alpine-coreutils: lstat /home/users/karim90/simages/databio/alpine-coreutils: no such file or directory
databio/pepatac|$ exit"

Thank you for your help

[nsheff]

singularity is not correctly pulling that image... can you do this independet of bulker?

singularity pull rpipe docker://databio/rpipe

[Kmrouj]

Hi Nathan,
I tried: "singularity pull rpipe docker://databio/rpipe" but it did not work
INFO: Creating SIF file...
"FATAL: While making image from oci registry: while building SIF from layers: while creating squashfs: create command failed: signal: killed:"

I managed to edit the "singularity_build.jinja2 " file to remove the -n flag:

#!/bin/sh

if [ ! -f "{{ pkg.singularity_fullpath }}" ]; then
singularity pull {{ pkg.singularity_image }} docker://{{ pkg.docker_image }}
mv {{ pkg.singularity_image }} {{ pkg.singularity_fullpath }}
fi
However when I started again loading and activating I had the same type of error:
[karim90@sh01-ln03 login ~]$ bulker activate databio/pepatac
Bulker config: /home/users/karim90/bulker_config.yaml
Activating bulker crate: databio/pepatac
Error for command "pull": unknown shorthand flag: 'n' in -n

Run 'singularity pull --help' for more detailed usage information.
mv: cannot stat ‘alpine-coreutils’: No such file or directory
FATAL: could not open image /home/users/karim90/simages/databio/alpine-coreutils: failed to retrieve path for /home/users/karim90/simages/databio/alpine-coreutils: lstat /home/users/karim90/simages/databio/alpine-coreutils: no such file or directory
databio/pepatac|~$

[nsheff]

If you can't do "singularity pull rpipe docker://databio/rpipe" in general, then there's some problem with singularity that is not related to bulker. might need to ask the sys admins or something, maybe they disabled creation of singularity images?

[Kmrouj]

Hi,
Good news, with the help of Stanford personal (cluster) we were able to figure out where was the problem (with singularity) we used 4 cores to pull the singularity image:

[karim90@sh02-ln01 login ~]$ srun -c 4 --pty bash
[karim90@sh02-01n30 ~]$ cd $PI_HOME
[karim90@sh02-01n30 /home/groups/pbeachy]$ singularity pull rpipe docker://databio/rpipe
INFO: Converting OCI blobs to SIF format
INFO: Starting build...

INFO: Creating SIF file...
INFO: Build complete: rpipe

I want to ask you since I pulled directly the singularity image and now I have it in my $PI_HOME/Karim Directory I guess I don’t need to do: bulker activate databio/pepatac

Can I start with the test project, or there’s another step to do before.

Thanks

[nsheff]

well, bulker puts it in a specific spot that it manages, so you'd be best of now using bulker to build it...
you can type: bulker load -b databio/pepatac, for example, and it will build all of the singularity containers and put them in the correct places.

[Kmrouj]

Just a quick question regarding my directory. Right now the pulled image is here (I made a special dir for this):
[karim90@sh02-01n30 /home/groups/pbeachy/Karim/singularity/pepatac]$ ls
pipe

Can I stay in this same directory to "bulker load -b databio/pepatac" or move to the parent directory in my case /home/groups/pbeachy/Karim

Thanks

[nsheff]

the directory shouldn't matter, bulker will manage the locations according what you have listed in your bulker config file.

[Kmrouj]

It did not work

karim90@sh02-01n30 /home/groups/pbeachy/Karim/singularity/pepatac]$ bulker load -b databio/pepatac
Using default config. No config found in env var: BULKERCFG
Bulker config: /home/users/karim90/.local/lib/python3.6/site-packages/bulker/templates/bulker_config.yaml
Building images with template: /home/users/karim90/.local/lib/python3.6/site-packages/bulker/templates/docker_build.jinja2
Importing crate 'bulker/alpine:default' from '/home/users/karim90/bulker_crates/bulker/alpine/default'.
Importing crate 'bulker/coreutils:default' from '/home/users/karim90/bulker_crates/bulker/coreutils/default'.
sh: line 2: docker: command not found
------ Error building. Build script used: ------
#!/bin/sh

docker pull quay.io/biocontainers/ucsc-bedgraphtobigwig:357--h35c10e6_3

Traceback (most recent call last):
File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/ordattmap.py", line 45, in getitem
return super(OrdAttMap, self).getitem(item)
KeyError: 'singularity_fullpath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/users/karim90/.local/bin/bulker", line 8, in
sys.exit(main())
File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 870, in main
force=args.force)
File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 426, in bulker_load
_LOGGER.info("Container available at: {cmd}".format(cmd=pkg["singularity_fullpath"]))
File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/pathex_attmap.py", line 51, in getitem
v = super(PathExAttMap, self).getitem(item)
File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/ordattmap.py", line 47, in getitem
return AttMap.getitem(self, item)
File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/attmap.py", line 32, in getitem
return self.dict[item]
KeyError: 'singularity_fullpath'

[Kmrouj]

Here's my configuration for bulker_config.yaml:

[karim90@sh02-01n30 /home/groups/pbeachy]$ cat bulker_config.yaml
bulker:
volumes: ['$HOME']
envvars: ['DISPLAY']
registry_url: http://hub.bulker.io/
shell_path: ${SHELL}
shell_rc: ${HOME}/.bashrc
rcfile: templates/start.sh
rcfile_strict: templates/start_strict.sh
default_crate_folder: ${HOME}/bulker_crates
singularity_image_folder: ${HOME}/simages
container_engine: singularity
default_namespace: bulker
executable_template: templates/singularity_executable.jinja2
shell_template: templates/singularity_shell.jinja2
build_template: templates/singularity_build.jinja2
crates:
bulker:
demo:
default: /home/users/karim90/bulker_crates/bulker/demo/default
[karim90@sh02-01n30 /home/groups/pbeachy]$

[nsheff]

hi, you need to make sure to set up the config correctly. See the error message: Using default config. No config found in env var: BULKERCFG

So, it's not using the config. and therefore trying to use docker (the default), which you don't have installed. You have to pass it as -c or set up a BULKERCFG environment variable. I'd recommend reading the documentation, docs.bulker.io. You can find the part on how to configure bulker here: https://bulker.databio.org/en/latest/install/

[Kmrouj]

I followed the documentation by defining BULKERCFG environment variable but it did not work

[karim90@sh01-ln04 login ]$ export BULKERCFG=/home/users/karim90/bulker_config.yaml
[karim90@sh01-ln04 login ~]$ export BULKERCFG="bulker_config.yaml"
[karim90@sh01-ln04 login ~]$ cd $PI_HOME
[karim90@sh01-ln04 login /home/groups/pbeachy]$ cd /home/groups/pbeachy/Karim
[karim90@sh01-ln04 login /home/groups/pbeachy/Karim]$ bulker load -b databio/pepatac
Using default config. No config found in env var: BULKERCFG
Bulker config: /home/users/karim90/.local/lib/python3.6/site-packages/bulker/templates/bulker_config.yaml
Building images with template: /home/users/karim90/.local/lib/python3.6/site-packages/bulker/templates/docker_build.jinja2
Importing crate 'bulker/alpine:default' from '/home/users/karim90/bulker_crates/bulker/alpine/default'.
Importing crate 'bulker/coreutils:default' from '/home/users/karim90/bulker_crates/bulker/coreutils/default'.
sh: line 2: docker: command not found
------ Error building. Build script used: ------
#!/bin/sh

docker pull quay.io/biocontainers/ucsc-bedgraphtobigwig:357--h35c10e6_3

Traceback (most recent call last):
File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/ordattmap.py", line 45, in getitem
return super(OrdAttMap, self).getitem(item)
KeyError: 'singularity_fullpath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/users/karim90/.local/bin/bulker", line 8, in
sys.exit(main())
File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 870, in main
force=args.force)
File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 426, in bulker_load
_LOGGER.info("Container available at: {cmd}".format(cmd=pkg["singularity_fullpath"]))
File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/pathex_attmap.py", line 51, in getitem
v = super(PathExAttMap, self).getitem(item)
File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/ordattmap.py", line 47, in getitem
return AttMap.getitem(self, item)
File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/attmap.py", line 32, in getitem
return self.dict[item]
KeyError: 'singularity_fullpath'

[Kmrouj]

To give you a clear idea; Here's the full list of commands I have done (mainly by following the documentation) I also included the error message after try to load:

ml python/3.6.1
export BULKERCFG="bulker_config.yaml"
bulker init -c $BULKERCFG
export BULKERCFG=/home/users/karim90/bulker_config.yaml
bulker load -b databio/pepatac

#Error
Populating host commands
Traceback (most recent call last):
File "/home/users/karim90/.local/bin/bulker", line 8, in
sys.exit(main())
File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 870, in main
force=args.force)
File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 441, in bulker_load
os.symlink(local_exe, path)
FileExistsError: [Errno 17] File exists: '/share/software/user/open/python/3.6.1/bin/python3' -> '/home/users/karim90/bulker_crates/databio/pepatac/default/python3'