===== Running containerized jobs with singularity. =====

==== Downloading pre-built images ====

You can use the search command to locate groups, collections, and containers of interest on the [[https://cloud.sylabs.io/library|Container Library]].

You can use the pull and build commands to download pre-built images from an external resource like the [[https://cloud.sylabs.io/library|Container Library]] or [[https://hub.docker.com/|Docker Hub]].

When called on a native Singularity image like those provided on the Container Library, pull simply downloads the image file to your system.

You can also use pull with the docker:/ / uri to reference Docker images served from a registry. In this case pull does not just download an image file. Docker images are stored in layers, so pull must also combine those layers into a usable Singularity file.

<code bash>
singularity pull docker://godlovedc/lolcow
</code>

Pulling Docker images reduces reproducibility. If you were to pull a Docker image today and then wait six months and pull again, you are not guaranteed to get the same image. If any of the source layers has changed the image will be altered. If reproducibility is a priority for you, try building your images from the Container Library.

You can also use the build command to download pre-built images from an external resource. When using build you must specify a name for your container like so:

<code bash>
singularity build lolcow.sif docker://godlovedc/lolcow
</code>

Unlike pull, build will convert your image to the latest Singularity image format after downloading it.

build is like a “Swiss Army knife” for container creation. In addition to downloading images, you can use build to create images from other images or from scratch using a definition file. You can also use build to convert an image between the container formats supported by Singularity.

==== Interact with images ====

You can interact with images in several ways. It is not actually necessary to pull or build an image to interact with it. The commands listed here will work with image URIs in addition to accepting a local path to an image.

For these examples we will use a lolcow_latest.sif image that can be pulled from the Container Library like so.
<code bash>
singularity pull docker://godlovedc/lolcow
</code>

=== Shell ===

The shell command allows you to spawn a new shell within your container and interact with it as though it were a small virtual machine.

<code bash>
$ singularity shell lolcow_latest.sif

Singularity lolcow_latest.sif:~>
</code>

The change in prompt indicates that you have entered the container (though you should not rely on that to determine whether you are in container or not).

Once inside of a Singularity container, you are the same user as you are on the host system.

<code bash>
Singularity lolcow_latest.sif:~> whoami
david

Singularity lolcow_latest.sif:~> id
uid=1000(david) gid=1000(david) groups=1000(david),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),116(lpadmin),126(sambashare)
</code>

=== Executing Commands ===

The exec command allows you to execute a custom command within a container by specifying the image file. For instance, to execute the cowsay program within the lolcow_latest.sif container:

<code bash>
$ singularity exec lolcow_latest.sif cowsay moo
 _____
< moo >
 -----
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
</code>

=== Running a container ===

Singularity containers contain runscripts. These are user defined scripts that define the actions a container should perform when someone runs it. The runscript can be triggered with the run command, or simply by calling the container as though it were an executable.

<code bash>
$ singularity run lolcow_latest.sif
 _____________________________________
/ You have been selected for a secret \
\ mission.                            /
 -------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

$ ./lolcow_latest.sif
 ____________________________________
/ Q: What is orange and goes "click, \
\ click?" A: A ball point carrot.    /
 ------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
</code>

=== Working with files ===
Files on the host are reachable from within the container.
<code bash>
$ echo "Hello from inside the container" > $HOME/hostfile.txt

$ singularity exec lolcow_latest.sif cat $HOME/hostfile.txt

Hello from inside the container
</code>
This example works because hostfile.txt exists in the user’s home directory. By default Singularity bind mounts /home/$USER, /tmp, and $PWD into your container at runtime.

You can specify additional directories to bind mount into your container with the --bind option. In this example, the data directory on the host system is bind mounted to the /mnt directory inside the container.
<code bash>
$ echo "Drink milk (and never eat hamburgers)." > /data/cow_advice.txt

$ singularity exec --bind /data:/mnt lolcow_latest.sif cat /mnt/cow_advice.txt
Drink milk (and never eat hamburgers).
</code>
Pipes and redirects also work with Singularity commands just like they do with normal Linux commands.
<code bash>
$ cat /data/cow_advice.txt | singularity exec lolcow_latest.sif cowsay
 ________________________________________
< Drink milk (and never eat hamburgers). >
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
</code>

==== Build images from scratch ====
Singularity v3.0 produces immutable images in the Singularity Image File (SIF) format. This ensures reproducible and verifiable images and allows for many extra benefits such as the ability to sign and verify your containers.

However, during testing and debugging you may want an image format that is writable. This way you can shell into the image and install software and dependencies until you are satisfied that your container will fulfill your needs. For these scenarios, Singularity also supports the sandbox format (which is really just a directory).

For more details about the different build options and best practices, read about the Singularity flow.

=== Sandbox Directories ===
To build into a sandbox (container in a directory) use the build --sandbox command and option:
<code bash>
$ sudo singularity build --sandbox ubuntu/ library://ubuntu
</code>
This command creates a directory called ubuntu/ with an entire Ubuntu Operating System and some Singularity metadata in your current working directory.

You can use commands like shell, exec , and run with this directory just as you would with a Singularity image. If you pass the --writable option when you use your container you can also write files within the sandbox directory (provided you have the permissions to do so).
<code bash>
$ sudo singularity exec --writable ubuntu touch /foo

$ singularity exec ubuntu/ ls /foo
/foo
</code>

=== Converting images from one format to another ===
The build command allows you to build a container from an existing container. This means that you can use it to convert a container from one format to another. For instance, if you have already created a sandbox (directory) and want to convert it to the default immutable image format (squashfs) you can do so:
<code bash>
$ singularity build new-sif sandbox
</code>
Doing so may break reproducibility if you have altered your sandbox outside of the context of a definition file, so you are advised to exercise care.

=== Singularity Definition Files ===
For a reproducible, production-quality container you should build a SIF file using a Singularity definition file. This also makes it easy to add files, environment variables, and install custom software, and still start from your base of choice (e.g., the Container Library).

A definition file has a header and a body. The header determines the base container to begin with, and the body is further divided into sections that do things like install software, setup the environment, and copy files into the container from the host system.

Here is an example of a definition file:
<code bash>
BootStrap: library
From: ubuntu:16.04

%post
    apt-get -y update
    apt-get -y install fortune cowsay lolcat

%environment
    export LC_ALL=C
    export PATH=/usr/games:$PATH

%runscript
    fortune | cowsay | lolcat

%labels
    Author GodloveD
</code>
To build a container from this definition file (assuming it is a file named lolcow.def), you would call build like so:
<code bash>
$ sudo singularity build lolcow.sif lolcow.def
</code>
In this example, the header tells Singularity to use a base Ubuntu 16.04 image from the Container Library.

The %post section executes within the container at build time after the base OS has been installed. The %post section is therefore the place to perform installations of new applications.

The %environment section defines some environment variables that will be available to the container at runtime.

The %runscript section defines actions for the container to take when it is executed.

And finally, the %labels section allows for custom metadata to be added to the container.

This is a very small example of the things that you can do with a definition file. In addition to building a container from the Container Library, you can start with base images from Docker Hub and use images directly from official repositories such as Ubuntu, Debian, CentOS, Arch, and BusyBox. You can also use an existing container on your host system as a base.

If you want to build Singularity images but you don’t have administrative (root) access on your build system, you can build images using the [[https://cloud.sylabs.io/builder|Remote Builder]].

This [[https://sylabs.io/guides/3.0/user-guide/quick_start.html|quickstart]] document just scratches the surface of all of the things you can do with Singularity!

For more help type :
<code bash>
singularity help
singularity help [command]
</code>