Reproducible Environments with Singularity

Steffen Ridderbusch

JuliaCon 2020

Link in github.com/Crown421/Singularity.jl

Imagine…

Reproducible results

Some reading
https://www.jarrodmillman.com/oss-chapter.html
https://www.nature.com/articles/d41586-018-05990-5

We need

Containers

Docker Docker Singularity

Potential caveats

• no layers: containers can’t share base

• read-only (for non-superuser)

  • good: preserves integrity
  • bad: must precompile at build

• run via

  • full Singularity
  • beta without builder
  • via WSL

Base Image

as small as possible

Demo > singularity pull –arch amd64 library://crown421/default/juliabase:1.4.2
INFO:    Container is trusted - run ‘singularity key list’ to list your trusted keys
INFO:    Download complete: juliabase_1.4.2.sif
Demo > du -sh juliabase_1.4.2.sif
138M    juliabase_1.4.2.sif
Demo > singularity shell –no-home juliabase_1.4.2.sif
Singularity> julia
               _
   _       _ ()_     |  Documentation: https://docs.julialang.org
  ()     | () (_)    |
   _ _   | |  __ _   |  Type “?” for help, “]?” for Pkg help.
  | | | | | | |/ _|  |
  | | |_| | | | (_| |  |  Version 1.4.2 (2020-05-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |
(@v1.4) pkg> add Distances
    Cloning default registries into/home/steffen/Code/Demo/containerhome/.julia
    Cloning registry from "https://github.com/JuliaRegistries/General.git"
      Added registryGeneralto/home/steffen/Code/Demo/containerhome/.julia/registries/General
  Resolving package versions...
  Installed Distances ─ v0.9.0
   Updating/home/steffen/Code/Demo/containerhome/.julia/environments/v1.4/Project.toml
  [b4f34e82] + Distances v0.9.0
   Updating/home/steffen/Code/Demo/containerhome/.julia/environments/v1.4/Manifest.toml`
  [b4f34e82] + Distances v0.9.0
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
julia> using Distances
[ Info: Precompiling Distances [b4f34e82-e78d-54a5-968a-f98e89d6e8f7]
julia> euclidean(rand(3), rand(3))
0.6122474761590728
julia> exit()
Singularity> exit
exit
Demo > tree -a -L 3
.
├── containerhome
│   └── .julia
│       ├── compiled
│       ├── environments
│       ├── logs
│       ├── packages
│       └── registries
└── juliabase_1.4.2.sif
7 directories, 1 file
Demo >

Goal

everything neatly packed

helper package github.com/Crown421/Singularity.jl

container a project

ContaineredExample >  master > tree -L 3
.
├── data
│   ├── exp_pro
│   ├── exp_raw
│   └── sims
├── Manifest.toml
├── notebooks
├── old_result.bson
├── papers
├── plots
├── Project.toml
├── README.md
├── _research
├── scripts
│   └── intro.jl
└── src
    └── ExampleModule
        ├── Manifest.toml
        ├── Project.toml
        └── src


12 directories, 8 files
ContaineredExample >  master > julia –project
               _
   _       _ ()_     |  Documentation: https://docs.julialang.org
  ()     | () (_)    |
   _ _   | |  __ _   |  Type “?” for help, “]?” for Pkg help.
  | | | | | | |/ _|  |
  | | |_| | | | (_| |  |  Version 1.4.1 (2020-04-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |
</br>
julia> using Singularity
julia> generate_deffile()
julia> buildsif()
runningsudo singularity build –force container/projectcontainer.sif container/Singularity.pack
[sudo] password for steffen: 
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Running setup scriptlet
++ pwd
+ dir=/home/steffen/Code/ContaineredExample
+ git clone file:///home/steffen/Code/ContaineredExample /tmp/rootfs-ee0425d8-c83c-11ea-9229-eee13580103b/Project
Cloning into '/tmp/rootfs-ee0425d8-c83c-11ea-9229-eee13580103b/Project'...
remote: Enumerating objects: 49, done.
remote: Counting objects: 100% (49/49), done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 49 (delta 12), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (49/49), 8.31 KiB | 2.08 MiB/s, done.
Resolving deltas: 100% (12/12), done.
INFO:    Running post scriptlet
+ export JULIA_DEPOT_PATH=/user/.julia
+ export PATH=/opt/julia/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
+ cd Project
+ git checkout master -- src/ scripts/ Project.toml Manifest.toml
+ julia --project -e using Pkg; Pkg.rm.(Any[])
+ julia --project -e using Pkg; Pkg.instantiate()
    Cloning default registries into/user/.julia
    Cloning registry from "https://github.com/JuliaRegistries/General.git"
      Added registryGeneralto/user/.julia/registries/General
  Installed Requires ── v1.0.1
  Installed UnPack ──── v1.0.1
  Installed Distances ─ v0.9.0
  Installed DrWatson ── v1.14.4
  Installed FileIO ──── v1.3.0
  Installed BSON ────── v0.2.6
+ julia --project -e using Pkg; Pkg.precompile()
Precompiling project...
+ chmod -R a+rX /user/.julia
+ chmod -R a+rX /Project/scripts
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: container/projectcontainer.sif
Process(sudo singularity build –force container/projectcontainer.sif container/Singularity.pack`, ProcessExited(0))
julia> exit()


ContaineredExample >  git status
On branch master
Your branch is ahead of ‘origin/master’ by 3 commits.
  (use “git push” to publish your local commits)
Changes not staged for commit:
  (use “git add …” to update what will be committed)
  (use “git restore …” to discard changes in working directory)
        modified:   .gitignore
Untracked files:
  (use “git add …” to include in what will be committed)
        container/
no changes added to commit (use “git add” and/or “git commit -a”)


ContaineredExample >  cd container
container >  tree
.
├── pack.log
├── projectcontainer.sif
└── Singularity.pack


0 directories, 3 files
container >  du -sh projectcontainer.sif
221M    projectcontainer.sif
container >  singularity run –bind=../data:/Project/data projectcontainer.sif intro.jl
container >  cd ..
ContaineredExample >  tree data container
data
├── exp_pro
├── exp_raw
└── sims
    └── compute_x0=7.bson
container
├── intro.jl-2020-07-17T160055.log
├── pack.log
├── projectcontainer.sif
└── Singularity.pack


3 directories, 5 files

Reproduce results

ContaineredExample >  julia –project
               _
   _       _ ()_     |  Documentation: https://docs.julialang.org
  ()     | () (_)    |
   _ _   | |  __ _   |  Type “?” for help, “]?” for Pkg help.
  | | | | | | |/ |  |
  | | |_| | | | (_| |  |  Version 1.4.1 (2020-04-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |
</br>
julia> using DrWatson
</br>
julia> load("data/sims/compute_x0=7.bson")
Dict{Symbol,Any} with 5 entries:
  :result    => 42
  :params    => (x0 = 7,)
  :gitcommit => "v0.2-3-g83b59df"
  :script    => "scripts/intro.jl#12"
  :gitpatch  => ""
</br>
julia> load("old_result.bson")
Dict{Symbol,Any} with 5 entries:
  :result    => 43
  :params    => (x0 = 7,)
  :gitcommit => "v0.2-1-g185718b"
  :script    => "scripts/intro.jl#12"
  :gitpatch  => ""
</br>
julia> using Singularity
julia> recreatedata("old_result.bson")
julia> buildsif()
runningsudo singularity build –force container/projectcontainer.sif container/Singularity.pack
[sudo] password for steffen: 
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Running setup scriptlet
++ pwd
+ dir=/home/steffen/Code/ContaineredExample
+ git clone file:///home/steffen/Code/ContaineredExample /tmp/rootfs-0d3b9d22-c846-11ea-9d0d-eee13580103b/Project
Cloning into '/tmp/rootfs-0d3b9d22-c846-11ea-9d0d-eee13580103b/Project'...
remote: Enumerating objects: 49, done.
remote: Counting objects: 100% (49/49), done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 49 (delta 12), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (49/49), 8.31 KiB | 8.31 MiB/s, done.
Resolving deltas: 100% (12/12), done.
INFO:    Running post scriptlet
+ export JULIA_DEPOT_PATH=/user/.julia
+ export PATH=/opt/julia/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
+ cd Project
+ git checkout v0.2-1-g185718b -- src/ scripts/ Project.toml Manifest.toml
+ julia --project -e using Pkg; Pkg.rm.(Any[])
+ julia --project -e using Pkg; Pkg.instantiate()
    Cloning default registries into/user/.julia
    Cloning registry from "https://github.com/JuliaRegistries/General.git"
      Added registryGeneralto/user/.julia/registries/General
  Installed Requires ── v1.0.1
  Installed UnPack ──── v1.0.1
  Installed Distances ─ v0.9.0
  Installed BSON ────── v0.2.6
  Installed FileIO ──── v1.3.0
  Installed DrWatson ── v1.14.4
+ julia --project -e using Pkg; Pkg.precompile()
Precompiling project...
+ chmod -R a+rX /user/.julia
+ chmod -R a+rX /Project/scripts
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: container/projectcontainer.sif
Process(sudo singularity build –force container/projectcontainer.sif container/Singularity.pack, ProcessExited(0))
</br>
julia> exit()
ContaineredExample >  cd container 
container >  singularity run --bind=.:/Project/data projectcontainer.sif
┌ Warning: The Git repository is dirty! Appending _dirty to the commit ID
└ @ DrWatson ~/.julia/packages/DrWatson/LGFKJ/src/saving_tools.jl:60
 container >  julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _ |  |
  | | || | | | (_| |  |  Version 1.4.1 (2020-04-14)
 / |_‘|||_‘_|  |  Official https://julialang.org/ release
|__/                   |


julia> using DrWatson
julia> load(“sims/compute_x0=7.bson”)
Dict{Symbol,Any} with 5 entries:
  :result    => 43
  :params    => (x0 = 7,)
  :gitcommit => “v0.2-3-g83b59df_dirty”
  :script    => “scripts/intro.jl#12”
  :gitpatch  => “diff –git a/src/ExampleModule/src/ExampleModule.jl …

Outlook

• soon
  • containers (basically working)
• experimental:
  • shallow clones in container
• next
  • improved robustness and tests
  • support for multiple def files