In certain situations you may want to run multiple sequential commands using the same job files, and some of those commands should be execute in different stacks. For example, consider a use case where one wants to pre-processes input files using stack A, then compute the output files using stack B, and finally generate figures using stack C. One advantage of separating these commands across three stacks is that it prevents you from requiring a single monolithic stack that contains all the dependencies.
The command job:exec ID command arg1 arg2 ... argN facilitates these types of workflows. It allows you to start a new job that uses the same files as an existing job, which we will refer to as the parent job.
The ID argument is the ID of the parent job with whom the files should be shared. The parent job can be a running job or an exited job.
Let’s see how to use the job:exec command with a simple example. First start an async job that runs our script
$ cjr job:start --async python3 script.py 5
a12d5bc00b965bf820cdcdecfd11804db13c8ece1d6cdcf9a2da37ac412b2378
Now let’s use exec job to run a new command that reruns the script again
$ cjr job:exec a12d5bc --async python3 script.py 3
779866e6d0b70486a6dcc2352ba28c34ceae2e18d17cab60a22c38255c97baf3
Finally, lets run a syncronous exec job that starts a bash shell and uses the ubuntu image
$ cjr job:exec a12d5bc --sync --stack=ubuntu:latest bash
root@9ee09cf8f98c:/example-project# ls
output-1600506296.txt output-1600506832.txt output-1600582790.txt output-1600582969.txt script.py
root@9ee09cf8f98c:/example-project# cat /etc/os-release
\NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
...
In each case we are accessing the job files of the parent job with id a12d5bc00b. In the final exec command we are using an interactive shell that is running ubuntu. This means that we can use any of the software in our ubuntu stack to post process the outputs.
When we are done manipulatin the files, we can sync the changes to our project directory by running cjr job:cp a12d5bc00b; notice that we are using the ID of the parent job and not the exec ids.
For more advanced tasks, it is possible to write bash scripts that start a new job, monitor the job states using the command cjr job:state ID, and then exec commands sequentially.
In the following example we run a job that launches our script, then follow the job with two exec jobs. The first runs in the image fedora:latest and the second runs in image ubuntu:latest.
#!/bin/bash
ID=$(cjr job:start --async python3 script.py 5)
while [ $(cjr job:state $ID) == "running" ]; do sleep 2; done
EID=$(cjr job:exec $ID --async --stack=fedora:latest bash -c 'cat /etc/os-release >> os.txt')
while [ $(cjr job:state $EID) == "running" ]; do sleep 2; done
EID=$(cjr job:exec $ID --async --stack=ubuntu:latest bash -c 'cat /etc/os-release >> os.txt')
while [ $(cjr job:state $EID) == "running" ]; do sleep 2; done
cjr job:cp $ID
The output file os.txt will shows that each exec was completed in a different operating system.
Advanced Users: If you are familiar with docker, then the cjr command job:exec should not be confused with the docker command exec. Exec jobs run in different containers and simply share a volume with the original job.