Container Assignment and AWS Batch Integration in Snakemake: Rule-Specific Container Configuration Options

[hmkim]

Hello, team!

Can I assign an individual container to each rule and run it on AWS Batch, as shown below? Or do I currently need the necessary tool installation for all rules in a single container?

https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html#running-jobs-in-containers

[phmoferring]

I made comment in another issue that addresses this use case: #31 (comment)

My solution is currently not the cleanest, but it is working to execute jobs in job-level containers on batch.

[radusuciu]

Also discovered this. So essentially, if you want to run rules in different containers, you have to do so through apptainer, within the executor container.

I'm probably going to go the monolithic container route, but looking at the plugin code it seems like it wouldn't be too hard to specify a per job container. Currently we have:

snakemake-executor-plugin-aws-batch/snakemake_executor_plugin_aws_batch/batch_job_builder.py

Line 79 in 2525ca0

"image": self.container_image,

In #36, I added support for configuring this like so:

    resources:
        aws_batch_container_image="my-custom-image:tag"

However, there is a caveat - the image you use must be based on snakemake/snakemake and not interfere with the environment that snakemake expects. For example, in the image I was running, I switched the USER in the Dockerfile and ended up with an error: PermissionError: [Errno 13] Permission denied: '/tmp/repo/.snakemake'. Perhaps it's possible to work around by creating a compatible image but that might be brittle.