Dupblaster insertion

[MattWellie]

Restart of #61

Purpose

• Picard MarkDups is expensive and fragile
• Closes Bin off MarkDuplicates #33
• Linked to Adds a new image, containing dragmap, ora, and dupblaster images#311

Proposed Changes

• Inserts DupBlaster as an in-stream tool, as a Picard alternative
• Updates the Dockerfile to install an updated samtools and dupblaster alongside the workflow
• 2 new tools are installed conservatively, using a compilation layer to reduce final foorprint
• Inserts new methods to name-sort data (used in single-alignment case, and between merging and dupblasting), and coordinate sort explicitly - used after dupblasting

Depends on

• Adds a new image, containing dragmap, ora, and dupblaster images#311 - required in the single-file alignment pathway, where the whole workflow is carried out in the DRAGMAP image, instead of splitting between alignment [1-n] -> merge [1]

Checklist

• Related GitHub Issue created
• Tests covering new change
• Linting checks pass

[Johnnyassaf]

I like the explicit aspect here, but be careful the machine setting isnt interfering with anything here. This would affect nthreads and permissons to use job.memory maybe

in the rnaseq jobs i set them like this:

from cpg_flow.resources import HIGHMEM
HIGHMEM.set_resources(j=j, ncpu=nthreads, storage_gb=50)

[MattWellie]

That's why I've dropped down to explicit resourcing - there's a contradiction in CPG-Flow's resource profiling.

Line 1: "GCP is one thread per vCPU"
Method: get ncpu by starting with threads, dividing by two, and getting the nearest power of two

Based on Google's documentation there are 2 threads available per CPU, so the fixed value of 1 is incorrect, and affects calculations.

This conflict was meaning that if I request a 16 core VM manually, and I run with 16 threads it's fine. If I try to use all the helper methods I request 16 threads, it's converted to a 8 CPU machine, and the profile gives me 8 threads instead.

I CBA using the helper methods which work and ignoring the ones which don't. It made sense when the code contained a decision about whether to use a highmem profile or a standard one, but it's a memory & cpu intensive process, so always using highmem feels defensible.

[Johnnyassaf]

This is quite literally a backbone of the pipeline so I agree that if anything deserves highmem it'll be here

[EddieLF]

Love to see it

[EddieLF]

hate this v

[MattWellie]

It hates u 2