Adds analysis registration script

[MattWellie]

A candidate collection of methods/standalone script for registering outputs in metamist. This general process has been trialled in NF/Seqera workflows, but required the script to leave inside each workflow. Migrating this code upstream will mean each CPG-Utils install makes this available.

This is not a script per se, it's a parameterised metamist interaction for registering analyses. The previous incarnation in the seqera-trial repository was a script, because that's what NextFlow required. This implementation takes that same interaction, keeps a script-like entrypoint, and also adds the same functionality as an importable method. The exact same logic is exposed both ways, making this solution usable for both code-accessed and cli-accessed registrations.

• this started with @dancoates' OG standalone metamist script here
• that was boosted here, adding flexibility to register any analysis type, populating analysis meta, and adding secondary files to analysis records, which didn't previously exist in CPG-Flow workflows.

This file contains a __main__ entrypoint, or a direct-use create_new method. If called as a script the CLI looks like:

python -m cpg_utils.metamist_registration \
    --project <project_name> \
    --output "<path to a primary output file>" \
     --type <analysis-type> \
     --cohorts COHABC COHDEF *or* --sgs CPGABC CPGDEF \
     [--meta stage=test sequencing_type=genome] \
     [--secondary index=<path to an index file>] 

If run via import in code, usage looks like:

from cpg_utils.metamist_registration import create_new

create_new(
    project=<project>,
    output=<path/to/primary/output>,
    analysis_type=cram,
    meta={'stage': 'name', 'sequencing_type': 'genome'},
    secondary={'index': <path to an index file>}
    cohorts=['COHABC', 'COHDEF'],
)

This interface takes:

• either SG or Cohorts (lists of one or more, but each analysis can only be aligned to SG or Cohorts)
• a main analysis output path (string, typically a GS-path)
• an unlimited number of secondary files, each with a corresponding type (the type must already exist in metamist)
• an unlimited number of additional metadata key=values to be appended to the analysis record's meta (meta is not populated for any secondary files)

The result is:

• an analysis record of the indicated type, inside the project
• the analysis.meta dictionary is populated with all key-value pairs
• if required, the analysis.outputs contains a nested secondary_files dict, each with a key identifying the secondary output type.

The analysis record 388733 has been generated by this process as a demonstration, go check it out.

NB by writing analysis records in a way which enables secondary files, the analysis.output field is always an empty String. This is a deprecated field, so we're probably comfortable with this, but some processes/workflows still prioritise reading the output String instead of analysis.outputs.path

[jmarshall]

At the moment cpg-utils does not depend on metamist; meanwhile metamist does use cpg-utils. Also cpg-utils is purely a library for use from Python code and does not currently contain any scripts.

So this doesn't seem like a great fit for cpg-utils. Are there other places this facility could be added instead to make it widely available?

[MattWellie]

Strategically this is both a script and an importable process - allowing its use as a standalone script doesn't affect anything else in the library - taking behaviour we only previously accessed via prod-pipes and CPG-Flow, and casting it as a generically accessible function with a simple interface.

This core function is required across all workflows during the migration away from CPG-Flow/hail batch as our runner, so it should live somewhere communal, and CPG-utils is already ubiquitous either directly or as a dependency in all our workflows.

[Johnnyassaf]

From a design perspective I understand not wanting to introduce a dependency on metamist, and not wanting to risk a cyclic dependency.

We don't really have a communal scripts repo though and I think this script fits in cpg-utils as CPG communal code that can live as a script (just register jobs when you're done with work) or as a function (register outputs as part of a job). As long as we ensure that the metamist functions imported here don't import this cpg-utils script in the future, it should be fine.

[jmarshall]

I guess the onus is on the PR's proponents to demonstrate that the cyclic dependency is not an issue.

I see this as yet more grist for the mill of sorting out a suitable location for scripts.

[MattWellie]

I'm not sure where this talk of cyclic dependencies is coming from. There's no install-time cyclic issues, pip or any other package manager treats this as an "other and self" dependency resolution, not a recursion risk. There are cpg-utils imports within the metamist library, but a new import of cpg_utils.metamist_registration would have to be added for that to be a risk. Specifically it would have to be added in the graphql or exceptions modules, which would be wildly inappropriate.

This is adding new functionality to be used globally across the CPG, which is the exact remit of cpg-utils. It is not a script like we have previously discussed for a scripts repository (i.e. a one-shot isolated process), it is a behaviour which we need in all of our separate workflows. The ability to operate it via import or as a standalone entrypoint is a benefit (cpg-flow sets up workflows in python, nextflow stages are a series of command line calls), but this should not be confused for a script

[Johnnyassaf]

LGTme

[Alexander-Stuckey]

Few comments, mostly looks good

[Alexander-Stuckey]

Looks good to me