Restructure biahub

[ieivanov]

Here I'm suggesting a restructure of the biahub repo to make it easier to understand, browse, and contribute to.

Make code easier to find

biahub currently has roughly the following structure:

biahub/
    analysis/
        deskew.py
        deconvolve.py
        ...
    cli/
        deskew.py
        deconvolve.py
        stabilize.py
        main.py
        ...
    tests/

The analysis module is a remnant from shrimpy where we had acquisition and analysis modules. Currently, it's confusing and hard to know what functions go under analysis vs cli. I suggest removing the analysis and cli modules to achieve a structure like this:

biahub/
    deskew.py
    deconvolve.py
    stabilize.py
    cli.py
    ...
    tests/

The new deskew.py module will combine functions from analysis/deskew.py and cli/deskew.py and define a deskew_cli() function which will be imported in cli.py, which will have the following structure:

import click

from biahub.deskew import deskew_cli
...

@click.group()
def main()

main.add_command(deskew_cli)
...

Make example config files easier to find

Our example configs, which are essential to find for any new user and buried in

biahub/
    analysis/
        settings/
            example_deskew_settings.yml
            example_deconvolve_settings.yml (actually currently missing)
            ...

We should instead move them to

biahub/
    deskew.py
    deconvolve.py
    stabilize.py
    cli.py
    tests/
settings/
    example_deskew_settings.yml
    example_deconvolve_settings.yml

@mattersoflight @edyoshikun @talonchandler @ziw-liu Let me know what you think about this plan. It may be helpful to merge large open PRs before this restructure, but I don't think that should be a blocker.

[ziw-liu]

This plan looks good to me! One minor thing thing that could be changed is where the example settings are. Is there a reason to distribute them in the package? It will be more discoverable if it lives outside, in a top-level examples directory in the repository.

[ieivanov]

@ziw-liu I agree, I was thinking about that too

[edyoshikun]

I agree with all these points in the first comment and this one regarding the location of the examples.

I would directly benefit from merging and reviewing the following:

• fix to channel names during concat #25 Concat CLI bug
• Patch segmentation #53 Segmentation CLI bug
• Add process-with-config CLI #29 Uni processing (Biahub with any czyx function)
• Track CLI using uni_processing #40 tracking CLI

[mattersoflight]

@ieivanov I do like the structure of keeping the methods and implementations of cli in same file. It will make refactors more efficient. It also makes sense to bring settings up in the hierarchy.

I understand there is fair bit of common machinery used for parallelization via submitit. If there are repeated code blocks used by all CLI to do so, it also makes sense to create a utils module that provides these functions.

[ziw-liu]

it also makes sense to create a utils module that provides these functions.

These can probably be put into the current cli module with specific names. For example printing job monitoring information is a function that every CLI needs:

biahub/
-  cli.py
+  cli/
+    __init__.py
+    main.py
+    monitoring.py

[ieivanov]

Thanks, I'll keep that in mind as I rearrange things. We do also have option_eat_all.py, and parsing.py - it seems like cli should remain a module.