Add blueprint-based pipeline construction utility with Sphinx documentation and asimov integration

[Copilot]

Gravitic lacked a pipeline construction utility for joining scripts into executable workflows. This adds YAML blueprint support similar to GitHub Actions/GitLab CI, enabling pipeline definition with sequential steps and asset tracking for integration with asimov workflows.

Implementation

Core module (gravitic/blueprint.py):

• Blueprint - Parses and validates YAML pipeline definitions
• StepExecutor - Executes scripts with configurable shells (bash, Python, etc.)
• PipelineRunner - Orchestrates sequential step execution in shared working directory
• Asset collection and verification for outputs

CLI enhancement:

• gravitic run <blueprint.yml> command with progress reporting

Asimov integration (gravitic/asimov.py):

• GraviticPipeline - Implements asimov Pipeline interface for workflow automation
• HTCondor DAG construction and job submission
• Asset-based completion detection
• Blueprint template system for production metadata

Documentation:

• Comprehensive Sphinx documentation with kentigern theme
• Automated GitHub Pages deployment via GitHub Actions
• User guides (quickstart, blueprints, asimov integration)
• API reference with autodoc

Dependencies:

• Added pyyaml to requirements

Blueprint Format

kind: analysis
pipeline: transientlunatic/gravitic
steps:
- name: Generate data
  script: |
    echo "data" > input.dat
- name: Analyze
  shell: /usr/bin/python3
  script: |
    import json
    with open('input.dat') as f:
        result = {"data": f.read()}
    with open('output.json', 'w') as f:
        json.dump(result, f)
assets:
  posterior:
    - output.json

Usage

Standalone

$ gravitic run analysis.yml
[gravitic] Starting pipeline: transientlunatic/gravitic
[gravitic] Running: Generate data
[gravitic] Running: Analyze
[gravitic] Pipeline completed successfully

With Asimov

from gravitic.asimov import GraviticPipeline

pipeline = GraviticPipeline(production)
pipeline.build_dag()
pipeline.submit_dag()

Documentation

Documentation is automatically built and deployed to GitHub Pages on push to master. Includes:

• Getting started guide
• Complete blueprint format reference
• Asimov integration examples
• API reference

Security

Blueprints execute arbitrary shell commands by design (like GitHub Actions). Documentation includes warnings about trust models and recommendations for containerized execution (Docker/Podman/Apptainer) in production environments.

Testing

Added comprehensive test suite covering blueprint loading, validation, step execution, pipeline orchestration, and asset collection (9 tests, all passing).

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Create gravitic</issue_title>
<issue_description>Gravitic is an old project which I started to design which would allow LVK-style analysis pipelines to be made more modular.
In the end this has been partly superceded by asimov (https://github.com/etive-io/asimov) which can run workflows composed of multiple pipelines glued together. You might find some incoming asimov features useful to (etive-io/asimov#86).

However, there is still no good pipeline construction utility for the ecosystem, and it relies on software already having a pipelining setup.

Gravitic should solve this problem, by allowing scripts to be joined together into pipelines which asimov can then run in workflows. These scripts can call e.g. a simple analysis, or run common shell commands.

My image for this is that the pipeline should be defined in a similar way to e.g. a github actions or gitlab CI workflow, and allows e.g. outputs to be specified in a blueprint file so that we can construct an interface with asimov similar to the one which is presented by pipelines like bilby.

Here's how I think a blueprint should look to the user:

kind: analysis
pipeline: transientlunatic/gravitic
comment: This is a test analysis using gravitic!
steps:
- name: This is an optional name for this step
   script: |
     cat data.dat | wc > output.txt
- name: Step 2
  script: |
    analysis.py output.txt > posterior.dat
assets:
  posterior:
    - posterior.dat

Each step should run in the same environment but a separate shell (and we should be able to specify that shell so the script can be e.g. python), so that files etc are accessible in each step.

We need to think carefully about security here; I think we probably ought to run the pipeline in either docker, apptainer, or podman.

</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Create gravitic #1

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[transientlunatic]

@copilot This looks like a proming start! But some things are missing, like sphinx documentation (use the kentigern theme, and deploy automatically to github pages with github actions), and more importantly, the integration with asimov!

You can see how other pages do that e.g. here: https://github.com/etive-io/asimov-cogwheel

[Copilot]

@copilot This looks like a proming start! But some things are missing, like sphinx documentation (use the kentigern theme, and deploy automatically to github pages with github actions), and more impor...

Added all requested features in commit f419d1b:

1. Sphinx documentation with kentigern theme - Comprehensive docs covering quickstart, blueprints, asimov integration, and API reference
2. GitHub Pages deployment - Automated workflow (.github/workflows/build-docs.yaml) builds and deploys docs on push to master
3. Asimov integration - GraviticPipeline class in gravitic/asimov.py implements the asimov Pipeline interface with DAG construction, job submission, and completion detection

The asimov integration follows the same pattern as asimov-cogwheel, with blueprint templates, HTCondor DAG support, and asset-based completion detection.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• docs.python.org
  • Triggering command: /home/REDACTED/.local/bin/sphinx-build sphinx-build -M html . _build ACCEPT (dns block)
  • Triggering command: /home/REDACTED/.local/bin/sphinx-build sphinx-build -M html . _build (dns block)
• ligo-asimov.readthedocs.io
  • Triggering command: /home/REDACTED/.local/bin/sphinx-build sphinx-build -M html . _build ACCEPT (dns block)
  • Triggering command: /home/REDACTED/.local/bin/sphinx-build sphinx-build -M html . _build (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[Copilot]

Pull request overview

This pull request adds a comprehensive blueprint-based pipeline construction utility to Gravitic, enabling users to define analysis pipelines using YAML files similar to GitHub Actions or GitLab CI workflows. The implementation includes core blueprint functionality, asimov integration for workflow automation, Sphinx documentation with automated GitHub Pages deployment, and a CLI interface for running pipelines.

Changes:

• Added blueprint-based pipeline system with YAML configuration support for defining sequential analysis steps with asset tracking
• Integrated with asimov workflow management via GraviticPipeline class implementing HTCondor DAG construction and job submission
• Comprehensive Sphinx documentation with automated GitHub Pages deployment, including quickstart, blueprint reference, asimov integration guide, and API documentation

Reviewed changes

Copilot reviewed 23 out of 25 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
gravitic/blueprint.py	Core blueprint functionality including Blueprint, StepExecutor, and PipelineRunner classes for parsing, validating, and executing YAML-based pipelines
gravitic/asimov.py	Asimov integration providing GraviticPipeline class for HTCondor DAG construction, job submission, and completion detection
gravitic/cli/__init__.py	Added gravitic run command for executing blueprint files from the command line
gravitic/__init__.py	Module exports for blueprint functionality with graceful import handling for optional dependencies
gravitic/data/__init__.py	Made waveform import optional to improve compatibility
gravitic/asimov_template.yaml	Jinja2 template for rendering blueprint files with asimov production metadata
setup.py	Updated to use find_packages, include template as package data, and set zip_safe=False
requirements.txt	Added pyyaml dependency
MANIFEST.in	Package manifest for including template and documentation files
.gitignore	Added Python build artifacts and documentation build directories
.github/workflows/build-docs.yaml	GitHub Actions workflow for automated Sphinx documentation build and deployment to GitHub Pages
docs/conf.py	Enhanced Sphinx configuration with autodoc, viewcode, and intersphinx extensions
docs/index.rst	Updated main documentation page with project overview and navigation
docs/quickstart.rst	Getting started guide with installation and basic usage examples
docs/blueprints.rst	Comprehensive blueprint format reference with multiple examples
docs/asimov.rst	Asimov integration guide with configuration and usage examples
docs/api.rst	API reference documentation using autodoc
README.rst	Added blueprint pipeline documentation and security notes
tests/test_blueprint.py	Comprehensive test suite with 9 tests covering blueprint loading, validation, execution, and asset collection
tests/__init__.py	Test package initialization
example_blueprint.yml	Example bash-based pipeline
example_python_blueprint.yml	Example Python-based pipeline
QUICK_REFERENCE.md	Quick reference guide for blueprint format and usage
IMPLEMENTATION_SUMMARY.md	Detailed implementation summary document
DOCUMENTATION_SUMMARY.md	Documentation and asimov integration summary

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The install_requires parameter is commented out, which means the package dependencies (pyyaml, click) won't be automatically installed when users install gravitic. Uncomment this line to ensure dependencies are installed correctly. The requirements list is already being read from requirements.txt on line 9.

Suggested change

	# install_requires=requirements,
	install_requires=requirements,

[Copilot]

The config_template is being set at class definition time using a context manager. However, this pattern may not work as expected because the context manager will exit immediately after the with block, potentially closing the resource. Consider using importlib.resources.files() (Python 3.9+) or read_text() to load the template content, or ensure the path object is properly handled outside the context manager.

Suggested change

	with importlib.resources.path("gravitic", "asimov_template.yaml") as template_file:
	config_template = template_file
	# Use importlib.resources.files() to obtain a stable reference to the
	# template resource without relying on a context manager whose scope
	# would end at class definition time.
	template_resource = importlib.resources.files("gravitic").joinpath("asimov_template.yaml")
	config_template = template_resource if template_resource.is_file() else None

[Copilot]

The getenv="True" setting in HTCondor submit description is deprecated in newer versions of HTCondor. Consider using getenv=True (boolean) instead of the string "True", or use the more modern approach with explicit environment variable passing via the environment parameter.

Suggested change

	getenv="True",
	getenv=True,

[Copilot]

When dryrun is True, this method doesn't return a value, which is inconsistent with the non-dryrun case. Consider returning None explicitly when dryrun is True to make the behavior clear, or return a sentinel value to indicate a dry run was performed.

Suggested change

	return cluster_id
	return cluster_id
	return None

[Copilot]

The blueprint variable is set to a path relative to cwd, but then passed to the gravitic run command without ensuring it will be accessible from the rundir. If the DAG runs from a different working directory than cwd, the blueprint file may not be found. Consider using an absolute path or copying the blueprint to the rundir.

Suggested change

	# Create the gravitic run command
	run_command = f"run {blueprint}"
	# Resolve blueprint to an absolute path to ensure it is accessible from the run directory
	blueprint_path = Path(blueprint).resolve()
	# Create the gravitic run command
	run_command = f"run {blueprint_path}"