New CTX repos

[jbcaillau]

@ocots @PierreMartinon @gergaud

New repositories for the control-toolbox ecosystem

Two complementary repositories are proposed: one fills the most significant algorithmic
gap in the ecosystem, the other addresses long-term developer infrastructure. They are
independent and can be pursued in parallel.

---

1. CTIndirect.jl — indirect methods for optimal control

Motivation

CTFlows.jl already provides the building blocks for indirect methods: it can integrate
Hamiltonian flows given an initial costate. What is missing is the layer above it —
the formulation and resolution of the shooting equations, and the continuation machinery
needed to handle hard problems. These currently live in research scripts and tutorials,
not in a reusable, versioned library. The effect is that indirect methods are a
second-class citizen in the ecosystem, despite being a primary method in practice for
problems in space mechanics, quantum control, and the calculus of variations.

CTIndirect.jl would give indirect methods the same standing as direct methods
(CTDirect.jl), occupying a symmetric slot in the algorithms layer.

Scope

Simple shooting. A ShootingProblem type wraps an OCP and an initial costate
guess. The shoot() function integrates the Hamiltonian flow via CTFlows, evaluates
the boundary conditions on the adjoint, and returns a residual. The residual is passed
to NonlinearSolve.jl for root-finding. This is the canonical single-shooting approach
for problems with a known switching structure.

Multiple shooting. An interior time mesh is introduced; state and costate are
simultaneously integrated on each arc and matched at interior nodes. This improves
numerical conditioning for long-horizon problems and those with complex switching.

Homotopy / continuation. A HomotopyPath type deforms a tractable reference
problem (e.g., quadratic cost, no state constraints) toward the target problem through
a sequence of intermediate problems. Each intermediate solution is used as the warm-start
for the next. Arc-length continuation is supported for turning points. This is the
core technique for the hardest problems — minimum-time orbital transfers, problems with
bang-bang or singular arcs, constrained quantum control — and has no current home in
the toolbox.

Proposed interface

using OptimalControl, CTIndirect

# Simple shooting
prob = ShootingProblem(ocp, p0_guess)
sol  = solve(prob, SimpleShooting(); nlp_solver=NewtonRaphson())

# Multiple shooting
prob = MultipleShootingProblem(ocp, p0_guess; nodes=10)
sol  = solve(prob, MultipleShooting(); nlp_solver=NewtonRaphson())

# Homotopy
path = HomotopyPath(ocp_easy => ocp_hard; steps=20)
sol  = solve(path)

The solve dispatch integrates with CTSolvers.jl so that indirect solvers appear
alongside direct ones in the unified solver registry. A combined workflow — indirect
warm-start from a direct solution, or vice versa — then requires no boilerplate.

Repository structure

CTIndirect.jl/
├── src/
│   ├── CTIndirect.jl           # module entry point, re-exports
│   ├── problems.jl             # ShootingProblem, MultipleShootingProblem, HomotopyPath
│   ├── shooting.jl             # residual computation, flow integration
│   ├── multiple_shooting.jl    # mesh, continuity conditions, interior nodes
│   └── homotopy.jl             # continuation loop, arc-length strategy
├── test/
│   ├── goddard.jl              # Goddard rocket — known analytical structure
│   ├── zermelo.jl              # Zermelo — simple bang-bang reference
│   └── mri.jl                  # MRI saturation — singular/homotopy showcase
├── docs/
│   ├── src/
│   │   ├── index.md
│   │   ├── shooting.md
│   │   ├── multiple_shooting.md
│   │   └── homotopy.md
│   └── make.jl
├── Project.toml
└── README.md

Dependencies

Dependency	Role
CTBase.jl	types, exceptions
CTModels.jl	OCP and solution data structures
CTFlows.jl	Hamiltonian flow integration
NonlinearSolve.jl	BVP root-finding
CTSolvers.jl	solver registry integration

No new heavy dependency is introduced. The package is a pure Julia layer on top
of existing infrastructure.

Position in the ecosystem

OptimalControl.jl
        │
   ┌────┴──────┬──────────────┐
CTParser  CTModels       CTSolvers
              │                │
         ┌────┴────┐     ┌─────┴──────┐
      CTDirect  CTFlows  CTIndirect ✦
         │          │         │
      (NLP)    (ODE/DAE)  (BVP/NLS)

---

2. CTDev — developer tooling for the multi-repo ecosystem

Context: the existing integration workflow

The multi-repo structure is already managed through a deliberate two-stage release
workflow. Beta versions of CTX packages are first registered in
ct-registry, a
LocalRegistry.jl-based Julia registry specific to the control-toolbox organization:

pkg> registry add https://github.com/control-toolbox/ct-registry.git
pkg> add CTDirect          # resolves from ct-registry before General

This local registry acts as the integration gate: a new version of, say, CTModels
can be published there as a beta and immediately consumed by downstream packages for
testing, without touching Julia's General registry. Only once the cross-package
combination is validated does the release propagate to General.

The crux: cascading updates inside the CT hierarchy

The real difficulty is not releasing individual packages — it is coordinating a
cascade of beta registrations triggered by a change anywhere in the dependency
hierarchy. The CT packages are not independent; they share a dependency graph with
several cross-edges:

CTBase
  ├── CTModels
  │     ├── CTParser
  │     ├── CTFlows
  │     │     └── CTDirect
  │     └── CTDirect
  │           └── CTSolvers
  └── (all of the above)
         └── OptimalControl

A change to CTBase — a new type, a renamed function, a revised abstract interface —
requires a coordinated update of every package that depends on it, directly or
transitively. In practice this means:

1. Register CTBase v0.19.0-beta to ct-registry.
2. Update [compat] in CTModels to include the new beta → register CTModels beta.
3. Update [compat] in CTParser and CTFlows → register their betas.
4. Update [compat] in CTDirect, which depends on both CTModels and CTFlows →
   register its beta.
5. Update [compat] in CTSolvers → register beta.
6. Update [compat] in OptimalControl → register beta.
7. Verify that the full combination works end-to-end.
8. When satisfied, re-register each package to General in topological order.

Done manually, this cascade is tedious and error-prone: it is easy to miss a compat
entry, register packages out of order, or lose track of which beta combination is
currently under test. This is the concrete problem CTDev is designed to solve.

Beta versions over branch-based development

The workflow described above deliberately favours pre-release registrations over the
alternative of using Pkg.develop() on local branches. This is the right choice,
for several reasons worth stating explicitly.

A registered beta — even one only in ct-registry — is an immutable, tagged
version. Julia's package resolver treats it like any other version: it checks compat
constraints, resolves transitive dependencies, and produces a reproducible Manifest.toml.
Every team member and every CI runner sees exactly the same thing by adding
ct-registry.

A Pkg.develop() path is none of those things. It points to a mutable local
directory. It bypasses the package resolver's compat checks, so a compat violation
in a downstream package goes undetected until release time. It is invisible to CI
unless you reproduce the exact local filesystem layout on the runner. And it is
fragile in ways that are hard to debug: uncommitted changes, wrong branch, stale
precompilation cache. The only legitimate use of Pkg.develop() is rapid
within-session experimentation before you know what you want to commit — not as a
cross-package integration strategy.

Concretely: when working on a change that touches CTBase and CTDirect together,
the right workflow is to commit the CTBase change, tag and register a beta to
ct-registry, then update CTDirect's compat entry and iterate. This is slightly
more ceremony than editing two local directories simultaneously, but it produces a
traceable, reproducible record of which combination was tested, and it exercises the
compat machinery that users will actually encounter.

Motivation for CTDev

What the existing workflow does not yet provide is a systematic, automated answer to
the question: is the current beta combination on ct-registry working end-to-end?
The local registry handles version pinning and staged release; CTBenchmarks.jl
tracks solver performance; CTActions centralizes CI workflow templates. But there
is no single repository that:

• automates the cascade of compat bumps and beta registrations triggered by a
  hierarchical change, keeping packages in dependency order and opening PRs
  consistently;
• runs cross-package integration tests (spanning the full parser → model → solver →
  solution pipeline) automatically against the current beta combination;
• provides a one-command developer environment bootstrap for contributors working
  across multiple CTX packages.

CTDev fills that role. It is not a Julia package and is not registered on any
registry. It is a developer-facing repository whose subject is the ecosystem as a whole.

Scope

Cascade tooling. Given a release plan describing which packages change and by how
much (CTBase: minor, CTModels: patch, ...), a Julia script walks the dependency
graph in topological order, bumps versions, updates all downstream [compat] entries,
opens PRs against each affected repository, and registers the new betas to ct-registry
via LocalRegistry.register(). The dependency graph is encoded once in
dependency_order.toml and reused for every release, rather than reconstructed from
memory.

Cross-package integration tests. A Project.toml that resolves against
ct-registry (where betas live) and hosts end-to-end tests of the full pipeline:
@def → parse → model → direct transcription → solve → solution extraction → plot,
as well as the Hamiltonian flow and GPU paths. These tests span multiple package
boundaries and cannot live inside any individual package's test suite. Running them
against ct-registry betas is exactly the validation step that currently happens
informally before each General release.

Nightly CI. A GitHub Actions workflow that resolves packages from ct-registry,
runs the integration suite, and reports failures per package. This makes the "does
the current beta combination work?" question answerable automatically, overnight,
rather than on each developer's machine before a release. Optionally triggered on
any push to ct-registry itself.

General promotion. Once the nightly suite is green on a beta combination, a
script triggers JuliaRegistrator for each package in topological order, respecting
the General registry's merge-delay convention.

Developer bootstrap. A script that clones all CTX repositories, adds
ct-registry to the local Julia environment, and calls Pkg.develop() on all
packages, for contributors who need to work on multiple packages in the same session.
Even then, the bootstrap starts from ct-registry versions so that the local
dev environment is consistent with what CI sees.

Repository structure

CTDev/
├── integration/
│   ├── Project.toml            # resolves from ct-registry
│   ├── Manifest.toml           # pinned; updated after each beta registration
│   ├── test_direct.jl          # @def → CTParser → CTModels → CTDirect → Ipopt
│   ├── test_indirect.jl        # CTFlows → CTIndirect → BVP solve  (once CTIndirect exists)
│   ├── test_gpu.jl             # ExaModels → MadNLP/CUDA path
│   └── test_plot.jl            # solution → RecipesBase rendering
├── release/
│   ├── dependency_order.toml   # canonical topological order and cross-edges of CTX packages
│   ├── cascade.jl              # bump versions, update all compat entries, open PRs
│   ├── register_beta.jl        # register one or more packages to ct-registry
│   ├── register_general.jl     # promote the validated beta combination to General
│   └── checklist.md            # human steps (tag, announce, Discourse post)
├── dev/
│   ├── bootstrap.sh            # clone all repos, add ct-registry, Pkg.develop() all
│   └── dev_env.jl              # Julia-side of the bootstrap
├── .github/
│   └── workflows/
│       ├── nightly.yml         # nightly integration suite against ct-registry
│       └── compat_check.yml    # warn when compat entries drift from dependency_order
└── README.md

Key scripts in detail

cascade.jl — the central piece. Takes a release plan as input:

# release_plan.toml
[bumps]
CTBase   = "minor"
CTModels = "patch"

Walks dependency_order.toml, computes the new version numbers, updates every
affected [compat] entry in every downstream Project.toml, commits the changes,
and calls register_beta.jl for each package in order. The result is a fully
consistent beta combination on ct-registry with a single command, rather than
six manual steps prone to ordering errors.

nightly.yml — resolves all CTX packages from ct-registry, picking up the
latest registered betas, runs integration/, and reports failures per package. Also
updates integration/Manifest.toml as a timestamped record of exactly which versions
were tested.

register_general.jl — once the nightly suite is green, promotes each package
to General via JuliaRegistrator in topological order, with configurable delay
between registrations to respect the General registry's merge conventions.

What this is not

CTDev does not replace the per-package CI. Each CTX package continues to have
its own CI.yml running its own unit and regression tests. CTDev tests only
cross-package behaviour that cannot live inside any single package.

It is also not a monorepo. Each CTX package remains an independent repository with
its own history, issues, and release cycle. CTDev observes and coordinates them;
it does not own them.

---

Relationship between the two

The two repositories are orthogonal and can be pursued independently. If CTIndirect.jl
is developed, CTDev immediately gains a new node in dependency_order.toml and a
new integration test (test_indirect.jl). The cascade tooling then handles it
automatically on the next release. If CTDev is established first, it provides the
infrastructure that makes launching CTIndirect.jl smoother from day one: the beta
registration workflow, the cascade script, the nightly signal, and the bootstrap all
apply to any new CTX package.

A natural sequencing:

1. Bootstrap CTDev with the existing packages (direct, GPU, plot paths).
2. Develop CTIndirect.jl, registering development betas to ct-registry and
   relying on the CTDev nightly suite as the integration signal.
3. Add test_indirect.jl to CTDev and a new entry to dependency_order.toml
   as CTIndirect stabilizes toward a first release.

Both repositories follow the conventions of the ecosystem (CTAppTemplate, Blue
style, Documenter.jl, Aqua.jl, codecov) and would live under the
control-toolbox GitHub organization.

---

Conclusion

The two proposals address distinct gaps. CTIndirect.jl closes the main algorithmic
asymmetry in the ecosystem: direct methods have a home (CTDirect), Hamiltonian flows
have a home (CTFlows), but shooting, multiple shooting, and homotopy — the workhorses
of hard optimal control problems — currently do not. CTDev closes the main operational
gap: the release cascade across eight interdependent packages is today a manual,
error-prone sequence, and the existing ct-registry / beta workflow, though sound in
principle, lacks the tooling to execute it reliably. Together, the two repositories
make the ecosystem stronger algorithmically and more robust to maintain, without
altering its multi-repo structure or its proven staged-release discipline.