Skip to content

DDIR: write the example programs as corgi lambdas - #935

Merged
frankmcsherry merged 9 commits into
master-nextfrom
ddir-lambda-examples
Oct 15, 2026
Merged

frankmcsherry merged 9 commits into
master-nextfrom
ddir-lambda-examples

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

It also carries a fix that missed #934: a DDIR -- comment inside a lambda now reaches corgi as a # comment, which corgi reads (before, corgi rejected it).

Every scalar slot in DDIR's .ddp programs is now a corgi lambda: the AoC programs, examples/programs, the tests/programs fixtures, and the server and LDBC programs.
No program here uses DDIR's term language ($n, ^n, tuple, list, if, fold, case, infix operators, constructors).
And and or of conditions are written with corgi's (c1, c2, ..) all and any.

Each program computes what it did before, checked against the original on the corgi backend:

  • all 33 AoC programs give identical inspect output, not only the checked answer;
  • the corgi gate's programs match their originals at 1 and 3 workers;
  • the server session scripts give identical output (times and thread ids aside);
  • the LDBC smoke run and the SNB suite verify their answers.

Three fixtures lose what tested only the term language: if_literal is gone (a literal-condition if whose branches differ in shape), scalar_ops drops tuple len, and sum_ops drops istag on a non-sum.

binders, case_ops and tour take sums apart in lambdas, which the vec backend cannot run (it types a lambda's input from the row alone).
The corgi gate checks them against recorded output instead: <prog>.expected beside the program, one row per line, from the original programs on vec.
The gate uses a recorded file where one exists and the vec backend otherwise.

A second commit makes each program's shapes plain.
The originals wrapped nearly every single-field key and value in a 1-tuple, because DDIR's projections always built tuples, and the first commit kept those shapes so each rewrite could be checked row for row.
Keys and values are now bare where only the program sees them, and lambdas bind a value whole when they do not take it apart.
Inputs keep their declared shapes, count still emits (n), and the server and LDBC programs keep the shapes of their exports and imports.
Each program computes what it did before up to 1-tuple wrappers, by the same checks, and the three .expected files are regenerated after confirming they differ only in wrappers.

Two more commits: the spread_values fixture is gone (it pinned how a bare $n spliced fields, which lambdas do not do), and count emits a bare Int rather than (n), in both backends and shape inference.
The programs that read counts take a bare n; the server and LDBC programs re-wrap a count where it reaches an export, adding one operator to count_gen.
These were checked against the previous commit's programs run on a server built before the change.

The empty_batch fixture is gone: it needed a projection that fell back to a row-wise path, which no longer exists, and it passed with its guard removed.

The last two commits bump corgi to frankmcsherry/WIP#100 and use it: constant lists are literals (day16's table, day18's powers of two, day03's offsets), computed lists are one n-ary append, and a lambda builds Some(x) as x inject Some.

Not converted here: inline programs in Rust tests (mostly tests/explain.rs), and the SNB program generator server/bench/ldbc/snb/rel.py.

cargo test -p interactive -p ddir-server passes.

🤖 Generated with Claude Code

frankmcsherry and others added 2 commits October 14, 2026 17:57
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every scalar slot in the `.ddp` programs (AoC, examples, test fixtures, server and LDBC programs) is a corgi lambda, so none uses DDIR's term language.
Each computes what it did before, checked against the originals on the corgi backend.
And and or of conditions are written with corgi's `all` and `any`.

The `if_literal` fixture is gone, and `scalar_ops` and `sum_ops` lose the columns that tested only term-language features (a literal-condition `if` of mixed shapes, tuple `len`, `istag` on a non-sum).
`binders`, `case_ops` and `tour` take sums apart in lambdas, which the vec backend cannot run, so the corgi gate checks them against recorded output (`<prog>.expected`, from the originals on vec).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frankmcsherry
frankmcsherry changed the base branch from ddir-corgi-scope to master-next October 14, 2026 21:57
frankmcsherry and others added 4 commits October 14, 2026 19:00
The term-syntax originals wrapped nearly every single-field key and value in a 1-tuple, as DDIR's projections always built tuples, and the lambda rewrite kept those shapes so it could be checked row for row.
Nothing in corgi or the engine needs the wrappers, so each program's own shapes are now bare values.
Lambdas bind a value whole when they do not take it apart.

Inputs keep their declared shapes, `count` still emits `(n)`, and the server and LDBC programs keep the shapes of their exports and imports.
The AoC and fixture programs' exports lose their 1-tuples too, so the three recorded `.expected` files and the `registered` test's reading of `cells` follow.

Each program computes what it did before, up to 1-tuple wrappers, checked against the previous commit on the corgi backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It pinned how a bare `$n` spliced a row's fields, which lambdas do not do; its programs now pass their values through unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`count` wrapped its count in a 1-tuple, `(n)`, to keep the convention that a value is a tuple so DDIR terms could read it as `$1[0]`.
Lambdas read a bare count directly, so both backends and shape inference now give an `Int`.

The programs that read counts take a bare `n`, and the server and LDBC programs re-wrap a count where it reaches one of their exports, so those keep their shapes.
Each program computes what it did before up to 1-tuple wrappers, checked against the previous commit's programs on a server built before this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It needed a projection that fell back to a row-wise path, which no longer exists; with the empty-batch guard in apply_ops removed it still passed, in its term and lambda forms alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frankmcsherry added a commit to frankmcsherry/WIP that referenced this pull request Oct 15, 2026
…lane, and a note for stray parentheses (#100)

* corgi: append takes any number of lists

`append` took a pair of lists. It now takes a tuple of one or more, as `zip`
already does: `(xs, ys, zs) append` is each row's xs, then ys, then zs. The pair
form is unchanged, and `(xs) append` is xs.

The point is lists of computed values. `enlist` moves no data (it puts bounds of
one element per row over a column), so `(a enlist, b enlist, c enlist) append` is
one pass that gathers n columns into a list of n per row. Written with the pair,
a left-nested chain of n appends copies each row's growing prefix at every step,
about n²/2 copies per row.

Lists of numbers keep the pair's direct loop for two lists. For more, a loop
copies each part's run per row; when every part is one width per row (as
enlisted columns are), it reads and writes no bounds per row and the result is
one width per row. Lists held at different storages meet at the wider, as the
pair did.

Measured on 1M rows (best of 15, release):
- (a, b) append, lists of 0-6 and 4 numbers: 7.3 ms before and after.
- the same of pairs: 24.5 ms before and after.
- three enlisted columns: ((c enlist, c enlist) append, c enlist) append
  takes 9.9 ms; (c enlist, c enlist, c enlist) append takes 2.8 ms.
- three lists: ((a, b) append, a) append takes 15.7 ms; (a, b, a) append
  takes 12.3 ms.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* corgi: list literals of constants, and ascriptions `: S` that check shapes

Two additions to the ML surface, asked for by DDIR, whose example programs are
now written in corgi lambdas (TimelyDataflow/differential-dataflow#935).

A list literal of constants, `[1, 2, 3]` or `[(0, "GET"), (1, "PUT")]`, is one
constant: the parser builds the list's value and emits one `Op::Lit`, which
fills it to the scope's length, exactly as a string is. `"ab"` and `[97, 98]`
build the same graph. Elements are numbers, strings, `()`, and tuples and lists
of these, all of one shape. Nothing else reaches that node today but a list of
bytes, so this is not a second spelling: the enlist/append chain DDIR wrote
builds each constant as a column and appends them row by row. A list of
computed values is not a literal; it is an append of enlisted columns, and a
bracket holding anything but constants is a parse error that says so.

An ascription `: S` says a value has the shape S: as a stage (`xs : List(int)`),
and on a pattern (`(acc : (int, int), x : int) -> ...`). `_` in a shape is a
hole, and an enum names its sum, a variant declared without a payload being a
hole. It is a check, made when the program is typed; it converts and steers
nothing. The one place it gives a shape rather than checking one is the empty
list, which has none of its own: `[] : List(int)`. Inside a literal, an empty
list may instead take its shape from the elements beside it, `[[1], []]`.

The core gains no op. `Pattern`, which checked the nesting of products a tuple
pattern takes apart, becomes a shape with holes (`Int`, `Float`, `List`, `Sum`
and `Ref` join `Tuple` and `_`), so `Op::Pattern` checks a whole shape. A
pattern's own nesting and its ascriptions are one check, as a tuple pattern was.
The check reads the value's structure, not its rows. A failed one reads, e.g.,
"in the pattern List(float), the pattern float wants a Float, not Int"; graph
nodes carry no source positions, so it has no line and column, as op errors
have none.

DDIR's cases, before and after:
- day16: 24 lines of `let l = (l, (0, 45, 0, 1, 0) enlist) append in` become
  one literal, `[(0, 46, 0, 1, 0), (0, 45, 0, 1, 0), ...]`.
- day18: `(1099511627776, (0, 40) range) scan ((a, _) -> (a, 2) div)` becomes
  the 40 powers of two written out, `[549755813888, ..., 2, 1]`.
- day03: `(-1, 2) range map (dr -> (dr, (-1, 2) range) cap_list) flatten` becomes
  `[(-1, -1), (-1, 0), (-1, 1), (0, -1), (0, 0), (0, 1), (1, -1), (1, 0), (1, 1)]`.
- unnest: `((a enlist, b enlist) append, (a, b) add enlist) append` becomes
  `(a enlist, b enlist, (a, b) add enlist) append`, with the previous commit.

Every .col program parses to the same graph as before, apart from two whose text
changed here: their comments described the fold's accumulator shape in prose
(naming U64, now retired), and their lambdas now say it with ascriptions. A new
corpus program shows the literals.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* corgi: inject fills a lane the enum leaves open with its payload's shape

`inject` built a sum only from an enum that declared every lane's payload shape.
The lanes it does not fill still need shapes: they are built as empty columns.
The lane it fills now need not have one: left open, it takes the payload's
shape, as `branch`'s lanes take theirs from the input. Declared, it is checked
as before.

So one declaration serves every payload, which is what DDIR's Option and Result
want:

    enum Option = None () | Some in (3 inject Some, "dog" inject Some)

builds a {() | Int} and a {() | List<Int>}, and both match by the names
`None`/`Some`. Before, DDIR wrote an unconditional Some(x) as
`(x, 1) branch Option map_variant None (_ -> ())`.

What it does not do: `() inject None` still needs Some's shape, since that
lane is built empty and nothing in the program gives its shape. A Result
generic in both lanes is the same case.

`Op::Inject` carries `Vec<Option<Shape>>`: `None` only for the lane the payload
fills. A host that builds the op directly (DDIR's term-language lowering does)
wraps its shapes in `Some`. `unwrap(inject x) = x` now cancels only when every
lane is declared alike, or there is one lane.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* corgi: a shape error on a one-field tuple of what was wanted says parentheses built it

Parentheses build a tuple wherever they appear, so `(e)` written for `e` is a
one-field tuple, and the error it causes names the right shapes but is easy to
misread. DDIR hit it in every batch of its rewrite, most recently as
`(([] : List(int)), xs) append`:

    Append operand: expected a list, got (List<Int>)

When the operand is a one-field tuple holding exactly what was wanted, the
error now ends with

    (a one-field tuple: parentheses build a tuple, so `(e)` is not `e`)

This covers the value accessors every op reads its operand through (a pair, a
list, a sum, a leaf, an Int, a mask, a Float), merges whose operands must share a
shape (`min`, `select`, the comparisons), ascriptions and tuple patterns, an
inject's declared lane, and a host word's input. A one-field tuple of something
else, as in `(x) fold_add` on an Int, is no such slip and gets no note.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
frankmcsherry and others added 3 commits October 14, 2026 20:13
…pen lane

Corgi (frankmcsherry/WIP#100) builds `Op::Inject` from optional lane shapes, so DDIR's term lowering passes its declared lanes as `Some`.
`Option` and `Result` leave their payload lanes open, so a lambda builds `Some(x)` as `x inject Some`.
A pattern that does not fit its input now says it "wants" a product of so many fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
day16's 24-row table, day18's powers of two and day03's neighbour offsets are list literals.
ast, unnest and tour build their computed lists with one n-ary `append` of enlisted values, in place of nested pairs.
Each computes what it did before, checked against the previous commit on the corgi backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its list is one append, nothing falls back to rows, and corgi orders integers by signed value, so its inputs need no contract (it agrees across backends on negative inputs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frankmcsherry
frankmcsherry merged commit 08c458b into master-next Oct 15, 2026
6 checks passed
@frankmcsherry
frankmcsherry deleted the ddir-lambda-examples branch October 15, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant