feat: introduces "Sub-lobsters": nested lobster workflows with optional loop

[ggondim]

This pull request introduces support for sub-workflow steps in the workflow engine, allowing users to embed .lobster files as steps and optionally loop them with conditions, like a Ralph Loop. It adds new fields to the workflow step schema, updates validation logic, implements execution of sub-workflows (including looping and conditional exit), and provides comprehensive tests and documentation.

🚀 Sub-workflow step support

• Added lobster, args, and loop fields to the WorkflowStep type, enabling steps to reference and run sub-workflows, pass arguments, and repeat execution with optional conditions. (src/workflows/file.ts src/workflows/file.tsR19-R31)
• Implemented sub-workflow execution logic, including argument resolution, looping with max iterations and shell condition, and result propagation to parent workflow. (src/workflows/file.ts [1] [2]

🔰 Validation improvements

• Updated workflow file validation to ensure lobster steps require a non-empty file path and that step IDs are unique. (src/workflows/file.ts src/workflows/file.tsR111-L105)

📝 Documentation updates

• Added detailed documentation for new step fields and usage examples for sub-workflow steps and looping in README.md:

Step fields

Field	Required	Description
id	yes	Unique step identifier; used to reference the step's output in subsequent steps via $id.stdout or $id.json.
command	yes (for regular steps)	Shell command to run. Mutually exclusive with lobster.
lobster	yes (for sub-workflow steps)	Path to a .lobster file to run as a sub-workflow (resolved relative to the parent workflow). Mutually exclusive with command.
args	no	Key/value map of input arguments passed to the sub-workflow. Values support ${arg} and $stepId.stdout/$stepId.json template syntax. Only valid when lobster is set.
loop	no	Repeat the sub-workflow step in a loop. Only valid when lobster is set.
loop.maxIterations	yes (when loop is set)	Maximum number of iterations.
loop.condition	no	Shell command evaluated after each iteration. Exit code 0 continues the loop; non-zero stops it early. Receives LOBSTER_LOOP_STDOUT, LOBSTER_LOOP_JSON, and LOBSTER_LOOP_ITERATION as environment variables.
stdin	no	Pass a previous step's output as stdin.
approval	no	Set to required to insert an approval gate before the step runs.
condition	no	Expression that must be truthy for the step to run.

Sub-workflow step example

Use the lobster field to embed another .lobster file as a step in your workflow, optionally passing arguments and looping until a condition is met:

steps:
  - id: prepare
    command: echo "hello"
  - id: process
    lobster: ./sub_workflow.lobster
    args:
      input: $prepare.stdout
    loop:
      maxIterations: 10
      condition: '! echo "$LOBSTER_LOOP_STDOUT" | grep -q "^done"'

The sub-workflow's last step result (stdout/json) is stored as the step result and is accessible via $process.stdout / $process.json in subsequent steps.
134 changes: 129 additions & 5 deletions134
src/workflows/file.ts

✅ Testing

• Added comprehensive tests covering sub-workflow execution, argument passing, looping behavior, loop conditions, and validation errors. (test/lobster_sub_step.test.ts test/lobster_sub_step.test.tsR1-R186)

[vignesh07]

The primary upside of lobster is visibility and approval gating. Having an invisible sub loop running - especially compared to a Ralph loop concerns me.

I don't think this is the intended direction for lobster.

[ggondim]

@vignesh07 Should Lobster be so opinionated and make the experience with deterministic workflows more challenging by imposing single-level-only workflows?

The arguments to merge it:

• "Invisible sub loops running", including Ralph loops, are already possible: using exec commands or tools installed to OpenClaw.
• Approval gates for "sub-lobsters" are still possible: users can use the condition config with previous approval gates or even nested ones

I understand if this doesn’t align with the team’s vision. Please let me know, and I’ll create a custom plugin for myself. Thank you for your time.