Intent-aware mutation descriptions should encourage optional metadata and clearly require strict metadata

[alondahari]

Summary

Generated descriptions for all intent-aware issue mutations should heavily encourage agents to include rationale and confidence even when issue-intent is omitted, while clearly stating that these fields are required when issue-intent: true.

The affected tools are:

• add_labels
• set_issue_type
• set_issue_field
• close_issue
• assign_to_user
• assign_to_agent

There is also a concrete strict-mode failure in add_labels: its generated schema correctly requires structured label objects with name, rationale, and confidence, but its description still says plain strings are allowed and gives a plain-string example. Agents can follow that contradictory description, causing the safe-output processor to reject the complete label mutation.

Version

• gh-aw: v0.82.13
• Engine: default Copilot engine

Configuration modes

safe-outputs:
  add-labels:
    max: 5
    # issue-intent omitted

Omitting issue-intent keeps intent metadata optional and forwards it when supplied. This mode should strongly encourage metadata without requiring it.

safe-outputs:
  add-labels:
    max: 5
    issue-intent: true

Setting issue-intent: true enables strict mode. Generated descriptions must state that rationale and confidence are required, not optional.

Current generated contracts

Strict mode rewrites labels.items to an object-only schema:

{
  "type": "object",
  "required": ["name", "rationale", "confidence"],
  "properties": {
    "name": { "type": "string" },
    "rationale": { "type": "string", "maxLength": 280 },
    "confidence": {
      "type": "string",
      "enum": ["LOW", "MEDIUM", "HIGH"]
    },
    "suggest": { "type": "boolean" }
  }
}

But the generated labels description remains:

Labels to add (e.g., ['bug', 'priority-high']). Each entry can be either a label name string or an object with name plus optional rationale/confidence/suggest intent metadata.

That text conflicts with the strict schema in two ways:

1. It explicitly permits plain strings.
2. It describes rationale and confidence as optional even though strict mode requires them.

The top-level tool suffix only says:

INTENT: Include rationale ... and confidence ... with each call.

For add_labels, those fields belong to each label object rather than the top-level call, so this guidance does not fully disambiguate the expected payload.

The other intent-aware mutation schemas expose top-level rationale, confidence, and suggest. When strict mode is enabled, rationale and confidence become required in the schema, but their field descriptions still call them optional. When issue-intent is omitted, the fields remain available and are forwarded, but no generated suffix actively encourages the agent to use them.

Observed output

Agents may emit:

{
  "type": "add_labels",
  "item_number": 123,
  "labels": ["bug", "question"]
}

The safe-output processor then fails with:

Plain string label names are not permitted when issue_intent is explicitly enabled.
Provide label objects with a "name" field and intent metadata (rationale, confidence).

Because the complete add_labels operation is rejected, none of its labels are applied.

Expected behavior by mode

issue-intent omitted: strongly encouraged, not required

For every intent-aware mutation, generated tool descriptions should strongly encourage including rationale and confidence whenever the agent can make a reasoned mutation.

The guidance should:

• explain that rationale improves transparency and should normally be included;
• explain that confidence communicates certainty and should normally be included;
• identify the allowed confidence values (LOW, MEDIUM, HIGH);
• show a complete example containing intent metadata;
• clearly state that metadata remains optional in this mode so omission does not fail validation;
• explain that suggest: true routes the mutation for human review when supported.

For optional add_labels, plain strings must remain accepted for compatibility, but the description and primary example should prefer structured label objects:

{
  "type": "add_labels",
  "item_number": 123,
  "labels": [
    {
      "name": "bug",
      "rationale": "The report describes reproducible incorrect behavior.",
      "confidence": "HIGH"
    }
  ]
}

issue-intent: true: required

When issue-intent: true is enabled:

• Every intent-aware mutation description should explicitly state that rationale and confidence are required.
• Field descriptions must not call required intent fields optional.
• Examples must include all required intent metadata.
• The generated description and examples should only show structured label objects.
• The description should state that name, rationale, and confidence are required for every label.
• add_labels should guide the agent to emit:

{
  "type": "add_labels",
  "item_number": 123,
  "labels": [
    {
      "name": "bug",
      "rationale": "The report describes reproducible incorrect behavior.",
      "confidence": "HIGH"
    }
  ]
}

issue-intent: false: disabled

Intent fields and intent guidance should remain omitted.

Suggested implementation

In generate_safe_outputs_tools.cjs, generate mode-specific descriptions for every intent-aware mutation:

1. Omitted config: append strong-but-optional guidance and complete examples.
2. Explicit true: append required guidance, update field descriptions to say required, and use complete examples.
3. Explicit false: strip intent fields and do not add intent guidance.

For strict label intent mode, when replacing labels.items with the object-only schema, also replace the labels field description with strict-mode-specific guidance and a structured example.

Add regression tests asserting that the generated strict-mode descriptions:

• does not say plain strings are accepted;
• does not call rationale or confidence optional;
• includes an object-array example with all required fields.

Add tests for every intent-aware mutation asserting that omitted mode strongly encourages rationale and confidence without adding them to the schema's required list.

If the agent output path can bypass MCP input-schema validation, consider adding earlier validation or retry feedback so a schema-invalid final output can be corrected before the safe-output job runs.

Acceptance criteria

• All intent-aware mutations strongly encourage rationale and confidence when issue-intent is omitted.
• Omitted mode keeps rationale and confidence optional and continues forwarding them when supplied.
• All intent-aware mutations explicitly identify rationale and confidence as required when issue-intent: true.
• Required-mode field descriptions do not call required fields optional.
• add-labels.issue-intent: true generates an object-only schema and matching object-only description.
• Strict label descriptions identify name, rationale, and confidence as required per label.
• Optional label descriptions prefer structured labels while continuing to permit plain strings.
• issue-intent: false omits intent fields and guidance.
• Tests cover omitted, required, and disabled descriptions and schemas for every supported mutation.

Rationale

Intent metadata is most useful when agents include it routinely, not only when workflows force it. Strong optional-mode guidance should improve transparency while preserving compatibility, and precise required-mode guidance prevents schema-validity failures. Mode-specific descriptions make the intended behavior clear without requiring workflow authors to duplicate tool-shape instructions in every prompt.

Confidence

HIGH for the strict label failure — reproduced across multiple independent workflow runs, with emitted plain-string payloads and downstream validation errors matching exactly.

HIGH for the description gap — current generation only adds intent guidance when issue-intent: true; omitted mode exposes optional fields but does not actively encourage their use.