[Schema Consistency] Schema Consistency Check - 2026-03-28

[github-actions[bot]]

Summary

• Inconsistencies Found: 3 new (+ persistent backlog)
• Categories Analyzed: Schema ↔ Parser/Compiler, Schema ↔ Documentation, Schema ↔ Workflows
• Strategy Used: GitHub App Permissions Schema Completeness Audit (NEW — day 87, 30% window)
• Approach: Systematically compared all permission scopes across main_workflow_schema.json (permissions object with additionalProperties:false), pkg/workflow/permissions.go (convertStringToPermissionScope + GetAllGitHubAppOnlyScopes), and docs/src/content/docs/reference/permissions.md

---

Critical Issues

🔴 [NEW] 29 GitHub App-Only Permission Scopes Block Schema Validation

Files: pkg/parser/schemas/main_workflow_schema.json · pkg/workflow/permissions.go · docs/src/content/docs/reference/permissions.md · pkg/parser/schema_errors.go:217

The schema's permissions object uses additionalProperties: false and defines only 19 properties (16 standard GITHUB_TOKEN scopes + organization-projects, vulnerability-alerts, all). However, pkg/workflow/permissions.go handles 29 additional GitHub App-only scopes that are also documented in permissions.md as fully supported.

Consequence: Any workflow using these scopes fails JSON schema validation (via validateWithSchema in pkg/parser/schema_compiler.go) before reaching the semantic validateGitHubAppOnlyPermissions check. The App-only permission validation logic is effectively unreachable for 29 of 31 declared App-only scopes in the full compilation pipeline.

29 scopes in code + docs but missing from schema

Repository-level (9):
administration, environments, git-signing, workflows, repository-hooks, single-file, codespaces, repository-custom-properties, (plus vulnerability-alerts which IS in schema)

Organization-level (17):
members, organization-administration, team-discussions, organization-hooks, organization-members, organization-packages, organization-self-hosted-runners, organization-custom-org-roles, organization-custom-properties, organization-custom-repository-roles, organization-announcement-banners, organization-events, organization-plan, organization-user-blocking, organization-personal-access-token-requests, organization-personal-access-tokens, organization-copilot, organization-codespaces, (plus organization-projects which IS in schema)

User-level (3):
email-addresses, codespaces-lifecycle-admin, codespaces-metadata

User impact: A user who writes permissions:\n administration: read (as documented in permissions.md) gets a confusing schema validation error listing only the 19 allowed GITHUB_TOKEN scopes. No guidance toward GitHub App-only scopes or the permissions.md docs page is provided.

Evidence of test isolation gap: pkg/workflow/github_app_permissions_validation_test.go tests administration: read by directly calling validateGitHubAppOnlyPermissions(workflowData) — bypassing schema validation entirely. The tests verify downstream behavior that is unreachable in practice for these 29 scopes.

Suggested fix: Add all 29 scopes to permissions.oneOf[1].properties in the schema (with appropriate descriptions indicating GitHub App requirement), or change additionalProperties: false to additionalProperties: { type: "string", enum: ["read", "write", "none"] } with a schema comment explaining App-only scopes. Also update knownFieldValidValues and knownFieldScopes in schema_errors.go.

---

Schema ↔ Documentation Mismatches

🟡 organization-projects Missing "GitHub App-Only" Note in Schema Description

File: pkg/parser/schemas/main_workflow_schema.json (permissions.oneOf[1].properties.organization-projects)

vulnerability-alerts correctly reads: "GitHub App-only permission: required to access Dependabot alerts via the GitHub MCP server. The GITHUB_TOKEN does not have this permission — a GitHub App must be configured."

organization-projects simply reads: "Permission level for organization projects (read/write/none). Controls access to manage organization-level GitHub Projects boards." — no mention of GitHub App requirement.

Both are in GetAllGitHubAppOnlyScopes() and listed under "Organization-level" GitHub App-only permissions in permissions.md. The schema description for organization-projects should be updated to match vulnerability-alerts's pattern.

🟡 Schema Error Message Omits GitHub App Permission Guidance

File: pkg/parser/schema_errors.go:217 (knownFieldValidValues) and :226 (knownFieldScopes) and :236 (knownFieldDocs)

When a user specifies an unknown permission scope (e.g., administration: read), the error hint says:

"Valid permission scopes: actions, all, attestations, checks, contents, deployments, discussions, id-token, issues, metadata, models, organization-projects, packages, pages, pull-requests, repository-projects, security-events, statuses, vulnerability-alerts"

The documentation URL links to GitHub's GITHUB_TOKEN reference, not to the repo's own permissions.md which lists all App-only scopes. Users hitting this error receive no guidance that GitHub App-only scopes exist or how to enable them.

---

Persistent Issues (Unresolved from Prior Runs)

View persistent backlog

Severity	Issue	First Found
🔴 CRITICAL	status-comment schema description says "Must be explicitly set to true" but compiler_safe_outputs.go:192-195 auto-enables for slash_command/label_command triggers. Two pending changesets (minor-decouple-status-comment, minor-enable-reaction-status-comment-by-default) will change this.	2026-03-19
🔴 CRITICAL	update-discussion labels is a DEAD FEATURE — schema + docs define labels:/allowed-labels: fields but update_discussion.cjs has zero label handling code.	2026-03-20
🔴 CRITICAL	mcp-gateway feature flag in constants.go / sandbox.md / schema description says "requires feature flag" but is never checked via isFeatureEnabled() — gateway auto-enables via HasMCPServers(). Schema has internal contradiction on this.	2026-03-25
🔴 HIGH	proxy-args top-level MCP server field handled in code (mcp_config_custom.go:661) but MISSING from $defs/stdio_mcp_tool.properties in schema (additionalProperties:false).	2026-03-19
🔴 HIGH	runtimesConfigToMap() silently drops dotnet, elixir, haskell, java, ruby runtimes when RuntimesTyped is used. All five are in schema, struct, and runtime_definitions.go.	2026-03-27
🟡 HIGH	set-issue-type and mark-pull-request-as-ready-for-review safe output types: fully implemented in schema + code but ABSENT from safe-outputs.md.	2026-03-24
🟡 HIGH	observability.job-summary field: implemented in schema + code + tests but absent from frontmatter.md and frontmatter-full.md.	2026-03-26
🟡 HIGH	imports object form (with aw: and apm-packages: subfields): in schema + code but imports.md only documents array form.	2026-03-26
🟡 HIGH	disable-xpia-prompt feature flag: actively used in unified_prompt_step.go:106 but completely absent from all docs/schema.	2026-03-25
🟡 HIGH	close-older-key field: implemented in schema + code + CJS runtime but absent from all docs.	2026-03-22
🟡 HIGH	Codemod ID sandbox-agent-false-removal listed in mcp_tools_management.go:171 and gh-aw-as-mcp-server.md:284 but doesn't exist in fix_codemods.go. Actual ID is sandbox-false-to-agent-false.	2026-03-21

---

Recommendations

1. Fix schema permissions gap — Add all 29 missing GitHub App-only scopes to permissions.oneOf[1].properties with descriptions noting they require a GitHub App. Update knownFieldValidValues + knownFieldScopes + knownFieldDocs in schema_errors.go to reflect this.
2. Add integration test — Add a test that compiles a full workflow file (not just calls validateGitHubAppOnlyPermissions in isolation) with an App-only scope like administration: read to catch future schema/code drift.
3. Fix organization-projects schema description — Mirror the GitHub App-only language from vulnerability-alerts.
4. Address proxy-args schema gap — High priority since it also blocks compilation for a documented feature.
5. Address runtimes serialization gap — dotnet/elixir/haskell/java/ruby are silently dropped.

---

Strategy Performance

• Strategy Used: GitHub App Permissions Schema Completeness Audit (NEW — 30% exploration window)
• Approach: Schema property enumeration × code switch statement × docs comparison
• Findings: 3 new findings, 1 critical
• Effectiveness: HIGH — new strategy uncovered a fundamental schema/code contract violation affecting all GitHub App-only permissions
• Should Reuse: YES — the property set comparison approach has proven effective across multiple strategies (proxy-args in strategy-30, runtimes in strategy-38, and now permissions in strategy-39)

---

References:

• §23677407220

AI generated by Schema Consistency Checker · history

• expires on Mar 29, 2026, 4:36 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Schema Consistency Checker.

A newer discussion is available at Discussion #23416.