Repository Quality Report: Validation System Compliance & Test Coverage (2026-03-27)

[github-actions[bot]]

🎯 Repository Quality Improvement Report — Validation System Compliance

Analysis Date: 2026-03-27
Focus Area: Validation System Compliance & Test Coverage
Strategy Type: Custom (first run, no history)
Custom Area: Yes — The pkg/workflow/ and pkg/cli/ directories contain 54 non-WASM validation files that are core to the gh aw compile pipeline. AGENTS.md explicitly documents a 300-line hard limit for validators. This focus area directly measures compliance with that architectural constraint and identifies critical testing gaps in the validation layer.

---

Executive Summary

The workflow compilation pipeline relies on a large system of 54 validators spread across pkg/workflow/ and pkg/cli/. Analysis reveals 9 validators (17%) exceed the documented 300-line hard limit, and 30 validators (56%) lack dedicated test files, including several of the largest and most critical ones.

The two safe_outputs validators are the most severe outliers: both at 407 lines with no test coverage. The permissions_validation.go (351 lines) and dispatch_workflow_validation.go (363 lines) also exceed the limit without tests. These files handle security-critical decisions (safe output targeting, permission scopes, tool authorization) and represent the highest risk for undetected regressions.

Full Analysis Report

Focus Area: Validation System Compliance & Test Coverage

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total non-WASM validators	54	✅
Validators exceeding 300-line hard limit	9 (17%)	❌
Validators exceeding 200-line target	22 (41%)	⚠️
Validators without test files	30 (56%)	❌
Validators >200 lines AND without tests	10	❌
Average AGENTS.md target size	100–200 lines	—
AGENTS.md hard limit	300 lines	—

Files Exceeding 300-Line Hard Limit:

File	Lines	Has Tests
pkg/workflow/safe_outputs_validation_config.go	407	❌ No
pkg/workflow/safe_outputs_validation.go	407	❌ No
pkg/workflow/dispatch_workflow_validation.go	363	✅ Yes
pkg/workflow/tools_validation.go	359	✅ Yes
pkg/workflow/permissions_validation.go	351	❌ No
pkg/cli/run_workflow_validation.go	344	✅ Yes
pkg/workflow/repository_features_validation.go	342	✅ Yes
pkg/workflow/template_injection_validation.go	306	✅ Yes
pkg/workflow/mcp_config_validation.go	301	✅ Yes

Large Validators Without Any Test Coverage (>200 lines):

File	Lines
pkg/workflow/safe_outputs_validation_config.go	407
pkg/workflow/safe_outputs_validation.go	407
pkg/workflow/permissions_validation.go	351
pkg/workflow/expression_safety_validation.go	295
pkg/workflow/runtime_validation.go	290
pkg/workflow/glob_validation.go	255
pkg/workflow/agent_validation.go	255
pkg/cli/compile_validation.go	237
pkg/workflow/expression_syntax_validation.go	236
pkg/workflow/pip_validation.go	205

Findings

Strengths

• Many newer validators have adopted the small-file pattern correctly (e.g., strict_mode_*.go validators are 72–190 lines each)
• Validators that exceed limits but have tests (e.g., tools_validation.go, mcp_config_validation.go) are lower risk
• Clear naming convention ({domain}_validation.go) is consistently applied
• WASM variants are correctly split into their own small files

Areas for Improvement

• [CRITICAL] safe_outputs_validation.go and safe_outputs_validation_config.go — both 407 lines, no tests. Safe Outputs is a security-critical feature and the absence of unit tests is a high-risk gap.
• [HIGH] permissions_validation.go — 351 lines, no tests. Permission validation affects all workflows using GitHub toolsets.
• [HIGH] expression_safety_validation.go — 295 lines, no tests. Template injection prevention depends on this.
• [MEDIUM] dispatch_workflow_validation.go — 363 lines, has tests but exceeds the 300-line limit; contains multiple distinct validation domains that could be split.
• [MEDIUM] agent_validation.go — 255 lines, no tests. Validates AI engine + tool combinations — important for correctness.

Detailed Analysis

The safe_outputs_validation_config.go is unique: it contains the ValidationConfig data structure (a table of field validation rules), plus the GetValidationConfigJSON function that serializes this config for use in shell scripts. This is a data + logic mix that could reasonably be split. The companion safe_outputs_validation.go handles domain pattern matching for network allowed-domains and safe-output targets.

permissions_validation.go defines ValidatePermissions(), FormatValidationMessage(), and ValidateIncludedPermissions() — three distinct concerns that each warrant their own test files. The absence of tests here means permission misconfiguration could silently pass through compilation.

The pattern of strict-mode validation being split correctly (5 files, all under 200 lines) shows the team knows how to apply the split architecture — it just hasn't been applied retroactively to older validators.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot coding agent execution. Each section represents an independent work item that can be assigned separately.

---

Task 1: Add Test Coverage for safe_outputs_validation.go

Priority: High
Estimated Effort: Medium
Focus Area: Test Coverage — Security-Critical Validation

Description:
pkg/workflow/safe_outputs_validation.go (407 lines) is one of the two largest validators and has zero test coverage. It validates network allowed-domains and safe-output target configurations. These validations are critical to the safe-outputs security model.

Acceptance Criteria:

• Create pkg/workflow/safe_outputs_validation_test.go with //go:build !integration build tag
• Test validateNetworkAllowedDomains() with valid domains, wildcard patterns, and invalid inputs
• Test validateSafeOutputsAllowedDomains() with ecosystem identifiers and invalid patterns
• Test validateSafeOutputsTarget() with all valid target types (empty, "triggering", "*", integer strings, GitHub expressions)
• Test validateDomainPattern() with multiple wildcard and malformed inputs
• Run make test-unit (or selective test) to confirm all pass
• Run make lint to confirm no linting issues

Code Region: pkg/workflow/safe_outputs_validation.go

Add a comprehensive test file for `pkg/workflow/safe_outputs_validation.go`.

The file contains these functions to test:
- `validateNetworkAllowedDomains(network *NetworkPermissions) error` — validates domains in network config
- `validateSafeOutputsAllowedDomains(config *SafeOutputsConfig) error` — validates safe-outputs allowed domains
- `validateDomainPattern(domain string) error` — validates a single domain pattern (wildcards, plain, etc.)
- `validateSafeOutputsTarget(config *SafeOutputsConfig) error` — validates "target" field values
- `validateTargetValue(configName, target string) error` — validates one target string
- `isGitHubExpression(s string) bool` — checks if string is a GitHub Actions expression
- `isEcosystemIdentifier(domain string) bool` — checks if a string is an ecosystem identifier like "node", "defaults"

Create `pkg/workflow/safe_outputs_validation_test.go` with:
1. Build tag `//go:build !integration` at the very top
2. Table-driven tests for each exported/unexported function
3. Cover valid inputs, invalid inputs, edge cases, and boundary conditions
4. Use `assert.*` for validations, `require.*` for setup
5. Include descriptive assertion messages

Reference AGENTS.md for test patterns: use `assert.NoError`, `assert.Error`, `assert.Equal` with messages.
Run `go test -v -run "Test.*SafeOutputs\|Test.*Domain\|Test.*Target\|Test.*Ecosystem" ./pkg/workflow/` to validate.

---

Task 2: Add Test Coverage for permissions_validation.go

Priority: High
Estimated Effort: Medium
Focus Area: Test Coverage — Permission Validation

Description:
pkg/workflow/permissions_validation.go (351 lines, no tests) validates GitHub Actions permissions for all workflows using GitHub toolsets. The exported ValidatePermissions() function is used throughout the compilation pipeline to enforce correct permission scopes.

Acceptance Criteria:

• Create pkg/workflow/permissions_validation_test.go with //go:build !integration build tag
• Test ValidatePermissions() with all supported toolset combinations
• Test FormatValidationMessage() with missing and extra permissions
• Test that ValidateIncludedPermissions() handles valid and invalid YAML permission blocks
• Cover the checkMissingPermissions() logic via ValidatePermissions() calls
• Run go test -v -run "TestValidatePermissions\|TestFormatValidation" ./pkg/workflow/ to confirm
• Run make lint to confirm no linting issues

Code Region: pkg/workflow/permissions_validation.go

Add a comprehensive test file for `pkg/workflow/permissions_validation.go`.

This file contains:
- `ValidatePermissions(permissions *Permissions, githubTool ValidatableTool) *PermissionsValidationResult`
- `checkMissingPermissions(permissions *Permissions, required map[PermissionScope]PermissionLevel, toolsets []string, result *PermissionsValidationResult)`
- `FormatValidationMessage(result *PermissionsValidationResult, strict bool) string`
- `formatMissingPermissionsMessage(result *PermissionsValidationResult) string`
- `ValidateIncludedPermissions(topPermissionsYAML string, importedPermissionsJSON string) error`

Create `pkg/workflow/permissions_validation_test.go` with:
1. Build tag `//go:build !integration` at the very top
2. Table-driven tests for `ValidatePermissions` covering: nil permissions, missing required scopes, sufficient permissions, extra permissions, various toolset combinations
3. Tests for `FormatValidationMessage` with both strict=true and strict=false modes
4. Tests for `ValidateIncludedPermissions` with valid YAML, missing permissions in imports, and malformed YAML
5. Use `assert.*` for validations, `require.*` for setup
6. Include descriptive assertion messages

Run `go test -v -run "TestValidatePermissions\|TestFormatValidation\|TestValidateIncluded" ./pkg/workflow/` to validate.

---

Task 3: Add Test Coverage for expression_safety_validation.go

Priority: High
Estimated Effort: Medium
Focus Area: Test Coverage — Template Injection Prevention

Description:
pkg/workflow/expression_safety_validation.go (295 lines, no tests) validates that workflow expressions don't introduce template injection vulnerabilities. This is a security-critical validator with no test coverage.

Acceptance Criteria:

• Create pkg/workflow/expression_safety_validation_test.go with //go:build !integration build tag
• Test safe expression patterns (constants, safe GitHub context vars)
• Test dangerous expression patterns (untrusted inputs like github.event.issue.title)
• Test edge cases: nested expressions, multi-line expressions, empty inputs
• Run go test -v -run "Test.*Expression.*Safety\|Test.*ExpressionSafety" ./pkg/workflow/ to confirm
• Run make lint to confirm no issues

Code Region: pkg/workflow/expression_safety_validation.go

Add a comprehensive test file for `pkg/workflow/expression_safety_validation.go`.

This file validates that workflow prompt expressions don't expose untrusted GitHub context values that could lead to prompt injection. Study the file to understand what patterns are considered safe vs. dangerous.

Create `pkg/workflow/expression_safety_validation_test.go` with:
1. Build tag `//go:build !integration` at the very top
2. Table-driven tests covering: safe expressions (constants, trusted context), dangerous expressions (untrusted event data like `github.event.issue.title`, `github.event.pull_request.body`), compound expressions mixing safe and unsafe parts
3. Edge cases: empty strings, GitHub Actions built-in functions (fromJSON, toJSON), nested expressions
4. Use `assert.*` for validations, `require.*` for setup  
5. Include descriptive assertion messages

Run `go test -v -run "Test" ./pkg/workflow/` focused on the expression safety functions to validate.

---

Task 4: Refactor safe_outputs_validation.go to Meet the 300-Line Limit

Priority: Medium
Estimated Effort: Medium
Focus Area: Architecture Compliance — 300-Line Hard Limit

Description:
pkg/workflow/safe_outputs_validation.go at 407 lines contains two distinct validation domains: (1) network domain validation (validateNetworkAllowedDomains, domain pattern helpers) and (2) safe-outputs target/domain validation (validateSafeOutputsAllowedDomains, validateSafeOutputsTarget). Per AGENTS.md, files over 300 lines should be split into separate files by domain.

Acceptance Criteria:

• Split into pkg/workflow/network_domains_validation.go (network.allowed domain logic) and retain safe-outputs-specific logic in safe_outputs_validation.go
• Or split by validateNetworkAllowedDomains + helpers → network_domains_validation.go, and move safe-outputs domain validation to existing file
• Both resulting files should be under 300 lines
• Shared helpers (isEcosystemIdentifier, validateDomainPattern) should be in the file where they're primarily used, or in validation_helpers.go if used by both
• Run make build and make test-unit to confirm no regressions
• Run make lint and make fmt before committing

Code Region: pkg/workflow/safe_outputs_validation.go

Refactor `pkg/workflow/safe_outputs_validation.go` (407 lines) to comply with the 300-line hard limit documented in AGENTS.md.

The file contains two logical groups:
1. **Network domain validation**: `validateNetworkAllowedDomains()`, `isEcosystemIdentifier()`, `isEcosystemIdentifierPattern` regex — validates `network.allowed` domains
2. **Safe outputs validation**: `validateSafeOutputsAllowedDomains()`, `validateDomainPattern()`, `validateSafeOutputsTarget()`, `validateTargetValue()`, `isGitHubExpression()` — validates `safe-outputs` configuration

Split the file:
- Move the network domain validation group to a new file: `pkg/workflow/network_domains_validation.go`
- Keep the safe-outputs validation group in `pkg/workflow/safe_outputs_validation.go`
- Shared helpers like `validateDomainPattern` and `isEcosystemIdentifier` may be needed by both — place them in whichever file uses them more, OR move to `pkg/workflow/validation_helpers.go` if used by both

After splitting:
1. Run `make build` to verify compilation
2. Run `go test -v ./pkg/workflow/` to verify no regressions
3. Run `make fmt` and `make lint`
4. Confirm both new files are under 300 lines

Follow the naming convention: `{domain}_validation.go`

---

Task 5: Add Test Coverage for agent_validation.go

Priority: Medium
Estimated Effort: Small
Focus Area: Test Coverage — Engine/Tool Compatibility Validation

Description:
pkg/workflow/agent_validation.go (255 lines, no tests) validates AI engine and tool combinations — e.g., that Copilot engine doesn't use tools unsupported for that engine, and that workflow_run triggers have appropriate branch restrictions. This is important for preventing silent misconfigurations.

Acceptance Criteria:

• Create pkg/workflow/agent_validation_test.go with //go:build !integration build tag
• Test engine + tool compatibility (valid combos, invalid combos)
• Test workflow_run trigger validation with and without branch filters
• Run go test -v -run "Test.*Agent\|Test.*Engine" ./pkg/workflow/ to confirm
• Run make lint to confirm no issues

Code Region: pkg/workflow/agent_validation.go

Add a test file for `pkg/workflow/agent_validation.go`.

This file validates AI engine and tool compatibility. Study the file to understand:
1. Which tools are checked for engine compatibility
2. What warnings/errors are emitted for incompatible combinations
3. How `workflow_run` trigger restrictions are validated

Create `pkg/workflow/agent_validation_test.go` with:
1. Build tag `//go:build !integration` at the very top
2. Table-driven tests covering: supported tool + engine combos, unsupported tool + engine combos (expect warning or error), workflow_run with branch filters, workflow_run without branch filters
3. Use `assert.*` for validations, `require.*` for setup
4. Include descriptive assertion messages

Run `go test -v -run "Test" ./pkg/workflow/` targeting agent validation functions to validate.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-03-27	Validation System Compliance & Test Coverage	First Run	Yes	9 files over limit, 30 without tests — 5 tasks generated

---

🎯 Recommendations

Immediate Actions (This Week)

1. Add tests for safe_outputs_validation.go — Priority: High (407 lines, security-critical, zero coverage)
2. Add tests for permissions_validation.go — Priority: High (351 lines, permission decisions, zero coverage)
3. Add tests for expression_safety_validation.go — Priority: High (295 lines, injection prevention, zero coverage)

Short-term Actions (This Month)

1. Refactor safe_outputs_validation.go to split network domain validation into its own file — Priority: Medium
2. Add tests for agent_validation.go — Priority: Medium (255 lines, engine/tool combos)

Long-term Actions (This Quarter)

1. Establish a CI gate that fails if any new validation file is created over 300 lines
2. Work through the remaining 20 test-less validators systematically
3. Consider a shared test helper for creating FrontmatterConfig fixtures for validation tests

---

📈 Success Metrics

Track these metrics to measure improvement in Validation System Compliance:

• Validators over 300 lines: 9 → 0 (target: eliminate all hard-limit violations)
• Validators without test files: 30 → 20 (initial target: cover all >200-line validators)
• Test-to-source ratio (validation only): Current: ~44% → Target: 70%+

---

Next Steps

1. Review and prioritize the 5 tasks above
2. Assign Tasks 1–3 (test coverage) to Copilot coding agent — these are independent and can be parallelized
3. Assign Task 4 (refactor) after Task 1 test coverage is in place for safety
4. Re-evaluate in 2 weeks — target: zero validators over 300 lines, all >200-line validators have tests

---

References:

• §23648542324

AI generated by Repository Quality Improvement Agent · history

• expires on Mar 28, 2026, 1:35 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-28T13:35:42.111Z.

Closed by Workflow