[Quality] Workflow Error Message Quality and Actionability - 2025-11-15

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Error Message Quality and Actionability

Analysis Date: 2025-11-15
Focus Area: Workflow Error Message Quality and Actionability
Strategy Type: Custom
Custom Area: Yes - This focus area was specifically chosen for gh-aw because clear, actionable error messages are critical for developer experience when compiling markdown workflows to GitHub Actions YAML. When validation fails, users need precise guidance to fix their workflow configurations.

Executive Summary

The gh-aw repository has established an excellent Error Message Style Guide (.github/instructions/error-messages.instructions.md) that defines a three-part template for error messages: [what's wrong] + [what's expected] + [example]. However, analysis reveals significant room for improvement in adoption and consistency across the codebase.

Key Findings:

• Only 5.6% of error messages include examples (54 out of 956 error sites)
• Strong foundation exists with exemplary error messages in pkg/workflow/time_delta.go and dedicated test coverage in error_message_quality_test.go
• Inconsistent CLI error formatting - only 35 of 95 CLI error messages use console formatting helpers
• Validation errors vary in quality - some follow the guide perfectly, while many lack context, examples, or type information

The codebase demonstrates best practices in isolated areas but needs systematic application of the error message template across all validation paths, especially in compiler validation, MCP configuration errors, and CLI command errors.

Full Analysis Report

Focus Area: Workflow Error Message Quality and Actionability

Current State Assessment

Codebase Scale:

• Total lines of code (non-test): 60,216 lines
• Error message sites: 956 locations
• Validation-specific files: 42 files

Error Message Quality Metrics:

Metric	Value	Status
Error sites with examples	54 / 956 (5.6%)	❌ Low coverage
Errors with format guidance	125 / 956 (13.1%)	⚠️ Needs improvement
Console-formatted CLI errors	35 / 95 (36.8%)	⚠️ Inconsistent
Error sites with type info (%T)	~150 / 956 (15.7%)	⚠️ Good for type errors
Dedicated test coverage	Yes (error_message_quality_test.go)	✅ Excellent
Style guide documentation	Yes (comprehensive)	✅ Excellent

Findings

Strengths

1. Excellent Style Guide - .github/instructions/error-messages.instructions.md provides comprehensive guidance with:

   • Clear three-part template structure
   • Good vs. bad examples with explanations
   • Category-specific patterns (format, type, enum, configuration validation)
   • Testing guidelines

2. Exemplary Reference Implementation - pkg/workflow/time_delta.go demonstrates best practices:

   fmt.Errorf("invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m", deltaStr)

3. Strong Test Coverage - pkg/workflow/error_message_quality_test.go validates error message quality with assertions for:

   • Presence of "Example:" text
   • Inclusion of valid options for enums
   • Descriptive length (>30 characters)
   • Avoidance of vague phrases

4. Console Formatting Infrastructure - The pkg/console package provides consistent formatting helpers that are used in some areas

Areas for Improvement

1. Low Example Inclusion Rate (5.6%)

   • Only 54 of 956 error messages include concrete examples
   • Many validation errors state what's wrong but not how to fix it
   • Users must search documentation or guess at correct formats

2. Inconsistent CLI Error Formatting

   • Only 36.8% of CLI errors use console.FormatErrorMessage()
   • Mix of raw fmt.Fprintf(os.Stderr) and formatted output
   • Some warnings bypass console formatting entirely

3. Vague Error Messages Without Context

   Examples needing improvement:

   // ❌ Too vague - no guidance
   fmt.Errorf("invalid SHA format for %s@%s: %s", repo, version, sha)
   
   // ❌ No example or valid format shown
   fmt.Errorf("invalid repository format: %s (expected owner/repo)", repo)
   
   // ❌ Wrapped error without user-facing context
   fmt.Errorf("invalid MCP configuration: %w", err)

4. Missing Type Information in Type Errors

   • Some "must be X type" errors don't show actual type received
   • Makes debugging more difficult for users
   • Should use %T verb to show concrete type

5. Validation Errors That Need Enhancement

   Files with improvement opportunities:

   • pkg/workflow/compiler.go - MCP config and engine validation errors
   • pkg/workflow/stop_after.go - Time-based validation errors
   • pkg/cli/trial_command.go - CLI argument validation
   • pkg/cli/add_command.go - Workflow specification parsing
   • pkg/parser/schema.go - JSON schema validation errors

Detailed Analysis

Pattern Analysis: What Works Well

Time Delta Validation (pkg/workflow/time_delta.go):

// ✅ Perfect example - clear, actionable, with examples
return nil, fmt.Errorf("invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m", deltaStr)

// ✅ Clear constraint with helpful suggestion
return nil, fmt.Errorf("minute unit 'm' is not allowed for stop-after. Minimum unit is hours 'h'. Use +%dh instead of +%dm", (value+59)/60, value)

// ✅ Specific error with context
return nil, fmt.Errorf("duplicate unit '%s' in time delta: +%s", unit, deltaStr)

Manual Approval Validation (pkg/workflow/manual_approval.go):

// ✅ Shows type received and provides YAML example
return "", fmt.Errorf("manual-approval value must be a string, got %T. Example: manual-approval: \"production\"", manualApproval)

// ✅ Multi-line example with proper YAML formatting
return "", fmt.Errorf("invalid on: section format, got %T. Expected string or object. Example: on: push or on:\n  push:\n    branches: [main]", onSection)

Pattern Analysis: Needs Improvement

Compiler Validation Errors:

// ❌ Wrapped error loses user-facing context
return nil, fmt.Errorf("invalid engine setting '%s': %w", engineSetting, err)

// ❌ No guidance on what makes it invalid
return nil, fmt.Errorf("invalid MCP configuration: %w", err)

// ✅ BETTER: Add context and example
return nil, fmt.Errorf("invalid engine setting '%s': %w. Valid engines: copilot, claude, codex, custom. Example: engine: copilot", engineSetting, err)

Repository Format Errors:

// ❌ States expected format but no example
return false, fmt.Errorf("invalid repository format: %s (expected owner/repo)", repo)

// ✅ BETTER: Add concrete example
return false, fmt.Errorf("invalid repository format: %s. Expected format: owner/repo. Example: githubnext/gh-aw", repo)

Stop-After Validation:

// ❌ Too vague - what's wrong with the format?
return "", fmt.Errorf("invalid on: section format")

// ✅ BETTER: Explain what's expected
return "", fmt.Errorf("invalid on: section format, got %T. Expected object with 'stop-after' field. Example:\non:\n  issues:\n    types: [opened]\n  stop-after: +24h", onSection)

CLI Error Message Consistency Issues

Current State:

• 95 total CLI error output sites
• 35 use console.FormatErrorMessage() (36.8%)
• 60 use raw fmt.Fprintf(os.Stderr) (63.2%)

Inconsistency Example from pkg/cli/compile_command.go:

// ✅ Good - uses console formatting
fmt.Fprintln(os.Stderr, console.FormatErrorMessage(fmt.Sprintf("failed to parse workflow file %s: %v", file, err)))

// ❌ Inconsistent - raw output without formatting
fmt.Fprintf(os.Stderr, "⚠️  Failed to save action cache: %v\n", err)

// ❌ Warning without console.FormatWarningMessage
fmt.Fprintf(os.Stderr, "Warning: Unable to fetch GitHub workflows: %v\n", err)

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

Task 1: Enhance Compiler Validation Error Messages

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Error Message Quality

Description:
Improve error messages in pkg/workflow/compiler.go to follow the error message style guide. Focus on validation errors that currently wrap underlying errors without providing user-facing context or examples.

Acceptance Criteria:

• All validation errors include "what's wrong" + "what's expected" + "example"
• Engine validation errors list valid options (copilot, claude, codex, custom)
• MCP configuration errors provide concrete YAML examples
• Type mismatch errors use %T to show received type
• Multi-line YAML examples use proper formatting with \n

Code Region: pkg/workflow/compiler.go (validation error messages)

Improve validation error messages in pkg/workflow/compiler.go to follow the error message style guide defined in .github/instructions/error-messages.instructions.md.

Focus on these specific patterns:
1. Lines with `fmt.Errorf("invalid engine setting")` - add valid engine list and example
2. Lines with `fmt.Errorf("invalid MCP configuration")` - add specific YAML example
3. Lines with `fmt.Errorf("invalid campaign")` - explain what makes it invalid and show valid format
4. Any error that wraps another error with %w - add user-facing context before the wrapped error

Follow the three-part template:
- What's wrong: Clearly state the validation error
- What's expected: Explain valid format/values
- How to fix: Provide concrete example

Examples from time_delta.go show the pattern well - use similar clarity and completeness.

---

Task 2: Standardize CLI Error Message Formatting

Priority: High
Estimated Effort: Large
Focus Area: Workflow Error Message Quality

Description:
Standardize all CLI error messages in pkg/cli/ to use console formatting helpers consistently. Currently only 36.8% of CLI errors use the console package, leading to inconsistent user experience.

Acceptance Criteria:

• All error messages use console.FormatErrorMessage(err.Error())
• All warnings use console.FormatWarningMessage(message)
• All info messages use console.FormatInfoMessage(message)
• Remove raw fmt.Fprintf(os.Stderr, "Error: ...") patterns
• Remove emoji-prefixed messages (⚠️, ❌) in favor of console formatting
• Maintain existing error context and information

Code Region: pkg/cli/*.go (all CLI command files)

Standardize error message formatting across all CLI command files in pkg/cli/ to use console formatting helpers consistently.

Replace patterns like:
- `fmt.Fprintf(os.Stderr, "⚠️  Failed to save: %v\n", err)` 
  with `fmt.Fprintln(os.Stderr, console.FormatWarningMessage(fmt.Sprintf("Failed to save: %v", err)))`

- `fmt.Fprintf(os.Stderr, "Warning: %v\n", err)`
  with `fmt.Fprintln(os.Stderr, console.FormatWarningMessage(err.Error()))`

- `fmt.Fprintf(os.Stderr, "Error: %v\n", err)`
  with `fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))`

Focus on files:
- pkg/cli/compile_command.go (multiple warning/error sites)
- pkg/cli/enable.go (workflow operation errors)
- pkg/cli/add_command.go (validation errors)
- pkg/cli/trial_command.go (specification errors)

Reference: See AGENTS.md section on "Console Message Formatting" for the canonical pattern.

---

Task 3: Add Examples to Repository Format Validation Errors

Priority: High
Estimated Effort: Small
Focus Area: Workflow Error Message Quality

Description:
Add concrete examples to repository format validation errors across the codebase. These are common user-facing errors that currently lack actionable guidance.

Acceptance Criteria:

• All "invalid repository format" errors include example (e.g., "githubnext/gh-aw")
• All "invalid repo slug" errors show expected format
• All "expected owner/repo" messages provide concrete example
• Error messages follow three-part template

Code Region:

• pkg/workflow/repository_features_validation.go
• pkg/cli/trial_command.go
• pkg/cli/repo.go

Enhance repository format validation errors to include concrete examples following the error message style guide.

Current pattern:
```go
return false, fmt.Errorf("invalid repository format: %s (expected owner/repo)", repo)

Improved pattern:

return false, fmt.Errorf("invalid repository format: %s. Expected format: owner/repo. Example: githubnext/gh-aw", repo)

Apply to all repository validation errors in:

1. pkg/workflow/repository_features_validation.go
2. pkg/cli/trial_command.go (multiple specification parsing errors)
3. pkg/cli/repo.go (slug parsing errors)

Each error should clearly state:

• What's wrong: "invalid repository format: %s"
• What's expected: "Expected format: owner/repo"
• How to fix: "Example: githubnext/gh-aw"

---

#### Task 4: Improve MCP Configuration Error Messages

**Priority**: Medium  
**Estimated Effort**: Medium  
**Focus Area**: Workflow Error Message Quality

**Description:**
Enhance MCP (Model Context Protocol) configuration validation errors to provide clear YAML examples and explain valid configuration options. MCP errors are particularly important as they involve complex nested YAML structures.

**Acceptance Criteria:**
- [ ] Missing required field errors show complete valid example
- [ ] Type validation errors show received type and expected type
- [ ] Invalid property errors list valid properties
- [ ] Multi-line YAML examples are properly formatted
- [ ] Errors explain mutual exclusivity (command vs container)

**Code Region:** `pkg/workflow/mcp-config.go`, `pkg/parser/mcp.go`

```markdown
Improve MCP configuration validation error messages to follow the error message style guide with emphasis on YAML examples.

Reference the excellent MCP error examples in .github/instructions/error-messages.instructions.md:

```go
// ✅ Example to follow
return fmt.Errorf("tool '%s' mcp configuration must specify either 'command' or 'container'. Example:\ntools:\n  %s:\n    command: \"npx `@my/tool`\"", toolName, toolName)

Apply similar quality to all MCP validation in:

1. pkg/workflow/mcp-config.go - tool configuration validation
2. pkg/parser/mcp.go - MCP schema parsing errors

Each error should include:

• Clear statement of what's wrong
• Explanation of valid configuration options
• Multi-line YAML example showing correct usage
• Use tool name in examples when available for personalization

---

#### Task 5: Add Type Information to Type Validation Errors

**Priority**: Medium  
**Estimated Effort**: Small  
**Focus Area**: Workflow Error Message Quality

**Description:**
Add actual type information using `%T` verb to all type mismatch validation errors. This helps users understand exactly what type was provided versus what was expected.

**Acceptance Criteria:**
- [ ] All "must be a string" errors show received type with `%T`
- [ ] All "must be a boolean" errors show received type
- [ ] All "must be an object" errors show received type
- [ ] All "must be an integer" errors show received type
- [ ] Each error includes example with correct type

**Code Region:** Search for `"must be a\|must be an\|should be a\|expected.*type"` in `pkg/workflow/*.go`

```markdown
Add type information to type validation errors across workflow validation code.

Pattern to find: Any error message containing "must be a", "must be an", "should be a", or "expected" + "type"

Current pattern:
```go
return fmt.Errorf("timeout-minutes must be an integer")

Improved pattern:

return fmt.Errorf("timeout-minutes must be an integer, got %T. Example: timeout-minutes: 10", value)

Search in pkg/workflow/ for all type validation errors and enhance them with:

1. Actual type received using %T verb
2. Expected type clearly stated
3. Concrete example showing correct usage

This makes errors immediately actionable - users can see "got string" vs "expected integer" and know exactly what to change.

---

## 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2025-11-15 | Workflow Error Message Quality | Custom | Y | First quality improvement run - established baseline for custom focus area strategy |

**Note**: This is the inaugural run of the Repository Quality Improvement Agent. Future runs will maintain diversity by alternating between custom repository-specific focus areas (60%), standard software quality categories (30%), and strategic reuse (10%).

</details>

---

## 🎯 Recommendations

### Immediate Actions (This Week)

1. **Standardize CLI error formatting** - Priority: High
   - Impact: Affects all user-facing command errors
   - Effort: Large but straightforward pattern replacement
   - Consistency benefit: Uniform error presentation across CLI

2. **Enhance compiler validation errors** - Priority: High
   - Impact: Most common workflow compilation failures
   - Effort: Medium - requires context for each validation
   - User benefit: Faster resolution of workflow syntax errors

3. **Add repository format examples** - Priority: High
   - Impact: Common user error when specifying repos
   - Effort: Small - simple pattern to apply
   - Quick win: Immediate improvement in error actionability

### Short-term Actions (This Month)

1. **Improve MCP configuration errors** - Priority: Medium
   - Impact: Complex YAML configurations need clear examples
   - Effort: Medium - requires understanding of valid MCP configs
   - Benefit: Reduces confusion around MCP server setup

2. **Add type information to validation errors** - Priority: Medium
   - Impact: Makes type mismatches immediately clear
   - Effort: Small - systematic application of %T verb
   - Benefit: Reduces debugging time for type errors

### Long-term Actions (This Quarter)

1. **Automated error message quality testing**
   - Extend `error_message_quality_test.go` to cover more validation paths
   - Add linting rules to catch new errors without examples
   - Create CI check to enforce error message standards

2. **Error message documentation in user docs**
   - Create troubleshooting guide with common errors and solutions
   - Link error messages to relevant documentation sections
   - Build error code system for easy reference

3. **Measure error message effectiveness**
   - Track which errors users encounter most frequently
   - Gather feedback on error message clarity
   - Iterate on examples based on actual user confusion points

---

## 📈 Success Metrics

Track these metrics to measure improvement in **Workflow Error Message Quality and Actionability**:

- **Example inclusion rate**: 5.6% → 50% (increase from 54 to ~478 error messages)
- **CLI error formatting consistency**: 36.8% → 95% (from 35 to 90+ console-formatted errors)
- **Type validation completeness**: 15.7% → 80% (ensure type errors show actual vs expected types)
- **Validation error coverage**: Track % of validation paths with quality error messages
- **User error resolution time**: Measure time from error to fix (via issue analysis)

**Baseline Metrics (2025-11-15):**
- Total error sites: 956
- Errors with examples: 54 (5.6%)
- Console-formatted CLI errors: 35/95 (36.8%)
- Files with validation logic: 42

**Target Metrics (End of Quarter):**
- Errors with examples: 478+ (50%)
- Console-formatted CLI errors: 90+/95 (95%)
- Type errors with %T: 760+/956 (80%)
- New validation error quality test coverage: 100%

---

## Next Steps

1. **Review and prioritize the tasks above** - Focus on high-priority items first
2. **Assign tasks to Copilot agent via planner agent** - Break down into manageable work items
3. **Track progress on improvement items** - Use task acceptance criteria as checkpoints
4. **Re-evaluate this focus area** - Check metrics in 30 days to assess improvement
5. **Schedule next quality run** - Tomorrow (2025-11-16) with different focus area following diversity algorithm

---

## About This Report

This analysis was generated by the **Repository Quality Improvement Agent** using a diversity-driven approach to repository quality assessment:

- **Strategy**: 60% custom focus areas, 30% standard categories, 10% strategic reuse
- **This run**: Custom focus area specific to gh-aw's workflow compilation and validation needs
- **Next run**: Will select a different aspect of the software development lifecycle based on the diversity algorithm
- **Goal**: Continuous, diverse improvement across all aspects of repository quality

The agent analyzes different facets of the codebase daily, creating actionable tasks for the Copilot agent while maintaining historical context for trend analysis.

---

*Generated by Repository Quality Improvement Agent*  
*Next analysis: 2025-11-16 - Focus area will be selected based on diversity algorithm*


> AI generated by [Repository Quality Improvement Agent](https://github.com/githubnext/gh-aw/actions/runs/19390373005)

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.