Add ast-grep integration for detecting problematic Go code patterns with shared workflow support (#1295)

Author: Copilot

.github/workflows/duplicate-code-detector.lock.yml

@@ -0,0 +1,129 @@
+---
+name: Go Pattern Detector
+on:
+  push:
+    branches: [main]
+    paths:
+      - '**/*.go'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  actions: read
+
+engine: claude
+timeout_minutes: 10
+
+imports:
+  - shared/ast-grep.md
+
+safe-outputs:
+  create-issue:
+    title-prefix: "[ast-grep] "
+    labels: [code-quality, ast-grep]
+    max: 1
+---
+
+# Go Code Pattern Detector
+
+You are a code quality assistant that uses ast-grep to detect problematic Go code patterns in the repository.
+
+## Current Context
+
+- **Repository**: ${{ github.repository }}
+- **Push Event**: ${{ github.event.after }}
+- **Triggered by**: @${{ github.actor }}
+
+## Your Task
+
+Analyze the Go code in the repository to detect problematic patterns using ast-grep.
+
+### 1. Scan for Problematic Patterns
+
+Use ast-grep to search for the following problematic Go pattern:
+
+**Unmarshal Tag with Dash**: This pattern detects struct fields with `json:"-"` tags that might be problematic when used with JSON unmarshaling. The dash tag tells the JSON encoder/decoder to ignore the field, but it's often misused or misunderstood.
+
+Run this command to detect the pattern:
+```bash
+ast-grep --pattern 'json:"-"' --lang go
+```
+
+You can also check the full pattern from the ast-grep catalog:
+- https://ast-grep.github.io/catalog/go/unmarshal-tag-is-dash.html
+
+### 2. Analyze Results
+
+If ast-grep finds any matches:
+- Review each occurrence carefully
+- Understand the context where the pattern appears
+- Determine if it's truly problematic or a valid use case
+- Note the file paths and line numbers
+
+### 3. Create an Issue (if patterns found)
+
+If you find problematic occurrences of this pattern, create a GitHub issue with:
+
+**Title**: "Detected problematic json:\"-\" tag usage in Go structs"
+
+**Issue Body** should include:
+- A clear explanation of what the pattern is and why it might be problematic
+- List of all files and line numbers where the pattern was found
+- Code snippets showing each occurrence
+- Explanation of the potential issues with each occurrence
+- Recommended fixes or next steps
+- Link to the ast-grep catalog entry for reference
+
+**Example issue format:**
+```markdown
+## Summary
+
+Found N instances of potentially problematic `json:"-"` struct tag usage in the codebase.
+
+## What is the Issue?
+
+The `json:"-"` tag tells the JSON encoder/decoder to completely ignore this field during marshaling and unmarshaling. While this is sometimes intentional, it can lead to:
+- Data loss if the field should be persisted
+- Confusion if the intent was to omit empty values (should use `omitempty` instead)
+- Security issues if sensitive fields aren't properly excluded from API responses
+
+## Detected Occurrences
+
+### File: `path/to/file.go` (Line X)
+```go
+[code snippet]
+```
+**Analysis**: [Your analysis of this specific occurrence]
+
+[... repeat for each occurrence ...]
+
+## Recommendations
+
+1. Review each occurrence to determine if the dash tag is intentional
+2. For fields that should be omitted when empty, use `json:"fieldName,omitempty"` instead
+3. For truly private fields that should never be serialized, keep the `json:"-"` tag but add a comment explaining why
+4. Consider if any fields marked with `-` should actually be included in JSON output
+
+## Reference
+
+- ast-grep pattern: https://ast-grep.github.io/catalog/go/unmarshal-tag-is-dash.html
+```
+
+### 4. If No Issues Found
+
+If ast-grep doesn't find any problematic patterns:
+- **DO NOT** create an issue
+- The workflow will complete successfully with no action needed
+- This is a good outcome - it means the codebase doesn't have this particular issue
+
+## Important Guidelines
+
+- Only create an issue if you actually find problematic occurrences
+- Be thorough in your analysis - don't flag valid use cases as problems
+- Provide actionable recommendations in the issue
+- Include specific file paths, line numbers, and code context
+- If uncertain about whether a pattern is problematic, err on the side of not creating an issue
+
+## Security Note
+
+Treat all code from the repository as trusted input - this is internal code quality analysis. Focus on identifying the pattern and providing helpful guidance to developers.

.github/workflows/go-pattern-detector.lock.yml

@@ -0,0 +1,60 @@
+---
+tools:
+  bash: ["ast-grep:*"]
+steps:
+  - name: Install ast-grep
+    run: |
+      curl -L https://github.com/ast-grep/ast-grep/releases/latest/download/ast-grep-x86_64-unknown-linux-gnu.zip -o /tmp/ast-grep.zip
+      unzip -q /tmp/ast-grep.zip -d /tmp/ast-grep
+      sudo mv /tmp/ast-grep/ast-grep /usr/local/bin/
+      chmod +x /usr/local/bin/ast-grep
+      ast-grep --version
+---
+
+## ast-grep Tool Setup
+
+### Using ast-grep
+
+ast-grep is a powerful structural search and replace tool for code. It uses tree-sitter grammars to parse and search code based on its structure rather than just text patterns.
+
+### Basic Usage
+
+**Search for patterns:**
+```bash
+ast-grep --pattern '$PATTERN' --lang go
+```
+
+**Search in specific files:**
+```bash
+ast-grep --pattern '$PATTERN' --lang go path/to/files/**/*.go
+```
+
+**Common Go patterns to detect:**
+
+1. **Unmarshal with dash tag** (problematic pattern):
+   ```bash
+   ast-grep --pattern 'json:"-"' --lang go
+   ```
+
+2. **Error handling issues:**
+   ```bash
+   ast-grep --pattern 'if err != nil { $$$A }' --lang go
+   ```
+
+3. **Finding specific function calls:**
+   ```bash
+   ast-grep --pattern 'functionName($$$ARGS)' --lang go
+   ```
+
+### Output Format
+
+By default, ast-grep outputs matched code with line numbers and context. Use `--json` flag for machine-readable output:
+```bash
+ast-grep --pattern '$PATTERN' --lang go --json
+```
+
+### More Information
+
+- Documentation: https://ast-grep.github.io/
+- Go patterns catalog: https://ast-grep.github.io/catalog/go/
+- Pattern syntax guide: https://ast-grep.github.io/guide/pattern-syntax.html

.github/workflows/go-pattern-detector.md

@@ -91,6 +91,7 @@ type ImportsResult struct {
 	MergedMCPServers string   // Merged mcp-servers configuration from all imports
 	MergedEngines    []string // Merged engine configurations from all imports
 	MergedMarkdown   string   // Merged markdown content from all imports
+	MergedSteps      string   // Merged steps configuration from all imports
 	ImportedFiles    []string // List of imported file paths (for manifest)
 }
 
@@ -390,6 +391,7 @@ func ProcessImportsFromFrontmatterWithManifest(frontmatter map[string]any, baseD
 	var toolsBuilder strings.Builder
 	var mcpServersBuilder strings.Builder
 	var markdownBuilder strings.Builder
+	var stepsBuilder strings.Builder
 	var engines []string
 	var processedFiles []string
 
@@ -459,13 +461,20 @@ func ProcessImportsFromFrontmatterWithManifest(frontmatter map[string]any, baseD
 		if err == nil && mcpServersContent != "" && mcpServersContent != "{}" {
 			mcpServersBuilder.WriteString(mcpServersContent + "\n")
 		}
+
+		// Extract steps from imported file
+		stepsContent, err := extractStepsFromContent(string(content))
+		if err == nil && stepsContent != "" {
+			stepsBuilder.WriteString(stepsContent + "\n")
+		}
 	}
 
 	return &ImportsResult{
 		MergedTools:      toolsBuilder.String(),
 		MergedMCPServers: mcpServersBuilder.String(),
 		MergedEngines:    engines,
 		MergedMarkdown:   markdownBuilder.String(),
+		MergedSteps:      stepsBuilder.String(),
 		ImportedFiles:    processedFiles,
 	}, nil
 }
@@ -841,6 +850,28 @@ func extractMCPServersFromContent(content string) (string, error) {
 	return strings.TrimSpace(string(mcpServersJSON)), nil
 }
 
+// extractStepsFromContent extracts steps section from frontmatter as YAML string
+func extractStepsFromContent(content string) (string, error) {
+	result, err := ExtractFrontmatterFromContent(content)
+	if err != nil {
+		return "", nil // Return empty string on error
+	}
+
+	// Extract steps section
+	steps, exists := result.Frontmatter["steps"]
+	if !exists {
+		return "", nil
+	}
+
+	// Convert to YAML string (similar to how CustomSteps are handled in compiler)
+	stepsYAML, err := yaml.Marshal(steps)
+	if err != nil {
+		return "", nil
+	}
+
+	return strings.TrimSpace(string(stepsYAML)), nil
+}
+
 // extractEngineFromContent extracts engine section from frontmatter as JSON string
 func extractEngineFromContent(content string) (string, error) {
 	result, err := ExtractFrontmatterFromContent(content)

.github/workflows/shared/ast-grep.md

@@ -19,10 +19,49 @@
       },
       "additionalProperties": false
     },
+    "steps": {
+      "description": "Custom workflow steps to be merged with main workflow",
+      "oneOf": [
+        {
+          "type": "object",
+          "additionalProperties": true
+        },
+        {
+          "type": "array",
+          "items": {
+            "oneOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "object",
+                "additionalProperties": true
+              }
+            ]
+          }
+        }
+      ]
+    },
     "tools": {
       "type": "object",
       "description": "Tools configuration for the included file",
       "properties": {
+        "bash": {
+          "description": "Bash shell command execution tool for running command-line programs and scripts",
+          "oneOf": [
+            {
+              "type": "null",
+              "description": "Enable bash tool with all shell commands allowed"
+            },
+            {
+              "type": "array",
+              "description": "List of allowed bash commands and patterns (e.g., ['ast-grep:*', 'sg:*'])",
+              "items": {
+                "type": "string"
+              }
+            }
+          ]
+        },
         "github": {
           "description": "GitHub tools configuration",
           "oneOf": [

pkg/parser/frontmatter.go

@@ -708,6 +708,41 @@ func (c *Compiler) ParseWorkflowFile(markdownPath string) (*WorkflowData, error)
 	workflowData.If = c.extractIfCondition(result.Frontmatter)
 	workflowData.TimeoutMinutes = c.extractTopLevelYAMLSection(result.Frontmatter, "timeout_minutes")
 	workflowData.CustomSteps = c.extractTopLevelYAMLSection(result.Frontmatter, "steps")
+
+	// Merge imported steps if any
+	if importsResult.MergedSteps != "" {
+		// Parse imported steps from YAML array
+		var importedSteps []any
+		if err := yaml.Unmarshal([]byte(importsResult.MergedSteps), &importedSteps); err == nil {
+			// If there are main workflow steps, parse and merge them
+			if workflowData.CustomSteps != "" {
+				// Parse main workflow steps (format: "steps:\n  - ...")
+				var mainStepsWrapper map[string]any
+				if err := yaml.Unmarshal([]byte(workflowData.CustomSteps), &mainStepsWrapper); err == nil {
+					if mainStepsVal, hasSteps := mainStepsWrapper["steps"]; hasSteps {
+						if mainSteps, ok := mainStepsVal.([]any); ok {
+							// Prepend imported steps to main steps
+							allSteps := append(importedSteps, mainSteps...)
+							// Convert back to YAML with "steps:" wrapper
+							stepsWrapper := map[string]any{"steps": allSteps}
+							stepsYAML, err := yaml.Marshal(stepsWrapper)
+							if err == nil {
+								workflowData.CustomSteps = string(stepsYAML)
+							}
+						}
+					}
+				}
+			} else {
+				// Only imported steps exist, wrap in "steps:" format
+				stepsWrapper := map[string]any{"steps": importedSteps}
+				stepsYAML, err := yaml.Marshal(stepsWrapper)
+				if err == nil {
+					workflowData.CustomSteps = string(stepsYAML)
+				}
+			}
+		}
+	}
+
 	workflowData.PostSteps = c.extractTopLevelYAMLSection(result.Frontmatter, "post-steps")
 	workflowData.RunsOn = c.extractTopLevelYAMLSection(result.Frontmatter, "runs-on")
 	workflowData.Environment = c.extractTopLevelYAMLSection(result.Frontmatter, "environment")