✨ feat(ci): create automated testing workflow for devcontainer features - #260
Merged
codekiln merged 2 commits intoNov 24, 2025
Conversation
Establishes foundational CI workflow structure for automated devcontainer feature testing following production best practices from research (#241). Changes: - Created .github/workflows/test-features.yml - Configured triggers: - Push to main (paths: .devcontainer/features/**) - Pull requests (paths: .devcontainer/features/**) - Version tags (v*) when binary releases happen - Manual workflow_dispatch for ad-hoc testing - Set up OS matrix testing (Ubuntu 22.04, 24.04) - Added Dev Container CLI installation - Included feature discovery step - Added placeholder for actual test implementation Implementation Notes: This workflow establishes the CI structure and triggers. Subsequent tasks from parent issue #240 will implement the actual testing: - #248: Dev Container CLI-based feature installation testing - #249: OS distribution matrix testing (already scaffolded) - #250: Smoke tests to verify langstar command - #251: Feature metadata linting Research Reference: Based on devcontainers/features CI patterns documented in: reference/research/241-devcontainer-feature-ci-testing/ devcontainer-feature-ci-testing-best-practices-2025-11-22.md (lines 66-90) Fixes #247 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR establishes the foundational CI infrastructure for automated devcontainer feature testing by creating a GitHub Actions workflow that will validate feature installations across multiple Ubuntu versions.
Key Changes:
- Created
.github/workflows/test-features.ymlwith triggers for push to main, pull requests, version tags, and manual dispatch - Configured OS matrix testing for Ubuntu 22.04 and 24.04 using Microsoft's devcontainer base images
- Set up scaffolding for feature discovery and testing with Dev Container CLI (actual test implementation deferred to issue #248)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Separate tag trigger from branch trigger to fix path filter issue - Add documentation comment explaining unused features output - Add error handling for missing/empty features directory Addresses Copilot review comments in PR #260 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Owner
Author
|
Thanks @copilot for the detailed review! All three issues have been addressed in commit 5df5172: 1. Trigger configuration fix (comment)
2. Documentation for unused output (comment)
3. Error handling for find command (comment)
|
Contributor
codekiln
merged commit Nov 24, 2025
35ec8dc
into
codekiln/240-implement-automated-ci-testing
4 checks passed
This was referenced Nov 24, 2025
codekiln
added a commit
that referenced
this pull request
Nov 24, 2025
…278) * ✨ feat(ci): create automated testing workflow for devcontainer features (#260) * ✨ feat(ci): create automated testing workflow for devcontainer features Establishes foundational CI workflow structure for automated devcontainer feature testing following production best practices from research (#241). Changes: - Created .github/workflows/test-features.yml - Configured triggers: - Push to main (paths: .devcontainer/features/**) - Pull requests (paths: .devcontainer/features/**) - Version tags (v*) when binary releases happen - Manual workflow_dispatch for ad-hoc testing - Set up OS matrix testing (Ubuntu 22.04, 24.04) - Added Dev Container CLI installation - Included feature discovery step - Added placeholder for actual test implementation Implementation Notes: This workflow establishes the CI structure and triggers. Subsequent tasks from parent issue #240 will implement the actual testing: - #248: Dev Container CLI-based feature installation testing - #249: OS distribution matrix testing (already scaffolded) - #250: Smoke tests to verify langstar command - #251: Feature metadata linting Research Reference: Based on devcontainers/features CI patterns documented in: reference/research/241-devcontainer-feature-ci-testing/ devcontainer-feature-ci-testing-best-practices-2025-11-22.md (lines 66-90) Fixes #247 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * 🩹 fix(ci): address Copilot PR review comments - Separate tag trigger from branch trigger to fix path filter issue - Add documentation comment explaining unused features output - Add error handling for missing/empty features directory Addresses Copilot review comments in PR #260 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> * ✨ feat(ci): implement Dev Container CLI testing for features (#263) * ✨ feat(ci): implement Dev Container CLI testing for features Implements headless, automated testing of devcontainer features using the official Dev Container CLI, following production best practices from devcontainers/features repository. Changes: - Replace TODO placeholder with full Dev Container CLI test implementation - Use 'devcontainer up' to build and start test containers with features - Use 'devcontainer exec' to verify feature installation (command presence and version) - Create test devcontainer.json for each feature with absolute feature paths - Add comprehensive build and execution logging with trace-level output - Capture and display logs on failure for debugging - Test each feature in isolated temporary directories - Clean up test directories after each feature test Testing approach: - For each feature, create temporary test workspace - Generate devcontainer.json referencing local feature by absolute path - Build container from base image (Ubuntu 22.04 or 24.04) with feature - Execute commands inside container to verify installation - Verify both command availability and version output - Display detailed logs on any failure This implements Best Practice #1 from research (#241): use official Dev Container CLI for reproducible, headless testing without VS Code. Fixes #248 Parent Issue: #240 (201.3-devcontainer-feature-ci) Epic: #201 (devcontainer-feature milestone) Milestone: devcontainer-feature 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update .github/workflows/test-features.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: Update .github/workflows/test-features.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ✨ feat(ci): expand OS distribution matrix testing (#266) Extends the test-features.yml workflow matrix to test devcontainer features across multiple Linux distributions beyond Ubuntu: - Ubuntu 22.04 and 24.04 (existing) - Debian 12 and 11 (new) - Alpine 3.19 and 3.18 (new) This ensures broad compatibility across different base images and catches OS-specific issues early (e.g., package differences, filesystem layouts, shell variations). Implements Best Practice #2 from CI testing research: test across multiple base images to ensure devcontainer features work universally. Fixes #249 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * 🧪 test(devcontainer): add smoke tests for langstar feature (#267) * 🧪 test(devcontainer): add smoke tests for langstar feature Creates automated smoke test script that verifies: - langstar binary is in PATH - langstar --version command works - langstar --help command works - installed version matches requested version (when VERSION is set) Tests fail fast with clear, actionable error messages to help diagnose installation issues. Fixes #250 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update .devcontainer/features/langstar/test.sh Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: Update .devcontainer/features/langstar/test.sh Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ✨ feat(ci): add feature metadata validation to CI workflow (#270) * ✨ feat(ci): add feature metadata validation to CI workflow Add comprehensive validation for devcontainer-feature.json files: - Validate JSON syntax using jq - Check required fields (id, version, name, description) - Verify option definitions have type and description - Run Dev Container CLI validate command - Fail fast on validation errors before running feature tests Fixes #251 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update .github/workflows/test-features.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: Update .github/workflows/test-features.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: Update .github/workflows/test-features.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: Update .github/workflows/test-features.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix duplicate failed_features entries in option validation (#271) * Initial plan * fix: prevent duplicate entries in failed_features list during option validation Co-authored-by: codekiln <140930+codekiln@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: codekiln <140930+codekiln@users.noreply.github.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> * 🧪 test(ci): add explicit container cleanup for test isolation (#273) Ensures complete test isolation by explicitly stopping and removing Docker containers after each feature test. This prevents: - Container accumulation during test runs - Resource conflicts between tests - Shared state contamination Changes: - Add container cleanup in success path using Docker labels - Add container cleanup in both error paths (build and exec failures) - Add test isolation guarantees documentation in workflow output - Create comprehensive TEST-ISOLATION.md documentation Container identification uses Dev Container CLI labels: - devcontainer.local_folder=${TEST_DIR} (primary) - devcontainer.config_file=${TEST_DIR}/.devcontainer.json (fallback) Aligns with Best Practice #4: "Use fresh, clean containers for each CI test, preventing contamination from previous runs" Fixes #252 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * 🔧 build(ci): enable strict mode for required status checks (#275) * 🔧 build(ci): enable strict mode for required status checks Configures the main branch ruleset to require branches to be up-to-date before merging. This ensures all CI tests run on the final code that will be merged, preventing integration issues and merge conflicts. Changes: - Updated main ruleset (ID: 9196293) via GitHub API - Set strict_required_status_checks_policy to true - Added comprehensive documentation in docs/dev/procedures.md - Created enable-strict-status-checks.sh script for applying changes When test-features workflow runs (on .devcontainer/features/** changes), it must pass before the PR can be merged. The strict mode ensures branches are up-to-date, so tests run on the exact code being merged. Fixes #253 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update scripts/enable-strict-status-checks.sh Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: Update scripts/enable-strict-status-checks.sh Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ♻️ refactor: improve enable-strict-status-checks.sh error handling and temp file usage - Replace /tmp hardcoded paths with mktemp for unique temporary files - Capture and display stderr instead of silencing with 2>/dev/null - Remove unnecessary use of 'cat' when piping to jq - Add proper cleanup of temporary files in all code paths This addresses PR review feedback about security, debugging, and best practices. * 📚 docs: fix git command pattern in procedures.md Replace 'git pull origin main' with more explicit pattern: - git fetch origin - git merge origin/main (or git rebase origin/main) This matches the workflow example from the PR description and provides clearer guidance on updating branches. * 📚 docs: fix invalid JSON comment in procedures.md Move inline comment outside JSON block to maintain valid JSON syntax. Comments are not valid in JSON. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ✨ feat(ci): add comprehensive logging infrastructure to test workflow Implements detailed logging for CI test runs as specified in issue #254: ## Changes ### Environment Information - Added dedicated step to log environment details - Includes OS, architecture, Docker version, Node.js, npm, and Dev Container CLI versions - Uses GitHub Actions groups for collapsible output ### Test Execution Logging - Created structured log files for each test run - Logs saved to `logs/test-<image-name>.log` with all test output - Individual feature logs saved separately for build and execution phases - All output uses `tee` to write to both console and log files ### GitHub Actions Step Summaries - Generate markdown summaries for each test run - Display test results in GitHub Actions UI - Include metrics table with pass/fail counts - List failed features with failure reasons ### Error Handling & Tracking - Track test failures with counters (total, passed, failed) - Continue testing after individual feature failures - Collect failed feature names with failure reasons - Exit with error only after all tests complete ### Log Artifact Upload - Upload logs as artifacts on both success and failure - Failure logs retained for 30 days - Success logs retained for 7 days - Separate artifact names by base image for easy identification ### Structured Output - All test phases wrapped in GitHub Actions groups - Clear section headers for build, test, and cleanup phases - Consistent log formatting across all steps - Final report step displays summary and references artifacts ## Testing This implementation establishes the logging infrastructure needed for effective debugging of CI test failures. The structured, searchable output reduces debugging time from hours to minutes. Fixes #254 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * ✨ feat(ci): add comprehensive logging infrastructure to test workflow (#276) * ✨ feat(ci): add comprehensive logging infrastructure to test workflow Implements detailed logging for CI test runs as specified in issue #254: ## Changes ### Environment Information - Added dedicated step to log environment details - Includes OS, architecture, Docker version, Node.js, npm, and Dev Container CLI versions - Uses GitHub Actions groups for collapsible output ### Test Execution Logging - Created structured log files for each test run - Logs saved to `logs/test-<image-name>.log` with all test output - Individual feature logs saved separately for build and execution phases - All output uses `tee` to write to both console and log files ### GitHub Actions Step Summaries - Generate markdown summaries for each test run - Display test results in GitHub Actions UI - Include metrics table with pass/fail counts - List failed features with failure reasons ### Error Handling & Tracking - Track test failures with counters (total, passed, failed) - Continue testing after individual feature failures - Collect failed feature names with failure reasons - Exit with error only after all tests complete ### Log Artifact Upload - Upload logs as artifacts on both success and failure - Failure logs retained for 30 days - Success logs retained for 7 days - Separate artifact names by base image for easy identification ### Structured Output - All test phases wrapped in GitHub Actions groups - Clear section headers for build, test, and cleanup phases - Consistent log formatting across all steps - Final report step displays summary and references artifacts ## Testing This implementation establishes the logging infrastructure needed for effective debugging of CI test failures. The structured, searchable output reduces debugging time from hours to minutes. Fixes #254 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * 🩹 fix(ci): sanitize base image name in artifact names Fixes artifact upload failures caused by invalid characters in artifact names. ## Changes - Export SAFE_IMAGE_NAME as step output for use across workflow steps - Use sanitized image name in artifact upload step names - Update artifact reference in report step to use sanitized name ## Issue GitHub Actions artifact names cannot contain `/` or `:` characters, but matrix.baseImage contains these (e.g., "mcr.microsoft.com/devcontainers/base:ubuntu-22.04"). This would cause artifact uploads to fail. ## Solution The SAFE_IMAGE_NAME variable is now: 1. Computed in bash: `sed 's/[\/:]/-/g'` 2. Exported as step output: `echo "safe_image_name=${SAFE_IMAGE_NAME}" >> $GITHUB_OUTPUT` 3. Used in YAML context: `${{ steps.test-features.outputs.safe_image_name }}` This ensures artifact names like: - `test-logs-failure-mcr.microsoft.com-devcontainers-base-ubuntu-22.04` - `test-logs-success-mcr.microsoft.com-devcontainers-base-debian-12` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: deduplicate tee calls Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * style: small change in formatting Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * style: remove double hyphen Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * 🩹 fix(ci): resolve merge conflict in artifact naming Resolved conflict between HEAD and remote in test-features.yml. Used safe_image_name (sanitized) instead of matrix.baseImage for artifact names to avoid invalid characters (/, :) in filenames. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * 🩹 fix(ci): address Copilot PR review comments - Combine duplicate push triggers in test-features.yml workflow - Support pre-release versions in langstar test.sh version regex - Remove unreachable validation code in enable-strict-status-checks.sh - Remove outdated comment referencing completed issues - Fix string trimming inconsistency in test summary output All fixes improve code quality and maintainability. Note: workspace tests fail due to pre-existing bug on main (cli/src/commands/prompt.rs:183) * 📚 docs(skill): clarify pre-commit checks require environment sourcing - Explicitly mention pre-commit checks in critical first step warning - Add 'Quick Start for Pre-Commit Checks' one-liner at top - Add Mistake #1 specifically about pre-commit checks without sourcing - Include exact error symptoms encountered (byte index 8 out of bounds) - Update Key Takeaways to emphasize pre-commit checks require sourcing - Make it crystal clear that ALL cargo commands may need environment vars This would have prevented the churn where pre-commit checks failed with cryptic panics due to missing LANGSMITH_API_KEY in worktree environment. * fix: incomplete backtick formatting in md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * 🩹 fix(tests): handle empty workspace IDs and improve CI error logging - Fix panic when slicing empty workspace_id/org_id strings in print_scope_info - Add length check before slicing to handle IDs shorter than 8 chars - Capture and display Dev Container CLI validation errors in CI workflow - All 13 prompt_scoping_test tests now pass Fixes string slice panic: 'byte index 8 is out of bounds of ``' Improves CI debugging by showing actual validation errors * 🩹 fix(ci): disable exit-on-error during validation to capture output - Add set +e before devcontainer validate command - Re-enable set -e after capturing exit code and output - Fixes issue where bash -e would exit before error output could be displayed - Now validation errors will be properly captured and shown in logs * 🩹 fix(ci): remove invalid devcontainer validate command - Remove step 4 'Dev Container CLI validation' entirely - 'devcontainer features validate' command does not exist - Dev Container CLI only supports: test, package, publish, info, resolve-dependencies, generate-docs - Manual JSON validation in steps 1-3 is sufficient for metadata validation Root cause identified: Unknown arguments: validate, .devcontainer/features/langstar --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Establishes the foundational CI workflow structure for automated devcontainer feature testing, following production best practices identified in research issue #241.
This PR creates the GitHub Actions workflow file and configures all necessary triggers for automated testing of devcontainer features.
Changes
Created Workflow File
.github/workflows/test-features.ymlConfigured Triggers
The workflow triggers in the following scenarios:
Push to main branch (when feature files change)
.devcontainer/features/**Pull requests (when feature files are modified)
.devcontainer/features/**Version tags (
v*)Manual dispatch (
workflow_dispatch)OS Matrix Testing
mcr.microsoft.com/devcontainers/base:ubuntu-22.04mcr.microsoft.com/devcontainers/base:ubuntu-24.04fail-fast: false(test all combinations even if one fails)Workflow Steps
@devcontainers/cli)Implementation Status
✅ Completed in this PR:
⏳ Pending in follow-up tasks:
langstarcommand worksResearch Reference
Based on production patterns from:
reference/research/241-devcontainer-feature-ci-testing/devcontainer-feature-ci-testing-best-practices-2025-11-22.mdRelated Issues
Fixes #247
Parent Issue: #240 (201.3-devcontainer-feature-ci)
Epic: #201 (devcontainer-feature milestone)
Milestone: devcontainer-feature
Testing
The workflow is valid YAML and follows the project's CI patterns established in:
.github/workflows/ci.yml(trigger and job structure).github/workflows/release.yml(version tag triggers)The workflow will be validated when:
codekiln/240-implement-automated-ci-testingNext Steps
After merging this PR, the next sub-issue (#248) will implement the actual test execution using the Dev Container CLI to:
langstarcommand is available🤖 Generated with Claude Code