Skip to content

✨ feat(ci): create automated testing workflow for devcontainer features - #260

Merged
codekiln merged 2 commits into
codekiln/240-implement-automated-ci-testingfrom
codekiln/247-create-automated-testing-workflow
Nov 24, 2025
Merged

✨ feat(ci): create automated testing workflow for devcontainer features#260
codekiln merged 2 commits into
codekiln/240-implement-automated-ci-testingfrom
codekiln/247-create-automated-testing-workflow

Conversation

@codekiln

Copy link
Copy Markdown
Owner

Summary

Establishes the foundational CI workflow structure for automated devcontainer feature testing, following production best practices identified in research issue #241.

This PR creates the GitHub Actions workflow file and configures all necessary triggers for automated testing of devcontainer features.

Changes

Created Workflow File

  • File: .github/workflows/test-features.yml
  • Purpose: Automated CI testing for devcontainer features

Configured Triggers

The workflow triggers in the following scenarios:

  1. Push to main branch (when feature files change)

    • Path filter: .devcontainer/features/**
    • Ensures main branch always has tested features
  2. Pull requests (when feature files are modified)

    • Path filter: .devcontainer/features/**
    • Catches issues before merge
  3. Version tags (v*)

    • Triggers when langstar binary releases happen
    • Validates feature compatibility with new releases
  4. Manual dispatch (workflow_dispatch)

    • Allows ad-hoc testing when needed
    • Useful for debugging and verification

OS Matrix Testing

  • Ubuntu 22.04: mcr.microsoft.com/devcontainers/base:ubuntu-22.04
  • Ubuntu 24.04: mcr.microsoft.com/devcontainers/base:ubuntu-24.04
  • Strategy: fail-fast: false (test all combinations even if one fails)

Workflow Steps

  1. Checkout code
  2. Set up Node.js 20
  3. Install Dev Container CLI (@devcontainers/cli)
  4. Verify CLI installation
  5. Discover features to test
  6. Test feature installation (placeholder for actual implementation)
  7. Report results

Implementation Status

Completed in this PR:

  • Workflow file structure
  • Trigger configuration
  • OS matrix setup
  • Dev Container CLI installation
  • Feature discovery scaffolding

Pending in follow-up tasks:

Research Reference

Based on production patterns from:

  • devcontainers/features repository CI workflows
  • Research document: reference/research/241-devcontainer-feature-ci-testing/devcontainer-feature-ci-testing-best-practices-2025-11-22.md
  • Specific patterns from lines 66-90 (workflow structure)
  • Best practices from lines 130-157 (automated testing, isolation, clear logs)

Related Issues

Fixes #247

Parent Issue: #240 (201.3-devcontainer-feature-ci)
Epic: #201 (devcontainer-feature milestone)
Milestone: devcontainer-feature

Testing

The workflow is valid YAML and follows the project's CI patterns established in:

  • .github/workflows/ci.yml (trigger and job structure)
  • .github/workflows/release.yml (version tag triggers)

The workflow will be validated when:

  1. This PR is merged to parent branch codekiln/240-implement-automated-ci-testing
  2. Feature changes are pushed or PRs created
  3. Manual dispatch is triggered via GitHub Actions UI

Next Steps

After merging this PR, the next sub-issue (#248) will implement the actual test execution using the Dev Container CLI to:

  • Build test containers with the langstar feature installed
  • Verify feature installation succeeds
  • Run smoke tests to ensure langstar command is available

🤖 Generated with Claude Code

Establishes foundational CI workflow structure for automated devcontainer
feature testing following production best practices from research (#241).

Changes:
- Created .github/workflows/test-features.yml
- Configured triggers:
  - Push to main (paths: .devcontainer/features/**)
  - Pull requests (paths: .devcontainer/features/**)
  - Version tags (v*) when binary releases happen
  - Manual workflow_dispatch for ad-hoc testing
- Set up OS matrix testing (Ubuntu 22.04, 24.04)
- Added Dev Container CLI installation
- Included feature discovery step
- Added placeholder for actual test implementation

Implementation Notes:
This workflow establishes the CI structure and triggers. Subsequent tasks
from parent issue #240 will implement the actual testing:
- #248: Dev Container CLI-based feature installation testing
- #249: OS distribution matrix testing (already scaffolded)
- #250: Smoke tests to verify langstar command
- #251: Feature metadata linting

Research Reference:
Based on devcontainers/features CI patterns documented in:
reference/research/241-devcontainer-feature-ci-testing/
devcontainer-feature-ci-testing-best-practices-2025-11-22.md (lines 66-90)

Fixes #247

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR establishes the foundational CI infrastructure for automated devcontainer feature testing by creating a GitHub Actions workflow that will validate feature installations across multiple Ubuntu versions.

Key Changes:

  • Created .github/workflows/test-features.yml with triggers for push to main, pull requests, version tags, and manual dispatch
  • Configured OS matrix testing for Ubuntu 22.04 and 24.04 using Microsoft's devcontainer base images
  • Set up scaffolding for feature discovery and testing with Dev Container CLI (actual test implementation deferred to issue #248)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/test-features.yml
Comment thread .github/workflows/test-features.yml
Comment thread .github/workflows/test-features.yml
- Separate tag trigger from branch trigger to fix path filter issue
- Add documentation comment explaining unused features output
- Add error handling for missing/empty features directory

Addresses Copilot review comments in PR #260

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@codekiln

Copy link
Copy Markdown
Owner Author

Thanks @copilot for the detailed review! All three issues have been addressed in commit 5df5172:

1. Trigger configuration fix (comment)

  • Separated the tag trigger from the branch trigger
  • Branch pushes to main now only trigger when feature files change
  • Tag pushes always trigger (testing feature compatibility with new releases)

2. Documentation for unused output (comment)

3. Error handling for find command (comment)

  • Added check for directory existence with clear error message
  • Added check for empty features directory with graceful exit

Copilot AI commented Nov 24, 2025

Copy link
Copy Markdown
Contributor

@codekiln I've opened a new pull request, #261, to work on those changes. Once the pull request is ready, I'll request review from you.

@codekiln
codekiln merged commit 35ec8dc into codekiln/240-implement-automated-ci-testing Nov 24, 2025
4 checks passed
@codekiln
codekiln deleted the codekiln/247-create-automated-testing-workflow branch November 24, 2025 12:11
codekiln added a commit that referenced this pull request Nov 24, 2025
…278)

* ✨ feat(ci): create automated testing workflow for devcontainer features (#260)

* ✨ feat(ci): create automated testing workflow for devcontainer features

Establishes foundational CI workflow structure for automated devcontainer
feature testing following production best practices from research (#241).

Changes:
- Created .github/workflows/test-features.yml
- Configured triggers:
  - Push to main (paths: .devcontainer/features/**)
  - Pull requests (paths: .devcontainer/features/**)
  - Version tags (v*) when binary releases happen
  - Manual workflow_dispatch for ad-hoc testing
- Set up OS matrix testing (Ubuntu 22.04, 24.04)
- Added Dev Container CLI installation
- Included feature discovery step
- Added placeholder for actual test implementation

Implementation Notes:
This workflow establishes the CI structure and triggers. Subsequent tasks
from parent issue #240 will implement the actual testing:
- #248: Dev Container CLI-based feature installation testing
- #249: OS distribution matrix testing (already scaffolded)
- #250: Smoke tests to verify langstar command
- #251: Feature metadata linting

Research Reference:
Based on devcontainers/features CI patterns documented in:
reference/research/241-devcontainer-feature-ci-testing/
devcontainer-feature-ci-testing-best-practices-2025-11-22.md (lines 66-90)

Fixes #247

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* 🩹 fix(ci): address Copilot PR review comments

- Separate tag trigger from branch trigger to fix path filter issue
- Add documentation comment explaining unused features output
- Add error handling for missing/empty features directory

Addresses Copilot review comments in PR #260

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

* ✨ feat(ci): implement Dev Container CLI testing for features (#263)

* ✨ feat(ci): implement Dev Container CLI testing for features

Implements headless, automated testing of devcontainer features using the
official Dev Container CLI, following production best practices from
devcontainers/features repository.

Changes:
- Replace TODO placeholder with full Dev Container CLI test implementation
- Use 'devcontainer up' to build and start test containers with features
- Use 'devcontainer exec' to verify feature installation (command presence and version)
- Create test devcontainer.json for each feature with absolute feature paths
- Add comprehensive build and execution logging with trace-level output
- Capture and display logs on failure for debugging
- Test each feature in isolated temporary directories
- Clean up test directories after each feature test

Testing approach:
- For each feature, create temporary test workspace
- Generate devcontainer.json referencing local feature by absolute path
- Build container from base image (Ubuntu 22.04 or 24.04) with feature
- Execute commands inside container to verify installation
- Verify both command availability and version output
- Display detailed logs on any failure

This implements Best Practice #1 from research (#241): use official Dev
Container CLI for reproducible, headless testing without VS Code.

Fixes #248

Parent Issue: #240 (201.3-devcontainer-feature-ci)
Epic: #201 (devcontainer-feature milestone)
Milestone: devcontainer-feature

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Update .github/workflows/test-features.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Update .github/workflows/test-features.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* ✨ feat(ci): expand OS distribution matrix testing (#266)

Extends the test-features.yml workflow matrix to test devcontainer features
across multiple Linux distributions beyond Ubuntu:

- Ubuntu 22.04 and 24.04 (existing)
- Debian 12 and 11 (new)
- Alpine 3.19 and 3.18 (new)

This ensures broad compatibility across different base images and catches
OS-specific issues early (e.g., package differences, filesystem layouts,
shell variations).

Implements Best Practice #2 from CI testing research: test across multiple
base images to ensure devcontainer features work universally.

Fixes #249

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>

* 🧪 test(devcontainer): add smoke tests for langstar feature (#267)

* 🧪 test(devcontainer): add smoke tests for langstar feature

Creates automated smoke test script that verifies:
- langstar binary is in PATH
- langstar --version command works
- langstar --help command works
- installed version matches requested version (when VERSION is set)

Tests fail fast with clear, actionable error messages to help
diagnose installation issues.

Fixes #250

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Update .devcontainer/features/langstar/test.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Update .devcontainer/features/langstar/test.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* ✨ feat(ci): add feature metadata validation to CI workflow (#270)

* ✨ feat(ci): add feature metadata validation to CI workflow

Add comprehensive validation for devcontainer-feature.json files:
- Validate JSON syntax using jq
- Check required fields (id, version, name, description)
- Verify option definitions have type and description
- Run Dev Container CLI validate command
- Fail fast on validation errors before running feature tests

Fixes #251

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Update .github/workflows/test-features.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Update .github/workflows/test-features.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Update .github/workflows/test-features.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Update .github/workflows/test-features.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix duplicate failed_features entries in option validation (#271)

* Initial plan

* fix: prevent duplicate entries in failed_features list during option validation

Co-authored-by: codekiln <140930+codekiln@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: codekiln <140930+codekiln@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>

* 🧪 test(ci): add explicit container cleanup for test isolation (#273)

Ensures complete test isolation by explicitly stopping and removing
Docker containers after each feature test. This prevents:
- Container accumulation during test runs
- Resource conflicts between tests
- Shared state contamination

Changes:
- Add container cleanup in success path using Docker labels
- Add container cleanup in both error paths (build and exec failures)
- Add test isolation guarantees documentation in workflow output
- Create comprehensive TEST-ISOLATION.md documentation

Container identification uses Dev Container CLI labels:
- devcontainer.local_folder=${TEST_DIR} (primary)
- devcontainer.config_file=${TEST_DIR}/.devcontainer.json (fallback)

Aligns with Best Practice #4: "Use fresh, clean containers for each
CI test, preventing contamination from previous runs"

Fixes #252

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>

* 🔧 build(ci): enable strict mode for required status checks (#275)

* 🔧 build(ci): enable strict mode for required status checks

Configures the main branch ruleset to require branches to be up-to-date
before merging. This ensures all CI tests run on the final code that will
be merged, preventing integration issues and merge conflicts.

Changes:
- Updated main ruleset (ID: 9196293) via GitHub API
- Set strict_required_status_checks_policy to true
- Added comprehensive documentation in docs/dev/procedures.md
- Created enable-strict-status-checks.sh script for applying changes

When test-features workflow runs (on .devcontainer/features/** changes),
it must pass before the PR can be merged. The strict mode ensures branches
are up-to-date, so tests run on the exact code being merged.

Fixes #253

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Update scripts/enable-strict-status-checks.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Update scripts/enable-strict-status-checks.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* ♻️ refactor: improve enable-strict-status-checks.sh error handling and temp file usage

- Replace /tmp hardcoded paths with mktemp for unique temporary files
- Capture and display stderr instead of silencing with 2>/dev/null
- Remove unnecessary use of 'cat' when piping to jq
- Add proper cleanup of temporary files in all code paths

This addresses PR review feedback about security, debugging, and best practices.

* 📚 docs: fix git command pattern in procedures.md

Replace 'git pull origin main' with more explicit pattern:
- git fetch origin
- git merge origin/main (or git rebase origin/main)

This matches the workflow example from the PR description and provides
clearer guidance on updating branches.

* 📚 docs: fix invalid JSON comment in procedures.md

Move inline comment outside JSON block to maintain valid JSON syntax.
Comments are not valid in JSON.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* ✨ feat(ci): add comprehensive logging infrastructure to test workflow

Implements detailed logging for CI test runs as specified in issue #254:

## Changes

### Environment Information
- Added dedicated step to log environment details
- Includes OS, architecture, Docker version, Node.js, npm, and Dev Container CLI versions
- Uses GitHub Actions groups for collapsible output

### Test Execution Logging
- Created structured log files for each test run
- Logs saved to `logs/test-<image-name>.log` with all test output
- Individual feature logs saved separately for build and execution phases
- All output uses `tee` to write to both console and log files

### GitHub Actions Step Summaries
- Generate markdown summaries for each test run
- Display test results in GitHub Actions UI
- Include metrics table with pass/fail counts
- List failed features with failure reasons

### Error Handling & Tracking
- Track test failures with counters (total, passed, failed)
- Continue testing after individual feature failures
- Collect failed feature names with failure reasons
- Exit with error only after all tests complete

### Log Artifact Upload
- Upload logs as artifacts on both success and failure
- Failure logs retained for 30 days
- Success logs retained for 7 days
- Separate artifact names by base image for easy identification

### Structured Output
- All test phases wrapped in GitHub Actions groups
- Clear section headers for build, test, and cleanup phases
- Consistent log formatting across all steps
- Final report step displays summary and references artifacts

## Testing

This implementation establishes the logging infrastructure needed for
effective debugging of CI test failures. The structured, searchable output
reduces debugging time from hours to minutes.

Fixes #254

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* ✨ feat(ci): add comprehensive logging infrastructure to test workflow (#276)

* ✨ feat(ci): add comprehensive logging infrastructure to test workflow

Implements detailed logging for CI test runs as specified in issue #254:

## Changes

### Environment Information
- Added dedicated step to log environment details
- Includes OS, architecture, Docker version, Node.js, npm, and Dev Container CLI versions
- Uses GitHub Actions groups for collapsible output

### Test Execution Logging
- Created structured log files for each test run
- Logs saved to `logs/test-<image-name>.log` with all test output
- Individual feature logs saved separately for build and execution phases
- All output uses `tee` to write to both console and log files

### GitHub Actions Step Summaries
- Generate markdown summaries for each test run
- Display test results in GitHub Actions UI
- Include metrics table with pass/fail counts
- List failed features with failure reasons

### Error Handling & Tracking
- Track test failures with counters (total, passed, failed)
- Continue testing after individual feature failures
- Collect failed feature names with failure reasons
- Exit with error only after all tests complete

### Log Artifact Upload
- Upload logs as artifacts on both success and failure
- Failure logs retained for 30 days
- Success logs retained for 7 days
- Separate artifact names by base image for easy identification

### Structured Output
- All test phases wrapped in GitHub Actions groups
- Clear section headers for build, test, and cleanup phases
- Consistent log formatting across all steps
- Final report step displays summary and references artifacts

## Testing

This implementation establishes the logging infrastructure needed for
effective debugging of CI test failures. The structured, searchable output
reduces debugging time from hours to minutes.

Fixes #254

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* 🩹 fix(ci): sanitize base image name in artifact names

Fixes artifact upload failures caused by invalid characters in artifact names.

## Changes

- Export SAFE_IMAGE_NAME as step output for use across workflow steps
- Use sanitized image name in artifact upload step names
- Update artifact reference in report step to use sanitized name

## Issue

GitHub Actions artifact names cannot contain `/` or `:` characters, but
matrix.baseImage contains these (e.g., "mcr.microsoft.com/devcontainers/base:ubuntu-22.04").
This would cause artifact uploads to fail.

## Solution

The SAFE_IMAGE_NAME variable is now:
1. Computed in bash: `sed 's/[\/:]/-/g'`
2. Exported as step output: `echo "safe_image_name=${SAFE_IMAGE_NAME}" >> $GITHUB_OUTPUT`
3. Used in YAML context: `${{ steps.test-features.outputs.safe_image_name }}`

This ensures artifact names like:
- `test-logs-failure-mcr.microsoft.com-devcontainers-base-ubuntu-22.04`
- `test-logs-success-mcr.microsoft.com-devcontainers-base-debian-12`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: deduplicate tee calls

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* style: small change in formatting

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* style: remove double hyphen

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 🩹 fix(ci): resolve merge conflict in artifact naming

Resolved conflict between HEAD and remote in test-features.yml.
Used safe_image_name (sanitized) instead of matrix.baseImage for
artifact names to avoid invalid characters (/, :) in filenames.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* 🩹 fix(ci): address Copilot PR review comments

- Combine duplicate push triggers in test-features.yml workflow
- Support pre-release versions in langstar test.sh version regex
- Remove unreachable validation code in enable-strict-status-checks.sh
- Remove outdated comment referencing completed issues
- Fix string trimming inconsistency in test summary output

All fixes improve code quality and maintainability. Note: workspace
tests fail due to pre-existing bug on main (cli/src/commands/prompt.rs:183)

* 📚 docs(skill): clarify pre-commit checks require environment sourcing

- Explicitly mention pre-commit checks in critical first step warning
- Add 'Quick Start for Pre-Commit Checks' one-liner at top
- Add Mistake #1 specifically about pre-commit checks without sourcing
- Include exact error symptoms encountered (byte index 8 out of bounds)
- Update Key Takeaways to emphasize pre-commit checks require sourcing
- Make it crystal clear that ALL cargo commands may need environment vars

This would have prevented the churn where pre-commit checks failed with
cryptic panics due to missing LANGSMITH_API_KEY in worktree environment.

* fix: incomplete backtick formatting in md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 🩹 fix(tests): handle empty workspace IDs and improve CI error logging

- Fix panic when slicing empty workspace_id/org_id strings in print_scope_info
- Add length check before slicing to handle IDs shorter than 8 chars
- Capture and display Dev Container CLI validation errors in CI workflow
- All 13 prompt_scoping_test tests now pass

Fixes string slice panic: 'byte index 8 is out of bounds of ``'
Improves CI debugging by showing actual validation errors

* 🩹 fix(ci): disable exit-on-error during validation to capture output

- Add set +e before devcontainer validate command
- Re-enable set -e after capturing exit code and output
- Fixes issue where bash -e would exit before error output could be displayed
- Now validation errors will be properly captured and shown in logs

* 🩹 fix(ci): remove invalid devcontainer validate command

- Remove step 4 'Dev Container CLI validation' entirely
- 'devcontainer features validate' command does not exist
- Dev Container CLI only supports: test, package, publish, info, resolve-dependencies, generate-docs
- Manual JSON validation in steps 1-3 is sufficient for metadata validation

Root cause identified: Unknown arguments: validate, .devcontainer/features/langstar

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants