Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 85 additions & 1 deletion skill/agent-ready/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Evaluate repository maturity for AI agent collaboration using the 10 Pillars / 5
| 1 | docs | README, AGENTS.md, API docs |
| 2 | style | Linting, formatting, types |
| 3 | build | Build scripts, CI/CD |
| 4 | test | Unit, integration, coverage |
| 4 | test | Unit, integration, coverage, **[BDT methodology](#testing-methodology-behavior-driven-testing)** |
| 5 | security | Secrets, dependabot, SAST |
| 6 | observability | Logging, tracing, metrics |
| 7 | env | .env.example, devcontainer |
Expand Down Expand Up @@ -164,9 +164,93 @@ npx agent-ready init . --level L2
npx agent-ready init . --level L2 --dry-run
```

## Testing Methodology (Behavior-Driven Testing)

The `test` pillar includes a full testing methodology based on **Behavior-Driven Testing (BDT)** — start from user behavior, not code structure. Every user-reachable path must be tested.

### Core Principles

1. **Behavior over Implementation** - Test what users see, not how code works
2. **Exhaustive Coverage** - Every branch, every condition, every edge case
3. **Context Awareness** - Every test must define its preconditions explicitly
4. **Real Environment Validation** - Mocks are tools, not destinations

### Workflow

```
Analysis → Design → Execution → Verify Coverage → Ship (or loop back)
```

- **Analysis**: Requirements definition, code change tracking, state machine analysis, branch mapping
- **Design**: Test case design (equivalence partitioning, boundary analysis), impact analysis, prioritization
- **Execution**: Test data preparation, implementation, execution, coverage verification

### Must-Test Branches (Quick Reference)

| Category | Test Cases | Priority |
|----------|------------|:--------:|
| **Empty values** | null, undefined, "", " " (whitespace), [], {} | P0 |
| **Boundaries** | min-1, min, min+1, max-1, max, max+1 | P1 |
| **Auth states** | logged in, logged out, loading, session expired | P0 |
| **API responses** | 200+data, 200+empty, 400, 401, 403, 404, 500, timeout, offline | P0 |
| **User chaos** | double-click, rapid navigation, refresh mid-action, back button | P1 |

### Branch Matrix Template

For each code change, create a branch matrix:

```markdown
| ID | Condition | True Behavior | False Behavior | Priority | Status |
|----|-----------|---------------|----------------|:--------:|:------:|
| B01 | condition_a | Do X | Do Y | P0 | ⬜ |
| B02 | condition_b | Proceed | Show error | P0 | ⬜ |
| B03 | boundary | Edge case | - | P1 | ⬜ |

Status: ⬜ Pending | ✅ Passed | ❌ Failed
```

### Common Mistakes

| Mistake | Why It's Bad | Fix |
|---------|--------------|-----|
| Only happy path | Error paths are 50% of code | Test ALL branches |
| Skip empty value tests | Most common production bugs | Test null, undefined, "", whitespace separately |
| Mock everything | Mocks hide real problems | Add integration + E2E tests |
| Ignore loading states | Users interact during load | Test loading behavior |

### Pre-Release Checklist

```markdown
## Branch Matrix
- [ ] All P0 branches tested
- [ ] All P1 branches tested
- [ ] No untested edge cases

## Test Types
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Coverage thresholds met

## Real Environment
- [ ] E2E tests pass on staging
- [ ] Core paths verified in real environment
```

### BDT Detailed References

Load these only when you need detailed guidance for a specific phase:

- **Analysis phase**: `references/testing/analysis-phase.md` — Gherkin specs, state machines, branch mapping
- **Design phase**: `references/testing/design-phase.md` — Equivalence partitioning, boundary analysis, decision tables
- **Execution phase**: `references/testing/execution-phase.md` — Fixtures, factories, test execution strategy
- **Branch matrices**: `references/testing/branch-matrices.md` — Templates for auth, API, input, error branches
- **Test templates**: `references/testing/test-templates.md` — Copy-paste unit, integration, E2E templates
- **Testing principles**: `references/testing/testing-principles.md` — Mock vs real, context matrices, progressive strategy

## References

- **Scoring rubric**: `references/scoring-rubric.md`
- **Analysis patterns**: `references/analysis-patterns.md`
- **Pillar details**: `references/pillars.md`
- **Level requirements**: `references/levels.md`
- **Testing methodology**: `references/testing/` (BDT — 6 reference files)
12 changes: 11 additions & 1 deletion skill/agent-ready/references/pillars.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ jobs:

## 4. Testing (test)

**Purpose**: Verify code works correctly, enabling confident changes.
**Purpose**: Verify code works correctly, enabling confident changes. Includes **Behavior-Driven Testing (BDT)** methodology for systematic branch coverage.

### Key Components
| Component | Level | Purpose |
Expand All @@ -123,11 +123,21 @@ jobs:
| Test command | L1 | `npm test` documented |
| Coverage | L3 | Track test coverage % |
| Coverage threshold | L4 | Enforce minimum coverage |
| Branch matrix | L4 | BDT: systematic branch tracking |
| BDT methodology | L5 | Full behavior-driven testing with P0/P1 coverage |

### Why It Matters for Agents
- Tests = validation that changes work
- Coverage = confidence in what's tested
- Test patterns = templates for new tests
- **BDT branch matrices** = ensures agents test ALL branches, not just happy paths

### BDT Quick Reference

P0 branches (must always test): empty values, auth states, API error responses
P1 branches (should test): boundary values, concurrent actions, loading states

See `references/testing/` for complete methodology, templates, and branch matrix examples.

---

Expand Down
11 changes: 9 additions & 2 deletions skill/agent-ready/references/scoring-rubric.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,22 @@ README.md exists?
| 0-20 | No tests or only placeholder |
| 21-40 | Some unit tests, low coverage |
| 41-60 | Good unit tests, >50% coverage |
| 61-80 | Unit + integration, >80% coverage |
| 81-100| Mutation testing, property tests |
| 61-80 | Unit + integration, >80% coverage, systematic branch coverage |
| 81-100| BDT methodology: branch matrix tracked, all P0/P1 branches covered, mutation/property tests |

**Key quality indicators:**
- Do tests actually run and pass?
- What's the code coverage percentage?
- Are edge cases tested?
- Are there integration/e2e tests?

**BDT methodology indicators (L4+):**
- Is there a branch matrix tracking coverage of all conditions?
- Are P0 branches (empty values, auth states, API errors) all tested?
- Are boundary values (min-1, min, min+1, max-1, max, max+1) tested?
- Are error paths tested, not just happy paths?
- See `references/testing/` for full BDT methodology

### 5. Security (security)

| Score | Criteria |
Expand Down
188 changes: 188 additions & 0 deletions skill/agent-ready/references/testing/analysis-phase.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
# Part I: Analysis Phase

## 1. Requirements Definition

**Before touching code, define what "correct" means.**

### 1.1 User Story Format

```
As a [role],
I want to [action],
So that [benefit].

Acceptance Criteria:
- Given [precondition], when [action], then [expected result]
- Given [precondition], when [action], then [expected result]
```

### 1.2 Gherkin Specification

```gherkin
Feature: [Feature Name]
As a [role]
I want [capability]
So that [benefit]

Background:
Given [common preconditions]

@critical @happy-path
Scenario: Successful operation
Given [specific preconditions]
When [user action]
Then [expected outcome]
And [additional verification]

@critical @error-handling
Scenario: Operation fails gracefully
Given [failure preconditions]
When [user action]
Then [error handling behavior]
And [system remains stable]

@edge-case
Scenario: Boundary condition handling
Given [boundary preconditions]
When [user action]
Then [boundary behavior]
```

### 1.3 Acceptance Criteria Checklist

| Category | Questions to Answer |
|----------|---------------------|
| **Functionality** | What should happen? What should NOT happen? |
| **Performance** | Response time requirements? Throughput limits? |
| **Security** | Authentication required? Authorization checks? |
| **Usability** | Error messages clear? Recovery path available? |
| **Compatibility** | Browser/device support? API versioning? |
| **Data Integrity** | Validation rules? Consistency requirements? |

---

## 2. Code Change Tracking

**Know exactly what changed before testing.**

### 2.1 Change Classification

| Change Type | Test Impact | Strategy |
|-------------|-------------|----------|
| **New Function** | New test cases needed | Full coverage of new code |
| **Modified Function** | Existing + new tests | Regression + new behavior |
| **Refactored Code** | Behavior verification | Before/after comparison |
| **Deleted Code** | Remove obsolete tests | Verify no dependencies |
| **Configuration** | Environment testing | Multi-environment validation |

### 2.2 Change Log Template

```markdown
## Change Summary

### Files Modified
| File | Type | Impact | Complexity |
|------|------|--------|:----------:|
| `path/to/file.ts` | Modified | High | ●●●○○ |

### Functions Changed
| Function | Change | Lines | Branches Added |
|----------|--------|:-----:|:--------------:|
| `functionName()` | Modified | 45-89 | +3 |

### Dependencies Affected
- Component A depends on modified function
- Service B calls refactored code
```

### 2.3 Complexity Scoring

```
●○○○○ (1) - Simple: Linear logic, no branches
●●○○○ (2) - Low: 1-2 branches, straightforward
●●●○○ (3) - Medium: 3-5 branches, some edge cases
●●●●○ (4) - High: 6-10 branches, complex logic
●●●●● (5) - Critical: 10+ branches, intricate state
```

---

## 3. State Machine Analysis

**Every UI is a state machine. Map all states and transitions.**

### 3.1 State Diagram Template

```dot
digraph state_machine {
rankdir=LR;
node [shape=circle];

idle [label="Idle\n(initial)"];
loading [label="Loading"];
success [label="Success"];
error [label="Error"];

idle -> loading [label="trigger"];
loading -> success [label="complete"];
loading -> error [label="fail"];
success -> idle [label="reset"];
error -> idle [label="dismiss"];
error -> loading [label="retry"];
}
```

### 3.2 State Transition Matrix

```markdown
| Current State | Event | Next State | Side Effects | Guards |
|---------------|-------|------------|--------------|--------|
| idle | submit | loading | show spinner | input valid |
| idle | submit | idle | show error | input invalid |
| loading | success | success | hide spinner | - |
| loading | failure | error | hide spinner | - |
| loading | timeout | error | show timeout msg | - |
| error | retry | loading | clear error | - |
| error | dismiss | idle | clear error | - |
```

### 3.3 Invalid Transitions (Must Reject)

| Current State | Invalid Event | Expected Behavior |
|---------------|---------------|-------------------|
| loading | submit | Ignore (button disabled) |
| loading | navigate away | Warn or cancel |
| error | auto-retry | No retry without user consent |

---

## 4. Branch Mapping

**The CORE artifact. Map ALL code paths systematically.**

### 4.1 Branch Identification Rules

| Code Pattern | Test Requirement | Minimum Cases |
|--------------|------------------|:-------------:|
| `if (A)` | Both outcomes | 2 |
| `if (A && B)` | All combinations that affect outcome | 3+ |
| `if (A \|\| B)` | All combinations that affect outcome | 3+ |
| `A ? B : C` | Both branches | 2 |
| `switch(x)` | Each case + default | N+1 |
| `try/catch` | Normal + exception | 2 |
| `?.` optional chain | Exists + null/undefined | 2 |
| `??` nullish coalesce | Has value + nullish | 2 |
| `async/await` | Resolve + reject + timeout | 3+ |
| `callback(ok, err)` | All callback paths | 2+ |

### 4.2 Branch Matrix Template

```markdown
| ID | Condition | True Behavior | False Behavior | Priority | Status |
|----|-----------|---------------|----------------|:--------:|:------:|
| B01 | `condition` | action A | action B | P0 | ⬜ |

Status: ⬜ Pending | ✅ Passed | ❌ Failed | ⏭️ Skipped
```

See [branch-matrices.md](branch-matrices.md) for complete standard branch categories.
Loading
Loading