📐 Ideas on how PAI could Implement Principle "Spec / Test / Evals First"

[mellanon]

Seeking guidance on spec-driven development approaches

---

The Question

PAI's Principle "Spec / Test / Evals First" states: "If you can't specify it, you can't test it. If you can't test it, you can't trust it."

This resonates deeply, but how do we actually implement it?

While working on PR #183: Context Skill, I experimented with spec-driven patterns—a release framework, change proposals, and a 4-layer test approach. It helped, but I'm curious: what methodology should PAI recommend for contributors?

There's an emerging ecosystem of tools addressing this exact problem. I'd love input on whether/how PAI should integrate any of them.

---

The Spec-Driven Development Ecosystem

Several tools have emerged to address the same problem: aligning humans and AI on intent before implementation.

Tool	Creator	Approach	Key Features
OpenSpec	Fission AI	Spec-first with change tracking	specs/ (truth) + changes/ (proposals), SHALL/MUST format
spec-kit	GitHub	Spec-first with constitution	Spec → Plan → Tasks, "constitution" as immutable rules
Kiro	AWS	Spec-anchored IDE	EARS notation, requirements.md → design.md → tasks.md
Cline Memory Bank	Community	Context persistence	.clinerules, structured markdown for cross-session memory

---

How They Compare

From Martin Fowler's analysis, SDD approaches range across a spectrum:

SPEC-FIRST ──────────────────► SPEC-ANCHORED ──────────────────► SPEC-AS-SOURCE
     │                               │                                 │
 "Write spec                    "Spec is living               "Code generated
  before code"                   reference"                   directly from spec"
     │                               │                                 │
 OpenSpec                         Kiro                            Tessl
 spec-kit                    Cline Memory Bank

---

Tool Deep Dives

OpenSpec (Fission AI)

• Philosophy: Brownfield-friendly change tracking
• Structure: specs/ (current truth) + changes/ (proposals with deltas)
• Format: SHALL/MUST language, ADDED/MODIFIED/REMOVED sections
• Claude Code: /openspec:proposal, /openspec:apply, /openspec:archive
• Strength: Explicit change tracking, works for existing systems (not just greenfield)

spec-kit (GitHub)

• Philosophy: Spec as source of truth with "constitution"
• Structure: Spec → Technical Plan → Tasks
• Format: Constitution (immutable rules) + specification + implementation tasks
• CLI: specify init <project> --ai claude
• Strength: Constitution concept aligns with PAI's CORE principles and CONSTITUTION.md

Kiro (AWS)

• Philosophy: Full IDE for spec-driven development
• Structure: requirements.md → design.md → tasks.md
• Format: EARS notation (Easy Approach to Requirements Syntax)
  • WHEN [condition] THE SYSTEM SHALL [behavior]
• Strength: Integrated IDE experience, hooks for automation

Cline Memory Bank (Community)

• Philosophy: Context persistence across sessions
• Structure: .clinerules, projectbrief.md, activeContext.md, progress.md
• Format: Structured markdown for rebuilding understanding
• Strength: Works with any AI that reads documentation

---

One Possible Approach

Here's what I've been thinking—not a proposal, just ideas to react to:

1. Define the principle (in CORE/SKILL.md):

## Principle 6: Spec/Test/Evals First

WHEN developing features or fixing bugs:
→ FOLLOW spec-first methodology
→ Specifications MUST exist before implementation
→ Tests MUST validate spec requirements

2. Provide methodology as skill:

.claude/skills/SpecFirst/
├── SKILL.md                    ← Entry point: "WHEN developing, follow spec-first"
│
├── openspec/                   ← Skill's own specs (dogfooding the approach)
│   ├── project.md
│   ├── specs/
│   │   ├── release-workflow/
│   │   │   └── spec.md         ← SHALL/MUST for release process
│   │   ├── test-pyramid/
│   │   │   └── spec.md         ← SHALL/MUST for test layers
│   │   └── change-tracking/
│   │       └── spec.md         ← SHALL/MUST for proposals
│   └── changes/
│       └── archive/
│
├── workflows/
│   ├── RELEASE-FRAMEWORK.md    ← Release methodology (see example below)
│   ├── propose.md              ← How to create change proposals
│   └── validate.md             ← How to validate against specs
│
├── templates/
│   ├── RELEASE-vX.Y.Z.md       ← Release plan template (see example below)
│   ├── requirement.md          ← SHALL/MUST format
│   └── test-spec.md            ← Test definition format
│
├── approaches/
│   ├── openspec.md             ← OpenSpec integration guide
│   ├── spec-kit.md             ← spec-kit integration guide
│   └── manual.md               ← Lightweight manual approach
│
└── docs/
    ├── test-pyramid.md         ← 4-layer approach documentation
    └── concepts.md             ← SDD theory and rationale

Real Examples from PR #183

The Context Skill contribution provides working examples:

PR #183 Pattern	What It Does	Location in SpecFirst Skill
RELEASE-FRAMEWORK.md	Reusable release methodology	workflows/RELEASE-FRAMEWORK.md
RELEASE-v1.0.0.md	Concrete change proposal	templates/RELEASE-vX.Y.Z.md
4-layer test pyramid	Spec validation	docs/test-pyramid.md

3. Flexible on tooling, strict on release process:

Spec format (flexible): Choose what works for you:

• OpenSpec (openspec init)
• spec-kit (specify init)
• Lightweight markdown
• Cline Memory Bank

Release management (strict): Follow a defined process:

• Explicit file inventory (what's included/excluded from release)
• Change proposal before implementation
• Pre-release checklist with validation gates
• Clear propagation workflow (branch → PR → upstream)

Why stricter releases matter with AI tools:

AI assistants are eager to help—they generate code quickly, add files liberally, and expand scope readily. This creates new risks:

• Data leakage: Personal config, API keys, or sensitive fixtures accidentally included
• Bloated deployments: Verbose output, debug files, or tangential changes bundled in
• Scope creep: PRs that touch far more than intended

The release process should be more controlled and more manual than development. Move fast during implementation (with test/eval frameworks providing confidence), but slow down at the release gate. The upfront investment in release discipline pays off—you ship faster with quality and without surprises.

---

The Core Pattern (Tool-Agnostic)

Regardless of tooling, the pattern is:

1. SPECIFY
   ┌────────────────────────────────────────────┐
   │ ### Requirement: [Name]                    │
   │ The system SHALL [behavior].               │
   │                                            │
   │ #### Scenario: [Use case]                  │
   │ - WHEN [condition]                         │
   │ - THEN [expected outcome]                  │
   └────────────────────────────────────────────┘
              │
              ▼
2. TEST
   ┌────────────────────────────────────────────┐
   │ TEST-XXX: [Requirement name]               │
   │ command: [test command]                    │
   │ expected: [matches THEN clause]            │
   └────────────────────────────────────────────┘
              │
              ▼
3. IMPLEMENT
   ┌────────────────────────────────────────────┐
   │ Code that makes tests pass                 │
   │ (implementation details)                   │
   └────────────────────────────────────────────┘
              │
              ▼
4. VALIDATE
   ┌────────────────────────────────────────────┐
   │ Tests pass → Spec is implemented           │
   │ Spec updated → Living documentation        │
   └────────────────────────────────────────────┘

---

Real Example: --name Flag

The same requirement expressed across tools:

Tool	Format
OpenSpec	### Requirement: Filename Override The system SHALL allow users to override filenames via --name flag.
Kiro (EARS)	WHEN user provides --name "Custom-Name" THE SYSTEM SHALL use "Custom-Name" as output filename
Manual/PAI	- WHEN user provides --name "X" - THEN filename SHALL be "X"

All express the same intent—the format is secondary to the principle.

---

PAI Development Lifecycle: Tooling vs Specs

A key architectural question: where does test tooling live?

The Three Layers

┌─────────────────────────────────────────────────────────────────┐
│                 PAI DEVELOPMENT LIFECYCLE                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  LAYER 1: METHODOLOGY (SpecFirst Skill)                         │
│  └── .claude/skills/SpecFirst/                                  │
│      ├── RELEASE-FRAMEWORK.md   ← How to release (reusable)     │
│      ├── openspec/              ← How to specify                │
│      └── templates/             ← Standard formats              │
│                                                                 │
│  LAYER 2: TOOLING (pai-tooling - independent repo)              │
│  └── pai-tooling/                                               │
│      ├── framework/             ← Test runners, validators      │
│      │   ├── cli-runner.ts      ← Runs CLI tests               │
│      │   ├── integration-runner.ts                              │
│      │   ├── acceptance-runner.ts                               │
│      │   └── llm-judge.ts       ← AI-powered evaluation        │
│      └── TEST-FRAMEWORK-SPEC.md ← Spec for the tooling itself   │
│                                                                 │
│  LAYER 3: SKILL-SPECIFIC (per skill, coupled to functionality)  │
│  └── pai-tooling/[skill]-test/                                  │
│      ├── specs/                 ← Test cases for THIS skill     │
│      │   ├── cli-direct.spec.ts ← TEST-CLI-014: --name flag    │
│      │   └── scope.spec.ts      ← TEST-SCOPE-001, etc.         │
│      ├── fixtures/              ← Captured test data (private)  │
│      └── output/                ← Test run results              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Why Separate Tooling from Specs?

Aspect	Tooling (Layer 2)	Specs (Layer 3)
Coupling	Independent of any skill	Tightly coupled to skill
Reuse	Shared across all skills	Specific to one skill
Visibility	Public (npm package)	Public (part of skill contribution)
Audience	Framework developers	Skill contributors
Example	cli-runner.ts	cli-direct.spec.ts

Real Example: pai-tooling Structure

pai-tooling/                      ← Could be public
├── ingest-test/
│   ├── framework/                ← PUBLIC: Reusable tooling
│   │   ├── cli-runner.ts
│   │   ├── integration-runner.ts
│   │   ├── llm-judge.ts
│   │   └── types.ts
│   ├── specs/                    ← PUBLIC: Test definitions
│   │   ├── cli-direct.spec.ts    ← "TEST-CLI-014 expects [name:X]"
│   │   └── scope.spec.ts
│   ├── fixtures/                 ← PRIVATE: Captured real data
│   │   └── (gitignored)
│   └── TEST-FRAMEWORK-SPEC.md
└── config/                       ← PRIVATE: Personal paths/IDs
    └── (gitignored)

What's Actually Sensitive?

Component	Sensitive?	Recommendation
framework/	No	Public npm package
specs/	No	Part of skill contribution
fixtures/	Should be No	Use synthetic/public test data, not captured personal content
config/	Yes	.env file, gitignored, never committed

Best practice: Design test fixtures to be non-sensitive from the start—use synthetic data or public documents. The only sensitive component should be config (API tokens, vault paths), which belongs in .env files outside source control.

Where This Fits in SpecFirst Skill

The SpecFirst skill would document this pattern but NOT contain the tooling:

.claude/skills/SpecFirst/
├── docs/
│   ├── test-pyramid.md           ← Documents the 4 layers
│   └── tooling-architecture.md   ← Documents Layer 2/3 separation
└── templates/
    └── test-spec.md              ← How to write specs (Layer 3)

The actual pai-tooling repo remains independent—it's infrastructure that implements the methodology, not the methodology itself.

---

What I'd Love to Hear

1. Does PAI need a formal methodology? Or is the principle enough guidance on its own?

2. Tool-agnostic or tool-specific? Recommend one tool (OpenSpec? spec-kit?) or define patterns that work with any?

3. Where should this live? A dedicated skill? Part of CORE? Just contributor docs?

4. How much structure? Full openspec init ceremony, or lightweight markdown?

5. What's worked for you? If you've implemented this principle, what approach did you take?

---

References

PAI Resources:

• PAI README - PAI architecture and principles
• PR #183: Context Skill - Source of spec-driven patterns
• RELEASE-FRAMEWORK.md - Release methodology example
• RELEASE-v1.0.0.md - Change proposal example

SDD Tools:

• OpenSpec - Fission AI
• spec-kit - GitHub
• Kiro - AWS
• Cline Memory Bank - Community
• Martin Fowler: SDD Tools Comparison

---

@mellanon | December 2025

[mellanon]

Update: Implementation Progress

The ideas discussed here have been developed into concrete implementations:

	PR #189 - SpecFirst	PR #190 - Jira
What	The methodology	Built using the methodology
Files	24 files (+4,965)	20 files (+5,724)
Status	🟢 OPEN	🟢 OPEN
URL	PR #189	PR #190

PR #189 implements the SpecFirst skill methodology outlined in this discussion, while PR #190 demonstrates the methodology in practice by building the Jira skill using spec-driven development.

Feedback welcome on both!

[mellanon]

How It Came Together

The development of these PRs demonstrates the methodology in action:

Foundation: PR #183 (Context Skill)

• Established the release framework, test pyramid, and change proposal patterns
• These patterns became the basis for the SpecFirst skill methodology

Dogfooding: Parallel Development

• PR feat(specfirst): Add SpecFirst skill for spec-driven development #189 (SpecFirst) codifies the methodology using OpenSpec
• PR feat(jira): Add Jira skill with Filters API, Quick Filters, and Ordering #190 (Jira Skill) was built simultaneously as acceptance testing for the methodology
• Both skills developed in parallel with strict guardrails — completed in ~4 hours

The Pattern:

PR #183 (Context)     →  Discovered patterns (release framework, test layers)
       ↓
PR #189 (SpecFirst)   →  Codified as methodology (OpenSpec specs, workflows)
       ↓
PR #190 (Jira)        →  Validated by building real skill in parallel

This validates the core principle: "If you can't specify it, you can't test it. If you can't test it, you can't trust it."

The Jira skill serves as living proof that the SpecFirst methodology works — it was built using the methodology while the methodology itself was being formalized.

[mellanon]

The Full Workflow: Context Skill → Research → Development

What made this possible was using PR #183 (Context Skill) as the knowledge foundation:

1. Research Phase

• Ingested PAI principles from Daniel Miessler's README into Obsidian vault
• Researched spec-first development approaches (OpenSpec, spec-kit, Kiro, Martin Fowler's analysis)
• Captured research as markdown files → ingested into vault

2. Context Loading

• Used Context Skill to load research + PAI principles into Claude's context
• This gave Claude Code the full picture: PAI philosophy + SDD methodology options

3. Development

• Built SpecFirst (PR feat(specfirst): Add SpecFirst skill for spec-driven development #189) and Jira (PR feat(jira): Add Jira skill with Filters API, Quick Filters, and Ordering #190) skills in parallel
• Claude Code + PAI operating system with research context loaded
• ~4 hours, strict guardrails, spec-driven approach

The Loop:

[Research] → ingest → [Vault] → obs search → [Context] → Claude Code → [Skills]
     ↑                                                              |
     └──────────────── methodology validated ←──────────────────────┘

This demonstrates PAI's Knowledge Layer principle in action: capture anything from anywhere, retrieve it with natural language, use it to build.

[mellanon]

@danielmiessler — Your 13 Principles in Action 🚀

Just wanted to share some excitement here. I've been putting PAI's principles to work this week and the results have been incredible.

The journey:

• Burned some midnight oil building the Context Skill (PR feat(context): Add Context Skill for knowledge capture and retrieval #183) — the knowledge layer, release framework, test patterns
• Used it to ingest research, specs, and architectural principles (including yours) into my vault with proper tagging
• During SpecFirst + Jira development, loaded relevant context from the vault on demand
• Culmination: Built two complete skills in parallel in ~4 hours using those building blocks

What emerged:

	PR #189 - SpecFirst	PR #190 - Jira
What	The methodology	Built using the methodology
Files	24 files (+4,965)	20 files (+5,724)
URL	PR #189	PR #190

Here's how the 13 founding principles showed up:

Principle	In Practice
1. Clear Thinking + Prompting	Researched SDD approaches, ingested into vault, loaded context before building
2. Scaffolding > Model	PAI itself is scaffolding. Then layered: OpenSpec + release management + multi-branch propagation + test framework = robust AI-powered dev process. (Learned the hard way — almost leaked private config. Guardrails matter.)
3. Deterministic	APIs wrapped in CLI commands (deterministic) → Skills compose into workflows → NLP routes to deterministic outcomes. Natural language in, predictable execution out.
4. Code Before Prompts	CLI tools (ingest, obs, jira) solve problems; skills orchestrate them
5. Spec / Test / Evals First	PR #189 is this principle — codified as a reusable skill
6. UNIX Philosophy	Small CLI tools + APIs underneath → Skills compose them → CORE routes between skills. Pipes all the way down.
7. ENG / SRE Principles	Release framework, change proposals, validation gates before merge
8. CLI as Interface	Every operation scriptable: ingest, obs search, jira
9. Goal → Code → CLI → Prompts → Agents	Followed this pipeline exactly in development
10. Meta / Self Update	SpecFirst improves how we build skills — the system improving itself
11. Custom Skill Management	Three new skills following the container pattern
12. Custom History System	(Not directly used here — but could trace back how this was built)

Food for thought: What I did use heavily was a Knowledge Layer — ingesting external research, specs, and principles into the vault, then loading them as context during development. That felt like a distinct capability from History (which captures your work). Maybe there's a 14th principle here? 🤔

The investment in Context Skill paid off — once the building blocks were solid, the 4-hour sprint to build SpecFirst + Jira was the payoff.

Well done on laying the foundation — this is becoming genuinely powerful. Excited to see where it goes!

[danielmiessler]

Hey, Thank you so much for the positive support here. Really appreciate you, and I'm glad it's working great for you! And we're only just getting started.

…

On Thu, Dec 11, 2025 at 10:39 PM, mellanon < ***@***.*** > wrote: @ danielmiessler ( https://github.com/danielmiessler ) — Your 13 Principles in Action 🚀 --------------------------------------------------------------------------------------- Just wanted to share some excitement here. I've been putting PAI's principles to work this week and the results have been incredible. *The journey:* * Burned some midnight oil building the Context Skill (PR #183 ( #183 ) ) — the knowledge layer, release framework, test patterns * Used it to ingest research, specs, and architectural principles (including yours) into my vault with proper tagging * During SpecFirst + Jira development, loaded relevant context from the vault on demand * *Culmination:* Built two complete skills in parallel in ~4 hours using those building blocks *What emerged:* PR #189 ( #189 ) - SpecFirst PR #190 ( #190 ) - Jira *What* The methodology Built using the methodology *Files* 24 files (+4,965) 20 files (+5,724) *URL* PR #189 ( #189 ) PR #190 ( #190 ) *Here's how the 13 founding principles showed up:* Principle In Practice *1. Clear Thinking + Prompting* Researched SDD approaches, ingested into vault, loaded context before building *2. Scaffolding > Model* PAI itself is scaffolding. Then layered: OpenSpec + release management + multi-branch propagation + test framework = robust AI-powered dev process. (Learned the hard way — almost leaked private config. Guardrails matter.) *3. Deterministic* APIs wrapped in CLI commands (deterministic) → Skills compose into workflows → NLP routes to deterministic outcomes. Natural language in, predictable execution out. *4. Code Before Prompts* CLI tools ( ingest , obs , jira ) solve problems; skills orchestrate them *5. Spec / Test / Evals First* PR #189 ( #189 ) is this principle — codified as a reusable skill *6. UNIX Philosophy* Small CLI tools + APIs underneath → Skills compose them → CORE routes between skills. Pipes all the way down. *7. ENG / SRE Principles* Release framework, change proposals, validation gates before merge *8. CLI as Interface* Every operation scriptable: ingest , obs search , jira *9. Goal → Code → CLI → Prompts → Agents* Followed this pipeline exactly in development *10. Meta / Self Update* SpecFirst improves how we build skills — the system improving itself *11. Custom Skill Management* Three new skills following the container pattern *12. Custom History System* (Not directly used here — but could trace back how this was built) *Food for thought:* What I did use heavily was a *Knowledge Layer* — ingesting external research, specs, and principles into the vault, then loading them as context during development. That felt like a distinct capability from History (which captures your work). Maybe there's a 14th principle here? 🤔 The investment in Context Skill paid off — once the building blocks were solid, the 4-hour sprint to build SpecFirst + Jira was the payoff. Well done on laying the foundation — this is becoming genuinely powerful. Excited to see where it goes! — Reply to this email directly, view it on GitHub ( #187 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAAMLXTGGSGAFTEYRCK2H734BJPI5AVCNFSM6AAAAACOZTYYZ2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMRTG4ZDKMY ). You are receiving this because you were mentioned. Message ID: <danielmiessler/Personal_AI_Infrastructure/repo-discussions/187/comments/15237253 @ github. com>

[danielmiessler]

Please refer to this major change coming to the repo that should address your issues. If it doesn't please resubmit. Thank you for the work!

#222

[TabishB]

@mellanon Just saw this, I'd say OpenSpec is spec-anchored rather than spec-first. We store specs and use them to guide creation of new specs and changes to the system.