Skip to content

test(config): mutation hardening to break 95 and release readiness #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
msalvatti merged 5 commits into from
Jul 16, 2026
Merged

test(config): mutation hardening to break 95 and release readiness #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2ed7d38
test(config): add mutation plan, baseline, and public-gated mutation …
msalvatti Jul 16, 2026
d2a8db8
test(config): harden suites against surviving mutants (7.2)
msalvatti Jul 16, 2026
1c74682
docs(config): record mutation results and equivalents (7.3)
msalvatti Jul 16, 2026
9cf0859
chore(config): gate release on public repo and document go-live (7.4)
msalvatti Jul 16, 2026
f17eae8
docs(config): close phase 7 dashboards (7.5)
msalvatti Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/mutation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Mutation

# Mutation testing is the deep release gate: it verifies the suite actually
# detects behavioral regressions, not just that lines execute. A full Stryker
# run takes several minutes and is memory-heavy, so it is deliberately kept out
# of the per-PR CI (which already enforces 100% line and branch coverage) and
# runs here instead: on demand from the Actions tab and on a weekly cron so the
# score stays fresh even when no code lands.
#
# The job is gated on repository visibility. While the repository is private it
# never runs; once the maintainer flips the repository public it activates
# automatically, matching the same gating used by the CodeQL and Scorecard
# workflows so every heavier gate turns on together at go-live.
on:
workflow_dispatch:
schedule:
# Mondays 05:00 UTC, between the Scorecard (04:00) and CodeQL (06:00) crons.
- cron: '0 5 * * 1'

concurrency:
group: mutation-${{ github.ref }}
cancel-in-progress: true

# Least privilege: mutation testing only reads the repository.
permissions:
contents: read

jobs:
mutation:
name: Stryker mutation run (break 95)
runs-on: ubuntu-latest
timeout-minutes: 30

# Activates only once the repository is public, consistent with the CodeQL
# and Scorecard workflows. Kept wired while private so it starts running the
# moment the repository visibility changes, with no further edit required.
if: ${{ github.event.repository.private == false }}

steps:
- name: Checkout
uses: actions/checkout@v6

- name: Setup pnpm
uses: pnpm/action-setup@v6
with:
version: 10.8.1
run_install: false

- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 24.x
cache: pnpm

- name: Install dependencies
run: pnpm install --frozen-lockfile

- name: Run mutation tests
# Exits non-zero when the score drops below the configured break
# threshold (95), turning a weakened suite into a red gate.
run: pnpm mutation
9 changes: 9 additions & 0 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,15 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 20

# Provenance (`--provenance`) requires a public repository: npm rejects a
# provenance publish sourced from a private repo, and the transparency-log
# attestation is only meaningful when the source is publicly verifiable. The
# job is therefore gated on visibility, consistent with the CodeQL,
# Scorecard, and mutation workflows. While private, a pushed tag cannot
# attempt a publish; once the repository is public the release activates
# automatically with no further edit.
if: ${{ github.event.repository.private == false }}

# The `npm-publish` environment gates the run behind a manual approval
# (configured in repo Settings, Environments). Even an accidentally
# pushed tag cannot publish without a reviewer clicking Approve.
Expand Down
25 changes: 25 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,31 @@ standard is Jest.
registration. Precedence between layers (defaults, env, secrets) is the
caller's composition (`{ ...a, ...b }`), kept explicit on purpose.

## Releasing

Publishing is driven entirely from CI through the `release.yml` workflow, which
uses npm OIDC trusted publishing (no long-lived npm token) and attaches build
provenance. Provenance and the security workflows (CodeQL, Scorecard, dependency
review) require a public repository, so the publish job is gated on repository
visibility and stays inactive while the repository is private.

Go-live is a deliberate, manual sequence performed by a maintainer:

1. **Verify the gates locally.** `pnpm mutation` (score at or above the break
threshold of 95), `pnpm prepublishOnly`, `pnpm build && pnpm test:e2e`, and
`pnpm publish --dry-run` (confirm the tarball packs only `dist/`, `LICENSE`,
`README.md`, and `CHANGELOG.md`).
2. **Confirm the version.** `package.json` `version` and the top dated
`CHANGELOG.md` heading must both read the version being released.
3. **Make the repository public.** This activates the provenance publish job and
the CodeQL, Scorecard, and dependency-review gates.
4. **Tag from `main`.** Push an annotated `vX.Y.Z` tag matching the manifest
version. The tag push triggers `release.yml`, which verifies the tag against
`package.json`, re-runs the release gates, and publishes with provenance.
5. **Post-publish smoke.** In a scratch directory outside the repository, install
the freshly published version alongside the NestJS 11 peers and boot a minimal
fixture to confirm the shipped artifact resolves and starts.

## License

[MIT](./LICENSE)
9 changes: 7 additions & 2 deletions docs/development_plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,14 @@

## 1. Progress Dashboard

> **Overall progress: 7 / 8 phases (88%)**
> **Overall progress: 8 / 8 phases (100%)**
> **Active phase:** none
> **Blocked phases:** none
>
> Note: the live publish of v0.1.0 is a deferred manual go-live (make the
> repository public, then push the `v0.1.0` tag) performed by the maintainer per
> maintainer direction; the phase delivers the mutation gate, the public-gated
> release wiring, and a verified publish dry run.

### Phase Table

Expand All @@ -38,7 +43,7 @@
| P4 | typed-accessor | ✅ | 100% | S | 2026-07-16 |
| P5 | testing-subpath | ✅ | 100% | M | 2026-07-16 |
| P6 | integration-docs-dogfood | ✅ | 100% | M | 2026-07-16 |
| P7 | mutation-hardening-release | 📋 | 0% | L | 2026-07-06 |
| P7 | mutation-hardening-release | ✅ | 100% | L | 2026-07-16 |

---

Expand Down
165 changes: 165 additions & 0 deletions docs/mutation_testing_plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
# Mutation Testing Plan: @bymax-one/nest-config

> **Purpose:** the release-gate plan for Stryker mutation testing. It records the
> thresholds, the mutated surface, the known-risk areas, the session strategy,
> and the baseline survivor catalog that drove the hardening pass.
> **Final results:** [`mutation_testing_results.md`](./mutation_testing_results.md)

---

## Thresholds

`stryker.config.json`:

```json
"thresholds": { "high": 99, "low": 95, "break": 95 }
```

| Threshold | Meaning |
| ----------- | -------------------------------------------------------------- |
| `break: 95` | `pnpm mutation` exits non-zero when the score drops below 95 % |
| `low: 95` | Score between low and high renders yellow in the HTML report |
| `high: 99` | Aspirational target; a score at or above 99 % renders green |

Mutation testing is a **release gate, not a per-commit gate**: a full run is
slow and memory-heavy, so it runs in concentrated sessions and in the dedicated
`mutation.yml` workflow (weekly cron plus manual dispatch), never inside
`prepublishOnly` or the per-PR `ci.yml`. Per-PR CI already enforces 100 % line,
branch, function, and statement coverage.

---

## Setup

Everything is in place; no install or config steps are needed.

| File | Purpose |
| ------------------------------------- | --------------------------------------------------------- |
| `stryker.config.json` | Thresholds, mutate globs, concurrency cap, reporters |
| `jest.stryker.config.ts` | Jest config used by Stryker (`perTest` coverage analysis) |
| `@stryker-mutator/core` | Core, in `devDependencies` |
| `@stryker-mutator/jest-runner` | Jest test-runner plugin |
| `@stryker-mutator/typescript-checker` | Type-checker plugin that discards type-invalid mutants |

### Running

```bash
# Full run. Writes reports/mutation/mutation.html and mutation.json
pnpm mutation

# Faster re-run using the cached results from the last full run
pnpm mutation:incremental

# Validate the config without running any mutants
pnpm mutation:dry-run
```

Concurrency is capped at 2 in `stryker.config.json` so a run stays within bounded
resources and never has to compete with a parallel test workload.

---

## Mutated surface

`stryker.config.json` mutates `src/**/*.ts` and excludes specs, test folders,
type declarations, and public barrels (`index.ts`). That leaves 16 files:

- `config.module-definition.ts`, `config.module.ts`, `config.options.ts`,
`config.providers.ts`, `config.service.ts`, `config.tokens.ts`
- `define-env.ts`, `deep-freeze.ts`, `env-validator.ts`, `errors.ts`,
`report-formatter.ts`, `source-mapping.ts`, `types.ts`
- `testing/config-testing.module.ts`, `testing/create-test-config.ts`,
`testing/placeholder-synthesizer.ts`

`disableTypeChecks` and the TypeScript checker discard mutants that cannot type
check (the large "errors" bucket in the report), so the score reflects only
mutants that produce runnable, type-valid programs.

---

## Known-risk areas

- **Validator branch density (`env-validator.ts`).** The single-pass validator
concentrates the branching: per-namespace candidate framing, missing-versus-
invalid classification, the deterministic first-issue collapse, the
longest-prefix unknown-key attribution, and the stable unknown-key sort. These
boundaries need behavioral assertions on the observable issue list, not on
intermediate structures.
- **Formatter string and boundary mutants (`report-formatter.ts`).** The report
is part of the public contract, so column-width and pluralization boundaries
are pinned by asserting the rendered lines.
- **Deep-freeze guard (`deep-freeze.ts`).** The `isFreezable` guard is followed
by an `Object.isFrozen` short-circuit; several guard mutants are equivalent
because the short-circuit reaches the same early return (see the catalog).
- **Placeholder synthesizer boundaries (`testing/placeholder-synthesizer.ts`).**
The synthesizer picks constraint-boundary values (canonical-versus-resized
strings, exclusive integer edges, exact lengths). Inclusive bounds and the
deterministic canonical placeholders are pinned by asserting the exact
synthesized value, which is part of the `./testing` contract.

---

## Session strategy

1. **Baseline.** Full run, score and survivor catalog recorded here.
2. **Hardening pass.** For each survivor, add or sharpen one behavioral assertion
that kills it, or classify it as a candidate equivalent with a technical
reason.
3. **Equivalents documentation.** Every genuine equivalent is documented in
`mutation_testing_results.md`; narrow, justified inline disables only where a
mutant is a proven equivalent.
4. **Final gated run.** `pnpm mutation` with `break: 95` enforced must exit green.

---

## Baseline run

- **Date:** 2026-07-16
- **Command:** `pnpm mutation`
- **Score:** 89.11 % (229 killed, 28 survived, 0 timeout; 167 type-invalid
mutants discarded by the checker and excluded from the score).
- **Result:** below the `break: 95` gate; drove the hardening pass below.

### Survivor catalog

Classification: **kill** means a behavioral assertion was added in the hardening
pass; **equivalent** means no observable behavioral difference exists within the
supported two-level convention (documented in `mutation_testing_results.md`).

| File | Line | Mutator | Classification | Note |
| ------------------------------------ | ---- | ------------------------------------------------ | -------------- | -------------------------------------------------------------------- |
| `env-validator.ts` | 91 | MethodExpression | kill | Dropping the per-namespace grouping breaks the parse candidate |
| `env-validator.ts` | 92 | ConditionalExpression | kill | Filter-always-true places a leaf under the wrong namespace |
| `env-validator.ts` | 186 | ConditionalExpression | kill | First-issue collapse: pin the minimum-length message |
| `env-validator.ts` | 218 | EqualityOperator | equivalent | Equal-length distinct prefixes cannot both match one key |
| `env-validator.ts` | 261 | ConditionalExpression (x2) | equivalent | Half-mutated comparator still sorts ascending for any realistic size |
| `env-validator.ts` | 261 | EqualityOperator (x2) | equivalent | Compared variables are always distinct, so `>`/`>=` agree |
| `testing/placeholder-synthesizer.ts` | 121 | ConditionalExpression | kill | Exact length must win over a redundant minimum |
| `testing/placeholder-synthesizer.ts` | 182 | EqualityOperator | kill | URL min bound is inclusive at the canonical length |
| `testing/placeholder-synthesizer.ts` | 183 | EqualityOperator | kill | URL max bound is inclusive at the canonical length |
| `testing/placeholder-synthesizer.ts` | 202 | ConditionalExpression | kill | Email canonical is used when it fits the bounds |
| `testing/placeholder-synthesizer.ts` | 203 | EqualityOperator | kill | Email min bound is inclusive at the canonical length |
| `testing/placeholder-synthesizer.ts` | 204 | EqualityOperator | kill | Email max bound is inclusive at the canonical length |
| `testing/placeholder-synthesizer.ts` | 205 | BlockStatement | kill | Email canonical return must not be skipped |
| `testing/placeholder-synthesizer.ts` | 220 | EqualityOperator | equivalent | `target > max` and `target >= max` yield the same clamped value |
| `testing/placeholder-synthesizer.ts` | 258 | ArithmeticOperator | kill | Exclusive integer upper edge: pin the lone in-range value |
| `testing/placeholder-synthesizer.ts` | 339 | StringLiteral | kill | Boolean leaf must resolve to the `true` token, not the filler |
| `deep-freeze.ts` | 17 | ConditionalExpression (x2), LogicalOperator (x2) | equivalent | `Object.isFrozen` short-circuit reaches the same early return |
| `errors.ts` | 117 | BooleanLiteral | kill | `name` must be non-configurable (delete throws) |
| `errors.ts` | 125 | BooleanLiteral | kill | `code` must be non-configurable (delete throws) |
| `errors.ts` | 126 | BooleanLiteral | kill | `issues` must be non-configurable (delete throws) |
| `source-mapping.ts` | 45 | StringLiteral | kill | Acronym-boundary replacement must not blank the match |
| `source-mapping.ts` | 45 | Regex (`[^a-z]`) | kill | Acronym boundary uses the following lowercase word start |
| `source-mapping.ts` | 45 | Regex (`[A-Z]+`) | equivalent | Underscore position is fixed by the group-2 boundary |

Total: 28 survivors. 17 targeted for killing behavioral assertions, 11 classified
as candidate equivalents.

### Hardening result

- **Date:** 2026-07-16
- **Score after the hardening pass:** 95.72 % (246 killed, 11 survived), exit 0.
- Every one of the 17 kill targets was closed by a behavioral assertion; the 11
remaining survivors are exactly the classified equivalents above. Full per-file
scores and equivalent reasons are in
[`mutation_testing_results.md`](./mutation_testing_results.md).
Loading
Loading