ci(test): coverage report with soft targets posted as sticky PR comment

james commented

2026-06-18 12:47:29 +00:00

Owner

Problem

Carol has 225 tests against a layered codebase but no coverage instrumentation. Two recurring questions can't be answered:

Where are the testing gaps? Branch coverage on the testable surface (excluding UI per CLAUDE.md) gives a useful signal that test count doesn't.
Does a new PR include tests for the new code? Patch coverage = covered/added lines on changed source files. A PR that ships behaviour without a test for it is reviewable signal.

Both want surfacing on the PR itself, not in a separate dashboard, mirroring the ADR-0016 sticky-scanner-comment pattern.

Scope

Install @vitest/coverage-v8 as a dev dep. Verify no new lavamoat allowscripts entries are needed.
Add test:coverage npm script.
Configure vitest.config.ts coverage block: v8 provider; text-summary + json-summary + json reporters; include: ["app/**", "lib/**", "db/**"]; explicit excludes for UI (page.tsx, *-client.tsx, components/, sw, manifest, etc.), pure types (db/entities, db/schema, db/index), and lib/query/** (browser-only).
Write scripts/ci/coverage-report.mjs: reads coverage/coverage-final.json (istanbul format) + git diff origin/<base>...HEAD, computes overall + patch coverage, emits a Markdown body. Strict source filter (regex) mirrors the vitest exclude list and is canonical.
Write scripts/ci/post-coverage-comment.mjs: Forgejo upsert with marker . Mirror post-pr-comment.mjs helpers; no shared lib today (YAGNI; refactor if a third sticky-comment use case appears).
New coverage job in .forgejo/workflows/pr.yml. SQLite only (engine-independent). fetch-depth: 0 for the diff. actionlint clean. Never blocks the workflow.
ADR-0018 documenting the policy: testable surface, soft thresholds (lines ≥ 50%, branches ≥ 75%, patch ≥ 80%), what "valuable" coverage means (branches over lines, decision points over imports), and why we picked v8 over istanbul, soft over hard.
docs/ci.md Coverage section under the sticky-comments section.
CLAUDE.md "Working in this repo" note (one paragraph; cross-reference ADR-0018).
Add coverage/ to .gitignore (already there) and eslint.config.mjs ignores (auto-generated HTML/JS files trigger lint).

Out of scope

Hard gates / blocking merges on coverage. Documented rejection in ADR-0018.
Codecov / Coveralls external integration.
Coverage diff visualization (line-by-line in the diff view). Forgejo doesn't support inline annotations the way some GitHub Apps do; the sticky comment is sufficient.
Coverage from npm run build or browser-driven E2E tests. CLAUDE.md keeps UI browser-tested by hand.

Acceptance

A PR that adds source code without tests sees a ⚠️ below 80% line in the sticky comment but the workflow still completes successfully.
A doc-only PR sees no testable lines changed with the overall numbers as confirmation.
An OIDC-equivalent PR (~75 added lines, well-tested) sees Patch coverage: ~90% ✅ and per-file table.
The exclusion list keeps the headline overall number honest (≥ 75% branches on testable code, not a misleading ~50% global figure that pulls UI in).

## Problem Carol has 225 tests against a layered codebase but no coverage instrumentation. Two recurring questions can't be answered: 1. **Where are the testing gaps?** Branch coverage on the testable surface (excluding UI per CLAUDE.md) gives a useful signal that test count doesn't. 2. **Does a new PR include tests for the new code?** Patch coverage = covered/added lines on changed source files. A PR that ships behaviour without a test for it is reviewable signal. Both want surfacing on the PR itself, not in a separate dashboard, mirroring the ADR-0016 sticky-scanner-comment pattern. ## Scope - [ ] Install `@vitest/coverage-v8` as a dev dep. Verify no new lavamoat allowscripts entries are needed. - [ ] Add `test:coverage` npm script. - [ ] Configure `vitest.config.ts` coverage block: `v8` provider; `text-summary` + `json-summary` + `json` reporters; `include: ["app/**", "lib/**", "db/**"]`; explicit excludes for UI (page.tsx, *-client.tsx, components/, sw, manifest, etc.), pure types (db/entities, db/schema, db/index), and `lib/query/**` (browser-only). - [ ] Write `scripts/ci/coverage-report.mjs`: reads `coverage/coverage-final.json` (istanbul format) + `git diff origin/<base>...HEAD`, computes overall + patch coverage, emits a Markdown body. Strict source filter (regex) mirrors the vitest exclude list and is canonical. - [ ] Write `scripts/ci/post-coverage-comment.mjs`: Forgejo upsert with marker ``. Mirror `post-pr-comment.mjs` helpers; no shared lib today (YAGNI; refactor if a third sticky-comment use case appears). - [ ] New `coverage` job in `.forgejo/workflows/pr.yml`. SQLite only (engine-independent). `fetch-depth: 0` for the diff. `actionlint` clean. **Never blocks the workflow.** - [ ] ADR-0018 documenting the policy: testable surface, soft thresholds (lines ≥ 50%, branches ≥ 75%, patch ≥ 80%), what "valuable" coverage means (branches over lines, decision points over imports), and why we picked v8 over istanbul, soft over hard. - [ ] `docs/ci.md` Coverage section under the sticky-comments section. - [ ] `CLAUDE.md` "Working in this repo" note (one paragraph; cross-reference ADR-0018). - [ ] Add `coverage/` to `.gitignore` (already there) and `eslint.config.mjs` ignores (auto-generated HTML/JS files trigger lint). ## Out of scope - Hard gates / blocking merges on coverage. Documented rejection in ADR-0018. - Codecov / Coveralls external integration. - Coverage diff visualization (line-by-line in the diff view). Forgejo doesn't support inline annotations the way some GitHub Apps do; the sticky comment is sufficient. - Coverage from `npm run build` or browser-driven E2E tests. CLAUDE.md keeps UI browser-tested by hand. ## Acceptance - A PR that adds source code without tests sees a `⚠️ below 80%` line in the sticky comment but the workflow still completes successfully. - A doc-only PR sees `no testable lines changed` with the overall numbers as confirmation. - An OIDC-equivalent PR (~75 added lines, well-tested) sees `Patch coverage: ~90% ✅` and per-file table. - The exclusion list keeps the headline overall number honest (≥ 75% branches on testable code, not a misleading ~50% global figure that pulls UI in).

james added the

area:ci

label

2026-06-18 12:47:34 +00:00

james referenced this issue from a commit

2026-06-18 12:48:06 +00:00

ci(test): coverage report with soft targets as sticky PR comment

james referenced this issue from a pull request that will close it,

2026-06-18 12:48:27 +00:00

ci(test): coverage report with soft targets as sticky PR comment (#110) #111

james referenced this issue from a commit

2026-06-18 13:02:33 +00:00