ci(test): coverage report with soft targets as sticky PR comment (#110) #111

Merged

james merged 2 commits from 110-ci-coverage into main 2026-06-18 13:48:05 +00:00

Conversation 1 Commits 2 Files changed 11 +1325

james commented 2026-06-18 12:48:27 +00:00

Owner

Copy link

Summary

A new coverage job in .forgejo/workflows/pr.yml runs vitest with @vitest/coverage-v8 on every PR and posts a sticky comment with overall + patch coverage. It never blocks the workflow. Coverage drops change the emoji in the comment from ✅ to ⚠️ but the job exits successfully — soft signal, not gate.

The sticky-comment mechanism reuses the ADR-0016 scanner pattern with two intentional differences:

• Single marker <!-- coverage-comment --> (no per-scanner variant — only one coverage comment per PR).
• No delete-on-clean. Coverage info is informational on every PR; reviewers want to see "yes this is fine" alongside the green CI badge.

What's reported

Surface	Soft target	Notes
Overall lines	≥ 50%	Anti-regression. Current baseline on the testable surface is ~80%, so the threshold sits well below baseline.
Overall branches	≥ 75%	The meaningful "are decisions tested?" metric. Current ~84%.
Patch coverage (lines on changed source files)	≥ 80%	New code should land tested.

UI is excluded from the calculation per CLAUDE.md ("UI changes are tested in the browser"). Canonical exclusion list lives as a regex in scripts/ci/coverage-report.mjs; the vitest.config.ts exclude is a hint for local dev output. ADR-0018 documents why.

End-to-end smoke check

Ran the report script locally against the past three commits (which includes the OIDC userinfo fallback PR) to verify the patch-coverage computation:

Patch coverage: 90.9% (70/77 added lines) ✅ (soft target ≥ 80%)
| File | Patch coverage | Overall lines | Branches |
| lib/auth/oidc-providers.ts | 87.5% (28/32) | 89.4% | 88.3% |
| lib/auth/oidc-verify.ts | 97.5% (39/40) | 97.3% | 89.5% |
| app/api/auth/oauth/callback/[provider]/route.ts | 60.0% (3/5) | 87.7% | 79.7% |

The 60% on the callback route reflects added error-class branches that aren't all directly exercised in the OAuth test suite — that's the kind of real signal the soft threshold was designed to surface.

Test plan

• npm run typecheck — clean.
• npm run lint — clean (added coverage/ to eslint ignores).
• npm test — 225 passed / 38 skipped.
• actionlint .forgejo/workflows/pr.yml — clean.
• npm run test:coverage locally — emits coverage/coverage-final.json + text summary.
• node scripts/ci/coverage-report.mjs reads JSON + git-diff, writes Markdown body, prints flag=<bool>.
• Watch the first CI run on this PR — the comment should appear with flag=false since the changes are all scripts / docs / config (no testable-surface source touched).

Files

• New: scripts/ci/coverage-report.mjs, scripts/ci/post-coverage-comment.mjs, docs/adr/0018-coverage-soft-targets.md.
• Modified: package.json (+ @vitest/coverage-v8, test:coverage), vitest.config.ts (coverage block), .forgejo/workflows/pr.yml (coverage job), docs/ci.md, CLAUDE.md, docs/adr/README.md, eslint.config.mjs.

Closes #110.

🤖 Generated with Claude Code

## Summary A new `coverage` job in `.forgejo/workflows/pr.yml` runs vitest with `@vitest/coverage-v8` on every PR and posts a sticky comment with overall + patch coverage. **It never blocks the workflow.** Coverage drops change the emoji in the comment from ✅ to ⚠️ but the job exits successfully — soft signal, not gate. The sticky-comment mechanism reuses the ADR-0016 scanner pattern with two intentional differences: - **Single marker** `<!-- coverage-comment -->` (no per-scanner variant — only one coverage comment per PR). - **No delete-on-clean.** Coverage info is informational on every PR; reviewers want to see "yes this is fine" alongside the green CI badge. ## What's reported | Surface | Soft target | Notes | |---|---|---| | Overall lines | ≥ 50% | Anti-regression. Current baseline on the testable surface is ~80%, so the threshold sits well below baseline. | | Overall branches | ≥ 75% | The meaningful "are decisions tested?" metric. Current ~84%. | | Patch coverage (lines on changed source files) | ≥ 80% | New code should land tested. | UI is excluded from the calculation per CLAUDE.md ("UI changes are tested in the browser"). Canonical exclusion list lives as a regex in `scripts/ci/coverage-report.mjs`; the `vitest.config.ts` exclude is a hint for local dev output. ADR-0018 documents why. ## End-to-end smoke check Ran the report script locally against the past three commits (which includes the OIDC userinfo fallback PR) to verify the patch-coverage computation: ``` Patch coverage: 90.9% (70/77 added lines) ✅ (soft target ≥ 80%) | File | Patch coverage | Overall lines | Branches | | lib/auth/oidc-providers.ts | 87.5% (28/32) | 89.4% | 88.3% | | lib/auth/oidc-verify.ts | 97.5% (39/40) | 97.3% | 89.5% | | app/api/auth/oauth/callback/[provider]/route.ts | 60.0% (3/5) | 87.7% | 79.7% | ``` The 60% on the callback route reflects added error-class branches that aren't all directly exercised in the OAuth test suite — that's the kind of real signal the soft threshold was designed to surface. ## Test plan - [x] `npm run typecheck` — clean. - [x] `npm run lint` — clean (added `coverage/` to eslint ignores). - [x] `npm test` — 225 passed / 38 skipped. - [x] `actionlint .forgejo/workflows/pr.yml` — clean. - [x] `npm run test:coverage` locally — emits `coverage/coverage-final.json` + text summary. - [x] `node scripts/ci/coverage-report.mjs` reads JSON + git-diff, writes Markdown body, prints `flag=<bool>`. - [ ] Watch the first CI run on this PR — the comment should appear with `flag=false` since the changes are all scripts / docs / config (no testable-surface source touched). ## Files - New: `scripts/ci/coverage-report.mjs`, `scripts/ci/post-coverage-comment.mjs`, `docs/adr/0018-coverage-soft-targets.md`. - Modified: `package.json` (+ `@vitest/coverage-v8`, `test:coverage`), `vitest.config.ts` (coverage block), `.forgejo/workflows/pr.yml` (coverage job), `docs/ci.md`, `CLAUDE.md`, `docs/adr/README.md`, `eslint.config.mjs`. Closes #110. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james added 1 commit 2026-06-18 12:48:27 +00:00

ci(test): coverage report with soft targets as sticky PR comment ... e628f086aa

A new `coverage` job in pr.yml runs vitest with `@vitest/coverage-v8`
on every PR and posts a sticky comment with overall + patch coverage.
It NEVER blocks the workflow — drops are flagged with ⚠️ in the
comment, not gated.

What's reported (testable surface only — UI excluded per CLAUDE.md):

- Overall lines / branches / functions across `app/`, `lib/`, `db/`.
- Patch coverage = (covered ∩ added) / coverable_added on changed
  source files, computed by intersecting `git diff origin/<base>...HEAD`
  with the istanbul statement map.

Soft targets (informational):
- Overall lines  ≥ 50%   (current baseline ~80% on testable surface)
- Overall branch ≥ 75%   (current ~84%)
- Patch lines    ≥ 80%

The sticky-comment mechanism mirrors ADR-0016 (the security scanners)
with two intentional differences: the comment is keyed by a single
marker `<!-- coverage-comment -->` (not per-scanner), and we don't
delete on clean — reviewers want to see "yes this is fine" alongside
the green CI badge.

ADR-0018 documents the policy, the exclusion list (canonical regex
in coverage-report.mjs; vitest.config.ts mirror is a hint), and what
"valuable" coverage means in Carol (branches on testable code over
lines for lines' sake).

Closes #110.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

forgejo-actions commented 2026-06-18 12:49:45 +00:00

Copy link

📊 Test coverage

Patch coverage: no testable lines changed.

Overall (app/, lib/, db/, excluding UI per ADR-0019):

Metric	Value	Soft target
Lines	83.7% ✅	≥ 50%
Branches	82.3% ✅	≥ 75%
Functions	90.8%	informational

Soft thresholds per ADR-0019. Coverage is informational and does not block merge.

<!-- coverage-comment --> ## 📊 Test coverage **Patch coverage:** no testable lines changed. **Overall** (`app/`, `lib/`, `db/`, excluding UI per ADR-0019): | Metric | Value | Soft target | |---|---|---| | Lines | 83.7% ✅ | ≥ 50% | | Branches | 82.3% ✅ | ≥ 75% | | Functions | 90.8% | informational | Soft thresholds per [ADR-0019](docs/adr/0019-coverage-soft-targets.md). Coverage is informational and does not block merge.

james added 1 commit 2026-06-18 12:57:54 +00:00

fix(ci): use execFileSync for git diff in coverage script ... 14b0cfbf12

semgrep's carol-no-shell-exec-interpolation rule flagged
`execSync(\`git diff ${baseRef}...\`)` in coverage-report.mjs.
`baseRef` originates from a CI env var; even though the workflow
sets it to `origin/<base>.ref`, the rule's "never interpolate into
a shell command" stance is correct — the spawn-via-argv pattern
makes the safety guarantee obvious at the call site instead of
relying on careful env hygiene.

`*.ts` / `*.tsx` here are git pathspecs (git's own wildcard
matching), not shell globs — they work identically with execFile.

james force-pushed 110-ci-coverage from 14b0cfbf12 to 53cf535f3b 2026-06-18 13:02:33 +00:00 Compare

james force-pushed 110-ci-coverage from 53cf535f3b to 92e4c55602 2026-06-18 13:17:26 +00:00 Compare

james merged commit dd99a7faa3 into main 2026-06-18 13:48:05 +00:00

james deleted branch 110-ci-coverage 2026-06-18 13:48:05 +00:00

james referenced this pull request from a commit 2026-06-18 13:48:06 +00:00

Merge pull request 'ci(test): coverage report with soft targets as sticky PR comment (#110)' (#111) from 110-ci-coverage into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area:auth

Authentication and authorization

  

area:ci

Forgejo Actions, build, test, security, release

  

area:db

Database abstraction, migrations, persistence

  

area:infra

Containerization, deployment, foundational tooling

  

area:native

React Native app (Android + Linux Flatpak)

  

area:pwa

PWA / web frontend

  

area:service

Backend service, API routes, persistence

  

epic

High-level tracker; links several feature tickets

  

feature

A user-facing feature

  

foundation

Foundational work that other tickets build on

No labels

area:auth

area:ci

area:db

area:infra

area:native

area:pwa

area:service

epic

feature

foundation

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

james/carol!111

No description provided.