feat(api): export user domain data as a tar.gz archive #322

Merged

james merged 2 commits from feat/data-export into main 2026-06-28 19:01:47 +00:00

Conversation 1 Commits 2 Files changed 12 +1455 -12

james commented 2026-06-28 00:32:44 +00:00

Owner

Copy link

What

Adds GET /api/export, the read side of #286 — the first slice of import/export. The authenticated user downloads all of their domain data as a portable carol-export-<date>.tar.gz: YAML docs at the archive root plus a media/ dir for blobs.

carol-export/
  manifest.yaml        # formatVersion: 1, exportedAt, appVersion, per-entity counts
  profile.yaml         # profile (+ contacts), picture -> media/profile-avatar.webp
  skills.yaml          # skill_sections (+ skills)
  education.yaml       # educations
  experience.yaml      # jobs -> positions -> contributions
  projects.yaml        # projects
  people.yaml          # people (+ contacts, notes, relatives, met_through, organizations)
  organizations.yaml   # organizations (+ links, key_people)
  notes.yaml           # notes
  media/profile-avatar.webp

How

• apps/api/lib/export/serialize.ts — buildExportData(db, userId): gathers every domain entity through the existing repositories, nests children under their parents (parent UUID is the nesting key), and omits user_id from every emitted row (it is re-scoped to the importer on import, and omitting it keeps the archive instance-portable). Pure data — no YAML/tar here. Resolves the profile-picture blob into media/ (skips silently if absent).
• apps/api/lib/export/archive.ts — buildExportArchive(data): YAML-serializes each doc (yaml), packs with tar-stream, gzips (zlib), and builds the manifest (formatVersion: 1, exportedAt, appVersion from apps/api/package.json, per-entity counts).
• apps/api/app/api/export/route.ts — authed GET (getAuthIdentity → unauthorized() on null), returns application/gzip with Content-Disposition: attachment; filename="carol-export-<YYYY-MM-DD>.tar.gz". Mirrors the binary response shape of GET /api/profile/picture. Not added to the public-routes allowlist — auth required.
• OpenAPI — registered under a new data tag; openapi.json regenerated; openapi:check + openapi:coverage pass.
• ADR-0028 — documents the gzipped-tar-of-a-directory format, domain-only scope (auth/identity/settings excluded and why), the three (future) import modes, dry-run-preview semantics, user_id re-scoping, and the transaction decision (real db.transaction() + a file-backed SQLite test for the future import apply path, superseding the libsql in-memory caveat at apps/api/db/repositories/skill-sections.ts).
• Tests — both-engine serializer test (tests/db/export-serialize.test.ts) and an API-level route test (tests/api/export.test.ts) that gunzips + untars the response and verifies manifest counts, YAML contents, media bytes, and a 401 unauth case.

Corrections to #286's sketch

• projects.yaml + contributions is a ticket error. contributions are children of positions (contributions.position_id → positions → jobs), so they nest under experience.yaml. projects.yaml holds projects only (projects have no child rows).
• Contracts. Contracts share the jobs table (is_contract), but JobsRepository.listByUserId filters them out. They are domain data and their positions would otherwise be orphaned, so the serializer reads every job for the user directly and keeps the is_contract flag on each emitted row.
• Media naming. Profiles have no surrogate id (the row is keyed by user_id), so the single avatar is stored at the fixed media/profile-avatar.webp rather than media/<uuid>-<name>. Documented in ADR-0028.

Out of scope (filed as follow-ups)

Import preview (#317), import apply with three modes (#318), settings UI (#319), and the cross-cutting i18n + remaining import tests (#320). This PR also lands ADR-0028 and the export tests (tracked under #320).

Verification

• pnpm -F @carol/api test — full suite green; new export tests pass (SQLite leg locally; the Postgres leg runs in CI).
• pnpm -F @carol/api openapi:check && openapi:coverage — no drift, route covered.
• pnpm -F @carol/api build — compiles; /api/export registered.
• pnpm -F @carol/api lint — clean.

Closes #316
Refs #286

🤖 Generated with Claude Code

## What Adds `GET /api/export`, the read side of #286 — the first slice of import/export. The authenticated user downloads **all of their domain data** as a portable `carol-export-<date>.tar.gz`: YAML docs at the archive root plus a `media/` dir for blobs. ``` carol-export/ manifest.yaml # formatVersion: 1, exportedAt, appVersion, per-entity counts profile.yaml # profile (+ contacts), picture -> media/profile-avatar.webp skills.yaml # skill_sections (+ skills) education.yaml # educations experience.yaml # jobs -> positions -> contributions projects.yaml # projects people.yaml # people (+ contacts, notes, relatives, met_through, organizations) organizations.yaml # organizations (+ links, key_people) notes.yaml # notes media/profile-avatar.webp ``` ## How - **`apps/api/lib/export/serialize.ts`** — `buildExportData(db, userId)`: gathers every domain entity through the existing repositories, nests children under their parents (parent UUID is the nesting key), and **omits `user_id` from every emitted row** (it is re-scoped to the importer on import, and omitting it keeps the archive instance-portable). Pure data — no YAML/tar here. Resolves the profile-picture blob into `media/` (skips silently if absent). - **`apps/api/lib/export/archive.ts`** — `buildExportArchive(data)`: YAML-serializes each doc (`yaml`), packs with `tar-stream`, gzips (`zlib`), and builds the manifest (`formatVersion: 1`, `exportedAt`, `appVersion` from `apps/api/package.json`, per-entity counts). - **`apps/api/app/api/export/route.ts`** — authed `GET` (`getAuthIdentity` → `unauthorized()` on null), returns `application/gzip` with `Content-Disposition: attachment; filename="carol-export-<YYYY-MM-DD>.tar.gz"`. Mirrors the binary response shape of `GET /api/profile/picture`. Not added to the public-routes allowlist — auth required. - **OpenAPI** — registered under a new `data` tag; `openapi.json` regenerated; `openapi:check` + `openapi:coverage` pass. - **ADR-0028** — documents the gzipped-tar-of-a-directory format, domain-only scope (auth/identity/settings excluded and why), the three (future) import modes, dry-run-preview semantics, `user_id` re-scoping, and the transaction decision (real `db.transaction()` + a file-backed SQLite test for the future import apply path, superseding the libsql in-memory caveat at `apps/api/db/repositories/skill-sections.ts`). - **Tests** — both-engine serializer test (`tests/db/export-serialize.test.ts`) and an API-level route test (`tests/api/export.test.ts`) that gunzips + untars the response and verifies manifest counts, YAML contents, media bytes, and a 401 unauth case. ## Corrections to #286's sketch - **`projects.yaml` + contributions is a ticket error.** `contributions` are children of `positions` (`contributions.position_id → positions → jobs`), so they nest under `experience.yaml`. `projects.yaml` holds projects only (projects have no child rows). - **Contracts.** Contracts share the `jobs` table (`is_contract`), but `JobsRepository.listByUserId` filters them out. They are domain data and their positions would otherwise be orphaned, so the serializer reads every job for the user directly and keeps the `is_contract` flag on each emitted row. - **Media naming.** Profiles have no surrogate id (the row is keyed by `user_id`), so the single avatar is stored at the fixed `media/profile-avatar.webp` rather than `media/<uuid>-<name>`. Documented in ADR-0028. ## Out of scope (filed as follow-ups) Import preview (#317), import apply with three modes (#318), settings UI (#319), and the cross-cutting i18n + remaining import tests (#320). This PR also lands ADR-0028 and the export tests (tracked under #320). ## Verification - `pnpm -F @carol/api test` — full suite green; new export tests pass (SQLite leg locally; the Postgres leg runs in CI). - `pnpm -F @carol/api openapi:check && openapi:coverage` — no drift, route covered. - `pnpm -F @carol/api build` — compiles; `/api/export` registered. - `pnpm -F @carol/api lint` — clean. Closes #316 Refs #286 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james added 1 commit 2026-06-28 00:32:44 +00:00

feat(api): export user domain data as a tar.gz archive ... 1c368ee0df

Add GET /api/export, the read side of #286. The authenticated user
downloads all of their domain data as a portable
carol-export-<date>.tar.gz: YAML docs at the archive root plus a
media/ dir for blobs.

- lib/export/serialize.ts gathers every domain entity via the
  repositories, nests children under parents, and omits user_id from
  every emitted row (re-scoped on import; keeps the archive portable).
  Reads all jobs (incl. contracts) directly so contract positions
  aren't orphaned.
- lib/export/archive.ts YAML-serializes each doc, packs with
  tar-stream, gzips, and builds the manifest (formatVersion 1,
  exportedAt, appVersion, per-entity counts).
- OpenAPI registration under a new `data` tag; regenerated openapi.json.
- ADR-0028 documents the archive format, domain-only scope, the three
  (future) import modes, dry-run preview, user_id re-scoping, and the
  transaction decision for the import apply path.
- Both-engine serializer test + API-level route test (gunzip+untar,
  manifest counts, YAML contents, media bytes, 401 unauth).

Closes #316
Refs #286

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

forgejo-actions commented 2026-06-28 00:36:59 +00:00

Copy link

📊 Test coverage

Patch coverage: no testable lines changed.

Overall (app/, lib/, db/, excluding UI per ADR-0019):

Metric	Value	Soft target
Lines	82.3% ✅	≥ 50%
Branches	73.4% ⚠️	≥ 75%
Functions	91.6%	informational

Soft thresholds per ADR-0019. Coverage is informational and does not block merge.

<!-- coverage-comment --> ## 📊 Test coverage **Patch coverage:** no testable lines changed. **Overall** (`app/`, `lib/`, `db/`, excluding UI per ADR-0019): | Metric | Value | Soft target | |---|---|---| | Lines | 82.3% ✅ | ≥ 50% | | Branches | 73.4% ⚠️ | ≥ 75% | | Functions | 91.6% | informational | Soft thresholds per [ADR-0019](docs/adr/0019-coverage-soft-targets.md). Coverage is informational and does not block merge.

james added 1 commit 2026-06-28 18:04:11 +00:00

fix(api): typecheck export test + regenerate api-client types ... a04e59bd63

The export-serialize test indexed arrays under noUncheckedIndexedAccess
(TS2532) and the api-client generated types were not regenerated after
adding GET /api/export, failing the Typecheck and OpenAPI CI jobs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

james force-pushed feat/data-export from a04e59bd63 to 966ca62988 2026-06-28 18:48:04 +00:00 Compare

james merged commit d13f7ca265 into main 2026-06-28 19:01:47 +00:00

james deleted branch feat/data-export 2026-06-28 19:01:48 +00:00

james referenced this pull request from a commit 2026-06-28 19:01:48 +00:00

Merge pull request 'feat(api): export user domain data as a tar.gz archive' (#322) from feat/data-export into main

james referenced this pull request 2026-06-30 12:24:39 +00:00

chore(tooling): pin/reconcile Prettier so pnpm format doesn't churn unrelated files #393

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area:auth

Authentication and authorization

  

area:ci

Forgejo Actions, build, test, security, release

  

area:db

Database abstraction, migrations, persistence

  

area:infra

Containerization, deployment, foundational tooling

  

area:native

React Native app (Android + Linux Flatpak)

  

area:pwa

PWA / web frontend

  

area:service

Backend service, API routes, persistence

  

epic

High-level tracker; links several feature tickets

  

feature

A user-facing feature

  

foundation

Foundational work that other tickets build on

No labels

area:auth

area:ci

area:db

area:infra

area:native

area:pwa

area:service

epic

feature

foundation

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

james/carol!322

No description provided.