com.gregkilmer.www/docs/content-service.md

# Content service

The content service sits between the vault and everything that shows or
changes content. It is the only thing that reads or writes the Site folder.

```
Obsidian ──edits──▶ Site folder ──watch──▶ rules ──▶ accepted store ──▶ site, admin page
                        ▲                                  ▲
                        └──── accepted API changes ────────┤
                                                    write API (HTTP)
                                                      ▲        ▲
                                              MCP server   admin page, later editors
```

- **The vault is the source of truth.** Greg edits in Obsidian; the service
  watches the folder and takes each change in.
- **Every change runs the same rules**, whether it came from Obsidian, an
  agent or the admin page. A change that passes is accepted. A change that
  fails is held for review, and the site keeps showing the last accepted
  version, so a bad edit never reaches a page.
- **Changes through the API are written back into the vault** once accepted,
  so Obsidian always shows what the site shows. The service marks its own
  writes as seen before writing, so the watcher does not take them in twice.
- **Deleting or moving a file unpublishes the note.** Its history stays, and
  any version can be rolled back.

## The rules

Pure functions in [`services/content/src/rules.ts`](../services/content/src/rules.ts),
run in this order. The rules live in code, not in the vault, so no agent can
loosen them.

| Code | Holds a change when |
| --- | --- |
| `invalid` | The note fails its type's schema. |
| `misplaced` | A settings note (`_site`, `Now/_kinds`, `Store/_settings`) is not at its fixed path. |
| `duplicate-name`, `duplicate-slug` | Another live note already has the file name, or the same type already uses the address. |
| `text-loss` | A body of 400 or more characters loses more than half in one edit. |
| `secret` | New text looks like a key, token or password. |
| `contact` | An agent adds an email address or phone number. |
| `agent-edit` | An agent changes a note it did not write, unless the note has `agentEdits: true`. |
| `agent-publish` | An agent publishes anything but a status or a Library item. |
| `batch` | An agent makes more than 10 changes in 10 minutes. |

A sync job writing notes (author `sync`) runs every agent rule above, the
`batch` limit counted for syncs on their own; see [Sync jobs](#sync-jobs).

Invalid input through the API is turned down at once with the problems, rather
than held, so callers fix it themselves. A change from Obsidian that fails is
held, because the file is already on disk.

## Addresses

A slug change or a rename keeps the old address working as a redirect. When a
note disappears and a note of the same type with the same text or title
appears within ten minutes, that is treated as a move: redirects to the old
path are pointed at the new one, so redirects never chain. A slug that comes
back into use belongs to its note, not to the redirect.

## The write API

`ContentService.apply` takes one change, `create` (type, fields, body) or
`update` (path, the fields to change, `null` to remove one, and optionally
the body), along with who made it. Every client uses it:

- the MCP server, as `agent`;
- the admin page and later editing tools, as `greg` (editor token) or `admin`;
- the review queue: `acceptHeld`, `rejectHeld` (optionally restoring the live
  version into the vault), and `rollback` to any earlier revision.

An update first takes in any unsaved Obsidian edit to that file, so an API
change never overwrites one. A note with a change waiting for review takes no
other API change until that one is decided. A note's type never changes.

### HTTP

Every route but `/health` needs `Authorization: Bearer <token>`. Tokens map to
roles in `CONTENT_TOKENS`.

| Route | Who | What |
| --- | --- | --- |
| `GET /health` | anyone | Note, published and held counts. |
| `GET /api/schema` | all | Every note type with its JSON Schema. |
| `GET /api/notes?type=&q=&limit=&offset=&full=` | all | List or search. Readers see published notes only. With `full=true` each note carries `synced` (see [Sync jobs](#sync-jobs)). |
| `GET /api/note?path=`, `?name=` or `?type=&slug=` | all | One note with its backlinks and `synced`. An old slug answers with `{ redirect }`. |
| `GET /api/backlinks?name=` | all | Notes linking to a note. |
| `GET /api/redirects` | all | Every redirect. |
| `GET /attachments/<file>` | all | An image from `_attachments`. |
| `GET /api/note/history?path=`, `/api/note/revision?path=&rev=` | agent, editor, admin | History and old versions. |
| `GET /api/problems` | agent, editor, admin | Broken links and other whole-folder problems. |
| `GET /api/held`, `/api/held/<id>` | agent, editor, admin | The review queue; one entry comes with the current text for comparison. |
| `GET /api/status` | agent, editor, admin | When the service started, its last full scan and last error, counts, and each sync job's latest run. |
| `GET /api/syncs`, `?job=` | agent, editor, admin | Each sync job's latest run, or one job's recent runs. |
| `POST /api/syncs` | agent, editor, admin | A sync job reports a run: `{ job, ok, message?, counts? }`. The last 50 runs per job are kept. |
| `POST /api/changes` | agent, editor, admin | One change. 422 with reasons when turned down. |
| `POST /api/held/<id>/accept`, `/reject` | editor, admin | Decide a held change. `{ "restore": true }` on reject puts the live version back into the vault. |
| `POST /api/note/rollback` | admin | `{ path, rev }` |
| `POST /api/scan` | editor, admin | Compare the whole folder now. |
| `POST /api/submissions` | intake, admin | File a saved copy of an inquiry or order. Answers `{ id, receipt }`. |
| `POST /api/submissions/<id>/delivery` | intake (with the receipt, once), admin | `{ receipt, status: sent, failed or skipped, error? }`: how the email went. |
| `GET /api/submissions?handled=`, `/api/submissions/<id>` | admin | Saved copies, unhandled first, newest first, with counts. |
| `POST /api/submissions/<id>/handled` | admin | `{ handled: true or false }` |
| `GET /api/submissions/<id>/attachment` | admin | The attached file, always as a download. |
| `/mcp` | agent, editor, admin | The MCP server. |

### MCP

Streamable HTTP at `/mcp`, stateless. Tools: `search`, `get_note`,
`get_schema`, `list_held`, `set_status`, `add_library_item`, `update_fields`
and `draft_post` (always unpublished). There is no tool to delete, rename,
move or write a raw file.

## Running it

| Variable | Default | |
| --- | --- | --- |
| `SITE_DIR` | required | The vault's Site folder. |
| `DATA_DIR` | `data` | Holds `content.db`. Back it up: history and the review queue cannot be rebuilt from the vault. |
| `PORT`, `HOST` | `4310`, `0.0.0.0` | |
| `CONTENT_TOKENS` | none | `role:token` pairs, comma separated. Roles: reader, agent, editor, admin, intake. Tokens are 24 characters or more. |
| `ALLOWED_EMAILS` | none | Addresses an agent may add without a hold. |
| `HEALTHCHECK_URL` | none | Pinged after every full scan. |
| `NOTIFY_WEBHOOK_URL` | none | Receives `{ content, event: "held", held }` when a change is held. `content` is a one-line message, so a Discord webhook URL works as is. |
| `SCAN_MINUTES` | `5` | Full scan interval, on top of the file watch. |
| `SUBMISSION_MAX_MB` | `10` | Largest attachment a form submission may carry. |
| `SYNC_JOBS` | none | Public sync jobs to run, comma separated: `archidekt`, `bgg`, `covers`. See [Sync jobs](#sync-jobs). |
| `SYNC_<JOB>_MINUTES` | per job | Overrides a job's interval, e.g. `SYNC_BGG_MINUTES=720`. |
| `SYNC_<JOB>_HEALTHCHECK_URL` | none | The job's Healthchecks ping URL, e.g. `SYNC_ARCHIDEKT_HEALTHCHECK_URL`. |
| `BGG_TOKEN` | none | BoardGameGeek application token, sent as `Authorization: Bearer`. Register an app with BGG if the XML API answers 401. |
| `TMDB_API_KEY` | none | TMDB v3 API key or v4 read access token. Without it, film and TV covers are skipped. |

```sh
SITE_DIR=/path/to/Site CONTENT_TOKENS="reader:…,agent:…,editor:…,admin:…" npm run content
```

The store is SQLite (`node:sqlite`) with full-text search, so the service has
no database server to run.

## Sync jobs

Outside sources sync on a schedule into a cache the site reads, so a source
being down never takes a page down. Code: [`services/content/src/sync/`](../services/content/src/sync/).

| Job | Every | Reads | Stores per note |
| --- | --- | --- | --- |
| `archidekt` | hour | Deck notes with `archidektId`: `GET https://archidekt.com/api/decks/<id>/` | `commanderArt` (Scryfall art crop of the commander, else the deck's featured image), `colors` (color identity, WUBRG letters), `commanders`, `deckSize` (cards, maybeboard excluded), `format`, `deckUrl` |
| `bgg` | day | Board-game notes' `bggId`: XML API2 `thing`, 20 ids a request, 5 s apart, waits out `202` | `minPlayers`, `maxPlayers`, `playTime`, `minPlayTime`, `maxPlayTime`, `cover` (box art) |
| `covers` | 15 min | Library items with no `cover`: Open Library (books), TMDB (film, TV; needs `TMDB_API_KEY`), AniList (anime, manga, manhwa, light novels, webnovels; by `anilistId` when set) | `cover` |
| `suwayomi` | hour | PersonalReader's manga and manhwa: Suwayomi's GraphQL library (category, furthest chapter read and when, AniList tracker id), matched to Library items by `anilistId`; AniList for covers | `chapterProgress` (`title`, `status`, `chapter`, `readAt`), `cover` (AniList); changes `state` (below) |
| `kavita` | day | PersonalReader's light novels and books: Kavita's API, matched by `kavitaSeriesId`, or by `anilistId` against an anilist.co link in the series' metadata | `bookProgress` (`title`, `percent`, `readAt`); changes `state` (below) |
| `bambuddy` | hour | Filament spools in stock (**unverified API**, below) | `filamentStock` on product lines with a Filament option |
| `homebox` | hour | Homebox items with the labels or locations in `HOMEBOX_LABELS` / `HOMEBOX_LOCATIONS` | `materialStock` on product lines with a Leather or Paper option |

The covers job looks every 15 minutes, so a new Library item gets a cover
soon after it appears, and sweeps slowly: at most 20 items a run, an item
with no cover found is asked again after a day, one with a cover after 30
days. Only image addresses are stored; nothing is downloaded. Video games
have no cover source.

### Homelab sources (Cosmos)

These four run on Cosmos over `homelab_network`, are **read only** against
their source, and are enabled by their own URL and login alone (not
`SYNC_JOBS`). Code: `sync/jobs/{suwayomi,kavita,bambuddy,homebox}.ts`,
settings in `sync/jobs/homelab.ts`. None of them has been run against a real
server from this repository: they are written from each project's API
documentation (Bambuddy: from nothing, see below) and tested with
hand-written sample payloads in `services/content/test/sync/fixtures`.

**PersonalReader (Suwayomi, Kavita)** is private. A job touches a Library
item only when the note carries the series' id: `anilistId` (Suwayomi's
AniList tracker record; Kavita's metadata link), or `kavitaSeriesId`. It never
matches by title. It caches only a title, a chapter or percent, when it was
read and an AniList cover: never a source site, a scan group, a thumbnail or
any address inside Suwayomi or Kavita.

| Reader says | Library `state` |
| --- | --- |
| Suwayomi category Reading | `now` (a note at `caught up` stays) |
| Plan to read | `up next` |
| Completed | `finished` |
| Dropped, or any other category | left alone |
| Kavita: every page read | `finished` when the series is Completed or Ended, else `caught up` (a note at `finished` stays) |
| Kavita: some pages read | `now` (a note at `caught up` stays) |
| Kavita: unread and on Want to Read | `up next` |

State changes go through `ctx.apply` as `sync`, so a note Greg wrote is held
for review (`agent-edit`), as is everything else in the rules above. A job
proposes each change once: it is not proposed again while it waits, nor after
Greg rejects it, until the reader says something different (the job keeps
that memo in its cache under `_memo`, which is not a note and never reaches
the site).

- **Suwayomi** logs in with its own read-only account. `SUWAYOMI_AUTH=ui_login`
  (the default with a user set) uses the GraphQL `login` mutation and sends the
  access token as `Authorization: Bearer`; `basic` sends HTTP basic auth (for
  `AUTH_MODE=basic_auth`); `none` sends nothing. Suwayomi's cookie-based
  `simple_login` mode is not supported. It then reads `categories { nodes {
  name mangas { nodes { title latestReadChapter { chapterNumber lastReadAt }
  trackRecords { nodes { trackerId remoteId title lastChapterRead } } } } } }`
  from `<SUWAYOMI_URL>/api/graphql`; the AniList record is tracker id 2. Covers,
  titles and media for suggestions come from one batched AniList query.
- **Library suggestions.** A Reading series with an AniList id and no Library
  item is listed at `GET /api/suggestions` (editor, admin, agent) and on the
  admin Overview, with one button that creates the Library item (title,
  medium from AniList, `anilistId`, `state: now`, published) through
  `POST /api/changes`, and a Dismiss (`POST /api/suggestions/<key>/dismiss`,
  editor or admin; kept in the `suggestion_dismissals` table). The list itself
  is the job's cache entry `_suggestions`; a series leaves it as soon as a
  Library item carries its AniList id.
- **Kavita** exchanges `KAVITA_API_KEY` at `POST /api/Plugin/authenticate` for
  a token, reads `GET /api/Series/<id>`, `GET /api/Series/metadata?seriesId=`,
  `POST /api/Series/all-v2` (only when a note needs an AniList match; one
  metadata request per series, a quarter second apart) and
  `POST /api/want-to-read/v2`. AniList matching applies to book, light novel
  and webnovel notes only, so a manga note is Suwayomi's.

**Store stock (Bambuddy, Homebox)** never writes notes. A job stores what is
in stock on every product line that has an option in a group it feeds
(`filamentStock` for Bambuddy's group, Filament by default;
`materialStock` for Homebox's), as `[{ group, options: [{ name, hex?, inStock }] }]`.
The site lays it under the note (`optionGroups` in
`services/site/src/lib/store.ts`, which the Store pages and the order check
both use): an option's own `inStock` and `hex` always win; an option that
leaves `inStock` out (it is optional) takes the synced value, and is out of
stock when the source does not have it; in-stock colors the note does not
list are added after the note's own. With nothing synced, options show
exactly as the note says (a missing `inStock` counts as in stock).

- **Bambuddy: unverified.** Its API is not documented anywhere this project
  has, and no request has been made to one. The endpoint is
  `BAMBUDDY_SPOOLS_PATH` (default `/api/v1/inventory/spools`, a guess), and all
  knowledge of the response is in `mapBambuddySpools` in
  `sync/jobs/bambuddy.ts`: it accepts a list (or one under `items`, `spools`,
  `data`, `results`) and reads color name, hex, material and grams left under
  common spellings. Check it against Greg's Bambuddy and fix that one function
  before relying on it. A color is in stock when some spool of it has at least
  `BAMBUDDY_MIN_GRAMS` (default 50) left, or does not say.
- **Homebox** logs in at `POST /api/v1/users/login`, looks up the label and
  location ids, and lists `GET /api/v1/items?labels=<id>` (or
  `locations=<id>`). Each `HOMEBOX_LABELS` / `HOMEBOX_LOCATIONS` entry is
  `name` or `name=Group` (`leather` feeds Leather). An item is in stock when
  its quantity is above zero; a `#rrggbb` in its description is its swatch
  color. A name that matches no label or location fails the run.

| Variable | |
| --- | --- |
| `SUWAYOMI_URL` | Enables `suwayomi`, like `http://suwayomi:4567`. |
| `SUWAYOMI_USER`, `SUWAYOMI_PASS` | Its own read-only Suwayomi account. |
| `SUWAYOMI_AUTH` | `ui_login` (default with a user), `basic`, or `none`. |
| `KAVITA_URL`, `KAVITA_API_KEY` | Enable `kavita`, like `http://kavita:5000`; the key of a Kavita user that can see the libraries. |
| `BAMBUDDY_URL` | Enables `bambuddy`. |
| `BAMBUDDY_SPOOLS_PATH` | Endpoint path (default `/api/v1/inventory/spools`, unverified). |
| `BAMBUDDY_API_KEY`, `BAMBUDDY_AUTH_HEADER` | Sent as `Authorization: Bearer <key>`, or as the raw key in the named header. |
| `BAMBUDDY_GROUP`, `BAMBUDDY_MATERIALS`, `BAMBUDDY_MIN_GRAMS` | Option group (Filament), materials to include (like `PLA,PETG`; all when empty), grams that count as in stock (50). |
| `HOMEBOX_URL`, `HOMEBOX_USER`, `HOMEBOX_PASS` | Enable `homebox`, with a read-only Homebox user. |
| `HOMEBOX_LABELS`, `HOMEBOX_LOCATIONS` | Which items count, `name` or `name=Group`, comma separated. Labels default to `leather,paper`. |

Each also takes `SYNC_<JOB>_MINUTES` and `SYNC_<JOB>_HEALTHCHECK_URL` like any
job (`SYNC_SUWAYOMI_HEALTHCHECK_URL`, and so on).

### How they run

- **Enabled only when configured.** The public sources run when named in
  `SYNC_JOBS`. A job that needs a URL or a login is enabled by those
  variables alone.
- **Scheduler.** Each job runs once shortly after start (20 seconds apart,
  so they do not all start together), then every interval. A job never runs
  twice at once; a run past its timeout (10 minutes unless the job says
  otherwise) is aborted and reported as failed.
- **Reporting.** Every run lands in `sync_runs` (`/api/syncs`, the admin
  Overview). With `SYNC_<JOB>_HEALTHCHECK_URL` set, the job pings
  `<url>/start`, then `<url>` or `<url>/fail` with the message, as the nightly
  backup does.
- **The cache.** Table `sync_cache` in `content.db`, keyed by job and note
  path, holding the values as JSON and when they were fetched. A job replaces
  a note's values only with values it fetched successfully, so a failed run
  or a failed item keeps the last good values. Values for notes that leave a
  job's scope (a deck loses its `archidektId`) are dropped. The cache can be
  rebuilt by running the jobs again.

### What the site gets

`/api/notes?full=true` and `/api/note` give every note a `synced` object:
each job's values for that note merged together (jobs do not share keys).
The site lays `synced` under the note's `data`, so **a value set in the note
always wins** (a note's own `cover` beats a synced one), and a note with
nothing synced renders exactly as before.

### Writing a job

A job is a `SyncJob` ([`sync/types.ts`](../services/content/src/sync/types.ts)),
added to `enabledJobs` in [`sync/index.ts`](../services/content/src/sync/index.ts)
behind the settings it needs (read in [`sync/settings.ts`](../services/content/src/sync/settings.ts)
from the environment, and documented above):

```ts
interface SyncJob {
  name: string;            // "bambuddy": sync_runs, the cache and SYNC_BAMBUDDY_* use it
  everyMinutes: number;
  timeoutMs?: number;      // default 10 minutes
  healthcheckUrl?: string;
  run(ctx: SyncContext): Promise<{ message?: string; counts?: Record<string, number>; ok?: boolean }>;
}
```

`run` gets a `SyncContext` and nothing else:

| | |
| --- | --- |
| `ctx.notes(type)` | Live notes of a type, read only. |
| `ctx.cached(path)`, `ctx.set(path, values)` | This job's cache for a note. Call `set` only with values fetched successfully. |
| `ctx.retain(paths)` | Drop this job's values for notes outside `paths`. |
| `ctx.apply(change)` | Propose a note change as `sync` (below). |
| `ctx.fetch(url, init)` | `fetch` with the sync User-Agent, the run's abort signal and a 30 second timeout (`timeoutMs` to change). |
| `ctx.sleep(ms)`, `ctx.signal`, `ctx.now()`, `ctx.log(msg)` | Polite pauses, cancellation, the clock, logging. |

Throw to fail the whole run; return `ok: false` when some items failed. A run
that throws changes nothing already cached.

**Read-only sources** (Bambuddy filament, Homebox materials) store what they
read with `ctx.set` on the note it belongs to and add the keys to the site's
readers; they never write notes.

**Jobs that change notes** (Suwayomi and Kavita progress for PersonalReader)
call `ctx.apply({ op: "update", path, fields })`, the same
`ContentService.apply` every client uses, with author `sync`, so the note
gets `updatedBy: sync`. A sync is held to every agent rule and can do nothing
an agent cannot:

- it cannot delete, move or rename a note, or change its type;
- a change to a note the sync did not write (Greg's, an agent's) is held for
  review unless the note has `agentEdits: true`;
- added email addresses or phone numbers, anything that looks like a secret,
  and large text loss are held;
- it may publish only statuses and Library items; anything else is held;
- more than 10 sync changes in 10 minutes are held (`batch`), so a job that
  updates many notes should change only what actually differs.

The outcome says `accepted`, `held` or `rejected`; count those in the run's
`counts`. A note with a change waiting for review takes no further change
until Greg decides it, which `apply` answers as `rejected` with `waiting`.

## Form submissions

The site's Inquiries and Store order forms are emailed to Greg and saved here
as well (`services/content/src/submissions.ts`), so a lost email never loses a
request. Each copy keeps its kind (inquiry or order), type (inquiry type or
product line), name, email, the answers in form order, the attachment if any,
when it came in, how its email went (`pending`, `sent`, `failed`, `skipped`
when SMTP is not set up) and whether Greg has handled it.

- The site files copies with an `intake` token. That token can do nothing
  else: every other route, `/mcp` included, answers 403, and it can never read
  a copy back, not even the one it just filed. It reports the email result
  once, with the receipt it got when filing.
- Only the `admin` token lists, reads, downloads and marks copies handled
  (the admin page's Inbox).
- Attachments are kept under `DATA_DIR/submissions` with a name the service
  makes up (id, random part, a known extension), up to `SUBMISSION_MAX_MB`.
  Allowed: png, jpg, jpeg, gif, webp, avif, heic, pdf, txt, csv, dek, dec. They
  are only ever served as downloads.
- The rows live in `content.db`, so the `backup.dump=sqlite` label covers
  them; back up `DATA_DIR/submissions` with the volume too.