com.gregkilmer.www/docs/content-service.md

Content service

The content service sits between the vault and everything that shows or changes content. It is the only thing that reads or writes the Site folder.

Obsidian ──edits──▶ Site folder ──watch──▶ rules ──▶ accepted store ──▶ site, admin page
                        ▲                                  ▲
                        └──── accepted API changes ────────┤
                                                    write API (HTTP)
                                                      ▲        ▲
                                              MCP server   admin page, later editors

• The vault is the source of truth. Greg edits in Obsidian; the service watches the folder and takes each change in.
• Every change runs the same rules, whether it came from Obsidian, an agent or the admin page. A change that passes is accepted. A change that fails is held for review, and the site keeps showing the last accepted version, so a bad edit never reaches a page.
• Changes through the API are written back into the vault once accepted, so Obsidian always shows what the site shows. The service marks its own writes as seen before writing, so the watcher does not take them in twice.
• Deleting or moving a file unpublishes the note. Its history stays, and any version can be rolled back.

The rules

Pure functions in services/content/src/rules.ts, run in this order. The rules live in code, not in the vault, so no agent can loosen them.

Code	Holds a change when
invalid	The note fails its type's schema.
misplaced	A settings note (_site, Now/_kinds, Store/_settings) is not at its fixed path.
duplicate-name, duplicate-slug	Another live note already has the file name, or the same type already uses the address.
text-loss	A body of 400 or more characters loses more than half in one edit.
secret	New text looks like a key, token or password.
contact	An agent adds an email address or phone number.
agent-edit	An agent changes a note it did not write, unless the note has agentEdits: true.
agent-publish	An agent publishes anything but a status or a Library item.
batch	An agent makes more than 10 changes in 10 minutes.

A sync job writing notes (author sync) runs every agent rule above, the batch limit counted for syncs on their own; see Sync jobs.

Invalid input through the API is turned down at once with the problems, rather than held, so callers fix it themselves. A change from Obsidian that fails is held, because the file is already on disk.

Addresses

A slug change or a rename keeps the old address working as a redirect. When a note disappears and a note of the same type with the same text or title appears within ten minutes, that is treated as a move: redirects to the old path are pointed at the new one, so redirects never chain. A slug that comes back into use belongs to its note, not to the redirect.

The write API

ContentService.apply takes one change, create (type, fields, body) or update (path, the fields to change, null to remove one, and optionally the body), along with who made it. Every client uses it:

• the MCP server, as agent;
• the admin page and later editing tools, as greg (editor token) or admin;
• the review queue: acceptHeld, rejectHeld (optionally restoring the live version into the vault), and rollback to any earlier revision.

An update first takes in any unsaved Obsidian edit to that file, so an API change never overwrites one. A note with a change waiting for review takes no other API change until that one is decided. A note's type never changes.

HTTP

Every route but /health needs Authorization: Bearer <token>. Tokens map to roles in CONTENT_TOKENS.

Route	Who	What
GET /health	anyone	Note, published and held counts.
GET /api/schema	all	Every note type with its JSON Schema.
GET /api/notes?type=&q=&limit=&offset=&full=	all	List or search. Readers see published notes only. With full=true each note carries synced (see Sync jobs).
GET /api/note?path=, ?name= or ?type=&slug=	all	One note with its backlinks and synced. An old slug answers with { redirect }.
GET /api/backlinks?name=	all	Notes linking to a note.
GET /api/redirects	all	Every redirect.
GET /attachments/<file>	all	An image from _attachments.
GET /api/note/history?path=, /api/note/revision?path=&rev=	agent, editor, admin	History and old versions.
GET /api/problems	agent, editor, admin	Broken links and other whole-folder problems.
GET /api/held, /api/held/<id>	agent, editor, admin	The review queue; one entry comes with the current text for comparison.
GET /api/status	agent, editor, admin	When the service started, its last full scan and last error, counts, and each sync job's latest run.
GET /api/syncs, ?job=	agent, editor, admin	Each sync job's latest run, or one job's recent runs.
POST /api/syncs	agent, editor, admin	A sync job reports a run: { job, ok, message?, counts? }. The last 50 runs per job are kept.
POST /api/changes	agent, editor, admin	One change. 422 with reasons when turned down.
POST /api/held/<id>/accept, /reject	editor, admin	Decide a held change. { "restore": true } on reject puts the live version back into the vault.
POST /api/note/rollback	admin	{ path, rev }
POST /api/scan	editor, admin	Compare the whole folder now.
POST /api/submissions	intake, admin	File a saved copy of an inquiry or order. Answers { id, receipt }.
POST /api/submissions/<id>/delivery	intake (with the receipt, once), admin	{ receipt, status: sent, failed or skipped, error? }: how the email went.
GET /api/submissions?handled=, /api/submissions/<id>	admin	Saved copies, unhandled first, newest first, with counts.
POST /api/submissions/<id>/handled	admin	{ handled: true or false }
GET /api/submissions/<id>/attachment	admin	The attached file, always as a download.
/mcp	agent, editor, admin	The MCP server.

MCP

Streamable HTTP at /mcp, stateless. Tools: search, get_note, get_schema, list_held, set_status, add_library_item, update_fields and draft_post (always unpublished). There is no tool to delete, rename, move or write a raw file.

Running it

Variable	Default
SITE_DIR	required	The vault's Site folder.
DATA_DIR	data	Holds content.db. Back it up: history and the review queue cannot be rebuilt from the vault.
PORT, HOST	4310, 0.0.0.0
CONTENT_TOKENS	none	role:token pairs, comma separated. Roles: reader, agent, editor, admin, intake. Tokens are 24 characters or more.
ALLOWED_EMAILS	none	Addresses an agent may add without a hold.
HEALTHCHECK_URL	none	Pinged after every full scan.
NOTIFY_WEBHOOK_URL	none	Receives { event: "held", held } when a change is held.
SCAN_MINUTES	5	Full scan interval, on top of the file watch.
SUBMISSION_MAX_MB	10	Largest attachment a form submission may carry.
SYNC_JOBS	none	Public sync jobs to run, comma separated: archidekt, bgg, covers. See Sync jobs.
SYNC_<JOB>_MINUTES	per job	Overrides a job's interval, e.g. SYNC_BGG_MINUTES=720.
SYNC_<JOB>_HEALTHCHECK_URL	none	The job's Healthchecks ping URL, e.g. SYNC_ARCHIDEKT_HEALTHCHECK_URL.
BGG_TOKEN	none	BoardGameGeek application token, sent as Authorization: Bearer. Register an app with BGG if the XML API answers 401.
TMDB_API_KEY	none	TMDB v3 API key or v4 read access token. Without it, film and TV covers are skipped.

SITE_DIR=/path/to/Site CONTENT_TOKENS="reader:…,agent:…,editor:…,admin:…" npm run content

The store is SQLite (node:sqlite) with full-text search, so the service has no database server to run.

Sync jobs

Outside sources sync on a schedule into a cache the site reads, so a source being down never takes a page down. Code: services/content/src/sync/.

Job	Every	Reads	Stores per note
archidekt	hour	Deck notes with archidektId: GET https://archidekt.com/api/decks/<id>/	commanderArt (Scryfall art crop of the commander, else the deck's featured image), colors (color identity, WUBRG letters), commanders, deckSize (cards, maybeboard excluded), format, deckUrl
bgg	day	Board-game notes' bggId: XML API2 thing, 20 ids a request, 5 s apart, waits out 202	minPlayers, maxPlayers, playTime, minPlayTime, maxPlayTime, cover (box art)
covers	15 min	Library items with no cover: Open Library (books), TMDB (film, TV; needs TMDB_API_KEY), AniList (anime, manga, manhwa, light novels, webnovels; by anilistId when set)	cover
suwayomi	hour	PersonalReader's manga and manhwa: Suwayomi's GraphQL library (category, furthest chapter read and when, AniList tracker id), matched to Library items by anilistId; AniList for covers	chapterProgress (title, status, chapter, readAt), cover (AniList); changes state (below)
kavita	day	PersonalReader's light novels and books: Kavita's API, matched by kavitaSeriesId, or by anilistId against an anilist.co link in the series' metadata	bookProgress (title, percent, readAt); changes state (below)
bambuddy	hour	Filament spools in stock (unverified API, below)	filamentStock on product lines with a Filament option
homebox	hour	Homebox items with the labels or locations in HOMEBOX_LABELS / HOMEBOX_LOCATIONS	materialStock on product lines with a Leather or Paper option

The covers job looks every 15 minutes, so a new Library item gets a cover soon after it appears, and sweeps slowly: at most 20 items a run, an item with no cover found is asked again after a day, one with a cover after 30 days. Only image addresses are stored; nothing is downloaded. Video games have no cover source.

Homelab sources (Cosmos)

These four run on Cosmos over homelab_network, are read only against their source, and are enabled by their own URL and login alone (not SYNC_JOBS). Code: sync/jobs/{suwayomi,kavita,bambuddy,homebox}.ts, settings in sync/jobs/homelab.ts. None of them has been run against a real server from this repository: they are written from each project's API documentation (Bambuddy: from nothing, see below) and tested with hand-written sample payloads in services/content/test/sync/fixtures.

PersonalReader (Suwayomi, Kavita) is private. A job touches a Library item only when the note carries the series' id: anilistId (Suwayomi's AniList tracker record; Kavita's metadata link), or kavitaSeriesId. It never matches by title. It caches only a title, a chapter or percent, when it was read and an AniList cover: never a source site, a scan group, a thumbnail or any address inside Suwayomi or Kavita.

Reader says	Library state
Suwayomi category Reading	now (a note at caught up stays)
Plan to read	up next
Completed	finished
Dropped, or any other category	left alone
Kavita: every page read	finished when the series is Completed or Ended, else caught up (a note at finished stays)
Kavita: some pages read	now (a note at caught up stays)
Kavita: unread and on Want to Read	up next

State changes go through ctx.apply as sync, so a note Greg wrote is held for review (agent-edit), as is everything else in the rules above. A job proposes each change once: it is not proposed again while it waits, nor after Greg rejects it, until the reader says something different (the job keeps that memo in its cache under _memo, which is not a note and never reaches the site).

• Suwayomi logs in with its own read-only account. SUWAYOMI_AUTH=ui_login (the default with a user set) uses the GraphQL login mutation and sends the access token as Authorization: Bearer; basic sends HTTP basic auth (for AUTH_MODE=basic_auth); none sends nothing. Suwayomi's cookie-based simple_login mode is not supported. It then reads categories { nodes { name mangas { nodes { title latestReadChapter { chapterNumber lastReadAt } trackRecords { nodes { trackerId remoteId title lastChapterRead } } } } } } from <SUWAYOMI_URL>/api/graphql; the AniList record is tracker id 2. Covers, titles and media for suggestions come from one batched AniList query.
• Library suggestions. A Reading series with an AniList id and no Library item is listed at GET /api/suggestions (editor, admin, agent) and on the admin Overview, with one button that creates the Library item (title, medium from AniList, anilistId, state: now, published) through POST /api/changes, and a Dismiss (POST /api/suggestions/<key>/dismiss, editor or admin; kept in the suggestion_dismissals table). The list itself is the job's cache entry _suggestions; a series leaves it as soon as a Library item carries its AniList id.
• Kavita exchanges KAVITA_API_KEY at POST /api/Plugin/authenticate for a token, reads GET /api/Series/<id>, GET /api/Series/metadata?seriesId=, POST /api/Series/all-v2 (only when a note needs an AniList match; one metadata request per series, a quarter second apart) and POST /api/want-to-read/v2. AniList matching applies to book, light novel and webnovel notes only, so a manga note is Suwayomi's.

Store stock (Bambuddy, Homebox) never writes notes. A job stores what is in stock on every product line that has an option in a group it feeds (filamentStock for Bambuddy's group, Filament by default; materialStock for Homebox's), as [{ group, options: [{ name, hex?, inStock }] }]. The site lays it under the note (optionGroups in services/site/src/lib/store.ts, which the Store pages and the order check both use): an option's own inStock and hex always win; an option that leaves inStock out (it is optional) takes the synced value, and is out of stock when the source does not have it; in-stock colors the note does not list are added after the note's own. With nothing synced, options show exactly as the note says (a missing inStock counts as in stock).

• Bambuddy: unverified. Its API is not documented anywhere this project has, and no request has been made to one. The endpoint is BAMBUDDY_SPOOLS_PATH (default /api/v1/inventory/spools, a guess), and all knowledge of the response is in mapBambuddySpools in sync/jobs/bambuddy.ts: it accepts a list (or one under items, spools, data, results) and reads color name, hex, material and grams left under common spellings. Check it against Greg's Bambuddy and fix that one function before relying on it. A color is in stock when some spool of it has at least BAMBUDDY_MIN_GRAMS (default 50) left, or does not say.
• Homebox logs in at POST /api/v1/users/login, looks up the label and location ids, and lists GET /api/v1/items?labels=<id> (or locations=<id>). Each HOMEBOX_LABELS / HOMEBOX_LOCATIONS entry is name or name=Group (leather feeds Leather). An item is in stock when its quantity is above zero; a #rrggbb in its description is its swatch color. A name that matches no label or location fails the run.

Variable
SUWAYOMI_URL	Enables suwayomi, like http://suwayomi:4567.
SUWAYOMI_USER, SUWAYOMI_PASS	Its own read-only Suwayomi account.
SUWAYOMI_AUTH	ui_login (default with a user), basic, or none.
KAVITA_URL, KAVITA_API_KEY	Enable kavita, like http://kavita:5000; the key of a Kavita user that can see the libraries.
BAMBUDDY_URL	Enables bambuddy.
BAMBUDDY_SPOOLS_PATH	Endpoint path (default /api/v1/inventory/spools, unverified).
BAMBUDDY_API_KEY, BAMBUDDY_AUTH_HEADER	Sent as Authorization: Bearer <key>, or as the raw key in the named header.
BAMBUDDY_GROUP, BAMBUDDY_MATERIALS, BAMBUDDY_MIN_GRAMS	Option group (Filament), materials to include (like PLA,PETG; all when empty), grams that count as in stock (50).
HOMEBOX_URL, HOMEBOX_USER, HOMEBOX_PASS	Enable homebox, with a read-only Homebox user.
HOMEBOX_LABELS, HOMEBOX_LOCATIONS	Which items count, name or name=Group, comma separated. Labels default to leather,paper.

Each also takes SYNC_<JOB>_MINUTES and SYNC_<JOB>_HEALTHCHECK_URL like any job (SYNC_SUWAYOMI_HEALTHCHECK_URL, and so on).

How they run

• Enabled only when configured. The public sources run when named in SYNC_JOBS. A job that needs a URL or a login is enabled by those variables alone.
• Scheduler. Each job runs once shortly after start (20 seconds apart, so they do not all start together), then every interval. A job never runs twice at once; a run past its timeout (10 minutes unless the job says otherwise) is aborted and reported as failed.
• Reporting. Every run lands in sync_runs (/api/syncs, the admin Overview). With SYNC_<JOB>_HEALTHCHECK_URL set, the job pings <url>/start, then <url> or <url>/fail with the message, as the nightly backup does.
• The cache. Table sync_cache in content.db, keyed by job and note path, holding the values as JSON and when they were fetched. A job replaces a note's values only with values it fetched successfully, so a failed run or a failed item keeps the last good values. Values for notes that leave a job's scope (a deck loses its archidektId) are dropped. The cache can be rebuilt by running the jobs again.

What the site gets

/api/notes?full=true and /api/note give every note a synced object: each job's values for that note merged together (jobs do not share keys). The site lays synced under the note's data, so a value set in the note always wins (a note's own cover beats a synced one), and a note with nothing synced renders exactly as before.

Writing a job

A job is a SyncJob (sync/types.ts), added to enabledJobs in sync/index.ts behind the settings it needs (read in sync/settings.ts from the environment, and documented above):

interface SyncJob {
  name: string;            // "bambuddy": sync_runs, the cache and SYNC_BAMBUDDY_* use it
  everyMinutes: number;
  timeoutMs?: number;      // default 10 minutes
  healthcheckUrl?: string;
  run(ctx: SyncContext): Promise<{ message?: string; counts?: Record<string, number>; ok?: boolean }>;
}

run gets a SyncContext and nothing else:

ctx.notes(type)	Live notes of a type, read only.
ctx.cached(path), ctx.set(path, values)	This job's cache for a note. Call set only with values fetched successfully.
ctx.retain(paths)	Drop this job's values for notes outside paths.
ctx.apply(change)	Propose a note change as sync (below).
ctx.fetch(url, init)	fetch with the sync User-Agent, the run's abort signal and a 30 second timeout (timeoutMs to change).
ctx.sleep(ms), ctx.signal, ctx.now(), ctx.log(msg)	Polite pauses, cancellation, the clock, logging.

Throw to fail the whole run; return ok: false when some items failed. A run that throws changes nothing already cached.

Read-only sources (Bambuddy filament, Homebox materials) store what they read with ctx.set on the note it belongs to and add the keys to the site's readers; they never write notes.

Jobs that change notes (Suwayomi and Kavita progress for PersonalReader) call ctx.apply({ op: "update", path, fields }), the same ContentService.apply every client uses, with author sync, so the note gets updatedBy: sync. A sync is held to every agent rule and can do nothing an agent cannot:

• it cannot delete, move or rename a note, or change its type;
• a change to a note the sync did not write (Greg's, an agent's) is held for review unless the note has agentEdits: true;
• added email addresses or phone numbers, anything that looks like a secret, and large text loss are held;
• it may publish only statuses and Library items; anything else is held;
• more than 10 sync changes in 10 minutes are held (batch), so a job that updates many notes should change only what actually differs.

The outcome says accepted, held or rejected; count those in the run's counts. A note with a change waiting for review takes no further change until Greg decides it, which apply answers as rejected with waiting.

Form submissions

The site's Inquiries and Store order forms are emailed to Greg and saved here as well (services/content/src/submissions.ts), so a lost email never loses a request. Each copy keeps its kind (inquiry or order), type (inquiry type or product line), name, email, the answers in form order, the attachment if any, when it came in, how its email went (pending, sent, failed, skipped when SMTP is not set up) and whether Greg has handled it.

• The site files copies with an intake token. That token can do nothing else: every other route, /mcp included, answers 403, and it can never read a copy back, not even the one it just filed. It reports the email result once, with the receipt it got when filing.
• Only the admin token lists, reads, downloads and marks copies handled (the admin page's Inbox).
• Attachments are kept under DATA_DIR/submissions with a name the service makes up (id, random part, a known extension), up to SUBMISSION_MAX_MB. Allowed: png, jpg, jpeg, gif, webp, avif, heic, pdf, txt, csv, dek, dec. They are only ever served as downloads.
• The rows live in content.db, so the backup.dump=sqlite label covers them; back up DATA_DIR/submissions with the volume too.