Antares/com.gregkilmer.www
1
0
Fork 0
mirror of https://github.com/Patiun/com.gregkilmer.www.git synced 2026-11-07 19:58:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Phase 6: sync jobs, inquiry and order forms

Browse Source 
- Sync scheduler in the content service with a per-job cache (sync_cache),
  sync_runs reporting and Healthchecks pings. Jobs: Archidekt, BoardGameGeek,
  cover art (Open Library, TMDB, AniList), Suwayomi and Kavita reading
  progress, Bambuddy filament and Homebox material stock. A note's own value
  always wins over a synced one.
- PersonalReader: updates only Library notes with a matching anilistId,
  changes to Greg's notes are held for review; Reading series with no note
  show as suggestions on the admin Overview.
- /api/inquiries and /api/orders: Turnstile, rate limits, a saved copy in the
  content service (intake token) plus email over SMTP. Orders are re-checked
  against the live notes: accepted only when the category's ordersOpen is
  true and the line has capacity.
- Admin Inbox for saved submissions. Options' inStock is optional so synced
  stock can fill it. All settings from process.env; .env.example per service.
- Sources could not be reached from the build environment; parsers are tested
  against fixtures labelled as not live.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aSbo2D1qP8cMFQCykf7kx
This commit is contained in:
Claude
2026-10-11 23:50:06 +00:00
parent 60dfeecd7f
commit e3b0cf60c5
76 changed files with 6050 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/content-service.md
+244 -3
View File
@@ -42,6 +42,9 @@ loosen them.
| `agent-publish` | An agent publishes anything but a status or a Library item. |
| `batch` | An agent makes more than 10 changes in 10 minutes. |
A sync job writing notes (author `sync`) runs every agent rule above, the
`batch` limit counted for syncs on their own; see [Sync jobs](#sync-jobs).
Invalid input through the API is turned down at once with the problems, rather
than held, so callers fix it themselves. A change from Obsidian that fails is
held, because the file is already on disk.
@@ -78,8 +81,8 @@ roles in `CONTENT_TOKENS`.
| --- | --- | --- |
| `GET /health` | anyone | Note, published and held counts. |
| `GET /api/schema` | all | Every note type with its JSON Schema. |
| `GET /api/notes?type=&q=&limit=&offset=&full=` | all | List or search. Readers see published notes only. |
| `GET /api/note?path=`, `?name=` or `?type=&slug=` | all | One note with its backlinks. An old slug answers with `{ redirect }`. |
| `GET /api/notes?type=&q=&limit=&offset=&full=` | all | List or search. Readers see published notes only. With `full=true` each note carries `synced` (see [Sync jobs](#sync-jobs)). |
| `GET /api/note?path=`, `?name=` or `?type=&slug=` | all | One note with its backlinks and `synced`. An old slug answers with `{ redirect }`. |
| `GET /api/backlinks?name=` | all | Notes linking to a note. |
| `GET /api/redirects` | all | Every redirect. |
| `GET /attachments/<file>` | all | An image from `_attachments`. |
@@ -93,6 +96,11 @@ roles in `CONTENT_TOKENS`.
| `POST /api/held/<id>/accept`, `/reject` | editor, admin | Decide a held change. `{ "restore": true }` on reject puts the live version back into the vault. |
| `POST /api/note/rollback` | admin | `{ path, rev }` |
| `POST /api/scan` | editor, admin | Compare the whole folder now. |
| `POST /api/submissions` | intake, admin | File a saved copy of an inquiry or order. Answers `{ id, receipt }`. |
| `POST /api/submissions/<id>/delivery` | intake (with the receipt, once), admin | `{ receipt, status: sent, failed or skipped, error? }`: how the email went. |
| `GET /api/submissions?handled=`, `/api/submissions/<id>` | admin | Saved copies, unhandled first, newest first, with counts. |
| `POST /api/submissions/<id>/handled` | admin | `{ handled: true or false }` |
| `GET /api/submissions/<id>/attachment` | admin | The attached file, always as a download. |
| `/mcp` | agent, editor, admin | The MCP server. |
### MCP
@@ -109,11 +117,17 @@ move or write a raw file.
| `SITE_DIR` | required | The vault's Site folder. |
| `DATA_DIR` | `data` | Holds `content.db`. Back it up: history and the review queue cannot be rebuilt from the vault. |
| `PORT`, `HOST` | `4310`, `0.0.0.0` | |
| `CONTENT_TOKENS` | none | `role:token` pairs, comma separated. Roles: reader, agent, editor, admin. Tokens are 24 characters or more. |
| `CONTENT_TOKENS` | none | `role:token` pairs, comma separated. Roles: reader, agent, editor, admin, intake. Tokens are 24 characters or more. |
| `ALLOWED_EMAILS` | none | Addresses an agent may add without a hold. |
| `HEALTHCHECK_URL` | none | Pinged after every full scan. |
| `NOTIFY_WEBHOOK_URL` | none | Receives `{ event: "held", held }` when a change is held. |
| `SCAN_MINUTES` | `5` | Full scan interval, on top of the file watch. |
| `SUBMISSION_MAX_MB` | `10` | Largest attachment a form submission may carry. |
| `SYNC_JOBS` | none | Public sync jobs to run, comma separated: `archidekt`, `bgg`, `covers`. See [Sync jobs](#sync-jobs). |
| `SYNC_<JOB>_MINUTES` | per job | Overrides a job's interval, e.g. `SYNC_BGG_MINUTES=720`. |
| `SYNC_<JOB>_HEALTHCHECK_URL` | none | The job's Healthchecks ping URL, e.g. `SYNC_ARCHIDEKT_HEALTHCHECK_URL`. |
| `BGG_TOKEN` | none | BoardGameGeek application token, sent as `Authorization: Bearer`. Register an app with BGG if the XML API answers 401. |
| `TMDB_API_KEY` | none | TMDB v3 API key or v4 read access token. Without it, film and TV covers are skipped. |
```sh
SITE_DIR=/path/to/Site CONTENT_TOKENS="reader:…,agent:…,editor:…,admin:…" npm run content
@@ -121,3 +135,230 @@ SITE_DIR=/path/to/Site CONTENT_TOKENS="reader:…,agent:…,editor:…,admin:…
The store is SQLite (`node:sqlite`) with full-text search, so the service has
no database server to run.
## Sync jobs
Outside sources sync on a schedule into a cache the site reads, so a source
being down never takes a page down. Code: [`services/content/src/sync/`](../services/content/src/sync/).
| Job | Every | Reads | Stores per note |
| --- | --- | --- | --- |
| `archidekt` | hour | Deck notes with `archidektId`: `GET https://archidekt.com/api/decks/<id>/` | `commanderArt` (Scryfall art crop of the commander, else the deck's featured image), `colors` (color identity, WUBRG letters), `commanders`, `deckSize` (cards, maybeboard excluded), `format`, `deckUrl` |
| `bgg` | day | Board-game notes' `bggId`: XML API2 `thing`, 20 ids a request, 5 s apart, waits out `202` | `minPlayers`, `maxPlayers`, `playTime`, `minPlayTime`, `maxPlayTime`, `cover` (box art) |
| `covers` | 15 min | Library items with no `cover`: Open Library (books), TMDB (film, TV; needs `TMDB_API_KEY`), AniList (anime, manga, manhwa, light novels, webnovels; by `anilistId` when set) | `cover` |
| `suwayomi` | hour | PersonalReader's manga and manhwa: Suwayomi's GraphQL library (category, furthest chapter read and when, AniList tracker id), matched to Library items by `anilistId`; AniList for covers | `chapterProgress` (`title`, `status`, `chapter`, `readAt`), `cover` (AniList); changes `state` (below) |
| `kavita` | day | PersonalReader's light novels and books: Kavita's API, matched by `kavitaSeriesId`, or by `anilistId` against an anilist.co link in the series' metadata | `bookProgress` (`title`, `percent`, `readAt`); changes `state` (below) |
| `bambuddy` | hour | Filament spools in stock (**unverified API**, below) | `filamentStock` on product lines with a Filament option |
| `homebox` | hour | Homebox items with the labels or locations in `HOMEBOX_LABELS` / `HOMEBOX_LOCATIONS` | `materialStock` on product lines with a Leather or Paper option |
The covers job looks every 15 minutes, so a new Library item gets a cover
soon after it appears, and sweeps slowly: at most 20 items a run, an item
with no cover found is asked again after a day, one with a cover after 30
days. Only image addresses are stored; nothing is downloaded. Video games
have no cover source.
### Homelab sources (Cosmos)
These four run on Cosmos over `homelab_network`, are **read only** against
their source, and are enabled by their own URL and login alone (not
`SYNC_JOBS`). Code: `sync/jobs/{suwayomi,kavita,bambuddy,homebox}.ts`,
settings in `sync/jobs/homelab.ts`. None of them has been run against a real
server from this repository: they are written from each project's API
documentation (Bambuddy: from nothing, see below) and tested with
hand-written sample payloads in `services/content/test/sync/fixtures`.
**PersonalReader (Suwayomi, Kavita)** is private. A job touches a Library
item only when the note carries the series' id: `anilistId` (Suwayomi's
AniList tracker record; Kavita's metadata link), or `kavitaSeriesId`. It never
matches by title. It caches only a title, a chapter or percent, when it was
read and an AniList cover: never a source site, a scan group, a thumbnail or
any address inside Suwayomi or Kavita.
| Reader says | Library `state` |
| --- | --- |
| Suwayomi category Reading | `now` (a note at `caught up` stays) |
| Plan to read | `up next` |
| Completed | `finished` |
| Dropped, or any other category | left alone |
| Kavita: every page read | `finished` when the series is Completed or Ended, else `caught up` (a note at `finished` stays) |
| Kavita: some pages read | `now` (a note at `caught up` stays) |
| Kavita: unread and on Want to Read | `up next` |
State changes go through `ctx.apply` as `sync`, so a note Greg wrote is held
for review (`agent-edit`), as is everything else in the rules above. A job
proposes each change once: it is not proposed again while it waits, nor after
Greg rejects it, until the reader says something different (the job keeps
that memo in its cache under `_memo`, which is not a note and never reaches
the site).
- **Suwayomi** logs in with its own read-only account. `SUWAYOMI_AUTH=ui_login`
(the default with a user set) uses the GraphQL `login` mutation and sends the
access token as `Authorization: Bearer`; `basic` sends HTTP basic auth (for
`AUTH_MODE=basic_auth`); `none` sends nothing. Suwayomi's cookie-based
`simple_login` mode is not supported. It then reads `categories { nodes {
name mangas { nodes { title latestReadChapter { chapterNumber lastReadAt }
trackRecords { nodes { trackerId remoteId title lastChapterRead } } } } } }`
from `<SUWAYOMI_URL>/api/graphql`; the AniList record is tracker id 2. Covers,
titles and media for suggestions come from one batched AniList query.
- **Library suggestions.** A Reading series with an AniList id and no Library
item is listed at `GET /api/suggestions` (editor, admin, agent) and on the
admin Overview, with one button that creates the Library item (title,
medium from AniList, `anilistId`, `state: now`, published) through
`POST /api/changes`, and a Dismiss (`POST /api/suggestions/<key>/dismiss`,
editor or admin; kept in the `suggestion_dismissals` table). The list itself
is the job's cache entry `_suggestions`; a series leaves it as soon as a
Library item carries its AniList id.
- **Kavita** exchanges `KAVITA_API_KEY` at `POST /api/Plugin/authenticate` for
a token, reads `GET /api/Series/<id>`, `GET /api/Series/metadata?seriesId=`,
`POST /api/Series/all-v2` (only when a note needs an AniList match; one
metadata request per series, a quarter second apart) and
`POST /api/want-to-read/v2`. AniList matching applies to book, light novel
and webnovel notes only, so a manga note is Suwayomi's.
**Store stock (Bambuddy, Homebox)** never writes notes. A job stores what is
in stock on every product line that has an option in a group it feeds
(`filamentStock` for Bambuddy's group, Filament by default;
`materialStock` for Homebox's), as `[{ group, options: [{ name, hex?, inStock }] }]`.
The site lays it under the note (`optionGroups` in
`services/site/src/lib/store.ts`, which the Store pages and the order check
both use): an option's own `inStock` and `hex` always win; an option that
leaves `inStock` out (it is optional) takes the synced value, and is out of
stock when the source does not have it; in-stock colors the note does not
list are added after the note's own. With nothing synced, options show
exactly as the note says (a missing `inStock` counts as in stock).
- **Bambuddy: unverified.** Its API is not documented anywhere this project
has, and no request has been made to one. The endpoint is
`BAMBUDDY_SPOOLS_PATH` (default `/api/v1/inventory/spools`, a guess), and all
knowledge of the response is in `mapBambuddySpools` in
`sync/jobs/bambuddy.ts`: it accepts a list (or one under `items`, `spools`,
`data`, `results`) and reads color name, hex, material and grams left under
common spellings. Check it against Greg's Bambuddy and fix that one function
before relying on it. A color is in stock when some spool of it has at least
`BAMBUDDY_MIN_GRAMS` (default 50) left, or does not say.
- **Homebox** logs in at `POST /api/v1/users/login`, looks up the label and
location ids, and lists `GET /api/v1/items?labels=<id>` (or
`locations=<id>`). Each `HOMEBOX_LABELS` / `HOMEBOX_LOCATIONS` entry is
`name` or `name=Group` (`leather` feeds Leather). An item is in stock when
its quantity is above zero; a `#rrggbb` in its description is its swatch
color. A name that matches no label or location fails the run.
| Variable | |
| --- | --- |
| `SUWAYOMI_URL` | Enables `suwayomi`, like `http://suwayomi:4567`. |
| `SUWAYOMI_USER`, `SUWAYOMI_PASS` | Its own read-only Suwayomi account. |
| `SUWAYOMI_AUTH` | `ui_login` (default with a user), `basic`, or `none`. |
| `KAVITA_URL`, `KAVITA_API_KEY` | Enable `kavita`, like `http://kavita:5000`; the key of a Kavita user that can see the libraries. |
| `BAMBUDDY_URL` | Enables `bambuddy`. |
| `BAMBUDDY_SPOOLS_PATH` | Endpoint path (default `/api/v1/inventory/spools`, unverified). |
| `BAMBUDDY_API_KEY`, `BAMBUDDY_AUTH_HEADER` | Sent as `Authorization: Bearer <key>`, or as the raw key in the named header. |
| `BAMBUDDY_GROUP`, `BAMBUDDY_MATERIALS`, `BAMBUDDY_MIN_GRAMS` | Option group (Filament), materials to include (like `PLA,PETG`; all when empty), grams that count as in stock (50). |
| `HOMEBOX_URL`, `HOMEBOX_USER`, `HOMEBOX_PASS` | Enable `homebox`, with a read-only Homebox user. |
| `HOMEBOX_LABELS`, `HOMEBOX_LOCATIONS` | Which items count, `name` or `name=Group`, comma separated. Labels default to `leather,paper`. |
Each also takes `SYNC_<JOB>_MINUTES` and `SYNC_<JOB>_HEALTHCHECK_URL` like any
job (`SYNC_SUWAYOMI_HEALTHCHECK_URL`, and so on).
### How they run
- **Enabled only when configured.** The public sources run when named in
`SYNC_JOBS`. A job that needs a URL or a login is enabled by those
variables alone.
- **Scheduler.** Each job runs once shortly after start (20 seconds apart,
so they do not all start together), then every interval. A job never runs
twice at once; a run past its timeout (10 minutes unless the job says
otherwise) is aborted and reported as failed.
- **Reporting.** Every run lands in `sync_runs` (`/api/syncs`, the admin
Overview). With `SYNC_<JOB>_HEALTHCHECK_URL` set, the job pings
`<url>/start`, then `<url>` or `<url>/fail` with the message, as the nightly
backup does.
- **The cache.** Table `sync_cache` in `content.db`, keyed by job and note
path, holding the values as JSON and when they were fetched. A job replaces
a note's values only with values it fetched successfully, so a failed run
or a failed item keeps the last good values. Values for notes that leave a
job's scope (a deck loses its `archidektId`) are dropped. The cache can be
rebuilt by running the jobs again.
### What the site gets
`/api/notes?full=true` and `/api/note` give every note a `synced` object:
each job's values for that note merged together (jobs do not share keys).
The site lays `synced` under the note's `data`, so **a value set in the note
always wins** (a note's own `cover` beats a synced one), and a note with
nothing synced renders exactly as before.
### Writing a job
A job is a `SyncJob` ([`sync/types.ts`](../services/content/src/sync/types.ts)),
added to `enabledJobs` in [`sync/index.ts`](../services/content/src/sync/index.ts)
behind the settings it needs (read in [`sync/settings.ts`](../services/content/src/sync/settings.ts)
from the environment, and documented above):
```ts
interface SyncJob {
name: string; // "bambuddy": sync_runs, the cache and SYNC_BAMBUDDY_* use it
everyMinutes: number;
timeoutMs?: number; // default 10 minutes
healthcheckUrl?: string;
run(ctx: SyncContext): Promise<{ message?: string; counts?: Record<string, number>; ok?: boolean }>;
}
```
`run` gets a `SyncContext` and nothing else:
| | |
| --- | --- |
| `ctx.notes(type)` | Live notes of a type, read only. |
| `ctx.cached(path)`, `ctx.set(path, values)` | This job's cache for a note. Call `set` only with values fetched successfully. |
| `ctx.retain(paths)` | Drop this job's values for notes outside `paths`. |
| `ctx.apply(change)` | Propose a note change as `sync` (below). |
| `ctx.fetch(url, init)` | `fetch` with the sync User-Agent, the run's abort signal and a 30 second timeout (`timeoutMs` to change). |
| `ctx.sleep(ms)`, `ctx.signal`, `ctx.now()`, `ctx.log(msg)` | Polite pauses, cancellation, the clock, logging. |
Throw to fail the whole run; return `ok: false` when some items failed. A run
that throws changes nothing already cached.
**Read-only sources** (Bambuddy filament, Homebox materials) store what they
read with `ctx.set` on the note it belongs to and add the keys to the site's
readers; they never write notes.
**Jobs that change notes** (Suwayomi and Kavita progress for PersonalReader)
call `ctx.apply({ op: "update", path, fields })`, the same
`ContentService.apply` every client uses, with author `sync`, so the note
gets `updatedBy: sync`. A sync is held to every agent rule and can do nothing
an agent cannot:
- it cannot delete, move or rename a note, or change its type;
- a change to a note the sync did not write (Greg's, an agent's) is held for
review unless the note has `agentEdits: true`;
- added email addresses or phone numbers, anything that looks like a secret,
and large text loss are held;
- it may publish only statuses and Library items; anything else is held;
- more than 10 sync changes in 10 minutes are held (`batch`), so a job that
updates many notes should change only what actually differs.
The outcome says `accepted`, `held` or `rejected`; count those in the run's
`counts`. A note with a change waiting for review takes no further change
until Greg decides it, which `apply` answers as `rejected` with `waiting`.
## Form submissions
The site's Inquiries and Store order forms are emailed to Greg and saved here
as well (`services/content/src/submissions.ts`), so a lost email never loses a
request. Each copy keeps its kind (inquiry or order), type (inquiry type or
product line), name, email, the answers in form order, the attachment if any,
when it came in, how its email went (`pending`, `sent`, `failed`, `skipped`
when SMTP is not set up) and whether Greg has handled it.
- The site files copies with an `intake` token. That token can do nothing
else: every other route, `/mcp` included, answers 403, and it can never read
a copy back, not even the one it just filed. It reports the email result
once, with the receipt it got when filing.
- Only the `admin` token lists, reads, downloads and marks copies handled
(the admin page's Inbox).
- Attachments are kept under `DATA_DIR/submissions` with a name the service
makes up (id, random part, a known extension), up to `SUBMISSION_MAX_MB`.
Allowed: png, jpg, jpeg, gif, webp, avif, heic, pdf, txt, csv, dek, dec. They
are only ever served as downloads.
- The rows live in `content.db`, so the `backup.dump=sqlite` label covers
them; back up `DATA_DIR/submissions` with the volume too.