Antares/com.gregkilmer.www
1
0
Fork 0
mirror of https://github.com/Patiun/com.gregkilmer.www.git synced 2026-11-07 19:58:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Phase 6: sync jobs, inquiry and order forms

Browse Source 
- Sync scheduler in the content service with a per-job cache (sync_cache),
  sync_runs reporting and Healthchecks pings. Jobs: Archidekt, BoardGameGeek,
  cover art (Open Library, TMDB, AniList), Suwayomi and Kavita reading
  progress, Bambuddy filament and Homebox material stock. A note's own value
  always wins over a synced one.
- PersonalReader: updates only Library notes with a matching anilistId,
  changes to Greg's notes are held for review; Reading series with no note
  show as suggestions on the admin Overview.
- /api/inquiries and /api/orders: Turnstile, rate limits, a saved copy in the
  content service (intake token) plus email over SMTP. Orders are re-checked
  against the live notes: accepted only when the category's ordersOpen is
  true and the line has capacity.
- Admin Inbox for saved submissions. Options' inStock is optional so synced
  stock can fill it. All settings from process.env; .env.example per service.
- Sources could not be reached from the build environment; parsers are tested
  against fixtures labelled as not live.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aSbo2D1qP8cMFQCykf7kx
This commit is contained in:
Claude
2026-10-11 23:50:06 +00:00
parent 60dfeecd7f
commit e3b0cf60c5
76 changed files with 6050 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/admin.md
+2 -1
View File
@@ -15,13 +15,14 @@ the vault so Obsidian shows them too. Its changes are recorded with the author
| Page | What it does |
| --- | --- |
| Overview | Live and total notes, what waits for review, folder problems, the service's last scan and last error, a Scan now button, and each sync job's last run and last success. |
| Overview | Live and total notes, what waits for review, folder problems, the service's last scan and last error, a Scan now button, each sync job's last run and last success, and Library suggestions: series moved to Reading in PersonalReader with no Library item, each with Add to the Library (title, medium, AniList id, state now, published) and Dismiss. |
| Review | Held changes with the rules' reasons and a line diff against the live version. Accept puts it live (and into the vault, for API changes). Reject keeps the live version; for an edit made in Obsidian, "Reject and put the live version back" also rewrites the file. Earlier decisions are listed by status. |
| Notes, Drafts | Browse and search every note, or only the unpublished ones. |
| A note | A preview of the body, its fields, what links to it, Publish or Unpublish, and its history. Any earlier version can be compared with the live one and rolled back. |
| Capture | A status for the Now log, or a Library item with the option to post a status about it in the same step. |
| Store | Orders open or closed per category, with the reopen date and turnaround, and open slots or stock per product line. |
| Problems | Whole-folder problems, like links to notes that do not exist. |
| Inbox | Inquiries and order requests from the site, unhandled first and newest first, with whether each email reached you. Open one to read it, reply by email, download its attachment, and mark it handled or not handled. |
## Safety
docs/content-service.md
+244 -3
View File
@@ -42,6 +42,9 @@ loosen them.
| `agent-publish` | An agent publishes anything but a status or a Library item. |
| `batch` | An agent makes more than 10 changes in 10 minutes. |
A sync job writing notes (author `sync`) runs every agent rule above, the
`batch` limit counted for syncs on their own; see [Sync jobs](#sync-jobs).
Invalid input through the API is turned down at once with the problems, rather
than held, so callers fix it themselves. A change from Obsidian that fails is
held, because the file is already on disk.
@@ -78,8 +81,8 @@ roles in `CONTENT_TOKENS`.
| --- | --- | --- |
| `GET /health` | anyone | Note, published and held counts. |
| `GET /api/schema` | all | Every note type with its JSON Schema. |
| `GET /api/notes?type=&q=&limit=&offset=&full=` | all | List or search. Readers see published notes only. |
| `GET /api/note?path=`, `?name=` or `?type=&slug=` | all | One note with its backlinks. An old slug answers with `{ redirect }`. |
| `GET /api/notes?type=&q=&limit=&offset=&full=` | all | List or search. Readers see published notes only. With `full=true` each note carries `synced` (see [Sync jobs](#sync-jobs)). |
| `GET /api/note?path=`, `?name=` or `?type=&slug=` | all | One note with its backlinks and `synced`. An old slug answers with `{ redirect }`. |
| `GET /api/backlinks?name=` | all | Notes linking to a note. |
| `GET /api/redirects` | all | Every redirect. |
| `GET /attachments/<file>` | all | An image from `_attachments`. |
@@ -93,6 +96,11 @@ roles in `CONTENT_TOKENS`.
| `POST /api/held/<id>/accept`, `/reject` | editor, admin | Decide a held change. `{ "restore": true }` on reject puts the live version back into the vault. |
| `POST /api/note/rollback` | admin | `{ path, rev }` |
| `POST /api/scan` | editor, admin | Compare the whole folder now. |
| `POST /api/submissions` | intake, admin | File a saved copy of an inquiry or order. Answers `{ id, receipt }`. |
| `POST /api/submissions/<id>/delivery` | intake (with the receipt, once), admin | `{ receipt, status: sent, failed or skipped, error? }`: how the email went. |
| `GET /api/submissions?handled=`, `/api/submissions/<id>` | admin | Saved copies, unhandled first, newest first, with counts. |
| `POST /api/submissions/<id>/handled` | admin | `{ handled: true or false }` |
| `GET /api/submissions/<id>/attachment` | admin | The attached file, always as a download. |
| `/mcp` | agent, editor, admin | The MCP server. |
### MCP
@@ -109,11 +117,17 @@ move or write a raw file.
| `SITE_DIR` | required | The vault's Site folder. |
| `DATA_DIR` | `data` | Holds `content.db`. Back it up: history and the review queue cannot be rebuilt from the vault. |
| `PORT`, `HOST` | `4310`, `0.0.0.0` | |
| `CONTENT_TOKENS` | none | `role:token` pairs, comma separated. Roles: reader, agent, editor, admin. Tokens are 24 characters or more. |
| `CONTENT_TOKENS` | none | `role:token` pairs, comma separated. Roles: reader, agent, editor, admin, intake. Tokens are 24 characters or more. |
| `ALLOWED_EMAILS` | none | Addresses an agent may add without a hold. |
| `HEALTHCHECK_URL` | none | Pinged after every full scan. |
| `NOTIFY_WEBHOOK_URL` | none | Receives `{ event: "held", held }` when a change is held. |
| `SCAN_MINUTES` | `5` | Full scan interval, on top of the file watch. |
| `SUBMISSION_MAX_MB` | `10` | Largest attachment a form submission may carry. |
| `SYNC_JOBS` | none | Public sync jobs to run, comma separated: `archidekt`, `bgg`, `covers`. See [Sync jobs](#sync-jobs). |
| `SYNC_<JOB>_MINUTES` | per job | Overrides a job's interval, e.g. `SYNC_BGG_MINUTES=720`. |
| `SYNC_<JOB>_HEALTHCHECK_URL` | none | The job's Healthchecks ping URL, e.g. `SYNC_ARCHIDEKT_HEALTHCHECK_URL`. |
| `BGG_TOKEN` | none | BoardGameGeek application token, sent as `Authorization: Bearer`. Register an app with BGG if the XML API answers 401. |
| `TMDB_API_KEY` | none | TMDB v3 API key or v4 read access token. Without it, film and TV covers are skipped. |
```sh
SITE_DIR=/path/to/Site CONTENT_TOKENS="reader:…,agent:…,editor:…,admin:…" npm run content
@@ -121,3 +135,230 @@ SITE_DIR=/path/to/Site CONTENT_TOKENS="reader:…,agent:…,editor:…,admin:…
The store is SQLite (`node:sqlite`) with full-text search, so the service has
no database server to run.
## Sync jobs
Outside sources sync on a schedule into a cache the site reads, so a source
being down never takes a page down. Code: [`services/content/src/sync/`](../services/content/src/sync/).
| Job | Every | Reads | Stores per note |
| --- | --- | --- | --- |
| `archidekt` | hour | Deck notes with `archidektId`: `GET https://archidekt.com/api/decks/<id>/` | `commanderArt` (Scryfall art crop of the commander, else the deck's featured image), `colors` (color identity, WUBRG letters), `commanders`, `deckSize` (cards, maybeboard excluded), `format`, `deckUrl` |
| `bgg` | day | Board-game notes' `bggId`: XML API2 `thing`, 20 ids a request, 5 s apart, waits out `202` | `minPlayers`, `maxPlayers`, `playTime`, `minPlayTime`, `maxPlayTime`, `cover` (box art) |
| `covers` | 15 min | Library items with no `cover`: Open Library (books), TMDB (film, TV; needs `TMDB_API_KEY`), AniList (anime, manga, manhwa, light novels, webnovels; by `anilistId` when set) | `cover` |
| `suwayomi` | hour | PersonalReader's manga and manhwa: Suwayomi's GraphQL library (category, furthest chapter read and when, AniList tracker id), matched to Library items by `anilistId`; AniList for covers | `chapterProgress` (`title`, `status`, `chapter`, `readAt`), `cover` (AniList); changes `state` (below) |
| `kavita` | day | PersonalReader's light novels and books: Kavita's API, matched by `kavitaSeriesId`, or by `anilistId` against an anilist.co link in the series' metadata | `bookProgress` (`title`, `percent`, `readAt`); changes `state` (below) |
| `bambuddy` | hour | Filament spools in stock (**unverified API**, below) | `filamentStock` on product lines with a Filament option |
| `homebox` | hour | Homebox items with the labels or locations in `HOMEBOX_LABELS` / `HOMEBOX_LOCATIONS` | `materialStock` on product lines with a Leather or Paper option |
The covers job looks every 15 minutes, so a new Library item gets a cover
soon after it appears, and sweeps slowly: at most 20 items a run, an item
with no cover found is asked again after a day, one with a cover after 30
days. Only image addresses are stored; nothing is downloaded. Video games
have no cover source.
### Homelab sources (Cosmos)
These four run on Cosmos over `homelab_network`, are **read only** against
their source, and are enabled by their own URL and login alone (not
`SYNC_JOBS`). Code: `sync/jobs/{suwayomi,kavita,bambuddy,homebox}.ts`,
settings in `sync/jobs/homelab.ts`. None of them has been run against a real
server from this repository: they are written from each project's API
documentation (Bambuddy: from nothing, see below) and tested with
hand-written sample payloads in `services/content/test/sync/fixtures`.
**PersonalReader (Suwayomi, Kavita)** is private. A job touches a Library
item only when the note carries the series' id: `anilistId` (Suwayomi's
AniList tracker record; Kavita's metadata link), or `kavitaSeriesId`. It never
matches by title. It caches only a title, a chapter or percent, when it was
read and an AniList cover: never a source site, a scan group, a thumbnail or
any address inside Suwayomi or Kavita.
| Reader says | Library `state` |
| --- | --- |
| Suwayomi category Reading | `now` (a note at `caught up` stays) |
| Plan to read | `up next` |
| Completed | `finished` |
| Dropped, or any other category | left alone |
| Kavita: every page read | `finished` when the series is Completed or Ended, else `caught up` (a note at `finished` stays) |
| Kavita: some pages read | `now` (a note at `caught up` stays) |
| Kavita: unread and on Want to Read | `up next` |
State changes go through `ctx.apply` as `sync`, so a note Greg wrote is held
for review (`agent-edit`), as is everything else in the rules above. A job
proposes each change once: it is not proposed again while it waits, nor after
Greg rejects it, until the reader says something different (the job keeps
that memo in its cache under `_memo`, which is not a note and never reaches
the site).
- **Suwayomi** logs in with its own read-only account. `SUWAYOMI_AUTH=ui_login`
(the default with a user set) uses the GraphQL `login` mutation and sends the
access token as `Authorization: Bearer`; `basic` sends HTTP basic auth (for
`AUTH_MODE=basic_auth`); `none` sends nothing. Suwayomi's cookie-based
`simple_login` mode is not supported. It then reads `categories { nodes {
name mangas { nodes { title latestReadChapter { chapterNumber lastReadAt }
trackRecords { nodes { trackerId remoteId title lastChapterRead } } } } } }`
from `<SUWAYOMI_URL>/api/graphql`; the AniList record is tracker id 2. Covers,
titles and media for suggestions come from one batched AniList query.
- **Library suggestions.** A Reading series with an AniList id and no Library
item is listed at `GET /api/suggestions` (editor, admin, agent) and on the
admin Overview, with one button that creates the Library item (title,
medium from AniList, `anilistId`, `state: now`, published) through
`POST /api/changes`, and a Dismiss (`POST /api/suggestions/<key>/dismiss`,
editor or admin; kept in the `suggestion_dismissals` table). The list itself
is the job's cache entry `_suggestions`; a series leaves it as soon as a
Library item carries its AniList id.
- **Kavita** exchanges `KAVITA_API_KEY` at `POST /api/Plugin/authenticate` for
a token, reads `GET /api/Series/<id>`, `GET /api/Series/metadata?seriesId=`,
`POST /api/Series/all-v2` (only when a note needs an AniList match; one
metadata request per series, a quarter second apart) and
`POST /api/want-to-read/v2`. AniList matching applies to book, light novel
and webnovel notes only, so a manga note is Suwayomi's.
**Store stock (Bambuddy, Homebox)** never writes notes. A job stores what is
in stock on every product line that has an option in a group it feeds
(`filamentStock` for Bambuddy's group, Filament by default;
`materialStock` for Homebox's), as `[{ group, options: [{ name, hex?, inStock }] }]`.
The site lays it under the note (`optionGroups` in
`services/site/src/lib/store.ts`, which the Store pages and the order check
both use): an option's own `inStock` and `hex` always win; an option that
leaves `inStock` out (it is optional) takes the synced value, and is out of
stock when the source does not have it; in-stock colors the note does not
list are added after the note's own. With nothing synced, options show
exactly as the note says (a missing `inStock` counts as in stock).
- **Bambuddy: unverified.** Its API is not documented anywhere this project
has, and no request has been made to one. The endpoint is
`BAMBUDDY_SPOOLS_PATH` (default `/api/v1/inventory/spools`, a guess), and all
knowledge of the response is in `mapBambuddySpools` in
`sync/jobs/bambuddy.ts`: it accepts a list (or one under `items`, `spools`,
`data`, `results`) and reads color name, hex, material and grams left under
common spellings. Check it against Greg's Bambuddy and fix that one function
before relying on it. A color is in stock when some spool of it has at least
`BAMBUDDY_MIN_GRAMS` (default 50) left, or does not say.
- **Homebox** logs in at `POST /api/v1/users/login`, looks up the label and
location ids, and lists `GET /api/v1/items?labels=<id>` (or
`locations=<id>`). Each `HOMEBOX_LABELS` / `HOMEBOX_LOCATIONS` entry is
`name` or `name=Group` (`leather` feeds Leather). An item is in stock when
its quantity is above zero; a `#rrggbb` in its description is its swatch
color. A name that matches no label or location fails the run.
| Variable | |
| --- | --- |
| `SUWAYOMI_URL` | Enables `suwayomi`, like `http://suwayomi:4567`. |
| `SUWAYOMI_USER`, `SUWAYOMI_PASS` | Its own read-only Suwayomi account. |
| `SUWAYOMI_AUTH` | `ui_login` (default with a user), `basic`, or `none`. |
| `KAVITA_URL`, `KAVITA_API_KEY` | Enable `kavita`, like `http://kavita:5000`; the key of a Kavita user that can see the libraries. |
| `BAMBUDDY_URL` | Enables `bambuddy`. |
| `BAMBUDDY_SPOOLS_PATH` | Endpoint path (default `/api/v1/inventory/spools`, unverified). |
| `BAMBUDDY_API_KEY`, `BAMBUDDY_AUTH_HEADER` | Sent as `Authorization: Bearer <key>`, or as the raw key in the named header. |
| `BAMBUDDY_GROUP`, `BAMBUDDY_MATERIALS`, `BAMBUDDY_MIN_GRAMS` | Option group (Filament), materials to include (like `PLA,PETG`; all when empty), grams that count as in stock (50). |
| `HOMEBOX_URL`, `HOMEBOX_USER`, `HOMEBOX_PASS` | Enable `homebox`, with a read-only Homebox user. |
| `HOMEBOX_LABELS`, `HOMEBOX_LOCATIONS` | Which items count, `name` or `name=Group`, comma separated. Labels default to `leather,paper`. |
Each also takes `SYNC_<JOB>_MINUTES` and `SYNC_<JOB>_HEALTHCHECK_URL` like any
job (`SYNC_SUWAYOMI_HEALTHCHECK_URL`, and so on).
### How they run
- **Enabled only when configured.** The public sources run when named in
`SYNC_JOBS`. A job that needs a URL or a login is enabled by those
variables alone.
- **Scheduler.** Each job runs once shortly after start (20 seconds apart,
so they do not all start together), then every interval. A job never runs
twice at once; a run past its timeout (10 minutes unless the job says
otherwise) is aborted and reported as failed.
- **Reporting.** Every run lands in `sync_runs` (`/api/syncs`, the admin
Overview). With `SYNC_<JOB>_HEALTHCHECK_URL` set, the job pings
`<url>/start`, then `<url>` or `<url>/fail` with the message, as the nightly
backup does.
- **The cache.** Table `sync_cache` in `content.db`, keyed by job and note
path, holding the values as JSON and when they were fetched. A job replaces
a note's values only with values it fetched successfully, so a failed run
or a failed item keeps the last good values. Values for notes that leave a
job's scope (a deck loses its `archidektId`) are dropped. The cache can be
rebuilt by running the jobs again.
### What the site gets
`/api/notes?full=true` and `/api/note` give every note a `synced` object:
each job's values for that note merged together (jobs do not share keys).
The site lays `synced` under the note's `data`, so **a value set in the note
always wins** (a note's own `cover` beats a synced one), and a note with
nothing synced renders exactly as before.
### Writing a job
A job is a `SyncJob` ([`sync/types.ts`](../services/content/src/sync/types.ts)),
added to `enabledJobs` in [`sync/index.ts`](../services/content/src/sync/index.ts)
behind the settings it needs (read in [`sync/settings.ts`](../services/content/src/sync/settings.ts)
from the environment, and documented above):
```ts
interface SyncJob {
name: string; // "bambuddy": sync_runs, the cache and SYNC_BAMBUDDY_* use it
everyMinutes: number;
timeoutMs?: number; // default 10 minutes
healthcheckUrl?: string;
run(ctx: SyncContext): Promise<{ message?: string; counts?: Record<string, number>; ok?: boolean }>;
}
```
`run` gets a `SyncContext` and nothing else:
| | |
| --- | --- |
| `ctx.notes(type)` | Live notes of a type, read only. |
| `ctx.cached(path)`, `ctx.set(path, values)` | This job's cache for a note. Call `set` only with values fetched successfully. |
| `ctx.retain(paths)` | Drop this job's values for notes outside `paths`. |
| `ctx.apply(change)` | Propose a note change as `sync` (below). |
| `ctx.fetch(url, init)` | `fetch` with the sync User-Agent, the run's abort signal and a 30 second timeout (`timeoutMs` to change). |
| `ctx.sleep(ms)`, `ctx.signal`, `ctx.now()`, `ctx.log(msg)` | Polite pauses, cancellation, the clock, logging. |
Throw to fail the whole run; return `ok: false` when some items failed. A run
that throws changes nothing already cached.
**Read-only sources** (Bambuddy filament, Homebox materials) store what they
read with `ctx.set` on the note it belongs to and add the keys to the site's
readers; they never write notes.
**Jobs that change notes** (Suwayomi and Kavita progress for PersonalReader)
call `ctx.apply({ op: "update", path, fields })`, the same
`ContentService.apply` every client uses, with author `sync`, so the note
gets `updatedBy: sync`. A sync is held to every agent rule and can do nothing
an agent cannot:
- it cannot delete, move or rename a note, or change its type;
- a change to a note the sync did not write (Greg's, an agent's) is held for
review unless the note has `agentEdits: true`;
- added email addresses or phone numbers, anything that looks like a secret,
and large text loss are held;
- it may publish only statuses and Library items; anything else is held;
- more than 10 sync changes in 10 minutes are held (`batch`), so a job that
updates many notes should change only what actually differs.
The outcome says `accepted`, `held` or `rejected`; count those in the run's
`counts`. A note with a change waiting for review takes no further change
until Greg decides it, which `apply` answers as `rejected` with `waiting`.
## Form submissions
The site's Inquiries and Store order forms are emailed to Greg and saved here
as well (`services/content/src/submissions.ts`), so a lost email never loses a
request. Each copy keeps its kind (inquiry or order), type (inquiry type or
product line), name, email, the answers in form order, the attachment if any,
when it came in, how its email went (`pending`, `sent`, `failed`, `skipped`
when SMTP is not set up) and whether Greg has handled it.
- The site files copies with an `intake` token. That token can do nothing
else: every other route, `/mcp` included, answers 403, and it can never read
a copy back, not even the one it just filed. It reports the email result
once, with the receipt it got when filing.
- Only the `admin` token lists, reads, downloads and marks copies handled
(the admin page's Inbox).
- Attachments are kept under `DATA_DIR/submissions` with a name the service
makes up (id, random part, a known extension), up to `SUBMISSION_MAX_MB`.
Allowed: png, jpg, jpeg, gif, webp, avif, heic, pdf, txt, csv, dek, dec. They
are only ever served as downloads.
- The rows live in `content.db`, so the `backup.dump=sqlite` label covers
them; back up `DATA_DIR/submissions` with the volume too.
docs/site.md
+38 -2
View File
@@ -54,7 +54,7 @@ gives "Oct 11, 2026".
| `/tabletop` | Systems, campaigns, card games and decks as anchors |
| `/now` | Every status; each is an anchor |
| `/blog`, `/blog/topic/<topic>`, `/blog/<slug>`, `/rss.xml` | Blog, topics, posts, feed |
| `/store`, `/store/<line>`, `/store/order` | Store, product lines, order form |
| `/store`, `/store/<line>`, `/store/order`, `/store/order/sent` | Store, product lines, order form and its confirmation |
| anything else | 404, Off the star chart |
## Rules for pages
@@ -111,4 +111,40 @@ The highlights row and Selected projects come from the `/resume` Page text note'
| Blog, topics, posts, RSS | `pages/blog/`, `rss.xml.ts` | `lib/blog.ts`, `components/blog` |
| Store, lines, order | `pages/store/` | `lib/store.ts`, `components/store` |
Forms post to `/api/inquiries` and `/api/orders`, which arrive in phase 6.
## Forms
The Inquiries form posts to `POST /api/inquiries` and the order form to
`POST /api/orders` (`src/pages/api/`). Both are plain multipart posts that
answer with a 303 redirect, so they work without JavaScript. The rules are
plain functions in `src/lib/forms.ts` (tested in `services/site/test`); the
network calls are in `src/lib/forms-server.ts`.
1. A filled honeypot field (`leave_blank`) gets the confirmation page and nothing else.
2. The answers are checked. Inquiries take only the chosen type's `job_*`,
`freelance_*` or `patiun_*` fields; others are ignored.
3. A per-address rate limit, then Cloudflare Turnstile when
`TURNSTILE_SECRET_KEY` is set (a missing or bad token, or Cloudflare being
unreachable, turns the post down).
4. Orders only: the line is checked against the live store notes, fetched
from the content service then and there. An order goes through only for a
line sold here, in a category with `ordersOpen: true`, with an open slot
(made to order) or stock, and with in-stock options. Otherwise it is
turned down, whatever was posted.
5. The copy is saved through the content service with `CONTENT_INTAKE_TOKEN`,
then emailed (SMTP via nodemailer) to `CONTACT_TO` or `ORDER_TO`, and the
email's result is recorded on the copy. A failed email still shows the
visitor the confirmation; the admin Inbox flags it. If saving fails the
email is still tried; only when neither works does the form show an error.
6. Success redirects to `/inquiries/sent` (cookie `gk-inquiry`) or
`/store/order/sent` (cookie `gk-order`): URL-encoded JSON, httpOnly,
SameSite Lax, ten minutes, secure in production. A turned-down post goes
back to the form with `?error=<code>` and a `gk-inquiry-draft` or
`gk-order-draft` cookie holding the reason, the marked fields and the
answers (an attachment has to be added again).
Settings are in `services/site/.env.example`: `CONTENT_INTAKE_TOKEN`,
`PUBLIC_TURNSTILE_SITE_KEY`, `TURNSTILE_SECRET_KEY`, `SMTP_HOST`, `SMTP_PORT`,
`SMTP_USER`, `SMTP_PASS`, `SMTP_FROM`, `SMTP_ALLOW_SELFSIGNED`, `CONTACT_TO`,
`ORDER_TO`, `CONTACT_RATE_LIMIT`, `CONTACT_RATE_WINDOW_MIN`, `ORDER_RATE_LIMIT`,
`ORDER_RATE_WINDOW_MIN`, `FORM_MAX_ATTACHMENT_MB`. The Turnstile widget needs
JavaScript; with the secret set, a visitor without it cannot send a form.
package-lock.json
Generated
+10
View File
@@ -4709,6 +4709,15 @@
"integrity": "sha512-KQyt/wLjG3TAc7DOUhpqWzgd4ERxR80JOlTK5VE5R1S12IaPVN5qkj4klBce9HPG1Njuup4Sb5bljaT34lIyjw==",
"license": "MIT"
},
"node_modules/nodemailer": {
"version": "10.0.10",
"resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.0.10.tgz",
"integrity": "sha512-He9XskOFms62SyAKkLu8CcGcYHAo+BSHSsORI8s4PJH8qZgZY4L4lDfvyN0twvmbpyG+eGrxpyBlskj9d9rJ8A==",
"license": "MIT-0",
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/normalize-path": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/normalize-path/-/normalize-path-3.0.0.tgz",
@@ -6878,6 +6887,7 @@
"@gk/ui": "*",
"astro": "7.3.5",
"marked": "18.0.14",
"nodemailer": "10.0.10",
"pdf-lib": "^1.17.1",
"pdfkit": "^0.20.2"
},
packages/schema/src/fields.ts
+1 -1
View File
@@ -87,7 +87,7 @@ export const baseFields = {
created: isoDate({ title: "Created" }).optional(),
updated: isoDate({ title: "Updated" }).optional(),
updatedBy: z
.enum(["greg", "agent", "admin"])
.enum(["greg", "agent", "admin", "sync"])
.meta({ title: "Last edited by", description: "Set by the content service; you never need to write it." })
.optional(),
agentEdits: z
packages/schema/src/types.ts
+9 -1
View File
@@ -116,6 +116,11 @@ export const libraryItem = def({
note: line({ title: "Note", description: "Context, like where you are or how you came to it." }).optional(),
progress: line({ title: "Progress", description: "Like 16/18 volumes. Synced from PersonalReader when linked." }).optional(),
anilistId: z.number().int().meta({ title: "AniList id", description: "Links the item to PersonalReader's progress." }).optional(),
kavitaSeriesId: z
.number()
.int()
.meta({ title: "Kavita series id", description: "Links a book or light novel to its series in Kavita, for PersonalReader's progress." })
.optional(),
}),
});
@@ -415,7 +420,10 @@ export const productLine = def({
group: line({ title: "Option group", description: "Like Filament, Leather or Thread." }),
name: line({ title: "Name" }),
hex: z.string().regex(/^#[0-9a-fA-F]{6}$/).meta({ title: "Color" }).optional(),
inStock: z.boolean().meta({ title: "In stock" }),
inStock: z
.boolean()
.meta({ title: "In stock", description: "Leave out to follow the synced stock (Bambuddy filament, Homebox leather and paper)." })
.optional(),
}),
)
.meta({ title: "Options" })
services/admin/src/client.ts
+42
View File
@@ -86,6 +86,37 @@ export type Change =
| { op: "create"; data: Record<string, unknown>; body?: string }
| { op: "update"; path: string; fields?: Record<string, unknown>; body?: string };
export interface Submission {
id: number;
kind: "inquiry" | "order";
type: string;
name: string;
email: string;
fields: { label: string; value: string }[];
attachment: { name: string; type: string; size: number } | null;
createdAt: string;
delivery: "pending" | "sent" | "failed" | "skipped";
deliveryError: string | null;
deliveryAt: string | null;
handled: boolean;
handledAt: string | null;
}
export interface SubmissionCounts {
total: number;
unhandled: number;
failed: number;
}
/** A series in PersonalReader's Reading with no Library item (content service /api/suggestions). */
export interface Suggestion {
key: string;
anilistId: number;
title: string;
medium: string;
cover?: string;
}
export type FullNote = Omit<Note, "text" | "backlinks" | "pending">;
export class ContentApi {
@@ -130,6 +161,17 @@ export class ContentApi {
rollback = (path: string, rev: number) => this.call<Outcome>("POST", "/api/note/rollback", { path, rev }, [200, 422]);
scan = () => this.call<{ changed: number; held: number }>("POST", "/api/scan", {});
submissions = (handled?: boolean) =>
this.get<{ counts: SubmissionCounts; submissions: Submission[] }>("/api/submissions", { handled: handled === undefined ? undefined : String(handled) });
submission = (id: number) => this.get<Submission>(`/api/submissions/${id}`);
setHandled = (id: number, handled: boolean) => this.call<Submission>("POST", `/api/submissions/${id}/handled`, { handled });
/** A submission's attachment, to pass through as a download. */
submissionFile = (id: number) =>
fetch(new URL(`/api/submissions/${id}/attachment`, this.base), { headers: { authorization: `Bearer ${this.token}` } });
suggestions = () => this.get<Suggestion[]>("/api/suggestions");
dismissSuggestion = (key: string) => this.call<{ ok: boolean }>("POST", `/api/suggestions/${encodeURIComponent(key)}/dismiss`, {});
/** Fetch an attachment to pass through to the browser. */
attachment = (name: string) =>
fetch(new URL(`/attachments/${encodeURIComponent(name)}`, this.base), { headers: { authorization: `Bearer ${this.token}` } });
services/admin/src/inbox.ts
+98
View File
@@ -0,0 +1,98 @@
/**
* The Inbox: saved copies of inquiries and order requests sent through the
* site, unhandled first and newest first. Each one shows whether its email
* reached Greg, so a failed email is never a lost request.
*/
import type { Submission, SubmissionCounts } from "./client.ts";
import { html, qs, when, type Html } from "./html.ts";
export const inboxHref = (id: number) => `/inbox/${id}`;
const kindLabel = (s: Submission) => (s.kind === "order" ? "Order" : "Inquiry");
const typeLabels: Record<string, string> = { job: "Job or role", freelance: "Freelance web work", patiun: "Patiun custom order" };
const typeLabel = (s: Submission) => (s.kind === "inquiry" ? (typeLabels[s.type] ?? s.type) : s.type);
export function deliveryPill(s: Pick<Submission, "delivery">): Html {
switch (s.delivery) {
case "sent":
return html`<span class="pill live">Emailed</span>`;
case "failed":
return html`<span class="pill bad">Email failed</span>`;
case "skipped":
return html`<span class="pill draft">Not emailed</span>`;
default:
return html`<span class="pill draft">Email unknown</span>`;
}
}
const handledPill = (s: Submission) => (s.handled ? html`<span class="pill">Handled</span>` : html`<span class="pill draft">New</span>`);
const size = (n: number) => (n >= 1024 * 1024 ? `${(n / 1024 / 1024).toFixed(1)} MB` : `${Math.max(1, Math.round(n / 1024))} KB`);
/** The first line of the longest field, as a hint of what it is about. */
function gist(s: Submission): string {
const d = s.fields.find((f) => /^(details|notes)$/i.test(f.label))?.value ?? s.fields[0]?.value ?? "";
const line = d.split("\n").find((l) => l.trim()) ?? "";
return line.length > 90 ? `${line.slice(0, 89)}…` : line;
}
export function inboxList(items: Submission[], counts: SubmissionCounts, show: string): Html {
const filters: [string, string][] = [
["open", "Not handled"],
["all", "All"],
["handled", "Handled"],
];
return html`<h1>Inbox</h1>
<p class="lede">Inquiries and order requests sent through the site. Each one is emailed to you and saved here, so nothing is lost if the email is.</p>
<div class="grid">
<div class="panel"><div class="stat">${counts.unhandled}</div>not handled yet</div>
<div class="panel"><div class="stat">${counts.failed}</div>not handled and the email failed</div>
<div class="panel"><div class="stat">${counts.total}</div>saved in all</div>
</div>
<p class="actions">${filters.map(([v, l]) => (v === show ? html`<strong>${l}</strong>` : html`<a href="/inbox${qs({ show: v === "open" ? undefined : v })}">${l}</a>`))}</p>
${items.length
? html`<div class="table-wrap"><table><thead><tr><th>From</th><th>What</th><th>About</th><th>Received</th><th>Email</th><th></th></tr></thead><tbody>
${items.map(
(s) => html`<tr>
<td><a href="${inboxHref(s.id)}">${s.name}</a><br><span class="muted">${s.email}</span></td>
<td>${kindLabel(s)}<br><span class="muted">${typeLabel(s)}</span></td>
<td>${gist(s)}${s.attachment ? html` <span class="pill">File</span>` : ""}</td>
<td>${when(s.createdAt)}</td>
<td>${deliveryPill(s)}</td>
<td>${handledPill(s)}</td>
</tr>`,
)}
</tbody></table></div>`
: html`<p class="muted">${show === "open" ? "Nothing waiting. Every request is handled." : "None yet."}</p>`}`;
}
export function inboxOne(s: Submission): Html {
const reply = `mailto:${s.email}?subject=${encodeURIComponent(s.kind === "order" ? "Your order request" : "Your inquiry")}`;
return html`<h1>${kindLabel(s)} from ${s.name}</h1>
<p class="lede"><a href="/inbox">Back to the inbox</a> · Received ${when(s.createdAt)} · ${handledPill(s)} ${s.handled ? when(s.handledAt) : ""}</p>
${s.delivery === "failed"
? html`<div class="flash err" role="alert">The email for this ${kindLabel(s).toLowerCase()} did not go out${s.deliveryError ? html`: ${s.deliveryError}` : "."} This saved copy is the only one.</div>`
: s.delivery === "skipped"
? html`<div class="flash" role="status">No email was sent (SMTP is not set up on the site). This saved copy is the only one.</div>`
: s.delivery === "pending"
? html`<div class="flash" role="status">The site did not report whether the email went out. Check your mail before relying on it.</div>`
: ""}
<div class="panel">
<dl class="fields">
<dt>Kind</dt><dd>${kindLabel(s)}: ${typeLabel(s)}</dd>
<dt>Name</dt><dd>${s.name}</dd>
<dt>Email</dt><dd><a href="${reply}">${s.email}</a></dd>
${s.fields.map((f) => html`<dt>${f.label}</dt><dd>${f.value.includes("\n") ? html`<pre>${f.value}</pre>` : f.value}</dd>`)}
${s.attachment ? html`<dt>Attachment</dt><dd><a href="/inbox/${s.id}/attachment" download>${s.attachment.name}</a> <span class="muted">${size(s.attachment.size)}, downloads as a file</span></dd>` : ""}
<dt>Email to you</dt><dd>${deliveryPill(s)} ${s.deliveryAt ? when(s.deliveryAt) : ""}</dd>
</dl>
</div>
<div class="actions">
<a class="button" href="${reply}">Reply by email</a>
<form method="post" action="/inbox/${s.id}/handled" class="inline">
<input type="hidden" name="handled" value="${s.handled ? "0" : "1"}">
<button class="${s.handled ? "" : "primary"}">${s.handled ? "Mark as not handled" : "Mark as handled"}</button>
</form>
</div>`;
}
services/admin/src/layout.ts
+3 -2
View File
@@ -9,6 +9,7 @@ export interface Flash {
const nav: [string, string][] = [
["/", "Overview"],
["/review", "Review"],
["/inbox", "Inbox"],
["/notes", "Notes"],
["/drafts", "Drafts"],
["/capture", "Capture"],
@@ -16,7 +17,7 @@ const nav: [string, string][] = [
["/problems", "Problems"],
];
export function layout(p: { title: string; path: string; held: number; flash?: Flash; body: Html }): Html {
export function layout(p: { title: string; path: string; held: number; inbox?: number; flash?: Flash; body: Html }): Html {
const current = (href: string) => (href === "/" ? p.path === "/" : p.path.startsWith(href));
return html`<!doctype html>
<html lang="en">
@@ -34,7 +35,7 @@ export function layout(p: { title: string; path: string; held: number; flash?: F
<a class="brand" href="/">GregKilmer.com admin</a>
<nav>${nav.map(
([href, label]) =>
html`<a href="${href}" ${current(href) ? html`aria-current="page"` : ""}>${label}${href === "/review" && p.held ? html`<span class="count">${p.held}</span>` : ""}</a>`,
html`<a href="${href}" ${current(href) ? html`aria-current="page"` : ""}>${label}${href === "/review" && p.held ? html`<span class="count">${p.held}</span>` : ""}${href === "/inbox" && p.inbox ? html`<span class="count">${p.inbox}</span>` : ""}</a>`,
)}</nav>
</header>
<main>
services/admin/src/server.ts
+58 -6
View File
@@ -15,6 +15,8 @@ import { ApiError, type ContentApi, type Outcome } from "./client.ts";
import { html, qs, type Html } from "./html.ts";
import { layout, type Flash } from "./layout.ts";
import * as pages from "./pages.ts";
import * as inbox from "./inbox.ts";
import { suggestionChange, suggestionsPanel } from "./suggestions.ts";
const css = readFileSync(new URL("../public/admin.css", import.meta.url), "utf8");
@@ -47,8 +49,13 @@ export function createAdminServer(api: ContentApi, opts: { now?: () => Date } =
async function get(path: string, q: URLSearchParams): Promise<Page | Redirect> {
switch (path) {
case "/": {
const [status, held, problems] = await Promise.all([api.status(), api.held(), api.problems()]);
return { title: "Overview", body: pages.overview(status, held, problems) };
const [status, held, problems, suggestions] = await Promise.all([
api.status(),
api.held(),
api.problems(),
api.suggestions().catch(() => []),
]);
return { title: "Overview", body: html`${pages.overview(status, held, problems)}${suggestionsPanel(suggestions)}` };
}
case "/syncs": {
const job = q.get("job") ?? "";
@@ -106,6 +113,16 @@ export function createAdminServer(api: ContentApi, opts: { now?: () => Date } =
case "/problems":
return { title: "Problems", body: pages.problemsView(await api.problems()) };
}
if (path === "/inbox") {
const show = q.get("show") ?? "open";
const { counts, submissions } = await api.submissions(show === "all" ? undefined : show === "handled");
return { title: "Inbox", body: inbox.inboxList(submissions, counts, show) };
}
const inboxOne = path.match(/^\/inbox\/(\d+)$/);
if (inboxOne) {
const s = await api.submission(Number(inboxOne[1]));
return { title: `${s.kind === "order" ? "Order" : "Inquiry"} from ${s.name}`, body: inbox.inboxOne(s) };
}
const review = path.match(/^\/review\/(\d+)$/);
if (review) {
const h = await api.heldOne(Number(review[1]));
@@ -179,6 +196,17 @@ export function createAdminServer(api: ContentApi, opts: { now?: () => Date } =
}
return after("/capture", o, "Added to the Library.");
}
case "/suggestions/add": {
const key = required(f, "key");
const s = (await api.suggestions()).find((x) => x.key === key);
if (!s) throw new ApiError(409, "That suggestion is gone: it was added or dismissed already.", null);
const o = await api.apply(suggestionChange(s));
return after("/", o, `Added ${s.title} to the Library as Now.`);
}
case "/suggestions/dismiss": {
await api.dismissSuggestion(required(f, "key"));
return new Redirect(`/${qs({ msg: "Dismissed. It will not be suggested again." })}`);
}
case "/store/category": {
const [settings] = await api.fullNotes("store-settings");
if (!settings) throw new ApiError(404, "There is no Store/_settings note yet.", null);
@@ -207,6 +235,12 @@ export function createAdminServer(api: ContentApi, opts: { now?: () => Date } =
return after("/store", o, "Saved.");
}
}
const handled = path.match(/^\/inbox\/(\d+)\/handled$/);
if (handled) {
const id = Number(handled[1]);
const s = await api.setHandled(id, f.get("handled") === "1");
return new Redirect(`${s.handled ? "/inbox" : inbox.inboxHref(id)}${qs({ msg: s.handled ? `Marked ${s.name}'s ${s.kind} as handled.` : "Marked as not handled." })}`);
}
const decide = path.match(/^\/review\/(\d+)\/(accept|reject)$/);
if (decide) {
const id = Number(decide[1]);
@@ -245,6 +279,21 @@ export function createAdminServer(api: ContentApi, opts: { now?: () => Date } =
return;
}
const inboxFile = url.pathname.match(/^\/inbox\/(\d+)\/attachment$/);
if (method === "GET" && inboxFile) {
// Always a download, never shown inline: a visitor chose this file.
const r = await api.submissionFile(Number(inboxFile[1]));
if (!r.ok || !r.body) return text(res, r.status === 404 ? 404 : 502, "No such attachment.");
res.writeHead(200, {
"content-type": r.headers.get("content-type") ?? "application/octet-stream",
"content-disposition": r.headers.get("content-disposition") ?? "attachment",
"x-content-type-options": "nosniff",
"cache-control": "no-store",
});
Readable.fromWeb(r.body as never).pipe(res);
return;
}
if (method === "POST") {
if (!sameOrigin(req)) return text(res, 403, "Form posts must come from the admin page itself.");
const form = new URLSearchParams(await readBody(req));
@@ -259,18 +308,21 @@ export function createAdminServer(api: ContentApi, opts: { now?: () => Date } =
res.writeHead(303, { location: out.to });
return res.end();
}
const held = (await api.held().catch(() => [])).length;
page(res, 200, out, url, held);
const [held, unhandled] = await Promise.all([
api.held().then((h) => h.length).catch(() => 0),
api.submissions(false).then((s) => s.counts.unhandled).catch(() => 0),
]);
page(res, 200, out, url, held, unhandled);
}
function page(res: ServerResponse, status: number, p: Page, url: URL, held: number) {
function page(res: ServerResponse, status: number, p: Page, url: URL, held: number, unhandled = 0) {
const flash: Flash = {
msg: url.searchParams.get("msg") ?? undefined,
err: url.searchParams.get("err") ?? undefined,
reasons: url.searchParams.getAll("r"),
};
res.writeHead(status, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store" });
res.end(layout({ title: p.title, path: url.pathname, held, flash, body: p.body }).toString());
res.end(layout({ title: p.title, path: url.pathname, held, inbox: unhandled, flash, body: p.body }).toString());
}
return createServer((req, res) => {
services/admin/src/suggestions.ts
+32
View File
@@ -0,0 +1,32 @@
/**
* Library suggestions on the Overview: a series moved to Reading in
* PersonalReader with no Library item yet. One button adds it (title,
* medium, AniList id, state now, published) through the normal write API;
* Dismiss hides it for good.
*/
import type { Change, Suggestion } from "./client.ts";
import { html, type Html } from "./html.ts";
export function suggestionsPanel(list: Suggestion[]): Html | "" {
if (!list.length) return "";
return html`<h2>Library suggestions</h2>
<p class="muted">Series you are reading in PersonalReader with no Library item. Adding one puts it on the Library as Now.</p>
<div class="table-wrap"><table>
<thead><tr><th>Series</th><th>Medium</th><th></th></tr></thead>
<tbody>${list.map(
(s) => html`<tr>
<td>${s.title} <span class="muted mono">AniList ${s.anilistId}</span></td>
<td>${s.medium}</td>
<td><form method="post" action="/suggestions/add" class="actions"><input type="hidden" name="key" value="${s.key}"><button class="primary">Add to the Library</button></form>
<form method="post" action="/suggestions/dismiss" class="actions"><input type="hidden" name="key" value="${s.key}"><button>Dismiss</button></form></td>
</tr>`,
)}</tbody></table></div>`;
}
/** The Library item a suggestion becomes. */
export function suggestionChange(s: Suggestion): Change {
return {
op: "create",
data: { type: "library-item", title: s.title, medium: s.medium, anilistId: s.anilistId, state: "now", publish: true },
};
}
services/admin/test/inbox.test.ts
+108
View File
@@ -0,0 +1,108 @@
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { parseTokens } from "../../content/src/config.ts";
import { createHttpServer } from "../../content/src/http.ts";
import { rig, type Rig } from "../../content/test/helpers.ts";
import { ContentApi } from "../src/client.ts";
import { createAdminServer } from "../src/server.ts";
const ADMIN = "d".repeat(24);
const INTAKE = "i".repeat(24);
let r: Rig;
let content: Server;
let admin: Server;
let base: string;
let contentUrl: string;
const listen = async (s: Server) => {
await new Promise<void>((ok) => s.listen(0, "127.0.0.1", ok));
return `http://127.0.0.1:${(s.address() as AddressInfo).port}`;
};
const file = (body: unknown) =>
fetch(`${contentUrl}/api/submissions`, {
method: "POST",
headers: { authorization: `Bearer ${INTAKE}`, "content-type": "application/json" },
body: JSON.stringify(body),
}).then((res) => res.json() as Promise<{ id: number; receipt: string }>);
let first: number;
let second: number;
beforeAll(async () => {
r = rig();
r.service.scan();
content = createHttpServer(r.service, { tokens: parseTokens(`admin:${ADMIN},intake:${INTAKE}`) });
contentUrl = await listen(content);
admin = createAdminServer(new ContentApi(contentUrl, ADMIN));
base = await listen(admin);
const a = await file({
kind: "inquiry",
type: "patiun",
name: "Ann <b>Bold</b>",
email: "ann@example.com",
fields: [{ label: "Details", value: "Sixty proxies\nfor a Golgari deck" }],
attachment: { name: "list.txt", data: Buffer.from("1 Sol Ring").toString("base64") },
});
first = a.id;
await fetch(`${contentUrl}/api/submissions/${a.id}/delivery`, {
method: "POST",
headers: { authorization: `Bearer ${INTAKE}`, "content-type": "application/json" },
body: JSON.stringify({ receipt: a.receipt, status: "failed", error: "Connection refused" }),
});
second = (await file({ kind: "order", type: "proxy-cards", name: "Bo", email: "bo@example.com", fields: [{ label: "Quantity", value: "75 cards" }] })).id;
});
afterAll(() => {
admin.close();
content.close();
r.done();
});
const get = async (path: string) => {
const res = await fetch(base + path, { redirect: "manual" });
return { status: res.status, text: await res.text(), headers: res.headers };
};
const post = async (path: string, form: Record<string, string>, headers: Record<string, string> = { "sec-fetch-site": "same-origin" }) => {
const res = await fetch(base + path, {
method: "POST",
redirect: "manual",
headers: { "content-type": "application/x-www-form-urlencoded", ...headers },
body: new URLSearchParams(form).toString(),
});
return { status: res.status, location: res.headers.get("location") ?? "" };
};
describe("admin inbox", () => {
it("lists submissions newest first with their email status, escaped", async () => {
const page = await get("/inbox");
expect(page.status).toBe(200);
expect(page.text.indexOf("Bo</a>")).toBeLessThan(page.text.indexOf("Ann &lt;b&gt;"));
expect(page.text).not.toContain("<b>Bold</b>");
expect(page.text).toContain("Email failed");
expect(page.text).toContain('<span class="count">2</span>');
});
it("shows one with the failure and the attachment", async () => {
const page = await get(`/inbox/${first}`);
expect(page.text).toContain("Connection refused");
expect(page.text).toContain("<pre>Sixty proxies\nfor a Golgari deck</pre>");
expect(page.text).toContain(`/inbox/${first}/attachment`);
const dl = await fetch(`${base}/inbox/${first}/attachment`);
expect(dl.headers.get("content-disposition")).toContain("attachment");
expect(await dl.text()).toBe("1 Sol Ring");
expect((await get(`/inbox/${second}/attachment`)).status).toBe(404);
});
it("marks handled and back, from this page only", async () => {
expect((await post(`/inbox/${second}/handled`, { handled: "1" }, { "sec-fetch-site": "cross-site" })).status).toBe(403);
const done = await post(`/inbox/${second}/handled`, { handled: "1" });
expect(done.status).toBe(303);
expect(done.location).toMatch(/^\/inbox\?msg=/);
const open = await get("/inbox");
expect(open.text).not.toContain("Bo</a>");
expect((await get("/inbox?show=handled")).text).toContain("Bo</a>");
await post(`/inbox/${second}/handled`, { handled: "0" });
expect((await get("/inbox")).text).toContain("Bo</a>");
});
});
services/admin/test/suggestions.test.ts
+87
View File
@@ -0,0 +1,87 @@
/**
* Library suggestions on the Overview: one button adds the Library item
* through the normal write API, Dismiss hides it. The suggestion list is
* seeded the way the Suwayomi job leaves it in the sync cache.
*/
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { parseTokens } from "../../content/src/config.ts";
import { createHttpServer } from "../../content/src/http.ts";
import { SUGGESTIONS_KEY } from "../../content/src/sync/jobs/reader.ts";
import { rig, type Rig } from "../../content/test/helpers.ts";
import { ContentApi } from "../src/client.ts";
import { createAdminServer } from "../src/server.ts";
const ADMIN = "s".repeat(24);
let r: Rig;
let content: Server;
let admin: Server;
let base: string;
const listen = async (s: Server) => {
await new Promise<void>((ok) => s.listen(0, "127.0.0.1", ok));
return `http://127.0.0.1:${(s.address() as AddressInfo).port}`;
};
const page = (path: string) => fetch(`${base}${path}`).then((res) => res.text());
const flash = (res: Response) => new URL(res.headers.get("location") ?? "/", "http://admin").searchParams;
const post = (path: string, form: Record<string, string>) =>
fetch(`${base}${path}`, {
method: "POST",
redirect: "manual",
headers: { "content-type": "application/x-www-form-urlencoded", "sec-fetch-site": "same-origin" },
body: new URLSearchParams(form).toString(),
});
beforeAll(async () => {
r = rig();
r.service.scan();
r.store.syncCache.set(
"suwayomi",
SUGGESTIONS_KEY,
{
items: [
{ anilistId: 100004, title: "New Sample Series", medium: "manhwa" },
{ anilistId: 100006, title: "Another <Sample>", medium: "manga" },
],
},
"2026-10-11T12:00:00Z",
);
content = createHttpServer(r.service, { tokens: parseTokens(`admin:${ADMIN}`) });
admin = createAdminServer(new ContentApi(await listen(content), ADMIN));
base = await listen(admin);
});
afterAll(() => {
admin.close();
content.close();
r.done();
});
describe("Library suggestions", () => {
it("show on the Overview, escaped", async () => {
const html = await page("/");
expect(html).toContain("Library suggestions");
expect(html).toContain("New Sample Series");
expect(html).toContain("Another &lt;Sample&gt;");
expect(html).toContain('value="anilist-100004"');
});
it("add creates the Library item through the write API, and the suggestion leaves", async () => {
const res = await post("/suggestions/add", { key: "anilist-100004" });
expect(res.status).toBe(303);
expect(flash(res).get("msg")).toBe("Added New Sample Series to the Library as Now.");
const note = r.store.list({ type: "library-item" }).find((n) => n.data.anilistId === 100004)!;
expect(note.data).toMatchObject({ title: "New Sample Series", medium: "manhwa", state: "now", publish: true, updatedBy: "admin" });
expect(r.vault.exists(note.path)).toBe(true);
expect(await page("/")).not.toContain("New Sample Series");
});
it("dismiss hides it for good; the panel goes when nothing is left", async () => {
expect((await post("/suggestions/dismiss", { key: "anilist-100006" })).status).toBe(303);
const html = await page("/");
expect(html).not.toContain("Library suggestions");
// A gone suggestion cannot be added.
const again = await post("/suggestions/add", { key: "anilist-100006" });
expect(flash(again).get("err")).toContain("That suggestion is gone");
});
});
services/content/.env.example
+55
View File
@@ -0,0 +1,55 @@
# Content service settings. Copy to .env (never commit it) or set them in the
# container's environment. See docs/content-service.md, "Running it".
# --- Sync jobs (docs/content-service.md, "Sync jobs") ------------------------
# Public sources to run: archidekt (hourly), bgg (daily), covers (every 15 min).
SYNC_JOBS=archidekt,bgg,covers
# Optional per-job interval override, in minutes.
# SYNC_ARCHIDEKT_MINUTES=60
# SYNC_BGG_MINUTES=1440
# SYNC_COVERS_MINUTES=15
# Healthchecks ping URLs, one per job (start, success and fail are pinged).
SYNC_ARCHIDEKT_HEALTHCHECK_URL=
SYNC_BGG_HEALTHCHECK_URL=
SYNC_COVERS_HEALTHCHECK_URL=
# BoardGameGeek application token (sent as Authorization: Bearer), if BGG asks for one.
BGG_TOKEN=
# TMDB v3 API key or v4 read access token, for film and TV covers.
TMDB_API_KEY=
# --- Homelab syncs on Cosmos (docs/content-service.md, "Homelab sources") ----
# Each is enabled by its own URL and login; all are read only against the source.
# PersonalReader, manga and manhwa (hourly). Its own read-only Suwayomi account.
SUWAYOMI_URL=
SUWAYOMI_USER=
SUWAYOMI_PASS=
# ui_login (default with a user), basic, or none.
# SUWAYOMI_AUTH=ui_login
# PersonalReader, light novels and books (daily).
KAVITA_URL=
KAVITA_API_KEY=
# Filament in stock (hourly). UNVERIFIED API: check the path and mapping first.
BAMBUDDY_URL=
# BAMBUDDY_SPOOLS_PATH=/api/v1/inventory/spools
# BAMBUDDY_API_KEY=
# BAMBUDDY_AUTH_HEADER=
# BAMBUDDY_GROUP=Filament
# BAMBUDDY_MATERIALS=PLA,PETG
# BAMBUDDY_MIN_GRAMS=50
# Leather and paper in stock (hourly). Entries are name or name=Group.
HOMEBOX_URL=
HOMEBOX_USER=
HOMEBOX_PASS=
HOMEBOX_LABELS=leather,paper
# HOMEBOX_LOCATIONS=
SYNC_SUWAYOMI_HEALTHCHECK_URL=
SYNC_KAVITA_HEALTHCHECK_URL=
SYNC_BAMBUDDY_HEALTHCHECK_URL=
SYNC_HOMEBOX_HEALTHCHECK_URL=
# --- Form submissions (docs/content-service.md, "Form submissions") ----------
# Add an intake token for the site's forms to CONTENT_TOKENS, e.g.
# CONTENT_TOKENS=reader:…,admin:…,intake:<24+ random characters>
# The site uses the same value as CONTENT_INTAKE_TOKEN. It can only file submissions.
# Largest attachment a submission may carry, in MB. Files go to DATA_DIR/submissions.
SUBMISSION_MAX_MB=10
services/content/src/config.ts
+11 -3
View File
@@ -4,8 +4,10 @@
* cannot change who may write or what the rules allow.
*/
import { resolve } from "node:path";
import { syncSettings, type SyncSettings } from "./sync/settings.ts";
export type Role = "reader" | "agent" | "editor" | "admin";
/** `intake` can only file submissions (the site's forms); it can read nothing. */
export type Role = "reader" | "agent" | "editor" | "admin" | "intake";
export interface Config {
/** The vault's Site folder. */
@@ -22,9 +24,13 @@ export interface Config {
notifyWebhookUrl?: string;
allowedEmails: string[];
scanMinutes: number;
/** Which sync jobs run, their intervals, Healthchecks URLs and keys. */
sync: SyncSettings;
/** Largest attachment a submission may carry, in bytes. */
submissionMaxBytes: number;
}
const roles = new Set<Role>(["reader", "agent", "editor", "admin"]);
const roles = new Set<Role>(["reader", "agent", "editor", "admin", "intake"]);
/** CONTENT_TOKENS is a comma separated list of role:token pairs. */
export function parseTokens(raw: string | undefined): Map<string, Role> {
@@ -53,12 +59,14 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {
notifyWebhookUrl: env.NOTIFY_WEBHOOK_URL || undefined,
allowedEmails: (env.ALLOWED_EMAILS ?? "").split(",").map((s) => s.trim().toLowerCase()).filter(Boolean),
scanMinutes: Number(env.SCAN_MINUTES ?? 5),
sync: syncSettings(env),
submissionMaxBytes: Math.round(Number(env.SUBMISSION_MAX_MB ?? 10) * 1024 * 1024),
};
}
/** What each role may do. Checked by the HTTP API and the MCP server alike. */
export const can = {
readDrafts: (r: Role) => r !== "reader",
readDrafts: (r: Role) => r !== "reader" && r !== "intake",
write: (r: Role) => r === "agent" || r === "editor" || r === "admin",
review: (r: Role) => r === "editor" || r === "admin",
rollback: (r: Role) => r === "admin",
services/content/src/http.ts
+15 -3
View File
@@ -13,6 +13,8 @@ import { can, type Config, type Role } from "./config.ts";
import { handleMcp } from "./mcp.ts";
import type { Change, ContentService } from "./service.ts";
import type { Author, StoredNote } from "./store.ts";
import { handleSubmissions } from "./submissions.ts";
import { handleSuggestions } from "./suggestions.ts";
import { heldView, noteView, outcomeView, revisionView, summaryView } from "./views.ts";
const MAX_BODY = 1_000_000;
@@ -66,7 +68,7 @@ export function schemaList() {
}));
}
export function createHttpServer(service: ContentService, config: Pick<Config, "tokens">) {
export function createHttpServer(service: ContentService, config: Pick<Config, "tokens"> & Partial<Pick<Config, "submissionMaxBytes">>) {
const { store } = service;
const visible = (n: StoredNote | null, role: Role) => (n && !n.removed && (n.publish || can.readDrafts(role)) ? n : null);
@@ -87,6 +89,12 @@ export function createHttpServer(service: ContentService, config: Pick<Config, "
if (!ok) throw new HttpError(403, `The ${role} token cannot do this`);
};
// Form submissions have their own rules; an intake token can do nothing else.
if (await handleSubmissions(req, res, { path, method, role, url, submissions: store.submissions, maxBytes: config.submissionMaxBytes })) return;
if (role === "intake") throw new HttpError(403, "The intake token can only file submissions");
// Library suggestions from PersonalReader (see suggestions.ts).
if (await handleSuggestions(req, res, { path, method, role, store })) return;
if (path === "/mcp") {
need(can.write(role));
const body = method === "POST" ? await readJson(req) : undefined;
@@ -105,8 +113,11 @@ export function createHttpServer(service: ContentService, config: Pick<Config, "
const notes = query
? store.search(query, { publishedOnly, limit }).filter((n) => !type || n.type === type)
: store.list({ type, publishedOnly, limit, offset: Number(q("offset") ?? 0) });
// full=true returns whole notes, so a list page needs one request.
return send(res, 200, notes.map((n) => (q("full") === "true" ? noteView(n) : summaryView(n))));
// full=true returns whole notes, so a list page needs one request,
// each with `synced`: what the sync jobs fetched for it (see docs).
if (q("full") !== "true") return send(res, 200, notes.map(summaryView));
const synced = store.syncCache.all();
return send(res, 200, notes.map((n) => ({ ...noteView(n), synced: synced.get(n.path) ?? {} })));
}
if (path === "/api/note") {
@@ -129,6 +140,7 @@ export function createHttpServer(service: ContentService, config: Pick<Config, "
if (!found) throw new HttpError(404, "No such note");
return send(res, 200, {
...noteView(found, { text: can.readDrafts(role) }),
synced: store.syncCache.forPath(found.path),
backlinks: store.backlinks(found.name).filter((b) => visible(b, role)).map(summaryView),
...(can.readDrafts(role) ? { pending: store.pendingFor(found.path) ? heldView(store.pendingFor(found.path)!) : null } : {}),
});
services/content/src/main.ts
+7
View File
@@ -6,6 +6,7 @@ import { loadConfig } from "./config.ts";
import { createHttpServer } from "./http.ts";
import { ContentService } from "./service.ts";
import { Store } from "./store.ts";
import { enabledJobs, Scheduler } from "./sync/index.ts";
import { Vault } from "./vault.ts";
import { Watcher } from "./watcher.ts";
import { heldView } from "./views.ts";
@@ -41,6 +42,11 @@ const watcher = new Watcher(service, config.siteDir, {
});
watcher.start();
// Sync jobs: each enabled source on its own schedule, into the sync cache.
const syncs = new Scheduler(service, enabledJobs(config.sync));
if (syncs.jobs.length) console.log(`Sync jobs: ${syncs.jobs.map((j) => `${j.name} every ${j.everyMinutes} min`).join(", ")}`);
syncs.start();
const server = createHttpServer(service, config);
server.listen(config.port, config.host, () => {
const c = store.count();
@@ -49,6 +55,7 @@ server.listen(config.port, config.host, () => {
const stop = () => {
watcher.stop();
syncs.stop();
server.close();
store.close();
process.exit(0);
services/content/src/rules.ts
+16 -8
View File
@@ -52,6 +52,12 @@ export interface RuleContext {
settings: RuleSettings;
}
/**
* Authors held to the agent rules. A sync job writes as `sync` and may do
* nothing an agent cannot: the same contact, publish, edit and batch limits.
*/
export const isLimited = (a: Author) => a === "agent" || a === "sync";
export type Verdict = { decision: "accept" } | { decision: "hold"; reasons: HeldReason[] };
/** Strings that look like credentials. Matching is on new text only. */
@@ -125,26 +131,28 @@ export function evaluate(p: Proposed, ctx: RuleContext): Verdict {
for (const [what, re] of SECRET_PATTERNS) {
if (newMatches(re, p.text, before).length) reasons.push({ code: "secret", message: `The note now contains what looks like ${what}` });
}
if (p.author === "agent") {
const who = p.author === "sync" ? "A sync" : "An agent";
if (isLimited(p.author)) {
const allowed = new Set(settings.allowedEmails.map((e) => e.toLowerCase()));
const emails = newMatches(EMAIL, p.text, before).filter((e) => !allowed.has(e.toLowerCase()));
if (emails.length) reasons.push({ code: "contact", message: `An agent added an email address: ${emails.join(", ")}` });
if (newMatches(PHONE, p.text, before).length) reasons.push({ code: "contact", message: "An agent added a phone number" });
if (emails.length) reasons.push({ code: "contact", message: `${who} added an email address: ${emails.join(", ")}` });
if (newMatches(PHONE, p.text, before).length) reasons.push({ code: "contact", message: `${who} added a phone number` });
}
// 6. Agents stay inside their lane.
if (p.author === "agent") {
if (live && live.author !== "agent" && live.data.agentEdits !== true) {
reasons.push({ code: "agent-edit", message: "An agent changed a note you wrote, which needs your review" });
if (isLimited(p.author)) {
// A sync may change notes it wrote itself; anything else needs agentEdits.
if (live && live.author !== p.author && live.data.agentEdits !== true) {
reasons.push({ code: "agent-edit", message: `${who} changed a note you wrote, which needs your review` });
}
const publishing = data.publish === true && !(live?.publish ?? false);
if (publishing && !settings.agentAutoPublish.includes(p.checked.type)) {
reasons.push({ code: "agent-publish", message: `An agent wants to publish a ${p.checked.type}, which needs your approval` });
reasons.push({ code: "agent-publish", message: `${who} wants to publish a ${p.checked.type}, which needs your approval` });
}
if (ctx.recentAgentChanges >= settings.agentBatchLimit) {
reasons.push({
code: "batch",
message: `More than ${settings.agentBatchLimit} agent changes in ${settings.agentBatchWindowMinutes} minutes`,
message: `More than ${settings.agentBatchLimit} ${p.author} changes in ${settings.agentBatchWindowMinutes} minutes`,
});
}
}
services/content/src/service.ts
+2 -2
View File
@@ -8,7 +8,7 @@
*/
import { basename } from "node:path";
import { checkNote, checkVault, formatNote, notePath, outgoingLinks, slugify, typeDef, type CheckedNote } from "@gk/schema";
import { defaultRules, evaluate, type RuleSettings } from "./rules.ts";
import { defaultRules, evaluate, isLimited, type RuleSettings } from "./rules.ts";
import type { Author, HeldChange, HeldReason, StoredNote, Store } from "./store.ts";
import { hashText, isNotePath, type Vault } from "./vault.ts";
@@ -264,7 +264,7 @@ export class ContentService {
current,
nameTakenBy,
slugTakenBy,
recentAgentChanges: author === "agent" ? this.store.changesSince("agent", window) : 0,
recentAgentChanges: isLimited(author) ? this.store.changesSince(author, window) : 0,
settings: this.rules,
},
);
services/content/src/store.ts
+11 -2
View File
@@ -8,9 +8,12 @@
*/
import { DatabaseSync } from "node:sqlite";
import { mkdirSync } from "node:fs";
import { dirname } from "node:path";
import { dirname, join } from "node:path";
import { Submissions } from "./submissions.ts";
import { SyncCache } from "./sync/cache.ts";
export type Author = "greg" | "agent" | "admin";
/** Who made a change. `sync` is a sync job writing through ContentService.apply; it runs the agent rules. */
export type Author = "greg" | "agent" | "admin" | "sync";
export interface StoredNote {
path: string;
@@ -176,12 +179,18 @@ const SYNC_RUNS_KEPT = 50;
export class Store {
readonly db: DatabaseSync;
readonly submissions: Submissions;
/** What sync jobs last fetched per note; the notes API shows it as `synced`. */
readonly syncCache: SyncCache;
constructor(file: string) {
if (file !== ":memory:") mkdirSync(dirname(file), { recursive: true });
this.db = new DatabaseSync(file);
this.db.exec("pragma journal_mode = wal; pragma foreign_keys = on;");
this.db.exec(SCHEMA);
// Saved copies of the site's forms; attachments sit beside content.db.
this.syncCache = new SyncCache(this.db);
this.submissions = new Submissions(this.db, file === ":memory:" ? null : join(dirname(file), "submissions"));
}
close() {
services/content/src/submissions.ts
+376
View File
@@ -0,0 +1,376 @@
/**
* Saved copies of what visitors send through the site's forms: inquiries and
* store order requests. Greg's decision is email plus a saved copy, so a lost
* email never loses a request. The rows live in content.db (backed up with
* the rest of it); an attachment is a file under DATA_DIR/submissions.
*
* Who may do what:
* - `intake` (the site) files a submission and reports, once, how its email
* went, using the receipt it got back. It can read nothing, not even what
* it just filed.
* - `admin` lists, reads and marks them handled, and downloads attachments.
* - Every other role gets 403 here.
*/
import { randomBytes } from "node:crypto";
import { createReadStream, mkdirSync, rmSync, statSync, writeFileSync } from "node:fs";
import type { IncomingMessage, ServerResponse } from "node:http";
import { join } from "node:path";
import type { DatabaseSync } from "node:sqlite";
import { z } from "zod";
import type { Role } from "./config.ts";
export const SUBMISSIONS_SCHEMA = `
create table if not exists submissions (
id integer primary key autoincrement,
kind text not null,
type text not null,
name text not null,
email text not null,
fields text not null,
attachment_name text,
attachment_file text,
attachment_type text,
attachment_size integer,
created_at text not null,
delivery text not null default 'pending',
delivery_error text,
delivery_at text,
receipt text not null,
handled integer not null default 0,
handled_at text
);
create index if not exists submissions_inbox on submissions(handled, id);
`;
export type Delivery = "pending" | "sent" | "failed" | "skipped";
export interface Submission {
id: number;
kind: "inquiry" | "order";
/** The inquiry type (job, freelance, patiun) or the product line's slug. */
type: string;
name: string;
email: string;
/** What the visitor sent, in form order. */
fields: { label: string; value: string }[];
attachment: { name: string; type: string; size: number } | null;
createdAt: string;
delivery: Delivery;
deliveryError: string | null;
deliveryAt: string | null;
handled: boolean;
handledAt: string | null;
}
/** File types an attachment may have, by extension, with the type it is served as. */
export const ATTACHMENT_TYPES: Record<string, string> = {
png: "image/png",
jpg: "image/jpeg",
jpeg: "image/jpeg",
gif: "image/gif",
webp: "image/webp",
avif: "image/avif",
heic: "image/heic",
pdf: "application/pdf",
txt: "text/plain; charset=utf-8",
csv: "text/csv; charset=utf-8",
dek: "text/plain; charset=utf-8",
dec: "text/plain; charset=utf-8",
};
export const DEFAULT_MAX_BYTES = 10 * 1024 * 1024;
export class SubmissionError extends Error {
constructor(readonly status: number, message: string) {
super(message);
}
}
/** A display name with nothing that could act as a path or a header break. */
export function safeName(name: string): string {
const base = name.split(/[\\/]/).pop() ?? "";
const clean = base
.normalize("NFKC")
.replace(/[^\p{L}\p{N}._ -]+/gu, "_")
.replace(/^[.\s]+/, "")
.slice(-120)
.trim();
return clean || "attachment";
}
export function extensionOf(name: string): string | null {
const m = /\.([A-Za-z0-9]{1,8})$/.exec(name);
const ext = m?.[1]?.toLowerCase();
return ext && ATTACHMENT_TYPES[ext] ? ext : null;
}
const fieldSchema = z.object({ label: z.string().trim().min(1).max(80), value: z.string().max(20_000) });
export const newSubmissionSchema = z.object({
kind: z.enum(["inquiry", "order"]),
type: z.string().trim().min(1).max(80),
name: z.string().trim().min(1).max(120),
email: z.email().max(200),
fields: z.array(fieldSchema).max(40).default([]),
attachment: z
.object({ name: z.string().min(1).max(300), data: z.string().min(1) })
.optional(),
});
export type NewSubmission = z.input<typeof newSubmissionSchema>;
const deliverySchema = z.object({
receipt: z.string().min(1).max(100).optional(),
status: z.enum(["sent", "failed", "skipped"]),
error: z.string().max(2000).optional(),
});
type Row = Record<string, any>;
const toSubmission = (r: Row): Submission => ({
id: r.id,
kind: r.kind,
type: r.type,
name: r.name,
email: r.email,
fields: JSON.parse(r.fields),
attachment: r.attachment_file ? { name: r.attachment_name, type: r.attachment_type, size: r.attachment_size } : null,
createdAt: r.created_at,
delivery: r.delivery,
deliveryError: r.delivery_error,
deliveryAt: r.delivery_at,
handled: !!r.handled,
handledAt: r.handled_at,
});
export class Submissions {
/**
* @param dir where attachment files go; null keeps no files (an in-memory
* store), and a submission with an attachment is then turned down.
*/
constructor(
private readonly db: DatabaseSync,
readonly dir: string | null,
private readonly now: () => Date = () => new Date(),
) {
db.exec(SUBMISSIONS_SCHEMA);
}
/** File a submission. Returns its id and the receipt for reporting delivery. */
create(input: NewSubmission, opts: { maxBytes?: number } = {}): { id: number; receipt: string } {
const parsed = newSubmissionSchema.safeParse(input);
if (!parsed.success) throw new SubmissionError(400, z.prettifyError(parsed.error));
const s = parsed.data;
const maxBytes = opts.maxBytes ?? DEFAULT_MAX_BYTES;
let file: { name: string; type: string; bytes: Buffer; ext: string } | null = null;
if (s.attachment) {
if (!this.dir) throw new SubmissionError(400, "This store keeps no attachments");
const ext = extensionOf(s.attachment.name);
if (!ext) throw new SubmissionError(415, `Attachments must be one of: ${Object.keys(ATTACHMENT_TYPES).join(", ")}`);
// Base64 is 4 characters per 3 bytes: refuse before decoding anything too large.
if (s.attachment.data.length > Math.ceil(maxBytes / 3) * 4 + 4) throw new SubmissionError(413, "The attachment is too large");
if (!/^[A-Za-z0-9+/]*={0,2}$/.test(s.attachment.data)) throw new SubmissionError(400, "The attachment is not base64");
const bytes = Buffer.from(s.attachment.data, "base64");
if (bytes.length > maxBytes) throw new SubmissionError(413, "The attachment is too large");
if (bytes.length === 0) throw new SubmissionError(400, "The attachment is empty");
file = { name: safeName(s.attachment.name), type: ATTACHMENT_TYPES[ext]!, bytes, ext };
}
const receipt = randomBytes(24).toString("base64url");
const at = this.now().toISOString();
const r = this.db
.prepare(
`insert into submissions (kind, type, name, email, fields, created_at, receipt)
values (?, ?, ?, ?, ?, ?, ?)`,
)
.run(s.kind, s.type, s.name, s.email, JSON.stringify(s.fields), at, receipt);
const id = Number(r.lastInsertRowid);
if (file) {
// The stored name is ours alone: the id, random bytes and a known extension.
const stored = `${id}-${randomBytes(8).toString("hex")}.${file.ext}`;
try {
mkdirSync(this.dir!, { recursive: true });
writeFileSync(join(this.dir!, stored), file.bytes, { flag: "wx" });
} catch (err) {
this.db.prepare("delete from submissions where id = ?").run(id);
throw err;
}
this.db
.prepare("update submissions set attachment_name = ?, attachment_file = ?, attachment_type = ?, attachment_size = ? where id = ?")
.run(file.name, stored, file.type, file.bytes.length, id);
}
return { id, receipt };
}
/**
* Record how the email went. With a receipt (the site), only once and only
* for the submission the receipt belongs to.
*/
setDelivery(id: number, d: { status: Exclude<Delivery, "pending">; error?: string; receipt?: string }, opts: { needReceipt: boolean }) {
const row = this.db.prepare("select receipt, delivery from submissions where id = ?").get(id) as Row | undefined;
if (opts.needReceipt) {
if (!row || !d.receipt || row.receipt !== d.receipt) throw new SubmissionError(403, "That receipt does not match");
if (row.delivery !== "pending") throw new SubmissionError(409, "Delivery is already recorded");
} else if (!row) throw new SubmissionError(404, "No such submission");
this.db
.prepare("update submissions set delivery = ?, delivery_error = ?, delivery_at = ? where id = ?")
.run(d.status, d.error ?? null, this.now().toISOString(), id);
}
/** Unhandled first, then newest first. */
list(opts: { handled?: boolean; limit?: number; offset?: number } = {}): Submission[] {
const where = opts.handled === undefined ? "" : `where handled = ${opts.handled ? 1 : 0}`;
return (
this.db
.prepare(`select * from submissions ${where} order by handled asc, id desc limit ? offset ?`)
.all(opts.limit ?? 200, opts.offset ?? 0) as Row[]
).map(toSubmission);
}
get(id: number): Submission | null {
const r = this.db.prepare("select * from submissions where id = ?").get(id) as Row | undefined;
return r ? toSubmission(r) : null;
}
counts(): { total: number; unhandled: number; failed: number } {
const r = this.db
.prepare("select count(*) total, coalesce(sum(handled = 0), 0) unhandled, coalesce(sum(delivery = 'failed' and handled = 0), 0) failed from submissions")
.get() as Row;
return { total: r.total, unhandled: r.unhandled, failed: r.failed };
}
setHandled(id: number, handled: boolean): Submission {
if (!this.get(id)) throw new SubmissionError(404, "No such submission");
this.db.prepare("update submissions set handled = ?, handled_at = ? where id = ?").run(handled ? 1 : 0, handled ? this.now().toISOString() : null, id);
return this.get(id)!;
}
/** The attachment's file on disk, with its display name and type. */
attachment(id: number): { file: string; name: string; type: string; size: number } | null {
const r = this.db.prepare("select attachment_file, attachment_name, attachment_type from submissions where id = ?").get(id) as Row | undefined;
if (!r?.attachment_file || !this.dir) return null;
const file = join(this.dir, r.attachment_file as string);
try {
return { file, name: r.attachment_name, type: r.attachment_type, size: statSync(file).size };
} catch {
return null;
}
}
/** For tests: drop the attachment folder. */
removeFiles() {
if (this.dir) rmSync(this.dir, { recursive: true, force: true });
}
}
// ------------------------------------------------------------ HTTP
const json = (res: ServerResponse, status: number, body: unknown) => {
res.writeHead(status, { "content-type": "application/json; charset=utf-8", "cache-control": "no-store" });
res.end(JSON.stringify(body));
};
async function readJson(req: IncomingMessage, limit: number): Promise<unknown> {
let size = 0;
const chunks: Buffer[] = [];
for await (const chunk of req) {
size += (chunk as Buffer).length;
if (size > limit) throw new SubmissionError(413, "The request is too large");
chunks.push(chunk as Buffer);
}
const text = Buffer.concat(chunks).toString("utf8");
if (!text) return undefined;
try {
return JSON.parse(text);
} catch {
throw new SubmissionError(400, "The body is not valid JSON");
}
}
/** RFC 6266 filename for a download, ASCII fallback plus the UTF-8 name. */
const disposition = (name: string) =>
`attachment; filename="${name.replace(/[^\x20-\x7e]|["\\]/g, "_")}"; filename*=UTF-8''${encodeURIComponent(name)}`;
/**
* Handles /api/submissions and everything under it. Returns false when the
* path is not a submissions route, so the main router carries on.
*/
export async function handleSubmissions(
req: IncomingMessage,
res: ServerResponse,
ctx: { path: string; method: string; role: Role; url: URL; submissions: Submissions; maxBytes?: number },
): Promise<boolean> {
const { path, method, role, submissions } = ctx;
if (path !== "/api/submissions" && !path.startsWith("/api/submissions/")) return false;
const maxBytes = ctx.maxBytes ?? DEFAULT_MAX_BYTES;
try {
const isIntake = role === "intake";
const isAdmin = role === "admin";
const forbid = () => {
throw new SubmissionError(403, `The ${role} token cannot do this`);
};
if (path === "/api/submissions" && method === "POST") {
if (!isIntake && !isAdmin) forbid();
const body = await readJson(req, Math.ceil(maxBytes / 3) * 4 + 256_000);
const out = submissions.create(body as NewSubmission, { maxBytes });
return (json(res, 201, out), true);
}
const delivery = path.match(/^\/api\/submissions\/(\d+)\/delivery$/);
if (delivery && method === "POST") {
if (!isIntake && !isAdmin) forbid();
const body = deliverySchema.safeParse(await readJson(req, 64_000));
if (!body.success) throw new SubmissionError(400, z.prettifyError(body.error));
submissions.setDelivery(Number(delivery[1]), body.data, { needReceipt: isIntake });
return (json(res, 200, { ok: true }), true);
}
// Everything else reads or changes saved copies: admin only.
if (!isAdmin) forbid();
if (path === "/api/submissions" && method === "GET") {
const h = ctx.url.searchParams.get("handled");
const limit = Math.min(Number(ctx.url.searchParams.get("limit") ?? 200) || 200, 1000);
return (json(res, 200, { counts: submissions.counts(), submissions: submissions.list({ handled: h === null ? undefined : h === "true", limit }) }), true);
}
const one = path.match(/^\/api\/submissions\/(\d+)$/);
if (one && method === "GET") {
const s = submissions.get(Number(one[1]));
if (!s) throw new SubmissionError(404, "No such submission");
return (json(res, 200, s), true);
}
const handled = path.match(/^\/api\/submissions\/(\d+)\/handled$/);
if (handled && method === "POST") {
const body = z.object({ handled: z.boolean() }).safeParse(await readJson(req, 10_000));
if (!body.success) throw new SubmissionError(400, z.prettifyError(body.error));
return (json(res, 200, submissions.setHandled(Number(handled[1]), body.data.handled)), true);
}
const file = path.match(/^\/api\/submissions\/(\d+)\/attachment$/);
if (file && method === "GET") {
const a = submissions.attachment(Number(file[1]));
if (!a) throw new SubmissionError(404, "No attachment");
res.writeHead(200, {
"content-type": a.type,
"content-length": a.size,
"content-disposition": disposition(a.name),
"x-content-type-options": "nosniff",
"content-security-policy": "default-src 'none'; sandbox",
"cache-control": "no-store",
});
createReadStream(a.file).pipe(res);
return true;
}
throw new SubmissionError(404, "Not found");
} catch (err) {
if (err instanceof SubmissionError) return (json(res, err.status, { error: err.message }), true);
throw err;
}
}
services/content/src/suggestions.ts
+79
View File
@@ -0,0 +1,79 @@
/**
* Library suggestions: a series Greg moved to Reading in PersonalReader that
* has no Library item yet. The Suwayomi sync job keeps the list in its cache
* (entry `_suggestions`); this module serves it to the admin page, leaves
* out series that now have a Library item, and remembers dismissals in a
* small table, so a dismissed series stays dismissed across runs.
*
* Adding one is not done here: the admin page creates the Library item with
* the normal write API (`POST /api/changes`), so every rule applies.
*
* - `GET /api/suggestions` (editor, admin, agent): `[{ key, anilistId, title, medium, cover? }]`
* - `POST /api/suggestions/<key>/dismiss` (editor, admin)
*/
import type { IncomingMessage, ServerResponse } from "node:http";
import type { DatabaseSync } from "node:sqlite";
import { can, type Role } from "./config.ts";
import type { Store } from "./store.ts";
import { SUGGESTIONS_KEY } from "./sync/jobs/reader.ts";
import type { Suggestion } from "./sync/jobs/suwayomi.ts";
/** Jobs whose cache may hold suggestions. */
const SOURCES = ["suwayomi"];
export interface SuggestionView extends Suggestion {
/** Stable id for dismissing: `anilist-<id>`. */
key: string;
}
const tables = new WeakMap<DatabaseSync, true>();
function ready(db: DatabaseSync) {
if (!tables.has(db)) {
db.exec(`create table if not exists suggestion_dismissals (key text primary key, at text not null);`);
tables.set(db, true);
}
return db;
}
export function listSuggestions(store: Store): SuggestionView[] {
const db = ready(store.db);
const dismissed = new Set((db.prepare("select key from suggestion_dismissals").all() as { key: string }[]).map((r) => r.key));
const linked = new Set(store.list({ type: "library-item" }).map((n) => n.data.anilistId).filter((id) => typeof id === "number"));
const out = new Map<string, SuggestionView>();
for (const job of SOURCES) {
const items = (store.syncCache.get(job, SUGGESTIONS_KEY)?.values.items as Suggestion[] | undefined) ?? [];
for (const s of items) {
if (typeof s?.anilistId !== "number" || linked.has(s.anilistId)) continue;
const key = `anilist-${s.anilistId}`;
if (!dismissed.has(key) && !out.has(key)) out.set(key, { ...s, key });
}
}
return [...out.values()].sort((a, b) => a.title.localeCompare(b.title));
}
export function dismissSuggestion(store: Store, key: string, at = new Date().toISOString()) {
ready(store.db).prepare("insert into suggestion_dismissals (key, at) values (?, ?) on conflict(key) do nothing").run(key, at);
}
export async function handleSuggestions(
_req: IncomingMessage,
res: ServerResponse,
o: { path: string; method: string; role: Role; store: Store },
): Promise<boolean> {
const send = (status: number, body: unknown) => {
res.writeHead(status, { "content-type": "application/json; charset=utf-8", "cache-control": "no-store" });
res.end(JSON.stringify(body));
return true;
};
if (o.path === "/api/suggestions" && o.method === "GET") {
if (!can.readDrafts(o.role)) return send(403, { error: `The ${o.role} token cannot do this` });
return send(200, listSuggestions(o.store));
}
const m = o.path.match(/^\/api\/suggestions\/(anilist-\d{1,10})\/dismiss$/);
if (m && o.method === "POST") {
if (!can.review(o.role)) return send(403, { error: `The ${o.role} token cannot do this` });
dismissSuggestion(o.store, m[1]!);
return send(200, { ok: true });
}
return false;
}
services/content/src/sync/cache.ts
+68
View File
@@ -0,0 +1,68 @@
/**
* The sync cache: what each job last fetched for each note, as JSON, with
* when it was fetched. The site reads it through the notes API (`synced`),
* so a source being down never takes a page down: the last good values stay
* until a later run replaces them.
*
* It lives in content.db but can be rebuilt by running the jobs again.
*/
import type { DatabaseSync } from "node:sqlite";
import type { CacheEntry, SyncedValues } from "./types.ts";
type Row = { job: string; path: string; vals: string; fetched_at: string };
export class SyncCache {
constructor(private readonly db: DatabaseSync) {
db.exec(`create table if not exists sync_cache (
job text not null,
path text not null,
vals text not null,
fetched_at text not null,
primary key (job, path)
);`);
}
get(job: string, path: string): CacheEntry | null {
const r = this.db.prepare("select * from sync_cache where job = ? and path = ?").get(job, path) as Row | undefined;
return r ? { values: JSON.parse(r.vals), fetchedAt: r.fetched_at } : null;
}
set(job: string, path: string, values: SyncedValues, fetchedAt: string) {
this.db
.prepare(
`insert into sync_cache (job, path, vals, fetched_at) values (?, ?, ?, ?)
on conflict(job, path) do update set vals = excluded.vals, fetched_at = excluded.fetched_at`,
)
.run(job, path, JSON.stringify(values), fetchedAt);
}
/** Drop a job's rows for notes outside `paths`. */
retain(job: string, paths: Iterable<string>) {
const keep = new Set(paths);
const del = this.db.prepare("delete from sync_cache where job = ? and path = ?");
for (const r of this.db.prepare("select path from sync_cache where job = ?").all(job) as { path: string }[]) {
if (!keep.has(r.path)) del.run(job, r.path);
}
}
/**
* Every note's synced values, each job's merged in job-name order. Jobs
* should not set the same key for one note; if two do, the later name wins.
*/
all(): Map<string, SyncedValues> {
const out = new Map<string, SyncedValues>();
for (const r of this.db.prepare("select * from sync_cache order by job").all() as Row[]) {
out.set(r.path, { ...out.get(r.path), ...JSON.parse(r.vals) });
}
return out;
}
/** One note's synced values, merged across jobs. */
forPath(path: string): SyncedValues {
let out: SyncedValues = {};
for (const r of this.db.prepare("select * from sync_cache where path = ? order by job").all(path) as Row[]) {
out = { ...out, ...JSON.parse(r.vals) };
}
return out;
}
}
services/content/src/sync/index.ts
+27
View File
@@ -0,0 +1,27 @@
/**
* The sync jobs this service can run, and which of them are enabled. To add
* a source, write a `SyncJob` (see types.ts and docs/content-service.md,
* "Sync jobs") and add it here behind the settings it needs.
*/
import { archidektJob } from "./jobs/archidekt.ts";
import { bggJob } from "./jobs/bgg.ts";
import { coversJob } from "./jobs/covers.ts";
import { homelabJobs } from "./jobs/homelab.ts";
import type { SyncSettings } from "./settings.ts";
import type { SyncJob } from "./types.ts";
export { Scheduler } from "./scheduler.ts";
export { SyncCache } from "./cache.ts";
export { syncSettings, type SyncSettings } from "./settings.ts";
export type { SyncJob, SyncContext, SyncResult, SyncedValues } from "./types.ts";
/** Public sources need nothing but their name in SYNC_JOBS. */
export function enabledJobs(s: SyncSettings): SyncJob[] {
const common = (name: string) => ({ everyMinutes: s.minutes[name], healthcheckUrl: s.healthchecks[name] });
const jobs: SyncJob[] = [];
if (s.enabled.has("archidekt")) jobs.push(archidektJob(common("archidekt")));
if (s.enabled.has("bgg")) jobs.push(bggJob({ ...common("bgg"), token: s.bggToken }));
if (s.enabled.has("covers")) jobs.push(coversJob({ ...common("covers"), tmdbApiKey: s.tmdbApiKey }));
if (s.homelab) jobs.push(...homelabJobs(s.homelab, common));
return jobs;
}
services/content/src/sync/jobs/archidekt.ts
+133
View File
@@ -0,0 +1,133 @@
/**
* Archidekt: for every deck note with an `archidektId`, the deck's commander
* art, color identity, a short list summary and its address. Hourly.
*
* API: GET https://archidekt.com/api/decks/<id>/ (public JSON, no key). The
* legacy site used the owner's deck list (/api/decks/v3/?ownerUsername=);
* the single-deck endpoint adds the cards, so commanders and color identity
* come from the list itself.
*/
import type { SyncJob, SyncedValues } from "../types.ts";
export const ARCHIDEKT_API = "https://archidekt.com/api/decks";
const COLOR_ORDER = ["W", "U", "B", "R", "G"] as const;
const COLOR_WORDS: Record<string, string> = { white: "W", blue: "U", black: "B", red: "R", green: "G" };
/** Archidekt deckFormat ids to labels, as the legacy site had them (legacy/src/lib/archidekt.ts). */
const FORMATS: Record<number, string> = {
1: "Casual",
2: "Modern",
3: "Commander",
5: "Standard",
7: "Legacy",
10: "Pauper",
14: "Historic",
16: "Pioneer",
37: "Oathbreaker",
45: "Brawl",
};
interface RawCard {
quantity?: number;
categories?: string[] | null;
card?: {
uid?: string;
oracleCard?: { name?: string; colorIdentity?: string[] };
};
}
export interface RawDeck {
id: number;
name?: string;
deckFormat?: number;
featured?: string;
customFeatured?: string;
private?: boolean;
categories?: { name: string; includedInDeck?: boolean; isPremier?: boolean }[];
cards?: RawCard[];
}
export interface DeckValues extends SyncedValues {
deckUrl: string;
/** Art crop of the (first) commander, else the deck's featured image. */
commanderArt?: string;
/** Color identity as WUBRG letters, in that order. Empty means colorless. */
colors: string[];
commanders: string[];
/** Cards in the deck, not counting the maybeboard or other excluded categories. */
deckSize: number;
format?: string;
}
export const deckAddress = (id: number) => `https://archidekt.com/decks/${id}`;
/** Scryfall's art crop for a card by its Scryfall id (Archidekt's `uid`). */
export const scryfallArt = (uid: string) => `https://cards.scryfall.io/art_crop/front/${uid[0]}/${uid[1]}/${uid}.jpg`;
const https = (url?: string) => (url && /^https?:\/\//.test(url) ? url.replace(/^http:/, "https:") : undefined);
const colorLetter = (c: string) => COLOR_WORDS[c.toLowerCase()] ?? (COLOR_ORDER.includes(c.toUpperCase() as "W") ? c.toUpperCase() : undefined);
/** Turn one Archidekt deck response into the values the site shows. */
export function parseDeck(raw: RawDeck): DeckValues {
const excluded = new Set((raw.categories ?? []).filter((c) => c.includedInDeck === false).map((c) => c.name));
const premier = new Set((raw.categories ?? []).filter((c) => c.isPremier).map((c) => c.name));
premier.add("Commander");
const inDeck = (raw.cards ?? []).filter((c) => !(c.categories ?? []).some((cat) => excluded.has(cat)));
const commanderCards = inDeck.filter((c) => (c.categories ?? []).some((cat) => premier.has(cat)));
const identitySource = commanderCards.length ? commanderCards : inDeck;
const letters = new Set(identitySource.flatMap((c) => (c.card?.oracleCard?.colorIdentity ?? []).map(colorLetter)));
const uid = commanderCards.find((c) => c.card?.uid)?.card?.uid;
const values: DeckValues = {
deckUrl: deckAddress(raw.id),
colors: COLOR_ORDER.filter((l) => letters.has(l)),
commanders: commanderCards.map((c) => c.card?.oracleCard?.name).filter((n): n is string => !!n),
deckSize: inDeck.reduce((n, c) => n + (c.quantity ?? 1), 0),
};
const art = uid ? scryfallArt(uid) : https(raw.customFeatured) ?? https(raw.featured);
if (art) values.commanderArt = art;
const format = raw.deckFormat !== undefined ? FORMATS[raw.deckFormat] : undefined;
if (format) values.format = format;
return values;
}
export function archidektJob(o: { everyMinutes?: number; healthcheckUrl?: string; delayMs?: number } = {}): SyncJob {
return {
name: "archidekt",
everyMinutes: o.everyMinutes ?? 60,
healthcheckUrl: o.healthcheckUrl,
async run(ctx) {
const decks = ctx.notes("deck").filter((n) => typeof n.data.archidektId === "number");
let synced = 0;
let missing = 0;
let failed = 0;
const errors: string[] = [];
for (const [i, note] of decks.entries()) {
if (ctx.signal.aborted) throw ctx.signal.reason;
if (i > 0) await ctx.sleep(o.delayMs ?? 1000);
const id = note.data.archidektId as number;
try {
const res = await ctx.fetch(`${ARCHIDEKT_API}/${id}/`, { headers: { accept: "application/json" } });
if (res.status === 404 || res.status === 403) {
missing++;
ctx.log(`${note.path}: deck ${id} is private or gone (${res.status})`);
continue;
}
if (!res.ok) throw new Error(`Archidekt answered ${res.status}`);
ctx.set(note.path, parseDeck((await res.json()) as RawDeck));
synced++;
} catch (err) {
if (ctx.signal.aborted) throw ctx.signal.reason;
failed++;
errors.push(`${id}: ${err instanceof Error ? err.message : String(err)}`);
}
}
ctx.retain(decks.map((d) => d.path));
return {
ok: failed === 0,
message: failed ? `${failed} of ${decks.length} decks failed: ${errors.slice(0, 3).join("; ")}` : `${synced} decks synced`,
counts: { decks: decks.length, synced, missing, failed },
};
},
};
}
services/content/src/sync/jobs/bambuddy.ts
+111
View File
@@ -0,0 +1,111 @@
/**
* Bambuddy: filament colors in stock, for the Store's Filament options.
* Hourly. Read only.
*
* UNVERIFIED: Bambuddy's API is not documented anywhere this project has, and
* no request has been made to a real Bambuddy. The endpoint path is a setting
* (BAMBUDDY_SPOOLS_PATH) and everything that depends on the response's shape
* is in `mapBambuddySpools` below, so fixing it against the real API is a
* change to one function. Check it against a live Bambuddy before relying on
* it (docs/content-service.md, "Sync jobs").
*/
import type { SyncJob } from "../types.ts";
import { mergeOptions, normalizeHex, storeOnLines, type StockOption } from "./stock.ts";
export const BAMBUDDY_DEFAULT_PATH = "/api/v1/inventory/spools";
export interface BambuddyOptions {
url: string;
/** Path of the spool or filament inventory endpoint. */
path?: string;
/** Sent as `Authorization: Bearer`, or in `authHeader` when that is set. */
apiKey?: string;
authHeader?: string;
/** Option group the colors feed. Default Filament. */
group?: string;
/** Only these materials (like PLA, PETG), when set. */
materials?: string[];
/** A spool with less filament than this, in grams, counts as out. Default 50. */
minGrams?: number;
everyMinutes?: number;
healthcheckUrl?: string;
}
export interface Spool {
name: string;
hex?: string;
material?: string;
/** Grams left, when the source says. */
remaining?: number;
}
const pick = (o: Record<string, unknown>, keys: string[]) => {
for (const k of keys) {
const v = o[k];
if (v !== undefined && v !== null && v !== "") return v;
}
return undefined;
};
/**
* THE ONE PLACE that knows Bambuddy's response shape (a guess; see above).
* Accepts a list, or an object holding one under items, spools, data or
* results, and reads each spool's color name, hex, material and grams left
* under the common spellings. Archived or empty-named entries are skipped.
*/
export function mapBambuddySpools(json: unknown): Spool[] {
const root = json as Record<string, unknown> | unknown[];
const list = Array.isArray(root)
? root
: (["items", "spools", "data", "results"].map((k) => (root as Record<string, unknown>)?.[k]).find(Array.isArray) as unknown[] | undefined);
if (!list) throw new Error("Bambuddy answered with no list of spools");
const out: Spool[] = [];
for (const raw of list) {
if (!raw || typeof raw !== "object") continue;
const o = raw as Record<string, unknown>;
if (o.archived === true || o.deleted === true) continue;
const name = pick(o, ["color_name", "colorName", "color", "name"]);
if (typeof name !== "string" || !name.trim()) continue;
const s: Spool = { name: name.trim() };
const hex = normalizeHex(pick(o, ["color_hex", "colorHex", "hex", "rgba", "color_code"]));
if (hex) s.hex = hex;
const material = pick(o, ["material", "filament_type", "type"]);
if (typeof material === "string") s.material = material.trim();
const grams = Number(pick(o, ["remaining_weight", "remainingWeight", "weight_remaining", "remaining", "remaining_grams"]));
if (Number.isFinite(grams)) s.remaining = grams;
out.push(s);
}
return out;
}
/** Spools as Filament options: in stock when some spool of that color has enough left. */
export function filamentOptions(spools: Spool[], o: Pick<BambuddyOptions, "materials" | "minGrams"> = {}): StockOption[] {
const materials = o.materials?.length ? new Set(o.materials.map((m) => m.toLowerCase())) : null;
const min = o.minGrams ?? 50;
return mergeOptions(
spools
.filter((s) => !materials || (s.material && materials.has(s.material.toLowerCase())))
.map((s) => ({ name: s.name, ...(s.hex ? { hex: s.hex } : {}), inStock: s.remaining === undefined || s.remaining >= min })),
);
}
export function bambuddyJob(o: BambuddyOptions): SyncJob {
const group = o.group ?? "Filament";
return {
name: "bambuddy",
everyMinutes: o.everyMinutes ?? 60,
healthcheckUrl: o.healthcheckUrl,
async run(ctx) {
const headers: Record<string, string> = { accept: "application/json" };
if (o.apiKey) headers[o.authHeader?.toLowerCase() || "authorization"] = o.authHeader ? o.apiKey : `Bearer ${o.apiKey}`;
const url = `${o.url.replace(/\/$/, "")}/${(o.path ?? BAMBUDDY_DEFAULT_PATH).replace(/^\//, "")}`;
const res = await ctx.fetch(url, { headers });
if (res.status === 401 || res.status === 403) throw new Error(`Bambuddy refused the request (${res.status}); check BAMBUDDY_API_KEY`);
if (!res.ok) throw new Error(`Bambuddy answered ${res.status}; check BAMBUDDY_SPOOLS_PATH`);
const options = filamentOptions(mapBambuddySpools(await res.json()), o);
const lines = storeOnLines(ctx, "filamentStock", [{ group, options }]);
const inStock = options.filter((x) => x.inStock).length;
return { message: `${inStock} of ${options.length} colors in stock, on ${lines} lines`, counts: { colors: options.length, inStock, lines } };
},
};
}
services/content/src/sync/jobs/bgg.ts
+139
View File
@@ -0,0 +1,139 @@
/**
* BoardGameGeek: for every board-game note, its player count, play time and
* box art, from the game's `bggId`. Daily.
*
* API: XML API2 `thing`, GET https://boardgamegeek.com/xmlapi2/thing?id=1,2,3
* Up to 20 ids per request, with a pause between requests. BGG may answer
* 202 (request queued: ask again shortly) or 429 (slow down). BGG now asks
* API users to register an application; when BGG_TOKEN is set it is sent as
* `Authorization: Bearer <token>`.
*/
import type { SyncJob, SyncedValues } from "../types.ts";
export const BGG_API = "https://boardgamegeek.com/xmlapi2";
const BATCH = 20;
export interface GameValues extends SyncedValues {
minPlayers?: number;
maxPlayers?: number;
/** BGG's playing time in minutes, with its range when given. */
playTime?: number;
minPlayTime?: number;
maxPlayTime?: number;
/** BGG's box art. */
cover?: string;
}
const decode = (s: string) =>
s
.replace(/&#(\d+);/g, (_, n) => String.fromCharCode(Number(n)))
.replace(/&quot;/g, '"')
.replace(/&apos;/g, "'")
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&amp;/g, "&");
const attr = (tag: string, name: string) => {
const m = new RegExp(`\\b${name}="([^"]*)"`).exec(tag);
return m ? decode(m[1]!) : undefined;
};
/** `<name value="3"/>` as a positive number. */
const valueOf = (xml: string, el: string) => {
const m = new RegExp(`<${el}\\b[^>]*>`).exec(xml);
const n = m ? Number(attr(m[0], "value")) : NaN;
return Number.isFinite(n) && n > 0 ? n : undefined;
};
/** Parse a `thing` response into values keyed by BGG id. */
export function parseThings(xml: string): Map<number, GameValues> {
const out = new Map<number, GameValues>();
for (const m of xml.matchAll(/<item\b([^>]*)>([\s\S]*?)<\/item>/g)) {
const id = Number(attr(m[1]!, "id"));
if (!Number.isInteger(id)) continue;
const body = m[2]!;
const v: GameValues = {};
const set = (k: keyof GameValues, n: number | undefined) => {
if (n !== undefined) v[k] = n;
};
set("minPlayers", valueOf(body, "minplayers"));
set("maxPlayers", valueOf(body, "maxplayers"));
set("playTime", valueOf(body, "playingtime"));
set("minPlayTime", valueOf(body, "minplaytime"));
set("maxPlayTime", valueOf(body, "maxplaytime"));
const image = /<image>\s*([^<]+?)\s*<\/image>/.exec(body)?.[1];
if (image) {
const url = decode(image).replace(/^\/\//, "https://").replace(/^http:/, "https:");
if (url.startsWith("https://")) v.cover = url;
}
out.set(id, v);
}
return out;
}
export function bggJob(o: { everyMinutes?: number; healthcheckUrl?: string; token?: string; delayMs?: number } = {}): SyncJob {
const delay = o.delayMs ?? 5000;
return {
name: "bgg",
everyMinutes: o.everyMinutes ?? 24 * 60,
healthcheckUrl: o.healthcheckUrl,
timeoutMs: 30 * 60_000,
async run(ctx) {
const games = ctx.notes("board-game").filter((n) => typeof n.data.bggId === "number");
const byId = new Map<number, string[]>();
for (const g of games) byId.set(g.data.bggId as number, [...(byId.get(g.data.bggId as number) ?? []), g.path]);
const ids = [...byId.keys()];
const headers: Record<string, string> = { accept: "application/xml" };
if (o.token) headers.authorization = `Bearer ${o.token}`;
let synced = 0;
let failed = 0;
const errors: string[] = [];
for (let i = 0; i < ids.length; i += BATCH) {
if (i > 0) await ctx.sleep(delay);
if (ctx.signal.aborted) throw ctx.signal.reason;
const batch = ids.slice(i, i + BATCH);
const url = `${BGG_API}/thing?id=${batch.join(",")}&type=boardgame,boardgameexpansion`;
let xml: string | null = null;
try {
for (let attempt = 0; attempt < 4 && xml === null; attempt++) {
const res = await ctx.fetch(url, { headers });
if (res.status === 202) {
await ctx.sleep(delay * (attempt + 1));
continue;
}
if (res.status === 429) throw new Error("BoardGameGeek asked us to slow down (429)");
if (res.status === 401 || res.status === 403) throw new Error(`BoardGameGeek refused the request (${res.status}); set BGG_TOKEN`);
if (!res.ok) throw new Error(`BoardGameGeek answered ${res.status}`);
xml = await res.text();
}
if (xml === null) throw new Error("BoardGameGeek kept the request queued");
} catch (err) {
if (ctx.signal.aborted) throw ctx.signal.reason;
failed += batch.length;
errors.push(err instanceof Error ? err.message : String(err));
if (/429|refused/.test(errors.at(-1)!)) break;
continue;
}
const found = parseThings(xml);
for (const id of batch) {
const v = found.get(id);
if (!v) {
failed++;
errors.push(`no game ${id}`);
continue;
}
for (const path of byId.get(id) ?? []) ctx.set(path, v);
synced++;
}
}
ctx.retain(games.map((g) => g.path));
const unsynced = ids.length - synced;
return {
ok: unsynced === 0,
message: unsynced ? `${unsynced} of ${ids.length} games not synced: ${[...new Set(errors)].slice(0, 3).join("; ")}` : `${synced} games synced`,
counts: { games: ids.length, synced, failed: unsynced },
};
},
};
}
services/content/src/sync/jobs/covers.ts
+176
View File
@@ -0,0 +1,176 @@
/**
* Cover art for Library items that have no `cover` of their own:
*
* - books: Open Library search (no key);
* - films and TV: TMDB search (needs TMDB_API_KEY; skipped without it);
* - anime, manga, manhwa, light novels and webnovels: AniList (no key),
* by the note's `anilistId` when it has one, else by title.
*
* Video games have no source yet. Only the image address is stored; nothing
* is downloaded. The job looks every 15 minutes, so a new item gets its
* cover soon after it appears, and it re-checks old answers slowly: an item
* with no cover found is asked about again after a day, one with a cover
* after 30 days, at most `perRun` items a run. This follows the legacy
* scripts in legacy/scripts/fetch-{book-covers,tmdb-art,anime-art}.mjs.
*/
import type { StoredNote } from "../../store.ts";
import type { SyncContext, SyncJob } from "../types.ts";
export const OPEN_LIBRARY_SEARCH = "https://openlibrary.org/search.json";
export const OPEN_LIBRARY_COVERS = "https://covers.openlibrary.org/b";
export const TMDB_API = "https://api.themoviedb.org/3";
export const TMDB_IMAGES = "https://image.tmdb.org/t/p/w500";
export const ANILIST_API = "https://graphql.anilist.co";
const DAY = 24 * 60 * 60_000;
const RETRY_FOUND_MS = 30 * DAY;
const RETRY_MISSING_MS = DAY;
type Source = "openlibrary" | "tmdb" | "anilist";
export function sourceFor(medium: string): Source | null {
if (medium === "book") return "openlibrary";
if (medium === "movie" || medium === "TV") return "tmdb";
if (["anime", "manga", "manhwa", "light novel", "webnovel"].includes(medium)) return "anilist";
return null;
}
class RateLimited extends Error {}
// ---------------------------------------------------------------- parsing
/** Open Library search.json: the first result with a cover. */
export function parseOpenLibrary(json: unknown): string | null {
const docs = ((json as { docs?: { cover_i?: number; cover_edition_key?: string }[] })?.docs ?? []);
const withId = docs.find((d) => typeof d.cover_i === "number" && d.cover_i > 0);
if (withId) return `${OPEN_LIBRARY_COVERS}/id/${withId.cover_i}-L.jpg`;
const withEdition = docs.find((d) => d.cover_edition_key);
return withEdition ? `${OPEN_LIBRARY_COVERS}/olid/${withEdition.cover_edition_key}-L.jpg` : null;
}
/** TMDB search/movie or search/tv: the first result with a poster. */
export function parseTmdb(json: unknown): string | null {
const results = (json as { results?: { poster_path?: string | null }[] })?.results ?? [];
const hit = results.find((r) => r.poster_path);
return hit ? `${TMDB_IMAGES}${hit.poster_path}` : null;
}
/** AniList Media query: the largest cover. */
export function parseAniList(json: unknown): string | null {
const media = (json as { data?: { Media?: { coverImage?: { extraLarge?: string; large?: string } } | null } })?.data?.Media;
const url = media?.coverImage?.extraLarge ?? media?.coverImage?.large;
return url && url.startsWith("https://") ? url : null;
}
export const ANILIST_QUERY = `query ($id: Int, $search: String, $type: MediaType, $format: MediaFormat) {
Media(id: $id, search: $search, type: $type, format: $format) {
id
title { english romaji }
coverImage { extraLarge large }
}
}`;
// ---------------------------------------------------------------- lookups
async function json(ctx: SyncContext, url: string, init: RequestInit = {}) {
const res = await ctx.fetch(url, { ...init, headers: { accept: "application/json", ...(init.headers as Record<string, string>) } });
if (res.status === 429) throw new RateLimited("rate limited (429)");
if (res.status === 404) return null;
if (!res.ok) throw new Error(`answered ${res.status}`);
return res.json();
}
export async function lookup(ctx: SyncContext, note: StoredNote, tmdbKey?: string): Promise<string | null> {
const d = note.data as { medium: string; creator?: string; year?: number; anilistId?: number };
const title = note.title;
switch (sourceFor(d.medium)) {
case "openlibrary": {
const p = new URLSearchParams({ title, limit: "5", fields: "title,cover_i,cover_edition_key" });
if (d.creator) p.set("author", d.creator);
return parseOpenLibrary(await json(ctx, `${OPEN_LIBRARY_SEARCH}?${p}`));
}
case "tmdb": {
if (!tmdbKey) return null;
const kind = d.medium === "movie" ? "movie" : "tv";
const p = new URLSearchParams({ query: title, include_adult: "false" });
if (d.year) p.set(kind === "movie" ? "year" : "first_air_date_year", String(d.year));
// A v4 read access token (a JWT) goes in the header; a v3 key in the query.
const bearer = tmdbKey.startsWith("eyJ");
if (!bearer) p.set("api_key", tmdbKey);
return parseTmdb(await json(ctx, `${TMDB_API}/search/${kind}?${p}`, bearer ? { headers: { authorization: `Bearer ${tmdbKey}` } } : {}));
}
case "anilist": {
const variables: Record<string, unknown> = { type: d.medium === "anime" ? "ANIME" : "MANGA" };
if (d.anilistId) variables.id = d.anilistId;
else {
variables.search = title;
if (d.medium === "light novel" || d.medium === "webnovel") variables.format = "NOVEL";
}
return parseAniList(
await json(ctx, ANILIST_API, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ query: ANILIST_QUERY, variables }),
}),
);
}
default:
return null;
}
}
// ---------------------------------------------------------------- the job
export function coversJob(
o: { everyMinutes?: number; healthcheckUrl?: string; tmdbApiKey?: string; perRun?: number; delayMs?: number } = {},
): SyncJob {
return {
name: "covers",
everyMinutes: o.everyMinutes ?? 15,
healthcheckUrl: o.healthcheckUrl,
async run(ctx) {
const now = ctx.now().getTime();
const items = ctx.notes("library-item").filter((n) => !n.data.cover && sourceFor(String(n.data.medium)) !== null);
const usable = items.filter((n) => sourceFor(String(n.data.medium)) !== "tmdb" || o.tmdbApiKey);
// New items first, then the oldest answers that are due again.
const due = usable
.map((n) => ({ n, c: ctx.cached(n.path) }))
.filter(({ c }) => !c || now - Date.parse(c.fetchedAt) >= (c.values.cover ? RETRY_FOUND_MS : RETRY_MISSING_MS))
.sort((a, b) => (a.c ? Date.parse(a.c.fetchedAt) : 0) - (b.c ? Date.parse(b.c.fetchedAt) : 0))
.slice(0, o.perRun ?? 20);
let found = 0;
let missing = 0;
let failed = 0;
const errors: string[] = [];
for (const [i, { n, c }] of due.entries()) {
if (i > 0) await ctx.sleep(o.delayMs ?? 1000);
if (ctx.signal.aborted) throw ctx.signal.reason;
try {
const cover = await lookup(ctx, n, o.tmdbApiKey);
if (cover) {
ctx.set(n.path, { cover });
found++;
} else {
// Nothing found: keep a cover found earlier, else remember the miss.
ctx.set(n.path, c?.values.cover ? c.values : {});
missing++;
}
} catch (err) {
if (ctx.signal.aborted) throw ctx.signal.reason;
failed++;
errors.push(`${n.title}: ${err instanceof Error ? err.message : String(err)}`);
if (err instanceof RateLimited) break;
}
}
ctx.retain(items.map((n) => n.path));
return {
ok: failed === 0,
message: failed
? `${failed} lookups failed: ${errors.slice(0, 3).join("; ")}`
: `${due.length} checked: ${found} covers found, ${missing} not found`,
counts: { checked: due.length, found, missing, failed, waiting: usable.length - due.length },
};
},
};
}
services/content/src/sync/jobs/homebox.ts
+144
View File
@@ -0,0 +1,144 @@
/**
* Homebox: leather and paper materials in stock, for the Store's Leather and
* Paper options. Hourly. Read only.
*
* API (Homebox's REST API, v1):
* - `POST /api/v1/users/login` with username and password gives a token
* (already prefixed `Bearer `), sent as `Authorization`.
* - `GET /api/v1/labels` and `GET /api/v1/locations` name the ids to filter on.
* - `GET /api/v1/items?labels=<id>&page=&pageSize=` (or `locations=<id>`)
* lists the items, each with name, quantity and description.
*
* Which items count is a setting: HOMEBOX_LABELS (like `leather,paper`) and
* HOMEBOX_LOCATIONS, each entry `name` or `name=Group`. Without `=Group` the
* option group is the name capitalized (leather: Leather). An item is in
* stock when its quantity is above zero; a `#rrggbb` in its description
* becomes the swatch color.
*
* Written from Homebox's API documentation; not checked against a live
* server from here (see docs/content-service.md).
*/
import type { SyncContext, SyncJob } from "../types.ts";
import { mergeOptions, normalizeHex, storeOnLines, type StockGroup, type StockOption } from "./stock.ts";
export interface HomeboxFilter {
/** Label or location name in Homebox, matched case-insensitively. */
name: string;
/** Store option group it feeds. */
group: string;
}
export interface HomeboxOptions {
url: string;
user: string;
pass: string;
labels: HomeboxFilter[];
locations?: HomeboxFilter[];
everyMinutes?: number;
healthcheckUrl?: string;
}
/** `leather,paper=Paper goods` as filters. */
export function parseFilters(raw: string | undefined): HomeboxFilter[] {
return (raw ?? "")
.split(",")
.map((s) => s.trim())
.filter(Boolean)
.map((s) => {
const [name, group] = s.split("=").map((x) => x.trim());
return { name: name!, group: group || name!.charAt(0).toUpperCase() + name!.slice(1) };
});
}
export interface HomeboxItem {
name: string;
quantity: number;
description?: string;
}
/** `GET /api/v1/items`: `{ items, page, pageSize, total }`. */
export function parseItems(json: unknown): { items: HomeboxItem[]; total: number } {
const r = json as { items?: { name?: string; quantity?: number; description?: string; archived?: boolean }[]; total?: number };
if (!Array.isArray(r?.items)) throw new Error("Homebox answered with no list of items");
const items = r.items
.filter((i) => i && typeof i.name === "string" && i.name.trim() && i.archived !== true)
.map((i) => ({ name: i.name!.trim(), quantity: Number(i.quantity ?? 0) || 0, ...(i.description ? { description: i.description } : {}) }));
return { items, total: Number(r.total ?? items.length) };
}
/** `GET /api/v1/labels` or `/locations`: `[{ id, name }]` to a name to id map. */
export function parseNamed(json: unknown): Map<string, string> {
const list = Array.isArray(json) ? json : ((json as { items?: unknown[] })?.items ?? []);
const out = new Map<string, string>();
for (const x of list as { id?: string; name?: string }[]) if (x?.id && x.name) out.set(x.name.trim().toLowerCase(), x.id);
return out;
}
export function materialOptions(items: HomeboxItem[]): StockOption[] {
return mergeOptions(
items.map((i) => {
const hex = normalizeHex(/#[0-9a-f]{6}\b/i.exec(i.description ?? "")?.[0]);
return { name: i.name, ...(hex ? { hex } : {}), inStock: i.quantity > 0 };
}),
);
}
async function call(ctx: SyncContext, url: string, init: RequestInit = {}) {
const res = await ctx.fetch(url, init);
if (res.status === 401 || res.status === 403) throw new Error(`Homebox refused the request (${res.status}); check HOMEBOX_USER and HOMEBOX_PASS`);
if (!res.ok) throw new Error(`Homebox answered ${res.status}`);
return res.json() as Promise<unknown>;
}
export function homeboxJob(o: HomeboxOptions): SyncJob {
const api = (path: string) => `${o.url.replace(/\/$/, "")}/api/v1${path}`;
return {
name: "homebox",
everyMinutes: o.everyMinutes ?? 60,
healthcheckUrl: o.healthcheckUrl,
async run(ctx) {
const login = (await call(ctx, api("/users/login"), {
method: "POST",
headers: { "content-type": "application/json", accept: "application/json" },
body: JSON.stringify({ username: o.user, password: o.pass, stayLoggedIn: false }),
})) as { token?: string };
if (!login?.token) throw new Error("Homebox login gave no token");
const headers = { accept: "application/json", authorization: /^Bearer /i.test(login.token) ? login.token : `Bearer ${login.token}` };
const filters: { kind: "labels" | "locations"; f: HomeboxFilter }[] = [
...o.labels.map((f) => ({ kind: "labels" as const, f })),
...(o.locations ?? []).map((f) => ({ kind: "locations" as const, f })),
];
const ids = { labels: new Map<string, string>(), locations: new Map<string, string>() };
for (const kind of ["labels", "locations"] as const) {
if (filters.some((x) => x.kind === kind)) ids[kind] = parseNamed(await call(ctx, api(`/${kind}`), { headers }));
}
const byGroup = new Map<string, HomeboxItem[]>();
const unknown: string[] = [];
for (const { kind, f } of filters) {
const id = ids[kind].get(f.name.toLowerCase());
if (!id) {
unknown.push(f.name);
continue;
}
const items = byGroup.get(f.group) ?? [];
for (let page = 1; page <= 20; page++) {
const p = new URLSearchParams({ [kind]: id, page: String(page), pageSize: "100" });
const r = parseItems(await call(ctx, api(`/items?${p}`), { headers }));
items.push(...r.items);
if (r.items.length < 100 || page * 100 >= r.total) break;
}
byGroup.set(f.group, items);
}
// A filter naming nothing in Homebox is a setup mistake: fail rather than mark everything out.
if (unknown.length) throw new Error(`No Homebox label or location named ${unknown.join(", ")}`);
const groups: StockGroup[] = [...byGroup].map(([group, items]) => ({ group, options: materialOptions(items) }));
const lines = storeOnLines(ctx, "materialStock", groups);
const counts: Record<string, number> = { lines };
for (const g of groups) counts[g.group.toLowerCase()] = g.options.filter((x) => x.inStock).length;
return { message: `${groups.map((g) => `${g.group}: ${g.options.filter((x) => x.inStock).length} in stock`).join(", ")}; on ${lines} lines`, counts };
},
};
}
services/content/src/sync/jobs/homelab.ts
+67
View File
@@ -0,0 +1,67 @@
/**
* Settings and jobs for the homelab sources on Cosmos: Suwayomi and Kavita
* (PersonalReader), Bambuddy and Homebox (Store stock). Each is enabled by
* its own URL and credentials alone, not by SYNC_JOBS. All are read only
* against their source.
*/
import type { SyncJob } from "../types.ts";
import { bambuddyJob, type BambuddyOptions } from "./bambuddy.ts";
import { homeboxJob, parseFilters, type HomeboxOptions } from "./homebox.ts";
import { kavitaJob, type KavitaOptions } from "./kavita.ts";
import { suwayomiJob, type SuwayomiAuth, type SuwayomiOptions } from "./suwayomi.ts";
type Base<T> = Omit<T, "everyMinutes" | "healthcheckUrl">;
export interface HomelabSettings {
suwayomi?: Base<SuwayomiOptions>;
kavita?: Base<KavitaOptions>;
bambuddy?: Base<BambuddyOptions>;
homebox?: Base<HomeboxOptions>;
}
const list = (v: string | undefined) => (v ?? "").split(",").map((s) => s.trim()).filter(Boolean);
export function homelabSettings(env: NodeJS.ProcessEnv): HomelabSettings {
const s: HomelabSettings = {};
if (env.SUWAYOMI_URL) {
const auth = env.SUWAYOMI_AUTH?.trim().toLowerCase().replace("basic_auth", "basic");
if (auth && !["ui_login", "basic", "none"].includes(auth)) throw new Error(`SUWAYOMI_AUTH: "${auth}" is not ui_login, basic or none`);
s.suwayomi = {
url: env.SUWAYOMI_URL,
user: env.SUWAYOMI_USER || undefined,
pass: env.SUWAYOMI_PASS || undefined,
auth: (auth as SuwayomiAuth) || undefined,
};
}
if (env.KAVITA_URL && env.KAVITA_API_KEY) s.kavita = { url: env.KAVITA_URL, apiKey: env.KAVITA_API_KEY };
if (env.BAMBUDDY_URL) {
s.bambuddy = {
url: env.BAMBUDDY_URL,
path: env.BAMBUDDY_SPOOLS_PATH || undefined,
apiKey: env.BAMBUDDY_API_KEY || undefined,
authHeader: env.BAMBUDDY_AUTH_HEADER || undefined,
group: env.BAMBUDDY_GROUP || undefined,
materials: list(env.BAMBUDDY_MATERIALS),
minGrams: env.BAMBUDDY_MIN_GRAMS ? Number(env.BAMBUDDY_MIN_GRAMS) : undefined,
};
}
if (env.HOMEBOX_URL && env.HOMEBOX_USER && env.HOMEBOX_PASS) {
s.homebox = {
url: env.HOMEBOX_URL,
user: env.HOMEBOX_USER,
pass: env.HOMEBOX_PASS,
labels: parseFilters(env.HOMEBOX_LABELS ?? "leather,paper"),
locations: parseFilters(env.HOMEBOX_LOCATIONS),
};
}
return s;
}
export function homelabJobs(s: HomelabSettings, common: (name: string) => { everyMinutes?: number; healthcheckUrl?: string }): SyncJob[] {
const jobs: SyncJob[] = [];
if (s.suwayomi) jobs.push(suwayomiJob({ ...s.suwayomi, ...common("suwayomi") }));
if (s.kavita) jobs.push(kavitaJob({ ...s.kavita, ...common("kavita") }));
if (s.bambuddy) jobs.push(bambuddyJob({ ...s.bambuddy, ...common("bambuddy") }));
if (s.homebox) jobs.push(homeboxJob({ ...s.homebox, ...common("homebox") }));
return jobs;
}
services/content/src/sync/jobs/kavita.ts
+238
View File
@@ -0,0 +1,238 @@
/**
* Kavita (PersonalReader's light novels and books): reading progress for
* Library items linked to a Kavita series. Daily. Read only against Kavita.
*
* A Library item is linked only explicitly, never by title:
* - `kavitaSeriesId` on the note names the Kavita series; or
* - a book, light novel or webnovel note's `anilistId` equals the AniList id
* in a Kavita series' metadata (an anilist.co address in its web links).
*
* Auth: the API key is exchanged for a token at
* `POST /api/Plugin/authenticate?apiKey=…&pluginName=…`, sent as
* `Authorization: Bearer`. Series come from `POST /api/Series/all-v2` (only
* when an AniList match is needed) or `GET /api/Series/<id>`, metadata from
* `GET /api/Series/metadata?seriesId=`, the Want to Read list from
* `POST /api/want-to-read/v2`.
*
* State: fully read is finished (or caught up while the series is still
* ongoing), partly read is now, unread and on Want to Read is up next;
* anything else leaves the note alone. Cached per note: `bookProgress`
* (title, percent read, when). Nothing that links into Kavita.
*
* Written from Kavita's API documentation; not checked against a live server
* from here (see docs/content-service.md).
*/
import type { StoredNote } from "../../store.ts";
import type { SyncContext, SyncJob, SyncResult } from "../types.ts";
import { errorText, isoTime, MEMO_KEY, StateProposer, targetState, type LibraryState } from "./reader.ts";
export interface KavitaOptions {
url: string;
apiKey: string;
everyMinutes?: number;
healthcheckUrl?: string;
delayMs?: number;
}
/** Media that may be linked to Kavita by AniList id. */
export const KAVITA_MEDIA = ["book", "light novel", "webnovel"];
/** Kavita's PublicationStatus: OnGoing 0, Hiatus 1, Completed 2, Cancelled 3, Ended 4. */
const DONE_STATUSES = new Set([2, 4]);
export interface KavitaSeries {
id: number;
name: string;
pages: number;
pagesRead: number;
latestReadDate?: string;
}
export interface KavitaMetadata {
anilistId?: number;
publicationStatus?: number;
}
export function parseSeries(raw: unknown): KavitaSeries | null {
const s = raw as { id?: number; name?: string; localizedName?: string; pages?: number; pagesRead?: number; latestReadDate?: string };
if (typeof s?.id !== "number" || !s.name) return null;
const out: KavitaSeries = { id: s.id, name: s.localizedName || s.name, pages: Math.max(0, Number(s.pages) || 0), pagesRead: Math.max(0, Number(s.pagesRead) || 0) };
const at = isoTime(s.latestReadDate);
if (at) out.latestReadDate = at;
return out;
}
const ANILIST_LINK = /anilist\.co\/(?:manga|anime)\/(\d+)/i;
/** AniList id from the series' web links (comma separated addresses), and its publication status. */
export function parseMetadata(raw: unknown): KavitaMetadata {
const m = raw as { webLinks?: string; publicationStatus?: number };
const out: KavitaMetadata = {};
const id = ANILIST_LINK.exec(m?.webLinks ?? "")?.[1];
if (id) out.anilistId = Number(id);
if (typeof m?.publicationStatus === "number") out.publicationStatus = m.publicationStatus;
return out;
}
/** The Library state Kavita's progress means, or null to leave the note alone. */
export function kavitaState(s: KavitaSeries, meta: KavitaMetadata, wantToRead: boolean): LibraryState | null {
if (s.pages > 0 && s.pagesRead >= s.pages) return DONE_STATUSES.has(meta.publicationStatus ?? -1) ? "finished" : "caught up";
if (s.pagesRead > 0) return "now";
return wantToRead ? "up next" : null;
}
export const percentRead = (s: KavitaSeries) => (s.pages > 0 ? Math.min(100, Math.floor((s.pagesRead / s.pages) * 100)) : 0);
// ---------------------------------------------------------------- requests
const FILTER = { statements: [], combination: 1, limitTo: 0, sortOptions: { sortField: 1, isAscending: true } };
class Kavita {
private headers: Record<string, string> = {};
constructor(private readonly ctx: SyncContext, private readonly o: KavitaOptions) {}
private url = (path: string) => `${this.o.url.replace(/\/$/, "")}${path}`;
async login() {
const p = new URLSearchParams({ apiKey: this.o.apiKey, pluginName: "GregKilmer.com" });
const res = await this.ctx.fetch(this.url(`/api/Plugin/authenticate?${p}`), { method: "POST", headers: { accept: "application/json" } });
if (res.status === 401 || res.status === 403) throw new Error(`Kavita refused the API key (${res.status})`);
if (!res.ok) throw new Error(`Kavita login answered ${res.status}`);
const token = ((await res.json()) as { token?: string })?.token;
if (!token) throw new Error("Kavita login gave no token");
this.headers = { authorization: `Bearer ${token}` };
}
async call(path: string, body?: unknown): Promise<unknown> {
const res = await this.ctx.fetch(this.url(path), {
method: body === undefined ? "GET" : "POST",
headers: { accept: "application/json", ...(body === undefined ? {} : { "content-type": "application/json" }), ...this.headers },
body: body === undefined ? undefined : JSON.stringify(body),
});
if (res.status === 404) return null;
if (!res.ok) throw new Error(`Kavita answered ${res.status} for ${path.split("?")[0]}`);
return res.json();
}
async allSeries(): Promise<KavitaSeries[]> {
const out: KavitaSeries[] = [];
for (let page = 1; page <= 50; page++) {
const list = (await this.call(`/api/Series/all-v2?PageNumber=${page}&PageSize=200`, FILTER)) as unknown[] | null;
if (!Array.isArray(list) || !list.length) break;
for (const r of list) {
const s = parseSeries(r);
if (s) out.push(s);
}
if (list.length < 200) break;
}
return out;
}
async series(id: number) {
return parseSeries(await this.call(`/api/Series/${id}`));
}
async metadata(id: number) {
return parseMetadata(await this.call(`/api/Series/metadata?seriesId=${id}`));
}
async wantToRead(): Promise<Set<number>> {
try {
const list = (await this.call("/api/want-to-read/v2?PageNumber=1&PageSize=1000", FILTER)) as { id?: number }[] | null;
return new Set((list ?? []).map((s) => s.id).filter((id): id is number => typeof id === "number"));
} catch (err) {
if (this.ctx.signal.aborted) throw this.ctx.signal.reason;
this.ctx.log(`Want to Read list not read: ${errorText(err)}`);
return new Set();
}
}
}
// ---------------------------------------------------------------- the job
export function kavitaJob(o: KavitaOptions): SyncJob {
const delay = o.delayMs ?? 250;
return {
name: "kavita",
everyMinutes: o.everyMinutes ?? 24 * 60,
healthcheckUrl: o.healthcheckUrl,
timeoutMs: 30 * 60_000,
async run(ctx): Promise<SyncResult> {
const items = ctx.notes("library-item");
const byKavita = items.filter((n) => typeof n.data.kavitaSeriesId === "number");
const byAniList = items.filter(
(n) => typeof n.data.kavitaSeriesId !== "number" && typeof n.data.anilistId === "number" && KAVITA_MEDIA.includes(String(n.data.medium)),
);
if (!byKavita.length && !byAniList.length) {
ctx.retain([MEMO_KEY]);
return { message: "No Library item is linked to Kavita", counts: { linked: 0 } };
}
const k = new Kavita(ctx, o);
await k.login();
// Pairs of (note, Kavita series id), with the metadata already read.
const pairs: { note: StoredNote; id: number }[] = byKavita.map((note) => ({ note, id: note.data.kavitaSeriesId as number }));
const meta = new Map<number, KavitaMetadata>();
const known = new Map<number, KavitaSeries>();
if (byAniList.length) {
const wanted = new Set(byAniList.map((n) => n.data.anilistId as number));
for (const [i, s] of (await k.allSeries()).entries()) {
if (i > 0) await ctx.sleep(delay);
if (ctx.signal.aborted) throw ctx.signal.reason;
const m = await k.metadata(s.id);
if (!m.anilistId || !wanted.has(m.anilistId)) continue;
meta.set(s.id, m);
known.set(s.id, s);
for (const note of byAniList.filter((n) => n.data.anilistId === m.anilistId)) pairs.push({ note, id: s.id });
}
}
const wantToRead = await k.wantToRead();
const proposer = new StateProposer(ctx);
const kept: string[] = [MEMO_KEY];
let synced = 0;
let missing = 0;
let failed = 0;
const errors: string[] = [];
for (const { note, id } of pairs) {
if (ctx.signal.aborted) throw ctx.signal.reason;
kept.push(note.path);
try {
let s = known.get(id) ?? null;
if (!s) {
await ctx.sleep(delay);
s = await k.series(id);
}
if (!s) {
missing++;
ctx.log(`${note.path}: no Kavita series ${id}`);
continue;
}
const m = meta.get(id) ?? (await k.metadata(id));
const progress: Record<string, unknown> = { title: s.name, percent: percentRead(s) };
if (s.latestReadDate) progress.readAt = s.latestReadDate;
ctx.set(note.path, { bookProgress: progress });
synced++;
proposer.propose(note, targetState(String(note.data.state), kavitaState(s, m, wantToRead.has(id))));
} catch (err) {
if (ctx.signal.aborted) throw ctx.signal.reason;
failed++;
errors.push(`${note.title}: ${errorText(err)}`);
}
}
proposer.save(kept);
ctx.retain(kept);
const c = proposer.counts;
const problems = [...errors, ...proposer.errors];
return {
ok: problems.length === 0,
message: problems.length
? `${problems.length} problems: ${problems.slice(0, 3).join("; ")}`
: `${synced} series synced: ${c.accepted} updated, ${c.held} held for review`,
counts: { linked: pairs.length, synced, missing, failed, accepted: c.accepted, held: c.held, waiting: c.waiting, rejected: c.rejected },
};
},
};
}
services/content/src/sync/jobs/reader.ts
+123
View File
@@ -0,0 +1,123 @@
/**
* What the two PersonalReader jobs (Suwayomi and Kavita) share: the state
* mapping, proposing a state change through `ctx.apply` without nagging, and
* the job-private cache entries.
*
* PersonalReader is private. A job updates a Library item only when the note
* carries the series' id (`anilistId`, or `kavitaSeriesId` for Kavita), and it
* stores nothing that names a source site, a scan group or a link into the
* reader: only title, chapter or percent, when it was read, and an AniList
* cover.
*/
import { libraryStates } from "@gk/schema";
import type { StoredNote } from "../../store.ts";
import type { SyncContext, SyncedValues } from "../types.ts";
export type LibraryState = (typeof libraryStates)[number];
/**
* Job-private cache entries. They are keyed like note paths but are not notes
* (no `.md`), so the notes API never shows them to the site.
*/
export const MEMO_KEY = "_memo";
export const SUGGESTIONS_KEY = "_suggestions";
/** Suwayomi category to Library state. `null`: leave the note alone (Dropped). */
const CATEGORY_STATES: Record<string, LibraryState | null> = {
reading: "now",
"plan to read": "up next",
completed: "finished",
dropped: null,
};
/** The Library state a reader category maps to; undefined for a category the site does not know. */
export function stateForCategory(category: string): LibraryState | null | undefined {
const key = category.trim().toLowerCase().replace(/\s+/g, " ");
return key in CATEGORY_STATES ? CATEGORY_STATES[key] : undefined;
}
/**
* The state to propose for a note, or null to leave it. Two refinements keep
* a distinction the reader cannot make: "caught up" (read to the latest of an
* ongoing series) is not turned back into "now", and "finished" is not
* turned into "caught up".
*/
export function targetState(current: string, mapped: LibraryState | null | undefined): LibraryState | null {
if (!mapped || mapped === current) return null;
if (mapped === "now" && current === "caught up") return null;
if (mapped === "caught up" && current === "finished") return null;
return mapped;
}
export type ProposeResult = "accepted" | "held" | "waiting" | "rejected" | "skipped";
interface Memo {
/** The state last proposed for each note path. */
proposed: Record<string, string>;
}
/**
* Proposes state changes, once each. A change Greg rejected is not proposed
* again until the reader says something different, so the review queue is not
* refilled every hour.
*/
export class StateProposer {
private readonly memo: Memo;
readonly counts = { accepted: 0, held: 0, waiting: 0, rejected: 0 };
readonly errors: string[] = [];
constructor(private readonly ctx: SyncContext) {
const m = ctx.cached(MEMO_KEY)?.values as Partial<Memo> | undefined;
this.memo = { proposed: { ...(m?.proposed ?? {}) } };
}
propose(note: StoredNote, target: LibraryState | null): ProposeResult {
const path = note.path;
if (!target) {
delete this.memo.proposed[path];
return "skipped";
}
if (this.memo.proposed[path] === target) return "skipped";
const o = this.ctx.apply({ op: "update", path, fields: { state: target } });
if (o.status === "accepted" || o.status === "unchanged") {
this.counts.accepted++;
this.memo.proposed[path] = target;
return "accepted";
}
if (o.status === "held") {
this.counts.held++;
this.memo.proposed[path] = target;
return "held";
}
// A change already waits for review: try again next run.
if (o.reasons.some((r) => r.code === "waiting")) {
this.counts.waiting++;
return "waiting";
}
this.counts.rejected++;
this.errors.push(`${note.title}: ${o.reasons.map((r) => r.message).join("; ")}`);
return "rejected";
}
/** Save the memo, forgetting notes no longer in scope. */
save(paths: Iterable<string>) {
const keep = new Set(paths);
const proposed = Object.fromEntries(Object.entries(this.memo.proposed).filter(([p]) => keep.has(p)));
this.ctx.set(MEMO_KEY, { proposed } satisfies SyncedValues);
}
}
/** Epoch seconds or milliseconds (or an ISO string) as ISO; undefined for zero or junk. */
export function isoTime(v: unknown): string | undefined {
if (v === null || v === undefined || v === "") return undefined;
if (typeof v === "string" && !/^\d+$/.test(v)) {
const t = Date.parse(v);
// Kavita says 0001-01-01 for "never".
return Number.isFinite(t) && t > 0 ? new Date(t).toISOString() : undefined;
}
const n = Number(v);
if (!Number.isFinite(n) || n <= 0) return undefined;
return new Date(n < 1e12 ? n * 1000 : n).toISOString();
}
export const errorText = (err: unknown) => (err instanceof Error ? err.message : String(err));
services/content/src/sync/jobs/stock.ts
+64
View File
@@ -0,0 +1,64 @@
/**
* What the two Store stock jobs (Bambuddy filament, Homebox leather and
* paper) share: the synced shape and how it lands on product lines.
*
* A job reads what is in stock and stores it on every product-line note that
* has an option in a group the job feeds (Filament for Bambuddy; the groups
* named in HOMEBOX_LABELS for Homebox). The site lays it under the note's own
* options (services/site/src/lib/store.ts): a note's option always wins, a
* note option that leaves `inStock` or `hex` out takes the synced value, and
* in-stock colors the note does not list are added after the note's own.
*/
import type { SyncContext, SyncedValues } from "../types.ts";
export interface StockOption {
name: string;
hex?: string;
inStock: boolean;
}
export interface StockGroup {
group: string;
options: StockOption[];
}
/** `#rrggbb` from `rrggbb`, `#RRGGBB` or `#rrggbbaa`; undefined otherwise. */
export function normalizeHex(v: unknown): string | undefined {
if (typeof v !== "string") return undefined;
const m = /^#?([0-9a-f]{6})(?:[0-9a-f]{2})?$/i.exec(v.trim());
return m ? `#${m[1]!.toLowerCase()}` : undefined;
}
/** One entry per name (case-insensitive): in stock when any entry is, keeping the first hex seen. */
export function mergeOptions(options: StockOption[]): StockOption[] {
const out = new Map<string, StockOption>();
for (const o of options) {
const key = o.name.trim().toLowerCase();
if (!key) continue;
const prev = out.get(key);
if (!prev) out.set(key, { ...o, name: o.name.trim() });
else {
prev.inStock ||= o.inStock;
prev.hex ??= o.hex;
}
}
return [...out.values()].sort((a, b) => Number(b.inStock) - Number(a.inStock) || a.name.localeCompare(b.name));
}
/**
* Store `groups` under `key` on every product line that uses one of them,
* forgetting lines that no longer do. Returns how many lines were updated.
*/
export function storeOnLines(ctx: SyncContext, key: string, groups: StockGroup[]): number {
const fed = new Map(groups.map((g) => [g.group.toLowerCase(), g]));
const kept: string[] = [];
for (const line of ctx.notes("product-line")) {
const used = new Set(((line.data.options as { group?: string }[] | undefined) ?? []).map((o) => String(o.group ?? "").toLowerCase()));
const mine = [...used].map((g) => fed.get(g)).filter((g): g is StockGroup => !!g);
if (!mine.length) continue;
ctx.set(line.path, { [key]: mine } satisfies SyncedValues);
kept.push(line.path);
}
ctx.retain(kept);
return kept.length;
}
services/content/src/sync/jobs/suwayomi.ts
+313
View File
@@ -0,0 +1,313 @@
/**
* Suwayomi (PersonalReader's manga and manhwa): reading progress for Library
* items that carry the series' `anilistId`. Hourly. Read only against
* Suwayomi.
*
* - Logs in with its own read-only account. With Suwayomi's `ui_login` auth
* mode that is the GraphQL `login` mutation, whose access token is sent as
* `Authorization: Bearer`; with `basic_auth` it is HTTP basic auth.
* - Reads the library over GraphQL (`<SUWAYOMI_URL>/api/graphql`): each
* category (Reading, Plan to read, Completed, Dropped) with its series, the
* furthest chapter read and when, and the AniList tracker record, whose
* remote id is the AniList id.
* - Maps the category to the note's state (Reading: now, Plan to read: up
* next, Completed: finished, Dropped: left alone) through `ctx.apply` as
* `sync`; Greg's notes are held for review.
* - Caches `chapterProgress` (title, chapter, when) and the AniList `cover`.
* Never a source name, scan group, thumbnail or any Suwayomi address.
* - A series in Reading with no Library item becomes a suggestion on the
* admin page.
*
* Written from Suwayomi-Server's GraphQL schema; not checked against a live
* server from here (see docs/content-service.md).
*/
import type { StoredNote } from "../../store.ts";
import type { SyncContext, SyncJob } from "../types.ts";
import { ANILIST_API } from "./covers.ts";
import { errorText, isoTime, StateProposer, stateForCategory, SUGGESTIONS_KEY, MEMO_KEY, targetState } from "./reader.ts";
/** Suwayomi's (and Mihon's) tracker id for AniList. */
export const ANILIST_TRACKER_ID = 2;
export type SuwayomiAuth = "ui_login" | "basic" | "none";
export interface SuwayomiOptions {
url: string;
user?: string;
pass?: string;
/** Default `ui_login` when a user is set, else `none`. */
auth?: SuwayomiAuth;
everyMinutes?: number;
healthcheckUrl?: string;
}
export const LOGIN_MUTATION = `mutation GkLogin($username: String!, $password: String!) {
login(input: { username: $username, password: $password }) { accessToken }
}`;
/** Only what the site needs: no ids, urls, thumbnails, sources or scanlators. */
export const LIBRARY_QUERY = `query GkLibrary {
categories {
nodes {
name
mangas {
nodes {
title
inLibrary
latestReadChapter { chapterNumber lastReadAt }
trackRecords { nodes { trackerId remoteId title lastChapterRead } }
}
}
}
}
}`;
/** Batched AniList lookup for the cover and what a suggestion needs. */
export const ANILIST_BATCH_QUERY = `query ($ids: [Int]) {
Page(perPage: 50) {
media(id_in: $ids, type: MANGA) {
id
title { english romaji }
format
countryOfOrigin
coverImage { extraLarge large }
}
}
}`;
export interface Series {
title: string;
category: string;
anilistId?: number;
/** Furthest chapter read. */
chapter?: number;
readAt?: string;
}
export interface ChapterProgress {
title: string;
/** Library state the category maps to, or the category itself for Dropped. */
status: string;
chapter?: number;
readAt?: string;
}
export interface AniListInfo {
title?: string;
medium: "manga" | "manhwa" | "light novel";
cover?: string;
}
const CATEGORY_ORDER = ["reading", "plan to read", "completed", "dropped"];
interface RawChapter {
chapterNumber?: number | string | null;
lastReadAt?: number | string | null;
}
interface RawManga {
title?: string;
inLibrary?: boolean;
latestReadChapter?: RawChapter | null;
trackRecords?: { nodes?: { trackerId?: number; remoteId?: number | string; title?: string; lastChapterRead?: number | null }[] } | null;
}
interface RawLibrary {
data?: { categories?: { nodes?: { name?: string; mangas?: { nodes?: RawManga[] } | null }[] } | null } | null;
errors?: { message?: string }[];
}
const positive = (v: unknown) => {
const n = Number(v);
return Number.isFinite(n) && n > 0 ? n : undefined;
};
/**
* The library response as series. A series in several known categories takes
* the first of Reading, Plan to read, Completed, Dropped; series only in
* categories the site does not know are left out.
*/
export function parseLibrary(json: unknown): Series[] {
const raw = json as RawLibrary;
if (raw?.errors?.length) throw new Error(`Suwayomi: ${raw.errors.map((e) => e.message).join("; ")}`);
const cats = raw?.data?.categories?.nodes;
if (!Array.isArray(cats)) throw new Error("Suwayomi answered without a library");
const best = new Map<string, Series & { rank: number }>();
for (const cat of cats) {
const name = (cat.name ?? "").trim();
const rank = CATEGORY_ORDER.indexOf(name.toLowerCase().replace(/\s+/g, " "));
if (rank < 0 || stateForCategory(name) === undefined) continue;
for (const m of cat.mangas?.nodes ?? []) {
if (m.inLibrary === false || !m.title) continue;
const track = (m.trackRecords?.nodes ?? []).find((t) => t.trackerId === ANILIST_TRACKER_ID);
const anilistId = positive(track?.remoteId);
const s: Series & { rank: number } = { title: (track?.title || m.title).trim(), category: name, rank };
if (anilistId && Number.isInteger(anilistId)) s.anilistId = anilistId;
const chapter = positive(m.latestReadChapter?.chapterNumber) ?? positive(track?.lastChapterRead);
if (chapter !== undefined) s.chapter = chapter;
const readAt = isoTime(m.latestReadChapter?.lastReadAt);
if (readAt) s.readAt = readAt;
// Key by AniList id when there is one, else by title.
const key = anilistId ? `a${anilistId}` : `t${m.title}`;
const prev = best.get(key);
if (!prev || rank < prev.rank) best.set(key, s);
}
}
return [...best.values()].map(({ rank: _, ...s }) => s);
}
/** AniList's batched Media answer, by id. */
export function parseAniListBatch(json: unknown): Map<number, AniListInfo> {
type M = { id: number; title?: { english?: string; romaji?: string }; format?: string; countryOfOrigin?: string; coverImage?: { extraLarge?: string; large?: string } };
const media = (json as { data?: { Page?: { media?: M[] } } })?.data?.Page?.media ?? [];
const out = new Map<number, AniListInfo>();
for (const m of media) {
if (typeof m?.id !== "number") continue;
const cover = m.coverImage?.extraLarge ?? m.coverImage?.large;
const info: AniListInfo = { medium: m.format === "NOVEL" ? "light novel" : m.countryOfOrigin === "KR" ? "manhwa" : "manga" };
const title = m.title?.english || m.title?.romaji;
if (title) info.title = title;
if (cover?.startsWith("https://")) info.cover = cover;
out.set(m.id, info);
}
return out;
}
// ---------------------------------------------------------------- requests
const endpoint = (base: string) => `${base.replace(/\/$/, "")}/api/graphql`;
async function graphql(ctx: SyncContext, url: string, headers: Record<string, string>, query: string, variables?: Record<string, unknown>) {
const res = await ctx.fetch(url, {
method: "POST",
headers: { "content-type": "application/json", accept: "application/json", ...headers },
body: JSON.stringify({ query, variables }),
});
if (res.status === 401 || res.status === 403) throw new Error(`Suwayomi refused the login (${res.status})`);
if (!res.ok) throw new Error(`Suwayomi answered ${res.status}`);
return res.json() as Promise<unknown>;
}
/** The headers that authenticate one run. */
export async function authHeaders(ctx: SyncContext, o: SuwayomiOptions): Promise<Record<string, string>> {
const mode = o.auth ?? (o.user ? "ui_login" : "none");
if (mode === "none") return {};
if (!o.user || !o.pass) throw new Error("SUWAYOMI_USER and SUWAYOMI_PASS are required for this auth mode");
if (mode === "basic") return { authorization: `Basic ${Buffer.from(`${o.user}:${o.pass}`).toString("base64")}` };
const json = (await graphql(ctx, endpoint(o.url), {}, LOGIN_MUTATION, { username: o.user, password: o.pass })) as {
data?: { login?: { accessToken?: string } };
errors?: { message?: string }[];
};
const token = json?.data?.login?.accessToken;
if (!token) throw new Error(`Suwayomi login failed${json?.errors?.length ? `: ${json.errors.map((e) => e.message).join("; ")}` : ""}`);
return { authorization: `Bearer ${token}` };
}
async function aniList(ctx: SyncContext, ids: number[]): Promise<Map<number, AniListInfo>> {
const out = new Map<number, AniListInfo>();
for (let i = 0; i < ids.length; i += 50) {
if (i > 0) await ctx.sleep(1000);
const res = await ctx.fetch(ANILIST_API, {
method: "POST",
headers: { "content-type": "application/json", accept: "application/json" },
body: JSON.stringify({ query: ANILIST_BATCH_QUERY, variables: { ids: ids.slice(i, i + 50) } }),
});
if (!res.ok) throw new Error(`AniList answered ${res.status}`);
for (const [k, v] of parseAniListBatch(await res.json())) out.set(k, v);
}
return out;
}
// ---------------------------------------------------------------- the job
export interface Suggestion {
anilistId: number;
title: string;
medium: AniListInfo["medium"];
cover?: string;
}
export function suwayomiJob(o: SuwayomiOptions): SyncJob {
return {
name: "suwayomi",
everyMinutes: o.everyMinutes ?? 60,
healthcheckUrl: o.healthcheckUrl,
async run(ctx) {
const headers = await authHeaders(ctx, o);
const series = parseLibrary(await graphql(ctx, endpoint(o.url), headers, LIBRARY_QUERY));
const items = ctx.notes("library-item");
const byAniList = new Map<number, StoredNote[]>();
for (const n of items) {
const id = n.data.anilistId;
if (typeof id === "number") byAniList.set(id, [...(byAniList.get(id) ?? []), n]);
}
// AniList: covers for matched series, titles and media for suggestions.
const tracked = series.filter((s) => s.anilistId);
const wanted = tracked.filter((s) => byAniList.has(s.anilistId!) || stateForCategory(s.category) === "now").map((s) => s.anilistId!);
let info = new Map<number, AniListInfo>();
let anilistError: string | null = null;
try {
if (wanted.length) info = await aniList(ctx, [...new Set(wanted)]);
} catch (err) {
if (ctx.signal.aborted) throw ctx.signal.reason;
anilistError = errorText(err);
}
const proposer = new StateProposer(ctx);
const kept: string[] = [MEMO_KEY, SUGGESTIONS_KEY];
const suggestions: Suggestion[] = [];
const oldSuggestions = (ctx.cached(SUGGESTIONS_KEY)?.values.items as Suggestion[] | undefined) ?? [];
let matched = 0;
for (const s of series) {
const mapped = stateForCategory(s.category);
const notes = s.anilistId ? (byAniList.get(s.anilistId) ?? []) : [];
if (!notes.length) {
if (mapped === "now" && s.anilistId) {
const a = info.get(s.anilistId);
const old = oldSuggestions.find((x) => x.anilistId === s.anilistId);
const sug: Suggestion = { anilistId: s.anilistId, title: a?.title ?? old?.title ?? s.title, medium: a?.medium ?? old?.medium ?? "manga" };
const cover = a?.cover ?? old?.cover;
if (cover) sug.cover = cover;
suggestions.push(sug);
}
continue;
}
for (const note of notes) {
matched++;
kept.push(note.path);
const progress: Record<string, unknown> = { title: s.title, status: mapped ?? s.category.toLowerCase() };
if (s.chapter !== undefined) progress.chapter = s.chapter;
if (s.readAt) progress.readAt = s.readAt;
// Keep the last good cover when AniList could not be asked.
const cover = info.get(s.anilistId!)?.cover ?? (ctx.cached(note.path)?.values.cover as string | undefined);
ctx.set(note.path, cover ? { chapterProgress: progress, cover } : { chapterProgress: progress });
proposer.propose(note, targetState(String(note.data.state), mapped));
}
}
ctx.set(SUGGESTIONS_KEY, { items: suggestions });
proposer.save(kept);
ctx.retain(kept);
const c = proposer.counts;
const problems = [...(anilistError ? [`AniList: ${anilistError}`] : []), ...proposer.errors];
return {
ok: problems.length === 0,
message: problems.length
? problems.slice(0, 3).join("; ")
: `${matched} series synced: ${c.accepted} updated, ${c.held} held for review, ${suggestions.length} suggested`,
counts: {
series: series.length,
matched,
untracked: series.length - tracked.length,
accepted: c.accepted,
held: c.held,
waiting: c.waiting,
rejected: c.rejected,
suggestions: suggestions.length,
},
};
},
};
}
services/content/src/sync/scheduler.ts
+180
View File
@@ -0,0 +1,180 @@
/**
* Runs each sync job on its interval: once shortly after start (staggered so
* the jobs do not all hit the network together), then every `everyMinutes`.
* A job never runs twice at once: a tick that finds it still running is
* skipped. A run longer than its timeout is aborted and reported as failed.
*
* Every run is reported to sync_runs (shown on /api/syncs and the admin
* Overview) and pinged to the job's Healthchecks URL when it has one.
*/
import type { ContentService } from "../service.ts";
import type { SyncCache } from "./cache.ts";
import type { SyncContext, SyncJob, SyncResult } from "./types.ts";
export const SYNC_USER_AGENT = "GregKilmer.com content sync (+https://gregkilmer.com)";
const DEFAULT_TIMEOUT_MS = 10 * 60_000;
export interface SchedulerOptions {
/** Delay between the jobs' first runs. Default 20 seconds. */
staggerMs?: number;
/** Injectable for tests. */
fetch?: typeof fetch;
now?: () => Date;
log?: (message: string) => void;
}
export interface RunReport {
job: string;
ok: boolean;
message: string;
counts: Record<string, number>;
/** True when the run was skipped because the job was already running. */
skipped?: boolean;
}
export class Scheduler {
readonly cache: SyncCache;
private readonly running = new Map<string, Promise<RunReport>>();
private readonly timers = new Set<NodeJS.Timeout>();
private readonly stopping = new AbortController();
private readonly fetchFn: typeof fetch;
private readonly now: () => Date;
private readonly log: (message: string) => void;
constructor(
private readonly service: ContentService,
readonly jobs: SyncJob[],
private readonly o: SchedulerOptions = {},
) {
this.cache = service.store.syncCache;
this.fetchFn = o.fetch ?? fetch;
this.now = o.now ?? (() => new Date());
this.log = o.log ?? ((m) => console.log(m));
}
/** Run every job once, staggered, then on its interval. */
start() {
const stagger = this.o.staggerMs ?? 20_000;
this.jobs.forEach((job, i) => {
const loop = () => {
void this.runNow(job.name);
this.later(loop, job.everyMinutes * 60_000);
};
this.later(loop, i * stagger);
});
}
stop() {
for (const t of this.timers) clearTimeout(t);
this.timers.clear();
this.stopping.abort();
}
/** Run one job now, unless it is already running. */
runNow(name: string): Promise<RunReport> {
const job = this.jobs.find((j) => j.name === name);
if (!job) return Promise.resolve({ job: name, ok: false, message: "No such job", counts: {} });
if (this.running.has(name)) {
return Promise.resolve({ job: name, ok: false, message: "Already running", counts: {}, skipped: true });
}
const run = this.execute(job);
this.running.set(name, run);
// Released only when the job's promise settles, even after a timeout, so
// a job that ignores its abort signal still never overlaps itself.
return run;
}
isRunning(name: string) {
return this.running.has(name);
}
private later(fn: () => void, ms: number) {
const t = setTimeout(() => {
this.timers.delete(t);
fn();
}, ms);
t.unref?.();
this.timers.add(t);
}
private async execute(job: SyncJob): Promise<RunReport> {
const abort = new AbortController();
const onStop = () => abort.abort(new Error("The service is stopping"));
this.stopping.signal.addEventListener("abort", onStop);
const timeoutMs = job.timeoutMs ?? DEFAULT_TIMEOUT_MS;
this.ping(job, "/start");
let timer: NodeJS.Timeout | undefined;
const work = job.run(this.context(job, abort.signal));
const timedOut = new Promise<never>((_, reject) => {
timer = setTimeout(() => {
const err = new Error(`Timed out after ${Math.round(timeoutMs / 1000)} seconds`);
abort.abort(err);
reject(err);
}, timeoutMs);
});
let report: RunReport;
try {
const result: SyncResult = await Promise.race([work, timedOut]);
const ok = result.ok !== false;
report = { job: job.name, ok, message: result.message ?? (ok ? "OK" : "Some items failed"), counts: result.counts ?? {} };
} catch (err) {
report = { job: job.name, ok: false, message: err instanceof Error ? err.message : String(err), counts: {} };
} finally {
clearTimeout(timer);
this.stopping.signal.removeEventListener("abort", onStop);
}
this.service.store.reportSync({ ...report, message: report.message.slice(0, 2000), at: this.now().toISOString() });
this.ping(job, report.ok ? "" : "/fail", report.message);
if (!report.ok) this.log(`Sync ${job.name} failed: ${report.message}`);
// Hold the running flag until the job itself has finished.
work.then(
() => this.running.delete(job.name),
() => this.running.delete(job.name),
);
return report;
}
private context(job: SyncJob, signal: AbortSignal): SyncContext {
const { service, cache } = this;
return {
signal,
now: this.now,
notes: (type) => service.store.list({ type }),
cached: (path) => cache.get(job.name, path),
set: (path, values) => cache.set(job.name, path, values, this.now().toISOString()),
retain: (paths) => cache.retain(job.name, paths),
apply: (change) => service.apply(change, "sync"),
fetch: (url, init = {}) => {
const { timeoutMs = 30_000, headers, ...rest } = init;
return this.fetchFn(url, {
...rest,
headers: { "user-agent": SYNC_USER_AGENT, ...(headers as Record<string, string> | undefined) },
signal: AbortSignal.any([signal, AbortSignal.timeout(timeoutMs)]),
});
},
sleep: (ms) =>
new Promise<void>((resolve) => {
if (signal.aborted) return resolve();
const t = setTimeout(done, ms);
function done() {
clearTimeout(t);
signal.removeEventListener("abort", done);
resolve();
}
signal.addEventListener("abort", done);
}),
log: (m) => this.log(`Sync ${job.name}: ${m}`),
};
}
/** Healthchecks: `<url>/start`, `<url>` on success, `<url>/fail` with the message. */
private ping(job: SyncJob, suffix: string, body?: string) {
if (!job.healthcheckUrl) return;
const url = job.healthcheckUrl.replace(/\/$/, "") + suffix;
this.fetchFn(url, { method: "POST", body: body ?? "", signal: AbortSignal.timeout(10_000) }).catch(() => {});
}
}
services/content/src/sync/settings.ts
+46
View File
@@ -0,0 +1,46 @@
/**
* Sync settings from the environment. A job is enabled only when its config
* is present: the public sources (no credentials) are named in SYNC_JOBS;
* jobs that need a URL or a login are enabled by those variables alone.
*/
import { homelabSettings, type HomelabSettings } from "./jobs/homelab.ts";
export interface SyncSettings {
/** Public jobs switched on in SYNC_JOBS, e.g. "archidekt,bgg,covers". */
enabled: Set<string>;
/** SYNC_<JOB>_MINUTES overrides a job's interval. */
minutes: Record<string, number>;
/** SYNC_<JOB>_HEALTHCHECK_URL. */
healthchecks: Record<string, string>;
bggToken?: string;
tmdbApiKey?: string;
/** Suwayomi, Kavita, Bambuddy and Homebox: enabled by their own URL and login. */
homelab: HomelabSettings;
}
const envKey = (job: string) => job.toUpperCase().replace(/[^A-Z0-9]/g, "_");
export function syncSettings(env: NodeJS.ProcessEnv = process.env): SyncSettings {
const enabled = new Set((env.SYNC_JOBS ?? "").split(",").map((s) => s.trim().toLowerCase()).filter(Boolean));
const minutes: Record<string, number> = {};
const healthchecks: Record<string, string> = {};
for (const [k, v] of Object.entries(env)) {
if (!v) continue;
const m = /^SYNC_([A-Z0-9_]+)_(MINUTES|HEALTHCHECK_URL)$/.exec(k);
if (!m) continue;
const job = m[1]!.toLowerCase().replace(/_/g, "-");
if (m[2] === "MINUTES" && Number(v) > 0) minutes[job] = Number(v);
if (m[2] === "HEALTHCHECK_URL") healthchecks[job] = v;
}
return {
enabled,
minutes,
healthchecks,
bggToken: env.BGG_TOKEN || undefined,
tmdbApiKey: env.TMDB_API_KEY || undefined,
homelab: homelabSettings(env),
};
}
/** The env var name for one job's setting, for docs and messages. */
export const syncEnvName = (job: string, what: "MINUTES" | "HEALTHCHECK_URL") => `SYNC_${envKey(job)}_${what}`;
services/content/src/sync/types.ts
+73
View File
@@ -0,0 +1,73 @@
/**
* The sync job interface. A job reads notes, asks one outside source about
* them, and either stores what it learned in the sync cache (shown on the
* site under the note's own values) or, for jobs that must change a note,
* proposes the change through `ContentService.apply` as `sync`.
*
* A job never touches the vault, the store's tables or another job's cache
* directly: everything goes through the context, so a sync can do no more
* than an agent can. See docs/content-service.md, "Sync jobs".
*/
import type { Change, Outcome } from "../service.ts";
import type { StoredNote } from "../store.ts";
/** Synced values for one note: plain JSON. */
export type SyncedValues = Record<string, unknown>;
export interface CacheEntry {
values: SyncedValues;
/** When these values were fetched. */
fetchedAt: string;
}
export interface SyncContext {
/** Aborted when the run times out or the service stops. Pass it to every request. */
readonly signal: AbortSignal;
/** The current time (injectable for tests). */
now(): Date;
/** Live notes of one type, published or not. Read only. */
notes(type: string): StoredNote[];
/** This job's cached values for a note, if any. */
cached(path: string): CacheEntry | null;
/**
* Store values fetched for a note, replacing this job's previous values for
* it. Only call this with values that were fetched successfully: whatever
* is not set keeps its last good values.
*/
set(path: string, values: SyncedValues): void;
/** Forget this job's values for every note not in `paths` (notes that left the job's scope). */
retain(paths: Iterable<string>): void;
/**
* Propose a note change as `sync`. It runs the same rules as an agent's
* change: contact details, secrets, text loss, notes the sync did not write
* (unless `agentEdits: true`), publishing and the batch limit all hold it
* for review. A sync cannot delete, move or rename a note.
*/
apply(change: Change): Outcome;
/** fetch with the sync User-Agent, the run's abort signal and a per-request timeout. */
fetch(url: string, init?: RequestInit & { timeoutMs?: number }): Promise<Response>;
/** A polite pause between requests; ends early if the run is aborted. */
sleep(ms: number): Promise<void>;
log(message: string): void;
}
export interface SyncResult {
/** Shown on the admin Overview. */
message?: string;
counts?: Record<string, number>;
/** Set false when the run finished but some items failed; the run is reported as failed. */
ok?: boolean;
}
export interface SyncJob {
/** Lowercase id, as in sync_runs and the cache: `archidekt`, `bgg`, `covers`. */
name: string;
/** Minutes between runs. */
everyMinutes: number;
/** A run longer than this is aborted and reported as failed. Default 10 minutes. */
timeoutMs?: number;
/** Healthchecks ping URL: `/start` before, the URL on success, `/fail` on failure. */
healthcheckUrl?: string;
/** One run. Throwing marks the run failed; the cache keeps its last good values. */
run(ctx: SyncContext): Promise<SyncResult>;
}
services/content/test/submissions.test.ts
+145
View File
@@ -0,0 +1,145 @@
import { readdirSync } from "node:fs";
import type { AddressInfo } from "node:net";
import { join } from "node:path";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { parseTokens } from "../src/config.ts";
import { createHttpServer } from "../src/http.ts";
import { extensionOf, safeName } from "../src/submissions.ts";
import { book, rig, type Rig } from "./helpers.ts";
const T = { reader: "r".repeat(24), agent: "a".repeat(24), editor: "e".repeat(24), admin: "d".repeat(24), intake: "i".repeat(24) };
const MAX = 1024;
let r: Rig;
let base: string;
let server: ReturnType<typeof createHttpServer>;
beforeAll(async () => {
r = rig();
r.put(book("Mistborn"));
r.service.scan();
server = createHttpServer(r.service, {
tokens: parseTokens(Object.entries(T).map(([k, v]) => `${k}:${v}`).join(",")),
submissionMaxBytes: MAX,
});
await new Promise<void>((ok) => server.listen(0, "127.0.0.1", ok));
base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
afterAll(() => {
server.close();
r.done();
});
const call = (method: string, path: string, token: string, body?: unknown) =>
fetch(base + path, {
method,
headers: { authorization: `Bearer ${token}`, ...(body !== undefined ? { "content-type": "application/json" } : {}) },
body: body !== undefined ? JSON.stringify(body) : undefined,
});
const inquiry = (extra: Record<string, unknown> = {}) => ({
kind: "inquiry",
type: "job",
name: "Jane Doe",
email: "jane@example.com",
fields: [{ label: "Details", value: "A role on the platform team." }],
...extra,
});
describe("submissions", () => {
it("lets intake file a submission and nothing else", async () => {
const res = await call("POST", "/api/submissions", T.intake, inquiry());
expect(res.status).toBe(201);
const { id, receipt } = (await res.json()) as { id: number; receipt: string };
expect(id).toBeGreaterThan(0);
expect(receipt.length).toBeGreaterThan(20);
for (const path of ["/api/submissions", `/api/submissions/${id}`, `/api/submissions/${id}/attachment`, "/api/notes", "/api/note?type=library-item&slug=mistborn", "/api/schema", "/api/redirects", "/api/status", "/api/held", "/attachments/x.png"]) {
expect((await call("GET", path, T.intake)).status, path).toBe(403);
}
expect((await call("POST", `/api/submissions/${id}/handled`, T.intake, { handled: true })).status).toBe(403);
expect((await call("POST", "/api/changes", T.intake, { op: "create", data: book("X") })).status).toBe(403);
expect((await call("POST", "/api/syncs", T.intake, { job: "x", ok: true })).status).toBe(403);
expect((await call("POST", "/mcp", T.intake, {})).status).toBe(403);
});
it("keeps readers, agents and editors out of the saved copies", async () => {
for (const t of [T.reader, T.agent, T.editor]) {
expect((await call("GET", "/api/submissions", t)).status).toBe(403);
expect((await call("POST", "/api/submissions", t, inquiry())).status).toBe(403);
expect((await call("GET", "/api/submissions/1", t)).status).toBe(403);
}
expect((await call("GET", "/api/submissions", "nope".repeat(8))).status).toBe(401);
});
it("records delivery once, only with the matching receipt", async () => {
const { id, receipt } = (await (await call("POST", "/api/submissions", T.intake, inquiry())).json()) as { id: number; receipt: string };
expect((await call("POST", `/api/submissions/${id}/delivery`, T.intake, { receipt: "wrong", status: "sent" })).status).toBe(403);
expect((await call("POST", `/api/submissions/${id}/delivery`, T.intake, { status: "sent" })).status).toBe(403);
expect((await call("POST", `/api/submissions/${id}/delivery`, T.intake, { receipt, status: "failed", error: "SMTP said no" })).status).toBe(200);
expect((await call("POST", `/api/submissions/${id}/delivery`, T.intake, { receipt, status: "sent" })).status).toBe(409);
const s = (await (await call("GET", `/api/submissions/${id}`, T.admin)).json()) as { delivery: string; deliveryError: string };
expect(s.delivery).toBe("failed");
expect(s.deliveryError).toBe("SMTP said no");
expect(JSON.stringify(s)).not.toContain(receipt);
});
it("lists unhandled first, newest first, and marks handled", async () => {
const list = async () =>
((await (await call("GET", "/api/submissions", T.admin)).json()) as { counts: { unhandled: number }; submissions: { id: number; handled: boolean }[] });
const before = await list();
const ids = before.submissions.map((s) => s.id);
expect(ids).toEqual([...ids].sort((a, b) => b - a));
const oldest = ids[ids.length - 1]!;
const res = await call("POST", `/api/submissions/${oldest}/handled`, T.admin, { handled: true });
expect(res.status).toBe(200);
const after = await list();
expect(after.submissions[after.submissions.length - 1]!.id).toBe(oldest);
expect(after.submissions[after.submissions.length - 1]!.handled).toBe(true);
expect(after.counts.unhandled).toBe(before.counts.unhandled - 1);
expect((await call("POST", "/api/submissions/9999/handled", T.admin, { handled: true })).status).toBe(404);
});
it("validates what it saves", async () => {
expect((await call("POST", "/api/submissions", T.intake, inquiry({ email: "not an email" }))).status).toBe(400);
expect((await call("POST", "/api/submissions", T.intake, inquiry({ kind: "spam" }))).status).toBe(400);
expect((await call("POST", "/api/submissions", T.intake, inquiry({ name: "" }))).status).toBe(400);
});
it("saves an attachment under a safe name and serves it as a download", async () => {
const data = Buffer.from("1 Sol Ring\n1 Command Tower\n").toString("base64");
const res = await call("POST", "/api/submissions", T.intake, inquiry({ attachment: { name: "../../etc/deck list.txt", data } }));
expect(res.status).toBe(201);
const { id } = (await res.json()) as { id: number };
const files = readdirSync(join(r.dir, "submissions"));
expect(files.some((f) => f.startsWith(`${id}-`) && f.endsWith(".txt"))).toBe(true);
const s = (await (await call("GET", `/api/submissions/${id}`, T.admin)).json()) as { attachment: { name: string; size: number } };
expect(s.attachment.name).toBe("deck list.txt");
const dl = await call("GET", `/api/submissions/${id}/attachment`, T.admin);
expect(dl.status).toBe(200);
expect(dl.headers.get("content-disposition")).toContain("attachment;");
expect(dl.headers.get("x-content-type-options")).toBe("nosniff");
expect(await dl.text()).toBe("1 Sol Ring\n1 Command Tower\n");
});
it("turns down attachments that are too large, of an unknown type, or not base64", async () => {
const big = Buffer.alloc(MAX + 1, 1).toString("base64");
const count = async () => ((await (await call("GET", "/api/submissions", T.admin)).json()) as { counts: { total: number } }).counts.total;
const n = await count();
expect((await call("POST", "/api/submissions", T.intake, inquiry({ attachment: { name: "a.png", data: big } }))).status).toBe(413);
expect((await call("POST", "/api/submissions", T.intake, inquiry({ attachment: { name: "a.exe", data: "AAAA" } }))).status).toBe(415);
expect((await call("POST", "/api/submissions", T.intake, inquiry({ attachment: { name: "a.html", data: "AAAA" } }))).status).toBe(415);
expect((await call("POST", "/api/submissions", T.intake, inquiry({ attachment: { name: "a.png", data: "<script>" } }))).status).toBe(400);
expect(await count()).toBe(n);
});
});
describe("attachment names", () => {
it("strips paths and odd characters", () => {
expect(safeName("C:\\Users\\x\\photo.png")).toBe("photo.png");
expect(safeName("..")).toBe("attachment");
expect(safeName("a\r\nb.txt")).toBe("a_b.txt");
expect(extensionOf("deck.DEK")).toBe("dek");
expect(extensionOf("x.svg")).toBeNull();
expect(extensionOf("noext")).toBeNull();
});
});
services/content/test/sync/fixtures/README.md
+22
View File
@@ -0,0 +1,22 @@
# Sync test fixtures
Small sample payloads written by hand from each API's documented response
shape, for the parsing tests. They are **not** live responses: no request was
made to Archidekt, BoardGameGeek, Open Library, TMDB, AniList, Suwayomi,
Kavita, Bambuddy or Homebox to make them,
and ids, names and image addresses are illustrative.
| File | Shape of |
| --- | --- |
| `archidekt-deck.sample.json` | `GET https://archidekt.com/api/decks/<id>/` |
| `bgg-thing.sample.xml` | `GET https://boardgamegeek.com/xmlapi2/thing?id=…` |
| `openlibrary-search.sample.json` | `GET https://openlibrary.org/search.json?title=…` |
| `tmdb-search.sample.json` | `GET https://api.themoviedb.org/3/search/movie?query=…` |
| `anilist-media.sample.json` | `POST https://graphql.anilist.co` with the `Media` query |
| `suwayomi-library.sample.json` | `POST <SUWAYOMI_URL>/api/graphql` with `LIBRARY_QUERY` (Suwayomi-Server's GraphQL schema) |
| `anilist-page.sample.json` | `POST https://graphql.anilist.co` with the batched `Page { media(id_in:) }` query |
| `kavita-series.sample.json` | `POST <KAVITA_URL>/api/Series/all-v2` (SeriesDto list) |
| `kavita-metadata.sample.json` | `GET <KAVITA_URL>/api/Series/metadata?seriesId=`, one answer per series id |
| `bambuddy-spools.sample.json` | **A guess.** Bambuddy's API is not documented anywhere this project has; this only exercises `mapBambuddySpools`' tolerant reading. |
| `homebox-labels.sample.json` | `GET <HOMEBOX_URL>/api/v1/labels` |
| `homebox-items.sample.json` | `GET <HOMEBOX_URL>/api/v1/items?labels=…` |
services/content/test/sync/fixtures/anilist-media.sample.json
+13
View File
@@ -0,0 +1,13 @@
{
"_sample": "Hand-written from AniList's GraphQL Media response shape; not a live response.",
"data": {
"Media": {
"id": 30013,
"title": { "english": "One Piece", "romaji": "ONE PIECE" },
"coverImage": {
"extraLarge": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/large/sample-xl.jpg",
"large": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/sample-l.jpg"
}
}
}
}
services/content/test/sync/fixtures/anilist-page.sample.json
+15
View File
@@ -0,0 +1,15 @@
{
"data": {
"Page": {
"media": [
{ "id": 100001, "title": { "english": "Sample Tower Climber", "romaji": "Sample Tou" }, "format": "MANGA", "countryOfOrigin": "KR",
"coverImage": { "extraLarge": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/large/sample-100001.jpg", "large": null } },
{ "id": 100003, "title": { "english": null, "romaji": "Sample Owari" }, "format": "MANGA", "countryOfOrigin": "JP",
"coverImage": { "extraLarge": null, "large": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/sample-100003.jpg" } },
{ "id": 100004, "title": { "english": "New Sample Series", "romaji": "Shin Sample" }, "format": "MANGA", "countryOfOrigin": "KR",
"coverImage": { "extraLarge": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/large/sample-100004.jpg" } },
{ "id": 100005, "title": { "english": "Sample Dropped" }, "format": "MANGA", "countryOfOrigin": "JP", "coverImage": {} }
]
}
}
}
services/content/test/sync/fixtures/archidekt-deck.sample.json
+39
View File
@@ -0,0 +1,39 @@
{
"_sample": "Hand-written from Archidekt's deck response shape; not a live response.",
"id": 1234567,
"name": "Sample Atraxa Superfriends",
"deckFormat": 3,
"featured": "https://storage.googleapis.com/topdekt-user/images/sample-featured.jpg",
"customFeatured": "",
"private": false,
"categories": [
{ "id": 1, "name": "Commander", "isPremier": true, "includedInDeck": true, "includedInPrice": true },
{ "id": 2, "name": "Ramp", "isPremier": false, "includedInDeck": true, "includedInPrice": true },
{ "id": 3, "name": "Maybeboard", "isPremier": false, "includedInDeck": false, "includedInPrice": false }
],
"cards": [
{
"quantity": 1,
"categories": ["Commander"],
"card": {
"uid": "d0d33d52-3d28-4635-b985-51e126289259",
"oracleCard": { "name": "Atraxa, Praetors' Voice", "colorIdentity": ["White", "Blue", "Black", "Green"] }
}
},
{
"quantity": 1,
"categories": ["Ramp"],
"card": { "uid": "aaaa1111-0000-0000-0000-000000000000", "oracleCard": { "name": "Sol Ring", "colorIdentity": [] } }
},
{
"quantity": 34,
"categories": ["Land"],
"card": { "uid": "bbbb2222-0000-0000-0000-000000000000", "oracleCard": { "name": "Forest", "colorIdentity": ["Green"] } }
},
{
"quantity": 1,
"categories": ["Maybeboard"],
"card": { "uid": "cccc3333-0000-0000-0000-000000000000", "oracleCard": { "name": "Lightning Bolt", "colorIdentity": ["Red"] } }
}
]
}
services/content/test/sync/fixtures/bambuddy-spools.sample.json
+11
View File
@@ -0,0 +1,11 @@
{
"_comment": "GUESSED SHAPE. Bambuddy's API is undocumented here; this only exercises mapBambuddySpools' tolerant reading.",
"items": [
{ "id": 1, "material": "PLA", "color_name": "Jade White", "color_hex": "F5F5DCFF", "remaining_weight": 640 },
{ "id": 2, "material": "PLA", "color_name": "Jade White", "color_hex": "F5F5DC", "remaining_weight": 20 },
{ "id": 3, "material": "PLA", "colorName": "Bambu Green", "colorHex": "#00AE42", "remainingWeight": 15 },
{ "id": 4, "material": "PETG", "color": "Translucent Orange", "hex": "#FF6A13", "remaining": 900 },
{ "id": 5, "material": "PLA", "color_name": "Old Spool", "archived": true, "remaining_weight": 500 },
{ "id": 6, "material": "PLA", "color_name": "Charcoal", "color_hex": "#2b2b2b" }
]
}
services/content/test/sync/fixtures/bgg-thing.sample.xml
+32
View File
@@ -0,0 +1,32 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Hand-written from BoardGameGeek's XML API2 "thing" response shape; not a live response. -->
<items termsofuse="https://boardgamegeek.com/xmlapi/termsofuse">
<item type="boardgame" id="266192">
<thumbnail>https://cf.geekdo-images.com/sample__thumb/img/wingspan.jpg</thumbnail>
<image>https://cf.geekdo-images.com/sample__original/img/wingspan.jpg</image>
<name type="primary" sortindex="1" value="Wingspan" />
<description>Attract a beautiful and diverse collection of birds &amp;amp; more.</description>
<yearpublished value="2019" />
<minplayers value="1" />
<maxplayers value="5" />
<playingtime value="70" />
<minplaytime value="40" />
<maxplaytime value="70" />
<minage value="10" />
</item>
<item type="boardgame" id="13">
<image>http://cf.geekdo-images.com/sample__original/img/catan.jpg?a=1&amp;b=2</image>
<name type="primary" sortindex="1" value="CATAN" />
<minplayers value="3" />
<maxplayers value="4" />
<playingtime value="120" />
<minplaytime value="60" />
<maxplaytime value="120" />
</item>
<item type="boardgame" id="99">
<name type="primary" sortindex="1" value="No Data Yet" />
<minplayers value="0" />
<maxplayers value="0" />
<playingtime value="0" />
</item>
</items>
services/content/test/sync/fixtures/homebox-items.sample.json
+10
View File
@@ -0,0 +1,10 @@
{
"page": 1,
"pageSize": 100,
"total": 3,
"items": [
{ "id": "a1", "name": "Veg tan, Walnut", "quantity": 2, "description": "Swatch #5c3a21, 3-4 oz", "archived": false },
{ "id": "a2", "name": "Chrome tan, Black", "quantity": 0, "description": "", "archived": false },
{ "id": "a3", "name": "Retired hide", "quantity": 4, "description": "", "archived": true }
]
}
services/content/test/sync/fixtures/homebox-labels.sample.json
+5
View File
@@ -0,0 +1,5 @@
[
{ "id": "6f1c0a52-0000-4000-8000-000000000001", "name": "Leather", "description": "" },
{ "id": "6f1c0a52-0000-4000-8000-000000000002", "name": "Paper", "description": "" },
{ "id": "6f1c0a52-0000-4000-8000-000000000003", "name": "Tools", "description": "" }
]
services/content/test/sync/fixtures/kavita-metadata.sample.json
+5
View File
@@ -0,0 +1,5 @@
{
"11": { "seriesId": 11, "publicationStatus": 0, "webLinks": "https://www.novelupdates.com/series/sample/,https://anilist.co/manga/200001/Sample-Light-Novel/" },
"12": { "seriesId": 12, "publicationStatus": 0, "webLinks": "" },
"13": { "seriesId": 13, "publicationStatus": 2, "webLinks": "https://anilist.co/manga/200003" }
}
services/content/test/sync/fixtures/kavita-series.sample.json
+5
View File
@@ -0,0 +1,5 @@
[
{ "id": 11, "name": "Sample Light Novel", "localizedName": "", "pages": 400, "pagesRead": 400, "latestReadDate": "2026-10-02T21:15:00", "libraryId": 2 },
{ "id": 12, "name": "Sample Ongoing Novel", "localizedName": "Sample Ongoing Novel (EN)", "pages": 300, "pagesRead": 120, "latestReadDate": "2026-10-09T08:00:00", "libraryId": 2 },
{ "id": 13, "name": "Sample Unread Novel", "pages": 250, "pagesRead": 0, "latestReadDate": "0001-01-01T00:00:00", "libraryId": 2 }
]
services/content/test/sync/fixtures/openlibrary-search.sample.json
+8
View File
@@ -0,0 +1,8 @@
{
"_sample": "Hand-written from Open Library's search.json response shape; not a live response.",
"numFound": 2,
"docs": [
{ "title": "The Way of Kings (study guide)" },
{ "title": "The Way of Kings", "cover_i": 8221255, "cover_edition_key": "OL24395542M" }
]
}
services/content/test/sync/fixtures/suwayomi-library.sample.json
+42
View File
@@ -0,0 +1,42 @@
{
"data": {
"categories": {
"nodes": [
{ "name": "Default", "mangas": { "nodes": [
{ "title": "Uncategorised Series", "inLibrary": true, "latestReadChapter": null, "trackRecords": { "nodes": [] } }
] } },
{ "name": "Reading", "mangas": { "nodes": [
{ "title": "Sample Tower Climber (Source Title)", "inLibrary": true,
"latestReadChapter": { "chapterNumber": 142, "lastReadAt": "1759900000" },
"trackRecords": { "nodes": [
{ "trackerId": 1, "remoteId": "555", "title": "Sample Tower Climber MAL", "lastChapterRead": 140 },
{ "trackerId": 2, "remoteId": "100001", "title": "Sample Tower Climber", "lastChapterRead": 142 }
] } },
{ "title": "New Sample Series", "inLibrary": true,
"latestReadChapter": { "chapterNumber": 3, "lastReadAt": "1759990000" },
"trackRecords": { "nodes": [ { "trackerId": 2, "remoteId": "100004", "title": "New Sample Series", "lastChapterRead": 3 } ] } },
{ "title": "Untracked Sample", "inLibrary": true,
"latestReadChapter": { "chapterNumber": 7, "lastReadAt": "0" },
"trackRecords": { "nodes": [] } }
] } },
{ "name": "Plan to read", "mangas": { "nodes": [
{ "title": "Sample Later", "inLibrary": true, "latestReadChapter": null,
"trackRecords": { "nodes": [ { "trackerId": 2, "remoteId": "100002", "title": "Sample Later", "lastChapterRead": 0 } ] } }
] } },
{ "name": "Completed", "mangas": { "nodes": [
{ "title": "Sample Tower Climber (Source Title)", "inLibrary": true,
"latestReadChapter": { "chapterNumber": 142, "lastReadAt": "1759900000" },
"trackRecords": { "nodes": [ { "trackerId": 2, "remoteId": "100001", "title": "Sample Tower Climber", "lastChapterRead": 142 } ] } },
{ "title": "Sample Finished", "inLibrary": true,
"latestReadChapter": { "chapterNumber": 88.5, "lastReadAt": "1750000000000" },
"trackRecords": { "nodes": [ { "trackerId": 2, "remoteId": "100003", "title": "Sample Finished", "lastChapterRead": 88 } ] } }
] } },
{ "name": "Dropped", "mangas": { "nodes": [
{ "title": "Sample Dropped", "inLibrary": true,
"latestReadChapter": { "chapterNumber": 12, "lastReadAt": "1740000000" },
"trackRecords": { "nodes": [ { "trackerId": 2, "remoteId": "100005", "title": "Sample Dropped", "lastChapterRead": 12 } ] } }
] } }
]
}
}
}
services/content/test/sync/fixtures/tmdb-search.sample.json
+9
View File
@@ -0,0 +1,9 @@
{
"_sample": "Hand-written from TMDB's search/movie response shape; not a live response.",
"page": 1,
"results": [
{ "id": 1, "title": "Arrival (short)", "poster_path": null },
{ "id": 329865, "title": "Arrival", "release_date": "2016-11-10", "poster_path": "/x2FJsf1ElAgr63Y3PNPtJrcmpoe.jpg" }
],
"total_results": 2
}
services/content/test/sync/homelab.test.ts
+410
View File
@@ -0,0 +1,410 @@
/**
* The homelab jobs (Suwayomi, Kavita, Bambuddy, Homebox) and Library
* suggestions, against the hand-written sample payloads in ./fixtures. Those
* are documented shapes (Bambuddy's is a guess), never live responses; see
* fixtures/README.md.
*/
import { readFileSync } from "node:fs";
import type { AddressInfo } from "node:net";
import { join } from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { parseTokens } from "../../src/config.ts";
import { createHttpServer } from "../../src/http.ts";
import { listSuggestions } from "../../src/suggestions.ts";
import { bambuddyJob, filamentOptions, mapBambuddySpools } from "../../src/sync/jobs/bambuddy.ts";
import { homelabSettings } from "../../src/sync/jobs/homelab.ts";
import { homeboxJob, materialOptions, parseFilters, parseItems } from "../../src/sync/jobs/homebox.ts";
import { kavitaJob, kavitaState, parseMetadata, parseSeries } from "../../src/sync/jobs/kavita.ts";
import { stateForCategory, targetState } from "../../src/sync/jobs/reader.ts";
import { ANILIST_TRACKER_ID, parseAniListBatch, parseLibrary, suwayomiJob } from "../../src/sync/jobs/suwayomi.ts";
import { enabledJobs, syncSettings } from "../../src/sync/index.ts";
import { Scheduler } from "../../src/sync/scheduler.ts";
import type { SyncJob } from "../../src/sync/types.ts";
import { rig, type Rig } from "../helpers.ts";
const fixture = (name: string) => JSON.parse(readFileSync(join(import.meta.dirname, "fixtures", name), "utf8"));
const ok = (body: unknown, status = 200) => new Response(JSON.stringify(body), { status });
type Call = { url: string; init?: RequestInit };
function fakeFetch(answer: (url: string, init?: RequestInit) => Response) {
const calls: Call[] = [];
const fn = (async (input: string | URL | Request, init?: RequestInit) => {
calls.push({ url: String(input), init });
return answer(String(input), init);
}) as typeof fetch;
return { fn, calls };
}
const headerOf = (c: Call, k: string) => (c.init?.headers as Record<string, string> | undefined)?.[k];
const bodyOf = (c: Call) => JSON.parse(String(c.init?.body ?? "{}"));
let r: Rig;
beforeEach(() => {
r = rig();
});
afterEach(() => r.done());
const scheduler = (job: SyncJob, fetchFn: typeof fetch) =>
new Scheduler(r.service, [job], { fetch: fetchFn, now: () => new Date(r.clock.t), log: () => {} });
const manga = (title: string, extra: Record<string, unknown> = {}) => ({
type: "library-item",
title,
medium: "manhwa",
state: "up next",
publish: true,
...extra,
});
// ---------------------------------------------------------------- state mapping
describe("PersonalReader state mapping", () => {
it("maps categories: Reading now, Plan to read up next, Completed finished, Dropped left alone", () => {
expect(stateForCategory("Reading")).toBe("now");
expect(stateForCategory(" plan to Read ")).toBe("up next");
expect(stateForCategory("Completed")).toBe("finished");
expect(stateForCategory("Dropped")).toBeNull();
expect(stateForCategory("Default")).toBeUndefined();
});
it("proposes only a different state, and keeps caught up and finished", () => {
expect(targetState("up next", "now")).toBe("now");
expect(targetState("now", "now")).toBeNull();
expect(targetState("now", null)).toBeNull();
expect(targetState("caught up", "now")).toBeNull();
expect(targetState("finished", "caught up")).toBeNull();
expect(targetState("dropped", "finished")).toBe("finished");
});
});
// ---------------------------------------------------------------- Suwayomi
describe("Suwayomi", () => {
it("parses the library: AniList tracker id, furthest chapter, when, best category", () => {
const s = parseLibrary(fixture("suwayomi-library.sample.json"));
const by = (t: string) => s.find((x) => x.title === t);
// In Reading and Completed: Reading wins. The AniList record's title is used, not the source's.
expect(by("Sample Tower Climber")).toEqual({ title: "Sample Tower Climber", category: "Reading", anilistId: 100001, chapter: 142, readAt: "2025-10-08T05:06:40.000Z" });
expect(by("Sample Finished")).toMatchObject({ category: "Completed", anilistId: 100003, chapter: 88.5, readAt: new Date(1750000000000).toISOString() });
expect(by("Untracked Sample")).toEqual({ title: "Untracked Sample", category: "Reading", chapter: 7 });
expect(by("Uncategorised Series")).toBeUndefined();
expect(s).toHaveLength(6);
expect(ANILIST_TRACKER_ID).toBe(2);
expect(() => parseLibrary({ errors: [{ message: "Unauthorized" }] })).toThrow(/Unauthorized/);
});
it("parses AniList's batch: cover, title and medium", () => {
const m = parseAniListBatch(fixture("anilist-page.sample.json"));
expect(m.get(100001)).toEqual({ title: "Sample Tower Climber", medium: "manhwa", cover: "https://s4.anilist.co/file/anilistcdn/media/manga/cover/large/sample-100001.jpg" });
expect(m.get(100003)).toMatchObject({ title: "Sample Owari", medium: "manga" });
expect(m.get(100005)?.cover).toBeUndefined();
});
function suwayomiFetch() {
return fakeFetch((url, init) => {
if (url === "https://graphql.anilist.co") return ok(fixture("anilist-page.sample.json"));
const q = String(JSON.parse(String(init?.body)).query);
if (q.includes("login")) return ok({ data: { login: { accessToken: "jwt-access" } } });
return ok(fixture("suwayomi-library.sample.json"));
});
}
it("logs in (ui_login), caches progress and cover, writes states as sync, holds Greg's notes, suggests", async () => {
// A note the sync wrote earlier (so it may change it) and notes Greg wrote.
const created = r.service.apply({ op: "create", data: manga("Sample Tower Climber", { anilistId: 100001, publish: true }) }, "sync");
expect(created.status).toBe("accepted");
const later = r.put(manga("Sample Later", { anilistId: 100002, state: "now" }));
const finished = r.put(manga("Sample Finished", { medium: "manga", anilistId: 100003, state: "now" }));
const dropped = r.put(manga("Sample Dropped", { medium: "manga", anilistId: 100005, state: "now" }));
r.service.scan();
const f = suwayomiFetch();
const job = suwayomiJob({ url: "http://suwayomi:4567/", user: "site-reader", pass: "pw" });
const report = await scheduler(job, f.fn).runNow("suwayomi");
expect(report).toMatchObject({ ok: true, counts: { series: 6, matched: 4, untracked: 1, accepted: 1, held: 2, suggestions: 1 } });
// Login, then the library with the access token.
expect(f.calls[0]!.url).toBe("http://suwayomi:4567/api/graphql");
expect(bodyOf(f.calls[0]!).variables).toEqual({ username: "site-reader", password: "pw" });
expect(headerOf(f.calls[1]!, "authorization")).toBe("Bearer jwt-access");
// The sync's own note changed; Greg's are held for review; Dropped is untouched.
const tower = created.status === "accepted" ? created.path : "";
expect(r.store.get(tower)!.data).toMatchObject({ state: "now", updatedBy: "sync" });
expect(r.store.get(later)!.data.state).toBe("now");
expect(r.store.pendingFor(later)?.reasons.map((x) => x.code)).toContain("agent-edit");
expect(r.store.pendingFor(finished)?.text).toContain("state: finished");
expect(r.store.pendingFor(dropped)).toBeNull();
// Cached for the site: title, chapter, status, when, AniList cover. Nothing else.
const synced = r.store.syncCache.forPath(tower);
expect(synced).toEqual({
chapterProgress: { title: "Sample Tower Climber", status: "now", chapter: 142, readAt: "2025-10-08T05:06:40.000Z" },
cover: "https://s4.anilist.co/file/anilistcdn/media/manga/cover/large/sample-100001.jpg",
});
const everything = JSON.stringify([...r.store.syncCache.all()]);
expect(everything).not.toMatch(/suwayomi|Source Title|4567|thumbnail|scanlator/i);
// The new Reading series is a suggestion; the untracked one cannot be (no AniList id).
expect(listSuggestions(r.store)).toEqual([
{
key: "anilist-100004",
anilistId: 100004,
title: "New Sample Series",
medium: "manhwa",
cover: "https://s4.anilist.co/file/anilistcdn/media/manga/cover/large/sample-100004.jpg",
},
]);
});
it("proposes a held change once: not again while it waits, nor after Greg rejects it", async () => {
const later = r.put(manga("Sample Later", { anilistId: 100002, state: "now" }));
r.service.scan();
const s = scheduler(suwayomiJob({ url: "http://suwayomi:4567", auth: "none" }), suwayomiFetch().fn);
expect((await s.runNow("suwayomi")).counts).toMatchObject({ held: 1 });
expect((await s.runNow("suwayomi")).counts).toMatchObject({ held: 0, waiting: 0, accepted: 0 });
r.service.rejectHeld(r.store.pendingFor(later)!.id);
expect((await s.runNow("suwayomi")).counts).toMatchObject({ held: 0 });
expect(r.store.pendingFor(later)).toBeNull();
expect(r.store.get(later)!.data.state).toBe("now");
});
it("keeps the last cover when AniList is down, and reports the run as failed", async () => {
r.service.apply({ op: "create", data: manga("Sample Tower Climber", { anilistId: 100001, state: "now" }) }, "sync");
const good = suwayomiFetch();
await scheduler(suwayomiJob({ url: "http://s", auth: "none" }), good.fn).runNow("suwayomi");
const down = fakeFetch((url) => (url.includes("anilist") ? ok({}, 503) : ok(fixture("suwayomi-library.sample.json"))));
const report = await scheduler(suwayomiJob({ url: "http://s", auth: "none" }), down.fn).runNow("suwayomi");
expect(report).toMatchObject({ ok: false, message: expect.stringContaining("AniList answered 503") });
const [path] = [...r.store.syncCache.all().keys()].filter((p) => p.endsWith(".md"));
expect(r.store.syncCache.forPath(path!).cover).toMatch(/sample-100001/);
});
it("uses basic auth when set, and fails the run on a refused login", async () => {
const f = fakeFetch(() => ok(fixture("suwayomi-library.sample.json")));
await scheduler(suwayomiJob({ url: "http://s", user: "u", pass: "p", auth: "basic" }), f.fn).runNow("suwayomi");
expect(headerOf(f.calls[0]!, "authorization")).toBe(`Basic ${Buffer.from("u:p").toString("base64")}`);
const no = fakeFetch(() => ok({ data: { login: null }, errors: [{ message: "Invalid credentials" }] }));
const report = await scheduler(suwayomiJob({ url: "http://s", user: "u", pass: "bad" }), no.fn).runNow("suwayomi");
expect(report).toMatchObject({ ok: false, message: expect.stringContaining("Invalid credentials") });
});
});
// ---------------------------------------------------------------- suggestions
describe("Library suggestions", () => {
it("are listed, dismissed for good, and leave once a Library item has the AniList id", async () => {
await scheduler(suwayomiJob({ url: "http://s", auth: "none" }), fakeFetch((url) =>
ok(url.includes("anilist") ? fixture("anilist-page.sample.json") : fixture("suwayomi-library.sample.json")),
).fn).runNow("suwayomi");
// Every Reading series with an AniList id and no note: 100001 and 100004.
const ADMIN = "a".repeat(24);
const READER = "r".repeat(24);
const server = createHttpServer(r.service, { tokens: parseTokens(`admin:${ADMIN},reader:${READER}`) });
await new Promise<void>((done) => server.listen(0, "127.0.0.1", done));
const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
const get = (token = ADMIN) => fetch(`${base}/api/suggestions`, { headers: { authorization: `Bearer ${token}` } });
try {
expect((await get(READER)).status).toBe(403);
const list = (await (await get()).json()) as { key: string; anilistId: number }[];
// By title: New Sample Series, Sample Tower Climber.
expect(list.map((s) => s.key)).toEqual(["anilist-100004", "anilist-100001"]);
// Dismiss one.
const dis = await fetch(`${base}/api/suggestions/anilist-100004/dismiss`, { method: "POST", headers: { authorization: `Bearer ${ADMIN}` } });
expect(dis.status).toBe(200);
// Add the other through the normal write path.
const add = await fetch(`${base}/api/changes`, {
method: "POST",
headers: { authorization: `Bearer ${ADMIN}`, "content-type": "application/json" },
body: JSON.stringify({ op: "create", data: { type: "library-item", title: "Sample Tower Climber", medium: "manhwa", anilistId: 100001, state: "now", publish: true } }),
});
expect(((await add.json()) as { status: string }).status).toBe("accepted");
expect(await (await get()).json()).toEqual([]);
} finally {
server.close();
}
});
});
// ---------------------------------------------------------------- Kavita
describe("Kavita", () => {
const series = fixture("kavita-series.sample.json") as unknown[];
const metadata = fixture("kavita-metadata.sample.json") as Record<string, unknown>;
it("parses series and metadata; AniList only from an explicit anilist.co link", () => {
expect(parseSeries(series[1])).toEqual({ id: 12, name: "Sample Ongoing Novel (EN)", pages: 300, pagesRead: 120, latestReadDate: new Date("2026-10-09T08:00:00").toISOString() });
expect(parseSeries(series[2])).not.toHaveProperty("latestReadDate");
expect(parseMetadata(metadata["11"])).toEqual({ anilistId: 200001, publicationStatus: 0 });
expect(parseMetadata(metadata["12"])).toEqual({ publicationStatus: 0 });
});
it("maps progress to a state", () => {
const s = (pages: number, pagesRead: number) => ({ id: 1, name: "x", pages, pagesRead });
expect(kavitaState(s(10, 10), { publicationStatus: 2 }, false)).toBe("finished");
expect(kavitaState(s(10, 10), { publicationStatus: 0 }, false)).toBe("caught up");
expect(kavitaState(s(10, 3), {}, false)).toBe("now");
expect(kavitaState(s(10, 0), {}, true)).toBe("up next");
expect(kavitaState(s(10, 0), {}, false)).toBeNull();
});
it("links by kavitaSeriesId or by AniList id in metadata, never by title", async () => {
const byId = r.put({ type: "library-item", title: "Sample Ongoing Novel", medium: "light novel", state: "up next", kavitaSeriesId: 12, publish: true });
const byAniList = r.service.apply(
{ op: "create", data: { type: "library-item", title: "Sample Light Novel", medium: "light novel", state: "now", anilistId: 200001 } },
"sync",
);
const sameTitle = r.put({ type: "library-item", title: "Sample Unread Novel", medium: "light novel", state: "now", publish: true });
r.service.scan();
const f = fakeFetch((url) => {
if (url.includes("/api/Plugin/authenticate")) return ok({ username: "greg", token: "kavita-jwt" });
if (url.includes("/api/Series/all-v2")) return ok(series);
if (url.includes("/api/Series/metadata")) return ok(metadata[new URL(url).searchParams.get("seriesId")!]);
if (url.endsWith("/api/Series/12")) return ok(series[1]);
if (url.includes("/api/want-to-read/v2")) return ok([series[2]]);
return ok({}, 404);
});
const report = await scheduler(kavitaJob({ url: "http://kavita:5000", apiKey: "key-123", delayMs: 0 }), f.fn).runNow("kavita");
expect(report).toMatchObject({ ok: true, counts: { linked: 2, synced: 2, accepted: 1, held: 1 } });
expect(f.calls[0]!.url).toBe("http://kavita:5000/api/Plugin/authenticate?apiKey=key-123&pluginName=GregKilmer.com");
expect(headerOf(f.calls[1]!, "authorization")).toBe("Bearer kavita-jwt");
// Read to the end of an ongoing series: caught up, written as sync.
const lnPath = byAniList.status === "accepted" ? byAniList.path : "";
expect(r.store.get(lnPath)!.data).toMatchObject({ state: "caught up", updatedBy: "sync" });
expect(r.store.syncCache.forPath(lnPath)).toEqual({ bookProgress: { title: "Sample Light Novel", percent: 100, readAt: new Date("2026-10-02T21:15:00").toISOString() } });
// Greg's note: partly read is now, held for review.
expect(r.store.pendingFor(byId)?.text).toContain("state: now");
// Same title, no link: untouched and nothing cached.
expect(r.store.syncCache.forPath(sameTitle)).toEqual({});
expect(r.store.pendingFor(sameTitle)).toBeNull();
});
});
// ---------------------------------------------------------------- Bambuddy
describe("Bambuddy (unverified API shape)", () => {
it("maps spools to colors: merged by name, enough grams is in stock, archived skipped", () => {
const spools = mapBambuddySpools(fixture("bambuddy-spools.sample.json"));
expect(spools.map((s) => s.name)).toEqual(["Jade White", "Jade White", "Bambu Green", "Translucent Orange", "Charcoal"]);
expect(filamentOptions(spools)).toEqual([
{ name: "Charcoal", hex: "#2b2b2b", inStock: true },
{ name: "Jade White", hex: "#f5f5dc", inStock: true },
{ name: "Translucent Orange", hex: "#ff6a13", inStock: true },
{ name: "Bambu Green", hex: "#00ae42", inStock: false },
]);
expect(filamentOptions(spools, { materials: ["PETG"] }).map((o) => o.name)).toEqual(["Translucent Orange"]);
expect(mapBambuddySpools([{ name: "Plain list" }])).toEqual([{ name: "Plain list" }]);
expect(() => mapBambuddySpools({ nothing: true })).toThrow(/no list/);
});
it("stores filamentStock on lines with a Filament option, at the configured path", async () => {
const line = (title: string, group: string) => ({
type: "product-line",
title,
category: "Tabletop",
mode: "made to order",
channel: "here",
summary: "s",
slots: { open: 1, total: 2 },
priceTiers: [],
options: [{ group, name: "Jade White", inStock: true }],
publish: true,
});
const printed = r.put(line("Printed terrain", "Filament"));
const cards = r.put(line("Cards", "Finish"));
r.service.scan();
const f = fakeFetch(() => ok(fixture("bambuddy-spools.sample.json")));
const report = await scheduler(bambuddyJob({ url: "http://bambuddy:8000/", path: "/api/v1/spools", apiKey: "k" }), f.fn).runNow("bambuddy");
expect(report).toMatchObject({ ok: true, counts: { colors: 4, inStock: 3, lines: 1 } });
expect(f.calls[0]!.url).toBe("http://bambuddy:8000/api/v1/spools");
expect(headerOf(f.calls[0]!, "authorization")).toBe("Bearer k");
expect(r.store.syncCache.forPath(printed).filamentStock).toEqual([{ group: "Filament", options: expect.arrayContaining([{ name: "Jade White", hex: "#f5f5dc", inStock: true }]) }]);
expect(r.store.syncCache.forPath(cards)).toEqual({});
// Notes are never written.
expect(r.store.get(printed)!.author).toBe("greg");
});
});
// ---------------------------------------------------------------- Homebox
describe("Homebox", () => {
it("parses filters, items and materials", () => {
expect(parseFilters("leather, paper=Paper goods")).toEqual([
{ name: "leather", group: "Leather" },
{ name: "paper", group: "Paper goods" },
]);
const { items, total } = parseItems(fixture("homebox-items.sample.json"));
expect(total).toBe(3);
expect(materialOptions(items)).toEqual([
{ name: "Veg tan, Walnut", hex: "#5c3a21", inStock: true },
{ name: "Chrome tan, Black", inStock: false },
]);
});
it("logs in, filters by label and stores materialStock on Leather lines", async () => {
const wallet = r.put({
type: "product-line",
title: "Wallet",
category: "Leather",
mode: "made to order",
channel: "here",
summary: "s",
slots: { open: 1, total: 2 },
priceTiers: [],
options: [{ group: "Leather", name: "Chrome tan, Black" }],
publish: true,
});
r.service.scan();
const f = fakeFetch((url) => {
if (url.endsWith("/api/v1/users/login")) return ok({ token: "Bearer hb-token", expiresAt: "2026-10-12T00:00:00Z" });
if (url.endsWith("/api/v1/labels")) return ok(fixture("homebox-labels.sample.json"));
if (url.includes("/api/v1/items?")) {
return new URL(url).searchParams.get("labels")?.endsWith("01") ? ok(fixture("homebox-items.sample.json")) : ok({ items: [], total: 0 });
}
return ok({}, 404);
});
const report = await scheduler(homeboxJob({ url: "http://homebox:7745", user: "u", pass: "p", labels: parseFilters("leather,paper") }), f.fn).runNow("homebox");
expect(report).toMatchObject({ ok: true, counts: { lines: 1, leather: 1, paper: 0 } });
expect(headerOf(f.calls[2]!, "authorization")).toBe("Bearer hb-token");
expect(r.store.syncCache.forPath(wallet).materialStock).toEqual([
{ group: "Leather", options: [{ name: "Veg tan, Walnut", hex: "#5c3a21", inStock: true }, { name: "Chrome tan, Black", inStock: false }] },
]);
const missing = await scheduler(homeboxJob({ url: "http://homebox:7745", user: "u", pass: "p", labels: parseFilters("vinyl") }), f.fn).runNow("homebox");
expect(missing).toMatchObject({ ok: false, message: expect.stringContaining("vinyl") });
});
});
// ---------------------------------------------------------------- settings
describe("homelab settings", () => {
it("enable each job by its own URL and login, not by SYNC_JOBS", () => {
expect(enabledJobs(syncSettings({})).map((j) => j.name)).toEqual([]);
const env = {
SUWAYOMI_URL: "http://suwayomi:4567",
SUWAYOMI_USER: "u",
SUWAYOMI_PASS: "p",
KAVITA_URL: "http://kavita:5000",
KAVITA_API_KEY: "k",
BAMBUDDY_URL: "http://bambuddy:8000",
HOMEBOX_URL: "http://homebox:7745",
HOMEBOX_USER: "u",
HOMEBOX_PASS: "p",
SYNC_KAVITA_MINUTES: "720",
SYNC_SUWAYOMI_HEALTHCHECK_URL: "https://hc/s",
};
const jobs = enabledJobs(syncSettings(env));
expect(jobs.map((j) => [j.name, j.everyMinutes])).toEqual([
["suwayomi", 60],
["kavita", 720],
["bambuddy", 60],
["homebox", 60],
]);
expect(jobs[0]!.healthcheckUrl).toBe("https://hc/s");
expect(homelabSettings(env).homebox?.labels).toEqual(parseFilters("leather,paper"));
expect(homelabSettings({ KAVITA_URL: "http://k" }).kavita).toBeUndefined();
expect(() => homelabSettings({ SUWAYOMI_URL: "http://s", SUWAYOMI_AUTH: "simple_login" })).toThrow(/SUWAYOMI_AUTH/);
});
});
services/content/test/sync/scheduler.test.ts
+204
View File
@@ -0,0 +1,204 @@
import type { AddressInfo } from "node:net";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { parseTokens } from "../../src/config.ts";
import { createHttpServer } from "../../src/http.ts";
import { Scheduler } from "../../src/sync/scheduler.ts";
import { syncSettings } from "../../src/sync/settings.ts";
import { enabledJobs } from "../../src/sync/index.ts";
import type { SyncContext, SyncJob, SyncResult } from "../../src/sync/types.ts";
import { book, rig, type Rig } from "../helpers.ts";
let r: Rig;
beforeEach(() => {
r = rig();
});
afterEach(() => {
vi.useRealTimers();
r.done();
});
const pings: string[] = [];
const fakeFetch = (async (url: string | URL | Request) => {
pings.push(String(url));
return new Response("OK");
}) as typeof fetch;
const job = (name: string, run: (ctx: SyncContext) => Promise<SyncResult>, extra: Partial<SyncJob> = {}): SyncJob => ({
name,
everyMinutes: 60,
run,
...extra,
});
const deferred = () => {
let resolve!: (v: SyncResult) => void;
const promise = new Promise<SyncResult>((ok) => (resolve = ok));
return { promise, resolve };
};
describe("scheduler", () => {
it("reports each run to sync_runs and pings Healthchecks start, success and fail", async () => {
pings.length = 0;
let fail = false;
const s = new Scheduler(
r.service,
[
job("demo", async () => {
if (fail) throw new Error("source down");
return { message: "2 synced", counts: { synced: 2 } };
}, { healthcheckUrl: "https://hc-ping.com/abc/" }),
],
{ fetch: fakeFetch, log: () => {} },
);
expect(await s.runNow("demo")).toMatchObject({ ok: true, message: "2 synced" });
fail = true;
expect(await s.runNow("demo")).toMatchObject({ ok: false, message: "source down" });
expect(r.store.syncRuns("demo").map((x) => [x.ok, x.message])).toEqual([
[false, "source down"],
[true, "2 synced"],
]);
expect(r.store.syncJobs()[0]).toMatchObject({ job: "demo", ok: false, counts: {} });
expect(pings).toEqual(["https://hc-ping.com/abc/start", "https://hc-ping.com/abc", "https://hc-ping.com/abc/start", "https://hc-ping.com/abc/fail"]);
});
it("never runs one job twice at once", async () => {
const d = deferred();
let runs = 0;
const s = new Scheduler(r.service, [job("slow", () => (runs++, d.promise))], { fetch: fakeFetch, log: () => {} });
const first = s.runNow("slow");
expect(await s.runNow("slow")).toMatchObject({ skipped: true });
expect(runs).toBe(1);
d.resolve({});
await first;
await Promise.resolve();
expect(s.isRunning("slow")).toBe(false);
});
it("aborts a run that passes its timeout and stays locked until the job really ends", async () => {
vi.useFakeTimers();
const d = deferred();
let aborted = false;
const s = new Scheduler(
r.service,
[
job("stuck", (ctx) => {
ctx.signal.addEventListener("abort", () => (aborted = true));
return d.promise;
}, { timeoutMs: 1000 }),
],
{ fetch: fakeFetch, log: () => {} },
);
const run = s.runNow("stuck");
await vi.advanceTimersByTimeAsync(1001);
expect(await run).toMatchObject({ ok: false, message: "Timed out after 1 seconds" });
expect(aborted).toBe(true);
expect(s.isRunning("stuck")).toBe(true);
d.resolve({});
await vi.advanceTimersByTimeAsync(0);
expect(s.isRunning("stuck")).toBe(false);
});
it("runs each job at start, staggered, then on its interval", async () => {
vi.useFakeTimers();
const ran: string[] = [];
const mk = (name: string) => job(name, async () => (ran.push(name), {}), { everyMinutes: 10 });
const s = new Scheduler(r.service, [mk("a"), mk("b")], { staggerMs: 1000, fetch: fakeFetch, log: () => {} });
s.start();
await vi.advanceTimersByTimeAsync(0);
expect(ran).toEqual(["a"]);
await vi.advanceTimersByTimeAsync(1000);
expect(ran).toEqual(["a", "b"]);
await vi.advanceTimersByTimeAsync(10 * 60_000);
expect(ran).toEqual(["a", "b", "a", "b"]);
s.stop();
await vi.advanceTimersByTimeAsync(60 * 60_000);
expect(ran).toHaveLength(4);
});
});
describe("sync cache", () => {
it("keeps the last good values when a run fails, and shows them on the notes API", async () => {
const path = r.put(book("Mistborn"));
r.service.scan();
let fail = false;
const s = new Scheduler(
r.service,
[
job("demo", async (ctx) => {
if (fail) throw new Error("down");
ctx.set(path, { cover: "https://example.com/a.jpg", pages: 541 });
return {};
}),
],
{ fetch: fakeFetch, log: () => {} },
);
await s.runNow("demo");
const first = r.store.syncCache.get("demo", path);
fail = true;
await s.runNow("demo");
expect(r.store.syncCache.get("demo", path)).toEqual(first);
// Values from several jobs merge per note.
r.store.syncCache.set("other", path, { year: 2006 }, "2026-10-11T00:00:00Z");
const token = "r".repeat(24);
const server = createHttpServer(r.service, { tokens: parseTokens(`reader:${token}`) });
await new Promise<void>((ok) => server.listen(0, "127.0.0.1", ok));
const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
const get = (p: string) => fetch(base + p, { headers: { authorization: `Bearer ${token}` } }).then((x) => x.json());
try {
const [listed] = (await get("/api/notes?full=true")) as { synced: unknown }[];
expect(listed!.synced).toEqual({ cover: "https://example.com/a.jpg", pages: 541, year: 2006 });
expect(((await get(`/api/note?path=${encodeURIComponent(path)}`)) as { synced: unknown }).synced).toMatchObject({ pages: 541 });
expect(await get("/api/notes")).toEqual([expect.not.objectContaining({ synced: expect.anything() })]);
} finally {
server.close();
}
});
it("retain drops values for notes that left a job's scope", () => {
r.store.syncCache.set("j", "a.md", { x: 1 }, "t");
r.store.syncCache.set("j", "b.md", { x: 2 }, "t");
r.store.syncCache.set("k", "b.md", { y: 3 }, "t");
r.store.syncCache.retain("j", ["a.md"]);
expect([...r.store.syncCache.all().entries()]).toEqual([
["a.md", { x: 1 }],
["b.md", { y: 3 }],
]);
});
});
describe("writing notes as sync", () => {
it("runs the agent rules: a note Greg wrote is held unless it allows agent edits", async () => {
const mine = r.put(book("Mine", { progress: "1/10" }));
const open = r.put(book("Open", { progress: "1/10", agentEdits: true }));
r.service.scan();
const outcomes: string[] = [];
const s = new Scheduler(
r.service,
[
job("reader", async (ctx) => {
for (const p of [mine, open]) outcomes.push(ctx.apply({ op: "update", path: p, fields: { progress: "2/10" } }).status);
outcomes.push(ctx.apply({ op: "update", path: open, fields: { note: "Mail me at someone@example.com" } }).status);
return {};
}),
],
{ fetch: fakeFetch, log: () => {} },
);
await s.runNow("reader");
expect(outcomes).toEqual(["held", "accepted", "held"]);
expect(r.store.heldList()[0]!.reasons.map((x) => x.code)).toContain("contact");
expect(r.store.get(open)!.data).toMatchObject({ progress: "2/10", updatedBy: "sync" });
expect(r.store.get(open)!.author).toBe("sync");
});
});
describe("settings", () => {
it("enables only the jobs whose config is present", () => {
const s = syncSettings({ SYNC_JOBS: "bgg, covers", SYNC_BGG_MINUTES: "720", SYNC_COVERS_HEALTHCHECK_URL: "https://hc/x", BGG_TOKEN: "t" });
expect(enabledJobs(s).map((j) => [j.name, j.everyMinutes, j.healthcheckUrl])).toEqual([
["bgg", 720, undefined],
["covers", 15, "https://hc/x"],
]);
expect(enabledJobs(syncSettings({}))).toEqual([]);
});
});
services/content/test/sync/sources.test.ts
+165
View File
@@ -0,0 +1,165 @@
/**
* Each source's parsing and its job, against the hand-written sample payloads
* in ./fixtures (documented shapes, not live responses; see fixtures/README.md).
*/
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { archidektJob, parseDeck, scryfallArt, type RawDeck } from "../../src/sync/jobs/archidekt.ts";
import { bggJob, parseThings } from "../../src/sync/jobs/bgg.ts";
import { coversJob, parseAniList, parseOpenLibrary, parseTmdb } from "../../src/sync/jobs/covers.ts";
import { Scheduler } from "../../src/sync/scheduler.ts";
import type { SyncJob } from "../../src/sync/types.ts";
import { book, rig, type Rig } from "../helpers.ts";
const fixture = (name: string) => readFileSync(join(import.meta.dirname, "fixtures", name), "utf8");
const json = (name: string) => JSON.parse(fixture(name));
type Call = { url: string; init?: RequestInit };
function fakeFetch(answer: (url: string, init?: RequestInit) => Response | Promise<Response>) {
const calls: Call[] = [];
const fn = (async (input: string | URL | Request, init?: RequestInit) => {
const url = String(input);
calls.push({ url, init });
return answer(url, init);
}) as typeof fetch;
return { fn, calls };
}
const ok = (body: unknown, status = 200) =>
new Response(typeof body === "string" ? body : JSON.stringify(body), { status });
describe("Archidekt", () => {
it("parses commander art, color identity and a list summary", () => {
const v = parseDeck(json("archidekt-deck.sample.json") as RawDeck);
expect(v).toEqual({
deckUrl: "https://archidekt.com/decks/1234567",
commanderArt: scryfallArt("d0d33d52-3d28-4635-b985-51e126289259"),
colors: ["W", "U", "B", "G"],
commanders: ["Atraxa, Praetors' Voice"],
deckSize: 36,
format: "Commander",
});
expect(v.commanderArt).toBe("https://cards.scryfall.io/art_crop/front/d/0/d0d33d52-3d28-4635-b985-51e126289259.jpg");
});
it("falls back to the featured image and the whole list's colors without a commander", () => {
const v = parseDeck({
id: 5,
featured: "http://example.com/f.jpg",
cards: [{ quantity: 4, categories: ["Main"], card: { oracleCard: { name: "Bolt", colorIdentity: ["R"] } } }],
});
expect(v).toMatchObject({ commanderArt: "https://example.com/f.jpg", colors: ["R"], commanders: [], deckSize: 4 });
});
});
describe("BoardGameGeek", () => {
it("parses players, play time and box art per id", () => {
const m = parseThings(fixture("bgg-thing.sample.xml"));
expect(m.get(266192)).toEqual({
minPlayers: 1,
maxPlayers: 5,
playTime: 70,
minPlayTime: 40,
maxPlayTime: 70,
cover: "https://cf.geekdo-images.com/sample__original/img/wingspan.jpg",
});
expect(m.get(13)?.cover).toBe("https://cf.geekdo-images.com/sample__original/img/catan.jpg?a=1&b=2");
// Zeros mean BGG has no value: left out, so nothing wrong shows.
expect(m.get(99)).toEqual({});
});
});
describe("cover art", () => {
it("parses Open Library, TMDB and AniList answers", () => {
expect(parseOpenLibrary(json("openlibrary-search.sample.json"))).toBe("https://covers.openlibrary.org/b/id/8221255-L.jpg");
expect(parseTmdb(json("tmdb-search.sample.json"))).toBe("https://image.tmdb.org/t/p/w500/x2FJsf1ElAgr63Y3PNPtJrcmpoe.jpg");
expect(parseAniList(json("anilist-media.sample.json"))).toBe(
"https://s4.anilist.co/file/anilistcdn/media/manga/cover/large/sample-xl.jpg",
);
expect(parseOpenLibrary({ docs: [] })).toBeNull();
expect(parseAniList({ data: { Media: null } })).toBeNull();
});
});
// ---------------------------------------------------------------- the jobs, end to end
let r: Rig;
beforeEach(() => {
r = rig();
});
afterEach(() => r.done());
const scheduler = (job: SyncJob, fetchFn: typeof fetch) =>
new Scheduler(r.service, [job], { fetch: fetchFn, now: () => new Date(r.clock.t), log: () => {} });
describe("jobs", () => {
it("Archidekt syncs each deck with an id and forgets decks that lose it", async () => {
r.put({ type: "card-game", title: "Magic", publish: true, stance: "Main event" });
const atraxa = r.put({ type: "deck", title: "Atraxa", game: "[[Magic]]", archidektId: 1234567, publish: true });
r.put({ type: "deck", title: "Paper only", game: "[[Magic]]", listUrl: "https://example.com/list", publish: true });
r.service.scan();
const f = fakeFetch(() => ok(fixture("archidekt-deck.sample.json")));
const report = await scheduler(archidektJob({ delayMs: 0 }), f.fn).runNow("archidekt");
expect(report).toMatchObject({ ok: true, counts: { decks: 1, synced: 1, failed: 0 } });
expect(f.calls.map((c) => c.url)).toEqual(["https://archidekt.com/api/decks/1234567/"]);
expect(r.store.syncCache.forPath(atraxa)).toMatchObject({ colors: ["W", "U", "B", "G"] });
});
it("BoardGameGeek batches 20 ids a request, sends the token and waits out a 202", async () => {
for (let i = 1; i <= 25; i++) r.put({ type: "board-game", title: `Game ${i}`, bggId: i === 1 ? 266192 : 1000 + i, blurb: "Fun.", publish: true });
r.service.scan();
let queued = true;
const f = fakeFetch(() => {
if (queued) {
queued = false;
return ok("", 202);
}
return ok(fixture("bgg-thing.sample.xml"));
});
const report = await scheduler(bggJob({ token: "app-token", delayMs: 0 }), f.fn).runNow("bgg");
expect(f.calls).toHaveLength(3); // 202, then the first 20, then the last 5
const ids = (c: Call) => new URL(c.url).searchParams.get("id")!.split(",");
expect(f.calls[1]!.url).toMatch(/^https:\/\/boardgamegeek\.com\/xmlapi2\/thing\?id=/);
expect([ids(f.calls[1]!).length, ids(f.calls[2]!).length]).toEqual([20, 5]);
expect(new Set([...ids(f.calls[1]!), ...ids(f.calls[2]!)]).size).toBe(25);
expect((f.calls[0]!.init?.headers as Record<string, string>).authorization).toBe("Bearer app-token");
// Only Wingspan is in the sample answer; the rest count as not synced.
expect(report).toMatchObject({ ok: false, counts: { games: 25, synced: 1, failed: 24 } });
expect(r.store.syncCache.forPath("Shelf/Game 1.md")).toMatchObject({ minPlayers: 1, maxPlayers: 5, playTime: 70 });
});
it("covers: picks the source by medium, skips TMDB without a key, keeps a found cover on a miss", async () => {
const b = r.put(book("The Way of Kings", { creator: "Brandon Sanderson" }));
const m = r.put(book("One Piece", { medium: "manga", anilistId: 30013 }));
r.put(book("Arrival", { medium: "movie" }));
r.put(book("Has its own", { cover: "https://example.com/own.jpg" }));
r.service.scan();
const f = fakeFetch((url) =>
url.startsWith("https://openlibrary.org/") ? ok(fixture("openlibrary-search.sample.json")) : ok(fixture("anilist-media.sample.json")),
);
const s = scheduler(coversJob({ delayMs: 0 }), f.fn);
expect(await s.runNow("covers")).toMatchObject({ ok: true, counts: { checked: 2, found: 2 } });
expect(f.calls.map((c) => new URL(c.url).host).sort()).toEqual(["graphql.anilist.co", "openlibrary.org"]);
expect(f.calls.find((c) => c.url.includes("openlibrary"))!.url).toContain("author=Brandon+Sanderson");
expect(JSON.parse(String(f.calls.find((c) => c.url.includes("anilist"))!.init?.body)).variables).toEqual({ type: "MANGA", id: 30013 });
expect(r.store.syncCache.forPath(b).cover).toBe("https://covers.openlibrary.org/b/id/8221255-L.jpg");
// Nothing is due again until its retry time.
expect(await s.runNow("covers")).toMatchObject({ counts: { checked: 0 } });
// A month on, the sources find nothing: the earlier covers stay.
r.clock.advance(31 * 24 * 60);
const empty = fakeFetch(() => ok({ docs: [], data: { Media: null } }));
await scheduler(coversJob({ delayMs: 0 }), empty.fn).runNow("covers");
expect(r.store.syncCache.forPath(m).cover).toMatch(/anilist/);
});
it("covers: TMDB with a key", async () => {
const a = r.put(book("Arrival", { medium: "movie", year: 2016 }));
r.service.scan();
const f = fakeFetch(() => ok(fixture("tmdb-search.sample.json")));
await scheduler(coversJob({ delayMs: 0, tmdbApiKey: "k".repeat(32) }), f.fn).runNow("covers");
expect(f.calls[0]!.url).toMatch(/^https:\/\/api\.themoviedb\.org\/3\/search\/movie\?query=Arrival&include_adult=false&year=2016&api_key=k+$/);
expect(r.store.syncCache.forPath(a).cover).toMatch(/image\.tmdb\.org/);
});
});
services/site/.env.example
+36
View File
@@ -0,0 +1,36 @@
# Site settings. Set them in the container's environment (never commit real
# values). Read from process.env at run time only. See docs/site.md.
# --- Content service ----------------------------------------------------------
CONTENT_URL=http://localhost:4310
# Reader token: published notes, and the live store check for orders.
CONTENT_TOKEN=
# Intake token: files saved copies of inquiries and orders, can read nothing.
CONTENT_INTAKE_TOKEN=
# --- Forms: Cloudflare Turnstile ---------------------------------------------
# The widget shows on both forms when the site key is set.
PUBLIC_TURNSTILE_SITE_KEY=
# When set, a post without a valid token is turned down (fails closed if
# Cloudflare cannot be reached). Unset in local development.
TURNSTILE_SECRET_KEY=
# --- Forms: email (nodemailer over SMTP) --------------------------------------
# Without SMTP_HOST no email is sent; the saved copy in the admin Inbox stands.
SMTP_HOST=
SMTP_PORT=587
SMTP_USER=
SMTP_PASS=
SMTP_FROM=GregKilmer.com <no-reply@gregkilmer.com>
# Only for a relay on the same private network with a self-signed certificate.
SMTP_ALLOW_SELFSIGNED=false
CONTACT_TO=contact@gregkilmer.com
ORDER_TO=orders@gregkilmer.com
# --- Forms: limits --------------------------------------------------------------
# Posts per visitor address per window.
CONTACT_RATE_LIMIT=5
CONTACT_RATE_WINDOW_MIN=15
# ORDER_RATE_LIMIT and ORDER_RATE_WINDOW_MIN default to the CONTACT_ values.
# Largest inquiry attachment in MB; keep it at or under the content service's SUBMISSION_MAX_MB.
FORM_MAX_ATTACHMENT_MB=10
services/site/package.json
+1
View File
@@ -16,6 +16,7 @@
"@gk/ui": "*",
"astro": "7.3.5",
"marked": "18.0.14",
"nodemailer": "10.0.10",
"pdf-lib": "^1.17.1",
"pdfkit": "^0.20.2"
},
services/site/src/components/library/GameCard.astro
+9 -1
View File
@@ -11,7 +11,7 @@ import StarChartArt from "@gk/ui/components/StarChartArt.astro";
import StatusPill from "@gk/ui/components/StatusPill.astro";
import StatusChip from "../StatusChip.astro";
import { formatDate, mediaSrc, type SiteIndex } from "../../lib/site.ts";
import type { Game } from "../../lib/library.ts";
import { gamePlayers, gamePlayTime, type Game } from "../../lib/library.ts";
interface Props {
game: Game;
@@ -22,6 +22,8 @@ const { game, site, hidden } = Astro.props;
const d = game.data;
const plays = d.howItPlays ?? [];
const status = site.statusFor(game);
// Players and play time sync from BoardGameGeek; nothing shows until they have.
const facts = [gamePlayers(game), gamePlayTime(game)].filter(Boolean);
---
<li
id={game.slug}
@@ -43,6 +45,7 @@ const status = site.statusFor(game);
{d.favorite && <><Star size={12} /><span class="gk-visually-hidden">Favorite</span></>}
</h3>
<p class="blurb">{d.blurb}</p>
{facts.length > 0 && <p class="facts">{facts.join(" · ")}</p>}
{
(status || d.wantToPlay || !d.owned || d.lastPlayed) && (
<div class="states">
@@ -110,6 +113,11 @@ const status = site.statusFor(game);
line-height: 1.45;
color: var(--gk-muted);
}
.facts {
margin: 0;
font-size: 12px;
color: var(--gk-soft);
}
.states {
display: flex;
flex-wrap: wrap;
services/site/src/components/library/LibraryRow.astro
+2 -2
View File
@@ -10,7 +10,7 @@ import Star from "@gk/ui/components/Star.astro";
import StarChartArt from "@gk/ui/components/StarChartArt.astro";
import StatusChip from "../StatusChip.astro";
import { mediaSrc, type SiteIndex } from "../../lib/site.ts";
import { mediaGroupOf, mediumLabel, sectionOf, type Item } from "../../lib/library.ts";
import { itemDetails, mediaGroupOf, mediumLabel, sectionOf, type Item } from "../../lib/library.ts";
interface Props {
item: Item;
@@ -22,7 +22,7 @@ const d = item.data;
const status = site.statusFor(item);
const post = site.backlinks(item, { type: "post" })[0];
const postHref = post && site.href(post);
const details = [d.creator, d.year, d.platform, d.progress, d.group].filter((x) => x !== undefined && x !== "");
const details = itemDetails(item);
---
<li
id={item.slug}
services/site/src/components/store/OrderForm.astro
+50 -16
View File
@@ -2,12 +2,15 @@
/**
* The order request form. Posts to /api/orders. Field names:
* name, email, line (a line slug), option.<line slug>.<group> (an option
* name), quantity, details, notes, terms ("accepted").
* name), quantity, details, notes, terms ("accepted"), plus the honeypot and,
* when a Turnstile site key is set, the widget's cf-turnstile-response.
* `values` and `errors` refill the form after a post was turned down.
* Without JS every line's options show, labelled with their line; a small
* script shows only the chosen line's options and terms.
*/
import Button from "@gk/ui/components/Button.astro";
import Field from "@gk/ui/components/Field.astro";
import { HONEYPOT } from "../../lib/forms.ts";
import Star from "@gk/ui/components/Star.astro";
import { optionGroups, type Line, type Term } from "../../lib/store.ts";
import Terms from "./Terms.astro";
@@ -16,20 +19,23 @@ interface Props {
lines: Line[];
selected?: string;
storeTerms: Term[];
values?: Record<string, string>;
errors?: Record<string, string>;
turnstileKey?: string;
}
const { lines, selected, storeTerms } = Astro.props;
const { lines, selected, storeTerms, values: v = {}, errors: err = {}, turnstileKey } = Astro.props;
const chosen = lines.find((l) => l.slug === selected)?.slug ?? (lines.length === 1 ? lines[0]!.slug : undefined);
const slugPart = (x: string) => x.toLowerCase().replace(/[^a-z0-9]+/g, "-");
---
<form method="post" action="/api/orders" class="order" aria-labelledby="order-title" data-order-form>
<h2 id="order-title" class="gk-visually-hidden">Order request form</h2>
<div class="row">
<Field label="Name" name="name" autocomplete="name" required class="half" />
<Field label="Email" name="email" type="email" autocomplete="email" required class="half" />
<Field label="Name" name="name" autocomplete="name" required maxlength={120} class="half" value={v.name} error={err.name} />
<Field label="Email" name="email" type="email" autocomplete="email" required maxlength={200} class="half" value={v.email} error={err.email} />
</div>
<fieldset>
<legend>What would you like?</legend>
<legend>What would you like?{err.line && <span class="err"> {err.line}</span>}</legend>
<div class="choices">
{
lines.map((l) => (
@@ -49,11 +55,12 @@ const slugPart = (x: string) => x.toLowerCase().replace(/[^a-z0-9]+/g, "-");
<fieldset class="opts" data-line={l.slug} hidden={chosen ? l.slug !== chosen : undefined} disabled={chosen ? l.slug !== chosen : undefined}>
<legend>
{group} {lines.length > 1 && <span class="for">({l.title} only)</span>}
{err[`option.${l.slug}.${group}`] && <span class="err"> {err[`option.${l.slug}.${group}`]}</span>}
</legend>
<div class="choices">
{opts.map((o, i) => (
<label class:list={["gk-filter", "swatch", { out: !o.inStock }]}>
<input type="radio" name={`option.${l.slug}.${group}`} value={o.name} disabled={!o.inStock} id={`opt-${l.slug}-${slugPart(group)}-${i}`} />
<input type="radio" name={`option.${l.slug}.${group}`} value={o.name} disabled={!o.inStock} checked={o.inStock && v[`option.${l.slug}.${group}`] === o.name} id={`opt-${l.slug}-${slugPart(group)}-${i}`} />
{o.hex && <span class="dot" style={`background:${o.hex}`} aria-hidden="true" />}
{o.name}
{!o.inStock && <span class="stock">(out of stock)</span>}
@@ -65,16 +72,18 @@ const slugPart = (x: string) => x.toLowerCase().replace(/[^a-z0-9]+/g, "-");
)
}
<Field label="Approximate quantity" name="quantity" placeholder="e.g. 75 cards, or 1 deck box" required />
<Field
label="Card list or deck box spec"
name="details"
as="textarea"
rows={6}
hint="For cards, paste your full card list, one per line."
class="big"
/>
<Field label="Notes (optional)" name="notes" as="textarea" rows={3} />
<Field label="Approximate quantity" name="quantity" placeholder="e.g. 75 cards, or 1 deck box" required maxlength={200} value={v.quantity} error={err.quantity} />
<div class="gk-label big">
<label class="gk-label-text" for="f-details">Card list or deck box spec</label>
<textarea id="f-details" name="details" class="gk-field" rows={6} maxlength={20000} aria-describedby="f-details-hint">{v.details ?? ""}</textarea>
<span class="gk-hint" id="f-details-hint">For cards, paste your full card list, one per line.</span>
{err.details && <span class="err">{err.details}</span>}
</div>
<div class="gk-label">
<label class="gk-label-text" for="f-notes">Notes (optional)</label>
<textarea id="f-notes" name="notes" class="gk-field" rows={3} maxlength={5000}>{v.notes ?? ""}</textarea>
{err.notes && <span class="err">{err.notes}</span>}
</div>
{
(lines.some((l) => l.data.terms?.length) || storeTerms.length > 0) && (
@@ -102,9 +111,18 @@ const slugPart = (x: string) => x.toLowerCase().replace(/[^a-z0-9]+/g, "-");
<input type="checkbox" name="terms" value="accepted" required />
<span>I have read and accept the order terms for this request.</span>
</label>
{err.terms && <span class="err">{err.terms}</span>}
<div class="hp" aria-hidden="true">
<label>Leave this empty <input type="text" name={HONEYPOT} tabindex="-1" autocomplete="off" /></label>
</div>
{turnstileKey && <div class="cf-turnstile" data-sitekey={turnstileKey} data-theme="dark" data-response-field-name="cf-turnstile-response"></div>}
{turnstileKey && <noscript><span class="tnote">The bot check needs JavaScript. Turn it on for this page to send the form.</span></noscript>}
<div class="submit">
<Button type="submit">Send order request</Button>
{turnstileKey && <span class="tnote">Protected by Cloudflare Turnstile.</span>}
</div>
</form>
@@ -280,6 +298,22 @@ const slugPart = (x: string) => x.toLowerCase().replace(/[^a-z0-9]+/g, "-");
align-items: center;
gap: 12px;
}
.err {
font-size: 12px;
font-weight: 400;
color: #f0857a;
}
.tnote {
font-size: 13px;
color: var(--gk-muted);
}
.hp {
position: absolute;
left: -10000px;
width: 1px;
height: 1px;
overflow: hidden;
}
@media (max-width: 640px) {
.order {
padding: 22px 16px;
services/site/src/lib/content.ts
+19 -1
View File
@@ -20,6 +20,24 @@ export interface Note<T extends NoteType = NoteType> {
data: NoteData<T>;
body: string;
acceptedAt: string;
/**
* What the content service's sync jobs fetched for this note (BoardGameGeek,
* Archidekt, cover art). Already merged under `data` by `withSynced`, so
* pages read `data`; kept here so a page can tell a synced value apart.
*/
synced?: Record<string, unknown>;
}
/**
* Lay the synced values under the note's own: a value set in the note always
* wins, and a note with nothing synced is returned as it came.
*/
export function withSynced<T extends Note>(note: T): T {
const synced = note.synced;
if (!synced || !Object.keys(synced).length) return note;
const data: Record<string, unknown> = { ...synced };
for (const [k, v] of Object.entries(note.data as Record<string, unknown>)) if (v !== undefined && v !== null) data[k] = v;
return { ...note, data: data as T["data"] };
}
export interface Redirect {
@@ -48,7 +66,7 @@ async function loadAll(): Promise<SiteIndex> {
for (let offset = 0; ; offset += 1000) {
const published = includeDrafts() ? "" : "&published=true";
const page = await api<(Note & { publish?: boolean })[]>(`/api/notes?full=true${published}&limit=1000&offset=${offset}`);
notes.push(...page);
notes.push(...page.map(withSynced));
if (page.length < 1000) break;
}
const redirects = await api<Redirect[]>("/api/redirects").catch(() => []);
services/site/src/lib/forms-server.ts
+155
View File
@@ -0,0 +1,155 @@
/**
* What the form endpoints do outside the process: Turnstile, the content
* service (live store notes, saving the copy, recording delivery) and SMTP.
* Settings come from process.env through env(); see services/site/.env.example.
*/
import type { Note } from "./content.ts";
import { env } from "./env.ts";
import { DEFAULT_ATTACHMENT_MB, RateLimiter, type Row, type Upload } from "./forms.ts";
import { SiteIndex } from "./site.ts";
import { storeState, type StoreState } from "./store.ts";
const contentUrl = () => env("CONTENT_URL", "http://localhost:4310").replace(/\/$/, "");
export const maxAttachmentBytes = () => Math.round(Number(env("FORM_MAX_ATTACHMENT_MB", String(DEFAULT_ATTACHMENT_MB))) * 1024 * 1024);
export const inquiryLimiter = new RateLimiter(
() => Number(env("CONTACT_RATE_LIMIT", "5")),
() => Number(env("CONTACT_RATE_WINDOW_MIN", "15")) * 60_000,
);
export const orderLimiter = new RateLimiter(
() => Number(env("ORDER_RATE_LIMIT", env("CONTACT_RATE_LIMIT", "5"))),
() => Number(env("ORDER_RATE_WINDOW_MIN", env("CONTACT_RATE_WINDOW_MIN", "15"))) * 60_000,
);
/** The public site key, for the widget. */
export const turnstileSiteKey = () => env("PUBLIC_TURNSTILE_SITE_KEY") ?? env("TURNSTILE_SITE_KEY");
/**
* Cloudflare Turnstile. Skipped when TURNSTILE_SECRET_KEY is unset (local
* development); when it is set, a missing token, a bad token or an error
* talking to Cloudflare all turn the post down.
*/
export async function verifyTurnstile(token: string, ip: string, fetcher: typeof fetch = fetch): Promise<boolean> {
const secret = env("TURNSTILE_SECRET_KEY");
if (!secret) return true;
if (!token) return false;
try {
const body = new URLSearchParams({ secret, response: token });
if (ip && ip !== "unknown") body.set("remoteip", ip);
const res = await fetcher("https://challenges.cloudflare.com/turnstile/v0/siteverify", {
method: "POST",
body,
signal: AbortSignal.timeout(8000),
});
const data = (await res.json()) as { success?: boolean };
return data.success === true;
} catch (err) {
console.error("Turnstile verification failed; turning the post down.", err);
return false;
}
}
/**
* The store's notes as they are right now, straight from the content
* service rather than the site's cached copy, so an order is checked against
* what Greg last saved. Throws when the service cannot answer.
*/
export async function liveStore(): Promise<StoreState> {
const get = async (type: string) => {
const res = await fetch(`${contentUrl()}/api/notes?type=${type}&full=true&published=true&limit=1000`, {
headers: { authorization: `Bearer ${env("CONTENT_TOKEN", "")}` },
signal: AbortSignal.timeout(8000),
});
if (!res.ok) throw new Error(`Content service answered ${res.status} for ${type}`);
return (await res.json()) as Note[];
};
const [settings, lines] = await Promise.all([get("store-settings"), get("product-line")]);
return storeState(new SiteIndex([...settings, ...lines]));
}
export interface SubmissionInput {
kind: "inquiry" | "order";
type: string;
name: string;
email: string;
fields: Row[];
attachment?: Upload | null;
}
const intake = async (path: string, body: unknown) => {
const token = env("CONTENT_INTAKE_TOKEN");
if (!token) throw new Error("CONTENT_INTAKE_TOKEN is not set");
const res = await fetch(`${contentUrl()}${path}`, {
method: "POST",
headers: { authorization: `Bearer ${token}`, "content-type": "application/json" },
body: JSON.stringify(body),
signal: AbortSignal.timeout(20_000),
});
const data = (await res.json().catch(() => null)) as Record<string, unknown> | null;
if (!res.ok) throw new Error(`Content service answered ${res.status}: ${String(data?.error ?? "")}`);
return data;
};
/** Save the copy with the intake token. Returns the id and the receipt for reporting delivery. */
export async function saveSubmission(s: SubmissionInput): Promise<{ id: number; receipt: string }> {
const attachment = s.attachment ? { name: s.attachment.name, data: Buffer.from(await s.attachment.arrayBuffer()).toString("base64") } : undefined;
const out = (await intake("/api/submissions", { ...s, attachment })) as { id: number; receipt: string };
return { id: out.id, receipt: out.receipt };
}
export async function reportDelivery(id: number, receipt: string, status: "sent" | "skipped" | "failed", error?: string) {
await intake(`/api/submissions/${id}/delivery`, { receipt, status, ...(error ? { error } : {}) });
}
export interface Mail {
to: string;
replyTo: { name: string; address: string };
subject: string;
text: string;
attachment?: Upload | null;
}
/**
* Send through SMTP (nodemailer). "skipped" when SMTP_HOST is unset, so a
* site without mail still saves copies. Throws when sending fails.
*/
export async function sendMail(m: Mail): Promise<"sent" | "skipped"> {
const host = env("SMTP_HOST");
if (!host) return "skipped";
const nodemailer = (await import("nodemailer")).default;
const port = Number(env("SMTP_PORT", "587"));
const user = env("SMTP_USER");
const transport = nodemailer.createTransport({
host,
port,
secure: port === 465,
auth: user ? { user, pass: env("SMTP_PASS", "") } : undefined,
// Only for a relay on the same private network that presents a self-signed certificate.
...(env("SMTP_ALLOW_SELFSIGNED") === "true" ? { tls: { rejectUnauthorized: false } } : {}),
connectionTimeout: 10_000,
greetingTimeout: 10_000,
socketTimeout: 20_000,
});
await transport.sendMail({
from: env("SMTP_FROM", "GregKilmer.com <no-reply@gregkilmer.com>"),
to: m.to,
replyTo: m.replyTo,
subject: m.subject.replace(/[\r\n]+/g, " "),
text: m.text,
attachments: m.attachment ? [{ filename: m.attachment.name, content: Buffer.from(await m.attachment.arrayBuffer()) }] : undefined,
});
return "sent";
}
export const contactTo = () => env("CONTACT_TO", "contact@gregkilmer.com");
export const orderTo = () => env("ORDER_TO", "orders@gregkilmer.com");
/** Short-lived cookie settings for the confirmation and try-again cookies. */
export const flashCookie = (url: URL) => ({
path: "/",
httpOnly: true,
sameSite: "lax" as const,
secure: import.meta.env.PROD || url.protocol === "https:",
maxAge: 600,
});
services/site/src/lib/forms.ts
+476
View File
@@ -0,0 +1,476 @@
/**
* The inquiry and order forms' rules, as plain functions: what a valid post
* looks like, what an order may be accepted for, what the confirmation and
* the "try again" pages are handed, and how saving and emailing combine.
* Nothing here touches the network, so tests run it without Astro.
*/
import type { InquiryType, SentSummary } from "./work.ts";
import { parseInquiryType } from "./work.ts";
import { channelOf, hasCapacity, optionGroups, type Line, type StoreState } from "./store.ts";
export interface Row {
label: string;
value: string;
}
/** What a form post gives: FormData, or anything shaped like it in tests. */
export interface FormLike {
get(name: string): FormDataEntryValue | null;
}
export type FieldErrors = Record<string, string>;
export type Invalid = { ok: false; code: ErrorCode; errors: FieldErrors };
export type ErrorCode = "invalid" | "bot" | "rate" | "closed" | "full" | "unavailable" | "failed";
/** What the visitor reads for each kind of failure. Nothing here leaks internals. */
export const errorMessages: Record<ErrorCode, string> = {
invalid: "Some answers need another look. Nothing was sent yet.",
bot: "The bot check did not go through. Please try again.",
rate: "Too many requests from here in a short time. Please try again a little later.",
closed: "Orders for that are closed right now, so nothing was sent.",
full: "That line has no open slots right now, so nothing was sent.",
unavailable: "The store could not confirm availability just now. Please try again in a few minutes.",
failed: "Your request could not be sent just now. Please try again a little later.",
};
export const parseErrorCode = (raw: string | null | undefined): ErrorCode | undefined =>
raw && raw in errorMessages ? (raw as ErrorCode) : undefined;
const text = (form: FormLike, key: string): string => {
const v = form.get(key);
return typeof v === "string" ? v.replace(/\r\n/g, "\n").trim() : "";
};
const EMAIL = /^[^\s@<>()[\]\\,;:"]+@[^\s@<>()[\]\\,;:"]+\.[^\s@<>()[\]\\,;:"]{2,}$/;
export const isEmail = (s: string) => s.length <= 200 && EMAIL.test(s);
export const isHttpUrl = (s: string) => {
try {
const u = new URL(s);
return (u.protocol === "http:" || u.protocol === "https:") && !!u.hostname;
} catch {
return false;
}
};
const isDate = (s: string) => /^\d{4}-\d{2}-\d{2}$/.test(s) && !Number.isNaN(Date.parse(`${s}T00:00:00Z`));
/** The hidden field bots fill and people never see. */
export const HONEYPOT = "leave_blank";
export const isBot = (form: FormLike) => text(form, HONEYPOT) !== "";
/** The Turnstile widget adds this field to the form it sits in. */
export const TURNSTILE_FIELD = "cf-turnstile-response";
// ---------------------------------------------------------------- attachments
/** Matches the content service's list (services/content/src/submissions.ts). */
export const ATTACHMENT_EXTENSIONS = ["png", "jpg", "jpeg", "gif", "webp", "avif", "heic", "pdf", "txt", "csv", "dek", "dec"];
export const DEFAULT_ATTACHMENT_MB = 10;
export interface Upload {
name: string;
size: number;
type: string;
arrayBuffer(): Promise<ArrayBuffer>;
}
const isUpload = (v: unknown): v is Upload =>
typeof v === "object" && v !== null && typeof (v as Upload).arrayBuffer === "function" && typeof (v as Upload).size === "number";
/** The attached file, or null for none; an error message when it cannot be taken. */
export function checkAttachment(form: FormLike, key: string, maxBytes: number): { file: Upload | null; error?: string } {
const v = form.get(key);
if (!isUpload(v) || v.size === 0) return { file: null };
const ext = /\.([A-Za-z0-9]{1,8})$/.exec(v.name)?.[1]?.toLowerCase();
if (!ext || !ATTACHMENT_EXTENSIONS.includes(ext)) {
return { file: null, error: "Attach an image, a PDF, or a text, CSV or deck list file." };
}
if (v.size > maxBytes) return { file: null, error: `Attachments can be up to ${Math.floor(maxBytes / 1024 / 1024)} MB.` };
return { file: v };
}
// ---------------------------------------------------------------- inquiries
export const choices = {
job_kind: ["Full-time", "Contract", "Not sure yet"],
freelance_need: ["A new site", "An app", "An integration", "Something else"],
patiun_item: ["Proxy cards", "Deck box", "Something else"],
} as const;
export const timing = [
{ value: "no-rush", label: "No rush" },
{ value: "within-a-month", label: "Within a month" },
{ value: "date", label: "A specific date" },
];
type Question =
| { key: string; label: string; kind: "text"; max: number }
| { key: string; label: string; kind: "url" }
| { key: string; label: string; kind: "choice"; options: readonly string[] }
| { key: string; label: string; kind: "timing"; dateKey: string };
/** Each type's own questions, in form order. Fields of other types are ignored. */
export const questions: Record<InquiryType, Question[]> = {
job: [
{ key: "job_company", label: "Company", kind: "text", max: 160 },
{ key: "job_role", label: "Role", kind: "text", max: 160 },
{ key: "job_kind", label: "Kind of role", kind: "choice", options: choices.job_kind },
{ key: "job_link", label: "Posting", kind: "url" },
],
freelance: [
{ key: "freelance_need", label: "Needs", kind: "choice", options: choices.freelance_need },
{ key: "freelance_site", label: "Current site", kind: "url" },
{ key: "freelance_timing", label: "Timeline", kind: "timing", dateKey: "freelance_date" },
],
patiun: [
{ key: "patiun_item", label: "Item", kind: "choice", options: choices.patiun_item },
{ key: "patiun_quantity", label: "How many", kind: "text", max: 120 },
{ key: "patiun_timing", label: "Need it by", kind: "timing", dateKey: "patiun_date" },
],
};
export interface Inquiry {
type: InquiryType;
name: string;
email: string;
details: string;
/** The chosen type's answers, in form order, empty ones left out. */
rows: Row[];
attachment: Upload | null;
}
const formatDay = (d: string) => new Date(`${d}T00:00:00Z`).toLocaleDateString("en-US", { month: "short", day: "numeric", year: "numeric", timeZone: "UTC" });
export function validateInquiry(form: FormLike, opts: { maxBytes: number }): { ok: true; value: Inquiry } | Invalid {
const errors: FieldErrors = {};
const type = parseInquiryType(text(form, "type"));
if (!type) errors.type = "Pick what this is about.";
const name = text(form, "name");
if (!name) errors.name = "Add your name.";
else if (name.length > 120) errors.name = "Keep your name under 120 characters.";
const email = text(form, "email");
if (!isEmail(email)) errors.email = "Add an email address I can reply to.";
const details = text(form, "details");
if (details.length < 10) errors.details = "Tell me a little more: a sentence or two is plenty.";
else if (details.length > 5000) errors.details = "Keep the details under 5,000 characters, or attach a file.";
const rows: Row[] = [];
for (const q of type ? questions[type] : []) {
const v = text(form, q.key);
if (!v) continue;
if (q.kind === "text") {
if (v.length > q.max) errors[q.key] = `Keep this under ${q.max} characters.`;
else rows.push({ label: q.label, value: v });
} else if (q.kind === "url") {
if (v.length > 500 || !isHttpUrl(v)) errors[q.key] = "Use a full link starting with https://.";
else rows.push({ label: q.label, value: v });
} else if (q.kind === "choice") {
if (!q.options.includes(v)) errors[q.key] = "Pick one of the options.";
else rows.push({ label: q.label, value: v });
} else {
const t = timing.find((x) => x.value === v);
if (!t) errors[q.key] = "Pick one of the options.";
else if (t.value === "date") {
const d = text(form, q.dateKey);
if (!isDate(d)) errors[q.dateKey] = "Pick the date.";
else rows.push({ label: q.label, value: formatDay(d) });
} else rows.push({ label: q.label, value: t.label });
}
}
const att = checkAttachment(form, "attachment", opts.maxBytes);
if (att.error) errors.attachment = att.error;
if (Object.keys(errors).length || !type) return { ok: false, code: "invalid", errors };
return { ok: true, value: { type, name, email, details, rows, attachment: att.file } };
}
const inquiryTypeLabels: Record<InquiryType, string> = { job: "A job or role", freelance: "Freelance web work", patiun: "Patiun custom order" };
/** Everything the saved copy and the email carry, in order. */
export const inquiryFields = (q: Inquiry): Row[] => [...q.rows, { label: "Details", value: q.details }];
export function inquiryEmail(q: Inquiry, savedId: number | null): { subject: string; text: string } {
return {
subject: `Inquiry: ${inquiryTypeLabels[q.type]} from ${q.name}`,
text: [
`New inquiry through gregkilmer.com`,
``,
`Type: ${inquiryTypeLabels[q.type]}`,
`Name: ${q.name}`,
`Email: ${q.email}`,
...q.rows.map((r) => `${r.label}: ${r.value}`),
``,
`Details:`,
q.details,
``,
q.attachment ? `Attached: ${q.attachment.name}` : "",
savedId ? `Saved in the admin inbox as #${savedId}.` : `Not saved in the admin inbox: the content service did not take it. This email is the only copy.`,
].join("\n"),
};
}
/** The `gk-inquiry` cookie the sent page reads: what the visitor sent, briefly. */
export function inquirySent(q: Inquiry): SentSummary {
const short = (v: string) => (v.length > 120 ? `${v.slice(0, 119)}…` : v);
return {
type: q.type,
name: q.name,
email: q.email,
rows: [...q.rows.map((r) => ({ label: r.label, value: short(r.value) })), ...(q.attachment ? [{ label: "Attachment", value: short(q.attachment.name) }] : [])].slice(0, 6),
};
}
// ---------------------------------------------------------------- orders
export interface OrderRequest {
name: string;
email: string;
line: string;
quantity: string;
details: string;
notes: string;
}
/** Check the fields that do not depend on the store. */
export function validateOrderFields(form: FormLike): { ok: true; value: OrderRequest } | Invalid {
const errors: FieldErrors = {};
const name = text(form, "name");
if (!name) errors.name = "Add your name.";
else if (name.length > 120) errors.name = "Keep your name under 120 characters.";
const email = text(form, "email");
if (!isEmail(email)) errors.email = "Add an email address I can reply to.";
const line = text(form, "line");
if (!/^[a-z0-9-]{1,120}$/.test(line)) errors.line = "Pick what you would like.";
const quantity = text(form, "quantity");
if (!quantity) errors.quantity = "Give a rough quantity.";
else if (quantity.length > 200) errors.quantity = "Keep this under 200 characters.";
const details = text(form, "details");
if (details.length > 20_000) errors.details = "Keep this under 20,000 characters.";
const notes = text(form, "notes");
if (notes.length > 5000) errors.notes = "Keep notes under 5,000 characters.";
if (text(form, "terms") !== "accepted") errors.terms = "Read and accept the order terms to send a request.";
if (Object.keys(errors).length) return { ok: false, code: "invalid", errors };
return { ok: true, value: { name, email, line, quantity, details, notes } };
}
export interface CheckedOrder extends OrderRequest {
lineNote: Line;
/** The chosen options, as { label: group, value: option }. */
options: Row[];
}
/**
* Whether the store takes this order now, against the live notes. An order
* is accepted only for a line sold here, in a category with ordersOpen true,
* with an open slot or stock, and only with options that exist and are in
* stock. Anything else is turned down, whatever the form said.
*/
export function checkOrder(req: OrderRequest, form: FormLike, st: StoreState): { ok: true; value: CheckedOrder } | Invalid {
const line = st.lines.find((l) => l.slug === req.line);
if (!line || channelOf(line) !== "here") return { ok: false, code: "invalid", errors: { line: "That is not something you can order here." } };
if (!st.category(line.data.category).ordersOpen) return { ok: false, code: "closed", errors: {} };
if (!hasCapacity(line)) return { ok: false, code: "full", errors: {} };
const options: Row[] = [];
const errors: FieldErrors = {};
for (const [group, opts] of optionGroups(line)) {
const key = `option.${line.slug}.${group}`;
const v = text(form, key);
if (!v) continue;
const o = opts.find((x) => x.name === v);
if (!o) errors[key] = `Pick one of the ${group.toLowerCase()} options.`;
else if (!o.inStock) errors[key] = `${o.name} is out of stock. Pick another.`;
else options.push({ label: group, value: o.name });
}
if (Object.keys(errors).length) return { ok: false, code: "invalid", errors };
return { ok: true, value: { ...req, lineNote: line, options } };
}
export const orderFields = (o: CheckedOrder): Row[] => [
{ label: "Line", value: o.lineNote.title },
...o.options,
{ label: "Quantity", value: o.quantity },
...(o.details ? [{ label: "Details", value: o.details }] : []),
...(o.notes ? [{ label: "Notes", value: o.notes }] : []),
{ label: "Terms", value: "Accepted" },
];
export function orderEmail(o: CheckedOrder, savedId: number | null): { subject: string; text: string } {
return {
subject: `Order request: ${o.lineNote.title} from ${o.name}`,
text: [
`New order request through gregkilmer.com. No payment was taken.`,
``,
`Name: ${o.name}`,
`Email: ${o.email}`,
...orderFields(o).flatMap((r) => (r.value.includes("\n") ? [``, `${r.label}:`, r.value, ``] : [`${r.label}: ${r.value}`])),
``,
savedId ? `Saved in the admin inbox as #${savedId}.` : `Not saved in the admin inbox: the content service did not take it. This email is the only copy.`,
].join("\n"),
};
}
export interface OrderSent {
name: string;
email: string;
line: string;
lineSlug: string;
rows: Row[];
}
/** The `gk-order` cookie the order confirmation reads. */
export function orderSent(o: CheckedOrder): OrderSent {
const short = (v: string) => (v.length > 120 ? `${v.slice(0, 119)}…` : v);
return {
name: o.name,
email: o.email,
line: o.lineNote.title,
lineSlug: o.lineNote.slug,
rows: [...o.options, { label: "Quantity", value: o.quantity }].map((r) => ({ label: r.label, value: short(r.value) })).slice(0, 6),
};
}
export function parseOrderSent(raw: string | undefined): OrderSent | undefined {
if (!raw) return undefined;
try {
const v = JSON.parse(raw.startsWith("%7B") || raw.startsWith("%7b") ? decodeURIComponent(raw) : raw) as Record<string, unknown>;
const str = (x: unknown) => (typeof x === "string" && x.trim() ? x.trim().slice(0, 300) : "");
const rows = Array.isArray(v.rows)
? v.rows
.map((r) => ({ label: str((r as Row)?.label), value: str((r as Row)?.value) }))
.filter((r) => r.label && r.value)
.slice(0, 8)
: [];
return { name: str(v.name), email: str(v.email), line: str(v.line), lineSlug: str(v.lineSlug), rows };
} catch {
return undefined;
}
}
// ---------------------------------------------------------------- try again
/**
* What goes back to the form when a post is turned down: the error, which
* fields to mark, and the visitor's answers. It travels as a short-lived
* cookie, so it has to fit: the longest answers are cut first.
*/
export interface Draft {
code: ErrorCode;
errors: FieldErrors;
values: Record<string, string>;
}
export const DRAFT_LIMIT = 3600;
/** Every text value a form posted, files and the bot fields left out. */
export function formValues(form: { entries(): IterableIterator<[string, FormDataEntryValue]> } | FormData): Record<string, string> {
const out: Record<string, string> = {};
for (const [k, v] of (form as FormData).entries()) {
if (typeof v !== "string" || k === HONEYPOT || k === TURNSTILE_FIELD || k === "terms") continue;
if (k.length > 160) continue;
out[k] = v.slice(0, 20_000);
}
return out;
}
export function encodeDraft(d: Draft, limit = DRAFT_LIMIT): string {
const values = { ...d.values };
const enc = () => encodeURIComponent(JSON.stringify({ ...d, values }));
let out = enc();
while (out.length > limit) {
const longest = Object.entries(values).sort((a, b) => b[1].length - a[1].length)[0];
if (!longest || longest[1].length === 0) break;
const [k, v] = longest;
values[k] = v.length > 200 ? v.slice(0, Math.floor(v.length / 2)) : "";
out = enc();
}
return JSON.stringify({ ...d, values });
}
export function parseDraft(raw: string | undefined): Draft | undefined {
if (!raw) return undefined;
try {
const v = JSON.parse(raw.startsWith("%7B") || raw.startsWith("%7b") ? decodeURIComponent(raw) : raw) as Record<string, unknown>;
const code = parseErrorCode(typeof v.code === "string" ? v.code : undefined);
if (!code) return undefined;
const strings = (x: unknown) =>
Object.fromEntries(Object.entries(typeof x === "object" && x ? x : {}).filter((e): e is [string, string] => typeof e[1] === "string"));
return { code, errors: strings(v.errors), values: strings(v.values) };
} catch {
return undefined;
}
}
// ---------------------------------------------------------------- send
export type EmailResult = "sent" | "skipped" | "failed";
/**
* Save the copy, then email, then record how the email went. The visitor
* hears it worked when either the saved copy or the email made it; when
* neither did, the form shows an error.
*/
export async function saveAndSend(deps: {
save(): Promise<{ id: number; receipt: string }>;
email(savedId: number | null): Promise<"sent" | "skipped">;
report(id: number, receipt: string, status: EmailResult, error?: string): Promise<void>;
log?(message: string, err: unknown): void;
}): Promise<{ ok: boolean; savedId: number | null; email: EmailResult }> {
const log = deps.log ?? ((m, e) => console.error(m, e));
let saved: { id: number; receipt: string } | null = null;
try {
saved = await deps.save();
} catch (err) {
log("Saving the copy failed; trying the email anyway.", err);
}
let email: EmailResult;
let error: string | undefined;
try {
email = await deps.email(saved?.id ?? null);
} catch (err) {
email = "failed";
error = err instanceof Error ? err.message.slice(0, 500) : String(err).slice(0, 500);
log("Sending the email failed.", err);
}
if (saved) {
await deps.report(saved.id, saved.receipt, email, error).catch((err: unknown) => log("Recording the email result failed.", err));
}
return { ok: !!saved || email === "sent", savedId: saved?.id ?? null, email };
}
// ---------------------------------------------------------------- rate limit
/** N posts per address per window, in memory. Enough for one site process. */
export class RateLimiter {
private readonly hits = new Map<string, { count: number; reset: number }>();
constructor(
private readonly limit: () => number,
private readonly windowMs: () => number,
private readonly now: () => number = Date.now,
) {}
/** Counts this post; true when it is over the limit. */
hit(key: string): boolean {
const t = this.now();
if (this.hits.size > 5000) for (const [k, e] of this.hits) if (t > e.reset) this.hits.delete(k);
const e = this.hits.get(key);
if (!e || t > e.reset) {
this.hits.set(key, { count: 1, reset: t + this.windowMs() });
return false;
}
e.count += 1;
return e.count > this.limit();
}
}
/** The visitor's address: the proxy's X-Forwarded-For first, else the socket's. */
export function clientIp(headers: Headers, fallback?: string): string {
const fwd = headers.get("x-forwarded-for");
if (fwd) return fwd.split(",")[0]!.trim();
return headers.get("x-real-ip") ?? fallback ?? "unknown";
}
services/site/src/lib/library.ts
+49
View File
@@ -5,6 +5,7 @@
*/
import type { Note } from "./content.ts";
import type { SiteIndex } from "./site.ts";
import { formatDate } from "./site.ts";
export type Item = Note<"library-item">;
export type Game = Note<"board-game">;
@@ -47,6 +48,29 @@ export function byStanding(a: Item, b: Item): number {
);
}
/** Progress PersonalReader syncs onto a linked item (see withSynced in content.ts). */
interface ReaderSynced {
chapterProgress?: { chapter?: number; readAt?: string };
bookProgress?: { percent?: number; readAt?: string };
}
/** "Chapter 42, read Oct 3, 2026" or "62% read", from PersonalReader; undefined when nothing is synced. */
export function readerProgress(item: Item): string | undefined {
const d = item.data as ReaderSynced;
const c = d.chapterProgress;
const b = d.bookProgress;
const what = c?.chapter ? `Chapter ${c.chapter}` : b && typeof b.percent === "number" && b.percent > 0 ? `${b.percent}% read` : undefined;
if (!what) return undefined;
const at = (c?.chapter ? c.readAt : b?.readAt)?.slice(0, 10);
return at ? `${what}, read ${formatDate(at)}` : what;
}
/** The row's details line. The note's own `progress` wins over the synced one. */
export const itemDetails = (item: Item): (string | number)[] => {
const d = item.data;
return [d.creator, d.year, d.platform, d.progress || readerProgress(item), d.group].filter((x): x is string | number => x !== undefined && x !== "");
};
export interface LibraryFilter {
type?: MediaKey;
q: string;
@@ -152,6 +176,31 @@ export function gameShows(g: Game, f: ShelfFilter, text: string): boolean {
return !f.q || text.includes(f.q);
}
/** Values BoardGameGeek syncs onto a game (see withSynced in content.ts). */
interface GameSynced {
minPlayers?: number;
maxPlayers?: number;
playTime?: number;
minPlayTime?: number;
maxPlayTime?: number;
}
const range = (lo?: number, hi?: number) => (lo && hi && lo !== hi ? `${lo}–${hi}` : String(lo || hi || ""));
/** "2–4 players", "1 player" or undefined when BoardGameGeek has not been synced. */
export function gamePlayers(g: Game): string | undefined {
const d = g.data as GameSynced;
const r = range(d.minPlayers, d.maxPlayers);
return r ? `${r} ${r === "1" ? "player" : "players"}` : undefined;
}
/** "60–90 min", "45 min" or undefined. */
export function gamePlayTime(g: Game): string | undefined {
const d = g.data as GameSynced;
const r = d.minPlayTime && d.maxPlayTime ? range(d.minPlayTime, d.maxPlayTime) : range(d.playTime);
return r ? `${r} min` : undefined;
}
/** Every "how it plays" value on the shelf, A to Z. */
export const playStyles = (games: Game[]) =>
[...new Set(games.flatMap((g) => g.data.howItPlays ?? []))].sort((a, b) => a.localeCompare(b));
services/site/src/lib/store.ts
+37 -4
View File
@@ -118,10 +118,43 @@ export function priceSummary(line: Line): { from: string; note?: string; more: s
return { from: low.price, note: low.note, more };
}
/** Option groups in note order, with their options. */
export function optionGroups(line: Line): [string, NonNullable<Line["data"]["options"]>][] {
const map = new Map<string, NonNullable<Line["data"]["options"]>>();
for (const o of line.data.options ?? []) map.set(o.group, [...(map.get(o.group) ?? []), o]);
export type Option = NonNullable<Line["data"]["options"]>[number] & { inStock: boolean };
/** Stock the content service syncs onto a line: Bambuddy filament, Homebox leather and paper. */
type StockGroup = { group: string; options: { name: string; hex?: string; inStock: boolean }[] };
interface StockSynced {
filamentStock?: StockGroup[];
materialStock?: StockGroup[];
}
/**
* Option groups in note order, with their options. Synced stock sits under
* the note's own values: a note option keeps every value it sets, takes the
* synced `inStock` and `hex` only where it leaves them out (out of stock
* when the source does not have it, in stock when nothing is synced), and
* in-stock colors the note does not list follow its own.
*/
export function optionGroups(line: Line): [string, Option[]][] {
const d = line.data as Line["data"] & StockSynced;
const synced = new Map<string, StockGroup["options"]>();
for (const g of [...(d.filamentStock ?? []), ...(d.materialStock ?? [])]) synced.set(g.group.toLowerCase(), g.options);
const map = new Map<string, Option[]>();
for (const o of line.data.options ?? []) {
const stock = synced.get(o.group.toLowerCase());
const match = stock?.find((x) => x.name.toLowerCase() === o.name.toLowerCase() || (!!o.hex && x.hex?.toLowerCase() === o.hex.toLowerCase()));
const hex = o.hex ?? match?.hex;
map.set(o.group, [...(map.get(o.group) ?? []), { ...o, ...(hex ? { hex } : {}), inStock: o.inStock ?? (stock ? !!match?.inStock : true) }]);
}
for (const [group, opts] of map) {
const stock = synced.get(group.toLowerCase()) ?? [];
const listed = new Set(opts.map((o) => o.name.toLowerCase()));
const listedHex = new Set(opts.map((o) => o.hex?.toLowerCase()).filter(Boolean));
for (const x of stock) {
if (!x.inStock || listed.has(x.name.toLowerCase()) || (x.hex && listedHex.has(x.hex.toLowerCase()))) continue;
opts.push({ group, name: x.name, ...(x.hex ? { hex: x.hex } : {}), inStock: true });
listed.add(x.name.toLowerCase());
}
}
return [...map.entries()];
}
services/site/src/lib/world.ts
+23
View File
@@ -91,6 +91,29 @@ export function statusesFor(s: SiteIndex, page: string) {
/** Archidekt deck address. */
export const archidektUrl = (id: number) => `https://archidekt.com/decks/${id}`;
/** Values Archidekt syncs onto a deck (see withSynced in content.ts). */
interface DeckSynced {
commanderArt?: string;
colors?: string[];
commanders?: string[];
}
/** The deck's own cover, else its synced commander art. */
export function deckArt(deck: Note<"deck">): string | undefined {
const d = deck.data as DeckSynced & { cover?: string };
return mediaSrc(d.cover ?? d.commanderArt);
}
const COLOR_NAMES: Record<string, string> = { W: "White", U: "Blue", B: "Black", R: "Red", G: "Green" };
/** The deck's color identity as names, WUBRG order; empty when not synced. */
export function deckColors(deck: Note<"deck">): string[] {
return ((deck.data as DeckSynced).colors ?? []).map((c) => COLOR_NAMES[c]).filter((c): c is string => !!c);
}
/** The deck's commanders, when synced. */
export const deckCommanders = (deck: Note<"deck">): string[] => (deck.data as DeckSynced).commanders ?? [];
/** Deck's list address: Archidekt when set, else its own list link. */
export function deckUrl(deck: Note<"deck">): string | undefined {
return deck.data.archidektId ? archidektUrl(deck.data.archidektId) : deck.data.listUrl;
services/site/src/pages/api/inquiries.ts
+76
View File
@@ -0,0 +1,76 @@
/**
* POST /api/inquiries: the Inquiries form (multipart, works without script).
* Checks the post, the rate limit and Turnstile, saves a copy through the
* content service, emails Greg, and redirects (303) to /inquiries/sent with
* the gk-inquiry cookie. A post that is turned down goes back to /inquiries
* with the reason and the visitor's answers.
*/
import type { APIRoute } from "astro";
import {
clientIp,
encodeDraft,
formValues,
inquiryEmail,
inquiryFields,
inquirySent,
isBot,
saveAndSend,
TURNSTILE_FIELD,
validateInquiry,
type ErrorCode,
type FieldErrors,
} from "../../lib/forms.ts";
import {
contactTo,
flashCookie,
inquiryLimiter,
maxAttachmentBytes,
reportDelivery,
saveSubmission,
sendMail,
verifyTurnstile,
} from "../../lib/forms-server.ts";
export const prerender = false;
export const POST: APIRoute = async ({ request, cookies, redirect, url, clientAddress }) => {
let form: FormData;
try {
form = await request.formData();
} catch {
return redirect("/inquiries?error=invalid", 303);
}
const type = String(form.get("type") ?? "");
const back = (code: ErrorCode, errors: FieldErrors = {}) => {
cookies.set("gk-inquiry-draft", encodeDraft({ code, errors, values: formValues(form) }), flashCookie(url));
const q = new URLSearchParams({ error: code });
if (/^[a-z]{1,20}$/.test(type)) q.set("type", type);
return redirect(`/inquiries?${q}`, 303);
};
// A bot filled the hidden field: answer as if it worked, keep nothing.
if (isBot(form)) return redirect("/inquiries/sent", 303);
const checked = validateInquiry(form, { maxBytes: maxAttachmentBytes() });
if (!checked.ok) return back(checked.code, checked.errors);
const q = checked.value;
let ip = "unknown";
try {
ip = clientIp(request.headers, clientAddress);
} catch {
ip = clientIp(request.headers);
}
if (inquiryLimiter.hit(ip)) return back("rate");
if (!(await verifyTurnstile(String(form.get(TURNSTILE_FIELD) ?? ""), ip))) return back("bot");
const result = await saveAndSend({
save: () => saveSubmission({ kind: "inquiry", type: q.type, name: q.name, email: q.email, fields: inquiryFields(q), attachment: q.attachment }),
email: (savedId) => sendMail({ to: contactTo(), replyTo: { name: q.name, address: q.email }, ...inquiryEmail(q, savedId), attachment: q.attachment }),
report: reportDelivery,
});
if (!result.ok) return back("failed");
cookies.set("gk-inquiry", JSON.stringify(inquirySent(q)), flashCookie(url));
return redirect("/inquiries/sent", 303);
};
services/site/src/pages/api/orders.ts
+80
View File
@@ -0,0 +1,80 @@
/**
* POST /api/orders: the Store's order request form (works without script).
* Checks the post, the rate limit and Turnstile, then re-checks the line
* against the live store notes: an order is taken only when its category's
* ordersOpen is true and the line has an open slot or stock, whatever the
* post says. Then it saves a copy, emails Greg, and redirects (303) to
* /store/order/sent with the gk-order cookie. No payment is taken.
*/
import type { APIRoute } from "astro";
import {
checkOrder,
clientIp,
encodeDraft,
formValues,
isBot,
orderEmail,
orderFields,
orderSent,
saveAndSend,
TURNSTILE_FIELD,
validateOrderFields,
type ErrorCode,
type FieldErrors,
} from "../../lib/forms.ts";
import { flashCookie, liveStore, orderLimiter, orderTo, reportDelivery, saveSubmission, sendMail, verifyTurnstile } from "../../lib/forms-server.ts";
import type { StoreState } from "../../lib/store.ts";
export const prerender = false;
export const POST: APIRoute = async ({ request, cookies, redirect, url, clientAddress }) => {
let form: FormData;
try {
form = await request.formData();
} catch {
return redirect("/store/order?error=invalid", 303);
}
const line = String(form.get("line") ?? "");
const back = (code: ErrorCode, errors: FieldErrors = {}) => {
cookies.set("gk-order-draft", encodeDraft({ code, errors, values: formValues(form) }), flashCookie(url));
const q = new URLSearchParams({ error: code });
if (/^[a-z0-9-]{1,120}$/.test(line)) q.set("line", line);
return redirect(`/store/order?${q}`, 303);
};
if (isBot(form)) return redirect("/store/order/sent", 303);
const fields = validateOrderFields(form);
if (!fields.ok) return back(fields.code, fields.errors);
let ip = "unknown";
try {
ip = clientIp(request.headers, clientAddress);
} catch {
ip = clientIp(request.headers);
}
if (orderLimiter.hit(ip)) return back("rate");
if (!(await verifyTurnstile(String(form.get(TURNSTILE_FIELD) ?? ""), ip))) return back("bot");
// The gate: always the live notes, never the cached copy or the form.
let st: StoreState;
try {
st = await liveStore();
} catch (err) {
console.error("Could not read the live store notes; turning the order down.", err);
return back("unavailable");
}
const checked = checkOrder(fields.value, form, st);
if (!checked.ok) return back(checked.code, checked.errors);
const o = checked.value;
const result = await saveAndSend({
save: () => saveSubmission({ kind: "order", type: o.lineNote.slug, name: o.name, email: o.email, fields: orderFields(o) }),
email: (savedId) => sendMail({ to: orderTo(), replyTo: { name: o.name, address: o.email }, ...orderEmail(o, savedId) }),
report: reportDelivery,
});
if (!result.ok) return back("failed");
cookies.set("gk-order", JSON.stringify(orderSent(o)), flashCookie(url));
return redirect("/store/order/sent", 303);
};
services/site/src/pages/inquiries/index.astro
+68 -23
View File
@@ -1,20 +1,31 @@
---
import Button from "@gk/ui/components/Button.astro";
import Field from "@gk/ui/components/Field.astro";
import Notice from "@gk/ui/components/Notice.astro";
import HeroHorizon from "@gk/ui/components/HeroHorizon.astro";
import PageShell from "@gk/ui/components/PageShell.astro";
import Seal from "@gk/ui/components/Seal.astro";
import Star from "@gk/ui/components/Star.astro";
import StatusPill from "@gk/ui/components/StatusPill.astro";
import ChoicePills from "../../components/work/ChoicePills.astro";
import { env } from "../../lib/env.ts";
import { choices, errorMessages, HONEYPOT, parseDraft, parseErrorCode, timing } from "../../lib/forms.ts";
import { turnstileSiteKey } from "../../lib/forms-server.ts";
import { site } from "../../lib/page.ts";
import { inquiryTypes, parseInquiryType, type InquiryType } from "../../lib/work.ts";
const s = await site();
const settings = s.settings();
const text = s.pageText("/inquiries");
const selected = parseInquiryType(Astro.url.searchParams.get("type"));
// A post that was turned down comes back with its reason and the answers given.
const draft = parseDraft(Astro.cookies.get("gk-inquiry-draft")?.value);
if (draft) {
Astro.cookies.delete("gk-inquiry-draft", { path: "/" });
Astro.response.headers.set("Cache-Control", "private, no-store");
}
const errorCode = draft?.code ?? parseErrorCode(Astro.url.searchParams.get("error"));
const v = draft?.values ?? {};
const err = draft?.errors ?? {};
const selected = parseInquiryType(v.type ?? Astro.url.searchParams.get("type"));
const lede = text?.data.lede ?? "Roles, freelance web work or a custom Patiun order. Pick one and the form asks the right questions.";
const quickFacts = text?.data.quickFacts?.map((f) => f.text) ?? [`Replies within ${s.settings()?.replyTime ?? "a few days"}`, "Read by me, not a bot"];
const socials = settings?.socials ?? [];
@@ -25,12 +36,7 @@ const placeholders: Record<InquiryType | "none", string> = {
freelance: "What you need built, who it is for, and any links.",
patiun: "Card list, deck colors, theme, or anything I should know.",
};
const timing = [
{ value: "no-rush", label: "No rush" },
{ value: "within-a-month", label: "Within a month" },
{ value: "date", label: "A specific date" },
];
const turnstileKey = env("PUBLIC_TURNSTILE_SITE_KEY") ?? env("TURNSTILE_SITE_KEY");
const turnstileKey = turnstileSiteKey();
---
<PageShell title="Inquiries" family="inquiries" current="inquiries" description="Ask about a role, freelance web work or a custom Patiun order.">
{turnstileKey && <script slot="head" src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer is:inline></script>}
@@ -74,8 +80,13 @@ const turnstileKey = env("PUBLIC_TURNSTILE_SITE_KEY") ?? env("TURNSTILE_SITE_KEY
<form class="form" method="post" action="/api/inquiries" enctype="multipart/form-data" aria-label="Inquiry" data-inquiry>
<div class="frame">
{errorCode && (
<Notice role="alert" title={errorMessages[errorCode]}>
{Object.keys(err).length > 0 ? "The marked answers are below." : ""}
</Notice>
)}
<fieldset class="types">
<legend>What's this about?</legend>
<legend>What's this about?{err.type && <span class="err"> {err.type}</span>}</legend>
<div class="tiles">
{
inquiryTypes.map((t) => (
@@ -91,46 +102,68 @@ const turnstileKey = env("PUBLIC_TURNSTILE_SITE_KEY") ?? env("TURNSTILE_SITE_KEY
</fieldset>
<div class="pair">
<Field label="Your name" name="name" placeholder="Jane Doe" autocomplete="name" required maxlength={120} />
<Field label="Email" name="email" type="email" placeholder="you@example.com" autocomplete="email" required maxlength={200} />
<Field label="Your name" name="name" placeholder="Jane Doe" autocomplete="name" required maxlength={120} value={v.name} error={err.name} />
<Field label="Email" name="email" type="email" placeholder="you@example.com" autocomplete="email" required maxlength={200} value={v.email} error={err.email} />
</div>
<div class="q q-job" data-q="job">
<div class="pair">
<Field label="Company" name="job_company" placeholder="Where is the role?" autocomplete="organization" maxlength={160} />
<Field label="Role" name="job_role" placeholder="e.g. Senior full-stack developer" maxlength={160} />
<Field label="Company" name="job_company" placeholder="Where is the role?" autocomplete="organization" maxlength={160} value={v.job_company} error={err.job_company} />
<Field label="Role" name="job_role" placeholder="e.g. Senior full-stack developer" maxlength={160} value={v.job_role} error={err.job_role} />
</div>
<ChoicePills legend="Kind of role" name="job_kind" options={["Full-time", "Contract", "Not sure yet"]} />
<Field label="Link to the posting" name="job_link" type="url" placeholder="https://" hint="Optional." maxlength={500} />
<ChoicePills legend="Kind of role" name="job_kind" options={[...choices.job_kind]} checked={v.job_kind} />
<Field label="Link to the posting" name="job_link" type="url" placeholder="https://" hint="Optional." maxlength={500} value={v.job_link} error={err.job_link} />
</div>
<div class="q q-freelance" data-q="freelance">
<ChoicePills legend="What do you need?" name="freelance_need" options={["A new site", "An app", "An integration", "Something else"]} />
<ChoicePills legend="What do you need?" name="freelance_need" options={[...choices.freelance_need]} checked={v.freelance_need} />
<div class="pair">
<Field label="Your current site" name="freelance_site" type="url" placeholder="https://" hint="Optional." maxlength={500} />
<Field label="Your current site" name="freelance_site" type="url" placeholder="https://" hint="Optional." maxlength={500} value={v.freelance_site} error={err.freelance_site} />
<Field label="Timeline" name="freelance_timing" as="select" options={timing} />
</div>
<Field label="Date" name="freelance_date" type="date" class="when-date" />
<Field label="Date" name="freelance_date" type="date" class="when-date" value={v.freelance_date} error={err.freelance_date} />
</div>
<div class="q q-patiun" data-q="patiun">
<ChoicePills legend="What would you like made?" name="patiun_item" options={["Proxy cards", "Deck box", "Something else"]} />
<ChoicePills legend="What would you like made?" name="patiun_item" options={[...choices.patiun_item]} checked={v.patiun_item} />
<div class="pair">
<Field label="How many?" name="patiun_quantity" placeholder="e.g. 60 cards" maxlength={120} />
<Field label="How many?" name="patiun_quantity" placeholder="e.g. 60 cards" maxlength={120} value={v.patiun_quantity} error={err.patiun_quantity} />
<Field label="Need it by" name="patiun_timing" as="select" options={timing} />
</div>
<Field label="Date" name="patiun_date" type="date" class="when-date" />
<Field label="Date" name="patiun_date" type="date" class="when-date" value={v.patiun_date} error={err.patiun_date} />
</div>
<Field label="Details" name="details" as="textarea" rows={5} required maxlength={5000} placeholder={placeholders[selected ?? "none"]} data-details />
<div class="gk-label">
<label class="gk-label-text" for="f-details">Details</label>
<textarea
id="f-details"
name="details"
class="gk-field"
rows={5}
required
minlength={10}
maxlength={5000}
placeholder={placeholders[selected ?? "none"]}
aria-describedby={err.details ? "f-details-error" : undefined}
aria-invalid={err.details ? "true" : undefined}
data-details>{v.details ?? ""}</textarea>
{err.details && <span class="err" id="f-details-error">{err.details}</span>}
</div>
<label class="upload">
<svg width="22" height="22" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M12 16V4M7 9l5-5 5 5M5 20h14" /></svg>
<span>Add a reference image or decklist <span class="browse">Browse</span><span class="file-name" data-file-name></span></span>
<input type="file" name="attachment" accept="image/*,.pdf,.txt,.csv,.dek,.dec" class="sr" data-file />
</label>
{err.attachment && <span class="err">{err.attachment}</span>}
{draft && !err.attachment && <span class="note">If you attached a file, add it again.</span>}
{turnstileKey && <div class="cf-turnstile" data-sitekey={turnstileKey} data-theme="dark"></div>}
<div class="hp" aria-hidden="true">
<label>Leave this empty <input type="text" name={HONEYPOT} tabindex="-1" autocomplete="off" /></label>
</div>
{turnstileKey && <div class="cf-turnstile" data-sitekey={turnstileKey} data-theme="dark" data-response-field-name="cf-turnstile-response"></div>}
{turnstileKey && <noscript><span class="note">The bot check needs JavaScript. Turn it on for this page to send the form.</span></noscript>}
<div class="submit">
<Button type="submit">Send inquiry</Button>
@@ -406,6 +439,18 @@ const turnstileKey = env("PUBLIC_TURNSTILE_SITE_KEY") ?? env("TURNSTILE_SITE_KEY
font-size: 13px;
color: var(--gk-muted);
}
.err {
font-size: 12px;
font-weight: 400;
color: #f0857a;
}
.hp {
position: absolute;
left: -10000px;
width: 1px;
height: 1px;
overflow: hidden;
}
@media (max-width: 640px) {
.tiles {
grid-template-columns: 1fr;
services/site/src/pages/library/shelf.astro
+2 -1
View File
@@ -3,7 +3,8 @@
* The Shelf: every board game. What is on the table (from the status log),
* then the collection, filtered by how it plays, owned, want to play and
* favorites, and sorted by last played, A to Z or recently added. Player
* counts and play times come from BoardGameGeek later; no filter for them yet.
* counts, play times and box art sync from BoardGameGeek (a cover set in the
* note wins); no filter for them yet.
*/
import Divider from "@gk/ui/components/Divider.astro";
import FilterGroup from "@gk/ui/components/FilterGroup.astro";
services/site/src/pages/store/order.astro
+23 -2
View File
@@ -15,10 +15,20 @@ import OrderForm from "../../components/store/OrderForm.astro";
import { site } from "../../lib/page.ts";
import { formatDate } from "../../lib/site.ts";
import { canOrder, channelOf, storeState } from "../../lib/store.ts";
import { errorMessages, parseDraft, parseErrorCode } from "../../lib/forms.ts";
import { turnstileSiteKey } from "../../lib/forms-server.ts";
const s = await site();
const st = storeState(s);
const selected = Astro.url.searchParams.get("line") ?? undefined;
// A post that was turned down comes back with its reason and the answers given.
const draft = parseDraft(Astro.cookies.get("gk-order-draft")?.value);
if (draft) {
Astro.cookies.delete("gk-order-draft", { path: "/" });
Astro.response.headers.set("Cache-Control", "private, no-store");
}
const errorCode = draft?.code ?? parseErrorCode(Astro.url.searchParams.get("error"));
const turnstileKey = turnstileSiteKey();
const selected = draft?.values.line ?? Astro.url.searchParams.get("line") ?? undefined;
const orderable = st.lines.filter((l) => canOrder(l, st));
const selling = st.lines.filter((l) => channelOf(l) === "here");
const shown = orderable.length ? orderable : selling;
@@ -26,6 +36,7 @@ const turnarounds = [...new Set(st.groups.map(([c]) => st.category(c).turnaround
const reopens = st.closed.filter((c) => c.reopens).map((c) => `${c.name} reopens ${formatDate(c.reopens)}.`);
---
<PageShell title="Order request" family="store" current="store" description="Send an order request for something from the Store.">
{turnstileKey && <script slot="head" src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer is:inline></script>}
<section class="intro" aria-labelledby="order-h1">
<Breadcrumb here="Order request" />
<h1 id="order-h1">Order request</h1>
@@ -34,9 +45,16 @@ const reopens = st.closed.filter((c) => c.reopens).map((c) => `${c.name} reopens
<div class="layout">
<div class="main">
{errorCode && (
<div class="problem">
<Notice role="alert" title={errorMessages[errorCode]}>
{draft && Object.keys(draft.errors).length > 0 ? "The marked answers are below." : ""}
</Notice>
</div>
)}
{
orderable.length > 0 ? (
<OrderForm lines={orderable} selected={selected} storeTerms={st.terms} />
<OrderForm lines={orderable} selected={selected} storeTerms={st.terms} values={draft?.values} errors={draft?.errors} turnstileKey={turnstileKey} />
) : (
<div class="closed">
{st.anyOpen ? (
@@ -104,6 +122,9 @@ const reopens = st.closed.filter((c) => c.reopens).map((c) => `${c.name} reopens
flex: 3 1 600px;
min-width: 0;
}
.problem {
margin-bottom: 16px;
}
.closed {
display: flex;
flex-direction: column;
services/site/src/pages/store/order/sent.astro
+204
View File
@@ -0,0 +1,204 @@
---
/**
* The confirmation after an order request. /api/orders sets the short-lived
* `gk-order` cookie (URL-encoded JSON: name, email, line, rows) before it
* redirects here; the page shows it once and clears it. Without the cookie
* the page still reads correctly.
*/
import Button from "@gk/ui/components/Button.astro";
import PageShell from "@gk/ui/components/PageShell.astro";
import Star from "@gk/ui/components/Star.astro";
import TickRing from "@gk/ui/components/TickRing.astro";
import TickScale from "@gk/ui/components/TickScale.astro";
import Breadcrumb from "../../../components/store/Breadcrumb.astro";
import { parseOrderSent } from "../../../lib/forms.ts";
import { site } from "../../../lib/page.ts";
import { storeState } from "../../../lib/store.ts";
const sent = parseOrderSent(Astro.cookies.get("gk-order")?.value);
if (sent) Astro.cookies.delete("gk-order", { path: "/" });
Astro.response.headers.set("Cache-Control", "private, no-store");
const s = await site().catch(() => null);
const replyTime = s?.settings()?.replyTime ?? "a few days";
const line = sent?.lineSlug && s ? storeState(s).lines.find((l) => l.slug === sent.lineSlug) : undefined;
const turnaround = line && s ? storeState(s).category(line.data.category).turnaround : undefined;
const rows = [
...(sent?.line ? [{ label: "Line", value: sent.line }] : []),
...(sent?.rows ?? []),
...(sent?.email ? [{ label: "Reply to", value: sent.email }] : []),
];
const firstName = sent?.name?.split(/\s+/)[0];
const steps = [
{ title: "I check the request", text: "The details, the options and my queue." },
{ title: "I confirm with you", text: "The total and the timing, by email. No payment is taken through the site." },
{ title: "I start once you say yes", text: turnaround ?? "Nothing is made until we agree." },
];
---
<PageShell title="Order request sent" family="store" current="store" description="Your order request is on its way.">
<meta slot="head" name="robots" content="noindex" />
<section class="intro" aria-labelledby="sent-h1">
<Breadcrumb here="Request sent" />
<h1 id="sent-h1">Your request is in</h1>
<p class="lede">Thanks{firstName ? `, ${firstName}` : ""}. I read every order request myself and reply within {replyTime}.</p>
</section>
<div class="layout">
<div class="main">
<ol class="steps" aria-label="What happens next">
{
steps.map((st, i) => (
<li>
<TickRing size={36}>{i + 1}</TickRing>
<span class="step-text">
<span class="step-title">{st.title}</span>
<span class="step-detail">{st.text}</span>
</span>
</li>
))
}
</ol>
<div class="actions">
<Button href="/store">Back to the Store</Button>
<Button variant="secondary" href="/inquiries">Something custom? Start an inquiry</Button>
</div>
</div>
{
rows.length > 0 && (
<aside class="sent" aria-labelledby="sent-heading">
<h2 id="sent-heading" class="kicker">
<Star size={11} />
What you sent
</h2>
<TickScale width={150} height={10} count={30} class="ticks" />
<dl>
{rows.map((r) => (
<div class="row">
<dt>{r.label}</dt>
<dd>{r.value}</dd>
</div>
))}
</dl>
</aside>
)
}
</div>
</PageShell>
<style>
.intro {
padding: 36px 0 8px;
display: flex;
flex-direction: column;
gap: 10px;
}
h1 {
margin: 0;
font-size: clamp(32px, 5vw, 44px);
letter-spacing: 0.05em;
}
.lede {
margin: 0;
font-size: 16px;
color: var(--gk-soft);
max-width: 64ch;
line-height: 1.6;
}
.layout {
padding: 16px 0 64px;
display: flex;
flex-wrap: wrap;
gap: 32px;
align-items: flex-start;
}
.main {
flex: 3 1 480px;
min-width: 0;
display: flex;
flex-direction: column;
gap: 20px;
}
.steps {
margin: 0;
padding: 0;
list-style: none;
border-bottom: 1px solid var(--gk-lines);
}
.steps li {
display: flex;
gap: 14px;
align-items: flex-start;
padding: 14px 0;
border-top: 1px solid var(--gk-lines);
font-size: 13px;
}
.step-text {
display: flex;
flex-direction: column;
gap: 3px;
}
.step-title {
font-weight: 700;
font-size: 15px;
}
.step-detail {
font-size: 14px;
color: var(--gk-muted);
line-height: 1.5;
}
.actions {
display: flex;
flex-wrap: wrap;
gap: 12px;
}
.sent {
flex: 1 1 300px;
max-width: 420px;
min-width: 0;
padding: 24px;
border-radius: var(--gk-radius);
background: var(--gk-midnight);
border: 1px solid rgba(201, 162, 74, 0.55);
}
.kicker {
margin: 0 0 8px;
display: flex;
align-items: center;
gap: 8px;
font-family: var(--gk-font-body);
font-size: 11px;
letter-spacing: 0.3em;
font-weight: 600;
color: var(--gk-gold-bright);
text-transform: uppercase;
}
.sent :global(.ticks) {
margin: -4px 0 2px;
}
dl {
margin: 8px 0 0;
}
.row {
display: flex;
justify-content: space-between;
gap: 16px;
padding: 10px 0;
border-top: 1px solid var(--gk-lines);
font-size: 14px;
}
dt {
color: var(--gk-muted);
}
dd {
margin: 0;
font-weight: 600;
text-align: right;
overflow-wrap: anywhere;
}
@media (max-width: 700px) {
.sent {
max-width: none;
}
}
</style>
services/site/src/pages/tabletop.astro
+8 -2
View File
@@ -11,7 +11,7 @@ import WorldSection from "../components/world/WorldSection.astro";
import type { Note } from "../lib/content.ts";
import { plural, site } from "../lib/page.ts";
import { slugify } from "@gk/schema";
import { byOrder, deckUrl, makePhotos, sentence, stancePill, systemPill, systemStates } from "../lib/world.ts";
import { byOrder, deckArt, deckColors, deckCommanders, deckUrl, makePhotos, sentence, stancePill, systemPill, systemStates } from "../lib/world.ts";
const s = await site();
const text = s.pageText("/tabletop");
@@ -151,7 +151,7 @@ const workshopImg = s.all("make").flatMap((m) => makePhotos(m))[0];
const game = gameOf(d);
return (
<li id={d.slug} class="deck" data-value={deckGroup(d)}>
<StarChartArt seal="astrolabe" ratio="auto" class="d-art" />
<StarChartArt seal="astrolabe" ratio="auto" class="d-art" src={deckArt(d)} alt={deckCommanders(d).join(" and ")} />
<div class="d-txt">
<span class="d-kick">{[shortGame(game.title), d.data.format].filter(Boolean).join(" · ")}</span>
<div class="e-top">
@@ -167,6 +167,7 @@ const workshopImg = s.all("make").flatMap((m) => makePhotos(m))[0];
</h3>
{deckOfMoment?.path === d.path && <StatusPill>Playing now</StatusPill>}
</div>
{deckColors(d).length > 0 && <p class="d-colors">{deckColors(d).join(" · ")}</p>}
{d.data.take && <p class="e-blurb">{d.data.take}</p>}
{url && (
<a href={url} class="gk-text d-link" rel="noopener" aria-label={`${d.title} deck list`}>
@@ -401,6 +402,11 @@ const workshopImg = s.all("make").flatMap((m) => makePhotos(m))[0];
.d-name a:hover {
color: var(--gk-gold-bright);
}
.d-colors {
margin: 0;
font-size: 12px;
color: var(--gk-muted);
}
.d-link {
align-self: flex-start;
font-size: 13px;
services/site/test/forms.test.ts
+261
View File
@@ -0,0 +1,261 @@
import { describe, expect, it, vi } from "vitest";
import type { Note } from "../src/lib/content.ts";
import {
checkOrder,
clientIp,
DRAFT_LIMIT,
encodeDraft,
formValues,
inquiryFields,
inquirySent,
isBot,
orderFields,
orderSent,
parseDraft,
parseOrderSent,
RateLimiter,
saveAndSend,
validateInquiry,
validateOrderFields,
} from "../src/lib/forms.ts";
import { SiteIndex } from "../src/lib/site.ts";
import { storeState } from "../src/lib/store.ts";
import { parseSentSummary } from "../src/lib/work.ts";
const MB = 1024 * 1024;
const form = (fields: Record<string, string | File>) => {
const f = new FormData();
for (const [k, v] of Object.entries(fields)) f.append(k, v);
return f;
};
const base = { name: "Jane Doe", email: "jane@example.com", details: "A senior role on the platform team." };
describe("inquiry validation", () => {
it("takes a job inquiry and only the job questions", () => {
const r = validateInquiry(
form({ ...base, type: "job", job_company: "Acme", job_kind: "Contract", job_link: "https://acme.example/jobs/1", patiun_quantity: "60", freelance_need: "An app" }),
{ maxBytes: MB },
);
expect(r.ok).toBe(true);
if (!r.ok) return;
expect(r.value.rows).toEqual([
{ label: "Company", value: "Acme" },
{ label: "Kind of role", value: "Contract" },
{ label: "Posting", value: "https://acme.example/jobs/1" },
]);
expect(inquiryFields(r.value).at(-1)).toEqual({ label: "Details", value: base.details });
});
it("marks what is wrong", () => {
const r = validateInquiry(form({ type: "nope", name: "", email: "x@", details: "hi", job_link: "javascript:alert(1)" }), { maxBytes: MB });
expect(r.ok).toBe(false);
if (r.ok) return;
expect(Object.keys(r.errors).sort()).toEqual(["details", "email", "name", "type"]);
});
it("checks choices, links and dates for the chosen type", () => {
const bad = validateInquiry(
form({ ...base, type: "freelance", freelance_need: "World peace", freelance_site: "ftp://x", freelance_timing: "date", freelance_date: "soon" }),
{ maxBytes: MB },
);
expect(bad.ok).toBe(false);
if (!bad.ok) expect(Object.keys(bad.errors).sort()).toEqual(["freelance_date", "freelance_need", "freelance_site"]);
const good = validateInquiry(form({ ...base, type: "patiun", patiun_item: "Deck box", patiun_timing: "date", patiun_date: "2026-12-01" }), { maxBytes: MB });
expect(good.ok && good.value.rows).toEqual([
{ label: "Item", value: "Deck box" },
{ label: "Need it by", value: "Dec 1, 2026" },
]);
});
it("limits attachments by type and size", () => {
const ok = validateInquiry(form({ ...base, type: "patiun", attachment: new File(["1 Sol Ring"], "deck.dek") }), { maxBytes: MB });
expect(ok.ok && ok.value.attachment?.name).toBe("deck.dek");
const exe = validateInquiry(form({ ...base, type: "patiun", attachment: new File(["MZ"], "x.exe") }), { maxBytes: MB });
expect(!exe.ok && exe.errors.attachment).toBeTruthy();
const big = validateInquiry(form({ ...base, type: "patiun", attachment: new File([new Uint8Array(MB + 1)], "big.png") }), { maxBytes: MB });
expect(!big.ok && big.errors.attachment).toContain("1 MB");
const empty = validateInquiry(form({ ...base, type: "patiun", attachment: new File([], "") }), { maxBytes: MB });
expect(empty.ok && empty.value.attachment).toBeNull();
});
it("builds the sent cookie the sent page reads", () => {
const r = validateInquiry(form({ ...base, type: "patiun", patiun_quantity: "60 cards" }), { maxBytes: MB });
if (!r.ok) throw new Error("should pass");
const parsed = parseSentSummary(encodeURIComponent(JSON.stringify(inquirySent(r.value))));
expect(parsed).toEqual({ type: "patiun", name: "Jane Doe", email: "jane@example.com", rows: [{ label: "How many", value: "60 cards" }] });
});
it("spots the honeypot", () => {
expect(isBot(form({ leave_blank: "http://spam" }))).toBe(true);
expect(isBot(form({ leave_blank: "" }))).toBe(false);
});
});
// Sample store notes, written for these tests.
const settings = (open: Record<string, boolean>) =>
({
path: "Store/_settings.md",
type: "store-settings",
name: "_settings",
slug: "settings",
title: "Store settings",
body: "",
acceptedAt: "2026-10-01",
data: { type: "store-settings", title: "Store settings", categories: Object.entries(open).map(([name, ordersOpen]) => ({ name, ordersOpen })) },
}) as unknown as Note;
const line = (slug: string, data: Record<string, unknown>) =>
({
path: `Store/${slug}.md`,
type: "product-line",
name: slug,
slug,
title: slug.replace(/-/g, " "),
body: "",
acceptedAt: "2026-10-01",
data: { type: "product-line", title: slug, category: "Tabletop", channel: "here", mode: "made to order", summary: "", priceTiers: [], ...data },
}) as unknown as Note;
const store = (open: Record<string, boolean>, lines: Note[]) => storeState(new SiteIndex([settings(open), ...lines]));
const cards = line("proxy-cards", {
slots: { open: 2, total: 5 },
options: [
{ group: "Finish", name: "Matte", inStock: true },
{ group: "Finish", name: "Foil", inStock: false },
],
});
const orderForm = (extra: Record<string, string> = {}) =>
form({ name: "Bo", email: "bo@example.com", line: "proxy-cards", quantity: "75 cards", terms: "accepted", ...extra });
const fieldsOf = (f: FormData) => {
const r = validateOrderFields(f);
if (!r.ok) throw new Error(JSON.stringify(r.errors));
return r.value;
};
describe("order validation", () => {
it("needs the terms, a line, a quantity and contact details", () => {
const r = validateOrderFields(form({ line: "../x" }));
expect(!r.ok && Object.keys(r.errors).sort()).toEqual(["email", "line", "name", "quantity", "terms"]);
});
it("takes an order for an open line with room and an in-stock option", () => {
const f = orderForm({ "option.proxy-cards.Finish": "Matte", "option.other-line.Color": "Red" });
const r = checkOrder(fieldsOf(f), f, store({ Tabletop: true }, [cards]));
expect(r.ok).toBe(true);
if (!r.ok) return;
expect(r.value.options).toEqual([{ label: "Finish", value: "Matte" }]);
expect(orderFields(r.value).map((x) => x.label)).toEqual(["Line", "Finish", "Quantity", "Terms"]);
expect(parseOrderSent(encodeURIComponent(JSON.stringify(orderSent(r.value))))?.rows).toEqual([
{ label: "Finish", value: "Matte" },
{ label: "Quantity", value: "75 cards" },
]);
});
it("turns down a closed category, whatever the post says", () => {
const f = orderForm();
expect(checkOrder(fieldsOf(f), f, store({ Tabletop: false }, [cards]))).toMatchObject({ ok: false, code: "closed" });
// No settings note at all counts as closed.
expect(checkOrder(fieldsOf(f), f, storeState(new SiteIndex([cards])))).toMatchObject({ ok: false, code: "closed" });
});
it("turns down a line with no open slot or no stock", () => {
const f = orderForm();
const full = line("proxy-cards", { slots: { open: 0, total: 5 } });
expect(checkOrder(fieldsOf(f), f, store({ Tabletop: true }, [full]))).toMatchObject({ ok: false, code: "full" });
const sold = line("proxy-cards", { mode: "in stock", quantity: 0 });
expect(checkOrder(fieldsOf(f), f, store({ Tabletop: true }, [sold]))).toMatchObject({ ok: false, code: "full" });
const stocked = line("proxy-cards", { mode: "in stock", quantity: 3 });
expect(checkOrder(fieldsOf(f), f, store({ Tabletop: true }, [stocked])).ok).toBe(true);
});
it("turns down lines sold elsewhere or unknown, and out-of-stock options", () => {
const f = orderForm();
const etsy = line("proxy-cards", { channel: "Patiun.com", slots: { open: 2, total: 5 } });
expect(checkOrder(fieldsOf(f), f, store({ Tabletop: true }, [etsy]))).toMatchObject({ ok: false, code: "invalid" });
expect(checkOrder(fieldsOf(f), f, store({ Tabletop: true }, []))).toMatchObject({ ok: false, code: "invalid" });
const foil = orderForm({ "option.proxy-cards.Finish": "Foil" });
expect(checkOrder(fieldsOf(foil), foil, store({ Tabletop: true }, [cards]))).toMatchObject({ ok: false, code: "invalid" });
const made = orderForm({ "option.proxy-cards.Finish": "Gold leaf" });
expect(checkOrder(fieldsOf(made), made, store({ Tabletop: true }, [cards]))).toMatchObject({ ok: false, code: "invalid" });
});
});
describe("try-again drafts", () => {
it("round-trips and fits in a cookie", () => {
const values = formValues(form({ name: "Jane", details: "x".repeat(20_000), leave_blank: "", "cf-turnstile-response": "tok", terms: "accepted" }));
expect(Object.keys(values)).toEqual(["name", "details"]);
const raw = encodeDraft({ code: "invalid", errors: { email: "Add an email" }, values });
expect(encodeURIComponent(raw).length).toBeLessThanOrEqual(DRAFT_LIMIT);
const d = parseDraft(encodeURIComponent(raw));
expect(d?.code).toBe("invalid");
expect(d?.values.name).toBe("Jane");
expect(d?.errors.email).toBe("Add an email");
expect(parseDraft("garbage")).toBeUndefined();
expect(parseDraft(JSON.stringify({ code: "nope" }))).toBeUndefined();
});
});
describe("saving and emailing", () => {
const quiet = { log: () => {} };
it("records a failed email and still succeeds when the copy was saved", async () => {
const report = vi.fn(async () => {});
const r = await saveAndSend({
...quiet,
save: async () => ({ id: 7, receipt: "rcpt" }),
email: async () => {
throw new Error("ECONNREFUSED");
},
report,
});
expect(r).toEqual({ ok: true, savedId: 7, email: "failed" });
expect(report).toHaveBeenCalledWith(7, "rcpt", "failed", "ECONNREFUSED");
});
it("still emails when saving fails, and fails only when both do", async () => {
const email = vi.fn(async (id: number | null) => (id === null ? ("sent" as const) : ("skipped" as const)));
const report = vi.fn(async () => {});
const failSave = async (): Promise<{ id: number; receipt: string }> => {
throw new Error("down");
};
expect(await saveAndSend({ ...quiet, save: failSave, email, report })).toEqual({ ok: true, savedId: null, email: "sent" });
expect(report).not.toHaveBeenCalled();
const r = await saveAndSend({
...quiet,
save: failSave,
email: async () => {
throw new Error("smtp down");
},
report,
});
expect(r.ok).toBe(false);
// Nothing configured for mail and nothing saved: not a success either.
expect((await saveAndSend({ ...quiet, save: failSave, email: async () => "skipped", report })).ok).toBe(false);
});
it("counts a skipped email as fine when the copy was saved", async () => {
const report = vi.fn(async () => {});
const r = await saveAndSend({ ...quiet, save: async () => ({ id: 1, receipt: "r" }), email: async () => "skipped", report });
expect(r.ok).toBe(true);
expect(report).toHaveBeenCalledWith(1, "r", "skipped", undefined);
});
});
describe("rate limit and address", () => {
it("allows N posts per window per address", () => {
let t = 0;
const rl = new RateLimiter(() => 2, () => 1000, () => t);
expect([rl.hit("a"), rl.hit("a"), rl.hit("a"), rl.hit("b")]).toEqual([false, false, true, false]);
t = 1001;
expect(rl.hit("a")).toBe(false);
});
it("reads the forwarded address first", () => {
expect(clientIp(new Headers({ "x-forwarded-for": "203.0.113.9, 10.0.0.1" }), "10.0.0.2")).toBe("203.0.113.9");
expect(clientIp(new Headers(), "10.0.0.2")).toBe("10.0.0.2");
});
});
services/site/test/homelab-sync.test.ts
+86
View File
@@ -0,0 +1,86 @@
/**
* PersonalReader progress on Library rows and synced stock on Store options.
* Synced values are the shapes the content service's jobs store (see
* services/content/src/sync/jobs/); a note's own values always win, and a
* note with nothing synced renders as before.
*/
import { describe, expect, it } from "vitest";
import { withSynced, type Note } from "../src/lib/content.ts";
import { itemDetails, readerProgress, type Item } from "../src/lib/library.ts";
import { optionGroups, type Line } from "../src/lib/store.ts";
const note = <T extends Note>(type: string, data: Record<string, unknown>, synced?: Record<string, unknown>) =>
withSynced({ path: `${type}/x.md`, type, name: "x", slug: "x", title: "X", body: "", acceptedAt: "", data: { type, ...data }, synced } as unknown as T);
describe("Library progress from PersonalReader", () => {
const base = { medium: "manhwa", state: "now", creator: "Someone", year: 2020 };
it("adds the chapter line to the details", () => {
const i = note<Item>("library-item", base, { chapterProgress: { title: "X", status: "now", chapter: 142, readAt: "2026-10-03T05:00:00.000Z" } });
expect(readerProgress(i)).toBe("Chapter 142, read Oct 3, 2026");
expect(itemDetails(i)).toEqual(["Someone", 2020, "Chapter 142, read Oct 3, 2026"]);
const book = note<Item>("library-item", { ...base, medium: "light novel" }, { bookProgress: { title: "X", percent: 62 } });
expect(readerProgress(book)).toBe("62% read");
});
it("lets the note's own progress win, and shows the same line when nothing is synced", () => {
const own = note<Item>("library-item", { ...base, progress: "16/18 volumes" }, { chapterProgress: { chapter: 3 } });
expect(itemDetails(own)).toEqual(["Someone", 2020, "16/18 volumes"]);
const bare = note<Item>("library-item", base);
expect(itemDetails(bare)).toEqual(["Someone", 2020]);
expect(readerProgress(note<Item>("library-item", base, { bookProgress: { percent: 0 } }))).toBeUndefined();
});
});
describe("Store options with synced stock", () => {
const lineData = (options: unknown[]) => ({ category: "Tabletop", mode: "made to order", channel: "here", summary: "", priceTiers: [], options });
const stock = {
filamentStock: [
{
group: "Filament",
options: [
{ name: "Jade White", hex: "#f5f5dc", inStock: true },
{ name: "Bambu Green", hex: "#00ae42", inStock: false },
{ name: "Charcoal", hex: "#2b2b2b", inStock: true },
],
},
],
};
it("is exactly the note's options when nothing is synced", () => {
const opts = [
{ group: "Filament", name: "Jade White", hex: "#ffffff", inStock: true },
{ group: "Finish", name: "Matte", inStock: false },
];
expect(optionGroups(note<Line>("product-line", lineData(opts)))).toEqual([
["Filament", [opts[0]]],
["Finish", [opts[1]]],
]);
// Left out with nothing synced: in stock, as before the field was optional.
expect(optionGroups(note<Line>("product-line", lineData([{ group: "Finish", name: "Gloss" }])))[0]![1][0]!.inStock).toBe(true);
});
it("sits under the note: note values win, gaps take the synced value, unlisted in-stock colors follow", () => {
const l = note<Line>(
"product-line",
lineData([
{ group: "Filament", name: "Jade White" },
{ group: "Filament", name: "Bambu Green", inStock: true },
{ group: "Filament", name: "Galaxy Black" },
{ group: "Finish", name: "Matte", inStock: true },
]),
stock,
);
const [[g, filament], [, finish]] = optionGroups(l) as [string, ReturnType<typeof optionGroups>[number][1]][];
expect(g).toBe("Filament");
expect(filament).toEqual([
{ group: "Filament", name: "Jade White", hex: "#f5f5dc", inStock: true },
// The note says in stock: the note wins over the source.
{ group: "Filament", name: "Bambu Green", hex: "#00ae42", inStock: true },
// Not at the source: out of stock.
{ group: "Filament", name: "Galaxy Black", inStock: false },
{ group: "Filament", name: "Charcoal", hex: "#2b2b2b", inStock: true },
]);
expect(finish).toEqual([{ group: "Finish", name: "Matte", inStock: true }]);
});
});
services/site/test/synced.test.ts
+38
View File
@@ -0,0 +1,38 @@
import { describe, expect, it } from "vitest";
import { withSynced, type Note } from "../src/lib/content.ts";
import { gamePlayers, gamePlayTime, type Game } from "../src/lib/library.ts";
import { deckArt, deckColors } from "../src/lib/world.ts";
const note = <T extends Note>(type: string, data: Record<string, unknown>, synced?: Record<string, unknown>) =>
({ path: `${type}/x.md`, type, name: "x", slug: "x", title: "X", body: "", acceptedAt: "", data: { type, ...data }, synced }) as unknown as T;
describe("synced values", () => {
it("sit under the note's own values: the note always wins", () => {
const n = withSynced(note("library-item", { cover: "[[mine.jpg]]", title: "X" }, { cover: "https://x/cover.jpg", year: 2006 }));
expect(n.data).toMatchObject({ cover: "[[mine.jpg]]", year: 2006 });
const bare = withSynced(note("library-item", { title: "X" }, { cover: "https://x/cover.jpg" }));
expect((bare.data as { cover?: string }).cover).toBe("https://x/cover.jpg");
});
it("leave a note with nothing synced exactly as it came", () => {
const n = note("board-game", { bggId: 1, blurb: "b" }, {});
expect(withSynced(n)).toBe(n);
const none = note<Game>("board-game", { bggId: 1, blurb: "b" });
expect(withSynced(none)).toBe(none);
expect([gamePlayers(none), gamePlayTime(none)]).toEqual([undefined, undefined]);
expect(deckArt(note("deck", { game: "[[Magic]]" }))).toBeUndefined();
expect(deckColors(note("deck", { game: "[[Magic]]" }))).toEqual([]);
});
it("give the Shelf players and play time, and the decks art and colors", () => {
const g = withSynced(note<Game>("board-game", { bggId: 1, blurb: "b" }, { minPlayers: 1, maxPlayers: 5, playTime: 70, minPlayTime: 40, maxPlayTime: 70 }));
expect([gamePlayers(g), gamePlayTime(g)]).toEqual(["1–5 players", "40–70 min"]);
const solo = withSynced(note<Game>("board-game", { bggId: 1, blurb: "b" }, { minPlayers: 1, maxPlayers: 1, playTime: 30 }));
expect([gamePlayers(solo), gamePlayTime(solo)]).toEqual(["1 player", "30 min"]);
const d = withSynced(note<Note<"deck">>("deck", { game: "[[Magic]]" }, { commanderArt: "https://cards.scryfall.io/a.jpg", colors: ["W", "U", "G"] }));
expect(deckArt(d)).toBe("https://cards.scryfall.io/a.jpg");
expect(deckColors(d)).toEqual(["White", "Blue", "Green"]);
const own = withSynced(note<Note<"deck">>("deck", { game: "[[Magic]]", cover: "[[deck.jpg]]" }, { commanderArt: "https://cards.scryfall.io/a.jpg" }));
expect(deckArt(own)).toBe("/media/deck.jpg");
});
});
vault-kit/24.01_PersonalWebsite/Site/_templates/Library item.md
+1
View File
@@ -13,6 +13,7 @@ verdict: # optional. A short review. No score.
note: # optional. Context, like where you are or how you came to it.
progress: # optional. Like 16/18 volumes. Synced from PersonalReader when linked.
anilistId: # optional. Links the item to PersonalReader's progress.
kavitaSeriesId: # optional. Links a book or light novel to its series in Kavita, for PersonalReader's progress.
---
<!-- Optional longer thoughts. -->