com.gregkilmer.www/docs/deploy.md

# Deploying on Cosmos

The stack is three containers on `homelab_network`, none with a host port:

| Container | Image target | Port | Reached as | Who can reach it |
| --- | --- | --- | --- | --- |
| `gk-content` | `content` | 4310 | `content.gregkilmer.com` (API and `/mcp`), `http://gk-content:4310` on homelab_network | Tailnet only |
| `gk-admin` | `admin` | 4320 | `admin.gregkilmer.com` | Tailnet only |
| `gk-site` | `site` | 4321 | `beta.gregkilmer.com` until cutover, then `gregkilmer.com` | Tailnet only until cutover, then public through Cloudflare |

The old site keeps serving gregkilmer.com until phase 8 (cutover).

Files in the repo:

- `Dockerfile`: one target per service, plus `check` (typecheck and every test). Node is pinned to `node:22.20.0-alpine`.
- `deploy/compose.yaml`: the stack. Containers run as the vault's owner, with a read-only root filesystem, no capabilities and `no-new-privileges`.
- `deploy/deploy.sh`: builds `check`, then the three images tagged with the commit, and restarts the stack. If the new containers aren't healthy within two minutes, it puts the last good tag back. Run it again on the same commit and it does nothing (`FORCE=1` rebuilds).
- `deploy/rollback.sh`: switches to the previous tag. Run it again to roll forward.
- `deploy/pull-and-deploy.sh`, with `deploy/gk-deploy.service` and `gk-deploy.timer`: every 5 minutes, Cosmos fetches master from GitHub into its own clone of the repo and runs `deploy.sh` when master has moved. A commit that failed to deploy isn't retried until master moves again.

## On Cosmos

Everything lives in `/home/greg/homelab/gregkilmer`:

```
/home/greg/homelab/gregkilmer/
  repo/             clone of github.com/Patiun/com.gregkilmer.www, for deploys only
  compose.yaml      copied from deploy/compose.yaml by every deploy
  .env              GK_TAG, GK_PREVIOUS_TAG, VAULT_SITE, PUID, PGID, DEPLOY_HEALTHCHECK_URL
  content.env       content service secrets and sync settings
  admin.env         ADMIN_TOKEN
  site.env          site secrets: tokens, Turnstile, SMTP
  data/content/     content.db and submissions/ (backed up)
```

Kopia already covers `/home/greg`, so the `.env` files and `data/` are in the encrypted backups.

## First-time setup

1. **Clone and settings.** The repo is private, so give Cosmos read-only access with a deploy key:

   ```sh
   ssh-keygen -t ed25519 -N "" -C cosmos-gregkilmer-deploy -f ~/.ssh/gregkilmer_deploy
   ```

   Add `~/.ssh/gregkilmer_deploy.pub` on GitHub under the repo's Settings, then Deploy keys, with write access left off. Then:

   ```sh
   mkdir -p /home/greg/homelab/gregkilmer/data/content
   cd /home/greg/homelab/gregkilmer
   GIT_SSH_COMMAND="ssh -i ~/.ssh/gregkilmer_deploy -o IdentitiesOnly=yes" \
     git clone git@github.com:Patiun/com.gregkilmer.www.git repo
   git -C repo config core.sshCommand "ssh -i ~/.ssh/gregkilmer_deploy -o IdentitiesOnly=yes"
   cp repo/deploy/stack.env.example .env
   cp repo/services/content/.env.example content.env
   cp repo/deploy/admin.env.example admin.env
   cp repo/services/site/.env.example site.env
   chmod 600 .env *.env
   ```

   In `.env`, check that `PUID` and `PGID` match `id -u greg` and `id -g greg`, so the content service can write notes back into the vault.

2. **Tokens.** Make one per role with `openssl rand -hex 24`:
   - In `content.env`, set `CONTENT_TOKENS=reader:<r>,admin:<a>,intake:<i>,agent:<g>`.
   - In `admin.env`, set `ADMIN_TOKEN=<a>`.
   - In `site.env`, set `CONTENT_TOKEN=<r>` and `CONTENT_INTAKE_TOKEN=<i>`.
   - Give agents `<g>`.

   Leave `CONTENT_URL` out of `site.env`; compose sets it.

3. **The rest of the settings.** Fill in Healthchecks, the Discord webhook, Turnstile, SMTP and the sync logins. The list is in `/mnt/project-files/website/phase6/setup-on-cosmos.md` and in each `.env.example`.

4. **First deploy.** Run it by hand once and watch it:

   ```sh
   /home/greg/homelab/gregkilmer/repo/deploy/pull-and-deploy.sh
   ```

   Then check:
   - `docker compose -f /home/greg/homelab/gregkilmer/compose.yaml ps` shows three healthy containers.
   - `docker logs gk-content` says `Content service on 0.0.0.0:4310: N notes`.

5. **Deploy timer.** This makes every later merge to master deploy on its own:

   ```sh
   sudo cp /home/greg/homelab/gregkilmer/repo/deploy/gk-deploy.{service,timer} /etc/systemd/system/
   sudo systemctl daemon-reload && sudo systemctl enable --now gk-deploy.timer
   ```

   - The service runs as `greg`, who must be in the `docker` group.
   - Make a Healthchecks check with a 5-minute period, and set its URL as `DEPLOY_HEALTHCHECK_URL` in `.env`. It turns red when GitHub can't be reached or a commit fails to deploy.
   - Logs: `journalctl -u gk-deploy`.

6. **Nginx Proxy Manager.** Add a proxy host for each hostname. Force SSL on each. For `content`, add `proxy_buffering off;` under Advanced, because MCP streams its responses.

   | Host | Forward to | Access list |
   | --- | --- | --- |
   | `admin.gregkilmer.com` | `http://gk-admin:4320` | Tailnet only |
   | `content.gregkilmer.com` | `http://gk-content:4310` | Tailnet only |
   | `beta.gregkilmer.com` | `http://gk-site:4321` | Tailnet only |

   - DNS: an A record for each pointing at 100.99.111.16 (TTL 300), the same as other tailnet hosts.
   - Certificates: Let's Encrypt can't reach a tailnet-only host over HTTP, so use NPM's DNS challenge with your DNS provider. This is the same open item as the `*.patiun.com` certificate.
   - Agents already on homelab_network can skip NPM and use `http://gk-content:4310/mcp`.

7. **Turnstile.** Add `beta.gregkilmer.com` and `gregkilmer.com` to the widget's hostnames.

8. **Healthchecks.** Create these checks:
   - one for the content service's full scan (`HEALTHCHECK_URL`), every 5 minutes;
   - one per sync job you turn on (Archidekt, Suwayomi, Bambuddy and Homebox hourly; BGG and Kavita daily; covers every 15 minutes).

9. **Inventory note.** Copy `vault-kit/24.01_PersonalWebsite/GregKilmer.com stack.md` into the vault's `24.01_PersonalWebsite` folder, so `cosmos-inventory` stops listing the containers as undocumented.

## Backups

- `gk-content` carries `backup.dump=sqlite`. `cosmos-dump` finds `content.db` in its `/data` mount and copies it with `VACUUM INTO` before Kopia runs.
  - `content.db` holds note history, the review queue, saved inquiries and orders, and the sync cache.
  - The vault mount has no SQLite files, so nothing else is dumped.
- Form attachments are plain files in `data/content/submissions/`, which Kopia picks up with the rest of `/home/greg`.
- The plan called for a cache folder in `exclusions.conf`. None is needed: the sync cache and search index live inside `content.db` and are small.
- `gk-admin` and `gk-site` keep no data.

## Cutover (phase 8)

1. In NPM, point `gregkilmer.com` and `www.gregkilmer.com` at `http://gk-site:4321` with no access list, and retire the old `greg-profile-site` container.
2. Turn on Cloudflare's proxy for both records with SSL mode Full (strict). Cloudflare doesn't cache HTML by default, which is what the site needs: pages change within a minute of a note changing.
3. The site already trusts `X-Forwarded-Host` and `X-Forwarded-Proto` for gregkilmer.com and its subdomains (`security.allowedDomains` in `services/site/astro.config.mjs`), so form posts pass Astro's origin check behind the proxy. It reads the visitor's address from `CF-Connecting-IP` for the form rate limits.

## Day to day

- **Deploy:** merge to master on GitHub. Cosmos picks it up within 5 minutes. To deploy straight away, run `sudo systemctl start gk-deploy`.
- **Retry a failed commit:** `FORCE=1 /home/greg/homelab/gregkilmer/repo/deploy/pull-and-deploy.sh`.
- **Roll back:** `/home/greg/homelab/gregkilmer/repo/deploy/rollback.sh`, or set `GK_TAG` in `.env` to any image tag still on the host and run `docker compose up -d`.
- **Logs:** `docker logs gk-content`. Sync runs also appear on the admin page's Overview.
- **Restart one service:** `docker compose -f /home/greg/homelab/gregkilmer/compose.yaml restart gk-site`.