com.gregkilmer.www/docs/deploy.md

Deploying on Cosmos

The stack is three containers on homelab_network, none with a host port:

Container	Image target	Port	Reached as	Who can reach it
gk-content	content	4310	content.gregkilmer.com (API and /mcp), http://gk-content:4310 on homelab_network	Tailnet only
gk-admin	admin	4320	admin.gregkilmer.com	Tailnet only
gk-site	site	4321	beta.gregkilmer.com until cutover, then gregkilmer.com	Tailnet only until cutover, then public through Cloudflare

The old site keeps serving gregkilmer.com until phase 8 (cutover).

Files in the repo:

• Dockerfile: one target per service, plus check (typecheck and every test). Node is pinned to node:22.20.0-alpine.
• deploy/compose.yaml: the stack. Containers run as the vault's owner, with a read-only root filesystem, no capabilities and no-new-privileges.
• deploy/deploy.sh: builds check, then the three images tagged with the commit, and restarts the stack. If the new containers aren't healthy within two minutes, it puts the last good tag back. Run it again on the same commit and it does nothing (FORCE=1 rebuilds).
• deploy/rollback.sh: switches to the previous tag. Run it again to roll forward.
• deploy/pull-and-deploy.sh, with deploy/gk-deploy.service and gk-deploy.timer: every 5 minutes, Cosmos fetches master from GitHub into its own clone of the repo and runs deploy.sh when master has moved. A commit that failed to deploy isn't retried until master moves again.

On Cosmos

Everything lives in /home/greg/homelab/gregkilmer:

/home/greg/homelab/gregkilmer/
  repo/             clone of github.com/Patiun/com.gregkilmer.www, for deploys only
  compose.yaml      copied from deploy/compose.yaml by every deploy
  .env              GK_TAG, GK_PREVIOUS_TAG, VAULT_SITE, PUID, PGID, DEPLOY_HEALTHCHECK_URL
  content.env       content service secrets and sync settings
  admin.env         ADMIN_TOKEN
  site.env          site secrets: tokens, Turnstile, SMTP
  data/content/     content.db and submissions/ (backed up)

Kopia already covers /home/greg, so the .env files and data/ are in the encrypted backups.

First-time setup

1. Clone and settings. The repo is private, so give Cosmos read-only access with a deploy key:

   ssh-keygen -t ed25519 -N "" -C cosmos-gregkilmer-deploy -f ~/.ssh/gregkilmer_deploy
   

   Add ~/.ssh/gregkilmer_deploy.pub on GitHub under the repo's Settings, then Deploy keys, with write access left off. Then:

   mkdir -p /home/greg/homelab/gregkilmer/data/content
   cd /home/greg/homelab/gregkilmer
   GIT_SSH_COMMAND="ssh -i ~/.ssh/gregkilmer_deploy -o IdentitiesOnly=yes" \
     git clone git@github.com:Patiun/com.gregkilmer.www.git repo
   git -C repo config core.sshCommand "ssh -i ~/.ssh/gregkilmer_deploy -o IdentitiesOnly=yes"
   cp repo/deploy/stack.env.example .env
   cp repo/services/content/.env.example content.env
   cp repo/deploy/admin.env.example admin.env
   cp repo/services/site/.env.example site.env
   chmod 600 .env *.env
   

   In .env, check that PUID and PGID match id -u greg and id -g greg, so the content service can write notes back into the vault.

2. Tokens. Make one per role with openssl rand -hex 24:

   • In content.env, set CONTENT_TOKENS=reader:<r>,admin:<a>,intake:<i>,agent:<g>.
   • In admin.env, set ADMIN_TOKEN=<a>.
   • In site.env, set CONTENT_TOKEN=<r> and CONTENT_INTAKE_TOKEN=<i>.
   • Give agents <g>.

   Leave CONTENT_URL out of site.env; compose sets it.

3. The rest of the settings. Fill in Healthchecks, the Discord webhook, Turnstile, SMTP and the sync logins. The list is in /mnt/project-files/website/phase6/setup-on-cosmos.md and in each .env.example.

4. First deploy. Run it by hand once and watch it:

   /home/greg/homelab/gregkilmer/repo/deploy/pull-and-deploy.sh
   

   Then check:

   • docker compose -f /home/greg/homelab/gregkilmer/compose.yaml ps shows three healthy containers.
   • docker logs gk-content says Content service on 0.0.0.0:4310: N notes.

5. Deploy timer. This makes every later merge to master deploy on its own:

   sudo cp /home/greg/homelab/gregkilmer/repo/deploy/gk-deploy.{service,timer} /etc/systemd/system/
   sudo systemctl daemon-reload && sudo systemctl enable --now gk-deploy.timer
   

   • The service runs as greg, who must be in the docker group.
   • Make a Healthchecks check with a 5-minute period, and set its URL as DEPLOY_HEALTHCHECK_URL in .env. It turns red when GitHub can't be reached or a commit fails to deploy.
   • Logs: journalctl -u gk-deploy.

6. Nginx Proxy Manager. Add a proxy host for each hostname. Force SSL on each. For content, add proxy_buffering off; under Advanced, because MCP streams its responses.

   Host	Forward to	Access list
   admin.gregkilmer.com	http://gk-admin:4320	Tailnet only
   content.gregkilmer.com	http://gk-content:4310	Tailnet only
   beta.gregkilmer.com	http://gk-site:4321	Tailnet only

   • DNS: an A record for each pointing at 100.99.111.16 (TTL 300), the same as other tailnet hosts.
   • Certificates: Let's Encrypt can't reach a tailnet-only host over HTTP, so use NPM's DNS challenge with your DNS provider. This is the same open item as the *.patiun.com certificate.
   • Agents already on homelab_network can skip NPM and use http://gk-content:4310/mcp.

7. Turnstile. Add beta.gregkilmer.com and gregkilmer.com to the widget's hostnames.

8. Healthchecks. Create these checks:

   • one for the content service's full scan (HEALTHCHECK_URL), every 5 minutes;
   • one per sync job you turn on (Archidekt, Suwayomi, Bambuddy and Homebox hourly; BGG and Kavita daily; covers every 15 minutes).

9. Inventory note. Copy vault-kit/24.01_PersonalWebsite/GregKilmer.com stack.md into the vault's 24.01_PersonalWebsite folder, so cosmos-inventory stops listing the containers as undocumented.

Backups

• gk-content carries backup.dump=sqlite. cosmos-dump finds content.db in its /data mount and copies it with VACUUM INTO before Kopia runs.
  • content.db holds note history, the review queue, saved inquiries and orders, and the sync cache.
  • The vault mount has no SQLite files, so nothing else is dumped.
• Form attachments are plain files in data/content/submissions/, which Kopia picks up with the rest of /home/greg.
• The plan called for a cache folder in exclusions.conf. None is needed: the sync cache and search index live inside content.db and are small.
• gk-admin and gk-site keep no data.

Cutover (phase 8)

1. In NPM, point gregkilmer.com and www.gregkilmer.com at http://gk-site:4321 with no access list, and retire the old greg-profile-site container.
2. Turn on Cloudflare's proxy for both records with SSL mode Full (strict). Cloudflare doesn't cache HTML by default, which is what the site needs: pages change within a minute of a note changing.
3. The site already trusts X-Forwarded-Host and X-Forwarded-Proto for gregkilmer.com and its subdomains (security.allowedDomains in services/site/astro.config.mjs), so form posts pass Astro's origin check behind the proxy. It reads the visitor's address from CF-Connecting-IP for the form rate limits.

Day to day

• Deploy: merge to master on GitHub. Cosmos picks it up within 5 minutes. To deploy straight away, run sudo systemctl start gk-deploy.
• Retry a failed commit: FORCE=1 /home/greg/homelab/gregkilmer/repo/deploy/pull-and-deploy.sh.
• Roll back: /home/greg/homelab/gregkilmer/repo/deploy/rollback.sh, or set GK_TAG in .env to any image tag still on the host and run docker compose up -d.
• Logs: docker logs gk-content. Sync runs also appear on the admin page's Overview.
• Restart one service: docker compose -f /home/greg/homelab/gregkilmer/compose.yaml restart gk-site.