Production

This is the operator runbook for a live Omadx host: first-run bootstrap (DNS, host nginx, Let's Encrypt, firewall) and day-to-day production run. Release gates, backups, and soak tests live in Production readiness. Container inventory is in Container architecture.

Public brand and hostnames are Omadx. Docker Compose still uses project name swaggbet, so containers, the main database, Grafana service labels, and on-host paths remain swaggbet-*.

Hosts

Production is two machines, not one:

Host	Role	Typical hostname
App	Player site, admin, support, API, workers, Postgres, Redis, app Alloy	omadx.com (178.104.28.19)
Observability	Grafana LGTM (Grafana, Loki, Tempo, Mimir, central Alloy)	stat.omavx.com / obs.{domain}

The app host terminates public HTTPS and runs the casino. It forwards logs, metrics, and traces to the observability host. Deploy the observability stack before or in parallel with the first app install so Alloy has somewhere to send telemetry.

TEST is a separate app host (test.omavx.com). Both app hosts can share one Grafana if OBS_INGEST_ALLOW_CIDRS includes both public IPs and each app sets OBSERVABILITY_CLUSTER (omadx vs test).

First-run setup

This is the gap that is easy to miss in the local-dev docs. The player stack does not expose 443 itself. A host-level nginx (Debian/Ubuntu package, not a Docker nginx) is the public edge. Certbot issues certificates on that host. Docker publishes application ports on loopback; nginx proxies to them.

Prerequisites

• Debian/Ubuntu VPS with apt-get, root (or passwordless sudo), and enough disk for images + Postgres + backups.
• DNS for the names below pointed at the app host public IP, except obs.{domain} which points at the observability host.
• An email for Let's Encrypt (--email).
• At least one office/VPN CIDR for admin and support (--admin-allowed-cidrs). Setup aborts in production without it unless you pass the audited break-glass flag.
• A CI release-manifest.json with digest-pinned images. Host compilation is an emergency path only (--allow-source-build).

DNS records

scripts/setup-production.sh expects these names. Apex plus role subdomains must resolve before certbot can succeed:

Name	Points at	Purpose
{domain}	App host	Player site
www.{domain}	App host	301 to apex
api.{domain}	App host	Hono API + provider callbacks
admin.{domain}	App host	Admin panel (CIDR-locked)
support.{domain}	App host	Support desk (CIDR-locked)
docs.{domain}	App host	These VitePress docs
pay.{domain}	App host	P2P / pay surface
promo.{domain}	App host	Promo prelanding
obs.{domain}	Observability host	Grafana + OTLP/Loki/Mimir ingest

Campaign hostnames that are not {role}.{domain} go in nginx/extra-hosts (today promo.omadx.com) and need a matching nginx/{host}.conf plus their own certificate.

www is issued on the shared Let's Encrypt cert but is not enabled as its own nginx site; the apex config redirects it.

Cloudflare

• App hostnames may sit behind Cloudflare as usual.
• obs.{domain} must be DNS-only (grey cloud). Proxied (orange cloud) TLS breaks Alloy ingest on 4318 / 3100 / 9009. Grafana at a dedicated stats host such as stat.omavx.com follows the same rule if it terminates those ports.

What the installer does

scripts/setup-production.sh (normally invoked by scripts/deploy-from-zip.sh):

1. Installs ca-certificates, curl, nginx, certbot, openssl, ufw, and Docker if missing.
2. Writes secrets and generated values into /opt/swaggbet/shared/main.env and copies them onto the current release .env.
3. Installs host nginx site files from nginx/*.conf, rewriting the template domain to the target domain.
4. Briefly enables an ACME-only HTTP vhost, issues/expands Let's Encrypt certificates with webroot (/var/www/html), then enables the TLS sites.
5. Opens UFW for 22 / 80 / 443 (and replication CIDRs when configured). Application ports stay on loopback.
6. Pulls digest-pinned images (or emergency-bakes), runs one-shot migrate containers, starts Compose project swaggbet.
7. Installs keep-warm, local DB dump cron, and the node-helper service.

Observability first-run is a sibling script: scripts/setup-observability.sh. See Monitoring.

Host nginx and certificates

Nginx lives on the host:

Path	Role
/etc/nginx/sites-available/{host}.conf	Generated from nginx/ templates
/etc/nginx/sites-enabled/	Symlinks for live vhosts (default site removed)
/etc/letsencrypt/live/{cert-name}/fullchain.pem	Shared cert for apex + role names
/var/www/html	ACME webroot

Issuer command shape (the script runs this; you should not need to by hand):

certbot certonly --webroot -w /var/www/html --non-interactive --agree-tos \
  --cert-name "$DOMAIN" --email [email protected] -d "$DOMAIN"
# then --expand for www, api, admin, support, docs, pay, promo

Useful flags:

Flag	When to use
--email {addr}	Let's Encrypt registration (preferred)
--skip-certbot	Nginx files only; fails if fullchain.pem is missing
--observability-host {host}	Override default obs.{domain} for app Alloy

After a successful first issue, certbot renews via systemd timer. Reloading nginx after renew is handled by the packaged hook.

Optional names that are not in DNS yet are skipped with a warning rather than aborting the whole install. Apex must succeed because every vhost points at that certificate path.

The installer enables the docs.{domain} vhost and includes that name on the shared certificate. It does not copy VitePress HTML. After first-run (and after doc edits), publish a subdomain build (VITEPRESS_BASE=/) to /var/www/swaggbet-docs and reload only nginx:

pnpm --filter docs run build:subdomain
rsync -a --delete apps/docs/.vitepress/dist/ root@{app-host}:/var/www/swaggbet-docs/
ssh root@{app-host} 'chown -R www-data:www-data /var/www/swaggbet-docs && nginx -t && systemctl reload nginx'

Pages are then https://docs.{domain}/production, /monitoring, /environment-variables. Bookmarks under /docs/... 301 to the same path without the prefix. Do not recreate player or API nginx when updating docs.

Firewall and admin lock

UFW on the app host: SSH, HTTP, HTTPS. Docker-published Postgres/Redis/metrics stay loopback (or CIDR-locked for analytics replication).

Admin and support vhosts are fail-closed:

sudo bash scripts/setup-production.sh example.com \
  --release-manifest /root/release-manifest.json \
  --email [email protected] \
  --admin-allowed-cidrs '203.0.113.10/32,2001:db8:1234::/64'

ADMIN_SUPPORT_ALLOWED_CIDRS is written into env and rendered as nginx allow / deny all. --disable-admin-network-restriction is an audited break-glass override and must not be used for a normal release.

On-disk layout

Path	Purpose
/opt/swaggbet/releases/{id}	Immutable unpacked zip for that release
/opt/swaggbet/current	Symlink to the active release
/opt/swaggbet/current/.env	Env file Compose interpolates
/opt/swaggbet/shared/main.env	Durable secrets (survives release swaps)
/opt/swaggbet/shared/observability.env	Copied on the monitoring host after zip deploy
/var/lib/swaggbet/db-backups	On-host dumps (SWAGGBET_DB_BACKUP_*)

First install commands

Preferred path (zip + digest-pinned manifest). Observability host first:

pnpm package:production

# Observability host
unzip -p casin-production.zip scripts/deploy-from-zip.sh |
  sudo bash -s -- observability obs.example.com ./casin-production.zip --detach --open-firewall

# App host
unzip -p casin-production.zip scripts/deploy-from-zip.sh |
  sudo bash -s -- main example.com ./casin-production.zip \
    --release-manifest /root/release-manifest.json \
    --email [email protected] \
    --admin-allowed-cidrs '203.0.113.10/32' \
    --detach

Already-extracted tree:

sudo bash scripts/setup-observability.sh example.com --email [email protected]
sudo bash scripts/setup-production.sh example.com \
  --release-manifest /root/release-manifest.json \
  --email [email protected] \
  --admin-allowed-cidrs '203.0.113.10/32'

--allow-source-build bakes images on the VPS. That is emergency-only; off-host backup credentials remain required for a normal production posture.

Fill third-party secrets in /opt/swaggbet/shared/main.env after first boot (Fundist, payments, email, CRM). The installer generates platform secrets (AUTH_SECRET, wallet keys, DB passwords) and will not overwrite credentials already present on --update.

The full variable catalog is Environment variables.

Production run

After first install, the casino is a Compose project on the app host. Nginx and certbot are systemd services; apps are Docker.

Process model

Layer	How it runs
nginx, certbot timer, UFW	systemd on the host
swaggbet-* containers	docker compose project swaggbet
DB migrate	one-shot db-migrate / p2p-migrate containers during deploy
Keep-warm / local dumps / node-helper	scripts installed by setup

Always pass the project directory:

docker compose --project-directory /opt/swaggbet/current \
  -f /opt/swaggbet/current/docker/compose/docker-compose.yml \
  ps

Compose reads /opt/swaggbet/current/.env, not shared/main.env by itself. Setup copies shared secrets into the current release env, but a later bake that only updates shared/main.env will not change running containers. Pin image digests in both files, then recreate.

Image pins

Production compose requires digest-pinned images, for example:

WEB_IMAGE=swaggbet-web@sha256:…
API_IMAGE=swaggbet-api@sha256:…

Same pattern for ADMIN_IMAGE, SUPPORT_IMAGE, P2P_IMAGE, PROMO_IMAGE, worker images, POSTGRES_IMAGE, and SCHEDULER_IMAGE. A floating :latest tag on current/.env can silently roll a surface back to an older bake even when shared/main.env still has the intended digest.

Health

GET /health/ready on the API (via https://api.{domain}/health/ready or loopback 127.0.0.1:8787) is the deploy gate. It checks PostgreSQL/schema/TOTP/KYC, Redis, KYC storage, and critical worker heartbeats.

Workers also expose /health/live, /health/ready, /health, and /metrics. See Background jobs and Monitoring.

Routine operations

# Status
docker compose --project-directory /opt/swaggbet/current \
  -f /opt/swaggbet/current/docker/compose/docker-compose.yml ps

# Logs (Compose still names containers swaggbet-*)
docker logs --tail=200 --since 30m swaggbet-api
docker logs --tail=200 swaggbet-web
docker logs --tail=200 swaggbet-alloy

# Recreate after env/image pin changes
docker compose --project-directory /opt/swaggbet/current \
  -f /opt/swaggbet/current/docker/compose/docker-compose.yml \
  up -d --force-recreate web api

Reload host nginx after site-file changes: nginx -t && systemctl reload nginx.

Updates

# Zip deploy (preferred)
unzip -p casin-production.zip scripts/deploy-from-zip.sh |
  sudo bash -s -- main example.com ./casin-production.zip \
    --release-manifest /root/release-manifest.json --detach

# In-place update of an already-extracted tree
sudo bash scripts/setup-production.sh example.com --update \
  --release-manifest /root/release-manifest.json

--update preserves third-party credentials already in env, bumps release/deployment IDs, fills missing observability keys, and restarts.

Maintenance window, drain, backup health, and 24/72-hour watch are in Production readiness.

Where to look when something is wrong

1. Host nginx: systemctl status nginx, error.log, nginx -t.
2. Certificates: certbot certificates, expiry, whether optional names were skipped at first run.
3. Compose: docker ps -a --filter name=swaggbet-, container health, OOM.
4. Env: compare current/.env vs shared/main.env (especially *_IMAGE and NEXT_PUBLIC_*).
5. Grafana: Monitoring — core service down, metrics/logs/traces absent.

NEXT_PUBLIC_* values are inlined when the web/admin/support images are baked. Changing them in .env and recreating without a new image does not update the player bundle.