Production readiness and release runbook

Production is a single hardened VPS with off-host recovery, not a high-availability deployment. A host failure causes an outage. The release objectives are RPO ≤5 minutes, RTO ≤1 hour, 1,000 concurrent player/WebSocket sessions, and 100 sustained HTTP requests per second.

Non-negotiable release gates

A release is blocked unless all of the following are attached to the release:

• green clean-checkout CI, PostgreSQL integration tests, security scans, SBOM, and BuildKit provenance;
• a clean release-manifest.json containing the Git SHA, lockfile and migration checksums, and digest-pinned images;
• a provider capability evidence file accepted by pnpm test:provider-capabilities -- --evidence <file>;
• a successful authenticated browser smoke/rapid-refresh run;
• a 60-minute staging soak satisfying the checked-in k6 thresholds;
• a successful staging resilience drill;
• a pgBackRest/WAL health check and a quarterly isolated restore completed in less than one hour.

scripts/setup-production.sh refuses production deployment without a digest-pinned release manifest and enabled off-host pgBackRest configuration. Production-host compilation is not supported.

Maintenance deployment

1. Announce the 15–30 minute window and drain public traffic at Nginx.
2. Disable creation of new wallet/provider mutations.
3. Verify financial jobs, outbox rows, and provider callbacks are settled.
4. Run scripts/check-backup-health.sh, then create and verify a pgBackRest checkpoint.
5. Confirm the migration preflight was run against restored production data.
6. Deploy the CI release manifest. The one-shot db-migrate and isolated p2p-migrate containers are the only migration runners and connect as dedicated DDL roles. API and worker processes use DML-only runtime roles.
7. Wait for /health/ready. It checks PostgreSQL/schema/TOTP/KYC migration gates, Redis, KYC storage, and critical worker heartbeat endpoints.
8. Run login/token renewal, wallet read, game launch capability, provider reconciliation, and realtime reconnect smoke tests.
9. Reopen mutation traffic, then public traffic.
10. Monitor the release for 24 hours and 72 hours with no unresolved P0/P1 alerts.

Migrations must use expand/contract sequencing and remain compatible with the previous application image through the next release. Application rollback uses the previous image digests; schema rollback is not part of the normal rollback path.

Authentication

Player, admin, and support logins create one stable hashed device session. Browser login state remains in Secure HttpOnly SameSite=Lax cookies. API access tokens last ten minutes and include sub, sid, ver, aud, iss, iat, exp, and jti. Current and previous signing keys permit controlled rotation.

Revoking one device revokes its sid. Revoking all devices also increments the principal token version. PostgreSQL, Redis, and key-service failures are dependency failures (503), never invalid credentials (401). Support accounts cannot pass readiness until MFA is enrolled. AUTH_ALLOW_MFA_SKIP is a local-only enrollment escape hatch and startup rejects it in production. Password changes and admin device-session revocation commit atomically. Password reset links use a short Redis claim lease, become consumed only after the password write, and return 503 on token-store or database outages so transient failures remain safely retryable.

Admin and support vhosts are fail-closed behind ADMIN_SUPPORT_ALLOWED_CIDRS. Production setup requires at least one explicit VPN/office IP or CIDR and generates the shared nginx allow/deny all policy. The --disable-admin-network-restriction option is an audited break-glass override and must not be used for a normal release. Loopback application health checks continue to target the bound container ports directly.

During the rollout, invalidate all pre-release player/support/admin sessions. Rotate the access-token signing key and remove the previous key after the longest old token lifetime has elapsed.

Recovery

pgBackRest continuously archives WAL to encrypted off-host S3-compatible storage. Full backups run weekly, differential backups daily, and health/WAL archive probes run every 15 minutes. Restic independently protects notification media, P2P receipts, and the deployment manifest. Redis is rebuildable and is not restored.

• Manual database backup: scripts/backup-postgres.sh full
• Local pull dumps (always on-host, even when pgBackRest is off): scripts/backup-postgres-local.sh. Files land in /var/lib/swaggbet/db-backups/latest (swaggbet.dump, p2p.dump, *.schema.sql). Pull them with scripts/pull-db-backup.sh user@host.
• WAL and backup verification: scripts/check-backup-health.sh
• Operational assets: scripts/backup-operational-assets.sh
• Isolated restore evidence: scripts/restore-drill.sh

The restore drill creates an internal Docker network and a new temporary volume, restores both main and P2P databases, verifies their schemas, records duration, and removes only those isolated resources. Evidence is retained under /var/log/swaggbet/restore-drills.

Sensitive data

KYC uploads are authenticated bounded raw uploads (POST /user/kyc/uploads/:documentType), magic-byte checked, size bounded (5 MB), re-encoded through Sharp to strip metadata and written to private local filesystem storage under KYC_DOCUMENTS_DIR (absolute path required in production). PostgreSQL stores opaque IDs, relative object keys, hashes, state, and audit metadata. Upload admission is rate- and concurrency-limited. Replacing a document marks the previous row superseded and deletes its file. Pending/approved submissions are rejected inside the row-locked DB transaction.

Admin views receive audited permissioned local document URLs with Cache-Control: no-store; bytes are streamed only through /api/admin/users/:id/kyc/documents/:documentId. Production readiness probes writable KYC storage.

In Docker, API, admin, and the sensitive-data migrator share the kyc_documents volume (uid 1001). Operational asset backups include that volume read-only. If cutting over from a previous S3-backed deployment, copy existing bucket objects into the volume while preserving their stored relative keys (for example kyc/<user>/<documentId>.webp) before enabling traffic.

The rollout migrator copies and verifies legacy base64 records before clearing their columns. Run its dry preflight with pnpm migrate:legacy-sensitive-data, then run pnpm migrate:legacy-sensitive-data -- --apply while the application is drained and expansion migrations 0106/0107 are present. The database migration runner supports the exact boundary with pnpm --filter @repo/db migrate -- --through 0107_kyc_object_storage.

The immutable production installer always performs this sequence:

docker compose up db-expand-sensitive-data --exit-code-from db-expand-sensitive-data
docker compose up sensitive-data-migrate --exit-code-from sensitive-data-migrate
docker compose up db-migrate --exit-code-from db-migrate

The runner is idempotent: TOTP envelopes are decrypted and compared before commit; KYC objects are re-read and checked against their stored size and SHA-256 before a source blob is nulled. Contract migration 0108 still refuses to drop legacy columns while any source data remains. Take and verify a PITR checkpoint before running it, retain its output as release evidence, and do not enable application traffic between the data migration and 0108.

Readiness remains false while any legacy KYC blob or plaintext TOTP secret remains. TOTP secrets use versioned envelope encryption; recovery codes are hashed and atomically consumed once.

Dependency failure

External API errors use { error, code, requestId, retryable? }. Credential failure is 401, permission denial 403, state/idempotency conflict 409, invalid business input 422, provider failure 502, unavailable dependency or required configuration 503, and unexpected invariant failure 500.

Durable money, provider, and player-state effects must remain transaction plus outbox/job operations. Optional notification failure may not roll back a money mutation, but it must produce a metric and structured redacted log.

Providers and capacity

Provider evidence must cover launch, wager, win, rollback, free rounds, duplicate/out-of-order callbacks, deposit lifecycle, chargeback, withdrawal lifecycle, reconciliation, and continuation of accepted in-flight work after a configuration is disabled.

The checked-in soak refuses to run unless TARGET_ENVIRONMENT=staging. It requires 1,000 isolated player tokens, signed callback fixtures, and an isolated idempotent mutation fixture. It enforces read p95 <300 ms, mutation/provider p95 <750 ms, and non-induced errors <0.5%. Afterward, verify queue age, zero duplicate/lost ledger movements, zero lost outbox events, no exhausted pools, and at least 30% CPU, memory, and database headroom.