Background jobs runbook

Casino background work is stored in PostgreSQL. domain_jobs is the durable queue for scheduled/domain work; analytics, notification, Customer.io and realtime use their own transactional outbox tables. Redis is transport for realtime delivery, not the source of truth for jobs.

The scheduler process runs separately from the HTTP API on port 8793. Notifier and realtime always run as separate consumers. Streamer and Customer.io run only when their complete credential pairs are configured. Each active container runs one Node.js process as PID 1 with restart: unless-stopped.

Health and metrics

Every worker exposes:

Endpoint	Meaning
GET /health/live	The HTTP event loop responds. Used by Docker health checks.
GET /health/ready	The consumer is started and its poll heartbeat is fresh.
GET /health	Sanitised operational state; no stack traces or secrets.
GET /metrics	Prometheus metrics scraped by Alloy.

The Service Health Grafana dashboard shows worker heartbeat age, queue depth, queue lag and matching Loki failures. Provisioned alerts cover a worker being down/stalled for two minutes, three starts in 15 minutes, three consecutive job failures, any dead letter, and queue lag (60 seconds for realtime, five minutes for other queues). Firing and recovery notifications are grouped by alertname, service and queue.

Set the alert receiver on the monitoring host:

GRAFANA_ALERT_WEBHOOK_URL=https://alerts.example.internal/grafana
GRAFANA_ALERT_WEBHOOK_BEARER_TOKEN=optional-secret

Triage

1. In Grafana Explore, select Loki and search (Compose still labels services swaggbet-*):

   {service_name=~"swaggbet-(scheduler|streamer|notifier|customerio|realtime)"}
     |~ "worker_fatal|job_failed|worker_watchdog_stalled"

2. Inspect the durable row's last_error, attempts, available_at, locked_at and locked_by. Do not copy stack traces or payload secrets into an incident ticket.

3. Check whether Docker killed or restarted the process:

   docker inspect --format '{{json .State}}' <container-name>
   docker logs --since 30m <container-name>

   Pay particular attention to OOMKilled, ExitCode, Error, StartedAt and FinishedAt. An exit code of 1 follows a fatal runtime error; a watchdog-confirmed stall is terminated with SIGKILL.

4. Confirm PostgreSQL/provider connectivity and current queue lag. A normal provider/job failure uses retry/backoff and must not restart the process.

Safe dead-letter replay

Fix the root cause first and verify that the handler's side effect is idempotent. For domain_jobs, replay a single reviewed row in a transaction; never bulk-reset every dead letter:

BEGIN;

SELECT id, job_type, idempotency_key, attempts, last_error, payload
FROM domain_jobs
WHERE id = '<reviewed-job-id>' AND status = 'dead_letter'
FOR UPDATE;

UPDATE domain_jobs
SET status = 'pending',
    attempts = 0,
    available_at = CURRENT_TIMESTAMP,
    locked_at = NULL,
    locked_by = NULL,
    processed_at = NULL,
    updated_at = CURRENT_TIMESTAMP
WHERE id = '<reviewed-job-id>' AND status = 'dead_letter';

COMMIT;

Preserve last_error until a successful completion clears it. For another outbox, use its domain operation/admin tooling rather than adapting this SQL: status names and retry semantics differ.

Failure and persistence behaviour

• Processing is at least once. A row is completed only after its side effect succeeds; a stale lease can be reclaimed after a hard kill.
• Long jobs renew their queue lease. Job/provider errors are stored and retried with backoff; exceeding max_attempts produces a dead letter.
• Processed outbox rows are retained for seven days and completed domain_jobs for 30 days. Cleanup is bounded; dead letters are not deleted.
• PostgreSQL and Redis use named volumes. Container restart/recreation preserves the queue, but host loss and docker compose down -v do not. Database backup/HA is a separate infrastructure control.

Rollout and rollback

Apply database migration 0096_domain_jobs_runtime.sql before starting the new scheduler. Start scheduler, verify /health/ready and its metrics, then deploy the API version with in-process timers disabled. Deterministic schedule keys and leases make overlap safe during the transition.

For rollback, stop scheduler before restoring the previous API consumer. Jobs remain in PostgreSQL, so rollback does not discard pending work.