Realtime messaging

Private realtime pushes use HTTP writes + WebSocket delivery. Mutations enqueue durable outbox rows in Postgres; the realtime worker publishes to Redis; the API WebSocket gateway fans events to subscribed clients. SSE for support/withdrawal/notifications was removed — one WebSocket stack serves all private topics.

Architecture

Client                    API                         Worker
  |  POST /realtime/ticket  |                            |
  |------------------------>|  store ticket in Redis     |
  |  WS /realtime/connect   |                            |
  |  auth + subscribe       |                            |
  |<------------------------|  Redis pub/sub fan-out     |
  |                         |<---------------------------|
  |                         |  claim outbox → publish    |
  |                         |         Postgres outbox    |

Write path: route handler or service mutates DB inside a transaction and enqueues either:

• private_state_outbox — revision-coalesced snapshots (wallet balance, shop coins, notification inbox summary).
• realtime_event_outbox — support, withdrawal, and other sequenced domain events.

Read path: client opens WebSocket, authenticates with a one-time ticket plus its last durable revision map, optionally subscribes to resource streams, and receives event frames. On reconnect, unchanged topics are acknowledged in ready.revisions without rebuilding or retransmitting their snapshots. Resource subscriptions are replayed; broad HTTP reconciliation is reserved for explicit forced recovery.

Topics and event types

Topic	Subscription kind	Typical events
wallet.balance	(auto on auth for player)	wallet.balance.snapshot
shop.coins	(auto on auth for player)	shop.coins.snapshot
notification.inbox	(auto on auth for player)	notification.inbox.snapshot
player.quests	(auto on auth for player)	player.query.changed (revision-only; HTTP refetch)
player.vip	(auto on auth for player)	player.vip.snapshot (+ pending/history HTTP on revision advance)
player.wheels	(auto on auth for player)	player.query.changed
player.bonuses	(auto on auth for player)	player.bonuses.snapshot
player.settings	(auto on auth for player)	player.settings.snapshot
player.transactions	(auto on auth for player)	player.query.changed
player.support-inbox	(auto on auth for player)	player.query.changed
support.conversation	conversation + conversation id	support.message.created, support.conversation.updated
support.staff-feed	staff-feed + self	support.staff-feed.changed (personal + broadcast pool channel)
withdrawal.thread	withdrawal + withdrawal id	withdrawal.message.created, withdrawal.status.updated

Contracts live in @repo/shared/realtime (topics.ts, protocol.ts, parsers.ts).

Snapshot-covered live state: wallet balances, shop coin balance, notification unread count, active bonus issuances, VIP status, and user settings are revision-coalesced private-state snapshots. Player auth bootstraps only shell-critical wallet and unread snapshots; shop, bonuses, VIP, and settings load over HTTP when their features open. Subsequent mutations still publish snapshots through private_state_outbox. This avoids materializing and sending unrelated private payloads on every route.

Request ownership matrix (player client)

Domain	Owner while WS READY	HTTP allowed when	Notes
Auth.js session	Event-driven (login / profile update / 401 recovery)	Never on a timer or window focus	SSR-seeded mount must not probe; provider-internal reads use broadcast: false
Wallet balances	wallet.balance.snapshot	Offline, connecting grace, READY without this-connection snapshot, payment/crypto settlement	Active fiat/crypto pending may keep a short visible-tab poll; per-connection receipt (not durable revision alone) gates HTTP
Notification unread	notification.inbox.snapshot	Same fallback rules as wallet	Revision watermarks are pinned (gcTime: Infinity); identity reset is the only clear
Shop coins	HTTP on feature open, then shop.coins.snapshot	Feature-local fallback while disconnected	No connect-time payload
Bonus list	HTTP on feature open, then player.bonuses.snapshot	Feature-local fallback while disconnected	No connect-time payload
VIP status	HTTP on feature open, then player.vip.snapshot	Feature-local fallback; pending/history refetch on revision advance	No connect-time payload
User settings	HTTP on feature open, then player.settings.snapshot	Feature-local fallback while disconnected	No connect-time payload
Quests / wheels / transactions / support inbox lists	HTTP on feature open; player.query.changed refetches active queries	Visible disconnected polling; forced recovery	Not snapshot payloads
Public config versions	Focus/mount with 60s staleTime	Admin publish freshness	Idle slot pages should not stampede
Exchange rates	Shared one-shot + 120m refresh	Visible-tab interval only	Legitimate slot bootstrap
Onboarding / registration offers	Marketing routes (+ guests)	Authenticated slot/profile pages skip	Route-gated in CasinoFeatureProviders

Allowed HTTP exceptions on an idle authenticated slot page: game catalog / favorite state, shared exchange rates, and at most one snapshot-domain GET if connect-time bootstrap omitted that domain. Recurring GET /api/auth/session and recurring unread/balance/coins/bonus/VIP/settings GETs while READY are regressions.

Revision-only query signals: quests, wheels, transactions, and the player support inbox publish player.query.changed with { revision } only. Clients send their watermarks in AUTH. The gateway delivers only newer signals before READY and includes every durable topic watermark in ready.revisions. Snapshot-backed features compare those acknowledgements with their local HTTP/WS cache and refetch only active stale data; unopened features remain lazy. Normal reconnect is therefore gap-covered by Redis buffering plus revision comparison and does not launch a broad HTTP refetch; reconcilePlayerLiveState is for forced or malformed-transport recovery.

HTTP bootstrap responses must not overwrite a newer WS revision (client merge helpers). Optimistic local count updates must preserve the cached revision watermark so only a strictly newer server revision (incoming.revision > cached.revision) can overwrite state. Per-bet coin accruals enqueue outbox rows like wallet mutations; the worker coalesces to the latest revision per (userId, topic) before publish.

Ticket principals

Tickets carry a principalKind resolved at mint time:

Kind	Who	Ticket endpoint
player	Signed-in casino user	POST /realtime/ticket
support_operator	Support staff (agent / manager / admin role)	POST /realtime/ticket
admin_panel	Admin panel session with unified token	POST /realtime/ticket or admin BFF POST /api/admin/realtime/ticket

Subscription auth (subscription-auth.ts) enforces access per kind — e.g. staff feed requires support_operator (admin panel principals are rejected), conversation access checks ownership/assignment/pool rules.

Tickets are one-time (Redis GETDEL), TTL 30s, auth lease 10 minutes per connection. The ticket endpoint stores the already-validated principal claims server-side, avoiding a duplicate user read during the immediately following WS AUTH. Before lease expiry the client mints one new ticket and sends refresh_auth; auth_refreshed extends the existing socket without reconnect or subscription replay.

Dual-run removed

Support, withdrawal, and notification delivery no longer use SSE. Do not reintroduce /support/sse, staff SSE tokens, or parallel SSE broadcast paths. Wallet never used SSE; it continues on private_state_outbox.

Production notes

• Staff pool fan-out is O(1): pool-wide events publish to realtime:support:staff:broadcast. Staff-feed subscribers auto-join that channel; do not enqueue one outbox row per online operator.
• Hub delivery is channel-indexed: Redis messages deliver only to sockets registered on that channel (not a full-scan of all connections).
• Redis subscribe/unsubscribe is serialized per channel to avoid lost subscriptions under concurrent connect/disconnect.
• Worker publish never races a timeout into markFailed: slow publishes are logged but still awaited so Redis delivery is not duplicated via retry.
• Client invalidations for support are batched (~200ms) to avoid HTTP storms under chat load.
• Assignment/claim/priority/status/pool enqueue realtime in the same transaction as the domain write (conditional claim UPDATE prevents double-claim races; reassignment decrements the previous assignee's active-ticket counter).
• Maintenance pool-overdue emits one O(1) staff-feed broadcast after pooling.
• Prefer supportConversationOperations.updateStatusWithRealtime / updatePriorityWithRealtime / claimFromPool over bare CRUD helpers when adding new call sites.

Outbox and dead-letter recovery

Both outbox tables use the same lifecycle: pending → processing → processed | failed | dead_letter.

The worker (apps/workers/src/realtime) polls both outboxes, coalesces every private-state topic by max revision, publishes JSON events to Redis channels, and marks rows processed.

Stale locks: releaseStaleLocks returns stuck processing rows to failed (or dead_letter when attempts exhausted).

Dead letters: rows in dead_letter are not retried automatically. Inspect last_error, fix root cause, then manually reset status to pending and clear lock fields if a replay is safe (same pattern as other platform outboxes). Worker /health exposes deadLetter counts for both outboxes.

Operational signals: API /health reports Redis readiness; worker /health reports pending, published, failed, deadLetter, lastError.

Reconnect and HTTP reconcile

RealtimeTransport (packages/client/src/realtime/transport/client.ts):

1. Mint fresh ticket (player API or admin BFF issuer).
2. Connect WebSocket and AUTH with cached durable revisions.
3. On network reconnect, re-auth and resend desired resource subscriptions.
4. Before lease expiry, refresh AUTH in-band on the same socket.

Durable revisions make ordinary reconnect gap-safe: equal wallet/unread revisions produce no payload, newer revisions produce authoritative snapshots, and buffered Redis events flush after READY. Query signals follow the same comparison. Conversation/withdrawal threads still refetch on resubscribe because they are ordered resource streams. Forced recovery remains the safety valve for malformed transport or explicit operator/user action.

Snapshot and query-signal domains use useRealtimeGatedPolling so HTTP polling resumes only while the socket is down (quests, VIP, wheels, bonuses list, transactions, support inbox, shop coins, notification unread). When realtime is enabled, browser online reconnect is owned by RealtimeProvider (not a parallel refreshActivePlayerQueries storm). Protected Hono calls wait for a one-shot client session bootstrap so a stale SSR Bearer never produces a startup 401 wave.

Ops alerts: watch app_realtime_outbox_lag_seconds (threshold ≈ REALTIME_MAX_READY_LAG_MS), app_realtime_outbox_dead_letter_total, and app_realtime_publish_total{outcome="failure"} per topic. Use app_realtime_frame_bytes_total, app_realtime_bootstrap_duration_seconds, app_realtime_bootstrap_events, and app_realtime_auth_refresh_total to catch payload, materialization, and lease-refresh regressions without high-cardinality labels.

Environment variables

Variable	Required	Notes
REDIS_URL	Yes (live pushes)	Ticket store + pub/sub. API boots without it but realtime is degraded.
DATABASE_URL	Yes	Both outbox tables.
REALTIME_PORT	No	Worker listen port (default 8792).
REALTIME_WORKER_ID	No	Stable worker identity for lock ownership.
REALTIME_BATCH_SIZE	No	Outbox claim batch (default 50).
REALTIME_POLL_INTERVAL_MS	No	Poll interval (default 500).
REALTIME_LOCK_TTL_MS	No	Stale lock threshold (default 300000).
REALTIME_PUBLISH_CONCURRENCY	No	Parallel Redis publishes (default 10).
REALTIME_PUBLISH_TIMEOUT_MS	No	Per-publish timeout (default 10000).
REALTIME_MAX_READY_LAG_MS	No	Readiness lag threshold (default 60000).
REALTIME_PROCESSED_RETENTION_MS	No	Processed row retention (default 24h).

See .env.example for the full list.

Local verification

docker compose --project-directory . -f docker/compose/docker-compose.dev.yml up -d postgres redis
pnpm --filter @repo/db migrate   # includes 0086 (support operator FKs), 0087 (realtime_event_outbox)
pnpm dev                         # API + default realtime worker on 8792
pnpm test:realtime-e2e           # wallet + notification inbox + bootstrap snapshots + support subscribe smoke

Bootstrap order (do not regress): player sockets receive changed wallet/unread snapshots and newer query-signal revisions before the ready frame. ready.revisions reports all durable domain watermarks. Live Redis publishes that arrive during snapshot load are buffered and flushed after ready. Feature payloads (shop, bonuses, VIP, settings) must not be materialized merely because a route connected. Credentials: AUDIT_EMAIL / AUDIT_PASSWORD.

Deployment notes

1. Apply migrations 0086 and 0087 before deploying API/worker builds that enqueue realtime_event_outbox.
2. Deploy Redis and the realtime worker container (realtime:8792) alongside API; set REDIS_URL on both.
3. Deploy API + client packages together — WebSocket protocol and subscription kinds must match.
4. Admin panel mints tickets via BFF (POST /api/admin/realtime/ticket) so browser calls stay same-origin.
5. Monitor worker deadLetter and pending after cutover; SSE endpoints are gone — clients must use RealtimeProvider.