Monitoring

Omadx uses a self-hosted Grafana LGTM stack on a dedicated monitoring host. Application servers run a lightweight Grafana Alloy agent and forward OTLP, logs, and Prometheus scrapes to that stack. Player browsers do not talk to Grafana; client diagnostics go to same-origin /api/track and show up as structured logs.

Repo detail and telemetry-loss commands live in observability/README.md. This page is the operator map; use the README when a pipeline is empty.

Architecture

App host                         Observability host
─────────                        ──────────────────
swaggbet-* ──/metrics──► Alloy ──OTLP :4318──────► central Alloy
container logs ────────► Alloy ──Loki  :3100/push─► Loki ──► Grafana
node-exporter ─────────► Alloy ──Mimir :9009/push─► Mimir
postgres-exporter ─────► Alloy                      Tempo (traces)

Service	Port (internal)	Purpose
Grafana	3000	Dashboards, Explore, alert rules
Loki	3100	Logs
Tempo	3200 / 4317 / 4318	Traces
Mimir	9009	Metrics
Alloy	12345 / 4317 / 4318	Central OTLP collector
node-exporter	9100	Host CPU / RAM / disk / network
postgres-exporter	9187	App-host Postgres (swaggbet_monitor)
MinIO	9000	Object storage for Loki / Tempo / Mimir

Public Grafana is https://obs.{domain} or a dedicated stats host such as https://stat.omavx.com. Docker services bind loopback; host nginx terminates TLS.

Retention defaults: Loki 14 days (336h), Tempo 7 days (168h), Mimir 30 days (720h). Tune observability/config/*.yaml if disk is tight.

How monitoring is deployed

Copy the zip (or observability/ tree) onto the monitoring host. Point DNS obs.{domain} at it. Cloudflare must be grey-cloud (DNS-only) for that name so ingest ports are not proxied.

# First install — Docker, nginx TLS, certbot, LGTM stack
sudo bash scripts/setup-observability.sh example.com --email [email protected]

# Hostname form also works
sudo bash scripts/setup-observability.sh obs.example.com --email [email protected]

# Redeploy from the local tree
sudo bash scripts/setup-observability.sh example.com --update

# Open UFW 80/443 (Grafana + ACME). Ingest stays IP-restricted.
sudo bash scripts/setup-observability.sh example.com --update --open-firewall

Zip path (from scripts/README.md):

unzip -p casin-production.zip scripts/deploy-from-zip.sh |
  sudo bash -s -- observability obs.example.com ./casin-production.zip --detach --open-firewall

The script:

1. Installs Docker, nginx, and certbot.
2. Generates Grafana/MinIO secrets into observability/.env (and /opt/swaggbet/shared/observability.env after a zip deploy).
3. Remaps Docker host ports so nginx owns public 443 / 4318 / 3100 / 9009.
4. Issues a Let's Encrypt cert for obs.{domain}.
5. Starts the stack and prints the OBSERVABILITY_CENTRAL_* values to paste onto the app host.
6. Verifies Mimir/Loki/Tempo datasource health and that alert rules are provisioned.

Local laptop (no nginx): cd observability && cp .env.example .env && docker compose up -d. Grafana is http://localhost:3300.

Security

• Change GRAFANA_ADMIN_PASSWORD and MINIO_ROOT_PASSWORD (generated on first install when placeholders remain).
• Firewall ingest (4318, 3100, 9009) to app-host IPs. Keep 443 for Grafana.
• Set OBS_INGEST_ALLOW_CIDRS to app-host public IPs when they are not RFC1918. Omadx production (178.104.28.19) and TEST (167.233.251.127) must both be listed or Grafana only sees one site.
• App hosts set OBSERVABILITY_CLUSTER (omadx vs test) so series do not mix under a generic production label.

Grafana login

Admin user/password are printed at the end of a successful setup run. For QA, create a Viewer user in Grafana — do not share the admin password.

App-host wiring

setup-production.sh enables OTEL and defaults central endpoints to obs.{domain} (override with --observability-host):

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4318
OBSERVABILITY_CENTRAL_ALLOY_ENDPOINT=https://obs.example.com:4318
OBSERVABILITY_CENTRAL_LOKI_ENDPOINT=https://obs.example.com:3100/loki/api/v1/push
OBSERVABILITY_CENTRAL_MIMIR_ENDPOINT=https://obs.example.com:9009/api/v1/push
OBSERVABILITY_ENVIRONMENT=production
OBSERVABILITY_CLUSTER=omadx
OBSERVABILITY_TRACE_SAMPLE_RATIO=0.1
OBSERVABILITY_OTLP_METRICS_ENABLED=false
OBSERVABILITY_METRICS_ENABLED=false
OBSERVABILITY_TRACES_ENABLED=true

HTTP RED metrics come from /metrics scrapes, not OTLP app metrics (those stay off). Without the OBSERVABILITY_CENTRAL_* URLs, app Alloy runs standalone and Grafana stays empty.

Metrics

Alloy scrape labels in Mimir:

Label service	Source
swaggbet-api, swaggbet-web, swaggbet-admin, swaggbet-support, swaggbet-p2p	App /metrics
swaggbet-realtime, swaggbet-notifier, swaggbet-scheduler, …	Worker /metrics
swaggbet-node	App-host node-exporter
swaggbet-postgres	App-host postgres-exporter
observability-node	Observability-host node-exporter

Platform and HTTP

Series	Meaning
up{service=~"swaggbet-.*"}	Scrape success per service
app_http_request_duration_seconds	HTTP latency histogram (RED)
app_http_requests_total	Request counters by route / status

Exact HTTP metric names follow the API/web /metrics exporters; the Service Health dashboard is the source of truth for panels.

Workers and queues

Series	Meaning
app_worker_running	Worker runtime is up
app_worker_last_heartbeat_timestamp_seconds	Event-loop heartbeat
app_worker_consecutive_failures	Consecutive poll failures
app_background_job_runs_total	Job runs by outcome
app_background_job_duration_seconds	Job duration histogram
app_background_job_last_success_timestamp	Last success
app_background_job_consecutive_failures	Consecutive job failures
app_queue_depth	Durable queue rows by status
app_queue_oldest_ready_age_seconds	Oldest ready-row lag

Worker health HTTP: /health/live, /health/ready, /health, /metrics. Runbook: Background jobs.

Host and Postgres

Series	Meaning
node_*	CPU, memory, disk, network, load
pg_*	Postgres exporter (connections, up)

postgres-exporter uses the read-only swaggbet_monitor role (MONITORING_DATABASE_PASSWORD). setup-production.sh re-applies pg_monitor grants on every deploy.

Dashboards

Provisioned under Grafana (folder Observability):

Dashboard	File / uid	What it answers
Collection Verification	collection-verification	Are metrics, logs, and traces arriving?
Service Health	service-health	HTTP RED, workers, queues, process health
Host & Database	host-database	Node + Postgres
Player Experience	player-experience	Player-facing latency and errors
Marketing Campaigns	marketing-campaigns	UTM / registration / deposit cohort (needs app Postgres datasource)
Mercado Pago Reconciliation	mercadopago-reconciliation	P2P MP polling health

Marketing Campaigns needs POSTGRES_* on the observability host (prefer a read-only role). Leave host/user/database empty to skip the datasource rather than shipping a broken Grafana plugin.

Alerts

Rules are provisioned from observability/grafana/provisioning/alerting/. Contact point: ops-alerts. Folder: Observability. Evaluate interval is 1 minute unless noted.

Platform (platform-alerts.yaml)

Alert	Fires when	Severity
Core service is down	up missing for swaggbet-(api|web|admin|support|p2p) for 2m	critical
Application metrics are absent	No app metrics in Mimir	critical
Application logs are absent	Log pipeline silence	critical
Application traces are absent	OTLP / Tempo silence	warning
swaggbet-api traces are absent	API traces specifically gone	warning
Loki service_name is unknown_service	Resource attributes not set	warning
Average HTTP latency is high	Mean app_http_request_duration_seconds above 1s	warning
Error logs appearing in Loki	{service_name=~"swaggbet-.+", level="error"}	warning

Infrastructure (infrastructure-alerts.yaml)

Alert	Fires when	Severity
Node exporter is down	App or obs node-exporter scrape failed	critical
Postgres exporter is down	swaggbet-postgres scrape failed	critical
Postgres is unreachable	Exporter pg_up missing	critical
Host disk space is low	Filesystem > 85% used	critical
Host memory pressure	Memory available < 10%	warning
Host CPU load average is high	node_load5 above CPU count for 5m	warning
Postgres connections are high	Connections > 80% of max	warning

Workers (worker-alerts.yaml)

Alert	Fires when	Severity
Background worker is down	Worker up / running missing for 2m	critical
Background worker event loop stalled	Heartbeat stale	critical
Background worker crash loop	Three starts in 15m	critical
Background job repeatedly failing	Three consecutive job failures	warning
Durable queue has dead letters	Any dead letter	warning
Durable queue processing lag	Ready-row age: 60s realtime, 5m other queues	warning

Sample / smoke (sample-alerts.yaml)

Alert	Notes
[SAMPLE] Always-firing smoke test	Provisioned paused — unpause to test Telegram
[SAMPLE] Always-OK evaluation check	Should stay Normal; does not need app scrapes

Telegram does not send RESOLVED follow-ups (disableResolveMessage: true). Recovered alerts remain visible in Grafana.

Telegram and webhooks

Provisioned contact point ops-alerts (zz-telegram-contact.yaml when Telegram is set).

# Observability host .env
TELEGRAM_ALERT_BOT_TOKEN=123456:ABC...
TELEGRAM_ALERT_CHAT_ID=-1001234567890

# Optional extra HTTPS sink on the same contact point
GRAFANA_ALERT_WEBHOOK_URL=https://hooks.example.com/alerts
GRAFANA_ALERT_WEBHOOK_BEARER_TOKEN=optional-bearer

Create the bot with @BotFather, start a chat or add the bot to a group, then --update the observability host. Chat id is quoted as a string so Grafana 11 accepts negative group ids.

Do not use http://127.0.0.1:9 as a dummy webhook — connection refused retries the whole contact point (including Telegram). Do not leave an empty Bearer token on the contact point.

Messages use the HTML template in 01-notification-templates.yaml: status, severity, instance summaries, one action line, and Rule / Dashboard / Silence links. Notifications group by alertname + severity.

Verify after deploy

On the observability host:

source observability/.env   # or /opt/swaggbet/shared/observability.env
GRAFANA_URL="http://127.0.0.1:${GRAFANA_HOST_PORT:-3000}" \
  GRAFANA_USER="$GRAFANA_ADMIN_USER" \
  GRAFANA_PASSWORD="$GRAFANA_ADMIN_PASSWORD" \
  bash scripts/verify-grafana-alerts.sh

Then open Collection Verification and confirm up series for swaggbet-* with the expected cluster label.

When Grafana is empty

Check both hosts. Typical app-host failures: Alloy not running, OBSERVABILITY_CENTRAL_* still host.docker.internal, UFW allowing Mimir but not Loki/OTLP, Alloy unable to read Docker *-json.log. Typical obs-host failures: grey-cloud DNS not applied, ingest CIDR missing the app IP, nginx not bound on 4318/3100/9009.

setup-production.sh prints a non-fatal TELEMETRY DEGRADED warning when Alloy health or central endpoints fail during verify.

Full command list: observability/README.md (telemetry-loss triage).