Observability & Monitoring

Orange uses OpenTelemetry for collecting application metrics, traces, and logs.

Overview

The monitoring stack:

1. Application Instrumentation - FastAPI, SQLAlchemy, HTTPx auto-instrumentation
2. OpenTelemetry Collector - Collects system, database, and application metrics
3. Custom Metrics - Heartbeat monitoring for service health

Collected Metrics

Application Metrics

Service Health Monitoring:

• service.status - Health status (1 = healthy, 0 = down)
• service.last_seen - Last heartbeat timestamp
• Heartbeat interval: 30 seconds

HTTP Request Metrics (FastAPI auto-instrumentation):

• http.server.request.duration - Request latency
• http.server.request.count - Request count by endpoint/status
• http.server.active_requests - Active concurrent requests

Database Query Metrics (SQLAlchemy instrumentation):

• Query duration and count
• Connection pool usage
• Slow query detection

System Metrics (Host)

Collected every 30 seconds via OpenTelemetry Collector:

Metric	Description
system.cpu.utilization	CPU usage percentage per core
system.cpu.time	CPU time by state (user, system, idle)
system.cpu.load_average.1m/5m/15m	Load averages
system.memory.usage	Memory used (bytes)
system.memory.utilization	Memory usage percentage
system.disk.io	Disk I/O bytes (read/write)
system.disk.operations	Disk operations count
system.network.io	Network I/O bytes
system.network.packet.count	Network packets sent/received
system.network.errors	Network transmission errors
system.filesystem.usage	Disk space used per mount
system.filesystem.utilization	Disk usage percentage
system.process.count	Number of processes
system.uptime	System uptime in seconds

PostgreSQL Metrics

Collected every 60 seconds:

Metric	Description
postgresql.backends	Active database connections
postgresql.commits	Transaction commits
postgresql.rollbacks	Transaction rollbacks
postgresql.db_size	Database size in bytes
postgresql.deadlocks	Deadlock count
postgresql.database.locks	Active locks by mode
postgresql.sequential_scans	Sequential table scans

Attributes: db.name, db.user, host.name, deployment.environment

Redis Metrics

Collected every 30 seconds:

Metric	Description
redis.clients.connected	Connected clients
redis.commands.processed	Total commands processed
redis.memory.used	Memory used (bytes)
redis.keyspace.keys	Number of keys per database

Configuration

Environment Variables

# Enable/Disable OpenTelemetry
OTEL_ENABLED=true

# Service Identification
OTEL_SERVICE_NAME=santra-be        # API service name
OTEL_WORKER_NAME=santra-worker     # Celery worker name
ENVIRONMENT=production              # Environment tag

# Heartbeat Interval
HEARTBEAT_TIME=30.0  # seconds

# Hostname Override (optional)
HOST_HOSTNAME=production-server-01

Dashboard

A pre-configured SigNoz dashboard is available at config/signoz-dashboard.json. This dashboard includes visualizations for:

• API request rate and latency
• Database connection pool usage
• System resource utilization
• Service health status
• Error rates by endpoint

Import this file into SigNoz to get started with monitoring.

Traces

All HTTP requests and database queries are automatically traced via OpenTelemetry instrumentation.

Logs

Logs are automatically correlated with traces, including trace ID and span ID for request tracing.

PostgreSQL Slow Query Logging (Azure Flexible Server)

Goal: capture every statement that runs longer than 2 s (log_min_duration_statement = 2000 ms) so we can find slow queries in prod.

Reality check (read first)

Prod Postgres is Azure Database for PostgreSQL Flexible Server — primary jeff-main-db-restore.* plus a read replica. Consequences:

• It is managed: there is no local log file on our VM. The collector's filelog/postgresql receiver (path POSTGRESQL_LOG_FILE) only works for the optional --profile postgres container and is dormant in prod — the log volume mount in docker-compose.prod.yml is commented out. Do not rely on it for managed PG.
• Slow queries leave Azure only through a Diagnostic Setting, not a file.
• The slow analytical reads run on the replica (≈22 s queries profiled there). log_min_duration_statement and the resulting logs are per-node — you must enable it on both the primary and the replica, and the replica is the one that matters most.
• This is the per-execution slow-query log (real SQL + duration). It is complementary to pg_stat_statements (aggregate, normalized top-N) — also per-node, queried separately on each server.

Phase 1 — minimal, stays in Azure (do this first)

1. Set the threshold on the primary and the replica. Dynamic — no restart.

   az postgres flexible-server parameter set \
     --resource-group <rg> --server-name jeff-main-db-restore \
     --name log_min_duration_statement --value 2000
   # repeat with --server-name <replica-server-name>

   Portal equivalent: Server parameters → log_min_duration_statement → 2000.

   Verify it landed on each node (we have read-only access to the replica):

   SHOW log_min_duration_statement;   -- expect "2s"

   On Flexible Server replicas most parameters inherit from the primary. If the replica doesn't reflect the value, set it explicitly on the replica too. Leave log_statement = none — that logs by type, not duration, and would flood logs.

2. Route logs out — add a Diagnostic Setting on each server exporting the PostgreSQLLogs category to a Log Analytics workspace (skip if one already exists):

   az monitor diagnostic-settings create --name pg-logs-to-law \
     --resource <flexible-server-resource-id> \
     --workspace <log-analytics-workspace-id> \
     --logs '[{"category":"PostgreSQLLogs","enabled":true}]'

3. Read the slow queries:

   • Query Performance Insight (portal, built on pg_stat_statements) — top queries by duration, zero query-writing.
   • KQL over Log Analytics for the actual >2 s statements:

     AzureDiagnostics
     | where Category == "PostgreSQLLogs" and Message has "duration:"
     | project TimeGenerated, Resource, Message   // Message: "duration: 2351.1 ms  statement: SELECT ..."
     | order by TimeGenerated desc

Phase 2 — optional, unify into SigNoz

Only if we want these alongside the rest of our telemetry. Replaces the dead filelog path for managed PG:

1. Point the Diagnostic Setting at an Event Hub (in addition to / instead of Log Analytics).
2. Add the contrib collector's azureeventhub receiver to otel-collector-config.yaml and a new logs/postgresql-azure pipeline → the existing SigNoz otlp exporter.

A 2 s threshold keeps volume tiny, so SigNoz ingest cost is negligible. If the threshold is ever lowered, use log_min_duration_sample + log_statement_sample_rate to cap volume (cf. EQU-174 cardinality/cost discipline).

Verify end-to-end

SELECT pg_sleep(3);   -- run against each server; should appear within ~1 min

Confirm it shows up in Query Performance Insight / the KQL query (Phase 1) or the SigNoz logs view (Phase 2).

Optional: correlate logs ↔ traces

• App-side: SQLAlchemyInstrumentor().instrument(enable_commenter=True, ...) in app/core/observability/telemetry.py injects the trace id into the SQL text.
• Or set log_line_prefix to include %Q (PG14+) to stamp the queryid into each log line, joinable to pg_stat_statements.