Skip to main content

Observability & Monitoring

Orange uses OpenTelemetry for collecting application metrics, traces, and logs.

Overview​

The monitoring stack:

  1. Application Instrumentation - FastAPI, SQLAlchemy, HTTPx auto-instrumentation
  2. OpenTelemetry Collector - Collects system, database, and application metrics
  3. Custom Metrics - Heartbeat monitoring for service health

Collected Metrics​

Application Metrics​

Service Health Monitoring:

  • service.status - Health status (1 = healthy, 0 = down)
  • service.last_seen - Last heartbeat timestamp
  • Heartbeat interval: 30 seconds

HTTP Request Metrics (FastAPI auto-instrumentation):

  • http.server.request.duration - Request latency
  • http.server.request.count - Request count by endpoint/status
  • http.server.active_requests - Active concurrent requests

Database Query Metrics (SQLAlchemy instrumentation):

  • Query duration and count
  • Connection pool usage
  • Slow query detection

System Metrics (Host)​

Collected every 30 seconds via OpenTelemetry Collector:

MetricDescription
system.cpu.utilizationCPU usage percentage per core
system.cpu.timeCPU time by state (user, system, idle)
system.cpu.load_average.1m/5m/15mLoad averages
system.memory.usageMemory used (bytes)
system.memory.utilizationMemory usage percentage
system.disk.ioDisk I/O bytes (read/write)
system.disk.operationsDisk operations count
system.network.ioNetwork I/O bytes
system.network.packet.countNetwork packets sent/received
system.network.errorsNetwork transmission errors
system.filesystem.usageDisk space used per mount
system.filesystem.utilizationDisk usage percentage
system.process.countNumber of processes
system.uptimeSystem uptime in seconds

PostgreSQL Metrics​

Collected every 60 seconds:

MetricDescription
postgresql.backendsActive database connections
postgresql.commitsTransaction commits
postgresql.rollbacksTransaction rollbacks
postgresql.db_sizeDatabase size in bytes
postgresql.deadlocksDeadlock count
postgresql.database.locksActive locks by mode
postgresql.sequential_scansSequential table scans

Attributes: db.name, db.user, host.name, deployment.environment

Redis Metrics​

Collected every 30 seconds:

MetricDescription
redis.clients.connectedConnected clients
redis.commands.processedTotal commands processed
redis.memory.usedMemory used (bytes)
redis.keyspace.keysNumber of keys per database

Configuration​

Environment Variables​

# Enable/Disable OpenTelemetry
OTEL_ENABLED=true

# Service Identification
OTEL_SERVICE_NAME=santra-be # API service name
OTEL_WORKER_NAME=santra-worker # Celery worker name
ENVIRONMENT=production # Environment tag

# Heartbeat Interval
HEARTBEAT_TIME=30.0 # seconds

# Hostname Override (optional)
HOST_HOSTNAME=production-server-01

Dashboard​

A pre-configured SigNoz dashboard is available at config/signoz-dashboard.json. This dashboard includes visualizations for:

  • API request rate and latency
  • Database connection pool usage
  • System resource utilization
  • Service health status
  • Error rates by endpoint

Import this file into SigNoz to get started with monitoring.

Traces​

All HTTP requests and database queries are automatically traced via OpenTelemetry instrumentation.

Logs​

Logs are automatically correlated with traces, including trace ID and span ID for request tracing.

PostgreSQL Slow Query Logging (Azure Flexible Server)​

Goal: capture every statement that runs longer than 2 s (log_min_duration_statement = 2000 ms) so we can find slow queries in prod.

Reality check (read first)​

Prod Postgres is Azure Database for PostgreSQL Flexible Server — primary jeff-main-db-restore.* plus a read replica. Consequences:

  • It is managed: there is no local log file on our VM. The collector's filelog/postgresql receiver (path POSTGRESQL_LOG_FILE) only works for the optional --profile postgres container and is dormant in prod — the log volume mount in docker-compose.prod.yml is commented out. Do not rely on it for managed PG.
  • Slow queries leave Azure only through a Diagnostic Setting, not a file.
  • The slow analytical reads run on the replica (≈22 s queries profiled there). log_min_duration_statement and the resulting logs are per-node — you must enable it on both the primary and the replica, and the replica is the one that matters most.
  • This is the per-execution slow-query log (real SQL + duration). It is complementary to pg_stat_statements (aggregate, normalized top-N) — also per-node, queried separately on each server.

Phase 1 — minimal, stays in Azure (do this first)​

  1. Set the threshold on the primary and the replica. Dynamic — no restart.

    az postgres flexible-server parameter set \
    --resource-group <rg> --server-name jeff-main-db-restore \
    --name log_min_duration_statement --value 2000
    # repeat with --server-name <replica-server-name>

    Portal equivalent: Server parameters → log_min_duration_statement → 2000.

    Verify it landed on each node (we have read-only access to the replica):

    SHOW log_min_duration_statement; -- expect "2s"

    On Flexible Server replicas most parameters inherit from the primary. If the replica doesn't reflect the value, set it explicitly on the replica too. Leave log_statement = none — that logs by type, not duration, and would flood logs.

  2. Route logs out — add a Diagnostic Setting on each server exporting the PostgreSQLLogs category to a Log Analytics workspace (skip if one already exists):

    az monitor diagnostic-settings create --name pg-logs-to-law \
    --resource <flexible-server-resource-id> \
    --workspace <log-analytics-workspace-id> \
    --logs '[{"category":"PostgreSQLLogs","enabled":true}]'
  3. Read the slow queries:

    • Query Performance Insight (portal, built on pg_stat_statements) — top queries by duration, zero query-writing.
    • KQL over Log Analytics for the actual >2 s statements:
      AzureDiagnostics
      | where Category == "PostgreSQLLogs" and Message has "duration:"
      | project TimeGenerated, Resource, Message // Message: "duration: 2351.1 ms statement: SELECT ..."
      | order by TimeGenerated desc

Phase 2 — optional, unify into SigNoz​

Only if we want these alongside the rest of our telemetry. Replaces the dead filelog path for managed PG:

  1. Point the Diagnostic Setting at an Event Hub (in addition to / instead of Log Analytics).
  2. Add the contrib collector's azureeventhub receiver to otel-collector-config.yaml and a new logs/postgresql-azure pipeline → the existing SigNoz otlp exporter.

A 2 s threshold keeps volume tiny, so SigNoz ingest cost is negligible. If the threshold is ever lowered, use log_min_duration_sample + log_statement_sample_rate to cap volume (cf. EQU-174 cardinality/cost discipline).

Verify end-to-end​

SELECT pg_sleep(3); -- run against each server; should appear within ~1 min

Confirm it shows up in Query Performance Insight / the KQL query (Phase 1) or the SigNoz logs view (Phase 2).

Optional: correlate logs ↔ traces​

  • App-side: SQLAlchemyInstrumentor().instrument(enable_commenter=True, ...) in app/core/observability/telemetry.py injects the trace id into the SQL text.
  • Or set log_line_prefix to include %Q (PG14+) to stamp the queryid into each log line, joinable to pg_stat_statements.