Observability Standards

Purpose: - Define mandatory telemetry standards before observability implementation starts. - Ensure consistent, secure, low-noise operational signals across services.

1. Required Telemetry Outputs

Every runtime component (cmd/api, all workers) must emit: - structured logs - OpenTelemetry traces - Prometheus-compatible metrics

All signals must include correlation context where applicable.

2. Required Fields

Required log fields: - timestamp - level - service - message - correlation_id

Required span attributes: - service.name - deployment.environment - correlation_id

Required metric dimensions: - Keep labels bounded and low-cardinality. - Allowed: service, route_template, status_code_class, operation. - Forbidden: raw UUIDs, session IDs, token IDs, email addresses.

3. Naming and Units

Rules: - Counters use _total. - Durations in seconds. - Monetary values in minor units (integers). - Retry/failure metrics must be explicit (*_failures_total, *_retries_total).

4. Redaction and Data Safety

Never emit into logs/spans/metrics labels: - access/refresh/id tokens - passwords and password hashes - SSH private key material - payment card data - full webhook payloads containing PII

Reference: - doc/governance/Coding_Standards.md sanitize/redaction rules are mandatory.

5. Error and Correlation Requirements

Requirements: - Every error response must include canonical ErrorResponse with correlation_id. - X-Correlation-ID must be returned on every HTTP response. - Async handlers must carry correlation IDs from event envelope to logs/spans.

6. Dashboards and Alerts (Minimum v1)

Must-have dashboards: - API SLO dashboard (latency/error/availability) - Billing + payment processing dashboard - Provisioning workflow dashboard - Queue/relay health dashboard

Must-have alerts: - API error-rate burn - high p95 latency - outbox relay publish failures/backlog growth - webhook processing failures - worker heartbeat loss

7. CI and Review Gates

Before merge for observability-impacting changes: - unit tests for new metrics/log fields (where practical) - integration smoke for telemetry endpoints where available - documentation updated for added/changed metrics - no new high-cardinality labels introduced

Review checklist: - [ ] Correlation preserved end-to-end - [ ] PII-safe telemetry - [ ] Dashboard/alert impact documented - [ ] Contract docs updated before code