Tenant/Project Authorization Incident Runbook

Trigger

• Spike in tenant/project membership authorization failures.
• Requests rejected for missing project context header.

Required Request Context

• X-Project-ID must be present for project-scoped operations.
• Missing X-Project-ID should be treated as request-context failure (not infrastructure outage).

Immediate Mitigation

1. Confirm whether failures are concentrated to one project or tenant.
2. Validate client/request path is sending X-Project-ID.
3. If widespread, pause risky admin mutations until scope checks are verified.

Diagnosis

1. Start with API error envelope code + correlation_id.
2. Query logs by correlation_id and confirm X-Project-ID presence.
3. Distinguish:
4. missing/invalid X-Project-ID context
5. valid context with denied tenant/project membership
6. Verify membership records and recent role/scope changes for impacted actor.
7. Review control-plane authz counters in Admin Ops:
8. platform_role_admin_denied_total
9. platform_role_mutation_failure_total
10. platform_role_service_unavailable_total to classify expected authorization denial vs backend/schema degradation.

Recovery

1. Restore correct project-context propagation (X-Project-ID) in caller path.
2. Correct tenant/project membership assignment where policy allows.
3. Re-run failing operation and confirm deterministic success/failure behavior.

Post-Incident

• Record affected tenant/project identifiers and correlation trail.
• Capture whether failure class was context-missing vs membership-denied.
• Add follow-up tasks for recurring client-side context propagation gaps.