Skip to content

Testing patterns

Implemented

Source: doc/governance/Testing_Standards.md (275 lines)

Test pyramid

flowchart TB
    classDef u  fill:#e8f5e9,stroke:#2e7d32
    classDef i  fill:#fff3e0,stroke:#e65100
    classDef c  fill:#e3f2fd,stroke:#1565c0
    classDef e  fill:#fce4ec,stroke:#c2185b

    E[E2E<br/>Playwright critical journeys]:::e
    C[Contract<br/>Schemathesis vs OpenAPI]:::c
    I[Integration<br/>real Postgres + Redis + NATS]:::i
    U[Unit<br/>domain + handler<br/>table-driven]:::u

    U --> I --> C --> E
Layer Tooling Trigger
Unit go test table-driven make test
Handler httptest make test
Integration real infra make test-integration
Contract schemathesis CI gate contracts_schemathesis_report.sh
Frontend Vitest + Playwright make verify-web + make frontend-e2e

Acceptance matrix

The PRD has 80+ acceptance tests (AT-001 through AT-083). Key examples:

AT Check
AT-001 Login success + token issuance
AT-002 Invalid login returns 401
AT-003 Non-admin blocked from admin APIs
AT-010 Marketplace capacity reflects online + unassigned only
AT-020 Provisioning fails offline / in-use / insufficient-funds
AT-023 Successful provision creates allocation + usage
AT-030 Release by owner/admin succeeds
AT-031 Release by unauthorized user fails
AT-032 Release unassigns node
AT-040 Billing loop accrues cost over time
AT-041 Low balance warning only once per low-state transition
AT-042 Depleted balance triggers forced release
AT-050 Stripe charge session returns checkout URL
AT-051 Webhook credits balance on valid event
AT-052 Duplicate webhook does not double-credit
AT-053 Webhook signature bypass attempt rejected with 400
AT-061 Storage traversal attempts rejected
AT-070 Rate limit enforced after threshold; 429
AT-071 Rate-limit response headers present
AT-080 Privileged mutations each produce a structured audit entry
AT-081 Audit log entries contain all required fields
AT-082 Failed authorization attempts recorded with result=failure
AT-083 Audit log entries immutable

Full list in Testing_Standards.md §Acceptance Matrix.

Evidence-first verification rules

From the standards:

  • Verification is reported relative to a baseline, not isolated pass/fail.
  • Baseline matches scope of change; targeted checks preferred over ritual full-suite.
  • Every non-trivial behavior change includes one direct proof.
  • A previously-passing scoped check that fails after the change is a regression until disproven.
  • Unexpected failures are evidence about dependencies — record them, don't wave away.

Sample table-driven test

func TestSliceVMPCIAddressPattern(t *testing.T) {
    cases := []struct {
        name  string
        in    string
        ok    bool
    }{
        {"valid", "0000:1a:00.0", true},
        {"valid hex", "0000:5e:00.7", true},
        {"missing domain", "1a:00.0", false},
        {"bad function", "0000:1a:00.8", false},
        {"uppercase", "0000:1A:00.0", true},
    }
    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            got := sliceVMPCIAddressPattern.MatchString(tc.in)
            if got != tc.ok {
                t.Fatalf("match(%q) = %v want %v", tc.in, got, tc.ok)
            }
        })
    }
}

Sample handler test (httptest)

func TestCreateAllocation_RequiresAuth(t *testing.T) {
    h := newTestHandler(t)
    req := httptest.NewRequest(http.MethodPost, "/api/v1/allocations", bytes.NewBufferString(`{}`))
    rr := httptest.NewRecorder()
    h.ServeHTTP(rr, req)
    if rr.Code != http.StatusUnauthorized {
        t.Fatalf("got %d want 401", rr.Code)
    }
    var e errors.ErrorResponse
    _ = json.Unmarshal(rr.Body.Bytes(), &e)
    if e.Code != "token_missing" {
        t.Fatalf("got code %q", e.Code)
    }
    if e.CorrelationID == "" {
        t.Fatal("missing correlation_id")
    }
}

Integration test pattern

//go:build integration

func TestSliceReservation_LocksSlots(t *testing.T) {
    ctx, db, cleanup := integration.Setup(t)
    defer cleanup()

    // seed: one node, 4 slots all available
    integration.SeedSliceNode(t, db, integration.NodeSpec{Slots: 4})

    // place a 2-GPU slice
    s := newOrchestrator(db)
    alloc, err := s.CreateRequested(ctx, in)
    require.NoError(t, err)

    // assert: 2 slots reserved, 2 still available
    var reserved, available int
    require.NoError(t, db.QueryRow(ctx, `
        SELECT COUNT(*) FILTER (WHERE status='reserved'),
               COUNT(*) FILTER (WHERE status='available')
        FROM node_resource_slots WHERE node_id=$1
    `, alloc.NodeID).Scan(&reserved, &available))
    require.Equal(t, 2, reserved)
    require.Equal(t, 2, available)
}

Coverage targets

Layer Target
Domain logic 80%+
HTTP handlers 100% of error paths
Workers All terminal states
Frontend Critical journeys + selector contract

SLO testing

SLO Test class
Authz membership p95 ≤ 20 ms Integration explain-plan + load test
Allocation create → outbox event < 1 s Integration
Webhook → ledger credit < 5 s Integration

Non-functional testing

  • Load/performance tests with SLO thresholds.
  • Chaos / failure tests for retries, DLQ, partial failures.
  • Migration forward/rollback tests.
  • Backup/restore verification.

Security testing

  • SAST / DAST.
  • Dependency + container scans.
  • Secret scanning.
  • IaC policy scanning.

Where to look next