Skip to content

GPU Slice End-to-End Readiness Decisions v1

Purpose

Close the remaining architecture and product decisions before continuing end-to-end GPU slice validation.

This document is the bridge between:

  1. Allocation_Capacity_Shapes_and_GPU_Slices_v1.md;
  2. Hierarchical_Placement_and_Node_Scheduler_v1.md;
  3. GPU_Slice_Implementation_Checklist_v1.md;
  4. Product_UI_System_Redesign_v1.md;
  5. UX_Redesign_Implementation_Plan.md;
  6. GPU_Slice_Infra_Enablement_Proposal_v1.md.

It does not replace those documents. It records the decisions that unblock the next implementation and test pass.

Finalized Decisions

1. Fabric Model

The v1 GPU VM slice product uses per-slot VF-backed fabric.

Each schedulable whole-GPU slice slot must have:

  1. one physical GPU PCI function;
  2. one raw NVMe device approved for tenant slice ownership;
  3. one unique fabric VF or equivalent isolated fabric attachment;
  4. one management MAC/private-IP reservation;
  5. SKU/profile-owned CPU and memory;
  6. cleanup/wipe policy metadata.

The scheduler must not treat a shared host BF/IB parent device as schedulable concurrency. Parent fabric metadata is operator context only. A slot is schedulable only when capacity_metadata.fabric_claim_mode=per_slot_vf and capacity_metadata.fabric_vf_pci_address is non-empty.

This preserves max performance and gives a clean extension point for future fractional or shared GPU products. Fractional/shared GPU support remains a future capacity shape and must not be hidden under gpu_slice.

2. Same-Node Placement

Multi-GPU slice allocations must not span nodes.

If a user asks for 4 GPUs and the platform has two nodes with two schedulable slots each, the request is unavailable. It must not reserve capacity across both nodes for one VM.

The user-facing message should distinguish:

  1. no capacity in region;
  2. capacity exists but no single node can satisfy the requested count;
  3. capacity exists but topology/fabric/storage readiness blocks scheduling.

The API can initially return sku_unavailable, but the product should evolve toward structured unavailable reasons so the catalog and wizard can explain the blocker without direct DB inspection.

3. Controller Scheduler and Node Scheduler Boundary

GPUaaS uses a hierarchical placement model:

Region scheduler -> Cluster scheduler -> Node scheduler -> Runtime executor

For the current code path, the controller scheduler already performs durable reservation using node_resource_slots and allocation_resource_claims. That is acceptable for the first end-to-end test, but it is not the final boundary.

The target model is:

  1. controller scheduler chooses region, pool, and candidate node;
  2. node scheduler validates host-local state and returns a concrete bundle plan;
  3. controller records the returned plan as durable claims;
  4. runtime executor creates or releases the VM from the accepted plan.

The node scheduler may live inside node-agent initially. It must be a bounded typed-task path, not an SSH shell or unstructured script runner.

4. Node Scheduler Lease Model

The node scheduler must take a host-local lease before reporting that a bundle is safe to use.

Minimum lease requirements:

  1. one lease per slot or per bundle;
  2. lease file under /var/lib/gpuaas/node-scheduler/leases;
  3. lease contains allocation ID, task ID, slot IDs, device identities, and expiry;
  4. stale leases are reconciled by node-agent before new placement;
  5. controller claims remain the durable source of truth after reservation.

This protects against same-node races where the controller has stale inventory or two node tasks are in flight concurrently.

5. Node Mode and Baremetal Preservation

Nodes can be tagged as slice-capable, baremetal-capable, or both-capable, but placement is mutually exclusive at runtime.

Initial states:

  1. baremetal_only: no slice profile tag or no approved slots;
  2. slice_candidate: profile tag exists but readiness or slot approval is incomplete;
  3. both_capable: node has profile tag, readiness evidence, and approved slots, with no active claims;
  4. slice_active: at least one slice claim is reserved/provisioning/active;
  5. baremetal_active: whole-node claim is active or reserved.

Scheduler policy should prefer already-sliced nodes for new slice requests so clean nodes remain available for baremetal. Freeing a sliced node for baremetal is a drain/release/reconcile operation, not transparent live migration.

6. Host Bootstrap and Cloud-Init

Standard MAAS commissioning remains the default.

GPUaaS-specific firmware/profile work runs through deploy-time cloud-init or an infra wrapper, selected by MAAS tags:

  1. gpuaas-profile-slice-vm;
  2. gpuaas-profile-baremetal;
  3. hardware tags such as gpu-nvidia-h200;
  4. server tags such as server-dell-xe9680l;
  5. fabric tags such as fabric-bf3.

The helper checks profile readiness, applies one-time BIOS/RACADM drift only when approved, performs at most one controlled reboot for that profile pass, and records evidence under /var/lib/gpuaas/firmware-profile/.

Runtime bootstrap remains separate from firmware readiness. Libvirt, OVS, VFIO, image cache, hugepages, host networking, and node-agent installation are deployed-host bootstrap responsibilities.

7. Slice Images

End-to-end testing needs at least two image tracks:

  1. base Ubuntu slice image;
  2. CUDA/NVIDIA developer image.

The slice runtime must resolve image metadata from SKU/profile or approved slot metadata. The first test may use a manually registered path, but productized launch must use an image catalog with compatibility metadata.

Images must have:

  1. immutable digest;
  2. compatibility shape (gpu_slice or baremetal);
  3. accelerator compatibility;
  4. default username;
  5. readiness contract;
  6. rollout state such as active/canary/deprecated.

8. Task Visibility

Allocation timeline pagination is sufficient for the immediate slice test, but the product direction is a first-class task read model.

The task model must cover:

  1. allocation provisioning and release;
  2. GPU slice slot reservation;
  3. image clone/cache;
  4. cloud-init and boot;
  5. readiness probes;
  6. MAAS onboarding and decommission;
  7. image build/import/verify.

The UI should initially load a small latest-event page and support loading older events. Backend pagination and sorting are required; the UI must not load unbounded task histories for long-lived allocations.

9. Launch UX

The product should expose one H200 launch flow with branching inside the wizard:

  1. baremetal SKU path: fixed GPU count, whole-node semantics;
  2. GPU slice path: selectable allowed counts, same-node availability, image selection, SSH/access, storage, network, and review.

Region is a top-bar and wizard context even while only one region is enabled. Initial display value is US Buffalo / us-buffalo-1 or the currently mapped platform region label for the MAAS site. Future regions can be disabled rather than hidden.

10. Test Node Scope

Current slice validation nodes:

Node MAAS IP Intended use
j22u15 10.177.36.197 Primary slice test host
j22u11 10.177.36.100 Secondary slice test host
j27u15 10.177.36.198 Baremetal or temporary slice test if retagged

j22u05 is being returned and should not be used for new slice validation.

Immediate Engineering Order

Complete these before broad UI redesign work resumes:

  1. make GPUaaS inventory reflect the MAAS slice tags for the active test nodes;
  2. run host readiness/bootstrap on j22u15;
  3. run topology discovery and confirm per-slot fabric VFs exist;
  4. approve only slots with raw NVMe slice ownership and unique per-slot fabric VF metadata;
  5. verify the base slice image path/digest exists on the node;
  6. launch one 1-GPU slice allocation;
  7. verify VM boot, cloud-init, SSH/terminal, and nvidia-smi;
  8. release allocation and verify wipe/cleanup evidence before slot reuse;
  9. repeat on j22u11 only after the primary path works.

Explicit Non-Goals For This Pass

  1. fractional GPU scheduling;
  2. live migration or automatic evacuation;
  3. multi-node slice VMs;
  4. shared host-owned fabric as a concurrency model;
  5. public ingress for slice VMs beyond the existing private management path;
  6. full product UI rewrite before the slice runtime path is proven.