Skip to content

Allocation Capacity Shapes and GPU Slices v1

Purpose

Define how GPUaaS should evolve from whole-node allocations to multiple capacity shapes on the same physical node:

  1. baremetal: one allocation claims an entire physical node.
  2. gpu_slice: one allocation claims one or more host-local resource slots on a physical node.

This document is a design proposal. It does not change the current API or schema until the OpenAPI, database schema, placement service, billing, and node-agent contracts are updated.

Companion architecture docs:

  1. doc/architecture/Slice_Networking_Architecture_v1.md: slice VM management network, BF3/OVS, IPAM, public ingress, console access, and networking drift.

Context

Current allocation placement is node-exclusive:

  1. allocation create selects a nodes row,
  2. allocations.node_id is the placement anchor,
  3. node availability is effectively binary,
  4. one active/releasing allocation blocks the whole node.

That model is correct for full bare-metal leasing, but it cannot represent an H200 host split into independent VM-backed slices where each slice owns a GPU, local NVMe device, IB/VF device, network identity, and NUMA placement.

The prototype in ~/Downloads/GPUaaS demonstrates the intended H200 product shape: an 8-GPU node can expose eight fixed slice bundles. Each bundle maps:

  1. one GPU PCI device,
  2. one fabric/network device or VF,
  3. one raw NVMe device or volume,
  4. one NUMA node,
  5. one VM network identity.

The prototype is useful for intent, but GPUaaS must model this centrally rather than as a node-local Flask/libvirt control plane. The platform model must also support non-H200 nodes, including AMD GPU hosts that use RoCE rather than InfiniBand.

Prototype-derived implementation constraints that must not be lost:

  1. slice inventory is a host-approved topology map, not a blind GPU count;
  2. node bootstrap has hard prerequisites such as IOMMU, VFIO, SR-IOV, OVS, libvirt, cloud-init tooling, and sometimes a reboot;
  3. VM disks may be destructive raw NVMe assignments, so disk safety and wipe verification are part of scheduling correctness;
  4. DHCP/NAT/overlay/public ingress state must be reconciled with control-plane allocation state;
  5. any VM console access must be authenticated and audited, not exposed as raw VNC from the node.

Terminology

Physical Node

A GPU server enrolled in GPUaaS and managed by node-agent. A node has lifecycle, identity, health, region, SKU class, topology, and host-level prerequisites.

Capacity Shape

The type of allocation realization:

  1. baremetal: exclusive node lease.
  2. gpu_slice: VM or isolated runtime lease on a subset of node-local resources.

Capacity shape is a platform primitive. It is not an app-specific concept. Future fractional or shared-GPU products should be added as explicit capacity shapes, for example gpu_partition or gpu_shared, only after the underlying hardware/runtime mechanism is selected and tested. Do not overload gpu_slice to mean arbitrary time sharing, MPS, MIG, vGPU, or software multiplexing.

Resource Slot

A schedulable bundle of host-local resources that can be safely assigned to one allocation. A slot is a platform-approved schedulable unit, not necessarily a whole physical GPU forever. For first H200 slice mode, each slot is expected to map to one full GPU device, but the abstraction must also leave room for future child slots that represent MIG/vGPU/fractional resources under a parent physical GPU. For H200 slice mode, a slot is not just a GPU count. It is a bundle:

  1. GPU device identity and PCI address,
  2. optional MIG profile or full-GPU flag,
  3. NVMe device or volume identity,
  4. network fabric and VF/device identity,
  5. NUMA locality,
  6. VM/runtime metadata such as MAC/IP pool reference,
  7. CPU and memory reservation metadata,
  8. health and lifecycle state.

Fractional or shared-GPU inventory, if introduced later, should still be modeled as approved slots or child slots with explicit parentage and capacity metadata. The scheduler must never infer fractional availability from raw GPU memory or device count alone.

Resource Claim

The durable binding between an allocation and one or more resource slots.

Allocation

The customer-facing lease and billing anchor. An allocation is no longer synonymous with "one node". It is a request for capacity that is realized by a node claim or slot claims.

Core Decision

Do not make allocations.node_id the only source of placement truth going forward.

Instead, model placement as:

  1. allocations: customer lease, lifecycle, billing anchor, request metadata.
  2. allocation_placements: optional resolved placement summary/read model for an allocation.
  3. allocation_resource_claims: one row per claimed physical node or resource slot.
  4. node_resource_slots: host-local schedulable inventory discovered or approved per node.

allocations.node_id can remain as a compatibility/read-model field during migration, but new scheduling logic should depend on placement/claim tables.

allocation_resource_claims is the required relational source of truth. allocation_placements is not required as a separate table for v1 if the same read-model fields can live safely on allocations, for example capacity_shape, placement_status, and a compatibility node_id. Treat the placement summary as an implementation choice:

  1. prefer columns on allocations if the summary remains 1:1 and simple;
  2. use a separate allocation_placements table only if the summary grows enough to justify its own ownership, lifecycle, or refresh semantics;
  3. never make scheduling correctness depend on the placement summary instead of claims and slots.

Placement Rules

Baremetal

A baremetal allocation:

  1. requires an active node matching region, SKU, tenant boundary, and health,
  2. claims the whole node,
  3. blocks all resource slots on that node,
  4. is blocked if any slot on that node is already reserved/provisioning/active,
  5. may use MAAS full reimage isolation on release when policy requires.

GPU Slice

A gpu_slice allocation:

  1. requires one or more compatible slots,
  2. must fit on a single physical node for the first implementation,
  3. cannot span node boundaries,
  4. must lock selected slots atomically,
  5. must fail with sku_unavailable when no same-node compatible slot set exists,
  6. should prefer topology-compatible placement such as same NUMA node for multi-GPU slice requests when possible.

Fabric compatibility is part of slot compatibility:

  1. NVIDIA H200/IB slice products should require slots with the expected InfiniBand/HCA capability when the SKU or app profile requires it.
  2. AMD/RoCE slice products should require RoCE-capable NIC/VF resources and should validate the RoCE path separately from IB checks.
  3. Apps that do not require low-latency fabric can use products whose fabric requirement is ethernet or none, but they must not be scheduled onto a SKU that promises IB/RoCE semantics unless the slot validation passed.

Mixed Node State

A node can be:

  1. fully available,
  2. partially allocated,
  3. fully allocated by slices,
  4. exclusively allocated by baremetal,
  5. draining,
  6. cleanup blocked,
  7. unavailable.

Node occupancy must become a derived aggregate of node status plus slot and claim states, not a single active-allocation predicate.

Mixed Node Transitions

Node mode is derived from active claims plus admin intent. The scheduler should apply these transition rules:

From To Rule
fully available slice use allowed when node supports slice mode and has approved slots
fully available baremetal use allowed when no slots or claims are reserved/active
partially allocated by slices baremetal use blocked until all slice claims are released and cleanup passes
partially allocated by slices draining allowed; no new slice claims are scheduled, existing allocations continue until user/admin release
fully allocated by slices draining allowed; node becomes available only after all slots release and cleanup succeeds
any active allocation unavailable allowed for health failure; billing/allocation policy decides whether to fail, migrate later, or keep assigned
slot cleanup failed cleanup blocked block only affected slot unless health evidence indicates node-wide corruption
cleanup blocked fully/partially available allowed after admin repair or successful probe clears blocked slots
baremetal active slice use blocked until baremetal allocation releases and node cleanup/reimage completes
host share/storage mounted slice use blocked until admin/infra drain unmounts or remaps the devices and approves destructive slice ownership
slice active baremetal/share use blocked until every slice claim releases, cleanup proof passes, and infra remounts or recreates the baremetal storage layout
slice-capable baremetal-capable mode is derived from claims plus storage ownership; admins may drain to force the next allocation shape

Admin drain should be non-disruptive by default. It prevents new placement but does not stop active customer allocations unless a separate retire/force-release policy is invoked.

Slice capability must be a promotion decision, not a hardware inference. The platform should treat the MAAS slice-host tag or equivalent resource-pool membership as admin intent, deploy-time firmware profile evidence as readiness evidence, and approved node_resource_slots as the scheduling source of truth. Passing the readiness checks without the slice-host tag leaves the node baremetal-only for GPUaaS. Having the tag without a ready firmware profile leaves the node a slice candidate, but not deployable. Having both without approved slots leaves the node infra-ready, but not schedulable for gpu_slice.

MAAS and platform inventory tags should be composable. Use firmware profile intent tags such as gpuaas-profile-slice-vm and gpuaas-profile-baremetal to select the deploy-time cloud-init profile, plus independent hardware tags such as gpu-nvidia-h200, server-dell-xe9680l, and fabric-bf3. SKU derivation should combine hardware tags, selected firmware profile, readiness state, and approved slots instead of overloading one SKU-specific tag. This keeps the same path usable for H100, B-series, AMD, or later accelerator hosts.

The slice VM and baremetal firmware profiles must be maintained independently because their RACADM settings may diverge. A machine should not carry both GPUaaS profile tags in automated onboarding unless infra is explicitly testing a lab override; otherwise the cloud-init helper should block the node to avoid applying conflicting firmware policy.

The user-facing catalog and admin views should distinguish capability from current occupancy:

  1. baremetal_only: no slice VM profile intent is recorded for the node;
  2. slice_candidate: slice VM profile intent exists but readiness or slot approval is incomplete;
  3. both_capable: baremetal and slice products can target the node, subject to mutually exclusive placement;
  4. baremetal_active: whole-node placement currently blocks slice placement;
  5. slice_active: one or more slice claims currently block baremetal placement.

Nodes with convertible NVMe pools require an explicit storage-ownership transition even if the compute claims are clear. A mounted host share partition is valid in baremetal/share mode, but it is not valid evidence for a schedulable slice slot. Slice approval must happen after the node is drained, the share storage is unmounted or remapped, and the destructive wipe policy is accepted.

Proposed Data Model

This is directional and must be translated into db_schema_v1.sql only after API contracts are updated.

node_resource_slots

Fields:

  1. id uuid primary key
  2. node_id uuid not null references nodes(id)
  3. slot_index integer not null
  4. shape text not null such as gpu_slice
  5. status text not null such as available, reserved, provisioning, active, releasing, cleanup, failed, disabled
  6. gpu_model text
  7. gpu_count integer not null default 1
  8. gpu_pci_addr text
  9. gpu_uuid text
  10. mig_profile text null
  11. nvme_device text null
  12. network_fabric text null such as infiniband, roce, ethernet, or none
  13. network_device text null
  14. fabric_pci_addr text null
  15. fabric_device_id text null
  16. fabric_metadata jsonb not null default '{}'
  17. numa_node integer null
  18. vcpu_count integer null
  19. memory_mib integer null
  20. mac_address text null
  21. private_ip inet null
  22. capacity_metadata jsonb not null default '{}'
  23. health_metadata jsonb not null default '{}'
  24. uniqueness on (node_id, slot_index)

Fractional/shared-GPU support should extend this model without changing the claim invariant. The schema and admin slot contract may carry reserved fields before the scheduler uses them:

  1. parent_slot_id uuid null references node_resource_slots(id) or equivalent parent resource identity for child slots under a physical GPU;
  2. sharing_model text such as exclusive_device, mig_partition, mdev_vgpu, time_sliced, mps, or software_shared;
  3. profile_name text for the vendor/runtime profile, for example a MIG or vGPU profile;
  4. quantitative capacity such as gpu_memory_mib, compute_milli, or max_claims;
  5. explicit compatibility and reset metadata in capacity_metadata.

For slice-capable slots, capacity_metadata.fabric_claim_mode is the fabric approval gate. The v1 GPU VM slice product is per-slot VF backed: schedulable whole-GPU slice slots must set fabric_claim_mode=per_slot_vf and include a non-empty fabric_vf_pci_address for the isolated VF or equivalent attachment. fabric_device may carry the parent BF/IB device for operator context, but duplicate parent fabric devices are not sufficient proof of safe concurrency.

Missing fabric_claim_mode, exclusive_device, and shared_host are not schedulable for GPU VM slices. Keep those slots disabled or cleanup-blocked until cloud-init/approval automation records the per-slot VF identity. Do not set per_slot_vf as a workaround for bad discovery; if the candidate map reused the same BF/IB device because discovery was incomplete, the node-agent/node-scheduler must validate and publish the VF-backed slot model before the marketplace can advertise concurrent slices.

These fields are reserved for model extensibility only. The first slice implementation must keep capacity_shape=gpu_slice as exclusive whole-GPU VM slots unless a separate fractional/shared-GPU phase changes the public capacity shape enum, scheduler, node-agent, and billing rules. Admin visibility of these fields does not make fractional/shared GPU products schedulable.

The v1 slice scheduler must explicitly ignore any slot with parent_slot_id set, sharing_model other than exclusive_device, max_claims greater than one, or compute_milli below one whole accelerator unit. This prevents early topology experiments from being sold as whole-GPU slices by mistake.

GPU_Slice_End_to_End_Readiness_Decisions_v1.md is the current decision record for the final v1 slice model: same-node placement, per-slot VF-backed fabric, node-scheduler validation, MAAS tag intent, and launch/task UX boundaries.

Do not hardcode the slot schema around InfiniBand. H200 SXM systems may use InfiniBand/HCA passthrough. AMD GPU systems we have used are expected to use RoCE over Ethernet NICs. Both need first-class fabric metadata because placement, network validation, VM attachment, and app compatibility differ.

The latest tested prototype uses fixed per-slot VM defaults of 24 vCPUs and 64 GiB memory for a one-GPU slice. Earlier scripts used or discussed 12 vCPUs, so these values must not be buried inside node-agent scripts. They should come from SKU/profile metadata or approved slot capacity so scheduling, billing, app validation, and benchmark expectations all see the same resource envelope.

Slot discovery must produce a candidate map, not an automatically trusted map. The prototype can derive GPU/NVMe ordering from host commands, but IB devices, BF3 VFs, skipped NVMe device numbers, MAC/IP assignment, and NUMA-safe grouping need admin approval or a site-specific topology profile before the slots become available.

Discovery requirements:

  1. enumerate GPUs from PCI/sysfs rather than hardcoded device lists;
  2. enumerate NVMe devices by topology and admin-approved disk identity, not by hardcoded size filters such as 3.5T|3.8T;
  3. enumerate IB/RDMA devices from /sys/class/infiniband and PCI metadata, including Mellanox vendor ID 15b3 where applicable;
  4. include IB/RDMA port state, link layer, GUID, PCI address, and NUMA metadata in the candidate bundle;
  5. surface the candidate map for admin approval before slots become schedulable.

The first platform API for this flow is POST /api/v1/admin/nodes/{node_id}/slice-topology/discovery, which queues a signed slice.topology_discover node task, and GET /api/v1/admin/nodes/{node_id}/slice-topology/discovery, which reads the latest advisory task output. This deliberately uses node_tasks.output as the candidate-report store in v1 so discovered topology remains separate from approved node_resource_slots; promotion into schedulable slots is still an explicit admin action through the resource-slots API.

allocation_placements

Fields:

  1. allocation_id uuid primary key references allocations(id)
  2. capacity_shape text not null
  3. node_id uuid not null references nodes(id)
  4. placement_status text not null
  5. placement_metadata jsonb not null default '{}'
  6. created_at, updated_at

This table gives fast read access for API/UI without overloading allocations.node_id.

allocation_resource_claims

Fields:

  1. id uuid primary key
  2. allocation_id uuid not null references allocations(id)
  3. node_id uuid not null references nodes(id)
  4. slot_id uuid null references node_resource_slots(id)
  5. claim_kind text not null such as node_exclusive or slot
  6. status text not null such as reserved, provisioning, active, releasing, released, failed
  7. resource_snapshot jsonb not null
  8. created_at, released_at

For baremetal, create one node_exclusive claim with slot_id = null. For slices, create one slot claim per selected slot.

Scheduling Transaction

Placement and allocation creation must remain atomic.

Longer-term placement responsibility is defined in doc/architecture/Hierarchical_Placement_and_Node_Scheduler_v1.md. The current control-plane slot-selection flow is a bootstrap implementation. The intended direction is region and cluster placement in the control plane, with node-agent-owned node scheduling for host-local GPU, fabric, disk, CPU, memory, and network bundle selection.

For gpu_slice:

  1. start transaction,
  2. select candidate active nodes by region, tenant boundary, SKU/product class, shape support, and not node-exclusive claimed,
  3. select compatible slots on each candidate node with FOR UPDATE SKIP LOCKED,
  4. require all requested slots to come from the same node,
  5. insert allocation,
  6. insert placement row,
  7. insert claim rows,
  8. mark selected slots reserved,
  9. write outbox event,
  10. commit.

For baremetal:

  1. lock candidate node,
  2. ensure no active/reserved slot claims exist,
  3. insert allocation,
  4. insert node-exclusive claim,
  5. mark placement,
  6. write outbox event,
  7. commit.

The current NOT EXISTS active allocation on node predicate should be replaced with claim-aware predicates.

Fragmentation Strategy

Fragmentation avoidance is a v1 scheduling requirement, not a later optimization. With eight-slot nodes, greedy small allocations can quickly make a future 4-GPU request impossible even when aggregate free GPUs exist.

Phase 3 should ship with a deterministic topology-aware best-fit policy:

  1. group candidate slots by node and topology group, such as NUMA domain or approved slot group;
  2. filter groups that satisfy the requested GPU count, fabric, storage, and health constraints;
  3. prefer the candidate that leaves the smallest usable remainder while still respecting allowed future shapes;
  4. pack from stable slot ordering inside each topology group;
  5. avoid placing 1-GPU requests into a clean 4-GPU group when another fragmented group can satisfy the request;
  6. expose a scheduling reason when aggregate capacity exists but topology-safe capacity does not.

For H200 v1 with allowed counts [1, 2, 4], the scheduler should understand approved groups such as two 4-slot NUMA groups when topology discovery supports that. A 4-GPU request should reserve a whole 4-slot group. A 2-GPU request should prefer a group that is already partially used or a paired subgroup that does not strand a 4-GPU placement.

Optional later controls:

  1. reservation/draining mode for preserving large shapes during high demand;
  2. per-SKU fragmentation budgets;
  3. admin visibility showing why free slots cannot satisfy a request;
  4. defragmentation through natural release only. Live migration is not a v1 defragmentation mechanism.

This same packing policy should preserve baremetal optionality. Prefer placing new slice requests on nodes that are already in slice mode before converting a clean baremetal-capable node. If a baremetal request needs a node currently occupied by slices, v1 should support drain plus natural release, and later a user-visible stop-and-recreate evacuation flow. Transparent live migration is not expected for GPU/NVMe/IB passthrough VMs.

Implementation note: keep concurrency control in PostgreSQL, but do not assume the whole best-fit decision must be one SQL statement. The likely v1 shape is:

  1. query candidate nodes and compatible free slots with enough metadata to score topology groups;
  2. score fragmentation/topology candidates in Go using deterministic ordering;
  3. start or continue a short transaction;
  4. lock the selected slot rows with FOR UPDATE SKIP LOCKED;
  5. revalidate the selected group still satisfies the request;
  6. commit claims, slot state changes, allocation state, and outbox atomically.

If the selected group cannot be locked because another scheduler won the race, retry with the next scored candidate. Prototype this query early because it is a core scheduler risk: SQL should narrow the candidate set, while Go may own the fragmentation scoring if the topology rules become too complex for maintainable SQL.

Marketplace Capacity Accounting

Catalog capacity must not treat slice slots as additional bare-metal nodes. A physical H200 host can be sold either as a full bare-metal node or as approved GPU VM slots, but those are competing views of the same underlying GPUs.

NodeSummary.slot_summary.by_sku[] reports schedulable slot capacity using the SKU stored in slot metadata. Marketplace and launch-wizard code should use this per-SKU breakdown for gpu_slice products and use node.sku plus in_use only for full bare-metal products. A node with any active or pending slice claim is not bare-metal free until every slice claim has released and cleanup proof has completed.

For j22u05, raw slot inventory can show seven available H200 slice slots while one slice is active. That number is not additive with H200 bare-metal capacity. The launchable count may be lower than raw available slots when fabric, NVMe, or BF ownership rules exclude otherwise-free slots. Until the BF/fabric model is settled, the UI should prefer an explicit "capacity blocked by topology/fabric policy" reason over silently advertising those slots as generally launchable.

Node-Agent Runtime Implications

Node-agent remains scoped to a physical node, but node tasks need placement details:

  1. allocation_id
  2. capacity_shape
  3. claim_ids
  4. slot_ids
  5. device bundle snapshot
  6. requested OS/image/runtime profile
  7. isolation/release policy

For gpu_slice, the node-agent should execute typed tasks such as:

  1. check host prerequisites such as IOMMU, VFIO, SR-IOV, OVS, libvirt, UEFI firmware, cloud-init tooling, RDMA tools, and reboot-required status;
  2. discover and report candidate topology bundles;
  3. prepare VM disk or clone image;
  4. create cloud-init;
  5. attach GPU/fabric/NVMe/network devices;
  6. start/stop/delete VM;
  7. report VM IP/access endpoint;
  8. reconcile libvirt domain, DHCP lease, device binding, and disk state;
  9. wipe/reclaim slot resources;
  10. verify VM service readiness before reporting active;
  11. verify slot health after release.

Do not turn node-agent into a generic remote shell. The VM/slice lifecycle must be implemented as bounded typed tasks with signed inputs and structured outputs.

Slice Guest Image Profiles

The first implementation can bootstrap drivers into an Ubuntu cloud image, but that path is slow and produces a confusing first-login experience when drivers are still installing. Production GPU VM launch should expose image/runtime profiles in the same launch wizard as SSH, storage, and network choices.

Minimum profiles:

  1. ubuntu-24.04-base: minimal Ubuntu image with GPU/RDMA driver bootstrap and health checks. Use when users want to bring their own CUDA/toolchain stack.
  2. ubuntu-24.04-cuda-dev: prebuilt CUDA-enabled developer image with NVIDIA utilities, CUDA toolkit/runtime, RDMA userland, headers, and common build tools. Use as the default for GPU VM slices.

Image selection is part of the allocation request model, not a host-local special case. The scheduler still selects slots by capacity and topology; the node-agent consumes the selected image/runtime profile while preparing the VM disk. Cached or prebuilt images must be invalidated by image version/digest so CI and deploy acceleration cannot serve stale guest artifacts.

Hypervisor Decision For v1

Use a bounded libvirt/QEMU implementation for the first VM-backed slice runtime, controlled by node-agent typed tasks. The node-agent may call libvirt or a small local slice manager, but the control-plane contract should remain task-based so we can replace the local implementation without changing API semantics.

Networking details are owned by doc/architecture/Slice_Networking_Architecture_v1.md.

Initial assumptions:

  1. GPU assignment uses VFIO passthrough of full GPU PCI devices for H200 slices.
  2. Management/public network assignment uses BF3 virtual functions attached through OVS, with one vNIC per VM and optional QoS throttling per vNIC.
  3. Fabric assignment uses the validated IB device exposed in the slot bundle. The IB card does not require an IP address for RDMA. IPoIB can be supported later when a workload requires it, but it is not required for v1.
  4. NVMe assignment uses a raw device, partition, volume, or host-backed disk declared in the slot bundle.
  5. Public ingress, when enabled, is not carried over IB. It is NAT or firewall mapping from public IP to the VM management vNIC, or a controlled overlay such as Tailscale/Funnel while firewall control is being resolved.
  6. MIG, SR-IOV variants, and vendor-specific partitioning are future extensions represented in slot metadata rather than separate allocation concepts.

GPU and fabric devices should normally remain bound to vfio-pci while a node is operating in slice mode. Reattaching to the host GPU driver on every slice release adds latency and device churn. Reattach devices only when a node is drained, transitioned back to baremetal use, or repaired by an admin workflow.

Known constraint: PCI passthrough pins the VM to a physical node and specific devices. Live migration is not supported for v1 and should not be promised by the API or UI. Recovery from node failure is a reprovision/recreate workflow, not a live migration workflow.

Host Bootstrap And Prerequisites

The slice-capable node bootstrap should be explicit and observable. Prototype setup requires packages and host configuration roughly equivalent to:

  1. qemu-kvm, libvirt, virt-install, bridge tooling, OVS;
  2. cloud-image-utils/genisoimage for cloud-init seed images;
  3. qemu-img image conversion support;
  4. RDMA/InfiniBand diagnostics and runtime packages;
  5. kernel arguments such as intel_iommu=on iommu=pt for Intel hosts;
  6. vfio-pci loaded at boot;
  7. host IP forwarding and controlled NAT when private management networking is used.
  8. optional hugepage reservation for tuned slice VM profiles that use QEMU hugepage-backed memory.
  9. optional persistent driverctl overrides for approved GPU and IB PCI devices when the node is dedicated to slice mode.

The first enablement of IOMMU/VFIO may require a host reboot. Node-agent bootstrap should report this as a lifecycle state such as reboot_required rather than silently continuing with partial capability. After reboot, bootstrap must be idempotent and able to resume.

Latest prototype tuning from GPUaaS_0416 adds host-side performance settings that should be treated as an infra-approved slice-node profile rather than blind application logic:

  1. default_hugepagesz=1G hugepagesz=1G hugepages=512 plus --memorybacking=hugepages=yes for the tested VM shape.
  2. persistent VFIO binding for both H200 GPUs and ConnectX-7/IB devices with driverctl, after stopping host NVIDIA/Fabric Manager/RDMA services.
  3. host network sysctls for larger TCP/RDMA buffers and backlog.
  4. guest cloud-init tuning for NVIDIA server drivers, RDMA packages, larger buffers, IPoIB queue sizes, and unlimited memlock.
  5. UEFI boot with Secure Boot disabled to avoid guest driver/MOK prompts.

Do not copy the prototype's storage behavior directly. Its host setup unmounts /dev/nvme*p1 share partitions and then treats the parent NVMe devices as raw slice disks. In GPUaaS production flow, this can be a valid node-mode transition only when infra has explicitly drained baremetal/share use, unmounted or remapped the devices, assigned the disk to tenant-slice use, and approved destructive wipe/reimage. Blind unmounting from node-agent lifecycle code is not allowed.

Slice Drift Reconciliation

The control plane cannot assume that VM state equals allocation state. The node-agent should periodically or on-demand reconcile:

  1. libvirt domain existence and runtime state;
  2. expected PCI devices detached/attached to the right VM;
  3. raw disk ownership, mount state, and wipe status;
  4. DHCP/IP lease for the expected MAC address;
  5. OVS port or VF attachment state;
  6. public NAT, firewall, or overlay exposure mappings;
  7. guest health and optional agent/SSH reachability.

Drift should surface in admin node details and block reuse only for the affected slot unless the evidence points to node-wide corruption.

VM Readiness Gate

The prototype assumes a slice VM is ready after a fixed wait. GPUaaS should report a slice allocation as active only after node-agent verifies the management endpoint is usable.

Initial readiness gate:

  1. wait for the expected DHCP/IPAM lease or configured static private IP;
  2. poll TCP :22 or the image/profile-specific management port;
  3. optionally verify guest-agent or cloud-init completion when the image supports it;
  4. fail the provisioning task with structured output if readiness times out.

The default timeout should be configurable by policy or SKU/profile. A practical starting point is 120 seconds for cloud-image slices.

SKU and Pricing Direction

SKU should become a sellable product/profile rather than only a node class. However, avoid one public SKU per slice size unless pricing or guarantees differ materially. The first H200 product model should keep the catalog simple:

  1. h200-sxm-baremetal-8g: exclusive 8-GPU H200 node.
  2. h200-sxm-slice: shared-node H200 GPU VM product with a constrained user-selected GPU count. The internal SKU may keep the slice suffix, but user-facing catalog copy should say GPU VM or GPU VM slice, not SXM Slice.

The user-facing slice flow should present h200-sxm-slice once, then let the user select from platform-approved GPU counts. It should remain a launch wizard: capacity, SSH keys, storage, and network/fabric choices belong inside the same flow so users do not leave provisioning to complete required setup. Do not expose raw 1..8 unless all counts are topology-safe and operationally useful.

Initial allowed counts:

  1. 1 GPU: smallest practical slice; maps cleanly to one GPU slot bundle.
  2. 2 GPUs: useful medium shape; should prefer or require same-NUMA placement depending on discovered topology.
  3. 4 GPUs: larger shape; should require a topology-aligned group, likely a same-NUMA or same-board boundary.

Initial disallowed or deferred counts:

  1. 3 GPUs, 5 GPUs, 6 GPUs, 7 GPUs: defer until a real workload justifies capacity fragmentation and placement complexity.
  2. 8 GPUs as slice is allowed only as a topology-aligned full-host VM shape for the current platform-control rollout. The scheduler still treats it as a slice allocation so users get the VM/runtime flow; operators must see that it consumes every GPU slot on the host.

This gives a clean product surface:

sku: h200-sxm-slice
capacity_shape: gpu_slice
gpu_model: h200
allowed_gpu_counts: [1, 2, 4, 8]
same_node_required: true
exclusive_node: false
topology_policy:
  gpu_count_1: any_healthy_slot
  gpu_count_2: numa_aligned_preferred_or_required
  gpu_count_4: numa_aligned_required
  gpu_count_8: full_host_slot_group_required
  fabric_claim_mode: per_slot_vf
  fabric_vf_pci_address: <slot VF PCI address>
  fabric_parent_device: <optional parent BF/IB PCI/device>

sku: h200-sxm-baremetal-8g
capacity_shape: baremetal
gpu_model: h200
gpu_count: 8
exclusive_node: true

SKU metadata should include:

  1. capacity_shape
  2. accelerator vendor, model, and class
  3. allowed GPU counts for slice products, or fixed GPU count for baremetal
  4. whether it requires exclusive node access
  5. local storage requirement
  6. network fabric requirement, for example infiniband, roce, ethernet, or none
  7. NIC/VF attachment and validation requirements
  8. NUMA constraints
  9. compatible node topology profile
  10. price model

For future fractional or shared-GPU SKUs, metadata must be more specific than gpu_count. It should include the partition/share mechanism, allowed profiles, GPU memory or compute-share units, whether multiple claims can coexist on the same parent GPU, and whether the profile is exclusive, overcommitted, or time-sliced. Do not publish a fractional SKU until those semantics are explicit and validated by node-agent reconciliation.

Slice VM Runtime Profiles

Do not encode the VM shape as a single hardcoded value such as vcpu_per_gpu=12 or vcpu_per_gpu=24. The prototype is still changing, and different workloads may need different CPU/memory envelopes for the same GPU count. Model the VM shape as a named runtime profile selected by SKU, app profile, admin override, or later user-facing product option.

Short-term implementation can store this in sku_catalog.resource_profile without adding a new table:

{
  "fabric": "ib",
  "slice_unit_gpu_count": 1,
  "default_slice_vm_profile": "h200_1g_24c_64g",
  "slice_vm_profiles": {
    "h200_1g_12c_64g": {
      "gpu_count": 1,
      "vcpu_count": 12,
      "memory_mib": 65536,
      "notes": "early prototype shape"
    },
    "h200_1g_24c_64g": {
      "gpu_count": 1,
      "vcpu_count": 24,
      "memory_mib": 65536,
      "hugepages": {
        "enabled": true,
        "page_size": "1G"
      },
      "guest_driver_profile": "ubuntu_24_04_nvidia_570_server_rdma",
      "notes": "latest tested prototype shape"
    }
  }
}

For multi-GPU counts, the default profile can scale from the selected one-GPU profile, but it should still be materialized as an explicit profile once tested so benchmark expectations, placement, and billing exports have stable names. Example derived profiles:

  1. h200_2g_48c_128g
  2. h200_4g_96c_256g

Selection rule for v1:

  1. if an app profile requires a specific slice VM profile, use that profile;
  2. otherwise use the SKU default_slice_vm_profile;
  3. allow admin-only overrides during lab tuning;
  4. write the selected profile name and resolved CPU/memory values into the allocation placement/claim snapshot so later seed changes do not mutate running allocations.
  5. record whether the profile requires hugepage-backed memory and fail provisioning if the host has not reported that capability.

When this stabilizes or needs per-region/per-node-pool variants, promote it to a first-class slice_vm_profiles table with audit history. Until then, keeping it in SKU metadata keeps the rollout fast and avoids schema churn while still preventing hidden constants in node-agent.

For NVIDIA H200 slice VMs, the v1 guest bootstrap profile is ubuntu_24_04_nvidia_570_server_rdma: Ubuntu 24.04 cloud image, UEFI with Secure Boot disabled, NVIDIA 570 server driver utilities, kernel headers for the running guest kernel, linux-modules-extra for the running guest kernel, pciutils, and RDMA/IB tools (rdma-core, ibverbs-utils, infiniband-diags). Provisioning readiness must not stop at SSH reachability; the node agent waits for cloud-init to finish and requires a successful nvidia-smi -L before it marks the slice VM guest ready.

Billing should stay allocation-based. There should be one usage record per allocation billing interval, not one independent billable record per slot. The billing worker should derive billable dimensions from immutable allocation placement/claim snapshots:

  1. rate source: SKU price model at allocation start, with any baremetal or slice premium encoded in the SKU/price record;
  2. GPU count: number of claimed GPU slots or fixed baremetal GPU count;
  3. capacity shape: baremetal or gpu_slice;
  4. node id and slot ids: recorded for audit/export, not as independent billing accounts;
  5. effective resource count changes: require an explicit resize or repair state transition, not silent billing drift.

If a 4-GPU slice cannot provide all four GPUs after launch, the allocation should enter a degraded/failed operational state according to provisioning policy. Do not silently bill it as 3 GPUs unless a future resize workflow changes the allocation contract and writes an auditable ledger adjustment.

Baremetal and slice can both be expressed as GPU-hour SKUs, but they do not need the same price. Slice may carry a markup for virtualization/isolation overhead, or baremetal may carry a node-exclusive minimum. The ledger should record the SKU, capacity shape, GPU count, slot count, and node id so usage exports explain why the price was charged.

For future fractional/shared GPU products, do not assume GPU count remains the only billable dimension. The usage snapshot should support billable accelerator units such as GPU-memory GiB-hours, compute-share hours, partition-profile hours, or vendor vGPU-profile hours. The rate still comes from the SKU/price record at allocation start, but the units must come from claim snapshots so a later profile change cannot rewrite historical billing.

Vendor And Fabric Variants

The SKU catalog should distinguish product family and fabric without forcing a separate SKU for every GPU count.

Examples:

sku: h200-sxm-slice
accelerator_vendor: nvidia
accelerator_model: h200
capacity_shape: gpu_slice
network_fabric: infiniband
allowed_gpu_counts: [1, 2, 4, 8]

sku: amd-mi300x-roce-slice
accelerator_vendor: amd
accelerator_model: mi300x
capacity_shape: gpu_slice
network_fabric: roce
allowed_gpu_counts: [1, 2, 4]

The exact AMD SKU name should match the actual GPU model we onboard. The important rule is that RoCE is not treated as "missing IB"; it is a different fabric capability with its own validation and placement rules.

OS Image Catalog

GPU slices are VM-like allocations, so the platform needs an OS image catalog before slices become generally usable. The image chosen for a slice is part of the allocation runtime contract, not a node-local implementation detail.

Do not rely on ad hoc files copied onto one node. Store image metadata in the control plane and make the artifact immutable by digest. The binary image can be hosted in platform object storage, an OCI artifact registry, a controlled HTTP location, or a MAAS image reference, but GPUaaS should track the digest and compatibility rules centrally.

Initial image targets:

  1. baremetal: image references are MAAS deploy images or MAAS-compatible image aliases. These are used for full-node deploy/reimage flows.
  2. vm_slice: image references are cloud images such as qcow2 or raw, suitable for node-agent download/cache/clone and cloud-init injection.

Directional table:

os_images
  id uuid primary key
  slug text unique not null
  display_name text not null
  version text not null
  target text not null              -- baremetal | vm_slice
  family text not null              -- ubuntu | rocky | custom
  architecture text not null        -- amd64 | arm64
  accelerator_vendor text null      -- nvidia | amd | any
  network_fabric text null          -- infiniband | roce | ethernet | none | any
  image_format text not null        -- maas | qcow2 | raw | oci
  source_type text not null         -- maas | platform_storage | oci_artifact | http
  source_ref text not null
  digest_sha256 text not null
  size_bytes bigint null
  cloud_init_supported boolean not null default true
  guest_agent_supported boolean not null default false
  default_username text not null
  driver_strategy text not null     -- none | preinstalled_nvidia | preinstalled_rocm | install_on_boot
  compatibility_metadata jsonb not null default '{}'
  active boolean not null default true
  created_at timestamptz not null
  updated_at timestamptz not null
  retired_at timestamptz null

First implementation default:

slug: ubuntu-24.04-gpuaas-slice
target: vm_slice
family: ubuntu
architecture: amd64
source_type: platform_storage
image_format: qcow2
driver_strategy: install_on_boot
cloud_init_supported: true

For GPU slice products, we should also support accelerator-specific optimized images:

slug: h200-ubuntu-24.04-gpuaas-slice
target: vm_slice
family: ubuntu
architecture: amd64
accelerator_vendor: nvidia
accelerator_model: h200
source_type: platform_storage
image_format: qcow2
driver_strategy: preinstalled_nvidia
cloud_init_supported: true
compatible_skus: ["h200-sxm-slice"]

This is likely the preferred production path for H200 slices because it reduces startup time and removes driver install variability from the hot allocation path. The tradeoff is image maintenance: every driver/CUDA/security baseline change becomes an image build, publish, verify, and rollout event. GPUaaS should therefore keep both concepts:

  1. generic slice image, useful for development, CPU-only validation, and fallback bootstrapping;
  2. accelerator-specific slice image, preferred for production GPU products where startup latency and repeatability matter.

Admin API direction, contract-first when implemented:

  1. GET /api/v1/admin/os-images: list images with filters for target, active, family, accelerator vendor, and fabric.
  2. POST /api/v1/admin/os-images: add an image metadata record and optionally trigger verification/import.
  3. GET /api/v1/admin/os-images/{image_id}: inspect image metadata, compatibility, verification status, and current usage.
  4. PATCH /api/v1/admin/os-images/{image_id}: update mutable fields such as display name, active flag, default marker, compatibility metadata, or notes.
  5. DELETE /api/v1/admin/os-images/{image_id}: retire the image by default. Hard delete should be allowed only when the image has never been referenced by an allocation, app instance, or audit event.

Privileged image mutations require audit logs. The API should reject missing digests for immutable artifact sources. A later verification task should download or inspect the artifact, validate digest/format/cloud-init support, and mark it usable for scheduling.

Slice provisioning should pass os_image_id, source_ref, digest_sha256, and compatibility metadata to node-agent. Node-agent then downloads or reuses a local cache under a controlled path such as /var/lib/gpuaas/images, verifies the digest, clones a fresh disk for the allocation, injects cloud-init, and never boots directly from the shared cached image.

The prototype clones an Ubuntu cloud image directly to the raw NVMe device with qemu-img convert -O raw. GPUaaS can use the same fast path, but it needs guardrails:

  1. verify the destination block device is the approved slot disk and is not mounted by the host;
  2. require the image digest to match the control-plane catalog entry before cloning;
  3. treat clone/write failures as provisioning failures with structured node-task output;
  4. never allow a user-provided path to select the destination disk;
  5. explicitly wipe or recreate the disk on release before the slot is reusable.

Manual ISO installer flows are useful for lab debugging, but they should not be part of the default customer slice product. If kept, expose them only as an admin/debug operation with console access, audit logging, and no billing promise until the VM reaches a managed-ready state.

Latest prototype runtime details worth carrying into implementation:

  1. guest boot uses UEFI with secure boot disabled because the driver and passthrough path currently assume that compatibility mode;
  2. optional cloud-init guest driver installation installs NVIDIA server driver packages and RDMA/IB packages, then reboots the guest;
  3. production should prefer accelerator-specific prebuilt images over hot-path guest driver installation, but the cloud-init install path remains useful for lab fallback;
  4. the prototype pins each one-GPU slice to NUMA-local CPU IDs, but the current sample command requests 24 vCPUs while its static cpuset examples list only 12 CPU IDs per NUMA side. GPUaaS should derive a valid CPU set from topology and requested vcpu_count, not copy the static list.

Latest prototype benchmark evidence:

Metric Bare metal One-GPU VM slice Delta
DCGM FP64 627 TFLOPS 614 TFLOPS -2.1%
HBM bandwidth 4000 GiB/s 4000 GiB/s 0%
IB write bandwidth 363 Gb/s 206 Gb/s -43%
vLLM OPT-125M 2404 tokens/s 1993 tokens/s -17%

Interpretation: PCI passthrough is strong enough for GPU compute and memory bandwidth, but fabric and vCPU tuning remain real scheduling/profile inputs. The first production profile should record expected benchmark thresholds and fail readiness or mark slots degraded when a node falls materially below them.

Follow-up tuning backlog:

  1. Fabric/BF3/IB: validate MTU, multi-queue, queue depth, IRQ affinity, NUMA locality, VF/representor mode, OVS offload, and per-vNIC QoS before deciding the production networking profile. The current IB delta is large enough that it should be tracked as a performance workstream, not treated as a normal virtualization cost.
  2. NVIDIA/GPU guest path: compare guest driver versions, CUDA stack, persistence mode, fabric manager behavior, and GPU reset behavior between bare metal and VM images. Prefer prebuilt accelerator-specific images once the working combination is known.
  3. CPU/vLLM: tune vCPU count, cpuset generation, CPU governor, hugepages, NUMA memory policy, tensor parallel settings, and container runtime options. The vLLM delta may be more CPU/orchestration-related than GPU-related.
  4. Benchmark contract: store benchmark profile name, command versions, driver versions, and expected thresholds with the slice VM runtime profile so readiness/acceptance tests are reproducible.

OS Image Build And Rollout

The image catalog needs a separate build/publish pipeline spec before implementation. This allocation design assumes the pipeline provides:

  1. repeatable image build inputs, including base OS, kernel, driver/CUDA or ROCm stack, guest agent, and cloud-init support;
  2. automated boot and smoke tests before an image can become active;
  3. digest and optional signature/attestation capture;
  4. canary rollout by SKU, region, or node pool before fleet-wide defaulting;
  5. rollback by changing the default image pointer while keeping old image metadata available for existing allocations;
  6. node-agent cache eviction rules that never delete an image backing a running VM.

For now, the admin API can store and retire image records manually. Production slice rollout should not depend on manual image creation long term.

App Runtime Compatibility

Apps should not care whether the underlying allocation is a node or a slice when their contract only needs allocation-local execution. The app should target an allocation, and the platform should decide whether that allocation is suitable.

Allow Apps on Slices?

Yes, but only when the app profile declares compatibility with the allocation shape and resource envelope.

Initial rule:

  1. Launchable OCI/Jupyter/vLLM-style single-node apps may run on gpu_slice allocations if:
  2. the allocation has enough GPU/CPU/memory/storage for the app request,
  3. the slice runtime exposes Docker/Compose or the required container runtime,
  4. network exposure mode is supported for slice VMs,
  5. the profile does not require host-level privileges.
  6. Single-node Slurm or Kubernetes profiles may run on gpu_slice allocations if the adapter keeps the entire control plane and worker runtime inside one allocation and does not require cross-node networking.
  7. Multi-node Slurm, Kubernetes, and other cluster-style control-plane app profiles should default to baremetal or remain unavailable until their adapter and the slice networking model explicitly support slice-hosted clusters.
  8. Apps that require multi-node workers, host networking, privileged device control, or full-node reimage must declare requires_capacity_shape: baremetal.

This rule does not prohibit infra-managed controller VMs on slice-capable hosts. For example, an operator may create a small Slurm controller or Kubernetes control-plane VM with no GPU claim, such as 24 vCPUs, 64 GiB RAM, and a 50 GiB disk, using virt-manager or a future platform-managed system_vm workflow. That is an infrastructure placement decision, not the same as advertising the Slurm or Kubernetes customer app adapter as slice-compatible.

The platform should eventually model these as system/control-plane placements:

  1. capacity_shape=system_vm or equivalent internal placement type;
  2. CPU, memory, disk, network, and host affinity requirements;
  3. no customer GPU slot claim unless explicitly needed;
  4. admin-only lifecycle and audit trail;
  5. exclusion from normal customer slice availability and billing unless product policy says otherwise.

Until project/private networking is available, the safe product boundary is:

  1. single-node Slurm/Kubernetes on one allocation can be considered slice compatible;
  2. multi-node Slurm/Kubernetes clusters should not be advertised on slices;
  3. baremetal remains the default for real multi-node cluster products.

App Manifest Additions

Phase 4 should start with a small manifest contract:

  1. requires_capacity_shape, for example ["baremetal"] or ["baremetal", "gpu_slice"];
  2. min_gpu_count;
  3. requires_exclusive_node.

The platform can infer most v1 fabric, device, runtime, and networking checks from the selected SKU, allocation placement, OCI profile, and app adapter. More specific declarations can be added later only when a real app needs them, such as explicit accelerator runtime, required fabric access, host networking, or slice NAT tolerance.

The app create flow should keep using placement_intent.target_allocation_id for allocation-local apps. The validation layer should inspect the target allocation placement and reject incompatible profiles before dispatching node tasks.

Future fractional/shared-GPU support should be opt-in per app adapter. Existing gpu_slice compatibility must not automatically imply fractional compatibility, because device visibility, isolation, driver behavior, performance guarantees, and reset semantics differ between MIG, vGPU, MPS, and time-sliced sharing. Initial fractional-compatible candidates are likely simple allocation-local Jupyter or vLLM profiles after validation. Slurm, Kubernetes, and other control-plane apps should remain on baremetal or whole-GPU gpu_slice allocations until their adapters explicitly support fractional devices.

App Instance Members

app_instance_members.bound_node_id is not sufficient for slices. Member views should eventually include:

  1. bound_allocation_id
  2. bound_node_id
  3. bound_slot_ids
  4. capacity_shape
  5. resource snapshot

This keeps app detail pages understandable when multiple app instances run on different slices of the same physical node.

UI Implications

User Allocation Flow

The allocation flow should ask for product shape before placement details:

  1. choose SKU/product: baremetal node or GPU slice,
  2. choose region,
  3. choose GPU count,
  4. optionally choose storage/network add-ons,
  5. review price and availability,
  6. create allocation.

Users should not be asked to pick raw PCI devices.

Admin Nodes

Node detail must show:

  1. physical node status,
  2. agent version/health,
  3. total slots,
  4. used/reserved/cleanup/disabled slots,
  5. active allocations per slot,
  6. whether baremetal is currently blocked by slice use,
  7. whether slice scheduling is blocked by node-exclusive/baremetal use,
  8. bootstrap capability status and whether a reboot is required,
  9. per-slot drift status for VM, device, disk, network, and cleanup evidence,
  10. optional admin console action when a slice VM requires manual debugging.

App Deploy Wizard

The app deploy wizard should show only compatible target allocations. If a Jupyter profile supports slices, slice allocations appear. If a Slurm profile requires baremetal, slice allocations are hidden or marked incompatible.

Slice Networking

Slice networking is a first-class part of slot compatibility, but the detailed network architecture is maintained separately in doc/architecture/Slice_Networking_Architecture_v1.md.

For placement purposes, this document depends on three decisions from that networking spec:

  1. slice VMs have a management/public plane and a workload fabric plane;
  2. public ingress is a management-plane IPAM/firewall/proxy concern, not an IB feature;
  3. console access, if provided, must use an authenticated/audited gateway rather than raw node-exposed VNC.

Isolation and Cleanup

Baremetal and slice release paths are different.

Baremetal:

  1. revoke user/session access,
  2. stop app runtimes,
  3. optionally MAAS full reimage,
  4. return node to active pool.

GPU slice:

  1. gracefully stop VM or slice runtime;
  2. hard-stop only after the configured graceful timeout expires;
  3. wipe raw disk or recreate image;
  4. reset GPU/network devices where supported;
  5. release DHCP/IP/MAC lease;
  6. validate the slot can run a health smoke;
  7. mark slot available.

Slice cleanup failure should block only the affected slot unless the failure indicates node-wide device/driver corruption.

Raw disk cleanup must be explicit. virsh undefine --remove-all-storage is not sufficient proof that a raw NVMe device was wiped or safe for another tenant. The release task should unmount any host-visible partitions, run the configured wipe/blkdiscard/reimage operation, and verify the disk no longer contains the previous tenant's filesystem signature before marking the slot available.

Raw NVMe wipe verification is a blocking release criterion for slice reuse. A slot with unverified wipe state must remain cleanup or cleanup_blocked and must not be scheduled to another tenant.

The VM shutdown sequence should be:

  1. request graceful shutdown, for example virsh shutdown;
  2. wait for a configurable timeout, defaulting to 120 seconds unless policy says otherwise;
  3. use hard stop, for example virsh destroy, only as fallback;
  4. continue disk, network, and device cleanup after the VM is confirmed stopped.

Capacity Read Models

Needed API/read-model surfaces:

  1. region/SKU availability by capacity shape,
  2. node summary with aggregate slot counts,
  3. node detail with per-slot state for admins,
  4. allocation detail with placement shape and claims,
  5. app target allocation picker with compatibility reasons.

For product UX, users should see product availability, not raw slot internals. For operators, slot internals must be visible.

Migration Strategy

Phase 1: Document and read-model groundwork.

  1. Add capacity shape fields to docs/contracts.
  2. Add slot/claim tables behind feature flags.
  3. Keep existing whole-node path unchanged.
  4. Backfill existing active allocations as capacity_shape=baremetal with a node-exclusive claim.

Phase 2: Admin inventory.

  1. Let node-agent/bootstrap report discovered resource topology.
  2. Let admins approve or correct slot bundles.
  3. Show slot inventory in Admin Nodes.
  4. Add admin OS image catalog CRUD for slice VM and baremetal image metadata.
  5. Add image verification/import tasks before images become selectable.
  6. Add topology discovery evidence for IB/RDMA devices, NVMe identity, NUMA, BF3/management network identity, MAC/IP reservations, and CPU/memory slot capacity.

Phase 3: Slice allocation.

  1. Add gpu_slice SKU entries.
  2. Add claim-aware placement.
  3. Add topology-aware best-fit placement for [1, 2, 4] H200 slice counts, with control-plane region/cluster placement and node-agent-owned host-local bundle validation.
  4. Add node-agent node-scheduler planning for complete slice bundles: GPU, IB/RDMA, raw NVMe, CPU, memory, MAC/IP, and NUMA policy.
  5. Add node-agent typed tasks for VM slice lifecycle.
  6. Add slice release/cleanup verification.
  7. Add usage-record dimensions sourced from allocation claim snapshots.
  8. Resolve default vm_slice OS image from SKU/profile when the user does not explicitly choose one.
  9. Enforce graceful shutdown, readiness gates, and blocking raw NVMe wipe verification before slot reuse.

Phase 3a: OS image pipeline.

  1. Add build/publish/verify pipeline spec for slice images.
  2. Add image canary/default/rollback model.
  3. Add node-agent image cache policy.
  4. Keep manual admin image registration as the bootstrap path only.

Phase 4: App compatibility.

  1. Add app profile compatibility declarations.
  2. Filter target allocation picker by compatibility.
  3. Enable launchable OCI/Jupyter/vLLM on slices first.
  4. Enable single-node Slurm/Kubernetes profiles on slices only when the adapter runs fully inside one allocation.
  5. Keep multi-node Slurm/Kubernetes cluster profiles baremetal-only until slice networking supports clusters.
  6. Separately evaluate infra-managed system_vm placements for Slurm controllers, Kubernetes control planes, and similar admin-owned services on slice-capable hosts.

Deferred Phase 5: Fractional/shared GPU model readiness.

  1. Keep the first implementation disabled for fractional/shared scheduling.
  2. Reserve model semantics for child slots under a parent physical GPU.
  3. Require explicit capacity shapes such as gpu_partition or gpu_shared before exposing a user-facing fractional product.
  4. Require node-agent support for the selected mechanism, for example MIG, vGPU/mdev, SR-IOV-style accelerator partitioning, MPS, or time slicing.
  5. Extend billing dimensions from whole-GPU count to claim-snapshotted accelerator units before charging for fractional products.
  6. Make app compatibility opt-in and adapter-specific; do not treat whole-GPU slice support as fractional support.

Open Questions

  1. Is the first slice implementation always one GPU per VM, or do we support multi-slot VM allocations immediately?
  2. Should CPU and memory be first-class slot resources or node-level soft capacity constraints in the first slice?
  3. Is local NVMe always exclusive per slot, or can some profiles use shared project storage instead?
  4. Is InfiniBand required for all H200 slices, or should network device requirements be SKU-specific?
  5. What RoCE validation is sufficient for AMD slices before a slot can be marked available?
  6. Should node-agent call libvirt/QEMU directly, or should it call a bounded local slice manager while preserving the same typed-task contract?
  7. Which VM network attachment should be the default for kind, MAAS lab, and production: bridge, macvtap, OVS, or another controlled model?
  8. How do we expose private slice endpoints through existing app access models?
  9. What is the minimum cleanup validation before a slot can be reused?
  10. Should OS images be stored primarily as OCI artifacts, platform object storage objects, or both?
  11. Do we require image signing/attestation before a slice image is active?
  12. What image cache eviction policy should node-agent use when a node hosts many slice image versions?
  13. Which GPU driver strategy is safest for each slice family: preinstalled driver, install-on-boot, or node-agent mounted driver payload?
  14. What is the exact source of truth for per-slice CPU and memory: SKU defaults, approved slot metadata, or both?
  15. Which node-local IPAM implementation should back private slice networking: dnsmasq, libvirt network DHCP, central DHCP/IPAM, or a small node-agent-owned allocator with reconciliation?
  16. Which admin-only console gateway should replace direct VNC exposure for manual slice debugging?
  17. What destructive-disk safety checks are mandatory before a raw NVMe device can be assigned or reused?
  18. Should topology discovery be entirely node-agent driven, or should each site maintain a reviewed topology profile for known server layouts?
  19. Which guest readiness signal is required per OS image family: SSH only, cloud-init completion, guest agent, or an app-specific probe?
  20. What policy key or SKU/profile value controls graceful VM shutdown timeout?
  21. Should infra-managed controller VMs be modeled as a separate internal system_vm capacity shape, or as reserved CPU/memory/disk placements on a slice-capable node?
  22. Which fractional/shared-GPU mechanism, if any, should be supported first: MIG, NVIDIA vGPU/mdev, SR-IOV-style accelerator partitioning, CUDA MPS, time slicing, or app-level software multiplexing?
  23. Should fractional inventory be represented as child rows in node_resource_slots, a separate accelerator partition table, or generated only after an admin approves a parent GPU partition profile?
  24. What billable units should fractional GPU products use: GPU-memory GiB-hour, compute-share hour, named partition-profile hour, or another SKU-defined unit?

Non-Goals For First Slice

  1. Cross-node sliced allocations.
  2. Arbitrary overcommit of GPUs, NVMe, or IB devices.
  3. Generic VM cloud product semantics.
  4. User-visible PCI/device selection.
  5. Running every app type on slices immediately.
  6. Replacing MAAS baremetal lifecycle.
  7. Live migration or live defragmentation of GPU slice VMs.
  8. Fractional/shared-GPU scheduling, including MIG, vGPU, MPS, time slicing, or software multiplexing.