Skip to content

MAAS Provisioning Time Optimization v1

Status: draft; first instrumentation, wait-loop improvements, and host image pipeline implemented Audience: GPUaaS engineering, infra/MAAS operators, platform-control reviewers Last updated: 2026-04-21

Purpose

Capture the concrete optimization directions for reducing MAAS-backed node provisioning time without weakening the existing lifecycle contract.

This document is intentionally narrower than the full MAAS lifecycle baseline:

It focuses on:

  1. where provisioning time is currently spent,
  2. which work belongs in the image versus first boot versus post-enrollment,
  3. which optimizations preserve control-plane correctness,
  4. what should be measured before and after each change.

It does not redefine:

  1. the MAAS workflow API,
  2. the public endpoint/public IP model,
  3. the embedded UI gateway or workload proxy model.

Problem Statement

Current provisioning paths still spend too much time after MAAS deploy completes. The likely bottlenecks are no longer just MAAS image transfer. They are spread across:

  1. boot-to-cloud-init timing,
  2. bootstrap package fetch and validation,
  3. package install or runtime prerequisite install during first boot,
  4. enrollment token/bootstrap token validation,
  5. node-agent install/start and first enrollment,
  6. post-deploy readiness polling and conservative completion gates.

The product goal is:

  • keep the control path safe and auditable,
  • keep the data path direct,
  • shift repeatable heavy work into the image or controlled site/profile bundles,
  • leave only small environment-specific configuration for first boot.

Guiding Position

Provisioning speed should come primarily from moving work earlier in the lifecycle, not from weakening readiness checks.

The preferred layering is:

  1. MAAS image owns the base OS and heavy/static prerequisites.
  2. Site/profile bootstrap bundle owns site-specific environment wiring.
  3. GPUaaS bootstrap owns node-agent enrollment and final platform handoff.
  4. Post-enrollment typed node tasks own ongoing drift repair or later runtime changes.

Avoid using first boot as a generic install script phase for large packages or repeated host bootstrap that could be prebuilt.

Current Execution Boundary

The existing MAAS docs already preserve several behaviors that must remain:

  1. deterministic discovery/adoption,
  2. PXE safety before deploy,
  3. bounded classified deploy retry,
  4. infra-owned site bootstrap before GPUaaS bootstrap,
  5. explicit completion gates before a node is considered active.

Those behaviors should stay. Optimization should change where work happens, not remove those guardrails.

Provisioning Time Buckets

For optimization work, measure at least these buckets separately:

  1. request accepted to MAAS workflow start,
  2. discovery/adoption time,
  3. commission time,
  4. ready-to-deploy wait,
  5. image deploy time,
  6. first boot to cloud-init start,
  7. cloud-init/bootstrap package fetch,
  8. site bootstrap execution,
  9. GPUaaS bootstrap execution,
  10. node-agent enrollment,
  11. post-enrollment runtime prereq checks,
  12. control-plane readiness confirmation.

Without this split, image and first-boot changes will be hard to evaluate.

Current implementation notes:

  1. site bootstrap can receive the bootstrap progress callback environment,
  2. GPUaaS node bootstrap now emits explicit node_bootstrap started, succeeded, and failed callbacks to the control plane when a progress URL is available,
  3. onboarding and decommission wait loops record MAAS INFO events into the workflow timeline for the active stage window,
  4. node enrollment polling now runs on a shorter interval than MAAS deploy polling so an already-enrolled node is not held behind a coarse wait tick,
  5. onboarding and decommission reenrollment loops record node status changes while waiting for active state.
  6. a platform-control hosted MAAS H200 host-image build/upload pipeline now exists for the first prebaked image experiment; see MAAS_H200_Host_Image_Pipeline_Runbook.md.
  7. onboarding and decommission release/deploy submission timeouts are treated as unknown-outcome MAAS mutations: the workflow records the timeout and polls MAAS for the target state before failing.

Optimization Layers

1. Image-level optimization

Move these into the MAAS-deployed image whenever they are stable enough:

  1. base OS packages required on every managed node,
  2. container runtime packages when they are standard for the target profile,
  3. GPU/driver support packages that do not need per-host dynamic install,
  4. large package repositories, apt sources, and trusted keys,
  5. common diagnostics used by first boot and incident workflows,
  6. any static bootstrap helper binaries or launchers.

Initial H200 host image content:

  1. Ubuntu 24.04 Noble cloud-image base expanded for H200 host use,
  2. Docker and compose plugin,
  3. NVIDIA host driver utilities and NVIDIA container toolkit where available,
  4. Docker daemon baseline config with BuildKit enabled and bounded JSON log retention,
  5. optional DCGM/fabric manager packages where available from configured repos,
  6. Netdata with bounded retention plus host nvidia-smi, Docker, cgroup, and systemd collectors,
  7. libvirt/qemu/OVS/dnsmasq/cloud-image tooling required by the slice host runtime,
  8. RDMA/IB diagnostics and perf tooling,
  9. slice passthrough kernel args (intel_iommu=on iommu=pt) and a bounded systemd-networkd-wait-online baseline to avoid waiting on unused secondary/fabric ports,
  10. the current h200-slice-vm site bootstrap bundle staged under /usr/local/share/gpuaas/site-bootstrap/.

The image does not pre-run hardware-specific configuration. VFIO binding, SR-IOV VF creation, slice bridge creation, and token/enrollment work remain first-boot responsibilities. IOMMU is the exception: the kernel args should be baked so first boot has IOMMU groups and the slice host does not need a second reboot before topology discovery can pass.

Benefits:

  1. removes repeated package download/install cost,
  2. reduces exposure to transient mirror/network delays at first boot,
  3. keeps first-boot cloud-init smaller and easier to debug.

Risks:

  1. image rebuild cadence becomes important,
  2. stale images can hide drift until deploy,
  3. multiple site/profile variants can multiply image count if unmanaged.

Recommendation:

  • use a small number of curated base images per host class/profile family,
  • do not let each site invent arbitrary one-off images,
  • version and publish image lineage explicitly.

The first H200 baseline should compare these two readiness moments separately:

  1. node active/enrolled: node-agent is up, authenticated, and reporting host identity,
  2. slice-ready: approved or discoverable slice slots have been reported and can be scheduled.

For slice hosts, slice-ready must be stricter than "node-agent is active" or "VFIO devices are bound." It also requires the fabric and VM runtime baseline:

  1. the slice profile tag in MAAS is gpuaas-profile-slice-vm;
  2. gpuaas-slice-fabric-vfs, gpuaas-slice-vfio-devices, gpuaas-slice-network-baseline, and gpuaas-slice-ovs-bridge are active;
  3. IPoIB netplan exists, an ibp* interface has the site fabric address, and a fabric route is installed;
  4. ovsbr0 has the guest subnet gateway;
  5. dnsmasq and libvirtd are active;
  6. node-agent topology reports the slice network evidence without blockers.

This is intentionally a convergence gate, not just an image-build gate. A prebaked image can remove package-install time, but the first-boot site/profile bootstrap still owns host-specific fabric, bridge, and scheduler readiness.

j22u11 showed that a node can be active while slice slots are still not reported. That makes slot discovery/reporting a distinct performance and correctness gate, not just part of generic onboarding completion.

2. Site/profile bootstrap optimization

The site/profile bootstrap bundle should stay infra-owned or tightly controlled, but it should be minimal.

It should do only what must remain site-specific at first boot:

  1. network/site wiring,
  2. site CA or trust bundle placement,
  3. hardware/site toggles not safe to bake globally,
  4. bootstrap launcher execution with versioned inputs,
  5. small site-specific configuration files.

It should not become the place where:

  1. Docker or large runtime packages are installed every time,
  2. full hardware-sync reseeds run without reason,
  3. ad hoc remote shell logic accumulates.

3. GPUaaS bootstrap optimization

GPUaaS bootstrap should be limited to:

  1. validating rendered bootstrap inputs,
  2. installing or starting node-agent if not already baked,
  3. enrolling with bootstrap/enrollment tokens,
  4. writing minimal control-plane managed config,
  5. confirming the first successful node heartbeat.

It should not own:

  1. large package installation by default,
  2. long-lived SSH-based host mutation,
  3. recurring repair logic after enrollment.

4. Post-enrollment repair split

Anything that is:

  1. repeatable,
  2. bounded,
  3. useful after the node is already under control,

should move to typed node-agent tasks instead of first boot.

Examples:

  1. runtime prerequisite drift repair,
  2. registry trust refresh,
  3. optional package/runtime updates,
  4. diagnostics collection,
  5. controlled service lifecycle actions.

This keeps MAAS first-boot responsibility smaller and faster.

Fastest Safe Path

The best near-term optimization path is:

  1. build a curated MAAS image with the heavy common packages already installed,
  2. keep site/profile bootstrap focused on environment configuration,
  3. keep GPUaaS bootstrap focused on enrollment,
  4. move any recurring post-enrollment setup into typed node tasks,
  5. preflight all bootstrap inputs before MAAS deploy starts.

This is the highest-signal path because it reduces time in the longest repeated steps without changing the public lifecycle model.

Predeploy Hardening

Before deploy starts, validate:

  1. bootstrap endpoint reachability from the MAAS-served node network,
  2. exact bootstrap artifact/package URL fetchability,
  3. CA bundle content is valid and final, not a template placeholder,
  4. bootstrap-token and enrollment-token persistence is internally consistent,
  5. site/profile bootstrap inputs resolve cleanly,
  6. selected image/profile version exists and is approved for the site.

The point is to fail before deploy when the inputs are known-bad instead of waiting for a full deploy plus cloud-init failure.

MAAS Mutation Timeout Rule

Some MAAS mutation calls can time out after MAAS has accepted the operation. Those failures must not be treated as terminal until the workflow verifies the machine's final state.

For long-running state transitions, use this rule:

  1. submit mutation,
  2. if the request succeeds, poll for the target state,
  3. if the request times out, record an unknown-outcome progress event and poll for the target state,
  4. fail only if the follow-up poll reaches a MAAS failure state or expires.

Currently covered transitions:

  1. onboarding release deployed machine to Ready,
  2. onboarding retry release to Ready,
  3. decommission release to Ready,
  4. onboarding commission to Ready,
  5. onboarding deploy to Deployed,
  6. decommission reimage deploy to Deployed.

Short configuration mutations such as boot-disk selection, storage layout, and RoCE link assignment still need explicit post-write verification before they can use this timeout rule safely.

Post-Deploy Completion Gates

The node should not transition to active until all are true:

  1. MAAS reports Deployed,
  2. site bootstrap completed successfully,
  3. GPUaaS bootstrap completed successfully,
  4. node-agent started,
  5. first enrollment succeeded,
  6. node row reached the expected active state,
  7. onboarding attempt metadata reflects the currently executing run.

These gates should remain explicit even if image/bootstrap work becomes faster.

Image Strategy Options

Option A: Generic Ubuntu + first-boot install

Pros:

  1. lowest image maintenance,
  2. easiest early experimentation.

Cons:

  1. slowest path,
  2. highest dependence on mirrors/network/package repos during deploy,
  3. largest first-boot variance.

This should not be the preferred production direction.

Option B: Curated MAAS base image per host/profile family

Pros:

  1. best balance of speed and control,
  2. predictable boot/runtime environment,
  3. first boot stays small.

Cons:

  1. image pipeline required,
  2. versioning and rollout discipline required.

This is the recommended default.

Option C: Fully site-custom image per site

Pros:

  1. maximum local optimization.

Cons:

  1. image sprawl,
  2. hard-to-debug drift,
  3. poor reproducibility across sites.

Avoid unless a site truly requires it.

Suggested Ownership Split

GPUaaS engineering owns:

  1. image/version policy,
  2. which prerequisites are image-level versus bootstrap-level,
  3. bootstrap token/enrollment flow correctness,
  4. typed post-enrollment repair tasks,
  5. timing instrumentation and read models.

Infra/MAAS operators own:

  1. MAAS image upload/availability process,
  2. site-local bootstrap bundle content where infra-owned,
  3. mirror/package source reliability,
  4. MAAS network/PXE health.

Instrumentation Requirements

Add or preserve timeline evidence for:

  1. image selected,
  2. deploy started,
  3. deploy finished,
  4. first boot observed,
  5. site bootstrap started/finished,
  6. GPUaaS bootstrap started/finished,
  7. node-agent enrollment started/finished,
  8. node became active.

Optimization work should not proceed without timing visibility for these stages.

Slice 1: Measure current path precisely

Before changing behavior broadly:

  1. capture timing for the buckets listed above on the current MAAS flow,
  2. identify which steps dominate the wall clock,
  3. record whether failures cluster around image deploy, cloud-init, package install, or enrollment.

Slice 2: Prebaked image experiment

Run a controlled comparison between:

  1. current image + full first-boot path,
  2. prebaked MAAS image with common prerequisites already installed.

Compare:

  1. deploy time,
  2. boot-to-enrollment time,
  3. failure rate,
  4. operational complexity.

Implementation entry point:

scripts/ops/gpuaas_maas_h200_host_image_platform_control_pipeline.sh

The pipeline builds on platform-control and uploads custom/gpuaas-h200-host-ubuntu2404 to MAAS. Use the current j22u11 run as the baseline before switching the host/profile to the prebaked image.

2026-04-21 update: the custom image must be a MAAS tgz root filesystem archive, not a ddgz full-disk artifact, for the current flat-storage deploy path. The ddgz experiment reached curtin but failed at GRUB/EFI because the disk image root and MAAS-created EFI partition were not aligned. The tgz artifact lets MAAS own partitioning and completed deployment successfully.

The first successful j22u11 custom-image run still spent most time in first-boot site bootstrap. The next image build therefore bakes the H200 site-bootstrap base package set and best-effort DOCA/OFED packages into the image, and the h200-ib site bootstrap skips those apt-heavy phases when /etc/gpuaas/maas-host-image-build.json is present and the expected packages are installed.

2026-04-26 follow-up: a later custom-image reimage of j22u11 proved that node active/enrolled can pass while slice-ready is still false. The host had node-agent, Netdata, VFIO, and OVS, but did not have the IPoIB netplan, fabric route, or gpuaas-slice-network-baseline service present on the known-good j22u15 slice host. The optimization path must therefore preserve or compose the h200-ib fabric-network baseline into the h200-slice-vm path before the custom image can be considered production-ready for slice scheduling.

Slice 3: Bootstrap minimization

Reduce first-boot cloud-init/bootstrap to:

  1. environment wiring,
  2. token materialization,
  3. node-agent start/enroll,
  4. final readiness report.

Slice 4: Post-enrollment drift repair extraction

Move anything still done in first boot but not required for first enrollment into typed node-agent tasks.

Explicit Non-Goals

  1. replacing the MAAS lifecycle contract,
  2. weakening safety checks just to improve timing,
  3. coupling public endpoint/network design into MAAS optimization,
  4. making the MAAS bootstrap path a generic shell execution channel,
  5. turning platform API services into a data-plane proxy.

Summary

The strongest provisioning-time reduction path is not more boot-time scripting. It is:

  1. prebuilt images for heavy/static prerequisites,
  2. smaller site/profile bootstrap,
  3. minimal GPUaaS bootstrap,
  4. typed post-enrollment repair,
  5. better timing instrumentation and predeploy validation.

That path keeps the control-plane model intact while cutting repeated work from the critical path.