Multi-Node Inference: Tensor Parallelism Across a DGX Spark and an x86_64 Node

Splitting a 70B-parameter model across a DGX Spark (GB10 Grace Blackwell superchip, arm64, 1 GPU) and an x86_64 box with 2 discrete GPUs — both running Talos Linux, in a single Kubernetes cluster — and why the “obvious” way to parallelize across them still wastes your best hardware.

Why this cluster is a harder problem than it looks

On paper this is a 3-GPU, 2-node Kubernetes cluster. In practice it’s still one of the more awkward topologies you can hand to a tensor-parallel (TP) inference engine, for three reasons that compound on each other:

Asymmetric compute per node. The DGX Spark’s GB10 is a single Blackwell GPU die fused to a Grace CPU on one package, sharing 128 GB of LPDDR5X unified memory at 273 GB/s between CPU and GPU (NVIDIA DGX Spark product page, Tom’s Hardware GB10 deep dive). The x86_64 node’s two GPUs are ordinary discrete cards with their own dedicated VRAM pools — almost certainly much smaller per-GPU than GB10’s 128 GB, and connected to each other over NVLink/PCIe, not to the Spark.
One cluster, two CPU architectures. Both nodes run Talos, which solves the control-plane problem — one kubectl get nodes, one etcd, one upgrade process, one RBAC surface. It does not solve the silicon problem: the DGX Spark is arm64 (Grace), the other node is amd64. Every container image that has to run on both — the NVIDIA driver extension, the device plugin, the vLLM/Ray image itself — needs an actual linux/arm64 build, not just a tag that happens to exist. Some of those images have historically bolted arm64 support on well after the amd64 build shipped, and Talos support for DGX Spark specifically is new enough that there are open compatibility issues upstream as of this writing. This is the section of the post that changed the most from the previous k3s+Talos revision — swapping “which control plane” for “which architecture” as the recurring gotcha.
Odd GPU counts break clean tensor-parallel math. TP shards attention heads and MLP columns evenly across ranks. With tp_size=3 you need num_attention_heads % 3 == 0 and hidden_size % 3 == 0 — a lot of popular checkpoints (64-head 70B-class Llama variants, for instance) fail that check outright. And even when it divides cleanly, symmetric TP forces every shard to be the same size, which means your usable capacity is tp_size × min(GPU_memory) — not the sum. Naive TP=3 across this cluster caps the whole 128 GB GB10 die at whatever your smallest x86 GPU has spare, because the shard sizes must match.

That third point is still the crux of this post, unchanged by the platform migration. The thesis: don’t force symmetric tensor parallelism across heterogeneous nodes — use pipeline parallelism (PP) to cross the node boundary, with an uneven layer split that lets the Spark carry the memory-heavy stage, and reserve tensor parallelism for the one place it’s actually free: the two matched GPUs sitting inside the x86 box.

We’ll build that primary configuration end to end on the new single-cluster topology, then show the “flat TP=3” alternative and exactly where it falls over, then close with a from-scratch PyTorch/NCCL prototype of heterogeneous tensor parallelism — uneven column splits proportional to each device’s memory — which is the direction the ecosystem is heading but doesn’t yet ship out of the box (see vLLM issue #27239, “Heterogeneous TP per Pipeline Stage”, still unsolved).

Stack for this build: vLLM + Ray for orchestration, Llama-3.1-70B-Instruct as the target model, plain TCP/Ethernet for the inter-node fabric (no InfiniBand/RoCE assumed), and Talos Linux on both nodes.

Why bother — the business case for one cluster, two architectures

Fair question, and it deserves a sharper answer than “because you can.” If you’re speccing a cluster from zero with a real budget, buy matched x86 nodes and skip the arm64 headache entirely. Homogeneous-architecture TP is simpler, every framework’s default path assumes it, and you’ll spend zero time chasing arm64 image variants. Nothing below argues against that. It argues for the situation this cluster is actually in: you own a DGX Spark, you own an existing x86 GPU box, and neither one is going away.

A few reasons that’s a real constraint, not a preference:

The DGX Spark’s hardware class isn’t available on x86. Grace-Blackwell’s unified-memory architecture — 128 GB shared between CPU and GPU at 273 GB/s — is an ARM product. There is no x86 SKU that replicates it. If you want that memory profile in the rack, arm64 comes with it; “just buy a matching x86 box” isn’t actually an option for what the Spark uniquely offers.
Standardizing the platform is the win, even when the silicon stays heterogeneous. The previous revision of this cluster ran k3s on the Spark and Talos on the x86 node specifically because Talos-on-arm64-DGX-Spark wasn’t something we’d stood up yet. Migrating the Spark onto Talos too collapses two sets of runbooks, two upgrade processes, and two security postures into one — immutable, API-managed, no SSH drift, on every node regardless of CPU vendor. That consolidation is the actual business value here, and it didn’t require making the hardware identical, just the platform.
Hardware arrives in waves. The Spark showed up as a devkit; the x86 box was already running production workloads. The competing option to pooling them wasn’t “buy a matched pair” — it was “let the Spark sit idle until next quarter’s budget.”
De-risking a future capex decision. Everything in this post — the layer-partition math, the NCCL interface pinning, the divisibility checks, the multi-arch image handling — transfers directly to a homogeneous production cluster later. Running it on hardware you already own is a cheap way to validate the architecture (and find out where Talos’s arm64/DGX-Spark support is still rough) before committing to a larger, matched buy.
Regulated and air-gapped environments. Whatever hardware passed accreditation is the hardware you have. A single hardened, immutable OS image across every architecture you’re allowed to run is often the only realistic way to keep one security posture across a mixed fleet, rather than maintaining separate compliance stories for “the ARM boxes” and “the x86 boxes.”

The common thread: in every case, the alternative isn’t a cleaner homogeneous cluster you could have had instead — it’s an idle Spark, two divergent platforms, or a procurement cycle you don’t control. If you’re free to provision fresh, identical x86 nodes, do that and skip straight to section 5. The rest of this post is for when you’re not.

1. Physical and logical topology

                      ┌───────────────────────────────┐
                      │        Flat L2/L3 network       │
                      │        (management VLAN)        │
                      └───────┬─────────────────┬───────┘
                              │                 │
                 10.0.0.10    │                 │   10.0.0.20
        ┌─────────────────────▼───┐   ┌─────────▼─────────────────┐
        │ spark-01 (DGX Spark)     │   │ x86-01                     │
        │ Talos — worker           │   │ Talos — controlplane+worker│
        │ arch: arm64              │   │ arch: amd64                 │
        │ 1× Blackwell GPU (GB10)  │   │ 2× discrete NVIDIA GPU      │
        │ 128GB unified CPU+GPU mem│   │ dedicated VRAM per GPU      │
        │ RuntimeClass: nvidia     │   │ RuntimeClass: nvidia        │
        └───────────┬──────────────┘   └────────────┬───────────────┘
                     │  hostNetwork pod: Ray WORKER   │ hostNetwork pod: Ray HEAD
                     │  vllm engine proc (PP stage 0) │ vllm serve (PP stage 1, TP=2)
                     └──────────────┬──────────────────┘
                                    │ NCCL over TCP (per-node interface pinned)
                                    ▼
                    ONE Talos-managed Kubernetes cluster
                    Ray GCS on x86-01:6379 · API on x86-01:8000

Two topology decisions worth calling out explicitly:

The controlplane role lives on x86-01, not the Spark. With only two nodes there’s no etcd quorum to speak of either way, but there’s no reason to run the cluster’s control plane on the newer, less-proven arm64/DGX-Spark Talos combination when a boring, well-trodden amd64 node is sitting right there. Keep the exotic hardware as a worker.
The Ray head also lives on x86-01, not the Spark — a change from the previous revision. With one cluster now, the head just needs to be reachable and stable; putting it on the more mature amd64 node reduces the number of “is this a Spark quirk or a real bug” debugging sessions during bring-up. The Spark runs a Ray worker and owns PP stage 0.

Because it’s a single cluster now, standard Kubernetes primitives are back on the table that weren’t available when this was two independently-managed control planes: real Service objects for discovery, a single kubectl get nodes -o wide to see both architectures at once, and gang-scheduling controllers like LeaderWorkerSet (see the callout in section 4). We still use hostNetwork: true for the Ray/vLLM pods specifically — not because we have to bridge clusters anymore, but because we want NCCL’s collective traffic on the real NIC, not the CNI overlay.

2. GPU enablement on both nodes — same OS, different architecture

Talos requires the NVIDIA driver and container toolkit as system extensions baked into the boot image (immutability means no post-boot package install) (Talos NVIDIA GPU docs). The extensions themselves are published for both amd64 and arm64 (siderolabs/extensions), so the process is identical on both nodes — but the image references and a couple of kernel details differ per architecture, so don’t copy-paste one node’s patch onto the other. Two hard requirements below came from actually crashing on this exact hardware, not from reading the docs in advance, so they’re stated more bluntly than the rest of this post.

Proprietary drivers only on spark-01 — this is not a preference. Talos ships two driver flavors: proprietary (nonfree-kmod-nvidia) and OSS/open kernel modules (nvidia-open-gpu-kernel-modules). Sidero’s own docs say plainly that Grace-Blackwell/GB10 devices require the proprietary driver — the OSS variant is not supported on this chip. If you build a schematic with nvidia-open-gpu-kernel-modules for spark-01, expect the kernel module to load “successfully” (it’ll show up fine in dmesg) while everything downstream of it — the container toolkit, CDI generation, the device plugin — behaves unpredictably, because the officially-supported combination for this hardware doesn’t exist on that path.

Build the boot image with Talos Image Factory, don’t hand-write extension image tags. An earlier draft of this section suggested patching machine.install.extensions with directly-typed ghcr.io/siderolabs/... image references and rebooting. In practice that produced a stale, wrong tag, and — more importantly — switching driver flavors (or adding/removing extensions at all) requires an actual talosctl upgrade to a new installer image, not just a machine-config patch and reboot, since the extension set is baked into the install image itself. The reliable way to get a correct, version-matched set of extensions is Talos Image Factory: pick your Talos version, select nonfree-kmod-nvidia + nvidia-container-toolkit (proprietary, matched versions — the factory UI enforces this pairing for you) for spark-01, or nonfree-kmod-nvidia/OSS-if-you-prefer + nvidia-container-toolkit for x86-01, copy the resulting installer image reference, and upgrade to it:

# 1. https://factory.talos.dev -> pick Talos version -> select extensions:
#    nonfree-kmod-nvidia + nvidia-container-toolkit (required on spark-01;
#    fine to use the same combo on x86-01 for consistency)
#    -> copy the installer image reference it generates, e.g.:
#       factory.talos.dev/installer/<schematic-id>:v1.13.x

# 2. Upgrade each node to its schematic's installer image -- this is what
#    actually swaps the baked-in extensions, patch+reboot alone will not:
talosctl upgrade -n 10.0.0.20 --image factory.talos.dev/installer/<x86-schematic-id>:v1.13.x
talosctl upgrade -n 10.0.0.10 --image factory.talos.dev/installer/<spark-schematic-id>:v1.13.x

2.1 x86-01 (amd64, controlplane+worker) — kernel modules

The extensions come from the Image Factory installer above; the machine config still needs the module list patched in separately:

# code/talos/nvidia-patch-x86.yaml — merged via talosctl patch
machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset

2.2 spark-01 (arm64, DGX Spark / GB10) — kernel modules plus `arm64.nobti`

Grace-Blackwell arm64 devices need one extra kernel argument (arm64.nobti) on top of the module list — without it the system either fails to boot cleanly or the CUDA libraries won’t load, per the same Sidero proprietary-drivers doc. DGX-Spark-on-Talos is new enough that it’s worth watching siderolabs/talos#12170 and siderolabs/talos#13019 for current compatibility status. The community microscaler/talos-dgx-spark overlay tracks DGX-Spark-specific boot and firmware quirks beyond the driver extension itself, and is worth a look if the stock installer doesn’t come up cleanly:

# code/talos/nvidia-patch-spark.yaml — merged via talosctl patch
machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset
    args:
      - arm64.nobti

Apply both, then confirm at the machine level before involving Kubernetes at all:

talosctl patch mc --patch @code/talos/nvidia-patch-x86.yaml   -n 10.0.0.20
talosctl patch mc --patch @code/talos/nvidia-patch-spark.yaml -n 10.0.0.10
talosctl reboot -n 10.0.0.20,10.0.0.10

talosctl get extensions -n 10.0.0.10   # expect nonfree-kmod-nvidia, NOT nvidia-open-gpu-kernel-modules
talosctl get extensions -n 10.0.0.20

2.3 Device plugin — one DaemonSet, and pin the version deliberately

The NVIDIA device plugin DaemonSet (code/talos/nvidia-device-plugin.yaml) gets applied once, against the single cluster, and schedules onto both nodes automatically. Don’t assume the image tag you pick actually ships an arm64 layer.

docker manifest inspect nvcr.io/nvidia/k8s-device-plugin:v0.17.4 | grep -A2 architecture
# confirm both amd64 and arm64 entries are present before you deploy

Pin v0.17.4 or newer — not v0.17.0, and this isn’t a style preference. GB10 has no discrete VRAM to query: CPU and GPU share a single unified memory pool, so the classic nvmlDeviceGetMemoryInfo() NVML call returns NVML_ERROR_NOT_SUPPORTED on spark-01. v0.17.0 treats that as fatal and crash-loops with error getting device memory: Not Supported on exactly this node — and critically, FAIL_ON_INIT_ERROR=false does not save you here, because that flag only guards NVML initialization failures, not this later per-device memory-query failure during device enumeration. v0.17.4 handles the NOT_SUPPORTED response gracefully and registers the GPU anyway. This is tracked upstream at NVIDIA/k8s-device-plugin#1482, and the same NVML behavior trips up other GPU-aware Kubernetes components on GB10 too, not just this plugin — see kubernetes-sigs/dra-driver-nvidia-gpu#1073 and NVIDIA/gpu-operator#1794 if you hit an equivalent crash somewhere else in the stack (DRA driver, GPU Operator’s own device plugin, DCGM exporter, etc.) — the fix in all of them is the same class: get to a component version released after this was patched.

kubectl apply -f code/talos/runtimeclass.yaml
kubectl apply -f code/talos/nvidia-device-plugin.yaml

kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture,GPUs:.status.allocatable.nvidia\\.com/gpu
# NAME       ARCH    GPUs
# x86-01     amd64   2
# spark-01   arm64   1

One RuntimeClass: nvidia object, applied once, covers both nodes — Talos’s recommended pattern is to leave containerd’s default runtime alone and opt individual pods in via runtimeClassName, which the Ray pods below do.

3. NCCL environment and multi-arch container images

Two separate gotchas live in this section, and it’s worth keeping them mentally distinct: one is about which network interface each node uses, the other is about which CPU architecture each node’s container image needs to match.

3.1 Per-node interface names (still true in one cluster)

Every distributed communication library used here — NCCL, Gloo, and Ray’s own transport — needs to be pinned to the correct routable interface. This does not get simpler just because there’s one cluster now: spark-01 and x86-01 are still different hardware on different NIC drivers, and will almost certainly enumerate their interfaces under different names. A shared config that hardcodes eth0 for both nodes is wrong on at least one of them, possibly both — GB10/Spark boxes in particular tend to show up under long PCIe-derived names (NVIDIA’s own DGX Spark documentation uses interfaces like enP2p1s0f1np1 for the QSFP link, not eth0).

code/nccl_env.sh (identical script, sourced by every node, deliberately does not hardcode an interface — it requires MN_IF_NAME to already be set per-node in the Pod spec, and fails loudly if it isn’t):

#!/usr/bin/env bash
# Pin every collective-comms library to the real NIC, not a container bridge.
# MN_IF_NAME is intentionally NOT hardcoded -- spark-01 and x86-01 are
# different hardware and will very likely enumerate NICs differently.
# Set it per-node in the Pod env (see ray-head.yaml / ray-worker.yaml).
set -euo pipefail

: "${MN_IF_NAME:?MN_IF_NAME is not set -- set it per-node, it will differ between spark-01 and x86-01}"

export NCCL_SOCKET_IFNAME="${MN_IF_NAME}"
export GLOO_SOCKET_IFNAME="${MN_IF_NAME}"
export TP_SOCKET_IFNAME="${MN_IF_NAME}"
export UCX_NET_DEVICES="${MN_IF_NAME}"
export OMPI_MCA_btl_tcp_if_include="${MN_IF_NAME}"

export NCCL_IB_DISABLE=1
export NCCL_NET_GDR_LEVEL=0
export NCCL_BUFFSIZE=8388608
export NCCL_NTHREADS=256
export NCCL_SOCKET_NTHREADS=4
export NCCL_NSOCKS_PERTHREAD=4
export NCCL_DEBUG="${NCCL_DEBUG:-INFO}"
export NCCL_DEBUG_SUBSYS="${NCCL_DEBUG_SUBSYS:-NET,INIT}"

# DGX Spark uses a unified memory architecture -- Ray's default memory
# monitor mis-reads this as pressure and kills workers. Disable it
# cluster-wide; it's a no-op on the x86 node.
export RAY_memory_monitor_refresh_ms=0

Find the real interface name on each node before deploying anything. On Talos there’s no shell, so this goes through talosctl, not ip addr:

talosctl get links -n 10.0.0.10 | grep -v -E 'lo|docker|cni|veth'   # spark-01 (arm64)
talosctl get links -n 10.0.0.20 | grep -v -E 'lo|docker|cni|veth'   # x86-01 (amd64)

Set the result as MN_IF_NAME in each node’s Pod spec — see section 4.

3.2 Container images have to actually run on both architectures

This is the part that’s new relative to the k3s+Talos revision: previously every workload ran on one architecture or the other, never both from the same image tag. Now vllm serve needs a container that boots on arm64 (Grace) and amd64. vLLM’s official images are published as multi-arch manifests (linux/amd64 + linux/arm64), so the same tag should resolve correctly on both nodes without any special handling on your end — but “should” is doing some work in that sentence given how new Blackwell arm64 (sm_121) support is across the CUDA/PyTorch stack. Verify before you trust it:

$ docker manifest inspect vllm/vllm-openai:v0.17.0 | grep -B2 -A3 architecture
         "digest": "sha256:29f77807c8784517f370f4fae5964a90fb2bf2473acb7608a8634fe50ab6f507",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
--
         "digest": "sha256:14ea8b431aaaf75eb873c46c8ebfbad2b4b0790d30c66126d789d8cb9bd0aab9",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }

4. Bootstrapping the single Talos cluster and the Ray cluster on top of it

4.1 Talos cluster bootstrap (one cluster, two nodes, two architectures)

talosctl gen config talos-cluster https://10.0.0.20:6443 --output-dir _out

talosctl apply-config --insecure -n 10.0.0.20 --file _out/controlplane.yaml
talosctl apply-config --insecure -n 10.0.0.10 --file _out/worker.yaml

talosctl bootstrap  -n 10.0.0.20 --talosconfig _out/talosconfig
talosctl kubeconfig -n 10.0.0.20 --talosconfig _out/talosconfig

kubectl get nodes -o wide
# x86-01     Ready    control-plane,worker   amd64
# spark-01   Ready    worker                 arm64

Apply the machine-config extension patches from section 2 either before or after bootstrap — Talos will reboot the affected node to pick up new extensions either way.

4.2 Ray head (x86-01) and Ray worker (spark-01)

Both Pods are hostNetwork: true for the NCCL traffic, scheduled with nodeSelector: kubernetes.io/arch — the standard, automatically-populated label, no custom node labels needed now that it’s one cluster. Two things about this environment specifically, both confirmed by hitting them, not just read about:

namespace-and-configmap.yaml labels the inference namespace pod-security.kubernetes.io/enforce: privileged. Without it, the default baseline PodSecurity level rejects both Pods outright — hostNetwork, hostPath, and hostPort are all disallowed under baseline/restricted, and this Pod spec needs all three.
The hf-cache hostPath lives under /var, not /srv. Talos’s host root filesystem is read-only squashfs everywhere except /var (the writable ephemeral partition). A hostPath under /srv (or almost anywhere outside /var) fails at container-creation time with mkdir /srv: read-only file system — a CreateContainerError that only shows up in kubectl describe pod’s events, not in kubectl get pods.

ray-head.yaml (x86-01):

apiVersion: v1
kind: Pod
metadata:
  name: ray-head
  namespace: inference
  labels: { role: ray-head }
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  runtimeClassName: nvidia
  nodeSelector:
    kubernetes.io/arch: amd64
  containers:
    - name: ray-head
      image: vllm/vllm-openai:v0.17.0
      command: ["/bin/bash", "-c"]
      args:
        - |
          set -euo pipefail
          source /etc/nccl/nccl_env.sh
          export VLLM_HOST_IP="${POD_IP}"
          ray start --head \
            --port=6379 \
            --dashboard-host=0.0.0.0 \
            --num-gpus=1 \
            --block
      env:
        - name: POD_IP
          valueFrom: { fieldRef: { fieldPath: status.podIP } }
        - name: MN_IF_NAME
          value: "enp3s0"   # replace with x86-01's real interface, see 3.1
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom: { secretKeyRef: { name: hf-token, key: token } }
      envFrom:
        - configMapRef: { name: hf-cache-env }
      resources:
        limits: { nvidia.com/gpu: 2 }
      volumeMounts:
        - { name: nccl-env, mountPath: /etc/nccl }
        - { name: shm, mountPath: /dev/shm }
        - { name: hf-cache, mountPath: /root/.cache/huggingface }
      ports:
        - { containerPort: 6379, name: gcs }
        - { containerPort: 8265, name: dashboard }
        - { containerPort: 8000, name: http }
  volumes:
    - { name: nccl-env, configMap: { name: nccl-env, defaultMode: 0o755 } }
    - { name: shm, emptyDir: { medium: Memory, sizeLimit: 16Gi } }
    - { name: hf-cache, hostPath: { path: /var/lib/hf-cache, type: DirectoryOrCreate } }

ray-worker.yaml (spark-01) — note the different nodeSelector and, if you went with option 1 from section 3.2, a different image:

apiVersion: v1
kind: Pod
metadata:
  name: ray-worker
  namespace: inference
  labels: { role: ray-worker }
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  runtimeClassName: nvidia
  nodeSelector:
    kubernetes.io/arch: arm64
  containers:
    - name: ray-worker
      image: vllm/vllm-openai:v0.17.0   # or your arm64/Spark-specific image, see 3.2
      command: ["/bin/bash", "-c"]
      args:
        - |
          set -euo pipefail
          source /etc/nccl/nccl_env.sh
          export VLLM_HOST_IP="${POD_IP}"
          ray start \
            --address="${RAY_HEAD_IP}:6379" \
            --num-gpus=1 \
            --block
      env:
        - name: POD_IP
          valueFrom: { fieldRef: { fieldPath: status.podIP } }
        - name: RAY_HEAD_IP
          value: "10.0.0.20"   # x86-01
        - name: MN_IF_NAME
          value: "enP2p1s0f1np1"   # replace with spark-01's real interface, see 3.1
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom: { secretKeyRef: { name: hf-token, key: token } }
      envFrom:
        - configMapRef: { name: hf-cache-env }
      resources:
        limits: { nvidia.com/gpu: 1 }
      volumeMounts:
        - { name: nccl-env, mountPath: /etc/nccl }
        - { name: shm, mountPath: /dev/shm }
        - { name: hf-cache, mountPath: /root/.cache/huggingface }
  volumes:
    - { name: nccl-env, configMap: { name: nccl-env, defaultMode: 0o755 } }
    - { name: shm, emptyDir: { medium: Memory, sizeLimit: 16Gi } }
    - { name: hf-cache, hostPath: { path: /var/lib/hf-cache, type: DirectoryOrCreate } }

Why ray-head.yaml advertises --num-gpus=1 to Ray even though x86-01 has 2 physical GPUs mounted (resources.limits.nvidia.com/gpu: 2 in the same manifest). This is deliberate, and getting it backwards is the easiest way to silently run this entire “multi-node” deployment on a single node without any error telling you so. Ray’s placement-group scheduler defaults to packing a group onto as few nodes as it can. Section 5’s launch only needs 2 total GPU workers (PP=2, TP=1) — if the head node advertises 2 logical GPUs, Ray will happily satisfy both workers from x86-01 alone and never touch spark-01, while ray status still reports a perfectly healthy 2.0/2.0 GPU in use. Nothing errors, nothing warns, and the curl test in section 6 still returns real completions — the only visible symptom is nvidia-smi on spark-01 sitting at 0% util for the entire run. Capping the head’s Ray-visible count to 1 forces the second placement-group slot onto spark-01. x86-01’s second physical GPU stays mounted and visible inside the container (nvidia.com/gpu: 2 in resources.limits, so it’s there for a future --tensor-parallel-size 2 stage or a second concurrent deployment) — it’s just deliberately withheld from this job’s scheduler.

kubectl apply -f code/talos/namespace-and-configmap.yaml
kubectl apply -f code/talos/ray-head.yaml
kubectl apply -f code/talos/ray-worker.yaml

kubectl exec -it ray-head -n inference -- ray status

Expect a total pool of 2.0 GPU in the Resources section — 1 from x86-01, 1 from spark-01 — with usage at 0.0/2.0 until section 5’s job actually runs, one kubectl, one cluster. If usage instead climbs to 2.0/2.0 while nvidia-smi on spark-01 never leaves 0%, you’ve hit the packing issue above, not a networking or scheduling bug.

Going further: now that both nodes are under one control plane, LeaderWorkerSet becomes a real option for gang scheduling this pair (start/stop together, automatic restart of the whole group on failure) instead of two independently-applied Pods. It wasn’t viable in the previous k3s+Talos revision because it assumes a single cluster. We’re keeping plain Pods in this post for clarity — the mechanics of PP/TP placement are the same either way — but if you’re operationalizing this beyond a one-off, look at LeaderWorkerSet next.

5. The main event: uneven pipeline parallelism (works today, no patches)

The topology changed; the parallelism strategy didn’t. We are not going to ask vLLM for --tensor-parallel-size 3. Instead:

--pipeline-parallel-size 2 — one stage per physical node, crossing the TCP link exactly once per forward pass (activations only, not full-layer all-reduce).
--tensor-parallel-size 2 inside the x86-01 stage, since its two GPUs are matched, NVLink/PCIe-connected, and TP’s per-layer all-reduce never has to leave the box.
spark-01’s stage runs at TP=1, because it’s a single physical device — and because vLLM requires the same TP degree at every pipeline stage in a single engine today (vLLM #27239 tracks lifting this), the real launch below uses pipeline_parallel_size=2 with tensor_parallel_size=1 everywhere, exploiting the Spark’s outsized memory through an uneven layer partition instead — giving spark-01’s stage more of the model since it’s carrying 128 GB against a single x86 GPU’s much smaller VRAM. Because this launch only needs 2 total GPU workers, ray-head.yaml (section 4.2) deliberately advertises just 1 of x86-01’s 2 physical GPUs to Ray — otherwise Ray’s scheduler packs both workers onto x86-01 alone and spark-01 never gets touched, with no error to tell you so.

#!/usr/bin/env bash
set -euo pipefail

MODEL="meta-llama/Llama-3.2-3B-Instruct"

export VLLM_PP_LAYER_PARTITION="25,3"

ray job submit --address=http://10.0.0.20:8265 -- \
  vllm serve "${MODEL}" \
    --host 0.0.0.0 --port 8000 \
    --pipeline-parallel-size 2 \
    --tensor-parallel-size 1 \
    --distributed-executor-backend ray \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.75 \
    --trust-remote-code

6. Benchmarking the cluster

benchmark.py drives the OpenAI-compatible endpoint with concurrent requests and reports throughput/latency — dependency-light (httpx + stdlib), unchanged from the previous revision except the default --url now points at x86-01 where the Ray head (and the API server) live. The numbers below are from the live meta-llama/Llama-3.2-3B-Instruct validation run referenced in the revision note up top, not the 70B target — swap --model for your actual checkpoint once you’ve re-run estimate_pp_split.py against it:

python code/benchmark.py --url http://10.0.0.20:8000 \
  --model meta-llama/Llama-3.2-3B-Instruct --concurrency 8 --total 64

requests=64 concurrency=8 wall_s=21.20
throughput_req_s=3.019
throughput_tok_s=371.40
latency_s  p50=2.57 p95=3.06 max=3.06

Read this as confirmation the pipeline works correctly, not as a performance number to chase. p50/p95/max sitting within half a second of each other across all 64 requests means no long-tail stalls and no silent retries — a healthy, stable deployment. But 371 tok/s aggregate at concurrency 8 is modest for a 3B model, and the reason is structural, not a bug: every decode step now pays a real network round-trip crossing the PP boundary between x86-01 and spark-01 over plain TCP (section 3.1), and a 3B model’s per-layer compute is small enough that this round-trip isn’t well hidden behind it the way it would be for a 70B-class checkpoint doing far more work per layer. Cross-node PP amortizes communication cost against compute cost — the smaller the model, the worse that trade works out, and a 3B model has no real reason to be split across two nodes in the first place (it fits comfortably in a single x86 GPU’s VRAM on its own). This run validates that the mechanics — the uneven layer partition, the placement-group fix in section 4.2, NCCL actually crossing the node boundary — are wired correctly. It is not evidence of what to expect at the scale this post’s argument is really about; expect a meaningfully better tokens-per-second-per-network-hop ratio once this points at a 70B-class checkpoint.

What to watch while this runs:

kubectl exec ray-head -n inference -- nvidia-smi and kubectl exec ray-worker -n inference -- nvidia-smi — confirm memory is actually split per your VLLM_PP_LAYER_PARTITION, not mirrored across every device (mirrored weights are the signature of an accidentally-replicated, not sharded, deployment). Note spark-01’s nvidia-smi will never show a process entry or per-process memory for the GB10, even under real load — its unified-memory NVML path doesn’t support that query (same root cause as the device-plugin bug in section 2.3). GPU-Util and power state (P0 active vs P8 idle) are the reliable liveness signals on that node, not the process table.
ray status --address=http://10.0.0.20:8265 for GPU utilization skew between stages — heavy skew means your layer partition doesn’t match the real compute-per-layer ratio between the Spark and the x86 GPU; re-run estimate_pp_split.py with updated numbers.
Per-token latency should scale roughly with 1/PP_size for the cross-node hop cost (one activation transfer per micro-batch) rather than the O(layers) all-reduce cost you’d pay under flat TP.

7. The road not taken: flat TP=3, and exactly where it breaks

Many 70B-class checkpoints use 64 attention heads, and 64 % 3 != 0 — the engine refuses to start. That’s failure mode one: loud, at init, not silent at inference time.

Failure mode two is the one worth internalizing even on a checkpoint that happens to divide evenly: symmetric TP forces every rank’s shard to be the same size, so the maximum model-plus-KV-cache footprint the whole 3-GPU group can hold is 3 × min(single_GPU_VRAM), not 128 GB + 2 × VRAM. If the x86 GPUs have, say, 24 GB each, a flat tp_size=3 engine can use at most 3 × 24 GB = 72 GB of aggregate weight+cache space — the Spark’s other ~104 GB of unified memory sits idle, because TP has no mechanism for giving one rank a bigger shard than its peers. That’s the concrete cost of forcing a heterogeneous cluster into a homogeneous-TP shape, arm64 or not.

8. Troubleshooting notes from actually running this

The first three of these are confirmed failures from actually bringing this cluster up, in the order they happened, not hypothetical ones — worth checking first if you’re following along and hit a wall on the device plugin specifically.

Device plugin crash-loops with error getting device memory: Not Supported, FAIL_ON_INIT_ERROR=false notwithstanding. This is NVIDIA/k8s-device-plugin#1482 — GB10 has no discrete VRAM, nvmlDeviceGetMemoryInfo() returns NOT_SUPPORTED, and versions before v0.17.4 treat that as fatal during device enumeration (a different code path than the one FAIL_ON_INIT_ERROR guards). Pin v0.17.4 or newer. Same underlying NVML behavior affects other GPU-aware components on this chip — see kubernetes-sigs/dra-driver-nvidia-gpu#1073 and NVIDIA/gpu-operator#1794 if it shows up elsewhere in your stack.
Failed to create pod sandbox: runtimeclass.node.k8s.io "nvidia" not found. The RuntimeClass object itself was never applied — kubectl apply -f code/talos/runtimeclass.yaml. This is a plain Kubernetes API resource, unrelated to which NVIDIA driver flavor is installed; every pod using runtimeClassName: nvidia (device plugin, Ray head/worker) is blocked until it exists.
Error from server (Forbidden): ... violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true), hostPath volumes ..., hostPort .... The inference namespace is enforcing the default baseline PodSecurity level, which rejects hostNetwork, hostPath, and hostPort outright — all three of which ray-head.yaml/ray-worker.yaml need. namespace-and-configmap.yaml in this repo now labels the namespace pod-security.kubernetes.io/enforce: privileged; if you’re re-applying against a namespace created before that label was added, kubectl label --overwrite ns inference pod-security.kubernetes.io/enforce=privileged fixes it without needing to delete/recreate the namespace.
Pod stuck CreateContainerError, kubectl describe pod events show mkdir /srv: read-only file system (or any path outside /var). A hostPath volume points somewhere other than /var. Talos’s host root is immutable, read-only squashfs everywhere except the /var ephemeral partition — hostPath targets have to live under /var (e.g. /var/lib/hf-cache), not /srv, /opt, or anywhere else that would be writable on a normal distro. This only surfaces in pod events, not kubectl get pods status, so it’s easy to miss if you’re only glancing at STATUS.
Kernel module loads fine (dmesg shows NVRM: loading NVIDIA UNIX Open Kernel Module... succeeding), but nothing downstream works. Check which extension is actually installed: talosctl get extensions -n 10.0.0.10 should say nonfree-kmod-nvidia. If it says nvidia-open-gpu-kernel-modules (OSS), that’s the problem — GB10 requires the proprietary driver per Sidero’s own docs, and the OSS variant loading “successfully” at the kernel level is not the same as it working. See section 2 for the Image-Factory-based fix; a talosctl patch mc + reboot alone will not swap driver flavors.
kubectl get nodes only shows one architecture. Confirm both talosctl apply-config calls targeted the right node IPs and that spark-01 actually rebooted into the extension-patched image — talosctl get extensions -n 10.0.0.10 should list the nvidia driver/toolkit; if it doesn’t, the machine config patch didn’t land before the node joined.
ImagePullBackOff on spark-01 only. The image tag doesn’t actually publish an arm64 manifest, or publishes one without Blackwell (sm_121) support compiled in. Re-run docker manifest inspect from section 3.2 before assuming the tag is multi-arch just because it works on x86-01.
ray status shows only 2 GPUs after the worker pod starts. Almost always spark-01’s Ray worker pod failing to start or failing to reach the head — check kubectl logs ray-worker -n inference before assuming a GPU-detection issue; confirm 10.0.0.10 can reach 10.0.0.20:6379 (nc -zv 10.0.0.20 6379 from inside the worker pod).
ray status reports a fully healthy GPU count and curl returns real completions, but one node’s nvidia-smi never leaves 0% util. Not a networking or scheduling failure — Ray’s placement-group scheduler packs a job onto as few nodes as it can satisfy. If a node advertises more logical GPUs (ray start --num-gpus=...) than the pipeline strictly needs to leave for other nodes, the whole job can land on that one node while ray status still reports the resource as fully “used.” Check ray status’s per-node breakdown, or just nvidia-smi on every node directly, rather than trusting the aggregate count. Fix by capping --num-gpus on the over-provisioned node so the placement group is forced to reach the other one — see the callout in section 4.2, which is exactly this bug.
NameResolutionError / Temporary failure in name resolution fetching the Hugging Face file list at startup, then it recovers a few seconds later on its own. vLLM’s loader retries the bulk tree/main API call up to twice, then falls back to downloading each allow_patterns entry individually — either path tolerates a transient DNS hiccup. Confirm CoreDNS itself is healthy (kubectl get pods -n kube-system -l k8s-app=kube-dns) and that kubectl exec <pod> -n inference -- python3 -c "import socket; print(socket.gethostbyname('huggingface.co'))" succeeds (note: these images generally don’t ship nslookup or getent-friendly tooling — use Python, it’s always present). If DNS resolves fine on manual check and the logs show the per-pattern fallback picking up and weights actually downloading, this was a momentary blip, not a config problem — let it retry rather than chasing it.
nvidia.com/gpu: 0 in a node’s Capacity/Allocatable (kubectl describe node) after deleting/recreating a Pod that held GPU resources. The device plugin lost its registration with kubelet — unrelated to the driver/extension setup in section 2, and different from the v0.17.0 crash-loop bug in 2.3. Check kubectl get pods -n kube-system -o wide | grep -i nvidia for a crashlooping or stale device-plugin pod on that node and kubectl delete it to force re-registration; capacity should repopulate within seconds. Worth a ray status re-check afterward too, since the new pod may re-schedule into a different placement than before.
NCCL hangs at init with no error. Check MN_IF_NAME actually matches an interface that exists in the container’s network namespace on that specific node — spark-01 and x86-01 will not have the same interface name; re-run the talosctl get links discovery from section 3.1 rather than assuming the value copied from the other node’s manifest is correct.
Engine starts but memory usage is mirrored across all devices, not sharded. You launched with --tensor-parallel-size equal to total GPU count without --pipeline-parallel-size, or --distributed-executor-backend fell back to a single-process mode. Confirm --distributed-executor-backend ray is set and ray status shows the expected 3-GPU, 2-node count before vllm serve runs.
/dev/shm OOM during weight loading. Bump the emptyDir: {medium: Memory} sizeLimit — 16 Gi is a floor, not a target, for 70B-class checkpoints; NCCL’s inter-node buffers on TCP transport are shared-memory-hungry.
Uneven GPU utilization between the two pipeline stages. Re-run estimate_pp_split.py — this almost always means the layer partition doesn’t match actual per-device throughput, not a networking problem.
Talos won’t boot cleanly on the DGX Spark at all. Confirm arm64.nobti is actually present in the applied machine config (talosctl get meta -n 10.0.0.10 or inspect the rendered config), and check the microscaler/talos-dgx-spark overlay and the open siderolabs/talos DGX Spark issues for anything specific to your Talos version — this integration is new enough that “which Talos release” matters more than usual.

Summary

Standardizing on Talos across both nodes solved the problem the previous revision of this post spent most of its length on — two independent control planes bridged by hand at the network layer — and replaced it with a narrower, more contained one: one cluster, two CPU architectures, and container images that have to actually work on both. The core inference-engineering problem is exactly where it was before: the Spark and the x86 node are not peers, one is a single 128 GB unified-memory die and the other is two matched discrete GPUs, and flat tensor parallelism still throws away the Spark’s memory advantage to match its smallest neighbor. Pipeline parallelism, split unevenly across the two physical nodes and paired with tensor parallelism confined to the one place it’s actually symmetric — inside the x86 box — remains the right shape for the hardware you actually have, now with one kubectl instead of two.

Sources referenced throughout: NVIDIA DGX Spark product page, Tom’s Hardware — GB10 Superchip deep dive, Sidero Docs — NVIDIA GPU (Proprietary drivers), siderolabs/extensions, siderolabs/talos issue #12170 — DGX Spark support, siderolabs/talos issue #13019 — Talos 1.13.0-beta on DGX Spark arm64, microscaler/talos-dgx-spark, NVIDIA/k8s-device-plugin issue #297 — arm64 support history, vLLM Parallelism and Scaling docs, vLLM issue #27239 — Heterogeneous TP per Pipeline Stage, Kubernetes Recipes — Inter-Node Tensor Parallelism on Kubernetes.