Zero-Copy RAG: Leveraging Unified Memory for Vector Search on the GB10

I started writing this article with a clear thesis: the GB10’s coherent unified memory eliminates the PCIe tax that x86+discrete-GPU systems pay on every RAG query, and I was going to measure exactly how much that saves you. I built a single-node K3s pipeline with Qdrant, BGE-small, and an 8B-parameter NVFP4 generator, all sharing the GB10’s 128 GB LPDDR5X pool. I built a “staging” configuration that simulates what a PCIe-attached GPU would have to do. I ran 100 queries against each.

The configurations came out within 1 ms of each other.

That is not what I expected, and the obvious conclusion — that the PCIe tax is imaginary — is wrong. The real conclusion is that my simulation didn’t simulate what I claimed it simulated, and I had no honest way to measure the actual PCIe overhead on a machine that has no PCIe bus.

This article documents what I actually built, what I actually measured, and where the architectural claims about coherent memory still hold up despite my benchmark not being able to prove them.

Many of the RAG pipeline has a standard recipe

The embedding model ships a query vector to the GPU.
The GPU computes it. The GPU sends the vector back to the CPU.
The CPU hands it to the vector database
The database returns neighbor IDs and payloads.
The CPU packages the retrieved text.
The LLM receives it. The LLM copies the prompt — tokenized context plus retrieved chunks — back to the GPU for prefill.

Count the hops. On an x86 box with a PCIe-attached GPU, at least four of them cross the PCIe bus. Small vectors, small payloads, latency-sensitive — never comes close to saturating it. What you feel instead is the per-transfer overhead: DMA setup, pinned-memory staging, driver synchronization, and the inevitable `cudaMemcpy` calls that show up as flat floors in your flame graphs.

This is the PCIe tax. On the GB10 Grace Blackwell Superchip, you don’t pay it. The GB10’s CPU and GPU dies share a single coherent pool of 128 GB of LPDDR5X, connected via NVLink-C2C at roughly five times PCIe Gen5 bandwidth. The GPU does not have its own VRAM. It talks to memory through controllers on the CPU die. That means a vector sitting in Qdrant’s mmap’d storage is, from the GPU’s perspective, already there. No staging. No migration. No copy.

I wanted to know how much that actually buys you on a real RAG pipeline. So I built one on my single-node K3s cluster, pinned the hot paths, profiled the cold paths, and measured Time to First Token (TTFT) against the only honest baseline I could think of: the same pipeline, same models, same data, but with the embeddings and the model weights living in deliberately non-coherent memory regions. The numbers surprised me in one direction and disappointed me in another. This article documents that build end to end, including the parts that broke.

What “Zero-Copy” Actually Means on Grace Blackwell

Before I write a single benchmark number, I need to be precise about what the GB10’s memory model actually does, because there is a lot of marketing noise. There are three distinct things people call “unified memory” in CUDA:

UVA (Unified Virtual Addressing): Every pointer is valid on every processor. This has been around since Kepler. It does not mean data is co-located — it just means the pointer arithmetic works.
Managed memory via `cudaMallocManaged: The driver migrates pages on demand between CPU DRAM and GPU VRAM. On a discrete GPU, a GPU access to a CPU-resident page still triggers a page fault and a PCIe transfer. The transfer is automatic; it is not free.
Hardware-coherent unified memory with ATS: On Grace Hopper and Grace Blackwell systems, the CPU and GPU share a single page table. The GPU accesses CPU-allocated memory at cache-line granularity over NVLink-C2C, with no page migration, no fault, and no driver involvement on the data path. This is what the NVIDIA docs call “full unified memory with hardware coherency,” and it is qualitatively different from the other two.

    nvidia-smi -q | grep -i addressing
    Addressing Mode                   : ATS

ATS — Address Translation Services — is the relevant detail. It means the GPU’s MMU consults the CPU’s page tables directly over NVLink-C2C. A malloc’d buffer on the Grace side is immediately, coherently readable from the Blackwell side without any CUDA API calls. `cudaMemcpy(dst, src, n, cudaMemcpyHostToDevice)` on this hardware is a hint to the driver at best and often a straight memory-to-memory copy within the same physical DRAM pool at worst.

This is the foundation everything else in this article rests on. If you are coming from a PCIe-GPU mental model, the most important adjustment to make is this: “host memory” and “device memory” are the same memory. The bandwidth number that matters is not PCIe Gen5; it is the 273 GB/s that LPDDR5X delivers to the unified pool, shared between CPU and GPU.

One nuance worth flagging: 273 GB/s is *aggregate* bandwidth. The CPU and GPU contend for it. If your CPU is running Qdrant’s HNSW graph traversal — which is memory-bandwidth-bound — while your GPU is running vLLM prefill — also memory-bandwidth-bound — you are splitting the same pipe. I will return to this when I explain the benchmark numbers.

Architecture: One Memory, Three Processes

The pipeline I built has three components, all running as pods on my single-node K3s cluster:

bge-embedder — a small embedding server running `BAAI/bge-small-en-v1.5` on vLLM’s embedding backend. Produces 384-dimensional query vectors. Pinned to the GPU.
qdrant — a single-node Qdrant deployment with on-disk storage backed by a `local-path` PVC. Serves HNSW vector search over a corpus of 250k chunked Wikipedia passages, each with a 384-dimension embedding. Pinned to the Grace CPU cores.
vllm-generator — a vLLM server running `Qwen/Qwen2.5-7B-Instruct`. Handles the generation half of RAG. Pinned to the GPU.

A thin Python orchestrator ties them together: embed → search Qdrant → stuff retrieved passages into a prompt template → send to vLLM → stream tokens back. On paper, this is a completely standard RAG topology. The only unusual thing is that all three processes share a single physical memory pool. Qdrant’s 250k-vector index (about 380 MB of dense vectors plus ~200 MB of HNSW graph) lives in the same LPDDR5X that holds Qwen2.5-7B’s ~4 GB of NVFP4 weights and the KV cache. When vLLM does prefill on a prompt that includes retrieved chunks, it is reading those chunks — as UTF-8 bytes — directly from pages that Qdrant’s Rust process wrote. No kernel copy. No network hop for the retrieval itself (Qdrant and vLLM talk over HTTP on localhost, which does incur a kernel-level socket copy, but the payload is tiny).

The interesting question is whether any of this actually matters for TTFT. My hypothesis going in was: probably yes for the retrieval step itself, marginally no for the generation step, and the surprise will be somewhere I didn’t expect.

Here are the manifests I created.

00-namespace.yaml

			
piVersion: v1
kind: Namespace
metadata:
  name: zerocopy-rag
  labels:
    app.kubernetes.io/part-of: gb10-ml-series

		

10-qdrant.yaml

			
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: qdrant-config
  namespace: zerocopy-rag
data:
  production.yaml: |
    log_level: INFO
    storage:
      storage_path: /qdrant/storage
      snapshots_path: /qdrant/snapshots
      on_disk_payload: false
      performance:
        max_search_threads: 8
        max_optimization_threads: 4
      hnsw_index:
        m: 16
        ef_construct: 128
        max_indexing_threads: 4
      optimizers:
        deleted_threshold: 0.2
        vacuum_min_vector_number: 1000
        default_segment_number: 4
        flush_interval_sec: 5
    service:
      host: 0.0.0.0
      http_port: 6333
      grpc_port: 6334
      enable_cors: true
    telemetry_disabled: true
---
apiVersion: v1
kind: Service
metadata:
  name: qdrant
  namespace: zerocopy-rag
  labels:
    app: qdrant
spec:
  selector:
    app: qdrant
  ports:
    - name: http
      port: 6333
      targetPort: 6333
    - name: grpc
      port: 6334
      targetPort: 6334
  type: ClusterIP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: qdrant
  namespace: zerocopy-rag
  labels:
    app: qdrant
spec:
  serviceName: qdrant
  replicas: 1
  selector:
    matchLabels:
      app: qdrant
  template:
    metadata:
      labels:
        app: qdrant
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "6333"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 6333
            - name: grpc
              containerPort: 6334
          env:
            - name: QDRANT__SERVICE__API_KEY
              valueFrom:
                secretKeyRef:
                  name: qdrant-api-key
                  key: api-key
                  optional: true
            - name: RUST_LOG
              value: "info,qdrant=info,collection=info"
          resources:
            requests:
              cpu: "8"
              memory: "16Gi"
            limits:
              cpu: "8"
              memory: "16Gi"
          volumeMounts:
            - name: storage
              mountPath: /qdrant/storage
            - name: snapshots
              mountPath: /qdrant/snapshots
            - name: config
              mountPath: /qdrant/config/production.yaml
              subPath: production.yaml
          readinessProbe:
            httpGet:
              path: /readyz
              port: 6333
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /livez
              port: 6333
            initialDelaySeconds: 30
            periodSeconds: 30
      volumes:
        - name: config
          configMap:
            name: qdrant-config
  volumeClaimTemplates:
    - metadata:
        name: storage
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: local-path
        resources:
          requests:
            storage: 50Gi
    - metadata:
        name: snapshots
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: local-path
        resources:
          requests:
            storage: 20Gi

		

20-bge-embedder.yaml

			
---
apiVersion: v1
kind: Service
metadata:
  name: bge-embedder
  namespace: zerocopy-rag
  labels:
    app: bge-embedder
spec:
  selector:
    app: bge-embedder
  ports:
    - name: http
      port: 8000
      targetPort: 8000
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bge-embedder
  namespace: zerocopy-rag
  labels:
    app: bge-embedder
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bge-embedder
  template:
    metadata:
      labels:
        app: bge-embedder
    spec:
      containers:
        - name: vllm
          image: nvcr.io/nvidia/vllm:26.03.post1-py3
          imagePullPolicy: IfNotPresent
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
            - "--model=BAAI/bge-small-en-v1.5"
            - "--host=0.0.0.0"
            - "--port=8000"
            - "--gpu-memory-utilization=0.05"
            - "--max-model-len=512"
            - "--dtype=float16"
          ports:
            - name: http
              containerPort: 8000
          env:
            - name: HF_HOME
              value: /cache/hf
            - name: VLLM_WORKER_MULTIPROC_METHOD
              value: spawn
          resources:
            requests:
              cpu: "2"
              memory: "4Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "2"
              memory: "4Gi"
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: hf-cache
              mountPath: /cache/hf
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 180
            periodSeconds: 30
      volumes:
        - name: hf-cache
          persistentVolumeClaim:
            claimName: hf-cache
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hf-cache
  namespace: zerocopy-rag
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 50Gi

		

30-vllm-generator.yaml

			
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-generator
  namespace: zerocopy-rag
  labels:
    app: vllm-generator
spec:
  selector:
    app: vllm-generator
  ports:
    - name: http
      port: 8000
      targetPort: 8000
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-generator
  namespace: zerocopy-rag
  labels:
    app: vllm-generator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-generator
  template:
    metadata:
      labels:
        app: vllm-generator
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: vllm
          image: nvcr.io/nvidia/vllm:26.03.post1-py3
          imagePullPolicy: IfNotPresent
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
            - "--model=Qwen/Qwen2.5-7B-Instruct"
            - "--host=0.0.0.0"
            - "--port=8000"
            - "--gpu-memory-utilization=0.45"
            - "--max-model-len=8192"
            # Prefix caching DISABLED for the benchmark in the article.
            # Turn this back ON for production: --enable-prefix-caching
            # Expect ~30-40% TTFT improvement on shared-prefix prompts.
          ports:
            - name: http
              containerPort: 8000
          env:
            - name: HF_HOME
              value: /cache/hf
            - name: VLLM_WORKER_MULTIPROC_METHOD
              value: spawn
          resources:
            requests:
              cpu: "4"
              memory: "8Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "4"
              memory: "8Gi"
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: hf-cache
              mountPath: /cache/hf
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10
            timeoutSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 300
            periodSeconds: 30
            timeoutSeconds: 10
      volumes:
        - name: hf-cache
          persistentVolumeClaim:
            claimName: hf-cache

		

40-rag-orchestrator.yaml

			
---
# RAG orchestrator: embed → search Qdrant → assemble prompt → generate.
# Pinned to efficiency cores; lightweight asyncio service.
# Exposes Prometheus metrics for end-to-end TTFT with per-stage breakdown.
apiVersion: v1
kind: ConfigMap
metadata:
  name: orchestrator-code
  namespace: zerocopy-rag
data:
  orchestrator.py: |
    # Minimal RAG orchestrator. See scripts/orchestrator.py in the tarball
    # for the full implementation with staging-mode toggles and MS MARCO
    # evaluation harness.
    import os
    import time
    import httpx
    from fastapi import FastAPI, Request
    from fastapi.responses import StreamingResponse
    from prometheus_client import Histogram, make_asgi_app
    EMBED_URL = os.environ["EMBED_URL"]
    QDRANT_URL = os.environ["QDRANT_URL"]
    VLLM_URL = os.environ["VLLM_URL"]
    COLLECTION = os.environ.get("QDRANT_COLLECTION", "wiki_250k")
    TOP_K = int(os.environ.get("TOP_K", "5"))
    app = FastAPI()
    app.mount("/metrics", make_asgi_app())
    H_EMBED = Histogram("rag_embed_seconds", "embedding latency",
                        buckets=(0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2))
    H_SEARCH = Histogram("rag_search_seconds", "qdrant search latency",
                         buckets=(0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2))
    H_TTFT = Histogram("rag_ttft_seconds", "end-to-end time to first token",
                       buckets=(0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.5, 1.0))
    client = httpx.AsyncClient(timeout=60.0)
    @app.post("/rag")
    async def rag(req: Request):
        body = await req.json()
        query = body["query"]
        t0 = time.perf_counter()
        # Stage 1: embed
        er = await client.post(f"{EMBED_URL}/v1/embeddings",
                               json={"model": "BAAI/bge-small-en-v1.5",
                                     "input": [query]})
        vec = er.json()["data"][0]["embedding"]
        t1 = time.perf_counter()
        H_EMBED.observe(t1 - t0)
        # Stage 2: search Qdrant
        sr = await client.post(
            f"{QDRANT_URL}/collections/{COLLECTION}/points/search",
            json={"vector": vec, "limit": TOP_K, "with_payload": True})
        hits = sr.json()["result"]
        t2 = time.perf_counter()
        H_SEARCH.observe(t2 - t1)
        # Stage 3: assemble prompt
        ctx = "\n\n".join(
            f"[{i+1}] {h['payload']['text']}" for i, h in enumerate(hits))
        messages = [
            {"role": "system",
             "content": "Answer using only the provided passages. "
                        "Cite passage numbers."},
            {"role": "user",
             "content": f"Passages:\n{ctx}\n\nQuestion: {query}"},
        ]
        # Stage 4: stream generation
        async def stream():
            first_token_seen = False
            async with client.stream(
                    "POST", f"{VLLM_URL}/v1/chat/completions",
                    json={"model": "Qwen/Qwen2.5-7B-Instruct",
                          "messages": messages, "stream": True,
                          "max_tokens": 512}) as r:
                async for line in r.aiter_lines():
                    if line.startswith("data:"):
                        if not first_token_seen:
                            first_token_seen = True
                            H_TTFT.observe(time.perf_counter() - t0)
                        yield line + "\n"
        return StreamingResponse(stream(), media_type="text/event-stream")
    @app.get("/health")
    def health():
        return {"ok": True}
---
apiVersion: v1
kind: Service
metadata:
  name: rag-orchestrator
  namespace: zerocopy-rag
spec:
  selector:
    app: rag-orchestrator
  ports:
    - name: http
      port: 8080
      targetPort: 8080
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-orchestrator
  namespace: zerocopy-rag
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rag-orchestrator
  template:
    metadata:
      labels:
        app: rag-orchestrator
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: orchestrator
          # Public multi-arch python image; the code is tiny and pure-python.
          image: python:3.11-slim
          command: ["/bin/sh", "-c"]
          args:
            - |
              pip install --no-cache-dir fastapi uvicorn httpx prometheus-client && \
              exec uvicorn orchestrator:app --host 0.0.0.0 --port 8080 --app-dir /app
          env:
            - name: EMBED_URL
              value: "http://bge-embedder:8000"
            - name: QDRANT_URL
              value: "http://qdrant:6333"
            - name: VLLM_URL
              value: "http://vllm-generator:8000"
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "2"
              memory: "512Mi"
            limits:
              cpu: "2"
              memory: "512Mi"
          volumeMounts:
            - name: code
              mountPath: /app
      volumes:
        - name: code
          configMap:
            name: orchestrator-code
            items:
              - key: orchestrator.py
                path: orchestrator.py

		

benchmarking script

			
#!/usr/bin/env python3
"""
Time to First Token benchmark for the zero-copy RAG pipeline.
Modes (matching the three configurations in the blog post):
    native   — straight through the orchestrator (Config A)
    staging  — orchestrator + artificial staging-copy overhead (Config B)
Draws queries from MS MARCO dev. Runs N queries, reports median and p95
TTFT along with per-stage breakdown.
Usage:
    python3 bench_ttft.py --config native --n 200
    python3 bench_ttft.py --config staging --n 200
"""
from __future__ import annotations
import argparse
import json
import os
import statistics
import sys
import time
from dataclasses import dataclass, field
import httpx
@dataclass
class StageTimings:
    embed_ms: float
    search_ms: float
    stage_overhead_ms: float
    ttft_ms: float
@dataclass
class Result:
    timings: list[StageTimings] = field(default_factory=list)
    def summarize(self) -> dict:
        def pct(xs, p):
            xs_sorted = sorted(xs)
            k = int(len(xs_sorted) * p / 100)
            return xs_sorted[min(k, len(xs_sorted) - 1)]
        ttfts = [t.ttft_ms for t in self.timings]
        embeds = [t.embed_ms for t in self.timings]
        searches = [t.search_ms for t in self.timings]
        return {
            "n": len(self.timings),
            "ttft_p50_ms": statistics.median(ttfts),
            "ttft_p95_ms": pct(ttfts, 95),
            "ttft_p99_ms": pct(ttfts, 99),
            "embed_p50_ms": statistics.median(embeds) if any(embeds) else 0,
            "search_p50_ms": statistics.median(searches) if any(searches) else 0,
        }
def load_queries(path: str, n: int) -> list[str]:
    if not os.path.exists(path):
        print(f"Query file {path} not found. Fetching from HuggingFace MS MARCO...",
              file=sys.stderr)
        from datasets import load_dataset
        ds = load_dataset("ms_marco", "v2.1", split="validation", streaming=True)
        qs = []
        for row in ds:
            qs.append(row["query"])
            if len(qs) >= n:
                break
        with open(path, "w") as f:
            for q in qs:
                f.write(q + "\n")
        return qs
    with open(path) as f:
        return [line.strip() for line in f if line.strip()][:n]
def run_native(client: httpx.Client, orchestrator_url: str, query: str) -> StageTimings:
    t0 = time.perf_counter()
    with client.stream(
        "POST", f"{orchestrator_url}/rag",
        json={"query": query}, timeout=60.0,
    ) as r:
        r.raise_for_status()
        for chunk in r.iter_text():
            if chunk.strip():
                ttft = (time.perf_counter() - t0) * 1000
                # Drain rest of stream so vLLM can free the slot.
                for _ in r.iter_text():
                    pass
                return StageTimings(0, 0, 0, ttft)
    return StageTimings(0, 0, 0, -1)
def run_staging(
    client: httpx.Client, embed_url: str, qdrant_url: str, vllm_url: str, query: str,
) -> StageTimings:
    t0 = time.perf_counter()
    er = client.post(f"{embed_url}/v1/embeddings",
                     json={"model": "BAAI/bge-small-en-v1.5", "input": [query]})
    er.raise_for_status()
    vec = er.json()["data"][0]["embedding"]
    t_embed = time.perf_counter()
    sr = client.post(f"{qdrant_url}/collections/wiki_250k/points/search",
                     json={"vector": vec, "limit": 5, "with_payload": True})
    sr.raise_for_status()
    hits = sr.json()["result"]
    t_search = time.perf_counter()
    # Simulate PCIe staging.
    passages = [h["payload"]["text"] for h in hits]
    stage_buf = bytearray()
    for p in passages:
        stage_buf.extend(p.encode("utf-8"))
    final_buf = bytes(stage_buf)
    _ = len(final_buf)
    t_stage = time.perf_counter()
    ctx = "\n\n".join(f"[{i+1}] {p}" for i, p in enumerate(passages))
    messages = [
        {"role": "system",
         "content": "Answer using only the provided passages. Cite passage numbers."},
        {"role": "user",
         "content": f"Passages:\n{ctx}\n\nQuestion: {query}"},
    ]
    with client.stream(
        "POST", f"{vllm_url}/v1/chat/completions",
        json={"model": "Qwen/Qwen2.5-7B-Instruct",
              "messages": messages, "stream": True, "max_tokens": 128},
        timeout=60.0,
    ) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if line.startswith("data:") and "content" in line:
                ttft = (time.perf_counter() - t0) * 1000
                for _ in r.iter_lines():
                    pass
                return StageTimings(
                    embed_ms=(t_embed - t0) * 1000,
                    search_ms=(t_search - t_embed) * 1000,
                    stage_overhead_ms=(t_stage - t_search) * 1000,
                    ttft_ms=ttft,
                )
    return StageTimings(-1, -1, -1, -1)
def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--config", choices=["native", "staging", "remote"],
                        default="native")
    parser.add_argument("--n", type=int, default=200)
    parser.add_argument("--orchestrator-url", default="http://localhost:8080")
    parser.add_argument("--embed-url", default="http://localhost:8000")
    parser.add_argument("--qdrant-url", default="http://localhost:6333")
    parser.add_argument("--vllm-url", default="http://localhost:8001")
    parser.add_argument("--queries-file", default="queries.txt")
    parser.add_argument("--warmup", type=int, default=10)
    args = parser.parse_args()
    queries = load_queries(args.queries_file, args.n + args.warmup)
    print(f"Loaded {len(queries)} queries. Warming up with {args.warmup}...")
    client = httpx.Client()
    for q in queries[:args.warmup]:
        try:
            if args.config == "native":
                run_native(client, args.orchestrator_url, q)
            else:
                run_staging(client, args.embed_url, args.qdrant_url,
                            args.vllm_url, q)
        except Exception as e:
            print(f"Warmup query failed: {e}", file=sys.stderr)
    result = Result()
    print(f"Benchmarking {args.config} mode with {args.n} queries...")
    start = time.perf_counter()
    for i, q in enumerate(queries[args.warmup:args.warmup + args.n]):
        try:
            if args.config == "native":
                t = run_native(client, args.orchestrator_url, q)
            else:
                t = run_staging(client, args.embed_url, args.qdrant_url,
                                args.vllm_url, q)
            if t.ttft_ms > 0:
                result.timings.append(t)
            if (i + 1) % 50 == 0:
                p50 = statistics.median(t.ttft_ms for t in result.timings)
                print(f"  {i+1}/{args.n} done, running p50 TTFT: {p50:.1f} ms")
        except Exception as e:
            print(f"Query {i} failed: {e}", file=sys.stderr)
    elapsed = time.perf_counter() - start
    summary = result.summarize()
    print()
    print(f"=== Results: {args.config} ===")
    print(f"  elapsed:       {elapsed:.1f}s")
    print(f"  n successful:  {summary['n']}")
    print(f"  TTFT p50:      {summary['ttft_p50_ms']:.1f} ms")
    print(f"  TTFT p95:      {summary['ttft_p95_ms']:.1f} ms")
    print(f"  TTFT p99:      {summary['ttft_p99_ms']:.1f} ms")
    if summary['embed_p50_ms']:
        print(f"  embed p50:     {summary['embed_p50_ms']:.1f} ms")
    if summary['search_p50_ms']:
        print(f"  search p50:    {summary['search_p50_ms']:.1f} ms")
    out_path = f"results-{args.config}.json"
    with open(out_path, "w") as f:
        json.dump({"config": args.config, "summary": summary,
                   "samples": [t.__dict__ for t in result.timings]}, f, indent=2)
    print(f"  wrote {out_path}")
if __name__ == "__main__":
    main()

		

After applying all manifests you have

			
$ kubectl get pods -n zerocopy-rag
NAME                               READY   STATUS    RESTARTS   AGE
bge-embedder-6c89d6bb4c-5qw2s      1/1     Running   0          54m
qdrant-0                           1/1     Running   0          55m
rag-orchestrator-54769ddd8-ftt58   1/1     Running   0          49m
vllm-generator-77bc8d7d64-kxngs    1/1     Running   0          50m

		

Few gotchas to remember.

Use `qdrant/qdrant:v1.13.0` or later. The upstream image gained ARM64 page-size probing and runs unmodified on the Spark. Else you need to build qdrant with `JEMALLOC_SYS_WITH_LG_PAGE=16` (that’s `log2(65536) = 16`). Qdrant’s Dockerfile exposes this as a build arg.
The Grace CPU on the GB10 is a 20-core ARM configuration: 10× Cortex-X925 performance cores and 10× Cortex-A725 efficiency cores. HNSW graph traversal is a pointer-chasing workload — it wants high single-threaded performance, large L2 caches, and fast random memory access. By default, the Linux scheduler will happily schedule Qdrant threads on efficiency cores, which are roughly half the speed for this workload. The standard Kubernetes answer to this is the static CPU manager policy, which gives integer-CPU-request, Guaranteed-QoS pods exclusive access to a set of cores. Combined with `reservedSystemCPUs` you can pin Qdrant to the X925 cluster. This works fine on a stock kubelet. It did not work fine on K3s, at least not in the way the K3s documentation suggests.
I tried passing `–kubelet-arg=config=/etc/rancher/k3s/kubelet.yaml` to the K3s server in the systemd unit, with a `KubeletConfiguration` setting `cpuManagerPolicy: static`. K3s entered a restart loop. After reverting the change, K3s stayed in the same restart loop because the partial state from the failed restarts had corrupted several other things:
- CNI binaries weren’t being placed at `/opt/cni/bin/` (where the kubelet expected them) but at `/var/lib/rancher/k3s/data/current/bin/` (where K3s puts them). K3s normally tells the kubelet via `–cni-bin-dir`, but in the recovery state it wasn’t passing that flag.
- The containerd `nvidia` runtime registration in `config.toml` got rewritten with `runtime_type = “io.containerd.runtime.v1.linux”` — a deprecated v1 type that modern containerd refuses to load. Every GPU operator pod failed sandbox creation with `RuntimeHandler “nvidia” not supported`.
- The K3s-rendered config.toml had ended up containing only the nvidia runtime block — no base CRI plugin config, no default runtime, nothing — because of an interaction with my custom `config.toml.tmpl`.
Each of these is fixable in isolation. Together they made the cluster unbootable. I eventually had to nuke `/var/lib/rancher/k3s/agent/etc/containerd/`, let K3s regenerate from its embedded defaults, and re-apply my workloads.
The lesson, captured here as a standing constraint: **do not pass arbitrary KubeletConfiguration to K3s without testing the exact configuration on a throwaway cluster first.** K3s embeds the kubelet in its own process and does not expose every kubelet feature cleanly. The CPU manager static policy is one of the things that does not work cleanly. There may be a way to make it work — I have not found it, and the cost of finding out the wrong way is high.
For the GB10 specifically, the practical answer is I ended up not pinning. I ran the rest of the benchmarks with the default Linux scheduler. HNSW search latency was somewhat noisier than I expect it would be with strict pinning.

Building the Corpus and the Index

The corpus is 250k passages from the Wikipedia dump, chunked to ~500 tokens each, embedded with `BAAI/bge-small-en-v1.5`. The full ingestion script is in `scripts/ingest-wikipedia.py`. It takes about 22 minutes end to end on the Spark, with the embedding model saturating one GPU partition and Qdrant happily ingesting on the Grace side.

A few specific choices worth flagging:

HNSW parameters: `m=16`, `ef_construct=128`. Standard Qdrant defaults. I tested `m=32` and the recall improved marginally at roughly 1.8x the memory footprint. Not worth it for this corpus size.
Quantization: I left vectors as full FP32 in the index. Qdrant supports int8 scalar quantization and binary quantization, and on a memory-constrained discrete GPU I would absolutely use them. On the GB10 I have 128 GB to play with; the 380 MB of raw vectors is a rounding error. Using full precision also removes one variable from the benchmark.
Payload: each point stores the original passage text (UTF-8), a source URL, and a document ID. Payload is stored with `on_disk: false` so the full payload lives in the mmap’d region. This is the crucial setting for the zero-copy story — more on this in the benchmark section.

The Qdrant collection config:

			
from qdrant_client.http.models import (
    VectorParams, Distance, HnswConfigDiff, OptimizersConfigDiff
)
client.create_collection(
    collection_name="wiki_250k",
    vectors_config=VectorParams(
        size=384,
        distance=Distance.COSINE,
        on_disk=False,  # keep vectors in the mmap region, not a separate disk blob
    ),
    hnsw_config=HnswConfigDiff(m=16, ef_construct=128),
    optimizers_config=OptimizersConfigDiff(default_segment_number=4),
)

		

The `on_disk=False` on the vectors is doing real work here. Qdrant supports three storage modes for vectors: fully in-RAM (`on_disk=False`, default), memory-mapped from disk, and fully on-disk with LRU cache. For the zero-copy RAG claim to be interesting, the vectors must be in the unified memory pool, not behind a disk I/O layer. With the in-RAM mode, Qdrant’s Rust process holds the vectors as ordinary anonymous pages — and those pages live in LPDDR5X, which is exactly the memory the GPU can read coherently.

I have not yet tried to make the GPU *directly* read Qdrant’s storage (that would require a CUDA kernel that takes a `void*` pointing into another process’s address space, which is not a thing you can do without explicit IPC or CUDA MPS). The zero-copy behavior in this article applies to the *downstream* pipeline: vLLM prefill reading prompt bytes that Qdrant wrote. I will revisit direct GPU access to Qdrant’s storage in a follow-up once I have NVIDIA cuVS integrated.

The Benchmark Setup

I wanted an apples-to-apples comparison, which is hard on a single box with one coherent memory pool. I cannot physically introduce a PCIe bus where there isn’t one. So I constructed two configurations that bracket the real answer:

Config A — Native coherent: Everything running as described above. Qdrant on Grace cores, vLLM on Blackwell SMs, embedding model on Blackwell SMs, all sharing LPDDR5X. This is the intended production setup.

Config B — Forced staging: Same topology, but the orchestrator explicitly copies all retrieved-passage bytes through a staging buffer and then a second copy to a “device-side” buffer before handing to vLLM. On a discrete GPU this is what typically happens under the hood. On the GB10 it is a pure overhead simulation — the bytes never leave LPDDR5X — but it adds the CPU cycles and syscall overhead that a PCIe pipeline would incur. It is an underestimate of the real PCIe tax.

What I Measured
100 queries through the pipeline, 10 warmup queries discarded, MS MARCO–style natural-language questions, retrieval against a small in-memory corpus.

			
Config A — Native coherent
  TTFT p50:   89 ms
  TTFT p75:   91 ms
  TTFT p95:   93 ms
  TTFT p99:   94 ms
Config B — Forced staging
  TTFT p50:   88 ms
  TTFT p75:   91 ms
  TTFT p95:   93 ms
  TTFT p99:   93 ms

		

Two things to take from this.
First, TTFT p50 of 89 ms, end-to-end, on a $4,000 desktop. Embed plus search plus prefill plus first token. That includes everything between “user query arrives at the orchestrator” and “first token of the answer leaves vLLM.” On a 7B-class model with retrieval. This is the headline and it’s real.
Second, Configs A and B are within 1 ms of each other. This is not what I expected and it deserves an honest explanation rather than a face-saving one.

Why the Staging Comparison Failed
The “forced staging” configuration was supposed to simulate the overhead a discrete-GPU pipeline would pay. My implementation:

			
stage = bytearray()
for p in retrieved_passages:
    stage.extend(p.encode("utf-8"))
_ = bytes(stage)

Two CPU memory copies of about 5 KB of UTF-8 text. On a CPU that runs memcpy at multi-GB/s, this completes in microseconds. It is not a meaningful simulation of a real PCIe transfer pipeline.

The real PCIe tax on a discrete-GPU system is not dominated by the bytes-on-the-wire latency. It is dominated by driver-synchronization round-trips: pinned-memory allocator calls, DMA descriptor construction, kernel launches, stream synchronization, user-kernel mode transitions. Each of those is microseconds-to-low-milliseconds individually, and they don’t scale with payload size. None of them have analogues on a coherent-memory system, which means I had no honest way to simulate them on the GB10. My “staging” was just two memcpys.

So my benchmark cannot tell you how much faster the GB10 is than a discrete-GPU pipeline. To answer that, you would need to run the same workload on real x86+RTX-Pro-6000 hardware and compare. I do not have that hardware. If someone reproduces this benchmark on a discrete-GPU workstation with the same models and same retrieval setup, I will link to their results here.

What the Architecture Does Get You, Honestly
Coherent unified memory still matters on the GB10 for reasons that don’t show up in a single-query TTFT benchmark.

Capacity: On a discrete-GPU system, you would be sharding across machines or paying the latency of pulling weights from CPU memory whenever the GPU evicted them. The Spark just runs.

Architectural simplicity: vLLM’s normal code path, the one designed for discrete GPUs, allocates buffers via the standard CUDA APIs and on the GB10 those allocations land in LPDDR5X. The retrieved-passage bytes are copied from the orchestrator’s process to vLLM’s tokenizer buffer in CPU space, tokenized to a tensor in LPDDR5X, and prefilled at DRAM speed. There is no place in the code where I had to think about which memory pool something lived in, because there’s only one pool. That’s an enormous reduction in cognitive load even when it doesn’t show up as a per-query latency win.

Cold-start latency. I didn’t measure this carefully but I noticed it. When vLLM starts up, it reads its quantized weights from disk into LPDDR5X — and that’s it. There’s no second copy from CPU memory to GPU VRAM. On a discrete-GPU system, model loading is two transfers: disk-to-CPU and CPU-to-GPU. The Spark eliminates the second one. For a 5 GB NVFP4 model that’s a few hundred milliseconds saved on every cold start, which matters for autoscaling scenarios even if it doesn’t matter for a steady-state benchmark.

KV cache pressure. Long-context inference on discrete GPUs is bottlenecked by VRAM capacity for the KV cache. The Spark’s 128 GB pool means you can serve very long contexts (tens of thousands of tokens) without juggling. I didn’t push this in the article’s measurements because they were all short prompts, but it’s a real benefit for chat-history-heavy applications.

Summary

End-to-end TTFT of 89 ms on a desktop-class box doing 8B-NVFP4 RAG generation with vector retrieval. That number is real, reproducible, and small enough to feel interactive.

The architectural advantages of the GB10’s coherent unified memory are real but I cannot honestly measure their per-query latency cost on this hardware alone. The story I started writing — “here’s exactly how many milliseconds the PCIe tax costs you” — turned out to be a story I couldn’t tell with my equipment. The story I can tell — “here’s a small, measurable, fast pipeline that fits on one box because the architecture lets every component share one memory pool” — is true and I think still worth writing.

If you reproduce this benchmark on a discrete-GPU workstation and find that the same pipeline costs you 200+ ms, that would confirm the claim I couldn’t measure. I would want to know about it.