Efficiency Gains or Moved Inefficiency?

Consider the connection between four developments that look unrelated at first. Inference is being cut by workload phase, with prefill separated from decode. Capacity is being cut by tenant, through fractional GPUs. Access is being cut by trust, through confidential computing. Performance is being cut by firmware policy, through the FP32-accumulation restriction on consumer cards. The common thread is that much of the GPU's theoretical capability goes unused, and much of the current engineering and commercial effort goes into recovering it.

Almost everything here was reported in the second half of October 2026, so the picture is current. Most claims come from a single source. I say where a claim has more support, and where the material contradicts itself.

The physical mismatch behind phase splitting

The starting point is an asymmetry inside a single request. LLM inference combines compute-bound prefill with memory-bound decoding ³. Prefill builds the KV cache using GEMM ³. A 2,048-token prefill performs 33 TFLOP ¹⁶, and each weight feeds 2,048 tokens ¹⁶. Decoding generates one token at a time using GEMV ³ and frequently accesses the KV cache ³.

The numbers show how lopsided this is. Decode at batch 1 takes 3.3 ms per token ¹⁶ and reads 16 GB of weights ¹⁶. That leaves tensor cores idle 99.5% of the time ¹⁶. At roughly 1 FLOP/byte, batch-1 decode reaches about 5 of 989 TFLOP/s ¹⁶. Two sources each support the 33 TFLOP prefill figure ¹⁶ and the 5-of-989 figure ¹⁶, which makes this some of the better-corroborated physics in the set.

Batching lifts weight reads toward the ridge ¹⁶, but it does not close the gap. GPT3-175B decoding at batch 32 needs 422 TFLOPS ³, and about 1.2 PFLOPS FP16 is nearly 3x that ³. The arXiv preprint 2610.26131 reaches the same conclusion from a roofline analysis. On a high-end GPU, prefill achieves near-peak throughput ³ while decoding uses far fewer compute resources ³. Decoding still maintains near-peak HBM bandwidth ³. The two phases stress different parts of the machine, and running them on the same hardware wastes one resource or the other.

Competing remedies and what they cost

Co-executing prefill and decoding causes interference ³. Continuous batching produces prefill-decode interference and TPOT tail latency ³. Chunked prefill is the obvious patch, but it only trades one problem for another, between TPOT and TTFT ³. Small chunks raise TTFT because they require repeated KV loading ³. Large chunks inflate TPOT ³. For MoE models it is also suboptimal because of repeated expert loading ³.

Disaggregation is the main alternative. It separates the phases into distinct GPU pools. It is the best-corroborated architectural claim in the material, with two sources spanning 2026-09-09 to 2026-10-20 ^1,3. Commenters described it as common ¹⁰, and llm-d 1.0 uses disaggregated P/D pools ¹². The costs are real, though. Moving the KV cache over InfiniBand ³ or NVLink ³ incurs significant overhead. Decoding also still underuses compute resources in PD-disaggregated serving ³. Splitting the phases therefore moves the inefficiency without removing it. Scope matters too. DeepInfer's benchmarks were both run within the same scale-up domain ⁶, and VE/Prefill/Decode disaggregation is not included in its release ⁶.

DuoPD: a promising but unreplicated answer

DuoPD is the most detailed attempt to fix the leftover decode inefficiency. It executes full decoding on Processing-Near-Memory (PNM) ³ while prefill runs concurrently on the GPU ³. The preprint reports up to 2.23x higher end-to-end throughput than conventional PD disaggregation ³ and lower latency under the same GPU budget ³. The 2.23x figure is described as the paper's simulation-based claim ³.

Against chunked prefill, the reported results depend on the workload:

• Up to 3.12x higher QPS at the QPS where TTFT rises 3x ³.
• 1.08–2.42x throughput for prefill-heavy workloads with Lo=32 ³.
• 1.31–3.92x for long outputs with Lo=512 ³.

At batch 16 and 16K input, GPU compute utilization is within <1% of prefill instances ³ and 9.50x–30.62x higher than decode instances ³. All of this comes from one preprint and has not been replicated. I read it as evidence of direction, not settled performance.

A tension over NVIDIA's own prefill hardware

The material disagrees about NVIDIA's own response. One account says NVIDIA Rubin CPX runs prefill on CPX with GDDR7, and two sources assert it ³. Another says NVIDIA shelved Rubin CPX, a GDDR7 prefill chip ¹⁴. That claim has one source, and both are dated mid-to-late October. The material does not reconcile them. The alternative it describes is Attention-FFN disaggregation. In that design Rubin GPUs handle self-attention over the KV cache ¹⁴, and activations are exchanged between chips for each token ¹⁴. Two sources support each of those claims. The safest reading is that NVIDIA's approach to prefill-specialized hardware looks unsettled.

Software levers are hardware-dependent

Software techniques attack the same decode inefficiency, but the gains vary widely. With DFlash, Nemotron 3.5 Bolt E4B 4GB reaches 2.74x per-user decode speed versus no speculative decoding at concurrency 1 ¹⁵. A DGX Spark with an NVFP4 W4A16 drafter shows only 1.13x per-user speedup at concurrency 1 ¹⁵. That second figure is the most heavily corroborated in the set, with six sources. The gap between the two shows that speculative-decoding gains depend heavily on the hardware and model in use.

Fractional GPUs: recovering capacity across tenants

The same waste appears at cluster level, and the commercial pressure to recover it is visible. Shadeform claims average utilization above 95% for Blackwell and Hopper GPUs across Flex VM providers to date ⁴. Another source argues that the monthly node revenue gap between 50% and 95% utilization is roughly double for every GPU generation ². Both claims come from single sources and may be interested. Still, they explain why sharing is attractive.

The tooling is maturing. A new open-source Kubernetes operator named gpu-fractioning ⁸ is hosted under the KAI Scheduler organization on GitHub ⁸. Starting with v0.18.0, the KAI Scheduler has a GPU-sharing mode called NvFractions ⁸. The division of labor is clean:

• The scheduler picks which slice of which GPU each workload gets and writes annotations ⁸.
• The operator enforces those annotations ⁸.
• The operator also sets up and runs MPS on each node ⁸.

The sharing mechanisms vary in strength. Time-slicing presents four virtual replicas per physical GPU ⁷. MIG instances are hardware-isolated and transparent to the application ⁸. CUDA 13.4 adds a transparent memory limit that shows each workload only a fraction of GPU memory ⁸. It also adds a compute mode that lets a workload choose sm-sharing over default time-sharing ⁸. Users request a fraction through an annotation such as gpu-fraction: "0.5" ⁸. A fractional request applies per device, so a 40Gi request across four devices gets 40Gi on each of four GPUs ⁸.

The constraints say as much as the features. The scheduler must still choose a GPU with enough free memory ⁸ and must never mix compute modes on one GPU ⁸. KAI never places time-slicing and sm-sharing workloads on the same GPU ⁸, although one node can host both modes on different GPUs ⁸. Fractioning therefore adds scheduling complexity. It does not remove it.

Confidential computing: where trust partitioning shrinks supply

Sharing and confidentiality pull in opposite directions. Arcium's survey reports that nearly all available confidential GPUs are Hopper-class single-GPU configurations ¹³. It also finds that the confidential GPUs that can actually be rented are about 0.04% of those capable of confidential computing ¹³. One structural reason is that GPUs under virtualization cannot enter confidential computing mode ¹³. That rules out the virtualized sharing model described above. Blackthorn expands the pool from about 2,000 GPUs to bare-metal Hopper and Blackwell fleets run by neoclouds and enterprises, according to two sources ¹³. Virtualized GPUs are still excluded ¹³.

Gaps remain beyond access. On Hopper, GPU-to-GPU traffic within an NVLink-connected node is in plaintext but invisible to the host ¹³. On Blackwell, cryptographic multi-GPU confidentiality can be completed with Blackthorn ¹³. GPU CC mode establishes keys via SPDM, according to two sources ¹³. Remote SPDM at GPU initialization takes about 1 to 2 seconds ¹³.

Firmware-gated performance and the FP32-accumulation dispute

The most contested material asks whether NVIDIA throttles consumer silicon. The halving of FP16 with FP32 accumulation on consumer cards is corroborated across generations. It has two sources for Ampere ¹¹, three for Ada ¹¹ and three for Blackwell 2.0 ¹¹. Turing is reported as halved as well ¹¹. The same post says FP32-accumulation performance is unlocked on all Hopper GPUs ¹¹ and all Blackwell Datacenter GPUs ¹¹. A post author says NVIDIA intentionally limits tensor-core throughput when accumulating in FP32 ⁹. The unlock uses NVIDIA's own Booter instructions to write the override ⁹, and only NVIDIA's privileged firmware can write that register ⁹.

The gpu-unlock project's performance claims are broad. It reports up to 2x FP16/BF16/TF32 tensor throughput ⁵. Its figures are specific, but all are project-reported:

• LLM prefill gains of +25–55% ⁵.
• Time to first token 25–30% lower ⁵.
• Image generation gains of +30–55% ⁵.
• FlashAttention/SDPA kernel gains of +50–60% ⁵.
• 0–15% for llama.cpp/Ollama with quantized GGUF, mostly at long context ⁵.
• No change for FP32 CUDA ⁵.

The project describes outputs as bit-identical before and after unlock ⁵. It also states that the realized gain depends on the FP16/BF16/TF32 share of the workload and on power headroom ⁵. A commenter's anecdote puts the gain for an unlocked 3080 at 50–100% more diffusion performance if inference uses FP32 accumulation ¹¹. Another commenter says a different open-gpu-kernel-modules patch unlocks roughly 20% card-to-card speed on consumer GPUs for multi-GPU P2P ⁹. Whether DLSS benefits is disputed. Some commenters claim it does not use FP32 accumulation ¹¹, and one author says he is almost sure it uses FP16 or lower precision ⁹. None of these gains is independently benchmarked in the material.

What this implies

One economic logic runs through all four seams: the cheapest performance gain is to stop paying for capability that sits idle. Idle tensor cores during decode, at 99.5% at batch 1 ¹⁶, underutilized nodes ² and throttled consumer tensor throughput ⁹ are three versions of the same problem. The strength of the evidence differs by seam.

• Phase mismatch is the best-corroborated problem. Prefill is compute-bound and decode is memory-bound. Disaggregation is widely adopted, but KV-transfer overhead ³ and underused decode compute ³ remain open. DuoPD's gains are so far unreplicated and partly simulation-based ³.
• Utilization is the commercial lever. Fractioning via KAI and gpu-fractioning is shipping ⁸ and recovers idle capacity, but it carries real scheduling rules ⁸. The >95% utilization ⁴ and revenue-gap ² claims each rest on a single source.
• Confidentiality and sharing conflict. Virtualized GPUs cannot enter confidential mode ¹³, and rentable confidential capacity is about 0.04% ¹³.
• Treat unlock gains as provisional. The FP32-accumulation restriction is supported by multiple sources across consumer generations ¹¹. The quantified gains are self-reported, and the project itself says they depend on workload mix and power headroom ⁵.

Much of the remaining headroom is governed by policy as well as by silicon. NVIDIA's firmware decides which parts of the tensor-core capability are available ⁹. Its roadmap appears unsettled on prefill-specialized hardware, given the conflicting Rubin CPX claims ^3,14. The practical lesson for anyone building serving systems is to measure where capability is going idle before choosing a seam to cut. The reported gains are largest where the workload, the hardware and the isolation requirements fit the cut.