Tensor-parallel inference on four clamshell-modded 3080s (stock GA102, 760 GB/s, no P2P) in a China datacenter. The verdict: run Q8 with the MTP head — ~95 tok/s decode, 2.4× the 8× RTX 5060 Ti box, at full quant quality.
Champion decode
95 tok/s
Q8 + MTP, tensor, 32K ctx
vs 8× 5060 Ti
2.4×
▲ faster single-user decode
MTP speedup
1.8×
~80% draft acceptance
256K ctx fits
11 GB/GPU
of 20 GB · hybrid-SSM KV
Speculative decode is the whole ballgame
Q8 · tensor · 220W
Decode throughput vs context, on realistic coding prompts (cache_prompt=false). The model's trained multi-token-prediction head nearly doubles single-stream speed; prompt-lookup n-gram backfires on this latency-bound interconnect.
data table
Quant & context scaling
synthetic · llama-bench
Q6_K_XL vs Q8_0 across the full 512→256K sweep. Prefill favors Q8 by 2–5%; decode is a dead heat — the 5060 Ti's Q6-decode edge vanishes because decode here is bound by the quant-independent no-P2P all-reduce latency, not weight streaming. Falloff is graceful (SSM layers are O(1) in context).
Token generation (decode)
tok/s · higher is better · log context axis
Prompt processing (prefill)
tok/s · higher is better · log context axis
Why it wins vs the 8× 5060 Ti box
matched · 4-GPU tensor · Q6 · short ctx
Cores/FLOPS explain prefill (+20%), memory bandwidth explains decode (+39%), and the shared no-P2P wall trims both. The 1.89× CUDA-core headline is the number not to quote — Blackwell's higher clocks erase most of it.
Throughput, same config
3080 vs 5060 Ti · tok/s
The silicon (measured)
CUDA cores8704 vs 4608 · 1.89×
FP3229.8 vs 23.7 · 1.26×
Memory BW760 vs 448 · 1.70×
InterconnectGen3×16 ≈ Gen4×8
P2Pdisabled (both)
Bin3080 · 68 SM · 320-bit
Interconnect, ubatch & thermals
the supporting cast
Intercard PCIe by phase
Gen3 ×16 ceiling ≈ 15.75 GB/s
n_ubatch → prefill
default 512 is near-optimal
Power & thermals @ 220W
Power draw220W CAPPED
Peak temp67°C COOL
SM clock~1770 MHz
VRAM used~11–12 GB
Power-limited, not thermal-limited — the locked-away 320W headroom would likely help, but the container denies nvidia-smi -pl.
Findings
01
-sm tensor works on no-P2P GeForce. Loads qwen35 across 4 cards; 256K fits at ~11 GB/GPU. (Row-split crashed on the 5060 Ti box.)
02
MTP ≈ 1.8× decode, ~80% acceptance, peaks 95 t/s @32K. The config to ship.
03
n-gram is a net loss (~2.7× slower, ~40% accept) — partly our cold-cache / short-gen methodology.
04
Just run Q8. Prefill +2–5% vs Q6, decode a tie — the bandwidth-bound quant advantage disappears behind the latency floor.
05
The interconnect is the ceiling. High-context prefill bursts saturate Gen3 ×16 (~15 GB/s) — that's the long-context falloff.
06
Sweet spot by default. 4 cards = no 8-way TP collapse; ubatch tuning ≤4%; cards power-capped but thermally idle.