Qwen3.8-27B: NVFP4 vs GGUF on an RTX 5090

I’d choose by workload

For short prompts, I’d start with Base GGUF UD-Q6_K for accuracy or Swift 1.5 GGUF Q6_K_S for speed. If you regularly feed the model long inputs, repaired Base NVFP4 is worth the extra setup.

On my RTX 5090, Base GGUF scored 60.81/100 on QuantBench and generated 95.48 tokens/sec on average. Base NVFP4 scored 59.82 at 93.64 tokens/sec. That gives me little reason to move a working short-prompt setup just for NVFP4.

Long inputs change my recommendation. Both Base configurations scored 95.83 on the reduced HomHaystack test, but the average wait before the answer started fell from 21.84 seconds with GGUF to 8.26 seconds with NVFP4. Generation also increased from 78.34 to 90.60 tokens/sec. That’s a difference you’ll feel while waiting for a response.

ThinkingCap NVFP4 reached the highest long-input score, 97.50, and the highest generation speed, 106.35 tokens/sec, among the complete results. It also took longer to start answering than the other NVFP4 configurations. I wouldn’t choose from the generation-speed column alone.

Nine configurations on one 32GB card

I compared nine current configurations across five Qwen3.8-27B variants on one RTX 5090 with 32GB VRAM. Base, Swift 1.5, ThinkingCap, and OrcaRouter each have a GGUF and an NVFP4 entry. NVIDIA has an NVFP4 entry only.

The model and quantization names matter:

• Base: Unsloth’s Qwen3.8-27B, using UD-Q6_K GGUF and its NVFP4 checkpoint.
• Swift 1.5: ukisai’s Swift-1.5-Qwen3.8-27b NVFP4, compared with Bartowski’s Q6_K_S GGUF conversion of that variant.
• ThinkingCap: bottlecapai’s ThinkingCap-Qwen3.8-27B in Q6_K GGUF and NVFP4.
• NVIDIA: NVIDIA’s Qwen3.8-27B NVFP4 checkpoint.
• OrcaRouter: orcarouter’s Qwen3.8-27B-Uncensored in Q6_K GGUF and NVFP4.

All main comparisons use thinking Off and native MTP1. MTP, or multi-token prediction, lets the model propose an extra token for verification. Accepted proposals can reduce the work needed to generate an answer. The speed benefit still needs measurement.

The GGUF configurations ran through LM Studio and llama.cpp with Q8_0 K/V cache. The repaired NVFP4 configurations used vLLM 0.29.0 in WSL/Docker, FP8 E4M3 K/V cache, and a fixed 4GiB cache pool. The historical NVFP4 timing baseline used vLLM 0.27.1 and remains a separate cohort. K/V cache holds the attention state that grows as you process and generate tokens.

These results compare complete configurations. The runtimes, publishers, quantizers, cache formats, and chat templates differ. Even tokenization can affect the work each request requires. An NVFP4 checkpoint name also doesn’t mean every tensor is stored in FP4.

Two workloads, two separate scores

QuantBench uses 50 questions with 32,768 tokens of configured context capacity, release QB3-3.3.8, and scorer QB3-SCORE-3.3.0. The prompts are much shorter than that capacity. This is my short-prompt accuracy and speed comparison.

Reduced HomHaystack tests retrieval and retrieval-plus-reasoning over long inputs. Its configured context is 90,112 tokens, with 4,096 tokens reserved for the output and a 2,048-token safety margin. The short and long datasets contain 3,405 and 6,314 records, respectively.

Each configuration has five Classic-Short requests, one Classic-Long request, and three Reasoning-Short requests. The HS-1.2 overall weights those groups 30%, 20%, and 50%. All nine requests must be valid and scorable for an official overall. This reduced profile, legacy-vram-mtp1-v1, has a different workload from my historical 262,144-context results; keep its scores separate.

I also keep QuantBench and HomHaystack separate here. A combined score would add a workload preference that these measurements don’t choose for you.

Explore the QuantBench configurations or reduced HomHaystack configurations for capture dates, runtime details, timing distributions and completion status.

GGUF remains competitive on short prompts

Base GGUF has the highest complete QuantBench score, but the gap to Base NVFP4 is 0.99 points. Their generation speeds are close too. Swift GGUF reaches 100.56 tokens/sec, with a 59.06 score, making it my first GGUF choice when short-prompt speed matters more than that last bit of accuracy.

ThinkingCap NVFP4 delivers 109.12 tokens/sec with a 59.76 score. That’s the fastest complete short-prompt result here. It’s a useful option if you’re willing to maintain the NVFP4 setup. NVIDIA and OrcaRouter NVFP4 both generate around 94–96 tokens/sec, but their QuantBench scores fall to 55.38 and 54.69. Similar speed doesn’t buy you the same task accuracy.

ThinkingCap GGUF completed 48/50 QuantBench questions and has no official overall score. Two questions failed after the allowed connection retries. Its 97.59 tokens/sec average covers only the 48 completions, so I’ve kept it out of the complete-results table.

These are observed differences across this question set. They don’t establish a universal ranking or statistical confidence from repeated identical trials.

Long inputs give NVFP4 a stronger case

All five NVFP4 entries generate faster than the three GGUF entries with complete long-input results. Accuracy still depends on the variant. Base preserves the 95.83 score across formats. Swift NVFP4 scores 94.17, compared with 95.83 for Swift GGUF. ThinkingCap NVFP4 rises to 97.50, compared with 95.83 for its GGUF configuration.

For this workload, I’d try Base NVFP4 first for the combination of accuracy and lower waiting time. ThinkingCap is the next configuration I’d test if the higher measured retrieval score matters more to me than how quickly the answer starts.

OrcaRouter GGUF completed only 8/9 requests. The remaining request reached the unchanged 4,096-token output cap on both allowed attempts. Its overall score is unavailable. Its 77.79 tokens/sec average covers eight completions only, so it doesn’t belong in the complete-suite speed ranking.

The model’s “Uncensored” label doesn’t change what this test measures. I didn’t measure refusal rates, and successful non-answering for missing keys is only a limited test behavior. It doesn’t establish dependable hallucination rejection. Thinking was Off throughout, so these results also don’t tell us which model uses reasoning tokens most efficiently with thinking enabled.

The wait changes the speed you experience

Generation speed starts counting after output begins. When your input is long, processing it can take a substantial part of the request. That initial wait is why I also compare speed over the whole request.

The original Base, Swift, and ThinkingCap GGUF waits use the runtime’s native first-token timing. NVFP4 uses the client’s first delivered token. Those boundaries aren’t identical, so treat the cross-runtime waits as recorded measurements rather than perfectly interchangeable stopwatch positions. Whole-request speed includes the initial wait.

Base NVFP4 improves speed including the wait from 31.58 to 54.54 tokens/sec against Base GGUF. Swift goes from 31.51 to 57.95. Both pairings show why a modest generation-speed difference can still make a long-input task feel much faster.

ThinkingCap shows the other side. It generates at 106.35 tokens/sec, ahead of NVIDIA’s 94.06, but waits 15.40 seconds before starting, compared with NVIDIA’s 7.49 seconds. Including that wait, NVIDIA reaches 58.14 tokens/sec and ThinkingCap reaches 43.72.

The average whole request takes 19.63 seconds for NVIDIA and 26.10 seconds for ThinkingCap. Their answers also differ in length, so this isn’t a fixed-output-duration test. It is a practical reason to measure your complete task before choosing the largest generation-speed number.

How I calculate these averages

Each table uses the arithmetic average of the per-request measurements. I don’t divide all tokens by all elapsed time and substitute that pooled rate for the average. The complete QuantBench rows contain 50 questions; the complete HomHaystack rows contain nine requests.

Generation speed excludes the initial wait and rejected speculative proposals. For NVFP4, it uses delivered tokens after the first batch and the measured first-to-last delivery interval. GGUF uses its saved native generation counts and timing. Speed including the wait divides each request’s output-token count by its full elapsed time before averaging.

I used cold-cache requests: prefix reuse was disabled for NVFP4, and GGUF was unloaded and reloaded between requests. The recorded response timing covers the request, not the separate model reload. Reusing an existing conversation or cached prefix could change your waiting time.

The original NVFP4 slowdown was real

My original NVFP4 results were roughly 17–23 tokens/sec. Those were generation rates that already excluded the initial wait. The timing audit confirmed the slowdown; the later speed came from repairing the runtime and rerunning the unchanged test suite.

On QuantBench, Base NVFP4 moved from 22.65 to 93.64 tokens/sec, about 4.13× faster. Swift moved from 17.01 to 95.65, about 5.62× faster. Accuracy didn’t move in one direction: Base rose from 56.68 to 59.82, while Swift fell from 60.03 to 58.43.

The original eager runtime was already executing native FP4/FP8 kernels. Confirming that the intended kernels run is useful, but it doesn’t prove the execution path is efficient.

I isolated compilation and CUDA graph replay with the same Base checkpoint, MTP1, FP8 cache, fixed 4GiB cache pool, inputs, output lengths, and sampling. Compilation can combine operations; CUDA graphs reduce repeated dispatch work. The controlled results were:

Compilation and graph replay both improved these measured controls. I verified actual graph replay during requests. A setting that says graphs are enabled doesn’t establish that they were used.

A separate profile of the compiled small-graph configuration reported CUDA runtime 13.0, CUDA driver API 13.4 and NCCL 2.29.7 under vLLM 0.29.0. That short-input diagnostic was captured on October 1, 2026; its profiled timing is excluded from the speed and score results. The driver API version is distinct from the installed NVIDIA driver version, and this diagnostic does not establish the loaded CUDA runtime for every scored request.

Cache allocation also affected performance. Fixing the cache pool at 4GiB improved the eager path compared with the utilization-based allocation tested during repair. The memory observations don’t identify which tensor, if any, was being paged, so I can’t turn that result into a specific spill diagnosis.

The final setup used supported compilation and small graph capture sizes. ThinkingCap also needed the supported older GPU runner because the newer runner encountered a WSL host-memory compatibility problem. Dependency fixes made the environment compatible, but I didn’t isolate a separate speed gain for each package change.

OrcaRouter GGUF needed its embedded chat template supplied explicitly, with thinking disabled. After that repair it completed all 50 QuantBench questions, but the long-input output-cap failure remained. Getting a model loaded correctly and getting every task finished are separate checks.

That’s the configuration effort behind these numbers. If you’re already getting good short-prompt performance in LM Studio, the extra runtime work needs to solve a problem you actually have.

MTP helped in the matched Base controls

On the repaired Base setup, average generation speed with MTP Off was 68.01 tokens/sec for the short input and 61.82 for the long input. With MTP1, those rates were 94.78 and 88.18. That’s approximately 1.39× and 1.43× faster.

Each input had three measured repeats per setting, with warm-ups excluded. These are controls for the Base checkpoint. They don’t establish the same gain for NVIDIA, OrcaRouter, or every other checkpoint.

Graph dispatch shapes change when speculative slots are added, and the Off/On measurements used separate sessions and observer versions. The measured benefit therefore includes those execution differences. I’d keep that qualification attached to the MTP1 speedup.

I’d verify speed with MTP enabled and disabled on the exact setup I planned to use. A high acceptance rate tells you how often proposals were accepted; it doesn’t tell you whether the extra machinery saved time.

NVFP4 didn’t always use less memory

Base NVFP4 reached 28.28GiB, compared with 25.52GiB for Base GGUF on this workload. ThinkingCap went the other way: 25.74GiB with NVFP4 versus 26.20GiB with GGUF. OrcaRouter NVFP4 reached 29.53GiB, leaving less room on this card.

These are the highest sampled whole-device readings during the included completed requests. They aren’t model-only allocations or a guarantee that every brief peak was captured. The partial OrcaRouter GGUF value excludes its failed attempts. I converted the recorded MiB to GiB by dividing by 1,024, without subtracting a desktop baseline.

No CPU weight or K/V offload was configured. CUDA placement checks and Windows/WSL memory counters still can’t prove the physical residency of every allocation. File size, configured placement, and measured device memory answer different questions.

All of this was measured on the 32GB RTX 5090. I wouldn’t use these readings to promise a fit on a smaller card. Check the exact model, cache precision, context, output reserve, and MTP settings on the hardware you intend to run.

What I’d try first

For short prompts, I’d keep Base GGUF UD-Q6_K as my accuracy starting point and try Swift 1.5 GGUF Q6_K_S when generation speed is the priority. ThinkingCap NVFP4 is worth testing if you want its higher measured generation speed and are comfortable maintaining the runtime.

For frequent long inputs, I’d start with repaired Base NVFP4. It kept the Base GGUF retrieval score while shortening the wait substantially. I’d test ThinkingCap NVFP4 next for its higher measured long-input score, while watching total request time. NVIDIA NVFP4 offers a quick start on this workload, with a lower short-prompt score to account for.

In the NVFP4 versus GGUF video, I walk through the results and the runtime changes behind them. Tell me in the YouTube comments which runtime, GPU, and context you’re using, and whether the wait or the generation speed is your bigger problem. Subscribe to DeepWakeLabs for the next comparison.