My Local LLM Picks for 12 GB and 16 GB GPUs

For a 12 GB GPU, I’d start with Ternary Bonsai 2 27B PTQ1_0. For a 16 GB GPU, I’d choose the specific Qwen3.8-27B Heretic GSQ-RCO IQ3_XXS file I tested. If you want a simpler LM Studio setup, Qwen3.5 9B Q5_K_M is my alternative.

Bonsai gives me a strong general-purpose score with enough memory headroom for a useful context window. The tested Qwen3.8-27B configuration scores higher and handles the long-context reasoning test much better. Qwen3.5 9B gives up general task accuracy, but its document lookup results make it worth considering.

There’s a limitation right at the start: I ran every benchmark on a single RTX 5090 with 32 GB of VRAM. The 12 GB and 16 GB recommendations are projections from measured memory use. I haven’t validated these configurations on physical cards in those capacities, and the speeds below are 5090 results.

My starting configuration for either main recommendation is 64K context, thinking on, Q4_0 K/V cache, flash attention on and multi-token prediction off. Keep room for the desktop and your other applications. A file fitting on disk tells you very little about how much room the running model leaves on your GPU.

The exact files and workloads matter

I tested eleven model files. Both Bonsai packings are included because a different packing changes memory use and can change the answers. The quantization suffixes identify the tested files; these results don’t apply automatically to another download with the same model family name.

The file sizes above are binary GiB. They are weight-file sizes, not running VRAM requirements.

I ran the non-Bonsai files through LM Studio 0.4.25 Build 1 and its llama.cpp engine. The selected CUDA12 extension varied across captures: 2.46.0, 2.47.0, 2.50.0 and 2.51.0. The result details retain each recorded selection; it does not establish the loaded upstream engine commit. Bonsai used PrismML’s llama.cpp fork, build b10709, commit 9a9394a. My Bonsai recommendation assumes you’re comfortable using that tested runtime. This campaign doesn’t establish how these particular Bonsai files perform in a different application or runtime.

QuantBench covers 50 tasks across knowledge, math, logic, planning, state tracking, Python, SQL, browser/UI work and writing. It awards partial credit. The configured context capacity was 65,536 tokens, with Q4_0 for both the key and value cache. That capacity includes the input and generated output; these were short prompts, not 64K documents. Responses that exhausted the context limit retained the benchmark’s cap penalties.

Seven files have separate thinking-on and thinking-off results. The Qwen3 Instruct file and the three Ministral Instruct files are the non-thinking variants in this comparison. A missing thinking score for those files isn’t a zero.

HomHaystack asks a different question: can the model find information in a very long input, and can it combine what it found correctly? The main comparison uses Q8_0 K/V cache, thinking off and temperature zero. Each suite contains five Classic 128K requests, one Classic 240K request and three Reasoning 128K requests. Those labels describe the input workload; the configured context capacity was 262,144 tokens. Historical requests capped generated output at 4,096 tokens; newer Bonsai, Ornith and Qwen3 Instruct captures removed that cap. Every selected 4,096-limited capture stopped normally below the limit. These remain different configurations.

There are also separate Bonsai experiments with Q4_0 K/V cache. I’ll keep those separate because changing the cache precision changes the tested configuration. Flash attention was enabled and MTP was disabled in the reported configurations.

Qwen3.8-27B leads QuantBench; Bonsai earns the smaller-card recommendation

These headline scores give each of the 50 distinct tasks equal weight. For tasks with eligible repeats, I average that task’s scores first, then average the 50 task means.

Qwen3.8-27B Heretic leads in both modes. With thinking on, it reaches 87.2, followed by Bonsai PTQ1_0 at 78.6 and PQ2_0 at 76.9. Gemma and Qwen3.5 9B land at 69.0 and 68.0.

That makes PTQ1_0 attractive for a smaller card. It outscored those two alternatives while using a smaller weight file and less measured process VRAM at 64K. PQ2_0 uses more memory without establishing a quality advantage in this task set, so PTQ1_0 is the packing I’d try first.

The tested thinking configurations also change the buying decision. Qwen3.5 9B rises from 31.8 to 68.0 in the tested thinking-on configuration. Qwen3.8-27B rises from 54.2 to 87.2. These changes also used different samplers. Original Off used temperature 0.7, top-p 0.9, top-k 40 and min-p 0. On used temperature 1 and top-p 0.95; top-k was 20 except Gemma at 64. Bonsai On used min-p 0.05; the others used 0. Bonsai and GSQ used controlled Medium effort, while Qwen3.5, Gemma and Ornith used native thinking without a matched effort limit. These are configuration comparisons, not an isolated toggle experiment.

The harder tasks explain why I still want a larger model squeezed into a smaller file. In the single full run, the hardest ten-task tier scored about 83 for Bonsai PTQ1_0, 81 for Qwen3.8-27B, 48 for Gemma and 44 for Qwen3.5 9B, with thinking on. Ten tasks don’t establish a universal ranking, but they do show capability differences that an easy-task score would miss.

Why the repeat weighting changed

An earlier leaderboard averaged one full 50-task run with three short runs that repeated the same ten tasks. Giving those four runs equal weight gives the repeated subset much more influence than the other forty tasks.

The headline table above corrects that imbalance by weighting tasks equally. It uses the full run and the available repeats with seeds 5091 and 5092, excluding the short repeat labeled seed 5090 as an aggregation choice. The saved metadata does not establish the full Off cohort’s seed, and repeated answers were not universally identical. Including the 5090 repeats leaves the model order unchanged. Ornith’s thinking-on series has only one of those retained repeat batches. These are observed scores with unequal repeat coverage, not confidence intervals from an equally replicated experiment.

The Instruct comparison below uses complete 50-task cohorts throughout, so its basis is explicit. The adjusted Ministral series changed the temperature to 0.05 and disabled generic reasoning parsing. It’s a separate configuration, not a matched repeat of the original settings.

The original Ministral 3B cohort joins 1 + 49 recovered answers; Gemma’s thinking-off cohort joins 36 + 14. Each covers all 50 tasks.

The adjusted settings improved all three Ministral full-cohort scores, but none challenged my two main recommendations on this workload. Ministral 14B does have a stronger result in long-context reasoning, which I’ll get to below.

Budget for the running model and its context

The memory figures below are sampled peaks of the model process’s dedicated GPU memory on Windows. The process samples were collected at roughly two-second intervals. They exclude other processes and shared system memory, and a sampled peak can miss a shorter transient allocation.

Whole-card readings include other GPU activity. I don’t treat a whole-card peak minus an idle reading as interchangeable with the model process’s dedicated allocation. That distinction mattered particularly for the Bonsai comparison.

The 32K column comes from saved context probes. The 64K column uses the highest recorded process peak across the available 64K measurements. The 128K column is an estimate, calculated from the change between the measured 32K and 64K context probes. It isn’t a 128K measurement or a guarantee that allocation remains linear in another runtime.

For planning, I allow 2 GiB for the desktop and other GPU use. That’s a chosen reserve, not a fixed overhead measured on every card. Your actual headroom depends on what else is running.

Bonsai PTQ1_0 used 7.48 GiB at 64K, leaving substantially more room under a 12 GB budget than Qwen3.8-27B’s 11.34 GiB. Qwen3.8-27B already used 10.64 GiB in the 32K probe, so it isn’t my comfortable 12 GB recommendation with that reserve.

On a 16 GB card, Qwen3.8-27B is the stronger projected choice at the measured 64K setting. Its 12.75 GiB estimate at 128K is useful for planning, but I’d begin with 64K and verify memory use on the actual card before increasing it.

There is one measured long-context Bonsai result worth separating from those estimates. PTQ1_0 reached 11.70 GiB of process VRAM with a 262,144-token allocation and Q4/Q4 cache. With Q8/Q8 cache at that allocation, it reached 15.70 GiB. Cache precision has a meaningful memory cost.

But allocating a large context doesn’t establish that the model uses it well. Bonsai’s long-context quality results are the main reason I wouldn’t choose it just to advertise the biggest context window.

Long-document lookup and reasoning give different answers

Classic HomHaystack includes 100 real keys and 20 nonexistent keys per request. Half the composite score comes from finding the real values; the other half comes from leaving nonexistent keys without a parsed numeric answer.

That scoring split can conceal very different behavior. At 240K, Qwen3.8-27B, Qwen3.5 9B, Qwen3.5 4B and Ornith all scored close to 49. Their real-value retrieval was 98–99%, but they returned numeric answers for all twenty nonexistent queries. They earned no credit on that half of the score.

So that cluster wasn’t evidence that those models retrieved almost nothing and received a 50-point floor. The raw component counts show the opposite: strong retrieval of present values, paired with poor handling of absent ones under this prompt.

There’s a prompt limitation too. The saved Classic prompt didn’t explicitly prescribe a special response marker for an absent key. I wouldn’t turn this particular absence score into a general claim about whether a model is truthful.

Here are the positive retrieval counts accepted by the required answer parser with Q8/Q8 cache:

Bonsai has four Q8 executions per packing of the same suite, so its denominators are larger. The other files have one selected suite each. These counts pool the recorded requests; they don’t imply equal replication across models.

The Qwen3 Instruct 128K count includes one request that reached its context limit. Treat that aggregate as observed coverage, not a fully converged suite. Ministral 14B’s 240K response used Markdown that the required parser rejected. Its official score remains zero. An offline check that removes only bullet and bold wrappers finds 98 of 100 real values correctly written, alongside numeric answers to all twenty absent queries. That diagnostic reading is separate from the official score. Ministral 3B also had two 128K responses rejected for line prefixes, so its strict-parser count understates the values a person can read. Gemma’s 240K response stopped normally but retrieved only 12 of 100 real values; its composite score of 6 follows from those retrieval results and no absence credit.

At 128K, Qwen3.5 9B is a useful alternative: it found 499 of 500 real values and left all 100 nonexistent queries without a parsed numeric answer. At 240K it still retrieved 98 of 100 real values, but it lost that absence handling. Good lookup at one input length doesn’t settle the result at another.

Finding the values isn’t enough

Reasoning 128K requires the model to find four values, associate them with the requested keys in order, and compute a sum, maximum, minimum or range. A fully correct answer needs the retrieval, ordering and final result to be right.

Qwen3.8-27B completed 53 of 60 question executions fully correctly. Ministral 14B was next at 31 of 60. Qwen3.5 9B managed 15 of 60, despite its excellent Classic lookup. That’s why I separate finding information from reasoning over it.

The failure details make that distinction concrete. Qwen3.5 9B had the four values correctly associated with their keys on 42 of 60 questions, but 27 of those 42 still had an incorrect final result. A retrieval-only score would miss that problem.

For Bonsai, each packing separately answered 24 of 240 question executions fully correctly across four Q8 suite executions. Those cover 60 distinct questions, with identical saved outputs across repeats. The separate Q4 experiments scored lower under the required format: PTQ1_0 completed 6 of 180 fully correctly, and PQ2_0 completed 3 of 180. Those Q4 runs retrieved 156/300 and 162/300 real values at 240K, respectively. Q4 reasoning answers sometimes used different separators, causing strict-format rejection. These scores do not isolate a reasoning loss caused by cache precision. The Q4 denominator likewise repeats the same 60 questions three times.

That limits my Bonsai recommendation. I like it for the general tasks in QuantBench within a smaller memory budget. I wouldn’t choose it for dependable reasoning across an extremely long input based on these results.

The main long-document results also used Q8 cache on the 32 GB test card. They don’t validate the same accuracy with Q4 cache at very long context on a 12 GB or 16 GB card. The memory and quality settings need to stay attached to their results.

Generation speed doesn’t tell you how long you’ll wait

These are thinking-on QuantBench results on the RTX 5090. Generation tok/s is the arithmetic mean of the recorded generation rates per completed task execution. It excludes prefill from the generation-rate metric. Bonsai’s rates come from llama.cpp’s evaluation timing logs; the other models use the recorded runtime generation rates.

Median task time measures elapsed task time, including the wait while the model thinks and answers. It isn’t the same as generation speed, and it isn’t an isolated short-output speed test.

These timing aggregates include the completed repeat executions, including the baseline-seed repeats excluded from the quality headline. They describe the recorded execution pool rather than the equal-task scoring formula. Ornith has fewer completed executions because I stopped further testing.

Qwen3.5 4B generates tokens quickly, but Bonsai PQ2_0 had the shortest median task time among these thinking-on results. Qwen3.8-27B took longer than either Bonsai packing while returning the stronger quality score. The useful choice depends on whether that extra wait buys you a better answer for your work.

Ornith shows why tok/s alone is a poor recommendation metric. It generated about 132 tokens/sec, yet its median task time was 144.2 seconds. In its full thinking-on run, 9 of 50 tasks reached the context cap. That’s 18% of that full run, not 18% of every execution across the campaign.

I stopped additional Ornith, Ministral 3B and Ministral 8B work because of the time involved and the results I was getting. Their completed captures remain in the relevant comparisons. An aborted follow-up isn’t a zero score, and a completed full run doesn’t mean every planned follow-up finished.

I also wouldn’t use this campaign to declare that every Ornith deployment performs this way. These are results for the tested file, runtime and settings. The saved thinking-on sampler and single-user prompt renderings matched the provider recommendations checked during the audit. That narrows the configuration concern, but does not establish the cause of the poor responses or replicate every provider benchmark condition.

Use the speed table to understand this test rig. I haven’t measured these workloads on your 12 GB or 16 GB GPU, and I wouldn’t promise a target-card token rate by scaling the 5090 number.

Downloads and detailed settings

The benchmark explorer carries the exact file identities, component counts, timing definitions and per-result runtime records. The AI tools expose the same published data. These repository links identify the tested sources; a later download may differ from the recorded file.

• Qwen3.8-27B Heretic GSQ-RCO IQ3_XXS: RVN-Qwen3.8-27B-Heretic-GSQ-RCO-IQ3_XXS-mtp.gguf. Repository revision was not recorded; the benchmark result details include the recorded file hash.
• Bonsai 2 27B PTQ1_0: Ternary-Bonsai-2-27B-PTQ1_0.gguf. Recorded revision: 6ed5e12bf84b7a63069882c91dd9e9218647d17b.
• Bonsai 2 27B PQ2_0: Ternary-Bonsai-2-27B-PQ2_0.gguf. Recorded revision: 6ed5e12bf84b7a63069882c91dd9e9218647d17b.
• Gemma 4 12B IT QAT Q4_0: gemma-4-12b-it-qat-q4_0.gguf. Repository revision was not recorded; the benchmark result details include the recorded file hash.
• Qwen3.5 9B Q5_K_M: Qwen3.5-9B-Q5_K_M.gguf. Repository revision was not recorded; the benchmark result details include the recorded file hash.
• Qwen3.5 4B Q5_K_M: Qwen3.5-4B-Q5_K_M.gguf. Repository revision: e87f176479d0855a907a41277aca2f8ee7a09523; the benchmark result details include the recorded file hash.
• Ornith 1.5 9B Q5_K_M: Ornith-1.5-9B-Q5_K_M.gguf. Repository revision was not recorded; the benchmark result details include the recorded file hash.
• Qwen3 4B Instruct 2507 Q5_K_M: Qwen_Qwen3-4B-Instruct-2507-Q5_K_M.gguf. Repository revision: ae44f08e1392f39c0e474af10c3ff8355c8b6688; the benchmark result details include the recorded file hash.
• Ministral 3 14B Q4_K_M: Ministral-3-14B-Instruct-2512-Q4_K_M.gguf. Repository revision: 74fac473c43357d7fb2671713608183cc72496d0; the benchmark result details include the recorded file hash.
• Ministral 3 8B Q4_K_M: Ministral-3-8B-Instruct-2512-Q4_K_M.gguf. Repository revision: 0102285ad796bd99af90f58de616092e5630e970; the benchmark result details include the recorded file hash.
• Ministral 3 3B Q5_K_M: Ministral-3-3B-Instruct-2512-Q5_K_M.gguf. Repository revision: eb599d408350ea2bb60452cb86be7c7b2fc28227; the benchmark result details include the recorded file hash.

Corrections accompanying the video

This article uses one consistent score basis for each table. Gemma’s task-weighted Off score rounds to 43.6. The Instruct table uses complete 50-task cohorts throughout. Bonsai’s Q8 reasoning result is 24/240 per packing, covering repeated executions of 60 distinct questions. Its Q4 results stay separate. The memory table distinguishes measured process peaks from 128K estimates. These clarifications govern the written results and linked benchmark data.

What I’d install

For 12 GB, I’d try Bonsai PTQ1_0 at 64K with thinking on and Q4/Q4 cache. Its combination of general task accuracy and measured memory use is the strongest reason to accept the PrismML runtime setup. I’d keep the context conservative until I’d checked actual GPU use during my own work.

For 16 GB, I’d choose the tested Qwen3.8-27B Heretic GSQ-RCO IQ3_XXS file, starting at the same 64K/Q4 settings. It leads this QuantBench comparison and the Q8 long-context reasoning test. Those are results for this particular community variant and quantization; they don’t establish that every Qwen3.8-27B file performs the same way, or isolate what its model modifications contributed.

For an LM Studio alternative, I’d choose Qwen3.5 9B Q5_K_M, especially for document lookup. I’d still verify answers that require calculations across retrieved information. Its lookup accuracy and its reasoning accuracy are very different results.

Qwen3.5 4B is also worth considering when memory is tighter. Its 4.35 GiB measured peak at 64K makes it a candidate for a projected 8 GB setup, with the same requirement to validate on the actual hardware.

In the companion YouTube video, I walk through the comparisons and the tradeoffs behind these picks. Let me know in the YouTube comments which GPU and workload you’re using, and how these configurations behave on your card. Subscribe to DeepWakeLabs if you want more local-model tests with the settings and limitations attached to the results.