Ten models, four domains

I spent two weeks benchmarking ten language models, all running on my own machine. Each one got the same questions across the four things my local coaching system covers - Hindustani classical music, tennis, photography and career coaching.

The one I kept is Google’s Gemma4 26B-A4B, a Mixture of Experts model, in a Quantization-Aware Training build. It was the fastest of the ten that gave me usable output. For sessions that need harder reasoning I use gemma4’s 12B, with extended thinking switched on.

Table of contents

1. What I was trying to solve
2. The ten models
3. Speed and VRAM, the full table
4. Hindustani classical music
5. Tennis coaching
6. Photography, text coaching
7. Career coaching
8. Photography, visual critique
9. The thinking mode question
10. The MoE detour
11. Extended conversations and OOM
12. What I chose
13. What I learnt

What I was trying to solve

The previous article describes the system, a Hindustani classical music practice companion that runs locally and draws on a Wikipedia retrieval database of 1,385 articles. It’s a retrieval-augmented generation setup that answers in the voice of one of two lineages - the Dagarvani bani when I’m practising dhrupad on the surbahar, or the Etawah gharana when it’s khayal.

I started on qwen3.6:27b, and it worked. With thinking on it managed around 16 tokens per second - a full reply took 30–50 seconds. With the knowledge base out of the picture it got a fact wrong, putting the Dagarvani bani in the Maihar gharana of Allauddin Khan, a different lineage altogether. The bigger problem was memory. The model took up nearly all of the card’s VRAM, leaving hardly any room for the context to grow over a long conversation. So I wanted to know whether any model beat it on quality or on speed, ideally both.

The ten models

Everything ran locally under Ollama on Bazzite Linux, on an AMD Ryzen 7 9850X3D with an RX 9070 XT (16GB of GDDR6). The models fell into three groups. Dense models came first, qwen3.6:27b (the baseline), qwen3:14b, gemma4:12b, gemma4:31b, gemma4:e4b, and phi4-reasoning:plus. Then there were the deepseek-r1 variants. I dropped those after reading that their training was tuned for maths and code, when what I needed was nuance in language. Last were the Mixture of Experts models, two quantization builds of gemma4’s 26B-A4B, and Qwen3’s 30B-A3B from HuggingFace.

Community benchmarks had gpt-oss:20b as the fastest model anyone had run on my exact GPU, at 101 tokens per second. On mine it ran just as fast, with empty output every time. It uses MXFP4, and Ollama can’t yet pull the content out of it properly.

Speed and VRAM, the full table

All of these numbers come from the RX 9070 XT (16GB GDDR6, ROCm, Ollama 0.30.10). Speed is how fast it generated a 280-token response. Wall time runs from sending the prompt to the last token.

Hindustani classical music

I asked two music questions. One needed real knowledge of raag grammar - how the slow oscillation Darbari Kanada puts on komal Ga ought to shape phrasing down in the lower octave. The other asked about the Dagarvani idea of dwelling in a note and how to handle an aalap that starts to feel restless.

qwen3.6:27b made the Maihar mistake again. gemma4:12b called the surbahar a bowed instrument, when it’s plucked. Both went away as soon as the knowledge base returned the right passages. They still show holes in what these models learnt in training. Neither mistake came from the QAT or the 31B.

The 31B’s prose was the best of the lot. It used vazan and nyasa in context and without showing off. The single best line from any model, though, came from the 12B, answering the restlessness question, “your mind is living in the next phrase while your voice is still in the current one.”

Tennis coaching

The tennis pair were a diagnosis of a match that fell apart (won 6-3, then lost the second set 3-6 from 3-1 up, after tightening) and a technique question on keeping topspin on the forehand while moving sideways at full stretch.

Every model has seen plenty of tennis in training, and the gaps here were narrower than in music. Of the five diagnoses the 31B’s “Shift from Process to Outcome” was the sharpest, and its forehand drill used the right biomechanical terms. The QAT came close on depth.

Photography, text coaching

Text only here, on black and white technique. One question asked how to keep neighbouring greys apart on industrial buildings under a flat overcast sky. The other asked how to get better at Zone System pre-visualisation.

Here the smaller models began to differ more visibly. gemma4:12b’s “texture as proxy for value” is exactly what someone shooting black and white at expert level needs to hear. The 31B reached for Ansel Adams’ own “score vs performance” analogy and applied it to pre-visualisation correctly. The QAT kept up at 85 tokens per second.

Career coaching

Two questions again - one from a software engineer seven years in who wants to move into AI product management, the other from a mid-level engineer with three weeks to position for a promotion.

Of the four, career is the most generic. Every model knew enough, and the scores sat closer together here than anywhere else. At 86 tokens per second against 7, the QAT gave me comparable advice twelve times sooner.

Photography, visual critique

Seven vision-capable models got the same standard greyscale gradient and a request for a critique covering contrast, composition and tonal range.

Qwen2.5-VL was quickest, at 116 tokens per second, and it and Qwen3-VL both read the tonal structure of the image accurately. MiniCPM-V 4.5 picked up that it was a gradient and described the move from shadow to highlight in specific terms. Pixtral 12B is good with artistic and photographic work, and its critique was competent, though no better than the two Qwen models here. gemma4:e4b got the file, seemed to throw it away, and then asked me for the photograph I’d just sent it.

For critiquing real photographs, Qwen3-VL:8b is the main model. MiniCPM-V 4.5 is there when very high-resolution analysis matters - it takes images up to 1.8 megapixels and gets by on fewer visual tokens than most.

The thinking mode question

Turning thinking on made the answers to nuanced questions better every time, and it was expensive. With a 300-token budget the thinking regularly used up every token before any visible answer appeared, and getting something useful took 1,200–1,500 tokens.

Put side by side, 51 seconds a turn against 8 means the thinking 12B is over six times slower, for answers of about the same quality. On hard analytical questions the thinking made the 12B noticeably better, just not often enough to beat the bigger QAT answering straight away.

The MoE detour

The gemma4:26b at the standard Q4_K_M (16.8GB) needed 13.4GB of VRAM. After its first response it ran out of memory and crashed. The KV cache grew with the conversation history until it filled whatever VRAM was left.

The QAT build is 14.4GB, smaller though it’s the same model. It needed 13.8GB of VRAM and stayed stable for the full ten turns of the test. The QAT left 3.1GB free for the KV cache, where the standard one left less than 1GB.

Qwen3’s 30B-A3B MoE did worse, at Q3_K_M from HuggingFace. Three-bit quantization is aggressive, and it hurt the model so much that its raw reasoning spilled into the output in place of a properly formatted answer.

Extended conversations and OOM

The last test was a practice session of ten exchanges in a row, each one building on the one before. Of everything I measured, this told me the most.

Neither model ran out of memory over the ten turns, and both kept giving substantive answers to the end. The 31B kept its musical vocabulary a little more precise, with vazan and nyasa still turning up naturally in the later turns. The 12B was done in 8.5 minutes and the 31B took 40, and for daily use that small gain in quality isn’t worth the wait.

What I chose

Daily sessions run on the gemma4:26b QAT. It comes from Unsloth’s HuggingFace repository by way of Ollama’s direct HuggingFace integration (hf.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL). It does 85–91 tokens per second with 64K of context and needs 13.8GB of VRAM, and a response takes 4–8 seconds. It took over both the fast tier and the deep reasoning tier - those used to be two separate models. For careful reasoning I switch to gemma4:12b, thinking on, with a budget of 1,500 tokens, at about 51 seconds a turn. Critiques of actual photographs go to Qwen3-VL 8B, with MiniCPM-V 4.5 as the high-resolution alternative. gemma4:31b stays installed for the odd session when prose quality is all I care about.

What I learnt

A smaller model that thought for 900 tokens was no more accurate than a bigger one that answered in 300. That model is an MoE. MoE cuts the compute per token and does nothing for memory. On a card with limited VRAM holding a long conversation, storing the whole model ate whatever it saved on compute. The QAT build beat the standard Q4_K_M build every time, smaller, faster and about as accurate. Qwen2.5-VL, at 5.6GB and 116 tokens per second, beat gemma4:12b at image analysis although it’s the smaller model.