Ten models, four domains

First, the short version. After building a local coaching system for Hindustani classical music, tennis, photography, and career coaching, I spent two weeks systematically benchmarking ten language models that run on my own machine to find the right ones. The system is a RAG one, retrieval-augmented generation, which means the model is handed passages from a knowledge base before it answers rather than relying on what it remembers. The winner was a Mixture of Experts model (Google’s Gemma4 26B-A4B in a Quantization-Aware Training build), running at 85–91 tokens per second while using only 13.8GB of VRAM. A Mixture of Experts model is one split into specialist parts, only a few of which do the work on any one word. Quantization is keeping each of the model’s numbers to fewer bits, which makes the whole thing smaller, and a Quantization-Aware Training build is one trained with that squeeze in mind. A token is a word or a piece of one, so tokens per second is the rate at which the model writes, and VRAM is the memory on the graphics card, where the whole model must sit while it runs. For sessions that need deeper reasoning, a smaller 12B model with extended thinking mode turned out to be the right call, 12B meaning twelve billion parameters, the numbers inside a model that training sets, and thinking mode meaning that the model writes out a chain of reasoning before it answers. Getting to those conclusions involved two hardware crashes, one architecture I’d never heard of, and a 40-minute test run that went nowhere useful.

Table of contents

1. What I was trying to solve
2. The ten models
3. Speed and VRAM, the full table
4. Hindustani classical music
5. Tennis coaching
6. Photography, text coaching
7. Career coaching
8. Photography, visual critique
9. The thinking mode question
10. The MoE detour
11. Extended conversations and OOM
12. What I chose
13. What I learned

What I was trying to solve

The system I described in the previous article (a local AI practice companion for Hindustani classical music, grounded in a Wikipedia retrieval database) works by pairing a language model with passages retrieved from 1,385 articles. The model’s job is to synthesise what the knowledge base returns and speak in a specific voice, namely the Dagarvani bani for dhrupad practice on the surbahar and the Etawah gharana for khayal. A bani or a gharana is a lineage of musicians with a style of its own, dhrupad and khayal are the two styles I practise, and the surbahar is the larger sitar with the deeper voice. The knowledge base handles factual accuracy, and the model handles how to say things.

The original model was qwen3.6:27b, a name that gives the model family, its version and its size in billions of parameters. It worked, but it was slow - around 16 tokens per second with thinking enabled, and 30–50 seconds for a full response. Without the knowledge base it also made a factual error, attributing the Dagarvani bani to the Maihar gharana of Allauddin Khan, which is wrong. More importantly, it was sitting right at the VRAM limit of the card, which left almost no headroom for the context window to grow across a long conversation, the context window being the amount of text the model holds in mind at once, which grows with every turn.

The question was whether something better existed (better on quality, faster, or both), and whether I could find it without spending a week reading specifications.

The ten models

I ran all the models locally on an AMD Ryzen 7 9850X3D with an RX 9070 XT (16GB GDDR6), using Ollama, the program that downloads and runs models on your own machine, on Bazzite Linux.

The candidates came in three categories. The first category was dense models of various sizes, dense meaning that every parameter works on every token, namely qwen3.6:27b (the baseline), qwen3:14b, gemma4:12b, gemma4:31b, gemma4:e4b, and phi4-reasoning:plus. The second was dedicated reasoning models, the deepseek-r1 variants, which I eliminated after research showed their training was optimised for mathematics and code rather than nuanced language tasks. The third was Mixture of Experts models, with gemma4’s 26B-A4B architecture in two quantization builds and Qwen3’s 30B-A3B from HuggingFace, the site where models are shared. In those names, 26B-A4B means 26 billion parameters of which 4 billion are active on any one token.

One model, gpt-oss:20b, appeared in community benchmarks as the fastest thing tested on the exact GPU I was using, at 101 tokens per second. It generated tokens happily but produced empty output - the MXFP4 quantization format it used, a newer four-bit one, is too new for Ollama to extract content from properly.

Speed and VRAM, the full table

I measured all the numbers on the RX 9070 XT (16GB GDDR6, ROCm, Ollama 0.30.10), ROCm being AMD’s software for running this kind of work on its graphics cards. Generation speed is in tokens per second for a 280-token response, and wall time is end-to-end, including prompt processing, the time the model spends reading the question before it writes. The quantization names in the first column (Q4_K_M, UD-Q4_K_XL, Q3_K_M) are recipes for how many bits each number is kept in, roughly four, four and three.

Hindustani classical music

The music test had two questions. The first required technical knowledge of raag grammar, specifically the relationship between the slow oscillation on komal Ga in Darbari Kanada and how it should inform phrasing in the lower octave, komal Ga being the flattened third note and Darbari Kanada a grave raag of the late night. The second asked about the Dagarvani tradition’s philosophy of dwelling in a note, and what to do when aalap, the unhurried opening of a raag before any rhythm begins, feels restless.

The two errors are instructive. qwen3.6:27b attributed the Dagarvani bani to the Maihar gharana of Allauddin Khan, which is the wrong lineage entirely, and gemma4:12b described the surbahar as a bowed instrument, whereas it is plucked. Both errors disappear when the retrieval knowledge base returns the correct passages, but they indicate gaps in baseline parametric knowledge that matter in edge cases. The QAT model and the 31B made neither error.

On prose quality, the 31B introduced terms like vazan (weight) and nyasa (the resting point of a note) naturally and in context, rather than decoratively. The 12B produced the best single phrase across all models on the restlessness question: “your mind is living in the next phrase while your voice is still in the current one.”

Tennis coaching

The two tennis questions were a match collapse diagnosis (won 6-3, lost 3-6 after tightening at 3-1 up) and a technical question about maintaining forehand topspin, the forward spin that dips the ball back into the court, while moving laterally at full stretch.

The tennis domain is well covered in training data across all models - the quality differences were smaller here than for music. The 31B’s diagnosis “Shift from Process to Outcome” was the sharpest frame, and its forehand drill description used correct biomechanical terminology. The QAT matched it in speed while producing nearly comparable depth.

Photography, text coaching

This was text-only coaching on B&W photographic technique, with one question about creating tonal separation from flat overcast light on industrial architecture, and one about developing Zone System pre-visualisation. Tonal separation is keeping the greys of one surface distinct from the next, and the Zone System is the method of deciding the tones of the final print before the picture is taken.

Photography coaching is where the smaller models started to diverge more visibly. gemma4:12b’s phrase “texture as proxy for value” (using surface detail to create tonal separation when light isn’t doing the work) is exactly the expert-level insight a B&W photographer needs. The 31B correctly used Adams’ own “score vs performance” analogy for pre-visualisation, and the QAT held its own at 85 tokens per second.

Career coaching

I tested two questions on the same models, one about a seven-year software engineer wanting to transition into AI product management, and one about a mid-level engineer wanting to position for promotion in a three-week window.

Career coaching is the most generic of the four domains, where all the models had adequate knowledge and the quality differences were the smallest. When quality is roughly equivalent across models, the QAT’s speed advantage becomes decisive. At 86 tokens per second versus 7, comparable advice arrives twelve times faster.

Photography, visual critique

This is a separate category, for models that can actually receive and analyse photographs rather than only give text advice about photography. I tested seven vision-capable models with a standardised greyscale gradient image, a test picture that runs smoothly from black to white, and asked for a B&W photograph critique covering tonal range, contrast, and composition.

Qwen2.5-VL was the fastest at 116 tokens per second, and both Qwen VL models produced accurate analysis of the image’s tonal structure. MiniCPM-V 4.5 was notable for its analytical framing, where it recognised the gradient composition and commented on the transition from shadow to highlight with specific language. Pixtral 12B, which handles artistic and photographic content well, produced a competent critique but did not meaningfully outperform the Qwen VL models on this test. gemma4:e4b received the image file but appeared to discard it, and asked the user to provide the photograph it had already been sent.

For photography critique in practice, Qwen3-VL:8b is the primary model (fast, accurate, stable), with MiniCPM-V 4.5 as an alternative for sessions where very high-resolution image analysis matters (it supports images up to 1.8 megapixels with fewer visual tokens than most models). A visual token is one of the pieces an image is cut into for the model.

The thinking mode question

Several models support a reasoning mode, where the model produces an internal chain of thought before answering. Enabling it consistently improved the quality of responses to nuanced questions, but with a significant cost. A 300-token budget was routinely consumed entirely by the thinking chain before a single word of visible response appeared, and useful results required 1,200–1,500 tokens.

The direct comparison settled the question. At 51 seconds per turn versus 8 seconds, the 12B with thinking takes more than six times as long for responses that are roughly equal in quality. The thinking chain improved the 12B’s outputs noticeably on genuinely hard analytical questions, but not enough to consistently beat the larger QAT model answering directly. For everyday journaling the QAT without thinking wins, and for specific hard questions where precision matters, the 12B with thinking is worth the wait.

The MoE detour

Mixture of Experts models were the most interesting detour and the most disappointing practical result. The architecture routes each token through a subset of expert networks rather than the full model, which in theory offers the quality of a large model at the compute cost of a small one.

The gemma4:26b in standard Q4_K_M quantization (16.8GB) used 13.4GB of VRAM and crashed with an out-of-memory error after the first response, because the growing KV cache from conversation history, the model’s running memory of everything said so far, filled the remaining VRAM.

The QAT build at 14.4GB (smaller despite containing the same model) used 13.8GB of VRAM and remained stable across all ten test turns. The difference was headroom - the QAT build left 3.1GB for the KV cache, and the standard build left under 1GB.

Qwen3’s 30B-A3B MoE, pulled from HuggingFace at Q3_K_M quantization, was worse. The aggressive 3-bit quantization degraded the model enough that it leaked its raw reasoning process as visible output rather than producing a properly formatted response. The speed was extraordinary at 100 tokens per second, but the output was unusable.

Extended conversations and OOM

The ten-turn conversation test (ten consecutive exchanges of a practice session, each building on the previous one) produced the most useful practical finding.

The num_ctx row is the context window setting, in tokens. Neither model crashed across the full ten turns. Both produced substantive responses throughout, with the 31B maintaining slightly more precise musical vocabulary (the terms vazan and nyasa appeared naturally on later turns). The 12B completed the session in 8.5 minutes, whereas the 31B took 40 minutes, and for daily use the quality gap does not justify that time cost.

What I chose

For daily sessions I chose the gemma4:26b QAT from Unsloth’s HuggingFace repository, pulled via Ollama’s direct HuggingFace integration (hf.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL). It runs at 85–91 tokens per second, supports 64K context, which is sixty-four thousand tokens held in mind at once, uses 13.8GB of VRAM, and produces responses in 4–8 seconds. It replaced both the fast and the deep reasoning tiers that previously ran separate models.

For sessions where careful reasoning matters, it’s gemma4:12b with thinking enabled and a 1,500-token budget, at ~51 seconds per turn.

For photography critique with actual images, Qwen3-VL 8B is the primary model and MiniCPM-V 4.5 is the high-resolution alternative.

The gemma4:31b stays installed for occasional use, when maximum prose quality is the priority.

What I learned

Running ten models across four domains taught me more about the tradeoffs than reading specifications ever would have.

Thinking mode is not always better. A model thinking for 900 tokens before answering is not more accurate than a larger model answering in 300 tokens. The reasoning chain adds precision on genuinely hard analytical tasks and adds time everywhere.

MoE architecture solves a different problem than the one I had. It reduces compute per token but not memory, and on a constrained VRAM card used for multi-turn conversation, where memory is the actual constraint, the storage cost of the full model cancels out the compute savings. The architecture makes more sense for server inference, where one machine answers many users at once, than for a personal practice companion running one conversation at a time.

Quantization-Aware Training is worth seeking out. The QAT build consistently outperformed the standard Q4_K_M of the same model (smaller, faster, and comparably accurate), because the quantization was part of training rather than applied afterwards.

The knowledge base still matters more than the model. The factual errors that appeared in models running without retrieval (the wrong gharana attribution and the wrong instrument classification) disappeared when the correct passages were returned by the knowledge base, hence the model’s parametric knowledge is a floor, not a ceiling. Parametric knowledge is what a model absorbed in training, as opposed to what it is handed when asked.

Vision models are a separate category. The best text models and the best vision models are not the same models at this hardware tier, and choosing a vision model requires different criteria than choosing a reasoning model. Qwen2.5-VL at 116 tokens per second and 5.6GB outperforms gemma4:12b for image analysis despite being smaller, because it was trained specifically for vision understanding.