I don’t host it exactly, just use it when I don’t use my graphics card for gaming. I run Qwen3.6-35b on my 16gb vram RX 9700 xt with 34t/s. I use it as an IT advisor, admin and Linux teacher for my cachyOS gaming PC.
I actually ran a series of A|B split tests (using GPT, Claude, Qwen 27B, Qwen 35B, GLM) on some code I’d written.
The Qwen models managed to find issues the others missed and offer useful suggestions.
Coding wise, they’re a little too eager to take the next step / be a helpful assistant, and context collapse is a real thing with them. I would say yes, they are capable, and probably even more so in the Qwen specific coding harness.
The thing is, small models can only hold so much in their latent space. If you give them a big job or free range task, they will find a way to monkey paw it. They need short leash and test gates.
Pretty simple. People keep going on about how useful these local models are for coding. So what I wanted to do was to create a standardized test for myself to see if that was true before committing to anything.
( I think the various benchmarks out there are a bit fluffy, so I wanted to try it against a real workload.)
What I did was throw a bunch of money up at OpenRouter and then used Roo to call in diff models, one at a time.
I gave each the same task - that is, here is a piece of code, here is my ticket, here is my repo. Investigate what you want and then do what my ticket says.
I already knew what was wrong with the code, but I wanted to see how obedient the models are at sticking to a scoped ticket and what they would find.
By far the best bang for buck was GPT 5.4 mini. It is exceptionally obedient at doing exactly what you tell it as long as you tell it exactly what to do.
It won’t go off piste if properly constrained.
I think for light - med workloads, $20 on ChatGPT is a crimal steal. Chat and Codex have a separate usage pool.
I’m also aware that this is open AI’s lock in phase where they provide the samples of crack for free to get you hooked. And, yes, they are crack dealers in every sense of the word.
Anyway, it’s good to know that with a little bit of elbow grease and some smarts, the smaller models, which could reasonably be self-hosted, could do a decent enough job if they are narrowly scoped.
You’re probably not going to be able to yeet an entire code base at them and go “figure out what’s wrong and fix it” while you snooze tho, but I think that’s probably a good thing from a human in the middle perspective.
That Qwen 35B model is going to remain the people’s champ for a long time I think. Surprisingly capable, even for code. I hear it loops badly at Q4 quant?
Probably that plus a higher quant solves it. Thing is most of us default to Q4_K_M as “precise enough”… and that seems to be kryptonite for the new Qwen’s.
That’s another thing with hosting AI that’s not often discussed. Sure, you can maybe run that 27B model…but if it’s at Q3_XS it’s going to be … “mentally challenged”.
I’ve heard the Gemma models with QAT are meant to be near full precision at Q4 size. Haven’t tried em yet.
Actually, on that topic - I’ve heard there’s a different architecture (RWKV), that’s supposed to be much more efficient for long context because it uses an entirely different KV system.
Sadly, there are few RWKV native models and retraining a standard transformer to RWKV seems like a pain in the ass. I’d need to hire a cloud GPU, distill into a different architecture, mess with datasets … honestly ICBF.
I don’t host it exactly, just use it when I don’t use my graphics card for gaming. I run Qwen3.6-35b on my 16gb vram RX 9700 xt with 34t/s. I use it as an IT advisor, admin and Linux teacher for my cachyOS gaming PC.
Do you find you can do coding tasks with it well?
I‘m not a coder, so I don’t know exactly. It is able to code, but I would say somebody with experience should guide it and have an eye on the results.
I actually ran a series of A|B split tests (using GPT, Claude, Qwen 27B, Qwen 35B, GLM) on some code I’d written.
The Qwen models managed to find issues the others missed and offer useful suggestions.
Coding wise, they’re a little too eager to take the next step / be a helpful assistant, and context collapse is a real thing with them. I would say yes, they are capable, and probably even more so in the Qwen specific coding harness.
The thing is, small models can only hold so much in their latent space. If you give them a big job or free range task, they will find a way to monkey paw it. They need short leash and test gates.
Neat, what did your setup look like? You mentioned the qwen harness? Run it all on one machine?
Pretty simple. People keep going on about how useful these local models are for coding. So what I wanted to do was to create a standardized test for myself to see if that was true before committing to anything.
( I think the various benchmarks out there are a bit fluffy, so I wanted to try it against a real workload.)
What I did was throw a bunch of money up at OpenRouter and then used Roo to call in diff models, one at a time.
I gave each the same task - that is, here is a piece of code, here is my ticket, here is my repo. Investigate what you want and then do what my ticket says.
I already knew what was wrong with the code, but I wanted to see how obedient the models are at sticking to a scoped ticket and what they would find.
By far the best bang for buck was GPT 5.4 mini. It is exceptionally obedient at doing exactly what you tell it as long as you tell it exactly what to do.
It won’t go off piste if properly constrained.
I think for light - med workloads, $20 on ChatGPT is a crimal steal. Chat and Codex have a separate usage pool.
I’m also aware that this is open AI’s lock in phase where they provide the samples of crack for free to get you hooked. And, yes, they are crack dealers in every sense of the word.
Anyway, it’s good to know that with a little bit of elbow grease and some smarts, the smaller models, which could reasonably be self-hosted, could do a decent enough job if they are narrowly scoped.
You’re probably not going to be able to yeet an entire code base at them and go “figure out what’s wrong and fix it” while you snooze tho, but I think that’s probably a good thing from a human in the middle perspective.
That Qwen 35B model is going to remain the people’s champ for a long time I think. Surprisingly capable, even for code. I hear it loops badly at Q4 quant?
Looping was a problem after reaching a certain context window size. The llama.cpp flags - -flash-attn on and looping penalties helped.
Probably that plus a higher quant solves it. Thing is most of us default to Q4_K_M as “precise enough”… and that seems to be kryptonite for the new Qwen’s.
That’s another thing with hosting AI that’s not often discussed. Sure, you can maybe run that 27B model…but if it’s at Q3_XS it’s going to be … “mentally challenged”.
I’ve heard the Gemma models with QAT are meant to be near full precision at Q4 size. Haven’t tried em yet.
Actually, on that topic - I’ve heard there’s a different architecture (RWKV), that’s supposed to be much more efficient for long context because it uses an entirely different KV system.
Sadly, there are few RWKV native models and retraining a standard transformer to RWKV seems like a pain in the ass. I’d need to hire a cloud GPU, distill into a different architecture, mess with datasets … honestly ICBF.
Yeah, a higher quant would be nice, I actually try not to go below Q5, but you can domino’s so much with 16GB of VRAM and the ddr4 system RAM.
But I must say I‘m pretty impressed by Qwen3.6-35b, not only from its capabilities but also from hardware requirements. MoE for the win I guess.
RWKV sounds interesting, have to look into it, thanks!