I actually ran a series of A|B split tests (using GPT, Claude, Qwen 27B, Qwen 35B, GLM) on some code I’d written.
The Qwen models managed to find issues the others missed and offer useful suggestions.
Coding wise, they’re a little too eager to take the next step / be a helpful assistant, and context collapse is a real thing with them. I would say yes, they are capable, and probably even more so in the Qwen specific coding harness.
The thing is, small models can only hold so much in their latent space. If you give them a big job or free range task, they will find a way to monkey paw it. They need short leash and test gates.
Pretty simple. People keep going on about how useful these local models are for coding. So what I wanted to do was to create a standardized test for myself to see if that was true before committing to anything.
( I think the various benchmarks out there are a bit fluffy, so I wanted to try it against a real workload.)
What I did was throw a bunch of money up at OpenRouter and then used Roo to call in diff models, one at a time.
I gave each the same task - that is, here is a piece of code, here is my ticket, here is my repo. Investigate what you want and then do what my ticket says.
I already knew what was wrong with the code, but I wanted to see how obedient the models are at sticking to a scoped ticket and what they would find.
By far the best bang for buck was GPT 5.4 mini. It is exceptionally obedient at doing exactly what you tell it as long as you tell it exactly what to do.
It won’t go off piste if properly constrained.
I think for light - med workloads, $20 on ChatGPT is a crimal steal. Chat and Codex have a separate usage pool.
I’m also aware that this is open AI’s lock in phase where they provide the samples of crack for free to get you hooked. And, yes, they are crack dealers in every sense of the word.
Anyway, it’s good to know that with a little bit of elbow grease and some smarts, the smaller models, which could reasonably be self-hosted, could do a decent enough job if they are narrowly scoped.
You’re probably not going to be able to yeet an entire code base at them and go “figure out what’s wrong and fix it” while you snooze tho, but I think that’s probably a good thing from a human in the middle perspective.
Do you find you can do coding tasks with it well?
I‘m not a coder, so I don’t know exactly. It is able to code, but I would say somebody with experience should guide it and have an eye on the results.
I actually ran a series of A|B split tests (using GPT, Claude, Qwen 27B, Qwen 35B, GLM) on some code I’d written.
The Qwen models managed to find issues the others missed and offer useful suggestions.
Coding wise, they’re a little too eager to take the next step / be a helpful assistant, and context collapse is a real thing with them. I would say yes, they are capable, and probably even more so in the Qwen specific coding harness.
The thing is, small models can only hold so much in their latent space. If you give them a big job or free range task, they will find a way to monkey paw it. They need short leash and test gates.
Neat, what did your setup look like? You mentioned the qwen harness? Run it all on one machine?
Pretty simple. People keep going on about how useful these local models are for coding. So what I wanted to do was to create a standardized test for myself to see if that was true before committing to anything.
( I think the various benchmarks out there are a bit fluffy, so I wanted to try it against a real workload.)
What I did was throw a bunch of money up at OpenRouter and then used Roo to call in diff models, one at a time.
I gave each the same task - that is, here is a piece of code, here is my ticket, here is my repo. Investigate what you want and then do what my ticket says.
I already knew what was wrong with the code, but I wanted to see how obedient the models are at sticking to a scoped ticket and what they would find.
By far the best bang for buck was GPT 5.4 mini. It is exceptionally obedient at doing exactly what you tell it as long as you tell it exactly what to do.
It won’t go off piste if properly constrained.
I think for light - med workloads, $20 on ChatGPT is a crimal steal. Chat and Codex have a separate usage pool.
I’m also aware that this is open AI’s lock in phase where they provide the samples of crack for free to get you hooked. And, yes, they are crack dealers in every sense of the word.
Anyway, it’s good to know that with a little bit of elbow grease and some smarts, the smaller models, which could reasonably be self-hosted, could do a decent enough job if they are narrowly scoped.
You’re probably not going to be able to yeet an entire code base at them and go “figure out what’s wrong and fix it” while you snooze tho, but I think that’s probably a good thing from a human in the middle perspective.