This is my upgraded rig:
- Ryzen9 5950x with 64gb DDR4
- Dual NVIDIA RTX A4000 (16+16GB VRAM)
to whom read my previous posts, i jumped the gun and upgraded my server, it was worthwhile and somewhat cheap given i already had the two GPUs and the DDR4 RAM.
Anyway, i am currently running Qwen3.6-35B-A3B-UD-Q5_K_XL all in VRAM with 65536 context and pretty happy with speed (80-90t/s) and overall responses (mostly chat).
I would like to experiment with something beefier, with CPU offload, that i can run with my llama.cpp. Of course t/s is not a goal here, but precision and accuracy of responses is.
I tried to find a good model with claude and gemini, but always got short. Once the model suggested fully crashed my server (guess fill up RAM and ended up in a swap loop), more then once i ended up chasing non existent models. Pretty annoying.
Considering i would only use between 32 and 48GB or system RAM, can you suggest (preferably with links to HF) some models?
I like qwen3.6, but open to anything.
I’m fond of this model. In “thinking” mode
gemma-4-31B-it-uncensored-heretic-BF16.gguf
Edit:
There’s also this beast I’ve been curious about. https://github.com/JustVugg/colibri using deep seek v4 flash. 280B-13b moe.
What are you wanting to do with the model?
Agentic development with that setup, you can easily run a good quant of Qwen3.6-27B at full context. unsloth/Qwen3.6-27B-MTP Q5_K_XL and use a fixed template
Creative writing? Probably need a different model. I’ve heard good things about Gemma 4 31B.
Both of those are dense models so won’t be as fast as Qwen3.6-35B-A3B. Also play around with MTP, quants, KV cache quant, KV caching, etc.
The RAM size will limit you on larger models unless you stream from storage.
