Hi all!
i have written a couple of posts in the past, i am an illiterate having fun with LLMs and AI in general, who is being pulled in in a deeper hole by the days…
I have extensive experience with Linux (Gentoo lover since 20 years here) i am a sw dev now “promoted” to management, and avid tech user, so not really illiterate, but i know very little about all this LLM game.
I started with OpenWebUI + Ollama and played as an idiot with random models. Then come across an NVIDIA RTX A4000 (16gb VDDR6) and plugged into my I7-8700 server with 64gb RAM. The server has a Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] too, unused at this time (server is 100% headless anyway).
I am currently installing LocalAI to run llama.cpp and improve my models capability and speed, planning to ditch OpenWebUI and Ollama, if LocalAI + llama.cpp works fine.
My first usage was chatting with random local models. Then i discovered Fooocus and quickly upgraded to ComfyUI. Last, i have set up my SubWave radio station and i am having so much fun…
I have a few questions:
- Can i leverage both my NVIDIA and the iGPU at the same time?
- If i use the iGPU do i need to fixedly allocate RAM from the BIOS to it? Or will it use system RAM as needed?
- Using llama.cpp i want to leverage also CPU usage, since i have 64gb ram (also shared by many more self hosted stuff, tough) is there anything special i need to do to achieve that?
- What are a set of models that you guys recommend for my setup? I am currently using qwen2.5-coder:14b-instruct-q5_K_M with ollama, and i am pretty satisfied with it’s coding capabilities, but i want something more general purpose for my SubWave (AI assisted web radio channel)
- I might have the opportunity to install a second RTX A4000, identical to the first, on my server (need to check pci-e slot availability and power supply specs), would that make any sense at all?
- Power consumption wise, do the NVIDIA cards suck power also when not in active use?
- No, not really. For a start, that CPU / Igpu could at best have Vulkan support…except the i7-8700 is AXV2…which is about 4 gen out of date, so have dicey vulkan support in Windows / a little better in linux… Ask me how I know.
But even if you could, it would be slower than a real GPU for AI compute due to lack of tensors in the Igpu. You’d probably (definitely?) be better off just running straight on CPU in that case. And this is before even talking about different OS issues in windows / Linux…
Save yourself the pain; don’t try to force inference via Igpu; you’re gonna have a bad time.
-
It’d be dynamically allocated (generally speaking)…but see 1.
-
Sadly, it won’t help so much as you think…because you are memory bandwidth bound. Though I suppose it does allow you to run larger models (slowly) on CPU…if you hate yourself :)
Incidentally, I can tell you that Qwen 3.6 35B-A3B, at Q4_K_M, --ctx 8192, runs at 6-8 tok/s on that CPU…because I tested it myself (llama.cpp, beellama and ik_llama), with llama.cpp being fastest overall.
If you already have a 16GB, DDR6 GPU, find something that plays nicely with it.
If you’re asking about spill over or offloading a MoE, that should be automatic / with launch flags.
-
The very easy answer to this is “anything by Qwen”, because they are the community darlings. But…is your use case specifically AI radio? Because that maybe speaks to something like ACE-Step 1.5 (…which is a Qwen 3 derivative)
-
Maybe … but you’ll then have to deal with split inference across 2 GPU (or I suppose run different things on different GPUs). Possible but…you’ll have to look into orchestration.
-
They do, but much less than at peak. For example, if your GPU sucks down 250w at peak, it might idle at 100w. So if you have 2 GPUs + everything else, it might idle at 300w and peak near 650w. I suppose that’s OK if you’re not hammering it 24/7
Qwen3.6-27b and qwen3.6-35b-a3b are your starting point. Ask those how to configure your stuff. You can then move to something larger like a 120b model using some offloading if you want.
Run pi.dev and llama.cpp, use always the latest builds or compile from source.
The card does not eat energy when it is not in active use.
