Hey everyone! I was just skimming through some inference benchmarks of other people and noticed the driver version is usually mentioned. It made me wonder how relevant this is. My prod server runs Debian 12 so the packaged nvidia drivers are rather old, but I’d prefer not to mess with the drivers if it won’t bring a benefit. Does any of you have any experience or did do some testing?
On AMD hardware, I moved from rocm 6 to 7 and the associated amdgpu driver release and saw pretty noticeable inference performance improvements on an RX 7900 XTX with llama.cpp (as of rocm 7.0.2, AMD has a Debian Trixie build, BTW).
But I imagine that AMD/Nvidia, specific hardware, application, and settings are probably all major inputs into that. YMMV.
EDIT: Not to mention model being run.