I have an HP Z2 G9 where I’ve replaced the original A2000 GPU with a 5060Ti. I’m pondering getting another 5060Ti to be able to go up to dense models instead of MoE, but the PCIe options on this motherboard aren’t the best. I will need to use a PCIe extender no matter what I opt for since there’s physically no room for another GPU (the 5060 I have is 2.5 slot sized) and so I’m wondering whether I would even gain something from using an M2 slot adapter instead.
Any insight welcome.
(1) PCI Express Gen5 slot x16 mechanical/ x16 electrical (full height, full length)
(1) PCI Express Gen3 slot x4 mechanical/ x1 electrical (full height, full length, open-ended)
(1) PCI Express Gen3 slot x16 mechanical/ x4 electrical (full height, full length)
(1) PCI Express Gen3 slot x4 mechanical/ x4 electrical (full height, full length, open-ended)
(1) M.2 2280 Storage (PCIe Gen4 x4)
(1) M.2 2280 Storage (PCIe Gen4 x4)
(1) M.2 2280 Storage (PCIe Gen4 x4)
Inference doesn’t actually need gobs of PCIe bandwidth, the model gets split in half at the layer level and you just pass the output of one card to the input of the next. I have 2x7800XTs going and one is on (chipset) PCIe4x4 and it works fine. External via eGPU can also be done.
Yes, if you use layer parallelism. When using tensor parallelism PCIe bandwith is a very important factor.
True, but not hugely relevant to models fitting in 32GB as I understand it, more for improving latencies in larger models (120B+) or batch processing. Nonetheless layer will work for OPs use case.
I use a lot of PCIe extenders to abuse my consumer motherboard with too many GPUs. You should be fine as long as your processor and chipset have enough lanes and power. Cheap extenders can be had for like $10.
