Estebiu

Uhh, a lot of big words here. I mostly just play around with it… Never used LLMs for anything more serious than a couple of test, so I don’t even know now many tokens can my setup generate…

You can still run smaller models on cheaper gpus, no need for the greatest gpu ever. Btw, I use it for other things too, not only LLMs

For llama 70B I’m using an rtx a6000; slightly older but it does the job magnificently with hers 48gb of vram.