Shifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 4 months agoSpent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.plus-squaremessage-squaremessage-square5linkfedilinkarrow-up126arrow-down13
arrow-up123arrow-down1message-squareSpent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.plus-squareShifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 4 months agomessage-square5linkfedilink
Shifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 4 months agoBitNet b1.58-2B-4T at 36 tok/s on Xeon - pure C99, no Python/BLAS, single binary for ternary + GGUFplus-squaremessage-squaremessage-square0linkfedilinkarrow-up16arrow-down12
arrow-up14arrow-down1message-squareBitNet b1.58-2B-4T at 36 tok/s on Xeon - pure C99, no Python/BLAS, single binary for ternary + GGUFplus-squareShifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 4 months agomessage-square0linkfedilink
