It’s been a while! Let us know what you’re currently working on, what exploded, what cool stuff you found, or just brag that everything is running fine unattended.

I haven’t been doing a lot, renovate auto upgrades for patches and sometimes minor updates work great so far. I got involved in my union to spin up a mastodon server soon, so there’s a new project, but I already run mastodon so it shouldn’t be much of a deal, I just need to get comfortable with the infrastructure. I also set up ntfy but only sent some test notifications and didn’t configure any app yet - planned for Element Chat.

[Meta] I’m not sure how to apply the new tags. In the recent versions of this thread, people sometimes posted about their ollama setup or the like, but I’m not mentioning AI in the post body (except this meta thingy) and it shouldn’t be much of the discussion, but it’s an open post. Maybe curbstickle can chime in to let me know if I should apply one of them.

  • brucethemoose@lemmy.worldEnglish
    6·
    3 hours ago

    [AIT] I know this isn’t everyone’s cup of tea, but I’m excited about Deepseek V4 flash. It’s (for me) the perfect size and architecture to self-host an LLM.

    My box (and brain) are chugging through a queue:

    • Figure out why my swap is going crazy, and how to ban processes from it [Done].

    • Figure out why Code OSS is unhappy [Partially Done].

    • Make an ik_llama.cpp iMatrix for Deepseek V4 [Done].

    • Figure out why quantization isn’t working [Done].

    • Make a test IQ2_KL/MXFP4_R8 quant to see how it does squeezed onto my box [in progress].

    • Test. Tune. Inevitably troubleshoot the dozen other things that go wrong. Figure out how much spare RAM that leaves me.

    • Make a higher quality IQ3_KT quantization. This will take all night on my CPU.

    • KLD test both of them vs the full precision, to quantify quantization loss. Likely an overnight task, too.

    • Try merging the new model release with the base model, 50/50, for a less “deep fried” model. imatrix, quant, test.

    The goal is to host it on a single RTX 3090, Ryzen 7000 with 128GB RAM, for anyone curious. Though I may try smaller models too, like Laguna S1.