Ollama + about 8 different models at the moment, hosted on a mac mini with open webui as a front end.
Predominantly for transcription, translation, an extra round of security checks on code, a more context friendly home assistant interface, and a daily run of context evaluation on property I’m looking for with a lot of specific needs (acreage, min elevation change, soil type, area, etc).
On the list but haven’t gotten to it yet, but I know I should. I could probably get a bit more out of that box with it, expand the context windows a bit…
Nice, I’ve got a Mac Studio M1 Max with 32GB of RAM that I use with Ollama and then I host OpenWebUI and OpenCode on my Arch Server. I use the Mac as a primary workstation, so it’s a little rough when I start running a model. I’m sure I could probably do and learn more about Ollama to improve my experience, but for now it works for certain tasks.
Yep.
Ollama + about 8 different models at the moment, hosted on a mac mini with open webui as a front end.
Predominantly for transcription, translation, an extra round of security checks on code, a more context friendly home assistant interface, and a daily run of context evaluation on property I’m looking for with a lot of specific needs (acreage, min elevation change, soil type, area, etc).
I have to recommend switching to llamacpp. It’s SO much faster than ollama.
On the list but haven’t gotten to it yet, but I know I should. I could probably get a bit more out of that box with it, expand the context windows a bit…
How? What is your average response time?
Apple silicon is pretty good at it as long as you’ve got the ram for it. I wouldn’t do less than 16GB.
A few seconds for most of the tasks
What spec Mini do you use?
Just an m2 w/ 16gb I repurposed.
Can’t really do a lot at once, and the context is limited, but it does the trick. I’d buy a few more if I saw them at the right price.
Nice, I’ve got a Mac Studio M1 Max with 32GB of RAM that I use with Ollama and then I host OpenWebUI and OpenCode on my Arch Server. I use the Mac as a primary workstation, so it’s a little rough when I start running a model. I’m sure I could probably do and learn more about Ollama to improve my experience, but for now it works for certain tasks.
I got mine a few years back for some iOS builds, don’t need to do them that often so it became the model host for me