ejs

i would say this is like if open code and open web ui had a baby it would be this. It’s a web interface for self hosting models but runs them through open code to make it agentic. Helpful for non developers to get into running models, but imo isn’t significant bc using open code tui and connecting it to a llama.cpp or vllm self hosted api is not difficult for devs

I don’t have any familiarity with using this kind of software, but I looked through the git repo of SavaPage. It looks like it has been actively developed for the past few years, which is a great sign, but it looks like almost all commits are done by one user. The issue tracker is also a little meager, with just one open issue, potentially pointing to a very small user base. Adoption heavily depends on as long as that one person keeps maintaining the project.

Honestly, you’re a few months late to the whole buying GPUs for local llms party, so expect exorbitant prices even for older cards

The name of the game is vram. For the most part, more is better. If you can get your hands on multiple matching (same model) 24gb or higher cards (within price range), you’re golden.

Going for more than 2 gpus can become challenging with motherboard pcie slot heights, so make sure either your cards aren’t too tall or you have widely spaced out pcie slots.

For inference, speed (tokens/second) is limited by memory bandwidth. Go for faster bandwidth memory cards if you can afford it (e.g. GDDR6 will be faster than GDDR5).

Also with multi gpus you will need an adequate power supply, and a large enough case.

If you want to be a bit eccentric and load huge models, you can also go the CPU route and fill up a motherboard with 256 GB ram, because then you’re in the several hundred B param model territory, which could, depending on your use case, be better than having faster inference on smaller/quantized models. Even then, DDR5 with high MHz is still way slower than gpus.

lol they already support running local models. wtf is the distro gonna do…? pre-install llama.cpp? this is so silly to me that people are resigning over this, too.

Has anyone compiled a list of where projects are moving to? I know many linux desktop applications are self hosting on gitlab, but i’ve also seen gitea and codeberg. If anyone has opinions about a preference, do comment. I have been enjoying self hosting gitea for my simple personal projects and for deploying simple web apps, all on $5 vps.

I suggest looking at llm arena leaderboards filtered by open weight models. It offers benchmarks at a very complete and statistically detailed level for models, and usually is quite up to date when new models come out. The new Gemma that just came out might be the best for 1x GPU, and if you have a bunch of vram check out the larger Chinese models