AI - Artificial intelligence

I know it's not exactly what you're asking for, but I've been using qwen3.8 flash-next, glm 5.3 flash, deepseek v4 flash, v4 pro and gemma-4 26b-a4b through Cortecs (openrouter alternative) and they've surprised me by how good they are for most tasks. I still have access to big models like Sonnet and so on but I rarely reach out for them.

If you're not completely ready to abandon the big models, perhaps try something like Cortecs or Openrouter as a middle ground? You can still self-host the smaller ones and reach out and use something like Sonnet through them when needed.

Some providers have strict data retention policies, are powered by green energy and so on... I use open-webui front-end to interact with Cortecs.

GLM 5.3 flash if i remember correctly had even some benchmarks that even beat Opus. I know there might be some benchmaxxing going on but still. And it's really cheap.

Is GLM too slow or not capable enough? It's quite a big model, if the runtime is setup correctly you should have good quality but getting a throughput similar to cloud models is going to be difficult. Did you give Qwen3.8-27B-FP8 a try?

Also you mentioned Pi as your harness, but what are you using as the inference backend?

If your spec is really what your computer is, you should be able to run the models you need to.

Just slow down, have patience, and be opinionated on what you want it to look like, from code to architecture to user experience.

I hit more failure modes than when I use Claude, but there are always failure modes, and the answer is always slowing down, learning what it's doing, and having an opinion.

I will use, sometimes, free web search tools to architect, plan, and generally get ideas. Their ability to search the web directly helps. And to code, I sometimes build a cheap RAG with the language documentation (nushell needs help most of the time) and that helps.

Again, in general, just slow down and take your time. Like a jr dev, just cut it into bite-sized pieces with clear goals, all of which a more capable model can help with.

GLM worked well and did some impressive one-shots. Gemma can be more sophisticated in what it writes, but Qwen is less acrobatic and stable. Ornith looks interesting but I was never able to run it.

I've got the same goal as you. My findings follow, I hope they're helpful:

Frontier is good for planning which takes relatively few tokens compared to writing the code. You can run a frontier model like Claude and tell it to use your local AI model as a coding resource - hand it tasks and check the output. Qwen-3.8:27B is my local. It's good, and fast.

If you want to break your dependency on American frontier models specifically, you can use GLM-5.3 through openrouter for planning. You might find yourself using it for everything, because it's pretty cheap.

I'm doing the first one, myself. Often I'll use Hermes running GLM-5.3-flash (on ollama cloud) directly instead of Claude Code for smaller tasks.

Apparently you just need to give them a good harness to use tools like write code and then debug it, that makes it way more efficient.

And please run them inside a VM or at least a container sandbox, not just unsandboxed on your machine.

I've had great success, but I have a pretty beefy gpu: radeon pro w7900 (48GB vram).

Definitely not as fast or good as frontier models, but quite good! I'm currently using qwen3.8-27b, but have used qwe3.6-35b-a3b and other models (including using openrouter and opencode go, but you asked about self-hosting).

Also, FYI, I use opencode.

Try posting this on !localllama@sh.itjust.works.