I have a Framework Desktop that I want to put AI models on. I keep hearing that local models are getting more and more impressive. I would love to one day get off of my Claude dependence for the sake of privacy and running things in house.
I am currently trying out using pi agent with GLM-4.7-Flash Q8_0.
I don't have much of a reference to say how it compares to other local models, but it's definitely not something I could switch over to as a main driver instead of Claude.
What are you guys thoughts?
This is also the first time I've tried Pi. I've also been recommended hermes, which I know very little about.
▗▄▄▄ ▗▄▄▄▄ ▄▄▄▖ root@nixos
▜███▙ ▜███▙ ▟███▛ ------------
▜███▙ ▜███▙▟███▛ OS: NixOS 26.05 (Yarara) x86_64
▜███▙ ▜██████▛ Host: Desktop (AMD Ryzen AI Max)
▟█████████████████▙ ▜████▛ ▟▙ Kernel: Linux 6.18.44
▟███████████████████▙ ▜███▙ ▟██▙ Uptime: 11 days, 21 hours, 26 ms
▄▄▄▄▖ ▜███▙ ▟███▛ Packages: 500 (nix-system)
▟███▛ ▜██▛ ▟███▛ Shell: bash 5.3.9
▟███▛ ▜▛ ▟███▛ Terminal: /dev/pts/7
▟███████████▛ ▟██████████▙ CPU: AMD RYZEN AI MAX+ 395 (32)z
▜██████████▛ ▟███████████▛ GPU: AMD Radeon 8060S Graphics ]
▟███▛ ▟▙ ▟███▛ Memory: 39.81 GiB / 125.09 GiB )
▟███▛ ▟██▙ ▟███▛ Swap: 6.73 MiB / 7.45 GiB (0%)
▟███▛ ▜███▙ ▝▀▀▀▀ Disk (/): 119.94 GiB / 3.57 TiB4
▜██▛ ▜███▙ ▜██████████████████▛ Local IP (enp191s0): 192.168.0.4
▜▛ ▟████▙ ▜████████████████▛ Locale: en_US.UTF-8
▟██████▙ ▜███▙
▟███▛▜███▙ ▜███▙
▟███▛ ▜███▙ ▜███▙
▝▀▀▀ ▀▀▀▀▘ ▀▀▀▘
If your spec is really what your computer is, you should be able to run the models you need to.
Just slow down, have patience, and be opinionated on what you want it to look like, from code to architecture to user experience.
I hit more failure modes than when I use Claude, but there are always failure modes, and the answer is always slowing down, learning what it's doing, and having an opinion.
I will use, sometimes, free web search tools to architect, plan, and generally get ideas. Their ability to search the web directly helps. And to code, I sometimes build a cheap RAG with the language documentation (nushell needs help most of the time) and that helps.
Again, in general, just slow down and take your time. Like a jr dev, just cut it into bite-sized pieces with clear goals, all of which a more capable model can help with.
GLM worked well and did some impressive one-shots. Gemma can be more sophisticated in what it writes, but Qwen is less acrobatic and stable. Ornith looks interesting but I was never able to run it.
Hey! thanks for the reply :) yeah the slowing down part is a big factor.. i'm certainly being a little impatient, so well received 🙏😁
I had also noticed I had a big mismatch in my pi
models.json'scontextWindowandmaxTokens, which I just fixed (shown in another comment) so going to test it out some more.