Programming

This seems like a silly workaround at first but it's really not. If the network is unreliable, you can't really use normal video streaming, you need to send full screenshots. Probably still a better idea to use only I-frames than a bunch of JPEGs but whatever.

But they did make some very silly mistakes. Par for the course of an AI coding company I guess.

1. WTF are you doing with 40mbps. Tone it down to like 8.
2. If the network is reliable but slow, just reduce bitrate and resolution. Don't use JPEGs unless packet loss is the problem.
3. WTF are you doing using a whole game streaming server for? It's meant for LAN, with minimal latency. Just capture the screen and encode it, send via WebSockets. Moonlight is completely unnecessary.
4. Only keep the latest frames on the server. Don't try to send them all immediately or it'll fall behind. Wait for the client to finish receiving before sending another one. This way it won't lag behind increasingly. This should have been extremely obvious.
5. H264 is so 2003, ask the client if it supports AV1 or HEVC then use that, more data for free.
6. Use WebTransport when available, it's basically made for live streaming
7. Why are you running a screenshot tool in terminal then grabbing the jpg... Unnecessary file overhead & dependency

I probably missed some but even for an AI company this is really bad

Uh, I'm pretty damn sure I have seen an office with hundreds of people, all connected remotely to workstations, on enterprise network, without any of the problems they are talking about. I've worked remotely from a coffee shop Wifi without any lag or issues. What the hell are they going on about? Have they never heard about VNC or RDP?

But our WebSocket streaming layer sits on top of the Moonlight protocol

Oh. I mean, I'm sitting on my own Wifi, one wall between me with a laptop (it is 10 years old, though) and my computer running Sunshite/Moonlight stream, and I run into issues pretty often even on 30FPS stream. It's made for super low-latency game streaming, that's expected. It's extremely wrong tool for the job.

We’re building Helix, an AI platform where autonomous coding agents...

Oh. So that's why.

Lol.

"we suck at coding so we did a funny but shitty workaround look at us we are amazing woohoo buy our vaporware"

UDP — Blocked. Deprioritized. Dropped. “Security risk.”

It's this actually a thing? I've never seen any corporate network that blocks UDP. HTTP/3 will even rely on it.

This sounded unhinged so I just had to check the article to confirm...
Yep, another AI startup.

Ok there's a whole lot of wtf going on here.

AI codebots in the cloud doing your code for you, cool, I guess.

So you need to watch them? And presumably intervene if necessary? Ok.

So then:

They decided that they'd stream a video of the AI codebots doing their thing.

At 40Mbps per stream.

For "enterprise use".

Where presumably they want lots of users.

And then they didn't know about locked down enterprise internet and had to engineer a fallback to jpeg for when things aren't great for them. Newsflash - with streaming video peaking at 40Mbs per user, things will never be great for your product in the real world.

How, in any way, does this scale to anything approaching success? Their back end now has to have the compute power to encode and serve up gigabits of streaming video for anything more than ~50 concurrent users, let alone the compute usage of the actual "useful" bit , the AI codebots.

For say, 5 users out of a site of 200, IT departments will now see hundreds of megabits of streaming traffic - and if they're proactive, they will choke those endpoints to a crawl so that their pathetic uplink has a chance to serve the other 195 users.

All of this for a system that is fundamentally working on maybe 5kB of visible unicode text at any particular moment.

hardware-accelerated, WebCodecs-powered, 60fps H.264 streaming pipeline over WebSockets and replaced it with grim | curl when the WiFi got a bit sketchy.

I think they took https://justuse.org/curl/ a bit too serious.

We’re building Helix, an AI platform where autonomous coding agents work in cloud sandboxes. Users need to watch their AI assistants work. Think “screen share, but the thing being shared is a robot writing code.”

Oh, they are all about useless tech. Why would you need to watch an agent code at 1080P 60fps using 40Mbps?

Laundering VC money for this. 😂

You are making it so people need to live watch AIs coding? This is insanity. This bubble is gonna hurt.

But our WebSocket streaming layer sits on top of the Moonlight protocol, which is reverse-engineered from NVIDIA GameStream.

Mf? The GameStream protocol is designed to be ultra low latency because it's made for Game Streaming, you do not need ultra low latency streaming to watch your agents typing, WTF?
They're gonna be "working" on a desktop, why the hell didn't you look into VNC instead? RDP??
You know, protocols with built-in compression and other techniques to reduce their bandwidth usage? Hello?? The fuck are you doing???

Why aren't they streaming the text changes and rendering them them client side?

I don't understand why they bother with the "modern" method if the fallback works so well and is much simpler and cheaper.

Would have been more sensible to stream h.264 at a lower fps. Or maybe stream the text directly

Yeah... I mean they should have just copied whatever video conferencing platforms do because they all work fine behind corporate proxies and they also don't suffer from this "increasing delay" problem.

I haven't actually looked into what they do but presumably it's something like webrtc with a fallback to HLS with closed loop feedback about the delay.

Though in fairness it doesn't sound like "watching an AI agent" is the most critical thing and mjpeg is surprisingly decent.