Programming

26102 readers

928 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 2 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

UlrikHD@programming.dev

bugsmith@programming.dev

Spyro@programming.dev

509

AI still doesn't work very well, businesses are faking it, and a reckoning is coming (www.theregister.com)

submitted 17 hours ago by brianpeiris@lemmy.ca to c/programming@programming.dev

127 comments fedilink hide all child comments

Excerpt:

"Even within the coding, it's not working well," said Smiley. "I'll give you an example. Code can look right and pass the unit tests and still be wrong. The way you measure that is typically in benchmark tests. So a lot of these companies haven't engaged in a proper feedback loop to see what the impact of AI coding is on the outcomes they care about. Lines of code, number of [pull requests], these are liabilities. These are not measures of engineering excellence."

Measures of engineering excellence, said Smiley, include metrics like deployment frequency, lead time to production, change failure rate, mean time to restore, and incident severity. And we need a new set of metrics, he insists, to measure how AI affects engineering performance.

"We don't know what those are yet," he said.

One metric that might be helpful, he said, is measuring tokens burned to get to an approved pull request – a formally accepted change in software. That's the kind of thing that needs to be assessed to determine whether AI helps an organization's engineering practice.

To underscore the consequences of not having that kind of data, Smiley pointed to a recent attempt to rewrite SQLite in Rust using AI.

"It passed all the unit tests, the shape of the code looks right," he said. It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."

All the optimism about using AI for coding, Smiley argues, comes from measuring the wrong things.

"Coding works if you measure lines of code and pull requests," he said. "Coding does not work if you measure quality and team performance. There's no evidence to suggest that that's moving in a positive direction."

you are viewing a single comment's thread
view the rest of the comments

[–] DerHans@lemmy.world 2 points 1 hour ago* (last edited 19 minutes ago) (2 children)

Yes it does not work right!

I agree.

also there are no new discoveries made by AI, we only see chat bots, self driving cars, automation in workplace, yet no discoveries. At some point I thought AI will help us solve cancer or way to travel in space, yet billionaires think of money.

We aren't there yet. ~~AI and research around it started, or rather really took off, around 2018 (at least relating to what we mean by AI today; ruled based approaches existed much longer). It is very much a new field, considering most other fields existed for over 30 years at this point.~~ Transformers, the current architecture of most models and what we consider when we speak of "AI", started with a paper in 2017. It is very much new ground, considering the fundamentals behind it are much older. And well, to be pedantic, large language models aren't really AI because there is no intelligence. They are just generating output that is the most probable continuation of the input and context provided. So yeah, "AI" cannot really research or make new discoveries yet. There may very well be a time, where AI helps us solve cancer. It definitely isn't today nor tomorrow.

I also don't think that billionaries make money with AI. I mean, if you look at OpenAI: they are actually burning money, at a fast rate measured in billions. They are believed to turn a profit in 2030. Without others investing in it, they would be long gone already. The people with money believe that OpenAI and other companies related to AI will someday make the world changing discovery. That could very well lead to AI making discoveries on its own AND to lots of money. Until then, they are obviously willing to burn a tremendous amount of money and that is keeping OpenAI in particular alive at this moment. Only time will tell what happens next. I keep my popcorn ready, once the bubble bursts :D

Edit: Connected AI making discoveries to lots of money gained or rather saved. That is the sole reason for investments from people with big money.
Edit 2: Clarified what I meant exactly by AI. Thanks everyone for pointing it out.

[–] hamid@crazypeople.online 2 points 1 hour ago

I took a class in what is ultimately the current approach in AI and Machine learning in 2002 using textbooks that had their first editions in the 90s. The field is in reality 30 years old.

[–] Saledovil@sh.itjust.works 1 points 57 minutes ago (1 children)

Neural networks existed since the 1970s.

[–] DerHans@lemmy.world 1 points 8 minutes ago

Yes, I meant the current state-of-the-art architecture by the term "AI" and partly the boom thereafter. The field "AI" is obviously much older. Sorry for that and thanks for pointing it out.