Programming

Did these developers not have experience with AI?

This is from the article

But Rush and Becker have shied away from making sweeping claims about what the results of their study mean for the future of AI. For one, the study’s sample was small and non-generalizable, including only a specialized group of people to whom these AI tools were brand new.

I'm not sure focusing on one aspect to scope a reasonable and doable study automatically makes it “really low effort”.

You are right, but I believe they should at least have chosen another use case, to make it interesting. I wouldn't have needed a study to know that an AI performs worse than a developer in a project the developer most likely built them self. The existing project might have some really weird code smells and work arounds that only the developer on the project knows about and understand. There might be relevant context external to the solution. The AI have to be a mind reader in these cases.

But, if you gave the AI and the developer a blank canvas a clear defined task, I just believe it would be a more interesting study. *

It kind of sounds like they were just handed a tool they knew nothing about and were asked to perform better with it. A mitter saw is way better and faster than a regular saw, if you know how to use it.

*edit

To make my point more clear, I don't mean the developer needed to solve an issue that's not related to his daily work, but a task that's not dependent on years of tech debt or context that is not provided to the AI. And yes, by that, I don't believe code generation from an AI have a big use case in scenarios where the project have too many dependencies and touches on niche solutions, but you can still use it for other purposes than building features.