Popularity is not correctness. You're using a sloppily defined term. I'm using the fundamental definition. Your (Verners) concept muddles matters pointlessly.

The fact is, self-refining LLMs can very possibly exhibit the intelligence explosion fundamental to Von Neumann or I.J. Good's definition. They are already beginning to alter the way human society operates (coding, school, replacing jobs). They easily pass the Turing test with the right prompts. Your whole point is that it's not "real" intelligence because they don't really "understand", but I can say the same for you. For all I know, you're an LLM and there's literally no way that you can prove you aren't.

Lines in the sand about "real" intelligence are purely philosophical, and that kind of hyperopic philosophizing is exactly the sort of behavior that dooms humanity via underestimation. I'd rather we didn't find ourselves under machine overlords because "technically they aren't even really intelligent".

@agamemnonymous

Look, I've had to watch it happen to "triggered", "mansplain", and "woke." You're going to have to accept that it happened to Singularity.

You don't honestly think that the improvement of an LLM's predictive algorithm is going to lead to it taking over the world? All it can do is produce words. Unless we stupidly do everything it says, thinking it's truly intelligent, it has no power.

We only have to worry about machine overlords if we PUT machines in charge of stuff, and we'll only do that if we think they are intelligent enough to make decisions. So yeah, determining whether it has real intelligent is a key thing here.