That’s because there’s absolutely reams of writing out there about Sonnet 18—it could draw from thousands of student essays and cheap study guides, which allowed it to remain at least vaguely coherent. But when forced away from a topic for which it has ample data to plagiarize, the illusion disintegrates.

Indeed, Any intelligence present is that of the pilfered commons, and that of the reader.

I had the same thought about the few times LLMs appear to be successful in translation, (where proper translation requires understanding), it's not exactly doing nothing, but a lot of the work is done by the reader striving to make sense of what he reads, and because humans are clever they can somtimes glimpse the meaning, through the filter of AI mapping a set of words unto another, given enough context. (Until they really can't, or the subtelties of language completely reverse the meaning when not handled with the proper care).