Ask Lemmy

Ask the author if they’d prefer

A. A fluffy bunny

B. A letter from their sweetie

C. A properly-formatted data file.

TLDR: The result of current LLMs will be very bandlimited and one-directional.

I hope that means something to you, because otherwise I'm going to try to explain this very specific thing, and I'm afraid I might not be able to express it in very understandable terms (sorry):

Firstly, one-directionality: when a human wants to write a story, we usually think about the plot twist beforehand and then pave the way by hinting at the upcoming twist without giving too much away. It's just nice when a first time reader is surprised, but struggles on a second time how they missed all the obvious clues.

This process requires a lot of back-and-forth while writing. Humans do this naturally. LLMs and other transformer networks have a huge problem with this. I often hear LLMs referred to as text prediction machines. This is not entirely accurate, but a similar enough. And to keep with this analogy: text prediction doesn't really work backwards to suggest a better start to the sentence, does it? LLMs tend to take a path, from start to finish, even in great detail, but that's it. There's no setup. It's very flat writing.

Secondly, bandlimiting: Over time LLMs tend to mush different characterizations and continuity into a smooth paste, leaving little grit to it. I really struggle to not say the word derivative (like in math). But LLMs just write average characters who do average things in an average way. And then spell out how everything was totally unpredictable, important and meaningful, while using superficially eloquent language. Nothing just is everything serves as. It's a poor writing style that often misses the appropriate tone, trying to sound sophisticated.

You can’t. The best indicator of human authorship at this point are mistakes like misspellings and poor grammar. The common advice — like looking for em-dashes — was garbage to begin with and has only become worse as LLMs evolve.

It isn't possible in a common case. There are bots that check text for specific wordage rarely used by humans, but those bots are unreliable. Especially if the text is written by a non-native-speaker.

If you’re concerned about detecting AI for an assignment or a competition, I suggest something that tracks changes, like a shared Google doc. It shows you how the document was written over time, which requires a lot more effort to fake.

But in general, no, you cannot reliably detect whether text was produced by AI or a human. There are some signs, but it would be difficult to prove.

EDIT: voice typing errors.

I dont, I just assume you are all robots. Nice robots, who usually say smart or nice things, but unless I can drive to a place an punch you without breaking my hand... Best to just assume yall are to avoid the disappointment later.

As for proving to the rest of the robots that im not a robot... For Lemmy, I completely turn off spellcheck and amy grammar assistance. The frequency of errors, poorly constructed sentences and bad use of frequent AI tells is hopefully enough to validate that im beep boop not a robot.

Everyone in this thread saying it’s impossible is basically correct, but that doesn’t mean you don’t have to do your best to identify ai-generated text sometimes.

If you’re focused on identifying whether a given text is ai generated, here’s how Wikipedia does it: https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing

If you’re focused on proving something you’ve written was written without ai assistance, I think your best bet is a screencast of the writing process.

Use an AI enough and you won't be able to stop noticing the telltale signs

It starts to get weird: I recognize that

It starts to have several of these: funny introduction phrases, logical mistakes/knots/holes, unneccessary redundancy, unneccessary jumps, funny summary towards the ending: highly suspicious, and I probably stop wasting my time with it.

There are some generic observations you can use to identify whether a story was AI generated or written by a human. However, there are no definitive criteria for identifying AI generated text except for text directed at the LLM user such as "certainly, here is a story that fits your criteria," or "as a large language model, I cannot..."

There are some signs that can be used to identify AI generated text though they might not always be accurate. For instance, the observation that AI tends to be superficial. It often puts undue emphasis on emotions that most humans would not focus on. It tends to be somewhat more ambiguous and abstract compared to humans.

A large language model often uses poetic language instead of factual (e.g., saying that something insignificant has "profound beauty"). It tends to focus too much on the overarching themes in the background even when not required (e.g., "this highlights the significance of xyz in revolutionizing the field of ...").

There are some grammatical traits that can be used to identify AI but they are even more ambiguous than judging the quality of the content, especially because someone might not be a native English speaker or they might be a native speaker whose natural grammar sounds like AI.

The only good methods of judging whether text was AI generated are judging the quality of the content (which one should do regardless of whether they want to use content quality to identify AI generated text) and looking for text directed at the AI user.

A faraday caged room, irl in-person supervision, test takers required to be fully nude.

Side note: We need a chess championship game where Hans Niemann plays nude and livestreamed to the world. 😏

I am only a dabbler, but every time I see a new "LLM-detection" tool, I have been able to defeat them with simple changes to parameters and/or prompts.

It is a whack-a-mole game where detection will lag evasion. For situations where authorship really matters, I suspect low-tech countermeasures like proctored exams are going to be necessary.

They'd have to write it in front of you. All the qualities people are describing here are just good writing or something that you can change by simply telling the LLM 'dont write like an llm'

There is no reliable way to tell whether a story was written by a human or an AI based on the final text alone. Current detection tools are inconsistent and can be wrong in both directions.

Drafts can offer some context, but they are not proof, since AI-generated text can also be produced in stages or edited to look human. Posting a draft and editing it live may demonstrate effort, but it still cannot conclusively prove authorship.

Because of this, most platforms focus on disclosure and rules about acceptable use rather than trying to definitively verify whether a text was written by a human or an AI.