Can AI detectors be wrong?

Two kinds of mistake

• A false alarm: text a person wrote is flagged as AI. This is the costly mistake, because someone can be accused of something they did not do.
• A miss: AI text passes as human. This is the common mistake, and it gets more common once the text has been edited.

A detector can trade one for the other. Set it to raise fewer false alarms and it misses more AI text. No setting removes both.

How often it happens

Our own checker is a simple one, and we publish its error rates. It calls about 1 in 30 human texts "Probably AI". It gives that verdict to 59% of AI texts, says "Can't tell" for 36%, and wrongly calls 4.5% "Probably not AI". The details are on the accuracy page.

Commercial detectors make mistakes too. In 2023 Turnitin's chief product officer estimated that its AI detector had wrongly flagged about 1% of documents and 4% of sentences, as reported by the University of Kansas Center for Teaching Excellence. One percent sounds small. Across a university that checks tens of thousands of essays, it is hundreds of students.

Who gets flagged unfairly

False alarms do not fall evenly. A 2023 Stanford study, GPT detectors are biased against non-native English writers, found that detectors consistently labelled writing by non-native English speakers as AI-generated while judging native writing correctly. Plain, formulaic or very formal prose is at more risk, whoever wrote it.

Why detectors cannot be sure

A detector looks at the text and nothing else. It does not know who typed it, when, or how. Models are trained to write the way people write, so the two overlap. Rewording, trimming or asking the model for a plainer style moves a text further into the overlap.

If your own writing was flagged

1. Ask which detector was used and what its published error rate is.
2. Show how the text came to be: version history, earlier drafts, notes and sources.
3. Offer to talk through the work. Explaining your own argument is something a detector score cannot contradict.

If you are checking someone else's writing

Treat a flag as a reason to look closer and to ask, never as proof. Compare the text with the person's other work, and remember that a clean result does not prove a person wrote it either.