AI has the world a bit shaken at the moment. Technology has made a huge leap and AI can write in a way that sounds coherent — albeit a bit bland and fully untrustworthy. So, how can we all determine whether something is AI-generated or not? AI writing detectors are attempting to fill the gap. Here’s a look at what they are, how they work, if they work, and where they may be headed.
What are AI writing detectors?
Many companies have quickly worked to launch tools that can detect the likelihood that a piece of writing was generated by AI. Notice I say “likelihood.” While the tools can make an educated assessment, they aren’t foolproof.
How do AI writing detectors work?
Generally speaking, AI detection tools use algorithms to identify patterns in AI-generated content like a lack of natural flow, predictable word choices, a generic tone, and the repetition of words.
GPTZero, for example, assesses the perplexity (randomness of the text) and burstiness (the variation in perplexity) of text.
To gain a better understanding, it can be helpful to look back a bit. The Giant Language Model Test Room (GLTR) was created in 2019 to identify GPT-2 text. It breaks down the following process of how the predictability of text is analyzed:
“Each text is analyzed by how likely it is that each word would be the predicted word given the context to the left. If the actual used word would be in the Top 10 predicted words the background is colored green, for Top 100 in yellow, Top 1000 red, otherwise violet.”
In this model, higher levels of predictability point to a higher likelihood that the text is machine-generated.
That said, as AI quickly advances, detection algorithms are growing ever more sophisticated.
How accurate are current AI writing detector tools?
The big question is — do AI detectors work?
The accuracy of AI writing detection tools varies.
Open AI has released its Classifier AI detection tool to get feedback from the public. Currently, the company says the tool should not be used as a primary decision-making tool. It correctly identifies just 26% of AI-written content as “likely AI-written” while it incorrectly labels human-written content as AI-written 9% of the time.
On the other hand, some of the other providers report much higher levels of accuracy.
- Originality.AI claims it detects ChatGPT-3 content with a 94%+ accuracy rate.
- GPTZero claims to classify 99% of human-written articles and 85% of AI-generated articles correctly.
- Turnitin claims 98% confidence.
- Content at Scale claims its free AI Detector works at a deeper level than Open AI’s Classifier, but doesn’t provide any stats or studies to back it up. In my testing, human writing came back “Highly likely to be human” while text from ChatGPT came back “Likely to be AI-generated.” However, that’s a small case study so take it with a grain of salt.
In summary, the accuracy varies greatly, isn’t always backed up by solid data, and is often based on the company’s own studies.
Is ChatGPT’s AI-generated content watermarked?
ChatGPT’s AI-generated content is not currently watermarked but it may be soon, according to Scott Aaronson, a computer scientist currently working at OpenAI. In a recent lecture, he shared that he’s currently working to create a tool that statistically watermarks the output of text models like ChatGPT.
“Basically, whenever GPT generates some long text, we want there to be an otherwise unnoticeable secret signal in its choices of words, which you can use to prove later that, yes, this came from GPT. We want it to be much harder to take a GPT output and pass it off as if it came from a human,” says Aaronson.
He adds, “This could be helpful for preventing academic plagiarism, obviously, but also, for example, mass generation of propaganda—you know, spamming every blog with seemingly on-topic comments supporting Russia’s invasion of Ukraine, without even a building full of trolls in Moscow. Or impersonating someone’s writing style in order to incriminate them. These are all things one might want to make harder, right?”
Definitely would be helpful!
8 AI writing detector tools
Wondering where to find AI detection tools? Here are a few leading the charge:
- Open AI’s AI Text Classifier: Open AI says this free tool is not fully reliable and was made public to get feedback.
- Content at Scale’s AI Detector: Content at Scale is in the business of selling long-form AI content that it says is undetectable. It also offers a free AI detector tool that checks up to 25,000 characters at a time.
- Writer AI’s Content Detector: This free detector checks up to 1,500 characters at a time.
- GPTZero: GPTZero checks up to 20,000 characters at a time. However, only the first 5,000 characters are shown for free. It provides an assessment and highlights sentences that are more likely to have been written by AI. Pricing isn’t disclosed on the website.
- TurnItIn AI Writing Indicator: A tool available to non-students using Turnitin’s products that assesses whether sentences were generated by AI or not.
- Winston AI Detection Tool: An AI detector lets you know the odds that copy is AI- or human-generated. You can scan up to 2,000 words free, then plans start at $14 per month.
- Originality.AI Free AI Detection Tool: An AI content detector Chrome extension that lets you check for AI on any page you’re reading or writing. You get charged by the number of scans you perform, $0.01 per 100-word scan.
- Crossplag AI Content Detector: This tool scans up to 1,000 words free, then plans start at $9.99 to scan up to 5,000 words.
In all honesty, it feels like the wild west in the AI world right now. While these detection tools may be able to help to some degree, they are not yet foolproof. That said, better help may be on the way in the form of watermarks — time will tell.
Know of any other AI detection tools? Have you found any to be really useful? Let us know in the comments!
FAQs about AI writing detection
What do teachers use to detect AI writing?
Turnitin has served the global education community since 1998 and is now leading the AI detection charge in the education space. The company claims 98% confidence in its current ability to detect AI-generated content and is focused on helping educators detect AI in their student’s writing.
Is there a way to detect AI writing?
A variety of tools now assess the likelihood that a selection of writing is AI-generated. Many self-report accuracy levels above 90%. The tools spot AI writing by analyzing factors about the text such as the predictability of the word choices.