How to tell if AI-written help articles are good enough to publish

Reviewing AI-generated documentation well comes down to people, not AI. Here's what we learned while building and testing the Pageloop agent, and how you can apply it to your own help center.

If your team now drafts help articles with Claude, ChatGPT or a docs tool, you might be living a version of this. Drafts arrive fast, people approve them, and the help center grows. But nobody's quite sure whether "looks fine" means the article is actually good.

Here's the short answer. To review AI-generated documentation properly, have someone with real documentation expertise judge each draft against what the best version would look like, and not against what your team usually accepts. Check the facts, look for what's missing, read for the tells of AI writing, and decide upfront which docs are allowed to be average.

Why "approved" doesn't mean "good"

When we rebuilt the Pageloop agent, the part that drafts new articles and suggests updates, we used evals to work out how close we were to shipping it to customers. The first eval's numbers said we were nowhere near ready, but as we dug in, we found it was measuring the wrong thing.

Phew.

So we dug into the raw data, and two facts jumped out:

1. Our evals are shaped by customer behaviour and setup. The suggestions customers accept or ignore, their integrations and so on all influence how the agent evolves.

2. 90% of our customers are not knowledge managers or technical writers. Docs aren't their day-to-day, because they're product managers, PMMs, support agents and customer education folks.

User activity on Pageloop tells us what they consider 'acceptable'. It's not always what is 'best'. Our evals were judging outputs against acceptable.

And we want the best.

The same thing happens inside your team. If a PM or support agent approves an AI draft, that tells you they found it acceptable on a busy afternoon. An approval rate measures how often drafts clear that bar, and it says very little about whether the article is as good as it could be.

Who should review AI-generated documentation

We now have experienced knowledge managers, people who spent years honing their skills, building our evals. AI can't replace domain expertise.

When those KMs reviewed the outputs, the new agent beat the old one in every possible way. The same new agent, judged by people with documentation expertise, got the opposite verdict.

For your team, that means the final call on quality belongs to whoever knows documentation best. That's usually a knowledge manager or a technical writer, and for some teams it's the support lead who has lived in the help center for years. Anyone can check whether a step matches the product. Judging whether an article is well structured, findable and pitched at the right reader is a skill, so route drafts to the person who has it.

"Nothing factually wrong" can still be average

I ran an experiment a while ago that changed how I review AI output. I wanted a refresher on our competitors: their target customers, positioning, pricing and marketing. So I did the research the old-school way first. I visited every competitor website, read and re-read their messaging and pricing, traced how their positioning evolved on the Wayback Machine, went through hundreds of G2 reviews, and spent ~10 hours down rabbit holes. I only opened Claude once I was completely done with my own notes.

Claude gave me fantastic summaries. Nothing factually wrong. It spotted 3 of the 8 insights I had noted.

The five it missed were the subtle ones. Here's what I noted next to what Claude gave me:

• Words. I noticed competitors chose different words, like self-maintaining docs versus self-updating docs. Claude gave me a summary of their positioning.

• Pricing. I read pricing as a peek into how each one's customers buy (seat-based, credit-based and so on). Claude gave me a table of pricing differences.

• Reviews. I spotted a pattern in G2 reviews where PMs loved one product and tech writers complained about it. Claude gave me an overview of what customers mainly like and dislike.

So nothing was wrong, and it was still an average understanding, the same summary available to every one of our competitors. Because average understanding leads to average outcomes.

A help article works the same way. An AI draft can describe every step correctly and still miss the workaround your customers always ask about, or the reason a setting exists. When you review, read the draft against what you know your customers struggle with, and ask what an expert would have added.

How to spot AI writing in a help article

Someone on our team once sent me a "spot the AI writing" quiz and predicted I'd ace it. I got 10/10, and apparently only 28% of people get a perfect score (who knows!). I'm not gifted at this. I just spend hours every week reviewing Pageloop's AI outputs.

The tells were in the words, like one answer that said "within a fortnight". Fortnight? Who's saying that in a casual conversation?!

Your customers notice these too, even if they couldn't name them. A word nobody on your team would say, an oddly formal phrase in the middle of a casual article, or a sentence that sounds polished and says nothing all make a help article feel less trustworthy. The more AI writing we read, the faster we spot it. It's simply a muscle, so build it by reading a lot of drafts and flagging the lines that don't sound like your team.

(It's also why I write my own LinkedIn posts. Look, I'm an average writer. But given the slop out there, I bet you my writing will scream authentic.)

Decide which docs are allowed to be average

Of course I'll still use AI for research. I'll just assess whether it's a thing that's allowed to be average.

The same question works for your help center. An internal note, a minor change log entry or a first draft for a PM to react to can be average, and Claude will get you there fast. The articles customers and your AI support agent lean on most can't be. Those deserve an expert review every time, because an average version of them becomes the answer your customers get.

A review routine you can start this week

Here's how I'd put it together for a team that drafts with AI today:

1. Sort the draft first. Decide whether this article is allowed to be average or needs to be the best version.

2. Check every fact against the source. Compare steps, settings and names with the PRD, release notes or the product itself.

3. Ask what's missing. Look for the workaround, the edge case and the reason behind the feature, the things a summary tends to leave out.

4. Read it out loud for the tells. Flag any word or phrase nobody on your team would actually say.

5. Send the important ones to your docs expert. The final call on articles that need to be the best goes to the person with documentation expertise, not to whoever is free.

If one person ends up holding all of that expertise, that's a different problem, and we wrote about it in should you hire a technical writer?

Where Pageloop fits

This is why we had experienced knowledge managers test our rebuilt agent, and didn't judge it on whatever customers happened to accept. Pageloop drafts new articles and suggests updates, and a person reviews every change before it goes live. Otherwise, what's even the difference between AI slop and tools like Pageloop? (I'm just a little bit biased, ok.)

So, who's reviewing your AI drafts today, and are they judging what's acceptable or what's best?


Untitled (New Zealand seascape), 1863, New Zealand, by Charles Decimus Barraud.