Chatbot Setup Checklist: What to Do Before You Go Live

Chances are, if you have a product and a website, a chatbot has crossed your mind at least once. They can take a significant chunk of repetitive questions off your support team, they're available 24/7 when your team isn't, and most of your customers would genuinely rather get an instant answer from a bot than wait in a queue for a human. (62% of them, according to Salesforce. We're not alone, you and I!)

But deploying one is not as straightforward as the vendor demos make it look. You can absolutely try, but don't be surprised if things go awry. What's tricky with chatbots is that when a chatbot gets something wrong, it doesn't flag it for you to see. The bot just answers confidently with whatever it found in your knowledge base, even if that information might be out of date today. Your customer reads it, believes it, acts on it, and you never hear about it because they didn't file a ticket. They just leave after they get what they want. That could lead to a problem tomorrow.

Now that I've sufficiently scared you, let me bring the good news. Once you sort out the foundation, the bot largely runs itself. The work is upfront, not ongoing. And the answer isn't to set up everything, but to set up the minimum that keeps you from getting blindsided, so any hiccups that do come through are fixable rather than catastrophic.

Here's what we've recommended you start with.

What to set up on day one

Three things, all of which can be wired up in an afternoon (especially if you're using Pageloop, since we can do all the heavy lifting for you)

1. Audit your content before the bot ever touches it

When you set up a chatbot, it is going to read your knowledge base and answer from whatever it finds. This is why auditing information is crucial. If the chatbot finds an article that still describes your old pricing tier, it will still quote that pricing to a customer. Similarly, if two articles contradict each other on your refund policy, the bot will pick one. Unfortunately, you won't be able to know which it chose.

Before you connect anything, go through what you have. You're looking for three things.

Outdated information. Articles that were accurate when they were written but haven't been touched since the product changed. We find that every team has these. The average help center has between 15% and 40% of its articles out of date at any given time, depending on how often the product ships.

Contradictions. This looks like two articles that say different things about the same topic. Happens more than you'd think, especially when different people wrote them at different times. The bot has no way to know which one is current.

(Optional but important nonetheless) Marketing copy in the knowledge base. If your help center articles include language pulled from landing pages or feature announcements, your chatbot will answer support questions in that same promotional tone. Customers asking how to fix something do not want to hear about how powerful the feature is. Strip whatever was written to sell rather than explain.

If you have 200 articles, you do not need to audit all 200 before launch. Start with the 20 that get the most traffic (you can check analytics for this) or cover the topics that generate the most tickets. Fix those, launch, and work through the rest over the following weeks.

2. Make your content readable by the bot, not just by humans

Your chatbot doesn't read an article top to bottom the way a person would. Most chatbots today use something called retrieval-augmented generation (RAG). It starts by breaking down articles into smaller sections (chunks) and then retrieves the chunk that best matches the question, passing it to an LLM to compose a response. That process works well when your content has clear headings, short sections, and each section covers one topic.

A few things that make a measurable difference to retrieval accuracy.

Use a clear heading hierarchy. H1 for the article title, H2 for each major section, H3 for sub-sections. This gives the retrieval system natural break points. One source found that structured content with clean headings improved retrieval accuracy by up to 35% compared to unstructured text.

Keep sections self-contained. The bot might retrieve one section from the middle of an article without any of the surrounding context, so each section needs to make sense on its own. This means being specific rather than relying on what came before it. An example of good writing would look like this

"Look at the feature box on the right" = Bad "Look for the "Legacy" button on the upper right hand corner of the top margin." = Good

Put the answer near the top. The bot uses whatever it retrieves to compose a response, so if your article opens with three paragraphs of context before getting to the point, that's what the bot works with too.

Add text alongside screenshots. Bots cannot read images too well, so don't rely on images alone to explain processes. If a critical step in your article is "see the screenshot below" with no text description of what the screenshot shows, the bot will likely skip it. Every visual needs a text equivalent, which is why alt text is pretty important.

3. Test with real questions before you go live

Take your 20 most common support questions, which your team answers every week. Ask your support team to grab those queries for you and start working on those. Type them into the chatbot in a sandbox or preview environment, exactly the way a customer would phrase them, and then check the answers against reality.

You're looking for three things.

1. Is the answer correct?

2. Is the answer complete?

3. Did the bot pull from the right article?

Also try asking the same question three different ways. Customers don't all phrase things identically. If "how do I cancel" gets a perfect answer but "I want to stop my subscription" gets nonsense, you have a terminology problem in your content. If it answered a billing question using your onboarding guide, the content might be fine but the retrieval is confused, and that usually means the articles aren't scoped clearly enough.

If you get these out of the way, you can deploy your chatbot. All the other issues are things you can fix a little later.

What to add once the bot is live

Track what the bot is actually saying. Most chatbot platforms have a conversation log that you can go through. Set aside some time each week to go through a random sample of those conversations. Over here you'll be able to find wrong answers and you'll start to see which topics your knowledge base handles well and which ones it fumbles. That's where you go first to fix.

Connect failed queries back to the source. When the bot gives a wrong answer, trace it. Locate which articles it's pulling from and check what information was incorrect.

Tie updates to your product release cycle (more long term). This is the part that separates teams whose chatbot stays accurate from teams whose chatbot degrades. Every time the product changes, some portion of the knowledge base becomes wrong. If your KB updates happen on a separate schedule from your product releases, decay starts immediately and accelerates with every sprint. The two need to be linked.

Set a system prompt that tells the bot how to behave. You can configure your chatbot to stick to a specific support tone and to not answer certain questions when unsure. This ensures that trickier questions get handed off to humans to clarify.

What you probably don't need

A few things that come up in every chatbot setup guide that you can skip for now.

1. Your chatbot platform already handles chunking, how it splits articles into smaller pieces for retrieval. If your content has decent headings, it works. You don't need to calculate overlap percentages or count words per section.

2. If one person is checking conversations weekly and updating articles when the product ships, you have enough process. A dedicated AI ops function can wait.

3. And if the bot is giving wrong answers, don't upgrade the model. It's almost always a content problem. A newer LLM will read the same stale article and give the same wrong answer, just more fluently.

Why Pageloop is useful here

A quick pitch, because we're the best tool for the job? (humble brag)

Pageloop keeps your knowledge base accurate so your chatbot doesn't have to work with stale content. It covers most of the prep work above and the ongoing maintenance after launch.

We monitor your signals from where product changes can be detected (including your support tickets, Slack conversations, and engineering tools - Linear, Jira, GitHub) and then flags which articles are affected and suggests updates. Isn't that neat? All you do is review the suggestion, edit it if you want to, and publish when you're ready.

It also audits your existing KB for the things we covered in step one (broken links, contradictions, outdated articles, and staleness scoring). Perfect if you're planning to set up a chatbot yourself.

You can run a free audit of your entire kb on Pageloop and see what your bot is going to be reading. Want to see what it looks like? Find out more here.

Some FAQ's

Do I need to do all of this before launching a chatbot?

Not all of these steps. We recommend the content audit and the 20-question test as non-negotiable steps. Heading structure and summaries make a difference but you can fix those in the first couple of weeks after launch. The metadata and screenshot descriptions can wait longer. Don't bog yourself down by aiming for a perfect setup.

Can I skip the content audit if my KB is relatively new?

New doesn't always mean accurate. A six-month-old KB can still have articles that describe features that have since changed, and conflicting information is lurking somewhere that the bot pulls from. The audit is faster on a smaller KB, which is a good reason to do it now rather than when you have 300 articles. Pageloop's audit tool can run through this in a few minutes if you'd rather not do it manually.

How often should I review chatbot conversations after launch?

Once a week on a random sample of conversations is a good starting point. You're not trying to read every conversation but what you're trying to find are patterns. It could be the same wrong answer coming up repeatedly or a topic where the bot consistently pulls from the wrong article.

After doing this for a while, you'll have a good sense of where the gaps are and can shift to checking after product releases instead. If you're using Pageloop, the product change monitoring handles a lot of this by catching stale content before the bot serves it rather than after.

Does formatting actually affect chatbot accuracy?

Yes, it does. One study found that structured content with clean heading hierarchy improved retrieval accuracy by up to 35% compared to unstructured text.

Image Courtesy Unsplash and Europeana
Hann von Weyhern, Adelaide (geb. Kahle): Märkischer Landsee, Adelaide Hann von Weyhern (1840-1919)