AI
How to train an AI agent on your company's knowledge base
"Training the AI on company data" is one of the most repeated phrases in meetings and one of the least understood. In most projects, nobody trains any model. What you do is give the agent the right material, at the right time, with clear rules about what it may say. The quality of the agent depends far more on that material than on the model you choose.
What "training" means in practice
The language model comes ready, with general knowledge. It knows nothing about your products, prices and rules. There are three ways to give it that knowledge:
- Instructions. A fixed text that goes with every conversation: who the agent is, how it speaks, what it does and what it does not do.
- Knowledge base. The company's documents and tables. For each question, the system searches for the related passages and hands them to the model along with the customer's message.
- Tools. Real-time lookups in your systems: calendar, stock, price, order status.
Adjusting the model itself, known as fine-tuning, is rare and almost never the first step. It does not solve the problem of information that changes every week.
What goes in the base and what stays out
The temptation is to throw in everything that exists: old manuals, presentations, exported conversations. The result is an agent that answers with outdated information and contradicts itself.
What goes in:
- descriptions of products and services, in the customer's language;
- frequently asked questions with the official answers;
- policies: exchanges, cancellation, warranty, payment methods;
- how the process works after the purchase;
- handoff rules: when to call a person and which one.
What stays out:
- internal documents the customer should not see, such as margin, cost and strategy;
- personal data of customers and employees;
- drafts, ideas and old versions;
- prices, dates and availability written as text, if they change often.
The last point deserves a section of its own.
Prices, deadlines and stock come from tools
Information that changes should not live in a document. If the price is written in a text in the base and someone forgets to update it, the agent will quote the old value with total confidence. The safe path is for the agent to check the system where the official data lives, at the moment of the question.
A protection rule is also worth having: if the agent's reply contains a price, a date or a link, the system checks whether that data came from an official lookup. If it did not, the message is not sent. We cover these safeguards in AI agents on WhatsApp: what they do well and where they need people.
The agent may choose the words. Not the facts. Facts come from an official document or from a system.
How to organize the material
Search works by passages. That is why the way the content is written weighs as much as the content itself.
- One subject per block. Each passage should make sense alone. "As mentioned above" does not work when the "above" was not retrieved.
- A title that says what it is about. "Cancellation policy for the annual plan" is better than "Section 4.2".
- Question and answer. The FAQ format matches the way customers write.
- Tables for structured content. Plans, locations, opening hours and use cases work better in an editable table than in a paragraph.
- One version only. If two documents cover the same topic, merge them. A contradiction in the base becomes a contradiction in the answer.
- Tags. Mark each item with type, topic and which agent it applies to. This lets the sales agent and the support agent use different parts of the same base.
There is also the choice between what always goes in the context and what comes in only when retrieved. Behavior rules and essential information always go. Details about each product come in on demand. That split directly affects the bill, as we show in What it costs to run an AI agent.
The behavior rules
Besides knowing things, the agent needs to know how to behave. The instructions should cover:
- Tone: how the brand speaks, with examples of good and bad sentences.
- Limits: subjects it does not discuss and promises it does not make.
- When it does not know: say it will check and hand off, instead of making something up.
- Handoff to a person: on which signals to pass the conversation and what summary to deliver.
- Privacy: which data it may ask for and which it never requests.
Write rules with the reason next to them. "Do not quote a price without checking the price list, because values change by location" works better than a bare prohibition.
How to test before going live
You do not validate an agent by chatting twice and deciding it looks good. Build a set of test questions:
- the thirty questions customers ask most often, with the expected answer;
- hard questions: out of scope, ambiguous, with wrong information built in;
- attempts to get the agent to promise what it cannot.
Run the set at every change to the base or the instructions. And start in test mode, with the agent replying only to approved numbers, before opening it to customers.
Maintenance: the base is a living product
Once published, the agent shows what is missing. Read conversations every week and note:
- questions it could not answer, which become new items in the base;
- answers that were correct but confusing, which call for rewriting the passage;
- subjects on which customers asked for a person.
Define an owner for the base, the same way a process needs an owner. The format we use for processes, in How to document a process people actually follow, applies here too.
Where to start
List the twenty questions your customer service receives most and write the official answer to each one. That exercise alone reveals where your own team does not have a single answer. This document is the seed of the base, and it serves the human team that same day.
delpra™'s Operational Maturity Assessment shows whether your operations already have processes and data organized enough for an agent to work well.
Written by the delpra™ teamProcesses, automation and AI so operations scale efficiently.