The main complaint about AI is this: "it invents things confidently". And it is fair — a language model on its own knows nothing about your price list, your delivery terms, or the fact that an item was discontinued last year. The technology that fixes this is called RAG, and it is what turns a general model into an agent that knows your company. At SUDUS·TEAM we build a RAG layer into every AI agent project, so here is the explanation without the academic language.
This article is about how an AI agent learns from your company's documents: what RAG is, why it works better than fine-tuning a model, what preparing a knowledge base looks like from the client's side, and the mistakes people make most often.
What RAG is, in plain words
RAG stands for Retrieval-Augmented Generation. The idea is simple: before answering, the agent goes and reads your documents.
Here is the comparison that explains everything. Picture a new hire with an excellent education who started this morning. They are smart and articulate, but they know nothing about your company. There are two routes:
- Make them memorise every internal document — slow, expensive, and every price change means learning it again. That is the equivalent of fine-tuning.
- Give them access to the company's files and teach them to search quickly — instant, cheap, and a price update is picked up automatically. That is RAG.
For 95% of business tasks the right answer is the second one.
How it works inside
Step 1. Splitting documents into fragments
Your documents are cut into pieces of a few hundred words, called chunks. The cutting has to be intelligent: not mid-sentence and not mid-table, but along meaningful boundaries — sections, clauses, FAQ entries. Half the result depends on the quality of this step.
Step 2. Turning them into vectors
Each fragment goes through a separate embedding model that turns text into a long list of numbers — a vector. The key property: texts with similar meaning get similar vectors, even when they share no words at all. So a question about "how long until my parcel arrives" finds the fragment about "delivery times".
Step 3. Storing them in a vector database
The vectors go into a database optimised for similarity search. That is what allows the right three fragments to be found among five thousand in a fraction of a second.
Step 4. Searching during the conversation
A customer asks a question. The question is turned into a vector too, and the database returns the nearest fragments. Good systems use hybrid search — semantic plus classic keyword — because part numbers, model codes and document names are found better literally.
Step 5. Composing the answer
The retrieved fragments are placed in the context alongside the question and a system instruction along the lines of "answer only from the material provided; if the answer is not there, say so". The model writes the reply. That instruction is the main safeguard against invention.
Why RAG beats fine-tuning
- Updates in minutes, not weeks. The price list changed — you replace one document in the base. Fine-tuning would require a full retraining cycle.
- An order of magnitude cheaper. Indexing a knowledge base costs tens of dollars; fine-tuning a model costs thousands.
- It can cite its source. The agent can point to a specific document and clause. A fine-tuned model simply "knows" and cannot explain where from.
- Less invention. The model answers from the supplied text rather than from a memory of a billion pages it once read.
- Access control is easy. Different departments see different parts of the base — trivial at the search layer, close to impossible inside a fine-tuned model.
Fine-tuning still has its uses, but for something else: teaching a model a specific style, tone or answer format. Knowing facts is RAG's job.
What is needed from you: preparing the knowledge base
This is where projects most often stall. The developer does not know your business, so only your team can assemble the right material.
What goes into a knowledge base
- Price lists and tariffs with a validity date
- Descriptions of services and products, technical specifications
- Delivery, payment, returns and warranty terms
- Internal policies and staff instructions
- A history of typical enquiries with the correct answers
- Document and contract templates
- The questions and answers support has already accumulated
Three requirements that save weeks
- One fact, one source. If a price appears differently in three files, the agent will quote a random one of the three. Contradictions have to be removed before indexing, not after customers complain.
- Text, not pictures. A price list scanned into a PDF as an image has to be recognised first. That is doable, but it is a separate stage and separate time.
- Currency with a date. Every document should carry a note of when it was last reviewed. Otherwise a year later nobody will remember whether that promotion still runs.
In our experience, cleaning up a knowledge base takes a client 10 to 40 hours. It is real work, and it is better to plan for it from the start — this is the stage at which whole projects slip.
What it looks like on a concrete example
Abstractions do not stick, so take a real document type — a service centre's price list — and watch what happens to it.
Before: one 40-page file
The price list has sections by device type, each with a table of services, prices and turnaround times, and at the bottom, in small print, warranty terms that apply to the whole document.
The naive split that does not work
Cut the file mechanically every 1,000 words and this happens: one fragment breaks off mid-table, the warranty terms land in a separate chunk with nothing tying them to the services, and the section heading "Laptop repairs" stays in the previous fragment. Asked "how much is a screen replacement and what is the warranty", the agent finds the price without the warranty, or the warranty without the price.
The right split
Cut along meaning — one fragment per service — and write into each the context that was implicit in the original file:
- The section heading is repeated in every fragment: "Laptop repairs → Screen replacement".
- Shared terms (warranty, turnaround, whether diagnostics are included) are added to each fragment rather than sitting on their own.
- Customer-language synonyms are written next to the technical name: "screen, display, panel, cracked screen".
- The validity date goes inside the fragment's text, so the agent can say "as of June".
Same document, same model — but now a question about screens brings back one precise fragment instead of three approximate ones. This work is the bulk of configuring RAG, and it is the part most often skipped.
Hybrid search: why semantic alone is not enough
Semantic search understands meaning beautifully but has a blind spot: exact identifiers. Three examples where it fails:
- Part numbers and models. The query "do you have NP-BX1?" resembles nothing semantically. The vector of such a string is close to random.
- References to clauses. "What does clause 4.2 say" needs a literal match, not similarity of meaning.
- Rare proper names. The name of your tariff or internal programme is something the model is seeing for the first time.
So in production systems the search is always double: classic keyword plus semantic, with the results of both merged into one list and re-scored. It costs almost nothing, and the difference is immediately visible on technical catalogues and contracts.
Access control: who sees what
The question that comes up the moment an agent is tried inside a company. RAG's answer is elegant: access is filtered at the retrieval stage, before the model has seen anything.
Each fragment in the base carries labels — department, access level, document type. When an employee asks a question, the search runs only over the fragments they are allowed. A sales manager physically cannot get an answer out of HR documents: those never enter the result set.
The same mechanism works for customer-facing agents — only the public part of the base is exposed, while internal policies and cost prices stay invisible. That is far more reliable than asking the model in the prompt "not to say too much".
Five mistakes that make RAG work badly
1. Fragments that are too big
Documents get cut every 2,000 words "so there is more context". The result: the retrieved fragment contains the one phrase you needed and 1,950 words of noise, the model gets lost, and you pay for all the tokens. The sweet spot is usually 200–500 words with a small overlap between neighbours.
2. Semantic search only
Without classic keyword search the agent will not find a product by part number and will not understand "clause 4.2 of the contract". Hybrid search is mandatory anywhere codes, numbers and names appear.
3. No reranking
Vector search returns ten candidates, of which three are relevant. Hand the model all ten and quality drops. A separate re-scoring stage that keeps the best three improves answers noticeably for almost nothing.
4. No "I don't know" answer
If the instructions do not say what to do when the information is missing, the model will start constructing something plausible. An explicit instruction to admit ignorance and offer to connect a human is an essential element.
5. The base was indexed once
Six months later the price list has changed three times and the base still holds the first version. Indexing has to be a process, not an event: either automatic updates from your system, or a written routine with a named owner.
How to tell whether RAG is set up well
Three metrics worth asking your contractor for and checking yourself:
- Accuracy on a control set. Take 50–100 real questions with known correct answers and count the hit rate. Below 85% the system is not ready for customers.
- The share of "I don't know" instead of invention. Deliberately ask questions the base has no answer to. The agent should own up every time.
- Average tokens per answer. This translates directly into money. If a simple question consumes 40,000 input tokens, the base is chunked wrongly.
Frequently asked questions
How many documents does RAG need?
It works on ten pages of good text. Our starter package includes a base of up to 100 pages, the professional one up to 1,000. Consistency matters more than volume: 30 verified pages beat 500 full of duplicates and stale data.
Do our documents end up training the model?
No. With RAG the documents live in your vector base and are passed to the model only as context for a specific request. The major commercial providers do not use business customers' data for training. For the most sensitive data we deploy a local model on your own infrastructure, and nothing leaves the building.
Can the agent still be wrong with RAG?
It can, but the errors change type: not invention, but the wrong fragment retrieved. That shows up in the logs and is fixed by tuning the search — unlike things made up from nowhere, which are close to impossible to fix.
How often should the knowledge base be updated?
As changes happen. Prices and promotions immediately; policies and service descriptions quarterly, with a review. For systems connected to a CRM or accounting system, the agent pulls part of the data live and there is nothing to update by hand at all.
Can RAG be added to an existing chatbot?
Yes, if the source code is available. It is a common pattern: the scripted side stays, and free-form questions go to the AI layer. The two approaches are compared in AI agent or Telegram bot.
In short
RAG is not exotic — it is the standard way to make an AI agent speak your company's facts rather than generalities from the internet. The technical side has long been worked out; the deciding factor is almost always the quality of your documents.
So the most useful thing you can do before a project even starts is to gather in one place the current price list, your terms of business, and the top 50 customer questions with the correct answers. With that material a first working agent appears in two weeks. Want to talk through your own case? Write to us.