Two Things an AI Answer Should Refuse to Do
Retrieval alone is not enough. An AI assistant pointed at a firm's documents will happily answer from a policy replaced two years ago, or from a file the person asking was never allowed to open. Docent adds two layers — a validity layer that keeps superseded versions out, and a permission gate that mirrors the source system's access rights, including for administrators. On 131 Dutch legal questions with superseded versions planted as traps, citations of outdated rules fell from 35% to 0% and answers containing the required facts rose from 60% to 90%, with the language model unchanged.
Point a competent AI assistant at a firm’s document library and two failures follow within the week. Both are ordinary. Both are the reason so many internal AI pilots quietly stop.
The first: someone asks about the expenses policy, and the answer is correct — for the version replaced in 2024. The old file is still in the drive, it reads as authoritative, and nothing in the text says it has been superseded.
The second: someone asks a general question, and the answer draws on a file they were never allowed to open. Nothing was hacked. The assistant simply indexed everything it could reach and answered from the best match.
Retrieval-augmented generation, on its own, solves neither. Grounding an answer in retrieved passages fixes invention; it does nothing about whether the retrieved passage should have been retrieved at all. Two layers sit between retrieval and the answer in Docent, and this is how they work.
The validity layer
Documents in a professional firm are not a flat set of facts. They have lifecycles: a procedure is issued, amended, replaced. The replacement rarely deletes its predecessor — the old file stays in the folder, sometimes in three folders.
Each document in Docent carries what it knows about its own lifecycle: an effective date, an expiry date where one exists, and a replaced by relationship pointing at its successor. Some of this comes from the source: an official register states which version of a law is in force, and when it stopped being so. Some is set by the customer, one field on the document. Some is inferred from the document itself.
Before retrieved passages reach the model, the validity filter removes those belonging to a document that has been superseded or has expired. Not down-ranked — removed. A down-ranked old version still competes; a removed one does not.
Where validity is unknown, the document is kept and the answer is marked as drawing on a source with no stated validity. That is the deliberate asymmetry: being wrong about what is current is worse than admitting what you do not know.
The permission gate
Access rights are the harder half, because they live in someone else’s system and they change without telling you.
Docent resolves them per connector, at synchronisation time:
- SharePoint and OneDrive. Library and item-level permissions through Microsoft Graph, including inherited and broken inheritance. Site groups are the difficulty: Graph will tell you a site group has access, and will not tell you who is in it. Resolving membership needs a second authenticated interface with its own token audience, so Docent obtains one and expands the groups.
- Confluence Cloud. Page restrictions and space permissions, read and stored per page, reconciled on the daily full listing so a restriction added yesterday is honoured today.
- Nextcloud. Checked against the asking person’s own rights at query time, inside their own instance.
At retrieval, candidate passages are filtered against the rights of the person asking, before the model sees anything. Three rules make the difference between a permission model and a permission gesture:
- Unresolved means withheld. If a group cannot be expanded or a permission cannot be read, the document does not enter the answer. Fail closed, every time.
- Administrators are not exempt. A document restricted in SharePoint stays restricted here for administrators too. An access model an administrator can step around is not one.
- Mirroring is opt-in per library. Turning it on for a library that was previously open is a visible change, not a silent one.
The cost of fail-closed is that a misconfigured group makes documents invisible until someone fixes it. That is the right cost. The alternative is a system that occasionally shows the wrong person the wrong file, which is not a bug report — it is an incident.
How we know it works
Claims about retrieval quality are easy to make and rarely tested in public, so we built an examination the way a regulator would.
The Dutch set is 131 questions across the Algemene wet bestuursrecht, Participatiewet, Wmo 2015, Jeugdwet, Gemeentewet and the UAVG, taken from the official BWB repository behind wetten.overheid.nl. Superseded 2019 and 2021 versions are planted in the corpus as traps, built from real differences between versions — the kostendelersnorm age change, the bijzondere-bijstand threshold, the voorschot percentage. Every answer key is validated against the article text; a required fact that is not in the named article fails validation before the question is ever asked. A parallel Danish set runs 150 questions over eight acts from retsinformation.dk.
| Measure (Dutch set, 131 questions) | Baseline | With the layers |
|---|---|---|
| Answer contains the facts the question requires | 60% | 90% |
| Correct source among the top five results | 53% | 90% |
| Correct source reached the answer | 66% | 92% |
| Answer cites a superseded version | 35% | 0% |
| Says “not in the documents” when the answer is genuinely absent | 100% | 100% |
The Danish set moved the same way: required facts 48% to 89%, correct legal provision named 17% to 89%, superseded citations 20% to 0%.
The language model was the same in both columns. That is the part worth sitting with. The improvement did not come from a better model, a larger context window or a cleverer prompt. It came from deciding what the model is allowed to see.
What we do not claim
The generation models are licensed, not ours: Google’s Gemini models through Vertex AI in EU regions, or Mistral in France when an organisation wants its provider under EU jurisdiction. The embedding model is licensed too. We do not train models, and customer content is never used to train one.
What is ours is the layer described above — retrieval, validity, permission mirroring, the anonymisation step that replaces personal data before any model call, the citation enforcement that refuses to pass an ungrounded answer to a reader, and the evaluation harness that keeps all of it honest. Two hundred and three automated checks run on every change, and a route guard fails the build if an endpoint ships without authentication.
An answer you cannot check is not an answer. An answer drawn from a document you should not have seen is worse than none. Everything above exists so that the system can say, precisely and provably, no.
The evaluation sets, corpora and method can be demonstrated on request — including to researchers who want to look at the questions themselves.
Frequently asked questions
What is a validity layer in a RAG system?
A validity layer is a filter applied before retrieval results reach the language model. Each document carries dates and, where known, a 'replaced by' relationship to a newer version. Passages from a document that has been superseded are withheld, so the current version does not have to compete with the old one for the model's attention. In Docent's Dutch evaluation set, adding this layer moved citations of outdated rules from 35% of answers to 0%.
How can an AI assistant respect document permissions?
By reading the access rights from the system the document came from and enforcing them at retrieval time, rather than at display time. Docent resolves SharePoint's item and library permissions, including site groups, and Confluence's page and space restrictions, then filters candidate passages against the rights of the person asking. Where the rights cannot be resolved, the document is withheld rather than guessed — the fail-closed choice.
Do administrators see everything?
No. Departmental restrictions are exempt for administrators, because an administrator configuring the system needs to see its structure. Mirrored source-system restrictions are not: if a file is restricted in SharePoint, it stays restricted in answers for administrators too. An access model that an administrator can step around is not an access model.
Does this depend on which language model is used?
No, and that is the point. The measured improvement from 60% to 90% on required facts, and from 35% to 0% on superseded citations, was achieved with the same models before and after. What changed was the retrieval, validity and permission layer around them.
See what your firm already knows.
Docent turns your documents into a queryable, teachable institutional memory — EU-native and GDPR-first.