- Define the questionsChoose a focused topic and expected answers.
- Confirm the sourcesName owners, current versions and precedence rules.
- Test extractionCompare scans and tables with their originals.
- Check the answersVerify evidence, missing-source behavior and access boundaries.
Start with a question, not the whole archive
Decide what people need to ask: how to onboard a supplier, follow a maintenance procedure or find an approved policy. This gives you a manageable starting set and a way to judge whether an answer is useful.
More files do not inevitably produce errors, but a broad archive makes scope, conflicting instructions and permissions harder to manage. Keep only relevant material in the initial search collection. Older versions can remain available for historical questions if their status is clear.
- Write down the questions the first release should answer.
Make the current version easy to identify
Give each document a responsible owner and record its version, approval status and effective date where applicable. Explain which source takes precedence when two documents disagree. A filename containing 'final' is not enough to settle that.
Agree who updates the collection when a policy changes and how withdrawn material is marked or removed from current answers. The search system cannot repair an unresolved business decision about which instruction is valid.
- Keep document ownership and update responsibility visible.
Check what the system actually reads
Digital text, scans, spreadsheets and diagrams need different extraction approaches. OCR can read scans, and multimodal tools can interpret images, but neither guarantees accurate results. Test representative files rather than judging them by their extension.
Compare extracted content with the original. Check column headings, row relationships, units, footnotes and merged cells. For scans, inspect unclear text and rotated pages. Preserve the structure needed to interpret a table instead of flattening it into an ambiguous list.
- Choose test files that include the difficult parts of your work.
Test answers and access together
A user should receive information only from sources they are authorized to access. Check retrieval, generated answers, citations and cached results against that boundary. Include a test where the user has no access to the relevant document.
Make references identifiable: document name or link, plus a page, section or cell when available. Check that the cited passage supports the answer. Ask questions with known answers, conflicting versions and no supporting source. When evidence is missing, the system should say so rather than invent an instruction.
- Have a document owner review answers before wider use.
A practical next step
Choose one topic and its owner, then collect current documents and a small set of representative questions to test together.
Continue with
Make the document readable
Verify that text can be extracted from PDFs and that table structures remain intact. A scan may require OCR, and complex tables might need distinct representations with column headers. Inspect the extracted text manually: having characters in a file does not guarantee that prices, units of measurement, and notes stayed connected.
Implementation details
Section headings, product names, and terms of use must be comprehensible without reading the entire archive. Do not detach exceptions from their general rules. For long documents, set chunk boundaries so the answer can cite a specific section and open it for the user.
How it works
One possible scenario
Question
An employee formulates the task in their own words.
Supporting document
Search locates the document accessible to this user.
Answer with source citation
You can open the source document and verify the context.
Define search access boundaries
Permission to view an answer must not be broader than permission to read its source. Check permissions during retrieval, not merely by hiding citations after generation. The pilot must include queries from various roles, including attempts to retrieve documents without permission.
Implementation details
Document text is raw material for answering, not an instruction for the assistant to alter its rules. OWASP lists prompt injection among key risks of LLM applications. A practical takeaway for the pilot: do not allow arbitrary source text to trigger actions or expose new data on its own.
Prepare questions with expected source citations
Collect real questions from your team and identify the document or section that substantiates each answer. Include ambiguously phrased questions, outdated versions of rules, and queries with no available answer. The latter verifies whether the system can honestly refrain from guessing.
Implementation details
Evaluate separately: whether the correct fragment was retrieved, whether the answer matches it, whether the link works, and whether access permissions were respected. A well-crafted answer drawn from the wrong document is an error. A single aggregated percentage without breakdown masks the root cause.
Agree on how knowledge is updated
Define who approves a new version and when the previous one ceases to be used in answers. If a source is unavailable or sync is delayed, the user must see the appropriate status. Silently relying on outdated instructions can be worse than no answer at all.
Implementation details
Before transferring documents, agree on where originals are stored, which parts are sent to external models, how long logs are retained, and who has access. Do not call a solution fully local just because files reside on your server: the query and processing path must be verified separately.
What is included in a substantive first pilot
The scope can be defined as one document set, agreed-upon roles, control questions, and an answer screen with citations. The outcome of the pilot is a report on specific correct and incorrect answers, along with decisions on what to fix next. This is far more useful than showcasing only successful queries.
Implementation details
If the main issue is conflicting documents, the next step is working with their owners. If sources are consistent but search misses relevant content, indexing can be improved and validated against the same control sample.