Your documents can make an AI answer worse

The answer looks unusually trustworthy. It names a product version, gives a direct response and includes a link to the company’s own documentation. The problem is that the linked guide describes an older release. The assistant has grounded its answer in a real source and still given the user the wrong information.

Consider a fictional drawing application called Sketchfield. A user of release 2.1 asks, Can I export SVG without a plug-in? Build a small teaching library around that question:

Fictional record	Applies to	What it says
SVG export questions	1.x	SVG export requires the add-on.
Version 2 manual: vector output	2.0 onward	Native vector export is available; consult format-specific notes.
SVG support notice	2.1 onward	SVG is included in native export without the add-on.

These records and their contents are invented, not a measured failure by a real product. The older page repeats the user’s wording; the manual uses a broader term. The short notice carries the decisive release-specific statement. Returning all three documents would provide the needed information, but it would not select the correct rule by itself.

Adding company documents to an AI system changes where an answer can come from. It does not automatically establish which document should govern, whether a passage answers the question or whether the generated response remains faithful to it. The distinction is easiest to see by following the answer through five decisions.

Decision one: what was allowed into the library?

The archive has a legitimate purpose. People using older releases still need their manuals. Deleting every historical document would make the collection less useful. The problem is not the existence of the version-one guide; it is the absence of a reliable way to distinguish its scope from that of the current instructions.

Before any model receives a passage, ingestion can preserve a document’s title, identifier, applicable version, publication or revision date and status where those fields exist. It should also record where those values came from. A date guessed from a filename is not equivalent to an explicit effective-date field supplied by the document’s owner.

The library also needs to reflect changes after initial ingestion. A search index can retain a deleted or superseded record even when the public source has changed. Conversely, a new document may exist on the website but not yet be available to retrieval. A system claiming current answers should be able to explain the freshness of the collection, rather than only the freshness of the model.

Access belongs at this stage as well. A document’s relevance does not establish that this user may receive it. Search results and source links should honor the underlying permissions, including after a person’s role changes. Grounding an answer in an inaccessible internal source can create a confidentiality problem even when the factual content is correct.

Decision two: which passages looked relevant?

Retrieval selects material to place before the model. It may use lexical matching, vector similarity, structured filters, reranking or a combination. These methods can be useful without being interchangeable. A matching product code can matter more than a broad semantic resemblance; an applicable version can matter more than either.

The foundational 2020 retrieval-augmented generation paper combined a learned generator with retrieved external memory and reported improvements on the tasks it evaluated. That result explains an important architectural opportunity. It does not certify every later enterprise search implementation or guarantee that supplying an arbitrary internal archive improves every answer.

For Sketchfield, similarity alone can favor the wrong guide because the old paragraph contains the exact phrase the user typed. The current notice might call the same feature by a new name. A useful retrieval design therefore has to consider applicability as well as resemblance. The nearest passage in a vector space is not necessarily the document entitled to answer the question.

Splitting documents into small chunks can complicate that decision. A paragraph about exporting might be detached from the heading that specifies an older version. A note that says the following limitation no longer applies might be separated from the limitation itself. A chunk can remain fluent after losing the context that determines its meaning.

Preserving nearby headings or attaching relevant metadata can help, but larger chunks have costs too. They may contain more irrelevant material and make it harder to locate the needed statement. The appropriate unit should be tested against the documents and questions at hand. A universal chunk size chosen without that evidence is a convenience setting, not an established answer-quality policy.

Decision three: what happened when the documents disagreed?

Suppose the retrieval stage returns all three fictional records. The system now has the old guide, the updated manual and the short amendment. Adding more passages has recovered the right evidence without yet determining the right conclusion. The assistant still needs a way to identify which statements apply together and which have been superseded.

Newest is not always sufficient. A recently edited overview may describe the whole product while a slightly older technical note supplies the exact limitation for a particular release. A future-dated announcement may not yet apply. A document updated to fix a typographical error may retain the same substantive rule. Effective scope and revision time answer different questions.

The safe result for an unresolved conflict is not a confident blend. An assistant that combines the old restriction with the new feature can produce an answer found in none of the sources. Instead, it should identify the conflict and seek the authoritative record or appropriate clarification. Saying what remains uncertain is useful information when the library itself cannot settle the issue.

Context arrangement can influence this process. Lost in the Middle, first submitted in 2023, found position-sensitive performance in the question-answering and retrieval tasks it tested. That is a reason to evaluate document ordering and context use. It should not be promoted into a timeless claim that every current model ignores the middle of every input.

For Sketchfield, hold the release 2.1 notice fixed while moving it before, between and after the other records. Keep the question, wording and other settings the same. A changed result would identify sensitivity to arrangement in this configuration. It would not show that every long-context model behaves that way.

Decision four: did the answer say what the evidence supported?

Once the right passages are present, generation can still introduce an unsupported detail. The assistant might infer that a feature available in one edition exists in every edition, or turn a conditional instruction into a universal one. These errors are not retrieval misses. They occur between the supplied evidence and the final claim.

The 2024 EMNLP paper Attribute or Abstain examined attribution across six long-document tasks and five language models. It distinguished long-document attribution from retrieval-augmented settings, where initial retrieval can confound the result. Evidence quality predicted response quality for simple responses, but not for complex ones in its tests. It also did not find the same positional effect for attribution that a casual generalization from earlier long-context work might predict. A test of selecting documents and a test of supporting claims inside supplied documents answer different questions.

For the fictional export question, an answer could separate the supported parts: the applicable release, the available format and any limitation explicitly stated in the current record. If an edition or account setting remains unspecified, the assistant should not fill it from habit. A short answer that preserves those conditions can be more useful than a polished paragraph that silently removes them.

Citation placement matters. A link at the end of a long paragraph may support its first sentence while leaving the rest ungrounded. An answer should make the relationship between claim and evidence easy to inspect. That does not require placing a citation after every word; it requires avoiding a visual impression of support broader than the source supplies.

Decision five: can the reader inspect the same evidence later?

A source title is not enough if the link opens a different version or lands at the top of a long document without the relevant passage. The system should retain stable identifiers and, where appropriate, version information that lets a reviewer reconstruct the basis of an answer. A search session is transient; the decision made from it may not be.

There is a tension between stable evidence and current information. An archived snapshot can show what the assistant used, while a live page can show what the publisher says now. Both can be useful if clearly identified. Quietly replacing the old evidence with the current page can make an earlier mistake impossible to diagnose and an earlier correct answer appear inexplicable.

The user-facing presentation should also distinguish a direct quotation, a paraphrase and the assistant’s own inference. An inference can be reasonable and valuable. Labeling it as an inference allows the reader to evaluate the reasoning instead of assuming that a source expressly stated the conclusion. Evidence-backed analysis is not the same as copying a sentence from a manual.

Repair the failed decision, then rerun the question

The fictional wrong answer can now have several causes: missing version metadata, stale ingestion, poor retrieval, an unresolved conflict, unsupported generation or an unhelpful citation. Replacing the language model might affect some of these. It cannot reliably repair a document that was never indexed or an authorization rule that was never applied.

A useful evaluation separates those stages. Did retrieval return the necessary current evidence? Did it retain the context needed to interpret it? Did the response use the evidence correctly? Could the user inspect it? Measuring only whether a final answer contains a source link would treat several distinct failures as successes.

The test collection should include questions with answerable current records, questions that require archived records and questions the library cannot settle. It should also include misleadingly similar documents and legitimate exceptions. A system that always cites the newest page may do well on a simplified test while failing the users who actually need version-specific guidance.

When the corpus changes, rerun the relevant questions. A new manual can alter retrieval rankings even for an old feature; a renamed heading can break a citation path. Document maintenance and answer maintenance are consequently connected work. A grounding system needs an owner for both, rather than treating the index as a one-time installation.

The same fictional question provides a small repair test. Release 1.8 should still lead to the add-on instruction; release 2.1 should lead to the native SVG notice. An unspecified release should prompt a version question rather than combine the two. Removing the 2.1 notice should change the justified confidence of the answer, even if the model happens to remember a yes. Record which passage made each answer supportable. This tests the library’s contribution, not merely the model’s ability to produce the expected word.

The corrected Sketchfield answer is not impressive because it has a citation. It is useful because the cited record applies to the requested release, the claim stays within the record’s scope and uncertainty remains visible where the evidence ends. Documents can make an assistant better informed. They can also give a wrong answer a convincing set of footnotes. The difference is the sequence of decisions between the archive and the reader.