10 Common RAG Mistakes We Keep Seeing in Production

Enterprise Document Intelligence [Vol.1 #4bis] – A coauthor note on the brick-by-brick pitfalls that justified the four-brick split, before Part II walks the fixes.

Introduction

I am the coauthor of this series with Angela Shi. This pitfalls article lists the failure modes we both kept seeing on production RAG systems, which pushed us toward the four-brick contract in the first place.

1. Parsing: how the document loses its shape

Parsing fails when the team treats the document as text rather than as a structured object. Three patterns keep showing up: discarding tables and layout (Pitfall 1), dumping the entire document into the prompt (Pitfall 2), and chunking the document into fixed-size windows that ignore its structure (Pitfall 3). The fix is a structural parser that produces typed tables instead of strings or arbitrary windows.

1.1 Pitfall 1: The PDF had a table. The parser returned a string.

The cost shows up the first time a table arrives with grouped row labels. Flattened to text, the categories disappear into the token stream and the LLM sees multiple valid answers without clear context.

1.2 Pitfall 2: Pay for 1200 pages on every question

On a real corpus, the approach of returning the whole PDF to the chat can become expensive rapidly as it scales with document size and user queries.

1.3 Pitfall 3: Tuning chunk_size. The PDF had structure.

The fix is a structural parser that keeps the PDF’s typography and structure intact, enabling downstream processes to rely on the original document layout.

2. Question parsing: how you ignore the user

Question parsing fails when the team treats the user’s natural-language question as if it were a query. Two reflexes keep coming back: passing the raw string straight to retrieval (Pitfall 4), and stopping at keyword extraction when the question carried answer shape, scope, and format constraints too (Pitfall 5).

2.1 Pitfall 4: “Just embed the question.”

Real questions take many shapes, and embedding can lose critical structure and constraints needed for proper retrieval and generation.

2.2 Pitfall 5: “Just use HyDE.” Or trust the embedding.

The keyword dictionary approach is superior, allowing a structured response that adheres to user-defined relationships and constraints.

3. Retrieval: the vector DB reflex and its blind spots

Retrieval fails when the only tool is embedding. This leads to important information being lost and means retrieval systems cannot effectively ground answers back to their sources.

3.1 Pitfall 6: “Just use a vector DB”

The reliance on a single retrieval technique can overlook the potential of keyword matches, resulting in lost opportunities for exact retrieval.

3.2 Pitfall 7: The chunk is right. The pipeline stopped there.

Extracting the specific line that contains the sought answer is a necessity for effective response generation.

3.3 Pitfall 8: “See Section 4.2” and never look

Retrieval must resolve references instead of just stopping at pointers within the source document, ensuring a complete audit trail of referenced material.

4. Generation: where the audit chain dies

The generation brick often fails to incorporate effective checks and balances that validate the output of LLMs, leading to potential inaccuracies in responses.

4.1 Pitfall 9: No flag, no schema, no audit. Just text.

Structured outputs from the generation phase with verifiable contacts enhance reliability significantly.

4.2 Pitfall 10: “Not in the chunks” is not “not in the corpus.”

Proving the absence of information based on a comprehensive corpus search as opposed to chunk-based searches is critical to ensure accuracy.

5. What you should expect from Part II

The ten mistakes detailed above stem from structural choices made early in development that can lead to significant operational failures later. Each pitfall highlights a need for improvement in the design of RAG systems.