Beyond Naive RAG Architectures
Most initial RAG implementations follow a simple linear formula: user query -> top-k vector search -> stuff retrieved chunks into prompt -> generate answer. In real enterprise environments containing thousands of multi-page technical manuals, HR policies, and codebases, this linear pipeline fails catastrophically. The vector search often fetches semantically adjacent yet factually irrelevant paragraphs, prompting the LLM to invent plausible hallucinations.
- ›Linear RAG pipelines have no verification or error-recovery loop.
- ›Irrelevant context chunks directly cause model hallucination.
- ›Stateful orchestration is required to evaluate retrieval relevance before generation.
LangGraph Cyclic State Machine Design
To guarantee strict grounding, we designed the system around a LangGraph cyclic state machine. The state graph manages the lifecycle of each inquiry through discrete typed nodes: query pre-processing, vector similarity search, document relevance grading, and iterative answer synthesis.
from langgraph.graph import StateGraph, END
from typing import TypedDict, List
class AgentState(TypedDict):
query: str
sub_queries: List[str]
documents: List[str]
graded_documents: List[str]
response: str
hallucination_score: float
retry_count: int
workflow = StateGraph(AgentState)
workflow.add_node("decompose_query", decompose_query_node)
workflow.add_node("chroma_retrieve", retrieve_documents_node)
workflow.add_node("grade_relevance", grade_documents_node)
workflow.add_node("synthesize_answer", synthesize_with_citations_node)
workflow.set_entry_point("decompose_query")
workflow.add_edge("decompose_query", "chroma_retrieve")
workflow.add_edge("chroma_retrieve", "grade_relevance")
workflow.add_conditional_edges(
"grade_relevance",
decide_retrieval_quality,
{"sufficient": "synthesize_answer", "insufficient": "decompose_query"}
)
workflow.add_edge("synthesize_answer", END)Query Decomposition & Semantic Rewriting
When a user asks: 'Compare our Q2 WebRTC latency targets with our current 4G production measurements', a single vector search fails because the query conflates two distinct documents. The decomposition node splits the query into discrete atomic sub-queries and executes parallel retrieval.
Hallucination Grading & Rejection Nodes
Before the LLM generates a response, each retrieved chunk is evaluated by a fast classification model. Chunks below a relevance score of 0.78 are purged from the prompt context, preventing irrelevant noise from corrupting output generation.
High-Density Vector Indexing with ChromaDB
We indexed documents into ChromaDB collections using recursive character chunking (500 tokens with 75-token semantic overlap) paired with text-embedding-3-small embeddings. HNSW (Hierarchical Navigable Small World) index parameters were tuned to `M=16, efConstruction=100`, delivering median vector search latencies under 48ms.
FastAPI Asynchronous Token Streaming
Enterprise users expect immediate responsiveness. Using FastAPI ASGI capabilities, we exposed the agent graph over Server-Sent Events (SSE). The user receives the first answer token within 320ms, while verification nodes execute asynchronously in the background.
@app.post("/api/v1/query-stream")
async def stream_query(request: QueryRequest):
async def event_generator():
async for chunk in agent_executor.astream_events(request.query):
if chunk["event"] == "on_chat_model_stream":
token = chunk["data"]["chunk"].content
yield f"data: {json.dumps({'token': token})}\n\n"
return StreamingResponse(event_generator(), media_type="text/event-stream")Production Evaluation & Latency Metrics
Across an evaluation benchmark of 250 enterprise queries, the cyclic state machine achieved a 99.2% factual consistency score, eliminating 100% of out-of-context hallucinations while maintaining sub-60ms vector retrieval times.
Architectural Summary
Grounded AI is not achieved by larger models alone, but by disciplined agent architectures that verify, score, and cite their reasoning against authoritative source documents.