At a mechanical level, retrieval in a RAG system runs through four stages:
- Query embedding — the incoming question gets converted into a numerical vector.
- Vector search — a similarity engine scans a vector database for the closest matching chunks of data.
- Context injection — the retrieved chunks get inserted into the LLM prompt as grounding context.
- Generation — the model produces an answer conditioned on that context, not on guesswork.
The rest of this article breaks down each stage, the infrastructure decisions behind them, and where retrieval systems actually fail in production — based on architecture we've shipped on live financial and trading platforms, not theory.
What Is Retrieval in RAG — and Why LLMs Can't Skip It
An LLM without retrieval answers from parametric memory: whatever got baked into its weights during training. That memory is static, unverifiable, and — for anything domain-specific or recent — frequently wrong. Ask a base model about your internal pricing logic or last week's market data, and it fills the gap with a statistically plausible guess. In a chatbot, that's an annoying hallucination. In a trading signal generator or a compliance tool, it's a liability.
Retrieval fixes this by giving the model a citable source. Instead of asking "what do you know about X", the system asks "find the most relevant facts about X, then reason over them". This single architectural shift is what separates a demo from a product you can actually put in front of institutional clients or regulators, because every output traces back to a specific document or data point instead of a black box.
We saw this play out directly while building a multi-agent AI trading system for a client who needed explainable signal generation, not a black-box prediction engine. The team couldn't ship "trust the model" to institutional users — every signal had to point back to the market data that produced it. Retrieval was the only architectural answer that satisfied that constraint.
The RAG Pipeline Step by Step
Query Embedding
The pipeline starts by converting the user's query into an embedding — a dense vector, typically 384 to 1536 dimensions depending on the model, that represents the semantic meaning of the text rather than its literal words. Embedding models (OpenAI's text-embedding-3, Cohere embed-v3, or open-source options like BGE) run this conversion in milliseconds. The critical engineering decision here is consistency: you embed your knowledge base and your incoming queries with the exact same model. Swap embedding models mid-project and every vector in your database becomes incompatible with new queries — a mistake we've seen kill retrieval accuracy overnight on projects that skipped this check.
Vector Search (Similarity Matching)
Once the query becomes a vector, the system searches a vector database for the stored embeddings closest to it — usually measured by cosine similarity or dot product. This is not a keyword match; it's a geometric nearest-neighbor search across potentially millions of vectors. The database returns the top-k closest chunks (commonly 3 to 10), each carrying a similarity score and a pointer back to the original source text.
Context Injection into the Prompt
The retrieved chunks get inserted directly into the LLM prompt, usually with a system instruction like "answer using only the following context". This step is where context window management becomes a real engineering constraint — cram in too many chunks and you burn tokens and dilute relevance; inject too few and the model doesn't have enough grounding to answer accurately. Teams running enterprise ChatGPT integrations hit this trade-off constantly: the fix isn't a bigger context window, it's better retrieval precision upstream.
Response Generation
The LLM generates its answer conditioned on the injected context. Model choice matters less here than most teams assume — Claude, GPT-4-class models, and strong open-source options on the open source LLM leaderboard all perform reasonably well once you feed them accurate, relevant context. Garbage retrieval produces garbage generation regardless of which model sits at the end of the pipeline.
costs to develop
your AI platform
Vector Databases — the Foundation of Retrieval
The vector database is where retrieval quality actually gets decided. We evaluate three options on almost every project that needs a RAG layer:
| Solution | Best for | Trade-off |
|---|---|---|
| Supabase (pgvector managed) | Fast MVP launch, teams already on Postgres | Less tuning control at very large scale |
| PostgreSQL + PgVector (self-hosted) | Teams that want full infrastructure control and no vendor lock-in | You own index tuning, backups, and scaling decisions |
| Pinecone / dedicated vector DB | Very large corpora, managed scaling out of the box | Recurring cost scales with vector count; another vendor dependency |
We picked PostgreSQL with PgVector on our AI trading system build precisely because it kept the whole data layer — relational and vector — inside a stack we already controlled, instead of introducing a separate managed service the team would need to maintain, monitor, and pay for independently.
Challenge: A client needed AI-generated trading signals that traders could actually trust, not a model producing plausible-looking predictions with no traceable source. A single LLM call with no grounding kept generating confident-sounding but unverifiable outputs.
Solution: Our team built a multi-agent architecture (a CrewAI-style setup) where each agent — market analysis, signal processing, decision-making — runs its own system prompt and queries a shared PgVector/Supabase store for grounding data. Market feeds (TraderMade, XE, crypto APIs) and historical price data get embedded and stored as the system's source of truth, so every agent retrieves from the same verified dataset instead of hallucinating from the LLM's parametric memory. The AI layer runs in Python with LangGraph orchestration, decoupled from the Node.js business logic layer, so the retrieval stack can evolve independently of the core product.
Result: The system produces explainable signals — every recommendation traces back to specific embedded market data, satisfying the transparency requirement institutional users demanded. The team shipped this as an MVP within a ~$40,000 budget using a Proof of Concept → MVP → Product rollout, validating the business hypothesis before investing in heavier ML optimization.
Indexing Strategy — Where Most RAG Systems Break at Scale
HNSW vs IVFFlat: Performance Trade-offs
| Index type | Query speed | Build cost | When to use it |
|---|---|---|---|
| HNSW | Fast, consistent recall at scale | Higher memory footprint, slower to build | Production systems with millions of vectors and stable data |
| IVFFlat | Fast once tuned, recall depends on cluster count | Cheaper to build and rebuild | Datasets that change frequently and need regular re-indexing |
Why Index Absence Becomes a Production Blocker
Retrieval speed is fundamentally a database indexing problem before it's an AI problem — a lesson that shows up constantly in high-load crypto exchange architecture work, not just in vector search.
Challenge: On a production crypto exchange, a single query tied to the login and authentication flow started timing out under load, blocking user access to the platform.
Solution: Our engineers ran a performance audit at the query level and found the root cause: a missing database index was forcing a full-table scan on every request. The team added targeted indexing and layered in throttling on older, heavier queries to stop them from re-triggering the same load spike.
Result: The production blocker disappeared without downtime, and a query-level performance audit became a standard checkpoint before any future scaling effort. The same principle applies directly to vector search: skip proper HNSW or IVFFlat indexing on your embedding store, and retrieval degrades exactly the way an un-indexed SQL query does — silently, then catastrophically under load.
Async Re-Indexing — Keeping Retrieval Fresh Without Blocking Production
Worker-Based Architecture for Embedding Pipelines
A retrieval system that only indexes data once at launch goes stale fast. Production RAG needs a re-indexing pipeline that ingests new documents, transactions, or market data continuously — and that pipeline can't run inline with the main API, or every re-embedding job blocks user-facing requests.
Kafka and Redis as a Messaging Layer
We solve this the same way we solve any high-throughput async processing problem: separate the system into web services (HTTP), worker nodes (queue processing), and scheduled jobs, then connect them with a messaging layer.
Challenge: A FinTech platform needed to process high volumes of transactional and market data continuously without blocking its main HTTP API — but the backend framework (Laravel) had no native Kafka support, which the team needed for horizontal scaling.
Solution: Our DevOps team split the architecture into web services, worker nodes for queue processing, and cron jobs running as Kubernetes deployments. They integrated Kafka through a custom library with careful attention to session lifecycle management, so connections close correctly instead of hanging under load. Secrets across all services run through HashiCorp Vault with JWT authentication via GitLab CI/CD, keeping the whole pipeline centrally secured.
Result: The team got an asynchronous processing layer that scales independently from the core API — the same pattern any RAG system needs so that re-embedding new content never slows down response time for the person actually asking a question.
Dense vs Sparse vs Hybrid Retrieval — Choosing the Right Approach
| Approach | How it matches | Strength | Weakness |
|---|---|---|---|
| Dense (embeddings) | Semantic similarity in vector space | Understands meaning, handles paraphrasing well | Can miss exact keyword or numeric matches |
| Sparse (BM25 / keyword) | Term-frequency matching | Precise on exact terms, ticker symbols, IDs | Blind to semantic meaning and synonyms |
| Hybrid | Combines both, often with a reranking step | Best of both — semantic recall plus keyword precision | More infrastructure and tuning overhead |
Most teams start with pure dense retrieval because it's simpler to stand up, then add a sparse or hybrid layer once they notice the system missing exact-match queries — ticker symbols, contract IDs, SKU numbers — that dense embeddings alone tend to blur together. This is also where RAG starts to diverge architecturally from newer patterns like RAG vs MCP for AI agents: MCP standardizes how an agent calls external tools and data sources, while RAG is specifically about grounding generation in retrieved context — the two are complementary, not competing.
Security & Access Control for Retrieval Systems
Vault-Based Secrets Management for Embedding Providers
Every RAG pipeline talks to at least one external API — an embedding provider, an LLM, a market data feed — and each of those needs credentials. We route secrets through HashiCorp Vault with JWT authentication rather than storing API keys in environment files or CI variables, which is standard practice on any financial product we build and a direct carryover from the same discipline we apply to crypto exchange security architecture.
Row-Level Security for Multi-Tenant Vector Stores
If your vector store holds data from multiple clients or business units, retrieval needs row-level security baked in at the database layer — not filtered in application code after the fact. A query that's supposed to search only Client A's embeddings should be structurally incapable of returning Client B's data, regardless of what the application logic does upstream.
What It Costs to Build a Production RAG Layer
Pricing a RAG layer depends heavily on whether it's single-agent or multi-agent, and how large the corpus is that needs indexing. Based on our delivery data across microservice-based FinTech platforms — the same stack a retrieval layer plugs into — here's what a comparable module runs:
| Tier | Scope | Typical cost | Timeline |
|---|---|---|---|
| Basic | Single-agent RAG, one vector DB, static corpus | $15,000 – $22,000 | 4 – 5 weeks |
| Standard | Multi-agent orchestration, hybrid retrieval, async re-indexing | $22,000 – $32,000 | 6 – 7 weeks |
| Advanced | Multi-tenant security, Vault-managed secrets, full observability stack | $32,000 – $40,000 | 7 – 8 weeks |
These figures extrapolate from our fielded rates on comparable trading-platform builds — a team of business analyst, project manager, front-end/back-end developers, blockchain or AI-integration engineers, project architect, and QA, at a front-end hourly rate around $25/h — applied to a standalone retrieval module rather than a full platform build.
For comparison, a full crypto trading platform in our Standard tier runs $49,000–$64,000 over 2.5 months; a RAG layer is a smaller, isolated slice of that same engineering effort. It tracks closely with published AI agent development cost ranges, since a multi-agent RAG system and an agentic AI product share most of the same infrastructure.
The ROI argument here isn't "more accurate predictions" — it's explainability. An enterprise AI development project that can show exactly which data point produced which output clears compliance review faster and earns trader or institutional trust faster than a model that just asserts an answer.
Production Readiness Checklist
- Vector index type (HNSW/IVFFlat) matches your data volatility and query volume.
- Re-indexing runs asynchronously through workers, never inline with the request path.
- Embedding model stays consistent between corpus indexing and query time.
- Secrets for embedding and LLM providers route through Vault or an equivalent secrets manager, not env files.
- Multi-tenant data enforces row-level security at the database layer.
- Observability (logs, query latency, retrieval hit rate) runs through a monitoring stack like Grafana, not ad hoc debugging.
- Context injection has a defined token budget so retrieval quality doesn't get diluted by over-stuffed prompts.
FAQ
How is retrieval different from fine-tuning an LLM?
Fine-tuning bakes knowledge into the model's weights permanently and requires retraining to update. Retrieval keeps knowledge external and swappable — you update the vector database, not the model, whenever your data changes.
What vector database should a CTO choose first?
Start with PostgreSQL + PgVector or Supabase if your team already runs Postgres — it avoids a new vendor dependency and scales well until you're indexing tens of millions of vectors, at which point a dedicated vector DB becomes worth the added cost.
How much does a production RAG system cost to build?
A single-agent RAG layer typically runs $15,000–$22,000, while a multi-agent system with hybrid retrieval and async re-indexing runs $22,000–$40,000, depending on corpus size and security requirements.
Does RAG eliminate hallucinations completely?
No — it reduces them significantly by grounding responses in retrieved data, but the model can still misinterpret context. Strong retrieval precision and clear prompt instructions to answer only from provided context cut hallucination rates substantially.
Can RAG and MCP work together in the same system?
Yes — RAG handles grounding generation in retrieved context, while MCP standardizes how an agent calls external tools and APIs. Many production agentic systems use both simultaneously.
