Enterprise RAG Case Study

We built an internal AI assistant for a US distributor of pumps, valves, and fluid handling equipment. Its service teams worked with 12,400 manuals, parts catalogs, and service bulletins. With the assistant, an employee could ask a technical question, open the cited PDF page, and check whether the answer applied to the equipment the technician was servicing.

In a four-week mvp with 48 users, median lookup time fell from 7 minutes 20 seconds to 2 minutes 5 seconds. When the documents could not support an answer, the assistant said what was missing and sent the employee to a specialist.

Why a simple question took seven minutes

The distributor supported equipment from 36 manufacturers, and its service desk handled about 1,600 requests a month. Most of them could be answered from existing documentation. The slow part was finding the right document and confirming that it applied to the specific unit.

SharePoint search worked when staff knew a manual's title or part number. It failed when someone used the customer's name for a component or typed a model number without its hyphen. Older catalogs were often scanned, so their compatibility tables never appeared in search results. Some manufacturers also reused model names across product generations.

Experienced staff knew which bulletin superseded which manual. Newer staff asked a parts specialist or opened PDFs one by one while the technician waited on the phone. The client wanted faster lookups without losing the checks that kept the desk from recommending the wrong part.

What the first release covered

We reviewed 300 closed service tickets with the client's parts and service leads. Three question types came up often, and staff could verify the answers in a document: whether a replacement kit fits a given model and serial range, where to find the manufacturer's instructions for a maintenance task, and which forms a warranty claim needs.

Each answer showed the document title, revision date, and page. Staff could open the source next to the answer and read a table or diagram in its original layout. If a model existed in two versions and the serial number decided the answer, the assistant asked for the number. Warranty decisions and approvals for substitute parts stayed in the service desk's existing workflow.

Why we chose RAG

The client's documentation changed all the time. Manufacturers issued new bulletins and revised compatibility tables, and every answer had to point to a traceable source and respect the employee's document permissions.

That set of requirements pointed to custom RAG development. The assistant retrieved approved passages for each question and tied the answer to a specific document revision. A new bulletin became available as soon as it was indexed, with no retraining. Removing a superseded document removed it from retrieval.

We considered fine-tuning the model for terminology and response format. During evaluation, better retrieval and clearer instructions covered both, so fine-tuning stayed out of the initial scope. The budget went to document extraction, compatibility tables, access control, and evaluation.

Why we built a custom application

During discovery we tested Microsoft 365 Copilot and Glean on a sample of the client's service questions. The client already used Microsoft 365, so an existing product was the obvious place to start.

The decision came down to how the service desk reached an answer. Compatibility checks needed exact matching on model and serial range. A bulletin could override a manual for one production range while the rest of the manual stayed valid. Staff also had to open the cited PDF page beside the answer and log unresolved questions against a specific product.

Supporting those workflows in the shortlisted products would have required substantial configuration and integration work. We built a custom enterprise AI application instead, on top of the client's Azure environment, Entra ID, and SharePoint libraries. That gave us direct control over retrieval rules, document precedence, the verification screen, and the evaluation process.

Where the data lived

The application ran in the client's Azure subscription, inside its Microsoft Entra tenant. Extracted documents were stored in Azure Blob Storage, searchable passages in Azure AI Search, and application records in Azure Database for PostgreSQL. The Azure OpenAI resource sat in the same environment (we compare the architecture options in our guide to ChatGPT integrations for enterprise). Private endpoints connected the application to storage, search, and the models.

We chose GPT-4.1 for answer generation and text-embedding-3-large for embeddings after testing them on representative questions, scanned tables, and ambiguous product references. Every evaluation run recorded the model and prompt versions, so we could compare changes before releasing them.

Preparing the documents for search

A Python ingestion service copied documents from SharePoint into Azure Blob Storage and kept the original file identifier, revision, and access permissions. Exact duplicates were caught by file hash. Conflicting revisions went to document owners, because a later upload date did not always mean a newer document.

PyMuPDF extracted text from searchable PDFs. Azure AI Document Intelligence handled scanned pages and tables. Each passage kept its page number, heading, product family, and revision, and table headers stayed attached to their rows so that a value like 250 PSI remained linked to the right model and operating condition.

The first pass missed some scanned pages where stamps or handwritten notes covered part numbers. A low-confidence flag now sends such pages to a review queue. The client's documentation team can correct the index entry or keep the page out of search until a better copy arrives.

After the initial import, indexing runs incrementally. A worker checks the configured SharePoint libraries every 15 minutes and queues new, modified, and deleted files. The previous version stays searchable until its replacement passes validation. Deleted files leave the index, and superseded revisions appear only when a user searches the archive explicitly. After publishing an urgent bulletin, a document owner can request an immediate refresh. Failed extractions show up in an admin queue with the affected pages and the reason.

How the assistant found and checked an answer

Search combined keyword and vector retrieval. Keyword matching carried more weight when a question contained a model, serial number, or part code. Vector search helped with questions like "What paperwork do I need for a motor that failed on startup?" when the warranty guide used different wording.

The backend normalized common variations in product codes and ranked candidate passages by document type, model match, and revision. If a service bulletin explicitly superseded a manual for the relevant serial range, the bulletin won. If two applicable sources disagreed and neither clearly took precedence, the assistant showed the conflict and referred the user to a specialist.

For missing or possibly outdated information, staff could search approved manufacturer domains. A separate service ran those searches with a sanitized query that contained product identifiers and technical terms only. Internal passages, customer names, and ticket contents never left the client environment. Web results appeared apart from internal sources, and the desk still checked any newer bulletin against its approved documents before recommending a part.

Permissions and security

Employees signed in through Microsoft Entra ID. During ingestion, each indexed document received access metadata: its SharePoint identifier and the users or groups allowed to see it. Before retrieval, the backend resolved the employee's identity and group membership and applied an access filter, so the model only received passages that employee could open. The source viewer ran the authorization check again when a document was opened.

Permission changes went through the same sync queue as content changes, and a scheduled reconciliation caught missed updates. If the connector could not resolve a document's permissions, the document stayed unavailable until the check succeeded. Access revocation had its own test scenario.

We treated retrieved documents and web pages as untrusted input. Their text could serve as evidence but could not change system instructions, grant access, or trigger an action. Ingestion stripped scripts and hidden page content, and retrieved text reached the model inside marked source boundaries. The answering component had no tools for sending email, editing records, or running code.

Before the mvp we tested pages with instructions to ignore previous rules, reveal internal content, or send users to unrelated sites. Release checks confirmed that the assistant kept its access restrictions and backed answers with relevant sources. These controls lower the risk without removing it, so injection scenarios stayed in regression testing and suspicious retrievals were reviewed.

What we logged

We designed logging so that any answer could be traced back later, which is the minimum AI governance requires of an internal assistant. For every request we recorded the user identifier, timestamp, model and prompt versions, retrieved document identifiers, access checks, response time, token usage, citations, and outcome.

The outcome showed whether the assistant answered, asked for clarification, or declined and referred the employee to a specialist. With that, the client could tell whether a weak response came from missing documentation, retrieval, or answer generation.

Full conversations were not stored by default. A user could submit a flagged conversation for review. Only authorized reviewers could see it, and it was kept for a defined retention period. Operational logs excluded document text and credentials.

MVP exposed a problem with near matches

Before MVP, we assembled 240 questions from historical tickets. Service specialists marked which documents supported each answer, which questions needed a follow-up, and which could only be resolved by escalation. We held back 80 questions for the final evaluation and used the other 160 to tune retrieval and response behavior.

An early test found an expensive failure. The assistant cited the right page of a compatibility table but read the row for an adjacent pump model. The answer looked credible because it came with a citation.

We fixed it in two ways. Compatibility rows were indexed together with their headers and model identifiers, and the assistant now needs an exact model match before it gives a part number from a table. We also added more pairs of easily confused models to the evaluation set, plus serial numbers on both sides of known production changes.

MVP started with 12 service desk staff. After reviewing the first week of feedback, we opened it to 48 users, including field technicians and parts specialists. Users could flag a wrong answer, an outdated source, or a missing document from the same screen. Those flags gave the documentation team a list of gaps and showed engineering whether each failure came from retrieval, extraction, or generation.

Results after four weeks

The timing included opening and checking the cited source. An answer counted as correct only when the cited passage supported the specific claim.

During MVP, the assistant handled 1,180 questions.

Of the 118 refusals, 71 came from missing documentation and 29 from conflicting applicable sources. The other 18 involved documents the employee had no access to. In those cases the assistant showed a generic availability message and did not reveal that the document existed.

Service specialists reviewed all 118 refusals and judged 104 of them appropriate. In the remaining 14, an accessible source existed but retrieval missed it. Those cases went into the next search-tuning cycle.

The refusal rate shows how often the system held back. It says nothing about unsupported answers, which is why answered questions were checked separately against the held-out set.

The remaining failures were mostly older scanned tables and local nicknames for equipment. Both became the next priorities for document cleanup and search tuning. The parts team also got a clearer picture of where its documentation slowed people down. A missing bulletin used to surface as a question to a colleague. Now it can be flagged against the actual model and question and assigned to a document owner.

Another RAG project of ours, built for insurance claims and customer service, is described in the AI insurance system case study.

How we delivered the project in 14 weeks

Merehead assigned six people from its AI development team. Two developers handled document ingestion, retrieval, permissions, and backend services. A third built the React application and the PDF verification screen. The QA engineer maintained the evaluation set and tested access changes, ambiguous product questions, and regressions.

The project manager worked through decisions with the client's service and documentation leads. Our DevOps engineer set up the Azure environment, private connectivity, deployments, monitoring, and recovery procedures. On the client side, specialists reviewed technical answers and approved the document precedence rules.

If you are estimating AI app development cost for a similar project, six people over 14 weeks is a useful reference point.