For most business use cases, building generative AI means building an application on top of an existing foundation model, not training a model of your own. The real decision is how much you customize that model: call it through an API with good prompts, ground it in your own data with retrieval-augmented generation (RAG), fine-tune it, or train a custom model.
Your data, accuracy requirements, budget and need for control decide which path fits. After that, the build follows the same sequence every time: define the use case, prepare data, select a model, build the pipeline, evaluate, deploy and monitor.
What "building generative AI" actually means
When a company says it is building generative AI, it is usually building the application layer: the prompts, data connections, business logic, interface and safety checks that turn a general-purpose model into a tool that does one job well. The model itself, such as GPT, Claude, Gemini, Llama or Mistral, already exists and is accessed through an API or hosted on your own infrastructure.
It helps to think in three layers:
Most of your engineering time goes into the top two layers. The bottom layer is something you choose, not something you build, unless you have a reason to train your own model. That case is rare, and the FAQ below covers what it involves.
The same structure applies whether the output is text, code, images or a mix. A support assistant, a contract summarizer and a product-image generator all sit on a foundation model with a customization layer and an app on top.
Choose your build approach: API, RAG, fine-tuning or a custom model
Pick the lightest approach that meets your accuracy, data and control requirements, and move to a heavier one only when testing shows the lighter one falls short.
The four approaches compared
| Approach | What it is | Use it when | Data you need | Relative cost | Time to first version | Control | Main risk |
| Prompting via API | Call a hosted model with well-designed prompts and examples | The task is general (drafting, summarizing, classifying) and public model knowledge is enough | A few examples and a test set | Low | Fastest | Low | Inconsistent output, vendor dependency |
| RAG | Retrieve relevant documents from your data and pass them to the model with each request | Answers must reflect your own, private or frequently changing information | A document corpus you can index and keep current | Low to medium | Fast | Medium | Poor retrieval quality leads to wrong or incomplete answers |
| Fine-tuning | Further train an existing model on your examples | You need a specific style, format, domain vocabulary or behavior that prompting can't produce reliably | Hundreds to thousands of high-quality labeled examples | Medium | Moderate | High | Overfitting, retraining when the base model changes |
| Custom model (pretraining) | Train a model from scratch or continue pretraining on a large corpus | You need a fundamentally new capability or a model built on a large proprietary corpus | Very large datasets | Very high | Slowest | Full | Cost, talent and compute requirements |
Cost and time in this table are relative to each other and depend heavily on scope. They are not price estimates.
How to decide
Work through these questions in order:
- Can a strong general model do the task with good prompts? Test it on 30 to 50 real examples before building anything. If quality is acceptable, stop here and ship the prompting approach.
- Do answers depend on your own or fresh information (policies, product catalogs, tickets, contracts)? Add RAG. Fine-tuning is a poor way to teach a model facts that change, because you would have to retrain every time they do.
- Is the problem behavior rather than knowledge? If the model knows enough but won't stick to your format, tone or domain conventions, fine-tune it.
- Do none of the above work, and do you have a large proprietary corpus plus a research-level budget? Only then consider training or continued pretraining.
These approaches stack. A common production setup is a fine-tuned model that follows your output format, fed by a RAG pipeline that supplies current facts. For a deeper look at retrieval and tool-based architectures, see our guide to RAG vs MCP.
How to build generative AI in 7 steps
The steps below apply to every approach. What changes is how much work lands in steps 2 and 3.
1. Define the use case and success metrics
Write down one specific job the system will do, who uses it and what "good" looks like. "An assistant that answers employee HR questions from our policy documents, with a cited source in every answer" is a use case. "Use AI for HR" is not.
Then set measurable targets before any code exists. Typical ones are answer accuracy on a test set, share of answers that need human correction, response latency, cost per request and adoption by the intended users. Without these numbers you can't tell whether a change to the prompt or model made things better or worse.
A common mistake is starting with the technology ("we want to use agents") instead of the job.
2. Prepare and govern your data
Data work depends on the approach you picked:
- Prompting: a set of real input examples and expected outputs for testing.
- RAG: a clean, deduplicated document corpus, split into chunks that make sense on their own, with metadata (source, date, access level) attached.
- Fine-tuning: labeled input-output pairs that reflect exactly the behavior you want. Quality matters far more than volume here.
Settle governance questions early. Who owns each data source? Can it be sent to a third-party model provider under your contracts and regulations? Which users are allowed to see which documents? Access rules you forget at this step tend to surface later as a data leak through the model's answers.
3. Select the model
Choose between proprietary models accessed by API (OpenAI, Anthropic, Google) and open-weight models you can host yourself (Llama, Mistral and others). Proprietary APIs usually give you the strongest general quality with no infrastructure work. Open-weight models give you more control over data residency, cost at high volume and customization.
If the product's advantage depends on internal data, decide where that data is allowed to go before you compare providers. When we scoped an AI assistant for a crypto trading platform that would use the exchange's own liquidity data, keeping that data inside an agreed project boundary was a hard requirement, and whether a hosted GPT-class API could be used at all depended on how the data flow was designed, not on answer quality.
Compare candidates on:
- quality on your task, measured with your own test set
- context window size, if you'll pass in long documents
- latency and throughput
- price per token, or hosting cost for self-hosted models
- license terms and data-handling policies
- where data is processed, if you have residency requirements
Public benchmarks and leaderboards are a useful shortlist, not a decision. Run the shortlisted models side by side on 50 or more of your real inputs and score the outputs. Our note on what to check in an open-source LLM leaderboard covers how to read those rankings.
4. Build the application pipeline
This is where the app takes shape. A typical RAG request flows like this:
The pieces you build:
- Prompts. Clear role, task, constraints and output format. Keep prompts in version control so you can trace which version produced which result.
- Retrieval. Embeddings, a vector store and a search step that returns the right passages. Hybrid search (keyword plus vector) often helps when users search by exact terms like product codes.
- Orchestration. The logic that chains steps together, calls tools or APIs, and handles failures. Frameworks like LangChain and LlamaIndex speed this up, though plain code is fine for simple flows.
- Integrations. Connections to the systems where the work happens: CRM, ticketing, document storage, internal APIs.
Agents, where the model decides which tools to call and in what order, belong here too. They add flexibility and also add failure modes, so introduce them only when a fixed pipeline can't handle the task.
Don't ask the LLM to do work that a deterministic tool or a specialized model does better. In the same trading-assistant scoping, the proposed design left price forecasting to dedicated time-series models and used the LLM to choose which tools a question needs and to explain their output. That split lets you test numeric accuracy separately from the quality of the explanation.
5. Evaluate and add guardrails
Build a test set from real or realistic user queries, each with an expected answer or a clear grading rule. Run it after every change to prompts, models or retrieval settings. Check for:
- Accuracy and completeness against the expected answers.
- Hallucinations: claims not supported by the provided context.
- Out-of-scope requests: does the system decline politely or improvise?
- Prompt injection: inputs that try to override instructions or extract the system prompt.
Automated scoring with a second model ("LLM-as-judge") scales well but needs spot-checking by people, because the judge model makes mistakes too.
Guardrails sit around the model on both sides. Input checks filter out-of-scope and malicious requests. Output checks validate format, block unsafe content and verify that cited sources exist. The OWASP Top 10 for LLM Applications is a practical checklist here: its 2025 edition lists prompt injection first and also covers sensitive information disclosure, excessive agency, system prompt leakage and unbounded consumption, among others.
6. Deploy
You have two broad options. A managed model API or cloud AI platform (such as Amazon Bedrock, Microsoft Foundry or Google Vertex AI) handles model hosting and scaling for you. Self-hosting an open-weight model gives you full control over data and cost at scale, but you take on GPU infrastructure, scaling and model updates.
Either way, production deployment needs authentication and role-based access, isolation of each customer's or department's data, rate limits, logging that doesn't store sensitive content you aren't allowed to keep, and a fallback when the model provider is slow or down.
7. Monitor, measure and improve
Once real users arrive, trace every request: input, retrieved context, prompt version, model response, latency and token cost. Tools such as LangSmith, MLflow and Opik do this for LLM apps.
Watch for quality drift (new kinds of questions your test set doesn't cover), cost creep and slow responses. Collect user feedback, even a simple thumbs up or down, and feed failing cases back into the test set. The system improves through this loop, not through the initial build.
The generative AI tech stack, layer by layer
Most teams assemble their stack from managed services and a few open-source components. The tools below are examples, not recommendations; the right picks depend on your cloud, team skills and data rules.
| Layer | What it does | Example tools |
| Foundation models | Generate text, code or images | GPT, Claude, Gemini (API); Llama, Mistral (open-weight) |
| Model access and hosting | Serve models via API or your own infrastructure | Provider APIs, Amazon Bedrock, Microsoft Foundry, Google Vertex AI, Hugging Face |
| Orchestration | Chain prompts, retrieval, tools and agents | LangChain, LlamaIndex, or custom code |
| Data and vector store | Store embeddings and retrieve relevant context | pgvector (PostgreSQL), Pinecone, Weaviate, Chroma |
| Application and UI | Backend, APIs and user interface | Python with FastAPI or Flask; Gradio or Streamlit for prototypes |
| Evaluation and observability | Test quality, trace requests, track cost | LangSmith, MLflow, Opik |
Python dominates the model, data and orchestration layers. The application layer can be built in whatever your team already uses, including TypeScript or Java.
What it takes to get to production
A working demo proves the idea is possible. Production readiness proves it holds up under real users, real data and real costs, and that gap is where most of the effort goes. Before launch, check that you have:
- An automated evaluation pipeline that runs on every change (step 5)
- Input and output guardrails, including prompt-injection defenses
- Versioned prompts, model settings and retrieval configuration
- Token cost controls: trimmed system prompts, summarized conversation history, retrieval that sends only relevant passages, and cheaper models for simple subtasks like classification
- Rules that skip the LLM entirely when deterministic code can do the job
- Load and latency testing at expected peak traffic
- Request tracing and alerting (step 7)
- Fallback behavior when the model or a dependency fails
- Data privacy and compliance review for every data flow to and from the model
How much it costs and how long it takes
Cost and timeline depend mostly on three things: the approach you choose, how ready your data is and how many systems the app has to integrate with. Prompting and RAG projects cost far less and ship far faster than fine-tuning, and training a custom model is in a different category altogether.
| Cost driver | Prompting / API | RAG | Fine-tuning | Custom model |
| Model usage or training | Pay per token | Pay per token, plus embedding costs | Training runs plus inference | Large-scale compute for training |
| Data work | Minimal | Corpus cleaning, chunking, indexing | Building labeled datasets | Collecting and curating very large datasets |
| Development | App and prompts | App, retrieval pipeline, access control | App plus training and evaluation pipeline | Dedicated ML research and engineering |
| Infrastructure | Usually none beyond the app | Vector store | Training and hosting resources | GPU clusters |
| Ongoing | Token spend, monitoring | Token spend, index updates | Retraining when data or base models change | Continuous training and operations |
| Relative timeline | Shortest | Short to moderate | Moderate | Longest |
Model usage is billed per token, and prices vary widely between models and change often. Check the providers' current pricing pages (OpenAI, Anthropic, Google) and estimate monthly spend from expected requests multiplied by average input and output tokens.
User count alone won't give you an estimate. For the trading-assistant MVP mentioned above, the target was about 100 users a day, yet the infrastructure couldn't be priced until three other questions were answered: how much historical market data to store, how often it refreshes, and whether a model of roughly 20 billion parameters would run on dedicated GPU servers or through an API.
Ongoing costs are easy to underestimate. Token spend grows with usage, indexes need updating as documents change, and model upgrades require re-running your evaluation suite. For budget ranges by project type, see our breakdown of AI app development cost.
Team and the build-vs-partner decision
A prompting or RAG application is mostly a software engineering project with an AI component. Fine-tuning and custom models add specialist ML and data roles.
| Approach | Core roles |
| Prompting / API | Backend developer, frontend developer, product owner with domain knowledge |
| RAG | The above, plus a data engineer for ingestion, indexing and access control |
| Fine-tuning | The above, plus an ML engineer for training, evaluation and model versioning |
| Custom model | ML researchers, ML infrastructure engineers and a dedicated data team |
QA with LLM-specific testing skills and someone accountable for security and compliance are needed at every level.
Building in-house makes sense when GenAI is core to your product, you already have engineers who can own the system long term and you can afford the learning curve. Working with a development partner makes sense when you need to validate an idea quickly, lack ML or LLM production experience on the team, or when hiring would take longer than the project itself. Many teams combine the two: a partner builds the first production version and the internal team takes ownership afterward.
Next steps
Pick one narrow use case, build a test set of real examples and try the simplest approach first. If prompting alone meets your targets, ship it and invest in evaluation and monitoring. If it doesn't, add retrieval or fine-tuning based on what the failures show. To learn the hands-on side, Microsoft Learn's Develop generative AI apps path and the IBM course Develop Generative AI Applications: Get Started on Coursera are solid starting points. If you need help taking a GenAI product to production, Merehead's AI development team can help you scope it.
FAQ
Yes, but it is rarely the right choice for a business application. Training a competitive general-purpose model requires very large datasets, large GPU clusters and specialized research teams. Stanford's 2024 AI Index estimated the training compute alone at about $78 million for GPT-4 and $191 million for Gemini Ultra. Smaller domain-specific models cost far less, but fine-tuning an open-weight model usually reaches a similar result faster and cheaper.
The models are trained on large datasets to predict the next token (for text) or to generate images from noise, and then refined with instruction tuning and human feedback. Applications are built on top of those trained models by adding prompts, your data, guardrails and an interface.
Python is the standard for model training, data pipelines and most orchestration frameworks. The application layer is often built in JavaScript or TypeScript, or in whatever language your existing backend uses, since it talks to the model over an API.
