Every team building an AI application faces the same architectural question early on: should we fine-tune a model, build a RAG system, or create an agent? The answer determines your cost structure, maintenance burden, accuracy profile, and development timeline. Getting it wrong means either over-engineering a simple problem or under-engineering a complex one.

There is no universal answer. Each approach occupies a different point in the design space, optimized for different requirements. What teams need is a decision framework – a systematic way to evaluate their specific requirements against the strengths and constraints of each approach.

At CONFLICT, we have built systems using all three patterns (and hybrids of them) across client engagements. This article presents the decision framework we use internally, grounded in the trade-offs we have observed in production.

The Three Approaches in Brief

Fine-tuning modifies a base model’s weights using your domain-specific data. The result is a model that has internalized your domain knowledge and behavioral patterns. It does not need external context at inference time – the knowledge is baked into the model.

RAG (Retrieval-Augmented Generation) keeps the base model unchanged and provides relevant information at inference time through a retrieval system. The model reasons over the retrieved context to produce grounded answers. The knowledge lives in an external data store, not in the model weights.

Agents combine an LLM with tools (APIs, databases, code execution) and a planning/execution loop. The agent decomposes tasks, calls tools to gather information or take actions, and iterates until the task is complete. The intelligence is in the orchestration, not just the generation.

These are not mutually exclusive – production systems often combine them. But understanding when each approach is the right starting point is the critical first decision.

Decision Axis 1: Knowledge Dynamics

How often does the knowledge your system needs change?

Static knowledge (changes rarely, measured in months or years). Examples: medical terminology, legal frameworks, programming language syntax, domain-specific jargon. Fine-tuning is well-suited here. The knowledge can be baked into the model during training and does not need to be updated frequently. The overhead of fine-tuning (data preparation, training, evaluation, deployment) is justified when the knowledge is stable.

Semi-dynamic knowledge (changes weekly or monthly). Examples: product catalogs, internal policies, documentation, knowledge bases. RAG is the natural fit. New and updated documents are added to the retrieval index without retraining the model. The system stays current as the knowledge base evolves.

Highly dynamic knowledge (changes hourly or daily). Examples: live pricing, inventory levels, customer account data, real-time analytics. Agents are the right approach because the system needs to fetch current data at execution time through API calls and database queries. Neither fine-tuning nor RAG can keep up with data that changes this frequently.

Decision Axis 2: Task Complexity

What is the system being asked to do?

Single-step generation. The system receives an input and produces an output in one pass. Classification, summarization, translation, content generation, question answering. Both fine-tuning and RAG handle single-step tasks well. The choice between them depends on the other axes (knowledge dynamics, data availability, cost).

Multi-step reasoning. The system needs to break a task into steps, gather information from multiple sources, and synthesize a result. Research tasks, comparative analysis, complex question answering that spans multiple documents. RAG with a strong retrieval pipeline handles this when the reasoning can be done in a single LLM call with the right context. Agents handle this when the reasoning requires iterative information gathering.

Action execution. The system needs to take actions in the world: call APIs, update databases, send messages, execute code. This is agent territory. Neither fine-tuning nor RAG can take actions – they can only generate text. If your system needs to do things rather than just say things, you need an agent (or at minimum, a structured output system that drives downstream actions).

Conversational interaction. The system maintains a multi-turn conversation, remembering context and adapting its responses. All three approaches support conversation, but agents handle open-ended conversations with tool use most naturally. Fine-tuned models excel at conversations with consistent persona and domain expertise. RAG supports conversations grounded in specific knowledge.

Decision Axis 3: Data Availability

What data do you have, and how much of it?

Fine-tuning requires training data. Specifically, it requires hundreds to thousands of high-quality input-output examples in the format you want the model to produce. Generating this data is expensive – it often requires domain expert review of every example. If you do not have this data and cannot create it efficiently, fine-tuning is not a practical starting point.

RAG requires a document corpus. The quality of a RAG system is bounded by the quality and completeness of its retrieval corpus. If your knowledge exists in well-structured documents, RAG is straightforward. If your knowledge is scattered across Slack messages, meeting recordings, and tribal knowledge, you need a significant data engineering effort before RAG becomes useful.

Agents require tool definitions. An agent needs well-defined tools with clear interfaces, documentation, and error handling. If the systems your agent needs to interact with have clean APIs, agent development is tractable. If they require screen scraping, undocumented endpoints, or manual workarounds, the tool integration effort dominates the project.

Decision Axis 4: Accuracy Requirements

How precise does the system need to be?

Fine-tuning produces the most consistent outputs for tasks within its training distribution. A fine-tuned model for medical report generation will produce reports with consistent structure, terminology, and formatting. The weakness is that fine-tuned models can be confidently wrong on inputs outside their training distribution.

RAG provides source-grounded accuracy. When the retrieval system finds the right documents, RAG systems produce accurate, verifiable answers with citations. When the retrieval system fails (the answer is not in the corpus, or the wrong documents are retrieved), accuracy drops sharply. RAG accuracy is a function of retrieval accuracy.

Agents provide the broadest coverage because they can gather information from multiple sources and cross-reference results. But each tool call introduces potential error, and multi-step chains compound errors. Agent accuracy requires robust error handling and validation at each step.

For high-stakes domains (medical, legal, financial), the accuracy requirements often point toward RAG with human review, because RAG provides source citations that humans can verify. Fine-tuning in high-stakes domains requires exhaustive evaluation to ensure the model does not hallucinate with confidence.

Decision Axis 5: Cost and Latency

What are your operational constraints?

Fine-tuning has high upfront cost and low inference cost. Training a fine-tuned model requires compute for training and human effort for data preparation. But inference is just a model call – no retrieval step, no tool calls. Per-request latency and cost are the lowest of the three approaches.

RAG has moderate upfront cost and moderate inference cost. Building the retrieval pipeline requires engineering effort but no model training. Each request involves a retrieval step (vector search, re-ranking) plus an LLM call with a larger context window. The retrieval adds latency (typically 100-500ms) and the larger context adds token cost.

Agents have low upfront cost for simple cases and high variable cost. A basic agent can be built quickly with a framework like LangChain or a custom loop. But each agent execution involves multiple LLM calls and tool calls, making per-request cost unpredictable and potentially high. A complex task might require 10-20 LLM calls, while a simple task requires 2-3. Cost containment is a first-class engineering concern for agents.

The Decision Framework

Putting the axes together, here is the framework we use:

Choose fine-tuning when:

• The domain knowledge is stable and well-defined
• You need consistent output format and style
• You have high-quality training data (or can create it)
• Low latency and low per-request cost are priorities
• The task is well-scoped and does not require external data

Choose RAG when:

• The knowledge base changes regularly
• Users need answers grounded in specific, citable sources
• The corpus is too large for the model’s context window
• Accuracy depends on accessing specific documents
• You need to update knowledge without retraining

Choose agents when:

• The task requires taking actions (API calls, database updates)
• The system needs to gather information from multiple live sources
• The task complexity varies and requires adaptive planning
• Multi-step reasoning with intermediate tool use is needed
• The system needs to interact with external services

Choose a hybrid when:

• You need a fine-tuned model’s consistency plus RAG’s grounding (fine-tune + RAG)
• You need an agent that retrieves knowledge before acting (agent + RAG)
• You need an agent with domain-specific reasoning (agent + fine-tuned model)

Common Mistakes

Choosing agents when RAG suffices. If your system only needs to answer questions from a knowledge base, you do not need an agent. RAG is simpler, cheaper, more predictable, and easier to debug. Agents add complexity that is only justified when you need planning, tool use, or multi-step execution.

Choosing fine-tuning for rapidly changing knowledge. If you fine-tune a model on your product catalog and the catalog changes weekly, you are either retraining constantly (expensive) or serving stale information (dangerous). RAG handles knowledge that changes.

Choosing RAG when fine-tuning is more appropriate. If your task is producing consistent outputs in a specific format (generating insurance reports, coding in a proprietary framework, translating between domain-specific vocabularies), fine-tuning produces better results than stuffing examples into the context window.

Underestimating agent cost. Agent demos look cheap because they process one task. In production with thousands of requests, the variable cost of agent execution compounds. Budget for worst-case execution paths, not average-case.

Ignoring the hybrid option. The three approaches are not mutually exclusive. A fine-tuned model for style consistency, backed by RAG for factual grounding, orchestrated by an agent for multi-step tasks – this is a legitimate architecture for complex applications. Do not force yourself into a single approach when the requirements span multiple quadrants of the decision space.

Maintenance and Evolution

The maintenance burden differs significantly across approaches:

Fine-tuned models need periodic retraining as your domain evolves, training data curation to address discovered failure modes, and evaluation infrastructure to catch regressions when you retrain. You also need to manage model versions and rollbacks.

RAG systems need document pipeline maintenance (ingestion, chunking, embedding updates), retrieval quality monitoring, and corpus curation (removing outdated content, resolving contradictions). The advantage is that updates are incremental – adding a document does not require retraining anything.

Agent systems need tool maintenance (APIs change, services get deprecated), prompt and planning logic updates as the task space evolves, and cost/performance monitoring. Agents are the highest-maintenance option because they have the most moving parts.

Choose the approach whose maintenance model fits your team’s capabilities. A small team maintaining an agent with 15 tool integrations will spend most of their time on tool maintenance. The same team maintaining a RAG system spends their time on corpus quality and retrieval tuning, which may be more sustainable.

A Real-World Example

A client needed an internal assistant for their engineering organization. The requirements:

• Answer questions about internal systems, architecture, and processes (knowledge base of ~5,000 documents)
• Look up real-time system status and metrics
• Create JIRA tickets for identified issues
• Generate incident reports in a specific format

Our architecture decision:

• RAG for knowledge base questions (semi-dynamic knowledge, need for citations)
• Agent with tool access for real-time status checks and JIRA ticket creation (action execution, live data)
• Fine-tuned model for incident report generation (consistent format, stable template)

The system routes incoming requests to the appropriate pattern based on intent classification. Knowledge questions go through the RAG pipeline. Status checks and ticket creation go through the agent. Report generation goes through the fine-tuned model with RAG-retrieved incident data.

This hybrid approach reflects the reality that production AI systems rarely fit neatly into a single architectural pattern. The decision framework helps you choose the right pattern for each capability, then compose them into a system that addresses the full requirement set.

Start with the framework. Map your requirements to the axes. Let the trade-offs guide you to the right architecture. And build the simplest version that works before adding complexity.