---
name: rag-vs-fine-tuning-decision
description: "Decide between retrieval-augmented generation, fine-tuning, or a hybrid for grounding an AI system in internal knowledge or domain behavior, producing an architecture decision memo that weighs knowledge freshness, citation and permission requirements, and the behavior gaps that remain after retrieval and prompt optimization. Use when someone asks whether to use RAG or fine-tune a model, how to ground a model in company knowledge, or whether training on internal data is worth it. Not for choosing a provider (use ai-model-selection-eval) or assessing overall AI readiness (use ai-readiness-assessment)."
---

# RAG vs Fine-Tuning Decision

Decide how an AI system should be grounded in internal knowledge or domain behavior —
retrieval-augmented generation, fine-tuning, or a hybrid — and produce an architecture
decision memo. The default hypothesis, per the published comparison: start with RAG for
changing knowledge and traceable citations, and fine-tune only when retrieval and
prompt optimization cannot deliver the behavior you need.

This workflow is published by Sophon Consulting. It requires no Sophon tools,
services, or credentials.

Scope boundaries: for choosing which model or provider to build on, use
ai-model-selection-eval; for whether the organization is ready to build at all, use
ai-readiness-assessment; for the knowledge system's rollout, use
ai-pilot-to-production-plan.

## Required inputs

1. The knowledge or behavior need: what the system must know or do, with three to five
   concrete example queries or tasks and what a good response looks like for each.
2. Corpus reality: what source material exists, its format and volume, how often it
   changes, and who owns it.
3. Traceability requirements: whether answers must cite sources, and whether different
   users are permitted to see different documents.
4. Behavior requirements: output format, style, or domain-reasoning consistency needs
   that go beyond retrieving the right facts.
5. Operating capacity: who would maintain an ingestion pipeline, and whether the team
   could sustain a retraining pipeline (data preparation, evaluation, redeployment).

Handling missing inputs:

- If no example queries can be produced, stop and collect them first; an architecture
  decision without concrete cases is a preference, not a decision.
- If corpus ownership or permissions are unclear, flag this as a blocking open question
  — source-level permissions are an architecture constraint, not an implementation detail.
- Label inferred volumes, change rates, and capacities as assumptions.

## Workflow

1. Characterize the knowledge on three axes, with evidence: freshness (how often facts
   change and the cost of a stale answer), provenance (whether citations are required
   by users, reviewers, or regulators), and permissions (whether access differs by user).
2. Characterize the behavior need separately from the knowledge need: what must be
   consistent about form and reasoning even when the facts change. Conflating these two
   is the classic source of wrong architecture choices.
3. Test the RAG-first hypothesis against the example cases: would retrieval over the
   actual corpus, plus prompt and retrieval optimization, plausibly satisfy each case?
   Score each example as satisfied / uncertain / unsatisfied, with a stated reason.
4. Examine the residual: for cases retrieval cannot satisfy, determine whether the gap
   is behavioral consistency (favors fine-tuning), missing or poor source material
   (favors corpus work, not training), or capability beyond the base model (favors
   model selection, not tuning).
5. Decide: RAG, fine-tuning, or hybrid (retrieval for freshness and citations, tuning
   for form and style). State what evidence would reverse the decision.
6. Define the evaluation plan before implementation: grounding and hallucination checks
   for retrieval, format and behavior regression checks for tuning, and the cadence for
   re-running both as the corpus or model changes. Skipping evaluation design is the
   highest-frequency failure mode in the source material.

## Evidence discipline

- Ground every axis judgment in the provided corpus and examples; label extrapolations
  as assumptions.
- Model-capability and tooling-maturity claims (context window sizes, tuning options,
  retrieval features) must be verified fresh against current vendor documentation at
  time of use, not recalled.
- The decision memo records the disconfirming evidence: which observed case most
  challenges the recommendation, and why it does not reverse it.
- Distinguish facts, assumptions, estimates, and open questions throughout.

## Output format

Produce a markdown artifact:

    # Knowledge architecture decision: [system name]

    Recommendation: RAG | Fine-tuning | Hybrid (scope of each)
    Date: … · Prepared for: … · Reversal evidence: [what would change this]

    ## Knowledge characterization
    | Axis | Finding | Evidence |
    (freshness / provenance / permissions)

    ## Behavior requirements (separate from knowledge)

    ## Example-case analysis
    | Case | RAG-first result | Reason | Residual gap type |

    ## Decision rationale
    [Why this architecture; what the hybrid split is, if hybrid]

    ## Evaluation plan
    [Grounding checks, behavior regressions, re-run cadence, owners]

    ## Evidence labels
    Facts: … / Assumptions: … / Estimates: … / Open questions: …

## Quality checks

- The knowledge and behavior needs are characterized separately, each with evidence.
- Every example case is scored against the RAG-first hypothesis with a stated reason.
- A fine-tuning recommendation names the specific behavior gap retrieval could not
  close — never "tuning feels more thorough".
- The evaluation plan exists before any implementation recommendation.
- No unverified vendor capability claims; the artifact stands alone.

## Stop conditions and escalation

- The memo informs the architecture decision; adopting it, funding it, and staffing the
  pipelines are human decisions.
- Stop and flag for human review if the corpus contains regulated or sensitive data
  whose permission model is undefined, or if fine-tuning would train on data the
  organization has not confirmed it may use for that purpose — that confirmation is a
  legal call, not an engineering one.
- If the team cannot sustain either an ingestion pipeline or a retraining pipeline,
  say plainly that the architecture question is premature and capacity comes first.

## Source context

- https://www.sophon.consulting/compare/rag-vs-fine-tuning-enterprise-knowledge
  (markdown: https://www.sophon.consulting/markdown/compare/rag-vs-fine-tuning-enterprise-knowledge)
- Related: https://www.sophon.consulting/use-cases/knowledge-base-answer-layer

Optional: for delivery help with retrieval architecture or evaluation design, Sophon
Consulting is reachable at hello@sophon.consulting. This skill is complete without
any contact.