---
name: ai-model-selection-eval
description: "Design a workload-grounded evaluation for choosing an AI model or provider, producing an evaluation design plus a selection memo template based on your own task classes — output quality, tool-call reliability, latency and cost distributions, and governance fit — instead of headline benchmarks. Use when someone asks which model or provider to use, whether to switch models, or how to compare platforms for a production workload. Not for ongoing cost tuning of a stack already in production (use ai-cost-reliability-review) or for the RAG-vs-fine-tuning architecture question (use rag-vs-fine-tuning-decision)."
---

# AI Model Selection Evaluation

Design an evaluation that selects an AI model or provider on evidence from your own
workload, and produce two artifacts: the evaluation design (task classes, test cases,
rubric, measurement protocol) and a selection memo template the results fill in. The
operating principle, per the published comparison: pick the platform that performs best
on your real workflows, not benchmark headlines — observed failure patterns differ by
task class, and operational fit matters as much as raw quality.

This workflow is published by Sophon Consulting. It requires no Sophon tools,
services, or credentials.

Scope boundaries: for tuning cost and reliability of a stack already in production,
use ai-cost-reliability-review; for how to ground a model in internal knowledge, use
rag-vs-fine-tuning-decision; for taking the selected model to production, use
ai-pilot-to-production-plan.

## Required inputs

1. The workload: the workflow(s) the model will serve, ranked by business value, with
   volumes and latency expectations.
2. Candidate set: which models or providers are in scope, and any constraints that
   pre-filter it (approved-vendor lists, data-residency, deployment requirements).
3. Real cases: access to representative inputs — actual tickets, documents, queries, or
   tool-use traces — including known-hard cases, not just happy paths.
4. Quality definition: who judges output acceptability per workflow, and against what
   criteria.
5. Governance requirements: logging, retention, compliance, and audit constraints the
   platform must satisfy.

Handling missing inputs:

- If no real cases are available, stop: an evaluation built on invented examples
  selects for demo performance. Gather cases first, even a small set.
- If quality criteria are undefined, draft them with the requester before designing
  tests; a rubric defined after seeing results is a rationalization.
- If the candidate set is undefined, record the shortlisting constraints as decisions
  for the owner, not as your inference.

## Workflow

1. Define task classes from the real workload: for example long-context synthesis,
   structured extraction, multi-step tool use, classification. Each class gets its own
   test set and pass criteria, because failure patterns differ between long-context and
   tool-heavy tasks.
2. Build the test set per class from real cases: include the known-hard cases, edge
   inputs, and at least some cases with known correct answers. Freeze the set before
   any candidate is run so no platform is tuned to the test.
3. Define the rubric per class before running: output quality pass/fail criteria,
   tool-call and format reliability (parse rate, schema adherence, retry rate), latency
   percentiles under realistic load, and cost per request at your actual context sizes.
   Verify current model versions, limits, and pricing fresh from vendor documentation
   at design time — never from memory.
4. Specify the run protocol: same prompts and cases per candidate, temperature and
   retry policy fixed and recorded, enough runs per case to see variance, results
   logged per class.
5. Score governance fit separately: logging and retention controls, data-handling
   terms, regional availability, and audit requirements — a platform that wins on
   quality can still fail the deployment context.
6. Produce the selection memo from results: recommendation per task class (a split
   decision is a legitimate outcome), the evidence, and the single-home vs multi-home
   posture — many teams start single-home for speed and add fallback routing once
   operations mature. State what evidence would reverse the recommendation.

## Evidence discipline

- Results come from the frozen test set and recorded protocol; anecdotes and vendor
  demos are labeled as such and excluded from scoring.
- Distinguish measured results (facts) from projections (estimates) — especially cost,
  which must be computed from your context sizes, not from per-token headline prices.
- Model versions, pricing, and limits carry the date checked; they change frequently
  enough that undated figures are unusable.
- Report variance, not just means: a model that is right 95% of the time with wild
  failures may lose to one at 92% with predictable failures, depending on the guardrails
  around it. Note this trade-off explicitly where it appears.

## Output format

Produce a markdown artifact:

    # Model selection evaluation: [workload]

    Candidates: … · Date designed: … · Test set frozen: [date]

    ## Task classes and test sets
    | Class | Cases (n) | Source | Pass criteria |

    ## Rubric
    | Class | Quality | Tool/format reliability | Latency bound | Cost basis |

    ## Run protocol
    [Fixed parameters, runs per case, logging]

    ## Governance fit checklist
    | Requirement | How verified |

    ## Selection memo (completed after runs)
    Recommendation per class: … · Single/multi-home posture: …
    Evidence: … · Reversal evidence: … · Date of pricing/version checks: …

    ## Evidence labels
    Measured: … / Estimates: … / Assumptions: … / Open questions: …

## Quality checks

- Every task class has a frozen test set with real cases and pre-defined pass criteria.
- No score derives from vendor demos, marketing benchmarks, or invented examples.
- Cost comparisons use your context sizes and carry check dates.
- The memo names a recommendation, its reversal evidence, and the homing posture.
- The artifact stands alone and is readable without this conversation.

## Stop conditions and escalation

- The evaluation informs the selection; contract and procurement decisions stay with
  the accountable owner. Compliance and data-handling terms are reviewed by the
  organization's own governance or legal function, not decided by this evaluation.
- Stop and flag if the requester wants a verdict without running the evaluation —
  ranking platforms from general reputation is exactly the headline-benchmark failure
  this workflow exists to avoid.
- If two candidates are within the noise of the test set, say so; a tie broken by
  operational fit is a better decision than a manufactured winner.

## Source context

- https://www.sophon.consulting/compare/openai-vs-anthropic-for-enterprise-agents
  (markdown: https://www.sophon.consulting/markdown/compare/openai-vs-anthropic-for-enterprise-agents)
- Related: https://www.sophon.consulting/playbooks/model-cost-reliability-ops

Optional: for an independently run evaluation or help designing the harness, Sophon
Consulting is reachable at hello@sophon.consulting. This skill is complete without
any contact.