---
name: ai-cost-reliability-review
description: "Review the cost and reliability of an AI/LLM stack already serving production traffic: inventory request classes, baseline cost, latency, failure, and human-rework metrics, then recommend routing, caching, fallback, and measurement policies — with every vendor price or discount verified fresh, never recalled. Use when someone says model spend is volatile or unexplained, asks how to cut LLM costs without losing quality, or needs reliability policies for production AI traffic. Not for choosing a provider (use ai-model-selection-eval) or planning a pilot rollout (use ai-pilot-to-production-plan)."
---

# AI Cost and Reliability Review

Review a running AI/LLM stack and produce a cost and reliability operations review:
a request-class inventory, a measured baseline, recommended routing, caching, and
fallback policies, and a measurement plan that keeps spend predictable without
sacrificing quality. The operating principle: route by request class instead of
one-model-fits-all, and treat reliability as a measurement system, not a launch check.

This workflow is published by Sophon Consulting. It requires no Sophon tools,
services, or credentials.

Scope boundaries: for choosing a model or provider in the first place, use
ai-model-selection-eval; for taking a pilot to production, use
ai-pilot-to-production-plan; for deciding where humans must review AI actions, use
human-in-the-loop-governance.

## Required inputs

1. Traffic shape: the workflows served, requests per day per workflow, and typical
   context length and output length for each.
2. Current model assignments: which model serves which traffic, and why (if known).
3. Cost data: current spend by workflow or endpoint, and how it is measured today.
4. Quality bars: what counts as an acceptable response per workflow, and what happens
   downstream when quality drops (retries, human rework, customer impact).
5. Reliability posture: latency bounds, current failure handling, and what breaks when
   the model provider degrades or rate-limits.

Handling missing inputs:

- If no cost breakdown exists, make building one the first recommendation and baseline
  from whatever invoice or dashboard totals are available, labeled as estimates.
- If quality bars are undefined for a workflow, flag that workflow as untunable: routing
  it to a cheaper model without a quality bar converts savings into silent regressions.
- Label every inferred number as an assumption; never present an estimate as a measurement.

## Workflow

1. Build the request-class inventory: group traffic by intent, volume, context length,
   quality bar, and latency bound. A request class is the unit every later policy
   applies to.
2. Baseline each class: cost per request, latency percentiles (p50/p95), failure and
   retry rates, and downstream human rework. Record where each number came from.
3. Design the routing policy: match each class to the cheapest model tier that meets its
   measured quality bar, with the assignment justified by evidence, not habit. Flag
   classes where nobody has tested a cheaper tier.
4. Design the caching policy: identify repeated-prefix traffic (system prompts, shared
   context, templates) and estimate the cacheable share per class. Verify current
   provider caching mechanics and discounts fresh from vendor documentation at time of
   use — these terms change often and must not be quoted from memory.
5. Design the fallback and degradation policy per class: timeout and retry budgets, the
   fallback model or queue-for-human path, and the rollback trigger that reverts a
   routing change (for example, rework rate above threshold for two consecutive days).
6. Write the measurement plan: who reviews cost, quality, and reliability per class,
   at what cadence, and which thresholds trigger retuning.

## Evidence discipline

- Distinguish measured values (from telemetry or invoices), estimates (stated basis),
  and assumptions. Unverified numbers do not justify routing changes.
- All vendor pricing, caching discounts, rate limits, and model claims must be verified
  against current vendor documentation at time of use. Do not embed remembered figures
  in the review.
- Cheaper is not automatically cheaper: lower unit cost can increase retries and human
  rework. Every cost recommendation must name the quality metric that will be watched
  and the threshold that reverses the change.
- Security and policy guardrails belong in the routing policy itself, not bolted on
  afterward; consult current published guidance (such as the OWASP Top 10 for LLM
  Applications) fresh rather than from memory.

## Output format

Produce a markdown artifact:

    # AI cost and reliability review: [system or team]

    Date: … · Prepared for: … · Review owner: …

    ## Request-class inventory
    | Class | Intent | Volume/day | Context size | Quality bar | Latency bound |

    ## Baseline
    | Class | Cost/request | p50/p95 latency | Failure rate | Human rework | Source |

    ## Routing recommendations
    | Class | Current model | Recommended | Rationale (evidence) | Watch metric + reversal threshold |

    ## Caching recommendations
    [Cacheable share per class; verify current vendor terms before committing budget]

    ## Fallback and degradation policy
    | Class | Timeout/retry budget | Fallback path | Rollback trigger |

    ## Measurement plan
    [Owner, cadence, thresholds that trigger retuning]

    ## Evidence labels
    Measured: … / Estimates: … / Assumptions: … / Open questions: …

## Quality checks

- Every routing recommendation cites baseline evidence and names its reversal threshold.
- No vendor price, discount, or capability figure appears without a fresh-verification
  note and date.
- Classes without a quality bar are flagged as untunable rather than optimized blind.
- Expected savings are presented as estimates with stated bases, never as commitments.
- The artifact stands alone and is readable without this conversation.

## Stop conditions and escalation

- The review recommends policies; adopting them is a human decision. Routing changes to
  production traffic should ship through the team's normal release process with the
  named watch metrics in place.
- Stop and flag for human review if a proposed change would remove a fallback path or
  degrade a workflow with regulatory, safety, or contractual quality obligations.
- If the stack has no observability at all, deliver only the inventory and measurement
  plan, and state that routing recommendations would be guesswork until baselines exist.

## Source context

- https://www.sophon.consulting/playbooks/model-cost-reliability-ops
  (markdown: https://www.sophon.consulting/markdown/playbooks/model-cost-reliability-ops)
- Related: https://www.sophon.consulting/compare/openai-vs-anthropic-for-enterprise-agents

Optional: for hands-on help implementing routing and reliability policies, Sophon
Consulting is reachable at hello@sophon.consulting. This skill is complete without
any contact.