The Real AI Stack: Models, Context, Tools, and Evaluation

An AI product is often described by its model: which provider, which size, which benchmark, which context window.

That is like describing a reporting system by its database. The choice matters. It does not describe the product.

The useful stack begins with a job and ends with evidence that the job happened. Between those points sit context, retrieval, tools, state, controls, and evaluation. Most production failures occur in those layers.

1. The job

Before architecture, define the responsibility.

“Answer questions about our policies” is not enough. Which policies are authoritative? Who is asking? What decisions will the answer inform? Must the answer cite a source? What should happen when material conflicts? How current must the corpus be?

A useful job definition includes the user, trigger, expected result, constraints, and observable completion condition. It should make sense without the phrase “powered by AI.”

2. The experience

The interface shapes model behavior and user trust.

A blank chat box pushes every responsibility onto the user. A designed workflow can collect the right inputs, expose available actions, show evidence, constrain choices, preserve state, and make review efficient.

Uncertainty should be visible in the experience. Missing evidence, conflicting sources, unverified actions, and out-of-bound cases deserve distinct states. “Something went wrong” and a confident guess are both insufficient.

3. Context

The model only knows what arrives in the request or its trained parameters. Application context makes the behavior specific to this user, account, case, and moment.

Context may include:

current application state,
user role and permissions,
relevant records,
policy and procedural material,
prior actions,
tool results,
examples of expected output,
and constraints for this step.

More context is not automatically better. Irrelevant material increases cost and can distract the model. Context construction is a ranking and product decision.

4. Retrieval

Retrieval finds evidence from a larger body of material. It includes ingestion, parsing, metadata, indexing, permission filters, query construction, ranking, and deletion.

Measure retrieval separately. Build a set of real questions with expected sources. Track whether the authoritative passage appears before evaluating answer style. Otherwise the team may tune generation around a search failure.

5. Tools

Tools let the system act or gather information beyond the model request. A tool should be narrow, typed, permission-aware, and explicit about success.

Bad tool: manage_customer_account.

Better tools: get_customer_balance, list_open_cases, draft_credit_request, submit_credit_request_for_approval, verify_credit_status.

Narrow tools reduce ambiguity and create natural approval boundaries. They also make evaluation possible: the team can inspect whether the correct tool and arguments were selected.

6. State

Long work needs durable state outside the model’s conversational memory.

The system should know which step is active, what was attempted, which tool result was accepted, what requires approval, and whether the underlying business state changed. If the process crashes, it should resume or fail cleanly without repeating an irreversible action.

This is conventional application and workflow engineering. It becomes more important when a model selects the path dynamically.

7. The model

Choose the model after the job and test cases exist.

Compare candidates on the actual behavior: extraction quality, tool selection, synthesis, instruction following, latency, cost, and failure. A benchmark can help narrow the field but cannot represent the organization’s source material, tools, and standards.

Preserve a provider boundary where practical. That does not mean pretending every model API is identical. It means keeping business behavior and evaluation independent enough that a new candidate can be tested without rebuilding the product.

8. Controls

Controls constrain consequence:

permission checks inside tools,
allowed actions by workflow state,
approval before material or irreversible changes,
budgets for tokens, time, and tool calls,
source and data boundaries,
fallback behavior,
and immediate disable paths.

Do not rely on the prompt for a rule the application can enforce.

9. Evaluation

Evaluation turns quality into an engineering artifact.

Use representative cases with explicit criteria. Combine deterministic assertions, model-based graders where appropriate, and human review. Measure the full task and the layers that explain it.

| Layer | Example measure | | --- | --- | | Retrieval | Expected source appears in top results | | Generation | Material claims supported by context | | Tool use | Correct tool and valid arguments | | Workflow | Required steps and approvals respected | | Outcome | Underlying task completed correctly | | Operation | Cost and latency within budget |

Every production failure worth fixing should become a test or monitoring rule.

10. Operations

Models, prompts, sources, tools, and user behavior all change. Operation requires versions, traces, release criteria, alerts, replay, cost visibility, and an owner who can interpret failure.

The system should answer: What happened? Which inputs and versions were involved? What changed downstream? Is the case reproducible? Did a recent release affect similar tasks?

The trace is not the outcome. A trace explains the path. Outcome verification determines whether the job finished.

The architecture in one sentence

The model proposes or interprets; the application provides context and state; tools expose narrow capability; controls limit consequence; evaluation defines quality; operations keep the definition current.

That is the real AI stack.

The model creates the discontinuity. The surrounding system turns that discontinuity into something a customer or operation can depend on.

Trace one request through the stack

The layers become easier to understand when they are attached to a specific job. Imagine a finance team asking an internal system to prepare the first draft of a monthly portfolio review.

The job layer defines completion: collect governed measures, compare them with the prior period and plan, identify material changes, connect those changes to approved source context, draft commentary, and route open questions to the analyst. “Write a summary” is not a sufficient definition.

The experience layer shows the analyst what is ready, what remains uncertain, and where every material statement came from. It allows a number, explanation, or source to be challenged without discarding the entire draft.

The context layer supplies the portfolio, period, user role, reporting definitions, thresholds, prior decisions, and current workflow state. The retrieval layer finds relevant investment notes and source documents while enforcing access and freshness. Tools query the governed semantic model, perform deterministic calculations, and create review tasks. State records which steps completed and which inputs were used.

The model interprets narrative material, compares context, and drafts explanations. Controls prevent it from inventing governed numbers, publishing without review, or reading material outside the user’s permissions. Evaluation cases compare its proposed package with representative historical reviews. Operations capture latency, cost, tool failures, reviewer corrections, and the model or prompt version behind the output.

If the generated commentary is weak, this trace gives the team somewhere to look. The source note may not have been retrieved. A metric definition may be ambiguous. The model may have ignored a threshold. The prompt may be poor. The analyst may disagree with the expected interpretation. Without separable layers, every failure is blamed on “the AI,” and improvement becomes guesswork.

The stack is also an ownership map

Architecture diagrams often show services but omit responsibility. In a production AI system, each layer needs an owner or at least a clear response path.

Product owns the job, user experience, and acceptable tradeoffs. Domain operators own the meaning of inputs, the review method, and the consequences of errors. Data teams may own sources, contracts, lineage, and access. Engineering owns application behavior, tool contracts, state, reliability, and deployment. Security and legal shape boundaries appropriate to the information and action. Someone must own evaluation as a release process rather than a one-time report.

The exact organizational chart varies. The principle does not: a layer with no owner becomes an invisible dependency. Retrieval quality declines because no one watches the corpus. Costs rise because no one owns budgets. Review queues become unusable because the human handoff was considered a safety checkbox rather than a product surface.

During design, place an owner and an operating question beside every layer:

| Layer | Operating question | |---|---| | Job | Are we still solving a valuable problem? | | Experience | Can users understand and correct the system? | | Context and retrieval | Is the right, permitted information arriving? | | Tools and state | Did the underlying work actually complete? | | Model | Is this model still the best fit for the bounded behavior? | | Controls | Are consequential actions appropriately limited? | | Evaluation | Did the proposed change earn release? | | Operations | Can the team detect and recover from failure? |

This keeps the stack from becoming a technology shopping list. Its purpose is to make a human and technical system legible enough to build, operate, and improve.