Building AI Agents in Production with Refonte Learning in 2026

Why Production Agents Are a Different Engineering Problem

Most teams in 2026 have already built an agent demo. The hard part — the part nobody warned them about — is keeping that agent useful, safe, and affordable once it serves real users. A demo agent answers a curated question on a developer's laptop. A production agent handles ambiguous input, calls live tools that occasionally fail, runs inside latency budgets, leaves an auditable trail, and costs money on every token. The skill set required to ship one is closer to distributed systems engineering than to prompt engineering.

This article is written for engineers who already understand what an agent is — a loop of LLM reasoning plus tool use plus memory — and who now need to put one in front of customers. We will skip the conceptual intro and go directly into the deployment surface: evaluation harnesses, guardrails, cost governance, observability, orchestration patterns, rollout strategy, and the operational disciplines that separate a demo from a service.

The reason production agents fail is rarely the model. It is almost always one of three things:

• The team has no way to measure whether a change made the agent better or worse.
• The agent has unbounded behavior — it can call any tool, spend any amount, or emit any output — because nobody put rails around it.
• The team cannot reproduce a customer-reported failure because nothing was logged at the right granularity.

Each of these is solvable, but only if you treat agents as software systems first and language models second. The architecture has more in common with a payment processor — idempotent operations, retries, audit logs, circuit breakers — than with a chatbot.

The economics have also shifted. In 2024 it was acceptable to burn tokens for novelty. In 2026, finance teams ask uncomfortable questions about per-conversation cost, and product teams ask why an agent's p95 latency tripled after a model upgrade. The engineers who can answer those questions with data are the ones whose agents stay in production.

A second economic shift: model providers now offer five or six capability tiers at very different price points, and routing between them has become a first-class engineering concern. A naive implementation that always calls the largest model will lose to a thoughtfully routed system that uses small models for classification, medium models for retrieval synthesis, and large models only when reasoning depth is actually required. We will return to routing later because it is one of the highest-leverage decisions in the stack.

Finally, agents in 2026 are no longer single-turn assistants. They orchestrate workflows that span minutes or hours, suspend and resume, call other agents, and write to systems of record. That changes everything about how you design state, recovery, and human-in-the-loop checkpoints. The rest of this piece walks through each of those concerns with concrete patterns we teach at Refonte Learning.

The Evaluation Layer Comes First, Not Last

If you take one thing from this article, take this: build the evaluation harness before you tune the agent. Teams that prompt-engineer first and evaluate later end up shipping regressions they cannot detect. Teams that build evals first move faster because every change is measurable.

A production evaluation suite for an agent has at least four layers. The first is unit evals on individual prompts and tool calls — given this input, did the model produce parseable JSON, choose the right tool, extract the right entity? These are cheap, fast, and run on every pull request. Frameworks like Promptfoo, DeepEval, and Inspect AI cover most of this ground, but rolling your own with pytest is fine for small teams.

The second layer is trajectory evals — given a multi-step task, did the agent reach a correct final state through a reasonable path? This is where most demo-quality agents fall apart. A trajectory eval needs a labeled dataset of tasks with acceptable end states, and ideally a judge model that scores intermediate steps. You will discover that an agent can produce the right answer through a wildly inefficient path, and that path costs you money. Trajectory evals expose that.

The third layer is regression evals against a held-out set of real production traces. Once your agent is live, every interesting failure becomes a new test case. We recommend a weekly cadence where on-call engineers add five to ten newly observed failure modes to the regression set. Within a quarter you have a dataset that reflects your actual user distribution, which no public benchmark will ever do.

The fourth layer is online evaluation — A/B tests, shadow traffic, and user feedback signals. Offline evals tell you whether a change is plausibly better; online evals tell you whether it actually is. The pattern we teach is: never promote a model or prompt change to 100% traffic without a 48-hour shadow run and a 10% canary, regardless of how good the offline numbers look.

A critical design point: use LLM-as-judge sparingly and with calibration. Judge models drift, agree with themselves more than with humans, and inherit the biases of the model they are evaluating. Calibrate every judge against a human-labeled gold set monthly, and never use the same model family as both the agent and its judge. If you teach a GPT-class model to grade GPT-class outputs, you are measuring family loyalty, not quality.

The operational discipline of evaluation maps closely to what good software teams already do for testing — see our deep dive on software engineering skills and roadmap for 2026 for the broader testing culture this builds on. Agent evaluation is testing, with stochastic outputs and fuzzier oracles, but the discipline is the same.

Guardrails: Input, Output, and Behavioral

Guardrails are the rails that keep your agent from doing something stupid, dangerous, or expensive. They sit at three layers and each has different failure modes.

Input guardrails screen what reaches the model. The cheapest version is a content classifier that rejects obvious prompt injection, jailbreak attempts, and personally identifiable information you do not want logged. The more sophisticated version uses small purpose-built models — Llama Guard, NeMo Guardrails, or Lakera's classifiers — to score inputs along multiple risk axes. Input guardrails should fail closed: if the classifier times out, reject the request rather than passing it through unscreened.

Output guardrails screen what reaches the user. This is where you catch hallucinated facts, leaked secrets, unsafe code, and outputs that violate brand or compliance constraints. A useful pattern is grounding-check guardrails: for any factual claim in an output, can the system point to a retrieved source that supports it? Claims without grounding get flagged or rewritten. This is more expensive than a simple classifier but dramatically reduces hallucination complaints.

Behavioral guardrails constrain what the agent can do. These are not LLM-based — they are deterministic policy code. Examples: an agent can call the refund tool but only for amounts under $100; an agent can read from the customer database but never write to it; an agent can make at most 12 tool calls per task before a human is paged. Behavioral guardrails are where you encode the actual safety contract with your business, and they belong in code that has unit tests, not in prompts that can be talked around.

A pattern we see repeatedly: teams put their entire safety story in the system prompt. This fails. System prompts are suggestions; tool-level authorization is enforcement. If you do not want the agent to delete user data, do not write "do not delete user data" in the prompt — write a tool wrapper that does not expose a delete endpoint to the agent at all. Capability restriction beats instruction every time.

Prompt injection deserves its own paragraph because it is the dominant attack surface in 2026. Any time your agent reads untrusted content — a webpage, an email, a PDF, a database field a user controlled — that content can contain instructions targeting your agent. Defenses include: marking untrusted content with explicit delimiters, running it through an instruction-stripping pass, never giving agents that read untrusted content the same tool permissions as agents that act on user intent, and structured logging so you can audit when an agent took an action that was not in the original user request.

The security mindset for agent systems overlaps significantly with broader detection and response practices — our guide to proactive monitoring tools in cybersecurity covers the telemetry side of this picture, and the same instincts apply when watching agent behavior.

Cost Engineering: The Discipline Nobody Teaches

An agent that costs $0.12 per conversation is fine. An agent that costs $4.80 per conversation is a business problem, and most production agents drift toward the latter without active engineering. Cost discipline is the single biggest differentiator we see between teams that scale agents and teams that quietly shelve them.

Start by instrumenting cost as a first-class metric per request, per user, per feature, and per model. If you cannot answer "what did the average conversation cost yesterday and what drove the variance" in under sixty seconds, you do not have cost observability. The fix is to emit a structured cost event from every model call and every tool call with non-trivial cost (vector search, embedding generation, external API fees) and aggregate them in the same observability backend as your latency and error metrics.

The next move is model routing. A well-designed agent uses three or four model tiers. Classification, intent detection, and simple extraction go to small fast models — Haiku-class, Gemini Flash, GPT-4o-mini, or a fine-tuned open model. Synthesis and tool selection go to mid-tier models. Hard reasoning, multi-step planning, and ambiguous resolution go to frontier models. A router — itself a small classifier — picks the tier per turn. Done well, routing cuts costs 60-80% with negligible quality loss. Done poorly, it cuts quality with negligible cost savings, which is why you need the evaluation harness in place first.

Caching is the next lever. Semantic caching of frequently-asked patterns, exact-match caching of tool-call results that do not change minute-to-minute, and prompt-prefix caching (which most providers now offer natively in 2026) all stack. A retrieval-heavy agent with no caching is leaving 30-50% of its bill on the table.

Context window discipline matters more than people think. The naive pattern of stuffing every prior turn, every retrieved document, and every tool result into the next call inflates costs quadratically as conversations grow. Teach the agent to summarize, prune, and reference rather than re-include. A rolling summary plus the last three turns plus top-k retrieved chunks usually beats a 60K-token transcript on both cost and quality.

Set budget circuit breakers at three levels: per-request (cap tokens and tool calls per turn), per-conversation (cap total spend on a single user session), and per-tenant or per-feature daily (cap aggregate spend so a runaway loop cannot bankrupt you overnight). Circuit breakers should degrade gracefully — fall back to a cheaper model, refuse the request with a clear message, or page a human — not crash. The number of horror stories about agents that looped overnight and burned five-figure bills is now large enough that this is non-negotiable.

Finally, measure cost-per-successful-outcome, not cost-per-token. A cheap agent that fails half the time and forces human follow-up is more expensive than an agent that costs 3x per call but resolves the task. Tie cost metrics to your eval metrics so you can have honest tradeoff conversations with product.

Observability: Traces, Spans, and the Replay Problem

You cannot operate what you cannot see. Agent observability has matured significantly in 2026, with tools like LangSmith, Langfuse, Arize Phoenix, Helicone, and Braintrust offering trace-level visibility into model calls, tool calls, and intermediate state. Pick one and instrument everything from day one.

The core data model is the trace: a single user interaction decomposed into spans for each model call, tool invocation, retrieval, and guardrail check. Each span carries inputs, outputs, latency, tokens, cost, and a status. A good trace lets an on-call engineer reproduce exactly what the agent saw and did, in order, with timing, without having to ask the user to repeat themselves.

The replay problem is specific to agents. Because models are stochastic and tools call live systems, you cannot deterministically re-run a failed trace and expect the same result. The workaround is to record tool-call inputs and outputs alongside the model spans, then offer a "replay with mocked tools" mode that lets you re-prompt the model against the same context to test fixes. Without this, debugging is guesswork.

Key metrics to dashboard:

• Task success rate, measured by your eval judge against production samples.
• p50 and p95 latency, broken down by model, tool, and feature.
• Cost per successful task.
• Tool error rates, especially for external APIs that go down without telling you.
• Guardrail trigger rates — sudden spikes mean either an attack, a regression, or a misconfigured rule.
• Loop and retry counts — agents that exceed expected step counts are almost always either confused or being manipulated.

Alerts should fire on rate-of-change, not absolute thresholds. A jailbreak attempt rate that doubles overnight matters even if the absolute number is small. A success rate that drops three percentage points in 24 hours matters even if the absolute rate looks fine.

The operational discipline here borrows directly from SRE practice. If you want the broader context on the deployment, monitoring, and incident-response side of modern infrastructure, our piece on DevOps engineering in 2026 covers how the same disciplines apply to AI-adjacent systems.

One practical note: trace storage gets expensive fast. Sample aggressively for happy-path traces (1-5% is usually fine) but keep 100% of traces that triggered guardrails, returned errors, hit budget caps, or received negative user feedback. Those are the ones you will actually look at.

Orchestration Patterns That Survive Production

Framework choice is less important than orchestration pattern. LangGraph, CrewAI, Autogen, LlamaIndex Workflows, Pydantic AI, and a half-dozen others all support the patterns that matter. What matters is choosing a pattern that fits your workload.

Single-agent with tools is the right starting point for most use cases. One model in a loop, a clear tool schema, a clear termination condition. Most teams that move to multi-agent architectures regret it because coordination overhead — token cost, latency, debuggability — exceeds the benefit. Only graduate to multi-agent when you have a genuinely decomposable workload and have hit a clear ceiling with a single agent.

Planner-executor splits planning from execution. A planner model produces a structured plan; a cheaper executor model carries out each step. This pattern is excellent for cost (planner runs once, executor runs many times on a cheap tier) and for auditability (the plan is a reviewable artifact). It struggles when plans need to adapt mid-execution, so include a replan trigger.

Hierarchical agents — a supervisor that delegates to specialist sub-agents — fit workloads where the specialists have genuinely different tool access or domain knowledge. The cost is communication overhead and a debugging surface that grows multiplicatively. Use sparingly.

Stateful long-running workflows are the 2026 frontier. Workflows that suspend for human approval, wait for an external event, or run for hours need durable execution. Temporal, Restate, Inngest, and the workflow primitives built into LangGraph all solve this. The non-negotiable requirement is idempotency: every step must be safely retriable because suspended workflows will resume after process restarts, deployments, and crashes.

Design your tool layer carefully. Tools should be typed, idempotent where possible, timeout-bounded, and observable. A tool that takes 30 seconds with no progress signal will be retried by a confused agent, blowing up cost and possibly causing duplicate side effects. Wrap every tool in a layer that enforces timeouts, emits structured logs, and surfaces failure modes the model can reason about ("the payment API returned a 503, retry in 10 seconds" beats a raw stack trace).

Memory deserves explicit design. Short-term memory (conversation buffer), working memory (current task state), and long-term memory (cross-session user knowledge) have different storage, retrieval, and privacy requirements. Mixing them in a single vector store is the most common architectural mistake we see. Use a relational store for structured state, a vector store for semantic retrieval, and a key-value store for short-term buffers. Each has different consistency, latency, and cost profiles.

Retrieval Is Still Where Most Quality Comes From

In 2026, with context windows over a million tokens, some teams have concluded retrieval is obsolete. They are wrong. Long context is expensive, slow, and often less accurate than well-tuned retrieval. The right mental model is that retrieval is now about precision and freshness, not just fitting documents into the prompt.

A production-quality retrieval stack typically combines:

• Hybrid search — dense embeddings plus BM25 or similar lexical scoring. Pure semantic search misses exact-match queries; pure lexical misses paraphrase. The combination beats either alone on most real datasets.
• Reranking with a cross-encoder on the top 30-50 candidates. This is the single highest-leverage retrieval upgrade most teams have not yet made.
• Query rewriting — using a small model to transform the user query into a retrieval-friendly form, generate hypothetical answers (HyDE), or decompose multi-hop questions into sub-queries.
• Metadata filtering — restricting retrieval to documents the user has permission to see, that are recent enough to matter, or that match structural criteria. Permission-aware retrieval is non-optional for B2B.
• Freshness handling — for fast-changing domains, time-decay scoring or explicit recency filters matter more than embedding quality.

Evaluate retrieval separately from agent quality. A standard retrieval evaluation reports recall@k, mean reciprocal rank, and nDCG against a labeled query-document set. If retrieval recall@10 is 60%, no amount of prompt engineering will save the agent — it cannot answer from documents it never saw. Diagnose retrieval before you diagnose generation.

Chunking strategy matters more than embedding model choice in our experience. Naive fixed-size chunking destroys document structure; semantic chunking and parent-document retrieval (chunk for retrieval, return parent for context) consistently outperform. Spend a day on chunking before you spend a week comparing embedding models.

The data engineering underneath retrieval — pipelines that ingest, normalize, chunk, embed, and refresh document stores — is where most production retrieval failures actually originate. A stale index, a silently failing embedding job, or a permission flag that did not propagate causes more incidents than model choice ever will. Treat the retrieval pipeline as a first-class data product with SLAs, monitoring, and on-call ownership — the discipline our data science and AI in 2026 coverage explores in more depth.

Deployment, Versioning, and Safe Rollout

An agent in production is the composition of at least four things that can change independently: the model, the prompts, the tool definitions, and the orchestration code. Each needs a version, and you need to be able to roll back any one of them without redeploying the others.

The pattern we teach is treating prompts and tool schemas as versioned artifacts, not strings in code. Store them in a registry (a database table or a tool like Langfuse or PromptLayer) with semantic versions, change history, and the ability to bind a specific version to a specific deployment. A prompt change should follow the same review-test-canary-promote lifecycle as a code change.

For models, do not pin to a provider's latest alias in production. Pin to a specific snapshot (gpt-4o-2024-08-06, claude-3-5-sonnet-20241022, and so on). Providers update behind aliases, and "the model got worse on Tuesday" is a real incident category. Pinning gives you control over when you evaluate the next snapshot.

Rollout strategy for agents looks like this:

1. Offline eval against your full regression suite. Reject changes that regress on any high-priority slice.
2. Shadow traffic for 24-72 hours. The new version processes live requests but its output is logged, not shown. Compare against the live version on cost, latency, and judge-scored quality.
3. Canary at 1-10% of traffic, with automatic rollback triggers on success rate, latency, cost, and guardrail trigger rate.
4. Gradual ramp to 100% over hours or days depending on traffic volume and confidence.
5. Post-deploy monitoring for at least 72 hours. Many agent regressions only surface on the long tail of user behavior.

Feature flags belong in the agent layer too. Wrap risky tool capabilities, model upgrades, and new prompt strategies behind flags you can flip in seconds without a deploy. When something goes wrong at 2am, you want a kill switch, not a redeploy.

The broader engineering practice of versioned artifacts, canary deploys, and reversibility is well-covered in AI developer engineering practices for 2026, and the patterns translate directly to agent systems.

Human-in-the-Loop and Escalation Design

Production agents that touch money, identity, or irreversible actions need humans in the loop by design, not as an afterthought. The question is not whether to escalate but when, to whom, and with what context.

Design three escalation modes:

• Soft escalation: the agent completes the task but flags it for async human review. Used for low-confidence completions, edge-case classifications, and anything the agent flagged but could not resolve.
• Hard escalation: the agent pauses and waits for human approval before acting. Used for high-value actions, irreversible changes, and policy-restricted operations.
• Handoff: the agent transfers the conversation to a human with full context. Used when the user explicitly asks, when guardrails fire repeatedly, or when the agent has looped without progress.

The interface for human reviewers matters as much as the agent itself. Reviewers need the full trace, the proposed action, the agent's reasoning (or a summary of it), one-click approve/reject/modify, and a feedback channel that flows back into the regression dataset. A clunky review interface kills throughput and creates an incentive to widen the agent's autonomy further than it should be.

Measure escalation rate as a first-class metric. Rising escalation rates on a stable workload usually mean model drift, retrieval degradation, or a new class of user request the agent was not trained for. Falling escalation rates on a stable workload sometimes mean the agent learned to suppress its own uncertainty, which is worse than escalating too often. Watch the trend, not the absolute number.

Confidence calibration is the underlying capability. The agent needs to know when it does not know. Techniques include self-consistency sampling (run the same prompt multiple times, escalate if outputs disagree), explicit confidence prompts (ask the model to rate its certainty, calibrate the ratings against outcomes), and grounding checks (escalate when the agent cannot point to a source). None of these are perfect; layer them.

Security, Compliance, and Data Governance

Agent systems concentrate risk: they read sensitive data, take actions on systems of record, and produce outputs that may be quoted as authoritative. The security posture needs to match.

Data handling: classify what the agent sees. PII, PHI, payment data, and trade secrets all have different handling requirements. Log redaction at the trace level is non-negotiable — your observability backend should never receive raw credit card numbers or unmasked health data. Most observability tools support redaction hooks; use them on day one, not after the breach.

Tenant isolation: in multi-tenant SaaS, an agent must never leak data between tenants. The dangerous failure mode is a retrieval bug that returns Tenant A's documents when Tenant B asks. Test for this explicitly with an isolation eval that asks an agent operating as Tenant B for information only Tenant A has, and verify the agent cannot retrieve or repeat it.

Audit logs: every agent action that modifies external state needs an audit log with user identity, agent version, prompt version, model snapshot, tool inputs, tool outputs, and timestamp. Regulators in finance, healthcare, and legal sectors increasingly require this, and your own incident response will require it long before regulators show up.

Model provider trust: understand your provider's data retention, training opt-out, and regional residency policies. Enterprise tiers of the major providers offer zero-retention modes; consumer tiers do not. The contract you have determines whether your prompts can be used for training, how long they are stored, and where.

Prompt injection deserves a security-team-level response, not just an engineering one. Threat-model your agent: what is the worst action an attacker who controls one of your input sources could cause? If the answer is "transfer funds" or "delete production data", you need defense in depth, not just an input classifier. Capability restriction, dual-control approvals for high-risk actions, and out-of-band confirmation are the real defenses.

Supply chain: agents pull in dependencies — frameworks, model SDKs, vector databases, tool integrations. Each is an attack surface. Pin versions, scan for vulnerabilities (Trivy, Snyk, GitHub Advanced Security), and maintain an SBOM. The agent-framework ecosystem is young; vulnerabilities are still being discovered regularly.

The Team and Skills That Ship Production Agents

The team that successfully runs agents in production looks different from the team that built the prototype. It needs:

• An AI engineer who owns the model layer: evals, prompts, model selection, routing, and quality metrics.
• A platform engineer who owns the runtime: orchestration framework, deployment pipeline, observability, and infrastructure cost.
• A data engineer who owns the retrieval and memory pipelines: ingestion, indexing, freshness, permissioning.
• A product owner who owns the success definition: what does "good" look like, what is the cost ceiling, what escalation rate is acceptable.
• A security or compliance partner for any system that touches regulated data or irreversible actions.

In small teams these roles compress into one or two people, but the responsibilities must all be owned somewhere. The single most common failure mode we see is no one owning eval quality — everyone assumes someone else will write the next regression test, and the dataset goes stale.

Skill-wise, the engineers who succeed in this space combine four things: solid software engineering fundamentals (typing, testing, observability, async patterns), distributed systems intuition (retries, idempotency, backpressure, partial failure), enough ML literacy to debug a retrieval pipeline or read an eval report, and product judgment about when the agent is good enough to ship. None of these is exotic, but the combination is rare, which is why teams that can hire for it move much faster than teams that cannot.

This is the gap Refonte Learning's AI Engineering program is designed to close. The curriculum walks engineers through training pipelines, LLMOps, evaluation harness design, production serving, and the observability stack — taught by practitioners who have shipped agents at scale, with hands-on internship projects that mirror the production patterns described in this article. We focus on the boring, durable skills: writing the eval before the prompt, instrumenting cost before scaling traffic, designing tools that fail safely. The flashy stuff changes every six months; the engineering discipline does not.

At Refonte Learning we also see engineers from adjacent disciplines — data engineers, backend engineers, SREs — convert into AI engineering roles faster than they expect, because the operational skills they already have transfer directly. If you can run a payment system, you can run an agent. The model layer is learnable; the systems thinking is the hard part, and you may already have it.

A Pragmatic Path to Production

If you are starting today, here is the order of operations we recommend, drawn from teams we have advised through this transition:

1. Pick one workflow with clear success criteria, bounded scope, and a defined escalation path. Avoid open-ended assistants for your first production agent.
2. Write the evals first. Twenty trajectory tests and fifty unit tests before any production traffic. This forces you to define success precisely.
3. Build the simplest possible single-agent loop with three to five tools. Resist multi-agent until you have hit a real ceiling.
4. Instrument cost and traces from request zero. Retrofitting observability is painful; building it in is free.
5. Add guardrails before you add features. Behavioral guardrails (capability restriction), then output guardrails (grounding checks), then input guardrails (injection screening).
6. Shadow-deploy for a week before any real user sees output. You will find things you cannot find offline.
7. Canary at 5% with automatic rollback on cost, latency, and success rate triggers.
8. Set up a weekly eval review where on-call engineers add new failure modes to the regression set. This is the flywheel.
9. Tune routing and caching once you have baseline cost and quality numbers. Not before — you will optimize the wrong thing.
10. Plan for the second agent to share infrastructure with the first. Every team eventually has multiple agents; build the platform now so the second one takes a week, not a quarter.

The teams that follow something like this sequence ship agents that survive contact with real users, scale gracefully, and do not generate 2am incidents. The teams that skip the evaluation and observability steps in favor of feature velocity ship demos that get quietly retired six months later when the cost or quality numbers come due.

Production AI agents in 2026 are an engineering discipline, not a prompt-craft hobby. The patterns are knowable, the tools are mature enough, and the operational playbook is converging. What separates the teams that succeed is not access to better models — everyone has the same model menu — but the discipline to treat agents as the distributed systems they actually are. Build the harness, set the rails, watch the meters, and ship something that earns the right to stay in production.

About Refonte Learning

Refonte Learning trains engineers and analysts for the AI-first economy through practitioner-led programs in AI engineering, data, cloud, DevOps, and software engineering. Our AI Engineering track covers the full production stack — from training pipelines and LLMOps to evaluation, serving, and observability — with internship projects that mirror the systems described in this article. Engineers who complete the program ship agents that survive production, not demos that survive a slide deck.