Agentic AI in Production — Phase 1: The Front Door

Part 1 of a new series on turning agentic AI proofs-of-concept into production-grade enterprise systems. This series walks an enterprise reference architecture layer by layer. Phase 1 covers Layer 1 — Experience & Channels and Layer 2 — Identity, Gateway & Governance. The question: who is allowed to reach the agent, through which door, on whose authority, and what does that door enforce before any model or tool is touched?

The POC has one caller. Production has many kinds.

A typical agent POC has a single entry point: a notebook, a Streamlit page, a curl command. The caller is the developer, the API key sits in an environment variable, and the model is whichever one looked best in the demo.

Production breaks each of those assumptions:

• The caller may be an employee in a copilot, a partner’s system, a schedule, a Kafka event, or a customer on a phone.
• The identity may be a user, a service, an agent acting on behalf of a user, or a chain of all three.
• The model will change, the provider will throttle, and the budget is owned by someone who was not in the demo.
• One request can fan out into many model calls and tool calls, so a single unauthenticated hit can cost real money or trigger a real action.

None of this is “model quality”. It is the front door, and in agentic systems it is where a large share of security, cost and accountability incidents begin. A chatbot that answers wrongly is a quality problem. An agent that acts wrongly, under the wrong identity, with no record of who authorized it, is an incident.

The forces that shape every layer

The RAG series used four forces: scalability, resilience, cost and latency, then added quality and security. Agentic systems add a seventh: accountability. Every action needs an answer to “who caused this, under whose authority, with what evidence, and can we prove it later?”

How this series is organized

The reference architecture has seven layers plus a set of cross-cutting concerns that run through all of them. Each phase of this series takes one or two layers and, for each, asks what the cross-cutting concerns look like at that layer rather than treating them as a separate chapter at the end.

The cross-cutting concerns we track at every layer

Layer 1: Experience & Channels

The architecture lists four channel types: Chat & Voice, Apps & APIs, Event Triggers and External Users. They look like UI variants. They are really different operating environments with different failure modes.

Channel comparison

Decisions that look small and are not

1. Streaming protocol. Server-sent events are simpler and pass through most proxies and CDNs. WebSockets support bidirectional flows (voice, interruption) but are harder to scale, authenticate per message and load balance. Whichever you pick, check the entire path (CDN, WAF, API gateway, ingress, service mesh) for response buffering and idle or absolute timeouts. Streaming that works on a laptop and dies at a 30-second gateway limit is one of the most common POC-to-production surprises.

2. Async by default for long-running work. An agent that plans, calls tools and waits for approval is not a single HTTP request. Model it as submit → job id → status / stream / webhook. Synchronous chat is the special case.

3. Event triggers need agent-safe semantics. Queue-driven agents inherit at-least-once delivery. Require idempotency keys on every side-effecting tool call, cap the maximum number of agent steps, and route repeated failures to a dead-letter queue with the full trace attached. An event-driven agent with no step cap is an unbounded cost generator.

4. Identity must travel with the channel. A chat UI that authenticates the user and then calls the backend with a shared service credential has silently discarded the user’s identity. Everything downstream (data access control, tool authorization, audit) then runs against the wrong principal. Layer 1 must hand a verifiable identity to Layer 2.

5. Not every channel deserves every capability. A voice channel with speech recognition errors and weak caller authentication should not be able to approve a payment, even if the same agent can do so from an authenticated web app. Define per-channel action ceilings.

6. External users change the threat model. Plan per-tenant quotas, abuse detection and bot protection from day one. External traffic should not share a rate-limit pool with internal users.

Cross-cutting lens: Layer 1

Cloud options for Layer 1

Where custom code is usually needed: the channel adapter. Managed bot frameworks are strong for scripted or intent-driven flows but constrain agentic, streaming, tool-using interactions. Most teams end up with a thin custom adapter that normalizes web chat, Teams/Slack, voice and events into one internal request envelope: identity, tenant, channel, correlation id, idempotency key, consent flags. It is a small amount of code that pays for itself as soon as the second channel is added.

Layer 2: Identity, Gateway & Governance

The architecture places five components here: Identity & Access, API Gateway, AI Gateway, Guardrails and Approvals. Together they answer: who is this, how much may they use, which model handles it, what is allowed, and does a human need to say yes?

2a. Identity & Access — the hardest problem in agentic AI

Classic applications have two identity questions: who is the user, and what may they do. Agents add a third: what may this agent do, and on whose behalf?

A useful rule: the effective permission of an agent action should be the intersection of what the agent is allowed to do and what the user is allowed to do, never the union.

Further decisions:

• RBAC vs ABAC. Roles are easy to reason about but multiply with per-tenant, per-document and per-context rules. Agent scenarios (“finance user, managed device, own accounts only, under a threshold”) push toward attribute- or policy-based decisions. A policy engine (OPA, Cedar-based services, cloud-native equivalents) keeps rules out of prompts and out of scattered if statements.
• Token passthrough is an anti-pattern. Do not forward the inbound user token to arbitrary tools or MCP servers. Exchange it for a narrowly scoped, audience-bound token per downstream service.
• Long-running agents outlive tokens. Design for refresh, re-consent or re-authorization at approval points rather than storing a long-lived credential.
• Secrets custody. Provider API keys and tool credentials belong in a vault behind the gateway, never in client code, prompts or agent memory.

Cloud options (identity):

Honest assessment: agent-to-tool delegated identity is the least standardized area in the whole architecture. Cloud-native services cover authentication well. Expect to build glue: a token-exchange service or gateway plugin, a per-tool scope map, and audit enrichment that records user, agent, tool, decision. Delegation semantics usually need custom design.

Cross-cutting lens: Identity & Access

2b. API Gateway vs AI Gateway — related, not interchangeable

Teams often ask whether their existing API gateway can also be the AI gateway. Sometimes yes, often partly. They protect different things.

The AI gateway in one line: the single choke point where model access is authenticated, metered, routed, protected and observed, so applications never talk to model providers directly.

Why it matters: without it, every team hardcodes a provider, holds its own key, implements its own retries (multiplying load during an outage), and reports cost differently. Swapping a model then becomes a migration project instead of a configuration change.

Options for the gateway layer

Product capabilities in this area change every quarter. Verify current features, streaming behavior, timeouts and limits against vendor documentation before committing.

When to go custom (and when not to)

Choose managed / cloud-native when one cloud and a small number of model providers cover your needs, governance already runs through that cloud’s API management, and the team has no capacity to operate another critical-path service.

Choose open source / self-hosted when multi-cloud or multi-provider routing is a real requirement (not a hedge), data must stay inside a controlled network boundary, or you need routing or budget logic managed products cannot express.

Go custom only for the parts that are truly yours:

1. Business-aware quota and chargeback: budgets by cost centre, product or tenant tier, tied to your billing model.
2. Delegated identity and token exchange for agent-to-tool calls.
3. Routing rules tied to business context: for example, “regulated-data requests may only use models in region X”.
4. The channel-to-gateway envelope: correlation ids, idempotency keys, tenant context.
5. Approval workflows integrated with internal systems.

A healthy common pattern is a managed or open-source gateway as the data plane, plus a small custom control plane (policy, budgets, identity mapping) that configures it. Avoid writing a full LLM proxy from scratch unless you have a specific reason: streaming, token accounting, provider quirks and failover are each harder than they look.

The gateway is now critical infrastructure

Everything routes through it, so the trade-offs are sharp:

• Latency: every hop adds time. Budget for it explicitly, and keep heavy checks (deep guardrail scans, semantic-cache lookups) off the path when a risk-based route can skip them.
• Single point of failure: run multi-instance and multi-zone at minimum. Decide each failure mode up front: for key custody and authorization the answer is usually fail closed; for optional analytics it is degrade and alert.
• Blast radius of a bad configuration: one wrong routing rule can send regulated data to the wrong model. Treat gateway configuration as code: reviewed, versioned, canary-deployed.
• Streaming token accounting: cost and quota logic must handle partial streams, client disconnects and retries without double-counting.
• Semantic cache versus authorization: a cache lookup must sit behind identity and tenant scope, or be keyed by it. Otherwise a cache hit can serve one tenant’s answer to another and the authorization check never runs.
• Retry ownership: if the client SDK, the gateway and the orchestrator each retry, one provider blip becomes a request storm. Decide who retries, and only for transient errors.

Cross-cutting lens: API Gateway & AI Gateway

2c. Guardrails at the gateway — what belongs here and what does not

The gateway is a natural place for coarse, cheap, uniform controls: input size and rate limits, PII detection on prompts leaving the trust boundary, known-pattern injection screening, output secret/PII scanning, and content policy. Options include provider-native safety services (Azure Content Safety and Prompt Shields, Bedrock Guardrails, Google Model Armor) and open-source tooling (NeMo Guardrails, Llama Guard, LLM Guard, Presidio).

It is the wrong place for controls that need application context: document-level authorization, groundedness against retrieved evidence, and business-rule validation. Those belong closer to retrieval, tools and output handling. Central guardrails give consistency, not completeness. A gateway that “handles safety” can give downstream teams false confidence that they can skip their own controls.

Cross-cutting lens: Guardrails

2d. Approvals — human-in-the-loop as an architecture component

Approvals are more than a UI button. They interrupt an agent mid-plan and resume it later, potentially hours or days later.

Design decisions:

• What triggers approval? Risk-based rules: action type, monetary threshold, data sensitivity, first-time actions, low confidence. Requiring approval for everything trains humans to click “yes” without reading.
• What does the approver see? The proposed action, its parameters, the evidence and reasoning, the requester and the consequences, not a raw model transcript.
• Who may approve? Separation of duties: the requester must not approve their own high-risk action.
• What happens on timeout? The default should be do nothing and notify, not auto-approve.
• State durability. The agent’s state must survive restarts while it waits, which pushes toward durable workflow engines instead of in-memory loops.
• Approver channel. Approving from chat is convenient but weaker evidence than a step-up authenticated approval.

Options: cloud workflow services (AWS Step Functions, Azure Durable Functions / Logic Apps, Google Workflows), durable-execution engines (Temporal), agent-framework interrupts (for example LangGraph), and ITSM or chat-based approvals (ServiceNow, Teams, Slack). The approval mechanism is easy to buy. The approval policy, meaning what needs a human and who decides, is always custom and is a governance conversation before it is a technical one.

Cross-cutting lens: Approvals

Cross-cutting summary matrix (Layers 1–2)

What breaks when you scale the front door

Technical debt you may be accepting — and whether it is worth it

The decision framework: seven questions before you greenlight Layers 1–2

1. Whose authority is every downstream action executed under? If the answer is “the agent’s service account” for anything user-specific, you have a confused-deputy problem.
2. Can any client reach a model or tool without passing the gateway? Direct provider access defeats every control described here. Block it at the network and credential level.
3. Can you attribute every token to a team, user and feature, and cap it? If not, a runaway loop is a budget incident waiting to happen.
4. What happens when the primary model or provider is unavailable or throttled? Is the fallback compatible in capability, region and data policy, or does it silently violate a compliance rule?
5. Which controls run at the gateway, and which must run closer to the data? Make sure central guardrails are not standing in for authorization in retrieval and tools.
6. What may the agent do without a human, and who decides? Approval thresholds need a named owner and a review cadence.
7. Is the gateway operated like critical infrastructure? HA, configuration review, canary deployment, capacity plan, on-call, and a defined failure mode for each dependency.

Choosing an approach

The consistent lesson: buy the data plane where you can, build the control plane where your business rules live, and never let the front door be the only place security is enforced.

The end-to-end picture

The diagram below follows one request through Layers 1–2: from any channel, through edge protection and the channel adapter into a single request envelope, then through authentication, authorization, input guardrails, the AI gateway and model providers, back through output guardrails, and finally to either automatic release or a human approval before it is allowed to reach agent orchestration. The bottom band shows how the eight cross-cutting concerns all hang off the same correlation context created at the edge.

The next phase moves one layer deeper into the architecture, to the point where a request stops being a call and becomes a plan: agent orchestration, routing, workflows and state.

Comments