AI agents on premise can work, but the model is the easy part. A working agent also needs tool calling, retrieval, memory, confidence gates, and an audit trail — and those are where self-hosted deployments succeed or fail. Here is what works today, component by component.

Pulling a 70B open-weight model onto a pair of GPUs and getting fluent text back is a weekend project. Getting that same model to reliably emit a valid tool call on the fourth turn of a conversation, against a schema it has never seen, while a compliance reviewer needs every step logged and reproducible, is the actual project. That gap is where most on-premise AI agents plans stall, usually after the demo and before the pilot.

This is written for architects and platform engineers scoping a controlled-environment deployment, not for anyone deciding whether on-prem beats cloud on principle. Assume that decision — air-gapped AI deployment, data residency, a contractual boundary — is already made. What's left is engineering.

Documentation on this exact problem is thin for a structural reason: nearly all agent tooling was built and tested against a frontier API endpoint. Function-calling accuracy, context handling, and failure recovery all look different once you swap in a self-hosted model behind vLLM or Ollama, running open weights on hardware you own. Almost nobody writing about agents runs one this way, so the failure surface stays undocumented.

What an On-Premise AI Agent Needs Beyond the Model

Say "the agent" and most people mean the model — the chat window, the demo, the model card. A production agent is a loop. Input arrives. The agent retrieves grounding context, decides what to do with it, calls a tool, reads the result, and decides again — answer, or call another tool — until it stops.

Six things happen in that loop, and only one is the model: a retriever that pulls the right context; a tool registry that knows what functions exist and what valid calls look like; a state store holding conversation memory across turns; a policy layer deciding whether an output is allowed to reach the customer; a log writer recording what happened in enough detail to reconstruct later; and the model, doing the reasoning step in between. Those core capabilities also depend on robust data pipelines for real-time processing across the ai infrastructure.

A cloud vendor hides five of these behind one API call. That's a reasonable trade for many use cases, and it's also why "agent" and "model" get used interchangeably — the vendor made five-sixths of the system invisible. On-prem, that convenience disappears, but so does the opacity: the system runs within an organization's physical environment, often in the organization's local data centers, giving teams full control, tighter data control, and stronger data privacy and regulatory compliance by keeping sensitive data and private data local. You own all six pieces, which means building or selecting all six on your ai platform — and it means nothing is hidden when a compliance reviewer asks how a specific answer was produced or inspects data handling. That's the actual argument for self-hosted AI agents in regulated industries, which often require on-prem AI deployments for compliance: not that it's easier, but that nothing is invisible, and that clarity is what makes mature ai capabilities possible.

This is the structural reason Clarity built its AI Agent Hub as separate components — models, tools and MCP, confidence gates, retrieval — rather than one opaque service. Separation is what lets you place some components inside a network boundary and others outside it, deliberately, instead of accepting an all-or-nothing deployment.

Component Map: What's Self-Hostable Today

Inference serving is a concurrency decision, not a single choice. A modern GPU such as an NVIDIA A100 is a common baseline for serious on-prem agent workloads. Ollama is good for a laptop or a small internal tool with a handful of simultaneous users; it isn't built for a production support queue's request pattern. vLLM and SGLang exist for that case specifically — continuous batching keeps the GPU fed as requests arrive and finish at different times, and paged KV-cache management avoids the memory fragmentation of naively pre-allocating a fixed cache per request. llama.cpp sits in its own lane: CPU inference and quantized models on constrained or air-gapped hardware. A demo on Ollama with one user in the room tells you nothing about fifty concurrent tickets.

Embeddings are the component people forget to threat-model. If retrieval calls a third-party embedding API while the rest of the stack stays inside your network, you've built an air-gapped-looking system with a hole in it — the text you're protecting gets sent out to generate the vectors representing it. Self-hosted embeddings matter for the same reason a self-hosted LLM does: the input text is the asset, and the retrieval stack should stay inside your boundary all the way through vector databases.

Tool Calling and MCP on Open-Weight Models: The Real State of It

Can open-weight models call tools reliably? It depends which tool-calling problem you mean, and on-prem teams tend to ask the easy version, especially when autonomous AI agents are software entities powered by machine learning models and expected to use tools correctly.

The most-cited benchmark, the Berkeley Function-Calling Leaderboard, scores models on AST accuracy, execution accuracy, and relevance detection across simple, multi-tool, parallel, and multi-turn categories. On single-turn, schema-fill tasks, open-weight models are competitive with frontier models. But the leaderboard's own maintainers reweighted the overall score so multi-turn and agentic tasks carry most of the weight, precisely because single-turn function-calling had become saturated. A documented real-world test drove a top-scoring open-weight fine-tune through an actual multi-step workflow against a task-management API, and it struggled with things the benchmark doesn't measure: resolving a relative date, looking up an ID on its own initiative, tracking which task it was supposed to be modifying across several turns. "Close" on a four-step task is a different reliability profile than "close" on one function call, and the distance compounds with every added turn.

Three places a tool call breaks, and they don't share a fix. Wrong tool selected: five options available, the model picks the wrong one, or a plausible one when it should have asked a clarifying question. Wrong arguments: the right tool called with a malformed payload — a hallucinated customer ID, a bad date format, a missing required field. Malformed output: the response isn't even valid JSON.

The third failure is close to solved. Constrained decoding enforces a grammar on the token stream so the model can only produce tokens on a valid path through a schema. vLLM exposes this as guided decoding (xgrammar as the default backend in its newer engine); SGLang implements the same idea. Set a JSON schema or regex as a decoding constraint and the model is structurally prevented from emitting invalid syntax. The cost is latency, worth paying deliberately rather than skipping because a demo felt fast enough without it.

Wrong tool and wrong arguments are reasoning problems, not decoding problems, and they get mitigated by design: fewer tools exposed per decision point rather than the full registry every turn; enums instead of free text wherever a parameter has a bounded set of values, since constrained decoding enforces those cleanly; tool descriptions written for the model's reasoning step, not as internal code comments; no overloaded dispatch tools that push selection into an argument-filling problem; and structured errors fed back into context with a capped retry budget, so a model told what was wrong with its last call can often fix it, without burning tokens for two minutes on an uncapped loop. In practice, that is the job of the orchestration layer when coordinating multiple agents on complex tasks, selecting relevant context and keeping autonomous agents from drifting.

Parallel tool calls carry a separate risk: constrained decoding validates that each call is individually well-formed, not that the set together is sound. Two individually valid write-scoped calls firing in parallel can still be jointly wrong, and no grammar constraint catches that — it needs an application-level check, and more conservative use of parallel calls the more a tool can write rather than read.

The Model Context Protocol standardizes the interface between a model-driven client and a tool server, so an agent doesn't need bespoke glue per integration. A local stdio server runs as a subprocess with no network hop — fine when a tool only needs resources already reachable from that host. Streamable HTTP is the transport for a self-hosted MCP server sitting behind an internal gateway in front of a database or ticketing backend, authorized through OAuth 2.1 against whatever identity provider you already run. That matters even more when on-premise orchestration coordinates multiple autonomous agents for jobs like triaging incidents or optimizing supply chains.

Support for MCP across serving stacks is uneven, and the protocol doesn't mandate that a client validate tool metadata before acting on it. Documented evaluation of major MCP clients found most did not implement static validation of server-provided tool descriptions — which matters because a compromised tool server can embed instructions inside its own metadata, and a client that doesn't inspect it hands that straight to the model as trusted context. That's the entry point for tool-result prompt injection: documented attack patterns include tool poisoning, confused-deputy exploitation of an agent's existing access, and real CVEs where injection through an MCP integration led to arbitrary command execution. This is not solved the way malformed JSON is.

What we tell teams to do: cap tool-loop depth so an agent that hasn't converged in N calls escalates instead of retrying; log every call and result, not just the final answer; treat any tool result as untrusted input, not the same trust class as the system prompt; add state management and error recovery because effective orchestration depends on both; and never let a write-scoped tool run without a gate in front of it. This is the specific job Clarity's AI Agent Hub separates out — tools and MCP sit behind the model as a discrete layer, paired with AI Safety Guardrails for the write-scoped cases needing a checkpoint before execution, which is where teams usually operationalize workflow automation for intelligent agents.

Confidence Gates and Handover: Why On-Prem Agents Need Tighter Thresholds

Smaller self-hosted models degrade faster across multi-turn tool loops, retrieval carries more of the grounding work with less parametric knowledge underneath it, and there's no vendor safety net absorbing bad outputs before they reach anyone. Confidence thresholds on a self-hosted deployment need to sit tighter than the cloud equivalent — a direct response to a worse starting error rate and the absence of anyone else's guardrail.

"Confidence" isn't one signal; it's at least five, and they don't always agree. Model self-reported probability is the weakest — models are poorly calibrated, especially smaller open-weight ones reasoning across several tool calls, and it should never be the sole gating signal. Retrieval grounding score checks whether a real source supports the answer, which is closer to ground truth than the model's opinion of itself. Schema and policy validation checks the mechanics: does the call parse, do arguments fall inside allowed bounds, does the action match a permitted policy. A separate verifier pass runs a second model or rules-based check over the primary answer. Intent-level historical resolution rate is the empirical signal — how often has this category of request actually resolved correctly through the autonomous path before.

Threshold setting is empirical, not a one-time design decision. Start conservative, run the gate in shadow mode against volume already handled by humans, and compare its decisions against what a person actually did. Two error rates matter and pull in opposite directions: false autonomous resolutions, where the gate let something close that shouldn't have, and unnecessary escalations, where it handed off something safely resolvable. Move the threshold based on which rate the data shows you're paying for, and keep moving it as volume accumulates.

When the gate fires, the handover matters as much as the decision. A bad handover drops the customer into a fresh queue with no context. A good one carries the full transcript, the retrieved sources, every tool call attempted and what it returned, and the gate score with the specific reason it fired. This is the deflection-with-handover pattern behind Clarity's AI Support Agent, enforced by the confidence gates inside the AI Agent Hub: the agent resolves 60–80% of inquiries autonomously in practice, and the rest is exactly the population the gate exists to catch. The human isn't a fallback bolted on after the fact — they're a defined stage the gate routes into deliberately, which is what a human-in-the-loop AI agent design actually means.

Retrieval and Per-Tenant Isolation in a Self-Hosted Vector Store

A frontier API model has enough parametric knowledge baked into its weights to often answer correctly with weak retrieval. A self-hosted model in the 7B–70B range most on-prem teams actually run has less to fall back on — when retrieval hands it the wrong passage, it's more likely to guess wrong. Retrieval quality drives answer quality more directly on-prem, because there's less of a safety net underneath it. This is what Clarity's AI Knowledge Agent is built around: grounding responses in the customer's own knowledge base rather than what the model happened to memorize. That retrieval stack matters especially in finance and healthcare, where on-prem AI is often required for data sovereignty.

Four parts of the retrieval path each fail in a specific, diagnosable way if skipped. Chunking strategy has to match document type — a support macro and a 40-page policy PDF don't chunk the same way. The embedding model needs to be version-pinned: embeddings are only comparable to others from the same model version, and swapping versions makes every existing vector incomparable to new queries with no obvious error, just quietly worse retrieval. In a typical rag pipeline, vector databases act as the storage and indexing layer for embedded documents before retrieval. Hybrid lexical-plus-vector search catches what pure vector similarity misses while pulling relevant context from private data — ask for "error E4021" and a vector-only retriever returns semantically related passages about errors in general rather than the one document containing that literal string; a lexical index (BM25 or equivalent) catches the exact match. Reranking retrieves a wider candidate set and reorders it by relevance before it reaches the model's context, a small compute cost that matters more on a smaller model with less room to compensate.

If more than one tenant shares the infrastructure, isolation stops being a nice-to-have. Three levels exist, in increasing strength and cost: logical partitioning within a shared collection, distinguished by a payload field and filtered at query time (Qdrant's payload-based multitenancy, Milvus's partition key); a separate collection per tenant, full physical separation inside the same cluster, which Milvus recommends for tenants needing role-based access control; and a separate instance per tenant, dedicated infrastructure with nothing in the request path one tenant's query could accidentally traverse into another's data. In practice, teams also need controls for sensitive information, clear data lineage, and a data fabric that supports seamless integration across retrieval services; the same on-prem pattern is common in manufacturing for real-time equipment monitoring.

The leaks that actually happen: a tenant filter applied at query time rather than enforced at the index, so any code path that forgets to attach it returns cross-tenant results silently; a shared reranker cache whose key doesn't include tenant identity, surfacing one tenant's cached rerank to another; and an evaluation set built from one tenant's data used to tune retrieval for the whole system, quietly optimizing everyone else's behavior against a corpus they never contributed to.

GPU Sizing for LLM Inference: What It Actually Costs

VRAM for serving a model breaks into three pieces, and only one is fixed by model choice. Weight memory is parameters times bytes per parameter at your quantization — a 70B model needs roughly 140GB in FP16, dropping to roughly 40GB near INT4. An 8B model needs roughly 18GB in BF16, about 8GB at INT4, and local execution can also deliver reduced latency for time-sensitive operations.

KV cache is the piece that actually varies with usage, scaling with concurrent sessions times context length, not parameter count. For a 70B-class model with grouped-query attention, that's roughly 0.33MB per token at BF16. At 4,096 tokens and one user, that's a little over a gigabyte; at 32,768 tokens and eight concurrent users, it's on the order of 86GB — larger than the model's own weights. At 128K context with eight users, KV cache alone can run to more than twice the size of the weights.

The point people size wrong: for agent workloads, concurrency and context length usually decide the hardware, not parameter count. An agent turn isn't a short prompt — it's a system prompt, several tool schemas, one or more tool results (often a large JSON payload), and running conversation history, all in context before a single token comes out. That's a long-context workload on every turn, and KV cache at that length times realistic concurrency dwarfs the weight-memory gap between a 30B and a 70B model.

Quantization is the lever everyone reaches for to shrink weight memory, and it costs something specific. FP8 and INT8 typically hold up well with minimal measurable quality loss. Aggressive 4-bit quantization saves more memory, but general benchmark scores don't tell you whether that saved memory came at the cost of tool-call reliability specifically — a model slightly fuzzier on word choice in chat is a minor issue; one slightly fuzzier on a write-scoped tool argument is a reliability problem. Test tool-call accuracy directly at your target quantization level rather than trusting a general benchmark score to cover it.

Published comparisons put the self-hosted-versus-API crossover somewhere around one to a few million tokens of steady daily traffic, with the zone near one to two million a genuine toss-up. But the number that actually determines which side of that line you land on is utilization, not the GPU's list price. High, steady utilization can land self-hosted inference at a small fraction of frontier API pricing per million tokens; low, spiky utilization can cost several times more than the naive calculation suggests, because idle GPU-hours are still paid hours. On-premise deployments also tend to require a high initial capital expenditure for specialized hardware and maintenance, even if they can make long-term costs more predictable. Run the math against your real traffic pattern before treating either side of the crossover as settled.

Scaling on-prem remains bounded by physical hardware limits, so adding capacity usually means more servers, more configuration work, and more gpu resources.

That also means carrying the operational load yourself: infrastructure management and hardware maintenance, the specialized talent for security and compliance, and the upside of independence from cloud-provider pricing changes.

Observability: Logs, Traces, and an Immutable Audit Trail

Three artifacts serve different purposes, and treating them as one is how audits go badly. Application logs are for debugging — free-form, high-volume, not built for anyone outside engineering to read. Distributed traces are for latency and loop analysis: a span per model call, per retrieval, per tool invocation, with token counts and the gate's decision recorded as attributes — useful for "where did the four extra seconds go," not for compliance. For on-prem systems, strong access controls and logging are baseline requirements, and that includes preserving audit logs separately from operational telemetry.

An immutable audit record is a different artifact with different requirements: append-only, retained on a defined schedule, tamper-evident, exportable, complete enough that a reviewer can reconstruct why the system produced a specific answer on a specific date, and supported by model provenance so AI supply chains remain reviewable under AI security controls. A trace built for debugging speed and a record built for evidentiary integrity are not the same file with different labels — conflating them is how a team discovers mid-audit that what they have is thorough but not tamper-evident, or retained but not exportable in a usable form with proper version control.

A compliance reviewer's questions are specific: what data went into this decision, what the model was shown, who or what verified the result, and whether the record has changed since it was written. This is the shape of Clarity's AI Quality Agent: scoring 100% of conversations against the customer's own rubric, not a sample, with audit-trail exports built in rather than assembled after a request comes in. Those records support governance in regulated industries, including public sector and critical infrastructure environments, where compliance with HIPAA and GDPR depends on durable audit trails and clear performance metrics. It sits on top of Clarity's SOC 2, HIPAA, ISO 27001, GDPR, and PDPL certification set — a documented, reviewable record of what the system did and why, not a claim of safety.

A Worked Example: Customer Service, End to End, Inside the Boundary

Put the six components together and a support interaction looks like this. A customer message arrives through chat, WhatsApp, email, or voice, entering through Clarity's Omnichannel Inbox. Keeping the workflow inside the boundary improves privacy and operational control through on-prem orchestration. It's classified by intent and language before anything else happens, so the right knowledge namespace and tool set get selected. The AI Knowledge Agent queries the customer's own knowledge base, scoped to their tenant partition, returning candidate sources alongside the passages retrieved. The agent calls a self-hosted MCP server sitting in front of whatever backend holds the real account data — a billing system, an outage tracker, a CRM — which supports seamless integration and can be customized to the workload instead of relying on a generic external service. The proposed action is checked against its schema before execution. The confidence gate inside the AI Agent Hub weighs the retrieval grounding score, the schema/policy check, and the intent's historical resolution rate together. If the gate clears, the agent answers and closes the conversation; if not, it hands off with the full transcript, sources, tool calls, and the specific reason the gate fired. After close, the AI Quality Agent scores the conversation against the customer's own rubric and writes the score to the audit record either way, which also preserves business continuity during internet outages because the core agent loop runs locally.

This is close to what happened at Saudi Electricity Company: 40% of power outage inquiries were resolved end to end by AI within four months, with no added headcount. That outcome works because outage inquiries are lookups against a real system of record with a bounded intent — a tool call that either confirms an outage or doesn't. It isn't an agent reasoning through an ambiguous request; it's the retrieval-through-gate steps running cleanly because the workload matches what they were built for, which is why teams use this pattern for workflow automation around sensitive data, stronger data security, deployment in a private cloud, and total operational control over model behavior in sensitive service workflows.

Failure Modes to Design For

Six components running well individually still fail in specific, recurring ways together, and none announces itself before it happens. Silent tool-schema drift: an internal API changes and the agent keeps calling it with the old argument shape, producing generic-looking errors instead of an obvious schema mismatch — validate the live schema against the tool registry on a schedule, not just at deployment. Retrieval poisoning: a stale knowledge base article ranks well and the agent repeats a wrong answer confidently — track which sources get cited most and put a review cadence on the knowledge base itself. KV cache exhaustion under burst: queue timeouts rather than a graceful slowdown, because cache headroom was sized for average rather than peak load — size against burst concurrency and alert on queue depth before it becomes a timeout. Model swap regressions: an upgraded model scores better on general benchmarks but starts failing tool calls it used to handle, because output formatting changed in a way the parser wasn't built for — re-run tool-call accuracy tests specifically before promoting any swap. Orphaned handovers: the gate fires and no one picks the conversation up, because no queue-level ownership or SLA is attached — treat a handover as a tracked queue item with an owner and a time bound. Unbounded tool loops: the agent keeps calling tools, burning context, and loses track of the original request — cap loop depth explicitly and force escalation once the cap is hit.

Every component in this trace is self-hostable today. The difficulty concentrates in the tool layer and the gate, because those are the two places a wrong call or a wrong threshold reaches a customer directly rather than just costing latency. Running ai agents on premise doesn't fail because the model can't run inside the boundary — it fails when a team scopes the project as running one model, when it's actually six components, each with its own failure modes and maintenance burden, especially when open source frameworks add more integration work for engineering teams. Cloud deployment is often the better fit for rapidly fluctuating workloads or for teams that do not want to operate hardware. The question worth asking isn't whether it can run inside your network. It's whether the team taking it on is prepared to own six components instead of one, since deploying intelligent systems on-premise increases responsibility for infrastructure, security, and operations, while cloud architectures make it easier to provision computing resources and integrate managed services. In that comparison, cloud based solutions and cloud ai can be easier to scale, but public cloud platforms and even a generic public cloud may still fall short where strict control, privacy, or compliance apply. Hybrid architectures can balance control, cost, scalability, and data governance requirements. Self-hosting contrasts with cloud-based AI agents, which run on external provider infrastructure and are accessed through APIs.

Request the detailed per-suite deployment diagrams from Clarity's on-premise page to see how each of these components maps to a specific deployment architecture.