Private LLM for Customer Service: Deployment Patterns, On-Prem Trade-offs, and What to Ask a Vendor

Private LLM for Customer Service: Deployment Patterns, On-Prem Trade-offs, and What to Ask a Vendor

Private LLM for Customer Service: Deployment Patterns, On-Prem Trade-offs, and What to Ask a Vendor

Guides

Default share icon

Private LLM for Customer Service: Deployment Patterns, On-Prem Trade-offs, and What to Ask a Vendor

Private LLM for Customer Service: Deployment Patterns, On-Prem Trade-offs, and What to Ask a Vendor

A private LLM for customer service means choosing where inference runs, where logs live, and which features survive on-prem. Compare deployment patterns.

A private LLM for customer service means choosing where inference runs, where logs live, and which features survive on-prem. Compare deployment patterns.

ยท

16

min

Guides

Private LLM for Customer Service: Deployment Patterns, On-Prem Trade-offs, and What to Ask a Vendor

Guides

Private LLM for Customer Service: Deployment Patterns, On-Prem Trade-offs, and What to Ask a Vendor
Summarize this page with your favourite AI assistant
ChatGPTClaudeGoogle GeminiGrokPerplexity

A private LLM for customer service is a deployment arrangement, not a model choice: prompts, completions, and logs stay inside a boundary the organization controls. An on-prem LLM is the strictest version, hardware you operate with no external network path. In-region cloud, private tenancy, and redacted public API are private LLMs too, just with the boundary drawn differently.

That line can sit in different places, and each placement changes what you actually get. A private LLM deployment might mean a dedicated instance in an approved cloud region, a virtual private cloud with no shared tenancy, or a server in a data center your IT team runs with no outbound connection at all. Treating all of these as the same thing is where procurement conversations go wrong. A CX leader asking โ€œcan we run this privatelyโ€ and an IT leader asking โ€œcan we run this on premโ€ are asking different questions, with different cost, latency, and feature consequences.

For a customer service organization, the decision comes down to three things: where inference executes, where conversation logs come to rest once a ticket closes, and which product features still work when nothing crosses the network edge. That third question is the one vendors answer least honestly. This piece uses OnClarity (www.onclarity.com) as a worked example, including the exact scope of its own on-premise deployment, because a vendorโ€™s constraints tell you more than its marketing does.

This is written for CX and IT leaders evaluating a private LLM for customer service at the organizational level, not for anyone comparing consumer chat apps on a laptop.

Where can public and private LLMs actually run, and what changes at each boundary?

Follow one customer message through the system. It hits a channel endpoint (chat widget, email inbox, voice gateway), gets matched to a conversation record, and triggers retrieval of relevant passages from a knowledge base, internal documents, or past tickets. In practice, that is retrieval augmented generation: an existing model is supplemented with internal data before it answers. Those passages, the customerโ€™s text, and recent history get assembled into a prompt. The prompt goes to a model somewhere. A response comes back, gets shown to the customer or drafted for an agent, and the full exchange is written to a transcript store and an audit log. Six hops: channel ingress, the retrieval index, prompt assembly, the model call, the response path, and logging. Each hop is a separate question about who can see the data and where it sits.

Most teams answer only the fourth question. Public and private LLMs differ here mainly in control and data handling: public LLMs are third-party services, while a private LLM keeps the organization closer to its own data boundary. โ€œWhere does the model runโ€ gets scrutinized in procurement, while the retrieval index and the logs, usually larger in volume and longer-lived than any single model call, get waved through. A model call is stateless and over in seconds. A retrieval index holds your whole knowledge base in searchable form. A transcript store holds every conversation your organization has ever had. If data residency matters, it has to apply to all six hops, not just the one with a recognizable vendor name attached.

Here are the four patterns teams actually choose between:

Pattern

Where data goes

Who runs the model

Operational effort

Best fit

Public API

Prompt text leaves the network to a vendor-operated endpoint

The model vendor

Lowest, no infrastructure to manage

Teams without data-residency constraints, governed by contract terms instead

Public API with PII removal

A redaction layer strips names, account numbers, and contact details before the call; pseudonymised text leaves the network

The model vendor, on text it cannot tie to an identity

Low to moderate, requires a redaction layer tuned to your data

Teams whose policy objects to personal data leaving, not to external compute itself

Private or in-region cloud

Data stays within a dedicated tenancy or a region pinned to a defined jurisdiction

The vendor or cloud provider, inside that boundary

Moderate, contractual and architectural controls, no hardware to run

Residency obligations tied to a jurisdiction, not a building

Fully on-premise or air-gapped

Inference executes on the organizationโ€™s own hardware behind its firewall

The organization, or a vendor operating inside that network under contract

Highest, GPU capacity, serving infrastructure, and update pipelines become your responsibility

Strict IT policy or regulated environments where no external network path is acceptable

The trade-off has the same shape at every tier, priced differently. Public LLMs usually lead on reasoning, while private setups can produce more accurate responses when grounded in internal data. Public API buys the least operational burden and the least control over where bytes travel. PII removal buys a real reduction in exposure at the cost of building a redaction layer that must catch identifiers reliably before anything leaves. In-region cloud buys jurisdictional guarantees at the cost of lock-in to whichever regions the vendor operates. Air-gapped on-prem buys the strongest guarantee, nothing crosses the boundary, at the cost of owning GPU capacity, patching, and monitoring that a cloud vendor would otherwise handle. The right row depends on whether your constraint is a jurisdiction, a contract clause, or an IT policy that draws the line at the firewall.

OnClarity offers all four patterns, SaaS, in-region cloud, private cloud, and fully on-premise, rather than forcing one architecture on every customer. Every AI and human action is logged in each configuration, because logging is its own data surface, not an afterthought. Full detail is on the deployment page.

Should we self-host an open-weight model or bring our own proprietary data model to the vendor?

โ€œOur IT policy does not allow external LLMsโ€ is really two requests wearing one sentence. One organization means it owns and operates the model itself. The other means it will accept a model running on an endpoint it already governs, even if a vendor built the application around it. Some companies also want a model that is fine-tuned or adapted through training on proprietary data rather than only hosted privately. Both are routes to a private LLM, and they carry different operational weight.

What you operate

Where weights live

Who patches and upgrades

Quality ceiling

Failure mode

Self-hosted open-weight model

GPU capacity, inference serving, monitoring, scaling, security patching

On your own infrastructure

Your team, on your schedule, against a checkpoint you froze

Bounded by that checkpoint until you manually upgrade it

An outage or jailbreak is yours to diagnose, usually fixed at the prompt or guardrail layer

Bring your own model

The endpoint itself, your cluster or a separately contracted private deployment

Wherever that endpoint sits, cloud or on-prem

Whoever operates that endpoint under your contract

Set by whichever model sits behind the endpoint

Depends on your endpointโ€™s own SLA, decoupled from the CX vendorโ€™s uptime

Self-hosting means owning the full inference stack. Bring-your-own-model means youโ€™ve solved that problem elsewhere and are pointing the vendorโ€™s application layer at an endpoint you govern. Neither route requires accepting a vendorโ€™s default model.

Open-weight versus proprietary is a separate axis, worth untangling on its own terms:

Quality shifts on a release cycle. At any given moment, frontier proprietary models tend to lead on hard reasoning and long-context tasks, and open-weight releases close that gap and sometimes reopen it with the next release. Treat any specific benchmark claim as dated the moment you read it. Still, private deployments can be customized to a brand voice and company-specific workflows, giving more relevant answers in internal large language model use cases.

Updates work differently in each direction. A proprietary endpoint can change behavior between your own releases, since the vendor controls the version behind the API. That is a silent evaluation problem: last quarterโ€™s regression suite may no longer describe todayโ€™s behavior. An open-weight checkpoint, once downloaded, is a frozen artifact. It does not drift. It also does not improve unless you deliberately swap it and re-run evaluation before cutover.

Licensing matters at the procurement stage, not after. Open-weight is not open-source. Several widely used open-weight families attach a usage-based ceiling, often a monthly-active-users or revenue threshold, above which commercial use needs a separate agreement. Others attach acceptable-use restrictions or attribution obligations. None of this is disqualifying, but it is legal reviewโ€™s job before a model is selected.

Patching follows ownership. When a jailbreak class surfaces against a hosted proprietary model, the provider ships a mitigation on its own schedule. When the same weakness surfaces against a self-hosted checkpoint, the weights donโ€™t change until you swap them; mitigation happens at the prompt, guardrail, or retrieval layer your team controls, slower to start but fully within your visibility.

This is why a model-agnostic application layer is an architectural decision, not a marketing line. The engineering value in a customer service deployment sits in routing, retrieval grounding, guardrails, escalation, and audit logging, not in the model call itself. OnClarity is built this way: a choice of leading LLMs, a self-hosted open-weight model, or your own model behind an endpoint you already govern. Some organizations use a hybrid architecture, keeping sensitive workloads on private systems and sending lower-risk tasks to public large language models. Swapping the model shouldnโ€™t require rebuilding the system around it.

If we use a global model, how is personal data actually protected with enhanced data security?

A PII removal layer intercepts the prompt after assembly and before the model call. It scans for names, addresses, phone numbers, account and card numbers, and other sensitive information, then replaces each one with a stable placeholder before the redacted prompt leaves the network. The model answers using placeholders. On the way back, the response is rehydrated: placeholders are swapped for real values so the customer sees their own account details, even though the model never saw them.

Detection is the hard part, and it is a probabilistic process, not a lookup. Structured identifiers like an email address or a 16-digit card number are easy to catch. An account number typed mid-sentence with a stray space, a misspelled street name, or an address pasted into one run-on line are the common misses, because the detector is pattern-matching free text rather than a validated field. Recall is never 100 percent here. A redaction layer trades a small amount of context quality, the model answers slightly blind, for a large reduction in exposure, since most identifying data never reaches the model vendor at all. Placeholder consistency within a conversation matters: if an account number maps to the same placeholder every time, the model can still reason about โ€œthe account mentioned earlierโ€ correctly.

This reduces exposure; it does not remove the legal obligation. GDPR Article 4(5) defines pseudonymisation, and Recital 26 makes clear that pseudonymised data which could be re-identified using additional information held separately is still personal data. A redaction layer with a reversible placeholder mapping is pseudonymisation, not anonymisation, and that distinction determines whether your organizationโ€™s data protection obligations continue to apply after redaction. They do. Private deployments can reduce the risk of data breaches compared with public models because prompts and logs stay inside an organization-controlled boundary.

Logging is the larger standing data surface, and most teams stop analyzing the problem before they get to it. Prompts, completions, tool calls, and full transcripts accumulate over time, in volume, at rest. A single redacted model call is brief. A transcript store holds every conversation your organization has ever had, which is exactly what makes it useful for QA review and exactly why it deserves the same scrutiny as the model call itself. OnClarity logs every AI and human action, producing an audit trail by design. Where that trail physically sits, in-region cloud, private cloud, or your own hardware, is a deployment question, not a property of the redaction layer. That choice directly affects data privacy and compliance with privacy regulations in regulated industries, where data security is a buying requirement and the 2023 average cost of a breach reached $4.45 million.

On certification: OnClarity is SOC 2 Type II certified. It is an independent auditorโ€™s attestation that specified controls operated effectively over a review window, typically six to twelve months. It is not a guarantee about any single data flow. Separately, OnClarity is GDPR and HIPAA ready, and aligned with Saudi PDPL. Ready and aligned describe a posture built to meet those frameworks; they are not third-party certifications in the way SOC 2 Type II is. Full scope and controls are on the enterprise security page.

What does running a private LLM in production actually take, and how do the costs and cost efficiency compare?

Running a private LLM in production takes six standing commitments: sized GPU capacity, an inference serving layer, continuous monitoring, a patching process, application-layer guardrails, and an evaluation harness. Cost-wise, per-token APIs are variable spend that tracks volume; self-hosted inference is reserved capacity billed whether or not itโ€™s used.

  1. GPU capacity. Memory has to cover both the modelโ€™s weights and the KV cache generated during inference, and these scale differently. Weights at FP16 run roughly 2 bytes per parameter, so a 70-billion-parameter model needs around 140GB before a single request arrives. The KV cache grows with context length and with concurrency: every simultaneous conversation holds its own cache footprint. A system sized for ten concurrent conversations can run out of memory at fifty, even though the weights havenโ€™t changed. Size for peak contact-center volume, not average load.

  2. Inference serving. Raw weights donโ€™t serve requests. A serving layer handles continuous batching, queueing, timeouts, and what happens under overload across cloud or internal servers. A well-built layer sheds load predictably, with a bounded queue and clear timeout, rather than degrading silently until every conversation in flight slows down at once.

  3. Monitoring. Averages hide the failures that matter. Track latency at p95 and p99, not the mean, because the slowest 5 percent of responses are what a customer notices. Track token throughput, queue depth, GPU utilization, and memory pressure alongside application-level signals like escalation rate and containment rate, because a server can look healthy while response quality drops. Ongoing monitoring also takes real resources to keep thresholds, dashboards, and alerts current.

  4. Updates. Three layers need patching: the serving stack, the runtime, and the model itself. In a fully air-gapped environment, none of this arrives automatically. A patch has to be staged and verified outside the boundary, signed, transferred across the gap, and re-verified before admission. That adds weeks of lag between a patch becoming available and reaching production. Budget that lag as a planned process.

  5. Guardrails. Grounding against a knowledge base, refusal behavior, output filtering, and escalation to a human live in the application layer around the model, not in the weights. That layer has to be ported deliberately alongside any model change.

  6. Evaluation. An offline evaluation set drawn from real production conversations, run as a regression check before any model or prompt change ships, plus ongoing drift monitoring. Whoever owns the model owns the eval harness. There is no vendor on the other side of an API call catching a regression for you once the model is yours to operate.

The cost comparison follows from that list. A per-token API is variable operating expense: cost tracks conversation volume, and idle periods cost close to nothing. Self-hosted inference inverts that. GPU capacity is reserved and paid for whether or not itโ€™s handling traffic, which turns a usage charge into a utilization problem: the economics work once sustained concurrency keeps that reserved capacity busy most of the time. Below that crossover, self-hosting costs more per conversation. Self-hosting can avoid ongoing subscription fees tied to public-model access, but that cost efficiency only pays off when utilization is high enough to justify the fixed infrastructure, staffing, and speed requirements. Above that crossover, the arithmetic can favor self-hosting, but only after accounting for the team paid to keep it running: on-call coverage, capacity planning, and maintenance of the eval harness itself. None of that shows up on a per-token invoice.

OnClarityโ€™s pricing is custom, scoped to the specific problem, and usage-based on conversation volume rather than charged per seat, regardless of deployment pattern. Request a demo for pricing scoped to your own conversation volume.

Which customer service features stop working on-premise, and what should we ask a vendor?

Fully on-premise deployment typically keeps conversation handling, retrieval, guardrails, and logging running inside the network. It typically loses or changes outbound email relay, cross-tenant analytics, and automatic model upgrades. The same setup can also support internal knowledge assistants or automated processing of regulated documents when those workflows must stay inside the network. OnClarity runs AI agents for voice and chat and Voice of Customer analytics fully on-premise today.

Features disappear at the network boundary for an architectural reason. Most CX platforms are built as a cloud control plane paired with a data plane that does the actual work of handling a conversation, retrieving a passage, and calling a model. The control plane manages configuration, routing rules, and coordinated upgrades across customers. When a deployment goes fully on-premise, the data plane moves inside the firewall and runs standalone. The control plane stays in the vendorโ€™s cloud to serve every customer at once. Anything depending on a live connection between the two either needs an explicit outbound allowance through the firewall, or stops working.

Usually keeps working inside the network: conversation handling for voice and chat, retrieval over an internal knowledge base for secure search across internal documents, guardrails, routing logic, escalation to a human agent, and audit logging of every AI and human action. The same pattern can ground answers in product details and other specific datasets, which improves accuracy when people ask for specific information.

Usually stays in the cloud or needs a deliberate exception: channel integrations calling third-party SaaS APIs over the public internet, outbound email delivery through hosted sending infrastructure, analytics depending on cross-tenant or cloud-side processing at scale, and model upgrades, which stop being a background change the vendor pushes silently and become a scheduled internal operation your team plans and executes.

That last point is worth sitting with. In a cloud deployment, a model upgrade happens to you, on the vendorโ€™s release calendar. On-premise, itโ€™s something your team does, on a calendar you control, with a regression suite you run before cutover. More control and more work, at the same time.

OnClarityโ€™s on-premise scope follows the same logic: AI agents for voice and chat and Voice of Customer analytics run fully on-premise today, while integrations that depend on the public internet need a deliberate outbound exception through the firewall.

Four questions to ask before you sign:

  1. Where does inference run, and can you show the request path including retrieval and logging? Ask for a diagram, not a sentence.

  2. Can we bring our own model or endpoint, and what breaks if we do? Ask specifically which guardrails or features are tied to the vendorโ€™s default model.

  3. What exactly is logged, where does it live, for how long, and who can read it? This is an audit-trail question, not a privacy-policy restatement.

  4. What features are unavailable in the deployment pattern we need, and is that a current limit or a roadmap item? A roadmap promise is not a current capability.

A vendor that cannot name a loss is either not actually deployed on-premise in production, or has not been asked the question directly enough. Further detail is on the AI agents page and in this on-premise deployment post.

How should a CX team decide between deployment patterns?

The decision is three conditional branches, because the specific policy objection blocking procurement determines which pattern is correct; healthcare, legal, and financial teams are common cases where confidentiality requirements push organizations toward private deployments.

If the objection is personal data leaving the network: PII removal in front of a global model usually satisfies it, at the lowest operational cost of the three branches. You add a redaction layer and a rehydration step; the model vendor never sees identity-linked text.

If the objection is jurisdiction: in-region or private cloud usually answers it without requiring you to take on inference operations. A dedicated tenancy or a region pinned to a defined jurisdiction satisfies a residency requirement tied to where data is processed, while the vendor still operates and patches the serving stack. This branch gets over-solved constantly; teams reach for fully on-premise when the actual requirement was โ€œstays in this jurisdiction,โ€ not โ€œtouches no external infrastructure at all.โ€

If the objection is any outbound traffic, full stop: fully on-premise is the only branch that answers it, and the feature trade-offs are real. Outbound email relay, cross-tenant analytics, and automatic model upgrades are the kinds of capabilities that typically change at that boundary. Price that trade-off into the decision before procurement starts, not after the deployment is live.

Most teams get further by scoping the deployment pattern to the workload that needs it, rather than applying the strictest pattern everywhere because one workload requires it. Voice transcripts handling account numbers and a public-facing FAQ bot do not carry the same risk, so the right choice is a business decision based on task sensitivity, not a one-size-fits-all rule. For many companies, forcing both through an air-gapped deployment means paying for on-prem GPU capacity, patching, and monitoring for traffic that never needed it. Scope the pattern per workload, then let the hardest constraint govern only the workload it applies to.

Request a demo to work through which pattern fits which workload in your environment. Pricing is custom, scoped to the problem, and usage-based on conversation volume rather than charged per seat.

Frequently asked questions

What is a private LLM?

A private LLM is a deployment arrangement where prompts, completions, and logs stay inside a boundary the organization controls, rather than a specific model or product. The boundary can sit in a vendorโ€™s dedicated cloud tenancy, an in-region data center, or fully on-premise. What makes it private is where data goes, not which model answers the prompt.

Is an on prem LLM the same as a private LLM?

No. On prem LLM is the strictest version of a private LLM, where inference runs on hardware the organization operates with no external network path. Private LLM is the broader category, including in-region cloud and private tenancy, both of which keep data under contractual or jurisdictional control without requiring you to run GPU infrastructure yourself.

Can we use our own model with a CX platform?

Yes, this is called bring your own model: the platformโ€™s application layer, routing, retrieval, guardrails, logging, points at a model endpoint you already govern, your own infrastructure or a separately contracted deployment. OnClarity supports this alongside a choice of leading LLMs or a self-hosted open-weight model, so the model becomes a configuration choice, not a fixed dependency.

Does PII removal make it safe to use a global model?

It substantially reduces exposure by stripping identifying data before the model call, but it does not remove the legal obligation. Under GDPR, pseudonymised data that could be re-identified using information held separately is still personal data. PII removal lowers risk; it does not convert personal data into anonymous data.

What do we lose by deploying fully on-premise?

Typically outbound email relay, cross-tenant analytics, and automatic model upgrades, since these depend on a cloud control plane. OnClarity runs AI agents for voice and chat and Voice of Customer analytics fully on-premise today; integrations that call third-party SaaS APIs over the public internet need an explicit outbound exception through the firewall.

OnClarity

5,000 free AI support resolutions for startups

Startups in the US, UK and Europe with under 200 staff get 5,000 free resolutions from OnClarityโ€™s AI support agent as their trial.

Apply for the Startup Program

Open until 31 March 2027 or until 50 startups are approved.

OnClarity

5,000 free AI support resolutions for startups

Startups in the US, UK and Europe with under 200 staff get 5,000 free resolutions from OnClarityโ€™s AI support agent as their trial.

Apply for the Startup Program

Open until 31 March 2027 or until 50 startups are approved.

OnClarity

5,000 free AI support resolutions for startups

Startups in the US, UK and Europe with under 200 staff get 5,000 free resolutions from OnClarityโ€™s AI support agent as their trial.

Apply for the Startup Program

Open until 31 March 2027 or until 50 startups are approved.

Latest topics

Latest topics