Air-Gapped LLM Deployment: An On-Prem Guide for Regulated Teams

An air-gapped LLM deployment runs a language model on hardware with no network path to the outside: the weights, the retrieval index, the application and every update cross into the environment deliberately, are verified, and are logged. An on-premises deployment is the looser cousin, running on servers you own and control but usually connected to your internal network. Both keep prompts, documents and outputs on your hardware, and both are now practical with open-weight models.

On-prem, private cloud or air-gapped?

• Private cloud. Your own cloud account, your own keys, no shared model endpoint. Fastest to stand up, and often acceptable when the data is sensitive but not restricted from cloud processing.
• On-premises. Your servers, your facility, connected to your internal network. Chosen when data must stay in your building or when you want predictable cost at steady load.
• Air-gapped. No outside connectivity at all. Required for some classified, critical infrastructure and highly regulated environments, and the most demanding to operate.

Pick the least restrictive option your data allows. Every step toward isolation adds operational work, especially for updates.

A reference architecture

1. Model serving. An inference server hosting one or more open-weight models on local GPUs, exposing an internal API. Smaller models often do the job and are cheaper to host and easier to evaluate.
2. Retrieval. A document pipeline that parses, chunks and embeds your corpus into a local vector index, with access controls carried through so users only retrieve what they are cleared to read.
3. Application layer. The chat, search, drafting or agent interface, with prompt templates, guardrails and tool permissions defined in code and reviewed like code.
4. Identity and access. Integration with your directory so every request is tied to a user and a role.
5. Logging and audit. Prompts, retrieved sources, outputs and tool actions logged to a store your security team controls.
6. Transfer station. For air-gapped sites, a controlled process and media path for bringing in weights, containers, packages and documents.

Sizing the hardware

The first-order estimate is memory. Model weights need roughly the parameter count multiplied by the bytes per parameter: about 2 bytes at 16-bit precision, about 1 byte at 8-bit and about half a byte at 4-bit. A 7-billion-parameter model therefore needs around 14 GB for weights at 16-bit and around 4 GB at 4-bit, before overhead. On top of the weights you need memory for the attention cache, which grows with context length and the number of simultaneous users, plus room for the serving software.

Size from the workload backward: the longest context you need, the number of concurrent users, and the response time they will accept. Then test quantized models against your own evaluation set, because the accuracy cost of quantization varies by model and task.

Getting models and updates across the gap

This is where air-gapped projects succeed or stall. A workable process:

• Download weights and containers on a connected staging system, and check them against the publisher's hashes or signatures.
• Scan, record a software bill of materials, and package with a manifest.
• Carry across on approved media, verify the hashes again inside, and log the transfer.
• Run the new version against the internal evaluation set before promoting it, and keep the previous version ready for rollback.

Plan a regular cadence. A model and a dependency stack frozen for years accumulates vulnerabilities and falls behind on quality.

Security controls that matter

• Least privilege for tools. If the model can call tools or run code, scope each permission narrowly and require approval for consequential actions.
• Prompt injection defenses. Treat retrieved documents as untrusted input. The OWASP Top 10 for LLM applications is a practical checklist.
• Secure development. Build and update under a secure software development framework, with an SBOM for every release.
• Framework alignment. Map controls to the requirements you are held to, such as NIST SP 800-171 families for controlled unclassified information.

Evaluate inside the perimeter

Public benchmarks tell you little about your documents and your users. Build an internal evaluation set of real questions with verified answers, labeled by your own experts, and run it on every model and configuration change. Track accuracy, refusal behavior, citation quality and latency. Keep the set inside the enclave.

Operating it day to day

A private model is a system to run, not a file to install. Assign an owner for the model stack, just as you would for a database. Monitor GPU utilization, queue depth and response times so you see capacity problems before users do. Review a sample of conversations and retrieved sources on a schedule, with appropriate access controls, to catch drift in answer quality and new kinds of misuse. Re-index documents when the corpus changes, and remove content that should no longer be retrievable. Keep a written runbook for restarts, rollbacks and incident response that works without outside access, because in an air-gapped site nobody can search for the fix.

When a smaller or different model is the answer

Not every task needs a large model. Classification, extraction and routing often run well on small language models, and some perception tasks at the edge are better served by event-driven hardware than by any language model. We cover that end of the spectrum on our neuromorphic computing page.

The AI Cowboys deploy language models on-premises, air-gapped, or in the customer's own cloud. See edge, on-prem and air-gapped AI, our private and on-prem LLM solutions, how LLMs work in AI engineering, and cybersecurity and AI trust. Defense contractors should also read CMMC and AI tools.