Self-hosted LLM security usually gets planned around the model: what it may say, what it may call, who may prompt it. NEAR AI's incident report from November 9 is a reminder that the inference engine underneath is ordinary network-facing code, and it can be the way in. An attacker used a remote code execution flaw in SGLang, the open-source engine serving NEAR AI's models, to run code on one inference server and reach the company's wider infrastructure. If you serve open models for your product or your agents, this article covers what happened, why confidential computing did not cover it, and the check we would run before an inference worker is allowed to start.

One inference server led to the rest of the infrastructure

NEAR AI's own account is short and specific. The attacker "exploited a remote code execution vulnerability in SGLang, the open-source inference engine running on our servers. This gave them code execution on one inference server and, from there, access to our infrastructure."

From that foothold, NEAR AI says, the attacker reached an internal system behind its deprecated Private Chat product and downloaded account details for about 100 users: email addresses, names and, for some accounts, plan and balance information. They also reached 18 agents on a legacy deployment of NEAR AI Agents, retrieving their chat history, settings and logs. And "Two Cloud customers had NEAR AI Cloud API keys exposed in logs." Those keys were revoked. The company says it found no indication that stored Private Chat conversations or uploaded files were accessed, and it has rotated credentials, patched SGLang across all inference servers and restored access to NEAR AI Cloud.

NEAR AI has not said which SGLang vulnerability was used. The engine has had several remote code execution reports this year. One example is CVE-2026-5760, which CERT/CC describes as template injection through a malicious model file reaching the /v1/rerank endpoint, where "Deployments that expose the affected interface to untrusted networks are at the highest risk of exploitation." CERT/CC also notes that "No response or patch was obtained during the coordination process." We are not saying that is the flaw NEAR AI hit; it shows the class of bug an inference engine carries.

An enclave protects you from the host, and only from the host

NEAR AI runs confidential inference in trusted execution environments (TEEs), hardware enclaves that keep data hidden from the machine's operator. Its report is unusually clear about the limit: "Confidential computing protects data from the operator and the host. It does not protect against a vulnerability in the software running inside the environment". The flaw let the attacker run code inside one inference server and see requests being processed there, including ones in confidential mode.

That is the general lesson for anyone serving models. A TEE, a private VPC or an air-gapped GPU box all limit who else is on the machine. None of them limit what the code on the machine can reach once it misbehaves. The inference process parses untrusted prompts, loads model files from the internet and renders templates that ship inside those files. Treat it like a public web server that happens to own a GPU.

The 18 legacy agents are the second lesson. They were not running in NEAR AI's TEE environment, and they held chat history and logs. Old deployments keep their data and their credentials long after anyone is watching them, which makes them easy places to land.

Give the worker nothing worth stealing

The goal is a blast radius small enough that code execution on an inference server gets an attacker a GPU and the requests passing through it, and nothing else. In practice that means four properties:

  1. No secrets in the process. Model weights are pulled at build time, so the running worker needs no Hugging Face token, no database URL and no cloud credentials.
  2. No route inward. The worker can answer the gateway in front of it and nothing else: no path to the app database, the session store, the agent runtime or the cloud metadata endpoint.
  3. No keys in its logs. Logs never carry request headers. API keys are checked at the gateway, which forwards an opaque request id instead.
  4. No orphans. A deployment that is deprecated is deleted, along with its data, rather than left running beside the new one.

The first two are easy to state and easy to lose in a later config change, so we check them every time the worker boots. A sketch of how we would do it, as a preflight that runs before the engine starts:

// preflight.ts: refuse to boot an inference worker
// that has something worth stealing. A sketch.
import { connect } from "node:net";

const SECRET = /KEY|TOKEN|SECRET|PASSWORD|DATABASE_URL/i;
const INWARD: Array<[string, number]> = [
  ["10.0.0.5", 5432],       // app database
  ["10.0.0.9", 6379],       // session store
  ["169.254.169.254", 80],  // cloud metadata
];

function reachable(host: string, port: number) {
  return new Promise<boolean>((resolve) => {
    const s = connect({ host, port, timeout: 1500 });
    s.on("connect", () => { s.destroy(); resolve(true); });
    s.on("timeout", () => { s.destroy(); resolve(false); });
    s.on("error", () => resolve(false));
  });
}

const secrets = Object.keys(process.env)
  .filter((name) => SECRET.test(name));
const open: string[] = [];
for (const [host, port] of INWARD) {
  if (await reachable(host, port)) open.push(`${host}:${port}`);
}
if (secrets.length > 0 || open.length > 0) {
  console.error("preflight failed", { secrets, open });
  process.exit(1);
}

Notice that the check tests what the worker can actually do from inside its own network namespace, rather than trusting the firewall config that was meant to apply. If a later change adds a database URL to the environment or opens a route to the session store, the worker stops at boot and the deploy fails where someone will see it.

Where this costs you

Pulling weights at build time makes images large, often tens of gigabytes, and slows rollouts. A shared read-only volume populated by a separate job is the usual compromise, and that job holds the download token instead of the worker.

The list of inward targets has to be maintained. It is only as good as your knowledge of what sits on the network, so generate it from your infrastructure code where you can rather than typing addresses by hand.

None of this stops the bug itself. The attacker still sees the requests on the server they reach, as NEAR AI's attacker could. Patch the engine quickly, keep its endpoints off public networks, and load model files only from sources you have reviewed. Containment decides how bad the day is once a patch comes too late.

If you call a hosted API and never run your own engine, most of this sits with your provider. Ask them how their inference workers are isolated, and keep your own agents' credentials out of anything you send them. Our AI agent guardrails page covers the agent side of the same boundary.

What to take from it

  • Treat the inference engine as untrusted, network-facing code. It loads files and renders templates from the internet.
  • Confidential computing protects requests from the host. It does nothing about code running inside the enclave.
  • Give the worker no secrets and no route inward, and check both at every boot.
  • Delete deprecated deployments together with their data.

If you are putting open models behind your product and want a second pair of eyes on the boundary, book a conversation.

Sources