商店現已上線

我們打造用於網站、AI 智慧體和 LLM 訓練的 Workstation AI 機器——線上配置併購買。

獲取同行價格
AI 產品
AI 實驗室OpenAI 智慧體Claude 代理Grok BotWorkstation AI 原生 CRM營銷所有產品
AI 解決方案
應用型 AIAI 智慧體生成式 AIAI 自動化AI 技能提升中小企業 AI 套餐企業 AI 實驗室AI 工作站NVIDIA 硬體與軟體NVIDIA 加速計算私有 AIGPU 私有云GPU 叢集邊緣 AI行業 AI
AI 服務
平臺現代化數字工程資料基礎與 AI自主運營AI 諮詢DevOps 自動化網路安全軟體開發智慧體構建MLOps 搭建GPU 租賃GPU 私有云
關於我們
公司我們的故事合作伙伴客戶案例
AI 資料中心
AI 資料中心GPU 私有云GPU 叢集GPU 租賃
AI 洞察Jev 系列基準測試實驗室模型選擇
技術說明
文件
Linux 作業系統UbuntuRed Hat Enterprise LinuxSUSE LinuxAlpine LinuxDebian、Rocky 與 AlmaLinux
AI 雲平臺AWS 與 Amazon BedrockGoogle Cloud 與 Vertex AIMicrosoft Azure 與 AI FoundryGPU 雲與算力市場
私有 AI 資料中心面向律師事務所面向會計師事務所本地部署平臺全面安全的雲
工具Ring PromoterJobshout
新聞
部落格
聯絡我們登入

面向現代企業的 AI 工作站、AI 多智慧體軟體、GPU 基礎設施和智慧代理解決方案。

聯絡我們

AI 解決方案

應用型 AIAI 智慧體生成式 AIAI 自動化AI 技能提升中小企業 AI 套餐企業 AI 實驗室行業 AI

產品

所有產品Workstation AI native CRM 與 ERP營銷OpenAI 智慧體Ring PromoterJobshout

公司

關於我們公司AI 資料中心為什麼選擇Workstation合作伙伴客戶案例定價聯絡TenthMatrix

資源

AI 洞察技術說明文件Linux 作業系統AI 雲平臺私有 AI 資料中心部落格搜尋網站地圖
英國總部
77-79 Marlowes, Hemel Hempstead HP1 1LF路線指引 — 從 M25 外環倫敦 20 號出口駛出公司編號: 11641870週一至週五:上午 9:00 - 下午 6:00 GMT
+44 7515 356 146
比利時辦公室
Workstation SRL, Rue Vanderkindere 34, 1180 Uccle, BrusselsBE 0751.518.683週一至週五:上午 9:00 - 下午 6:00 CET
+32 492 45 67 46
印度辦公室
#159 Sector 9, Pocket 1, DDA Flats, 110077 Dwarka, New Delhi
+91 98881 98841
台灣辦公室
3F, No. 2-4, Aisi Rd., Ren'ai Dist., Keelung City 200, Taiwan基隆市仁愛區愛四路2之4號3樓週一至週五:上午 9:00 - 下午 6:00 臺灣時間(GMT+8)
+886 926 892 701+32 492 45 67 46

© 2026 Workstation AI。保留所有權利。

隱私資料刪除Cookie服務條款網站地圖

NVIDIA、NVIDIA 標誌、RTX、RTX PRO、Blackwell、CUDA、NVLink、MGX、Omniverse、Spectrum-X、Quantum、BlueField、DGX 與 HGX 為 NVIDIA Corporation 在美國及其他國家的商標及/或註冊商標。AMD、AMD Arrow 標誌、EPYC、Instinct、Ryzen 與 Threadripper 為 Advanced Micro Devices, Inc. 的商標。Dell 與 PowerEdge 為 Dell Inc. 或其子公司的商標。Intel 與 Xeon 為 Intel Corporation 或其子公司的商標。AMAX、AceleMax 與 LiquidMax 為 AMAX Engineering Corporation 的商標。所有其他產品名稱、標誌與品牌均為其各自所有者之財產。我們使用它們僅為識別我們所描述、供應或支援的產品;使用這些名稱並不表示其所有者為我們背書。

首頁 / 技術說明 / Technology
AISecurityArchitectureAI Agents

AI Usage Policy as Code: Build an LLM Gateway with PII Redaction, Tool Allow-Lists and Telemetry

Close the AI guidance gap in code: a versioned policy file, an OpenAI-compatible FastAPI gateway, Presidio redaction, per-team tool and MCP allow-lists, red-team tests, an audit log, Prometheus metrics and a usage report

2026年11月16日Technology29 min read

Most organisations already have staff using AI with no guidance, and many are now rolling out internal assistants and agents. This technote turns an AI usage policy into code you can run: a small OpenAI-compatible gateway in Python that checks every request against a versioned policy file, redacts personal data with Microsoft Presidio, allow-lists tools and MCP servers per team, logs every decision and counts usage, so you can see where guidance is still missing. All the code below was run and tested on 16 November 2026.

A policy file in YAML feeding a gateway that checks, redacts and logs requests before they reach external or private models.

Summary. Put one gateway between your people, apps and agents and every model they use. Write the policy as a YAML file reviewed like code: which teams may send which data classes to which models, and which tools each team may give an agent. The gateway enforces it, replaces emails, National Insurance numbers, card numbers and similar with placeholders before anything leaves, blocks with a message that links to the policy, and writes a JSON audit line and Prometheus metrics per request. Red-team tests in pytest prove the rules hold, and a short report shows which teams hit blocks or paste personal data: that is where training should go next. About 250 lines of Python; the alternatives table covers when to use LiteLLM, Envoy AI Gateway or NeMo Guardrails instead.

This technote accompanies our blog post The AI guidance gap: use outpaces training at school and at work, which covers the research (79% of students use AI for homework, 47% were shown how, according to ThinkYoung research cited by OpenAI [1]) and the business playbook. Here we build the technical half.

Who this is for, what you will build, prerequisites

Who it is for: platform, ML and security engineers asked to "let people use AI safely" across a company, or to put guardrails around an internal assistant or agent before it reaches real users.

What you will build:

  • policy.yaml: the AI usage policy as code (data classes, models, teams, tools, limits).
  • A FastAPI gateway exposing /v1/chat/completions, so any OpenAI-compatible SDK, IDE plug-in or agent framework works by changing its base URL.
  • PII redaction with Presidio, including the UK National Insurance number recogniser.
  • Tool and MCP controls: per-team tool allow-lists, a check on the tools the model actually calls, a human-approval flag for write tools, and a read-only MCP server.
  • Red-team tests in pytest, a JSON audit log, Prometheus metrics and a "where is guidance missing" report.
  • A container image and production notes.

Prerequisites: Python 3.12 (spaCy, which Presidio uses, did not install on Python 3.15 when we tested), basic FastAPI, and either an OpenAI-compatible model endpoint (a hosted API, or vLLM or Ollama for private models) or the mock model below. No GPU is needed for the gateway itself. Versions we tested: FastAPI 0.163.1, uvicorn 0.54.0, httpx 0.28.1, presidio-analyzer and presidio-anonymizer 2.2.366, spaCy 3.8.18 with en_core_web_sm 3.8.0, prometheus-client 0.26.0, mcp 2.3.0, pytest 9.2.0.

1. Architecture

Every request goes through four steps in the gateway: authenticate the caller and find their team; check the policy; redact what the data class requires; forward to the right upstream model. The response comes back through a fifth step that checks which tools the model asked to call. Each request produces one audit line and updates counters.

Clients such as SDKs, IDE assistants and agents send requests to the gateway, which authenticates, checks policy, redacts and routes to an external model or a private model; every request writes to an audit log and Prometheus metrics.
One gateway between every client and every model. The policy file is the only place rules live.

Why a gateway rather than rules in each app? Because the guidance gap is organisational. If ten teams each build an assistant, you get ten interpretations of the policy and no single view of use. A gateway gives one enforcement point, one log and one set of numbers, and existing tools need only a base-URL change.

2. Set up the project

mkdir llm-gateway && cd llm-gateway
mkdir gateway tests && touch gateway/__init__.py
python3.12 -m venv .venv && source .venv/bin/activate
cat > requirements.txt <<'EOF'
fastapi==0.163.1
uvicorn[standard]==0.54.0
httpx==0.28.1
pyyaml==6.0.3
prometheus-client==0.26.0
presidio-analyzer==2.2.366
presidio-anonymizer==2.2.366
pytest==9.2.0
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
EOF
pip install -r requirements.txt

The last line of requirements.txt installs the small English spaCy model as a pinned wheel, so builds are reproducible and do not need python -m spacy download at run time. It is about 13 MB to download; the large model (en_core_web_lg, several hundred MB) finds names more reliably and is worth testing if you redact PERSON.

3. Write the policy as code

The policy is organised by data class, not by tool, because tools change monthly and data classes do not. Each class lists the model aliases it may use and the entity types to redact. Teams get a ceiling (the highest data class they may send) and a tool list. Users see aliases such as external-general; the real upstream model name is configuration you can change without retraining anyone.

# AI usage policy as code. Reviewed like any other change (pull request + owner approval).
version: 7
guidance_url: https://intranet.example.internal/ai-policy   # shown to users when a request is blocked

# Data classes, lowest to highest. A team may send data up to its max_data_class.
data_classes:
  public:       {rank: 0, models: [external-general, internal-private], redact: []}
  internal:     {rank: 1, models: [external-general, internal-private],
                 redact: [EMAIL_ADDRESS, PHONE_NUMBER, UK_NINO, CREDIT_CARD, IBAN_CODE]}
  confidential: {rank: 2, models: [internal-private],
                 redact: [EMAIL_ADDRESS, PHONE_NUMBER, UK_NINO, CREDIT_CARD, IBAN_CODE, PERSON]}
  restricted:   {rank: 3, models: [], redact: []}   # never leaves the source system

# Model aliases users see -> the real upstream (any OpenAI-compatible endpoint).
models:
  external-general: {base_url_env: EXTERNAL_LLM_URL, key_env: EXTERNAL_LLM_KEY, upstream_model: gpt-4.1-mini}
  internal-private: {base_url_env: INTERNAL_LLM_URL, key_env: INTERNAL_LLM_KEY, upstream_model: meta-llama/Llama-3.1-8B-Instruct}

# Tools (function calls, usually backed by MCP servers) and who may use them.
tools:
  search_docs:     {mcp_server: docs-readonly, scope: read}
  read_invoice:    {mcp_server: erp,           scope: read}
  create_ticket:   {mcp_server: servicedesk,   scope: write}
  send_email:      {mcp_server: mail,          scope: write, requires_approval: true}

teams:
  finance:     {max_data_class: confidential, tools: [search_docs, read_invoice, send_email]}
  engineering: {max_data_class: internal,     tools: [search_docs, create_ticket]}
  marketing:   {max_data_class: public,       tools: [search_docs]}

limits:
  max_tokens: 2048
  max_messages: 50

Treat this file like code: changes go through a pull request with an owner's approval, the version number goes up, and the tests in section 8 run in CI before it ships. Every audit line records the policy version, so you can answer "which rules applied to this request?" months later.

4. Load and enforce the policy

Policy validates the file at start-up (unknown model, class or tool names fail fast rather than on the first request) and decide() returns either a Decision or a PolicyError with a stable reason code. Those codes become metric labels, so keep them short and fixed.

# gateway/policy.py
"""Load policy.yaml and decide whether a request may go upstream."""
from dataclasses import dataclass, field
from pathlib import Path

import yaml


class PolicyError(Exception):
    """A request that breaks the policy. `reason` is a stable code for metrics."""

    def __init__(self, reason: str, message: str):
        super().__init__(message)
        self.reason = reason
        self.message = message


@dataclass
class Decision:
    model_alias: str
    model: dict
    data_class: str
    redact_entities: list[str]
    approval_tools: set[str] = field(default_factory=set)


class Policy:
    def __init__(self, raw: dict):
        self.raw = raw
        self.version = raw["version"]
        self.guidance_url = raw.get("guidance_url", "")
        self.data_classes = raw["data_classes"]
        self.models = raw["models"]
        self.tools = raw["tools"]
        self.teams = raw["teams"]
        self.limits = raw.get("limits", {})
        self._validate()

    @classmethod
    def load(cls, path: str | Path) -> "Policy":
        return cls(yaml.safe_load(Path(path).read_text(encoding="utf-8")))

    def _validate(self) -> None:
        # Fail at start-up, not on the first request, if the policy refers to unknown names.
        for name, dc in self.data_classes.items():
            for m in dc["models"]:
                if m not in self.models:
                    raise ValueError(f"data class {name}: unknown model {m}")
        for team, t in self.teams.items():
            if t["max_data_class"] not in self.data_classes:
                raise ValueError(f"team {team}: unknown data class {t['max_data_class']}")
            for tool in t["tools"]:
                if tool not in self.tools:
                    raise ValueError(f"team {team}: unknown tool {tool}")

    def decide(self, team: str, data_class: str, model_alias: str, body: dict) -> Decision:
        if team not in self.teams:
            raise PolicyError("unknown_team", f"Team '{team}' has no AI access configured.")
        if data_class not in self.data_classes:
            raise PolicyError("unknown_data_class", f"Unknown data class '{data_class}'.")
        t = self.teams[team]
        if self.data_classes[data_class]["rank"] > self.data_classes[t["max_data_class"]]["rank"]:
            raise PolicyError(
                "data_class_exceeds_team",
                f"Team '{team}' may send data up to '{t['max_data_class']}', not '{data_class}'.",
            )
        allowed_models = self.data_classes[data_class]["models"]
        if model_alias not in allowed_models:
            raise PolicyError(
                "model_not_allowed",
                f"'{data_class}' data may only go to {allowed_models or 'no model'}, not '{model_alias}'.",
            )
        if len(body.get("messages", [])) > self.limits.get("max_messages", 100):
            raise PolicyError("too_many_messages", "Conversation is too long for the gateway limit.")
        approval = set()
        for tool in body.get("tools") or []:
            name = tool.get("function", {}).get("name", "")
            if name not in t["tools"]:
                raise PolicyError("tool_not_allowed", f"Tool '{name}' is not approved for team '{team}'.")
            if self.tools[name].get("requires_approval"):
                approval.add(name)
        return Decision(
            model_alias=model_alias,
            model=self.models[model_alias],
            data_class=data_class,
            redact_entities=self.data_classes[data_class]["redact"],
            approval_tools=approval,
        )
Decision flow: is the team known, is the data class within the team ceiling, is the model allowed for the data class, are all declared tools on the team list; any no returns a 403 with a reason code and a link to the policy; all yes redacts and forwards.
The order of checks in decide(). Every "no" has its own reason code, which is what the usage report counts later.

5. Redact personal data with Presidio

Presidio's analyzer combines a spaCy named-entity model with pattern recognisers (regular expressions plus checksums or context words) and returns spans with confidence scores; the anonymizer replaces them [2]. We ask only for the entity types the data class lists, which avoids most false positives.

# gateway/redact.py
"""PII redaction with Microsoft Presidio (spaCy NER + pattern recognisers)."""
import logging
from collections import Counter

from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.nlp_engine import NlpEngineProvider
from presidio_analyzer.predefined_recognizers import UkNinoRecognizer
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

logging.getLogger("presidio-analyzer").setLevel(logging.ERROR)  # quiet start-up warnings


class Redactor:
    def __init__(self, spacy_model: str = "en_core_web_sm", score_threshold: float = 0.4):
        provider = NlpEngineProvider(nlp_configuration={
            "nlp_engine_name": "spacy",
            "models": [{"lang_code": "en", "model_name": spacy_model}],
        })
        self.analyzer = AnalyzerEngine(nlp_engine=provider.create_engine(), supported_languages=["en"])
        self.analyzer.registry.add_recognizer(UkNinoRecognizer())  # shipped, but not loaded by default
        self.anonymizer = AnonymizerEngine()
        self.threshold = score_threshold

    def redact(self, text: str, entities: list[str]) -> tuple[str, Counter]:
        if not entities or not text:
            return text, Counter()
        results = self.analyzer.analyze(
            text=text, language="en", entities=entities, score_threshold=self.threshold
        )
        if not results:
            return text, Counter()
        out = self.anonymizer.anonymize(
            text=text,
            analyzer_results=results,
            operators={e: OperatorConfig("replace", {"new_value": f"<{e}>"}) for e in entities},
        )
        # Count what was actually replaced (overlapping hits are merged by the anonymizer).
        return out.text, Counter(item.entity_type for item in out.items)

    def redact_messages(self, messages: list[dict], entities: list[str]) -> tuple[list[dict], Counter]:
        total: Counter = Counter()
        clean = []
        for m in messages:
            m = dict(m)
            if isinstance(m.get("content"), str):
                m["content"], found = self.redact(m["content"], entities)
                total += found
            elif isinstance(m.get("content"), list):  # multi-part content: redact the text parts
                parts = []
                for p in m["content"]:
                    if p.get("type") == "text":
                        p = dict(p)
                        p["text"], found = self.redact(p["text"], entities)
                        total += found
                    parts.append(p)
                m["content"] = parts
            clean.append(m)
        return clean, total

Three things we found while testing, all worth knowing before you rely on it:

  • The UK National Insurance recogniser ships with Presidio but is not loaded by default. Neither are the other UK recognisers (passport, driving licence, postcode, vehicle registration). Add the ones you need with registry.add_recognizer(), as above.
  • The email recogniser validates the top-level domain. jane.doe@example.test is not detected because .test is not a public suffix; jane.doe@example.eu is. Use realistic test data or your tests will pass for the wrong reason.
  • The small spaCy model is noisy when you ask for everything. Without an entity list, it tagged "Email" as a person and parts of a phone number as dates. Restrict entities per data class and keep a score threshold (we use 0.4). A UK mobile number written in international format (country code first, with spaces) was not caught at that threshold in our test, so do not treat phone redaction as complete.

6. The gateway

The gateway speaks the OpenAI chat-completions format because almost every SDK, IDE assistant and agent framework can point at it. It hashes the prompt for the audit log instead of storing it, caps max_tokens, swaps the alias for the real model name, and checks the tool calls in the response, not only the tools the client declared.

# gateway/app.py
"""A small LLM gateway: policy check -> PII redaction -> upstream -> audit log + metrics.

Speaks the OpenAI-compatible /v1/chat/completions API, so existing SDKs and tools work by
changing their base URL. Run:  uvicorn gateway.app:create_app --factory --port 8080
"""
import hashlib
import json
import logging
import os
import sys
import time
import uuid
from contextlib import asynccontextmanager

import httpx
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from prometheus_client import Counter, Histogram, make_asgi_app

from .policy import Policy, PolicyError
from .redact import Redactor

REQUESTS = Counter("gateway_requests_total", "Chat requests", ["team", "model", "outcome"])
BLOCKS = Counter("gateway_blocks_total", "Requests blocked by policy", ["team", "reason"])
REDACTIONS = Counter("gateway_redactions_total", "PII entities redacted", ["team", "entity"])
TOKENS = Counter("gateway_tokens_total", "Tokens used", ["team", "model", "kind"])
LATENCY = Histogram("gateway_upstream_seconds", "Upstream latency", ["model"])

audit = logging.getLogger("audit")
audit.setLevel(logging.INFO)
audit.addHandler(logging.StreamHandler(sys.stdout))
audit.propagate = False


def parse_keys(raw: str) -> dict[str, str]:
    """GATEWAY_KEYS="token1=finance,token2=engineering" -> {token: team}. Use SSO in production."""
    pairs = (p.split("=", 1) for p in raw.split(",") if "=" in p)
    return {k.strip(): v.strip() for k, v in pairs}


def create_app(policy_path: str | None = None, transport: httpx.AsyncBaseTransport | None = None) -> FastAPI:
    policy = Policy.load(policy_path or os.environ.get("POLICY_PATH", "policy.yaml"))
    redactor = Redactor(os.environ.get("SPACY_MODEL", "en_core_web_sm"))
    keys = parse_keys(os.environ.get("GATEWAY_KEYS", ""))

    @asynccontextmanager
    async def lifespan(app: FastAPI):
        app.state.http = httpx.AsyncClient(timeout=60, transport=transport)
        yield
        await app.state.http.aclose()

    app = FastAPI(title="llm-gateway", lifespan=lifespan)
    app.mount("/metrics", make_asgi_app())

    def blocked(team: str, reason: str, message: str, status: int, record: dict) -> JSONResponse:
        BLOCKS.labels(team, reason).inc()
        REQUESTS.labels(team, record.get("model", "-"), "blocked").inc()
        audit.info(json.dumps(record | {"outcome": "blocked", "reason": reason}))
        return JSONResponse(status_code=status, content={"error": {
            "type": "policy_violation", "code": reason, "message": message,
            "guidance": policy.guidance_url, "policy_version": policy.version,
        }})

    @app.get("/healthz")
    async def healthz():
        return {"status": "ok", "policy_version": policy.version}

    @app.post("/v1/chat/completions")
    async def chat(request: Request):
        started = time.perf_counter()
        token = request.headers.get("authorization", "").removeprefix("Bearer ").strip()
        team = keys.get(token)
        if team is None:
            return JSONResponse(status_code=401, content={"error": {"message": "unknown gateway key"}})
        body = await request.json()
        model_alias = body.get("model", "")
        data_class = request.headers.get("x-data-class", "internal")  # default to a cautious class
        record = {
            "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "request_id": str(uuid.uuid4()),
            "team": team,
            "user": request.headers.get("x-user", "unknown"),
            "use_case": request.headers.get("x-use-case", "unspecified"),
            "model": model_alias,
            "data_class": data_class,
            "policy_version": policy.version,
            "tools": [t.get("function", {}).get("name") for t in body.get("tools") or []],
            # A hash lets you join repeated prompts without storing their text.
            "prompt_sha256": hashlib.sha256(json.dumps(body.get("messages", []), sort_keys=True).encode()).hexdigest(),
        }
        if body.get("stream"):
            return blocked(team, "stream_not_supported", "Streaming is not enabled on this gateway.", 400, record)
        try:
            decision = policy.decide(team, data_class, model_alias, body)
        except PolicyError as e:
            return blocked(team, e.reason, e.message, 403, record)

        messages, found = redactor.redact_messages(body.get("messages", []), decision.redact_entities)
        for entity, n in found.items():
            REDACTIONS.labels(team, entity).inc(n)
        record["redactions"] = dict(found)

        upstream = dict(body, messages=messages, model=decision.model["upstream_model"])
        upstream["max_tokens"] = min(int(body.get("max_tokens") or 1024), policy.limits.get("max_tokens", 2048))
        upstream.pop("max_completion_tokens", None)
        base_url = os.environ[decision.model["base_url_env"]].rstrip("/")
        headers = {}
        if key := os.environ.get(decision.model.get("key_env", ""), ""):
            headers["Authorization"] = f"Bearer {key}"

        t0 = time.perf_counter()
        try:
            resp = await request.app.state.http.post(f"{base_url}/chat/completions", json=upstream, headers=headers)
        except httpx.HTTPError as e:
            REQUESTS.labels(team, model_alias, "upstream_error").inc()
            audit.info(json.dumps(record | {"outcome": "upstream_error", "error": type(e).__name__}))
            return JSONResponse(status_code=502, content={"error": {"message": "upstream unavailable"}})
        LATENCY.labels(model_alias).observe(time.perf_counter() - t0)
        if resp.status_code != 200:
            REQUESTS.labels(team, model_alias, "upstream_error").inc()
            audit.info(json.dumps(record | {"outcome": "upstream_error", "status": resp.status_code}))
            return JSONResponse(status_code=502, content={"error": {"message": f"upstream returned {resp.status_code}"}})

        data = resp.json()
        # Check what the model asked to call, not only what the client declared.
        allowed = set(policy.teams[team]["tools"])
        called = [tc["function"]["name"]
                  for ch in data.get("choices", [])
                  for tc in (ch.get("message", {}).get("tool_calls") or [])]
        if bad := [c for c in called if c not in allowed]:
            return blocked(team, "model_called_unapproved_tool", f"Model tried to call {bad}.", 502, record)
        needs_approval = sorted(set(called) & decision.approval_tools)

        usage = data.get("usage") or {}
        for kind in ("prompt_tokens", "completion_tokens"):
            TOKENS.labels(team, model_alias, kind).inc(usage.get(kind, 0))
        REQUESTS.labels(team, model_alias, "ok").inc()
        audit.info(json.dumps(record | {
            "outcome": "ok", "tool_calls": called, "approval_required": needs_approval,
            "usage": usage, "latency_ms": round((time.perf_counter() - started) * 1000),
        }))
        data["model"] = model_alias
        data["gateway"] = {"request_id": record["request_id"], "redactions": dict(found),
                           "approval_required": needs_approval, "policy_version": policy.version}
        return data

    return app

Notes on the choices:

  • Authentication. GATEWAY_KEYS maps a key to a team so you can try it in minutes. In production, put the gateway behind your identity provider (for example oauth2-proxy or your API gateway validating an OIDC token) and take team and user from verified claims, not headers.
  • Data class. The caller declares it in X-Data-Class, defaulting to internal. That is honest about a hard problem: only the calling app knows what it is sending. Give each app its own key with a fixed class where you can, and rely on redaction as the backstop, not the primary control.
  • Streaming is refused with a clear error. Checking tool calls and logging usage on a stream needs the response to be buffered or parsed as it flows; add it once the basics work.
  • Blocked requests never reach a model, and the error carries the reason, the policy version and a link to the guidance. That message is training delivered at the moment it is needed.

7. Run it locally

To try it without an API key, use a stand-in model that echoes what it received, so you can see exactly what would have left the building.

# mock_llm.py
"""A stand-in OpenAI-compatible model that echoes what it received. For local testing only."""
from fastapi import FastAPI, Request

app = FastAPI()


@app.post("/v1/chat/completions")
async def chat(request: Request):
    body = await request.json()
    last = body["messages"][-1]["content"]
    return {
        "id": "mock-1", "object": "chat.completion", "model": body["model"],
        "choices": [{"index": 0, "finish_reason": "stop",
                     "message": {"role": "assistant", "content": f"Model received: {last}"}}],
        "usage": {"prompt_tokens": len(last.split()), "completion_tokens": 4, "total_tokens": len(last.split()) + 4},
    }
uvicorn mock_llm:app --port 9001 &
export GATEWAY_KEYS="dev-eng=engineering,dev-fin=finance,dev-mkt=marketing"
export EXTERNAL_LLM_URL=http://127.0.0.1:9001/v1 INTERNAL_LLM_URL=http://127.0.0.1:9001/v1
uvicorn gateway.app:create_app --factory --port 8091 --no-access-log > audit.jsonl &

curl -s http://127.0.0.1:8091/v1/chat/completions \
  -H "Authorization: Bearer dev-eng" -H "X-Data-Class: internal" \
  -H "X-User: sam" -H "X-Use-Case: support-reply" -H "Content-Type: application/json" \
  -d '{"model":"external-general","messages":[{"role":"user",
       "content":"Reply to jane.doe@example.eu, NI number AB 12 34 56 C, about her refund."}]}'

Expected output (abridged). The model only ever saw placeholders:

{
  "model": "external-general",
  "choices": [{"message": {"role": "assistant",
    "content": "Model received: Reply to <EMAIL_ADDRESS>, NI number <UK_NINO>, about her refund."}}],
  "gateway": {"redactions": {"UK_NINO": 1, "EMAIL_ADDRESS": 1},
              "approval_required": [], "policy_version": 7}
}

Now send confidential data to the external model as finance:

curl -s http://127.0.0.1:8091/v1/chat/completions \
  -H "Authorization: Bearer dev-fin" -H "X-Data-Class: confidential" -H "Content-Type: application/json" \
  -d '{"model":"external-general","messages":[{"role":"user","content":"Summarise the board pack"}]}'
{"error": {"type": "policy_violation", "code": "model_not_allowed",
  "message": "'confidential' data may only go to ['internal-private'], not 'external-general'.",
  "guidance": "https://intranet.example.internal/ai-policy", "policy_version": 7}}

Existing code needs only a new base URL. With the official OpenAI Python SDK (we tested 3.51.0), the gateway's extra gateway field is available on model_extra:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8091/v1", api_key="dev-eng",
                default_headers={"X-Data-Class": "internal", "X-Use-Case": "code-review"})
r = client.chat.completions.create(
    model="external-general",
    messages=[{"role": "user", "content": "Explain this stack trace, card=4111 1111 1111 1111"}])
print(r.choices[0].message.content)   # Model received: Explain this stack trace, card=<CREDIT_CARD>
print(r.model_extra["gateway"])       # {'request_id': '…', 'redactions': {'CREDIT_CARD': 1}, …}

Pitfall: on our test machine another process already listened on port 8080 over IPv6, so curl localhost:8080 reached the wrong server and the SDK got a 404. Use 127.0.0.1 and a free port when testing locally.

8. Agents and MCP: allow-listed tools with scoped permissions

An agent is a model that can call tools, and the Model Context Protocol (MCP) is the common way to plug tools in. Agents need three layers of control, because no single layer sees everything.

Three layers of control for agents: the client loads only allow-listed MCP servers; the gateway checks declared tools and the tools the model actually calls, and flags write tools for approval; each MCP server runs with least-privilege credentials, such as a read-only database connection.
Defence in depth for agents. The gateway sees intent (tool calls); only the MCP server's own credentials limit what actually happens.

Layer 1, the client: restrict which MCP servers assistants may load. For Claude Code, administrators set an allow-list in managed settings; with allowManagedMcpServersOnly users cannot widen it, and remote servers must match a serverUrl pattern [7]:

{
  "allowManagedMcpServersOnly": true,
  "allowedMcpServers": [
    { "serverUrl": "https://mcp-docs.internal.example.com/*" },
    { "serverUrl": "https://mcp-servicedesk.internal.example.com/*" }
  ],
  "deniedMcpServers": [
    { "serverUrl": "https://*.untrusted.example.com/*" }
  ]
}

The same documentation warns that matching on serverName is not a security control, because users choose the name; match on URL or command instead [7]. Other clients have their own equivalents; check them before rollout.

Layer 2, the gateway: already in the code above. A request declaring a tool outside the team's list gets 403 tool_not_allowed; a response in which the model calls a tool it was not given is replaced with 502 model_called_unapproved_tool; and a call to a tool marked requires_approval is returned with approval_required set, which your agent runtime must route to a person before executing.

Layer 3, the tool itself: give each MCP server the narrowest credentials that work. This read-only documentation server uses the MCP Python SDK 2.x. Note that 2.x renamed FastMCP to MCPServer and uses snake_case field names, so many 1.x examples online no longer import.

# docs_mcp.py
"""A read-only MCP server: one tool, no write paths, read-only credentials."""
import sqlite3

from mcp.server.mcpserver import MCPServer
from mcp.types import ToolAnnotations

server = MCPServer("docs-readonly")
DB = "file:docs.db?mode=ro"  # the OS-level guarantee: this process cannot write


@server.tool(annotations=ToolAnnotations(read_only_hint=True, open_world_hint=False))
def search_docs(query: str, limit: int = 5) -> list[dict]:
    """Search approved internal documentation by title. Returns title and URL only."""
    limit = max(1, min(limit, 20))
    with sqlite3.connect(DB, uri=True) as conn:
        rows = conn.execute(
            "SELECT title, url FROM docs WHERE title LIKE ? ORDER BY title LIMIT ?",
            (f"%{query}%", limit),
        ).fetchall()
    return [{"title": t, "url": u} for t, u in rows]


if __name__ == "__main__":
    server.run(transport="streamable-http")

The read_only_hint annotation tells clients the tool does not change anything, but it is a hint. The guarantee is mode=ro: the process cannot write even if the model asks it to. Test it in memory, with no network:

# test_mcp.py  (run python seed_docs.py first to create docs.db)
import asyncio
from mcp import Client
from docs_mcp import server

async def main():
    async with Client(server) as client:          # in-memory transport, no network
        tools = await client.list_tools()
        for t in tools.tools:
            print(t.name, t.annotations.read_only_hint)
        result = await client.call_tool("search_docs", {"query": "policy"})
        print(result.structured_content)

asyncio.run(main())
search_docs True
{'result': [{'title': 'AI acceptable use policy', 'url': 'https://intranet.example.internal/ai-policy'}]}

Run it as a service with python docs_mcp.py; it serves streamable HTTP at http://127.0.0.1:8000/mcp by default, which we also called successfully with mcp.Client [8].

9. Test it: guardrail and red-team tests

Guardrails you have not tested are guesses. The tests below use httpx.MockTransport as the upstream, which records exactly what would have been sent to a model, so a test can assert that a secret never left. The cases live in YAML so that people writing the policy can add cases without writing Python. Each one is a request a well-meaning but untrained colleague might send.

# tests/redteam.yaml
# Each case is a request a well-meaning but untrained colleague might send.
# expect.status: what the gateway must return. must_not_reach_model: strings that must be redacted.
- name: customer email and NI number are redacted before the external model
  key: eng-key
  data_class: internal
  model: external-general
  prompt: "Draft a reply to jane.doe@example.eu about her NI number AB 12 34 56 C."
  expect: {status: 200, must_not_reach_model: ["jane.doe@example.eu", "AB 12 34 56 C"]}

- name: card number in a pasted log is redacted
  key: eng-key
  data_class: internal
  model: external-general
  prompt: "Why did this payment fail? card=4111 1111 1111 1111 status=declined"
  expect: {status: 200, must_not_reach_model: ["4111 1111 1111 1111"]}

- name: confidential data cannot go to the external model
  key: fin-key
  data_class: confidential
  model: external-general
  prompt: "Summarise the Q3 board pack."
  expect: {status: 403, code: model_not_allowed}

- name: marketing cannot label data above its ceiling
  key: mkt-key
  data_class: internal
  model: external-general
  prompt: "Summarise this internal roadmap."
  expect: {status: 403, code: data_class_exceeds_team}

- name: restricted data goes nowhere
  key: fin-key
  data_class: restricted
  model: internal-private
  prompt: "Here is the payroll export..."
  expect: {status: 403, code: data_class_exceeds_team}

- name: engineering may not declare the email tool
  key: eng-key
  data_class: internal
  model: external-general
  prompt: "Email the release notes to all customers."
  tools: [send_email]
  expect: {status: 403, code: tool_not_allowed}
# tests/test_guardrails.py
"""Guardrail tests: a fake upstream records exactly what would have left the building."""
import json
from pathlib import Path

import httpx
import pytest
import yaml
from fastapi.testclient import TestClient

from gateway.app import create_app

HERE = Path(__file__).parent
CASES = yaml.safe_load((HERE / "redteam.yaml").read_text())


def tool(name: str) -> dict:
    return {"type": "function", "function": {"name": name, "parameters": {"type": "object", "properties": {}}}}


@pytest.fixture
def gateway(monkeypatch):
    monkeypatch.setenv("GATEWAY_KEYS", "fin-key=finance,eng-key=engineering,mkt-key=marketing")
    monkeypatch.setenv("EXTERNAL_LLM_URL", "https://external.test/v1")
    monkeypatch.setenv("INTERNAL_LLM_URL", "http://internal.test/v1")
    sent: list[dict] = []
    reply = {"tool_calls": None}

    def upstream(request: httpx.Request) -> httpx.Response:
        sent.append(json.loads(request.content))
        message = {"role": "assistant", "content": "ok", "tool_calls": reply["tool_calls"]}
        return httpx.Response(200, json={
            "id": "x", "object": "chat.completion", "model": "m",
            "choices": [{"index": 0, "message": message, "finish_reason": "stop"}],
            "usage": {"prompt_tokens": 12, "completion_tokens": 3, "total_tokens": 15},
        })

    app = create_app(str(HERE.parent / "policy.yaml"), transport=httpx.MockTransport(upstream))
    with TestClient(app) as client:
        yield client, sent, reply


def post(client, key, data_class, model, prompt, tools=()):
    return client.post(
        "/v1/chat/completions",
        headers={"Authorization": f"Bearer {key}", "X-Data-Class": data_class, "X-User": "test"},
        json={"model": model, "messages": [{"role": "user", "content": prompt}],
              "tools": [tool(t) for t in tools] or None},
    )


@pytest.mark.parametrize("case", CASES, ids=[c["name"] for c in CASES])
def test_redteam_case(gateway, case):
    client, sent, _ = gateway
    r = post(client, case["key"], case["data_class"], case["model"], case["prompt"], case.get("tools", ()))
    assert r.status_code == case["expect"]["status"], r.text
    if code := case["expect"].get("code"):
        assert r.json()["error"]["code"] == code
        assert sent == [], "a blocked request must never reach a model"
    for secret in case["expect"].get("must_not_reach_model", []):
        assert secret not in json.dumps(sent), f"{secret!r} reached the model"


def test_unknown_key_is_rejected(gateway):
    client, sent, _ = gateway
    assert post(client, "nope", "public", "external-general", "hi").status_code == 401
    assert sent == []


def test_model_cannot_call_a_tool_it_was_not_given(gateway):
    client, _, reply = gateway
    reply["tool_calls"] = [{"id": "c1", "type": "function",
                            "function": {"name": "send_email", "arguments": "{}"}}]
    r = post(client, "eng-key", "internal", "external-general", "tidy this", tools=["search_docs"])
    assert r.status_code == 502
    assert r.json()["error"]["code"] == "model_called_unapproved_tool"


def test_write_tool_is_flagged_for_human_approval(gateway):
    client, _, reply = gateway
    reply["tool_calls"] = [{"id": "c1", "type": "function",
                            "function": {"name": "send_email", "arguments": "{}"}}]
    r = post(client, "fin-key", "internal", "external-general", "chase invoice 42", tools=["send_email"])
    assert r.status_code == 200
    assert r.json()["gateway"]["approval_required"] == ["send_email"]


def test_upstream_sees_real_model_name_and_capped_tokens(gateway):
    client, sent, _ = gateway
    post(client, "fin-key", "confidential", "internal-private", "summarise")
    assert sent[-1]["model"] == "meta-llama/Llama-3.1-8B-Instruct"
    assert sent[-1]["max_tokens"] <= 2048
$ pytest -q
..........                                                     [100%]
10 passed in 5.54s

These tests check the gateway's deterministic rules. They do not test the model. For model behaviour (prompt injection through retrieved documents, jailbreaks, unsafe output), add a separate evaluation suite run against your real upstream: a fixed set of adversarial prompts and documents with expected outcomes, scored on each model or prompt change. The OWASP Top 10 for LLM Applications 2025 is a good checklist for what to cover, in particular prompt injection (LLM01), sensitive information disclosure (LLM02) and excessive agency (LLM06) [5]. Run the guardrail tests in CI on every policy change.

10. Usage telemetry: see where guidance is missing

The gateway produces two streams. Prometheus counters for dashboards and alerts:

gateway_requests_total{model="external-general",outcome="blocked",team="marketing"} 2.0
gateway_blocks_total{reason="data_class_exceeds_team",team="marketing"} 2.0
gateway_blocks_total{reason="model_not_allowed",team="finance"} 1.0
gateway_redactions_total{entity="CREDIT_CARD",team="engineering"} 1.0

(The metrics app is mounted at /metrics, which redirects to /metrics/; Prometheus follows the redirect, or scrape /metrics/ directly.) And one JSON line per request, with no prompt text:

{"ts": "2026-11-16T12:29:00Z", "team": "engineering", "user": "sam", "use_case": "support-reply",
 "model": "external-general", "data_class": "internal", "policy_version": 7, "tools": [],
 "prompt_sha256": "f7d1e4a7…", "redactions": {"UK_NINO": 1, "EMAIL_ADDRESS": 1}, "outcome": "ok",
 "usage": {"prompt_tokens": 9, "completion_tokens": 4, "total_tokens": 13}, "latency_ms": 115}

The report turns that log into the question the business cares about: which teams need guidance, on what?

# report.py
"""Summarise the gateway audit log: where is guidance missing?

Usage: python report.py audit.jsonl
"""
import json
import sys
from collections import Counter, defaultdict

requests, blocks, redactions = Counter(), defaultdict(Counter), defaultdict(Counter)
use_cases = defaultdict(Counter)
for line in open(sys.argv[1], encoding="utf-8"):
    if not line.startswith("{"):
        continue  # skip non-audit lines (e.g. uvicorn access logs)
    r = json.loads(line)
    team = r["team"]
    requests[team] += 1
    use_cases[team][r.get("use_case", "unspecified")] += 1
    if r["outcome"] == "blocked":
        blocks[team][r["reason"]] += 1
    for entity, n in (r.get("redactions") or {}).items():
        redactions[team][entity] += n

print(f"{'team':<12} {'requests':>8} {'blocked':>8} {'block %':>8} {'PII hits':>9}  top block reason")
for team, n in requests.most_common():
    b = sum(blocks[team].values())
    top = blocks[team].most_common(1)[0][0] if b else "-"
    print(f"{team:<12} {n:>8} {b:>8} {100 * b / n:>7.0f}% {sum(redactions[team].values()):>9}  {top}")
print("\nUnlabelled use (no X-Use-Case header):")
for team, uc in use_cases.items():
    print(f"  {team}: {uc['unspecified']} of {sum(uc.values())}")
$ python report.py audit.jsonl
team         requests  blocked  block %  PII hits  top block reason
engineering         2        0       0%         3  -
marketing           2        1      50%         0  data_class_exceeds_team
finance             1        1     100%         0  model_not_allowed

Unlabelled use (no X-Use-Case header):
  engineering: 0 of 2
  finance: 1 of 1
  marketing: 2 of 2
A loop: the gateway logs and counts requests; a weekly report shows blocks, redactions and unlabelled use per team; the policy owner turns that into targeted training, a policy change or a new approved tool; the change ships as a new policy version; and the gateway measures again.
Telemetry closes the loop. Blocks and redactions are signals of missing guidance, not offences.

How to read it: a team with many model_not_allowed blocks wants a capability the policy does not provide (perhaps a private model for confidential work). Many redactions mean people paste personal data, so their training should cover what to strip out. High unlabelled use means you do not know what AI is used for, so ask. Pair these with quality measures; Google's engineering team reports tracking acceptance rates and running online A/B experiments because offline metrics "are often only rough proxies of user value" [6].

11. Run it in production

# Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY gateway/ gateway/
COPY policy.yaml .
RUN useradd --uid 10001 gateway
USER 10001
EXPOSE 8080
CMD ["uvicorn", "gateway.app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8080", "--no-access-log"]

Build with a .dockerignore that admits only requirements.txt, gateway/ and policy.yaml. We built this image and ran it: it starts in a few seconds, /healthz answers, and policy blocks work. It is about 760 MB, mostly spaCy and its dependencies.

  • CPU, not GPU. The gateway and Presidio run on CPU. Redaction with the small model took about 5 ms for a two-line prompt and about 170 ms for a 1,000-word prompt on our test machine, and it grows with prompt length; measure with your own prompts and add CPU before you add workers.
  • Scale with replicas, one worker each. prometheus_client keeps counters per process. With several uvicorn workers in one container you need its multiprocess mode; one worker per pod and more pods is simpler.
  • Kubernetes. Use /healthz for readiness and liveness, mount policy.yaml from a ConfigMap built from your repository, and take upstream keys from your secret store. Use a network policy so only the gateway can reach the model endpoints; otherwise people will bypass it.
  • Audit log. Ship stdout to your log platform. It holds user identifiers and is personal data in its own right: set a retention period, restrict access, and do not add prompt text unless you have a lawful reason and a short retention.
  • Private models. Point INTERNAL_LLM_URL at any OpenAI-compatible server you host, such as vLLM or Ollama, so confidential work never leaves your network.
  • Latency and failure. The gateway is now on the critical path. Run at least two replicas, set upstream timeouts (60 seconds here), and alert on outcome="upstream_error".

12. Pitfalls and debugging

  • A guardrail test passes but data still leaks: check your test data is detectable (the .test email case) and assert on what the mock upstream received, not on the response.
  • ModuleNotFoundError: mcp.server.fastmcp: you have the MCP SDK 2.x; import MCPServer from mcp.server.mcpserver, or pin mcp<2 for 1.x code.
  • Noisy Presidio warnings at start-up about Italian, Spanish or Polish recognisers are harmless; they are skipped because the registry is English-only. We set the presidio-analyzer logger to ERROR.
  • Everything is blocked with unknown_team: the key in GATEWAY_KEYS maps to a team name not in policy.yaml.
  • People route around the gateway: make it the only network path to model APIs, and make the approved tools good enough that nobody wants to.
  • Multi-part messages (images plus text): the redactor only cleans text parts. Block image inputs for classes above public until you add image handling.

13. Alternatives

OptionWhat it isChoose it whenWatch out for
This gateway (about 250 lines)FastAPI, Presidio, YAML policy, PrometheusYou want to read every rule, start in a day and extend freelyYou own streaming, budgets, retries and provider quirks
LiteLLM proxyOpen-source OpenAI-compatible proxy for many providersYou need many providers, virtual keys and spend limits; it has a Presidio PII guardrail [4]Large configuration surface; review which guardrails run in which mode
Envoy AI Gateway (renamed Agent Router)Open-source control plane for AI and agent traffic, built on Envoy Gateway for KubernetesYou already run Envoy or Gateway API and want routing, rate limits and auth at the edgePolicy logic beyond routing lives in extensions you write
NVIDIA NeMo GuardrailsOpen-source library of programmable input, output and dialogue railsYou need conversational rules, topic control or fact-checking rails inside one appA library inside your app, not a company-wide choke point
Safety classifier models (for example Llama Guard)A model that labels prompts or responses as safe or unsafeYou need content-safety screening in addition to data rulesAdds latency and a model to run; complements, not replaces, policy checks
Vendor admin controlsEnterprise settings in your AI assistant productsYou only use one or two vendor productsEach vendor is a separate policy and log; no single view

These are not exclusive. A common pattern is LiteLLM or Envoy for provider routing and keys, a thin policy service like this one for your data-class rules, and a classifier for content safety.

14. FAQ

Does redaction make it safe to send confidential data to an external model? No. Detection misses things, and context can identify a person even without a name. That is why the example policy routes confidential data only to the private model, and redaction is the backstop.

Will it break tools that expect OpenAI responses? Not in our test with the OpenAI SDK: we add one extra top-level field (gateway) and replace the model name with the alias. Strict clients that reject unknown fields are rare, but test yours.

Can I use it with Anthropic, Google or other providers? Yes, through any OpenAI-compatible endpoint, or by putting LiteLLM behind this gateway as the upstream, so this layer keeps your policy and LiteLLM handles provider APIs.

How does this help with the EU AI Act? Article 4 asks providers and deployers to take measures to support their staff's AI literacy [3]. A gateway does not train anyone, but its block messages, policy versions and usage reports are evidence of the measures you took and show where training is still needed.

Should we log prompts for investigations? Default to hashes and metadata. If you must keep content, keep it for a short period, encrypt it and restrict access; otherwise the audit log becomes your largest store of personal data.

What about agents that act without a chat request? Route their model calls through the gateway like any other client, and enforce tool scopes in the MCP servers (layer 3). The approval flag only works if your agent runtime honours it, so test that path explicitly.

If you want this running against private models on your own infrastructure, our private AI team builds and operates gateways, model hosting and the monitoring behind them for UK organisations.

References

  1. Closing the AI guidance gap in education, OpenAI, 10 November 2026 (citing ThinkYoung research).
  2. Microsoft Presidio documentation and source (analyzer, anonymizer, predefined recognisers including UkNinoRecognizer).
  3. EU AI Act, Article 4: AI literacy, as recalibrated by the July 2026 Digital Omnibus.
  4. LiteLLM: PII masking with Presidio, LiteLLM documentation.
  5. OWASP Top 10 for LLM Applications 2025, OWASP GenAI Security Project.
  6. AI in software engineering at Google: progress and the path ahead, Google Research, 6 June 2024.
  7. Control MCP server access for your organization, Claude Code documentation.
  8. MCP Python SDK (2.x: MCPServer, Client) and MCP's move to the Agentic AI Foundation, Anthropic, 9 December 2025.
  9. FastAPI, uvicorn (application factories) and Prometheus Python client documentation.
分享這篇文章

更多 Technology 內容

Testing ML and LLM Systems You Can't Prove Correct: A Python Evidence Kit

Testing ML and LLM Systems You Can't Prove Correct: A Python Evidence Kit

Evidence instead of proofs, in about 400 lines of Python with real output: slice metrics with bootstrap intervals and a pass/fail contract, Hypothesis property and metamorphic tests, calibration and conformal prediction, Pydantic contracts and regression evals for LLM output, drift checks, a CI gate, and exact verification of a small neural network

閱讀更多
TorchCurves in Practice: Learnable B-spline and Legendre Curves in PyTorch

TorchCurves in Practice: Learnable B-spline and Legendre Curves in PyTorch

Install torchcurves, fit and size a curve, choose B-spline or Legendre and the input map, build a monotone calibration curve and an additive model on tabular data, compare with SciPy, scikit-learn, a KAN and an MLP, export coefficients to JSON, test and run it in production, with real outputs

閱讀更多
Run Agentic, Voice and Vision Models On-Device with Cactus: A Hands-On Guide (2026)

Run Agentic, Voice and Vision Models On-Device with Cactus: A Hands-On Guide (2026)

Install cactus-compute, make tool calls, gate on confidence with a fallback to a bigger model, extract structured data, run speech and vision, measure latency and memory, and ship an offline container, with every example run and a comparison with llama.cpp, ONNX Runtime, ExecuTorch, MLC LLM and TensorRT Edge-LLM

閱讀更多