Skip to main content
CID222 Docs

Deploy the local LLM Analyst stack

Connect the read-only MCP evidence service, risk analyst and a local OpenAI-compatible model on an online or air-gapped deployment.

  • Version: 0.4
  • Feature: risk.llm_review
  • Role: admin_user
  • Type: task

The Analyst stack has three separate trust boundaries:

Dashboard → gateway → risk-analyst → cid-mcp-server → read-only Postgres role
                              └────→ OpenAI-compatible local model

The model never receives a database connection. It can request only the MCP server's typed tools; the MCP server pins scope, excludes secret columns and runs through cid_mcp_ro. The analyst then verifies figures and labels against those tool results before the gateway releases an answer.

What do I need?

Licence
Enterprise risk.llm_review; mcp.server for the evidence service
Role
admin_user

Prerequisites

  • The base appliance is healthy and its migrations are current.
  • You have chosen Ollama, the signed cid-llm GPU slice, or a separately managed llama-server.
  • The model endpoint supports OpenAI-compatible tool_calls; plain text function names are not enough.
  • Air-gapped deployments have the signed model and engine payloads before the maintenance window.

Choose the inference placement

PlacementSupported useMinimum evidence-backed sizing
Host-native OllamaDevelopment and connected PoCs14B Q4 model needs roughly 16–23 GB host RAM; below 16 GB is unsupported
Separate cid-llm GPU VMSupported GPU deploymentOne NVIDIA card with 20 GB VRAM, 40 GB free for staging on both /var/lib/cid-llm and Docker root
CPU llama-serverSupported analyst protocol, custom deployment16–23 GB RAM for Q4_K_M; 24 GB or more for Q5_K_M; no compose overlay ships

The measured GPU reference is Qwen3-14B-AWQ on an RTX 4000 SFF Ada 20 GB: 9.36 GiB weights, 32768-token context and 26.7 tokens/s for one stream. Treat those as one tested configuration, not a promise for every card or concurrency level. See optional service sizing for the total appliance plus analyst arithmetic.

Start the evidence and analyst services

The profile is off by default:

docker compose --profile llm-analyst up -d mcp-server risk-analyst

Set matching MCP_SHARED_TOKEN values and apply MCP_DB_PASSWORD to the database role. The MCP server builds its baked reference indexes at startup and can answer 503 for roughly 80 seconds; its compose health allowance is 180 seconds. The analyst starts only after MCP is healthy.

RISK_ANALYST_SERVICE_ENABLED controls automatic review dispatch. It does not gate interactive Analyst chat. Keep it false until the health and grounding checks below pass.

Connect an inference engine

Run Ollama on the host, pull qwen3:14b, and set:

LLM_ENGINE=ollama
LLM_BASE_URL=http://host.docker.internal:11434/v1
LLM_MODEL=qwen3:14b
ANALYST_CONTEXT_WINDOW=32768

Start Ollama with OLLAMA_CONTEXT_LENGTH=32768 and OLLAMA_KEEP_ALIVE=-1. The /v1 request cannot set either value, so a default server may unload the model or expose a shorter context.

For vLLM, enable automatic tool selection with the Hermes parser. Do not enable request logging: analyst prompts contain customer evidence.

Air-gapped sequence

  1. On a connected build host, fetch the pinned weights and engine, produce the inner model manifest, and build the signed .cidllm bundle.
  2. Transfer the bundle and public update key through the customer's approved media process.
  3. Run the GPU-host preflight: Docker Compose 2.24 or later, NVIDIA driver/toolkit, nvidia-smi, staging space on both filesystems and a routable private bind address.
  4. Install the bundle. Signature, image identity and every model file digest are checked before the active symlink changes.
  5. Point the appliance at the private endpoint. No internet route is required at runtime.

Verify before enabling automatic reviews

  1. GET http://mcp-server:8020/healthz is healthy, not degraded, and reports the expected tools.
  2. GET http://risk-analyst:8019/health reports both llm_reachable and mcp_connected true.
  3. Run the engine acceptance probe: bearer refusal, model identity, parsed tool call, thinking suppression, streamed usage and a $defs/$ref tool schema.
  4. Ask Which input filters fired most often in the last 7 days? in AI Analyst. Require a rendered evidence table, tool trail and Verified result.
  5. Ask the same question as a group-scoped administrator. Confirm every tool call stays inside the permitted tenant set.
  6. Only then set RISK_ANALYST_SERVICE_ENABLED=true and exercise one automatic review window.

If it fails

Runbookperformance

The LLM risk analyst is disabled, unreachable, or times out

You might see: The risk analyst service is not enabled on this deployment. · The risk analyst could not be reached. · The analyst is already running an analysis. · The analysis did not start.

Before you start

Checks

  1. 1

    Read whether the gateway believes the analyst is enabled

    expected · The flag is set.

  2. 2

    Read whether the analyst service answered

    diagnostics · services[risk-analyst].reachable = true

  3. 3

    Read the MCP server, which the analyst reads the estate through

    diagnostics · services[mcp-server].reachable = true

  4. 4

    Check whether an analysis is already running

    expected · No analysis is in flight.

  5. 5

    For timeouts, compare the configured deadline against how long a reading actually takes

    expected · The deadline is longer than a typical completed review took.

  • A healthy model that returns no tool_calls is incompatible or misconfigured. For vLLM check --enable-auto-tool-choice --tool-call-parser hermes; for llama-server check --jinja.
  • MCP degraded means the read-only transaction or grants are wrong. Repair the role; do not give the analyst a broader database credential.
  • Repeated quarantine means the verifier cannot trace the answer to tool results. Inspect the tool trail and scope before changing prompts or disabling verification.
  • GPU out-of-memory at startup means weights plus KV cache exceed the card. Reduce context or concurrency under a measured test; do not advertise the new limit until the acceptance probe passes.
  • Use AI Analyst — conversations, evidence trail and quarantine behaviour.
  • Risk analysis — automatic sliding-window reviews.
  • Optional services — RAM, VRAM and combined sizing.
  • MCP servers — why the placeholder inventory page is unrelated to this private evidence service.

Last updated on

On this page