Best practices

Three facts about the gateway shape almost every integration decision. It buffers the model's answer and filters it before it sends anything, so there are no token deltas to render. It rate-limits only the sign-in routes, so there is no 429 to back off from on the request path. And it accepts two kinds of bearer token that reach different endpoints. Design around those and the rest is ordinary HTTP.

In the examples, <gateway-host> is the address the gateway listens on — port 3000 by default, or whatever the reverse proxy in front of it publishes.

Hold credentials server-side

A gateway API key is a bearer credential in the form cid_key_ followed by 64 hex characters, and it is presented as Authorization: Bearer cid_key_…. There is no X-API-Key header. Anything holding that string can spend the tenant's provider budget.

• Never ship a key to a browser or a mobile binary. Call the gateway from your own server and let the browser talk to your server.
• Read the key from the environment. Do not commit it, and do not bake it into an image layer.
• Use a separate key per environment. Development, staging and production get their own keys so one can be revoked without an outage elsewhere.
• Rotate on a schedule and on departure. An administrator regenerates a key under API Keys → Regenerate API Key; the old value stops working immediately.
• Set expires_at where you can. A key with no expiry never lapses on its own.

// Server-side route. The key never leaves the server.
export default async function handler(req, res) {
  const upstream = await fetch('https://<gateway-host>/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.CID222_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ model: 'gpt-4o', messages: req.body.messages }),
  });
 
  // The gateway answers with text/event-stream. Forward it, do not buffer it.
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('X-Accel-Buffering', 'no');
  for await (const chunk of upstream.body) res.write(chunk);
  res.end();
}

Validate input before you send it

The gateway applies a global validation pipe with transform: true and whitelist: true. Unknown body fields are stripped silently rather than rejected, so a misspelt field name produces a successful request that ignores your setting. Check the field names against the DTO rather than trusting a 200.

The limits that exist are these:

There is no per-message character limit on /chat/completions. Impose your own before you spend a provider call on a runaway prompt.

function validateMessage(content) {
  if (typeof content !== 'string' || !content.trim()) {
    throw new Error('Message cannot be empty');
  }
  // Your own budget, not a gateway limit — the gateway caps the body at 50 MB.
  if (content.length > 10000) {
    throw new Error('Message too long for this application');
  }
  return content;
}

Expect one content event, not token deltas

POST /chat/completions always responds with text/event-stream. The stream field in the request body is accepted by the DTO and read by nothing — there is no non-streaming mode to opt into, and no way to opt out.

What travels over that stream is not a token feed. The orchestrator consumes the provider's deltas, accumulates the whole answer, runs the output filter over it, and emits the result as a single event:

{"id":"filtered-response","content":"…the whole reply…","finish_reason":"stop","filtered":false,"entities_masked":0}

So the client sees any control events, then silence while the model generates, then one large content event, then data: [DONE]. Plan the user interface around that: a progress indicator that resolves in one step, not a typewriter effect. A client that waits for delta.text, content_block_delta or a choices[] body waits forever.

Reuse connections

Detection runs before the provider call, so a request holds a connection for as long as the model takes to answer plus the filter time. Opening a fresh TLS connection per request adds a handshake to that. Keep a pooled dispatcher alive for the life of the process.

// Node 18+, native fetch. `agent` is a node-fetch v2 option and is ignored here.
import { Agent, fetch } from 'undici';
 
const dispatcher = new Agent({ keepAliveTimeout: 60_000, connections: 32 });
 
export function callGateway(body) {
  return fetch('https://<gateway-host>/chat/completions', {
    method: 'POST',
    dispatcher,
    headers: {
      Authorization: `Bearer ${process.env.CID222_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(body),
  });
}

Choose sessions or the stateless endpoint deliberately

Sessions keep the conversation on the server, so you send one message instead of the whole history, and the gateway manages the context window, summarisation and per-session token accounting for you.

They cost you the API key. /sessions/* is guarded by JwtAuthGuard alone: a cid_key_ bearer cannot create a session or post a message to one. Session traffic needs a user JWT from POST /auth/login, which expires after JWT_EXPIRES_IN — 24 hours by default.

GET /sessions takes no query parameters and does not paginate. List responses return context: [] with a message_count; fetch GET /sessions/:id for the full context.

Retry only what is retryable

There is no rate limiting on the request path. /chat/completions, /api/v1/guardrails/detect, /sessions/*, /image-analysis/*, /document-analysis/* and /models are not throttled at all. Only three unauthenticated routes are: POST /auth/login at 10 requests per minute per client IP, and POST /auth/forgot-password and POST /auth/reset-password at 5 per minute. Nothing in the gateway emits a Retry-After header or a rate-limit header set, so retry logic that reads one is reading null.

That leaves a short list worth retrying:

• A connection failure or a timeout before any bytes arrive. Retry with exponential backoff.
• A 5xx from a route that answers with JSON — the admin and analysis routes. Retry with backoff.
• A 429 from /auth/login. Wait out the one-minute window; there is no header telling you how long is left.

Everything else is a client error to fix, not to repeat. And note what a retry cannot recover: /chat/completions sets its status and SSE headers before it does any work, so a failure after that point arrives as an event on a 200 response, not as a status code. Repeating the request repeats the provider spend.

async function withRetry(fn, maxAttempts = 3) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      // 4xx is a request to fix. A content block is not an error at all —
      // it arrives as an SSE event on a 200.
      if (error.status >= 400 && error.status < 500) throw error;
      if (attempt === maxAttempts - 1) throw error;
      await new Promise((r) => setTimeout(r, 2 ** attempt * 1000));
    }
  }
}

Choose the model deliberately

Ask for a model that the seed data actually registers, and send provider when the name is ambiguous. gpt-4o, gpt-4o-mini and gpt-4-turbo are registered by both openai and azure_openai, and provider is what separates them. Azure's 3.5 model is gpt-35-turbo, without the dot.

• Call GET /models first. It returns only the models whose provider has an active credential for your tenant, so it is the authoritative list for that caller.
• Prefer the smaller model where it answers the question. gpt-4o-mini and gpt-4o are both registered against openai.
• Set max_tokens on anything user-facing. It is the only ceiling on a runaway completion.
• Put the behaviour you expect in a system message rather than repeating it in every user turn.

Monitor what the gateway detects

• Watch the detection feed. All Detections shows what was masked, flagged and rejected across every inspection path. A rise in rejections usually means a filter change, not an attack.
• Track token usage. The token_usage event carries the real prompt_tokens / completion_tokens split, including cached_input_tokens when the provider reported a prompt-cache discount. Record it per request; it is what the cost reports are built from.
• Alert on end-to-end latency, not on time-to-first-token. Because the answer is buffered, first byte and last byte are effectively the same moment.
• Test filter changes before they ship. Filter Testing runs a rule set against sample content without spending a provider call.

Limits and known gaps

• No rate limiting on the request path. A misbehaving client can exhaust a provider quota unimpeded. Impose a limit in your own application or in the proxy in front of the gateway.
• No incremental streaming to the client. Output filtering needs the whole response before it can judge it, so a per-token user interface is not possible through the gateway today.
• stream is inert. It validates and is then ignored, which makes it look like a supported switch in request logs.
• Unknown fields are dropped without a warning. A typo in a field name degrades silently.
• An API key cannot use sessions, image analysis or document analysis. Those surfaces are JWT-only, so a machine integration that needs them has to hold a user credential and refresh it.
• A group key borrows a member's identity. Removing that member transfers the key rather than revoking it.
• Detection quality depends on text length, detector and filter configuration. Treat any accuracy or latency figure as conditional on those; see Performance and accuracy.

Related pages

• Error handling — what each status code and SSE rejection event means, and which are worth retrying.
• Integration examples — working clients for the SSE contract described here.
• Chat API — the full request DTO and event list.