Toxicity and attack detection

Two detectors decide whether a prompt is harmful rather than merely sensitive: a thirteen-label toxicity classifier and a two-label attack classifier, plus a deterministic scan for SQL injection and cross-site scripting. All three read text that has already been decoded.

The problem

Harm is not one category. "How do I make a bomb" and "you people are vermin" are both unacceptable and need different answers, and an organisation that treats them as one score cannot set a policy that blocks the first and reviews the second. Attacks are worse: a prompt that overrides the system instruction looks like ordinary text, and an attacker who knows a classifier is watching will spell the payload in base64, leetspeak or zero-width characters. A single score over the raw string answers neither problem.

How CID222 does it

Decoding comes first

Eight normalisers run over the text before any model sees it, in a fixed order — cheap structural cleanups first, so the semantic decoders work on already-stripped text.

The pass repeats up to NORMALIZATION_MAX_PASSES times (default 2), so one layer of encoding inside another is unwrapped. NORMALIZATION_ENABLED turns the whole stage off.

Thirteen toxicity labels

cid-hap-guard-v2 runs a multilingual ONNX classifier and returns a score for every one of thirteen labels on every request: violence, crime, sexual crime, child exploitation, defamation, dangerous advice, privacy, intellectual property, weapons, hate, self-harm, sexual content and cyber crimes. The labels are independent — a prompt can score high on several at once.

Each label carries its own three-level threshold triple — block, flag and log-only — in a thresholds file that ships inside the toxicity service image. The values are tuned per label rather than shared: dangerous_advice blocks at a high score because career and medical questions otherwise trip it, while cyber_crimes blocks at a much lower one. The exact numbers, and the default action each label maps to, are in the generated entity and label types reference.

Two attack labels, plus a deterministic scan

cid-attack-guard runs a second ONNX classifier with sigmoid activation over two labels — injection and jailbreak — so both can fire on one prompt. The gateway emits one detection per label that crossed its block threshold, which is why a single prompt can produce both a prompt_injection and a jailbreak row.

Beside the model, the same service exposes a deterministic scan built on libinjection, the tokeniser used by the OWASP Core Rule Set. It emits sql_injection and xss with a score of 1.0 — a fingerprint match, not a probability.

Trigger words change the confidence, not the verdict

The attack detector asks the service for the words that drove the score, then combines them with the model score into one hybrid confidence:

• Model score at or above 0.65 — the model is trusted as-is.
• Model score below 0.65 — each qualifying trigger word adds 0.05, capped at a 0.2 bonus and a 0.75 result.
• Model did not flag, but trigger words are present — at least two qualifying words are required, and the result is capped below the level a confirmed detection reaches. A single weak word yields zero, and the detection is dropped.

A detection whose hybrid confidence lands at zero never becomes a row.

Where the thresholds live

Both threshold files are copied into their service image at build time, over whatever the model shipped with. There is no environment variable that overrides either. Retuning means editing the file and rebuilding the image.

Limits and known gaps

• Eight trained languages. Both models are trained on English, Turkish, German, French, Spanish, Arabic, Italian and Dutch. A prompt in another language still reaches them and still returns a score; that score has no measured reliability.
• The decoder set is finite. ROT13, base64, leetspeak, Unicode confusables, hex/URL/HTML entities, invisible characters and letter padding are covered. An encoding outside that list reaches the detectors intact.
• Long text is not decoded at all. Above the input cap the normalisation stage is skipped, so a large document body is scanned as written.
• Attack-guard label names are not verifiable from the repository. Until label_map.json is committed or exported, an operator reading the threshold file cannot tell which row governs which label.
• The log_only tier is unused on the toxicity side. Every one of the thirteen toxicity labels ships log_only: 0.00, so that tier never fires there. The attack file does set it — both of its rows carry a non-zero log_only — so the tier is live for attack scoring and dead for toxicity.
• Accuracy is conditional. It varies with label, language and text length. Quote the qualified figures in performance and accuracy, never a bare percentage.
• A failed model call produces no detections. Both detectors log the error, report the service as degraded and return an empty list. Nothing is blocked on the strength of a detector that did not answer.

Related

• The content safety pipeline — where these detectors sit and how their verdicts resolve into one action.
• Entity and label types — every label with its thresholds and default action.
• Hallucination detection — the response-side check that runs after the reply is delivered.