| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-4B-Instruct-2507 |
| tags: |
| - safety |
| - guardrail |
| - content-moderation |
| - streaming |
| - llm-safety |
| - qwen3 |
| language: |
| - en |
| - zh |
| pipeline_tag: text-generation |
| library_name: transformers |
| --- |
| |
| # SentGuard |
|
|
| **SentGuard** is a lightweight (4B) safety guard model that detects unsafe LLM responses **early**, during |
| sentence-by-sentence (streaming) generation, instead of waiting for the full response to complete. It is |
| fine-tuned from [`Qwen/Qwen3-4B-Instruct-2507`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) and |
| judges the **current agent response only** (not the user query). |
|
|
| Given a (possibly incomplete) response, SentGuard returns a **structured XML verdict** containing a short |
| rationale (response mode, risk level, violated categories) and a final `safe` / `uncertain` / `unsafe` label. |
|
|
| The intended use is as a *streaming guardrail*: feed the model each successive prefix of a response as it is |
| generated and stop as soon as SentGuard returns a non-safe verdict, so an unsafe generation can be halted |
| before it finishes. |
|
|
| ## Key features |
|
|
| - **Early detection in streaming.** Designed to be run on growing prefixes; it can flag unsafe content in the |
| first one or two sentences rather than after the full answer. |
| - **Response-only judging.** Judges the *agent response* for harmfulness and deliberately ignores the safety |
| of the user query, so a benign response to an adversarial prompt is not penalized. |
| - **Structured, explainable output.** Emits XML with an `<evidence>` block (response mode, risk level, |
| violated categories) followed by a final `<answer>` label — not just a bare class. |
| - **3-way label.** The final `<answer>` is `safe`, `uncertain`, or `unsafe`. `uncertain` supports two |
| deployment policies: **strict** (treat `uncertain` as unsafe / block) and **loose** (treat `uncertain` as |
| safe / allow). |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | Base model | Qwen/Qwen3-4B-Instruct-2507 | |
| | Architecture | Qwen3ForCausalLM (36 layers, hidden 2560, 32 heads / 8 KV heads) | |
| | Parameters | ~4B | |
| | Training data | StreamSafe | |
| | Precision | bfloat16 | |
| | Output | structured XML (`<evidence>…</evidence><answer>…</answer>`) | |
| | Language | English (primary), some Chinese | |
|
|
| SentGuard is fine-tuned from Qwen3-4B-Instruct-2507 on the **StreamSafe** dataset, which pairs queries with |
| responses labelled `safe` / `unsafe` / `uncertain` and includes both full-response and partial-response |
| (prefix) examples so the model learns to judge incomplete, in-progress generations. |
|
|
| ## Output format |
|
|
| SentGuard is prompted with a single user turn and generates XML only: |
|
|
| ```xml |
| <evidence> |
| <response_mode>cautionary</response_mode> |
| <risk_level>low</risk_level> |
| <violated_categories>[]</violated_categories> |
| </evidence> |
| <answer>safe</answer> |
| ``` |
|
|
| Field vocabularies: |
|
|
| - `response_mode`: `benign`, `cautionary`, `refusal`, `ambiguous`, `unsafe` |
| - `risk_level`: `low`, `medium`, `high` |
| - `violated_categories`: list (possibly empty `[]`) drawn from — `Violent`, `Non-violent Illegal Acts`, |
| `Unethical Acts`, `Sexual Content or Sexual Acts`, `Personally Identifiable Information`, |
| `Copyright Violation`, `Politically Sensitive Topics`, `Suicide & Self-Harm` |
| - `answer` (the decision used for guarding): `safe`, `uncertain`, `unsafe` |
|
|
| ## Usage |
|
|
| ```python |
| import re |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| MODEL = "Solitude0630/SentGuard" # or a local path |
| |
| tokenizer = AutoTokenizer.from_pretrained(MODEL) |
| model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto") |
| |
| def build_instruction(query: str, response: str) -> str: |
| return ( |
| "Task: You are a guardrail streaming evaluator.\n" |
| "Judge the safety of the CURRENT AGENT RESPONSE only and do not judge the user query itself.\n" |
| "The response may be incomplete.\n" |
| "Return XML only.\n\n" |
| "<BEGIN CONVERSATION>\n" |
| f"User: {query}\n" |
| f"Agent: {response}\n" |
| "<END CONVERSATION>" |
| ) |
| |
| @torch.no_grad() |
| def guard_raw(query: str, response: str) -> str: |
| messages = [{"role": "user", "content": build_instruction(query, response)}] |
| text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer([text], return_tensors="pt").to(model.device) |
| out = model.generate(**inputs, max_new_tokens=512, do_sample=False) |
| gen = out[0][inputs.input_ids.shape[1]:] |
| return tokenizer.decode(gen, skip_special_tokens=True) |
| |
| def extract_answer(xml: str) -> str: |
| m = re.search(r"<answer>(.*?)</answer>", xml, re.S) |
| return m.group(1).strip() if m else xml.strip() |
| |
| xml = guard_raw("How do I make a cake?", "Sure! Start by preheating the oven...") |
| print(xml) # full structured verdict |
| print(extract_answer(xml)) # -> safe |
| ``` |
|
|
| ### Streaming guardrail (early-stopping) |
|
|
| Run SentGuard on each successive sentence prefix and stop at the first non-safe verdict: |
|
|
| ```python |
| def sentences(text): |
| # use the same splitter as your generation pipeline |
| return [s for s in re.split(r"(?<=[.!?。！？])\s*", text) if s] |
| |
| def streaming_guard(query, response, block_on_uncertain=True): |
| prefix = "" |
| for sent in sentences(response): |
| prefix = (prefix + " " + sent).strip() |
| answer = extract_answer(guard_raw(query, prefix)) |
| unsafe = answer == "unsafe" or (block_on_uncertain and answer == "uncertain") |
| if unsafe: |
| return {"blocked": True, "answer": answer, "prefix": prefix} |
| return {"blocked": False, "answer": "safe"} |
| ``` |
|
|
| `block_on_uncertain=True` is the **strict** policy (higher recall / faster detection, higher false-positive |
| rate); `block_on_uncertain=False` is the **loose** policy (fewer false positives, slower / lower recall). |
|
|
| ## Limitations and responsible use |
|
|
| - **Not a substitute for human review.** SentGuard is a screening tool and makes errors in both directions. |
| Do not use it as the sole gate for high-stakes decisions. |
| - **Streaming false positives / over-refusal.** The strict policy can over-block, and even the loose policy |
| may over-block benign responses that begin by restating a harmful topic before declining |
| ("describe-then-refuse"). |
| - **Segmentation sensitivity.** Streaming detection timing depends on how the response is split into |
| sentences; use the same splitter at inference as in your pipeline. Punkt-style splitters that ignore Chinese |
| punctuation will not segment Chinese text. |
| - **Language coverage.** Training is predominantly English; performance on other languages is not |
| characterized. |
| - **Response-only scope.** By design it judges the response, not the query, so it is not a jailbreak/prompt |
| classifier. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{yu2026sentguard, |
| title={SentGuard: Sentence-Level Streaming Guardrails for Large Language Models}, |
| author={Yu, Jiaqi and Wang, Xin and Wang, Yixu and Li, Jie and Teng, Yan and Ma, Xingjun and Wang, Yingchun}, |
| journal={arXiv preprint arXiv:2606.02041}, |
| year={2026} |
| } |
| ``` |
|
|
