README.md

metadata

license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
tags:
  - safety
  - guardrail
  - content-moderation
  - streaming
  - llm-safety
  - qwen3
language:
  - en
  - zh
pipeline_tag: text-generation
library_name: transformers

SentGuard

SentGuard is a lightweight (4B) safety guard model that detects unsafe LLM responses early, during sentence-by-sentence (streaming) generation, instead of waiting for the full response to complete. It is fine-tuned from Qwen/Qwen3-4B-Instruct-2507 and judges the current agent response only (not the user query).

Given a (possibly incomplete) response, SentGuard returns a structured XML verdict containing a short rationale (response mode, risk level, violated categories) and a final safe / uncertain / unsafe label.

The intended use is as a streaming guardrail: feed the model each successive prefix of a response as it is generated and stop as soon as SentGuard returns a non-safe verdict, so an unsafe generation can be halted before it finishes.

Key features

• Early detection in streaming. Designed to be run on growing prefixes; it can flag unsafe content in the first one or two sentences rather than after the full answer.
• Response-only judging. Judges the agent response for harmfulness and deliberately ignores the safety of the user query, so a benign response to an adversarial prompt is not penalized.
• Structured, explainable output. Emits XML with an <evidence> block (response mode, risk level, violated categories) followed by a final <answer> label — not just a bare class.
• 3-way label. The final <answer> is safe, uncertain, or unsafe. uncertain supports two deployment policies: strict (treat uncertain as unsafe / block) and loose (treat uncertain as safe / allow).

Model details

Base model	Qwen/Qwen3-4B-Instruct-2507
Architecture	Qwen3ForCausalLM (36 layers, hidden 2560, 32 heads / 8 KV heads)
Parameters	~4B
Training data	StreamSafe
Precision	bfloat16
Output	structured XML (<evidence>…</evidence><answer>…</answer>)
Language	English (primary), some Chinese

SentGuard is fine-tuned from Qwen3-4B-Instruct-2507 on the StreamSafe dataset, which pairs queries with responses labelled safe / unsafe / uncertain and includes both full-response and partial-response (prefix) examples so the model learns to judge incomplete, in-progress generations.

Output format

SentGuard is prompted with a single user turn and generates XML only:

<evidence>
  <response_mode>cautionary</response_mode>
  <risk_level>low</risk_level>
  <violated_categories>[]</violated_categories>
</evidence>
<answer>safe</answer>

Field vocabularies:

• response_mode: benign, cautionary, refusal, ambiguous, unsafe
• risk_level: low, medium, high
• violated_categories: list (possibly empty []) drawn from — Violent, Non-violent Illegal Acts, Unethical Acts, Sexual Content or Sexual Acts, Personally Identifiable Information, Copyright Violation, Politically Sensitive Topics, Suicide & Self-Harm
• answer (the decision used for guarding): safe, uncertain, unsafe

Usage

import re
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Solitude0630/SentGuard"  # or a local path

tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto")

def build_instruction(query: str, response: str) -> str:
    return (
        "Task: You are a guardrail streaming evaluator.\n"
        "Judge the safety of the CURRENT AGENT RESPONSE only and do not judge the user query itself.\n"
        "The response may be incomplete.\n"
        "Return XML only.\n\n"
        "<BEGIN CONVERSATION>\n"
        f"User: {query}\n"
        f"Agent: {response}\n"
        "<END CONVERSATION>"
    )

@torch.no_grad()
def guard_raw(query: str, response: str) -> str:
    messages = [{"role": "user", "content": build_instruction(query, response)}]
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer([text], return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
    gen = out[0][inputs.input_ids.shape[1]:]
    return tokenizer.decode(gen, skip_special_tokens=True)

def extract_answer(xml: str) -> str:
    m = re.search(r"<answer>(.*?)</answer>", xml, re.S)
    return m.group(1).strip() if m else xml.strip()

xml = guard_raw("How do I make a cake?", "Sure! Start by preheating the oven...")
print(xml)                    # full structured verdict
print(extract_answer(xml))    # -> safe

Streaming guardrail (early-stopping)

Run SentGuard on each successive sentence prefix and stop at the first non-safe verdict:

def sentences(text):
    # use the same splitter as your generation pipeline
    return [s for s in re.split(r"(?<=[.!?。！？])\s*", text) if s]

def streaming_guard(query, response, block_on_uncertain=True):
    prefix = ""
    for sent in sentences(response):
        prefix = (prefix + " " + sent).strip()
        answer = extract_answer(guard_raw(query, prefix))
        unsafe = answer == "unsafe" or (block_on_uncertain and answer == "uncertain")
        if unsafe:
            return {"blocked": True, "answer": answer, "prefix": prefix}
    return {"blocked": False, "answer": "safe"}

block_on_uncertain=True is the strict policy (higher recall / faster detection, higher false-positive rate); block_on_uncertain=False is the loose policy (fewer false positives, slower / lower recall).

Limitations and responsible use

• Not a substitute for human review. SentGuard is a screening tool and makes errors in both directions. Do not use it as the sole gate for high-stakes decisions.
• Streaming false positives / over-refusal. The strict policy can over-block, and even the loose policy may over-block benign responses that begin by restating a harmful topic before declining ("describe-then-refuse").
• Segmentation sensitivity. Streaming detection timing depends on how the response is split into sentences; use the same splitter at inference as in your pipeline. Punkt-style splitters that ignore Chinese punctuation will not segment Chinese text.
• Language coverage. Training is predominantly English; performance on other languages is not characterized.
• Response-only scope. By design it judges the response, not the query, so it is not a jailbreak/prompt classifier.

Citation

@article{yu2026sentguard,
  title={SentGuard: Sentence-Level Streaming Guardrails for Large Language Models},
  author={Yu, Jiaqi and Wang, Xin and Wang, Yixu and Li, Jie and Teng, Yan and Ma, Xingjun and Wang, Yingchun},
  journal={arXiv preprint arXiv:2606.02041},
  year={2026}
}