| | --- |
| | license: mit |
| | datasets: |
| | - saiteja33/DAMASHA |
| | language: |
| | - en |
| | base_model: |
| | - FacebookAI/roberta-base |
| | - answerdotai/ModernBERT-base |
| | pipeline_tag: token-classification |
| | --- |
| | |
| | # DAMASHA-MAS: Mixed-Authorship Adversarial Segmentation (Token Classification) |
| |
|
| | This repository contains a **token-classification model** trained on the **DAMASHA-MAS** benchmark, introduced in: |
| |
|
| | > **DAMASHA: Detecting AI in Mixed Adversarial Texts via Segmentation with Human-interpretable Attribution** |
| |
|
| | The model aims to **segment mixed human–AI text** at *token level* – i.e., decide for each token whether it was written by a *human* or an *LLM*, even under **syntactic adversarial attacks**. |
| |
|
| | - **Base encoders:** |
| | - [`FacebookAI/roberta-base`](https://huggingface.co/FacebookAI/roberta-base) |
| | - [`answerdotai/ModernBERT-base`](https://huggingface.co/answerdotai/ModernBERT-base) |
| | - **Architecture (high level):** RoBERTa + ModernBERT feature fusion → BiGRU + CRF with the **Info-Mask** gating mechanism from the paper. |
| | - **Task:** Token classification (binary authorship: human vs AI). |
| | - **Language:** English |
| | - **License (this model):** MIT |
| | - **Training data license:** CC-BY-4.0 via the DAMASHA dataset. |
| |
|
| | If you use this model, **please also cite the DAMASHA paper and dataset** (see Citation section). |
| |
|
| | --- |
| |
|
| | ## 1. Model Highlights |
| |
|
| | - **Fine-grained mixed-authorship detection** |
| | Predicts authorship **per token**, allowing reconstruction of human vs AI **spans** in long documents. |
| |
|
| | - **Adversarially robust** |
| | Trained and evaluated on **syntactically attacked texts** (misspelling, Unicode substitutions, invisible characters, punctuation swaps, case perturbations, and “all-mixed” attacks). |
| |
|
| | - **Human-interpretable Info-Mask** |
| | The architecture incorporates **stylometric features** (perplexity, POS density, punctuation density, lexical diversity, readability) via an **Info-Mask** module that gates token representations in an interpretable way. |
| |
|
| | - **Strong reported performance (from the paper)** |
| | On DAMASHA-MAS, the **RMC\*** model (RoBERTa + ModernBERT + CRF + Info-Mask) achieves: |
| | - **Token-level**: Accuracy / Precision / Recall / F1 ≈ **0.98** |
| | - **Span-level (strict)**: SBDA ≈ **0.45**, SegPre ≈ **0.41** |
| | - **Span-level (relaxed IoU ≥ 0.5)**: ≈ **0.82** |
| |
|
| | > ⚠️ The exact numbers for *this* specific checkpoint may differ depending on training run and configuration. The values above are from the paper’s best configuration (RMC\*). |
| | |
| | --- |
| | |
| | ## 2. Intended Use |
| | |
| | ### What this model is for |
| | |
| | - **Research on human–AI co-authorship** |
| | - Studying where LLMs “take over” in mixed texts. |
| | - Analysing robustness of detectors under adversarial perturbations. |
| | |
| | - **Tooling / applications (with human oversight)** |
| | - Assisting editors, educators, or moderators to **highlight suspicious spans** rather than making final decisions. |
| | - Exploring **interpretability overlays** (e.g., heatmaps over tokens) when combined with Info-Mask outputs. |
| | |
| | ### What this model is *not* for |
| |
|
| | - Automated “cheating detector” / plagiarism court. |
| | - High-stakes decisions affecting people’s livelihood, grades, or reputation **without human review**. |
| | - Non-English or heavily code-mixed text (training data is English-centric). |
| |
|
| | Use this model as a **signal**, not a judge. |
| |
|
| | --- |
| |
|
| | ## 3. Data: DAMASHA-MAS |
| |
|
| | The model is trained on the **MAS** benchmark released with the DAMASHA paper and hosted as the Hugging Face dataset: |
| |
|
| | - **Dataset:** [`saiteja33/DAMASHA`](https://huggingface.co/datasets/saiteja33/DAMASHA) |
| |
|
| | ### 3.1 What’s in MAS? |
| |
|
| | MAS consists of **mixed human–AI texts with explicit span tags**: |
| |
|
| | - Human text comes from several corpora for **domain diversity**, including: |
| | - Reddit (M4-Reddit) |
| | - Yelp & /r/ChangeMyView (MAGE-YELP, MAGE-CMV) |
| | - News summaries (XSUM) |
| | - Wikipedia (M4-Wiki, MAGE-SQuAD) |
| | - ArXiv abstracts (MAGE-SciGen) |
| | - QA texts (MAGE-ELI5) |
| |
|
| | - AI text is generated by multiple modern LLMs: |
| | - **DeepSeek-V3-671B** (open-source) |
| | - **GPT-4o, GPT-4.1, GPT-4.1-mini** (closed-source) |
| |
|
| | ### 3.2 Span tagging |
| |
|
| | Authorship is marked using **explicit tags** around AI spans: |
| |
|
| | - `<AI_Start>` … `</AI_End>` denote AI-generated segments within otherwise human text. |
| | - The dataset stores text in a `hybrid_text` column, plus metadata such as `has_pair`, and adversarial variants include `attack_name`, `tag_count`, and `attacked_text`. |
| | - Tags are sentence-level in annotation, but the model is trained to output **token-level** predictions for finer segmentation. |
| |
|
| | > During training, these tags are converted into **token labels** (2 labels total; see `config.id2label` in the model files). |
| |
|
| | ### 3.3 Adversarial attacks |
| |
|
| | MAS includes multiple **syntactic attacks** applied to the mixed text: |
| |
|
| | - Misspelling |
| | - Unicode character substitution |
| | - Invisible characters |
| | - Punctuation substitution |
| | - Upper/lower case swapping |
| | - All-mixed combinations of the above |
| |
|
| | These perturbations make tokenization brittle and test robustness of detectors in realistic settings. |
| |
|
| | --- |
| |
|
| | ## 4. Model Architecture & Training |
| |
|
| | ### 4.1 Architecture (conceptual) |
| |
|
| | The model follows the **Info-Mask RMC\*** architecture described in the DAMASHA paper: |
| |
|
| | 1. **Dual encoders** |
| | - RoBERTa-base and ModernBERT-base encode the same input sequence. |
| | 2. **Feature fusion** |
| | - Hidden states from both encoders are fused into a shared representation. |
| | 3. **Stylometric Info-Mask** |
| | - Hand-crafted style features (perplexity, POS density, punctuation density, lexical diversity, readability) are projected, passed through multi-head attention, and turned into a **scalar mask per token**. |
| | - This mask gates the fused encoder states, down-weighting style-irrelevant tokens and emphasizing style-diagnostic ones. :contentReference[oaicite:16]{index=16} |
| | 4. **Sequence model + CRF** |
| | - A BiGRU layer captures sequential dependencies, followed by a **CRF** layer for structured token labeling with a sequence-level loss. :contentReference[oaicite:17]{index=17} |
| |
|
| | ### 4.2 Training setup (from the paper) |
| |
|
| | Key hyperparameters used for the Info-Mask models on MAS: |
| |
|
| | - **Number of labels:** 2 |
| | - **Max sequence length:** 512 |
| | - **Batch size:** 64 |
| | - **Epochs:** 5 |
| | - **Optimizer:** AdamW (with cosine annealing LR schedule) |
| | - **Weight decay:** 0.01 |
| | - **Gradient clipping:** 1.0 |
| | - **Dropout:** Dynamic 0.1–0.3 (initial 0.1) |
| | - **Warmup ratio:** 0.1 |
| | - **Early stopping patience:** 2 |
| |
|
| | **Hardware & compute** (as reported): |
| |
|
| | - AWS EC2 g6e.xlarge, NVIDIA L40S (48GB) GPU, Ubuntu 24.04 |
| | - ≈ 400 GPU hours for experiments. |
| |
|
| | > The exact training script used for this checkpoint is available in the project GitHub: |
| | > <https://github.com/saitejalekkala33/DAMASHA> |
| |
|
| | --- |
| |
|
| | --- |
| | license: mit |
| | --- |
