Add comprehensive EmObserver model card

README.md

@@ -0,0 +1,313 @@
+---
+license: apache-2.0
+base_model:
+- Qwen/Qwen3-VL-8B-Thinking
+pipeline_tag: image-text-to-text
+library_name: transformers
+language:
+- en
+tags:
+- qwen3-vl
+- vision-language
+- multimodal
+- visual-emotion
+- affective-computing
+- emotional-intelligence
+- reasoning
+datasets:
+- wudq/MVEI_PLUS
+model-index:
+- name: EmObserver
+  results:
+  - task:
+      type: image-text-to-text
+      name: Emotion Statement Judgement
+    dataset:
+      type: wudq/MVEI_PLUS
+      name: MVEI
+      split: test
+    metrics:
+    - type: accuracy
+      name: Accuracy
+      value: 86.23
+  - task:
+      type: image-text-to-text
+      name: Visual Emotion Classification
+    dataset:
+      type: wudq/MVEI_PLUS
+      name: VECBench
+      split: test
+    metrics:
+    - type: accuracy
+      name: Overall accuracy
+      value: 63.42
+  - task:
+      type: image-text-to-text
+      name: Single-image Emotion Perception
+    dataset:
+      type: wudq/MVEI_PLUS
+      name: EEmo-Bench Single Perception
+      split: test
+    metrics:
+    - type: accuracy
+      name: Overall accuracy
+      value: 71.94
+  - task:
+      type: image-text-to-text
+      name: Image-pair Emotion Perception
+    dataset:
+      type: wudq/MVEI_PLUS
+      name: EEmo-Bench Pair Perception
+      split: test
+    metrics:
+    - type: accuracy
+      name: Overall accuracy
+      value: 71.47
+---
+
+# EmObserver
+
+Official model release for **“MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement.”**
+
+EmObserver is an emotion-oriented multimodal large language model built from [Qwen3-VL-8B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking). It is optimized through a four-stage training recipe for visual emotion understanding, Emotion Statement Judgement (ESJ), emotion classification, and single-/multi-image affective reasoning.
+
+The model accepts one or more images with a natural-language question and produces a reasoning-style response followed by a concise answer:
+
+```text
+<think>...</think><answer>...</answer>
+```
+
+## 🔗 Project map
+
+| Resource | Description | Link |
+| --- | --- | --- |
+| Paper | MVEI & EmObserver | [arXiv:2607.21061](https://arxiv.org/abs/2607.21061) |
+| Official code | Inference, evaluation, and four-stage training | [wdqqdw/EmObserver](https://github.com/wdqqdw/EmObserver) |
+| Expanded release | Benchmarks, images, predictions, and training data | [wudq/MVEI_PLUS](https://huggingface.co/datasets/wudq/MVEI_PLUS) |
+| EmObserver results | Released predictions and metric summaries | [MVEI_PLUS/baselines/Qwen3-VL-EMOBSERVER](https://huggingface.co/datasets/wudq/MVEI_PLUS/tree/main/baselines/Qwen3-VL-EMOBSERVER) |
+| Evaluation data | Standardized MVEI, EEmo-Bench, and VECBench metadata | [MVEI_PLUS/benchmarks](https://huggingface.co/datasets/wudq/MVEI_PLUS/tree/main/benchmarks) |
+| Training data | Four-stage data and GPT-5.5-filtered INSETS-462k | [MVEI_PLUS/training_data](https://huggingface.co/datasets/wudq/MVEI_PLUS/tree/main/training_data) |
+| Original MVEI dataset | Original benchmark release | [wudq/MVEI](https://huggingface.co/datasets/wudq/MVEI) |
+| Original INSETS-462k | Original training-data release | [wudq/INSETS-462k](https://huggingface.co/datasets/wudq/INSETS-462k) |
+| Original conference code | Earlier MVEI codebase and project release | [wdqqdw/MVEI](https://github.com/wdqqdw/MVEI) |
+| Base model | Qwen3-VL-8B-Thinking | [Qwen/Qwen3-VL-8B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking) |
+
+## 🚀 Quick start with Transformers
+
+Install a compatible PyTorch build first, then install the model dependencies:
+
+```bash
+python3 -m pip install -U "transformers>=5.9.0" accelerate pillow
+```
+
+The following example uses the same MVEI question form as the released test metadata:
+
+```python
+import torch
+from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
+
+model_id = "wudq/EmObserver"
+
+model = Qwen3VLForConditionalGeneration.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto",
+)
+processor = AutoProcessor.from_pretrained(model_id)
+
+image_url = (
+    "https://huggingface.co/datasets/wudq/MVEI_PLUS/resolve/main/"
+    "benchmarks/MVEI/images/contentment/contentment_14236.jpg"
+)
+prompt = (
+    "Is the following statement correct about the image? Upon viewing this image, "
+    "observers, despite various individual or contextual factors, are most likely "
+    "to experience negative emotions. Choose the answer from "
+    "{'A': 'Correct.', 'B': 'Incorrect.'}. Answer in the format of "
+    "<think>...</think><answer>...</answer>."
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": image_url},
+            {"type": "text", "text": prompt},
+        ],
+    }
+]
+
+inputs = processor.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_dict=True,
+    return_tensors="pt",
+).to(model.device)
+
+generated = model.generate(**inputs, max_new_tokens=1024)
+trimmed = [output[len(input_ids):] for input_ids, output in zip(inputs.input_ids, generated)]
+response = processor.batch_decode(
+    trimmed,
+    skip_special_tokens=True,
+    clean_up_tokenization_spaces=False,
+)[0]
+print(response)
+```
+
+For multi-image questions, add each image to `content` before the text item:
+
+```python
+"content": [
+    {"type": "image", "url": first_image_url},
+    {"type": "image", "url": second_image_url},
+    {"type": "text", "text": pair_prompt},
+]
+```
+
+FlashAttention 2 can reduce memory use on supported hardware. Install a build compatible with your CUDA/PyTorch stack, then pass `attn_implementation="flash_attention_2"` to `from_pretrained`.
+
+## 🧪 Full evaluation with the official code
+
+The official repository provides the vLLM inference wrapper, benchmark loaders, output parsing, and metric calculators:
+
+```bash
+git clone https://github.com/wdqqdw/EmObserver.git
+cd EmObserver
+
+python3 -m pip install -r requirements.txt
+hf download wudq/MVEI_PLUS --repo-type dataset --local-dir public_data
+hf download wudq/EmObserver --local-dir models/EmObserver
+
+python3 -m evaluate.infer_and_eval \
+  --engine qwen3_vl_vllm \
+  --size emobserver \
+  --gpu_n 8
+```
+
+Adjust `--gpu_n` to the visible GPUs that can hold the checkpoint. Use `--bench` to select a subset from `MVEI`, `VECBench`, `EEmo-Bench-Single-Perception`, and `EEmo-Bench-Pair-Perception`.
+
+## 💬 Benchmark-style examples
+
+The examples below reproduce question forms from the standardized test metadata. The listed answer is the reference final answer; reasoning text may vary.
+
+### 1. Emotion Statement Judgement — MVEI
+
+**Image:** [contentment_14236.jpg](https://huggingface.co/datasets/wudq/MVEI_PLUS/resolve/main/benchmarks/MVEI/images/contentment/contentment_14236.jpg)
+
+```text
+Is the following statement correct about the image? Upon viewing this image,
+observers, despite various individual or contextual factors, are most likely
+to experience negative emotions. Choose the answer from
+{'A': 'Correct.', 'B': 'Incorrect.'}. Answer in the format of
+<think>...</think><answer>...</answer>.
+```
+
+Reference final answer:
+
+```text
+<answer>Incorrect.</answer>
+```
+
+### 2. Single-image emotion perception — EEmo-Bench
+
+**Image:** [EEmo-Bench single image 1](https://huggingface.co/datasets/wudq/MVEI_PLUS/resolve/main/benchmarks/EEmo-Bench/EEmo-Bench_single/images/1.jpg)
+
+```text
+What do you think of surprise as one of the three main emotions you felt from
+this image? Choose the answer from {'A': 'No', 'B': 'Yes'}. Answer in the
+format of <think>...</think><answer>...</answer>.
+```
+
+Reference final answer:
+
+```text
+<answer>Yes.</answer>
+```
+
+### 3. Image-pair emotion comparison — EEmo-Bench
+
+**Images:** [image 579](https://huggingface.co/datasets/wudq/MVEI_PLUS/resolve/main/benchmarks/EEmo-Bench/EEmo-Bench_pair/images/579.jpg) and [image 100](https://huggingface.co/datasets/wudq/MVEI_PLUS/resolve/main/benchmarks/EEmo-Bench/EEmo-Bench_pair/images/100.jpg)
+
+```text
+Which emotion is evoked in both images and is common to them? Choose the
+answer from {'A': 'Neutral', 'B': 'Fear', 'C': 'Sadness', 'D': 'Joy'}.
+Answer in the format of <think>...</think><answer>...</answer>.
+```
+
+Reference final answer:
+
+```text
+<answer>Joy.</answer>
+```
+
+### 4. Visual emotion classification — VECBench
+
+**Image:** [abstract_0001.jpg](https://huggingface.co/datasets/wudq/MVEI_PLUS/resolve/main/benchmarks/VECBench/images/Abstract/testImages_abstract/abstract_0001.jpg)
+
+```text
+Which emotion might this image evoke? Choose the most likely one from
+['Amusement', 'Anger', 'Awe', 'Content', 'Disgust', 'Excitement', 'Fear',
+'Sad']. Think step by step. Respond in the format:
+<think>{your reasoning}</think><answer>{your final answer}</answer>.
+```
+
+Reference final answer:
+
+```text
+<answer>Content.</answer>
+```
+
+## 📊 Released evaluation results
+
+These values are taken from the released EmObserver inference outputs in [MVEI_PLUS](https://huggingface.co/datasets/wudq/MVEI_PLUS/tree/main/baselines/Qwen3-VL-EMOBSERVER). They are intended to make the checkpoint release traceable; environment or decoding changes may produce different results.
+
+| Benchmark | Metric | Result |
+| --- | --- | ---: |
+| MVEI | Overall accuracy | 86.23% |
+| MVEI | Sentiment polarity | 86.47% |
+| MVEI | Emotion interpretation | 83.55% |
+| MVEI | Scene context | 90.30% |
+| MVEI | Perception subjectivity | 86.06% |
+| VECBench | Overall accuracy | 63.42% |
+| EEmo-Bench Single | Overall accuracy | 71.94% |
+| EEmo-Bench Pair | Overall accuracy | 71.47% |
+
+## 🏋️ Training overview
+
+EmObserver is produced by the four-stage public training pipeline:
+
+1. supervised fine-tuning;
+2. GRPO-based reinforcement learning;
+3. on-policy self-distillation (OPSD);
+4. GRPO with an LLM-consistency reward.
+
+The runnable scripts are in the [official code repository](https://github.com/wdqqdw/EmObserver/tree/main/training), and their released inputs are under [MVEI_PLUS/training_data](https://huggingface.co/datasets/wudq/MVEI_PLUS/tree/main/training_data). External judge credentials are not included; users must provide their own endpoint when reproducing the final stage.
+
+## ⚠️ Intended use and limitations
+
+EmObserver is intended for research on visual emotion understanding, affective image analysis, multimodal reasoning, and benchmark development.
+
+- Visual emotion is inherently subjective and culturally/contextually dependent. A model prediction should not be treated as a universal description of how every person will feel.
+- The model may inherit social, cultural, demographic, and content biases from its base model and training data.
+- The checkpoint is not designed for mental-health diagnosis, psychological profiling, surveillance, hiring, education assessment, or other high-stakes decisions about individuals.
+- Generated reasoning can be plausible but factually or emotionally incorrect. Verify outputs when reliability matters.
+- Users are responsible for reviewing the licenses, attribution requirements, privacy constraints, and redistribution terms of input images and downstream datasets.
+
+## 📄 License
+
+The checkpoint is released under the Apache 2.0 license, following its [Qwen3-VL-8B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking) base model. Third-party datasets and images retain their respective licenses and terms.
+
+## 📝 Citation
+
+If you use EmObserver, MVEI, or the expanded release, please cite:
+
+```bibtex
+@article{wu2026mvei_emobserver,
+  title   = {MVEI \& EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement},
+  author  = {Wu, Daiqing and Yang, Dongbao and Yao, Jiashu and Zhang, Hongrui and Ma, Can and Zhou, Yu and Zhao, Sicheng},
+  journal = {arXiv preprint arXiv:2607.21061},
+  year    = {2026}
+}
+```
+