README.md

# VLAlert — Code & Models

Source code for **VLAlert**, a vision-language driver-alerting framework that
produces structured per-frame safety `<|BELIEF|>` tokens from dashcam video and
maps them to three alert actions: **SILENT / OBSERVE / ALERT**.

This repository contains the **training and evaluation code** for all model
variants. Model weights / checkpoints are **not** included. The benchmark data
and experimental results are hosted separately at
[`AsianPlayer/VLAlert-Bench`](https://huggingface.co/datasets/AsianPlayer/VLAlert-Bench).

## Architecture

```
8 dashcam frames
      │
      ▼
Qwen3-VL-4B + LoRA  ──►  [Analysis] reasoning  +  [Safety Assessment]
                          <|BELIEF|> ... </|BELIEF|> <|ACTION|>  (per frame)
      │
      ├─ belief span (mean-pool layers {20,24,28,32}) → z_t ∈ ℝ^10240 ─► DangerHead (14.8M)
      └─ close-tag hidden state (layer 33)            → r_t ∈ ℝ^2560  ─► PolicyHead (7.0M)
                                                                            │
                                            a_{t-1} feedback ◄──── FSM Decoder ──► Action a_t
```

## Repository Structure

```
lkalert/
  models/                  # model architectures
    danger_head.py         #   per-frame + clip danger regressor (PMA aggregator)
    policy_head_v2.py      #   GRU 3-class policy head (SILENT/OBSERVE/ALERT)
    adaptive_window.py     #   adaptive temporal-window selection (VLAlert-X)
    components.py          #   MultiQueryPMA aggregator, legacy heads
    belief_vlm.py          #   integrated VLM + belief/action heads
    multichannel_belief.py #   LKAlert-MCB gated multi-channel fusion
    lora.py                #   LoRA implementation
  utils/, data/            # core library

training/
  VLA/                     # belief-token SFT on Qwen3-VL-4B
    train_cot_belief_v2.py #   v2 SFT (belief + action per frame)
    train_vlalert_sft_v3.py#   v3 SFT (reasoning → belief, embedding loss option)
    cot_belief_dataset_v2.py
  Policy/                  # downstream head training
    train_danger_head.py   #   DangerHead (5-seed)
    train_policy_head_v2.py#   PolicyHead (5-seed)
    train_vlalert_x.py     #   VLAlert-X adaptive-window end-to-end
    train_head_dpo.py      #   DPO preference fine-tuning
    train_head_kto.py      #   KTO fine-tuning
    train_head_ppo.py      #   PPO fine-tuning
  SFT/                     # Qwen2.5-VL-3B monolithic SFT (VLAlert-2.5)
  DPO/                     # preference-pair training
  pretrain*/               # 2-stage vision-language pretraining
  Nexar/                   # CNN baselines (ResNet50-LSTM, R3D-18, MViT-V2-S)

tools/
  # data preparation
  relabel_dada_nexar.py    # action labels via risky_time + 2s rule
  relabel_dota_corpus.py   # BADAS-gated OBSERVE labels
  generate_beliefs.py      # rule-based belief content
  run_v1_gpt5_cot.py       # GPT-4o belief generation
  build_v5_benchmark.py    # unified benchmark builder
  # belief cache extraction
  make_cache_x_v2.py       # dual-stream cache (belief_content + policy_position)
  run_qwen3_cache_fast.py  # cache extraction with Conv3d→Linear patch
  # evaluation
  demo_compare_pipeline.py # multi-model demo scoring + visualization
  score_*.py, compute_daus_v6.py
  # figures
  render_modelarchi_v4.py, render_belief_span.py

PATCH_conv3d_linear.md     # Conv3d→Linear acceleration (64× on Blackwell GPUs)
requirements.txt
```

## The Conv3d → Linear Patch

`PATCH_conv3d_linear.md` documents a 64× end-to-end speedup of Qwen3-VL vision
patch embedding on Blackwell GPUs (RTX 5090), by replacing the degenerate
`nn.Conv3d(kernel=stride)` patchification with a mathematically equivalent
`nn.Linear`. This makes large-scale belief-cache extraction feasible
(6 days → ~2 hours). Equivalence is proven and verified
(`tools/verify_patch_embed_correctness.py`).

## Reproduction

1. Prepare benchmark annotations from
   [`AsianPlayer/VLAlert-Bench`](https://huggingface.co/datasets/AsianPlayer/VLAlert-Bench).
2. **Stage 1 — SFT**: `training/VLA/train_vlalert_sft_v3.py`
3. **Stage 2 — cache extraction**: `tools/make_cache_x_v2.py`
4. **Stage 3 — heads**: `training/Policy/train_danger_head.py`, `train_policy_head_v2.py`
5. **Evaluation**: `tools/score_*.py`, `tools/compute_daus_v6.py`

Paths in scripts use `PROJECT_ROOT` as a placeholder for the repository root.

## License

Code released for research review. The benchmark builds on Nexar, DADA-2000,
DoTA, and DAD source datasets; see the dataset repository for source licenses
and citations.