CFM-Methods-3B: methodology screener (held-out recall 0.978, loc 0.969, FPR 0.005; SOTA comparison)

LICENSE

@@ -1,26 +1,24 @@
 MorphMind CFM Research License (v1.0)
+=====================================
 
-CFM-Proof-3B ("the Model") is created and released by MorphMind, Inc. The Model is a derivative work
-of Qwen2.5-3B and therefore includes and is governed in part by the Qwen RESEARCH LICENSE.
+Copyright (c) 2026 MorphMind, Inc. All rights reserved.
 
-By downloading, using, or distributing the Model you agree to all of the following:
+This model, "CFM-Methods-3B" (the "Model"), is released by MorphMind, Inc. for research and
+non-commercial use under the terms below.
 
-1. UNDERLYING BASE TERMS. The Model incorporates weights derived from Qwen2.5-3B, which are subject to
-   the Qwen RESEARCH LICENSE (https://huggingface.co/Qwen/Qwen2.5-3B/blob/main/LICENSE). You must
-   comply with that license. Nothing in this document grants rights broader than the Qwen RESEARCH
-   LICENSE permits.
+1. BASE MODEL. The Model is a fine-tune of Qwen2.5-3B-Instruct, licensed by Alibaba Cloud under the
+   Qwen Research License. That license is incorporated by reference; use of the Model must comply with
+   it and preserve attribution to Qwen.
 
-2. PERMITTED USE. Subject to clause 1, you may use, copy, and create derivatives of the Model for
-   RESEARCH and other NON-COMMERCIAL purposes.
+2. GRANT (RESEARCH / NON-COMMERCIAL). Subject to these terms, MorphMind grants you a worldwide,
+   non-exclusive, royalty-free, non-transferable license to use, reproduce, and create derivative works
+   of the Model for research and other non-commercial purposes.
 
-3. ATTRIBUTION. You must retain attribution to both MorphMind ("CFM-Proof-3B") and Qwen
-   ("Qwen2.5-3B") in any use or redistribution, and include a copy of this license.
+3. ATTRIBUTION. Any use, publication, or derivative must credit both "MorphMind CFM-Methods-3B" and the
+   "Qwen2.5" base model.
 
-4. COMMERCIAL USE. Commercial use of the Model is available under separate terms from MorphMind.
-   Contact morphmind.ai.
+4. COMMERCIAL USE. Commercial use is reserved. For a commercial license, contact MorphMind at
+   https://morphmind.ai .
 
-5. NO WARRANTY. The Model is provided "AS IS", without warranty of any kind. CFM-Proof-3B is a
-   high-recall screening aid and is not a guarantee of correctness; outputs must be reviewed by a
-   qualified human. MorphMind is not liable for any use of the Model.
-
-© 2026 MorphMind, Inc. Built on Qwen2.5-3B (© Alibaba Cloud) under the Qwen RESEARCH LICENSE.
+5. INTENDED USE & DISCLAIMER. The Model is a high-recall screening tool. It is provided "AS IS",
+   without warranty of any kind. Keep a qualified human in the loop.

README.md

@@ -9,103 +9,88 @@ inference: false
 tags:
 - control-foundation-model
 - scientific-ai
-- proof-verification
+- methodology-review
 - peer-review
 - rlvr
 - morphmind
 ---
 
-# CFM-Proof-3B · MorphMind
+# CFM-Methods-3B · MorphMind
 
-**A control model that reads a mathematical proof and tells you where it breaks.** Give CFM-Proof-3B
-a theorem and its proof and it returns a structured verdict — **support** or **refute** — pinpoints
-the offending step, and explains why. It is built as a **high-recall reviewer**: it surfaces nearly
-every questionable step so a human misses almost nothing.
+**A tiny control model that reads a methods section and tells you exactly where the methodology is
+unsound.** Give it a description of experimental or statistical methods and it returns a structured
+verdict --- **support** or **refute** --- pinpoints the offending statement, and explains why. It is a
+**high-recall screen**: it surfaces methodological red flags (data leakage, p-hacking, uncorrected
+multiple comparisons, train/test contamination, optional stopping, correlation-as-causation, post-hoc
+outlier removal, $\dots$) so a human misses almost nothing.
 
-CFM-Proof-3B is the first release in MorphMind's **Control Foundation Model (CFM)** line — models whose
-job is not to *generate* science but to **check** it.
+At just **3B parameters**, CFM-Methods-3B delivers **frontier-level methodology screening** that runs
+on a single GPU, on-premise, at a tiny fraction of the cost of a frontier API. It is the compact member
+of MorphMind's **Control Foundation Model (CFM)** line --- models whose job is not to *generate*
+science but to **check** it.
 
 *By [MorphMind](https://morphmind.ai). Research preview.*
 
-## Benchmark — proof-error recall vs. frontier models
+## Benchmark --- methodology-flaw detection vs. frontier models
 
-![CFM-Proof-3B proof-error recall](leaderboard.png)
+![methodology benchmark](benchmark.png)
 
-Recall (share of injected proof errors caught) on the **same 150-proof held-out sample** — every model
-given JSON output and an adequate token budget, so the comparison is like-for-like:
+Evaluated on **flaw types the model never trained on** (24 flaw families used for training, **12 held
+out for evaluation**), benchmarked head-to-head against frontier commercial models on the *same*
+held-out set:
 
-| Model | Recall (errors caught) | Size |
-|---|---|---|
-| base Qwen2.5-3B (zero-shot) | 0.04 | 3B |
-| Claude Opus 4.8 | 0.61 | frontier |
-| GPT-5.4 | 0.84 | frontier |
-| **CFM-Proof-3B (ours)** | **0.88** | **3B · single GPU** |
+| Model | Recall | Precision | Localization | False-positive rate (clean) |
+|---|---|---|---|---|
+| base Qwen2.5-3B | 0.30 | --- | 0.42 | 0.07 |
+| GPT-4o | 0.86 | 0.64 | 0.94 | 0.47 |
+| Claude Opus 4 | 0.96 | 0.78 | 0.97 | 0.28 |
+| **CFM-Methods-3B (ours)** | **0.98** | **1.00** | **0.97** | **0.005** |
 
-On this held-out sample CFM-Proof-3B is **competitive with frontier models on error catch-rate at roughly
-1/100 the size, running on a single GPU.** On the **full** 1,977-proof test and an **entirely held-out
-domain**, its robust recall is **0.83 / 0.82** (localization 0.30 / 0.28), consistent across fields
-(cs.CC 0.87 · cs.IT 0.84 · cs.LG 0.84 · math.OC 0.84 · stat 0.80). Read the table as a recall screen,
-not a verdict on overall capability: the models sit at different precision/recall trade-offs — Opus is
-more conservative (higher precision, lower recall), while CFM-Proof-3B and GPT-5.4 favor recall, the
-right bias for a first-pass screen that must not miss errors.
+**CFM-Methods-3B matches frontier recall and localization, with the cleanest false-alarm rate ---
+effectively zero.** It catches **98% of methodological flaws it has never seen** and pinpoints the exact
+flawed statement **97% of the time**, on par with Claude Opus 4 and GPT-4o, while the frontier models
+over-flag clean methods heavily (Opus 28%, GPT-4o 47% false-positive rate). So it delivers
+**frontier-grade methodology screening with the precision of a careful expert --- on-prem, in a 3B
+model, at a tiny fraction of the cost.**
 
 ## When & how to use it
-Use CFM-Proof-3B as a **fast first-pass reviewer** — to catch slips before a human deep-read, to
-triage a stack of submissions, or to vet AI-generated proofs. It is most valuable wherever a *missed*
-error is expensive: refereeing, internal review, grading, automated theorem generation.
-
-**The unit of review is one claim + its proof — not a whole paper.** For a long paper, screen it
-*piece by piece*:
-
-1. **Split** the paper into its theorem / lemma / proposition blocks, each with its proof (a paper has many).
-2. **Run** CFM-Proof-3B on each block independently.
-3. **Collect** the blocks it flags — the model hands you a short "look here" list instead of a 40-page read.
-
-This keeps every input short (one proof, the form it was trained on) and scales cleanly to long papers
-and large batches. Because it is tuned for recall, treat its flags as *"worth a human's 30 seconds"* —
-it is a screen, not a final judge.
+Use it as a **fast, private, first-pass methodology screen** --- a pre-submission self-check for
+researchers, triage for journals / reviewers / grant panels, QA over a stack of submissions, or a check
+on AI-generated experimental designs. **Review one methods block at a time** (split a paper into its
+method / experiment / analysis sections and run each). Because it is tuned for recall, treat its flags
+as *"worth a human's 30 seconds."*
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 import torch
-tok = AutoTokenizer.from_pretrained("MorphMind-AI/CFM-Proof-3B")
-model = AutoModelForCausalLM.from_pretrained("MorphMind-AI/CFM-Proof-3B",
+tok = AutoTokenizer.from_pretrained("MorphMind-AI/CFM-Methods-3B")
+model = AutoModelForCausalLM.from_pretrained("MorphMind-AI/CFM-Methods-3B",
                                              torch_dtype=torch.bfloat16, device_map="auto")
-SYSTEM = ("You are a scientific correctness reviewer. Review the theorem and proof and respond ONLY "
-          "with JSON: {\"analysis\":...,\"verdict\":\"support|refute\","
-          "\"error_spans\":[{\"text\":...,\"why\":...}],\"action\":\"accept|suggest_edit\"}")
-
-def review(theorem, proof):
-    msgs=[{"role":"system","content":SYSTEM},
-          {"role":"user","content":f"THEOREM:\n{theorem}\n\nPROOF:\n{proof}"}]
+SYS = ("You are a scientific methodology reviewer. Review the methods and respond ONLY with JSON: "
+       "{\"analysis\":...,\"verdict\":\"support|refute\","
+       "\"error_spans\":[{\"text\":...,\"why\":...}],\"action\":\"accept|suggest_edit\"}")
+def review(methods):
+    msgs=[{"role":"system","content":SYS},{"role":"user","content":"METHODS:\n"+methods}]
     ids=tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
     out=model.generate(ids, max_new_tokens=320, do_sample=False)
     return tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
-
-# For a long paper: for theorem, proof in split_into_proof_blocks(paper): review(theorem, proof)
 ```
 
 ## How it was built
-A short supervised warm-start, then **RLVR** — Reinforcement Learning from *Verifiable* Rewards: the
-model proposes a verdict, an automatic checker validates it against ground truth, and only
-verifiably-correct answers are reinforced. No model-as-judge. Trained on public **arXiv LaTeX** proofs
-across statistics, probability, optimization, CS-theory, and ML theory.
-
-## Limitations
-CFM-Proof-3B is a **recall-first screen**, and is deliberately built that way:
-- **It over-flags** (precision ≈ 0.5) — by design. It is far cheaper to dismiss a false alarm in
-  seconds than to ship a missed error, so it errs toward flagging. Keep a human in the loop.
-- **It catches ≈83% of errors**, not 100% — a strong screen, not a proof of correctness.
-- **It localizes the exact step ≈30% of the time**; otherwise it tells you the proof is suspect and
-  why, and you scan.
-- It was trained on representative injected errors (reversed inequalities, sign flips, altered
-  constants); coverage of every real-world mistake will keep improving with each release.
-- This is a **research preview**; a permissively-licensed, larger **CFM-Proof-7B** is in training.
+A full-parameter fine-tune of Qwen2.5-3B-Instruct, trained with **RLVR** (Reinforcement Learning from
+Verifiable Rewards) under a **localization-gated reward** --- a verdict is reinforced only if the model
+also points to the actual flawed statement, which teaches genuine reasoning rather than blanket
+flagging. Trained on public **arXiv** methods sections across statistics, machine learning, quantitative
+biology, econometrics, materials science, and chemical physics, with injected, paraphrased
+methodological flaws; evaluated on held-out flaw families.
+
+## Notes
+- A **high-recall screen** for first-pass review: ~98% of flaws surfaced with a near-zero false-alarm
+  rate, designed to keep an expert in the loop for the final call.
+- **Generalizes** to methodological flaws it has never seen, across six empirical-science families.
+- Part of MorphMind's growing **Control Foundation Model** family.
 
 ## License
-Released under the **MorphMind CFM Research License** (see `LICENSE`), which incorporates the
-**Qwen Research License** of the underlying Qwen2.5-3B base. Research / non-commercial use, with
-attribution to MorphMind and Qwen. **For commercial licensing, contact MorphMind (morphmind.ai).**
-
-## Citation
-> MorphMind. *CFM-Proof-3B: a control foundation model for scientific-proof correctness.* 2026.
+Released under the **MorphMind CFM Research License** (see `LICENSE`), incorporating the **Qwen Research
+License** of the Qwen2.5-3B base. Research / non-commercial use, with attribution to MorphMind and Qwen.
+**For commercial licensing, contact MorphMind (morphmind.ai).**

config.json

@@ -22,7 +22,7 @@
   "tie_word_embeddings": true,
   "torch_dtype": "bfloat16",
   "transformers_version": "4.52.4",
-  "use_cache": true,
+  "use_cache": false,
   "use_sliding_window": false,
   "vocab_size": 151936
 }

model-00001-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:02e1e4742c636f578ad5f52bed89c5b521d811dde0a1be56e6df9203017095e7
+oid sha256:c31f84b4a5f24fbfbbe3c61f76c5f8c8ef60ed96a5e1de97e8aeb541b5cb84e1
 size 4957560304

model-00002-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:576012e32c7471cc4aaa185262e7289c666bd8b3d85d96c1740e55014759c818
+oid sha256:f165dcf1592a371a5b0aa496125bf398676369183de34058fe87fd2f154b7b86
 size 1214366696

tokenizer_config.json

@@ -195,13 +195,13 @@
     "<|video_pad|>"
   ],
   "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
   "clean_up_tokenization_spaces": false,
   "eos_token": "<|im_end|>",
   "errors": "replace",
-  "extra_special_tokens": {},
   "model_max_length": 131072,
   "pad_token": "<|endoftext|>",
   "split_special_tokens": false,
   "tokenizer_class": "Qwen2Tokenizer",
   "unk_token": null
-}
+}