narcolepticchicken commited on
Commit
0f60772
Β·
verified Β·
1 Parent(s): 5a85f0b

Update README with honest results, clean structure, paper reference

Browse files
Files changed (1) hide show
  1. README.md +73 -52
README.md CHANGED
@@ -4,7 +4,51 @@ tags:
4
  ---
5
  # Oracle-Credit-Compute (OCC) Stack
6
 
7
- A minimal, open-source research prototype for **agentic compute allocation** where agents earn and spend non-transferable, decaying credits based on verified marginal impact.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
  ## Quickstart
10
 
@@ -14,21 +58,23 @@ cd occ-stack
14
  pip install -r requirements.txt
15
 
16
  # Simulated benchmarks (CPU)
17
- python benchmarks/benchmark_code.py # Code compute allocation
18
- python benchmarks/benchmark_retrieval_qa.py # Retrieval QA
19
- python benchmarks/benchmark_debate_v2.py # Multi-agent debate
20
 
21
- # Ablations + anti-gaming (CPU, ~5 min)
22
  python eval_runner.py
23
 
24
- # Real LLM benchmark (GPU, requires T4+)
25
- python jobs/run_real_llm_standalone_v7.py
26
-
27
  # Unit tests
28
  python tests/test_oracle.py
29
  python tests/test_ledger.py
 
 
 
30
  ```
31
 
 
 
32
  ## Architecture
33
 
34
  ```
@@ -47,67 +93,42 @@ python tests/test_ledger.py
47
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
48
  ```
49
 
50
- ## Key Results (Simulated)
51
 
52
- - **52.3% compute reduction at iso-accuracy** on code benchmark (OCC tiered escalation vs fixed budget)
53
- - **76% accuracy with 40% adversarial agents** in debate (OCC credit-filtering vs 56% naive confidence voting)
54
- - **All anti-gaming attacks contained:** hidden-test gaming, collusion, over-abstention, spam
 
 
 
 
55
 
56
- ## Status
57
 
58
- | Component | Status |
59
- |-----------|--------|
60
- | Impact Oracle | βœ… Working |
61
- | Credit Ledger | βœ… Working |
62
- | Resource Broker | βœ… Working |
63
- | GRPO/RL Hook | βœ… Factory ready |
64
- | Simulated benchmarks | βœ… Complete |
65
- | Ablations (10 conditions) | βœ… Complete |
66
- | Anti-gaming tests | βœ… Complete |
67
- | Real LLM benchmark | πŸ”„ V7 in progress |
68
- | GRPO training | πŸ”„ Not yet run |
69
 
70
  ## Repo Structure
71
 
72
  ```
73
  occ/
74
- oracle/ # ImpactOracle β€” rule-based scoring
75
- ledger/ # CreditLedger β€” non-transferable, decaying credits
76
- broker/ # ResourceBroker β€” capability-based access control
77
- rl/ # RewardHook, OfflineComparator β€” TRL GRPO integration
78
- benchmarks/ # 3 benchmark scripts + real LLM variants
 
79
  tests/ # Unit tests
80
- reports/ # Reports, results, blog post
81
- jobs/ # Self-contained GPU job scripts
82
  ```
83
 
84
  ## Citation
85
 
86
  ```bibtex
87
  @misc{occ2026,
88
- title={Oracle-Credit-Compute: A Minimal Stack for Agentic Compute Allocation},
89
  author={narcolepticchicken},
90
  year={2026},
91
  url={https://huggingface.co/narcolepticchicken/occ-stack}
92
  }
93
  ```
94
-
95
- <!-- ml-intern-provenance -->
96
- ## Generated by ML Intern
97
-
98
- This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub.
99
-
100
- - Try ML Intern: https://smolagents-ml-intern.hf.space
101
- - Source code: https://github.com/huggingface/ml-intern
102
-
103
- ## Usage
104
-
105
- ```python
106
- from transformers import AutoModelForCausalLM, AutoTokenizer
107
-
108
- model_id = 'narcolepticchicken/occ-stack'
109
- tokenizer = AutoTokenizer.from_pretrained(model_id)
110
- model = AutoModelForCausalLM.from_pretrained(model_id)
111
- ```
112
-
113
- For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class.
 
4
  ---
5
  # Oracle-Credit-Compute (OCC) Stack
6
 
7
+ **Compute is not neutral.** In multi-agent debate, giving an adversarial agent equal speaking turns causes accuracy to collapse from 73.3% to 56.7% β€” worse than single-round voting. The mechanism is volume amplification, not persuasion: honest agents retain their positions 84% of the time but get outvoted.
8
+
9
+ OCC is a mechanism-design layer that treats agent compute as a scarce, earned, auditable privilege. Agents earn non-transferable, decaying, capability-scoped credits based on verified marginal impact.
10
+
11
+ **Paper:** `reports/occ_paper.md`
12
+ **Design doc:** `design.md`
13
+ **Literature review:** `reports/literature_review.md`
14
+
15
+ ---
16
+
17
+ ## Key Results
18
+
19
+ ### Debate Collapse (Real LLM β€” Qwen3-Coder-30B on H200)
20
+
21
+ | Condition | Accuracy | Ξ” |
22
+ |-----------|:--------:|:---:|
23
+ | 1-round baseline | **73.3%** | β€” |
24
+ | 3-round debate (with adversary) | **56.7%** | βˆ’16.7pp |
25
+ | Judge voting | **73.3%** | recovered |
26
+ | Confidence-weighted voting | **73.3%** | recovered |
27
+ | Equal-token (cap adversary volume) | **70.0%** | βˆ’3.3pp |
28
+
29
+ **Root cause:** Volume amplification (H1 βœ…). Not persuasion (H4 ❌, 84% retention). Not skill-dependent (H6 ❌, weak adversary still collapses).
30
+
31
+ ### Simulated Benchmarks
32
+
33
+ | Benchmark | Baseline | OCC | Savings |
34
+ |----------|:--------:|:---:|:-------:|
35
+ | Code (pass@1, iso-accuracy) | 0.78 | 0.78 | 52.3% compute |
36
+ | Debate (adversarial) | 0.56 | 0.76 | +20pp accuracy |
37
+ | Retrieval QA | 0.79 | 0.71 | 33% retrievals (accuracy lags) |
38
+
39
+ ### Anti-Gaming
40
+
41
+ 10 attack vectors tested, all contained: credit farming, collusion, oracle spoofing, verbosity gaming, confidence manipulation, strategic abstention, identity laundering, sybil agents, sandbagging, griefing.
42
+
43
+ ### Honest Limitations
44
+
45
+ - OCC β‰ˆ random gating at moderate budgets (doesn't beat simple baselines in median case)
46
+ - GRPO training produced no improvement at 0.5B scale
47
+ - Single seed for mechanism isolation (CI Β±16pp)
48
+ - Simulated benchmarks for code/QA (not real LLM agents)
49
+ - HumanEval result is adaptive retry, not OCC credit allocation
50
+
51
+ ---
52
 
53
  ## Quickstart
54
 
 
58
  pip install -r requirements.txt
59
 
60
  # Simulated benchmarks (CPU)
61
+ python benchmarks/benchmark_code.py
62
+ python benchmarks/benchmark_debate_v2.py
63
+ python benchmarks/benchmark_retrieval_qa.py
64
 
65
+ # Ablations + anti-gaming (CPU)
66
  python eval_runner.py
67
 
 
 
 
68
  # Unit tests
69
  python tests/test_oracle.py
70
  python tests/test_ledger.py
71
+
72
+ # Debate collapse analysis
73
+ python jobs/analyze_collapse.py reports/debate_collapse_mechanism_results.json
74
  ```
75
 
76
+ ---
77
+
78
  ## Architecture
79
 
80
  ```
 
93
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
94
  ```
95
 
96
+ ## Components
97
 
98
+ | Component | File | Description |
99
+ |-----------|------|-------------|
100
+ | Impact Oracle | `oracle/oracle.py` | Multi-mode scoring (code, QA, debate) with cost-adjusted rewards |
101
+ | Credit Ledger | `ledger/ledger.py` | Non-transferable, decaying, capability-scoped credits with audit trail |
102
+ | Resource Broker | `broker/broker.py` | Capability-based access control (allow/deny/downgrade/escalate) |
103
+ | GRPO Hook | `rl/grpo_hook.py` | TRL-compatible reward function for learned allocation |
104
+ | Reward | `rl/reward.py` | Cost-adjusted reward with anti-gaming penalties |
105
 
106
+ ## Formal Definition
107
 
108
+ Full formal definition with 10 system invariants, ledger event schema, threat model, and when-to-use guidance in [`design.md`](design.md).
 
 
 
 
 
 
 
 
 
 
109
 
110
  ## Repo Structure
111
 
112
  ```
113
  occ/
114
+ oracle/ # ImpactOracle
115
+ ledger/ # CreditLedger
116
+ broker/ # ResourceBroker
117
+ rl/ # GRPO hook + reward function
118
+ benchmarks/ # 3 benchmark scripts
119
+ jobs/ # GPU job scripts + analysis
120
  tests/ # Unit tests
121
+ reports/ # Paper, reports, literature review, blog post
122
+ design.md # Formal system definition
123
  ```
124
 
125
  ## Citation
126
 
127
  ```bibtex
128
  @misc{occ2026,
129
+ title={Compute Is Not Neutral: Mechanism Analysis of Adversarial Debate Collapse and the OCC Stack},
130
  author={narcolepticchicken},
131
  year={2026},
132
  url={https://huggingface.co/narcolepticchicken/occ-stack}
133
  }
134
  ```