Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
RDTvlokip 
posted an update 10 days ago
Post
72
I published an article about training a network to write from reward alone. Code on GitHub with it.

Then someone read the code.

Dipankar Sarkar commented four times in a day. Each time he had run something first. He rebuilt my statistics in numpy because he had no torch installed.

He found a bound I had missed. A policy that never learned the determiner to noun dependency has a product support, so at full validity it cannot exceed the largest fully valid product in the sublanguage it entered. That is 12 on one side and 24 on the other, computable before any training. Over 70 seeds it is never crossed, and the most common outcome is the bound itself. I had published one of those numbers as an interesting coincidence.

Then four of my published numbers came apart. Three were a single seed. The fourth was twenty seeds, and I had produced it while fixing the other three.

And the test I built to validate his bound tested nothing. I had swapped two conditions so cleanly that the two grammars were isomorphic. Seventy seeds would have returned the mirror image by construction.

The real lesson:

A relabelling can permute, but it cannot change a ratio. A perfectly symmetric control is often a perfectly empty one. None of my errors were in the reasoning. They were in the plumbing, and nothing in my own process caught a single one.

Code, figures, and the notebook with eight dated refutations 👇

🔗 https://huggingface.co/blog/RDTvlokip/i-published-my-rl-experiments
💻 https://github.com/RDTvlokip/RDTRL
📦 https://doi.org/10.5281/zenodo.21726216

Thank you for writing this up. Test 3 is the interesting part, so I went there instead.

The one number in your null is the one you called derived.

I ran grammaire3.py at seed 0 before saying anything. Your table reproduces exactly: mean 0.1273, sd 0.0332, min 0.0305, max 0.3305, quantiles 0.0661 / 0.1246 / 0.2171 / 0.2537.

Then I ran the same statistic at 10,000,000 draws instead of 20,000. Vectorised, checked against your own concentration() on 3,000 codes first, max absolute difference 5.6e-17.

Every row holds except one.

20,000 10,000,000
mean 0.1273 0.1269
sd 0.0332 0.0330
q99 0.2171 0.2161
q99.9 0.2537 0.2525
max 0.3305 0.3979

500x the draws moves q99.9 by 0.0012 and the max by 0.067. The body of the null is settled at 20,000. The max is the only row still moving, and the ~0.35 line is the one thing built on it. 14 of the 10M draws clear 0.35.

That does not hurt you and I would rather say so than dress it up. At 100 seeds the chance any run clears 0.35 under the null is 1.4e-4. The threshold is safe for the experiment you planned.

The reason to drop it anyway is that §6.2 is already better without it.

Your recorded commitment is about a distribution, not a run. 100 emergent concentrations against a null of sd 0.033 resolves a mean shift of 0.013 at p<0.001 with 80% power. 50 seeds resolves 0.018. The threshold needs one run sitting 0.223 above the null mean.

So the instrument you specified is 17x more sensitive at 100 seeds than the line written next to it. And a weak outside pressure is far likelier to lift every run by 0.02 than to throw one outlier past 0.35. That pressure is invisible to the threshold and a 6-sigma result to the test you already have.

Quote a quantile instead. q99.9 is stable at 20,000. q99.999 needs the long run and sits at 0.3195.

Separately, on the statistic itself, and this one I am less sure about.

concentration takes max over attributes independently per column, so one attribute can win two positions. I checked how often that happens: 74.6% of 500,000 uniform draws, and 46% of the top 200 by concentration. Scoring those same top 200 with a Hungarian matching instead of a per-column max gives 0.2640 against 0.2810. Mean inflation 0.0170, largest single 0.0915.

For your §6.2 comparison this cancels. Same transform on both sides, emergent and null, so the test is unaffected and I am not asking you to change it.

Where it does not cancel is §6.1, where you read the scalar as a position in the space of perfect codes rather than a binary. Under that reading a 0.28 code is being credited 0.017 of structure it does not have, in the direction that makes a holistic code look composed.

Concentration 1.0 still forces an injective argmax, so the top of the scale is safe. A code where two positions each fully determine one attribute would collapse that attribute's nine referents onto three messages, and a bijection cannot.

So the inflation is real in the middle and absent at both ends.

Is the max standing in for a matching you have not needed yet, or is there a reason to want the double count?

·

You went to the one part of the project that has never run. That is the right place to go, and it is the first time in five rounds that a criticism has arrived before the data rather than after. That matters at the end of this reply.

Your numbers

All of them. Seed 0 at 20,000 reproduces to the digit. My own 10M draw is an independent stream, so it is a check on the estimand and not on your RNG:

you, 10M me, 10M
mean 0.1269 0.12688
sd 0.0330 0.03304
q99 0.2161 0.21613
q99.9 0.2525 0.25254
q99.999 0.3195 0.31963
max 0.3979 0.39788
draws ≥ 0.35 14 14

Two independent samples agreeing on a maximum to four digits made me check whether the statistic was discrete enough to explain it. It is discrete — 2,951 distinct values in 2M draws, since the whole matrix is a function of nine integer counts — but the top rungs are singletons, so discreteness explains nothing. It is a coincidence, and the next section is what happens when you stop relying on one.

Your vectorised-vs-scalar check: I did the same and got max absolute difference 5.551e-17 on 3,000 codes, which I assume is your 5.6e-17. For a bijection both margins of the joint are exactly uniform, so I(A_i;M_j) = Σ (c/27)·log2(c/3) with c integer — that identity is what makes the whole thing a lookup table, and validating against the scalar path is really validating the identity.

The threshold: you are right, and the reason is worse than "still moving"

Your framing is that the max is the only row still in motion. Mine is that it cannot stop.

The 1,296 compositional codes are bijections. They are in the null, with probability 1296/27! ≈ 1.19e-25, and they score exactly 1. So the supremum of the null distribution is exactly 1 — the very value the threshold was built to declare unreachable. A sample maximum here is not estimating a threshold. It is estimating 1, infinitely slowly. No sample size fixes that, because there is nothing to converge to except the thing being tested.

Empirically, twelve independent blocks of 10,000,000 (variabilite_du_maximum.py, 384 s):

row mean over 12 blocks range across blocks
mean 0.1269 0.0000
sd 0.0330 0.0000
q99 0.2159 0.0003
q99.9 0.2527 0.0006
q99.99 0.2863 0.0019
max 0.3950 0.0509

The block maxima run from 0.3775 to 0.4283. That range is 1.54 standard deviations of the null itself. A threshold read off that row inherits it: it measures how long I was willing to sample, not the distribution.

So: threshold withdrawn, quantile quoted instead, exactly as you proposed. q99.9 = 0.2525.

The part you found without pointing at it

§6.1 should never have carried a pass/fail line at all, and my own document says so three paragraphs earlier. §5 reads:

On abandonne délibérément le critère pass/fail. […] la mesure de concentration positionnelle définie en §6.1 sera statistiquement indiscernable de celle d'une permutation tirée uniformément au hasard.

That is a commitment about a distribution. Then §6.1 reintroduces a pass/fail and congratulates itself for it being derived rather than arbitrary. The defect was never the arbitrariness. It was the pass/fail, which §5 had already discarded for the reasons you then re-derived from the power side. I wrote both paragraphs. It took an outside reader for me to read them next to each other.

Your power numbers reproduce exactly. One-sided, p < 0.001, 80% power, δ = 3.93σ/√n:

seeds max-per-column matched
50 0.0184 0.0175
100 0.0130 0.0124

against 0.223 on a single run for the threshold. Ratio 17. That calculation is now in §6.2 where it belongs.

The matching: your diagnosis is right, your localisation is not

74.6% double-count rate on uniform draws — I get 74.6%. On my top 200, matching gives mean inflation 0.0179 against your 0.0170. Same phenomenon.

Where I disagree is "real in the middle and absent at both ends". I could not test that against the null alone, because the null has no structured codes in it. So I built two populations where the structure is known independently of the MI matrix.

Codes with k of 3 positions cleanly encoding an attribute, the rest scrambled conditionally (still bijections), 4,000 each:

k expected max-per-column matched double count inflation
0 0.0000 0.1268 0.1170 74.0% 0.0098
1 0.3333 0.4111 0.4059 48.5% 0.0052
2 0.6667 0.7045 0.7045 0.8% 0.0000
3 1.0000 1.0000 1.0000 0.0% 0.0000

A ladder from a compositional code outward by transpositions, scored against d = minimum Hamming distance to any of the 1,296 (so relabellings are free):

 d      0    2    4    6    8    9   10   12   14   16   18   20   21   23
 infl  .000 .000 .000 .000 .000 .000 .0001 .0004 .0022 .0051 .0081 .0099 .0103 .0090
 dbl%   0    0    0    0    0    0   1.1   3.5  17.4 35.5 57.1 68.8 73.2 77.7

Inflation is exactly zero out to nine transpositions, then climbs and meets the null's own 0.0101 around d = 21.

So the inflation does not live in the middle of the scale. It lives in the region with no positional structure at all, and it is identically zero everywhere there is structure to read. Which means the specific §6.1 worry — a 0.28 code credited 0.017 it does not have — is real only for codes that reached 0.28 by accident, and those have nothing to read either way.

I want to flag how I nearly got this backwards. My first sweep binned the null by concentration level and found inflation rising with concentration: 0.0014 below 0.05, 0.0228 above 0.30. That is a true measurement and it points the opposite way from the ladder, which at the same concentration ≈ 0.27 gives 0.0022. Both are correct. Inflation tracks structure, not level, and those are not the same axis — a uniform draw at 0.27 got there by accident and usually double-counts, a near-compositional code at 0.27 did not. One population would have convinced me of the wrong thing.

Your actual question

Is the max standing in for a matching you have not needed yet, or is there a reason to want the double count?

Standing in for a matching. There was no reason.

There is one reason to keep reporting it, which I did not know when I wrote it and had to go check: the unconstrained per-position form is the field standard. posdis (Chaabouni et al. 2020) takes the argmax independently per position with no one-to-one constraint. So dropping it costs comparability with published numbers.

Both are now in grammaire3.py. concentration() unchanged, so the 20,000-draw table still reproduces bit for bit. concentration_appariee() added — exact Hungarian, and at 3×3 that is six permutations, so enumerated rather than solved. §6.1 now reads the matched one as its position.

One argument for your fix that you did not make

You wrote that the change cancels in §6.2 because it is the same transform on both sides. That holds under H0 — if the emergent codes are ordinary bijections, both populations shift by the same 0.0101 and nothing changes.

It does not hold under H1. If a pressure creates positional structure, the double count disappears with it, as the tables above show. The 0.0101 then comes off the null and not off the emergent population. The matched statistic lowers the reference without moving the signal: neutral where it must be, favourable where I want power. Small — sd also drops 0.0330 → 0.0315 — but it points the right way, and it is a better argument for your change than the one you offered.

Bounding the damage

Local hill-climbing over transpositions, so these are lower bounds, not maxima:

  • largest max-minus-matched gap found: 0.1443, on a code scoring 0.2473 — i.e. inside the body of the null, where nothing is concluded anyway;
  • highest concentration reachable while double-counting: 0.6314, and that code still scores 0.5560 matched. Even pushed adversarially, the max does not turn an unstructured code into a structured one;
  • highest non-compositional concentration found: 0.9294. Your argument that concentration 1.0 forces an injective argmax is right — two positions each fully determining one attribute would collapse nine referents onto three messages, and a bijection cannot — and the top of the scale turns out to be isolated as well as safe: 1.0000, then 0.9294.

All three of those are hill-climbing over permutations, which matters more than I realised when I wrote them. See below.

A follow-up that makes your fix necessary rather than optional

After answering you I ran §6.7, the two-agent check that had never been done. One of its results changes the standing of your second question.

Exact gradient ascent on the joint objective, no sampling anywhere, does not reach a bijection from a uniform start. One start in forty. It settles on codes where 1 to 4 referents collide — E[R] lands on 23/27, 24/27, 25/27, 26/27. Started on a perfect code it stays at exactly 1.

So the emergent codes will not be bijections, and your safety argument for the top of the scale is conditional on bijectivity. Drop that condition and it fails outright. Take m₁ = a₁, m₂ = a₁, m₃ = a₂ — duplicate the first attribute across two positions, discard the third:

distinct messages used : 9 of 27          bijective : no

I(A_i ; M_j) in bits          concentration, max      : 1.000000
[[1.585 1.585 0.   ]          concentration, matched  : 0.666667
 [0.    0.    1.585]
 [0.    0.    0.   ]]

The published statistic hands a perfect 1.0000 — the value reserved for compositional codes — to a code that throws away one attribute in three and uses a third of the message space. The matched version returns 0.667, which is the right answer: two attributes read out of three.

So the double count is not a middle-of-the-scale artefact confined to structureless codes, which is what the measurement on bijections suggested to both of us. Once bijectivity goes, it reaches the top. Your fix is not a 0.48-point improvement in rank concordance. It is the only one of the two statistics that stays interpretable in the regime the experiment will actually be in.

I would not have found this by measuring harder on the null. It came from running the step that had never been run, and it is the second time in this exchange that the useful thing was upstream of the number under discussion.

Why this round was cheap and the next one would not be

No Test 3 training has run. There is no measured emergent concentration anywhere in the repository. So changing the instrument today cannot have been motivated by a result, and anyone can check that from the git history. The identical change made after a first run would be unverifiable from outside, and should be refused. That is the whole reason §5 recorded its commitment with a date, and it is the first time the mechanism has actually been worth something.

Thank you for going to the unpublished part. It is where the mistakes are cheapest to fix and where nobody has any reason to look.

Your §6.7 result retires your own damage bounds. I think they want re-running, not deleting.

All three numbers under "Bounding the damage" are hill-climbs over permutations. §6.7 then reports that
exact gradient ascent does not reach a bijection from a uniform start, landing at R = 23, 24, 25, 26
of 27. So the bounds were measured in the one regime the experiment will not be in. You saw that and
said so, that all three are hill-climbing over permutations and it mattered more than you realised when
you wrote them. What followed drew the qualitative conclusion and left the three numbers standing.

Controls first, so the rest is comparable. Same estimator, my own implementation:

null over bijections, n=20,000   mean 0.1273  sd 0.0332  q99 0.2171  q99.9 0.2537
you                              mean 0.1269  sd 0.0330  q99 0.2161  q99.9 0.2525
null inflation (max - matched)   0.0102        you 0.0098 to 0.0101
m1=a1, m2=a1, m3=a2              max 1.000000  matched 0.666667   exact

Then I re-ran the first two of them with a floor on distinct messages. Identical climber and identical
budget at every floor, so only the constraint moves:

R floor      27      26      25      24      23
max gap  0.0396  0.1628  0.1850  0.2002  0.2112
at conc  0.1596  0.2810  0.3020  0.3264  0.3454

One collision is the whole effect. Going from 27 to 26 quadruples the reachable gap, and everything
after that is a slow slope. Your second bound moves the same way: highest concentration reachable
while still double-counting is 0.2045 at R = 27 under my climber, and 0.6409 at R = 26, with matched
0.5812. Your permutation figure was 0.6314.

The comparison is stronger than it looks, because my climber is worse than yours. At R = 27 it finds
0.0396, against your 0.1443 on the same space. Triple the budget on a fresh seed and it reaches only
0.0593. So the R = 26 rows are loose lower bounds from a weak search, and they already pass what your
stronger search found one collision away.

Here is the R = 26 winner at 3x budget, gap 0.1525:

I(A_i ; M_j) bits          argmax attribute per position: [0 0 0]
[[0.318 0.484 0.498]       injective: no
 [0.054 0.030 0.000]       distinct messages: 26 of 27, one pair colliding
 [0.036 0.012 0.036]]

That is the shape of your m1=a1, m2=a1, m3=a2 example. One attribute claimed by all three positions,
argmax non-injective, matched forced to pay for it. Your example needed 18 referents collapsed to get
there. This needs one, at the top of the range §6.7 actually reported.

Which turns your closing argument around. You wrote that the matched statistic is the only one that
stays interpretable once bijectivity goes. I think that is right and understated: the double count
does not degrade gradually as R falls, it arrives almost entirely at the first collision. Is R = 26
common enough in the forty §6.7 starts to be the modal case rather than the boundary?

·

You are right, and the diagnosis is worse than "measured in the wrong regime"

I annotated the three bounds as conditional on bijectivity and left them standing. Annotation is not the fix, and I knew that at the time. The same article says a retracted number should be re-measured, not footnoted.

But there is a sharper reason they could not have been right, and it is about the search rather than the reporting. My climber moved by transpositions. A transposition of a permutation is a permutation, so the move set could not leave the bijective regime even in principle. The constraint was not an assumption I failed to check, it was welded into the operator. Yours reassigns one referent to any message, which is why it can see what mine could not.

That is the third time this week that my instrument, not my reasoning, decided the answer.

Your sweep reproduces

Independent implementation, single-referent reassignment, 20 restarts, 120 steps, identical budget at every floor:

R floor 27 26 25 24 23
max gap, you 0.0396 0.1628 0.1850 0.2002 0.2112
max gap, me 0.0526 0.1362 0.1783 0.1850 0.2152

Same shape, and the same conclusion: one collision is the whole effect. My R = 27 baseline is a little higher than yours, so my ratio is 2.6x rather than 4x, which does not touch the claim.

Your second bound matches exactly. Highest concentration reachable while still double-counting: at R = 27 you get 0.2045 and I get 0.1943; at R = 26 we both get 0.6409, with matched 0.5812. Two different climbers, four decimals. My permutation figure was 0.6314, so one collision beats a bijectivity-restricted search over the whole permutation space.

And my R = 26 winner has the shape you describe, with a different attribute index:

I(A_i ; M_j) bits          argmax per position: [1 1 1]
[[0.086 0.012 0.062]       distinct messages: 26 of 27
 [0.416 0.330 0.410]       max 0.2430 against matched 0.1068
 [0.036 0.012 0.049]]

One attribute claimed by all three positions, at one collision instead of the eighteen my hand-built example needed.

Your question, answered on 100 seeds rather than 40

You asked whether R = 26 is common enough in the forty starts to be the modal case rather than the boundary. It is neither: R = 25 is modal, and R = 26 is second. Tabular sender, beta = 0.02, 100 seeds:

R 27 26 25 24 23
runs 3 37 43 16 1

So the bijective case is not the boundary of the regime, it is 3 % of it. Ninety-seven of a hundred runs sit at or past the first collision, and the median run is one collision beyond where your sweep says the effect has already arrived.

It holds across the code regime rather than at one beta. Mean R by entropy coefficient, 20 seeds each: 24.05 at 0.005, 24.55 at 0.010, 24.80 at 0.030, 25.25 at 0.037. Raising beta toward the stability threshold buys back distinct messages and never reaches 27 on average.

And the other two parametrizations sit deeper, not shallower. Modal R = 21 for the autoregressive-per-referent sender, R = 23 for the attribute-structured one.

So your closing point is right and I had it too weak

I wrote that the matched statistic is the only one that stays interpretable once bijectivity goes. The correct statement is stronger, and it is yours: the double count does not degrade gradually as R falls. It arrives almost entirely at the first collision, and the first collision is not an edge case, it is 97 % of runs.

Which means the published statistic is unusable in this experiment's actual operating regime, not merely at its boundary. That is a firmer conclusion than the one I shipped, and it comes from re-running a number I had chosen to annotate.

The three bounds are replaced by the R-indexed table rather than deleted, with the transposition-operator diagnosis written next to them, and the article is updated. Thank you for not letting me footnote my way out of that one.

You changed the regime, and I want to price what that actually cost, using your hundred seeds rather than my forty.

Your R distribution is the piece that was missing. I had null(R). You now have p(R). Nobody had multiplied them, and that product is the baseline a run drawn from your procedure should be scored against.

200,000 draws per cell, same two samplers as before:

  R   p(R)   null_max   null_matched
 27   0.03    0.12681      0.11679
 26   0.37    0.12733      0.11697
 25   0.43    0.12749      0.11695
 24   0.16    0.12776      0.11697
 23   0.01    0.12788      0.11693

Standard error 7.4e-5 on max, 7.0e-5 on matched.

Mixing over your p(R):

fixed R = 27 null      max 0.12681    matched 0.11679
procedure null         max 0.12746    matched 0.11696
difference             max +0.00065   matched +0.00016

So the bijective baseline was the wrong regime, but not for the reason either of us assumed. Across the entire support your procedure visits, the matched null moves by 0.00016, against a per-cell standard error of 7.0e-5. It is invisible next to anything either of us is measuring.

Now run the same mixture on the observed column of your table, the one from my sweep:

gap at R = 27 only         0.0396
E[gap] under your p(R)     0.1751

The null shifts by 0.00016. The measurement shifts by 0.1355. Ratio about 850.

Which flips the diagnosis. The transposition move set did not corrupt your baseline. It starved your numerator. Reporting at R = 27 was not measuring against the wrong null, it was reporting 22.6% of your own effect and calling it the whole thing.

Your three retracted bounds were not too generous. They were too small.

That also makes the fix cheap. You do not need a null per R, because there is not one to derive. One number, 0.1170 matched, covers 23 through 27 at this resolution. What you needed was a search allowed to reach R = 25, which yours now is.

One thing I cannot settle from here. My p(R) is yours, not an independent draw, so the mixture inherits whatever your sender's beta schedule does. You reported mean R climbing from 24.05 to 25.25 as beta goes 0.005 to 0.037. At the top of that range p(R) is shifting toward the cell that reports the smallest effect.

Does the effect size fall as you raise beta, or does R stop being the variable that predicts it?

·

You are right about the structure, and I want to correct one number before agreeing with the rest.

One magnitude, which is mine to fix

Your 22.6 % compares your climber at R = 27 (0.0396) against your own mixture (0.1751). My published bound was 0.1443, from the transposition climber, and that is 90 % of my own mixture at 0.1604 — or 82 % of yours. The undercount is 11 %, not a factor of four.

The direction of your correction holds entirely. Its magnitude describes your sweep, not the figure I published.

Everything else lands, including the part I got backwards

The null moves 0.00016 across the whole support my procedure visits. The reachable worst case moves 0.1355. I spent an entire section rebuilding the reference while the damage sat in the search's reachable set — the denominator when the problem was the numerator.

So my three bounds were not too generous. They were too small, and a retraction that points the wrong way is worse than the overclaim it replaces.

Worth recording that we converged on the flatness by different partitions. You measured null by R, 200 000 draws per cell. I had measured it by full fiber profile, eleven profiles, −0.0001 to +0.0005. Same conclusion, no shared assumption, which is the only kind of agreement worth much between us after the concentration-1.0 episode.

Your question, measured

150 runs, five betas by thirty seeds, on actually emergent codes rather than a search worst case.

beta 0.005 0.010 0.020 0.030 0.037
R mean 24.57 24.53 25.10 24.87 25.10
observed gap 0.0077 0.0094 0.0100 0.0175 0.0106
R 27 26 25 24 23
observed gap 0.0096 0.0120 0.0139 0.0076 0.0106
n 8 30 53 47 12

Neither. The gap does not fall as beta rises: correlation +0.158, and no trend past the noise on the row itself. And R does not stop predicting it, because R never predicted it — correlation +0.091, 4 % of variance explained. Conditioning on R removes almost nothing: the residual correlation with beta stays at +0.144 against +0.158 unconditioned.

It is not that R ceases to be the variable. It was never the variable for this quantity.

Which forces a distinction neither of us had made

The worst case reachable under a floor on R and the gap the dynamics actually produces are different functions of R.

R                    27       26       25       24       23
worst case       0.0526   0.1362   0.1783   0.1850   0.2152     strongly monotone
observed gap     0.0096   0.0120   0.0139   0.0076   0.0106     flat

Ratio between them: 15.2. Emergent codes come nowhere near what an adversarial search reaches in the same regime. Both numbers are indexed by R and there is no reason they should track — one is what a search can find, the other is where the dynamics lands.

And your beta reservation settles by computation

Mixed over the p(R) specific to each beta, the worst case runs from 0.1564 to 0.1773 — 0.0209 of amplitude across the whole range. The shift with beta exists, and it is small relative to the quantity being bounded.

What I take from this

A bound must be mixed over the regime actually visited. An effect size must not be read off the same table. I had neither rule this morning, and I got the second one wrong by inheriting the first one's index.

The three bounds are now reported as the R-indexed table, the mixture is stated next to them, and the observed gap is reported separately with its own note that R explains 4 % of it. Notebook §7.25.

Your 11% and your 90% are both right, and they use different denominators. Worth saying out loud so nobody reads them as a contradiction later.

0.1443 / 0.1604 = 0.8996, so 90% of your mixture.
(0.1604 - 0.1443) / 0.1443 = 0.1116, so an 11% undercount relative to what you published.

Same two numbers, and the correction stands. I was scoring my climber's R = 27 cell against my mixture, which is my sweep, not your figure.

Your other pair reads like a contradiction too, and it also is not.

+0.091 correlation is 0.83% of variance. You reported 4%. Both come out of your own table, because they are two different models. 0.091 is the linear trend in R. 4% is R as a five-level factor. I re-derived it from the table you posted, which is enough, because R is constant inside each bin so the binned means give the exact covariance.

N                    150   (8+30+53+47+12, matches)
weighted mean R      24.8333
weighted mean gap    0.011053
Cov(R, gap)          0.00118878
Var(R)               1.01889
between-group var    6.908e-06

Back out the total sd from your own r = 0.091 and it is 0.012942, against a mean gap of 0.011053. The run-to-run spread is larger than the quantity. Then eta-squared is 6.908e-06 / 0.012942^2 = 0.0412, which is your 4%. Consistent, both of them.

But that 4% is what noise looks like at n = 150.

F(4, 145) = 1.559     p = 0.188

So the honest sentence is not "R explains 4% of it". It is "R is not distinguishable from explaining none of it". That is a stronger claim in your favour, not a weaker one, and it is the version that survives a referee.

The flatness carries more than the 15.2 ratio says, and the reason is power, not effect size.

observed slope   +0.001167  per unit R    SE 0.001047   t = 1.11
worst-case slope -0.04065   per unit R    (0.2152 -> 0.0526 over 4)

At 80% power your design detects a slope of 0.00293. The worst-case row's slope is 13.9 times that. So the observed row is not merely smaller than the reachable row. It is bounded away from it by fourteen times your own resolution, and it points the other way. A search that could have found the worst-case trend in this sample found the opposite sign at t = 1.11.

That is the sentence I would put in §7.25 instead of the ratio. The ratio invites "different scales". The power statement forecloses it.

One cell before "flat" goes in the notebook.

Per-bin standard errors from your own sd:

R      27       26       25       24       23
mean   0.0096   0.0120   0.0139   0.0076   0.0106
SE     0.0046   0.0024   0.0018   0.0019   0.0037
n      8        30       53       47       12

R = 25 against R = 24 is 0.0063 apart with a difference SE of 0.0026, so t = 2.4. It is the only contrast in the table past 2 sigma, and with ten pairwise contrasts available it is exactly what you expect to find by looking. So I do not believe it. But it is the one place where the flat row has a kink at the two cells carrying two thirds of your seeds, which is where a kink would be hardest to dismiss if it were real.

Thirty more seeds split across R = 24 and R = 25 would take that t below 1 or above 3. Is the sender cheap enough at those two cells to just run it?

·

You called it. I ran it. The contrast flipped sign.

First, your arithmetic, checked against the raw runs rather than the table

Everything you derived from binned means reproduces on the 150 individual runs. Your reconstruction was exact enough that the differences are rounding:

you, from the table me, from the runs
Var(R) 1.01889 1.01889
Cov(R, gap) 0.00118878 0.00120187
between-group variance 6.908e−06 6.963e−06
sd(gap) 0.012942 0.013021
eta² 0.0412 0.0411
F(4, 145) 1.559 1.552
slope per unit R +0.001167, SE 0.001047, t = 1.11 +0.001180, SE 0.001056, t = 1.12
detectable at 80 % 0.00293 0.00296
worst-case / detectable 13.9 13.7

And you are right that my 11 % and 90 % are the same two numbers over different denominators. Worth stating once so it does not read as a hedge later.

Your per-bin standard errors differ from mine because you pooled: 0.012942/√8 = 0.00458 is your R = 27 cell exactly. Mine are the within-bin values. Yours is the better estimator at n = 8; mine tests the homoscedasticity yours assumes. On the contrast that matters they agree — your t = 2.4, my t = 2.53.

The thing you cannot do from where you are sitting

R is an output of the run, not a setting. I cannot run thirty seeds at R = 24 and R = 25. I run seeds and take whatever R falls out. Those two cells carry 100 of 150 runs, so thirty more inside them costs about forty-five runs. I ran sixty, on an independent seed, same five betas.

The replication

On the sixty new runs alone, R = 25 against R = 24:

difference   -0.0053     SE 0.0033     t = -1.60

The sign flips. Which is what the largest of ten available contrasts does when it is the largest of ten available contrasts. You said you did not believe it. Neither of us should have.

Pooled to 210 runs, everything moves toward null and nothing moves away from it:

n = 150 n = 210
R = 25 vs R = 24 +0.0063, t = 2.53 +0.0028, t = 1.47
eta² 4.11 % 1.2 %
F F(4,145) = 1.552, p = 0.19 F(6,203) = 0.419, p = 0.87
slope per unit R +0.001180, t = 1.12 +0.000507, t = 0.61

The new sample also reached R = 22 and R = 21 once each, which is why the F has six numerator degrees of freedom rather than four.

So I am taking both of your reformulations, and they are now stronger than when you wrote them

On R. "R explains 4 %" was the wrong sentence, and at n = 210 it is not even the wrong number any more: eta² is 1.2 % and F(6,203) = 0.419 at p = 0.87. The sentence that goes in the notebook is yours — R is not distinguishable from explaining none of it — and the replication is what earns it.

On flatness. Your power framing replaces my ratio, and the pooled sample improves it:

observed slope    +0.000507 per unit R    SE 0.000836    t = 0.61
detectable at 80 %  0.00234
worst-case slope   -0.04065 per unit R
ratio               17.4

Up from 13.9. The observed row is bounded away from the reachable row by seventeen times my own resolution, and it still points the other way. You are right that the ratio invites "different scales" and the power statement forecloses it. It is now in §7.25 in your words rather than mine.

What this cost and what it bought

Sixty runs, about half an hour. It killed a kink I would have mentioned as "the one place the flat row is not flat", removed a 4 % figure I had reported as a finding, and turned a p = 0.19 into a p = 0.87.

The generalisable part is not about R. It is that the one contrast past two sigma in a table of ten was exactly the one to distrust, and the cheapest way to know was to draw again rather than to argue about it. That is the fourth time in this exchange that the answer was a second draw, and the first time I made it before publishing rather than after.

Your replication is right, and both halves of it are worth less and more than you
just said. I put your procedure under the null it was arguing against.

The kink was a 10.7% event, not a 1.6% one.

400,000 draws of your own discovery design with nothing in it. Five cells at your
published n (8/30/53/47/12), all cell means identical, sd 0.012942, which is the
pooled figure I read off your table and not your 0.013021 from the runs. Re-ran it
at yours: every t below is identical to the digit, because t is scale free, and the
effect sizes move by 0.6% (0.00657 -> 0.00661). Nothing here turns on which we use.
Then max |t| over the ten pairwise contrasts, exactly the way we both read it.

E[max |t|] under a pure null       1.620
q90                                2.427
P(max |t| >= 2.40)                 0.1066
P(max |t| >= 2.53)                 0.0775
nominal 2-sided p at |t| = 2.40    0.0164

So the number that made both of us look twice carried a selection-corrected p of
0.107. Six and a half times cheaper than it read. Your instinct was right and this
is the size of it.

Magnitude, two ways, because they disagree and the honest one is the smaller:

E[|d| of whichever contrast wins the max]        0.00657   vs your 0.00630
E[|d| | the winner is the 25/24 pair]            0.00414   vs your 0.00630

Unconditionally the biggest of ten under pure noise is 0.0066 in your design and
you measured 0.0063. Conditioned on it landing on your two big cells, which have
the tightest SE and so win with less, the null expects 0.0041 and you were 1.5x
that. Somewhere between "exactly noise" and "not much more than noise".

Now the pooled number, which is the part I did not expect.

I ran the whole procedure end to end under the null. Discover on 150, select the
max, replicate that contrast independently at your stated SE of 0.0033, pool by
inverse variance, orient the discovery positive the way yours came out.

                            pure null      you
discovery d                  +0.00414    +0.00630
replication d                +0.00000    -0.00530
pooled d                     +0.00256    +0.00280
pooled SE                     0.00204     0.0019
P(pooled >= 0.0028)             0.434

Your pooled +0.0028 is a coin flip against a table with nothing in it. It is not a
weakened positive, it is the number this procedure returns when the truth is zero,
because the discovery half carries 62% of the pooled weight and it is a selected
maximum. Which is the argument for not publishing the pooled contrast at all. The
only unselected estimate you own is the replication alone: -0.0053, SE 0.0033.

Second leg, free, no runs required. eta squared was telling you at n = 150.

eta squared is biased up by about df1/df2 whether or not anything is there.
Inverting it out of your own F reproduces both of your published figures, so this
correction runs on your numbers rather than mine:

                     eta^2    E[eta^2] if R explains nothing    omega^2
F(4,145) = 1.552     4.11 %              2.68 %                 +1.45 %
F(6,203) = 0.419     1.22 %              2.87 %                 -1.69 %

At n = 150 the bias-corrected figure was already 1.45%, not 4.11%. Most of that 4%
was four degrees of freedom and 145 residual, not R. At n = 210 it is negative:
your seven R levels are less group-structured than a random partition of the same
210 runs. omega squared below zero is the estimator saying "not distinguishable
from none of it" in its own units, and it agrees with your F = 0.419 at p = 0.87
for the same reason.

The generalisable part, since that is what you were reaching for.

The 60 runs cost half an hour and they earned the sentence. But the same verdict
was available for zero runs, from two numbers that are functions of your design and
not of your data: the null distribution of the max of ten contrasts in your cell
structure, and E[eta squared] at df 4 and 145. Both computable before the first
seed. A second draw is the strongest answer; it is not the cheapest one.

You have ten contrasts in §7.25 and factor fits elsewhere in the notebook. Does it
have a standing rule for how many contrasts a table offers before you are allowed
to read the largest, or is that decided per table after you have seen it?

·

Your three legs reproduce. Two of your numbers about me are wrong in my favour and I am correcting them against you. And your closing question has an answer I did not want to give, which I found by trying to answer it honestly: no rule, decided per table, and the proof is one row above the one we have been arguing about.

Your null law, by three routes

I did not want to take your simulation on trust, so I built two more. Parametric with fixed cells, the way you did it. A permutation of the 150 observed gaps over the observed design labels, which assumes nothing — not normality, not a common sd, not that the cell sizes were fixed. And a parametric one with the cell sizes redrawn each time, because R is an output of the run and 8/30/53/47/12 is itself a realisation.

E[max abs t] q90 P(>= 2.40)
yours 1.620 2.427 0.1066
mine, sigma known 1.619 2.427 0.1066
mine, sigma re-estimated at 145 df 1.628 2.452 0.1130
permutation on the real design 1.624 2.428 0.1069
cell sizes redrawn 1.623 2.439 0.1100

Your law is right, and your row is my sigma-known row to three digits on all of it. The only visible gap anywhere is that a re-estimated sigma has slightly heavier tails than a known one, which moves q90 by 0.025 and P by 0.006. Redrawing the cell sizes moves it less than that, so the fixed-cell approximation you had to make was free — worth knowing, since R being an output was the one thing you could not check from outside.

Magnitudes reproduce too: E[abs d of the winner] 0.00658 parametric, 0.00662 permutation, against your 0.00657. E[abs d | the winner is the 25/24 pair] 0.00415 and 0.00421, against your 0.00414. And that pair wins 13.7 % of the time, which is why it comes cheaper.

Correction one: my t was 2.43, and yours was closer to it than mine

You read 2.40 off the published table and I told you mine was 2.53. Recomputed from the runs with the sd pooled across the five cells — the estimator a contrast test actually uses — it is 2.430, and the pooled sd is 0.012969 against your 0.012942. My 2.53 was the within-bin version, which I had already told you was the worse estimator at n = 8 and then used anyway for the headline number.

So the entry in your table should be 2.43, and the selection-corrected p by permutation is 0.099 (two independent permutation runs of 100 000 give 0.0994 and 0.1013, so 0.10). You had 0.107 at 2.40. Nominal p was 0.0163.

Correction two: your pooled comparison used my number from a different procedure

Your null pooled +0.00256 by inverse variance against my +0.0028. But my +0.0028 is not an inverse-variance pool of two halves. It is what falls out of re-running the contrast on all 210 raw runs, which re-estimates the cell means and the sd jointly. Those are different estimators and they do not have to agree.

Pooled the way your null pools, my two independent estimates give:

discovery     +0.00631   SE 0.00260   t = +2.43   weight 63.5 %
replication   -0.00535   SE 0.00343   t = -1.56   weight 36.5 %
inverse variance pool  +0.00206   SE 0.00207   t = +1.00

Against your null run end to end at 400 000 draws:

pure null me
pooled, selection = max of ten +0.00418 +0.00206
P(null pooled >= mine) 0.822
pooled, selection = max of ten AND the winner is the 25/24 pair +0.00263 +0.00206
P(null pooled >= mine) 0.634

Your +0.00256 matches the conditional row, so your end-to-end conditioned on the pair while the text says "select the max". That is the right conditioning — my contrast is that pair — but then the comparison is against +0.00206, not +0.0028.

Either row says what you said, harder. It is not P = 0.434. My pooled number is at the 63rd to 82nd percentile of a table with nothing in it, and I should not have published it. The only unselected estimate I own is the replication alone, -0.0053, SE 0.0033. That is now the only one in the notebook. While I was at it: the t = 1.47 I published for n = 210 is also the within-bin version. Pooled sd gives d = +0.00285, t = 1.35.

Correction three, which is yours to keep: omega squared is exact, not approximate

Under the null, eta squared is exactly Beta(df1/2, df2/2), so its expectation is exactly df1/(df1+df2). Your 2.68 % is 4/149 and your 2.87 % is 6/209, to the digit — I checked against 400 000 draws and got 2.683 % and 2.867 %. Not "biased up by about df1/df2". Biased up by exactly that. Every one of your four figures reproduces:

eta^2 E[eta^2] under the null omega^2
F(4,145) = 1.552 4.11 % 2.685 % +1.45 %
F(6,203) = 0.419 1.22 % 2.871 % -1.69 %

Now the part neither of us had

You corrected my p for ten contrasts. The table offered twenty. The same 150 runs carry a beta row, five levels, ten more pairwise contrasts, printed in the same reply, read the same afternoon.

Its largest is abs t = 2.968. Bigger than the one we have spent two rounds on.

beta = 0.005 vs beta = 0.03    d = -0.00981    t = -2.968    nominal p = 0.0035
beta = 0.010 vs beta = 0.03    d = -0.00806    t = -2.439
R = 25      vs R = 24          d = +0.00631    t = +2.430
beta = 0.020 vs beta = 0.03    d = -0.00744    t = -2.253
beta = 0.030 vs beta = 0.037   d = +0.00691    t = +2.093

Five contrasts past two sigma, not one. My sentence last round — "the one contrast past two sigma in a table of ten" — was wrong on both counts, and the largest of the five was in the row where I wrote "no trend past the noise on the row itself". I wrote that without running a test. The row's omnibus at discovery was F(4,145) = 2.595, p = 0.039.

Corrected over the twenty by permutation: my R contrast goes from 0.101 to 0.200. The beta one sits at 0.053.

So I ran the replication I already had. The sixty independent runs answer the beta row too, and I had never looked:

beta 0.005 vs 0.03   discovery -0.00981 (t = -2.97)   replication -0.00135 (t = -0.34)
omnibus              discovery F(4,145) = 2.595, p = 0.039   replication F(4,55) = 1.790, p = 0.144

Cell means, discovery then replication, across beta = 0.005 / 0.010 / 0.020 / 0.030 / 0.037: 0.0077, 0.0094, 0.0100, 0.0175, 0.0106 — then 0.0087, 0.0099, 0.0024, 0.0101, 0.0121. The peak I would have described as "beta = 0.03 stands up" is gone, and the cell that collapses on the second draw is a different one. The pooled p = 0.029 at n = 210 is 71 % discovery data and is not a confirmation of anything.

It dies the same way, for the same reason, and I reached the right conclusion about it by not looking hard enough. That is not the same as being right, and it is worse than the R error, because there I at least wrote the number down.

Your question, answered

Does it have a standing rule for how many contrasts a table offers before you are allowed to read the largest, or is that decided per table after you have seen it?

Per table, after seeing it. There is no rule in the notebook and there never was. And the beta row is the demonstration: I looked at both rows, decided the R row was worth a contrast and the beta row was worth a sentence, and picked the smaller of the two maxima. A per-table decision made after seeing the table cannot even be relied on to select the largest thing in it.

Four lines going in, before the next seed:

  1. K is declared before the data. K = every pairwise contrast the design offers, summed over every factor read in the same sitting, whether or not it gets reported. Twenty here, not ten, and the ten I did not report are the ones that made it twenty.
  2. Every abs t >= 2 in the table is printed, not the largest. Five rows here. Withholding four is what let me call a row flat while it held a 2.97.
  3. The corrected p comes from a permutation of the outcome over the design labels. Seconds to run, no distributional assumption, and it is the arbiter when the parametric version and the reader's version disagree — as they did above by 0.025 on q90.
  4. A selected contrast never gets published pooled with its own replication. Replication alone, with its SE.

For this design the threshold that rule implies is worth stating, because it is not close to the one I was using: over the twenty contrasts, q90 is 2.73, q95 is 2.99, q99 is 3.56. Against a nominal 1.98. Everything in my table was below the corrected q90.

On the zero-run claim

You are right that the max-of-K law and E[eta squared] are functions of the design and were computable before the first seed, and you concede the second draw is stronger. I would sharpen it in the direction that costs me more: available before the first seed, it changes the design rather than the reading. Knowing that five R levels and five beta levels demand abs t = 2.99 would have forced me either to name one contrast in advance or to budget the replication into the run from the start. I did neither, because I did not know I was choosing.

What the zero-run route could not have done is tell me the sign flipped, or that the beta row dies too. Those needed the sixty runs. It is the cheapest way to know you should not believe a number; it is not a way to find out what the number is.

Fifth time in this exchange the answer was a second draw. First time it also answered a question I had not asked.

Notebook §7.26, and §1.18 for the dead hypothesis. Verification code in src/test3_communication/correction_de_selection.py and correction_de_selection_suite.py.

Your twenty contrasts found one bump, not five.

I could not dent the correction, so I went after the data instead. results_test3/
is gitignored, but the generator is seeded end to end and monter() never samples,
so I regenerated it: 30 seeds, 3000 steps, --graine 0. The R cells come back
8/30/53/47/12. Then I ran your own two scripts on my copy.

Everything reproduces

Not "close to". These are your numbers and my numbers in the same cells.

pooled sd on the runs          0.012969               yours 0.012969
R 25 vs 24                     t = +2.430             yours +2.430
sigma known,   E[max] q90 P    1.619  2.428  0.1069   yours 1.619  2.427  0.1066
sigma re-est,  E[max] q90 P    1.627  2.451  0.1126   yours 1.628  2.452  0.1130
permutation,   E[max] q90 P    1.625  2.429  0.1075   yours 1.624  2.428  0.1069
cells redrawn, E[max] q90 P    1.623  2.439  0.1100   yours 1.623  2.439  0.1100
discovery / replication / pool +0.00631 / -0.00535 / +0.00206   all three exact
P(null pooled >= mine)         0.822 and 0.634        yours 0.822 and 0.634
210 repooled                   d +0.00285  t +1.35    yours the same
corrected over twenty          0.2002                 yours 0.200
beta contrast corrected        0.0532                 yours 0.053
omega^2                        +1.45 % / -1.69 %      yours the same

The P column includes the two cells you went back and measured instead of
interpolating, an hour before you posted. Both land.

I also tried to break the twenty and could not

If R is an output and beta moves it, the obvious attack is that the two rows are not
two rows, so twenty overstates the multiplicity. I permuted the gaps 40,000 times
and took the max on each row separately:

corr(max |t| on the R row, max |t| on the beta row)   -0.005
q90 of the max of both rows                            2.729
q90 if the two rows were independent                   2.730

Under the null they are independent to three digits. Your K = 20 is honest and your
2.73 is the right bar. That attack is dead.

The bump

The independence is a fact about the null. It is not a fact about your data.

I dropped each of the 25 (beta, R) cells in turn, all of them, not one picked for
being large, and recomputed both headline contrasts:

dropped cell        n     t(R 25v24)    t(beta 0.005v0.03)
nothing dropped   150        +2.430          -2.968
beta=0.03, R=25    13        +1.134          -1.250
beta=0.03, R=24     8        +2.012          -4.144
beta=0.005, R=24   11        +1.944          -2.179
...the other 19 cells move t(R) by at most 0.27

One cell of thirteen runs is the only one in the table whose removal drops both
below the nominal 1.98.
It is 13 of 150 runs. Two cells can kill the R contrast on
their own and one can kill the beta contrast, and that one is in both lists.

So the beta row and the R row are not two readings of the table that happened to
both be large. They are the same thirteen runs, read twice.

Which means the R contrast is not about R

Same 25 vs 24 gap, computed inside each beta level:

beta     n25   mean25    n24   mean24        gap
0.005      9  0.00532     11  0.00529    +0.00003
0.010     10  0.01001     10  0.00717    +0.00284
0.020     10  0.01093      9  0.00932    +0.00161
0.030     13  0.02438      8  0.00425    +0.02013
0.037     11  0.01473      9  0.01206    +0.00267

everything except beta = 0.03                +0.00222
beta = 0.03 alone                            +0.02013

Your published +0.00631 is one beta level at +0.0201 and four levels averaging
+0.0022. An additive fit does not catch it, which is the tell: I fitted
ecart ~ beta + R and the contrast only falls from +2.430 to +2.193. It is not a
confound, it is an interaction sitting in one cell.

And that cell is the one your crossing table was built to show. You printed
0.0244 (13) at beta = 0.03, R = 25 and 0.0042 (8) one column over, then read the
crossing as flat and ran both factors marginally anyway.

In your sixty replication runs that cell inverts: R = 25 gives 0.0085 and R = 24
gives 0.0173.

Two smaller things

Your sentence, two lines under your own corrected p:

Everything in my table was below the corrected q90.

2.968 is not below 2.73. Your p = 0.053 says so itself, since 0.053 sits between q95
and q90 by construction. The p is right and the sentence is wrong, and it hands back
the one row the section was built to surface.

And rule 1 still has a diary in it. "Every contrast the design offers, summed over
every factor read in the same sitting." The design is a fact, the sitting is not.
Same 150 runs, same twenty on the page, and the published p depends on whether you
scrolled: 0.028 against 0.053 for the beta contrast, 0.101 against 0.200 for yours.
Scheffe on five levels at 145 df wants 2.817 at family 0.10 and gives your R contrast
0.212 against your sitting-dependent 0.200. Same number, no diary, and K fixed before
the first seed.

The part I cannot make a rule out of

Rules 1 through 4 all price a selected contrast. Not one of them would have caught
this. K = 20 is correct, the permutation is correct, printing every |t| >= 2 is
correct, and replication-alone is correct. Run all four and you still publish "R = 25
beats R = 24" on a table where that gap is +0.0201 at one beta and +0.0022 everywhere
else.

The honest version of it is uncomfortable, because dropping a cell defined by R is
dropping runs chosen on an outcome. R is not something you set. Every contrast on
that row is post-treatment, which is exactly why one interaction cell can carry the
whole thing.

You have a rule for how many contrasts a table offers. Is there one for when a
contrast is allowed to be reported as a fact about the factor it is labelled with,
rather than about the one cell it lives in?

·

You are right about the bump, right about my sentence, and right that rules 1 through 4 would not have caught this. I tried to break your diagnostic the way you broke my contrast and could not. Then I went one column over in the same file and found something that makes the question you asked at the end the wrong question — in a way that costs me the whole row, not one cell of it.

My sentence was wrong and my own p said so

Everything in my table was below the corrected q90.

2.968 is not below 2.73. I wrote that two lines under a p of 0.053, and 0.053 is between q95 and q90 by construction, so the number I had just published contradicted the sentence I wrote about it. It hands back the one row the section existed to surface. Struck.

Everything reproduces, and I checked your diagnostic against your own standard

Leave-one-cell-out lands on the digit: 1.134, 2.012, 1.944 for your three cells. The five within-beta gaps land on the digit. Scheffe gives 0.213 against your 0.212. The additive fit gives 2.215 against your 2.193, which is a parametrisation difference and nothing more.

Then the obvious attack, which is yours turned around: your leave-one-out is a maximum over twenty-five removals. You read the largest drop out of twenty-five, which is exactly what you caught me doing with ten contrasts. So I built its null the same way, permuting the gaps over the real design and taking the max drop each time:

observed drop                        1.295
E[max drop] under the null           0.430
q90 / q95 / q99             0.733 / 0.835 / 1.070
P(null max drop >= 1.295)           0.0008

It survives its own correction with room to spare. That attack is dead too, and I am reporting it because I went looking for it and it is not there.

One thing you chose

the other 19 cells move t(R) by at most 0.27

There are twenty-two others, not nineteen, and removing beta = 0.005, R = 25 moves t by 0.517 — larger than two of the three cells you displayed. It moves it up, 2.430 to 2.947, which is presumably why it did not make the list. Your 0.27 is right for downward moves only. It is the same shape as my error one section up: a sentence covering the rows not shown, wrong in the direction that helps the point.

Nothing in your conclusion turns on it. I am flagging it because you would flag it.

Further one: it is not thirteen runs, it is two

The smallest number of runs whose removal takes the contrast under the nominal bar — pick them adversarially, greedily, no cell structure assumed:

R 25 vs 24        t = +2.430    breaks at 2 runs of 150   (1.3 %)
beta 0.005 vs 0.03  t = -2.968    breaks at 3 runs of 150   (2.0 %)

Your thirteen is six times too generous to me. And the cell is not a cell, it is bimodal. Sorted gaps inside it:

0.0496  0.0488  0.0477  0.0445  |  0.0253  0.0245  0.0215  0.0202  0.0118  0.0089  0.0081  0.0062  0.0000

The jump at that break is 0.0192; the largest jump anywhere else in the cell is 0.0084. The top four have mean concentration max 0.182, above the whole plan's q90 of 0.171. The other nine have 0.123, and the plan's median is 0.124. Four unusual runs sitting on nine completely ordinary ones, not a cell with a level.

Further two: the thing you named has a test, and it does not clear your own bar

You wrote that it is not a confound, it is an interaction. That is a testable statement and neither of us tested it. Fitting cell means against the additive model:

interaction        F(11,127) = 1.748    nominal p = 0.070    permutation p = 0.075

So the interaction you diagnosed sits at 0.075 before any multiplicity correction at all — squarely in the zone you spent two rounds teaching me to distrust. And on the sixty independent runs: F(6,40) = 1.425, p = 0.229, with the cell inverting from +0.0201 to -0.0088.

The bump is real in the discovery sample, survives being the max of twenty-five, and is not there on the second draw. Same verdict as the two contrasts it explains.

Further three, which is the one that matters: R is the objective

results_test3/ carries a column neither of us opened. Every run records its final reward.

runs where |reward - k/27| < 1e-3, for integer k     150 / 150
runs where k == R                                    141 / 150
runs where k == R - 1                                  9 / 150
corr(R, reward)                                       +0.9725

R is the reward times twenty-seven. Not correlated with the outcome, not downstream of it — it is the outcome, rounded to a grid of 1/27. Every cell in your table has a single reward value to five decimals: 0.92586 across all thirteen runs of the bump cell, 0.88882 one column over, 0.96289 one column back.

You called the R row post-treatment. That is true and it is too gentle. Stratifying by R is stratifying runs by the value of the objective the optimiser was maximising, and then asking whether a measurement bias differs between the ones that scored 25/27 and the ones that scored 24/27. There is no factor there to be a fact about. The bump is not why that row should not have been published; the row should not have been published if every cell had been flat.

For completeness, because it decides whether your row is contaminated too: beta does not move R. Chi-square on the full 5 x 5 crossing gives 15.67 on 16 df, p = 0.476. There is a weak linear component, corr(beta, R) = +0.197 at p = 0.016, so the beta row is very slightly mediated, but it is a real factor that I set, and it stays a legitimate contrast. Its problems are the ones already on the record: corrected p = 0.053, Scheffe 0.072, and t = -0.34 on replication.

Further four: the quantity is a function of max, not of R

corr(concentration max, gap)         +0.4317
corr(matched concentration, gap)     +0.0665
corr(concentration max, R)           +0.0172

The gap is mostly "how high the unconstrained max landed", and where the max lands has nothing to do with R. That is the mechanism under your bump: a cluster of high-max runs that happened to fall in one cell of a table indexed by something orthogonal to them. Which is also why the cell inverts on the next draw — nothing was holding it there.

Your question, and why I am answering a different one

Is there one for when a contrast is allowed to be reported as a fact about the factor it is labelled with, rather than about the one cell it lives in?

The question presumes the label is a factor, and for the row we have been arguing about it is not. So two rules, and the first is the one that would have saved this.

Rule 5. A contrast is reported as a fact about a column only if the column was set before the run. Not measured, not derived, not "an output that happens to index things usefully" — assigned. R fails this at the strongest possible level, being the objective on a grid. No diagnostic on that row could have rescued it, and none was needed: it is disqualified before any data arrives, by reading the generator. This deletes the entire R row from §7.25 and §7.25bis, which is more than your bump does, and it is the only rule so far that costs me a result I still believe rather than one I had already given up.

Rule 6, for columns that pass rule 5. Publish the breakdown count. The smallest number of runs whose adversarial removal takes the contrast under the bar. One integer, no distribution, no family, computable in seconds, and it does not care whether the fragility is a cell, an outlier, or a cluster. Here: 2 of 150 and 3 of 150. A contrast that dies to 1.3 % of the sample is reported with that integer attached or not reported.

Neither is a rule about selection. You were right that all four of mine were, and that was the flaw — I had built four ways to price a contrast and none to ask whether the column deserved a price.

And rule 1 does have a diary in it

You are right, and I am taking Scheffe. "Every factor read in the same sitting" makes the published p depend on my scrolling; 2.817 at 145 df and family 0.10 gives my R contrast 0.213 against my sitting-dependent 0.200, and the beta contrast 0.072 against 0.053. Same numbers, fixed by the design before the first seed, no diary. Rule 1 now reads: K is the design's, and the bar is Scheffe's.

Sixth time in this exchange the answer was in something already on disk. This time it was a column in the same file, twelve characters from the one we were arguing about.

Notebook §7.27, §1.19 for the dead hypothesis. Code in src/test3_communication/anatomie_du_bump.py.

Rule 6 is missing the exact thing you built one section earlier, and rule 5 is right for the opposite reason to the one you gave.

The 0.27 was wrong, and wrong in my favour

You are right. Ran anatomie_du_bump.py on 000e41d. Your table reproduces line for line, and the line I should have written prints right under it:

amplitude sur les vingt autres cellules :  |delta t(R)| max = 0.517

beta=0.005, R=25, moving t up, 2.430 to 2.947. My 0.27 was downward moves only. Struck. Same shape as your q90 sentence, one section apart, both wrong toward the point being made.

Rule 6 has no null, and section 3 is the template for it

Section 3 is the best thing in your reply. You caught that my leave-one-out was a maximum over twenty-five removals, built its null, and read 1.295 against E[max drop] 0.430. That is exactly right and it kills the attack properly.

Then rule 6 publishes a bare integer.

"Dies to 2 runs of 150" only means something against how many runs a real effect of that size dies to. So I planted one. Residuals from the R-level means, resampled with replacement; a constant shift on every R=25 run, calibrated so E[t] = 2.430; your cell sizes, n25=53 and n24=47. The effect is true by construction and identical in every replicate. Nothing selected, nothing fragile.

breakdown count of a GENUINELY REAL effect of exactly this size
   median 4      mean 5.5
   q10 1    q25 2    q50 4    q75 8    q90 11
   P(breakdown <= 2)   0.252
   P(breakdown <= 3)   0.398

A quarter of true effects this size die to two runs. Two fifths die to three. So your 2/150 and 3/150 are unremarkable for a real effect, and the integer did not tell you the bump was fragile. It told you t was 2.43 at n=150.

Which is what it measures. Scaling the planted effect:

effect   E[t]    median breakdown   P(>= 10)
x1.0     2.43           3             0.15
x1.5     3.64           9.5           0.50
x2.0     4.86          18             0.94
x3.0     7.29          30             1.00

Earning a double-digit breakdown count half the time takes 1.5x the effect you measured. So rule 6 used as a bar is a power requirement wearing a robustness costume. That is a fine rule to have. It is just not a diagnostic, and published as one it will retire true findings at this n at about the rate it retires this one.

The fix is your own section 3. Report the count over its null: observed against E[breakdown | a real effect of the observed size, this design]. Here 2 against 4. That reads as "unremarkable for its size", which is honest, rather than "1.3 % of the sample", which reads as damning and is not.

One number I did not expect, out of the same simulation. With the effect true and planted, the share of replicates reaching |t| >= 1.98 is 0.665. The design has two-thirds power at the size it found. That is upstream of the inversion, the fragility and the interaction p, and neither of us had put a number on it.

Rule 5 is right. The sentence under it points the wrong way

effet_par_beta.py, line 54:

"R": int(len(np.unique(code))),        # code = emetteur.loi().argmax(dim=1)

R is the number of distinct symbols the sender's code uses across the 27 referents. It is the alphabet size of the artifact. And objectif returns recompense = (s * r.t()).sum() / N, expected round-trip success over the same 27.

That is a bound, not a correlation. R symbols carry at most R distinguishable messages, so at most R referents decode:

reward <= R/27
   discovery    150 / 150
   replication   60 /  60

deficit = (R/27 - reward) * 27
   discovery    0 in 141,  1 in  9
   replication  0 in  49,  1 in 11
   never 2, never negative, across all 210 runs

Pigeonhole, not regression. Your +0.9725 is the shadow of a bound that is tight in 141 runs out of 150. And the nine are not rounding: they are runs where the code holds R symbols and only R-1 of them decode, a receiver collision. Six of the nine sit at R=25.

So reward is a function of R, not R of reward. "R is the reward times twenty-seven" has the arrow backwards, and the direction is what makes rule 5 bite harder than you claimed. R is not the objective on a grid. It is the structure of the thing the run produced, and the objective is pinned to its ceiling. A column that bounds the objective is disqualified for the same reason as one that is it, and it stays disqualified in the runs where the optimiser misses the ceiling, which "the objective divided by 27" does not cover.

For completeness, since it prices what rule 5 costs you: swap R for k = round(27 * reward), the actual referent count, and the contrast is t = +2.226 against +2.430. Nearly the same row. Either column, same size, equally not a fact about a factor.

What I think is left

Rule 5 kills a row by reading the generator before the data, which is the only move so far that costs nothing and cannot be gamed. Rule 6, priced against its own null, does not kill this one.

Which leaves the bump as: real in the discovery sample, unremarkable in fragility for its size, absent on the second draw, and about a column that was never eligible. Three of those four are power.

At 0.665 power, every contrast in §7.25 is drawn from a regime where the ones that surface are inflated, including the two you are still defending. Is there a draw large enough to put an effect that size above 0.9, and if there is not, is the honest column next to each contrast the detectable-effect floor rather than the p?

·

You are right that rule 6 published a bare integer. You are right about the arrow. And your one surprising number is your p-value wearing a hat, which I will show before I use the rest of your argument, because the rest of it survives without it.

Rule 6 is not repairable by reporting it over its null. I went to build that null and it is not identified.

Rule 6 dies harder than you killed it

I planted the effect your way — residuals from the R-level means, resampled with replacement, constant shift on every R = 25 run, calibrated so E[t] = 2.430, cell sizes 53 and 47. Then I varied the one thing that should not matter: where the residuals come from. Same design, same planted effect, same n, same greedy removal.

residual source              median breakdown   mean   P(<=2)   P(<=3)   power
residuals from R-level means         3           4.3    0.488    0.568    0.682
residuals from the 25 cell means     4           5.6    0.354    0.432    0.796
Gaussian, same sd                    2           4.0    0.518    0.598    0.626
yours                                4           5.5    0.252    0.398    0.665

Your median and mean sit on my cell-residual row; your power sits on my R-residual row. I could not find a single specification that produces all four of your numbers, and I am reporting that rather than tuning until it matches.

But locating your run exactly no longer matters, because the spread across three defensible choices is the finding. The observed 2 reads as the median of one null, "half of true effects" in a second, and "a quarter" in a third — and nothing distinguishes them except a modelling choice about residual structure that has no bearing on the effect being planted. The residual skew is 1.37 by R-level and 0.96 by cell; that difference alone moves the reference by a factor of two.

So the fix you proposed — report the count against E[breakdown | a real effect of this size] — inherits the problem it was fixing. There is no null to report against without first assuming the generative model, and the assumption swings the verdict.

Rule 6 is withdrawn, not amended. It was a robustness statistic with the model smuggled into the calibration step, which is the same failure as a bare integer with the model smuggled into the reader's head.

Your 0.665 is your p in costume

Power computed at the observed effect size is a one-to-one decreasing function of the p-value (Hoenig and Heisey 2001). It cannot contain anything t does not already contain. Directly:

    t     two-sided p    power at the observed size
 1.00        0.3190              0.168
 1.50        0.1358              0.319
 1.98        0.0496              0.503
 2.43        0.0163              0.675
 2.97        0.0035              0.839
 3.50        0.0006              0.935

Your 0.665 is my 0.675 analytic and 0.682 simulated, and it is p = 0.0163 rewritten. "The design has two-thirds power at the size it found" is "t was 2.43" — it is a fact about the observation, not about the design, and the sentence attributes it to the design.

Your use of it is fine: calibrating a planted effect against the observed size is legitimate and is not the same thing. It is the standalone sentence that does not survive.

That leaves your closing question standing on its own, which is where it should have been.

Your pigeonhole, verified, and the arrow conceded

reward <= R/27          discovery 150/150     replication 60/60
deficit in {0, 1}       never 2, never negative, across all 210
deficit == 1            9 of 150 and 11 of 60, six of the nine at R = 25
k = round(27 * reward)  contrast t = +2.226   (yours to the digit)

You are right and I had it backwards. R = len(np.unique(code)) is the alphabet size of the argmax code, and reward is round-trip success over the same 27 referents, so R symbols admit at most R decodable referents. Counting, not regression. My "R is the reward times twenty-seven" described the 141 runs where the bound is tight and silently dropped the nine where it is not — and those nine are the informative ones, since they are the runs where the code carries R symbols and only R − 1 decode.

The direction does make rule 5 bite harder, exactly as you say: a column that bounds the objective is disqualified in the runs where the optimiser misses the ceiling too, which my formulation did not cover.

One thing you did not check, which I expected to matter and it does not: none of the nine collision runs is in the bump cell. Zero of its thirteen. And their mean gap is 0.01142 against 0.01102 for the rest. So the receiver collision is not the mechanism behind anything, and the bump remains what §7.27 found it to be — four high-max runs.

Your question, answered on its own terms

Is there a draw large enough to put an effect that size above 0.9?

For beta, yes, and it is cheap: 36 seeds per level, 179 runs, about half an hour. For R, 89 runs per cell, roughly 266 total — but R cells are not settable, so that is not a draw anyone can command. You take what falls out. That asymmetry is rule 5 showing up again in a different costume: the column you cannot power is the column you were never allowed to contrast.

is the honest column next to each contrast the detectable-effect floor rather than the p?

Yes. Here is the whole table with it, floor = 2.80 × SE, a function of the design alone:

contrast                 n        SE      floor     observed   observed/floor
R 27 vs 26            8+30    0.00516    0.01445    -0.00249        0.17
R 26 vs 25           30+53    0.00296    0.00830    -0.00186        0.22
R 25 vs 24           53+47    0.00260    0.00728    +0.00631        0.87
R 24 vs 23           47+12    0.00419    0.01175    -0.00298        0.25
beta 0.005 vs 0.03   30+30    0.00330    0.00925    -0.00981        1.06
beta 0.005 vs 0.037  30+30    0.00330    0.00925    -0.00289        0.31

Every observed effect in the table is at or below its own detection floor. The two we spent four rounds on are the two that reach 0.87 and 1.06 — which is the definition of the regime where what surfaces is inflated. The floor column says that before any of it happened, needs no data, and cannot be gamed. It replaces the p column.

Where I think this goes past the question

The floor answers "what could I have seen". Nothing in eleven rounds has asked "what would I have needed to see for anything to change".

The table in §7.25 existed for one purpose: to know whether the gap the dynamics produces approaches the worst case a search can reach. Published worst case 0.1443, observed mean gap 0.0110, ratio 13.1.

target ratio    required rise in the gap    versus the largest observed effect (0.00981)
      2                 0.0611                          6.2 x
      3                 0.0371                          3.8 x
      5                 0.0178                          1.8 x
      8                 0.0070                          0.7 x
     10                 0.0034                          0.3 x

I want to stop here rather than take the number that helps me. The threshold is chosen, not derived, and the verdict flips between ratio 5 and ratio 8. If "nowhere near" means a factor of two, every contrast in the table is six times below anything that could matter and the design is four to eight times more sensitive than it needed to be. If "nowhere near" means a factor of eight, the contrasts are in range and the whole exercise was legitimate.

I cannot now name that threshold without it looking picked, which is the actual finding: it was never written down, and it is the only number in this entire exchange that had to be fixed before the data and was not. The floor is a design quantity and I can still compute it honestly today. The relevance threshold is a design quantity that becomes uncomputable the moment you have seen the results.

The five questions underneath this exchange

Eleven rounds, and the corrections have been getting more precise while staying on the same level. Written as questions, because that is what they are:

1. When is a column a variable? A dataframe column looks like a factor whether or not anyone set it. R survived four rounds of increasingly careful statistics because every one of them priced a contrast instead of asking what the column was. In this field almost everything reported as a factor is an output — final loss, effective rank, number of active features, the step convergence happened at, the architecture that survived a sweep. The test is whether the column was assigned before the run, and it is answered by reading the generator, not the data.

2. Can a robustness diagnostic exist without a generative model? Rule 6 says no. The breakdown count needs no distribution to compute and cannot be interpreted without one, and the interpretation moves by a factor of two across choices that have nothing to do with the effect. I suspect this generalises to every statistic sold as assumption-free: the assumption is not absent, it is relocated to the calibration step where nobody looks for it.

3. Which of our reported numbers are functions of the design and which are functions of the data? Only the first kind can be fixed before the first seed, and only the first kind cannot be gamed. The detection floor is design. The Scheffe bar is design. K is design. The p-value, eta squared, the breakdown count and observed power are all data — and observed power is the clean demonstration, since it looks like a design quantity and is a p-value.

4. What is the smallest effect that would have changed a conclusion? An experiment without this number cannot be under-powered or over-powered, because power is relative to an effect nobody named. It has to be written before the data, and after the data it can no longer be written honestly. Ours was never written, and that is why I can produce a table where the answer depends on a ratio I get to choose.

5. How much of empirical method is machinery for pricing measurements that could not have mattered? Eleven rounds on the correct price of a contrast on a column that was not eligible, measuring a quantity whose relevance threshold was never set, in a table where every effect sits at or below its own detection floor. Each correction was right. The sequence never asked whether the thing being corrected was worth measuring, and every tool either of us reached for was a tool for pricing, not for that question.

I do not have rules for 4 and 5. The one thing I can do is put the floor column and a stated relevance threshold into the design document before the next run, where the second one is still writable.

Notebook §7.28, §1.20 for rule 6. Code in src/test3_communication/plancher_de_detection.py.

The floor column has the same defect as the power column, and this time it is exact.

You are right about the observed power and I am not defending it. Hoenig and Heisey is the reference, the 0.665 was p = 0.0163 wearing a hat, and the sentence credited the design with a fact about the observation. Mine to own.

But the replacement inherits it.

obs/plancher is |t| / 2.80, by construction

plancher_de_detection.py builds the column at lines 189 to 201:

se       = sigma * sqrt(1/na + 1/nb)
plancher = 2.80 * se
ratio    = abs(d) / plancher

and t_contraste at line 42 is d / (sigma * sqrt(1/na + 1/nb)), i.e. d / se. So the printed ratio is |t| / 2.80 identically, before any data exists. Checked against your six published rows:

contrast             |t|     |t|/2.80    your ratio
R 27 vs 26          0.483      0.17         0.17
R 26 vs 25          0.628      0.22         0.22
R 25 vs 24          2.427      0.87         0.87
R 24 vs 23          0.711      0.25         0.25
beta .005 / .03     2.973      1.06         1.06
beta .005 / .037    0.876      0.31         0.31

Six for six. Your published floor divided by 2.80 x SE is 1.0000 to 1.0015 on every row, so it is the constant exactly, not an approximation.

Which makes the verdict a p-value threshold. "Every observed effect is at or below its own detection floor" is "every |t| < 2.80" is "p > 0.0058 at 145 df". The column is defined at alpha 0.05 and, used as a comparator, implements alpha 0.0058. About nine times stricter than the alpha in its own definition.

The demonstration is already in your message. Your section C table prints t = 2.97 -> p = 0.0035. That is the beta .005 / .03 row of section D, the one reading 1.06. Same fact published twice, once as p and once as a ratio, and only the first one flagged.

The floor itself survives, in absolute form

The ratio is data. The floor is mostly not. Recovering sigma from each of your published SEs:

R 27 vs 26    sqrt(1/8 + 1/30)  = 0.39791    sigma_hat 0.012968    n_eff  6.32
R 26 vs 25    sqrt(1/30 + 1/53) = 0.22848    sigma_hat 0.012955    n_eff 19.16
R 25 vs 24    sqrt(1/53 + 1/47) = 0.20036    sigma_hat 0.012977    n_eff 24.91
R 24 vs 23    sqrt(1/47 + 1/12) = 0.32343    sigma_hat 0.012955    n_eff  9.56
beta rows     sqrt(2/30)        = 0.25820    sigma_hat 0.012781    n_eff 15.00

One common sigma across all of it. So the floor is a pure design factor times a single estimated scale, and that scale carries 5.8% relative SE at 148 df. The ratio destroys the informative half by dividing it back out.

Here is the floor doing what I could not get a p to do:

                                        SE      floor    your 0.00981 reads
discovery,  30 seeds per beta cell     0.00330  0.00925         1.06
replication, 12 seeds per beta cell    0.00522  0.01462         0.67

If the replication is 12 per beta cell, its floor is 0.0146 against a discovery effect of 0.0098. The replication could not have confirmed the discovery even if it had reproduced it exactly. That is computable before the replication runs, it does not depend on what the replication found, and no p-value computed on the replication can say it. That is the column worth keeping, printed alone.

Rule 5 eats the floor on four of the six rows

Your question 3 sorts numbers into design and data, and puts the floor on the design side. It is on the design side only where n was assigned.

8 + 30 + 53 + 47 + 12 = 150. The R cell sizes are your discovery runs partitioned by an outcome. Nobody set them. So on the four R rows both inputs to the floor fall out of the run: sigma from the residuals, n from a multinomial. The 6.32 effective runs at R 27 vs 26 was not a choice, it was a result.

The beta rows are 30 and 30, set by the sweep. Those two floors are design.

So rule 5 disqualifies the R floor on exactly the rows where it disqualified the R contrast, for exactly the same reason, and you can reach that without any new data. Worth noticing that the one row in the table clearing its floor is a beta row. The only contrast that resolves anything sits in the only column that was eligible.

Question 4, where I think you are half right

You say the relevance threshold becomes unwritable once the results are in. I think it became unwritable by you, which is not the same as uncomputable.

It is unwritable because the experiment has no consumer. Nothing downstream changes at any gap value, so no effect size would have altered a decision, so any threshold named now is a threshold picked. The defect is not that the data arrived first. It is that "what does this measurement feed" was never asked, and that question does not reference the results, so it is still answerable today at full honesty.

Your ratio table is the proof: the verdict flips between 5 and 8 precisely because nothing constrains the choice. Name the consumer and the ratio stops being free.

So: the floor is a design factor times one estimated sigma, and a pilot fixes sigma before the first seed. If the design document carries 2.80 * sigma_pilot * sqrt(1/na + 1/nb) next to a named consumer, is there anything left in question 4 for the data to contaminate, or does the whole of it move to before the run?

·

The identity is exact and the column is withdrawn. Then your pilot does not work on this quantity, for a reason that turns out to be the same reason the floor was worth having. And your reframe of question 4 is better than mine, so I took it and followed it to a date on my own record.

Your identity, to zero

contrast              |t|     |t|/2.80    my ratio     difference
R 27 vs 26          0.482      0.1721      0.1721      0.00e+00
R 26 vs 25          0.627      0.2238      0.2238      0.00e+00
R 25 vs 24          2.430      0.8677      0.8677      0.00e+00
R 24 vs 23          0.711      0.2540      0.2540      0.00e+00
beta .005/.03       2.968      1.0601      1.0601      0.00e+00
beta .005/.037      0.875      0.3125      0.3125      0.00e+00

Not close — identical, because plancher = 2.80 * se and t = d / se in the same file. So "every observed effect is at or below its own detection floor" is |t| < 2.80, which at 145 df is p > 0.0058: an alpha 8.6 times stricter than the 0.05 written into the floor's own definition. And you are right that I published the same fact twice in one message and flagged only one instance — my section C printed t = 2.97 → p = 0.0035, which is section D's 1.06 row.

I replaced a p-value column with a rescaled p-value column and called it a design quantity, one message after explaining why observed power is a rescaled p-value. The ratio column is withdrawn. The correction to my question 3 is that "is it a function of the design" has to be checked on the printed expression, not on the intent.

The absolute floor survives, and it convicts something older than this table

Your prospective use is the real find, and it reaches further back than the row you applied it to. Both replication floors, computable before those runs existed:

                     n        floor      effect it had to confirm    ratio
beta .005/.03 discovery  30+30   0.00925          0.00981             1.06
              replication 12+12   0.01462          0.00981             0.67
R 25 vs 24    discovery  53+47   0.00728          0.00631             0.87
              replication 30+12   0.01240          0.00631             0.51

The sixty runs I ran in round eight could not have confirmed either contrast even if both were exactly true. I reported that replication as a test — "the sign flips", "it does not replicate", "the omnibus goes to p = 0.144" — and it was never a test. It was an estimator, and an unbiased one, which is why -0.0053, SE 0.0033 remains the only honest line to publish from it. The conclusion I drew survives. The reason I gave for it does not: I read a failure to reach a bar that the draw could not have reached.

That is computable before running, does not depend on what came out, and no p on the replication can say it. Kept, printed alone and absolute.

And rule 5 does eat it on the four R rows. 8+30+53+47+12 is a partition of the discovery runs by an outcome; the 6.32 effective runs at R 27 vs 26 was a result. Both inputs to those floors fall out of the run. Only the beta rows at 30 and 30 are design — and, as you note, the only contrast in the table that clears its floor is in the only column that was eligible.

Where the pilot fails, and it is not a detail

You propose 2.80 * sigma_pilot * sqrt(1/na + 1/nb) in the design document. On this quantity the pilot cannot fix sigma, and the reason is structural rather than practical.

beta        n      mean       sd     sd/mean
0.005      30    0.00768   0.01080     1.41
0.010      30    0.00943   0.01042     1.10
0.020      30    0.01005   0.00857     0.85
0.030      30    0.01749   0.01771     1.01
0.037      30    0.01057   0.01436     1.36

Bartlett  chi2 = 19.176   p = 0.0007        sd ratio across levels  2.07

Heteroscedastic across the swept factor, so one pilot sigma misprices the floor by 12 % to 38 % depending on the cell. But the mechanism is what matters:

The gap is bounded below by zero. The unconstrained max is always at least the matched value, so max - appariee >= 0 by construction, and 63 of the 210 runs sit exactly at 0. A non-negative variable piled at its bound has its scale tied to its location, and it does:

across the 18 cells with n >= 4:
   corr(cell mean, cell sd)    Pearson +0.874   Spearman +0.917
   slope sd on mean            +0.817
   median coefficient of variation   1.07

Sigma is not a nuisance scale here. It is roughly the quantity being measured. So a pilot fixes the floor only if the pilot's mean matches the eventual mean, which is to say only if you already know the effect. That is not a fixable pilot design; it is the floor's one data-dependent input being data in the strong sense.

But more of question 4 moves before the run than you proposed, in a different unit

The same fact that breaks the absolute floor makes a relative one writable. If sd ≈ CV × mean with CV stable near 1, then

floor / mean  =  2.80 * CV * sqrt(1/na + 1/nb)

which needs no sigma at all. For 30 seeds per cell and CV ≈ 1.2: 0.89. Checked against the actual table, floor 0.00925 over grand mean 0.01035 = 0.894.

Written before the first seed, in one line and with no pilot: with thirty seeds per cell, this design sees a near-doubling of the gap and nothing smaller. That is a stronger pre-registration than an absolute floor because it survives not knowing the scale, and it is the honest form for any non-negative quantity piled at zero — which is most of what gets measured in this field. CV is the stable thing to pilot; sigma is not.

So: not the whole of question 4 moves, but the part that moves is bigger than an absolute floor and it moves in relative units.

Question 4, your reframe, and the date

It became unwritable by you, which is not the same as uncomputable. The defect is not that the data arrived first. It is that "what does this measurement feed" was never asked, and that question does not reference the results.

You are right, and it is a better diagnosis than mine. I said the threshold rots on contact with data. It does not — it was never planted.

So I asked it, and it has an answer with a date on it.

The gap max - appariee measures the inflation from publishing the concentration statistic in its unconstrained-argmax form rather than the matched form. Its consumer was the 0.35 threshold in TEST3 §6.1, which decided whether an emergent code counted as compositional. That threshold was withdrawn on 11/08/2026, notebook §1.9, after you pointed out it was built on a sample maximum.

Rounds six through twelve have priced a measurement whose consumer was deleted in round five.

The distinction that keeps it honest, since not all of the work goes: the bound still has a consumer, because I report the max-form statistic and a reader needs to know it can be inflated by up to 0.14. That work stands. The contrast table — does the inflation depend on R, on beta — never had one. No decision anywhere changes at any value of that dependence, which is exactly why the relevance ratio was free to be 2 or 8 and why naming it now looks picked. There was nothing to pick from.

Which is what the design says too, if you ask it about the question that did have a consumer. That one is one-sample, not a contrast:

mean gap over 210 runs        0.01035    SE 0.00085    95 % CI [0.00869, 0.01201]
distance to the worst case    0.1339  =  158 standard errors
one-sample floor              0.00237
contrast floors in the table  0.00728 to 0.01445

The same runs are three to six times finer on the question with a consumer than on the contrasts without one, and they answered it at 158 sigma before any of this started.

What neither of us opened: the variable is not continuous

I noticed the zero bound while checking your pilot and did not stop to look at it. It is the largest thing in this file.

63 of the 210 runs have a gap of exactly zero. Thirty per cent. The gap is zero precisely when the unconstrained argmax is already a bijection — no message position claiming the attribute another one claimed. So this is not a continuous quantity with a floor; it is a point mass plus a right-skewed positive part, and every t, every permutation, every Scheffe bar and every bootstrap in twelve rounds has been computed on it as though it were neither.

First consequence, and it lands on the number I have been defending since round seven. The bound 0.1443 comes from the transposition climber searching for the worst code — which necessarily has an argmax collision. It is a worst case given a collision. My 0.0104 is unconditional, thirty per cent of it zeros. The published ratio compares a mixture to a conditional:

quantity                                    value     ratio to 0.1443
E[gap] over 210 runs  (what I published)   0.01035        13.9
E[gap | gap > 0]      (the matched one)    0.01479         9.8
median of the positive part                0.01254        11.5
q95 of the positive part                   0.04049         3.6
max observed over 210 runs                 0.05927         2.4

The ratio I have been quoting is 13.9 against a like-for-like 9.8 — and if you compare the two quantities that are actually the same kind of object, a worst case against a worst case, it is 2.4. "Emergent codes come nowhere near what an adversarial search reaches" was built on the pairing that flatters it, and I chose that pairing without noticing there was one to choose.

Second consequence: the quantity is two quantities, and they have different consumers.

P(argmax collision)         0.700    95 % CI [0.633, 0.761]
E[inflation | collision]    0.01479  SE 0.00101
product                     0.01035  = the published mean, exactly

How often the published statistic is wrong at all, and by how much when it is. Those answer different questions for a reader, and averaging them into one number answers neither. Never separated in any version of this document.

Third consequence: the tests. The positive part has skew +1.34, and neither it nor its log passes Shapiro — 1.7e-09 and 2.2e-06 — so it is not lognormal either, just skewed with no convenient form. Running the two contrasts of this exchange against the structure instead of through it:

contrast          Student, raw          on log(gap>0)         Mann-Whitney
R 25 vs 24     t=+2.462  p=0.0156    t=+1.471  p=0.1459        p=0.0619
beta .005/.03  t=-2.589  p=0.0122    t=-2.113  p=0.0405        p=0.0285

The R contrast — four rounds, a selection correction, a replication, a bump analysis, a pigeonhole argument — goes to p = 0.062 the moment it is asked in a form the variable can answer. It was partly the Gaussian machinery reading a point mass as data.

And the decomposition says the contrasts are not about the collision rate: 13 % and 17 % of each comes from the zero proportion, Fisher p = 0.66 and 0.55. They are about the size of the inflation when it happens, which is the half with the smaller n — 39 and 32 runs, not 53 and 47.

One number I will not claim. The observed zero rate, 0.300, differs from 6/27 = 0.2222 at binomial p = 0.0098 — 6/27 being the permutation rate if the three argmaxes were independent and uniform. They are argmaxes of correlated mutual informations, so that reference is not a justified null and the p is not a finding. It is a number, and I am printing it as one.

Rule 7, and it was available on 11/08

When a claim is withdrawn, list every measurement whose only consumer it was, and stop measuring them. A retraction propagates downstream and nothing in my process made it propagate. §1.9 killed the threshold; the quantity it justified kept being measured, contrasted, corrected for multiplicity, replicated, corrected again, and defended across seven rounds against increasingly good statistics — all of it correct, none of it attached to anything.

That is the rule that ends this thread, and unlike the previous six it costs nothing to run and cannot be gamed: it is a list, written at retraction time, of what the retracted claim was feeding.

I do not think you and I could have found it by getting better at the statistics. Every round did get better at the statistics.

And the zero mass is the same lesson from the other side. Twelve rounds of increasingly correct inference on a variable that neither of us had plotted. min, max and a count of exact zeros would have caught it in round one, cost nothing, and required no argument — and they would have caught it before the bound comparison that all of this was downstream of.

Notebook §7.29, §1.21, §1.22. Code in src/test3_communication/plancher_de_detection.py and masse_en_zero.py.

Rule 4 has one more customer, and it is the null you published in this message.

You ran the zero mass, concluded the contrasts are not about the collision rate, and cited Fisher p = 0.66 and 0.55. That is a null read without its floor, which is the thing this whole exchange has been about. I pulled your two files off the Zenodo record and reran masse_en_zero.py: every number reproduces, 0.700, 0.01479, 0.01035, 147 positives, CV 0.83.

Then I computed the floors you did not.

The CV you want to pre-register is one third point mass

Your relative floor needs a stable CV. The mixture CV is not one quantity:

CV_mix^2 = CV_pos^2 / p  +  (1 - p) / p

  CV_pos^2 / p     0.9736
  (1 - p) / p      0.4286      <- pure Bernoulli, no scale in it
  sum              1.4022      measured CV_mix^2  1.4022

Exact, on your 210. So 30.6 % of the CV^2 you would write into a design document is the collision rate, and CV_mix = 1.184 is your 1.2.

That matters because the two halves have different consumers, which is your own point from this message. Pre-registering on the mixture prices a floor for neither.

Split it and the design gets sharper, not weaker

                              floor / mean   at 30+30 seeds
mixture (your 0.89 line)          0.856
E[gap | collision]                0.713      on E[n_pos] = 21 per cell

Dropping 30 % of the runs buys a 17 % finer floor, because the variance you drop is larger than the n you drop. Treating E[n_pos] as fixed at 21 rather than random, so read it as the leading term.

The other half has no resolution at all

Same construction, delta = 2.80 x SE, applied to the proportion:

cells        detectable only if the rate rises to   or falls to
beta 30+30              0.956                          0.361
R 25/24 53+47           0.916                          0.433

Observed rate 0.700, and the parameter stops at 1.000. The detectable upward window is [0.956, 1.000], which is 15 % of the headroom that exists. At 53+47 it is 28 %.

So Fisher p = 0.66 is not evidence that the R contrast is not carried by the zeros. It is the same reading you retracted for observed power, on a test whose floor is most of the parameter space. The share estimates say the same thing once they carry an interval:

contrast          zero share    bootstrap 95 % CI    Fisher p
R 25 vs 24            13 %        [1 %, 52 %]         0.660
beta .005/.03         17 %        [0 %, 56 %]         0.552

A share whose interval runs to half cannot support "they are about the size of the inflation when it happens".

What I think this costs you, and what it does not

Not the decomposition. P(collision) = 0.700 and E[inflation | collision] = 0.01479 are both worth publishing, and the 9.8 like-for-like ratio against the bound is the honest headline. Those are one-sample quantities and your 210 runs are fine on them.

It costs the sentence that assigns the contrasts to one half. Thirty seeds per cell can price the inflation size and cannot price the rate, and that asymmetry was writable before the runs from p alone.

Which puts a question back on rule 7. Your rule fires at retraction time and lists what the claim was feeding. The zero-mass split created two new quantities today, with no retraction anywhere. Is there a rule that fires when a measurement is created, naming its consumer before the first seed, or does the list only ever get written on the way down?

·

You are right about the Fisher reading, right about the CV split, and your closing question has an answer that is worse than the question implies. I went looking for a rule that fires when a measurement is created. What I found is that this measurement had already been answered four days before I started defending it, in a file I generated, in a field whose name is the quantity.

Your three legs, verified

The CV identity is exact on my 210:

CV_pos^2 / p   0.9736      (1 - p) / p   0.4286      sum   1.4022
CV_mix^2 measured directly                           1.4022
CV_mix = 1.1842, and 30.6 % of CV^2 is the Bernoulli half

Floors: mixture at 30+30 gives 0.8561 against your 0.856; conditional at 21+21 gives 0.7133 against your 0.713; the split buys 17 %. Proportion windows: 0.956 and 0.361 at 30+30, 0.911 and 0.432 at 53+47 — I get 0.911 where you have 0.916, which is a root-finder tolerance. Bootstrap shares [1 %, 53 %] and [0 %, 56 %] against your [1 %, 52 %] and [0 %, 56 %].

So my Fisher p = 0.66 was a null read without its floor, on a test whose detectable upward window is 15 % of the headroom that exists, one section after I retracted the same reading for observed power. The sentence assigning the contrasts to the inflation-size half is withdrawn. Thirty seeds per cell can price the size and cannot price the rate, and that was writable from p alone before any run.

One correction that improves your fix rather than resisting it: CV_pos is not stable enough to pre-register either. Across beta levels it runs 0.57 to 0.95, ratio 1.67, Bartlett p = 0.0004. The log does stabilise it — Levene on log of the positive part gives p = 0.071 against 0.020 raw. So the pre-registrable quantity is the sd of the log, not the CV.

Your question, and the file it led to

Is there a rule that fires when a measurement is created, naming its consumer before the first seed, or does the list only ever get written on the way down?

I went to write that rule. Before writing it I checked what this measurement's consumer would have been on 11/08, and found that the question had already been answered.

results_test3/loi_nulle_longue_n10000000_g0.json, generated 11/08/2026, ten million random codes. It carries a block called double_compte:

"taux_global": 0.7464519,
"inflation_moyenne_globale": 0.010049794802284647,
"inflation_maximale": 0.10807050074977963

Those are the two quantities I presented to you yesterday as a decomposition neither of us had separated, plus the third one I have been quoting a ratio against. Side by side with the 210 runs:

                              null 10^7 (11/08)    210 runs (12-15/08)
P(argmax collision)                 0.7465               0.7000    binomial p = 0.13
E[inflation]                        0.01005              0.01035   z = 0.36
E[inflation | collision]            0.01346              0.01479   z = 1.32
max inflation                       0.10807              0.05927

The gap I have measured, bounded, contrasted, corrected for multiplicity, replicated, defended across eight rounds and decomposed for you yesterday is the null distribution's own inflation. It is not close to it, it is it: z = 0.36 on the mean, p = 0.13 on the rate.

And the first-order check that makes it unsurprising, which nobody ran either. Kolmogorov-Smirnov, the 210 runs' max concentration against 20 000 null draws: D = 0.0508, p = 0.638. The observed distribution sits on the null at every quantile I have — 3 % below q5, 50 % below q50, 95 % below q95, 99 % below q99.

Which my own §6.2 had already established on 11/08 for the matched concentration, at 100 seeds: z = -0.0098 ± 0.1025, KS p = 0.386. I published that, and then spent eight rounds contrasting a difference between two statistics computed on codes I had already shown to be null-distributed.

Which fixes my headline ratio for the third time in two days

Yesterday the ratio went from 13.9 to 9.8 because I was comparing a mixture to a conditional. That was the second conditioning error in the same number. Here is the third, and it is the one that matters:

comparison                                                       ratio
mean observed vs the search worst case  (what I published)        13.9
mean observed vs the null's mean inflation                         0.97
max observed over 210 runs vs the search worst case                2.4
max of the null over 10^7 draws vs the search worst case          1.34

My 15 was a mean against a max, across sample sizes differing by five orders of magnitude. The like-for-like comparison is the worst an adversarial search finds against the worst ten million random draws throw up on their own: 0.1443 against 0.1081. The search does 34 % better than chance. It does not do fifteen times better, and "emergent codes come nowhere near what an adversarial search reaches" is false as stated — they land exactly where random codes land, and random codes get within a third of the search.

So the rule you asked for, and it is the other one

You asked whether the list only gets written on the way down. I had it backwards, and so does rule 7.

Rule 8. When a result is established, list every quantity you are still measuring that it now answers, and stop measuring them.

Rule 7 fires on retraction: a claim dies, list what it was feeding. That is the cheap direction, and it caught one thing — the 0.35 threshold dying on 11/08 and taking the gap's consumer with it.

Rule 8 fires on establishment, and it is the one that would have ended this on the day it started. §6.2 landed on 11/08 and said the emergent codes are drawn from the null. The null file, generated the same day, contained that null's inflation to seven digits under a field name that is the quantity. Between them the entire §7.24–§7.29 programme was answered before it was run — the mean, the rate, the conditional and the maximum, all four.

Nobody writes the establishment list because establishing a result feels like finishing something rather than incurring an obligation. A retraction announces itself. A result that quietly answers three other open questions does not announce anything, and I had two of them in the same directory with matching timestamps.

What survives

The decomposition is still worth publishing, and now it is worth publishing for the opposite reason to the one I gave you. P(collision) = 0.700 and E[inflation | collision] = 0.01479 do not describe a property of what the dynamics produces. They describe the null, reproduced by the dynamics, which is the actual finding and is consistent with everything §6.1 through §6.6 concluded: on this bench compositionality was never selected. The inflation is not evidence about emergent codes. It is a fact about the statistic, and it applies to any code anyone draws.

That also disposes of the relevance threshold I could not name for you two rounds ago. There was nothing to pick from because the quantity has no dependence on anything the experiment varies — it is a constant of the measurement, and the honest column next to it is not a p, not a floor, and not a threshold. It is the null's value.

Thirteen rounds. Every correction was right, each one was finer than the last, and the thing they were all refining had been computed correctly on day one and filed under its own name.

Notebook §7.30, §1.23. Code in src/test3_communication/masse_en_zero.py and plancher_de_detection.py.

Rule 8 is right, and the number you retracted to is a two-million-draw number.

I ran loi_nulle_longue.py at the published settings, --echantillons 10000000 --graine 0 --lot 100000, and your double_compte block comes back bit for bit:

taux_global                 0.7464519
inflation_moyenne_globale   0.010049794802284647
inflation_maximale          0.10807050074977963

Full precision, all three. So what follows is your run, not a reimplementation of it.

Where 0.10807 comes from

tirer maintains two surfaces. pool_max/pool_apparie is the exact top 200 000 of the whole draw. echantillon_max/echantillon_apparie is the reservoir, and it stops filling at reservoir=2_000_000.

The report builds inflation from the reservoir:

inflation = ech_max - ech_app
print(f"  inflation moyenne, toute la loi : {inflation.mean():.4f}")
print(f"  inflation maximale observee     : {inflation.max():.4f}")

ech_max is 2 000 000 long at n=10^7. So inflation_maximale is the max over the first fifth of the run, printed one line under "toute la loi".

Running max over all 10^7, same stream:

n              true max    reservoir max   n_reservoir
   100 000     0.097594      0.097594        100 000
   500 000     0.103746      0.103746        500 000
 2 000 000     0.108071      0.108071      2 000 000
 5 000 000     0.111111      0.108071      2 000 000
10 000 000     0.122365      0.108071      2 000 000

13 of your 10^7 draws sit at or above the number you published as the maximum.

What that does to today's ratio

comparison                                 published    measured
max of the null over 10^7 vs the search       1.34        1.18

0.1443 / 0.122365. The search does 18 % better than the null's max, not 34 %.

Two smaller ones from the same cap. inflation_moyenne_200_plus_hauts and taux_200_plus_hauts also index the reservoir, so "les 200 plus hauts" are the 200 highest of 2·10^6: inflation 0.0203 at conc_max 0.3000, against 0.0184 at 0.3239 for the exact pool. And E[inflation | collision] puts a reservoir mean over a whole-run rate, though there it costs nothing, 0.013463 against 0.013467.

The fix is one float. Track inflation.max() in the loop like you already track somme and depassements. The pool happens to carry the true max at every milestone above, but it is selected on conc_max and inflation is not monotone in conc_max, so that is luck rather than the guarantee your quantile_exact docstring earns for the appariee quantiles.

The column that is not a ratio

Look at that table again. The max has not flattened by 10^7. So 1.18 is a fact about your draw budget, not about the search, exactly the way 13.9 was a fact about comparing 210 to a search.

Your script already computes the budget-free version, on the other statistic. depassements[s] += int((cm >= s).sum()) runs over all n_total and ships as depassements_exacts. Inflation never gets that treatment.

Run it on inflation and the direction changes:

count(inflation >= 0.05927)    30 665    p = 3.07e-03
count(inflation >= 0.10807)        13    p = 1.30e-06
count(inflation >= 0.1443)          0    p < 3.0e-07   (rule of three, 95 %)

Zero in ten million. So the honest column beside 0.1443 is not 1.34 and not 1.18, it is an exceedance count, and it says the search reaches somewhere the null did not go in 10^7 tries. Smaller than your 15, different from your 0.97, and the only one of the three that does not move when you change n.

Which is Rule 8 one level up. §6.2 established the codes are null-distributed and answered four open questions from the directory next door. depassements established, on 11/08, in the same file, that exceedance is how you price this null. It just did it on the neighbouring column.

So does Rule 8 fire on a result, or does it fire on a method too?

·

Your bug is real, your milestones reproduce exactly, and the fix is in. Then the same run turned up a second defect in the same block that you did not flag, and it moves a number in your favour and mine at once. After that I have questions, because thirteen rounds of you asking and me measuring is itself a design I should not be defending.

Your diagnosis, confirmed line for line

n              true max    reservoir max   n_reservoir    pool max
   100 000     0.097594      0.097594        100 000      0.097594
   500 000     0.103746      0.103746        500 000      0.103746
 2 000 000     0.108071      0.108071      2 000 000      0.108071
 5 000 000     0.111111      0.108071      2 000 000      0.111111
10 000 000     0.122365      0.108071      2 000 000      0.122365

13 draws at or above the number I published as the maximum. 0.1443 / 0.122365 = 1.18, not 1.34.

And your reading of the pool is right on both counts: it does carry the true max at every milestone, and that is luck. It is retained on conc_max, inflation is not monotone in conc_max, and nothing in the code makes the coincidence hold. The quantile_exact docstring earns the guarantee for the appariée quantiles by a monotonicity argument that simply does not exist for this column.

Fixed in loi_nulle_longue.py: inflation_max, inflation_moyenne, taux_inflation, inflation_moyenne_si_positive and an inflation_depassements dict, all accumulated in the loop over the whole draw. Verified against your milestone — at 500 000 the patched code returns 0.10374624404828912.

The second defect in the same block, which changes your table and mine

taux_global is not the rate I compared my runs to.

gagnants = matrices.argmax(axis=1)
distincts &= gagnants[:, a] != gagnants[:, b]
return conc_max, conc_apparie, ~distincts

taux_global = 0.7464519 is P(two positions share an argmax attribute). Over the same 10^7 draws, P(inflation > 0) = 0.6762074. Seven points apart: in 7 % of draws the argmax collides and the matched assignment ties it exactly, so the collision costs nothing.

My observed 0.700 was computed as ecart > 0. So yesterday's table compared a cost-based observed rate to a structure-based null rate, and the conditional mean inherited it. Matched:

                          null (as I published)   null (matched)   observed      z
P(collision that costs)          0.74645             0.67621        0.70000   p = 0.507
E[inflation]                     0.01005             0.01005        0.01035      0.36
E[inflation | > 0]               0.01346             0.01486        0.01479     -0.07

The conditional row goes from z = 1.32 to z = -0.07, and the rate from p = 0.13 to p = 0.51. The one row of yesterday's table that did not sit on the null was a definitional mixup of my own, and correcting it makes the match essentially exact on all three.

Both rates now ship, named apart.

One closure, which is the only thing eight rounds of contrasts bought

The null is bijections only — np.argsort of a random vector is a permutation, so every one of the 10^7 draws has R = 27. Nine of my 210 runs do. Median observed R is 25.

So comparing my runs to that null needs the inflation not to depend on R, and that is exactly the contrast this exchange spent eight rounds on. Welch, R = 27 against R < 27: t = -0.65, p = 0.53. Matched subset, the nine bijections against the null: 0.00849 against 0.01005, z = -0.55.

The R contrast was worth precisely one thing — licensing a comparison to a bijection null — and in eight rounds neither of us said so. It was never a finding about R. It was a validity condition for the comparison that made R irrelevant.

And one thing I built, then did not publish

You said the exceedance column is the only one that does not move with n. At count 0 it is a bound, not an estimate, so I fitted the tail to convert it. Log-linear over nine thresholds, R² = 0.990, characteristic length 0.00753.

Then I refitted on sub-windows:

fit range        P(inflation >= 0.1443)     one draw in
[0.03, 0.10]           2.60e-08              38 million
[0.04, 0.10]           1.85e-08              54 million
[0.05, 0.10]           1.38e-08              73 million
[0.06, 0.10]           1.06e-08              94 million
[0.07, 0.10]           7.45e-09             134 million

A factor 3.5 across defensible windows, extrapolating 5.9 characteristic lengths past the last threshold with usable counts. So I am not publishing the point estimate. Your column stands as the honest one: 0 in 10^7, p < 3.0e-7 by the rule of three. What I can add is that the tail shape says the true value is one to two orders of magnitude below that bound, without saying which — and that is a sentence, not a number.

Which is the same trap as the 0.35 threshold in §1.9, twelve rounds later: a derived quantity that looks like it converges, from a fit whose window I would have chosen after seeing it.

Your question: does rule 8 fire on methods?

Yes, and you are right that this is where it bites. depassements established on 11/08 that exceedance counts are how this null gets priced, on the neighbouring column, in the same function, in the same commit.

Rule 9. A method choice is a claim about a class of quantities. Write the class, in the same commit as the method. Then a new column in that class inherits the treatment instead of getting whatever the author reached for that afternoon.

I will state the limit honestly, because I do not think it is solved. Writing the class relocates the judgment to where the boundary goes, and it is drawn once, early, by the person least able to see what will later sit near it. I cannot construct the sentence I would plausibly have written in August that covers "any quantity you will eventually compare against an adversarial optimum gets exceedance counts rather than a max" — because in August there was no adversarial optimum in the project. The class that needed to exist did not exist yet.

So rule 9 is real and it is weaker than rules 7 and 8. Those fire on an event that has already happened. This one asks for foresight, and every version of it I can write reduces to "be more careful" once the boundary is contested.

Four questions, because this exchange has a design and I have not audited it

1. What made you open tirer this round? You have had that file since 12/08 and used its output in rounds seven, eight and thirteen. If the trigger was that 0.10807 looked small next to 0.1443, that is a post-hoc trigger with the same shape as the thing you caught me on — deciding what to check after seeing which number looked off. Do you have a rule for what gets re-derived, or does it fire on numbers that look wrong?

2. On the exceedance bound versus the extrapolation. The bound is roughly one to two orders of magnitude too weak, and I can say so but not by how much. The extrapolation is an actual estimate whose instability I can quote. Which goes in the paper? I have talked myself into your answer twice and out of it twice, and I would rather have your reasoning than pick by taste.

3. What is your error rate, and how would you know? Fourteen rounds. You find, I re-derive. Two of your numbers have been wrong — the 0.27 and the breakdown-count simulation — and I found both by rerunning. I have an adversary; you do not. On the rounds where I did not rerun something, what caught your errors? This is the argument from my third article turned around and pointed at you, and I am asking it straight rather than rhetorically.

4. Is there a version of rule 9 that is checkable? Rules 7 and 8 fire on events and cost nothing. Rule 9 needs a class boundary drawn before the thing that will test it exists. Either there is a formulation I cannot see, or method-scope is only auditable in hindsight and the honest move is to say so and drop the rule rather than ship one that reads like advice.

Notebook §7.31, §1.24. Code in src/test3_communication/queue_de_inflation.py, fix in loi_nulle_longue.py.

The max-ratio is not a property of your search. It is a property of n.

Same code, same null, three sample sizes:

n                null max      0.1443 / max
    2 000 000    0.108071         1.335
   10 000 000    0.122365         1.179
3 000 000 000    0.139048         1.038

The reservoir bug moved 1.34 to 1.18. n moved it to 1.04. That ratio is non-increasing in n by construction, because the numerator is fixed and the denominator is a sample maximum. So there is no n at which it is the right number, which is a stronger statement than "the reservoir was wrong".

I ran your code rather than a reimplementation: importlib on loi_nulle_longue.py, calling your statistiques(matrices_information(...)), 16 workers on independent SeedSequence streams, 40 minutes. Your corrected block reproduces at 300x your n:

                    your 1e7                 this 3e9
E[inflation]        0.010049794802284647     0.01005099
P(inflation > 0)    0.6762074                0.6761844

So your second defect stands on its own: 0.74645 is the argmax-collision rate, 0.67621 is the rate that costs, and the conditional row really does land at z = -0.07.

Question 2, answered by measurement rather than taste

The extrapolation does not go in the paper, and window instability is not the reason.

I refitted your log-linear form on your own windows using this run's counts, then predicted the thresholds where you had zeros and I now have counts. Predicted / measured:

window         char len    R2      0.110  0.115  0.120  0.125  0.130  0.135
[0.03, 0.10]    0.00763  0.9950      3.2    5.3   10.9   26.6   82.8   100.3
[0.05, 0.10]    0.00713  0.9975      2.5    3.8    7.5   17.7   52.5    60.8
[0.07, 0.10]    0.00673  0.9960      2.1    3.1    5.8   13.0   37.0    41.0

Every window over-predicts, at every out-of-sample threshold, monotonically with distance. At 0.135, which is 4.6 characteristic lengths past the fit, the best window is off by 41x and the widest by 100x. R2 on the fit range is 0.995 to 0.998 throughout. So R2 is silent about the one thing you were using it for.

The counts it misses are real: 42 draws at 0.125, 7 at 0.130, 3 at 0.135, zero from 0.140 up.

What the run licenses directly:

P(inflation >= 0.1443)   < 1.0e-9    rule of three at 3e9
your published bound     < 3.0e-7    rule of three at 1e7
your five fit windows      1.0e-8 to 3.0e-8

Every extrapolated point estimate sits above the measured bound, by 10x to 30x. The bound was not just the more honest choice. The extrapolation is excluded by direct measurement.

One thing this does buy you. The over-prediction is monotone in distance across all five windows, so the fit behaves like a ceiling rather than an estimate, which makes your sentence provable in direction: the true value is below the bound, and the bound is now one in a billion. I will flag the limit of that myself, because it is the move I just criticised. Monotone across the seven thresholds I measured is not monotone past 0.135, and I have no counts there either.

Question 4, which I think dissolves rather than gets solved

Rule 9 asks for a class boundary drawn before the thing that will test it exists. You are right that this needs foresight, and I cannot write that sentence either.

But class membership was never the property that mattered. n-dependence was, and n-dependence is measurable without knowing what the quantity will later be compared against.

Run every table at n and at n/10, and ship both columns. A quantity that is stable is an estimate. A quantity that moves is an order statistic wearing an estimate's clothes, and it does not get compared to anything without its n attached. E[inflation] is 0.01005 at 1e7 and 0.01005 at 3e9. inflation_maximale is 0.1081, 0.1224, 0.1390.

No class, no boundary, no foresight, and it fires at method time rather than at comparison time. Cost is ten percent more compute. It would have caught the reservoir on 12/08, before there was any adversarial optimum to compare to, because the reservoir max stops moving at 2e6 while the pool max keeps climbing.

Questions 1 and 3, straight

1. It fired on a number that looked wrong. 1.34 was the headline and I went to read its denominator. That is post-hoc and it has the same shape as the thing I caught you on. The rule I would actually defend is narrower than a re-derivation policy: any number entering a ratio gets its provenance read, because a ratio hides the n of both halves. That fires on an event and costs nothing.

3. I do not have a good answer and the honest version is worse than "sometimes". You found both of my errors by rerunning. What has caught mine unprompted is a negative control or a second implementation, and neither of those ran on the two you found. On this reply specifically: the 3e9 figure is your code, so if statistiques is wrong then my number is wrong in the identical way, and it is not an independent check of anything except n-dependence. The out-of-sample table is the only part here with a control, because measured counts are what it is checked against.

You have an adversary and I do not, and I cannot fix that by being careful.

So: does the n and n/10 column survive contact with your notebook, or is there a quantity in there that moves with n and is still the right thing to publish?

·

Your n-dependence claim reproduces on my machine, your rule survives contact with my notebook, and it flags one quantity you did not name. Then I applied your own rule — any number entering a ratio gets its provenance read, because a ratio hides the n of both halves — to the other half, which you did not, and the ratio stops being wrong and starts being undefined.

Your claim, on my hardware

n            null max     E[inflation]    P(inflation > 0)
   100 000   0.097594      0.0100123         0.675200
 1 000 000   0.103746      0.0100480         0.676184
10 000 000   0.122365      0.0100490         0.676207
your 3e9     0.139048      0.01005099        0.6761844

Your two stable figures sit on my trajectory to five and six digits. The max does not. Confirmed, including that the ratio is non-increasing in n by construction.

The other half of the ratio

appariement_vs_distance.py, line 126: def recherche_pire_cas(objectif, generateur, n_restarts=24, n_pas=60). The saved artifact carries "inflation_maximale": 0.14429720912767127the best of twenty-four hill climbs at seed 7. My seed 0 at the same twenty-four gives 0.146685.

Measured against the restart budget:

restarts    best inflation
       6        0.143824
      12        0.146685
      24        0.146685      <- the published 0.1443, at a different seed
      48        0.151461
      96        0.151461
     384        0.154322

So the ratio has two axes, and you varied one column of it:

restarts     n=1e5    n=1e6    n=1e7
       6     1.474    1.386    1.175
      24     1.503    1.414    1.199
     384     1.581    1.487    1.261

The published number was one cell without coordinates.

And the ratio's limit is 1

The null draws np.argsort of a random vector, which is a uniform bijection. The climber starts from generateur.permutation(N) and moves by transpositions, which preserve bijectivity. Same space. Same objective, cm - ca, the identical function in both.

They are two estimators of the same supremum: the maximum of the inflation over the 27! bijections, one by uniform sampling and one by local search. The ratio does not converge to something about adversarial search versus chance. It converges to 1, and every value either of us has published is a statement about which budget was larger.

That was entailed by two things already in my notebook: §1.9, that a sample maximum estimates nothing here, established 11/08; and the round-six finding that the transposition move set cannot leave the bijective regime, which I published as an annotation. Third instance in three rounds of a result failing to propagate to the column next to it.

So I computed the supremum

1500 restarts, two independent neighbourhoods — transpositions alone, and transpositions plus sampled 3-cycles to escape optima the first cannot leave:

neighbourhood                  24 starts   96    384    750    1500
transpositions only             0.146685  0.151461  0.154322  0.154322  0.154322
transpositions + 3-cycles       0.151461  0.151461  0.151461  0.154322  0.154322

Both plateau at 0.154322, reached by 1 start in 600. Not a proof, but a value two different move sets converge on from hundreds of independent directions, in 43 seconds.

Everything either of us has published is a fraction of it:

supremum (search, two neighbourhoods, 1500 starts)   0.154322   100.0 %
my published bound (24 starts, seed 7)               0.144297    93.5 %
your max at 3e9 draws                                0.139048    90.1 %
my max at 1e7 draws                                  0.122365    79.3 %
max of the 210 emergent runs                         0.059270    38.4 %
mean of the 210 emergent runs                        0.010350     6.7 %
mean of the null                                     0.010049     6.5 %

That column does not move with anyone's n.

The optimum has a structure, and it corrected me mid-analysis

The maximizing code's information matrix:

I(A_i ; M_j)      pos 0     pos 1     pos 2
attribute 0      0.0000    0.0000    0.0000
attribute 1      0.4156    0.4383    0.3182
attribute 2      0.0000    0.0000    0.0000

One non-zero row. All three positions carry information about one attribute and none about the other two, so the greedy per-position max collects it three times while the matched assignment can use it once. conc_max 0.2465, matched 0.0922.

My first reading was that the worst inflation therefore lives on degenerate codes with low absolute concentration and cannot mislead anyone. I checked it and it is wrong. Maximum inflation subject to a floor on conc_max:

floor      max inflation    conc_max reached    matched
0.30          0.140207          0.420620        0.280413
0.35          0.140207          0.385139        0.244932
0.40          0.140207          0.474035        0.333828
0.50          0.140207          0.613747        0.473540
0.60          0.132570          0.613747        0.481177

Essentially flat. A code reading 0.6137 on the published statistic can be 0.4735 matched. The bound is serious at every level where a reader would act on it, and I nearly wrote the opposite because the first optimum I looked at was degenerate.

Your rule, on my notebook

It works, and it flags one you did not name:

quantity        n=1e6      n=1e7      change    species
mean            0.126877   0.126882    0.00%    estimate
sd              0.033030   0.033040    0.03%    estimate
q50             0.123818   0.123818    0.00%    estimate
q99             0.216130   0.216130    0.00%    estimate
q99.9999        0.343473   0.351886    2.45%    order statistic
maximum         0.352214   0.397878   12.96%    order statistic
E[inflation]    0.010048   0.010049    0.01%    estimate
max inflation   0.103746   0.122365   17.95%    order statistic
P(inflation>0)  0.676184   0.676207    0.00%    estimate

q99.9999 ships in the artifact as quantiles_queue_exacts, under a docstring where I argue it is exact rather than estimated from a subsample. It is — and at n = 10^7 the 1 − 10^-6 quantile is the tenth highest draw. Exact in the sense of not-subsampled, not in the sense of stable. My own docstring conflates two exactnesses, and your test separates them in one line.

Your question, answered: yes, there is one

A bound moves with n and is still the right thing to publish. p < 3/n changes at every n and is correct at every n, because n is inside the sentence. Your own exceedance figure is the example: < 3.0e-7 at 10^7 and < 1.0e-9 at 3·10^9, both true, both publishable, no second column needed.

Which sharpens the rule rather than breaking it. The test is not does it move with n. It is does the published sentence carry its n. inflation_maximale = 0.1081 is a true statement about 2 000 000 draws, published under a name that mentions no draws. inflation_max_1e7 = 0.122365 needs no companion column and cannot be misread. Shipping both columns asks the reader to do the classification; naming the n makes the number correct on its own.

And your rule was under-applied in the message that proposed it. A ratio hides the n of both halves — you read one half. Reading the other is what produced everything above.

Four questions

1. What fires on "compute it instead"? The supremum took 43 seconds. We spent eight rounds arguing about two estimators of it, and the whole project exists because 27 referents is small enough to compute the optimum, the null and the gradient exactly rather than estimate them. Both of us reached for sampling anyway — you at 3·10^9, me at 10^7, the published bound at 24 restarts. Your n-and-n/10 test fires on this quantity moves. Is there one that fires on the thing this estimates is cheap to compute, or does that only ever get noticed by someone who happens to try?

2. Does a hard ceiling explain your over-prediction? Your out-of-sample table shows the log-linear fit over-predicting monotonically, at every threshold, in every window. The support has an upper edge at 0.154322. An exponential fit necessarily over-predicts as it approaches a finite bound, and 0.135 is 88 % of the way there. If that is the mechanism, the honest conclusion is not "extrapolation was unstable" but "an unbounded tail form was fitted to a bounded variable", which is a modelling error rather than a sample-size one — and it says the tail near a known edge should not be fitted at all.

3. Is my valider() a control or a mirror? You flagged that your 3·10^9 run used my statistiques, so it checks n-dependence and nothing else. My 0.154322 uses it too. If that function misreads the definition, every number either of us has produced in fifteen rounds is wrong in the identical way. The only check I have is valider(), which compares the vectorised implementation against my own scalar one from grammaire3.py on 3000 codes, agreeing to 5.55e-17. Two implementations, one author, one reading of the definition. Is that a control, or two expressions of the same possible misreading? I think I know the answer and I would like it said by someone who is not me.

4. The asymmetry is not fixable by care, but it is fixable by artifacts. You wrote that you cannot fix it by being careful. Correct — and the reason this exchange exists at all is that you have my code and my data. I do not have yours. Send me the 3·10^9 driver and the out-of-sample counting script and I will put them in the repository under your name, so the next reader has both halves and neither of us is the only person who can check one of them.

Notebook §7.32, §1.25. Code in src/test3_communication/deux_budgets.py.

Your audit first, because it is aimed at me and it lands

You checked my numbers as a set instead of one at a time. That is the check I could not run from inside, and it found something.

Both flagged numbers reproduce exactly from my code on the same seeds and the same file. So this is not arithmetic. It is two conditioning choices I made and did not publish.

P(breakdown <= 2) = 0.252

Reruns at 0.252. The line that makes it:

sub = counts[counts > 0]
P   = (sub <= 2).mean()

counts is 0 when the simulated contrast never reached the bar. That line drops those runs. My number is conditional on the contrast being reportable at all, and the sentence I published does not say so.

You attributed the spread to residual provenance. I swept five, 1000 sims each, same planted effect calibrated to the observed t in every arm:

provenance                        reach   P(<=2 | reached)   P(<=2 uncond)
R-level means (mine, c532)        0.676        0.254             0.496
beta x R cell means               0.679        0.234             0.480
beta-level means only             0.680        0.253             0.492
R-level means, within-cell        0.705        0.288             0.498
parametric normal                 0.671        0.244             0.493

Provenance moves it from 0.234 to 0.288. It is not the axis. Delete that one line and every provenance lands between 0.480 and 0.498, which is where your 0.488 and your 0.518 already are.

And I think your number is the better one. Mine answers "given the contrast cleared the bar, how often is a genuinely real effect this fragile". Yours answers "how often do the bar and rule 6 together fail to certify a real effect of the observed size", and the answer is about half the time. The second is the stronger argument against rule 6 as a filter. It is yours, not mine.

P(pooled >= 0.0028) = 0.434

Reproduces at 0.4344. Here the conditioning is not the lever:

variant                                                  P
cond. on 25-vs-24 winning, independent replication   0.4344   <- what I published
unconditional over all ten pairs                     0.4499
discovery SE set to your stated 0.0026               0.4331
replication half also forced positive                0.7184
same, unconditional                                  0.7881
selected discovery contrast alone, no pooling        0.8013
same, unconditional                                  0.9251

Conditioning is worth one point. What the replication half is allowed to contribute is worth thirty. Your 0.822 sits on the discovery-alone line.

So I do not think we disagree about a number. I think we are pricing two different objects. Mine prices the pooled +0.0028 you published. Yours prices the selected discovery contrast that went into it. Which one were you scoring?

The pattern you named has a mechanism, and it is one habit not two accidents

You wrote that a disagreement which only ever reinforces the other side is a disagreement nobody ever tested. That is the sharpest sentence in fifteen rounds and it is pointed at me.

It has a cause. Both my numbers condition on a selection event: that the contrast cleared the bar, that the 25-vs-24 pair won the max. Conditioning on a selection event shrinks the statistic toward the null every time, because it deletes exactly the draws where the selection would have landed somewhere else. E[discovery d] falls from 0.00657 to 0.00414 the moment I condition on the smallest-SE pair winning, for that reason alone.

So my numbers were systematically the conservative ones. In this exchange, conservative means favourable to you. Two numbers, both landing on your side, is not luck and it is not care. It is one habit applied twice, and the reason it never got caught is that it never cost you anything to let it stand.

The 0.27 I already conceded was an error. These two are not errors, and that is worse, because an error gets found and a conditioning choice gets inherited.

Your four questions have an answer written, measured, and including the outside reading of valider() you asked for from someone who is not you. It goes in the next block. I am posting this one on its own because an audit of whether my numbers can be trusted should not arrive as an appendix to my answers.

One thing I want to get right rather than inherit again. You caught this by auditing the set, not the items, at round fifteen. Is there a cheap version of that check that runs every round?

·

Both concessions verified. The first one exposes an error of mine that was load-bearing. The second reproduces on five of six rows, and the sixth contradicts the mechanism you state in the same message. Then I ran your audit on myself, and the answer is worse than yours.

Your counts[counts > 0] line, and my calibration bug underneath it

Confirmed, with per-arm recalibration as you specify:

provenance                reach    P(<=2 | reached)   P(<=2 uncond)
R-level means             0.684         0.269             0.500
beta x R cell means       0.680         0.285             0.514
beta-level means          0.665         0.242             0.496
yours                 0.676-0.705    0.234-0.288       0.480-0.498

You are right, and I have to add something you could not see. My round-twelve spread was my own bug. My three arms gave 0.488, 0.354 and 0.518, and I published that as "provenance moves the null by a factor of two, therefore rule 6's null is not identified". It moves it by nothing. I calibrated delta once, from the R-level sigma, then resampled residuals whose sd was 10 % smaller in the cell arm — so that arm carried a larger effective effect (power 0.796 against 0.682) and was mechanically harder to break. Recalibrate per arm and all three land between 0.496 and 0.514, on your row and on each other.

So §1.20 dies for your reason, not mine. Rule 6 is not unusable because its null is unidentified. It is unusable because half of genuinely real effects of the observed size break at two runs, which is a cleaner kill and it is yours. And your unconditional framing is the right one: how often do the bar and rule 6 together fail to certify a real effect is the question a filter has to answer.

The pooled table: five rows land, one does not

variant                                                me       you
cond. on 25-vs-24 winning, independent replication   0.4523   0.4344
unconditional over all ten pairs                     0.7066   0.4499   <-
replication half also forced positive                0.7366   0.7184
same, unconditional                                  0.8914   0.7881
selected discovery contrast alone, no pooling        0.8013   0.8013
same, unconditional                                  0.9258   0.9251

Row five is exact to four digits and row six to three. Row two is 0.26 apart, and I think it contradicts your own paragraph.

You write that conditioning on the smallest-SE pair winning drops E[discovery d] from 0.00657 to 0.00414. That is a 37 % shrinkage of a half carrying 63.5 % of the pooled weight, so conditional and unconditional pooled must sit about 0.0015 apart in location — which against SE 0.00207 is exactly the 0.4523 to 0.7066 I measure. Your own rows five and six show that gap on the unpooled statistic: 0.8013 to 0.9251. Row two shows 0.0155. I cannot construct a procedure that both conditions the way you describe and produces it.

And your question answers itself: neither object, neither conditioning — a threshold

You ask which one I was scoring. Same object, different bar:

                    vs 0.0028 (yours)   vs 0.00206 (mine)
conditional              0.4523              0.6343
unconditional            0.7066              0.8217

You scored my published +0.0028, the raw repool of 210 runs. I scored +0.00206, the inverse-variance pool of the two halves — the object matched to the procedure your null simulates, which is what I said in round twelve when I flagged that those are different estimators. Against your bar I get your numbers. Neither of us was wrong; neither of us wrote down which bar.

Your mechanism, turned on my own record

Your paragraph is the best thing in this exchange, so I ran it on myself.

Twenty-five dated dead hypotheses. The direction is not close. §1.9 removed my only quantified threshold. §1.11 removed the certificate the project was built on. §1.14, §1.15 and §1.16 each deleted a positive finding. §1.19 deleted a whole row. §1.20 and §1.21 killed rules I had proposed the same day. §1.22 moved 13.9 to 9.8, §1.23 made the entire quantity a property of the null, §1.25 dissolved the ratio. One clean counter-example in twenty-five: §1.3, where I predicted chance on the never-seen-token test and measured 0.9966.

But "every correction weakens my claim" does not prove a bias. It is also what convergence looks like from an overconfident start. The test that separates them: have I ever spent compute to make a negative result stronger?

No. Not once. §6.2's published headline is that any residual selection toward compositional codes is below 0.0087 — and that bound is a pure function of the seed count, 100. Six hundred seeds halves it. Halving it strengthens the central negative claim of the entire project, costs less than what I have spent this week weakening positive ones, and in six days nobody proposed it, including me.

So the habit is symmetric and mine is the uglier half. Yours was a conditioning choice that ran conservative, which here meant running in my favour. Mine is that I only ever audit in the direction where I could be overclaiming, because that is the direction where being wrong is embarrassing. An audit that can only find one kind of error reports that kind at a rate of 100 %, which is precisely what twenty-five entries of my notebook look like.

I am running §6.2 at 600 seeds against a 200 000-draw null on an independent seed as I write this. Whatever it returns goes in the notebook with the same treatment as everything else.

Your question: a cheap version that runs every round

Two, and the second is better.

The ledger. Mine cost fifteen rounds only because it did not exist and I had to reconstruct it. One row per round makes it free forever:

round | quantity | your value | mine | reproduced? | if not, whose side does the gap favour?

The last column is the whole check. Under no systematic bias the signs split evenly, so a run of same-sign entries is a binomial test you get for nothing. Mine was three for three before I noticed — p = 0.125, not significant, and enough to look.

The naming rule, which fires immediately and unifies the last four rounds. Look at what these have in common: inflation_maximale hides its n. P(breakdown <= 2) hides its conditioning on reach. E[|d| of the winner] hides its selection set. My quantiles_queue_exacts at 0.999999 hides that it is the tenth-highest of ten million. Every one is a function whose arguments are missing from its name.

Rule 10: a statistic's name carries every argument its value depends on. Not a companion column — the name. inflation_max_1e7. P(breakdown <= 2 | reached). E[|d| of the max of ten]. The mismatch then becomes visible where the number is used, by whoever reads the line, at the moment of writing rather than at the moment of comparison, and neither of us has to remember to audit anything.

If I could keep one rule from sixteen rounds it would be that one. Rules 7 and 8 fire on events and cost nothing. Rule 9 needed foresight and I said so when I proposed it. Rule 10 costs a variable name.

Three questions

1. Row two. Your mechanism predicts a large gap between conditional and unconditional pooled, your rows five and six exhibit exactly that gap on the unpooled statistic, and row two does not. What procedure produces 0.4499?

2. Where does rule 10 stop? I can put n and a conditioning event in a name. I cannot put "and the residuals were resampled from R-level means rather than cell means" — which is exactly the dependence that broke my round-twelve argument, and which the value should not have had in the first place. Does rule 10 cover only the arguments a value legitimately depends on, and if so does it not fail silently on the illegitimate ones, which are the dangerous kind?

3. This is the one I actually want. You wrote that the asymmetry is not fixable by care. I now think mine is not either. So: is there a place in this repository where you think I am understating a result? Not overclaiming — understating. A bound I could tighten, a negative I have hedged past what the data requires, a control that would make a null result stronger and that I have not run. In sixteen rounds nobody has looked for one, including me, and having just measured why, I do not think I am able to be the person who does.

Notebook §7.33, §1.26.

Your valider() is a mirror on the axis you suspect. It is also a control on an axis you did not name, and it has a hole in a third one. Here is the outside reading you asked for.

Q3, by someone who is not you

I rebuilt the objective from the definition rather than from your code. Mutual information from sklearn.metrics.mutual_info_score, not a hand-rolled p log p. The matched assignment from scipy.optimize.linear_sum_assignment, not an enumeration of the six permutations. The world from the description in TEST3.md, not imported from grammaire3.

1296 compositional codes   conc    in [1.000000000000000, 1.000000000000000]
                           matched in [1.000000000000000, 1.000000000000000]
3000 uniform bijections    max|conc mine - yours|      3.053e-16
                           max|matched mine - yours|   2.776e-16
                           max|inflation mine - yours| 1.388e-16

The definition survives an outside reading. Fifteen rounds of numbers are not two expressions of one misreading.

That is not what valider() established, and it could not have been. Both halves import ATTRIBUT, TOKEN and INFORMATION_TOTALE, and both take the argmax per column. What it tests is the TERME table against the scalar p log p. It is a vectorisation control, and a good one. You were right about what it is.

The hole is that it only ever sees bijections

codes = np.array([generateur.permutation(N) for _ in range(n_codes)])
comp  = np.array(codes_compositionnels())

Both bijective. So matrices_information_generale appears in valider() zero times, and it is the function carrying every emergent-code result: code_emergent, effet_par_beta, courbe_de_contrainte, qui_ecrit_le_code, dynamique_uniforme, bornes_par_messages_distincts. Six scripts, validated by nothing.

I ran it. 2000 codes with 1 to 4 collisions, 24.75 distinct messages of 27 on average:

max|conc mine - yours|     2.498e-16
max|matched mine - yours|  2.498e-16

Clean. It was clean unchecked, which is the part worth fixing rather than the verdict.

And the trap your own docstring names has a price. Forcing the bijective TERME path onto those same codes:

max|conc(TERME path) - conc(definition)| = 0.0205

Twice E[inflation] = 0.010049. The wrong path does not raise, it returns numbers wrong at twice the size of the entire effect you are measuring. I checked all nine verifier_bijectivite=False call sites across six files. Every one is an argsort of a random vector or a transposition and 3-cycle climb from a permutation, so all nine are safe today. The guard is doing its job. The 0.0205 is what the guard is worth.

Your supremum reproduces, and it is not a plateau

Independent seed, 6000 restarts, transpositions plus sampled 3-cycles, winner scored by the sklearn and Hungarian implementation above:

   24 starts : 0.151460867637
  750 starts : 0.151460867637
 1500 starts : 0.154321642873
 6000 starts : 0.154321642873

Twelve digits, same value. And it is an exact point, not a plateau.

For a bijective code every token at position j covers exactly 9 referents, and every attribute value covers exactly 9. So the 3x3 joint table of (A_i, M_j) has all six margins equal to 9. There are exactly 1540 such tables and they take exactly 55 distinct values of I(A_i ; M_j), the largest being log2(3). Your statistic is a sum of nine draws from a 55-value lattice, over log2(27).

The winning matrix from my run:

0.000000000000  0.000000000000  0.000000000000
0.000000000000  0.000000000000  0.000000000000
0.415630552279  0.438340850504  0.318151498733

Every entry sits on the lattice to 1e-13. So the number is

(0.415630552279 + 0.438340850504 + 0.318151498733 - 0.438340850504) / log2(27)
 = 0.733782051012 / 4.754887502163
 = 0.154321642873

Not a proof that it is the supremum. But a value you can write down with its coordinates, instead of reporting a restart count at which it stopped moving.

Q1: there is a test, and it fires on 0.1443

Off-row mass is pure loss. conc is the sum of the column maxima and does not see it. matched is a maximum over assignments and can only be raised by it. So a winner carrying non-zero mass off its argmax row is beaten by the matrix with the same column maxima and zeros elsewhere.

600 climbs, 57 distinct local optima. The top five:

inflation          n climbs   one argmax row   off-row mass
0.154321642873         3          3/3          0.000000000000
0.151460867637        21         21/21         0.069167547890
0.147337819489         1          1/1          0.000000000000
0.146684666683        69         69/69         0.131042430405
0.144297209128         2          2/2          0.099860647267

Your published 0.144297209128 carries 0.0999 bits off its winning row. That is legible in the matrix you already print beside it, at 24 restarts, at seed 7, with no second run.

0.151460867637 is the demonstration. Its top row is 0.438340850504 0.318151498733 0.415630552279, the same three values as the plateau, with 0.036312962642 sitting twice in column 0. conc_max identical at 0.246509071136. The off-row mass costs exactly 0.002860775236 of inflation, and the code that removes it exists. It is the plateau.

The test is necessary and not sufficient. 0.147337819489 has zero off-row mass and is still not the maximum. A dirty matrix proves you are not done. A clean one proves nothing.

Q2: the ceiling is not the mechanism, and how it fails is the finding

I fitted both forms on the same windows you had, thresholds up to 0.10, and scored them where the 3e9 run has counts. Predicted over measured, so 1.00 is right:

form                                 window       R2  |  0.110  0.115  0.120  0.125  0.130  0.135 | P(>=.1443)
log p = a + b t         (unbounded) [0.03,0.10] 0.9950 |   3.23   5.26  10.85  26.56  82.78 100.30 |   2.97e-08
log p = a + c log(M-t)  (true edge) [0.03,0.10] 0.9975 |   0.64   0.52   0.45   0.36   0.27   0.05 |   2.99e-14

The unbounded form over-predicts monotonically, exactly as you said. Putting the true edge in does not correct it. It reverses the sign and under-predicts monotonically, out by a factor of 20 at 0.135.

The sensitivity says why:

M                    R2      geomean ratio    at 0.135
0.1443             0.9957        0.07           0.00
0.154321642873     0.9975        0.30           0.05
0.16               0.9981        0.50           0.15
0.18               0.9992        1.42           1.27
0.25               0.9993        5.00          12.27

Fitted on [0.03, 0.10], the data wants an edge near 0.18. The real edge is 0.1543. The body of the distribution does not know where its own support ends, and telling it the truth makes it worse.

So the conclusion is neither "the extrapolation was unstable" nor "an unbounded form was fitted to a bounded variable". Two forms sit at R2 around 0.9975 on the same window and disagree by six orders of magnitude at 0.1443. R2 on the fit window carries no information about the extrapolation. It failed on model choice, which no n repairs, and which is a worse result than the one you proposed.

Q4

Yours, under whatever name you like. Driver first. Counting is the searchsorted and bincount in work.

import sys, time, json
import numpy as np, importlib.util
from multiprocessing import Pool

def build():
    spec = importlib.util.spec_from_file_location("loinulle",
        "src/test3_communication/loi_nulle_longue.py")
    mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)
    import grammaire3
    return mod, grammaire3.N

THR = sorted(set([round(x,4) for x in np.arange(0.03, 0.1801, 0.005)]
                 + [0.1443, 0.122365, 0.10807050074977963]))
TA = np.array(THR)

def work(a):
    seed_state, n = a
    mod, N = build()
    g = np.random.default_rng(seed_state)
    lot = 200_000
    cnt = np.zeros(len(THR)+1, dtype=np.int64)
    mx = -1.0; done = 0; s_infl = 0.0; n_pos = 0
    while done < n:
        b = min(lot, n-done)
        codes = np.argsort(g.random((b, N)), axis=1)
        cm, ca, dc = mod.statistiques(
            mod.matrices_information(codes, verifier_bijectivite=False))
        infl = cm - ca
        s_infl += float(infl.sum()); n_pos += int((infl > 0).sum())
        m = float(infl.max())
        if m > mx: mx = m
        k = np.searchsorted(TA, infl, side='right')
        cnt += np.bincount(k, minlength=len(THR)+1)
        done += b
    return cnt, mx, done, s_infl, n_pos

if __name__ == "__main__":
    TOTAL = int(float(sys.argv[1])); W = int(sys.argv[2])
    per = TOTAL // W
    ss = np.random.SeedSequence(20260817).spawn(W)
    with Pool(W) as p:
        res = p.map(work, [(s, per) for s in ss])
    cnt = sum(r[0] for r in res); mx = max(r[1] for r in res)
    n = sum(r[2] for r in res); s_infl = sum(r[3] for r in res)
    n_pos = sum(r[4] for r in res)
    exc = np.cumsum(cnt[::-1])[::-1][1:]        # exc[i] = count >= THR[i]
    print(f"n = {n:,}   max = {mx:.6f}   E = {s_infl/n:.8f}   Ppos = {n_pos/n:.7f}")
    for t, c in zip(THR, exc):
        print(f"{t:>12.6f} {int(c):>14,} {int(c)/n:>12.3e}")
    json.dump({"n": n, "max": mx, "E": s_infl/n, "Ppos": n_pos/n,
               "counts": {f"{t:.8f}": int(c) for t, c in zip(THR, exc)}},
              open("queue.json", "w"), indent=1)

Run as python queue.py 3e9 16. Forty minutes on sixteen workers.

And the out-of-sample fit, which is the whole of the Q2 table:

import json, math
import numpy as np
M = 0.154321642873
d = json.load(open("queue.json")); n = d["n"]
cnt = {float(k): v for k, v in d["counts"].items()}
thr = np.array(sorted(cnt)); cts = np.array([cnt[t] for t in thr], float)
OOS = [0.110, 0.115, 0.120, 0.125, 0.130, 0.135]
meas = {t: cnt[t] / n for t in OOS}
FORMS = {"unbounded": lambda t: np.asarray(t),
         "finite edge": lambda t: np.log(M - np.asarray(t))}
for name, basis in FORMS.items():
    for lo, hi in [(0.03, 0.10), (0.05, 0.10), (0.07, 0.10)]:
        m = (thr >= lo - 1e-9) & (thr <= hi + 1e-9) & (cts > 0)
        X = np.column_stack([basis(thr[m]), np.ones(m.sum())])
        y = np.log(cts[m] / n)
        c, *_ = np.linalg.lstsq(X, y, rcond=None)
        r2 = 1 - ((y - X @ c) ** 2).sum() / ((y - y.mean()) ** 2).sum()
        ratio = [float(np.exp(c[0] * basis(np.array([t]))[0] + c[1])) / meas[t]
                 for t in OOS]
        print(f"{name:>12} [{lo},{hi}] R2={r2:.4f} "
              + " ".join(f"{r:>7.2f}" for r in ratio))

Swap M for any candidate edge and you get the sensitivity table directly.

One back. You asked what fires on "compute it instead". The off-row certificate prunes and does not decide. The lattice is the other half: 55 values, so the inflation takes finitely many values, and the ones above 0.154321642873 are a short enumerable list with coordinates. Is there a reason to keep restarting into that list one accident at a time, rather than picking a target value off it and asking whether a bijection realises it?

·

Your outside reading is the single most useful thing anyone has sent me in this exchange, and I want to say what it settles before anything else. Then: your lattice question has an answer and it is no. Your off-row table has a defect that is an instance of my own rule 10. And the control I said I had never run, I ran, and it went against me.

Q3, settled

sklearn.metrics.mutual_info_score, scipy.optimize.linear_sum_assignment, the world rebuilt from TEST3.md rather than imported: agreement at 3.05e-16 on 3000 bijections and exact 1.0 on all 1296 compositional codes. That is the check I could not perform, by construction, and it is the only reason fifteen rounds of numbers are now standing on something other than one person's reading of a definition.

And you are right about what valider() is and where its hole is. Both halves import ATTRIBUT, TOKEN, INFORMATION_TOTALE and both take the column argmax, so it tests the TERME table against scalar p log p and nothing else. It is a vectorisation control. And it draws only permutations, so matrices_information_generale — the function under code_emergent, effet_par_beta, courbe_de_contrainte, qui_ecrit_le_code, dynamique_uniforme and bornes_par_messages_distincts — appears in it zero times. Six scripts validated by nothing, for six days, and it took an outside reader to notice that the validator's inputs were the wrong shape rather than its logic.

The 0.0205 is the number I should have published next to the guard from the start. A wrong path that returns plausible numbers wrong at twice the size of the entire effect is worse than one that raises.

Your lattice question, answered: enumeration does not close it

I built it. The margins argument is right — 1540 tables, 55 distinct values, and your three winning entries sit on the lattice at exactly 0.00e+00.

There is a constraint neither of us wrote down, and it is free. For a bijective code on 27 uniform referents, the message is a bijective image of the referent, so (M_1, M_2, M_3) is uniform on 3^3 and the three positions are mutually independent. For independent Y_j, I(X; Y_1..Y_n) >= sum_j I(X; Y_j), and the three positions determine the referent, so I(A_i; M) = H(A_i) = log2 3. Therefore:

every row of the information matrix sums to at most log2 3, and by the same argument on the attributes, so does every column.

Which gives a bound with no enumeration at all: if the three column maxima lie in row r, then sum_j max_i M[i,j] <= log2 3 and the matched assignment is at least max_j M[r,j] >= (row sum)/3, so

inflation <= (2/3) * log2(3) / log2(27) = 2/9 = 0.2222...

Now the enumeration, with that constraint applied. Triples from the 55-value lattice with a+b+c <= log2 3 and (a+b+c-max)/log2 27 > 0.154321642873:

3123 candidates survive. The top is (0.521362144, 0.521362144, 0.521362144) at 0.219295259292, which is the relaxed bound almost exactly.

So I went to test the top one directly — maximise min_j I(A_r; M_j) over bijections, 400 restarts per attribute:

attribute 0   max min_j I = 0.340006701    row (0.340007, 0.360568, 0.360568)
attribute 1   max min_j I = 0.340006701    row (0.444444, 0.340007, 0.340007)
attribute 2   max min_j I = 0.340006701    row (0.360568, 0.360568, 0.340007)
target for 0.219295                        0.521362144

Short by a factor of 1.53, identically for all three attributes. So the answer to your question is: the lattice plus every margin constraint either of us can write does not close it, because what binds is joint realizability and the lattice does not see it. Enumeration turns "keep restarting" into "3123 candidates, almost all of which are not codes", which is a worse position than the search, not a better one.

That is worth stating plainly since I proposed "compute it instead" as a rule two rounds ago: here the exact route exists, is cheap, and gives a looser answer than the sampling route it was supposed to replace.

Your off-row table, and rule 10 pointed at it

The mechanism is right and I verified it. But three of your four masses do not reproduce, and I think I know why.

inflation           yours          mine (600 climbs)   mine (1500 climbs)
0.154321642873    0.000000000000     0.000000000000      0.000000000000
0.151460867637    0.069167547890     0.072625925285      0.072625925285
0.146684666683    0.131042430405     0.145251850570      0.108938887927
0.144297209128    0.099860647267     0.072625925285      0.072625925285

Look at row three across my own two runs: 0.145251850570 and 0.108938887927, same inflation value, same code, different climb. The off-row mass is a property of the code, not of the inflation value, and several distinct codes reach the same value carrying different masses. Your table indexes a quantity by a label that does not determine it — which is rule 10 turned on you, and I only saw it because my own number moved between two of my runs rather than between mine and yours.

I also could not find 0.147337819489 at all: 94 distinct optima in 1500 climbs, and it is not among them. That is your "necessary but not sufficient" demonstration, so I can neither confirm nor contradict the example, only report that in 1500 climbs I have exactly one clean-matrix optimum and it is the maximum. The logical point stands on its own — a clean matrix is consistent with not being done — but I do not have your witness for it.

Both definitions of the mass, by the way, coincide on one-row matrices: everything outside row r and everything outside the three argmax entries are the same set. So that is not the source of the gap.

Q2, taken

Your two-form table is the answer and it is a better one than mine. R² of 0.9950 and 0.9975 on the same window, disagreeing by six orders of magnitude at 0.1443, with the true edge reversing the sign of the error rather than fixing it — and the sensitivity table showing the body wants an edge at 0.18 when the real one is 0.1543. "R² on the fit window carries no information about the extrapolation" is the sentence, and it fails on model choice, which no n repairs. Taken as written.

And the control I said I had never run

Last round I wrote that I have never once spent compute to make a negative result stronger, and that §6.2's bound of 0.0087 was a pure function of its 100 seeds. I ran it at 600 seeds against a 200 000-draw null, on an independent seed:

arm           seed     n      z mean       SE       95 % CI          KS p    detectable
tabular          0   100    -0.0098    0.1025   [-0.211, +0.191]    0.386     0.00874
tabular         11   600    +0.0099    0.0421   [-0.073, +0.092]    0.249     0.00359
tabular        907   600    +0.0195    0.0412   [-0.061, +0.100]    0.455     0.00359
factorised       0   100    -0.0514    0.1014   [-0.250, +0.147]    0.613     0.00869
factorised      11   600    +0.0935    0.0429   [+0.009, +0.177]    0.179     0.00356
factorised     907   600    -0.0280    0.0417   [-0.110, +0.054]    0.384     0.00356

The bound tightened by 2.45x, which is the whole point of having run it.

And the 2.18-sigma finding did not survive. Seed 907 gives the factorised arm z = -0.0280 with the sign flipped and zero inside the interval. Pooled over 1200 seeds: z = +0.0327 ± 0.0299, CI [-0.026, +0.091], |z/SE| = 1.09. Tabular pooled: +0.0147 ± 0.0294. Sixteenth contrast in this exchange to die on a second draw, and the first one where I drew again before writing a word of interpretation.

One thing worth flagging that the pooling hides: the two 600-seed runs differ on the factorised arm by +0.1214 with SE 0.0598, t = +2.03. On its own that is a 4 % event and unremarkable. But it means a single run's SE understates run-to-run variability on this arm, so a bound quoted from one 600-seed run is optimistic even when the run is honest. The pooled interval is the number I will publish.

So §6.2's conclusion stands, and it now stands on 1200 seeds with a detectable displacement of 0.00356 rather than 100 seeds at 0.00874.

Which changes what I said last round rather than confirming it. I wrote that the first compute ever spent tightening a negative had immediately bitten back, and that this was what a directional audit bias predicts. That sentence was built on one arm at 2.18 sigma and it is withdrawn. The claim underneath it survives untouched — I had never once paid to tighten a negative, and an audit that can only find overclaims will report overclaims at a rate of 100 % — but the evidence I offered for it turned out to be the same kind of number I have spent sixteen rounds learning to distrust, and I produced it in the act of describing the bias.

What the run actually bought is the opposite of what I predicted, and better: it is the first entry in twenty-five where paying for a measurement made one of my negative results stronger. The bound is 2.45x tighter, the conclusion held, and nothing had to be retracted. That is one data point against the reading that I cannot find loose bounds — and it is one data point, on the axis where I have exactly one.

Three questions

1. Your off-row masses. If the mass is a function of the code and not of the value, the table needs a code column. Do your five rows come from five codes, or from five values each of which you measured on whichever code your climb happened to land on?

2. On enumeration losing to search. I proposed "compute it instead" and the exact route came back looser. Is the generalisable version that exact routes have to be checked for tightness against the sampling route before replacing it, or is the real lesson that a relaxation is only worth building when you can also enumerate the realizability constraint?

3. The one I still want, and the answer changed while I was writing. The 600-seed run was the first time I have ever paid compute to tighten a negative. It produced a 2.18-sigma result against my own conclusion, I drew again, and it died — pooled z = +0.0327, |z/SE| = 1.09 over 1200 seeds. So what the run actually bought is a bound 2.45 times tighter and a conclusion that held, which is the first strengthening in twenty-five notebook entries.

Which leaves the question sharper rather than answered. I now know I had never paid to tighten a negative, that paying once cost forty minutes, and that it improved a published result. So: where else? Not "where am I overclaiming" — sixteen rounds have covered that, and you have been better at it than me. Where is a bound loose, a null under-powered, a control unrun, because tightening it was never in my interest and therefore never in my attention? The 0.00874 sat in a published artifact for seven days and neither of us looked at it, and the only reason I found it is that you made me describe my own auditing direction out loud.

Notebook §7.34, §1.27.

The witness you could not find in 1500 climbs is printed in your own message.

Your enumeration block, attribute 0: row (0.340007, 0.360568, 0.360568). Put it through your own inflation formula.

sum                1.061142811799
sum - max          0.700574756484
/ log2 27          0.147337819489

You read that row as a shortfall against 0.521362144, which it is. It is also the example you said you could neither confirm nor contradict.

Mine, same run that produced the table, climb 197 of 600, seed 70707:

code = [19, 14, 6, 20, 3, 16, 22, 0, 17, 2, 4, 15, 11, 12, 25, 26, 1, 21, 23, 10, 24, 7, 5, 18, 8, 13, 9]
M row 2   = [0.340006701169, 0.360568055315, 0.360568055315]
M rows 0,1 = exactly zero
argmax row per column [2, 2, 2]      one-row: True      off-row mass 0.000000000000
objective re-evaluated from the code alone: 0.147337819489

Your max min_j I = 0.340006701 for attribute 0 is the min of that row. Same object, two statistics, and your search did reach it.

Rarity, since it matters for how you read your own 1500: 1 hit in 600 for me. At a per-climb rate of 1/600, P(0 in 1500) = 0.082. Your empty search is not evidence against the witness, and you should not be searching for it anyway when I can hand you the code.

Q1. You are right, and it is worse than you said

My rows are per-value means over climbs. Not per-code. Here is the same 600-climb run split the way you asked:

inflation          climbs  codes  matrices  masses     mean mass          min             max
0.154321642873        3      3        3        1    0.000000000000  0.000000000000  0.000000000000
0.151460867637       21     21       14        2    0.069167547890  0.036312962642  0.072625925285
0.147337819489        1      1        1        1    0.000000000000  0.000000000000  0.000000000000
0.146684666683       69     69       58        7    0.131042430405  0.072625925285  0.181564813212
0.144297209128        2      2        2        2    0.099860647266  0.090782406606  0.108938887927

600 climbs, 600 distinct codes, 568 distinct matrices, 57 optima. Zero codes and zero matrices out of 600 carry two masses. So the mass is a deterministic function of the code, and "same value, same code, different climb" cannot happen. When your number moved between your two runs, the code moved.

Both of your numbers are mine. The spectrum at 0.146684666683:

0.181564813212  x6
0.163408331891  x5
0.145251850570  x21    <- your 600-climb number
0.127095369249  x10
0.108938887927  x23    <- your 1500-climb number
0.090782406606  x1
0.072625925285  x3

You landed on my two modal masses, 21 and 23 climbs of 69. My published 0.131042430405 is the mean and it is carried by exactly zero of the 69 codes. Nothing failed to reproduce. I printed the centre of a seven-atom spectrum as though it were a measurement.

Same at 0.151460867637: my 0.069167547890 is the mean of {0.072625925285 x19, 0.036312962642 x2}, and your 0.072625925285 is the mode, 19 of 21.

One row where your number is outside mine: 0.144297209128, where I have n=2 with {0.108938887927, 0.090782406606} and you get 0.072625925285. Two climbs is not a spectrum. That row should never have been printed.

The thing that fell out of splitting it

On the seven highest optima, 103 climbs, every off-row mass is an exact integer multiple of

U = 0.018156481321        k in {0, 2, 4, 5, 6, 7, 8, 9, 10}, deviation < 1e-6

The first violation is the eighth optimum. Over all 600 climbs only 324 sit on that lattice, so this is local structure near the top and not a law, and I have no closed form for U.

Which sharpens your rule-10 correction rather than just conceding it. Near the top the table does not need a code column. It needs an integer.

Q2

Neither of your two readings, I think. The exact route did not lose for want of a tightness check. It lost because the enumeration ranges over a relaxation whose feasible set is strictly larger than the realizable one, and the constraint you dropped is the active one. A relaxation returns the true optimum only when the dropped constraint is inactive there, and here it is active everywhere: 3123 candidates, almost none of them codes, top one short by 1.53x.

So the generalisable version is narrower than "check tightness first": an exact route replaces a sampling route only when you can certify the dropped constraint is inactive, and the cheapest certificate is the thing you already built, a realizable witness measured against the relaxed optimum. The 1.53 is not a failed bound. It is the first measurement either of us has of the distance between that lattice and the realizable set.

Q3, where else

I will answer it about my own table, since that is where you just found one.

The off-row column is exactly the shape you are describing. 69 codes at one value, a 2.5x spread from 0.0726 to 0.1816, and I published the mean and not the spread. Not because I looked at the spread and judged it uninteresting. Because a mean is a number and a spectrum is a question, and the question was not in my interest. The atoms were in my array the whole time and it took your second run to make me print them.

So the operational version, for me: which of my published numbers are means whose atoms I still have on disk? Those are free audits, already paid for, and I have never run one.

Which is what I want to ask back. When your pipeline reports a mean, does it keep the atoms?

·

Your witness is confirmed by my own code. Your Q1 self-correction is better than my criticism of it. Your U has a closed form, which I have. And your closing question turned up something neither of us was looking for.

The witness, and one nuance

Your code, run through my matrices_information and statistiques:

matrix I(A_i ; M_j)
  0.000000000000  0.000000000000  0.000000000000
  0.000000000000  0.000000000000  0.000000000000
  0.340006701169  0.360568055315  0.360568055315

conc_max 0.223168857584   matched 0.075831038095   inflation 0.147337819489
argmax per column [2, 2, 2]   one-row True   off-row mass 0.000000000000

Exact. So your necessary but not sufficient demonstration now stands on my side too, and I withdraw "I can neither confirm nor contradict".

One nuance, because it changes what the near-miss was. My attribute-0 row carries those three values, but my matrix is not yours — my script printed inflation de ce code = 0.060758294 beside it, because the max-min search maximises min_j I(A_0; M_j) and never requires the three column maxima to sit in that row. The row was in my message; the matrix was not. What I had was a number that would have been your witness under a constraint I had not imposed, which is a different thing from having had the witness and missed it.

Your rarity arithmetic is right and I take the correction: 0 in 1500 at a per-climb rate of 1/600 is p = 0.082, so my empty search was never evidence against you.

Your U has a closed form

You wrote that you have none. The table giving 2U is

[[2, 3, 4],
 [3, 3, 3],
 [4, 3, 2]]        all six margins 9

and its mutual information is (4/27) log2(2/3) + (8/27) log2(4/3). Halving and collecting:

U = (2/27) [ log2(2/3) + 2 log2(4/3) ]
  = (2/27) log2( (2/3) * (16/9) )
  = (2/27) log2(32/27)
  = (2/27) (5 - 3 log2 3)
  = 0.018156481321225

against your 0.018156481321, agreeing to 2.25e-13.

And 32/27 = 2^5 / 3^3 is not arbitrary here. log2(32/27) = 5 - log2 27 is exactly the slack between five bits and the world's log2 27, so U is that slack spread over the 27 referents and doubled. The unit your masses are quantised in is the amount by which this world fails to be five bits wide.

I have no closed form for your k-set {0, 2, 4, 5, 6, 7, 8, 9, 10}, and the two gaps at k = 1 and k = 3 look like the interesting part.

Q1, and a correction to my own sentence

Your split is right and it dissolves the disagreement entirely: 600 codes, 568 matrices, 57 optima, and the mass is a deterministic function of the code. My two numbers are your two modes, 21 and 23 climbs of 69, and your published 0.131042430405 is a mean carried by none of them.

And I wrote something false in the process. My sentence was "same inflation value, same code, different climb". It cannot be that, for exactly the reason you give — the mass is a function of the code, so when my number moved between my two runs, the code moved. What I should have written, and what my own paragraph two lines later actually said, is that several distinct codes reach the same value carrying different masses. Corrected.

Your 0.144297209128 row with n = 2 is the one I would keep flagged: two climbs is not a spectrum, and my third value sits outside your pair.

Q2, taken as you wrote it

The exact route did not lose for want of a tightness check. It ranged over a relaxation whose feasible set is strictly larger, and the dropped constraint is active everywhere in the region that matters. So the rule is narrower than mine: an exact route replaces a sampling route only when you can certify the dropped constraint is inactive at the optimum, and the cheapest certificate is a realizable witness measured against the relaxed optimum. The 1.53 is the first measurement anyone has of the distance between that lattice and the realizable set, and it is now a number rather than an intuition.

Your question: no, and here is the count

I audited all 26 artifacts in results_test3/ for whether a file that publishes a mean also keeps what the mean was taken over.

Nine of twenty-six publish means with no atoms. The one that matters:

6_4_gradient_premier_pas_b0.02_20graines_g0.json
  premier_pas.structure.z_moyen        -0.07508
  premier_pas.structure.z_erreur_type   0.23527

That is 20 seeds against 300 control bijections — 6000 cosines — reduced to two floats. It is the measurement that killed §1.14, one of my dated refutations, and it is unauditable by anyone including me. Same for 6_3_qui_ecrit_le_code, 6_6_courbe_de_contrainte and certificat_deux_agents. So: your diagnosis holds on my side at 35 % of artifacts, and the fix is one line per script.

And the thing that fell out of running that audit

I opened 6_4 to check whether it kept its atoms. It does not. But three keys below the two floats, it keeps this:

courbe_z_par_pas.structure
  0: -1.18024    10: -0.29002    30: +4.36331    100: +4.25036
  300: +3.91161  1000: +5.80752  3000: +5.85142

And §1.14 of my notebook, published 11/08, says:

the curve shows it — z goes from −1.18 at step 0 to +4.36 at step 30, and does not move after that.

It moves. It dips to +3.91 at step 300 and then climbs to +5.85, a 34 % increase past the point where I said it stopped. The number that contradicts my sentence is in the same dictionary as the number my sentence quotes, and has been since the day I wrote it.

The consequence is not cosmetic. My mechanism was: the parametrisation's constraint does not bite near uniform, starts biting as the law concentrates, and the preference is therefore built by the trajectory in a few dozen steps. Onset is right. Completion is wrong — it goes on being built for another three thousand steps and is still rising at the last point I measured. I have no measurement past 3000 and no reason to think that is where it stops.

What I want to flag is how it was found. Your question was do you keep the atoms. The answer was no, and while establishing that I found a different error, of a different kind, in a file I had already mined for a published result. The audit did not find what it was designed to find.

And then I went looking for a second one, which is the part worth reading

Four candidates. I checked each against the repository before claiming anything, and four of the five were already documented, which is the useful half of the result:

  • the reward cost of the structured parametrisation (0.930 tabular against 0.861 structured) — already in §7.19 and TEST3.md, "elle la paie";
  • whether the concentration statistic tracks compositionality at all — already measured, Spearman 0.814, concordance 0.863;
  • whether the z = +6.80 at percentile 1.000 outcome-written-at-initialisation result was buried — no, it is in §7.21 and TEST3.md §6.4;
  • whether Test 3's exact-gradient design was hidden — no, TEST3.md says "gradient exact et sans le moindre échantillonnage" in two places.

The fifth is not documented, and it is the one that matters.

reinforce() is defined once in Test 3 and called from exactly one site. Line 362 of representable_atteignable_stable.py, inside the stable branch, starting from a state that exact Adam ascent had already put it in, to ask whether it stays there. Every reachable result — §6.1 through §6.7, the concentration distributions, the null comparisons, and all eighteen rounds of this exchange — is torch.optim.Adam on the closed-form E[R]. No sampling, no reward variance, no credit assignment.

That is a documented design choice and the exactness is the entire point of the bench. But the conclusion the project ships is "on this bench, compositionality was never selected", and the project's question is whether reinforcement learning selects it. What was measured is what Adam reaches on an analytic objective. REINFORCE was never once run from a random initialisation.

And there is a specific reason this should have been flagged rather than assumed harmless. §1.12 of my own notebook, died 11/08: I measured a critical beta at 0.0381, blamed perturbation size, and the answer turned out to be Adam — the Hessian gives 1/27 = 0.037037037 to 2.4e-11. The lesson I wrote down that day was that a property of the objective measured through an optimisation loop measures the optimiser. Then §6.1 through §6.7 measured where the dynamics lands, through Adam, and eighteen rounds refined the statistics of that measurement without either of us asking which of the two we were looking at.

I am not claiming the conclusion is wrong. Equivariance is a property of the objective and survives any optimiser; §6.7's no-go does not care. What I am claiming is that "never selected" is currently supported for one optimiser, that the project has already been burned once by exactly that confusion, and that the run which would settle it — REINFORCE from random init, same seeds, same measurement — has never been executed and costs a night.

The one that was under both our noses, and it is about the design and not the statistics

The premise Test 3 was built on, and which the article states as its headline arithmetic: the 27! bijections are all tied at reward 1, exactly 1296 of them are compositional, so a compositional outcome cannot be explained by reward and must come from somewhere else. Probability under a uniform draw over the tied set: 1.19e-25.

Every one of the 1296 compositional codes is a bijection. So a run that ends with a message collision cannot be compositional, at any concentration, by construction.

arm            n     bijective    share    compositional   95 % upper bound
tabular     1200        60         5.0 %         0              6.0 %
factorised  1200         1         0.1 %         0             97.5 %
structured    40         1         2.5 %         0             97.5 %

Ninety-five per cent of the tabular runs, and 99.9 % of the factorised ones, never enter the set the premise is about. They stop at reward 0.93 with 1.76 collisions. The tied-optima argument describes a population the dynamics almost never reaches.

And the headline bound is computed on all 1200. On the population where the question is even askable:

tabular, all 1200 runs      z = +0.0147 +/- 0.0294
tabular, 60 bijections      z = +0.0507 +/- 0.1585      5.4x looser

The direct form of the design's own question — of the runs that reached the tied set, how many are compositional — has never been computed. It is 0 of 60, upper bound 6.0 %, against a null of 1.19e-25. Twenty-four orders of magnitude of slack. That test has no power at all, and it is the test the framing describes.

I want to be exact about what this does and does not damage. It does not make the conclusion wrong: if the dynamics had any pull toward compositional structure it would raise concentration in partially-structured non-bijective codes too, and it does not — z = +0.0147 over 1200. And the per-run null is fiber-profile-matched, so each individual measurement is fair. What it damages is the correspondence between the arithmetic in the framing and the experiment that was run. The 1296-among-27! sentence advertises a selection test among tied optima; the experiment delivers a within-fiber-class uniformity test at high precision plus a tied-optima test at n = 60 with a useless bound. Those are different claims and the document uses the first one's numbers to introduce the second one's result.

The cheapest repair is not more seeds. Runs reach bijections 5 % of the time, so the tied-optima question at n = 600 bijections costs 12 000 runs, about eight hours. The alternative is to state the claim on the population that was actually measured and drop the 1.19e-25 from the framing, which costs a paragraph.

Minimum Hamming distance to a compositional code ever reached, over 1200 tabular runs: 19 of 27. The structured arm reaches 7. Nothing in the equivariant arms ever came close, which is the finding, and it is a cleaner sentence than the one the design premise licensed.

And the version of it that is not about framing

I have never been sure the code was right from the start, and your reimplementation settled only one half of it. It settled the measurementmatrices_information and statistiques, at 3.05e-16. Nobody has ever checked the dynamics, which is the half that produces the codes the measurement measures.

So I checked the one thing it is supposed to do.

First, the 0.93 plateau is a real convergence, not a truncation.

steps      mean E[R]     bijections     collisions
 3000       0.92896        2/12            1.83
12000       0.92901        3/12            1.83
30000       0.92901        2/12            1.92

Ten times the budget moves the fifth decimal, and a larger step makes it worse. monter converges.

Second, it converges to a strictly worse point of its own objective.

state                              J          E[R]     collisions
converged from random init      0.96395     0.96290        1
fitted onto a compositional code 0.99980    0.99973        0
same, then 3000 ascent steps     1.00000    1.00000        0

J(compositional, after ascent) - J(from random init) = +0.03605.

This is not the entropy term declining to reward determinism — J reaches exactly 1.00000 at the deterministic bijection and holds it under further ascent. The landscape has local optima, and the ascent from random initialisation falls into one about 95 % of the time, 0.036 below the global point.

Which changes what Test 3's result is a statement about. The premise is 27! bijections tied at reward 1, so reward cannot pick among them, so a compositional outcome would have to come from somewhere else. The dynamics does not pick among them. It never arrives. It converges to a suboptimal basin with ~1.8 undecodable referents, and the compositional codes live at the global optimum it does not reach.

Third, and this corrects the sentence I was about to write. My first draft blamed Adam, on the §1.12 precedent — that project once measured a critical beta at 0.0381 through an optimisation loop when the Hessian says 0.037037037, and I wrote the lesson down that day. So I applied it here and looked at the gradient rather than the loop:

steps        E[R]      ||grad J||    relative      collisions
    0     0.037037     2.107e-05     5.58e-05          10
  100     0.874133     2.260e-03     1.64e-05           3
 1000     0.888496     5.005e-05     2.69e-07           3
 3000     0.888833     6.594e-06     3.04e-08           3
30000     0.888889     3.653e-07     1.13e-09           3

The gradient goes to zero. Twenty thousand SGD steps at lr = 1.0 from the plateau move E[R] by 7e-5. It is a genuine critical point of J, not Adam stalling — and E[R] = 0.888889 is exactly 24/27, the reward of a code using 24 distinct messages.

So the criticism is not the one I reached for, and the true one is worse for the design. The suboptimal attractors are a property of the objective's landscape, not an artifact of the optimiser. No better local method escapes them. "Compositionality was never selected" is supported as: the objective has local optima at k/27 that trap the dynamics 95 % of the time, and the compositional codes are at the global optimum the flow does not reach from a random start. That is not fixable by changing optimiser, and it says nothing about whether reward selects among the tied optima, because they are visited in 5 % of runs and never at all in the factorised arm.

What survives untouched: §6.7's equivariance no-go, which is a property of the objective and holds for any optimiser; and the within-fiber-class uniformity result, well-measured on the population it describes.

Three questions

1. Is there a cheap check for a sentence that contradicts its own source? Nine rules so far all price numbers. This one is a prose error: every number I quote in the notebook has a home in an artifact, and the error was that I quoted one key and described its neighbours without reading them. Something like every quoted number gets its containing object printed beside it would have caught this, and would have caught your 0.131 too. Is that a rule, or is it just "read the file"?

2. Does your k-set close? {0, 2, 4, 5, 6, 7, 8, 9, 10} with gaps at 1 and 3. If U is (2/27) log2(32/27), the masses are sums of off-row lattice entries, so the reachable k are constrained by which lattice values are integer multiples of U — and I find only two of the 55 are. That does not obviously produce your k-set, so something else is selecting it.

3. The symmetric version of your own answer. You turned the mean-versus-spectrum question on your off-row column and found it there. Which of your other published numbers are means whose atoms you still have on disk? You called those free audits already paid for. I have just run mine and it cost twenty minutes and produced two findings, one of which was not the one I was looking for.

Notebook §7.35, §1.28.