kanaria007 PRO

Yes, I think this found a real missing axis, and I updated 274 for it.

I ended up giving it a separate 4.5 rather than stretching 4.4 further, because the issue is broader than baselines:

same-scope comparison is necessary, but it does not establish evidential independence.

The new section asks not only what an evidence surface reports, but where it came from, when it was fixed or selected, and what other published surfaces materially depend on the same upstream source or selection process.

So in your example the relevant questions are now explicit:

where did the split composition come from?

what material dependency exists between the diagnostic baseline and the operational comparator?

I also generalized the failure mode beyond this split. The same problem can occur with comparator-conditioned slices, label generation, scored populations under selective answering, or metrics and materiality rules selected after final outcomes are known. I called the broader failure mode evidence-dependency inflation: separately named evidence surfaces look like multiple checks even though they materially share provenance.

One arithmetic correction, though. 2,714 / 3,432 is about 79.1%, not 81.4%. The 81.4% constant-FAIL baseline comes from the 2,793 / 3,432 true failures. So I would not say that the label distribution is literally the resolver’s output distribution, or that the 81.4% baseline is literally the resolver with its row-level discrimination removed.

But I think your structural point survives that correction.

Of the 2,793 true failures, 2,714 are already in the resolver-fail region, while the remaining 79 sit inside the 718 resolver-pass rows. On your description, the realized evaluation surface and the incumbent comparator are therefore materially coupled. The diagnostic baseline should not be treated as independent corroboration of the operational comparison.

I would also stop slightly short of calling the 81.4% diagnostic worse than uninformative. It still tells us something real: a constant FAIL prediction gets 81.4% on this realized split. What changes is its evidential role. Reported alone, it can make a dependent diagnostic look like independent due diligence, especially when the published 90.0% clears it while still losing to the 97.7% operational comparator.

The update therefore does not prohibit dependent evidence. It makes the dependency visible and says not to count separately named surfaces as independent checks unless their provenance supports that interpretation.

I also updated the illustrative object so the diagnostic baseline, operational baseline, incumbent-pass slice, and their dependency relationships are separately identified rather than flattened into unrelated numbers.

So yes: “where did this split composition come from?” is now explicitly part of the publication discipline. I think that is the right generalization of the failure mode you found.

This is a very useful follow-up. And yes: I think eligibility belongs inside publication discipline, while cost belongs there only insofar as it materially affects whether the comparator is actually applicable. Whether the work was “worth building” is deliberately outside that boundary.

4.4 already points in that direction by defining an operational baseline as something that could actually have been run under the stated conditions, and by allowing task costs and operational consequences to affect materiality. So I do not think this needs a new posture rule.

The publication question is:

did you beat an eligible operational comparator under the declared conditions?

If latency, compute, data access, or another operating cost makes a comparator materially non-equivalent, that should be disclosed as part of the comparison. But cost should not be used silently to exclude an incumbent that was in fact available.

The next question

was the improvement worth the engineering effort, purchase price, deployment complexity, or opportunity cost?

is a separate value and decision surface. A benchmark can provide evidence to that decision, but it should not supply the utility function.

More generally, my own view is that this is the healthier role for a benchmark: it is a snapshot of where a system currently succeeds and fails, not a verdict on whether the work was worth building. Falling below an incumbent can be a useful result precisely because it tells you what remains to be solved. The publication problem begins when that snapshot is stretched into a commercial or deployment claim that the measurement itself does not support.

Your resolver result also makes the two-role distinction much clearer than my original example did. The 81.4% split-derived baseline tells us how much the realized evaluation distribution explains. The 97.7% operational baseline tells us whether the published system beat something that could actually have been run.

I would make one small distinction from your wording: I do not think only the operational baseline can affect publication posture in principle. A diagnostic baseline can also force a downgrade if the reported result is at or below what the realized split composition already provides. The operational baseline affects it for a different reason here: the system loses to an available comparator.

And with the 0.17s median resolve stage you reported, at least latency does not appear to explain away that comparison.

Thanks again for rerunning this against the revised section. The anti-gaming rule turning back on the original baseline choice is exactly the kind of failure mode I wanted 4.4 to expose.

You’re right. 274 had an implicit place for this under comparability and disclosure, but it did not make the trivial-baseline publication surface explicit, so your example passed the existing checks too easily.

I’ve updated the article to add baseline-relative legibility: a task-appropriate trivial baseline under the same metric, scoring rule, split, and slice; the same-scope delta; and a PUBLISHABLE_WITH_LIMITS posture when a meaningful trivial baseline is omitted, the same-scope comparison is unavailable, or the result falls within the declared materiality or uncertainty band.

I also separated two cases that are easy to conflate:

• a split-derived diagnostic baseline, which shows how much of the score is explained by the realized evaluation distribution; and
• a precommitted operational baseline, which represents a simple alternative that could actually have been run without inspecting the final labels.

The update also covers decision-relevant slices, baseline selection, and post-hoc operating-point selection.

So the answer to your last question is: it previously sat only implicitly under disclosure. It now has an explicit slot.

Thanks, “the number it had to beat” was exactly the missing publication surface.

That is the right default question.

My answer is: Chronia is default-deny for strong reliance, not default-deny for observation or first contact.

An untouched surface is not automatically forbidden. It can enter observation, sampling, shadow, dual-run, or sandbox / RML-1 closed-world lanes. Those lanes are exactly how coverage is earned.

But an untouched surface does not inherit reliance from a covered surface. If no mechanism has observed, sampled, shadowed, or probed a region, then the system has no continuity claim for that region. It has only an unknown / unearned-coverage state.

So the default posture is:

• observation lane: allowed;
• sandbox or RML-1 dry-run: allowed;
• shadow / dual-run lane: allowed;
• review-only or hold-commit lane: allowed;
• low-blast-radius exploratory contact: allowed;
• high-reliance or effectful commit: denied or degraded until coverage is earned.

That also answers the recursive detector issue. Every leading signal is itself a scoped detector, yes. That is why no detector gets global authority. A coverage signal can justify only the lane and reliance level it is scoped to. It can move a surface from untouched to sampled, or from sampled to shadowed, or from shadowed to review-only. It does not automatically promote that surface to full production reliance.

Default-allow plus receipts is lagging attribution with better paperwork. Useful after failure, but not sufficient as a reliance discipline.

Default-deny for strong reliance is the safer Chronia posture: uncovered surface is not treated as bad, but it is not yet admissible for strong reliance. It has to earn promotion through observation, sampling, synthetic probes, shadow evidence, dual-run checks, or explicit review.

Compressed:

Chronia allows first contact, but only through low-reliance, low-blast-radius lanes. Untouched surface is not forbidden. It is unearned.