docs/validation.md

Validation checklist

Release checks prevent software, deployment, and provenance mistakes; they do not pretend the analytical profile has measured-hardware fidelity.

Automated software checks

• deterministic constant/Poisson/bursty workload generation
• exact trace replay and trace validation
• prefill latency monotonicity
• quantization footprint ordering
• latency sensitivity-scale behavior
• colocated end-to-end completion
• static vs continuous behavioral difference
• component TTFT/E2E SLO accounting
• robust repetition-aware capacity search
• bottleneck-diagnosis provenance
• prefix hits do not alter the underlying generated request trace
• prefix reuse reduces modeled prefill work
• P/D pipeline completion and transfer telemetry
• configurable P/D worker counts
• four-scenario topology/cache comparison
• bounded design-space sweep and two Pareto objectives
• paired common-seed research study
• bootstrap paired-effect intervals
• analytical-profile perturbation study
• external-measurement import, normalization, and validation report generation
• train-only calibration with held-out evaluation when enough cases exist
• repeated-seed robust policy ranking and bootstrap intervals
• Pareto-stability accounting across matched seeds
• bounded full-trace oracle candidate search and policy-regret accounting
• Markdown research-report generation with interpretation guardrails
• deterministic stateful agent-session trace generation
• zero cross-turn hits under immediate eviction
• KV reuse and reduced recomputation under retention + affinity
• common-trace four-policy agent comparison
• TTL latency/memory Pareto frontier
• ASCII-only public UI labels
• chart export controls present
• explicit planner worst-repetition and target columns
• no product-style footer
• Hugging Face short_description <= 60 characters
• sdk: static metadata
• canonical Python source equals browser mirror
• every browser Python module is included by the worker
• provenance remains analytical-reference
• JavaScript syntax parse
• Python compilation

Empirical validation protocol

A future hardware run should be stored as external validation cases, not copied into analytical profiles without provenance. At minimum each case should record:

• model / revision
• serving topology and scheduler
• accelerator / count
• precision
• workload arrival process or trace ID
• prompt/output distribution or exact trace
• SLO definition
• p95 TTFT / p95 E2E / goodput
• runtime/software versions

For already-normalized cases run:

python scripts/validate_measurements.py measured_cases.json --output validation_report.json

For serving-tool artifacts run:

python scripts/import_measurements.py result.json --source auto --output measurement_cases.json
python scripts/calibrate_profiles.py measurement_cases.json --holdout 0.33 --output calibration_result.json

The calibration script fits only on the training cases and reports baseline/calibrated residuals on held-out cases when at least three cases are available. A smaller case set is explicitly marked as resubstitution rather than held-out validation.

Not claimed

• empirical L4/A10G/A100 latency accuracy
• exact vLLM/SGLang scheduler equivalence
• CUDA-kernel modeling
• real network-protocol fidelity
• radix-tree prefix-cache eviction/scheduling
• speculative decoding
• Attention-FFN disaggregation
• production-fidelity agentic serving or dynamic batching inside Stateful Sessions
• real tool execution or production agent-runtime integration
• claims that the bounded offline oracle is globally optimal
• claims that simulator bootstrap intervals quantify real-hardware error

These are explicit scope boundaries, not hidden assumptions.