docs/methodology.md

Methodology and limitations

What is simulated

InferScale models request arrival, queueing, admission, prefill, autoregressive decode, dynamic batch membership, KV-cache memory, and request completion.

For P/D disaggregation it additionally models independent prefill/decode worker pools and explicit prompt-KV transfer before decode admission. Metrics are computed from per-request virtual timestamps.

Scheduler semantics

• static_fcfs: admits one colocated batch and drains it before admitting new requests.
• continuous_fcfs: admits FCFS work whenever decode slots become available.
• continuous_sjf: prioritizes shorter estimated jobs at admission.
• continuous_slo: uses least-slack-style ordering from the E2E deadline and an analytical remaining-service estimate.
• chunked_slo: combines least-slack ordering with chunked prompt prefill.

These are transparent approximations, not line-by-line reproductions of vLLM or SGLang. Static batching is excluded from P/D comparisons; a requested static P/D run is converted to continuous semantics with an explicit warning.

Workload semantics

Poisson arrivals use exponentially distributed inter-arrival times. Constant arrivals are evenly spaced. Bursty arrivals alternate lower and higher rate periods. Prompt/output lengths use log-normal distributions parameterized by mean and coefficient of variation.

All generated arrival modes are open-loop: arrival scheduling does not wait for prior responses to finish. This preserves queueing delay under overload instead of allowing client backpressure to hide tail latency.

Exact trace replay accepts rows with:

arrival_time, prompt_tokens, output_tokens

and preserves those values directly. Capacity search is intentionally disabled for exact traces because changing request_rate_rps would not change a fixed arrival sequence.

Shared-prefix model

The simulator models one reusable exact prefix:

1. cache hits use a separate seeded RNG so enabling cache does not alter the arrival/prompt/output trace;
2. a hit reduces prefill work by the reusable-prefix length, bounded by prompt length;
3. shared prefix KV consumes one persistent allocation per serving worker instead of being duplicated per request;
4. decode still uses the full logical context for attention-cost estimation.

This isolates exact prefix reuse without implementing radix-tree lookup, eviction, or cache-aware routing.

P/D disaggregation

P/D uses a global event queue with four main event classes:

arrival
prefill_done
transfer_done
decode_done

Prefill workers batch queued requests independently. Completed prompt state enters a serialized transfer link. Decode workers admit transferred requests only between decode iterations, preserving continuous-batching semantics.

The transfer model is:

(base_latency + bytes / bandwidth) x transfer_scale

where transferred bytes correspond to newly computed prompt KV. With a cache hit, shared-prefix state is assumed resident in both role pools and only the uncached suffix is transferred.

The model does not claim protocol-level fidelity to PCIe, NVLink, RDMA, NIXL, NCCL, or any particular production transport.

Latency model

The default reference model is roofline-inspired:

• dense transformer FLOPs scale with parameter count and processed tokens;
• attention adds context-length-dependent work;
• decode includes weight traffic and context-dependent KV reads;
• time is approximated from compute/memory costs plus a launch/scheduling proxy;
• conservative efficiency factors prevent peak hardware specifications from being treated as achieved throughput.

Prefill and decode expose multiplicative sensitivity scales. These default to 1.0 and are used only by research stress tests unless explicitly supplied.

This creates useful qualitative dynamics but is not empirically calibrated.

KV cache

KV bytes per token are approximated as:

2 x layers x KV heads x head dimension x 2 bytes

for K and V with FP16 KV state. Paged allocation rounds live sequence lengths to kv_block_tokens. Static batching reserves full prompt+requested-output capacity.

Capacity search

Each offered rate is evaluated over deterministic seed offsets and is feasible only when every repetition:

1. reaches the configured SLO-attainment target; and
2. fully drains all generated requests.

The result retains mean, worst, best, and standard-deviation evidence. A bounded binary search estimates the highest feasible rate and a recommended load after user-selected headroom.

Design-space explorer

The browser-safe sweep evaluates:

• three colocated continuous schedulers over batch sizes 8/16/32;
• one cached SLO-aware colocated point at the current batch size;
• optionally P/D worker splits 1P:1D, 1P:2D, and 2P:1D, each with cache off/on.

A candidate is performance-Pareto-optimal when no other candidate has both at least as much goodput and no worse p95 TTFT, with at least one strict improvement. An independent efficiency frontier replaces raw goodput with goodput per accelerator.

This is a bounded interactive design study, not exhaustive global optimization.

Paired A/B studies

A paired study compares one controlled system change repeatedly. Baseline and treatment receive the same seed on every repetition. This is the common-random-numbers variance-reduction idea: both alternatives see the same sampled workload, so the paired delta is less contaminated by workload randomness.

For each metric InferScale reports:

• baseline and treatment means;
• mean and median paired delta;
• treatment win rate;
• mean relative change;
• 95% percentile-bootstrap interval over paired deltas.

The bootstrap interval is an uncertainty summary for the simulated repeated experiment. It is not evidence that the analytical hardware profile itself is empirically correct.

Analytical-model sensitivity

The robustness study draws shared multiplicative prefill/decode/transfer scales inside a user-defined band and applies each draw to both alternatives. It reports how often the treatment wins TTFT/goodput/E2E and how often each alternative satisfies the SLO.

The perturbation distribution is deliberately described as a sensitivity analysis, not a calibrated posterior over real hardware. Its purpose is to identify conclusions that reverse under modest model error.

Stateful agent sessions

Agent mode is a separate program-level discrete-event model. Session arrivals are open-loop. Each session receives a deterministic number of turns, token increments, outputs, and tool gaps from a seeded trace. A later turn cannot become ready until the prior turn completes and its tool gap elapses.

Each simulated replica is a serial analytical service station in this mode. This is deliberate: dynamic batching remains covered by Serving section, while Stateful Sessions isolates four stateful effects:

1. cross-turn reuse: a resident KV entry means the next turn prefills only appended tokens;
2. tool-gap residency: retained KV occupies memory while the program waits outside the model;
3. routing locality: session-affinity can preserve reuse but may trade against load balance;
4. eviction: TTL expiry and LRU-style memory-pressure eviction can force full-history recomputation.

The cache working set is tracked independently on each replica. The simulator integrates occupancy over virtual time and reports HBM GB-seconds, peak KV, mean KV, cache-hit rate, recomputed history tokens, routing-locality rate, and eviction counts.

The TTL sweep replays one identical agent-program trace for every TTL and reports the non-dominated frontier minimizing both p95 turn TTFT and mean KV residency. It is a controlled what-if study, not an optimizer over a measured production system.

Empirical validation and calibration

measurements.py accepts external serving artifacts, normalizes them into explicit validation cases, and then delegates residual computation to validation.py. Supported observations include p95/p99 TTFT, p95/p99 E2E, goodput, request throughput, and SLO attainment.

When at least three cases are available, calibration uses a deterministic train/held-out split. Global prefill and decode timing multipliers are fit only on the training cases using robust median ratios, then baseline and calibrated predictions are evaluated on the held-out cases. With fewer than three cases, InferScale falls back to resubstitution and labels that weaker protocol explicitly.

The fitted scales can correct a global timing bias. They do not establish fidelity for unseen models, devices, schedulers, topologies, or workload regimes. No measured benchmark values are bundled as truth with the project.

Stateful host-tier KV model

Agent-session experiments can move idle cross-turn KV from the modeled HBM tier to a host-memory tier. Transfer time is represented as:

transfer_time = base_latency + KV_size / host_bandwidth

Offload begins when an LLM turn completes and can overlap the simulated tool gap. If the next turn becomes ready before offload completion, the remaining offload time is exposed before restore. Restore is serialized with the turn service model in this reference implementation. Host capacity is finite and uses LRU-style pressure eviction.

This is a deliberately transparent what-if model, not a PCIe/NVLink/NIXL/DMA simulator. Its purpose is to compare three costs under the same program trace:

• recompute history after eviction;
• retain KV in scarce HBM during tool gaps;
• offload KV and pay data movement on reuse.

Bounded-affinity routing

Strict session affinity always routes a turn to the replica that already holds its HBM KV. Least-load routing ignores locality. Bounded affinity interpolates between them:

1. find the least-loaded replica;
2. find the replica holding the session KV, if any;
3. estimate the extra queue/busy-horizon penalty of following locality;
4. keep affinity only if that penalty is at most affinity_slack_ms.

The estimator is intentionally simple and labeled as such. The Affinity Frontier sweeps the slack on one common program trace to show where additional locality stops being worth the load imbalance.

Finite-HBM budget stress

The memory-budget experiment first runs a full-retention reference trace and records the unconstrained peak per-replica KV working set. It then expresses stress budgets as multiples of that trace-specific peak rather than arbitrary fractions of total accelerator VRAM. This makes the experiment meaningful even for small models whose default VRAM headroom would otherwise be far larger than the generated KV working set.

Failed turns and incomplete sessions count against SLO attainment; latency percentiles remain conditional on turns/sessions that actually complete and are accompanied by failure counts in the experiment table.

Adaptive tool-gap prediction

Agent traces attach a tool_kind to each non-final LLM turn. Reference tool families have different duration scales (filesystem, search, database, and remote_api) so a tool-aware predictor has an identifiable structure to learn. Tool identity is synthetic metadata used only by the agent-session simulator.

The adaptive policy uses an online exponentially weighted moving average (EWMA). Two scopes are supported:

• global_ema: one duration estimate shared by all tool families;
• per_tool_ema: one estimate per tool family, with the global estimate used until the per-tool warm-up count is reached.

No future tool duration is visible when the retention decision is made. When an LLM turn completes, the adaptive policy predicts the upcoming gap and chooses HBM retention or host offload using the same threshold as the oracle gap-aware baseline. The actual tool duration is added to predictor history only when the simulated tool call completes through a separate tool_observed event. A predicted HBM-retain decision is still capped by kv_ttl_s; this safety TTL prevents a severe under-prediction from pinning HBM indefinitely.

This sequencing is important: directly updating the predictor at LLM-turn completion would leak the already-sampled future gap into the policy and make the adaptive result clairvoyant.

Non-stationary predictive-tiering study

The Predictive tiering experiment generates one common program trace and replays it across fixed and adaptive policies. At a configured fraction of the arrival horizon, the duration scale of slower external tools (database and remote_api) is multiplied by a user-selected factor. Faster local-style tools remain unchanged.

The study compares:

1. fixed TTL;
2. unconditional host offload;
3. adaptive global EWMA;
4. adaptive per-tool EWMA;
5. oracle gap-aware tiering.

The oracle is not a deployable contender. It reads the realized future tool gap and exists only to expose an upper bound for the threshold-based decision rule.

For adaptive candidates InferScale reports:

• mean absolute tool-gap prediction error;
• pre-shift and post-shift MAE;
• agreement with the oracle retain/offload action;
• serving p95 TTFT and session E2E;
• SLO attainment;
• HBM/host residency and recomputation.

Prediction accuracy is intentionally not treated as the sole objective. Two predictors can have similar MAE while producing different queueing and memory behavior because only errors near the retention threshold change the tiering action.

Adaptation-rate sweep

The EWMA alpha sweep replays the exact same shifted trace for every candidate alpha. Small alpha values smooth noise but react slowly after distribution shift; large values react quickly but can overfit individual long-tailed tool calls. InferScale therefore reports both pre-shift and post-shift error alongside system-level latency and memory metrics rather than presenting one global optimum.

Online execution-transition learning

Execution Model models a different reuse object from Stateful Sessions. Stateful Sessions tracks one user's growing cross-turn KV state. Execution Model tracks reusable static agent prefixes (for example planner/retriever/reasoner system context) that can be shared across workflows.

A synthetic workflow is generated from a first-order role-transition matrix over five agent roles plus an END state. At shift_fraction of the arrival horizon a second transition matrix becomes active. Every candidate in a controlled study replays the exact same workflows.

The learner stores a transition row for each current role. Prediction occurs before the next role is observable, and the model is updated only when that next role becomes ready after the tool gap. transition_decay = 1 is cumulative counting; smaller values exponentially forget older evidence.

Multi-step forecast

For a current role, InferScale rolls the learned transition matrix forward for forecast_horizon steps. Future-step distributions are accumulated with geometric discount forecast_discount. This produces an expected discounted future-visit score for each role. No realized future workflow step is read during online policies.

multistep chooses up to prefetch_top_k roles from that ranking when their normalized score exceeds forecast_min_score. utility uses the same learned forecast but requires positive decision value above utility_threshold_ms.

The reference utility for role r is:

forecast_score(r) * avoided_static_prefix_prefill_ms(r)
- host_transfer_ms(r)
- forecast_weighted_eviction_recompute_ms

The eviction term estimates the opportunity cost of removing currently cached prefixes that the same forecast considers likely to be needed. This is a transparent heuristic objective, not a claim of globally optimal cache control.

Prediction calibration and action quality

InferScale keeps prediction and action metrics separate. The transition model reports:

• top-1 accuracy before/after the shift;
• multiclass Brier score;
• multiclass log loss;
• top-1 expected calibration error (ECE).

The serving policy reports:

• forecast-set recall over the configured horizon;
• immediate next-step prefetch precision;
• eventual prefetch utilization (a prefetched prefix was actually consumed before eviction/end);
• unused prefetch volume;
• prefix hit rate and prefill tokens saved;
• HBM GB-seconds and pressure evictions;
• p95 step TTFT and workflow E2E.

This distinction matters because multi-step prefetch can be useful even when a prefetched role is not the immediate next step, while a highly accurate top-1 predictor can still make poor resource decisions.

Controlled studies

The confidence-threshold sweep varies when one-step learned predictions are acted on. The transition-decay sweep varies transition decay. The prefetch-planning comparison compares top-1, multi-step top-k, utility-aware multi-step, and a clairvoyant realized-future-set information bound. The forecast-horizon sweep varies rollout depth on a common trace. The prefix-cache budget sweep runs top-1, multi-step, and utility-aware planners at multiple shared-prefix working-set budgets.

Oracle next-role and oracle future-set candidates are clairvoyant information bounds, not guaranteed serving-performance upper bounds. Perfect future information may still produce poor global resource behavior when multiple workflows share a small cache and serialized transfer path.

Research consolidation and robust policy ranking

The final research layer deliberately stops adding new serving mechanisms and asks whether policy conclusions survive workload variation. repeated_seed_policy_study generates matched workflow traces across multiple deterministic seeds and replays every deployable Execution Model policy on each seed. It reports:

• median and mean p95 step TTFT;
• a bootstrap interval over seed-level TTFT;
• paired TTFT deltas relative to the top-1-decayed baseline;
• fraction of seeds won on TTFT;
• worst-seed TTFT;
• Pareto stability, the fraction of seeds on which a policy remains non-dominated over p95 TTFT, unused speculative prefetch, and mean prefix-HBM residency.

Bootstrap intervals quantify uncertainty over the sampled simulator workloads. They do not quantify analytical-profile error or real-hardware uncertainty.

Bounded full-trace serving oracle

For each matched seed InferScale also computes a bounded information upper bound. It evaluates the same deployable policies plus clairvoyant future-set plans over forecast horizons H=1..5 and prefetch widths K=1..3, all under the same prefix-cache capacity, transfer bandwidth, workload trace, model, and analytical device profile. The best completed candidate on that full trace defines the per-seed reference.

Policy latency regret is:

policy_regret_ms = policy_p95_TTFT - bounded_oracle_p95_TTFT

This reference is intentionally named bounded, because exhaustive search over the declared candidate family is not a proof of globally optimal action scheduling. The oracle may expose information advantage, but the simulator does not solve arbitrary cache-control sequences.

Research report export

The consolidation result and optional measurement calibration can be rendered into one Markdown report. The report carries the study protocol, robust ranking, bounded-oracle definition, held-out calibration summary when present, and explicit interpretation guardrails. This export is intended to make experiments reviewable outside the web UI rather than turn the UI itself into the only artifact.