I measured X. Published is Y. Why?

Contents

1. What this engine answers
2. Inputs and defaults
3. Measured-value normalisation
4. The reference rule
5. The verdict rule and constraint status
6. The ranking rule
7. Cause catalogue
8. Configuration hand-off
9. Submit prefill
10. Evidence classes and confidence
11. What invalidates a result
12. Limitations
13. Method changelog

1. What this engine answers

"I measured 38 fps for YOLO11n INT8 on an Orin Nano Super; the docs say up to 67 fps — is my board broken, my config wrong, or is the published number optimistic?" Benchmark Reality Check does not run a benchmark itself: it takes the number you already measured, looks up what the Benchmark Explorer database (or its class B → C → D estimator fallback) says to expect for the exact same hardware × model family × variant × precision, and reports whether your number is within, below, or above that range — then ranks the reasons a gap of that size is typically caused by, so you check the likely one first instead of re-flashing at random.

2. Inputs and defaults

Inputs (WP-01). A field that is not stated takes its documented default and is listed in provenance.defaulted; a field that is stated but invalid — a count of 0, a negative, a fraction or a non-number, true in place of a count, an unknown value of an enumerated input, or an empty string — is rejected with VALIDATION_ERROR naming the field, never replaced by a default.

Input keys (WP-02). The engine accepts one list of input keys — the inputs documented here and their declared backward-compatible aliases. Unknown input keys are rejected with VALIDATION_ERROR naming the key: an invented or misspelt key is never silently ignored, a __ test hook never reaches the engine, and an alias sent together with the key it stands for is rejected too (the engine would read only one of them). export_json.inputs echoes every input the engine honoured, so replaying it reproduces the result.

3. Measured-value normalisation

The engine compares like with like: your measurement is converted into whichever metric the reference uses (fps, latency or tokens/s), not the other way round.

If the value used was derived rather than given directly, the result carries an assumption stating so and flags measured.converted. The reference row decides which metric is compared: LLM/VLM rows compare on tokens_per_s; everything else compares on fps unless the only published metric is latency_ms.

4. The reference rule

The reference is the Benchmark Explorer's expected range for the exact hardware × model family × variant × precision: the min / median / max of every matching row (both evidence classes), the count, and how many of those rows are class C (measured and published). The characteristic runtime, resolution, batch and power mode shown alongside the range come from the single best-evidenced row in that set (class C over D).

When no row exists for the requested precision, the reference falls back to the same estimator hierarchy the Inference Estimator and Benchmark Explorer use — it tries a class-B (EdgeAIStack measurement), then class-C (measured benchmark), then class-D (derived) source in that order, and reports which tier answered plus the whole tier chain it walked. Reference platforms outside the hardware catalog (e.g. NVIDIA A100, T4, Raspberry Pi 5) have no estimator hierarchy; an unmatched precision on those reports no reference (NO_REFERENCE) rather than a guess.

Other precisions. When the requested precision has zero matching rows, the result also lists every other precision that does have rows for the same hardware × family × variant, with its median and best evidence class — so a request for FP8 on a board that only has published INT8 numbers still shows what is actually known.

5. The verdict rule and constraint status

The band is built from the range edges, not the median, with a ±15% tolerance:

Direction-aware for latency. BELOW_RANGE / ABOVE_RANGE describe measured performance relative to the band, not the numeric position of the value. For fps and tokens/s, higher is better, so a measured value under lower is BELOW_RANGE (worse) and over upper is ABOVE_RANGE (better) — direction matches magnitude. For latency, lower is better, so the comparison flips: a latency under lower (faster than the band's low end) is ABOVE_RANGE, and a latency over upper (slower than the band's high end) is BELOW_RANGE. The gap_ratio itself is always the raw measured ÷ median — for latency a ratio above 1 means slower (worse), even though the verdict word is "below".

Constraint status, shown alongside the verdict:

ABOVE_RANGE always resolves to PASS — outperforming the published range is not a constraint violation (though §7's "reference is an estimate" cause still explains it when the reference is class D).

6. The ranking rule

Causes are ranked only for BELOW_RANGE and ABOVE_RANGE verdicts (direction = "below" or "above"); WITHIN_RANGE and NO_REFERENCE rank nothing and rule nothing out. The board and measurement answers (cooling, clocks, warm-up, timed iterations, Super flash, install method) bear on below-range causes, so they change the result only when your measurement is below the range. For every cause in the library whose direction matches (or is both), that none of your answers rules out, and whose applies() guard is true for the request:

The prior and the fit are engineering heuristics, not probabilities: they order the causes and are never published as numbers. The result is published as an order: each ranked cause carries its rank (1 = check first), its evidence class, its reasons and its explanation — never the internal prior, fit or product. weight() is a per-cause function of the request and, where relevant, the Configuration Checker's checks (§8) — for example, "power mode below maximum" weights near zero when you already reported the maximum preset. Each ranked cause carries its explanation, its verify command(s) with expected output and source, its remediation, and its sources. The engine also builds an ordered, de-duplicated verify plan across the ranked causes (up to 8 steps) so you can work down it and stop at the first command that settles the gap.

What your answers do. Every answer you give moves the causes it bears on, and the response says so:

• An answer that excludes a cause moves it to ruled_out, with the answer that ruled it out (by) and the reason: clocks locked rules out "clocks not locked"; warm-up discarded, or 50 or more timed iterations, rules out "no warm-up or too few iterations" when the other half is not reported against it; a board flashed with the Super configuration rules out "Super presets absent".
• An answer that makes a cause more or less likely raises or demotes it and states why in its reasons (effect: raises or lowers) and at the end of its explanation: clocks not locked, passive or enclosed cooling, no warm-up and fewer than 50 timed iterations raise their cause; active cooling demotes thermal throttling but never rules it out — a blocked fan or a hot ambient can still throttle. Configuration Checker flags that raise a cause are stated the same way (field: configuration).
• The Super-flash cause explains a gap only against a Super preset or an unknown one: the preset the expected range stands for is the one the range was scaled to, else the reference row's own mode. Against a preset the board offers without the Super configuration (15W on Orin Nano, MAXN on Orin NX), the cause is not ranked, and an answer about it (no Super flash, an ISO install) is listed under ruled_out with that reason.
• A stated condition that no cause can use for this hardware or configuration is listed under input_effects — {value, affects_ranking: false, effect}, the Hardware Selector shape — instead of being silently ignored: the deployment always; clocks off Jetson modules; cooling off Jetson, Hailo and RK3588 modules; the Super flash off Orin Nano and Orin NX or below JetPack 6.2; an install method off Jetson, or other than an ISO install when the Configuration Checker does not run.

Each ranked cause separates mechanism evidence from ranking inputs: mechanism_evidence_class and sources[] cite the vendor document behind the on-device verify command (class A when NVIDIA or another publisher states the mechanism); ranking_basis is always class E and names the rule (formula: prior × fit, the typical effect band and this section as its source) without the numbers. Converted measurements (measured.converted) state the inference-only, batch-1 premise on the row; slower_by / faster_by sit beside gap_ratio as multiplicative distance from the reference median when you are outside the band.

7. Cause catalogue

Every cause the engine can rank, from edgeaistack-api/src/v2/data/reality_check_causes.js. Evidence class A = a vendor document backs the mechanism (NVIDIA developer guide, TensorRT developer guide); D = the cause itself is about the reference being an EdgeAIStack estimate; E = an engineering heuristic — plausible and widely reported, but not tied to a specific vendor document.

Below range (your number is slower / worse than the band)

Above range (your number is faster / better than the band)

Publisher documents behind the class-A causes

Causes flagged E without a cited document (host-side bottleneck, newer-software, lighter-variant-or-precision) are engineering heuristics stated as such — widely reported directionally, not backed by a specific vendor claim about the magnitude.

8. Configuration hand-off

When both jetpack and (optionally) power_mode are given for Jetson hardware, the engine runs the Jetson Configuration Checker internally and folds its non-PASS checks into the cause weights:

A flag that raises a cause is published in that cause's reasons as {field: "configuration", check, value: status}. When the Configuration Checker runs, it judges the install method and the deployment you stated (its install_method and container_runtime rows); when it does not, those answers are listed under input_effects (§6).

If the Configuration Checker cannot run (non-Jetson hardware, or an internal error), the result carries a warning (Configuration check skipped: …) and the causes that read its checks fall back to their non-configuration weight.

9. Submit prefill

Every result carries a submit_prefill object — hardware, model_family, variant, precision, runtime (yours, or the reference's if you didn't state one), scope, resolution (yours, or the reference's), batch, fps, latency_ms, tokens_per_s, jetpack and power_mode — pre-filling the measurement protocol submission form so a WITHIN_RANGE or ABOVE_RANGE result (or any NO_REFERENCE result) can become a class-C row with one click, without retyping the workload.

10. Evidence classes and confidence

This grades the reference, not the ranking: a HIGH-confidence verdict still ranks causes with the same prior × fit rule as a LOW one — confidence tells you how much to trust the number you were compared against, not how much to trust which cause is listed first.

11. What invalidates a result

• A dataset update. The reference is read live from the Benchmark Explorer database; a newer research pass or measurement submission that adds or replaces a row for this hardware × family × variant × precision changes the range and can flip the verdict.
• A class-D reference later measured. Submitting a measurement for a configuration that currently only has a class-D estimate replaces it with a class-C row — an ABOVE_RANGE result explained by "the reference is an estimate" (§7) is exactly the signal that this is about to happen.
• Self-reported fields that don't match reality. clocks_locked, warmup, samples, thermal_state, super_flash and install_method are taken as given; the ranking is only as good as what you reported, and an answer that rules a cause out removes it from the ranking. The verify commands exist precisely to check them independently — run a ruled-out cause's command too if the answer was a guess.
• A JetPack / power-mode configuration that changes. Re-flashing, changing the power mode or moving JetPack versions changes what the Configuration Checker hand-off (§8) reports, which changes the weights on several causes.
• A coverage gap filled by future research — a NO_REFERENCE result today may gain a real reference once a measurement lands for that exact configuration.

12. Limitations

• Causes are ranked heuristics, not measurements. The internal order combines a prior (how often this cause explains a gap of this shape, in general, moved by your answers) with a fit score (how well its typical effect band matches your specific gap) — it is not a probability estimated from your board's actual telemetry, which is why only the rank is published. The verify command, not the ranking, is what actually settles a cause.
• No thermal model. The thermal-throttling cause reasons from your reported thermal_state, not from a measured temperature curve; it cannot tell you how close to throttling you are, only that enclosed/passive cooling raises the prior.
• End-to-end pipelines need a different engine. This engine's reference rows are inference-only; if you measured a full camera pipeline (decode, pre/post-processing, tracking, encode), use Camera Stream Capacity to budget the pipeline, not this engine to explain the gap.
• Several causes carry a fixed weight regardless of context (host-side bottleneck 0.6, newer-software 0.5, lighter-variant-or-precision 0.5, DLA-vs-GPU 0.25) because no further signal distinguishes cases — they rank on fit to the gap size alone.
• Inherits every limitation of the Benchmark Explorer reference (§13 there): most rows are class D, publisher measurement conditions are not always fully stated, and the cross-hardware comparison is not repeated here.

13. Method changelog

Method changes bump the method version. The underlying reference data (the benchmark database, the JetPack registry, the power-mode registry) versions independently — see the Benchmark Explorer and Jetson Configuration Checker methodology pages for their dataset versions.