How the Model Memory Fit engine decides what fits.

Contents

1. What it computes, and what it does not
2. Inputs and defaults
3. Weights
4. KV cache
5. VLM image tokens and vision tower
6. Runtime overhead
7. OS headroom and reserve
8. Vision-pipeline mode
9. Verdict thresholds and constraint statuses
10. Capacity landmarks and requested position
11. Evidence classes and the confidence rule
12. Model registry
13. Module memory
14. Documents
15. What invalidates a result
16. Limitations and gaps
17. Method changelog

1. What it computes, and what it does not

The engine builds a memory budget for one model on one Jetson module at one quantisation, context length, concurrency and runtime, and compares the total against the module's installed memory. Four modes share the same budget: llm (weights + KV cache + runtime overhead + OS + reserve), vlm (llm plus a vision tower and image tokens added to the context), asr (Whisper-style encoder–decoder, a small decoder KV cache), and vision (detector pipelines, whose model-side parts come from the shared V2 vision-memory module and are rebuilt on this same budget, so this engine and Model Match agree on what a detector weighs as well as on OS headroom and reserve).

It does not measure your actual runtime's memory use, predict tokens/s, model KV-cache eviction or paging strategies, or account for a container runtime's own overhead beyond the stated runtime constant. Where the registry has no data for a field the engine falls back to a stated formula (never a silent guess) and labels the result's evidence class accordingly.

2. Inputs and defaults

Every input has a stated default so a minimal request (platform + model) still returns a result.

Inputs (WP-01). A field that is not stated takes its documented default and is listed in provenance.defaulted; a field that is stated but invalid — a count of 0, a negative, a fraction or a non-number, true in place of a count, an unknown value of an enumerated input, or an empty string — is rejected with VALIDATION_ERROR naming the field, never replaced by a default.

Input keys (WP-02). The engine accepts one list of input keys — the inputs documented here and their declared backward-compatible aliases. Unknown input keys are rejected with VALIDATION_ERROR naming the key: an invented or misspelt key is never silently ignored, a __ test hook never reaches the engine, and an alias sent together with the key it stands for is rejected too (the engine would read only one of them). export_json.inputs echoes every input the engine honoured, so replaying it reproduces the result.

3. Weights

The engine prefers a measured artefact: if the registry has a published quantised file for this model and quant (a GGUF from bartowski, unsloth or ggml-org, or a safetensors shard set), its published size is the weight figure, class A. Otherwise weights are parameters × bytes-per-parameter.

bytes_per_param is a fixed storage width for FP16/BF16 (2 bytes) and FP8 (1 byte) — class A, true by definition. For block quants (INT8/GGUF Q8_0, Q4_K_M, NVFP4) the nominal bit width understates the real size because each block also stores a scale, so the engine uses the median measured artefact-size ÷ parameters ratio across the registry's own rows for that quant — class D, a EdgeAIStack-derived figure, not a vendor one.

Source for the Q8_0 block layout: ggml-org (llama.cpp) · github:ggml-org/llama.cpp/ggml-common.h §block_q8_0 · verified 2026-09-07 — “QK8_0 32 … block_q8_0 { ggml_half d; int8_t qs[QK8_0]; }”.

4. KV cache

The KV cache formula is the standard transformer key-value cache size, applied per sequence and summed across concurrent sequences. ASR mode caps the token count at the decoder's text limit (448 tokens for Whisper), since the encoder side has no growing cache.

Source: NVIDIA · nvidia-blog:mastering-llm-techniques-inference-optimization §Key-value caching · verified 2026-09-07 — “KV cache size per token = 2 × (num_layers) × (num_heads × dim_head) × precision_in_bytes”. Class A: the formula is the architecture-derived cache size; class D overall because layers/kv_heads/head_dim for a given model may themselves be class-A config.json fields or, when config.json omits head_dim, derived as hidden_size ÷ num_attention_heads (flagged per-model as head_dim_derived).

5. VLM image tokens and vision tower

A VLM adds two things beyond the text backbone: image tokens appended to the context (so the KV cache grows), and the vision tower's own weights.

image_tokens_per_image is model-specific and stated on each model's page: for example Qwen2.5-VL uses (H/28)×(W/28) tokens after a 2×2 spatial merge (1280 assumed for a ~1 MP image); Gemma 3 pools its SigLIP patches to a fixed 256 tokens per image. Vision-tower parameters, when not directly stated, are estimated with the standard ViT parameter count 12 × layers × hidden², kept in FP16 regardless of the text quantisation (vision towers are rarely quantised below FP16 in practice).

6. Runtime overhead

A fixed base plus a percentage of weights, per runtime — engineering heuristics (class E), not vendor-published figures. These cover the CUDA context, compute buffers and (for llama.cpp/MLC) the paged KV allocator's bookkeeping beyond the KV bytes already counted in §4.

runtime_mb = base_mb + workspace_pct × (weights_mb + vision_mb) / 100.

6a. ASR serving overhead from the published landmark

An ASR row whose publisher states a serving memory figure — OpenAI's Whisper README gives ~10 GB for large — is not sized from weights × width alone, because that figure is measured on a running transcription and this engine's modelled parts are not. The difference is real memory: encoder activations, audio and beam buffers and allocator slack. It is measured once, against a fixed reference configuration — the one the publisher measured, and nothing of what you asked for: FP16 weights, one sequence, an FP16 KV cache under the publisher's reference runtime, the decoder's 448-token limit, a headless OS, the default 10 % reserve and nothing co-located. It is then carried as one more part of the budget:

The reference is fixed because it describes a measurement someone else made. Rebuild it from your inputs instead and the overhead becomes a residual that shrinks as your request grows — raising the reserve to 50 % would then lower the total, and a workload that was TIGHT at the defaults would read FITS under strictly worse assumptions. Every deployment cost you add — reserve, OS, co-located memory, context, concurrency, a wider quantisation, a heavier runtime — is charged on your own budget, and this fixed term is added on top of it.

So at the published configuration the total is the published landmark (Whisper large-v3 FP16 at 448 tokens under the reference runtime: 10 GB, which does not fit an 8 GB module), while a narrower quantisation or a longer concurrency still moves the total by exactly what the weights and KV cache move. One consequence is worth stating plainly: the single published figure is carried across runtimes, because no publisher states an equivalent for MLC or llama.cpp, so a leaner runtime lowers the total by its own modelled workspace and not by this term. The overhead is class A by measurement and class E by attribution: the landmark is published, the split between "what we model" and "what we do not" is ours. Because it is part of the budget, every figure built from that budget — the verdict, the concurrency ladder, the largest-context sweep, the quantisation alternatives and the per-module table — carries it, and the alternatives table names published serving landmark as the basis for those rows.

7. OS headroom and reserve

Shared with Camera Stream Capacity so the two engines never disagree about what the OS costs.

Jetson modules share one memory pool between CPU and GPU (class A: NVIDIA · nvidia-docs:cuda-for-tegra-appnote §Memory Management · verified 2026-09-07 — “the iGPU and the CPU share the same SoC DRAM”), so the OS, the runtime and the model all draw on the same total; there is no separate VRAM.

8. Vision-pipeline mode

With no precision named in vision mode, the precision is the planning default of the task the model registry files the model under (method 1.3.0) — FP16 for a YOLO11-seg or YOLO11-pose model, INT8 for every other model — and on a platform that runs only INT8 (Coral, Hailo) the FP16 default falls back to INT8, the platform’s supported precision. It is listed in defaulted, and a task-specific default is marked workload.precision_default (DERIVED) with an assumption; a named precision is used as named.

For detector pipelines (YOLO, ResNet and similar), the engine does not duplicate the weight/activation/workspace model — the model-side parts come from the one shared V2 vision-memory module that Model Match also reads: the registry weight size at the chosen precision, and the published TensorRT activation and workspace split where one exists (class C), or that same published curve interpolated on to a parameter count it was not measured at (class E, and labelled as one). The workload-side parts — per-stream decode buffers (resolution mapped to the same buckets as Camera Stream Capacity: 720p 8 · 1080p 16 · 4K 64 MB), preprocess buffers (32 MB per stream), tracker memory and deployment overhead — are added here, and the total is rebuilt on this engine's shared budget function, so OS headroom and reserve are computed identically to every other mode. This keeps the two engines from silently disagreeing on a workload that could be scored by either.

9. Verdict thresholds and constraint statuses

The memory figure is also reported as a constraint object on the shared V2 framework (status PASS below 80%, NEAR_LIMIT from 80% through 100%, FAIL above 100%) — a slightly stricter early-warning line than the FITS/TIGHT/DOES_NOT_FIT verdict, which is tuned for the headline sentence. Two additional constraint types can appear: a software constraint (class A, HIGH confidence) fails when the requested context exceeds the model's max_position_embeddings, and a software constraint reports UNSUPPORTED when runtime is llama_cpp and the registry notes that no GGUF has been published for the model (VILA 1.5 3B and 8B, as of the current dataset). The headline verdict applies the kernel rollup across every material constraint; memory_verdict keeps the memory-only band above so a context or artefact failure cannot read FITS while memory alone would fit.

A memory total over the capacity is a FAIL only when the evidence can prove it (elimination authority). The weights, the KV cache and anything you co-locate are exact arithmetic on published architecture facts; the runtime overhead, activations, workspace, OS footprint and reserve are estimates. So the memory constraint FAILs — and the headline reads DOES_NOT_FIT — when the floor alone exceeds the memory: the weights at the format's storage width (the published artefact when there is one, else every parameter at the format's integer width), plus the KV cache for the text context, plus co-located memory. When only the estimated overheads take the total over, the constraint is REQUIRES VALIDATION with the modelled shortfall stated, memory_verdict still reads DOES_NOT_FIT and the headline reads UNKNOWN — not a pass, not an elimination.

10. Capacity landmarks and requested position

Headroom uses the same landmark vocabulary as Camera Stream Capacity: comfortable, near_limit, modeled_ceiling, and fails_at. For LLM workloads the ladder steps concurrent sequences; for vision-pipeline mode it steps camera streams. Each landmark records the count and the memory utilisation at that step. near_limit is the highest count whose utilisation sits in the memory tight band (85 % through 100 % inclusive). A genuinely empty comfortable rung is { sequences: 0, reason: "no configuration below the comfortable threshold" } rather than null; fails_at always carries the utilisation that placed that rung. requested_position states whether the caller's request sits below comfortable, between comfortable and near limit, at the modeled ceiling, or above what fits; the utilisation in that statement is always the requested configuration's own figure.

Alternative quantisations list weights_basis as either published_file_size (class A artefact on disk) or parameter_count (registry parameters × bytes per parameter).

11. Evidence classes and the confidence rule

Every figure carries one of five evidence classes.

Confidence: a result is MEDIUM when the weight figure came from a published artefact (class A) and module capacity is known; otherwise LOW. This is deliberately conservative — every result mixes at least one class D/E part (KV cache, runtime, OS, reserve), so nothing from this engine is ever labelled HIGH. The software constraint for a context beyond max_position_embeddings is HIGH confidence on its own (it is a direct comparison against a class-A config field), but does not raise the overall result above MEDIUM.

12. Model registry

22 rows: 10 LLM, 9 VLM, 3 ASR. Every field is copied from the publisher's config.json, safetensors index or GGUF repository listing; gated repos (meta-llama, google) are read from a byte-identical community mirror (unsloth, NousResearch) and the mirror is named in the source. Full detail per model, including quantised artefact sizes and Jetson tokens/s measurements: /model-fit/.

13. Module memory

Installed memory per Jetson module, from NVIDIA's technical specifications table (class A). jetson_orin_nano_super is an alias of jetson_orin_nano (same silicon, same memory) and is not listed separately.

14. Documents

15. What invalidates a result

• A new model release or GGUF re-quantisation. Every result URL carries mv (method version) and dv (dataset version); when either differs from the current values the result page says the data has changed since the link was made.
• A different runtime than assumed. The runtime-overhead table is an engineering heuristic (§6); a real llama.cpp build with a larger n_batch, or vLLM's pre-allocated gpu_memory_utilization, can use meaningfully more or less than the stated base + percentage.
• Container and driver overhead. The OS headroom figure (§7) is a bare-JetPack baseline; a Docker/L4T container runtime, X11/Wayland desktop session or other resident services add to it. The coexisting_mb input exists for exactly this.
• Context beyond max_position_embeddings. The engine still computes a KV cache for it and flags a class-A software constraint, but tokens/s and quality beyond the trained context are not modelled at all.
• No GGUF for llama.cpp. VILA 1.5 3B and 8B have no published GGUF as of this dataset; the engine reports UNSUPPORTED for that runtime rather than a number.
• Vision-tower and image-token estimates. Any model whose vision tower has no published mmproj/file size uses the class-D 12×L×H² estimate (§5), which can be off by 20–30% against the model's real parameter count.

16. Limitations and gaps

• No Jetson-measured memory rows yet. The registry's jetson_measurements carry tokens/s (mostly class C, from Jetson AI Lab archives), not measured RAM footprints; every memory total on this site is computed from the formulas above, not read off a running board. Validate with sudo tegrastats.
• Gated repositories. meta-llama and google repos on Hugging Face return 401 for unauthenticated reads; the registry uses byte-identical community mirrors (unsloth, NousResearch) instead and names the mirror in the source.
• VLM parameter counts include the vision tower. A model card's headline size (e.g. “Qwen2.5-VL 3B”) usually names the text backbone only; the registry's params_b is the full safetensors total, so it reads higher than the marketing name. The engine separates text and vision parameters internally (§3, §5) so weight pricing is not double-counted.
• Non-Jetson platforms. The engine falls back to the shared memory-hardware table for module capacity (class D) when a platform is not in the Model Memory Fit module registry; that path is LOW confidence.
• No accuracy or quality modelling. A quantisation that fits says nothing about output quality at that quant; that tradeoff is out of scope for this engine.

17. Method changelog

Method changes bump the method version; registry updates (new models, new measurements, corrected sources) bump the dataset version. Both appear in the citation block under every result and in the provenance object of every API response.