Whisper large-v3 on Jetson Orin NX 8GB: FITS

Memory breakdown

Weights source

• OpenAI · github:openai/whisper/README.md · verified 2026-09-07 · source · class A

  1.55 B parameters

Headroom

Largest context that fits at 1 sequence: — tokens (recommended —). Max concurrent sequences at 448 tokens: 20 (recommended 4). Sequence ladder: 0 comfortable · 20 near limit · 20 modeled ceiling · fails at 21. 1 sequences is near the limit (82 % of module memory)

Other quantisations on Jetson Orin NX 8GB

Whisper large-v3 on other modules

Constraints

Sources

• NVIDIA · nvidia.com:jetson-modules-spec-table · verified 2026-09-07 · source · class A

  Jetson Orin NX 8GB: 8 GB

• OpenAI · hf:openai/whisper-large-v3 · config.json · verified 2026-09-07 · source · class A

  layers 32, kv_heads 20, head_dim 64, max context 448

• OpenAI · github:openai/whisper/README.md · verified 2026-09-07 · source · class A

  1.55 B parameters

• NVIDIA · nvidia-blog:mastering-llm-techniques-inference-optimization · Key-value caching · verified 2026-09-07 · source · class A

  KV cache size per token = 2 × (num_layers) × (num_heads × dim_head) × precision_in_bytes

• NVIDIA · nvidia-docs:cuda-for-tegra-appnote · Memory Management · verified 2026-09-07 · source · class A

  Jetson iGPU and CPU share the same DRAM: OS, runtime and model all draw on one pool.

Assumptions

• Weights: 1.55 B parameters × 0.625 bytes/parameter (median artefact size ÷ parameters across 22 registry rows) (class D).
• KV cache: 2 × 32 layers × 20 KV heads × 64 head dim × 448 tokens × 2 bytes × 1 sequence = 70 MB (class D formula).
• Runtime overhead: llama.cpp / Ollama: 400 MB base + 6% of weights (CUDA context + compute buffer for the prompt batch (n_batch 512)) (class E).
• OS 900 MB (headless) + 10% reserve from the shared memory budget (class E).
• ASR (encoder–decoder): KV cache uses the decoder text limit (448 tokens); encoder activations are inside the runtime workspace. Weights×width is not the published serving budget.
• ASR published memory landmark ~10 GB (OpenAI · github:openai/whisper/README.md · verified 2026-09-07) measured at FP16 weights and one sequence, where this engine's modelled parts for that same published configuration (FP16, one sequence, headless OS, 10% reserve, nothing co-located) come to 6122 MB: the difference, 4118 MB, is serving overhead the publisher saw and this engine does not model (encoder activations, audio and beam buffers, allocator slack). That reference is fixed — it describes what the publisher measured, not what is asked for here — so this overhead does not move with the request; it is carried on top of every quantisation, headroom step and alternative on this result, whose own parts carry the runtime workspace, reserve, OS and co-located workloads asked for here. No publisher states the same figure for llama.cpp / Ollama, so the one measurement is carried across runtimes. Weights × width alone would understate the serving budget.
• Memory evidence class D: registry-backed model data.

Method and limitations

total = weights (published artefact size, else parameters × bytes per parameter) + KV cache (2 × layers × KV heads × head dim × context × bytes × sequences) + vision tower (VLM) + runtime overhead + OS headroom + reserve, on the memory budget shared with Camera Stream Capacity. Verdict: FITS < 85% of module memory ≤ TIGHT ≤ 100% < DOES_NOT_FIT. Vision pipelines take their per-part figures from the memory estimator and are rebuilt on the same budget.

• Weights are class A when a published quantised artefact matches; otherwise parameters × bytes-per-parameter is class D for block quants (Q4, INT8, NVFP4) and class A for fixed-width types (FP16, FP8).
• KV cache, runtime overhead, OS headroom and reserve are class D/E formulas — engineering models, not vendor measurements. Validate on device with sudo tegrastats; llama.cpp prints its own KV and compute-buffer sizes at load.
• Full method, evidence classes and the confidence rule: Model Memory Fit methodology.

Links

• Live engine (this exact workload): https://edgeaistack.ai/engines/model-memory-fit/?mode=asr&hw=jetson_orin_nx_8gb&model=whisper-large-v3&quant=q4&ctx=448&seq=1&rt=llama_cpp&os=headless&mv=1.5.0&dv=2026-09-07
• Methodology: /methodology/model-memory-fit/
• Dataset: /datasets/model-memory.json
• Hugging Face repository: openai/whisper-large-v3