Storage Layout and Ring Buffer Design for Edge AI

TL;DR

A well-designed storage layout on an edge AI node separates OS, model, log, and video partitions explicitly. The video partition operates as a ring buffer: fixed capacity, oldest data overwritten first. Retention period, write endurance impact, and over-provisioning are all calculated from camera count, bitrate, and drive TBW rating. This guide provides the formulas, partition sizing tables, and design patterns to get this right at build time.

Why Storage Layout Matters

An edge AI node that writes video continuously has a dramatically different storage profile than a desktop or server. On a server, storage layout is largely an operational convenience. On an edge node, the wrong layout causes:

• OS partition filled by video data, crashing the system
• Log data consuming space allocated for video, causing recording gaps
• Uncontrolled write growth exhausting SSD endurance ahead of schedule
• Model storage growing (new model versions deployed) until space runs out

Explicit partition boundaries with fixed maximum sizes for each data category prevent all of these failure modes. Design the layout at provisioning time and enforce it.

Ring Buffer Logic

A ring buffer (circular buffer) for video operates on a simple principle: when the buffer fills, writing wraps around to the beginning, overwriting the oldest data. The buffer has a fixed maximum size, so total disk usage is bounded and predictable.

For video, this is typically implemented at the segment level rather than the byte level: video is written in fixed-duration segments (e.g., 1-minute or 5-minute files). The ring buffer manager deletes the oldest segment file when total used space exceeds the configured maximum. This is simpler and more robust than byte-level circular writes, especially for video codecs that require complete segment files for playback.

Segment-level ring buffer behavior:

1. Video is written to segments: cam1_20260221_1430.mp4, cam1_20260221_1431.mp4, etc.
2. A monitor process checks total video partition usage every N minutes.
3. When usage exceeds 90% of the partition maximum, oldest segments are deleted until usage drops below 80%.
4. This 80/90% hysteresis prevents thrashing (constant delete-write cycles).

Retention Calculations

Given a video partition size, calculate retention period:

Retention (days) = Partition size (GB) ÷ Daily write volume (GB/day)

Daily write volume for continuous recording:

Daily write (GB) = Bitrate (Mbps) × Camera count × 86400 ÷ 8 ÷ 1000

Examples:

For event-triggered recording (e.g., record only when a detection event occurs), multiply daily write volume by the fraction of time recording is active. At 25% duty cycle, daily write volume drops by 75%, quadrupling retention at the same partition size.

Write Endurance Math

Write endurance determines how long the SSD will last under the ring buffer's sustained write load. Full derivation is covered in the SSD endurance guide; the key formula for quick sizing:

Drive life (years) = TBW ÷ (Daily writes × WAF × 365)

Where WAF (write amplification factor) is typically 1.2–1.5 for sequential video writes.

Example: 8 cameras at 4 Mbps continuous → 345 GB/day host writes × 1.3 WAF = 448 GB/day NAND wear. At 600 TBW drive: 600 ÷ (448 × 365 ÷ 1000) = 3.67 years. At 1200 TBW: 7.3 years.

For 8-camera continuous recording, a prosumer or enterprise NVMe rated 600 TBW minimum is required. Consumer drives at 300 TBW would last under 2 years at this write load.

Partition Design

Recommended partition layout for a 2 TB NVMe on an 8-camera Jetson node:

Mount the video partition with noatime to avoid read-triggered metadata writes that add unnecessary write amplification. For XFS on high-write video partitions, logbufs=8 and logbsize=256k improve journal write efficiency.

Storage Layout Patterns Comparison

Implementation Patterns

Segment-based ring buffer with bash + cron: The simplest implementation. A cron job runs every 5 minutes, checks video partition usage with df, and deletes the oldest segment files (sorted by creation time) until usage drops below the threshold. Robust for single-camera deployments and prototypes.

Application-managed ring buffer: The inference application (DeepStream pipeline, Python RTSP recorder, etc.) manages its own segment files, deleting oldest segments as new ones are created. More integrated but couples storage management to the application lifecycle.

LVM thin provisioning: For deployments needing flexible partition resizing without reformatting, LVM thin pools allow partitions to grow dynamically within a pool. Adds complexity but enables online resize if retention requirements change.

For the full node context including compute and networking layers, see the 8-camera reference architecture. For the SSD endurance foundation underlying these calculations, see SSD endurance for edge AI.

Common Pitfalls

• No partition separation: Writing video to a single root partition means a video write surge can fill the OS partition, preventing log writes, crashing systemd services, and potentially rendering the node unresponsive. Always separate OS and video storage.
• Setting ring buffer threshold too high: A ring buffer that deletes only when 99% full leaves very little time between deletion runs. Use 80–90% fill as the deletion trigger to keep continuous headroom for incoming writes.
• Not accounting for segment write-in-progress: A segment file currently being written is not yet complete. Ring buffer cleanup code must skip files that were modified in the last N seconds to avoid deleting an in-progress write.
• Forgetting log rotation: Inference logs grow indefinitely without log rotation. Configure logrotate on all log directories with size-based rotation limits that match the log partition size.
• Using a single NVMe for OS and high-write video on a consumer drive: OS writes (package updates, container layers, docker builds) and continuous video writes together accelerate consumer drive wear significantly. For sustained workloads, use prosumer or enterprise drives.
• Not testing ring buffer behavior at full capacity: The ring buffer must be tested by filling the video partition completely during development. This reveals edge cases in deletion logic before they occur in production.

FAQ

Should I use ext4 or XFS for the video partition?

Both are viable. XFS has better performance for large sequential writes and large files, making it slightly preferable for high-bitrate video. ext4 is more familiar and well-documented. Either choice is acceptable; the difference in practice is minor for this use case.

How do I prevent the ring buffer from deleting footage that triggered an alert?

When an alert fires, copy the relevant segments to a protected directory outside the ring buffer partition (e.g., the log partition). The ring buffer deletes only from the video partition; protected alert clips are managed separately with their own retention policy.

Can I use a network share (NFS, SMB) instead of local NVMe for the ring buffer?

Technically yes, but not recommended for production. Network latency and bandwidth variability make network storage unreliable for continuous high-bitrate video writes. Local NVMe provides deterministic write performance that network shares cannot match.

What is the minimum video partition size for a useful ring buffer?

Plan for at least 48–72 hours of retention at your expected write volume. Less than 24 hours of retention makes it very difficult to retrieve footage for incidents that are reported with a delay. Size accordingly using the retention calculation formulas above.

How do I handle time gaps in the ring buffer (camera offline, network down)?

Gaps are inevitable. Segment filenames include timestamps, so gaps are visible in the file listing. Some NVR systems write gap marker files or update a metadata database to track recording continuity. For compliance requirements, document gap handling explicitly in the system design.

Does the ring buffer design change for event-triggered recording?

The partition layout stays the same. The ring buffer cleanup thresholds can be raised since daily write volume is lower. Consider also implementing pre-trigger buffering: keep a short in-memory ring buffer of recent video so event clips include a few seconds before the trigger event.

Recommended Reading

• SSD Endurance for Edge AI: TBW and DWPD Explained
• NVMe for Jetson Orin Nano: Drive Selection and Thermals
• 8-Camera Edge AI Reference Architecture: Storage Layout
• Networking for Edge AI: Uplink and Bandwidth Sizing
• Power and UPS: Protecting Storage From Sudden Outages