Latency vs Throughput: The Two Metrics That Define Real-Time AI Performance
Why optimizing for one often hurts the other — and how to find the right balance for video AI
If you've ever deployed a real-time AI system and watched it fall apart under production load, you've already met the latency-throughput tradeoff — even if you didn't have a name for it. This tension sits at the center of every video AI architecture decision: batching strategy, model choice, hardware selection, and deployment topology all come back to the same question: are you optimizing for speed on individual requests, or for volume across all of them?
The honest answer is that you need both — but you can't fully maximize both at the same time. Understanding why, and knowing when to prioritize which, is what separates a video AI pipeline that works in demos from one that holds up in production.
Defining the Terms Precisely
- Latency vs Throughput (Real-Time AI)
Latency is the elapsed time between a stimulus entering your AI pipeline — a video frame, a sensor event, an audio segment — and a result being produced. It is typically measured as a percentile distribution: p50 (median), p95, and p99. Throughput is the number of inputs your system can process per unit of time, usually expressed as frames per second (FPS) for video AI or requests per second (RPS) for API-based systems. The two metrics are related: for a fixed compute budget, higher throughput is generally achieved by batching inputs together, which increases per-request latency.
The reason percentiles matter more than averages: averages hide the tail. A system with a 50ms average latency might have a p99 of 800ms — meaning 1 in 100 frames takes nearly a full second to process. For safety-critical use cases like real-time object detection, that tail behavior is the failure mode you're designing against, not the median.
Why Latency and Throughput Trade Off
The root cause is batching. Modern GPU inference is dramatically more efficient when processing multiple inputs simultaneously than when processing them one at a time. A single image through a vision model might take 40ms. Eight images batched together might take 55ms total — 7ms per image, but with a 55ms wait for the last image in the batch.
That's the tradeoff in its purest form. If you care about the absolute fastest response to any single frame, you run batch size 1 — maximum latency consistency, minimum GPU utilization. If you care about processing the most frames per second with a fixed GPU, you run large batches — maximum throughput, with each individual frame waiting in the queue for the batch to fill.
8x
typical GPU throughput improvement from batch size 1 to batch size 8 on a vision inference workload — at the cost of 2-4x higher p99 latency
This relationship isn't linear and it doesn't scale indefinitely. There's a sweet spot for every model and hardware combination where the batch size delivers most of the throughput benefit without pushing latency into unacceptable territory. Finding that sweet spot is one of the core tasks in edge AI vs cloud AI deployment decisions.
The Pipeline Parallelism Alternative
Batching isn't the only lever. Pipeline parallelism is the other major strategy, and it operates differently.
Instead of waiting to accumulate a batch, you overlap the stages of your inference pipeline so that while one stage is processing frame N, the next stage is already working on frame N-1. In a video AI pipeline with four stages — decode, preprocess, infer, postprocess — you can have all four stages active simultaneously on different frames.
The effect on metrics is the opposite of batching: pipeline parallelism improves throughput without increasing per-frame latency. In fact, at sufficient parallelism depth, it can improve both — because idle compute time between stages is eliminated.
The catch is implementation complexity. Pipelining requires careful buffer management, synchronization primitives, and handling the case where a downstream stage is slower than an upstream one (backpressure). It's one of the reasons that purpose-built real-time video analytics architectures differ substantially from simple sequential inference scripts.
P50 vs P99: Why the Tail Is What Kills You
Most engineering teams measure average latency and report that as their system's performance. This is a mistake.
Average latency is a smoothed signal. In video AI pipelines, latency is rarely Gaussian — it has a long right tail caused by garbage collection pauses, network jitter, frame decode failures, and GPU memory pressure under concurrent load. Your median latency might be 90ms. Your p99 might be 1,400ms.
For a security monitoring use case, that 1,400ms p99 means that 1% of intrusion events take 1.4 seconds to detect. If your alert SLA is "under 500ms," you're out of compliance on 1% of events — which at 30fps across 10 cameras is roughly 9 events per second, or 54 missed-SLA events per minute.
1 in 100
frames exceeding your p99 latency threshold — across a 10-camera, 30fps deployment, that's over 50 SLA breaches per minute
The discipline of measuring and optimizing p99 (and p999 for the most demanding use cases) is what distinguishes production-grade GPU vs CPU inference decisions from prototypes. Edge deployment with a local NPU that delivers consistent 45ms p99 often beats cloud inference at 30ms p50 with a 900ms p99 — especially for time-sensitive detection tasks.
When Latency Is the Priority
Certain use cases have a hard latency ceiling. Below the ceiling, the system is useful. Above it, it isn't — regardless of throughput.
The clearest example is safety alerting. If a worker enters a restricted zone, the window for a useful alert is roughly 0-500ms. A 600ms response might still enable an intervention, but a 3-second response is a post-incident report, not a prevention. The same logic applies to quality inspection on fast-moving production lines: if a defect passes the rejection gate before the AI flags it, the throughput metric is meaningless.
For these use cases, the engineering strategy is: first, identify the latency ceiling. Second, work backward from that ceiling through the entire pipeline — stream protocol, frame extraction, model inference, alert delivery — to ensure every stage has a budget that sums to less than the ceiling. Third, choose model size and hardware that fit within that budget before considering throughput optimization.
This is also where the choice between RTSP, WebRTC, and HLS becomes a latency decision, not just a compatibility decision: RTSP adds 20-50ms of streaming latency vs WebRTC's sub-100ms, but both are meaningfully different from HLS's 5-30 second buffer.
When Throughput Is the Priority
Not every video AI application is time-sensitive. A significant class of real-world use cases cares primarily about processing volume:
Archival analysis. Processing hours of recorded footage to find specific events, tag objects, or build search indexes. Latency is irrelevant — the footage already happened. Throughput determines cost and time-to-results.
Batch quality reporting. Aggregating defect counts, dwell times, or occupancy metrics across a shift. The daily report doesn't need sub-second data — it needs accurate counts across thousands of frames.
Training data generation. Using AI to auto-label video frames for supervised learning. Speed per frame matters; individual frame latency doesn't.
For these use cases, maximize batch size, run larger models that would be too slow for real-time use, and process asynchronously. The model optimization decisions for edge deployment flip entirely: instead of choosing a fast, smaller model that fits within a latency budget, you choose the most accurate model that fits within a cost budget.
Measuring Both in Practice
The right instrumentation for a real-time AI pipeline tracks both metrics simultaneously, because they can diverge in ways that aren't visible from monitoring either alone.
For latency, instrument at the pipeline boundary: timestamp when a frame enters the system (or when the triggering event occurs), timestamp when the result is emitted, compute the delta. Track this as a rolling percentile — p50, p95, p99 — not as an average. Alert when p99 exceeds your SLA threshold.
For throughput, measure at the output: how many results does the system emit per second, sustained over a 5-minute window? Compare this to your camera count times your target analysis frequency. If you have 20 cameras and want to analyze every 2 seconds, you need at least 10 results per second sustained throughput — and you need that to hold under the load of all 20 cameras operating simultaneously.
The metric that often gets missed is the queue depth — how many frames are waiting to be processed at any given time. A growing queue is an early warning that your throughput is insufficient for your input rate, and that latency will soon blow out. This is the leading indicator; p99 latency is the lagging one. The video-to-LLM gap manifests precisely in systems where throughput was sized for average load but not peak load, causing queue growth and latency collapse under real conditions.
Choosing AI Models for Video Analytics
Model selection is where latency and throughput concerns converge most directly. Larger models are more accurate but slower; smaller models are faster but less accurate. The right model for video analytics depends on your latency ceiling and throughput target before it depends on accuracy benchmarks.
A practical framework: start with your latency budget and work backward. If your SLA is 300ms end-to-end and your pipeline overhead (streaming, decode, postprocessing) accounts for 100ms, your inference budget is 200ms. Benchmark models at batch size 1 on your target hardware — edge GPU, NPU, or cloud instance — and select the most accurate model that fits within 200ms at p99. Then run throughput benchmarks to confirm the model sustains your required FPS at that hardware configuration.
This is the sequence that matters: latency ceiling first, hardware selection second, model size third, accuracy evaluation last. Reversing the order — choosing the most accurate model first, then trying to squeeze it into a latency budget — is the source of most video AI deployment failures.
Keep Reading
- Edge AI vs Cloud AI: Where Should You Process Your Video Streams? — How deployment location determines your latency floor and throughput ceiling, with a decision framework for camera-heavy deployments.
- GPU vs CPU for AI Inference: What Actually Matters for Video Workloads — The hardware tradeoffs that set the upper bound on what your latency and throughput targets can be.
- Scaling Video AI Architecture: From 5 Cameras to 500 — How pipeline architecture evolves as camera count grows — and the throughput bottlenecks that appear at each scale inflection point.