Scaling Video AI from 10 to 10,000 Cameras: An Architecture Guide
Message queues, load balancing, and the infrastructure patterns that separate pilot projects from production
Running AI on 10 cameras is a weekend project. Running it on 10,000 is a distributed systems problem. Between those two numbers lies every uncomfortable truth about production infrastructure: the message queues that back up under load, the GPU pools that need to autoscale at 3am, the frame sampling strategies that determine whether your system is cost-effective or financially ruinous.
This guide is for engineering teams who have proved the concept — video AI works for their use case — and now need to scale it. Whether you're moving from pilot to production or planning an architecture that won't need a complete rewrite at 500 cameras, the patterns here apply.
Why Scaling Video AI Is Hard
Video AI pipelines combine three properties that individually are manageable and together are brutal: high data volume, strict latency requirements, and stateful processing.
A single 1080p camera at 30fps produces roughly 3 GB of raw pixel data per minute. You cannot send all of that to a Vision LLM — you need to sample intelligently, which requires stateful logic per stream. Each stream also needs its own connection management, reconnection logic, and context window. And all of this has to run continuously, with the kind of reliability you'd expect from infrastructure rather than an application.
3 TB
of raw video data generated per day by a single 1080p/30fps camera — before any AI inference is applied
Most teams underestimate this data gravity problem. The challenge isn't teaching the AI to understand video — modern Vision LLMs handle that well. The challenge is getting the right frames to the right inference workers at the right time, at scale, without either dropping critical events or spending a fortune on unnecessary API calls.
The good news: the industry has solved these problems in adjacent domains. The message queue and stream processing patterns that power financial trading systems, IoT telemetry platforms, and ad auction engines all apply here. Video AI is a specialization of a well-understood category.
Tier 1: The 10-Camera Architecture
At 10 cameras, your architecture can be simple — and it should be. Premature optimization at this tier wastes engineering time and creates complexity you'll have to unwind later.
- Frame Sampling
Frame sampling is the practice of selecting a subset of video frames for AI inference rather than processing every frame from a live stream. Effective sampling strategies balance event-detection coverage (not missing important moments) against inference cost and latency. Common approaches include fixed-rate temporal sampling (every N seconds), motion-triggered sampling (only when pixel change exceeds a threshold), and scene-change detection (when the visual content shifts significantly from the previous analyzed frame).
The 10-camera architecture looks like this: one application server, direct RTSP connections, a simple job queue (even a database table works), and a single pool of inference workers. You analyze frames at a fixed interval — typically 1 frame per second for most monitoring use cases — and POST results to a webhook or write them to a database.
The critical design decision at this tier is your frame sampling rate. One frame per second per camera means 864,000 inference calls per day across 10 cameras. At $0.002 per Vision LLM call (typical for GPT-4o mini or Gemini Flash), that's $1,728/month — already a meaningful cost. Motion-triggered sampling typically reduces this by 60–80% without meaningful loss in detection coverage.
For the stream connection layer, consider how your video protocol choice affects reconnection complexity. RTSP is the most common for IP cameras but requires robust reconnection handling. Starting with a managed stream API like Trio eliminates this complexity entirely at this tier.
Tier 2: The 100-Camera Architecture
At 100 cameras, three things break simultaneously if you haven't designed for them: connection management, inference throughput, and operational visibility.
Your single application server can technically hold 100 RTSP connections, but you now need to think about what happens when it restarts. You need a connection registry — some persistent store of which streams are being monitored — so that your workers can reconnect to the right cameras after any failure.
More importantly, you need a proper message queue between your stream ingestion layer and your inference layer. The queue decouples the two concerns: ingestion can run at whatever rate the cameras produce data, and inference workers can process at whatever rate your GPU budget allows. Without this decoupling, a spike in camera activity or a slow inference response will cascade across the system.
Redis Streams is the right choice at this tier. It's operationally simple, supports consumer groups for parallel processing, and handles the throughput of 100 cameras with ease. You don't yet need the complexity of Kafka.
Load balancing at 100 cameras is also a new concern. You need to distribute streams across workers in a way that's aware of each worker's capacity, and you need to rebalance when workers go down or new ones come up. A simple consistent hash of the camera ID to a worker — with a fallback reassignment on worker failure — handles this tier adequately.
This is also the tier where you decide between building this infrastructure yourself or using a managed stream API. At 100 cameras, building is still feasible but takes 6–10 weeks of focused engineering. A managed API is often the right call unless video AI is your core product.
Tier 3: The 1,000-Camera Architecture
A thousand cameras is a qualitatively different problem. At this scale, you need a real distributed message queue, horizontal autoscaling, and a cost model that won't bankrupt your organization.
Apache Kafka as the Central Nervous System
At 1,000 cameras, Kafka becomes the right choice. Its primary advantage over Redis at this scale is durability and replay: if your inference workers fall behind (due to a traffic spike or a deployment), frames are retained in Kafka and can be processed in order. Redis Streams has retention too, but Kafka's topic/partition model makes it dramatically easier to parallelize processing and reason about consumer lag.
The topic design matters: do not use a single topic for all cameras. A topic-per-camera-group (grouped by location, camera type, or priority tier) lets you set different retention policies, different consumer group sizes, and different sampling rates per group. High-priority cameras (entry/exit points, safety zones) can have aggressive sampling and dedicated consumers; lower-priority cameras (empty corridors, parking lots) can share consumer groups and use more conservative sampling.
GPU Worker Pools
At 1,000 cameras with motion-triggered sampling, you might be sending 500–2,000 frames per second to inference workers at peak. A single GPU can handle about 50–200 frames per second depending on model size and batch efficiency. You need a pool.
GPU autoscaling is more complex than CPU autoscaling because GPU instances take 2–4 minutes to provision and warm up. Your autoscaling trigger needs to be predictive, not reactive. Scale up based on Kafka consumer lag reaching a threshold, not based on CPU or memory utilization of existing workers.
Batching frames across cameras before sending to a Vision LLM is one of the highest-leverage optimizations at this tier. If you're sending 10 frames individually and then 10 frames as a batch, the batch approach typically delivers 3–5x cost reduction and 2–3x throughput improvement. This is because the per-request overhead (network round trip, model loading) is amortized across the batch. Most Vision LLM APIs support multi-image requests — use them.
For teams evaluating the edge vs. cloud tradeoff at this tier: 1,000 cameras is usually the inflection point where a hybrid architecture becomes compelling. Run fast, lightweight models at the edge to filter frames (motion detection, scene change detection), and send only the interesting frames to cloud-based Vision LLMs for complex reasoning. This can reduce cloud inference costs by 70–90%.
85%
reduction in cloud inference costs achievable with edge pre-filtering at the 1,000-camera tier, based on typical scene activity rates
Adaptive Frame Sampling
Fixed-rate sampling (1 frame per second) is a blunt instrument. At 1,000 cameras, adaptive sampling is essential for cost control.
A solid adaptive sampling system has three layers: a baseline rate (1 frame per 30 seconds for idle scenes), a motion trigger (bump to 2fps when pixel difference exceeds a threshold), and an event trigger (analyze every frame for up to 30 seconds when a high-confidence detection occurs). This approach typically reduces inference calls by 70% compared to fixed 1fps sampling while actually improving event coverage — because you analyze more frames during the moments that matter.
Tier 4: The 10,000-Camera Architecture
At 10,000 cameras, you are running a distributed systems organization, not just a video AI product. The architecture at this tier requires multi-region deployment, sophisticated SLO tracking, and a dedicated infrastructure team.
The Kafka deployment becomes multi-region. A single Kafka cluster cannot reliably handle 10,000 concurrent producers without significant operational risk. You partition cameras across regional clusters — typically by geography — with cross-region replication for any camera feeds that need multi-region processing. Cloud-managed Kafka services (Confluent Cloud, AWS MSK, Google Managed Kafka) become the right choice here unless you have deep Kafka operational expertise.
GPU capacity planning becomes a continuous process. At 10,000 cameras, your inference workload varies by 10–20x between 3am and peak afternoon hours. Reserved instances handle your baseline, spot/preemptible instances handle burst. Your autoscaling system needs to predict workload increases 5–10 minutes in advance to provision spot capacity before it's needed.
The latency vs. throughput tradeoff becomes explicit at this tier. You likely have camera feeds with different SLA requirements: safety-critical feeds need sub-5-second alert latency, while business-intelligence feeds (foot traffic, shelf occupancy) can tolerate 60-second latency in exchange for higher throughput batch processing. Separate inference queues with separate worker pools and separate SLOs for each priority tier is the right model.
Load Balancing Strategies for Video Streams
Standard HTTP load balancers are not the right tool for video AI pipelines. Video streams are long-lived, stateful connections — assigning a stream to a worker is more like a database shard assignment than a web request route.
Consistent hashing is the standard approach: hash each camera's ID to a worker, so the same camera always goes to the same worker (enabling per-stream state like motion baselines and temporal context). When a worker fails, only the cameras assigned to it need to be reassigned, not all cameras.
Capacity-aware placement adds a second dimension: don't just hash to a worker, hash to a worker that has capacity. Track each worker's current stream count and GPU utilization, and route new streams to the least-loaded worker in the appropriate hash ring segment. This prevents hot spots where one worker handles many high-activity cameras while another handles mostly idle ones.
Geographic affinity matters at scale: route cameras to inference workers in the same region to minimize the latency of frame transmission. A camera in Tokyo sending frames to an inference worker in Virginia adds 150ms of unnecessary round-trip latency — meaningful when your SLA is sub-5-second alert delivery.
Monitoring a Video AI Pipeline at Scale
You cannot operate a 1,000-camera video AI system on intuition. The metrics that matter are different from a typical web application, and teams often discover this the hard way when their queue backs up over a weekend and nobody notices.
The key metrics to instrument, grouped by layer:
Stream layer: Stream connection count per worker, reconnection rate per camera (high reconnection rate = camera problem or network issue), frames ingested per second, and frame drop rate (frames skipped because the queue was full).
Queue layer: Consumer lag per topic/partition group (the most important metric — this tells you if your workers are keeping up), message age at consumption (how old is the oldest unprocessed frame), and dead-letter queue volume (frames that failed processing and were moved to DLQ).
Inference layer: Frames processed per second per worker, GPU utilization per worker, inference latency (p50/p95/p99), batch size distribution, and Vision LLM API error rate.
Output layer: Alert delivery latency (time from frame capture to webhook delivery), alert volume by camera and category, and false positive rate by detection type.
For GPU vs. CPU inference decisions, your monitoring data will make this obvious: if GPU utilization is consistently below 40%, you're over-provisioned; if it's consistently above 80%, you're at risk of queue backup during traffic spikes.
The data privacy implications of your monitoring setup also deserve attention at scale. Frame-level telemetry that includes thumbnails or metadata about detected individuals needs to follow the same data handling policies as your primary inference outputs.
Cost Modeling at Scale
The cost structure of a video AI system changes significantly as you scale. At 10 cameras, inference cost dominates. At 10,000 cameras, infrastructure (compute, networking, storage) often exceeds inference cost.
A realistic cost model for 1,000 cameras with adaptive sampling:
- Stream ingestion and processing: ~$2,000/month (4 x
m5.xlargeinstances) - Kafka (managed): ~$800/month
- GPU inference workers: ~$6,000/month (10 x
g4dn.xlargespot instances with 30% uptime buffer) - Vision LLM API calls (estimated 200 calls/camera/hour with adaptive sampling): ~$8,600/month at $0.002/call
- Storage (frame thumbnails for audit): ~$400/month
- Monitoring and observability: ~$600/month
Total: ~$18,400/month for 1,000 cameras, or about $18.40/camera/month.
Comparison: the same deployment at fixed 1fps sampling would cost ~$58,000/month — adaptive sampling delivers a 68% cost reduction at this scale. The ROI calculation for video AI needs to account for this infrastructure cost alongside the value generated.
For teams building on Trio's stream API, the infrastructure tier is managed — you pay per stream and per inference call rather than provisioning and operating the queue/worker layer yourself. At 1,000 cameras, the break-even point between managed and self-hosted depends heavily on your engineering team's time cost, which is why the build vs. buy decision deserves a rigorous cost comparison.
The Path From Pilot to Production
The pattern that separates successful video AI deployments from failed ones is not the model choice or the camera hardware — it is whether the team designed the pipeline for the tier they are at while building the abstractions that allow them to reach the next tier.
Start at Tier 1. Prove the use case. Understand your actual frame sampling requirements and detection SLAs from real data, not projections. Move to Tier 2 when your camera count or reliability requirements demand it. Introduce Kafka only when Redis genuinely cannot keep up, not as a speculative future-proofing exercise.
The Video-to-LLM gap is real, but it is solvable tier by tier. The teams that scale successfully are the ones that resist the temptation to build a 10,000-camera architecture for their 50-camera pilot.
Keep Reading
- Build vs. Buy: Should You Build Your Own Video Analytics Pipeline? — A decision framework for teams choosing between custom infrastructure and a managed stream API, with cost and timeline comparisons.
- Edge AI vs Cloud AI: Where Should You Process Your Video Streams? — How to design the edge/cloud split that minimizes latency, cost, and bandwidth for your camera network.
- Getting Started with the Trio Stream API — Connect a live camera feed, define what to watch for, and get AI-powered insights back in minutes — without building the scaling infrastructure yourself.