What is Edge AI?

The four reasons that justify it

Latency. A network round trip costs tens to hundreds of milliseconds. For collision avoidance, robotic control, or interactive audio, that is unacceptable regardless of bandwidth.

Connectivity. Ships, mines, farms, factory floors, and moving vehicles cannot assume a reliable link. A system that fails without connectivity is not a system in those settings.

Privacy and regulation. Processing locally means raw data never leaves the device. For cameras in private spaces, clinical devices, and voice capture, this can be the difference between a deployable product and a compliance problem — and it is often the most compelling argument of the four.

Cost at scale. Streaming continuous video from thousands of cameras to the cloud is expensive in bandwidth and inference alike. Processing locally and transmitting only events inverts the economics.

What the constraints actually are

Edge hardware is meaningfully limited compared to a cloud GPU. Memory is measured in megabytes on microcontrollers, and power budgets matter when a device runs on battery or in a sealed enclosure with no active cooling — sustained inference generates heat, and thermal throttling silently degrades throughput in ways that never appear in a bench test.

Fitting a model into that envelope uses a familiar toolkit. Quantisation to 8-bit or lower is essentially standard, and some accelerators only support integer arithmetic. Pruning removes redundant weights. Distillation trains a small model to imitate a larger one and generally preserves more accuracy than shrinking an architecture directly. Purpose-built compact architectures such as the MobileNet family are designed for this regime from the outset.

Expect to give up some accuracy, and measure how much on your own task rather than trusting a published figure — degradation from compression is uneven, hitting precise and rare-case behaviour harder than average performance.

Deployment is the hard part

The modelling is usually not what makes edge projects difficult. Fleet operations are.

Updating a model on ten thousand devices in the field is a genuine distributed systems problem: staged rollouts, version tracking, verified integrity, and a reliable rollback path for devices that may be intermittently connected. Monitoring is harder than in the cloud, because the telemetry you would want may exceed the bandwidth you were trying to conserve — so decide early what minimal signal you need to detect drift. Hardware heterogeneity compounds everything, since a fleet accumulated over years spans several device generations with different accelerators.

Because of this, the pattern that usually wins is hybrid rather than pure edge. Run a small, fast model locally to handle the common cases and to filter the stream; escalate uncertain or high-value cases to a larger cloud model. You get edge latency and bandwidth economics for the bulk of traffic and cloud accuracy where it matters, while keeping the heavy model in one place where it is easy to update. Train centrally, deploy compressed models to the edge, and be deliberate about which decisions genuinely need to be made locally.