On-Device AI Explained: Running Edge Inference Locally

On-device AI is a deployment model that runs inference on the local hardware, letting smartphones, laptops, and embedded systems generate outputs without sending data to a remote server. The pattern has moved from research demo to shipping product in the last two release cycles, driven by dedicated neural processing unit (NPU) silicon from Apple, Qualcomm, and Intel, by aggressive quantization that compresses model weights into int8 and int4 formats, and by a maturing runtime stack built around Core ML, TensorFlow Lite, ONNX Runtime, and Google AI Edge. The result is a credible alternative to cloud inference for a growing slice of workloads: speech transcription, on-screen translation, image segmentation, and short-form text generation through small language models like Gemini Nano and Llama. For broader context on how AI models are served at scale in cloud infrastructure, see the AI infrastructure explainer on how AI models are served at scale.

What On-Device AI Actually Means

On-device AI means the model weights and compute graph execute on the chip inside the user's own device rather than on a remote server in a data center. The distinction matters because it changes the data path, the latency profile, and the failure mode. With cloud inference, a request leaves the device over HTTPS, hits a load balancer, runs on a GPU or TPU in a hyperscaler region, and returns a response, with every hop adding round-trip latency and exposing the payload to the network. With on-device AI, the same input reaches the NPU on the local SoC, executes against weights stored in device memory, and returns a result without crossing a network boundary. Edge inference is the umbrella term that covers both true on-device AI and nearby edge servers such as a base-station accelerator or a private appliance, while local inference is often used interchangeably with on-device inference in vendor documentation. The three categories share an architectural goal, which is to push computation toward the data rather than the reverse. The trade-offs differ by deployment surface, and when edge computing outperforms cloud deployments depends on latency, privacy, and bandwidth constraints that vary across applications.

• On-device AI: inference executed entirely on the chip inside the user's phone, laptop, or embedded device, with no network call.
• Edge inference: inference run on hardware physically close to the user, either on-device or on a nearby edge server, contrasted with central cloud inference.
• Cloud inference: inference executed on a remote GPU or TPU farm, reached over the network, with the model and data crossing the public internet.
• Local inference: functional synonym for on-device inference used across vendor documentation from Apple, Google, and Qualcomm.

The Hardware Layer: NPUs, CPUs, and GPUs on Device

On-device AI performance depends almost entirely on which dedicated silicon the device exposes for matrix multiply workloads. The Apple Neural Engine ships on every A-series and M-series chip and handles Core ML graphs at fixed-function speed, while Qualcomm Hexagon serves the same role on Snapdragon mobile SoCs and is the target for the Google AI Edge LLM Inference API on Android. Intel Core Ultra integrates an NPU alongside the iGPU and CPU on Meteor Lake and later laptops, and AMD Ryzen AI builds an XDNA NPU into recent mobile parts. The mobile GPU is a fallback when the NPU lacks a needed operator, and the CPU runs the long tail of small operations that neither accelerator handles efficiently. NVIDIA TensorRT documents the parallel pattern on discrete GPUs, where the SDK optimizes deep learning inference for NVIDIA hardware and supports mixed precision including FP32, FP16, BF16, FP8, and INT8 (NVIDIA, TensorRT documentation). The architectural overview confirms that TensorRT uses ONNX as the primary import format and ships an ONNX parser library to assist in importing models (NVIDIA, TensorRT architecture overview).

Quantization: Shrinking Models to Fit on Device

On-device AI deployments almost always require quantization because full-precision model weights are too large for the RAM available on consumer hardware. A 1B parameter model stored as FP32 occupies roughly 4 GB of memory, which exceeds the available RAM budget on most flagship smartphones once the operating system and the application itself are accounted for. Quantization reduces the bit width of each weight, dropping from FP32 to FP16, then to int8, and at the aggressive end to int4. The model size scales proportionally, so the same 1B parameter network compresses to about 500 MB at int4, which fits within the working set of a phone with 8 GB of RAM. The accuracy cost is real but bounded: int8 typically loses one to two percentage points on standard benchmarks, while int4 with calibration loses three to five. NVIDIA TensorRT supports the full mixed-precision pipeline including INT8 as part of its optimization toolkit, and its performance guidance describes benchmarking and optimization as two pillars treated as a feedback loop, measure first, optimize, then measure again (NVIDIA, TensorRT performance best practices). The mechanics of weight compression are covered in more depth in how AI model quantization and optimization work in depth.

1. FP32 baseline: the trained model in full precision, used as the accuracy reference and never deployed to consumer devices.
2. FP16 conversion: halves model size with negligible accuracy loss and is the default on mobile GPUs that support it.
3. int8 quantization: quarters model size against FP32, runs natively on most NPUs, and is the most common production target.
4. int4 weight compression: the aggressive frontier for on-device LLMs, packing two weights per byte and enabling 1B to 3B parameter models on phones.
5. Calibration: a post-training step that uses sample data to choose quantization scales that preserve accuracy at lower bit widths.

Runtimes and Frameworks for On-Device AI

On-device AI requires a runtime that translates a trained model graph into operations the local chip can execute efficiently. Core ML is the Apple-native runtime, converting PyTorch or TensorFlow graphs into a format the Apple Neural Engine, GPU, and CPU can run, and is the path Apple recommends for iOS and macOS apps. TensorFlow Lite, now folded into the LiteRT and Google AI Edge stack, targets Android, embedded Linux, and microcontroller boards through delegates that route operators onto Hexagon, Mali, Adreno, or the CPU. Open Neural Network Exchange (ONNX) is the cross-vendor interchange format, and ONNX Runtime executes those graphs across Windows, Linux, Android, and iOS using execution providers that map operators to NPU, GPU, or CPU backends. MediaPipe wraps perception pipelines around these runtimes, and the Google AI Edge LLM Inference API uses MediaPipe to run large language models completely on-device on Android (Google, LLM Inference API for Android). TensorRT serves the discrete GPU end of the spectrum on workstation and edge-server hardware. The runtime layer sits below the modeling layer, and how machine learning models are built and trained covers the training side that produces the graph the runtime then deploys.

Models That Run Locally Today

On-device AI model selection is constrained by the memory ceiling of the target device, which today sits between 4 GB and 16 GB on flagship smartphones. Gemini Nano is the Google-built small language model that ships on Pixel phones and is exposed to Android apps through the Google AI Edge stack, with the LLM Inference API documented as optimized for high-end Android devices such as recent Pixel and Samsung Galaxy phones (Google, LLM Inference API for Android). Llama 1B and Llama 3B are the Meta-released checkpoints sized for on-device inference deployment, distributed as open weights that downstream vendors quantize and integrate. Apple Intelligence ships Apple-built foundation models that run on the Apple Neural Engine for on-device features and fall back to Private Cloud Compute for heavier prompts. Microsoft Phi-3 mini and Phi-3.5 mini target the same on-device niche on Windows NPUs and have been packaged for ONNX Runtime. Whisper small and Whisper tiny are the open-weight speech models that fit comfortably on phones for transcription. Model size and capability remain in tension, and when small language models outperform larger cloud alternatives walks through the cases where a 1B to 3B parameter network is the right architectural choice.

• Gemini Nano: Google's small language model for Android, exposed through the AI Edge LLM Inference API on recent Pixel and Samsung Galaxy generation devices.
• Llama 1B and 3B: Meta's open-weight checkpoints sized for mobile and laptop NPUs, distributed in quantized form by partner vendors.
• Apple Intelligence on-device model: Apple's foundation model that runs on the Apple Neural Engine for short-form text and summarization.
• Phi-3.5 mini: Microsoft's small language model packaged for ONNX Runtime on Windows Copilot+ PCs with NPUs.
• Whisper small and Whisper tiny: open-weight speech recognition models that quantize cleanly for on-device transcription.
• MobileBERT and DistilBERT variants: compact transformer classifiers used for on-device search, intent detection, and content filtering.

Privacy and Offline Advantages of On-Device AI

On-device AI eliminates the network hop to a remote API, which means sensitive input data such as voice audio, camera frames, and typed text never leaves the device. The privacy gain is structural rather than policy-based: there is no third-party server log to subpoena, no payload to intercept in transit, and no provider retention window to negotiate. Data residency regulations that constrain cross-border transfers become moot when inference is local, since the data never crosses a border in the first place. The offline property is the second structural advantage. An on-device AI feature continues to function on an airplane, in a tunnel, on a ferry, or in a country where the cloud provider is blocked, because no network round trip is required to produce an output. The trade-off is that the model on the device is fixed at install time and updates ship as part of the OS or app release rather than continuously on the server. The Google AI Edge LLM Inference API documents that LLMs can run completely on-device for Android applications, which is the canonical statement of the offline-capable design point (Google, LLM Inference API for Android).

• No payload in transit: voice, image, and text inputs stay on the device, removing the wire-interception risk inherent to cloud inference.
• No server-side logs: the inference provider has no record of the prompt or the output because the inference provider is the device itself.
• Data residency by default: local inference avoids cross-border data transfer questions under GDPR, India's DPDP Act, and similar regimes.
• Offline operation: features work without a network connection once the model is installed, useful in transit, in field work, and in poor-coverage regions.
• Deterministic cost: no per-token billing, since the compute is amortized across device hardware the user already paid for.

When to Choose On-Device AI Over Cloud Inference

On-device AI is the better architectural choice when the application requires sub-100 ms response times, must operate without a network connection, or handles data the user cannot send to a third-party server. Latency is the most measurable axis. A round trip to a cloud GPU adds 50 to 300 ms before any inference begins, while an NPU on the same device can return a first token in under 20 ms once the model is loaded. Bandwidth is the second axis: an on-device AI feature does not consume cellular data on every interaction, which matters on metered plans and in low-bandwidth markets. Privacy and regulatory constraints push specific verticals such as healthcare transcription, legal note-taking, and financial document summarization toward local inference by default. The counter-case is also clear. Cloud inference wins when the workload requires a model too large to fit on device, when the application benefits from continuous model updates without a client release, or when the per-request compute cost is small enough that the network hop is not the bottleneck. The economics of the cloud path are covered in what drives the cost of cloud AI inference. OpenAI documents that its latest hosted models support text and image input, text output, multilingual capabilities, and vision, which is the workload profile cloud inference is optimized for and which on-device AI cannot match on consumer hardware (OpenAI, models documentation).

1. Latency budget under 100 ms: interactive features such as live captions, predictive keyboards, and AR overlays cannot tolerate a cloud round trip.
2. Offline requirement: apps that must work in transit, in the field, or in restricted networks need on-device AI to function at all.
3. Privacy-sensitive payloads: voice notes, medical dictation, and on-device document scans should not leave the device without explicit user action.
4. Bandwidth-constrained users: markets with metered cellular or low-quality fixed-line access benefit from edge inference that does not consume data per request.
5. Stable model and small payload: when the model can fit on device and the input size is bounded, local inference is both faster and cheaper to operate.
6. Workload too large for device: if the model exceeds the device's memory ceiling, or the request needs tool use against fresh web data, cloud inference remains the right call.

References

• NVIDIA, TensorRT documentation
• NVIDIA, TensorRT architecture overview
• NVIDIA, TensorRT performance best practices
• Google, LLM Inference API for Android
• OpenAI, models documentation
• NVIDIA TensorRT, GitHub releases

Frequently Asked Questions