A few years from now we may look back at this period and say that this is where it all started. With on-device models, a large company and a solo developer alike can build products without the privacy worries that come with sending user data to a server.
Let me say up front that I don't think we're quite there yet. But I don't think it's wrong to expect a wave of apps built for specific tasks over the next few years. When it comes to cost and to decentralised AI, these are important days.
Once on-device models stop making things up so often, AI may stop feeling like a "service" and start feeling like ordinary "software" that simply sits installed on your computer. That idea became much clearer to me over the past few weeks, while I was playing with the new small open models and reading the survey on on-device AI by Wang and colleagues. This post sits somewhere between the two: a bit of literature, a bit of notes from the field.
What exactly is on-device AI?
Put simply, on-device AI means the model runs inference directly on the device instead of sending the data to the cloud. It doesn't matter whether that device is a phone, a watch, a car, a sensor on a factory floor or a laptop. The literature points to three things these models have in common: they respond in real time, they have to make do with limited compute, memory and energy, and because the data stays local, they strengthen privacy.
A Gartner forecast from a few years ago sums up why this matters now: by 2025, around 75% of the data enterprises generate would be created outside traditional data centres. If data is already being produced at the edge, moving all of it to the centre and processing it there makes less and less sense in terms of latency, bandwidth and cost.
On the other side is model size. GPT-3, with 175 billion parameters, needed about 800 GB of storage, and putting a model like that on a phone was out of the question. Today there are language models designed for devices from the start, such as Gemma 2B and Llama 3.2 1B, and even multimodal models under a billion parameters. The gap closed much faster than I expected.
Cloud or device? It comes down to trade-offs
Running on the device isn't free: what you gain in latency and privacy, you pay back in scaling and maintenance.
| Aspect | On-device | Cloud-based |
|---|---|---|
| Compute | Limited by the device's hardware; optimisation is a must | Handles large, complex models comfortably |
| Latency | Low, no network round trip | Depends on the network; risky for time-critical work |
| Privacy | Data never leaves the device | Data goes to a server; the attack surface grows |
| Scaling | Fixed by the hardware you have | Resources grow with demand |
| Updates and maintenance | Rolling out to thousands of devices is tedious | Managed from one place |
| Cost model | Inference cost sits on the user's device | Pay per request or per token |
| Offline use | Yes | No |
What I see in practice is that this table isn't an either/or choice. The healthiest architectures are hybrid: work that is sensitive to latency or privacy stays on the device, and the genuinely heavy work goes to the cloud. The literature calls this cloud-edge collaboration or offloading; in practice it often starts with a small router model on the device.
Three layers of fitting a model on a device: data, model, system
To me the most useful idea in the survey is to spread the optimisation across three layers instead of relying on a single trick: the data, the model, and the system that runs it. Neglect one of them and much of what you gained in the other two tends to be left on the table.
- Noise filtering
- Dimensionality reduction
- Sensor fusion
- 32 → 8-bit precision
- Quantization
- Pruning
- Knowledge distillation
- Low-rank, NAS
- Inference frameworks
- NPU, GPU, CPU
- Scheduling
- Memory management
Data: cutting the work before it starts
"Garbage in, garbage out" hurts more on a device, because every unnecessary piece of data costs both compute and battery. Filtering out noisy measurements, keeping only meaningful features through dimensionality reduction, merging data from several sensors and lowering precision from 32 to 8 bits are the main tools in this layer. Each has a price: filter too aggressively and you throw away valuable signal along with the noise.
Model: doing the same job with fewer parameters
This is the layer people talk about most:
- Quantization: Reducing weights from 32-bit floating point to 8-bit integers or lower. The memory and speed gains are large; push it too far and accuracy drops, and not every piece of hardware supports every format.
- Pruning: Removing unimportant weights or neurons. Methods such as DropNet report cutting network complexity by up to 90% without hurting accuracy, but heavy pruning usually needs retraining.
- Knowledge distillation: Training a small "student" model on the outputs of a large "teacher". The small versions of Gemma 2 and Llama 3.2 used this approach in training too.
- Low-rank factorisation and parameter sharing: Approximating weight matrices with smaller ones, or reusing the same weights across layers.
- Hardware-aware architecture search (NAS): Searching for an architecture directly against the latency and energy budget of the target device.
These techniques work best together. The classic example is Deep Compression, which chains pruning, quantization and Huffman coding into one pipeline and shrinks the network considerably.
Architecture design itself makes a big difference, and the MobileNet series tells that story well. V1 reaches 70.6% accuracy on ImageNet with 569 million multiply-accumulate operations (MACs); V2 gets to 72% with 300 million, and V3 to 75.2% with about 219 million. The cost of architecture search followed a similar path: NASNet-A burned 48,000 GPU hours, while the hardware-aware ProxylessNAS reaches slightly higher accuracy (74.6%) in about 200 GPU hours.
The trend I find most interesting lately is "one model, many budgets". Once-for-All (OFA) trains a network once and extracts sub-networks for different devices from it; DynaBERT adjusts BERT's width and depth to the resources of the device. Today we see the same idea in embedding models through Matryoshka Representation Learning (MRL).
Google's EmbeddingGemma 2, released at the start of October, is a good example. It lets you cut its 768-dimensional vectors down to 128 dimensions and add or drop the image and audio encoders as needed. As a rough calculation, a float32 index for a gallery of 50,000 photos takes about 154 MB at 768 dimensions and 26 MB at 128. Shipping one model and letting each device trim it to its own budget is, to me, the most pragmatic answer to device diversity.
System: thinking about software and hardware together
On the software side, inference frameworks build the bridge between the model and the hardware. TensorFlow Lite (now called LiteRT) and PyTorch Mobile, which is steadily giving way to ExecuTorch, convert and optimise models for mobile; ONNX Runtime, OpenVINO, TensorRT, NCNN, MNN and Apache TVM focus on fast inference across different hardware. With language models, llama.cpp, MLX and Ollama joined the list.
On the hardware side the options are CPU, GPU, NPU, FPGA and ASIC. The CPU is everywhere but inefficient for AI work; the GPU is strong at parallel work but hungry for power; the FPGA is flexible but laborious to develop for; ASICs and NPUs offer unmatched efficiency for specific tasks and give up flexibility in return. That almost every phone above the mid-range now has an NPU is the development that really shifted the balance in practice.
Where on-device AI really makes a difference
When I look at the application areas, the common thread always comes down to one of three things: work where milliseconds matter, data that shouldn't leave the device, and places where the connection can't be trusted.
| Area | Typical task | Why on the device? |
|---|---|---|
| Autonomous driving | Object detection, lane and speed decisions | No time to wait for a round trip to the cloud |
| Wearable health | Tracking heart rate, rhythm and blood pressure | Health data is sensitive, battery is limited |
| Industrial IoT | Predictive maintenance, line monitoring | Bandwidth is expensive, the line must not stop |
| Smart home | Thermostats, cameras, automation | Video and audio from inside the home shouldn't leave it |
| Smart farming | Optimising irrigation and fertilising | Weak connectivity in the field |
| Smartphones | Voice assistant, photo processing, personal search | Instant response, personal data |
There is a legal upside too: when data never leaves the device, many of the questions raised by the GDPR (or KVKK, its Turkish counterpart) get simpler as early as the design stage.
The most interesting addition to this list in the past year, to me, is multimodal search. Small embedding models that put text, images, audio and video into the same vector space make it possible to index and query everything on the device, from your photo gallery to meeting recordings. Add a small language model and you get a RAG (retrieval-augmented generation) pipeline that runs entirely locally:
- Photos, documents, audio, video
- Embedding model
- Local vector index
- The user's question
- Same embedding model
- Similarity search in the index
- Small language model retrieved content + question
- Answer
Easy on paper, hard in the field
In benchmark tables everything looks fine; the real problems start when the model reaches real devices and real users.
Device diversity. Same model, same framework, different phone: in one measurement cited in the survey, ResNet50 on PyTorch Mobile takes 333 ms on a Galaxy S10e and 1,260 ms on an Oppo R17. That's roughly a fourfold difference. The number you measure on a flagship doesn't tell you what your average user will experience.
Energy and heat. The battery is the most underestimated constraint of on-device AI. A model working through thousands of files in the background drains the charge and heats the device until it slows itself down (thermal throttling). Simple scheduling decisions, like doing the work while the phone is charging and idle, often make more difference than the model itself.
Updates and maintenance. In the cloud you change the model once; on devices, dozens of different versions keep living in the field. Embedding-based systems make it harder still: vectors from different models aren't compatible with each other, so when the model changes, the user's entire index has to be rebuilt from scratch.
The cost in accuracy. Every compression technique gives something up. Even when average accuracy holds, you need to look at which examples you're losing. For languages that are less represented in training data, Turkish among them, testing on your own data is the only way to avoid surprises.
Security. Keeping data on the device is good for privacy, but it isn't a guarantee on its own: the device can be stolen, malware can get in, and the weights of a model shipped inside an app can be extracted. In the literature, trusted execution environments (TEEs), homomorphic encryption and federated learning are the main approaches to closing these gaps.
Small models still make things up. On the generative side, small language models hallucinate more easily than their bigger cousins. A good retrieval layer reduces that risk but doesn't remove it. This is exactly what I meant at the start about models making things up less often.
The next few years: trends I'm watching
The change I'm most looking forward to is foundation models getting smaller. The idea of pre-training a large model on broad data and then adapting it to a specific task with fine-tuning is now coming down to device scale. It's also why I expect a wave of apps built for specific tasks: the large model carries the general knowledge, and the small one on the device only has to do its own job well.
Hybrid architectures are maturing as well. 5G and whatever follows it make the split of work between device and cloud more flexible. When the device does the first pass and sends only what's needed, latency, bandwidth and cost all drop at once.
The hardest of these, I think, is learning on the device. Today most systems only run inference locally; training still happens in the cloud. Models that adapt to a user's habits on the device itself, along with federated learning, look like the next step, but given the memory and energy limits it's a much harder problem than inference.
Two debates are also getting louder. One is about energy: efficient models and low-power accelerators are now discussed not just for battery life but for the power data centres consume, and an architecture where not every query goes to a data centre has the stronger hand there. The other is about oversight. Bias and explainability in small models have become a research area of their own, and a model running on the user's device is harder to audit. Testing and transparency have to be designed in from the start rather than added later.
When AI turns from a service into software
Nobody thinks of SQLite as a "service". It runs on almost every phone and in almost every browser, yet nobody gets an API key for it, pays a bill for it or wonders where their data goes. It's just an ordinary component that ships inside the app.
AI models reaching the same place doesn't seem far off to me, and academia is looking in the same direction. The "Mobile Foundation Model as Firmware" paper, published at MobiCom 2024, proposes treating the foundation model as an infrastructure layer built into the device, rather than something every app packages separately. The maths changes for developers too: the inference cost moves to the user's device, and the per-token bill goes away.
For me this is the most concrete form the decentralised AI debate takes: the model's weights are open, and the hardware it runs on is in the user's pocket. Once generative models are reliable enough, saying "my app has AI in it" will sound as ordinary as "my app has a database" does today. Until then, my advice is to pick a small, specific problem and start getting to know these tools now.
Frequently asked questions
What is on-device AI?
It means an AI model runs inference directly on a phone, computer, car or IoT device without sending the data to the cloud. Its defining traits are low latency, the ability to work within limited resources, and data that stays local.
Is edge AI the same as on-device AI?
Close, but not the same. Edge AI is the broader umbrella for moving processing to any point near the data source (edge servers, gateways, devices); on-device AI describes the case where the work happens directly on the end user's device.
How do you fit an AI model on a device?
Usually by combining several techniques: quantization, pruning, knowledge distillation and efficient architecture design. A suitable inference framework and hardware accelerators such as an NPU complete the picture.
Can a large language model run on a phone?
Small language models can. Models in the 1 to 3 billion parameter range run on current phones with quantization, but speed, battery use and answer quality vary a lot from device to device.
Does on-device AI guarantee privacy?
Not on its own. Keeping data on the device lowers the risk, but device security, protecting the model and how the app stores data still have to be dealt with separately.
Sources and further reading
- Wang, X. et al. (2025). Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models. ACM Computing Surveys. arXiv:2503.06027
- Luo, C. et al. (2020). Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices. arXiv:2005.05085
- Yuan, J. et al. (2024). Mobile Foundation Model as Firmware. MobiCom 2024.
- Google DeepMind (6 October 2026). EmbeddingGemma 2 announcement and model card.
