On-device AI: how small language models run on your phone without the cloud

On-device AI means running an AI model directly on a phone, laptop or other gadget instead of sending the request to a data center. The model has to be small enough to fit in the device’s memory and frugal enough not to drain its battery. In return, the data stays local, answers need no network round trip, and the feature works offline.

Why the same model can be shrunk to fit in a pocket

A large cloud model is like a central library: huge, well stocked, and reached by sending a request across the network. An on-device model is more like a well-chosen shelf of reference books in your own room. It knows less, but it answers instantly and nobody sees what you look up.

Three levers make that shelf small enough. The first is fewer parameters, the numbers a model learns during training. The second is lower numeric precision, usually called quantization: storing each weight in 4 bits instead of 16, for example, which cuts memory sharply. The third is a narrow job. A model asked to do one well-defined task needs far less general knowledge than a chatbot.

Quantization normally costs some quality. Google’s Gemma 3 270M release in August 2025 included checkpoints trained with quantization-aware training (QAT), where the model learns during training to cope with the lower precision. Google says this allows INT4 precision with minimal performance loss, which suits resource-constrained devices.

Gemma and the push to make small models phone-sized

Gemma is Google DeepMind’s family of lightweight open models. The Gemma 3 technical report (Google, March 2025, arXiv 2503.19786) covers models from 1 billion to 27 billion parameters, trained with distillation, and describes changes to attention layers that reduce the memory needed for the KV cache during long inputs.

Google then pushed further toward hardware-limited devices. Its Gemma 3n family, described in a Google developer post dated June 26, 2025, comes in two sizes, E2B and E4B. Their raw parameter counts are 5 billion and 8 billion, but they run with memory footprints similar to 2B and 4B models, about 2 GB and 3 GB according to Google. Two ideas do the work. Per-Layer Embeddings move much of the parameter count to be computed on the CPU, so only the core transformer weights must sit in accelerator memory. MatFormer, short for Matryoshka Transformer, nests a smaller working model inside a larger one, like Russian dolls, so developers can extract the E2B model from E4B or build sizes in between.

At the small end, Gemma 3 270M, published August 14, 2025, has 270 million parameters, about 170 million of them embeddings for its 256k-token vocabulary and about 100 million in the transformer blocks. In Google’s internal tests on a Pixel 9 Pro, the INT4 version used 0.75% of the battery for 25 conversations. Google positions it for well-defined, high-volume tasks such as sentiment analysis, entity extraction, query routing and compliance checks, after fine-tuning.

A cheap gatekeeper in front of a smarter model

The newest public example is Singapore’s scam filter. In a post dated October 14, 2026, Google DeepMind and GovTech Singapore describe a scam detection model that runs on the phone, built by fine-tuning Gemma on local scam conversations, including impersonation of public institutions, banks and local figures.

Its design shows a common on-device pattern: a cascade. A lightweight classifier screens incoming messages using minimal power, and messages it judges safe pass through. Only suspicious ones go to the fine-tuned Gemma model, which examines a larger snapshot of the conversation and either confirms or dismisses the threat. Because the expensive step runs rarely, battery cost stays low. If manipulation is detected, the user sees a real-time warning urging them to pause before sharing sensitive information or sending money.

DeepMind says the analysis happens on the device, that the model accesses conversations only with the user’s consent, and that neither the Singapore government nor Google can read users’ conversations through it. Mark Chen of GovTech Singapore said: “Fighting scams shouldn’t mean sacrificing personal privacy.” The post does not say which Gemma variant is used.

Who else runs models locally

Google builds Gemma 3n on an architecture it says is shared with the next generation of Gemini Nano, the model family Google ships on Android devices, and it names Qualcomm, MediaTek and Samsung System LSI as hardware partners. Developers can run Gemma-family models with tools such as Hugging Face Transformers, llama.cpp, Google AI Edge, Ollama and MLX, according to Google’s launch materials. The common thread across these efforts is that the model, the compression method and the chip are designed together.

What a small model gives up

The first cost is capability. A model that fits in 2 GB of memory will generally reason less deeply and know fewer facts than a frontier model in the cloud, which is why on-device systems tend to target narrow tasks or hand hard cases elsewhere. The second is verification. The Singapore announcement, as published, gives no accuracy figures or false-positive rate, so outsiders cannot yet judge how often it misses a scam or flags an honest message. For a scam filter, both errors matter: a miss leaves a person exposed, and false alarms teach people to ignore warnings.

Privacy is also not automatic. Keeping inference local removes one exposure, but what matters is the whole system: what the host app permits, what is logged, and how the model was trained. In this case DeepMind says training ran in GovTech Singapore’s managed cloud environment with isolated data, personally identifiable information excluded, and anonymized reports used. Those are the labs’ own statements, not independent audits.

Finally, updating is harder. A cloud model can be changed overnight; a model on millions of phones depends on app and system updates, while scammers change tactics quickly.

What to watch next

Several things are worth tracking. One is whether DeepMind and GovTech publish evaluation numbers and name the Gemma size they ship, and when the phased rollout reaches the public. Another is whether Singapore’s scam losses, which Singapore Police Force data put at S$913.1 million in 2025 as reported by Malay Mail, fall further in the categories the model targets, though any change will be hard to attribute to one tool. More broadly, watch how small the useful models become: the move from 5B-class to 270M-class Gemma releases within about two months of 2025 shows the direction, and fine-tuning small models for one high-volume task is now a routine recipe rather than a research project.