How to Run a Local LLM on Your Phone (and Why You’d Want To)

There are now enough freely available Large Language Models (LLMs) that running them locally on a computer is commonplace, and a better idea in some ways—they're more private, as you don't have to send anything to the cloud, and they work offline.

Getting these advanced AI models running on phones hasn't been quite as straightforward (hence why the most advanced Siri AI features need one of the latest iPhones), but we're now at the point where most of the handsets are powerful enough, and some of the models are small and efficient enough that it's actually practically feasible.

The benefits are the same as on the desktop—you get an LLM that's always available and private to you, with nothing sent back to Google, OpenAI, Anthropic, or anyone else. The downsides are that you're going to get slower and more limited performance from your AI, as you're dealing with smaller and less powerful models, and you might see a hit with battery life (these AI chats can be quite resource-intensive).

Even with those compromises, the local AI you'll be able to get up and running on your phone is still going to be more than capable of everyday tasks and chats—and that means you can give Gemini or Siri (and perhaps your AI subscription) a rest.

The phone you need to run a local LLM

Any iPhone or Android phone launched in the last couple of years should do a decent job of running a local LLM; the AI boom means manufacturers have started building their devices with this kind of usage specifically in mind. Older phones can still apply, but you might need smaller models and get worse performance.

RAM is actually a bigger consideration than chipset performance when it comes to running an AI model. You're going to need 6GB or above for this to work at a satisfactory level, and 8GB or more is better for the larger models. Below 8GB, stick to 1-2B models—that B is for billion, and refers to the number of parameters in a model (essentially, how smart and versatile it's going to be).

In terms of storage space, you don't have too much to worry about. Even the largest models that use 7-8B are going to top out at around 5GB for how much room they take up on your phone, so you might even want to keep several AI models on your handset and switch between them as required.

If you buy a new flagship-level iPhone or Android phone today, it will actually come with local AI models preinstalled for quick tasks that can be done without pinging the cloud. However, these Apple and Google-made models aren't available directly to you, the user—they're used as and when Siri AI and Gemini think they're the best option.

The apps and models you need to run a local LLM

There are several apps that will do the job of getting AI models on your phone and running them. A couple of the most popular are Atomic Chat (Android or iOS) and PocketPal AI (Android or iOS). There are some differences—it's easier to extend Atomic Chat to desktop LLMs, for example, while PocketPal AI is a little more lightweight—but they're both great for getting started with local AI on your phone.

As for models themselves—sometimes called SLMs or Small Language Models because they're designed to work in more limited environments—you've got plenty of choice here, too. For instance, Gemma is the umbrella name of the open-source models that Google makes available for free, and some of these are specifically engineered for working in tighter spaces with fewer resources (like on your phone).

Meta has its own open-source AI models given the Llama moniker, while Microsoft has its Phi-4 models, which are also highly rated for efficiency. You just need to look for the versions with the lowest number of parameters in front of the "B" (or with "mini" in the name) to find the packages that'll work best with your phone.

At the moment, these SLMs are mostly text-only, although some of the newer, larger, and more advanced ones can analyze images and files. If you want to be able to generate images and videos, you're going to have to use the conventional cloud-based AI models, at least until the next leap forward in the technology.

Putting a local LLM to the test

To see how useful one of these local AI models might be on a phone, I installed PocketPal AI and one of the smaller Google Gemma models on my Pixel 9 Pro—not really a flagship any more, but not too old. I did like the little selection wizard that PocketPal AI opens with, that directs you straight to a suitable AI model for your phone.

Once I'd downloaded and installed a couple of models, it was simple enough to load them up and get started. PocketPal AI also gives you access to optional "pals" (hence the app name) that can tailor AI models for your use—the default Pip option seems fine as a stand-in for what you might be used to with the standard Gemini or Siri AI apps. Prompting and following-up works as normal, with your chat history saved by default.

There is a noticeable (and expected) slowness to the responses when you run LLMs on your phone, and a noticeable difference between AI model sizes—choosing a smaller model will get you answers significantly faster, even if they're not quite as smart or complete. It's worth experimenting with a few models just to find your own sweet spot between performance and speed.

With no web search or up-to-date knowledge available, this is best for brainstorming ideas, analyzing and refining existing text, composing new text, and getting fast facts or comparisons ("give me a film like..."). As always, watch out for those hallucinations, and don't take the word of any AI to be guaranteed as accurate.