Running large language models locally
Model Shop
Image © sdecoret, 123RF.com
Ollama and Open WebUI let you join the AI revolution without relying on the cloud.
Large language models (LLMs) such as the ones used by OpenAI's [1] ChatGPT [2] are too resource intensive to run locally on your own computer. That's why they're deployed as online services that you pay for. However, since ChatGPT's release, some significant advancements have occurred around smaller LLMs. Many of these smaller LLMs are open source or have a liberal license (see the "Licenses" box). You can run them on your own computer without having to send your input to a cloud server and without having to pay a fee to an online service.
Because these LLMs are computationally intensive and need a lot of RAM, running them on your CPU can be slow. For optimal performance, you need a GPU – GPUs have many parallel compute cores and a lot of dedicated RAM. An NVIDIA or AMD GPU with 8GB RAM or more is recommended.
In addition to the hardware and the models, you also need software that enables you to run the models. One popular package is Ollama [3], named for Meta AI's large language model Llama [4]. Ollama is a command-line application that runs on Linux, macOS, and Windows, and you can also run it as a server that other software connects to.
[...]
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Is Linus Torvalds Vibe Coding?
The creator of Linux has added AI into his process for Linux kernel development, and it's more than just for debugging.
-
openKylin 3.0 Released with Expanded AI Integration
The latest release features a pre-installed AI agent to help perform tasks.
-
DigitalOcean Ends Open Source Credits Program
In a change that has surprised many open source developers, DigitalOcean has decided to sunset its Open Source Credits Program.
-
Office 365 Available for Testing on Linux via Bottles
Linux users can now test the official Office 365, thanks to Mirko Brombin's work on Bottles.
-
Dutch Government Turns to Open Source for Digital Sovereignty
Following in the footsteps of other European countries, the Netherlands is looking to increase its sovereign digital infrastructure with open source software.
-
Linux May Soon Work on Snapdragon X2-Based PCs
If you're thinking about buying a Snapdragon X2-based desktop or laptop, Linux may soon be an option.
-
KDE for People Initiative Calls for AI Ban in Plasma
A new community of KDE wants the project to adopt a strict no-AI policy for KDE Plasma.
-
KDE Sets Ambitious Goals for 2026 and Beyond
KDE Connect reveals the goals for the Linux desktop darling, with one of those goals long overdue.
-
It’s Time to Test Fedora 45 Beta
Fedora 45 beta has been released with updated Gnome, KDE Plasma, and kernel.
-
Ubuntu Stonking Stingray Gets Even Rustier
Ubuntu 26.10 has completed its migration to the Rust-based coreutils.