Safe at Home

Cloud-based Large Language Models (LLMs) pose a number of problems for the user. For one thing, you might accidentally share sensitive data that you didn't mean to share. You might also have high latency issues or discover that token limits restrict your productivity. Cloud-based services could also limit your ability to choose which models to work with.

As a security professional, my interest is in learning about LLM security, and working offline permits me to study possible attacks without breaching terms of service from cloud-based LLMs. My security interests aside, running local LLMs in an air-gapped environment is a great way to keep your precious code safe from prying eyes and bots while you work with sensitive data. It is possible to achieve total privacy and still reap the rewards of the eye-watering functionality afforded by LLMs.

In this article, I will demonstrate how I created my own local AI rig from scratch. This article walks through some of the gotchas and unapologetically spends some time getting the LLM performance working optimally courtesy of a few simple benchmarking tests.

[...]