Self-hosting large language models

1 · What it is

With a cloud-hosted model, your prompts go to the provider, which processes them to run the service. Self-hosting flips this. You run the model on a computer you control. When it runs locally, the tool maker does not see your prompts. llama.cpp is built to run models on many kinds of hardware, whether on your own computer or in the cloud.

The hard part is size. A model’s weights are the numbers it learned in training. They are usually stored as 32-bit or 16-bit floating-point numbers (numbers with decimals). Quantization stores them with fewer bits (the 0s and 1s a computer uses), so the model needs less memory. Some methods go down to 8-bit or 4-bit integers, and llama.cpp supports levels from 1.5-bit up to 8-bit. Quantization methods try to keep as much accuracy as they can, so it is a trade.

Here is an everyday example. With Ollama installed, typing ollama run gemma4 starts a chat with the Gemma 4 model. The server waits at address 127.0.0.1, on port 11434. A port is like a numbered door. By default, only that computer can reach it. The ollama ps command shows whether the model sits on the GPU (the graphics chip), the CPU (the main chip) or both. Apps written for OpenAI can switch to it by changing the base address (the web address the app sends its requests to).

For engineers, there are dedicated serving tools. vLLM is a library for running and serving models. It batches incoming requests. It also uses a method called PagedAttention to manage the memory the model uses to keep track of the text so far. Hugging Face’s Text Generation Inference server is now in maintenance mode, and its docs recommend vLLM, SGLang, llama.cpp or MLX going forward.