Running a Local LLM on Linux with llama.cpp and ROCm

AI-generated

I am using local LLMs more often for small coding tasks, note search and private text processing. With llama.cpp and ROCm it is fairly easy to use an AMD GPU on Arch or Gentoo without sending prompts to an external API.

My main local machines run Arch or Gentoo, and some of them have AMD GPUs available. For a single user setup I prefer running llama.cpp directly on the host. Containers are possible too, but ROCm device access and permissions add a bit more complexity. I use containers for long running services in my homelab, similar to my rootless Podman and Quadlet setup, but for an LLM server on my desktop I just use a systemd user service later.

Check that ROCm can see the GPU

First make sure that the normal amdgpu kernel driver is loaded and that your GPU is supported by your installed ROCm version. ROCm support is still more selective than the general Linux AMD driver support.

On Arch I install the HIP SDK and ROCm BLAS libraries:

$ sudo pacman -S --needed base-devel cmake git rocm-hip-sdk rocblas

I also make sure my user can access the GPU render devices. A logout and login is needed after changing groups.

$ sudo usermod -aG video,render yves
$ groups
yves wheel video render

Now rocminfo should show a GPU agent and its architecture name.

$ rocminfo | grep -m1 gfx
  Name:                    gfx1030

The gfx value is important when building llama.cpp. Mine can be different from yours, especially on newer RDNA GPUs.

Do not blindly use a ROCm package version from an old blog post. ROCm, the Linux kernel and Mesa move quite quickly. If rocminfo cannot see the card, llama.cpp will not fix that problem.

On Gentoo the package names and required USE flags depend a bit on the current tree and installed ROCm slot. I normally make sure HIP and rocBLAS are available first, then use the same llama.cpp build steps below.

Build llama.cpp with HIP support

llama.cpp supports AMD GPUs through HIP. I build it from source because it makes selecting the correct GPU target explicit.

$ git clone https://github.com/ggml-org/llama.cpp.git
$ cd llama.cpp
$ cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release
$ cmake --build build --config Release -j"$(nproc)"

Replace gfx1030 with the value from rocminfo. The resulting server binary is in build/bin/llama-server.

A quick check should show that the binary starts and lists its options:

$ ./build/bin/llama-server --help | head
usage: ./build/bin/llama-server [options]

If the build fails while finding HIP or rocBLAS, I usually check that my shell is using the same ROCm installation as CMake. Having an old manually installed ROCm release below /opt/rocm and newer distribution packages at the same time is a common source of strange errors.

Download a GGUF model

llama.cpp uses GGUF models. For local coding and note work I usually choose a reasonably sized instruct model with a Q4 or Q5 quantization. The right size depends much more on available VRAM than on GPU compute performance.

I use the Hugging Face CLI for downloads:

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install -U huggingface_hub
$ mkdir -p ~/models/qwen3
$ hf download bartowski/Qwen_Qwen3-8B-GGUF \
  Qwen_Qwen3-8B-Q4_K_M.gguf \
  --local-dir ~/models/qwen3

For an 8B model, Q4 is a useful starting point on GPUs with moderate VRAM. If you have more VRAM, a Q5 or Q6 model can make sense. Do not forget that the context cache also needs VRAM, especially with long notes or a large codebase.

I keep models outside my Git checkout. They are large files and it makes rebuilding llama.cpp much less annoying.

Start the OpenAI-compatible server

The server exposes OpenAI-compatible endpoints under /v1. I bind it only to localhost and use an API key even though the service is local. Some clients require one, and it stops accidental access from another local process.

$ ./build/bin/llama-server \
  -m ~/models/qwen3/Qwen_Qwen3-8B-Q4_K_M.gguf \
  -ngl 999 \
  -c 8192 \
  --host 127.0.0.1 \
  --port 8080 \
  --alias local-qwen \
  --api-key local-only

-ngl 999 asks llama.cpp to offload all possible layers to the GPU. Watch the startup log. It tells you how many layers actually moved to the GPU.

I normally start with an 8192 token context. Setting -c 32768 just because the model supports it can eat a surprising amount of VRAM. If the server runs out of memory, reduce the context size first, then use a smaller quantization or lower -ngl.

Test the API with curl:

$ curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer local-only' \
  -d '{
    "model": "local-qwen",
    "messages": [
      {"role": "user", "content": "Explain what a systemd user service is in two sentences."}
    ],
    "temperature": 0.2
  }'

The API shape is useful because most coding assistants and note tools already support an OpenAI provider. There is no need to write a custom llama.cpp integration.

Use the server from note and coding tools

For tools based on the official OpenAI Python SDK, the connection is straightforward:

$ export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
$ export OPENAI_API_KEY=local-only

Most graphical coding tools have equivalent fields named API base URL, endpoint or OpenAI-compatible URL. I set the model name to local-qwen, API base URL to http://127.0.0.1:8080/v1, and the key to local-only.

A typical provider configuration looks like this:

models:
  - name: Local Qwen
    provider: openai
    model: local-qwen
    apiBase: http://127.0.0.1:8080/v1
    apiKey: local-only

The exact option names depend on the tool, but the values are the same. For local Markdown notes, an MCP based setup can be useful too. I found this guide about connecting local Markdown notes to a local AI assistant with MCP a good addition when the model should search files instead of only answering from its prompt.

For coding, I keep the model limited to the current project directory. A local model is private, but it can still produce bad shell commands or modify files you did not intend to touch. The same applies to remote assistants, of course.

Do not bind llama-server to 0.0.0.0 unless you really need network access. The API key is not a replacement for proper network isolation.

If I need access from another machine in my homelab, I rather use Tailscale and firewall rules than publishing the port. My Tailscale exit node policy routing setup covers some of the routing details around that.

Keep the server running

After testing the command manually, I put it into a systemd user service. This starts it after login and makes logs available through journalctl --user.

[Unit]
Description=llama.cpp local LLM server

[Service]
ExecStart=%h/src/llama.cpp/build/bin/llama-server -m %h/models/qwen3/Qwen_Qwen3-8B-Q4_K_M.gguf -ngl 999 -c 8192 --host 127.0.0.1 --port 8080 --alias local-qwen --api-key local-only
Restart=on-failure

[Install]
WantedBy=default.target

For larger multi-user GPU servers, it is worth learning more about Linux service hardening and GPU resource management. The German Linux server hardening overview from cmt.de is useful background for that.

For my desktop use case, llama.cpp with ROCm is enough. It is fast, simple, and my notes and code prompts stay on my machine. If you know a better ROCm setup for llama.cpp, let me know.