GPU Setup

This guide provides detailed instructions for configuring Ollama to utilize GPU acceleration on different hardware platforms including NVIDIA, AMD, and Intel GPUs.

GPU Acceleration Overview

GPU acceleration dramatically improves Ollama's performance, enabling:

  • Faster model loading times

  • Increased inference speed (token generation)

  • Higher throughput for concurrent requests

  • Ability to run larger models efficiently

Hardware Requirements

GPU Manufacturer
Minimum Requirements
Recommended

NVIDIA

CUDA-capable GPU (Compute 5.0+) Pascal/10xx series or newer

RTX series (30xx/40xx)

AMD

ROCm-compatible GPU (CDNA/RDNA) Radeon RX 6000+ series

Radeon RX 7000 series

Intel

Intel Arc GPUs with OneAPI support

Intel Arc A770/A750

NVIDIA GPU Setup

NVIDIA GPUs offer the best performance and compatibility with Ollama through CUDA integration.

Prerequisites

  1. Install the NVIDIA driver:

    # Ubuntu/Debian
sudo apt update
sudo apt install -y nvidia-driver-535 nvidia-utils-535

# RHEL/CentOS/Fedora
sudo dnf install -y akmod-nvidia

  2. Install the CUDA toolkit (11.4 or newer recommended):

    # Download and install CUDA toolkit
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudo sh cuda_11.8.0_520.61.05_linux.run

  3. Add CUDA to your PATH:

    echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Configuring Ollama for NVIDIA GPUs

Ollama automatically detects NVIDIA GPUs when available. You can customize GPU utilization with environment variables:

# Use specific GPUs (zero-indexed)
export CUDA_VISIBLE_DEVICES=0,1

# Limit memory usage per GPU (in MiB)
export GPU_MEMORY_UTILIZATION=90

# Start Ollama with GPU acceleration
ollama serve

Verifying GPU Usage

# Check if CUDA is detected
ollama run mistral "Are you using my GPU?" --verbose

# Monitor GPU usage
nvidia-smi -l 1

NVIDIA Docker Setup

For Docker-based deployments:

# Install NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

# Configure Docker
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Run Ollama with GPU support
docker run --gpus all -p 11434:11434 ollama/ollama

AMD GPU Setup

AMD GPU support in Ollama uses the ROCm platform.

Prerequisites

  1. Install the ROCm driver stack:

    # Add ROCm apt repository
wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | sudo apt-key add -
echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/5.4.3/ ubuntu main' | sudo tee /etc/apt/sources.list.d/rocm.list

# Install ROCm
sudo apt update
sudo apt install -y rocm-dev rocm-libs miopen-hip

  2. Add your user to the render group:

    sudo usermod -aG render $USER
sudo usermod -aG video $USER

  3. Set up environment variables:

    echo 'export PATH=/opt/rocm/bin:$PATH' >> ~/.bashrc
echo 'export HSA_OVERRIDE_GFX_VERSION=10.3.0' >> ~/.bashrc
source ~/.bashrc

Configuring Ollama for AMD GPUs

# Configure Ollama for AMD GPUs
export OLLAMA_COMPUTE_TYPE=rocm

# For specific AMD GPU settings
export HSA_OVERRIDE_GFX_VERSION=10.3.0

# Start Ollama
ollama serve

Verifying AMD GPU Support

# Check if ROCm is detected
rocm-smi

# Check Ollama logs
ollama run mistral "Are you using my GPU?" --verbose

AMD Docker Setup

# Set up Docker container with ROCm
docker run --device=/dev/kfd --device=/dev/dri \
    --security-opt seccomp=unconfined \
    --group-add render \
    -p 11434:11434 \
    -e OLLAMA_COMPUTE_TYPE=rocm \
    -e HSA_OVERRIDE_GFX_VERSION=10.3.0 \
    ollama/ollama

Intel GPU Setup

Intel Arc GPUs can accelerate Ollama through OneAPI integration.

Prerequisites

  1. Install the Intel GPU drivers:

    # Ubuntu
sudo apt update
sudo apt install -y intel-opencl-icd intel-level-zero-gpu level-zero

# Install Intel oneAPI base toolkit
wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18673/l_BaseKit_p_2022.2.0.262_offline.sh
sudo sh l_BaseKit_p_2022.2.0.262_offline.sh

  2. Add OneAPI to your PATH:

    echo 'source /opt/intel/oneapi/setvars.sh' >> ~/.bashrc
source ~/.bashrc

Configuring Ollama for Intel GPUs

# Enable Intel GPU acceleration
export NEOCommandLine="-cl-intel-greater-than-4GB-buffer-required"
export OLLAMA_COMPUTE_TYPE=sycl

# Start Ollama
ollama serve

Verifying Intel GPU Support

# Check OneAPI configuration
sycl-ls

# Test with Ollama
ollama run mistral "Are you using my GPU?" --verbose

Troubleshooting GPU Issues

Common NVIDIA Issues

Issue
Solution

CUDA not found

Verify CUDA installation: nvcc --version

Insufficient memory

Reduce model size or context window: ollama run mistral:7b-q4_0 -c 2048

Multiple GPU conflict

Specify device: export CUDA_VISIBLE_DEVICES=0

Driver/CUDA mismatch

Install compatible versions: NVIDIA Compatibility

Common AMD Issues

Issue
Solution

ROCm device not found

Check installation: rocm-smi

Hip runtime error

Set HSA_OVERRIDE_GFX_VERSION=10.3.0

Permission issues

Add user to render group: sudo usermod -aG render $USER

Common Intel Issues

Issue
Solution

GPU not detected

Verify driver installation: clinfo

Memory allocation failed

Set -cl-intel-greater-than-4GB-buffer-required

Driver too old

Update Intel GPU driver

Performance Optimization

NVIDIA Performance Tips

# Use mixed precision for better performance
export OLLAMA_COMPUTE_TYPE=float16

# For large models on limited VRAM
export OLLAMA_GPU_LAYERS=35

AMD Performance Tips

# Adjust compute type for better performance
export OLLAMA_COMPUTE_TYPE=float16

# For large models on limited VRAM
export HIP_VISIBLE_DEVICES=0
export OLLAMA_GPU_LAYERS=28

Intel Performance Tips

# Optimize for Intel GPUs
export OLLAMA_COMPUTE_TYPE=float16
export SYCL_CACHE_PERSISTENT=1

Multi-GPU Configuration

For systems with multiple GPUs:

# Use specific GPUs (comma-separated, zero-indexed)
export CUDA_VISIBLE_DEVICES=0,1  # NVIDIA
export HIP_VISIBLE_DEVICES=0,1   # AMD

# Set number of GPUs to use
export OLLAMA_NUM_GPU=2

Real-World Deployment Examples

High-Performance Server (4x NVIDIA RTX 4090)

# Create a systemd service
sudo nano /etc/systemd/system/ollama.service
[Unit]
Description=Ollama Service
After=network.target

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="CUDA_VISIBLE_DEVICES=0,1,2,3"
Environment="OLLAMA_COMPUTE_TYPE=float16"
Environment="OLLAMA_NUM_GPU=4"
ExecStart=/usr/local/bin/ollama serve
Restart=always
User=ollama

[Install]
WantedBy=multi-user.target

Mixed GPU Environment (NVIDIA + AMD)

For environments with both NVIDIA and AMD GPUs:

# For NVIDIA
CUDA_VISIBLE_DEVICES=0 ollama serve

# For AMD (in separate instance)
HIP_VISIBLE_DEVICES=0 OLLAMA_COMPUTE_TYPE=rocm ollama serve --port 11435

NixOS GPU Configuration

For NixOS users, configure GPU acceleration in configuration.nix:

{ config, pkgs, ... }:

{
  # Enable NVIDIA driver and CUDA
  hardware.opengl.enable = true;
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.stable;
  hardware.nvidia.modesetting.enable = true;

  # Enable Ollama service with GPU acceleration
  services.ollama = {
    enable = true;
    acceleration = "cuda"; # Options: none, cuda, rocm, or oneapi
    package = pkgs.ollama;
    environmentFiles = [ "/etc/ollama/env.conf" ]; # Custom environment variables
  };
  
  # Create file with: 
  # OLLAMA_COMPUTE_TYPE=float16
  # OLLAMA_HOST=0.0.0.0:11434
}

Next Steps

After configuring GPU acceleration for Ollama:

  1. Explore available models optimized for GPU acceleration

  2. Set up advanced configurations for optimal performance

  3. Try real-world DevOps usage examples

  4. Set up Open WebUI for a graphical interface

PreviousDocker SetupNextOpen WebUI

Last updated