> ## Documentation Index
> Fetch the complete documentation index at: https://qualcomm-3.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# LLMs/VLMs using Llama.cpp

> Run large language and vision-language models locally on Dragonwing devices using llama.cpp, with CPU, GPU (OpenCL), or NPU (Hexagon HTP) backends.

You can run a wide range of Large Language Models (LLMs) and Vision Language Models (VLMs) on your Dragonwing development boards using [llama.cpp](https://github.com/ggml-org/llama.cpp). llama.cpp supports three backends on Dragonwing devices: CPU, GPU (via OpenCL), and NPU (via the Hexagon HTP backend). You can run a subset of models on the NPU via [GENIE](/ai-workflows/genie) as well.

## Choose a backend

* **CPU**: no special drivers or extra build flags needed. Useful as a baseline to compare against GPU or NPU performance.
* **GPU (OpenCL)**: build llama.cpp with the OpenCL backend to offload layers to the Adreno GPU. The same build also runs on CPU by skipping GPU offload.
* **NPU (Hexagon HTP)**: build llama.cpp with the Snapdragon toolchain container to offload layers to the Hexagon HTP device (`HTP0`) for the best performance on supported models.

<Tabs>
  <Tab title="NPU (Hexagon HTP)">
    ### Overview

    Build llama.cpp for Dragonwing devices with the Hexagon HTP backend, then run GGUF large language models on `HTP0`.

    <Note>
      These instructions focus on the Snapdragon and Dragonwing llama.cpp build that exposes the Hexagon HTP device as `HTP0`. This is different from the OpenCL GPU workflow in the GPU tab.
    </Note>

    ### Prerequisites

    Before you begin, make sure you have:

    * Completed the first time setup for your Dragonwing device:
      * [Dragonwing IQ8 setup](/devices/iq8275-evk/setup)
      * [Dragonwing IQ9 setup](/devices/iq9075-evk/setup)
    * Access to the device by SSH, or by a directly connected display, keyboard, and mouse.
      * The setup guides linked above include instructions for networking, serial console access, display setup, and SSH access.
    * Installed the required Dragonwing software packages on the device:
      * [Dragonwing IQ8](/devices/iq8275-evk/Install_required_software_packages)
      * [Dragonwing IQ9](/devices/iq9075-evk/Install_required_software_packages)
    * Installed Docker on the build host.
    * Enough free space for the build output and models. Plan for several GB per model.

    The required software package setup installs the QNN runtime and tools, including `libqnn-dev` and `qnn-tools`, that llama.cpp needs for accelerated inference.

    ### Prepare the build host

    On your build host, clone llama.cpp or update an existing checkout.

    ```shell theme={null}
    mkdir -p ~/src
    cd ~/src

    # Clone if this is your first build
    git clone https://github.com/ggml-org/llama.cpp.git
    cd llama.cpp

    # For repeat builds, update to the latest upstream changes
    git fetch origin
    git checkout master
    git pull --ff-only
    ```

    If you need a reproducible build, record the commit you built:

    ```shell theme={null}
    git rev-parse HEAD
    ```

    ### Build llama.cpp with the Snapdragon toolchain container

    The easiest way to build llama.cpp for Dragonwing is to use the Snapdragon ARM64 Linux toolchain container. The container includes the ARM64 cross compiler, CMake, OpenCL SDK, and Hexagon SDK pieces needed by the Snapdragon preset. The Docker command below explicitly requests the `linux/amd64` image.

    From the root of your llama.cpp checkout, start the container:

    ```shell theme={null}
    docker run -it \
      -u $(id -u):$(id -g) \
      --volume $(pwd):/workspace \
      --platform linux/amd64 \
      ghcr.io/snapdragon-toolchain/arm64-linux:v0.1
    ```

    Inside the container, configure and build llama.cpp:

    ```shell theme={null}
    cd /workspace
    cp docs/backend/snapdragon/CMakeUserPresets.json .

    cmake --preset arm64-linux-snapdragon-release -B build-snapdragon
    cmake --build build-snapdragon -j $(nproc)
    ```

    Create an installable package:

    ```shell theme={null}
    cmake --install build-snapdragon --prefix pkg-snapdragon
    zip -r pkg-snapdragon.zip pkg-snapdragon
    ```

    Exit the container when the package is complete:

    ```shell theme={null}
    exit
    ```

    The package archive is now available on the host at:

    ```text theme={null}
    ~/src/llama.cpp/pkg-snapdragon.zip
    ```

    ### Rebuild when upstream llama.cpp changes

    llama.cpp changes frequently. To rebuild with the latest upstream code, repeat this update and build flow from your host checkout:

    ```shell theme={null}
    cd ~/src/llama.cpp
    git fetch origin
    git checkout master
    git pull --ff-only

    docker run -it \
      -u $(id -u):$(id -g) \
      --volume $(pwd):/workspace \
      --platform linux/amd64 \
      ghcr.io/snapdragon-toolchain/arm64-linux:v0.1
    ```

    Then, inside the container:

    ```shell theme={null}
    cd /workspace
    cp docs/backend/snapdragon/CMakeUserPresets.json .

    rm -rf build-snapdragon pkg-snapdragon pkg-snapdragon.zip
    cmake --preset arm64-linux-snapdragon-release -B build-snapdragon
    cmake --build build-snapdragon -j $(nproc)
    cmake --install build-snapdragon --prefix pkg-snapdragon
    zip -r pkg-snapdragon.zip pkg-snapdragon
    ```

    <Note>
      If you are testing a branch, tag, or local llama.cpp changes, check out that source before running the Docker build command.
    </Note>

    ### Copy the package to the Dragonwing device

    Replace `ubuntu@DEVICE_IP` with your SSH user and target IP address.

    ```shell theme={null}
    scp ~/src/llama.cpp/pkg-snapdragon.zip ubuntu@DEVICE_IP:/home/ubuntu/
    ```

    Log in to the target device:

    ```shell theme={null}
    ssh ubuntu@DEVICE_IP
    ```

    Unpack the package:

    ```shell theme={null}
    cd ~
    unzip pkg-snapdragon.zip
    cd pkg-snapdragon
    ```

    Set the runtime library paths for the current shell:

    ```shell theme={null}
    export LD_LIBRARY_PATH=$PWD/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
    export ADSP_LIBRARY_PATH=$PWD/lib${ADSP_LIBRARY_PATH:+:$ADSP_LIBRARY_PATH}
    ```

    Verify that the package runs:

    ```shell theme={null}
    ./bin/llama-cli --version
    ```

    List the available llama.cpp devices:

    ```shell theme={null}
    ./bin/llama-cli --list-devices
    ```

    Expected output includes:

    ```text theme={null}
    Available devices:
      HTP0: Hexagon
    ```

    ### Make this the default llama.cpp install on the device

    The packaged binaries need `LD_LIBRARY_PATH` and `ADSP_LIBRARY_PATH` so they can find the packaged llama.cpp and Hexagon backend libraries. The safest way to make this the default install is to move the package into `/opt` and create wrapper commands in `/usr/local/bin`.

    Run the following on the Dragonwing device:

    ```shell theme={null}
    cd ~
    sudo rm -rf /opt/llama.cpp-snapdragon
    sudo mkdir -p /opt
    sudo cp -a ~/pkg-snapdragon /opt/llama.cpp-snapdragon
    sudo chown -R root:root /opt/llama.cpp-snapdragon
    ```

    Create a shared environment file:

    ```shell theme={null}
    sudo tee /etc/llama.cpp-snapdragon.env >/dev/null <<'EOF'
    LLAMA_CPP_SNAPDRAGON_HOME=/opt/llama.cpp-snapdragon
    LD_LIBRARY_PATH=/opt/llama.cpp-snapdragon/lib
    ADSP_LIBRARY_PATH=/opt/llama.cpp-snapdragon/lib
    EOF
    ```

    Create wrapper commands for `llama-cli` and `llama-server`:

    ```shell theme={null}
    sudo tee /usr/local/bin/llama-cli >/dev/null <<'EOF'
    #!/usr/bin/env bash
    set -euo pipefail
    source /etc/llama.cpp-snapdragon.env
    exec "$LLAMA_CPP_SNAPDRAGON_HOME/bin/llama-cli" "$@"
    EOF

    sudo tee /usr/local/bin/llama-server >/dev/null <<'EOF'
    #!/usr/bin/env bash
    set -euo pipefail
    source /etc/llama.cpp-snapdragon.env
    exec "$LLAMA_CPP_SNAPDRAGON_HOME/bin/llama-server" "$@"
    EOF

    sudo chmod +x /usr/local/bin/llama-cli /usr/local/bin/llama-server
    hash -r
    ```

    Confirm that the default commands now resolve to the wrappers:

    ```shell theme={null}
    which llama-cli
    which llama-server
    llama-cli --version
    llama-cli --list-devices
    ```

    Expected paths:

    ```text theme={null}
    /usr/local/bin/llama-cli
    /usr/local/bin/llama-server
    ```

    <Tip>
      `/usr/local/bin` usually appears before `/usr/bin` in `PATH`, so these wrappers become the default commands without replacing system packages.
    </Tip>

    ### Download a model

    llama.cpp uses models in GGUF format. A small instruct model is a good first test.

    Create a model directory on the Dragonwing device:

    ```shell theme={null}
    mkdir -p ~/models
    cd ~/models
    ```

    Download a Llama 3.2 3B instruct GGUF model:

    ```shell theme={null}
    wget https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_0.gguf
    ```

    <Note>
      Model support and performance vary by architecture, quantization, context length, and llama.cpp commit. If you find a model or quantization that runs particularly well on Dragonwing devices, share it with the community.
    </Note>

    ### Run your first prompt on HTP0

    Run `llama-cli` and offload layers to the Hexagon HTP device:

    ```shell theme={null}
    llama-cli \
      -m ~/models/Llama-3.2-3B-Instruct-Q4_0.gguf \
      --device HTP0 \
      -ngl 99 \
      -p "What is the most popular cookie in the world?"
    ```

    Useful options:

    * `--device HTP0` selects the Hexagon HTP backend.
    * `-ngl 99` asks llama.cpp to offload model layers to the selected device.
    * `-m` points to your GGUF model file.
    * `-p` passes a prompt for single prompt testing.

    ### Start llama-server

    `llama-server` exposes a local web UI and an OpenAI compatible API.

    Start a server on the Dragonwing device:

    ```shell theme={null}
    llama-server \
      -m ~/models/Llama-3.2-3B-Instruct-Q4_0.gguf \
      --device HTP0 \
      -ngl 99 \
      --host 0.0.0.0 \
      --port 8080
    ```

    Find the device IP address:

    ```shell theme={null}
    hostname -I
    ```

    From another machine on the same network, open:

    ```text theme={null}
    http://DEVICE_IP:8080
    ```

    You can also test the API with `curl`:

    ```shell theme={null}
    curl http://DEVICE_IP:8080/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
        "messages": [
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Explain Dragonwing in one sentence."}
        ],
        "temperature": 0.7,
        "max_tokens": 128
      }'
    ```

    ### Update the default install after a rebuild

    After you rebuild and copy a new `pkg-snapdragon.zip` to the device, update `/opt/llama.cpp-snapdragon`:

    ```shell theme={null}
    cd ~
    rm -rf pkg-snapdragon
    unzip pkg-snapdragon.zip

    sudo rm -rf /opt/llama.cpp-snapdragon
    sudo cp -a ~/pkg-snapdragon /opt/llama.cpp-snapdragon
    sudo chown -R root:root /opt/llama.cpp-snapdragon

    llama-cli --version
    llama-cli --list-devices
    ```

    The wrapper commands in `/usr/local/bin` do not need to be recreated unless you change the install path.

    ### Troubleshooting

    **`error while loading shared libraries`**

    If you run binaries directly from the package directory, set the library paths first:

    ```shell theme={null}
    cd ~/pkg-snapdragon
    export LD_LIBRARY_PATH=$PWD/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
    export ADSP_LIBRARY_PATH=$PWD/lib${ADSP_LIBRARY_PATH:+:$ADSP_LIBRARY_PATH}
    ./bin/llama-cli --version
    ```

    If you are using the default install wrappers, confirm the wrapper is being used:

    ```shell theme={null}
    which llama-cli
    ```

    It should print `/usr/local/bin/llama-cli`.

    **`HTP0` does not appear**

    Confirm that the required Dragonwing software packages are installed, especially `libqnn-dev` and `qnn-tools`. Then check devices again:

    ```shell theme={null}
    llama-cli --list-devices
    ```

    Also confirm the package contains Hexagon backend libraries:

    ```shell theme={null}
    ls /opt/llama.cpp-snapdragon/lib/libggml-hexagon.so*
    ls /opt/llama.cpp-snapdragon/lib/libggml-htp-*.so
    ```

    **The command uses the wrong llama.cpp binary**

    Check command resolution:

    ```shell theme={null}
    type -a llama-cli
    type -a llama-server
    ```

    If another path appears before `/usr/local/bin`, update your `PATH` or call `/usr/local/bin/llama-cli` explicitly.
  </Tab>

  <Tab title="GPU (OpenCL)">
    ### Building llama.cpp

    You'll need to build some dependencies for llama.cpp. Open the terminal on your development board, or an ssh session to your development board, and run:

    1. Install build dependencies:

       ```
       sudo apt update
       sudo apt install -y cmake ninja-build curl libcurl4-openssl-dev build-essential
       ```

    2. Install the OpenCL headers and ICD loader library:

       ```shell theme={null}
       mkdir -p ~/dev/llm

       # Symlink the OpenCL shared library
       sudo rm -f /usr/lib/libOpenCL.so
       sudo ln -s /lib/aarch64-linux-gnu/libOpenCL.so.1.0.0 /usr/lib/libOpenCL.so

       # OpenCL headers
       cd ~/dev/llm
       git clone https://github.com/KhronosGroup/OpenCL-Headers
       cd OpenCL-Headers
       git checkout 5d52989617e7ca7b8bb83d7306525dc9f58cdd46
       mkdir -p build && cd build
       cmake .. -G Ninja \
           -DBUILD_TESTING=OFF \
           -DOPENCL_HEADERS_BUILD_TESTING=OFF \
           -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF \
           -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
       cmake --build . --target install

       # ICD Loader
       cd ~/dev/llm
       git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader
       cd OpenCL-ICD-Loader
       git checkout 02134b05bdff750217bf0c4c11a9b13b63957b04
       mkdir -p build && cd build
       cmake .. -G Ninja \
           -DCMAKE_BUILD_TYPE=Release \
           -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \
           -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
       cmake --build . --target install

       # Symlink OpenCL headers
       sudo rm -rf /usr/include/CL
       sudo ln -s ~/dev/llm/opencl/include/CL/ /usr/include/CL
       ```

    3. Build llama.cpp with the OpenCL backend:

       ```
       cd ~/dev/llm

       # Clone repository
       git clone https://github.com/ggml-org/llama.cpp
       cd llama.cpp

       # We've tested this commit explicitly, you can try master if you want bleeding edge
       git checkout f6da8cb86a28f0319b40d9d2a957a26a7d875f8c
       git rev-parse HEAD
       # Expected: f6da8cb86a28f0319b40d9d2a957a26a7d875f8c

       # Build
       mkdir -p build
       cd build
       cmake .. -G Ninja \
           -DCMAKE_BUILD_TYPE=Release \
           -DBUILD_SHARED_LIBS=OFF \
           -DGGML_OPENCL=ON
       ninja -j`nproc`
       ```

    4. Add the llama.cpp paths to your PATH:

       ```
       cd ~/dev/llm/llama.cpp/build/bin

       echo "" >> ~/.bash_profile
       echo "# Begin llama.cpp" >> ~/.bash_profile
       echo "export PATH=\$PATH:$PWD" >> ~/.bash_profile
       echo "# End llama.cpp" >> ~/.bash_profile
       echo "" >> ~/.bash_profile

       # To use the llama.cpp files in your current session
       source ~/.bash_profile
       ```

    5. You now have llama.cpp:

       ```
       llama-cli --version
       # ggml_opencl: selected platform: 'QUALCOMM Snapdragon(TM)'
       # ggml_opencl: device: 'QUALCOMM Adreno(TM) 663 (OpenCL 3.0 Adreno(TM) 663)'
       # ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: 0808.0.7 Compiler E031.49.02.00
       # ggml_opencl: vector subgroup broadcast support: true
       ```

    ### Downloading and quantizing a model

    To run GPU-accelerated models you'll want pure 4-bit quantized (`Q4_0`) models in GGUF format (the llama.cpp format, [conversion guide](https://github.com/ggml-org/llama.cpp/discussions/2948)). You can either find pre-quantized models, or quantize a model yourself using `llama-quantize`. For example, for Qwen2-1.5B-Instruct:

    ```
    # Download fp16 model (https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF)
    wget https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF/resolve/main/qwen2-1_5b-instruct-fp16.gguf

    # Quantize (pure Q4_0)
    llama-quantize --pure qwen2-1_5b-instruct-fp16.gguf qwen2-1_5b-instruct-q4_0-pure.gguf Q4_0
    ```

    ### Running your first LLM using llama-cli

    You're now ready to run the LLM via `llama-cli`. It'll automatically offload layers to the GPU:

    ```
    llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off

    # ... You'll see:
    # load_tensors: offloaded 29/29 layers to GPU
    # ...
    # Knock knock, 11:59 pm ... rest of the story
    ```

    🚀 You now have an LLM running on the GPU of your device!

    ### Serving LLMs using llama-server

    Next, you can use `llama-server` to start a web server with a chat interface, and an OpenAI compatible chat completions API.

    1. First, find the IP address of your development board:

       ```
       ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'

       # ... Example:
       # 192.168.1.253
       ```

    2. Start the server via:

       ```
       llama-server -m ./qwen2-1_5b-instruct-q4_0-pure.gguf --no-warmup -b 128 -c 2048 -s 11 -n 128 --host 0.0.0.0 --port 9876
       ```

    3. On your computer, open a web browser and navigate to `http://192.168.1.253:9876` (replace the IP address with the one you found in 1.):

           <Frame caption="Serving LLMs using llama-server">
             <img src="https://mintlify.s3.us-west-1.amazonaws.com/qualcomm-3/images/ai-workflows/llamacpp1.png" />
           </Frame>

    4. You can also programmatically access this server using the OpenAI Chat Completions API. E.g. from Python:

       1. Create a new venv and install `requests`:

          ```
          python3 -m venv .venv-chat
          source .venv-chat/bin/activate
          pip3 install requests
          ```

       2. Create a new file `chat.py`:

          ```
          import requests

          # if running from your own computer, replace localhost with the IP address of your development board
          url = "http://localhost:9876/v1/chat/completions"

          payload = {
              "messages": [
                  {"role": "system", "content": "You are a helpful assistant."},
                  {"role": "user", "content": "Explain Qualcomm in one sentence."}
              ],
              "temperature": 0.7,
              "max_tokens": 200
          }

          response = requests.post(url, headers={ "Content-Type": "application/json" }, json=payload)
          print(response.json())
          ```

       3. Run `chat.py`:

          ```
          python3 chat.py

          # ...
          # {'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': 'Qualcomm is a leading global technology company that designs, develops, licenses, and markets semiconductor-based products and mobile platform technologies to major telecommunications and consumer electronics manufacturers worldwide.'}}], 'created': 1757073340, 'model': 'gpt-3.5-turbo', 'system_fingerprint': 'b6362-f6da8cb8', 'object': 'chat.completion', 'usage': {'completion_tokens': 34, 'prompt_tokens': 26, 'total_tokens': 60}, 'id': 'chatcmpl-3O7l005WG1DzN191FTNomJNweHMoH8Is', 'timings': {'prompt_n': 12, 'prompt_ms': 303.581, 'prompt_per_token_ms': 25.298416666666668, 'prompt_per_second': 39.52816546490064, 'predicted_n': 34, 'predicted_ms': 4052.23, 'predicted_per_token_ms': 119.18323529411765, 'predicted_per_second': 8.390441806116632}}
          ```

    ### Serving multi-modal LLMs

    You can also use multi-modal LLMs. For example [SmolVLM-500M-Instruct-GGUF](https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF). Download both the Q4\_0 quantized weights (or quantize them yourself), and download the CLIP encoder `mmproj-*.gguf` file. For example:

    ```
    # Download weights
    wget https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF/resolve/main/SmolVLM-500M-Instruct-f16.gguf
    wget https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF/resolve/main/mmproj-SmolVLM-500M-Instruct-f16.gguf

    # Quantize model (mmproj- models are not quantizable via llama-quantize, see below)
    llama-quantize --pure SmolVLM-500M-Instruct-f16.gguf SmolVLM-500M-Instruct-q4_0-pure.gguf Q4_0

    # Serve the model
    llama-server -m ./SmolVLM-500M-Instruct-q4_0-pure.gguf --mmproj ./mmproj-SmolVLM-500M-Instruct-f16.gguf --no-warmup -b 128 -c 2048 -s 11 -n 128 --host 0.0.0.0 --port 9876
    ```

    <Frame caption="Serving multi-modal LLMs using llama-server">
      <img src="https://mintlify.s3.us-west-1.amazonaws.com/qualcomm-3/images/ai-workflows/llamacpp2.png" />
    </Frame>

    **CLIP model is still fp16:** The `mmproj` model is still fp16; and thus processing images will be slow. There is code to quantize the CLIP encoder in [older versions of llama.cpp](https://github.com/ggml-org/llama.cpp/pull/11644), that you can explore.
  </Tab>

  <Tab title="CPU">
    ### Running on CPU

    You don't need a separate build to try CPU inference. The same OpenCL-enabled build from the GPU tab runs on CPU too. Add `-ngl 0` to any `llama-*` command to skip offloading layers to the GPU, which is a quick way to compare CPU and GPU performance on the same model and build.

    For example, the Qwen2-1.5B-Instruct Q4\_0 model (see the GPU tab for how to download and quantize it):

    **GPU:**

    ```
    llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off

    # llama_perf_sampler_print:    sampling time =      26.33 ms /   133 runs   (    0.20 ms per token,  5050.70 tokens per second)
    # llama_perf_context_print:        load time =    3535.69 ms
    # llama_perf_context_print: prompt eval time =     192.38 ms /     5 tokens (   38.48 ms per token,    25.99 tokens per second)
    # llama_perf_context_print:        eval time =    5679.81 ms /   127 runs   (   44.72 ms per token,    22.36 tokens per second)
    # llama_perf_context_print:       total time =    9276.10 ms /   132 tokens
    # llama_perf_context_print:    graphs reused =        122
    ```

    **CPU:**

    ```
    llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -ngl 0 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off

    # llama_perf_sampler_print:    sampling time =      15.44 ms /   133 runs   (    0.12 ms per token,  8615.66 tokens per second)
    # llama_perf_context_print:        load time =    1061.95 ms
    # llama_perf_context_print: prompt eval time =      51.75 ms /     5 tokens (   10.35 ms per token,    96.62 tokens per second)
    # llama_perf_context_print:        eval time =    2789.13 ms /   127 runs   (   21.96 ms per token,    45.53 tokens per second)
    # llama_perf_context_print:       total time =    3885.55 ms /   132 tokens
    # llama_perf_context_print:    graphs reused =        122
    ```

    Here the CPU evaluates tokens about twice as fast as the GPU for this small model. Smaller models can be memory-bandwidth bound enough that CPU keeps up with or beats the GPU, so it's worth comparing both for your specific model and quantization.

    `llama-server` also honors `-ngl 0`, so you can serve on CPU using the same command shown in the GPU tab's serving section.
  </Tab>
</Tabs>