Adamant Tools

Adamant Tools › Blog

  • Ollama
  • Docker
  • Local models
  • CPU
  • Deployment

Ollama in Docker on a server without a GPU: what to expect and how to measure it

Run Ollama in Docker on a CPU-only server, keep the model loaded, limit memory and queue, and measure load time and tokens per second from the API response.

Published 7 min readGitHubLeggi in italiano

Ollama in Docker on a server without a GPU: what to expect and how to measure it
Foto: Franklsf95, CC BY-SA 3.0, Wikimedia Commons

A small local model is often enough for classification, extraction or an internal assistant, and a virtual server without a GPU is the cheapest place to run it. Ollama publishes an official Docker image, so the setup takes a few minutes. What takes longer is learning how the server behaves under load: the first request is slow, an idle model is unloaded after a while, and a second concurrent user can double the memory use.

This guide gives a pinned Docker Compose file for a CPU-only host, explains the settings that matter, and provides a small benchmark client that measures your server using the numbers Ollama itself returns. It does not contain benchmark results. Speed on a CPU depends on the processor, the memory bandwidth, the model and its quantization, and any figure copied from another machine would be a guess. Measure on your own hardware.

Versions checked on 3 November 2026: Ollama 0.40.14 (the ollama/ollama:0.40.14 tag on Docker Hub, pushed that day). The benchmark client uses only the Python standard library.

Start Ollama without a GPU

The official Docker documentation gives this command for CPU-only use:

bash
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

The named volume ollama holds the downloaded models in /root/.ollama; without it, every new container would download them again. Port 11434 is the API. For a server that stays up I prefer Compose, with the image pinned to a version instead of latest, so that an update is a decision and not a surprise:

yaml
services:
  ollama:
    image: ollama/ollama:0.40.14
    ports:
      - "127.0.0.1:11434:11434"   # reachable only from this machine
    volumes:
      - ollama:/root/.ollama      # models survive container recreation
    environment:
      OLLAMA_KEEP_ALIVE: "30m"    # default is 5m: keep the model in RAM longer
      OLLAMA_NUM_PARALLEL: "1"    # default 1: parallel requests multiply the context memory
      OLLAMA_MAX_QUEUE: "16"      # default 512: reject early instead of queueing for minutes
    mem_limit: 8g                 # model + context must fit, or the container is OOM-killed
    restart: unless-stopped
volumes:
  ollama:

Then download a model and call it:

bash
docker compose up -d
docker compose exec ollama ollama pull llama3.2:1b
curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.2:1b","prompt":"Hello","stream":false}'

The model name here is only an example of a small model. Which model fits your task and your RAM is your choice, and the model library lists sizes. The pull endpoint of the API resumes cancelled downloads, so a dropped SSH session does not mean starting from zero.

The settings that matter

The FAQ documents the environment variables. These are the ones I set in the file above, with the defaults as the FAQ states them on the day of writing:

Variable Default Why change it on a small server
OLLAMA_KEEP_ALIVE 5 minutes After this idle time the model is unloaded from memory, and the next request pays the load time again. For a service used a few times an hour, 5 minutes means frequent cold starts.
OLLAMA_NUM_PARALLEL 1 Number of requests each model processes at the same time. Raising it needs more memory, and on a CPU it does not make the machine faster: requests share the same cores.
OLLAMA_MAX_QUEUE 512 Requests queued while the server is busy before it starts rejecting. With a CPU generating a few tokens per second, a queue of 512 means minutes of waiting. A short queue turns overload into an error you can handle.
OLLAMA_MAX_LOADED_MODELS 3 for CPU inference How many different models stay in memory. Each one occupies RAM; loading three models of 4 GB each needs about 12 GB before any context.

OLLAMA_HOST controls the bind address of the server process; Ollama binds 127.0.0.1:11434 by default. In the Compose file the exposure is decided by the port mapping: 127.0.0.1:11434:11434 makes the API reachable only from the host itself. The Ollama API has no authentication. If other machines need it, put a reverse proxy with authentication in front of it, and do not publish the port to the internet. A container with -p 11434:11434 on a public server is reachable by anyone who finds it.

The mem_limit line is a Docker setting, not an Ollama one. If the model plus the context do not fit, the kernel kills the container; a limit makes that failure predictable and protects the other services on the same host. How much memory a model needs depends on its size, the quantization and the context length you request, so check with docker stats while the model is loaded and answering.

Cold start and warm start

The first request after a pull, or after the keep-alive time, has to read the model from disk into memory. That is the cold case. Later requests find the model loaded: the warm case. On a CPU server with a slow disk the difference can be large, and users notice it.

There are two ways to control it, both documented in the FAQ. You can preload a model by sending an empty request, and you can set keep_alive per request: 0 unloads the model right after the answer, a negative number such as -1 keeps it loaded. A simple deployment pattern is to preload the model when the container starts:

bash
curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.2:1b","keep_alive":"30m"}'

The benchmark client has the same call as a function, so a health check or a start script can use it.

Read the numbers Ollama gives you

The final response of /api/generate contains durations and token counts, as described in the API reference. All durations are in nanoseconds:

Field Meaning
total_duration Total time of the request
load_duration Time spent loading the model
prompt_eval_count, prompt_eval_duration Tokens in the prompt and time spent evaluating the uncached ones
eval_count, eval_duration Tokens generated and time spent generating them

The documentation gives the formula for generation speed: eval_count divided by eval_duration, times 10^9, which is tokens per second. The same calculation on the prompt fields gives the prompt processing speed. On a CPU these two behave differently: reading a long prompt is usually much faster per token than generating, but a long prompt still costs seconds, and a RAG application that sends several thousand tokens of context pays for it on every request. Our guide on counting tokens shows how to know the prompt size before the call.

Here is the client. It uses stream: false to get a single response object with all the fields, and options.num_predict to bound the length of the answer so that the measurement stays short:

python
"""Measure an Ollama server: load time, prompt speed, generation speed. Standard library only."""
import json
import sys
import urllib.request

NS = 1_000_000_000


def post(base_url: str, path: str, body: dict, timeout: float = 600) -> dict:
    req = urllib.request.Request(
        base_url + path, data=json.dumps(body).encode(),
        headers={"content-type": "application/json"}, method="POST")
    with urllib.request.urlopen(req, timeout=timeout) as res:
        return json.load(res)


def preload(base_url: str, model: str, keep_alive: str = "30m") -> dict:
    """An empty request loads the model into memory without generating anything."""
    return post(base_url, "/api/generate", {"model": model, "keep_alive": keep_alive})


def generate(base_url: str, model: str, prompt: str, num_predict: int = 64) -> dict:
    return post(base_url, "/api/generate", {
        "model": model, "prompt": prompt, "stream": False,
        "options": {"num_predict": num_predict}})


def summarize(r: dict) -> dict:
    """Turn the final response fields (all durations in nanoseconds) into readable numbers."""
    def rate(count_key: str, dur_key: str) -> float | None:
        count, dur = r.get(count_key, 0), r.get(dur_key, 0)
        return round(count / dur * NS, 2) if dur else None

    return {
        "load_s": round(r.get("load_duration", 0) / NS, 2),
        "total_s": round(r.get("total_duration", 0) / NS, 2),
        "prompt_tokens": r.get("prompt_eval_count", 0),
        "prompt_tokens_per_s": rate("prompt_eval_count", "prompt_eval_duration"),
        "output_tokens": r.get("eval_count", 0),
        "output_tokens_per_s": rate("eval_count", "eval_duration"),
    }


def main(base_url: str, model: str) -> None:
    prompt = "Explain in three sentences what a reverse proxy does."
    cold = summarize(generate(base_url, model, prompt))   # may include the model load
    warm = summarize(generate(base_url, model, prompt))   # model already in memory
    print(json.dumps({"cold": cold, "warm": warm}, indent=2))


if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:11434",
         sys.argv[2] if len(sys.argv) > 2 else "llama3.2:1b")

Run it against your server and read the two blocks:

bash
python bench.py http://127.0.0.1:11434 llama3.2:1b

The output has a cold and a warm entry with load_s, prompt_tokens_per_s and output_tokens_per_s. If the first call finds the model already in memory, the two entries will look alike: to see a real cold start, restart the container or wait for the keep-alive time first. Repeat the measurement a few times, at different times of day if the host is shared, because virtual servers can have neighbours that use the same physical cores.

What the numbers mean for your application

  • Output speed is the user experience. A person reads at a few words per second. If your measured generation speed is lower than that, stream the answer to the browser instead of waiting for the whole text; the guide on streaming with SSE shows how.
  • Total time is prompt time plus generation time plus (maybe) load time. Estimate it as prompt_tokens / prompt speed + max output tokens / output speed, and set your client timeout well above that.
  • Concurrency is a capacity question. With OLLAMA_NUM_PARALLEL=1, a second user waits for the first. If your peak is more than one request at a time, work out how many requests per second the server really sustains and compare it with the expected traffic; our capacity sizing tool does that arithmetic. Past a certain point the answer is a smaller model, a shorter prompt, a GPU or a hosted API, not a larger queue.
  • Overload should be an error. With a short OLLAMA_MAX_QUEUE, a rejected request can be retried later with backoff; see HTTP 429 and Retry-After for the pattern. I did not verify which HTTP status Ollama returns when the queue is full, so check it on your version and map it in your client; our HTTP errors reference helps to read what you get.

Test it without a model

The example folder has a fake Ollama server written with the standard library. It answers /api/generate with fixed numbers: a slow first call that includes a load time, and faster ones afterwards. The tests check the request your client sends (stream: false, num_predict, an empty body for preload), the unit conversion from nanoseconds, the division by zero case, and that the cold run is slower than the warm one:

text
test_cold_slower_than_warm ... ok
test_generate_request_shape ... ok
test_preload_sends_no_prompt ... ok
test_summarize_units ... ok
test_zero_duration_gives_none ... ok

The suite runs in GitHub Actions on every change.

Limits of this guide

  • I could not run Docker or a real model in the environment where this article was prepared. The CI tests prove that the client sends the right requests and computes the right numbers from the documented fields; they do not prove the Compose file. I wrote it from the official documentation, so test docker compose up on your server before relying on it.
  • There are no speed figures in this article on purpose. They would depend on a machine you do not have.
  • The Ollama API has no built-in authentication, and a model answering from a CPU is easy to overload. Both are reasons to keep the port closed and to put limits in front of it.
  • Defaults change between versions. The values above are those documented on 3 November 2026 for 0.40.x.
Full code on GitHub
The complete, tested example for this article is in our public repository: adamant-solutions/ai-dev-examples/ollama-docker

About the author

Written by the Adamant Solutions engineering team: senior engineers working on APIs, identity and enterprise integration, certified on WSO2 and Spring. Code is tested before publication; if you spot an error, write to administrata@adamant-solutions.com.