Ollama in Docker on a server without a GPU: what to expect and how to measure it
Run Ollama in Docker on a CPU-only server, keep the model loaded, limit memory and queue, and measure load time and tokens per second from the API response.

A small local model is often enough for classification, extraction or an internal assistant, and a virtual server without a GPU is the cheapest place to run it. Ollama publishes an official Docker image, so the setup takes a few minutes. What takes longer is learning how the server behaves under load: the first request is slow, an idle model is unloaded after a while, and a second concurrent user can double the memory use.
This guide gives a pinned Docker Compose file for a CPU-only host, explains the settings that matter, and provides a small benchmark client that measures your server using the numbers Ollama itself returns. It does not contain benchmark results. Speed on a CPU depends on the processor, the memory bandwidth, the model and its quantization, and any figure copied from another machine would be a guess. Measure on your own hardware.
Versions checked on 3 November 2026: Ollama 0.40.14 (the ollama/ollama:0.40.14 tag on Docker Hub, pushed that day). The benchmark client uses only the Python standard library.
Start Ollama without a GPU
The official Docker documentation gives this command for CPU-only use:
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
The named volume ollama holds the downloaded models in /root/.ollama; without it, every new container would download them again. Port 11434 is the API. For a server that stays up I prefer Compose, with the image pinned to a version instead of latest, so that an update is a decision and not a surprise:
services:
ollama:
image: ollama/ollama:0.40.14
ports:
- "127.0.0.1:11434:11434" # reachable only from this machine
volumes:
- ollama:/root/.ollama # models survive container recreation
environment:
OLLAMA_KEEP_ALIVE: "30m" # default is 5m: keep the model in RAM longer
OLLAMA_NUM_PARALLEL: "1" # default 1: parallel requests multiply the context memory
OLLAMA_MAX_QUEUE: "16" # default 512: reject early instead of queueing for minutes
mem_limit: 8g # model + context must fit, or the container is OOM-killed
restart: unless-stopped
volumes:
ollama:
Then download a model and call it:
docker compose up -d
docker compose exec ollama ollama pull llama3.2:1b
curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.2:1b","prompt":"Hello","stream":false}'
The model name here is only an example of a small model. Which model fits your task and your RAM is your choice, and the model library lists sizes. The pull endpoint of the API resumes cancelled downloads, so a dropped SSH session does not mean starting from zero.
The settings that matter
The FAQ documents the environment variables. These are the ones I set in the file above, with the defaults as the FAQ states them on the day of writing:
| Variable | Default | Why change it on a small server |
|---|---|---|
OLLAMA_KEEP_ALIVE |
5 minutes | After this idle time the model is unloaded from memory, and the next request pays the load time again. For a service used a few times an hour, 5 minutes means frequent cold starts. |
OLLAMA_NUM_PARALLEL |
1 | Number of requests each model processes at the same time. Raising it needs more memory, and on a CPU it does not make the machine faster: requests share the same cores. |
OLLAMA_MAX_QUEUE |
512 | Requests queued while the server is busy before it starts rejecting. With a CPU generating a few tokens per second, a queue of 512 means minutes of waiting. A short queue turns overload into an error you can handle. |
OLLAMA_MAX_LOADED_MODELS |
3 for CPU inference | How many different models stay in memory. Each one occupies RAM; loading three models of 4 GB each needs about 12 GB before any context. |
OLLAMA_HOST controls the bind address of the server process; Ollama binds 127.0.0.1:11434 by default. In the Compose file the exposure is decided by the port mapping: 127.0.0.1:11434:11434 makes the API reachable only from the host itself. The Ollama API has no authentication. If other machines need it, put a reverse proxy with authentication in front of it, and do not publish the port to the internet. A container with -p 11434:11434 on a public server is reachable by anyone who finds it.
The mem_limit line is a Docker setting, not an Ollama one. If the model plus the context do not fit, the kernel kills the container; a limit makes that failure predictable and protects the other services on the same host. How much memory a model needs depends on its size, the quantization and the context length you request, so check with docker stats while the model is loaded and answering.
Cold start and warm start
The first request after a pull, or after the keep-alive time, has to read the model from disk into memory. That is the cold case. Later requests find the model loaded: the warm case. On a CPU server with a slow disk the difference can be large, and users notice it.
There are two ways to control it, both documented in the FAQ. You can preload a model by sending an empty request, and you can set keep_alive per request: 0 unloads the model right after the answer, a negative number such as -1 keeps it loaded. A simple deployment pattern is to preload the model when the container starts:
curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.2:1b","keep_alive":"30m"}'
The benchmark client has the same call as a function, so a health check or a start script can use it.
Read the numbers Ollama gives you
The final response of /api/generate contains durations and token counts, as described in the API reference. All durations are in nanoseconds:
| Field | Meaning |
|---|---|
total_duration |
Total time of the request |
load_duration |
Time spent loading the model |
prompt_eval_count, prompt_eval_duration |
Tokens in the prompt and time spent evaluating the uncached ones |
eval_count, eval_duration |
Tokens generated and time spent generating them |
The documentation gives the formula for generation speed: eval_count divided by eval_duration, times 10^9, which is tokens per second. The same calculation on the prompt fields gives the prompt processing speed. On a CPU these two behave differently: reading a long prompt is usually much faster per token than generating, but a long prompt still costs seconds, and a RAG application that sends several thousand tokens of context pays for it on every request. Our guide on counting tokens shows how to know the prompt size before the call.
Here is the client. It uses stream: false to get a single response object with all the fields, and options.num_predict to bound the length of the answer so that the measurement stays short:
"""Measure an Ollama server: load time, prompt speed, generation speed. Standard library only."""
import json
import sys
import urllib.request
NS = 1_000_000_000
def post(base_url: str, path: str, body: dict, timeout: float = 600) -> dict:
req = urllib.request.Request(
base_url + path, data=json.dumps(body).encode(),
headers={"content-type": "application/json"}, method="POST")
with urllib.request.urlopen(req, timeout=timeout) as res:
return json.load(res)
def preload(base_url: str, model: str, keep_alive: str = "30m") -> dict:
"""An empty request loads the model into memory without generating anything."""
return post(base_url, "/api/generate", {"model": model, "keep_alive": keep_alive})
def generate(base_url: str, model: str, prompt: str, num_predict: int = 64) -> dict:
return post(base_url, "/api/generate", {
"model": model, "prompt": prompt, "stream": False,
"options": {"num_predict": num_predict}})
def summarize(r: dict) -> dict:
"""Turn the final response fields (all durations in nanoseconds) into readable numbers."""
def rate(count_key: str, dur_key: str) -> float | None:
count, dur = r.get(count_key, 0), r.get(dur_key, 0)
return round(count / dur * NS, 2) if dur else None
return {
"load_s": round(r.get("load_duration", 0) / NS, 2),
"total_s": round(r.get("total_duration", 0) / NS, 2),
"prompt_tokens": r.get("prompt_eval_count", 0),
"prompt_tokens_per_s": rate("prompt_eval_count", "prompt_eval_duration"),
"output_tokens": r.get("eval_count", 0),
"output_tokens_per_s": rate("eval_count", "eval_duration"),
}
def main(base_url: str, model: str) -> None:
prompt = "Explain in three sentences what a reverse proxy does."
cold = summarize(generate(base_url, model, prompt)) # may include the model load
warm = summarize(generate(base_url, model, prompt)) # model already in memory
print(json.dumps({"cold": cold, "warm": warm}, indent=2))
if __name__ == "__main__":
main(sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:11434",
sys.argv[2] if len(sys.argv) > 2 else "llama3.2:1b")
Run it against your server and read the two blocks:
python bench.py http://127.0.0.1:11434 llama3.2:1b
The output has a cold and a warm entry with load_s, prompt_tokens_per_s and output_tokens_per_s. If the first call finds the model already in memory, the two entries will look alike: to see a real cold start, restart the container or wait for the keep-alive time first. Repeat the measurement a few times, at different times of day if the host is shared, because virtual servers can have neighbours that use the same physical cores.
What the numbers mean for your application
- Output speed is the user experience. A person reads at a few words per second. If your measured generation speed is lower than that, stream the answer to the browser instead of waiting for the whole text; the guide on streaming with SSE shows how.
- Total time is prompt time plus generation time plus (maybe) load time. Estimate it as
prompt_tokens / prompt speed + max output tokens / output speed, and set your client timeout well above that. - Concurrency is a capacity question. With
OLLAMA_NUM_PARALLEL=1, a second user waits for the first. If your peak is more than one request at a time, work out how many requests per second the server really sustains and compare it with the expected traffic; our capacity sizing tool does that arithmetic. Past a certain point the answer is a smaller model, a shorter prompt, a GPU or a hosted API, not a larger queue. - Overload should be an error. With a short
OLLAMA_MAX_QUEUE, a rejected request can be retried later with backoff; see HTTP 429 and Retry-After for the pattern. I did not verify which HTTP status Ollama returns when the queue is full, so check it on your version and map it in your client; our HTTP errors reference helps to read what you get.
Test it without a model
The example folder has a fake Ollama server written with the standard library. It answers /api/generate with fixed numbers: a slow first call that includes a load time, and faster ones afterwards. The tests check the request your client sends (stream: false, num_predict, an empty body for preload), the unit conversion from nanoseconds, the division by zero case, and that the cold run is slower than the warm one:
test_cold_slower_than_warm ... ok
test_generate_request_shape ... ok
test_preload_sends_no_prompt ... ok
test_summarize_units ... ok
test_zero_duration_gives_none ... ok
The suite runs in GitHub Actions on every change.
Limits of this guide
- I could not run Docker or a real model in the environment where this article was prepared. The CI tests prove that the client sends the right requests and computes the right numbers from the documented fields; they do not prove the Compose file. I wrote it from the official documentation, so test
docker compose upon your server before relying on it. - There are no speed figures in this article on purpose. They would depend on a machine you do not have.
- The Ollama API has no built-in authentication, and a model answering from a CPU is easy to overload. Both are reasons to keep the port closed and to put limits in front of it.
- Defaults change between versions. The values above are those documented on 3 November 2026 for 0.40.x.
The complete, tested example for this article is in our public repository: adamant-solutions/ai-dev-examples/ollama-docker