Running an Open-Source LLM on My Own GPU

I run a small open-source language model on my laptop's 4 GB GPU, and it has one job: checking the answers of my Spring Boot app. Each answer goes in next to the answer I expected, and the model returns PASS or FAIL with a short reason.

Why a model and not a normal test? My app reads a software incident and suggests its root cause. A normal test compares exact values, and that works when code gives the same output every time. A language model can give the right root cause in completely different words, and an exact comparison fails it. Checking meaning takes something that can read, so the evaluator is another language model.

Why on my laptop and not Claude or GPT? For two reasons. Every answer my app writes stays on my machine. And a local model is a fixed file, pinned by a SHA-256 hash, so when a score changes, I know my app changed and the evaluator did not. The cost is size: a 4-billion-parameter model is far weaker than Claude or GPT.

How does it work? Ollama runs the model inside Docker, Spring AI calls it over HTTP, and Qwen3 4B does the checking. Building it showed me two things I did not expect. Only 80% of the model fit on the GPU, and one verdict took 9.8 seconds. This article explains both, with every number measured on my laptop.

Here is the whole set-up, first as an idea and then as the machinery:

Tools and tech stack

Everything in this article ran on one laptop:

Setting up Ollama

Ollama is the server that runs the model. It loads a model file, keeps it in memory, and answers HTTP requests on port 11434. If you come from Java, think of it as Tomcat for models.

My laptop needed three things: an NVIDIA GPU with a current driver, WSL2, and Docker Desktop. I installed nothing else on Windows, because Ollama runs in a container that brings its own CUDA libraries.

Why a container and not the Windows installer? For the same reason I ship Spring Boot services in containers. The whole server becomes one file I can read, review, and later run unchanged on a Linux box. This is the Ollama part of my docker-compose.yml:

Three parts of this file matter. The deploy block hands the GPU to the container. Without it, the container cannot see the card, and Ollama quietly falls back to the CPU. The two environment variables shrink the memory the model needs while it runs, and I come back to them in the memory math below. The volume keeps downloaded models when the container is rebuilt, so I never download 2.5 GB twice.

One docker compose up -d later, the server was up and healthy:

Before loading anything, I took one more reading. Every measurement in this article starts from this line, the GPU with nothing on it:

Choosing the model

On ollama.com, I filter by Local first. The cloud models in the same list run on Ollama's servers, and that defeats the whole point. Then I read the size tag, which is the parameter count. At 4-bit storage, each billion parameters costs about 0.6 GB, so a 4B model is about 2.5 GB.

My laptop had two Qwen3 chat models of the same size, with nearly the same name:

They behave nothing alike. qwen3:4b is a thinking model. Before it answers, it writes out its reasoning, and every word of that reasoning costs the same time as the answer. I asked it a one-line question in Open WebUI, and it thought for two minutes before it replied.

At the 63 ms per token I measured later, two minutes is about 1,900 tokens written and thrown away. The amount also changes from question to question. An evaluator runs thousands of times, so I cannot afford that. I picked qwen3:4b-instruct-2507, which answers directly:

Two lines in that output decide most of what follows.

Q4_K_M is how the weights are stored. Each of the 4 billion weights takes about 5 bits instead of 16. At 16 bits, the weights alone would be 8 GB, twice my card. At 5 bits, they are 2.5 GB. It is the same kind of trade as keeping a number in a float instead of a double: a little less precision for much less memory.

temperature 0.7 is the model's default randomness. With it, the same test case can get a different verdict on the next run. That is fine for chat and wrong for an evaluator, so I override it in Spring Boot.

On disk, the model is a manifest of five files, 2,497,293,803 bytes in total, and each file is named by its SHA-256 hash. For a Java developer, this is a dependency with a fixed version and a checksum. Nothing can swap it underneath you, and that is what makes the evaluator stable.

Loading it onto the GPU

A running Ollama server holds no model. It loads one the first time a request names it. So I sent an empty request from Postman, only to load it:

Then I asked Ollama where the model went:

This was the screen that surprised me. size is everything the model needs while it runs: 2.94 GB. size_vram is the part that made it onto the GPU: 2.36 GB. That is only 80.4%. The other 575 MB sits in ordinary RAM and runs on the CPU.

Ollama's load log said the same thing in layers. The model has 37 layers. Twenty-nine went to the GPU, and eight stayed on the CPU. The driver agreed:

The real arithmetic

My first question was simple. The file is 2.5 GB, so why does the loaded model need 2.94 GB? Because running a model needs more than its weights:

The KV cache works like any cache in a Java service: compute a value once, keep it, and never compute it again. For every token the model has read, each layer keeps a key and a value, so the next token does not redo that work.

This is where the two Compose settings pay off. At 16 bits, the cache would be 604 MB. OLLAMA_KV_CACHE_TYPE: q8_0 stores it in 8 bits, for 321 MB. Flash attention does the attention math in small tiles instead of one giant matrix, and Ollama needs it switched on before it will store the cache in 8 bits at all.

My second question: 2.94 GB on a 4 GB card should fit, so why didn't it? Because the card never offered 4 GB:

Ollama's load log reported only 3.2 GiB free before the model arrived. nvidia-smi showed nothing using the missing 0.8 GiB. On Windows, the display driver keeps video memory for the desktop, and that is the likely holder.

The same log shows Ollama keeping 457 MiB back as a safety reserve. Then it placed the model one whole layer at a time, using its own estimate of each layer's size. That estimate ran out with 501 MiB still unused.

I built this calculator from those measurements, so you can try other models, context sizes and cards. The defaults reproduce my laptop:

Two things came out of the arithmetic. Halving the context frees only 160 MB, which is not enough to pull the eight layers back onto the GPU. And by the same numbers, a 16 GB card holds the whole evaluator, even at higher precision, with room to spare.

The rule I took away works on any machine. If size_vram is smaller than size, part of your model runs on the CPU.

Calling it from Spring Boot

This part felt like home. If you have used Spring Data, you know the pattern: no client code, only configuration. Spring AI's Ollama starter works the same way. My code makes one method call, and the starter turns it into an HTTP request to Ollama. All the settings live in one file, which application.yml imports:

I read this file top to bottom as the path of a request. base-url is where the call goes, Ollama's port on my laptop. pull-model-strategy: when_missing means a fresh machine downloads the models on first start, and nobody has to type ollama pull. The timeout gives that first download ten minutes instead of the default five. chat.model names the evaluator, and temperature: 0.1 replaces the model's own 0.7, so a verdict does not wander between runs.

num-ctx: 4096 is the line most people miss. Ollama's documentation says that if you leave it out, Ollama picks a context size based on the GPU's memory. A request that asks for a different size than the loaded model then forces a reload. Pinning it keeps the model identical on every machine.

One request, measured

To see exactly what goes over the wire, and to time the model with nothing else in the way, I sent the same kind of request straight to Ollama from Postman. It is plain JSON. The test case asks why customers were charged twice. The evaluator gets the expected root cause and my app's root cause, and must decide whether they describe the same cause:

POST http://localhost:11434/api/chat
{
  "model": "qwen3:4b-instruct-2507-q4_K_M",
  "stream": false,
  "keep_alive": "30m",
  "options": { "temperature": 0, "num_ctx": 4096 },
  "format": {
    "type": "object",
    "properties": {
      "reason":  { "type": "string" },
      "verdict": { "type": "string", "enum": ["PASS", "FAIL"] }
    },
    "required": ["reason", "verdict"]
  },
  "messages": [
    { "role": "system", "content": "You are an evaluator. Compare the candidate root cause with the reference root cause. Answer PASS only if both describe the same underlying cause. Reply in JSON." },
    { "role": "user", "content": "Incident: Customers were charged twice for one order.\nReference root cause: The payment client retried after a timeout without an idempotency key, so the provider processed the charge twice.\nCandidate root cause: A network timeout made the service resend the payment request, and the provider had no way to detect the duplicate." }
  ]
}

Ollama answered with the verdict and its own stopwatch, in nanoseconds:

The verdict is right. Both answers blame a retry that nothing could recognize as a duplicate.

The timing is the real lesson, so look at the two speeds: 60 tokens a second for reading, 16 for writing. Reading the prompt is like a batch insert. All 109 tokens go through each layer together, as one big matrix multiplication, and that is exactly what a GPU is built for.

Writing is like inserting rows one at a time, where each row needs the one before it. Token 2 cannot start until token 1 exists, and every token is a full trip through all 37 layers. So the 125-token verdict took more than four times as long as reading the 109-token prompt.

For a local evaluator, the length of the verdict is the bill. Drag the slider to change it:

Sent back to back at 9.82 seconds a verdict, one evaluator on my laptop tops out at about 8,800 verdicts a day. If I ask for a one-sentence reason of about 20 tokens instead, the same measured speeds give about 3.2 seconds a verdict, or about 27,000 a day.

What makes it an evaluator

A chat model becomes an evaluator through three settings in that request.

The prompt asks one question only: do these two describe the same cause? It answers PASS only if they do.

The format field carries a JSON schema, and Ollama only lets the model produce output that matches it. My Java code reads a field instead of guessing at prose. I put reason before verdict on purpose. A model writes left to right, so with the reason first, it decides only after it has argued.

Temperature 0 takes the randomness out of word choice, so the same case should get the same verdict on the same machine. Confirming it is simple: send the case twice and compare. The code that reads the result checks the verdict and keeps the reason, so every FAIL explains itself.

One more limit showed up while I was testing. The context size is reserved when the model loads, so it works like a wall. I found this through Open WebUI, a chat front end pointed at the same server. I asked a ten-word question, and it was rejected. By the time it reached Ollama, Open WebUI had added more than 6,000 tokens of its own:

Ollama refused instead of silently cutting off the start of the prompt. That is exactly what I want from an evaluator, because it never evaluates half an answer. The evaluator prompt, the expected answer and my app's answer must all fit, with room left for the verdict.

Why it is slow

The first request moves the model

Every Java developer knows the cold start: the first request after a deploy is slow while the JVM warms up. Ollama has its own version. A first request must copy 2.9 GB from the model files into GPU memory and RAM, set up the KV cache, and only then start reading the prompt.

Ollama reports that cost in load_duration. In my warm Postman request above, it was 2,547,133 nanoseconds, just 2.5 milliseconds, because the model was already in place.

The keep-alive decides whether a request finds the model cold or warm. By default, Ollama unloads a model after five idle minutes, so the first test case after a coffee break pays the full load and looks mysteriously slow. I use two fixes. I send the empty load request before a batch of cases, and I raise the keep-alive with OLLAMA_KEEP_ALIVE on the container or keep_alive on the request. My Postman call sets it to 30 minutes.

Every token costs 63 milliseconds

To write one token, the model multiplies its current state by every one of its weights. So for each token it reads all 2.5 GB of weights from memory, and does only two arithmetic operations with each weight. The math is cheap, and the reading is expensive.

That makes memory bandwidth the speed limit: how many gigabytes per second the chip can pull in. My laptop has two kinds of memory in the path. The bandwidths below are the hardware's rated peaks, and the weights per part are estimates from the model's layer sizes:

A fifth of the model sits in memory three times slower than the GPU's, and even in the best case it takes 42% of every token's time. The measured 62.9 ms is more than double that best case. Every token also crosses the PCIe bus between the CPU and the GPU, and the CPU side runs on six laptop cores.

On a 16 GB NVIDIA T4, rated at 320 GB/s with the whole model in GPU memory, the same arithmetic gives an upper bound of about 128 tokens per second.

What I learned, and what comes next

The lesson I keep from this is a habit I already had as a backend engineer: measure before you trust. The file size said the model would fit, and /api/ps said it did not. The speed I expected came from the GPU's specification sheet. The speed I got came from the slowest memory in the path.

The same habit decides what comes next. A 4B evaluator is only worth trusting once it has been measured against a person. My next step is to grade a hundred cases by hand, run the evaluator on the same cases, and report how often the two agree. I will put Cohen's kappa next to the raw agreement, because an evaluator that always says PASS looks accurate when most answers pass.

Right now, my app and the evaluator also run the same model. Once a second model family fits, it should do the evaluating, so the evaluator is not checking its own kind of writing. The laptop is where I build and calibrate the evaluator. A 16 GB GPU, where the arithmetic says the whole model fits, is where it should run every day.

Where every number comes from

None of the numbers in this article come from a specification sheet. Each one came from a command or a screen on my laptop, so you can get the same reading on yours:

Everything else is arithmetic on these readings: the KV cache size, the best-case time per token, the slider's projections, and the 16 GB estimate.

Sources

1. Ollama API reference: the endpoints used here (/api/chat, /api/generate, /api/tags, /api/ps), the JSON-schema format field, and what each timing field measures.
2. Ollama FAQ: the five-minute default keep-alive, and why the 8-bit KV cache needs flash attention.
3. Ollama: context length: how Ollama picks a default context from the GPU's memory when the client does not set one.
4. Qwen3-4B-Instruct-2507 model card: the architecture behind the KV cache arithmetic, 36 layers and 8 key-value heads.
5. Spring AI: Ollama chat: the base-url, model, temperature, num-ctx and pull-model-strategy settings in ollama.yml.
6. Docker Desktop: GPU support: how a container reaches an NVIDIA GPU through WSL2, which the deploy block relies on.