• Local LLMs
  • Claude
  • Spring AI

Ollama on a 4 GB GPU vs Claude: when a local model stops being enough

I ran my RCA assistant with Ollama on a 4 GB laptop GPU, measured where it ran out, and moved its reasoning to Claude while keeping embeddings local.

Vishnu VardhanProject: KanbanAssist

KanbanAssist is my open-source root-cause-analysis (RCA) assistant for Azure DevOps tickets. A webhook brings in each new ticket, pgvector finds the five most similar past tickets, and a model writes the likely cause and fix. That model was qwen3:4b, running in Ollama on my laptop's 4 GB GPU. When I started building toward an autonomous agent, where the model has to reason across several steps, the local model stopped being a learning tool and became the bottleneck.

1. My laptop and Claude, side by side

This table is the short version of the story. Rows marked measured come from my own laptop; the sections after it explain the numbers that decided the switch.

Table 1. My laptop and Claude, side by side. Measured values come from my earlier article; the rest is derived or from Anthropic's documentation.
MetricWhat it meansqwen3:4b in Ollama, my laptopClaude Sonnet 5.5
Hardware
Where it runsThe machine doing the workDell laptop, NVIDIA Quadro T1000, 4 GBAnthropic's data centers
Model sizeHow many learned numbers (weights) it has4 billion weights, 2.44 GB after 4‑bit compressionNot published
Memory
GPU memory freeWhat the card offers after Windows takes its share3.2 GiB of 4 GB (measured)Not my concern
Where the model sitsLayers on the fast GPU versus slow system RAM29 layers on GPU, 8 on CPU: 80% fit (measured)All on Anthropic's hardware
Memory per tokenKV cache: what the model keeps about each earlier token147,456 bytesHandled by Anthropic
Tokens and context
Context windowMost tokens in one request, prompt plus answer262,144 trained; about 6,800 fit on my card1,000,000
Longest answerMost tokens one reply can containLimited by the same memory128,000
Prompt too longWhat happens past the limitOllama quietly cuts the promptThe API returns an error
Speed
Reading the promptTokens processed before the answer starts60.4 tokens/s: 109 tokens in 1.80 s (measured)Not published
Writing the answerTokens generated per second15.9 tokens/s: 125 tokens in 7.86 s (measured)About 42 tokens/s (third‑party)
One short answerTotal time for one request9.82 s (measured)Fast on Investigate; Langfuse timing next
Parallel requestsCan it answer several at once?One at a time; a second cache does not fitMany, within rate limits
Cost
PriceWhat each token costs$0; I pay for the GPU and power$2 per million input, $10 per million output
One RCA call8,000 tokens in, 800 out$0$0.024
DiscountsWays to pay lessNot neededCached input $0.10 per million; Batch API half price
Limits and quality
Rate limitsCaps on how much you can sendNone; the GPU is the capPer minute: 1,000 requests, 2M input, 400K output tokens
Reasoning qualityHow well it reasons over the retrieved ticketsBecame my bottleneck while building the agentFast, usable answers on Investigate; formal evaluation next
Data and control
Data leaves my machineWhere ticket text goesNoYes, to Anthropic
Works offlineRuns without internetYesNo
Model changesCan the model change under you?Never; the file is pinnedPinned IDs; old models are retired
Fine‑tuningTraining it further on your own dataAllowed: open weights, Apache 2.0No self‑serve option
EmbeddingsTurning text into vectors for searchnomic‑embed‑text, localNot offered; use another provider

The measured column tells the real story. My card held only 80 percent of the model, so it wrote 15.9 tokens per second and took 9.82 seconds for a 125-token answer. Claude's constraints are money and rate limits, not memory, and at $0.024 per call the money is small.

2. The moment it stopped working

When an RCA answer from the local model was wrong, it had three possible causes, and I had no way to tell them apart. An agent that loops an Analyst and a Critic over several steps would multiply that doubt at every step.

Three suspects for one wrong answer A wrong RCA answer could come from the prompt, the code or the small model, so every failure was ambiguous. Wrong RCA answer from the local model My prompt? My code? The 4B model's limits? No signal nothing to learn
Figure 1. A weak model turns every failure into a three-way guess. I was debugging the model instead of my design.

A strong model removes the third suspect, so a failure points at my prompt or my code, which are the things I can fix. To see why the small model was weak and slow on my machine, I worked the numbers from the hardware up.

3. Why a 4 GB GPU stalls

To write one token, the GPU must read every weight of the model from its memory. My model has 4.02 billion weights, compressed to about 4.85 bits each, so every token moves 4.02 B × 4.85 ÷ 8 = 2.44 GB. A laptop card with a 128-bit bus at 12 Gbps moves 16 bytes × 12 billion = 192 GB/s.

Time budget for one token Reading 2.44 GB of weights takes 12.7 ms; the arithmetic takes 0.8 ms. One token on a 192 GB/s GPU Read 2.44 GB of weights 12.7 ms Arithmetic, 8 billion ops 0.8 ms Same time scale on both bars. Ceiling: 192 ÷ 2.44 = 79 tokens per second.
Figure 2. Moving data takes 16 times longer than computing with it. Generation is limited by memory bandwidth, not by the GPU's cores.

The second cost is the KV cache, the model's memory of earlier tokens. Every token keeps a Key and a Value vector in each of 36 layers: 36 × 2 × (8 heads × 128 numbers) × 2 bytes = 147,456 bytes per token. My RCA prompt is about 8,000 tokens, so a context of 8,192 tokens needs 8,192 × 147,456 bytes = 1.21 GB.

GPU memory budget Weights 2.44 GB, KV cache 1.21 GB, runtime 0.50 GB and display 0.35 GB total 4.50 GB, beyond the 4.29 GB card. What must fit on a 4 GiB card (4.29 GB) Card limit 4.29 GB Weights 2.44 GB KV cache, 8,192 tokens 1.21 GB Runtime 0.50 GB Windows display 0.35 GB Total 4.50 GB: 0.21 GB more than the card holds
Figure 3. The model plus one RCA prompt does not fit. The memory left for the KV cache is 4.29 − 2.44 − 0.50 − 0.35 = 1.00 GB, which holds 6,780 tokens.

When memory runs out, Ollama moves some layers to system RAM, where the CPU runs them. Laptop RAM moves 51.2 GB/s, a quarter of the GPU's speed, and my earlier article measured exactly this: only 80 percent of the model fit on the GPU.

Cost of spilling layers to the CPU All layers on the GPU take 12.7 ms per token; with 8 layers on the CPU it takes 9.9 plus 10.6, 20.5 ms. Time per token GPU only 12.7 ms 8 layers on CPU 28 layers, GPU 9.9 ms 8 layers, CPU 10.6 ms 20.5 ms 22 percent of the layers on slow memory add 61 percent to every token.
Figure 4. The slowest memory sets the pace for the whole model.

Context tells the same story. The model was trained for 262,144 tokens, but my card can hold the cache for 6,780 of them. Claude Sonnet 5.5 accepts 1,000,000, and its cache lives on Anthropic's hardware, not mine.

Context window, to scale Usable on the 4 GB GPU 6,780 tokens; qwen3-4B trained for 262,144; Claude Sonnet 5.5 accepts 1,000,000. Tokens one request can hold, drawn to scale Usable on my 4 GB GPU 6,780 qwen3-4B trained for 262,144 Claude Sonnet 5.5 1,000,000
Figure 5. The model's limit is not the limit you get. On a small card, memory decides how much a prompt can hold.

4. Time to a correct answer

Claude writes about 42 tokens per second in third‑party measurements, close to my laptop's 40. Speed per token is the wrong metric. What matters is how long it takes to get an answer you can use, which adds the time to read the prompt and multiplies by the attempts.

Time to a usable RCA answer Local: three attempts of 12.9 s reading and 20 s writing, 99 s. Claude: one attempt of about 20 s. Attempt 1 Critic retry Critic retry qwen3:4b, local 99 s Claude about 20 s, one attempt Reading the 8,000-token prompt (12.9 s) Writing 800 tokens (20 s)
Figure 6. With two Critic retries, an illustrative count for a weak model, the local path takes five times longer. Claude's reading time is too short to draw at this scale.

Retries also compound. If each agent step succeeds 85 percent of the time, four steps succeed 0.85⁴ = 52 percent of the time; at 97 percent per step, 0.97⁴ = 88 percent.

5. What Claude costs, and where the limits bite

Claude Sonnet 5.5 costs $2 per million input tokens and $10 per million output tokens. Output costs five times more because input is read in one parallel pass, while every output token needs its own pass through the model. One RCA call sends 8,000 tokens and receives 800.

Cost of one RCA call Standard $0.0240; with 2,000 cached tokens $0.0202; Batch API plus caching $0.0101. One RCA call: 8,000 tokens in, 800 out Standard $0.0240 2,000 tokens cached $0.0202 Batch API + cached $0.0101 Input Output
Figure 7. Standard: 8,000 × $2 ÷ 1M + 800 × $10 ÷ 1M = $0.016 + $0.008. A cached prefix costs $0.10 per million, and the Batch API halves everything for work nobody waits on.

Once the Analyst and Critic loop is built, a ticket that takes three calls will cost 3 × $0.024 = $0.072. Money is not the hard limit, though. The hard limit is how much Anthropic lets one account send per minute.

What a rate limit is

Anthropic's servers are shared by every customer, so each account gets a fixed share per minute. On the starting tier, Claude Sonnet 5.5 allows three amounts per minute, and a request goes through only if it fits all three. Table 2 lists them, with the same number spread over a second.

Table 2. Start-tier limits for Claude Sonnet 5.5
What is countedPer minutePer second
Requests1,00016.7
Input tokens: everything I send2,000,00033,333
Output tokens: everything Claude writes400,0006,667

Anthropic enforces each amount like a bucket of tokens. The bucket refills a little every second instead of resetting once a minute, and every call takes tokens out. Figure 8 follows the input-token bucket through one cycle.

How the input-token bucket works The bucket refills 33,333 tokens every second up to 2,000,000; each call takes its input tokens; when not enough is left, the API answers HTTP 429. 1. Refill +33,333 tokens every second 2. The bucket holds at most 2,000,000 tokens 3. Each call takes its input one RCA call takes 8,000 tokens 4. Not enough left? HTTP 429: wait, then retry
Figure 8. A 429 answer means "too many requests". It usually carries a retry-after header with the seconds to wait, and Spring AI can retry the call automatically.

What it means for KanbanAssist

For live tickets the limit never matters. One ticket through a three-call loop sends 3 × 8,000 = 24,000 input tokens, which is 1.2 percent of a minute's 2,000,000. My planned evaluation is different, because it replays hundreds of closed tickets in one go. Figure 9 shows what happens if I send 300 of them at the same moment, and three ways around it.

A planned evaluation against the input limit 300 tickets times 8,000 tokens is 2,400,000, which is 400,000 over the 2,000,000 limit, so 50 calls are rejected. Fixes: pace calls, cache the fixed prefix, or use the Batch API. Planned evaluation: 300 tickets sent at once 300 × 8,000 = 2,400,000 tokens, 400,000 over the limit: 400,000 ÷ 8,000 = 50 calls rejected Send at refill pace 33,333 ÷ 8,000 = 4.17 calls/s 300 calls in 72 s Cache the fixed part cached tokens are not counted 300 × 6,000 = 1,800,000 Use the Batch API one job, own queue half price, done within 24 h
Figure 9. Caching works because the first 2,000 tokens of every RCA call, the instructions and tool definitions, are identical. Only the 6,000 tokens of tickets change from call to call.

My laptop has no rate limit, but it has a ceiling of its own: one GPU answers one request at a time, because a second KV cache does not fit in its memory. At my measured 9.82 seconds for one short answer, 300 tickets in a row would take 300 × 9.82 = 2,946 seconds, or 49 minutes, and full RCA prompts would take longer still.

6. Embeddings stay local

Anthropic has no embedding model, and it does not need one here. Retrieval and generation meet only at plain text, so each side can use a different provider.

Retrieval and generation are separate The embedding model and pgvector find similar tickets; only their text reaches Claude. Retrieval, local New ticket text nomic-embed-text Ollama, 768 numbers pgvector search top 5 similar tickets Generation, Claude Ticket text + 5 tickets plain text, no vectors Claude Sonnet 5.5 RCA answer
Figure 10. Claude never sees a vector. The chat model can change without touching a single stored embedding.

Vectors from different embedding models never mix. Each model learns its own coordinate system, so a 768-number nomic vector and a 1,024-number Voyage vector for the same sentence share no meaning, and switching models means re-embedding every stored ticket. Embedding models are also small, frequent and close in quality to paid APIs, which makes them the clearest case where a local model wins.

7. What the switch changed

The application code did not change. Everything happened in configuration.

Files touched by the switch pom.xml gained one starter, anthropic.yml is new, application.yml gained one import and two keys, .env gained four keys; application code is unchanged. pom.xml +1 starter anthropic.yml new file application.yml +1 import, 2 keys .env +4 keys Java code: unchanged
Figure 11. One dependency, one new file, one import line and four environment keys.

The two properties that matter sit in application.yml. Both read from .env with no default, so a missing key stops the application at startup and names the key, instead of running on a hidden fallback.

spring:
  ai:
    model:
      chat: ${CHAT_PROVIDER}             # anthropic
      embedding: ${EMBEDDING_PROVIDER}   # ollama

# config/anthropic.yml
spring:
  ai:
    anthropic:
      api-key: ${ANTHROPIC_API_KEY}
      chat:
        options:
          model: ${ANTHROPIC_CHAT_MODEL}  # claude-sonnet-5-5
          max-tokens: 2048

8. How Spring Boot picks the model

Nothing in my code names Ollama or Anthropic. Each starter JAR lists its auto-configuration classes in a file named AutoConfiguration.imports, and each class runs only when the property spring.ai.model.chat names its provider.

Auto-configuration loading flow Starter JARs list auto-configuration classes; the condition on spring.ai.model.chat creates AnthropicChatModel and skips OllamaChatModel; ChatClient.Builder wraps the one ChatModel and is injected into the controller. Starter JARs on the classpath ...AutoConfiguration.imports Spring checks each class's condition spring.ai.model.chat=anthropic AnthropicChatModel condition matches: created OllamaChatModel condition fails: skipped ChatClient.Builder wraps the one ChatModel Investigation code receives the builder; unchanged Dashed border: on the classpath but not created.
Figure 12. One property decides which bean exists. With CHAT_PROVIDER=ollama, the two middle boxes swap and nothing below them changes.

At run time, the same interface call becomes two different HTTP requests. Spring AI translates the prompt into each provider's format and converts both replies back into one ChatResponse object.

One call, two wire formats chatClient.call() goes through OllamaChatModel to localhost:11434/api/chat, or through AnthropicChatModel to api.anthropic.com/v1/messages; both return a ChatResponse. chatClient.prompt()...call() OllamaChatModel AnthropicChatModel POST localhost:11434/api/chat system prompt inside messages POST api.anthropic.com/v1/messages system field, max_tokens, API key ChatResponse → .content()
Figure 13. Two request shapes and two reply shapes, hidden behind one interface. The dashed path is the one switched off.

9. Where a local model is the right call

Companies do run models on their own hardware, usually self-hosted in their cloud account with a serving engine such as vLLM rather than Ollama on a laptop. Figure 14 is the decision I would apply to any single step of an AI system.

Where should this step run? Data must stay inside: self-hosted. Narrow, very high volume: small local model. Offline: on-device. Otherwise: frontier model first, then move passing steps down. Must the data stay in your network? Narrow task at very high volume? Must it work offline? Frontier model first prove the task, build an evaluation Self-hosted model vLLM or private endpoint Small local model embeddings, classification On-device model phone, car, factory floor Move passing steps down only where the eval holds yes yes yes no no no
Figure 14. KanbanAssist lands in two boxes: embeddings in "small local model", reasoning in "frontier model first".

Sending tickets to Claude also means their text leaves my machine, including any log line or secret pasted into a comment. Redacting secrets before the call, and reading the provider's data-retention terms, belongs in the first question of Figure 14.

What I learned

Running a model on a 4 GB GPU taught me more about how models run than any course, and it also showed me where that lesson ends. These are the points I would carry into any AI system:

The rule I ended up withRun a model locally when the data cannot leave your network, when the task is narrow and very high-volume, or when the system must work offline. Otherwise, prove the task on a frontier model first, and move steps to a smaller model only when an evaluation shows they still pass.

KanbanAssist now reasons with Claude Sonnet 5.5 and embeds locally with nomic-embed-text. My next step is an automated evaluation over closed tickets, so that every change to the agent, and any future move of a step back to a smaller model, is decided by a score instead of by how an answer feels.

Sources