How to Use Claude Code & Coding Agents with a Local LLM

"Can I run Claude Codeon a local LLM?" is one of the most-asked questions in self-hosted AI, and most answers online are either "no" or a config snippet that quietly doesn't work in practice. The honest answer is: technically yes, but there are two real catches— and once you understand them, you'll probably pick a different tool for local coding and keep Claude Code for the cloud work it's great at. This page walks the real arc, from someone who actually tried it.

Catch #1 — Claude Code speaks the Anthropic API, not OpenAI

Almost every local inference server (llama.cpp, Ollama, LM Studio, vLLM) exposes an OpenAI-compatible endpoint — /v1/chat/completions. Claude Code does not speak that. It talks to whatever ANTHROPIC_BASE_URL points at using Anthropic's Messages API (/v1/messages), a different request and response shape. So you can't simply point Claude Code at localhost:11434and have it work — the formats don't match.

The workaround is a translation proxy: a small service that accepts Anthropic-format requests and re-emits them as OpenAI calls to your local model. claude-code-router and LiteLLM both do this. You set ANTHROPIC_BASE_URL to the proxy, and the proxy forwards to your local server or a gateway upstream.

# Claude Code talks to whatever ANTHROPIC_BASE_URL points at,
# using the Anthropic Messages API format — NOT OpenAI's.
export ANTHROPIC_BASE_URL="http://localhost:8787"   # a translation proxy
export ANTHROPIC_AUTH_TOKEN="anything"              # the proxy checks/forwards this

# The proxy (e.g. claude-code-router or LiteLLM) converts Anthropic
# /v1/messages requests into OpenAI /v1/chat/completions calls and
# forwards them to your local model or gateway upstream.

Catch #2 — the context-window trap (the real gotcha)

This is the one that bites people after they've solved the API format. Claude Code is designed for frontier cloud models with very large context windows, so it ships a large system prompt plus extensive tool definitions and sends them on every request. On a cloud model with a huge window, that overhead is negligible. On a local model, it can be most of your budget — before you've added a single line of your own code.

It gets worse because of defaults. Many local runtimes ship a small default context window — Ollama's default num_ctxis modest, and LM Studio loads at a conservative context length unless you raise it. Combine a small default window with Claude Code's heavy built-in prompt and the model can be effectively full on arrival, with no room to actually read your files or hold a conversation.

# The trap: Claude Code's own system prompt + tool defs are large.
# On a runtime that defaults to a small context window, they can fill it.

# Ollama defaults num_ctx to a small value — raise it explicitly:
#   (in a Modelfile)  PARAMETER num_ctx 32768
# LM Studio: set the context length slider up (e.g. 32768) before loading.
# Either way, more context = more VRAM for the KV cache.

To make Claude Code usable locally you therefore need both a large-context local model (32k at the very least, ideally 128k) and to explicitly configure the runtime to allocate that context (Ollama num_ctx, LM Studio context length). That extra context is real KV-cache memory on top of the weights — so this path realistically wants a 24GB-class card running a large-context model, just to give Claude Code's own overhead somewhere to live.

The honest recommendation: right tool for each job

After fighting both catches, the pragmatic conclusion is simple: don't force Claude Code onto a local model. Keep Claude Code for cloud / Anthropic work, where its big context budget is free and it genuinely shines. For local coding on your own GPU, switch to a lighter-weight terminal agent whose system prompt doesn't blow the context budget — that one change makes a small-context local model feel usable instead of suffocated.

The specific recommendation is oh-my-pi (the binary is omp) by can1357. It's an open-source terminal coding agent that loads custom providers from ~/.omp/agent/models.yml, so you declare a Wide Area Intelligence provider and code against a model on hardware you own — zero per-token cost, code never leaving your machines.

# The lighter local path: oh-my-pi (omp) with a WideAreaAI provider.
bun install -g @oh-my-pi/pi-coding-agent

# ~/.omp/agent/models.yml — point omp at your own GPU via the gateway
providers:
  wideareaai:
    baseUrl: https://wideareaai.com/api/v1
    apiKey: WIDEAREAAI_API_KEY        # resolved from the agent .env
    api: openai-completions
    models:
      - id: Qwen3-Coder-30B-A3B-Instruct
        name: WideAreaAI Qwen3 Coder
        contextWindow: 32768
        cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }

Full step-by-step: run the oh-my-pi coding agent on your own GPU with Wide Area Intelligence.

Where the gateway fits — and fixes the failure case

Whichever OpenAI-compatible agent you choose, a gateway is what turns "a model on one machine" into "a model my agent can rely on." Wide Area Intelligence serves your chosen model from a GPU node behind one OpenAI-compatible endpoint (https://wideareaai.com/api/v1) with revocable keys and multi-node load balancing — reachable from any machine, no port forwarding.

It also directly answers the context-window trap's worst case: capability-aware cloud failover.When a small-context local model genuinely can't handle a task, the gateway fails the request over to a cloud model on prepaid credits instead of hard-failing — so your agent keeps working and you only pay for the requests that truly needed the cloud. That's the "right tool for each job" principle enforced at the routing layer. How capability-aware failover works →

Not sure your GPU can serve a large-context coder model in the first place? Check with can I run it? and the context window calculator, then bring your GPU online with Wide Area Intelligence →It's free for up to two nodes.