What a gateway does

An API gateway is the single front door to your services. Requests arrive at the gateway; it decides what to do with them before anything reaches your code.

The common jobs:

• Routing. /orders goes to one service, /users to another.
• TLS termination. Certificates are handled in one place.
• Authentication. API keys, JWT validation or OAuth, checked once at the edge rather than in every service.
• Rate limiting and quotas. Protect your backend and enforce plan limits.
• Request and response transformation. Headers, versioning, protocol translation.
• Observability. One place to log, measure and trace every call.
• Developer portal and monetisation. Documentation, key issuance and billing for external API consumers. Where “gateway” turns into “API management”.

Most small teams need the first four. Very few need the last.

Where it sits

There are four broad places to put this function, in rising order of complexity:

1. A reverse proxy you already run. Caddy, Nginx or Traefik in front of your services. Routing, TLS and basic rate limiting come free; authentication needs a plugin or a few lines of config.
2. A self-hosted gateway. Kong Gateway (open source) or KrakenD Community Edition (Apache 2.0) on your own instances. More features, a plugin ecosystem, still yours to run.
3. A managed cloud gateway. AWS API Gateway, Google Cloud API Gateway, or an edge platform such as Cloudflare Workers. No servers, priced per request, tied to that provider.
4. A full API management platform. Apigee or Kong Konnect: policies, analytics, developer portals, monetisation, with pricing to match.

Complexity

Run cost

Request-related cost per month at list prices (checked 1 October 2026). Prices are shown in the provider’s own currency. This table excludes egress, logging, tax and free-tier credits; egress follows below.

Check them yourself: AWS API Gateway, Google Cloud API Gateway, Apigee, Kong Konnect, Cloudflare Workers, Scaleway Instances and Scaleway Load Balancer.

How to read it:

• Per-request pricing is cheap until it isn’t. AWS’s HTTP API is the bargain of the managed options; its REST API, which many tutorials still default to, costs 3.5 times as much per request.
• API management platforms have a floor. Apigee’s Base environment starts at $365 a month before a single call, and caps at 50 requests a second, about 130M a month. Kong Konnect Plus costs $25 a month for a serverless gateway, which includes 1M requests, then charges $200 for each additional million and stops at 10M a month.
• Self-hosting is flat. Two PLAY2-PICO instances at about €10.42 a month each and an LB-S load balancer at about €16.79 cost the same at 5M requests as at 300M. The instances sit on a free private network behind the load balancer, so they don’t need public IPv4 addresses; if you give them one, add about €2.92 a month each. That 300M is an average of about 115 requests a second, comfortably within what a lightweight proxy handles on modest hardware, though you should load-test your own traffic before relying on it.
• Edge workers are surprisingly cheap. Cloudflare’s Workers Paid plan includes 10M requests for $5 and charges $0.30 per million after that, and doesn’t charge for egress. CPU time is billed separately, so a gateway doing heavy work per request will cost more than this. You write the gateway logic yourself, so treat it as a programmable proxy rather than an off-the-shelf gateway.

Then add egress…

Egress is the charge for data leaving the provider, and it has always been the elephant in the room with AWS and the other hyperscalers. It rarely shows up on a pricing page headline, and it’s designed to scale with your success.

Here’s the same comparison with an assumption of 10 KB per average response, which is modest for a JSON API. That’s 50 GB a month at 5M requests and 3,000 GB at 300M.

AWS charges $0.09 per GB and Google Cloud $0.105 per GB within Europe on its premium tier. Cloudflare doesn’t charge for egress on Workers, and Scaleway’s list prices include it. The AWS figures ignore its free allowance of 100 GB a month, which covers the whole 5M column but barely dents the 300M one. On AWS’s HTTP API, egress almost doubles the bill at every volume. Larger responses, file downloads or a chatty mobile app make it worse.

The Scaleway setup has bandwidth limits rather than a meter: 100 Mbps per PLAY2-PICO instance and 200 Mbps on the LB-S load balancer. 300M requests at 10 KB averages about 9 Mbps, so there’s headroom, but plan for your peaks rather than your average.

And then the shopping basket…

The gateway is rarely the only line item. Turn on access logging and every request becomes a log entry, billed for ingestion and again for storage. Add metrics, tracing, a WAF, a cache, a custom domain, a private endpoint. Each one is a small item and each looks reasonable on its own, like dropping things into a shopping basket. The total at the till is what surprises people, and it grows with your traffic.

Before choosing, list every add-on you’ll actually switch on and price the basket, not just the headline item. Sample your logs, or keep only errors and slow requests.

Avoiding lock-in. A managed gateway’s configuration doesn’t move. If you want to be able to change provider in the future, keep routing and auth rules in code you can port, and don’t build business logic into gateway policies.

Running it yourself takes some ongoing attention: patching, certificates, upgrades, a second instance for availability. How much that’s worth depends on your team, so I haven’t put a number on it. With infrastructure in code and a managed load balancer in front, it’s less than people expect.

What I’d pick

• One product, a handful of services, your own frontend as the main client: a reverse proxy. You may never need more.
• Several services, external API consumers, real rate-limit and key management: a self-hosted gateway on cheap European instances, or AWS’s HTTP API if you’re already serverless on AWS.
• Very spiky or global traffic, small team: an edge platform.
• You sell API access to third parties and need portals and billing: now an API management platform earns its price tag, and not before.

Most people I’ve talked to are in the first two groups and are being sold the fourth.

If you’re not sure which group you’re in, it’s one of the questions a Cloud Health Check answers.

What I Built

Learn Codebase is a small tool I built to get up to speed on someone else’s code. You point it at a GitHub or GitLab repository, or a folder on your machine, and a language model reads the code. It then writes a set of flashcards and a multiple-choice test about how the project works. It runs as a simple Jekyll site, and talks to any OpenAI-compatible endpoint: LM Studio locally, or OpenRouter for hosted models. There is also a Claude Code version (a /learn-codebase skill) that runs on my Claude subscription.

Which raised the question: which model should I pick to do the work?

Why Test?

I had four questions:

• Local or cloud? I’ve written on my personal blog about developing with local LLMs and about cloud inference endpoints. This was a chance to put numbers on that trade-off for a real task.
• Is my go-to model good enough? GLM-5.3-Flash has been my everyday model, because it is fast, reasonably capable and very cheap. But it has lost coherence on me a couple of times, once deleting its own skill files, which had to be recovered from backup. Does it have challengers at its price?
• How big is the gap to the frontier, and what does closing it cost?
• What does an agent harness add? Claude Code runs Claude Opus 5.5. How much of its result comes from the model, and how much from Claude Code itself?

How I Tested

• Three codebases: Learn Codebase itself (JavaScript), Pallets’ itsdangerous (Python) and chi (Go), from 44 to 110 files.
• The same job every time: 20 flashcards and 10 multiple-choice questions with four options each.
• Blind judging against the code: headless Claude Code read each codebase and scored every item for correctness and insight. For questions, it also checked whether the answer key was right and whether the wrong options were plausible. It never knew which model wrote what, and every verdict had to cite a file and line.
• Coverage: before judging, Claude Code listed the 20 key concepts a newcomer needs for each codebase. Each set was scored on how many of them it taught correctly.
• One number: each result is a percentage of the score the Claude Code skill achieved on the same codebase.
• Repeats: the important comparisons were run up to nine times, because the judge alone can move a score by about 10 points.

Local models ran in LM Studio on my Apple M2 Max with 64GB RAM. Everything hosted went through OpenRouter, with my zero-data-retention guardrail switched on. The whole exercise cost about $25 on OpenRouter, plus a bit of my Claude Pro subscription time for the judging.

A caveat up front: the judge is Claude inside the Claude Code harness. The sets were anonymised and every verdict needed evidence, but if anything the method favours Claude. Treat differences under 10 points as noise.

The Results

What I Learned

1. The harness matters more than the model.

This was the most interesting result. Called through the API at the same effort, Claude Opus 5.5 scored 89% of what it achieves inside Claude Code. It was every bit as accurate and insightful; the whole gap was coverage. Claude Code’s sets taught 87% of the key concepts, the API’s about 70%.

My first thought was that the API version simply saw less of the code, so I sent it two to four times as much. It barely moved: 90%, at twice the cost. On the Go codebase it read all 108 files and scored lower. The API runs had the code in front of them. What they lacked was a way to tell which parts mattered. Claude Code explores like an engineer: it looks at the structure, picks what matters, follows references across files and checks its own draft. That is worth more than a bigger model or a bigger context window.

I may investigate harnesses further with smaller models to see just what you can get out of them, especially when combining with targetted post training.

2. Price buys reliability, not much more quality.

The curve is steep. Half a cent buys 80% (GLM-5.3-Flash), four cents buys 85% (GLM-5.3), and 55 cents buys 89% (Opus 5.5). What the frontier tier really bought was reliability: every frontier model got every answer key right, and missed fewer important ideas. The budget models were 93-97% accurate, but occasionally keyed the wrong answer, or left two defensible answers to a question.

3. Max effort is mostly overthinking.

I started the frontier models at the settings they’re rated at on the Artificial Analysis leaderboard: “max” or “xhigh” effort. That was a mistake. GPT-6 Astra at max cost $3.50 a set and took 20 minutes. At high effort it scored the same for $1.04 in under four minutes. Around 90% of its max-effort output was reasoning that never reached the answer. For a well-defined task with limited context, effort is a cost lever in its own right, and probably the next thing worth testing in the other direction: a frontier model at low effort.

4. Reading strategy is the biggest lever for smaller models.

The app has three reading modes. Quick sends what fits in one request. Thorough summarises the codebase in parts first. Progressive reads in rounds, writing as it goes. Moving from Quick to Thorough lifted almost every model, typically by 10 to 25 points, mostly through coverage. For hosted models, Thorough was the best mode; it is now the default.

5. Some models can’t stop thinking.

The app originally asked every model not to reason, which keeps local models fast. GLM-5.3-Flash ignored that, thought anyway, ran out of budget and returned nothing: 8 out of 9 sets failed. With reasoning on, it was faultless. I can’t prove it, but a model that keeps reasoning past its budget and then returns nothing could well explain the odd incoherent moment in longer agentic sessions.

6. A data policy is part of the model choice.

Muse Wonder never ran. Its only provider on OpenRouter is Meta, with no zero-data-retention endpoint, so my guardrail refused every request before anything was sent. IMHO that speaks for itself. Check where your code goes, not just what it costs.

7. Local models are accurate, but narrow and slow.

The best local result, gemma-4-31b-qat, reached about 70% but took 25 minutes a set. qwen3.6-35b-a3b managed about 55% in three or four minutes. A humbling footnote to my January post on local LLMs: qwen3-coder-30b, which I recommended there for autocomplete, scored only 33% here. And devstral-small-2 wouldn’t load at all with KV cache quantisation enabled in LM Studio, the very setting that post suggested switching on. The right model depends on the task, not the leaderboard.

8. Benchmarking your own tool finds bugs.

The test found four genuine defects in the app. Weaker models repeated their first batch word for word. Some replied in a JSON form the parser couldn’t read. There was no way to turn reasoning on. And there was no fallback for models that can’t turn it off. All are now fixed, and several models went from half a set to a full one.

My Take: What Each Tier Buys You

My Iron Triangle post on jimm.uk says you can pick two of fast, cheap and good. Language models bend that a little, but it still holds. Here is how I’d now choose for this kind of task.

Local (free, private, slow). You gain privacy and zero marginal cost; you pay in time and completeness.

• Use the biggest model your memory allows (gemma-4-31b-qat on 64GB), not a small or code-specialised one.
• Use Progressive reading, and give it at least 16k tokens of context.
• Let it run unattended. It’s the right choice when the code genuinely can’t leave the machine.

Budget hosted (GLM-5.3-Flash, Claude Haiku 5.5, MiMo-V2.6-Pro). You gain 80% of the quality for pocket change, in a couple of minutes. For personal learning, this is good enough most of the time.

• Keep reasoning on, and use Thorough reading.
• Glance at the answer keys before trusting them.
• If mistakes matter more than pennies, Haiku 5.5 and MiMo-V2.6-Pro made the fewest at this price. GLM-5.3-Flash stays my default, and nothing at its price beat it.

Mid-price (GLM-5.3). This is the sweet spot. Frontier-level results (it matched GPT-6 Astra) for 4 cents a set.

• If I’m handing a study set to someone else, this is where I’d start.

Frontier via the API (Claude Opus 5.5, Gemini 4 Argon). You gain reliability: every answer right, and the fewest gaps.

• Opus 5.5 was also the most consistent and the fastest.
• Use high effort, not max. Max is overthinking you pay for by the token.
• It’s worth it when the output will be relied on, say onboarding a team or assessing understanding. Otherwise it’s a lot of money for the last few points.

Agent (Claude Code). You gain completeness, because it reads a codebase the way an engineer would. Nothing else came close on coverage, and it is included in a Claude subscription.

• For a large or unfamiliar codebase, this is the one to use.
• Point it at the repository, let it run, and review the result.

In short: with models, you can now have fast, cheap and good enough for most jobs. What you pay more for is reliability and completeness, and the cheapest way to get completeness is a better harness, not a bigger model.

What’s Next

• Frontier models at low or medium effort. If Opus 5.5 at low effort keeps its accuracy, it could give near-frontier quality at close to budget prices.
• Agent-like reading in the app. Let the model ask for specific files, or search for symbols, before it writes. That’s the gap the harness comparison exposed.
• A long-running agentic test of GLM-5.3-Flash against its challengers, to see whether the coherence problems show up under controlled conditions.

References

• Artificial Analysis - model intelligence and cost leaderboard
• OpenRouter - hosted models, pricing and data policies
• LM Studio - local inference
• Claude Code - the agent harness
• itsdangerous and chi - two of the test codebases

Summary

I tested local, budget, mid-price and frontier models, plus Claude Code, on writing study material about three codebases, with every item judged blind against the code. Budget models got 80% of the way for half a cent, and the mid-price GLM-5.3 matched the frontier models for 4 cents. Max reasoning effort mostly wasted money. Most of all, the same frontier model scored noticeably better inside an agent harness than through a raw API, and feeding it more code didn’t close the gap. How a model explores the code mattered more than which model it was.

This is a fast-moving field. The models, prices and providers here are as tested in October 2026, and some will have changed by the time you read this.

An inference gateway is a small service that sits between your application and every model you use. Your code talks to one endpoint. The gateway decides where each request goes.

It’s not exotic. It’s the same idea as a load balancer or an API gateway, applied to models. For a small team it’s usually the first piece of AI infrastructure worth building, because it makes every later decision cheaper.

What it does

One API for every model. Your application speaks one format, usually the OpenAI-compatible chat API that most providers and open-source servers now accept. Behind the gateway that might be a frontier API, an open-weight model through OpenRouter, or a model you host yourself on vLLM. Swapping between them is a configuration change.

Routing. Rules decide which model handles which request. Classification and extraction go to a small, cheap model. Long reasoning goes to a large one. Internal tools can use a cheaper tier than customer-facing features.

Fallbacks. When a provider times out or rate-limits you, the gateway retries against a second provider. Your users see a slightly slower response instead of an error.

Cost tracking. Every request is logged with its model, token counts and cost, tagged by feature or customer. For the first time you can answer “what does this feature cost us per user?” with a number rather than a guess.

Limits and keys. Budgets per team or per feature, and one place to rotate provider keys, rather than secrets scattered across services.

Caching. Identical or near-identical requests can be answered from cache. How much this saves depends entirely on your traffic, so measure before you count on it.

What it doesn’t do

It won’t make a bad prompt good or a small model clever. It adds a hop, so it has to be close to your application and properly monitored. It’s one more service to run, which is why it should be boring: a single container, stateless apart from its log store, deployed the same way as everything else.

Build or adopt

You don’t need to write one from scratch. LiteLLM is the common open-source starting point: it speaks the OpenAI-compatible API, supports most providers and open-weight servers, and handles routing, fallbacks and spend tracking. Managed routers such as OpenRouter cover part of the same ground as a hosted service.

The work isn’t installing it. It’s deciding the routing rules, wiring cost data into something people actually look at, and making sure the gateway itself never becomes the outage.

When it’s worth it

As soon as you use more than one model, or spend enough on inference that someone has asked why. In practice, that’s most teams within a few months of shipping their first AI feature.

It’s also the right first step before any self-hosting decision. Once traffic goes through a gateway, trying an open-weight model on 5% of requests is a configuration change, and you get real cost and quality numbers instead of a guess.

Setting one up is usually the first week of my AI Inference Setup engagement.