Kog’s $5M Bet: 3,000 Tokens a Second Could Humiliate AI’s GPU Spend
AI companies are spending fortunes on chips while Kog says the hardware is already being wasted. Its $5M bet is that better software, not another GPU order, is where the real margin sits.
Most AI companies are responding to slow models by buying more chips. Kog has raised just $5 million to argue that a frightening amount of that spending is lazy engineering.
The Paris startup says it can generate 3,000 output tokens per second for one request on a standard eight-GPU AMD MI300X server. If that translates from demo to production on the big models businesses actually use, it is not a cute technical win. It is a direct attack on the assumption underpinning half the AI infrastructure gold rush: that more compute is the only answer. ([blog.kog.ai](https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/))
The core story: Kog is selling speed, not another model
Kog is a Paris-based AI-infrastructure startup founded in 2023 by Gaël Delalleau. It has an 11-person team, including engineers and researchers, and says it has raised $5 million from Varsity VC and Bpifrance’s Deep Tech program. Its product is the Kog Inference Engine, designed to make language models respond dramatically faster on the data-centre GPUs companies already own. ([blog.kog.ai](https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/))
That last bit matters. The AI world has become addicted to hardware theatre. Bigger GPU clusters. Bigger capex plans. Bigger promises that the next generation of silicon will rescue unit economics.
Kog’s pitch is less glamorous and far more useful: quit treating a GPU as a magic black box. Understand how its memory, communications, scheduling and physical layout actually behave, then write software that stops wasting them.
Its May technical preview ran a purpose-built 2-billion-parameter coding model, Laneformer 2B, at 3,000 tokens per second per request on eight AMD MI300X GPUs. Kog also reported 2,100 tokens per second on eight Nvidia H200 GPUs. Those figures were for batch size one, FP16, without quantisation or speculative decoding. That is important because it isolates the experience of one person waiting for one agent to do a job, rather than hiding sluggishness behind a flattering aggregate-throughput benchmark. ([blog.kog.ai](https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/))
For founders building coding agents, research tools or workflow software, that is the number that matters. Your customer does not care that your fleet produced a heroic number of tokens across 10,000 requests. They care whether their agent finishes the task before they lose patience and open another tab.
Kog told TechCrunch it collected 200 tangible business leads after its preview. It is initially targeting software-engineering workflows, plus design partners building prompt-driven game and app creation tools. ([techcrunch.com](https://techcrunch.com/2026/08/14/kog-is-going-deeper-to-squeeze-more-inference-out-of-gpus/))
That is the commercial opportunity. Fast inference is not merely a nicer user interface. In agentic software, every slow output delays the next action: inspect code, make a plan, write a patch, run tests, find the bug, try again. A few seconds of latency multiplied across a long chain becomes an unusable product.
The number nobody should ignore
Kog gives a brutally simple example. At 100 tokens per second, an agent generating 50,000 tokens takes roughly eight minutes. At 3,000 tokens per second, it takes under 20 seconds. Sure, a real agent also waits on tools, test suites, browsers and databases. But when generation is the bottleneck, shrinking it by that magnitude changes what a customer will tolerate and what a product can charge for. ([blog.kog.ai](https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/))
This is what founders routinely miss: performance is not an engineering vanity metric when it changes customer behaviour.
A customer who waits eight minutes asks the agent to do one narrow thing. A customer who gets an answer in seconds asks it to explore options, retry failures and do more work. Faster systems create more usage because people stop rationing requests. More usage can create more value. More value gives you room to charge.
That is why Anthropic can charge a premium for its Fast Mode. Speed has become a product feature with a price tag, not a backend detail that belongs in a DevOps slide deck. ([techcrunch.com](https://techcrunch.com/2026/08/14/kog-is-going-deeper-to-squeeze-more-inference-out-of-gpus/))
Why Kog believes standard GPUs have more in them
Kog’s technical case is that single-request language-model decoding is predominantly constrained by memory bandwidth, not headline compute power. In plain English: for each generated token, the machine spends much of its life moving model weights around. If those movements stall, all the expensive mathematical horsepower sits around looking decorative.
The company’s answer is unusually deep in the stack. It combines a persistent “monokernel” runtime, custom inter-GPU communications software called KCCL, and a model architecture designed around multi-GPU data movement. Instead of constantly bouncing between many kernels and CPU instructions, it aims to keep the critical decoding process running as one tightly controlled GPU-resident program. ([blog.kog.ai](https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/))
This is proper hard engineering. It is also the sort of work startups avoid because it is difficult to demo in a pitch deck and difficult to hire for. Building another wrapper around a model API is easier. Understanding the physical topology inside an AMD MI300X, locating nanoseconds of delay, and reorganising memory access around it is much less sexy — until it cuts your customer’s wait time from minutes to seconds.
Kog says a 25-layer model running at 3,000 tokens per second has only about 333 microseconds of budget per token. Waste one microsecond in each layer and you burn 7.5% of the time budget. That is the whole game: tiny inefficiencies, repeated endlessly, become a bloody big bill. ([blog.kog.ai](https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/))
The overlooked angle: this could be terrible news for lazy AI infrastructure
Here is the contrarian take: the best outcome for AI startups may be less hardware scarcity, not more hardware supply.
Everyone is excited by the next GPU because hardware is easy to understand. Buy more, get more. But software that extracts more work from the same server is financially superior for the buyer and strategically awkward for anyone selling capacity by the hour.
If Kog or companies like it can materially improve inference on existing Nvidia and AMD fleets, then some of the infrastructure spend now considered inevitable turns into optionality. A startup can delay a cluster purchase. An AI cloud can support more customers per machine. An enterprise can use the hardware it already bought before the CFO asks why the shiny rack is underutilised.
That does not mean GPUs suddenly become worthless. Don’t be silly. Better models, more users and more agentic workloads will still eat extraordinary compute. But it does mean hardware demand and software efficiency are not enemies. They compound. More memory bandwidth plus smarter inference software can make bigger models viable at usable speeds without every company needing to build a private power station.
Kog itself expects future GPU generations with higher memory bandwidth to help it run bigger models quickly, potentially with fewer GPUs. That is the more interesting investment thesis: not “AI needs infinite chips,” but “AI needs a better operating system for the chips it has.” ([blog.kog.ai](https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/))
The bit that should make you cautious
Now for the part where founders need to put the champagne back in the fridge.
Kog’s headline result is a technical preview using its own small, purpose-built model. It is not yet proof that a frontier third-party model will perform at the same level in a messy production environment with long contexts, real customer traffic, tool calls, reliability requirements and competing workloads.
Kog is candid about that gap. Its 3,000-token result used a model of roughly 2 billion parameters. Its next challenge is bringing the same approach to much larger open-weight mixture-of-experts models, where it estimates real-world performance will be lower than theoretical ceilings once communication and other overheads are counted. ([blog.kog.ai](https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/))
That is not a flaw in the story. It is the story.
Early-stage companies do not win by proving every last thing on day one. They win by identifying the expensive bottleneck everyone else accepts, then showing enough evidence that customers want to help solve it with them. Kog’s 200 reported leads suggest the pain is real. The next proof point is whether those leads become paying design partners, then durable revenue. ([techcrunch.com](https://techcrunch.com/2026/08/14/kog-is-going-deeper-to-squeeze-more-inference-out-of-gpus/))
What this means for you
If you are a founder building with AI, stop bragging about which model you use and start measuring where your customer waits.
Tomorrow, pull up your product telemetry and answer four questions:
1. What is the wall-clock time to a completed customer outcome? Not time to first token. Not model benchmark score. The finished job. 2. How much of that time is model generation? Separate it from tool calls, database queries, browser automation and human approval. 3. What does delay cost you? Lower conversion, lower retention, fewer tasks per user, more support tickets or a lower price ceiling. 4. Could a faster, cheaper inference stack change the product itself? If the answer is yes, performance belongs in your company strategy, not just your engineering backlog.
And if you are an investor, be wary of startups whose whole moat is access to expensive compute. Compute is an input. The durable value is often in making that input do more work, producing a better workflow, or owning the customer relationship on top.
Kog may or may not become the company that cracks inference at scale. But its underlying point is dead right: spending more money on hardware is not a strategy when your software is still leaving the engine idling in the driveway.