OpenAI Jalapeño vs Blackwell: perf per watt is real, the HBM4 asterisk is too

On August 25 OpenAI put its own inference chip, Jalapeño, on a public benchmark for the first time. The numbers from SemiAnalysis InferenceX: 1.5-1.9x more compute per watt than Nvidia Blackwell, 1.7-3.6x lower latency, and on the most interactive workloads 2.1-4.1x. The headline comparison is GPT-OSS 120B on Jalapeño versus a GB200 system: 85,448 versus 44,960 mixed TPS per kW, about 1.9x peak performance per kilowatt. OpenAI plans to deploy it in its own infrastructure by the end of 2026. CNBC and SemiAnalysis both carried the results.

Perf per watt is the operating cost

For an inference operator, performance per watt is not an engineering vanity metric; it is the bill. Power and cooling are the constraint on every new site, so a chip that does 1.9x the tokens per kilowatt roughly halves the marginal cost of serving, before you count that OpenAI no longer pays Nvidia's margin on that capacity. Custom silicon from a lab is a second data point, after Cerebras CS-4 on August 20, that inference is moving off general-purpose GPUs. For anyone buying API tokens the long-run implication is that prices keep falling, which strengthens the routing and cost thesis rather than the buy-hardware thesis.

The HBM4 asterisk

Jalapeño uses HBM4. Blackwell uses HBM3e. Decode is memory-bandwidth bound, so a chip with a newer, wider memory system wins on tokens per second and on latency partly because of the memory, not the architecture. The like-for-like Nvidia part is Rubin, which also carries HBM4 and is not what OpenAI benchmarked against. Some fraction of the 1.9x is a memory-generation gap that closes the moment Rubin ships in volume. I cannot tell you what fraction, and neither can the press release.

A perf-per-watt number without the memory generation attached is a marketing number; the architecture only gets credit for what is left after you subtract the HBM.

How to read InferenceX and vendor-first benchmarks

InferenceX reports mixed TPS per kW, which blends prefill and decode. That matters because the two phases stress different things: prefill is compute-bound and favours raw FLOPs, decode is bandwidth-bound and favours memory. A mixed number can be dominated by whichever phase the chosen workload mix leans on, and interactive workloads lean on decode, which is exactly where HBM4 helps most. So the 2.1-4.1x on interactive is the least surprising and the least transferable number in the set.

• Ask which phase: prefill throughput and decode throughput separately, per kW.
• Ask which memory: HBM generation and bandwidth, then compare against the part with the same memory.
• Ask which model and precision: GPT-OSS 120B at whatever quantisation OpenAI chose; the NVFP4 pipeline shows how much precision alone shifts throughput on Blackwell.
• Ask who ran it: first-party results on a public harness beat a slide and lose to a third party's rerun.
• Convert to quality-adjusted tokens: raw tokens per second without an accuracy check is half a metric.

The Hopper vs Blackwell write-up went through the same exercise for Nvidia's own generational claims, and the lesson was identical: the memory system explains most of the delta on decode.

What it means if you are not OpenAI

You cannot buy Jalapeño. Its effect on you is indirect: cheaper inference for OpenAI translates into pricing room, and a credible non-Nvidia serving part in one lab's fleet is leverage in every other negotiation with Nvidia. If you run your own inference, the actionable version is to stop reading perf-per-watt headlines and start measuring your own decode tokens per kW on the hardware you actually have, with the memory bandwidth written next to it. The same Samsung LPDDR5X-PIM preview from Hot Chips the following day, 3x token throughput on paper, deserves the same treatment: preliminary until measured.

The honest limitation

These are OpenAI's numbers on a public harness, published on OpenAI's schedule, against the Nvidia part that flatters the comparison. Until SemiAnalysis or someone else runs Jalapeño against Rubin under the same workload mix, treat the 1.9x as an upper bound on the architecture's contribution, not a measurement of it.