At the Hot Chips conference on Tuesday, OpenAI finally pulled back the curtain on its custom inference chip, Jalapeño, and - surprise, surprise - it's pretty good at what it does. Tested on SemiAnalysis' InferenceX benchmark, Jalapeño delivered more tokens per user and more throughput per kilowatt than the current state-of-the-art inference processors. That's a fancy way of saying it serves more AI per watt, which is the kind of efficiency that makes accountants and environmentalists both do a little happy dance.

“The bottom line is that the results show a very, very significant performance advance over state of the art,” said Richard Ho, OpenAI’s head of hardware, in a press call. “Jalapeño can serve more AI work per unit of power, while also returning responses more quickly. It’s very efficient to serve a lot of customers, but it can also be very low latency.”

But before you start planning a parade, here's the kicker: that comparison is against an Nvidia Blackwell system - which, by the time Jalapeño actually hits the market, might have a few more tricks up its sleeve. Ho estimated that Jalapeño would deploy at the end of 2026 “in very small volumes,” with more significant deployment coming in 2027. So, we've got a while before we taste the full flavor.

First announced last October, Jalapeño was developed by OpenAI in close collaboration with Broadcom, with OpenAI’s own models assisting in the development process. The company plans to make Jalapeño a multigenerational platform, allowing AI products, models, chips, and memory all developed in concert. Because of that full-stack approach, OpenAI was able to address specific phases in the inference process that often cause friction during inference processing. In particular, Jalapeño is designed to minimize delays during the prefill and communication phases of processing, which OpenAI says often act as bottlenecks.

“We designed Jalapeño to minimize data movement and communication delays,” the company said in a blog post presenting the results. “This means that model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase.” In other words, they've made the chip smart about where it puts things, so it doesn't have to run around looking for them. Revolutionary, we know.