I attended this year’s Hot Chips end to end, starting from Sunday’s tutorial. The most impressive presentation is OpenAI’s Jalapeño. Seeing so much of their team on stage gave me a strong impression. Overall I learned and will share 3 things in this post:
What makes their team effective?
What is different about the Jalapeño architecture?
How is AI going to completely disrupt the chip design industry?
The first question is relevant to every CEO, because AI has redefined the shape of an effective organization. Specifically: with AI, a small dense team whose people each span several areas will beat a big org with the same amount of talent spread thin.
The second and third questions are relevant to every entrepreneur because it’s a textbook example for how to use AI to disrupt your field. In this case, it’s the supposedly most undisruptable company, NVIDIA, that is getting disrupted. NVIDIA’s proud ecosystem CUDA might actually become their innovator’s dilemma. And OpenAI, previously a nobody in the chip field, is transforming the chip industry by rethinking the problem from first principles and leveraging AI.
It’s bold.
1. What makes the team so effective?
They are bold, in 2 ways
When I was below the stage hearing the Jalapeño talk, I had this very intuitive feeling: these people are really bold. Then later, I think a better way to put it is what Fei-Fei Li calls “intellectual fearlessness”. And I think them being bold is embodied in 2 ways: in how they think and how they do things:
How they think
No. 1: the people on stage — Richard Ho, Ravi Narayanaswami, Chris Leary — constantly challenge what the industry thinks. By industry, I mean the overall perception of the industry of the right way to do things. First they questioned the basic assumptions of the problems. There are 2 examples of this: most of the industry thought that per-user decode speed is capped by the bandwidth of HBM, the memory stacks next to the chip — that’s why the “fast” chips have been SRAM-based. But Ravi said flatly “HBM is not the bottleneck”: the problem is operands, the numbers a compute unit needs in its registers before it can multiply, not showing up in time. It’s an orchestration problem, not a bandwidth problem; existing bandwidth is far from being fully utilized. Chris illustrated this with a napkin calculation, which I cover in the architecture section below. The second example is that prefill/decode disaggregation and heterogeneous clusters were presumed to be the future; many players like Groq and Cerebras already split prefill onto other hardware to cut latency while increasing throughput. But Ravi simply said: “We do not believe that the long-term solution involves moving large KV states.” And because the prefill / draft / verify ratio keeps changing with every new model, context length and algorithm, a fleet of specialized chips ends up with idle hardware, while one balanced chip just gates off the blocks it isn’t using. Their slide puts it as “dark silicon is cheaper than idle accelerators.”
So by bold, I mean these people are first-principles thinkers. Most of us in the industry tend to think inductively rather than deductively: we see/hear about a trend A/B/C and we inductively say that A, B, C are true. But OpenAI folks are ones who say: no, A is not true, and B, C are only true in certain cases with strong trade-offs that make them not worth pursuing. No doubt, this requires lots of work, but it also requires lots of boldness to question basic assumptions that the whole industry takes for granted.
How they do things
No. 2: they completely changed how a chip’s full stack should be designed and built. This spans both the chip’s software and hardware: how kernels, the small programs that each run one step of the model on the chip, should be written for each new generation of models, and how to accelerate the chip development cycle.
What enabled the new opportunity is that our world has shifted so fundamentally that an old assumption from when NVIDIA was building chips no longer holds: intelligence today comes cheaply and in abundance. It’s now AI alongside humans developing, optimizing, and even using chips. And this shifted under their feet mid-program: they started in the o3 era, when it was a struggle to get the AI to write kernels for the new architecture, and by now the frontier models are “really good” at it. So it only makes sense to rethink building chips under this new assumption, and to share the driver seat with AI across the entire stack: AI simulates the chip, writes the software kernels, optimizes kernels for specific new model architectures, tests their performance and iterates.
By doing this loop, the AI takes over the part that is onerous for humans, planning which of the chip’s 64 cores gets which operands and when, so the team could afford a chip that gives up the GPU’s conveniences — unified memory, hardware scheduling — for explicit placement. Chris was clear they still designed it so humans can program it; the AI just makes the tedious part cheap. That’s what opens a new Pareto curve. This is how they could beat NVIDIA… The whole last section is about this loop.
Three things make the team work
I think it comes down to 3 things: talent density and cross-functional people, great objectives and non-objectives, and short feedback loops.
Talent density
Talent density mattered before AI, but if anything AI has amplified the effect: a small team with highly talented players beats anything else, or simply: talent density wins. Richard Ho’s version: “an extremely small team on a very insane timeline.” And the reason a small team can now cover the ground of a big one is that the grunt sub-teams are AI. Chris put it like this: “It was like having a sub team of people who are optimizing PPA [power, performance, area] for us in the background.” The humans span the boundaries; the AI fills in the depth behind each of them. The other trait is being cross-functional: people who think in complementary ways but can still talk to each other at a very high level. That way each person’s specialist view flows freely through the whole team. Here I quote one of the speakers, Chris Leary:
OK, so you want to build the best chip in the world, and you want to go real fast. You have two choices, build a really big team or build a really small team. We picked the latter […] having as many people be cross-functional across these boundaries, partitions was an essential part of what made Jalapeno possible to do in the amount of time that it was done in.
Great objectives and non-objectives
The second thing is having great objectives, and great objectives have something in common, it’s about having less rather than more. And it’s as much about what not to do as what to do. And they embody this perfectly on this slide, which shows that they have 2 objectives: latency and energy. Not even a third one! That’s how much focus, and how much saying no, you need. And everything else is explicitly listed as NOT an objective: # chips used, throughput per chip, TTFT. TTFT is time to first token; what they measure instead is TTLT, time to LAST token, when the whole answer is done.
Short feedback loops
It turns out that to improve a chip’s performance is simple! Like all things, it’s really just about iterations. A chip’s final performance comes down to how many end-to-end iterations you get to run on it, from the hardware architecture down to the software kernels. The reason it’s hard is because each iteration cycle is humongously long as chip design involves many stages: architecture, microarchitecture, RTL, synthesis, physical design. RTL is the actual source code of the chip’s logic; the two stages after it turn that code into gates and then into a layout on silicon. None of these give you a real chip yet — you are working against a software simulator or an FPGA emulation of the chip, plus the synthesis and physical-design numbers that tell you whether it fits and makes timing. And next we need to work out how a specific kernel will run on the chip, which is the software part of the chip. Then we break down running a model’s architecture into multiple kernels, and then start to optimize that entire system. From there we can get the performance metrics for our objectives: latency and energy.
Given that time is fixed, how many iterations we can do is simply the inverse of how long each iteration takes. So the whole problem is about how to cut the feedback cycle. The Jalapeño team spent lots of effort creating a specialized stack to support this flow. Here I quote Chris Leary:
[…] we actually kind of discovered over the course of this program what makes that work well and what we discovered was really the key to that really fast cadence that you saw at the beginning of the talk was being able to run this loop you might call it agile, you might call it co-design, co-design can mean lots of different things but really running this loop as many times as possible an insight into how to deliver the performance turning into workload mapping turning into something that went into our cycle approximate simulator, turning into RTL implementation, getting vetted through the DV and PD [design verification and physical design] process, we were running changes right up until the very end, which is not always making everybody happy, but it was actually essential for reaching the performance level that we ended up achieving. And so really we think that iterating this loop and running it relentlessly and having as many people be cross-functional across these boundaries, partitions was an essential part of what made Jalapeno possible to do in the amount of time that it was done in.
And it’s written on their slide #27:
None of these 3 things is about chips. They’re about how you run a team when intelligence is cheap.
2. What is different about the Jalapeño architecture?
The short version
The short version of why Jalapeño is faster and more energy-efficient than a GPU — and, as a side effect rather than a goal, gets today’s best throughput at a fraction of the latency:
OpenAI found that the real bottleneck on a GPU isn’t raw memory speed, it’s that the operands aren’t there when the compute unit needs them — because every unit is queuing for the same shared memory. Think 64 chefs and one pantry. So OpenAI gives every core its own private slice of memory, stocks it in advance with the numbers that core will need, and choreographs the unavoidable cross-core traffic over a dedicated contention-free collective network. They call it a memory-sliced, NUMA-style architecture. Less running around means less energy and lower latency.
But there’s a trade-off: now someone has to plan ahead which operand gets planted in which HBM slice, and when and how it will be used, so that it’s there exactly when it’s needed.
How do we know when each operand arrives, so we can orchestrate them? We know this by looking at and co-optimizing a specific model’s inference workload. And since most model architectures today are quite different — slightly different attention, a different MoE, and so on — we have to write the mapping per kernel type. MoE is mixture of experts, where only a few sub-networks fire per token. The kernels are parameterized over shapes, so a new model that shares an architecture mostly just reuses them. OpenAI was explicit that co-design “does not mean overfitting to a single model.”
So for DeepSeek R1, they worked out how the operands for its MLA attention should be orchestrated and dance on the hardware. MLA is multi-head latent attention, DeepSeek’s compressed-attention trick. Kimi K2.5 is basically the same architecture, so those kernels just ran on Kimi’s shapes; Ravi said it took a few days. GPT-OSS, OpenAI’s open-weight model, uses a different attention, so its attention and MoE kernels get re-optimized — and there the AI beat the existing expert-written ones by 1.5-1.8x. The hardware is designed to be more explicit, not more complex — sliced memory instead of one shared pool, a dedicated collective network, a deliberately anemic general NoC — which kills the contention but pushes the hard part into the mapping: someone has to plan which operand lives where and when it moves. That’s the trade-off Chris said they made. History picked CUDA’s simplicity, meaning dynamic hardware scheduling and unified memory, over explicitly scheduled spatial designs that needed a very smart compiler or a very patient programmer, for a good reason: nobody could afford expensive programmers spending weeks optimizing one workload. So you needed a general architecture that’s efficient and easy for humans to program. Fundamentally, the trade-off OpenAI made is to spend extra time writing more specialized software for each workload. And this is a good problem to have because the iteration speed for software writing is getting increasingly better with the advancement of more intelligent LLM agents. So what fundamentally changed is that AI is writing code rather than humans!
My take: OpenAI is willing to let its internal Codex tune the kernels for every model family it runs and every phase of inference, rather than hand-optimizing a handful of workloads the way everyone else has to.
For scale: on the public InferenceX benchmark, Jalapeño hits ~1,460 tokens/s per user on GPT-OSS-120B where NVIDIA’s GB200 tops out around 535, and ~700 on DeepSeek R1 vs ~170 on GB300 — at a 700 W package vs 1,200–1,400 W. Those are single-token-prediction numbers on both sides; when you let GB300 use multi-token prediction (speculative decoding) and keep Jalapeño without it, Jalapeño still leads (700 vs 329 tokens/s per user).
What’s wrong with existing GPUs?
Now to look at the above in a more detailed picture, I found this slide to perfectly illustrate OpenAI’s first-principles thinking towards the fundamental problems of GPUs.
On it, they used a napkin calculation to illustrate that there’s something wrong with saying that high latency is because of memory bandwidth not being enough. In fact HBM is far from being fully utilized and the theoretical, bandwidth-only token rate is far from being achieved. Specifically, given a rack of 128 Jalapeño chips with 1+ PB/s of aggregate HBM bandwidth (15.4 TB/s each), and a 1T-parameter LLM at 4-bit weights, i.e. 0.5 TB, then every second the rack can read the whole model 2000+ times. And in the answer-writing phase every token you generate needs one pass over the weights, so that’s a ceiling of ~2000 tokens/s for a single user with no speculative-decoding tricks. Then why cannot we see anything close to 1000-2000 tokens per second for a single user on a model that size in reality?
So the problem is operands showing up late, because everything is fighting over shared paths:
The fix: sliced memory
OpenAI’s answer is the memory-sliced architecture: each of the 64 cores owns its own private slice of HBM instead of everyone sharing one pool.
3. How is AI going to disrupt chip design?
A chip is a loop
As I mentioned under “short feedback loops”, designing a chip is all about how many iterations you can run end to end, and each iteration is long because it goes through many stages. So the key is to cut the feedback cycle within each stage, and use the signals to inform the next round of iteration. Here’s their whole loop on one slide:
The sections below follow the five boxes on this slide.
From workload to kernels to mapping
Because we care about the overall performance of the workload, which depends on the software running on the hardware, we naturally need to first define what the workload looks like. This part involves looking into what the model’s architecture looks like: how the model’s inner components communicate with each other. This is the workload.
So how do we describe the workload on the hardware? With kernels. By hardware I just mean a simplified model of the chip for now: some memory local to each core, some compute, a way to move data between cores. Which memory, how big, and where it sits are decided in the hardware design phase. The kernel says what each step computes and how it’s tiled against that model; the mapping, 2 paragraphs down, is where the real chip comes in.
So what’s a kernel? One unit of work you launch on the accelerator. Simplest case it’s a single op: a matmul kernel, a softmax kernel. But usually you fuse neighboring ops into one program, like an attention kernel that does QKᵀ, softmax and the multiply by V in one go, or an MoE kernel that does the routing plus the expert matmuls. That’s a fused kernel. Why fuse? Less data movement. With 2 separate kernels, the first has to write its whole output to HBM and the second has to read it all back. Inside a fused kernel that intermediate never leaves the chip. Fusing also means fewer launches, and every launch costs a fixed overhead plus a sync point.
Once the workload is a chain of kernels, the next question is how each kernel gets laid out on the hardware: which HBM slice holds each tensor, which cores take which tiles of the computation, in what order they run, and which exchanges between cores go over the fast collective network. This is the mapping, and it’s where the shape of the chip starts to matter. The same kernel can be mapped many ways, all giving the same result at very different speeds. On a GPU the unified memory and hardware scheduler absorb a poor mapping, at some cost. On Jalapeño’s sliced-memory design the only cushion is the slow global view over the general NoC, which they made deliberately anemic and try never to touch — so the mapping decides whether the chip hits its roofline or sits waiting for operands.
Simulate: Chilisim
This is the part that surprised me most. So far this reads like a straight pipeline: workload, then kernels, then mapping, then chip. But it can’t be — mapping needs a chip to map onto, and the chip doesn’t exist yet. So the workload isn’t a blueprint, it’s a guide to what to measure. Instead, start from an architectural hypothesis: sliced memory, a collective network, and one balanced chip for every phase instead of a heterogeneous fleet — the prefill/decode point from section 1. Build a cycle-accurate simulator of it, which they call Chilisim and which lands within 5% of real silicon [3]. Map the kernels onto that simulated chip, measure, change the architecture, re-map, repeat. The chip is whatever that loop converges to. And the workload isn’t the only constraint. The other is physics and cost: a 700 W package budget, the die area you get on TSMC’s N3P node [2], whatever HBM4 stacks and interface IP you can actually get, and standard cells only.
Back to Chilisim. It’s a cycle-accurate model of the proposed chip architecture, i.e. a chip simulator: 64 core slices, each with its own local HBM view, a collective network, a thin general NoC, and out-of-order cores [3] with whatever matrix-unit width you picked. Run the kernels on it and you get cycle counts, stall reasons, and how much of roofline you actually hit. If even the best mapping can’t reach the memory roofline, they change the architecture (slice size, collective topology, unit width), edit the simulator to match, and re-tune the kernels against it. So Chilisim is the bridge between software and hardware: the one version of the chip that’s cheap enough to change every day, while the architecture is still fluid. It spits out the numbers the team uses to decide whether to change the chip, and the numbers the AI optimizes against when it changes the mapping.
RTL + QoR
Chilisim is only a simulator, good to about 5% of what real silicon does. To get higher fidelity, they move to RTL. The goal here is to implement the architecture the simulator embodies as an actual hardware description: more than half of the core in XLS, Google’s open-source hardware language that Chris Leary co-founded, the rest in Verilog, plus licensed interface IP for the HBM, SerDes and PCIe links. The AI liked XLS because “it kind of looks like Rust.” Their internal Codex model [3] proposes and PPA-optimizes blocks — the BF16 multiplier, FP4 dot, FP32 accumulator — alongside the designers. Two checks run continuously: DV, which checks the RTL actually does what the spec says, and synthesis, which maps the RTL onto standard cells and returns QoR, quality of results: area, timing, power. If QoR says the design doesn’t fit or doesn’t make 1.8 GHz, the feedback goes up: shrink the block, or change the architecture, which changes Chilisim, which changes what the kernels are tuned against. This upward feedback is the “convergence” part — “track DV, PD, and performance together”, as their slide puts it — and it runs until RTL freeze in July 2025.
Physical design
This is the “PD” from Chris’s quote. Floorplanning, deciding where each block sits on the die, runs alongside the RTL work — that’s how the team found out Jalapeño’s content wasn’t going to fit, and the fix came back as AI-PPA-optimized RTL before freeze. After freeze, behavior is fixed and the remaining questions are geometric: placement and routing of cells and wires across the metal layers, timing closure, power and IR-drop analysis, and design-rule checking against TSMC’s rules. Findings here can still force ECOs on the frozen RTL, but architectural changes are off the table by now; the features they wanted but couldn’t fit into this train got rolled into Gen 2. This stage ends at tapeout, sending the final layout to the fab, in November 2025.
Where AI sits in the loop
So where does the AI sit in this loop? First, the kernel-optimization loop: starting from a correct-but-slow functional kernel, the AI proposes changes one after another — FP8 attention matmuls, attention rescaling with tiled lookahead, value-matmul scheduling, key-tile prefetch + coalesced scale access — and their internal harness scores each one on the simulator or on silicon. For DeepSeek’s MLA attention, which OpenAI had never needed before, this went from “functional only” to ~89% of roofline in about 48 hours:
For kernel mapping, humans write the kernel in Gluon against a simple model: local tensors, explicit communication. The AI searches the mapping: placement, scheduling, collective orchestration. And the hardware is built with a clean local-vs-global hierarchy so that the search has something predictable to optimize against.
Next, the chip as a kernel sees it: each core computes on its own HBM slice through a fast local view, the collective network is the planned path between cores, global memory is the slow fallback, and TensorInfo is where the mapping (which slice, which core) is written down.
And the same loop one layer down, at the RTL + QoR step: the AI proposes hardware blocks in XLS, verification rejects wrong ones, synthesis returns QoR, and the winners — a 56% better BF16 multiplier and a 10% smaller matrix unit — are what squeezed the content into the floorplan, in Chris’s words, and let changes keep landing until the day of RTL freeze.
So, back to my 3 questions. The team works because it’s small, dense and cross-functional, with 2 objectives and a loop it runs relentlessly. The chip is different because they questioned what the whole industry took for granted, and moved the hard part out of the hardware and into the mapping, where AI does the tedious work.
It’s bold.
quReferences:
[1] Richard Ho, Ravi Narayanaswami, Chris Leary (OpenAI), “You Can Just Build Things … Chips”, Hot Chips 2026, Stanford, Aug 25, 2026 — slides + video are attendee-only until Hot Chips posts them publicly in December 2026.
[2] zartbot, in Chinese: “重新设计推理芯片：从 Nvidia GPU 的缺陷到 OpenAI Jalapeño”, i.e. “Redesigning the inference chip: from the flaws of Nvidia’s GPU to OpenAI’s Jalapeño” (my translation), https://zartbot.github.io/blog/arch/jalapeno/index.html
[3] SemiAnalysis, “OpenAI Jalapeño: Better Than Nvidia Blackwell”, Aug 25, 2026, https://newsletter.semianalysis.com/p/openai-jalapeno-better-than-nvidia — source for the Chilisim name, the 5% figure, the out-of-order cores and the internal Codex model.