Modular Inc. says its MAX inference stack ran 5.5 times faster than vLLM at P50 time to first token when serving Moonshot AI’s Kimi-K2.5 model on Nvidia B200 GPUs. The figure comes from chief scientist Abdul Dakkak’s lightning talk at MLSys 2026 on May 18, and Modular restated it in a company blog post, Three Trends From MLSys 2026, published June 12 and updated October 2. For a company that has spent three years selling developers on a single compiler stack that targets any chip, a benchmark this specific is the clearest pitch yet: stop writing separate serving code for every GPU vendor.
The claim lands at a moment when inference cost, not training cost, has become the line item CFOs actually watch. Every major model lab has shipped a flagship release in the past few months, from Kimi K3 to Nvidia’s own Nemotron line, and all of them need somewhere to run at scale. Modular’s bet is that the serving layer, not the model, is where the next cost war gets fought.
What Modular Actually Announced
Modular’s MAX platform is a compiler and runtime designed to take one container and run it across Nvidia GPUs, AMD GPUs, and CPUs without a rewrite. The company’s benchmark claims, as laid out in its MLSys recap, compare MAX against vLLM, the widely used open-source inference server that many startups and research labs treat as a default choice.
On Kimi-K2.5 running on Nvidia B200 hardware, Modular reports a 5.5x advantage in P50 time to first token over vLLM. On a second model, Gemma-4-31B-it, it reports a 2.5x advantage in P99 TTFT. Across both models, Modular claims 1.5x higher throughput than vLLM. Those are the three numbers the company has put its name behind in writing, and they form the backbone of this story.
What the public material does not include is the full test harness: batch sizes, quantization settings, sequence lengths, or the exact vLLM version used as the baseline. Benchmark gaps like this are common in vendor-published numbers, and they matter because a 5.5x gain under one batch size can shrink or vanish under another. Readers should treat the figure as a vendor claim backed by a named conference presentation, not as an independently reproduced result.
Breaking Down P50 TTFT and Why It’s the Metric That Matters
Time to first token measures how long a user waits between sending a prompt and seeing the first word come back. P50 means the median case, the typical request a real user experiences, rather than a worst-case tail. For chat products, coding assistants, and agent workflows, TTFT is often the number that decides whether an app feels instant or sluggish, even when total generation time is similar across stacks.
Modular also published a P99 figure, the slowest 1% of requests, where it claims a 2.5x edge on Gemma-4-31B-it. P99 numbers tend to expose scheduling and memory-management weaknesses that don’t show up in median-case testing, so citing both metrics is a stronger signal than citing P50 alone. Still, without the underlying methodology published alongside the headline ratios, the numbers function as a marketing claim with a technical pedigree, not a peer-reviewed result.
Who Is Abdul Dakkak, and Why MLSys 2026 Is the Venue That Matters
Abdul Dakkak is Modular’s chief scientist, and he presented the company’s “SOTA Perf on MAX” talk as a sponsor lightning session at MLSys 2026 on May 18. MLSys is a peer-adjacent systems conference where machine learning infrastructure researchers and vendors present performance work, which gives the talk more technical credibility than a press release, even though the specific benchmark numbers themselves have not been independently reproduced by a third party as of this writing.
Dakkak also appeared at the AI Engineer World’s Fair 2026, where his session described MAX’s cross-hardware positioning to a developer audience rather than a systems-research one. The repeated venue choice, back to back industry and academic stages, suggests Modular is trying to win over both camps at once: the engineers who buy inference infrastructure and the researchers who validate performance claims.
Kimi K2.5: The Model Modular Chose as Its Benchmark
Kimi-K2.5 comes from Chinese lab Moonshot AI, released January 27, 2026. It’s a mixture-of-experts model with roughly 1.04 trillion total parameters and about 32 billion active parameters per token, spread across 384 experts. It handles text and images natively through a vision encoder called MoonViT, supports a context window in the range of 256,000 tokens, and ships under a modified MIT license that keeps it open-weight.
Moonshot built the model with agentic workloads in mind, including a feature the lab calls Agent Swarm, which can coordinate up to 100 sub-agents through as many as 1,500 parallel steps. Picking a model this large and this agent-oriented as the benchmark subject is a deliberate choice by Modular: it signals that MAX’s performance claims are aimed at the kind of heavyweight, multi-step inference workloads enterprises are starting to deploy, not toy chatbot demos.
AMD’s MI355X Enters the Conversation
Modular’s pitch has never been Nvidia-only. The company positions MAX as hardware-portable, meaning the same container image can target Nvidia GPUs, AMD GPUs including the MI355X, and CPUs without separate serving code for each. That portability claim is where AMD enters the story, and it’s a meaningful one: most inference stacks are built and tuned around Nvidia’s CUDA ecosystem first, with AMD support arriving later or performing worse.
Modular’s published numbers here are narrower than the headline 5.5x TTFT figure. For FLUX.2 image generation specifically, Modular lists a starting price of $0.009 per image on MI355X versus $0.011 per image on Nvidia B200, a roughly 18% difference, with Black Forest Labs’ commercial licensing fee folded into both figures. That’s a cost-per-image comparison on one image-generation workload, not a discount on MAX software itself, and not a blanket AMD pricing cut across all workloads.
The AMD Discount Question: What’s Confirmed and What Isn’t
Some coverage circulating this week has framed Modular’s AMD story as a flat 30% discount. That specific figure does not appear in Modular’s own published blog post, its deployment pricing page, or the MLSys conference materials reviewed for this story. The closest documented comparison is the FLUX.2 cost-per-image example above, an 18% gap on one workload, not a general 30% software or hardware discount.
Modular does describe its dedicated-endpoint pricing as a straightforward dollars-per-minute rate, deployable on B200s, MI355Xs, or a mix of both, and it notes that standard cloud mechanisms like reserved capacity, committed-use discounts, and startup credits can apply to bring-your-own-cloud inference spend. Those are infrastructure-level savings any customer might negotiate, not an AMD-specific markdown tied to this announcement. Until Modular or AMD publishes a specific percentage tied to a date and a product, readers should treat a “30% AMD discount” framing as unverified.
MAX vs vLLM vs SGLang vs TensorRT-LLM: Where Each Stack Stands
Modular’s benchmark story actually spans two separate comparisons, and conflating them overstates the claim. The vLLM comparison uses B200 GPUs and the two named models above. A second comparison, against SGLang, runs on Nvidia B300 GPUs against models larger than 400 billion parameters, where Modular reports sub-500 millisecond mean TTFT, roughly 30% faster P99 end-to-end latency, and 22% faster mean end-to-end latency than SGLang. No comparable published number exists yet against Nvidia’s own TensorRT-LLM stack, which remains the default choice for teams fully committed to Nvidia hardware.
| Inference Stack | Maker | Hardware Targets | Modular’s Stated Comparison | Independent Verification |
|---|---|---|---|---|
| MAX | Modular Inc. | Nvidia, AMD, CPU | Baseline (claims 5.5x P50 TTFT vs. vLLM) | Not yet reproduced by a third party |
| vLLM | Open-source community / UC Berkeley origin | Primarily Nvidia | 5.5x slower P50 TTFT on Kimi-K2.5 per Modular’s B200 test | N/A (baseline in Modular’s own test) |
| SGLang | Open-source community | Primarily Nvidia | ~30% slower P99 end-to-end latency per Modular’s B300 test | Not yet reproduced by a third party |
| TensorRT-LLM | Nvidia | Nvidia only | No published Modular comparison found | N/A |
The table above reflects what Modular has put in writing, nothing more. vLLM and SGLang are both open-source projects with large contributor bases, and either could publish counter-benchmarks at any time, which is common when a vendor makes a splashy comparative claim.
Historical Context: How the Inference Stack Race Got Here
For most of 2023 and 2024, the industry’s performance conversation centered on model architecture and training compute. That shifted through 2025 and into 2026 as the cost of running models at scale overtook the cost of training them for many companies. vLLM became the default open-source serving layer for exactly that reason: it was free, good enough, and widely supported. SGLang carved out a niche in high-throughput production serving. Nvidia’s TensorRT-LLM stayed the performance ceiling for teams willing to lock into Nvidia hardware and accept the engineering overhead that comes with it.
Modular’s pitch since its founding has been different: write once, compile everywhere, using the Mojo language and the MAX runtime to avoid the vendor lock-in that TensorRT-LLM implies and the performance ceiling that general-purpose stacks like vLLM can hit. The B200 generation of GPUs, alongside rising interest in AMD’s MI300 and MI355X lines as a cost alternative to Nvidia, created the conditions for this argument to matter commercially rather than just academically. Cloud buyers increasingly want one deployment artifact that works wherever capacity happens to be available, especially as GPU shortages and price surges have made single-vendor dependence riskier.
Market Impact: Why Inference Speed Is Now a Boardroom Metric
Nvidia B200 cloud rental prices have already climbed sharply this year, and infrastructure buyers are under pressure to extract more useful work per GPU-hour rather than simply renting more hardware. A inference stack that genuinely delivers a multiple-x speedup on time to first token changes the economics of serving large MoE models like Kimi-K2.5 in production, because faster TTFT at the same hardware spend effectively lowers the per-request cost.
That’s also why the AMD angle matters commercially, independent of the unverified discount figure. If Modular’s compiler approach genuinely lets enterprises run the same workload on AMD MI355X hardware at comparable or lower cost than Nvidia B200, it gives buyers real leverage in price negotiations with Nvidia, which has historically enjoyed minimal competitive pressure on inference-grade GPUs. Whether that leverage materializes depends entirely on whether Modular’s numbers hold up outside its own test environment.
Benchmark Claims at a Glance
| Claim | Comparison | Hardware | Model(s) | Source |
|---|---|---|---|---|
| 5.5x faster P50 TTFT | MAX vs. vLLM | Nvidia B200 | Kimi-K2.5 | Modular blog, MLSys 2026 talk |
| 2.5x faster P99 TTFT | MAX vs. vLLM | Nvidia B200 | Gemma-4-31B-it | Modular blog, MLSys 2026 talk |
| 1.5x higher throughput | MAX vs. vLLM | Nvidia B200 | Both models above | Modular blog |
| ~30% faster P99 end-to-end latency | MAX vs. SGLang | Nvidia B300 | Models over 400B params | Modular blog |
| $0.009 vs. $0.011 per image | MI355X vs. B200 | AMD MI355X, Nvidia B200 | FLUX.2 image generation | Modular pricing page |
The Vendor-Reported Caveat: Reading These Numbers Critically
Every figure in the two tables above came from Modular’s own blog, pricing pages, or conference slide deck. No outlet reviewed for this story found an independent lab or competing vendor that has reproduced the 5.5x TTFT figure using its own test harness. That doesn’t make the number false, but it does mean the appropriate level of confidence is “a specific, named, conference-presented vendor claim,” not “an industry-verified benchmark.”
Engineers evaluating MAX for their own workloads should ask Modular directly for the missing methodology details: exact vLLM version, batch size, quantization format, and sequence length distribution used in the test. Those details determine whether a 5.5x gain generalizes to a different workload or was specific to the exact conditions Modular chose to publish.
What This Means for Enterprise AI Buyers Right Now
Teams running large MoE models like Kimi-K2.5, or comparably sized open-weight models such as Nvidia’s Nemotron 3.5, now have a documented reason to benchmark MAX against their current vLLM or SGLang deployment rather than assuming the incumbent stack is optimal. Even if Modular’s exact multiplier doesn’t reproduce one-to-one, a hardware-portable compiler stack that narrows the Nvidia-AMD performance gap is worth testing given how tight GPU supply has been.
Cost-sensitive teams evaluating alternatives have more options than a year ago. Smaller, cheaper decision models like Cloudflare’s Clef-omni show that not every workload needs a trillion-parameter model in the first place, which changes the inference-stack calculus entirely: a lighter model on a well-optimized stack can beat a heavyweight model on a mediocre one for many production use cases.
A Simple Way to Think About TTFT in Your Own Testing
Teams that want to sanity-check any inference stack’s TTFT claim, Modular’s included, typically time the gap between sending a request and receiving the first streamed token, repeated across many requests to get a distribution rather than a single sample. The pseudocode below illustrates the general measurement pattern used across the industry; it is not Modular’s actual benchmark harness, which the company has not published in full.
import time
def measure_ttft(client, prompt, n_requests=200):
ttfts = []
for _ in range(n_requests):
start = time.perf_counter()
stream = client.generate_stream(prompt)
first_token = next(stream) # blocks until first token arrives
ttfts.append(time.perf_counter() - start)
ttfts.sort()
p50 = ttfts[int(len(ttfts) * 0.50)]
p99 = ttfts[int(len(ttfts) * 0.99)]
return p50, p99Open Questions That Still Need Answers
Three things remain unresolved as of October 13, 2026: whether an independent party reproduces the 5.5x figure, whether Modular or AMD ever publishes a specific discount percentage tied to a date, and whether Modular benchmarks directly against TensorRT-LLM, the one major stack missing from its public comparisons. Any of those three developments would meaningfully change how much weight the market should put on this story.
Predictions: Where This Goes Over the Next Year
- Expect vLLM and SGLang maintainers to respond publicly with their own benchmark rebuttals or methodology requests within weeks, given how specific and public Modular’s claim is.
- Expect Modular to eventually publish a direct TensorRT-LLM comparison, since skipping Nvidia’s own stack in a credibility-building campaign is a gap competitors and customers will keep asking about.
- Expect more inference vendors to lean on AMD MI355X support as a negotiating chip against Nvidia pricing, following the pattern set by deals like AMD’s Helios rack rollout, regardless of whether Modular’s specific cost numbers hold up.
- Expect enterprise buyers to start requesting hardware-portable inference benchmarks as a standard RFP line item, rather than treating Nvidia-only deployment as the default assumption.
- Expect continued ambiguity on Modular’s funding and valuation, since no reliable public figures currently exist, which will keep questions about the company’s long-term staying power open even as its technical claims draw attention.
The Bigger Picture for the Inference Market
Whether or not Modular’s 5.5x number survives independent scrutiny, the underlying trend it points to is real: the serving layer has become a genuine competitive battleground, not an afterthought bolted onto whichever model a company happens to use. A year ago, the open-source default of vLLM was rarely questioned by teams without a dedicated infrastructure group. Now, a named chief scientist is standing on a systems-conference stage with specific numbers attached to a specific model, and that alone signals how much money is riding on shaving milliseconds off time to first token at scale.
Frequently Asked Questions
What is Modular’s MAX inference stack?
MAX is Modular Inc.’s compiler and runtime platform designed to run AI inference workloads across Nvidia GPUs, AMD GPUs, and CPUs from a single deployment artifact, rather than requiring separate serving code for each hardware vendor.
What did Modular actually claim about Kimi K2.5?
Modular says MAX achieved 5.5x faster P50 time to first token than vLLM when serving Kimi-K2.5 on Nvidia B200 GPUs, a figure presented by chief scientist Abdul Dakkak at MLSys 2026 and republished in the company’s June 12 blog post.
Is there really a 30% AMD discount tied to this announcement?
No specific 30% discount figure appears in Modular’s published blog post, pricing page, or conference materials. The closest documented AMD comparison is an 18% cost-per-image gap between MI355X and B200 pricing for FLUX.2 image generation, which is a workload-specific example, not a general discount.
Who made Kimi K2.5?
Kimi-K2.5 is a mixture-of-experts model from Chinese lab Moonshot AI, released January 27, 2026, with roughly 1.04 trillion total parameters and about 32 billion active parameters per token.
Has anyone independently verified Modular’s 5.5x benchmark?
Not as of this writing. The claim originates entirely from Modular’s own blog post and its MLSys 2026 conference presentation. No independent lab or rival vendor has published a reproduction using its own test setup.
What is P50 vs. P99 time to first token?
P50 TTFT is the median wait time before a user sees the first generated token, representing a typical request. P99 TTFT is the 99th percentile, representing the slowest 1% of requests, which is often where inference stacks reveal scheduling or memory-management weaknesses.
Does MAX support AMD GPUs today?
Yes. Modular’s own materials confirm MAX support for AMD GPUs, specifically naming the MI355X, alongside Nvidia GPU and CPU deployment targets.
How does MAX compare to TensorRT-LLM?
There is no published Modular benchmark comparing MAX directly against Nvidia’s TensorRT-LLM stack. Modular’s public comparisons so far cover vLLM and SGLang only.
