📝 摘要 #

本文系统分析了 LLM 推理行业过去两年中价值链的迁移路径：从推理引擎的软件护城河，到服务运营与平台能力，再到资本、GPU 容量和最终的数据中心所有权。文章以 NVIDIA 三代硬件（Ampere/Hopper/Blackwell）为线索，论证推理引擎层正在快速商品化——每一代硬件的优化清单从"差异化优势"变为"公开知识"的速度越来越快，Blackwell 时代更是将 NVFP4 优化收敛到 TRT-LLM 参考栈一条路。编码智能体进一步压缩了专有优势的独占时间窗口。开源引擎（vLLM、SGLang）实质上成为公共品，引擎公司自身也在向上游服务层迁移。文章最终预测：推理服务商终将收购数据中心，行业竞争从软件比拼变为资本密集型基础设施博弈，而拥有物理资产的 neocloud（如 Nebius）从底部向上攀登的速度，将快于软件公司从顶部向下建设资产的速度。

核心概念： LLM 推理引擎层正在商品化，可防御的价值已从软件内核迁移到服务运营、资本和物理基础设施；推理行业的终局是资本密集型基础设施竞争，而非软件竞争。

关键技术栈 / 关键要点：

1. 三代硬件清单收敛 — Ampere（CUDA Graphs + EAGLE + INT8）→ Hopper（FlashAttention-3 + FP8 + PD 分离 + EP）→ Blackwell（NVFP4 优化，TRT-LLM 成为唯一路径），每代清单从"差异化"到"常识"的速度递增
2. TRT-LLM 成为 Blackwell 基线 — NVIDIA 的商业激励（硬件 benchmark、launch-day 性能）使 TRT-LLM 在新硬件上拥有不可复制的首日优势，独立引擎的 make-or-buy 天平已倾向 buy
3. 编码智能体加速优势消散 — Intent Lab 用一周 agent 辅助完成原本需一季度的优化，使专有优势的有效生命周期从"月"缩短到"周"
4. 开源引擎的公共品困境 — vLLM/SGLang 持续提升免费基线，但每提升一次就压缩了自身和其他引擎的差异化空间
5. Day-0 支持的关系护城河 — 模型实验室越来越多地将推理服务商而非开源引擎作为首发合作伙伴，早期访问权是关系而非技术，不可复制
6. 服务层的终局是资本游戏 — 资本→GPU 容量→大合同→ARR→融资→更多容量的飞轮，决定规模的是资产负债表而非内核优化
7. 向上游的终局：数据中心 — 推理服务商将购买数据中心和电力合同，从软件公司变为基础设施公司；Nebius 等已有资产者从底部向上攀登比软件公司从顶部向下建设更快

关键结论 / 实践建议： 推理行业竞争的核心已从"谁的引擎更快"转向"谁有资本和容量来承接大合同"；对工程师而言，开源引擎释放的时间应投向基础设施可靠性、平台体验和 RL 基础设施，而非继续在引擎层重复造轮子。

LLM Inference — Past, Now, and Future: Where Does the Value Move? #

For the past two years, LLM inference has been one of the most contested layers in AI infrastructure. Dozens of inference providers, GPU clouds, open-source projects, and chip vendors have all chased the same goal: serve a trained model faster and more cheaply than the next company.

The attraction rested on a plausible idea. Inference looked like a software problem with a software moat. Better kernels, a cleverer scheduler, or a stronger speculative decoding could support premium pricing or better margins at the same price. For a time, an in-house engine was a real competitive weapon, and benchmarks won deals.

That period is ending. Inference matters more than ever, and the market continues to grow, but defensible value has moved away from the engine. The engine layer is commoditizing quickly. Value is shifting first into serving operations and platforms, then toward capital, GPU capacity, and eventually the data centers themselves.

Software performance still counts. A slow or unreliable engine can disqualify a provider. Good performance, however, has become widely available, making it harder for any one company to charge for it. The question is no longer whether the engine creates value, but who captures that value once the engine becomes common infrastructure.

Three hardware generations make the shift visible. The same pattern explains the competition among inference providers today and where it is likely to go next.

Part I: The Past — When the Engine Was the Moat #

Three generations, three checklists

Each NVIDIA hardware generation came with a checklist for an in-house engine. What changed was how quickly the checklist became public knowledge, and how little advantage remained once everyone had completed it.

The Ampere era (A100). The early bar was concrete. An engine that supported CUDA Graphs to remove launch overhead, implemented speculative decoding such as EAGLE-1 or Medusa, and shipped solid W8A8 INT8 quantization stood ahead of most of the field. The engineering was difficult but bounded, and completing that short list put a provider in the top tier. Those features won deals.

The Hopper era (H100/H200). The list grew and split in two. For a single deployment, whether one replica, one node, or a few, the differentiators were FlashAttention-3, FP8 attention, EAGLE-3 speculative decoding, and W8A8 FP8 quantization. Strong implementations produced standout single-node results.

Hopper also opened a second front in disaggregated deployment. Support for prefill–decode (PD) disaggregation, expert parallelism (EP) for the increasingly dominant MoE architectures, and KV cache offload across the memory hierarchy mattered at cluster scale, where the largest contracts lived. This tier demanded systems engineering as well as kernel work. For a while, it separated the strongest providers from everyone else.

The two tiers rewarded different capabilities. Kernel teams could still win a benchmark on a contained deployment, while the largest production workloads demanded coordination across machines, memory pools, and failure domains. That second tier took longer to copy and gave providers a larger window in which to turn engineering work into revenue.

The Blackwell era (B200/B300/GB200/GB300). Here the checklist has narrowed to one main item: NVFP4 optimization. The leading route is no longer an independent implementation. Teams integrate the trtllm-gen cubins shipped by NVIDIA or build on TensorRT-LLM directly. Speculative decoding, FP8 and FP4 paths, disaggregation support, and MoE parallelism are already in the reference stack.

Blackwell changes the make-or-buy decision. Building the stack once demonstrated technical depth; now it can amount to recreating vendor work while competitors spend the same engineers elsewhere. A proprietary implementation may still suit an unusual model or deployment, but it is no longer the default path to leading performance.

Inference has no secrets. Every important technique has a paper, an open-source implementation, or a vendor binary. Knowledge that once circulated through a handful of performance teams is packaged into code that any capable group can inspect or integrate. The checklist still qualifies an engine, but no longer sets it apart.

Why TRT-LLM became the Blackwell baseline

TensorRT-LLM's position in the Blackwell era follows from the incentives on both sides of the market.

NVIDIA needs TRT-LLM to perform well. The software underpins its new-hardware benchmarks, including InferenceX submissions, launch-day claims, and keynote results. Optimizations for a new chip therefore arrive in TRT-LLM on day one, backed by a large kernel-engineering organization and a capable runtime.

Independent engines cannot reproduce that advantage through effort alone. NVIDIA sees the hardware roadmap, controls the lowest software layers, and has a direct commercial reason to make each generation look strong at launch. TRT-LLM is where those incentives meet.

At the same time, a handful of frontier open-weight model families now account for the overwhelming majority of serious production traffic. Providers have less need to support hundreds of architectures. Given that narrower model set and Blackwell hardware, TRT-LLM offers the highest available performance ceiling. Breadth of model support, the traditional case for a general-purpose engine, matters less when demand itself has narrowed.

Production now rewards specialization. An engine that handles a long tail gracefully is useful, but a provider earns most of its revenue from the models customers actually request. On a small matrix of popular models and current NVIDIA hardware, peak performance outweighs architectural breadth.

Since mid-2025, more inference teams have stopped maintaining fully independent engines and moved to secondary development on TRT-LLM. Its persistent weakness is usability. The developer experience is rough, but a team paid to extract the last 20 percent from a GPU fleet will tolerate a difficult toolchain. Usability breaks ties; TRT-LLM on Blackwell has no equals to break ties with.

Those teams have not stopped engineering. They still tune models, patch runtime behavior, and build the production systems around the engine. What changed is the layer at which they begin. Starting from NVIDIA's baseline directs more effort toward their workload and less toward reproducing general machinery.

The accelerant: coding agents

Open-source convergence and NVIDIA's vertical push were already shortening the life of proprietary advantages. Over the past six months, coding agents have shortened it further by reducing the cost of inference engineering.

Kernel work, runtime changes, and serving infrastructure can all be completed faster with AI assistance. Intent Lab spent about a week, working agent-assisted on TRT-LLM, to ship optimizations that produced very large end-to-end gains. Previously, work of that scope might have occupied a dedicated engineer for a quarter.

The economics of proprietary engine work change with that speed. A technique that required six engineer-months and bought nine months of exclusivity could justify the investment. If it takes two weeks to build and competitors reproduce it in three, the result is not a moat; it is a treadmill. The work remains technically hard, but the useful life of the advantage is approaching zero.

What matters economically is how long the lead lasts. A difficult optimization can still be commercially weak if it diffuses before the company recovers the cost of building it. Coding agents do not make the engineering trivial; they make exclusivity expire faster.

What's left for the open engines: community, usability, and very little loyalty

vLLM, SGLang, and other open-source engines serve the market left open by TRT-LLM's rough developer experience. Many users outside providers with dedicated inference teams are researchers or run offline generation: throughput-oriented batch workloads rather than latency-sensitive online serving. In those settings, the performance gap with TRT-LLM is modest, often negligible.

Those users optimize for a different workflow. They need to bring up a model quickly, change architectures without rewriting the stack, and find answers when something breaks. A few percentage points of throughput rarely justify days spent fighting a runtime, especially when the workload has no interactive latency target.

Adoption turns instead on ease of use, documentation, and community. Install the package, point it at a Hugging Face repository, and expose an OpenAI-compatible endpoint. For a researcher or a batch-generation pipeline, that is the whole purchase decision.

Adoption is not loyalty. Standardization on the OpenAI-compatible API reduces an engine switch, in the simplest case, to a change in base_url. A team can run vLLM today, try SGLang tomorrow, and benchmark both later in the week. Open engines must keep competing for workloads they have already won.

Users benefit from that portability; projects seeking durable control do not. Community size can attract a workload, and documentation can keep it for a while, but neither prevents a team from rerunning the comparison when a competitor releases a faster version.

The real switching costs appear in online production. They come from monitoring and alerting, deployment pipelines, disaster recovery, automated failure recovery, accumulated bug fixes, and the long tail of production edge cases. Tool-call parsing alone supplies plenty of them. This operational layer creates lock-in, but it is migration friction rather than a capability moat. A competent team can reproduce it around another engine in weeks. More important, that knowledge belongs to the user's SRE organization, so the engine project captures none of it.

Commercially, operational knowledge makes a deployment sticky without giving the engine vendor pricing power. The user bears the migration cost and owns most of the surrounding systems. Even where replacement is annoying, the engine itself has not secured the account.

Open source as a public good

The major open-source engines have largely accepted this role. vLLM and SGLang return nearly all of their work to the community. Their strategic aim is adoption; the result is a steadily better free baseline in features, performance, and stability. In practice, the ecosystem is subsidizing state-of-the-art inference for everyone.

There is a cost to the projects themselves. Each improvement to the baseline shrinks the room for differentiation, including for in-house engines and the open engines that made the improvement. By raising the floor, these projects also compress the value of their own layer.

The tell: engine companies are moving up the stack

The behavior of engine authors offers the clearest evidence that an engine alone cannot capture much value.

The companies behind the two leading open-source engines have begun taking production serving work, reportedly signing inference contracts with prominent model labs and consumer platforms. Their stated reason makes sense: running production is the fastest way to expose the failures an engine must handle. Dogfooding at scale finds the corner cases.

It also puts those companies in competition with providers that were once their most important users and advocates. The project drives adoption, while serving contracts produce revenue. Those roles are difficult to reconcile when the users of the project sell the same service.

Engine companies are understandably careful about the provider label. Their ecosystem depends on companies that want a neutral upstream project, not a subsidized competitor. Yet once the engine company operates production for customers, the overlap is real regardless of how the work is described.

Moving into serving says more than the accompanying explanation. Engine authors do not expect the engine layer to support a business by itself. The question is whether serving is any more defensible.

Part II: The Present — What Actually Distinguishes Providers #

The incumbents' real advantages have nothing to do with kernels

The durable advantages of leading inference providers such as Together AI, Fireworks, and Baseten do not appear on a engine benchmark.

First-mover advantage and brand. When a model lab needs a launch partner or an AI-native startup needs production inference, these companies make the first shortlist. Mindshare sounds soft until it decides contract after contract. Default consideration in a fast market is worth more than a narrow benchmark lead.

The platform. Years of work have accumulated in deployment tooling, observability, enterprise controls, and compliance. A new entrant must rebuild that surface one piece at a time while incumbents continue to extend it.

None of these features wins a public speed chart, but together they decide whether a customer can move a workload into production. They also compound. Each deployment exposes another missing control or failure mode, and the fix becomes part of the platform offered to the next customer.

GPU capacity and its supply chain. This is the hardest advantage and the least discussed. Providers must secure allocations across hardware generations, negotiate with clouds and neoclouds, manage heterogeneous fleets, and plan capacity against uneven demand. Incumbents have learned to do this under load. When a large contract arrives, the deciding question is often not whose engine is faster, but who can bring thousands of GPUs online next month.

Late entrants are responding to those economics. Modal has launched an inference service. Nebius acquired Eigen AI to add serving to its cloud. Inference carries better margins than raw GPU-hours, so GPU clouds are moving up while engine companies move into serving from below. Providers, clouds, and engine companies are converging on the same layer because that is where value currently collects.

Each group begins with a different advantage. Engine companies bring software expertise, GPU clouds bring capacity, and established providers bring customers and operating experience. Their movement toward the same product makes the remaining differences easier to see: distribution, capital, and the ability to run a reliable service at scale.

Why small-batch TPS rankings will fade

The small-concurrency output-speed ranking on Artificial Analysis still attracts attention, and for years it was a fair proxy for engineering quality. It says less with each hardware cycle. The standard API tests use either one request or ten parallel requests. At that load, a provider can combine newer hardware with load-aware speculative decoding such as DSpark, spend more batch capacity on each request while the machine is otherwise idle, and post extraordinary per-user TPS. The number is real but narrow: it shows how fast a lightly loaded endpoint can emit one user's tokens, not how efficiently a fleet serves a business.

Production has a different objective. First, keep per-user TPS above the level the application requires. Then maximize total tokens per minute per GPU (TPM/GPU) without dropping below that floor. Once the user experience is fast enough, another increment of lightly loaded TPS may be worth far less than serving more simultaneous users on the same GPU. Cost per delivered token matters more than the headline rank.

Speculative decoding sharpens the distinction. Verification work that is cheap at low batch can consume valuable batch capacity under concurrency, so a configuration tuned for the highest lightly loaded TPS need not sit on the best production cost curve. The useful result is a Pareto frontier: per-user TPS on one axis and TPM/GPU on the other, with the application's speed floor selecting the operating point and cost per token following from it.

Small-batch TPS is not useless. It establishes an interactivity floor and exposes endpoints that are plainly too slow. Artificial Analysis's own ramped-concurrency testing points in the more useful direction, measuring what the system as a whole can sustain. What will fade is the winner-takes-all reading of the small-batch ranking. Production buyers will care less about who posts the highest TPS and more about how much paid traffic each GPU carries while per-user TPS remains above the required floor. It is the essay's pattern in miniature: the most visible number stops predicting where the money goes.

Day-0 support: the relationship moat tightens

A change in 2026 has strengthened the incumbents further: model developers increasingly partner with inference providers directly.

Day-0 support once ran through open-source engines. Before a launch, a lab would coordinate with vLLM or SGLang; the engine would merge support; providers downstream would pick it up. This gave open engines a central place in the distribution chain.

The old dependence is fading. Major labs now grant inference providers early access before release, sometimes ahead of open-source engine teams. A lab needs more than a merged pull request on launch day. It needs tuned and load-tested production capacity with an SLA. A provider can deliver the package; an engine supplies only its first component.

Early access is a relationship rather than a published technique, so competitors cannot reproduce it from a paper or a kernel. Labs grant it to partners they already trust, reinforcing providers that already have brand and capacity. Open-source engines, meanwhile, are losing their place at the front of the release pipeline. Providers increasingly lead Day-0 work and engines follow.

A successful launch makes a provider more likely to receive early access for the next one, and early access improves its odds again. Unlike an optimization, the relationship does not become available to the rest of the market after publication.

The endgame at the serving layer: a capital game

Once competitors reach the serving layer, they face a common bar. A credible provider must hold its SLA under pressure, deliver top-tier latency, and preserve accuracy. These are difficult operational requirements, and they separate providers from GPU resellers. They are also table stakes: meeting them grants entry, while missing them eliminates a provider.

Shared engines and public techniques pull serious providers toward comparable performance, reliability, and pricing. Above that line, GPU capacity becomes the deciding variable. In other words, the contest turns into a contest for capital.

The flywheel is direct. Capital secures GPU capacity. Capacity enables larger model-lab and enterprise contracts that constrained competitors must decline. Those contracts produce inference volume and ARR. Higher ARR supports a larger financing round and valuation, which pays for the next block of capacity.

A ready fleet also changes the sales conversation. A customer choosing a provider for a major launch cannot rely on capacity that may appear later. It needs machines reserved, networked, and ready. A provider that can commit those resources has an advantage before latency and token price enter the negotiation.

For engineers, this is an uncomfortable outcome. The serving endgame looks less like a software competition and more like capital-intensive infrastructure, closer to early cloud computing, airlines, or telecoms. Operational excellence determines survival; access to capital determines scale.

Operations still separate viable companies from failed ones. Poor utilization, weak reliability, or bad capacity planning can destroy a well-funded provider. Once every serious competitor meets the technical bar, though, another kernel improvement cannot substitute for the balance sheet required to accept the next large contract.

The market has already moved this way. Leading providers have raised large rounds and seen their valuations multiply over the past year. Investors are paying for position in a capacity land grab, not for kernel engineering.

Part III: The Future — Value Sinks Into Concrete #

Prediction: they will all buy data centers

The capital logic points toward ownership. At sufficient scale, a company whose cost of goods is dominated by GPUs and electricity loses margin whenever it rents compute or data-center space. Each rented layer transfers part of the gross margin to another balance sheet.

Leading inference providers will therefore buy or acquire data centers, including the facilities and the power contracts beneath them, rather than stopping at GPU allocations or long-term cloud commitments. The war chests they have raised make sense under no other endgame.

The inference provider began as a software company whose engine was the product. It became a managed-service company selling a platform and an SLA. It is now becoming an infrastructure company selling capacity. Across that progression, value has moved from kernels to serving operations and then to steel, land, and power-purchase agreements.

Each step changes the skills and economics of the company. Software iteration gives way to fleet management, procurement, financing, and energy strategy. The interface may remain an API, but the business underneath it becomes heavier and more exposed to utilization.

The story began with the loss of a software moat. It closes with the same companies competing for the oldest moats in the world: land, energy, and capital. The moat was never really in the software.

Still "inference providers," never "neoclouds"

Even after buying data centers, these companies will continue to describe themselves as inference providers rather than neoclouds. Capital markets award a higher multiple to an AI infrastructure company that sells tokens than to a neocloud that rents GPU-hours.

Under the label, they will resemble a new kind of neocloud: the same asset-heavy foundation with a different interface, selling tokens, SLAs, and APIs instead of bare GPUs. The CoreWeaves of the world built the asset base first and are moving toward serving. The Fireworks of the world built the interface first and are moving toward the asset base. Both paths lead to the same corporate shape; the starting point determines the label and the multiple attached to it.

Selling through an API changes the packaging. Tokens bundle software, operations, and capacity into a product a customer can consume directly. GPU-hours expose more of the underlying commodity. Two companies may own similar assets yet receive very different valuations because one has packaged those assets higher in the stack.

They will own the data centers and still call themselves inference providers because the label is worth more than the building.

The real competitor is coming from below

If the destination is a capital-intensive, data-center-owning token business, ranking the current players by their distance from it changes the competitive picture.

Together AI, Fireworks, and Baseten have brands, platforms, and contracts. They still need to build or buy an asset base of facilities, power, and a neocloud-scale supply chain, all constrained by multiyear construction and power lead times.

Nebius already owns data centers and operates its own GPU supply chain. As a public company, it also has a financing channel that private providers can approximate only through repeated large rounds. Its missing piece was the serving layer. The acquisition of Eigen AI brought in software and a team, and a focused year closes most of the software gap because inference has no secrets.

By contrast, a software-first provider cannot compress the physical timetable in the same way. Power interconnects, construction, and equipment delivery follow multiyear schedules. Capital can secure a place in those queues, but it cannot turn them into a software release cycle.

Software can be added to an asset base much faster than an asset base can be added to a software company. Engine commoditization helped providers grow, but it also equips their most dangerous challenger.

The biggest rival facing Together AI, Fireworks, and Baseten is not another provider. It is Nebius, followed by any neocloud willing to climb the stack.

Where the engineering hours go instead

Inference engineering will still matter inside providers, but its allocation will change. Open-source engines make that reallocation possible.

As the free engine baseline improves, providers can move expensive engineering time out of kernels and runtimes. The hours go into infrastructure reliability, which gives an SLA substance; platform experience, which supports renewals; and increasingly RL. Reinforcement-learning rollouts are inference-intensive, so serving expertise transfers directly into RL infrastructure for the growing post-training market.

Providers are not retreating from technical work. Reliability across a heterogeneous fleet, fast recovery under load, and efficient RL rollouts are difficult systems problems. Their solutions remain closer to the provider's customers and operations, making them more useful as sources of differentiation.

The value in the engine layer has not vanished. Open source has folded it into a free foundation, allowing engineering value to migrate into the systems built above it. The strategic question for most companies is no longer whether to build an engine. It is where to spend the time that a free engine saves. The beneficiaries of open-source public-goods work show up in other companies' roadmaps.

The story is not over

The endgame remains unsettled. Vera Rubin, MI455, LPU, and other inference-first hardware will reset the checklist, as every previous hardware transition has done. Each reset briefly reopens room for engine differentiation through new numeric formats, memory hierarchies, and parallelism trade-offs. The techniques then spread and the window closes.

New model architectures and workloads will create similar openings. Agents with large reusable contexts, and RL at production scale all impose new constraints. Engine commoditization is a recurring cycle. The interval between invention and diffusion is shrinking, but it has not disappeared.

During each opening, a fast team can still win meaningful business. Temporary advantages matter when they arrive alongside a major hardware or model transition, because the revenue and customer relationships can persist after competitors catch up. What has changed is the time available to recognize and exploit the opening.

Engines and providers still face two tests. The first is reaction speed: can they catch a transition while the advantage remains available? Every generation has left behind well-run companies that arrived one hardware cycle late. Teams that moved early on disaggregation or Blackwell's NVFP4 economics earned temporary but compounding gains.

The second test is financial: can they preserve margin while ARR grows? A provider can book more revenue and lose more money by buying growth with underpriced capacity. As competition tightens and gross margins approach infrastructure economics, operational discipline becomes a condition of survival. Fleet utilization, power costs, demand forecasting, capacity planning, and the balance between committed and spot revenue will decide the next group of winners. Raising billions is only the beginning; spending those billions well is the craft.

Value in this market keeps moving: from models to engines, from engines to serving, and from serving toward capacity and power. It leaves layers where no durable moat remains. The next winners will recognize the shift before the checklist becomes common knowledge.