Every ML engineering team I know has a love-hate relationship with NVIDIA. The H100 is the gold standard for training and inference, but at over $30,000 per GPU — and often more on the secondary market — it strains budgets. AMD's MI300X offers comparable memory and compute at a lower sticker price, but its software stack has lagged. Yet a growing number of teams are discovering that splitting inference across both vendors can cut total cost by a third. This isn't a theoretical exercise. It's happening in production clusters right now, and the economics are compelling.

The $100K GPU Duel No One Talks About

NVIDIA's H100 dominates the AI hardware market, but its price tag — often exceeding $30,000 per unit — makes large-scale deployment a serious financial commitment. AMD's MI300X, with 192 GB of HBM3 memory and comparable FP16 performance, costs roughly 20% less in volume purchases. The gap widens when you factor in supply constraints: NVIDIA's allocation delays have pushed spot prices even higher, while AMD has been more readily available. For instance, a mid-sized AI startup recently reported waiting over six months for a 500-GPU H100 order, while an equivalent MI300X order shipped in under two months. That delay cost them an estimated $1.5 million in lost productivity — a hidden cost that never appears on a GPU spec sheet.

Split inference — the practice of distributing neural network layers across different GPU architectures — lets teams mix and match. A cluster might run the first few layers on an H100 for precision-sensitive attention, then offload the remaining feed-forward layers to an MI300X. This approach reduces the average cost per GPU-hour without sacrificing overall throughput. The key insight is that inference workloads, especially for large language models, are memory-bound rather than compute-bound. Both GPUs have similar memory bandwidth, so the performance gap narrows. In one benchmark, a 50/50 split of Mixtral 8x7B inference across H100 and MI300X achieved only 8% lower throughput than an all-H100 cluster — while cutting GPU cost by 34%.

Vendor lock-in is the hidden risk. Teams that standardize on NVIDIA lose negotiation power and face higher prices at renewal. A dual-source strategy introduces operational complexity but shifts the leverage balance. AMD knows it's the underdog and is willing to offer aggressive pricing to win footholds. Some large buyers have reported 25% discounts on MI300X bundles that include support contracts, effectively making the per-chip cost roughly $20,000 versus NVIDIA's $30,000.

Why Split Inference Works on AMD Too

The common objection is software maturity. AMD's ROCm stack has historically been less polished than CUDA, with fewer optimized libraries and spotty PyTorch support. But recent releases — ROCm 6.x and the Triton compiler — have changed the game. Triton abstracts kernel writing from the underlying architecture, letting developers write once and run on both NVIDIA and AMD GPUs. This is not a theoretical promise; several teams have reported successful deployment of Triton-based models on MI300X with minimal code changes, achieving within 90% of the performance of hand-tuned CUDA kernels.

Model parallelism naturally lends itself to split inference. Each GPU handles a subset of layers, communicating activations via high-bandwidth interconnects — NVLink on NVIDIA, Infinity Fabric on AMD. The real bottleneck is cross-vendor interconnect, which currently relies on PCIe or Ethernet. That adds latency, but for latency-tolerant batch inference, the hit is acceptable. For example, a team serving a 70B-parameter model for offline summarization found that using PCIe-based communication between H100 and MI300X added about 5 milliseconds per token, which was well within their 200ms latency budget for batch sizes of 32.

Mixtral 8x7B, a mixture-of-experts model, is a strong candidate. Its experts can be distributed across GPUs independently. Early adopters report running the model on a mixed cluster at 30% lower cost per token compared to an all-NVIDIA setup, with throughput degradation under 10% for batch sizes above 16. Another team working with a custom 13B-parameter dense model observed similar savings, though they needed to tune the split ratio to account for the MI300X's larger memory — it could handle more layers per GPU, reducing cross-vendor communication.

ROCm's open-source nature reduces DevOps overhead. Teams can debug driver issues without NVIDIA's proprietary black box. AMD also contributes to the MLIR-based IREE compiler, which optimizes model graphs for both architectures. The ecosystem is not yet as seamless as CUDA, but it's good enough for production inference — and getting better each quarter. For instance, AMD's recent contributions to the ONNX Runtime have closed the performance gap in common operators like LayerNorm and Softmax, which previously ran 20% slower on ROCm.

The Hidden Costs of Homogeneous Clusters

Sticking with a single vendor creates hidden costs beyond the GPU sticker price. NVIDIA's supply constraints have led to allocation games: teams wait months for H100s, or pay premiums on the gray market. AMD's MI300X has been more consistently available, reducing project delays and the cost of idle engineering time. A financial services firm running LLM-based fraud detection reported that a 4-month delay in H100 delivery forced them to use cloud instances at 3x the cost of on-prem, erasing any hardware savings. In contrast, a mixed fleet with AMD as a fallback could have mitigated that risk.

Power and cooling differ by architecture. The H100 has a TDP of 700W, while the MI300X sits at 750W. But AMD's chip can be undervolted more aggressively in inference workloads, reducing power draw by up to 15% with minimal performance impact. Over a 1,000-GPU cluster, that translates to roughly $100,000 in annual electricity savings. Additionally, AMD's memory can be clocked down when not fully utilized, further reducing power. Some teams have reported that in practice, the MI300X consumes about 10% less power than the H100 under typical inference loads, despite the higher TDP rating.

Single-vendor negotiation weakens leverage. When you only buy from NVIDIA, they know you have no alternative. A second-source clause in procurement contracts — promising a portion of spend to AMD — has been shown to reduce per-unit pricing by 15–20% from both vendors. The threat of switching is real, even if you never fully exercise it. One hyperscaler's procurement team told me they saved $4 million on a 2,000-GPU order simply by including AMD in the RFP, without ever intending to buy a single MI300X.

EOL risk is another factor. NVIDIA's H100 successor, the B100, is expected soon, and the H100's resale value may drop sharply. AMD's CDNA 4 architecture promises better FP8 support and improved memory bandwidth, potentially closing the gap further. Teams that invest in a mixed fleet today can transition more smoothly to next-gen hardware. For example, if CDNA 4 delivers on its promise of 2x FP8 performance, a mixed fleet with existing MI300X infrastructure can upgrade only the AMD portion, while NVIDIA users may need to replace entire clusters.

There's also the operational overhead of managing two driver stacks and monitoring tools. But many teams already run multi-cloud; a multi-vendor on-prem cluster is a similar skill set. The savings often outweigh the complexity. One team estimated that the additional DevOps effort added about $50,000 per year in engineer time, but the hardware savings exceeded $500,000 annually. That's a 10x return on the complexity investment.

Real-World Economics: A 1000-GPU Case Study

Consider a hypothetical but realistic 1,000-GPU cluster used for batch inference and fine-tuning. An all-NVIDIA configuration of 1,000 H100s costs roughly $30 million upfront. An all-AMD configuration of 1,000 MI300X costs around $24 million. But a mixed fleet — 600 H100s and 400 MI300Xs — costs $18 million plus $9.6 million, totaling $27.6 million, a savings of $2.4 million (8%) on hardware alone.

The real savings come in operational costs. With a mixed fleet, reserved instance pricing from cloud providers drops because you can commit to both vendors. On-prem, the ability to route latency-sensitive workloads to H100s and batch inference to MI300Xs reduces the number of expensive H100s needed. Annual savings on electricity, cooling, and maintenance can reach $1.2 million. For instance, if the MI300X nodes use 10% less power, that's roughly $80,000 saved annually on a 400-GPU cluster. Maintenance contracts for AMD are typically 15% cheaper than NVIDIA's, adding another $50,000 in savings.

Throughput impact is modest. For a typical LLM serving pipeline with batch sizes of 32, the mixed cluster achieves 92% of the all-H100 throughput. Since inference is often over-provisioned to handle spikes, the extra capacity from 400 MI300Xs compensates. Total cost per token falls below $0.0001, compared to $0.00015 for the all-NVIDIA cluster. Over a year of serving 1 trillion tokens, that difference amounts to $50,000 — not huge, but combined with hardware savings, the total cost of ownership is about 25% lower for the mixed fleet.

Pre-training large models remains NVIDIA's stronghold due to CUDA's mature distributed training libraries — NCCL, Megatron-LM, and NeMo. But fine-tuning and serving are AMD's sweet spot. These workloads are less communication-intensive and benefit from the MI300X's larger memory, which can hold bigger batches. A team fine-tuning a 7B-parameter model found that the MI300X could handle a batch size of 128 versus 96 on the H100, thanks to its 192 GB memory. That meant fewer gradient accumulation steps and faster convergence, partially offsetting the raw compute speed difference.

Negotiating with Two Vendors Changes Everything

Procurement teams have discovered that a dual-source strategy transforms vendor relationships. When AMD knows it's competing with NVIDIA for a slice of your budget, it offers 20% lower per-chip pricing. NVIDIA, in turn, matches or improves its service-level agreements to retain share. One large-scale buyer reported that simply mentioning AMD in a negotiation reduced NVIDIA's quote by 12%. Another firm secured a 3-year support contract from AMD at no extra cost after showing them a competing NVIDIA offer.

Service-level agreements become competitive. NVIDIA historically offered limited uptime guarantees on on-prem hardware. With AMD offering 99.9% uptime SLAs on its ROCm stack, NVIDIA has started matching those terms. Reserved instance discounts grow by 15% when you commit to a two-vendor mix, because both vendors want the guaranteed revenue. In one case, a cloud provider offered a 20% discount on reserved H100 instances after the buyer committed to also using AMD instances for 30% of their workload.

A second-source clause protects against allocation delays. If NVIDIA can't deliver H100s on schedule, you shift more spend to AMD without disrupting operations. This flexibility is valuable given the persistent supply-demand imbalance in AI hardware. For example, a startup that had ordered 200 H100s with a 6-month lead time was able to divert half the order to MI300Xs when NVIDIA pushed the delivery to 9 months. They received the AMD GPUs in 3 months, keeping their training pipeline on track.

Ownership of the software stack also shifts. With CUDA, you depend on NVIDIA's proprietary drivers and libraries. With ROCm, you can inspect source code, patch issues, and even contribute fixes. That level of control reduces risk of vendor abandonment or forced migration. One team discovered a performance bug in ROCm's GEMM kernel and fixed it themselves, gaining a 5% performance improvement — something impossible with CUDA's closed source.

Three Rules for a Cost-Effective Mixed Fleet

First, keep training workloads on NVIDIA's tensor cores. The H100's Transformer Engine and FP8 support provide a 2x speedup over AMD for large-scale training. Trying to train a 70B-parameter model on a mixed cluster introduces too much communication overhead. Save AMD for inference and fine-tuning.

Second, route inference through AMD for latency-tolerant tasks. Batch processing, offline evaluation, and asynchronous serving are ideal. For real-time applications with sub-100ms latency requirements, the cross-vendor interconnect adds too much jitter. Use NVIDIA for those paths. For instance, a chatbot serving team routes all user-facing requests to H100s, while nightly batch summarization jobs run on MI300Xs.

Third, use ONNX Runtime or TensorRT for cross-platform optimization. ONNX Runtime's execution providers for CUDA and ROCm let you deploy the same model on both architectures with minimal code changes. TensorRT can optimize for NVIDIA, while AMD's Composable Kernel library provides similar optimizations. Monitor utilization religiously; idle GPUs waste the savings you fought for. One team implemented a simple scheduler that routes requests based on current utilization, ensuring no GPU sits idle for more than a few seconds.

Fourth, plan for next-gen. AMD's CDNA 4, expected in 2026, may include hardware support for FP8 and improved memory bandwidth, potentially closing the gap with NVIDIA's B100. A mixed fleet today builds the operational muscle to adopt whichever architecture offers the best price-performance tomorrow. For example, if CDNA 4 delivers 2x the inference throughput of MI300X, you can upgrade the AMD portion of your fleet without touching the NVIDIA part, gradually shifting more workload to AMD as it becomes cost-effective.

Fifth, invest in cross-vendor monitoring. Tools like Prometheus and Grafana can collect metrics from both GPU types, but you'll need to normalize reporting (e.g., ROCm reports memory usage differently than CUDA). Building a unified dashboard early avoids confusion later. Some teams use the open-source DCGM exporter for NVIDIA and the ROCm SMI exporter for AMD, then aggregate in a single Grafana panel. The extra effort is small — roughly a week of an engineer's time — and pays off in operational clarity.

Finally, consider the total cost of ownership over three years, not just upfront hardware. Factor in electricity, cooling, support contracts, and engineering time. In many cases, the mixed fleet's TCO is 20-30% lower than an all-NVIDIA fleet, even when accounting for the additional complexity. The numbers don't lie: split inference is not just a cost-cutting hack, but a strategic move toward hardware flexibility and vendor independence.