The GPU Server Buyer's Guide: H100 vs H200 vs B200 for AI Commerce Workloads

The Thermal Throttling Problem Nobody Talks About

Last spring, I got a call from a research computing team at a major university — one I had worked with before during my time at EKWB USA. They had just taken delivery of a new 8-GPU H100 SXM5 cluster, air-cooled, from a well-known OEM. The system looked great on paper. On the bench, it looked less great: sustained inference workloads were thermal throttling inside 90 minutes. Junction temps were hitting 84°C. The GPUs were backing off clock speeds. Their benchmark scores, which had looked excellent during short burst tests, were not reproducible under real sustained workloads.

This is not a rare story. It is the story I have seen repeat itself at university research labs, at hedge funds running quantitative models, at enterprise teams trying to serve 70-billion-parameter models to production traffic. The GPU market moves at a pace that makes careful evaluation difficult. Vendors have every incentive to quote peak numbers from 30-second benchmark windows. Buyers have every incentive to believe them, because the numbers are extraordinary.

My job — when I was running EKWB USA and when I work with clients now — has always been the same: cut through the marketing, match the hardware to the actual workload, and make sure the cooling infrastructure can sustain performance under real conditions. That is what this guide is about. Not the spec sheets. The deployments.

I am going to walk you through the three GPU generations that matter right now for AI commerce workloads — H100, H200, and B200 — give you the real thermal data, work through the cloud versus on-prem math honestly, and tell you what I would actually recommend for each type of buyer. No upselling. No overbuilding. If you need two GPUs, I will tell you two GPUs.

NVIDIA H100 SXM5: The Current Workhorse

The H100 is the GPU that kicked off the modern AI infrastructure arms race, and it remains the most widely deployed training and inference accelerator in production today. Understanding it in detail matters because it sets the baseline against which everything else gets compared — and because a significant percentage of what you can actually purchase and receive in the next 90 days will be H100-based.

What the Spec Sheet Doesn't Tell You

At 700W TDP per GPU, an 8-GPU H100 system draws 5.6 kilowatts from the GPU array alone — before you account for CPUs, memory, NVMe drives, networking, and chassis power. Total system draw in a dense configuration runs 10–14kW. That heat has to go somewhere.

In air-cooled deployments, which is what most OEM servers ship by default, sustained workloads push junction temperatures into the 80–85°C range. I have measured 84°C on sustained inference runs in data center environments with adequate airflow. At that temperature, the GPU's thermal management system starts throttling clock speeds to protect the silicon.

In liquid-cooled configurations — direct-to-chip or full immersion — those same GPUs run at 42–48°C under sustained load. No throttling. The peak performance numbers become actual sustained numbers. This is not a marginal difference. It is the difference between the system you paid for and the system you actually get.

H100 Inference Capacity for Real Workloads

For AI commerce applications, a single H100 SXM5 can sustain: serving a 70-billion-parameter LLM in FP8 quantization, 50–100 requests per second at typical prompt/completion lengths. Two H100s in NVLink configuration push above 150 req/sec with tensor parallelism. For most mid-market agentic commerce deployments — an intelligent checkout assistant, a real-time product recommendation engine, a multi-agent transaction processor — 2–4 H100s covers the load with headroom.

UCP and ACP protocol processing is CPU-bound rather than GPU-intensive. Do not buy GPU capacity for protocol overhead. Buy it for the inference workloads those protocols invoke.

NVIDIA H200: When the Memory Wall Matters

The H200 shipped in Q2 2025 and became the correct choice for large-context inference workloads. The architectural change is not in the compute cores — the GPU die is largely the same. The change is in the memory subsystem.

Running a 70B model in full FP16 precision requires ~140GB of GPU memory — two H100s at capacity, or one H200 with headroom. If your workloads involve large-context RAG, multi-document summarization, or long-session conversation memory, the H200's 141GB HBM3e changes the architecture. You fit more on fewer GPUs, which simplifies tensor parallelism and reduces inter-GPU communication overhead.

The 4.8 TB/s memory bandwidth drives the 1.5–1.9x inference improvement. LLM inference is memory-bound — the bottleneck is moving data between memory and compute. More bandwidth means faster token generation, lower latency per request. For AI commerce where response time matters, this counts.

The caveat: if your workloads are purely compute-bound — dense training, small-batch inference at high throughput — the H200's advantages shrink. Profile your workload before you spec the hardware.

NVIDIA B200: Blackwell and the Question of Availability

The B200 is genuinely revolutionary hardware. At 1,000W TDP per GPU, an 8-GPU cluster draws 8kW from the GPU array at sustained load — 15–18kW total system. The B200 is effectively liquid-cooling-mandatory for sustained workloads.

Limited availability, allocation queues from hyperscalers, and lead times stretching months mean if you need hardware in the next 90 days, B200s are largely off the table. For organizations doing serious multi-model fine-tuning or frontier-scale training, the B200 is where you want to be. Everyone else: wait for availability to normalize, or buy H200s now and upgrade in 18–24 months.

Liquid vs Air Cooling: The Decision That Changes Everything

Air cooling runs H100 junction temps at 80–85°C under sustained load. Liquid cooling runs those same GPUs at 42–48°C. The performance difference is not subtle.

Supermicro ships air-cooled by default. Dell's XE9680 is available liquid-cooled and is the most accessible enterprise option. For purpose-built liquid cooling — direct-to-chip cold plates, custom manifold designs — EK and CoolIT Systems both offer enterprise-grade solutions. My recommendation: spec liquid cooling from the start. The retrofit cost is higher than speccing it correctly at purchase.

Right-Sizing: Match the Hardware to the Workload

Agentic Commerce Pilot (1–2 GPUs): Single LLM for checkout assistance, product recommendation, or transaction processing. 70B parameter model in FP8, 50–100 concurrent users. One or two H100s. Do not buy an 8-GPU server for this workload.

Mid-Market Production (2–4 GPUs): Multiple inference endpoints — customer-facing agent, internal knowledge retrieval, fine-tuned vertical model. 2–4 H100s or H200s in a half-populated server with a natural expansion path.

Enterprise Multi-Model Infrastructure (8–16 GPUs): Model zoo with foundation models, fine-tuned variants, embedding models, reranking models. Training running in parallel with inference. SLA commitments. 8-GPU H200 server or dual rack.

Frontier Training (B200 when available): Pre-training or large-scale fine-tuning above 7B parameters. Plan for liquid cooling, higher power infrastructure, and longer procurement lead times.

Cloud vs On-Prem: The Break-Even Analysis

On-prem 8-GPU H100 server with liquid cooling: $350K–$400K capital + ~$50K/yr operations. Three-year total: $500K–$550K. Cloud at 100% utilization: $858K/yr on AWS. The break-even is ~40–50% GPU utilization. Above that, on-prem wins. Below it, cloud wins on flexibility.

For most mid-market AI commerce: hybrid. 2–4 GPUs on-prem for continuous baseline inference (high utilization), cloud burst for peak demand and training (variable utilization). Expand on-prem as baseline grows.

Let's Build It Right the First Time

The GPU decision is not primarily a hardware decision. It is a workload characterization problem, a cooling infrastructure problem, and a three-year total cost of ownership problem. Get those three things right and the hardware choice becomes straightforward.

If you are sizing a GPU infrastructure deployment for an AI commerce workload and want a second opinion from someone who has put these systems into production at MIT, Cornell, Princeton, Texas A&M, and CrowdStrike — reach out. I will look at your workload profile, your utilization projections, and your facility constraints, and I will give you a recommendation I can stand behind. No vendor relationships influencing the answer. No overbuilding. Just the right system for what you are actually trying to do.