Preparing for and keeping up with AI/HPC infrastructure needs has been an ongoing challenge for years. You’ve likely felt the strain in lead times that keep stretching or facility designs that constantly need adjusting. With AHEAD heading to SC26 next week in our headquarters city of Chicago, I thought I’d share some initial thoughts on facility infrastructure.
Walking the floor, you’ll probably notice how much has changed by looking at the cables alone. For newer AI/HPC deployments, data cabling now sits up front, where traditional racks keep it in the rear. There are no individual power cords for each server either, due to DC bus bars in rack. For your facility, the increased density means the whole room is changing along with the rack.
If you run a hyperscale or neocloud environment, this probably isn’t new to you. If you’ve spent decades operating enterprise data centers, some of it might be. Those environments were built well, just for a different era.
During SC26, use my checklist as a reference on what to look for, why it matters, and what ready looks like for your team.
1. Power Delivery for 150 to 250 kW Racks
Since I mentioned cabling and power first, let’s start there. Most of these racks are MGX or ORv3 compliant, with a DC bus bar in the rear. Instead of rack-mounted PDUs distributing AC power to servers, power shelves convert AC to DC and feed the bus bar. That also means the compute and networking nodes inside don’t have individual power supplies the way traditional equipment does.
These racks pull 150 to 250 kilowatts each. That used to be enough for an entire row of cabinets. Vendor roadmaps show a single rack eclipsing 600kW in the next 1-2 years, with 1MW on the horizon. The old 208 volt North American data center standard is giving way to 400 volts, or 415v in the US. If your facility can’t support that draw, it’s your first big hurdle to getting these racks running.
Many of these racks also take 480 volts, which can be easier for a facility running 208v today, since even legacy data centers often run UPS systems at 480v. If you feed 480v directly instead of stepping down through a transformer, the uplift potentially shrinks to new wiring, some switchgear, and a decision about bus bars. That matters because transformer lead times are often eclipsing the time to receive GPU systems. Nuances in downstream equipment play into whether 480v is viable directly to the racks, and it’s details like these that make the difference between success and delays.
What Ready Looks Like: 415 or 480 volts at the rack location, so your distribution plan isn’t waiting on a transformer you can’t get.
2. A Cooling Approach that Fits Your Site
Most of this gear is moving to direct liquid cooling, which usually means at least 80 to 85 percent of the power these systems use is liquid-cooled. The catch is that your facility has to support it. Most facilities already use chilled water for primary cooling, but new warmer water temperature requirements and coolant distribution units (CDUs) have recently come into play. Air cooling still matters too, since even 15 percent of 150kW is more air cooling than many legacy facilities can support.
In a retrofit, you can often feed existing chilled water to a CDU, as long as your building systems have the capacity. The more power-efficient move is free cooling, ideally a dry cooler, which can cut water usage. That’s important as data center projects face public pushback everywhere. Most new facilities have left evaporative cooling behind, even though it was cheap.
The newest systems run warm enough for dry coolers or adiabatic cooling, which uses a small amount of water only at the hottest times of year. Geography helps too. A northern site can free cool for a much bigger slice of the year than, say, Phoenix. Warmer climates can still go hybrid, using free-cooling to shed most of the heat and a refrigerant-based chiller for the last bit of heat rejection.
What Ready Looks Like: A cooling design that matches your climate, with chilled water capacity confirmed for your CDUs or a free-cooling plan to take its place.
3. Floor Loading and Rack Layout
Rack weight is climbing fast. Some options already top 3,500 pounds, and roadmaps point toward 4,000+. Many raised flooring systems can’t handle that without reinforcing the substructure. Newer data centers skip that problem by setting racks on concrete, with a coating or high-density tile for electro-dissipative qualities.
Layout gets interesting too. One rack can draw the same power as an entire row used to. Without a wholesale reorganization of your power distribution, you could end up with one or two racks in a row originally built for 10 to 12. Sometimes, if you already have the total power, spreading out gets your racks online faster. It’s the core tradeoff of any retrofit: time and money spent on site modifications versus getting equipment online as fast as possible.
What Ready Looks Like: A floor rated for the weights of new and upcoming racks, and a purposeful layout.
4. Network Fabrics and Cabling that Match End to End
These systems can have as many as four discrete networks: management, north-south (roughly your old production network), east-west for GPU connectivity, and a separate storage network. Then you add the complexity of Infiniband versus three slightly (but importantly) different variants of Ethernet. Transceiver form factors have gone from basically two (QSFP and SFP) to upwards of five: QSFP, SFP, QSFP-DD, QSFP-112, OSFP-Finned, OSFP-Nonfinned, etc.
From a speeds and feeds perspective, even the same OEM’s switches can be incompatible at what appears to be the same bitrate. For instance, not all 400 gigabit connections are created equal. Are we talking 8x50G lanes, or PAM4 with 100G lanes? The port form factor and the ultimate speed are the same, but the links won’t work if that next level of detail isn’t properly evaluated during BOM creation. Even with one vendor, the NICs going into your servers may or may not work with an upstream switch choice.
Cabling has its own surprises. MPO cables look the same on the outside, but the fiber is cut and polished differently, with angled physical contact (APC) required for most newer high-speed connections and ultra physical contact (UPC) left to yesterday’s technology. Your facility team might see a transceiver with a familiar-looking MPO interface, but UPC cables or patch panels introduced anywhere in the light path are problematic. Then come further form factor differences, with MMC and MDC connectors bringing higher density fiber and XPO modules bringing extreme density optics.
What Ready Looks Like: Every link specified, from NIC to switch to connector polish, and the facility cabling spec validated before an order is placed.
5. Integration and Validation Before Anything Ships
Deployment lives and dies by lead times. You can have the servers but not the rack to mount them in, and suddenly a $5,000 rack is holding up $1.5M of servers. And that’s just one rack.
The full bill of materials should be tracked against your facility schedule, with alternative manufacturers lined up for long-lead items. Power and liquid cooling readiness must be settled upfront. The goal is to validate as much as possible across power, cooling, networking, and system workloads before anything ships.
For an NVL72-type solution, the rack is the computer. The OEM builds those entire racks, much like it once built complete servers. Storage often comes from a different vendor than your compute OEM. Since networking fabrics are extremely complicated, integrating them with the supporting racks can take complexity off your floor. And with facility requirements, rack spacing, and ladder rack height established, cabling packages can be pre-built to length and installed much faster than an on-site build once the racks land.
Many facilities won’t be ready to host this equipment out of the gate. They don’t have to be. While you change power distribution and add liquid cooling, the network and compute cabinets and storage can be built and tested in parallel. That way, when everything lands, the two workstreams can click into place together.
What Ready Looks Like: Facility work and equipment build moving in parallel, so neither one is left waiting on the other.
6. Ready on Arrival: No Surprises, Just Success
The last thing anyone needs is for the final piece of equipment to arrive, only to find the cooling falls short, a part is missing, or the facility ran the wrong power to the wrong place. Any one of these can cost you weeks, or even months.
Success means no delays once the equipment is on site. Every part that lands is compatible and can be connected and operational quickly. Cabinets get placed, the collaborative teams make the power and liquid cooling connections, and the remaining cabling between racks gets connected. Pre-planned, pre-staged cables make each rack a drop-in, plug-in finish.
What Ready Looks Like: The equipment lands, everything fits, and no one stumbles onto a last-minute surprise.
One Big Picture, Built Together
In the old days, facilities had a baseline target for power and cooling. Technology owners picked servers, storage, and networking. The data center made those choices work, because there was enough power and cooling to go around.
That model is changing. AI/HPC systems are now so dense that you can’t architect them the way you used to, sketching them out at a high level and diving into the details later. These designs need to be fully baked, often before the first purchase order is cut.
Splitting responsibilities can also sink a project. Procurement is often mandated to save every dollar, so networking gets bought from one vendor, servers from another, with facilities handled separately. While that traditional approach makes sense as far as price, it tends to cost more in the end. Maybe a team prepares a facility without accounting for the detailed equipment specs. Or Network Vendor A builds switch and optic BOMs that aren’t fully compatible with Compute Vendor B’s server BOMs. Suddenly you’re facing project delays of weeks or months, plus emergency orders for the correct optics, cables, NICs, or even switches. Saving those pennies up front can suddenly (and quite painfully) end up costing your business measurable time and money.
If you take anything away from SC26, it’s this: everyone now must build the same big picture together. Your teams can no longer function in silos. Every person who touches this deployment must be seated at the table and communicating openly.
Why AHEAD for AI/HPC Infrastructure: Integrated, Tested, and Ready to Run
The approach I’ve described is AHEAD’s go-to for AI/HPC solutions, because we know it works.
We know the difference a good facility can make. AHEAD’s own Foundry integration facility in Libertyville, Illinois, has 10 megawatts of power and direct liquid cooling integration. It’s how we power up and test your systems before they ship, air-cooled or liquid-cooled, while you prep your facility. With partnerships spanning some of the biggest names in the ecosystem, AHEAD can offer agnostic advice on power and cooling readiness. The result is capacity that’s ready to run as soon as it lands.
Our technical depth goes as deep as the details demand. The awards we’ve earned show how much trust major OEMs have in what AHEAD develops and deploys, with public accolades from NVIDIA, Dell Technologies, Cisco, VAST Data, NetApp, and Everpure. AHEAD Professional Services teams stand by you through deployment; getting networking operational, building storage sub-systems, establishment management planes, and setting up clusters so everything interoperates. And when you need help running it after it goes live, AHEAD Managed Services can take that on.
Everyone’s setup is a little different. AHEAD meets your team exactly where you need us, ensuring your project works as intended.
If you’re interested in talking big picture, find me and the rest of the AHEAD team at SC26 in booth 6720.
;
;
;