POV on Top Considerations for Designing AI Inference Chips
AI inference workloads are becoming more compute-intensive, with higher demands on throughput, memory access, and power efficiency.
AI inference chips are placing new demands on physical design. Compute is becoming denser, SRAM is moving closer to compute, data paths are becoming wider, and local power density is increasing.
As a result, floorplanning, power delivery, clocking, routing, and thermal management all must work together. The physical design decisions made early in the flow can directly affect performance, power, and scalability.
How should floorplanning change when AI inference chips contain large compute clusters with tightly coupled SRAM and high-bandwidth data paths?
The chip floorplan methods that have worked for decades, even for ‘complex’ CPU-based designs, have seen a tectonic shift with the advent of AI inference chips.
Golden rules while freezing the floorplan for traditional designs:
- Digital logic to occupy central area with least obstructions
- Memory macros around the edges, preferably without intruding into the central core area
- Relatively narrow buses and less constrained data path latency
But AI inference chips have different needs: high data throughput, low latency, hundreds of compute units processing data in parallel, and they are power-hungry as well.
While doing floorplanning for AI inference chips:
- Ensure compute logic and memory macros are co-located in dataflow-driven floorplanning
- Repetitive tiles (compute + memory) rather than a few big chunks of logic and macros
- Wide data bus can save power and reduce latency, optimizing the physical connection length and reserving routing tracks, ensuring smooth scalability, just by adding more tiles
- Use a more complex power-delivery network—including TSVs, microbumps, and backside power delivery—and identify and distribute power hotspots across the chip.
- Clock distribution network must balance within each tile and from the root to each tile, and support gating
How do you manage IR drop and local power density when thousands of MACs or AI compute units switch simultaneously?
Managing power in AI inference chips is one of the hardest sign-off steps. Thousands of chips processing data together act like power guzzlers, pulling current from the power grid at once and causing voltage drops. Usually, high current consumption issues were tackled either by compromising on frequency of operation or by reducing operational voltage.
AI chips are too power-hungry for these approaches to be practical.
The solution is:
- Compute tiles not triggered at the same time but with staggered start
- Clock phase shifting – one set of tiles gets the clock edge a little bit later than another set. Hence, processing logic is active at different timestamps
- Lower the voltage/frequency of the idle region using DVFS (Dynamic Voltage and Frequency Scaling)
- Vertical Power Delivery (VPD) – power comes from directly underneath via TSVs
- Backside Power Delivery – moves power routing to the backside of the silicon wafer
- Plan for sufficient decoupling capacitors
How should clock tree design account for large AI compute arrays, high fanout control signals, and multiple performance-critical domains?
Clock tree design for AI compute arrays must find a sweet spot considering low skew across thousands of MACs, power efficiency since the clock network contributes heavily to dynamic current consumption, and multi-domain integrity (in case of DVFS)
For regular or irregular tile arrays, no single topology is sufficient. The implementation may use a mix of H-tree, multi-point CTS, and clock mesh.
- Fine-grain clock gating reduces clock power in inactive compute clusters
- Medium-sized buffers handle clock propagation instead of large ones
- Correctly sized clock cells and high-Vt cells, where timing permits, can reduce leakage
- Multiple power and voltage domains require isolation and clock integrity. Each domain can use a dedicated clock tree, while minimizing clock crossings
What physical design trade-offs are most important when deciding between higher compute density, larger on-chip memory, and manageable thermal and routing congestion?
High compute density gives high throughput but bumps up power and routing track requirements.
High on-chip RAM reduces chip traffic, enables faster data transfer, better power bit disproportionately increases the chip area occupied by RAM. Close attention towards thermal control and routability will help with easier design closure but losing on effective functionality of the chip.
- Limit compute performance and throughput to a limit can support and manage generated heat
- Managing power hotspots bloom – uniform power consumption profile is easier to manage
- Voltage and frequency require careful calculations. Both curves are non-linear. Increasing voltage might provide performance advantage, but the power profile might deteriorate more aggressively
- SRAM does not scale like logic. If the workload is compute-bound, on-chip RAM is a bad trade. If it is memory-bound, it is essential
- SRAM blocks have high leakage and dynamic power for reads/writes. Weigh the energy cost of accessing off-chip DRAM against keeping data in on-chip RAM
- The best floorplan is the one that takes care of the workload’s actual bottleneck, not the one that maximises any single metric
From a physical design perspective, what is the biggest challenge that makes AI inference chips different from conventional SoC designs?
Power delivery and thermal density!
Routing, clock skew, and SRAM placement are hard but solvable with known techniques. Feeding and cooling a massively parallel array under simultaneous switching load is what makes AI inference chips fundamentally different.
What is one consideration that AI chip designers often underestimate during the physical design stage, but can have a significant impact on the final chip?
How much the floorplan controls data movement energy compared to compute energy.
Most teams obsess over MAC efficiency, TOPS/mm2, and compute density. But in real AI inference chips, moving a byte of data is often costlier than multiply-accumulate.
During architecture and RTL stages, data movement looks “free”. It is just wires. Teams size SRAM and buses based on bandwidth (GB/s), not energy per byte per mm. By the time physical design reveals the true wire lengths, the floorplan is locked, and there is no room to fix it.
MosChip has experience across the physical design flow for complex Turnkey ASIC and SoC designs, from floorplanning and power planning through placement, routing, clock tree synthesis, timing closure, and physical sign-off. The work includes macro placement, congestion optimization, IR drop and power integrity analysis, multi-voltage and low-power implementation, ECOs, DRC, LVS, and extraction across process nodes from 180nm to 2nm.
