Memory-Aware ASIC Design for AI Workloads
Every AI accelerator is limited by two main things: computation and data movement.
Recently, AI processors have seen significant improvements in computing power due to specialized units such as NPUs, larger MAC arrays, and more parallel structures. But as AI tasks have grown from CNNs to transformer models and multimodal systems, memory has become the main problem. In many AI chips, the processing part can do more operations than the memory can support.
This has changed how AI chips are built.
Instead of seeing memory as just a helper part, companies now focus on memory levels, how data flows, and how data is reused when designing the chip. The goal is not just to make the most operations possible but to keep the processing capability by refining data efficiently.
The memory wall starts with the workload
Memory-aware design begins by understanding the workload.
CNNs have predictable memory access, and they reuse the same model parts often, so they work well with local SRAM. Transformer models are different. They have big parameter sets, attention operations, and growing key-value caches, which means they need a lot of memory bandwidth and frequent access.
These features shape how the chip is built.
Workload analysis helps estimate tensor sizes, how much data can be reused, memory bandwidth needs, and how much memory is needed at any time. Before RTL design and development begin, this information is gathered. If the memory cannot meet the threshold, adding more cores is not helpful. This problem is called the memory wall.
Building the memory hierarchy
The memory system is designed to reduce unnecessary data movement.
Register files are the fastest access for active data. Local SRAM holds data that is reused often. Shared memory helps different parts of the chip communicate with each other. External memory like LPDDR or HBM gives space for bigger models.
Deciding how much local and external memory to include is a balance, not just about adding more memory.
More SRAM reduces data movement but takes up more space, power, and makes routing harder. Using HBM improves speed but affects the package, heat, and cost. So, architects balance performance, power, area, and the needs of the application instead of focusing only on one thing.
Reducing data movement through the architecture
After setting up the memory hierarchy, the next step is to reduce how much data moves between memory levels.
The way data flows through the computing units affects how much it is reused and how much memory is accessed. One common method is to keep often-used model parts in local memory while moving input data through the processing parts, which lowers the need for external memory and improves bandwidth. The best data flow depends on the type of AI task.
Loop tiling keeps data in local memory for longer, reducing the need to access external memory again.
Memory banking allows many processors to access memory at the same time. Double buffering helps keep the processing moving by overlapping data transfer with calculation. DMA engines move data between memory levels while computation is happening, improving overall chip performance.
Memory is more than SRAM and DRAM
Efficient data movement also depends on the internal connections.
Even if there is enough memory bandwidth, a bad internal network can cause traffic jams and slow things down. So, memory controllers, access rules, and how data is scheduled are just as important as the memory itself.
For larger models, new packaging like HBM, 2.5D, and chiplet-based designs extend memory beyond what traditional interfaces can handle.
Processing-in-Memory is still being studied, but most commercial AI chips rely on careful use of the existing memory structure to get the best results.
Hardware-Software Co-Design
Memory efficiency cannot be achieved just with hardware design alone.
AI compilers decide where to place data, how to combine operations, and how to schedule tasks to maximize reuse and reduce memory access. Using less data (quantization) and focusing on sparse data also help lower memory use and bandwidth.
Because software controls memory behavior, the chip design and compiler must work together to move data efficiently.
Hardware-software co-design is now an important part of building AI chips.
From Architecture to Tapeout
Memory decisions stay important even after the design is set.
Where SRAM is placed affects the layout, the design process, routing, and timing. Testing must check both how the chip works and how the memory system handles many requests, data movement, and peak loads. A system that works well in simulations must also work well after the physical chip is made.
In short, memory-aware ASIC design is not just about using faster memory or more bandwidth.
It is a way of thinking that guides the design from understanding the workload to building the chip. Choices about memory levels, data flow, internal connections, data movement, compiler settings, and physical design all affect how efficiently the processing engine gets data.
As AI tasks keep changing, successful AI-enabled ASICs will not just be fast, but also smart about how they move data around the entire system.
Keeping data movement to a minimum, using data effectively, and balancing speed, power, and size will continue to be key to building scalable AI chips.
Designing memory-aware AI ASICs needs skills across different areas.
MosChip offers full Turnkey ASIC services to help clients build AI chips optimized for modern tasks. Our services cover everything from design and testing to packaging and mass production.
By combining experience in all parts of silicon design and development and product engineering, we help organizations create scalable, power-efficient AI chips that use computing resources well while keeping memory bottlenecks to a minimum.
