The Technology Behind Intel's Neural Processing Units

Michael Langan, from Intel, recently discussed the fascinating developments taking place within the neural processing unit (NPU) team and how the architecture is evolving.

As interest in artificial intelligence (AI) continues to surge, news coverage surrounding the technology is ubiquitous. Whether it concerns the launch of new large language models (LLMs) or investment announcements for AI startups, the buzz is everywhere.

This excitement has even given rise to "AI washing," a term that describes the tendency of companies to exaggerate their AI capabilities in order to draw in customers and investors. Consequently, regulatory bodies like the U.S. Federal Trade Commission are actively working to address misleading AI practices aimed at deceiving users.

Amid this turmoil, the need for robust hardware to support AI functionalities remains critical. This is precisely where neural processing units (NPUs) come into play. Often referred to as AI accelerators, NPUs are specialized hardware designed to speed up the computations essential for AI model performance, emulating the processing abilities of the human brain.

To gain a deeper insight into the creation of NPUs, Langan shared his thoughts during the annual Midas conference held in November 2024. Having spent 14 years at Intel, he now leads the NPU intellectual property team based in Ireland. He noted that the team is responsible for central IP related to all client options, like laptops and desktops, contributing to a substantial $30 billion revenue market each year with AI being a vital component.

Ireland’s Contribution to Neural Processing Units

The NPU IP team at Intel consists of about 500 members worldwide, but Langan highlighted that their technology's roots trace back to the Irish startup Movidius, which Intel acquired in 2016.

The surge in neural processing technology can be linked to the introduction of convolutional neural networks in 2012, predominantly utilized in computer vision for tasks like image recognition.

A pivotal moment arrived in 2017 when Google published a groundbreaking paper titled "Attention is All You Need," which introduced the transformer model architecture. Langan remarked that this shift altered the landscape overnight, leading to the development of many applications we now associate with AI, such as Chat GPT and LLMs. "Design efforts are now concentrated on accelerating workloads stemming from this architecture," he elaborated.

Within Intel, Langan highlighted that his team handles everything from traditional hardware design using Verilog RTL to extensive verification processes. They work across different manufacturing processes at TSMC and Intel, ensuring their designs can be integrated into any application. Additionally, a significant software and compiler support team is critical for their endeavors. "There’s fierce competition for optimized AI compilers, and we have a robust team focused on this in Ireland," he mentioned.

Intel employs around 250-300 individuals in this area in Leixlip. While this number is modest compared to the site’s total workforce of approximately 5,000, their contributions to the larger Intel strategy are substantial.

The Challenges of Rapid Change and Talent Shortages

One significant challenge facing those developing NPUs is the rapid pace of technological advancements, particularly in recent times. Major players like Microsoft, Dell, and HP frequently launch new models and features, creating a backlog for those who support the underlying tech.

Langan reflected on the change in dynamics, stating, "Previously, our customer-facing teams would promote new features to the market. Now, it’s the reverse, with clients approaching us for new applications and features."

Another pressing issue is the shortage of skilled talent required for NPU development. Langan emphasized the ongoing need for individuals proficient in deep learning hardware, software, and AI compilers.

To address this talent gap, Intel initiated an internship program with universities over a decade ago. "We’ve developed a strong pipeline and fostered excellent relationships with universities, resulting in high-caliber candidates joining us," he noted, adding that Ireland boasts a wealth of talented individuals recognized by Intel globally.

Looking forward, while Langan remains focused on AI models and related hardware/software, he acknowledged the looming question of what the next architectural evolution will be beyond transformers. "New research emerges weekly, with many referring to potential successors as ‘transformer killers.’ Although these technologies haven't yet materialized, advancements such as the Mamba and Hymba model architectures aim to enhance training efficiency while reducing power consumption. We monitor these developments closely to incorporate them into our hardware future," he stated.

AI, NPU, Intel