Neural Processing Unit (NPU) Explained

What is an NPU?

Neural Processing Unit (NPU) is a dedicated processor optimized for neural network and AI workloads. With the explosive growth of deep learning and AI applications, traditional CPUs and GPUs, while capable of performing similar tasks, have a significant disadvantage in terms of efficiency and power consumption. the NPU has been architecturally designed specifically for neural network computation through hardware-level optimization, and is able to deliver higher AI computation performance with lower power consumption.

It is just at the beginning of the explosion of demand for neural networks and machine learning processing. Traditional CPUs/GPUs can do similar tasks, but NPU specifically optimized for neural networks can perform much better than CPUs/GPUs. Gradually, similar neural network tasks will be done by dedicated NPU units.

NPU (neural processing unit) is a specialized processor for network application packets, using a "data-driven parallel computing" architecture, especially good at processing massive multimedia data such as video and images.

NPU is also a kind of integrated circuit, but different from the single function of special-purpose integrated circuit (ASIC), network processing is more complex, more flexible. Generally, we can use software or hardware in accordance with the characteristics of network computing special programming to achieve the special purpose of the network.

NPU

Core Advantages of NPU

The main advantage of NPU is its specialized parallel computing architecture, which is manifested in the following ways:

The highlight of the NPU is the ability to run multiple parallel threads -- The NPU is taken to another level with some special hardware-level optimizations, such as providing some easily accessible caching systems for some really different processing cores. These high-capacity cores are simpler than typical "regular" processors because they don't need to perform multiple types of tasks. This set of "optimizations" makes NPUs more efficient, which is why so much R&D is being put into ASICs.

One of the advantages of NPUs is that they spend most of their time focusing on low-precision algorithms, new data flow architectures, or in-memory computing capabilities. Unlike GPUs, they are more concerned with throughput than latency.

Processor modules of NPU

NPU is specially designed for IoT AI to accelerate neural network operations and solve the problem of inefficiency of traditional chips in neural network operations. The NPU processor includes modules for multiplication and addition, activation functions, 2D data operations, decompression, etc.

The multiplication and addition module is used to calculate matrix multiplication and addition, convolution, dot product, and other functions. There are 64 MACs inside the NPU and 32 in the SNPU.

The activation function module is used to implement the activation function in the neural network by using the highest 12th order parameter fitting, with 6 MACs inside the NPU and 3 in the SNPU.

The 2D data operation module is used to implement operations on a plane, such as downsampling and plane data copying, etc. There are 1 MAC and 1 SNPU inside the NPU.

The decompression module is used to decompress the weighted data. In order to solve the characteristics of small memory bandwidth in IoT devices, the weights in the neural network are compressed in the NPU compiler, which can achieve a 6-10 times compression effect with almost no impact on accuracy.

Modern NPUs typically contain the following core processing modules that work together to accomplish efficient computation in neural networks:

Neural Processing Unit (NPU) architecture Diagram：

NPU: the core carrier of cell phone AI

As we all know, the normal operation of cell phones is inseparable from the SoC chip, which is only the size of a fingernail cover but has all the "guts". Its integrated modules work together to support the implementation of cell phone functions. The CPU is responsible for smooth switching of mobile applications, the GPU supports fast loading of game screens, and the NPU is specifically responsible for the implementation of AI computing and AI applications.

It is also necessary to start with Huawei, which is the first company to use NPU (neural-network processing units) on cell phones, and the first company to integrate NPU into cell phone CPUs.

Evolution of NPUs in mobile devices

The evolution of mobile NPUs:

The application of NPU in smartphones began in 2017, when Huawei first integrated NPU in a commercial cell phone.Since then, major chip vendors have launched their own NPU solutions:

2017: Huawei first integrated NPU in the Kirin 970 processor

2018: Apple introduces a neural network engine in the A12 Bionic chip

2019-2020: Qualcomm, Samsung, MediaTek and other vendors have launched mobile SoCs with integrated NPUs

2021-2023: NPU performance becomes a key competitive point for flagship mobile chips

2024-2025: the performance of mobile NPU is greatly improved to support the operation of large AI models on the device side

Comparison of Current Mainstream Mobile NPUs

NPU has become standard in the latest mobile processors, and the performance difference is significant:

Apple A17 Pro: 26-core neural network engine, AI performance increased by about 40% compared to A16

Qualcomm Snapdragon 8 Gen 3: new Hexagon processor, AI performance increased by about 45% over the previous generation

MediaTek Tiangui 9300: 6th generation APU, AI performance increased by more than 100 percent

Samsung Exynos 2400: new generation NPU, AI performance increase of about 14.7 times

NPU vs. GPU

Although GPU has the advantage in parallel computing capability, it does not work alone and needs the CPU's co-processing. The construction of neural network models and data streams are still carried out on the CPU. There is also the problem of high power consumption and large size. The higher the performance, the larger the GPU, the higher the power consumption, and the more expensive it is, which will not be available for some small devices and mobile devices. Therefore, a small size, low power consumption, high computational performance, and high computational efficiency of a dedicated chip NPU was born.

NPU works by simulating human neurons and synapses at the circuit layer and directly processing large-scale neurons and synapses with a deep learning instruction set, where one instruction completes the processing of a set of neurons. Compared with CPUs and GPUs, NPUs integrate storage and computation through synaptic weights, thus improving operational efficiency.

CPU and GPU processors need to use thousands of instructions to complete the neuron processing. NPU can be completed with just one or a few instructions, so it has obvious advantages in the processing efficiency of deep learning. Experimental results show that the performance of NPU is 118 times that of GPU with the same power consumption.

CPU, GPU, and NPU Architecture Comparison

The characteristics of different processing units

1. CPU -- 70% of the transistors are used to build Cache, and part of the control unit. Few computational units, suitable for logic control operations.

2. GPU -- Transistors are mostly used to build computational units, with low computational complexity, suitable for large-scale parallel computing. Mainly used in big data, backend server, image processing.

3. NPU -- simulate neurons in circuit layer, realize storage and computation integration by synaptic weights. One instruction completes the processing of a group of neurons, improve operation efficiency. Mainly used in the communication field, big data, image processing.

4. FPGA -- programmable logic, high computational efficiency, closer to the underlying IO. Logic editable through redundant transistors and linkages. Essentially instructionless, no shared memory required, and more computationally efficient than CPU and GPU. Mainly used in smartphones, portable mobile devices, and automobiles.

Practical applications of NPU

• AI scene recognition by NPU when taking photos, and retouching pictures with NPU computing.

• NPU judges light source and dark light details to synthesize super night scenes.

• Realize voice assistant operation by NPU.

• NPU with GPU Turbo pre-determine the next frame to achieve early rendering to improve the smoothness of the game.

• NPU pre-determines the touch to improve the following hand and sensitivity.

• NPU judges the difference of front and back-end network speed demand with Link Turbo.

• NPU judge the game rendering load to adjust the resolution intelligently.

• Give the NPU the power saving by reducing the computing load of AI during the game.

• NPU realizes the dynamic scheduling of CPU and GPU.

• NPU assists in big data advertising push.

• Implement AI intelligent word association function of input method through NPU.

Explanation of each type of processing unit

APU: Accelerated Processing Unit, AMD product for accelerated image processing

BPU: Brain Processing Unit, Horizon's embedded processor architecture

CPU: Central Processing Unit, a mainstream product for PC cores

DPU: Data Stream Processing Unit, the AI architecture proposed by Wave Computing.

FPU: Floating Point Processing Unit, floating point module in general-purpose processors

GPU: Graphics Processing Unit, multi-threaded SIMD architecture, designed for graphics processing

HPU: Holographic Processing Unit, Microsoft's holographic computing chip and device

IPU: Intelligent Processing Unit, the AI processor product of Graphcore, a DeepMind investment

MPU/MCU: Microprocessor/Microcontroller Unit, typically used for low-computing applications in RISC computer architecture products

NPU: Neural Network Processing Unit, a new type of processor based on neural network algorithms and acceleration

TPU: Tensor Processing Unit, Google's processor dedicated to accelerating artificial intelligence algorithms

VPU: Vector Processing Unit, a chip introduced by Movidius, acquired by Intel, dedicated to accelerating image processing and artificial intelligence.

Future development trend of NPU

With the rapid development of AI technology and the continuous expansion of application scenarios, NPU, as the core hardware of AI computing, is experiencing unprecedented innovation and change. The following is an in-depth analysis of the future development trend of NPU:

Architecture Innovation

Further Convergence of Heterogeneous Computing Architectures

Deep integration of NPU with other computing units: future SoC designs will realize tighter integration of computing units such as NPU, CPU, GPU and DSP, and seamless switching and collaborative work between different computing tasks through unified memory architecture and intelligent scheduling system.

Reconfigurable Computing Architecture: The new generation NPU will adopt a dynamic reconfigurable architecture, which can adaptively adjust hardware resource allocation according to the characteristics of different AI tasks, and strike a better balance between generality and specialization.

Mixed-precision computing: Supporting mixed-precision computing modes ranging from INT4 to FP16, the NPU will automatically select the most suitable computing precision according to different parts of the model, balancing performance and precision requirements.

Mature In-Memory Computing Technology

Non-volatile memory computing: Utilizing new non-volatile memories such as ReRAM, MRAM, etc. to realize in-memory computing will significantly reduce the data handling overhead and increase the energy efficiency ratio by 10-100 times.

Analog in-store computing: By executing analog computation directly in the storage array, energy efficiency and computational density can be further improved, which is especially suitable for computationally intensive tasks such as convolutional neural networks.

3D Integrated In-Store Computing: Adopting 3D stacking technology, vertically integrating computing units with storage units, significantly increasing bandwidth and reducing latency, providing more efficient hardware support for large-scale neural networks.

Commercialization of brain-like computing architectures

Specialized Hardware for Stimulated Neural Networks (SNN): Specialized NPUs designed for biologically-inspired stimulated neural networks, capable of achieving computational efficiencies close to those of biological neural systems with very low power consumption.

Neuromorphic chips: Neuromorphic computing architectures that mimic the working mechanism of neurons and synapses in the human brain will move from research to commercialization, showing unique advantages in scenarios such as time series prediction and anomaly detection.

Adaptive Learning Hardware: NPU architecture that supports online learning and self-adaptation can continuously optimize models based on new data without relying on the cloud, realizing true edge intelligence.

Performance Enhancement

Specialized Operators Further Optimized

Domain-specific operators: Highly optimized dedicated operators are developed for different AI application areas, such as computer vision, natural language processing, recommendation systems, etc., providing 10-100 times performance improvement.

Sparse Computing Acceleration: Hardware acceleration units specifically designed for sparsity in neural networks, capable of skipping zero-value computation and significantly improving computational efficiency.

Dynamic operator fusion: Intelligent compiler works with NPU to realize dynamic fusion of multiple operators, reducing intermediate result storage and access, and improving computational throughput.

Wide Application of Quantization Technology

Extremely low-bitwidth quantization: 2-bit or even 1-bit extremely low-bitwidth quantization techniques will be widely used in NPUs to achieve several to dozens of times performance improvement with acceptable accuracy loss.

Adaptive quantization: Automatically adjust the quantization strategy according to the sensitivity of different layers of the model to maximize the performance improvement under the premise of guaranteeing accuracy.

Hardware support for quantization-aware training: NPU will directly support quantization-aware training so that the model can better adapt to the low-precision hardware characteristics and reduce the loss of precision caused by quantization.

Continuous Advancement of Chip Processes

Advanced process nodes: with the application of 2nm or even more advanced processes, the transistor density of NPUs will continue to increase, and the computing power per unit area will be significantly enhanced.

Advanced Packaging Technology: Advanced packaging technologies such as chiplet design and 3D stacking will enable NPUs to break through single-chip area limitations and realize larger-scale integration of parallel computing units.

New semiconductor materials: the application of carbon nanotubes, graphene and other new semiconductor materials will bring NPU higher energy efficiency and lower operating temperature.

Application Expansion

Dramatic increase in edge AI capability

End-side training: future NPUs will support lightweight end-side training, enabling devices to perform personalized model tuning based on user data while protecting privacy.

Multi-device co-computing: Multiple smart devices in a home or office environment will realize distributed co-computing through NPUs, working together to complete complex AI tasks.

All-weather low-power AI: The new generation of NPUs will support ultra-low-power modes to realize all-weather AI sensing and analysis while controlling power consumption at the milliwatt level.

Device-side large model reasoning becomes standardized

Mobile-side large language model (LLM): optimized large language models with 10-100B parameter scales will be able to run smoothly on mobile devices, providing a local interaction experience similar to ChatGPT.

Multi-Round Dialogue Understanding: The NPU will be specifically optimized for multi-round dialogue understanding to support more natural and coherent human-computer interactions.

Knowledge Base Integration: The device-side NPU will support efficient integration with the local knowledge base, enabling the AI assistant to access and understand the user's personal data and provide more relevant services.

Popularization of multimodal AI applications

Audio-visual fusion understanding: the NPU will support real-time fusion analysis of multimodal data such as video, audio, and text to achieve more comprehensive scene understanding.

Generative AI acceleration: Generative AI applications such as text generation image, video editing, music creation, etc. will realize real-time response under the acceleration of NPU.

Immersive experience: combined with AR/VR devices, NPU will support real-time environment understanding and virtual content generation to create a more natural and immersive mixed reality experience.

AI Security and Privacy Protection

Federated Learning Hardware Acceleration: the NPU will provide a specialized federated learning acceleration unit, enabling devices to participate in model training without sharing raw data.

Differential Privacy Computing: NPUs with built-in differential privacy mechanisms will ensure data privacy protection during AI computation, balancing utility and privacy protection needs.

Model anti-tampering mechanism: the NPU will integrate a secure computing unit to prevent the model from being maliciously tampered with or extracted, protecting AI intellectual property.

Vertical Industry Application Deepening

Medical health monitoring: Dedicated NPUs will enable wearable devices to monitor health indicators in real time and perform complex analysis, warning potential health risks in advance.

Intelligent Manufacturing Optimization: Industrial-grade NPUs will support real-time production line optimization and predictive maintenance to improve manufacturing efficiency and product quality.

Automated Driving Decision Making: Vehicle-grade NPUs will process multi-sensor fusion data to realize millisecond-level environment sensing and decision making, promoting the development of automated driving technology.

Ecosystem Evolution

Development tool chain maturity

Unified programming model: cross-platform and cross-device NPU unified programming model will significantly reduce the difficulty of AI application development and accelerate application innovation.

Automated model optimization: AI-assisted compilers will automatically optimize models for different NPU architectures, realizing the vision of “train once, deploy everywhere”.

Visual debugging and analysis tools: Professional NPU performance analysis and debugging tools will help developers pinpoint performance bottlenecks and optimize resource utilization.

Standardization and Open Ecology

Standardization of hardware interface: Standardization of NPU hardware interface will promote the prosperity of hardware and software ecosystem and reduce development and adaptation costs.

Open-source NPU design: open-source NPU architecture will lower the threshold of AI hardware innovation and promote the rapid development of NPUs in specific fields.

NPU-as-a-Service (NPUaaS): the cloud will provide customizable NPU services, enabling developers to customize virtual NPU resources according to application requirements.

References

Sze, V., Chen, Y. H., Yang, T. J., & Emer, J. S. (2020). Efficient processing of deep neural networks. Synthetic Lectures on Computer Architecture, 15(2), 1-341.

Dally, W. J., Turakhia, Y., & Han, S. (2022). Domain-specific hardware gas pedals. communications of the ACM, 65(9), 48-57.

Technical White Papers and Product Specification Notes from Major Chip Vendors (2023-2025)

AI Chip Market Research Report (2024)

Mobile Computing Platform Performance Review Report (2025)

Last updated: 2025-04-22