An open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, ONNX Runtime, and MXNet)
Intel® Neural Compressor, formerly known as Intel® Low Precision Optimization Tool, is an open-source Python library that runs on Intel CPUs and GPUs, which delivers unified interfaces across multiple deep-learning frameworks for popular network compression technologies such as quantization, pruning, and knowledge distillation. This tool supports automatic accuracy-driven tuning strategies to help the user quickly find out the best quantized model. It also implements different weight-pruning algorithms to generate a pruned model with predefined sparsity goal. It also supports knowledge distillation to distill the knowledge from the teacher model to the student model. Intel® Neural Compressor is a critical AI software component in the Intel® oneAPI AI Analytics Toolkit.
Visit the Intel® Neural Compressor online document website at: https://intel.github.io/neural-compressor.
Python version: 3.7, 3.8, 3.9, 3.10
- Release binary install
# install stable basic version from pip pip install neural-compressor # Or install stable full version from pip (including GUI) pip install neural-compressor-full
- Nightly binary install
git clone https://github.com/intel/neural-compressor.git cd neural-compressor pip install -r requirements.txt # install nightly basic version from pip pip install -i https://test.pypi.org/simple/ neural-compressor # Or install nightly full version from pip (including GUI) pip install -i https://test.pypi.org/simple/ neural-compressor-full
More installation methods can be found at Installation Guide. Please check out our FAQ for more details.
# A TensorFlow Example
pip install tensorflow
# Prepare fp32 model
wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/mobilenet_v1_1.0_224_frozen.pbimport tensorflow as tf
from neural_compressor.experimental import Quantization, common
quantizer = Quantization()
quantizer.model = './mobilenet_v1_1.0_224_frozen.pb'
dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()Quantization with JupyterLab Extension
Search for jupyter-lab-neural-compressor in the Extension Manager in JupyterLab and install with one click:
Quantization with GUI
# An ONNX Example
pip install onnxruntime-extensions
# Prepare fp32 model
wget https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx
# Start GUI
inc_benchIntel® Neural Compressor supports CPUs based on Intel 64 architecture or compatible processors:
- Intel Xeon Scalable processor (formerly Skylake, Cascade Lake, Cooper Lake, and Icelake)
- Future Intel Xeon Scalable processor (code name Sapphire Rapids)
Intel® Neural Compressor quantized ONNX models support multiple hardware vendors through ONNX Runtime:
- Intel CPU, AMD/ARM CPU, and NVidia GPU. Please refer to the validated model list.
- OS version: CentOS 8.4, Ubuntu 20.04
- Python version: 3.7, 3.8, 3.9, 3.10
| Framework | TensorFlow | Intel TensorFlow | PyTorch | Intel® Extension for PyTorch* | ONNX Runtime | MXNet |
|---|---|---|---|---|---|---|
| Version | 2.10.0 2.9.1 2.8.2 | 2.10.0 2.9.1 2.8.0 | 1.12.1+cpu 1.11.0+cpu 1.10.0+cpu |
1.12.0 1.11.0 1.10.0 |
1.12.1 1.11.0 1.10.0 |
1.8.0 1.7.0 1.6.0 |
Note: Set the environment variable
TF_ENABLE_ONEDNN_OPTS=1to enable oneDNN optimizations if you are using TensorFlow v2.6 to v2.8. oneDNN is the default for TensorFlow v2.9.
Intel® Neural Compressor validated 420+ examples for quantization with a performance speedup geomean of 2.2x and up to 4.2x on VNNI while minimizing accuracy loss. Over 30 pruning and knowledge distillation samples are also available. More details for validated models are available here.
- Intel together with Tencent deepens the cooperation to build a cloud foundation for digital and intelligent industry (Dec 2022)
- Running Fast Transformers on CPUs: Intel Approach Achieves Significant Speed Ups and SOTA Performance (Nov 2022)
- Intel Neural Compressor for TF Virtual Appliance packaged by Bitnami (Nov 2022)
- Neural Compressor: an open-source Python library for network compression (Nov 2022)
- Running Fast Transformers on CPUs: Intel Approach Achieves Significant Speed Ups and SOTA Performance (Nov 2022)
- Personalized Stable Diffusion with Few-Shot Fine-Tuning (Nov 2022)
- Fast DistilBERT on CPUs (Oct 2022)
- Meet the Innovation of Intel AI Software: Intel® Extension for TensorFlow* (Oct 2022)
- PyTorch* Inference Acceleration with Intel® Neural Compressor (Oct 2022)
View our full publication list.
- Release Information
- Contribution Guidelines
- Legal Information
- Security Policy
- Intel® Neural Compressor Website
We are actively hiring. Send your resume to inc.maintainers@intel.com if you are interested in model compression techniques.