§ ONNX Runtime for Engineers
9 lessons · every lesson has a Colab notebook

Quantization

Nine lessons on running trained models with ONNX Runtime. Each lesson links to a Colab notebook for hands-on practice. The core exercises use a CPU; GPU work is optional.

Before you begin

You should be comfortable with Python functions and arrays. Inference means using a trained model to calculate an answer. A graph describes the operations used in that calculation. An operator is one kind of operation, such as addition or matrix multiplication.

Read the plain-language introduction, try its question, then open the notebook. Keep your measured results separate from the illustrative simulations on this page.

Technical references: ONNX Runtime Python API and quantization guidance.

07 · Quantization: Smaller Numbers, Measured Trade-offs

Introduction

Quantization stores a range of real values with small integers. With scale 0.1 and zero point 0, value 1.2 becomes integer 12, while nearby values may round to the same level.

Learning goal

Explain integer quantization and compare dynamic and static methods using measured size, speed, and accuracy.

Before you start

Floating-point numbers, rounding, scales, model weights, activations, and basic evaluation metrics.

Lesson plan

  1. Encode and decode one real value using scale, zero point, and rounding.
  2. Compare dynamic weight-focused quantization with carefully calibrated static QDQ quantization methods.
  3. Measure artifact size, latency, and output error instead of assuming a universal gain.

Calculate one quantized value. With scale 0.1 and zero point 0, real value 1.2 maps to integer 12. A value of 1.24 also rounds to 12, so both decode near 1.2. Quantization saves space by accepting this controlled loss of precision.

Neural networks tolerate small weight perturbations, and most hardware has several times the INT8 throughput of FP32. Quantization converts fp32 models to int8 — roughly 4× smaller, often faster — for a bounded accuracy cost you get to measure.

DynamicStatic (QDQ)
WeightsINT8INT8
Activationsquantized on the flyscales fixed from calibration data
Data needednonea few hundred samples
Typical latency winmoderatelarger — EPs optimize this path hardest
When to usequick win, cold-start limits, transformer weightsedge / mobile / CPU serving default
 
from onnxruntime.quantization import quantize_dynamic, quantize_static, QuantType, QuantFormat, CalibrationDataReader

quantize_dynamic("fp32.onnx", "int8_dynamic.onnx", weight_type=QuantType.QInt8)   # no data

quantize_static("fp32.onnx", "int8_qdq.onnx", calibration_data_reader=MyCalib(),
                quant_format=QuantFormat.QDQ, activation_type=QuantType.QUInt8,
                weight_type=QuantType.QInt8, per_channel=True)                     # calibrated
Interactive · Quantization error explorer
What the notebook measures

A small MLP trained on two-moons data, then three variants — fp32, dynamic INT8, static QDQ — compared on size, latency, test accuracy, and maximum relative output drift. The lesson: check task metrics and output drift; a model can hold accuracy while getting brittle at the edges of its input distribution.

▶ Run notebook 07 in Colab 07-quantization.ipynb CPU runtime · trains a tiny model, then quantizes it