Execution providers

The one-sentence version

ONNX Runtime (ORT) is a cross-platform inference engine that executes ONNX graphs — it is a runtime, not a framework. If PyTorch and TensorFlow are languages plus compilers, then ONNX is the bytecode and ORT is the VM that runs that bytecode on CPUs, GPUs, NPUs, phones, and browsers.

Two things follow from that split, and they explain almost everything else in this guide:

1. The artifact is the contract. Training frameworks stop at the file boundary. Deployment starts there. You can train in PyTorch, export once, and execute the same bytes in a C++ service, a browser tab, and an Android app.
2. The runtime owns everything after the file. Optimization, kernel selection, memory planning, device placement — all deployment-time concerns, all invisible until they slow you down. This guide makes them visible.

01 · What ONNX Runtime Actually Is

Trace one request. A client sends a float tensor shaped (1,4). An inference session validates its name and shape, runs the stored graph on available hardware, and returns the named output tensor. The runtime executes learned computation; it does not train the model.

A session, not a process

You do not "run" a model in ORT. You create an InferenceSession once, and call it many times. Session creation is where the runtime does its expensive work:

1. Parse and validate the graph (structure, opset, types)
2. Optimize it — fusion, constant folding, layout transforms (lesson 04)
3. Partition and assign kernels across Execution Providers with CPU fallback (lesson 05)
4. Plan memory for activations, keyed by input shape (lesson 06)

Per inference call, what remains is copying inputs, launching kernels, and copying outputs. Everything else is amortized. This is why "startup was fast, throughput is fine" and "throughput is fine, latency has overhead" are different problems with different fixes.

import onnxruntime as ort

sess = ort.InferenceSession("tiny_linear.onnx", providers=["CPUExecutionProvider"])
y = sess.run(None, {"X": x})[0]        # None = fetch every output

Framework vs. runtime: who owns what

02 · The ONNX Format Up Close

Read the graph as data. Each node names an operation and connects named tensors. Initializers store learned constants such as weights. Opset and IR versions describe different compatibility layers, so matching one does not guarantee matching the other.

Anatomy of a graph

An .onnx file is protobuf. A GraphProto holds:

Fail: [ONNXRuntimeError] : 1 : FAIL : Unsupported model IR version: 14, max supported IR version: 13

import onnx
from onnx import shape_inference

inferred = shape_inference.infer_shapes(model)     # propagate shapes through the graph
json_like = onnx.printer.to_text(model)            # read the protobuf as text
total_params = sum(t.dims == t.dims and int(np.prod(t.dims)) for t in model.graph.initializer)

The errors you will actually see

03 · Exporting from PyTorch

Predict the acceptance check: for the same fixed input, PyTorch and ONNX Runtime should return the same shape and numerically close values. A successful export call only proves that a file was written.

The pipeline is three steps, and only the middle one is magic:

nn.Module (eval)  ──torch.onnx.export──▶  model.onnx  ──onnx.checker + ORT──▶  verified artifact

The export knobs that matter

torch.onnx.export(
    model,                      # .eval() — training layers behave differently
    (dummy_input,),             # shapes get traced through this path
    "model.onnx",
    input_names=["input"],      # NAME YOUR TENSORS
    output_names=["logits"],
    opset_version=17,           # pin it; verify the runtime supports it
    do_constant_folding=True,   # fold constant subgraphs at export time
    dynamic_axes={"input": {0: "batch"}, "logits": {0: "batch"}},
)

with torch.no_grad():
    ref = model(x).numpy()
got = sess.run(None, {"input": x.numpy()})[0]
assert np.allclose(ref, got, atol=1e-5)      # never trust an unchecked export

What a clean export looks like

Linear → Gemm, ReLU → Relu, no aten:: remnants, no Python control flow. If you see chains of Slice/Squeeze/Unsqueeze/Concat around what should be one op, your module is fragmented — fix the module, not the runtime.

Other paths in

04 · Graph Optimizations

Consider y = relu(xW+b). A runtime may fuse these operations or precompute constants while preserving the output. Verify equivalence before and after optimization. A smaller node count is useful only when the model still answers correctly.

At session creation ORT rewrites your graph. The trigger to see it: point optimized_model_filepath at a file and the session writes the post-optimization graph to disk.

so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
so.optimized_model_filepath = "opt_all.onnx"     # audit what the runtime actually runs
sess = ort.InferenceSession("convnet.onnx", so, providers=["CPUExecutionProvider"])

The notebook proves this on a real model: node lists at each optimization level, a constant-folding demo, and a latency comparison. Expect the fusion levels to collapse Conv + BatchNormalization + Relu — the exact resulting op names are runtime- and EP-dependent, which is why you dump the graph instead of trusting folklore.

05 · Execution Providers and Graph Partitioning

Predict a partition: if a GPU provider supports MatMul and Add but not a custom final operation, the first nodes can run on GPU while the last falls back to CPU. The session may succeed while device transfers reduce performance.

An Execution Provider is a backend that implements kernels. At session creation ORT walks the graph and asks each EP, in priority order, can you run this node? Contiguous supported runs become subgraphs; everything else falls back — ultimately to the CPU. At boundaries, tensors are copied between devices.

avail = ort.get_available_providers()          # what this wheel can do
providers = (["TensorrtExecutionProvider", "CUDAExecutionProvider", "CPUExecutionProvider"]
             if "CUDAExecutionProvider" in avail else ["CPUExecutionProvider"])
sess = ort.InferenceSession("model.onnx", providers=providers)
print(sess.get_providers())                     # the truth after partitioning

Provider cheat sheet

06 · Sessions, IOBinding and Memory

A session owns the loaded graph and execution plan. Reuse it across requests when practical. IOBinding controls where input and output buffers live; it helps only when it avoids real copies or allocations in the measured path.

A session owns three expensive things: the optimized graph, the kernels, and memory arenas. Per call, the costs are input copy-in, output allocation, and kernel launches. Two session options and one API control nearly all of it:

IOBinding: stop copying

Normal sess.run() copies inputs into runtime buffers and allocates fresh outputs. IOBinding hands over memory directly. On CPU the win is modest; the same API is how GPU inference avoids PCIe round-trips.

io = sess.io_binding()
io.bind_cpu_input("X", x)            # input stays where it is
io.bind_output("Y", "cpu")           # runtime-owned output buffer
sess.run_with_iobinding(io)
y = io.copy_outputs_to_cpu()[0]      # one copy, only when you ask

# fixed shapes: own the output buffer yourself
out = ort.OrtValue.ortvalue_from_numpy(np.empty((32, 128), dtype=np.float32))
io.bind_output(out)                  # runtime writes straight into your array

07 · Quantization: Smaller Numbers, Measured Trade-offs

Calculate one quantized value. With scale 0.1 and zero point 0, real value 1.2 maps to integer 12. A value of 1.24 also rounds to 12, so both decode near 1.2. Quantization saves space by accepting this controlled loss of precision.

Neural networks tolerate small weight perturbations, and most hardware has several times the INT8 throughput of FP32. Quantization converts fp32 models to int8 — roughly 4× smaller, often faster — for a bounded accuracy cost you get to measure.

from onnxruntime.quantization import quantize_dynamic, quantize_static, QuantType, QuantFormat, CalibrationDataReader

quantize_dynamic("fp32.onnx", "int8_dynamic.onnx", weight_type=QuantType.QInt8)   # no data

quantize_static("fp32.onnx", "int8_qdq.onnx", calibration_data_reader=MyCalib(),
                quant_format=QuantFormat.QDQ, activation_type=QuantType.QUInt8,
                weight_type=QuantType.QInt8, per_channel=True)                     # calibrated

08 · Dynamic Shapes and the ORT Format

Dynamic means a dimension may change between runs; it does not mean every dimension is unconstrained. A model can accept batch sizes 1 and 8 while requiring feature width 768. Name which axes vary before configuring profiles or testing exports.

Weights are fixed at export; shapes are a deployment decision. A dimension is either fixed ([4, 512] — fastest to plan, breaks on other sizes) or symbolic (["batch", 512] — flexible, re-plans per shape).

Shape-specialized EPs turn this into a contract. TensorRT builds an engine per shape range, and the profile you give it decides which batch sizes get optimized kernels:

("TensorrtExecutionProvider", {
    "trt_fp16_enable": True,
    "trt_engine_cache_enable": True,          # rebuilds happen once, not per process
    "trt_engine_cache_path": "./trt_cache",
    "trt_profile_min_shapes": "input:1x256",
    "trt_profile_opt_shapes": "input:16x256", # optimize for your MEDIAN batch
    "trt_profile_max_shapes": "input:64x256", # required — never leave unbounded
})

The .ort format: freeze the optimization

python -m onnxruntime.tools.convert_onnx_models_to_ort model.onnx     # writes model.ort
python -m onnxruntime.tools.symbolic_shape_infer --input m.onnx --output m_shaped.onnx --auto_merge

The conversion runs the optimizer and freezes the result into ORT's flatbuffer format — faster loads, no dead graph. But it is a build artifact, not a portable model: it is tied to the runtime build and EP set that produced it. Generate it in CI for a known target.

09 · Profiling and Debugging

Move from symptom to evidence. If latency rises, first confirm input shapes and provider placement. Then inspect the trace for expensive nodes, copies, or repeated allocations. A profile is a timeline of measured events, not an automatic diagnosis.

Three tools replace guessing: the profiler, the log severity knob, and a mental catalog of error messages.

so = ort.SessionOptions()
so.enable_profiling = True
so.profile_file_prefix = "ortprof"
sess = ort.InferenceSession(model, so, providers=["CPUExecutionProvider"])

for _ in range(5):
    sess.run(None, {"X": x})

path = sess.end_profiling()      # Chrome-trace JSON: every kernel, with duration
                                 # opens in chrome://tracing, Perfetto, or in pandas

Read the profile like a budget: for a MatMul-heavy model, MatMul should dominate and elementwise ops should be noise. If a Cast chain, a Transpose pair, or a Memcpy tops the table, you have found your optimization target. On multi-EP setups, each event's provider argument tells you where the node actually ran — the tool for proving silent CPU fallback.

Where It Earns Its Place

The full notebook set

1. What ONNX Runtime Actually Is — hand-built graph, session lifecycle
2. The ONNX Format Up Close — protobuf, shape inference, IR/opset skew
3. Exporting from PyTorch — dynamic axes, verification
4. Graph Optimizations — levels, fusion, constant folding
5. Execution Providers — selection, fallback, proof via profiler
6. Sessions, IOBinding and Memory — threads, arenas, zero-copy
7. Quantization — dynamic vs static QDQ, measured
8. Dynamic Shapes and ORT Format — shape inference, TRT profiles, .ort
9. Profiling and Debugging — traces, severity, error field guide