Edge Inference Benchmark Lab — FP32 vs FP16 vs INT8 on Jetson Orin Nano

Introduction

In this article I am going to use the MobileNetV2 deep learning model from the ONNX model zoo [1] to benchmark the inference performance on NVIDIA Jetson Orin Nano Developer Kit. The main focus is to compare the performance of FP32, TF32, FP16 and INT8 precision modes in terms of latency, throughput, accuracy and power consumption.

The goal is to provide insights into efficient model deployment on edge devices, and to share lessons learned from the benchmarking process. The article is intended for Edge AI enthusiasts, engineers, and researchers who are interested in optimizing deep learning models for edge devices.

This is a learning project, and all data presented in the article comes from my own measurements on the NVIDIA Jetson Orin Nano Developer Kit. The repository containing the code and results is available in my GitHub repository.

This article focuses on three main measurements:

1. Latency measurement: how much time we gain by reducing precision, with and without using CUDA Graph.
2. Accuracy evaluation using the ImageNetV2 dataset.
3. Measuring power consumption at idle and during inference.

About the project setup

Model

MobileNetV2 is a very efficient computer vision model in terms of speed and size and hence are ideal for embedded mobile applications, which makes it a good choice for the benchmark test on NVIDIA Jetson Orin Nano. It performs image classification, it takes images as input and classifies the major object in the image into a set of pre-defined classes. It is trained on the ImageNet dataset, which contains images from 1000 classes. Its characteristic is the inverted residual with linear bottleneck. It replaces a full convolutional operator with a factorized version that splits convolution into two separate layers. The first layer is called a depthwise convolution, it performs lightweight filtering by applying a single convolutional filter per input channel. The second layer is a 1 × 1 convolution, called a pointwise convolution, which is responsible for building new features through computing linear combinations of the input channels [2].

Hardware & Software

For the measurements I use my NVIDIA Jetson Orin Nano Developer Kit, which is a small form factor edge device with a 6-core Arm CPU and a 1024-core Ampere GPU. The Jetson Orin Nano Developer Kit is designed for AI applications and provides a good balance between performance and power efficiency. Compared to the Orin NX or AGX Orin, the Nano has no Deep Learning Accelerator (DLA), so all DLA-based methodologies should be avoided. Memory is unified, which means no PCIe ‘host <-> device’ copy, GPU and the CPU address the same LPDDR5. [3]

If you are using an NVIDIA Jetson Orin for the first time, make sure that you have the latest bootloder version on your device. After the first run of my computer I realised that Super Mode was not active and could not be activated. The JetPack 7.2 installer misdetected the Orin Nano Developer Kit and omitted the 25 W and MAXN SUPER power profiles, capping the board at 15 W / 624 MHz. I tried editing the nvpmodel config, symlinking the correct configuration files, but it did not help. Although nvpmodel -q showed 25 W, the actual clocks did not change, which makes me think that the clock ceilings come from the bootloader. In the end, a firmware update including installing the latest nvidia-l4t-bootloader version fixed the issue, moving L4T from 39.2.0 to 39.2.1 (R39 revision 2.0 to 2.1). The environment setup before and after.

Use the following commands to check the current power mode and the GPU/EMC clocks. The GPU clock is the clock of the CUDA cores, while the EMC clock is the clock of the LPDDR5 memory controller. The EMC clock is important because it determines the memory bandwidth, which is a bottleneck for many deep learning models.

sudo nvpmodel -q
# 0: 25W, 1: 15W, 2: MAXN-SUPER
sudo jetson_clocks --show | grep -E 'GPU|EMC'
# For 25 W:, should be: GPU 918000000   EMC 3199000000

The board actively manages frequency based on the power mode (nvpmodel) and thermal state, optizing using a technique called dvfs (dynamic voltage and frequency scaling). It is a very useful technique to save energy during idle time, when the board is not under load. However, for latency benchmarking it can introduce extra latency increase, that we can eliminate by pinning the clock to the maximum using the following command:

sudo jetson_clocks

Therefore, every single measurement row records the nvpmodel mode and whether jetson_clocks was applied. Later we will see its effect on power consumption and the throughput.

The benchmark included FP32, TF32, FP16 and INT8, where TF32 is a Tensor Core format with FP32 range and FP16-ish precision. By default TensorRT uses TF32 in an FP32 engine unless --noTF32 is passed. Making this distinction consciously is what stops us from reporting a TF32 run as the FP32 baseline.

Dataset & Preprocessing

For accuracy evaluation I used the ImageNetV2 matched-frequency dataset [6], [7]. It contains 10,000 images 10 images per 1000 classes and I use a 1000-image calibration set, 9000 for eval without any overlap between those two. I deliberately used PIL instead of OpenCV during the preprocessing, because the two produce different pixel values for the same resize, and torchvision’s reference accuracy came from the PIL path. Mixing them might manufacture a 1-2% “quantization loss” that is really just a resize bug. [9] The results include a fingerprint for the split to prove that two runs used the same images. The split is also saved in a json file with the fingerprint, so it is traceable in the repository.

Build pipeline

Let’s consider the following build pipeline:

  PyTorch nn.Module          ← source code (Python, eager, dynamic)
         │  torch.onnx.export / torch.export
         ▼
  ONNX graph (.onnx)         ← portable IR (static graph, typed tensors, opset-versioned)
         │  TensorRT builder  (optimizing compiler + autotuner)
         ▼
  TensorRT engine (.plan)    ← target-specific binary (NOT portable)
         │  TensorRT runtime
         ▼
  CUDA kernels               ← the actual machine code, dispatched to SMs
         │
         ▼
  Ampere SMs + Tensor Cores  ← silicon

MobileNetV2 is available in an intermediate representation (IR) static graph format (.onnx extension) on Hugging Face. In the model used here, mobilenetv2-12.onnx, the 12 describes the opset version, which defines the semantics of operators such as Conv. An ONNX graph contains:

• Nodes: operations such as Conv, Add, and Clip
• Initializers: learned weights and biases stored as constants

To understand the architecture of the network I used polygraphy.

To make the model specific to the target hardware, I used trtexec to create the engine files (.plan) for the different data types: FP32, TF32, FP16 and INT8. I fixed the min, opt and max shape to 1x3x224x224 and added the --noTF32 flag to build the FP32 engine.

Quantization for INT8

INT8 has 256 values to work with, so every tensor needs a scale: a number mapping its floating-point range onto that grid. Compared to weights and their ranges, activations are not known and depend on the input, so they need to be measured. That measurement is called calibration. We push a sample of real images through the network and record how large each activation actually gets.

I used 512 images from the calibration split, preprocessed through the same code path evaluation uses. For choosing where to clip I used the entropy method. Entropy picks the clip point that loses the least information.

python scripts/quantize_int8.py --n 512 --method entropy

scripts/quantize_int8.py — calibration set selection, the ModelOpt [5] call and the node census it prints afterwards.

The output is an ONNX file, with QuantizeLinear/DequantizeLinear pairs inserted that carry the scales. This is called explicit path: the scales live in the graph, so TensorRT has no discretion about which layers run in INT8. The older approach called implicit path (a calibrator handed to the builder, deprecated since TensorRT 10.1 [4]) lets TensorRT quietly keep layers in FP16 where that is faster — sensible when shipping, useless for a benchmark whose whole question is what actually executed.

After quantization:

Each quantized tensor gets a QuantizeLinear + DequantizeLinear pair, so 105 pairs = 210 extra nodes. These are not extra work at runtime — TensorRT reads them as precision annotations and folds them into the kernels. The graph gets bigger; the engine shouldn’t. There are 105 Q/DQ pairs, but only 64 nodes are considered “quantized”.

FP32, TF32, FP16, INT8 Latency check

To measure only inference latency we measure how fast the model performs without any pre-processing or post-processing steps. At this point we don’t use real photos, only memory garbage to see how fast the neural network performs from the input layer to the output layer.

For each run we:

• Fix the min, opt and max shape to: 1x3x224x224
• Define 2000 ms warmup
• Run the inference for 30 seconds
• Pin GPU clock to max

Performance summary without CUDA Graph

Source: results/run/20260919T211117Z_7eb0b09/summary.md for FP32/TF32/FP16 and results/run/20260920T102737Z_bff3273/summary.md for INT8. Both carry the build and benchmark commands, the raw trtexec logs and the per-layer exports.

Performance summary with CUDA Graph

• FP16 remains effectively free from an accuracy perspective while significantly improving latency and throughput.
• INT8 keeps high performance with a small accuracy loss, around 0.78 points of Top-1 on this dataset.
• CUDA Graph reduces the host-side overhead substantially in the INT8 run, improving the median latency from about 0.809 ms to about 0.626 ms.
• The current run confirms the expected pattern: the INT8 graph-enabled configuration is the fastest measured variant in this benchmark set.

Learned after running CUDA Graph:

The enqueue median dropped from 0.5371 ms to 0.0078 ms, which is a ~69× reduction for INT8. This enqueue collapse is expected as one graph launch replaces 57 individual ones.

FP32, TF32, FP16, INT8 accuracy check

MobileNetV2’s original validation set is ImageNet and torchvision reports a Top-1 accuracy of 71.878% and a Top-5 accuracy of 90.286% on the original ImageNet validation set. [8] The ImageNetV2 dataset is designed to be more challenging, so the Top-1 accuracy on ImageNetV2 is expected to be lower than on the original ImageNet validation set. [6] I saw reports of natural 11% to 14% drop in top-1 accuracy with ImageNetV2, which is also reflected in my own measurements.

The FP32 accuracy baseline

The ran the FP32 accuracy baseline test both on TensorRT and through ONNX Runtime on CPU. The latter one is an independent implementation. With this measurement, I can validate that my TensorRT FP32 engine is correct, so the baseline can be locked to:

FP32, TF32, FP16, INT8 accuracy

Source: results/run/20260920T102737Z_bff3273/summary.md. The evaluation split is pinned by fingerprint in data/split_v1.json

• FP16 costs nothing measurable in accuracy and delivers 2.32× the throughput
• INT8 costs 0.78 points of Top-1 accuracy and delivers 2.99× the throughput
• As we lower precision, we capture a smaller share of the theoretical gain — 34%, 29%, 19%. The hardware’s arithmetic ceiling rises 16× while the delivered speedup rises 2.99×, which clearly indicates that the bottleneck is not arithmetic but memory bandwidth.

Power consumption

In the previous chapter we could observe the trade-off between latency, throughput and accuracy. Discussing only quality and speed trade-offs assumes that energy is available without any limits. However, as soon as we run devices on battery with limited energy resources, we introduce a new variable called power, measured in watts. This brings us to the next measurements.

The sampler is framecost/telemetry.py and the measurement script is scripts/measure_energy.py.

NVIDIA Jetson Orin Nano is equipped with many sensors that can be read out in real time by running the built-in tegrastats function. It captures a lot of useful telemetry: instantaneous temperature, power, EMC and GPU frequency, used and available RAM. From this data, we can calculate mean and maximum values, energy per frame, as well as behavior at idle and during inference.

In addition to the onboard measurement, I installed a power meter between the Jetson and the socket. This gives the total power that the Jetson consumes from the grid (or would do from a battery).

I used the WM03-DE model from Ningbo with the following parameters:

• EU type: Imax = 16A, Vn= 230V ~, P_max = 3680W
• Power display range: 0.5W-3680W
• Power Accuracy: ±2%

P_board is what VDD_IN measures: the DC input to the devkit after the barrel jack, sensed by the carrier-board INA3221, so it covers module and carrier loads. P_wall is what the WM03-DE measures at the socket, including PSU loss.

A note on the naming: tegrastats labels that channel VDD_IN, which names the rail rather than the quantity flowing on it. Since every number in these tables is a power in watts, I call it P_board and keep VDD_IN for the raw field in the code and the result files.

The decisive factor is the energy per frame, which we can call efficiency. It is calculated as follows:

n_inferences differs per run, since the throughput is different for each precision. For example for INT8 at 25 W: 9.754 W × 123.5 s = 1204.6 J, divided by 184,488 = 6.53 mJ.

Repo reference: all results are documented in result.json, and the calculation is made by telemetry.py, which is called by measure_energy.py, the script used for the following measurements. The results are automatically saved with the following naming convention: <timestamp_in_ISO_form>_<model>_<precision>_<power_mode>_<dvfs|pinned>_<last_short_commit_hash>

Efficiency with pinned vs unpinned GPU clock

Source: results/energy/summary-e3e0867e26c9.md.

Before benchmarking the models, it is worth measuring what difference it makes to pin the GPU clock to maximum versus using the built-in DVFS (dynamic voltage and frequency scaling) setting.

Unpinned vs. pinned

P_wall / P_board ratio

• The results show that pinning clocks costs 2.29 W at idle and buys only 0.7% throughput. Under continuous load it’s 0.7% more work for 1.5% more power, so energy per frame is marginally worse pinned. The entire cost lands at idle, the board consumes 2.9 W more while doing nothing.
• That means 13.8 mJ/frame at the wall, against 10.8 mJ/frame from tegrastats.

Efficiency with FP16 and INT8 for different power modes

Source: results/energy/summary-6fc2633c8be2.md, six runs.

As we concluded in the previous section, the DVFS version is more efficient, so all six of the following runs were executed with DVFS on. All six started at 46–47 °C, with idle baselines within 1% of each other.

FP16

INT8

Interpretation

• FP16 behaves as predicted — 15 W is the most efficient: 62% of MAXN’s throughput for 57% of the power, giving 7.3% better energy per frame. Slower is more efficient.
• INT8 inverts it: 25 W is the optimum on P_board, and MAXN_SUPER at the wall. 15 W is the worst of the three. That’s the opposite of the FP16 ordering. The difference between 25 W and MAXN_SUPER is very small, under 1%, which can be inside noise. But the direction of the shift is systematic, because the fixed 1.73 W is paid per second and so it penalises slower configurations.
• The practical statement: the optimal power mode depends on the precision you deploy.

Conclusion

NVIDIA Jetson Orin Nano is a very capable edge device, and the MobileNetV2 model is a good candidate to get started with model benchmarking. The results show that FP16 and INT8 precision modes can significantly improve inference performance while maintaining acceptable accuracy levels. CUDA Graph can further reduce host-side overhead, improving latency and throughput.

The power consumption measurements show that the optimal power mode depends on the precision used, and not always the lower power mode results in better efficiency. As we saw, pinning the GPU clock to maximum is a sound approach to get consistent accuracy results, however for deployment scenarios where efficiency is key, it is not recommended to pin the clocks to maximum, as it increases idle power consumption.

References

External sources only. Everything measured in this article links inline to the artifact in the repository that produced it.

[1] ONNX model zoo, mobilenetv2-12. https://huggingface.co/onnxmodelzoo/mobilenetv2-12

[2] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR 2018. arXiv:1801.04381. https://arxiv.org/abs/1801.04381

[3] NVIDIA. Jetson Orin Nano Super Developer Kit. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit/

[4] NVIDIA. TensorRT 10.1.0 Release Notes. https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/release-notes-10/10.1.0.html — deprecates the INT8 implicit quantization and calibrator APIs, IInt8EntropyCalibrator2 among them, recommending explicit quantization instead.

[5] NVIDIA Model Optimizer, ONNX quantization guide. https://nvidia.github.io/Model-Optimizer/guides/_onnx_quantization.html (repository: https://github.com/NVIDIA/Model-Optimizer, formerly TensorRT-Model-Optimizer)

[6] B. Recht, R. Roelofs, L. Schmidt, V. Shankar. Do ImageNet Classifiers Generalize to ImageNet? ICML 2019. arXiv:1902.10811. https://arxiv.org/abs/1902.10811

[7] ImageNetV2 matched-frequency set. https://huggingface.co/datasets/vaishaal/ImageNetV2/resolve/main/imagenetv2-matched-frequency.tar.gz

[8] torchvision, mobilenet_v2 — MobileNet_V2_Weights.IMAGENET1K_V1, 71.878% top-1 / 90.286% top-5 on ImageNet-1K. https://docs.pytorch.org/vision/main/models/generated/torchvision.models.mobilenet_v2.html

[9] Saksham Sinha’s Medium article on image reading and resizing. https://medium.com/data-science/image-read-and-resize-with-opencv-tensorflow-and-pil-3e0f29b992be