Issue with Onnx'd detectron2 mask-rcnn bounding boxes

[cwheeler-rr]

I have build a mask-rcnn in detectron2 with COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml as a back bone but have a bounding box miss match compared to the original image

When using the detectron2 model through the inbuilt methods like so:

cfg = get_cfg()
# Use a pre-trained Mask R-CNN model from the model zoo
cfg.merge_from_file("../../outputs/models/config_custom_2025-09-13 05:57:14.510549.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # confidence threshold
cfg.MODEL.WEIGHTS = "../../outputs/models/best_model.pth"
cfg.MODEL.DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

predictor = DefaultPredictor(cfg)

img = get_masks_no_norm("../temp_data/full_inspection/img1.png")
outputs = predictor(img)
instances = outputs["instances"].to("cpu")
boxes = instances.pred_boxes.tensor.numpy()
boxes_dict[name] = boxes

This will give bonding boxes for an image that are accurate and line up well with the prediction visulisations from training and testing.

However when using an onnx model created using the commands from detectron2s documentation:

python3 ./detectron2/tools/deploy/export_model.py --config "../../outputs/models/config_custom_2025-09-12 17:13:50.286475.yaml" --output ./ --export-method tracing --sample-image ../datasets/dataset1/test.png --format onnx MODEL.WEIGHTS ../../outputs/models/best_model.pth MODEL.DEVICE cuda

I get an onnx output but it performs differently to the original model.

I am then running this model with:

session = ort.InferenceSession(model_path)

ai_image = image_pre_process()

# Run inference
inp_dict = {"x.1": ai_image}
outputs = session.run(None, inp_dict)

Ive given an example of how these detection are different below the object we are detecting are small so even being 2-3 pixels out could lead to missing the thing we want to find.

Detecron2 boxes:

[277.0877,  222.06036, 291.78217, 248.23471
 [253.8955, 175.23312. 271.17606. 197.9112 ]

Onnx boxes:

[274.79288, 219.97232, 289.00266, 246.60608]
[250.75594, 174.43523, 268.79462, 197.45988]

Is there any reason you can think of why there would be a difference in the outputs from the Onnx model and the original detectron2 model?

Thank you for any help

[github-actions]

You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template.
The following information is missing: "Instructions To Reproduce the Issue and Full Logs"; "Your Environment";

[cwheeler-rr]

Environment:

---

sys.platform linux
Python 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]
numpy 2.2.6
detectron2 0.6 @/usr/local/lib/python3.10/dist-packages/detectron2
Compiler GCC 11.4
CUDA compiler not available
DETECTRON2_ENV_MODULE
PyTorch 2.7.1+cu126 @/usr/local/lib/python3.10/dist-packages/torch
PyTorch debug build False
torch._C._GLIBCXX_USE_CXX11_ABI True
GPU available Yes
GPU 0 NVIDIA GeForce RTX 4090 (arch=8.9)
Driver version 535.247.01
CUDA_HOME None - invalid!
Pillow 11.2.1
torchvision 0.22.1+cu126 @/usr/local/lib/python3.10/dist-packages/torchvision
torchvision arch flags /usr/local/lib/python3.10/dist-packages/torchvision/_C.so
fvcore 0.1.5.post20221221
iopath 0.1.9
cv2 4.11.0

---

PyTorch built with:

• GCC 11.2
• C++ Version: 201703
• Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 12.6
• NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
• CuDNN 90.5.1
• Magma 2.6.1
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=e2d141dbde55c2a4370fac5165b0561b6af4798b, CUDA_VERSION=12.6, CUDNN_VERSION=9.5.1, CXX_COMPILER=/opt/rh/gcc-toolset-11/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.7.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

Instructions To Reproduce the Issue and Full Logs:

Unsure if this is a common problem when any model is onnx'd with the script from detectron2 but the command bellow was the only one used when creating the model:

python3 ./detectron2/tools/deploy/export_model.py --config "../../outputs/models/config_custom_2025-09-12 17:13:50.286475.yaml" --output ./ --export-method tracing --sample-image ../datasets/dataset1/test.png --format onnx MODEL.WEIGHTS ../../outputs/models/best_model.pth MODEL.DEVICE cuda

I can provide the config file if needed