Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RuntimeError: list index out of range on RetinaNet #2103

Open
kingjuno opened this issue Feb 11, 2025 · 1 comment
Open

RuntimeError: list index out of range on RetinaNet #2103

kingjuno opened this issue Feb 11, 2025 · 1 comment

Comments

@kingjuno
Copy link

(mlc) ➜ mlcommons mlc run script --tags=run-mlperf,inference,_r5.0-dev
--model=retinanet
--implementation=reference
--framework=pytorch
--category=edge
--scenario=Offline
--execution_mode=valid
--device=cpu
--quiet

[2025-02-11 16:55:23,348 main.py:159 WARNING] - Warning: /home/junikki/MLC/repos/GATEOverflow@mlperf-automations/meta.yaml not found. Skipping...
....
...
......
.......
Using MLCommons Inference source from '/home/junikki/MLC/repos/local/cache/get-git-repo_509d019e/inference'

[2025-02-11 16:56:38,692 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-dataset-openimages_77ccb076
[2025-02-11 16:56:38,693 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-dataset-openimages/run.sh from tmp-run.sh
./openimages_mlperf.sh -d /home/junikki/MLC/repos/local/cache/get-dataset-openimages_77ccb076/install -m 50
Downloading images: 100%|███████████████████████| 50/50 [00:13<00:00,  3.80it/s]
Converting annotations to COCO format...
/home/junikki/MLC/repos/local/cache/get-git-repo_509d019e/inference/vision/classification_and_detection/tools/openimages.py:159: FutureWarning: factorize with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version.
  annotations["image_id"] = pd.factorize(annotations["ImageID"].tolist())[0]
[2025-02-11 16:57:07,463 module.py:5481 INFO] -              ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-dataset-openimages/customize.py
[2025-02-11 16:57:07,517 module.py:560 INFO] -       * mlcr mlperf,mlcommons,inference,source,src
[2025-02-11 16:57:07,525 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-mlperf-inference-src_7e7276f9/mlc-cached-state.json
[2025-02-11 16:57:07,593 module.py:560 INFO] -       * mlcr get,generic-python-lib,_pycocotools
[2025-02-11 16:57:07,609 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:07,610 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:07,611 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:07,611 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:07,612 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:07,612 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:07,794 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 2.0.8
[2025-02-11 16:57:07,819 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:07,821 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:07,822 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:07,822 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:07,825 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_1077e419/mlc-cached-state.json
[2025-02-11 16:57:07,891 module.py:560 INFO] -       * mlcr get,generic-python-lib,_opencv-python
[2025-02-11 16:57:07,905 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:07,906 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:07,907 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:07,907 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:07,908 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:07,908 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:08,031 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 4.11.0.86
[2025-02-11 16:57:08,050 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:08,051 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,052 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,052 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,055 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_1791e3ac/mlc-cached-state.json
[2025-02-11 16:57:08,113 module.py:560 INFO] -       * mlcr get,generic-python-lib,_pillow
[2025-02-11 16:57:08,128 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:08,129 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,129 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,130 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,130 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:08,130 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:08,240 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 11.1.0
[2025-02-11 16:57:08,259 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:08,261 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,261 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,261 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,264 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_7fe994bd/mlc-cached-state.json
[2025-02-11 16:57:08,333 module.py:560 INFO] -       * mlcr get,generic-python-lib,_package.ujson
[2025-02-11 16:57:08,347 module.py:560 INFO] -         * mlcr detect,os
[2025-02-11 16:57:08,353 module.py:5334 INFO] -                ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,354 module.py:5335 INFO] -                ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-11 16:57:08,403 module.py:5481 INFO] -                ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-11 16:57:08,421 module.py:560 INFO] -         * mlcr detect,cpu
[2025-02-11 16:57:08,430 module.py:560 INFO] -           * mlcr detect,os
[2025-02-11 16:57:08,436 module.py:5334 INFO] -                  ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,436 module.py:5335 INFO] -                  ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-11 16:57:08,466 module.py:5481 INFO] -                  ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-11 16:57:08,483 module.py:5334 INFO] -                ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,483 module.py:5335 INFO] -                ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/run.sh from tmp-run.sh
[2025-02-11 16:57:08,542 module.py:5481 INFO] -                ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/customize.py
[2025-02-11 16:57:08,559 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:08,560 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,561 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,561 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,635 module.py:560 INFO] -         * mlcr get,generic-python-lib,_pip
[2025-02-11 16:57:08,647 module.py:560 INFO] -           * mlcr get,python3
[2025-02-11 16:57:08,648 module.py:1274 INFO] -                ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,649 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,649 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,650 module.py:5334 INFO] -                ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,650 module.py:5335 INFO] -                ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:08,771 module.py:5481 INFO] -                ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
              Detected version: 24.0
[2025-02-11 16:57:08,789 module.py:560 INFO] -           * mlcr get,python3
[2025-02-11 16:57:08,790 module.py:1274 INFO] -                ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,791 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,791 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,792 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_9e5b1656/mlc-cached-state.json
[2025-02-11 16:57:08,802 module.py:5334 INFO] -                  ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,802 module.py:5335 INFO] -                  ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh from tmp-run.sh
[2025-02-11 16:57:08,923 module.py:5481 INFO] -                  ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py

            Extra PIP CMD:   --break-system-packages 

[2025-02-11 16:57:08,931 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,931 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh from tmp-run.sh

/media/junikki/research/mlcommons/mlc/bin/python3 -m pip install "ujson" --break-system-packages
Collecting ujson
  Using cached ujson-5.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.3 kB)
Using cached ujson-5.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (53 kB)
Installing collected packages: ujson
Successfully installed ujson-5.10.0
[2025-02-11 16:57:11,081 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:11,082 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh from tmp-run.sh
[2025-02-11 16:57:11,185 module.py:5481 INFO] -              ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 5.10.0
[2025-02-11 16:57:11,278 module.py:560 INFO] -       * mlcr get,generic-python-lib,_numpy
[2025-02-11 16:57:11,291 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,292 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,293 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,293 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,294 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,294 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:11,400 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 1.26.4
[2025-02-11 16:57:11,415 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,416 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,416 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,417 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,417 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,417 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:11,523 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 1.26.4
[2025-02-11 16:57:11,550 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,552 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,552 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,552 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,555 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_b0eede39/mlc-cached-state.json
[2025-02-11 16:57:11,644 module.py:560 INFO] -       * mlcr get,generic-python-lib,_numpy
[2025-02-11 16:57:11,660 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,661 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,661 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,662 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,662 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,662 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:11,772 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 1.26.4
[2025-02-11 16:57:11,790 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,791 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,792 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,792 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,793 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,793 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:11,891 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 1.26.4
[2025-02-11 16:57:11,919 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,921 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,921 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,921 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,924 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_b0eede39/mlc-cached-state.json
Using MLCommons Inference source from '/home/junikki/MLC/repos/local/cache/get-git-repo_509d019e/inference'
[2025-02-11 16:57:11,933 module.py:5334 INFO] -            ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,933 module.py:5335 INFO] -            ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-preprocessed-dataset-openimages/run.sh from tmp-run.sh
INFO:coco:loaded 50 images, cache=True, already_preprocessed=False, took=4.2sec
[2025-02-11 16:57:16,447 module.py:5481 INFO] -            ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-preprocessed-dataset-openimages/customize.py
[2025-02-11 16:57:16,480 module.py:560 INFO] -     * mlcr generate,user-conf,mlperf,inference
[2025-02-11 16:57:16,492 module.py:560 INFO] -       * mlcr detect,os
[2025-02-11 16:57:16,501 module.py:5334 INFO] -              ! cd /media/junikki/research/mlcommons
[2025-02-11 16:57:16,501 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
....
....
***************************************************************************
CM script::benchmark-program/run.sh

Run Directory: /home/junikki/MLC/repos/local/cache/get-git-repo_509d019e/inference/vision/classification_and_detection

CMD: ./run_local.sh pytorch retinanet cpu --scenario Offline    --max-batchsize 1 --threads 8 --user_conf '/home/junikki/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/d60c5984361c4900b3dcedf09242df27.conf' --use_preprocessed_dataset --cache_dir /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9 --dataset-list /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/annotations/openimages-mlperf.json 2>&1 | tee '/home/junikki/MLC/repos/local/cache/get-mlperf-inference-results-dir_7dbef0e6/valid_results/junikki-reference-cpu-pytorch-v2.6.0-default_config/retinanet/offline/performance/run_1/console.out'; echo \${PIPESTATUS[0]} > exitstatus

[2025-02-11 16:57:19,099 module.py:5334 INFO] -          ! cd /media/junikki/research/mlcommons
[2025-02-11 16:57:19,099 module.py:5335 INFO] -          ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program/run-ubuntu.sh from tmp-run.sh

./run_local.sh pytorch retinanet cpu --scenario Offline    --max-batchsize 1 --threads 8 --user_conf '/home/junikki/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/d60c5984361c4900b3dcedf09242df27.conf' --use_preprocessed_dataset --cache_dir /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9 --dataset-list /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/annotations/openimages-mlperf.json 2>&1 | tee '/home/junikki/MLC/repos/local/cache/get-mlperf-inference-results-dir_7dbef0e6/valid_results/junikki-reference-cpu-pytorch-v2.6.0-default_config/retinanet/offline/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
python3 python/main.py --profile retinanet-pytorch --model "/home/junikki/MLC/repos/local/cache/download-file_3313d856/resnext50_32x4d_fpn.pth" --dataset-path /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9 --output "/home/junikki/MLC/repos/local/cache/get-mlperf-inference-results-dir_7dbef0e6/valid_results/junikki-reference-cpu-pytorch-v2.6.0-default_config/retinanet/offline/performance/run_1" --scenario Offline --max-batchsize 1 --threads 8 --user_conf /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/d60c5984361c4900b3dcedf09242df27.conf --use_preprocessed_dataset --cache_dir /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9 --dataset-list /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/annotations/openimages-mlperf.json
INFO:main:Namespace(dataset='openimages-800-retinanet', dataset_path='/home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9', dataset_list='/home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/annotations/openimages-mlperf.json', data_format=None, profile='retinanet-pytorch', scenario='Offline', max_batchsize=1, model='/home/junikki/MLC/repos/local/cache/download-file_3313d856/resnext50_32x4d_fpn.pth', output='/home/junikki/MLC/repos/local/cache/get-mlperf-inference-results-dir_7dbef0e6/valid_results/junikki-reference-cpu-pytorch-v2.6.0-default_config/retinanet/offline/performance/run_1', inputs=['image'], outputs=['boxes', 'labels', 'scores'], backend='pytorch-native', device=None, model_name='retinanet', threads=8, qps=None, cache=0, cache_dir='/home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9', preprocessed_dir=None, use_preprocessed_dataset=True, accuracy=False, find_peak_performance=False, debug=False, user_conf='/home/junikki/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/d60c5984361c4900b3dcedf09242df27.conf', audit_conf='audit.config', time=None, count=None, performance_sample_count=None, max_latency=None, samples_per_query=8)
INFO:coco:loaded 50 images, cache=0, already_preprocessed=True, took=0.0sec
/media/junikki/research/mlcommons/mlc/lib/python3.12/site-packages/torch/serialization.py:1434: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)
  warnings.warn(
/media/junikki/research/mlcommons/mlc/lib/python3.12/site-packages/torch/nn/modules/module.py:1750: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3637.)
  return forward_call(*args, **kwargs)
code/__torch__/model/retinanet.py:159: UserWarning: RetinaNet always returns a (Losses, Detections) tuple in scripting
INFO:main:starting TestScenario.Offline
ERROR:main:thread: failed on contentid=['/home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/01e714e8fbbc0d0c.jpg'], The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/model/anchor_utils.py", line 76, in forward
      size = grid_sizes[_26]
      stride = strides[_26]
      base_anchors = cell_anchors[_26]
                     ~~~~~~~~~~~~~~~~~ <--- HERE
      grid_height, grid_width, = size
      stride_height, stride_width, = stride

Traceback of TorchScript, original code (most recent call last):
  File "/Users/pablogonzalez/Desktop/MLCommons/training/single_stage_detector/ssd/model/anchor_utils.py", line 91, in forward
                             "feature maps passed and the number of sizes / aspect ratios specified.")
    
        for size, stride, base_anchors in zip(
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            grid_sizes, strides, cell_anchors
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        ):
        ~~
            grid_height, grid_width = size
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            stride_height, stride_width = stride
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            device = base_anchors.device
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
            # For output anchor, compute [x_center, y_center, x_center, y_center]
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            shifts_x = torch.arange(
            ~~~~~~~~~~~~~~~~~~~~~~~~
                0, grid_width, dtype=torch.float32, device=device
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            ) * stride_width
            ~~~~~~~~~~~~~~~~
            shifts_y = torch.arange(
            ~~~~~~~~~~~~~~~~~~~~~~~~
                0, grid_height, dtype=torch.float32, device=device
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            ) * stride_height
            ~~~~~~~~~~~~~~~~~
            shift_y, shift_x = torch.meshgrid(shifts_y, shifts_x)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            shift_x = shift_x.reshape(-1)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            shift_y = shift_y.reshape(-1)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            shifts = torch.stack((shift_x, shift_y, shift_x, shift_y), dim=1)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
            # For every (base anchor, output anchor) pair,
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            # offset each zero-centered base anchor by the center of the output anchor.
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            anchors.append(
            ~~~~~~~~~~~~~~~
                (shifts.view(-1, 1, 4) + base_anchors.view(1, -1, 4)).reshape(-1, 4)
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
            )
    
RuntimeError: list index out of range
@arjunsuresh
Copy link
Contributor

This is because of not using the "_full" variation and so only 50 images are loaded. Can you please try with this variation?

@anandhu-eng - can we make "_full" the default for retinanet as otherwise it always fails for >50 inputs.

@anandhu-eng anandhu-eng mentioned this issue Feb 18, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arjunsuresh @kingjuno