Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automation failing when pulling from huggingface #197

Closed
stopiamwarren opened this issue Feb 7, 2025 · 7 comments
Closed

Automation failing when pulling from huggingface #197

stopiamwarren opened this issue Feb 7, 2025 · 7 comments

Comments

@stopiamwarren
Copy link

When I attempt to run the llama2-70B benchmark natively, it will get to the hugging face login, ask me to login twice, and then error out.

mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev    --model=llama2-70b-99    --implementation=reference    --framework=pytorch    --category=datacenter    --scenario=Offline    --execution_mode=test    --device=cpu     --quiet    --test_query_count=10
[2025-02-07 13:12:32,751 module.py:560 INFO] - * mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
[2025-02-07 13:12:32,759 module.py:560 INFO] -   * mlcr detect,os
[2025-02-07 13:12:32,763 module.py:5340 INFO] -          ! cd /home/user1
[2025-02-07 13:12:32,763 module.py:5341 INFO] -          ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:32,809 module.py:5487 INFO] -          ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:32,829 module.py:560 INFO] -   * mlcr detect,cpu
[2025-02-07 13:12:32,836 module.py:560 INFO] -     * mlcr detect,os
[2025-02-07 13:12:32,839 module.py:5340 INFO] -            ! cd /home/user1
[2025-02-07 13:12:32,839 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:32,884 module.py:5487 INFO] -            ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:32,901 module.py:5340 INFO] -          ! cd /home/user1
[2025-02-07 13:12:32,901 module.py:5341 INFO] -          ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/run.sh from tmp-run.sh
[2025-02-07 13:12:33,119 module.py:5487 INFO] -          ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/customize.py
[2025-02-07 13:12:33,131 module.py:560 INFO] -   * mlcr get,python3
[2025-02-07 13:12:33,132 module.py:1274 INFO] -        ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:33,132 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:33,132 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:33,148 module.py:560 INFO] -   * mlcr get,mlcommons,inference,src
[2025-02-07 13:12:33,149 module.py:1274 INFO] -        ! load /home/user1/MLC/repos/local/cache/get-mlperf-inference-src_8c5be2df/mlc-cached-state.json
[2025-02-07 13:12:33,158 module.py:560 INFO] -   * mlcr get,sut,description
[2025-02-07 13:12:33,165 module.py:560 INFO] -     * mlcr detect,os
[2025-02-07 13:12:33,168 module.py:5340 INFO] -            ! cd /home/user1
[2025-02-07 13:12:33,168 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:33,215 module.py:5487 INFO] -            ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:33,236 module.py:560 INFO] -     * mlcr detect,cpu
[2025-02-07 13:12:33,243 module.py:560 INFO] -       * mlcr detect,os
[2025-02-07 13:12:33,246 module.py:5340 INFO] -              ! cd /home/user1
[2025-02-07 13:12:33,246 module.py:5341 INFO] -              ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:33,292 module.py:5487 INFO] -              ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:33,308 module.py:5340 INFO] -            ! cd /home/user1
[2025-02-07 13:12:33,308 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/run.sh from tmp-run.sh
[2025-02-07 13:12:33,480 module.py:5487 INFO] -            ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/customize.py
[2025-02-07 13:12:33,491 module.py:560 INFO] -     * mlcr get,python3
[2025-02-07 13:12:33,492 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:33,492 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:33,492 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:33,503 module.py:560 INFO] -     * mlcr get,compiler
[2025-02-07 13:12:33,504 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-llvm_ecc06aac/mlc-cached-state.json
[2025-02-07 13:12:33,558 module.py:560 INFO] -     * mlcr get,generic-python-lib,_package.dmiparser
[2025-02-07 13:12:33,568 module.py:560 INFO] -       * mlcr get,python3
[2025-02-07 13:12:33,569 module.py:1274 INFO] -            ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:33,569 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:33,569 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:33,569 module.py:5340 INFO] -            ! cd /home/user1
[2025-02-07 13:12:33,569 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-07 13:12:33,621 module.py:5487 INFO] -            ! call "detect_version" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
          Detected version: 5.1
[2025-02-07 13:12:33,633 module.py:560 INFO] -       * mlcr get,python3
[2025-02-07 13:12:33,633 module.py:1274 INFO] -            ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:33,634 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:33,634 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:33,634 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-generic-python-lib_40c47c3e/mlc-cached-state.json
[2025-02-07 13:12:33,641 module.py:560 INFO] -     * mlcr get,cache,dir,_name.mlperf-inference-sut-descriptions
[2025-02-07 13:12:33,642 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-cache-dir_e1579bbb/mlc-cached-state.json
Generating SUT description file for vkcompute0-pytorch
[2025-02-07 13:12:33,645 module.py:5487 INFO] -          ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-description/customize.py
[2025-02-07 13:12:33,656 module.py:560 INFO] -   * mlcr get,mlperf,inference,results,dir,_version.r5.0-dev
[2025-02-07 13:12:33,657 module.py:1274 INFO] -        ! load /home/user1/MLC/repos/local/cache/get-mlperf-inference-results-dir_ea370d70/mlc-cached-state.json
[2025-02-07 13:12:33,663 module.py:560 INFO] -   * mlcr install,pip-package,for-mlc-python,_package.tabulate
[2025-02-07 13:12:33,664 module.py:1274 INFO] -        ! load /home/user1/MLC/repos/local/cache/install-pip-package-for-mlc-python_4b52afc4/mlc-cached-state.json
[2025-02-07 13:12:33,670 module.py:560 INFO] -   * mlcr get,mlperf,inference,utils
[2025-02-07 13:12:33,687 module.py:560 INFO] -     * mlcr get,mlperf,inference,src
[2025-02-07 13:12:33,688 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-mlperf-inference-src_8c5be2df/mlc-cached-state.json
[2025-02-07 13:12:33,691 module.py:5487 INFO] -          ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /home/user1/MLC/repos/local/cache/get-git-repo_81be0e09/inference

Running loadgen scenario: Offline and mode: performance
[2025-02-07 13:12:33,819 module.py:560 INFO] - * mlcr app,mlperf,inference,generic,_reference,_llama2-70b-99,_pytorch,_cpu,_test,_r5.0-dev_default,_offline
[2025-02-07 13:12:33,830 module.py:560 INFO] -   * mlcr detect,os
[2025-02-07 13:12:33,833 module.py:5340 INFO] -          ! cd /home/user1
[2025-02-07 13:12:33,833 module.py:5341 INFO] -          ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:33,882 module.py:5487 INFO] -          ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:33,906 module.py:560 INFO] -   * mlcr get,sys-utils-cm
[2025-02-07 13:12:33,914 module.py:560 INFO] -     * mlcr detect,os
[2025-02-07 13:12:33,917 module.py:5340 INFO] -            ! cd /home/user1/MLC/repos/local/cache/get-sys-utils-cm_e571d819
[2025-02-07 13:12:33,917 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:33,963 module.py:5487 INFO] -            ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:33,991 module.py:560 INFO] -   * mlcr get,python
[2025-02-07 13:12:33,992 module.py:1274 INFO] -        ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:33,992 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:33,992 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:34,008 module.py:560 INFO] -   * mlcr get,mlcommons,inference,src
[2025-02-07 13:12:34,009 module.py:1274 INFO] -        ! load /home/user1/MLC/repos/local/cache/get-mlperf-inference-src_ed2d4648/mlc-cached-state.json
[2025-02-07 13:12:34,016 module.py:560 INFO] -   * mlcr get,mlperf,inference,utils
[2025-02-07 13:12:34,033 module.py:560 INFO] -     * mlcr get,mlperf,inference,src
[2025-02-07 13:12:34,034 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-mlperf-inference-src_ed2d4648/mlc-cached-state.json
[2025-02-07 13:12:34,037 module.py:5487 INFO] -          ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
[2025-02-07 13:12:34,141 module.py:560 INFO] -   * mlcr app,mlperf,reference,inference,_pytorch,_cpu,_llama2-70b-99,_offline,_fp32
[2025-02-07 13:12:34,150 module.py:560 INFO] -     * mlcr detect,os
[2025-02-07 13:12:34,153 module.py:5340 INFO] -            ! cd /home/user1/MLC/repos/local/cache/get-sys-utils-cm_e571d819
[2025-02-07 13:12:34,153 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:34,201 module.py:5487 INFO] -            ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:34,225 module.py:560 INFO] -     * mlcr detect,cpu
[2025-02-07 13:12:34,232 module.py:560 INFO] -       * mlcr detect,os
[2025-02-07 13:12:34,235 module.py:5340 INFO] -              ! cd /home/user1/MLC/repos/local/cache/get-sys-utils-cm_e571d819
[2025-02-07 13:12:34,235 module.py:5341 INFO] -              ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:34,281 module.py:5487 INFO] -              ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:34,301 module.py:5340 INFO] -            ! cd /home/user1/MLC/repos/local/cache/get-sys-utils-cm_e571d819
[2025-02-07 13:12:34,301 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/run.sh from tmp-run.sh
[2025-02-07 13:12:34,515 module.py:5487 INFO] -            ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/customize.py
[2025-02-07 13:12:34,526 module.py:560 INFO] -     * mlcr get,sys-utils-cm
[2025-02-07 13:12:34,534 module.py:560 INFO] -       * mlcr detect,os
[2025-02-07 13:12:34,538 module.py:5340 INFO] -              ! cd /home/user1/MLC/repos/local/cache/get-sys-utils-cm_e571d819
[2025-02-07 13:12:34,538 module.py:5341 INFO] -              ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:34,587 module.py:5487 INFO] -              ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:34,620 module.py:560 INFO] -     * mlcr get,python
[2025-02-07 13:12:34,620 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:34,621 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:34,621 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:34,673 module.py:560 INFO] -     * mlcr get,generic-python-lib,_torch
[2025-02-07 13:12:34,682 module.py:560 INFO] -       * mlcr get,python3
[2025-02-07 13:12:34,683 module.py:1274 INFO] -            ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:34,683 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:34,683 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:34,683 module.py:5340 INFO] -            ! cd /home/user1/MLC/repos/local/cache/get-sys-utils-cm_e571d819
[2025-02-07 13:12:34,683 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-07 13:12:34,736 module.py:5487 INFO] -            ! call "detect_version" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
          Detected version: 2.6.0
[2025-02-07 13:12:34,748 module.py:560 INFO] -       * mlcr get,python3
[2025-02-07 13:12:34,749 module.py:1274 INFO] -            ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:34,749 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:34,749 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:34,749 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-generic-python-lib_d557edb9/mlc-cached-state.json
[2025-02-07 13:12:34,802 module.py:560 INFO] -     * mlcr get,generic-python-lib,_torchvision
[2025-02-07 13:12:34,811 module.py:560 INFO] -       * mlcr get,python3
[2025-02-07 13:12:34,812 module.py:1274 INFO] -            ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:34,812 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:34,812 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:34,813 module.py:5340 INFO] -            ! cd /home/user1/MLC/repos/local/cache/get-sys-utils-cm_e571d819
[2025-02-07 13:12:34,813 module.py:5341 INFO] -            ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-07 13:12:34,866 module.py:5487 INFO] -            ! call "detect_version" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
          Detected version: 0.21.0
[2025-02-07 13:12:34,878 module.py:560 INFO] -       * mlcr get,python3
[2025-02-07 13:12:34,879 module.py:1274 INFO] -            ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:34,879 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:34,879 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:34,879 module.py:1274 INFO] -          ! load /home/user1/MLC/repos/local/cache/get-generic-python-lib_db560766/mlc-cached-state.json
[2025-02-07 13:12:34,898 module.py:560 INFO] -     * mlcr get,ml-model,llama2,raw,_pytorch,_fp32
[2025-02-07 13:12:34,911 module.py:560 INFO] -       * mlcr get,ml-model,huggingface,zoo,_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf
[2025-02-07 13:12:34,921 module.py:560 INFO] -         * mlcr get,python3
[2025-02-07 13:12:34,922 module.py:1274 INFO] -              ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:34,922 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:34,922 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:34,975 module.py:560 INFO] -         * mlcr get,generic-python-lib,_huggingface_hub
[2025-02-07 13:12:34,985 module.py:560 INFO] -           * mlcr get,python3
[2025-02-07 13:12:34,985 module.py:1274 INFO] -                ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:34,985 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:34,985 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:34,986 module.py:5340 INFO] -                ! cd /home/user1/MLC/repos/local/cache/get-ml-model-huggingface-zoo_1a5f58d4
[2025-02-07 13:12:34,986 module.py:5341 INFO] -                ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-07 13:12:35,039 module.py:5487 INFO] -                ! call "detect_version" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
              Detected version: 0.28.1
[2025-02-07 13:12:35,051 module.py:560 INFO] -           * mlcr get,python3
[2025-02-07 13:12:35,051 module.py:1274 INFO] -                ! load /home/user1/MLC/repos/local/cache/get-python3_e12aeb9b/mlc-cached-state.json
[2025-02-07 13:12:35,051 module.py:2220 INFO] - Path to Python: /home/user1/mlc/bin/python3
[2025-02-07 13:12:35,052 module.py:2220 INFO] - Python version: 3.9.16
[2025-02-07 13:12:35,052 module.py:1274 INFO] -              ! load /home/user1/MLC/repos/local/cache/get-generic-python-lib_0dcd4fa3/mlc-cached-state.json
[2025-02-07 13:12:35,064 module.py:560 INFO] -         * mlcr get,git,repo,_lfs,_repo.https://huggingface.co/meta-llama/Llama-2-70b-chat-hf
[2025-02-07 13:12:35,072 module.py:560 INFO] -           * mlcr detect,os
[2025-02-07 13:12:35,075 module.py:5340 INFO] -                  ! cd /home/user1/MLC/repos/local/cache/get-git-repo_c328db7d
[2025-02-07 13:12:35,075 module.py:5341 INFO] -                  ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-07 13:12:35,125 module.py:5487 INFO] -                  ! call "postprocess" from /home/user1/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-07 13:12:35,197 module.py:560 INFO] -           * mlcr get,generic,sys-util,_git-lfs
[2025-02-07 13:12:35,199 module.py:1274 INFO] -                ! load /home/user1/MLC/repos/local/cache/get-generic-sys-util_9dc070c1/mlc-cached-state.json
[2025-02-07 13:12:35,202 module.py:5340 INFO] -                ! cd /home/user1/MLC/repos/local/cache/get-git-repo_c328db7d
[2025-02-07 13:12:35,202 module.py:5341 INFO] -                ! call /home/user1/MLC/repos/mlcommons@mlperf-automations/script/get-git-repo/run.sh from tmp-run.sh
/home/user1/MLC/repos/local/cache/get-git-repo_c328db7d
rm -rf repo
******************************************************
Current directory: /home/user1/MLC/repos/local/cache/get-git-repo_c328db7d

Cloning Llama-2-70b-chat-hf from https://huggingface.co/meta-llama/Llama-2-70b-chat-hf

git clone  --recurse-submodules https://huggingface.co/meta-llama/Llama-2-70b-chat-hf --depth 5 repo

Cloning into 'repo'...
Username for 'https://huggingface.co': <username removed>
Password for 'https://<username removed>@huggingface.co':
Cloning into 'repo'...
Username for 'https://huggingface.co': <username removed>
Password for 'https://<username removed>@huggingface.co':
Traceback (most recent call last):
  File "/home/user1/mlc/bin/mlcr", line 8, in <module>
    sys.exit(mlcr())
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1354, in mlcr
    main()
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1441, in main
    res = method(run_args)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1196, in run
    return self.call_script_module_function("run", run_args)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1176, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
    r = self._run(i)
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1772, in _run
    r = customize_code.preprocess(ii)
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 286, in preprocess
    r = mlc.access(ii)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 96, in access
    result = method(self, options)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1196, in run
    return self.call_script_module_function("run", run_args)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1176, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
    r = self._run(i)
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1842, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3538, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3708, in _run_deps
    r = self.action_object.access(ii)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 96, in access
    result = method(self, options)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1196, in run
    return self.call_script_module_function("run", run_args)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1176, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
    r = self._run(i)
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1631, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3538, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3708, in _run_deps
    r = self.action_object.access(ii)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 96, in access
    result = method(self, options)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1196, in run
    return self.call_script_module_function("run", run_args)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1176, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
    r = self._run(i)
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1842, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3538, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3708, in _run_deps
    r = self.action_object.access(ii)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 96, in access
    result = method(self, options)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1196, in run
    return self.call_script_module_function("run", run_args)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1176, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
    r = self._run(i)
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1631, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3538, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/user1/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3708, in _run_deps
    r = self.action_object.access(ii)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 96, in access
    result = method(self, options)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1196, in run
    return self.call_script_module_function("run", run_args)
  File "/home/user1/mlc/lib64/python3.9/site-packages/mlc/main.py", line 1186, in call_script_module_function
    raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.main.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-git-repo, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.
@arjunsuresh
Copy link
Collaborator

@anandhu-eng can you please replace the huggingface download with the MLCommons download for llama2-70b?

@arjunsuresh
Copy link
Collaborator

@stopiamwarren have you got the access to the llama2 models in huggingface?

@stopiamwarren
Copy link
Author

@anandhu-eng can you please replace the huggingface download with the MLCommons download for llama2-70b?

I'm not sure how I would do this

@stopiamwarren
Copy link
Author

@stopiamwarren have you got the access to the llama2 models in huggingface?

Yes

Image

@arjunsuresh
Copy link
Collaborator

@stopiamwarren Then thats strange as the git clone should have worked. Did it fail immediately or after a while? The download is over 300G and so it can take hours.

One other thing to try is to using hugglingface-cli to authenticate and after that you won't need to put password during the run.
https://github.com/mlcommons/mlperf-automations/blob/main/.github/workflows/test-mlperf-inference-llama2.yml#L33

@anandhu-eng
Copy link
Contributor

Hi @stopiamwarren , were you able to execute the script successfully? One other possibility could be insufficient storage. As @arjunsuresh pointed out, did it fail after a while?

@stopiamwarren
Copy link
Author

The issue resolved itself with no explanation after I pulled the most recent version of the repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@arjunsuresh @stopiamwarren @anandhu-eng