Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Automotive 3d Object Detection Benchmark Automation #142

Closed
10 of 11 tasks
Tracked by #141
arjunsuresh opened this issue Jan 24, 2025 · 2 comments
Closed
10 of 11 tasks
Tracked by #141

Support Automotive 3d Object Detection Benchmark Automation #142

arjunsuresh opened this issue Jan 24, 2025 · 2 comments

Comments

@arjunsuresh
Copy link
Collaborator

arjunsuresh commented Jan 24, 2025

  • Add MLC scripts for validation dataset download
  • Add MLC scripts for calibration dataset download
  • Add MLC scripts for model download
  • Add MLC scripts for benchmark run
  • Add MLC scripts for accuracy verification
  • Add MLC scripts for compliance runs
  • Document the benchmark details in inference docs page
  • Document the benchmark run commands in inference docs
  • Update the README in the Inference GitHub repo with the relevant MLC commands
  • Update the GitHub action for calibration dataset download (calibration dataset download is not checked in the GitHub action for reference implementation)
  • Upload valid submission log files to the unofficial results repository
@arjunsuresh arjunsuresh changed the title Support Automotive 3d Object Detection Automation Support Automotive 3d Object Detection Benchmark Automation Jan 24, 2025
@arjunsuresh arjunsuresh moved this to In progress in MLPerf Automation Jan 28, 2025
@anandhu-eng
Copy link
Contributor

README in inference repo has been updated in this PR

@anandhu-eng
Copy link
Contributor

anandhu-eng commented Feb 25, 2025

Hi @arjunsuresh , I think this issue can be closed. The GitHub runner storage is insufficient for downloading the waymo calibration dataset. We could open a separate issue for adding the calibration dataset downloads for large datasets(to be implemented when we get run-pod instance).

Edit: Created a separate issue for larger calibration datasets here

@github-project-automation github-project-automation bot moved this from In progress to Done in MLPerf Automation Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

2 participants