nsys perf and eval

[malay-nagda]

What does this PR do ?

Evaluate perf between given boundaries.

Changelog

start = performance_config.get("eval_time_start_step")
    if start is None:
        start = max(1, int(len(steps) * performance_config.get("skip_first_percent_time", 0.1)))
    end = performance_config.get("eval_time_end_step")
    performance_result["metrics"]["current_avg_iter_time_ms"] = float(np.nanmean(current_iter_time_values[start:end]))
    performance_result["metrics"]["golden_avg_iter_time_ms"] = float(np.nanmean(golden_iter_time_values[start:end]))

GitHub Actions CI

See the CI sectionin the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

• Related to # (issue)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[ko3n1g]

Should we just refactor this to only use eval window instead of skip_n_steps? I'm wondering if there will be a case in future where we still need skip_n_steps?

[malay-nagda]

Should we just refactor this to only use eval window instead of skip_n_steps? I'm wondering if there will be a case in future where we still need skip_n_steps?

Wouldn't it be useful in case of variable number of steps like for convergence testing or when the set number of steps were not completed due to time limit or some crash- skip_n_percent can still calculate based on available steps?