Speedup Ninja backend with many extract_objects or targets #13879
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
get_target_generated_sources often calls File.from_built_relative on the same file, if it is used by many sources. This is a somewhat expensive call both CPU- and memory-wise, so cache the creation of build-directory files as well. Signed-off-by: Paolo Bonzini <[email protected]>
The deps and orderdeps are sorted on output, so there is no need to preserve their order. Signed-off-by: Paolo Bonzini <[email protected]>
bonzini
force-pushed
the
speedups
branch
3 times, most recently
from
03c53a4
to
c7bd527
Compare
Accumulate into lists that are passed by the caller, thus avoiding allocations and calls to extend() on recursive extract_objects(). Signed-off-by: Paolo Bonzini <[email protected]>
The proj_dir_to_build_root argument of determine_ext_objs() is always empty, remove it. Signed-off-by: Paolo Bonzini <[email protected]>
eli-schwartz
requested changes
Nov 6, 2024
mesonbuild/backend/backends.py
Outdated
Comment on lines
479
to
484
| proj_dir_to_build_root: str) -> T.Tuple[T.List[str], T.List[build.BuildTargetTypes]]: | ||
| obj_list: T.List[str] = [] | ||
| deps: T.List[build.BuildTargetTypes] = [] | ||
| proj_dir_to_build_root: str, | ||
| obj_list: T.List[str], deps: T.List[build.BuildTargetTypes]) -> None: |
There was a problem hiding this comment.
@dcbaker I suspect you may have some opinions about this.
Cache the results of is_source(), which is called almost 900000 times in a QEMU setup, and avoid reinventing the File.from_built_relative() wheel. Together this basically removes _determine_ext_objs() from the profile when building QEMU. While at it, fix the existing wrong mypy annotation on cached_object_by_name(). Signed-off-by: Paolo Bonzini <[email protected]>
proj_dir_to_build_root is empty by default, in fact always except on some cases of the VS2010 backend. Add it after the fact in flatten_object_list(), which reduces the numbers of os.path.join(). Signed-off-by: Paolo Bonzini <[email protected]>
bonzini
force-pushed
the
speedups
branch
3 times, most recently
from
46cb2e6
to
47d8542
Compare
version_compare can take a few milliseconds. If you have a thousand object files or a multiple thereof, it adds up. Signed-off-by: Paolo Bonzini <[email protected]>
bonzini
changed the title
Avoid expensive calls and loops, instead relying as much on Python
builtins as possible. Track whether any options need to be deduped
at flush_pre_post() time, and if not just concatenate pre, _container
and post.
Before:
ncalls tottime cumtime
19268 0.163 3.586 arglist.py:97(__init__)
45127 0.251 4.530 arglist.py:142(__iter__)
81866 3.623 5.013 arglist.py:108(flush_pre_post)
76618 3.793 5.338 arglist.py:273(__iadd__)
After:
35647 0.156 0.627 arglist.py:160(__iter__)
18674 0.211 3.442 arglist.py:97(__init__)
78998 2.627 3.603 arglist.py:116(flush_pre_post)
73774 3.605 5.049 arglist.py:292(__iadd__)
Signed-off-by: Paolo Bonzini <[email protected]>
Regexes can be surprisingly slow. This small change brings ninja_quote() from 12 to 3 seconds when building QEMU. Before: ncalls tottime percall cumtime percall 3734443 4.872 0.000 11.944 0.000 After: ncalls tottime percall cumtime percall 3595590 3.193 0.000 3.196 0.000 Signed-off-by: Paolo Bonzini <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This speeds up the target generation of QEMU, which is very slow.
2921 seconds (total time150134 seconds)