Skip to content

PerfFlowAspect Reader + CPU/Mem Usage #144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
spencer-gre wants to merge 54 commits into
base: develop
Choose a base branch
from
Draft

PerfFlowAspect Reader + CPU/Mem Usage #144

spencer-gre wants to merge 54 commits into from

Conversation

@spencer-gre
Copy link

@spencer-gre spencer-gre commented Jul 17, 2024

Improves upon #101 adding ability to read in a PerfFlowAspect log with cpu and memory usage.

r-yin and others added 30 commits July 3, 2024 16:56
perf_reader initial commit
50afc48
graph updated
3f71b2e
@slabasan
add front-level reader API, add PFA reader example
4f5223e 
- add node object to dataframe
@r-yin
graph updated
7a6dcc2
@r-yin
added pfw testfile with one iteration of laghos output
0d306b5
@r-yin
pfw file processor added
5e93a00
@r-yin
pfw to literal converter
bb9c9c9
@r-yin
sample json
587a17a
@r-yin
pfw folder processor
105c4c9
@r-yin
finalized pfw to caliper reader
99cf379
@slabasan
cleaning up code
37d1b05
@slabasan
update example for pfa reader
65f6aa3
@slabasan
add perfflowaspect unit test
d0cd0dc 
- reorganize data files
- restore ts and dur into frame
@slabasan
add AMS MPI pfa unit test (concat pfw files per rank), flake, black
078d83d
@slabasan
restore kwargs in from_hdf
dea17c0
@slabasan
update comment
1b13b2e
Add perfflowaspect object reader and update related tests and scripts
8f73af7
updated the test file. add extra fields for the time unit and matedat…
6c94f8f 
…a. Renamed and moved pfw test file to the correct folder
perf_reader initial commit
73414e2
graph updated
41a1281
@slabasan
add front-level reader API, add PFA reader example
4659c1f 
- add node object to dataframe
@r-yin
graph updated
d8d8faf
@r-yin
added pfw testfile with one iteration of laghos output
ff65cd0
@r-yin
pfw file processor added
48cf77b
@r-yin
pfw to literal converter
700eba0
@r-yin
sample json
57a471c
@r-yin
pfw folder processor
cf6fdb8
@r-yin
finalized pfw to caliper reader
9e67d6f
@slabasan
cleaning up code
68065f6
@slabasan
update example for pfa reader
4ed1e7e
@spencer-gre spencer-gre marked this pull request as ready for review July 17, 2024 17:22
@ilumsden ilumsden added area-readers Issues and PRs involving Hatchet's data readers priority-normal Normal priority issues and PRs type-feature Requests for new features or PRs which implement new features labels Jul 17, 2024
@spencer-gre
Read All PFW Files
b07fbf3 
The PerfFlowAspect Reader used to require that .pfw files
(PerfFlowAspect Traces) conform to JSON standard. However,
PerfFlowAspect does not generate files that are consistent with JSON.
This commit modifies a PerfFlowAspect trace file that does not conform
to JSON standard - by adding the closing brace and removing the trailing
comma.
@spencer-gre
Copy link
Author

spencer-gre commented Jul 17, 2024

I've added b07fbf3 to this pull request. This commit ensures that all PerfFlowAspect Trace files can be read without manual modification.

@spencer-gre spencer-gre marked this pull request as draft July 25, 2024 21:22
@tpatki
Copy link
Member

tpatki commented Jul 25, 2024 •
edited
Loading

This PR will supersede #101 (we will close #101).
This is currently a draft PR and not ready for review yet.

@spencer-gre @alizalisan @loudsun1997:

  • Add the object format reader to this PR. Convert time to seconds for reporting as per Hatchet/Thicket's convention.
  • Rename the readers to be perfflowaspect_array_reader and perfflowaspect_object_reader
  • Add CPU and Memory Usage statistics to object format reader.
  • Remove any JSON example files from here
  • Add adiak metadata parsing and test cases (new pfw files) for just the object format reader in order to support Thicket
  • Add documentation on the two different formats for PFA and compact format usage.

@tpatki tpatki mentioned this pull request Jul 25, 2024
Closed
@spencer-gre
Cleanup/Prep for Object Reader
b1461b5 
This commit follows remarks that llnl#144 will supersede llnl#101, and
specifically prepares for the integration of both an array reader for
PerfFlowAspect Array Trace Files (already present) and PerfFlowAspect
Object Trace Files (to be added). Additionally removes .json files in
the pytest data directory as no longer needed.
@spencer-gre
Removing PFW JSON Files
faf48b2 
Missed two in the prior commit. Refer there for full explanation.
@spencer-gre
Copy link
Author

spencer-gre commented Jul 25, 2024

@tpatki

b1461b5 and faf48b2 address the following points of the roadmap:

  1. Remove any JSON example files from here
  2. Renamed perfflowaspect_reader to perfflowaspect_array_reader (and once @loudsun1997 adds object reader, that will be named accordingly).

loudsun1997 and others added 7 commits July 28, 2024 23:51
@loudsun1997
added us to s converter. fixed object reader tests
6a90b58
@loudsun1997
Merge branch 'pfa-reader-pr' into pfa-hatchet-pr-conflict-resolved
84cf2f7
@spencer-gre
Cleanup after merge
f5723ba 
This branch cleans up conflicts/redundancies introduced after object
reader added. Specifically, JSON files were reintroduced, and references
to the original perfflowaspect_reader was being used in the graphframe
(and not perfflowaspect_array_reader)
@loudsun1997
added tests. validated things I wroked on are properly merged and wor…
636086b 
…king
@loudsun1997
Merge branch 'pfa-hatchet-pr-conflict-resolved' of https://github.com…
b11f3dd 
…/spencer-gre/hatchet into pfa-reader-pr
@loudsun1997
Merge pull request #1 from spencer-gre/pfa-hatchet-pr-conflict-resolved
8f09fd0 
Added test for us to s conversion
@spencer-gre
flake8
a1e5771
@ilumsden ilumsden added the status-work-in-progress PR is currently being worked on label Sep 4, 2024
@spencer-gre
Copy link
Author

spencer-gre commented Dec 4, 2025

as of e99b000

-> adiak metadata parsing added
-> object format supported via singular perfflowaspect reader that automatically detects object vs array
-> cpu/mem usage

@tpatki @slabasan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area-readers Issues and PRs involving Hatchet's data readers priority-normal Normal priority issues and PRs status-work-in-progress PR is currently being worked on type-feature Requests for new features or PRs which implement new features

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@spencer-gre @tpatki @ilumsden @slabasan @r-yin @loudsun1997