Skip to content

Run regression tests on Travis #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 54 commits into
base: develop
Choose a base branch
from
Open

Run regression tests on Travis #35

wants to merge 54 commits into from

Conversation

@nightlark
Copy link
Contributor

Runs stencil4d-otf example using the dragonfly, fattree, and hypre, and torus network configurations, and compares the resulting output files to a baseline set of output files.

  • Some times check to make sure the value is within a tolerance; some floating point time values are very slightly (10^-6) different on Travis than when the same test and code is run on LC
  • Busy time in msg-stats files are ignored; they vary significantly between runs even on the same machine

nightlark and others added 30 commits July 10, 2019 11:57
@nightlark
Test mpirun and mpiexec commands
aeef947
@nightlark
Install mpich on Travis
8529034
@nightlark
Run TraceR simulation on Travis
2ddcb92
@nightlark
Test install TraceR on Travis
2026548
@nightlark
Pass arguments to make install on Travis
ee6280f
@nightlark
Return to starting directory before running TraceR
93c96c3
@nightlark
Add baseline regression test output files
399fa1e
@nightlark
Bash script to compare output files with regression test files
630c750
@nightlark
Merge pull request #1 from nightlark/travis-test-mpirun
c14fb6e 
Run TraceR with a dfly network on the stencil4d-otf example on Travis
@nightlark
Output files from dfly example run in tests/output folder
25ac0f3
@nightlark
Run check files script to compare regression files against output fil…
063b7c4 
…es for stencil4d dfly test
@nightlark
Move check to its own entry under script (exit status needs to fail b…
6d661fd 
…uild)
@nightlark
Create an output folder for results from running tests
30478ba
@nightlark
Output the diff for files that regression tests that don't match
5e50139
@nightlark
Switch to using git diff tool for word-level diff
6d00002
@nightlark
Merge branch 'ci-test' of https://github.com/nightlark/TraceR into ci…
2065c81 
…-test
@nightlark
Check that removing the git diff header works on Travis
c49cb9e
@nightlark
Test out float val extraction
7c35187
@nightlark
Tweak the detection of allowable diff lines and extraction of floatin…
17d3ac8 
…g point time num
@nightlark
Switch to alternate form of checking if return code for diff was 0
d988019
@nightlark
Add floating point value check
fe96166
@nightlark
Set tolerance check to something that should fail
0467fe1
@nightlark
Change tolerance check to something that should pass
ad28bdb
@nightlark
Clean-up recv_time check output
9bca43b
@nightlark
Fix bash error on echo
83ffb93
@nightlark
Add missing closing quote
dd630d3
@nightlark
Tweak debug output for diff comparison
821d797
@nightlark
Bash debug output
f553136
@nightlark
Tweak one of the checks
0e87168
@nightlark
Fix up the logic for detecting non-recv_time diff
c9a51f6
nightlark and others added 20 commits August 26, 2019 22:50
@nightlark
Typo in variable name fix
98060d2
@nightlark
Turn off verbose debug output for file diff checking
a49fab6
@nightlark
Run test with other network configurations
b1ffc26
@nightlark
Use the right conf file for hypreX_expressMesh, and use mpirun exit c…
2ea7191 
…ode for success/fail
@nightlark
Update setting return value for multi-line steps to avoid premature s…
3f871d8 
…cript exit
@nightlark
Apply suggestions from ShellCheck.net
f80bf19
@nightlark
Check msg-stats file for CI tests
63214f2
@nightlark
Tweak checking for failure
0c7a9ad
@nightlark
Fix failing for msg-stats file mismatches
d1b7ef0
@nightlark
Refactor msg-stat checking to handle different types of time fields
366e456
@nightlark
Add time tolerance check for very small variations on Travis
1ebaac0
@nightlark
Apply suggestions from ShellCheck.net
ed87517
@nightlark
Switch line splitting to use read -a
956c43b
@nightlark
Add files for checking TraceR stdout simulation prints
f21bf42
@nightlark
Create test output directory before writing to it
79b443c
@nightlark
Dump filtered output to log
7a564e3
@nightlark
Try outputting directly to console after piping output
6534c9c
@nightlark
Test running set
f8b3276
@nightlark
Remove early creation of output directory
34cecb6
@nightlark
Reduce the set of lines checked
8c5f8f2
@nightlark
Copy link
Contributor Author

The push build for this branch passes: https://travis-ci.com/nightlark/TraceR/builds/129162778

The failure is because #38 resulted in values changing in the CODES output files, as well as the TraceR WallTime values printed out.

@bhatele bhatele changed the base branch from master to develop November 19, 2020 13:13
@nightlark
Copy link
Contributor Author

Given recent changes to Travis, I don't think running tests there is as good as it used to be. Would be better to use GitHub Actions now.

@bhatele
Copy link
Member

bhatele commented Feb 24, 2021

Given recent changes to Travis, I don't think running tests there is as good as it used to be. Would be better to use GitHub Actions now.

Yes we are planning to move to GitHub actions soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nightlark @bhatele