Fix Hyper scripts to perform hot runs #688
Conversation
A recent commit aimed at "Prevent cheating by Munich-based databases" (4a2f3fa) incorrectly modified Hyper's `query.py` scripts to restart the database server on each iteration, essentially making every run a cold run. This unfairly brought our results down by a large margin. This commit reverts the previous behavior which was already correct and there was absolutely no cheating involved.
|
@caetanosauer There was some discussion about what "cold" runs really mean in #667. I pushed #692 in response. Tl;dr is that today's Hyper submission is fine - now we'll only need to migrate all other submissions one-by-one from lukewarm to cold runs for fair comparison. Allow me to close this PR - thanks. PS: I removed "cheating" from the message of #659. This term was unnecessarily dramatic. |
|
@rschu1ze Thanks for your response and the update on the lukewarm definitions. The problem I mentioned in this PR still persists, though: There are no hot runs being measured for Hyper. Last time I measured, Hyper was second place behind Umbra in the "Hot Run" category. Now we are way down below, because all 3 repetitions are run on a freshly restarted Hyper. |
2 participants
A recent commit aimed at "Prevent cheating by Munich-based databases" (4a2f3fa) incorrectly modified Hyper's
query.pyscripts to restart the database server on each iteration, essentially making every run a cold run. This unfairly brought our results down by a large margin.This commit reverts the previous behavior which was already correct and there was absolutely no cheating involved.