-
criterion performance measurements
-
-
overview
-
-
want to understand this report?
-
-
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 0.4847128241570614 |
- 0.4908948086617951 |
- 0.5109107636187348 |
-
-
- Standard deviation |
- 4.176360362081773e-3 |
- 1.3673801077204834e-2 |
- 2.0399166016711115e-2 |
-
-
-
-
-
- Outlying measurements have moderate
- (0.12244897959183673%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 8.799614949608082e-2 |
- 8.8629525776332e-2 |
- 8.900730617232591e-2 |
-
-
- Standard deviation |
- 5.807767007341921e-4 |
- 9.972547118101713e-4 |
- 1.5795410789369389e-3 |
-
-
-
-
-
- Outlying measurements have slight
- (5.536332179930795e-2%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 0.47682691878108013 |
- 0.506753976247042 |
- 0.5443583530510778 |
-
-
- Standard deviation |
- 3.818451484421398e-2 |
- 5.0367134962777595e-2 |
- 5.789626276543197e-2 |
-
-
-
-
-
- Outlying measurements have moderate
- (0.2672689853230438%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 9.506872849859133e-2 |
- 9.602444657299489e-2 |
- 9.696212148375674e-2 |
-
-
- Standard deviation |
- 1.615210336380169e-3 |
- 2.114096929002399e-3 |
- 2.8168711478317784e-3 |
-
-
-
-
-
- Outlying measurements have slight
- (5.53633217993079e-2%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 0.4247120479408722 |
- 0.4280377244169049 |
- 0.43268214948319816 |
-
-
- Standard deviation |
- 3.241674649906193e-3 |
- 5.614496247321413e-3 |
- 8.471046026295666e-3 |
-
-
-
-
-
- Outlying measurements have moderate
- (0.12244897959183673%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 5.923397785509686e-2 |
- 5.948915604019876e-2 |
- 5.979412212483465e-2 |
-
-
- Standard deviation |
- 4.0861921805163213e-4 |
- 6.414232614480645e-4 |
- 1.0769747057569558e-3 |
-
-
-
-
-
- Outlying measurements have slight
- (4.5351473922902494e-2%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 0.9063111016500747 |
- 0.9137464516100469 |
- 0.9240750379100428 |
-
-
- Standard deviation |
- 5.393903981006103e-3 |
- 1.1205636220453591e-2 |
- 1.542422323897618e-2 |
-
-
-
-
-
- Outlying measurements have moderate
- (0.16000000000000003%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 0.15430268911035178 |
- 0.1556137329307144 |
- 0.15695866184990012 |
-
-
- Standard deviation |
- 1.6966469325334782e-3 |
- 2.5261353593158393e-3 |
- 3.811514561890576e-3 |
-
-
-
-
-
- Outlying measurements have slight
- (7.100591715976311e-2%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 0.6628789246833398 |
- 0.7216568147112058 |
- 0.8090466152946546 |
-
-
- Standard deviation |
- 4.873506144831662e-2 |
- 9.838676547648781e-2 |
- 0.1460107560009224 |
-
-
-
-
-
- Outlying measurements have moderate
- (0.31776599608044276%)
- effect on estimated standard deviation.
-
-
-
-
-
-
- |
- lower bound |
- estimate |
- upper bound |
-
-
-
- OLS regression |
- xxx |
- xxx |
- xxx |
-
-
- R² goodness-of-fit |
- xxx |
- xxx |
- xxx |
-
-
- Mean execution time |
- 0.1600520425469252 |
- 0.162661628086318 |
- 0.16561311435429607 |
-
-
- Standard deviation |
- 3.8545745739126222e-3 |
- 5.242671865187316e-3 |
- 7.0081533232793924e-3 |
-
-
-
-
-
- Outlying measurements have slight
- (7.100591715976326e-2%)
- effect on estimated standard deviation.
-
-
-
-
-
In this report, each function benchmarked by criterion is assigned
- a section of its own. The charts in each section are active; if
- you hover your mouse over data points and annotations, you will see
- more details.
-
-
- - The chart on the left is a
- kernel
- density estimate (also known as a KDE) of time
- measurements. This graphs the probability of any given time
- measurement occurring. A spike indicates that a measurement of a
- particular time occurred; its height indicates how often that
- measurement was repeated.
-
- - The chart on the right is the raw data from which the kernel
- density estimate is built. The x axis indicates the
- number of loop iterations, while the y axis shows measured
- execution time for the given number of loop iterations. The
- line behind the values is the linear regression prediction of
- execution time for a given number of iterations. Ideally, all
- measurements will be on (or very near) this line.
-
-
-
Under the charts is a small table.
- The first two rows are the results of a linear regression run
- on the measurements displayed in the right-hand chart.
-
-
- - OLS regression indicates the
- time estimated for a single loop iteration using an ordinary
- least-squares regression model. This number is more accurate
- than the mean estimate below it, as it more effectively
- eliminates measurement overhead and other constant factors.
- - R² goodness-of-fit is a measure of how
- accurately the linear regression model fits the observed
- measurements. If the measurements are not too noisy, R²
- should lie between 0.99 and 1, indicating an excellent fit. If
- the number is below 0.99, something is confounding the accuracy
- of the linear model.
- - Mean execution time and standard deviation are
- statistics calculated from execution time
- divided by number of iterations.
-
-
-
We use a statistical technique called
- the bootstrap
- to provide confidence intervals on our estimates. The
- bootstrap-derived upper and lower bounds on estimates let you see
- how accurate we believe those estimates to be. (Hover the mouse
- over the table headers to see the confidence levels.)
-
-
A noisy benchmarking environment can cause some or many
- measurements to fall far from the mean. These outlying
- measurements can have a significant inflationary effect on the
- estimate of the standard deviation. We calculate and display an
- estimate of the extent to which the standard deviation has been
- inflated by outliers.
-
-
-
-