Re: micro-benchmarking

From:

Patricia Shanahan <pats@acm.org>

Newsgroups:

comp.lang.java.programmer

Date:

Sat, 02 May 2009 13:04:43 -0700

Message-ID:

<KbWdnXt9yJTDOmHUnZ2dnUVZ_gednZ2d@earthlink.com>

Arved Sandstrom wrote:

Lew wrote:

Giovanni Azua wrote:

[ SNIP ]

A good idea (I think brought up by Tom) would be to measure each
iteration separately and then discard outliers by e.g. [sic]
discarding those that exceed the abs [sic] diff [sic] between the
mean and the stddev [sic].

That technique doesn't seem statistically valid.

In the first place, you'd have to use the outliers to calculate the
mean and "stddev".

I've seen techniques before that discard the endmost data points, but
never ones that required statistical analysis to decide what to
include or reject for the statistical analysis.

Doing this is acceptable if it's a step in identifying outliers for
examination, rather than being an automatic elimination step. What
Giovanni suggested might not be the statistical procedure of choice
however; something like Grubb's test would be common enough if your
clean data is normally distributed.

The outlier may represent something that needs doing periodically, and
that needs to be taken into account.

For example, one source of outliers I found when benchmarking some
memory intensive work was memory refresh. The operating system needs to
do routine timer work periodically. The JVM needs to run its garbage
collector.

I prefer to do enough runs that including or excluding any one run does
not make much difference in the mean. Of course, there can be an outlier
so far out that it has to be investigated.

Patricia