Re: Interesting observation with network event notification and batching

[Wei Liu <wei.liu2@citrix.com>]

On Mon, Jul 01, 2013 at 05:53:27PM +0100, Andrew Bennieston wrote:
[...]
> >
> >Sadly enough, I found out today these sort of test seems to be quite
> >inconsistent. On a Intel 10G Nic the throughput is actually higher
> >without enforcing iperf / netperf to generate large packets.
> 
> When I have made performance measurements using iperf, I found that
> for a given point in the parameter space (e.g. for a fixed number of
> guests, interfaces, fixed parameters to iperf, fixed test run
> duration, etc.) the variation was typically _smaller than_ +/- 1
> Gbit/s on a 10G NIC.
> 

I was talking about virtual interface v.s. real hardware. The parameters
that maximize throughput for one case don't seem to be working for the
other case. The deviation for a specific interface is rather small.

> I notice that your results don't include any error bars or
> indication of standard deviation...
> 
> With this sort of data (or, really, any data) measuring at least 5
> times will help to get an idea of the fluctuations present (i.e. a
> measure of statistical uncertainty) by quoting a mean +/- standard
> deviation. Having the standard deviation (or other estimator for the
> uncertainty in the results) allows us to better determine how
> significant this difference in results really is.
> 
> For example, is the high throughput you quoted (~ 14 Gbit/s) an
> upward fluctuation, and the low value (~6) a downward fluctuation?
> Having a mean and standard deviation would allow us to determine
> just how (in)compatible these values are.
> 

I ran those tests for several times and picked the number that appeared
most. Anyway I will try to come up with better visualized graphs.

> Assuming a Gaussian distribution (and when sampled sufficient times,
> "everything" tends to a Gaussian) you have an almost 5% chance that
> a result lies more than 2 standard deviations from the mean (and a
> 0.3% chance that it lies more than 3 s.d. from the mean!). Results
> that appear "high" or "low" may, therefore, not be entirely
> unexpected. Having a measure of the standard deviation provides some
> basis against which to determine how likely it is that a measured
> value is just statistical fluctuation, or whether it is a
> significant result.
> 
> Another thing I noticed is that you're running the iperf test for
> only 5 seconds. I have found in the past that iperf (or, more
> likely, TCP) takes a while to "ramp up" (even with all parameters
> fixed e.g. "-l <size> -w <size>") and that tests run for 2 minutes
> or more (e.g. "-t 120") give much more stable results.
> 

Hmm... for me the lenght of the test doesn't make much difference,
that's why I've chosen such a short time. As you mentioned this I intend
to run the tests a big longer.

> Andrew.
> 
> >
> >
> >Wei.
> >
> >>Thanks
> >>Annie