On Mon, Sep 21, 2020 at 09:39:44AM -0400, Vivek Goyal wrote:
> On Mon, Sep 21, 2020 at 09:39:23AM +0100, Stefan Hajnoczi wrote:
> > On Fri, Sep 18, 2020 at 05:34:36PM -0400, Vivek Goyal wrote:
> > > And here are the comparision results. To me it seems that by default
> > > we should switch to 1 thread (Till we can figure out how to make
> > > multi thread performance better even when single process is doing
> > > I/O in client).
> > 
> > Let's understand the reason before making changes.
> > 
> > Questions:
> >  * Is "1-thread" --thread-pool-size=1?
> 
> Yes.

Okay, I wanted to make sure 1-thread is still going through the glib
thread pool. So it's the same code path regardless of the
--thread-pool-size= value. This suggests the performance issue is
related to timing side-effects like lock contention, thread scheduling,
etc.

> >  * How do the kvm_stat vmexit counters compare?
> 
> This should be same, isn't it. Changing number of threads serving should
> not change number of vmexits?

There is batching at the virtio and eventfd levels. I'm not sure if it's
coming into play here but you would see it by comparing vmexits and
eventfd reads. Having more threads can increase the number of
notifications and completion interrupt, which can make overall
performance worse in some cases.

> >  * How does host mpstat -P ALL compare?
> 
> Never used mpstat. Will try running it and see if I can get something
> meaningful.

Tools like top, vmstat, etc can give similar information. I'm wondering
what the host CPU utilization (guest/sys/user) looks like.

> But I suepct it has to do with thread pool implementation and possibly
> extra cacheline bouncing.

I think perf can record cacheline bounces if you want to check.

Stefan