From: Stefan Hajnoczi <stefanha@redhat.com>
To: Vivek Goyal <vgoyal@redhat.com>
Cc: virtio-fs-list <virtio-fs@redhat.com>,
qemu-devel@nongnu.org,
"Dr. David Alan Gilbert" <dgilbert@redhat.com>
Subject: Re: tools/virtiofs: Multi threading seems to hurt performance
Date: Mon, 21 Sep 2020 17:57:33 +0100 [thread overview]
Message-ID: <20200921165733.GA189810@stefanha-x1.localdomain> (raw)
In-Reply-To: <20200921133944.GB13362@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]
On Mon, Sep 21, 2020 at 09:39:44AM -0400, Vivek Goyal wrote:
> On Mon, Sep 21, 2020 at 09:39:23AM +0100, Stefan Hajnoczi wrote:
> > On Fri, Sep 18, 2020 at 05:34:36PM -0400, Vivek Goyal wrote:
> > > And here are the comparision results. To me it seems that by default
> > > we should switch to 1 thread (Till we can figure out how to make
> > > multi thread performance better even when single process is doing
> > > I/O in client).
> >
> > Let's understand the reason before making changes.
> >
> > Questions:
> > * Is "1-thread" --thread-pool-size=1?
>
> Yes.
Okay, I wanted to make sure 1-thread is still going through the glib
thread pool. So it's the same code path regardless of the
--thread-pool-size= value. This suggests the performance issue is
related to timing side-effects like lock contention, thread scheduling,
etc.
> > * How do the kvm_stat vmexit counters compare?
>
> This should be same, isn't it. Changing number of threads serving should
> not change number of vmexits?
There is batching at the virtio and eventfd levels. I'm not sure if it's
coming into play here but you would see it by comparing vmexits and
eventfd reads. Having more threads can increase the number of
notifications and completion interrupt, which can make overall
performance worse in some cases.
> > * How does host mpstat -P ALL compare?
>
> Never used mpstat. Will try running it and see if I can get something
> meaningful.
Tools like top, vmstat, etc can give similar information. I'm wondering
what the host CPU utilization (guest/sys/user) looks like.
> But I suepct it has to do with thread pool implementation and possibly
> extra cacheline bouncing.
I think perf can record cacheline bounces if you want to check.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
next prev parent reply other threads:[~2020-09-21 16:58 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-18 21:34 tools/virtiofs: Multi threading seems to hurt performance Vivek Goyal
2020-09-21 8:39 ` Stefan Hajnoczi
2020-09-21 13:39 ` Vivek Goyal
2020-09-21 16:57 ` Stefan Hajnoczi [this message]
2020-09-21 8:50 ` Dr. David Alan Gilbert
2020-09-21 13:35 ` Vivek Goyal
2020-09-21 14:08 ` Daniel P. Berrangé
2020-09-21 15:32 ` Dr. David Alan Gilbert
2020-09-22 10:25 ` Dr. David Alan Gilbert
2020-09-22 17:47 ` Vivek Goyal
2020-09-24 21:33 ` Venegas Munoz, Jose Carlos
2020-09-24 22:10 ` virtiofs vs 9p performance(Re: tools/virtiofs: Multi threading seems to hurt performance) Vivek Goyal
2020-09-25 8:06 ` virtiofs vs 9p performance Christian Schoenebeck
2020-09-25 13:13 ` Vivek Goyal
2020-09-25 15:47 ` Christian Schoenebeck
2021-02-19 16:08 ` Can not set high msize with virtio-9p (Was: Re: virtiofs vs 9p performance) Vivek Goyal
2021-02-19 17:33 ` Christian Schoenebeck
2021-02-19 19:01 ` Vivek Goyal
2021-02-20 15:38 ` Christian Schoenebeck
2021-02-22 12:18 ` Greg Kurz
2021-02-22 15:08 ` Christian Schoenebeck
2021-02-22 17:11 ` Greg Kurz
2021-02-23 13:39 ` Christian Schoenebeck
2021-02-23 14:07 ` Michael S. Tsirkin
2021-02-24 15:16 ` Christian Schoenebeck
2021-02-24 15:43 ` Dominique Martinet
2021-02-26 13:49 ` Christian Schoenebeck
2021-02-27 0:03 ` Dominique Martinet
2021-03-03 14:04 ` Christian Schoenebeck
2021-03-03 14:50 ` Dominique Martinet
2021-03-05 14:57 ` Christian Schoenebeck
2020-09-25 12:41 ` virtiofs vs 9p performance(Re: tools/virtiofs: Multi threading seems to hurt performance) Dr. David Alan Gilbert
2020-09-25 13:04 ` Christian Schoenebeck
2020-09-25 13:05 ` Dr. David Alan Gilbert
2020-09-25 16:05 ` Christian Schoenebeck
2020-09-25 16:33 ` Christian Schoenebeck
2020-09-25 18:51 ` Dr. David Alan Gilbert
2020-09-27 12:14 ` Christian Schoenebeck
2020-09-29 13:03 ` Vivek Goyal
2020-09-29 13:28 ` Christian Schoenebeck
2020-09-29 13:49 ` Vivek Goyal
2020-09-29 13:59 ` Christian Schoenebeck
2020-09-29 13:17 ` Vivek Goyal
2020-09-29 13:49 ` [Virtio-fs] " Miklos Szeredi
2020-09-29 14:01 ` Vivek Goyal
2020-09-29 14:54 ` Miklos Szeredi
2020-09-29 15:28 ` Vivek Goyal
2020-09-25 12:11 ` tools/virtiofs: Multi threading seems to hurt performance Dr. David Alan Gilbert
2020-09-25 13:11 ` Vivek Goyal
2020-09-21 20:16 ` Vivek Goyal
2020-09-22 11:09 ` Dr. David Alan Gilbert
2020-09-22 22:56 ` Vivek Goyal
2020-09-23 12:50 ` [Virtio-fs] " Chirantan Ekbote
2020-09-23 12:59 ` Vivek Goyal
2020-09-25 11:35 ` Dr. David Alan Gilbert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200921165733.GA189810@stefanha-x1.localdomain \
--to=stefanha@redhat.com \
--cc=dgilbert@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=vgoyal@redhat.com \
--cc=virtio-fs@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).