From: Vivek Goyal <vgoyal@redhat.com> To: Stefan Hajnoczi <stefanha@redhat.com> Cc: virtio-fs-list <virtio-fs@redhat.com>, qemu-devel@nongnu.org, "Dr. David Alan Gilbert" <dgilbert@redhat.com> Subject: Re: tools/virtiofs: Multi threading seems to hurt performance Date: Mon, 21 Sep 2020 09:39:44 -0400 [thread overview] Message-ID: <20200921133944.GB13362@redhat.com> (raw) In-Reply-To: <20200921083923.GA71121@stefanha-x1.localdomain> On Mon, Sep 21, 2020 at 09:39:23AM +0100, Stefan Hajnoczi wrote: > On Fri, Sep 18, 2020 at 05:34:36PM -0400, Vivek Goyal wrote: > > And here are the comparision results. To me it seems that by default > > we should switch to 1 thread (Till we can figure out how to make > > multi thread performance better even when single process is doing > > I/O in client). > > Let's understand the reason before making changes. > > Questions: > * Is "1-thread" --thread-pool-size=1? Yes. > * Was DAX enabled? No. > * How does cache=none perform? I just ran random read workload with cache=none. cache-none randread-psync 45(MiB/s) 11k cache-none-1-thread randread-psync 63(MiB/s) 15k With 1 thread it offers more IOPS. > * Does commenting out vu_queue_get_avail_bytes() + fuse_log("%s: > Queue %d gave evalue: %zx available: in: %u out: %u\n") in > fv_queue_thread help? Will try that. > * How do the kvm_stat vmexit counters compare? This should be same, isn't it. Changing number of threads serving should not change number of vmexits? > * How does host mpstat -P ALL compare? Never used mpstat. Will try running it and see if I can get something meaningful. > * How does host perf record -a compare? Will try it. I feel this might be too big and too verbose to get something meaningful. > * Does the Rust virtiofsd show the same pattern (it doesn't use glib > thread pools)? No idea. Never tried rust implementation of virtiofsd. But I suepct it has to do with thread pool implementation and possibly extra cacheline bouncing. Thanks Vivek
WARNING: multiple messages have this Message-ID (diff)
From: Vivek Goyal <vgoyal@redhat.com> To: Stefan Hajnoczi <stefanha@redhat.com> Cc: virtio-fs-list <virtio-fs@redhat.com>, qemu-devel@nongnu.org Subject: Re: [Virtio-fs] tools/virtiofs: Multi threading seems to hurt performance Date: Mon, 21 Sep 2020 09:39:44 -0400 [thread overview] Message-ID: <20200921133944.GB13362@redhat.com> (raw) In-Reply-To: <20200921083923.GA71121@stefanha-x1.localdomain> On Mon, Sep 21, 2020 at 09:39:23AM +0100, Stefan Hajnoczi wrote: > On Fri, Sep 18, 2020 at 05:34:36PM -0400, Vivek Goyal wrote: > > And here are the comparision results. To me it seems that by default > > we should switch to 1 thread (Till we can figure out how to make > > multi thread performance better even when single process is doing > > I/O in client). > > Let's understand the reason before making changes. > > Questions: > * Is "1-thread" --thread-pool-size=1? Yes. > * Was DAX enabled? No. > * How does cache=none perform? I just ran random read workload with cache=none. cache-none randread-psync 45(MiB/s) 11k cache-none-1-thread randread-psync 63(MiB/s) 15k With 1 thread it offers more IOPS. > * Does commenting out vu_queue_get_avail_bytes() + fuse_log("%s: > Queue %d gave evalue: %zx available: in: %u out: %u\n") in > fv_queue_thread help? Will try that. > * How do the kvm_stat vmexit counters compare? This should be same, isn't it. Changing number of threads serving should not change number of vmexits? > * How does host mpstat -P ALL compare? Never used mpstat. Will try running it and see if I can get something meaningful. > * How does host perf record -a compare? Will try it. I feel this might be too big and too verbose to get something meaningful. > * Does the Rust virtiofsd show the same pattern (it doesn't use glib > thread pools)? No idea. Never tried rust implementation of virtiofsd. But I suepct it has to do with thread pool implementation and possibly extra cacheline bouncing. Thanks Vivek
next prev parent reply other threads:[~2020-09-21 13:41 UTC|newest] Thread overview: 107+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-09-18 21:34 tools/virtiofs: Multi threading seems to hurt performance Vivek Goyal 2020-09-18 21:34 ` [Virtio-fs] " Vivek Goyal 2020-09-21 8:39 ` Stefan Hajnoczi 2020-09-21 8:39 ` [Virtio-fs] " Stefan Hajnoczi 2020-09-21 13:39 ` Vivek Goyal [this message] 2020-09-21 13:39 ` Vivek Goyal 2020-09-21 16:57 ` Stefan Hajnoczi 2020-09-21 16:57 ` [Virtio-fs] " Stefan Hajnoczi 2020-09-21 8:50 ` Dr. David Alan Gilbert 2020-09-21 8:50 ` [Virtio-fs] " Dr. David Alan Gilbert 2020-09-21 13:35 ` Vivek Goyal 2020-09-21 13:35 ` [Virtio-fs] " Vivek Goyal 2020-09-21 14:08 ` Daniel P. Berrangé 2020-09-21 14:08 ` [Virtio-fs] " Daniel P. Berrangé 2020-09-21 15:32 ` Dr. David Alan Gilbert 2020-09-21 15:32 ` [Virtio-fs] " Dr. David Alan Gilbert 2020-09-22 10:25 ` Dr. David Alan Gilbert 2020-09-22 10:25 ` [Virtio-fs] " Dr. David Alan Gilbert 2020-09-22 17:47 ` Vivek Goyal 2020-09-22 17:47 ` [Virtio-fs] " Vivek Goyal 2020-09-24 21:33 ` Venegas Munoz, Jose Carlos 2020-09-24 21:33 ` [Virtio-fs] " Venegas Munoz, Jose Carlos 2020-09-24 22:10 ` virtiofs vs 9p performance(Re: tools/virtiofs: Multi threading seems to hurt performance) Vivek Goyal 2020-09-24 22:10 ` [Virtio-fs] " Vivek Goyal 2020-09-25 8:06 ` virtiofs vs 9p performance Christian Schoenebeck 2020-09-25 8:06 ` [Virtio-fs] " Christian Schoenebeck 2020-09-25 13:13 ` Vivek Goyal 2020-09-25 13:13 ` [Virtio-fs] " Vivek Goyal 2020-09-25 15:47 ` Christian Schoenebeck 2020-09-25 15:47 ` [Virtio-fs] " Christian Schoenebeck 2021-02-19 16:08 ` Can not set high msize with virtio-9p (Was: Re: virtiofs vs 9p performance) Vivek Goyal 2021-02-19 16:08 ` [Virtio-fs] " Vivek Goyal 2021-02-19 17:33 ` Christian Schoenebeck 2021-02-19 17:33 ` [Virtio-fs] " Christian Schoenebeck 2021-02-19 19:01 ` Vivek Goyal 2021-02-19 19:01 ` [Virtio-fs] " Vivek Goyal 2021-02-20 15:38 ` Christian Schoenebeck 2021-02-20 15:38 ` [Virtio-fs] " Christian Schoenebeck 2021-02-22 12:18 ` Greg Kurz 2021-02-22 12:18 ` [Virtio-fs] " Greg Kurz 2021-02-22 15:08 ` Christian Schoenebeck 2021-02-22 15:08 ` [Virtio-fs] " Christian Schoenebeck 2021-02-22 17:11 ` Greg Kurz 2021-02-22 17:11 ` [Virtio-fs] " Greg Kurz 2021-02-23 13:39 ` Christian Schoenebeck 2021-02-23 13:39 ` [Virtio-fs] " Christian Schoenebeck 2021-02-23 14:07 ` Michael S. Tsirkin 2021-02-23 14:07 ` [Virtio-fs] " Michael S. Tsirkin 2021-02-24 15:16 ` Christian Schoenebeck 2021-02-24 15:16 ` [Virtio-fs] " Christian Schoenebeck 2021-02-24 15:43 ` Dominique Martinet 2021-02-24 15:43 ` [Virtio-fs] " Dominique Martinet 2021-02-26 13:49 ` Christian Schoenebeck 2021-02-26 13:49 ` [Virtio-fs] " Christian Schoenebeck 2021-02-27 0:03 ` Dominique Martinet 2021-02-27 0:03 ` [Virtio-fs] " Dominique Martinet 2021-03-03 14:04 ` Christian Schoenebeck 2021-03-03 14:04 ` [Virtio-fs] " Christian Schoenebeck 2021-03-03 14:50 ` Dominique Martinet 2021-03-03 14:50 ` [Virtio-fs] " Dominique Martinet 2021-03-05 14:57 ` Christian Schoenebeck 2021-03-05 14:57 ` [Virtio-fs] " Christian Schoenebeck 2020-09-25 12:41 ` virtiofs vs 9p performance(Re: tools/virtiofs: Multi threading seems to hurt performance) Dr. David Alan Gilbert 2020-09-25 12:41 ` [Virtio-fs] " Dr. David Alan Gilbert 2020-09-25 13:04 ` Christian Schoenebeck 2020-09-25 13:04 ` [Virtio-fs] " Christian Schoenebeck 2020-09-25 13:05 ` Dr. David Alan Gilbert 2020-09-25 13:05 ` [Virtio-fs] " Dr. David Alan Gilbert 2020-09-25 16:05 ` Christian Schoenebeck 2020-09-25 16:05 ` [Virtio-fs] " Christian Schoenebeck 2020-09-25 16:33 ` Christian Schoenebeck 2020-09-25 16:33 ` [Virtio-fs] " Christian Schoenebeck 2020-09-25 18:51 ` Dr. David Alan Gilbert 2020-09-25 18:51 ` [Virtio-fs] " Dr. David Alan Gilbert 2020-09-27 12:14 ` Christian Schoenebeck 2020-09-27 12:14 ` [Virtio-fs] " Christian Schoenebeck 2020-09-29 13:03 ` Vivek Goyal 2020-09-29 13:03 ` [Virtio-fs] " Vivek Goyal 2020-09-29 13:28 ` Christian Schoenebeck 2020-09-29 13:28 ` [Virtio-fs] " Christian Schoenebeck 2020-09-29 13:49 ` Vivek Goyal 2020-09-29 13:49 ` [Virtio-fs] " Vivek Goyal 2020-09-29 13:59 ` Christian Schoenebeck 2020-09-29 13:59 ` [Virtio-fs] " Christian Schoenebeck 2020-09-29 13:17 ` Vivek Goyal 2020-09-29 13:17 ` [Virtio-fs] " Vivek Goyal 2020-09-29 13:49 ` Miklos Szeredi 2020-09-29 13:49 ` Miklos Szeredi 2020-09-29 14:01 ` Vivek Goyal 2020-09-29 14:01 ` Vivek Goyal 2020-09-29 14:54 ` Miklos Szeredi 2020-09-29 14:54 ` Miklos Szeredi 2020-09-29 15:28 ` Vivek Goyal 2020-09-29 15:28 ` Vivek Goyal 2020-09-25 12:11 ` tools/virtiofs: Multi threading seems to hurt performance Dr. David Alan Gilbert 2020-09-25 12:11 ` [Virtio-fs] " Dr. David Alan Gilbert 2020-09-25 13:11 ` Vivek Goyal 2020-09-25 13:11 ` [Virtio-fs] " Vivek Goyal 2020-09-21 20:16 ` Vivek Goyal 2020-09-21 20:16 ` [Virtio-fs] " Vivek Goyal 2020-09-22 11:09 ` Dr. David Alan Gilbert 2020-09-22 11:09 ` [Virtio-fs] " Dr. David Alan Gilbert 2020-09-22 22:56 ` Vivek Goyal 2020-09-22 22:56 ` [Virtio-fs] " Vivek Goyal 2020-09-23 12:50 ` Chirantan Ekbote 2020-09-23 12:59 ` Vivek Goyal 2020-09-25 11:35 ` Dr. David Alan Gilbert
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200921133944.GB13362@redhat.com \ --to=vgoyal@redhat.com \ --cc=dgilbert@redhat.com \ --cc=qemu-devel@nongnu.org \ --cc=stefanha@redhat.com \ --cc=virtio-fs@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.