From: Christian Schoenebeck <qemu_oss@crudebyte.com> To: qemu-devel@nongnu.org Cc: "Stefan Hajnoczi" <stefanha@redhat.com>, "Kevin Wolf" <kwolf@redhat.com>, "Laurent Vivier" <lvivier@redhat.com>, qemu-block@nongnu.org, "Michael S. Tsirkin" <mst@redhat.com>, "Jason Wang" <jasowang@redhat.com>, "Amit Shah" <amit@kernel.org>, "David Hildenbrand" <david@redhat.com>, "Greg Kurz" <groug@kaod.org>, virtio-fs@redhat.com, "Eric Auger" <eric.auger@redhat.com>, "Hanna Reitz" <hreitz@redhat.com>, "Gonglei (Arei)" <arei.gonglei@huawei.com>, "Gerd Hoffmann" <kraxel@redhat.com>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, "Paolo Bonzini" <pbonzini@redhat.com>, "Fam Zheng" <fam@euphon.net>, "Raphael Norwitz" <raphael.norwitz@nutanix.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com> Subject: Re: [PATCH v2 0/3] virtio: increase VIRTQUEUE_MAX_SIZE to 32k Date: Thu, 11 Nov 2021 18:54:03 +0100 [thread overview] Message-ID: <2186445.RDNOEBgitd@silver> (raw) In-Reply-To: <YY1FeMyvTOOWYEhV@stefanha-x1.localdomain> On Donnerstag, 11. November 2021 17:31:52 CET Stefan Hajnoczi wrote: > On Wed, Nov 10, 2021 at 04:53:33PM +0100, Christian Schoenebeck wrote: > > On Mittwoch, 10. November 2021 16:14:19 CET Stefan Hajnoczi wrote: > > > On Wed, Nov 10, 2021 at 02:14:43PM +0100, Christian Schoenebeck wrote: > > > > On Mittwoch, 10. November 2021 11:05:50 CET Stefan Hajnoczi wrote: > > > > As you are apparently reluctant for changing the virtio specs, what > > > > about > > > > introducing those discussed virtio capabalities either as experimental > > > > ones > > > > without specs changes, or even just as 9p specific device capabilities > > > > for > > > > now. I mean those could be revoked on both sides at any time anyway. > > > > > > I would like to understand the root cause before making changes. > > > > > > "It's faster when I do X" is useful information but it doesn't > > > necessarily mean doing X is the solution. The "it's faster when I do X > > > because Y" part is missing in my mind. Once there is evidence that shows > > > Y then it will be clearer if X is a good solution, if there's a more > > > general solution, or if it was just a side-effect. > > > > I think I made it clear that the root cause of the observed performance > > gain with rising transmission size is latency (and also that performance > > is not the only reason for addressing this queue size issue). > > > > Each request roundtrip has a certain minimum latency, the virtio ring > > alone > > has its latency, plus latency of the controller portion of the file server > > (e.g. permissions, sandbox checks, file IDs) that is executed with *every* > > request, plus latency of dispatching the request handling between threads > > several times back and forth (also for each request). > > > > Therefore when you split large payloads (e.g. reading a large file) into > > smaller n amount of chunks, then that individual latency per request > > accumulates to n times the individual latency, eventually leading to > > degraded transmission speed as those requests are serialized. > > It's easy to increase the blocksize in benchmarks, but real applications > offer less control over the I/O pattern. If latency in the device > implementation (QEMU) is the root cause then reduce the latency to speed > up all applications, even those that cannot send huge requests. Which I did, still do, and also mentioned before, e.g.: 8d6cb100731c4d28535adbf2a3c2d1f29be3fef4 9pfs: reduce latency of Twalk 0c4356ba7dafc8ecb5877a42fc0d68d45ccf5951 9pfs: T_readdir latency optimization Reducing overall latency is a process that is ongoing and will still take a very long development time. Not because of me, but because of lack of reviewers. And even then, it does not make the effort to support higher transmission sizes obsolete. > One idea is request merging on the QEMU side. If the application sends > 10 sequential read or write requests, coalesce them together before the > main part of request processing begins in the device. Process a single > large request to spread the cost of the file server over the 10 > requests. (virtio-blk has request merging to help with the cost of lots > of small qcow2 I/O requests.) The cool thing about this is that the > guest does not need to change its I/O pattern to benefit from the > optimization. > > Stefan Ok, don't get me wrong: I appreciate that you are suggesting approaches that could improve things. But I could already hand you over a huge list of mine. The limiting factor here is not the lack of ideas of what could be improved, but rather the lack of people helping out actively on 9p side: https://lists.gnu.org/archive/html/qemu-devel/2021-10/msg06452.html The situation on kernel side is the same. I already have a huge list of what could & should be improved. But there is basically no reviewer for 9p patches on Linux kernel side either. The much I appreciate suggestions of what could be improved, I would appreciate much more if there was *anybody* actively assisting as well. In the time being I have to work the list down in small patch chunks, priority based. Best regards, Christian Schoenebeck
WARNING: multiple messages have this Message-ID (diff)
From: Christian Schoenebeck <qemu_oss@crudebyte.com> To: qemu-devel@nongnu.org Cc: "Kevin Wolf" <kwolf@redhat.com>, "Laurent Vivier" <lvivier@redhat.com>, qemu-block@nongnu.org, "Michael S. Tsirkin" <mst@redhat.com>, "Jason Wang" <jasowang@redhat.com>, "Amit Shah" <amit@kernel.org>, "David Hildenbrand" <david@redhat.com>, "Raphael Norwitz" <raphael.norwitz@nutanix.com>, virtio-fs@redhat.com, "Eric Auger" <eric.auger@redhat.com>, "Hanna Reitz" <hreitz@redhat.com>, "Gonglei (Arei)" <arei.gonglei@huawei.com>, "Gerd Hoffmann" <kraxel@redhat.com>, "Paolo Bonzini" <pbonzini@redhat.com>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, "Fam Zheng" <fam@euphon.net> Subject: Re: [Virtio-fs] [PATCH v2 0/3] virtio: increase VIRTQUEUE_MAX_SIZE to 32k Date: Thu, 11 Nov 2021 18:54:03 +0100 [thread overview] Message-ID: <2186445.RDNOEBgitd@silver> (raw) In-Reply-To: <YY1FeMyvTOOWYEhV@stefanha-x1.localdomain> On Donnerstag, 11. November 2021 17:31:52 CET Stefan Hajnoczi wrote: > On Wed, Nov 10, 2021 at 04:53:33PM +0100, Christian Schoenebeck wrote: > > On Mittwoch, 10. November 2021 16:14:19 CET Stefan Hajnoczi wrote: > > > On Wed, Nov 10, 2021 at 02:14:43PM +0100, Christian Schoenebeck wrote: > > > > On Mittwoch, 10. November 2021 11:05:50 CET Stefan Hajnoczi wrote: > > > > As you are apparently reluctant for changing the virtio specs, what > > > > about > > > > introducing those discussed virtio capabalities either as experimental > > > > ones > > > > without specs changes, or even just as 9p specific device capabilities > > > > for > > > > now. I mean those could be revoked on both sides at any time anyway. > > > > > > I would like to understand the root cause before making changes. > > > > > > "It's faster when I do X" is useful information but it doesn't > > > necessarily mean doing X is the solution. The "it's faster when I do X > > > because Y" part is missing in my mind. Once there is evidence that shows > > > Y then it will be clearer if X is a good solution, if there's a more > > > general solution, or if it was just a side-effect. > > > > I think I made it clear that the root cause of the observed performance > > gain with rising transmission size is latency (and also that performance > > is not the only reason for addressing this queue size issue). > > > > Each request roundtrip has a certain minimum latency, the virtio ring > > alone > > has its latency, plus latency of the controller portion of the file server > > (e.g. permissions, sandbox checks, file IDs) that is executed with *every* > > request, plus latency of dispatching the request handling between threads > > several times back and forth (also for each request). > > > > Therefore when you split large payloads (e.g. reading a large file) into > > smaller n amount of chunks, then that individual latency per request > > accumulates to n times the individual latency, eventually leading to > > degraded transmission speed as those requests are serialized. > > It's easy to increase the blocksize in benchmarks, but real applications > offer less control over the I/O pattern. If latency in the device > implementation (QEMU) is the root cause then reduce the latency to speed > up all applications, even those that cannot send huge requests. Which I did, still do, and also mentioned before, e.g.: 8d6cb100731c4d28535adbf2a3c2d1f29be3fef4 9pfs: reduce latency of Twalk 0c4356ba7dafc8ecb5877a42fc0d68d45ccf5951 9pfs: T_readdir latency optimization Reducing overall latency is a process that is ongoing and will still take a very long development time. Not because of me, but because of lack of reviewers. And even then, it does not make the effort to support higher transmission sizes obsolete. > One idea is request merging on the QEMU side. If the application sends > 10 sequential read or write requests, coalesce them together before the > main part of request processing begins in the device. Process a single > large request to spread the cost of the file server over the 10 > requests. (virtio-blk has request merging to help with the cost of lots > of small qcow2 I/O requests.) The cool thing about this is that the > guest does not need to change its I/O pattern to benefit from the > optimization. > > Stefan Ok, don't get me wrong: I appreciate that you are suggesting approaches that could improve things. But I could already hand you over a huge list of mine. The limiting factor here is not the lack of ideas of what could be improved, but rather the lack of people helping out actively on 9p side: https://lists.gnu.org/archive/html/qemu-devel/2021-10/msg06452.html The situation on kernel side is the same. I already have a huge list of what could & should be improved. But there is basically no reviewer for 9p patches on Linux kernel side either. The much I appreciate suggestions of what could be improved, I would appreciate much more if there was *anybody* actively assisting as well. In the time being I have to work the list down in small patch chunks, priority based. Best regards, Christian Schoenebeck
next prev parent reply other threads:[~2021-11-11 17:55 UTC|newest] Thread overview: 97+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-10-04 19:38 [PATCH v2 0/3] virtio: increase VIRTQUEUE_MAX_SIZE to 32k Christian Schoenebeck 2021-10-04 19:38 ` [Virtio-fs] " Christian Schoenebeck 2021-10-04 19:38 ` [PATCH v2 1/3] virtio: turn VIRTQUEUE_MAX_SIZE into a variable Christian Schoenebeck 2021-10-04 19:38 ` [Virtio-fs] " Christian Schoenebeck 2021-10-05 7:36 ` Greg Kurz 2021-10-05 7:36 ` [Virtio-fs] " Greg Kurz 2021-10-05 12:45 ` Stefan Hajnoczi 2021-10-05 12:45 ` [Virtio-fs] " Stefan Hajnoczi 2021-10-05 13:15 ` Christian Schoenebeck 2021-10-05 13:15 ` [Virtio-fs] " Christian Schoenebeck 2021-10-05 15:10 ` Stefan Hajnoczi 2021-10-05 15:10 ` [Virtio-fs] " Stefan Hajnoczi 2021-10-05 16:32 ` Christian Schoenebeck 2021-10-05 16:32 ` [Virtio-fs] " Christian Schoenebeck 2021-10-06 11:06 ` Stefan Hajnoczi 2021-10-06 11:06 ` [Virtio-fs] " Stefan Hajnoczi 2021-10-06 12:50 ` Christian Schoenebeck 2021-10-06 12:50 ` [Virtio-fs] " Christian Schoenebeck 2021-10-06 14:42 ` Stefan Hajnoczi 2021-10-06 14:42 ` [Virtio-fs] " Stefan Hajnoczi 2021-10-07 13:09 ` Christian Schoenebeck 2021-10-07 13:09 ` [Virtio-fs] " Christian Schoenebeck 2021-10-07 15:18 ` Stefan Hajnoczi 2021-10-07 15:18 ` [Virtio-fs] " Stefan Hajnoczi 2021-10-08 14:48 ` Christian Schoenebeck 2021-10-08 14:48 ` [Virtio-fs] " Christian Schoenebeck 2021-10-04 19:38 ` [PATCH v2 2/3] virtio: increase VIRTQUEUE_MAX_SIZE to 32k Christian Schoenebeck 2021-10-04 19:38 ` [Virtio-fs] " Christian Schoenebeck 2021-10-05 7:16 ` Michael S. Tsirkin 2021-10-05 7:16 ` [Virtio-fs] " Michael S. Tsirkin 2021-10-05 7:35 ` Greg Kurz 2021-10-05 7:35 ` [Virtio-fs] " Greg Kurz 2021-10-05 11:17 ` Christian Schoenebeck 2021-10-05 11:17 ` [Virtio-fs] " Christian Schoenebeck 2021-10-05 11:24 ` Michael S. Tsirkin 2021-10-05 11:24 ` [Virtio-fs] " Michael S. Tsirkin 2021-10-05 12:01 ` Christian Schoenebeck 2021-10-05 12:01 ` [Virtio-fs] " Christian Schoenebeck 2021-10-04 19:38 ` [PATCH v2 3/3] virtio-9p-device: switch to 32k max. transfer size Christian Schoenebeck 2021-10-04 19:38 ` [Virtio-fs] " Christian Schoenebeck 2021-10-05 7:38 ` [PATCH v2 0/3] virtio: increase VIRTQUEUE_MAX_SIZE to 32k David Hildenbrand 2021-10-05 7:38 ` [Virtio-fs] " David Hildenbrand 2021-10-05 11:10 ` Christian Schoenebeck 2021-10-05 11:10 ` [Virtio-fs] " Christian Schoenebeck 2021-10-05 11:19 ` Michael S. Tsirkin 2021-10-05 11:19 ` [Virtio-fs] " Michael S. Tsirkin 2021-10-05 11:43 ` Christian Schoenebeck 2021-10-05 11:43 ` [Virtio-fs] " Christian Schoenebeck 2021-10-07 5:23 ` Stefan Hajnoczi 2021-10-07 5:23 ` [Virtio-fs] " Stefan Hajnoczi 2021-10-07 12:51 ` Christian Schoenebeck 2021-10-07 12:51 ` [Virtio-fs] " Christian Schoenebeck 2021-10-07 15:42 ` Stefan Hajnoczi 2021-10-07 15:42 ` [Virtio-fs] " Stefan Hajnoczi 2021-10-08 7:25 ` Greg Kurz 2021-10-08 7:25 ` [Virtio-fs] " Greg Kurz 2021-10-08 8:27 ` Greg Kurz 2021-10-08 14:24 ` Christian Schoenebeck 2021-10-08 14:24 ` [Virtio-fs] " Christian Schoenebeck 2021-10-08 16:08 ` Christian Schoenebeck 2021-10-08 16:08 ` [Virtio-fs] " Christian Schoenebeck 2021-10-21 15:39 ` Christian Schoenebeck 2021-10-21 15:39 ` [Virtio-fs] " Christian Schoenebeck 2021-10-25 10:30 ` Stefan Hajnoczi 2021-10-25 10:30 ` [Virtio-fs] " Stefan Hajnoczi 2021-10-25 15:03 ` Christian Schoenebeck 2021-10-25 15:03 ` [Virtio-fs] " Christian Schoenebeck 2021-10-28 9:00 ` Stefan Hajnoczi 2021-10-28 9:00 ` [Virtio-fs] " Stefan Hajnoczi 2021-11-01 20:29 ` Christian Schoenebeck 2021-11-01 20:29 ` [Virtio-fs] " Christian Schoenebeck 2021-11-03 11:33 ` Stefan Hajnoczi 2021-11-03 11:33 ` [Virtio-fs] " Stefan Hajnoczi 2021-11-04 14:41 ` Christian Schoenebeck 2021-11-04 14:41 ` [Virtio-fs] " Christian Schoenebeck 2021-11-09 10:56 ` Stefan Hajnoczi 2021-11-09 10:56 ` [Virtio-fs] " Stefan Hajnoczi 2021-11-09 13:09 ` Christian Schoenebeck 2021-11-09 13:09 ` [Virtio-fs] " Christian Schoenebeck 2021-11-10 10:05 ` Stefan Hajnoczi 2021-11-10 10:05 ` [Virtio-fs] " Stefan Hajnoczi 2021-11-10 13:14 ` Christian Schoenebeck 2021-11-10 13:14 ` [Virtio-fs] " Christian Schoenebeck 2021-11-10 15:14 ` Stefan Hajnoczi 2021-11-10 15:14 ` [Virtio-fs] " Stefan Hajnoczi 2021-11-10 15:53 ` Christian Schoenebeck 2021-11-10 15:53 ` [Virtio-fs] " Christian Schoenebeck 2021-11-11 16:31 ` Stefan Hajnoczi 2021-11-11 16:31 ` [Virtio-fs] " Stefan Hajnoczi 2021-11-11 17:54 ` Christian Schoenebeck [this message] 2021-11-11 17:54 ` Christian Schoenebeck 2021-11-15 11:54 ` Stefan Hajnoczi 2021-11-15 11:54 ` [Virtio-fs] " Stefan Hajnoczi 2021-11-15 14:32 ` Christian Schoenebeck 2021-11-15 14:32 ` [Virtio-fs] " Christian Schoenebeck 2021-11-16 11:13 ` Stefan Hajnoczi 2021-11-16 11:13 ` [Virtio-fs] " Stefan Hajnoczi
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=2186445.RDNOEBgitd@silver \ --to=qemu_oss@crudebyte.com \ --cc=amit@kernel.org \ --cc=arei.gonglei@huawei.com \ --cc=david@redhat.com \ --cc=dgilbert@redhat.com \ --cc=eric.auger@redhat.com \ --cc=fam@euphon.net \ --cc=groug@kaod.org \ --cc=hreitz@redhat.com \ --cc=jasowang@redhat.com \ --cc=kraxel@redhat.com \ --cc=kwolf@redhat.com \ --cc=lvivier@redhat.com \ --cc=marcandre.lureau@redhat.com \ --cc=mst@redhat.com \ --cc=pbonzini@redhat.com \ --cc=qemu-block@nongnu.org \ --cc=qemu-devel@nongnu.org \ --cc=raphael.norwitz@nutanix.com \ --cc=stefanha@redhat.com \ --cc=virtio-fs@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.