RE: issue about virtio-blk queue size

[Wangyong <wang.yongD@h3c.com>]

> From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
> Sent: Friday, December 20, 2019 5:53 PM
> To: wangyong (Cloud) <wang.yongD@h3c.com>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>; pbonzini@redhat.com;
> mark.kanda@oracle.com; hch@lst.de; qemu-devel@nongnu.org
> Subject: Re: issue about virtio-blk queue size
>
> On Thu, Dec 05, 2019 at 01:30:09AM +0000, Wangyong wrote:
> > >
> > > On Thu, Nov 28, 2019 at 08:44:43AM +0000, Wangyong wrote:
> > > > Hi all,
> > >
> > > This looks interesting, please continue this discussion on the QEMU
> > > mailing list <qemu-devel@nongnu.org> so that others can participate.
> > >
> > > >
> > > > This patch makes virtio_blk queue size configurable
> > > >
> > > > commit 6040aedddb5f474a9c2304b6a432a652d82b3d3c
> > > > Author: Mark Kanda <mark.kanda@oracle.com>
> > > > Date:   Mon Dec 11 09:16:24 2017 -0600
> > > >
> > > >     virtio-blk: make queue size configurable
> > > >
> > > > But when we set the queue size to more than 128, it will not take effect.
> > > >
> > > > That's because linux aio's maximum outstanding requests at a time
> > > > is always less than or equal to 128
> > > >
> > > > The following code limits the outstanding requests at a time:
> > > >
> > > > #define MAX_EVENTS 128
> > > >
> > > > laio_do_submit()
> > > > {
> > > >
> > > >     if (!s->io_q.blocked &&
> > > >         (!s->io_q.plugged ||
> > > >          s->io_q.in_flight + s->io_q.in_queue >= MAX_EVENTS)) {
> > > >         ioq_submit(s);
> > > >     }
> > > > }
> > > >
> > > > Should we make the value of MAX_EVENTS configurable ?
> > >
> > > Increasing MAX_EVENTS to a larger hardcoded value seems reasonable
> > > as a shortterm fix.  Please first check how /proc/sys/fs/aio-max-nr
> > > and
> > > io_setup(2) handle this resource limit.  The patch must not break
> > > existing systems where 128 works today.
> > [root@node2 ~]# cat /etc/centos-release CentOS Linux release 7.5.1804
> > (Core)
> >
> > [root@node2 ~]# cat /proc/sys/fs/aio-max-nr
> > 4294967296
> >
> > > > MAX_EVENTS should have the same value as queue size ?
> > >
> > > Multiple virtio-blk devices can share a single AioContext,
> > Is multiple virtio-blk configured with one IOThread?
> > Multiple virtio-blk performance will be worse.
>
> Yes.  By default IOThreads are not used and all virtio-blk devices share the
> main loop's AioContext.
>
> When IOThreads are configured it's up to the user how to assign devices to
> IOThreads.  Assigning multiple devices to one IOThread is realistic because
> it's common to create only num_vcpus IOThreads.
>
> A good starting point would be a patch that raises the limit to a higher
> hardcoded number.  Then you can investigate how to size the AioContext
> appropriately (maybe dynamically?) for a full fix.
>
Okay.  Raising MAX_EVENT to 1024 may be more appropriate? If possible I can try to make a patch.

> Stefan
-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from New H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!