From: Stefan Hajnoczi <stefanha@redhat.com>
To: Greg Kurz <groug@kaod.org>
Cc: Fam Zheng <fam@euphon.net>, Kevin Wolf <kwolf@redhat.com>,
qemu-block@nongnu.org, "Michael S. Tsirkin" <mst@redhat.com>,
qemu-devel@nongnu.org, Max Reitz <mreitz@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>,
David Gibson <david@gibson.dropbear.id.au>
Subject: Re: [RFC 0/8] virtio: Improve boot time of virtio-scsi-pci and virtio-blk-pci
Date: Mon, 29 Mar 2021 18:35:16 +0100 [thread overview]
Message-ID: <YGIP1HKa5NsHqnj7@stefanha-x1.localdomain> (raw)
In-Reply-To: <20210325150735.1098387-1-groug@kaod.org>
[-- Attachment #1: Type: text/plain, Size: 2801 bytes --]
On Thu, Mar 25, 2021 at 04:07:27PM +0100, Greg Kurz wrote:
> Now that virtio-scsi-pci and virtio-blk-pci map 1 virtqueue per vCPU,
> a serious slow down may be observed on setups with a big enough number
> of vCPUs.
>
> Exemple with a pseries guest on a bi-POWER9 socket system (128 HW threads):
>
> 1 0m20.922s 0m21.346s
> 2 0m21.230s 0m20.350s
> 4 0m21.761s 0m20.997s
> 8 0m22.770s 0m20.051s
> 16 0m22.038s 0m19.994s
> 32 0m22.928s 0m20.803s
> 64 0m26.583s 0m22.953s
> 128 0m41.273s 0m32.333s
> 256 2m4.727s 1m16.924s
> 384 6m5.563s 3m26.186s
>
> Both perf and gprof indicate that QEMU is hogging CPUs when setting up
> the ioeventfds:
>
> 67.88% swapper [kernel.kallsyms] [k] power_pmu_enable
> 9.47% qemu-kvm [kernel.kallsyms] [k] smp_call_function_single
> 8.64% qemu-kvm [kernel.kallsyms] [k] power_pmu_enable
> =>2.79% qemu-kvm qemu-kvm [.] memory_region_ioeventfd_before
> =>2.12% qemu-kvm qemu-kvm [.] address_space_update_ioeventfds
> 0.56% kworker/8:0-mm [kernel.kallsyms] [k] smp_call_function_single
>
> address_space_update_ioeventfds() is called when committing an MR
> transaction, i.e. for each ioeventfd with the current code base,
> and it internally loops on all ioventfds:
>
> static void address_space_update_ioeventfds(AddressSpace *as)
> {
> [...]
> FOR_EACH_FLAT_RANGE(fr, view) {
> for (i = 0; i < fr->mr->ioeventfd_nb; ++i) {
>
> This means that the setup of ioeventfds for these devices has
> quadratic time complexity.
>
> This series introduce generic APIs to allow batch creation and deletion
> of ioeventfds, and converts virtio-blk and virtio-scsi to use them. This
> greatly improves the numbers:
>
> 1 0m21.271s 0m22.076s
> 2 0m20.912s 0m19.716s
> 4 0m20.508s 0m19.310s
> 8 0m21.374s 0m20.273s
> 16 0m21.559s 0m21.374s
> 32 0m22.532s 0m21.271s
> 64 0m26.550s 0m22.007s
> 128 0m29.115s 0m27.446s
> 256 0m44.752s 0m41.004s
> 384 1m2.884s 0m58.023s
Excellent numbers!
I wonder if the code can be simplified since
memory_region_transaction_begin/end() supports nesting. Why not call
them directly from the device model instead of introducing callbacks in
core virtio and virtio-pci code?
Also, do you think there are other opportunities to have a long
transaction to batch up machine init, device hotplug, etc? It's not
clear to me when transactions must be ended. Clearly it's necessary to
end the transaction if we need to do something that depends on the
MemoryRegion, eventfd, etc being updated. But most of the time there is
no immediate need to end the transaction and more code could share the
same transaction before we go back to the event loop or vcpu thread.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
next prev parent reply other threads:[~2021-03-29 17:37 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-25 15:07 [RFC 0/8] virtio: Improve boot time of virtio-scsi-pci and virtio-blk-pci Greg Kurz
2021-03-25 15:07 ` [RFC 1/8] memory: Allow eventfd add/del without starting a transaction Greg Kurz
2021-03-29 17:03 ` Stefan Hajnoczi
2021-03-30 7:47 ` Greg Kurz
2021-03-30 10:16 ` Stefan Hajnoczi
2021-03-25 15:07 ` [RFC 2/8] virtio: Introduce virtio_bus_set_host_notifiers() Greg Kurz
2021-03-29 17:06 ` Stefan Hajnoczi
2021-03-25 15:07 ` [RFC 3/8] virtio: Add API to batch set host notifiers Greg Kurz
2021-03-29 17:10 ` Stefan Hajnoczi
2021-03-30 10:17 ` Greg Kurz
2021-03-30 13:55 ` Stefan Hajnoczi
2021-03-30 14:17 ` Greg Kurz
2021-03-31 14:47 ` Stefan Hajnoczi
2021-03-31 16:21 ` Greg Kurz
2021-03-25 15:07 ` [RFC 4/8] virtio-pci: Batch add/del ioeventfds in a single MR transaction Greg Kurz
2021-03-29 17:24 ` Stefan Hajnoczi
2021-03-30 10:29 ` Greg Kurz
2021-03-25 15:07 ` [RFC 5/8] virtio-blk: Fix rollback path in virtio_blk_data_plane_start() Greg Kurz
2021-03-29 17:25 ` Stefan Hajnoczi
2021-03-25 15:07 ` [RFC 6/8] virtio-blk: Use virtio_bus_set_host_notifiers() Greg Kurz
2021-03-29 17:26 ` Stefan Hajnoczi
2021-03-25 15:07 ` [RFC 7/8] virtio-scsi: Set host notifiers and callbacks separately Greg Kurz
2021-03-25 15:07 ` [RFC 8/8] virtio-scsi: Use virtio_bus_set_host_notifiers() Greg Kurz
2021-03-29 17:28 ` Stefan Hajnoczi
2021-03-25 17:05 ` [RFC 0/8] virtio: Improve boot time of virtio-scsi-pci and virtio-blk-pci Michael S. Tsirkin
2021-03-25 17:43 ` Stefan Hajnoczi
2021-03-25 20:51 ` Greg Kurz
2021-03-25 18:05 ` Greg Kurz
2021-03-29 17:35 ` Stefan Hajnoczi [this message]
2021-03-30 13:15 ` Greg Kurz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YGIP1HKa5NsHqnj7@stefanha-x1.localdomain \
--to=stefanha@redhat.com \
--cc=david@gibson.dropbear.id.au \
--cc=fam@euphon.net \
--cc=groug@kaod.org \
--cc=kwolf@redhat.com \
--cc=mreitz@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).