From: "Longpeng (Mike, Cloud Infrastructure Service Product Dept.)" <longpeng2@huawei.com>
To: "Philippe Mathieu-Daudé" <philmd@redhat.com>,
alex.williamson@redhat.com, mst@redhat.com,
marcel.apfelbaum@gmail.com, pbonzini@redhat.com
Cc: arei.gonglei@huawei.com, huangzhichao@huawei.com, qemu-devel@nongnu.org
Subject: Re: [PATCH 3/5] vfio: defer to enable msix in migration resume phase
Date: Wed, 25 Aug 2021 18:06:52 +0800 [thread overview]
Message-ID: <1504cd9a-0d90-db90-a942-78186886af13@huawei.com> (raw)
In-Reply-To: <5137d6c9-b917-2bed-cdcf-7b7072632c6c@redhat.com>
在 2021/8/25 17:57, Philippe Mathieu-Daudé 写道:
> On 8/25/21 9:56 AM, Longpeng(Mike) wrote:
>> The vf's unmasked msix vectors will be enable one by one in
>> migraiton resume phase, VFIO_DEVICE_SET_IRQS will be called
>
> Typo "migration"
>
Ok.
>> for each vector, it's a bit expensive if the vf has more
>> vectors.
>>
>> We can call VFIO_DEVICE_SET_IRQS once outside the loop of set
>> vector notifiers to reduce the cost.
>>
>> The test VM has 128 vcpus and 8 VF (with 65 vectors enabled),
>> we mesure the cost of the vfio_msix_enable for each one, and
>
> Typo "measure"
>
Ok.
>> we can see 10% costs can be reduced.
>>
>> Origin Apply this patch
>> 1st 8 4
>> 2nd 15 11
>> 3rd 22 18
>> 4th 24 25
>> 5th 36 33
>> 6th 44 40
>> 7th 51 47
>> 8th 58 54
>> Total 258ms 232ms
>>
>> Signed-off-by: Longpeng(Mike) <longpeng2@huawei.com>
>> ---
>> hw/vfio/pci.c | 22 ++++++++++++++++++++++
>> hw/vfio/pci.h | 1 +
>> 2 files changed, 23 insertions(+)
>>
>> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
>> index 7cc43fe..ca37fb7 100644
>> --- a/hw/vfio/pci.c
>> +++ b/hw/vfio/pci.c
>> @@ -372,6 +372,10 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix)
>> int ret = 0, i, argsz;
>> int32_t *fds;
>>
>> + if (!vdev->nr_vectors) {
>> + return 0;
>> + }
>> +
>> argsz = sizeof(*irq_set) + (vdev->nr_vectors * sizeof(*fds));
>>
>> irq_set = g_malloc0(argsz);
>> @@ -495,6 +499,11 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr,
>> }
>> }
>>
>> + if (vdev->defer_add_virq) {
>> + vdev->nr_vectors = MAX(vdev->nr_vectors, nr + 1);
>> + goto clear_pending;
>> + }
>> +
>> /*
>> * We don't want to have the host allocate all possible MSI vectors
>> * for a device if they're not in use, so we shutdown and incrementally
>> @@ -524,6 +533,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr,
>> }
>> }
>>
>> +clear_pending:
>> /* Disable PBA emulation when nothing more is pending. */
>> clear_bit(nr, vdev->msix->pending);
>> if (find_first_bit(vdev->msix->pending,
>> @@ -608,6 +618,16 @@ static void vfio_msix_enable(VFIOPCIDevice *vdev)
>> if (msix_set_vector_notifiers(pdev, vfio_msix_vector_use,
>> vfio_msix_vector_release, NULL)) {
>> error_report("vfio: msix_set_vector_notifiers failed");
>> + return;
>> + }
>> +
>> + if (!pdev->msix_function_masked && vdev->defer_add_virq) {
>> + int ret;
>> + vfio_disable_irqindex(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDEX);
>> + ret = vfio_enable_vectors(vdev, true);
>> + if (ret) {
>> + error_report("vfio: failed to enable vectors, %d", ret);
>> + }
>> }
>>
>> trace_vfio_msix_enable(vdev->vbasedev.name);
>> @@ -2456,7 +2476,9 @@ static int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f)
>> if (msi_enabled(pdev)) {
>> vfio_msi_enable(vdev);
>> } else if (msix_enabled(pdev)) {
>> + vdev->defer_add_virq = true;
>> vfio_msix_enable(vdev);
>
> What about passing defer_add_virq as boolean argument
> to vfio_msix_enable()?
>
We'll use defer_add_virq in the deep of the calltrace, it need to change more
functions to support the parameter passing in this way.
>> + vdev->defer_add_virq = false;
>> }
>>
>> return ret;
>> diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h
>> index 6477751..4235c83 100644
>> --- a/hw/vfio/pci.h
>> +++ b/hw/vfio/pci.h
>> @@ -171,6 +171,7 @@ struct VFIOPCIDevice {
>> bool no_kvm_ioeventfd;
>> bool no_vfio_ioeventfd;
>> bool enable_ramfb;
>> + bool defer_add_virq;
>> VFIODisplay *dpy;
>> Notifier irqchip_change_notifier;
>> };
>>
>
> .
>
next prev parent reply other threads:[~2021-08-25 10:07 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-25 7:56 [PATCH 0/5] optimize the downtime for vfio migration Longpeng(Mike)
2021-08-25 7:56 ` [PATCH 1/5] vfio: use helper to simplfy the failure path in vfio_msi_enable Longpeng(Mike)
2021-09-03 21:55 ` Alex Williamson
2021-09-07 2:11 ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
2021-08-25 7:56 ` [PATCH 2/5] msix: simplfy the conditional in msix_set/unset_vector_notifiers Longpeng(Mike)
2021-08-25 9:52 ` Philippe Mathieu-Daudé
2021-08-25 9:56 ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
2021-09-03 21:55 ` Alex Williamson
2021-08-25 7:56 ` [PATCH 3/5] vfio: defer to enable msix in migration resume phase Longpeng(Mike)
2021-08-25 9:57 ` Philippe Mathieu-Daudé
2021-08-25 10:06 ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.) [this message]
2021-09-03 21:56 ` Alex Williamson
2021-09-07 2:12 ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
2021-08-25 7:56 ` [PATCH 4/5] kvm: irqchip: support defer to commit the route Longpeng(Mike)
2021-09-03 21:57 ` Alex Williamson
2021-09-07 2:13 ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
2021-08-25 7:56 ` [PATCH 5/5] vfio: defer to commit kvm route in migraiton resume phase Longpeng(Mike)
2021-09-03 21:57 ` Alex Williamson
2021-09-07 2:14 ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
2021-08-25 10:05 ` [PATCH 0/5] optimize the downtime for vfio migration Philippe Mathieu-Daudé
2021-08-25 10:09 ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
2021-09-02 9:43 ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1504cd9a-0d90-db90-a942-78186886af13@huawei.com \
--to=longpeng2@huawei.com \
--cc=alex.williamson@redhat.com \
--cc=arei.gonglei@huawei.com \
--cc=huangzhichao@huawei.com \
--cc=marcel.apfelbaum@gmail.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=philmd@redhat.com \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.