Re: [PATCH v10 57/59] hw/xen: Support MSI mapping to PIRQ

From: Paul Durrant <xadimgnik@gmail.com>
To: David Woodhouse <dwmw2@infradead.org>,
	Peter Maydell <peter.maydell@linaro.org>,
	qemu-devel@nongnu.org
Cc: "Paolo Bonzini" <pbonzini@redhat.com>,
	"Joao Martins" <joao.m.martins@oracle.com>,
	"Ankur Arora" <ankur.a.arora@oracle.com>,
	"Philippe Mathieu-Daudé" <philmd@linaro.org>,
	"Thomas Huth" <thuth@redhat.com>,
	"Alex Bennée" <alex.bennee@linaro.org>,
	"Juan Quintela" <quintela@redhat.com>,
	"Dr . David Alan Gilbert" <dgilbert@redhat.com>,
	"Claudio Fontana" <cfontana@suse.de>,
	"Julien Grall" <julien@xen.org>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	"Marcel Apfelbaum" <marcel.apfelbaum@gmail.com>,
	armbru@redhat.com
Subject: Re: [PATCH v10 57/59] hw/xen: Support MSI mapping to PIRQ
Date: Tue, 14 Feb 2023 16:14:14 +0000	[thread overview]
Message-ID: <0e7501e1-15b2-f309-1160-4561f43096a0@xen.org> (raw)
In-Reply-To: <20230201143148.1744093-58-dwmw2@infradead.org>

On 01/02/2023 14:31, David Woodhouse wrote:
> From: David Woodhouse <dwmw@amazon.co.uk>
> 
> The way that Xen handles MSI PIRQs is kind of awful.
> 
> There is a special MSI message which targets a PIRQ. The vector in the
> low bits of data must be zero. The low 8 bits of the PIRQ# are in the
> destination ID field, the extended destination ID field is unused, and
> instead the high bits of the PIRQ# are in the high 32 bits of the address.
> 
> Using the high bits of the address means that we can't intercept and
> translate these messages in kvm_send_msi(), because they won't be caught
> by the APIC — addresses like 0x1000fee46000 aren't in the APIC's range.
> 
> So we catch them in pci_msi_trigger() instead, and deliver the event
> channel directly.
> 
> That isn't even the worst part. The worst part is that Xen snoops on
> writes to devices' MSI vectors while they are *masked*. When a MSI
> message is written which looks like it targets a PIRQ, it remembers
> the device and vector for later.
> 
> When the guest makes a hypercall to bind that PIRQ# (snooped from a
> marked MSI vector) to an event channel port, Xen *unmasks* that MSI
> vector on the device. Xen guests using PIRQ delivery of MSI don't
> ever actually unmask the MSI for themselves.
> 
> Now that this is working we can finally enable XENFEAT_hvm_pirqs and
> let the guest use it all.
> 
> Tested with passthrough igb and emulated e1000e + AHCI.
> 
>             CPU0       CPU1
>    0:         65          0   IO-APIC   2-edge      timer
>    1:          0         14  xen-pirq   1-ioapic-edge  i8042
>    4:          0        846  xen-pirq   4-ioapic-edge  ttyS0
>    8:          1          0  xen-pirq   8-ioapic-edge  rtc0
>    9:          0          0  xen-pirq   9-ioapic-level  acpi
>   12:        257          0  xen-pirq  12-ioapic-edge  i8042
>   24:       9600          0  xen-percpu    -virq      timer0
>   25:       2758          0  xen-percpu    -ipi       resched0
>   26:          0          0  xen-percpu    -ipi       callfunc0
>   27:          0          0  xen-percpu    -virq      debug0
>   28:       1526          0  xen-percpu    -ipi       callfuncsingle0
>   29:          0          0  xen-percpu    -ipi       spinlock0
>   30:          0       8608  xen-percpu    -virq      timer1
>   31:          0        874  xen-percpu    -ipi       resched1
>   32:          0          0  xen-percpu    -ipi       callfunc1
>   33:          0          0  xen-percpu    -virq      debug1
>   34:          0       1617  xen-percpu    -ipi       callfuncsingle1
>   35:          0          0  xen-percpu    -ipi       spinlock1
>   36:          8          0   xen-dyn    -event     xenbus
>   37:          0       6046  xen-pirq    -msi       ahci[0000:00:03.0]
>   38:          1          0  xen-pirq    -msi-x     ens4
>   39:          0         73  xen-pirq    -msi-x     ens4-rx-0
>   40:         14          0  xen-pirq    -msi-x     ens4-rx-1
>   41:          0         32  xen-pirq    -msi-x     ens4-tx-0
>   42:         47          0  xen-pirq    -msi-x     ens4-tx-1
> 
> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> ---
>   hw/i386/kvm/meson.build    |   7 +
>   hw/i386/kvm/trace-events   |   1 +
>   hw/i386/kvm/xen-stubs.c    |  27 ++++
>   hw/i386/kvm/xen_evtchn.c   | 261 ++++++++++++++++++++++++++++++++++++-
>   hw/i386/kvm/xen_evtchn.h   |   8 ++
>   hw/pci/msi.c               |  11 ++
>   hw/pci/msix.c              |   7 +
>   hw/pci/pci.c               |  17 +++
>   include/hw/pci/msi.h       |   1 +
>   target/i386/kvm/kvm.c      |  19 ++-
>   target/i386/kvm/kvm_i386.h |   2 +
>   target/i386/kvm/xen-emu.c  |   3 +-
>   12 files changed, 354 insertions(+), 10 deletions(-)
>   create mode 100644 hw/i386/kvm/xen-stubs.c

Reviewed-by: Paul Durrant <paul@xen.org>