From: Reinette Chatre <reinette.chatre@intel.com>
To: jgg@nvidia.com, yishaih@nvidia.com,
shameerali.kolothum.thodi@huawei.com, kevin.tian@intel.com,
alex.williamson@redhat.com
Cc: kvm@vger.kernel.org, dave.jiang@intel.com, jing2.liu@intel.com,
ashok.raj@intel.com, fenghua.yu@intel.com,
tom.zanussi@linux.intel.com, reinette.chatre@intel.com,
linux-kernel@vger.kernel.org, patches@lists.linux.dev
Subject: [RFC PATCH V2 00/18] vfio/pci: Back guest interrupts from Interrupt Message Store (IMS)
Date: Fri, 6 Oct 2023 09:40:55 -0700 [thread overview]
Message-ID: <cover.1696609476.git.reinette.chatre@intel.com> (raw)
Changes since RFC V1:
- RFC V1: https://lore.kernel.org/lkml/cover.1692892275.git.reinette.chatre@intel.com/
- This is a complete rewrite based on feedback from Jason and Kevin.
Primarily the transition is to make IMS a new backend of MSI-X
emulation: VFIO PCI transitions to be an interrupt management frontend
with existing interrupt management for PCI passthrough devices as a
backend and IMS interrupt management introduced as a new backend.
The first part of the series splits VFIO PCI interrupt
management into a "frontend" and "backend" with the existing PCI
interrupt management as its first backend. The second part of the
series adds IMS interrupt management as a new interrupt management
backend.
This is a significant change from RFC V1 as well as in the impact of
the changes on existing VFIO PCI. This was done in response to
feedback that I hope I understood as intended. If I did not get it
right, please do point out to me where I went astray and I'd be
happy to rewrite. Of course, suggestions for improvement will
be much appreciated.
Hi Everybody,
With Interrupt Message Store (IMS) support introduced in
commit 0194425af0c8 ("PCI/MSI: Provide IMS (Interrupt Message Store)
support") a device can create a secondary interrupt domain that works
side by side with MSI-X on the same device. IMS allows for
implementation-specific interrupt storage that is managed by the
implementation specific interrupt chip associated with the IMS domain
at the time it (the IMS domain) is created for the device via
pci_create_ims_domain().
An example usage of IMS is for devices that can have their resources
assigned to guests with varying granularity. For example, an
accelerator device may support many workqueues and a single workqueue
can be composed into a virtual device for use by a guest. Using
IMS interrupts for the guest preserves MSI-X for host usage while
allowing a significantly larger number of interrupt vectors than
allowed by MSI-X. All while enabling usage of the same device driver
within the host and guest.
This series introduces IMS support to VFIO PCI for use by
virtual devices that support MSI-X interrupts that are backed by IMS
interrupts on the host. Specifically, that means that when the virtual
device's VFIO_DEVICE_SET_IRQS ioctl() receives a "trigger interrupt"
(VFIO_IRQ_SET_ACTION_TRIGGER) for a MSI-X index then VFIO PCI IMS
allocates/frees an IMS interrupt on the host.
VFIO PCI assumes that it is managing interrupts of a passthrough PCI
device. VFIO PCI is split into a "frontend" and "backend" to support
interrupt management for virtual devices that are not passthrough PCI
devices. The VFIO PCI frontend directs guest requests to the
appropriate backend. Existing interrupt management for passthrough PCI
devices is the first backend, guest MSI-X interrupts backed by
IMS interrupts on the host is the new backend (VFIO PCI IMS).
An IMS interrupt is allocated via pci_ims_alloc_irq() that requires
an implementation specific cookie that is opaque to VFIO PCI IMS. This
can be a PASID, queue ID, pointer etc. During initialization
VFIO PCI IMS learns which PCI device to operate on and what the
default cookie should be for any new interrupt allocation. VFIO PCI
IMS can also associate a unique cookie with each vector and to maintain
this association the backend maintains interrupt contexts for the virtual
device's lifetime.
Guests may access a virtual device via both 'direct-path', where the
guest interacts directly with the underlying hardware, and 'intercepted
path', where the virtual device emulates operations. VFIO PCI
supports emulated interrupts (better naming suggestions are welcome) to
handle 'intercepted path' operations where completion interrupts are
signaled from the virtual device, not the underlying hardware. Backend
support is required for emulated interrupts and only VFIO PCI IMS
backend supports emulated interrupts in this series.
This has been tested with a yet to be published VFIO driver for the
Intel Data Accelerators (IDXD) present in Intel Xeon CPUs.
While this series contains a working implementation it is presented
as an RFC with the goal to obtain feedback on whether VFIO PCI IMS
is appropriate for inclusion into VFIO and whether it is
(or could be adapted to be) appropriate for support of other
planned IMS usages you may be aware of.
Any feedback will be greatly appreciated.
Reinette
Reinette Chatre (18):
PCI/MSI: Provide stubs for IMS functions
vfio/pci: Move PCI specific check from wrapper to PCI function
vfio/pci: Use unsigned int instead of unsigned
vfio/pci: Make core interrupt callbacks accessible to all virtual
devices
vfio/pci: Split PCI interrupt management into front and backend
vfio/pci: Separate MSI and MSI-X handling
vfio/pci: Move interrupt eventfd to interrupt context
vfio/pci: Move mutex acquisition into function
vfio/pci: Move interrupt contexts to generic interrupt struct
vfio/pci: Move IRQ type to generic interrupt context
vfio/pci: Split interrupt context initialization
vfio/pci: Provide interrupt context to generic ops
vfio/pci: Make vfio_pci_set_irqs_ioctl() available
vfio/pci: Add core IMS support
vfio/pci: Support emulated interrupts
vfio/pci: Support emulated interrupts in IMS backend
vfio/pci: Add accessor for IMS index
vfio/pci: Support IMS cookie modification
drivers/vfio/pci/vfio_pci_config.c | 2 +-
drivers/vfio/pci/vfio_pci_core.c | 50 +--
drivers/vfio/pci/vfio_pci_intrs.c | 658 ++++++++++++++++++++++++++---
drivers/vfio/pci/vfio_pci_priv.h | 2 +-
include/linux/pci.h | 31 +-
include/linux/vfio_pci_core.h | 70 ++-
6 files changed, 706 insertions(+), 107 deletions(-)
base-commit: 8a749fd1a8720d4619c91c8b6e7528c0a355c0aa
--
2.34.1
next reply other threads:[~2023-10-06 16:41 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-06 16:40 Reinette Chatre [this message]
2023-10-06 16:40 ` [RFC PATCH V2 01/18] PCI/MSI: Provide stubs for IMS functions Reinette Chatre
2023-10-06 16:40 ` [RFC PATCH V2 02/18] vfio/pci: Move PCI specific check from wrapper to PCI function Reinette Chatre
2023-10-06 16:40 ` [RFC PATCH V2 03/18] vfio/pci: Use unsigned int instead of unsigned Reinette Chatre
2023-10-06 16:40 ` [RFC PATCH V2 04/18] vfio/pci: Make core interrupt callbacks accessible to all virtual devices Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 05/18] vfio/pci: Split PCI interrupt management into front and backend Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 06/18] vfio/pci: Separate MSI and MSI-X handling Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 07/18] vfio/pci: Move interrupt eventfd to interrupt context Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 08/18] vfio/pci: Move mutex acquisition into function Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 09/18] vfio/pci: Move interrupt contexts to generic interrupt struct Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 10/18] vfio/pci: Move IRQ type to generic interrupt context Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 11/18] vfio/pci: Split interrupt context initialization Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 12/18] vfio/pci: Provide interrupt context to generic ops Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 13/18] vfio/pci: Make vfio_pci_set_irqs_ioctl() available Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 14/18] vfio/pci: Add core IMS support Reinette Chatre
2023-10-13 8:10 ` Tian, Kevin
2023-10-16 17:48 ` Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 15/18] vfio/pci: Support emulated interrupts Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 16/18] vfio/pci: Support emulated interrupts in IMS backend Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 17/18] vfio/pci: Add accessor for IMS index Reinette Chatre
2023-10-06 16:41 ` [RFC PATCH V2 18/18] vfio/pci: Support IMS cookie modification Reinette Chatre
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1696609476.git.reinette.chatre@intel.com \
--to=reinette.chatre@intel.com \
--cc=alex.williamson@redhat.com \
--cc=ashok.raj@intel.com \
--cc=dave.jiang@intel.com \
--cc=fenghua.yu@intel.com \
--cc=jgg@nvidia.com \
--cc=jing2.liu@intel.com \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=patches@lists.linux.dev \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=tom.zanussi@linux.intel.com \
--cc=yishaih@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).