From: Brett Creeley <brett.creeley@amd.com>
To: <kvm@vger.kernel.org>, <netdev@vger.kernel.org>,
<alex.williamson@redhat.com>, <jgg@nvidia.com>,
<yishaih@nvidia.com>, <shameerali.kolothum.thodi@huawei.com>,
<kevin.tian@intel.com>
Cc: <brett.creeley@amd.com>, <shannon.nelson@amd.com>
Subject: [PATCH v10 vfio 0/7] pds_vfio driver
Date: Fri, 2 Jun 2023 15:03:11 -0700 [thread overview]
Message-ID: <20230602220318.15323-1-brett.creeley@amd.com> (raw)
This is a patchset for a new vendor specific VFIO driver
(pds_vfio) for use with the AMD/Pensando Distributed Services Card
(DSC). This driver makes use of the pds_core driver.
This driver will use the pds_core device's adminq as the VFIO
control path to the DSC. In order to make adminq calls, the VFIO
instance makes use of functions exported by the pds_core driver.
In order to receive events from pds_core, the pds_vfio driver
registers to a private notifier. This is needed for various events
that come from the device.
An ASCII diagram of a VFIO instance looks something like this and can
be used with the VFIO subsystem to provide the VF device VFIO and live
migration support.
.------. .-----------------------.
| QEMU |--| VM .-------------. |
'......' | | Eth VF | |
| | .-------------. |
| | | SR-IOV VF | |
| | '-------------' |
| '------------||---------'
.--------------. ||
|/dev/<vfio_fd>| ||
'--------------' ||
Host Userspace | ||
=================================================== ||
Host Kernel | ||
.--------. ||
|vfio-pci| ||
'--------' ||
.------------------. || ||
| | exported API |<----+ || ||
| '--------------| | || ||
| | .-------------. ||
| pds_core |--->| pds_vfio | ||
'------------------' | '-------------' ||
|| | || ||
09:00.0 notifier 09:00.1 ||
== PCI ===============================================||=====
|| || ||
.----------. .----------. ||
,-----| PF |----------| VF |-------------------,
| '----------' '----------' | VF |
| DSC | data/control |
| | path |
-----------------------------------------------------------
The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds.
It makes use of and introduces new files in the common include/linux/pds
include directory.
Changes:
v10:
- Various fixes/suggestions by Jason Gunthorpe
- Simplify pds_vfio_get_lm_file() based on fpga_mgr_buf_load()
- Clean-ups/fixes based on clang-format
- Remove any double goto labels
- Name goto labels baesed on what needs to be cleaned/freed
instead of a "call from" scheme
- Fix any goto unwind ordering issues
- Make sure call dma_map_single() after data is written to
memory in pds_vfio_dma_map_lm_file()
- Don't use bitmap_zalloc() for the dirty bitmaps
- Use vzalloc() for dirty bitmaps and refactor how the bitmaps are DMA'd
to and from the device in pds_vfio_dirty_seq_ack()
- Remove unnecessary goto in pds_vfio_dirty_disable()
v9:
https://lore.kernel.org/netdev/20230422010642.60720-1-brett.creeley@amd.com/
- Various fixes/suggestions by Alex Williamson
- Fix how ID is generated in client registration
- Add helper functions to get the VF's struct device and struct
pci_dev pointers instead of caching the struct pci dev
- Remove redundant pds_vfio_lm_state() function and remove any
places this was being called
- Fix multi-line comments to follow standard convention
- Remove confusing comments in
pds_vfio_step_device_state_locked() since the driver's
migration states align with the VFIO documentation
- Validate pdsc returned from pdsc_get_pf_struct()
- Various fixes/suggestions by Jason Gunthorpe
- Use struct pdsc instead of void *
- Use {} instead of {0} for structure initialization
- Use unions on the stack instead of casting to the union when
sending AQ commands, which required including pds_lm.h in
pds_adminq.h
- Replace use of dma_alloc_coherent() when creating the sgl DMA
entries for the LM file
- Remove cached struct device *coredev and instead use
pci_physfn() to get the pds_core's struct device pointer
- Drop the recovery work item and call pds_vfio_recovery()
directly from the notifier callback
- Remove unnecessary #define for "pds_vfio_lm" and just use the
string inline to the anon_inode_getfile() argument
- Fix LM file reference counting
- Move initialization of some struct members to when the struct is being
initialized for AQ commands
- Make use of GFP_KERNEL_ACCOUNT where it makes sense
- Replace PDS_VFIO_DRV_NAME with KBUILD_MODNAME
- Update to latest pds_core exported functions
- Remove duplicated prototypes for
pds_vfio_dma_logging_[start|stop|report] from lm.h
- Hold pds_vfio->state_mutex while starting, stopping, and reporting
dirty page tracking in pds_vfio_dma_logging_[start|stop|report]
- Remove duplicate PDS_DEV_TYPE_LM_STR define from pds_lm.h that's
already included in pds_common.h
- Replace use of dma_alloc_coherent() when creating the sgl DMA
entries for the dirty bitmaps
v8:
https://lore.kernel.org/netdev/20230404190141.57762-1-brett.creeley@amd.com/
- provide default iommufd callbacks for bind_iommufd, unbind_iommufd, and
attach_ioas for the VFIO device as suggested by Shameerali Kolothum
Thodi
v7:
https://lore.kernel.org/netdev/20230331003612.17569-1-brett.creeley@amd.com/
- Disable and clean up dirty page tracking when the VFIO device is closed
- Various improvements suggested by Simon Horman:
- Fix RCT in vfio_combine_iova_ranges()
- Simplify function exit paths by removing unnecessary goto
labels
- Cleanup pds_vifo_print_guest_region_info() by adding a goto
label for freeing memory, which allowed for reduced
indentation on a for loop
- Where possible use C99 style for loops
v6:
https://lore.kernel.org/netdev/20230327200553.13951-1-brett.creeley@amd.com/
- As suggested by Alex Williamson, use pci_domain_nr() macro to make sure
the pds_vfio client's devname is unique
- Remove unnecessary forward declaration and include
- Fix copyright comment to use correct company name
- Remove "." from struct documentation for consistency
v5:
https://lore.kernel.org/netdev/20230322203442.56169-1-brett.creeley@amd.com/
- Fix SPDX comments in .h files
- Remove adminqcq argument from pdsc_post_adminq() uses
- Unregister client on vfio_pci_core_register_device() failure
- Other minor checkpatch issues
v4:
https://lore.kernel.org/netdev/20230308052450.13421-1-brett.creeley@amd.com/
- Update cover letter ASCII diagram to reflect new driver architecture
- Remove auxiliary driver implementation
- Use pds_core's exported functions to communicate with the device
- Implement and register notifier for events from the device/pds_core
- Use module_pci_driver() macro since auxiliary driver configuration is
no longer needed in __init/__exit
v3:
https://lore.kernel.org/netdev/20230219083908.40013-1-brett.creeley@amd.com/
- Update copyright year to 2023 and use "Advanced Micro Devices, Inc."
for the company name
- Clarify the fact that AMD/Pensando's VFIO solution is device type
agnostic, which aligns with other current VFIO solutions
- Add line in drivers/vfio/pci/Makefile to build pds_vfio
- Move documentation to amd sub-directory
- Remove some dead code due to the pds_core implementation of
listening to BIND/UNBIND events
- Move a dev_dbg() to a previous patch in the series
- Add implementation for vfio_migration_ops.migration_get_data_size to
return the maximum possible device state size
RFC to v2:
https://lore.kernel.org/all/20221214232136.64220-1-brett.creeley@amd.com/
- Implement state transitions for VFIO_MIGRATION_P2P flag
- Improve auxiliary driver probe by returning EPROBE_DEFER
when the PCI driver is not set up correctly
- Add pointer to docs in
Documentation/networking/device_drivers/ethernet/index.rst
RFC:
https://lore.kernel.org/all/20221207010705.35128-1-brett.creeley@amd.com/
Brett Creeley (7):
vfio: Commonize combine_ranges for use in other VFIO drivers
vfio/pds: Initial support for pds_vfio VFIO driver
vfio/pds: register with the pds_core PF
vfio/pds: Add VFIO live migration support
vfio/pds: Add support for dirty page tracking
vfio/pds: Add support for firmware recovery
vfio/pds: Add Kconfig and documentation
.../device_drivers/ethernet/amd/pds_vfio.rst | 79 +++
.../device_drivers/ethernet/index.rst | 1 +
MAINTAINERS | 7 +
drivers/vfio/pci/Kconfig | 2 +
drivers/vfio/pci/Makefile | 2 +
drivers/vfio/pci/mlx5/cmd.c | 48 +-
drivers/vfio/pci/pds/Kconfig | 20 +
drivers/vfio/pci/pds/Makefile | 11 +
drivers/vfio/pci/pds/cmds.c | 487 +++++++++++++++
drivers/vfio/pci/pds/cmds.h | 25 +
drivers/vfio/pci/pds/dirty.c | 577 ++++++++++++++++++
drivers/vfio/pci/pds/dirty.h | 38 ++
drivers/vfio/pci/pds/lm.c | 421 +++++++++++++
drivers/vfio/pci/pds/lm.h | 41 ++
drivers/vfio/pci/pds/pci_drv.c | 206 +++++++
drivers/vfio/pci/pds/pci_drv.h | 9 +
drivers/vfio/pci/pds/vfio_dev.c | 234 +++++++
drivers/vfio/pci/pds/vfio_dev.h | 45 ++
drivers/vfio/vfio_main.c | 47 ++
include/linux/pds/pds_adminq.h | 395 ++++++++++++
include/linux/pds/pds_common.h | 2 +
include/linux/vfio.h | 3 +
22 files changed, 2653 insertions(+), 47 deletions(-)
create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_vfio.rst
create mode 100644 drivers/vfio/pci/pds/Kconfig
create mode 100644 drivers/vfio/pci/pds/Makefile
create mode 100644 drivers/vfio/pci/pds/cmds.c
create mode 100644 drivers/vfio/pci/pds/cmds.h
create mode 100644 drivers/vfio/pci/pds/dirty.c
create mode 100644 drivers/vfio/pci/pds/dirty.h
create mode 100644 drivers/vfio/pci/pds/lm.c
create mode 100644 drivers/vfio/pci/pds/lm.h
create mode 100644 drivers/vfio/pci/pds/pci_drv.c
create mode 100644 drivers/vfio/pci/pds/pci_drv.h
create mode 100644 drivers/vfio/pci/pds/vfio_dev.c
create mode 100644 drivers/vfio/pci/pds/vfio_dev.h
--
2.17.1
next reply other threads:[~2023-06-02 22:03 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-02 22:03 Brett Creeley [this message]
2023-06-02 22:03 ` [PATCH v10 vfio 1/7] vfio: Commonize combine_ranges for use in other VFIO drivers Brett Creeley
2023-06-16 6:52 ` Tian, Kevin
2023-06-16 18:37 ` Brett Creeley
2023-06-02 22:03 ` [PATCH v10 vfio 2/7] vfio/pds: Initial support for pds_vfio VFIO driver Brett Creeley
2023-06-14 21:31 ` Alex Williamson
2023-06-14 21:41 ` Brett Creeley
2023-06-16 6:56 ` Tian, Kevin
2023-06-16 18:42 ` Brett Creeley
2023-06-02 22:03 ` [PATCH v10 vfio 3/7] vfio/pds: register with the pds_core PF Brett Creeley
2023-06-15 21:05 ` Shameerali Kolothum Thodi
2023-06-15 21:30 ` Brett Creeley
2023-06-16 7:04 ` Tian, Kevin
2023-06-16 19:01 ` Brett Creeley
2023-06-20 2:11 ` Tian, Kevin
2023-06-02 22:03 ` [PATCH v10 vfio 4/7] vfio/pds: Add VFIO live migration support Brett Creeley
2023-06-15 21:07 ` Shameerali Kolothum Thodi
2023-06-15 21:36 ` Brett Creeley
2023-06-16 8:06 ` Tian, Kevin
2023-06-17 4:45 ` Brett Creeley
2023-06-20 2:19 ` Tian, Kevin
2023-06-19 12:46 ` Jason Gunthorpe
2023-06-20 2:02 ` Tian, Kevin
2023-06-20 12:31 ` Jason Gunthorpe
2023-06-21 6:49 ` Tian, Kevin
2023-06-21 13:27 ` Jason Gunthorpe
2023-06-26 7:31 ` Tian, Kevin
2023-06-26 18:13 ` Jason Gunthorpe
2023-06-27 6:03 ` Tian, Kevin
2023-06-02 22:03 ` [PATCH v10 vfio 5/7] vfio/pds: Add support for dirty page tracking Brett Creeley
2023-06-02 22:03 ` [PATCH v10 vfio 6/7] vfio/pds: Add support for firmware recovery Brett Creeley
2023-06-16 8:24 ` Tian, Kevin
2023-06-17 0:47 ` Brett Creeley
2023-06-02 22:03 ` [PATCH v10 vfio 7/7] vfio/pds: Add Kconfig and documentation Brett Creeley
2023-06-16 8:25 ` Tian, Kevin
2023-06-16 20:05 ` Brett Creeley
2023-06-14 20:20 ` [PATCH v10 vfio 0/7] pds_vfio driver Alex Williamson
2023-06-16 6:47 ` Tian, Kevin
2023-06-16 20:06 ` Brett Creeley
2023-06-17 4:49 ` Brett Creeley
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230602220318.15323-1-brett.creeley@amd.com \
--to=brett.creeley@amd.com \
--cc=alex.williamson@redhat.com \
--cc=jgg@nvidia.com \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=shannon.nelson@amd.com \
--cc=yishaih@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).