[GIT PULL] Please pull mlx5 vfio changes

[GIT PULL] Please pull mlx5 vfio changes

[Leon Romanovsky]

Hi Alex,

This series is based on clean 6.0-rc4 as such it causes to two small merge
conficts whis vfio-next. One is in thrird patch where you should take whole
chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
header includes, which you should take too.

----------------------------------------------------------------
The following changes since commit 7e18e42e4b280c85b76967a9106a13ca61c16179:

  Linux 6.0-rc4 (2022-09-04 13:10:01 -0700)

are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git tags/mlx5-dma-logging

for you to fetch changes up to 7f5dc74155a9de8bf6adc86d84e26c5a1e762431:

  vfio/mlx5: Set the driver DMA logging callbacks (2022-09-07 12:11:22 +0300)

----------------------------------------------------------------
Add device DMA logging support for mlx5 driver

From Yishai:

This series adds device DMA logging uAPIs and their implementation as
part of mlx5 driver.

DMA logging allows a device to internally record what DMAs the device is
initiating and report them back to userspace. It is part of the VFIO
migration infrastructure that allows implementing dirty page tracking
during the pre copy phase of live migration. Only DMA WRITEs are logged,
and this API is not connected to VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE.

The uAPIs are based on the FEATURE ioctl as were introduced earlier by
the below RFC [1] and follows the notes that were discussed in the
mailing list.

It includes:
- A PROBE option to detect if the device supports DMA logging.
- A SET option to start device DMA logging in given IOVAs ranges.
- A GET option to read back and clear the device DMA log.
- A SET option to stop device DMA logging that was previously started.

Extra details exist as part of relevant patches in the series.

In addition, the series adds some infrastructure support for managing an
IOVA bitmap done by Joao Martins.

It abstracts how an IOVA range is represented in a bitmap that is
granulated by a given page_size. So it translates all the lifting of
dealing with user pointers into its corresponding kernel addresses.
This new functionality abstracts the complexity of user/kernel bitmap
pointer usage and finally enables an API to set some bits.

This functionality will be used as part of IOMMUFD series for the system
IOMMU tracking.

Finally, we come with mlx5 implementation based on its device
specification for the DMA logging APIs.

The matching qemu changes can be previewed here [2].
They come on top of the v2 migration protocol patches that were sent
already to the mailing list.

Note:
- As this series touched mlx5_core parts we may need to send the
  net/mlx5 patches as a pull request format to VFIO to avoid conflicts
  before acceptance.

[1] https://lore.kernel.org/all/20220501123301.127279-1-yishaih@nvidia.com/T/
[2] https://github.com/avihai1122/qemu/commits/device_dirty_tracking

Link: https://lore.kernel.org/all/20220905105852.26398-1-yishaih@nvidia.com/
Signed-of-by: Leon Romanovsky <leon@kernel.org>

----------------------------------------------------------------
Joao Martins (1):
      vfio: Add an IOVA bitmap support

Yishai Hadas (9):
      net/mlx5: Introduce ifc bits for page tracker
      net/mlx5: Query ADV_VIRTUALIZATION capabilities
      vfio: Introduce DMA logging uAPIs
      vfio: Introduce the DMA logging feature support
      vfio/mlx5: Init QP based resources for dirty tracking
      vfio/mlx5: Create and destroy page tracker object
      vfio/mlx5: Report dirty pages from tracker
      vfio/mlx5: Manage error scenarios on tracker
      vfio/mlx5: Set the driver DMA logging callbacks

 drivers/net/ethernet/mellanox/mlx5/core/fw.c   |   6 +
 drivers/net/ethernet/mellanox/mlx5/core/main.c |   1 +
 drivers/vfio/Kconfig                           |   1 +
 drivers/vfio/Makefile                          |   6 +-
 drivers/vfio/iova_bitmap.c                     | 422 +++++++++++
 drivers/vfio/pci/mlx5/cmd.c                    | 995 ++++++++++++++++++++++++-
 drivers/vfio/pci/mlx5/cmd.h                    |  63 +-
 drivers/vfio/pci/mlx5/main.c                   |   9 +-
 drivers/vfio/pci/vfio_pci_core.c               |   5 +
 drivers/vfio/vfio_main.c                       | 175 +++++
 include/linux/iova_bitmap.h                    |  26 +
 include/linux/mlx5/device.h                    |   9 +
 include/linux/mlx5/mlx5_ifc.h                  |  83 ++-
 include/linux/vfio.h                           |  28 +-
 include/uapi/linux/vfio.h                      |  86 +++
 15 files changed, 1895 insertions(+), 20 deletions(-)
 create mode 100644 drivers/vfio/iova_bitmap.c
 create mode 100644 include/linux/iova_bitmap.h

Re: [GIT PULL] Please pull mlx5 vfio changes

[Alex Williamson]

On Wed,  7 Sep 2022 12:43:44 +0300
Leon Romanovsky <leon@kernel.org> wrote:

> Hi Alex,
> 
> This series is based on clean 6.0-rc4 as such it causes to two small merge
> conficts whis vfio-next. One is in thrird patch where you should take whole
> chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
> header includes, which you should take too.

Is there any reason you can't provide a topic branch for the two
net/mlx5 patches and the remainder are rebased and committed through
the vfio tree?  I don't see this as an exceptional case that requires
resolving conflicts in merge commits.  Thanks,

Alex


> ----------------------------------------------------------------
> The following changes since commit 7e18e42e4b280c85b76967a9106a13ca61c16179:
> 
>   Linux 6.0-rc4 (2022-09-04 13:10:01 -0700)
> 
> are available in the Git repository at:
> 
>   https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git tags/mlx5-dma-logging
> 
> for you to fetch changes up to 7f5dc74155a9de8bf6adc86d84e26c5a1e762431:
> 
>   vfio/mlx5: Set the driver DMA logging callbacks (2022-09-07 12:11:22 +0300)
> 
> ----------------------------------------------------------------
> Add device DMA logging support for mlx5 driver
> 
> From Yishai:
> 
> This series adds device DMA logging uAPIs and their implementation as
> part of mlx5 driver.
> 
> DMA logging allows a device to internally record what DMAs the device is
> initiating and report them back to userspace. It is part of the VFIO
> migration infrastructure that allows implementing dirty page tracking
> during the pre copy phase of live migration. Only DMA WRITEs are logged,
> and this API is not connected to VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE.
> 
> The uAPIs are based on the FEATURE ioctl as were introduced earlier by
> the below RFC [1] and follows the notes that were discussed in the
> mailing list.
> 
> It includes:
> - A PROBE option to detect if the device supports DMA logging.
> - A SET option to start device DMA logging in given IOVAs ranges.
> - A GET option to read back and clear the device DMA log.
> - A SET option to stop device DMA logging that was previously started.
> 
> Extra details exist as part of relevant patches in the series.
> 
> In addition, the series adds some infrastructure support for managing an
> IOVA bitmap done by Joao Martins.
> 
> It abstracts how an IOVA range is represented in a bitmap that is
> granulated by a given page_size. So it translates all the lifting of
> dealing with user pointers into its corresponding kernel addresses.
> This new functionality abstracts the complexity of user/kernel bitmap
> pointer usage and finally enables an API to set some bits.
> 
> This functionality will be used as part of IOMMUFD series for the system
> IOMMU tracking.
> 
> Finally, we come with mlx5 implementation based on its device
> specification for the DMA logging APIs.
> 
> The matching qemu changes can be previewed here [2].
> They come on top of the v2 migration protocol patches that were sent
> already to the mailing list.
> 
> Note:
> - As this series touched mlx5_core parts we may need to send the
>   net/mlx5 patches as a pull request format to VFIO to avoid conflicts
>   before acceptance.
> 
> [1] https://lore.kernel.org/all/20220501123301.127279-1-yishaih@nvidia.com/T/
> [2] https://github.com/avihai1122/qemu/commits/device_dirty_tracking
> 
> Link: https://lore.kernel.org/all/20220905105852.26398-1-yishaih@nvidia.com/
> Signed-of-by: Leon Romanovsky <leon@kernel.org>
> 
> ----------------------------------------------------------------
> Joao Martins (1):
>       vfio: Add an IOVA bitmap support
> 
> Yishai Hadas (9):
>       net/mlx5: Introduce ifc bits for page tracker
>       net/mlx5: Query ADV_VIRTUALIZATION capabilities
>       vfio: Introduce DMA logging uAPIs
>       vfio: Introduce the DMA logging feature support
>       vfio/mlx5: Init QP based resources for dirty tracking
>       vfio/mlx5: Create and destroy page tracker object
>       vfio/mlx5: Report dirty pages from tracker
>       vfio/mlx5: Manage error scenarios on tracker
>       vfio/mlx5: Set the driver DMA logging callbacks
> 
>  drivers/net/ethernet/mellanox/mlx5/core/fw.c   |   6 +
>  drivers/net/ethernet/mellanox/mlx5/core/main.c |   1 +
>  drivers/vfio/Kconfig                           |   1 +
>  drivers/vfio/Makefile                          |   6 +-
>  drivers/vfio/iova_bitmap.c                     | 422 +++++++++++
>  drivers/vfio/pci/mlx5/cmd.c                    | 995 ++++++++++++++++++++++++-
>  drivers/vfio/pci/mlx5/cmd.h                    |  63 +-
>  drivers/vfio/pci/mlx5/main.c                   |   9 +-
>  drivers/vfio/pci/vfio_pci_core.c               |   5 +
>  drivers/vfio/vfio_main.c                       | 175 +++++
>  include/linux/iova_bitmap.h                    |  26 +
>  include/linux/mlx5/device.h                    |   9 +
>  include/linux/mlx5/mlx5_ifc.h                  |  83 ++-
>  include/linux/vfio.h                           |  28 +-
>  include/uapi/linux/vfio.h                      |  86 +++
>  15 files changed, 1895 insertions(+), 20 deletions(-)
>  create mode 100644 drivers/vfio/iova_bitmap.c
>  create mode 100644 include/linux/iova_bitmap.h
> 

Re: [GIT PULL] Please pull mlx5 vfio changes

[Leon Romanovsky]

On Wed, Sep 07, 2022 at 01:21:19PM -0600, Alex Williamson wrote:
> On Wed,  7 Sep 2022 12:43:44 +0300
> Leon Romanovsky <leon@kernel.org> wrote:
> 
> > Hi Alex,
> > 
> > This series is based on clean 6.0-rc4 as such it causes to two small merge
> > conficts whis vfio-next. One is in thrird patch where you should take whole
> > chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
> > header includes, which you should take too.
> 
> Is there any reason you can't provide a topic branch for the two
> net/mlx5 patches and the remainder are rebased and committed through
> the vfio tree?  

You added your Acked-by to vfio/mlx5 patches and for me it is a sign to
prepare clean PR with whole series.

I reset mlx5-vfio topic to have only two net/mlx5 commits without
special tag.

https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git topic/mlx5-vfio
Everything else can go directly to your tree without my intervention.

Thanks

Re: [GIT PULL] Please pull mlx5 vfio changes

[Alex Williamson]

On Thu, 8 Sep 2022 09:31:13 +0300
Leon Romanovsky <leon@kernel.org> wrote:

> On Wed, Sep 07, 2022 at 01:21:19PM -0600, Alex Williamson wrote:
> > On Wed,  7 Sep 2022 12:43:44 +0300
> > Leon Romanovsky <leon@kernel.org> wrote:
> >   
> > > Hi Alex,
> > > 
> > > This series is based on clean 6.0-rc4 as such it causes to two small merge
> > > conficts whis vfio-next. One is in thrird patch where you should take whole
> > > chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
> > > header includes, which you should take too.  
> > 
> > Is there any reason you can't provide a topic branch for the two
> > net/mlx5 patches and the remainder are rebased and committed through
> > the vfio tree?    
> 
> You added your Acked-by to vfio/mlx5 patches and for me it is a sign to
> prepare clean PR with whole series.
> 
> I reset mlx5-vfio topic to have only two net/mlx5 commits without
> special tag.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git topic/mlx5-vfio
> Everything else can go directly to your tree without my intervention.

Sorry, I knew the intention initially was to send a PR and I didn't
think about the conflicts we'd have versus the base you'd use.  Thanks
for splitting this out, I think it'll make for a cleaner upstream path
given the clear code split.

Yishai, can you post a v7 rebased on the vfio next branch?  The comment
I requested is now ephemeral since it only existed in the commits Leon
dropped.  Also feel free to drop my Acks since I'll add new Sign-offs.
Thanks,

Alex

Re: [GIT PULL] Please pull mlx5 vfio changes

[Yishai Hadas]

On 08/09/2022 19:53, Alex Williamson wrote:
> On Thu, 8 Sep 2022 09:31:13 +0300
> Leon Romanovsky <leon@kernel.org> wrote:
>
>> On Wed, Sep 07, 2022 at 01:21:19PM -0600, Alex Williamson wrote:
>>> On Wed,  7 Sep 2022 12:43:44 +0300
>>> Leon Romanovsky <leon@kernel.org> wrote:
>>>    
>>>> Hi Alex,
>>>>
>>>> This series is based on clean 6.0-rc4 as such it causes to two small merge
>>>> conficts whis vfio-next. One is in thrird patch where you should take whole
>>>> chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
>>>> header includes, which you should take too.
>>> Is there any reason you can't provide a topic branch for the two
>>> net/mlx5 patches and the remainder are rebased and committed through
>>> the vfio tree?
>> You added your Acked-by to vfio/mlx5 patches and for me it is a sign to
>> prepare clean PR with whole series.
>>
>> I reset mlx5-vfio topic to have only two net/mlx5 commits without
>> special tag.
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git topic/mlx5-vfio
>> Everything else can go directly to your tree without my intervention.
> Sorry, I knew the intention initially was to send a PR and I didn't
> think about the conflicts we'd have versus the base you'd use.  Thanks
> for splitting this out, I think it'll make for a cleaner upstream path
> given the clear code split.
>
> Yishai, can you post a v7 rebased on the vfio next branch?


Sure

Do you want me to include in V7 the two net/mlx5 patches that are part 
of the PR or that you'll take them first from the PR, publish your 
vfio/next tree and I'll drop them from V7 ?

> The comment
> I requested is now ephemeral since it only existed in the commits Leon
> dropped.  Also feel free to drop my Acks since I'll add new Sign-offs.
> Thanks,

OK, will drop them.

>
> Alex
>

Re: [GIT PULL] Please pull mlx5 vfio changes

[Alex Williamson]

On Thu, 8 Sep 2022 20:31:35 +0300
Yishai Hadas <yishaih@nvidia.com> wrote:

> On 08/09/2022 19:53, Alex Williamson wrote:
> > On Thu, 8 Sep 2022 09:31:13 +0300
> > Leon Romanovsky <leon@kernel.org> wrote:
> >  
> >> On Wed, Sep 07, 2022 at 01:21:19PM -0600, Alex Williamson wrote:  
> >>> On Wed,  7 Sep 2022 12:43:44 +0300
> >>> Leon Romanovsky <leon@kernel.org> wrote:
> >>>      
> >>>> Hi Alex,
> >>>>
> >>>> This series is based on clean 6.0-rc4 as such it causes to two small merge
> >>>> conficts whis vfio-next. One is in thrird patch where you should take whole
> >>>> chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
> >>>> header includes, which you should take too.  
> >>> Is there any reason you can't provide a topic branch for the two
> >>> net/mlx5 patches and the remainder are rebased and committed through
> >>> the vfio tree?  
> >> You added your Acked-by to vfio/mlx5 patches and for me it is a sign to
> >> prepare clean PR with whole series.
> >>
> >> I reset mlx5-vfio topic to have only two net/mlx5 commits without
> >> special tag.
> >>
> >> https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git topic/mlx5-vfio
> >> Everything else can go directly to your tree without my intervention.  
> > Sorry, I knew the intention initially was to send a PR and I didn't
> > think about the conflicts we'd have versus the base you'd use.  Thanks
> > for splitting this out, I think it'll make for a cleaner upstream path
> > given the clear code split.
> >
> > Yishai, can you post a v7 rebased on the vfio next branch?  
> 
> 
> Sure
> 
> Do you want me to include in V7 the two net/mlx5 patches that are part 
> of the PR or that you'll take them first from the PR, publish your 
> vfio/next tree and I'll drop them from V7 ?

For the sake of having a series that compiles and doesn't confuse
anyone with buildbot errors, please include them, but I'll pull them
from Leon's topic branch rather than applying them directly.  Thanks,

Alex

[GIT PULL] Please pull mlx5 vfio changes

[Leon Romanovsky]

The following changes since commit 2984287c4c19949d7eb451dcad0bd5c54a2a376f:

  net/mlx5: Remove not-implemented IPsec capabilities (2022-04-09 08:25:07 +0300)

are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/ tags/mlx5-lm-parallel

for you to fetch changes up to 4d67783344e8380ec3c0810e4f67c003646ccd7e:

  vfio/mlx5: Run the SAVE state command in an async mode (2022-05-10 15:46:42 +0300)

----------------------------------------------------------------
Improve mlx5 live migration driver

From Yishai:

This series improves mlx5 live migration driver in few aspects as of
below.

Refactor to enable running migration commands in parallel over the PF
command interface.

To achieve that we exposed from mlx5_core an API to let the VF be
notified before that the PF command interface goes down/up. (e.g. PF
reload upon health recovery).

Once having the above functionality in place mlx5 vfio doesn't need any
more to obtain the global PF lock upon using the command interface but
can rely on the above mechanism to be in sync with the PF.

This can enable parallel VFs migration over the PF command interface
from kernel driver point of view.

In addition,
Moved to use the PF async command mode for the SAVE state command.
This enables returning earlier to user space upon issuing successfully
the command and improve latency by let things run in parallel.

Alex, as this series touches mlx5_core we may need to send this in a
pull request format to VFIO to avoid conflicts before acceptance.

Link: https://lore.kernel.org/all/20220510090206.90374-1-yishaih@nvidia.com
Signed-of-by: Leon Romanovsky <leonro@nvidia.com>

----------------------------------------------------------------
Yishai Hadas (4):
      net/mlx5: Expose mlx5_sriov_blocking_notifier_register / unregister APIs
      vfio/mlx5: Manage the VF attach/detach callback from the PF
      vfio/mlx5: Refactor to enable VFs migration in parallel
      vfio/mlx5: Run the SAVE state command in an async mode

 drivers/net/ethernet/mellanox/mlx5/core/sriov.c |  65 ++++++-
 drivers/vfio/pci/mlx5/cmd.c                     | 236 ++++++++++++++++++------
 drivers/vfio/pci/mlx5/cmd.h                     |  52 +++++-
 drivers/vfio/pci/mlx5/main.c                    | 122 +++++-------
 include/linux/mlx5/driver.h                     |  12 ++
 5 files changed, 351 insertions(+), 136 deletions(-)