From: Keith Busch <kbusch@kernel.org>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Christoph Hellwig <hch@lst.de>, Lei Rao <lei.rao@intel.com>,
axboe@fb.com, kch@nvidia.com, sagi@grimberg.me,
alex.williamson@redhat.com, cohuck@redhat.com,
yishaih@nvidia.com, shameerali.kolothum.thodi@huawei.com,
kevin.tian@intel.com, mjrosato@linux.ibm.com,
linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
kvm@vger.kernel.org, eddie.dong@intel.com, yadong.li@intel.com,
yi.l.liu@intel.com, Konrad.wilk@oracle.com,
stephen@eideticom.com, hang.yuan@intel.com
Subject: Re: [RFC PATCH 1/5] nvme-pci: add function nvme_submit_vf_cmd to issue admin commands for VF driver.
Date: Tue, 6 Dec 2022 13:51:32 +0000 [thread overview]
Message-ID: <Y49I5MRsgHdNkIff@kbusch-mbp> (raw)
In-Reply-To: <Y49HKHP9NrId39iH@ziepe.ca>
On Tue, Dec 06, 2022 at 09:44:08AM -0400, Jason Gunthorpe wrote:
> On Tue, Dec 06, 2022 at 07:19:40AM +0100, Christoph Hellwig wrote:
> > On Tue, Dec 06, 2022 at 01:58:12PM +0800, Lei Rao wrote:
> > > The new function nvme_submit_vf_cmd() helps the host VF driver to issue
> > > VF admin commands. It's helpful in some cases that the host NVMe driver
> > > does not control VF's admin queue. For example, in the virtualization
> > > device pass-through case, the VF controller's admin queue is governed
> > > by the Guest NVMe driver. Host VF driver relies on PF device's admin
> > > queue to control VF devices like vendor-specific live migration commands.
> >
> > WTF are you even smoking when you think this would be acceptable?
>
> Not speaking to NVMe - but this driver is clearly copying mlx5's live
> migration driver, almost completely - including this basic function.
>
> So, to explain why mlx5 works this way..
>
> The VFIO approach is to fully assign an entire VF to the guest OS. The
> entire VF assignment means every MMIO register *and all the DMA* of
> the VF is owned by the guest operating system.
>
> mlx5 needs to transfer hundreds of megabytes to gigabytes of in-device
> state to perform a migration.
For storage, though, you can't just transfer the controller state. You have to
transfer all the namespace user data, too. So potentially many terabytes?
next prev parent reply other threads:[~2022-12-06 13:51 UTC|newest]
Thread overview: 65+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-06 5:58 [RFC PATCH 0/5] Add new VFIO PCI driver for NVMe devices Lei Rao
2022-12-06 5:58 ` [RFC PATCH 1/5] nvme-pci: add function nvme_submit_vf_cmd to issue admin commands for VF driver Lei Rao
2022-12-06 6:19 ` Christoph Hellwig
2022-12-06 13:44 ` Jason Gunthorpe
2022-12-06 13:51 ` Keith Busch [this message]
2022-12-06 14:27 ` Jason Gunthorpe
2022-12-06 13:58 ` Christoph Hellwig
2022-12-06 15:22 ` Jason Gunthorpe
2022-12-06 15:38 ` Christoph Hellwig
2022-12-06 15:51 ` Jason Gunthorpe
2022-12-06 16:55 ` Christoph Hellwig
2022-12-06 19:15 ` Jason Gunthorpe
2022-12-07 2:30 ` Max Gurtovoy
2022-12-07 7:58 ` Christoph Hellwig
2022-12-09 2:11 ` Tian, Kevin
2022-12-12 7:41 ` Christoph Hellwig
2022-12-07 7:54 ` Christoph Hellwig
2022-12-07 10:59 ` Max Gurtovoy
2022-12-07 13:46 ` Christoph Hellwig
2022-12-07 14:50 ` Max Gurtovoy
2022-12-07 16:35 ` Christoph Hellwig
2022-12-07 13:34 ` Jason Gunthorpe
2022-12-07 13:52 ` Christoph Hellwig
2022-12-07 15:07 ` Jason Gunthorpe
2022-12-07 16:38 ` Christoph Hellwig
2022-12-07 17:31 ` Jason Gunthorpe
2022-12-07 18:33 ` Christoph Hellwig
2022-12-07 20:08 ` Jason Gunthorpe
2022-12-09 2:50 ` Tian, Kevin
2022-12-09 18:56 ` Dong, Eddie
2022-12-11 11:39 ` Max Gurtovoy
2022-12-12 7:55 ` Christoph Hellwig
2022-12-12 14:49 ` Max Gurtovoy
2022-12-12 7:50 ` Christoph Hellwig
2022-12-13 14:01 ` Jason Gunthorpe
2022-12-13 16:08 ` Christoph Hellwig
2022-12-13 17:49 ` Jason Gunthorpe
2022-12-06 5:58 ` [RFC PATCH 2/5] nvme-vfio: add new vfio-pci driver for NVMe device Lei Rao
2022-12-06 5:58 ` [RFC PATCH 3/5] nvme-vfio: enable the function of VFIO live migration Lei Rao
2023-01-19 10:21 ` Max Gurtovoy
2023-02-09 9:09 ` Rao, Lei
2022-12-06 5:58 ` [RFC PATCH 4/5] nvme-vfio: check if the hardware supports " Lei Rao
2022-12-06 13:47 ` Keith Busch
2022-12-06 5:58 ` [RFC PATCH 5/5] nvme-vfio: Add a document for the NVMe device Lei Rao
2022-12-06 6:26 ` Christoph Hellwig
2022-12-06 13:05 ` Jason Gunthorpe
2022-12-06 13:09 ` Christoph Hellwig
2022-12-06 13:52 ` Jason Gunthorpe
2022-12-06 14:00 ` Christoph Hellwig
2022-12-06 14:20 ` Jason Gunthorpe
2022-12-06 14:31 ` Christoph Hellwig
2022-12-06 14:48 ` Jason Gunthorpe
2022-12-06 15:01 ` Christoph Hellwig
2022-12-06 15:28 ` Jason Gunthorpe
2022-12-06 15:35 ` Christoph Hellwig
2022-12-06 18:00 ` Dong, Eddie
2022-12-12 7:57 ` Christoph Hellwig
2022-12-11 12:05 ` Max Gurtovoy
2022-12-11 13:21 ` Rao, Lei
2022-12-11 14:51 ` Max Gurtovoy
2022-12-12 1:20 ` Rao, Lei
2022-12-12 8:09 ` Christoph Hellwig
2022-12-09 2:05 ` Tian, Kevin
2022-12-09 16:53 ` Li, Yadong
2022-12-12 8:11 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y49I5MRsgHdNkIff@kbusch-mbp \
--to=kbusch@kernel.org \
--cc=Konrad.wilk@oracle.com \
--cc=alex.williamson@redhat.com \
--cc=axboe@fb.com \
--cc=cohuck@redhat.com \
--cc=eddie.dong@intel.com \
--cc=hang.yuan@intel.com \
--cc=hch@lst.de \
--cc=jgg@ziepe.ca \
--cc=kch@nvidia.com \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=lei.rao@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=mjrosato@linux.ibm.com \
--cc=sagi@grimberg.me \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=stephen@eideticom.com \
--cc=yadong.li@intel.com \
--cc=yi.l.liu@intel.com \
--cc=yishaih@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).