From: Alex Williamson <alex.williamson@redhat.com>
To: Max Gurtovoy <mgurtovoy@nvidia.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Leon Romanovsky <leon@kernel.org>,
"Doug Ledford" <dledford@redhat.com>,
Yishai Hadas <yishaih@nvidia.com>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Kirti Wankhede <kwankhede@nvidia.com>, <kvm@vger.kernel.org>,
<linux-kernel@vger.kernel.org>, <linux-pci@vger.kernel.org>,
<linux-rdma@vger.kernel.org>, <netdev@vger.kernel.org>,
Saeed Mahameed <saeedm@nvidia.com>,
Cornelia Huck <cohuck@redhat.com>
Subject: Re: [PATCH mlx5-next 2/7] vfio: Add an API to check migration state transition validity
Date: Wed, 29 Sep 2021 09:17:12 -0600 [thread overview]
Message-ID: <20210929091712.6390141c.alex.williamson@redhat.com> (raw)
In-Reply-To: <d2e94241-a146-c57d-cf81-8b7d8d00e62d@nvidia.com>
On Wed, 29 Sep 2021 17:36:59 +0300
Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> On 9/29/2021 4:50 PM, Alex Williamson wrote:
> > On Wed, 29 Sep 2021 16:26:55 +0300
> > Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> >
> >> On 9/29/2021 3:35 PM, Alex Williamson wrote:
> >>> On Wed, 29 Sep 2021 13:44:10 +0300
> >>> Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> >>>
> >>>> On 9/28/2021 2:12 AM, Jason Gunthorpe wrote:
> >>>>> On Mon, Sep 27, 2021 at 04:46:48PM -0600, Alex Williamson wrote:
> >>>>>>> + enum { MAX_STATE = VFIO_DEVICE_STATE_RESUMING };
> >>>>>>> + static const u8 vfio_from_state_table[MAX_STATE + 1][MAX_STATE + 1] = {
> >>>>>>> + [VFIO_DEVICE_STATE_STOP] = {
> >>>>>>> + [VFIO_DEVICE_STATE_RUNNING] = 1,
> >>>>>>> + [VFIO_DEVICE_STATE_RESUMING] = 1,
> >>>>>>> + },
> >>>>>> Our state transition diagram is pretty weak on reachable transitions
> >>>>>> out of the _STOP state, why do we select only these two as valid?
> >>>>> I have no particular opinion on specific states here, however adding
> >>>>> more states means more stuff for drivers to implement and more risk
> >>>>> driver writers will mess up this uAPI.
> >>>> _STOP == 000b => Device Stopped, not saving or resuming (from UAPI).
> >>>>
> >>>> This is the default initial state and not RUNNING.
> >>>>
> >>>> The user application should move device from STOP => RUNNING or STOP =>
> >>>> RESUMING.
> >>>>
> >>>> Maybe we need to extend the comment in the UAPI file.
> >>> include/uapi/linux/vfio.h:
> >>> ...
> >>> * +------- _RESUMING
> >>> * |+------ _SAVING
> >>> * ||+----- _RUNNING
> >>> * |||
> >>> * 000b => Device Stopped, not saving or resuming
> >>> * 001b => Device running, which is the default state
> >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>> ...
> >>> * State transitions:
> >>> *
> >>> * _RESUMING _RUNNING Pre-copy Stop-and-copy _STOP
> >>> * (100b) (001b) (011b) (010b) (000b)
> >>> * 0. Running or default state
> >>> * |
> >>> ^^^^^^^^^^^^^
> >>> ...
> >>> * 0. Default state of VFIO device is _RUNNING when the user application starts.
> >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>>
> >>> The uAPI is pretty clear here. A default state of _STOP is not
> >>> compatible with existing devices and userspace that does not support
> >>> migration. Thanks,
> >> Why do you need this state machine for userspace that doesn't support
> >> migration ?
> > For userspace that doesn't support migration, there's one state,
> > _RUNNING. That's what we're trying to be compatible and consistent
> > with. Migration is an extension, not a base requirement.
>
> Userspace without migration doesn't care about this state.
>
> We left with kernel now. vfio-pci today doesn't support migration, right
> ? state is in theory is 0 (STOP).
>
> This state machine is controlled by the migration SW. The drivers don't
> move state implicitly.
>
> mlx5-vfio-pci support migration and will work fine with non-migration SW
> (it will stay with state = 0 unless someone will move it. but nobody
> will) exactly like vfio-pci does today.
>
> So where is the problem ?
So you have a device that's actively modifying its internal state,
performing I/O, including DMA (thereby dirtying VM memory), all while
in the _STOP state? And you don't see this as a problem?
There's a major inconsistency if the migration interface is telling us
something different than we can actually observe through the behavior of
the device. Thanks,
Alex
next prev parent reply other threads:[~2021-09-29 15:17 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <cover.1632305919.git.leonro@nvidia.com>
2021-09-22 10:38 ` [PATCH mlx5-next 1/7] PCI/IOV: Provide internal VF index Leon Romanovsky
2021-09-22 21:59 ` Bjorn Helgaas
2021-09-23 6:35 ` Leon Romanovsky
2021-09-24 13:08 ` Bjorn Helgaas
2021-09-25 10:10 ` Leon Romanovsky
2021-09-25 17:41 ` Bjorn Helgaas
2021-09-26 6:36 ` Leon Romanovsky
2021-09-26 20:23 ` Bjorn Helgaas
2021-09-27 11:55 ` Leon Romanovsky
2021-09-27 14:47 ` Bjorn Helgaas
2021-09-22 10:38 ` [PATCH mlx5-next 2/7] vfio: Add an API to check migration state transition validity Leon Romanovsky
2021-09-23 10:33 ` Shameerali Kolothum Thodi
2021-09-23 11:17 ` Leon Romanovsky
2021-09-23 13:55 ` Max Gurtovoy
2021-09-24 7:44 ` Shameerali Kolothum Thodi
2021-09-24 9:37 ` Kirti Wankhede
2021-09-26 9:09 ` Max Gurtovoy
2021-09-26 16:17 ` Shameerali Kolothum Thodi
2021-09-27 18:24 ` Max Gurtovoy
2021-09-27 18:29 ` Shameerali Kolothum Thodi
2021-09-27 22:46 ` Alex Williamson
2021-09-27 23:12 ` Jason Gunthorpe
2021-09-28 19:19 ` Alex Williamson
2021-09-28 19:35 ` Jason Gunthorpe
2021-09-28 20:18 ` Alex Williamson
2021-09-29 16:16 ` Jason Gunthorpe
2021-09-29 18:06 ` Alex Williamson
2021-09-29 18:26 ` Jason Gunthorpe
2021-09-29 10:57 ` Max Gurtovoy
2021-09-29 10:44 ` Max Gurtovoy
2021-09-29 12:35 ` Alex Williamson
2021-09-29 13:26 ` Max Gurtovoy
2021-09-29 13:50 ` Alex Williamson
2021-09-29 14:36 ` Max Gurtovoy
2021-09-29 15:17 ` Alex Williamson [this message]
2021-09-29 15:28 ` Max Gurtovoy
2021-09-29 16:14 ` Jason Gunthorpe
2021-09-29 21:48 ` Max Gurtovoy
2021-09-29 22:44 ` Alex Williamson
2021-09-30 9:25 ` Max Gurtovoy
2021-09-30 12:41 ` Alex Williamson
2021-09-29 23:21 ` Jason Gunthorpe
2021-09-30 9:34 ` Max Gurtovoy
2021-09-30 14:47 ` Jason Gunthorpe
2021-09-30 15:32 ` Max Gurtovoy
2021-09-30 16:24 ` Jason Gunthorpe
2021-09-30 16:51 ` Max Gurtovoy
2021-09-30 17:01 ` Jason Gunthorpe
2021-09-22 10:38 ` [PATCH mlx5-next 3/7] vfio/pci_core: Make the region->release() function optional Leon Romanovsky
2021-09-23 13:57 ` Max Gurtovoy
2021-09-22 10:38 ` [PATCH mlx5-next 4/7] net/mlx5: Introduce migration bits and structures Leon Romanovsky
2021-09-24 5:48 ` Mark Zhang
2021-09-22 10:38 ` [PATCH mlx5-next 5/7] net/mlx5: Expose APIs to get/put the mlx5 core device Leon Romanovsky
2021-09-22 10:38 ` [PATCH mlx5-next 6/7] mlx5_vfio_pci: Expose migration commands over mlx5 device Leon Romanovsky
2021-09-28 20:22 ` Alex Williamson
2021-09-29 5:36 ` Leon Romanovsky
2021-09-22 10:38 ` [PATCH mlx5-next 7/7] mlx5_vfio_pci: Implement vfio_pci driver for mlx5 devices Leon Romanovsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210929091712.6390141c.alex.williamson@redhat.com \
--to=alex.williamson@redhat.com \
--cc=bhelgaas@google.com \
--cc=cohuck@redhat.com \
--cc=davem@davemloft.net \
--cc=dledford@redhat.com \
--cc=jgg@ziepe.ca \
--cc=kuba@kernel.org \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mgurtovoy@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=saeedm@nvidia.com \
--cc=yishaih@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).