* Re: [PATCH 1/2] vdpa_sim: not reset state in vdpasim_queue_ready
[not found] ` <20230118164359.1523760-2-eperezma@redhat.com>
@ 2023-01-19 3:16 ` Jason Wang
[not found] ` <CAJaqyWejkbiwKK4O0qOObdf294YrzMeSRTVoWNzv75yQCvkJqQ@mail.gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Jason Wang @ 2023-01-19 3:16 UTC (permalink / raw)
To: Eugenio Pérez
Cc: Laurent Vivier, lulu, mst, linux-kernel, Gautam Dawar,
virtualization, leiyang
On Thu, Jan 19, 2023 at 12:44 AM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> vdpasim_queue_ready calls vringh_init_iotlb, which resets split indexes.
> But it can be called after setting a ring base with
> vdpasim_set_vq_state.
>
> Fix it by stashing them. They're still resetted in vdpasim_vq_reset.
>
> This was discovered and tested live migrating the vdpa_sim_net device.
>
> Fixes: 2c53d0f64c06 ("vdpasim: vDPA device simulator")
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
> drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> index cb88891b44a8..8839232a3fcb 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> @@ -66,6 +66,7 @@ static void vdpasim_vq_notify(struct vringh *vring)
> static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
> {
> struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
> + uint16_t last_avail_idx = vq->vring.last_avail_idx;
>
> vringh_init_iotlb(&vq->vring, vdpasim->features, vq->num, false,
> (struct vring_desc *)(uintptr_t)vq->desc_addr,
> @@ -74,6 +75,7 @@ static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
> (struct vring_used *)
> (uintptr_t)vq->device_addr);
>
> + vq->vring.last_avail_idx = last_avail_idx;
Does this need to be serialized with the datapath?
E.g in set_vq_state() we do:
spin_lock(&vdpasim->lock);
vrh->last_avail_idx = state->split.avail_index;
spin_unlock(&vdpasim->lock);
Thanks
> vq->vring.notify = vdpasim_vq_notify;
> }
>
> --
> 2.31.1
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb
[not found] ` <20230118164359.1523760-3-eperezma@redhat.com>
@ 2023-01-19 3:20 ` Jason Wang
[not found] ` <CAJaqyWetovvndcU=pu_kPNUNYkgao=HsENnrKCzoHdK7RBjyAQ@mail.gmail.com>
2023-02-01 16:11 ` Michael S. Tsirkin
1 sibling, 1 reply; 8+ messages in thread
From: Jason Wang @ 2023-01-19 3:20 UTC (permalink / raw)
To: Eugenio Pérez
Cc: Laurent Vivier, lulu, mst, linux-kernel, Gautam Dawar,
virtualization, leiyang
On Thu, Jan 19, 2023 at 12:44 AM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> Starting from an used_idx different than 0 is needed in use cases like
> virtual machine migration. Not doing so and letting the caller set an
> avail idx different than 0 causes destination device to try to use old
> buffers that source driver already recover and are not available
> anymore.
>
> While callers like vdpa_sim set avail_idx directly it does not set
> used_idx. Instead of let the caller do the assignment, fetch it from
> the guest at initialization like vhost-kernel do.
>
> To perform the same at vring_kernel_init and vring_user_init is left for
> the future.
>
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
> drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
> 1 file changed, 23 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/vhost/vringh.c b/drivers/vhost/vringh.c
> index 33eb941fcf15..0eed825197f2 100644
> --- a/drivers/vhost/vringh.c
> +++ b/drivers/vhost/vringh.c
> @@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
> return 0;
> }
>
> +/**
> + * vringh_update_used_idx - fetch used idx from driver's used split vring
> + * @vrh: The vring.
> + *
> + * Returns -errno or 0.
> + */
> +static inline int vringh_update_used_idx(struct vringh *vrh)
> +{
> + return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
> +}
> +
> /**
> * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
> * @vrh: the vringh to initialize.
> @@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
> struct vring_avail *avail,
> struct vring_used *used)
> {
While at this, I wonder if it's better to have a dedicated parameter
for last_avail_idx?
> - return vringh_init_kern(vrh, features, num, weak_barriers,
> - desc, avail, used);
> + int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
> + avail, used);
> +
> + if (r != 0)
> + return r;
> +
> + /* Consider the ring not initialized */
> + if ((void *)desc == used)
> + return 0;
I don't understand when we can get this (actually it should be a bug
of the caller).
Thanks
> +
> + return vringh_update_used_idx(vrh);
> +
> }
> EXPORT_SYMBOL(vringh_init_iotlb);
>
> --
> 2.31.1
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 0/2] Fix expected set_vq_state behavior on vdpa_sim
[not found] <20230118164359.1523760-1-eperezma@redhat.com>
[not found] ` <20230118164359.1523760-2-eperezma@redhat.com>
@ 2023-01-27 10:53 ` Michael S. Tsirkin
[not found] ` <20230118164359.1523760-3-eperezma@redhat.com>
2 siblings, 0 replies; 8+ messages in thread
From: Michael S. Tsirkin @ 2023-01-27 10:53 UTC (permalink / raw)
To: Eugenio Pérez
Cc: Laurent Vivier, lulu, linux-kernel, Gautam Dawar, virtualization,
leiyang
On Wed, Jan 18, 2023 at 05:43:57PM +0100, Eugenio Pérez wrote:
> The use of set_vq_state is to indicate vdpa device the state of a virtqueue.
> In the case of split, it means the avail_idx. This is mandatory for use
> cases like live migration.
>
> However, vdpa_sim reset the vq state at vdpasim_queue_ready since it calls
> vringh_init_iotlb.
>
> Also, to starting from an used_idx different than 0 is needed in use cases like
> virtual machine migration. Not doing so and letting the caller set an avail
> idx different than 0 causes destination device to try to use old buffers that
> source driver already recover and are not available anymore.
>
> This series fixes both problems allowing to migrate to a vdpa_sim_net device.
Jason problems you pointed out are all consmetic do you ack
the patchset? Or expect another revision?
> Eugenio Pérez (2):
> vdpa_sim: not reset state in vdpasim_queue_ready
> vringh: fetch used_idx from vring at vringh_init_iotlb
>
> drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 ++
> drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
> 2 files changed, 25 insertions(+), 2 deletions(-)
>
> --
> 2.31.1
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] vdpa_sim: not reset state in vdpasim_queue_ready
[not found] ` <CAJaqyWejkbiwKK4O0qOObdf294YrzMeSRTVoWNzv75yQCvkJqQ@mail.gmail.com>
@ 2023-01-29 5:56 ` Jason Wang
0 siblings, 0 replies; 8+ messages in thread
From: Jason Wang @ 2023-01-29 5:56 UTC (permalink / raw)
To: Eugenio Perez Martin
Cc: Laurent Vivier, lulu, mst, linux-kernel, Gautam Dawar,
virtualization, leiyang
在 2023/1/19 17:14, Eugenio Perez Martin 写道:
> On Thu, Jan 19, 2023 at 4:16 AM Jason Wang <jasowang@redhat.com> wrote:
>> On Thu, Jan 19, 2023 at 12:44 AM Eugenio Pérez <eperezma@redhat.com> wrote:
>>> vdpasim_queue_ready calls vringh_init_iotlb, which resets split indexes.
>>> But it can be called after setting a ring base with
>>> vdpasim_set_vq_state.
>>>
>>> Fix it by stashing them. They're still resetted in vdpasim_vq_reset.
>>>
>>> This was discovered and tested live migrating the vdpa_sim_net device.
>>>
>>> Fixes: 2c53d0f64c06 ("vdpasim: vDPA device simulator")
>>> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
>>> ---
>>> drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 ++
>>> 1 file changed, 2 insertions(+)
>>>
>>> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
>>> index cb88891b44a8..8839232a3fcb 100644
>>> --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
>>> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
>>> @@ -66,6 +66,7 @@ static void vdpasim_vq_notify(struct vringh *vring)
>>> static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
>>> {
>>> struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
>>> + uint16_t last_avail_idx = vq->vring.last_avail_idx;
>>>
>>> vringh_init_iotlb(&vq->vring, vdpasim->features, vq->num, false,
>>> (struct vring_desc *)(uintptr_t)vq->desc_addr,
>>> @@ -74,6 +75,7 @@ static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
>>> (struct vring_used *)
>>> (uintptr_t)vq->device_addr);
>>>
>>> + vq->vring.last_avail_idx = last_avail_idx;
>> Does this need to be serialized with the datapath?
>>
>> E.g in set_vq_state() we do:
>>
>> spin_lock(&vdpasim->lock);
>> vrh->last_avail_idx = state->split.avail_index;
>> spin_unlock(&vdpasim->lock);
>>
> vdpasim_queue_ready is called from vdpasim_set_vq_ready, which holds
> these locks.
>
> Maybe it's too much indirection and to embed vdpasim_queue_ready in
> vdpasim_set_vq_ready would be clearer for the future?
Nope, I miss the caller.
Acked-by: Jason Wang <jasowang@redhat.com>
Thanks
>
> Thanks!
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb
[not found] ` <CAJaqyWetovvndcU=pu_kPNUNYkgao=HsENnrKCzoHdK7RBjyAQ@mail.gmail.com>
@ 2023-01-29 6:00 ` Jason Wang
[not found] ` <CAJaqyWeWaddX9KjZWs8n9eqx8u-Lk4Nj+VVH_jDh38URuZWJdA@mail.gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Jason Wang @ 2023-01-29 6:00 UTC (permalink / raw)
To: Eugenio Perez Martin
Cc: Laurent Vivier, lulu, mst, linux-kernel, Gautam Dawar,
virtualization, leiyang
On Thu, Jan 19, 2023 at 4:11 PM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Thu, Jan 19, 2023 at 4:20 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Thu, Jan 19, 2023 at 12:44 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > >
> > > Starting from an used_idx different than 0 is needed in use cases like
> > > virtual machine migration. Not doing so and letting the caller set an
> > > avail idx different than 0 causes destination device to try to use old
> > > buffers that source driver already recover and are not available
> > > anymore.
> > >
> > > While callers like vdpa_sim set avail_idx directly it does not set
> > > used_idx. Instead of let the caller do the assignment, fetch it from
> > > the guest at initialization like vhost-kernel do.
> > >
> > > To perform the same at vring_kernel_init and vring_user_init is left for
> > > the future.
> > >
> > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > ---
> > > drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
> > > 1 file changed, 23 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/vhost/vringh.c b/drivers/vhost/vringh.c
> > > index 33eb941fcf15..0eed825197f2 100644
> > > --- a/drivers/vhost/vringh.c
> > > +++ b/drivers/vhost/vringh.c
> > > @@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
> > > return 0;
> > > }
> > >
> > > +/**
> > > + * vringh_update_used_idx - fetch used idx from driver's used split vring
> > > + * @vrh: The vring.
> > > + *
> > > + * Returns -errno or 0.
> > > + */
> > > +static inline int vringh_update_used_idx(struct vringh *vrh)
> > > +{
> > > + return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
> > > +}
> > > +
> > > /**
> > > * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
> > > * @vrh: the vringh to initialize.
> > > @@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
> > > struct vring_avail *avail,
> > > struct vring_used *used)
> > > {
> >
> > While at this, I wonder if it's better to have a dedicated parameter
> > for last_avail_idx?
> >
>
> I also had that thought. To directly assign last_avail_idx is not a
> specially elegant API IMO.
>
> Maybe expose a way to fetch used_idx from device vring and pass
> used_idx as parameter too?
If I was not wrong, we can start from last_avail_idx, for used_idx it
is only needed for inflight descriptors which might require other
APIs?
(All the current vDPA user of vringh is doing in order processing)
>
> > > - return vringh_init_kern(vrh, features, num, weak_barriers,
> > > - desc, avail, used);
> > > + int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
> > > + avail, used);
> > > +
> > > + if (r != 0)
> > > + return r;
> > > +
> > > + /* Consider the ring not initialized */
> > > + if ((void *)desc == used)
> > > + return 0;
> >
> > I don't understand when we can get this (actually it should be a bug
> > of the caller).
> >
>
> You can see it in vdpasim_vq_reset.
>
> Note that to consider desc == 0 to be an uninitialized ring is a bug
> IMO. QEMU considers it that way also, but the standard does not forbid
> any ring to be at address 0. Especially if we use vIOMMU.
>
> So I think the best way to know if we can use the vringh is either
> this way, or provide an explicit "initialized" boolean attribute.
> Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to
> add new attributes.
I wonder if we can avoid this in the simulator level instead of the
vringh (anyhow it only exposes a vringh_init_xxx() helper now).
Thanks
>
> Thanks!
>
> > Thanks
> >
> > > +
> > > + return vringh_update_used_idx(vrh);
> > > +
> > > }
> > > EXPORT_SYMBOL(vringh_init_iotlb);
> > >
> > > --
> > > 2.31.1
> > >
> >
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb
[not found] ` <CAJaqyWeWaddX9KjZWs8n9eqx8u-Lk4Nj+VVH_jDh38URuZWJdA@mail.gmail.com>
@ 2023-01-31 3:16 ` Jason Wang
[not found] ` <CAJaqyWcqnHAkUv9UZF=dOukg0C5u2g+eB-G5g8p+adcGZwRB_Q@mail.gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Jason Wang @ 2023-01-31 3:16 UTC (permalink / raw)
To: Eugenio Perez Martin
Cc: Laurent Vivier, lulu, mst, linux-kernel, Gautam Dawar,
virtualization, leiyang, Longpeng (Mike,
Cloud Infrastructure Service Product Dept.)
On Tue, Jan 31, 2023 at 12:39 AM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Sun, Jan 29, 2023 at 7:01 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Thu, Jan 19, 2023 at 4:11 PM Eugenio Perez Martin
> > <eperezma@redhat.com> wrote:
> > >
> > > On Thu, Jan 19, 2023 at 4:20 AM Jason Wang <jasowang@redhat.com> wrote:
> > > >
> > > > On Thu, Jan 19, 2023 at 12:44 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > > >
> > > > > Starting from an used_idx different than 0 is needed in use cases like
> > > > > virtual machine migration. Not doing so and letting the caller set an
> > > > > avail idx different than 0 causes destination device to try to use old
> > > > > buffers that source driver already recover and are not available
> > > > > anymore.
> > > > >
> > > > > While callers like vdpa_sim set avail_idx directly it does not set
> > > > > used_idx. Instead of let the caller do the assignment, fetch it from
> > > > > the guest at initialization like vhost-kernel do.
> > > > >
> > > > > To perform the same at vring_kernel_init and vring_user_init is left for
> > > > > the future.
> > > > >
> > > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > > ---
> > > > > drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
> > > > > 1 file changed, 23 insertions(+), 2 deletions(-)
> > > > >
> > > > > diff --git a/drivers/vhost/vringh.c b/drivers/vhost/vringh.c
> > > > > index 33eb941fcf15..0eed825197f2 100644
> > > > > --- a/drivers/vhost/vringh.c
> > > > > +++ b/drivers/vhost/vringh.c
> > > > > @@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
> > > > > return 0;
> > > > > }
> > > > >
> > > > > +/**
> > > > > + * vringh_update_used_idx - fetch used idx from driver's used split vring
> > > > > + * @vrh: The vring.
> > > > > + *
> > > > > + * Returns -errno or 0.
> > > > > + */
> > > > > +static inline int vringh_update_used_idx(struct vringh *vrh)
> > > > > +{
> > > > > + return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
> > > > > +}
> > > > > +
> > > > > /**
> > > > > * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
> > > > > * @vrh: the vringh to initialize.
> > > > > @@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
> > > > > struct vring_avail *avail,
> > > > > struct vring_used *used)
> > > > > {
> > > >
> > > > While at this, I wonder if it's better to have a dedicated parameter
> > > > for last_avail_idx?
> > > >
> > >
> > > I also had that thought. To directly assign last_avail_idx is not a
> > > specially elegant API IMO.
> > >
> > > Maybe expose a way to fetch used_idx from device vring and pass
> > > used_idx as parameter too?
> >
> > If I was not wrong, we can start from last_avail_idx, for used_idx it
> > is only needed for inflight descriptors which might require other
> > APIs?
> >
> > (All the current vDPA user of vringh is doing in order processing)
> >
>
> That was actually my first attempt and it works equally well for the
> moment, but it diverges from vhost-kernel behavior for little benefit.
>
> To assign both values at set_vring_base mean that if vDPA introduces
> an (hypothetical) VHOST_VDPA_F_INFLIGHT backend feature in the future,
> the initialization process would vary a lot:
> * Without that feature, the used_idx starts with 0, and the avail one
> is 0 or whatever value the user set with vring_set_base.
> * With that feature, the device will read guest's used_idx as
> vhost-kernel? We would enable a new ioctl to set it, or expand
> set_base to include used_idx, effectively diverting from vhost-kernel?
Adding Longpeng who is looking at this.
We can leave this to the caller to decide.
Btw, a question, at which case the device used index does not equal to
the used index in the vring when the device is suspended? I think the
correct way is to flush the pending used indexes before suspending.
Otherwise we need an API to get pending used indices?
>
> To me the wisest option is to move this with vhost-kernel. Maybe we
> need to add a feature bit to know that the hypervisor can trust the
> device will do "the right thing" (VHOST_VDPA_F_FETCH_USED_AT_ENABLE?),
> but we should keep it orthogonal to inflight descriptor migration in
> my opinion.
I think we need to understand if there are any other possible use
cases for setting used idx other than inflight stuff.
>
> Having said that, I'm totally ok to do it otherwise (or to expand the
> patch message if needed).
I tend to do that in another series (not mix with the fixes).
>
> > >
> > > > > - return vringh_init_kern(vrh, features, num, weak_barriers,
> > > > > - desc, avail, used);
> > > > > + int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
> > > > > + avail, used);
> > > > > +
> > > > > + if (r != 0)
> > > > > + return r;
> > > > > +
> > > > > + /* Consider the ring not initialized */
> > > > > + if ((void *)desc == used)
> > > > > + return 0;
> > > >
> > > > I don't understand when we can get this (actually it should be a bug
> > > > of the caller).
> > > >
> > >
> > > You can see it in vdpasim_vq_reset.
> > >
> > > Note that to consider desc == 0 to be an uninitialized ring is a bug
> > > IMO. QEMU considers it that way also, but the standard does not forbid
> > > any ring to be at address 0. Especially if we use vIOMMU.
> > >
> > > So I think the best way to know if we can use the vringh is either
> > > this way, or provide an explicit "initialized" boolean attribute.
> > > Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to
> > > add new attributes.
> >
> > I wonder if we can avoid this in the simulator level instead of the
> > vringh (anyhow it only exposes a vringh_init_xxx() helper now).
> >
>
> In my opinion that is a mistake if other drivers will use it to
> implement the emulated control virtqueue. And it requires more
> changes. But it is doable for sure.
The problem is, there's no reset API in vringh, that's why you need to
do if ((void *)desc == used) which depends on behaviour of the vringh
user.
So I think we should either:
1) move that check in vdpa_sim (since it's not guaranteed that all the
vringh users will make desc equal to used during reset)
or
2) introduce a vringh_reset_xxx()
1) seems a good step for -stable.
Thanks
>
> Thanks!
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb
[not found] ` <20230118164359.1523760-3-eperezma@redhat.com>
2023-01-19 3:20 ` [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb Jason Wang
@ 2023-02-01 16:11 ` Michael S. Tsirkin
1 sibling, 0 replies; 8+ messages in thread
From: Michael S. Tsirkin @ 2023-02-01 16:11 UTC (permalink / raw)
To: Eugenio Pérez
Cc: Laurent Vivier, lulu, linux-kernel, Gautam Dawar, virtualization,
leiyang
On Wed, Jan 18, 2023 at 05:43:59PM +0100, Eugenio Pérez wrote:
> Starting from an used_idx different than 0 is needed in use cases like
> virtual machine migration. Not doing so and letting the caller set an
> avail idx different than 0 causes destination device to try to use old
> buffers that source driver already recover and are not available
> anymore.
>
> While callers like vdpa_sim set avail_idx directly it does not set
> used_idx. Instead of let the caller do the assignment, fetch it from
> the guest at initialization like vhost-kernel do.
>
> To perform the same at vring_kernel_init and vring_user_init is left for
> the future.
>
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
So I applied 1/2 and dropped 2/2 for now, right?
> ---
> drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
> 1 file changed, 23 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/vhost/vringh.c b/drivers/vhost/vringh.c
> index 33eb941fcf15..0eed825197f2 100644
> --- a/drivers/vhost/vringh.c
> +++ b/drivers/vhost/vringh.c
> @@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
> return 0;
> }
>
> +/**
> + * vringh_update_used_idx - fetch used idx from driver's used split vring
> + * @vrh: The vring.
> + *
> + * Returns -errno or 0.
> + */
> +static inline int vringh_update_used_idx(struct vringh *vrh)
> +{
> + return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
> +}
> +
> /**
> * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
> * @vrh: the vringh to initialize.
> @@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
> struct vring_avail *avail,
> struct vring_used *used)
> {
> - return vringh_init_kern(vrh, features, num, weak_barriers,
> - desc, avail, used);
> + int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
> + avail, used);
> +
> + if (r != 0)
> + return r;
> +
> + /* Consider the ring not initialized */
> + if ((void *)desc == used)
> + return 0;
> +
> + return vringh_update_used_idx(vrh);
> +
> }
> EXPORT_SYMBOL(vringh_init_iotlb);
>
> --
> 2.31.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb
[not found] ` <CAJaqyWcqnHAkUv9UZF=dOukg0C5u2g+eB-G5g8p+adcGZwRB_Q@mail.gmail.com>
@ 2023-02-13 12:03 ` Michael S. Tsirkin
0 siblings, 0 replies; 8+ messages in thread
From: Michael S. Tsirkin @ 2023-02-13 12:03 UTC (permalink / raw)
To: Eugenio Perez Martin
Cc: Laurent Vivier, lulu, linux-kernel, Gautam Dawar, virtualization,
leiyang, Longpeng (Mike,
Cloud Infrastructure Service Product Dept.)
On Tue, Jan 31, 2023 at 08:58:37AM +0100, Eugenio Perez Martin wrote:
> On Tue, Jan 31, 2023 at 4:16 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Tue, Jan 31, 2023 at 12:39 AM Eugenio Perez Martin
> > <eperezma@redhat.com> wrote:
> > >
> > > On Sun, Jan 29, 2023 at 7:01 AM Jason Wang <jasowang@redhat.com> wrote:
> > > >
> > > > On Thu, Jan 19, 2023 at 4:11 PM Eugenio Perez Martin
> > > > <eperezma@redhat.com> wrote:
> > > > >
> > > > > On Thu, Jan 19, 2023 at 4:20 AM Jason Wang <jasowang@redhat.com> wrote:
> > > > > >
> > > > > > On Thu, Jan 19, 2023 at 12:44 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > > > > >
> > > > > > > Starting from an used_idx different than 0 is needed in use cases like
> > > > > > > virtual machine migration. Not doing so and letting the caller set an
> > > > > > > avail idx different than 0 causes destination device to try to use old
> > > > > > > buffers that source driver already recover and are not available
> > > > > > > anymore.
> > > > > > >
> > > > > > > While callers like vdpa_sim set avail_idx directly it does not set
> > > > > > > used_idx. Instead of let the caller do the assignment, fetch it from
> > > > > > > the guest at initialization like vhost-kernel do.
> > > > > > >
> > > > > > > To perform the same at vring_kernel_init and vring_user_init is left for
> > > > > > > the future.
> > > > > > >
> > > > > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > > > > ---
> > > > > > > drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
> > > > > > > 1 file changed, 23 insertions(+), 2 deletions(-)
> > > > > > >
> > > > > > > diff --git a/drivers/vhost/vringh.c b/drivers/vhost/vringh.c
> > > > > > > index 33eb941fcf15..0eed825197f2 100644
> > > > > > > --- a/drivers/vhost/vringh.c
> > > > > > > +++ b/drivers/vhost/vringh.c
> > > > > > > @@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
> > > > > > > return 0;
> > > > > > > }
> > > > > > >
> > > > > > > +/**
> > > > > > > + * vringh_update_used_idx - fetch used idx from driver's used split vring
> > > > > > > + * @vrh: The vring.
> > > > > > > + *
> > > > > > > + * Returns -errno or 0.
> > > > > > > + */
> > > > > > > +static inline int vringh_update_used_idx(struct vringh *vrh)
> > > > > > > +{
> > > > > > > + return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
> > > > > > > +}
> > > > > > > +
> > > > > > > /**
> > > > > > > * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
> > > > > > > * @vrh: the vringh to initialize.
> > > > > > > @@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
> > > > > > > struct vring_avail *avail,
> > > > > > > struct vring_used *used)
> > > > > > > {
> > > > > >
> > > > > > While at this, I wonder if it's better to have a dedicated parameter
> > > > > > for last_avail_idx?
> > > > > >
> > > > >
> > > > > I also had that thought. To directly assign last_avail_idx is not a
> > > > > specially elegant API IMO.
> > > > >
> > > > > Maybe expose a way to fetch used_idx from device vring and pass
> > > > > used_idx as parameter too?
> > > >
> > > > If I was not wrong, we can start from last_avail_idx, for used_idx it
> > > > is only needed for inflight descriptors which might require other
> > > > APIs?
> > > >
> > > > (All the current vDPA user of vringh is doing in order processing)
> > > >
> > >
> > > That was actually my first attempt and it works equally well for the
> > > moment, but it diverges from vhost-kernel behavior for little benefit.
> > >
> > > To assign both values at set_vring_base mean that if vDPA introduces
> > > an (hypothetical) VHOST_VDPA_F_INFLIGHT backend feature in the future,
> > > the initialization process would vary a lot:
> > > * Without that feature, the used_idx starts with 0, and the avail one
> > > is 0 or whatever value the user set with vring_set_base.
> > > * With that feature, the device will read guest's used_idx as
> > > vhost-kernel? We would enable a new ioctl to set it, or expand
> > > set_base to include used_idx, effectively diverting from vhost-kernel?
> >
> > Adding Longpeng who is looking at this.
> >
>
> Sorry, I'll CC longpeng2@huawei.com in future series like this one.
>
> > We can leave this to the caller to decide.
> >
> > Btw, a question, at which case the device used index does not equal to
> > the used index in the vring when the device is suspended? I think the
> > correct way is to flush the pending used indexes before suspending.
> > Otherwise we need an API to get pending used indices?
> >
> > >
> > > To me the wisest option is to move this with vhost-kernel. Maybe we
> > > need to add a feature bit to know that the hypervisor can trust the
> > > device will do "the right thing" (VHOST_VDPA_F_FETCH_USED_AT_ENABLE?),
> > > but we should keep it orthogonal to inflight descriptor migration in
> > > my opinion.
> >
> > I think we need to understand if there are any other possible use
> > cases for setting used idx other than inflight stuff.
> >
>
> Answering this and the previous comment, I cannot think in any case
> outside of inflight migration. I'm just trying to avoid different
> behavior between vhost-kernel and vhost-vdpa, and to make features as
> orthogonal as possible.
>
> > >
> > > Having said that, I'm totally ok to do it otherwise (or to expand the
> > > patch message if needed).
> >
> > I tend to do that in another series (not mix with the fixes).
> >
> > >
> > > > >
> > > > > > > - return vringh_init_kern(vrh, features, num, weak_barriers,
> > > > > > > - desc, avail, used);
> > > > > > > + int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
> > > > > > > + avail, used);
> > > > > > > +
> > > > > > > + if (r != 0)
> > > > > > > + return r;
> > > > > > > +
> > > > > > > + /* Consider the ring not initialized */
> > > > > > > + if ((void *)desc == used)
> > > > > > > + return 0;
> > > > > >
> > > > > > I don't understand when we can get this (actually it should be a bug
> > > > > > of the caller).
> > > > > >
> > > > >
> > > > > You can see it in vdpasim_vq_reset.
> > > > >
> > > > > Note that to consider desc == 0 to be an uninitialized ring is a bug
> > > > > IMO. QEMU considers it that way also, but the standard does not forbid
> > > > > any ring to be at address 0. Especially if we use vIOMMU.
> > > > >
> > > > > So I think the best way to know if we can use the vringh is either
> > > > > this way, or provide an explicit "initialized" boolean attribute.
> > > > > Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to
> > > > > add new attributes.
> > > >
> > > > I wonder if we can avoid this in the simulator level instead of the
> > > > vringh (anyhow it only exposes a vringh_init_xxx() helper now).
> > > >
> > >
> > > In my opinion that is a mistake if other drivers will use it to
> > > implement the emulated control virtqueue. And it requires more
> > > changes. But it is doable for sure.
> >
> > The problem is, there's no reset API in vringh, that's why you need to
> > do if ((void *)desc == used) which depends on behaviour of the vringh
> > user.
> >
>
> That's a very good point indeed.
>
> > So I think we should either:
> >
> > 1) move that check in vdpa_sim (since it's not guaranteed that all the
> > vringh users will make desc equal to used during reset)
> >
> > or
> >
> > 2) introduce a vringh_reset_xxx()
> >
> > 1) seems a good step for -stable.
> >
>
> We can go to 1 for sure. So let's set last_used_idx at
> vdpasim_set_vq_state, same value as last_avail_idx, and stash it at
> vdpasim_queue_ready.
>
> Do I need to resend the previous patch in this series?
>
> Do we need to offer a new feature flag indicating we will set used_idx
> with avail_idx?
>
> Thanks!
Jason did you forget to answer or did I miss it?
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2023-02-13 12:03 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20230118164359.1523760-1-eperezma@redhat.com>
[not found] ` <20230118164359.1523760-2-eperezma@redhat.com>
2023-01-19 3:16 ` [PATCH 1/2] vdpa_sim: not reset state in vdpasim_queue_ready Jason Wang
[not found] ` <CAJaqyWejkbiwKK4O0qOObdf294YrzMeSRTVoWNzv75yQCvkJqQ@mail.gmail.com>
2023-01-29 5:56 ` Jason Wang
2023-01-27 10:53 ` [PATCH 0/2] Fix expected set_vq_state behavior on vdpa_sim Michael S. Tsirkin
[not found] ` <20230118164359.1523760-3-eperezma@redhat.com>
2023-01-19 3:20 ` [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb Jason Wang
[not found] ` <CAJaqyWetovvndcU=pu_kPNUNYkgao=HsENnrKCzoHdK7RBjyAQ@mail.gmail.com>
2023-01-29 6:00 ` Jason Wang
[not found] ` <CAJaqyWeWaddX9KjZWs8n9eqx8u-Lk4Nj+VVH_jDh38URuZWJdA@mail.gmail.com>
2023-01-31 3:16 ` Jason Wang
[not found] ` <CAJaqyWcqnHAkUv9UZF=dOukg0C5u2g+eB-G5g8p+adcGZwRB_Q@mail.gmail.com>
2023-02-13 12:03 ` Michael S. Tsirkin
2023-02-01 16:11 ` Michael S. Tsirkin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).