* [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress @ 2022-03-03 15:19 Lee Jones 2022-03-03 19:14 ` Leon Romanovsky 0 siblings, 1 reply; 8+ messages in thread From: Lee Jones @ 2022-03-03 15:19 UTC (permalink / raw) To: lee.jones, mst, jasowang; +Cc: linux-kernel, kvm, virtualization, netdev All workers/users should be halted before any clean-up should take place. Suggested-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Lee Jones <lee.jones@linaro.org> --- drivers/vhost/vhost.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index bbaff6a5e21b8..d935d2506963f 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,9 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i; for (i = 0; i < dev->nvqs; ++i) { + /* Ideally all workers should be stopped prior to clean-up */ + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); + mutex_lock(&dev->vqs[i]->mutex); if (dev->vqs[i]->error_ctx) eventfd_ctx_put(dev->vqs[i]->error_ctx); -- 2.35.1.574.g5d30c73bfb-goog ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress 2022-03-03 15:19 [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress Lee Jones @ 2022-03-03 19:14 ` Leon Romanovsky 2022-03-03 19:38 ` Lee Jones 2022-03-03 21:01 ` Michael S. Tsirkin 0 siblings, 2 replies; 8+ messages in thread From: Leon Romanovsky @ 2022-03-03 19:14 UTC (permalink / raw) To: Lee Jones; +Cc: mst, jasowang, linux-kernel, kvm, virtualization, netdev On Thu, Mar 03, 2022 at 03:19:29PM +0000, Lee Jones wrote: > All workers/users should be halted before any clean-up should take place. > > Suggested-by: Michael S. Tsirkin <mst@redhat.com> > Signed-off-by: Lee Jones <lee.jones@linaro.org> > --- > drivers/vhost/vhost.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > index bbaff6a5e21b8..d935d2506963f 100644 > --- a/drivers/vhost/vhost.c > +++ b/drivers/vhost/vhost.c > @@ -693,6 +693,9 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > int i; > > for (i = 0; i < dev->nvqs; ++i) { > + /* Ideally all workers should be stopped prior to clean-up */ > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > + > mutex_lock(&dev->vqs[i]->mutex); I know nothing about vhost, but this construction and patch looks strange to me. If all workers were stopped, you won't need mutex_lock(). The mutex_lock here suggests to me that workers can still run here. Thanks > if (dev->vqs[i]->error_ctx) > eventfd_ctx_put(dev->vqs[i]->error_ctx); > -- > 2.35.1.574.g5d30c73bfb-goog > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress 2022-03-03 19:14 ` Leon Romanovsky @ 2022-03-03 19:38 ` Lee Jones 2022-03-03 21:01 ` Michael S. Tsirkin 1 sibling, 0 replies; 8+ messages in thread From: Lee Jones @ 2022-03-03 19:38 UTC (permalink / raw) To: Leon Romanovsky; +Cc: mst, jasowang, linux-kernel, kvm, virtualization, netdev On Thu, 03 Mar 2022, Leon Romanovsky wrote: > On Thu, Mar 03, 2022 at 03:19:29PM +0000, Lee Jones wrote: > > All workers/users should be halted before any clean-up should take place. > > > > Suggested-by: Michael S. Tsirkin <mst@redhat.com> > > Signed-off-by: Lee Jones <lee.jones@linaro.org> > > --- > > drivers/vhost/vhost.c | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > index bbaff6a5e21b8..d935d2506963f 100644 > > --- a/drivers/vhost/vhost.c > > +++ b/drivers/vhost/vhost.c > > @@ -693,6 +693,9 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > int i; > > > > for (i = 0; i < dev->nvqs; ++i) { > > + /* Ideally all workers should be stopped prior to clean-up */ > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > + > > mutex_lock(&dev->vqs[i]->mutex); > > I know nothing about vhost, but this construction and patch looks > strange to me. > > If all workers were stopped, you won't need mutex_lock(). The mutex_lock > here suggests to me that workers can still run here. The suggestion for this patch came from the maintainer. Please see the conversation here: https://lore.kernel.org/all/20220302082021-mutt-send-email-mst@kernel.org/ -- Lee Jones [李琼斯] Principal Technical Lead - Developer Services Linaro.org │ Open source software for Arm SoCs Follow Linaro: Facebook | Twitter | Blog ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress 2022-03-03 19:14 ` Leon Romanovsky 2022-03-03 19:38 ` Lee Jones @ 2022-03-03 21:01 ` Michael S. Tsirkin 2022-03-04 7:08 ` Leon Romanovsky 2022-03-04 7:50 ` Stefano Garzarella 1 sibling, 2 replies; 8+ messages in thread From: Michael S. Tsirkin @ 2022-03-03 21:01 UTC (permalink / raw) To: Leon Romanovsky Cc: Lee Jones, jasowang, linux-kernel, kvm, virtualization, netdev On Thu, Mar 03, 2022 at 09:14:36PM +0200, Leon Romanovsky wrote: > On Thu, Mar 03, 2022 at 03:19:29PM +0000, Lee Jones wrote: > > All workers/users should be halted before any clean-up should take place. > > > > Suggested-by: Michael S. Tsirkin <mst@redhat.com> > > Signed-off-by: Lee Jones <lee.jones@linaro.org> > > --- > > drivers/vhost/vhost.c | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > index bbaff6a5e21b8..d935d2506963f 100644 > > --- a/drivers/vhost/vhost.c > > +++ b/drivers/vhost/vhost.c > > @@ -693,6 +693,9 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > int i; > > > > for (i = 0; i < dev->nvqs; ++i) { > > + /* Ideally all workers should be stopped prior to clean-up */ > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > + > > mutex_lock(&dev->vqs[i]->mutex); > > I know nothing about vhost, but this construction and patch looks > strange to me. > > If all workers were stopped, you won't need mutex_lock(). The mutex_lock > here suggests to me that workers can still run here. > > Thanks "Ideally" here is misleading, we need a bigger detailed comment along the lines of: /* * By design, no workers can run here. But if there's a bug and the * driver did not flush all work properly then they might, and we * encountered such bugs in the past. With no proper flush guest won't * work correctly but avoiding host memory corruption in this case * sounds like a good idea. */ > > if (dev->vqs[i]->error_ctx) > > eventfd_ctx_put(dev->vqs[i]->error_ctx); > > -- > > 2.35.1.574.g5d30c73bfb-goog > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress 2022-03-03 21:01 ` Michael S. Tsirkin @ 2022-03-04 7:08 ` Leon Romanovsky 2022-03-04 8:11 ` Lee Jones 2022-03-04 7:50 ` Stefano Garzarella 1 sibling, 1 reply; 8+ messages in thread From: Leon Romanovsky @ 2022-03-04 7:08 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Lee Jones, jasowang, linux-kernel, kvm, virtualization, netdev On Thu, Mar 03, 2022 at 04:01:06PM -0500, Michael S. Tsirkin wrote: > On Thu, Mar 03, 2022 at 09:14:36PM +0200, Leon Romanovsky wrote: > > On Thu, Mar 03, 2022 at 03:19:29PM +0000, Lee Jones wrote: > > > All workers/users should be halted before any clean-up should take place. > > > > > > Suggested-by: Michael S. Tsirkin <mst@redhat.com> > > > Signed-off-by: Lee Jones <lee.jones@linaro.org> > > > --- > > > drivers/vhost/vhost.c | 3 +++ > > > 1 file changed, 3 insertions(+) > > > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > > index bbaff6a5e21b8..d935d2506963f 100644 > > > --- a/drivers/vhost/vhost.c > > > +++ b/drivers/vhost/vhost.c > > > @@ -693,6 +693,9 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > int i; > > > > > > for (i = 0; i < dev->nvqs; ++i) { > > > + /* Ideally all workers should be stopped prior to clean-up */ > > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > > + > > > mutex_lock(&dev->vqs[i]->mutex); > > > > I know nothing about vhost, but this construction and patch looks > > strange to me. > > > > If all workers were stopped, you won't need mutex_lock(). The mutex_lock > > here suggests to me that workers can still run here. > > > > Thanks > > > "Ideally" here is misleading, we need a bigger detailed comment > along the lines of: > > /* > * By design, no workers can run here. But if there's a bug and the > * driver did not flush all work properly then they might, and we > * encountered such bugs in the past. With no proper flush guest won't > * work correctly but avoiding host memory corruption in this case > * sounds like a good idea. > */ This description looks better, but the check is inherently racy. Why don't you add a comment and mutex_lock()? The WARN_ON here is more distraction than actual help. Thanks > > > > if (dev->vqs[i]->error_ctx) > > > eventfd_ctx_put(dev->vqs[i]->error_ctx); > > > -- > > > 2.35.1.574.g5d30c73bfb-goog > > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress 2022-03-04 7:08 ` Leon Romanovsky @ 2022-03-04 8:11 ` Lee Jones 0 siblings, 0 replies; 8+ messages in thread From: Lee Jones @ 2022-03-04 8:11 UTC (permalink / raw) To: Leon Romanovsky Cc: Michael S. Tsirkin, jasowang, linux-kernel, kvm, virtualization, netdev On Fri, 04 Mar 2022, Leon Romanovsky wrote: > On Thu, Mar 03, 2022 at 04:01:06PM -0500, Michael S. Tsirkin wrote: > > On Thu, Mar 03, 2022 at 09:14:36PM +0200, Leon Romanovsky wrote: > > > On Thu, Mar 03, 2022 at 03:19:29PM +0000, Lee Jones wrote: > > > > All workers/users should be halted before any clean-up should take place. > > > > > > > > Suggested-by: Michael S. Tsirkin <mst@redhat.com> > > > > Signed-off-by: Lee Jones <lee.jones@linaro.org> > > > > --- > > > > drivers/vhost/vhost.c | 3 +++ > > > > 1 file changed, 3 insertions(+) > > > > > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > > > index bbaff6a5e21b8..d935d2506963f 100644 > > > > --- a/drivers/vhost/vhost.c > > > > +++ b/drivers/vhost/vhost.c > > > > @@ -693,6 +693,9 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > > int i; > > > > > > > > for (i = 0; i < dev->nvqs; ++i) { > > > > + /* Ideally all workers should be stopped prior to clean-up */ > > > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > > > + > > > > mutex_lock(&dev->vqs[i]->mutex); HERE ---^ > > > I know nothing about vhost, but this construction and patch looks > > > strange to me. > > > > > > If all workers were stopped, you won't need mutex_lock(). The mutex_lock > > > here suggests to me that workers can still run here. > > > > > > Thanks > > > > > > "Ideally" here is misleading, we need a bigger detailed comment > > along the lines of: > > > > /* > > * By design, no workers can run here. But if there's a bug and the > > * driver did not flush all work properly then they might, and we > > * encountered such bugs in the past. With no proper flush guest won't > > * work correctly but avoiding host memory corruption in this case > > * sounds like a good idea. > > */ > > This description looks better, but the check is inherently racy. > Why don't you add a comment and mutex_lock()? We do, look up. ^ > The WARN_ON here is more distraction than actual help. The WARN() is just an indication that something else has gone wrong. Stefano patched one problem in: vhost: Protect the virtqueue from being cleared whilst still in use ... but others may crop up and the WARN() is how we'll be informed. -- Lee Jones [李琼斯] Principal Technical Lead - Developer Services Linaro.org │ Open source software for Arm SoCs Follow Linaro: Facebook | Twitter | Blog ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress 2022-03-03 21:01 ` Michael S. Tsirkin 2022-03-04 7:08 ` Leon Romanovsky @ 2022-03-04 7:50 ` Stefano Garzarella 2022-03-04 8:12 ` Lee Jones 1 sibling, 1 reply; 8+ messages in thread From: Stefano Garzarella @ 2022-03-04 7:50 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Leon Romanovsky, Lee Jones, jasowang, linux-kernel, kvm, virtualization, netdev On Thu, Mar 03, 2022 at 04:01:06PM -0500, Michael S. Tsirkin wrote: >On Thu, Mar 03, 2022 at 09:14:36PM +0200, Leon Romanovsky wrote: >> On Thu, Mar 03, 2022 at 03:19:29PM +0000, Lee Jones wrote: >> > All workers/users should be halted before any clean-up should take place. >> > >> > Suggested-by: Michael S. Tsirkin <mst@redhat.com> >> > Signed-off-by: Lee Jones <lee.jones@linaro.org> >> > --- >> > drivers/vhost/vhost.c | 3 +++ >> > 1 file changed, 3 insertions(+) >> > >> > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c >> > index bbaff6a5e21b8..d935d2506963f 100644 >> > --- a/drivers/vhost/vhost.c >> > +++ b/drivers/vhost/vhost.c >> > @@ -693,6 +693,9 @@ void vhost_dev_cleanup(struct vhost_dev *dev) >> > int i; >> > >> > for (i = 0; i < dev->nvqs; ++i) { >> > + /* Ideally all workers should be stopped prior to clean-up */ >> > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); >> > + >> > mutex_lock(&dev->vqs[i]->mutex); >> >> I know nothing about vhost, but this construction and patch looks >> strange to me. >> >> If all workers were stopped, you won't need mutex_lock(). The mutex_lock >> here suggests to me that workers can still run here. >> >> Thanks > > >"Ideally" here is misleading, we need a bigger detailed comment >along the lines of: > >/* > * By design, no workers can run here. But if there's a bug and the > * driver did not flush all work properly then they might, and we > * encountered such bugs in the past. With no proper flush guest won't > * work correctly but avoiding host memory corruption in this case > * sounds like a good idea. > */ Can we use vhost_vq_get_backend() to check this situation? IIUC all the vhost devices clear the backend to stop the workers. This is not racy (if we do after the mutex_lock) and should cover all cases. Thanks, Stefano ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress 2022-03-04 7:50 ` Stefano Garzarella @ 2022-03-04 8:12 ` Lee Jones 0 siblings, 0 replies; 8+ messages in thread From: Lee Jones @ 2022-03-04 8:12 UTC (permalink / raw) To: Stefano Garzarella Cc: Michael S. Tsirkin, Leon Romanovsky, jasowang, linux-kernel, kvm, virtualization, netdev On Fri, 04 Mar 2022, Stefano Garzarella wrote: > On Thu, Mar 03, 2022 at 04:01:06PM -0500, Michael S. Tsirkin wrote: > > On Thu, Mar 03, 2022 at 09:14:36PM +0200, Leon Romanovsky wrote: > > > On Thu, Mar 03, 2022 at 03:19:29PM +0000, Lee Jones wrote: > > > > All workers/users should be halted before any clean-up should take place. > > > > > > > > Suggested-by: Michael S. Tsirkin <mst@redhat.com> > > > > Signed-off-by: Lee Jones <lee.jones@linaro.org> > > > > --- > > > > drivers/vhost/vhost.c | 3 +++ > > > > 1 file changed, 3 insertions(+) > > > > > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > > > index bbaff6a5e21b8..d935d2506963f 100644 > > > > --- a/drivers/vhost/vhost.c > > > > +++ b/drivers/vhost/vhost.c > > > > @@ -693,6 +693,9 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > > int i; > > > > > > > > for (i = 0; i < dev->nvqs; ++i) { > > > > + /* Ideally all workers should be stopped prior to clean-up */ > > > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > > > + > > > > mutex_lock(&dev->vqs[i]->mutex); > > > > > > I know nothing about vhost, but this construction and patch looks > > > strange to me. > > > > > > If all workers were stopped, you won't need mutex_lock(). The mutex_lock > > > here suggests to me that workers can still run here. > > > > > > Thanks > > > > > > "Ideally" here is misleading, we need a bigger detailed comment > > along the lines of: > > > > /* > > * By design, no workers can run here. But if there's a bug and the > > * driver did not flush all work properly then they might, and we > > * encountered such bugs in the past. With no proper flush guest won't > > * work correctly but avoiding host memory corruption in this case > > * sounds like a good idea. > > */ > > Can we use vhost_vq_get_backend() to check this situation? > > IIUC all the vhost devices clear the backend to stop the workers. > This is not racy (if we do after the mutex_lock) and should cover all cases. I can look into this too if you like. -- Lee Jones [李琼斯] Principal Technical Lead - Developer Services Linaro.org │ Open source software for Arm SoCs Follow Linaro: Facebook | Twitter | Blog ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-03-04 8:12 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-03-03 15:19 [PATCH 1/1] vhost: Provide a kernel warning if mutex is held whilst clean-up in progress Lee Jones 2022-03-03 19:14 ` Leon Romanovsky 2022-03-03 19:38 ` Lee Jones 2022-03-03 21:01 ` Michael S. Tsirkin 2022-03-04 7:08 ` Leon Romanovsky 2022-03-04 8:11 ` Lee Jones 2022-03-04 7:50 ` Stefano Garzarella 2022-03-04 8:12 ` Lee Jones
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).