[PATCH] virtio_console: break out of buf poll on remove

[PATCH] virtio_console: break out of buf poll on remove

[Michael S. Tsirkin]

A common pattern for device reset is currently:
vdev->config->reset(vdev);
.. cleanup ..

reset prevents new interrupts from arriving and waits for interrupt
handlers to finish.

However if - as is common - the handler queues a work request which is
flushed during the cleanup stage, we have code adding buffers / trying
to get buffers while device is reset. Not good.

This was reproduced by running
	modprobe virtio_console
	modprobe -r virtio_console
in a loop.

Fixing this comprehensively needs some thought, and new APIs.
Let's at least handle the specific case of virtio_console
removal that was reported in the field.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1786239
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 drivers/char/virtio_console.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 7eaf303a7a86..c852ce0b4d56 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1956,6 +1956,12 @@ static void virtcons_remove(struct virtio_device *vdev)
 	list_del(&portdev->list);
 	spin_unlock_irq(&pdrvdata_lock);
 
+	/* Device is going away, exit any polling for buffers */
+	virtio_break_device(vdev);
+	if (use_multiport(portdev))
+		flush_work(&portdev->control_work);
+	else
+		flush_work(&portdev->config_work);
 	/* Disable interrupts for vqs */
 	vdev->config->reset(vdev);
 	/* Finish up work that's lined up */
-- 
MST

[PATCH] virtio_console: break out of buf poll on remove

[Michael S. Tsirkin]

A common pattern for device reset is currently:
vdev->config->reset(vdev);
.. cleanup ..

reset prevents new interrupts from arriving and waits for interrupt
handlers to finish.

However if - as is common - the handler queues a work request which is
flushed during the cleanup stage, we have code adding buffers / trying
to get buffers while device is reset. Not good.

This was reproduced by running
	modprobe virtio_console
	modprobe -r virtio_console
in a loop.

Fixing this comprehensively needs some thought, and new APIs.
Let's at least handle the specific case of virtio_console
removal that was reported in the field.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1786239
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 drivers/char/virtio_console.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 7eaf303a7a86..c852ce0b4d56 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1956,6 +1956,12 @@ static void virtcons_remove(struct virtio_device *vdev)
 	list_del(&portdev->list);
 	spin_unlock_irq(&pdrvdata_lock);
 
+	/* Device is going away, exit any polling for buffers */
+	virtio_break_device(vdev);
+	if (use_multiport(portdev))
+		flush_work(&portdev->control_work);
+	else
+		flush_work(&portdev->config_work);
 	/* Disable interrupts for vqs */
 	vdev->config->reset(vdev);
 	/* Finish up work that's lined up */
-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] virtio_console: break out of buf poll on remove

[Greg Kroah-Hartman]

On Tue, Oct 05, 2021 at 03:04:07AM -0400, Michael S. Tsirkin wrote:
> A common pattern for device reset is currently:
> vdev->config->reset(vdev);
> .. cleanup ..
> 
> reset prevents new interrupts from arriving and waits for interrupt
> handlers to finish.
> 
> However if - as is common - the handler queues a work request which is
> flushed during the cleanup stage, we have code adding buffers / trying
> to get buffers while device is reset. Not good.
> 
> This was reproduced by running
> 	modprobe virtio_console
> 	modprobe -r virtio_console
> in a loop.

That's a pathological case that is not "in the field" except by people
who want to abuse the system as root.  And they can do much worse things
than that.

> Fixing this comprehensively needs some thought, and new APIs.
> Let's at least handle the specific case of virtio_console
> removal that was reported in the field.

Let's fix this correctly, don't just hack it up now.

> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1786239
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  drivers/char/virtio_console.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> index 7eaf303a7a86..c852ce0b4d56 100644
> --- a/drivers/char/virtio_console.c
> +++ b/drivers/char/virtio_console.c
> @@ -1956,6 +1956,12 @@ static void virtcons_remove(struct virtio_device *vdev)
>  	list_del(&portdev->list);
>  	spin_unlock_irq(&pdrvdata_lock);
>  
> +	/* Device is going away, exit any polling for buffers */
> +	virtio_break_device(vdev);
> +	if (use_multiport(portdev))
> +		flush_work(&portdev->control_work);
> +	else
> +		flush_work(&portdev->config_work);
>  	/* Disable interrupts for vqs */

newline before comment?

thanks,

greg k-h

Re: [PATCH] virtio_console: break out of buf poll on remove

[Greg Kroah-Hartman]

On Tue, Oct 05, 2021 at 03:04:07AM -0400, Michael S. Tsirkin wrote:
> A common pattern for device reset is currently:
> vdev->config->reset(vdev);
> .. cleanup ..
> 
> reset prevents new interrupts from arriving and waits for interrupt
> handlers to finish.
> 
> However if - as is common - the handler queues a work request which is
> flushed during the cleanup stage, we have code adding buffers / trying
> to get buffers while device is reset. Not good.
> 
> This was reproduced by running
> 	modprobe virtio_console
> 	modprobe -r virtio_console
> in a loop.

That's a pathological case that is not "in the field" except by people
who want to abuse the system as root.  And they can do much worse things
than that.

> Fixing this comprehensively needs some thought, and new APIs.
> Let's at least handle the specific case of virtio_console
> removal that was reported in the field.

Let's fix this correctly, don't just hack it up now.

> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1786239
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  drivers/char/virtio_console.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> index 7eaf303a7a86..c852ce0b4d56 100644
> --- a/drivers/char/virtio_console.c
> +++ b/drivers/char/virtio_console.c
> @@ -1956,6 +1956,12 @@ static void virtcons_remove(struct virtio_device *vdev)
>  	list_del(&portdev->list);
>  	spin_unlock_irq(&pdrvdata_lock);
>  
> +	/* Device is going away, exit any polling for buffers */
> +	virtio_break_device(vdev);
> +	if (use_multiport(portdev))
> +		flush_work(&portdev->control_work);
> +	else
> +		flush_work(&portdev->config_work);
>  	/* Disable interrupts for vqs */

newline before comment?

thanks,

greg k-h
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] virtio_console: break out of buf poll on remove

[Michael S. Tsirkin]

On Tue, Oct 05, 2021 at 03:33:42PM +0200, Greg Kroah-Hartman wrote:
> On Tue, Oct 05, 2021 at 03:04:07AM -0400, Michael S. Tsirkin wrote:
> > A common pattern for device reset is currently:
> > vdev->config->reset(vdev);
> > .. cleanup ..
> > 
> > reset prevents new interrupts from arriving and waits for interrupt
> > handlers to finish.
> > 
> > However if - as is common - the handler queues a work request which is
> > flushed during the cleanup stage, we have code adding buffers / trying
> > to get buffers while device is reset. Not good.
> > 
> > This was reproduced by running
> > 	modprobe virtio_console
> > 	modprobe -r virtio_console
> > in a loop.
> 
> That's a pathological case that is not "in the field" except by people
> who want to abuse the system as root.  And they can do much worse things
> than that.
> 
> > Fixing this comprehensively needs some thought, and new APIs.
> > Let's at least handle the specific case of virtio_console
> > removal that was reported in the field.
> 
> Let's fix this correctly, don't just hack it up now.

Well I poked at it some more, and things are not as bad
as I thought. It's mostly just console and possibly virtio-mem.
Well and virtio-bt has a completely borken cleanup that
does not even bother to reset the device, but that's
a separate issue, discussing it with the maintainer.

So I wrote some patches to document the requirements better, added a
wrapper for reset and generally cleaned the API up a bit, and added a
patch for mem, but generally I no longer think we need a major API
change.


> > Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1786239
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  drivers/char/virtio_console.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> > index 7eaf303a7a86..c852ce0b4d56 100644
> > --- a/drivers/char/virtio_console.c
> > +++ b/drivers/char/virtio_console.c
> > @@ -1956,6 +1956,12 @@ static void virtcons_remove(struct virtio_device *vdev)
> >  	list_del(&portdev->list);
> >  	spin_unlock_irq(&pdrvdata_lock);
> >  
> > +	/* Device is going away, exit any polling for buffers */
> > +	virtio_break_device(vdev);
> > +	if (use_multiport(portdev))
> > +		flush_work(&portdev->control_work);
> > +	else
> > +		flush_work(&portdev->config_work);
> >  	/* Disable interrupts for vqs */
> 
> newline before comment?

sure

> thanks,
> 
> greg k-h

Re: [PATCH] virtio_console: break out of buf poll on remove

[Michael S. Tsirkin]

On Tue, Oct 05, 2021 at 03:33:42PM +0200, Greg Kroah-Hartman wrote:
> On Tue, Oct 05, 2021 at 03:04:07AM -0400, Michael S. Tsirkin wrote:
> > A common pattern for device reset is currently:
> > vdev->config->reset(vdev);
> > .. cleanup ..
> > 
> > reset prevents new interrupts from arriving and waits for interrupt
> > handlers to finish.
> > 
> > However if - as is common - the handler queues a work request which is
> > flushed during the cleanup stage, we have code adding buffers / trying
> > to get buffers while device is reset. Not good.
> > 
> > This was reproduced by running
> > 	modprobe virtio_console
> > 	modprobe -r virtio_console
> > in a loop.
> 
> That's a pathological case that is not "in the field" except by people
> who want to abuse the system as root.  And they can do much worse things
> than that.
> 
> > Fixing this comprehensively needs some thought, and new APIs.
> > Let's at least handle the specific case of virtio_console
> > removal that was reported in the field.
> 
> Let's fix this correctly, don't just hack it up now.

Well I poked at it some more, and things are not as bad
as I thought. It's mostly just console and possibly virtio-mem.
Well and virtio-bt has a completely borken cleanup that
does not even bother to reset the device, but that's
a separate issue, discussing it with the maintainer.

So I wrote some patches to document the requirements better, added a
wrapper for reset and generally cleaned the API up a bit, and added a
patch for mem, but generally I no longer think we need a major API
change.


> > Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1786239
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  drivers/char/virtio_console.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> > index 7eaf303a7a86..c852ce0b4d56 100644
> > --- a/drivers/char/virtio_console.c
> > +++ b/drivers/char/virtio_console.c
> > @@ -1956,6 +1956,12 @@ static void virtcons_remove(struct virtio_device *vdev)
> >  	list_del(&portdev->list);
> >  	spin_unlock_irq(&pdrvdata_lock);
> >  
> > +	/* Device is going away, exit any polling for buffers */
> > +	virtio_break_device(vdev);
> > +	if (use_multiport(portdev))
> > +		flush_work(&portdev->control_work);
> > +	else
> > +		flush_work(&portdev->config_work);
> >  	/* Disable interrupts for vqs */
> 
> newline before comment?

sure

> thanks,
> 
> greg k-h

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization