[PATCH 0/3] Fixing races in probe/remove

[PATCH 0/3] Fixing races in probe/remove

[Jason Wang]

Hi all:

This series tries to fix races spotted during code review for
caif_virtio in probe() and remove().

Compile test only. (I don't have setup to test them).

Please review.

Thanks

Jason Wang (3):
  caif_virtio: remove virtqueue_disable_cb() in probe
  caif_virtio: fix the race between virtio_device_ready() and ndo_open()
  caif_virtio: fix the race between reset and netdev unregister

 drivers/net/caif/caif_virtio.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

-- 
2.25.1

[PATCH 1/3] caif_virtio: remove virtqueue_disable_cb() in probe

[Jason Wang]

This disabling is a just a hint with best effort, there's no guarantee
that device doesn't send notification. The driver should survive with
that, so let's remove it.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/net/caif/caif_virtio.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
index 5458f57177a0..c677ded81133 100644
--- a/drivers/net/caif/caif_virtio.c
+++ b/drivers/net/caif/caif_virtio.c
@@ -705,9 +705,6 @@ static int cfv_probe(struct virtio_device *vdev)
 	netdev->needed_headroom = cfv->tx_hr;
 	netdev->needed_tailroom = cfv->tx_tr;
 
-	/* Disable buffer release interrupts unless we have stopped TX queues */
-	virtqueue_disable_cb(cfv->vq_tx);
-
 	netdev->mtu = cfv->mtu - cfv->tx_tr;
 	vdev->priv = cfv;
 
-- 
2.25.1

[PATCH 2/3] caif_virtio: fix the race between virtio_device_ready() and ndo_open()

[Jason Wang]

We used to depend on the virtio_device_ready() that is called after
probe() by virtio_dev_probe() after netdev registration. This
cause a race between ndo_open() and virtio_device_ready(): if
ndo_open() is called before virtio_device_ready(), the driver may
start to use the device (e.g TX) before DRIVER_OK which violates the
spec.

Fixing this by switching to use register_netdevice() and protect the
virtio_device_ready() with rtnl_lock() to make sure ndo_open() can
only be called after virtio_device_ready().

Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/net/caif/caif_virtio.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
index c677ded81133..66375bea2fcd 100644
--- a/drivers/net/caif/caif_virtio.c
+++ b/drivers/net/caif/caif_virtio.c
@@ -719,13 +719,21 @@ static int cfv_probe(struct virtio_device *vdev)
 	/* Carrier is off until netdevice is opened */
 	netif_carrier_off(netdev);
 
+	/* serialize netdev register + virtio_device_ready() with ndo_open() */
+	rtnl_lock();
+
 	/* register Netdev */
-	err = register_netdev(netdev);
+	err = register_netdevice(netdev);
 	if (err) {
+		rtnl_unlock();
 		dev_err(&vdev->dev, "Unable to register netdev (%d)\n", err);
 		goto err;
 	}
 
+	virtio_device_ready(vdev);
+
+	rtnl_unlock();
+
 	debugfs_init(cfv);
 
 	return 0;
-- 
2.25.1

[PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

[Jason Wang]

We use to do the following steps during .remove():

static void cfv_remove(struct virtio_device *vdev)
{
	struct cfv_info *cfv = vdev->priv;

	rtnl_lock();
	dev_close(cfv->ndev);
	rtnl_unlock();

	tasklet_kill(&cfv->tx_release_tasklet);
	debugfs_remove_recursive(cfv->debugfs);

	vringh_kiov_cleanup(&cfv->ctx.riov);
	virtio_reset_device(vdev);
	vdev->vringh_config->del_vrhs(cfv->vdev);
	cfv->vr_rx = NULL;
	vdev->config->del_vqs(cfv->vdev);
	unregister_netdev(cfv->ndev);
}

This is racy since device could be re-opened after dev_close() but
before unregister_netdevice():

1) RX vringh is cleaned before resetting the device, rx callbacks that
   is called after the vringh_kiov_cleanup() will result a UAF
2) Network stack can still try to use TX virtqueue even if it has been
   deleted after dev_vqs()

Fixing this by unregistering the network device first to make sure not
device access from both TX and RX side.

Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/net/caif/caif_virtio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
index 66375bea2fcd..a29f9b2df5b1 100644
--- a/drivers/net/caif/caif_virtio.c
+++ b/drivers/net/caif/caif_virtio.c
@@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
 {
 	struct cfv_info *cfv = vdev->priv;
 
-	rtnl_lock();
-	dev_close(cfv->ndev);
-	rtnl_unlock();
+	/* Make sure NAPI/TX won't try to access the device */
+	unregister_netdev(cfv->ndev);
 
 	tasklet_kill(&cfv->tx_release_tasklet);
 	debugfs_remove_recursive(cfv->debugfs);
@@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
 	vdev->vringh_config->del_vrhs(cfv->vdev);
 	cfv->vr_rx = NULL;
 	vdev->config->del_vqs(cfv->vdev);
-	unregister_netdev(cfv->ndev);
 }
 
 static struct virtio_device_id id_table[] = {
-- 
2.25.1

Re: [PATCH 1/3] caif_virtio: remove virtqueue_disable_cb() in probe

[Michael S. Tsirkin]

On Mon, Jun 20, 2022 at 01:11:13PM +0800, Jason Wang wrote:
> This disabling is a just a hint with best effort, there's no guarantee
> that device doesn't send notification. The driver should survive with
> that, so let's remove it.
> 
> Signed-off-by: Jason Wang <jasowang@redhat.com>

I guess, but frankly this change feels gratituous, and just might
uncover latent bugs. Which would be fine if we were out to
find and fix them, but given this was compile-tested only,
I'm not sure that's the case.


> ---
>  drivers/net/caif/caif_virtio.c | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> index 5458f57177a0..c677ded81133 100644
> --- a/drivers/net/caif/caif_virtio.c
> +++ b/drivers/net/caif/caif_virtio.c
> @@ -705,9 +705,6 @@ static int cfv_probe(struct virtio_device *vdev)
>  	netdev->needed_headroom = cfv->tx_hr;
>  	netdev->needed_tailroom = cfv->tx_tr;
>  
> -	/* Disable buffer release interrupts unless we have stopped TX queues */
> -	virtqueue_disable_cb(cfv->vq_tx);
> -
>  	netdev->mtu = cfv->mtu - cfv->tx_tr;
>  	vdev->priv = cfv;
>  
> -- 
> 2.25.1

Re: [PATCH 2/3] caif_virtio: fix the race between virtio_device_ready() and ndo_open()

[Michael S. Tsirkin]

On Mon, Jun 20, 2022 at 01:11:14PM +0800, Jason Wang wrote:
> We used to depend on the virtio_device_ready() that is called after

"We used to" implies it's not the case currently.
I think you mean "we currently depend".

> probe() by virtio_dev_probe() after netdev registration. This
> cause 

causes

>a race between ndo_open() and virtio_device_ready(): if
> ndo_open() is called before virtio_device_ready(), the driver may
> start to use the device (e.g TX) before DRIVER_OK which violates the
> spec.
> 
> Fixing this

Fix this

> by switching to use register_netdevice() and protect the
> virtio_device_ready() with rtnl_lock() to make sure ndo_open() can
> only be called after virtio_device_ready().
> 
> Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
> Signed-off-by: Jason Wang <jasowang@redhat.com>

Acked-by: Michael S. Tsirkin <mst@redhat.com>

> ---
>  drivers/net/caif/caif_virtio.c | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> index c677ded81133..66375bea2fcd 100644
> --- a/drivers/net/caif/caif_virtio.c
> +++ b/drivers/net/caif/caif_virtio.c
> @@ -719,13 +719,21 @@ static int cfv_probe(struct virtio_device *vdev)
>  	/* Carrier is off until netdevice is opened */
>  	netif_carrier_off(netdev);
>  
> +	/* serialize netdev register + virtio_device_ready() with ndo_open() */
> +	rtnl_lock();
> +
>  	/* register Netdev */
> -	err = register_netdev(netdev);
> +	err = register_netdevice(netdev);
>  	if (err) {
> +		rtnl_unlock();
>  		dev_err(&vdev->dev, "Unable to register netdev (%d)\n", err);
>  		goto err;
>  	}
>  
> +	virtio_device_ready(vdev);
> +
> +	rtnl_unlock();
> +
>  	debugfs_init(cfv);
>  
>  	return 0;
> -- 
> 2.25.1

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

[Michael S. Tsirkin]

On Mon, Jun 20, 2022 at 01:11:15PM +0800, Jason Wang wrote:
> We use to do the following steps during .remove():

We currently do


> static void cfv_remove(struct virtio_device *vdev)
> {
> 	struct cfv_info *cfv = vdev->priv;
> 
> 	rtnl_lock();
> 	dev_close(cfv->ndev);
> 	rtnl_unlock();
> 
> 	tasklet_kill(&cfv->tx_release_tasklet);
> 	debugfs_remove_recursive(cfv->debugfs);
> 
> 	vringh_kiov_cleanup(&cfv->ctx.riov);
> 	virtio_reset_device(vdev);
> 	vdev->vringh_config->del_vrhs(cfv->vdev);
> 	cfv->vr_rx = NULL;
> 	vdev->config->del_vqs(cfv->vdev);
> 	unregister_netdev(cfv->ndev);
> }
> This is racy since device could be re-opened after dev_close() but
> before unregister_netdevice():
> 
> 1) RX vringh is cleaned before resetting the device, rx callbacks that
>    is called after the vringh_kiov_cleanup() will result a UAF
> 2) Network stack can still try to use TX virtqueue even if it has been
>    deleted after dev_vqs()
> 
> Fixing this by unregistering the network device first to make sure not
> device access from both TX and RX side.
> 
> Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/net/caif/caif_virtio.c | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> index 66375bea2fcd..a29f9b2df5b1 100644
> --- a/drivers/net/caif/caif_virtio.c
> +++ b/drivers/net/caif/caif_virtio.c
> @@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
>  {
>  	struct cfv_info *cfv = vdev->priv;
>  
> -	rtnl_lock();
> -	dev_close(cfv->ndev);
> -	rtnl_unlock();
> +	/* Make sure NAPI/TX won't try to access the device */
> +	unregister_netdev(cfv->ndev);
>  
>  	tasklet_kill(&cfv->tx_release_tasklet);
>  	debugfs_remove_recursive(cfv->debugfs);
> @@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
>  	vdev->vringh_config->del_vrhs(cfv->vdev);
>  	cfv->vr_rx = NULL;
>  	vdev->config->del_vqs(cfv->vdev);
> -	unregister_netdev(cfv->ndev);
>  }


This gives me pause, callbacks can now trigger after device
has been unregistered. Are we sure this is safe?
Won't it be safer to just keep the rtnl_lock around
the whole process?

>  static struct virtio_device_id id_table[] = {
> -- 
> 2.25.1

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

[Jason Wang]

On Mon, Jun 20, 2022 at 5:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jun 20, 2022 at 01:11:15PM +0800, Jason Wang wrote:
> > We use to do the following steps during .remove():
>
> We currently do
>
>
> > static void cfv_remove(struct virtio_device *vdev)
> > {
> >       struct cfv_info *cfv = vdev->priv;
> >
> >       rtnl_lock();
> >       dev_close(cfv->ndev);
> >       rtnl_unlock();
> >
> >       tasklet_kill(&cfv->tx_release_tasklet);
> >       debugfs_remove_recursive(cfv->debugfs);
> >
> >       vringh_kiov_cleanup(&cfv->ctx.riov);
> >       virtio_reset_device(vdev);
> >       vdev->vringh_config->del_vrhs(cfv->vdev);
> >       cfv->vr_rx = NULL;
> >       vdev->config->del_vqs(cfv->vdev);
> >       unregister_netdev(cfv->ndev);
> > }
> > This is racy since device could be re-opened after dev_close() but
> > before unregister_netdevice():
> >
> > 1) RX vringh is cleaned before resetting the device, rx callbacks that
> >    is called after the vringh_kiov_cleanup() will result a UAF
> > 2) Network stack can still try to use TX virtqueue even if it has been
> >    deleted after dev_vqs()
> >
> > Fixing this by unregistering the network device first to make sure not
> > device access from both TX and RX side.
> >
> > Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > ---
> >  drivers/net/caif/caif_virtio.c | 6 ++----
> >  1 file changed, 2 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> > index 66375bea2fcd..a29f9b2df5b1 100644
> > --- a/drivers/net/caif/caif_virtio.c
> > +++ b/drivers/net/caif/caif_virtio.c
> > @@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
> >  {
> >       struct cfv_info *cfv = vdev->priv;
> >
> > -     rtnl_lock();
> > -     dev_close(cfv->ndev);
> > -     rtnl_unlock();
> > +     /* Make sure NAPI/TX won't try to access the device */
> > +     unregister_netdev(cfv->ndev);
> >
> >       tasklet_kill(&cfv->tx_release_tasklet);
> >       debugfs_remove_recursive(cfv->debugfs);
> > @@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
> >       vdev->vringh_config->del_vrhs(cfv->vdev);
> >       cfv->vr_rx = NULL;
> >       vdev->config->del_vqs(cfv->vdev);
> > -     unregister_netdev(cfv->ndev);
> >  }
>
>
> This gives me pause, callbacks can now trigger after device
> has been unregistered. Are we sure this is safe?

It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled
after tasklet_kill(). I can add a comment to explain this.

> Won't it be safer to just keep the rtnl_lock around
> the whole process?

It looks to me we rtnl_lock can't help in synchronizing with the
callbacks, anything I miss?

Thanks

>
> >  static struct virtio_device_id id_table[] = {
> > --
> > 2.25.1
>

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

[Michael S. Tsirkin]

On Mon, Jun 20, 2022 at 05:18:29PM +0800, Jason Wang wrote:
> On Mon, Jun 20, 2022 at 5:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, Jun 20, 2022 at 01:11:15PM +0800, Jason Wang wrote:
> > > We use to do the following steps during .remove():
> >
> > We currently do
> >
> >
> > > static void cfv_remove(struct virtio_device *vdev)
> > > {
> > >       struct cfv_info *cfv = vdev->priv;
> > >
> > >       rtnl_lock();
> > >       dev_close(cfv->ndev);
> > >       rtnl_unlock();
> > >
> > >       tasklet_kill(&cfv->tx_release_tasklet);
> > >       debugfs_remove_recursive(cfv->debugfs);
> > >
> > >       vringh_kiov_cleanup(&cfv->ctx.riov);
> > >       virtio_reset_device(vdev);
> > >       vdev->vringh_config->del_vrhs(cfv->vdev);
> > >       cfv->vr_rx = NULL;
> > >       vdev->config->del_vqs(cfv->vdev);
> > >       unregister_netdev(cfv->ndev);
> > > }
> > > This is racy since device could be re-opened after dev_close() but
> > > before unregister_netdevice():
> > >
> > > 1) RX vringh is cleaned before resetting the device, rx callbacks that
> > >    is called after the vringh_kiov_cleanup() will result a UAF
> > > 2) Network stack can still try to use TX virtqueue even if it has been
> > >    deleted after dev_vqs()
> > >
> > > Fixing this by unregistering the network device first to make sure not
> > > device access from both TX and RX side.
> > >
> > > Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
> > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > ---
> > >  drivers/net/caif/caif_virtio.c | 6 ++----
> > >  1 file changed, 2 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> > > index 66375bea2fcd..a29f9b2df5b1 100644
> > > --- a/drivers/net/caif/caif_virtio.c
> > > +++ b/drivers/net/caif/caif_virtio.c
> > > @@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
> > >  {
> > >       struct cfv_info *cfv = vdev->priv;
> > >
> > > -     rtnl_lock();
> > > -     dev_close(cfv->ndev);
> > > -     rtnl_unlock();
> > > +     /* Make sure NAPI/TX won't try to access the device */
> > > +     unregister_netdev(cfv->ndev);
> > >
> > >       tasklet_kill(&cfv->tx_release_tasklet);
> > >       debugfs_remove_recursive(cfv->debugfs);
> > > @@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
> > >       vdev->vringh_config->del_vrhs(cfv->vdev);
> > >       cfv->vr_rx = NULL;
> > >       vdev->config->del_vqs(cfv->vdev);
> > > -     unregister_netdev(cfv->ndev);
> > >  }
> >
> >
> > This gives me pause, callbacks can now trigger after device
> > has been unregistered. Are we sure this is safe?
> 
> It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled
> after tasklet_kill(). I can add a comment to explain this.

that waits for outstanding tasklets but does it really prevent
future ones?

> > Won't it be safer to just keep the rtnl_lock around
> > the whole process?
> 
> It looks to me we rtnl_lock can't help in synchronizing with the
> callbacks, anything I miss?
> 
> Thanks

good point.


> >
> > >  static struct virtio_device_id id_table[] = {
> > > --
> > > 2.25.1
> >

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

[Jason Wang]

On Mon, Jun 20, 2022 at 6:18 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jun 20, 2022 at 05:18:29PM +0800, Jason Wang wrote:
> > On Mon, Jun 20, 2022 at 5:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, Jun 20, 2022 at 01:11:15PM +0800, Jason Wang wrote:
> > > > We use to do the following steps during .remove():
> > >
> > > We currently do
> > >
> > >
> > > > static void cfv_remove(struct virtio_device *vdev)
> > > > {
> > > >       struct cfv_info *cfv = vdev->priv;
> > > >
> > > >       rtnl_lock();
> > > >       dev_close(cfv->ndev);
> > > >       rtnl_unlock();
> > > >
> > > >       tasklet_kill(&cfv->tx_release_tasklet);
> > > >       debugfs_remove_recursive(cfv->debugfs);
> > > >
> > > >       vringh_kiov_cleanup(&cfv->ctx.riov);
> > > >       virtio_reset_device(vdev);
> > > >       vdev->vringh_config->del_vrhs(cfv->vdev);
> > > >       cfv->vr_rx = NULL;
> > > >       vdev->config->del_vqs(cfv->vdev);
> > > >       unregister_netdev(cfv->ndev);
> > > > }
> > > > This is racy since device could be re-opened after dev_close() but
> > > > before unregister_netdevice():
> > > >
> > > > 1) RX vringh is cleaned before resetting the device, rx callbacks that
> > > >    is called after the vringh_kiov_cleanup() will result a UAF
> > > > 2) Network stack can still try to use TX virtqueue even if it has been
> > > >    deleted after dev_vqs()
> > > >
> > > > Fixing this by unregistering the network device first to make sure not
> > > > device access from both TX and RX side.
> > > >
> > > > Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
> > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > ---
> > > >  drivers/net/caif/caif_virtio.c | 6 ++----
> > > >  1 file changed, 2 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> > > > index 66375bea2fcd..a29f9b2df5b1 100644
> > > > --- a/drivers/net/caif/caif_virtio.c
> > > > +++ b/drivers/net/caif/caif_virtio.c
> > > > @@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
> > > >  {
> > > >       struct cfv_info *cfv = vdev->priv;
> > > >
> > > > -     rtnl_lock();
> > > > -     dev_close(cfv->ndev);
> > > > -     rtnl_unlock();
> > > > +     /* Make sure NAPI/TX won't try to access the device */
> > > > +     unregister_netdev(cfv->ndev);
> > > >
> > > >       tasklet_kill(&cfv->tx_release_tasklet);
> > > >       debugfs_remove_recursive(cfv->debugfs);
> > > > @@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
> > > >       vdev->vringh_config->del_vrhs(cfv->vdev);
> > > >       cfv->vr_rx = NULL;
> > > >       vdev->config->del_vqs(cfv->vdev);
> > > > -     unregister_netdev(cfv->ndev);
> > > >  }
> > >
> > >
> > > This gives me pause, callbacks can now trigger after device
> > > has been unregistered. Are we sure this is safe?
> >
> > It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled
> > after tasklet_kill(). I can add a comment to explain this.
>
> that waits for outstanding tasklets but does it really prevent
> future ones?

I think so, it tries to test and set TASKLET_STATE_SCHED which blocks
the future scheduling of a tasklet.

Thanks

>
> > > Won't it be safer to just keep the rtnl_lock around
> > > the whole process?
> >
> > It looks to me we rtnl_lock can't help in synchronizing with the
> > callbacks, anything I miss?
> >
> > Thanks
>
> good point.
>
>
> > >
> > > >  static struct virtio_device_id id_table[] = {
> > > > --
> > > > 2.25.1
> > >
>

Re: [PATCH 1/3] caif_virtio: remove virtqueue_disable_cb() in probe

[Jason Wang]

On Mon, Jun 20, 2022 at 5:02 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jun 20, 2022 at 01:11:13PM +0800, Jason Wang wrote:
> > This disabling is a just a hint with best effort, there's no guarantee
> > that device doesn't send notification. The driver should survive with
> > that, so let's remove it.
> >
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
>
> I guess, but frankly this change feels gratituous, and just might
> uncover latent bugs. Which would be fine if we were out to
> find and fix them, but given this was compile-tested only,
> I'm not sure that's the case.

Ok, let me drop this from the next version (no way to test).

Thanks

>
>
> > ---
> >  drivers/net/caif/caif_virtio.c | 3 ---
> >  1 file changed, 3 deletions(-)
> >
> > diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> > index 5458f57177a0..c677ded81133 100644
> > --- a/drivers/net/caif/caif_virtio.c
> > +++ b/drivers/net/caif/caif_virtio.c
> > @@ -705,9 +705,6 @@ static int cfv_probe(struct virtio_device *vdev)
> >       netdev->needed_headroom = cfv->tx_hr;
> >       netdev->needed_tailroom = cfv->tx_tr;
> >
> > -     /* Disable buffer release interrupts unless we have stopped TX queues */
> > -     virtqueue_disable_cb(cfv->vq_tx);
> > -
> >       netdev->mtu = cfv->mtu - cfv->tx_tr;
> >       vdev->priv = cfv;
> >
> > --
> > 2.25.1
>

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

[Michael S. Tsirkin]

On Tue, Jun 21, 2022 at 11:09:45AM +0800, Jason Wang wrote:
> On Mon, Jun 20, 2022 at 6:18 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, Jun 20, 2022 at 05:18:29PM +0800, Jason Wang wrote:
> > > On Mon, Jun 20, 2022 at 5:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, Jun 20, 2022 at 01:11:15PM +0800, Jason Wang wrote:
> > > > > We use to do the following steps during .remove():
> > > >
> > > > We currently do
> > > >
> > > >
> > > > > static void cfv_remove(struct virtio_device *vdev)
> > > > > {
> > > > >       struct cfv_info *cfv = vdev->priv;
> > > > >
> > > > >       rtnl_lock();
> > > > >       dev_close(cfv->ndev);
> > > > >       rtnl_unlock();
> > > > >
> > > > >       tasklet_kill(&cfv->tx_release_tasklet);
> > > > >       debugfs_remove_recursive(cfv->debugfs);
> > > > >
> > > > >       vringh_kiov_cleanup(&cfv->ctx.riov);
> > > > >       virtio_reset_device(vdev);
> > > > >       vdev->vringh_config->del_vrhs(cfv->vdev);
> > > > >       cfv->vr_rx = NULL;
> > > > >       vdev->config->del_vqs(cfv->vdev);
> > > > >       unregister_netdev(cfv->ndev);
> > > > > }
> > > > > This is racy since device could be re-opened after dev_close() but
> > > > > before unregister_netdevice():
> > > > >
> > > > > 1) RX vringh is cleaned before resetting the device, rx callbacks that
> > > > >    is called after the vringh_kiov_cleanup() will result a UAF
> > > > > 2) Network stack can still try to use TX virtqueue even if it has been
> > > > >    deleted after dev_vqs()
> > > > >
> > > > > Fixing this by unregistering the network device first to make sure not
> > > > > device access from both TX and RX side.
> > > > >
> > > > > Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
> > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > ---
> > > > >  drivers/net/caif/caif_virtio.c | 6 ++----
> > > > >  1 file changed, 2 insertions(+), 4 deletions(-)
> > > > >
> > > > > diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> > > > > index 66375bea2fcd..a29f9b2df5b1 100644
> > > > > --- a/drivers/net/caif/caif_virtio.c
> > > > > +++ b/drivers/net/caif/caif_virtio.c
> > > > > @@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
> > > > >  {
> > > > >       struct cfv_info *cfv = vdev->priv;
> > > > >
> > > > > -     rtnl_lock();
> > > > > -     dev_close(cfv->ndev);
> > > > > -     rtnl_unlock();
> > > > > +     /* Make sure NAPI/TX won't try to access the device */
> > > > > +     unregister_netdev(cfv->ndev);
> > > > >
> > > > >       tasklet_kill(&cfv->tx_release_tasklet);
> > > > >       debugfs_remove_recursive(cfv->debugfs);
> > > > > @@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
> > > > >       vdev->vringh_config->del_vrhs(cfv->vdev);
> > > > >       cfv->vr_rx = NULL;
> > > > >       vdev->config->del_vqs(cfv->vdev);
> > > > > -     unregister_netdev(cfv->ndev);
> > > > >  }
> > > >
> > > >
> > > > This gives me pause, callbacks can now trigger after device
> > > > has been unregistered. Are we sure this is safe?
> > >
> > > It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled
> > > after tasklet_kill(). I can add a comment to explain this.
> >
> > that waits for outstanding tasklets but does it really prevent
> > future ones?
> 
> I think so, it tries to test and set TASKLET_STATE_SCHED which blocks
> the future scheduling of a tasklet.
> 
> Thanks

But then in the end it clears it, does it not?

> >
> > > > Won't it be safer to just keep the rtnl_lock around
> > > > the whole process?
> > >
> > > It looks to me we rtnl_lock can't help in synchronizing with the
> > > callbacks, anything I miss?
> > >
> > > Thanks
> >
> > good point.
> >
> >
> > > >
> > > > >  static struct virtio_device_id id_table[] = {
> > > > > --
> > > > > 2.25.1
> > > >
> >

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

[Jason Wang]

On Tue, Jun 21, 2022 at 2:00 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 21, 2022 at 11:09:45AM +0800, Jason Wang wrote:
> > On Mon, Jun 20, 2022 at 6:18 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, Jun 20, 2022 at 05:18:29PM +0800, Jason Wang wrote:
> > > > On Mon, Jun 20, 2022 at 5:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Mon, Jun 20, 2022 at 01:11:15PM +0800, Jason Wang wrote:
> > > > > > We use to do the following steps during .remove():
> > > > >
> > > > > We currently do
> > > > >
> > > > >
> > > > > > static void cfv_remove(struct virtio_device *vdev)
> > > > > > {
> > > > > >       struct cfv_info *cfv = vdev->priv;
> > > > > >
> > > > > >       rtnl_lock();
> > > > > >       dev_close(cfv->ndev);
> > > > > >       rtnl_unlock();
> > > > > >
> > > > > >       tasklet_kill(&cfv->tx_release_tasklet);
> > > > > >       debugfs_remove_recursive(cfv->debugfs);
> > > > > >
> > > > > >       vringh_kiov_cleanup(&cfv->ctx.riov);
> > > > > >       virtio_reset_device(vdev);
> > > > > >       vdev->vringh_config->del_vrhs(cfv->vdev);
> > > > > >       cfv->vr_rx = NULL;
> > > > > >       vdev->config->del_vqs(cfv->vdev);
> > > > > >       unregister_netdev(cfv->ndev);
> > > > > > }
> > > > > > This is racy since device could be re-opened after dev_close() but
> > > > > > before unregister_netdevice():
> > > > > >
> > > > > > 1) RX vringh is cleaned before resetting the device, rx callbacks that
> > > > > >    is called after the vringh_kiov_cleanup() will result a UAF
> > > > > > 2) Network stack can still try to use TX virtqueue even if it has been
> > > > > >    deleted after dev_vqs()
> > > > > >
> > > > > > Fixing this by unregistering the network device first to make sure not
> > > > > > device access from both TX and RX side.
> > > > > >
> > > > > > Fixes: 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
> > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > ---
> > > > > >  drivers/net/caif/caif_virtio.c | 6 ++----
> > > > > >  1 file changed, 2 insertions(+), 4 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> > > > > > index 66375bea2fcd..a29f9b2df5b1 100644
> > > > > > --- a/drivers/net/caif/caif_virtio.c
> > > > > > +++ b/drivers/net/caif/caif_virtio.c
> > > > > > @@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
> > > > > >  {
> > > > > >       struct cfv_info *cfv = vdev->priv;
> > > > > >
> > > > > > -     rtnl_lock();
> > > > > > -     dev_close(cfv->ndev);
> > > > > > -     rtnl_unlock();
> > > > > > +     /* Make sure NAPI/TX won't try to access the device */
> > > > > > +     unregister_netdev(cfv->ndev);
> > > > > >
> > > > > >       tasklet_kill(&cfv->tx_release_tasklet);
> > > > > >       debugfs_remove_recursive(cfv->debugfs);
> > > > > > @@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
> > > > > >       vdev->vringh_config->del_vrhs(cfv->vdev);
> > > > > >       cfv->vr_rx = NULL;
> > > > > >       vdev->config->del_vqs(cfv->vdev);
> > > > > > -     unregister_netdev(cfv->ndev);
> > > > > >  }
> > > > >
> > > > >
> > > > > This gives me pause, callbacks can now trigger after device
> > > > > has been unregistered. Are we sure this is safe?
> > > >
> > > > It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled
> > > > after tasklet_kill(). I can add a comment to explain this.
> > >
> > > that waits for outstanding tasklets but does it really prevent
> > > future ones?
> >
> > I think so, it tries to test and set TASKLET_STATE_SCHED which blocks
> > the future scheduling of a tasklet.
> >
> > Thanks
>
> But then in the end it clears it, does it not?

Right, so we need to reset before taskset_kill().

Thanks

>
> > >
> > > > > Won't it be safer to just keep the rtnl_lock around
> > > > > the whole process?
> > > >
> > > > It looks to me we rtnl_lock can't help in synchronizing with the
> > > > callbacks, anything I miss?
> > > >
> > > > Thanks
> > >
> > > good point.
> > >
> > >
> > > > >
> > > > > >  static struct virtio_device_id id_table[] = {
> > > > > > --
> > > > > > 2.25.1
> > > > >
> > >
>