[PATCH RFC 0/2] vsock/virtio: fix issues on device hot-unplug

[PATCH RFC 0/2] vsock/virtio: fix issues on device hot-unplug

[Stefano Garzarella]

These patches try to handle the hot-unplug of vsock virtio transport device in
a proper way.

Maybe move the vsock_core_init()/vsock_core_exit() functions in the module_init
and module_exit of vsock_virtio_transport module can't be the best way, but the
architecture of vsock_core forces us to this approach for now.

The vsock_core proto_ops expect a valid pointer to the transport device, so we
can't call vsock_core_exit() until there are open sockets.

Another (little more complex) approach during the device removal, could be to
unregister the AF_VSOCK protocol, then reset all sockets and wait for their
destruction. At this point, we can set the transport pointer to NULL.

Any suggestions would be helpful.

Stefano Garzarella (2):
  vsock/virtio: fix kernel panic after device hot-unplug
  vsock/virtio: reset connected sockets on device removal

 net/vmw_vsock/virtio_transport.c | 29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)

-- 
2.19.2

[PATCH RFC 1/2] vsock/virtio: fix kernel panic after device hot-unplug

[Stefano Garzarella]

virtio_vsock_remove() invoked the vsock_core_exit() also if there
are opened socket for the AF_VSOCK protocol family. In this way
the vsock "transport" pointer was set to NULL, triggering the
kernel panic at the first socket activity.

This patch move the vsock_core_init()/vsock_core_exit() in the
virtio_vsock respectively in module_init and module_exit functions,
that cannot be invoked until there are open sockets.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1609699
Reported-by: Yan Fu <yafu@redhat.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
---
 net/vmw_vsock/virtio_transport.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
index 5d3cce9e8744..9dae54698737 100644
--- a/net/vmw_vsock/virtio_transport.c
+++ b/net/vmw_vsock/virtio_transport.c
@@ -75,6 +75,9 @@ static u32 virtio_transport_get_local_cid(void)
 {
 	struct virtio_vsock *vsock = virtio_vsock_get();
 
+	if (!vsock)
+		return VMADDR_CID_ANY;
+
 	return vsock->guest_cid;
 }
 
@@ -584,10 +587,6 @@ static int virtio_vsock_probe(struct virtio_device *vdev)
 
 	virtio_vsock_update_guest_cid(vsock);
 
-	ret = vsock_core_init(&virtio_transport.transport);
-	if (ret < 0)
-		goto out_vqs;
-
 	vsock->rx_buf_nr = 0;
 	vsock->rx_buf_max_nr = 0;
 	atomic_set(&vsock->queued_replies, 0);
@@ -618,8 +617,6 @@ static int virtio_vsock_probe(struct virtio_device *vdev)
 	mutex_unlock(&the_virtio_vsock_mutex);
 	return 0;
 
-out_vqs:
-	vsock->vdev->config->del_vqs(vsock->vdev);
 out:
 	kfree(vsock);
 	mutex_unlock(&the_virtio_vsock_mutex);
@@ -669,7 +666,6 @@ static void virtio_vsock_remove(struct virtio_device *vdev)
 
 	mutex_lock(&the_virtio_vsock_mutex);
 	the_virtio_vsock = NULL;
-	vsock_core_exit();
 	mutex_unlock(&the_virtio_vsock_mutex);
 
 	vdev->config->del_vqs(vdev);
@@ -702,14 +698,28 @@ static int __init virtio_vsock_init(void)
 	virtio_vsock_workqueue = alloc_workqueue("virtio_vsock", 0, 0);
 	if (!virtio_vsock_workqueue)
 		return -ENOMEM;
+
 	ret = register_virtio_driver(&virtio_vsock_driver);
 	if (ret)
-		destroy_workqueue(virtio_vsock_workqueue);
+		goto out_wq;
+
+	ret = vsock_core_init(&virtio_transport.transport);
+	if (ret)
+		goto out_vdr;
+
+	return 0;
+
+out_vdr:
+	unregister_virtio_driver(&virtio_vsock_driver);
+out_wq:
+	destroy_workqueue(virtio_vsock_workqueue);
 	return ret;
+
 }
 
 static void __exit virtio_vsock_exit(void)
 {
+	vsock_core_exit();
 	unregister_virtio_driver(&virtio_vsock_driver);
 	destroy_workqueue(virtio_vsock_workqueue);
 }
-- 
2.19.2

[PATCH RFC 2/2] vsock/virtio: reset connected sockets on device removal

[Stefano Garzarella]

When the virtio transport device disappear, we should reset all
connected sockets in order to inform the users.

Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
---
 net/vmw_vsock/virtio_transport.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
index 9dae54698737..15eb5d3d4750 100644
--- a/net/vmw_vsock/virtio_transport.c
+++ b/net/vmw_vsock/virtio_transport.c
@@ -634,6 +634,9 @@ static void virtio_vsock_remove(struct virtio_device *vdev)
 	flush_work(&vsock->event_work);
 	flush_work(&vsock->send_pkt_work);
 
+	/* Reset all connected sockets when the device disappear */
+	vsock_for_each_connected_socket(virtio_vsock_reset_sock);
+
 	vdev->config->reset(vdev);
 
 	mutex_lock(&vsock->rx_lock);
-- 
2.19.2

Re: [PATCH RFC 1/2] vsock/virtio: fix kernel panic after device hot-unplug

[Stefano Garzarella]

I fixed some mistakes in the commit text (for v2)

On Thu, Dec 20, 2018 at 1:16 PM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> virtio_vsock_remove() invoked the vsock_core_exit() also if there
> are opened socket for the AF_VSOCK protocol family. In this way
> the vsock "transport" pointer was set to NULL, triggering the
> kernel panic at the first socket activity.

virtio_vsock_remove() invokes the vsock_core_exit() also if there
are opened sockets for the AF_VSOCK protocol family. In this way
the vsock "transport" pointer is set to NULL, triggering the
kernel panic at the first socket activity.

>
> This patch move the vsock_core_init()/vsock_core_exit() in the
> virtio_vsock respectively in module_init and module_exit functions,
> that cannot be invoked until there are open sockets.
>
> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1609699
> Reported-by: Yan Fu <yafu@redhat.com>
> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
> ---
>  net/vmw_vsock/virtio_transport.c | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
>
> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> index 5d3cce9e8744..9dae54698737 100644
> --- a/net/vmw_vsock/virtio_transport.c
> +++ b/net/vmw_vsock/virtio_transport.c
> @@ -75,6 +75,9 @@ static u32 virtio_transport_get_local_cid(void)
>  {
>         struct virtio_vsock *vsock = virtio_vsock_get();
>
> +       if (!vsock)
> +               return VMADDR_CID_ANY;
> +
>         return vsock->guest_cid;
>  }
>
> @@ -584,10 +587,6 @@ static int virtio_vsock_probe(struct virtio_device *vdev)
>
>         virtio_vsock_update_guest_cid(vsock);
>
> -       ret = vsock_core_init(&virtio_transport.transport);
> -       if (ret < 0)
> -               goto out_vqs;
> -
>         vsock->rx_buf_nr = 0;
>         vsock->rx_buf_max_nr = 0;
>         atomic_set(&vsock->queued_replies, 0);
> @@ -618,8 +617,6 @@ static int virtio_vsock_probe(struct virtio_device *vdev)
>         mutex_unlock(&the_virtio_vsock_mutex);
>         return 0;
>
> -out_vqs:
> -       vsock->vdev->config->del_vqs(vsock->vdev);
>  out:
>         kfree(vsock);
>         mutex_unlock(&the_virtio_vsock_mutex);
> @@ -669,7 +666,6 @@ static void virtio_vsock_remove(struct virtio_device *vdev)
>
>         mutex_lock(&the_virtio_vsock_mutex);
>         the_virtio_vsock = NULL;
> -       vsock_core_exit();
>         mutex_unlock(&the_virtio_vsock_mutex);
>
>         vdev->config->del_vqs(vdev);
> @@ -702,14 +698,28 @@ static int __init virtio_vsock_init(void)
>         virtio_vsock_workqueue = alloc_workqueue("virtio_vsock", 0, 0);
>         if (!virtio_vsock_workqueue)
>                 return -ENOMEM;
> +
>         ret = register_virtio_driver(&virtio_vsock_driver);
>         if (ret)
> -               destroy_workqueue(virtio_vsock_workqueue);
> +               goto out_wq;
> +
> +       ret = vsock_core_init(&virtio_transport.transport);
> +       if (ret)
> +               goto out_vdr;
> +
> +       return 0;
> +
> +out_vdr:
> +       unregister_virtio_driver(&virtio_vsock_driver);
> +out_wq:
> +       destroy_workqueue(virtio_vsock_workqueue);
>         return ret;
> +
>  }
>
>  static void __exit virtio_vsock_exit(void)
>  {
> +       vsock_core_exit();
>         unregister_virtio_driver(&virtio_vsock_driver);
>         destroy_workqueue(virtio_vsock_workqueue);
>  }
> --
> 2.19.2
>

Re: [PATCH RFC 1/2] vsock/virtio: fix kernel panic after device hot-unplug

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

On Thu, Dec 20, 2018 at 01:15:34PM +0100, Stefano Garzarella wrote:
> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> index 5d3cce9e8744..9dae54698737 100644
> --- a/net/vmw_vsock/virtio_transport.c
> +++ b/net/vmw_vsock/virtio_transport.c
> @@ -75,6 +75,9 @@ static u32 virtio_transport_get_local_cid(void)
>  {
>  	struct virtio_vsock *vsock = virtio_vsock_get();
>  
> +	if (!vsock)
> +		return VMADDR_CID_ANY;
> +
>  	return vsock->guest_cid;
>  }

This looks unrelated to the rest of the patch.  Why is it necessary?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [PATCH RFC 2/2] vsock/virtio: reset connected sockets on device removal

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

On Thu, Dec 20, 2018 at 01:15:35PM +0100, Stefano Garzarella wrote:
> When the virtio transport device disappear, we should reset all
> connected sockets in order to inform the users.
> 
> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
> ---
>  net/vmw_vsock/virtio_transport.c | 3 +++
>  1 file changed, 3 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [PATCH RFC 1/2] vsock/virtio: fix kernel panic after device hot-unplug

[Stefano Garzarella]

On Wed, Jan 2, 2019 at 10:39 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Thu, Dec 20, 2018 at 01:15:34PM +0100, Stefano Garzarella wrote:
> > diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> > index 5d3cce9e8744..9dae54698737 100644
> > --- a/net/vmw_vsock/virtio_transport.c
> > +++ b/net/vmw_vsock/virtio_transport.c
> > @@ -75,6 +75,9 @@ static u32 virtio_transport_get_local_cid(void)
> >  {
> >       struct virtio_vsock *vsock = virtio_vsock_get();
> >
> > +     if (!vsock)
> > +             return VMADDR_CID_ANY;
> > +
> >       return vsock->guest_cid;
> >  }
>
> This looks unrelated to the rest of the patch.  Why is it necessary?

It is needed because the "the_virtio_vsock" returned by
virtio_vsock_get() is initialized during the probe and freed during
the removal.
So, if we move the vsock_core_exit() in the virtio_vsock_exit(), can
happen that the virtio_transport_get_local_cid() is called when the
"the_virtio_vsock" is NULL.

Do you think is better to split this patch?

Thanks,
Stefano

Re: [PATCH RFC 1/2] vsock/virtio: fix kernel panic after device hot-unplug

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1371 bytes --]

On Wed, Jan 02, 2019 at 11:01:27AM +0100, Stefano Garzarella wrote:
> On Wed, Jan 2, 2019 at 10:39 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Thu, Dec 20, 2018 at 01:15:34PM +0100, Stefano Garzarella wrote:
> > > diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> > > index 5d3cce9e8744..9dae54698737 100644
> > > --- a/net/vmw_vsock/virtio_transport.c
> > > +++ b/net/vmw_vsock/virtio_transport.c
> > > @@ -75,6 +75,9 @@ static u32 virtio_transport_get_local_cid(void)
> > >  {
> > >       struct virtio_vsock *vsock = virtio_vsock_get();
> > >
> > > +     if (!vsock)
> > > +             return VMADDR_CID_ANY;
> > > +
> > >       return vsock->guest_cid;
> > >  }
> >
> > This looks unrelated to the rest of the patch.  Why is it necessary?
> 
> It is needed because the "the_virtio_vsock" returned by
> virtio_vsock_get() is initialized during the probe and freed during
> the removal.
> So, if we move the vsock_core_exit() in the virtio_vsock_exit(), can
> happen that the virtio_transport_get_local_cid() is called when the
> "the_virtio_vsock" is NULL.
> 
> Do you think is better to split this patch?

I'm curious which code paths reach virtio_transport_get_local_cid()
after the virtio device has been removed.  ioctl
IOCTL_VM_SOCKETS_GET_LOCAL_CID does.  Anything else?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [PATCH RFC 1/2] vsock/virtio: fix kernel panic after device hot-unplug

[Stefano Garzarella]

On Thu, Jan 3, 2019 at 11:03 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Wed, Jan 02, 2019 at 11:01:27AM +0100, Stefano Garzarella wrote:
> > On Wed, Jan 2, 2019 at 10:39 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > >
> > > On Thu, Dec 20, 2018 at 01:15:34PM +0100, Stefano Garzarella wrote:
> > > > diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> > > > index 5d3cce9e8744..9dae54698737 100644
> > > > --- a/net/vmw_vsock/virtio_transport.c
> > > > +++ b/net/vmw_vsock/virtio_transport.c
> > > > @@ -75,6 +75,9 @@ static u32 virtio_transport_get_local_cid(void)
> > > >  {
> > > >       struct virtio_vsock *vsock = virtio_vsock_get();
> > > >
> > > > +     if (!vsock)
> > > > +             return VMADDR_CID_ANY;
> > > > +
> > > >       return vsock->guest_cid;
> > > >  }
> > >
> > > This looks unrelated to the rest of the patch.  Why is it necessary?
> >
> > It is needed because the "the_virtio_vsock" returned by
> > virtio_vsock_get() is initialized during the probe and freed during
> > the removal.
> > So, if we move the vsock_core_exit() in the virtio_vsock_exit(), can
> > happen that the virtio_transport_get_local_cid() is called when the
> > "the_virtio_vsock" is NULL.
> >
> > Do you think is better to split this patch?
>
> I'm curious which code paths reach virtio_transport_get_local_cid()
> after the virtio device has been removed.  ioctl
> IOCTL_VM_SOCKETS_GET_LOCAL_CID does.  Anything else?
>

Looking at af_vsock.c another path should be through the vsock_bind(),
if we initialize the vsock_core in the module_init, a user can bind
the socket when the virtio_vsock device is not probed or it is
removed.

Stefano

Re: [PATCH RFC 1/2] vsock/virtio: fix kernel panic after device hot-unplug

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2020 bytes --]

On Thu, Jan 03, 2019 at 03:09:39PM +0100, Stefano Garzarella wrote:
> On Thu, Jan 3, 2019 at 11:03 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Wed, Jan 02, 2019 at 11:01:27AM +0100, Stefano Garzarella wrote:
> > > On Wed, Jan 2, 2019 at 10:39 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > >
> > > > On Thu, Dec 20, 2018 at 01:15:34PM +0100, Stefano Garzarella wrote:
> > > > > diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> > > > > index 5d3cce9e8744..9dae54698737 100644
> > > > > --- a/net/vmw_vsock/virtio_transport.c
> > > > > +++ b/net/vmw_vsock/virtio_transport.c
> > > > > @@ -75,6 +75,9 @@ static u32 virtio_transport_get_local_cid(void)
> > > > >  {
> > > > >       struct virtio_vsock *vsock = virtio_vsock_get();
> > > > >
> > > > > +     if (!vsock)
> > > > > +             return VMADDR_CID_ANY;
> > > > > +
> > > > >       return vsock->guest_cid;
> > > > >  }
> > > >
> > > > This looks unrelated to the rest of the patch.  Why is it necessary?
> > >
> > > It is needed because the "the_virtio_vsock" returned by
> > > virtio_vsock_get() is initialized during the probe and freed during
> > > the removal.
> > > So, if we move the vsock_core_exit() in the virtio_vsock_exit(), can
> > > happen that the virtio_transport_get_local_cid() is called when the
> > > "the_virtio_vsock" is NULL.
> > >
> > > Do you think is better to split this patch?
> >
> > I'm curious which code paths reach virtio_transport_get_local_cid()
> > after the virtio device has been removed.  ioctl
> > IOCTL_VM_SOCKETS_GET_LOCAL_CID does.  Anything else?
> >
> 
> Looking at af_vsock.c another path should be through the vsock_bind(),
> if we initialize the vsock_core in the module_init, a user can bind
> the socket when the virtio_vsock device is not probed or it is
> removed.

I've audited the bind code path and that looks alright.

There are not many other users of the_virtio_vsock and they look safe
too.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [PATCH RFC 0/2] vsock/virtio: fix issues on device hot-unplug

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1309 bytes --]

On Thu, Dec 20, 2018 at 01:15:33PM +0100, Stefano Garzarella wrote:
> These patches try to handle the hot-unplug of vsock virtio transport device in
> a proper way.
> 
> Maybe move the vsock_core_init()/vsock_core_exit() functions in the module_init
> and module_exit of vsock_virtio_transport module can't be the best way, but the
> architecture of vsock_core forces us to this approach for now.
> 
> The vsock_core proto_ops expect a valid pointer to the transport device, so we
> can't call vsock_core_exit() until there are open sockets.
> 
> Another (little more complex) approach during the device removal, could be to
> unregister the AF_VSOCK protocol, then reset all sockets and wait for their
> destruction. At this point, we can set the transport pointer to NULL.
> 
> Any suggestions would be helpful.
> 
> Stefano Garzarella (2):
>   vsock/virtio: fix kernel panic after device hot-unplug
>   vsock/virtio: reset connected sockets on device removal
> 
>  net/vmw_vsock/virtio_transport.c | 29 +++++++++++++++++++++--------
>  1 file changed, 21 insertions(+), 8 deletions(-)
> 
> -- 
> 2.19.2
> 

Fine in the current model.  Once we tackle nested virtualization
(two transports at once) we'll have to revisit this.

Acked-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [PATCH RFC 0/2] vsock/virtio: fix issues on device hot-unplug

[Stefano Garzarella]

On Fri, Jan 4, 2019 at 12:44 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Thu, Dec 20, 2018 at 01:15:33PM +0100, Stefano Garzarella wrote:
> > These patches try to handle the hot-unplug of vsock virtio transport device in
> > a proper way.
> >
> > Maybe move the vsock_core_init()/vsock_core_exit() functions in the module_init
> > and module_exit of vsock_virtio_transport module can't be the best way, but the
> > architecture of vsock_core forces us to this approach for now.
> >
> > The vsock_core proto_ops expect a valid pointer to the transport device, so we
> > can't call vsock_core_exit() until there are open sockets.
> >
> > Another (little more complex) approach during the device removal, could be to
> > unregister the AF_VSOCK protocol, then reset all sockets and wait for their
> > destruction. At this point, we can set the transport pointer to NULL.
> >
> > Any suggestions would be helpful.
> >
> > Stefano Garzarella (2):
> >   vsock/virtio: fix kernel panic after device hot-unplug
> >   vsock/virtio: reset connected sockets on device removal
> >
> >  net/vmw_vsock/virtio_transport.c | 29 +++++++++++++++++++++--------
> >  1 file changed, 21 insertions(+), 8 deletions(-)
> >
> > --
> > 2.19.2
> >
>
> Fine in the current model.  Once we tackle nested virtualization
> (two transports at once) we'll have to revisit this.

I completely agree with you!

>
> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>

I'll send a v2 fixing the commit message of patch 1.

Thanks,
Stefano