* [PATCH] usbip: fix vhci races in connection tear down
@ 2021-03-12 2:27 Shuah Khan
2021-03-12 10:45 ` Johan Hovold
[not found] ` <20210312070806.383-1-hdanton@sina.com>
0 siblings, 2 replies; 4+ messages in thread
From: Shuah Khan @ 2021-03-12 2:27 UTC (permalink / raw)
To: valentina.manea.m, shuah, gregkh
Cc: Shuah Khan, linux-usb, linux-kernel, syzbot+a93fba6d384346a761e3
vhci_shutdown_connection() references connection state (tcp_socket,
tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the
lock.
Current connection tear down sequence:
Step 1: shutdown the socket
Step 2: stop rx thread and reset tcp_rx pointer
Step 3: stop tx thread and reset tcp_tx pointer
Step 4: Reset tcp_socket and sockfd
There are several race windows between these steps. In addition, device
reset routine (vhci_device_reset) resets tcp_socket and sockfd holding
the lock.
Fix these races:
- Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs
only once.
- Change attach_store() to initialize in_disconnect to false while
initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd)
- Change vhci_shutdown_connection() to check in_disconnect and bail
out if disconnect is in progress.
- Change vhci_shutdown_connection() to
-- hold lock to save connection state pointers and unlock.
-- Shutdown the socket and stop threads.
-- Hold lock to clear connection status and in_disconnect flag.
- Change vhci_device_reset() to reset tcp_socket and sockfd.
if !in_disconnect
Tested syzbot and the reproducer did not trigger any issue.
Reported-and-tested-by: syzbot+a93fba6d384346a761e3@syzkaller.appspotmail.com
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
---
drivers/usb/usbip/usbip_common.h | 1 +
drivers/usb/usbip/vhci_hcd.c | 55 +++++++++++++++++++++++---------
drivers/usb/usbip/vhci_sysfs.c | 4 +++
3 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/drivers/usb/usbip/usbip_common.h b/drivers/usb/usbip/usbip_common.h
index d60ce17d3dd2..f6261c5a8c91 100644
--- a/drivers/usb/usbip/usbip_common.h
+++ b/drivers/usb/usbip/usbip_common.h
@@ -268,6 +268,7 @@ struct usbip_device {
struct task_struct *tcp_rx;
struct task_struct *tcp_tx;
+ bool in_disconnect; /* run device disconnect just once */
unsigned long event;
wait_queue_head_t eh_waitq;
diff --git a/drivers/usb/usbip/vhci_hcd.c b/drivers/usb/usbip/vhci_hcd.c
index 3209b5ddd30c..c1917efe5737 100644
--- a/drivers/usb/usbip/vhci_hcd.c
+++ b/drivers/usb/usbip/vhci_hcd.c
@@ -1007,31 +1007,54 @@ static void vhci_device_unlink_cleanup(struct vhci_device *vdev)
static void vhci_shutdown_connection(struct usbip_device *ud)
{
struct vhci_device *vdev = container_of(ud, struct vhci_device, ud);
+ unsigned long flags;
+ struct socket *socket;
+ struct task_struct *tcp_rx = NULL;
+ struct task_struct *tcp_tx = NULL;
+ int sockfd = 0;
+
+ spin_lock_irqsave(&ud->lock, flags);
+ if (vdev->ud.in_disconnect) {
+ pr_info("%s: Disconnect in progress for sockfd %d\n",
+ __func__, ud->sockfd);
+ spin_unlock_irqrestore(&ud->lock, flags);
+ return;
+ }
+ vdev->ud.in_disconnect = true;
+ socket = ud->tcp_socket;
+ tcp_rx = vdev->ud.tcp_rx;
+ tcp_tx = vdev->ud.tcp_tx;
+ sockfd = ud->sockfd;
+ spin_unlock_irqrestore(&ud->lock, flags);
/* need this? see stub_dev.c */
- if (ud->tcp_socket) {
- pr_debug("shutdown tcp_socket %d\n", ud->sockfd);
- kernel_sock_shutdown(ud->tcp_socket, SHUT_RDWR);
+ if (socket) {
+ pr_info("%s: shutdown tcp_socket %d\n", __func__, sockfd);
+ kernel_sock_shutdown(socket, SHUT_RDWR);
}
- /* kill threads related to this sdev */
- if (vdev->ud.tcp_rx) {
- kthread_stop_put(vdev->ud.tcp_rx);
- vdev->ud.tcp_rx = NULL;
+ /* kill threads related to this vdev */
+ if (tcp_rx) {
+ pr_info("%s: stop rx thread\n", __func__);
+ kthread_stop_put(tcp_rx);
}
- if (vdev->ud.tcp_tx) {
- kthread_stop_put(vdev->ud.tcp_tx);
- vdev->ud.tcp_tx = NULL;
+ if (tcp_tx) {
+ pr_info("%s: stop tx thread\n", __func__);
+ kthread_stop_put(tcp_tx);
}
- pr_info("stop threads\n");
+ spin_lock_irqsave(&ud->lock, flags);
/* active connection is closed */
- if (vdev->ud.tcp_socket) {
+ if (ud->tcp_socket) {
+ vdev->ud.tcp_rx = NULL;
+ vdev->ud.tcp_tx = NULL;
sockfd_put(vdev->ud.tcp_socket);
vdev->ud.tcp_socket = NULL;
vdev->ud.sockfd = -1;
}
- pr_info("release socket\n");
+ vdev->ud.in_disconnect = false;
+ spin_unlock_irqrestore(&ud->lock, flags);
+ pr_info("%s: release socket\n", __func__);
vhci_device_unlink_cleanup(vdev);
@@ -1057,7 +1080,7 @@ static void vhci_shutdown_connection(struct usbip_device *ud)
*/
rh_port_disconnect(vdev);
- pr_info("disconnect device\n");
+ pr_info("%s: disconnect device\n", __func__);
}
static void vhci_device_reset(struct usbip_device *ud)
@@ -1073,7 +1096,9 @@ static void vhci_device_reset(struct usbip_device *ud)
usb_put_dev(vdev->udev);
vdev->udev = NULL;
- if (ud->tcp_socket) {
+ if (!vdev->ud.in_disconnect && ud->tcp_socket) {
+ pr_info("%s: release tcp_socket %d\n",
+ __func__, ud->sockfd);
sockfd_put(ud->tcp_socket);
ud->tcp_socket = NULL;
ud->sockfd = -1;
diff --git a/drivers/usb/usbip/vhci_sysfs.c b/drivers/usb/usbip/vhci_sysfs.c
index c4b4256e5dad..c4457026d5ad 100644
--- a/drivers/usb/usbip/vhci_sysfs.c
+++ b/drivers/usb/usbip/vhci_sysfs.c
@@ -412,6 +412,10 @@ static ssize_t attach_store(struct device *dev, struct device_attribute *attr,
vdev->ud.tcp_rx = tcp_rx;
vdev->ud.tcp_tx = tcp_tx;
vdev->ud.status = VDEV_ST_NOTASSIGNED;
+
+ /* used to run socket shutdown just once */
+ vdev->ud.in_disconnect = false;
+
usbip_kcov_handle_init(&vdev->ud);
spin_unlock(&vdev->ud.lock);
--
2.27.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] usbip: fix vhci races in connection tear down
2021-03-12 2:27 [PATCH] usbip: fix vhci races in connection tear down Shuah Khan
@ 2021-03-12 10:45 ` Johan Hovold
2021-03-12 18:43 ` Shuah Khan
[not found] ` <20210312070806.383-1-hdanton@sina.com>
1 sibling, 1 reply; 4+ messages in thread
From: Johan Hovold @ 2021-03-12 10:45 UTC (permalink / raw)
To: Shuah Khan
Cc: valentina.manea.m, shuah, gregkh, linux-usb, linux-kernel,
syzbot+a93fba6d384346a761e3
On Thu, Mar 11, 2021 at 07:27:37PM -0700, Shuah Khan wrote:
> vhci_shutdown_connection() references connection state (tcp_socket,
> tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the
> lock.
>
> Current connection tear down sequence:
> Step 1: shutdown the socket
> Step 2: stop rx thread and reset tcp_rx pointer
> Step 3: stop tx thread and reset tcp_tx pointer
> Step 4: Reset tcp_socket and sockfd
>
> There are several race windows between these steps. In addition, device
> reset routine (vhci_device_reset) resets tcp_socket and sockfd holding
> the lock.
>
> Fix these races:
> - Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs
> only once.
> - Change attach_store() to initialize in_disconnect to false while
> initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd)
> - Change vhci_shutdown_connection() to check in_disconnect and bail
> out if disconnect is in progress.
> - Change vhci_shutdown_connection() to
> -- hold lock to save connection state pointers and unlock.
> -- Shutdown the socket and stop threads.
> -- Hold lock to clear connection status and in_disconnect flag.
> - Change vhci_device_reset() to reset tcp_socket and sockfd.
> if !in_disconnect
>
> Tested syzbot and the reproducer did not trigger any issue.
>
> Reported-and-tested-by: syzbot+a93fba6d384346a761e3@syzkaller.appspotmail.com
> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
> ---
> drivers/usb/usbip/usbip_common.h | 1 +
> drivers/usb/usbip/vhci_hcd.c | 55 +++++++++++++++++++++++---------
> drivers/usb/usbip/vhci_sysfs.c | 4 +++
> 3 files changed, 45 insertions(+), 15 deletions(-)
> diff --git a/drivers/usb/usbip/vhci_hcd.c b/drivers/usb/usbip/vhci_hcd.c
> index 3209b5ddd30c..c1917efe5737 100644
> --- a/drivers/usb/usbip/vhci_hcd.c
> +++ b/drivers/usb/usbip/vhci_hcd.c
> @@ -1007,31 +1007,54 @@ static void vhci_device_unlink_cleanup(struct vhci_device *vdev)
> static void vhci_shutdown_connection(struct usbip_device *ud)
> {
> struct vhci_device *vdev = container_of(ud, struct vhci_device, ud);
> + unsigned long flags;
> + struct socket *socket;
> + struct task_struct *tcp_rx = NULL;
> + struct task_struct *tcp_tx = NULL;
> + int sockfd = 0;
> +
> + spin_lock_irqsave(&ud->lock, flags);
> + if (vdev->ud.in_disconnect) {
> + pr_info("%s: Disconnect in progress for sockfd %d\n",
> + __func__, ud->sockfd);
Looks like you forgot to remove all you debug printks like this one
before submitting.
> + spin_unlock_irqrestore(&ud->lock, flags);
> + return;
> + }
Johan
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] usbip: fix vhci races in connection tear down
2021-03-12 10:45 ` Johan Hovold
@ 2021-03-12 18:43 ` Shuah Khan
0 siblings, 0 replies; 4+ messages in thread
From: Shuah Khan @ 2021-03-12 18:43 UTC (permalink / raw)
To: Johan Hovold
Cc: valentina.manea.m, shuah, gregkh, linux-usb, linux-kernel,
syzbot+a93fba6d384346a761e3, Shuah Khan
On 3/12/21 3:45 AM, Johan Hovold wrote:
> On Thu, Mar 11, 2021 at 07:27:37PM -0700, Shuah Khan wrote:
>> vhci_shutdown_connection() references connection state (tcp_socket,
>> tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the
>> lock.
>>
>> Current connection tear down sequence:
>> Step 1: shutdown the socket
>> Step 2: stop rx thread and reset tcp_rx pointer
>> Step 3: stop tx thread and reset tcp_tx pointer
>> Step 4: Reset tcp_socket and sockfd
>>
>> There are several race windows between these steps. In addition, device
>> reset routine (vhci_device_reset) resets tcp_socket and sockfd holding
>> the lock.
>>
>> Fix these races:
>> - Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs
>> only once.
>> - Change attach_store() to initialize in_disconnect to false while
>> initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd)
>> - Change vhci_shutdown_connection() to check in_disconnect and bail
>> out if disconnect is in progress.
>> - Change vhci_shutdown_connection() to
>> -- hold lock to save connection state pointers and unlock.
>> -- Shutdown the socket and stop threads.
>> -- Hold lock to clear connection status and in_disconnect flag.
>> - Change vhci_device_reset() to reset tcp_socket and sockfd.
>> if !in_disconnect
>>
>> Tested syzbot and the reproducer did not trigger any issue.
>>
>> Reported-and-tested-by: syzbot+a93fba6d384346a761e3@syzkaller.appspotmail.com
>> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
>> ---
>> drivers/usb/usbip/usbip_common.h | 1 +
>> drivers/usb/usbip/vhci_hcd.c | 55 +++++++++++++++++++++++---------
>> drivers/usb/usbip/vhci_sysfs.c | 4 +++
>> 3 files changed, 45 insertions(+), 15 deletions(-)
>
>> diff --git a/drivers/usb/usbip/vhci_hcd.c b/drivers/usb/usbip/vhci_hcd.c
>> index 3209b5ddd30c..c1917efe5737 100644
>> --- a/drivers/usb/usbip/vhci_hcd.c
>> +++ b/drivers/usb/usbip/vhci_hcd.c
>> @@ -1007,31 +1007,54 @@ static void vhci_device_unlink_cleanup(struct vhci_device *vdev)
>> static void vhci_shutdown_connection(struct usbip_device *ud)
>> {
>> struct vhci_device *vdev = container_of(ud, struct vhci_device, ud);
>> + unsigned long flags;
>> + struct socket *socket;
>> + struct task_struct *tcp_rx = NULL;
>> + struct task_struct *tcp_tx = NULL;
>> + int sockfd = 0;
>> +
>> + spin_lock_irqsave(&ud->lock, flags);
>> + if (vdev->ud.in_disconnect) {
>> + pr_info("%s: Disconnect in progress for sockfd %d\n",
>> + __func__, ud->sockfd);
>
> Looks like you forgot to remove all you debug printks like this one
> before submitting.
>
Some printks were already in there and helped with debug. Yes I added
a few more when I submitted for syzbot testing.
I will clean them up i v2.
thanks,
-- Shuah
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] usbip: fix vhci races in connection tear down
[not found] ` <20210312070806.383-1-hdanton@sina.com>
@ 2021-03-12 20:42 ` Shuah Khan
0 siblings, 0 replies; 4+ messages in thread
From: Shuah Khan @ 2021-03-12 20:42 UTC (permalink / raw)
To: Hillf Danton
Cc: valentina.manea.m, shuah, gregkh, linux-usb, linux-kernel,
syzbot+a93fba6d384346a761e3, Shuah Khan
On 3/12/21 12:08 AM, Hillf Danton wrote:
> On Thu, 11 Mar 2021 19:27:37 -0700 Shuah Khan wrote:
>> vhci_shutdown_connection() references connection state (tcp_socket,
>> tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the
>> lock.
>>
>> Current connection tear down sequence:
>> Step 1: shutdown the socket
>> Step 2: stop rx thread and reset tcp_rx pointer
>> Step 3: stop tx thread and reset tcp_tx pointer
>> Step 4: Reset tcp_socket and sockfd
>>
>> There are several race windows between these steps. In addition, device
>> reset routine (vhci_device_reset) resets tcp_socket and sockfd holding
>> the lock.
>
> Can you specify the scenario where reset runs in race with teardown as
> both are parts of usbip_work on a singlethread workqueue?
>>
Hmm. I can't think of one. I was concerned about any async paths that
potentially interfere with shutdown. With vhci_shutdown_connection()
being so relaxed with locking, this is a cautious approach on my part.
I am also keeping in mind that this problem shows up in a limited
scope fuzzing test that doesn't trigger any other normal paths that
would be active if there is real device on the other side.
As for the tcp_socket check in the reset routine, I am not positive
what purpose it serves. I introduced the in_disconnect flag so
shutdown and reset don't collide, in case I am missing some scenario
in the normal path when we actually have a actual device attached.
With the other locking and error path problems in addressed, both
shutdown and reset could be made simpler.
In any case, I think in_disconnect might be too big a hammer. I will
redo the patch without it and also remove tcp_socket handling from
the reset routine. I don't see USBIP_EH_RESET getting set without
USBIP_EH_SHUTDOWN.
thanks,
-- Shuah
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-03-12 20:43 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-12 2:27 [PATCH] usbip: fix vhci races in connection tear down Shuah Khan
2021-03-12 10:45 ` Johan Hovold
2021-03-12 18:43 ` Shuah Khan
[not found] ` <20210312070806.383-1-hdanton@sina.com>
2021-03-12 20:42 ` Shuah Khan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).