[PATCH] nbd: Fix hung on disconnect request if socket is closed before

[PATCH] nbd: Fix hung on disconnect request if socket is closed before

[Xie Yongji]

When userspace closes the socket before sending a disconnect
request, the following I/O requests will be blocked in
wait_for_reconnect() until dead timeout. This will cause the
following disconnect request also hung on blk_mq_quiesce_queue().
That means we have no way to disconnect a nbd device if there
are some I/O requests waiting for reconnecting until dead timeout.
It's not expected. So let's wake up the thread waiting for
reconnecting directly when a disconnect request is sent.

Reported-by: Xu Jianhai <zero.xu@bytedance.com>
Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
---
 drivers/block/nbd.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 5a1f98494ddd..284557041336 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -947,11 +947,15 @@ static int wait_for_reconnect(struct nbd_device *nbd)
 	struct nbd_config *config = nbd->config;
 	if (!config->dead_conn_timeout)
 		return 0;
-	if (test_bit(NBD_RT_DISCONNECTED, &config->runtime_flags))
+
+	if (!wait_event_timeout(config->conn_wait,
+				test_bit(NBD_RT_DISCONNECTED,
+					 &config->runtime_flags) ||
+				atomic_read(&config->live_connections) > 0,
+				config->dead_conn_timeout))
 		return 0;
-	return wait_event_timeout(config->conn_wait,
-				  atomic_read(&config->live_connections) > 0,
-				  config->dead_conn_timeout) > 0;
+
+	return !test_bit(NBD_RT_DISCONNECTED, &config->runtime_flags);
 }
 
 static int nbd_handle_cmd(struct nbd_cmd *cmd, int index)
@@ -2082,6 +2086,7 @@ static void nbd_disconnect_and_put(struct nbd_device *nbd)
 	mutex_lock(&nbd->config_lock);
 	nbd_disconnect(nbd);
 	sock_shutdown(nbd);
+	wake_up(&nbd->config->conn_wait);
 	/*
 	 * Make sure recv thread has finished, we can safely call nbd_clear_que()
 	 * to cancel the inflight I/Os.
-- 
2.20.1

Re: [PATCH] nbd: Fix hung on disconnect request if socket is closed before

[Josef Bacik]

On Tue, Mar 22, 2022 at 04:06:39PM +0800, Xie Yongji wrote:
> When userspace closes the socket before sending a disconnect
> request, the following I/O requests will be blocked in
> wait_for_reconnect() until dead timeout. This will cause the
> following disconnect request also hung on blk_mq_quiesce_queue().
> That means we have no way to disconnect a nbd device if there
> are some I/O requests waiting for reconnecting until dead timeout.
> It's not expected. So let's wake up the thread waiting for
> reconnecting directly when a disconnect request is sent.
> 
> Reported-by: Xu Jianhai <zero.xu@bytedance.com>
> Signed-off-by: Xie Yongji <xieyongji@bytedance.com>

Reviewed-by: Josef Bacik <josef@toxicpanda.com>

Thanks,

Josef

Re: [PATCH] nbd: Fix hung on disconnect request if socket is closed before

[Yongji Xie]

Ping.

On Wed, Mar 23, 2022 at 3:56 AM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Tue, Mar 22, 2022 at 04:06:39PM +0800, Xie Yongji wrote:
> > When userspace closes the socket before sending a disconnect
> > request, the following I/O requests will be blocked in
> > wait_for_reconnect() until dead timeout. This will cause the
> > following disconnect request also hung on blk_mq_quiesce_queue().
> > That means we have no way to disconnect a nbd device if there
> > are some I/O requests waiting for reconnecting until dead timeout.
> > It's not expected. So let's wake up the thread waiting for
> > reconnecting directly when a disconnect request is sent.
> >
> > Reported-by: Xu Jianhai <zero.xu@bytedance.com>
> > Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
>
> Thanks,
>
> Josef

Re: [PATCH] nbd: Fix hung on disconnect request if socket is closed before

[Jens Axboe]

On Tue, 22 Mar 2022 16:06:39 +0800, Xie Yongji wrote:
> When userspace closes the socket before sending a disconnect
> request, the following I/O requests will be blocked in
> wait_for_reconnect() until dead timeout. This will cause the
> following disconnect request also hung on blk_mq_quiesce_queue().
> That means we have no way to disconnect a nbd device if there
> are some I/O requests waiting for reconnecting until dead timeout.
> It's not expected. So let's wake up the thread waiting for
> reconnecting directly when a disconnect request is sent.
> 
> [...]

Applied, thanks!

[1/1] nbd: Fix hung on disconnect request if socket is closed before
      commit: 491bf8f236fdeec698fa6744993f1ecf3fafd1a5

Best regards,
-- 
Jens Axboe