From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756907AbcIHVNS (ORCPT ); Thu, 8 Sep 2016 17:13:18 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:42272 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757833AbcIHVNH (ORCPT ); Thu, 8 Sep 2016 17:13:07 -0400 From: Josef Bacik To: , , , , Subject: [PATCH 4/5] nbd: allow block mq to deal with timeouts Date: Thu, 8 Sep 2016 17:12:09 -0400 Message-ID: <1473369130-22986-5-git-send-email-jbacik@fb.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1473369130-22986-1-git-send-email-jbacik@fb.com> References: <1473369130-22986-1-git-send-email-jbacik@fb.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [2620:10d:c091:180::1:8c0f] X-ClientProxiedBy: YQBPR01CA0065.CANPRD01.PROD.OUTLOOK.COM (10.169.139.33) To MWHPR15MB1328.namprd15.prod.outlook.com (10.175.3.142) X-MS-Office365-Filtering-Correlation-Id: 945ff590-35f2-45fa-812f-08d3d82cdc13 X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1328;2:+a9sn8povDi/+vzWhGXKFpQsVziL0e/6fjDglA9ugBh7kggwi8vOGj+pt3SvcgwQ/geNbJ4ebB+Fp+FZuK9QeR4CSW+xoIjsbXyZq2RrpoUm4OyhnsB/tz9mQpu93CgrQjsgPK1lWIl8vdS/KBVWTSh7+XMcRcan7c8Zlmda8W1ZMFjIouZM0FXG+5na7hVB;3:ZFPGdvIgtZtqoKNVTVhHp2Mtu/QktYtmEAjU3yS5xTfzE0ohOq11SV5BDcDZpr4VH0qLcRgHkFWhahwqqKjQRMKhTGqSpmegSiXiQNXUzjs037xnBdLtKoBy/fhWBK4k X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1328; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1328;25:ttOf1ZYXVXpzT8WzuznYQlOPLNfc4if2gpYgkCbuW1wLKCW7qTPzCTkUsXueev2dP4rEuczzefdMdO/gWkpXaGRZakckemacyxX6gUMjKuOQXXtCU9+Rfomrlc4yVpJ9Ur5qz67SAiN7vRzmys+qtGY+cYyqJEAIXW/Jq8ESGxAhAFVddaVXmaiW0FgXB8U77I0Rf80X4ljgDg4IqHVTVUkcxMhYawt9b01pLj6rG7wmlOybxOT7xiHCcydJisuBHVP6+ax5S5EwEoeIOb5OLNiHe3ZN9qEYwVA40AVr3XZ6hYNc2iPhECPgjKGE1mrOZHPOHJUBSiUetwEphbJitYnt/qMh+G3KFh1VKhXFx5zDGyntjrfE3sXnDd6dD2ss7PYtWeSJuH62kJP15b9GofjweWoRIIhwB03RG4610hUYjuFmr3InN60S7BN1FvDNU3RiQgxsuDlHfyom5Z+LfkKoLszu6t0lOgSRysyzLZk83FP2yYn5NCGJFscJmvbqF1710dxAmEzhx0Cn6LasAqlvA7U1X3X6gWaHQPVbTDJfaJu1A4wCLIuq0mo+i/NYSBgeU5ubIX/6YABQ5cELlAIe2ZRKAYDc5P5n6UvCk2E+6FGVpgPYc6dEywh/MIAIQj68K7V9izFt99bTosoZBtpdK5cNhSN2zHk0V7t9jISsRB4cF0YKaRMCYEIy8Kb1bBPJn+BXtxxem8YPhxz3trxXBh1JCTohepSH1MEy6yo= X-LD-Processed: 8ae927fe-1255-47a7-a2af-5f3a069daaa2,ExtAddr X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1328;31:7/uqd55iWUYPVC1wFRtBpkXzczgBzwnwCHTMJjQ7du05vaEYqWCmn/ulNMoM5vCf/Wpb5MRWZXjw+rL5KQGV2WIAXaqxCDRz9e5LzO+Xt31hoM0AJnIDpZpqR94mWqwkAII/zRdqw0w7SipPDEJ/0iWC8jHgE+2oFzV+0GiRJaXmgbSvYL0fJlJOAm5140PGU62PLrOUn0Z1g4Xrc8bOKA4fHSD5T4QsHgg54AJdYE0=;20:/QzksBmBbbbgor4koFsAMjJ2gYdHc1BQSRQwaOfkJimbldVCwc2v8GlLZ3RTYRRP4FsDncCWCpGSbQwiSbz5BU7pnU60fSoW4BQ1NRweJCajy91ehHX6T/eseFLhYflA5RaT5LyAJwylR/B/vUpNwLBHJz+cbOAiIyh9VbaqXo0=;4:jyB4xH9qQMTNGiCHtz5KRKqcL6pTSw5EdPCnii0RnTEcbcETfEX2wqeYehzdZ9scXQCMJ7iQD4YeqO/C77RuSHoy93a52UTrHAg1ev6h+15uX6WmhgC9cU6ADWGyKJmib48P9xqiXMQTpr55ezKg0j62dEKaEukCsci4CEuQsvLV8H9BLuLlvTCyM4tF3kxQ6HFtd96R1pvp0bAG7aQ8SDuiRCCfZ4unjlq6zFKq3My+sTwqtG9BZQeShhDs3PPudn7jwX5u/EuIz6XSEK1aXf/4j1TEh6O6x6WhmZnwpbKjFvmSKEnHf+Yxb4zmbSkmiZrlBYZVvR2hALJcwhaYPwI3s+6XZZi9sPlkrCNmp036n17enF8Fl3CFUwORL21sUrqb2OEANgk+xdh5DiCzO1ULDqFXws86QpKzfKmoc9cbamJrKPTqEF23cplJ3Qk/ X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(67672495146484); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046);SRVR:MWHPR15MB1328;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1328; X-Forefront-PRVS: 00594E8DBA X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6069001)(7916002)(45904002)(199003)(189002)(189998001)(19580405001)(92566002)(68736007)(107886002)(19580395003)(36756003)(6116002)(8676002)(5003940100001)(77096005)(76506005)(101416001)(7736002)(33646002)(586003)(76176999)(305945005)(50986999)(5660300001)(7846002)(47776003)(50466002)(97736004)(48376002)(2906002)(229853001)(50226002)(2950100001)(106356001)(42186005)(105586002)(2201001)(86362001)(5001770100001)(81166006)(81156014)(3826002);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR15MB1328;H:localhost;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;MWHPR15MB1328;23:n3guEQv2A8MpHHAhCd1FfVvMnoRsPri6bLTOGYx8U?= =?us-ascii?Q?HAIWJVJBbcVXmXKcC/QjOiQzpcsDQScdJ7Gl0ZcnnCEzaqGcOxgpRD/g7cu/?= =?us-ascii?Q?EPUDDE+rrBLjZjUhYosj0L8faoEWuH/4zS8mNXM8iRNlC1V5A5K8Os3HelEH?= =?us-ascii?Q?ySqiBWgdlzWGUPYJY1OmLCO4umvu1MKaT+Xlbn9nxsT8vOvCwUV9c9ncFLyL?= =?us-ascii?Q?9eexPXi0U9xAhOZQv8Ycyp6CfiXdvEe3XopkGWR+aCz9LfzHFNnmLBcASX+q?= =?us-ascii?Q?ZW9iDZnpJwDc0i3FI4I99Do63vn+Omq5/DAwVJuL5rtEMRP9XJ5jfGlCq7p7?= =?us-ascii?Q?cudOzkDJnk0uW8/gaQvxBT6cD8IxAqJWH6p9V9VuGkPJqC+NCG28dGyPXtci?= =?us-ascii?Q?TgIYPYp2FY12Kqit5ZY1kgTSbBwqEq1GnV4mrk1i3EEVav4xGeyvLmJg/5C+?= =?us-ascii?Q?bDmzFbwp2taVfAgbboET8aj4daV1+d9ZvXi9PmrU/zKbxwnZDXgTXOikH41q?= =?us-ascii?Q?L1ZutL8JqnWHSy4jmmuztNYFyc7sZwKhCOS+bf6HDr6oe4fq27SUCBfC9Giu?= =?us-ascii?Q?7MqsXDdpRf8ttbkvggDvJCBz+vzNX68mkKINr1L5y7FvwOWfjTT3OnR8PisL?= =?us-ascii?Q?9JOzCvf+flbV857FDb3rfKbHBj0n6LzwbhWA5l0fySmhhysA86JuipL1i0ZN?= =?us-ascii?Q?pERGU93o5uAu3i/OTmfM4x8//5vgWGPZ4M9iESTHXAsz0fyIW7P0rlVoLlQB?= =?us-ascii?Q?eAv4vbbeuByR30LA2V1crx4lW94NQAHiyGr/21goxGJv8AcHmDETATs9c/QI?= =?us-ascii?Q?ujFBa2ELsGRV83fAf4EM6eow05jbwBhm+1JrfpfyOI5wdU53PhasciBlk3h+?= =?us-ascii?Q?J0Gep90vP8ezNPCZrfkmQgQKLBFVpWuh38e6lhLZnJ06Gad89m2V8EV/4l5G?= =?us-ascii?Q?HnWAwKExaGha/uXhYFOMlgjax1hAmG7taENHcwALCicS5Ivc6PElau+YA9c0?= =?us-ascii?Q?XBSEplnqaWfuWJi97m146kz9B94c+c0WdqFuWXmv4cp2hOKZ9mjDvxVJnFHI?= =?us-ascii?Q?4+4+5jLka1BpCA6/6kL0/4nEsrur9dN79mhbpFyKAwsizTSYeBUIupezWpcK?= =?us-ascii?Q?4InBzam2NgkgagErlVbHvoQ2wq51klj?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1328;6:XatsPwA54aVg2hpEPyojN6MYN5zyUcMtpJsWXsh5JiirKTQCwkNtRXlIjLhGKIbqnYcRPDbpyM90s1zJnQy83waBsRqRLVyl+nmrie3jJM1/efb2/9Qr/EiSdIOTxVVk08FXhTf3E8NRQN8rz06iEsebRCYiRbg9xYEr1uUoK72R8NDKaPUmd0kZw/fYowa/JfYPDVVQ/2Of5RT0mBFDcSMCbf8JcbscW5cAaCjHkqSlNAYTEe452hOGbYZP2Lvx1WHI3LUI6QyAI50VQ9fdGghfRZ65E7gaWTfRuSOVrtM=;5:/01TTzwZUZKwBui0Y1LHRXcFMfUSjQgcCzriy6o37j3++NoVXxPbLAK3Bd/VU6i+St9l+jSqs72qQscNZdhvkgbSAbKFWIzBZYE0ZEpRS5A1SOmDqndYp24NWjjF9t4bV2f8jltnBJkPfYWS4nPdjA==;24:5XKrk3mxup1ukfDK6WWzUOD2l+SVHbHGqKgSi9885Ry1rp29V5vH77tWrnb2DgnKxJJfLDzb4qqO+g+hJ3MdhFAwv4pQfP8EN/iwLicXgrk=;7:XLIPOe2qLW/bZX6wqg0T2FGzViMVsG28kJO6Uyb6Lpz2T5YY3n9fHM01HwD4hXHx02m/rZtjfn4/Uz6+4gkQVgLK1GwlRwH4Rq8vTqGwx8mhR9stTu7h35ANL8Uwl08O1IfC04thIgsrGc87gWiHYUdO18w5kR5S2LD+6zkMKy/aVqwlCKPGNuVDHHC+Ur7qhRHDrDo/wHcNSqh/96Z1H1ZUrbmyAxegei5vdvfzENsIyjArx/9BIpuuopgqcyF5 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1328;20:5H8ggt1TpcVa1k1svW+bwWSdDPF2xSjP34PIEnyE/Ck7vPTfzx6lGQ/jtJZcXSjt17Kc3Fdsycw/WIykCcxHw37O89m4jaD30Ip4n5jzsSdTd5ved4Qy3SuQ4XaYMMIZu/5v0VSSOIZ58RCdw0iUoKrZBYWmxo7cMH8nFG1bUzw= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2016 21:12:36.7644 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1328 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-09-08_10:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Instead of rolling our own timer, just utilize the blk mq req timeout and do the disconnect if any of our commands timeout. Signed-off-by: Josef Bacik --- drivers/block/nbd.c | 51 ++++++++++++++------------------------------------- 1 file changed, 14 insertions(+), 37 deletions(-) diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index cf855a1..4c6dd1a 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -50,16 +50,13 @@ struct nbd_device { struct socket * sock; /* If == NULL, device is not ready, yet */ int magic; - atomic_t outstanding_cmds; struct blk_mq_tag_set tag_set; struct mutex tx_lock; struct gendisk *disk; int blksize; loff_t bytesize; - int xmit_timeout; - struct timer_list timeout_timer; /* protects initialization and shutdown of the socket */ spinlock_t sock_lock; struct task_struct *task_recv; @@ -154,7 +151,6 @@ static void nbd_end_request(struct nbd_cmd *cmd) dev_dbg(nbd_to_dev(nbd), "request %p: %s\n", cmd, error ? "failed" : "done"); - atomic_dec(&nbd->outstanding_cmds); blk_mq_complete_request(req, error); } @@ -165,7 +161,7 @@ static void sock_shutdown(struct nbd_device *nbd) { struct socket *sock; - spin_lock_irq(&nbd->sock_lock); + spin_lock(&nbd->sock_lock); if (!nbd->sock) { spin_unlock_irq(&nbd->sock_lock); @@ -175,24 +171,20 @@ static void sock_shutdown(struct nbd_device *nbd) sock = nbd->sock; dev_warn(disk_to_dev(nbd->disk), "shutting down socket\n"); nbd->sock = NULL; - spin_unlock_irq(&nbd->sock_lock); + spin_unlock(&nbd->sock_lock); kernel_sock_shutdown(sock, SHUT_RDWR); sockfd_put(sock); - - del_timer(&nbd->timeout_timer); } -static void nbd_xmit_timeout(unsigned long arg) +static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, + bool reserved) { - struct nbd_device *nbd = (struct nbd_device *)arg; + struct nbd_cmd *cmd = blk_mq_rq_to_pdu(req); + struct nbd_device *nbd = cmd->nbd; struct socket *sock = NULL; - unsigned long flags; - - if (!atomic_read(&nbd->outstanding_cmds)) - return; - spin_lock_irqsave(&nbd->sock_lock, flags); + spin_lock(&nbd->sock_lock); set_bit(NBD_TIMEDOUT, &nbd->runtime_flags); @@ -201,13 +193,15 @@ static void nbd_xmit_timeout(unsigned long arg) get_file(sock->file); } - spin_unlock_irqrestore(&nbd->sock_lock, flags); + spin_unlock(&nbd->sock_lock); if (sock) { kernel_sock_shutdown(sock, SHUT_RDWR); sockfd_put(sock); } + req->errors++; dev_err(nbd_to_dev(nbd), "Connection timed out, shutting down connection\n"); + return BLK_EH_HANDLED; } /* @@ -257,9 +251,6 @@ static int sock_xmit(struct nbd_device *nbd, int send, void *buf, int size, tsk_restore_flags(current, pflags, PF_MEMALLOC); - if (!send && nbd->xmit_timeout) - mod_timer(&nbd->timeout_timer, jiffies + nbd->xmit_timeout); - return result; } @@ -512,10 +503,6 @@ static void nbd_handle_cmd(struct nbd_cmd *cmd) goto error_out; } - if (nbd->xmit_timeout && !atomic_read(&nbd->outstanding_cmds)) - mod_timer(&nbd->timeout_timer, jiffies + nbd->xmit_timeout); - - atomic_inc(&nbd->outstanding_cmds); if (nbd_send_cmd(nbd, cmd) != 0) { dev_err(disk_to_dev(nbd->disk), "Request send failed\n"); req->errors++; @@ -569,9 +556,8 @@ static void nbd_reset(struct nbd_device *nbd) nbd->bytesize = 0; set_capacity(nbd->disk, 0); nbd->flags = 0; - nbd->xmit_timeout = 0; + nbd->tag_set.timeout = 0; queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, nbd->disk->queue); - del_timer_sync(&nbd->timeout_timer); } static void nbd_bdev_reset(struct block_device *bdev) @@ -668,13 +654,7 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, return nbd_size_set(nbd, bdev, nbd->blksize, arg); case NBD_SET_TIMEOUT: - nbd->xmit_timeout = arg * HZ; - if (arg) - mod_timer(&nbd->timeout_timer, - jiffies + nbd->xmit_timeout); - else - del_timer_sync(&nbd->timeout_timer); - + nbd->tag_set.timeout = arg * HZ; return 0; case NBD_SET_FLAGS: @@ -836,7 +816,7 @@ static int nbd_dev_dbg_init(struct nbd_device *nbd) debugfs_create_file("tasks", 0444, dir, nbd, &nbd_dbg_tasks_ops); debugfs_create_u64("size_bytes", 0444, dir, &nbd->bytesize); - debugfs_create_u32("timeout", 0444, dir, &nbd->xmit_timeout); + debugfs_create_u32("timeout", 0444, dir, &nbd->tag_set.timeout); debugfs_create_u32("blocksize", 0444, dir, &nbd->blksize); debugfs_create_file("flags", 0444, dir, nbd, &nbd_dbg_flags_ops); @@ -903,6 +883,7 @@ static struct blk_mq_ops nbd_mq_ops = { .queue_rq = nbd_queue_rq, .map_queue = blk_mq_map_queue, .init_request = nbd_init_request, + .timeout = nbd_xmit_timeout, }; /* @@ -1007,10 +988,6 @@ static int __init nbd_init(void) nbd_dev[i].magic = NBD_MAGIC; spin_lock_init(&nbd_dev[i].sock_lock); mutex_init(&nbd_dev[i].tx_lock); - init_timer(&nbd_dev[i].timeout_timer); - nbd_dev[i].timeout_timer.function = nbd_xmit_timeout; - nbd_dev[i].timeout_timer.data = (unsigned long)&nbd_dev[i]; - atomic_set(&nbd_dev[i].outstanding_cmds, 0); disk->major = NBD_MAJOR; disk->first_minor = i << part_shift; disk->fops = &nbd_fops; -- 2.8.0.rc2