From: Bart Van Assche <bvanassche@acm.org>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Sagi Grimberg <sagi@grimberg.me>,
Yamin Friedman <yaminf@mellanox.com>,
kernel test robot <rong.a.chen@intel.com>,
Max Gurtovoy <maxg@mellanox.com>,
LKML <linux-kernel@vger.kernel.org>,
Doug Ledford <dledford@redhat.com>,
Jason Gunthorpe <jgg+lists@ziepe.ca>,
linux-rdma@vger.kernel.org, lkp@lists.01.org
Subject: Re: [IB/srpt] c804af2c1d: last_state.test.blktests.exit_code.143
Date: Sun, 6 Sep 2020 19:58:12 -0700 [thread overview]
Message-ID: <2a2ff3a5-f58e-8246-fd09-87029b562347@acm.org> (raw)
In-Reply-To: <20200824133019.GH1152540@nvidia.com>
On 2020-08-24 06:30, Jason Gunthorpe wrote:
> On Sun, Aug 23, 2020 at 02:18:41PM -0700, Bart Van Assche wrote:>
>> The patch below is sufficient to unbreak blktests. I think that the
>> deadlock while unloading rdma_rxe happens because the RDMA core waits for
>> all ib_dev references to be dropped before dealloc_driver is called.
>
> Which is required, yes
>
>> The rdma_rxe dealloc_driver implementation drops an ib_dev
>> reference.
>
> Where does it do that? I didn't notice it?
That last statement was wrong.
Anyway, with the following debugging patch applied:
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index c36b4d2b61e0..b976dd30f727 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -332,6 +332,8 @@ struct ib_device *ib_device_get_by_index(const struct net *net, u32 index)
*/
void ib_device_put(struct ib_device *device)
{
+ WARN(device->warn_on_refcount_drop, "%s refcnt = %d\n",
+ dev_name(&device->dev), refcount_read(&device->refcount));
if (refcount_dec_and_test(&device->refcount))
complete(&device->unreg_completion);
}
@@ -1287,7 +1289,10 @@ static void disable_device(struct ib_device *device)
/* Pairs with refcount_set in enable_device */
ib_device_put(device);
+ device->warn_on_refcount_drop = true;
+#if 0
wait_for_completion(&device->unreg_completion);
+#endif
/*
* compat devices must be removed after device refcount drops to zero.
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index c868609a4ffa..2d050e3ee55a 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2704,6 +2704,7 @@ struct ib_device {
* registered and cannot be unregistered.
*/
refcount_t refcount;
+ bool warn_on_refcount_drop;
struct completion unreg_completion;
struct work_struct unregistration_work;
The following appeared:
WARNING: CPU: 5 PID: 1760 at drivers/infiniband/core/device.c:335 ib_device_put+0xf2/0x100 [ib_core]
Call Trace:
rxe_elem_release+0x76/0x90 [rdma_rxe]
rxe_destroy_cq+0x4f/0x70 [rdma_rxe]
ib_free_cq_user+0x12b/0x2b0 [ib_core]
ib_cq_pool_destroy+0xa8/0x140 [ib_core]
__ib_unregister_device+0x9c/0x1c0 [ib_core]
ib_unregister_driver+0x181/0x1a0 [ib_core]
rxe_module_exit+0x31/0x50 [rdma_rxe]
__x64_sys_delete_module+0x22a/0x310
do_syscall_64+0x36/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xa9
Do you agree that the above proves that the hang-on-unload is a
regression that has been introduced by the cq-pool patches? Is the patch
below a good way to fix this?
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -1454,8 +1454,8 @@ static void __ib_unregister_device(struct ib_device *ib_dev)
if (!refcount_read(&ib_dev->refcount))
goto out;
- disable_device(ib_dev);
ib_cq_pool_destroy(ib_dev);
+ disable_device(ib_dev);
/* Expedite removing unregistered pointers from the hash table */
free_netdevs(ib_dev);
Thanks,
Bart.
next prev parent reply other threads:[~2020-09-07 2:58 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-02 6:09 [IB/srpt] c804af2c1d: last_state.test.blktests.exit_code.143 kernel test robot
2020-08-02 15:05 ` Bart Van Assche
2020-08-03 6:19 ` Yamin Friedman
2020-08-03 7:27 ` Sagi Grimberg
2020-08-23 21:18 ` Bart Van Assche
2020-08-24 13:30 ` Jason Gunthorpe
2020-09-07 2:58 ` Bart Van Assche [this message]
2020-09-08 18:22 ` Jason Gunthorpe
2020-09-09 2:01 ` Bart Van Assche
2020-09-11 22:00 ` Bart Van Assche
2020-09-12 4:46 ` Yi Zhang
2020-09-14 18:26 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2a2ff3a5-f58e-8246-fd09-87029b562347@acm.org \
--to=bvanassche@acm.org \
--cc=dledford@redhat.com \
--cc=jgg+lists@ziepe.ca \
--cc=jgg@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=lkp@lists.01.org \
--cc=maxg@mellanox.com \
--cc=rong.a.chen@intel.com \
--cc=sagi@grimberg.me \
--cc=yaminf@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).