From: Maksym Planeta <mplaneta@os.inf.tu-dresden.de>
To: Moni Shoua <monis@mellanox.com>,
Doug Ledford <dledford@redhat.com>,
Jason Gunthorpe <jgg@ziepe.ca>,
linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org
Cc: Maksym Planeta <mplaneta@os.inf.tu-dresden.de>
Subject: [PATCH 00/10] Refactor rxe driver to remove multiple race conditions
Date: Mon, 22 Jul 2019 17:14:16 +0200 [thread overview]
Message-ID: <20190722151426.5266-1-mplaneta@os.inf.tu-dresden.de> (raw)
This patchset helps to get rid of following race condition situations:
1. Tasklet functions were incrementing reference counting after entering
running the tasklet.
2. Getting a pointer to reference counted object (kref) was done without
protecting kref_put with a lock.
3. QP cleanup was sometimes scheduling cleanup for later execution in
rxe_qp_do_cleaunpm, although this QP's memory could be freed immediately after
returning from rxe_qp_cleanup.
4. Non-atomic cleanup functions could be called in SoftIRQ context
5. Manipulating with reference counter inside a critical section could have
been done both inside and outside of SoftIRQ region. Such behavior may end up
in a deadlock.
The easiest way to observe these problems is to compile the kernel with KASAN
and lockdep and abruptly stop an application using SoftRoCE during the
communication phase. For my system this often resulted in kernel crash of a
deadlock inside the kernel.
To fix the above mentioned problems, this patch does following things:
1. Replace tasklets with workqueues
2. Adds locks to kref_put
3. Aquires reference counting in an appropriate place
As a shortcomming, the performance is slightly reduced, because instead of
trying to execute tasklet function directly the new version always puts it onto
the queue.
TBH, I'm not sure that I removed all of the problems, but the driver
deffinetely behaves much more stable now. I would be glad to get some
help with additional testing.
drivers/infiniband/sw/rxe/rxe_comp.c | 38 ++----
drivers/infiniband/sw/rxe/rxe_cq.c | 17 ++-
drivers/infiniband/sw/rxe/rxe_hw_counters.c | 1 -
drivers/infiniband/sw/rxe/rxe_hw_counters.h | 1 -
drivers/infiniband/sw/rxe/rxe_loc.h | 3 +-
drivers/infiniband/sw/rxe/rxe_mcast.c | 22 ++--
drivers/infiniband/sw/rxe/rxe_mr.c | 10 +-
drivers/infiniband/sw/rxe/rxe_net.c | 21 ++-
drivers/infiniband/sw/rxe/rxe_pool.c | 40 ++++--
drivers/infiniband/sw/rxe/rxe_pool.h | 16 ++-
drivers/infiniband/sw/rxe/rxe_qp.c | 130 +++++++++---------
drivers/infiniband/sw/rxe/rxe_recv.c | 8 +-
drivers/infiniband/sw/rxe/rxe_req.c | 17 +--
drivers/infiniband/sw/rxe/rxe_resp.c | 54 ++++----
drivers/infiniband/sw/rxe/rxe_task.c | 139 +++++++-------------
drivers/infiniband/sw/rxe/rxe_task.h | 40 ++----
drivers/infiniband/sw/rxe/rxe_verbs.c | 81 ++++++------
drivers/infiniband/sw/rxe/rxe_verbs.h | 8 +-
18 files changed, 302 insertions(+), 344 deletions(-)
--
2.20.1
next reply other threads:[~2019-07-22 15:14 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-22 15:14 Maksym Planeta [this message]
2019-07-22 15:14 ` [PATCH 01/10] Simplify rxe_run_task interface Maksym Planeta
2019-07-22 15:14 ` [PATCH 02/10] Remove counter that does not have a meaning anymore Maksym Planeta
2019-07-22 15:14 ` [PATCH 03/10] Make pool interface more type safe Maksym Planeta
2019-07-22 15:14 ` [PATCH 04/10] Protect kref_put with the lock Maksym Planeta
2019-07-22 15:25 ` Jason Gunthorpe
2019-07-22 15:28 ` Maksym Planeta
2019-07-22 15:14 ` [PATCH 05/10] Fix reference counting for rxe tasklets Maksym Planeta
2019-07-22 15:27 ` Jason Gunthorpe
2019-07-22 15:14 ` [PATCH 06/10] Remove pd form rxe_ah Maksym Planeta
2019-07-22 15:14 ` [PATCH 07/10] Pass the return value of kref_put further Maksym Planeta
2019-07-22 15:29 ` Jason Gunthorpe
2019-07-22 15:31 ` Maksym Planeta
2019-07-22 15:14 ` [PATCH 08/10] Move responsibility of cleaning up pool elements Maksym Planeta
2019-07-22 15:14 ` [PATCH 09/10] Consolidate resetting of QP's tasks into one place Maksym Planeta
2019-07-22 15:14 ` [PATCH 10/10] Replace tasklets with workqueues Maksym Planeta
2019-07-22 15:32 ` Jason Gunthorpe
2019-07-25 14:36 ` Maksym Planeta
2019-07-25 18:50 ` Jason Gunthorpe
2019-07-30 19:20 ` Maksym Planeta
2019-10-11 10:41 ` Maksym Planeta
2019-10-22 19:04 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190722151426.5266-1-mplaneta@os.inf.tu-dresden.de \
--to=mplaneta@os.inf.tu-dresden.de \
--cc=dledford@redhat.com \
--cc=jgg@ziepe.ca \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=monis@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).