Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Yixian Liu <liuyixian@huawei.com>
To: <dledford@redhat.com>, <jgg@ziepe.ca>, <leon@kernel.org>
Cc: <linux-rdma@vger.kernel.org>, <linuxarm@huawei.com>
Subject: [PATCH v5 for-next 0/2] RDMA/hns: Add the workqueue framework for flush cqe handler
Date: Sat, 28 Dec 2019 11:28:53 +0800
Message-ID: <1577503735-26685-1-git-send-email-liuyixian@huawei.com> (raw)

Earlier Background:
HiP08 RoCE hardware lacks ability(a known hardware problem) to flush
outstanding WQEs if QP state gets into errored mode for some reason.
To overcome this hardware problem and as a workaround, when QP is
detected to be in errored state during various legs like post send,
post receive etc [1], flush needs to be performed from the driver.

These data-path legs might get called concurrently from various context,
like thread and interrupt as well (like NVMe driver). Hence, these need
to be protected with spin-locks for the concurrency. This code exists
within the driver.

Problem:
Earlier The patch[1] sent to solve the hardware limitation explained
in the background section had a bug in the software flushing leg. It
acquired mutex while modifying QP state to errored state and while
conveying it to the hardware using the mailbox. This caused leg to
sleep while holding spin-lock and caused crash.

Suggested Solution:
In this patch, we have proposed to defer the flushing of the QP in
Errored state using the workqueue.

We do understand that this might have an impact on the recovery times
as scheduling of the workqueue handler depends upon the occupancy of
the system. Therefore to roughly mitigate this affect we have tried
to use Concurrency Managed workqueue to give worker thread (and
hence handler) a chance to run over more than one core.


[1] https://patchwork.kernel.org/patch/10534271/


This patch-set consists of:
[Patch 001] Introduce workqueue based WQE Flush Handler
[Patch 002] Call WQE flush handler in post {send|receive|poll}

v5 changes:
1. Remove WQ_MEM_RECLAIM flag according to Leon's suggestion.
2. Change to ordered workqueue for the requirement of flush work.

v4 changes:
1. Add flag for PI is being pushed according to Jason's suggestion
   to reduce unnecessary works submitted to workqueue.

v3 changes:
1. Fall back to dynamically allocate flush_work.

v2 changes:
1. Remove new created workqueue according to Jason's comment
2. Remove dynamic allocation for flush_work according to Jason's comment
3. Change current irq singlethread workqueue to concurrency management
   workqueue to ensure work unblocked.

Yixian Liu (2):
  RDMA/hns: Add the workqueue framework for flush cqe handler
  RDMA/hns: Delayed flush cqe process with workqueue

 drivers/infiniband/hw/hns/hns_roce_device.h |  4 ++
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c  | 97 +++++++++++++++--------------
 drivers/infiniband/hw/hns/hns_roce_qp.c     | 45 +++++++++++++
 3 files changed, 98 insertions(+), 48 deletions(-)

-- 
2.7.4


             reply index

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-28  3:28 Yixian Liu [this message]
2019-12-28  3:28 ` [PATCH v5 for-next 1/2] " Yixian Liu
2020-01-10 15:26   ` Jason Gunthorpe
2020-01-11  9:49     ` Liuyixian (Eason)
2020-01-13 11:26     ` Liuyixian (Eason)
2020-01-13 14:04       ` Jason Gunthorpe
2020-01-15  9:36         ` Liuyixian (Eason)
2020-01-15  9:39         ` Liuyixian (Eason)
2019-12-28  3:28 ` [PATCH v5 for-next 2/2] RDMA/hns: Delayed flush cqe process with workqueue Yixian Liu

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1577503735-26685-1-git-send-email-liuyixian@huawei.com \
    --to=liuyixian@huawei.com \
    --cc=dledford@redhat.com \
    --cc=jgg@ziepe.ca \
    --cc=leon@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=linuxarm@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git