From: Bart Van Assche <bvanassche@acm.org>
To: "Martin K . Petersen" <martin.petersen@oracle.com>
Cc: linux-scsi@vger.kernel.org, Jaegeuk Kim <jaegeuk@kernel.org>,
Bart Van Assche <bvanassche@acm.org>,
Adrian Hunter <adrian.hunter@intel.com>,
Stanley Chu <stanley.chu@mediatek.com>,
Can Guo <cang@codeaurora.org>,
Asutosh Das <asutoshd@codeaurora.org>,
Avri Altman <avri.altman@wdc.com>,
"James E.J. Bottomley" <jejb@linux.ibm.com>,
Matthias Brugger <matthias.bgg@gmail.com>,
Bean Huo <beanhuo@micron.com>,
Kiwoong Kim <kwmad.kim@samsung.com>,
Keoseong Park <keosung.park@samsung.com>
Subject: [PATCH v3 13/18] scsi: ufs: Optimize SCSI command processing
Date: Wed, 21 Jul 2021 20:34:34 -0700 [thread overview]
Message-ID: <20210722033439.26550-14-bvanassche@acm.org> (raw)
In-Reply-To: <20210722033439.26550-1-bvanassche@acm.org>
Use a spinlock to protect hba->outstanding_reqs instead of using atomic
operations to update this member variable.
This patch is a performance improvement because it reduces the number of
atomic operations in the hot path (test_and_clear_bit()) and because it
reduces the lock contention on the SCSI host lock. On my test setup this
patch improves IOPS by about 1%.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Stanley Chu <stanley.chu@mediatek.com>
Cc: Can Guo <cang@codeaurora.org>
Cc: Asutosh Das <asutoshd@codeaurora.org>
Cc: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
drivers/scsi/ufs/ufshcd.c | 29 ++++++++++++++++++-----------
drivers/scsi/ufs/ufshcd.h | 2 ++
2 files changed, 20 insertions(+), 11 deletions(-)
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 436d814f4c1e..a3ad83a3bae0 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2095,12 +2095,14 @@ void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag)
ufshcd_clk_scaling_start_busy(hba);
if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
ufshcd_start_monitor(hba, lrbp);
- spin_lock_irqsave(hba->host->host_lock, flags);
+
+ spin_lock_irqsave(&hba->outstanding_lock, flags);
if (hba->vops && hba->vops->setup_xfer_req)
hba->vops->setup_xfer_req(hba, task_tag, !!lrbp->cmd);
- set_bit(task_tag, &hba->outstanding_reqs);
+ __set_bit(task_tag, &hba->outstanding_reqs);
ufshcd_writel(hba, 1 << task_tag, REG_UTP_TRANSFER_REQ_DOOR_BELL);
- spin_unlock_irqrestore(hba->host->host_lock, flags);
+ spin_unlock_irqrestore(&hba->outstanding_lock, flags);
+
/* Make sure that doorbell is committed immediately */
wmb();
}
@@ -2882,7 +2884,9 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
* we also need to clear the outstanding_request
* field in hba
*/
- clear_bit(lrbp->task_tag, &hba->outstanding_reqs);
+ spin_lock_irqsave(&hba->outstanding_lock, flags);
+ __clear_bit(lrbp->task_tag, &hba->outstanding_reqs);
+ spin_unlock_irqrestore(&hba->outstanding_lock, flags);
}
return err;
@@ -5194,8 +5198,6 @@ static void __ufshcd_transfer_req_compl(struct ufs_hba *hba,
bool update_scaling = false;
for_each_set_bit(index, &completed_reqs, hba->nutrs) {
- if (!test_and_clear_bit(index, &hba->outstanding_reqs))
- continue;
lrbp = &hba->lrb[index];
lrbp->compl_time_stamp = ktime_get();
cmd = lrbp->cmd;
@@ -5250,10 +5252,14 @@ static irqreturn_t ufshcd_transfer_req_compl(struct ufs_hba *hba)
!(hba->quirks & UFSHCI_QUIRK_SKIP_RESET_INTR_AGGR))
ufshcd_reset_intr_aggr(hba);
- spin_lock_irqsave(hba->host->host_lock, flags);
+ spin_lock_irqsave(&hba->outstanding_lock, flags);
tr_doorbell = ufshcd_readl(hba, REG_UTP_TRANSFER_REQ_DOOR_BELL);
- completed_reqs = tr_doorbell ^ hba->outstanding_reqs;
- spin_unlock_irqrestore(hba->host->host_lock, flags);
+ completed_reqs = ~tr_doorbell & hba->outstanding_reqs;
+ WARN_ONCE(completed_reqs & ~hba->outstanding_reqs,
+ "completed: %#lx; outstanding: %#lx\n", completed_reqs,
+ hba->outstanding_reqs);
+ hba->outstanding_reqs &= ~completed_reqs;
+ spin_unlock_irqrestore(&hba->outstanding_lock, flags);
if (completed_reqs) {
__ufshcd_transfer_req_compl(hba, completed_reqs);
@@ -9340,10 +9346,11 @@ int ufshcd_alloc_host(struct device *dev, struct ufs_hba **hba_handle)
hba = shost_priv(host);
hba->host = host;
hba->dev = dev;
- *hba_handle = hba;
hba->dev_ref_clk_freq = REF_CLK_FREQ_INVAL;
-
INIT_LIST_HEAD(&hba->clk_list_head);
+ spin_lock_init(&hba->outstanding_lock);
+
+ *hba_handle = hba;
out_error:
return err;
diff --git a/drivers/scsi/ufs/ufshcd.h b/drivers/scsi/ufs/ufshcd.h
index 6df847facd1d..91b0b278469d 100644
--- a/drivers/scsi/ufs/ufshcd.h
+++ b/drivers/scsi/ufs/ufshcd.h
@@ -695,6 +695,7 @@ struct ufs_hba_monitor {
* @lrb: local reference block
* @cmd_queue: Used to allocate command tags from hba->host->tag_set.
* @outstanding_tasks: Bits representing outstanding task requests
+ * @outstanding_lock: Protects @outstanding_reqs.
* @outstanding_reqs: Bits representing outstanding transfer requests
* @capabilities: UFS Controller Capabilities
* @nutrs: Transfer Request Queue depth supported by controller
@@ -781,6 +782,7 @@ struct ufs_hba {
struct ufshcd_lrb *lrb;
unsigned long outstanding_tasks;
+ spinlock_t outstanding_lock;
unsigned long outstanding_reqs;
u32 capabilities;
next prev parent reply other threads:[~2021-07-22 3:35 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-22 3:34 [PATCH v3 00/18] UFS patches for kernel v5.15 Bart Van Assche
2021-07-22 3:34 ` [PATCH v3 01/18] scsi: ufs: Fix memory corruption by ufshcd_read_desc_param() Bart Van Assche
2021-07-25 12:40 ` Avri Altman
2021-07-22 3:34 ` [PATCH v3 02/18] scsi: ufs: Reduce power management code duplication Bart Van Assche
2021-07-31 14:44 ` Stanley Chu
2021-07-22 3:34 ` [PATCH v3 03/18] scsi: ufs: Only include power management code if necessary Bart Van Assche
2021-07-22 3:34 ` [PATCH v3 04/18] scsi: ufs: Rename the second ufshcd_probe_hba() argument Bart Van Assche
2021-08-02 8:17 ` Stanley Chu
2021-07-22 3:34 ` [PATCH v3 05/18] scsi: ufs: Use DECLARE_COMPLETION_ONSTACK() where appropriate Bart Van Assche
2021-07-22 3:34 ` [PATCH v3 06/18] scsi: ufs: Remove ufshcd_valid_tag() Bart Van Assche
2021-07-22 3:34 ` [PATCH v3 07/18] scsi: ufs: Verify UIC locking requirements at runtime Bart Van Assche
2021-07-22 3:34 ` [PATCH v3 08/18] scsi: ufs: Improve static type checking for the host controller state Bart Van Assche
2021-07-22 3:34 ` [PATCH v3 09/18] scsi: ufs: Remove several wmb() calls Bart Van Assche
2021-07-25 13:20 ` Avri Altman
2021-07-22 3:34 ` [PATCH v3 10/18] scsi: ufs: Inline ufshcd_outstanding_req_clear() Bart Van Assche
2021-07-29 7:42 ` Bean Huo
2021-07-22 3:34 ` [PATCH v3 11/18] scsi: ufs: Revert "Utilize Transfer Request List Completion Notification Register" Bart Van Assche
2021-07-29 8:03 ` Bean Huo
2021-07-29 16:10 ` Bart Van Assche
2021-07-29 16:13 ` Bart Van Assche
2021-07-29 21:14 ` Bean Huo
2021-08-02 15:24 ` Bean Huo
2021-08-03 18:49 ` Bart Van Assche
2021-07-22 3:34 ` [PATCH v3 12/18] scsi: ufs: Optimize serialization of setup_xfer_req() calls Bart Van Assche
2021-07-29 8:07 ` Bean Huo
2021-07-22 3:34 ` Bart Van Assche [this message]
2021-07-29 9:12 ` [PATCH v3 13/18] scsi: ufs: Optimize SCSI command processing Bean Huo
2021-07-29 16:11 ` Bart Van Assche
2021-08-02 12:11 ` Bean Huo
2021-07-22 3:34 ` [PATCH v3 14/18] scsi: ufs: Fix the SCSI abort handler Bart Van Assche
2021-08-02 13:15 ` Bean Huo
2021-07-22 3:34 ` [PATCH v3 15/18] scsi: ufs: Request sense data asynchronously Bart Van Assche
2021-08-02 13:16 ` Bean Huo
2021-07-22 3:34 ` [PATCH v3 16/18] scsi: ufs: Synchronize SCSI and UFS error handling Bart Van Assche
2021-08-02 14:24 ` Bean Huo
2021-08-28 9:47 ` Adrian Hunter
2021-08-29 7:17 ` Avri Altman
2021-08-29 21:33 ` Bart Van Assche
2021-08-29 9:57 ` Adrian Hunter
2021-08-29 22:18 ` Bart Van Assche
2021-08-31 7:24 ` Adrian Hunter
2021-08-31 10:04 ` Adrian Hunter
2021-08-31 17:18 ` Bart Van Assche
2021-09-01 7:42 ` Adrian Hunter
2021-09-01 20:46 ` Bart Van Assche
2021-09-02 6:02 ` Adrian Hunter
2021-07-22 3:34 ` [PATCH v3 17/18] scsi: ufs: Retry aborted SCSI commands instead of completing these successfully Bart Van Assche
2021-08-02 15:03 ` Bean Huo
2021-07-22 3:34 ` [PATCH v3 18/18] scsi: ufs: Add fault injection support Bart Van Assche
2021-08-02 15:03 ` Bean Huo
[not found] ` <CGME20210722033524epcas2p31e41c1db6883aaa644edf23bbe8a1ca2@epcms2p4>
2021-07-28 6:48 ` [PATCH v3 06/18] scsi: ufs: Remove ufshcd_valid_tag() Daejun Park
2021-07-28 22:48 ` Bart Van Assche
[not found] ` <CGME20210722033524epcas2p31e41c1db6883aaa644edf23bbe8a1ca2@epcms2p7>
2021-07-29 0:26 ` Daejun Park
[not found] ` <CGME20210722033530epcas2p4c76293e5fc5163fed3995acdd02678ce@epcms2p1>
2021-07-28 7:56 ` [PATCH v3 08/18] scsi: ufs: Improve static type checking for the host controller state Keoseong Park
[not found] ` <CGME20210722033504epcas2p1cc3c6f61e81814004c36b89c7c9e3dd5@epcms2p5>
2021-07-29 0:56 ` [PATCH v3 01/18] scsi: ufs: Fix memory corruption by ufshcd_read_desc_param() Daejun Park
[not found] ` <CGME20210722033510epcas2p410be4f2f387e98babeefc754a9fc1414@epcms2p2>
2021-07-29 0:56 ` [PATCH v3 02/18] scsi: ufs: Reduce power management code duplication Daejun Park
[not found] ` <CGME20210722033513epcas2p22e4c2e6ea644992ede2739ebe381d53f@epcms2p8>
2021-07-29 0:56 ` [PATCH v3 03/18] scsi: ufs: Only include power management code if necessary Daejun Park
2021-07-31 14:48 ` Stanley Chu
[not found] ` <CGME20210722033520epcas2p31c6f801eda7f100491c85e3f9c7d6600@epcms2p6>
2021-07-29 0:56 ` [PATCH v3 04/18] scsi: ufs: Rename the second ufshcd_probe_hba() argument Daejun Park
[not found] ` <CGME20210722033523epcas2p22ea9a4afaeb46870638ff4429010a3c1@epcms2p7>
2021-07-29 0:57 ` [PATCH v3 05/18] scsi: ufs: Use DECLARE_COMPLETION_ONSTACK() where appropriate Daejun Park
[not found] ` <CGME20210722033527epcas2p384eefb77dff85f5d8d59beede98b6bdc@epcms2p4>
2021-07-29 0:57 ` [PATCH v3 07/18] scsi: ufs: Verify UIC locking requirements at runtime Daejun Park
[not found] ` <CGME20210722033531epcas2p4a4a975689ad7966d3db56dd81a7a255f@epcms2p1>
2021-07-29 0:57 ` [PATCH v3 08/18] scsi: ufs: Improve static type checking for the host controller state Daejun Park
[not found] ` <CGME20210722033536epcas2p133eef1f5e2e5a1022ccef23e9c1035aa@epcms2p5>
2021-07-29 1:24 ` [PATCH v3 09/18] scsi: ufs: Remove several wmb() calls Daejun Park
[not found] ` <CGME20210722033552epcas2p39f68ea806091ffa9755a25b778d70101@epcms2p2>
2021-07-29 1:25 ` [PATCH v3 12/18] scsi: ufs: Optimize serialization of setup_xfer_req() calls Daejun Park
[not found] ` <CGME20210722033553epcas2p2818d9c1f046e8514415a72a4ddddc3db@epcms2p1>
2021-07-29 1:25 ` [PATCH v3 13/18] scsi: ufs: Optimize SCSI command processing Daejun Park
2021-08-03 2:13 ` [PATCH v3 00/18] UFS patches for kernel v5.15 Martin K. Petersen
2021-08-10 5:20 ` Martin K. Petersen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210722033439.26550-14-bvanassche@acm.org \
--to=bvanassche@acm.org \
--cc=adrian.hunter@intel.com \
--cc=asutoshd@codeaurora.org \
--cc=avri.altman@wdc.com \
--cc=beanhuo@micron.com \
--cc=cang@codeaurora.org \
--cc=jaegeuk@kernel.org \
--cc=jejb@linux.ibm.com \
--cc=keosung.park@samsung.com \
--cc=kwmad.kim@samsung.com \
--cc=linux-scsi@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=matthias.bgg@gmail.com \
--cc=stanley.chu@mediatek.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).