From: hch@lst.de (Christoph Hellwig)
Subject: [PATCH 13/15] nvme-pci: optimize mapping of small single segment requests
Date: Thu, 21 Mar 2019 16:10:35 -0700 [thread overview]
Message-ID: <20190321231037.25104-14-hch@lst.de> (raw)
In-Reply-To: <20190321231037.25104-1-hch@lst.de>
If a request is single segment and fits into one or two PRP entries we
do not have to create a scatterlist for it, but can just map the bio_vec
directly.
Signed-off-by: Christoph Hellwig <hch at lst.de>
---
drivers/nvme/host/pci.c | 45 ++++++++++++++++++++++++++++++++++++-----
1 file changed, 40 insertions(+), 5 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index c6047935e825..47fc4d653961 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -221,6 +221,7 @@ struct nvme_iod {
int npages; /* In the PRP list. 0 means small pool in use */
int nents; /* Used in scatterlist */
dma_addr_t first_dma;
+ unsigned int dma_len; /* length of single DMA segment mapping */
dma_addr_t meta_dma;
struct scatterlist *sg;
};
@@ -576,13 +577,18 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req)
dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
int i;
- if (iod->nents) {
- /* P2PDMA requests do not need to be unmapped */
- if (!is_pci_p2pdma_page(sg_page(iod->sg)))
- dma_unmap_sg(dev->dev, iod->sg, iod->nents, dma_dir);
-
+ if (iod->dma_len) {
+ dma_unmap_page(dev->dev, dma_addr, iod->dma_len, dma_dir);
+ return;
}
+ WARN_ON_ONCE(!iod->nents);
+
+ /* P2PDMA requests do not need to be unmapped */
+ if (!is_pci_p2pdma_page(sg_page(iod->sg)))
+ dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req));
+
+
if (iod->npages == 0)
dma_pool_free(dev->prp_small_pool, nvme_pci_iod_list(req)[0],
dma_addr);
@@ -795,6 +801,24 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev,
return BLK_STS_OK;
}
+static blk_status_t nvme_setup_prp_simple(struct nvme_dev *dev,
+ struct request *req, struct nvme_rw_command *cmnd,
+ struct bio_vec *bv)
+{
+ struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+ unsigned int first_prp_len = dev->ctrl.page_size - bv->bv_offset;
+
+ iod->first_dma = dma_map_bvec(dev->dev, bv, rq_dma_dir(req), 0);
+ if (dma_mapping_error(dev->dev, iod->first_dma))
+ return BLK_STS_RESOURCE;
+ iod->dma_len = bv->bv_len;
+
+ cmnd->dptr.prp1 = cpu_to_le64(iod->first_dma);
+ if (bv->bv_len > first_prp_len)
+ cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma + first_prp_len);
+ return 0;
+}
+
static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
struct nvme_command *cmnd)
{
@@ -805,6 +829,17 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
blk_status_t ret = BLK_STS_IOERR;
int nr_mapped;
+ if (blk_rq_nr_phys_segments(req) == 1) {
+ struct bio_vec bv = req_bvec(req);
+
+ if (!is_pci_p2pdma_page(bv.bv_page)) {
+ if (bv.bv_offset + bv.bv_len <= dev->ctrl.page_size * 2)
+ return nvme_setup_prp_simple(dev, req,
+ &cmnd->rw, &bv);
+ }
+ }
+
+ iod->dma_len = 0;
iod->sg = mempool_alloc(dev->iod_mempool, GFP_ATOMIC);
if (!iod->sg)
return BLK_STS_RESOURCE;
--
2.20.1
next prev parent reply other threads:[~2019-03-21 23:10 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-03-21 23:10 [RFC] optimize nvme single segment I/O Christoph Hellwig
2019-03-21 23:10 ` [PATCH 01/15] block: add a req_bvec helper Christoph Hellwig
2019-03-25 5:07 ` Chaitanya Kulkarni
2019-03-27 14:16 ` Christoph Hellwig
2019-03-21 23:10 ` [PATCH 02/15] block: add a rq_integrity_vec helper Christoph Hellwig
2019-03-25 5:10 ` Chaitanya Kulkarni
2019-03-27 14:19 ` Christoph Hellwig
2019-03-21 23:10 ` [PATCH 03/15] block: add a rq_dma_dir helper Christoph Hellwig
2019-03-22 13:06 ` Johannes Thumshirn
2019-03-27 14:20 ` Christoph Hellwig
2019-03-28 10:26 ` Johannes Thumshirn
2019-03-25 5:11 ` Chaitanya Kulkarni
2019-03-21 23:10 ` [PATCH 04/15] block: add dma_map_bvec helper Christoph Hellwig
2019-03-25 5:13 ` Chaitanya Kulkarni
2019-03-21 23:10 ` [PATCH 05/15] nvme-pci: remove the unused iod->length field Christoph Hellwig
2019-03-25 5:14 ` Chaitanya Kulkarni
2019-03-21 23:10 ` [PATCH 06/15] nvme-pci: remove nvme_init_iod Christoph Hellwig
2019-03-25 5:19 ` Chaitanya Kulkarni
2019-03-27 14:21 ` Christoph Hellwig
2019-03-21 23:10 ` [PATCH 07/15] nvme-pci: move the call to nvme_cleanup_cmd out of nvme_unmap_data Christoph Hellwig
2019-03-25 5:21 ` Chaitanya Kulkarni
2019-03-21 23:10 ` [PATCH 08/15] nvme-pci: merge nvme_free_iod into nvme_unmap_data Christoph Hellwig
2019-03-25 5:22 ` Chaitanya Kulkarni
2019-03-21 23:10 ` [PATCH 09/15] nvme-pci: only call nvme_unmap_data for requests transferring data Christoph Hellwig
2019-03-25 5:23 ` Chaitanya Kulkarni
2019-03-21 23:10 ` [PATCH 10/15] nvme-pci: do not build a scatterlist to map metadata Christoph Hellwig
2019-03-25 5:27 ` Chaitanya Kulkarni
[not found] ` <20190828092057.GA15524@ming.t460p>
2019-09-12 1:02 ` Ming Lei
2019-09-12 8:20 ` Christoph Hellwig
2019-03-21 23:10 ` [PATCH 11/15] nvme-pci: split metadata handling from nvme_map_data / nvme_unmap_data Christoph Hellwig
2019-03-25 5:29 ` Chaitanya Kulkarni
2019-03-21 23:10 ` [PATCH 12/15] nvme-pci: remove the inline scatterlist optimization Christoph Hellwig
2019-03-25 5:30 ` Chaitanya Kulkarni
2019-03-21 23:10 ` Christoph Hellwig [this message]
2019-03-25 5:36 ` [PATCH 13/15] nvme-pci: optimize mapping of small single segment requests Chaitanya Kulkarni
2019-03-21 23:10 ` [PATCH 14/15] nvme-pci: optimize mapping single segment requests using SGLs Christoph Hellwig
2019-03-25 5:39 ` Chaitanya Kulkarni
2019-04-30 14:17 ` Klaus Birkelund
2019-04-30 14:32 ` Christoph Hellwig
2019-03-21 23:10 ` [PATCH 15/15] nvme-pci: tidy up nvme_map_data Christoph Hellwig
2019-03-25 5:40 ` Chaitanya Kulkarni
2019-03-22 15:44 ` [RFC] optimize nvme single segment I/O Jens Axboe
2019-03-27 14:24 ` Christoph Hellwig
2019-03-22 17:37 ` Keith Busch
2019-03-22 18:55 ` Sagi Grimberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190321231037.25104-14-hch@lst.de \
--to=hch@lst.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).