From: Logan Gunthorpe <logang@deltatee.com>
To: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org,
linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org,
linux-rdma@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@lst.de>,
Bjorn Helgaas <bhelgaas@google.com>,
Dan Williams <dan.j.williams@intel.com>,
Sagi Grimberg <sagi@grimberg.me>, Keith Busch <kbusch@kernel.org>,
Jason Gunthorpe <jgg@ziepe.ca>,
Stephen Bates <sbates@raithlin.com>,
Logan Gunthorpe <logang@deltatee.com>
Subject: [RFC PATCH 20/28] IB/core: Introduce API for initializing a RW ctx from a DMA address
Date: Thu, 20 Jun 2019 10:12:32 -0600 [thread overview]
Message-ID: <20190620161240.22738-21-logang@deltatee.com> (raw)
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
Introduce rdma_rw_ctx_dma_init() and rdma_rw_ctx_dma_destroy() which
peform the same operation as rdma_rw_ctx_init() and
rdma_rw_ctx_destroy() respectively except they operate on a DMA
address and length instead of an SGL.
This will be used for struct page-less P2PDMA, but there's also
been opinions expressed to migrate away from SGLs and struct
pages in the RDMA APIs and this will likely fit with that
effort.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/infiniband/core/rw.c | 74 ++++++++++++++++++++++++++++++------
include/rdma/rw.h | 6 +++
2 files changed, 69 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index 32ca8429eaae..cefa6b930bc8 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -319,6 +319,39 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
}
EXPORT_SYMBOL(rdma_rw_ctx_init);
+/**
+ * rdma_rw_ctx_dma_init - initialize a RDMA READ/WRITE context from a
+ * DMA address instead of SGL
+ * @ctx: context to initialize
+ * @qp: queue pair to operate on
+ * @port_num: port num to which the connection is bound
+ * @addr: DMA address to READ/WRITE from/to
+ * @len: length of memory to operate on
+ * @remote_addr:remote address to read/write (relative to @rkey)
+ * @rkey: remote key to operate on
+ * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
+ *
+ * Returns the number of WQEs that will be needed on the workqueue if
+ * successful, or a negative error code.
+ */
+int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
+ u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr,
+ u32 rkey, enum dma_data_direction dir)
+{
+ struct scatterlist sg;
+
+ sg_dma_address(&sg) = addr;
+ sg_dma_len(&sg) = len;
+
+ if (rdma_rw_io_needs_mr(qp->device, port_num, dir, 1))
+ return rdma_rw_init_mr_wrs(ctx, qp, port_num, &sg, 1, 0,
+ remote_addr, rkey, dir);
+ else
+ return rdma_rw_init_single_wr(ctx, qp, &sg, 0, remote_addr,
+ rkey, dir);
+}
+EXPORT_SYMBOL(rdma_rw_ctx_dma_init);
+
/**
* rdma_rw_ctx_signature_init - initialize a RW context with signature offload
* @ctx: context to initialize
@@ -566,17 +599,7 @@ int rdma_rw_ctx_post(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
}
EXPORT_SYMBOL(rdma_rw_ctx_post);
-/**
- * rdma_rw_ctx_destroy - release all resources allocated by rdma_rw_ctx_init
- * @ctx: context to release
- * @qp: queue pair to operate on
- * @port_num: port num to which the connection is bound
- * @sg: scatterlist that was used for the READ/WRITE
- * @sg_cnt: number of entries in @sg
- * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
- */
-void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
- struct scatterlist *sg, u32 sg_cnt, enum dma_data_direction dir)
+static void __rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp)
{
int i;
@@ -596,6 +619,21 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
BUG();
break;
}
+}
+
+/**
+ * rdma_rw_ctx_destroy - release all resources allocated by rdma_rw_ctx_init
+ * @ctx: context to release
+ * @qp: queue pair to operate on
+ * @port_num: port num to which the connection is bound
+ * @sg: scatterlist that was used for the READ/WRITE
+ * @sg_cnt: number of entries in @sg
+ * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
+ */
+void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
+ struct scatterlist *sg, u32 sg_cnt, enum dma_data_direction dir)
+{
+ __rdma_rw_ctx_destroy(ctx, qp);
/* P2PDMA contexts do not need to be unmapped */
if (!is_pci_p2pdma_page(sg_page(sg)))
@@ -603,6 +641,20 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
}
EXPORT_SYMBOL(rdma_rw_ctx_destroy);
+/**
+ * rdma_rw_ctx_dma_destroy - release all resources allocated by
+ * rdma_rw_ctx_dma_init
+ * @ctx: context to release
+ * @qp: queue pair to operate on
+ * @port_num: port num to which the connection is bound
+ */
+void rdma_rw_ctx_dma_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
+ u8 port_num)
+{
+ __rdma_rw_ctx_destroy(ctx, qp);
+}
+EXPORT_SYMBOL(rdma_rw_ctx_dma_destroy);
+
/**
* rdma_rw_ctx_destroy_signature - release all resources allocated by
* rdma_rw_ctx_init_signature
diff --git a/include/rdma/rw.h b/include/rdma/rw.h
index 494f79ca3e62..e47f8053af6e 100644
--- a/include/rdma/rw.h
+++ b/include/rdma/rw.h
@@ -58,6 +58,12 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
struct scatterlist *sg, u32 sg_cnt,
enum dma_data_direction dir);
+int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
+ u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr,
+ u32 rkey, enum dma_data_direction dir);
+void rdma_rw_ctx_dma_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
+ u8 port_num);
+
int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
u8 port_num, struct scatterlist *sg, u32 sg_cnt,
struct scatterlist *prot_sg, u32 prot_sg_cnt,
--
2.20.1
next prev parent reply other threads:[~2019-06-20 16:13 UTC|newest]
Thread overview: 89+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-20 16:12 [RFC PATCH 00/28] Removing struct page from P2PDMA Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 01/28] block: Introduce DMA direct request type Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 02/28] block: Add dma_vec structure Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 03/28] block: Warn on mis-use of dma-direct bios Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 04/28] block: Never bounce " Logan Gunthorpe
2019-06-20 17:23 ` Jason Gunthorpe
2019-06-20 18:38 ` Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 05/28] block: Skip dma-direct bios in bio_integrity_prep() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 06/28] block: Support dma-direct bios in bio_advance_iter() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 07/28] block: Use dma_vec length in bio_cur_bytes() for dma-direct bios Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 08/28] block: Introduce dmavec_phys_mergeable() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 09/28] block: Introduce vec_gap_to_prev() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 10/28] block: Create generic vec_split_segs() from bvec_split_segs() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 11/28] block: Create blk_segment_split_ctx Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 12/28] block: Create helper for bvec_should_split() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 13/28] block: Generalize bvec_should_split() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 14/28] block: Support splitting dma-direct bios Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 15/28] block: Support counting dma-direct bio segments Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 16/28] block: Implement mapping dma-direct requests to SGs in blk_rq_map_sg() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 17/28] block: Introduce queue flag to indicate support for dma-direct bios Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 18/28] block: Introduce bio_add_dma_addr() Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 19/28] nvme-pci: Support dma-direct bios Logan Gunthorpe
2019-06-20 16:12 ` Logan Gunthorpe [this message]
2019-06-20 16:49 ` [RFC PATCH 20/28] IB/core: Introduce API for initializing a RW ctx from a DMA address Jason Gunthorpe
2019-06-20 16:59 ` Logan Gunthorpe
2019-06-20 17:11 ` Jason Gunthorpe
2019-06-20 18:24 ` Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 21/28] nvmet: Split nvmet_bdev_execute_rw() into a helper function Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 22/28] nvmet: Use DMA addresses instead of struct pages for P2P Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 23/28] nvme-pci: Remove support for PCI_P2PDMA requests Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 24/28] block: Remove PCI_P2PDMA queue flag Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 25/28] IB/core: Remove P2PDMA mapping support in rdma_rw_ctx Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 26/28] PCI/P2PDMA: Remove SGL helpers Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 27/28] PCI/P2PDMA: Remove struct pages that back P2PDMA memory Logan Gunthorpe
2019-06-20 16:12 ` [RFC PATCH 28/28] memremap: Remove PCI P2PDMA page memory type Logan Gunthorpe
2019-06-20 18:45 ` [RFC PATCH 00/28] Removing struct page from P2PDMA Dan Williams
2019-06-20 19:33 ` Jason Gunthorpe
2019-06-20 20:18 ` Dan Williams
2019-06-20 20:51 ` Logan Gunthorpe
2019-06-21 17:47 ` Jason Gunthorpe
2019-06-21 17:54 ` Dan Williams
2019-06-24 7:31 ` Christoph Hellwig
2019-06-24 13:46 ` Jason Gunthorpe
2019-06-24 13:50 ` Christoph Hellwig
2019-06-24 13:55 ` Jason Gunthorpe
2019-06-24 16:53 ` Logan Gunthorpe
2019-06-24 18:16 ` Jason Gunthorpe
2019-06-24 18:28 ` Logan Gunthorpe
2019-06-24 18:54 ` Jason Gunthorpe
2019-06-24 19:37 ` Logan Gunthorpe
2019-06-24 16:10 ` Logan Gunthorpe
2019-06-25 7:18 ` Christoph Hellwig
2019-06-20 19:34 ` Logan Gunthorpe
2019-06-20 23:40 ` Dan Williams
2019-06-20 23:42 ` Logan Gunthorpe
2019-06-24 7:27 ` Christoph Hellwig
2019-06-24 16:07 ` Logan Gunthorpe
2019-06-25 7:20 ` Christoph Hellwig
2019-06-25 15:57 ` Logan Gunthorpe
2019-06-25 17:01 ` Christoph Hellwig
2019-06-25 19:54 ` Logan Gunthorpe
2019-06-26 6:57 ` Christoph Hellwig
2019-06-26 18:31 ` Logan Gunthorpe
2019-06-26 20:21 ` Jason Gunthorpe
2019-06-26 20:39 ` Dan Williams
2019-06-26 20:54 ` Jason Gunthorpe
2019-06-26 20:55 ` Logan Gunthorpe
2019-06-26 20:45 ` Logan Gunthorpe
2019-06-26 21:00 ` Jason Gunthorpe
2019-06-26 21:18 ` Logan Gunthorpe
2019-06-27 6:32 ` Jason Gunthorpe
2019-06-27 16:09 ` Logan Gunthorpe
2019-06-27 16:35 ` Jason Gunthorpe
2019-06-27 16:49 ` Logan Gunthorpe
2019-06-28 4:57 ` Jason Gunthorpe
2019-06-28 16:22 ` Logan Gunthorpe
2019-06-28 17:29 ` Jason Gunthorpe
2019-06-28 18:29 ` Logan Gunthorpe
2019-06-28 19:09 ` Jason Gunthorpe
2019-06-28 19:35 ` Logan Gunthorpe
2019-07-02 22:45 ` Jason Gunthorpe
2019-07-02 22:52 ` Logan Gunthorpe
2019-06-27 9:08 ` Christoph Hellwig
2019-06-27 16:30 ` Logan Gunthorpe
2019-06-27 17:00 ` Christoph Hellwig
2019-06-27 18:00 ` Logan Gunthorpe
2019-06-28 13:38 ` Christoph Hellwig
2019-06-28 15:54 ` Logan Gunthorpe
2019-06-27 9:01 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190620161240.22738-21-logang@deltatee.com \
--to=logang@deltatee.com \
--cc=axboe@kernel.dk \
--cc=bhelgaas@google.com \
--cc=dan.j.williams@intel.com \
--cc=hch@lst.de \
--cc=jgg@ziepe.ca \
--cc=kbusch@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=sagi@grimberg.me \
--cc=sbates@raithlin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).