lustre-devel-lustre.org archive mirror
 help / color / mirror / Atom feed
From: James Simmons <jsimmons@infradead.org>
To: Andreas Dilger <adilger@whamcloud.com>,
	Oleg Drokin <green@whamcloud.com>, NeilBrown <neilb@suse.de>
Cc: Amir Shehata <ashehata@whamcloud.com>,
	Lustre Development List <lustre-devel@lists.lustre.org>
Subject: [lustre-devel] [PATCH 11/18] lnet: RMDA infrastructure updates
Date: Mon, 19 Jul 2021 08:32:06 -0400	[thread overview]
Message-ID: <1626697933-6971-12-git-send-email-jsimmons@infradead.org> (raw)
In-Reply-To: <1626697933-6971-1-git-send-email-jsimmons@infradead.org>

From: Amir Shehata <ashehata@whamcloud.com>

Add infrastructure to force RDMA for payloads < 4K.
Add infrastructure to extract the first page in a
payload. Useful for determining the type of the payload
to be transmitted.

WC-bug-id: https://jira.whamcloud.com/browse/LU-14798
Lustre-commit: 7ac839837c1c6cd1f ("LU-14798 lnet: RMDA infrastructure updates")
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Lustre-change: https://review.whamcloud.com/37453
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Whamcloud-bug-id: EX-773
Reviewed-on: https://review.whamcloud.com/44109
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 include/linux/lnet/lib-lnet.h       |  1 +
 include/linux/lnet/lib-types.h      |  2 ++
 net/lnet/klnds/o2iblnd/o2iblnd_cb.c |  4 ++--
 net/lnet/lnet/lib-md.c              | 29 +++++++++++++++++++++++------
 4 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/include/linux/lnet/lib-lnet.h b/include/linux/lnet/lib-lnet.h
index 6b9e926..f56ecab 100644
--- a/include/linux/lnet/lib-lnet.h
+++ b/include/linux/lnet/lib-lnet.h
@@ -711,6 +711,7 @@ void lnet_copy_kiov2iter(struct iov_iter *to,
 void lnet_md_unlink(struct lnet_libmd *md);
 void lnet_md_deconstruct(struct lnet_libmd *lmd, struct lnet_event *ev);
 struct page *lnet_kvaddr_to_page(unsigned long vaddr);
+struct page *lnet_get_first_page(struct lnet_libmd *md, unsigned int offset);
 int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset);
 
 unsigned int lnet_get_lnd_timeout(void);
diff --git a/include/linux/lnet/lib-types.h b/include/linux/lnet/lib-types.h
index 64d7472..e951e02 100644
--- a/include/linux/lnet/lib-types.h
+++ b/include/linux/lnet/lib-types.h
@@ -122,6 +122,8 @@ struct lnet_msg {
 	enum lnet_msg_hstatus	msg_health_status;
 	/* This is a recovery message */
 	bool			msg_recovery;
+	/* force an RDMA even if the message size is < 4K */
+	bool			msg_rdma_force;
 	/* the number of times a transmission has been retried */
 	int			msg_retry_count;
 	/* flag to indicate that we do not want to resend this message */
diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c
index ec0d05a..c66acc51 100644
--- a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1553,7 +1553,7 @@ static int kiblnd_map_tx(struct lnet_ni *ni, struct kib_tx *tx,
 
 		/* is the REPLY message too small for RDMA? */
 		nob = offsetof(struct kib_msg, ibm_u.immediate.ibim_payload[lntmsg->msg_md->md_length]);
-		if (nob <= IBLND_MSG_SIZE)
+		if (nob <= IBLND_MSG_SIZE && !lntmsg->msg_rdma_force)
 			break;		/* send IMMEDIATE */
 
 		tx = kiblnd_get_idle_tx(ni, target.nid);
@@ -1599,7 +1599,7 @@ static int kiblnd_map_tx(struct lnet_ni *ni, struct kib_tx *tx,
 	case LNET_MSG_PUT:
 		/* Is the payload small enough not to need RDMA? */
 		nob = offsetof(struct kib_msg, ibm_u.immediate.ibim_payload[payload_nob]);
-		if (nob <= IBLND_MSG_SIZE)
+		if (nob <= IBLND_MSG_SIZE && !lntmsg->msg_rdma_force)
 			break;			/* send IMMEDIATE */
 
 		tx = kiblnd_get_idle_tx(ni, target.nid);
diff --git a/net/lnet/lnet/lib-md.c b/net/lnet/lnet/lib-md.c
index fbee4e0..affa921 100644
--- a/net/lnet/lnet/lib-md.c
+++ b/net/lnet/lnet/lib-md.c
@@ -87,9 +87,9 @@ struct page *lnet_kvaddr_to_page(unsigned long vaddr)
 }
 EXPORT_SYMBOL(lnet_kvaddr_to_page);
 
-int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset)
+struct page *
+lnet_get_first_page(struct lnet_libmd *md, unsigned int offset)
 {
-	int cpt = CFS_CPT_ANY;
 	unsigned int niov;
 	struct bio_vec *kiov;
 
@@ -102,7 +102,7 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset)
 		md = lnet_handle2md(&md->md_bulk_handle);
 
 	if (!md || md->md_niov == 0)
-		return CFS_CPT_ANY;
+		return NULL;
 
 	kiov = md->md_kiov;
 	niov = md->md_niov;
@@ -113,12 +113,29 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset)
 		kiov++;
 		if (niov == 0) {
 			CERROR("offset %d goes beyond iov\n", offset);
-			goto out;
+			return NULL;
 		}
 	}
 
-	cpt = cfs_cpt_of_node(lnet_cpt_table(),
-			      page_to_nid(kiov->bv_page));
+	return kiov->bv_page;
+}
+
+int
+lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset)
+{
+	struct page *page;
+	int cpt = CFS_CPT_ANY;
+
+	page = lnet_get_first_page(md, offset);
+	if (!page) {
+		CDEBUG(D_NET,
+		       "Couldn't resolve first page of md %p with offset %u\n",
+		       md, offset);
+		goto out;
+	}
+
+	cpt = cfs_cpt_of_node(lnet_cpt_table(), page_to_nid(page));
+
 out:
 	return cpt;
 }
-- 
1.8.3.1

_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

  parent reply	other threads:[~2021-07-19 12:32 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-19 12:31 [lustre-devel] [PATCH 00/18] lustre: sync to OpenSFS as of July 18, 2021 James Simmons
2021-07-19 12:31 ` [lustre-devel] [PATCH 01/18] lustre: statahead: update task management code James Simmons
2021-07-19 12:31 ` [lustre-devel] [PATCH 02/18] lustre: llite: simplify callback handling for async getattr James Simmons
2021-07-19 12:31 ` [lustre-devel] [PATCH 03/18] lustre: uapi: per-user changelog names and mask James Simmons
2021-07-19 12:31 ` [lustre-devel] [PATCH 04/18] lnet: Correct peer NI recovery age out calculation James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 05/18] lustre: lmv: compare space to mkdir on parent MDT James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 06/18] lnet: annotate LNET_WIRE_HANDLE_COOKIE_NONE as u64 James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 07/18] lnet: libcfs: Add checksum speed under /sys/fs James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 08/18] lnet: use ni fatal error when calculating net health James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 09/18] lustre: quota: add get/set project support for non-dir/file James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 10/18] lustre: readahead: fix to reserve min pages James Simmons
2021-07-19 12:32 ` James Simmons [this message]
2021-07-19 12:32 ` [lustre-devel] [PATCH 12/18] lnet: o2iblnd: Move racy NULL assignment James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 13/18] lnet: o2iblnd: Avoid double posting invalidate James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 14/18] lustre: quota: nodemap squashed root cannot bypass quota James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 15/18] lustre: llite: reset pfid after dir migration James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 16/18] lustre: llite: failed ASSERTION(ldlm_has_layout(lock)) James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 17/18] lustre: pcc: introducing OBD_CONNECT2_PCCRO flag James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 18/18] lustre: sec: migrate/extend/split on encrypted file James Simmons

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1626697933-6971-12-git-send-email-jsimmons@infradead.org \
    --to=jsimmons@infradead.org \
    --cc=adilger@whamcloud.com \
    --cc=ashehata@whamcloud.com \
    --cc=green@whamcloud.com \
    --cc=lustre-devel@lists.lustre.org \
    --cc=neilb@suse.de \
    --subject='Re: [lustre-devel] [PATCH 11/18] lnet: RMDA infrastructure updates' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).