From: James Simmons <jsimmons@infradead.org>
To: Andreas Dilger <adilger@whamcloud.com>,
Oleg Drokin <green@whamcloud.com>, NeilBrown <neilb@suse.de>
Cc: Amir Shehata <ashehata@whamcloud.com>,
Lustre Development List <lustre-devel@lists.lustre.org>
Subject: [lustre-devel] [PATCH 11/18] lnet: RMDA infrastructure updates
Date: Mon, 19 Jul 2021 08:32:06 -0400 [thread overview]
Message-ID: <1626697933-6971-12-git-send-email-jsimmons@infradead.org> (raw)
In-Reply-To: <1626697933-6971-1-git-send-email-jsimmons@infradead.org>
From: Amir Shehata <ashehata@whamcloud.com>
Add infrastructure to force RDMA for payloads < 4K.
Add infrastructure to extract the first page in a
payload. Useful for determining the type of the payload
to be transmitted.
WC-bug-id: https://jira.whamcloud.com/browse/LU-14798
Lustre-commit: 7ac839837c1c6cd1f ("LU-14798 lnet: RMDA infrastructure updates")
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Lustre-change: https://review.whamcloud.com/37453
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Whamcloud-bug-id: EX-773
Reviewed-on: https://review.whamcloud.com/44109
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
include/linux/lnet/lib-lnet.h | 1 +
include/linux/lnet/lib-types.h | 2 ++
net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 4 ++--
net/lnet/lnet/lib-md.c | 29 +++++++++++++++++++++++------
4 files changed, 28 insertions(+), 8 deletions(-)
diff --git a/include/linux/lnet/lib-lnet.h b/include/linux/lnet/lib-lnet.h
index 6b9e926..f56ecab 100644
--- a/include/linux/lnet/lib-lnet.h
+++ b/include/linux/lnet/lib-lnet.h
@@ -711,6 +711,7 @@ void lnet_copy_kiov2iter(struct iov_iter *to,
void lnet_md_unlink(struct lnet_libmd *md);
void lnet_md_deconstruct(struct lnet_libmd *lmd, struct lnet_event *ev);
struct page *lnet_kvaddr_to_page(unsigned long vaddr);
+struct page *lnet_get_first_page(struct lnet_libmd *md, unsigned int offset);
int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset);
unsigned int lnet_get_lnd_timeout(void);
diff --git a/include/linux/lnet/lib-types.h b/include/linux/lnet/lib-types.h
index 64d7472..e951e02 100644
--- a/include/linux/lnet/lib-types.h
+++ b/include/linux/lnet/lib-types.h
@@ -122,6 +122,8 @@ struct lnet_msg {
enum lnet_msg_hstatus msg_health_status;
/* This is a recovery message */
bool msg_recovery;
+ /* force an RDMA even if the message size is < 4K */
+ bool msg_rdma_force;
/* the number of times a transmission has been retried */
int msg_retry_count;
/* flag to indicate that we do not want to resend this message */
diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c
index ec0d05a..c66acc51 100644
--- a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1553,7 +1553,7 @@ static int kiblnd_map_tx(struct lnet_ni *ni, struct kib_tx *tx,
/* is the REPLY message too small for RDMA? */
nob = offsetof(struct kib_msg, ibm_u.immediate.ibim_payload[lntmsg->msg_md->md_length]);
- if (nob <= IBLND_MSG_SIZE)
+ if (nob <= IBLND_MSG_SIZE && !lntmsg->msg_rdma_force)
break; /* send IMMEDIATE */
tx = kiblnd_get_idle_tx(ni, target.nid);
@@ -1599,7 +1599,7 @@ static int kiblnd_map_tx(struct lnet_ni *ni, struct kib_tx *tx,
case LNET_MSG_PUT:
/* Is the payload small enough not to need RDMA? */
nob = offsetof(struct kib_msg, ibm_u.immediate.ibim_payload[payload_nob]);
- if (nob <= IBLND_MSG_SIZE)
+ if (nob <= IBLND_MSG_SIZE && !lntmsg->msg_rdma_force)
break; /* send IMMEDIATE */
tx = kiblnd_get_idle_tx(ni, target.nid);
diff --git a/net/lnet/lnet/lib-md.c b/net/lnet/lnet/lib-md.c
index fbee4e0..affa921 100644
--- a/net/lnet/lnet/lib-md.c
+++ b/net/lnet/lnet/lib-md.c
@@ -87,9 +87,9 @@ struct page *lnet_kvaddr_to_page(unsigned long vaddr)
}
EXPORT_SYMBOL(lnet_kvaddr_to_page);
-int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset)
+struct page *
+lnet_get_first_page(struct lnet_libmd *md, unsigned int offset)
{
- int cpt = CFS_CPT_ANY;
unsigned int niov;
struct bio_vec *kiov;
@@ -102,7 +102,7 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset)
md = lnet_handle2md(&md->md_bulk_handle);
if (!md || md->md_niov == 0)
- return CFS_CPT_ANY;
+ return NULL;
kiov = md->md_kiov;
niov = md->md_niov;
@@ -113,12 +113,29 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset)
kiov++;
if (niov == 0) {
CERROR("offset %d goes beyond iov\n", offset);
- goto out;
+ return NULL;
}
}
- cpt = cfs_cpt_of_node(lnet_cpt_table(),
- page_to_nid(kiov->bv_page));
+ return kiov->bv_page;
+}
+
+int
+lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset)
+{
+ struct page *page;
+ int cpt = CFS_CPT_ANY;
+
+ page = lnet_get_first_page(md, offset);
+ if (!page) {
+ CDEBUG(D_NET,
+ "Couldn't resolve first page of md %p with offset %u\n",
+ md, offset);
+ goto out;
+ }
+
+ cpt = cfs_cpt_of_node(lnet_cpt_table(), page_to_nid(page));
+
out:
return cpt;
}
--
1.8.3.1
_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
next prev parent reply other threads:[~2021-07-19 12:32 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-19 12:31 [lustre-devel] [PATCH 00/18] lustre: sync to OpenSFS as of July 18, 2021 James Simmons
2021-07-19 12:31 ` [lustre-devel] [PATCH 01/18] lustre: statahead: update task management code James Simmons
2021-07-19 12:31 ` [lustre-devel] [PATCH 02/18] lustre: llite: simplify callback handling for async getattr James Simmons
2021-07-19 12:31 ` [lustre-devel] [PATCH 03/18] lustre: uapi: per-user changelog names and mask James Simmons
2021-07-19 12:31 ` [lustre-devel] [PATCH 04/18] lnet: Correct peer NI recovery age out calculation James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 05/18] lustre: lmv: compare space to mkdir on parent MDT James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 06/18] lnet: annotate LNET_WIRE_HANDLE_COOKIE_NONE as u64 James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 07/18] lnet: libcfs: Add checksum speed under /sys/fs James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 08/18] lnet: use ni fatal error when calculating net health James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 09/18] lustre: quota: add get/set project support for non-dir/file James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 10/18] lustre: readahead: fix to reserve min pages James Simmons
2021-07-19 12:32 ` James Simmons [this message]
2021-07-19 12:32 ` [lustre-devel] [PATCH 12/18] lnet: o2iblnd: Move racy NULL assignment James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 13/18] lnet: o2iblnd: Avoid double posting invalidate James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 14/18] lustre: quota: nodemap squashed root cannot bypass quota James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 15/18] lustre: llite: reset pfid after dir migration James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 16/18] lustre: llite: failed ASSERTION(ldlm_has_layout(lock)) James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 17/18] lustre: pcc: introducing OBD_CONNECT2_PCCRO flag James Simmons
2021-07-19 12:32 ` [lustre-devel] [PATCH 18/18] lustre: sec: migrate/extend/split on encrypted file James Simmons
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1626697933-6971-12-git-send-email-jsimmons@infradead.org \
--to=jsimmons@infradead.org \
--cc=adilger@whamcloud.com \
--cc=ashehata@whamcloud.com \
--cc=green@whamcloud.com \
--cc=lustre-devel@lists.lustre.org \
--cc=neilb@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).