lustre-devel-lustre.org archive mirror
 help / color / mirror / Atom feed
From: James Simmons <jsimmons@infradead.org>
To: Andreas Dilger <adilger@whamcloud.com>,
	Oleg Drokin <green@whamcloud.com>, NeilBrown <neilb@suse.de>
Cc: Chris Horn <chris.horn@hpe.com>,
	Lustre Development List <lustre-devel@lists.lustre.org>
Subject: [lustre-devel] [PATCH 25/27] lnet: Fix destination NID for discovery PUSH
Date: Sun, 13 Jun 2021 19:11:35 -0400	[thread overview]
Message-ID: <1623625897-17706-26-git-send-email-jsimmons@infradead.org> (raw)
In-Reply-To: <1623625897-17706-1-git-send-email-jsimmons@infradead.org>

From: Chris Horn <chris.horn@hpe.com>

If we're sending a discovery PUSH after receiving a discovery
REPLY then we want to send via the same NID that the reply was
sent to. This introduces a challenge in selecting an appropriate
destination NID for the PUSH because lnet_select_pathway() will not
run the MR selection algorithm for choosing a peer NI if the source
NI has been specified.

It is reasonable to assume that the NID used by the message
originator in sending the REPLY is a suitable destination for the
discovery PUSH. Thus, we record this NID in the same location we
currently record the lp_disc_src_nid, and use it when sending the
PUSH. With this change, the only other user of lnet_peer_select_nid()
is lnet_peer_send_ping(). In the ping case we do not set a source NID,
so lnet_select_pathway() is free to choose any peer NI. So this change
allows us to get rid of lnet_peer_select_nid() altogether.

Alternatively, we would need to reproduce a lot of the path selection
algorithm inside lnet_peer_select_nid() in order to avoid sending to
unhealthy NIDs. It seems undesirable and unnecessary to duplicate that
logic.

HPE-bug-id: LUS-9333
WC-bug-id: https://jira.whamcloud.com/browse/LU-14660
Lustre-commit: dce2f7d1987711dfd ("LU-14660 lnet: Fix destination NID for discovery PUSH")
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Reviewed-on: https://review.whamcloud.com/43507
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 include/linux/lnet/lib-types.h |  2 ++
 net/lnet/lnet/peer.c           | 52 ++++++++++--------------------------------
 2 files changed, 14 insertions(+), 40 deletions(-)

diff --git a/include/linux/lnet/lib-types.h b/include/linux/lnet/lib-types.h
index d898066..cb0a950 100644
--- a/include/linux/lnet/lib-types.h
+++ b/include/linux/lnet/lib-types.h
@@ -640,6 +640,8 @@ struct lnet_peer {
 
 	/* source NID to use during discovery */
 	lnet_nid_t		lp_disc_src_nid;
+	/* destination NID to use during discovery */
+	lnet_nid_t		lp_disc_dst_nid;
 
 	/* net to perform discovery on */
 	u32			lp_disc_net_id;
diff --git a/net/lnet/lnet/peer.c b/net/lnet/lnet/peer.c
index d66a302..7630aff 100644
--- a/net/lnet/lnet/peer.c
+++ b/net/lnet/lnet/peer.c
@@ -221,6 +221,7 @@
 	spin_lock_init(&lp->lp_lock);
 	lp->lp_primary_nid = nid;
 	lp->lp_disc_src_nid = LNET_NID_ANY;
+	lp->lp_disc_dst_nid = LNET_NID_ANY;
 	if (lnet_peers_start_down())
 		lp->lp_alive = false;
 	else
@@ -2515,6 +2516,7 @@ static void lnet_peer_clear_discovery_error(struct lnet_peer *lp)
 	spin_lock(&lp->lp_lock);
 
 	lp->lp_disc_src_nid = ev->target.nid;
+	lp->lp_disc_dst_nid = ev->source.nid;
 
 	/*
 	 * If some kind of error happened the contents of message
@@ -3221,8 +3223,10 @@ static int lnet_peer_data_present(struct lnet_peer *lp)
 			 * received by lp, we need to set the discovery source
 			 * NID for new_lp to the NID stored in lp.
 			 */
-			if (lp->lp_disc_src_nid != LNET_NID_ANY)
+			if (lp->lp_disc_src_nid != LNET_NID_ANY) {
 				new_lp->lp_disc_src_nid = lp->lp_disc_src_nid;
+				new_lp->lp_disc_dst_nid = lp->lp_disc_dst_nid;
+			}
 			spin_unlock(&new_lp->lp_lock);
 			spin_unlock(&lp->lp_lock);
 
@@ -3273,41 +3277,10 @@ static int lnet_peer_ping_failed(struct lnet_peer *lp)
 	return rc ? rc : LNET_REDISCOVER_PEER;
 }
 
-/*
- * Select NID to send a Ping or Push to.
- */
-static lnet_nid_t lnet_peer_select_nid(struct lnet_peer *lp)
-{
-	struct lnet_peer_ni *lpni;
-
-	/* Look for a direct-connected NID for this peer. */
-	lpni = NULL;
-	while ((lpni = lnet_get_next_peer_ni_locked(lp, NULL, lpni)) != NULL) {
-		if (!lnet_get_net_locked(lpni->lpni_peer_net->lpn_net_id))
-			continue;
-		break;
-	}
-	if (lpni)
-		return lpni->lpni_nid;
-
-	/* Look for a routed-connected NID for this peer. */
-	lpni = NULL;
-	while ((lpni = lnet_get_next_peer_ni_locked(lp, NULL, lpni)) != NULL) {
-		if (!lnet_find_rnet_locked(lpni->lpni_peer_net->lpn_net_id))
-			continue;
-		break;
-	}
-	if (lpni)
-		return lpni->lpni_nid;
-
-	return LNET_NID_ANY;
-}
-
 /* Active side of ping. */
 static int lnet_peer_send_ping(struct lnet_peer *lp)
 __must_hold(&lp->lp_lock)
 {
-	lnet_nid_t pnid;
 	int nnis;
 	int rc;
 	int cpt;
@@ -3319,12 +3292,11 @@ static int lnet_peer_send_ping(struct lnet_peer *lp)
 	cpt = lnet_net_lock_current();
 	/* Refcount for MD. */
 	lnet_peer_addref_locked(lp);
-	pnid = lnet_peer_select_nid(lp);
 	lnet_net_unlock(cpt);
 
 	nnis = max_t(int, lp->lp_data_nnis, LNET_INTERFACES_MIN);
 
-	rc = lnet_send_ping(pnid, &lp->lp_ping_mdh, nnis, lp,
+	rc = lnet_send_ping(lp->lp_primary_nid, &lp->lp_ping_mdh, nnis, lp,
 			    the_lnet.ln_dc_handler, false);
 	/* if LNetMDBind in lnet_send_ping fails we need to decrement the
 	 * refcount on the peer, otherwise LNetMDUnlink will be called
@@ -3445,18 +3417,17 @@ static int lnet_peer_send_push(struct lnet_peer *lp)
 		CERROR("Can't bind push source MD: %d\n", rc);
 		goto fail_error;
 	}
+
 	cpt = lnet_net_lock_current();
 	/* Refcount for MD. */
 	lnet_peer_addref_locked(lp);
 	id.pid = LNET_PID_LUSTRE;
-	id.nid = lnet_peer_select_nid(lp);
+	if (lp->lp_disc_dst_nid != LNET_NID_ANY)
+		id.nid = lp->lp_disc_dst_nid;
+	else
+		id.nid = lp->lp_primary_nid;
 	lnet_net_unlock(cpt);
 
-	if (id.nid == LNET_NID_ANY) {
-		rc = -EHOSTUNREACH;
-		goto fail_unlink;
-	}
-
 	rc = LNetPut(lp->lp_disc_src_nid, lp->lp_push_mdh,
 		     LNET_ACK_REQ, id, LNET_RESERVED_PORTAL,
 		     LNET_PROTO_PING_MATCHBITS, 0, 0);
@@ -3466,6 +3437,7 @@ static int lnet_peer_send_push(struct lnet_peer *lp)
 	 * scratch
 	 */
 	lp->lp_disc_src_nid = LNET_NID_ANY;
+	lp->lp_disc_dst_nid = LNET_NID_ANY;
 	if (rc)
 		goto fail_unlink;
 
-- 
1.8.3.1

_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

  parent reply	other threads:[~2021-06-13 23:13 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-13 23:11 [lustre-devel] [PATCH 00/27] lustre: sync to 2.14.52 James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 01/27] lustre: uapi: add mdt_hash_name James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 02/27] lustre: uapi: rename CONFIG_T_* to MGS_CFG_T_* James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 03/27] lnet: o2iblnd: fix bug in list_first_entry() change James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 04/27] lustre: flr: mmap write/punch does not stale other mirrors James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 05/27] lustre: llite: default lsm update may memory leak James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 06/27] lustre: pcc: don't alloc FID in LLITE for pcc open James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 07/27] lustre: quota: default OST Pool Quotas James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 08/27] lustre: rename tgt_pool_* functions James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 09/27] lustre: llite: refresh layout after mirror merge/split James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 10/27] lustre: ptlrpc: do not match reply with resent RPC James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 11/27] lustre: vvp: wait for nrpages to be updated James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 12/27] lustre: obd: check if sbi->ll_md_exp is initialized James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 13/27] lustre: osc: Batch gang_lookup cbs James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 14/27] lustre: llite: Return errors for aio James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 15/27] lnet: do not crash if lnet_sock_getaddr returns error James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 16/27] lustre: sec: forbid file rename from enc to unencrypted dir James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 17/27] lustre: mdc: start changelog thread upon first access James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 18/27] lustre: llog: changelog purge deletes plain llog James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 19/27] lnet: libcfs: allow comma-separated masks James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 20/27] lustre: osc: cleanup comment in osc_object_is_contended James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 21/27] lnet: simplify lnet_ni_add_interface James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 22/27] lustre: lmv: change default hash type to crush James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 23/27] lustre: ptlrpc: move more members in PTLRPC request into pill James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 24/27] lustre: llite: add selinux testing James Simmons
2021-06-13 23:11 ` James Simmons [this message]
2021-06-13 23:11 ` [lustre-devel] [PATCH 26/27] lnet: Check if discovery toggled off in ping reply James Simmons
2021-06-13 23:11 ` [lustre-devel] [PATCH 27/27] lustre: update version to 2.14.52 James Simmons

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1623625897-17706-26-git-send-email-jsimmons@infradead.org \
    --to=jsimmons@infradead.org \
    --cc=adilger@whamcloud.com \
    --cc=chris.horn@hpe.com \
    --cc=green@whamcloud.com \
    --cc=lustre-devel@lists.lustre.org \
    --cc=neilb@suse.de \
    --subject='Re: [lustre-devel] [PATCH 25/27] lnet: Fix destination NID for discovery PUSH' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).