* [PATCH 0/3] Fix up a couple of issues around layout handling
@ 2017-05-02 16:41 Trond Myklebust
2017-05-02 16:41 ` [PATCH 1/3] pNFS: Don't clear the layout return info if there are segments to return Trond Myklebust
0 siblings, 1 reply; 5+ messages in thread
From: Trond Myklebust @ 2017-05-02 16:41 UTC (permalink / raw)
To: linux-nfs
The main issue to be dealt with is a deadlock that can occur due to
an ABBA-type of situation between layoutget and layoutreturn.
(Resend - apologies for the confused first send)
Trond Myklebust (3):
pNFS: Don't clear the layout return info if there are segments to
return
pNFS: Fix a deadlock when coalescing writes and returning the layout
pNFS: Fix a typo in pnfs_generic_alloc_ds_commits
fs/nfs/pnfs.c | 10 +++++++---
fs/nfs/pnfs_nfs.c | 2 +-
2 files changed, 8 insertions(+), 4 deletions(-)
--
2.9.3
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/3] pNFS: Don't clear the layout return info if there are segments to return 2017-05-02 16:41 [PATCH 0/3] Fix up a couple of issues around layout handling Trond Myklebust @ 2017-05-02 16:41 ` Trond Myklebust 2017-05-02 16:41 ` [PATCH 2/3] pNFS: Fix a deadlock when coalescing writes and returning the layout Trond Myklebust 0 siblings, 1 reply; 5+ messages in thread From: Trond Myklebust @ 2017-05-02 16:41 UTC (permalink / raw) To: linux-nfs In pnfs_clear_layoutreturn_info, ensure that we don't clear the layout return info if there are new segments queued for return due to, for instance, a race between a LAYOUTRETURN and a failed I/O attempt. Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> --- fs/nfs/pnfs.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index 140ecd7d350f..cea1e838efae 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -322,9 +322,15 @@ pnfs_set_plh_return_info(struct pnfs_layout_hdr *lo, enum pnfs_iomode iomode, static void pnfs_clear_layoutreturn_info(struct pnfs_layout_hdr *lo) { + struct pnfs_layout_segment *lseg; lo->plh_return_iomode = 0; lo->plh_return_seq = 0; clear_bit(NFS_LAYOUT_RETURN_REQUESTED, &lo->plh_flags); + list_for_each_entry(lseg, &lo->plh_segs, pls_list) { + if (!test_bit(NFS_LSEG_LAYOUTRETURN, &lseg->pls_flags)) + continue; + pnfs_set_plh_return_info(lo, lseg->pls_range.iomode, 0); + } } static void pnfs_clear_layoutreturn_waitbit(struct pnfs_layout_hdr *lo) @@ -367,9 +373,9 @@ pnfs_mark_layout_stateid_invalid(struct pnfs_layout_hdr *lo, struct pnfs_layout_segment *lseg, *next; set_bit(NFS_LAYOUT_INVALID_STID, &lo->plh_flags); - pnfs_clear_layoutreturn_info(lo); list_for_each_entry_safe(lseg, next, &lo->plh_segs, pls_list) pnfs_clear_lseg_state(lseg, lseg_list); + pnfs_clear_layoutreturn_info(lo); pnfs_free_returned_lsegs(lo, lseg_list, &range, 0); if (test_bit(NFS_LAYOUT_RETURN, &lo->plh_flags) && !test_and_set_bit(NFS_LAYOUT_RETURN_LOCK, &lo->plh_flags)) -- 2.9.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/3] pNFS: Fix a deadlock when coalescing writes and returning the layout 2017-05-02 16:41 ` [PATCH 1/3] pNFS: Don't clear the layout return info if there are segments to return Trond Myklebust @ 2017-05-02 16:41 ` Trond Myklebust 2017-05-02 16:41 ` [PATCH 3/3] pNFS: Fix a typo in pnfs_generic_alloc_ds_commits Trond Myklebust 0 siblings, 1 reply; 5+ messages in thread From: Trond Myklebust @ 2017-05-02 16:41 UTC (permalink / raw) To: linux-nfs Consider the following deadlock: Process P1 Process P2 Process P3 ========== ========== ========== lock_page(page) lseg = pnfs_update_layout(inode) lo = NFS_I(inode)->layout pnfs_error_mark_layout_for_return(lo) lock_page(page) lseg = pnfs_update_layout(inode) In this scenario, - P1 has declared the layout to be in error, but P2 holds a reference to a layout segment on that inode, so the layoutreturn is deferred. - P2 is waiting for a page lock held by P3. - P3 is asking for a new layout segment, but is blocked waiting for the layoutreturn. The fix is to ensure that pnfs_error_mark_layout_for_return() does not set the NFS_LAYOUT_RETURN flag, which blocks P3. Instead, we allow the latter to call LAYOUTGET so that it can make progress and unblock P2. Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> --- fs/nfs/pnfs.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index cea1e838efae..adc6ec28d4b5 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -2063,8 +2063,6 @@ void pnfs_error_mark_layout_for_return(struct inode *inode, return; } pnfs_set_plh_return_info(lo, range.iomode, 0); - /* Block LAYOUTGET */ - set_bit(NFS_LAYOUT_RETURN, &lo->plh_flags); /* * mark all matching lsegs so that we are sure to have no live * segments at hand when sending layoutreturn. See pnfs_put_lseg() -- 2.9.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/3] pNFS: Fix a typo in pnfs_generic_alloc_ds_commits 2017-05-02 16:41 ` [PATCH 2/3] pNFS: Fix a deadlock when coalescing writes and returning the layout Trond Myklebust @ 2017-05-02 16:41 ` Trond Myklebust 0 siblings, 0 replies; 5+ messages in thread From: Trond Myklebust @ 2017-05-02 16:41 UTC (permalink / raw) To: linux-nfs If the layout segment is invalid, we want to just resend the remaining writes. Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> --- fs/nfs/pnfs_nfs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/nfs/pnfs_nfs.c b/fs/nfs/pnfs_nfs.c index 7697ac0ff81a..ae600ab1a646 100644 --- a/fs/nfs/pnfs_nfs.c +++ b/fs/nfs/pnfs_nfs.c @@ -223,7 +223,7 @@ pnfs_generic_alloc_ds_commits(struct nfs_commit_info *cinfo, */ if (!pnfs_is_valid_lseg(bucket->clseg) && !test_bit(NFS_LSEG_LAYOUTRETURN, &bucket->clseg->pls_flags)) - continue; + break; data = nfs_commitdata_alloc(false); if (!data) break; -- 2.9.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 0/3] Fix up a couple of issues around layout handling
@ 2017-05-02 16:38 Trond Myklebust
2017-05-02 16:38 ` [RFC PATCH 1/5] SUNRPC: Allow creation of RPC clients with multiple connections Trond Myklebust
0 siblings, 1 reply; 5+ messages in thread
From: Trond Myklebust @ 2017-05-02 16:38 UTC (permalink / raw)
To: linux-nfs
The main issue to be dealt with is a deadlock that can occur due to
an ABBA-type of situation between layoutget and layoutreturn.
Trond Myklebust (3):
pNFS: Don't clear the layout return info if there are segments to
return
pNFS: Fix a deadlock when coalescing writes and returning the layout
pNFS: Fix a typo in pnfs_generic_alloc_ds_commits
fs/nfs/pnfs.c | 10 +++++++---
fs/nfs/pnfs_nfs.c | 2 +-
2 files changed, 8 insertions(+), 4 deletions(-)
--
2.9.3
^ permalink raw reply [flat|nested] 5+ messages in thread* [RFC PATCH 1/5] SUNRPC: Allow creation of RPC clients with multiple connections 2017-05-02 16:38 [PATCH 0/3] Fix up a couple of issues around layout handling Trond Myklebust @ 2017-05-02 16:38 ` Trond Myklebust 2017-05-02 16:38 ` [PATCH 1/3] pNFS: Don't clear the layout return info if there are segments to return Trond Myklebust 0 siblings, 1 reply; 5+ messages in thread From: Trond Myklebust @ 2017-05-02 16:38 UTC (permalink / raw) To: linux-nfs Add an argument to struct rpc_create_args that allows the specification of how many transport connections you want to set up to the server. Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> --- include/linux/sunrpc/clnt.h | 1 + net/sunrpc/clnt.c | 17 ++++++++++++++++- net/sunrpc/xprtmultipath.c | 3 +-- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h index 6095ecba0dde..8c3cb38a385b 100644 --- a/include/linux/sunrpc/clnt.h +++ b/include/linux/sunrpc/clnt.h @@ -120,6 +120,7 @@ struct rpc_create_args { u32 prognumber; /* overrides program->number */ u32 version; rpc_authflavor_t authflavor; + u32 nconnect; unsigned long flags; char *client_name; struct svc_xprt *bc_xprt; /* NFSv4.1 backchannel */ diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c index 673046c64e48..0ff97288b43f 100644 --- a/net/sunrpc/clnt.c +++ b/net/sunrpc/clnt.c @@ -522,6 +522,8 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args) .bc_xprt = args->bc_xprt, }; char servername[48]; + struct rpc_clnt *clnt; + int i; if (args->bc_xprt) { WARN_ON_ONCE(!(args->protocol & XPRT_TRANSPORT_BC)); @@ -584,7 +586,15 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args) if (args->flags & RPC_CLNT_CREATE_NONPRIVPORT) xprt->resvport = 0; - return rpc_create_xprt(args, xprt); + clnt = rpc_create_xprt(args, xprt); + if (IS_ERR(clnt) || args->nconnect <= 1) + return clnt; + + for (i = 0; i < args->nconnect - 1; i++) { + if (rpc_clnt_add_xprt(clnt, &xprtargs, NULL, NULL) < 0) + break; + } + return clnt; } EXPORT_SYMBOL_GPL(rpc_create); @@ -2605,6 +2615,10 @@ int rpc_clnt_test_and_add_xprt(struct rpc_clnt *clnt, return -ENOMEM; data->xps = xprt_switch_get(xps); data->xprt = xprt_get(xprt); + if (rpc_xprt_switch_has_addr(data->xps, (struct sockaddr *)&xprt->addr)) { + rpc_cb_add_xprt_release(data); + goto success; + } cred = authnull_ops.lookup_cred(NULL, NULL, 0); task = rpc_call_null_helper(clnt, xprt, cred, @@ -2614,6 +2628,7 @@ int rpc_clnt_test_and_add_xprt(struct rpc_clnt *clnt, if (IS_ERR(task)) return PTR_ERR(task); rpc_put_task(task); +success: return 1; } EXPORT_SYMBOL_GPL(rpc_clnt_test_and_add_xprt); diff --git a/net/sunrpc/xprtmultipath.c b/net/sunrpc/xprtmultipath.c index 95064d510ce6..486819d0c58b 100644 --- a/net/sunrpc/xprtmultipath.c +++ b/net/sunrpc/xprtmultipath.c @@ -51,8 +51,7 @@ void rpc_xprt_switch_add_xprt(struct rpc_xprt_switch *xps, if (xprt == NULL) return; spin_lock(&xps->xps_lock); - if ((xps->xps_net == xprt->xprt_net || xps->xps_net == NULL) && - !rpc_xprt_switch_has_addr(xps, (struct sockaddr *)&xprt->addr)) + if (xps->xps_net == xprt->xprt_net || xps->xps_net == NULL) xprt_switch_add_xprt_locked(xps, xprt); spin_unlock(&xps->xps_lock); } -- 2.9.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 1/3] pNFS: Don't clear the layout return info if there are segments to return 2017-05-02 16:38 ` [RFC PATCH 1/5] SUNRPC: Allow creation of RPC clients with multiple connections Trond Myklebust @ 2017-05-02 16:38 ` Trond Myklebust 0 siblings, 0 replies; 5+ messages in thread From: Trond Myklebust @ 2017-05-02 16:38 UTC (permalink / raw) To: linux-nfs In pnfs_clear_layoutreturn_info, ensure that we don't clear the layout return info if there are new segments queued for return due to, for instance, a race between a LAYOUTRETURN and a failed I/O attempt. Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> --- fs/nfs/pnfs.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index 140ecd7d350f..cea1e838efae 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -322,9 +322,15 @@ pnfs_set_plh_return_info(struct pnfs_layout_hdr *lo, enum pnfs_iomode iomode, static void pnfs_clear_layoutreturn_info(struct pnfs_layout_hdr *lo) { + struct pnfs_layout_segment *lseg; lo->plh_return_iomode = 0; lo->plh_return_seq = 0; clear_bit(NFS_LAYOUT_RETURN_REQUESTED, &lo->plh_flags); + list_for_each_entry(lseg, &lo->plh_segs, pls_list) { + if (!test_bit(NFS_LSEG_LAYOUTRETURN, &lseg->pls_flags)) + continue; + pnfs_set_plh_return_info(lo, lseg->pls_range.iomode, 0); + } } static void pnfs_clear_layoutreturn_waitbit(struct pnfs_layout_hdr *lo) @@ -367,9 +373,9 @@ pnfs_mark_layout_stateid_invalid(struct pnfs_layout_hdr *lo, struct pnfs_layout_segment *lseg, *next; set_bit(NFS_LAYOUT_INVALID_STID, &lo->plh_flags); - pnfs_clear_layoutreturn_info(lo); list_for_each_entry_safe(lseg, next, &lo->plh_segs, pls_list) pnfs_clear_lseg_state(lseg, lseg_list); + pnfs_clear_layoutreturn_info(lo); pnfs_free_returned_lsegs(lo, lseg_list, &range, 0); if (test_bit(NFS_LAYOUT_RETURN, &lo->plh_flags) && !test_and_set_bit(NFS_LAYOUT_RETURN_LOCK, &lo->plh_flags)) -- 2.9.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-05-02 16:41 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-05-02 16:41 [PATCH 0/3] Fix up a couple of issues around layout handling Trond Myklebust 2017-05-02 16:41 ` [PATCH 1/3] pNFS: Don't clear the layout return info if there are segments to return Trond Myklebust 2017-05-02 16:41 ` [PATCH 2/3] pNFS: Fix a deadlock when coalescing writes and returning the layout Trond Myklebust 2017-05-02 16:41 ` [PATCH 3/3] pNFS: Fix a typo in pnfs_generic_alloc_ds_commits Trond Myklebust -- strict thread matches above, loose matches on Subject: below -- 2017-05-02 16:38 [PATCH 0/3] Fix up a couple of issues around layout handling Trond Myklebust 2017-05-02 16:38 ` [RFC PATCH 1/5] SUNRPC: Allow creation of RPC clients with multiple connections Trond Myklebust 2017-05-02 16:38 ` [PATCH 1/3] pNFS: Don't clear the layout return info if there are segments to return Trond Myklebust
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).