* [PATCH 0/5] Fix up delegation management
@ 2020-01-27 14:58 Trond Myklebust
2020-01-27 14:58 ` [PATCH 1/5] NFSv4: nfs_inode_evict_delegation() should set NFS_DELEGATION_RETURNING Trond Myklebust
0 siblings, 1 reply; 6+ messages in thread
From: Trond Myklebust @ 2020-01-27 14:58 UTC (permalink / raw)
To: Anna Schumaker; +Cc: linux-nfs
Delegations allow the client to cache various objects aggressively,
however they can also be a source of scalability issues on the server
if not returned when not it use.
The following patchset allows the client to set a limit on the number
of delegations that it holds. Once it hits that limit, it starts to
return the delegations on close in order to avoid swamping the server
with state.
Trond Myklebust (5):
NFSv4: nfs_inode_evict_delegation() should set
NFS_DELEGATION_RETURNING
NFS: Clear NFS_DELEGATION_RETURN_IF_CLOSED when the delegation is
returned
NFSv4: Try to return the delegation immediately when marked for return
on close
NFSv4: Add accounting for the number of active delegations held
NFSv4: Limit the total number of cached delegations
fs/nfs/delegation.c | 80 +++++++++++++++++++++++++++++++++++++--------
fs/nfs/delegation.h | 1 +
fs/nfs/nfs4state.c | 1 +
3 files changed, 69 insertions(+), 13 deletions(-)
--
2.24.1
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/5] NFSv4: nfs_inode_evict_delegation() should set NFS_DELEGATION_RETURNING 2020-01-27 14:58 [PATCH 0/5] Fix up delegation management Trond Myklebust @ 2020-01-27 14:58 ` Trond Myklebust 2020-01-27 14:58 ` [PATCH 2/5] NFS: Clear NFS_DELEGATION_RETURN_IF_CLOSED when the delegation is returned Trond Myklebust 0 siblings, 1 reply; 6+ messages in thread From: Trond Myklebust @ 2020-01-27 14:58 UTC (permalink / raw) To: Anna Schumaker; +Cc: linux-nfs In particular, the pnfs return-on-close code will check for that flag, so ensure we set it appropriately. Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com> --- fs/nfs/delegation.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c index fe57b2b5314a..a7e42725c3b1 100644 --- a/fs/nfs/delegation.c +++ b/fs/nfs/delegation.c @@ -607,6 +607,7 @@ void nfs_inode_evict_delegation(struct inode *inode) delegation = nfs_inode_detach_delegation(inode); if (delegation != NULL) { + set_bit(NFS_DELEGATION_RETURNING, &delegation->flags); set_bit(NFS_DELEGATION_INODE_FREEING, &delegation->flags); nfs_do_return_delegation(inode, delegation, 1); nfs_free_delegation(delegation); -- 2.24.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/5] NFS: Clear NFS_DELEGATION_RETURN_IF_CLOSED when the delegation is returned 2020-01-27 14:58 ` [PATCH 1/5] NFSv4: nfs_inode_evict_delegation() should set NFS_DELEGATION_RETURNING Trond Myklebust @ 2020-01-27 14:58 ` Trond Myklebust 2020-01-27 14:58 ` [PATCH 3/5] NFSv4: Try to return the delegation immediately when marked for return on close Trond Myklebust 0 siblings, 1 reply; 6+ messages in thread From: Trond Myklebust @ 2020-01-27 14:58 UTC (permalink / raw) To: Anna Schumaker; +Cc: linux-nfs If a delegation is marked as needing to be returned when the file is closed, then don't clear that marking until we're ready to return it. Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com> --- fs/nfs/delegation.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c index a7e42725c3b1..b5b14618b73e 100644 --- a/fs/nfs/delegation.c +++ b/fs/nfs/delegation.c @@ -479,7 +479,7 @@ static bool nfs_delegation_need_return(struct nfs_delegation *delegation) if (test_and_clear_bit(NFS_DELEGATION_RETURN, &delegation->flags)) ret = true; - if (test_and_clear_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags) && !ret) { + else if (test_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags)) { struct inode *inode; spin_lock(&delegation->lock); @@ -488,6 +488,8 @@ static bool nfs_delegation_need_return(struct nfs_delegation *delegation) ret = true; spin_unlock(&delegation->lock); } + if (ret) + clear_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags); if (test_bit(NFS_DELEGATION_RETURNING, &delegation->flags) || test_bit(NFS_DELEGATION_REVOKED, &delegation->flags)) ret = false; -- 2.24.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/5] NFSv4: Try to return the delegation immediately when marked for return on close 2020-01-27 14:58 ` [PATCH 2/5] NFS: Clear NFS_DELEGATION_RETURN_IF_CLOSED when the delegation is returned Trond Myklebust @ 2020-01-27 14:58 ` Trond Myklebust 2020-01-27 14:58 ` [PATCH 4/5] NFSv4: Add accounting for the number of active delegations held Trond Myklebust 0 siblings, 1 reply; 6+ messages in thread From: Trond Myklebust @ 2020-01-27 14:58 UTC (permalink / raw) To: Anna Schumaker; +Cc: linux-nfs Add a routine to return the delegation immediately upon close of the file if it was marked for return-on-close. Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com> --- fs/nfs/delegation.c | 33 +++++++++++++++++++++++++++++++++ fs/nfs/delegation.h | 1 + fs/nfs/nfs4state.c | 1 + 3 files changed, 35 insertions(+) diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c index b5b14618b73e..90e50f32f3e0 100644 --- a/fs/nfs/delegation.c +++ b/fs/nfs/delegation.c @@ -639,6 +639,39 @@ int nfs4_inode_return_delegation(struct inode *inode) return err; } +/** + * nfs_inode_return_delegation_on_close - asynchronously return a delegation + * @inode: inode to process + * + * This routine is called on file close in order to determine if the + * inode delegation needs to be returned immediately. + */ +void nfs4_inode_return_delegation_on_close(struct inode *inode) +{ + struct nfs_delegation *delegation; + struct nfs_delegation *ret = NULL; + + if (!inode) + return; + rcu_read_lock(); + delegation = nfs4_get_valid_delegation(inode); + if (!delegation) + goto out; + if (test_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags)) { + spin_lock(&delegation->lock); + if (delegation->inode && + list_empty(&NFS_I(inode)->open_files) && + !test_and_set_bit(NFS_DELEGATION_RETURNING, &delegation->flags)) { + clear_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags); + ret = delegation; + } + spin_unlock(&delegation->lock); + } +out: + rcu_read_unlock(); + nfs_end_delegation_return(inode, ret, 0); +} + /** * nfs4_inode_make_writeable * @inode: pointer to inode diff --git a/fs/nfs/delegation.h b/fs/nfs/delegation.h index 15d3484be028..31b84604d383 100644 --- a/fs/nfs/delegation.h +++ b/fs/nfs/delegation.h @@ -42,6 +42,7 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred, void nfs_inode_reclaim_delegation(struct inode *inode, const struct cred *cred, fmode_t type, const nfs4_stateid *stateid, unsigned long pagemod_limit); int nfs4_inode_return_delegation(struct inode *inode); +void nfs4_inode_return_delegation_on_close(struct inode *inode); int nfs_async_inode_return_delegation(struct inode *inode, const nfs4_stateid *stateid); void nfs_inode_evict_delegation(struct inode *inode); diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c index 34552329233d..958ed4e4cde2 100644 --- a/fs/nfs/nfs4state.c +++ b/fs/nfs/nfs4state.c @@ -766,6 +766,7 @@ void nfs4_put_open_state(struct nfs4_state *state) list_del(&state->open_states); spin_unlock(&inode->i_lock); spin_unlock(&owner->so_lock); + nfs4_inode_return_delegation_on_close(inode); iput(inode); nfs4_free_open_state(state); nfs4_put_state_owner(owner); -- 2.24.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 4/5] NFSv4: Add accounting for the number of active delegations held 2020-01-27 14:58 ` [PATCH 3/5] NFSv4: Try to return the delegation immediately when marked for return on close Trond Myklebust @ 2020-01-27 14:58 ` Trond Myklebust 2020-01-27 14:58 ` [PATCH 5/5] NFSv4: Limit the total number of cached delegations Trond Myklebust 0 siblings, 1 reply; 6+ messages in thread From: Trond Myklebust @ 2020-01-27 14:58 UTC (permalink / raw) To: Anna Schumaker; +Cc: linux-nfs In order to better manage our delegation caching, add a counter to track the number of active delegations. Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com> --- fs/nfs/delegation.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c index 90e50f32f3e0..a777b3d0e720 100644 --- a/fs/nfs/delegation.c +++ b/fs/nfs/delegation.c @@ -25,13 +25,29 @@ #include "internal.h" #include "nfs4trace.h" -static void nfs_free_delegation(struct nfs_delegation *delegation) +static atomic_long_t nfs_active_delegations; + +static void __nfs_free_delegation(struct nfs_delegation *delegation) { put_cred(delegation->cred); delegation->cred = NULL; kfree_rcu(delegation, rcu); } +static void nfs_mark_delegation_revoked(struct nfs_delegation *delegation) +{ + if (!test_and_set_bit(NFS_DELEGATION_REVOKED, &delegation->flags)) { + delegation->stateid.type = NFS4_INVALID_STATEID_TYPE; + atomic_long_dec(&nfs_active_delegations); + } +} + +static void nfs_free_delegation(struct nfs_delegation *delegation) +{ + nfs_mark_delegation_revoked(delegation); + __nfs_free_delegation(delegation); +} + /** * nfs_mark_delegation_referenced - set delegation's REFERENCED flag * @delegation: delegation to process @@ -343,7 +359,8 @@ nfs_update_inplace_delegation(struct nfs_delegation *delegation, delegation->stateid.seqid = update->stateid.seqid; smp_wmb(); delegation->type = update->type; - clear_bit(NFS_DELEGATION_REVOKED, &delegation->flags); + if (test_and_clear_bit(NFS_DELEGATION_REVOKED, &delegation->flags)) + atomic_long_inc(&nfs_active_delegations); } } @@ -423,6 +440,8 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred, rcu_assign_pointer(nfsi->delegation, delegation); delegation = NULL; + atomic_long_inc(&nfs_active_delegations); + trace_nfs4_set_delegation(inode, type); spin_lock(&inode->i_lock); @@ -432,7 +451,7 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred, out: spin_unlock(&clp->cl_lock); if (delegation != NULL) - nfs_free_delegation(delegation); + __nfs_free_delegation(delegation); if (freeme != NULL) { nfs_do_return_delegation(inode, freeme, 0); nfs_free_delegation(freeme); @@ -796,13 +815,6 @@ static void nfs_client_mark_return_unused_delegation_types(struct nfs_client *cl rcu_read_unlock(); } -static void nfs_mark_delegation_revoked(struct nfs_server *server, - struct nfs_delegation *delegation) -{ - set_bit(NFS_DELEGATION_REVOKED, &delegation->flags); - delegation->stateid.type = NFS4_INVALID_STATEID_TYPE; -} - static void nfs_revoke_delegation(struct inode *inode, const nfs4_stateid *stateid) { @@ -830,7 +842,7 @@ static void nfs_revoke_delegation(struct inode *inode, } spin_unlock(&delegation->lock); } - nfs_mark_delegation_revoked(NFS_SERVER(inode), delegation); + nfs_mark_delegation_revoked(delegation); ret = true; out: rcu_read_unlock(); @@ -869,7 +881,7 @@ void nfs_delegation_mark_returned(struct inode *inode, delegation->stateid.seqid = stateid->seqid; } - nfs_mark_delegation_revoked(NFS_SERVER(inode), delegation); + nfs_mark_delegation_revoked(delegation); out_clear_returning: clear_bit(NFS_DELEGATION_RETURNING, &delegation->flags); -- 2.24.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 5/5] NFSv4: Limit the total number of cached delegations 2020-01-27 14:58 ` [PATCH 4/5] NFSv4: Add accounting for the number of active delegations held Trond Myklebust @ 2020-01-27 14:58 ` Trond Myklebust 0 siblings, 0 replies; 6+ messages in thread From: Trond Myklebust @ 2020-01-27 14:58 UTC (permalink / raw) To: Anna Schumaker; +Cc: linux-nfs Delegations can be expensive to return, and can cause scalability issues for the server. Let's therefore try to limit the number of inactive delegations we hold. Once the number of delegations is above a certain threshold, start to return them on close. Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com> --- fs/nfs/delegation.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c index a777b3d0e720..4a841071d8a7 100644 --- a/fs/nfs/delegation.c +++ b/fs/nfs/delegation.c @@ -25,7 +25,10 @@ #include "internal.h" #include "nfs4trace.h" +#define NFS_DEFAULT_DELEGATION_WATERMARK (5000U) + static atomic_long_t nfs_active_delegations; +static unsigned nfs_delegation_watermark = NFS_DEFAULT_DELEGATION_WATERMARK; static void __nfs_free_delegation(struct nfs_delegation *delegation) { @@ -676,7 +679,8 @@ void nfs4_inode_return_delegation_on_close(struct inode *inode) delegation = nfs4_get_valid_delegation(inode); if (!delegation) goto out; - if (test_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags)) { + if (test_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags) || + atomic_long_read(&nfs_active_delegations) >= nfs_delegation_watermark) { spin_lock(&delegation->lock); if (delegation->inode && list_empty(&NFS_I(inode)->open_files) && @@ -1365,3 +1369,5 @@ bool nfs4_delegation_flush_on_close(const struct inode *inode) rcu_read_unlock(); return ret; } + +module_param_named(delegation_watermark, nfs_delegation_watermark, uint, 0644); -- 2.24.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-01-27 15:00 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-01-27 14:58 [PATCH 0/5] Fix up delegation management Trond Myklebust 2020-01-27 14:58 ` [PATCH 1/5] NFSv4: nfs_inode_evict_delegation() should set NFS_DELEGATION_RETURNING Trond Myklebust 2020-01-27 14:58 ` [PATCH 2/5] NFS: Clear NFS_DELEGATION_RETURN_IF_CLOSED when the delegation is returned Trond Myklebust 2020-01-27 14:58 ` [PATCH 3/5] NFSv4: Try to return the delegation immediately when marked for return on close Trond Myklebust 2020-01-27 14:58 ` [PATCH 4/5] NFSv4: Add accounting for the number of active delegations held Trond Myklebust 2020-01-27 14:58 ` [PATCH 5/5] NFSv4: Limit the total number of cached delegations Trond Myklebust
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).