automatic freeing of space on ENOSPC

automatic freeing of space on ENOSPC

[J. Bruce Fields]

Is there anything analogous to a "shrinker", but for disk space?  So,
some hook that a filesystem could call to say "I'm running out of space,
could you please free something?", before giving up and returning
ENOSPC?

The NFS server currently revokes a client's state if the client fails to
contact it within a lease period (90 seconds by default).  That's
harsher than necessary--if a network partition lasts longer than a lease
period, but if nobody else needs that client's resources, it'd be nice
to be able to hang on to them so that the client could resume normal
operation after the network comes back.  So we'd delay revoking the
client's state until there's an actual conflict.  But that means we need
a way to clean up the client as soon as there is a conflict, to avoid
unnecessarily failing operations that conflict with resources held by an
expired client.

At first I thought we only needed to worry about file locks, but then I
realized clients can also hold references to files, which might be
unlinked.  I don't want a long-expired client to result in ENOSPC to
other filesystem users.

Any ideas?

I searched around and found this discussion of volatile ranges
https://lwn.net/Articles/522135/, which seems close, but I don't know if
anything came of that in the end.

--b.

Re: automatic freeing of space on ENOSPC

[Trond Myklebust]

On Mon, 2021-06-28 at 15:49 -0400, J. Bruce Fields wrote:
> Is there anything analogous to a "shrinker", but for disk space?  So,
> some hook that a filesystem could call to say "I'm running out of
> space,
> could you please free something?", before giving up and returning
> ENOSPC?
> 
> The NFS server currently revokes a client's state if the client fails
> to
> contact it within a lease period (90 seconds by default).  That's
> harsher than necessary--if a network partition lasts longer than a
> lease
> period, but if nobody else needs that client's resources, it'd be
> nice
> to be able to hang on to them so that the client could resume normal
> operation after the network comes back.  So we'd delay revoking the
> client's state until there's an actual conflict.  But that means we
> need
> a way to clean up the client as soon as there is a conflict, to avoid
> unnecessarily failing operations that conflict with resources held by
> an
> expired client.
> 
> At first I thought we only needed to worry about file locks, but then
> I
> realized clients can also hold references to files, which might be
> unlinked.  I don't want a long-expired client to result in ENOSPC to
> other filesystem users.
> 
> Any ideas?
> 

How about just setting up a notification for unlink on those files, the
same way we set up notifications for close with the NFSv3 filecache in
nfsd?

> I searched around and found this discussion of volatile ranges
> https://lwn.net/Articles/522135/, which seems close, but I don't know
> if
> anything came of that in the end.
> 
> --b.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: automatic freeing of space on ENOSPC

[bfields]

On Tue, Jun 29, 2021 at 12:43:14AM +0000, Trond Myklebust wrote:
> How about just setting up a notification for unlink on those files, the
> same way we set up notifications for close with the NFSv3 filecache in
> nfsd?

Yes, that'd probably work.  It'd be better if we didn't have to throw
away unlinked files when the client expires, but it'd still be an
incremental improvement over what we do now.

--b.

Re: automatic freeing of space on ENOSPC

[NeilBrown]

On Tue, 29 Jun 2021, bfields@fieldses.org wrote:
> On Tue, Jun 29, 2021 at 12:43:14AM +0000, Trond Myklebust wrote:
> > How about just setting up a notification for unlink on those files, the
> > same way we set up notifications for close with the NFSv3 filecache in
> > nfsd?
> 
> Yes, that'd probably work.  It'd be better if we didn't have to throw
> away unlinked files when the client expires, but it'd still be an
> incremental improvement over what we do now.

I wonder how important this is.  If an NFS client unlinks a file that it
has open, it will be silly_renamed, and if the client then goes silent,
it might never be removed.  So we already theoretically have a
possibilty of ENOSPC due to silent clients.  Have we heard of this
becoming a problem?
Is there reason to think that the Courteous server changes will make
this problem more likely? 

NeilBrown

Re: automatic freeing of space on ENOSPC

[Amir Goldstein]

On Tue, Jun 29, 2021 at 4:45 AM NeilBrown <neilb@suse.de> wrote:
>
> On Tue, 29 Jun 2021, bfields@fieldses.org wrote:
> > On Tue, Jun 29, 2021 at 12:43:14AM +0000, Trond Myklebust wrote:
> > > How about just setting up a notification for unlink on those files, the
> > > same way we set up notifications for close with the NFSv3 filecache in
> > > nfsd?
> >
> > Yes, that'd probably work.  It'd be better if we didn't have to throw
> > away unlinked files when the client expires, but it'd still be an
> > incremental improvement over what we do now.
>
> I wonder how important this is.  If an NFS client unlinks a file that it
> has open, it will be silly_renamed, and if the client then goes silent,
> it might never be removed.  So we already theoretically have a
> possibilty of ENOSPC due to silent clients.  Have we heard of this
> becoming a problem?
> Is there reason to think that the Courteous server changes will make
> this problem more likely?
>

To me, stale silly renamed files sounds like a problem worth fixing
not as an excuse to create another similar problem.

w.r.t pre-ENOSPC notification, I don't know of such notification
in filesystems. It exists for some thin-provisioned storage devices
(thinp as well I think), but that is not very useful for nfsd.

OTOH, ENOSPC is rarely a surprising event.
I believe you can get away with tunable for nfsd, such as
% of available storage space that may consumed for
"opportunistic caching".

Polling for available storage space every least time or so
in case there are possibly forgotten unlinked files should be
sufficient for any practical purpose IMO.

Thanks,
Amir.

Re: automatic freeing of space on ENOSPC

[bfields]

On Tue, Jun 29, 2021 at 07:07:47AM +0300, Amir Goldstein wrote:
> To me, stale silly renamed files sounds like a problem worth fixing
> not as an excuse to create another similar problem.

Yeah, it's ugly, I'd like to fix it some day, but given that people have
lived with it since forever it's not the highest priority.

> w.r.t pre-ENOSPC notification, I don't know of such notification
> in filesystems. It exists for some thin-provisioned storage devices
> (thinp as well I think), but that is not very useful for nfsd.
> 
> OTOH, ENOSPC is rarely a surprising event.
> I believe you can get away with tunable for nfsd, such as
> % of available storage space that may consumed for
> "opportunistic caching".
> 
> Polling for available storage space every least time or so
> in case there are possibly forgotten unlinked files should be
> sufficient for any practical purpose IMO.

Makes sense, thanks for the perspective.

--b.

Re: automatic freeing of space on ENOSPC

[bfields]

On Tue, Jun 29, 2021 at 11:43:45AM +1000, NeilBrown wrote:
> I wonder how important this is.  If an NFS client unlinks a file that it
> has open, it will be silly_renamed, and if the client then goes silent,
> it might never be removed.  So we already theoretically have a
> possibilty of ENOSPC due to silent clients.  Have we heard of this
> becoming a problem?

Oh, that's a good point.  I've seen complaints about sillyrename files,
but I can't recall ever seen a single complaint about their causing
ENOSPC.

> Is there reason to think that the Courteous server changes will make
> this problem more likely? 

So I guess the only new cases the courteous server will introduce are
even less likely (probably just the case where a file is unlinked by a
client other than the one that has it open).

So I think doing nothing for now is an acceptable alternative....

--b.

Re: automatic freeing of space on ENOSPC

[Chuck Lever III]

> On Jun 29, 2021, at 2:32 PM, bfields@fieldses.org wrote:
> 
> On Tue, Jun 29, 2021 at 11:43:45AM +1000, NeilBrown wrote:
>> I wonder how important this is.  If an NFS client unlinks a file that it
>> has open, it will be silly_renamed, and if the client then goes silent,
>> it might never be removed.  So we already theoretically have a
>> possibilty of ENOSPC due to silent clients.  Have we heard of this
>> becoming a problem?
> 
> Oh, that's a good point.  I've seen complaints about sillyrename files,
> but I can't recall ever seen a single complaint about their causing
> ENOSPC.
> 
>> Is there reason to think that the Courteous server changes will make
>> this problem more likely? 
> 
> So I guess the only new cases the courteous server will introduce are
> even less likely (probably just the case where a file is unlinked by a
> client other than the one that has it open).
> 
> So I think doing nothing for now is an acceptable alternative....

I'm comfortable with that too. We can keep a careful eye on this
as the courteous server moves further along.

--
Chuck Lever

Re: automatic freeing of space on ENOSPC

[Dave Chinner]

On Mon, Jun 28, 2021 at 03:49:08PM -0400, J. Bruce Fields wrote:
> Is there anything analogous to a "shrinker", but for disk space?  So,
> some hook that a filesystem could call to say "I'm running out of space,
> could you please free something?", before giving up and returning
> ENOSPC?

The only thing the filesystem can do at this point is run internal
garbage collection operations to free up space that it has either
speculatively allocated or has deferred the freeing and/or cleanupi
of. In general, filesystems already do this when they are
approaching/at ENOSPC, so it seems to me like there's little scope
for an external "free some space" trigger to be able to make much
difference.

I mean, we do have an ioctl in XFS to trigger garbage collection of
speculative preallocation from userspace (XFS_IOC_FREE_EOFBLOCKS
and the prealloc command in xfs_spaceman(8)) but this was written
because some people want accurate space accounting for things like
quota reports....

Regardless, we run these same GC triggers internally before we
declare ENOSPC, so there's no real need for applications to run them
when they get an ENOSPC error....

> The NFS server currently revokes a client's state if the client fails to
> contact it within a lease period (90 seconds by default).  That's
> harsher than necessary--if a network partition lasts longer than a lease
> period, but if nobody else needs that client's resources, it'd be nice
> to be able to hang on to them so that the client could resume normal
> operation after the network comes back.  So we'd delay revoking the
> client's state until there's an actual conflict.  But that means we need
> a way to clean up the client as soon as there is a conflict, to avoid
> unnecessarily failing operations that conflict with resources held by an
> expired client.

I'm not sure what you are asking for filesystems to do here.  This
seems like an application problem - revoking the client's open file
state and cleaning up silly rename files is application level
garbage collection, not filesystem level stuff. Maybe I've
misunderstood what you are trying to do, perhaps you could clarify
what you're expecting the filesystems to be able to clean up here?

> I searched around and found this discussion of volatile ranges
> https://lwn.net/Articles/522135/, which seems close, but I don't know if
> anything came of that in the end.

Nothing that I know of.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: automatic freeing of space on ENOSPC

[J. Bruce Fields]

On Tue, Jun 29, 2021 at 03:11:49PM +1000, Dave Chinner wrote:
> On Mon, Jun 28, 2021 at 03:49:08PM -0400, J. Bruce Fields wrote:
> > The NFS server currently revokes a client's state if the client fails to
> > contact it within a lease period (90 seconds by default).  That's
> > harsher than necessary--if a network partition lasts longer than a lease
> > period, but if nobody else needs that client's resources, it'd be nice
> > to be able to hang on to them so that the client could resume normal
> > operation after the network comes back.  So we'd delay revoking the
> > client's state until there's an actual conflict.  But that means we need
> > a way to clean up the client as soon as there is a conflict, to avoid
> > unnecessarily failing operations that conflict with resources held by an
> > expired client.
> 
> I'm not sure what you are asking for filesystems to do here.  This
> seems like an application problem - revoking the client's open file
> state and cleaning up silly rename files is application level
> garbage collection, not filesystem level stuff.

Right, the "application" in this case is knfsd.  It may be keeping some
unlinked files around that it doesn't really need to.  So I'm basically
wondering if I could get a notification from the filesystem that now
would be a good time to close those files.

I think Neil's convinced me this isn't a priority, though....

--b.

Re: automatic freeing of space on ENOSPC

[Theodore Ts'o]

On Mon, Jun 28, 2021 at 03:49:08PM -0400, J. Bruce Fields wrote:
> Is there anything analogous to a "shrinker", but for disk space?  So,
> some hook that a filesystem could call to say "I'm running out of space,
> could you please free something?", before giving up and returning
> ENOSPC?

In addition to the issues raised by Neil, Amir, Dave, and others, the
other challenge with the file system calling a "please try to free
something before I return ENOSPC" is that this would almost certainly
require blocking a system call while some userspace daemon tried to
free up some space --- or were you thinking that the nfsd kernel code
would be tracking all of the silly-rename files so it could release
space really quickly on demand?

Even if this is only a kernel callback, I'd be concerned about
potential locking hierarchy problems if we are calling out from block
allocation subsystem to nfsd, only to have nfsd call back in to
request unlinking a silly-renamed file.

So the suggestion that we not wait until we're down to 0 blocks free,
but when we hit some threshold (say, free space dips below N minutes
worth of worst or average case block allocations), trigger code which
deletes silly-renamed files, is probably the best way to go.  In which
case, a callback is not what is needed; and if N is large enough, this
could done via a pure user-space-only solution.

		      	     		     - Ted

Re: automatic freeing of space on ENOSPC

[J. Bruce Fields]

On Tue, Jun 29, 2021 at 12:47:11PM -0400, Theodore Ts'o wrote:
> On Mon, Jun 28, 2021 at 03:49:08PM -0400, J. Bruce Fields wrote:
> > Is there anything analogous to a "shrinker", but for disk space?  So,
> > some hook that a filesystem could call to say "I'm running out of space,
> > could you please free something?", before giving up and returning
> > ENOSPC?
> 
> In addition to the issues raised by Neil, Amir, Dave, and others, the
> other challenge with the file system calling a "please try to free
> something before I return ENOSPC" is that this would almost certainly
> require blocking a system call while some userspace daemon tried to
> free up some space --- or were you thinking that the nfsd kernel code
> would be tracking all of the silly-rename files so it could release
> space really quickly on demand?

Something like that, yep.

> Even if this is only a kernel callback, I'd be concerned about
> potential locking hierarchy problems if we are calling out from block
> allocation subsystem to nfsd, only to have nfsd call back in to
> request unlinking a silly-renamed file.
> 
> So the suggestion that we not wait until we're down to 0 blocks free,
> but when we hit some threshold (say, free space dips below N minutes
> worth of worst or average case block allocations), trigger code which
> deletes silly-renamed files, is probably the best way to go.  In which
> case, a callback is not what is needed; and if N is large enough, this
> could done via a pure user-space-only solution.

Makes sense, thanks!

--b.