Making statfs return qgroup info (for container use case etc.)

Making statfs return qgroup info (for container use case etc.)

[Tim Small]

Hello,

I use btrfs for containers (e.g. full OS style containers using lxc/lxd
etc. rather than application containers).  btrfs de-duplication and
send/receive are very useful features for this use-case.

Each container runs in its own subvolume, and I use quotas to stop one
container exhausting the disk space of the others.

df shows the total space + free space for the entire filesystem - which
is confusing for users within the containers.  Worse - they don't have
any way of knowing how much quota they actually have left (other than du
-xs / vs. whatever I've told them).

It'd be really nice if running e.g. statfs("/home", ...) within a
container could be made to return the qgroup usage + limit for the path
(e.g. by passing in an option at mount time - such as qgroup level
maybe?) , instead of the global filesystem data in f_bfree f_blocks etc.

A useful tweak might be to only return the qgroup info within non-root
user namespaces, or perhaps just if the root UID != 0 (since the normal
use case for this is unprivileged containers), but I've no idea how
viable that idea is.

Yet another idea - (since btrfs always returns 0 for f_files - used for
inodes by other FS), then perhaps having the option of returning quota
information there would be useful (or maybe return quota referenced size
info in f_blocks and f_bfree, and exclusive size info in f_files and
f_free)?

Googling shows that this question has come up briefly on the #btrfs IRC
channel in the past.

Does sound like something that could be added to btrfs?

What I need to do currently only works with zfs, but I don't really want
to go there if I can avoid it...

ref:  search for btrfs in this web page:

https://www.stgraber.org/2016/03/26/lxd-2-0-resource-control-412/

... the shown working example is on zfs...

Tim.

Re: Making statfs return qgroup info (for container use case etc.)

[Dave Chinner]

On Thu, Oct 06, 2016 at 02:51:58PM +0100, Tim Small wrote:
> Hello,
> 
> I use btrfs for containers (e.g. full OS style containers using lxc/lxd
> etc. rather than application containers).  btrfs de-duplication and
> send/receive are very useful features for this use-case.
> 
> Each container runs in its own subvolume, and I use quotas to stop one
> container exhausting the disk space of the others.
> 
> df shows the total space + free space for the entire filesystem - which
> is confusing for users within the containers.  Worse - they don't have
> any way of knowing how much quota they actually have left (other than du
> -xs / vs. whatever I've told them).
> 
> It'd be really nice if running e.g. statfs("/home", ...) within a
> container could be made to return the qgroup usage + limit for the path
> (e.g. by passing in an option at mount time - such as qgroup level
> maybe?) , instead of the global filesystem data in f_bfree f_blocks etc.

XFS does this with directory tree quotas. It was implmented 10 years
ago or so, IIRC...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Making statfs return qgroup info (for container use case etc.)

[David Sterba]

On Thu, Oct 06, 2016 at 02:51:58PM +0100, Tim Small wrote:
> I use btrfs for containers (e.g. full OS style containers using lxc/lxd
> etc. rather than application containers).  btrfs de-duplication and
> send/receive are very useful features for this use-case.
> 
> Each container runs in its own subvolume, and I use quotas to stop one
> container exhausting the disk space of the others.
> 
> df shows the total space + free space for the entire filesystem - which
> is confusing for users within the containers.  Worse - they don't have
> any way of knowing how much quota they actually have left (other than du
> -xs / vs. whatever I've told them).
> 
> It'd be really nice if running e.g. statfs("/home", ...) within a
> container could be made to return the qgroup usage + limit for the path
> (e.g. by passing in an option at mount time - such as qgroup level
> maybe?) , instead of the global filesystem data in f_bfree f_blocks etc.

The container usecase will not go away anytime soon, and I think we can
implement the proposal.

Geting the limits of qgroup attached to a subvolume is lightweight
enough for the statfs call.  Statfs gets the dentry as a parameter, so
we can iterate up to the first subvolume which would be marked as "stop
here" and use the limits for statfs reporting. The remaining part is to
add a new ioctl to mark the subvolume 'stop here'.

For the first implementation, this would be just a runtime flag, ie.
manually set by the user. I'd need to check wheter the flag persistnce
would be ok.

> A useful tweak might be to only return the qgroup info within non-root
> user namespaces, or perhaps just if the root UID != 0 (since the normal
> use case for this is unprivileged containers), but I've no idea how
> viable that idea is.

I'm not sure if we should rely on an arbitrary UID to change the output.

> Yet another idea - (since btrfs always returns 0 for f_files - used for
> inodes by other FS), then perhaps having the option of returning quota
> information there would be useful (or maybe return quota referenced size
> info in f_blocks and f_bfree, and exclusive size info in f_files and
> f_free)?

That's a misuse, the programs reading df would special case to interpret
the values different from the documented meaning. But I think we can
avoid that.

> Googling shows that this question has come up briefly on the #btrfs IRC
> channel in the past.

Yeah, and by that time I didn't have an idea how to implement that.

> Does sound like something that could be added to btrfs?
> 
> What I need to do currently only works with zfs, but I don't really want
> to go there if I can avoid it...
> 
> ref:  search for btrfs in this web page:
> 
> https://www.stgraber.org/2016/03/26/lxd-2-0-resource-control-412/
> 
> ... the shown working example is on zfs...

Thanks for the link, that's actually useful as the usecase description.

Re: Making statfs return qgroup info (for container use case etc.)

[David Sterba]

On Fri, Oct 07, 2016 at 09:40:11AM +1100, Dave Chinner wrote:
> XFS does this with directory tree quotas. It was implmented 10 years
> ago or so, IIRC...

Sometimes, the line between a historical remark and trolling is thin....

Re: Making statfs return qgroup info (for container use case etc.)

[Dave Chinner]

On Fri, Oct 07, 2016 at 06:58:47PM +0200, David Sterba wrote:
> On Fri, Oct 07, 2016 at 09:40:11AM +1100, Dave Chinner wrote:
> > XFS does this with directory tree quotas. It was implmented 10 years
> > ago or so, IIRC...
> 
> Sometimes, the line between a historical remark and trolling is thin....

All I've done is quickly point out that we've already got an
implementation of the exact functionality being asked for and given
a rough indication of when the commits hit the tree. i.e. so anyone
wanting to implment this in btrfs can search for it easily and work
out how to use the existing VFS infrastructure to report this
information.

You can take that as trolling if you want, but if you think I have
time for playing stupid, petty, idiotic games like that then you
need to ask yourself why you have had such a defensive and
paranoid reaction...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com