"quota-df" feature

"quota-df" feature

[Chengguang Xu]

Hello folks,

Recently I'm trying to implement a feature on overlayfs for our users.
I'd like to call it "quota-df" which means when calling statfs(2),
override/update some fields(e.g., total/used/avail of both block/inode counts）
using proper project quota information if the quota has already set to
underlying directories of overlayfs.

This feature is probably useful for most of container users and there is
no serious side effect to the other parts. We can also provide a new
mount option to enable/disable the function according to different use cases.

Although I'm still working on it but I'll post the RFC patch here for review
in recent days and hope to get some feedback from you. Any kind of suggestions
are welcomed.


Here are some test examples share with you (based on ext4):

1) project quota enabled && without hard-limit

$ df -h /mnt/test3 /mnt/test3/df/merged
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb3        99G  2.5G   91G   3% /mnt/test3
overlay          92G  201M   91G   1% /mnt/test3/df/merged

$ df -hi /mnt/test3 /mnt/test3/df/merged
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged

2) project quota enabled && with hard-limit

$ df -h /mnt/test3 /mnt/test3/df/merged
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb3        99G  2.5G   91G   3% /mnt/test3
overlay         1.0G  201M  824M  20% /mnt/test3/df/merged

$ df -hi /mnt/test3 /mnt/test3/df/merged
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
overlay          1000     8   992    1% /mnt/test3/df/merged


Thanks,
Chengguang.

Re: "quota-df" feature

[Amir Goldstein]

On Wed, Mar 7, 2018 at 2:52 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
> Hello folks,
>
> Recently I'm trying to implement a feature on overlayfs for our users.
> I'd like to call it "quota-df" which means when calling statfs(2),
> override/update some fields(e.g., total/used/avail of both block/inode counts）
> using proper project quota information if the quota has already set to
> underlying directories of overlayfs.
>

Do you mean project quota setting on underlying fs directories or
something else?


> This feature is probably useful for most of container users and there is
> no serious side effect to the other parts. We can also provide a new
> mount option to enable/disable the function according to different use cases.
>
> Although I'm still working on it but I'll post the RFC patch here for review
> in recent days and hope to get some feedback from you. Any kind of suggestions
> are welcomed.
>

I was asked about quota accounting for the merged content of overalyfs,
so quota would display a number resembling du -s on the overlay mount.
This means that "covered" lower files are not accounted for in the merge.
Is that what you aim to achieve? If not, would that also be an interesting case
for you to explore?
Basically, the index directory could be used to reconstruct the entire quota
information for upper and lower (and copied up) files if full indexing
is enabled
with nfs_export=on.

>
> Here are some test examples share with you (based on ext4):
>
> 1) project quota enabled && without hard-limit
>
> $ df -h /mnt/test3 /mnt/test3/df/merged
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> overlay          92G  201M   91G   1% /mnt/test3/df/merged
>
> $ df -hi /mnt/test3 /mnt/test3/df/merged
> Filesystem     Inodes IUsed IFree IUse% Mounted on
> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>
> 2) project quota enabled && with hard-limit
>
> $ df -h /mnt/test3 /mnt/test3/df/merged
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>
> $ df -hi /mnt/test3 /mnt/test3/df/merged
> Filesystem     Inodes IUsed IFree IUse% Mounted on
> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> overlay          1000     8   992    1% /mnt/test3/df/merged
>
>

I can't follow from the example below what is the expected result and why
if you add the quota setup commands that could be useful.

Thanks,
Amir.

Re: "quota-df" feature

[Chengguang Xu]

> Sent: Wednesday, March 07, 2018 at 9:28 PM
> From: "Amir Goldstein" <amir73il@gmail.com>
> To: "Chengguang Xu" <cgxu519@gmx.com>
> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> Subject: Re: "quota-df" feature
>
> On Wed, Mar 7, 2018 at 2:52 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
> > Hello folks,
> >
> > Recently I'm trying to implement a feature on overlayfs for our users.
> > I'd like to call it "quota-df" which means when calling statfs(2),
> > override/update some fields(e.g., total/used/avail of both block/inode counts）
> > using proper project quota information if the quota has already set to
> > underlying directories of overlayfs.
> >
> 
> Do you mean project quota setting on underlying fs directories or
> something else?

Underlying fs directories, more accurately the directory include
upperdir && workdir.

> 
> > This feature is probably useful for most of container users and there is
> > no serious side effect to the other parts. We can also provide a new
> > mount option to enable/disable the function according to different use cases.
> >
> > Although I'm still working on it but I'll post the RFC patch here for review
> > in recent days and hope to get some feedback from you. Any kind of suggestions
> > are welcomed.
> >
> 
> I was asked about quota accounting for the merged content of overalyfs,
> so quota would display a number resembling du -s on the overlay mount.
> This means that "covered" lower files are not accounted for in the merge.
> Is that what you aim to achieve? If not, would that also be an interesting case
> for you to explore?
> Basically, the index directory could be used to reconstruct the entire quota
> information for upper and lower (and copied up) files if full indexing
> is enabled
> with nfs_export=on.

To be honest, our use case is standard docker container with overlayfs and
our requirement of quota-df is accurately counting usage/avail inside container.

Lowerdirs could be share with different overlayfs, so I'm not sure counting
the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
to one overlayfs then maybe setting same project id with upperdir can resolve
"covered" problem that you mentioned above. Counting usage information without
quota might be a hard work when having plenty of files.

I don't know exactly what will happen when combining index and nfs_export options,
I need to read and understand related code later.

> 
> >
> > Here are some test examples share with you (based on ext4):
> >
> > 1) project quota enabled && without hard-limit
> >
> > $ df -h /mnt/test3 /mnt/test3/df/merged
> > Filesystem      Size  Used Avail Use% Mounted on
> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> > overlay          92G  201M   91G   1% /mnt/test3/df/merged
> >
> > $ df -hi /mnt/test3 /mnt/test3/df/merged
> > Filesystem     Inodes IUsed IFree IUse% Mounted on
> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> > overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
> >
> > 2) project quota enabled && with hard-limit
> >
> > $ df -h /mnt/test3 /mnt/test3/df/merged
> > Filesystem      Size  Used Avail Use% Mounted on
> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> > overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
> >
> > $ df -hi /mnt/test3 /mnt/test3/df/merged
> > Filesystem     Inodes IUsed IFree IUse% Mounted on
> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> > overlay          1000     8   992    1% /mnt/test3/df/merged
> >
> >
> 
> I can't follow from the example below what is the expected result and why
> if you add the quota setup commands that could be useful.

Underlying fs using ext4 and mkfs/mount with project quota option and
all needed directories(lowerdir, upperdir, workdir, merged) are set same
project quota. Current quota-df implementation only adjust the couting
information when having upperdir.

In case 1:
There is no hardlimit/softlimit, so the expected result as below.
Used:  The used count in project quota which set to the directory
       include upperdir && workdir.
Avail: The avail of underlying fs
Total: Used + Avail

#upperdir used 201M and /mnt/test3 used 2.5G

In case 2:
Project quota hardlimits are block count = 1G, inode count = 1000.
So the expected result as below.

Used:  The used count in project quota which set to the directory
       include upperdir && workdir.
Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
Total: Used + Avail

Thanks,
Chengguang.

Re: "quota-df" feature

[Amir Goldstein]

On Thu, Mar 8, 2018 at 4:21 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> Sent: Wednesday, March 07, 2018 at 9:28 PM
>> From: "Amir Goldstein" <amir73il@gmail.com>
>> To: "Chengguang Xu" <cgxu519@gmx.com>
>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> Subject: Re: "quota-df" feature
>>
>> On Wed, Mar 7, 2018 at 2:52 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> > Hello folks,
>> >
>> > Recently I'm trying to implement a feature on overlayfs for our users.
>> > I'd like to call it "quota-df" which means when calling statfs(2),
>> > override/update some fields(e.g., total/used/avail of both block/inode counts）
>> > using proper project quota information if the quota has already set to
>> > underlying directories of overlayfs.
>> >
>>
>> Do you mean project quota setting on underlying fs directories or
>> something else?
>
> Underlying fs directories, more accurately the directory include
> upperdir && workdir.
>
>>
>> > This feature is probably useful for most of container users and there is
>> > no serious side effect to the other parts. We can also provide a new
>> > mount option to enable/disable the function according to different use cases.
>> >
>> > Although I'm still working on it but I'll post the RFC patch here for review
>> > in recent days and hope to get some feedback from you. Any kind of suggestions
>> > are welcomed.
>> >
>>
>> I was asked about quota accounting for the merged content of overalyfs,
>> so quota would display a number resembling du -s on the overlay mount.
>> This means that "covered" lower files are not accounted for in the merge.
>> Is that what you aim to achieve? If not, would that also be an interesting case
>> for you to explore?
>> Basically, the index directory could be used to reconstruct the entire quota
>> information for upper and lower (and copied up) files if full indexing
>> is enabled
>> with nfs_export=on.
>
> To be honest, our use case is standard docker container with overlayfs and
> our requirement of quota-df is accurately counting usage/avail inside container.
>

My confusion is because I thought this was working correctly today.
When I posted docker support for project quota over year ago I remeber
the tests inside container showed reasonable behavior:
https://github.com/moby/moby/pull/24771

So what I am missing is "what is wrong now?".

> Lowerdirs could be share with different overlayfs, so I'm not sure counting
> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
> to one overlayfs then maybe setting same project id with upperdir can resolve
> "covered" problem that you mentioned above. Counting usage information without
> quota might be a hard work when having plenty of files.
>

Well, its a different use case, but a very well known use case -
When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
blocks, but each clone is fully accounted to the user/group/project.
This is the "thin provisioning" use case - every user gets accounted by files
that the user can reference, but the host does not pay the cost of sum of all
user quotas.

Without accounting of "covered" files to begin with, is it possible to
get to a state
where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
provisioned block,
but a situation that filesystems and administrators try really hard to avoid.

All I am saying is that it is "not hard" (TM) to keep track of
"covered" files disk usage
and "not hard" to re-calculate "covered" files disk usage when full
indexing is enabled.

> I don't know exactly what will happen when combining index and nfs_export options,
> I need to read and understand related code later.
>

nfs_export REQUIRES index and implies indexing of all files on copy up, not
only lower hardlinks.

>>
>> >
>> > Here are some test examples share with you (based on ext4):
>> >
>> > 1) project quota enabled && without hard-limit
>> >
>> > $ df -h /mnt/test3 /mnt/test3/df/merged
>> > Filesystem      Size  Used Avail Use% Mounted on
>> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>> > overlay          92G  201M   91G   1% /mnt/test3/df/merged
>> >
>> > $ df -hi /mnt/test3 /mnt/test3/df/merged
>> > Filesystem     Inodes IUsed IFree IUse% Mounted on
>> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>> > overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>> >
>> > 2) project quota enabled && with hard-limit
>> >
>> > $ df -h /mnt/test3 /mnt/test3/df/merged
>> > Filesystem      Size  Used Avail Use% Mounted on
>> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>> > overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>> >
>> > $ df -hi /mnt/test3 /mnt/test3/df/merged
>> > Filesystem     Inodes IUsed IFree IUse% Mounted on
>> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>> > overlay          1000     8   992    1% /mnt/test3/df/merged
>> >
>> >
>>
>> I can't follow from the example below what is the expected result and why
>> if you add the quota setup commands that could be useful.
>
> Underlying fs using ext4 and mkfs/mount with project quota option and
> all needed directories(lowerdir, upperdir, workdir, merged) are set same
> project quota. Current quota-df implementation only adjust the couting
> information when having upperdir.
>

Really? why is lowerdir on the same project id? For containers quota
only upper/work should be on the same project id. lowerdir should belong
to a shared project or no project at all.
In current docker implementation, every overlay2 driver root dir
is assigned a different project id, but lowerdir are symlinks to another
image root dir.

> In case 1:
> There is no hardlimit/softlimit, so the expected result as below.
> Used:  The used count in project quota which set to the directory
>        include upperdir && workdir.
> Avail: The avail of underlying fs
> Total: Used + Avail
>
> #upperdir used 201M and /mnt/test3 used 2.5G
>
> In case 2:
> Project quota hardlimits are block count = 1G, inode count = 1000.
> So the expected result as below.
>
> Used:  The used count in project quota which set to the directory
>        include upperdir && workdir.
> Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
> Total: Used + Avail
>

Are those "expected" results inline with what df shows on project quota
directories without overlayfs? If not, and the new behavior makes sense,
why change overlayfs and not change 'df'?

Thanks,
Amir.

Re: "quota-df" feature

[Chengguang Xu]

> Sent: Thursday, March 08, 2018 at 3:23 PM
> From: "Amir Goldstein" <amir73il@gmail.com>
> To: "Chengguang Xu" <cgxu519@gmx.com>
> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> Subject: Re: "quota-df" feature
>
> On Thu, Mar 8, 2018 at 4:21 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> Sent: Wednesday, March 07, 2018 at 9:28 PM
> >> From: "Amir Goldstein" <amir73il@gmail.com>
> >> To: "Chengguang Xu" <cgxu519@gmx.com>
> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >> Subject: Re: "quota-df" feature
> >>
> >> On Wed, Mar 7, 2018 at 2:52 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> > Hello folks,
> >> >
> >> > Recently I'm trying to implement a feature on overlayfs for our users.
> >> > I'd like to call it "quota-df" which means when calling statfs(2),
> >> > override/update some fields(e.g., total/used/avail of both block/inode counts）
> >> > using proper project quota information if the quota has already set to
> >> > underlying directories of overlayfs.
> >> >
> >>
> >> Do you mean project quota setting on underlying fs directories or
> >> something else?
> >
> > Underlying fs directories, more accurately the directory include
> > upperdir && workdir.
> >
> >>
> >> > This feature is probably useful for most of container users and there is
> >> > no serious side effect to the other parts. We can also provide a new
> >> > mount option to enable/disable the function according to different use cases.
> >> >
> >> > Although I'm still working on it but I'll post the RFC patch here for review
> >> > in recent days and hope to get some feedback from you. Any kind of suggestions
> >> > are welcomed.
> >> >
> >>
> >> I was asked about quota accounting for the merged content of overalyfs,
> >> so quota would display a number resembling du -s on the overlay mount.
> >> This means that "covered" lower files are not accounted for in the merge.
> >> Is that what you aim to achieve? If not, would that also be an interesting case
> >> for you to explore?
> >> Basically, the index directory could be used to reconstruct the entire quota
> >> information for upper and lower (and copied up) files if full indexing
> >> is enabled
> >> with nfs_export=on.
> >
> > To be honest, our use case is standard docker container with overlayfs and
> > our requirement of quota-df is accurately counting usage/avail inside container.
> >
> 
> My confusion is because I thought this was working correctly today.
> When I posted docker support for project quota over year ago I remeber
> the tests inside container showed reasonable behavior:
> https://github.com/moby/moby/pull/24771
> 
> So what I am missing is "what is wrong now?".

Seems OK except I'm not fully satisfied the adjusting mechanism in specific fs.
 
> > Lowerdirs could be share with different overlayfs, so I'm not sure counting
> > the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
> > to one overlayfs then maybe setting same project id with upperdir can resolve
> > "covered" problem that you mentioned above. Counting usage information without
> > quota might be a hard work when having plenty of files.
> >
> 
> Well, its a different use case, but a very well known use case -
> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
> blocks, but each clone is fully accounted to the user/group/project.
> This is the "thin provisioning" use case - every user gets accounted by files
> that the user can reference, but the host does not pay the cost of sum of all
> user quotas.
> 
> Without accounting of "covered" files to begin with, is it possible to
> get to a state
> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
> provisioned block,
> but a situation that filesystems and administrators try really hard to avoid.

Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
because of same named file exists in upperdir? or is it meaning the contents in
lowerdir but in the merged dir of overlayfs?


> All I am saying is that it is "not hard" (TM) to keep track of
> "covered" files disk usage
> and "not hard" to re-calculate "covered" files disk usage when full
> indexing is enabled.
> > I don't know exactly what will happen when combining index and nfs_export options,
> > I need to read and understand related code later.
> >
> 
> nfs_export REQUIRES index and implies indexing of all files on copy up, not
> only lower hardlinks.
> 
> >>
> >> >
> >> > Here are some test examples share with you (based on ext4):
> >> >
> >> > 1) project quota enabled && without hard-limit
> >> >
> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
> >> > Filesystem      Size  Used Avail Use% Mounted on
> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> >> > overlay          92G  201M   91G   1% /mnt/test3/df/merged
> >> >
> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> >> > overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
> >> >
> >> > 2) project quota enabled && with hard-limit
> >> >
> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
> >> > Filesystem      Size  Used Avail Use% Mounted on
> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> >> > overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
> >> >
> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> >> > overlay          1000     8   992    1% /mnt/test3/df/merged
> >> >
> >> >
> >>
> >> I can't follow from the example below what is the expected result and why
> >> if you add the quota setup commands that could be useful.
> >
> > Underlying fs using ext4 and mkfs/mount with project quota option and
> > all needed directories(lowerdir, upperdir, workdir, merged) are set same
> > project quota. Current quota-df implementation only adjust the couting
> > information when having upperdir.
> >
> 
> Really? why is lowerdir on the same project id? For containers quota
> only upper/work should be on the same project id. lowerdir should belong
> to a shared project or no project at all.
> In current docker implementation, every overlay2 driver root dir
> is assigned a different project id, but lowerdir are symlinks to another
> image root dir.

For the setting of docker, you are completely right. My description of tesing
environment in previous email is only for simple kernel testing and explaining
the condition of testing result above, not specific to docker.

> 
> > In case 1:
> > There is no hardlimit/softlimit, so the expected result as below.
> > Used:  The used count in project quota which set to the directory
> >        include upperdir && workdir.
> > Avail: The avail of underlying fs
> > Total: Used + Avail
> >
> > #upperdir used 201M and /mnt/test3 used 2.5G
> >
> > In case 2:
> > Project quota hardlimits are block count = 1G, inode count = 1000.
> > So the expected result as below.
> >
> > Used:  The used count in project quota which set to the directory
> >        include upperdir && workdir.
> > Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
> > Total: Used + Avail
> >
> 
> Are those "expected" results inline with what df shows on project quota
> directories without overlayfs? If not, and the new behavior makes sense,
> why change overlayfs and not change 'df'?

No, those results are based on my change of overlayfs. At the first,
I didn't notice that underlying filesystems had already implemented 
'quota-df' function. So I plan to do something in overlayfs because
it's better than doing same things in every low level filesystems. 
But now, I'm more willing to persuade xfs/ext4 people to midify the
detail mechanism of 'quota-df' in specific filesystems.

'df', hmm... Both solutions could work I think.

Thanks,
Chengguang.

Re: "quota-df" feature

[Amir Goldstein]

On Thu, Mar 8, 2018 at 11:23 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> Sent: Thursday, March 08, 2018 at 3:23 PM
>> From: "Amir Goldstein" <amir73il@gmail.com>
>> To: "Chengguang Xu" <cgxu519@gmx.com>
>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> Subject: Re: "quota-df" feature
>>
>> On Thu, Mar 8, 2018 at 4:21 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> >> Sent: Wednesday, March 07, 2018 at 9:28 PM
>> >> From: "Amir Goldstein" <amir73il@gmail.com>
>> >> To: "Chengguang Xu" <cgxu519@gmx.com>
>> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> >> Subject: Re: "quota-df" feature
>> >>
>> >> On Wed, Mar 7, 2018 at 2:52 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> >> > Hello folks,
>> >> >
>> >> > Recently I'm trying to implement a feature on overlayfs for our users.
>> >> > I'd like to call it "quota-df" which means when calling statfs(2),
>> >> > override/update some fields(e.g., total/used/avail of both block/inode counts）
>> >> > using proper project quota information if the quota has already set to
>> >> > underlying directories of overlayfs.
>> >> >
>> >>
>> >> Do you mean project quota setting on underlying fs directories or
>> >> something else?
>> >
>> > Underlying fs directories, more accurately the directory include
>> > upperdir && workdir.
>> >
>> >>
>> >> > This feature is probably useful for most of container users and there is
>> >> > no serious side effect to the other parts. We can also provide a new
>> >> > mount option to enable/disable the function according to different use cases.
>> >> >
>> >> > Although I'm still working on it but I'll post the RFC patch here for review
>> >> > in recent days and hope to get some feedback from you. Any kind of suggestions
>> >> > are welcomed.
>> >> >
>> >>
>> >> I was asked about quota accounting for the merged content of overalyfs,
>> >> so quota would display a number resembling du -s on the overlay mount.
>> >> This means that "covered" lower files are not accounted for in the merge.
>> >> Is that what you aim to achieve? If not, would that also be an interesting case
>> >> for you to explore?
>> >> Basically, the index directory could be used to reconstruct the entire quota
>> >> information for upper and lower (and copied up) files if full indexing
>> >> is enabled
>> >> with nfs_export=on.
>> >
>> > To be honest, our use case is standard docker container with overlayfs and
>> > our requirement of quota-df is accurately counting usage/avail inside container.
>> >
>>
>> My confusion is because I thought this was working correctly today.
>> When I posted docker support for project quota over year ago I remeber
>> the tests inside container showed reasonable behavior:
>> https://github.com/moby/moby/pull/24771
>>
>> So what I am missing is "what is wrong now?".
>
> Seems OK except I'm not fully satisfied the adjusting mechanism in specific fs.
>
>> > Lowerdirs could be share with different overlayfs, so I'm not sure counting
>> > the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>> > to one overlayfs then maybe setting same project id with upperdir can resolve
>> > "covered" problem that you mentioned above. Counting usage information without
>> > quota might be a hard work when having plenty of files.
>> >
>>
>> Well, its a different use case, but a very well known use case -
>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>> blocks, but each clone is fully accounted to the user/group/project.
>> This is the "thin provisioning" use case - every user gets accounted by files
>> that the user can reference, but the host does not pay the cost of sum of all
>> user quotas.
>>
>> Without accounting of "covered" files to begin with, is it possible to
>> get to a state
>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>> provisioned block,
>> but a situation that filesystems and administrators try really hard to avoid.
>
> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
> because of same named file exists in upperdir? or is it meaning the contents in
> lowerdir but in the merged dir of overlayfs?
>
>

The former. "covered" file size does not show up in du -s, so the
merged disk usage
is <upper used> + <lower used> - <covered used>

>> All I am saying is that it is "not hard" (TM) to keep track of
>> "covered" files disk usage
>> and "not hard" to re-calculate "covered" files disk usage when full
>> indexing is enabled.
>> > I don't know exactly what will happen when combining index and nfs_export options,
>> > I need to read and understand related code later.
>> >
>>
>> nfs_export REQUIRES index and implies indexing of all files on copy up, not
>> only lower hardlinks.
>>
>> >>
>> >> >
>> >> > Here are some test examples share with you (based on ext4):
>> >> >
>> >> > 1) project quota enabled && without hard-limit
>> >> >
>> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
>> >> > Filesystem      Size  Used Avail Use% Mounted on
>> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>> >> > overlay          92G  201M   91G   1% /mnt/test3/df/merged
>> >> >
>> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
>> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
>> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>> >> > overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>> >> >
>> >> > 2) project quota enabled && with hard-limit
>> >> >
>> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
>> >> > Filesystem      Size  Used Avail Use% Mounted on
>> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>> >> > overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>> >> >
>> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
>> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
>> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>> >> > overlay          1000     8   992    1% /mnt/test3/df/merged
>> >> >
>> >> >
>> >>
>> >> I can't follow from the example below what is the expected result and why
>> >> if you add the quota setup commands that could be useful.
>> >
>> > Underlying fs using ext4 and mkfs/mount with project quota option and
>> > all needed directories(lowerdir, upperdir, workdir, merged) are set same
>> > project quota. Current quota-df implementation only adjust the couting
>> > information when having upperdir.
>> >
>>
>> Really? why is lowerdir on the same project id? For containers quota
>> only upper/work should be on the same project id. lowerdir should belong
>> to a shared project or no project at all.
>> In current docker implementation, every overlay2 driver root dir
>> is assigned a different project id, but lowerdir are symlinks to another
>> image root dir.
>
> For the setting of docker, you are completely right. My description of tesing
> environment in previous email is only for simple kernel testing and explaining
> the condition of testing result above, not specific to docker.
>
>>
>> > In case 1:
>> > There is no hardlimit/softlimit, so the expected result as below.
>> > Used:  The used count in project quota which set to the directory
>> >        include upperdir && workdir.
>> > Avail: The avail of underlying fs
>> > Total: Used + Avail
>> >
>> > #upperdir used 201M and /mnt/test3 used 2.5G
>> >
>> > In case 2:
>> > Project quota hardlimits are block count = 1G, inode count = 1000.
>> > So the expected result as below.
>> >
>> > Used:  The used count in project quota which set to the directory
>> >        include upperdir && workdir.
>> > Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
>> > Total: Used + Avail
>> >
>>
>> Are those "expected" results inline with what df shows on project quota
>> directories without overlayfs? If not, and the new behavior makes sense,
>> why change overlayfs and not change 'df'?
>
> No, those results are based on my change of overlayfs. At the first,
> I didn't notice that underlying filesystems had already implemented
> 'quota-df' function. So I plan to do something in overlayfs because
> it's better than doing same things in every low level filesystems.
> But now, I'm more willing to persuade xfs/ext4 people to midify the
> detail mechanism of 'quota-df' in specific filesystems.
>
> 'df', hmm... Both solutions could work I think.
>

I don't thing it is the underlying filesystem that implements quota-df
I think it is 'df' itself. I was also surprised to learn than, but at least when
you set project quota with xfs_quota via /etc/projects /etc/projid
df just shows you the project directories as if they were mount points.
I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
is no different.

Thanks,
Amir.

Re: "quota-df" feature

[Chengguang Xu]

> Sent: Thursday, March 08, 2018 at 7:06 PM
> From: "Amir Goldstein" <amir73il@gmail.com>
> To: "Chengguang Xu" <cgxu519@gmx.com>
> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> Subject: Re: "quota-df" feature
>
> On Thu, Mar 8, 2018 at 11:23 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> Sent: Thursday, March 08, 2018 at 3:23 PM
> >> From: "Amir Goldstein" <amir73il@gmail.com>
> >> To: "Chengguang Xu" <cgxu519@gmx.com>
> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >> Subject: Re: "quota-df" feature
> >>
> >> On Thu, Mar 8, 2018 at 4:21 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> >> Sent: Wednesday, March 07, 2018 at 9:28 PM
> >> >> From: "Amir Goldstein" <amir73il@gmail.com>
> >> >> To: "Chengguang Xu" <cgxu519@gmx.com>
> >> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >> >> Subject: Re: "quota-df" feature
> >> >>
> >> >> On Wed, Mar 7, 2018 at 2:52 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> >> > Hello folks,
> >> >> >
> >> >> > Recently I'm trying to implement a feature on overlayfs for our users.
> >> >> > I'd like to call it "quota-df" which means when calling statfs(2),
> >> >> > override/update some fields(e.g., total/used/avail of both block/inode counts）
> >> >> > using proper project quota information if the quota has already set to
> >> >> > underlying directories of overlayfs.
> >> >> >
> >> >>
> >> >> Do you mean project quota setting on underlying fs directories or
> >> >> something else?
> >> >
> >> > Underlying fs directories, more accurately the directory include
> >> > upperdir && workdir.
> >> >
> >> >>
> >> >> > This feature is probably useful for most of container users and there is
> >> >> > no serious side effect to the other parts. We can also provide a new
> >> >> > mount option to enable/disable the function according to different use cases.
> >> >> >
> >> >> > Although I'm still working on it but I'll post the RFC patch here for review
> >> >> > in recent days and hope to get some feedback from you. Any kind of suggestions
> >> >> > are welcomed.
> >> >> >
> >> >>
> >> >> I was asked about quota accounting for the merged content of overalyfs,
> >> >> so quota would display a number resembling du -s on the overlay mount.
> >> >> This means that "covered" lower files are not accounted for in the merge.
> >> >> Is that what you aim to achieve? If not, would that also be an interesting case
> >> >> for you to explore?
> >> >> Basically, the index directory could be used to reconstruct the entire quota
> >> >> information for upper and lower (and copied up) files if full indexing
> >> >> is enabled
> >> >> with nfs_export=on.
> >> >
> >> > To be honest, our use case is standard docker container with overlayfs and
> >> > our requirement of quota-df is accurately counting usage/avail inside container.
> >> >
> >>
> >> My confusion is because I thought this was working correctly today.
> >> When I posted docker support for project quota over year ago I remeber
> >> the tests inside container showed reasonable behavior:
> >> https://github.com/moby/moby/pull/24771
> >>
> >> So what I am missing is "what is wrong now?".
> >
> > Seems OK except I'm not fully satisfied the adjusting mechanism in specific fs.
> >
> >> > Lowerdirs could be share with different overlayfs, so I'm not sure counting
> >> > the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
> >> > to one overlayfs then maybe setting same project id with upperdir can resolve
> >> > "covered" problem that you mentioned above. Counting usage information without
> >> > quota might be a hard work when having plenty of files.
> >> >
> >>
> >> Well, its a different use case, but a very well known use case -
> >> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
> >> blocks, but each clone is fully accounted to the user/group/project.
> >> This is the "thin provisioning" use case - every user gets accounted by files
> >> that the user can reference, but the host does not pay the cost of sum of all
> >> user quotas.
> >>
> >> Without accounting of "covered" files to begin with, is it possible to
> >> get to a state
> >> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
> >> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
> >> provisioned block,
> >> but a situation that filesystems and administrators try really hard to avoid.
> >
> > Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
> > because of same named file exists in upperdir? or is it meaning the contents in
> > lowerdir but in the merged dir of overlayfs?
> >
> >
> 
> The former. "covered" file size does not show up in du -s, so the
> merged disk usage
> is <upper used> + <lower used> - <covered used>

Thanks, got it.
But IIUC, "covered" file does not have chance to copy-up, so 
I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?

> 
> >> All I am saying is that it is "not hard" (TM) to keep track of
> >> "covered" files disk usage
> >> and "not hard" to re-calculate "covered" files disk usage when full
> >> indexing is enabled.
> >> > I don't know exactly what will happen when combining index and nfs_export options,
> >> > I need to read and understand related code later.
> >> >
> >>
> >> nfs_export REQUIRES index and implies indexing of all files on copy up, not
> >> only lower hardlinks.
> >>
> >> >>
> >> >> >
> >> >> > Here are some test examples share with you (based on ext4):
> >> >> >
> >> >> > 1) project quota enabled && without hard-limit
> >> >> >
> >> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
> >> >> > Filesystem      Size  Used Avail Use% Mounted on
> >> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> >> >> > overlay          92G  201M   91G   1% /mnt/test3/df/merged
> >> >> >
> >> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
> >> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
> >> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> >> >> > overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
> >> >> >
> >> >> > 2) project quota enabled && with hard-limit
> >> >> >
> >> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
> >> >> > Filesystem      Size  Used Avail Use% Mounted on
> >> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> >> >> > overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
> >> >> >
> >> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
> >> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
> >> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> >> >> > overlay          1000     8   992    1% /mnt/test3/df/merged
> >> >> >
> >> >> >
> >> >>
> >> >> I can't follow from the example below what is the expected result and why
> >> >> if you add the quota setup commands that could be useful.
> >> >
> >> > Underlying fs using ext4 and mkfs/mount with project quota option and
> >> > all needed directories(lowerdir, upperdir, workdir, merged) are set same
> >> > project quota. Current quota-df implementation only adjust the couting
> >> > information when having upperdir.
> >> >
> >>
> >> Really? why is lowerdir on the same project id? For containers quota
> >> only upper/work should be on the same project id. lowerdir should belong
> >> to a shared project or no project at all.
> >> In current docker implementation, every overlay2 driver root dir
> >> is assigned a different project id, but lowerdir are symlinks to another
> >> image root dir.
> >
> > For the setting of docker, you are completely right. My description of tesing
> > environment in previous email is only for simple kernel testing and explaining
> > the condition of testing result above, not specific to docker.
> >
> >>
> >> > In case 1:
> >> > There is no hardlimit/softlimit, so the expected result as below.
> >> > Used:  The used count in project quota which set to the directory
> >> >        include upperdir && workdir.
> >> > Avail: The avail of underlying fs
> >> > Total: Used + Avail
> >> >
> >> > #upperdir used 201M and /mnt/test3 used 2.5G
> >> >
> >> > In case 2:
> >> > Project quota hardlimits are block count = 1G, inode count = 1000.
> >> > So the expected result as below.
> >> >
> >> > Used:  The used count in project quota which set to the directory
> >> >        include upperdir && workdir.
> >> > Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
> >> > Total: Used + Avail
> >> >
> >>
> >> Are those "expected" results inline with what df shows on project quota
> >> directories without overlayfs? If not, and the new behavior makes sense,
> >> why change overlayfs and not change 'df'?
> >
> > No, those results are based on my change of overlayfs. At the first,
> > I didn't notice that underlying filesystems had already implemented
> > 'quota-df' function. So I plan to do something in overlayfs because
> > it's better than doing same things in every low level filesystems.
> > But now, I'm more willing to persuade xfs/ext4 people to midify the
> > detail mechanism of 'quota-df' in specific filesystems.
> >
> > 'df', hmm... Both solutions could work I think.
> >
> 
> I don't thing it is the underlying filesystem that implements quota-df
> I think it is 'df' itself. I was also surprised to learn than, but at least when
> you set project quota with xfs_quota via /etc/projects /etc/projid
> df just shows you the project directories as if they were mount points.
> I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
> is no different.

That sounds really interesting and that must be special version of df command.
On CentOS I have never seen that. :-(
In any case if you take a quick look at below functions in the code, you will probably
believe what I said before. If you stop calling those functions in the kernel code, 
then I guess all magic will be gone and never turn back again. :-)

xfs:
xfs_qm_statvfs

ext4:
ext4_statfs_project

f2fs:
f2fs_statfs_project


Thanks,
Chengguang.

Re: "quota-df" feature

[Amir Goldstein]

On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
[...]
>> >> > Lowerdirs could be share with different overlayfs, so I'm not sure counting
>> >> > the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>> >> > to one overlayfs then maybe setting same project id with upperdir can resolve
>> >> > "covered" problem that you mentioned above. Counting usage information without
>> >> > quota might be a hard work when having plenty of files.
>> >> >
>> >>
>> >> Well, its a different use case, but a very well known use case -
>> >> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>> >> blocks, but each clone is fully accounted to the user/group/project.
>> >> This is the "thin provisioning" use case - every user gets accounted by files
>> >> that the user can reference, but the host does not pay the cost of sum of all
>> >> user quotas.
>> >>
>> >> Without accounting of "covered" files to begin with, is it possible to
>> >> get to a state
>> >> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>> >> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>> >> provisioned block,
>> >> but a situation that filesystems and administrators try really hard to avoid.
>> >
>> > Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>> > because of same named file exists in upperdir? or is it meaning the contents in
>> > lowerdir but in the merged dir of overlayfs?
>> >
>> >
>>
>> The former. "covered" file size does not show up in du -s, so the
>> merged disk usage
>> is <upper used> + <lower used> - <covered used>
>
> Thanks, got it.
> But IIUC, "covered" file does not have chance to copy-up, so
> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?

Suppose your "image" (i.e. total disk usage of lower) is 1GB
and you want to allow user to touch all the files in the image and
create 1GB of new files.

If your only tool is project quota on upper then you need to set project
quota hard limit to 2GB, but then user can create 2GB of new files and
later when touching a lower file, will get EQUOTA on copy up.

If you account lower uncovered files to overlay merged quota then
you set the merged quota to 2GB and start with 50% used.
- Copy up will not change used
- Remove of lower will reduce used
- You can never get EQUOTA from touching a file

>
>>
>> >> All I am saying is that it is "not hard" (TM) to keep track of
>> >> "covered" files disk usage
>> >> and "not hard" to re-calculate "covered" files disk usage when full
>> >> indexing is enabled.
>> >> > I don't know exactly what will happen when combining index and nfs_export options,
>> >> > I need to read and understand related code later.
>> >> >
>> >>
>> >> nfs_export REQUIRES index and implies indexing of all files on copy up, not
>> >> only lower hardlinks.
>> >>
>> >> >>
>> >> >> >
>> >> >> > Here are some test examples share with you (based on ext4):
>> >> >> >
>> >> >> > 1) project quota enabled && without hard-limit
>> >> >> >
>> >> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
>> >> >> > Filesystem      Size  Used Avail Use% Mounted on
>> >> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>> >> >> > overlay          92G  201M   91G   1% /mnt/test3/df/merged
>> >> >> >
>> >> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
>> >> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
>> >> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>> >> >> > overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>> >> >> >
>> >> >> > 2) project quota enabled && with hard-limit
>> >> >> >
>> >> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
>> >> >> > Filesystem      Size  Used Avail Use% Mounted on
>> >> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>> >> >> > overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>> >> >> >
>> >> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
>> >> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
>> >> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>> >> >> > overlay          1000     8   992    1% /mnt/test3/df/merged
>> >> >> >
>> >> >> >
>> >> >>
>> >> >> I can't follow from the example below what is the expected result and why
>> >> >> if you add the quota setup commands that could be useful.
>> >> >
>> >> > Underlying fs using ext4 and mkfs/mount with project quota option and
>> >> > all needed directories(lowerdir, upperdir, workdir, merged) are set same
>> >> > project quota. Current quota-df implementation only adjust the couting
>> >> > information when having upperdir.
>> >> >
>> >>
>> >> Really? why is lowerdir on the same project id? For containers quota
>> >> only upper/work should be on the same project id. lowerdir should belong
>> >> to a shared project or no project at all.
>> >> In current docker implementation, every overlay2 driver root dir
>> >> is assigned a different project id, but lowerdir are symlinks to another
>> >> image root dir.
>> >
>> > For the setting of docker, you are completely right. My description of tesing
>> > environment in previous email is only for simple kernel testing and explaining
>> > the condition of testing result above, not specific to docker.
>> >
>> >>
>> >> > In case 1:
>> >> > There is no hardlimit/softlimit, so the expected result as below.
>> >> > Used:  The used count in project quota which set to the directory
>> >> >        include upperdir && workdir.
>> >> > Avail: The avail of underlying fs
>> >> > Total: Used + Avail
>> >> >
>> >> > #upperdir used 201M and /mnt/test3 used 2.5G
>> >> >
>> >> > In case 2:
>> >> > Project quota hardlimits are block count = 1G, inode count = 1000.
>> >> > So the expected result as below.
>> >> >
>> >> > Used:  The used count in project quota which set to the directory
>> >> >        include upperdir && workdir.
>> >> > Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
>> >> > Total: Used + Avail
>> >> >
>> >>
>> >> Are those "expected" results inline with what df shows on project quota
>> >> directories without overlayfs? If not, and the new behavior makes sense,
>> >> why change overlayfs and not change 'df'?
>> >
>> > No, those results are based on my change of overlayfs. At the first,
>> > I didn't notice that underlying filesystems had already implemented
>> > 'quota-df' function. So I plan to do something in overlayfs because
>> > it's better than doing same things in every low level filesystems.
>> > But now, I'm more willing to persuade xfs/ext4 people to midify the
>> > detail mechanism of 'quota-df' in specific filesystems.
>> >
>> > 'df', hmm... Both solutions could work I think.
>> >
>>
>> I don't thing it is the underlying filesystem that implements quota-df
>> I think it is 'df' itself. I was also surprised to learn than, but at least when
>> you set project quota with xfs_quota via /etc/projects /etc/projid
>> df just shows you the project directories as if they were mount points.
>> I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
>> is no different.
>
> That sounds really interesting and that must be special version of df command.
> On CentOS I have never seen that. :-(

I donno, maybe I dreamed of seeing it... most likely I am confusing seeing
the correct df usage on overlayfs mount with upper that has project quota.

> In any case if you take a quick look at below functions in the code, you will probably
> believe what I said before. If you stop calling those functions in the kernel code,
> then I guess all magic will be gone and never turn back again. :-)
>
> xfs:
> xfs_qm_statvfs
>
> ext4:
> ext4_statfs_project
>
> f2fs:
> f2fs_statfs_project
>
>

I see. so I'll wait for your RFC patch to see what ovl_statfs_project brings
to the table.

Cheers,
Amir.

Re: "quota-df" feature

[Chengguang Xu]

> Sent: Thursday, March 08, 2018 at 10:36 PM
> From: "Amir Goldstein" <amir73il@gmail.com>
> To: "Chengguang Xu" <cgxu519@gmx.com>
> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> Subject: Re: "quota-df" feature
>
> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
> [...]
> >> >> > Lowerdirs could be share with different overlayfs, so I'm not sure counting
> >> >> > the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
> >> >> > to one overlayfs then maybe setting same project id with upperdir can resolve
> >> >> > "covered" problem that you mentioned above. Counting usage information without
> >> >> > quota might be a hard work when having plenty of files.
> >> >> >
> >> >>
> >> >> Well, its a different use case, but a very well known use case -
> >> >> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
> >> >> blocks, but each clone is fully accounted to the user/group/project.
> >> >> This is the "thin provisioning" use case - every user gets accounted by files
> >> >> that the user can reference, but the host does not pay the cost of sum of all
> >> >> user quotas.
> >> >>
> >> >> Without accounting of "covered" files to begin with, is it possible to
> >> >> get to a state
> >> >> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
> >> >> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
> >> >> provisioned block,
> >> >> but a situation that filesystems and administrators try really hard to avoid.
> >> >
> >> > Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
> >> > because of same named file exists in upperdir? or is it meaning the contents in
> >> > lowerdir but in the merged dir of overlayfs?
> >> >
> >> >
> >>
> >> The former. "covered" file size does not show up in du -s, so the
> >> merged disk usage
> >> is <upper used> + <lower used> - <covered used>
> >
> > Thanks, got it.
> > But IIUC, "covered" file does not have chance to copy-up, so
> > I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
> 
> Suppose your "image" (i.e. total disk usage of lower) is 1GB
> and you want to allow user to touch all the files in the image and
> create 1GB of new files.
> 
> If your only tool is project quota on upper then you need to set project
> quota hard limit to 2GB, but then user can create 2GB of new files and
> later when touching a lower file, will get EQUOTA on copy up.
> 
> If you account lower uncovered files to overlay merged quota then
> you set the merged quota to 2GB and start with 50% used.
> - Copy up will not change used
> - Remove of lower will reduce used
> - You can never get EQUOTA from touching a file

I get your point, I'd like to think it as "reservation" feature we can implement
in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up. 
Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
block reservation? Are there many people hope to have this kind of function?


> >> >> All I am saying is that it is "not hard" (TM) to keep track of
> >> >> "covered" files disk usage
> >> >> and "not hard" to re-calculate "covered" files disk usage when full
> >> >> indexing is enabled.
> >> >> > I don't know exactly what will happen when combining index and nfs_export options,
> >> >> > I need to read and understand related code later.
> >> >> >
> >> >>
> >> >> nfs_export REQUIRES index and implies indexing of all files on copy up, not
> >> >> only lower hardlinks.
> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> > Here are some test examples share with you (based on ext4):
> >> >> >> >
> >> >> >> > 1) project quota enabled && without hard-limit
> >> >> >> >
> >> >> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
> >> >> >> > Filesystem      Size  Used Avail Use% Mounted on
> >> >> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> >> >> >> > overlay          92G  201M   91G   1% /mnt/test3/df/merged
> >> >> >> >
> >> >> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
> >> >> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
> >> >> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> >> >> >> > overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
> >> >> >> >
> >> >> >> > 2) project quota enabled && with hard-limit
> >> >> >> >
> >> >> >> > $ df -h /mnt/test3 /mnt/test3/df/merged
> >> >> >> > Filesystem      Size  Used Avail Use% Mounted on
> >> >> >> > /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
> >> >> >> > overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
> >> >> >> >
> >> >> >> > $ df -hi /mnt/test3 /mnt/test3/df/merged
> >> >> >> > Filesystem     Inodes IUsed IFree IUse% Mounted on
> >> >> >> > /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
> >> >> >> > overlay          1000     8   992    1% /mnt/test3/df/merged
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >> I can't follow from the example below what is the expected result and why
> >> >> >> if you add the quota setup commands that could be useful.
> >> >> >
> >> >> > Underlying fs using ext4 and mkfs/mount with project quota option and
> >> >> > all needed directories(lowerdir, upperdir, workdir, merged) are set same
> >> >> > project quota. Current quota-df implementation only adjust the couting
> >> >> > information when having upperdir.
> >> >> >
> >> >>
> >> >> Really? why is lowerdir on the same project id? For containers quota
> >> >> only upper/work should be on the same project id. lowerdir should belong
> >> >> to a shared project or no project at all.
> >> >> In current docker implementation, every overlay2 driver root dir
> >> >> is assigned a different project id, but lowerdir are symlinks to another
> >> >> image root dir.
> >> >
> >> > For the setting of docker, you are completely right. My description of tesing
> >> > environment in previous email is only for simple kernel testing and explaining
> >> > the condition of testing result above, not specific to docker.
> >> >
> >> >>
> >> >> > In case 1:
> >> >> > There is no hardlimit/softlimit, so the expected result as below.
> >> >> > Used:  The used count in project quota which set to the directory
> >> >> >        include upperdir && workdir.
> >> >> > Avail: The avail of underlying fs
> >> >> > Total: Used + Avail
> >> >> >
> >> >> > #upperdir used 201M and /mnt/test3 used 2.5G
> >> >> >
> >> >> > In case 2:
> >> >> > Project quota hardlimits are block count = 1G, inode count = 1000.
> >> >> > So the expected result as below.
> >> >> >
> >> >> > Used:  The used count in project quota which set to the directory
> >> >> >        include upperdir && workdir.
> >> >> > Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
> >> >> > Total: Used + Avail
> >> >> >
> >> >>
> >> >> Are those "expected" results inline with what df shows on project quota
> >> >> directories without overlayfs? If not, and the new behavior makes sense,
> >> >> why change overlayfs and not change 'df'?
> >> >
> >> > No, those results are based on my change of overlayfs. At the first,
> >> > I didn't notice that underlying filesystems had already implemented
> >> > 'quota-df' function. So I plan to do something in overlayfs because
> >> > it's better than doing same things in every low level filesystems.
> >> > But now, I'm more willing to persuade xfs/ext4 people to midify the
> >> > detail mechanism of 'quota-df' in specific filesystems.
> >> >
> >> > 'df', hmm... Both solutions could work I think.
> >> >
> >>
> >> I don't thing it is the underlying filesystem that implements quota-df
> >> I think it is 'df' itself. I was also surprised to learn than, but at least when
> >> you set project quota with xfs_quota via /etc/projects /etc/projid
> >> df just shows you the project directories as if they were mount points.
> >> I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
> >> is no different.
> >
> > That sounds really interesting and that must be special version of df command.
> > On CentOS I have never seen that. :-(
> 
> I donno, maybe I dreamed of seeing it... most likely I am confusing seeing
> the correct df usage on overlayfs mount with upper that has project quota.
> 
> > In any case if you take a quick look at below functions in the code, you will probably
> > believe what I said before. If you stop calling those functions in the kernel code,
> > then I guess all magic will be gone and never turn back again. :-)
> >
> > xfs:
> > xfs_qm_statvfs
> >
> > ext4:
> > ext4_statfs_project
> >
> > f2fs:
> > f2fs_statfs_project
> =【--07 
> >
> 
> I see. so I'll wait for your RFC patch to see what ovl_statfs_project brings
> to the table.

Seems there is nothing more to do unless we add more features like 'reservation' we discussed above.
In this case I think we should consider adding 'reservation amount' to bfree, and bavail represents
the real free space amount that can be utilized by new files.


Thanks,
Chengguang.

Re: "quota-df" feature

[Amir Goldstein]

On Sat, Mar 10, 2018 at 8:37 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> Sent: Thursday, March 08, 2018 at 10:36 PM
>> From: "Amir Goldstein" <amir73il@gmail.com>
>> To: "Chengguang Xu" <cgxu519@gmx.com>
>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> Subject: Re: "quota-df" feature
>>
>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> [...]
>> >> >> > Lowerdirs could be share with different overlayfs, so I'm not sure counting
>> >> >> > the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>> >> >> > to one overlayfs then maybe setting same project id with upperdir can resolve
>> >> >> > "covered" problem that you mentioned above. Counting usage information without
>> >> >> > quota might be a hard work when having plenty of files.
>> >> >> >
>> >> >>
>> >> >> Well, its a different use case, but a very well known use case -
>> >> >> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>> >> >> blocks, but each clone is fully accounted to the user/group/project.
>> >> >> This is the "thin provisioning" use case - every user gets accounted by files
>> >> >> that the user can reference, but the host does not pay the cost of sum of all
>> >> >> user quotas.
>> >> >>
>> >> >> Without accounting of "covered" files to begin with, is it possible to
>> >> >> get to a state
>> >> >> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>> >> >> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>> >> >> provisioned block,
>> >> >> but a situation that filesystems and administrators try really hard to avoid.
>> >> >
>> >> > Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>> >> > because of same named file exists in upperdir? or is it meaning the contents in
>> >> > lowerdir but in the merged dir of overlayfs?
>> >> >
>> >> >
>> >>
>> >> The former. "covered" file size does not show up in du -s, so the
>> >> merged disk usage
>> >> is <upper used> + <lower used> - <covered used>
>> >
>> > Thanks, got it.
>> > But IIUC, "covered" file does not have chance to copy-up, so
>> > I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
>>
>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
>> and you want to allow user to touch all the files in the image and
>> create 1GB of new files.
>>
>> If your only tool is project quota on upper then you need to set project
>> quota hard limit to 2GB, but then user can create 2GB of new files and
>> later when touching a lower file, will get EQUOTA on copy up.
>>
>> If you account lower uncovered files to overlay merged quota then
>> you set the merged quota to 2GB and start with 50% used.
>> - Copy up will not change used
>> - Remove of lower will reduce used
>> - You can never get EQUOTA from touching a file
>
> I get your point, I'd like to think it as "reservation" feature we can implement
> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
> block reservation? Are there many people hope to have this kind of function?
>

Not sure. I heard this requirement once, but I have a feeling people
still don't know
what they can get out of overlayfs, so they are used to getting along
with what they
have. I think now that docker supports project quotas with overlay2
driver, people
will start using project quota and sooner or later, they will start
hitting EQUOTA on
copy up. That's my prophecy anyway ;-)

Cheers,
Amir.

Re: "quota-df" feature

[Chengguang Xu]

> Sent: Sunday, March 11, 2018 at 4:03 AM
> From: "Amir Goldstein" <amir73il@gmail.com>
> To: "Chengguang Xu" <cgxu519@gmx.com>
> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> Subject: Re: "quota-df" feature
>
> On Sat, Mar 10, 2018 at 8:37 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> Sent: Thursday, March 08, 2018 at 10:36 PM
> >> From: "Amir Goldstein" <amir73il@gmail.com>
> >> To: "Chengguang Xu" <cgxu519@gmx.com>
> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >> Subject: Re: "quota-df" feature
> >>
> >> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> [...]
> >> >> >> > Lowerdirs could be share with different overlayfs, so I'm not sure counting
> >> >> >> > the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
> >> >> >> > to one overlayfs then maybe setting same project id with upperdir can resolve
> >> >> >> > "covered" problem that you mentioned above. Counting usage information without
> >> >> >> > quota might be a hard work when having plenty of files.
> >> >> >> >
> >> >> >>
> >> >> >> Well, its a different use case, but a very well known use case -
> >> >> >> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
> >> >> >> blocks, but each clone is fully accounted to the user/group/project.
> >> >> >> This is the "thin provisioning" use case - every user gets accounted by files
> >> >> >> that the user can reference, but the host does not pay the cost of sum of all
> >> >> >> user quotas.
> >> >> >>
> >> >> >> Without accounting of "covered" files to begin with, is it possible to
> >> >> >> get to a state
> >> >> >> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
> >> >> >> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
> >> >> >> provisioned block,
> >> >> >> but a situation that filesystems and administrators try really hard to avoid.
> >> >> >
> >> >> > Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
> >> >> > because of same named file exists in upperdir? or is it meaning the contents in
> >> >> > lowerdir but in the merged dir of overlayfs?
> >> >> >
> >> >> >
> >> >>
> >> >> The former. "covered" file size does not show up in du -s, so the
> >> >> merged disk usage
> >> >> is <upper used> + <lower used> - <covered used>
> >> >
> >> > Thanks, got it.
> >> > But IIUC, "covered" file does not have chance to copy-up, so
> >> > I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
> >>
> >> Suppose your "image" (i.e. total disk usage of lower) is 1GB
> >> and you want to allow user to touch all the files in the image and
> >> create 1GB of new files.
> >>
> >> If your only tool is project quota on upper then you need to set project
> >> quota hard limit to 2GB, but then user can create 2GB of new files and
> >> later when touching a lower file, will get EQUOTA on copy up.
> >>
> >> If you account lower uncovered files to overlay merged quota then
> >> you set the merged quota to 2GB and start with 50% used.
> >> - Copy up will not change used
> >> - Remove of lower will reduce used
> >> - You can never get EQUOTA from touching a file
> >
> > I get your point, I'd like to think it as "reservation" feature we can implement
> > in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
> > Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
> > block reservation? Are there many people hope to have this kind of function?
> >
> 
> Not sure. I heard this requirement once, but I have a feeling people
> still don't know
> what they can get out of overlayfs, so they are used to getting along
> with what they
> have. I think now that docker supports project quotas with overlay2
> driver, people
> will start using project quota and sooner or later, they will start
> hitting EQUOTA on
> copy up. That's my prophecy anyway ;-)

I believe in the near future, most container users will experience this.
Indoe will be worse compare to space, not only fail on creating new file
but also on deleting old uncovered files because of plenty whiteout files. 
I remembered someone has already posted fix patch to reduce whiteout files,
so after that merged, we can consider block and inode in the same way.

Hi Zhang yi,

Is that possible to count space and inode of "covered"/"uncovered"/"upper"
via userspace command tool? I think those info will be useful/important reference
to admin to set reservation when mounting overlayfs. Then maybe I can add
some reservation logics in kernel side to avoid EQUOTA error as much as possible.  


Thanks,
Chengguang.

Re: "quota-df" feature

[zhangyi (F)]

On 2018/3/11 11:16, Chengguang Xu Wrote:
>> Sent: Sunday, March 11, 2018 at 4:03 AM
>> From: "Amir Goldstein" <amir73il@gmail.com>
>> To: "Chengguang Xu" <cgxu519@gmx.com>
>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> Subject: Re: "quota-df" feature
>>
>> On Sat, Mar 10, 2018 at 8:37 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
>>>> Sent: Thursday, March 08, 2018 at 10:36 PM
>>>> From: "Amir Goldstein" <amir73il@gmail.com>
>>>> To: "Chengguang Xu" <cgxu519@gmx.com>
>>>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>>>> Subject: Re: "quota-df" feature
>>>>
>>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>>>> [...]
>>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
>>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
>>>>>>>>> "covered" problem that you mentioned above. Counting usage information without
>>>>>>>>> quota might be a hard work when having plenty of files.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Well, its a different use case, but a very well known use case -
>>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>>>>>>>> blocks, but each clone is fully accounted to the user/group/project.
>>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
>>>>>>>> that the user can reference, but the host does not pay the cost of sum of all
>>>>>>>> user quotas.
>>>>>>>>
>>>>>>>> Without accounting of "covered" files to begin with, is it possible to
>>>>>>>> get to a state
>>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>>>>>>>> provisioned block,
>>>>>>>> but a situation that filesystems and administrators try really hard to avoid.
>>>>>>>
>>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>>>>>>> because of same named file exists in upperdir? or is it meaning the contents in
>>>>>>> lowerdir but in the merged dir of overlayfs?
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> The former. "covered" file size does not show up in du -s, so the
>>>>>> merged disk usage
>>>>>> is <upper used> + <lower used> - <covered used>
>>>>>
>>>>> Thanks, got it.
>>>>> But IIUC, "covered" file does not have chance to copy-up, so
>>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
>>>>
>>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
>>>> and you want to allow user to touch all the files in the image and
>>>> create 1GB of new files.
>>>>
>>>> If your only tool is project quota on upper then you need to set project
>>>> quota hard limit to 2GB, but then user can create 2GB of new files and
>>>> later when touching a lower file, will get EQUOTA on copy up.
>>>>
>>>> If you account lower uncovered files to overlay merged quota then
>>>> you set the merged quota to 2GB and start with 50% used.
>>>> - Copy up will not change used
>>>> - Remove of lower will reduce used
>>>> - You can never get EQUOTA from touching a file
>>>
>>> I get your point, I'd like to think it as "reservation" feature we can implement
>>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
>>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
>>> block reservation? Are there many people hope to have this kind of function?
>>>
>>
>> Not sure. I heard this requirement once, but I have a feeling people
>> still don't know
>> what they can get out of overlayfs, so they are used to getting along
>> with what they
>> have. I think now that docker supports project quotas with overlay2
>> driver, people
>> will start using project quota and sooner or later, they will start
>> hitting EQUOTA on
>> copy up. That's my prophecy anyway ;-)
> 
> I believe in the near future, most container users will experience this.
> Indoe will be worse compare to space, not only fail on creating new file
> but also on deleting old uncovered files because of plenty whiteout files. 
> I remembered someone has already posted fix patch to reduce whiteout files,
> so after that merged, we can consider block and inode in the same way.

And copy up directory with nfs_export=on, which will cost one more inode in
work/index dir in upper layer.

> 
> Hi Zhang yi,
> 
> Is that possible to count space and inode of "covered"/"uncovered"/"upper"
> via userspace command tool? I think those info will be useful/important reference
> to admin to set reservation when mounting overlayfs. Then maybe I can add
> some reservation logics in kernel side to avoid EQUOTA error as much as possible.  
> 
> 
Hi Chengguang,

I think it is not hard to count thses 3 kinds of space and inode in userspace tool,
just figure out the already copied-up target in lower is enough. But I have some
questions about the "reservation" feature:

1. If we impliment this tool, does it means that the user should run this tool every
   time before mount overlayfs? and how to pass this info to overlayfs?

2. IIUC, we don't allow the user to create new file and write new data into upper layer
   if "real upper available space/inode" < "uncovered space/inode"?
   If so, think about the the most extreme case: the lower image is bigger than the upper
   space at the first mount, maybe user have this use case and still want to write
   something into upper layer.

3. We should take care of the "copy up directory case" mentioned above, it will cost
   1 inode and 4K space more(on ext4) each directory while copy up if nfs_export=on.

Thanks,
Yi.

Re: "quota-df" feature

[Chengguang Xu]

> Sent: Monday, March 12, 2018 at 2:50 PM
> From: "zhangyi (F)" <yi.zhang@huawei.com>
> To: "Chengguang Xu" <cgxu519@gmx.com>, "Amir Goldstein" <amir73il@gmail.com>
> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> Subject: Re: "quota-df" feature
>
> On 2018/3/11 11:16, Chengguang Xu Wrote:
> >> Sent: Sunday, March 11, 2018 at 4:03 AM
> >> From: "Amir Goldstein" <amir73il@gmail.com>
> >> To: "Chengguang Xu" <cgxu519@gmx.com>
> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >> Subject: Re: "quota-df" feature
> >>
> >> On Sat, Mar 10, 2018 at 8:37 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >>>> Sent: Thursday, March 08, 2018 at 10:36 PM
> >>>> From: "Amir Goldstein" <amir73il@gmail.com>
> >>>> To: "Chengguang Xu" <cgxu519@gmx.com>
> >>>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >>>> Subject: Re: "quota-df" feature
> >>>>
> >>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >>>> [...]
> >>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
> >>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
> >>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
> >>>>>>>>> "covered" problem that you mentioned above. Counting usage information without
> >>>>>>>>> quota might be a hard work when having plenty of files.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Well, its a different use case, but a very well known use case -
> >>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
> >>>>>>>> blocks, but each clone is fully accounted to the user/group/project.
> >>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
> >>>>>>>> that the user can reference, but the host does not pay the cost of sum of all
> >>>>>>>> user quotas.
> >>>>>>>>
> >>>>>>>> Without accounting of "covered" files to begin with, is it possible to
> >>>>>>>> get to a state
> >>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
> >>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
> >>>>>>>> provisioned block,
> >>>>>>>> but a situation that filesystems and administrators try really hard to avoid.
> >>>>>>>
> >>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
> >>>>>>> because of same named file exists in upperdir? or is it meaning the contents in
> >>>>>>> lowerdir but in the merged dir of overlayfs?
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> The former. "covered" file size does not show up in du -s, so the
> >>>>>> merged disk usage
> >>>>>> is <upper used> + <lower used> - <covered used>
> >>>>>
> >>>>> Thanks, got it.
> >>>>> But IIUC, "covered" file does not have chance to copy-up, so
> >>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
> >>>>
> >>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
> >>>> and you want to allow user to touch all the files in the image and
> >>>> create 1GB of new files.
> >>>>
> >>>> If your only tool is project quota on upper then you need to set project
> >>>> quota hard limit to 2GB, but then user can create 2GB of new files and
> >>>> later when touching a lower file, will get EQUOTA on copy up.
> >>>>
> >>>> If you account lower uncovered files to overlay merged quota then
> >>>> you set the merged quota to 2GB and start with 50% used.
> >>>> - Copy up will not change used
> >>>> - Remove of lower will reduce used
> >>>> - You can never get EQUOTA from touching a file
> >>>
> >>> I get your point, I'd like to think it as "reservation" feature we can implement
> >>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
> >>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
> >>> block reservation? Are there many people hope to have this kind of function?
> >>>
> >>
> >> Not sure. I heard this requirement once, but I have a feeling people
> >> still don't know
> >> what they can get out of overlayfs, so they are used to getting along
> >> with what they
> >> have. I think now that docker supports project quotas with overlay2
> >> driver, people
> >> will start using project quota and sooner or later, they will start
> >> hitting EQUOTA on
> >> copy up. That's my prophecy anyway ;-)
> > 
> > I believe in the near future, most container users will experience this.
> > Indoe will be worse compare to space, not only fail on creating new file
> > but also on deleting old uncovered files because of plenty whiteout files. 
> > I remembered someone has already posted fix patch to reduce whiteout files,
> > so after that merged, we can consider block and inode in the same way.
> 
> And copy up directory with nfs_export=on, which will cost one more inode in
> work/index dir in upper layer.
> 
> > 
> > Hi Zhang yi,
> > 
> > Is that possible to count space and inode of "covered"/"uncovered"/"upper"
> > via userspace command tool? I think those info will be useful/important reference
> > to admin to set reservation when mounting overlayfs. Then maybe I can add
> > some reservation logics in kernel side to avoid EQUOTA error as much as possible.  
> > 
> > 
> Hi Chengguang,
> 
> I think it is not hard to count thses 3 kinds of space and inode in userspace tool,
> just figure out the already copied-up target in lower is enough. But I have some
> questions about the "reservation" feature:
> 
> 1. If we impliment this tool, does it means that the user should run this tool every
>    time before mount overlayfs? and how to pass this info to overlayfs?

I propose adding a new mount option to pass this info to overlayfs from userspace,
so it means user/admin can decide how many spaces/inodes they can reserve. Although,
user can set the value of reservation without any information of overlayfs, but I guess
they will be happy/grateful if there a tool for checking that even if they don't use
'reservation' function.

I think it would be better counting those numbers before mounting, if we do the same
thing during mounting or inside filesystem, users will think weird by taking much time
on mounting because mount should be finish in a short period in a common sense.

> 
> 2. IIUC, we don't allow the user to create new file and write new data into upper layer
>    if "real upper available space/inode" < "uncovered space/inode"?
>    If so, think about the the most extreme case: the lower image is bigger than the upper
>    space at the first mount, maybe user have this use case and still want to write
>    something into upper layer.

Yes, that's reasonable. So we let user decide how many resource should be reserved.
I'd like to provide a function to help user but don't want to limit anything.


> 
> 3. We should take care of the "copy up directory case" mentioned above, it will cost
>    1 inode and 4K space more(on ext4) each directory while copy up if nfs_export=on.

I haven't thought very carefully for the limiting logic in detail, but I think we don't
have to control things so accurately, as you know, whiteout files are also wasting of
free inode, even if we make several deleted files link to one whiteout file, we still
don't know accurately how many whiteout files will be created.


Thanks,
Chengguang.

Re: "quota-df" feature

[Amir Goldstein]

On Mon, Mar 12, 2018 at 9:36 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> Sent: Monday, March 12, 2018 at 2:50 PM
>> From: "zhangyi (F)" <yi.zhang@huawei.com>
>> To: "Chengguang Xu" <cgxu519@gmx.com>, "Amir Goldstein" <amir73il@gmail.com>
>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> Subject: Re: "quota-df" feature
>>
>> On 2018/3/11 11:16, Chengguang Xu Wrote:
>> >> Sent: Sunday, March 11, 2018 at 4:03 AM
>> >> From: "Amir Goldstein" <amir73il@gmail.com>
>> >> To: "Chengguang Xu" <cgxu519@gmx.com>
>> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> >> Subject: Re: "quota-df" feature
>> >>
>> >> On Sat, Mar 10, 2018 at 8:37 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> >>>> Sent: Thursday, March 08, 2018 at 10:36 PM
>> >>>> From: "Amir Goldstein" <amir73il@gmail.com>
>> >>>> To: "Chengguang Xu" <cgxu519@gmx.com>
>> >>>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> >>>> Subject: Re: "quota-df" feature
>> >>>>
>> >>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> >>>> [...]
>> >>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
>> >>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>> >>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
>> >>>>>>>>> "covered" problem that you mentioned above. Counting usage information without
>> >>>>>>>>> quota might be a hard work when having plenty of files.
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Well, its a different use case, but a very well known use case -
>> >>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>> >>>>>>>> blocks, but each clone is fully accounted to the user/group/project.
>> >>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
>> >>>>>>>> that the user can reference, but the host does not pay the cost of sum of all
>> >>>>>>>> user quotas.
>> >>>>>>>>
>> >>>>>>>> Without accounting of "covered" files to begin with, is it possible to
>> >>>>>>>> get to a state
>> >>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>> >>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>> >>>>>>>> provisioned block,
>> >>>>>>>> but a situation that filesystems and administrators try really hard to avoid.
>> >>>>>>>
>> >>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>> >>>>>>> because of same named file exists in upperdir? or is it meaning the contents in
>> >>>>>>> lowerdir but in the merged dir of overlayfs?
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> The former. "covered" file size does not show up in du -s, so the
>> >>>>>> merged disk usage
>> >>>>>> is <upper used> + <lower used> - <covered used>
>> >>>>>
>> >>>>> Thanks, got it.
>> >>>>> But IIUC, "covered" file does not have chance to copy-up, so
>> >>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
>> >>>>
>> >>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
>> >>>> and you want to allow user to touch all the files in the image and
>> >>>> create 1GB of new files.
>> >>>>
>> >>>> If your only tool is project quota on upper then you need to set project
>> >>>> quota hard limit to 2GB, but then user can create 2GB of new files and
>> >>>> later when touching a lower file, will get EQUOTA on copy up.
>> >>>>
>> >>>> If you account lower uncovered files to overlay merged quota then
>> >>>> you set the merged quota to 2GB and start with 50% used.
>> >>>> - Copy up will not change used
>> >>>> - Remove of lower will reduce used
>> >>>> - You can never get EQUOTA from touching a file
>> >>>
>> >>> I get your point, I'd like to think it as "reservation" feature we can implement
>> >>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
>> >>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
>> >>> block reservation? Are there many people hope to have this kind of function?
>> >>>
>> >>
>> >> Not sure. I heard this requirement once, but I have a feeling people
>> >> still don't know
>> >> what they can get out of overlayfs, so they are used to getting along
>> >> with what they
>> >> have. I think now that docker supports project quotas with overlay2
>> >> driver, people
>> >> will start using project quota and sooner or later, they will start
>> >> hitting EQUOTA on
>> >> copy up. That's my prophecy anyway ;-)
>> >
>> > I believe in the near future, most container users will experience this.
>> > Indoe will be worse compare to space, not only fail on creating new file
>> > but also on deleting old uncovered files because of plenty whiteout files.
>> > I remembered someone has already posted fix patch to reduce whiteout files,
>> > so after that merged, we can consider block and inode in the same way.
>>
>> And copy up directory with nfs_export=on, which will cost one more inode in
>> work/index dir in upper layer.
>>
>> >
>> > Hi Zhang yi,
>> >
>> > Is that possible to count space and inode of "covered"/"uncovered"/"upper"
>> > via userspace command tool? I think those info will be useful/important reference
>> > to admin to set reservation when mounting overlayfs. Then maybe I can add
>> > some reservation logics in kernel side to avoid EQUOTA error as much as possible.
>> >
>> >
>> Hi Chengguang,
>>
>> I think it is not hard to count thses 3 kinds of space and inode in userspace tool,
>> just figure out the already copied-up target in lower is enough. But I have some
>> questions about the "reservation" feature:
>>
>> 1. If we impliment this tool, does it means that the user should run this tool every
>>    time before mount overlayfs? and how to pass this info to overlayfs?
>
> I propose adding a new mount option to pass this info to overlayfs from userspace,
> so it means user/admin can decide how many spaces/inodes they can reserve. Although,
> user can set the value of reservation without any information of overlayfs, but I guess
> they will be happy/grateful if there a tool for checking that even if they don't use
> 'reservation' function.
>
> I think it would be better counting those numbers before mounting, if we do the same
> thing during mounting or inside filesystem, users will think weird by taking much time
> on mounting because mount should be finish in a short period in a common sense.
>

Before proposing an overlay specific interface for reservation via
mount options, we
should try to use quotatools existing APIs and common quota practices
used by other
file systems. If we do propose to extend the quota APIs for reservation (I'm not
convinced at all that we do) then we should start a thread on fsdevel
to consider other
file systems needs for space reservation (e.g. tune2fs -r).

Cheers,
Amir.

Re: "quota-df" feature

[Chengguang Xu]

> Sent: Monday, March 12, 2018 at 4:55 PM
> From: "Amir Goldstein" <amir73il@gmail.com>
> To: "Chengguang Xu" <cgxu519@gmx.com>
> Cc: "zhangyi (F)" <yi.zhang@huawei.com>, overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> Subject: Re: "quota-df" feature
>
> On Mon, Mar 12, 2018 at 9:36 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> Sent: Monday, March 12, 2018 at 2:50 PM
> >> From: "zhangyi (F)" <yi.zhang@huawei.com>
> >> To: "Chengguang Xu" <cgxu519@gmx.com>, "Amir Goldstein" <amir73il@gmail.com>
> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >> Subject: Re: "quota-df" feature
> >>
> >> On 2018/3/11 11:16, Chengguang Xu Wrote:
> >> >> Sent: Sunday, March 11, 2018 at 4:03 AM
> >> >> From: "Amir Goldstein" <amir73il@gmail.com>
> >> >> To: "Chengguang Xu" <cgxu519@gmx.com>
> >> >> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >> >> Subject: Re: "quota-df" feature
> >> >>
> >> >> On Sat, Mar 10, 2018 at 8:37 AM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> >>>> Sent: Thursday, March 08, 2018 at 10:36 PM
> >> >>>> From: "Amir Goldstein" <amir73il@gmail.com>
> >> >>>> To: "Chengguang Xu" <cgxu519@gmx.com>
> >> >>>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
> >> >>>> Subject: Re: "quota-df" feature
> >> >>>>
> >> >>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
> >> >>>> [...]
> >> >>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
> >> >>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
> >> >>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
> >> >>>>>>>>> "covered" problem that you mentioned above. Counting usage information without
> >> >>>>>>>>> quota might be a hard work when having plenty of files.
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> Well, its a different use case, but a very well known use case -
> >> >>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
> >> >>>>>>>> blocks, but each clone is fully accounted to the user/group/project.
> >> >>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
> >> >>>>>>>> that the user can reference, but the host does not pay the cost of sum of all
> >> >>>>>>>> user quotas.
> >> >>>>>>>>
> >> >>>>>>>> Without accounting of "covered" files to begin with, is it possible to
> >> >>>>>>>> get to a state
> >> >>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
> >> >>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
> >> >>>>>>>> provisioned block,
> >> >>>>>>>> but a situation that filesystems and administrators try really hard to avoid.
> >> >>>>>>>
> >> >>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
> >> >>>>>>> because of same named file exists in upperdir? or is it meaning the contents in
> >> >>>>>>> lowerdir but in the merged dir of overlayfs?
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>> The former. "covered" file size does not show up in du -s, so the
> >> >>>>>> merged disk usage
> >> >>>>>> is <upper used> + <lower used> - <covered used>
> >> >>>>>
> >> >>>>> Thanks, got it.
> >> >>>>> But IIUC, "covered" file does not have chance to copy-up, so
> >> >>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
> >> >>>>
> >> >>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
> >> >>>> and you want to allow user to touch all the files in the image and
> >> >>>> create 1GB of new files.
> >> >>>>
> >> >>>> If your only tool is project quota on upper then you need to set project
> >> >>>> quota hard limit to 2GB, but then user can create 2GB of new files and
> >> >>>> later when touching a lower file, will get EQUOTA on copy up.
> >> >>>>
> >> >>>> If you account lower uncovered files to overlay merged quota then
> >> >>>> you set the merged quota to 2GB and start with 50% used.
> >> >>>> - Copy up will not change used
> >> >>>> - Remove of lower will reduce used
> >> >>>> - You can never get EQUOTA from touching a file
> >> >>>
> >> >>> I get your point, I'd like to think it as "reservation" feature we can implement
> >> >>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
> >> >>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
> >> >>> block reservation? Are there many people hope to have this kind of function?
> >> >>>
> >> >>
> >> >> Not sure. I heard this requirement once, but I have a feeling people
> >> >> still don't know
> >> >> what they can get out of overlayfs, so they are used to getting along
> >> >> with what they
> >> >> have. I think now that docker supports project quotas with overlay2
> >> >> driver, people
> >> >> will start using project quota and sooner or later, they will start
> >> >> hitting EQUOTA on
> >> >> copy up. That's my prophecy anyway ;-)
> >> >
> >> > I believe in the near future, most container users will experience this.
> >> > Indoe will be worse compare to space, not only fail on creating new file
> >> > but also on deleting old uncovered files because of plenty whiteout files.
> >> > I remembered someone has already posted fix patch to reduce whiteout files,
> >> > so after that merged, we can consider block and inode in the same way.
> >>
> >> And copy up directory with nfs_export=on, which will cost one more inode in
> >> work/index dir in upper layer.
> >>
> >> >
> >> > Hi Zhang yi,
> >> >
> >> > Is that possible to count space and inode of "covered"/"uncovered"/"upper"
> >> > via userspace command tool? I think those info will be useful/important reference
> >> > to admin to set reservation when mounting overlayfs. Then maybe I can add
> >> > some reservation logics in kernel side to avoid EQUOTA error as much as possible.
> >> >
> >> >
> >> Hi Chengguang,
> >>
> >> I think it is not hard to count thses 3 kinds of space and inode in userspace tool,
> >> just figure out the already copied-up target in lower is enough. But I have some
> >> questions about the "reservation" feature:
> >>
> >> 1. If we impliment this tool, does it means that the user should run this tool every
> >>    time before mount overlayfs? and how to pass this info to overlayfs?
> >
> > I propose adding a new mount option to pass this info to overlayfs from userspace,
> > so it means user/admin can decide how many spaces/inodes they can reserve. Although,
> > user can set the value of reservation without any information of overlayfs, but I guess
> > they will be happy/grateful if there a tool for checking that even if they don't use
> > 'reservation' function.
> >
> > I think it would be better counting those numbers before mounting, if we do the same
> > thing during mounting or inside filesystem, users will think weird by taking much time
> > on mounting because mount should be finish in a short period in a common sense.
> >
> 
> Before proposing an overlay specific interface for reservation via
> mount options, we
> should try to use quotatools existing APIs and common quota practices
> used by other
> file systems. If we do propose to extend the quota APIs for reservation (I'm not
> convinced at all that we do) then we should start a thread on fsdevel
> to consider other
> file systems needs for space reservation (e.g. tune2fs -r).

IMO, I don't think reservation should be a part of common quota function
or at least not in current stage. Why? I've not seen good benefit by
introducing it for common block based filesystems, current quota has already
have soft and hard limits for flexibly handling temporary burst. Reservation
might be important only for overlayfs in quota environment, so I think I don't
have enough reason to persuade people to accept adding new interface which
they are never used.

For space reservation, if underlying filesystems support then overlayfs can try
to utilize the function, though I'm not sure if underlying filesystem allows to do.


Thanks,
Chengguang.

Re: "quota-df" feature

[cgxu519]

在 2018年3月10日，下午2:37，Chengguang Xu <cgxu519@gmx.com> 写道：
> 
>> Sent: Thursday, March 08, 2018 at 10:36 PM
>> From: "Amir Goldstein" <amir73il@gmail.com>
>> To: "Chengguang Xu" <cgxu519@gmx.com>
>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>> Subject: Re: "quota-df" feature
>> 
>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>> [...]
>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
>>>>>>> "covered" problem that you mentioned above. Counting usage information without
>>>>>>> quota might be a hard work when having plenty of files.
>>>>>>> 
>>>>>> 
>>>>>> Well, its a different use case, but a very well known use case -
>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>>>>>> blocks, but each clone is fully accounted to the user/group/project.
>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
>>>>>> that the user can reference, but the host does not pay the cost of sum of all
>>>>>> user quotas.
>>>>>> 
>>>>>> Without accounting of "covered" files to begin with, is it possible to
>>>>>> get to a state
>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>>>>>> provisioned block,
>>>>>> but a situation that filesystems and administrators try really hard to avoid.
>>>>> 
>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>>>>> because of same named file exists in upperdir? or is it meaning the contents in
>>>>> lowerdir but in the merged dir of overlayfs?
>>>>> 
>>>>> 
>>>> 
>>>> The former. "covered" file size does not show up in du -s, so the
>>>> merged disk usage
>>>> is <upper used> + <lower used> - <covered used>
>>> 
>>> Thanks, got it.
>>> But IIUC, "covered" file does not have chance to copy-up, so
>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
>> 
>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
>> and you want to allow user to touch all the files in the image and
>> create 1GB of new files.
>> 
>> If your only tool is project quota on upper then you need to set project
>> quota hard limit to 2GB, but then user can create 2GB of new files and
>> later when touching a lower file, will get EQUOTA on copy up.
>> 
>> If you account lower uncovered files to overlay merged quota then
>> you set the merged quota to 2GB and start with 50% used.
>> - Copy up will not change used
>> - Remove of lower will reduce used
>> - You can never get EQUOTA from touching a file
> 
> I get your point, I'd like to think it as "reservation" feature we can implement
> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up. 
> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
> block reservation? Are there many people hope to have this kind of function?
> 
> 
>>>>>> All I am saying is that it is "not hard" (TM) to keep track of
>>>>>> "covered" files disk usage
>>>>>> and "not hard" to re-calculate "covered" files disk usage when full
>>>>>> indexing is enabled.
>>>>>>> I don't know exactly what will happen when combining index and nfs_export options,
>>>>>>> I need to read and understand related code later.
>>>>>>> 
>>>>>> 
>>>>>> nfs_export REQUIRES index and implies indexing of all files on copy up, not
>>>>>> only lower hardlinks.
>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Here are some test examples share with you (based on ext4):
>>>>>>>>> 
>>>>>>>>> 1) project quota enabled && without hard-limit
>>>>>>>>> 
>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>> overlay          92G  201M   91G   1% /mnt/test3/df/merged
>>>>>>>>> 
>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>> overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>>>>>>>>> 
>>>>>>>>> 2) project quota enabled && with hard-limit
>>>>>>>>> 
>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>> overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>>>>>>>>> 
>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>> overlay          1000     8   992    1% /mnt/test3/df/merged
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> I can't follow from the example below what is the expected result and why
>>>>>>>> if you add the quota setup commands that could be useful.
>>>>>>> 
>>>>>>> Underlying fs using ext4 and mkfs/mount with project quota option and
>>>>>>> all needed directories(lowerdir, upperdir, workdir, merged) are set same
>>>>>>> project quota. Current quota-df implementation only adjust the couting
>>>>>>> information when having upperdir.
>>>>>>> 
>>>>>> 
>>>>>> Really? why is lowerdir on the same project id? For containers quota
>>>>>> only upper/work should be on the same project id. lowerdir should belong
>>>>>> to a shared project or no project at all.
>>>>>> In current docker implementation, every overlay2 driver root dir
>>>>>> is assigned a different project id, but lowerdir are symlinks to another
>>>>>> image root dir.
>>>>> 
>>>>> For the setting of docker, you are completely right. My description of tesing
>>>>> environment in previous email is only for simple kernel testing and explaining
>>>>> the condition of testing result above, not specific to docker.
>>>>> 
>>>>>> 
>>>>>>> In case 1:
>>>>>>> There is no hardlimit/softlimit, so the expected result as below.
>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>       include upperdir && workdir.
>>>>>>> Avail: The avail of underlying fs
>>>>>>> Total: Used + Avail
>>>>>>> 
>>>>>>> #upperdir used 201M and /mnt/test3 used 2.5G
>>>>>>> 
>>>>>>> In case 2:
>>>>>>> Project quota hardlimits are block count = 1G, inode count = 1000.
>>>>>>> So the expected result as below.
>>>>>>> 
>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>       include upperdir && workdir.
>>>>>>> Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
>>>>>>> Total: Used + Avail
>>>>>>> 
>>>>>> 
>>>>>> Are those "expected" results inline with what df shows on project quota
>>>>>> directories without overlayfs? If not, and the new behavior makes sense,
>>>>>> why change overlayfs and not change 'df'?
>>>>> 
>>>>> No, those results are based on my change of overlayfs. At the first,
>>>>> I didn't notice that underlying filesystems had already implemented
>>>>> 'quota-df' function. So I plan to do something in overlayfs because
>>>>> it's better than doing same things in every low level filesystems.
>>>>> But now, I'm more willing to persuade xfs/ext4 people to midify the
>>>>> detail mechanism of 'quota-df' in specific filesystems.
>>>>> 
>>>>> 'df', hmm... Both solutions could work I think.
>>>>> 
>>>> 
>>>> I don't thing it is the underlying filesystem that implements quota-df
>>>> I think it is 'df' itself. I was also surprised to learn than, but at least when
>>>> you set project quota with xfs_quota via /etc/projects /etc/projid
>>>> df just shows you the project directories as if they were mount points.
>>>> I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
>>>> is no different.
>>> 
>>> That sounds really interesting and that must be special version of df command.
>>> On CentOS I have never seen that. :-(
>> 
>> I donno, maybe I dreamed of seeing it... most likely I am confusing seeing
>> the correct df usage on overlayfs mount with upper that has project quota.
>> 
>>> In any case if you take a quick look at below functions in the code, you will probably
>>> believe what I said before. If you stop calling those functions in the kernel code,
>>> then I guess all magic will be gone and never turn back again. :-)
>>> 
>>> xfs:
>>> xfs_qm_statvfs
>>> 
>>> ext4:
>>> ext4_statfs_project
>>> 
>>> f2fs:
>>> f2fs_statfs_project
>> =【--07 
>>> 
>> 
>> I see. so I'll wait for your RFC patch to see what ovl_statfs_project brings
>> to the table.
> 
> Seems there is nothing more to do unless we add more features like 'reservation' we discussed above.
> In this case I think we should consider adding 'reservation amount' to bfree, and bavail represents
> the real free space amount that can be utilized by new files.


Most of time underlying filesystem’s quota-df works well, but when real filesystem’s avail is lower than
project quota’s avail then the result is quite confusing. I’ve only tested on xfs but I think ext4 is
similar because they have same quota-df logic.

For example, if we have 100GB xfs filesystem(/mnt/test2) and we have
3 directories(pq1, pq2, pq3) inside it, each directory sets project quota. 
(block hard limit up to 10GB)

When avail space of real filesystem is only left 3.2MB, but when running df for
pg1,pg2,pg3 then avail space is 9.5GB, this is much more than real filesystem. 


Detail output:

$ df -h /mnt/test2
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2       100G  100G  3.2M 100% /mnt/test2

$ df -h /mnt/test2/pq1
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2        10G  570M  9.5G   6% /mnt/test2

$ df -h /mnt/test2/pq2
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2        10G  570M  9.5G   6% /mnt/test2

$ df -h /mnt/test2/pq3
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2        10G  570M  9.5G   6% /mnt/test2


So I just think if we can adjust size/used/avail in overlayfs layer like below,
it maybe a little bit more helpful for our users. What do you think for this? 


$ df -h /mnt/test2
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2       100G  100G  3.2M 100% /mnt/test2

$ df -h /mnt/test2/pq1
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2       574M  570M  3.2M 100% /mnt/test2

$ df -h /mnt/test2/pq2
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2       574M  570M  3.2M 100% /mnt/test2

$ df -h /mnt/test2/pq3
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2       574M  570M  3.2M 100% /mnt/test2


Thanks,
Chengguang.

Re: "quota-df" feature

[Amir Goldstein]

On Sun, Apr 1, 2018 at 9:35 AM,  <cgxu519@gmx.com> wrote:
> 在 2018年3月10日，下午2:37，Chengguang Xu <cgxu519@gmx.com> 写道：
>>
>>> Sent: Thursday, March 08, 2018 at 10:36 PM
>>> From: "Amir Goldstein" <amir73il@gmail.com>
>>> To: "Chengguang Xu" <cgxu519@gmx.com>
>>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>>> Subject: Re: "quota-df" feature
>>>
>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>>> [...]
>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
>>>>>>>> "covered" problem that you mentioned above. Counting usage information without
>>>>>>>> quota might be a hard work when having plenty of files.
>>>>>>>>
>>>>>>>
>>>>>>> Well, its a different use case, but a very well known use case -
>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>>>>>>> blocks, but each clone is fully accounted to the user/group/project.
>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
>>>>>>> that the user can reference, but the host does not pay the cost of sum of all
>>>>>>> user quotas.
>>>>>>>
>>>>>>> Without accounting of "covered" files to begin with, is it possible to
>>>>>>> get to a state
>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>>>>>>> provisioned block,
>>>>>>> but a situation that filesystems and administrators try really hard to avoid.
>>>>>>
>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>>>>>> because of same named file exists in upperdir? or is it meaning the contents in
>>>>>> lowerdir but in the merged dir of overlayfs?
>>>>>>
>>>>>>
>>>>>
>>>>> The former. "covered" file size does not show up in du -s, so the
>>>>> merged disk usage
>>>>> is <upper used> + <lower used> - <covered used>
>>>>
>>>> Thanks, got it.
>>>> But IIUC, "covered" file does not have chance to copy-up, so
>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
>>>
>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
>>> and you want to allow user to touch all the files in the image and
>>> create 1GB of new files.
>>>
>>> If your only tool is project quota on upper then you need to set project
>>> quota hard limit to 2GB, but then user can create 2GB of new files and
>>> later when touching a lower file, will get EQUOTA on copy up.
>>>
>>> If you account lower uncovered files to overlay merged quota then
>>> you set the merged quota to 2GB and start with 50% used.
>>> - Copy up will not change used
>>> - Remove of lower will reduce used
>>> - You can never get EQUOTA from touching a file
>>
>> I get your point, I'd like to think it as "reservation" feature we can implement
>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
>> block reservation? Are there many people hope to have this kind of function?
>>
>>
>>>>>>> All I am saying is that it is "not hard" (TM) to keep track of
>>>>>>> "covered" files disk usage
>>>>>>> and "not hard" to re-calculate "covered" files disk usage when full
>>>>>>> indexing is enabled.
>>>>>>>> I don't know exactly what will happen when combining index and nfs_export options,
>>>>>>>> I need to read and understand related code later.
>>>>>>>>
>>>>>>>
>>>>>>> nfs_export REQUIRES index and implies indexing of all files on copy up, not
>>>>>>> only lower hardlinks.
>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Here are some test examples share with you (based on ext4):
>>>>>>>>>>
>>>>>>>>>> 1) project quota enabled && without hard-limit
>>>>>>>>>>
>>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>>> overlay          92G  201M   91G   1% /mnt/test3/df/merged
>>>>>>>>>>
>>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>>> overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>>>>>>>>>>
>>>>>>>>>> 2) project quota enabled && with hard-limit
>>>>>>>>>>
>>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>>> overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>>>>>>>>>>
>>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>>> overlay          1000     8   992    1% /mnt/test3/df/merged
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I can't follow from the example below what is the expected result and why
>>>>>>>>> if you add the quota setup commands that could be useful.
>>>>>>>>
>>>>>>>> Underlying fs using ext4 and mkfs/mount with project quota option and
>>>>>>>> all needed directories(lowerdir, upperdir, workdir, merged) are set same
>>>>>>>> project quota. Current quota-df implementation only adjust the couting
>>>>>>>> information when having upperdir.
>>>>>>>>
>>>>>>>
>>>>>>> Really? why is lowerdir on the same project id? For containers quota
>>>>>>> only upper/work should be on the same project id. lowerdir should belong
>>>>>>> to a shared project or no project at all.
>>>>>>> In current docker implementation, every overlay2 driver root dir
>>>>>>> is assigned a different project id, but lowerdir are symlinks to another
>>>>>>> image root dir.
>>>>>>
>>>>>> For the setting of docker, you are completely right. My description of tesing
>>>>>> environment in previous email is only for simple kernel testing and explaining
>>>>>> the condition of testing result above, not specific to docker.
>>>>>>
>>>>>>>
>>>>>>>> In case 1:
>>>>>>>> There is no hardlimit/softlimit, so the expected result as below.
>>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>>       include upperdir && workdir.
>>>>>>>> Avail: The avail of underlying fs
>>>>>>>> Total: Used + Avail
>>>>>>>>
>>>>>>>> #upperdir used 201M and /mnt/test3 used 2.5G
>>>>>>>>
>>>>>>>> In case 2:
>>>>>>>> Project quota hardlimits are block count = 1G, inode count = 1000.
>>>>>>>> So the expected result as below.
>>>>>>>>
>>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>>       include upperdir && workdir.
>>>>>>>> Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
>>>>>>>> Total: Used + Avail
>>>>>>>>
>>>>>>>
>>>>>>> Are those "expected" results inline with what df shows on project quota
>>>>>>> directories without overlayfs? If not, and the new behavior makes sense,
>>>>>>> why change overlayfs and not change 'df'?
>>>>>>
>>>>>> No, those results are based on my change of overlayfs. At the first,
>>>>>> I didn't notice that underlying filesystems had already implemented
>>>>>> 'quota-df' function. So I plan to do something in overlayfs because
>>>>>> it's better than doing same things in every low level filesystems.
>>>>>> But now, I'm more willing to persuade xfs/ext4 people to midify the
>>>>>> detail mechanism of 'quota-df' in specific filesystems.
>>>>>>
>>>>>> 'df', hmm... Both solutions could work I think.
>>>>>>
>>>>>
>>>>> I don't thing it is the underlying filesystem that implements quota-df
>>>>> I think it is 'df' itself. I was also surprised to learn than, but at least when
>>>>> you set project quota with xfs_quota via /etc/projects /etc/projid
>>>>> df just shows you the project directories as if they were mount points.
>>>>> I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
>>>>> is no different.
>>>>
>>>> That sounds really interesting and that must be special version of df command.
>>>> On CentOS I have never seen that. :-(
>>>
>>> I donno, maybe I dreamed of seeing it... most likely I am confusing seeing
>>> the correct df usage on overlayfs mount with upper that has project quota.
>>>
>>>> In any case if you take a quick look at below functions in the code, you will probably
>>>> believe what I said before. If you stop calling those functions in the kernel code,
>>>> then I guess all magic will be gone and never turn back again. :-)
>>>>
>>>> xfs:
>>>> xfs_qm_statvfs
>>>>
>>>> ext4:
>>>> ext4_statfs_project
>>>>
>>>> f2fs:
>>>> f2fs_statfs_project
>>> =【--07
>>>>
>>>
>>> I see. so I'll wait for your RFC patch to see what ovl_statfs_project brings
>>> to the table.
>>
>> Seems there is nothing more to do unless we add more features like 'reservation' we discussed above.
>> In this case I think we should consider adding 'reservation amount' to bfree, and bavail represents
>> the real free space amount that can be utilized by new files.
>
>
> Most of time underlying filesystem’s quota-df works well, but when real filesystem’s avail is lower than
> project quota’s avail then the result is quite confusing. I’ve only tested on xfs but I think ext4 is
> similar because they have same quota-df logic.
>
> For example, if we have 100GB xfs filesystem(/mnt/test2) and we have
> 3 directories(pq1, pq2, pq3) inside it, each directory sets project quota.
> (block hard limit up to 10GB)
>
> When avail space of real filesystem is only left 3.2MB, but when running df for
> pg1,pg2,pg3 then avail space is 9.5GB, this is much more than real filesystem.
>
>
> Detail output:
>
> $ df -h /mnt/test2
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb2       100G  100G  3.2M 100% /mnt/test2
>
> $ df -h /mnt/test2/pq1
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>
> $ df -h /mnt/test2/pq2
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>
> $ df -h /mnt/test2/pq3
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>
>
> So I just think if we can adjust size/used/avail in overlayfs layer like below,
> it maybe a little bit more helpful for our users. What do you think for this?
>
>
> $ df -h /mnt/test2
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb2       100G  100G  3.2M 100% /mnt/test2
>
> $ df -h /mnt/test2/pq1
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>
> $ df -h /mnt/test2/pq2
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>
> $ df -h /mnt/test2/pq3
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>
>
> Thanks,
> Chengguang.
>

Re: "quota-df" feature

[Amir Goldstein]

On Sun, Apr 1, 2018 at 9:53 AM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Sun, Apr 1, 2018 at 9:35 AM,  <cgxu519@gmx.com> wrote:
>> 在 2018年3月10日，下午2:37，Chengguang Xu <cgxu519@gmx.com> 写道：
>>>
>>>> Sent: Thursday, March 08, 2018 at 10:36 PM
>>>> From: "Amir Goldstein" <amir73il@gmail.com>
>>>> To: "Chengguang Xu" <cgxu519@gmx.com>
>>>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>>>> Subject: Re: "quota-df" feature
>>>>
>>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>>>> [...]
>>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
>>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
>>>>>>>>> "covered" problem that you mentioned above. Counting usage information without
>>>>>>>>> quota might be a hard work when having plenty of files.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Well, its a different use case, but a very well known use case -
>>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>>>>>>>> blocks, but each clone is fully accounted to the user/group/project.
>>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
>>>>>>>> that the user can reference, but the host does not pay the cost of sum of all
>>>>>>>> user quotas.
>>>>>>>>
>>>>>>>> Without accounting of "covered" files to begin with, is it possible to
>>>>>>>> get to a state
>>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>>>>>>>> provisioned block,
>>>>>>>> but a situation that filesystems and administrators try really hard to avoid.
>>>>>>>
>>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>>>>>>> because of same named file exists in upperdir? or is it meaning the contents in
>>>>>>> lowerdir but in the merged dir of overlayfs?
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> The former. "covered" file size does not show up in du -s, so the
>>>>>> merged disk usage
>>>>>> is <upper used> + <lower used> - <covered used>
>>>>>
>>>>> Thanks, got it.
>>>>> But IIUC, "covered" file does not have chance to copy-up, so
>>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
>>>>
>>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
>>>> and you want to allow user to touch all the files in the image and
>>>> create 1GB of new files.
>>>>
>>>> If your only tool is project quota on upper then you need to set project
>>>> quota hard limit to 2GB, but then user can create 2GB of new files and
>>>> later when touching a lower file, will get EQUOTA on copy up.
>>>>
>>>> If you account lower uncovered files to overlay merged quota then
>>>> you set the merged quota to 2GB and start with 50% used.
>>>> - Copy up will not change used
>>>> - Remove of lower will reduce used
>>>> - You can never get EQUOTA from touching a file
>>>
>>> I get your point, I'd like to think it as "reservation" feature we can implement
>>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
>>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
>>> block reservation? Are there many people hope to have this kind of function?
>>>
>>>
>>>>>>>> All I am saying is that it is "not hard" (TM) to keep track of
>>>>>>>> "covered" files disk usage
>>>>>>>> and "not hard" to re-calculate "covered" files disk usage when full
>>>>>>>> indexing is enabled.
>>>>>>>>> I don't know exactly what will happen when combining index and nfs_export options,
>>>>>>>>> I need to read and understand related code later.
>>>>>>>>>
>>>>>>>>
>>>>>>>> nfs_export REQUIRES index and implies indexing of all files on copy up, not
>>>>>>>> only lower hardlinks.
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Here are some test examples share with you (based on ext4):
>>>>>>>>>>>
>>>>>>>>>>> 1) project quota enabled && without hard-limit
>>>>>>>>>>>
>>>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>>>> overlay          92G  201M   91G   1% /mnt/test3/df/merged
>>>>>>>>>>>
>>>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>>>> overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>>>>>>>>>>>
>>>>>>>>>>> 2) project quota enabled && with hard-limit
>>>>>>>>>>>
>>>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>>>> overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>>>>>>>>>>>
>>>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>>>> overlay          1000     8   992    1% /mnt/test3/df/merged
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I can't follow from the example below what is the expected result and why
>>>>>>>>>> if you add the quota setup commands that could be useful.
>>>>>>>>>
>>>>>>>>> Underlying fs using ext4 and mkfs/mount with project quota option and
>>>>>>>>> all needed directories(lowerdir, upperdir, workdir, merged) are set same
>>>>>>>>> project quota. Current quota-df implementation only adjust the couting
>>>>>>>>> information when having upperdir.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Really? why is lowerdir on the same project id? For containers quota
>>>>>>>> only upper/work should be on the same project id. lowerdir should belong
>>>>>>>> to a shared project or no project at all.
>>>>>>>> In current docker implementation, every overlay2 driver root dir
>>>>>>>> is assigned a different project id, but lowerdir are symlinks to another
>>>>>>>> image root dir.
>>>>>>>
>>>>>>> For the setting of docker, you are completely right. My description of tesing
>>>>>>> environment in previous email is only for simple kernel testing and explaining
>>>>>>> the condition of testing result above, not specific to docker.
>>>>>>>
>>>>>>>>
>>>>>>>>> In case 1:
>>>>>>>>> There is no hardlimit/softlimit, so the expected result as below.
>>>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>>>       include upperdir && workdir.
>>>>>>>>> Avail: The avail of underlying fs
>>>>>>>>> Total: Used + Avail
>>>>>>>>>
>>>>>>>>> #upperdir used 201M and /mnt/test3 used 2.5G
>>>>>>>>>
>>>>>>>>> In case 2:
>>>>>>>>> Project quota hardlimits are block count = 1G, inode count = 1000.
>>>>>>>>> So the expected result as below.
>>>>>>>>>
>>>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>>>       include upperdir && workdir.
>>>>>>>>> Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
>>>>>>>>> Total: Used + Avail
>>>>>>>>>
>>>>>>>>
>>>>>>>> Are those "expected" results inline with what df shows on project quota
>>>>>>>> directories without overlayfs? If not, and the new behavior makes sense,
>>>>>>>> why change overlayfs and not change 'df'?
>>>>>>>
>>>>>>> No, those results are based on my change of overlayfs. At the first,
>>>>>>> I didn't notice that underlying filesystems had already implemented
>>>>>>> 'quota-df' function. So I plan to do something in overlayfs because
>>>>>>> it's better than doing same things in every low level filesystems.
>>>>>>> But now, I'm more willing to persuade xfs/ext4 people to midify the
>>>>>>> detail mechanism of 'quota-df' in specific filesystems.
>>>>>>>
>>>>>>> 'df', hmm... Both solutions could work I think.
>>>>>>>
>>>>>>
>>>>>> I don't thing it is the underlying filesystem that implements quota-df
>>>>>> I think it is 'df' itself. I was also surprised to learn than, but at least when
>>>>>> you set project quota with xfs_quota via /etc/projects /etc/projid
>>>>>> df just shows you the project directories as if they were mount points.
>>>>>> I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
>>>>>> is no different.
>>>>>
>>>>> That sounds really interesting and that must be special version of df command.
>>>>> On CentOS I have never seen that. :-(
>>>>
>>>> I donno, maybe I dreamed of seeing it... most likely I am confusing seeing
>>>> the correct df usage on overlayfs mount with upper that has project quota.
>>>>
>>>>> In any case if you take a quick look at below functions in the code, you will probably
>>>>> believe what I said before. If you stop calling those functions in the kernel code,
>>>>> then I guess all magic will be gone and never turn back again. :-)
>>>>>
>>>>> xfs:
>>>>> xfs_qm_statvfs
>>>>>
>>>>> ext4:
>>>>> ext4_statfs_project
>>>>>
>>>>> f2fs:
>>>>> f2fs_statfs_project
>>>> =【--07
>>>>>
>>>>
>>>> I see. so I'll wait for your RFC patch to see what ovl_statfs_project brings
>>>> to the table.
>>>
>>> Seems there is nothing more to do unless we add more features like 'reservation' we discussed above.
>>> In this case I think we should consider adding 'reservation amount' to bfree, and bavail represents
>>> the real free space amount that can be utilized by new files.
>>
>>
>> Most of time underlying filesystem’s quota-df works well, but when real filesystem’s avail is lower than
>> project quota’s avail then the result is quite confusing. I’ve only tested on xfs but I think ext4 is
>> similar because they have same quota-df logic.
>>
>> For example, if we have 100GB xfs filesystem(/mnt/test2) and we have
>> 3 directories(pq1, pq2, pq3) inside it, each directory sets project quota.
>> (block hard limit up to 10GB)
>>
>> When avail space of real filesystem is only left 3.2MB, but when running df for
>> pg1,pg2,pg3 then avail space is 9.5GB, this is much more than real filesystem.
>>
>>
>> Detail output:
>>
>> $ df -h /mnt/test2
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vdb2       100G  100G  3.2M 100% /mnt/test2
>>
>> $ df -h /mnt/test2/pq1
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>
>> $ df -h /mnt/test2/pq2
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>
>> $ df -h /mnt/test2/pq3
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>
>>
>> So I just think if we can adjust size/used/avail in overlayfs layer like below,
>> it maybe a little bit more helpful for our users. What do you think for this?
>>
>>
>> $ df -h /mnt/test2
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vdb2       100G  100G  3.2M 100% /mnt/test2
>>
>> $ df -h /mnt/test2/pq1
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>
>> $ df -h /mnt/test2/pq2
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>
>> $ df -h /mnt/test2/pq3
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>
>>

OOPS sent by mistake.

Chengguang,

People on this list may have not been following the similar thread on xfs list
and will wonder why the heck total size 574M

You did not explain that is is a limitation of the current statfs() API and how
df presents the information.

My opinion is that at least for overlayfs use case, it is better to change 'df'
then to work around its current limitation and present another missleading
type of information.

It was the same story with btrfs. 'df' was just to simple to meet the demands
of reporting to user the real status of btrfs disk space, so btrfs tools needed
to provide a better 'df'.

IMO, the best course of action is to integrate 'df' with quota tools
information,
so that 'df' has more information and can display more accurate data.

Overlayfs users are used to work in a container environment where not all
utilities work out of the box and may need to use alternative tools or hacks
inside the container, so it won't be a bug hurdle, even if this needs a special
flavor of 'df'.

Cheers,
Amir.

Re: "quota-df" feature

[cgxu519]

在 2018年4月1日，下午3:00，Amir Goldstein <amir73il@gmail.com> 写道：
> 
> On Sun, Apr 1, 2018 at 9:53 AM, Amir Goldstein <amir73il@gmail.com> wrote:
>> On Sun, Apr 1, 2018 at 9:35 AM,  <cgxu519@gmx.com> wrote:
>>> 在 2018年3月10日，下午2:37，Chengguang Xu <cgxu519@gmx.com> 写道：
>>>> 
>>>>> Sent: Thursday, March 08, 2018 at 10:36 PM
>>>>> From: "Amir Goldstein" <amir73il@gmail.com>
>>>>> To: "Chengguang Xu" <cgxu519@gmx.com>
>>>>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>>>>> Subject: Re: "quota-df" feature
>>>>> 
>>>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>>>>> [...]
>>>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
>>>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>>>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
>>>>>>>>>> "covered" problem that you mentioned above. Counting usage information without
>>>>>>>>>> quota might be a hard work when having plenty of files.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Well, its a different use case, but a very well known use case -
>>>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>>>>>>>>> blocks, but each clone is fully accounted to the user/group/project.
>>>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
>>>>>>>>> that the user can reference, but the host does not pay the cost of sum of all
>>>>>>>>> user quotas.
>>>>>>>>> 
>>>>>>>>> Without accounting of "covered" files to begin with, is it possible to
>>>>>>>>> get to a state
>>>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>>>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>>>>>>>>> provisioned block,
>>>>>>>>> but a situation that filesystems and administrators try really hard to avoid.
>>>>>>>> 
>>>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>>>>>>>> because of same named file exists in upperdir? or is it meaning the contents in
>>>>>>>> lowerdir but in the merged dir of overlayfs?
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> The former. "covered" file size does not show up in du -s, so the
>>>>>>> merged disk usage
>>>>>>> is <upper used> + <lower used> - <covered used>
>>>>>> 
>>>>>> Thanks, got it.
>>>>>> But IIUC, "covered" file does not have chance to copy-up, so
>>>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
>>>>> 
>>>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
>>>>> and you want to allow user to touch all the files in the image and
>>>>> create 1GB of new files.
>>>>> 
>>>>> If your only tool is project quota on upper then you need to set project
>>>>> quota hard limit to 2GB, but then user can create 2GB of new files and
>>>>> later when touching a lower file, will get EQUOTA on copy up.
>>>>> 
>>>>> If you account lower uncovered files to overlay merged quota then
>>>>> you set the merged quota to 2GB and start with 50% used.
>>>>> - Copy up will not change used
>>>>> - Remove of lower will reduce used
>>>>> - You can never get EQUOTA from touching a file
>>>> 
>>>> I get your point, I'd like to think it as "reservation" feature we can implement
>>>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
>>>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
>>>> block reservation? Are there many people hope to have this kind of function?
>>>> 
>>>> 
>>>>>>>>> All I am saying is that it is "not hard" (TM) to keep track of
>>>>>>>>> "covered" files disk usage
>>>>>>>>> and "not hard" to re-calculate "covered" files disk usage when full
>>>>>>>>> indexing is enabled.
>>>>>>>>>> I don't know exactly what will happen when combining index and nfs_export options,
>>>>>>>>>> I need to read and understand related code later.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> nfs_export REQUIRES index and implies indexing of all files on copy up, not
>>>>>>>>> only lower hardlinks.
>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Here are some test examples share with you (based on ext4):
>>>>>>>>>>>> 
>>>>>>>>>>>> 1) project quota enabled && without hard-limit
>>>>>>>>>>>> 
>>>>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>>>>> overlay          92G  201M   91G   1% /mnt/test3/df/merged
>>>>>>>>>>>> 
>>>>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>>>>> overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>>>>>>>>>>>> 
>>>>>>>>>>>> 2) project quota enabled && with hard-limit
>>>>>>>>>>>> 
>>>>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>>>>> overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>>>>>>>>>>>> 
>>>>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>>>>> overlay          1000     8   992    1% /mnt/test3/df/merged
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I can't follow from the example below what is the expected result and why
>>>>>>>>>>> if you add the quota setup commands that could be useful.
>>>>>>>>>> 
>>>>>>>>>> Underlying fs using ext4 and mkfs/mount with project quota option and
>>>>>>>>>> all needed directories(lowerdir, upperdir, workdir, merged) are set same
>>>>>>>>>> project quota. Current quota-df implementation only adjust the couting
>>>>>>>>>> information when having upperdir.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Really? why is lowerdir on the same project id? For containers quota
>>>>>>>>> only upper/work should be on the same project id. lowerdir should belong
>>>>>>>>> to a shared project or no project at all.
>>>>>>>>> In current docker implementation, every overlay2 driver root dir
>>>>>>>>> is assigned a different project id, but lowerdir are symlinks to another
>>>>>>>>> image root dir.
>>>>>>>> 
>>>>>>>> For the setting of docker, you are completely right. My description of tesing
>>>>>>>> environment in previous email is only for simple kernel testing and explaining
>>>>>>>> the condition of testing result above, not specific to docker.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> In case 1:
>>>>>>>>>> There is no hardlimit/softlimit, so the expected result as below.
>>>>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>>>>      include upperdir && workdir.
>>>>>>>>>> Avail: The avail of underlying fs
>>>>>>>>>> Total: Used + Avail
>>>>>>>>>> 
>>>>>>>>>> #upperdir used 201M and /mnt/test3 used 2.5G
>>>>>>>>>> 
>>>>>>>>>> In case 2:
>>>>>>>>>> Project quota hardlimits are block count = 1G, inode count = 1000.
>>>>>>>>>> So the expected result as below.
>>>>>>>>>> 
>>>>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>>>>      include upperdir && workdir.
>>>>>>>>>> Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
>>>>>>>>>> Total: Used + Avail
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Are those "expected" results inline with what df shows on project quota
>>>>>>>>> directories without overlayfs? If not, and the new behavior makes sense,
>>>>>>>>> why change overlayfs and not change 'df'?
>>>>>>>> 
>>>>>>>> No, those results are based on my change of overlayfs. At the first,
>>>>>>>> I didn't notice that underlying filesystems had already implemented
>>>>>>>> 'quota-df' function. So I plan to do something in overlayfs because
>>>>>>>> it's better than doing same things in every low level filesystems.
>>>>>>>> But now, I'm more willing to persuade xfs/ext4 people to midify the
>>>>>>>> detail mechanism of 'quota-df' in specific filesystems.
>>>>>>>> 
>>>>>>>> 'df', hmm... Both solutions could work I think.
>>>>>>>> 
>>>>>>> 
>>>>>>> I don't thing it is the underlying filesystem that implements quota-df
>>>>>>> I think it is 'df' itself. I was also surprised to learn than, but at least when
>>>>>>> you set project quota with xfs_quota via /etc/projects /etc/projid
>>>>>>> df just shows you the project directories as if they were mount points.
>>>>>>> I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
>>>>>>> is no different.
>>>>>> 
>>>>>> That sounds really interesting and that must be special version of df command.
>>>>>> On CentOS I have never seen that. :-(
>>>>> 
>>>>> I donno, maybe I dreamed of seeing it... most likely I am confusing seeing
>>>>> the correct df usage on overlayfs mount with upper that has project quota.
>>>>> 
>>>>>> In any case if you take a quick look at below functions in the code, you will probably
>>>>>> believe what I said before. If you stop calling those functions in the kernel code,
>>>>>> then I guess all magic will be gone and never turn back again. :-)
>>>>>> 
>>>>>> xfs:
>>>>>> xfs_qm_statvfs
>>>>>> 
>>>>>> ext4:
>>>>>> ext4_statfs_project
>>>>>> 
>>>>>> f2fs:
>>>>>> f2fs_statfs_project
>>>>> =【--07
>>>>>> 
>>>>> 
>>>>> I see. so I'll wait for your RFC patch to see what ovl_statfs_project brings
>>>>> to the table.
>>>> 
>>>> Seems there is nothing more to do unless we add more features like 'reservation' we discussed above.
>>>> In this case I think we should consider adding 'reservation amount' to bfree, and bavail represents
>>>> the real free space amount that can be utilized by new files.
>>> 
>>> 
>>> Most of time underlying filesystem’s quota-df works well, but when real filesystem’s avail is lower than
>>> project quota’s avail then the result is quite confusing. I’ve only tested on xfs but I think ext4 is
>>> similar because they have same quota-df logic.
>>> 
>>> For example, if we have 100GB xfs filesystem(/mnt/test2) and we have
>>> 3 directories(pq1, pq2, pq3) inside it, each directory sets project quota.
>>> (block hard limit up to 10GB)
>>> 
>>> When avail space of real filesystem is only left 3.2MB, but when running df for
>>> pg1,pg2,pg3 then avail space is 9.5GB, this is much more than real filesystem.
>>> 
>>> 
>>> Detail output:
>>> 
>>> $ df -h /mnt/test2
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/vdb2       100G  100G  3.2M 100% /mnt/test2
>>> 
>>> $ df -h /mnt/test2/pq1
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>> 
>>> $ df -h /mnt/test2/pq2
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>> 
>>> $ df -h /mnt/test2/pq3
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>> 
>>> 
>>> So I just think if we can adjust size/used/avail in overlayfs layer like below,
>>> it maybe a little bit more helpful for our users. What do you think for this?
>>> 
>>> 
>>> $ df -h /mnt/test2
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/vdb2       100G  100G  3.2M 100% /mnt/test2
>>> 
>>> $ df -h /mnt/test2/pq1
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>> 
>>> $ df -h /mnt/test2/pq2
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>> 
>>> $ df -h /mnt/test2/pq3
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>> 
>>> 
> 
> OOPS sent by mistake.
> 
> Chengguang,
> 
> People on this list may have not been following the similar thread on xfs list
> and will wonder why the heck total size 574M
> 
> You did not explain that is is a limitation of the current statfs() API and how
> df presents the information.

Sorry for lack of detail information. Because statfs(2) only collects size/avail
information for df and calculate used by size - avail. So here we have to adjust
flowing size to meet the df mechanism.


> 
> My opinion is that at least for overlayfs use case, it is better to change 'df'
> then to work around its current limitation and present another missleading
> type of information.
> 
> It was the same story with btrfs. 'df' was just to simple to meet the demands
> of reporting to user the real status of btrfs disk space, so btrfs tools needed
> to provide a better 'df'.
> 
> IMO, the best course of action is to integrate 'df' with quota tools
> information,
> so that 'df' has more information and can display more accurate data.
> 
> Overlayfs users are used to work in a container environment where not all
> utilities work out of the box and may need to use alternative tools or hacks
> inside the container, so it won't be a bug hurdle, even if this needs a special
> flavor of 'df’.

All right, I get your point and I agree with your proposal.
Only one thing, I really hope put real avail info into quota-tool instead of df. :)

Thanks,
Chengguang.

Re: "quota-df" feature

[Amir Goldstein]

On Sun, Apr 1, 2018 at 10:49 AM,  <cgxu519@gmx.com> wrote:
> 在 2018年4月1日，下午3:00，Amir Goldstein <amir73il@gmail.com> 写道：
>>
>> On Sun, Apr 1, 2018 at 9:53 AM, Amir Goldstein <amir73il@gmail.com> wrote:
>>> On Sun, Apr 1, 2018 at 9:35 AM,  <cgxu519@gmx.com> wrote:
>>>> 在 2018年3月10日，下午2:37，Chengguang Xu <cgxu519@gmx.com> 写道：
>>>>>
>>>>>> Sent: Thursday, March 08, 2018 at 10:36 PM
>>>>>> From: "Amir Goldstein" <amir73il@gmail.com>
>>>>>> To: "Chengguang Xu" <cgxu519@gmx.com>
>>>>>> Cc: overlayfs <linux-unionfs@vger.kernel.org>, "Miklos Szeredi" <miklos@szeredi.hu>
>>>>>> Subject: Re: "quota-df" feature
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@gmx.com> wrote:
>>>>>> [...]
>>>>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting
>>>>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated
>>>>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve
>>>>>>>>>>> "covered" problem that you mentioned above. Counting usage information without
>>>>>>>>>>> quota might be a hard work when having plenty of files.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Well, its a different use case, but a very well known use case -
>>>>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same
>>>>>>>>>> blocks, but each clone is fully accounted to the user/group/project.
>>>>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files
>>>>>>>>>> that the user can reference, but the host does not pay the cost of sum of all
>>>>>>>>>> user quotas.
>>>>>>>>>>
>>>>>>>>>> Without accounting of "covered" files to begin with, is it possible to
>>>>>>>>>> get to a state
>>>>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that
>>>>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin
>>>>>>>>>> provisioned block,
>>>>>>>>>> but a situation that filesystems and administrators try really hard to avoid.
>>>>>>>>>
>>>>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir
>>>>>>>>> because of same named file exists in upperdir? or is it meaning the contents in
>>>>>>>>> lowerdir but in the merged dir of overlayfs?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> The former. "covered" file size does not show up in du -s, so the
>>>>>>>> merged disk usage
>>>>>>>> is <upper used> + <lower used> - <covered used>
>>>>>>>
>>>>>>> Thanks, got it.
>>>>>>> But IIUC, "covered" file does not have chance to copy-up, so
>>>>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error?
>>>>>>
>>>>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB
>>>>>> and you want to allow user to touch all the files in the image and
>>>>>> create 1GB of new files.
>>>>>>
>>>>>> If your only tool is project quota on upper then you need to set project
>>>>>> quota hard limit to 2GB, but then user can create 2GB of new files and
>>>>>> later when touching a lower file, will get EQUOTA on copy up.
>>>>>>
>>>>>> If you account lower uncovered files to overlay merged quota then
>>>>>> you set the merged quota to 2GB and start with 50% used.
>>>>>> - Copy up will not change used
>>>>>> - Remove of lower will reduce used
>>>>>> - You can never get EQUOTA from touching a file
>>>>>
>>>>> I get your point, I'd like to think it as "reservation" feature we can implement
>>>>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up.
>>>>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports
>>>>> block reservation? Are there many people hope to have this kind of function?
>>>>>
>>>>>
>>>>>>>>>> All I am saying is that it is "not hard" (TM) to keep track of
>>>>>>>>>> "covered" files disk usage
>>>>>>>>>> and "not hard" to re-calculate "covered" files disk usage when full
>>>>>>>>>> indexing is enabled.
>>>>>>>>>>> I don't know exactly what will happen when combining index and nfs_export options,
>>>>>>>>>>> I need to read and understand related code later.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> nfs_export REQUIRES index and implies indexing of all files on copy up, not
>>>>>>>>>> only lower hardlinks.
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here are some test examples share with you (based on ext4):
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) project quota enabled && without hard-limit
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>>>>>> overlay          92G  201M   91G   1% /mnt/test3/df/merged
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>>>>>> overlay          6.3M     8  6.3M    1% /mnt/test3/df/merged
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) project quota enabled && with hard-limit
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ df -h /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>>>>>>> /dev/vdb3        99G  2.5G   91G   3% /mnt/test3
>>>>>>>>>>>>> overlay         1.0G  201M  824M  20% /mnt/test3/df/merged
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ df -hi /mnt/test3 /mnt/test3/df/merged
>>>>>>>>>>>>> Filesystem     Inodes IUsed IFree IUse% Mounted on
>>>>>>>>>>>>> /dev/vdb3        6.3M  2.4K  6.3M    1% /mnt/test3
>>>>>>>>>>>>> overlay          1000     8   992    1% /mnt/test3/df/merged
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I can't follow from the example below what is the expected result and why
>>>>>>>>>>>> if you add the quota setup commands that could be useful.
>>>>>>>>>>>
>>>>>>>>>>> Underlying fs using ext4 and mkfs/mount with project quota option and
>>>>>>>>>>> all needed directories(lowerdir, upperdir, workdir, merged) are set same
>>>>>>>>>>> project quota. Current quota-df implementation only adjust the couting
>>>>>>>>>>> information when having upperdir.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Really? why is lowerdir on the same project id? For containers quota
>>>>>>>>>> only upper/work should be on the same project id. lowerdir should belong
>>>>>>>>>> to a shared project or no project at all.
>>>>>>>>>> In current docker implementation, every overlay2 driver root dir
>>>>>>>>>> is assigned a different project id, but lowerdir are symlinks to another
>>>>>>>>>> image root dir.
>>>>>>>>>
>>>>>>>>> For the setting of docker, you are completely right. My description of tesing
>>>>>>>>> environment in previous email is only for simple kernel testing and explaining
>>>>>>>>> the condition of testing result above, not specific to docker.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> In case 1:
>>>>>>>>>>> There is no hardlimit/softlimit, so the expected result as below.
>>>>>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>>>>>      include upperdir && workdir.
>>>>>>>>>>> Avail: The avail of underlying fs
>>>>>>>>>>> Total: Used + Avail
>>>>>>>>>>>
>>>>>>>>>>> #upperdir used 201M and /mnt/test3 used 2.5G
>>>>>>>>>>>
>>>>>>>>>>> In case 2:
>>>>>>>>>>> Project quota hardlimits are block count = 1G, inode count = 1000.
>>>>>>>>>>> So the expected result as below.
>>>>>>>>>>>
>>>>>>>>>>> Used:  The used count in project quota which set to the directory
>>>>>>>>>>>      include upperdir && workdir.
>>>>>>>>>>> Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b)
>>>>>>>>>>> Total: Used + Avail
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Are those "expected" results inline with what df shows on project quota
>>>>>>>>>> directories without overlayfs? If not, and the new behavior makes sense,
>>>>>>>>>> why change overlayfs and not change 'df'?
>>>>>>>>>
>>>>>>>>> No, those results are based on my change of overlayfs. At the first,
>>>>>>>>> I didn't notice that underlying filesystems had already implemented
>>>>>>>>> 'quota-df' function. So I plan to do something in overlayfs because
>>>>>>>>> it's better than doing same things in every low level filesystems.
>>>>>>>>> But now, I'm more willing to persuade xfs/ext4 people to midify the
>>>>>>>>> detail mechanism of 'quota-df' in specific filesystems.
>>>>>>>>>
>>>>>>>>> 'df', hmm... Both solutions could work I think.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't thing it is the underlying filesystem that implements quota-df
>>>>>>>> I think it is 'df' itself. I was also surprised to learn than, but at least when
>>>>>>>> you set project quota with xfs_quota via /etc/projects /etc/projid
>>>>>>>> df just shows you the project directories as if they were mount points.
>>>>>>>> I tested with xfs on Ubuntu, but suppose with ext4 and other distro it
>>>>>>>> is no different.
>>>>>>>
>>>>>>> That sounds really interesting and that must be special version of df command.
>>>>>>> On CentOS I have never seen that. :-(
>>>>>>
>>>>>> I donno, maybe I dreamed of seeing it... most likely I am confusing seeing
>>>>>> the correct df usage on overlayfs mount with upper that has project quota.
>>>>>>
>>>>>>> In any case if you take a quick look at below functions in the code, you will probably
>>>>>>> believe what I said before. If you stop calling those functions in the kernel code,
>>>>>>> then I guess all magic will be gone and never turn back again. :-)
>>>>>>>
>>>>>>> xfs:
>>>>>>> xfs_qm_statvfs
>>>>>>>
>>>>>>> ext4:
>>>>>>> ext4_statfs_project
>>>>>>>
>>>>>>> f2fs:
>>>>>>> f2fs_statfs_project
>>>>>> =【--07
>>>>>>>
>>>>>>
>>>>>> I see. so I'll wait for your RFC patch to see what ovl_statfs_project brings
>>>>>> to the table.
>>>>>
>>>>> Seems there is nothing more to do unless we add more features like 'reservation' we discussed above.
>>>>> In this case I think we should consider adding 'reservation amount' to bfree, and bavail represents
>>>>> the real free space amount that can be utilized by new files.
>>>>
>>>>
>>>> Most of time underlying filesystem’s quota-df works well, but when real filesystem’s avail is lower than
>>>> project quota’s avail then the result is quite confusing. I’ve only tested on xfs but I think ext4 is
>>>> similar because they have same quota-df logic.
>>>>
>>>> For example, if we have 100GB xfs filesystem(/mnt/test2) and we have
>>>> 3 directories(pq1, pq2, pq3) inside it, each directory sets project quota.
>>>> (block hard limit up to 10GB)
>>>>
>>>> When avail space of real filesystem is only left 3.2MB, but when running df for
>>>> pg1,pg2,pg3 then avail space is 9.5GB, this is much more than real filesystem.
>>>>
>>>>
>>>> Detail output:
>>>>
>>>> $ df -h /mnt/test2
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/vdb2       100G  100G  3.2M 100% /mnt/test2
>>>>
>>>> $ df -h /mnt/test2/pq1
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>>>
>>>> $ df -h /mnt/test2/pq2
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>>>
>>>> $ df -h /mnt/test2/pq3
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/vdb2        10G  570M  9.5G   6% /mnt/test2
>>>>
>>>>
>>>> So I just think if we can adjust size/used/avail in overlayfs layer like below,
>>>> it maybe a little bit more helpful for our users. What do you think for this?
>>>>
>>>>
>>>> $ df -h /mnt/test2
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/vdb2       100G  100G  3.2M 100% /mnt/test2
>>>>
>>>> $ df -h /mnt/test2/pq1
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>>>
>>>> $ df -h /mnt/test2/pq2
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>>>
>>>> $ df -h /mnt/test2/pq3
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/vdb2       574M  570M  3.2M 100% /mnt/test2
>>>>
>>>>
>>
>> OOPS sent by mistake.
>>
>> Chengguang,
>>
>> People on this list may have not been following the similar thread on xfs list
>> and will wonder why the heck total size 574M
>>
>> You did not explain that is is a limitation of the current statfs() API and how
>> df presents the information.
>
> Sorry for lack of detail information. Because statfs(2) only collects size/avail
> information for df and calculate used by size - avail. So here we have to adjust
> flowing size to meet the df mechanism.
>
>
>>
>> My opinion is that at least for overlayfs use case, it is better to change 'df'
>> then to work around its current limitation and present another missleading
>> type of information.
>>
>> It was the same story with btrfs. 'df' was just to simple to meet the demands
>> of reporting to user the real status of btrfs disk space, so btrfs tools needed
>> to provide a better 'df'.
>>
>> IMO, the best course of action is to integrate 'df' with quota tools
>> information,
>> so that 'df' has more information and can display more accurate data.
>>
>> Overlayfs users are used to work in a container environment where not all
>> utilities work out of the box and may need to use alternative tools or hacks
>> inside the container, so it won't be a bug hurdle, even if this needs a special
>> flavor of 'df’.
>
> All right, I get your point and I agree with your proposal.
> Only one thing, I really hope put real avail info into quota-tool instead of df. :)
>

Of course. Makes sense.

Thanks,
Amir.