btrfs quota issues

btrfs quota issues

[Rakesh Sankeshi]

I set 200GB limit to one user and 100GB to another user.

as soon as I reached 139GB and 53GB each, hitting the quota errors.
anyway to workaround quota functionality on btrfs LZO compressed
filesystem?



4.7.0-040700-generic #201608021801 SMP

btrfs-progs v4.7


Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8

Total devices 2 FS bytes used 150.62GiB

devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc

devid    2 size 1.00TiB used 78.01GiB path /dev/xvde


Data, RAID0: total=150.00GiB, used=149.12GiB

System, RAID1: total=8.00MiB, used=16.00KiB

Metadata, RAID1: total=3.00GiB, used=1.49GiB

GlobalReserve, single: total=512.00MiB, used=0.00B


Filesystem      Size  Used Avail Use% Mounted on

/dev/xvdc       2.0T  153G  1.9T   8% /test_lzo

Re: btrfs quota issues

[Duncan]

Rakesh Sankeshi posted on Thu, 11 Aug 2016 10:32:03 -0700 as excerpted:

> I set 200GB limit to one user and 100GB to another user.
> 
> as soon as I reached 139GB and 53GB each, hitting the quota errors.
> anyway to workaround quota functionality on btrfs LZO compressed
> filesystem?

The btrfs quota subsystem remains somewhat buggy and unstable.  A lot of 
work has gone into it to fix the problems, including rewrites of the 
entire subsystem, and it's much better than it used to be, but it's still 
a feature that I would recommend not using on btrfs.

My general position is this.  Either you need quotas for your use-case or 
you don't.  If you truly need them, you're far better off using a more 
mature filesystem with proven quota subsystem reliability.  If you don't 
really need them, simply keep the feature off for now, and for however 
long it takes to stabilize the feature, which could be some time.

Of course if you're specifically testing quotas in ordered to report 
issues and test bugfixes, that's a specific case of needing quota 
functionality, and your work is greatly appreciated as it'll help to 
eventually make that feature stable and workable for all. =:^)



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs quota issues

[Rakesh Sankeshi]

Thanks for your inputs.

Another question I had was, is there any way to check what's the
directory/file sizes prior to compression and how much copression
btrfs did, etc? Basicaly some stats around compression and/or dedupe
from btrfs.


On Thu, Aug 11, 2016 at 12:13 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Rakesh Sankeshi posted on Thu, 11 Aug 2016 10:32:03 -0700 as excerpted:
>
>> I set 200GB limit to one user and 100GB to another user.
>>
>> as soon as I reached 139GB and 53GB each, hitting the quota errors.
>> anyway to workaround quota functionality on btrfs LZO compressed
>> filesystem?
>
> The btrfs quota subsystem remains somewhat buggy and unstable.  A lot of
> work has gone into it to fix the problems, including rewrites of the
> entire subsystem, and it's much better than it used to be, but it's still
> a feature that I would recommend not using on btrfs.
>
> My general position is this.  Either you need quotas for your use-case or
> you don't.  If you truly need them, you're far better off using a more
> mature filesystem with proven quota subsystem reliability.  If you don't
> really need them, simply keep the feature off for now, and for however
> long it takes to stabilize the feature, which could be some time.
>
> Of course if you're specifically testing quotas in ordered to report
> issues and test bugfixes, that's a specific case of needing quota
> functionality, and your work is greatly appreciated as it'll help to
> eventually make that feature stable and workable for all. =:^)
>
>
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs quota issues

[Duncan]

Rakesh Sankeshi posted on Fri, 12 Aug 2016 08:47:13 -0700 as excerpted:

> Another question I had was, is there any way to check what's the
> directory/file sizes prior to compression and how much copression btrfs
> did, etc? Basicaly some stats around compression and/or dedupe from
> btrfs.

There's some dedupe reportes I've seen posted, that basically show shared 
vs. unique extents, but that's out of both my area of usage /and/ my area 
of interest, so that's pretty much all I can say on that.

Compression...  AFAIK there's no nice neat admin-level command for it 
(yet?), but it's possible to get the raw information via filefrag -v.  
The output is a list of extents for the file in question, with both their 
(uncompressed) size and start/end addresses.  If the size is greater than 
the space they take up based on their start/end addresses, then obviously 
that extent is compressed.  People have posted python scripts and the 
like that process that information into something higher level that an 
admin could digest, but I believe you'll have to mine the list archives 
to find them.

In the context of filefrag, it is worth mentioning, however, that btrfs 
compression blocks are 128 KiB in (uncompressed) size, and that filefrag 
considers each such reported block its own extent, even where it's 
contiguous with the next one.  Put another way, filefrag doesn't 
understand btrfs compression, tho it's possible to use its output to 
figure out whether a file is compressed or not, and by how much.

It's also relatively easy to quickly scan the filefrag output to see if 
all the extents are 128 KiB in size and consider that compressed, and/or 
to divide the file size by 128 KiB and see if that matches the number of 
reported extents, considering it compressed if so.  For a few one-off 
files, that's easy enough to do manually, but you'd definitely want to 
automate the process if you wanted that information on more than a few 
individual files.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs quota issues

[Qu Wenruo]

At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote:
> I set 200GB limit to one user and 100GB to another user.
>
> as soon as I reached 139GB and 53GB each, hitting the quota errors.
> anyway to workaround quota functionality on btrfs LZO compressed
> filesystem?
>

Please paste "btrfs qgroup show -prce <mnt>" output if you are using 
btrfs qgroup/quota function.

And, AFAIK btrfs qgroup is applied to subvolume, not user.

So did you mean limit it to one subvolume belongs to one user?

Thanks,
Qu

>
>
> 4.7.0-040700-generic #201608021801 SMP
>
> btrfs-progs v4.7
>
>
> Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8
>
> Total devices 2 FS bytes used 150.62GiB
>
> devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc
>
> devid    2 size 1.00TiB used 78.01GiB path /dev/xvde
>
>
> Data, RAID0: total=150.00GiB, used=149.12GiB
>
> System, RAID1: total=8.00MiB, used=16.00KiB
>
> Metadata, RAID1: total=3.00GiB, used=1.49GiB
>
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
>
> Filesystem      Size  Used Avail Use% Mounted on
>
> /dev/xvdc       2.0T  153G  1.9T   8% /test_lzo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: btrfs quota issues

[Rakesh Sankeshi]

yes, subvol level.

qgroupid         rfer         excl     max_rfer     max_excl parent  child

--------         ----         ----     --------     -------- ------  -----

0/5          16.00KiB     16.00KiB         none         none ---     ---

0/258       119.48GiB    119.48GiB    200.00GiB         none ---     ---

0/259        92.57GiB     92.57GiB    200.00GiB         none ---     ---


although I have 200GB limit on 2 subvols, running into issue at about
120 and 92GB itself


On Sun, Aug 14, 2016 at 7:11 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote:
>>
>> I set 200GB limit to one user and 100GB to another user.
>>
>> as soon as I reached 139GB and 53GB each, hitting the quota errors.
>> anyway to workaround quota functionality on btrfs LZO compressed
>> filesystem?
>>
>
> Please paste "btrfs qgroup show -prce <mnt>" output if you are using btrfs
> qgroup/quota function.
>
> And, AFAIK btrfs qgroup is applied to subvolume, not user.
>
> So did you mean limit it to one subvolume belongs to one user?
>
> Thanks,
> Qu
>
>>
>>
>> 4.7.0-040700-generic #201608021801 SMP
>>
>> btrfs-progs v4.7
>>
>>
>> Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8
>>
>> Total devices 2 FS bytes used 150.62GiB
>>
>> devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc
>>
>> devid    2 size 1.00TiB used 78.01GiB path /dev/xvde
>>
>>
>> Data, RAID0: total=150.00GiB, used=149.12GiB
>>
>> System, RAID1: total=8.00MiB, used=16.00KiB
>>
>> Metadata, RAID1: total=3.00GiB, used=1.49GiB
>>
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>>
>> Filesystem      Size  Used Avail Use% Mounted on
>>
>> /dev/xvdc       2.0T  153G  1.9T   8% /test_lzo
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
>

Re: btrfs quota issues

[Qu Wenruo]

At 08/16/2016 03:11 AM, Rakesh Sankeshi wrote:
> yes, subvol level.
>
> qgroupid         rfer         excl     max_rfer     max_excl parent  child
>
> --------         ----         ----     --------     -------- ------  -----
>
> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>
> 0/258       119.48GiB    119.48GiB    200.00GiB         none ---     ---
>
> 0/259        92.57GiB     92.57GiB    200.00GiB         none ---     ---
>
>
> although I have 200GB limit on 2 subvols, running into issue at about
> 120 and 92GB itself

1) About workload
Would you mind to mention the work pattern of your write?

Just dd data with LZO compression?
For compression part, it's a little complicated, as the reserved data 
size and on disk extent size are different.

It's possible that at some code we leaked some reserved data space.


2) Behavior after EDQUOT
And, after EDQUOT happens, can you write data into the subvolume?
If you can still write a lot of data (at least several giga), it seems 
to be something related with temporary reserved space.

If not, and even can't remove any file due to EQUOTA, then it's almost 
sure we have underflowed the reserved data.
In that case, unmount and mount again will be the only workaround.
(In fact, not workaround at all)

3) Behavior without compression

If it's OK for you, would you mind to test it without compression?
Currently we mostly use the assumption that on-disk extent size are the 
same with in-memory extent size (non-compression).

So qgroup + compression is not the main concern before and is buggy.

If without compression, qgroup works sanely, at least we can be sure 
that the cause is qgroup + compression.

Thanks,
Qu

>
>
> On Sun, Aug 14, 2016 at 7:11 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>>
>> At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote:
>>>
>>> I set 200GB limit to one user and 100GB to another user.
>>>
>>> as soon as I reached 139GB and 53GB each, hitting the quota errors.
>>> anyway to workaround quota functionality on btrfs LZO compressed
>>> filesystem?
>>>
>>
>> Please paste "btrfs qgroup show -prce <mnt>" output if you are using btrfs
>> qgroup/quota function.
>>
>> And, AFAIK btrfs qgroup is applied to subvolume, not user.
>>
>> So did you mean limit it to one subvolume belongs to one user?
>>
>> Thanks,
>> Qu
>>
>>>
>>>
>>> 4.7.0-040700-generic #201608021801 SMP
>>>
>>> btrfs-progs v4.7
>>>
>>>
>>> Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8
>>>
>>> Total devices 2 FS bytes used 150.62GiB
>>>
>>> devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc
>>>
>>> devid    2 size 1.00TiB used 78.01GiB path /dev/xvde
>>>
>>>
>>> Data, RAID0: total=150.00GiB, used=149.12GiB
>>>
>>> System, RAID1: total=8.00MiB, used=16.00KiB
>>>
>>> Metadata, RAID1: total=3.00GiB, used=1.49GiB
>>>
>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>
>>>
>>> Filesystem      Size  Used Avail Use% Mounted on
>>>
>>> /dev/xvdc       2.0T  153G  1.9T   8% /test_lzo
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>>
>
>

Re: btrfs quota issues

[Rakesh Sankeshi]

2) after EDQUOT, can't write anymore.

I can delete the data, but still can't write further

3) tested it without compression and also with LZO and ZLIB.. all
behave same way with qgroup. no consistency on when it hits the quota
limit and don't understand on how it's calculating the numbers.

In case of ext4 and xfs, I can see visually that it's hitting the quota limit.



On Mon, Aug 15, 2016 at 6:01 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> At 08/16/2016 03:11 AM, Rakesh Sankeshi wrote:
>>
>> yes, subvol level.
>>
>> qgroupid         rfer         excl     max_rfer     max_excl parent  child
>>
>> --------         ----         ----     --------     -------- ------  -----
>>
>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>>
>> 0/258       119.48GiB    119.48GiB    200.00GiB         none ---     ---
>>
>> 0/259        92.57GiB     92.57GiB    200.00GiB         none ---     ---
>>
>>
>> although I have 200GB limit on 2 subvols, running into issue at about
>> 120 and 92GB itself
>
>
> 1) About workload
> Would you mind to mention the work pattern of your write?
>
> Just dd data with LZO compression?
> For compression part, it's a little complicated, as the reserved data size
> and on disk extent size are different.
>
> It's possible that at some code we leaked some reserved data space.
>
>
> 2) Behavior after EDQUOT
> And, after EDQUOT happens, can you write data into the subvolume?
> If you can still write a lot of data (at least several giga), it seems to be
> something related with temporary reserved space.
>
> If not, and even can't remove any file due to EQUOTA, then it's almost sure
> we have underflowed the reserved data.
> In that case, unmount and mount again will be the only workaround.
> (In fact, not workaround at all)
>
> 3) Behavior without compression
>
> If it's OK for you, would you mind to test it without compression?
> Currently we mostly use the assumption that on-disk extent size are the same
> with in-memory extent size (non-compression).
>
> So qgroup + compression is not the main concern before and is buggy.
>
> If without compression, qgroup works sanely, at least we can be sure that
> the cause is qgroup + compression.
>
> Thanks,
> Qu
>
>
>>
>>
>> On Sun, Aug 14, 2016 at 7:11 PM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>> wrote:
>>>
>>>
>>>
>>> At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote:
>>>>
>>>>
>>>> I set 200GB limit to one user and 100GB to another user.
>>>>
>>>> as soon as I reached 139GB and 53GB each, hitting the quota errors.
>>>> anyway to workaround quota functionality on btrfs LZO compressed
>>>> filesystem?
>>>>
>>>
>>> Please paste "btrfs qgroup show -prce <mnt>" output if you are using
>>> btrfs
>>> qgroup/quota function.
>>>
>>> And, AFAIK btrfs qgroup is applied to subvolume, not user.
>>>
>>> So did you mean limit it to one subvolume belongs to one user?
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>>
>>>> 4.7.0-040700-generic #201608021801 SMP
>>>>
>>>> btrfs-progs v4.7
>>>>
>>>>
>>>> Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8
>>>>
>>>> Total devices 2 FS bytes used 150.62GiB
>>>>
>>>> devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc
>>>>
>>>> devid    2 size 1.00TiB used 78.01GiB path /dev/xvde
>>>>
>>>>
>>>> Data, RAID0: total=150.00GiB, used=149.12GiB
>>>>
>>>> System, RAID1: total=8.00MiB, used=16.00KiB
>>>>
>>>> Metadata, RAID1: total=3.00GiB, used=1.49GiB
>>>>
>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>
>>>>
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>
>>>> /dev/xvdc       2.0T  153G  1.9T   8% /test_lzo
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>> in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Re: btrfs quota issues

[Rakesh Sankeshi]

also is there any timeframe on when the qgroup / quota issues would be
stabilized in btrfs?

Thanks!


On Tue, Aug 16, 2016 at 9:05 AM, Rakesh Sankeshi
<rakesh.sankeshi@gmail.com> wrote:
> 2) after EDQUOT, can't write anymore.
>
> I can delete the data, but still can't write further
>
> 3) tested it without compression and also with LZO and ZLIB.. all
> behave same way with qgroup. no consistency on when it hits the quota
> limit and don't understand on how it's calculating the numbers.
>
> In case of ext4 and xfs, I can see visually that it's hitting the quota limit.
>
>
>
> On Mon, Aug 15, 2016 at 6:01 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>>
>> At 08/16/2016 03:11 AM, Rakesh Sankeshi wrote:
>>>
>>> yes, subvol level.
>>>
>>> qgroupid         rfer         excl     max_rfer     max_excl parent  child
>>>
>>> --------         ----         ----     --------     -------- ------  -----
>>>
>>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>>>
>>> 0/258       119.48GiB    119.48GiB    200.00GiB         none ---     ---
>>>
>>> 0/259        92.57GiB     92.57GiB    200.00GiB         none ---     ---
>>>
>>>
>>> although I have 200GB limit on 2 subvols, running into issue at about
>>> 120 and 92GB itself
>>
>>
>> 1) About workload
>> Would you mind to mention the work pattern of your write?
>>
>> Just dd data with LZO compression?
>> For compression part, it's a little complicated, as the reserved data size
>> and on disk extent size are different.
>>
>> It's possible that at some code we leaked some reserved data space.
>>
>>
>> 2) Behavior after EDQUOT
>> And, after EDQUOT happens, can you write data into the subvolume?
>> If you can still write a lot of data (at least several giga), it seems to be
>> something related with temporary reserved space.
>>
>> If not, and even can't remove any file due to EQUOTA, then it's almost sure
>> we have underflowed the reserved data.
>> In that case, unmount and mount again will be the only workaround.
>> (In fact, not workaround at all)
>>
>> 3) Behavior without compression
>>
>> If it's OK for you, would you mind to test it without compression?
>> Currently we mostly use the assumption that on-disk extent size are the same
>> with in-memory extent size (non-compression).
>>
>> So qgroup + compression is not the main concern before and is buggy.
>>
>> If without compression, qgroup works sanely, at least we can be sure that
>> the cause is qgroup + compression.
>>
>> Thanks,
>> Qu
>>
>>
>>>
>>>
>>> On Sun, Aug 14, 2016 at 7:11 PM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> wrote:
>>>>
>>>>
>>>>
>>>> At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote:
>>>>>
>>>>>
>>>>> I set 200GB limit to one user and 100GB to another user.
>>>>>
>>>>> as soon as I reached 139GB and 53GB each, hitting the quota errors.
>>>>> anyway to workaround quota functionality on btrfs LZO compressed
>>>>> filesystem?
>>>>>
>>>>
>>>> Please paste "btrfs qgroup show -prce <mnt>" output if you are using
>>>> btrfs
>>>> qgroup/quota function.
>>>>
>>>> And, AFAIK btrfs qgroup is applied to subvolume, not user.
>>>>
>>>> So did you mean limit it to one subvolume belongs to one user?
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>
>>>>>
>>>>> 4.7.0-040700-generic #201608021801 SMP
>>>>>
>>>>> btrfs-progs v4.7
>>>>>
>>>>>
>>>>> Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8
>>>>>
>>>>> Total devices 2 FS bytes used 150.62GiB
>>>>>
>>>>> devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc
>>>>>
>>>>> devid    2 size 1.00TiB used 78.01GiB path /dev/xvde
>>>>>
>>>>>
>>>>> Data, RAID0: total=150.00GiB, used=149.12GiB
>>>>>
>>>>> System, RAID1: total=8.00MiB, used=16.00KiB
>>>>>
>>>>> Metadata, RAID1: total=3.00GiB, used=1.49GiB
>>>>>
>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>>
>>>>>
>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>
>>>>> /dev/xvdc       2.0T  153G  1.9T   8% /test_lzo
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>> in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>

Re: btrfs quota issues

[Tim Walberg]

On 08/16/2016 16:33 -0700, Rakesh Sankeshi wrote:
>>	also is there any timeframe on when the qgroup / quota issues would be
>>	stabilized in btrfs?
>>	
>>	Thanks!

This may or may not be of interest to you, but for the record, since at least
linux 4.2, I've had pretty good luck with what I'd loosely call "non-enforcing
quotas" with btrfs - i.e. qgroups enabled so that you can actually track usage,
but no limits set, so none of the "deny allocation" logic ever gets hit. It's
understandably not the desired end-state - I still have to use daily reports
and manual enforcement to keep things in balance, but it's better than not
being able to use them at all, especially with the other benefits btrfs brings
to the table. If you really need the enforcement, I'm afraid a different FS
is the best option right now...



-- 
twalberg@gmail.com

Re: btrfs quota issues

[Qu Wenruo]

At 08/17/2016 12:05 AM, Rakesh Sankeshi wrote:
> 2) after EDQUOT, can't write anymore.
>
> I can delete the data, but still can't write further

So it's a underflow now.

>
> 3) tested it without compression and also with LZO and ZLIB.. all
> behave same way with qgroup. no consistency on when it hits the quota
> limit and don't understand on how it's calculating the numbers.

Even without compression?!

That's a really big problem then.
Workload please, it's an urgent bug now.

It's better to provide the scripts to reproduce it.



And for the meaning of the numbers, for rfer(reference) it means the 
size of all extents the subvolume has referred to, including both data 
and metadata.

For excl(exclusive), it means the size of all extents that only belongs 
to the subvolume.

And since it's all about size of extents(on-disk), for compression case, 
it's the size after compression.

Also, if one subvolume only referred to part of an extent, the whole 
extent size will be accounted.


Last but not least.
Considering there is quite a lot of report about hitting ENOSPC while 
there is still a lot of unallocated space,
is it reporting error message like "No space left on device" (ENOSPC) or 
"Quota exceeded"(EDQUOT)?

Thanks,
Qu

>
> In case of ext4 and xfs, I can see visually that it's hitting the quota limit.
>
>
>
> On Mon, Aug 15, 2016 at 6:01 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>>
>> At 08/16/2016 03:11 AM, Rakesh Sankeshi wrote:
>>>
>>> yes, subvol level.
>>>
>>> qgroupid         rfer         excl     max_rfer     max_excl parent  child
>>>
>>> --------         ----         ----     --------     -------- ------  -----
>>>
>>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>>>
>>> 0/258       119.48GiB    119.48GiB    200.00GiB         none ---     ---
>>>
>>> 0/259        92.57GiB     92.57GiB    200.00GiB         none ---     ---
>>>
>>>
>>> although I have 200GB limit on 2 subvols, running into issue at about
>>> 120 and 92GB itself
>>
>>
>> 1) About workload
>> Would you mind to mention the work pattern of your write?
>>
>> Just dd data with LZO compression?
>> For compression part, it's a little complicated, as the reserved data size
>> and on disk extent size are different.
>>
>> It's possible that at some code we leaked some reserved data space.
>>
>>
>> 2) Behavior after EDQUOT
>> And, after EDQUOT happens, can you write data into the subvolume?
>> If you can still write a lot of data (at least several giga), it seems to be
>> something related with temporary reserved space.
>>
>> If not, and even can't remove any file due to EQUOTA, then it's almost sure
>> we have underflowed the reserved data.
>> In that case, unmount and mount again will be the only workaround.
>> (In fact, not workaround at all)
>>
>> 3) Behavior without compression
>>
>> If it's OK for you, would you mind to test it without compression?
>> Currently we mostly use the assumption that on-disk extent size are the same
>> with in-memory extent size (non-compression).
>>
>> So qgroup + compression is not the main concern before and is buggy.
>>
>> If without compression, qgroup works sanely, at least we can be sure that
>> the cause is qgroup + compression.
>>
>> Thanks,
>> Qu
>>
>>
>>>
>>>
>>> On Sun, Aug 14, 2016 at 7:11 PM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> wrote:
>>>>
>>>>
>>>>
>>>> At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote:
>>>>>
>>>>>
>>>>> I set 200GB limit to one user and 100GB to another user.
>>>>>
>>>>> as soon as I reached 139GB and 53GB each, hitting the quota errors.
>>>>> anyway to workaround quota functionality on btrfs LZO compressed
>>>>> filesystem?
>>>>>
>>>>
>>>> Please paste "btrfs qgroup show -prce <mnt>" output if you are using
>>>> btrfs
>>>> qgroup/quota function.
>>>>
>>>> And, AFAIK btrfs qgroup is applied to subvolume, not user.
>>>>
>>>> So did you mean limit it to one subvolume belongs to one user?
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>
>>>>>
>>>>> 4.7.0-040700-generic #201608021801 SMP
>>>>>
>>>>> btrfs-progs v4.7
>>>>>
>>>>>
>>>>> Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8
>>>>>
>>>>> Total devices 2 FS bytes used 150.62GiB
>>>>>
>>>>> devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc
>>>>>
>>>>> devid    2 size 1.00TiB used 78.01GiB path /dev/xvde
>>>>>
>>>>>
>>>>> Data, RAID0: total=150.00GiB, used=149.12GiB
>>>>>
>>>>> System, RAID1: total=8.00MiB, used=16.00KiB
>>>>>
>>>>> Metadata, RAID1: total=3.00GiB, used=1.49GiB
>>>>>
>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>>
>>>>>
>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>
>>>>> /dev/xvdc       2.0T  153G  1.9T   8% /test_lzo
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>> in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Re: btrfs quota issues

[Rakesh Sankeshi]

sorry, was out of the town.

not much load on the system at all.

As we are hitting many issues in production, just using this system
for my test purpose. Built few different filesystems. 1 with LZO
compression, second one with ZLIB and third one without any
compression. All has issues related to quota.

whenever there is an issue, I am getting quota exceeded error (EDQUOT).

Please let me know if you still need more info.



On Tue, Aug 16, 2016 at 5:56 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> At 08/17/2016 12:05 AM, Rakesh Sankeshi wrote:
>>
>> 2) after EDQUOT, can't write anymore.
>>
>> I can delete the data, but still can't write further
>
>
> So it's a underflow now.
>
>>
>> 3) tested it without compression and also with LZO and ZLIB.. all
>> behave same way with qgroup. no consistency on when it hits the quota
>> limit and don't understand on how it's calculating the numbers.
>
>
> Even without compression?!
>
> That's a really big problem then.
> Workload please, it's an urgent bug now.
>
> It's better to provide the scripts to reproduce it.
>
>
>
> And for the meaning of the numbers, for rfer(reference) it means the size of
> all extents the subvolume has referred to, including both data and metadata.
>
> For excl(exclusive), it means the size of all extents that only belongs to
> the subvolume.
>
> And since it's all about size of extents(on-disk), for compression case,
> it's the size after compression.
>
> Also, if one subvolume only referred to part of an extent, the whole extent
> size will be accounted.
>
>
> Last but not least.
> Considering there is quite a lot of report about hitting ENOSPC while there
> is still a lot of unallocated space,
> is it reporting error message like "No space left on device" (ENOSPC) or
> "Quota exceeded"(EDQUOT)?
>
> Thanks,
> Qu
>
>
>>
>> In case of ext4 and xfs, I can see visually that it's hitting the quota
>> limit.
>>
>>
>>
>> On Mon, Aug 15, 2016 at 6:01 PM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>> wrote:
>>>
>>>
>>>
>>> At 08/16/2016 03:11 AM, Rakesh Sankeshi wrote:
>>>>
>>>>
>>>> yes, subvol level.
>>>>
>>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>>> child
>>>>
>>>> --------         ----         ----     --------     -------- ------
>>>> -----
>>>>
>>>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>>>>
>>>> 0/258       119.48GiB    119.48GiB    200.00GiB         none ---     ---
>>>>
>>>> 0/259        92.57GiB     92.57GiB    200.00GiB         none ---     ---
>>>>
>>>>
>>>> although I have 200GB limit on 2 subvols, running into issue at about
>>>> 120 and 92GB itself
>>>
>>>
>>>
>>> 1) About workload
>>> Would you mind to mention the work pattern of your write?
>>>
>>> Just dd data with LZO compression?
>>> For compression part, it's a little complicated, as the reserved data
>>> size
>>> and on disk extent size are different.
>>>
>>> It's possible that at some code we leaked some reserved data space.
>>>
>>>
>>> 2) Behavior after EDQUOT
>>> And, after EDQUOT happens, can you write data into the subvolume?
>>> If you can still write a lot of data (at least several giga), it seems to
>>> be
>>> something related with temporary reserved space.
>>>
>>> If not, and even can't remove any file due to EQUOTA, then it's almost
>>> sure
>>> we have underflowed the reserved data.
>>> In that case, unmount and mount again will be the only workaround.
>>> (In fact, not workaround at all)
>>>
>>> 3) Behavior without compression
>>>
>>> If it's OK for you, would you mind to test it without compression?
>>> Currently we mostly use the assumption that on-disk extent size are the
>>> same
>>> with in-memory extent size (non-compression).
>>>
>>> So qgroup + compression is not the main concern before and is buggy.
>>>
>>> If without compression, qgroup works sanely, at least we can be sure that
>>> the cause is qgroup + compression.
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>>>
>>>>
>>>> On Sun, Aug 14, 2016 at 7:11 PM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> I set 200GB limit to one user and 100GB to another user.
>>>>>>
>>>>>> as soon as I reached 139GB and 53GB each, hitting the quota errors.
>>>>>> anyway to workaround quota functionality on btrfs LZO compressed
>>>>>> filesystem?
>>>>>>
>>>>>
>>>>> Please paste "btrfs qgroup show -prce <mnt>" output if you are using
>>>>> btrfs
>>>>> qgroup/quota function.
>>>>>
>>>>> And, AFAIK btrfs qgroup is applied to subvolume, not user.
>>>>>
>>>>> So did you mean limit it to one subvolume belongs to one user?
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>>
>>>>>>
>>>>>> 4.7.0-040700-generic #201608021801 SMP
>>>>>>
>>>>>> btrfs-progs v4.7
>>>>>>
>>>>>>
>>>>>> Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8
>>>>>>
>>>>>> Total devices 2 FS bytes used 150.62GiB
>>>>>>
>>>>>> devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc
>>>>>>
>>>>>> devid    2 size 1.00TiB used 78.01GiB path /dev/xvde
>>>>>>
>>>>>>
>>>>>> Data, RAID0: total=150.00GiB, used=149.12GiB
>>>>>>
>>>>>> System, RAID1: total=8.00MiB, used=16.00KiB
>>>>>>
>>>>>> Metadata, RAID1: total=3.00GiB, used=1.49GiB
>>>>>>
>>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>>>
>>>>>>
>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>
>>>>>> /dev/xvdc       2.0T  153G  1.9T   8% /test_lzo
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>>> in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Re: btrfs quota issues

[Qu Wenruo]

At 08/24/2016 02:38 AM, Rakesh Sankeshi wrote:
> sorry, was out of the town.
>
> not much load on the system at all.
>
> As we are hitting many issues in production, just using this system
> for my test purpose. Built few different filesystems. 1 with LZO
> compression, second one with ZLIB and third one without any
> compression. All has issues related to quota.
>
> whenever there is an issue, I am getting quota exceeded error (EDQUOT).
>
> Please let me know if you still need more info.
>

Would you please try this patch and see if it has any improvement?
https://patchwork.kernel.org/patch/9201685/

BTW, is balance/relocate involved in your workload?

Also, for non-compressed case, what's the threshold to trigger the bug?
Is it always about 100 and 90G?

Or is it related to the sum of the 2 subvolumes?
(In your initial report, the limit is 200 for each subvolume while the 
sum of the rfer of these 2 subvolumes seems to be 200G, maybe just 
coincident?)

Thanks,
Qu

>
>
> On Tue, Aug 16, 2016 at 5:56 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>>
>> At 08/17/2016 12:05 AM, Rakesh Sankeshi wrote:
>>>
>>> 2) after EDQUOT, can't write anymore.
>>>
>>> I can delete the data, but still can't write further
>>
>>
>> So it's a underflow now.
>>
>>>
>>> 3) tested it without compression and also with LZO and ZLIB.. all
>>> behave same way with qgroup. no consistency on when it hits the quota
>>> limit and don't understand on how it's calculating the numbers.
>>
>>
>> Even without compression?!
>>
>> That's a really big problem then.
>> Workload please, it's an urgent bug now.
>>
>> It's better to provide the scripts to reproduce it.
>>
>>
>>
>> And for the meaning of the numbers, for rfer(reference) it means the size of
>> all extents the subvolume has referred to, including both data and metadata.
>>
>> For excl(exclusive), it means the size of all extents that only belongs to
>> the subvolume.
>>
>> And since it's all about size of extents(on-disk), for compression case,
>> it's the size after compression.
>>
>> Also, if one subvolume only referred to part of an extent, the whole extent
>> size will be accounted.
>>
>>
>> Last but not least.
>> Considering there is quite a lot of report about hitting ENOSPC while there
>> is still a lot of unallocated space,
>> is it reporting error message like "No space left on device" (ENOSPC) or
>> "Quota exceeded"(EDQUOT)?
>>
>> Thanks,
>> Qu
>>
>>
>>>
>>> In case of ext4 and xfs, I can see visually that it's hitting the quota
>>> limit.
>>>
>>>
>>>
>>> On Mon, Aug 15, 2016 at 6:01 PM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> wrote:
>>>>
>>>>
>>>>
>>>> At 08/16/2016 03:11 AM, Rakesh Sankeshi wrote:
>>>>>
>>>>>
>>>>> yes, subvol level.
>>>>>
>>>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>>>> child
>>>>>
>>>>> --------         ----         ----     --------     -------- ------
>>>>> -----
>>>>>
>>>>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>>>>>
>>>>> 0/258       119.48GiB    119.48GiB    200.00GiB         none ---     ---
>>>>>
>>>>> 0/259        92.57GiB     92.57GiB    200.00GiB         none ---     ---
>>>>>
>>>>>
>>>>> although I have 200GB limit on 2 subvols, running into issue at about
>>>>> 120 and 92GB itself
>>>>
>>>>
>>>>
>>>> 1) About workload
>>>> Would you mind to mention the work pattern of your write?
>>>>
>>>> Just dd data with LZO compression?
>>>> For compression part, it's a little complicated, as the reserved data
>>>> size
>>>> and on disk extent size are different.
>>>>
>>>> It's possible that at some code we leaked some reserved data space.
>>>>
>>>>
>>>> 2) Behavior after EDQUOT
>>>> And, after EDQUOT happens, can you write data into the subvolume?
>>>> If you can still write a lot of data (at least several giga), it seems to
>>>> be
>>>> something related with temporary reserved space.
>>>>
>>>> If not, and even can't remove any file due to EQUOTA, then it's almost
>>>> sure
>>>> we have underflowed the reserved data.
>>>> In that case, unmount and mount again will be the only workaround.
>>>> (In fact, not workaround at all)
>>>>
>>>> 3) Behavior without compression
>>>>
>>>> If it's OK for you, would you mind to test it without compression?
>>>> Currently we mostly use the assumption that on-disk extent size are the
>>>> same
>>>> with in-memory extent size (non-compression).
>>>>
>>>> So qgroup + compression is not the main concern before and is buggy.
>>>>
>>>> If without compression, qgroup works sanely, at least we can be sure that
>>>> the cause is qgroup + compression.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>
>>>>>
>>>>>
>>>>> On Sun, Aug 14, 2016 at 7:11 PM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I set 200GB limit to one user and 100GB to another user.
>>>>>>>
>>>>>>> as soon as I reached 139GB and 53GB each, hitting the quota errors.
>>>>>>> anyway to workaround quota functionality on btrfs LZO compressed
>>>>>>> filesystem?
>>>>>>>
>>>>>>
>>>>>> Please paste "btrfs qgroup show -prce <mnt>" output if you are using
>>>>>> btrfs
>>>>>> qgroup/quota function.
>>>>>>
>>>>>> And, AFAIK btrfs qgroup is applied to subvolume, not user.
>>>>>>
>>>>>> So did you mean limit it to one subvolume belongs to one user?
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 4.7.0-040700-generic #201608021801 SMP
>>>>>>>
>>>>>>> btrfs-progs v4.7
>>>>>>>
>>>>>>>
>>>>>>> Label: none  uuid: 66a78faf-2052-4864-8a52-c5aec7a56ab8
>>>>>>>
>>>>>>> Total devices 2 FS bytes used 150.62GiB
>>>>>>>
>>>>>>> devid    1 size 1.00TiB used 78.01GiB path /dev/xvdc
>>>>>>>
>>>>>>> devid    2 size 1.00TiB used 78.01GiB path /dev/xvde
>>>>>>>
>>>>>>>
>>>>>>> Data, RAID0: total=150.00GiB, used=149.12GiB
>>>>>>>
>>>>>>> System, RAID1: total=8.00MiB, used=16.00KiB
>>>>>>>
>>>>>>> Metadata, RAID1: total=3.00GiB, used=1.49GiB
>>>>>>>
>>>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>>>>
>>>>>>>
>>>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>>>>
>>>>>>> /dev/xvdc       2.0T  153G  1.9T   8% /test_lzo
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>>>> in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>