Exclusive quota of snapshot exceeded despite no space used

Exclusive quota of snapshot exceeded despite no space used

[Johannes Henninger]

I'm having a weird problem with snapshots and exclusive quotas. After
creating a snapshot of a subvolume and setting an exclusive quota of
50MB for the snapshot, everything seems to work fine. I can write
approximately 50MB before the quota kicks in.

However, if I create a snapshot, set an exclusive quota and just wait
for some time, I suddenly cannot even create an empty file because I'm
getting a "quota exceeded" error. The time until the bug appears seems
to vary. During the waiting time, I'm changing neither the snapshot nor
the original subvolume. "qgroup show -e" reports an exclusive use of
only a few kilobytes for the snapshot, which is nowhere near the limit.

Steps to reproduce (/media/extern is a fresh and empty btrfs partition):

Enable quota and create an empty subvolume:
    root@t420:/media/extern# btrfs quota enable .
    root@t420:/media/extern# btrfs subvolume create sub
    Create subvolume './sub'

Snapshot the subvolume and set a limit:
    root@t420:/media/extern# btrfs subvolume snapshot sub snap
    Create a snapshot of 'sub' in './snap'
    root@t420:/media/extern# cd snap/
    root@t420:/media/extern/snap# btrfs qgroup limit -e 50M .

Sometimes it takes "longer" for the quota to kick in, so I'm touching a
file every 5 minutes here:

    root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
sleep 5m; done
    touch: cannot touch ‘7’: Disk quota exceeded
    ^C
    root@t420:/media/extern/snap# btrfs qgroup show -e .
    qgroupid         rfer         excl     max_excl
    --------         ----         ----     --------
    0/5          16.00KiB     16.00KiB         none
    0/257        16.00KiB     16.00KiB         none
    0/258        16.00KiB     16.00KiB     50.00MiB

Any idea why this happens?

Thanks,
Johannes

System info:
    
    Linux t420 4.3.0-rc5 #1 SMP Tue Oct 13 13:21:02 CEST 2015 x86_64
GNU/Linux
    
    Label: none  uuid: 9551e3ca-1608-469c-9d8c-77b99ce0e8ec
        Total devices 1 FS bytes used 816.00KiB
        devid    1 size 931.51GiB used 2.04GiB path /dev/sdb1

    btrfs-progs v4.1.2

    Data, single: total=8.00MiB, used=256.00KiB
    System, DUP: total=8.00MiB, used=16.00KiB
    System, single: total=4.00MiB, used=0.00B
    Metadata, DUP: total=1.00GiB, used=544.00KiB
    Metadata, single: total=8.00MiB, used=0.00B
    GlobalReserve, single: total=16.00MiB, used=0.00B

    [249174.151820]  sdb: sdb1
    [249184.387377]  sdb: sdb1
    [249184.573096]  sdb: sdb1
    [249184.656274] BTRFS: device fsid
9551e3ca-1608-469c-9d8c-77b99ce0e8ec devid 1 transid 3 /dev/sdb1
    [249186.323915]  sdb: sdb1
    [249186.534505]  sdb: sdb1
    [249186.538420]  sdb: sdb1
    [249196.781978] BTRFS info (device sdb1): disk space caching is enabled
    [249196.781986] BTRFS: has skinny extents
    [249196.781990] BTRFS: flagging fs with big metadata feature
    [249196.818164] BTRFS: creating UUID tree
    [249202.311983] BTRFS info (device sdb1): qgroup scan completed
(inconsistency flag cleared)

Re: Exclusive quota of snapshot exceeded despite no space used

[Qu Wenruo]

在 2015年10月23日 04:38, Johannes Henninger 写道:
> I'm having a weird problem with snapshots and exclusive quotas. After
> creating a snapshot of a subvolume and setting an exclusive quota of
> 50MB for the snapshot, everything seems to work fine. I can write
> approximately 50MB before the quota kicks in.
>
> However, if I create a snapshot, set an exclusive quota and just wait
> for some time, I suddenly cannot even create an empty file because I'm
> getting a "quota exceeded" error. The time until the bug appears seems
> to vary. During the waiting time, I'm changing neither the snapshot nor
> the original subvolume. "qgroup show -e" reports an exclusive use of
> only a few kilobytes for the snapshot, which is nowhere near the limit.
>
> Steps to reproduce (/media/extern is a fresh and empty btrfs partition):
>
> Enable quota and create an empty subvolume:
>      root@t420:/media/extern# btrfs quota enable .
>      root@t420:/media/extern# btrfs subvolume create sub
>      Create subvolume './sub'
>
> Snapshot the subvolume and set a limit:
>      root@t420:/media/extern# btrfs subvolume snapshot sub snap
>      Create a snapshot of 'sub' in './snap'
>      root@t420:/media/extern# cd snap/
>      root@t420:/media/extern/snap# btrfs qgroup limit -e 50M .
>
> Sometimes it takes "longer" for the quota to kick in, so I'm touching a
> file every 5 minutes here:
>
>      root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
> sleep 5m; done
>      touch: cannot touch ‘7’: Disk quota exceeded
>      ^C
>      root@t420:/media/extern/snap# btrfs qgroup show -e .
>      qgroupid         rfer         excl     max_excl
>      --------         ----         ----     --------
>      0/5          16.00KiB     16.00KiB         none
>      0/257        16.00KiB     16.00KiB         none
>      0/258        16.00KiB     16.00KiB     50.00MiB
>
> Any idea why this happens?
BTW, to make btrfs qgroup show work, it's better to call sync before 
qgroup show.

It's a known bug that even after qgroup accounting rework, qgroup 
reserve still has bug and can cause reserved space to underflow, making 
such problem happen.

For such case, btrfs qgroup show won't help as reserved space is not 
shown in the output.

One workaround would be, umount the filesystem and mount again.
Which will reset the underflow reserved space and work for sometime.

If it's OK for you to recompile the kernel, you can try the following 
patchset:
[PATCH v3 00/21] Rework btrfs qgroup reserved space framework

Which should solve the problem.

Thanks,
Qu

>
> Thanks,
> Johannes
>
> System info:
>
>      Linux t420 4.3.0-rc5 #1 SMP Tue Oct 13 13:21:02 CEST 2015 x86_64
> GNU/Linux
>
>      Label: none  uuid: 9551e3ca-1608-469c-9d8c-77b99ce0e8ec
>          Total devices 1 FS bytes used 816.00KiB
>          devid    1 size 931.51GiB used 2.04GiB path /dev/sdb1
>
>      btrfs-progs v4.1.2
>
>      Data, single: total=8.00MiB, used=256.00KiB
>      System, DUP: total=8.00MiB, used=16.00KiB
>      System, single: total=4.00MiB, used=0.00B
>      Metadata, DUP: total=1.00GiB, used=544.00KiB
>      Metadata, single: total=8.00MiB, used=0.00B
>      GlobalReserve, single: total=16.00MiB, used=0.00B
>
>      [249174.151820]  sdb: sdb1
>      [249184.387377]  sdb: sdb1
>      [249184.573096]  sdb: sdb1
>      [249184.656274] BTRFS: device fsid
> 9551e3ca-1608-469c-9d8c-77b99ce0e8ec devid 1 transid 3 /dev/sdb1
>      [249186.323915]  sdb: sdb1
>      [249186.534505]  sdb: sdb1
>      [249186.538420]  sdb: sdb1
>      [249196.781978] BTRFS info (device sdb1): disk space caching is enabled
>      [249196.781986] BTRFS: has skinny extents
>      [249196.781990] BTRFS: flagging fs with big metadata feature
>      [249196.818164] BTRFS: creating UUID tree
>      [249202.311983] BTRFS info (device sdb1): qgroup scan completed
> (inconsistency flag cleared)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Exclusive quota of snapshot exceeded despite no space used

[Johannes Henninger]

On 23.10.2015 01:47, Qu Wenruo wrote:
> 在 2015年10月23日 04:38, Johannes Henninger 写道:
>> I'm having a weird problem with snapshots and exclusive quotas. After
>> creating a snapshot of a subvolume and setting an exclusive quota of
>> 50MB for the snapshot, everything seems to work fine. I can write
>> approximately 50MB before the quota kicks in.
>>
>> However, if I create a snapshot, set an exclusive quota and just wait
>> for some time, I suddenly cannot even create an empty file because I'm
>> getting a "quota exceeded" error. The time until the bug appears seems
>> to vary. During the waiting time, I'm changing neither the snapshot nor
>> the original subvolume. "qgroup show -e" reports an exclusive use of
>> only a few kilobytes for the snapshot, which is nowhere near the limit.
>>
>> Steps to reproduce (/media/extern is a fresh and empty btrfs partition):
>>
>> Enable quota and create an empty subvolume:
>>      root@t420:/media/extern# btrfs quota enable .
>>      root@t420:/media/extern# btrfs subvolume create sub
>>      Create subvolume './sub'
>>
>> Snapshot the subvolume and set a limit:
>>      root@t420:/media/extern# btrfs subvolume snapshot sub snap
>>      Create a snapshot of 'sub' in './snap'
>>      root@t420:/media/extern# cd snap/
>>      root@t420:/media/extern/snap# btrfs qgroup limit -e 50M .
>>
>> Sometimes it takes "longer" for the quota to kick in, so I'm touching a
>> file every 5 minutes here:
>>
>>      root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
>> sleep 5m; done
>>      touch: cannot touch ‘7’: Disk quota exceeded
>>      ^C
>>      root@t420:/media/extern/snap# btrfs qgroup show -e .
>>      qgroupid         rfer         excl     max_excl
>>      --------         ----         ----     --------
>>      0/5          16.00KiB     16.00KiB         none
>>      0/257        16.00KiB     16.00KiB         none
>>      0/258        16.00KiB     16.00KiB     50.00MiB
>>
>> Any idea why this happens?
> BTW, to make btrfs qgroup show work, it's better to call sync before
> qgroup show.
>
> It's a known bug that even after qgroup accounting rework, qgroup
> reserve still has bug and can cause reserved space to underflow,
> making such problem happen.
>
> For such case, btrfs qgroup show won't help as reserved space is not
> shown in the output.
>
> One workaround would be, umount the filesystem and mount again.
> Which will reset the underflow reserved space and work for sometime.
>
> If it's OK for you to recompile the kernel, you can try the following
> patchset:
> [PATCH v3 00/21] Rework btrfs qgroup reserved space framework
>
> Which should solve the problem.
>
> Thanks,
> Qu
>

Thanks a lot for your reply!

While remounting the filesystem fixes the issue temporary, it doesn't
take very long for the bug to happen again so it's not really a
workaround I can work with.

I did recompile the kernel using your patches, but unfortunately the
problem still appears.

Thanks,
Johannes

Re: Exclusive quota of snapshot exceeded despite no space used

[Qu Wenruo]

在 2015年10月23日 23:05, Johannes Henninger 写道:
> On 23.10.2015 01:47, Qu Wenruo wrote:
>> 在 2015年10月23日 04:38, Johannes Henninger 写道:
>>> I'm having a weird problem with snapshots and exclusive quotas. After
>>> creating a snapshot of a subvolume and setting an exclusive quota of
>>> 50MB for the snapshot, everything seems to work fine. I can write
>>> approximately 50MB before the quota kicks in.
>>>
>>> However, if I create a snapshot, set an exclusive quota and just wait
>>> for some time, I suddenly cannot even create an empty file because I'm
>>> getting a "quota exceeded" error. The time until the bug appears seems
>>> to vary. During the waiting time, I'm changing neither the snapshot nor
>>> the original subvolume. "qgroup show -e" reports an exclusive use of
>>> only a few kilobytes for the snapshot, which is nowhere near the limit.
>>>
>>> Steps to reproduce (/media/extern is a fresh and empty btrfs partition):
>>>
>>> Enable quota and create an empty subvolume:
>>>       root@t420:/media/extern# btrfs quota enable .
>>>       root@t420:/media/extern# btrfs subvolume create sub
>>>       Create subvolume './sub'
>>>
>>> Snapshot the subvolume and set a limit:
>>>       root@t420:/media/extern# btrfs subvolume snapshot sub snap
>>>       Create a snapshot of 'sub' in './snap'
>>>       root@t420:/media/extern# cd snap/
>>>       root@t420:/media/extern/snap# btrfs qgroup limit -e 50M .
>>>
>>> Sometimes it takes "longer" for the quota to kick in, so I'm touching a
>>> file every 5 minutes here:
>>>
>>>       root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
>>> sleep 5m; done
>>>       touch: cannot touch ‘7’: Disk quota exceeded
>>>       ^C
>>>       root@t420:/media/extern/snap# btrfs qgroup show -e .
>>>       qgroupid         rfer         excl     max_excl
>>>       --------         ----         ----     --------
>>>       0/5          16.00KiB     16.00KiB         none
>>>       0/257        16.00KiB     16.00KiB         none
>>>       0/258        16.00KiB     16.00KiB     50.00MiB
>>>
>>> Any idea why this happens?
>> BTW, to make btrfs qgroup show work, it's better to call sync before
>> qgroup show.
>>
>> It's a known bug that even after qgroup accounting rework, qgroup
>> reserve still has bug and can cause reserved space to underflow,
>> making such problem happen.
>>
>> For such case, btrfs qgroup show won't help as reserved space is not
>> shown in the output.
>>
>> One workaround would be, umount the filesystem and mount again.
>> Which will reset the underflow reserved space and work for sometime.
>>
>> If it's OK for you to recompile the kernel, you can try the following
>> patchset:
>> [PATCH v3 00/21] Rework btrfs qgroup reserved space framework
>>
>> Which should solve the problem.
>>
>> Thanks,
>> Qu
>>
>
> Thanks a lot for your reply!
>
> While remounting the filesystem fixes the issue temporary, it doesn't
> take very long for the bug to happen again so it's not really a
> workaround I can work with.
>
> I did recompile the kernel using your patches, but unfortunately the
> problem still appears.
>
> Thanks,
> Johannes
>
Interesting, just touching file will cause EQUOTA is quite a big problem.

I'll try to reproduce it with my patchset and see what really caused the 
problem.
The problem seems to do with snapshot qgroup hacking.
But I'm not completely sure yet.

BTW, does "sync; btrfs qgroup show -prce" still show excl as 16K?
16K is the correct number with only 6 empty files, just in case.

Thanks,
Qu

Re: Exclusive quota of snapshot exceeded despite no space used

[Johannes Henninger]

On 25.10.2015 02:44, Qu Wenruo wrote:
>
>
> 在 2015年10月23日 23:05, Johannes Henninger 写道:
>> On 23.10.2015 01:47, Qu Wenruo wrote:
>>> 在 2015年10月23日 04:38, Johannes Henninger 写道:
>>>> I'm having a weird problem with snapshots and exclusive quotas. After
>>>> creating a snapshot of a subvolume and setting an exclusive quota of
>>>> 50MB for the snapshot, everything seems to work fine. I can write
>>>> approximately 50MB before the quota kicks in.
>>>>
>>>> However, if I create a snapshot, set an exclusive quota and just wait
>>>> for some time, I suddenly cannot even create an empty file because I'm
>>>> getting a "quota exceeded" error. The time until the bug appears seems
>>>> to vary. During the waiting time, I'm changing neither the snapshot
>>>> nor
>>>> the original subvolume. "qgroup show -e" reports an exclusive use of
>>>> only a few kilobytes for the snapshot, which is nowhere near the
>>>> limit.
>>>>
>>>> Steps to reproduce (/media/extern is a fresh and empty btrfs
>>>> partition):
>>>>
>>>> Enable quota and create an empty subvolume:
>>>>       root@t420:/media/extern# btrfs quota enable .
>>>>       root@t420:/media/extern# btrfs subvolume create sub
>>>>       Create subvolume './sub'
>>>>
>>>> Snapshot the subvolume and set a limit:
>>>>       root@t420:/media/extern# btrfs subvolume snapshot sub snap
>>>>       Create a snapshot of 'sub' in './snap'
>>>>       root@t420:/media/extern# cd snap/
>>>>       root@t420:/media/extern/snap# btrfs qgroup limit -e 50M .
>>>>
>>>> Sometimes it takes "longer" for the quota to kick in, so I'm
>>>> touching a
>>>> file every 5 minutes here:
>>>>
>>>>       root@t420:/media/extern/snap# for file in {1..100}; do touch
>>>> $file;
>>>> sleep 5m; done
>>>>       touch: cannot touch ‘7’: Disk quota exceeded
>>>>       ^C
>>>>       root@t420:/media/extern/snap# btrfs qgroup show -e .
>>>>       qgroupid         rfer         excl     max_excl
>>>>       --------         ----         ----     --------
>>>>       0/5          16.00KiB     16.00KiB         none
>>>>       0/257        16.00KiB     16.00KiB         none
>>>>       0/258        16.00KiB     16.00KiB     50.00MiB
>>>>
>>>> Any idea why this happens?
>>> BTW, to make btrfs qgroup show work, it's better to call sync before
>>> qgroup show.
>>>
>>> It's a known bug that even after qgroup accounting rework, qgroup
>>> reserve still has bug and can cause reserved space to underflow,
>>> making such problem happen.
>>>
>>> For such case, btrfs qgroup show won't help as reserved space is not
>>> shown in the output.
>>>
>>> One workaround would be, umount the filesystem and mount again.
>>> Which will reset the underflow reserved space and work for sometime.
>>>
>>> If it's OK for you to recompile the kernel, you can try the following
>>> patchset:
>>> [PATCH v3 00/21] Rework btrfs qgroup reserved space framework
>>>
>>> Which should solve the problem.
>>>
>>> Thanks,
>>> Qu
>>>
>>
>> Thanks a lot for your reply!
>>
>> While remounting the filesystem fixes the issue temporary, it doesn't
>> take very long for the bug to happen again so it's not really a
>> workaround I can work with.
>>
>> I did recompile the kernel using your patches, but unfortunately the
>> problem still appears.
>>
>> Thanks,
>> Johannes
>>
> Interesting, just touching file will cause EQUOTA is quite a big problem.
>
> I'll try to reproduce it with my patchset and see what really caused
> the problem.
> The problem seems to do with snapshot qgroup hacking.
> But I'm not completely sure yet.
>
> BTW, does "sync; btrfs qgroup show -prce" still show excl as 16K?
> 16K is the correct number with only 6 empty files, just in case.
>
> Thanks,
> Qu

I ran my example from the first mail again and managed to write 7 files
this time, "qgroup show" still shows 16kB after sync:

root@t420:/media/extern/snap# btrfs qg limit -e 50M .
root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
sleep 5m; done
touch: cannot touch ‘8’: Disk quota exceeded
^C
root@t420:/media/extern/snap# sync
root@t420:/media/extern/snap# btrfs qgroup show -pcre .
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     --- 
0/257        16.00KiB     16.00KiB         none         none ---     --- 
0/258        16.00KiB     16.00KiB         none     50.00MiB ---     --- 
root@t420:/media/extern/snap# btrfs fi sync .
FSSync '.'
root@t420:/media/extern/snap# btrfs qgroup show -pcre .
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     --- 
0/257        16.00KiB     16.00KiB         none         none ---     --- 
0/258        16.00KiB     16.00KiB         none     50.00MiB ---     --- 

By the way, I don't if its relevant but the problem is not limited to
exclusive quotas, but also happens when setting a "referenced" limit
(qgroup limit without "-e").

Thanks,
Johannes

Re: Exclusive quota of snapshot exceeded despite no space used

[Qu Wenruo]

在 2015年10月25日 19:10, Johannes Henninger 写道:
> On 25.10.2015 02:44, Qu Wenruo wrote:
>>
>>
>> 在 2015年10月23日 23:05, Johannes Henninger 写道:
>>> On 23.10.2015 01:47, Qu Wenruo wrote:
>>>> 在 2015年10月23日 04:38, Johannes Henninger 写道:
>>>>> I'm having a weird problem with snapshots and exclusive quotas. After
>>>>> creating a snapshot of a subvolume and setting an exclusive quota of
>>>>> 50MB for the snapshot, everything seems to work fine. I can write
>>>>> approximately 50MB before the quota kicks in.
>>>>>
>>>>> However, if I create a snapshot, set an exclusive quota and just wait
>>>>> for some time, I suddenly cannot even create an empty file because I'm
>>>>> getting a "quota exceeded" error. The time until the bug appears seems
>>>>> to vary. During the waiting time, I'm changing neither the snapshot
>>>>> nor
>>>>> the original subvolume. "qgroup show -e" reports an exclusive use of
>>>>> only a few kilobytes for the snapshot, which is nowhere near the
>>>>> limit.
>>>>>
>>>>> Steps to reproduce (/media/extern is a fresh and empty btrfs
>>>>> partition):
>>>>>
>>>>> Enable quota and create an empty subvolume:
>>>>>        root@t420:/media/extern# btrfs quota enable .
>>>>>        root@t420:/media/extern# btrfs subvolume create sub
>>>>>        Create subvolume './sub'
>>>>>
>>>>> Snapshot the subvolume and set a limit:
>>>>>        root@t420:/media/extern# btrfs subvolume snapshot sub snap
>>>>>        Create a snapshot of 'sub' in './snap'
>>>>>        root@t420:/media/extern# cd snap/
>>>>>        root@t420:/media/extern/snap# btrfs qgroup limit -e 50M .
>>>>>
>>>>> Sometimes it takes "longer" for the quota to kick in, so I'm
>>>>> touching a
>>>>> file every 5 minutes here:
>>>>>
>>>>>        root@t420:/media/extern/snap# for file in {1..100}; do touch
>>>>> $file;
>>>>> sleep 5m; done
>>>>>        touch: cannot touch ‘7’: Disk quota exceeded
>>>>>        ^C
>>>>>        root@t420:/media/extern/snap# btrfs qgroup show -e .
>>>>>        qgroupid         rfer         excl     max_excl
>>>>>        --------         ----         ----     --------
>>>>>        0/5          16.00KiB     16.00KiB         none
>>>>>        0/257        16.00KiB     16.00KiB         none
>>>>>        0/258        16.00KiB     16.00KiB     50.00MiB
>>>>>
>>>>> Any idea why this happens?
>>>> BTW, to make btrfs qgroup show work, it's better to call sync before
>>>> qgroup show.
>>>>
>>>> It's a known bug that even after qgroup accounting rework, qgroup
>>>> reserve still has bug and can cause reserved space to underflow,
>>>> making such problem happen.
>>>>
>>>> For such case, btrfs qgroup show won't help as reserved space is not
>>>> shown in the output.
>>>>
>>>> One workaround would be, umount the filesystem and mount again.
>>>> Which will reset the underflow reserved space and work for sometime.
>>>>
>>>> If it's OK for you to recompile the kernel, you can try the following
>>>> patchset:
>>>> [PATCH v3 00/21] Rework btrfs qgroup reserved space framework
>>>>
>>>> Which should solve the problem.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>
>>> Thanks a lot for your reply!
>>>
>>> While remounting the filesystem fixes the issue temporary, it doesn't
>>> take very long for the bug to happen again so it's not really a
>>> workaround I can work with.
>>>
>>> I did recompile the kernel using your patches, but unfortunately the
>>> problem still appears.
>>>
>>> Thanks,
>>> Johannes
>>>
>> Interesting, just touching file will cause EQUOTA is quite a big problem.
>>
>> I'll try to reproduce it with my patchset and see what really caused
>> the problem.
>> The problem seems to do with snapshot qgroup hacking.
>> But I'm not completely sure yet.
>>
>> BTW, does "sync; btrfs qgroup show -prce" still show excl as 16K?
>> 16K is the correct number with only 6 empty files, just in case.
>>
>> Thanks,
>> Qu
>
> I ran my example from the first mail again and managed to write 7 files
> this time, "qgroup show" still shows 16kB after sync:
>
> root@t420:/media/extern/snap# btrfs qg limit -e 50M .
> root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
> sleep 5m; done
> touch: cannot touch ‘8’: Disk quota exceeded
> ^C
> root@t420:/media/extern/snap# sync
> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
> qgroupid         rfer         excl     max_rfer     max_excl parent  child
> --------         ----         ----     --------     -------- ------  -----
> 0/5          16.00KiB     16.00KiB         none         none ---     ---
> 0/257        16.00KiB     16.00KiB         none         none ---     ---
> 0/258        16.00KiB     16.00KiB         none     50.00MiB ---     ---
> root@t420:/media/extern/snap# btrfs fi sync .
> FSSync '.'
> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
> qgroupid         rfer         excl     max_rfer     max_excl parent  child
> --------         ----         ----     --------     -------- ------  -----
> 0/5          16.00KiB     16.00KiB         none         none ---     ---
> 0/257        16.00KiB     16.00KiB         none         none ---     ---
> 0/258        16.00KiB     16.00KiB         none     50.00MiB ---     ---
>
Thanks for the report.

At least, the rfer/excl number is correct.

So the problem is limited to qgroup reserved space.
I'll debug it tomorrow to see what's going wrong inside the reserved space.

> By the way, I don't if its relevant but the problem is not limited to
> exclusive quotas, but also happens when setting a "referenced" limit
> (qgroup limit without "-e").

This also indicates that the problem is in reserved space, not rfer/excl 
accounting, as reserved space is used for both rfer or excl, so if it's 
going wrong, no matter to which the limit is set, it will cause EDQUOT.

Thanks,
Qu

>
> Thanks,
> Johannes
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Exclusive quota of snapshot exceeded despite no space used

[Qu Wenruo]

>>> Thanks a lot for your reply!
>>>
>>> While remounting the filesystem fixes the issue temporary, it doesn't
>>> take very long for the bug to happen again so it's not really a
>>> workaround I can work with.
>>>
>>> I did recompile the kernel using your patches, but unfortunately the
>>> problem still appears.
>>>
>>> Thanks,
>>> Johannes
>>>
>> Interesting, just touching file will cause EQUOTA is quite a big problem.
>>
>> I'll try to reproduce it with my patchset and see what really caused
>> the problem.
>> The problem seems to do with snapshot qgroup hacking.
>> But I'm not completely sure yet.
>>
>> BTW, does "sync; btrfs qgroup show -prce" still show excl as 16K?
>> 16K is the correct number with only 6 empty files, just in case.
>>
>> Thanks,
>> Qu
>
> I ran my example from the first mail again and managed to write 7 files
> this time, "qgroup show" still shows 16kB after sync:
>
> root@t420:/media/extern/snap# btrfs qg limit -e 50M .
> root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
> sleep 5m; done
> touch: cannot touch ‘8’: Disk quota exceeded
> ^C
> root@t420:/media/extern/snap# sync
> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
> qgroupid         rfer         excl     max_rfer     max_excl parent  child
> --------         ----         ----     --------     -------- ------  -----
> 0/5          16.00KiB     16.00KiB         none         none ---     ---
> 0/257        16.00KiB     16.00KiB         none         none ---     ---
> 0/258        16.00KiB     16.00KiB         none     50.00MiB ---     ---
> root@t420:/media/extern/snap# btrfs fi sync .
> FSSync '.'
> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
> qgroupid         rfer         excl     max_rfer     max_excl parent  child
> --------         ----         ----     --------     -------- ------  -----
> 0/5          16.00KiB     16.00KiB         none         none ---     ---
> 0/257        16.00KiB     16.00KiB         none         none ---     ---
> 0/258        16.00KiB     16.00KiB         none     50.00MiB ---     ---
>
> By the way, I don't if its relevant but the problem is not limited to
> exclusive quotas, but also happens when setting a "referenced" limit
> (qgroup limit without "-e").
>
> Thanks,
> Johannes
>

The bug is located, and turns out to be quite a stupid problem caused by 
myself.

I just forgot to include a cleanup patch during rebase!!!! AGAIN!!!

You can apply the following patch to resolve it:
[PATCH 3/3] btrfs: qgroup: Fix a rebase bug which will cause qgroup 
double free

Or just apply the whole patchset:
[4.4][PATCH 0/3] btrfs: Qgroup hotfix

At least, with the patchset based on Chris' integration-4.4 branch, it 
succeeded in touching all the 100 files in my test box.

Thanks,
Qu


> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Exclusive quota of snapshot exceeded despite no space used

[Johannes Henninger]

On 26.10.2015 08:12, Qu Wenruo wrote:
>
>>>> Thanks a lot for your reply!
>>>>
>>>> While remounting the filesystem fixes the issue temporary, it doesn't
>>>> take very long for the bug to happen again so it's not really a
>>>> workaround I can work with.
>>>>
>>>> I did recompile the kernel using your patches, but unfortunately the
>>>> problem still appears.
>>>>
>>>> Thanks,
>>>> Johannes
>>>>
>>> Interesting, just touching file will cause EQUOTA is quite a big
>>> problem.
>>>
>>> I'll try to reproduce it with my patchset and see what really caused
>>> the problem.
>>> The problem seems to do with snapshot qgroup hacking.
>>> But I'm not completely sure yet.
>>>
>>> BTW, does "sync; btrfs qgroup show -prce" still show excl as 16K?
>>> 16K is the correct number with only 6 empty files, just in case.
>>>
>>> Thanks,
>>> Qu
>>
>> I ran my example from the first mail again and managed to write 7 files
>> this time, "qgroup show" still shows 16kB after sync:
>>
>> root@t420:/media/extern/snap# btrfs qg limit -e 50M .
>> root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
>> sleep 5m; done
>> touch: cannot touch ‘8’: Disk quota exceeded
>> ^C
>> root@t420:/media/extern/snap# sync
>> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
>> qgroupid         rfer         excl     max_rfer     max_excl parent 
>> child
>> --------         ----         ----     --------     -------- ------ 
>> -----
>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>> 0/257        16.00KiB     16.00KiB         none         none ---     ---
>> 0/258        16.00KiB     16.00KiB         none     50.00MiB ---     ---
>> root@t420:/media/extern/snap# btrfs fi sync .
>> FSSync '.'
>> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
>> qgroupid         rfer         excl     max_rfer     max_excl parent 
>> child
>> --------         ----         ----     --------     -------- ------ 
>> -----
>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>> 0/257        16.00KiB     16.00KiB         none         none ---     ---
>> 0/258        16.00KiB     16.00KiB         none     50.00MiB ---     ---
>>
>> By the way, I don't if its relevant but the problem is not limited to
>> exclusive quotas, but also happens when setting a "referenced" limit
>> (qgroup limit without "-e").
>>
>> Thanks,
>> Johannes
>>
>
> The bug is located, and turns out to be quite a stupid problem caused
> by myself.
>
> I just forgot to include a cleanup patch during rebase!!!! AGAIN!!!
>
> You can apply the following patch to resolve it:
> [PATCH 3/3] btrfs: qgroup: Fix a rebase bug which will cause qgroup
> double free
>
> Or just apply the whole patchset:
> [4.4][PATCH 0/3] btrfs: Qgroup hotfix
>
> At least, with the patchset based on Chris' integration-4.4 branch, it
> succeeded in touching all the 100 files in my test box.
>
> Thanks,
> Qu
>

It's working! Thank you so much for fixing this bug, you don't even know
how much this has helped me!

Thanks!
Johannes

Re: Exclusive quota of snapshot exceeded despite no space used

[Qu Wenruo]

Johannes Henninger wrote on 2015/10/27 01:15 +0100:
> On 26.10.2015 08:12, Qu Wenruo wrote:
>>
>>>>> Thanks a lot for your reply!
>>>>>
>>>>> While remounting the filesystem fixes the issue temporary, it doesn't
>>>>> take very long for the bug to happen again so it's not really a
>>>>> workaround I can work with.
>>>>>
>>>>> I did recompile the kernel using your patches, but unfortunately the
>>>>> problem still appears.
>>>>>
>>>>> Thanks,
>>>>> Johannes
>>>>>
>>>> Interesting, just touching file will cause EQUOTA is quite a big
>>>> problem.
>>>>
>>>> I'll try to reproduce it with my patchset and see what really caused
>>>> the problem.
>>>> The problem seems to do with snapshot qgroup hacking.
>>>> But I'm not completely sure yet.
>>>>
>>>> BTW, does "sync; btrfs qgroup show -prce" still show excl as 16K?
>>>> 16K is the correct number with only 6 empty files, just in case.
>>>>
>>>> Thanks,
>>>> Qu
>>>
>>> I ran my example from the first mail again and managed to write 7 files
>>> this time, "qgroup show" still shows 16kB after sync:
>>>
>>> root@t420:/media/extern/snap# btrfs qg limit -e 50M .
>>> root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
>>> sleep 5m; done
>>> touch: cannot touch ‘8’: Disk quota exceeded
>>> ^C
>>> root@t420:/media/extern/snap# sync
>>> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>> child
>>> --------         ----         ----     --------     -------- ------
>>> -----
>>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>>> 0/257        16.00KiB     16.00KiB         none         none ---     ---
>>> 0/258        16.00KiB     16.00KiB         none     50.00MiB ---     ---
>>> root@t420:/media/extern/snap# btrfs fi sync .
>>> FSSync '.'
>>> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>> child
>>> --------         ----         ----     --------     -------- ------
>>> -----
>>> 0/5          16.00KiB     16.00KiB         none         none ---     ---
>>> 0/257        16.00KiB     16.00KiB         none         none ---     ---
>>> 0/258        16.00KiB     16.00KiB         none     50.00MiB ---     ---
>>>
>>> By the way, I don't if its relevant but the problem is not limited to
>>> exclusive quotas, but also happens when setting a "referenced" limit
>>> (qgroup limit without "-e").
>>>
>>> Thanks,
>>> Johannes
>>>
>>
>> The bug is located, and turns out to be quite a stupid problem caused
>> by myself.
>>
>> I just forgot to include a cleanup patch during rebase!!!! AGAIN!!!
>>
>> You can apply the following patch to resolve it:
>> [PATCH 3/3] btrfs: qgroup: Fix a rebase bug which will cause qgroup
>> double free
>>
>> Or just apply the whole patchset:
>> [4.4][PATCH 0/3] btrfs: Qgroup hotfix
>>
>> At least, with the patchset based on Chris' integration-4.4 branch, it
>> succeeded in touching all the 100 files in my test box.
>>
>> Thanks,
>> Qu
>>
>
> It's working! Thank you so much for fixing this bug, you don't even know
> how much this has helped me!
>
> Thanks!
> Johannes
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
Glad to hear that.

If it's working for you, it would be better to add a 'Tested-by' tag for 
the 3rd patch.

Thanks,
Qu

Re: Exclusive quota of snapshot exceeded despite no space used

[Johannes Henninger]

On 27.10.2015 02:06, Qu Wenruo wrote:
>
>
> Johannes Henninger wrote on 2015/10/27 01:15 +0100:
>> On 26.10.2015 08:12, Qu Wenruo wrote:
>>>
>>>>>> Thanks a lot for your reply!
>>>>>>
>>>>>> While remounting the filesystem fixes the issue temporary, it
>>>>>> doesn't
>>>>>> take very long for the bug to happen again so it's not really a
>>>>>> workaround I can work with.
>>>>>>
>>>>>> I did recompile the kernel using your patches, but unfortunately the
>>>>>> problem still appears.
>>>>>>
>>>>>> Thanks,
>>>>>> Johannes
>>>>>>
>>>>> Interesting, just touching file will cause EQUOTA is quite a big
>>>>> problem.
>>>>>
>>>>> I'll try to reproduce it with my patchset and see what really caused
>>>>> the problem.
>>>>> The problem seems to do with snapshot qgroup hacking.
>>>>> But I'm not completely sure yet.
>>>>>
>>>>> BTW, does "sync; btrfs qgroup show -prce" still show excl as 16K?
>>>>> 16K is the correct number with only 6 empty files, just in case.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>
>>>> I ran my example from the first mail again and managed to write 7
>>>> files
>>>> this time, "qgroup show" still shows 16kB after sync:
>>>>
>>>> root@t420:/media/extern/snap# btrfs qg limit -e 50M .
>>>> root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
>>>> sleep 5m; done
>>>> touch: cannot touch ‘8’: Disk quota exceeded
>>>> ^C
>>>> root@t420:/media/extern/snap# sync
>>>> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
>>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>>> child
>>>> --------         ----         ----     --------     -------- ------
>>>> -----
>>>> 0/5          16.00KiB     16.00KiB         none         none
>>>> ---     ---
>>>> 0/257        16.00KiB     16.00KiB         none         none
>>>> ---     ---
>>>> 0/258        16.00KiB     16.00KiB         none     50.00MiB
>>>> ---     ---
>>>> root@t420:/media/extern/snap# btrfs fi sync .
>>>> FSSync '.'
>>>> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
>>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>>> child
>>>> --------         ----         ----     --------     -------- ------
>>>> -----
>>>> 0/5          16.00KiB     16.00KiB         none         none
>>>> ---     ---
>>>> 0/257        16.00KiB     16.00KiB         none         none
>>>> ---     ---
>>>> 0/258        16.00KiB     16.00KiB         none     50.00MiB
>>>> ---     ---
>>>>
>>>> By the way, I don't if its relevant but the problem is not limited to
>>>> exclusive quotas, but also happens when setting a "referenced" limit
>>>> (qgroup limit without "-e").
>>>>
>>>> Thanks,
>>>> Johannes
>>>>
>>>
>>> The bug is located, and turns out to be quite a stupid problem caused
>>> by myself.
>>>
>>> I just forgot to include a cleanup patch during rebase!!!! AGAIN!!!
>>>
>>> You can apply the following patch to resolve it:
>>> [PATCH 3/3] btrfs: qgroup: Fix a rebase bug which will cause qgroup
>>> double free
>>>
>>> Or just apply the whole patchset:
>>> [4.4][PATCH 0/3] btrfs: Qgroup hotfix
>>>
>>> At least, with the patchset based on Chris' integration-4.4 branch, it
>>> succeeded in touching all the 100 files in my test box.
>>>
>>> Thanks,
>>> Qu
>>>
>>
>> It's working! Thank you so much for fixing this bug, you don't even know
>> how much this has helped me!
>>
>> Thanks!
>> Johannes
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> Glad to hear that.
>
> If it's working for you, it would be better to add a 'Tested-by' tag
> for the 3rd patch.
>
> Thanks,
> Qu

Sure! Is there anything I have to do? I'm a kernel and mailing list noob :)

Thanks,
Johannes

Re: Exclusive quota of snapshot exceeded despite no space used

[Qu Wenruo]

Johannes Henninger wrote on 2015/10/28 15:02 +0100:
> On 27.10.2015 02:06, Qu Wenruo wrote:
>>
>>
>> Johannes Henninger wrote on 2015/10/27 01:15 +0100:
>>> On 26.10.2015 08:12, Qu Wenruo wrote:
>>>>
>>>>>>> Thanks a lot for your reply!
>>>>>>>
>>>>>>> While remounting the filesystem fixes the issue temporary, it
>>>>>>> doesn't
>>>>>>> take very long for the bug to happen again so it's not really a
>>>>>>> workaround I can work with.
>>>>>>>
>>>>>>> I did recompile the kernel using your patches, but unfortunately the
>>>>>>> problem still appears.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Johannes
>>>>>>>
>>>>>> Interesting, just touching file will cause EQUOTA is quite a big
>>>>>> problem.
>>>>>>
>>>>>> I'll try to reproduce it with my patchset and see what really caused
>>>>>> the problem.
>>>>>> The problem seems to do with snapshot qgroup hacking.
>>>>>> But I'm not completely sure yet.
>>>>>>
>>>>>> BTW, does "sync; btrfs qgroup show -prce" still show excl as 16K?
>>>>>> 16K is the correct number with only 6 empty files, just in case.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>
>>>>> I ran my example from the first mail again and managed to write 7
>>>>> files
>>>>> this time, "qgroup show" still shows 16kB after sync:
>>>>>
>>>>> root@t420:/media/extern/snap# btrfs qg limit -e 50M .
>>>>> root@t420:/media/extern/snap# for file in {1..100}; do touch $file;
>>>>> sleep 5m; done
>>>>> touch: cannot touch ‘8’: Disk quota exceeded
>>>>> ^C
>>>>> root@t420:/media/extern/snap# sync
>>>>> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
>>>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>>>> child
>>>>> --------         ----         ----     --------     -------- ------
>>>>> -----
>>>>> 0/5          16.00KiB     16.00KiB         none         none
>>>>> ---     ---
>>>>> 0/257        16.00KiB     16.00KiB         none         none
>>>>> ---     ---
>>>>> 0/258        16.00KiB     16.00KiB         none     50.00MiB
>>>>> ---     ---
>>>>> root@t420:/media/extern/snap# btrfs fi sync .
>>>>> FSSync '.'
>>>>> root@t420:/media/extern/snap# btrfs qgroup show -pcre .
>>>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>>>> child
>>>>> --------         ----         ----     --------     -------- ------
>>>>> -----
>>>>> 0/5          16.00KiB     16.00KiB         none         none
>>>>> ---     ---
>>>>> 0/257        16.00KiB     16.00KiB         none         none
>>>>> ---     ---
>>>>> 0/258        16.00KiB     16.00KiB         none     50.00MiB
>>>>> ---     ---
>>>>>
>>>>> By the way, I don't if its relevant but the problem is not limited to
>>>>> exclusive quotas, but also happens when setting a "referenced" limit
>>>>> (qgroup limit without "-e").
>>>>>
>>>>> Thanks,
>>>>> Johannes
>>>>>
>>>>
>>>> The bug is located, and turns out to be quite a stupid problem caused
>>>> by myself.
>>>>
>>>> I just forgot to include a cleanup patch during rebase!!!! AGAIN!!!
>>>>
>>>> You can apply the following patch to resolve it:
>>>> [PATCH 3/3] btrfs: qgroup: Fix a rebase bug which will cause qgroup
>>>> double free
>>>>
>>>> Or just apply the whole patchset:
>>>> [4.4][PATCH 0/3] btrfs: Qgroup hotfix
>>>>
>>>> At least, with the patchset based on Chris' integration-4.4 branch, it
>>>> succeeded in touching all the 100 files in my test box.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>
>>> It's working! Thank you so much for fixing this bug, you don't even know
>>> how much this has helped me!
>>>
>>> Thanks!
>>> Johannes
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>> Glad to hear that.
>>
>> If it's working for you, it would be better to add a 'Tested-by' tag
>> for the 3rd patch.
>>
>> Thanks,
>> Qu
>
> Sure! Is there anything I have to do? I'm a kernel and mailing list noob :)
>
> Thanks,
> Johannes
>
Find the mail
"[PATCH 3/3] btrfs: qgroup: Fix a rebase bug which will cause qgroup
double free",
and reply to it with the following contents if you tested the patch:

Tested-by: Johannes Hennigner <johannes+btrfs@henniger.io>


Also refer to kernel documentation/SubmittingPatches.

Thanks,
Qu