Re: [question] free space of disk with btrfs is too different from other du

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2021/6/25 下午1:28, Qu Wenruo wrote:
>
>
> On 2021/6/25 上午8:59, Qu Wenruo wrote:
>>
>>
>> On 2021/6/25 上午7:46, Qu Wenruo wrote:
>>>
>>>
>>> On 2021/6/24 下午10:38, Zhenyu Wu wrote:
>>>> ```
>>>> $ btrfs ins dump-tree -t data_reloc
>>>> btrfs-progs v5.10.1
>>>> data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0)
>>>> leaf 57458688 items 2 free space 16061 generation 941500 owner
>>>> DATA_RELOC_TREE
>>>> leaf 57458688 flags 0x1(WRITTEN) backref revision 1
>>>> fs uuid 644a9e91-5e05-4f5d-a79b-e0eccd21a1a8
>>>> chunk uuid fa782e34-63ae-4917-ac63-bbbe56851d8b
>>>> item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
>>>> generation 3 transid 0 size 0 nbytes 16384
>>>> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>>>> sequence 0 flags 0x0(none)
>>>> atime 1579910090.0 (2020-01-24 23:54:50)
>>>> ctime 1579910090.0 (2020-01-24 23:54:50)
>>>> mtime 1579910090.0 (2020-01-24 23:54:50)
>>>> otime 1579910090.0 (2020-01-24 23:54:50)
>>>> item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
>>>> index 0 namelen 2 name: ..
>>>> total bytes 499972575232
>>>> bytes used 466593501184
>>>> uuid 644a9e91-5e05-4f5d-a79b-e0eccd21a1a8
>>>> $ btrfs ins dump-tree -t root 2>&1|tee root.txt
>>>> # attachment
>>>
>>> This explains everything!
>>>
>>> There are 58 root items! Meaning all of those snapshots are still taking
>>> space.
>>>
>>> Furthermore, all those roots have no backref, meaning it's already
>>> unlinked from the fs, thus btrfs subvolume list won't list them.
>>>
>>> For the worst part, they also have no orphan item marking them to be
>>> deleted.
>>> Thus they all become real ghost subvolumes.
>>>
>>> And since qgroup rescan only create subvolumes based on its ROOT_REF_KEY
>>> to create 0 level qgroups, all these ghost snapshots are not created,
>>> causing the super weird result.
>>>
>>> I have no idea why this could happen, need to dig further into the code.
>>
>> A quick glance into the code, and it seems that for such ghost
>> subvolumes, "btrfs subv delete -i" may not work.
>>
>> For that case, I'll craft you a special btrfs-progs branch so that we
>> can add orphan items for those ghost roots and let them go for good.
>>
>> Thus if above "delete -i" doesn't work, please prepare a liveUSB in
>> which you can compile btrfs-progs.
>
> Here you go, please use the branch of btrfs-progs in LiveUSB:
>
> https://github.com/adam900710/btrfs-progs/tree/dirty_fixes

Forgot to mention, I have tested the tool locally without problem.

But even if the tool crashes by somehow, it shouldn't change any thing
of your fs, so feel free to use it.

After digging into the kernel code, kernel will just drop snapshots one
by one, thus the limit is not really needed.

You can change the line of "u64 orphan_roots[4] = { 0 };" to
"u64 orphan_roots[64] = { 0 }", which will cover all your ghosts roots
in one go.

For the formal fix, I'll enhance btrfs-check to detect and fix the
problem soon, but I'm afraid you have to use the dirty fix branch for now.

Thanks,
Qu
>
> It enhances btrfs-progs/btrfs-corrupt-block command (I know it's scary,
> but trust me, I make the extra option to be extra safe).
>
> Now in your liveUSB, make sure your fs is not mounted first.
>
> Then go the following loop:
>
> # ./btrfs-corrupt-block -X <device>
>
> It will output which roots are ghosts and will add back orphan items for
> them.
>
> # mount <device> <mnt>
>
> # btrfs subv sync <mnt>
>
> It will wait for the 4 subvolumes to be deleted.
>
> # umount <mnt>
>
> Then go back to "corrupt-block -X" command and continue.
> I made the command to only add orphans to at most 4 subvolumes, to make
> sure we won't trigger too many deletions at the same time.
>
> If you don't want above loop (you need to go at least 15 loops), nor
> want to craft a small script to do that, I can change the code to do all
> of them in one go, but I'm not sure if it's OK for the kernel to handle
> so many orphan roots.
>
> BTW, since you have qgroup enabled, it will make subvolume deletion
> pretty slow, it's recommended to disable qgroup first. >
> Thanks,
> Qu
>>
>> Thanks,
>> Qu
>>>
>>> But for now, you can try to delete all these 58 subvolumes by their
>>> subvolume ID.
>>>
>>> You can try this as a quick test:
>>>
>>> # btrfs subv delete -i 6317 <mnt>
>>> # btrfs subv sync <mnt>
>>>
>>> Then check if you still have "6317 ROOT_ITEM" in your root tree dump.
>>>
>>> If it's gone, repeat the process for all roots, I have extracted the
>>> root list and attached, so that you can craft a bash script to delete
>>> all of them.
>>> (Note that above "subv sync" is not needed for each deletion if you want
>>> to delete them in a batch)
>>>
>>> Thanks,
>>> Qu
>>>
>>>> $ btrfs ins dump-tree -t extent 2>&1|tee extent.txt
>>>> # it's too large, i'll upload it to a cloud disk
>>>> ```
>>>>
>>>> thanks!
>>>>
>>>>
>>>>
>>>> On Thu, Jun 24, 2021 at 8:30 PM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2021/6/24 下午7:33, Zhenyu Wu wrote:
>>>>>> the output has changed. do i need `btrfs ins dump-tree -t root`?
>>>>>> ```
>>>>>> $ sudo btrfs quota disable /
>>>>>> $ sudo btrfs quota enable /
>>>>>> $ sudo btrfs quota rescan -w /
>>>>>> # after 22m11s
>>>>>> $ sudo btrfs qgroup show -pcre /
>>>>>> qgroupid         rfer         excl     max_rfer     max_excl parent
>>>>>> child
>>>>>> --------         ----         ----     --------     -------- ------
>>>>>> -----
>>>>>> 0/5          81.23GiB      6.90GiB         none         none ---
>>>>>> ---
>>>>>> ```
>>>>>> thanks!
>>>>>
>>>>> Yes, dump tree output for both root and data_reloc is needed.
>>>>>
>>>>> There may be a larger dump needed, "btrfs ins dump-tree -t extent
>>>>> <device>", as I guess there is some ghost trees not properly
>>>>> deleted at
>>>>> all...
>>>>>
>>>>> The whole thing is going crazy now.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>>
>>>>>> On Thu, Jun 24, 2021 at 6:07 PM Qu Wenruo <quwenruo.btrfs@gmx.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2021/6/24 下午5:52, Zhenyu Wu wrote:
>>>>>>>> i have rescan quota but it looks like nothing change...
>>>>>>>> ```
>>>>>>>> $ sudo btrfs quota rescan -w /
>>>>>>>> quota rescan started
>>>>>>>> # after 8m39s
>>>>>>>> $ sudo btrfs qgroup show -pcre /
>>>>>>>> qgroupid         rfer         excl     max_rfer     max_excl
>>>>>>>> parent   child
>>>>>>>> --------         ----         ----     --------     --------
>>>>>>>> ------   -----
>>>>>>>> 0/5          81.23GiB      6.89GiB         none         none
>>>>>>>> ---      ---
>>>>>>>> 0/265           0.00B        0.00B         none         none
>>>>>>>> ---      ---
>>>>>>>> 0/266           0.00B        0.00B         none         none
>>>>>>>> ---      ---
>>>>>>>> 0/267           0.00B        0.00B         none         none
>>>>>>>> ---      ---
>>>>>>>> 0/6482          0.00B        0.00B         none         none
>>>>>>>> ---      ---
>>>>>>>> 0/7501       16.00KiB     16.00KiB         none         none
>>>>>>>> ---      ---
>>>>>>>> 0/7502       16.00KiB     16.00KiB         none         none
>>>>>>>> 255/7502 ---
>>>>>>>> 0/7503       16.00KiB     16.00KiB         none         none
>>>>>>>> 255/7503 ---
>>>>>>>
>>>>>>> This is now getting super weird now.
>>>>>>>
>>>>>>> Firstly, if there are some snapshots of subvolume 5, it should
>>>>>>> show up
>>>>>>> here, with size over several GiB.
>>>>>>>
>>>>>>> Thus there seems to be some ghost/hidden subvolumes that even don't
>>>>>>> show
>>>>>>> up in qgroup.
>>>>>>>
>>>>>>> It's possible that some qgroup is intentionally deleted, thus not
>>>>>>> being
>>>>>>> properly updated.
>>>>>>>
>>>>>>> For that case, you may want to disable qgroup, re-enable, then do a
>>>>>>> rescan: (Can all be done on the running system)
>>>>>>>
>>>>>>> # btrfs quota disable <mnt>
>>>>>>> # btrfs quota enable <mnt>
>>>>>>> # btrfs quota rescan -w <mnt>
>>>>>>>
>>>>>>> Then provide the output of 'btrfs qgroup show -prce <mnt>"
>>>>>>>
>>>>>>> If the output doesn't change at all, after the full 10min rescan,
>>>>>>> then I
>>>>>>> guess you have to dump the root tree, along with the data_reloc
>>>>>>> tree.
>>>>>>>
>>>>>>> # btrfs ins dump-tree -t root <device>
>>>>>>> # btrfs ins dump-tree -t data_reloc <device>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>> 1/0             0.00B        0.00B         none         none
>>>>>>>> ---      ---
>>>>>>>> 255/7502     16.00KiB     16.00KiB         none         none
>>>>>>>> ---      0/7502
>>>>>>>> 255/7503     16.00KiB     16.00KiB         none         none
>>>>>>>> ---      0/7503
>>>>>>>> ```
>>>>>>>>
>>>>>>>> and i try to mount with option subvolid:
>>>>>>>> ```
>>>>>>>> $ mkdir /tmp/fulldisk
>>>>>>>> $ sudo mount -o subvolid=5 /dev/sda2 /tmp/fulldisk
>>>>>>>> $ ls -lA /tmp/fulldisk
>>>>>>>> total 4
>>>>>>>> drwxr-xr-x 1 root   root   1936 May  3 21:34 bin
>>>>>>>> drwxr-xr-x 1 root   root      0 Jan 25  2020 boot
>>>>>>>> drwxr-xr-x 1 root   root   1686 Jan 20  2020 dev
>>>>>>>> drwxr-xr-x 1 wzy    wzy    5756 Jun 24 13:51 etc
>>>>>>>> drwxr-xr-x 1 root   root     22 Oct 17  2020 home
>>>>>>>> drwxr-xr-x 1 root   root   1332 May 18 14:13 lib
>>>>>>>> drwxr-xr-x 1 root   root   6606 May 18 14:13 lib64
>>>>>>>> lrwxrwxrwx 1 root   root     10 Jan 24 20:23 media -> /run/media
>>>>>>>> drwxr-xr-x 1 wzy    wzy      38 Jan 27 16:51 mnt
>>>>>>>> drwxr-xr-x 1 root   root    234 Jun 18 14:29 opt
>>>>>>>> drwxr-xr-x 1 root   root      0 Jan 20  2020 proc
>>>>>>>> drwx------ 1 wzy    wzy     536 Jun 15 20:26 root
>>>>>>>> drwxr-xr-x 1 root   root     48 May 30 14:14 run
>>>>>>>> drwxr-xr-x 1 root   root   4926 May 18 14:12 sbin
>>>>>>>> drwxr-xr-x 1 root   root     10 Jan 20  2020 sys
>>>>>>>> drwxrwxrwx 1 nobody nobody    0 Jun 18 14:34 tftproot
>>>>>>>> drwxrwxrwt 1 root   root      0 May 30 14:25 tmp
>>>>>>>> drwxr-xr-x 1 root   root    198 Mar 31 19:32 usr
>>>>>>>> drwxr-xr-x 1 root   root    150 Apr  1 18:21 var
>>>>>>>> $ sudo btrfs fi du -s /tmp/fulldisk/*
>>>>>>>>         Total   Exclusive  Set shared  Filename
>>>>>>>>      10.66MiB       0.00B    10.66MiB  /tmp/fulldisk/bin
>>>>>>>>         0.00B       0.00B       0.00B  /tmp/fulldisk/boot
>>>>>>>>         0.00B       0.00B       0.00B  /tmp/fulldisk/dev
>>>>>>>>      33.34MiB    36.00KiB    33.30MiB  /tmp/fulldisk/etc
>>>>>>>>      13.79GiB   784.05MiB    12.96GiB  /tmp/fulldisk/home
>>>>>>>>     922.28MiB       0.00B   922.28MiB  /tmp/fulldisk/lib
>>>>>>>>      23.11MiB       0.00B    23.11MiB  /tmp/fulldisk/lib64
>>>>>>>> ERROR: cannot check space of '/tmp/fulldisk/media': Inappropriate
>>>>>>>> ioctl for device
>>>>>>>>         0.00B       0.00B       0.00B  /tmp/fulldisk/mnt
>>>>>>>>      11.08GiB       0.00B    11.08GiB  /tmp/fulldisk/opt
>>>>>>>>         0.00B       0.00B       0.00B  /tmp/fulldisk/proc
>>>>>>>>      40.38MiB     4.35MiB    36.03MiB  /tmp/fulldisk/root
>>>>>>>>         0.00B       0.00B       0.00B  /tmp/fulldisk/run
>>>>>>>>      26.62MiB       0.00B    26.62MiB  /tmp/fulldisk/sbin
>>>>>>>>         0.00B       0.00B       0.00B  /tmp/fulldisk/sys
>>>>>>>>         0.00B       0.00B       0.00B  /tmp/fulldisk/tftproot
>>>>>>>>         0.00B       0.00B       0.00B  /tmp/fulldisk/tmp
>>>>>>>>      47.22GiB     1.03GiB    46.15GiB  /tmp/fulldisk/usr
>>>>>>>>       5.80GiB     4.35GiB     1.45GiB  /tmp/fulldisk/var
>>>>>>>> ```
>>>>>>>>
>>>>>>>> because media is a symbolic link so the ERROR should be normal.
>>>>>>>> according to `btrfs fi du` it looks like i only use about 80GiB.
>>>>>>>> it is
>>>>>>>> too weird.
>>>>>>>> thanks!
>>>>>>>>
>>>>>>>> On Thu, Jun 24, 2021 at 4:32 PM Graham Cobb <g.btrfs@cobb.uk.net>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On 24/06/2021 08:45, Zhenyu Wu wrote:
>>>>>>>>>> Thanks! this is some information of some programs.
>>>>>>>>>> ```
>>>>>>>>>> # boot from liveUSB
>>>>>>>>>> $ btrfs check /dev/sda2
>>>>>>>>>> [1/7] checking root items
>>>>>>>>>> [2/7] checking extents
>>>>>>>>>> [3/7] checking free space cache
>>>>>>>>>> [4/7] checking fs roots
>>>>>>>>>> [5/7] checking only csums items (without verifying data)
>>>>>>>>>> [6/7] checking root refs
>>>>>>>>>> [7/7] checking quota groups
>>>>>>>>>> # after mount /dev/sda2 to /mnt/gentoo
>>>>>>>>>
>>>>>>>>> Did you do 'mount -o subvolid=5 /dev/sda2 /mnt/gentoo' to make
>>>>>>>>> sure you
>>>>>>>>> can see all subvolumes, not just the default subvolume? I guess it
>>>>>>>>> doesn't matter for quota, but it matters if you are using tools
>>>>>>>>> like du.
>>>>>>>>>
>>>>>>>>> You don't even need to use a liveUSB. On your normal mounted
>>>>>>>>> system you
>>>>>>>>> can do...
>>>>>>>>>
>>>>>>>>> mkdir /tmp/fulldisk
>>>>>>>>> mount -o subvolid=5 /dev/sda2 /tmp/fulldisk
>>>>>>>>> ls -lA /tmp/fulldisk
>>>>>>>>>
>>>>>>>>> to see if there are other subvolumes which are not visible in /