Re: Files/Folders invisibles with 'ls -a' but can 'cd' to folder

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2021/8/12 上午5:54, k g wrote:
> Hi Qu ,
>
>
> Thanks for your valuable answers,
>
>
> The synology system I'm using has btrfs tools v4. I compiled a v5
> version because "btrfs check" version 4 returns "Couldn't open file system"
>
>
> a little bit late, (I'm 1000 km away from my crashed server, I'm doing
> all of this remotly) ,Here is some output of btrfs check I made today
> (output is very long , here is some samples of the messages returned)
>
>
> Opening filesystem to check...
> Checking filesystem on /dev/md2
> UUID: 306faa08-9e17-406b-924e-57e06e2c2763
> [1/7] checking root items                      (0:03:47 elapsed, 4389038
> items checked)
> [2/7] checking extents                         (0:06:51 elapsed, 3092
> items checked)
> cache and super generation don't match, space cache will be invalidated
> [3/7] checking free space cache                (0:00:00 elapsed)
> [4/7] checking fs roots                        (0:00:00 elapsed, 2 items
> checked)
> found 196033421312 bytes used, error(s) found
> total csum bytes: 15533120
> total tree bytes: 50102272
> total fs tree bytes: 17711104
> total extent tree bytes: 31784960
> btree space waste bytes: 10681343
> file data blocks allocated: 0
>   referenced 0
>
>
> ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent
> level=2 child bytenr=229931827200 child level=0

This is the worst case, metadata corruption between nodes and leaves.

> ERROR: extent[193738493952, 16384] referencer count mismatch (root: 257,
> owner: 226748, offset: 0) wanted: 1, have: 0
> parent transid verify failed on 229931827200 wanted 11406678 found 11406670
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent
> level=2 child bytenr=229931827200 child level=0
> ERROR: extent[193738510336, 16384] referencer count mismatch (root: 257,
> owner: 226749, offset: 0) wanted: 1, have: 0
>
> ERROR: extent [197244026880 16384] referencer bytenr mismatch, wanted:
> 197244026880, have: 229859393536
> ERROR: extent [197244043264 16384] referencer bytenr mismatch, wanted:
> 197244043264, have: 229859393536
>
> parent transid verify failed on 229931827200 wanted 11406678 found 11406670
> Ignoring transid failure

Furthermore, transid mismatch...

>
>
> WARNING: tree block [197151997952, 197152014336) crosses 64K page
> boudnary, may cause problem for 64K page system
> WARNING: tree block [197152063488, 197152079872) crosses 64K page
> boudnary, may cause problem for 64K page system
>
> Wrong key of child node/leaf, wanted: (197244633088, 169, 0), have:
> (40699577, 108, 0)
> Wrong generation of child node/leaf, wanted: 11406682, have: 11406670
> parent transid verify failed on 229934661632 wanted 11406672 found 11406677
> Ignoring transid failure
>
> ERROR: block group[9778059280384 10737418240] used 0 but extent items
> used 10735706112
> ERROR: block group[9788796698624 10737418240] used 0 but extent items
> used 10736922624
>
> leaf parent key incorrect 229908168704
> parent transid verify failed on 229908168704 wanted 11406668 found 11406678
> Ignoring transid failure
>
>
> ERROR: free space cache inode 41584067 has invalid mode: has 0100644
> expect 0100600
> parent transid verify failed on 229926993920 wanted 11406669 found 11406676
> Ignoring transid failure
> parent transid verify failed on 229958598656 wanted 11406672 found 11406678
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=229823332352 item=5 parent
> level=1 child bytenr=229958598656 child level=1
> ERROR: errors found in fs roots
> extent buffer leak: start 229934661632 len 16384
> extent buffer leak: start 229823561728 len 16384
> extent buffer leak: start 229931532288 len 16384
> extent buffer leak: start 229931827200 len 16384

Overall, the metadata is mostly screwed up, thus the DIR_ITEM/DIR_INDEX
mismatch happens.

>
>
>
> By "luck" I have a sql dump that contain 80% of the paths of the lost
> files, so I can make a bash or python script to recover them (by doing a
> copy elsewhere or mv two times the hidden folders)
>
> unfortunately, these path are samba paths and some of them are mangled
> (I did not manage, or had the time to reverse engineer the samba source
> code to rebuild linux paths from samba mangled path despite that I asked
> some help in stackoverflow...)
>
> But before starting building these scripts , I want to have  your
> feedback before launching any maintenance operation, or at best a
> procedure to relink these DIR_INDEX

It's way worse than just DIR_INDEX/DIR_ITEM missing, it's some metadata
corruption that mostly screw up the filesystem.

I would go btrfs-restore directly to salvage as much data as possible.

Thanks,
Qu

>
>
> all the best.
>
> .k.
>
>
>
> On 05/08/2021 00:55, Qu Wenruo wrote:
>>
>>
>> On 2021/8/4 下午9:19, k g wrote:
>>> Hi
>>>
>>>
>>> As I say in my subject, I'm facing a weird problem with my btrfs
>>> partition (I already sent this message on reddit /r/btrfs/ )
>>
>> Sorry, reddit is not really the go-to place for technical discussion nor
>> bug report.
>>
>>>
>>> It's in fact a btrfs partition in a raid5 synology system.
>>
>> We don't know how heavily backported the synology kernel is, thus it's
>> normally better to ask for help from synology.
>>
>>>
>>>
>>>
>>> 3 days ago, the volume 'crashed' (synology terms) ,however SMART data is
>>> ok and I don't have sector relcocation errors or CRC.... I rebooted
>>> several times , and after dozen of reboots my partition shows up , but 3
>>> TB of 10 TB are missing, I made a scrub but it did made my missing files
>>> appears.
>>>
>>>
>>>
>>> desperately I made a 'cd xyz' in a directory (I remember some of the
>>> folder names) and it works ; and inside this folder I can do "ls" and
>>> all files and subfolders appears .
>>>
>>> I made a copy elsewhere of some files and these ones are not corrupted
>>> or bit roted.
>>>
>>>
>>>
>>> I don't want to make a btrfs check --repair of course.
>>
>> But "btrfs check" without --repair should be the best tool to show
>> what's going wrong.
>>
>> Alternatively, "btrfs check --mode=lowmem" could provide a better human
>> readable output.
>>
>>>
>>>
>>>
>>> Is there a way to "relink" indexes/root or whatever it is called to
>>> bring back these files/folder visible and accessible with a safe
>>> command ?
>>
>> It's not that simple, from your description, it looks like the dir has
>> some DIR_ITEM but no DIR_INDEX, thus it doesn't shows up in ls, but cd
>> still work.
>>
>> This normally indicates much bigger problem.
>>
>> Thanks,
>> Qu
>>>
>>> I'm planning to backup all , is 'btrfs restore' will access to these
>>> "non visible" directories ?
>>>
>>>
>>>
>>> "I saw similar case here : The Directory Who Wasn't There : btrfs
>>> (reddit.com) , but I can't find a reply that solve the problem"
>>>
>>>
>>>
>>> cdly
>>>