Files/Folders invisibles with 'ls -a' but can 'cd' to folder

Files/Folders invisibles with 'ls -a' but can 'cd' to folder

[k g]

Hi


As I say in my subject, I'm facing a weird problem with my btrfs
partition (I already sent this message on reddit /r/btrfs/ )

It's in fact a btrfs partition in a raid5 synology system.



3 days ago, the volume 'crashed' (synology terms) ,however SMART data is
ok and I don't have sector relcocation errors or CRC.... I rebooted
several times , and after dozen of reboots my partition shows up , but 3
TB of 10 TB are missing, I made a scrub but it did made my missing files
appears.



desperately I made a 'cd xyz' in a directory (I remember some of the
folder names) and it works ; and inside this folder I can do "ls" and
all files and subfolders appears .

I made a copy elsewhere of some files and these ones are not corrupted
or bit roted.



I don't want to make a btrfs check --repair of course.



Is there a way to "relink" indexes/root or whatever it is called to
bring back these files/folder visible and accessible with a safe command ?

I'm planning to backup all , is 'btrfs restore' will access to these
"non visible" directories ?



"I saw similar case here : The Directory Who Wasn't There : btrfs
(reddit.com) , but I can't find a reply that solve the problem"



cdly

Re: Files/Folders invisibles with 'ls -a' but can 'cd' to folder

[Qu Wenruo]

On 2021/8/4 下午9:19, k g wrote:
> Hi
>
>
> As I say in my subject, I'm facing a weird problem with my btrfs
> partition (I already sent this message on reddit /r/btrfs/ )

Sorry, reddit is not really the go-to place for technical discussion nor
bug report.

>
> It's in fact a btrfs partition in a raid5 synology system.

We don't know how heavily backported the synology kernel is, thus it's
normally better to ask for help from synology.

>
>
>
> 3 days ago, the volume 'crashed' (synology terms) ,however SMART data is
> ok and I don't have sector relcocation errors or CRC.... I rebooted
> several times , and after dozen of reboots my partition shows up , but 3
> TB of 10 TB are missing, I made a scrub but it did made my missing files
> appears.
>
>
>
> desperately I made a 'cd xyz' in a directory (I remember some of the
> folder names) and it works ; and inside this folder I can do "ls" and
> all files and subfolders appears .
>
> I made a copy elsewhere of some files and these ones are not corrupted
> or bit roted.
>
>
>
> I don't want to make a btrfs check --repair of course.

But "btrfs check" without --repair should be the best tool to show
what's going wrong.

Alternatively, "btrfs check --mode=lowmem" could provide a better human
readable output.

>
>
>
> Is there a way to "relink" indexes/root or whatever it is called to
> bring back these files/folder visible and accessible with a safe command ?

It's not that simple, from your description, it looks like the dir has
some DIR_ITEM but no DIR_INDEX, thus it doesn't shows up in ls, but cd
still work.

This normally indicates much bigger problem.

Thanks,
Qu
>
> I'm planning to backup all , is 'btrfs restore' will access to these
> "non visible" directories ?
>
>
>
> "I saw similar case here : The Directory Who Wasn't There : btrfs
> (reddit.com) , but I can't find a reply that solve the problem"
>
>
>
> cdly
>

Re: Files/Folders invisibles with 'ls -a' but can 'cd' to folder

[k g]

Hi Qu ,


Thanks for your valuable answers, 


The synology system I'm using has btrfs tools v4. I compiled a v5
version because "btrfs check" version 4 returns "Couldn't open file system"


a little bit late, (I'm 1000 km away from my crashed server, I'm doing
all of this remotly) ,Here is some output of btrfs check I made today
(output is very long , here is some samples of the messages returned)


Opening filesystem to check...
Checking filesystem on /dev/md2
UUID: 306faa08-9e17-406b-924e-57e06e2c2763
[1/7] checking root items                      (0:03:47 elapsed, 4389038
items checked)
[2/7] checking extents                         (0:06:51 elapsed, 3092
items checked)
cache and super generation don't match, space cache will be invalidated
[3/7] checking free space cache                (0:00:00 elapsed)
[4/7] checking fs roots                        (0:00:00 elapsed, 2 items
checked)
found 196033421312 bytes used, error(s) found
total csum bytes: 15533120
total tree bytes: 50102272
total fs tree bytes: 17711104
total extent tree bytes: 31784960
btree space waste bytes: 10681343
file data blocks allocated: 0
 referenced 0


ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent
level=2 child bytenr=229931827200 child level=0
ERROR: extent[193738493952, 16384] referencer count mismatch (root: 257,
owner: 226748, offset: 0) wanted: 1, have: 0
parent transid verify failed on 229931827200 wanted 11406678 found 11406670
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent
level=2 child bytenr=229931827200 child level=0
ERROR: extent[193738510336, 16384] referencer count mismatch (root: 257,
owner: 226749, offset: 0) wanted: 1, have: 0

ERROR: extent [197244026880 16384] referencer bytenr mismatch, wanted:
197244026880, have: 229859393536
ERROR: extent [197244043264 16384] referencer bytenr mismatch, wanted:
197244043264, have: 229859393536

parent transid verify failed on 229931827200 wanted 11406678 found 11406670
Ignoring transid failure


WARNING: tree block [197151997952, 197152014336) crosses 64K page
boudnary, may cause problem for 64K page system
WARNING: tree block [197152063488, 197152079872) crosses 64K page
boudnary, may cause problem for 64K page system

Wrong key of child node/leaf, wanted: (197244633088, 169, 0), have:
(40699577, 108, 0)
Wrong generation of child node/leaf, wanted: 11406682, have: 11406670
parent transid verify failed on 229934661632 wanted 11406672 found 11406677
Ignoring transid failure

ERROR: block group[9778059280384 10737418240] used 0 but extent items
used 10735706112
ERROR: block group[9788796698624 10737418240] used 0 but extent items
used 10736922624

leaf parent key incorrect 229908168704
parent transid verify failed on 229908168704 wanted 11406668 found 11406678
Ignoring transid failure


ERROR: free space cache inode 41584067 has invalid mode: has 0100644
expect 0100600
parent transid verify failed on 229926993920 wanted 11406669 found 11406676
Ignoring transid failure
parent transid verify failed on 229958598656 wanted 11406672 found 11406678
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=229823332352 item=5 parent
level=1 child bytenr=229958598656 child level=1
ERROR: errors found in fs roots
extent buffer leak: start 229934661632 len 16384
extent buffer leak: start 229823561728 len 16384
extent buffer leak: start 229931532288 len 16384
extent buffer leak: start 229931827200 len 16384

 

By "luck" I have a sql dump that contain 80% of the paths of the lost
files, so I can make a bash or python script to recover them (by doing a
copy elsewhere or mv two times the hidden folders)

unfortunately, these path are samba paths and some of them are mangled
(I did not manage, or had the time to reverse engineer the samba source
code to rebuild linux paths from samba mangled path despite that I asked
some help in stackoverflow...)

But before starting building these scripts , I want to have  your
feedback before launching any maintenance operation, or at best a
procedure to relink these DIR_INDEX


all the best.

.k.



On 05/08/2021 00:55, Qu Wenruo wrote:
>
>
> On 2021/8/4 下午9:19, k g wrote:
>> Hi
>>
>>
>> As I say in my subject, I'm facing a weird problem with my btrfs
>> partition (I already sent this message on reddit /r/btrfs/ )
>
> Sorry, reddit is not really the go-to place for technical discussion nor
> bug report.
>
>>
>> It's in fact a btrfs partition in a raid5 synology system.
>
> We don't know how heavily backported the synology kernel is, thus it's
> normally better to ask for help from synology.
>
>>
>>
>>
>> 3 days ago, the volume 'crashed' (synology terms) ,however SMART data is
>> ok and I don't have sector relcocation errors or CRC.... I rebooted
>> several times , and after dozen of reboots my partition shows up , but 3
>> TB of 10 TB are missing, I made a scrub but it did made my missing files
>> appears.
>>
>>
>>
>> desperately I made a 'cd xyz' in a directory (I remember some of the
>> folder names) and it works ; and inside this folder I can do "ls" and
>> all files and subfolders appears .
>>
>> I made a copy elsewhere of some files and these ones are not corrupted
>> or bit roted.
>>
>>
>>
>> I don't want to make a btrfs check --repair of course.
>
> But "btrfs check" without --repair should be the best tool to show
> what's going wrong.
>
> Alternatively, "btrfs check --mode=lowmem" could provide a better human
> readable output.
>
>>
>>
>>
>> Is there a way to "relink" indexes/root or whatever it is called to
>> bring back these files/folder visible and accessible with a safe
>> command ?
>
> It's not that simple, from your description, it looks like the dir has
> some DIR_ITEM but no DIR_INDEX, thus it doesn't shows up in ls, but cd
> still work.
>
> This normally indicates much bigger problem.
>
> Thanks,
> Qu
>>
>> I'm planning to backup all , is 'btrfs restore' will access to these
>> "non visible" directories ?
>>
>>
>>
>> "I saw similar case here : The Directory Who Wasn't There : btrfs
>> (reddit.com) , but I can't find a reply that solve the problem"
>>
>>
>>
>> cdly
>>

Re: Files/Folders invisibles with 'ls -a' but can 'cd' to folder

[Qu Wenruo]

On 2021/8/12 上午5:54, k g wrote:
> Hi Qu ,
>
>
> Thanks for your valuable answers,
>
>
> The synology system I'm using has btrfs tools v4. I compiled a v5
> version because "btrfs check" version 4 returns "Couldn't open file system"
>
>
> a little bit late, (I'm 1000 km away from my crashed server, I'm doing
> all of this remotly) ,Here is some output of btrfs check I made today
> (output is very long , here is some samples of the messages returned)
>
>
> Opening filesystem to check...
> Checking filesystem on /dev/md2
> UUID: 306faa08-9e17-406b-924e-57e06e2c2763
> [1/7] checking root items                      (0:03:47 elapsed, 4389038
> items checked)
> [2/7] checking extents                         (0:06:51 elapsed, 3092
> items checked)
> cache and super generation don't match, space cache will be invalidated
> [3/7] checking free space cache                (0:00:00 elapsed)
> [4/7] checking fs roots                        (0:00:00 elapsed, 2 items
> checked)
> found 196033421312 bytes used, error(s) found
> total csum bytes: 15533120
> total tree bytes: 50102272
> total fs tree bytes: 17711104
> total extent tree bytes: 31784960
> btree space waste bytes: 10681343
> file data blocks allocated: 0
>   referenced 0
>
>
> ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent
> level=2 child bytenr=229931827200 child level=0

This is the worst case, metadata corruption between nodes and leaves.

> ERROR: extent[193738493952, 16384] referencer count mismatch (root: 257,
> owner: 226748, offset: 0) wanted: 1, have: 0
> parent transid verify failed on 229931827200 wanted 11406678 found 11406670
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent
> level=2 child bytenr=229931827200 child level=0
> ERROR: extent[193738510336, 16384] referencer count mismatch (root: 257,
> owner: 226749, offset: 0) wanted: 1, have: 0
>
> ERROR: extent [197244026880 16384] referencer bytenr mismatch, wanted:
> 197244026880, have: 229859393536
> ERROR: extent [197244043264 16384] referencer bytenr mismatch, wanted:
> 197244043264, have: 229859393536
>
> parent transid verify failed on 229931827200 wanted 11406678 found 11406670
> Ignoring transid failure

Furthermore, transid mismatch...

>
>
> WARNING: tree block [197151997952, 197152014336) crosses 64K page
> boudnary, may cause problem for 64K page system
> WARNING: tree block [197152063488, 197152079872) crosses 64K page
> boudnary, may cause problem for 64K page system
>
> Wrong key of child node/leaf, wanted: (197244633088, 169, 0), have:
> (40699577, 108, 0)
> Wrong generation of child node/leaf, wanted: 11406682, have: 11406670
> parent transid verify failed on 229934661632 wanted 11406672 found 11406677
> Ignoring transid failure
>
> ERROR: block group[9778059280384 10737418240] used 0 but extent items
> used 10735706112
> ERROR: block group[9788796698624 10737418240] used 0 but extent items
> used 10736922624
>
> leaf parent key incorrect 229908168704
> parent transid verify failed on 229908168704 wanted 11406668 found 11406678
> Ignoring transid failure
>
>
> ERROR: free space cache inode 41584067 has invalid mode: has 0100644
> expect 0100600
> parent transid verify failed on 229926993920 wanted 11406669 found 11406676
> Ignoring transid failure
> parent transid verify failed on 229958598656 wanted 11406672 found 11406678
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=229823332352 item=5 parent
> level=1 child bytenr=229958598656 child level=1
> ERROR: errors found in fs roots
> extent buffer leak: start 229934661632 len 16384
> extent buffer leak: start 229823561728 len 16384
> extent buffer leak: start 229931532288 len 16384
> extent buffer leak: start 229931827200 len 16384

Overall, the metadata is mostly screwed up, thus the DIR_ITEM/DIR_INDEX
mismatch happens.

>
>
>
> By "luck" I have a sql dump that contain 80% of the paths of the lost
> files, so I can make a bash or python script to recover them (by doing a
> copy elsewhere or mv two times the hidden folders)
>
> unfortunately, these path are samba paths and some of them are mangled
> (I did not manage, or had the time to reverse engineer the samba source
> code to rebuild linux paths from samba mangled path despite that I asked
> some help in stackoverflow...)
>
> But before starting building these scripts , I want to have  your
> feedback before launching any maintenance operation, or at best a
> procedure to relink these DIR_INDEX

It's way worse than just DIR_INDEX/DIR_ITEM missing, it's some metadata
corruption that mostly screw up the filesystem.

I would go btrfs-restore directly to salvage as much data as possible.

Thanks,
Qu

>
>
> all the best.
>
> .k.
>
>
>
> On 05/08/2021 00:55, Qu Wenruo wrote:
>>
>>
>> On 2021/8/4 下午9:19, k g wrote:
>>> Hi
>>>
>>>
>>> As I say in my subject, I'm facing a weird problem with my btrfs
>>> partition (I already sent this message on reddit /r/btrfs/ )
>>
>> Sorry, reddit is not really the go-to place for technical discussion nor
>> bug report.
>>
>>>
>>> It's in fact a btrfs partition in a raid5 synology system.
>>
>> We don't know how heavily backported the synology kernel is, thus it's
>> normally better to ask for help from synology.
>>
>>>
>>>
>>>
>>> 3 days ago, the volume 'crashed' (synology terms) ,however SMART data is
>>> ok and I don't have sector relcocation errors or CRC.... I rebooted
>>> several times , and after dozen of reboots my partition shows up , but 3
>>> TB of 10 TB are missing, I made a scrub but it did made my missing files
>>> appears.
>>>
>>>
>>>
>>> desperately I made a 'cd xyz' in a directory (I remember some of the
>>> folder names) and it works ; and inside this folder I can do "ls" and
>>> all files and subfolders appears .
>>>
>>> I made a copy elsewhere of some files and these ones are not corrupted
>>> or bit roted.
>>>
>>>
>>>
>>> I don't want to make a btrfs check --repair of course.
>>
>> But "btrfs check" without --repair should be the best tool to show
>> what's going wrong.
>>
>> Alternatively, "btrfs check --mode=lowmem" could provide a better human
>> readable output.
>>
>>>
>>>
>>>
>>> Is there a way to "relink" indexes/root or whatever it is called to
>>> bring back these files/folder visible and accessible with a safe
>>> command ?
>>
>> It's not that simple, from your description, it looks like the dir has
>> some DIR_ITEM but no DIR_INDEX, thus it doesn't shows up in ls, but cd
>> still work.
>>
>> This normally indicates much bigger problem.
>>
>> Thanks,
>> Qu
>>>
>>> I'm planning to backup all , is 'btrfs restore' will access to these
>>> "non visible" directories ?
>>>
>>>
>>>
>>> "I saw similar case here : The Directory Who Wasn't There : btrfs
>>> (reddit.com) , but I can't find a reply that solve the problem"
>>>
>>>
>>>
>>> cdly
>>>