* Files/Folders invisibles with 'ls -a' but can 'cd' to folder @ 2021-08-04 13:19 k g 2021-08-04 22:55 ` Qu Wenruo 0 siblings, 1 reply; 4+ messages in thread From: k g @ 2021-08-04 13:19 UTC (permalink / raw) To: linux-btrfs Hi As I say in my subject, I'm facing a weird problem with my btrfs partition (I already sent this message on reddit /r/btrfs/ ) It's in fact a btrfs partition in a raid5 synology system. 3 days ago, the volume 'crashed' (synology terms) ,however SMART data is ok and I don't have sector relcocation errors or CRC.... I rebooted several times , and after dozen of reboots my partition shows up , but 3 TB of 10 TB are missing, I made a scrub but it did made my missing files appears. desperately I made a 'cd xyz' in a directory (I remember some of the folder names) and it works ; and inside this folder I can do "ls" and all files and subfolders appears . I made a copy elsewhere of some files and these ones are not corrupted or bit roted. I don't want to make a btrfs check --repair of course. Is there a way to "relink" indexes/root or whatever it is called to bring back these files/folder visible and accessible with a safe command ? I'm planning to backup all , is 'btrfs restore' will access to these "non visible" directories ? "I saw similar case here : The Directory Who Wasn't There : btrfs (reddit.com) , but I can't find a reply that solve the problem" cdly ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Files/Folders invisibles with 'ls -a' but can 'cd' to folder 2021-08-04 13:19 Files/Folders invisibles with 'ls -a' but can 'cd' to folder k g @ 2021-08-04 22:55 ` Qu Wenruo 2021-08-11 21:54 ` k g 0 siblings, 1 reply; 4+ messages in thread From: Qu Wenruo @ 2021-08-04 22:55 UTC (permalink / raw) To: k g, linux-btrfs On 2021/8/4 下午9:19, k g wrote: > Hi > > > As I say in my subject, I'm facing a weird problem with my btrfs > partition (I already sent this message on reddit /r/btrfs/ ) Sorry, reddit is not really the go-to place for technical discussion nor bug report. > > It's in fact a btrfs partition in a raid5 synology system. We don't know how heavily backported the synology kernel is, thus it's normally better to ask for help from synology. > > > > 3 days ago, the volume 'crashed' (synology terms) ,however SMART data is > ok and I don't have sector relcocation errors or CRC.... I rebooted > several times , and after dozen of reboots my partition shows up , but 3 > TB of 10 TB are missing, I made a scrub but it did made my missing files > appears. > > > > desperately I made a 'cd xyz' in a directory (I remember some of the > folder names) and it works ; and inside this folder I can do "ls" and > all files and subfolders appears . > > I made a copy elsewhere of some files and these ones are not corrupted > or bit roted. > > > > I don't want to make a btrfs check --repair of course. But "btrfs check" without --repair should be the best tool to show what's going wrong. Alternatively, "btrfs check --mode=lowmem" could provide a better human readable output. > > > > Is there a way to "relink" indexes/root or whatever it is called to > bring back these files/folder visible and accessible with a safe command ? It's not that simple, from your description, it looks like the dir has some DIR_ITEM but no DIR_INDEX, thus it doesn't shows up in ls, but cd still work. This normally indicates much bigger problem. Thanks, Qu > > I'm planning to backup all , is 'btrfs restore' will access to these > "non visible" directories ? > > > > "I saw similar case here : The Directory Who Wasn't There : btrfs > (reddit.com) , but I can't find a reply that solve the problem" > > > > cdly > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Files/Folders invisibles with 'ls -a' but can 'cd' to folder 2021-08-04 22:55 ` Qu Wenruo @ 2021-08-11 21:54 ` k g 2021-08-12 1:23 ` Qu Wenruo 0 siblings, 1 reply; 4+ messages in thread From: k g @ 2021-08-11 21:54 UTC (permalink / raw) To: Qu Wenruo, linux-btrfs Hi Qu , Thanks for your valuable answers, The synology system I'm using has btrfs tools v4. I compiled a v5 version because "btrfs check" version 4 returns "Couldn't open file system" a little bit late, (I'm 1000 km away from my crashed server, I'm doing all of this remotly) ,Here is some output of btrfs check I made today (output is very long , here is some samples of the messages returned) Opening filesystem to check... Checking filesystem on /dev/md2 UUID: 306faa08-9e17-406b-924e-57e06e2c2763 [1/7] checking root items (0:03:47 elapsed, 4389038 items checked) [2/7] checking extents (0:06:51 elapsed, 3092 items checked) cache and super generation don't match, space cache will be invalidated [3/7] checking free space cache (0:00:00 elapsed) [4/7] checking fs roots (0:00:00 elapsed, 2 items checked) found 196033421312 bytes used, error(s) found total csum bytes: 15533120 total tree bytes: 50102272 total fs tree bytes: 17711104 total extent tree bytes: 31784960 btree space waste bytes: 10681343 file data blocks allocated: 0 referenced 0 ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent level=2 child bytenr=229931827200 child level=0 ERROR: extent[193738493952, 16384] referencer count mismatch (root: 257, owner: 226748, offset: 0) wanted: 1, have: 0 parent transid verify failed on 229931827200 wanted 11406678 found 11406670 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent level=2 child bytenr=229931827200 child level=0 ERROR: extent[193738510336, 16384] referencer count mismatch (root: 257, owner: 226749, offset: 0) wanted: 1, have: 0 ERROR: extent [197244026880 16384] referencer bytenr mismatch, wanted: 197244026880, have: 229859393536 ERROR: extent [197244043264 16384] referencer bytenr mismatch, wanted: 197244043264, have: 229859393536 parent transid verify failed on 229931827200 wanted 11406678 found 11406670 Ignoring transid failure WARNING: tree block [197151997952, 197152014336) crosses 64K page boudnary, may cause problem for 64K page system WARNING: tree block [197152063488, 197152079872) crosses 64K page boudnary, may cause problem for 64K page system Wrong key of child node/leaf, wanted: (197244633088, 169, 0), have: (40699577, 108, 0) Wrong generation of child node/leaf, wanted: 11406682, have: 11406670 parent transid verify failed on 229934661632 wanted 11406672 found 11406677 Ignoring transid failure ERROR: block group[9778059280384 10737418240] used 0 but extent items used 10735706112 ERROR: block group[9788796698624 10737418240] used 0 but extent items used 10736922624 leaf parent key incorrect 229908168704 parent transid verify failed on 229908168704 wanted 11406668 found 11406678 Ignoring transid failure ERROR: free space cache inode 41584067 has invalid mode: has 0100644 expect 0100600 parent transid verify failed on 229926993920 wanted 11406669 found 11406676 Ignoring transid failure parent transid verify failed on 229958598656 wanted 11406672 found 11406678 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=229823332352 item=5 parent level=1 child bytenr=229958598656 child level=1 ERROR: errors found in fs roots extent buffer leak: start 229934661632 len 16384 extent buffer leak: start 229823561728 len 16384 extent buffer leak: start 229931532288 len 16384 extent buffer leak: start 229931827200 len 16384 By "luck" I have a sql dump that contain 80% of the paths of the lost files, so I can make a bash or python script to recover them (by doing a copy elsewhere or mv two times the hidden folders) unfortunately, these path are samba paths and some of them are mangled (I did not manage, or had the time to reverse engineer the samba source code to rebuild linux paths from samba mangled path despite that I asked some help in stackoverflow...) But before starting building these scripts , I want to have your feedback before launching any maintenance operation, or at best a procedure to relink these DIR_INDEX all the best. .k. On 05/08/2021 00:55, Qu Wenruo wrote: > > > On 2021/8/4 下午9:19, k g wrote: >> Hi >> >> >> As I say in my subject, I'm facing a weird problem with my btrfs >> partition (I already sent this message on reddit /r/btrfs/ ) > > Sorry, reddit is not really the go-to place for technical discussion nor > bug report. > >> >> It's in fact a btrfs partition in a raid5 synology system. > > We don't know how heavily backported the synology kernel is, thus it's > normally better to ask for help from synology. > >> >> >> >> 3 days ago, the volume 'crashed' (synology terms) ,however SMART data is >> ok and I don't have sector relcocation errors or CRC.... I rebooted >> several times , and after dozen of reboots my partition shows up , but 3 >> TB of 10 TB are missing, I made a scrub but it did made my missing files >> appears. >> >> >> >> desperately I made a 'cd xyz' in a directory (I remember some of the >> folder names) and it works ; and inside this folder I can do "ls" and >> all files and subfolders appears . >> >> I made a copy elsewhere of some files and these ones are not corrupted >> or bit roted. >> >> >> >> I don't want to make a btrfs check --repair of course. > > But "btrfs check" without --repair should be the best tool to show > what's going wrong. > > Alternatively, "btrfs check --mode=lowmem" could provide a better human > readable output. > >> >> >> >> Is there a way to "relink" indexes/root or whatever it is called to >> bring back these files/folder visible and accessible with a safe >> command ? > > It's not that simple, from your description, it looks like the dir has > some DIR_ITEM but no DIR_INDEX, thus it doesn't shows up in ls, but cd > still work. > > This normally indicates much bigger problem. > > Thanks, > Qu >> >> I'm planning to backup all , is 'btrfs restore' will access to these >> "non visible" directories ? >> >> >> >> "I saw similar case here : The Directory Who Wasn't There : btrfs >> (reddit.com) , but I can't find a reply that solve the problem" >> >> >> >> cdly >> ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Files/Folders invisibles with 'ls -a' but can 'cd' to folder 2021-08-11 21:54 ` k g @ 2021-08-12 1:23 ` Qu Wenruo 0 siblings, 0 replies; 4+ messages in thread From: Qu Wenruo @ 2021-08-12 1:23 UTC (permalink / raw) To: k g, linux-btrfs On 2021/8/12 上午5:54, k g wrote: > Hi Qu , > > > Thanks for your valuable answers, > > > The synology system I'm using has btrfs tools v4. I compiled a v5 > version because "btrfs check" version 4 returns "Couldn't open file system" > > > a little bit late, (I'm 1000 km away from my crashed server, I'm doing > all of this remotly) ,Here is some output of btrfs check I made today > (output is very long , here is some samples of the messages returned) > > > Opening filesystem to check... > Checking filesystem on /dev/md2 > UUID: 306faa08-9e17-406b-924e-57e06e2c2763 > [1/7] checking root items (0:03:47 elapsed, 4389038 > items checked) > [2/7] checking extents (0:06:51 elapsed, 3092 > items checked) > cache and super generation don't match, space cache will be invalidated > [3/7] checking free space cache (0:00:00 elapsed) > [4/7] checking fs roots (0:00:00 elapsed, 2 items > checked) > found 196033421312 bytes used, error(s) found > total csum bytes: 15533120 > total tree bytes: 50102272 > total fs tree bytes: 17711104 > total extent tree bytes: 31784960 > btree space waste bytes: 10681343 > file data blocks allocated: 0 > referenced 0 > > > ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent > level=2 child bytenr=229931827200 child level=0 This is the worst case, metadata corruption between nodes and leaves. > ERROR: extent[193738493952, 16384] referencer count mismatch (root: 257, > owner: 226748, offset: 0) wanted: 1, have: 0 > parent transid verify failed on 229931827200 wanted 11406678 found 11406670 > Ignoring transid failure > ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent > level=2 child bytenr=229931827200 child level=0 > ERROR: extent[193738510336, 16384] referencer count mismatch (root: 257, > owner: 226749, offset: 0) wanted: 1, have: 0 > > ERROR: extent [197244026880 16384] referencer bytenr mismatch, wanted: > 197244026880, have: 229859393536 > ERROR: extent [197244043264 16384] referencer bytenr mismatch, wanted: > 197244043264, have: 229859393536 > > parent transid verify failed on 229931827200 wanted 11406678 found 11406670 > Ignoring transid failure Furthermore, transid mismatch... > > > WARNING: tree block [197151997952, 197152014336) crosses 64K page > boudnary, may cause problem for 64K page system > WARNING: tree block [197152063488, 197152079872) crosses 64K page > boudnary, may cause problem for 64K page system > > Wrong key of child node/leaf, wanted: (197244633088, 169, 0), have: > (40699577, 108, 0) > Wrong generation of child node/leaf, wanted: 11406682, have: 11406670 > parent transid verify failed on 229934661632 wanted 11406672 found 11406677 > Ignoring transid failure > > ERROR: block group[9778059280384 10737418240] used 0 but extent items > used 10735706112 > ERROR: block group[9788796698624 10737418240] used 0 but extent items > used 10736922624 > > leaf parent key incorrect 229908168704 > parent transid verify failed on 229908168704 wanted 11406668 found 11406678 > Ignoring transid failure > > > ERROR: free space cache inode 41584067 has invalid mode: has 0100644 > expect 0100600 > parent transid verify failed on 229926993920 wanted 11406669 found 11406676 > Ignoring transid failure > parent transid verify failed on 229958598656 wanted 11406672 found 11406678 > Ignoring transid failure > ERROR: child eb corrupted: parent bytenr=229823332352 item=5 parent > level=1 child bytenr=229958598656 child level=1 > ERROR: errors found in fs roots > extent buffer leak: start 229934661632 len 16384 > extent buffer leak: start 229823561728 len 16384 > extent buffer leak: start 229931532288 len 16384 > extent buffer leak: start 229931827200 len 16384 Overall, the metadata is mostly screwed up, thus the DIR_ITEM/DIR_INDEX mismatch happens. > > > > By "luck" I have a sql dump that contain 80% of the paths of the lost > files, so I can make a bash or python script to recover them (by doing a > copy elsewhere or mv two times the hidden folders) > > unfortunately, these path are samba paths and some of them are mangled > (I did not manage, or had the time to reverse engineer the samba source > code to rebuild linux paths from samba mangled path despite that I asked > some help in stackoverflow...) > > But before starting building these scripts , I want to have your > feedback before launching any maintenance operation, or at best a > procedure to relink these DIR_INDEX It's way worse than just DIR_INDEX/DIR_ITEM missing, it's some metadata corruption that mostly screw up the filesystem. I would go btrfs-restore directly to salvage as much data as possible. Thanks, Qu > > > all the best. > > .k. > > > > On 05/08/2021 00:55, Qu Wenruo wrote: >> >> >> On 2021/8/4 下午9:19, k g wrote: >>> Hi >>> >>> >>> As I say in my subject, I'm facing a weird problem with my btrfs >>> partition (I already sent this message on reddit /r/btrfs/ ) >> >> Sorry, reddit is not really the go-to place for technical discussion nor >> bug report. >> >>> >>> It's in fact a btrfs partition in a raid5 synology system. >> >> We don't know how heavily backported the synology kernel is, thus it's >> normally better to ask for help from synology. >> >>> >>> >>> >>> 3 days ago, the volume 'crashed' (synology terms) ,however SMART data is >>> ok and I don't have sector relcocation errors or CRC.... I rebooted >>> several times , and after dozen of reboots my partition shows up , but 3 >>> TB of 10 TB are missing, I made a scrub but it did made my missing files >>> appears. >>> >>> >>> >>> desperately I made a 'cd xyz' in a directory (I remember some of the >>> folder names) and it works ; and inside this folder I can do "ls" and >>> all files and subfolders appears . >>> >>> I made a copy elsewhere of some files and these ones are not corrupted >>> or bit roted. >>> >>> >>> >>> I don't want to make a btrfs check --repair of course. >> >> But "btrfs check" without --repair should be the best tool to show >> what's going wrong. >> >> Alternatively, "btrfs check --mode=lowmem" could provide a better human >> readable output. >> >>> >>> >>> >>> Is there a way to "relink" indexes/root or whatever it is called to >>> bring back these files/folder visible and accessible with a safe >>> command ? >> >> It's not that simple, from your description, it looks like the dir has >> some DIR_ITEM but no DIR_INDEX, thus it doesn't shows up in ls, but cd >> still work. >> >> This normally indicates much bigger problem. >> >> Thanks, >> Qu >>> >>> I'm planning to backup all , is 'btrfs restore' will access to these >>> "non visible" directories ? >>> >>> >>> >>> "I saw similar case here : The Directory Who Wasn't There : btrfs >>> (reddit.com) , but I can't find a reply that solve the problem" >>> >>> >>> >>> cdly >>> ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-08-12 1:23 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-08-04 13:19 Files/Folders invisibles with 'ls -a' but can 'cd' to folder k g 2021-08-04 22:55 ` Qu Wenruo 2021-08-11 21:54 ` k g 2021-08-12 1:23 ` Qu Wenruo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).