BTRFS error: bad tree block start 0 623771648

BTRFS error: bad tree block start 0 623771648

[marcel.cochem]

Hello,

I had to shut down my PC because it hang up, after the next reboot i
got the following error:
    BTRFS error on /dev/nvme0n1p4 bad tree block start 0 623771648
    open_ctree failed
    mount wrong fs type
    bad superblock
    missing codepage or helper program, or other error.

I had no backup (schame on my, now i made a backup of my failed parition)
I can't check or rescure any data, because all btrfs-progs stop with
messages like:

    root@marcel-debug:/mnt# btrfs restore  /dev/nvme0n1p4 /mnt/1
    checksum verify failed on 623771648 found E4E3BDB6 wanted 00000000
    checksum verify failed on 623771648 found E4E3BDB6 wanted 00000000
    bytenr mismatch, want=623771648, have=0
    Couldn't read tree root

or

    root@marcel-debug:/home/marcel/Desktop/btrfs-progs#
./btrfs-find-root /dev/nvme0n1p4
    Couldn't read tree root
    Superblock thinks the generation is 82096
    Superblock thinks the level is 1

My bad installation had kernel 4.11 and my rescue installation has
also 4.11 installed. However the btrfs-progs had some problems
(endless loop for btrfs-find-root), so i decided to compile them
myself with the newest version 4.12 . However neither version works.

I am pretty sure that not all data is lost as i can grep thorugh the
100 GB SSD partition. But my question is, if there is a tool to rescue
all (intact) data and maybe have only a few corrupt files which can't
be recovered.

I put detailed information and commands outputs in pastebin:
https://pastebin.com/7kzjb4Ae

Best regards,
Marcel

Re: BTRFS error: bad tree block start 0 623771648

[Liu Bo]

On Sun, Jul 30, 2017 at 06:14:35PM +0200, marcel.cochem wrote:
> Hello,
> 
> I had to shut down my PC because it hang up, after the next reboot i
> got the following error:
>     BTRFS error on /dev/nvme0n1p4 bad tree block start 0 623771648
>     open_ctree failed
>     mount wrong fs type
>     bad superblock
>     missing codepage or helper program, or other error.
> 
> I had no backup (schame on my, now i made a backup of my failed parition)
> I can't check or rescure any data, because all btrfs-progs stop with
> messages like:
> 
>     root@marcel-debug:/mnt# btrfs restore  /dev/nvme0n1p4 /mnt/1
>     checksum verify failed on 623771648 found E4E3BDB6 wanted 00000000
>     checksum verify failed on 623771648 found E4E3BDB6 wanted 00000000
>     bytenr mismatch, want=623771648, have=0
>     Couldn't read tree root

Superblock and chunk tree root is OK, looks like the header part of
the tree root is now all-zero, but I'm unable to think of a btrfs bug
which can lead to that (if there is, it is a serious enough one), and
on ssd like disks, by default there is only one copy for metadata.

Now that all progs has been stuck in reading tree root node, one
workaround is to hijack the btrfs code to bypass that whole
check(including checksum checking) for reading metadata blocks to let
us read the block no matter what mismatch it may have and see if the
content is meaningful to us.  The function that needs to be hijacked
is btree_readpage_end_io_hook() in fs/btrfs/disk-io.c

But please note you should play this trick on the backup copy disk
you've aleady made, instead of the original one.  The best situation
would be that tree root node is the only corrupted node.  Good luck.

thanks,

-liubo

> 
> or
> 
>     root@marcel-debug:/home/marcel/Desktop/btrfs-progs#
> ./btrfs-find-root /dev/nvme0n1p4
>     Couldn't read tree root
>     Superblock thinks the generation is 82096
>     Superblock thinks the level is 1
> 
> My bad installation had kernel 4.11 and my rescue installation has
> also 4.11 installed. However the btrfs-progs had some problems
> (endless loop for btrfs-find-root), so i decided to compile them
> myself with the newest version 4.12 . However neither version works.
> 
> I am pretty sure that not all data is lost as i can grep thorugh the
> 100 GB SSD partition. But my question is, if there is a tool to rescue
> all (intact) data and maybe have only a few corrupt files which can't
> be recovered.
> 
> I put detailed information and commands outputs in pastebin:
> https://pastebin.com/7kzjb4Ae
> 
> Best regards,
> Marcel
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS error: bad tree block start 0 623771648

[Roman Mamedov]

On Mon, 31 Jul 2017 11:12:01 -0700
Liu Bo <bo.li.liu@oracle.com> wrote:

> Superblock and chunk tree root is OK, looks like the header part of
> the tree root is now all-zero, but I'm unable to think of a btrfs bug
> which can lead to that (if there is, it is a serious enough one)

I see that the FS is being mounted with "discard". So maybe it was a TRIM gone
bad (wrong location or in a wrong sequence).

Generally it appears to be not recommended to use "discard" by now (because of
its performance impact, and maybe possible issues like this), instead schedule
to call "fstrim <mountpoint>" once a day or so, and/or on boot-up.

> on ssd like disks, by default there is only one copy for metadata.

Time and time again, the default of "single" metadata for SSD is a terrible
idea. Most likely DUP metadata would save the FS in this case.

-- 
With respect,
Roman

Re: BTRFS error: bad tree block start 0 623771648

[Roman Mamedov]

On Sun, 30 Jul 2017 18:14:35 +0200
"marcel.cochem" <marcel.cochem@googlemail.com> wrote:

> I am pretty sure that not all data is lost as i can grep thorugh the
> 100 GB SSD partition. But my question is, if there is a tool to rescue
> all (intact) data and maybe have only a few corrupt files which can't
> be recovered.

There is such a tool, see https://btrfs.wiki.kernel.org/index.php/Restore

-- 
With respect,
Roman

Re: BTRFS error: bad tree block start 0 623771648

[Liu Bo]

On Tue, Aug 01, 2017 at 11:04:10AM +0500, Roman Mamedov wrote:
> On Mon, 31 Jul 2017 11:12:01 -0700
> Liu Bo <bo.li.liu@oracle.com> wrote:
> 
> > Superblock and chunk tree root is OK, looks like the header part of
> > the tree root is now all-zero, but I'm unable to think of a btrfs bug
> > which can lead to that (if there is, it is a serious enough one)
> 
> I see that the FS is being mounted with "discard". So maybe it was a TRIM gone
> bad (wrong location or in a wrong sequence).
>

By checking discard path in btrfs, looks OK to me, more likely it's
caused by problems from underlying stuff.

Thanks,

-liubo

> Generally it appears to be not recommended to use "discard" by now (because of
> its performance impact, and maybe possible issues like this), instead schedule
> to call "fstrim <mountpoint>" once a day or so, and/or on boot-up.
> 
> > on ssd like disks, by default there is only one copy for metadata.
> 
> Time and time again, the default of "single" metadata for SSD is a terrible
> idea. Most likely DUP metadata would save the FS in this case.
> 
> -- 
> With respect,
> Roman
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS error: bad tree block start 0 623771648

[Duncan]

Roman Mamedov posted on Tue, 01 Aug 2017 11:08:05 +0500 as excerpted:

> On Sun, 30 Jul 2017 18:14:35 +0200 "marcel.cochem"
> <marcel.cochem@googlemail.com> wrote:
> 
>> I am pretty sure that not all data is lost as i can grep thorugh the
>> 100 GB SSD partition. But my question is, if there is a tool to rescue
>> all (intact) data and maybe have only a few corrupt files which can't
>> be recovered.
> 
> There is such a tool, see
> https://btrfs.wiki.kernel.org/index.php/Restore

I was going to suggest that too... and even started a reply to do so... 
upon which I read a bit closer and saw he'd actually tried restore 
already...

And before you suggest it, he tried btrfs-find-root as well, and it 
didn't work either, so he can't do the advanced/technical mode of 
restore, feeding it addresses from btrfs-find-root, either. =:^(

It's in the post...

So unfortunately he's pretty much left with manual hacking and scraping, 
and that's at a level beyond what I at least am able to help him with...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS error: bad tree block start 0 623771648

[marcel.cochem]

Thanks you for the information. It looks indeed like there are some
important parts all zerod... I will try to hijack to code to get easy
access to some config directories.
I already reinstalled the Operating System, now with dup Metadata and
will have a deeper look at the discard flag.
The restore tools don't work out of the box, because like Liu Bo
mentioned, they'll check metadata and exit on an error.

Thanks for your support
marcel


On Tue, Aug 1, 2017 at 11:45 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
> On Tue, Aug 01, 2017 at 11:04:10AM +0500, Roman Mamedov wrote:
>> On Mon, 31 Jul 2017 11:12:01 -0700
>> Liu Bo <bo.li.liu@oracle.com> wrote:
>>
>> > Superblock and chunk tree root is OK, looks like the header part of
>> > the tree root is now all-zero, but I'm unable to think of a btrfs bug
>> > which can lead to that (if there is, it is a serious enough one)
>>
>> I see that the FS is being mounted with "discard". So maybe it was a TRIM gone
>> bad (wrong location or in a wrong sequence).
>>
>
> By checking discard path in btrfs, looks OK to me, more likely it's
> caused by problems from underlying stuff.
>
> Thanks,
>
> -liubo
>
>> Generally it appears to be not recommended to use "discard" by now (because of
>> its performance impact, and maybe possible issues like this), instead schedule
>> to call "fstrim <mountpoint>" once a day or so, and/or on boot-up.
>>
>> > on ssd like disks, by default there is only one copy for metadata.
>>
>> Time and time again, the default of "single" metadata for SSD is a terrible
>> idea. Most likely DUP metadata would save the FS in this case.
>>
>> --
>> With respect,
>> Roman
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html