Btrfs suddenly unmountable, open_ctree failed

Btrfs suddenly unmountable, open_ctree failed

[Mike Hartman]

Distro: Linux Mint 16
Kernel: 3.11.0-12-generic
Btrfs (used to create): 0.20-rc1 (current version in Mint repo)
Btrfs (used to troubleshoot): 3.14.2

Setup: sda6 is a luks container with btrfs inside. Btrfs has
subvolumes @ (/) and @home (/home).

I was doing nothing in particular on my laptop (normal web browsing)
when I noticed my system had gone read-only. dmesg showed an IO error
of some kind between btrfs and the underlying drive. I didn't think to
save it out on the web.

I rebooted and successfully unlocked the luks container, but the btrfs
volumes failed to mount. dmesg showed output like:

[92039.112690] device fsid b1205e06-7943-4aaf-9e13-4df0cf1394bc devid
1 transid 214454 /dev/mapper/sda6_crypt
[92039.113633] btrfs: disk space caching is enabled
[92039.116754] btrfs bad tree block start 10446918091381484860 760696832
[92039.116956] btrfs bad tree block start 7222353813307377896 760696832
[92039.141820] btrfs: open_ctree failed

I dded an image of the partition, grabbed the latest btrfs-progs from
git, and then tried every troubleshooting/recovery method I could find
online with no luck. The complete transcript is available here:

http://pastebin.com/PSHL7UxZ

Summary:

"mount -o ro,recovery", "btrfs-image", "btrfs-debug-tree",
"btrfs-zero-log", "btrfs check" and "btrfs restore" all failed.

"btrfs rescue super-recover" said the supers are fine.
"btrfs-find-root" said it found the tree root. "btrfs rescue
chunk-recover" said it was successful.

The drive itself seems to be ok, but I haven't tried any tests that
would write to it. (The original btrfs data is still there and
untouched.)

Can anyone offer any suggestions? Is this data really unrecoverable? I
have no idea what could have gone so severely wrong.

Re: Btrfs suddenly unmountable, open_ctree failed

[Chris Murphy]

On Jun 23, 2014, at 4:18 PM, Mike Hartman <mike@hartmanipulation.com> wrote:

> Can anyone offer any suggestions? Is this data really unrecoverable? I
> have no idea what could have gone so severely wrong.

"btrfs check --repair /media/mint/usb_data/sda6_check.img
"btrfs check --repair --init-csum-tree --init-extent-tree /media/mint/usb_data/sda6_check.img

I think these things are going too far on actual file system without guidance, so it's superb you imaged the original drive first. Too many people get aggressive writing to the actual drive without backup images. You have both a dd image as well as btrfs-image which is really great. Hopefully someone can help find out what happened, it seems abruptly catastrophic, but also mixes messages where some information is being found but this one:
	• read block failed check_tree_block
makes me think of partial media failure. It would be damn bad luck if your metadata is set to DUP, yet both copies are toast. Is this an HDD or SSD?

Can you post the result from smartctl -x /dev/sdX   #for the disk in question, maybe from a live system?


Chris Murphy

Re: Btrfs suddenly unmountable, open_ctree failed

[Wang Shilong]

On 06/24/2014 09:17 AM, Chris Murphy wrote:
> On Jun 23, 2014, at 4:18 PM, Mike Hartman <mike@hartmanipulation.com> wrote:
>
>> Can anyone offer any suggestions? Is this data really unrecoverable? I
>> have no idea what could have gone so severely wrong.
> "btrfs check --repair /media/mint/usb_data/sda6_check.img
> "btrfs check --repair --init-csum-tree --init-extent-tree /media/mint/usb_data/sda6_check.img
My two cents:-)

If you really want to use btrfs check  --init-csum-tree 
--init-extent-tree, i'd suggest you use
David Latest btrfs-progs branch which includes some latest bug fixes.

>
> I think these things are going too far on actual file system without guidance, so it's superb you imaged the original drive first. Too many people get aggressive writing to the actual drive without backup images. You have both a dd image as well as btrfs-image which is really great. Hopefully someone can help find out what happened, it seems abruptly catastrophic, but also mixes messages where some information is being found but this one:
> 	• read block failed check_tree_block
> makes me think of partial media failure. It would be damn bad luck if your metadata is set to DUP, yet both copies are toast. Is this an HDD or SSD?
>
> Can you post the result from smartctl -x /dev/sdX   #for the disk in question, maybe from a live system?
>
>
> Chris Murphy
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Btrfs suddenly unmountable, open_ctree failed

[Mike Hartman]

I have a dd image, but not a btrfs-image. I ran the btrfs-image
command, but it threw the same errors as everything else and generated
a 0 byte file.

I agree that it SOUNDS like some kind of media failure, but if so it
seems odd to me that I was able to dd the entire partition with no
read errors. Even if there was something wrong with the drive that
prevented writing you'd think the ability to read it all would result
in a recoverable image.

smartctl -x /dev/sda (shortly after running smartctl -t long
/dev/sda): http://pastebin.com/kqSMZXsj

(The older failed tests are from several months ago. Completely
zeroing out the drive cleared them up and I haven't seen any issues
since.)

On Mon, Jun 23, 2014 at 9:17 PM, Chris Murphy <lists@colorremedies.com> wrote:
>
> On Jun 23, 2014, at 4:18 PM, Mike Hartman <mike@hartmanipulation.com> wrote:
>
>> Can anyone offer any suggestions? Is this data really unrecoverable? I
>> have no idea what could have gone so severely wrong.
>
> "btrfs check --repair /media/mint/usb_data/sda6_check.img
> "btrfs check --repair --init-csum-tree --init-extent-tree /media/mint/usb_data/sda6_check.img
>
> I think these things are going too far on actual file system without guidance, so it's superb you imaged the original drive first. Too many people get aggressive writing to the actual drive without backup images. You have both a dd image as well as btrfs-image which is really great. Hopefully someone can help find out what happened, it seems abruptly catastrophic, but also mixes messages where some information is being found but this one:
>         * read block failed check_tree_block
> makes me think of partial media failure. It would be damn bad luck if your metadata is set to DUP, yet both copies are toast. Is this an HDD or SSD?
>
> Can you post the result from smartctl -x /dev/sdX   #for the disk in question, maybe from a live system?
>
>
> Chris Murphy
>

Re: Btrfs suddenly unmountable, open_ctree failed

[Mike Hartman]

I have no particular desire to use it. I just tried everything else
first and thought it was worth a shot.

If you think that version would help, can you point me to the git
repo? The one I grabbed was
git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git.

On Mon, Jun 23, 2014 at 10:20 PM, Wang Shilong
<wangsl.fnst@cn.fujitsu.com> wrote:
> On 06/24/2014 09:17 AM, Chris Murphy wrote:
>>
>> On Jun 23, 2014, at 4:18 PM, Mike Hartman <mike@hartmanipulation.com>
>> wrote:
>>
>>> Can anyone offer any suggestions? Is this data really unrecoverable? I
>>> have no idea what could have gone so severely wrong.
>>
>> "btrfs check --repair /media/mint/usb_data/sda6_check.img
>> "btrfs check --repair --init-csum-tree --init-extent-tree
>> /media/mint/usb_data/sda6_check.img
>
> My two cents:-)
>
> If you really want to use btrfs check  --init-csum-tree --init-extent-tree,
> i'd suggest you use
> David Latest btrfs-progs branch which includes some latest bug fixes.
>
>>
>> I think these things are going too far on actual file system without
>> guidance, so it's superb you imaged the original drive first. Too many
>> people get aggressive writing to the actual drive without backup images. You
>> have both a dd image as well as btrfs-image which is really great. Hopefully
>> someone can help find out what happened, it seems abruptly catastrophic, but
>> also mixes messages where some information is being found but this one:
>>         * read block failed check_tree_block
>> makes me think of partial media failure. It would be damn bad luck if your
>> metadata is set to DUP, yet both copies are toast. Is this an HDD or SSD?
>>
>> Can you post the result from smartctl -x /dev/sdX   #for the disk in
>> question, maybe from a live system?
>>
>>
>> Chris Murphy
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: Btrfs suddenly unmountable, open_ctree failed

[Wang Shilong]

Hi Mike,

On 06/24/2014 11:04 AM, Mike Hartman wrote:
> I have no particular desire to use it. I just tried everything else
> first and thought it was worth a shot.
>
> If you think that version would help, can you point me to the git
> repo? The one I grabbed was
> git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git.
Of course,  You could pull from the following url:

https://github.com/kdave/btrfs-progs.git  integration-20140619

>
> On Mon, Jun 23, 2014 at 10:20 PM, Wang Shilong
> <wangsl.fnst@cn.fujitsu.com> wrote:
>> On 06/24/2014 09:17 AM, Chris Murphy wrote:
>>> On Jun 23, 2014, at 4:18 PM, Mike Hartman <mike@hartmanipulation.com>
>>> wrote:
>>>
>>>> Can anyone offer any suggestions? Is this data really unrecoverable? I
>>>> have no idea what could have gone so severely wrong.
>>> "btrfs check --repair /media/mint/usb_data/sda6_check.img
>>> "btrfs check --repair --init-csum-tree --init-extent-tree
>>> /media/mint/usb_data/sda6_check.img
>> My two cents:-)
>>
>> If you really want to use btrfs check  --init-csum-tree --init-extent-tree,
>> i'd suggest you use
>> David Latest btrfs-progs branch which includes some latest bug fixes.
>>
>>> I think these things are going too far on actual file system without
>>> guidance, so it's superb you imaged the original drive first. Too many
>>> people get aggressive writing to the actual drive without backup images. You
>>> have both a dd image as well as btrfs-image which is really great. Hopefully
>>> someone can help find out what happened, it seems abruptly catastrophic, but
>>> also mixes messages where some information is being found but this one:
>>>          * read block failed check_tree_block
>>> makes me think of partial media failure. It would be damn bad luck if your
>>> metadata is set to DUP, yet both copies are toast. Is this an HDD or SSD?
>>>
>>> Can you post the result from smartctl -x /dev/sdX   #for the disk in
>>> question, maybe from a live system?
>>>
>>>
>>> Chris Murphy
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> .
>

Re: Btrfs suddenly unmountable, open_ctree failed

[Chris Murphy]

On Jun 23, 2014, at 8:58 PM, Mike Hartman <mike@hartmanipulation.com> wrote:

> I have a dd image, but not a btrfs-image. I ran the btrfs-image
> command, but it threw the same errors as everything else and generated
> a 0 byte file.
> 
> I agree that it SOUNDS like some kind of media failure, but if so it
> seems odd to me that I was able to dd the entire partition with no
> read errors. Even if there was something wrong with the drive that
> prevented writing you'd think the ability to read it all would result
> in a recoverable image.

I've read of too many SSD failure cases to trust a graceful failure of an SSD. I guess I don't really trust an HDD either but at least they don't self destruct upon reaching end of life as apparently some SSDs do:
http://techreport.com/review/26523/the-ssd-endurance-experiment-casualties-on-the-way-to-a-petabyte

Anyway, it could be ECC failure where it says pass but it's actually corrupt, in which case it's silent data corruption which neither triggers ECC errors or read failures. You just get bad data. And really bad luck if this happens with Btrfs metadata that isn't DUP but is fundamental for mounting and/or repairing the system so it can be mounted.

Of course it could just be a bug so it's worth trying David's integration branch.


	• Firmware Version: 0006

Firmware 0007 is current for this SSD.


	• 173 Wear_Leveling_Count     PO--CK   086   086   000    -    728
	•   1  0x018  6      49304625928  Logical Sectors Written
	• 202 Perc_Rated_Life_Used    ---RC-   086   086   000    -    14

Those are all reasonable.

	• 181 Non4k_Aligned_Access    -O---K   100   100   000    -    36 0 35

Probably unrelated, but that's a curious attribute and value.

	• 199 UDMA_CRC_Error_Count    -OS-CK   100   100   000    -    15

That's not good in that it means interface problems have happened at some point. But they can happen and just not get caught, which results in corruption. Drive ECC will not correct these problems. So how many good writes on the way to the drive but were corrupted by the time they got there? Obviously there's no way to know this from available information.

 6  0x008  4             9821  Number of Hardware Resets

Why is the hardware being reset so many times?


Chris Murphy

Re: Btrfs suddenly unmountable, open_ctree failed

[Mike Hartman]

>> Of course it could just be a bug so it's worth trying David's integration branch.
>
>I'll try that shortly.
>
>>         * Firmware Version: 0006
>>
>> Firmware 0007 is current for this SSD.
>
>I assume that's probably not something I should mess with right now
though, right?
>
>>  6  0x008  4             9821  Number of Hardware Resets
>>
>> Why is the hardware being reset so many times?
>
>What exactly is that stat measuring? This is a 3 year old laptop with
>a 3 year old SSD. The whole system certainly gets restarted more often
>than a desktop or server would, but not an average of 9 times a day.
>Probably not even a maximum of 9 times a day. Could that reset be
>something related to acpi?

Actually, now that I think about it, that may be related to the errors
I saw months ago. I had some bad blocks or something, and when the
system attempted to access them I would get a bunch of errors in the
logs. I believe one of the things the system would do was attempt to
reset the drive connection (maybe multiple times). So that could have
driven that number up substantially. The "Lifetime Power-On Resets" is
only 1033, which seems about right for the age of the drive.

I zeroed out the drive and ran every smartctl test on it I could find
and it never threw any more errors. I really pounded it, a good solid
week or so of scans and data writes. Finally decided that perhaps
those blocks went bad but didn't get properly marked as bad and
removed from active duty until I zeroed everything out.

Whatever the underlying issue was this time, I don't think it's
exactly the same, because in that previous scenario the problem blocks
would consistently throw errors. I couldn't get a clean dd image of
the drive. I had to use ddrescue and live with a few small gaps. But I
was still able to mount that rescued image (it was ext4) and get
pretty much all of the data back. This time I'm encountering no read
errors at the drive level and had no trouble creating the dd image.

I'll see if I can track down any of the smartctl results from back
then and compare the "Number of Hardware Resets" value, as well as
UDMA_CRC_Error_Count.

Re: Btrfs suddenly unmountable, open_ctree failed

[Chris Murphy]

On Jun 23, 2014, at 10:34 PM, Mike Hartman <mike@hartmanipulation.com> wrote:

>> 
>> Firmware 0007 is current for this SSD.
> 
> I assume that's probably not something I should mess with right now
> though, right?

I would deal with that later. A firmware change now might make things worse if you care about this file system or any data on the drive. But seeing as it "improves reliability" I have no way of knowing if it's related.

> Actually, now that I think about it, that may be related to the errors
> I saw months ago. I had some bad blocks or something, and when the
> system attempted to access them I would get a bunch of errors in the
> logs. I believe one of the things the system would do was attempt to
> reset the drive connection (maybe multiple times).

Yes Ok that's normal.

> I zeroed out the drive and ran every smartctl test on it I could find
> and it never threw any more errors.

Zeroing SSDs isn't a good way to do it. Use ATA Secure Erase instead. The drive is overprovisioned, so there are pages without LBAs assigned, which means they can't be written to by software. Plus "zeros" make SSD pages full of zeros, rather than being empty and ready to be written to. ATA Secure Erase is supposed to make them empty (write ready) and does it for all pages.

http://mackonsti.wordpress.com/2011/11/22/ssd-secure-erase-ata-command/

> I'll see if I can track down any of the smartctl results from back
> then and compare the "Number of Hardware Resets" value, as well as
> UDMA_CRC_Error_Count.

It could be a loose or intermittent connection; maybe they are one off events as the laptop was in motion. Hard to say.




Chris Murphy

Re: Btrfs suddenly unmountable, open_ctree failed

[Mike Hartman]

>>> My two cents:-)
>>>
>>> If you really want to use btrfs check  --init-csum-tree
>>> --init-extent-tree,
>>> i'd suggest you use
>>> David Latest btrfs-progs branch which includes some latest bug fixes.
>>
>> I have no particular desire to use it. I just tried everything else
>> first and thought it was worth a shot.
>>
>> If you think that version would help, can you point me to the git
>> repo? The one I grabbed was
>> git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git.
>
> Of course,  You could pull from the following url:
>
> https://github.com/kdave/btrfs-progs.git  integration-20140619

Thanks. I pulled that version and retried everything in my original
transcript, including the "btrfs check --init-csum-tree
--init-extent-tree". Results are identical across the board.

Re: Btrfs suddenly unmountable, open_ctree failed

[Mike Hartman]

>> I zeroed out the drive and ran every smartctl test on it I could find
>> and it never threw any more errors.
>
> Zeroing SSDs isn't a good way to do it. Use ATA Secure Erase instead. The drive is overprovisioned, so there are pages without LBAs assigned, which means they can't be written to by software. Plus "zeros" make SSD pages full of zeros, rather than being empty and ready to be written to. ATA Secure Erase is supposed to make them empty (write ready) and does it for all pages.
>
> http://mackonsti.wordpress.com/2011/11/22/ssd-secure-erase-ata-command/

Thanks, that's good information to have. I'll definitely do that next time.

So, setting aside potential issues with the drive itself... Anything
else you can recommend in regards to accessing the data in my image?
If I can't repair/rebuild the whole file system I can live with that,
but at the very least I have some specific files I'd like to salvage.
I've got a backup setup in place, but the files in question are pretty
big and recently modified, so the latest and greatest hadn't made it
into the backup yet.

Even if the metadata got garbled somehow, the actual data should still
be buried in there somewhere, right? Are any forensic tools up to the
task of finding it? Or is their operation so entwined with the details
of the file system that they wouldn't do me any good?

Re: Btrfs suddenly unmountable, open_ctree failed

[Duncan]

Chris Murphy posted on Mon, 23 Jun 2014 23:19:37 -0600 as excerpted:

>> I zeroed out the drive and ran every smartctl test on it I could find
>> and it never threw any more errors.
> 
> Zeroing SSDs isn't a good way to do it. Use ATA Secure Erase instead.
> The drive is overprovisioned, so there are pages without LBAs assigned,
> which means they can't be written to by software. Plus "zeros" make SSD
> pages full of zeros, rather than being empty and ready to be written to.
> ATA Secure Erase is supposed to make them empty (write ready) and does
> it for all pages.
> 
> http://mackonsti.wordpress.com/2011/11/22/ssd-secure-erase-ata-command/

Tho in this case, the bad section was obviously assigned as it was 
triggering errors, and writing zeros to the entire thing might just have 
triggered the drive to cycle the bad area out of use, replacing it from 
the reserves.

So while zeroing an SSD is a bad idea in general, it still might be 
worthwhile to try in a case like this, particularly if after a a secure-
erase command the media is still giving errors.

IOW, YMMV. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Btrfs suddenly unmountable, open_ctree failed

[Chris Murphy]

On Jun 23, 2014, at 11:39 PM, Mike Hartman <mike@hartmanipulation.com> wrote:
>> 
>> https://github.com/kdave/btrfs-progs.git  integration-20140619
> 
> Thanks. I pulled that version and retried everything in my original
> transcript, including the "btrfs check --init-csum-tree
> --init-extent-tree". Results are identical across the board.

Does this version's btrfs-image allow you to make an image of the file system?


Chris Murphy

Re: Btrfs suddenly unmountable, open_ctree failed

[Chris Murphy]

On Jun 23, 2014, at 11:49 PM, Mike Hartman <mike@hartmanipulation.com> wrote:

>>> I zeroed out the drive and ran every smartctl test on it I could find
>>> and it never threw any more errors.
>> 
>> Zeroing SSDs isn't a good way to do it. Use ATA Secure Erase instead. The drive is overprovisioned, so there are pages without LBAs assigned, which means they can't be written to by software. Plus "zeros" make SSD pages full of zeros, rather than being empty and ready to be written to. ATA Secure Erase is supposed to make them empty (write ready) and does it for all pages.
>> 
>> http://mackonsti.wordpress.com/2011/11/22/ssd-secure-erase-ata-command/
> 
> Thanks, that's good information to have. I'll definitely do that next time.
> 
> So, setting aside potential issues with the drive itself... Anything
> else you can recommend in regards to accessing the data in my image?

I have no ideas.


> If I can't repair/rebuild the whole file system I can live with that,
> but at the very least I have some specific files I'd like to salvage.

https://btrfs.wiki.kernel.org/index.php/Restore

Your superblocks are good according to btrfs rescue super-recover. And various tree roots are found by btrfs-find-root including a tree root that also matches the latest latest generation:
	• Super think's the tree root is at 763420672, chunk root 20971520
	• Found tree root at 763420672 gen 214454 level 1

And yet btrfs restore fails. I'm not sure why but try:

btrfs restore -t 763420672 /dev/mapper/sda6_crypt /media/mint/usb_data/sda6_btrfs_recovery/

If that command doesn't quickly fail or spit out a bunch of messages, then it's probably restoring files. If you confirm in another shell that it is restoring, cancel the restore, and then figure out how to use the --path-regex option with the above command to just get the files you're looking for. If it fails, consider -i to ignore errors. Hopefully any errors stopping restore won't apply to the files you're looking for.

If this still fails, then go back up one generation in your btrfs-find-root list and use that tree root:
btrfs restore -t 762122240 -i --path-regex '<blah>' /dev/mapper/sda6_crypt /media/mint/usb_data/sda6_btrfs_recovery/

And so on up the list. Maybe one of them will work (?).

> Even if the metadata got garbled somehow, the actual data should still
> be buried in there somewhere, right?

Yeah but without the metadata you don't know where the data is or how to put its extents back together again if it's in multiple extents. Any file with portions being changed will have fragments on a COW file system. And while I don't know what happened or what's corrupt it's possible some tiny bit of really critical metadata is toast and that prevents locating other metadata needed to locate data.

> Are any forensic tools up to the
> task of finding it? Or is their operation so entwined with the details
> of the file system that they wouldn't do me any good?

I think you're in #btrfs channel on freenode territory. Try the above, and if all fail, head to #btrfs.


Chris Murphy

Re: Btrfs suddenly unmountable, open_ctree failed

[Mike Hartman]

> Does this version's btrfs-image allow you to make an image of the file system?

Nope, same errors and no output.

> https://btrfs.wiki.kernel.org/index.php/Restore
>
> Your superblocks are good according to btrfs rescue super-recover. And various tree roots are found by btrfs-find-root including a tree root that also matches the latest latest generation:
>         * Super think's the tree root is at 763420672, chunk root 20971520
>         * Found tree root at 763420672 gen 214454 level 1
>
> And yet btrfs restore fails. I'm not sure why but try:
>
> btrfs restore -t 763420672 /dev/mapper/sda6_crypt /media/mint/usb_data/sda6_btrfs_recovery/
>

First of all, a new log of all my restore attempts is here:
http://pastebin.com/7QcNMahG

This gives me the same output as omitting the -t parameter. (Which it
should since that's the root it detects on its own.)

> If that command doesn't quickly fail or spit out a bunch of messages, then it's probably restoring files. If you confirm in another shell that it is restoring, cancel the restore, and then figure out how to use the --path-regex option with the above command to just get the files you're looking for. If it fails, consider -i to ignore errors. Hopefully any errors stopping restore won't apply to the files you're looking for.
>

Adding -i gets it a bit further. It doesn't quit when the @home volume
throws an error. It proceeds to try the @ volume, which also throws an
error. And then it mentions that it's skipping the snapshots since I
didn't specify -s:

.....
read block failed check_tree_block
Error reading subvolume
/media/mint/usb_data/sda6_btrfs_restore_t_763420672/@home:
18446744073709551611
Check tree block failed, want=756572160, have=6711487709046970189
Check tree block failed, want=756572160, have=6711487709046970189
Check tree block failed, want=756572160, have=5305625995405044180
Check tree block failed, want=756572160, have=5305625995405044180
Check tree block failed, want=756572160, have=5305625995405044180
read block failed check_tree_block
Error reading subvolume
/media/mint/usb_data/sda6_btrfs_restore_t_763420672/@:
18446744073709551611
Skipping snapshot @_2014-05-22_pre-Firefox-29-upgrade
Skipping snapshot @home_2014-05-22_pre-Firefox-29-upgrade
Skipping snapshot @_2014-05-29-pre-Add-Intel-Graphics-Repo

> If this still fails, then go back up one generation in your btrfs-find-root list and use that tree root:
> btrfs restore -t 762122240 -i --path-regex '<blah>' /dev/mapper/sda6_crypt /media/mint/usb_data/sda6_btrfs_recovery/
>
> And so on up the list. Maybe one of them will work (?).

Tried them all, but none worked. They all gave the same error, except
the generation varied:

parent transid verify failed on 761872384 wanted 214454 found 214447
parent transid verify failed on 761872384 wanted 214454 found 214447
parent transid verify failed on 761872384 wanted 214454 found 214447
parent transid verify failed on 761872384 wanted 214454 found 214447
Ignoring transid failure
Couldn't setup extent tree
Couldn't setup device tree
Couldn't read fs root: -2

It seems like even though I'm explicitly telling it to use tree X
which is generation Y, it's failing because the generation isn't the Z
it expects. Which seems odd, because if you're passing the -t
parameter it's because you specifically DON'T want the most recent
tree. In which case the generation of the -t you specify will never
match the generation it's looking for. If it really does work that way
(generation of the tree you specify must match the generation it
thinks the system should be on) I don't see how the -t option could
ever work at all.

Anyway, I DID have some luck with "-is". It still refuses to load the
current state of the volumes, but it's able to recover a lot of stuff
from the 3 snapshots I have in there. Unfortunately, the recent
content I'm most interested in rescuing is in @home, and the only
snapshot I have of that is several weeks older than that content. I
usually only snapshot the root when I'm upgrading something or making
other system changes.

To be clear, I have two goals:

    - Retrieve an intact version of @ with as few errors as possible,
even if it's quite old. This is my / partition, and if I can get it
back on the drive I'll have a working system again. I can always
re-upgrade/install whatever I've touched in the last few weeks. I
think what I retrieved from the snapshots should suffice for this,
assuming there's not too much missing.

    - Retrieve as much of @home from the last week as possible. This
can tolerate more errors than @ since it's mostly data files, but the
data I'm looking for is also more recent.

> I think you're in #btrfs channel on freenode territory. Try the above, and if all fail, head to #btrfs.

I think that's the point I'm at with the @home retrieval. I've got the
dd image, those files aren't going anywhere and I don't need them
immediately. I will see if anyone in #btrfs can help. (Although if
anyone on the list has further ideas, please chime in! I'm listening.)

I do have a few more questions about how to leverage my recovered
snapshots into a working root though.

- Most of the output from "-isvvv" consists of "Restoring..." lines
for various files it was able to recover. But I have 4264 lines that
just look like "offset is <some number>". Do those represent files
that should be there that it couldn't retrieve? Some other error? I'm
trying to get a sense of what might be missing.

- The output shows files that were recovered and (I assume this is
what the "offset" lines are) recovery attempts that failed. Is there
any possibility there were other files in there that it just plain
didn't detect in the tree at all, so that it wouldn't even throw an
error for them? Or does the fact that the tree is obviously intact
enough to traverse rule that out? Again, trying to get a sense of how
complete the recovery was.

- Assuming I have a directory that represents an intact @ (whether
it's a single one of the snapshots or a merged version of both of them
to fill in missing files) how would I go about getting that back on
the original drive so I can boot into it? Would I just
delete/overwrite the existing btrfs fs on sda6, create a btrfs fs with
@ and @home volumes from scratch, and then just copy my directory onto
@?

Thanks for all the help.

Re: Btrfs suddenly unmountable, open_ctree failed

[Chris Murphy]

On Jun 24, 2014, at 11:53 PM, Mike Hartman <mike@hartmanipulation.com> wrote:

>> Does this version's btrfs-image allow you to make an image of the file system?
> 
> Nope, same errors and no output.
> 
>> https://btrfs.wiki.kernel.org/index.php/Restore
>> 
>> Your superblocks are good according to btrfs rescue super-recover. And various tree roots are found by btrfs-find-root including a tree root that also matches the latest latest generation:
>>        * Super think's the tree root is at 763420672, chunk root 20971520
>>        * Found tree root at 763420672 gen 214454 level 1
>> 
>> And yet btrfs restore fails. I'm not sure why but try:
>> 
>> btrfs restore -t 763420672 /dev/mapper/sda6_crypt /media/mint/usb_data/sda6_btrfs_recovery/
>> 
> 
> First of all, a new log of all my restore attempts is here:
> http://pastebin.com/7QcNMahG
> 
> This gives me the same output as omitting the -t parameter. (Which it
> should since that's the root it detects on its own.)
> 
>> If that command doesn't quickly fail or spit out a bunch of messages, then it's probably restoring files. If you confirm in another shell that it is restoring, cancel the restore, and then figure out how to use the --path-regex option with the above command to just get the files you're looking for. If it fails, consider -i to ignore errors. Hopefully any errors stopping restore won't apply to the files you're looking for.
>> 
> 
> Adding -i gets it a bit further. It doesn't quit when the @home volume
> throws an error. It proceeds to try the @ volume, which also throws an
> error. And then it mentions that it's skipping the snapshots since I
> didn't specify -s:
> 
> .....
> read block failed check_tree_block
> Error reading subvolume
> /media/mint/usb_data/sda6_btrfs_restore_t_763420672/@home:
> 18446744073709551611
> Check tree block failed, want=756572160, have=6711487709046970189
> Check tree block failed, want=756572160, have=6711487709046970189
> Check tree block failed, want=756572160, have=5305625995405044180
> Check tree block failed, want=756572160, have=5305625995405044180
> Check tree block failed, want=756572160, have=5305625995405044180
> read block failed check_tree_block
> Error reading subvolume
> /media/mint/usb_data/sda6_btrfs_restore_t_763420672/@:
> 18446744073709551611
> Skipping snapshot @_2014-05-22_pre-Firefox-29-upgrade
> Skipping snapshot @home_2014-05-22_pre-Firefox-29-upgrade
> Skipping snapshot @_2014-05-29-pre-Add-Intel-Graphics-Repo
> 
>> If this still fails, then go back up one generation in your btrfs-find-root list and use that tree root:
>> btrfs restore -t 762122240 -i --path-regex '<blah>' /dev/mapper/sda6_crypt /media/mint/usb_data/sda6_btrfs_recovery/
>> 
>> And so on up the list. Maybe one of them will work (?).
> 
> Tried them all, but none worked. They all gave the same error, except
> the generation varied:
> 
> parent transid verify failed on 761872384 wanted 214454 found 214447
> parent transid verify failed on 761872384 wanted 214454 found 214447
> parent transid verify failed on 761872384 wanted 214454 found 214447
> parent transid verify failed on 761872384 wanted 214454 found 214447
> Ignoring transid failure
> Couldn't setup extent tree
> Couldn't setup device tree
> Couldn't read fs root: -2
> 
> It seems like even though I'm explicitly telling it to use tree X
> which is generation Y, it's failing because the generation isn't the Z
> it expects. Which seems odd, because if you're passing the -t
> parameter it's because you specifically DON'T want the most recent
> tree. In which case the generation of the -t you specify will never
> match the generation it's looking for. If it really does work that way
> (generation of the tree you specify must match the generation it
> thinks the system should be on) I don't see how the -t option could
> ever work at all.
> 
> Anyway, I DID have some luck with "-is". It still refuses to load the
> current state of the volumes, but it's able to recover a lot of stuff
> from the 3 snapshots I have in there. Unfortunately, the recent
> content I'm most interested in rescuing is in @home, and the only
> snapshot I have of that is several weeks older than that content. I
> usually only snapshot the root when I'm upgrading something or making
> other system changes.
> 
> To be clear, I have two goals:
> 
>    - Retrieve an intact version of @ with as few errors as possible,
> even if it's quite old. This is my / partition, and if I can get it
> back on the drive I'll have a working system again. I can always
> re-upgrade/install whatever I've touched in the last few weeks. I
> think what I retrieved from the snapshots should suffice for this,
> assuming there's not too much missing.
> 
>    - Retrieve as much of @home from the last week as possible. This
> can tolerate more errors than @ since it's mostly data files, but the
> data I'm looking for is also more recent.
> 
>> I think you're in #btrfs channel on freenode territory. Try the above, and if all fail, head to #btrfs.
> 
> I think that's the point I'm at with the @home retrieval. I've got the
> dd image, those files aren't going anywhere and I don't need them
> immediately. I will see if anyone in #btrfs can help. (Although if
> anyone on the list has further ideas, please chime in! I'm listening.)
> 
> I do have a few more questions about how to leverage my recovered
> snapshots into a working root though.
> 
> - Most of the output from "-isvvv" consists of "Restoring..." lines
> for various files it was able to recover. But I have 4264 lines that
> just look like "offset is <some number>". Do those represent files
> that should be there that it couldn't retrieve? Some other error? I'm
> trying to get a sense of what might be missing.
> 
> - The output shows files that were recovered and (I assume this is
> what the "offset" lines are) recovery attempts that failed. Is there
> any possibility there were other files in there that it just plain
> didn't detect in the tree at all, so that it wouldn't even throw an
> error for them? Or does the fact that the tree is obviously intact
> enough to traverse rule that out? Again, trying to get a sense of how
> complete the recovery was.
> 
> - Assuming I have a directory that represents an intact @ (whether
> it's a single one of the snapshots or a merged version of both of them
> to fill in missing files) how would I go about getting that back on
> the original drive so I can boot into it? Would I just
> delete/overwrite the existing btrfs fs on sda6, create a btrfs fs with
> @ and @home volumes from scratch, and then just copy my directory onto
> @?

All of the questions are good ones I'd ask too, but I don't know the answer to any of them.

I don't know all states of this file system, and copies you have. Right now the earliest copy is obviously broken, and the latest copy is probably more broken because at the least its csum tree has been blown away meaning there's no checksums to confirm whether any data extracted/copied from the file system is OK. You'd just have to open the file and look at it to see if it behaves as it should, and even then depending on what kind of file it is, corruption may not be obvious immediately. So… catch22. But seems to me in any case sda6 needs to be reformatted, the only question is if you reformat it Btrfs or something else (honestly - it doesn't matter whether it was the chicken soup that made you sick, the association is understandable).

I'm very interesting in how this turns out. Is it a Btrfs bug, is it a hardware bug, is it some bad/unlucky collision of more than one problem? Is it possible metadata DUP might have changed the outcome? If so are we better off in the short term using metadata DUP on SSDs? Or is the hindsight takeway "metadata DUP isn't worth it on SSD, better is more frequent backups than what was done" and you just have to weigh whether that's practical for your workflow.


Chris Murphy

Re: Btrfs suddenly unmountable, open_ctree failed

[Mike Hartman]

> I don't know all states of this file system, and copies you have. Right now the earliest copy is obviously broken, and the latest copy is probably more broken because at the least its csum tree has been blown away meaning there's no checksums to confirm whether any data extracted/copied from the file system is OK. You'd just have to open the file and look at it to see if it behaves as it should, and even then depending on what kind of file it is, corruption may not be obvious immediately. So... catch22.

I'm not sure I follow you. I have a clean dd image, and I made a fresh
copy of it every time I tried something new, so no copy should be any
more broken than whatever the current recovery operation did to it.
(There shouldn't be a distinction between earliest and latest if I
understand what you mean by "copy".)

I get all those tree errors when it tries to access the "active"
versions of the @ and @home volumes. But by the time it starts
restoring the snapshots I don't see those anymore. So my
interpretation of the output was that the trees rooted at @ and @home
are messed up, but the tree structures for the snapshots are ok. Am I
misunderstanding that output, or are we both equally in the dark about
what it's telling us?

Setting aside the tree integrity issue, does anyone know what
condition causes that "offset is <number>" output during a restore
operation? I've seen several wiki pages and posts about how to use the
restore feature, but nothing that details what's going on under the
hood or how to interpret the output. If it's not complaining about the
tree, and there are none of those "offset" errors, would the
assumption be that the entire snapshot was recovered successfully?

>But seems to me in any case sda6 needs to be reformatted, the only question is if you reformat it Btrfs or something else (honestly - it doesn't matter whether it was the chicken soup that made you sick, the association is understandable).

Yep. I'll probably stick with Btrfs and just try to improve my backup
strategy. I think I need to start a) replicating my snapshots to
another file system and b) more aggressively snapshotting @home. I've
been avoiding that because I have so many big files that come and go
so often there the snapshots would take up a lot of space. But maybe a
single rolling snapshot hourly or daily would make this kind of issue
easier to deal with.

> I'm very interesting in how this turns out. Is it a Btrfs bug, is it a hardware bug, is it some bad/unlucky collision of more than one problem? Is it possible metadata DUP might have changed the outcome? If so are we better off in the short term using metadata DUP on SSDs? Or is the hindsight takeway "metadata DUP isn't worth it on SSD, better is more frequent backups than what was done" and you just have to weigh whether that's practical for your workflow.

I'll definitely report back here if #btrfs provides any insight. And
anyone else who comes across this and has suggestions, please, chime
in!

Re: Btrfs suddenly unmountable, open_ctree failed

[Chris Murphy]

On Jun 25, 2014, at 1:32 PM, Mike Hartman <mike@hartmanipulation.com> wrote:

>> I don't know all states of this file system, and copies you have. Right now the earliest copy is obviously broken, and the latest copy is probably more broken because at the least its csum tree has been blown away meaning there's no checksums to confirm whether any data extracted/copied from the file system is OK. You'd just have to open the file and look at it to see if it behaves as it should, and even then depending on what kind of file it is, corruption may not be obvious immediately. So... catch22.
> 
> I'm not sure I follow you. I have a clean dd image, and I made a fresh
> copy of it every time I tried something new, so no copy should be any
> more broken than whatever the current recovery operation did to it.

e.g. --init-csum blows away checksums, so any file read will show checksum mismatches. And I'm not sure the state of btrfs check --repair but in the recent past it has sometimes made problems worse. So that's what I mean. If you're making new copies each time, then of course the original isn't affected…


> (There shouldn't be a distinction between earliest and latest if I
> understand what you mean by "copy".)
> 
> I get all those tree errors when it tries to access the "active"
> versions of the @ and @home volumes. But by the time it starts
> restoring the snapshots I don't see those anymore. So my
> interpretation of the output was that the trees rooted at @ and @home
> are messed up, but the tree structures for the snapshots are ok. Am I
> misunderstanding that output, or are we both equally in the dark about
> what it's telling us?

I'm definitely in the dark about what it's saying.


Chris Murphy