Mount issue, mount /dev/sdc2: can't read superblock

Mount issue, mount /dev/sdc2: can't read superblock

[Peter Chant]

I managed to break my root partition today.  Playing with GPU
passthrough to a second graphics card, unsuccessfully, I suspect lead to
some lockups and/or unclean mounts.

I've Googled a bit and tried a number of things stopping just before
'btrfs check --repair'.  I'm running kernel 4.19.10.  I now have the
latest version of btrfs-progs, but I do admit to 'having a go' with
btrfs-progs 4.4.something from a Mint installation.

I've tried btrfs-find-root - and got the following:

btrfs-find-root /dev/sdc2
Superblock thinks the generation is 793794
Superblock thinks the level is 1
Found tree root at 1113905790976 gen 7937947 level 1

(transcribed by reading off PC and typing into laptop, any errors are mine)

Not sure what to do with the above info.  Should I just reformat and
re-install (inc backups) or is there something worth trying?

Pete

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Chris Murphy]

On Thu, Dec 20, 2018 at 3:23 PM Peter Chant <pete@petezilla.co.uk> wrote:
>
> I managed to break my root partition today.  Playing with GPU
> passthrough to a second graphics card, unsuccessfully, I suspect lead to
> some lockups and/or unclean mounts.

Should not matter, in theory.


> I've Googled a bit and tried a number of things stopping just before
> 'btrfs check --repair'.  I'm running kernel 4.19.10.  I now have the
> latest version of btrfs-progs, but I do admit to 'having a go' with
> btrfs-progs 4.4.something from a Mint installation.
>
> I've tried btrfs-find-root - and got the following:
>
> btrfs-find-root /dev/sdc2
> Superblock thinks the generation is 793794
> Superblock thinks the level is 1
> Found tree root at 1113905790976 gen 7937947 level 1

For all supers to be off by nearly 50 generations is really weird.
That's a lot of transactions to happen, and fail, and only for the
most recent super commits to happen successfully.

What do you get for ' btrfs rescue super -v <dev>'? That's a read only
command, I wonder if all the supers are really valid. And if they are,
then I'd like to see 'btrfs insp dump-s -f <dev>' and see if there's a
log tree. And then also the output from 'btrfs check --mode=lowmem
<dev>' which is also read-only, don't use --repair unless a dev
recommends it.


-- 
Chris Murphy

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Peter Chant]

On 12/21/2018 10:25 PM, Chris Murphy wrote:
> On Thu, Dec 20, 2018 at 3:23 PM Peter Chant <pete@petezilla.co.uk> wrote:
>>
>> I managed to break my root partition today.  Playing with GPU
>> passthrough to a second graphics card, unsuccessfully, I suspect lead to
>> some lockups and/or unclean mounts.
> 
> Should not matter, in theory.

Up until now it has seemed to be rather robust in this respect.  Any I
don't know if it was an unclean shutdown or something else.

I think lesson learned is to have a 'doing' system which I leave alone
and a 'play'  system that does not cause inconvenience if it breaks.
However, that take up extra space and gives two PCs to update...

> What do you get for ' btrfs rescue super -v <dev>'? That's a read only
> command, I wonder if all the supers are really valid. And if they are,
> then I'd like to see 'btrfs insp dump-s -f <dev>' and see if there's a
> log tree. And then also the output from 'btrfs check --mode=lowmem
> <dev>' which is also read-only, don't use --repair unless a dev
> recommends it.
> 
> 

Had to put the disk into another machine to do this.  Results below:


btrfs rescue super -v /dev/sdb2

All Devices:
	Device: id = 2, name = /dev/sdb2

Before Recovering:
	[All good supers]:
		device name = /dev/sdb2
		superblock bytenr = 65536

		device name = /dev/sdb2
		superblock bytenr = 67108864

		device name = /dev/sdb2
		superblock bytenr = 274877906944

	[All bad supers]:

All supers are valid, no need to recover


btrfs insp dump-s -f <dev>

superblock: bytenr=65536, device=/dev/sdb2
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0x939edd8c [match]
bytenr			65536
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			6496aabd-d6aa-49e0-96ca-e49c316edd8e
label			desk-system
generation		7937947
root			1113905790976
sys_array_size		97
chunk_root_generation	7743856
root_level		1
chunk_root		1066627579904
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		375809638400
bytes_used		352796758016
sectorsize		4096
nodesize		16384
leafsize (deprecated)		16384
stripesize		4096
root_dir		6
num_devices		1
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x6b
			( MIXED_BACKREF |
			  DEFAULT_SUBVOL |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF )
cache_generation	7937937
uuid_tree_generation	7937937
dev_item.uuid		d10d601a-54d6-4dce-9e2b-bb90f37c9b84
dev_item.fsid		6496aabd-d6aa-49e0-96ca-e49c316edd8e [match]
dev_item.type		0
dev_item.total_bytes	375809638400
dev_item.bytes_used	375808589824
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		2
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 1066627563520)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 2 offset 106301489152
			dev_uuid d10d601a-54d6-4dce-9e2b-bb90f37c9b84
backup_roots[4]:
	backup 0:
		backup_tree_root:	1113909100544	gen: 7937935	level: 1
		backup_chunk_root:	1066627579904	gen: 7743856	level: 1
		backup_extent_root:	1113909166080	gen: 7937935	level: 2
		backup_fs_root:		1113906577408	gen: 7762749	level: 0
		backup_dev_root:	1113911820288	gen: 7937933	level: 1
		backup_csum_root:	1113909755904	gen: 7937935	level: 2
		backup_total_bytes:	375809638400
		backup_bytes_used:	352796151808
		backup_num_devices:	1

	backup 1:
		backup_tree_root:	1113907347456	gen: 7937936	level: 1
		backup_chunk_root:	1066627579904	gen: 7743856	level: 1
		backup_extent_root:	1113906987008	gen: 7937936	level: 2
		backup_fs_root:		1113906921472	gen: 7937936	level: 0
		backup_dev_root:	1113907412992	gen: 7937936	level: 1
		backup_csum_root:	1113907658752	gen: 7937936	level: 2
		backup_total_bytes:	375809638400
		backup_bytes_used:	352796151808
		backup_num_devices:	1

	backup 2:
		backup_tree_root:	1113911951360	gen: 7937937	level: 1
		backup_chunk_root:	1066627579904	gen: 7743856	level: 1
		backup_extent_root:	1113911967744	gen: 7937937	level: 2
		backup_fs_root:		1113906921472	gen: 7937936	level: 0
		backup_dev_root:	1113907412992	gen: 7937936	level: 1
		backup_csum_root:	1113912492032	gen: 7937937	level: 2
		backup_total_bytes:	375809638400
		backup_bytes_used:	352796151808
		backup_num_devices:	1

	backup 3:
		backup_tree_root:	1113907494912	gen: 7937934	level: 1
		backup_chunk_root:	1066627579904	gen: 7743856	level: 1
		backup_extent_root:	1113907544064	gen: 7937934	level: 2
		backup_fs_root:		1113906577408	gen: 7762749	level: 0
		backup_dev_root:	1113911820288	gen: 7937933	level: 1
		backup_csum_root:	1113907855360	gen: 7937934	level: 2
		backup_total_bytes:	375809638400
		backup_bytes_used:	352796151808
		backup_num_devices:	1


btrfs check --mode=lowmem > <dev>

OK, this one threw an error:
[1/7] checking root items
Error: could not find extent items for root 290
ERROR: failed to repair root items: No such file or directory

And then gave this output:
Opening filesystem to check...
Checking filesystem on /dev/sdb2
UUID: 6496aabd-d6aa-49e0-96ca-e49c316edd8e


I'm getting suspicious of the drive as when I was trying the various
btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
I also have a separate basic install on ext4 on the same disk.  Though
e2fsck shows no errors and mounts fine I cannot log into that install.
Maybe a coincidence, but too many bad things thrown up make me
suspicious.  Whatever is happening this seems to be really fighting me.

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Chris Murphy]

On Sat, Dec 22, 2018 at 10:22 AM Peter Chant <pete@petezilla.co.uk> wrote:

> btrfs rescue super -v /dev/sdb2
...
> All supers are valid, no need to recover
>
>
> btrfs insp dump-s -f <dev>
...
> generation              7937947
...
>         backup 0:
>                 backup_tree_root:       1113909100544   gen: 7937935    level: 1
...
>         backup 1:
>                 backup_tree_root:       1113907347456   gen: 7937936    level: 1
...
>         backup 2:
>                 backup_tree_root:       1113911951360   gen: 7937937    level: 1
...
>         backup 3:
>                 backup_tree_root:       1113907494912   gen: 7937934    level: 1
...


The kernel wrote out three valid checksummed supers, with what seems
to be a rather significant sanity violation. The super generation and
tree root address do not match any of the backup tree roots. The
*current* tree root is supposed to be in one of the backups as well.

Qu, any idea how this is even theoretically possible? Bit flip right
before the super is computed and checksummed? Seems like some kind of
corruption before checksum is computed.


> I'm getting suspicious of the drive as when I was trying the various
> btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
> I also have a separate basic install on ext4 on the same disk.  Though
> e2fsck shows no errors and mounts fine I cannot log into that install.
> Maybe a coincidence, but too many bad things thrown up make me
> suspicious.  Whatever is happening this seems to be really fighting me.

I'm not sure how even a bad device accounts for the super generation
and backup mismatches. That's damn strange.

If you get bored with the back and forth and just want to give up,
that's fine. I suggest that if you have the time and space, to take a
btrfs-image in case Qu or some other developer wants to look at this
file system at some point. The btrfs-image is a read only process, can
be set to scrub filenames, and only contains metadata. Size of the
resulting file is around 1/2 of the size of metadata, when doing
'btrfs filesystem usage' or 'btrfs filesystem df'. So you'll need that
much free space to direct the command to.

btrfs-image -ss -c9 -t4 <devicetoimage> pathtofile

It might fail, if so you can try adding -w and see if that helps.

There is no log listed in the super so zero-log isn't indicated, and
also tells me there were no fsync's still flushing at the time of the
crash. The loss should be at most a minute of data, not an
inconsistent file system that can't be mounted anymore. Pretty weird.

What were your mount options? Defaults? Anything custom like discard,
commit=, notreelog? Any non-default mount options themselves would not
be the cause of the problem, but might suggest partial ideas for what
might have happened.



-- 
Chris Murphy

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3787 bytes --]



On 2018/12/24 上午8:58, Chris Murphy wrote:
> On Sat, Dec 22, 2018 at 10:22 AM Peter Chant <pete@petezilla.co.uk> wrote:
> 
>> btrfs rescue super -v /dev/sdb2
> ...
>> All supers are valid, no need to recover
>>
>>
>> btrfs insp dump-s -f <dev>
> ...
>> generation              7937947
> ...
>>         backup 0:
>>                 backup_tree_root:       1113909100544   gen: 7937935    level: 1
> ...
>>         backup 1:
>>                 backup_tree_root:       1113907347456   gen: 7937936    level: 1
> ...
>>         backup 2:
>>                 backup_tree_root:       1113911951360   gen: 7937937    level: 1
> ...
>>         backup 3:
>>                 backup_tree_root:       1113907494912   gen: 7937934    level: 1
> ...
> 
> 
> The kernel wrote out three valid checksummed supers, with what seems
> to be a rather significant sanity violation. The super generation and
> tree root address do not match any of the backup tree roots. The
> *current* tree root is supposed to be in one of the backups as well.

Oh, I missed this.

Indeed, super generation should match one one backup root.
There are cases where we won't record backup roots, but that's only for
fsync() calls, and for such case it should only update log_root, not
root generation.
(But I'm not that familiar with fsync() nor log tree codes, I could be
totally wrong)

But at least, its log_root is 0, so it's less possible to be caused by
fsync() routine.

> 
> Qu, any idea how this is even theoretically possible? Bit flip right
> before the super is computed and checksummed?

7937947 = 0x791f9b
7937937 = 0x791f91

The last low half byte, it's 0xb vs 0x1
0xb = 1011
0x1 = 0001

2 bits flipped. I'm not so sure if it's possible.

> Seems like some kind of
> corruption before checksum is computed.
> 
> 
>> I'm getting suspicious of the drive as when I was trying the various
>> btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
>> I also have a separate basic install on ext4 on the same disk.  Though
>> e2fsck shows no errors and mounts fine I cannot log into that install.
>> Maybe a coincidence, but too many bad things thrown up make me
>> suspicious.  Whatever is happening this seems to be really fighting me.
> 
> I'm not sure how even a bad device accounts for the super generation
> and backup mismatches. That's damn strange.

Yes, indeed.

I'd recommend to use "btrfs check -r 1113911951360" to verify if it's
only superblock generation corrupted.

Thanks,
Qu
> 
> If you get bored with the back and forth and just want to give up,
> that's fine. I suggest that if you have the time and space, to take a
> btrfs-image in case Qu or some other developer wants to look at this
> file system at some point. The btrfs-image is a read only process, can
> be set to scrub filenames, and only contains metadata. Size of the
> resulting file is around 1/2 of the size of metadata, when doing
> 'btrfs filesystem usage' or 'btrfs filesystem df'. So you'll need that
> much free space to direct the command to.
> 
> btrfs-image -ss -c9 -t4 <devicetoimage> pathtofile
> 
> It might fail, if so you can try adding -w and see if that helps.
> 
> There is no log listed in the super so zero-log isn't indicated, and
> also tells me there were no fsync's still flushing at the time of the
> crash. The loss should be at most a minute of data, not an
> inconsistent file system that can't be mounted anymore. Pretty weird.
> 
> What were your mount options? Defaults? Anything custom like discard,
> commit=, notreelog? Any non-default mount options themselves would not
> be the cause of the problem, but might suggest partial ideas for what
> might have happened.
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Peter Chant]

On 12/24/18 2:00 AM, Qu Wenruo wrote:

> I'd recommend to use "btrfs check -r 1113911951360" to verify if it's
> only superblock generation corrupted.

(cancelled  "btrfs-image -ss -c9 -t4 -w /dev/sdd2
/mnt/backup/btrfs_issue_dec_2018/btrfs_root_image_error_with_w_20181224.img"

as it said it was generating garbage).

OK, running the command:

bash-4.3# btrfs check -r 1113911951360 /dev/sdd2
Opening filesystem to check...
parent transid verify failed on 1113911951360 wanted 7937947 found 7937937
parent transid verify failed on 1113911951360 wanted 7937947 found 7937937
Ignoring transid failure
parent transid verify failed on 1113906003968 wanted 7937933 found 7937942
parent transid verify failed on 1113906003968 wanted 7937933 found 7937942
Ignoring transid failure
ERROR: cannot open file system
bash-4.3#


Not a dev so don't know what that means, but I'm assuming it is not good.


Pete

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Peter Chant]

On 12/24/18 12:58 AM, Chris Murphy wrote:
> On Sat, Dec 22, 2018 at 10:22 AM Peter Chant <pete@petezilla.co.uk> wrote:
> 
>> btrfs rescue super -v /dev/sdb2
> ...
>> All supers are valid, no need to recover
>>
>>
>> btrfs insp dump-s -f <dev>
> ...
>> generation              7937947
> ...
>>         backup 0:
>>                 backup_tree_root:       1113909100544   gen: 7937935    level: 1
> ...
>>         backup 1:
>>                 backup_tree_root:       1113907347456   gen: 7937936    level: 1
> ...
>>         backup 2:
>>                 backup_tree_root:       1113911951360   gen: 7937937    level: 1
> ...
>>         backup 3:
>>                 backup_tree_root:       1113907494912   gen: 7937934    level: 1
> ...
> 
> 
> The kernel wrote out three valid checksummed supers, with what seems
> to be a rather significant sanity violation. The super generation and
> tree root address do not match any of the backup tree roots. The
> *current* tree root is supposed to be in one of the backups as well.
> 

I wonder if this is a result of my trying to fix things?  E.g. btrfs
rescue super-recover or my attempts using the tools (and kernel) in Mint
18.1 at one point?

I must admit, early on I had assumed that either this file system was a
simple fix or was completely trashed, so I thought I'd have a quick go
at fixing it, or wipe it and start again.  But then I seemed to get
close with only the one error, but unmountable.


> Qu, any idea how this is even theoretically possible? Bit flip right
> before the super is computed and checksummed? Seems like some kind of
> corruption before checksum is computed.
> 
> 
>> I'm getting suspicious of the drive as when I was trying the various
>> btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
>> I also have a separate basic install on ext4 on the same disk.  Though
>> e2fsck shows no errors and mounts fine I cannot log into that install.
>> Maybe a coincidence, but too many bad things thrown up make me
>> suspicious.  Whatever is happening this seems to be really fighting me.
> 
> I'm not sure how even a bad device accounts for the super generation
> and backup mismatches. That's damn strange.

I'm less suspicious of the drive now.  I've been using an ext4 partition
on the same drive for a few days now, having reinstalled on that and
everything _seems_ fine.  Mind you, apart from usb sticks, I've not
experienced a ssd failure.  Perhaps my hdd failure experience is not
relevent, i.e. they work until they start throwing errors and then
rapidly fail?


> 
> If you get bored with the back and forth and just want to give up,
> that's fine. I suggest that if you have the time and space, to take a
> btrfs-image in case Qu or some other developer wants to look at this
> file system at some point. The btrfs-image is a read only process, can
> be set to scrub filenames, and only contains metadata. Size of the
> resulting file is around 1/2 of the size of metadata, when doing
> 'btrfs filesystem usage' or 'btrfs filesystem df'. So you'll need that
> much free space to direct the command to.
> 
> btrfs-image -ss -c9 -t4 <devicetoimage> pathtofile

Just done that:
bash-4.3# btrfs-image -ss -c9 -t4 /dev/sdd2
/mnt/backup/btrfs_issue_dec_2018/btrfs_root_image_error_20181224.img
WARNING: cannot find a hash collision for '..', generating garbage, it
won't match indexes



> 
> It might fail, if so you can try adding -w and see if that helps.


OK, try with -w:

OK, many many complaints about hash collisions:
...
ARNING: cannot find a hash collision for 'ifup', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'catv', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'FDPC', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'LIBS', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'INTC', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'SPI', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'PDCA', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'EBI', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'SMC', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'WIFI', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'LWIP', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'HID', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'yun', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'avr4', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'avr6', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'WiFi', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'TFT', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'Knob', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'FP.h', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'SD.h', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'Beep', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'FORK', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'CHM', generating garbage, it
won't match indexes
WARNING: cannot find a hash collision for 'HandS', generating garbage,
it won't match indexes
WARNING: cannot find a hash collision for 'dm-0', generating garbage, it
won't match indexes


Now seems to stopped producing output.  Can't see if it is doing
something useful.  (note, started again, more such messages)


> 
> There is no log listed in the super so zero-log isn't indicated, and
> also tells me there were no fsync's still flushing at the time of the
> crash. The loss should be at most a minute of data, not an
> inconsistent file system that can't be mounted anymore. Pretty weird.
> 

I think I ran zero-log to see if that helped.  Given that there was no
important data and I'd assume I'd either easily fix it, or wipe it and
start over I may have taken the 'monkey radomly pounding the buttons'
approach, short of 'btrfs check --repair'.  I only posted here as I
though I'd fixed it apart from the one error!  If it were a simple fix
then it was worth asking.


> What were your mount options? Defaults? Anything custom like discard,
> commit=, notreelog? Any non-default mount options themselves would not
> be the cause of the problem, but might suggest partial ideas for what
> might have happened.
> 
fstab states:
autodefrag,ssd,discard,noatime,defaults,subvol=_r_sl14.
2,compress=lzo

However, I used an initrd, so I'm not sure if that is correct?

Ok, digging into init within my initrd, the line where the root partion
is mounted:
  mount -o ro -t $ROOTFS $ROOTDEV /mnt

Where $ROOTFS is:
btrfs -o subvol=_r_sl14.2

and $ROOTDEV is:
/dev/disk/by-uuid/6496aabd-d6aa-49e0-96ca-e49c316edd8e



Pete

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 8707 bytes --]



On 2018/12/24 下午7:31, Peter Chant wrote:
> On 12/24/18 12:58 AM, Chris Murphy wrote:
>> On Sat, Dec 22, 2018 at 10:22 AM Peter Chant <pete@petezilla.co.uk> wrote:
>>
>>> btrfs rescue super -v /dev/sdb2
>> ...
>>> All supers are valid, no need to recover
>>>
>>>
>>> btrfs insp dump-s -f <dev>
>> ...
>>> generation              7937947
>> ...
>>>         backup 0:
>>>                 backup_tree_root:       1113909100544   gen: 7937935    level: 1
>> ...
>>>         backup 1:
>>>                 backup_tree_root:       1113907347456   gen: 7937936    level: 1
>> ...
>>>         backup 2:
>>>                 backup_tree_root:       1113911951360   gen: 7937937    level: 1
>> ...
>>>         backup 3:
>>>                 backup_tree_root:       1113907494912   gen: 7937934    level: 1
>> ...
>>
>>
>> The kernel wrote out three valid checksummed supers, with what seems
>> to be a rather significant sanity violation. The super generation and
>> tree root address do not match any of the backup tree roots. The
>> *current* tree root is supposed to be in one of the backups as well.
>>
> 
> I wonder if this is a result of my trying to fix things?  E.g. btrfs
> rescue super-recover or my attempts using the tools (and kernel) in Mint
> 18.1 at one point?

At least super-recover is not responsible for this.
While btrfs check --repair could indeed cause problems.

So it may be the case.

> 
> I must admit, early on I had assumed that either this file system was a
> simple fix or was completely trashed, so I thought I'd have a quick go
> at fixing it, or wipe it and start again.  But then I seemed to get
> close with only the one error, but unmountable.
> 
> 
>> Qu, any idea how this is even theoretically possible? Bit flip right
>> before the super is computed and checksummed? Seems like some kind of
>> corruption before checksum is computed.
>>
>>
>>> I'm getting suspicious of the drive as when I was trying the various
>>> btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
>>> I also have a separate basic install on ext4 on the same disk.  Though
>>> e2fsck shows no errors and mounts fine I cannot log into that install.
>>> Maybe a coincidence, but too many bad things thrown up make me
>>> suspicious.  Whatever is happening this seems to be really fighting me.
>>
>> I'm not sure how even a bad device accounts for the super generation
>> and backup mismatches. That's damn strange.
> 
> I'm less suspicious of the drive now.  I've been using an ext4 partition
> on the same drive for a few days now, having reinstalled on that and
> everything _seems_ fine.  Mind you, apart from usb sticks, I've not
> experienced a ssd failure.  Perhaps my hdd failure experience is not
> relevent, i.e. they work until they start throwing errors and then
> rapidly fail?

I don't really believe a drive can be so easily corrupted to certain
bits while all other bits are OK.

> 
> 
>>
>> If you get bored with the back and forth and just want to give up,
>> that's fine. I suggest that if you have the time and space, to take a
>> btrfs-image in case Qu or some other developer wants to look at this
>> file system at some point. The btrfs-image is a read only process, can
>> be set to scrub filenames, and only contains metadata. Size of the
>> resulting file is around 1/2 of the size of metadata, when doing
>> 'btrfs filesystem usage' or 'btrfs filesystem df'. So you'll need that
>> much free space to direct the command to.
>>
>> btrfs-image -ss -c9 -t4 <devicetoimage> pathtofile
> 
> Just done that:
> bash-4.3# btrfs-image -ss -c9 -t4 /dev/sdd2
> /mnt/backup/btrfs_issue_dec_2018/btrfs_root_image_error_20181224.img
> WARNING: cannot find a hash collision for '..', generating garbage, it
> won't match indexes
> 
> 
> 
>>
>> It might fail, if so you can try adding -w and see if that helps.
> 
> 
> OK, try with -w:
> 
> OK, many many complaints about hash collisions:
> ...
> ARNING: cannot find a hash collision for 'ifup', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'catv', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'FDPC', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'LIBS', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'INTC', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'SPI', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'PDCA', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'EBI', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'SMC', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'WIFI', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'LWIP', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'HID', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'yun', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'avr4', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'avr6', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'WiFi', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'TFT', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'Knob', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'FP.h', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'SD.h', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'Beep', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'FORK', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'CHM', generating garbage, it
> won't match indexes
> WARNING: cannot find a hash collision for 'HandS', generating garbage,
> it won't match indexes
> WARNING: cannot find a hash collision for 'dm-0', generating garbage, it
> won't match indexes
> 
> 
> Now seems to stopped producing output.  Can't see if it is doing
> something useful.  (note, started again, more such messages)

I don't know about other developers, normally I don't like btrfs-image
-ss at all.

Even plain btrfs-image isn't so helpful, especially considering its size.

Anyway, from all the data you collected, I suspect it's a corruption in
tree blocks allocation, maybe a btrfs bug in older kernels, which buried
a dangerous seed into the fs, breaking the metadata CoW.

And one day, an unexpected powerloss makes the seed grow and screw up
the fs.

Just a personal recommendation, for btrfs especially used with older
kernels, after a powerloss, it's highly recommended to run btrfs check
--readonly before mounting it.

Thanks,
Qu

> 
> 
>>
>> There is no log listed in the super so zero-log isn't indicated, and
>> also tells me there were no fsync's still flushing at the time of the
>> crash. The loss should be at most a minute of data, not an
>> inconsistent file system that can't be mounted anymore. Pretty weird.
>>
> 
> I think I ran zero-log to see if that helped.  Given that there was no
> important data and I'd assume I'd either easily fix it, or wipe it and
> start over I may have taken the 'monkey radomly pounding the buttons'
> approach, short of 'btrfs check --repair'.  I only posted here as I
> though I'd fixed it apart from the one error!  If it were a simple fix
> then it was worth asking.
> 
> 
>> What were your mount options? Defaults? Anything custom like discard,
>> commit=, notreelog? Any non-default mount options themselves would not
>> be the cause of the problem, but might suggest partial ideas for what
>> might have happened.
>>
> fstab states:
> autodefrag,ssd,discard,noatime,defaults,subvol=_r_sl14.
> 2,compress=lzo
> 
> However, I used an initrd, so I'm not sure if that is correct?
> 
> Ok, digging into init within my initrd, the line where the root partion
> is mounted:
>   mount -o ro -t $ROOTFS $ROOTDEV /mnt
> 
> Where $ROOTFS is:
> btrfs -o subvol=_r_sl14.2
> 
> and $ROOTDEV is:
> /dev/disk/by-uuid/6496aabd-d6aa-49e0-96ca-e49c316edd8e
> 
> 
> 
> Pete
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Tomáš Metelka]

Hi Qu,

just 1 curious question (maybe 2) about your statement "log_root is 0":

What does it mean when log_root is non-zero? Because I have similar 
problem (unmountable FS ... I don't know how much but I know there's 
corrupted 2 subsequent items in chunk tree node) and when I have made 
"btrfs inspect-internal dump-super":

superblock: bytenr=65536, device=/dev/sda4
...
generation		2488742
root			232408301568
sys_array_size		97
chunk_root_generation	2487902
root_level		1
chunk_root		242098421760
chunk_root_level	1
log_root		232433811456
log_root_transid	0
log_root_level		0

superblock: bytenr=67108864, device=/dev/sda4
...
generation		2488742
root			232408301568
sys_array_size		97
chunk_root_generation	2487902
root_level		1
chunk_root		242098421760
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0

Unfortunately when I try to do "btrfs rescue chunk-recover" I get 
(beside others):

"...

Unrecoverable Chunks:
   Chunk: start = 0, len = 4194304, type = 2, num_stripes = 1
       Stripes list:
       [ 0] Stripe: devid = 1, offset = 0
       No block group.
       No device extent.

Total Chunks:		184
   Recoverable:		183
   Unrecoverable:	1

Orphan Block Groups:

Orphan Device Extents:

Chunk tree recovery failed
"

And when I try "btrfs restore -m -S -v -i -D <dev>" I get only:
Could not open root, trying backup super
Could not open root, trying backup super
ERROR: superblock bytenr 274877906944 is larger than device size 
212000047104
Could not open root, trying backup super

Is it possible to recover data (at least some of them)? And is it worth 
to upgrade to newest btrfs-progs?

uname -a:
Linux tisc5 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
x86_64 x86_64 x86_64 GNU/Linux

btrfs-progs v4.15.1

Thanks
Metaliza


On 24. 12. 18 13:02, Qu Wenruo wrote:
> 
> 
> On 2018/12/24 下午7:31, Peter Chant wrote:
>> On 12/24/18 12:58 AM, Chris Murphy wrote:
>>> On Sat, Dec 22, 2018 at 10:22 AM Peter Chant <pete@petezilla.co.uk> wrote:
>>>
>>>> btrfs rescue super -v /dev/sdb2
>>> ...
>>>> All supers are valid, no need to recover
>>>>
>>>>
>>>> btrfs insp dump-s -f <dev>
>>> ...
>>>> generation              7937947
>>> ...
>>>>          backup 0:
>>>>                  backup_tree_root:       1113909100544   gen: 7937935    level: 1
>>> ...
>>>>          backup 1:
>>>>                  backup_tree_root:       1113907347456   gen: 7937936    level: 1
>>> ...
>>>>          backup 2:
>>>>                  backup_tree_root:       1113911951360   gen: 7937937    level: 1
>>> ...
>>>>          backup 3:
>>>>                  backup_tree_root:       1113907494912   gen: 7937934    level: 1
>>> ...
>>>
>>>
>>> The kernel wrote out three valid checksummed supers, with what seems
>>> to be a rather significant sanity violation. The super generation and
>>> tree root address do not match any of the backup tree roots. The
>>> *current* tree root is supposed to be in one of the backups as well.
>>>
>>
>> I wonder if this is a result of my trying to fix things?  E.g. btrfs
>> rescue super-recover or my attempts using the tools (and kernel) in Mint
>> 18.1 at one point?
> 
> At least super-recover is not responsible for this.
> While btrfs check --repair could indeed cause problems.
> 
> So it may be the case.
> 
>>
>> I must admit, early on I had assumed that either this file system was a
>> simple fix or was completely trashed, so I thought I'd have a quick go
>> at fixing it, or wipe it and start again.  But then I seemed to get
>> close with only the one error, but unmountable.
>>
>>
>>> Qu, any idea how this is even theoretically possible? Bit flip right
>>> before the super is computed and checksummed? Seems like some kind of
>>> corruption before checksum is computed.
>>>
>>>
>>>> I'm getting suspicious of the drive as when I was trying the various
>>>> btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
>>>> I also have a separate basic install on ext4 on the same disk.  Though
>>>> e2fsck shows no errors and mounts fine I cannot log into that install.
>>>> Maybe a coincidence, but too many bad things thrown up make me
>>>> suspicious.  Whatever is happening this seems to be really fighting me.
>>>
>>> I'm not sure how even a bad device accounts for the super generation
>>> and backup mismatches. That's damn strange.
>>
>> I'm less suspicious of the drive now.  I've been using an ext4 partition
>> on the same drive for a few days now, having reinstalled on that and
>> everything _seems_ fine.  Mind you, apart from usb sticks, I've not
>> experienced a ssd failure.  Perhaps my hdd failure experience is not
>> relevent, i.e. they work until they start throwing errors and then
>> rapidly fail?
> 
> I don't really believe a drive can be so easily corrupted to certain
> bits while all other bits are OK.
> 
>>
>>
>>>
>>> If you get bored with the back and forth and just want to give up,
>>> that's fine. I suggest that if you have the time and space, to take a
>>> btrfs-image in case Qu or some other developer wants to look at this
>>> file system at some point. The btrfs-image is a read only process, can
>>> be set to scrub filenames, and only contains metadata. Size of the
>>> resulting file is around 1/2 of the size of metadata, when doing
>>> 'btrfs filesystem usage' or 'btrfs filesystem df'. So you'll need that
>>> much free space to direct the command to.
>>>
>>> btrfs-image -ss -c9 -t4 <devicetoimage> pathtofile
>>
>> Just done that:
>> bash-4.3# btrfs-image -ss -c9 -t4 /dev/sdd2
>> /mnt/backup/btrfs_issue_dec_2018/btrfs_root_image_error_20181224.img
>> WARNING: cannot find a hash collision for '..', generating garbage, it
>> won't match indexes
>>
>>
>>
>>>
>>> It might fail, if so you can try adding -w and see if that helps.
>>
>>
>> OK, try with -w:
>>
>> OK, many many complaints about hash collisions:
>> ...
>> ARNING: cannot find a hash collision for 'ifup', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'catv', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'FDPC', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'LIBS', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'INTC', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'SPI', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'PDCA', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'EBI', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'SMC', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'WIFI', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'LWIP', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'HID', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'yun', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'avr4', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'avr6', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'WiFi', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'TFT', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'Knob', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'FP.h', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'SD.h', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'Beep', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'FORK', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'CHM', generating garbage, it
>> won't match indexes
>> WARNING: cannot find a hash collision for 'HandS', generating garbage,
>> it won't match indexes
>> WARNING: cannot find a hash collision for 'dm-0', generating garbage, it
>> won't match indexes
>>
>>
>> Now seems to stopped producing output.  Can't see if it is doing
>> something useful.  (note, started again, more such messages)
> 
> I don't know about other developers, normally I don't like btrfs-image
> -ss at all.
> 
> Even plain btrfs-image isn't so helpful, especially considering its size.
> 
> Anyway, from all the data you collected, I suspect it's a corruption in
> tree blocks allocation, maybe a btrfs bug in older kernels, which buried
> a dangerous seed into the fs, breaking the metadata CoW.
> 
> And one day, an unexpected powerloss makes the seed grow and screw up
> the fs.
> 
> Just a personal recommendation, for btrfs especially used with older
> kernels, after a powerloss, it's highly recommended to run btrfs check
> --readonly before mounting it.
> 
> Thanks,
> Qu
> 
>>
>>
>>>
>>> There is no log listed in the super so zero-log isn't indicated, and
>>> also tells me there were no fsync's still flushing at the time of the
>>> crash. The loss should be at most a minute of data, not an
>>> inconsistent file system that can't be mounted anymore. Pretty weird.
>>>
>>
>> I think I ran zero-log to see if that helped.  Given that there was no
>> important data and I'd assume I'd either easily fix it, or wipe it and
>> start over I may have taken the 'monkey radomly pounding the buttons'
>> approach, short of 'btrfs check --repair'.  I only posted here as I
>> though I'd fixed it apart from the one error!  If it were a simple fix
>> then it was worth asking.
>>
>>
>>> What were your mount options? Defaults? Anything custom like discard,
>>> commit=, notreelog? Any non-default mount options themselves would not
>>> be the cause of the problem, but might suggest partial ideas for what
>>> might have happened.
>>>
>> fstab states:
>> autodefrag,ssd,discard,noatime,defaults,subvol=_r_sl14.
>> 2,compress=lzo
>>
>> However, I used an initrd, so I'm not sure if that is correct?
>>
>> Ok, digging into init within my initrd, the line where the root partion
>> is mounted:
>>    mount -o ro -t $ROOTFS $ROOTDEV /mnt
>>
>> Where $ROOTFS is:
>> btrfs -o subvol=_r_sl14.2
>>
>> and $ROOTDEV is:
>> /dev/disk/by-uuid/6496aabd-d6aa-49e0-96ca-e49c316edd8e
>>
>>
>>
>> Pete
>>
> 

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 12994 bytes --]



On 2018/12/24 下午8:48, Tomáš Metelka wrote:
> Hi Qu,
> 
> just 1 curious question (maybe 2) about your statement "log_root is 0":
> 
> What does it mean when log_root is non-zero?

This means there are some dirty log, namely caused by fsync().

You could consider log tree as some kind of journal used in ext/xfs.

Btrfs doesn't rely on log tree to keep its metadata consistent, but uses
it as a faster way to implement fsync().

For fs with dirty log, btrfs itself should be consistent no matter if we
replay the log or not.


The only certain thing a non-zero log tree shows is, there is definitely
an unexpected powerloss happened.
(But not vice verse, it's completely possible to hit a unexpected
powerloss without a dirty log, either it's lucky that no fsync() called
during that trans, or notreelog mount option is used)

> Because I have similar
> problem (unmountable FS ... I don't know how much but I know there's
> corrupted 2 subsequent items in chunk tree node)

Then the problem is not related to log root.
But either chunk tree get corrupted or metadata cow get exploited.

> and when I have made
> "btrfs inspect-internal dump-super":
> 
> superblock: bytenr=65536, device=/dev/sda4
> ...
> generation        2488742
> root            232408301568
> sys_array_size        97
> chunk_root_generation    2487902
> root_level        1
> chunk_root        242098421760
> chunk_root_level    1
> log_root        232433811456

log_root is only recorded in the primary super block.
So it's fine that your backup super block doesn't contain log root.

It's the designed behavior.

> log_root_transid    0
> log_root_level        0
> 
> superblock: bytenr=67108864, device=/dev/sda4
> ...
> generation        2488742
> root            232408301568
> sys_array_size        97
> chunk_root_generation    2487902
> root_level        1
> chunk_root        242098421760
> chunk_root_level    1
> log_root        0
> log_root_transid    0
> log_root_level        0
> 
> Unfortunately when I try to do "btrfs rescue chunk-recover" I get
> (beside others):
> 
> "...
> 
> Unrecoverable Chunks:
>   Chunk: start = 0, len = 4194304, type = 2, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 1, offset = 0
>       No block group.
>       No device extent.
> 
> Total Chunks:        184
>   Recoverable:        183
>   Unrecoverable:    1
> 
> Orphan Block Groups:
> 
> Orphan Device Extents:
> 
> Chunk tree recovery failed
> "
> 
> And when I try "btrfs restore -m -S -v -i -D <dev>" I get only:
> Could not open root, trying backup super
> Could not open root, trying backup super
> ERROR: superblock bytenr 274877906944 is larger than device size
> 212000047104
> Could not open root, trying backup super
> 
> Is it possible to recover data (at least some of them)? And is it worth
> to upgrade to newest btrfs-progs?

btrfs check --readonly output please.

btrfs check --readonly is always the most reliable and detailed output
for any possible recovery.

Also kernel message for the mount failure could help.

btrfs ins dump-tree/super is only useful when we have some ideas to verify.

Thanks,
Qu

> 
> uname -a:
> Linux tisc5 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018
> x86_64 x86_64 x86_64 GNU/Linux
> 
> btrfs-progs v4.15.1
> 
> Thanks
> Metaliza
> 
> 
> On 24. 12. 18 13:02, Qu Wenruo wrote:
>>
>>
>> On 2018/12/24 下午7:31, Peter Chant wrote:
>>> On 12/24/18 12:58 AM, Chris Murphy wrote:
>>>> On Sat, Dec 22, 2018 at 10:22 AM Peter Chant <pete@petezilla.co.uk>
>>>> wrote:
>>>>
>>>>> btrfs rescue super -v /dev/sdb2
>>>> ...
>>>>> All supers are valid, no need to recover
>>>>>
>>>>>
>>>>> btrfs insp dump-s -f <dev>
>>>> ...
>>>>> generation              7937947
>>>> ...
>>>>>          backup 0:
>>>>>                  backup_tree_root:       1113909100544   gen:
>>>>> 7937935    level: 1
>>>> ...
>>>>>          backup 1:
>>>>>                  backup_tree_root:       1113907347456   gen:
>>>>> 7937936    level: 1
>>>> ...
>>>>>          backup 2:
>>>>>                  backup_tree_root:       1113911951360   gen:
>>>>> 7937937    level: 1
>>>> ...
>>>>>          backup 3:
>>>>>                  backup_tree_root:       1113907494912   gen:
>>>>> 7937934    level: 1
>>>> ...
>>>>
>>>>
>>>> The kernel wrote out three valid checksummed supers, with what seems
>>>> to be a rather significant sanity violation. The super generation and
>>>> tree root address do not match any of the backup tree roots. The
>>>> *current* tree root is supposed to be in one of the backups as well.
>>>>
>>>
>>> I wonder if this is a result of my trying to fix things?  E.g. btrfs
>>> rescue super-recover or my attempts using the tools (and kernel) in Mint
>>> 18.1 at one point?
>>
>> At least super-recover is not responsible for this.
>> While btrfs check --repair could indeed cause problems.
>>
>> So it may be the case.
>>
>>>
>>> I must admit, early on I had assumed that either this file system was a
>>> simple fix or was completely trashed, so I thought I'd have a quick go
>>> at fixing it, or wipe it and start again.  But then I seemed to get
>>> close with only the one error, but unmountable.
>>>
>>>
>>>> Qu, any idea how this is even theoretically possible? Bit flip right
>>>> before the super is computed and checksummed? Seems like some kind of
>>>> corruption before checksum is computed.
>>>>
>>>>
>>>>> I'm getting suspicious of the drive as when I was trying the various
>>>>> btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
>>>>> I also have a separate basic install on ext4 on the same disk.  Though
>>>>> e2fsck shows no errors and mounts fine I cannot log into that install.
>>>>> Maybe a coincidence, but too many bad things thrown up make me
>>>>> suspicious.  Whatever is happening this seems to be really fighting
>>>>> me.
>>>>
>>>> I'm not sure how even a bad device accounts for the super generation
>>>> and backup mismatches. That's damn strange.
>>>
>>> I'm less suspicious of the drive now.  I've been using an ext4 partition
>>> on the same drive for a few days now, having reinstalled on that and
>>> everything _seems_ fine.  Mind you, apart from usb sticks, I've not
>>> experienced a ssd failure.  Perhaps my hdd failure experience is not
>>> relevent, i.e. they work until they start throwing errors and then
>>> rapidly fail?
>>
>> I don't really believe a drive can be so easily corrupted to certain
>> bits while all other bits are OK.
>>
>>>
>>>
>>>>
>>>> If you get bored with the back and forth and just want to give up,
>>>> that's fine. I suggest that if you have the time and space, to take a
>>>> btrfs-image in case Qu or some other developer wants to look at this
>>>> file system at some point. The btrfs-image is a read only process, can
>>>> be set to scrub filenames, and only contains metadata. Size of the
>>>> resulting file is around 1/2 of the size of metadata, when doing
>>>> 'btrfs filesystem usage' or 'btrfs filesystem df'. So you'll need that
>>>> much free space to direct the command to.
>>>>
>>>> btrfs-image -ss -c9 -t4 <devicetoimage> pathtofile
>>>
>>> Just done that:
>>> bash-4.3# btrfs-image -ss -c9 -t4 /dev/sdd2
>>> /mnt/backup/btrfs_issue_dec_2018/btrfs_root_image_error_20181224.img
>>> WARNING: cannot find a hash collision for '..', generating garbage, it
>>> won't match indexes
>>>
>>>
>>>
>>>>
>>>> It might fail, if so you can try adding -w and see if that helps.
>>>
>>>
>>> OK, try with -w:
>>>
>>> OK, many many complaints about hash collisions:
>>> ...
>>> ARNING: cannot find a hash collision for 'ifup', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'catv', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'FDPC', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'LIBS', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'INTC', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'SPI', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'PDCA', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'EBI', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'SMC', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'WIFI', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'LWIP', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'HID', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'yun', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'avr4', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'avr6', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'WiFi', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'TFT', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'Knob', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'FP.h', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'SD.h', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'Beep', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'FORK', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'CHM', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'HandS', generating garbage,
>>> it won't match indexes
>>> WARNING: cannot find a hash collision for 'dm-0', generating garbage, it
>>> won't match indexes
>>>
>>>
>>> Now seems to stopped producing output.  Can't see if it is doing
>>> something useful.  (note, started again, more such messages)
>>
>> I don't know about other developers, normally I don't like btrfs-image
>> -ss at all.
>>
>> Even plain btrfs-image isn't so helpful, especially considering its size.
>>
>> Anyway, from all the data you collected, I suspect it's a corruption in
>> tree blocks allocation, maybe a btrfs bug in older kernels, which buried
>> a dangerous seed into the fs, breaking the metadata CoW.
>>
>> And one day, an unexpected powerloss makes the seed grow and screw up
>> the fs.
>>
>> Just a personal recommendation, for btrfs especially used with older
>> kernels, after a powerloss, it's highly recommended to run btrfs check
>> --readonly before mounting it.
>>
>> Thanks,
>> Qu
>>
>>>
>>>
>>>>
>>>> There is no log listed in the super so zero-log isn't indicated, and
>>>> also tells me there were no fsync's still flushing at the time of the
>>>> crash. The loss should be at most a minute of data, not an
>>>> inconsistent file system that can't be mounted anymore. Pretty weird.
>>>>
>>>
>>> I think I ran zero-log to see if that helped.  Given that there was no
>>> important data and I'd assume I'd either easily fix it, or wipe it and
>>> start over I may have taken the 'monkey radomly pounding the buttons'
>>> approach, short of 'btrfs check --repair'.  I only posted here as I
>>> though I'd fixed it apart from the one error!  If it were a simple fix
>>> then it was worth asking.
>>>
>>>
>>>> What were your mount options? Defaults? Anything custom like discard,
>>>> commit=, notreelog? Any non-default mount options themselves would not
>>>> be the cause of the problem, but might suggest partial ideas for what
>>>> might have happened.
>>>>
>>> fstab states:
>>> autodefrag,ssd,discard,noatime,defaults,subvol=_r_sl14.
>>> 2,compress=lzo
>>>
>>> However, I used an initrd, so I'm not sure if that is correct?
>>>
>>> Ok, digging into init within my initrd, the line where the root partion
>>> is mounted:
>>>    mount -o ro -t $ROOTFS $ROOTDEV /mnt
>>>
>>> Where $ROOTFS is:
>>> btrfs -o subvol=_r_sl14.2
>>>
>>> and $ROOTDEV is:
>>> /dev/disk/by-uuid/6496aabd-d6aa-49e0-96ca-e49c316edd8e
>>>
>>>
>>>
>>> Pete
>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Tomáš Metelka]

On 24. 12. 18 14:02, Qu Wenruo wrote:
> btrfs check --readonly output please.
> 
> btrfs check --readonly is always the most reliable and detailed output
> for any possible recovery.

This is very weird because it prints only:
ERROR: cannot open file system

I've tried also "btrfs check -r 75152310272" but it only says:
parent transid verify failed on 75152310272 wanted 2488742 found 2488741
parent transid verify failed on 75152310272 wanted 2488742 found 2488741
Ignoring transid failure
ERROR: cannot open file system

I've tried that because:
	backup 3:
  backup_tree_root:	75152310272	gen: 2488741 level: 1

> Also kernel message for the mount failure could help.

Sorry, my fault, I should start from this point:

Dec 23 21:59:07 tisc5 kernel: [10319.442615] BTRFS: device fsid 
be557007-42c9-4079-be16-568997e94cd9 devid 1 transid 2488742 /dev/loop0
Dec 23 22:00:49 tisc5 kernel: [10421.167028] BTRFS info (device loop0): 
disk space caching is enabled
Dec 23 22:00:49 tisc5 kernel: [10421.167034] BTRFS info (device loop0): 
has skinny extents
Dec 23 22:00:50 tisc5 kernel: [10421.807564] BTRFS critical (device 
loop0): corrupt node: root=1 block=75150311424 slot=245, invalid NULL 
node pointer
Dec 23 22:00:50 tisc5 kernel: [10421.807653] BTRFS error (device loop0): 
failed to read block groups: -5
Dec 23 22:00:50 tisc5 kernel: [10421.877001] BTRFS error (device loop0): 
open_ctree failed


So i tried to do:
1) btrfs inspect-internal dump-super (with the snippet posted above)
2) btrfs inspect-internal dump-tree -b 75150311424

And it showed (header + snippet for items 243-248):
node 75150311424 level 1 items 249 free 244 generation 2488741 owner 2
fs uuid be557007-42c9-4079-be16-568997e94cd9
chunk uuid dbe69c7e-2d50-4001-af31-148c5475b48b
...
   key (14799519744 EXTENT_ITEM 4096) block 233423224832 (14247023) gen 
2484894
   key (14811271168 EXTENT_ITEM 135168) block 656310272 (40058) gen 2488049
   key (1505328190277054464 UNKNOWN.4 366981796979539968) block 0 (0) gen 0
   key (0 UNKNOWN.0 1419267647995904) block 6468220747776 (394788864) 
gen 7786775707648
   key (12884901888 EXTENT_ITEM 24576) block 816693248 (49847) gen 2484931
   key (14902849536 EXTENT_ITEM 131072) block 75135844352 (4585928) gen 
2488739


I looked at that numbers quite a while (also in hex) trying to figure 
out what has happened (bit flips (it was on SSD), byte shifts (I 
suspected bad CPU also ... because it has died after 2 months from 
that)) and tried to guess "correct" values for that items ... but no idea:-(

So this why I have asked about that log_root and whether there is a 
chance to "log-replay things":-)


Thanks
M.

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4262 bytes --]



On 2018/12/24 下午9:52, Tomáš Metelka wrote:
> On 24. 12. 18 14:02, Qu Wenruo wrote:
>> btrfs check --readonly output please.
>>
>> btrfs check --readonly is always the most reliable and detailed output
>> for any possible recovery.
> 
> This is very weird because it prints only:
> ERROR: cannot open file system

A new place to enhance ;)

> 
> I've tried also "btrfs check -r 75152310272" but it only says:
> parent transid verify failed on 75152310272 wanted 2488742 found 2488741
> parent transid verify failed on 75152310272 wanted 2488742 found 2488741
> Ignoring transid failure
> ERROR: cannot open file system
> 
> I've tried that because:
>     backup 3:
>  backup_tree_root:    75152310272    gen: 2488741 level: 1
> 
>> Also kernel message for the mount failure could help.
> 
> Sorry, my fault, I should start from this point:
> 
> Dec 23 21:59:07 tisc5 kernel: [10319.442615] BTRFS: device fsid
> be557007-42c9-4079-be16-568997e94cd9 devid 1 transid 2488742 /dev/loop0
> Dec 23 22:00:49 tisc5 kernel: [10421.167028] BTRFS info (device loop0):
> disk space caching is enabled
> Dec 23 22:00:49 tisc5 kernel: [10421.167034] BTRFS info (device loop0):
> has skinny extents
> Dec 23 22:00:50 tisc5 kernel: [10421.807564] BTRFS critical (device
> loop0): corrupt node: root=1 block=75150311424 slot=245, invalid NULL
> node pointer
This explains the problem.

Your root tree has one node pointer which is not correct.
For pointer it should never points to 0.

This is pretty weird, at least some corruption pattern I have never seen.

Since your tree root get corrupted, there isn't much thing we can do,
but try to use older tree roots.

You could go try all backup roots, starting from the newest backup (with
highest generation), and check the backup root bytenr using:
# btrfs check -r <backup root bytenr> <device>

To see which one get least error, but normally the chance is near 0.

> Dec 23 22:00:50 tisc5 kernel: [10421.807653] BTRFS error (device loop0):
> failed to read block groups: -5
> Dec 23 22:00:50 tisc5 kernel: [10421.877001] BTRFS error (device loop0):
> open_ctree failed
> 
> 
> So i tried to do:
> 1) btrfs inspect-internal dump-super (with the snippet posted above)
> 2) btrfs inspect-internal dump-tree -b 75150311424
> 
> And it showed (header + snippet for items 243-248):
> node 75150311424 level 1 items 249 free 244 generation 2488741 owner 2
> fs uuid be557007-42c9-4079-be16-568997e94cd9
> chunk uuid dbe69c7e-2d50-4001-af31-148c5475b48b
> ...
>   key (14799519744 EXTENT_ITEM 4096) block 233423224832 (14247023) gen
> 2484894
>   key (14811271168 EXTENT_ITEM 135168) block 656310272 (40058) gen 2488049


>   key (1505328190277054464 UNKNOWN.4 366981796979539968) block 0 (0) gen 0
>   key (0 UNKNOWN.0 1419267647995904) block 6468220747776 (394788864) gen
> 7786775707648

Pretty obviously, these two nodes are garbage.
Something corrupted the memory at runtime, and we don't have runtime
check against corruption yet.

So IMHO, I think the problem is, some kernel code, either btrfs or other
parts, corrupted the memory.
And then btrfs fails to detect it, write it back to disk, and finally
kernel get its chance to read the tree block from disk and finally
caught the problem.

I could add such check for node, but normally it needs
CONFIG_BTRFS_FS_CHECK_INTEGRITY, so makes no sense for normal user.

>   key (12884901888 EXTENT_ITEM 24576) block 816693248 (49847) gen 2484931
>   key (14902849536 EXTENT_ITEM 131072) block 75135844352 (4585928) gen
> 2488739
> 
> 
> I looked at that numbers quite a while (also in hex) trying to figure
> out what has happened (bit flips (it was on SSD), byte shifts (I
> suspected bad CPU also ... because it has died after 2 months from
> that)) and tried to guess "correct" values for that items ... but no
> idea:-(

I'm not that sure, unless you're super lucky (or unlucky in this case),
or it will normally get caught by csum first.

> 
> So this why I have asked about that log_root and whether there is a
> chance to "log-replay things":-)

For your case, definitely not related to log replay.

Thanks,
Qu

> 
> 
> Thanks
> M.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Broken chunk tree - Was: Mount issue, mount /dev/sdc2: can't read superblock

[Tomáš Metelka]

Ok, I've got it:-(

But just a few questions: I've tried (with btrfs-progs v4.19.1) to 
recover files through btrfs restore -s -m -S -v -i ... and following 
events occurred:

1) Just 1 "hard" error:
ERROR: cannot map block logical 117058830336 length 1073741824: -2
Error copying data for /mnt/...
(file which absence really doesn't pain me:-))

2) For 24 files a I got "too much loops" warning (U mean this: "if 
(loops >= 0 && loops++ >= 1024) { ..."). I've always answered yes but 
I'm afraid these files are corrupted (at least 2 of them seems corrupted).

How much bad is this? Does the error mentioned in #1 mean that it's the 
only file which is totally lost? I can live without those 24 + 1 files 
so if #1 and #2 would be the only errors then I could say the recovery 
was successful ... but I'm afraid things aren't such easy:-)

Thanks
M.


   Tomáš Metelka
   Business & IT Analyst

   Tel: +420 728 627 252
   Email: tomas.metelka@metaliza.cz



On 24. 12. 18 15:19, Qu Wenruo wrote:
> 
> 
> On 2018/12/24 下午9:52, Tomáš Metelka wrote:
>> On 24. 12. 18 14:02, Qu Wenruo wrote:
>>> btrfs check --readonly output please.
>>>
>>> btrfs check --readonly is always the most reliable and detailed output
>>> for any possible recovery.
>>
>> This is very weird because it prints only:
>> ERROR: cannot open file system
> 
> A new place to enhance ;)
> 
>>
>> I've tried also "btrfs check -r 75152310272" but it only says:
>> parent transid verify failed on 75152310272 wanted 2488742 found 2488741
>> parent transid verify failed on 75152310272 wanted 2488742 found 2488741
>> Ignoring transid failure
>> ERROR: cannot open file system
>>
>> I've tried that because:
>>      backup 3:
>>   backup_tree_root:    75152310272    gen: 2488741 level: 1
>>
>>> Also kernel message for the mount failure could help.
>>
>> Sorry, my fault, I should start from this point:
>>
>> Dec 23 21:59:07 tisc5 kernel: [10319.442615] BTRFS: device fsid
>> be557007-42c9-4079-be16-568997e94cd9 devid 1 transid 2488742 /dev/loop0
>> Dec 23 22:00:49 tisc5 kernel: [10421.167028] BTRFS info (device loop0):
>> disk space caching is enabled
>> Dec 23 22:00:49 tisc5 kernel: [10421.167034] BTRFS info (device loop0):
>> has skinny extents
>> Dec 23 22:00:50 tisc5 kernel: [10421.807564] BTRFS critical (device
>> loop0): corrupt node: root=1 block=75150311424 slot=245, invalid NULL
>> node pointer
> This explains the problem.
> 
> Your root tree has one node pointer which is not correct.
> For pointer it should never points to 0.
> 
> This is pretty weird, at least some corruption pattern I have never seen.
> 
> Since your tree root get corrupted, there isn't much thing we can do,
> but try to use older tree roots.
> 
> You could go try all backup roots, starting from the newest backup (with
> highest generation), and check the backup root bytenr using:
> # btrfs check -r <backup root bytenr> <device>
> 
> To see which one get least error, but normally the chance is near 0.
> 
>> Dec 23 22:00:50 tisc5 kernel: [10421.807653] BTRFS error (device loop0):
>> failed to read block groups: -5
>> Dec 23 22:00:50 tisc5 kernel: [10421.877001] BTRFS error (device loop0):
>> open_ctree failed
>>
>>
>> So i tried to do:
>> 1) btrfs inspect-internal dump-super (with the snippet posted above)
>> 2) btrfs inspect-internal dump-tree -b 75150311424
>>
>> And it showed (header + snippet for items 243-248):
>> node 75150311424 level 1 items 249 free 244 generation 2488741 owner 2
>> fs uuid be557007-42c9-4079-be16-568997e94cd9
>> chunk uuid dbe69c7e-2d50-4001-af31-148c5475b48b
>> ...
>>    key (14799519744 EXTENT_ITEM 4096) block 233423224832 (14247023) gen
>> 2484894
>>    key (14811271168 EXTENT_ITEM 135168) block 656310272 (40058) gen 2488049
> 
> 
>>    key (1505328190277054464 UNKNOWN.4 366981796979539968) block 0 (0) gen 0
>>    key (0 UNKNOWN.0 1419267647995904) block 6468220747776 (394788864) gen
>> 7786775707648
> 
> Pretty obviously, these two nodes are garbage.
> Something corrupted the memory at runtime, and we don't have runtime
> check against corruption yet.
> 
> So IMHO, I think the problem is, some kernel code, either btrfs or other
> parts, corrupted the memory.
> And then btrfs fails to detect it, write it back to disk, and finally
> kernel get its chance to read the tree block from disk and finally
> caught the problem.
> 
> I could add such check for node, but normally it needs
> CONFIG_BTRFS_FS_CHECK_INTEGRITY, so makes no sense for normal user.
> 
>>    key (12884901888 EXTENT_ITEM 24576) block 816693248 (49847) gen 2484931
>>    key (14902849536 EXTENT_ITEM 131072) block 75135844352 (4585928) gen
>> 2488739
>>
>>
>> I looked at that numbers quite a while (also in hex) trying to figure
>> out what has happened (bit flips (it was on SSD), byte shifts (I
>> suspected bad CPU also ... because it has died after 2 months from
>> that)) and tried to guess "correct" values for that items ... but no
>> idea:-(
> 
> I'm not that sure, unless you're super lucky (or unlucky in this case),
> or it will normally get caught by csum first.
> 
>>
>> So this why I have asked about that log_root and whether there is a
>> chance to "log-replay things":-)
> 
> For your case, definitely not related to log replay.
> 
> Thanks,
> Qu
> 
>>
>>
>> Thanks
>> M.
> 

Re: Broken chunk tree - Was: Mount issue, mount /dev/sdc2: can't read superblock

[Duncan]

Tomáš Metelka posted on Sun, 30 Dec 2018 01:48:23 +0100 as excerpted:

> Ok, I've got it:-(
> 
> But just a few questions: I've tried (with btrfs-progs v4.19.1) to
> recover files through btrfs restore -s -m -S -v -i ... and following
> events occurred:
> 
> 1) Just 1 "hard" error:
> ERROR: cannot map block logical 117058830336 length 1073741824: -2 Error
> copying data for /mnt/...
> (file which absence really doesn't pain me:-))
> 
> 2) For 24 files a I got "too much loops" warning (U mean this: "if
> (loops >= 0 && loops++ >= 1024) { ..."). I've always answered yes but
> I'm afraid these files are corrupted (at least 2 of them seems
> corrupted).
> 
> How much bad is this? Does the error mentioned in #1 mean that it's the
> only file which is totally lost? I can live without those 24 + 1 files
> so if #1 and #2 would be the only errors then I could say the recovery
> was successful ... but I'm afraid things aren't such easy:-)

In this context, the biggest thing to know about btrfs restore is that 
because it's meant as a recovery measure and if it comes to using 
restore, the assumption is that the priority is recovery of /anything/ 
even if the checksums don't match (a chance of recovery of possibly bad 
data is considered better than rejecting possibly bad data entirely), it 
doesn't worry too much about checksums (AFAIK it ignores data checksums 
entirely, not sure whether it checks metadata checksums or not, but it 
probably ignores failure in at least some cases if it does, because 
that's the point for a tool like this).

Which means that anything recovered using btrfs recover doesn't have the 
usual btrfs checksums validation guarantees, and could very possibly be 
corrupt.

However, that's mitigated by the fact that most filesystems don't even 
have built-in checksumming and validation in the first place, so the data 
on them could go bad even in normal operation, and unless it was 
obviously corrupted into not working, you'd likely never even know it, so 
btrfs restore ignoring checksums simply returns the data to the state 
it's /normally/ in on most filesystems, completely unverified.

But if you happen to have checksums independently stored somewhere, or 
even just ordinary unvalidated backups you can compare against, and 
you're worried about the possibility of undiscovered corruption due to 
the restore, and/or you were using btrfs in part /because/ of its built-
in checksum verification, it could be worth doing that verification run 
against your old checksum database or backups.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Broken chunk tree - Was: Mount issue, mount /dev/sdc2: can't read superblock

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 6033 bytes --]



On 2018/12/30 上午8:48, Tomáš Metelka wrote:
> Ok, I've got it:-(
> 
> But just a few questions: I've tried (with btrfs-progs v4.19.1) to
> recover files through btrfs restore -s -m -S -v -i ... and following
> events occurred:
> 
> 1) Just 1 "hard" error:
> ERROR: cannot map block logical 117058830336 length 1073741824: -2
> Error copying data for /mnt/...
> (file which absence really doesn't pain me:-))

This means one data extent can't be recovered due to missing chunk mapping.

Not impossible for heavily damaged fs, but nothing serious.

> 
> 2) For 24 files a I got "too much loops" warning (U mean this: "if
> (loops >= 0 && loops++ >= 1024) { ..."). I've always answered yes but
> I'm afraid these files are corrupted (at least 2 of them seems corrupted).
> 
> How much bad is this?

Not sure, but I don't think store is robust enough for such case.
Maybe false alert.

> Does the error mentioned in #1 mean that it's the
> only file which is totally lost?

Not even total lost, as it's just one file extent, maybe other part is OK.

Thanks,
Qu

> I can live without those 24 + 1 files
> so if #1 and #2 would be the only errors then I could say the recovery
> was successful ... but I'm afraid things aren't such easy:-)
> 
> Thanks
> M.
> 
> 
>   Tomáš Metelka
>   Business & IT Analyst
> 
>   Tel: +420 728 627 252
>   Email: tomas.metelka@metaliza.cz
> 
> 
> 
> On 24. 12. 18 15:19, Qu Wenruo wrote:
>>
>>
>> On 2018/12/24 下午9:52, Tomáš Metelka wrote:
>>> On 24. 12. 18 14:02, Qu Wenruo wrote:
>>>> btrfs check --readonly output please.
>>>>
>>>> btrfs check --readonly is always the most reliable and detailed output
>>>> for any possible recovery.
>>>
>>> This is very weird because it prints only:
>>> ERROR: cannot open file system
>>
>> A new place to enhance ;)
>>
>>>
>>> I've tried also "btrfs check -r 75152310272" but it only says:
>>> parent transid verify failed on 75152310272 wanted 2488742 found 2488741
>>> parent transid verify failed on 75152310272 wanted 2488742 found 2488741
>>> Ignoring transid failure
>>> ERROR: cannot open file system
>>>
>>> I've tried that because:
>>>      backup 3:
>>>   backup_tree_root:    75152310272    gen: 2488741 level: 1
>>>
>>>> Also kernel message for the mount failure could help.
>>>
>>> Sorry, my fault, I should start from this point:
>>>
>>> Dec 23 21:59:07 tisc5 kernel: [10319.442615] BTRFS: device fsid
>>> be557007-42c9-4079-be16-568997e94cd9 devid 1 transid 2488742 /dev/loop0
>>> Dec 23 22:00:49 tisc5 kernel: [10421.167028] BTRFS info (device loop0):
>>> disk space caching is enabled
>>> Dec 23 22:00:49 tisc5 kernel: [10421.167034] BTRFS info (device loop0):
>>> has skinny extents
>>> Dec 23 22:00:50 tisc5 kernel: [10421.807564] BTRFS critical (device
>>> loop0): corrupt node: root=1 block=75150311424 slot=245, invalid NULL
>>> node pointer
>> This explains the problem.
>>
>> Your root tree has one node pointer which is not correct.
>> For pointer it should never points to 0.
>>
>> This is pretty weird, at least some corruption pattern I have never seen.
>>
>> Since your tree root get corrupted, there isn't much thing we can do,
>> but try to use older tree roots.
>>
>> You could go try all backup roots, starting from the newest backup (with
>> highest generation), and check the backup root bytenr using:
>> # btrfs check -r <backup root bytenr> <device>
>>
>> To see which one get least error, but normally the chance is near 0.
>>
>>> Dec 23 22:00:50 tisc5 kernel: [10421.807653] BTRFS error (device loop0):
>>> failed to read block groups: -5
>>> Dec 23 22:00:50 tisc5 kernel: [10421.877001] BTRFS error (device loop0):
>>> open_ctree failed
>>>
>>>
>>> So i tried to do:
>>> 1) btrfs inspect-internal dump-super (with the snippet posted above)
>>> 2) btrfs inspect-internal dump-tree -b 75150311424
>>>
>>> And it showed (header + snippet for items 243-248):
>>> node 75150311424 level 1 items 249 free 244 generation 2488741 owner 2
>>> fs uuid be557007-42c9-4079-be16-568997e94cd9
>>> chunk uuid dbe69c7e-2d50-4001-af31-148c5475b48b
>>> ...
>>>    key (14799519744 EXTENT_ITEM 4096) block 233423224832 (14247023) gen
>>> 2484894
>>>    key (14811271168 EXTENT_ITEM 135168) block 656310272 (40058) gen
>>> 2488049
>>
>>
>>>    key (1505328190277054464 UNKNOWN.4 366981796979539968) block 0 (0)
>>> gen 0
>>>    key (0 UNKNOWN.0 1419267647995904) block 6468220747776 (394788864)
>>> gen
>>> 7786775707648
>>
>> Pretty obviously, these two nodes are garbage.
>> Something corrupted the memory at runtime, and we don't have runtime
>> check against corruption yet.
>>
>> So IMHO, I think the problem is, some kernel code, either btrfs or other
>> parts, corrupted the memory.
>> And then btrfs fails to detect it, write it back to disk, and finally
>> kernel get its chance to read the tree block from disk and finally
>> caught the problem.
>>
>> I could add such check for node, but normally it needs
>> CONFIG_BTRFS_FS_CHECK_INTEGRITY, so makes no sense for normal user.
>>
>>>    key (12884901888 EXTENT_ITEM 24576) block 816693248 (49847) gen
>>> 2484931
>>>    key (14902849536 EXTENT_ITEM 131072) block 75135844352 (4585928) gen
>>> 2488739
>>>
>>>
>>> I looked at that numbers quite a while (also in hex) trying to figure
>>> out what has happened (bit flips (it was on SSD), byte shifts (I
>>> suspected bad CPU also ... because it has died after 2 months from
>>> that)) and tried to guess "correct" values for that items ... but no
>>> idea:-(
>>
>> I'm not that sure, unless you're super lucky (or unlucky in this case),
>> or it will normally get caught by csum first.
>>
>>>
>>> So this why I have asked about that log_root and whether there is a
>>> chance to "log-replay things":-)
>>
>> For your case, definitely not related to log replay.
>>
>> Thanks,
>> Qu
>>
>>>
>>>
>>> Thanks
>>> M.
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Mount issue, mount /dev/sdc2: can't read superblock

[Chris Murphy]

On Mon, Dec 24, 2018 at 4:31 AM Peter Chant <pete@petezilla.co.uk> wrote:

>
> Just done that:
> bash-4.3# btrfs-image -ss -c9 -t4 /dev/sdd2
> /mnt/backup/btrfs_issue_dec_2018/btrfs_root_image_error_20181224.img
> WARNING: cannot find a hash collision for '..', generating garbage, it
> won't match indexes

normal for some files including .. and files with really short names


> Now seems to stopped producing output.  Can't see if it is doing
> something useful.  (note, started again, more such messages)

You can watch if the file size of the image is increasing. With SSD
it's maybe around 4MB/s throughput, depending on the CPU.

>
>
> >
> > There is no log listed in the super so zero-log isn't indicated, and
> > also tells me there were no fsync's still flushing at the time of the
> > crash. The loss should be at most a minute of data, not an
> > inconsistent file system that can't be mounted anymore. Pretty weird.
> >
>
> I think I ran zero-log to see if that helped.  Given that there was no
> important data and I'd assume I'd either easily fix it, or wipe it and
> start over I may have taken the 'monkey radomly pounding the buttons'
> approach, short of 'btrfs check --repair'.  I only posted here as I
> though I'd fixed it apart from the one error!  If it were a simple fix
> then it was worth asking.

Yeah I don't know how zero-log actually works, if it just removes the
log tree address in the superblock or if it's more involved than that,
and also changes root tree address+generation.



>
>
> > What were your mount options? Defaults? Anything custom like discard,
> > commit=, notreelog? Any non-default mount options themselves would not
> > be the cause of the problem, but might suggest partial ideas for what
> > might have happened.
> >
> fstab states:
> autodefrag,ssd,discard,noatime,defaults,subvol=_r_sl14.
> 2,compress=lzo

I suggest not using discard when it passes through to the SSD. There
are firmware bugs abound that can cause weird problems. And Btrfs does
not delay discarding any backup metadata once it's dereferenced. So
all the backup trees in the super, once dereferenced also get TRIM'd
by the SSD, rendering them useless.

I was running with discard mount option on an NVMe drive in my laptop,
with no problems. But once I learned how aggressively it TRIMs metdata
subject to the discard mount option, I dropped it. These days I just
depend on the included systemd fstrim.service to issue TRIM once a
week.

I supposed it's possible your problem could be discard/TRIM related,
but somehow I kinda doubt it. Usually such bugs show up with entire
block ranges being wiped out when they shouldn't be (either with zeros
or returning corrupt data). In the meantime, drop it for all file
systems. And also check to make sure the SSD has the latest firmware
version.


-- 
Chris Murphy