btrfs check segfaults after flipping 2 Bytes

btrfs check segfaults after flipping 2 Bytes

[Niklas Fischer]

Hello,

I was trying to determine how btrfs reacts to disk errors, when I
discovered, that flipping two Bytes, supposedly inside of a file can
render the filesystem unusable. Here is what I did:

1. dd if=/dev/zero of=/dev/sdg2 bs=1M
2. mkfs.btrfs /dev/sdg2
3. mount /dev/sdg2 /tmp/btrfs
4. echo "hello world this is some text" > /tmp/btrfs/hello
5. umount /dev/sdg2

this should result in this image [1]

in the following steps /dev/sdg2 is altered. Basically, we want to apply
s/world/wirld/g

6. dd if=/dev/sdg2 of=/tmp/sdg2 bs=1M
7. edit /tmp/sdg2 in ghex, find & replace all occurences of "world" with
"wirld" (two occurences found), save as /tmp/sdg2_new
8. dd if=/tmp/sdg2_new of=/dev/sdg2 bs=1M

/dev/sdg2 now looks like this [2]

now, when I try mounting /dev/sdg2:

"""22:21 root@localhost ~#
LANG=EN mount /dev/sdg2 /tmp/btrfs/
mount: mount /dev/sdg2 on /tmp/btrfs failed: Cannot allocate memory"""

ok, so let us try btrfs check

"""22:21 root@localhost ~#
LANG=EN btrfs check /dev/sdg2
checksum verify failed on 4222976 found F22E71BD wanted CE334502
checksum verify failed on 4222976 found F22E71BD wanted CE334502
Csum didn't match
Checking filesystem on /dev/sdg2
UUID: ea2843f1-e8aa-420f-acf8-6ef13f6c6753
checking extents
Speicherzugriffsfehler (Speicherabzug geschrieben)"""

that last line reads "Segfault, memory dumped". Is this behaviour expected?

Kind regards,
Niklas

[1] tgz: https://gigamove.rz.rwth-aachen.de/d/id/R5CKikxbYTBdqs
    raw: https://gigamove.rz.rwth-aachen.de/d/id/VgMXAicS7CGEjt
[2] tgz: https://gigamove.rz.rwth-aachen.de/d/id/cEebNYne9ppFy4
    raw: https://gigamove.rz.rwth-aachen.de/d/id/ATJa7wpfEWdfL2

Re: btrfs check segfaults after flipping 2 Bytes

[Niklas Fischer]

What I forgot to mention:

`uname -r`: 3.17.0-0.rc6.git2.1.fc22.x86_64
`btrfs --version`: Btrfs v3.16

regards,
Niklas

Am 01.10.2014 um 22:29 schrieb Niklas Fischer:
> Hello,
> 
> I was trying to determine how btrfs reacts to disk errors, when I
> discovered, that flipping two Bytes, supposedly inside of a file can
> render the filesystem unusable. Here is what I did:
> 
> 1. dd if=/dev/zero of=/dev/sdg2 bs=1M
> 2. mkfs.btrfs /dev/sdg2
> 3. mount /dev/sdg2 /tmp/btrfs
> 4. echo "hello world this is some text" > /tmp/btrfs/hello
> 5. umount /dev/sdg2
> 
> this should result in this image [1]
> 
> in the following steps /dev/sdg2 is altered. Basically, we want to apply
> s/world/wirld/g
> 
> 6. dd if=/dev/sdg2 of=/tmp/sdg2 bs=1M
> 7. edit /tmp/sdg2 in ghex, find & replace all occurences of "world" with
> "wirld" (two occurences found), save as /tmp/sdg2_new
> 8. dd if=/tmp/sdg2_new of=/dev/sdg2 bs=1M
> 
> /dev/sdg2 now looks like this [2]
> 
> now, when I try mounting /dev/sdg2:
> 
> """22:21 root@localhost ~#
> LANG=EN mount /dev/sdg2 /tmp/btrfs/
> mount: mount /dev/sdg2 on /tmp/btrfs failed: Cannot allocate memory"""
> 
> ok, so let us try btrfs check
> 
> """22:21 root@localhost ~#
> LANG=EN btrfs check /dev/sdg2
> checksum verify failed on 4222976 found F22E71BD wanted CE334502
> checksum verify failed on 4222976 found F22E71BD wanted CE334502
> Csum didn't match
> Checking filesystem on /dev/sdg2
> UUID: ea2843f1-e8aa-420f-acf8-6ef13f6c6753
> checking extents
> Speicherzugriffsfehler (Speicherabzug geschrieben)"""
> 
> that last line reads "Segfault, memory dumped". Is this behaviour expected?
> 
> Kind regards,
> Niklas
> 
> [1] tgz: https://gigamove.rz.rwth-aachen.de/d/id/R5CKikxbYTBdqs
>     raw: https://gigamove.rz.rwth-aachen.de/d/id/VgMXAicS7CGEjt
> [2] tgz: https://gigamove.rz.rwth-aachen.de/d/id/cEebNYne9ppFy4
>     raw: https://gigamove.rz.rwth-aachen.de/d/id/ATJa7wpfEWdfL2
> 

Re: btrfs check segfaults after flipping 2 Bytes

[Duncan]

Niklas Fischer posted on Wed, 01 Oct 2014 22:29:55 +0200 as excerpted:

> I was trying to determine how btrfs reacts to disk errors, when I
> discovered, that flipping two Bytes, supposedly inside of a file can
> render the filesystem unusable. Here is what I did:
> 
> 1. dd if=/dev/zero of=/dev/sdg2 bs=1M
> 2. mkfs.btrfs /dev/sdg2
> 3. mount /dev/sdg2 /tmp/btrfs
> 4. echo "hello world this is some text" > /tmp/btrfs/hello
> 5. umount /dev/sdg2

Keep in mind that on btrfs, small enough files will not be written to 
file extents but instead will be written directly into the metadata.

That's a small enough file I guess that's what you were seeing, which 
would explain the two instances of the string, since on a single device 
btrfs, metadata is dup mode by default.

That metadata block would then fail checksum, and an attempt would be 
made to use the second copy, which of course would fail it the same way.

And that being the only file in the filesystem, I'd /guess/ (not being a 
developer myself, just a btrfs testing admin and list regular) that 
metadata block is still the original one, which very likely contains 
critical filesystem information as well, thus explaining the mount 
failure when the block failed checksum verify.

In theory at least, with a less synthetic test case there'd be enough 
more metadata on the filesystem that the affected metadata block would be 
further down the chain, and corrupting it wouldn't corrupt critical 
filesystem information as it wouldn't be in the same block.

That might explain the problem, but I don't know enough about btrfs to 
know how reasonable a solution would be.  I guess a btrfs dev should know.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs check segfaults after flipping 2 Bytes

[Brendan Hide]

On 2014/10/02 01:31, Duncan wrote:
> Niklas Fischer posted on Wed, 01 Oct 2014 22:29:55 +0200 as excerpted:
>
>> I was trying to determine how btrfs reacts to disk errors, when I
>> discovered, that flipping two Bytes, supposedly inside of a file can
>> render the filesystem unusable. Here is what I did:
>>
>> 1. dd if=/dev/zero of=/dev/sdg2 bs=1M
>> 2. mkfs.btrfs /dev/sdg2
>> 3. mount /dev/sdg2 /tmp/btrfs
>> 4. echo "hello world this is some text" > /tmp/btrfs/hello
>> 5. umount /dev/sdg2
> Keep in mind that on btrfs, small enough files will not be written to
> file extents but instead will be written directly into the metadata.
>
> That's a small enough file I guess that's what you were seeing, which
> would explain the two instances of the string, since on a single device
> btrfs, metadata is dup mode by default.
>
> That metadata block would then fail checksum, and an attempt would be
> made to use the second copy, which of course would fail it the same way.
At least a very unlikely scenario in production.
> And that being the only file in the filesystem, I'd /guess/ (not being a
> developer myself, just a btrfs testing admin and list regular) that
> metadata block is still the original one, which very likely contains
> critical filesystem information as well, thus explaining the mount
> failure when the block failed checksum verify.
This is a possible use-case for an equivalent to ZFS's ditto blocks. An 
alternative strategy would be to purposefully "sparsify" early metadata 
blocks (this is thinking out loud - whether or not that is a viable or 
easy strategy is debatable).
> In theory at least, with a less synthetic test case there'd be enough
> more metadata on the filesystem that the affected metadata block would be
> further down the chain, and corrupting it wouldn't corrupt critical
> filesystem information as it wouldn't be in the same block.
>
> That might explain the problem, but I don't know enough about btrfs to
> know how reasonable a solution would be.
> [snip]
A reasonable workaround to get the filesystem back into a usable or 
recoverable state might be to mount read-only and ignore checksums. That 
would keep the filesystem intact, though the system has no way to know 
whether or not the folder structures are also corrupt.

I'm not sure if there is a mount option for this use case however. The 
option descriptions for "nodatasum" and "nodatacow" imply that *new* 
checksums are not generated. In this case the checksums already exist.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: btrfs check segfaults after flipping 2 Bytes

[Duncan]

Brendan Hide posted on Thu, 02 Oct 2014 07:51:08 +0200 as excerpted:

> A reasonable workaround to get the filesystem back into a usable or
> recoverable state might be to mount read-only and ignore checksums. That
> would keep the filesystem intact, though the system has no way to know
> whether or not the folder structures are also corrupt.
> 
> I'm not sure if there is a mount option for this use case however. The
> option descriptions for "nodatasum" and "nodatacow" imply that *new*
> checksums are not generated. In this case the checksums already exist.

>From a certain viewpoint that's sort of what btrfs restore does, except 
that it doesn't mount the filesystem; it simply lets you retrieve files 
already there without mounting, if the filesystem isn't mountable.

Also see btrfs check --init-csum-tree, which basically wipes out the 
csums, after which the filesystem should mount, but entirely without 
checksums.  Of course this option kills all the csums and there's 
presently nothing that actually recalculates and rebuilds the csum tree 
(tho there's a very recent patch that I believe adds that functionality), 
so once they're gone, there' gone.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs check segfaults after flipping 2 Bytes

[Brendan Hide]

On 2014/10/02 07:51, Brendan Hide wrote:
> On 2014/10/02 01:31, Duncan wrote:
> [snip]
>
> I'm not sure if there is a mount option for this use case however. The 
> option descriptions for "nodatasum" and "nodatacow" imply that *new* 
> checksums are not generated. In this case the checksums already exist.
>
Looks like btrfsck has a relevant option, albeit likely more destructive 
than absolutely necessary:
--init-csum-tree
    create a new CRC tree.

^ Also, mail was sent as HTML 12 hours ago thus was never delivered. 
Thunderbird has been disciplined.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: btrfs check segfaults after flipping 2 Bytes

[Qu Wenruo]

-------- Original Message --------
Subject: btrfs check segfaults after flipping 2 Bytes
From: Niklas Fischer <niklas@niklasfi.de>
To: <linux-btrfs@vger.kernel.org>
Date: 2014年10月02日 04:29
> Hello,
>
> I was trying to determine how btrfs reacts to disk errors, when I
> discovered, that flipping two Bytes, supposedly inside of a file can
> render the filesystem unusable. Here is what I did:
>
> 1. dd if=/dev/zero of=/dev/sdg2 bs=1M
> 2. mkfs.btrfs /dev/sdg2
> 3. mount /dev/sdg2 /tmp/btrfs
> 4. echo "hello world this is some text" > /tmp/btrfs/hello
> 5. umount /dev/sdg2
>
> this should result in this image [1]
>
> in the following steps /dev/sdg2 is altered. Basically, we want to apply
> s/world/wirld/g
>
> 6. dd if=/dev/sdg2 of=/tmp/sdg2 bs=1M
> 7. edit /tmp/sdg2 in ghex, find & replace all occurences of "world" with
> "wirld" (two occurences found), save as /tmp/sdg2_new
2 problems here.

[1] csum mismatch
As already mentioned by Ducan and Brendan, the csum does not match.

What makes thing much worse, since small file's extent is inlined, the 
data is stored in metadata tree blocks,
and the file system is almost empty so the inline extent lies in the 
*root* leaf of fs_tree.
These two unfortunate facts makes the whole fs_tree corrupted(only one 
leaf, and its cusm dismatch),
which cause btrfs-progs segfault.

The good news is that, the bug in btrfs-progs is already fixed by Wang's 
patch:
https://patchwork.kernel.org/patch/4254631/
So at least, btrfs-progs will not segfault anymore.

[2] two occurences?
So you definitely changed something you should not touch... maybe 
another tree root?

Anyway, almost everything in btrfs is csums, so any manual modification 
may cause problem.

> 8. dd if=/tmp/sdg2_new of=/dev/sdg2 bs=1M
>
> /dev/sdg2 now looks like this [2]
>
> now, when I try mounting /dev/sdg2:
>
> """22:21 root@localhost ~#
> LANG=EN mount /dev/sdg2 /tmp/btrfs/
> mount: mount /dev/sdg2 on /tmp/btrfs failed: Cannot allocate memory"""
>
> ok, so let us try btrfs check
>
> """22:21 root@localhost ~#
> LANG=EN btrfs check /dev/sdg2
> checksum verify failed on 4222976 found F22E71BD wanted CE334502
> checksum verify failed on 4222976 found F22E71BD wanted CE334502
> Csum didn't match
> Checking filesystem on /dev/sdg2
> UUID: ea2843f1-e8aa-420f-acf8-6ef13f6c6753
> checking extents
> Speicherzugriffsfehler (Speicherabzug geschrieben)"""
>
> that last line reads "Segfault, memory dumped". Is this behaviour expected?
>
> Kind regards,
> Niklas
>
> [1] tgz: https://gigamove.rz.rwth-aachen.de/d/id/R5CKikxbYTBdqs
>      raw: https://gigamove.rz.rwth-aachen.de/d/id/VgMXAicS7CGEjt
> [2] tgz: https://gigamove.rz.rwth-aachen.de/d/id/cEebNYne9ppFy4
>      raw: https://gigamove.rz.rwth-aachen.de/d/id/ATJa7wpfEWdfL2
BTW, [2]'s tgz is in fact raw, and raw is tgz....

Thanks,
Qu
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs check segfaults after flipping 2 Bytes

[Duncan]

Qu Wenruo posted on Mon, 06 Oct 2014 10:29:53 +0800 as excerpted:

> 2 problems here.
> 
> [1] csum mismatch As already mentioned by Ducan and Brendan, the csum
> does not match.
> 
> What makes thing much worse, since small file's extent is inlined, the
> data is stored in metadata tree blocks,
> and the file system is almost empty so the inline extent lies in the
> *root* leaf of fs_tree.
> These two unfortunate facts makes the whole fs_tree corrupted(only one
> leaf, and its cusm dismatch),
> which cause btrfs-progs segfault.

So I nailed it. =:^)

> The good news is that, the bug in btrfs-progs is already fixed by Wang's
> patch:
> https://patchwork.kernel.org/patch/4254631/
> So at least, btrfs-progs will not segfault anymore.
> 
> [2] two occurences?
> So you definitely changed something you should not touch... maybe
> another tree root?

1) Single device btrfs, therefore...
2) DUP metadata by default.

3) Small file, therefore...
4) Inlined in (DUP) metadata, therefore...

5) Two occurrences, each a copy of the (metadata-inlined) file in its own 
instance of the (duped) metadata block.

I nailed it again. =:^)

But the connection between the mentioned tree root patch and this 
particular bug had escaped me, so that's useful new information to me 
too, resolving the problem that I spotted but had no clue whether it was 
even fixable.  Thanks.  =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs check segfaults after flipping 2 Bytes

[Qu Wenruo]

-------- Original Message --------
Subject: Re: btrfs check segfaults after flipping 2 Bytes
From: Duncan <1i5t5.duncan@cox.net>
To: <linux-btrfs@vger.kernel.org>
Date: 2014年10月06日 12:10
> Qu Wenruo posted on Mon, 06 Oct 2014 10:29:53 +0800 as excerpted:
>
>> 2 problems here.
>>
>> [1] csum mismatch As already mentioned by Ducan and Brendan, the csum
>> does not match.
>>
>> What makes thing much worse, since small file's extent is inlined, the
>> data is stored in metadata tree blocks,
>> and the file system is almost empty so the inline extent lies in the
>> *root* leaf of fs_tree.
>> These two unfortunate facts makes the whole fs_tree corrupted(only one
>> leaf, and its cusm dismatch),
>> which cause btrfs-progs segfault.
> So I nailed it. =:^)
>
>> The good news is that, the bug in btrfs-progs is already fixed by Wang's
>> patch:
>> https://patchwork.kernel.org/patch/4254631/
>> So at least, btrfs-progs will not segfault anymore.
>>
>> [2] two occurences?
>> So you definitely changed something you should not touch... maybe
>> another tree root?
> 1) Single device btrfs, therefore...
> 2) DUP metadata by default.
Oh, I forgot this, default DUP for metadata...

Thanks
Qu.
>
> 3) Small file, therefore...
> 4) Inlined in (DUP) metadata, therefore...
>
> 5) Two occurrences, each a copy of the (metadata-inlined) file in its own
> instance of the (duped) metadata block.
>
> I nailed it again. =:^)
>
> But the connection between the mentioned tree root patch and this
> particular bug had escaped me, so that's useful new information to me
> too, resolving the problem that I spotted but had no clue whether it was
> even fixable.  Thanks.  =:^)
>