Fwd: btrfsck: checksum verify failed

Fwd: btrfsck: checksum verify failed

[Evert Vorster]

---------- Forwarded message ----------
=46rom: Evert Vorster <evorster@gmail.com>
Date: Sun, Aug 8, 2010 at 1:18 PM
Subject: btrfsck: checksum verify failed
To: linux-btrfs@vger.kernel.org


Hi there.

I have a btrfs on a raw device. (/dev/sda insted of in a partition,
like /dev/sda1 )
The device in question is a USB hard drive with a 1TB size.

The device worked fine for a week or so, and was mounted and unmounted
quite a few times without trouble.

The last time the device was unmounted it was a normal unmount.

Now, when I try to mount the device, I get the following in dmesg:

usb 2-1: new high speed USB device using ehci_hcd and address 4
scsi6 : usb-storage 2-1:1.0
scsi 6:0:0:0: Direct-Access=A0=A0=A0=A0 Seagate=A0 FreeAgent GoFlex 014=
8 PQ: 0 ANSI: 4
sd 6:0:0:0: Attached scsi generic sg2 type 0
sd 6:0:0:0: [sdb] 1953525167 512-byte logical blocks: (1.00 TB/931 GiB)
sd 6:0:0:0: [sdb] Write Protect is off
sd 6:0:0:0: [sdb] Mode Sense: 1c 00 00 00
sd 6:0:0:0: [sdb] Assuming drive cache: write through
sd 6:0:0:0: [sdb] Assuming drive cache: write through
=A0sdb: unknown partition table
sd 6:0:0:0: [sdb] Assuming drive cache: write through
sd 6:0:0:0: [sdb] Attached SCSI disk
device label TeraByte1 devid 1 transid 5352 /dev/sdb
parent transid verify failed on 986800398336 wanted 5352 found 5355
parent transid verify failed on 986800398336 wanted 5352 found 5355
parent transid verify failed on 986800398336 wanted 5352 found 5355
btrfs: open_ctree failed


Also, if I try to fsck the device, this is the output:

bin@dora:~# btrfsck /dev/sdb
parent transid verify failed on 986800398336 wanted 5352 found 5355
parent transid verify failed on 986800398336 wanted 5352 found 5355
parent transid verify failed on 986800398336 wanted 5352 found 5355
btrfsck: disk-io.c:739: open_ctree_fd: Assertion `!(!tree_root->node)' =
failed.
Aborted


I am using v 0.19 of the btrfs-tools, and kernel version 2.6.34.1

While a little sad that I can't get at my data anymore, I do have
backups of it. I also want to use the device again, but want to stop
anybody else running into the same problem. So, if you have any
requests like maybe a dd of some small part of the filesystem to study
and see what went wrong, and maybe how to fix it, I would be happy to
oblige. However, If I hear nothing for a day or so, I'll wipe the
filesystem and use the device again.

Kind regards,
-Evert Vorster-



--
http://magnatune.com - Music shared the way it should be.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Evert Vorster]

Is anyone interested in some part of this filesystem to figure out how
it failed?

Or can I erase and start again?

Kind regards,
-Evert Vorster-

On Sun, Aug 8, 2010 at 8:18 AM, Evert Vorster <evorster@gmail.com> wrote:
> Hi there.
>
> I have a btrfs on a raw device. (/dev/sda insted of in a partition, like
> /dev/sda1 )
> The device in question is a USB hard drive with a 1TB size.
>
> The device worked fine for a week or so, and was mounted and unmounted quite
> a few times without trouble.
>
> The last time the device was unmounted it was a normal unmount.
>
> Now, when I try to mount the device, I get the following in dmesg:
>
> usb 2-1: new high speed USB device using ehci_hcd and address 4
> scsi6 : usb-storage 2-1:1.0
> scsi 6:0:0:0: Direct-Access     Seagate  FreeAgent GoFlex 0148 PQ: 0 ANSI: 4
> sd 6:0:0:0: Attached scsi generic sg2 type 0
> sd 6:0:0:0: [sdb] 1953525167 512-byte logical blocks: (1.00 TB/931 GiB)
> sd 6:0:0:0: [sdb] Write Protect is off
> sd 6:0:0:0: [sdb] Mode Sense: 1c 00 00 00
> sd 6:0:0:0: [sdb] Assuming drive cache: write through
> sd 6:0:0:0: [sdb] Assuming drive cache: write through
>  sdb: unknown partition table
> sd 6:0:0:0: [sdb] Assuming drive cache: write through
> sd 6:0:0:0: [sdb] Attached SCSI disk
> device label TeraByte1 devid 1 transid 5352 /dev/sdb
> parent transid verify failed on 986800398336 wanted 5352 found 5355
> parent transid verify failed on 986800398336 wanted 5352 found 5355
> parent transid verify failed on 986800398336 wanted 5352 found 5355
> btrfs: open_ctree failed
>
>
> Also, if I try to fsck the device, this is the output:
>
> bin@dora:~# btrfsck /dev/sdb
> parent transid verify failed on 986800398336 wanted 5352 found 5355
> parent transid verify failed on 986800398336 wanted 5352 found 5355
> parent transid verify failed on 986800398336 wanted 5352 found 5355
> btrfsck: disk-io.c:739: open_ctree_fd: Assertion `!(!tree_root->node)'
> failed.
> Aborted
>
>
> I am using v 0.19 of the btrfs-tools, and kernel version 2.6.34.1
>
> While a little sad that I can't get at my data anymore, I do have backups of
> it. I also want to use the device again, but want to stop anybody else
> running into the same problem. So, if you have any requests like maybe a dd
> of some small part of the filesystem to study and see what went wrong, and
> maybe how to fix it, I would be happy to oblige. However, If I hear nothing
> for a day or so, I'll wipe the filesystem and use the device again.
>
> Kind regards,
> -Evert Vorster-
>



-- 
http://magnatune.com - Music shared the way it should be.

Re: btrfsck: checksum verify failed

[Chris Mason]

On Fri, Nov 13, 2009 at 12:44:47AM +0300, Alexander Beregalov wrote:
> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> > On Thu, Nov 12, 2009 at 08:41:44PM +0300, Alexander Beregalov wrote=
:
> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> >> > On Thu, Nov 12, 2009 at 06:43:00PM +0300, Alexander Beregalov wr=
ote:
> >> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> >> >> > On Thu, Nov 12, 2009 at 05:16:34PM +0300, Alexander Beregalov=
 wrote:
> >> >> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> >> >> >> > On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Berega=
lov wrote:
> >> >> >> >> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum=
 verify
> >> >> >> >> failed";echo; done
> >> >> >> >> checksum verify failed on 31945617408 wanted 6607632D fou=
nd 9CC3ED0
> >> >> >> >> checksum verify failed on 31945617408 wanted 6607632D fou=
nd AFBBF41
> >> >> >> >
> >> >> >> > Was your filesystem mounted at the time? =A0It's strange t=
hat the checksum
> >> >> >> > keeps changing, that points to the data in the block chang=
ing.
> >> >> >>
> >> >> >> It was not.
> >> >> >> Any tips how I can try to find a root cause?
> >> >> >> It is raid0 of two disks.
> >> >> >
> >> >> > Well, I'd start by looking for anything else that could be to=
uching the
> >> >> > FS or the devices. =A0You shouldn't see more than two differe=
nt csums in this
> >> >> > configuration (one from each disk).
> >> >> I always see only these two blocks in output.
> >> >
> >> > The wanted is always the same but the found is always different?
> >>
> >> Yes, exactly. The bits marked as X look as random.
> >
> > Ok, I've added a new tool to the btrfs-unstable-progs repo. =A0It i=
s
> > called btrfs-map-logical and it will print the logical->physical ma=
pping
> > for any metadata or data block in the FS. =A0The FS should not be m=
ounted
> > when you run this program...it'll mostly work mounted but the resul=
ts
> > won't be 100% accurate.
> >
> > For your case:
> >
> > btrfs-map-logical -l 31945617408 -c 1 -o mirror1 /dev/sde
> > btrfs-map-logical -l 31945617408 -c 2 -o mirror2 /dev/sde
> >
> > This will print to the screen the mappings for that block and also =
save
> > the contents of the first mirror of the block to a file called mirr=
or1.
> > The second command will save the contents of the second mirror.
> >
> > If the FS isn't mounted and isn't being changed by someone else, yo=
u
> > should get exactly the same mappings every time and exactly the sam=
e
> > contents in the files mirror1 and mirror2 every time.
> >
> > Just let me know if you have any problems. =A0The most recent commi=
t in
> > the btrfs-progs repo is:
> >
> > commit ab8fb4c99516c186641bda1dbc0e788f68b4dc77
> > Author: Chris Mason <chris.mason@oracle.com>
> > Date: =A0 Thu Nov 12 14:34:09 2009 -0500
> >
> > =A0 =A0Add btrfs-map-logical program to map and read logical block =
numbers
> >
> > If you check kernel.org and this commit isn't there, just try again=
 ;)
> > The mirror takes a few minutes to update. =A0I'm pretty sure it goe=
s
> > faster if you hit reload over and over again, but that's just a gue=
ss.
> >
> > -chris
> >
>=20
> sh-4.0$  sudo ./btrfs-map-logical -l 31945617408 -c 1 -o /tmp/mirror1
> /dev/sde; md5sum /tmp/mirror1
> mirror 1 logical 31945617408 physical 16913231872 device /dev/sde
> mirror 2 logical 31945617408 physical 16893308928 device /dev/sdf
> bcce5ab8cc65c948650376e936952547  /tmp/mirror1
>=20
> sh-4.0$  sudo ./btrfs-map-logical -l 31945617408 -c 2 -o /tmp/mirror2
> /dev/sde; md5sum /tmp/mirror2
> mirror 1 logical 31945617408 physical 16913231872 device /dev/sde
> mirror 2 logical 31945617408 physical 16893308928 device /dev/sdf
> 347034331d8879096e262a736fd02ba0  /tmp/mirror2
>=20
> sh-4.0$  sudo ./btrfs-map-logical -l 31945617408 -c 1 -o /tmp/mirror1
> /dev/sde; md5sum /tmp/mirror1
> mirror 1 logical 31945617408 physical 16913231872 device /dev/sde
> mirror 2 logical 31945617408 physical 16893308928 device /dev/sdf
> bcce5ab8cc65c948650376e936952547  /tmp/mirror1
>=20
> sh-4.0$  sudo ./btrfs-map-logical -l 31945617408 -c 2 -o /tmp/mirror2
> /dev/sde; md5sum /tmp/mirror2
> mirror 1 logical 31945617408 physical 16913231872 device /dev/sde
> mirror 2 logical 31945617408 physical 16893308928 device /dev/sdf
> 347034331d8879096e262a736fd02ba0  /tmp/mirror2

You're getting different answers for the two blocks, which is ok (one
could be corrupt).  What doesn't make sense is that btrfsck isn't
dealing with this correctly.  When you read both copies again, you get
the same answers as the first time, so this looks like a btrfsck bug to
me.

I'll try to reproduce.  If you're in a hurry, you could do something
like this (with the FS unmounted).  Keep a good copy of both mirror1 an=
d
mirror2

dd if=3D/tmp/mirror1 of=3D/dev/sde bs=3D1 seek=3D16913231872 count=3D40=
96
dd if=3D/tmp/mirror1 of=3D/dev/sdf bs=3D1 seek=3D16893308928 count=3D40=
96
sync

btrfsck.  If that doesn't work:

dd if=3D/tmp/mirror2 of=3D/dev/sde bs=3D1 seek=3D16913231872 count=3D40=
96
dd if=3D/tmp/mirror2 of=3D/dev/sdf bs=3D1 seek=3D16893308928 count=3D40=
96

btrfsck again.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Alexander Beregalov]

2009/11/12 Chris Mason <chris.mason@oracle.com>:
> On Thu, Nov 12, 2009 at 08:41:44PM +0300, Alexander Beregalov wrote:
>> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
>> > On Thu, Nov 12, 2009 at 06:43:00PM +0300, Alexander Beregalov wrot=
e:
>> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
>> >> > On Thu, Nov 12, 2009 at 05:16:34PM +0300, Alexander Beregalov w=
rote:
>> >> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
>> >> >> > On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Beregalo=
v wrote:
>> >> >> >> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum v=
erify
>> >> >> >> failed";echo; done
>> >> >> >> checksum verify failed on 31945617408 wanted 6607632D found=
 9CC3ED0
>> >> >> >> checksum verify failed on 31945617408 wanted 6607632D found=
 AFBBF41
>> >> >> >
>> >> >> > Was your filesystem mounted at the time? =C2=A0It's strange =
that the checksum
>> >> >> > keeps changing, that points to the data in the block changin=
g.
>> >> >>
>> >> >> It was not.
>> >> >> Any tips how I can try to find a root cause?
>> >> >> It is raid0 of two disks.
>> >> >
>> >> > Well, I'd start by looking for anything else that could be touc=
hing the
>> >> > FS or the devices. =C2=A0You shouldn't see more than two differ=
ent csums in this
>> >> > configuration (one from each disk).
>> >> I always see only these two blocks in output.
>> >
>> > The wanted is always the same but the found is always different?
>>
>> Yes, exactly. The bits marked as X look as random.
>
> Ok, I've added a new tool to the btrfs-unstable-progs repo. =C2=A0It =
is
> called btrfs-map-logical and it will print the logical->physical mapp=
ing
> for any metadata or data block in the FS. =C2=A0The FS should not be =
mounted
> when you run this program...it'll mostly work mounted but the results
> won't be 100% accurate.
>
> For your case:
>
> btrfs-map-logical -l 31945617408 -c 1 -o mirror1 /dev/sde
> btrfs-map-logical -l 31945617408 -c 2 -o mirror2 /dev/sde
>
> This will print to the screen the mappings for that block and also sa=
ve
> the contents of the first mirror of the block to a file called mirror=
1.
> The second command will save the contents of the second mirror.
>
> If the FS isn't mounted and isn't being changed by someone else, you
> should get exactly the same mappings every time and exactly the same
> contents in the files mirror1 and mirror2 every time.
>
> Just let me know if you have any problems. =C2=A0The most recent comm=
it in
> the btrfs-progs repo is:
>
> commit ab8fb4c99516c186641bda1dbc0e788f68b4dc77
> Author: Chris Mason <chris.mason@oracle.com>
> Date: =C2=A0 Thu Nov 12 14:34:09 2009 -0500
>
> =C2=A0 =C2=A0Add btrfs-map-logical program to map and read logical bl=
ock numbers
>
> If you check kernel.org and this commit isn't there, just try again ;=
)
> The mirror takes a few minutes to update. =C2=A0I'm pretty sure it go=
es
> faster if you hit reload over and over again, but that's just a guess=
=2E
>
> -chris
>

sh-4.0$  sudo ./btrfs-map-logical -l 31945617408 -c 1 -o /tmp/mirror1
/dev/sde; md5sum /tmp/mirror1
mirror 1 logical 31945617408 physical 16913231872 device /dev/sde
mirror 2 logical 31945617408 physical 16893308928 device /dev/sdf
bcce5ab8cc65c948650376e936952547  /tmp/mirror1

sh-4.0$  sudo ./btrfs-map-logical -l 31945617408 -c 2 -o /tmp/mirror2
/dev/sde; md5sum /tmp/mirror2
mirror 1 logical 31945617408 physical 16913231872 device /dev/sde
mirror 2 logical 31945617408 physical 16893308928 device /dev/sdf
347034331d8879096e262a736fd02ba0  /tmp/mirror2

sh-4.0$  sudo ./btrfs-map-logical -l 31945617408 -c 1 -o /tmp/mirror1
/dev/sde; md5sum /tmp/mirror1
mirror 1 logical 31945617408 physical 16913231872 device /dev/sde
mirror 2 logical 31945617408 physical 16893308928 device /dev/sdf
bcce5ab8cc65c948650376e936952547  /tmp/mirror1

sh-4.0$  sudo ./btrfs-map-logical -l 31945617408 -c 2 -o /tmp/mirror2
/dev/sde; md5sum /tmp/mirror2
mirror 1 logical 31945617408 physical 16913231872 device /dev/sde
mirror 2 logical 31945617408 physical 16893308928 device /dev/sdf
347034331d8879096e262a736fd02ba0  /tmp/mirror2

Does that mean that the problem is not in FS and I need to check
discs, IO controller, memory.

The same result with mount FS, but when I unmounted it there was a bug:

BUG: unable to handle kernel paging request at 6b6b6c9f
IP: [<f9a3c696>] caching_kthread+0x286/0x410 [btrfs]
*pde =3D 00000000
Oops: 0002 [#1]
last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed
Modules linked in: hwmon_vid btrfs zlib_deflate zlib_inflate sata_sil
i2c_nforce2

Pid: 15258, comm: btrfs-cache-311 Not tainted (2.6.32-rc6-00346-gaa021b=
a #4)
EIP: 0060:[<f9a3c696>] EFLAGS: 00010286 CPU: 0
EIP is at caching_kthread+0x286/0x410 [btrfs]
EAX: 6b6b6b6b EBX: f6f9e098 ECX: 0000000e EDX: effc5f00
ESI: f4d95ab0 EDI: f26f4000 EBP: e84e7f98 ESP: e84e7f20
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process btrfs-cache-311 (pid: 15258, ti=3De84e6000 task=3Df6450000 task=
=2Eti=3De84e6000)
Stack:
 00000000 ffffffff 81c00000 00000007 4324c000 effc5508 effc54b8 f26f5bd=
0
<0> f62c20c0 000b6000 00000000 effc54b0 ffffffff ffffffff 00000032 e7b1=
4678
<0> f26f4000 f4d95ab0 c0007f98 00074324 00a80000 00000010 00000000 0743=
24d0
Call Trace:
 [<f9a3c410>] ? caching_kthread+0x0/0x410 [btrfs]
 [<c103ce4c>] ? kthread+0x6c/0x80
 [<c103cde0>] ? kthread+0x0/0x80
 [<c1003513>] ? kernel_thread_helper+0x7/0x14
Code: c7 8b 45 9c b9 01 00 00 00 ba 03 00 00 00 c7 04 24 00 00 00 00
e8 fb 4d 5e c7 8b 45 b4 e8 53 d5 ff ff 8b 75 cc 8b 86 a0 00 00 00 <ff>
88 34 01 00 00 31 c0 83 c4 6c 5b 5e 5f c9 c3 66 90 77 09 3b
EIP: [<f9a3c696>] caching_kthread+0x286/0x410 [btrfs] SS:ESP 0068:e84e7=
f20
CR2: 000000006b6b6c9f
---[ end trace 72d6516d4c697ae6 ]---
device: 'btrfs-4': device_unregister
device: 'btrfs-4': device_create_release
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Chris Mason]

On Thu, Nov 12, 2009 at 08:41:44PM +0300, Alexander Beregalov wrote:
> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> > On Thu, Nov 12, 2009 at 06:43:00PM +0300, Alexander Beregalov wrote=
:
> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> >> > On Thu, Nov 12, 2009 at 05:16:34PM +0300, Alexander Beregalov wr=
ote:
> >> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> >> >> > On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Beregalov=
 wrote:
> >> >> >> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum ve=
rify
> >> >> >> failed";echo; done
> >> >> >> checksum verify failed on 31945617408 wanted 6607632D found =
9CC3ED0
> >> >> >> checksum verify failed on 31945617408 wanted 6607632D found =
AFBBF41
> >> >> >
> >> >> > Was your filesystem mounted at the time? =A0It's strange that=
 the checksum
> >> >> > keeps changing, that points to the data in the block changing=
=2E
> >> >>
> >> >> It was not.
> >> >> Any tips how I can try to find a root cause?
> >> >> It is raid0 of two disks.
> >> >
> >> > Well, I'd start by looking for anything else that could be touch=
ing the
> >> > FS or the devices. =A0You shouldn't see more than two different =
csums in this
> >> > configuration (one from each disk).
> >> I always see only these two blocks in output.
> >
> > The wanted is always the same but the found is always different?
>=20
> Yes, exactly. The bits marked as X look as random.

Ok, I've added a new tool to the btrfs-unstable-progs repo.  It is
called btrfs-map-logical and it will print the logical->physical mappin=
g
for any metadata or data block in the FS.  The FS should not be mounted
when you run this program...it'll mostly work mounted but the results
won't be 100% accurate.

=46or your case:

btrfs-map-logical -l 31945617408 -c 1 -o mirror1 /dev/sde
btrfs-map-logical -l 31945617408 -c 2 -o mirror2 /dev/sde

This will print to the screen the mappings for that block and also save
the contents of the first mirror of the block to a file called mirror1.
The second command will save the contents of the second mirror.

If the FS isn't mounted and isn't being changed by someone else, you
should get exactly the same mappings every time and exactly the same
contents in the files mirror1 and mirror2 every time.

Just let me know if you have any problems.  The most recent commit in
the btrfs-progs repo is:

commit ab8fb4c99516c186641bda1dbc0e788f68b4dc77
Author: Chris Mason <chris.mason@oracle.com>
Date:   Thu Nov 12 14:34:09 2009 -0500

    Add btrfs-map-logical program to map and read logical block numbers

If you check kernel.org and this commit isn't there, just try again ;)
The mirror takes a few minutes to update.  I'm pretty sure it goes
faster if you hit reload over and over again, but that's just a guess.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Alexander Beregalov]

2009/11/12 Chris Mason <chris.mason@oracle.com>:
> On Thu, Nov 12, 2009 at 06:43:00PM +0300, Alexander Beregalov wrote:
>> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
>> > On Thu, Nov 12, 2009 at 05:16:34PM +0300, Alexander Beregalov wrot=
e:
>> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
>> >> > On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Beregalov w=
rote:
>> >> >> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum veri=
fy
>> >> >> failed";echo; done
>> >> >> checksum verify failed on 31945617408 wanted 6607632D found 9C=
C3ED0
>> >> >> checksum verify failed on 31945617408 wanted 6607632D found AF=
BBF41
>> >> >
>> >> > Was your filesystem mounted at the time? =C2=A0It's strange tha=
t the checksum
>> >> > keeps changing, that points to the data in the block changing.
>> >>
>> >> It was not.
>> >> Any tips how I can try to find a root cause?
>> >> It is raid0 of two disks.
>> >
>> > Well, I'd start by looking for anything else that could be touchin=
g the
>> > FS or the devices. =C2=A0You shouldn't see more than two different=
 csums in this
>> > configuration (one from each disk).
>> I always see only these two blocks in output.
>
> The wanted is always the same but the found is always different?

Yes, exactly. The bits marked as X look as random.

>
>> >
>> > Somehow the disk is giving us different answers each time, which p=
oints
>> > to a problem in the storage stack.
>> Hm, I will try to check discs by "badblocks" with few cycles of writ=
e-read.
>> >
>> > Otherwise it is possible that block 31945617408 is multiply linked=
 and
>> > is actually in use somehow else. =C2=A0btrfsck is a readonly opera=
tion
>> > though, it doesn't change the FS while it is running.
>>
>> Can it be a bug in the filesystem itself - in checksum calculation
>> code, wrong pointer to data or something else?
>
> We can check, =C2=A0I'll make a program to map a logical block number=
 to the
> physical sector.

Ok, thanks, I will not reboot it.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Chris Mason]

On Thu, Nov 12, 2009 at 06:43:00PM +0300, Alexander Beregalov wrote:
> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> > On Thu, Nov 12, 2009 at 05:16:34PM +0300, Alexander Beregalov wrote=
:
> >> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> >> > On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Beregalov wr=
ote:
> >> >> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum verif=
y
> >> >> failed";echo; done
> >> >> checksum verify failed on 31945617408 wanted 6607632D found 9CC=
3ED0
> >> >> checksum verify failed on 31945617408 wanted 6607632D found AFB=
BF41
> >> >
> >> > Was your filesystem mounted at the time? =A0It's strange that th=
e checksum
> >> > keeps changing, that points to the data in the block changing.
> >>
> >> It was not.
> >> Any tips how I can try to find a root cause?
> >> It is raid0 of two disks.
> >
> > Well, I'd start by looking for anything else that could be touching=
 the
> > FS or the devices. =A0You shouldn't see more than two different csu=
ms in this
> > configuration (one from each disk).
> I always see only these two blocks in output.

The wanted is always the same but the found is always different?

> >
> > Somehow the disk is giving us different answers each time, which po=
ints
> > to a problem in the storage stack.
> Hm, I will try to check discs by "badblocks" with few cycles of write=
-read.
> >
> > Otherwise it is possible that block 31945617408 is multiply linked =
and
> > is actually in use somehow else. =A0btrfsck is a readonly operation
> > though, it doesn't change the FS while it is running.
>=20
> Can it be a bug in the filesystem itself - in checksum calculation
> code, wrong pointer to data or something else?

We can check,  I'll make a program to map a logical block number to the
physical sector.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Alexander Beregalov]

2009/11/12 Chris Mason <chris.mason@oracle.com>:
> On Thu, Nov 12, 2009 at 05:16:34PM +0300, Alexander Beregalov wrote:
>> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
>> > On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Beregalov wrot=
e:
>> >> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum verify
>> >> failed";echo; done
>> >> checksum verify failed on 31945617408 wanted 6607632D found 9CC3E=
D0
>> >> checksum verify failed on 31945617408 wanted 6607632D found AFBBF=
41
>> >
>> > Was your filesystem mounted at the time? =C2=A0It's strange that t=
he checksum
>> > keeps changing, that points to the data in the block changing.
>>
>> It was not.
>> Any tips how I can try to find a root cause?
>> It is raid0 of two disks.
>
> Well, I'd start by looking for anything else that could be touching t=
he
> FS or the devices. =C2=A0You shouldn't see more than two different cs=
ums in this
> configuration (one from each disk).
I always see only these two blocks in output.
>
> Somehow the disk is giving us different answers each time, which poin=
ts
> to a problem in the storage stack.
Hm, I will try to check discs by "badblocks" with few cycles of write-r=
ead.
>
> Otherwise it is possible that block 31945617408 is multiply linked an=
d
> is actually in use somehow else. =C2=A0btrfsck is a readonly operatio=
n
> though, it doesn't change the FS while it is running.

Can it be a bug in the filesystem itself - in checksum calculation
code, wrong pointer to data or something else?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Chris Mason]

On Thu, Nov 12, 2009 at 05:16:34PM +0300, Alexander Beregalov wrote:
> 2009/11/12 Chris Mason <chris.mason@oracle.com>:
> > On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Beregalov wrote=
:
> >> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum verify
> >> failed";echo; done
> >> checksum verify failed on 31945617408 wanted 6607632D found 9CC3ED=
0
> >> checksum verify failed on 31945617408 wanted 6607632D found AFBBF4=
1
> >
> > Was your filesystem mounted at the time? =A0It's strange that the c=
hecksum
> > keeps changing, that points to the data in the block changing.
>=20
> It was not.
> Any tips how I can try to find a root cause?
> It is raid0 of two disks.

Well, I'd start by looking for anything else that could be touching the
=46S or the devices.  You shouldn't see more than two different csums i=
n this
configuration (one from each disk).

Somehow the disk is giving us different answers each time, which points
to a problem in the storage stack.

Otherwise it is possible that block 31945617408 is multiply linked and
is actually in use somehow else.  btrfsck is a readonly operation
though, it doesn't change the FS while it is running.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Alexander Beregalov]

2009/11/12 Chris Mason <chris.mason@oracle.com>:
> On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Beregalov wrote:
>> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum verify
>> failed";echo; done
>> checksum verify failed on 31945617408 wanted 6607632D found 9CC3ED0
>> checksum verify failed on 31945617408 wanted 6607632D found AFBBF41
>
> Was your filesystem mounted at the time? =C2=A0It's strange that the =
checksum
> keeps changing, that points to the data in the block changing.

It was not.
Any tips how I can try to find a root cause?
It is raid0 of two disks.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfsck: checksum verify failed

[Chris Mason]

On Thu, Nov 12, 2009 at 05:08:53AM +0300, Alexander Beregalov wrote:
> # for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum verify
> failed";echo; done
> checksum verify failed on 31945617408 wanted 6607632D found 9CC3ED0
> checksum verify failed on 31945617408 wanted 6607632D found AFBBF41

Was your filesystem mounted at the time?  It's strange that the checksum
keeps changing, that points to the data in the block changing.

-chris

btrfsck: checksum verify failed

[Alexander Beregalov]

# for i in `seq 1 20`; do btrfsck /dev/sde|grep "checksum verify
failed";echo; done
checksum verify failed on 31945617408 wanted 6607632D found 9CC3ED0
checksum verify failed on 31945617408 wanted 6607632D found AFBBF41

checksum verify failed on 31945617408 wanted 6607632D found A4F0ED0
checksum verify failed on 31945617408 wanted 6607632D found B7E8F41

checksum verify failed on 31945617408 wanted 6607632D found A83DED0
checksum verify failed on 31945617408 wanted 6607632D found BB35F41

checksum verify failed on 31945617408 wanted 6607632D found A0D5ED0
checksum verify failed on 31945617408 wanted 6607632D found B3CDF41

checksum verify failed on 31945617408 wanted 6607632D found A37BED0
checksum verify failed on 31945617408 wanted 6607632D found B673F41

checksum verify failed on 31945617408 wanted 6607632D found AB7FED0
checksum verify failed on 31945617408 wanted 6607632D found BE77F41

checksum verify failed on 31945617408 wanted 6607632D found AB73ED0
checksum verify failed on 31945617408 wanted 6607632D found BE6BF41

checksum verify failed on 31945617408 wanted 6607632D found A76DED0
checksum verify failed on 31945617408 wanted 6607632D found BA65F41

checksum verify failed on 31945617408 wanted 6607632D found 98E6ED0
checksum verify failed on 31945617408 wanted 6607632D found ABDEF41

checksum verify failed on 31945617408 wanted 6607632D found 9038ED0
checksum verify failed on 31945617408 wanted 6607632D found A330F41

checksum verify failed on 31945617408 wanted 6607632D found A150ED0
checksum verify failed on 31945617408 wanted 6607632D found B448F41

checksum verify failed on 31945617408 wanted 6607632D found 916CED0
checksum verify failed on 31945617408 wanted 6607632D found A464F41

checksum verify failed on 31945617408 wanted 6607632D found 9277ED0
checksum verify failed on 31945617408 wanted 6607632D found A56FF41

checksum verify failed on 31945617408 wanted 6607632D found A9C4ED0
checksum verify failed on 31945617408 wanted 6607632D found BCBCF41

checksum verify failed on 31945617408 wanted 6607632D found A81CED0
checksum verify failed on 31945617408 wanted 6607632D found BB14F41

checksum verify failed on 31945617408 wanted 6607632D found A886ED0
checksum verify failed on 31945617408 wanted 6607632D found BB7EF41

checksum verify failed on 31945617408 wanted 6607632D found 944AED0
checksum verify failed on 31945617408 wanted 6607632D found A742F41

It looks like
10XX XXXXXXXX XXXX1110 11010000
and
10XX XXXXXXXX XXXX1111 01000001