btrfs no csum found for inode X start 0

btrfs no csum found for inode X start 0

[Leszek Ciesielski]

(My previous post seems to have been discarded because of the
attachment size, I'm resending it without the dmesg output - which can
be found @ http://pastebin.com/T0J3z59j )

Hi,

yesterday I updated my kernel (clean clone from
mason/btrfs-unstable.gi), pulling in the single latest change I have
been missing ( http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=3f6fae9559225741c91f1320090b285da1413290
) and adding my patch from http://patchwork.kernel.org/patch/81547/ .
Previous kernel version (without my patch - could this be my fault?)
has been running fine for 14 days, but after recompiling and
rebooting, my dmesg output is full of "btrfs no csum found for inode
386 start 0" and "btrfs csum failed ino 386 extent 65191274496 csum
1851253866 wanted 0 mirror 1" and "btrfs csum failed ino 82619 off
8749056 csum 2686054019 private 0", repeating with different values.
Also, accessing pretty much any file ends with "read error" - only
small text files remained readable. Newly written files seem to behave
correctly. I have tried reverting back to the version from 10.02.2010
(last change on 04.02, without my patch) that worked well before, but
now it also spits out the same errors and files remain unreadable.
Dmesg output is attached. Anything more I can do to help diagnose the
problem?

Regards,

Leszek 'skolima' Ciesielski

Re: btrfs no csum found for inode X start 0

[Leszek Ciesielski]

I have changed the btrfs code to ignore checksum failures and now I
can read files correctly from the filesystem. Also, moving them onto
another volume and then back into btrfs fixes the checksums and no
more errors are reported for the file in question.

Quick and dirty code I used for getting my files out:

diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index a11a320..d6e6aa9 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -140,8 +140,9 @@ static int check_compressed_csum(struct inode *inode,
                               "wanted %u mirror %d\n", inode->i_ino,
                               (unsigned long long)disk_start,
                               csum, *cb_sum, cb->mirror_num);
-                       ret = -EIO;
-                       goto fail;
+                       /*ret = -EIO;
+                       goto fail;*/
+                       printk("btrfs ignoring compressed csum mismatch");
                }
                cb_sum++;

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 4deb280..f1572ce 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -1955,8 +1955,9 @@ static int btrfs_readpage_end_io_hook(struct
page *page, u64 start, u64 end,

        csum = btrfs_csum_data(root, kaddr + offset, csum,  end - start + 1);
        btrfs_csum_final(csum, (char *)&csum);
-       if (csum != private)
-               goto zeroit;
+       if (csum != private && printk_ratelimit())
+               printk(KERN_INFO "btrfs ignoring csum mismatch");
+//             goto zeroit;

        kunmap_atomic(kaddr, KM_USER0);
 good:


On Thu, Feb 25, 2010 at 10:34 AM, Leszek Ciesielski <skolima@gmail.com> wrote:
> (My previous post seems to have been discarded because of the
> attachment size, I'm resending it without the dmesg output - which can
> be found @ http://pastebin.com/T0J3z59j )
>
> Hi,
>
> yesterday I updated my kernel (clean clone from
> mason/btrfs-unstable.gi), pulling in the single latest change I have
> been missing ( http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=3f6fae9559225741c91f1320090b285da1413290
> ) and adding my patch from http://patchwork.kernel.org/patch/81547/ .
> Previous kernel version (without my patch - could this be my fault?)
> has been running fine for 14 days, but after recompiling and
> rebooting, my dmesg output is full of "btrfs no csum found for inode
> 386 start 0" and "btrfs csum failed ino 386 extent 65191274496 csum
> 1851253866 wanted 0 mirror 1" and "btrfs csum failed ino 82619 off
> 8749056 csum 2686054019 private 0", repeating with different values.
> Also, accessing pretty much any file ends with "read error" - only
> small text files remained readable. Newly written files seem to behave
> correctly. I have tried reverting back to the version from 10.02.2010
> (last change on 04.02, without my patch) that worked well before, but
> now it also spits out the same errors and files remain unreadable.
> Dmesg output is attached. Anything more I can do to help diagnose the
> problem?
>
> Regards,
>
> Leszek 'skolima' Ciesielski
>

Re: btrfs no csum found for inode X start 0

[Mike Fedyk]

On Thu, Feb 25, 2010 at 12:52 PM, Leszek Ciesielski <skolima@gmail.com> wrote:
> I have changed the btrfs code to ignore checksum failures and now I
> can read files correctly from the filesystem. Also, moving them onto
> another volume and then back into btrfs fixes the checksums and no
> more errors are reported for the file in question.
>
> Quick and dirty code I used for getting my files out:

Yes, but did you verify your data?

Re: btrfs no csum found for inode X start 0

[Chris Mason]

On Thu, Feb 25, 2010 at 10:34:22AM +0100, Leszek Ciesielski wrote:
> (My previous post seems to have been discarded because of the
> attachment size, I'm resending it without the dmesg output - which can
> be found @ http://pastebin.com/T0J3z59j )
> 
> Hi,
> 
> yesterday I updated my kernel (clean clone from
> mason/btrfs-unstable.gi), pulling in the single latest change I have
> been missing ( http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=3f6fae9559225741c91f1320090b285da1413290
> ) and adding my patch from http://patchwork.kernel.org/patch/81547/ .
> Previous kernel version (without my patch - could this be my fault?)
> has been running fine for 14 days, but after recompiling and
> rebooting, my dmesg output is full of "btrfs no csum found for inode
> 386 start 0" and "btrfs csum failed ino 386 extent 65191274496 csum
> 1851253866 wanted 0 mirror 1" and "btrfs csum failed ino 82619 off
> 8749056 csum 2686054019 private 0",

I don't think your patch alone could have caused this.  Has anything
else strange been happening on this machine?

The fact that all your files are wrong is especially strange.  Have you
ever mounted this FS with mount -o nodatasum?

-chris

Re: btrfs no csum found for inode X start 0

[Leszek Ciesielski]

On Fri, Feb 26, 2010 at 1:45 AM, Chris Mason <chris.mason@oracle.com> w=
rote:
> On Thu, Feb 25, 2010 at 10:34:22AM +0100, Leszek Ciesielski wrote:
>> (My previous post seems to have been discarded because of the
>> attachment size, I'm resending it without the dmesg output - which c=
an
>> be found @ http://pastebin.com/T0J3z59j )
>>
>> Hi,
>>
>> yesterday I updated my kernel (clean clone from
>> mason/btrfs-unstable.gi), pulling in the single latest change I have
>> been missing ( http://git.kernel.org/?p=3Dlinux/kernel/git/mason/btr=
fs-unstable.git;a=3Dcommit;h=3D3f6fae9559225741c91f1320090b285da1413290
>> ) and adding my patch from http://patchwork.kernel.org/patch/81547/ =
=2E
>> Previous kernel version (without my patch - could this be my fault?)
>> has been running fine for 14 days, but after recompiling and
>> rebooting, my dmesg output is full of "btrfs no csum found for inode
>> 386 start 0" and "btrfs csum failed ino 386 extent 65191274496 csum
>> 1851253866 wanted 0 mirror 1" and "btrfs csum failed ino 82619 off
>> 8749056 csum 2686054019 private 0",
>
> Yes, but did you verify your data?

Part of the data stored on the volume consisted of video recordings -
after copying out and back onto the volume, they play back fine,
without video or audio glitches. Which I am aware does not mean they
are intact, just "good enough to work". I had also some important data
there, which is backed up to another location - I will verify it's
integrity with rsync during the weekend.

>
> I don't think your patch alone could have caused this. =A0Has anythin=
g
> else strange been happening on this machine?

Not really. The FS was created with metadata=3Dmirror data=3Dmirror on =
a
single drive, then a second (larger) drive was added and the fs was
rebalanced. Compression is enabled. No problems until the last kernel
update. After the recovery - no new csum failures.

>
> The fact that all your files are wrong is especially strange. =A0Have=
 you
> ever mounted this FS with mount -o nodatasum?

Only once, after the problem occured - yesterday, when I was trying to
copy the data out. Only read access was performed (although the fs was
mounted rw).

Regards,

Leszek 'skolima' Ciesielski
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs no csum found for inode X start 0

[Chris Mason]

On Fri, Feb 26, 2010 at 11:51:35AM +0100, Leszek Ciesielski wrote:
> On Fri, Feb 26, 2010 at 1:45 AM, Chris Mason <chris.mason@oracle.com>=
 wrote:
> > On Thu, Feb 25, 2010 at 10:34:22AM +0100, Leszek Ciesielski wrote:
> >> (My previous post seems to have been discarded because of the
> >> attachment size, I'm resending it without the dmesg output - which=
 can
> >> be found @ http://pastebin.com/T0J3z59j )
> >>
> >> Hi,
> >>
> >> yesterday I updated my kernel (clean clone from
> >> mason/btrfs-unstable.gi), pulling in the single latest change I ha=
ve
> >> been missing ( http://git.kernel.org/?p=3Dlinux/kernel/git/mason/b=
trfs-unstable.git;a=3Dcommit;h=3D3f6fae9559225741c91f1320090b285da14132=
90
> >> ) and adding my patch from http://patchwork.kernel.org/patch/81547=
/ .
> >> Previous kernel version (without my patch - could this be my fault=
?)
> >> has been running fine for 14 days, but after recompiling and
> >> rebooting, my dmesg output is full of "btrfs no csum found for ino=
de
> >> 386 start 0" and "btrfs csum failed ino 386 extent 65191274496 csu=
m
> >> 1851253866 wanted 0 mirror 1" and "btrfs csum failed ino 82619 off
> >> 8749056 csum 2686054019 private 0",
> >
> > Yes, but did you verify your data?
>=20
> Part of the data stored on the volume consisted of video recordings -
> after copying out and back onto the volume, they play back fine,
> without video or audio glitches. Which I am aware does not mean they
> are intact, just "good enough to work". I had also some important dat=
a
> there, which is backed up to another location - I will verify it's
> integrity with rsync during the weekend.
>=20
> >
> > I don't think your patch alone could have caused this. =A0Has anyth=
ing
> > else strange been happening on this machine?
>=20
> Not really. The FS was created with metadata=3Dmirror data=3Dmirror o=
n a
> single drive, then a second (larger) drive was added and the fs was
> rebalanced. Compression is enabled. No problems until the last kernel
> update. After the recovery - no new csum failures.

Ok, what does btrfsck say about the FS now?

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs no csum found for inode X start 0

[Leszek Ciesielski]

On Fri, Feb 26, 2010 at 5:19 PM, Chris Mason <chris.mason@oracle.com> w=
rote:
> On Fri, Feb 26, 2010 at 11:51:35AM +0100, Leszek Ciesielski wrote:
>> On Fri, Feb 26, 2010 at 1:45 AM, Chris Mason <chris.mason@oracle.com=
> wrote:
>> > On Thu, Feb 25, 2010 at 10:34:22AM +0100, Leszek Ciesielski wrote:
>> >> (My previous post seems to have been discarded because of the
>> >> attachment size, I'm resending it without the dmesg output - whic=
h can
>> >> be found @ http://pastebin.com/T0J3z59j )
>> >>
>> >> Hi,
>> >>
>> >> yesterday I updated my kernel (clean clone from
>> >> mason/btrfs-unstable.gi), pulling in the single latest change I h=
ave
>> >> been missing ( http://git.kernel.org/?p=3Dlinux/kernel/git/mason/=
btrfs-unstable.git;a=3Dcommit;h=3D3f6fae9559225741c91f1320090b285da1413=
290
>> >> ) and adding my patch from http://patchwork.kernel.org/patch/8154=
7/ .
>> >> Previous kernel version (without my patch - could this be my faul=
t?)
>> >> has been running fine for 14 days, but after recompiling and
>> >> rebooting, my dmesg output is full of "btrfs no csum found for in=
ode
>> >> 386 start 0" and "btrfs csum failed ino 386 extent 65191274496 cs=
um
>> >> 1851253866 wanted 0 mirror 1" and "btrfs csum failed ino 82619 of=
f
>> >> 8749056 csum 2686054019 private 0",
>> >
>> > Yes, but did you verify your data?
>>
>> Part of the data stored on the volume consisted of video recordings =
-
>> after copying out and back onto the volume, they play back fine,
>> without video or audio glitches. Which I am aware does not mean they
>> are intact, just "good enough to work". I had also some important da=
ta
>> there, which is backed up to another location - I will verify it's
>> integrity with rsync during the weekend.
>>
>> >
>> > I don't think your patch alone could have caused this. =A0Has anyt=
hing
>> > else strange been happening on this machine?
>>
>> Not really. The FS was created with metadata=3Dmirror data=3Dmirror =
on a
>> single drive, then a second (larger) drive was added and the fs was
>> rebalanced. Compression is enabled. No problems until the last kerne=
l
>> update. After the recovery - no new csum failures.
>
> Ok, what does btrfsck say about the FS now?
>
> -chris
>

49MB of errors, 3.5MB compressed file. So I guess it's not too good
;-) Log uploaded here:
http://cid-3e1bf92365ec19a2.skydrive.live.com/self.aspx/.Public/btrfscf=
k.txt.gz
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html