From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout-de.gmx.net ([213.165.64.22]:53652 "HELO mailout-de.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1751928Ab2GJG5U (ORCPT ); Tue, 10 Jul 2012 02:57:20 -0400 Message-ID: <4FFBD24E.2050902@gmx.net> Date: Tue, 10 Jul 2012 08:57:18 +0200 From: Arne Jansen MIME-Version: 1.0 To: Jan Schmidt , linux-btrfs@vger.kernel.org Subject: Re: *** GMX Spamverdacht *** Re: btrfs GPF in read_extent_buffer() while scrubbing with kernel 3.4.2 References: <20120704160340.GA9770@sli.dy.fi> <4FF47168.9010405@jan-o-sch.net> <20120704202452.GA8996@sli.dy.fi> <4FF5998D.8020205@jan-o-sch.net> <20120705234739.GA9736@sli.dy.fi> <4FF6C12A.10305@jan-o-sch.net> <20120706143350.GA10427@sli.dy.fi> <4FF6FF8C.4010007@jan-o-sch.net> <20120706234405.GB8933@sli.dy.fi> <4FFA9EEB.7060008@gmx.net> <20120710041625.GA8909@sli.dy.fi> In-Reply-To: <20120710041625.GA8909@sli.dy.fi> Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-btrfs-owner@vger.kernel.org List-ID: On 10.07.2012 06:16, Sami Liedes wrote: > On Mon, Jul 09, 2012 at 11:05:47AM +0200, Arne Jansen wrote: >>> * Just before the crash: >>> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2261163409408, level 100, generation 4412718571037421157, nritems 538968254. len param 17. debug 2/989/538968254/4412718571037421157/0x0/0/0x0/0x0 >>> >> >> At a first glance: the generation converted to ascii is: "ent() ==", >> so someone is patching the memory with ascii text, possibly C source. >> It might be interesting to dump the full contents of the eb, to get >> a clue on the source of the data. > > I changed the code to dump the contents of the eb struct at the point > where that error (btrfs: invalid parameters...) is printed, at the > "checksum mismatch 4" site and at the "node seems invalid now" site. > Now I have a big log of 1795 corrupted ebs. So far nothing that looks > remotely like ascii text, though. But I have two different versions of > the eb that caused that warning, a less corrupted one and a more > corrupted one: > > ------------------------------------------------------------ > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ > btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... > btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... > btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... > btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. > btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. > btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... > btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... > btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... > btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. > btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ this one looks good so far. > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > btrfs: dm-6 checksum verify failed on 2239404212224 wanted B5F632BC found 3579FB59 level 160 > btrfs: node seems invalid now. checksum ok = 1 > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > [... identical dump to above ...] > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2239404212224, level 160, generation 4716553384049587249, nritems 295705211. len param 17. debug 2/989/295705211/4716553384049587249/0x0/0/0x0/0x0 > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ > btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... > btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... > btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... > btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. > btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. > btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... > btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... > btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... > btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. > btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... > btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > ------------[ cut here ]------------ > WARNING: at fs/btrfs/extent_io.c:4533 read_extent_buffer+0x1ef/0x220 [btrfs]() > Hardware name: System Product Name > Modules linked in: [last unloaded: scsi_wait_scan] > Pid: 25829, comm: btrfs-endio-met Tainted: G W 3.4.4+btrfsdebug3+ #4 > [...] > ------------------------------------------------------------ > > Here's the difference (lines reordered from diff to make comparing > easier): > > ------------------------------------------------------------ > -btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. > +btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. > -btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... > -btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q ------------------------------------------------------------ The diff is just the eb->debug[] Jan added. > > If there's one pattern that catches eye in the dumps, it's that in > many places where the same eb is dumped multiple times due to multiple > "checksum mismatch 4"s, there are bytes at offsets 0x30 and 0x31 that > always seem to have the same value and both separately increase > between the dumps, usually(?) by two. Then that might be normal, I > haven't looked at what should be at a struct eb :) There are some > instances where they wrap, for example > > f8 f8 -> fa fa -> fc fc -> fe fe -> 00 00 -> 02 02 -> ..., this is the spinlock to protect the refcnt, so some reference taking/releasing has been going on. > > with no other changes visible in the corrupted ebs. But that is by no > means the only change in the struct eb contents that can be observed, > only the most obvious pattern. > > For example, here's a 11-long string of eb contents. First the dmesg: > > ------------------------------------------------------------ > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 1] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 2] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 3] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 4] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 5] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 6] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 7] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 8] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 9] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 10] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 11] > ------------------------------------------------------------ > > Where DUMP 1 contents (file cc1.txt if you'd want to apply the diffs > below) is > > ------------------------------------------------------------ > btrfs: --- start eb contents at ffff8801cafbacc0 --- > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > btrfs: ffff8801cafbad30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad40: 00 00 10 00 00 00 00 00 04 04 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad50: 50 ad fb ca 01 88 ff ff 50 ad fb ca 01 88 ff ff P.......P....... > btrfs: ffff8801cafbad60: 02 02 00 00 00 00 00 00 68 ad fb ca 01 88 ff ff ........h....... > btrfs: ffff8801cafbad70: 68 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 h............... > btrfs: ffff8801cafbad80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad90: 00 a2 20 08 00 ea ff ff c0 2a f4 07 00 ea ff ff .. ......*...... > btrfs: ffff8801cafbada0: 00 ac 89 07 00 ea ff ff c0 42 79 07 00 ea ff ff .........By..... > btrfs: ffff8801cafbadb0: 80 af 77 07 00 ea ff ff 40 37 49 08 00 ea ff ff ..w.....@7I..... > btrfs: ffff8801cafbadc0: 80 6d f3 07 00 ea ff ff c0 a7 e8 07 00 ea ff ff .m.............. > btrfs: ffff8801cafbadd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801cafbacc0 --- > ------------------------------------------------------------ > > And the successive diffs are > > ------------------------------------------------------------ > --- cc1.txt 2012-07-10 06:57:21.564665577 +0300 > +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ the spinlock > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > @@ -20,9 +20,9 @@ > btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 M.......N....... > +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00 .x.............. and eb->debug > btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801cafbacc0 --- > --- cc2.txt 2012-07-10 06:59:10.272634578 +0300 > +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc3.txt 2012-07-10 06:59:10.016634663 +0300 > +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc4.txt 2012-07-10 06:59:09.752634749 +0300 > +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc5.txt 2012-07-10 06:59:09.504634831 +0300 > +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc6.txt 2012-07-10 06:59:09.240634917 +0300 > +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc7.txt 2012-07-10 06:59:08.944635015 +0300 > +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc8.txt 2012-07-10 06:59:08.656635109 +0300 > +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. > +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc9.txt 2012-07-10 06:59:08.328635216 +0300 > +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. > +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc10.txt 2012-07-10 06:58:33.180646115 +0300 > +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. > +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ''.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > ------------------------------------------------------------ All spinlocks. I'll go through this again with Jan to see if his eb->debug might give any clue here. It might be good to have a look at the contents of the pages though, as written in the previous mail. -Arne > > And since it's of course possible there's something wrong with my > modifications, here's the patch I used: > > ------------------------------------------------------------ > diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c > index 7452ecb..aadb82c 100644 > --- a/fs/btrfs/extent_io.c > +++ b/fs/btrfs/extent_io.c > @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void *dstv, > len, > eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3], > eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > WARN_ON(1); > } > WARN_ON(start + len > eb->start + eb->len); > diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c > index ea81bd4..663d6c4 100644 > --- a/fs/btrfs/reada.c > +++ b/fs/btrfs/reada.c > @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, > kref_get(&re->refcnt); > spin_unlock(&fs_info->reada_lock); > > - if (!err && btrfs_csum_tree_block(root, eb)) > + if (!err && btrfs_csum_tree_block(root, eb)) { > printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n", > eb->start); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > + } > > if (!re) > return -1; > @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, > if (err == 0) { > nritems = level ? btrfs_header_nritems(eb) : 0; > if (level > BTRFS_MAX_LEVEL || > - nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) > + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) { > printk(KERN_ERR "btrfs: node seems invalid now. checksum ok = %d\n", > btrfs_csum_tree_block(root, eb)); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > + > + } > generation = btrfs_header_generation(eb); > /* > * FIXME: currently we just set nritems to 0 if this is a leaf, > > ------------------------------------------------------------ > > Sami