[3.2.1] BUG at fs/btrfs/inode.c:1588

[3.2.1] BUG at fs/btrfs/inode.c:1588

[Kai Krakow]

Just happened while writing a huge avi file to my usb3 backup disk:

[356036.596292] ------------[ cut here ]------------
[356036.596300] kernel BUG at fs/btrfs/inode.c:1588!
[356036.596304] invalid opcode: 0000 [#1] SMP 
[356036.596307] CPU 2 
[356036.596309] Modules linked in: vmnet(O) vmblock(O) vsock(O) vmci(O) 
vmmon(O) af_packet snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss 
snd_mixer_oss nls_iso8859_15 nls_cp437 vfat fat btusb bluetooth zram(C) loop 
snd_usb_audio snd_hwdep snd_usbmidi_lib snd_rawmidi snd_seq_device 
gspca_sonixj gspca_main videodev v4l2_compat_ioctl32 pcspkr i2c_i801 evdev 
unix fuse xfs nfs nfs_acl auth_rpcgss lockd sunrpc reiserfs scsi_wait_scan 
hid_monterey hid_microsoft hid_logitech hid_ezkey hid_cypress hid_chicony 
hid_cherry hid_belkin hid_apple hid_a4tech usbhid usb_storage hid sr_mod 
cdrom sg pata_cmd64x [last unloaded: microcode]
[356036.596346] 
[356036.596349] Pid: 28747, comm: btrfs-fixup-1 Tainted: G         C O 
3.2.1-gentoo-r2 #1 To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Pro3
[356036.596355] RIP: 0010:[<ffffffff811605fe>]  [<ffffffff811605fe>] 
btrfs_writepage_fixup_worker+0xde/0x121
[356036.596363] RSP: 0000:ffff8801e2379de0  EFLAGS: 00010246
[356036.596366] RAX: 0000000000000000 RBX: ffffea00019b1a00 RCX: 
0000000000000000
[356036.596370] RDX: 0000000000000000 RSI: ffffffffffffffff RDI: 
ffff88008a1bbb40
[356036.596373] RBP: 00000000033fd000 R08: ffff8801e2379d2c R09: 
0000000180240024
[356036.596377] R10: 0000000000000000 R11: bf800000bf800000 R12: 
ffff88008a1bbc10
[356036.596380] R13: 0000000000000000 R14: ffff8801e2379df8 R15: 
00000000033fdfff
[356036.596384] FS:  0000000000000000(0000) GS:ffff88043fb00000(0000) 
knlGS:0000000000000000
[356036.596387] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[356036.596391] CR2: 00007f5ef9660000 CR3: 00000003253f2000 CR4: 
00000000000406e0
[356036.596394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[356036.596398] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[356036.596401] Process btrfs-fixup-1 (pid: 28747, threadinfo 
ffff8801e2378000, task ffff8802d2160650)
[356036.596405] Stack:
[356036.596407]  ffff88008a1bbab0 ffff88026d847540 ffffffffffffffff 
ffff88003a7c1f50
[356036.596412]  0000000000000000 ffff88019bf62d80 ffff88019bf62dd0 
ffff88019bf62d98
[356036.596417]  ffff88019bf62da8 ffff8802d2160650 ffff88019bf62d88 
ffffffff8117f23f
[356036.596421] Call Trace:
[356036.596426]  [<ffffffff8117f23f>] ? worker_loop+0x170/0x485
[356036.596431]  [<ffffffff8117f0cf>] ? btrfs_queue_worker+0x272/0x272
[356036.596435]  [<ffffffff8117f0cf>] ? btrfs_queue_worker+0x272/0x272
[356036.596439]  [<ffffffff810489fb>] ? kthread+0x7a/0x82
[356036.596445]  [<ffffffff81444634>] ? kernel_thread_helper+0x4/0x10
[356036.596449]  [<ffffffff81048981>] ? kthread_worker_fn+0x135/0x135
[356036.596453]  [<ffffffff81444630>] ? gs_change+0xb/0xb
[356036.596456] Code: 00 00 4c 89 f1 48 8b 3c 24 e8 67 4f 01 00 48 89 df e8 
b2 70 f2 ff ba 01 00 00 00 4c 89 ee 4c 89 e7 e8 bf 29 01 00 e9 4b ff ff ff 
<0f> 0b 41 b8 50 00 00 00 4c 89 f1 4c 89 fa 48 89 ee 48 8b 3c 24 
[356036.596478] RIP  [<ffffffff811605fe>] 
btrfs_writepage_fixup_worker+0xde/0x121
[356036.596483]  RSP <ffff8801e2379de0>
[356036.653626] ---[ end trace 9fa19a7644192fb6 ]---

btrfsck now finds many of these:

jupiter ~ # btrfsck /dev/sde1
root 256 inode 12746 errors 400
root 256 inode 12747 errors 400
root 256 inode 12748 errors 400
root 256 inode 12749 errors 400
root 256 inode 17141 errors 400
root 256 inode 219966 errors 400
root 256 inode 224243 errors 400
root 256 inode 225245 errors 400
root 256 inode 225354 errors 400
root 256 inode 290639 errors 2000
root 256 inode 291751 errors 2000

This disk is used for nothing else then the following cycle:

1. mount it (compress-force=gzip)
2. run rsync to backup my system (using cow-friendly rsync flags)
3. create a snapshot
4. unmount it

So that error must have been introduced simply by running rsync. It has 
plenty of free space (about 800 GB).

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Kai Krakow]

Interestingly, the filesystem was not unmountable - system hung. After 
reisub and checking again with "btrfs scrub" no errors where reported and it 
just rsync'ed fine this time. This does not make sense to me.

In any case here's my backup script although I see nothing special with it:

#!/bin/bash
DATE=$(date +%Y%m%d-%H%M)
BASEDIR=/mnt/usb-backup
ionice -c3 -p$$
mount ${BASEDIR} && (
  [ -d "${BASEDIR}/snapshots/system-${DATE}" ] || (
    time rsync -avAXH --delete --inplace --no-whole-file --stats \
      --exclude "/proc/" \
      --exclude "/dev/" \
      --exclude "/sys/" \
      --exclude "/boot/" \
      --exclude "/media/" \
      --exclude "/mnt/" \
      / ${BASEDIR}/current/
    btrfs subvolume snapshot \
      "${BASEDIR}/current" \
      "${BASEDIR}/snapshots/system-${DATE}"
    btrfs filesystem sync "${BASEDIR}"
  )
  umount /mnt/usb-backup
)


Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:

> Just happened while writing a huge avi file to my usb3 backup disk:
> 
> [356036.596292] ------------[ cut here ]------------
> [356036.596300] kernel BUG at fs/btrfs/inode.c:1588!
> [356036.596304] invalid opcode: 0000 [#1] SMP
> [356036.596307] CPU 2
> [356036.596309] Modules linked in: vmnet(O) vmblock(O) vsock(O) vmci(O)
> vmmon(O) af_packet snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss
> snd_mixer_oss nls_iso8859_15 nls_cp437 vfat fat btusb bluetooth zram(C)
> loop snd_usb_audio snd_hwdep snd_usbmidi_lib snd_rawmidi snd_seq_device
> gspca_sonixj gspca_main videodev v4l2_compat_ioctl32 pcspkr i2c_i801 evdev
> unix fuse xfs nfs nfs_acl auth_rpcgss lockd sunrpc reiserfs scsi_wait_scan
> hid_monterey hid_microsoft hid_logitech hid_ezkey hid_cypress hid_chicony
> hid_cherry hid_belkin hid_apple hid_a4tech usbhid usb_storage hid sr_mod
> cdrom sg pata_cmd64x [last unloaded: microcode]
> [356036.596346]
> [356036.596349] Pid: 28747, comm: btrfs-fixup-1 Tainted: G         C O
> 3.2.1-gentoo-r2 #1 To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Pro3
> [356036.596355] RIP: 0010:[<ffffffff811605fe>]  [<ffffffff811605fe>]
> btrfs_writepage_fixup_worker+0xde/0x121
> [356036.596363] RSP: 0000:ffff8801e2379de0  EFLAGS: 00010246
> [356036.596366] RAX: 0000000000000000 RBX: ffffea00019b1a00 RCX:
> 0000000000000000
> [356036.596370] RDX: 0000000000000000 RSI: ffffffffffffffff RDI:
> ffff88008a1bbb40
> [356036.596373] RBP: 00000000033fd000 R08: ffff8801e2379d2c R09:
> 0000000180240024
> [356036.596377] R10: 0000000000000000 R11: bf800000bf800000 R12:
> ffff88008a1bbc10
> [356036.596380] R13: 0000000000000000 R14: ffff8801e2379df8 R15:
> 00000000033fdfff
> [356036.596384] FS:  0000000000000000(0000) GS:ffff88043fb00000(0000)
> knlGS:0000000000000000
> [356036.596387] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [356036.596391] CR2: 00007f5ef9660000 CR3: 00000003253f2000 CR4:
> 00000000000406e0
> [356036.596394] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [356036.596398] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [356036.596401] Process btrfs-fixup-1 (pid: 28747, threadinfo
> ffff8801e2378000, task ffff8802d2160650)
> [356036.596405] Stack:
> [356036.596407]  ffff88008a1bbab0 ffff88026d847540 ffffffffffffffff
> ffff88003a7c1f50
> [356036.596412]  0000000000000000 ffff88019bf62d80 ffff88019bf62dd0
> ffff88019bf62d98
> [356036.596417]  ffff88019bf62da8 ffff8802d2160650 ffff88019bf62d88
> ffffffff8117f23f
> [356036.596421] Call Trace:
> [356036.596426]  [<ffffffff8117f23f>] ? worker_loop+0x170/0x485
> [356036.596431]  [<ffffffff8117f0cf>] ? btrfs_queue_worker+0x272/0x272
> [356036.596435]  [<ffffffff8117f0cf>] ? btrfs_queue_worker+0x272/0x272
> [356036.596439]  [<ffffffff810489fb>] ? kthread+0x7a/0x82
> [356036.596445]  [<ffffffff81444634>] ? kernel_thread_helper+0x4/0x10
> [356036.596449]  [<ffffffff81048981>] ? kthread_worker_fn+0x135/0x135
> [356036.596453]  [<ffffffff81444630>] ? gs_change+0xb/0xb
> [356036.596456] Code: 00 00 4c 89 f1 48 8b 3c 24 e8 67 4f 01 00 48 89 df
> [e8
> b2 70 f2 ff ba 01 00 00 00 4c 89 ee 4c 89 e7 e8 bf 29 01 00 e9 4b ff ff ff
> <0f> 0b 41 b8 50 00 00 00 4c 89 f1 4c 89 fa 48 89 ee 48 8b 3c 24
> [356036.596478] RIP  [<ffffffff811605fe>]
> btrfs_writepage_fixup_worker+0xde/0x121
> [356036.596483]  RSP <ffff8801e2379de0>
> [356036.653626] ---[ end trace 9fa19a7644192fb6 ]---
> 
> btrfsck now finds many of these:
> 
> jupiter ~ # btrfsck /dev/sde1
> root 256 inode 12746 errors 400
> root 256 inode 12747 errors 400
> root 256 inode 12748 errors 400
> root 256 inode 12749 errors 400
> root 256 inode 17141 errors 400
> root 256 inode 219966 errors 400
> root 256 inode 224243 errors 400
> root 256 inode 225245 errors 400
> root 256 inode 225354 errors 400
> root 256 inode 290639 errors 2000
> root 256 inode 291751 errors 2000
> 
> This disk is used for nothing else then the following cycle:
> 
> 1. mount it (compress-force=gzip)
> 2. run rsync to backup my system (using cow-friendly rsync flags)
> 3. create a snapshot
> 4. unmount it
> 
> So that error must have been introduced simply by running rsync. It has
> plenty of free space (about 800 GB).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Kai Krakow]

Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:

> Interestingly, the filesystem was not unmountable - system hung. After
> reisub and checking again with "btrfs scrub" no errors where reported and
> it just rsync'ed fine this time. This does not make sense to me.

btrfsck still shows a lot of errors, while scrubbing says everything is 
okay... *sigh

Now, how should one fix it if there is still no repair utility? I'm pretty 
sure sooner or later I will run into a BUG_ON again...

Thanks,
Kai

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Duncan]

Kai Krakow posted on Thu, 02 Feb 2012 04:54:45 +0100 as excerpted:

> Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:
> 
>> Interestingly, the filesystem was not unmountable - system hung. After
>> reisub and checking again with "btrfs scrub" no errors where reported
>> and it just rsync'ed fine this time. This does not make sense to me.
> 
> btrfsck still shows a lot of errors, while scrubbing says everything is
> okay... *sigh
> 
> Now, how should one fix it if there is still no repair utility? I'm
> pretty sure sooner or later I will run into a BUG_ON again...

I had hoped someone else better qualified would answer, and they may 
still do so, but in the meantime, a couple notes...

1) This is unfortunately quite handwavy as I don't understand the details 
myself, but if I'm reading the list right, there's a known "phantom ENOSPC 
bug" that some are hitting when trying to write a large file (gigs, think 
dvd image), or do an rsync of several gigs, or...  It may be that you hit 
it, and on retry, enough of the file was already there that it didn't 
trigger the second time around.

I gather they've not traced what is apparently a timeout or race 
condition fully and are working on a temporary throttling-based 
workaround.  That's supposed to work for the time being, but it's only a 
workaround, and the throttling controls themselves are apparently rather 
rough at this point, so part of the testing is to make it a bit easier 
for ordinary users without the esoteric knowledge of a btrfs dev to use.  

So there's a short-to-medium-term workaround coming and a longer term 
fix, once they trace down the problem itself.  Meanwhile, don't be too 
worried about ENOSPC errors the occur under heavy write load and that go 
away on retry, as there's apparently others having the same issue.

2) Just a couple days ago I read an article that claimed Oracle has a Feb 
16 deadline for a working btrfsck as that's the deadline for getting it 
in their next shipping Unbreakable Linux release.  I won't claim to know 
if the article is correct or not, but if so, a reasonably working btrfsck 
should be available within two weeks. =:^)  Of course it may continue to 
improve after that...

Meanwhile, there's a tool already available that should allow retrieving 
the undamaged data off of unmountable filesystems, at least, and there's 
another tool that allows rollback to an earlier root node if necessary, 
thus allowing recovery of most filesystems at the cost of losing the last 
few seconds of work.  Given the experimental nature of btrfs and the 
known lack of a proper btrfsck at this point anyway, that's... actually 
quite reasonable, and the reason I decided it was time to start checking 
out btrfs myself (I'm still researching but have been on the list about a 
week now and had read a couple weeks worth of posts before I responded to 
anything).

Don't ask me what the names of those tools are.  I could certainly look 
them up, but so can you, now that you know they exist. =:^) (assuming you 
didn't before, of course.)

Hopefully that's somewhat helpful in pointing you in the right direction, 
at least. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Kai Krakow]

Thank you, Duncon...

Duncan <1i5t5.duncan@cox.net> schrieb:

> Kai Krakow posted on Thu, 02 Feb 2012 04:54:45 +0100 as excerpted:
> 
>> Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:
>> 
>>> Interestingly, the filesystem was not unmountable - system hung. After
>>> reisub and checking again with "btrfs scrub" no errors where reported
>>> and it just rsync'ed fine this time. This does not make sense to me.
>> 
>> btrfsck still shows a lot of errors, while scrubbing says everything is
>> okay... *sigh
>> 
>> Now, how should one fix it if there is still no repair utility? I'm
>> pretty sure sooner or later I will run into a BUG_ON again...
> 
> I had hoped someone else better qualified would answer, and they may
> still do so, but in the meantime, a couple notes...

Still I think you gained good insight by reading all those posts. I'm using 
btrfs for a few weeks now and it is pretty solid since 3.2. I've been 
reading the list a few weeks before starting btrfs but only looked at 
articles about corruption and data loss. I started using btrfs when rescue 
tools became available and the most annoying corruption bugs had been fixed. 
But I've been hit by corruption and freezes a few times so I decided to have 
that big usb3 disk which I rsync every now and then using snapshots for 
rollback.

> 1) This is unfortunately quite handwavy as I don't understand the details
> myself, but if I'm reading the list right, there's a known "phantom ENOSPC
> bug" that some are hitting when trying to write a large file (gigs, think
> dvd image), or do an rsync of several gigs, or...  It may be that you hit
> it, and on retry, enough of the file was already there that it didn't
> trigger the second time around.

On my first thought this was my suspicion, too. But otoh there was no ENOSPC 
message, neither in dmesg nor in rsync. Rsync just froze, I was able to kill 
it and my script continued to create a snapshot afterwards and unmount. I 
tried to mount again after btrfsck, it worked fine, I unmounted, system 
hung. I rebooted, scrubbed my two-disk array, no problems, I mounted the 
backup disk again, rsync'ed it, went fine, unmounted. But btrfsck still 
shows the same errors for this disk. *sigh

[...]
> So there's a short-to-medium-term workaround coming and a longer term
> fix, once they trace down the problem itself.  Meanwhile, don't be too
> worried about ENOSPC errors the occur under heavy write load and that go
> away on retry, as there's apparently others having the same issue.

I reported it here in the hope that it helps tracking down the bug or find 
people with similar experiences. Since "only" my backup disk is affected it 
is not a big problem. I can create it from scratch but try not to do it 
until someone guides me in tracking it down a bit. I think btrfs should try 
to fix such corruptions online while using it. From what I've learned here 
this is the long-term target and a working btrfsck should just be a helper 
tool. And the reason for the long delayed btrfsck is that Chris wants to 
have proper online fixing in place first.

At least I can tell this corruption was introduced by bad logic in the 
kernel, and not by some crash. The usb3 disk is solely mounted for the 
purpose of rsync'ing and unmounted all the other times.

> 2) Just a couple days ago I read an article that claimed Oracle has a Feb
> 16 deadline for a working btrfsck as that's the deadline for getting it
> in their next shipping Unbreakable Linux release.  I won't claim to know
> if the article is correct or not, but if so, a reasonably working btrfsck
> should be available within two weeks. =:^)  Of course it may continue to
> improve after that...

Sounds good. I wonder if Chris could tell anything on that point. ;-)

> Meanwhile, there's a tool already available that should allow retrieving
> the undamaged data off of unmountable filesystems, at least, and there's
> another tool that allows rollback to an earlier root node if necessary,
> thus allowing recovery of most filesystems at the cost of losing the last
> few seconds of work.  Given the experimental nature of btrfs and the
> known lack of a proper btrfsck at this point anyway, that's... actually
> quite reasonable, and the reason I decided it was time to start checking
> out btrfs myself (I'm still researching but have been on the list about a
> week now and had read a couple weeks worth of posts before I responded to
> anything).

The tools are btrfs-rescue and btrfs-repair from Josef's btrfs-progs 
available from github.

> Hopefully that's somewhat helpful in pointing you in the right direction,
> at least. =:^)

Well, no real news to me. But it is good having another soul here interested 
in gaining some knowledge. I think btrfs will become a great filesystem.

But if you could provide a link for the Feb 16 deadline I'd be eager to read 
the article.

Regards,
Kai

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Kai Krakow]

Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:

> Just happened while writing a huge avi file to my usb3 backup disk:
> 
> [356036.596292] ------------[ cut here ]------------
> [356036.596300] kernel BUG at fs/btrfs/inode.c:1588!
[...]
> 
> btrfsck now finds many of these:
> 
> jupiter ~ # btrfsck /dev/sde1
> root 256 inode 12746 errors 400
> root 256 inode 12747 errors 400
> root 256 inode 12748 errors 400
> root 256 inode 12749 errors 400
> root 256 inode 17141 errors 400
> root 256 inode 219966 errors 400
> root 256 inode 224243 errors 400
> root 256 inode 225245 errors 400
> root 256 inode 225354 errors 400
> root 256 inode 290639 errors 2000
> root 256 inode 291751 errors 2000

It looks like the latter isn't a consequence of the former. I found kernel 
commit f70a9a6b9 which is supposed to do something about "inode errors 400":

commit f70a9a6b94af86fca069a7552ab672c31b457786
Author: Miao Xie <miaox@cn.fujitsu.com>
Date:   Thu Jan 12 19:10:12 2012 -0500
Btrfs: fix btrfsck error 400 when truncating a compressed

It's actually the case for me that rsync writes to the device using mount 
options "compress-force=zlib" and that rsync probably truncates files 
sometimes when using the inplace option.

So that occurence is explained. Can anyone tell how bad it is to have this 
error? May the fs explode at some point or is it just an error I could 
safely ignore for the moment?

And what about the "inode errors 2000"? What's the 2000 standing for?

Thanks,
Kai

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Mitch Harder]

On Sat, Feb 4, 2012 at 5:40 AM, Kai Krakow <hurikhan77+btrfs@gmail.com>=
 wrote:
> Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:
>
>> Just happened while writing a huge avi file to my usb3 backup disk:
>>
>> [356036.596292] ------------[ cut here ]------------
>> [356036.596300] kernel BUG at fs/btrfs/inode.c:1588!
> [...]
>>
>> btrfsck now finds many of these:
>>
>> jupiter ~ # btrfsck /dev/sde1
>> root 256 inode 12746 errors 400
>> root 256 inode 12747 errors 400
>> root 256 inode 12748 errors 400
>> root 256 inode 12749 errors 400
>> root 256 inode 17141 errors 400
>> root 256 inode 219966 errors 400
>> root 256 inode 224243 errors 400
>> root 256 inode 225245 errors 400
>> root 256 inode 225354 errors 400
>> root 256 inode 290639 errors 2000
>> root 256 inode 291751 errors 2000
>
> It looks like the latter isn't a consequence of the former. I found k=
ernel
> commit f70a9a6b9 which is supposed to do something about "inode error=
s 400":
>
> commit f70a9a6b94af86fca069a7552ab672c31b457786
> Author: Miao Xie <miaox@cn.fujitsu.com>
> Date: =A0 Thu Jan 12 19:10:12 2012 -0500
> Btrfs: fix btrfsck error 400 when truncating a compressed
>
> It's actually the case for me that rsync writes to the device using m=
ount
> options "compress-force=3Dzlib" and that rsync probably truncates fil=
es
> sometimes when using the inplace option.
>
> So that occurence is explained. Can anyone tell how bad it is to have=
 this
> error? May the fs explode at some point or is it just an error I coul=
d
> safely ignore for the moment?
>
> And what about the "inode errors 2000"? What's the 2000 standing for?
>

As I understand it, Miao Xie's "Btrfs: fix btrfsck error 400 when
truncating a compressed" patch was intended to address only one
scenario that would lead to 400 errors.  It was not intended to
comprehensively address problems that generate inode 400 errors, nor
will this patch fix the error once it encountered.

As it stands right now, if you have errors reported by btrfsck, and
you've exhausted the tools available for addressing errors (scrub and
zero-ing out the log are the only two I know of), you only really have
two options (until further tools such as btrfsck are released):

(1)  Run with the errors and cross your fingers.
(2)  Backup and restore onto a freshly formatted  volume (there are
some new tools available to extract data if you encounter errors).

My personal preference is to backup and restore.

With respect to the error codes, you have to look in the source for
btrfsck.c.  Inode errors are defined as I_ERR_<error>.

0x400 (1 << 8) errors are I_ERR_FILE_NBYTES_WRONG
0x2000 (1 << 13) errors are I_ERR_LINK_COUNT_WRONG

The 400 inode errors have been common lately.  I haven't seen as many
2000 inode errors.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Duncan]

Kai Krakow posted on Fri, 03 Feb 2012 00:25:51 +0100 as excerpted:

> Duncan <1i5t5.duncan@cox.net> schrieb:
> 
>> I had hoped someone else better qualified would answer, and they may
>> still do so, but in the meantime, a couple notes...
> 
> Still I think you gained good insight by reading all those posts. I'm
> using btrfs for a few weeks now and it is pretty solid since 3.2. I've
> been reading the list a few weeks before starting btrfs but only looked
> at articles about corruption and data loss. I started using btrfs when
> rescue tools became available and the most annoying corruption bugs had
> been fixed.
> But I've been hit by corruption and freezes a few times so I decided to
> have that big usb3 disk which I rsync every now and then using snapshots
> for rollback.

Agreed on the insight from reading, and good to read that it has been 
pretty solid for you since 3.2

What I seem to be seeing is that the normal single-disk/dual-metadata 
setup seems to be reasonable, a few weird reports here and there 
including the ENOSPC stuff, but nothing huge.

But my primary interest is raid1 both data and metadata, more than two-
copies (3-4), and > 2-copy just doesn't appear to be available yet (but 
see the article linked below, which says 3.4 timeframe).  Even the two-
copy setup looks like it still has major problems, including a recent 
thread reporting an inability to allocate further chunks when in degraded 
mode, so as soon as currently in-use chunks get full (maybe a gig or so 
instead of 20 in that test), ENOSPC.

So I think I'll wait another kernel cycle or two... but now that I'm 
here, I'm going to continue tracking the list, so I'll be ready to go 
when the time comes.

>> 1) "phantom ENOSPC bug"

> On my first thought this was my suspicion, too. But otoh there was no
> ENOSPC message, neither in dmesg nor in rsync. Rsync just froze, I was
> able to kill it and my script continued to create a snapshot afterwards
> and unmount. I tried to mount again after btrfsck, it worked fine, I
> unmounted, system hung. I rebooted, scrubbed my two-disk array, no
> problems, I mounted the backup disk again, rsync'ed it, went fine,
> unmounted. But btrfsck still shows the same errors for this disk. *sigh

Good point.  It looks similar to the ENOSPC bug, but without the ENOSPC.

But keep in mind that they're apparently simply throttling as a near-term 
workaround and haven't fully traced the bug, yet.  Given the otherwise 
similar trigger and symptoms, your reported problem could thus be a 
variant of the same bug, that happened to freeze rsync instead of erroring 
out with ENOSPC.  If so, when they do finally nail that one, it could 
well either nail yours or at least make it easier to trace, as well.


> I think btrfs should try to fix such corruptions online while using it.
> From what I've learned here this is the long-term target and a working
> btrfsck should just be a helper tool. And the reason for the long
> delayed btrfsck is that Chris wants to have proper online fixing in
> place first.

I had seen articles pointing out that the mount-time and online fixing 
tools were indeed taking up some of the slack, but this is the first time 
I've seen it claimed as a major strategy, vs. the problems they've been 
seeing simply being easy enough to fix online once they track them down 
sufficiently to fix them at all, online or off.  However, it does make a 
lot of sense to do what you can online, and until the last couple weeks I 
could have easily missed that it was deliberate since I wasn't following 
btrfs closely enough to be sure to catch it before that, so it well may 
/be/ a deliberate strategy.  I believe you're correct.

> At least I can tell this corruption was introduced by bad logic in the
> kernel, and not by some crash. The usb3 disk is solely mounted for the
> purpose of rsync'ing and unmounted all the other times.

That's a good point, as well.

>> 2) Just a couple days ago I read an article that claimed Oracle has a
>> Feb 16 deadline for a working btrfsck as that's the deadline for
>> getting it in their next shipping Unbreakable Linux release.  I won't
>> claim to know if the article is correct or not, but if so, a reasonably
>> working btrfsck should be available within two weeks. =:^)  Of course
>> it may continue to improve after that...
> 
> Sounds good. I wonder if Chris could tell anything on that point. ;-)
> 
>> Meanwhile, there's a tool already available that should allow
>> retrieving the undamaged data off of unmountable filesystems, at least,
>> and there's another tool that allows rollback to an earlier root node
>> if necessary

> The tools are btrfs-rescue and btrfs-repair from Josef's btrfs-progs
> available from github.

Thanks.  

> But if you could provide a link for the Feb 16 deadline I'd be eager to
> read the article.

It was a couple days before I could go looking, thus the delay in this 
post, and it might be Feb 14 not 16, but...

The basis seems to be Chris Mason's talk at SCALE 10x LA, so there should 
be independent coverage on various Linux new sites.  Here's the one I 
googled up first (using the Feb 16 date, that at least here appears to be 
Feb 14, which might explain why I had trouble googling it).  Phoronix.

http://www.phoronix.com/scan.php?page=news_item&px=MTA0ODU

That might have been the one I read, originally.  (I subscribe to
lxer.com 's feed, which covers Linux and Android stories from around the 
net, including phoronix, and would have clicked that if it had come up, 
but don't remember for sure whether that was it or if there was another.)

There's a bit more tech detail, including the new tidbit about multiple-
mirroring that I mentioned above, in a different article.

http://www.phoronix.com/scan.php?page=news_item&px=MTA0Njk

I had discovered much to my dismay that so-called btrfs-raid1 only does 
dual-copy, not full raid1 to an arbitrary number of copies.  My current 
disks are old enough that I really don't want to risk two-copy-only, 
especially since I'm currently running 4-spindle md/raid1 for most of my 
system so I already have the disks.  I originally installed md/raid6 for 
most of my data, thus the quad-spindle, but after running it for awhile, 
decided raid-1 fit my needs better.  If I'd have known about raid5/6 at 
purchase and setup time what I know now, I'd have probably gone 2-spindle 
raid1 then, with a third as a hot-spare, and saved myself the money on 
the 4th one.  The two-way would have been fine for current btrfs, but 
given that I'm running 4-way raid1 now and the disks are about mid-life 
operating hours, according to SMART, I simply don't want to risk 
switching to two-way-only mirroring, only to then have both mirrors of 
after all aging disks die at once.

So I've been debating whether btrfs DUAL mode (dual metadata on the same 
device, single data) on 4-way md/raid1s would be better, or btrfs-raid1s 
(two-way-mirrored data and metadata both, since two-way is all that's 
possible ATM) layered on pairs of 2-way md/raids would be better.  The 
latter would play to btrfs' ability to recover data from a different 
mirror when necessary.  But I already run a dozen md/raids on partitions 
across the same four physical devices, and that would double it to two-
dozen.  At some point it's no longer a workable solution...

But if triple-way mirroring (and one assumes N-way mirroring can't be far 
behind that if it's not what was meant) will show up in 3.4 or 3.5, as 
that article suggests, and with the writing-fsck being out for awhile by 
then if it's coming out later this month, that might well be my upgrade 
time.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Kai Krakow]

Mitch Harder <mitch.harder@sabayonlinux.org> schrieb:

> On Sat, Feb 4, 2012 at 5:40 AM, Kai Krakow <hurikhan77+btrfs@gmail.com>
> wrote:
>> It's actually the case for me that rsync writes to the device using mount
>> options "compress-force=zlib" and that rsync probably truncates files
>> sometimes when using the inplace option.
>>
>> So that occurence is explained. Can anyone tell how bad it is to have
>> this error? May the fs explode at some point or is it just an error I
>> could safely ignore for the moment?
>>
>> And what about the "inode errors 2000"? What's the 2000 standing for?
>>
> 
> As I understand it, Miao Xie's "Btrfs: fix btrfsck error 400 when
> truncating a compressed" patch was intended to address only one
> scenario that would lead to 400 errors.  It was not intended to
> comprehensively address problems that generate inode 400 errors, nor
> will this patch fix the error once it encountered.

I already understood that this patch wont magically fix errors already on 
disk. But my hope is, in 3.3, it would not introduce such errors any longer. 
But from what you write it could still generate inode 400 errors?

I conclude that there exist many points in the btrfs code which could 
generate inode 400 errors, and Miao Xie's patch only fixes the one scenario 
about truncating compressed files? Well, that should at least fix the 
problems for me because I believe this is coming from using compress-force 
with rsync-inplace... I don't think these errors are originating from 
somewhere else because this disk is only mounted for the few minutes I'm 
running the rsync to it, then unmounted. Any maybe 3-4 weeks ago there have 
been no errors in btrfsck...

> As it stands right now, if you have errors reported by btrfsck, and
> you've exhausted the tools available for addressing errors (scrub and
> zero-ing out the log are the only two I know of),

I think I will try btrfs-repair from Josef... That's one other tool for 
addressing errors.

> you only really have
> two options (until further tools such as btrfsck are released):
> 
> (1)  Run with the errors and cross your fingers.

I don't like crossing fingers for computers... ;-)

> (2)  Backup and restore onto a freshly formatted  volume (there are
> some new tools available to extract data if you encounter errors).

This is my backup disk so I could easily purge it and restart with a backup 
from scratch - it will just take some hours for the first sync.

> My personal preference is to backup and restore.

This is why I got this external backup disk. ;-)

> With respect to the error codes, you have to look in the source for
> btrfsck.c.  Inode errors are defined as I_ERR_<error>.
> 
> 0x400 (1 << 8) errors are I_ERR_FILE_NBYTES_WRONG
> 0x2000 (1 << 13) errors are I_ERR_LINK_COUNT_WRONG

Thanks for the pointers. I probably could live with the wrong link count. 
Not sure what kind of problems and wrong nbytes value introduces but at 
least my latest snapshot seems to be okay. Well, and at least these kind of 
errors seem like a proper candidate for btrfs-repair from Josef's btrs-progs 
tree.

> The 400 inode errors have been common lately.  I haven't seen as many
> 2000 inode errors.

I report back if btrfs-repair could fix it for me...

Thanks,
Kai

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Duncan]

Kai Krakow posted on Sun, 05 Feb 2012 09:01:25 +0100 as excerpted:

> This is why I got this external backup disk. ;-)

You've tested that backup, I hope?  If you're using it as a system disk, 
be sure you can boot from it without the main disk running so the system 
can't pull anything from it.

I mention this because I have an external that's not directly bootable.  
I had to create a USB stick to boot from, that can then load the system 
off the unbootable USB drive after the kernel is loaded.  People could be 
caught out if they weren't aware it wasn't bootable...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [3.2.1] BUG at fs/btrfs/inode.c:1588

[Andrea Gelmini]

2012/2/1 Kai Krakow <hurikhan77+btrfs@gmail.com>:
> Just happened while writing a huge avi file to my usb3 backup disk:

Same problem here, I try to give the filesystem history:
a) three days ago I format a 219GB partition:
   1) latest Linus' git kernel tree;
   2) two nested subvolumes;
   3) options: defaults,noatime,nobarrier,ssd,noacl,compress,subvol=root,autodefrag
b) I copy ~90GB of data;
c) I mount same as above, without compress;
d) I copy other data, to ~140GB;
e) run balance, after a while I had to poweroff;
f) two days ago, I boot and it finishes the balance;
g) I put in cron a snapshot every hour;
h) today (after a lot of clean resume/suspend in RAM) I run VirtualBox
and start an Ubuntu 12.04 install in Guest;
i) near the end of installation VirtualBox get stuck (but everything
else works) and kernel complain:

[16661.706465] ------------[ cut here ]------------
[16661.706514] kernel BUG at fs/btrfs/inode.c:1588!
[16661.706556] invalid opcode: 0000 [#1] SMP
[16661.706597] CPU 0
[16661.706615] Modules linked in: zram(C) xfs exportfs usbhid hid
binfmt_misc pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O)
rfcomm bnep joydev snd_hda_codec_realtek snd_hda_intel snd_hda_codec
snd_hwdep uvcvideo r852 sm_common nand videobuf2_core snd_pcm videodev
nand_ids btusb v4l2_compat_ioctl32 bluetooth mtd videobuf2_vmalloc
videobuf2_memops nand_bch bch option usb_wwan nand_ecc usbserial
psmouse snd_timer iwl3945 snd_page_alloc iwlegacy raid10 raid456
async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq
async_tx raid0 linear dm_mirror dm_region_hash dm_log usb_storage
sdhci_pci sdhci i915 drm_kms_helper mmc_core drm intel_agp intel_gtt
sky2 agpgart
[16661.707016]
[16661.707016] Pid: 710, comm: btrfs-fixup-1 Tainted: G         C O
3.3.0-rc3g+ #13 SAMSUNG ELECTRONICS CO., LTD. SQ45S70S/SQ45S70S
[16661.707016] RIP: 0010:[<ffffffff811aedc5>]  [<ffffffff811aedc5>]
btrfs_writepage_fixup_worker+0x145/0x150
[16661.707016] RSP: 0000:ffff8800b9bb3df0  EFLAGS: 00010246
[16661.707016] RAX: 0000000000000000 RBX: ffffea0002185900 RCX: 0000000002488000
[16661.707016] RDX: ffff8800897ae2a8 RSI: ffffffffffffffff RDI: ffff8800897ae428
[16661.707016] RBP: 0000000002488000 R08: 0000000000000008 R09: ffff8800b9bb3da8
[16661.707016] R10: 0000000000001000 R11: 0000000000000000 R12: ffff8800b75f5ff0
[16661.707016] R13: 0000000000000000 R14: 0000000002488fff R15: ffff8800b75f5e70
[16661.707016] FS:  0000000000000000(0000) GS:ffff8800bf400000(0000)
knlGS:0000000000000000
[16661.707016] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[16661.707016] CR2: 00007fa504fd40e0 CR3: 00000000ba515000 CR4: 00000000000006f0
[16661.707016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[16661.707016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[16661.707016] Process btrfs-fixup-1 (pid: 710, threadinfo
ffff8800b9bb2000, task ffff8800373edcd0)
[16661.707016] Stack:
[16661.707016]  ffffffff81046a30 ffff880085a0d960 00000000ffffffff
ffff88008e3602a0
[16661.707016]  ffff8800bf410b40 ffff8800b9ce3840 ffff880085a0d968
ffff8800b9ce3858
[16661.707016]  ffff8800b9ce3890 ffff8800b9bb3e90 ffff8800b9ce3858
ffffffff811d6972
[16661.707016] Call Trace:
[16661.707016]  [<ffffffff81046a30>] ? run_timer_softirq+0x220/0x220
[16661.707016]  [<ffffffff811d6972>] ? worker_loop+0xa2/0x500
[16661.707016]  [<ffffffff811d68d0>] ? btrfs_queue_worker+0x340/0x340
[16661.707016]  [<ffffffff81056966>] ? kthread+0x96/0xa0
[16661.707016]  [<ffffffff8150b534>] ? kernel_thread_helper+0x4/0x10
[16661.707016]  [<ffffffff810568d0>] ? kthread_freezable_should_stop+0x60/0x60
[16661.707016]  [<ffffffff8150b530>] ? gs_change+0xb/0xb
[16661.707016] Code: 41 5f c3 0f 1f 00 41 b8 50 00 00 00 48 8d 4c 24
18 4c 89 f2 48 89 ee 4c 89 ff e8 d7 a4 01 00 eb b8 48 89 df e8 3d 01
ef ff eb 9c <0f> 0b 66 0f 1f 84 00 00 00 00 00 41 55 4c 8d 6e 40 41 54
55 48
[16661.707016] RIP  [<ffffffff811aedc5>]
btrfs_writepage_fixup_worker+0x145/0x150
[16661.707016]  RSP <ffff8800b9bb3df0>
[16661.731968] ---[ end trace 9b36ae9483fc03e3 ]---

l) I can't remove ~/VirtualBox VMs/Ubuntu (command "rm -rf" doesn't
return), but I can cleanly close others apps;
m) booting from recovery partition I can mount BTRFS and rm the directory above;
n) run btrfsck

fs tree 454 refs 12
	unresolved ref root 454 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 455 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 458 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 459 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 466 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 498 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 500 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 501 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 503 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 504 dir 844801 index 5 namelen 9 name snap-0212 error 600
	unresolved ref root 507 dir 844801 index 5 namelen 9 name snap-0212 error 600
fs tree 455 refs 11
	unresolved ref root 455 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 458 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 459 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 466 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 498 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 500 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 501 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 503 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 504 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
	unresolved ref root 507 dir 844801 index 6 namelen 24 name
snap-2012-02-12.15:23:24 error 600
fs tree 458 refs 10
	unresolved ref root 458 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
	unresolved ref root 459 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
	unresolved ref root 466 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
	unresolved ref root 498 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
	unresolved ref root 500 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
	unresolved ref root 501 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
	unresolved ref root 503 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
	unresolved ref root 504 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
	unresolved ref root 507 dir 844801 index 7 namelen 24 name
snap-2012-02-13.00:44:53 error 600
fs tree 459 refs 9
	unresolved ref root 459 dir 844801 index 8 namelen 24 name
snap-2012-02-13.01:00:01 error 600
	unresolved ref root 466 dir 844801 index 8 namelen 24 name
snap-2012-02-13.01:00:01 error 600
	unresolved ref root 498 dir 844801 index 8 namelen 24 name
snap-2012-02-13.01:00:01 error 600
	unresolved ref root 500 dir 844801 index 8 namelen 24 name
snap-2012-02-13.01:00:01 error 600
	unresolved ref root 501 dir 844801 index 8 namelen 24 name
snap-2012-02-13.01:00:01 error 600
	unresolved ref root 503 dir 844801 index 8 namelen 24 name
snap-2012-02-13.01:00:01 error 600
	unresolved ref root 504 dir 844801 index 8 namelen 24 name
snap-2012-02-13.01:00:01 error 600
	unresolved ref root 507 dir 844801 index 8 namelen 24 name
snap-2012-02-13.01:00:01 error 600
fs tree 466 refs 8
	unresolved ref root 466 dir 844801 index 9 namelen 24 name
snap-2012-02-13.11:00:01 error 600
	unresolved ref root 498 dir 844801 index 9 namelen 24 name
snap-2012-02-13.11:00:01 error 600
	unresolved ref root 500 dir 844801 index 9 namelen 24 name
snap-2012-02-13.11:00:01 error 600
	unresolved ref root 501 dir 844801 index 9 namelen 24 name
snap-2012-02-13.11:00:01 error 600
	unresolved ref root 503 dir 844801 index 9 namelen 24 name
snap-2012-02-13.11:00:01 error 600
	unresolved ref root 504 dir 844801 index 9 namelen 24 name
snap-2012-02-13.11:00:01 error 600
	unresolved ref root 507 dir 844801 index 9 namelen 24 name
snap-2012-02-13.11:00:01 error 600
fs tree 498 refs 7
	unresolved ref root 498 dir 844801 index 10 namelen 24 name
snap-2012-02-13.12:00:01 error 600
	unresolved ref root 500 dir 844801 index 10 namelen 24 name
snap-2012-02-13.12:00:01 error 600
	unresolved ref root 501 dir 844801 index 10 namelen 24 name
snap-2012-02-13.12:00:01 error 600
	unresolved ref root 503 dir 844801 index 10 namelen 24 name
snap-2012-02-13.12:00:01 error 600
	unresolved ref root 504 dir 844801 index 10 namelen 24 name
snap-2012-02-13.12:00:01 error 600
	unresolved ref root 507 dir 844801 index 10 namelen 24 name
snap-2012-02-13.12:00:01 error 600
fs tree 500 refs 6
	unresolved ref root 500 dir 844801 index 11 namelen 24 name
snap-2012-02-13.14:00:01 error 600
	unresolved ref root 501 dir 844801 index 11 namelen 24 name
snap-2012-02-13.14:00:01 error 600
	unresolved ref root 503 dir 844801 index 11 namelen 24 name
snap-2012-02-13.14:00:01 error 600
	unresolved ref root 504 dir 844801 index 11 namelen 24 name
snap-2012-02-13.14:00:01 error 600
	unresolved ref root 507 dir 844801 index 11 namelen 24 name
snap-2012-02-13.14:00:01 error 600
fs tree 501 refs 5
	unresolved ref root 501 dir 844801 index 12 namelen 24 name
snap-2012-02-13.18:00:01 error 600
	unresolved ref root 503 dir 844801 index 12 namelen 24 name
snap-2012-02-13.18:00:01 error 600
	unresolved ref root 504 dir 844801 index 12 namelen 24 name
snap-2012-02-13.18:00:01 error 600
	unresolved ref root 507 dir 844801 index 12 namelen 24 name
snap-2012-02-13.18:00:01 error 600
fs tree 503 refs 4
	unresolved ref root 503 dir 844801 index 13 namelen 24 name
snap-2012-02-13.19:00:01 error 600
	unresolved ref root 504 dir 844801 index 13 namelen 24 name
snap-2012-02-13.19:00:01 error 600
	unresolved ref root 507 dir 844801 index 13 namelen 24 name
snap-2012-02-13.19:00:01 error 600
fs tree 504 refs 3
	unresolved ref root 504 dir 844801 index 14 namelen 24 name
snap-2012-02-13.20:00:01 error 600
	unresolved ref root 507 dir 844801 index 14 namelen 24 name
snap-2012-02-13.20:00:01 error 600
fs tree 507 refs 2
	unresolved ref root 507 dir 844801 index 15 namelen 24 name
snap-2012-02-13.21:00:01 error 600
found 148321431552 bytes used err is 1
total csum bytes: 143183508
total tree bytes: 1695162368
total fs tree bytes: 1380012032
btree space waste bytes: 395268619
file data blocks allocated: 181930553344
 referenced 190255210496

o) remove all snapshots, so btrsck stop complain:

found 144518303744 bytes used err is 0
total csum bytes: 139577232
total tree bytes: 1588137984
total fs tree bytes: 1281392640
btree space waste bytes: 370101450
file data blocks allocated: 143005290496
 referenced 152667561984

p) scrub is still running, but everything seems fine.

Thanks a lot for your time,
Andrea