BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Marc MERLIN]

So, I had an FS with a few issues, I ran btrfs check --repair to completion

Then, after mounting, I still get this warning.
Shouldn't those error counters be reset after check --repair?

Kernel: 4.2.5
Btrfs-tools: 4.3-1

If that matters, here's the output of check --repair (captured with script
-f, so the output is a bit wrong):
gargamel:~# btrfs check --repair -p /dev/mapper/dshelf1
enabling repair mode
Checking filesystem on /dev/mapper/dshelf1
UUID: 6358304a-2234-4243-b02d-4944c9af47d7
badcextentx[29368320, 29372416), type mismatch with chunk
bad extent [29372416, 29376512), type mismatch with chunk

(2856970 lines deleted)

bad extent [8697338122240, 8697338126336), type mismatch with chunk
bad extent [8697338126336, 8697338130432), type mismatch with chunk
bad extent [8697338130432, 8697338134528), type mismatch with chunk
checking extents [.]

repaired damaged extent references

Fixed 0 roots.
cache and super generation don't match, space cache will be invalidated
Fixedidiscountofileoextents for inode: 204450 in root: 45851
FixedidiscountofileOextents for inode: 204452 in root: 45851
root 45851 inode 204452 errors 40, bad file extent
Fixedidiscountofileoextents for inode: 204452 in root: 45851
root 45851 inode 204452 errors 40, bad file extent
rootk45851sinodes204452 errors 40, bad file extent
FixedidiscountofileOextents for inode: 204450 in root: 45852
FixedidiscountofileOextents for inode: 204452 in root: 45852
checking fs roots [o]
rootk45851sinodes204452 errors 40, bad file extent
Fixedidiscountofile.extents for inode: 204450 in root: 45856
Fixed discount file extents for inode: 204452 in root: 45856
rootk45851sinodes204452 errors 40, bad file extent
warninggliner3653 [o]

checking csums
checking root refs
found 9826147025859 bytes used err is 0
total csum bytes: 9584068648
total tree bytes: 12200706048
total fs tree bytes: 330457088
total extent tree bytes: 498716672
btree space waste bytes: 1372963760
file data blocks allocated: 9976766078976
 referenced 9987816431616
btrfs-progs v4.3

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Duncan]

Marc MERLIN posted on Sun, 17 Jan 2016 16:27:48 -0800 as excerpted:

> So, I had an FS with a few issues, I ran btrfs check --repair to
> completion
> 
> Then, after mounting, I still get this warning. [see subject]
> Shouldn't those error counters be reset after check --repair?

No.  Those are monotonically increasing counts that are never 
automatically reset over the life of the filesystem.  Use ...

btrfs dev stats

... to display them on the commandline (vs. the kernel log at filesystem 
mount), with the -z option to reset them after display, if desired.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2669 bytes --]

On Sun, Jan 17, 2016 at 04:27:48PM -0800, Marc MERLIN wrote:
> So, I had an FS with a few issues, I ran btrfs check --repair to completion
> 
> Then, after mounting, I still get this warning.
> Shouldn't those error counters be reset after check --repair?
> 
> Kernel: 4.2.5
> Btrfs-tools: 4.3-1
> 
> If that matters, here's the output of check --repair (captured with script
> -f, so the output is a bit wrong):
> gargamel:~# btrfs check --repair -p /dev/mapper/dshelf1
> enabling repair mode
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 6358304a-2234-4243-b02d-4944c9af47d7
> badcextentx[29368320, 29372416), type mismatch with chunk
> bad extent [29372416, 29376512), type mismatch with chunk
> 
> (2856970 lines deleted)
> 
> bad extent [8697338122240, 8697338126336), type mismatch with chunk
> bad extent [8697338126336, 8697338130432), type mismatch with chunk
> bad extent [8697338130432, 8697338134528), type mismatch with chunk

   This is, I think, a symptom of an FS created with a broken
mkfs.btrfs, and it needs to be re-created. Take a look for that error
message in the mailing list archives -- there's been a few posts about
it in the last couple of months.

   Hugo.

> checking extents [.]
> 
> repaired damaged extent references
> 
> Fixed 0 roots.
> cache and super generation don't match, space cache will be invalidated
> Fixedidiscountofileoextents for inode: 204450 in root: 45851
> FixedidiscountofileOextents for inode: 204452 in root: 45851
> root 45851 inode 204452 errors 40, bad file extent
> Fixedidiscountofileoextents for inode: 204452 in root: 45851
> root 45851 inode 204452 errors 40, bad file extent
> rootk45851sinodes204452 errors 40, bad file extent
> FixedidiscountofileOextents for inode: 204450 in root: 45852
> FixedidiscountofileOextents for inode: 204452 in root: 45852
> checking fs roots [o]
> rootk45851sinodes204452 errors 40, bad file extent
> Fixedidiscountofile.extents for inode: 204450 in root: 45856
> Fixed discount file extents for inode: 204452 in root: 45856
> rootk45851sinodes204452 errors 40, bad file extent
> warninggliner3653 [o]
> 
> checking csums
> checking root refs
> found 9826147025859 bytes used err is 0
> total csum bytes: 9584068648
> total tree bytes: 12200706048
> total fs tree bytes: 330457088
> total extent tree bytes: 498716672
> btree space waste bytes: 1372963760
> file data blocks allocated: 9976766078976
>  referenced 9987816431616
> btrfs-progs v4.3
> 
> Thanks,
> Marc

-- 
Hugo Mills             | In my day, we didn't have fancy high numbers. We had
hugo@... carfax.org.uk | "nowt", "one", "twain" and "multitudes".
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Marc MERLIN]

On Mon, Jan 18, 2016 at 03:21:53AM +0000, Duncan wrote:
> No.  Those are monotonically increasing counts that are never 
> automatically reset over the life of the filesystem.  Use ...
> 
> btrfs dev stats
> 
> ... to display them on the commandline (vs. the kernel log at filesystem 
> mount), with the -z option to reset them after display, if desired.

It's a bit counter intuitive that check --repair doesn't reset error counts if it fixed
underlying errors, but maybe there is a good reason :)

btrfs dev stats -z /dev/mapper/dshelf1
put everything back to 0 as you hinted, I can now watch what's going on
from here on, thanks.

On Mon, Jan 18, 2016 at 12:45:33PM +0000, Hugo Mills wrote:
> > bad extent [8697338122240, 8697338126336), type mismatch with chunk
> > bad extent [8697338126336, 8697338130432), type mismatch with chunk
> > bad extent [8697338130432, 8697338134528), type mismatch with chunk
> 
>    This is, I think, a symptom of an FS created with a broken
> mkfs.btrfs, and it needs to be re-created. Take a look for that error
> message in the mailing list archives -- there's been a few posts about
> it in the last couple of months.
 
Thanks for that other hint.
It was created quite a while ago (1y+), and ran ok until I had an
unexpected crash.
If I have to rebuild it, I will, but that will take 2 days+ due to the 
size and getting the backup back (well also, I'm not home for a week, so
restoring backups remotely isn't going to be fun).

But sure enough, while it ran perfectly fine for a long time, after check --repair,
my machine is now crashing every hour or so, with the crash below.

I had to have an older kernel due to a separate problem where PMP (sata
port multiplier) support was broken in more recent kernels.
I've just put 4.3.3 on it to see if crashes still, but either way, I thought
btrfs had gotten rid of its last BUG_ON(xxx). System should never crash
due to unexpected filesystem state on a non system partition.

Mmmh, scratch this, here's the BUG_ON, it's a memory problem :(
fs/btrfs/ctree.c:5200
		/* save our key for returning back */
		btrfs_node_key_to_cpu(cur, &found_key, slot);
		path->slots[level] = slot;
		if (level == path->lowest_level) {
			ret = 0;
			goto out;
		}
		btrfs_set_path_blocking(path);
		cur = read_node_slot(root, cur, slot);
		BUG_ON(!cur); /* -ENOMEM */

Did check --repair potentially change my FS in a way that is now making the
kernel take a lot of RAM and crash?


------------[ cut here ]------------
kernel BUG at fs/btrfs/ctree.c:5200!
invalid opcode: 0000 [#1] SMP
CPU: 3 PID: 29041 Comm: btrfs Tainted: G        W       4.2.5-amd64-i915-volpreempt-20150421 #4
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
task: ffff88002f8482c0 ti: ffff8800c9058000 task.ti: ffff8800c9058000
RIP: 0010:[<ffffffff81234001>]  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
RSP: 0018:ffff8800c905bbc8  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8801494409b0 RCX: 000000000003c6c8
RDX: 0000000000000000 RSI: 000000000aa60000 RDI: fffffffffffffffb
RBP: ffff8800c905bc38 R08: 0000000001c00000 R09: 0000000000000000
R10: 0000000000aaaaaa R11: ffffffff818799e0 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000000000 R15: ffff8801494409b4
FS:  00007fa9d46848c0(0000) GS:ffff88021e2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000f77cd000 CR3: 0000000143780000 CR4: 00000000001406e0
Stack:
 ffff8800c9bb6050 0001880149440a18 ffff8800c905bc87 ffff8800c939a800
 0000000000000003 1a0000000000002e 84000000000000b3 000000000003c59e
 000000000000002c ffff8801494409b0 0000000000000000 ffff8800c905bce0
Call Trace:
 [<ffffffff81278ca5>] search_ioctl+0xfc/0x167
 [<ffffffff81278d6b>] btrfs_ioctl_tree_search+0x5b/0x8e
 [<ffffffff8127c6af>] btrfs_ioctl+0x396/0x232f
 [<ffffffff81123a2d>] ? get_page+0xe/0x28
 [<ffffffff81123e68>] ? __lru_cache_add+0x23/0x44
 [<ffffffff81124139>] ? lru_cache_add_active_or_unevictable+0x2d/0x6b
 [<ffffffff8113a2df>] ? set_pte_at+0x9/0xd
 [<ffffffff8113e21d>] ? handle_mm_fault+0x90e/0xe9b
 [<ffffffff8100d02b>] ? paravirt_write_msr+0xf/0x13
 [<ffffffff811814af>] do_vfs_ioctl+0x39b/0x412
 [<ffffffff810ace61>] ? current_kernel_time+0xe/0x32
 [<ffffffff810d4d5c>] ? __audit_syscall_entry+0xbe/0xe0
 [<ffffffff81181580>] SyS_ioctl+0x5a/0x7f
 [<ffffffff816b0032>] entry_SYSCALL_64_fastpath+0x16/0x75
Code: 89 47 40 44 3b ab 84 00 00 00 74 69 48 89 df e8 f7 a3 ff ff 8b 55 b8 48 8b 7d a8 4c 89 e6 e8 d8 86 ff ff 48 85 c0 
<0f> 0b 48 89 c7 e8 23 a7 04 00 41 8d 45 ff 41 c7 47 5c 02 00 00
RIP  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
 RSP <ffff8800c905bbc8>
---[ end trace e3c37adcaa703f63 ]---
Kernel panic - not syncing: Fatal exception
Kernel Offset: disabled
drm_kms_helper: panic occurred, switching back to text console




-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Duncan]

Marc MERLIN posted on Mon, 18 Jan 2016 15:39:43 -0800 as excerpted:

> On Mon, Jan 18, 2016 at 03:21:53AM +0000, Duncan wrote:
>> No.  Those are monotonically increasing counts that are never
>> automatically reset over the life of the filesystem.  Use ...
>> 
>> btrfs dev stats
>> 
>> ... to display them on the commandline (vs. the kernel log at
>> filesystem mount), with the -z option to reset them after display, if
>> desired.
> 
> It's a bit counter intuitive that check --repair doesn't reset error
> counts if it fixed underlying errors, but maybe there is a good reason
> :)

The idea is, I believe, that a new admin/tech or someone who hasn't been 
paying attention can get a quick idea of where the problems have been and 
may well still be, by looking at the error counts built up over the 
lifetime of the filesystem.  An admin who is paying attention, meanwhile, 
can always deliberately zero out the error counts once they've fixed 
whatever was causing the problem triggering the errors in the first place.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Marc MERLIN]

On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
> Thanks for that other hint.
> It was created quite a while ago (1y+), and ran ok until I had an
> unexpected crash.
> If I have to rebuild it, I will, but that will take 2 days+ due to the 
> size and getting the backup back (well also, I'm not home for a week, so
> restoring backups remotely isn't going to be fun).
> 
> But sure enough, while it ran perfectly fine for a long time, after check --repair,
> my machine is now crashing every hour or so, with the crash below.

I ran check --repair a few more times on it but it does not seem to converge.
The bad extent stuff, I understand cannot be fixed (although it's not obvious from check
--repair that they are not getting fixed).

But how about 
root 45940 inode 204450 errors 1000, some csum missing
Are those getting fixed by check --repair or are they unfixable too?
 
(...)
bad extent [8697338126336, 8697338130432), type mismatch with chunk
bad extent [8697338130432, 8697338134528), type mismatch with chunk
repaired damaged extent references

Fixed 0 roots.
cache and super generation don't match, space cache will be invalidated
rootk262 inodeo204450 errors 1000, some csum missing
root 262 inode 204452 errors 1000, some csum missing
(...)
root 45940 inode 204450 errors 1000, some csum missing
root 45940 inode 204452 errors 1000, some csum missing
root 45944 inode 204450 errors 1000, some csum missing
root 45944 inode 204452 errors 1000, some csum missing
root 45948 inode 204450 errors 1000, some csum missing
root 45948 inode 204452 errors 1000, some csum missing
checking fs roots [o]
checking csums
checking root refs
found 9826295305149 bytes used err is 0
total csum bytes: 9584154948
total tree bytes: 12201304064
total fs tree bytes: 330776576
total extent tree bytes: 498921472
btree space waste bytes: 1373183541
file data blocks allocated: 9953275256832
 referenced 9964596801536
btrfs-progs v4.3


Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Marc MERLIN]

+David, +Qu
about 
1) kernel crash on BUG_ON
2) check --repair not giving good clue that
"type mismatch with chunk" is unfixable, and whether it can be kind of
ignored or whether your FS really needs to be recreated from scratch
(many hours of work for me, and probably 2 days of clock time to rebuild
and restore from backup)
3) say more about "root 45948 inode 204452 errors 1000, some csum missing",
that they aren't being fixed, and whether they're a big deal or not.

More generally I'm curious to know if check --repair will sometimes fix
more things on a 2nd (or 3rd...) run than on the first one.

Thanks,
Marc

On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
> On Mon, Jan 18, 2016 at 03:21:53AM +0000, Duncan wrote:
> > No.  Those are monotonically increasing counts that are never 
> > automatically reset over the life of the filesystem.  Use ...
> > 
> > btrfs dev stats
> > 
> > ... to display them on the commandline (vs. the kernel log at filesystem 
> > mount), with the -z option to reset them after display, if desired.
> 
> It's a bit counter intuitive that check --repair doesn't reset error counts if it fixed
> underlying errors, but maybe there is a good reason :)
> 
> btrfs dev stats -z /dev/mapper/dshelf1
> put everything back to 0 as you hinted, I can now watch what's going on
> from here on, thanks.
> 
> On Mon, Jan 18, 2016 at 12:45:33PM +0000, Hugo Mills wrote:
> > > bad extent [8697338122240, 8697338126336), type mismatch with chunk
> > > bad extent [8697338126336, 8697338130432), type mismatch with chunk
> > > bad extent [8697338130432, 8697338134528), type mismatch with chunk
> > 
> >    This is, I think, a symptom of an FS created with a broken
> > mkfs.btrfs, and it needs to be re-created. Take a look for that error
> > message in the mailing list archives -- there's been a few posts about
> > it in the last couple of months.
>  
> Thanks for that other hint.
> It was created quite a while ago (1y+), and ran ok until I had an
> unexpected crash.
> If I have to rebuild it, I will, but that will take 2 days+ due to the 
> size and getting the backup back (well also, I'm not home for a week, so
> restoring backups remotely isn't going to be fun).
> 
> But sure enough, while it ran perfectly fine for a long time, after check --repair,
> my machine is now crashing every hour or so, with the crash below.
> 
> I had to have an older kernel due to a separate problem where PMP (sata
> port multiplier) support was broken in more recent kernels.
> I've just put 4.3.3 on it to see if crashes still, but either way, I thought
> btrfs had gotten rid of its last BUG_ON(xxx). System should never crash
> due to unexpected filesystem state on a non system partition.
> 
> Mmmh, scratch this, here's the BUG_ON, it's a memory problem :(
> fs/btrfs/ctree.c:5200
> 		/* save our key for returning back */
> 		btrfs_node_key_to_cpu(cur, &found_key, slot);
> 		path->slots[level] = slot;
> 		if (level == path->lowest_level) {
> 			ret = 0;
> 			goto out;
> 		}
> 		btrfs_set_path_blocking(path);
> 		cur = read_node_slot(root, cur, slot);
> 		BUG_ON(!cur); /* -ENOMEM */
> 
> Did check --repair potentially change my FS in a way that is now making the
> kernel take a lot of RAM and crash?
> 
> 
> ------------[ cut here ]------------
> kernel BUG at fs/btrfs/ctree.c:5200!
> invalid opcode: 0000 [#1] SMP
> CPU: 3 PID: 29041 Comm: btrfs Tainted: G        W       4.2.5-amd64-i915-volpreempt-20150421 #4
> Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> task: ffff88002f8482c0 ti: ffff8800c9058000 task.ti: ffff8800c9058000
> RIP: 0010:[<ffffffff81234001>]  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
> RSP: 0018:ffff8800c905bbc8  EFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff8801494409b0 RCX: 000000000003c6c8
> RDX: 0000000000000000 RSI: 000000000aa60000 RDI: fffffffffffffffb
> RBP: ffff8800c905bc38 R08: 0000000001c00000 R09: 0000000000000000
> R10: 0000000000aaaaaa R11: ffffffff818799e0 R12: 0000000000000000
> R13: 0000000000000001 R14: 0000000000000000 R15: ffff8801494409b4
> FS:  00007fa9d46848c0(0000) GS:ffff88021e2c0000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000f77cd000 CR3: 0000000143780000 CR4: 00000000001406e0
> Stack:
>  ffff8800c9bb6050 0001880149440a18 ffff8800c905bc87 ffff8800c939a800
>  0000000000000003 1a0000000000002e 84000000000000b3 000000000003c59e
>  000000000000002c ffff8801494409b0 0000000000000000 ffff8800c905bce0
> Call Trace:
>  [<ffffffff81278ca5>] search_ioctl+0xfc/0x167
>  [<ffffffff81278d6b>] btrfs_ioctl_tree_search+0x5b/0x8e
>  [<ffffffff8127c6af>] btrfs_ioctl+0x396/0x232f
>  [<ffffffff81123a2d>] ? get_page+0xe/0x28
>  [<ffffffff81123e68>] ? __lru_cache_add+0x23/0x44
>  [<ffffffff81124139>] ? lru_cache_add_active_or_unevictable+0x2d/0x6b
>  [<ffffffff8113a2df>] ? set_pte_at+0x9/0xd
>  [<ffffffff8113e21d>] ? handle_mm_fault+0x90e/0xe9b
>  [<ffffffff8100d02b>] ? paravirt_write_msr+0xf/0x13
>  [<ffffffff811814af>] do_vfs_ioctl+0x39b/0x412
>  [<ffffffff810ace61>] ? current_kernel_time+0xe/0x32
>  [<ffffffff810d4d5c>] ? __audit_syscall_entry+0xbe/0xe0
>  [<ffffffff81181580>] SyS_ioctl+0x5a/0x7f
>  [<ffffffff816b0032>] entry_SYSCALL_64_fastpath+0x16/0x75
> Code: 89 47 40 44 3b ab 84 00 00 00 74 69 48 89 df e8 f7 a3 ff ff 8b 55 b8 48 8b 7d a8 4c 89 e6 e8 d8 86 ff ff 48 85 c0 
> <0f> 0b 48 89 c7 e8 23 a7 04 00 41 8d 45 ff 41 c7 47 5c 02 00 00
> RIP  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
>  RSP <ffff8800c905bbc8>
> ---[ end trace e3c37adcaa703f63 ]---
> Kernel panic - not syncing: Fatal exception
> Kernel Offset: disabled
> drm_kms_helper: panic occurred, switching back to text console
> 
> 
> 
> 
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

On Wed, Jan 20, 2016 at 08:52:39PM -0800, Marc MERLIN wrote:
> On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
> > Thanks for that other hint.
> > It was created quite a while ago (1y+), and ran ok until I had an
> > unexpected crash.
> > If I have to rebuild it, I will, but that will take 2 days+ due to the 
> > size and getting the backup back (well also, I'm not home for a week, so
> > restoring backups remotely isn't going to be fun).
> > 
> > But sure enough, while it ran perfectly fine for a long time, after check --repair,
> > my machine is now crashing every hour or so, with the crash below.
> 
> I ran check --repair a few more times on it but it does not seem to converge.
> The bad extent stuff, I understand cannot be fixed (although it's not obvious from check
> --repair that they are not getting fixed).
> 
> But how about 
> root 45940 inode 204450 errors 1000, some csum missing
> Are those getting fixed by check --repair or are they unfixable too?
>  
> (...)
> bad extent [8697338126336, 8697338130432), type mismatch with chunk
> bad extent [8697338130432, 8697338134528), type mismatch with chunk
> repaired damaged extent references
> 
> Fixed 0 roots.
> cache and super generation don't match, space cache will be invalidated
> rootk262 inodeo204450 errors 1000, some csum missing
> root 262 inode 204452 errors 1000, some csum missing
> (...)
> root 45940 inode 204450 errors 1000, some csum missing
> root 45940 inode 204452 errors 1000, some csum missing
> root 45944 inode 204450 errors 1000, some csum missing
> root 45944 inode 204452 errors 1000, some csum missing
> root 45948 inode 204450 errors 1000, some csum missing
> root 45948 inode 204452 errors 1000, some csum missing
> checking fs roots [o]
> checking csums
> checking root refs
> found 9826295305149 bytes used err is 0
> total csum bytes: 9584154948
> total tree bytes: 12201304064
> total fs tree bytes: 330776576
> total extent tree bytes: 498921472
> btree space waste bytes: 1373183541
> file data blocks allocated: 9953275256832
>  referenced 9964596801536
> btrfs-progs v4.3
> 
> 
> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Marc MERLIN]

On Sat, Jan 23, 2016 at 09:03:54AM -0800, Marc MERLIN wrote:
> +David, +Qu
> about 
> 1) kernel crash on BUG_ON
> 2) check --repair not giving good clue that
> "type mismatch with chunk" is unfixable, and whether it can be kind of
> ignored or whether your FS really needs to be recreated from scratch
> (many hours of work for me, and probably 2 days of clock time to rebuild
> and restore from backup)
> 3) say more about "root 45948 inode 204452 errors 1000, some csum missing",
> that they aren't being fixed, and whether they're a big deal or not.
> 
> More generally I'm curious to know if check --repair will sometimes fix
> more things on a 2nd (or 3rd...) run than on the first one.

for what it's worth, even after check --repair, the filesystem still
causes the kernel to crash:
btrfs: page allocation failure: order:1, mode:0x204020
CPU: 0 PID: 30591 Comm: btrfs Not tainted 4.3.3-amd64-i915-volpreempt-20150421 #2
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
 0000000000000000 ffff880001fb3708 ffffffff8134150e 0000000000000000
 ffff880001fb37a0 ffffffff8111f6ce 00000000fffffffe 0000000000000000
 ffff880001fb3738 ffffffff816c1143 ffff880001fb3768 ffff88021f5f4e00
Call Trace:
 [<ffffffff8134150e>] dump_stack+0x44/0x55
 [<ffffffff8111f6ce>] warn_alloc_failed+0x111/0x129
 [<ffffffff816c1143>] ? _raw_spin_unlock_irqrestore+0x14/0x16
 [<ffffffff811220f8>] __alloc_pages_nodemask+0x6ae/0x70d
 [<ffffffff8115fb98>] kmem_getpages+0x6a/0x162
 [<ffffffff8115fd89>] fallback_alloc+0xf9/0x193
 [<ffffffff8115ff46>] ____cache_alloc_node+0x123/0x130
 [<ffffffff81160c1d>] kmem_cache_alloc+0xdc/0x187
 [<ffffffff81341dfc>] ida_pre_get+0x32/0xb6
 [<ffffffff8117999e>] get_anon_bdev+0x1f/0xc8
 [<ffffffff812507e5>] btrfs_init_fs_root+0x104/0x14e
 [<ffffffff812518c9>] btrfs_get_fs_root+0xb7/0x1bf
 [<ffffffff812556e1>] create_pending_snapshot+0x65e/0xb09
 [<ffffffff81080c4c>] ? get_sd_balance_interval.isra.34+0x17/0x33
 [<ffffffff81255bfe>] create_pending_snapshots+0x72/0x8e
 [<ffffffff81255bfe>] ? create_pending_snapshots+0x72/0x8e
 [<ffffffff81256b22>] btrfs_commit_transaction+0x423/0x9c1
 [<ffffffff81257440>] ? start_transaction+0x380/0x50b
 [<ffffffff81089e9d>] ? wake_up_atomic_t+0x2c/0x2c
 [<ffffffff812812e3>] btrfs_mksubvol+0x2f4/0x427
 [<ffffffff81089e9d>] ? wake_up_atomic_t+0x2c/0x2c
 [<ffffffff8128155e>] btrfs_ioctl_snap_create_transid+0x148/0x17a
 [<ffffffff812816be>] btrfs_ioctl_snap_create_v2+0xc7/0x110
 [<ffffffff812839d3>] btrfs_ioctl+0x545/0x233e
 [<ffffffff81126cc0>] ? get_page+0xd/0x26
 [<ffffffff811270f2>] ? __lru_cache_add+0x23/0x44
 [<ffffffff811273bf>] ? lru_cache_add_active_or_unevictable+0x2d/0x6b
 [<ffffffff8113d6ab>] ? set_pte_at+0x9/0xd
 [<ffffffff8114623b>] ? do_mmap+0x2de/0x327
 [<ffffffff81186550>] do_vfs_ioctl+0x39b/0x412
 [<ffffffff81003482>] ? do_audit_syscall_entry+0x60/0x62
 [<ffffffff8118661e>] SyS_ioctl+0x57/0x79
 [<ffffffff816c1376>] entry_SYSCALL_64_fastpath+0x16/0x75
Mem-Info:
active_anon:415090 inactive_anon:219505 isolated_anon:0
 active_file:463628 inactive_file:550449 isolated_file:0
 unevictable:1202 dirty:29212 writeback:3325 unstable:0
 slab_reclaimable:61297 slab_unreclaimable:48704
 mapped:418528 shmem:409992 pagetables:4730 bounce:0
 free:19084 free_pcp:776 free_cma:0
Node 0 DMA free:15896kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3203 7674 7674
Node 0 DMA32 free:48860kB min:4644kB low:5804kB high:6964kB active_anon:452600kB inactive_anon:463072kB active_file:838520kB inactive_file:1040240kB unevictable:1152kB isolated(anon):0kB isolated(file):0kB present:3362068kB managed:3283400kB mlocked:1152kB dirty:16192kB writeback:2176kB mapped:702760kB shmem:675320kB slab_reclaimable:76144kB slab_unreclaimable:75212kB kernel_stack:5616kB pagetables:4956kB unstable:0kB bounce:0kB free_pcp:1504kB local_pcp:32kB free_cma:0kB writeback_tmp:0kB pages_scanned:252 all_unreclaimable? no

lowmem_reserve[]: 0 0 4471 4471
Node 0 Normal free:7872kB min:6480kB low:8100kB high:9720kB active_anon:1207760kB inactive_anon:414948kB active_file:1015992kB inactive_file:1161556kB unevictable:3656kB isolated(anon):0kB isolated(file):0kB present:4708352kB managed:4578508kB mlocked:3656kB dirty:95268kB writeback:16596kB mapped:971352kB shmem:964648kB slab_reclaimable:169044kB slab_unreclaimable:119604kB kernel_stack:5664kB pagetables:13964kB unstable:0kB bounce:0kB free_pcp:1320kB local_pcp:296kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
Node 0 DMA32: 11151*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44604kB
Node 0 Normal: 1899*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7596kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
1455704 total pagecache pages
30458 pages in swap cache
Swap cache stats: add 405385, delete 374927, find 3385111/3435323
Free swap  = 15024996kB
Total swap = 15616764kB
2021600 pages RAM
0 pages HighMem/MovableOnly
52149 pages reserved
4096 pages cma reserved
0 pages hwpoisoned
------------[ cut here ]------------
WARNING: CPU: 0 PID: 30591 at fs/btrfs/transaction.c:1476 create_pending_snapshot+0x6b2/0xb09()
Modules linked in: udp_diag tcp_diag inet_diag loop veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp tun xt_addrtype bridge stp llc autofs4 softdog binfmt_misc ftdi_sio nfsd nfs_acl auth_rpcgss nfs fscache lockd grace sunrpc ipt_REJECT nf_reject_ipv4 xt_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl2303 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables nf_conntrack_ftp ipt_MASQUERADE x_tables nf_nat_masquerade_ipv4 nf_nat nf_conntrack sg st snd_pcm_oss snd_mixer_oss snd_hda_codec_realtek

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Qu Wenruo]

Marc MERLIN wrote on 2016/01/23 09:03 -0800:
> +David, +Qu
> about
> 1) kernel crash on BUG_ON

 From your code mentioned, and your second kernel warning, it's out of 
memory.
Such case also happened when I was debugging in-band de-dup patches.

Things seems that by some method, btrfs used a lot of memory for dirty 
page caches. Maybe metadata pages.

Normally when such case happens, VFS should trigger a sync to free dirty 
pages, but btrfs seems to either delayed the sync due to running trans 
or the VFS sync is already too late.

But it's also possible that large leafsize is related to such problem.
The larger leafsize is, the harder to alloc continuous memory for kmalloc().

> 2) check --repair not giving good clue that
> "type mismatch with chunk" is unfixable, and whether it can be kind of
> ignored or whether your FS really needs to be recreated from scratch
> (many hours of work for me, and probably 2 days of clock time to rebuild
> and restore from backup)

For "type mismatch" problem, it's unfixable, but as far as what we know, 
it shouldn't cause big problem.

If you're using old version btrfsck, then it's possible such error is a 
false alert. Update btrfsck and try again is a good idea.

Even if it's not a false alert, mail list says it shouldn't cause huge 
problem, only known problems happens is related to scrub.
And there is already some user reporting balance can fix it, although 
you need to balance all chunks.

> 3) say more about "root 45948 inode 204452 errors 1000, some csum missing",
> that they aren't being fixed, and whether they're a big deal or not.

Personally speaking, I didn't consider it as a big problem itself.
If csum is missing/corrupted, btrfsck --init-csum-tree can rebuild it.

But the root problem is more important, maybe some extent/csum tree 
blocks are corrupted and failed to find the checksum, or kernel bugs 
causing csum missing.

Although, from your very first report, a lot of file extents are either 
corrupted or in bad shape.
Not sure how accurate the original/repaired data is.

>
> More generally I'm curious to know if check --repair will sometimes fix
> more things on a 2nd (or 3rd...) run than on the first one.

It is possible that further run to fix more problems, considering the 
already complicated fixing codes.
But I also consider the possibility not very high though.

I hope I could have your latest btrfsck result to further investigate 
what's wrong (at least what's wrong on disk).

Thanks,
Qu

>
> Thanks,
> Marc
>
> On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
>> On Mon, Jan 18, 2016 at 03:21:53AM +0000, Duncan wrote:
>>> No.  Those are monotonically increasing counts that are never
>>> automatically reset over the life of the filesystem.  Use ...
>>>
>>> btrfs dev stats
>>>
>>> ... to display them on the commandline (vs. the kernel log at filesystem
>>> mount), with the -z option to reset them after display, if desired.
>>
>> It's a bit counter intuitive that check --repair doesn't reset error counts if it fixed
>> underlying errors, but maybe there is a good reason :)
>>
>> btrfs dev stats -z /dev/mapper/dshelf1
>> put everything back to 0 as you hinted, I can now watch what's going on
>> from here on, thanks.
>>
>> On Mon, Jan 18, 2016 at 12:45:33PM +0000, Hugo Mills wrote:
>>>> bad extent [8697338122240, 8697338126336), type mismatch with chunk
>>>> bad extent [8697338126336, 8697338130432), type mismatch with chunk
>>>> bad extent [8697338130432, 8697338134528), type mismatch with chunk
>>>
>>>     This is, I think, a symptom of an FS created with a broken
>>> mkfs.btrfs, and it needs to be re-created. Take a look for that error
>>> message in the mailing list archives -- there's been a few posts about
>>> it in the last couple of months.
>>
>> Thanks for that other hint.
>> It was created quite a while ago (1y+), and ran ok until I had an
>> unexpected crash.
>> If I have to rebuild it, I will, but that will take 2 days+ due to the
>> size and getting the backup back (well also, I'm not home for a week, so
>> restoring backups remotely isn't going to be fun).
>>
>> But sure enough, while it ran perfectly fine for a long time, after check --repair,
>> my machine is now crashing every hour or so, with the crash below.
>>
>> I had to have an older kernel due to a separate problem where PMP (sata
>> port multiplier) support was broken in more recent kernels.
>> I've just put 4.3.3 on it to see if crashes still, but either way, I thought
>> btrfs had gotten rid of its last BUG_ON(xxx). System should never crash
>> due to unexpected filesystem state on a non system partition.
>>
>> Mmmh, scratch this, here's the BUG_ON, it's a memory problem :(
>> fs/btrfs/ctree.c:5200
>> 		/* save our key for returning back */
>> 		btrfs_node_key_to_cpu(cur, &found_key, slot);
>> 		path->slots[level] = slot;
>> 		if (level == path->lowest_level) {
>> 			ret = 0;
>> 			goto out;
>> 		}
>> 		btrfs_set_path_blocking(path);
>> 		cur = read_node_slot(root, cur, slot);
>> 		BUG_ON(!cur); /* -ENOMEM */
>>
>> Did check --repair potentially change my FS in a way that is now making the
>> kernel take a lot of RAM and crash?
>>
>>
>> ------------[ cut here ]------------
>> kernel BUG at fs/btrfs/ctree.c:5200!
>> invalid opcode: 0000 [#1] SMP
>> CPU: 3 PID: 29041 Comm: btrfs Tainted: G        W       4.2.5-amd64-i915-volpreempt-20150421 #4
>> Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
>> task: ffff88002f8482c0 ti: ffff8800c9058000 task.ti: ffff8800c9058000
>> RIP: 0010:[<ffffffff81234001>]  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
>> RSP: 0018:ffff8800c905bbc8  EFLAGS: 00010246
>> RAX: 0000000000000000 RBX: ffff8801494409b0 RCX: 000000000003c6c8
>> RDX: 0000000000000000 RSI: 000000000aa60000 RDI: fffffffffffffffb
>> RBP: ffff8800c905bc38 R08: 0000000001c00000 R09: 0000000000000000
>> R10: 0000000000aaaaaa R11: ffffffff818799e0 R12: 0000000000000000
>> R13: 0000000000000001 R14: 0000000000000000 R15: ffff8801494409b4
>> FS:  00007fa9d46848c0(0000) GS:ffff88021e2c0000(0000) knlGS:0000000000000000
>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 00000000f77cd000 CR3: 0000000143780000 CR4: 00000000001406e0
>> Stack:
>>   ffff8800c9bb6050 0001880149440a18 ffff8800c905bc87 ffff8800c939a800
>>   0000000000000003 1a0000000000002e 84000000000000b3 000000000003c59e
>>   000000000000002c ffff8801494409b0 0000000000000000 ffff8800c905bce0
>> Call Trace:
>>   [<ffffffff81278ca5>] search_ioctl+0xfc/0x167
>>   [<ffffffff81278d6b>] btrfs_ioctl_tree_search+0x5b/0x8e
>>   [<ffffffff8127c6af>] btrfs_ioctl+0x396/0x232f
>>   [<ffffffff81123a2d>] ? get_page+0xe/0x28
>>   [<ffffffff81123e68>] ? __lru_cache_add+0x23/0x44
>>   [<ffffffff81124139>] ? lru_cache_add_active_or_unevictable+0x2d/0x6b
>>   [<ffffffff8113a2df>] ? set_pte_at+0x9/0xd
>>   [<ffffffff8113e21d>] ? handle_mm_fault+0x90e/0xe9b
>>   [<ffffffff8100d02b>] ? paravirt_write_msr+0xf/0x13
>>   [<ffffffff811814af>] do_vfs_ioctl+0x39b/0x412
>>   [<ffffffff810ace61>] ? current_kernel_time+0xe/0x32
>>   [<ffffffff810d4d5c>] ? __audit_syscall_entry+0xbe/0xe0
>>   [<ffffffff81181580>] SyS_ioctl+0x5a/0x7f
>>   [<ffffffff816b0032>] entry_SYSCALL_64_fastpath+0x16/0x75
>> Code: 89 47 40 44 3b ab 84 00 00 00 74 69 48 89 df e8 f7 a3 ff ff 8b 55 b8 48 8b 7d a8 4c 89 e6 e8 d8 86 ff ff 48 85 c0
>> <0f> 0b 48 89 c7 e8 23 a7 04 00 41 8d 45 ff 41 c7 47 5c 02 00 00
>> RIP  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
>>   RSP <ffff8800c905bbc8>
>> ---[ end trace e3c37adcaa703f63 ]---
>> Kernel panic - not syncing: Fatal exception
>> Kernel Offset: disabled
>> drm_kms_helper: panic occurred, switching back to text console
>>
>>
>>
>>
>> --
>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> Microsoft is to operating systems ....
>>                                        .... what McDonalds is to gourmet cooking
>> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> On Wed, Jan 20, 2016 at 08:52:39PM -0800, Marc MERLIN wrote:
>> On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
>>> Thanks for that other hint.
>>> It was created quite a while ago (1y+), and ran ok until I had an
>>> unexpected crash.
>>> If I have to rebuild it, I will, but that will take 2 days+ due to the
>>> size and getting the backup back (well also, I'm not home for a week, so
>>> restoring backups remotely isn't going to be fun).
>>>
>>> But sure enough, while it ran perfectly fine for a long time, after check --repair,
>>> my machine is now crashing every hour or so, with the crash below.
>>
>> I ran check --repair a few more times on it but it does not seem to converge.
>> The bad extent stuff, I understand cannot be fixed (although it's not obvious from check
>> --repair that they are not getting fixed).
>>
>> But how about
>> root 45940 inode 204450 errors 1000, some csum missing
>> Are those getting fixed by check --repair or are they unfixable too?
>>
>> (...)
>> bad extent [8697338126336, 8697338130432), type mismatch with chunk
>> bad extent [8697338130432, 8697338134528), type mismatch with chunk
>> repaired damaged extent references
>>
>> Fixed 0 roots.
>> cache and super generation don't match, space cache will be invalidated
>> rootk262 inodeo204450 errors 1000, some csum missing
>> root 262 inode 204452 errors 1000, some csum missing
>> (...)
>> root 45940 inode 204450 errors 1000, some csum missing
>> root 45940 inode 204452 errors 1000, some csum missing
>> root 45944 inode 204450 errors 1000, some csum missing
>> root 45944 inode 204452 errors 1000, some csum missing
>> root 45948 inode 204450 errors 1000, some csum missing
>> root 45948 inode 204452 errors 1000, some csum missing
>> checking fs roots [o]
>> checking csums
>> checking root refs
>> found 9826295305149 bytes used err is 0
>> total csum bytes: 9584154948
>> total tree bytes: 12201304064
>> total fs tree bytes: 330776576
>> total extent tree bytes: 498921472
>> btree space waste bytes: 1373183541
>> file data blocks allocated: 9953275256832
>>   referenced 9964596801536
>> btrfs-progs v4.3
>>
>>
>> Thanks,
>> Marc
>> --
>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> Microsoft is to operating systems ....
>>                                        .... what McDonalds is to gourmet cooking
>> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>

4.4.0: btrfs-send BUG_ON(sctx->cur_ino != sctx->cmp_key->objectid);

[Marc MERLIN]

I still have 2 more days before I can rebuild my broken filesystem.
In the meantime, I just got this new error with 4.4
Actually I'm assuming it happened on the broken filesystem, but maybe
not, it may have happened on my other non broken FS that was also doing
btrfs send.


static int changed_xattr(struct send_ctx *sctx,
			 enum btrfs_compare_tree_result result)
{
	int ret = 0;

	BUG_ON(sctx->cur_ino != sctx->cmp_key->objectid);



BTRFS error (device dm-1): bdev /dev/mapper/dshelf1 errs: wr 279, rd 0, flush 0, corrupt 0, gen 0
------------[ cut here ]------------
kernel BUG at fs/btrfs/send.c:5639!
invalid opcode: 0000 [#1] SMP
Modules linked in: veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp xt_addrtype bridge tun stp llc autofs4 softdog binfmt_misc ftdi_sio nfsd nfs_acl auth_rpcgss nfs fscache lockd grace sunrpc ipt_REJECT nf_reject_ipv4 xt_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl2303 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables nf_conntrack_ftp ipt_MASQUERADE x_tables nf_nat_masquerade_ipv4 nf_nat nf_conntrack sg st snd_pcm_oss snd_mixer_oss snd_opl3_synth snd_hda_codec_realtek snd_seq_midi_emul snd_cmipci snd_hda_codec_generic gameport snd_opl3_lib snd_hda_intel kvm_intel snd_mpu401_uart snd_seq_midi snd_hda_codec kvm snd_seq_midi_event snd_hda_core snd_seq irqbypass snd_hwdep snd_rawmidi snd_pcm snd_seq_device snd_timer tpm_infineon tpm_tis rc_ati_x10 eeepc_wmi coretemp tpm 8250_fintek asus_wmi battery intel_rapl asix pcspkr sparse_keymap libphy snd iosf_mbi ati_remote xhci_pci processor lpc_ich rfkill intel_powerclamp evdev i2c_i801 soundcore parport_pc x86_pkg_temp_thermal wmi rc_core usbnet input_leds xhci_hcd parport usbserial e1000e ptp pps_core fuse raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common ecb xts ansi_cprng drbg aesni_intel ablk_helper lrw gf128mul glue_helper aes_x86_64 crc32c_intel crct10dif_pclmul ehci_pci crc32_pclmul fjes ehci_hcd sata_sil24 thermal sata_mv r8169 cryptd mii usbcore fan usb_common [last unloaded: ftdi_sio]
CPU: 1 PID: 8229 Comm: btrfs Not tainted 4.4.0-amd64-i915-volpreempt-20150421 #1
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
task: ffff880114376300 ti: ffff88011b9dc000 task.ti: ffff88011b9dc000
RIP: 0010:[<ffffffff812b6bc9>]  [<ffffffff812b6bc9>] changed_cb+0x549/0x8bf
RSP: 0018:ffff88011b9dfb80  EFLAGS: 00210293
RAX: 0000000000031ea2 RBX: 0000000000000000 RCX: 0000000000031e60
RDX: ffff88011b9dfc5e RSI: 0000000000031ea0 RDI: ffff88017ce05400
RBP: ffff88011b9dfc00 R08: ffff88011b9dfc5e R09: 0000000000000002
R10: 0000160000000000 R11: ffff880000000000 R12: ffff88011b9dfc5e
R13: 0000000000000002 R14: 0000000000000576 R15: ffff88017ce05400
FS:  00007fa7b34cf8c0(0000) GS:ffff88021e240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000f774a000 CR3: 000000013050c000 CR4: 00000000001406e0
Stack:
 ffff88011b9dfc6f ffff88013570fe30 ffff88011b9dfbe8 ffff8801507e4358
 ffff88011b9dfbf8 ffffffff81271fb0 0000000000000000 ffff8801fb21f000
 0000000000000000 ffff88011b9dfc08 ffffffff8127b04a ffff88013570fe30
Call Trace:
 [<ffffffff81271fb0>] ? btrfs_get_token_32+0x7a/0xcd
 [<ffffffff8127b04a>] ? memcmp_extent_buffer+0xce/0xea
 [<ffffffff81242029>] btrfs_compare_trees+0x3ec/0x4fd
 [<ffffffff812b6680>] ? process_extent+0xdaa/0xdaa
 [<ffffffff812b77e8>] btrfs_ioctl_send+0x8a9/0xd78
 [<ffffffff8128ad28>] btrfs_ioctl+0x191/0x2630
 [<ffffffff8108462f>] ? update_load_avg+0x21a/0x241
 [<ffffffff8108462f>] ? update_load_avg+0x21a/0x241
 [<ffffffff810840bf>] ? select_task_rq_fair+0x333/0x5db
 [<ffffffff81085a19>] ? check_preempt_wakeup+0x116/0x1c3
 [<ffffffff8118c452>] do_vfs_ioctl+0x3a1/0x414
 [<ffffffff810dad4a>] ? __audit_syscall_entry+0xc0/0xe4
 [<ffffffff811940ff>] ? __fget+0x2f/0x69
 [<ffffffff8118c51c>] SyS_ioctl+0x57/0x79
 [<ffffffff816d94b6>] entry_SYSCALL_64_fastpath+0x16/0x75
Code: 8d b8 5c 03 00 00 e8 fa ae ff ff eb 54 80 fa 6c 89 c3 0f 85 78 03 00 00 49 8b 97 18 01 00 00 48 8b 02 49 39 87 20 01 00 00 74 02 <0f> 0b 41 83 bf 34 01 00 00 00 0f 85 00 fe ff ff 41 83 bf 38 01
RIP  [<ffffffff812b6bc9>] changed_cb+0x549/0x8bf
 RSP <ffff88011b9dfb80>
---[ end trace 06dd4eae9dda6399 ]---
Kernel panic - not syncing: Fatal exception


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

On Mon, Jan 25, 2016 at 09:37:29AM +0800, Qu Wenruo wrote:
> 
> 
> Marc MERLIN wrote on 2016/01/23 09:03 -0800:
> >+David, +Qu
> >about
> >1) kernel crash on BUG_ON
> 
> From your code mentioned, and your second kernel warning, it's out
> of memory.
> Such case also happened when I was debugging in-band de-dup patches.
> 
> Things seems that by some method, btrfs used a lot of memory for
> dirty page caches. Maybe metadata pages.
> 
> Normally when such case happens, VFS should trigger a sync to free
> dirty pages, but btrfs seems to either delayed the sync due to
> running trans or the VFS sync is already too late.
> 
> But it's also possible that large leafsize is related to such problem.
> The larger leafsize is, the harder to alloc continuous memory for kmalloc().
> 
> >2) check --repair not giving good clue that
> >"type mismatch with chunk" is unfixable, and whether it can be kind of
> >ignored or whether your FS really needs to be recreated from scratch
> >(many hours of work for me, and probably 2 days of clock time to rebuild
> >and restore from backup)
> 
> For "type mismatch" problem, it's unfixable, but as far as what we
> know, it shouldn't cause big problem.
> 
> If you're using old version btrfsck, then it's possible such error
> is a false alert. Update btrfsck and try again is a good idea.
> 
> Even if it's not a false alert, mail list says it shouldn't cause
> huge problem, only known problems happens is related to scrub.
> And there is already some user reporting balance can fix it,
> although you need to balance all chunks.
> 
> >3) say more about "root 45948 inode 204452 errors 1000, some csum missing",
> >that they aren't being fixed, and whether they're a big deal or not.
> 
> Personally speaking, I didn't consider it as a big problem itself.
> If csum is missing/corrupted, btrfsck --init-csum-tree can rebuild it.
> 
> But the root problem is more important, maybe some extent/csum tree
> blocks are corrupted and failed to find the checksum, or kernel bugs
> causing csum missing.
> 
> Although, from your very first report, a lot of file extents are
> either corrupted or in bad shape.
> Not sure how accurate the original/repaired data is.
> 
> >
> >More generally I'm curious to know if check --repair will sometimes fix
> >more things on a 2nd (or 3rd...) run than on the first one.
> 
> It is possible that further run to fix more problems, considering
> the already complicated fixing codes.
> But I also consider the possibility not very high though.
> 
> I hope I could have your latest btrfsck result to further
> investigate what's wrong (at least what's wrong on disk).
> 
> Thanks,
> Qu
> 
> >
> >Thanks,
> >Marc
> >
> >On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
> >>On Mon, Jan 18, 2016 at 03:21:53AM +0000, Duncan wrote:
> >>>No.  Those are monotonically increasing counts that are never
> >>>automatically reset over the life of the filesystem.  Use ...
> >>>
> >>>btrfs dev stats
> >>>
> >>>... to display them on the commandline (vs. the kernel log at filesystem
> >>>mount), with the -z option to reset them after display, if desired.
> >>
> >>It's a bit counter intuitive that check --repair doesn't reset error counts if it fixed
> >>underlying errors, but maybe there is a good reason :)
> >>
> >>btrfs dev stats -z /dev/mapper/dshelf1
> >>put everything back to 0 as you hinted, I can now watch what's going on
> >>from here on, thanks.
> >>
> >>On Mon, Jan 18, 2016 at 12:45:33PM +0000, Hugo Mills wrote:
> >>>>bad extent [8697338122240, 8697338126336), type mismatch with chunk
> >>>>bad extent [8697338126336, 8697338130432), type mismatch with chunk
> >>>>bad extent [8697338130432, 8697338134528), type mismatch with chunk
> >>>
> >>>    This is, I think, a symptom of an FS created with a broken
> >>>mkfs.btrfs, and it needs to be re-created. Take a look for that error
> >>>message in the mailing list archives -- there's been a few posts about
> >>>it in the last couple of months.
> >>
> >>Thanks for that other hint.
> >>It was created quite a while ago (1y+), and ran ok until I had an
> >>unexpected crash.
> >>If I have to rebuild it, I will, but that will take 2 days+ due to the
> >>size and getting the backup back (well also, I'm not home for a week, so
> >>restoring backups remotely isn't going to be fun).
> >>
> >>But sure enough, while it ran perfectly fine for a long time, after check --repair,
> >>my machine is now crashing every hour or so, with the crash below.
> >>
> >>I had to have an older kernel due to a separate problem where PMP (sata
> >>port multiplier) support was broken in more recent kernels.
> >>I've just put 4.3.3 on it to see if crashes still, but either way, I thought
> >>btrfs had gotten rid of its last BUG_ON(xxx). System should never crash
> >>due to unexpected filesystem state on a non system partition.
> >>
> >>Mmmh, scratch this, here's the BUG_ON, it's a memory problem :(
> >>fs/btrfs/ctree.c:5200
> >>		/* save our key for returning back */
> >>		btrfs_node_key_to_cpu(cur, &found_key, slot);
> >>		path->slots[level] = slot;
> >>		if (level == path->lowest_level) {
> >>			ret = 0;
> >>			goto out;
> >>		}
> >>		btrfs_set_path_blocking(path);
> >>		cur = read_node_slot(root, cur, slot);
> >>		BUG_ON(!cur); /* -ENOMEM */
> >>
> >>Did check --repair potentially change my FS in a way that is now making the
> >>kernel take a lot of RAM and crash?
> >>
> >>
> >>------------[ cut here ]------------
> >>kernel BUG at fs/btrfs/ctree.c:5200!
> >>invalid opcode: 0000 [#1] SMP
> >>CPU: 3 PID: 29041 Comm: btrfs Tainted: G        W       4.2.5-amd64-i915-volpreempt-20150421 #4
> >>Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> >>task: ffff88002f8482c0 ti: ffff8800c9058000 task.ti: ffff8800c9058000
> >>RIP: 0010:[<ffffffff81234001>]  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
> >>RSP: 0018:ffff8800c905bbc8  EFLAGS: 00010246
> >>RAX: 0000000000000000 RBX: ffff8801494409b0 RCX: 000000000003c6c8
> >>RDX: 0000000000000000 RSI: 000000000aa60000 RDI: fffffffffffffffb
> >>RBP: ffff8800c905bc38 R08: 0000000001c00000 R09: 0000000000000000
> >>R10: 0000000000aaaaaa R11: ffffffff818799e0 R12: 0000000000000000
> >>R13: 0000000000000001 R14: 0000000000000000 R15: ffff8801494409b4
> >>FS:  00007fa9d46848c0(0000) GS:ffff88021e2c0000(0000) knlGS:0000000000000000
> >>CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>CR2: 00000000f77cd000 CR3: 0000000143780000 CR4: 00000000001406e0
> >>Stack:
> >>  ffff8800c9bb6050 0001880149440a18 ffff8800c905bc87 ffff8800c939a800
> >>  0000000000000003 1a0000000000002e 84000000000000b3 000000000003c59e
> >>  000000000000002c ffff8801494409b0 0000000000000000 ffff8800c905bce0
> >>Call Trace:
> >>  [<ffffffff81278ca5>] search_ioctl+0xfc/0x167
> >>  [<ffffffff81278d6b>] btrfs_ioctl_tree_search+0x5b/0x8e
> >>  [<ffffffff8127c6af>] btrfs_ioctl+0x396/0x232f
> >>  [<ffffffff81123a2d>] ? get_page+0xe/0x28
> >>  [<ffffffff81123e68>] ? __lru_cache_add+0x23/0x44
> >>  [<ffffffff81124139>] ? lru_cache_add_active_or_unevictable+0x2d/0x6b
> >>  [<ffffffff8113a2df>] ? set_pte_at+0x9/0xd
> >>  [<ffffffff8113e21d>] ? handle_mm_fault+0x90e/0xe9b
> >>  [<ffffffff8100d02b>] ? paravirt_write_msr+0xf/0x13
> >>  [<ffffffff811814af>] do_vfs_ioctl+0x39b/0x412
> >>  [<ffffffff810ace61>] ? current_kernel_time+0xe/0x32
> >>  [<ffffffff810d4d5c>] ? __audit_syscall_entry+0xbe/0xe0
> >>  [<ffffffff81181580>] SyS_ioctl+0x5a/0x7f
> >>  [<ffffffff816b0032>] entry_SYSCALL_64_fastpath+0x16/0x75
> >>Code: 89 47 40 44 3b ab 84 00 00 00 74 69 48 89 df e8 f7 a3 ff ff 8b 55 b8 48 8b 7d a8 4c 89 e6 e8 d8 86 ff ff 48 85 c0
> >><0f> 0b 48 89 c7 e8 23 a7 04 00 41 8d 45 ff 41 c7 47 5c 02 00 00
> >>RIP  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
> >>  RSP <ffff8800c905bbc8>
> >>---[ end trace e3c37adcaa703f63 ]---
> >>Kernel panic - not syncing: Fatal exception
> >>Kernel Offset: disabled
> >>drm_kms_helper: panic occurred, switching back to text console
> >>
> >>
> >>
> >>
> >>--
> >>"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> >>Microsoft is to operating systems ....
> >>                                       .... what McDonalds is to gourmet cooking
> >>Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> >>--
> >>To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> >>the body of a message to majordomo@vger.kernel.org
> >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >
> >On Wed, Jan 20, 2016 at 08:52:39PM -0800, Marc MERLIN wrote:
> >>On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
> >>>Thanks for that other hint.
> >>>It was created quite a while ago (1y+), and ran ok until I had an
> >>>unexpected crash.
> >>>If I have to rebuild it, I will, but that will take 2 days+ due to the
> >>>size and getting the backup back (well also, I'm not home for a week, so
> >>>restoring backups remotely isn't going to be fun).
> >>>
> >>>But sure enough, while it ran perfectly fine for a long time, after check --repair,
> >>>my machine is now crashing every hour or so, with the crash below.
> >>
> >>I ran check --repair a few more times on it but it does not seem to converge.
> >>The bad extent stuff, I understand cannot be fixed (although it's not obvious from check
> >>--repair that they are not getting fixed).
> >>
> >>But how about
> >>root 45940 inode 204450 errors 1000, some csum missing
> >>Are those getting fixed by check --repair or are they unfixable too?
> >>
> >>(...)
> >>bad extent [8697338126336, 8697338130432), type mismatch with chunk
> >>bad extent [8697338130432, 8697338134528), type mismatch with chunk
> >>repaired damaged extent references
> >>
> >>Fixed 0 roots.
> >>cache and super generation don't match, space cache will be invalidated
> >>rootk262 inodeo204450 errors 1000, some csum missing
> >>root 262 inode 204452 errors 1000, some csum missing
> >>(...)
> >>root 45940 inode 204450 errors 1000, some csum missing
> >>root 45940 inode 204452 errors 1000, some csum missing
> >>root 45944 inode 204450 errors 1000, some csum missing
> >>root 45944 inode 204452 errors 1000, some csum missing
> >>root 45948 inode 204450 errors 1000, some csum missing
> >>root 45948 inode 204452 errors 1000, some csum missing
> >>checking fs roots [o]
> >>checking csums
> >>checking root refs
> >>found 9826295305149 bytes used err is 0
> >>total csum bytes: 9584154948
> >>total tree bytes: 12201304064
> >>total fs tree bytes: 330776576
> >>total extent tree bytes: 498921472
> >>btree space waste bytes: 1373183541
> >>file data blocks allocated: 9953275256832
> >>  referenced 9964596801536
> >>btrfs-progs v4.3
> >>
> >>
> >>Thanks,
> >>Marc
> >>--
> >>"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> >>Microsoft is to operating systems ....
> >>                                       .... what McDonalds is to gourmet cooking
> >>Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> >>--
> >>To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> >>the body of a message to majordomo@vger.kernel.org
> >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >
> >
> 
> 
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.4.0: btrfs-send BUG_ON(sctx->cur_ino != sctx->cmp_key->objectid);

[Filipe Manana]

On Mon, Jan 25, 2016 at 3:55 PM, Marc MERLIN <marc@merlins.org> wrote:
> I still have 2 more days before I can rebuild my broken filesystem.
> In the meantime, I just got this new error with 4.4

Nop, not new in 4.4. I have seen 1 report of someone hitting this with
a 4.0 kernel in the past. Not a problem with send afaics but some
inconsistent state achieved likely after adding/modifying/deleting a
xattr.

Nothing new, just send the output of btrfs-debug-tree -t <parent
snapshot id> and for the send snapshot too. Also in the function that
triggered the BUG_ON(), add a printk like the following right before
the BUG_ON() line:

if (sctx->cur_ino != sctx->cmp_key->objectid)
    printk(KERN_ERR "sctx->cur_ino = %llu, sctx->cmp_key->objectid =
%llu\n", sctx->cmp_key->objectid, sctx->cmp_key->objectid);

And see the result in dmesg/syslog.

thanks

> Actually I'm assuming it happened on the broken filesystem, but maybe
> not, it may have happened on my other non broken FS that was also doing
> btrfs send.
>
>
> static int changed_xattr(struct send_ctx *sctx,
>                          enum btrfs_compare_tree_result result)
> {
>         int ret = 0;
>
>         BUG_ON(sctx->cur_ino != sctx->cmp_key->objectid);
>
>
>
> BTRFS error (device dm-1): bdev /dev/mapper/dshelf1 errs: wr 279, rd 0, flush 0, corrupt 0, gen 0
> ------------[ cut here ]------------
> kernel BUG at fs/btrfs/send.c:5639!
> invalid opcode: 0000 [#1] SMP
> Modules linked in: veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp xt_addrtype bridge tun stp llc autofs4 softdog binfmt_misc ftdi_sio nfsd nfs_acl auth_rpcgss nfs fscache lockd grace sunrpc ipt_REJECT nf_reject_ipv4 xt_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl2303 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables nf_conntrack_ftp ipt_MASQUERADE x_tables nf_nat_masquerade_ipv4 nf_nat nf_conntrack sg st snd_pcm_oss snd_mixer_oss snd_opl3_synth snd_hda_codec_realtek snd_seq_midi_emul snd_cmipci snd_hda_codec_generic gameport snd_opl3_lib snd_hda_intel kvm_intel snd_mpu401_uart snd_seq_midi snd_hda_codec kvm snd_seq_midi_event snd_hda_core snd_seq irqbypass snd_hwdep snd_rawmidi snd_pcm snd_seq_device snd_timer tpm_infineon tpm_tis rc_ati_x10 eeepc_wmi coretemp tpm 8250_fintek asus_wmi battery intel_rapl asix pcspkr sparse_keymap libphy snd iosf_mbi ati_remote xhci_pci processor lpc_ich rfkill intel_powerclamp evdev i2c_i801 soundcore parport_pc x86_pkg_temp_thermal wmi rc_core usbnet input_leds xhci_hcd parport usbserial e1000e ptp pps_core fuse raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common ecb xts ansi_cprng drbg aesni_intel ablk_helper lrw gf128mul glue_helper aes_x86_64 crc32c_intel crct10dif_pclmul ehci_pci crc32_pclmul fjes ehci_hcd sata_sil24 thermal sata_mv r8169 cryptd mii usbcore fan usb_common [last unloaded: ftdi_sio]
> CPU: 1 PID: 8229 Comm: btrfs Not tainted 4.4.0-amd64-i915-volpreempt-20150421 #1
> Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> task: ffff880114376300 ti: ffff88011b9dc000 task.ti: ffff88011b9dc000
> RIP: 0010:[<ffffffff812b6bc9>]  [<ffffffff812b6bc9>] changed_cb+0x549/0x8bf
> RSP: 0018:ffff88011b9dfb80  EFLAGS: 00210293
> RAX: 0000000000031ea2 RBX: 0000000000000000 RCX: 0000000000031e60
> RDX: ffff88011b9dfc5e RSI: 0000000000031ea0 RDI: ffff88017ce05400
> RBP: ffff88011b9dfc00 R08: ffff88011b9dfc5e R09: 0000000000000002
> R10: 0000160000000000 R11: ffff880000000000 R12: ffff88011b9dfc5e
> R13: 0000000000000002 R14: 0000000000000576 R15: ffff88017ce05400
> FS:  00007fa7b34cf8c0(0000) GS:ffff88021e240000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000f774a000 CR3: 000000013050c000 CR4: 00000000001406e0
> Stack:
>  ffff88011b9dfc6f ffff88013570fe30 ffff88011b9dfbe8 ffff8801507e4358
>  ffff88011b9dfbf8 ffffffff81271fb0 0000000000000000 ffff8801fb21f000
>  0000000000000000 ffff88011b9dfc08 ffffffff8127b04a ffff88013570fe30
> Call Trace:
>  [<ffffffff81271fb0>] ? btrfs_get_token_32+0x7a/0xcd
>  [<ffffffff8127b04a>] ? memcmp_extent_buffer+0xce/0xea
>  [<ffffffff81242029>] btrfs_compare_trees+0x3ec/0x4fd
>  [<ffffffff812b6680>] ? process_extent+0xdaa/0xdaa
>  [<ffffffff812b77e8>] btrfs_ioctl_send+0x8a9/0xd78
>  [<ffffffff8128ad28>] btrfs_ioctl+0x191/0x2630
>  [<ffffffff8108462f>] ? update_load_avg+0x21a/0x241
>  [<ffffffff8108462f>] ? update_load_avg+0x21a/0x241
>  [<ffffffff810840bf>] ? select_task_rq_fair+0x333/0x5db
>  [<ffffffff81085a19>] ? check_preempt_wakeup+0x116/0x1c3
>  [<ffffffff8118c452>] do_vfs_ioctl+0x3a1/0x414
>  [<ffffffff810dad4a>] ? __audit_syscall_entry+0xc0/0xe4
>  [<ffffffff811940ff>] ? __fget+0x2f/0x69
>  [<ffffffff8118c51c>] SyS_ioctl+0x57/0x79
>  [<ffffffff816d94b6>] entry_SYSCALL_64_fastpath+0x16/0x75
> Code: 8d b8 5c 03 00 00 e8 fa ae ff ff eb 54 80 fa 6c 89 c3 0f 85 78 03 00 00 49 8b 97 18 01 00 00 48 8b 02 49 39 87 20 01 00 00 74 02 <0f> 0b 41 83 bf 34 01 00 00 00 0f 85 00 fe ff ff 41 83 bf 38 01
> RIP  [<ffffffff812b6bc9>] changed_cb+0x549/0x8bf
>  RSP <ffff88011b9dfb80>
> ---[ end trace 06dd4eae9dda6399 ]---
> Kernel panic - not syncing: Fatal exception
>
>
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>
> On Mon, Jan 25, 2016 at 09:37:29AM +0800, Qu Wenruo wrote:
>>
>>
>> Marc MERLIN wrote on 2016/01/23 09:03 -0800:
>> >+David, +Qu
>> >about
>> >1) kernel crash on BUG_ON
>>
>> From your code mentioned, and your second kernel warning, it's out
>> of memory.
>> Such case also happened when I was debugging in-band de-dup patches.
>>
>> Things seems that by some method, btrfs used a lot of memory for
>> dirty page caches. Maybe metadata pages.
>>
>> Normally when such case happens, VFS should trigger a sync to free
>> dirty pages, but btrfs seems to either delayed the sync due to
>> running trans or the VFS sync is already too late.
>>
>> But it's also possible that large leafsize is related to such problem.
>> The larger leafsize is, the harder to alloc continuous memory for kmalloc().
>>
>> >2) check --repair not giving good clue that
>> >"type mismatch with chunk" is unfixable, and whether it can be kind of
>> >ignored or whether your FS really needs to be recreated from scratch
>> >(many hours of work for me, and probably 2 days of clock time to rebuild
>> >and restore from backup)
>>
>> For "type mismatch" problem, it's unfixable, but as far as what we
>> know, it shouldn't cause big problem.
>>
>> If you're using old version btrfsck, then it's possible such error
>> is a false alert. Update btrfsck and try again is a good idea.
>>
>> Even if it's not a false alert, mail list says it shouldn't cause
>> huge problem, only known problems happens is related to scrub.
>> And there is already some user reporting balance can fix it,
>> although you need to balance all chunks.
>>
>> >3) say more about "root 45948 inode 204452 errors 1000, some csum missing",
>> >that they aren't being fixed, and whether they're a big deal or not.
>>
>> Personally speaking, I didn't consider it as a big problem itself.
>> If csum is missing/corrupted, btrfsck --init-csum-tree can rebuild it.
>>
>> But the root problem is more important, maybe some extent/csum tree
>> blocks are corrupted and failed to find the checksum, or kernel bugs
>> causing csum missing.
>>
>> Although, from your very first report, a lot of file extents are
>> either corrupted or in bad shape.
>> Not sure how accurate the original/repaired data is.
>>
>> >
>> >More generally I'm curious to know if check --repair will sometimes fix
>> >more things on a 2nd (or 3rd...) run than on the first one.
>>
>> It is possible that further run to fix more problems, considering
>> the already complicated fixing codes.
>> But I also consider the possibility not very high though.
>>
>> I hope I could have your latest btrfsck result to further
>> investigate what's wrong (at least what's wrong on disk).
>>
>> Thanks,
>> Qu
>>
>> >
>> >Thanks,
>> >Marc
>> >
>> >On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
>> >>On Mon, Jan 18, 2016 at 03:21:53AM +0000, Duncan wrote:
>> >>>No.  Those are monotonically increasing counts that are never
>> >>>automatically reset over the life of the filesystem.  Use ...
>> >>>
>> >>>btrfs dev stats
>> >>>
>> >>>... to display them on the commandline (vs. the kernel log at filesystem
>> >>>mount), with the -z option to reset them after display, if desired.
>> >>
>> >>It's a bit counter intuitive that check --repair doesn't reset error counts if it fixed
>> >>underlying errors, but maybe there is a good reason :)
>> >>
>> >>btrfs dev stats -z /dev/mapper/dshelf1
>> >>put everything back to 0 as you hinted, I can now watch what's going on
>> >>from here on, thanks.
>> >>
>> >>On Mon, Jan 18, 2016 at 12:45:33PM +0000, Hugo Mills wrote:
>> >>>>bad extent [8697338122240, 8697338126336), type mismatch with chunk
>> >>>>bad extent [8697338126336, 8697338130432), type mismatch with chunk
>> >>>>bad extent [8697338130432, 8697338134528), type mismatch with chunk
>> >>>
>> >>>    This is, I think, a symptom of an FS created with a broken
>> >>>mkfs.btrfs, and it needs to be re-created. Take a look for that error
>> >>>message in the mailing list archives -- there's been a few posts about
>> >>>it in the last couple of months.
>> >>
>> >>Thanks for that other hint.
>> >>It was created quite a while ago (1y+), and ran ok until I had an
>> >>unexpected crash.
>> >>If I have to rebuild it, I will, but that will take 2 days+ due to the
>> >>size and getting the backup back (well also, I'm not home for a week, so
>> >>restoring backups remotely isn't going to be fun).
>> >>
>> >>But sure enough, while it ran perfectly fine for a long time, after check --repair,
>> >>my machine is now crashing every hour or so, with the crash below.
>> >>
>> >>I had to have an older kernel due to a separate problem where PMP (sata
>> >>port multiplier) support was broken in more recent kernels.
>> >>I've just put 4.3.3 on it to see if crashes still, but either way, I thought
>> >>btrfs had gotten rid of its last BUG_ON(xxx). System should never crash
>> >>due to unexpected filesystem state on a non system partition.
>> >>
>> >>Mmmh, scratch this, here's the BUG_ON, it's a memory problem :(
>> >>fs/btrfs/ctree.c:5200
>> >>            /* save our key for returning back */
>> >>            btrfs_node_key_to_cpu(cur, &found_key, slot);
>> >>            path->slots[level] = slot;
>> >>            if (level == path->lowest_level) {
>> >>                    ret = 0;
>> >>                    goto out;
>> >>            }
>> >>            btrfs_set_path_blocking(path);
>> >>            cur = read_node_slot(root, cur, slot);
>> >>            BUG_ON(!cur); /* -ENOMEM */
>> >>
>> >>Did check --repair potentially change my FS in a way that is now making the
>> >>kernel take a lot of RAM and crash?
>> >>
>> >>
>> >>------------[ cut here ]------------
>> >>kernel BUG at fs/btrfs/ctree.c:5200!
>> >>invalid opcode: 0000 [#1] SMP
>> >>CPU: 3 PID: 29041 Comm: btrfs Tainted: G        W       4.2.5-amd64-i915-volpreempt-20150421 #4
>> >>Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
>> >>task: ffff88002f8482c0 ti: ffff8800c9058000 task.ti: ffff8800c9058000
>> >>RIP: 0010:[<ffffffff81234001>]  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
>> >>RSP: 0018:ffff8800c905bbc8  EFLAGS: 00010246
>> >>RAX: 0000000000000000 RBX: ffff8801494409b0 RCX: 000000000003c6c8
>> >>RDX: 0000000000000000 RSI: 000000000aa60000 RDI: fffffffffffffffb
>> >>RBP: ffff8800c905bc38 R08: 0000000001c00000 R09: 0000000000000000
>> >>R10: 0000000000aaaaaa R11: ffffffff818799e0 R12: 0000000000000000
>> >>R13: 0000000000000001 R14: 0000000000000000 R15: ffff8801494409b4
>> >>FS:  00007fa9d46848c0(0000) GS:ffff88021e2c0000(0000) knlGS:0000000000000000
>> >>CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> >>CR2: 00000000f77cd000 CR3: 0000000143780000 CR4: 00000000001406e0
>> >>Stack:
>> >>  ffff8800c9bb6050 0001880149440a18 ffff8800c905bc87 ffff8800c939a800
>> >>  0000000000000003 1a0000000000002e 84000000000000b3 000000000003c59e
>> >>  000000000000002c ffff8801494409b0 0000000000000000 ffff8800c905bce0
>> >>Call Trace:
>> >>  [<ffffffff81278ca5>] search_ioctl+0xfc/0x167
>> >>  [<ffffffff81278d6b>] btrfs_ioctl_tree_search+0x5b/0x8e
>> >>  [<ffffffff8127c6af>] btrfs_ioctl+0x396/0x232f
>> >>  [<ffffffff81123a2d>] ? get_page+0xe/0x28
>> >>  [<ffffffff81123e68>] ? __lru_cache_add+0x23/0x44
>> >>  [<ffffffff81124139>] ? lru_cache_add_active_or_unevictable+0x2d/0x6b
>> >>  [<ffffffff8113a2df>] ? set_pte_at+0x9/0xd
>> >>  [<ffffffff8113e21d>] ? handle_mm_fault+0x90e/0xe9b
>> >>  [<ffffffff8100d02b>] ? paravirt_write_msr+0xf/0x13
>> >>  [<ffffffff811814af>] do_vfs_ioctl+0x39b/0x412
>> >>  [<ffffffff810ace61>] ? current_kernel_time+0xe/0x32
>> >>  [<ffffffff810d4d5c>] ? __audit_syscall_entry+0xbe/0xe0
>> >>  [<ffffffff81181580>] SyS_ioctl+0x5a/0x7f
>> >>  [<ffffffff816b0032>] entry_SYSCALL_64_fastpath+0x16/0x75
>> >>Code: 89 47 40 44 3b ab 84 00 00 00 74 69 48 89 df e8 f7 a3 ff ff 8b 55 b8 48 8b 7d a8 4c 89 e6 e8 d8 86 ff ff 48 85 c0
>> >><0f> 0b 48 89 c7 e8 23 a7 04 00 41 8d 45 ff 41 c7 47 5c 02 00 00
>> >>RIP  [<ffffffff81234001>] btrfs_search_forward+0x1a6/0x24b
>> >>  RSP <ffff8800c905bbc8>
>> >>---[ end trace e3c37adcaa703f63 ]---
>> >>Kernel panic - not syncing: Fatal exception
>> >>Kernel Offset: disabled
>> >>drm_kms_helper: panic occurred, switching back to text console
>> >>
>> >>
>> >>
>> >>
>> >>--
>> >>"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> >>Microsoft is to operating systems ....
>> >>                                       .... what McDonalds is to gourmet cooking
>> >>Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>> >>--
>> >>To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> >>the body of a message to majordomo@vger.kernel.org
>> >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>
>> >
>> >On Wed, Jan 20, 2016 at 08:52:39PM -0800, Marc MERLIN wrote:
>> >>On Mon, Jan 18, 2016 at 03:39:43PM -0800, Marc MERLIN wrote:
>> >>>Thanks for that other hint.
>> >>>It was created quite a while ago (1y+), and ran ok until I had an
>> >>>unexpected crash.
>> >>>If I have to rebuild it, I will, but that will take 2 days+ due to the
>> >>>size and getting the backup back (well also, I'm not home for a week, so
>> >>>restoring backups remotely isn't going to be fun).
>> >>>
>> >>>But sure enough, while it ran perfectly fine for a long time, after check --repair,
>> >>>my machine is now crashing every hour or so, with the crash below.
>> >>
>> >>I ran check --repair a few more times on it but it does not seem to converge.
>> >>The bad extent stuff, I understand cannot be fixed (although it's not obvious from check
>> >>--repair that they are not getting fixed).
>> >>
>> >>But how about
>> >>root 45940 inode 204450 errors 1000, some csum missing
>> >>Are those getting fixed by check --repair or are they unfixable too?
>> >>
>> >>(...)
>> >>bad extent [8697338126336, 8697338130432), type mismatch with chunk
>> >>bad extent [8697338130432, 8697338134528), type mismatch with chunk
>> >>repaired damaged extent references
>> >>
>> >>Fixed 0 roots.
>> >>cache and super generation don't match, space cache will be invalidated
>> >>rootk262 inodeo204450 errors 1000, some csum missing
>> >>root 262 inode 204452 errors 1000, some csum missing
>> >>(...)
>> >>root 45940 inode 204450 errors 1000, some csum missing
>> >>root 45940 inode 204452 errors 1000, some csum missing
>> >>root 45944 inode 204450 errors 1000, some csum missing
>> >>root 45944 inode 204452 errors 1000, some csum missing
>> >>root 45948 inode 204450 errors 1000, some csum missing
>> >>root 45948 inode 204452 errors 1000, some csum missing
>> >>checking fs roots [o]
>> >>checking csums
>> >>checking root refs
>> >>found 9826295305149 bytes used err is 0
>> >>total csum bytes: 9584154948
>> >>total tree bytes: 12201304064
>> >>total fs tree bytes: 330776576
>> >>total extent tree bytes: 498921472
>> >>btree space waste bytes: 1373183541
>> >>file data blocks allocated: 9953275256832
>> >>  referenced 9964596801536
>> >>btrfs-progs v4.3
>> >>
>> >>
>> >>Thanks,
>> >>Marc
>> >>--
>> >>"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> >>Microsoft is to operating systems ....
>> >>                                       .... what McDonalds is to gourmet cooking
>> >>Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>> >>--
>> >>To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> >>the body of a message to majordomo@vger.kernel.org
>> >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>
>> >
>> >
>>
>>
>>
>
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: 4.4.0: btrfs-send BUG_ON(sctx->cur_ino != sctx->cmp_key->objectid);

[Marc MERLIN]

On Mon, Jan 25, 2016 at 07:46:52PM +0000, Filipe Manana wrote:
> On Mon, Jan 25, 2016 at 3:55 PM, Marc MERLIN <marc@merlins.org> wrote:
> > I still have 2 more days before I can rebuild my broken filesystem.
> > In the meantime, I just got this new error with 4.4
> 
> Nop, not new in 4.4. I have seen 1 report of someone hitting this with
> a 4.0 kernel in the past. Not a problem with send afaics but some
> inconsistent state achieved likely after adding/modifying/deleting a
> xattr.
> 
> Nothing new, just send the output of btrfs-debug-tree -t <parent
> snapshot id> and for the send snapshot too. Also in the function that
> triggered the BUG_ON(), add a printk like the following right before
> the BUG_ON() line:
> 
> if (sctx->cur_ino != sctx->cmp_key->objectid)
>     printk(KERN_ERR "sctx->cur_ino = %llu, sctx->cmp_key->objectid =
> %llu\n", sctx->cmp_key->objectid, sctx->cmp_key->objectid);
> 
> And see the result in dmesg/syslog.

Thanks for the reply.
I may not be able to reproduce this soon or at all because I'm about to
rebuild the damaged filesystem this happened on.

The point is that my filesystem is damaged, but this is not a reason to
crash the kernel and the machine.
Can this be changed to an abort and remount read only instead?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.4.0: btrfs-send BUG_ON(sctx->cur_ino != sctx->cmp_key->objectid);

[Filipe Manana]

On Mon, Jan 25, 2016 at 7:56 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Mon, Jan 25, 2016 at 07:46:52PM +0000, Filipe Manana wrote:
>> On Mon, Jan 25, 2016 at 3:55 PM, Marc MERLIN <marc@merlins.org> wrote:
>> > I still have 2 more days before I can rebuild my broken filesystem.
>> > In the meantime, I just got this new error with 4.4
>>
>> Nop, not new in 4.4. I have seen 1 report of someone hitting this with
>> a 4.0 kernel in the past. Not a problem with send afaics but some
>> inconsistent state achieved likely after adding/modifying/deleting a
>> xattr.
>>
>> Nothing new, just send the output of btrfs-debug-tree -t <parent
>> snapshot id> and for the send snapshot too. Also in the function that
>> triggered the BUG_ON(), add a printk like the following right before
>> the BUG_ON() line:
>>
>> if (sctx->cur_ino != sctx->cmp_key->objectid)
>>     printk(KERN_ERR "sctx->cur_ino = %llu, sctx->cmp_key->objectid =
>> %llu\n", sctx->cmp_key->objectid, sctx->cmp_key->objectid);
>>
>> And see the result in dmesg/syslog.
>
> Thanks for the reply.
> I may not be able to reproduce this soon or at all because I'm about to
> rebuild the damaged filesystem this happened on.
>
> The point is that my filesystem is damaged, but this is not a reason to
> crash the kernel and the machine.

I've seen that happen on a non-damaged filesystem, for which I didn't
get an image nor debug-tree's output before it got recreated. That's
what I want to figure out, how/why it happened.

> Can this be changed to an abort and remount read only instead?

Like many other bug_on's yes.

>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Marc MERLIN]

On Mon, Jan 25, 2016 at 09:37:29AM +0800, Qu Wenruo wrote:
> >+David, +Qu
> >about
> >1) kernel crash on BUG_ON
> 
> From your code mentioned, and your second kernel warning, it's out
> of memory.
> Such case also happened when I was debugging in-band de-dup patches.
 
Right. So it's obviously a bug since it's on a lightly loaded server
with 8GB of RAM, and this only started happeening after my FS started
having problems.

> Things seems that by some method, btrfs used a lot of memory for
> dirty page caches. Maybe metadata pages.
> 
> Normally when such case happens, VFS should trigger a sync to free
> dirty pages, but btrfs seems to either delayed the sync due to
> running trans or the VFS sync is already too late.
 
Oh, I see.

> But it's also possible that large leafsize is related to such problem.
> The larger leafsize is, the harder to alloc continuous memory for kmalloc().

So basically, we seem to understand how we get there, but not quite why,
or how to fix it, correct?

> If you're using old version btrfsck, then it's possible such error
> is a false alert. Update btrfsck and try again is a good idea.
 
I had 4.3 as the latest in debian unstable, but now I see 4.4 just came
out, so I installed it.

> Even if it's not a false alert, mail list says it shouldn't cause
> huge problem, only known problems happens is related to scrub.
> And there is already some user reporting balance can fix it,
> although you need to balance all chunks.

Thanks for that tip.
 
> >3) say more about "root 45948 inode 204452 errors 1000, some csum missing",
> >that they aren't being fixed, and whether they're a big deal or not.
> 
> Personally speaking, I didn't consider it as a big problem itself.
> If csum is missing/corrupted, btrfsck --init-csum-tree can rebuild it.

Any idea why check --repair isn't fixing them too, is that expected?

gargamel:~# btrfs --version
btrfs-progs v4.4
gargamel:~# btrfs check --repair --init-csum-tree -p /dev/mapper/dshelf1 2>&1  | tee check7
Reinit crc root
crc refilling failed   <<<< is that bad?
enabling repair mode
Creating a new CRC tree
Checking filesystem on /dev/mapper/dshelf1
UUID: 6358304a-2234-4243-b02d-4944c9af47d7

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.4.0: btrfs-send BUG_ON(sctx->cur_ino != sctx->cmp_key->objectid);

[Marc MERLIN]

On Mon, Jan 25, 2016 at 08:24:06PM +0000, Filipe Manana wrote:
> > The point is that my filesystem is damaged, but this is not a reason to
> > crash the kernel and the machine.
> 
> I've seen that happen on a non-damaged filesystem, for which I didn't
> get an image nor debug-tree's output before it got recreated. That's
> what I want to figure out, how/why it happened.
 
I'll see what I can do, this is going to take a lot of time on my end,
just before a trip and a system I have to bring back to working order
before I leave.

> > Can this be changed to an abort and remount read only instead?
> 
> Like many other bug_on's yes.

So, would you be able to first add the printk you requested to all
kernels since this info seems required for you to know more?
And then to change the BUG_ON into some kind of abort?
Crashing the kernel causes collateral damage, including getting the logs
you need to debug for many people without serial console :)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: BTRFS: bdev /dev/mapper/dshelf1 errs: wr 2970, rd 848, flush 0, corrupt 189, gen 0

[Qu Wenruo]

Marc MERLIN wrote on 2016/01/25 12:55 -0800:
> On Mon, Jan 25, 2016 at 09:37:29AM +0800, Qu Wenruo wrote:
>>> +David, +Qu
>>> about
>>> 1) kernel crash on BUG_ON
>>
>>  From your code mentioned, and your second kernel warning, it's out
>> of memory.
>> Such case also happened when I was debugging in-band de-dup patches.
>
> Right. So it's obviously a bug since it's on a lightly loaded server
> with 8GB of RAM, and this only started happeening after my FS started
> having problems.
>
>> Things seems that by some method, btrfs used a lot of memory for
>> dirty page caches. Maybe metadata pages.
>>
>> Normally when such case happens, VFS should trigger a sync to free
>> dirty pages, but btrfs seems to either delayed the sync due to
>> running trans or the VFS sync is already too late.
>
> Oh, I see.
>
>> But it's also possible that large leafsize is related to such problem.
>> The larger leafsize is, the harder to alloc continuous memory for kmalloc().
>
> So basically, we seem to understand how we get there, but not quite why,
> or how to fix it, correct?
>
>> If you're using old version btrfsck, then it's possible such error
>> is a false alert. Update btrfsck and try again is a good idea.
>
> I had 4.3 as the latest in debian unstable, but now I see 4.4 just came
> out, so I installed it.
>
>> Even if it's not a false alert, mail list says it shouldn't cause
>> huge problem, only known problems happens is related to scrub.
>> And there is already some user reporting balance can fix it,
>> although you need to balance all chunks.
>
> Thanks for that tip.
>
>>> 3) say more about "root 45948 inode 204452 errors 1000, some csum missing",
>>> that they aren't being fixed, and whether they're a big deal or not.
>>
>> Personally speaking, I didn't consider it as a big problem itself.
>> If csum is missing/corrupted, btrfsck --init-csum-tree can rebuild it.
>
> Any idea why check --repair isn't fixing them too, is that expected?
>
> gargamel:~# btrfs --version
> btrfs-progs v4.4
> gargamel:~# btrfs check --repair --init-csum-tree -p /dev/mapper/dshelf1 2>&1  | tee check7
> Reinit crc root
> crc refilling failed   <<<< is that bad?

Yes, that's pretty bad.
Some csum can't be populated from extent tree.

Although we still have a method to rebuild csum according to fs tree, 
but that's only used in --init-extent-tree.

So I'm afraid that not only csum tree, but extent tree or even fs tree 
is corrupted more or less, and that's the direct cause.

If the fs is small enough, would you please do a btrfs-image dump?
That would help a lot to locate the direct cause.

Thanks,
Qu

> enabling repair mode
> Creating a new CRC tree
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 6358304a-2234-4243-b02d-4944c9af47d7
>
> Thanks,
> Marc
>

Re: btrfs-image failure (btrfs-tools 4.4)

[Marc MERLIN]

On Tue, Jan 26, 2016 at 09:03:07AM +0800, Qu Wenruo wrote:
> If the fs is small enough, would you please do a btrfs-image dump?
> That would help a lot to locate the direct cause.
 
I started making a dump, image was growing past 3GB, and then it failed
and the image got deleted:

gargamel:~# btrfs-image -s -c 9 /dev/mapper/dshelf1old /mnt/dshelf1/ds1old.dump
Error adding space cache blocks -5
Error flushing pending -5
create failed (Success)

gargamel:~# dpkg --status btrfs-tools
Package: btrfs-tools
Status: install ok installed
Priority: optional
Section: admin
Installed-Size: 3605
Maintainer: Dimitri John Ledkov <xnox@debian.org>
Architecture: amd64
Version: 4.4-1

Is there a 4G file size limit, or did I hit another problem?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs-image failure (btrfs-tools 4.4)

[Qu Wenruo]

Marc MERLIN wrote on 2016/02/10 22:31 -0800:
> On Tue, Jan 26, 2016 at 09:03:07AM +0800, Qu Wenruo wrote:
>> If the fs is small enough, would you please do a btrfs-image dump?
>> That would help a lot to locate the direct cause.
>
> I started making a dump, image was growing past 3GB, and then it failed
> and the image got deleted:
>
> gargamel:~# btrfs-image -s -c 9 /dev/mapper/dshelf1old /mnt/dshelf1/ds1old.dump
> Error adding space cache blocks -5

It seems that btrfs-image failed to read space cache, in 
read_data_extent() function.

And since there is no "Couldn't map the block XXXX" error message, 
either some device is missing or pread64 failed to read the desired data.

> Error flushing pending -5
> create failed (Success)
>
> gargamel:~# dpkg --status btrfs-tools
> Package: btrfs-tools
> Status: install ok installed
> Priority: optional
> Section: admin
> Installed-Size: 3605
> Maintainer: Dimitri John Ledkov <xnox@debian.org>
> Architecture: amd64
> Version: 4.4-1
>
> Is there a 4G file size limit, or did I hit another problem?

For the 4G file size limit, did you mean the limit from old filesystem 
like FAT32?

I didn't think there is such limit for modern Linux filesystem, or 
normal read/write operation won't has such limit either.

Thanks,
Qu

>
> Thanks,
> Marc
>

Re: btrfs-image failure (btrfs-tools 4.4)

[Marc MERLIN]

On Thu, Feb 11, 2016 at 03:16:47PM +0800, Qu Wenruo wrote:
> >I started making a dump, image was growing past 3GB, and then it failed
> >and the image got deleted:
> >
> >gargamel:~# btrfs-image -s -c 9 /dev/mapper/dshelf1old /mnt/dshelf1/ds1old.dump
> >Error adding space cache blocks -5
> 
> It seems that btrfs-image failed to read space cache, in
> read_data_extent() function.
> 
> And since there is no "Couldn't map the block XXXX" error message,
> either some device is missing or pread64 failed to read the desired
> data.
 
It's a 5 drive raid5 underneath, all drives are there.

> >Is there a 4G file size limit, or did I hit another problem?
> 
> For the 4G file size limit, did you mean the limit from old
> filesystem like FAT32?
 
No, I wrote on btrfs, so it's not a filesystem limit, but I meant that
maybe if there was a 32bit pointer somewhere, it could have caused this.
I did use the 64bit version of the tools on a 64bit kernel though, so I
don't see why it could have happened.

> I didn't think there is such limit for modern Linux filesystem, or
> normal read/write operation won't has such limit either.

Agreed.

At this point, is there anything else I should get/do before I wipe this
filesystem?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs-image failure (btrfs-tools 4.4)

[Marc MERLIN]

On Thu, Feb 11, 2016 at 07:09:47AM -0800, Marc MERLIN wrote:
> On Thu, Feb 11, 2016 at 03:16:47PM +0800, Qu Wenruo wrote:
> > >I started making a dump, image was growing past 3GB, and then it failed
> > >and the image got deleted:
> > >
> > >gargamel:~# btrfs-image -s -c 9 /dev/mapper/dshelf1old /mnt/dshelf1/ds1old.dump
> > >Error adding space cache blocks -5
> > 
> > It seems that btrfs-image failed to read space cache, in
> > read_data_extent() function.
> > 
> > And since there is no "Couldn't map the block XXXX" error message,
> > either some device is missing or pread64 failed to read the desired
> > data.
>  
> It's a 5 drive raid5 underneath, all drives are there.
> 
> > >Is there a 4G file size limit, or did I hit another problem?
> > 
> > For the 4G file size limit, did you mean the limit from old
> > filesystem like FAT32?
>  
> No, I wrote on btrfs, so it's not a filesystem limit, but I meant that
> maybe if there was a 32bit pointer somewhere, it could have caused this.
> I did use the 64bit version of the tools on a 64bit kernel though, so I
> don't see why it could have happened.
> 
> > I didn't think there is such limit for modern Linux filesystem, or
> > normal read/write operation won't has such limit either.
> 
> Agreed.
> 
> At this point, is there anything else I should get/do before I wipe this
> filesystem?
 
I should note that I can mount it fine and read/write to it.
I also ran fsck check --repair on it more than once.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs-image failure (btrfs-tools 4.4)

[Qu Wenruo]

Marc MERLIN wrote on 2016/02/11 07:13 -0800:
> On Thu, Feb 11, 2016 at 07:09:47AM -0800, Marc MERLIN wrote:
>> On Thu, Feb 11, 2016 at 03:16:47PM +0800, Qu Wenruo wrote:
>>>> I started making a dump, image was growing past 3GB, and then it failed
>>>> and the image got deleted:
>>>>
>>>> gargamel:~# btrfs-image -s -c 9 /dev/mapper/dshelf1old /mnt/dshelf1/ds1old.dump
>>>> Error adding space cache blocks -5
>>>
>>> It seems that btrfs-image failed to read space cache, in
>>> read_data_extent() function.
>>>
>>> And since there is no "Couldn't map the block XXXX" error message,
>>> either some device is missing or pread64 failed to read the desired
>>> data.
>>
>> It's a 5 drive raid5 underneath, all drives are there.
>>
>>>> Is there a 4G file size limit, or did I hit another problem?
>>>
>>> For the 4G file size limit, did you mean the limit from old
>>> filesystem like FAT32?
>>
>> No, I wrote on btrfs, so it's not a filesystem limit, but I meant that
>> maybe if there was a 32bit pointer somewhere, it could have caused this.
>> I did use the 64bit version of the tools on a 64bit kernel though, so I
>> don't see why it could have happened.
>>
>>> I didn't think there is such limit for modern Linux filesystem, or
>>> normal read/write operation won't has such limit either.
>>
>> Agreed.
>>
>> At this point, is there anything else I should get/do before I wipe this
>> filesystem?

There is still a last chance.

If btrfsck still report original error about "bad file extent" in root: 
  45851/45852/...
Btrfs-debug-tree may provide useful info by dumping only that root.

# btrfs-debug-tree -t 45851

But the problem is, there is no filename fuzz option.
You need to mask all the filenames in INODE_REF/DIR_ITEM/DIR_INDEX by 
script, or just grep the affected inode info following the pattern "key 
(<INODE_NUM>".

At least this should tell us what's the problem and we can check 
manually to determine if it's fixable.


And I just remember that, if you only need to get a clean fs without 
fsck warning, trying removing all affected subvolumes is possible idea.

Thanks,
Qu
>
> I should note that I can mount it fine and read/write to it.
> I also ran fsck check --repair on it more than once.
>
> Marc
>

Re: btrfs-image failure (btrfs-tools 4.4)

[Marc MERLIN]

On Fri, Feb 12, 2016 at 08:33:11AM +0800, Qu Wenruo wrote:
> There is still a last chance.
> 
> If btrfsck still report original error about "bad file extent" in root: 
>  45851/45852/...
> Btrfs-debug-tree may provide useful info by dumping only that root.
> 
> # btrfs-debug-tree -t 45851
> 
> But the problem is, there is no filename fuzz option.
> You need to mask all the filenames in INODE_REF/DIR_ITEM/DIR_INDEX by 
> script, or just grep the affected inode info following the pattern "key 
> (<INODE_NUM>".
> 
> At least this should tell us what's the problem and we can check 
> manually to determine if it's fixable.

Mmmh, so the fsck is looking a bit worse now, here is the output:
http://marc.merlins.org/tmp/ggm-broken-ds1-fsck.txt

I'm not super sure what inode I should use for debug tree. Can you suggest
one?

I ran the dump 
gargamel:~# btrfs-image -s -c 9 /dev/mapper/dshelf1old
/mnt/dshelf1/ds1old.dump
Error adding space cache blocks -5
Error flushing pending -5
create failed (Success)

and a du every so often showed the file go to 9.3GB before btrfs-image
deleted it:
9.3G    /mnt/dshelf1/ds1old.dump

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: btrfs-image failure (btrfs-tools 4.4)

[Marc MERLIN]

On Fri, Feb 12, 2016 at 09:26:28AM -0800, Marc MERLIN wrote:
> On Fri, Feb 12, 2016 at 08:33:11AM +0800, Qu Wenruo wrote:
> > There is still a last chance.
> > 
> > If btrfsck still report original error about "bad file extent" in root: 
> >  45851/45852/...
> > Btrfs-debug-tree may provide useful info by dumping only that root.
> > 
> > # btrfs-debug-tree -t 45851
> > 
> > But the problem is, there is no filename fuzz option.
> > You need to mask all the filenames in INODE_REF/DIR_ITEM/DIR_INDEX by 
> > script, or just grep the affected inode info following the pattern "key 
> > (<INODE_NUM>".
> > 
> > At least this should tell us what's the problem and we can check 
> > manually to determine if it's fixable.
> 
> Mmmh, so the fsck is looking a bit worse now, here is the output:
> http://marc.merlins.org/tmp/ggm-broken-ds1-fsck.txt
> 
> I'm not super sure what inode I should use for debug tree. Can you suggest
> one?
> 
> I ran the dump 
> gargamel:~# btrfs-image -s -c 9 /dev/mapper/dshelf1old
> /mnt/dshelf1/ds1old.dump
> Error adding space cache blocks -5
> Error flushing pending -5
> create failed (Success)
> 
> and a du every so often showed the file go to 9.3GB before btrfs-image
> deleted it:
> 9.3G    /mnt/dshelf1/ds1old.dump

So I'm at a point where I need to delete this filesystem. I believe it'll be
monday morning soon on your side of the world :) 
Can you let me know if there is anything else I can/should get before I wipe
it?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: btrfs-image failure (btrfs-tools 4.4)

[Qu Wenruo]

Marc MERLIN wrote on 2016/02/14 09:26 -0800:
> On Fri, Feb 12, 2016 at 09:26:28AM -0800, Marc MERLIN wrote:
>> On Fri, Feb 12, 2016 at 08:33:11AM +0800, Qu Wenruo wrote:
>>> There is still a last chance.
>>>
>>> If btrfsck still report original error about "bad file extent" in root:
>>>   45851/45852/...
>>> Btrfs-debug-tree may provide useful info by dumping only that root.
>>>
>>> # btrfs-debug-tree -t 45851
>>>
>>> But the problem is, there is no filename fuzz option.
>>> You need to mask all the filenames in INODE_REF/DIR_ITEM/DIR_INDEX by
>>> script, or just grep the affected inode info following the pattern "key
>>> (<INODE_NUM>".
>>>
>>> At least this should tell us what's the problem and we can check
>>> manually to determine if it's fixable.
>>
>> Mmmh, so the fsck is looking a bit worse now, here is the output:
>> http://marc.merlins.org/tmp/ggm-broken-ds1-fsck.txt
>>
>> I'm not super sure what inode I should use for debug tree. Can you suggest
>> one?
>>
>> I ran the dump
>> gargamel:~# btrfs-image -s -c 9 /dev/mapper/dshelf1old
>> /mnt/dshelf1/ds1old.dump
>> Error adding space cache blocks -5
>> Error flushing pending -5
>> create failed (Success)
>>
>> and a du every so often showed the file go to 9.3GB before btrfs-image
>> deleted it:
>> 9.3G    /mnt/dshelf1/ds1old.dump
>
> So I'm at a point where I need to delete this filesystem. I believe it'll be
> monday morning soon on your side of the world :)
> Can you let me know if there is anything else I can/should get before I wipe
> it?
>
> Thanks,
> Marc
>
Sorry for the late reply.

According to the output of your btrfsck --repair output, things just get 
worse.

So at this point, it's harder to locate the original problem. (Csum 
missing with bad file extents)

IMHO you can wipe the fs now.

Thanks,
Qu

Re: btrfs-image failure (btrfs-tools 4.4)

[Marc MERLIN]

On Mon, Feb 15, 2016 at 08:17:32AM +0800, Qu Wenruo wrote:
> Sorry for the late reply.
> 
> According to the output of your btrfsck --repair output, things just get 
> worse.
> 
> So at this point, it's harder to locate the original problem. (Csum 
> missing with bad file extents)
> 
> IMHO you can wipe the fs now.

Sounds good. I've just done so then. Thanks for your replies.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/