3.11.4: kernel BUG at fs/buffer.c:1268

3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

This is a newly built machine (although out of "tested" parts), so RAM
problems are not unthinkable, but I had the chance to capture this so
it seemed worth reporting.

i7-2xxx CPU, 8GB RAM, file system is ext4 on RAID-1.
The local patches are to a char device driver (remote control/rf
subsystem) that isn't even active ATM.

The BUG, BTW, is
static inline void check_irqs_on(void)
{
#ifdef irqs_disabled
        BUG_ON(irqs_disabled());
#endif
}

I'm not sure which config options are most important.
One that comes to mind is CONFIG_PREEMPT_VOLUNTARY=y


[88395.501925] ------------[ cut here ]------------
[88395.501952] kernel BUG at fs/buffer.c:1268!
[88395.501970] invalid opcode: 0000 [#1] SMP 
[88395.501992] Modules linked in: battery nfsd exportfs fuse ftdi_sio usbserial r8169 aesni_intel aes_x86_64 ablk_helper cryptd iTCO_wdt lrw gf128mul glue_helper mii
[88395.502089] CPU: 0 PID: 4971 Comm: iceweasel Not tainted 3.11.4-00008-g9838365 #97
[88395.502125] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[88395.502168] task: ffff880210b62080 ti: ffff8802014cc000 task.ti: ffff8802014cc000
[88395.502194] RIP: 0010:[<ffffffff810e115a>]  [<ffffffff810e115a>] check_irqs_on+0xb/0xf
[88395.502226] RSP: 0018:ffff8802014cd6e0  EFLAGS: 00210046
[88395.502245] RAX: 0000000000200086 RBX: 0000000000001000 RCX: ffff8802146e8000
[88395.502269] RDX: 0000000000001000 RSI: 0000000000d00206 RDI: ffff8802165789c0
[88395.502293] RBP: ffff8802014cd6e0 R08: 00000000000001a3 R09: 0000000000000003
[88395.502317] R10: 0000000000000003 R11: ffff88020b265ae0 R12: ffff8802165789c0
[88395.502341] R13: 0000000000d00206 R14: ffff88020092c920 R15: ffff880216ace400
[88395.502365] FS:  0000000000000000(0000) GS:ffff88021fa00000(0000) knlGS:0000000000000000
[88395.502393] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[88395.502413] CR2: 0000000000000000 CR3: 0000000001614000 CR4: 00000000000407f0
[88395.502436] Stack:
[88395.502444]  ffff8802014cd750 ffffffff810e136d ffffea000305e900 ffff8802014cd7a8
[88395.502473]  0000000000200292 ffff8802014cd788 ffff8802014cd720 ffffffff811af692
[88395.502501]  ffff8802014cd750 ffffffff8108aeba 0000000000000010 0000000000001000
[88395.502530] Call Trace:
[88395.502541]  [<ffffffff810e136d>] __find_get_block+0x1c/0x176
[88395.502563]  [<ffffffff811af692>] ? radix_tree_lookup_slot+0xe/0x10
[88395.502586]  [<ffffffff8108aeba>] ? find_get_page+0x41/0x63
[88395.502606]  [<ffffffff810e24cd>] __getblk+0x20/0x27e
[88395.502625]  [<ffffffff8111411d>] __ext4_get_inode_loc+0xf5/0x32f
[88395.502646]  [<ffffffff81115ba7>] ext4_get_inode_loc+0x29/0x2e
[88395.502667]  [<ffffffff81117347>] ext4_reserve_inode_write+0x1f/0x7a
[88395.502690]  [<ffffffff811173d8>] ext4_mark_inode_dirty+0x36/0x19b
[88395.502713]  [<ffffffff8113f773>] ? jbd2_journal_dirty_metadata+0x1b5/0x1f0
[88395.502737]  [<ffffffff81128f99>] __ext4_ext_dirty+0x5a/0x63
[88395.502758]  [<ffffffff8112a67b>] ext4_ext_insert_extent+0xd8f/0xdcf
[88395.502780]  [<ffffffff8112c9ab>] ext4_ext_map_blocks+0xc68/0xe01
[88395.502802]  [<ffffffff81115607>] ext4_map_blocks+0x27b/0x42b
[88395.502823]  [<ffffffff811178f5>] ext4_writepages+0x3b8/0x814
[88395.502844]  [<ffffffff81436b02>] ? _raw_spin_lock+0x9/0xb
[88395.502865]  [<ffffffff81092550>] do_writepages+0x19/0x27
[88395.502884]  [<ffffffff8108baf1>] __filemap_fdatawrite_range+0x50/0x52
[88395.502907]  [<ffffffff8108bb0a>] filemap_flush+0x17/0x19
[88395.502926]  [<ffffffff81115a21>] ext4_alloc_da_blocks+0x21/0x23
[88395.502947]  [<ffffffff81110c0b>] ext4_release_file+0x20/0x95
[88395.502968]  [<ffffffff810c14cd>] __fput+0xf2/0x1cb
[88395.502985]  [<ffffffff810c15d2>] ____fput+0x9/0xb
[88395.503003]  [<ffffffff81041d3a>] task_work_run+0x78/0x8e
[88395.503023]  [<ffffffff8102ea67>] do_exit+0x378/0x841
[88395.503042]  [<ffffffff81036712>] ? __sigqueue_free+0x34/0x37
[88395.503062]  [<ffffffff81036b15>] ? __dequeue_signal+0xa8/0xfd
[88395.503083]  [<ffffffff8102fa32>] do_group_exit+0x3f/0x95
[88395.503103]  [<ffffffff81038d53>] get_signal_to_deliver+0x423/0x443
[88395.503125]  [<ffffffff81001cf0>] do_signal+0x44/0x5c3
[88395.503144]  [<ffffffff81037d59>] ? do_send_sig_info+0x58/0x6d
[88395.503165]  [<ffffffff81002294>] do_notify_resume+0x25/0x58
[88395.503185]  [<ffffffff814376e0>] int_signal+0x12/0x17
[88395.503203] Code: 80 4d 00 20 4d 8b 6d 08 48 ff c3 4c 3b 6d d0 75 b7 5a 4c 89 e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 55 48 89 e5 9c 58 f6 c4 02 75 02 <0f> 0b 5d c3 55 48 81 fa ff 0f 00 00 48 89 e5 48 89 77 10 76 02 
[88395.503336] RIP  [<ffffffff810e115a>] check_irqs_on+0xb/0xf
[88395.503356]  RSP <ffff8802014cd6e0>
[88395.511861] ---[ end trace 2480df9f92ab983b ]---
[88395.511862] Fixing recursive fault but reboot is needed!

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Jan Kara]

On Wed 09-10-13 07:55:02, George Spelvin wrote:
> This is a newly built machine (although out of "tested" parts), so RAM
> problems are not unthinkable, but I had the chance to capture this so
> it seemed worth reporting.
> 
> i7-2xxx CPU, 8GB RAM, file system is ext4 on RAID-1.
> The local patches are to a char device driver (remote control/rf
> subsystem) that isn't even active ATM.
> 
> The BUG, BTW, is
> static inline void check_irqs_on(void)
> {
> #ifdef irqs_disabled
>         BUG_ON(irqs_disabled());
> #endif
> }
> 
> I'm not sure which config options are most important.
> One that comes to mind is CONFIG_PREEMPT_VOLUNTARY=y
  This is really weird. We are delivering a signal to a task. While task is
returning from kernel space we are running queued task works and one of
that works is dropping last file reference. Ext4 then does some data
flushing and at that point we find out irqs are disabled. It isn't really
clear to me where in that call chain got irqs disabled. I went through it
and didn't find any such place... If this is reproducible, there would be
ways to debug this (like irq tracing). Otherwise I'm not sure... I'm CCing
Al since he was digging in this code recently. Maybe he will have some
idea.

								Honza 
 
 
> [88395.501925] ------------[ cut here ]------------
> [88395.501952] kernel BUG at fs/buffer.c:1268!
> [88395.501970] invalid opcode: 0000 [#1] SMP 
> [88395.501992] Modules linked in: battery nfsd exportfs fuse ftdi_sio usbserial r8169 aesni_intel aes_x86_64 ablk_helper cryptd iTCO_wdt lrw gf128mul glue_helper mii
> [88395.502089] CPU: 0 PID: 4971 Comm: iceweasel Not tainted 3.11.4-00008-g9838365 #97
> [88395.502125] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
> [88395.502168] task: ffff880210b62080 ti: ffff8802014cc000 task.ti: ffff8802014cc000
> [88395.502194] RIP: 0010:[<ffffffff810e115a>]  [<ffffffff810e115a>] check_irqs_on+0xb/0xf
> [88395.502226] RSP: 0018:ffff8802014cd6e0  EFLAGS: 00210046
> [88395.502245] RAX: 0000000000200086 RBX: 0000000000001000 RCX: ffff8802146e8000
> [88395.502269] RDX: 0000000000001000 RSI: 0000000000d00206 RDI: ffff8802165789c0
> [88395.502293] RBP: ffff8802014cd6e0 R08: 00000000000001a3 R09: 0000000000000003
> [88395.502317] R10: 0000000000000003 R11: ffff88020b265ae0 R12: ffff8802165789c0
> [88395.502341] R13: 0000000000d00206 R14: ffff88020092c920 R15: ffff880216ace400
> [88395.502365] FS:  0000000000000000(0000) GS:ffff88021fa00000(0000) knlGS:0000000000000000
> [88395.502393] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
> [88395.502413] CR2: 0000000000000000 CR3: 0000000001614000 CR4: 00000000000407f0
> [88395.502436] Stack:
> [88395.502444]  ffff8802014cd750 ffffffff810e136d ffffea000305e900 ffff8802014cd7a8
> [88395.502473]  0000000000200292 ffff8802014cd788 ffff8802014cd720 ffffffff811af692
> [88395.502501]  ffff8802014cd750 ffffffff8108aeba 0000000000000010 0000000000001000
> [88395.502530] Call Trace:
> [88395.502541]  [<ffffffff810e136d>] __find_get_block+0x1c/0x176
> [88395.502563]  [<ffffffff811af692>] ? radix_tree_lookup_slot+0xe/0x10
> [88395.502586]  [<ffffffff8108aeba>] ? find_get_page+0x41/0x63
> [88395.502606]  [<ffffffff810e24cd>] __getblk+0x20/0x27e
> [88395.502625]  [<ffffffff8111411d>] __ext4_get_inode_loc+0xf5/0x32f
> [88395.502646]  [<ffffffff81115ba7>] ext4_get_inode_loc+0x29/0x2e
> [88395.502667]  [<ffffffff81117347>] ext4_reserve_inode_write+0x1f/0x7a
> [88395.502690]  [<ffffffff811173d8>] ext4_mark_inode_dirty+0x36/0x19b
> [88395.502713]  [<ffffffff8113f773>] ? jbd2_journal_dirty_metadata+0x1b5/0x1f0
> [88395.502737]  [<ffffffff81128f99>] __ext4_ext_dirty+0x5a/0x63
> [88395.502758]  [<ffffffff8112a67b>] ext4_ext_insert_extent+0xd8f/0xdcf
> [88395.502780]  [<ffffffff8112c9ab>] ext4_ext_map_blocks+0xc68/0xe01
> [88395.502802]  [<ffffffff81115607>] ext4_map_blocks+0x27b/0x42b
> [88395.502823]  [<ffffffff811178f5>] ext4_writepages+0x3b8/0x814
> [88395.502844]  [<ffffffff81436b02>] ? _raw_spin_lock+0x9/0xb
> [88395.502865]  [<ffffffff81092550>] do_writepages+0x19/0x27
> [88395.502884]  [<ffffffff8108baf1>] __filemap_fdatawrite_range+0x50/0x52
> [88395.502907]  [<ffffffff8108bb0a>] filemap_flush+0x17/0x19
> [88395.502926]  [<ffffffff81115a21>] ext4_alloc_da_blocks+0x21/0x23
> [88395.502947]  [<ffffffff81110c0b>] ext4_release_file+0x20/0x95
> [88395.502968]  [<ffffffff810c14cd>] __fput+0xf2/0x1cb
> [88395.502985]  [<ffffffff810c15d2>] ____fput+0x9/0xb
> [88395.503003]  [<ffffffff81041d3a>] task_work_run+0x78/0x8e
> [88395.503023]  [<ffffffff8102ea67>] do_exit+0x378/0x841
> [88395.503042]  [<ffffffff81036712>] ? __sigqueue_free+0x34/0x37
> [88395.503062]  [<ffffffff81036b15>] ? __dequeue_signal+0xa8/0xfd
> [88395.503083]  [<ffffffff8102fa32>] do_group_exit+0x3f/0x95
> [88395.503103]  [<ffffffff81038d53>] get_signal_to_deliver+0x423/0x443
> [88395.503125]  [<ffffffff81001cf0>] do_signal+0x44/0x5c3
> [88395.503144]  [<ffffffff81037d59>] ? do_send_sig_info+0x58/0x6d
> [88395.503165]  [<ffffffff81002294>] do_notify_resume+0x25/0x58
> [88395.503185]  [<ffffffff814376e0>] int_signal+0x12/0x17
> [88395.503203] Code: 80 4d 00 20 4d 8b 6d 08 48 ff c3 4c 3b 6d d0 75 b7 5a 4c 89 e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 55 48 89 e5 9c 58 f6 c4 02 75 02 <0f> 0b 5d c3 55 48 81 fa ff 0f 00 00 48 89 e5 48 89 77 10 76 02 
> [88395.503336] RIP  [<ffffffff810e115a>] check_irqs_on+0xb/0xf
> [88395.503356]  RSP <ffff8802014cd6e0>
> [88395.511861] ---[ end trace 2480df9f92ab983b ]---
> [88395.511862] Fixing recursive fault but reboot is needed!
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Al Viro]

On Wed, Oct 09, 2013 at 05:18:53PM +0200, Jan Kara wrote:

>   This is really weird. We are delivering a signal to a task. While task is

ITYM "a fatal signal"

> returning from kernel space we are running queued task works and one of

get_signal_to_deliver() notices that the signal has to be dealt with
via default reaction, which happens to be "die, you bastard".  So it
calls do_group_exit().  Which means that we'll never be returning to userland,
so the time to run pending __fput() is now.

> that works is dropping last file reference. Ext4 then does some data
> flushing and at that point we find out irqs are disabled. It isn't really
> clear to me where in that call chain got irqs disabled. I went through it
> and didn't find any such place... If this is reproducible, there would be
> ways to debug this (like irq tracing). Otherwise I'm not sure... I'm CCing
> Al since he was digging in this code recently. Maybe he will have some
> idea.

Note that do_group_exit() is preceded by
                spin_unlock_irq(&sighand->siglock);
so no matter what happened in callers, irq is enabled.  I'd suggest sticking
such BUG_ON() into __fput() and trying to reproduce that crap...

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

Al Viro wrote:
> Note that do_group_exit() is preceded by
>                spin_unlock_irq(&sighand->siglock);
> so no matter what happened in callers, irq is enabled.  I'd suggest sticking
> such BUG_ON() into __fput() and trying to reproduce that crap...

Well, it happened again (error appended).  Can you please clarify what you mean
by "such BUG_ON()"; I'm having a hard time following the RCU code and determining
all the situations under which __fput() might be called.

Given that __fput() includes might_sleep(), how about I enable CONFIG_DEBUG_ATOMIC_SLEEP?

[280344.098552] ------------[ cut here ]------------
[280344.098575] kernel BUG at fs/buffer.c:1268!
[280344.098590] invalid opcode: 0000 [#1] SMP 
[280344.098608] Modules linked in: fuse ftdi_sio usbserial iTCO_wdt
[280344.098635] CPU: 1 PID: 4298 Comm: iceweasel Not tainted 3.11.5-00008-ga1818c5 #98
[280344.098661] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[280344.098692] task: ffff88020abb2080 ti: ffff880204852000 task.ti: ffff880204852000
[280344.098717] RIP: 0010:[<ffffffff810eac61>]  [<ffffffff810eac61>] check_irqs_on+0xb/0xf
[280344.098748] RSP: 0018:ffff880204853788  EFLAGS: 00210046
[280344.098767] RAX: 0000000000200082 RBX: 0000000000001000 RCX: ffff8802145ca800
[280344.098792] RDX: 0000000000001000 RSI: 0000000000b800f2 RDI: ffff8802165a0d00
[280344.098816] RBP: ffff880204853788 R08: 0000000000000171 R09: 0000000000000002
[280344.098840] R10: 0000000000000002 R11: ffff88011e738ff0 R12: ffff8802165a0d00
[280344.098864] R13: 0000000000b800f2 R14: ffff88008da3ef40 R15: ffff880216acbc00
[280344.098889] FS:  0000000000000000(0000) GS:ffff88021fa40000(0000) knlGS:0000000000000000
[280344.098916] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[280344.098936] CR2: 00000000f7609bb0 CR3: 00000002117b6000 CR4: 00000000000407e0
[280344.098959] Stack:
[280344.098967]  ffff8802048537f8 ffffffff810eae74 0000000000000000 0000000000000000
[280344.098996]  0000000000000000 0000000000000000 0000000000000001 0000000000000001
[280344.099025]  0000000000000000 0000000000000000 0000000000000010 0000000000001000
[280344.099054] Call Trace:
[280344.099065]  [<ffffffff810eae74>] __find_get_block+0x1c/0x176
[280344.099086]  [<ffffffff810ebfd4>] __getblk+0x20/0x27e
[280344.099106]  [<ffffffff8109d193>] ? put_page+0x21/0x30
[280344.099124]  [<ffffffff810eaac7>] ? __find_get_block_slow+0x123/0x13a
[280344.099147]  [<ffffffff8111dc57>] __ext4_get_inode_loc+0xf5/0x32f
[280344.099169]  [<ffffffff8111f6e1>] ext4_get_inode_loc+0x29/0x2e
[280344.099190]  [<ffffffff81120e81>] ext4_reserve_inode_write+0x1f/0x7a
[280344.099212]  [<ffffffff81120f12>] ext4_mark_inode_dirty+0x36/0x19b
[280344.099234]  [<ffffffff81122dda>] ext4_dirty_inode+0x3b/0x54
[280344.099254]  [<ffffffff810e5e88>] __mark_inode_dirty+0x2d/0x196
[280344.099275]  [<ffffffff8113d430>] ext4_free_blocks+0x636/0x6dd
[280344.099296]  [<ffffffff8113528b>] ext4_ext_remove_space+0x568/0xa53
[280344.099319]  [<ffffffff81142002>] ? ext4_es_free_extent+0x52/0x55
[280344.099340]  [<ffffffff81142699>] ? __es_remove_extent+0x1fb/0x2a1
[280344.099362]  [<ffffffff81136702>] ext4_ext_truncate+0x84/0xa8
[280344.099382]  [<ffffffff81121d2b>] ext4_truncate+0x187/0x21c
[280344.099402]  [<ffffffff811223d0>] ext4_evict_inode+0x1ab/0x2a0
[280344.099423]  [<ffffffff810dc927>] evict+0xa2/0x151
[280344.099440]  [<ffffffff810dcf0a>] iput+0x121/0x12a
[280344.099458]  [<ffffffff810d96f6>] dentry_kill+0x109/0x123
[280344.099478]  [<ffffffff810d97e3>] dput+0xd3/0xe2
[280344.099495]  [<ffffffff810cb097>] __fput+0x1b5/0x1cb
[280344.099513]  [<ffffffff810cb0d9>] ____fput+0x9/0xb
[280344.099531]  [<ffffffff8104b841>] task_work_run+0x78/0x8e
[280344.099551]  [<ffffffff81038557>] do_exit+0x378/0x841
[280344.099569]  [<ffffffff81040202>] ? __sigqueue_free+0x34/0x37
[280344.099590]  [<ffffffff81040605>] ? __dequeue_signal+0xa8/0xfd
[280344.099610]  [<ffffffff81039522>] do_group_exit+0x3f/0x95
[280344.099630]  [<ffffffff81042843>] get_signal_to_deliver+0x423/0x443
[280344.099652]  [<ffffffff81001cf0>] do_signal+0x44/0x5c3
[280344.099671]  [<ffffffff81041849>] ? do_send_sig_info+0x58/0x6d
[280344.099691]  [<ffffffff81002294>] do_notify_resume+0x25/0x58
[280344.099712]  [<ffffffff814478a0>] int_signal+0x12/0x17
[280344.099729] Code: 80 4d 00 20 4d 8b 6d 08 48 ff c3 4c 3b 6d d0 75 b7 5a 4c 89 e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 55 48 89 e5 9c 58 f6 c4 02 75 02 <0f> 0b 5d c3 55 48 81 fa ff 0f 00 00 48 89 e5 48 89 77 10 76 02 
[280344.099862] RIP  [<ffffffff810eac61>] check_irqs_on+0xb/0xf
[280344.099883]  RSP <ffff880204853788>
[280344.108415] ---[ end trace 68f14daef5901df3 ]---
[280344.108416] Fixing recursive fault but reboot is needed!

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Jan Kara]

On Thu 17-10-13 17:11:43, George Spelvin wrote:
> Al Viro wrote:
> > Note that do_group_exit() is preceded by
> >                spin_unlock_irq(&sighand->siglock);
> > so no matter what happened in callers, irq is enabled.  I'd suggest sticking
> > such BUG_ON() into __fput() and trying to reproduce that crap...
> 
> Well, it happened again (error appended).  Can you please clarify what you mean
> by "such BUG_ON()"; I'm having a hard time following the RCU code and determining
> all the situations under which __fput() might be called.
> 
> Given that __fput() includes might_sleep(), how about I enable
> CONFIG_DEBUG_ATOMIC_SLEEP?
  Yes, that should work as well.

									Honza
> 
> [280344.098552] ------------[ cut here ]------------
> [280344.098575] kernel BUG at fs/buffer.c:1268!
> [280344.098590] invalid opcode: 0000 [#1] SMP 
> [280344.098608] Modules linked in: fuse ftdi_sio usbserial iTCO_wdt
> [280344.098635] CPU: 1 PID: 4298 Comm: iceweasel Not tainted 3.11.5-00008-ga1818c5 #98
> [280344.098661] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
> [280344.098692] task: ffff88020abb2080 ti: ffff880204852000 task.ti: ffff880204852000
> [280344.098717] RIP: 0010:[<ffffffff810eac61>]  [<ffffffff810eac61>] check_irqs_on+0xb/0xf
> [280344.098748] RSP: 0018:ffff880204853788  EFLAGS: 00210046
> [280344.098767] RAX: 0000000000200082 RBX: 0000000000001000 RCX: ffff8802145ca800
> [280344.098792] RDX: 0000000000001000 RSI: 0000000000b800f2 RDI: ffff8802165a0d00
> [280344.098816] RBP: ffff880204853788 R08: 0000000000000171 R09: 0000000000000002
> [280344.098840] R10: 0000000000000002 R11: ffff88011e738ff0 R12: ffff8802165a0d00
> [280344.098864] R13: 0000000000b800f2 R14: ffff88008da3ef40 R15: ffff880216acbc00
> [280344.098889] FS:  0000000000000000(0000) GS:ffff88021fa40000(0000) knlGS:0000000000000000
> [280344.098916] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
> [280344.098936] CR2: 00000000f7609bb0 CR3: 00000002117b6000 CR4: 00000000000407e0
> [280344.098959] Stack:
> [280344.098967]  ffff8802048537f8 ffffffff810eae74 0000000000000000 0000000000000000
> [280344.098996]  0000000000000000 0000000000000000 0000000000000001 0000000000000001
> [280344.099025]  0000000000000000 0000000000000000 0000000000000010 0000000000001000
> [280344.099054] Call Trace:
> [280344.099065]  [<ffffffff810eae74>] __find_get_block+0x1c/0x176
> [280344.099086]  [<ffffffff810ebfd4>] __getblk+0x20/0x27e
> [280344.099106]  [<ffffffff8109d193>] ? put_page+0x21/0x30
> [280344.099124]  [<ffffffff810eaac7>] ? __find_get_block_slow+0x123/0x13a
> [280344.099147]  [<ffffffff8111dc57>] __ext4_get_inode_loc+0xf5/0x32f
> [280344.099169]  [<ffffffff8111f6e1>] ext4_get_inode_loc+0x29/0x2e
> [280344.099190]  [<ffffffff81120e81>] ext4_reserve_inode_write+0x1f/0x7a
> [280344.099212]  [<ffffffff81120f12>] ext4_mark_inode_dirty+0x36/0x19b
> [280344.099234]  [<ffffffff81122dda>] ext4_dirty_inode+0x3b/0x54
> [280344.099254]  [<ffffffff810e5e88>] __mark_inode_dirty+0x2d/0x196
> [280344.099275]  [<ffffffff8113d430>] ext4_free_blocks+0x636/0x6dd
> [280344.099296]  [<ffffffff8113528b>] ext4_ext_remove_space+0x568/0xa53
> [280344.099319]  [<ffffffff81142002>] ? ext4_es_free_extent+0x52/0x55
> [280344.099340]  [<ffffffff81142699>] ? __es_remove_extent+0x1fb/0x2a1
> [280344.099362]  [<ffffffff81136702>] ext4_ext_truncate+0x84/0xa8
> [280344.099382]  [<ffffffff81121d2b>] ext4_truncate+0x187/0x21c
> [280344.099402]  [<ffffffff811223d0>] ext4_evict_inode+0x1ab/0x2a0
> [280344.099423]  [<ffffffff810dc927>] evict+0xa2/0x151
> [280344.099440]  [<ffffffff810dcf0a>] iput+0x121/0x12a
> [280344.099458]  [<ffffffff810d96f6>] dentry_kill+0x109/0x123
> [280344.099478]  [<ffffffff810d97e3>] dput+0xd3/0xe2
> [280344.099495]  [<ffffffff810cb097>] __fput+0x1b5/0x1cb
> [280344.099513]  [<ffffffff810cb0d9>] ____fput+0x9/0xb
> [280344.099531]  [<ffffffff8104b841>] task_work_run+0x78/0x8e
> [280344.099551]  [<ffffffff81038557>] do_exit+0x378/0x841
> [280344.099569]  [<ffffffff81040202>] ? __sigqueue_free+0x34/0x37
> [280344.099590]  [<ffffffff81040605>] ? __dequeue_signal+0xa8/0xfd
> [280344.099610]  [<ffffffff81039522>] do_group_exit+0x3f/0x95
> [280344.099630]  [<ffffffff81042843>] get_signal_to_deliver+0x423/0x443
> [280344.099652]  [<ffffffff81001cf0>] do_signal+0x44/0x5c3
> [280344.099671]  [<ffffffff81041849>] ? do_send_sig_info+0x58/0x6d
> [280344.099691]  [<ffffffff81002294>] do_notify_resume+0x25/0x58
> [280344.099712]  [<ffffffff814478a0>] int_signal+0x12/0x17
> [280344.099729] Code: 80 4d 00 20 4d 8b 6d 08 48 ff c3 4c 3b 6d d0 75 b7 5a 4c 89 e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 55 48 89 e5 9c 58 f6 c4 02 75 02 <0f> 0b 5d c3 55 48 81 fa ff 0f 00 00 48 89 e5 48 89 77 10 76 02 
> [280344.099862] RIP  [<ffffffff810eac61>] check_irqs_on+0xb/0xf
> [280344.099883]  RSP <ffff880204853788>
> [280344.108415] ---[ end trace 68f14daef5901df3 ]---
> [280344.108416] Fixing recursive fault but reboot is needed!
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Al Viro]

On Thu, Oct 17, 2013 at 05:11:43PM -0400, George Spelvin wrote:
> Al Viro wrote:
> > Note that do_group_exit() is preceded by
> >                spin_unlock_irq(&sighand->siglock);
> > so no matter what happened in callers, irq is enabled.  I'd suggest sticking
> > such BUG_ON() into __fput() and trying to reproduce that crap...
> 
> Well, it happened again (error appended).  Can you please clarify what you mean
> by "such BUG_ON()"; I'm having a hard time following the RCU code and determining
> all the situations under which __fput() might be called.

__fput() can be called via task_work_run() or via schedule_work().  That's
all.  And it certainly should never be called with interrupts disabled.
So stick BUG_ON(irqs_disabled()) in it (WARN_ON() might be better, but
not by much).

There are two ways these traces could've happened:
	* exit_task_work() called by do_exit() with irqs disabled.
Definitely buggy (and would do really nasty things to several functions
called by do_exit() before that one).  If such BUG_ON() triggers in
__fput(), this is what happens and the next step will be sticking
several BUG_ON(irqs_disabled()) in do_exit() - something like
	BUG_ON(irqs_disabled());
        exit_sem(tsk);
	BUG_ON(irqs_disabled());
        exit_shm(tsk);
	BUG_ON(irqs_disabled());
        exit_files(tsk);
	BUG_ON(irqs_disabled());
        exit_fs(tsk);
	BUG_ON(irqs_disabled());
        exit_task_namespaces(tsk);
	BUG_ON(irqs_disabled());
        exit_task_work(tsk);
and checking which of those triggers; that'll tell us who has disabled it
and forgot to enable.
	* __fput() is called with irqs enabled, but somewhere on the
way into ext4 (dput -> iput -> evict inode -> free blocks, now that
unlinked file got closed -> ...) we manage to disable irqs and forget
to enable them.

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

Sorry for the long delay between updates, but it took a while to
re-trigger the bug.  It seems to be caused by iceweasel crashing due to
some OOM condition.

Anyway, here's the stack dump with CONFIG_DEBUG_ATOMIC_SLEEP enabled.
(x = 1166866 seconds of uptime.)

[x.908244] BUG: sleeping function called from invalid context at fs/ext4/ext4_jbd2.c:45
[x.908248] in_atomic(): 0, irqs_disabled(): 1, pid: 15216, name: iceweasel
[x.908250] CPU: 6 PID: 15216 Comm: iceweasel Not tainted 3.11.5-00008-ga1818c5 #99
[x.908252] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[x.908253]  0000000000000002 ffff88010d249908 ffffffff81561d7f ffff88021549a000
[x.908255]  ffff88010d249918 ffffffff81069d2f ffff88010d249930 ffffffff8119079b
[x.908257]  ffff88021549a000 ffff88010d249968 ffffffff81190871 ffff8800cc7b8c20
[x.908259] Call Trace:
[x.908265]  [<ffffffff81561d7f>] dump_stack+0x54/0x74
[x.908268]  [<ffffffff81069d2f>] __might_sleep+0xcf/0xf0
[x.908271]  [<ffffffff8119079b>] ext4_journal_check_start+0x1b/0xa0
[x.908273]  [<ffffffff81190871>] __ext4_journal_start_sb+0x21/0x80
[x.908276]  [<ffffffff81177795>] ext4_dirty_inode+0x25/0x60
[x.908280]  [<ffffffff811296ed>] __mark_inode_dirty+0x2d/0x230
[x.908283]  [<ffffffff811992bc>] ext4_free_blocks+0x73c/0xa30
[x.908285]  [<ffffffff8118d936>] ext4_ext_remove_space+0x806/0xe20
[x.908287]  [<ffffffff8119fb14>] ? ext4_es_free_extent+0x54/0x60
[x.908289]  [<ffffffff8118fc18>] ext4_ext_truncate+0xb8/0xe0
[x.908291]  [<ffffffff81176065>] ext4_truncate+0x2b5/0x300
[x.908292]  [<ffffffff81176b18>] ext4_evict_inode+0x3f8/0x430
[x.908295]  [<ffffffff8111c69a>] evict+0xba/0x1c0
[x.908297]  [<ffffffff8111d04b>] iput+0x10b/0x1b0
[x.908298]  [<ffffffff81118e38>] dput+0x278/0x350
[x.908301]  [<ffffffff81104d0a>] __fput+0x16a/0x240
[x.908303]  [<ffffffff81104e19>] ____fput+0x9/0x10
[x.908306]  [<ffffffff8105e30c>] task_work_run+0x9c/0xd0
[x.908309]  [<ffffffff810451f7>] do_exit+0x2a7/0x9d0
[x.908311]  [<ffffffff8104f8ce>] ? __sigqueue_free.part.13+0x2e/0x40
[x.908312]  [<ffffffff8104679e>] do_group_exit+0x3e/0xb0
[x.908315]  [<ffffffff81052740>] get_signal_to_deliver+0x1b0/0x5f0
[x.908317]  [<ffffffff81002133>] do_signal+0x43/0x940
[x.908319]  [<ffffffff81051698>] ? do_send_sig_info+0x58/0x80
[x.908320]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
[x.908323]  [<ffffffff81569ca0>] int_signal+0x12/0x17
[x.908329] ------------[ cut here ]------------
[x.908352] kernel BUG at fs/buffer.c:1268!
[x.908370] invalid opcode: 0000 [#1] SMP 
[x.908391] Modules linked in: pl2303 fuse ftdi_sio usbserial iTCO_wdt
[x.908425] CPU: 6 PID: 15216 Comm: iceweasel Not tainted 3.11.5-00008-ga1818c5 #99
[x.908460] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[x.908484] task: ffff8801124ae800 ti: ffff88010d248000 task.ti: ffff88010d248000
[x.908504] RIP: 0010:[<ffffffff815612a6>]  [<ffffffff815612a6>] check_irqs_on.part.19+0x4/0x6
[x.908529] RSP: 0018:ffff88010d249798  EFLAGS: 00210046
[x.908543] RAX: 0000000000200082 RBX: ffff88010d249928 RCX: ffff880215a5c000
[x.908562] RDX: 0000000000001000 RSI: 000000000038005b RDI: ffff8802164296c0
[x.908580] RBP: ffff88010d249798 R08: 0000000000000000 R09: 0000000000000000
[x.908599] R10: ffff880215a5c000 R11: ffff88010d24947e R12: ffff8802164296c0
[x.908617] R13: 0000000000001000 R14: ffff88021fbdbe00 R15: ffff88021549a000
[x.908635] FS:  0000000000000000(0000) GS:ffff88021fb80000(0000) knlGS:0000000000000000
[x.908656] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[x.908672] CR2: 0000000000000000 CR3: 000000000180c000 CR4: 00000000000407e0
[x.908690] Stack:
[x.908696]  ffff88010d249808 ffffffff8112feb7 000000000000b760 ffff88010d2498f0
[x.908720]  ffff88021fb83fc0 ffff88010d2498e8 ffff88010d249840 ffffffff81004b7f
[x.908743]  ffff88010d24980c 000000000d2498f0 ffff88010d249928 ffff8802164296c0
[x.908766] Call Trace:
[x.908776]  [<ffffffff8112feb7>] __find_get_block+0x1d7/0x1e0
[x.908793]  [<ffffffff81004b7f>] ? dump_trace+0x17f/0x2d0
[x.908808]  [<ffffffff8112fee0>] __getblk+0x20/0x2f0
[x.908823]  [<ffffffff81171516>] __ext4_get_inode_loc+0x106/0x410
[x.908840]  [<ffffffff81004d7f>] ? show_stack_log_lvl+0xaf/0x1a0
[x.908857]  [<ffffffff811734b8>] ext4_get_inode_loc+0x18/0x20
[x.908874]  [<ffffffff81174c61>] ext4_reserve_inode_write+0x21/0x90
[x.908891]  [<ffffffff81561d7f>] ? dump_stack+0x54/0x74
[x.908906]  [<ffffffff81174d19>] ext4_mark_inode_dirty+0x49/0x1a0
[x.908924]  [<ffffffff811777ab>] ext4_dirty_inode+0x3b/0x60
[x.908940]  [<ffffffff811296ed>] __mark_inode_dirty+0x2d/0x230
[x.908957]  [<ffffffff811992bc>] ext4_free_blocks+0x73c/0xa30
[x.908974]  [<ffffffff8118d936>] ext4_ext_remove_space+0x806/0xe20
[x.908991]  [<ffffffff8119fb14>] ? ext4_es_free_extent+0x54/0x60
[x.909008]  [<ffffffff8118fc18>] ext4_ext_truncate+0xb8/0xe0
[x.909025]  [<ffffffff81176065>] ext4_truncate+0x2b5/0x300
[x.909041]  [<ffffffff81176b18>] ext4_evict_inode+0x3f8/0x430
[x.909057]  [<ffffffff8111c69a>] evict+0xba/0x1c0
[x.909071]  [<ffffffff8111d04b>] iput+0x10b/0x1b0
[x.909084]  [<ffffffff81118e38>] dput+0x278/0x350
[x.909099]  [<ffffffff81104d0a>] __fput+0x16a/0x240
[x.909113]  [<ffffffff81104e19>] ____fput+0x9/0x10
[x.909127]  [<ffffffff8105e30c>] task_work_run+0x9c/0xd0
[x.909143]  [<ffffffff810451f7>] do_exit+0x2a7/0x9d0
[x.909157]  [<ffffffff8104f8ce>] ? __sigqueue_free.part.13+0x2e/0x40
[x.909175]  [<ffffffff8104679e>] do_group_exit+0x3e/0xb0
[x.909190]  [<ffffffff81052740>] get_signal_to_deliver+0x1b0/0x5f0
[x.909207]  [<ffffffff81002133>] do_signal+0x43/0x940
[x.909222]  [<ffffffff81051698>] ? do_send_sig_info+0x58/0x80
[x.909238]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
[x.909254]  [<ffffffff81569ca0>] int_signal+0x12/0x17
[x.909267] Code: 4d 85 e4 74 1d 41 80 44 24 58 01 65 48 8b 04 25 b0 b7 00 00 ff 88 44 e0 ff ff 4c 89 e7 e8 23 79 bb ff 5b 41 5c 5d c3 55 48 89 e5 <0f> 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 
[x.909390] RIP  [<ffffffff815612a6>] check_irqs_on.part.19+0x4/0x6
[x.909408]  RSP <ffff88010d249798>
[x.915643] ---[ end trace 379d96cb0444fcb3 ]---
[x.915645] Fixing recursive fault but reboot is needed!

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Jan Kara]

  Hello,

On Thu 31-10-13 05:58:16, George Spelvin wrote:
> Sorry for the long delay between updates, but it took a while to
> re-trigger the bug.  It seems to be caused by iceweasel crashing due to
> some OOM condition.
> 
> Anyway, here's the stack dump with CONFIG_DEBUG_ATOMIC_SLEEP enabled.
> (x = 1166866 seconds of uptime.)
  Thanks!

> [x.908244] BUG: sleeping function called from invalid context at fs/ext4/ext4_jbd2.c:45
> [x.908248] in_atomic(): 0, irqs_disabled(): 1, pid: 15216, name: iceweasel
> [x.908250] CPU: 6 PID: 15216 Comm: iceweasel Not tainted 3.11.5-00008-ga1818c5 #99
> [x.908252] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
> [x.908253]  0000000000000002 ffff88010d249908 ffffffff81561d7f ffff88021549a000
> [x.908255]  ffff88010d249918 ffffffff81069d2f ffff88010d249930 ffffffff8119079b
> [x.908257]  ffff88021549a000 ffff88010d249968 ffffffff81190871 ffff8800cc7b8c20
> [x.908259] Call Trace:
> [x.908265]  [<ffffffff81561d7f>] dump_stack+0x54/0x74
> [x.908268]  [<ffffffff81069d2f>] __might_sleep+0xcf/0xf0
> [x.908271]  [<ffffffff8119079b>] ext4_journal_check_start+0x1b/0xa0
> [x.908273]  [<ffffffff81190871>] __ext4_journal_start_sb+0x21/0x80
> [x.908276]  [<ffffffff81177795>] ext4_dirty_inode+0x25/0x60
> [x.908280]  [<ffffffff811296ed>] __mark_inode_dirty+0x2d/0x230
> [x.908283]  [<ffffffff811992bc>] ext4_free_blocks+0x73c/0xa30
> [x.908285]  [<ffffffff8118d936>] ext4_ext_remove_space+0x806/0xe20
> [x.908287]  [<ffffffff8119fb14>] ? ext4_es_free_extent+0x54/0x60
> [x.908289]  [<ffffffff8118fc18>] ext4_ext_truncate+0xb8/0xe0
> [x.908291]  [<ffffffff81176065>] ext4_truncate+0x2b5/0x300
> [x.908292]  [<ffffffff81176b18>] ext4_evict_inode+0x3f8/0x430
> [x.908295]  [<ffffffff8111c69a>] evict+0xba/0x1c0
> [x.908297]  [<ffffffff8111d04b>] iput+0x10b/0x1b0
> [x.908298]  [<ffffffff81118e38>] dput+0x278/0x350
> [x.908301]  [<ffffffff81104d0a>] __fput+0x16a/0x240
> [x.908303]  [<ffffffff81104e19>] ____fput+0x9/0x10
> [x.908306]  [<ffffffff8105e30c>] task_work_run+0x9c/0xd0
> [x.908309]  [<ffffffff810451f7>] do_exit+0x2a7/0x9d0
> [x.908311]  [<ffffffff8104f8ce>] ? __sigqueue_free.part.13+0x2e/0x40
> [x.908312]  [<ffffffff8104679e>] do_group_exit+0x3e/0xb0
> [x.908315]  [<ffffffff81052740>] get_signal_to_deliver+0x1b0/0x5f0
> [x.908317]  [<ffffffff81002133>] do_signal+0x43/0x940
> [x.908319]  [<ffffffff81051698>] ? do_send_sig_info+0x58/0x80
> [x.908320]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
> [x.908323]  [<ffffffff81569ca0>] int_signal+0x12/0x17
  This is really fishy. So ext4_free_blocks() has might_sleep() just at its
beginning so at that point irqs were enabled. ext4_dirty_inode() ends up
having the might_sleep() check also at its beginning (from
ext4_journal_check_start()) so the disabling must have happened somewhere
inbetween.

The __mark_inode_dirty() call likely comes from dquot_free_block(). Can you
attach your current .config and also output of /proc/mounts? Depending on
that I'll see what other points checked for sleepable context. Definitely
ext4_journal_get_write_access() and ext4_mb_load_buddy() check for
might_sleep() as well and there's not much happening between that and the
call to dquot_free_block() in ext4_free_blocks(). Strange. 

								Honza

> [x.908329] ------------[ cut here ]------------
> [x.908352] kernel BUG at fs/buffer.c:1268!
> [x.908370] invalid opcode: 0000 [#1] SMP 
> [x.908391] Modules linked in: pl2303 fuse ftdi_sio usbserial iTCO_wdt
> [x.908425] CPU: 6 PID: 15216 Comm: iceweasel Not tainted 3.11.5-00008-ga1818c5 #99
> [x.908460] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
> [x.908484] task: ffff8801124ae800 ti: ffff88010d248000 task.ti: ffff88010d248000
> [x.908504] RIP: 0010:[<ffffffff815612a6>]  [<ffffffff815612a6>] check_irqs_on.part.19+0x4/0x6
> [x.908529] RSP: 0018:ffff88010d249798  EFLAGS: 00210046
> [x.908543] RAX: 0000000000200082 RBX: ffff88010d249928 RCX: ffff880215a5c000
> [x.908562] RDX: 0000000000001000 RSI: 000000000038005b RDI: ffff8802164296c0
> [x.908580] RBP: ffff88010d249798 R08: 0000000000000000 R09: 0000000000000000
> [x.908599] R10: ffff880215a5c000 R11: ffff88010d24947e R12: ffff8802164296c0
> [x.908617] R13: 0000000000001000 R14: ffff88021fbdbe00 R15: ffff88021549a000
> [x.908635] FS:  0000000000000000(0000) GS:ffff88021fb80000(0000) knlGS:0000000000000000
> [x.908656] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
> [x.908672] CR2: 0000000000000000 CR3: 000000000180c000 CR4: 00000000000407e0
> [x.908690] Stack:
> [x.908696]  ffff88010d249808 ffffffff8112feb7 000000000000b760 ffff88010d2498f0
> [x.908720]  ffff88021fb83fc0 ffff88010d2498e8 ffff88010d249840 ffffffff81004b7f
> [x.908743]  ffff88010d24980c 000000000d2498f0 ffff88010d249928 ffff8802164296c0
> [x.908766] Call Trace:
> [x.908776]  [<ffffffff8112feb7>] __find_get_block+0x1d7/0x1e0
> [x.908793]  [<ffffffff81004b7f>] ? dump_trace+0x17f/0x2d0
> [x.908808]  [<ffffffff8112fee0>] __getblk+0x20/0x2f0
> [x.908823]  [<ffffffff81171516>] __ext4_get_inode_loc+0x106/0x410
> [x.908840]  [<ffffffff81004d7f>] ? show_stack_log_lvl+0xaf/0x1a0
> [x.908857]  [<ffffffff811734b8>] ext4_get_inode_loc+0x18/0x20
> [x.908874]  [<ffffffff81174c61>] ext4_reserve_inode_write+0x21/0x90
> [x.908891]  [<ffffffff81561d7f>] ? dump_stack+0x54/0x74
> [x.908906]  [<ffffffff81174d19>] ext4_mark_inode_dirty+0x49/0x1a0
> [x.908924]  [<ffffffff811777ab>] ext4_dirty_inode+0x3b/0x60
> [x.908940]  [<ffffffff811296ed>] __mark_inode_dirty+0x2d/0x230
> [x.908957]  [<ffffffff811992bc>] ext4_free_blocks+0x73c/0xa30
> [x.908974]  [<ffffffff8118d936>] ext4_ext_remove_space+0x806/0xe20
> [x.908991]  [<ffffffff8119fb14>] ? ext4_es_free_extent+0x54/0x60
> [x.909008]  [<ffffffff8118fc18>] ext4_ext_truncate+0xb8/0xe0
> [x.909025]  [<ffffffff81176065>] ext4_truncate+0x2b5/0x300
> [x.909041]  [<ffffffff81176b18>] ext4_evict_inode+0x3f8/0x430
> [x.909057]  [<ffffffff8111c69a>] evict+0xba/0x1c0
> [x.909071]  [<ffffffff8111d04b>] iput+0x10b/0x1b0
> [x.909084]  [<ffffffff81118e38>] dput+0x278/0x350
> [x.909099]  [<ffffffff81104d0a>] __fput+0x16a/0x240
> [x.909113]  [<ffffffff81104e19>] ____fput+0x9/0x10
> [x.909127]  [<ffffffff8105e30c>] task_work_run+0x9c/0xd0
> [x.909143]  [<ffffffff810451f7>] do_exit+0x2a7/0x9d0
> [x.909157]  [<ffffffff8104f8ce>] ? __sigqueue_free.part.13+0x2e/0x40
> [x.909175]  [<ffffffff8104679e>] do_group_exit+0x3e/0xb0
> [x.909190]  [<ffffffff81052740>] get_signal_to_deliver+0x1b0/0x5f0
> [x.909207]  [<ffffffff81002133>] do_signal+0x43/0x940
> [x.909222]  [<ffffffff81051698>] ? do_send_sig_info+0x58/0x80
> [x.909238]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
> [x.909254]  [<ffffffff81569ca0>] int_signal+0x12/0x17
> [x.909267] Code: 4d 85 e4 74 1d 41 80 44 24 58 01 65 48 8b 04 25 b0 b7 00 00 ff 88 44 e0 ff ff 4c 89 e7 e8 23 79 bb ff 5b 41 5c 5d c3 55 48 89 e5 <0f> 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 
> [x.909390] RIP  [<ffffffff815612a6>] check_irqs_on.part.19+0x4/0x6
> [x.909408]  RSP <ffff88010d249798>
> [x.915643] ---[ end trace 379d96cb0444fcb3 ]---
> [x.915645] Fixing recursive fault but reboot is needed!
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

Jan Kara <jack@suse.cz> wrote:
> On Thu 31-10-13 05:58:16, George Spelvin wrote:
>> [x.908259] Call Trace:
>> [x.908265]  [<ffffffff81561d7f>] dump_stack+0x54/0x74
>> [x.908268]  [<ffffffff81069d2f>] __might_sleep+0xcf/0xf0
>> [x.908271]  [<ffffffff8119079b>] ext4_journal_check_start+0x1b/0xa0
>> [x.908273]  [<ffffffff81190871>] __ext4_journal_start_sb+0x21/0x80
>> [x.908276]  [<ffffffff81177795>] ext4_dirty_inode+0x25/0x60
>> [x.908280]  [<ffffffff811296ed>] __mark_inode_dirty+0x2d/0x230
>> [x.908283]  [<ffffffff811992bc>] ext4_free_blocks+0x73c/0xa30
>> [x.908285]  [<ffffffff8118d936>] ext4_ext_remove_space+0x806/0xe20
>> [x.908287]  [<ffffffff8119fb14>] ? ext4_es_free_extent+0x54/0x60
>> [x.908289]  [<ffffffff8118fc18>] ext4_ext_truncate+0xb8/0xe0
>> [x.908291]  [<ffffffff81176065>] ext4_truncate+0x2b5/0x300
>> [x.908292]  [<ffffffff81176b18>] ext4_evict_inode+0x3f8/0x430
>> [x.908295]  [<ffffffff8111c69a>] evict+0xba/0x1c0
>> [x.908297]  [<ffffffff8111d04b>] iput+0x10b/0x1b0
>> [x.908298]  [<ffffffff81118e38>] dput+0x278/0x350
>> [x.908301]  [<ffffffff81104d0a>] __fput+0x16a/0x240
>> [x.908303]  [<ffffffff81104e19>] ____fput+0x9/0x10
>> [x.908306]  [<ffffffff8105e30c>] task_work_run+0x9c/0xd0
>> [x.908309]  [<ffffffff810451f7>] do_exit+0x2a7/0x9d0
>> [x.908311]  [<ffffffff8104f8ce>] ? __sigqueue_free.part.13+0x2e/0x40
>> [x.908312]  [<ffffffff8104679e>] do_group_exit+0x3e/0xb0
>> [x.908315]  [<ffffffff81052740>] get_signal_to_deliver+0x1b0/0x5f0
>> [x.908317]  [<ffffffff81002133>] do_signal+0x43/0x940
>> [x.908319]  [<ffffffff81051698>] ? do_send_sig_info+0x58/0x80
>> [x.908320]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
>> [x.908323]  [<ffffffff81569ca0>] int_signal+0x12/0x17
>
>   This is really fishy. So ext4_free_blocks() has might_sleep() just at its
> beginning so at that point irqs were enabled. ext4_dirty_inode() ends up
> having the might_sleep() check also at its beginning (from
> ext4_journal_check_start()) so the disabling must have happened somewhere
> in between.

Thanks a lot for your debugging help!

> The __mark_inode_dirty() call likely comes from dquot_free_block(). Can you
> attach your current .config and also output of /proc/mounts? Depending on
> that I'll see what other points checked for sleepable context. Definitely
> ext4_journal_get_write_access() and ext4_mb_load_buddy() check for
> might_sleep() as well and there's not much happening between that and the
> call to dquot_free_block() in ext4_free_blocks(). Strange. 

"grep -v '^#' .config | cat -s" appended, and here's /proc/mounts.
The NFS mount with hostname, path, and IP address redacted is a a
read-only mount of "useful stuff" that was completely idle at the time.
(It's not a home directory or /usr/share or anything.)

rootfs / rootfs rw 0 0
/dev/root / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=805136k,mode=755 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=10240k,nr_inodes=1006234,mode=755 0 0
tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=6643400k 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/md2 /home ext4 rw,relatime,data=ordered 0 0
tmpfs /tmp tmpfs rw,relatime,size=16777216k 0 0
rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
server:/export/redacted /red/acted nfs ro,nosuid,nodev,noexec,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=0.1.2.3,mountvers=3,mountport=2050,mountproto=udp,local_lock=none,addr=0.1.2.3 0 0

CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_BZIP2=y
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_HAVE_GENERIC_HARDIRQS=y

CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

CONFIG_IRQ_TIME_ACCOUNTING=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y

CONFIG_TREE_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
CONFIG_IKCONFIG=y
CONFIG_LOG_BUF_SHIFT=16
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
CONFIG_UIDGID_CONVERTED=y
CONFIG_UIDGID_STRICT_TYPE_CHECKS=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_EXPERT=y
CONFIG_UID16=y
CONFIG_KALLSYMS=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_PCI_QUIRKS=y
CONFIG_HAVE_PERF_EVENTS=y

CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_SLUB_CPU_PARTIAL=y
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_JUMP_LABEL=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y

CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y

CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_BLOCK_COMPAT=y

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_DEFAULT_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y

CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_NO_BOOTMEM=y
CONFIG_MCORE2=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_PROCESSOR_SELECT=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_PREEMPT_COUNT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_SECCOMP=y
CONFIG_HZ_300=y
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION="/dev/hda1"
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=y
CONFIG_ACPI_FAN=m
CONFIG_ACPI_I2C=y
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_THERMAL=m
CONFIG_ACPI_CUSTOM_DSDT_FILE=""
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y

CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEFAULT=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y

CONFIG_ISA_DMA_API=y
CONFIG_PCCARD=m
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_PCCARD_NONSTATIC=y

CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_SCRIPT=y
CONFIG_COREDUMP=y
CONFIG_IA32_EMULATION=y
CONFIG_X86_X32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y

CONFIG_PACKET=y
CONFIG_PACKET_DIAG=y
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=m
CONFIG_XFRM_USER=m
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_INET_UDP_DIAG=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_IP_SCTP=m
CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5=y
CONFIG_SCTP_COOKIE_HMAC_MD5=y
CONFIG_HAVE_NET_DSA=y
CONFIG_DNS_RESOLVER=y
CONFIG_NETLINK_DIAG=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
CONFIG_NET_FLOW_LIMIT=y

CONFIG_WIRELESS=y

CONFIG_HAVE_BPF_JIT=y

CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_DMA_SHARED_BUFFER=y

CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8

CONFIG_EEPROM_LEGACY=y

CONFIG_HAVE_IDE=y

CONFIG_SCSI_MOD=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y

CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_CONSTANTS=y

CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y

CONFIG_SATA_AHCI=y
CONFIG_ATA_SFF=y

CONFIG_ATA_BMDMA=y

CONFIG_PATA_JMICRON=y

CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y

CONFIG_NETDEVICES=y
CONFIG_MII=y
CONFIG_NET_CORE=y

CONFIG_ETHERNET=y
CONFIG_NET_VENDOR_INTEL=y
CONFIG_E1000E=y
CONFIG_NET_VENDOR_REALTEK=y
CONFIG_R8169=y
CONFIG_PHYLIB=y

CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
CONFIG_BCM87XX_PHY=m
CONFIG_ICPLUS_PHY=m
CONFIG_REALTEK_PHY=m
CONFIG_NATIONAL_PHY=m
CONFIG_STE10XP=m
CONFIG_LSI_ET1011C_PHY=m
CONFIG_MICREL_PHY=m
CONFIG_MDIO_BITBANG=m

CONFIG_INPUT=y

CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1920
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1080
CONFIG_INPUT_EVDEV=y

CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y

CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m

CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_UNIX98_PTYS=y

CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_DETECT_IRQ=y

CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_NVRAM=m

CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y

CONFIG_I2C_I801=y

CONFIG_I2C_SCMI=m

CONFIG_PPS=y

CONFIG_PTP_1588_CLOCK=y

CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_GPIO_DEVRES=y
CONFIG_POWER_SUPPLY=y
CONFIG_HWMON=y
CONFIG_HWMON_VID=y

CONFIG_SENSORS_IT87=y

CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
CONFIG_THERMAL_GOV_STEP_WISE=y
CONFIG_THERMAL_GOV_USER_SPACE=y
CONFIG_X86_PKG_TEMP_THERMAL=y

CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y

CONFIG_SOFT_WATCHDOG=m
CONFIG_ITCO_WDT=m
CONFIG_IT87_WDT=m

CONFIG_SSB_POSSIBLE=y

CONFIG_BCMA_POSSIBLE=y

CONFIG_MFD_CORE=y
CONFIG_LPC_ICH=y
CONFIG_MEDIA_SUPPORT=y

CONFIG_MEDIA_RC_SUPPORT=y

CONFIG_RC_CORE=y
CONFIG_RC_MAP=m
CONFIG_RC_DEVICES=y
CONFIG_RC_ATI_REMOTE=m

CONFIG_AGP=y
CONFIG_AGP_INTEL=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=2
CONFIG_DRM=y
CONFIG_DRM_KMS_HELPER=y

CONFIG_DRM_I915=y
CONFIG_DRM_I915_KMS=y
CONFIG_VIDEO_OUTPUT_CONTROL=y
CONFIG_HDMI=y
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_MODE_HELPERS=y

CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_BACKLIGHT_CLASS_DEVICE=y

CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=y
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_PCI=y
CONFIG_SND_HDA_INTEL=y
CONFIG_SND_HDA_PREALLOC_SIZE=256
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=3

CONFIG_HID=y
CONFIG_HIDRAW=y
CONFIG_HID_GENERIC=y

CONFIG_HID_DRAGONRISE=y
CONFIG_HID_KYE=y
CONFIG_HID_GYRATION=y
CONFIG_HID_TWINHAN=y
CONFIG_HID_NTRIG=y
CONFIG_HID_ORTEK=y
CONFIG_HID_PANTHERLORD=y
CONFIG_HID_PETALYNX=y
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SUNPLUS=y
CONFIG_HID_GREENASIA=y
CONFIG_HID_SMARTJOYPLUS=y
CONFIG_HID_TOPSEED=y
CONFIG_HID_THRUSTMASTER=y
CONFIG_HID_ZEROPLUS=y

CONFIG_USB_HID=y
CONFIG_USB_HIDDEV=y

CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

CONFIG_USB_DEFAULT_PERSIST=y
CONFIG_USB_MON=y

CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=y
CONFIG_USB_UHCI_HCD=m

CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m

CONFIG_USB_STORAGE=m

CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_CH341=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_IUU=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_MOS7720=m
CONFIG_USB_SERIAL_MOS7840=m
CONFIG_USB_SERIAL_MOTOROLA=m
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_SPCP8X5=m
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_SERIAL_DEBUG=m

CONFIG_USB_EZUSB_FX2=m
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_SYSTOHC=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"

CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y

CONFIG_RTC_DRV_CMOS=y

CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_INTEL_IPS=y

CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_DMAR_TABLE=y
CONFIG_INTEL_IOMMU=y
CONFIG_INTEL_IOMMU_DEFAULT_ON=y
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
CONFIG_IRQ_REMAP=y

CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DMIID=y

CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_JBD2=y
CONFIG_FS_MBCACHE=y
CONFIG_FS_POSIX_ACL=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FUSE_FS=m
CONFIG_GENERIC_ACL=y

CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="utf8"

CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m

CONFIG_TRACE_IRQFLAGS_SUPPORT=y

CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4

CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=1024
CONFIG_STRIP_ASM_SYMS=y
CONFIG_DEBUG_FS=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_KERNEL=y

CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
CONFIG_HAVE_ARCH_KMEMCHECK=y

CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_TIMER_STATS=y

CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_BUGVERBOSE=y

CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y

CONFIG_HAVE_ARCH_KGDB=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_RODATA=y
CONFIG_DOUBLEFAULT=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_NONE=y
CONFIG_DEFAULT_IO_DELAY_TYPE=3
CONFIG_OPTIMIZE_INLINING=y

CONFIG_KEYS=y
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_CRYPTO=y

CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=y
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRYPTO_ABLK_HELPER_X86=y
CONFIG_CRYPTO_GLUE_HELPER_X86=y

CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m

CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=y
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=y

CONFIG_CRYPTO_CMAC=m
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m

CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=y
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256_SSSE3=m
CONFIG_CRYPTO_SHA512_SSSE3=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m

CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=y
CONFIG_CRYPTO_AES_NI_INTEL=y
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_X86_64=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX_X86_64=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=m

CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=m
CONFIG_CRYPTO_LZ4=m
CONFIG_CRYPTO_LZ4HC=m

CONFIG_HAVE_KVM=y

CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=m
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC32_SLICEBY8=y
CONFIG_CRC7=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=m
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_COMPRESS=m
CONFIG_LZ4HC_COMPRESS=m
CONFIG_LZ4_DECOMPRESS=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
CONFIG_OID_REGISTRY=y
CONFIG_FONT_SUPPORT=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Andreas Dilger]

On Oct 17, 2013, at 4:14 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Thu, Oct 17, 2013 at 05:11:43PM -0400, George Spelvin wrote:
>> 
>> Well, it happened again (error appended).  Can you please clarify what you mean
>> by "such BUG_ON()"; I'm having a hard time following the RCU code and determining
>> all the situations under which __fput() might be called.
> 
> __fput() can be called via task_work_run() or via schedule_work().  That's
> all.  And it certainly should never be called with interrupts disabled.
> So stick BUG_ON(irqs_disabled()) in it (WARN_ON() might be better, but
> not by much).
> 
> There are two ways these traces could've happened:
> 	* exit_task_work() called by do_exit() with irqs disabled.
> Definitely buggy (and would do really nasty things to several functions
> called by do_exit() before that one). 
> 	* __fput() is called with irqs enabled, but somewhere on the
> way into ext4 (dput -> iput -> evict inode -> free blocks, now that
> unlinked file got closed -> ...) we manage to disable irqs and forget
> to enable them.

IMHO the most common case of "BUG: sleeping function called from
invalid context” is due to stack overflow.  This corrupts the task
struct, and incorrectly sets the “in_interrupt” bit.

What kind of storage stack is underneath this filesystem?  If
it is deep (e.g. DM + LVM + iSCSI) then the stack overflow is
definitely possible.

There were also a discussion by Christoph of page allocation
recursing into the fs again (in "xfs: prevent stack overflows
from page cache allocation”) though I’m not sure if that applies
to ext4 or not.

Cheers, Andreas

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

Andreas Dilger asked:
> What kind of storage stack is underneath this filesystem?  If
> it is deep (e.g. DM + LVM + iSCSI) then the stack overflow is
> definitely possible.

ext4 on md raid1 on SATA.  Nothing too complicated.

Personalities : [raid0] [raid1] 
md1 : active raid1 sdb2[1] sda2[0]	# Root partition
      41942976 blocks [2/2] [UU]
      
md2 : active raid1 sdb3[1] sda3[0]	# /home partition
      100663232 blocks [2/2] [UU]
      
md0 : active raid1 sdb1[1] sda1[0]	# Swap partition
      25165696 blocks [2/2] [UU]
      
unused devices: <none>


There's 8G of RAM, and a large swap partition because I download
a lot of videos to tmpfs on /tmp.  (/tmp is capped at 16G, and swap
is 24G, so I shouldn't run out, but there was certainly quite
a few GB of stuff there.)

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Jan Kara]

On Thu 31-10-13 12:30:51, George Spelvin wrote:
> Jan Kara <jack@suse.cz> wrote:
> > On Thu 31-10-13 05:58:16, George Spelvin wrote:
> >> [x.908259] Call Trace:
> >> [x.908265]  [<ffffffff81561d7f>] dump_stack+0x54/0x74
> >> [x.908268]  [<ffffffff81069d2f>] __might_sleep+0xcf/0xf0
> >> [x.908271]  [<ffffffff8119079b>] ext4_journal_check_start+0x1b/0xa0
> >> [x.908273]  [<ffffffff81190871>] __ext4_journal_start_sb+0x21/0x80
> >> [x.908276]  [<ffffffff81177795>] ext4_dirty_inode+0x25/0x60
> >> [x.908280]  [<ffffffff811296ed>] __mark_inode_dirty+0x2d/0x230
> >> [x.908283]  [<ffffffff811992bc>] ext4_free_blocks+0x73c/0xa30
> >> [x.908285]  [<ffffffff8118d936>] ext4_ext_remove_space+0x806/0xe20
> >> [x.908287]  [<ffffffff8119fb14>] ? ext4_es_free_extent+0x54/0x60
> >> [x.908289]  [<ffffffff8118fc18>] ext4_ext_truncate+0xb8/0xe0
> >> [x.908291]  [<ffffffff81176065>] ext4_truncate+0x2b5/0x300
> >> [x.908292]  [<ffffffff81176b18>] ext4_evict_inode+0x3f8/0x430
> >> [x.908295]  [<ffffffff8111c69a>] evict+0xba/0x1c0
> >> [x.908297]  [<ffffffff8111d04b>] iput+0x10b/0x1b0
> >> [x.908298]  [<ffffffff81118e38>] dput+0x278/0x350
> >> [x.908301]  [<ffffffff81104d0a>] __fput+0x16a/0x240
> >> [x.908303]  [<ffffffff81104e19>] ____fput+0x9/0x10
> >> [x.908306]  [<ffffffff8105e30c>] task_work_run+0x9c/0xd0
> >> [x.908309]  [<ffffffff810451f7>] do_exit+0x2a7/0x9d0
> >> [x.908311]  [<ffffffff8104f8ce>] ? __sigqueue_free.part.13+0x2e/0x40
> >> [x.908312]  [<ffffffff8104679e>] do_group_exit+0x3e/0xb0
> >> [x.908315]  [<ffffffff81052740>] get_signal_to_deliver+0x1b0/0x5f0
> >> [x.908317]  [<ffffffff81002133>] do_signal+0x43/0x940
> >> [x.908319]  [<ffffffff81051698>] ? do_send_sig_info+0x58/0x80
> >> [x.908320]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
> >> [x.908323]  [<ffffffff81569ca0>] int_signal+0x12/0x17
> >
> >   This is really fishy. So ext4_free_blocks() has might_sleep() just at its
> > beginning so at that point irqs were enabled. ext4_dirty_inode() ends up
> > having the might_sleep() check also at its beginning (from
> > ext4_journal_check_start()) so the disabling must have happened somewhere
> > in between.
> 
> Thanks a lot for your debugging help!
> 
> > The __mark_inode_dirty() call likely comes from dquot_free_block(). Can you
> > attach your current .config and also output of /proc/mounts? Depending on
> > that I'll see what other points checked for sleepable context. Definitely
> > ext4_journal_get_write_access() and ext4_mb_load_buddy() check for
> > might_sleep() as well and there's not much happening between that and the
> > call to dquot_free_block() in ext4_free_blocks(). Strange. 
> 
> "grep -v '^#' .config | cat -s" appended, and here's /proc/mounts.
> The NFS mount with hostname, path, and IP address redacted is a a
> read-only mount of "useful stuff" that was completely idle at the time.
> (It's not a home directory or /usr/share or anything.)
> 
> rootfs / rootfs rw 0 0
> /dev/root / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=805136k,mode=755 0 0
> tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
> proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
> sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
> devtmpfs /dev devtmpfs rw,relatime,size=10240k,nr_inodes=1006234,mode=755 0 0
> tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=6643400k 0 0
> devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
> fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
> /dev/md2 /home ext4 rw,relatime,data=ordered 0 0
> tmpfs /tmp tmpfs rw,relatime,size=16777216k 0 0
> rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
> server:/export/redacted /red/acted nfs ro,nosuid,nodev,noexec,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=0.1.2.3,mountvers=3,mountport=2050,mountproto=udp,local_lock=none,addr=0.1.2.3 0 0
  Thanks for info. So ext4 mount options look pretty normal, quota is
disabled meaning that really the last place doing might_sleep() check is
ext4_mb_load_buddy(). The only thing that somewhat catched my eye is
CONFIG_SLUB.

So can you add attached patch which adds couple more might_sleep() into
ext4_free_blocks(). Also you can enable CONFIG_DEBUG_STACKOVERWLOW just to
make sure we aren't really overflowing the stack. Also you can try using
CONFIG_SLAB instead of CONFIG_SLUB to rule out some oddity in that
allocator.

								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Jan Kara]

[-- Attachment #1: Type: text/plain, Size: 4758 bytes --]

On Thu 31-10-13 21:37:25, Jan Kara wrote:
> On Thu 31-10-13 12:30:51, George Spelvin wrote:
> > Jan Kara <jack@suse.cz> wrote:
> > > On Thu 31-10-13 05:58:16, George Spelvin wrote:
> > >> [x.908259] Call Trace:
> > >> [x.908265]  [<ffffffff81561d7f>] dump_stack+0x54/0x74
> > >> [x.908268]  [<ffffffff81069d2f>] __might_sleep+0xcf/0xf0
> > >> [x.908271]  [<ffffffff8119079b>] ext4_journal_check_start+0x1b/0xa0
> > >> [x.908273]  [<ffffffff81190871>] __ext4_journal_start_sb+0x21/0x80
> > >> [x.908276]  [<ffffffff81177795>] ext4_dirty_inode+0x25/0x60
> > >> [x.908280]  [<ffffffff811296ed>] __mark_inode_dirty+0x2d/0x230
> > >> [x.908283]  [<ffffffff811992bc>] ext4_free_blocks+0x73c/0xa30
> > >> [x.908285]  [<ffffffff8118d936>] ext4_ext_remove_space+0x806/0xe20
> > >> [x.908287]  [<ffffffff8119fb14>] ? ext4_es_free_extent+0x54/0x60
> > >> [x.908289]  [<ffffffff8118fc18>] ext4_ext_truncate+0xb8/0xe0
> > >> [x.908291]  [<ffffffff81176065>] ext4_truncate+0x2b5/0x300
> > >> [x.908292]  [<ffffffff81176b18>] ext4_evict_inode+0x3f8/0x430
> > >> [x.908295]  [<ffffffff8111c69a>] evict+0xba/0x1c0
> > >> [x.908297]  [<ffffffff8111d04b>] iput+0x10b/0x1b0
> > >> [x.908298]  [<ffffffff81118e38>] dput+0x278/0x350
> > >> [x.908301]  [<ffffffff81104d0a>] __fput+0x16a/0x240
> > >> [x.908303]  [<ffffffff81104e19>] ____fput+0x9/0x10
> > >> [x.908306]  [<ffffffff8105e30c>] task_work_run+0x9c/0xd0
> > >> [x.908309]  [<ffffffff810451f7>] do_exit+0x2a7/0x9d0
> > >> [x.908311]  [<ffffffff8104f8ce>] ? __sigqueue_free.part.13+0x2e/0x40
> > >> [x.908312]  [<ffffffff8104679e>] do_group_exit+0x3e/0xb0
> > >> [x.908315]  [<ffffffff81052740>] get_signal_to_deliver+0x1b0/0x5f0
> > >> [x.908317]  [<ffffffff81002133>] do_signal+0x43/0x940
> > >> [x.908319]  [<ffffffff81051698>] ? do_send_sig_info+0x58/0x80
> > >> [x.908320]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
> > >> [x.908323]  [<ffffffff81569ca0>] int_signal+0x12/0x17
> > >
> > >   This is really fishy. So ext4_free_blocks() has might_sleep() just at its
> > > beginning so at that point irqs were enabled. ext4_dirty_inode() ends up
> > > having the might_sleep() check also at its beginning (from
> > > ext4_journal_check_start()) so the disabling must have happened somewhere
> > > in between.
> > 
> > Thanks a lot for your debugging help!
> > 
> > > The __mark_inode_dirty() call likely comes from dquot_free_block(). Can you
> > > attach your current .config and also output of /proc/mounts? Depending on
> > > that I'll see what other points checked for sleepable context. Definitely
> > > ext4_journal_get_write_access() and ext4_mb_load_buddy() check for
> > > might_sleep() as well and there's not much happening between that and the
> > > call to dquot_free_block() in ext4_free_blocks(). Strange. 
> > 
> > "grep -v '^#' .config | cat -s" appended, and here's /proc/mounts.
> > The NFS mount with hostname, path, and IP address redacted is a a
> > read-only mount of "useful stuff" that was completely idle at the time.
> > (It's not a home directory or /usr/share or anything.)
> > 
> > rootfs / rootfs rw 0 0
> > /dev/root / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> > tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=805136k,mode=755 0 0
> > tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
> > proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
> > sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
> > devtmpfs /dev devtmpfs rw,relatime,size=10240k,nr_inodes=1006234,mode=755 0 0
> > tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=6643400k 0 0
> > devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
> > fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
> > /dev/md2 /home ext4 rw,relatime,data=ordered 0 0
> > tmpfs /tmp tmpfs rw,relatime,size=16777216k 0 0
> > rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
> > server:/export/redacted /red/acted nfs ro,nosuid,nodev,noexec,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=0.1.2.3,mountvers=3,mountport=2050,mountproto=udp,local_lock=none,addr=0.1.2.3 0 0
>   Thanks for info. So ext4 mount options look pretty normal, quota is
> disabled meaning that really the last place doing might_sleep() check is
> ext4_mb_load_buddy(). The only thing that somewhat catched my eye is
> CONFIG_SLUB.
> 
> So can you add attached patch which adds couple more might_sleep() into
> ext4_free_blocks(). Also you can enable CONFIG_DEBUG_STACKOVERWLOW just to
> make sure we aren't really overflowing the stack. Also you can try using
> CONFIG_SLAB instead of CONFIG_SLUB to rule out some oddity in that
> allocator.
  Forgot to attach the patch...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

[-- Attachment #2: 0001-Sprinkle-some-might_sleep-checks-into-the-code.patch --]
[-- Type: text/x-patch, Size: 1704 bytes --]

>From 0babe605cb5b71bec811c7c8676c3f6a97b148d4 Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Thu, 31 Oct 2013 21:22:57 +0100
Subject: [PATCH] Sprinkle some might_sleep() checks into the code.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/ext4/mballoc.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index a41e3ba8cfaa..8b6bdb975e10 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4757,6 +4757,7 @@ do_more:
 	if (err)
 		goto error_return;
 
+	might_sleep();
 	if ((flags & EXT4_FREE_BLOCKS_METADATA) && ext4_handle_valid(handle)) {
 		struct ext4_free_data *new_entry;
 		/*
@@ -4764,6 +4765,7 @@ do_more:
 		 * be used until this transaction is committed
 		 */
 	retry:
+		might_sleep();
 		new_entry = kmem_cache_alloc(ext4_free_data_cachep, GFP_NOFS);
 		if (!new_entry) {
 			/*
@@ -4779,6 +4781,7 @@ do_more:
 		new_entry->efd_count = count_clusters;
 		new_entry->efd_tid = handle->h_transaction->t_tid;
 
+		might_sleep();
 		ext4_lock_group(sb, block_group);
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
 		ext4_mb_free_metadata(handle, &e4b, new_entry);
@@ -4796,7 +4799,7 @@ do_more:
 					 err);
 		}
 
-
+		might_sleep();
 		ext4_lock_group(sb, block_group);
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
 		mb_free_blocks(inode, &e4b, bit, count_clusters);
@@ -4807,6 +4810,7 @@ do_more:
 	ext4_block_bitmap_csum_set(sb, block_group, gdp, bitmap_bh);
 	ext4_group_desc_csum_set(sb, block_group, gdp);
 	ext4_unlock_group(sb, block_group);
+	might_sleep();
 
 	if (sbi->s_log_groups_per_flex) {
 		ext4_group_t flex_group = ext4_flex_group(sbi, block_group);
-- 
1.8.1.4

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

Due to wanting to stick with 3.11.x baseline, as opposed to whatever you
based your diff on, I had to amend the last hunk slightly.  Included
just FYI.

Compiled, rebooting now.  It may take some days to get a
bug report.

diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 4bbbf13b..e6f0d6b 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4734,6 +4734,7 @@ do_more:
 	if (err)
 		goto error_return;
 
+	might_sleep();
 	if ((flags & EXT4_FREE_BLOCKS_METADATA) && ext4_handle_valid(handle)) {
 		struct ext4_free_data *new_entry;
 		/*
@@ -4741,6 +4742,7 @@ do_more:
 		 * be used until this transaction is committed
 		 */
 	retry:
+		might_sleep();
 		new_entry = kmem_cache_alloc(ext4_free_data_cachep, GFP_NOFS);
 		if (!new_entry) {
 			/*
@@ -4756,6 +4758,7 @@ do_more:
 		new_entry->efd_count = count_clusters;
 		new_entry->efd_tid = handle->h_transaction->t_tid;
 
+		might_sleep();
 		ext4_lock_group(sb, block_group);
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
 		ext4_mb_free_metadata(handle, &e4b, new_entry);
@@ -4773,7 +4776,7 @@ do_more:
 					 err);
 		}
 
-
+		might_sleep();
 		ext4_lock_group(sb, block_group);
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
 		mb_free_blocks(inode, &e4b, bit, count_clusters);
@@ -4785,6 +4788,7 @@ do_more:
 	ext4_group_desc_csum_set(sb, block_group, gdp);
 	ext4_unlock_group(sb, block_group);
 	percpu_counter_add(&sbi->s_freeclusters_counter, count_clusters);
+	might_sleep();
 
 	if (sbi->s_log_groups_per_flex) {
 		ext4_group_t flex_group = ext4_flex_group(sbi, block_group);

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

Well, it finally triggered.


Not *that* long before, I fiddled with a USB thumb drive, which
I'll mention here, but I don't think it's connected.

[2328294.996152] usb 1-1.3: new high-speed USB device number 6 using ehci-pci
[2328295.080347] usb 1-1.3: New USB device found, idVendor=0781, idProduct=556c
[2328295.080351] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2328295.080352] usb 1-1.3: Product: Ultra
[2328295.080353] usb 1-1.3: Manufacturer: SanDisk
[2328295.080354] usb 1-1.3: SerialNumber: 20054861120C8D407604
[2328295.829526] usb-storage 1-1.3:1.0: USB Mass Storage device detected
[2328295.829571] scsi6 : usb-storage 1-1.3:1.0
[2328295.829615] usbcore: registered new interface driver usb-storage
[2328296.832215] scsi 6:0:0:0: Direct-Access     SanDisk  Ultra            1.20 PQ: 0 ANSI: 5
[2328296.832343] sd 6:0:0:0: Attached scsi generic sg3 type 0
[2328296.833579] sd 6:0:0:0: [sdc] 15633408 512-byte logical blocks: (8.00 GB/7.45 GiB)
[2328296.834942] sd 6:0:0:0: [sdc] Write Protect is off
[2328296.834944] sd 6:0:0:0: [sdc] Mode Sense: 43 00 00 00
[2328296.835947] sd 6:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[2328296.848345]  sdc: sdc1
[2328296.851338] sd 6:0:0:0: [sdc] Attached SCSI removable disk
[2328361.332585] FAT-fs (sdc1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
[2335705.838834] usb 1-1.3: USB disconnect, device number 6


The next thing in the kernel log is the might_sleep() warning (followed by
the oops):

[2348070.539862] BUG: sleeping function called from invalid context at fs/ext4/mballoc.c:4791
[2348070.539865] in_atomic(): 0, irqs_disabled(): 1, pid: 4635, name: iceweasel
[2348070.539867] CPU: 4 PID: 4635 Comm: iceweasel Tainted: G        W    3.11.5-00009-g06a2442 #100
[2348070.539868] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[2348070.539870]  ffff88011379f4e0 ffff8801a30179a8 ffffffff81561017 0000000000000002
[2348070.539872]  ffff8801a30179b8 ffffffff8106a06f ffff8801a3017a90 ffffffff81197a3e
[2348070.539874]  ffff880011306f68 ffff8801a3017fd8 00000001a30179f8 0000000000f15fd1
[2348070.539876] Call Trace:
[2348070.539881]  [<ffffffff81561017>] dump_stack+0x54/0x74
[2348070.539884]  [<ffffffff8106a06f>] __might_sleep+0xcf/0xf0
[2348070.539887]  [<ffffffff81197a3e>] ext4_free_blocks+0x53e/0xa90
[2348070.539889]  [<ffffffff8118c286>] ext4_ext_remove_space+0x806/0xe20
[2348070.539891]  [<ffffffff8118e568>] ext4_ext_truncate+0xb8/0xe0
[2348070.539894]  [<ffffffff811749a5>] ext4_truncate+0x2b5/0x300
[2348070.539895]  [<ffffffff81175458>] ext4_evict_inode+0x3f8/0x430
[2348070.539898]  [<ffffffff8111acca>] evict+0xba/0x1c0
[2348070.539899]  [<ffffffff8111b67b>] iput+0x10b/0x1b0
[2348070.539901]  [<ffffffff81117468>] dput+0x278/0x350
[2348070.539904]  [<ffffffff811032da>] __fput+0x16a/0x240
[2348070.539905]  [<ffffffff811033e9>] ____fput+0x9/0x10
[2348070.539909]  [<ffffffff8105e63c>] task_work_run+0x9c/0xd0
[2348070.539911]  [<ffffffff81045547>] do_exit+0x2a7/0x9d0
[2348070.539914]  [<ffffffff8104fc1e>] ? __sigqueue_free.part.13+0x2e/0x40
[2348070.539915]  [<ffffffff81046aee>] do_group_exit+0x3e/0xb0
[2348070.539917]  [<ffffffff81052a90>] get_signal_to_deliver+0x1b0/0x5f0
[2348070.539919]  [<ffffffff81002133>] do_signal+0x43/0x940
[2348070.539921]  [<ffffffff810519e8>] ? do_send_sig_info+0x58/0x80
[2348070.539923]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
[2348070.539925]  [<ffffffff81568f60>] int_signal+0x12/0x17
[2348070.539931] ------------[ cut here ]------------
[2348070.539950] kernel BUG at fs/buffer.c:1268!
[2348070.539962] invalid opcode: 0000 [#1] SMP 
[2348070.539976] Modules linked in: nls_utf8 nls_cp437 vfat fat usb_storage fuse pl2303 ftdi_sio usbserial iTCO_wdt
[2348070.540018] CPU: 4 PID: 4635 Comm: iceweasel Tainted: G        W    3.11.5-00009-g06a2442 #100
[2348070.540040] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[2348070.540063] task: ffff88021688cf00 ti: ffff8801a3016000 task.ti: ffff8801a3016000
[2348070.540082] RIP: 0010:[<ffffffff8156053e>]  [<ffffffff8156053e>] check_irqs_on.part.16+0x4/0x6
[2348070.540108] RSP: 0018:ffff8801a3017798  EFLAGS: 00010046
[2348070.540122] RAX: 0000000000000082 RBX: ffff8801a3017928 RCX: ffff8802162bd000
[2348070.540141] RDX: 0000000000001000 RSI: 0000000000980080 RDI: ffff8802171b6b00
[2348070.540159] RBP: ffff8801a3017798 R08: 0000000000000002 R09: 0000000000000002
[2348070.540177] R10: ffff8802162bd000 R11: ffff8801a301751e R12: ffff8802171b6b00
[2348070.540195] R13: 0000000000001000 R14: ffff880213a70600 R15: ffff880215228c00
[2348070.540214] FS:  0000000000000000(0000) GS:ffff88021fb00000(0000) knlGS:0000000000000000
[2348070.540235] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[2348070.540250] CR2: 00000000f4832a3c CR3: 000000000180c000 CR4: 00000000000407e0
[2348070.540268] Stack:
[2348070.540274]  ffff8801a3017808 ffffffff8112e547 ffff8801a30177b0 0000000000000092
[2348070.540297]  ffffffff81568f60 ffffffff81732ff4 ffffffff81568f60 ffffffff810987bd
[2348070.540320]  00000000ffe743c4 ffff8801a30177f0 ffff8801a3017928 ffff8802171b6b00
[2348070.540342] Call Trace:
[2348070.540352]  [<ffffffff8112e547>] __find_get_block+0x1d7/0x1e0
[2348070.540369]  [<ffffffff81568f60>] ? int_signal+0x12/0x17
[2348070.540384]  [<ffffffff81568f60>] ? int_signal+0x12/0x17
[2348070.540400]  [<ffffffff810987bd>] ? __module_text_address+0xd/0x60
[2348070.540417]  [<ffffffff8112e570>] __getblk+0x20/0x2f0
[2348070.540432]  [<ffffffff8116fe46>] __ext4_get_inode_loc+0x106/0x410
[2348070.540448]  [<ffffffff81171df8>] ext4_get_inode_loc+0x18/0x20
[2348070.540465]  [<ffffffff811735a1>] ext4_reserve_inode_write+0x21/0x90
[2348070.540482]  [<ffffffff81173659>] ext4_mark_inode_dirty+0x49/0x1a0
[2348070.540499]  [<ffffffff811760eb>] ext4_dirty_inode+0x3b/0x60
[2348070.540515]  [<ffffffff81127d3d>] __mark_inode_dirty+0x2d/0x230
[2348070.540532]  [<ffffffff81197cbc>] ext4_free_blocks+0x7bc/0xa90
[2348070.540548]  [<ffffffff8118c286>] ext4_ext_remove_space+0x806/0xe20
[2348070.540565]  [<ffffffff8118e568>] ext4_ext_truncate+0xb8/0xe0
[2348070.540581]  [<ffffffff811749a5>] ext4_truncate+0x2b5/0x300
[2348070.540597]  [<ffffffff81175458>] ext4_evict_inode+0x3f8/0x430
[2348070.540613]  [<ffffffff8111acca>] evict+0xba/0x1c0
[2348070.540627]  [<ffffffff8111b67b>] iput+0x10b/0x1b0
[2348070.540641]  [<ffffffff81117468>] dput+0x278/0x350
[2348070.540655]  [<ffffffff811032da>] __fput+0x16a/0x240
[2348070.540669]  [<ffffffff811033e9>] ____fput+0x9/0x10
[2348070.540683]  [<ffffffff8105e63c>] task_work_run+0x9c/0xd0
[2348070.540698]  [<ffffffff81045547>] do_exit+0x2a7/0x9d0
[2348070.540713]  [<ffffffff8104fc1e>] ? __sigqueue_free.part.13+0x2e/0x40
[2348070.540730]  [<ffffffff81046aee>] do_group_exit+0x3e/0xb0
[2348070.540745]  [<ffffffff81052a90>] get_signal_to_deliver+0x1b0/0x5f0
[2348070.540762]  [<ffffffff81002133>] do_signal+0x43/0x940
[2348070.540777]  [<ffffffff810519e8>] ? do_send_sig_info+0x58/0x80
[2348070.540793]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
[2348070.540809]  [<ffffffff81568f60>] int_signal+0x12/0x17
[2348070.540823] Code: 4d 85 e4 74 1d 41 80 44 24 58 01 65 48 8b 04 25 b0 b7 00 00 ff 88 44 e0 ff ff 4c 89 e7 e8 bb 6c bb ff 5b 41 5c 5d c3 55 48 89 e5 <0f> 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 
[2348070.540943] RIP  [<ffffffff8156053e>] check_irqs_on.part.16+0x4/0x6
[2348070.540961]  RSP <ffff8801a3017798>
[2348070.547170] ---[ end trace 41094c9a65f45b94 ]---
[2348070.547171] Fixing recursive fault but reboot is needed!


Here's the debug patch from last month's e-mail, for reference.
The one that tripped is the last hunk.

diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 4bbbf13b..e6f0d6b 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4734,6 +4734,7 @@ do_more:
 	if (err)
 		goto error_return;
 
+	might_sleep();
 	if ((flags & EXT4_FREE_BLOCKS_METADATA) && ext4_handle_valid(handle)) {
 		struct ext4_free_data *new_entry;
 		/*
@@ -4741,6 +4742,7 @@ do_more:
 		 * be used until this transaction is committed
 		 */
 	retry:
+		might_sleep();
 		new_entry = kmem_cache_alloc(ext4_free_data_cachep, GFP_NOFS);
 		if (!new_entry) {
 			/*
@@ -4756,6 +4758,7 @@ do_more:
 		new_entry->efd_count = count_clusters;
 		new_entry->efd_tid = handle->h_transaction->t_tid;
 
+		might_sleep();
 		ext4_lock_group(sb, block_group);
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
 		ext4_mb_free_metadata(handle, &e4b, new_entry);
@@ -4773,7 +4776,7 @@ do_more:
 					 err);
 		}
 
-
+		might_sleep();
 		ext4_lock_group(sb, block_group);
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
 		mb_free_blocks(inode, &e4b, bit, count_clusters);
@@ -4785,6 +4788,7 @@ do_more:
 	ext4_group_desc_csum_set(sb, block_group, gdp);
 	ext4_unlock_group(sb, block_group);
 	percpu_counter_add(&sbi->s_freeclusters_counter, count_clusters);
+	might_sleep();
 
 	if (sbi->s_log_groups_per_flex) {
 		ext4_group_t flex_group = ext4_flex_group(sbi, block_group);

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Jan Kara]

[-- Attachment #1: Type: text/plain, Size: 10041 bytes --]

On Thu 28-11-13 00:09:06, George Spelvin wrote:
> Well, it finally triggered.
> 
> 
> Not *that* long before, I fiddled with a USB thumb drive, which
> I'll mention here, but I don't think it's connected.
> 
> [2328294.996152] usb 1-1.3: new high-speed USB device number 6 using ehci-pci
> [2328295.080347] usb 1-1.3: New USB device found, idVendor=0781, idProduct=556c
> [2328295.080351] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
> [2328295.080352] usb 1-1.3: Product: Ultra
> [2328295.080353] usb 1-1.3: Manufacturer: SanDisk
> [2328295.080354] usb 1-1.3: SerialNumber: 20054861120C8D407604
> [2328295.829526] usb-storage 1-1.3:1.0: USB Mass Storage device detected
> [2328295.829571] scsi6 : usb-storage 1-1.3:1.0
> [2328295.829615] usbcore: registered new interface driver usb-storage
> [2328296.832215] scsi 6:0:0:0: Direct-Access     SanDisk  Ultra            1.20 PQ: 0 ANSI: 5
> [2328296.832343] sd 6:0:0:0: Attached scsi generic sg3 type 0
> [2328296.833579] sd 6:0:0:0: [sdc] 15633408 512-byte logical blocks: (8.00 GB/7.45 GiB)
> [2328296.834942] sd 6:0:0:0: [sdc] Write Protect is off
> [2328296.834944] sd 6:0:0:0: [sdc] Mode Sense: 43 00 00 00
> [2328296.835947] sd 6:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> [2328296.848345]  sdc: sdc1
> [2328296.851338] sd 6:0:0:0: [sdc] Attached SCSI removable disk
> [2328361.332585] FAT-fs (sdc1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
> [2335705.838834] usb 1-1.3: USB disconnect, device number 6
> 
> 
> The next thing in the kernel log is the might_sleep() warning (followed by
> the oops):
> 
> [2348070.539862] BUG: sleeping function called from invalid context at fs/ext4/mballoc.c:4791
> [2348070.539865] in_atomic(): 0, irqs_disabled(): 1, pid: 4635, name: iceweasel
> [2348070.539867] CPU: 4 PID: 4635 Comm: iceweasel Tainted: G        W    3.11.5-00009-g06a2442 #100
> [2348070.539868] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
> [2348070.539870]  ffff88011379f4e0 ffff8801a30179a8 ffffffff81561017 0000000000000002
> [2348070.539872]  ffff8801a30179b8 ffffffff8106a06f ffff8801a3017a90 ffffffff81197a3e
> [2348070.539874]  ffff880011306f68 ffff8801a3017fd8 00000001a30179f8 0000000000f15fd1
> [2348070.539876] Call Trace:
> [2348070.539881]  [<ffffffff81561017>] dump_stack+0x54/0x74
> [2348070.539884]  [<ffffffff8106a06f>] __might_sleep+0xcf/0xf0
> [2348070.539887]  [<ffffffff81197a3e>] ext4_free_blocks+0x53e/0xa90
  Thanks for the report. So this is the last might_sleep() we have added.
Interesting. I've also realized it isn't a stack overflow problem because
irqs_disabled() check in might_sleep() checks real setting of a flag in the
processor flags.

I went again through the several calls between last two might_sleep() calls
but I didn't find anything suspicious. So let's narrow this down a bit more
by the attached patch (to be applied on top of your previous patch).
Thanks!

								Honza

> [2348070.539889]  [<ffffffff8118c286>] ext4_ext_remove_space+0x806/0xe20
> [2348070.539891]  [<ffffffff8118e568>] ext4_ext_truncate+0xb8/0xe0
> [2348070.539894]  [<ffffffff811749a5>] ext4_truncate+0x2b5/0x300
> [2348070.539895]  [<ffffffff81175458>] ext4_evict_inode+0x3f8/0x430
> [2348070.539898]  [<ffffffff8111acca>] evict+0xba/0x1c0
> [2348070.539899]  [<ffffffff8111b67b>] iput+0x10b/0x1b0
> [2348070.539901]  [<ffffffff81117468>] dput+0x278/0x350
> [2348070.539904]  [<ffffffff811032da>] __fput+0x16a/0x240
> [2348070.539905]  [<ffffffff811033e9>] ____fput+0x9/0x10
> [2348070.539909]  [<ffffffff8105e63c>] task_work_run+0x9c/0xd0
> [2348070.539911]  [<ffffffff81045547>] do_exit+0x2a7/0x9d0
> [2348070.539914]  [<ffffffff8104fc1e>] ? __sigqueue_free.part.13+0x2e/0x40
> [2348070.539915]  [<ffffffff81046aee>] do_group_exit+0x3e/0xb0
> [2348070.539917]  [<ffffffff81052a90>] get_signal_to_deliver+0x1b0/0x5f0
> [2348070.539919]  [<ffffffff81002133>] do_signal+0x43/0x940
> [2348070.539921]  [<ffffffff810519e8>] ? do_send_sig_info+0x58/0x80
> [2348070.539923]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
> [2348070.539925]  [<ffffffff81568f60>] int_signal+0x12/0x17
> [2348070.539931] ------------[ cut here ]------------
> [2348070.539950] kernel BUG at fs/buffer.c:1268!
> [2348070.539962] invalid opcode: 0000 [#1] SMP 
> [2348070.539976] Modules linked in: nls_utf8 nls_cp437 vfat fat usb_storage fuse pl2303 ftdi_sio usbserial iTCO_wdt
> [2348070.540018] CPU: 4 PID: 4635 Comm: iceweasel Tainted: G        W    3.11.5-00009-g06a2442 #100
> [2348070.540040] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
> [2348070.540063] task: ffff88021688cf00 ti: ffff8801a3016000 task.ti: ffff8801a3016000
> [2348070.540082] RIP: 0010:[<ffffffff8156053e>]  [<ffffffff8156053e>] check_irqs_on.part.16+0x4/0x6
> [2348070.540108] RSP: 0018:ffff8801a3017798  EFLAGS: 00010046
> [2348070.540122] RAX: 0000000000000082 RBX: ffff8801a3017928 RCX: ffff8802162bd000
> [2348070.540141] RDX: 0000000000001000 RSI: 0000000000980080 RDI: ffff8802171b6b00
> [2348070.540159] RBP: ffff8801a3017798 R08: 0000000000000002 R09: 0000000000000002
> [2348070.540177] R10: ffff8802162bd000 R11: ffff8801a301751e R12: ffff8802171b6b00
> [2348070.540195] R13: 0000000000001000 R14: ffff880213a70600 R15: ffff880215228c00
> [2348070.540214] FS:  0000000000000000(0000) GS:ffff88021fb00000(0000) knlGS:0000000000000000
> [2348070.540235] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
> [2348070.540250] CR2: 00000000f4832a3c CR3: 000000000180c000 CR4: 00000000000407e0
> [2348070.540268] Stack:
> [2348070.540274]  ffff8801a3017808 ffffffff8112e547 ffff8801a30177b0 0000000000000092
> [2348070.540297]  ffffffff81568f60 ffffffff81732ff4 ffffffff81568f60 ffffffff810987bd
> [2348070.540320]  00000000ffe743c4 ffff8801a30177f0 ffff8801a3017928 ffff8802171b6b00
> [2348070.540342] Call Trace:
> [2348070.540352]  [<ffffffff8112e547>] __find_get_block+0x1d7/0x1e0
> [2348070.540369]  [<ffffffff81568f60>] ? int_signal+0x12/0x17
> [2348070.540384]  [<ffffffff81568f60>] ? int_signal+0x12/0x17
> [2348070.540400]  [<ffffffff810987bd>] ? __module_text_address+0xd/0x60
> [2348070.540417]  [<ffffffff8112e570>] __getblk+0x20/0x2f0
> [2348070.540432]  [<ffffffff8116fe46>] __ext4_get_inode_loc+0x106/0x410
> [2348070.540448]  [<ffffffff81171df8>] ext4_get_inode_loc+0x18/0x20
> [2348070.540465]  [<ffffffff811735a1>] ext4_reserve_inode_write+0x21/0x90
> [2348070.540482]  [<ffffffff81173659>] ext4_mark_inode_dirty+0x49/0x1a0
> [2348070.540499]  [<ffffffff811760eb>] ext4_dirty_inode+0x3b/0x60
> [2348070.540515]  [<ffffffff81127d3d>] __mark_inode_dirty+0x2d/0x230
> [2348070.540532]  [<ffffffff81197cbc>] ext4_free_blocks+0x7bc/0xa90
> [2348070.540548]  [<ffffffff8118c286>] ext4_ext_remove_space+0x806/0xe20
> [2348070.540565]  [<ffffffff8118e568>] ext4_ext_truncate+0xb8/0xe0
> [2348070.540581]  [<ffffffff811749a5>] ext4_truncate+0x2b5/0x300
> [2348070.540597]  [<ffffffff81175458>] ext4_evict_inode+0x3f8/0x430
> [2348070.540613]  [<ffffffff8111acca>] evict+0xba/0x1c0
> [2348070.540627]  [<ffffffff8111b67b>] iput+0x10b/0x1b0
> [2348070.540641]  [<ffffffff81117468>] dput+0x278/0x350
> [2348070.540655]  [<ffffffff811032da>] __fput+0x16a/0x240
> [2348070.540669]  [<ffffffff811033e9>] ____fput+0x9/0x10
> [2348070.540683]  [<ffffffff8105e63c>] task_work_run+0x9c/0xd0
> [2348070.540698]  [<ffffffff81045547>] do_exit+0x2a7/0x9d0
> [2348070.540713]  [<ffffffff8104fc1e>] ? __sigqueue_free.part.13+0x2e/0x40
> [2348070.540730]  [<ffffffff81046aee>] do_group_exit+0x3e/0xb0
> [2348070.540745]  [<ffffffff81052a90>] get_signal_to_deliver+0x1b0/0x5f0
> [2348070.540762]  [<ffffffff81002133>] do_signal+0x43/0x940
> [2348070.540777]  [<ffffffff810519e8>] ? do_send_sig_info+0x58/0x80
> [2348070.540793]  [<ffffffff81002a8d>] do_notify_resume+0x5d/0x80
> [2348070.540809]  [<ffffffff81568f60>] int_signal+0x12/0x17
> [2348070.540823] Code: 4d 85 e4 74 1d 41 80 44 24 58 01 65 48 8b 04 25 b0 b7 00 00 ff 88 44 e0 ff ff 4c 89 e7 e8 bb 6c bb ff 5b 41 5c 5d c3 55 48 89 e5 <0f> 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 
> [2348070.540943] RIP  [<ffffffff8156053e>] check_irqs_on.part.16+0x4/0x6
> [2348070.540961]  RSP <ffff8801a3017798>
> [2348070.547170] ---[ end trace 41094c9a65f45b94 ]---
> [2348070.547171] Fixing recursive fault but reboot is needed!
> 
> 
> Here's the debug patch from last month's e-mail, for reference.
> The one that tripped is the last hunk.
> 
> diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
> index 4bbbf13b..e6f0d6b 100644
> --- a/fs/ext4/mballoc.c
> +++ b/fs/ext4/mballoc.c
> @@ -4734,6 +4734,7 @@ do_more:
>  	if (err)
>  		goto error_return;
>  
> +	might_sleep();
>  	if ((flags & EXT4_FREE_BLOCKS_METADATA) && ext4_handle_valid(handle)) {
>  		struct ext4_free_data *new_entry;
>  		/*
> @@ -4741,6 +4742,7 @@ do_more:
>  		 * be used until this transaction is committed
>  		 */
>  	retry:
> +		might_sleep();
>  		new_entry = kmem_cache_alloc(ext4_free_data_cachep, GFP_NOFS);
>  		if (!new_entry) {
>  			/*
> @@ -4756,6 +4758,7 @@ do_more:
>  		new_entry->efd_count = count_clusters;
>  		new_entry->efd_tid = handle->h_transaction->t_tid;
>  
> +		might_sleep();
>  		ext4_lock_group(sb, block_group);
>  		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
>  		ext4_mb_free_metadata(handle, &e4b, new_entry);
> @@ -4773,7 +4776,7 @@ do_more:
>  					 err);
>  		}
>  
> -
> +		might_sleep();
>  		ext4_lock_group(sb, block_group);
>  		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
>  		mb_free_blocks(inode, &e4b, bit, count_clusters);
> @@ -4785,6 +4788,7 @@ do_more:
>  	ext4_group_desc_csum_set(sb, block_group, gdp);
>  	ext4_unlock_group(sb, block_group);
>  	percpu_counter_add(&sbi->s_freeclusters_counter, count_clusters);
> +	might_sleep();
>  
>  	if (sbi->s_log_groups_per_flex) {
>  		ext4_group_t flex_group = ext4_flex_group(sbi, block_group);
> 
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

[-- Attachment #2: 0001-Debug-irqs.patch --]
[-- Type: text/x-patch, Size: 1612 bytes --]

>From 0868c52e3a67c43b2e0f2eb3275e6d994b5f3bdd Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Thu, 28 Nov 2013 15:36:53 +0100
Subject: [PATCH] Debug irqs

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/ext4/mballoc.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 267c14e3a19e..d22f9690aaf6 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4760,8 +4760,11 @@ do_more:
 
 		might_sleep();
 		ext4_lock_group(sb, block_group);
+		WARN_ON(irqs_disabled());
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
+		WARN_ON(irqs_disabled());
 		ext4_mb_free_metadata(handle, &e4b, new_entry);
+		WARN_ON(irqs_disabled());
 	} else {
 		/* need to update group_info->bb_free and bitmap
 		 * with group lock held. generate_buddy look at
@@ -4779,15 +4782,23 @@ do_more:
 
 		might_sleep();
 		ext4_lock_group(sb, block_group);
+		WARN_ON(irqs_disabled());
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
+		WARN_ON(irqs_disabled());
 		mb_free_blocks(inode, &e4b, bit, count_clusters);
+		WARN_ON(irqs_disabled());
 	}
 
 	ret = ext4_free_group_clusters(sb, gdp) + count_clusters;
+	WARN_ON(irqs_disabled());
 	ext4_free_group_clusters_set(sb, gdp, ret);
+	WARN_ON(irqs_disabled());
 	ext4_block_bitmap_csum_set(sb, block_group, gdp, bitmap_bh);
+	WARN_ON(irqs_disabled());
 	ext4_group_desc_csum_set(sb, block_group, gdp);
+	WARN_ON(irqs_disabled());
 	ext4_unlock_group(sb, block_group);
+	WARN_ON(irqs_disabled());
 	percpu_counter_add(&sbi->s_freeclusters_counter, count_clusters);
 	might_sleep();
 
-- 
1.8.1.4

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

One of those additional WARN_ON tests tripped, hooray!
And it turned out to be in the ext4 metadata checksumming.  To be
precise, ext4_block_bitmap_csum_set() returned with irqs disabled,
and kaboom.

Since I have this experimental feature turned on and most people don't,
this explains why I'm finding it and World+Dog aren't.

I appear to be the designated finder of ext4 metadata_csum bugs, so tytso
notified on general principles.  I dropped the generic linux-fsdevel
list from the Cc: list.

But looking at the code, it just calls into the linux-crypto layer and
Tim Chen's SSE CRC32C implementation which uses kernel_fpu_begin()
and kernel_fpu_end() if the block is large enough.

I was going to add and Herbert Xu and Tim Chen and all those mailing
lists, but looking at the code, it sure *looks* like they're Doing The
right Thing, so I'm holding off for a bit.

I'm not sure quite where to pass th buck on this one.

Relevant platform info:
- Intel i7-2700K processor, with SSE4.2 and thus the CRC32C instruction.
- CONFIG_PREEMPT_VOLUNTARY=y
- # CONFIG_PREEMPT_NONE is not set
- CONFIG_PREEMPT_VOLUNTARY=y
- # CONFIG_PREEMPT is not set
- CONFIG_PREEMPT_COUNT=y
- CONFIG_DEBUG_ATOMIC_SLEEP=y
- CONFIG_DEBUG_BUGVERBOSE=y


[475059.561544] ------------[ cut here ]------------
[475059.561553] WARNING: CPU: 4 PID: 24642 at fs/ext4/mballoc.c:4796 ext4_free_blocks+0xb77/0xc10()
[475059.561554] Modules linked in: fuse pl2303 ftdi_sio usbserial iTCO_wdt
[475059.561560] CPU: 4 PID: 24642 Comm: iceweasel Not tainted 3.11.5-00010-g5626386 #101
[475059.561561] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[475059.561562]  0000000000000009 ffff880204e99970 ffffffff8156050d 0000000000000000
[475059.561565]  ffff880204e999a8 ffffffff81044103 ffff88021fb9b360 ffff8800a809eb60
[475059.561567]  ffff880168efda20 ffff88021667f000 ffff88021610fec0 ffff880204e999b8
[475059.561569] Call Trace:
[475059.561573]  [<ffffffff8156050d>] dump_stack+0x54/0x74
[475059.561576]  [<ffffffff81044103>] warn_slowpath_common+0x73/0x90
[475059.561578]  [<ffffffff810441d5>] warn_slowpath_null+0x15/0x20
[475059.561580]  [<ffffffff81197da7>] ext4_free_blocks+0xb77/0xc10
[475059.561582]  [<ffffffff8118bfd6>] ext4_ext_remove_space+0x806/0xe20
[475059.561585]  [<ffffffff8119e3c4>] ? ext4_es_free_extent+0x54/0x60
[475059.561587]  [<ffffffff8118e2b8>] ext4_ext_truncate+0xb8/0xe0
[475059.561590]  [<ffffffff81174705>] ext4_truncate+0x2b5/0x300
[475059.561592]  [<ffffffff811751b8>] ext4_evict_inode+0x3f8/0x430
[475059.561596]  [<ffffffff8111aa4a>] evict+0xba/0x1c0
[475059.561598]  [<ffffffff8111b3fb>] iput+0x10b/0x1b0
[475059.561600]  [<ffffffff81117208>] dput+0x278/0x350
[475059.561604]  [<ffffffff8110308a>] __fput+0x16a/0x240
[475059.561606]  [<ffffffff81103199>] ____fput+0x9/0x10
[475059.561609]  [<ffffffff8105e57c>] task_work_run+0x9c/0xd0
[475059.561612]  [<ffffffff810454a7>] do_exit+0x2a7/0x9d0
[475059.561615]  [<ffffffff8104fb6e>] ? __sigqueue_free.part.13+0x2e/0x40
[475059.561618]  [<ffffffff81046a4e>] do_group_exit+0x3e/0xb0
[475059.561620]  [<ffffffff810529e0>] get_signal_to_deliver+0x1b0/0x5f0
[475059.561623]  [<ffffffff81002113>] do_signal+0x43/0x940
[475059.561626]  [<ffffffff81051938>] ? do_send_sig_info+0x58/0x80
[475059.561628]  [<ffffffff81002a6d>] do_notify_resume+0x5d/0x80
[475059.561632]  [<ffffffff81568420>] int_signal+0x12/0x17
[475059.561633] ---[ end trace b093418eaa9338ce ]---
[475059.561635] ------------[ cut here ]------------
[475059.561637] WARNING: CPU: 4 PID: 24642 at fs/ext4/mballoc.c:4798 ext4_free_blocks+0xb61/0xc10()
[475059.561638] Modules linked in: fuse pl2303 ftdi_sio usbserial iTCO_wdt
[475059.561642] CPU: 4 PID: 24642 Comm: iceweasel Tainted: G        W    3.11.5-00010-g5626386 #101
[475059.561644] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[475059.561644]  0000000000000009 ffff880204e99970 ffffffff8156050d 0000000000000000
[475059.561647]  ffff880204e999a8 ffffffff81044103 ffff88021fb9b360 ffff8800a809eb60
[475059.561649]  ffff880168efda20 ffff88021667f000 ffff88021610fec0 ffff880204e999b8
[475059.561651] Call Trace:
[475059.561654]  [<ffffffff8156050d>] dump_stack+0x54/0x74
[475059.561656]  [<ffffffff81044103>] warn_slowpath_common+0x73/0x90
[475059.561658]  [<ffffffff810441d5>] warn_slowpath_null+0x15/0x20
[475059.561660]  [<ffffffff81197d91>] ext4_free_blocks+0xb61/0xc10
[475059.561662]  [<ffffffff8118bfd6>] ext4_ext_remove_space+0x806/0xe20
[475059.561665]  [<ffffffff8119e3c4>] ? ext4_es_free_extent+0x54/0x60
[475059.561667]  [<ffffffff8118e2b8>] ext4_ext_truncate+0xb8/0xe0
[475059.561669]  [<ffffffff81174705>] ext4_truncate+0x2b5/0x300
[475059.561672]  [<ffffffff811751b8>] ext4_evict_inode+0x3f8/0x430
[475059.561674]  [<ffffffff8111aa4a>] evict+0xba/0x1c0
[475059.561675]  [<ffffffff8111b3fb>] iput+0x10b/0x1b0
[475059.561677]  [<ffffffff81117208>] dput+0x278/0x350
[475059.561679]  [<ffffffff8110308a>] __fput+0x16a/0x240
[475059.561681]  [<ffffffff81103199>] ____fput+0x9/0x10
[475059.561684]  [<ffffffff8105e57c>] task_work_run+0x9c/0xd0
[475059.561686]  [<ffffffff810454a7>] do_exit+0x2a7/0x9d0
[475059.561688]  [<ffffffff8104fb6e>] ? __sigqueue_free.part.13+0x2e/0x40
[475059.561691]  [<ffffffff81046a4e>] do_group_exit+0x3e/0xb0
[475059.561693]  [<ffffffff810529e0>] get_signal_to_deliver+0x1b0/0x5f0
[475059.561695]  [<ffffffff81002113>] do_signal+0x43/0x940
[475059.561697]  [<ffffffff81051938>] ? do_send_sig_info+0x58/0x80
[475059.561700]  [<ffffffff81002a6d>] do_notify_resume+0x5d/0x80
[475059.561702]  [<ffffffff81568420>] int_signal+0x12/0x17
[475059.561704] ---[ end trace b093418eaa9338cf ]---
[475059.561705] ------------[ cut here ]------------
[475059.561708] WARNING: CPU: 4 PID: 24642 at fs/ext4/mballoc.c:4800 ext4_free_blocks+0xad0/0xc10()
[475059.561709] Modules linked in: fuse pl2303 ftdi_sio usbserial iTCO_wdt
[475059.561713] CPU: 4 PID: 24642 Comm: iceweasel Tainted: G        W    3.11.5-00010-g5626386 #101
[475059.561714] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[475059.561715]  0000000000000009 ffff880204e99970 ffffffff8156050d 0000000000000000
[475059.561717]  ffff880204e999a8 ffffffff81044103 ffff88021fb9b360 ffff8800a809eb60
[475059.561720]  ffff880168efda20 ffff88021667f000 ffff88021610fec0 ffff880204e999b8
[475059.561723] Call Trace:
[475059.561726]  [<ffffffff8156050d>] dump_stack+0x54/0x74
[475059.561728]  [<ffffffff81044103>] warn_slowpath_common+0x73/0x90
[475059.561730]  [<ffffffff810441d5>] warn_slowpath_null+0x15/0x20
[475059.561733]  [<ffffffff81197d00>] ext4_free_blocks+0xad0/0xc10
[475059.561735]  [<ffffffff8118bfd6>] ext4_ext_remove_space+0x806/0xe20
[475059.561738]  [<ffffffff8119e3c4>] ? ext4_es_free_extent+0x54/0x60
[475059.561740]  [<ffffffff8118e2b8>] ext4_ext_truncate+0xb8/0xe0
[475059.561742]  [<ffffffff81174705>] ext4_truncate+0x2b5/0x300
[475059.561745]  [<ffffffff811751b8>] ext4_evict_inode+0x3f8/0x430
[475059.561747]  [<ffffffff8111aa4a>] evict+0xba/0x1c0
[475059.561748]  [<ffffffff8111b3fb>] iput+0x10b/0x1b0
[475059.561750]  [<ffffffff81117208>] dput+0x278/0x350
[475059.561753]  [<ffffffff8110308a>] __fput+0x16a/0x240
[475059.561755]  [<ffffffff81103199>] ____fput+0x9/0x10
[475059.561757]  [<ffffffff8105e57c>] task_work_run+0x9c/0xd0
[475059.561760]  [<ffffffff810454a7>] do_exit+0x2a7/0x9d0
[475059.561762]  [<ffffffff8104fb6e>] ? __sigqueue_free.part.13+0x2e/0x40
[475059.561765]  [<ffffffff81046a4e>] do_group_exit+0x3e/0xb0
[475059.561767]  [<ffffffff810529e0>] get_signal_to_deliver+0x1b0/0x5f0
[475059.561768]  [<ffffffff81002113>] do_signal+0x43/0x940
[475059.561770]  [<ffffffff81051938>] ? do_send_sig_info+0x58/0x80
[475059.561771]  [<ffffffff81002a6d>] do_notify_resume+0x5d/0x80
[475059.561773]  [<ffffffff81568420>] int_signal+0x12/0x17
[475059.561774] ---[ end trace b093418eaa9338d0 ]---
[475059.561775] BUG: sleeping function called from invalid context at fs/ext4/mballoc.c:4802
[475059.561776] in_atomic(): 0, irqs_disabled(): 1, pid: 24642, name: iceweasel
[475059.561778] CPU: 4 PID: 24642 Comm: iceweasel Tainted: G        W    3.11.5-00010-g5626386 #101
[475059.561779] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[475059.561779]  ffff8800a809eb60 ffff880204e999a8 ffffffff8156050d 0000000000000079
[475059.561781]  ffff880204e999b8 ffffffff81069faf ffff880204e99a90 ffffffff811977c0
[475059.561783]  ffff880131ba3b58 ffff880204e99fd8 0000000104e999f8 00000000008de1f9
[475059.561784] Call Trace:
[475059.561786]  [<ffffffff8156050d>] dump_stack+0x54/0x74
[475059.561789]  [<ffffffff81069faf>] __might_sleep+0xcf/0xf0
[475059.561791]  [<ffffffff811977c0>] ext4_free_blocks+0x590/0xc10
[475059.561793]  [<ffffffff8118bfd6>] ext4_ext_remove_space+0x806/0xe20
[475059.561795]  [<ffffffff8119e3c4>] ? ext4_es_free_extent+0x54/0x60
[475059.561796]  [<ffffffff8118e2b8>] ext4_ext_truncate+0xb8/0xe0
[475059.561798]  [<ffffffff81174705>] ext4_truncate+0x2b5/0x300
[475059.561800]  [<ffffffff811751b8>] ext4_evict_inode+0x3f8/0x430
[475059.561801]  [<ffffffff8111aa4a>] evict+0xba/0x1c0
[475059.561803]  [<ffffffff8111b3fb>] iput+0x10b/0x1b0
[475059.561804]  [<ffffffff81117208>] dput+0x278/0x350
[475059.561806]  [<ffffffff8110308a>] __fput+0x16a/0x240
[475059.561807]  [<ffffffff81103199>] ____fput+0x9/0x10
[475059.561809]  [<ffffffff8105e57c>] task_work_run+0x9c/0xd0
[475059.561811]  [<ffffffff810454a7>] do_exit+0x2a7/0x9d0
[475059.561812]  [<ffffffff8104fb6e>] ? __sigqueue_free.part.13+0x2e/0x40
[475059.561814]  [<ffffffff81046a4e>] do_group_exit+0x3e/0xb0
[475059.561816]  [<ffffffff810529e0>] get_signal_to_deliver+0x1b0/0x5f0
[475059.561818]  [<ffffffff81002113>] do_signal+0x43/0x940
[475059.561821]  [<ffffffff81051938>] ? do_send_sig_info+0x58/0x80
[475059.561823]  [<ffffffff81002a6d>] do_notify_resume+0x5d/0x80
[475059.561825]  [<ffffffff81568420>] int_signal+0x12/0x17
[475059.561830] ------------[ cut here ]------------
[475059.561847] kernel BUG at fs/buffer.c:1268!
[475059.561860] invalid opcode: 0000 [#1] SMP 
[475059.561873] Modules linked in: fuse pl2303 ftdi_sio usbserial iTCO_wdt
[475059.561897] CPU: 4 PID: 24642 Comm: iceweasel Tainted: G        W    3.11.5-00010-g5626386 #101
[475059.561919] Hardware name: Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3, BIOS F13 03/20/2012
[475059.561942] task: ffff880216684380 ti: ffff880204e98000 task.ti: ffff880204e98000
[475059.561961] RIP: 0010:[<ffffffff8155fa34>]  [<ffffffff8155fa34>] check_irqs_on.part.16+0x4/0x6
[475059.561987] RSP: 0018:ffff880204e99798  EFLAGS: 00210046
[475059.562001] RAX: 0000000000200082 RBX: ffff880204e99928 RCX: ffff8802160fb000
[475059.562019] RDX: 0000000000001000 RSI: 000000000038005b RDI: ffff8802170f4ac0
[475059.562038] RBP: ffff880204e99798 R08: 0000000000000000 R09: 0000000000000000
[475059.562056] R10: ffff8802160fb000 R11: ffff880204e9951e R12: ffff8802170f4ac0
[475059.562079] R13: 0000000000001000 R14: ffff88021fbdbe00 R15: ffff88021667f000
[475059.562097] FS:  0000000000000000(0000) GS:ffff88021fb00000(0000) knlGS:0000000000000000
[475059.562118] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[475059.562134] CR2: 0000000099d3b000 CR3: 000000000180c000 CR4: 00000000000407e0
[475059.562152] Stack:
[475059.562158]  ffff880204e99808 ffffffff8112e2b7 ffff880204e997b0 0000000000200092
[475059.562181]  ffffffff81568420 ffffffff81732ff4 ffffffff81568420 ffffffff8109861d
[475059.562208]  00000000ff8b8e44 ffff880204e997f0 ffff880204e99928 ffff8802170f4ac0
[475059.562231] Call Trace:
[475059.562241]  [<ffffffff8112e2b7>] __find_get_block+0x1d7/0x1e0
[475059.562257]  [<ffffffff81568420>] ? int_signal+0x12/0x17
[475059.562272]  [<ffffffff81568420>] ? int_signal+0x12/0x17
[475059.562289]  [<ffffffff8109861d>] ? __module_text_address+0xd/0x60
[475059.562306]  [<ffffffff8112e2e0>] __getblk+0x20/0x2f0
[475059.562324]  [<ffffffff8116fbb6>] __ext4_get_inode_loc+0x106/0x410
[475059.562347]  [<ffffffff81171b68>] ext4_get_inode_loc+0x18/0x20
[475059.562363]  [<ffffffff81173311>] ext4_reserve_inode_write+0x21/0x90
[475059.562380]  [<ffffffff811733c9>] ext4_mark_inode_dirty+0x49/0x1a0
[475059.562397]  [<ffffffff81175e4b>] ext4_dirty_inode+0x3b/0x60
[475059.562413]  [<ffffffff81127aad>] __mark_inode_dirty+0x2d/0x230
[475059.562430]  [<ffffffff81197a7c>] ext4_free_blocks+0x84c/0xc10
[475059.562447]  [<ffffffff8118bfd6>] ext4_ext_remove_space+0x806/0xe20
[475059.562464]  [<ffffffff8119e3c4>] ? ext4_es_free_extent+0x54/0x60
[475059.562480]  [<ffffffff8118e2b8>] ext4_ext_truncate+0xb8/0xe0
[475059.562497]  [<ffffffff81174705>] ext4_truncate+0x2b5/0x300
[475059.562512]  [<ffffffff811751b8>] ext4_evict_inode+0x3f8/0x430
[475059.562528]  [<ffffffff8111aa4a>] evict+0xba/0x1c0
[475059.562542]  [<ffffffff8111b3fb>] iput+0x10b/0x1b0
[475059.562555]  [<ffffffff81117208>] dput+0x278/0x350
[475059.562573]  [<ffffffff8110308a>] __fput+0x16a/0x240
[475059.563369]  [<ffffffff81103199>] ____fput+0x9/0x10
[475059.564164]  [<ffffffff8105e57c>] task_work_run+0x9c/0xd0
[475059.564939]  [<ffffffff810454a7>] do_exit+0x2a7/0x9d0
[475059.565728]  [<ffffffff8104fb6e>] ? __sigqueue_free.part.13+0x2e/0x40
[475059.566518]  [<ffffffff81046a4e>] do_group_exit+0x3e/0xb0
[475059.567296]  [<ffffffff810529e0>] get_signal_to_deliver+0x1b0/0x5f0
[475059.568087]  [<ffffffff81002113>] do_signal+0x43/0x940
[475059.568895]  [<ffffffff81051938>] ? do_send_sig_info+0x58/0x80
[475059.569812]  [<ffffffff81002a6d>] do_notify_resume+0x5d/0x80
[475059.570907]  [<ffffffff81568420>] int_signal+0x12/0x17
[475059.572011] Code: 4d 85 e4 74 1d 41 80 44 24 58 01 65 48 8b 04 25 b0 b7 00 00 ff 88 44 e0 ff ff 4c 89 e7 e8 65 75 bb ff 5b 41 5c 5d c3 55 48 89 e5 <0f> 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 
[475059.573325] RIP  [<ffffffff8155fa34>] check_irqs_on.part.16+0x4/0x6
[475059.574581]  RSP <ffff880204e99798>
[475059.583933] ---[ end trace b093418eaa9338d1 ]---
[475059.583936] Fixing recursive fault but reboot is needed!

For a reminder, here's the second debug patch:

commit 5626386a9d37c990977a12f833d6e6fc2e89d8bf
Author: Jan Kara <jack@suse.cz>
Date:   Thu Nov 28 15:36:53 2013 +0100

    Debug irqs
    
    Signed-off-by: Jan Kara <jack@suse.cz>

diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index e6f0d6b..ece38b4 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4760,8 +4760,11 @@ do_more:
 
 		might_sleep();
 		ext4_lock_group(sb, block_group);
+		WARN_ON(irqs_disabled());
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
+		WARN_ON(irqs_disabled());
 		ext4_mb_free_metadata(handle, &e4b, new_entry);
+		WARN_ON(irqs_disabled());
 	} else {
 		/* need to update group_info->bb_free and bitmap
 		 * with group lock held. generate_buddy look at
@@ -4778,15 +4781,23 @@ do_more:
 
 		might_sleep();
 		ext4_lock_group(sb, block_group);
+		WARN_ON(irqs_disabled());
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
+		WARN_ON(irqs_disabled());
 		mb_free_blocks(inode, &e4b, bit, count_clusters);
+		WARN_ON(irqs_disabled());
 	}
 
 	ret = ext4_free_group_clusters(sb, gdp) + count_clusters;
+	WARN_ON(irqs_disabled());
 	ext4_free_group_clusters_set(sb, gdp, ret);
+	WARN_ON(irqs_disabled());
 	ext4_block_bitmap_csum_set(sb, block_group, gdp, bitmap_bh);
+	WARN_ON(irqs_disabled());		/***### BUG HERE ###***/
 	ext4_group_desc_csum_set(sb, block_group, gdp);
+	WARN_ON(irqs_disabled());
 	ext4_unlock_group(sb, block_group);
+	WARN_ON(irqs_disabled());
 	percpu_counter_add(&sbi->s_freeclusters_counter, count_clusters);
 	might_sleep();

The problem is metadata_csum enabled on this file system:

Filesystem volume name:   root
Last mounted on:          /
Filesystem UUID:          xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1049600
Block count:              10485744
Reserved block count:     524287
Free blocks:              6196830
Free inodes:              727373
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1021
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         3280
Inode blocks per group:   205
Flex block group size:    16
Filesystem created:       Mon Oct  7 23:15:31 2013
Last mount time:          Tue Dec 10 01:02:40 2013
Last write time:          Tue Dec 10 01:02:40 2013
Mount count:              2
Maximum mount count:      -1
Last checked:             Wed Dec  4 12:46:49 2013
Check interval:           0 (<none>)
Lifetime writes:          155 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
First orphan inode:       735089
Default directory hash:   half_md4
Directory Hash Seed:      xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0xfeaba085
Journal features:         journal_incompat_revoke
Journal size:             128M
Journal length:           32768
Journal sequence:         0x0004373b
Journal start:            22384

Looking at what happens where,
ext4_block_bitmap_csum_set() is a simple wrapper around ext4_chksum:

=== fs/ext4/bitmap.c:86 ==
void ext4_block_bitmap_csum_set(struct super_block *sb, ext4_group_t group,
				struct ext4_group_desc *gdp,
				struct buffer_head *bh)
{
	int sz = EXT4_CLUSTERS_PER_GROUP(sb) / 8;
	__u32 csum;
	struct ext4_sb_info *sbi = EXT4_SB(sb);

	if (!EXT4_HAS_RO_COMPAT_FEATURE(sb,
			EXT4_FEATURE_RO_COMPAT_METADATA_CSUM))
		return;

	csum = ext4_chksum(sbi, sbi->s_csum_seed, (__u8 *)bh->b_data, sz);
	gdp->bg_block_bitmap_csum_lo = cpu_to_le16(csum & 0xFFFF);
	if (sbi->s_desc_size >= EXT4_BG_BLOCK_BITMAP_CSUM_HI_END)
		gdp->bg_block_bitmap_csum_hi = cpu_to_le16(csum >> 16);
}

which in turn calls only crypto_shash_update:
=== fs/ext4/ext4.h:1701 ==
static inline u32 ext4_chksum(struct ext4_sb_info *sbi, u32 crc,
			      const void *address, unsigned int length)
{
	struct {
		struct shash_desc shash;
		char ctx[4];
	} desc;
	int err;

	BUG_ON(crypto_shash_descsize(sbi->s_chksum_driver)!=sizeof(desc.ctx));

	desc.shash.tfm = sbi->s_chksum_driver;
	desc.shash.flags = 0;
	*(u32 *)desc.ctx = crc;

	err = crypto_shash_update(&desc.shash, address, length);
	BUG_ON(err);

	return *(u32 *)desc.ctx;
}

=== crypto/shash.c:97 ===
int crypto_shash_update(struct shash_desc *desc, const u8 *data,
			unsigned int len)
{
	struct crypto_shash *tfm = desc->tfm;
	struct shash_alg *shash = crypto_shash_alg(tfm);
	unsigned long alignmask = crypto_shash_alignmask(tfm);

	if ((unsigned long)data & alignmask)
		return shash_update_unaligned(desc, data, len);

	return shash->update(desc, data, len);
}



=== Discussion ===
desc.shash.tfm is filled in from sbi->s_chksum_driver, which is filled in at
ext4_fill_super() time by crypto_alloc_shash("crc32c", 0, 0).

Thus, shash->update should turn into a call to crypto/crc32c.c:chksum_update(),
which calls lib/crc32.c:__crc32c_le().

Now, I happen to be running an i7-2700k which has sse4_2, and thus calls
into the x86 specific code, and apparently for large blocks it uses PCLMULQDQ,
which requires kernel_fpu_begin/end.

At least that makes some degree of sense.  The low level code, though
uses the functions in a very simple way that I can't see how it could fail
to unlock at the end.

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Jan Kara]

On Tue 10-12-13 04:35:28, George Spelvin wrote:
> One of those additional WARN_ON tests tripped, hooray!
> And it turned out to be in the ext4 metadata checksumming.  To be
> precise, ext4_block_bitmap_csum_set() returned with irqs disabled,
> and kaboom.
  Ha, great. Thanks for the persistence in testing.

> Since I have this experimental feature turned on and most people don't,
> this explains why I'm finding it and World+Dog aren't.
> 
> I appear to be the designated finder of ext4 metadata_csum bugs, so tytso
> notified on general principles.  I dropped the generic linux-fsdevel
> list from the Cc: list.
> 
> But looking at the code, it just calls into the linux-crypto layer and
> Tim Chen's SSE CRC32C implementation which uses kernel_fpu_begin()
> and kernel_fpu_end() if the block is large enough.
  Yup, that code was also my last hope but I can't say I see any problem in
there either.

> I was going to add and Herbert Xu and Tim Chen and all those mailing
> lists, but looking at the code, it sure *looks* like they're Doing The
> right Thing, so I'm holding off for a bit.
> 
> I'm not sure quite where to pass th buck on this one.
> 
> Relevant platform info:
> - Intel i7-2700K processor, with SSE4.2 and thus the CRC32C instruction.
> - CONFIG_PREEMPT_VOLUNTARY=y
> - # CONFIG_PREEMPT_NONE is not set
> - CONFIG_PREEMPT_VOLUNTARY=y
> - # CONFIG_PREEMPT is not set
> - CONFIG_PREEMPT_COUNT=y
> - CONFIG_DEBUG_ATOMIC_SLEEP=y
> - CONFIG_DEBUG_BUGVERBOSE=y
> 
...
> 
> === Discussion ===
> desc.shash.tfm is filled in from sbi->s_chksum_driver, which is filled in at
> ext4_fill_super() time by crypto_alloc_shash("crc32c", 0, 0).
> 
> Thus, shash->update should turn into a call to crypto/crc32c.c:chksum_update(),
> which calls lib/crc32.c:__crc32c_le().
> 
> Now, I happen to be running an i7-2700k which has sse4_2, and thus calls
> into the x86 specific code, and apparently for large blocks it uses PCLMULQDQ,
> which requires kernel_fpu_begin/end.
> 
> At least that makes some degree of sense.  The low level code, though
> uses the functions in a very simple way that I can't see how it could fail
> to unlock at the end.
  Hum, can you try disabling the HW support of CRC32C implementation
(CRYPTO_CRC32C_INTEL)? If the problem disappears, we know there's some
problem in the HW support code...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[Jan Kara]

On Tue 10-12-13 16:27:01, Jan Kara wrote:
> On Tue 10-12-13 04:35:28, George Spelvin wrote:
> > One of those additional WARN_ON tests tripped, hooray!
> > And it turned out to be in the ext4 metadata checksumming.  To be
> > precise, ext4_block_bitmap_csum_set() returned with irqs disabled,
> > and kaboom.
>   Ha, great. Thanks for the persistence in testing.
> 
> > Since I have this experimental feature turned on and most people don't,
> > this explains why I'm finding it and World+Dog aren't.
> > 
> > I appear to be the designated finder of ext4 metadata_csum bugs, so tytso
> > notified on general principles.  I dropped the generic linux-fsdevel
> > list from the Cc: list.
> > 
> > But looking at the code, it just calls into the linux-crypto layer and
> > Tim Chen's SSE CRC32C implementation which uses kernel_fpu_begin()
> > and kernel_fpu_end() if the block is large enough.
>   Yup, that code was also my last hope but I can't say I see any problem in
> there either.
  BTW, given you always see the problem when ext4_truncate() gets called 
as a response to application catching a deadly signal and thus
task_work_run() gets called, I think there's something in irq_fpu_usable()
which isn't exactly right. But I know nothing about the logic there. Or
maybe the signal is caught in some unlucky moment when FPU is in some
strange state?

								Honza

> > I was going to add and Herbert Xu and Tim Chen and all those mailing
> > lists, but looking at the code, it sure *looks* like they're Doing The
> > right Thing, so I'm holding off for a bit.
> > 
> > I'm not sure quite where to pass th buck on this one.
> > 
> > Relevant platform info:
> > - Intel i7-2700K processor, with SSE4.2 and thus the CRC32C instruction.
> > - CONFIG_PREEMPT_VOLUNTARY=y
> > - # CONFIG_PREEMPT_NONE is not set
> > - CONFIG_PREEMPT_VOLUNTARY=y
> > - # CONFIG_PREEMPT is not set
> > - CONFIG_PREEMPT_COUNT=y
> > - CONFIG_DEBUG_ATOMIC_SLEEP=y
> > - CONFIG_DEBUG_BUGVERBOSE=y
> > 
> ...
> > 
> > === Discussion ===
> > desc.shash.tfm is filled in from sbi->s_chksum_driver, which is filled in at
> > ext4_fill_super() time by crypto_alloc_shash("crc32c", 0, 0).
> > 
> > Thus, shash->update should turn into a call to crypto/crc32c.c:chksum_update(),
> > which calls lib/crc32.c:__crc32c_le().
> > 
> > Now, I happen to be running an i7-2700k which has sse4_2, and thus calls
> > into the x86 specific code, and apparently for large blocks it uses PCLMULQDQ,
> > which requires kernel_fpu_begin/end.
> > 
> > At least that makes some degree of sense.  The low level code, though
> > uses the functions in a very simple way that I can't see how it could fail
> > to unlock at the end.
>   Hum, can you try disabling the HW support of CRC32C implementation
> (CRYPTO_CRC32C_INTEL)? If the problem disappears, we know there's some
> problem in the HW support code...
> 
> 								Honza
> -- 
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 3.11.4: kernel BUG at fs/buffer.c:1268

[George Spelvin]

>  Hum, can you try disabling the HW support of CRC32C implementation
> (CRYPTO_CRC32C_INTEL)? If the problem disappears, we know there's some
> problem in the HW support code...

To isolate it even better, I left in the hardware support, but commented
out the CLMUL code.  I could have just upped the threshold, but opted
for commenting it out completely.

commit 8ebf85f2c151e4568d948bda29f39396ccc11096
Author: George Spelvin <linux@horizon.com>
Date:   Tue Dec 10 21:06:40 2013 +0000

    Disable Intel crc32c PCLMULQ usage
    
    It's a great hack, but uses the FPU, and that appears to cause problems.

diff --git a/arch/x86/crypto/crc32c-intel_glue.c b/arch/x86/crypto/crc32c-intel_glue.c
index 0c8b502..e7b44fd 100644
--- a/arch/x86/crypto/crc32c-intel_glue.c
+++ b/arch/x86/crypto/crc32c-intel_glue.c
@@ -183,11 +183,11 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,
 	 * use faster PCL version if datasize is large enough to
 	 * overcome kernel fpu state save/restore overhead
 	 */
-	if (len >= crc32c_pcl_breakeven && irq_fpu_usable()) {
-		kernel_fpu_begin();
-		*crcp = crc_pcl(data, len, *crcp);
-		kernel_fpu_end();
-	} else
+//	if (len >= crc32c_pcl_breakeven && irq_fpu_usable()) {
+//		kernel_fpu_begin();
+//		*crcp = crc_pcl(data, len, *crcp);
+//		kernel_fpu_end();
+//	} else
 		*crcp = crc32c_intel_le_hw(*crcp, data, len);
 	return 0;
 }
@@ -195,11 +195,11 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,
 static int __crc32c_pcl_intel_finup(u32 *crcp, const u8 *data, unsigned int len,
 				u8 *out)
 {
-	if (len >= crc32c_pcl_breakeven && irq_fpu_usable()) {
-		kernel_fpu_begin();
-		*(__le32 *)out = ~cpu_to_le32(crc_pcl(data, len, *crcp));
-		kernel_fpu_end();
-	} else
+//	if (len >= crc32c_pcl_breakeven && irq_fpu_usable()) {
+//		kernel_fpu_begin();
+//		*(__le32 *)out = ~cpu_to_le32(crc_pcl(data, len, *crcp));
+//		kernel_fpu_end();
+//	} else
 		*(__le32 *)out =
 			~cpu_to_le32(crc32c_intel_le_hw(*crcp, data, len));
 	return 0;