linux-next: slab shrinkers: BUG at mm/list_lru.c:92

linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

Hi,
I managed to trigger:
[ 1015.776029] kernel BUG at mm/list_lru.c:92!
[ 1015.776029] invalid opcode: 0000 [#1] SMP
[ 1015.776029] Modules linked in: edd nfsv3 nfs_acl nfs fscache lockd sunrpc af_packet bridge stp llc cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop dm_mod ohci_pci ohci_hcd ehci_hcd usbcore e1000 kvm_amd kvm tg3 usb_common ptp pps_core sg shpchp edac_core pci_hotplug sr_mod k8temp i2c_amd8111 i2c_amd756 amd_rng edac_mce_amd button serio_raw cdrom pcspkr processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[ 1015.776029] CPU: 5 PID: 10480 Comm: cc1 Not tainted 3.10.0-rc4-next-20130607nextbadpagefix+ #1
[ 1015.776029] Hardware name: AMD A8440/WARTHOG, BIOS PW2A00-5 09/23/2005
[ 1015.776029] task: ffff8800327fc240 ti: ffff88003a59a000 task.ti: ffff88003a59a000
[ 1015.776029] RIP: 0010:[<ffffffff81122d9c>]  [<ffffffff81122d9c>] list_lru_walk_node+0x10c/0x140
[ 1015.776029] RSP: 0018:ffff88003a59b7a8  EFLAGS: 00010286
[ 1015.776029] RAX: ffffffffffffffff RBX: ffff880002f7ae80 RCX: ffff880002f7ae80
[ 1015.776029] RDX: 0000000000000000 RSI: ffff8800370dacc0 RDI: ffff880002f7ad88
[ 1015.776029] RBP: ffff88003a59b808 R08: 0000000000000000 R09: ffff88001ffeafc0
[ 1015.776029] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8800370dacc0
[ 1015.776029] R13: 0000000000000227 R14: ffff880002fb6850 R15: ffff8800370dacc8
[ 1015.776029] FS:  00002aaaaaada600(0000) GS:ffff88001f300000(0000) knlGS:0000000000000000
[ 1015.776029] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1015.776029] CR2: 00000000025aac5c CR3: 000000001cf9d000 CR4: 00000000000007e0
[ 1015.776029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1015.776029] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1015.776029] Stack:
[ 1015.776029]  ffff8800151c6440 ffff88003a59b820 ffff88003a59b828 ffffffff8117e4a0
[ 1015.776029]  000000008117e6e1 ffff88003e174c48 00ff88003a59b828 ffff88003a59b828
[ 1015.776029]  000000000000021f ffff88003e174800 ffff88003a59b9f8 0000000000000220
[ 1015.776029] Call Trace:
[ 1015.776029]  [<ffffffff8117e4a0>] ? insert_inode_locked+0x160/0x160
[ 1015.776029]  [<ffffffff8117e74c>] prune_icache_sb+0x3c/0x60
[ 1015.776029]  [<ffffffff81167a2e>] super_cache_scan+0x12e/0x1b0
[ 1015.776029]  [<ffffffff8110f3da>] shrink_slab_node+0x13a/0x250
[ 1015.776029]  [<ffffffff8111256b>] shrink_slab+0xab/0x120
[ 1015.776029]  [<ffffffff81113784>] do_try_to_free_pages+0x264/0x360
[ 1015.776029]  [<ffffffff81113bd0>] try_to_free_pages+0x130/0x180
[ 1015.776029]  [<ffffffff81106fce>] __alloc_pages_slowpath+0x39e/0x790
[ 1015.776029]  [<ffffffff811075ba>] __alloc_pages_nodemask+0x1fa/0x210
[ 1015.776029]  [<ffffffff81147470>] alloc_pages_vma+0xa0/0x120
[ 1015.776029]  [<ffffffff81124c33>] do_anonymous_page+0x133/0x300
[ 1015.776029]  [<ffffffff8112a10d>] handle_pte_fault+0x22d/0x240
[ 1015.776029]  [<ffffffff81122f58>] ? list_lru_add+0x68/0xe0
[ 1015.776029]  [<ffffffff8112a3e3>] handle_mm_fault+0x2c3/0x3e0
[ 1015.776029]  [<ffffffff815a4997>] __do_page_fault+0x227/0x4e0
[ 1015.776029]  [<ffffffff81002930>] ? do_notify_resume+0x90/0x1d0
[ 1015.776029]  [<ffffffff81163d18>] ? fsnotify_access+0x68/0x80
[ 1015.776029]  [<ffffffff811660a4>] ? file_sb_list_del+0x44/0x50
[ 1015.776029]  [<ffffffff81060b05>] ? task_work_add+0x55/0x70
[ 1015.776029]  [<ffffffff81166214>] ? fput+0x74/0xd0
[ 1015.776029]  [<ffffffff815a4c59>] do_page_fault+0x9/0x10
[ 1015.776029]  [<ffffffff815a1632>] page_fault+0x22/0x30
[ 1015.776029] Code: b3 66 0f 1f 44 00 00 48 8b 03 48 8b 53 08 48 89 50 08 48 89 02 49 8b 44 24 10 49 89 5c 24 10 4c 89 3b 48 89 43 08 48 89 18 eb 89 <0f> 0b eb fe 8b 55 c4 48 8b 45 c8 f0 0f b3 10 e9 69 ff ff ff 66
[ 1015.776029] RIP  [<ffffffff81122d9c>] list_lru_walk_node+0x10c/0x140

with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
on top. 

This is obviously BUG_ON(nlru->nr_items < 0) and 
ffffffff81122d0b:       48 85 c0                test   %rax,%rax
ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
[...]
ffffffff81122d9c:       0f 0b                   ud2

RAX is -1UL.

I assume that the current backtrace is of no use and it would most
probably be some shrinker which doesn't behave.

Any idea how to pin this down?

Thanks!
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Mon, Jun 17, 2013 at 04:18:22PM +0200, Michal Hocko wrote:
> Hi,

Hi,

> I managed to trigger:
> [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> [ 1015.776029] invalid opcode: 0000 [#1] SMP
> with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> on top. 
> 
> This is obviously BUG_ON(nlru->nr_items < 0) and 
> ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> [...]
> ffffffff81122d9c:       0f 0b                   ud2
> 
> RAX is -1UL.
Yes, fearing those kind of imbalances, we decided to leave the counter as a signed quantity
and BUG, instead of an unsigned quantity.

> 
> I assume that the current backtrace is of no use and it would most
> probably be some shrinker which doesn't behave.
> 
There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
Assuming you are not using xfs, we are left with dentries and inodes.

The first thing to do is to find which one of them is misbehaving. You can try finding
this out by the address of the list_lru, and where it lays in the superblock.

Once we know each of them is misbehaving, then we'll have to figure out why.

Any special filesystem workload ?

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Mon 17-06-13 19:14:12, Glauber Costa wrote:
> On Mon, Jun 17, 2013 at 04:18:22PM +0200, Michal Hocko wrote:
> > Hi,
> 
> Hi,
> 
> > I managed to trigger:
> > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > on top. 
> > 
> > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > [...]
> > ffffffff81122d9c:       0f 0b                   ud2
> > 
> > RAX is -1UL.
>
> Yes, fearing those kind of imbalances, we decided to leave the counter
> as a signed quantity and BUG, instead of an unsigned quantity.
> 
> > I assume that the current backtrace is of no use and it would most
> > probably be some shrinker which doesn't behave.
> > 
> There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> Assuming you are not using xfs, we are left with dentries and inodes.
> 
> The first thing to do is to find which one of them is misbehaving. You
> can try finding this out by the address of the list_lru, and where it
> lays in the superblock.

I am not sure I understand. Care to prepare a debugging patch for me?
 
> Once we know each of them is misbehaving, then we'll have to figure
> out why.
> 
> Any special filesystem workload ?

This is two parallel kernel builds with separate kernel trees running
under 2 hard unlimitted groups (with 0 soft limit) followed by rm -rf
source trees + drop caches. Sometimes I have to repeat this multiple
times. I can also see some timer specific crashes which are most
probably not related so I am getting back to my mm tree and will hope
the tree is healthy.

I have seen some other traces as well (mentioning ext3 dput paths) but I
cannot reproduce them anymore.

Thanks!
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Mon, Jun 17, 2013 at 05:33:02PM +0200, Michal Hocko wrote:
> On Mon 17-06-13 19:14:12, Glauber Costa wrote:
> > On Mon, Jun 17, 2013 at 04:18:22PM +0200, Michal Hocko wrote:
> > > Hi,
> > 
> > Hi,
> > 
> > > I managed to trigger:
> > > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > > on top. 
> > > 
> > > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > > [...]
> > > ffffffff81122d9c:       0f 0b                   ud2
> > > 
> > > RAX is -1UL.
> >
> > Yes, fearing those kind of imbalances, we decided to leave the counter
> > as a signed quantity and BUG, instead of an unsigned quantity.
> > 
> > > I assume that the current backtrace is of no use and it would most
> > > probably be some shrinker which doesn't behave.
> > > 
> > There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> > Assuming you are not using xfs, we are left with dentries and inodes.
> > 
> > The first thing to do is to find which one of them is misbehaving. You
> > can try finding this out by the address of the list_lru, and where it
> > lays in the superblock.
> 
> I am not sure I understand. Care to prepare a debugging patch for me?
>  
> > Once we know each of them is misbehaving, then we'll have to figure
> > out why.
> > 
> > Any special filesystem workload ?
> 
> This is two parallel kernel builds with separate kernel trees running
> under 2 hard unlimitted groups (with 0 soft limit) followed by rm -rf
> source trees + drop caches. Sometimes I have to repeat this multiple
> times. I can also see some timer specific crashes which are most
> probably not related so I am getting back to my mm tree and will hope
> the tree is healthy.
> 
> I have seen some other traces as well (mentioning ext3 dput paths) but I
> cannot reproduce them anymore.
> 

Do you have those traces? If there is a bug in the ext3 dput, then it is
most likely the culprit. dput() is when we insert things into the LRU. So
if we are not fully inserting an element that we should have - and later
on try to remove it, we'll go negative.

Can we see those traces?

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Mon 17-06-13 20:54:10, Glauber Costa wrote:
> On Mon, Jun 17, 2013 at 05:33:02PM +0200, Michal Hocko wrote:
[...]
> > I have seen some other traces as well (mentioning ext3 dput paths) but I
> > cannot reproduce them anymore.
> > 
> 
> Do you have those traces? If there is a bug in the ext3 dput, then it is
> most likely the culprit. dput() is when we insert things into the LRU. So
> if we are not fully inserting an element that we should have - and later
> on try to remove it, we'll go negative.
> 
> Can we see those traces?

Unfortunatelly I don't because the machine where I saw those didn't have
a serial console and the traces where scrolling like crazy. Anyway I am
working on reproducing this. Linux next is hard to debug due to
unrelated crashes so I am still with my -mm git tree.

Anyway, I was able to reproduce one of those hangs which smells like the
same/similar issue:
 4659 pts/0    S+     0:00                 /bin/sh ./run_batch.sh mmotm
 4661 pts/0    S+     0:00                   /bin/bash ./start.sh
 4666 pts/0    S+     5:08                     /bin/bash ./start.sh
18294 pts/0    S+     0:00                       sleep 1s
 4682 pts/0    S+     0:00                     /bin/bash ./run_test.sh /dev/cgroup B 2
 4683 pts/0    S+     5:16                       /bin/bash ./run_test.sh /dev/cgroup B 2
18293 pts/0    S+     0:00                         sleep 1s
 8509 pts/0    S+     0:00                       /usr/bin/time -v make -j4 vmlinux
 8510 pts/0    S+     0:00                         make -j4 vmlinux
11730 pts/0    S+     0:00                           make -f scripts/Makefile.build obj=drivers
13135 pts/0    S+     0:00                             make -f scripts/Makefile.build obj=drivers/net
13415 pts/0    S+     0:00                               make -f scripts/Makefile.build obj=drivers/net/wireless
13657 pts/0    S+     0:00                                 make -f scripts/Makefile.build obj=drivers/net/wireless/rtl818x
13665 pts/0    D+     0:00                                   make -f scripts/Makefile.build obj=drivers/net/wireless/rtl818x/rtl8180
13737 pts/0    S+     0:00                                 make -f scripts/Makefile.build obj=drivers/net/wireless/rtlwifi
13754 pts/0    D+     0:00                                   make -f scripts/Makefile.build obj=drivers/net/wireless/rtlwifi/rtl8192de
13917 pts/0    D+     0:00                                   make -f scripts/Makefile.build obj=drivers/net/wireless/rtlwifi/rtl8192se

demon:/home/mhocko # cat /proc/13917/stack 
[<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

demon:/home/mhocko # cat /proc/13754/stack 
[<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118526f>] iget_locked+0x4f/0x180
[<ffffffff811ef9f3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a2c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81175254>] __lookup_hash+0x34/0x40
[<ffffffff81179872>] path_lookupat+0x7a2/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

demon:/home/mhocko # cat /proc/13665/stack 
[<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118526f>] iget_locked+0x4f/0x180
[<ffffffff811ef9f3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a2c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

Sysrq+l doesn't show only idle CPUs. Ext4 is showing in the traces
because of CONFIG_EXT4_USE_FOR_EXT23=y.
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Andrew Morton]

On Mon, 17 Jun 2013 19:14:12 +0400 Glauber Costa <glommer@gmail.com> wrote:

> > I managed to trigger:
> > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > on top. 
> > 
> > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > [...]
> > ffffffff81122d9c:       0f 0b                   ud2
> > 
> > RAX is -1UL.
> Yes, fearing those kind of imbalances, we decided to leave the counter as a signed quantity
> and BUG, instead of an unsigned quantity.
> 
> > 
> > I assume that the current backtrace is of no use and it would most
> > probably be some shrinker which doesn't behave.
> > 
> There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> Assuming you are not using xfs, we are left with dentries and inodes.
> 
> The first thing to do is to find which one of them is misbehaving. You can try finding
> this out by the address of the list_lru, and where it lays in the superblock.
> 
> Once we know each of them is misbehaving, then we'll have to figure out why.

The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
it's inodes?

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

[-- Attachment #1: Type: text/plain, Size: 3848 bytes --]

On Mon, Jun 17, 2013 at 02:35:08PM -0700, Andrew Morton wrote:
> On Mon, 17 Jun 2013 19:14:12 +0400 Glauber Costa <glommer@gmail.com> wrote:
> 
> > > I managed to trigger:
> > > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > > on top. 
> > > 
> > > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > > [...]
> > > ffffffff81122d9c:       0f 0b                   ud2
> > > 
> > > RAX is -1UL.
> > Yes, fearing those kind of imbalances, we decided to leave the counter as a signed quantity
> > and BUG, instead of an unsigned quantity.
> > 
> > > 
> > > I assume that the current backtrace is of no use and it would most
> > > probably be some shrinker which doesn't behave.
> > > 
> > There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> > Assuming you are not using xfs, we are left with dentries and inodes.
> > 
> > The first thing to do is to find which one of them is misbehaving. You can try finding
> > this out by the address of the list_lru, and where it lays in the superblock.
> > 
> > Once we know each of them is misbehaving, then we'll have to figure out why.
> 
> The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
> it's inodes?
> 
Assuming there is no memory corruption of any sort going on , let's check the code.
nr_item is only manipulated in 3 places:

1) list_lru_add, where it is increased
2) list_lru_del, where it is decreased in case the user have voluntarily removed the
   element from the list
3) list_lru_walk_node, where an element is removing during shrink.

All three excerpts seem to be correctly locked, so something like this indicates an imbalance.
Either the element was never added to the list, or it was added, removed, and we didn't notice
it. (Again, your backing storage is not XFS, is it? If it is , we have another user to look for)

I will assume that Andrew is correct and this is inode related. list_lru_del reads as follows:
        spin_lock(&nlru->lock);
        if (!list_empty(item)) { ... }

So one possibility is that we are manipulating this list outside this lock somewhere. Going to
inode.c... We always manipulate the LRU inside the lock, but the element is not always in the LRU,
if it is in a list. Could it be possible that the element is in the dispose_list, and at the same
time someone calls list_lru_del at it, creating the imbalance ?

callers:
iput_final, evict_inodes, invalidate_inodes.
Both evict_inodes and invalidate_inodes will do the following pattern:

                inode->i_state |= I_FREEING;                                            
                inode_lru_list_del(inode);
                spin_unlock(&inode->i_lock);
                list_add(&inode->i_lru, &dispose);

IOW, they will remove the element from the LRU, and add it to the dispose list.
Both of them will also bail out if they see I_FREEING already set, so they are safe
against each other - because the flag is manipulated inside the lock.

But how about iput_final? It seems to me that if we are calling iput_final at the
same time as the other two, this *could* happen (maybe there is some extra protection
that can be seen from Australia but not from here. Dave?)

Right now this is my best theory.

I am attaching a patch that should make a difference in case I am right.





[-- Attachment #2: inode.patch --]
[-- Type: text/plain, Size: 636 bytes --]

diff --git a/fs/inode.c b/fs/inode.c
index 00b804e..c46c92e 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -419,6 +419,8 @@ void inode_add_lru(struct inode *inode)
 
 static void inode_lru_list_del(struct inode *inode)
 {
+	if (inode->i_state & I_FREEING)
+		return;
 
 	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
 		this_cpu_dec(nr_unused);
@@ -1381,9 +1383,8 @@ static void iput_final(struct inode *inode)
 		inode->i_state &= ~I_WILL_FREE;
 	}
 
+	inode_lru_list_del(inode);
 	inode->i_state |= I_FREEING;
-	if (!list_empty(&inode->i_lru))
-		inode_lru_list_del(inode);
 	spin_unlock(&inode->i_lock);
 
 	evict(inode);

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Tue, Jun 18, 2013 at 02:30:05AM +0400, Glauber Costa wrote:
> On Mon, Jun 17, 2013 at 02:35:08PM -0700, Andrew Morton wrote:
> > On Mon, 17 Jun 2013 19:14:12 +0400 Glauber Costa <glommer@gmail.com> wrote:
> > 
> > > > I managed to trigger:
> > > > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > > > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > > > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > > > on top. 
> > > > 
> > > > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > > > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > > > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > > > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > > > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > > > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > > > [...]
> > > > ffffffff81122d9c:       0f 0b                   ud2
> > > > 
> > > > RAX is -1UL.
> > > Yes, fearing those kind of imbalances, we decided to leave the counter as a signed quantity
> > > and BUG, instead of an unsigned quantity.
> > > 
> > > > 
> > > > I assume that the current backtrace is of no use and it would most
> > > > probably be some shrinker which doesn't behave.
> > > > 
> > > There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> > > Assuming you are not using xfs, we are left with dentries and inodes.
> > > 
> > > The first thing to do is to find which one of them is misbehaving. You can try finding
> > > this out by the address of the list_lru, and where it lays in the superblock.
> > > 
> > > Once we know each of them is misbehaving, then we'll have to figure out why.
> > 
> > The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
> > it's inodes?
> > 
> Assuming there is no memory corruption of any sort going on , let's check the code.
> nr_item is only manipulated in 3 places:
> 
> 1) list_lru_add, where it is increased
> 2) list_lru_del, where it is decreased in case the user have voluntarily removed the
>    element from the list
> 3) list_lru_walk_node, where an element is removing during shrink.
> 
> All three excerpts seem to be correctly locked, so something like this indicates an imbalance.

inode_lru_isolate() looks suspicious to me:

        WARN_ON(inode->i_state & I_NEW);
        inode->i_state |= I_FREEING;
        spin_unlock(&inode->i_lock);

        list_move(&inode->i_lru, freeable);
        this_cpu_dec(nr_unused);
	return LRU_REMOVED;
}

All the other cases where I_FREEING is set and the inode is removed
from the LRU are completely done under the inode->i_lock. i.e. from
an external POV, the state change to I_FREEING and removal from LRU
are supposed to be atomic, but they are not here.

I'm not sure this is the source of the problem, but it definitely
needs fixing.

> callers:
> iput_final, evict_inodes, invalidate_inodes.
> Both evict_inodes and invalidate_inodes will do the following pattern:
> 
>                 inode->i_state |= I_FREEING;                                            
>                 inode_lru_list_del(inode);
>                 spin_unlock(&inode->i_lock);
>                 list_add(&inode->i_lru, &dispose);
> 
> IOW, they will remove the element from the LRU, and add it to the dispose list.
> Both of them will also bail out if they see I_FREEING already set, so they are safe
> against each other - because the flag is manipulated inside the lock.
> 
> But how about iput_final? It seems to me that if we are calling iput_final at the
> same time as the other two, this *could* happen (maybe there is some extra protection
> that can be seen from Australia but not from here. Dave?)

If I_FREEING is set before we enter iput_final(), then something
else is screwed up. I_FREEING is only set once the last reference
has gone away and we are killing the inode. All the other callers
that set I_FREEING check that the reference count on the inode is
zero before they set I_FREEING. Hence I_FREEING cannot be set on the
transition of i_count from 1 to 0 when iput_final() is called. So
the patch won't do anything to avoid the problem being seen.

Keep in mind that we this is actually a new warning on the count of
inodes on the LRU - we never had a check that it didn't go negative
before....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Tue, Jun 18, 2013 at 12:46:23PM +1000, Dave Chinner wrote:
> On Tue, Jun 18, 2013 at 02:30:05AM +0400, Glauber Costa wrote:
> > On Mon, Jun 17, 2013 at 02:35:08PM -0700, Andrew Morton wrote:
> > > On Mon, 17 Jun 2013 19:14:12 +0400 Glauber Costa <glommer@gmail.com> wrote:
> > > 
> > > > > I managed to trigger:
> > > > > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > > > > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > > > > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > > > > on top. 
> > > > > 
> > > > > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > > > > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > > > > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > > > > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > > > > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > > > > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > > > > [...]
> > > > > ffffffff81122d9c:       0f 0b                   ud2
> > > > > 
> > > > > RAX is -1UL.
> > > > Yes, fearing those kind of imbalances, we decided to leave the counter as a signed quantity
> > > > and BUG, instead of an unsigned quantity.
> > > > 
> > > > > 
> > > > > I assume that the current backtrace is of no use and it would most
> > > > > probably be some shrinker which doesn't behave.
> > > > > 
> > > > There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> > > > Assuming you are not using xfs, we are left with dentries and inodes.
> > > > 
> > > > The first thing to do is to find which one of them is misbehaving. You can try finding
> > > > this out by the address of the list_lru, and where it lays in the superblock.
> > > > 
> > > > Once we know each of them is misbehaving, then we'll have to figure out why.
> > > 
> > > The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
> > > it's inodes?
> > > 
> > Assuming there is no memory corruption of any sort going on , let's check the code.
> > nr_item is only manipulated in 3 places:
> > 
> > 1) list_lru_add, where it is increased
> > 2) list_lru_del, where it is decreased in case the user have voluntarily removed the
> >    element from the list
> > 3) list_lru_walk_node, where an element is removing during shrink.
> > 
> > All three excerpts seem to be correctly locked, so something like this indicates an imbalance.
> 
> inode_lru_isolate() looks suspicious to me:
> 
>         WARN_ON(inode->i_state & I_NEW);
>         inode->i_state |= I_FREEING;
>         spin_unlock(&inode->i_lock);
> 
>         list_move(&inode->i_lru, freeable);
>         this_cpu_dec(nr_unused);
> 	return LRU_REMOVED;
> }
> 
> All the other cases where I_FREEING is set and the inode is removed
> from the LRU are completely done under the inode->i_lock. i.e. from
> an external POV, the state change to I_FREEING and removal from LRU
> are supposed to be atomic, but they are not here.
> 
> I'm not sure this is the source of the problem, but it definitely
> needs fixing.
> 
Yes, I missed that yesterday, but that does look suspicious to me as well.

Michal, if you can manually move this one inside the lock as well and see
if it fixes your problem as well... Otherwise I can send you a patch as well
so we don't get lost on what is patched and what is not.

Let us at least know if this is the problem.

> > callers:
> > iput_final, evict_inodes, invalidate_inodes.
> > Both evict_inodes and invalidate_inodes will do the following pattern:
> > 
> >                 inode->i_state |= I_FREEING;                                            
> >                 inode_lru_list_del(inode);
> >                 spin_unlock(&inode->i_lock);
> >                 list_add(&inode->i_lru, &dispose);
> > 
> > IOW, they will remove the element from the LRU, and add it to the dispose list.
> > Both of them will also bail out if they see I_FREEING already set, so they are safe
> > against each other - because the flag is manipulated inside the lock.
> > 
> > But how about iput_final? It seems to me that if we are calling iput_final at the
> > same time as the other two, this *could* happen (maybe there is some extra protection
> > that can be seen from Australia but not from here. Dave?)
> 
> If I_FREEING is set before we enter iput_final(), then something
> else is screwed up. I_FREEING is only set once the last reference
> has gone away and we are killing the inode. All the other callers
> that set I_FREEING check that the reference count on the inode is
> zero before they set I_FREEING. Hence I_FREEING cannot be set on the
> transition of i_count from 1 to 0 when iput_final() is called. So
> the patch won't do anything to avoid the problem being seen.
> 
Yes, but isn't things like evict_inodes and invalidate_inodes called at
umount time, for instance? Can't it be that we drop the last reference
to a valid in use inode while someone else is invalidating them all?

> Keep in mind that we this is actually a new warning on the count of
> inodes on the LRU - we never had a check that it didn't go negative
> before....
>
 

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 18-06-13 10:31:05, Glauber Costa wrote:
> On Tue, Jun 18, 2013 at 12:46:23PM +1000, Dave Chinner wrote:
> > On Tue, Jun 18, 2013 at 02:30:05AM +0400, Glauber Costa wrote:
> > > On Mon, Jun 17, 2013 at 02:35:08PM -0700, Andrew Morton wrote:
> > > > On Mon, 17 Jun 2013 19:14:12 +0400 Glauber Costa <glommer@gmail.com> wrote:
> > > > 
> > > > > > I managed to trigger:
> > > > > > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > > > > > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > > > > > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > > > > > on top. 
> > > > > > 
> > > > > > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > > > > > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > > > > > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > > > > > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > > > > > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > > > > > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > > > > > [...]
> > > > > > ffffffff81122d9c:       0f 0b                   ud2
> > > > > > 
> > > > > > RAX is -1UL.
> > > > > Yes, fearing those kind of imbalances, we decided to leave the counter as a signed quantity
> > > > > and BUG, instead of an unsigned quantity.
> > > > > 
> > > > > > 
> > > > > > I assume that the current backtrace is of no use and it would most
> > > > > > probably be some shrinker which doesn't behave.
> > > > > > 
> > > > > There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> > > > > Assuming you are not using xfs, we are left with dentries and inodes.
> > > > > 
> > > > > The first thing to do is to find which one of them is misbehaving. You can try finding
> > > > > this out by the address of the list_lru, and where it lays in the superblock.
> > > > > 
> > > > > Once we know each of them is misbehaving, then we'll have to figure out why.
> > > > 
> > > > The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
> > > > it's inodes?
> > > > 
> > > Assuming there is no memory corruption of any sort going on , let's check the code.
> > > nr_item is only manipulated in 3 places:
> > > 
> > > 1) list_lru_add, where it is increased
> > > 2) list_lru_del, where it is decreased in case the user have voluntarily removed the
> > >    element from the list
> > > 3) list_lru_walk_node, where an element is removing during shrink.
> > > 
> > > All three excerpts seem to be correctly locked, so something like this indicates an imbalance.
> > 
> > inode_lru_isolate() looks suspicious to me:
> > 
> >         WARN_ON(inode->i_state & I_NEW);
> >         inode->i_state |= I_FREEING;
> >         spin_unlock(&inode->i_lock);
> > 
> >         list_move(&inode->i_lru, freeable);
> >         this_cpu_dec(nr_unused);
> > 	return LRU_REMOVED;
> > }
> > 
> > All the other cases where I_FREEING is set and the inode is removed
> > from the LRU are completely done under the inode->i_lock. i.e. from
> > an external POV, the state change to I_FREEING and removal from LRU
> > are supposed to be atomic, but they are not here.
> > 
> > I'm not sure this is the source of the problem, but it definitely
> > needs fixing.
> > 
> Yes, I missed that yesterday, but that does look suspicious to me as well.
> 
> Michal, if you can manually move this one inside the lock as well and see
> if it fixes your problem as well... Otherwise I can send you a patch as well
> so we don't get lost on what is patched and what is not.

OK, I am testing with this now:
diff --git a/fs/inode.c b/fs/inode.c
index 604c15e..95e598c 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -733,9 +733,9 @@ inode_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
 
 	WARN_ON(inode->i_state & I_NEW);
 	inode->i_state |= I_FREEING;
+	list_move(&inode->i_lru, freeable);
 	spin_unlock(&inode->i_lock);
 
-	list_move(&inode->i_lru, freeable);
 	this_cpu_dec(nr_unused);
 	return LRU_REMOVED;
 }

> Let us at least know if this is the problem.
> 
> > > callers:
> > > iput_final, evict_inodes, invalidate_inodes.
> > > Both evict_inodes and invalidate_inodes will do the following pattern:
> > > 
> > >                 inode->i_state |= I_FREEING;                                            
> > >                 inode_lru_list_del(inode);
> > >                 spin_unlock(&inode->i_lock);
> > >                 list_add(&inode->i_lru, &dispose);
> > > 
> > > IOW, they will remove the element from the LRU, and add it to the dispose list.
> > > Both of them will also bail out if they see I_FREEING already set, so they are safe
> > > against each other - because the flag is manipulated inside the lock.
> > > 
> > > But how about iput_final? It seems to me that if we are calling iput_final at the
> > > same time as the other two, this *could* happen (maybe there is some extra protection
> > > that can be seen from Australia but not from here. Dave?)
> > 
> > If I_FREEING is set before we enter iput_final(), then something
> > else is screwed up. I_FREEING is only set once the last reference
> > has gone away and we are killing the inode. All the other callers
> > that set I_FREEING check that the reference count on the inode is
> > zero before they set I_FREEING. Hence I_FREEING cannot be set on the
> > transition of i_count from 1 to 0 when iput_final() is called. So
> > the patch won't do anything to avoid the problem being seen.
> > 
> Yes, but isn't things like evict_inodes and invalidate_inodes called at
> umount time, for instance?

JFYI No unmount is going on in my test case.
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 18-06-13 10:24:14, Michal Hocko wrote:
> On Tue 18-06-13 10:31:05, Glauber Costa wrote:
> > On Tue, Jun 18, 2013 at 12:46:23PM +1000, Dave Chinner wrote:
> > > On Tue, Jun 18, 2013 at 02:30:05AM +0400, Glauber Costa wrote:
> > > > On Mon, Jun 17, 2013 at 02:35:08PM -0700, Andrew Morton wrote:
> > > > > On Mon, 17 Jun 2013 19:14:12 +0400 Glauber Costa <glommer@gmail.com> wrote:
> > > > > 
> > > > > > > I managed to trigger:
> > > > > > > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > > > > > > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > > > > > > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > > > > > > on top. 
> > > > > > > 
> > > > > > > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > > > > > > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > > > > > > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > > > > > > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > > > > > > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > > > > > > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > > > > > > [...]
> > > > > > > ffffffff81122d9c:       0f 0b                   ud2
> > > > > > > 
> > > > > > > RAX is -1UL.
> > > > > > Yes, fearing those kind of imbalances, we decided to leave the counter as a signed quantity
> > > > > > and BUG, instead of an unsigned quantity.
> > > > > > 
> > > > > > > 
> > > > > > > I assume that the current backtrace is of no use and it would most
> > > > > > > probably be some shrinker which doesn't behave.
> > > > > > > 
> > > > > > There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> > > > > > Assuming you are not using xfs, we are left with dentries and inodes.
> > > > > > 
> > > > > > The first thing to do is to find which one of them is misbehaving. You can try finding
> > > > > > this out by the address of the list_lru, and where it lays in the superblock.
> > > > > > 
> > > > > > Once we know each of them is misbehaving, then we'll have to figure out why.
> > > > > 
> > > > > The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
> > > > > it's inodes?
> > > > > 
> > > > Assuming there is no memory corruption of any sort going on , let's check the code.
> > > > nr_item is only manipulated in 3 places:
> > > > 
> > > > 1) list_lru_add, where it is increased
> > > > 2) list_lru_del, where it is decreased in case the user have voluntarily removed the
> > > >    element from the list
> > > > 3) list_lru_walk_node, where an element is removing during shrink.
> > > > 
> > > > All three excerpts seem to be correctly locked, so something like this indicates an imbalance.
> > > 
> > > inode_lru_isolate() looks suspicious to me:
> > > 
> > >         WARN_ON(inode->i_state & I_NEW);
> > >         inode->i_state |= I_FREEING;
> > >         spin_unlock(&inode->i_lock);
> > > 
> > >         list_move(&inode->i_lru, freeable);
> > >         this_cpu_dec(nr_unused);
> > > 	return LRU_REMOVED;
> > > }
> > > 
> > > All the other cases where I_FREEING is set and the inode is removed
> > > from the LRU are completely done under the inode->i_lock. i.e. from
> > > an external POV, the state change to I_FREEING and removal from LRU
> > > are supposed to be atomic, but they are not here.
> > > 
> > > I'm not sure this is the source of the problem, but it definitely
> > > needs fixing.
> > > 
> > Yes, I missed that yesterday, but that does look suspicious to me as well.
> > 
> > Michal, if you can manually move this one inside the lock as well and see
> > if it fixes your problem as well... Otherwise I can send you a patch as well
> > so we don't get lost on what is patched and what is not.
> 
> OK, I am testing with this now:
> diff --git a/fs/inode.c b/fs/inode.c
> index 604c15e..95e598c 100644
> --- a/fs/inode.c
> +++ b/fs/inode.c
> @@ -733,9 +733,9 @@ inode_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
>  
>  	WARN_ON(inode->i_state & I_NEW);
>  	inode->i_state |= I_FREEING;
> +	list_move(&inode->i_lru, freeable);
>  	spin_unlock(&inode->i_lock);
>  
> -	list_move(&inode->i_lru, freeable);
>  	this_cpu_dec(nr_unused);
>  	return LRU_REMOVED;
>  }

And this hung again:
 4434 pts/0    S+     0:00                 /bin/sh ./run_batch.sh mmotmdebug
 4436 pts/0    S+     0:00                   /bin/bash ./start.sh
 4441 pts/0    S+     0:26                     /bin/bash ./start.sh
 1919 pts/0    S+     0:00                       sleep 1s
 4457 pts/0    S+     0:00                     /bin/bash ./run_test.sh /dev/cgroup A 2
 4459 pts/0    S+     0:27                       /bin/bash ./run_test.sh /dev/cgroup A 2
 1913 pts/0    S+     0:00                         sleep 1s
 5626 pts/0    S+     0:00                       /usr/bin/time -v make -j4 vmlinux
 5628 pts/0    S+     0:00                         make -j4 vmlinux
 2676 pts/0    S+     0:00                           make -f scripts/Makefile.build obj=sound
 2893 pts/0    S+     0:00                             make -f scripts/Makefile.build obj=sound/pci
 2998 pts/0    D+     0:00                               make -f scripts/Makefile.build obj=sound/pci/emu10k1
 6590 pts/0    D+     0:00                           make -f scripts/Makefile.build obj=net
 4458 pts/0    S+     0:00                     /bin/bash ./run_test.sh /dev/cgroup B 2
 4464 pts/0    S+     0:27                       /bin/bash ./run_test.sh /dev/cgroup B 2
 1914 pts/0    S+     0:00                         sleep 1s
 5625 pts/0    S+     0:00                       /usr/bin/time -v make -j4 vmlinux
 5627 pts/0    S+     0:00                         make -j4 vmlinux
13010 pts/0    D+     0:00                           make -f scripts/Makefile.build obj=kernel
 3933 pts/0    Z+     0:00                             [sh] <defunct>
14459 pts/0    S+     0:00                           make -f scripts/Makefile.build obj=fs
  784 pts/0    D+     0:00                             make -f scripts/Makefile.build obj=fs/romfs
 2401 pts/0    S+     0:00                           make -f scripts/Makefile.build obj=crypto
 4614 pts/0    D+     0:00                             make -f scripts/Makefile.build obj=crypto/asymmetric_keys
 3343 pts/0    S+     0:00                           make -f scripts/Makefile.build obj=block
 5167 pts/0    D+     0:00                             make -f scripts/Makefile.build obj=block/partitions

demon:/home/mhocko # cat /proc/2998/stack 
[<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118525f>] iget_locked+0x4f/0x180
[<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
demon:/home/mhocko # cat /proc/6590/stack 
[<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118525f>] iget_locked+0x4f/0x180
[<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81175254>] __lookup_hash+0x34/0x40
[<ffffffff81179872>] path_lookupat+0x7a2/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
demon:/home/mhocko # cat /proc/13010/stack 
[<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118525f>] iget_locked+0x4f/0x180
[<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81175254>] __lookup_hash+0x34/0x40
[<ffffffff81179872>] path_lookupat+0x7a2/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
demon:/home/mhocko # cat /proc/784/stack 
[<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118525f>] iget_locked+0x4f/0x180
[<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81175254>] __lookup_hash+0x34/0x40
[<ffffffff81179872>] path_lookupat+0x7a2/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
demon:/home/mhocko # cat /proc/4614/stack 
[<ffffffff8117d0ca>] vfs_readdir+0x7a/0xc0
[<ffffffff8117d1a6>] sys_getdents64+0x96/0x100
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
demon:/home/mhocko # cat /proc/5167/stack 
[<ffffffff8117d0ca>] vfs_readdir+0x7a/0xc0
[<ffffffff8117d1a6>] sys_getdents64+0x96/0x100
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

JFYI: This is still with my -mm git tree + the above diff + referenced
patch from Mel (mm: Clear page active before releasing pages).
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

And again, another hang. It looks like the inode deletion never
finishes. The good thing is that I do not see any LRU related BUG_ONs
anymore. I am going to test with the other patch in the thread.

2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118525f>] iget_locked+0x4f/0x180
[<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
11214 [<ffffffff81178144>] do_last+0x2c4/0x780			<<< blocked on i_mutex
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
11217 [<ffffffff81178144>] do_last+0x2c4/0x780			<<< blocked on i_mutex
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
11288 [<ffffffff81178144>] do_last+0x2c4/0x780			<<< blocked on i_mutex
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
11453 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118525f>] iget_locked+0x4f/0x180
[<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
12439 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118525f>] iget_locked+0x4f/0x180
[<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81175254>] __lookup_hash+0x34/0x40
[<ffffffff81179872>] path_lookupat+0x7a2/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
12542 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
12588 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
12589 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
13098 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118525f>] iget_locked+0x4f/0x180
[<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
19091 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
19092 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> And again, another hang. It looks like the inode deletion never
> finishes. The good thing is that I do not see any LRU related BUG_ONs
> anymore. I am going to test with the other patch in the thread.
> 
> 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> [<ffffffff8118525f>] iget_locked+0x4f/0x180
> [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> [<ffffffff81174ad0>] lookup_real+0x20/0x60
> [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff

I don't think this has anything to do with LRUs.

__wait_on_freeing_inode() only blocks once the inode is being freed
(i.e. I_FREEING is set), and that happens when a lookup is done when
the inode is still in the inode hash.

I_FREEING is set on the inode at the same time it is removed from
the LRU, and from that point onwards the LRUs play no part in the
inode being freed and anyone waiting on the inode being freed
getting woken.

The only way I can see this happening, is if there is a dispose list
that is not getting processed properly. e.g., we move a bunch on
inodes to the dispose list setting I_FREEING, then for some reason
it gets dropped on the ground and so the wakeup call doesn't happen
when the inode has been removed from the hash.

I can't see anywhere in the code that this happens, though, but it
might be some pre-existing race in the inode hash that you are now
triggering because freeing will be happening in parallel on multiple
nodes rather than serialising on a global lock...

I won't have seen this on XFS stress testing, because it doesn't use
the VFS inode hashes for inode lookups. Given that XFS is not
triggering either problem you are seeing, that makes me think
that it might be a pre-existing inode hash lookup/reclaim race
condition, not a LRU problem.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > And again, another hang. It looks like the inode deletion never
> > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > anymore. I am going to test with the other patch in the thread.
> > 
> > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> 
> I don't think this has anything to do with LRUs.

I am not claiming that. It might be a timing issue which never mattered
but it is strange I can reproduce this so easily and repeatedly with the
shrinkers patchset applied.
As I said earlier, this might be breakage in my -mm tree as well
(missing some patch which didn't go via Andrew or misapplied patch). The
situation is worsen by the state of linux-next which has some unrelated
issues.

I really do not want to delay the whole patchset just because of some
problem on my side. Do you have any tree that I should try to test?

> __wait_on_freeing_inode() only blocks once the inode is being freed
> (i.e. I_FREEING is set), and that happens when a lookup is done when
> the inode is still in the inode hash.
> 
> I_FREEING is set on the inode at the same time it is removed from
> the LRU, and from that point onwards the LRUs play no part in the
> inode being freed and anyone waiting on the inode being freed
> getting woken.
> 
> The only way I can see this happening, is if there is a dispose list
> that is not getting processed properly. e.g., we move a bunch on
> inodes to the dispose list setting I_FREEING, then for some reason
> it gets dropped on the ground and so the wakeup call doesn't happen
> when the inode has been removed from the hash.
> 
> I can't see anywhere in the code that this happens, though, but it
> might be some pre-existing race in the inode hash that you are now
> triggering because freeing will be happening in parallel on multiple
> nodes rather than serialising on a global lock...
> 
> I won't have seen this on XFS stress testing, because it doesn't use
> the VFS inode hashes for inode lookups. Given that XFS is not
> triggering either problem you are seeing, that makes me think

I haven't tested with xfs.

> that it might be a pre-existing inode hash lookup/reclaim race
> condition, not a LRU problem.
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Wed, Jun 26, 2013 at 10:15:09AM +0200, Michal Hocko wrote:
> On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > > And again, another hang. It looks like the inode deletion never
> > > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > > anymore. I am going to test with the other patch in the thread.
> > > 
> > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > > [<ffffffffffffffff>] 0xffffffffffffffff
> > 
> > I don't think this has anything to do with LRUs.
> 
> I am not claiming that. It might be a timing issue which never mattered
> but it is strange I can reproduce this so easily and repeatedly with the
> shrinkers patchset applied.
> As I said earlier, this might be breakage in my -mm tree as well
> (missing some patch which didn't go via Andrew or misapplied patch). The
> situation is worsen by the state of linux-next which has some unrelated
> issues.
> 
> I really do not want to delay the whole patchset just because of some
> problem on my side. Do you have any tree that I should try to test?

No, I've just been testing Glauber's tree and sending patches for
problems back to him based on it.

> > I won't have seen this on XFS stress testing, because it doesn't use
> > the VFS inode hashes for inode lookups. Given that XFS is not
> > triggering either problem you are seeing, that makes me think
> 
> I haven't tested with xfs.

That might be worthwhile if you can easily do that - another data
point indicating a hang or absence of a hang will help point us in
the right direction here...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Thu 27-06-13 09:24:26, Dave Chinner wrote:
> On Wed, Jun 26, 2013 at 10:15:09AM +0200, Michal Hocko wrote:
> > On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> > > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > > > And again, another hang. It looks like the inode deletion never
> > > > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > > > anymore. I am going to test with the other patch in the thread.
> > > > 
> > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > > > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > > > [<ffffffffffffffff>] 0xffffffffffffffff
> > > 
> > > I don't think this has anything to do with LRUs.
> > 
> > I am not claiming that. It might be a timing issue which never mattered
> > but it is strange I can reproduce this so easily and repeatedly with the
> > shrinkers patchset applied.
> > As I said earlier, this might be breakage in my -mm tree as well
> > (missing some patch which didn't go via Andrew or misapplied patch). The
> > situation is worsen by the state of linux-next which has some unrelated
> > issues.
> > 
> > I really do not want to delay the whole patchset just because of some
> > problem on my side. Do you have any tree that I should try to test?
> 
> No, I've just been testing Glauber's tree and sending patches for
> problems back to him based on it.
> 
> > > I won't have seen this on XFS stress testing, because it doesn't use
> > > the VFS inode hashes for inode lookups. Given that XFS is not
> > > triggering either problem you are seeing, that makes me think
> > 
> > I haven't tested with xfs.
> 
> That might be worthwhile if you can easily do that - another data
> point indicating a hang or absence of a hang will help point us in
> the right direction here...

OK, still hanging (with inode_lru_isolate-fix.patch). It is not the same
thing, though, as xfs seem to do lookup slightly differently.
12467 [<ffffffffa02ca03e>] xfs_iget+0xbe/0x190 [xfs]
[<ffffffffa02d6e98>] xfs_lookup+0xe8/0x110 [xfs]
[<ffffffffa02cdad9>] xfs_vn_lookup+0x49/0x90 [xfs]
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
12667 [<ffffffffa02ca03e>] xfs_iget+0xbe/0x190 [xfs]
[<ffffffffa02d6e98>] xfs_lookup+0xe8/0x110 [xfs]
[<ffffffffa02cdad9>] xfs_vn_lookup+0x49/0x90 [xfs]
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
13830 [<ffffffffa02ca03e>] xfs_iget+0xbe/0x190 [xfs]
[<ffffffffa02d6e98>] xfs_lookup+0xe8/0x110 [xfs]
[<ffffffffa02cdad9>] xfs_vn_lookup+0x49/0x90 [xfs]
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81175254>] __lookup_hash+0x34/0x40
[<ffffffff81179872>] path_lookupat+0x7a2/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff81169c30>] sys_faccessat+0xe0/0x230
[<ffffffff81169d93>] sys_access+0x13/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
13913 [<ffffffffa02ca03e>] xfs_iget+0xbe/0x190 [xfs]
[<ffffffffa02d6e98>] xfs_lookup+0xe8/0x110 [xfs]
[<ffffffffa02cdad9>] xfs_vn_lookup+0x49/0x90 [xfs]
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
14245 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
14365 [<ffffffffa02ca03e>] xfs_iget+0xbe/0x190 [xfs]
[<ffffffffa02d6e98>] xfs_lookup+0xe8/0x110 [xfs]
[<ffffffffa02cdad9>] xfs_vn_lookup+0x49/0x90 [xfs]
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
14384 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
14413 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

I have just triggered this one.

[37955.354041] BUG: unable to handle kernel paging request at 000000572ead7838
[37955.356032] IP: [<ffffffff81127e5b>] list_lru_walk_node+0xab/0x140
[37955.364062] PGD 2bf0a067 PUD 0 
[37955.364062] Oops: 0000 [#1] SMP 
[37955.364062] Modules linked in: edd nfsv3 nfs_acl nfs fscache lockd sunrpc af_packet bridge stp llc cpufreq_conservative cpufreq_userspace cpufreq_powersave fuse xfs libcrc32c loop dm_mod tg3 ptp powernow_k8 pps_core e1000 kvm_amd shpchp kvm edac_core i2c_amd756 pci_hotplug i2c_amd8111 sg edac_mce_amd amd_rng k8temp sr_mod pcspkr cdrom serio_raw button ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[37955.364062] CPU 3 
[37955.364062] Pid: 3351, comm: as Not tainted 3.9.0mmotm+ #1490 AMD A8440/WARTHOG
[37955.364062] RIP: 0010:[<ffffffff81127e5b>]  [<ffffffff81127e5b>] list_lru_walk_node+0xab/0x140
[37955.364062] RSP: 0000:ffff8800374af7b8  EFLAGS: 00010286
[37955.364062] RAX: 0000000000000106 RBX: ffff88002ead7838 RCX: ffff8800374af830
[37955.364062] RDX: 0000000000000107 RSI: ffff88001d250dc0 RDI: ffff88002ead77d0
[37955.364062] RBP: ffff8800374af818 R08: 0000000000000000 R09: ffff88001ffeafc0
[37955.364062] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88001d250dc0
[37955.364062] R13: 00000000000000a0 R14: 000000572ead7838 R15: ffff88001d250dc8
[37955.364062] FS:  00002aaaaaadb100(0000) GS:ffff88003fd00000(0000) knlGS:0000000000000000
[37955.364062] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[37955.364062] CR2: 000000572ead7838 CR3: 0000000036f61000 CR4: 00000000000007e0
[37955.364062] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[37955.364062] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[37955.364062] Process as (pid: 3351, threadinfo ffff8800374ae000, task ffff880036d665c0)
[37955.364062] Stack:
[37955.364062]  ffff88001da3e700 ffff8800374af830 ffff8800374af838 ffffffff811846d0
[37955.364062]  0000000000000000 ffff88001ce75c48 01ff8800374af838 ffff8800374af838
[37955.364062]  0000000000000000 ffff88001ce75800 ffff8800374afa08 0000000000001014
[37955.364062] Call Trace:
[37955.364062]  [<ffffffff811846d0>] ? insert_inode_locked+0x160/0x160
[37955.364062]  [<ffffffff8118496c>] prune_icache_sb+0x3c/0x60
[37955.364062]  [<ffffffff8116dcbe>] super_cache_scan+0x12e/0x1b0
[37955.364062]  [<ffffffff8111354a>] shrink_slab_node+0x13a/0x250
[37955.364062]  [<ffffffff8111671b>] shrink_slab+0xab/0x120
[37955.364062]  [<ffffffff81117944>] do_try_to_free_pages+0x264/0x360
[37955.364062]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[37955.364062]  [<ffffffff81001974>] ? __switch_to+0x1b4/0x550
[37955.364062]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[37955.364062]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[37955.364062]  [<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[37955.364062]  [<ffffffff81129ebb>] do_anonymous_page+0x16b/0x350
[37955.364062]  [<ffffffff8112f9c5>] handle_pte_fault+0x235/0x240
[37955.364062]  [<ffffffff8107b8b0>] ? set_next_entity+0xb0/0xd0
[37955.364062]  [<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[37955.364062]  [<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[37955.364062]  [<ffffffff8116a8a8>] ? fsnotify_access+0x68/0x80
[37955.364062]  [<ffffffff8116b0b8>] ? vfs_read+0xd8/0x130
[37955.364062]  [<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[37955.364062]  [<ffffffff8157b348>] page_fault+0x28/0x30
[37955.364062] Code: 44 24 18 0f 84 87 00 00 00 49 83 7c 24 18 00 78 7b 49 83 c5 01 48 8b 4d a8 48 8b 11 48 8d 42 ff 48 85 d2 48 89 01 74 78 4d 39 f7 <49> 8b 06 4c 89 f3 74 6d 49 89 c6 eb a6 0f 1f 84 00 00 00 00 00 
[37955.364062] RIP  [<ffffffff81127e5b>] list_lru_walk_node+0xab/0x140

ffffffff81127e0e:       48 8b 55 b0             mov    -0x50(%rbp),%rdx
ffffffff81127e12:       4c 89 e6                mov    %r12,%rsi
ffffffff81127e15:       48 89 df                mov    %rbx,%rdi
ffffffff81127e18:       ff 55 b8                callq  *-0x48(%rbp)		# isolate(item, &nlru->lock, cb_arg)
ffffffff81127e1b:       83 f8 01                cmp    $0x1,%eax
ffffffff81127e1e:       74 78                   je     ffffffff81127e98 <list_lru_walk_node+0xe8>
ffffffff81127e20:       73 4e                   jae    ffffffff81127e70 <list_lru_walk_node+0xc0>
[...]
ffffffff81127e45:       48 8b 4d a8             mov    -0x58(%rbp),%rcx		# LRU_ROTATE:
ffffffff81127e49:       48 8b 11                mov    (%rcx),%rdx
ffffffff81127e4c:       48 8d 42 ff             lea    -0x1(%rdx),%rax	
ffffffff81127e50:       48 85 d2                test   %rdx,%rdx		# if ((*nr_to_walk)-- == 0)
ffffffff81127e53:       48 89 01                mov    %rax,(%rcx)
ffffffff81127e56:       74 78                   je     ffffffff81127ed0 <list_lru_walk_node+0x120>
ffffffff81127e58:       4d 39 f7                cmp    %r14,%r15
ffffffff81127e5b:       49 8b 06                mov    (%r14),%rax		<<< BANG
ffffffff81127e5e:       4c 89 f3                mov    %r14,%rbx
ffffffff81127e61:       74 6d                   je     ffffffff81127ed0 <list_lru_walk_node+0x120>
ffffffff81127e63:       49 89 c6                mov    %rax,%r14
ffffffff81127e66:       eb a6                   jmp    ffffffff81127e0e <list_lru_walk_node+0x5e>
[...]
ffffffff81127e70:       83 f8 02                cmp    $0x2,%eax
ffffffff81127e73:       74 d0                   je     ffffffff81127e45 <list_lru_walk_node+0x95>
ffffffff81127e75:       83 f8 03                cmp    $0x3,%eax
ffffffff81127e78:       74 06                   je     ffffffff81127e80 <list_lru_walk_node+0xd0>
ffffffff81127e7a:       0f 0b                   ud2
[...]
ffffffff81127ed0:       66 41 83 04 24 01       addw   $0x1,(%r12)
ffffffff81127ed6:       48 83 c4 38             add    $0x38,%rsp
ffffffff81127eda:       4c 89 e8                mov    %r13,%rax
ffffffff81127edd:       5b                      pop    %rbx
ffffffff81127ede:       41 5c                   pop    %r12
ffffffff81127ee0:       41 5d                   pop    %r13
ffffffff81127ee2:       41 5e                   pop    %r14
ffffffff81127ee4:       41 5f                   pop    %r15
ffffffff81127ee6:       c9                      leaveq 
ffffffff81127ee7:       c3                      retq

We are tripping over in list_for_each_safe and r14(000000572ead7838) is
obviously a garbage. So the lru is clobbered?
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Fri, Jun 28, 2013 at 10:39:43AM +0200, Michal Hocko wrote:
> I have just triggered this one.
> 
> [37955.364062] RIP: 0010:[<ffffffff81127e5b>]  [<ffffffff81127e5b>] list_lru_walk_node+0xab/0x140
> [37955.364062] RSP: 0000:ffff8800374af7b8  EFLAGS: 00010286
> [37955.364062] RAX: 0000000000000106 RBX: ffff88002ead7838 RCX: ffff8800374af830
Note ebx

> [37955.364062] RDX: 0000000000000107 RSI: ffff88001d250dc0 RDI: ffff88002ead77d0
> [37955.364062] RBP: ffff8800374af818 R08: 0000000000000000 R09: ffff88001ffeafc0
> [37955.364062] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88001d250dc0
> [37955.364062] R13: 00000000000000a0 R14: 000000572ead7838 R15: ffff88001d250dc8
Note r14


> [37955.364062] Process as (pid: 3351, threadinfo ffff8800374ae000, task ffff880036d665c0)
> [37955.364062] Stack:
> [37955.364062]  ffff88001da3e700 ffff8800374af830 ffff8800374af838 ffffffff811846d0
> [37955.364062]  0000000000000000 ffff88001ce75c48 01ff8800374af838 ffff8800374af838
> [37955.364062]  0000000000000000 ffff88001ce75800 ffff8800374afa08 0000000000001014
> [37955.364062] Call Trace:
> [37955.364062]  [<ffffffff811846d0>] ? insert_inode_locked+0x160/0x160
> [37955.364062]  [<ffffffff8118496c>] prune_icache_sb+0x3c/0x60
> [37955.364062]  [<ffffffff8116dcbe>] super_cache_scan+0x12e/0x1b0
> [37955.364062]  [<ffffffff8111354a>] shrink_slab_node+0x13a/0x250
> [37955.364062]  [<ffffffff8111671b>] shrink_slab+0xab/0x120
> [37955.364062]  [<ffffffff81117944>] do_try_to_free_pages+0x264/0x360
> [37955.364062]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
> [37955.364062]  [<ffffffff81001974>] ? __switch_to+0x1b4/0x550
> [37955.364062]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
> [37955.364062]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
> [37955.364062]  [<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
> [37955.364062]  [<ffffffff81129ebb>] do_anonymous_page+0x16b/0x350
> [37955.364062]  [<ffffffff8112f9c5>] handle_pte_fault+0x235/0x240
> [37955.364062]  [<ffffffff8107b8b0>] ? set_next_entity+0xb0/0xd0
> [37955.364062]  [<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
> [37955.364062]  [<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
> [37955.364062]  [<ffffffff8116a8a8>] ? fsnotify_access+0x68/0x80
> [37955.364062]  [<ffffffff8116b0b8>] ? vfs_read+0xd8/0x130
> [37955.364062]  [<ffffffff8157ebe9>] do_page_fault+0x9/0x10ffff88002ead7838
> [37955.364062]  [<ffffffff8157b348>] page_fault+0x28/0x30
> [37955.364062] Code: 44 24 18 0f 84 87 00 00 00 49 83 7c 24 18 00 78 7b 49 83 c5 01 48 8b 4d a8 48 8b 11 48 8d 42 ff 48 85 d2 48 89 01 74 78 4d 39 f7 <49> 8b 06 4c 89 f3 74 6d 49 89 c6 eb a6 0f 1f 84 00 00 00 00 00 
> [37955.364062] RIP  [<ffffffff81127e5b>] list_lru_walk_node+0xab/0x140
> 
> ffffffff81127e0e:       48 8b 55 b0             mov    -0x50(%rbp),%rdx
> ffffffff81127e12:       4c 89 e6                mov    %r12,%rsi
> ffffffff81127e15:       48 89 df                mov    %rbx,%rdi
> ffffffff81127e18:       ff 55 b8                callq  *-0x48(%rbp)		# isolate(item, &nlru->lock, cb_arg)
> ffffffff81127e1b:       83 f8 01                cmp    $0x1,%eax
> ffffffff81127e1e:       74 78                   je     ffffffff81127e98 <list_lru_walk_node+0xe8>
> ffffffff81127e20:       73 4e                   jae    ffffffff81127e70 <list_lru_walk_node+0xc0>
> [...]
One interesting thing I have noted here, is that r14 is basically the lower half of rbx, with
the upper part borked.

Because we are talking about a single word, this does not seem the usual update-half-of-double-word
without locking issue.

>From your excerpt, it is not totally clear what r14 is. But by looking at rdi which
is 0xffff88002ead77d0 and very probable nlru->lock due to the calling convention,
that would indicate that this is nlru->list in case you have spinlock debugging enabled.

So yes, someone destroyed our next pointer, and amazingly only half of it.

Still, the only time we ever release this lock is when isolate returns LRU_RETRY. Maybe the
way we restart is wrong? (although I can't see how)

An iput() happens outside the lock in that case, but it seems safe : if that ends up manipulating
the lru it will do so through our accessors.

I will have to think a bit more... Any other strange thing happening before it ?

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Fri 28-06-13 18:31:26, Glauber Costa wrote:
> On Fri, Jun 28, 2013 at 10:39:43AM +0200, Michal Hocko wrote:
> > I have just triggered this one.
> > 
> > [37955.364062] RIP: 0010:[<ffffffff81127e5b>]  [<ffffffff81127e5b>] list_lru_walk_node+0xab/0x140
> > [37955.364062] RSP: 0000:ffff8800374af7b8  EFLAGS: 00010286
> > [37955.364062] RAX: 0000000000000106 RBX: ffff88002ead7838 RCX: ffff8800374af830
> Note ebx
> 
> > [37955.364062] RDX: 0000000000000107 RSI: ffff88001d250dc0 RDI: ffff88002ead77d0
> > [37955.364062] RBP: ffff8800374af818 R08: 0000000000000000 R09: ffff88001ffeafc0
> > [37955.364062] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88001d250dc0
> > [37955.364062] R13: 00000000000000a0 R14: 000000572ead7838 R15: ffff88001d250dc8
> Note r14

Hmm the upper part is 57 which is also weird. Do you think this might be
a HW issue?

It would be strange that I cannot reproduce it without the series
applied and I was testing the "good" case for a long time to be sure
that this is just not a "consistent good luck".

> > [37955.364062] Process as (pid: 3351, threadinfo ffff8800374ae000, task ffff880036d665c0)
> > [37955.364062] Stack:
> > [37955.364062]  ffff88001da3e700 ffff8800374af830 ffff8800374af838 ffffffff811846d0
> > [37955.364062]  0000000000000000 ffff88001ce75c48 01ff8800374af838 ffff8800374af838
> > [37955.364062]  0000000000000000 ffff88001ce75800 ffff8800374afa08 0000000000001014
> > [37955.364062] Call Trace:
> > [37955.364062]  [<ffffffff811846d0>] ? insert_inode_locked+0x160/0x160
> > [37955.364062]  [<ffffffff8118496c>] prune_icache_sb+0x3c/0x60
> > [37955.364062]  [<ffffffff8116dcbe>] super_cache_scan+0x12e/0x1b0
> > [37955.364062]  [<ffffffff8111354a>] shrink_slab_node+0x13a/0x250
> > [37955.364062]  [<ffffffff8111671b>] shrink_slab+0xab/0x120
> > [37955.364062]  [<ffffffff81117944>] do_try_to_free_pages+0x264/0x360
> > [37955.364062]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
> > [37955.364062]  [<ffffffff81001974>] ? __switch_to+0x1b4/0x550
> > [37955.364062]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
> > [37955.364062]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
> > [37955.364062]  [<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
> > [37955.364062]  [<ffffffff81129ebb>] do_anonymous_page+0x16b/0x350
> > [37955.364062]  [<ffffffff8112f9c5>] handle_pte_fault+0x235/0x240
> > [37955.364062]  [<ffffffff8107b8b0>] ? set_next_entity+0xb0/0xd0
> > [37955.364062]  [<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
> > [37955.364062]  [<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
> > [37955.364062]  [<ffffffff8116a8a8>] ? fsnotify_access+0x68/0x80
> > [37955.364062]  [<ffffffff8116b0b8>] ? vfs_read+0xd8/0x130
> > [37955.364062]  [<ffffffff8157ebe9>] do_page_fault+0x9/0x10ffff88002ead7838
> > [37955.364062]  [<ffffffff8157b348>] page_fault+0x28/0x30
> > [37955.364062] Code: 44 24 18 0f 84 87 00 00 00 49 83 7c 24 18 00 78 7b 49 83 c5 01 48 8b 4d a8 48 8b 11 48 8d 42 ff 48 85 d2 48 89 01 74 78 4d 39 f7 <49> 8b 06 4c 89 f3 74 6d 49 89 c6 eb a6 0f 1f 84 00 00 00 00 00 
> > [37955.364062] RIP  [<ffffffff81127e5b>] list_lru_walk_node+0xab/0x140
> > 
> > ffffffff81127e0e:       48 8b 55 b0             mov    -0x50(%rbp),%rdx
> > ffffffff81127e12:       4c 89 e6                mov    %r12,%rsi
> > ffffffff81127e15:       48 89 df                mov    %rbx,%rdi
> > ffffffff81127e18:       ff 55 b8                callq  *-0x48(%rbp)		# isolate(item, &nlru->lock, cb_arg)
> > ffffffff81127e1b:       83 f8 01                cmp    $0x1,%eax
> > ffffffff81127e1e:       74 78                   je     ffffffff81127e98 <list_lru_walk_node+0xe8>
> > ffffffff81127e20:       73 4e                   jae    ffffffff81127e70 <list_lru_walk_node+0xc0>
> > [...]
> One interesting thing I have noted here, is that r14 is basically the lower half of rbx, with
> the upper part borked.
> 
> Because we are talking about a single word, this does not seem the usual update-half-of-double-word
> without locking issue.
> 
> From your excerpt, it is not totally clear what r14 is. But by looking at rdi which
> is 0xffff88002ead77d0 and very probable nlru->lock due to the calling convention,
> that would indicate that this is nlru->list in case you have spinlock debugging enabled.
> 
> So yes, someone destroyed our next pointer, and amazingly only half of it.
> 
> Still, the only time we ever release this lock is when isolate returns LRU_RETRY. Maybe the
> way we restart is wrong? (although I can't see how)
> 
> An iput() happens outside the lock in that case, but it seems safe : if that ends up manipulating
> the lru it will do so through our accessors.
> 
> I will have to think a bit more... Any other strange thing happening before it ?

Nothing. I basically see two cases here. One is hang and the other is
this crash followed up by many soft lockups as we died with the lock
held.

When I think about it, the hang case might still be a false positive
(for xfs at least) because it might just happen that those processes I
have listed are in D state just too long. I have gave them tens of
seconds and than compared traces and consider them hung if they didn't
change. I can retest and try to give them hours to be absolutely sure.

-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Thu, Jun 27, 2013 at 04:54:11PM +0200, Michal Hocko wrote:
> On Thu 27-06-13 09:24:26, Dave Chinner wrote:
> > On Wed, Jun 26, 2013 at 10:15:09AM +0200, Michal Hocko wrote:
> > > On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> > > > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > > > > And again, another hang. It looks like the inode deletion never
> > > > > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > > > > anymore. I am going to test with the other patch in the thread.
> > > > > 
> > > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > > > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > > > > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > > > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > > > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > > > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > > > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > > > > [<ffffffffffffffff>] 0xffffffffffffffff
> > > > 
> > > > I don't think this has anything to do with LRUs.
> > > 
> > > I am not claiming that. It might be a timing issue which never mattered
> > > but it is strange I can reproduce this so easily and repeatedly with the
> > > shrinkers patchset applied.
> > > As I said earlier, this might be breakage in my -mm tree as well
> > > (missing some patch which didn't go via Andrew or misapplied patch). The
> > > situation is worsen by the state of linux-next which has some unrelated
> > > issues.
> > > 
> > > I really do not want to delay the whole patchset just because of some
> > > problem on my side. Do you have any tree that I should try to test?
> > 
> > No, I've just been testing Glauber's tree and sending patches for
> > problems back to him based on it.
> > 
> > > > I won't have seen this on XFS stress testing, because it doesn't use
> > > > the VFS inode hashes for inode lookups. Given that XFS is not
> > > > triggering either problem you are seeing, that makes me think
> > > 
> > > I haven't tested with xfs.
> > 
> > That might be worthwhile if you can easily do that - another data
> > point indicating a hang or absence of a hang will help point us in
> > the right direction here...
> 
> OK, still hanging (with inode_lru_isolate-fix.patch). It is not the same
> thing, though, as xfs seem to do lookup slightly differently.
> 12467 [<ffffffffa02ca03e>] xfs_iget+0xbe/0x190 [xfs]
> [<ffffffffa02d6e98>] xfs_lookup+0xe8/0x110 [xfs]
> [<ffffffffa02cdad9>] xfs_vn_lookup+0x49/0x90 [xfs]
> [<ffffffff81174ad0>] lookup_real+0x20/0x60
> [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> [<ffffffff8117815e>] do_last+0x2de/0x780
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff

What are the full traces? This could be blocking on IO or locks
here if it's a cache miss and we are reading an inode....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

[-- Attachment #1: Type: text/plain, Size: 6849 bytes --]

On Sat 29-06-13 12:55:09, Dave Chinner wrote:
> On Thu, Jun 27, 2013 at 04:54:11PM +0200, Michal Hocko wrote:
> > On Thu 27-06-13 09:24:26, Dave Chinner wrote:
> > > On Wed, Jun 26, 2013 at 10:15:09AM +0200, Michal Hocko wrote:
> > > > On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> > > > > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > > > > > And again, another hang. It looks like the inode deletion never
> > > > > > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > > > > > anymore. I am going to test with the other patch in the thread.
> > > > > > 
> > > > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > > > > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > > > > > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > > > > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > > > > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > > > > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > > > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > > > > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > > > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > > > > > [<ffffffffffffffff>] 0xffffffffffffffff
> > > > > 
> > > > > I don't think this has anything to do with LRUs.
> > > > 
> > > > I am not claiming that. It might be a timing issue which never mattered
> > > > but it is strange I can reproduce this so easily and repeatedly with the
> > > > shrinkers patchset applied.
> > > > As I said earlier, this might be breakage in my -mm tree as well
> > > > (missing some patch which didn't go via Andrew or misapplied patch). The
> > > > situation is worsen by the state of linux-next which has some unrelated
> > > > issues.
> > > > 
> > > > I really do not want to delay the whole patchset just because of some
> > > > problem on my side. Do you have any tree that I should try to test?
> > > 
> > > No, I've just been testing Glauber's tree and sending patches for
> > > problems back to him based on it.
> > > 
> > > > > I won't have seen this on XFS stress testing, because it doesn't use
> > > > > the VFS inode hashes for inode lookups. Given that XFS is not
> > > > > triggering either problem you are seeing, that makes me think
> > > > 
> > > > I haven't tested with xfs.
> > > 
> > > That might be worthwhile if you can easily do that - another data
> > > point indicating a hang or absence of a hang will help point us in
> > > the right direction here...
> > 
> > OK, still hanging (with inode_lru_isolate-fix.patch). It is not the same
> > thing, though, as xfs seem to do lookup slightly differently.
> > 12467 [<ffffffffa02ca03e>] xfs_iget+0xbe/0x190 [xfs]
> > [<ffffffffa02d6e98>] xfs_lookup+0xe8/0x110 [xfs]
> > [<ffffffffa02cdad9>] xfs_vn_lookup+0x49/0x90 [xfs]
> > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > [<ffffffff8117815e>] do_last+0x2de/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> 
> What are the full traces?

Do you mean sysrq+t? It is attached. 

Btw. I was able to reproduce this again. The stuck processes were
sitting in the same traces for more than 28 hours without any change so
I do not think this is a temporal condition.

Traces of all processes in the D state:
7561 [<ffffffffa029c03e>] xfs_iget+0xbe/0x190 [xfs]
[<ffffffffa02a8e98>] xfs_lookup+0xe8/0x110 [xfs]
[<ffffffffa029fad9>] xfs_vn_lookup+0x49/0x90 [xfs]
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
8156 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
8913 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
9100 [<ffffffffa029c03e>] xfs_iget+0xbe/0x190 [xfs]
[<ffffffffa02a8e98>] xfs_lookup+0xe8/0x110 [xfs]
[<ffffffffa029fad9>] xfs_vn_lookup+0x49/0x90 [xfs]
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
9158 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
11247 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
12161 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
12585 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
-- 
Michal Hocko
SUSE Labs

[-- Attachment #2: demon.log.bz2 --]
[-- Type: application/octet-stream, Size: 20228 bytes --]

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Sun, Jun 30, 2013 at 08:33:49PM +0200, Michal Hocko wrote:
> On Sat 29-06-13 12:55:09, Dave Chinner wrote:
> > On Thu, Jun 27, 2013 at 04:54:11PM +0200, Michal Hocko wrote:
> > > On Thu 27-06-13 09:24:26, Dave Chinner wrote:
> > > > On Wed, Jun 26, 2013 at 10:15:09AM +0200, Michal Hocko wrote:
> > > > > On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> > > > > > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > > > > > > And again, another hang. It looks like the inode deletion never
> > > > > > > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > > > > > > anymore. I am going to test with the other patch in the thread.
> > > > > > > 
> > > > > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > > > > > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > > > > > > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > > > > > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > > > > > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > > > > > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > > > > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > > > > > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > > > > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > > > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > > > > > > [<ffffffffffffffff>] 0xffffffffffffffff

.....
> Do you mean sysrq+t? It is attached. 
> 
> Btw. I was able to reproduce this again. The stuck processes were
> sitting in the same traces for more than 28 hours without any change so
> I do not think this is a temporal condition.
> 
> Traces of all processes in the D state:
> 7561 [<ffffffffa029c03e>] xfs_iget+0xbe/0x190 [xfs]
> [<ffffffffa02a8e98>] xfs_lookup+0xe8/0x110 [xfs]
> [<ffffffffa029fad9>] xfs_vn_lookup+0x49/0x90 [xfs]
> [<ffffffff81174ad0>] lookup_real+0x20/0x60
> [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> [<ffffffff8117815e>] do_last+0x2de/0x780
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff

This looks like it may be equivalent to the ext4 trace above, though
I'm not totally sure on that yet. Can you get me the line of code
where the above code is sleeping - 'gdb> l *(xfs_iget+0xbe)' output
is sufficient.

If it's where I suspect it is, we are hitting a VFS inode that
igrab() is failing on because I_FREEING is set and that is returning
EAGAIN. Hence xfs_iget() sleeps for a short period and retries the
lookup. If you've still got a system in this state, can you dump the
xfs stats a few times about 5s apart i.e.

$ for i in `seq 0 1 5`; do echo ; date; cat /proc/fs/xfs/stat ; sleep 5 ; done

Depending on what stat is changing (i'm looking for skip vs recycle
in the inode cache stats), that will tell us why the lookup is
failing...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Mon 01-07-13 11:25:58, Dave Chinner wrote:
> On Sun, Jun 30, 2013 at 08:33:49PM +0200, Michal Hocko wrote:
> > On Sat 29-06-13 12:55:09, Dave Chinner wrote:
> > > On Thu, Jun 27, 2013 at 04:54:11PM +0200, Michal Hocko wrote:
> > > > On Thu 27-06-13 09:24:26, Dave Chinner wrote:
> > > > > On Wed, Jun 26, 2013 at 10:15:09AM +0200, Michal Hocko wrote:
> > > > > > On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> > > > > > > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > > > > > > > And again, another hang. It looks like the inode deletion never
> > > > > > > > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > > > > > > > anymore. I am going to test with the other patch in the thread.
> > > > > > > > 
> > > > > > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > > > > > > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > > > > > > > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > > > > > > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > > > > > > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > > > > > > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > > > > > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > > > > > > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > > > > > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > > > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > > > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > > > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > > > > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > > > > > > > [<ffffffffffffffff>] 0xffffffffffffffff
> 
> .....
> > Do you mean sysrq+t? It is attached. 
> > 
> > Btw. I was able to reproduce this again. The stuck processes were
> > sitting in the same traces for more than 28 hours without any change so
> > I do not think this is a temporal condition.
> > 
> > Traces of all processes in the D state:
> > 7561 [<ffffffffa029c03e>] xfs_iget+0xbe/0x190 [xfs]
> > [<ffffffffa02a8e98>] xfs_lookup+0xe8/0x110 [xfs]
> > [<ffffffffa029fad9>] xfs_vn_lookup+0x49/0x90 [xfs]
> > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > [<ffffffff8117815e>] do_last+0x2de/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> 
> This looks like it may be equivalent to the ext4 trace above, though
> I'm not totally sure on that yet. Can you get me the line of code
> where the above code is sleeping - 'gdb> l *(xfs_iget+0xbe)' output
> is sufficient.

OK, this is a bit tricky because I have xfs built as a module so objdump
on xfs.ko shows nonsense
   19039:       e8 00 00 00 00          callq  1903e <xfs_iget+0xbe>
   1903e:       48 8b 75 c0             mov    -0x40(%rbp),%rsi

crash was more clever though and it says:
0xffffffffa029c034 <xfs_iget+180>:      mov    $0x1,%edi
0xffffffffa029c039 <xfs_iget+185>:      callq  0xffffffff815776d0
<schedule_timeout_uninterruptible>
/dev/shm/mhocko-build/BUILD/kernel-3.9.0mmotm+/fs/xfs/xfs_icache.c: 423
0xffffffffa029c03e <xfs_iget+190>:      mov    -0x40(%rbp),%rsi

which maps to:
out_error_or_again:
        if (error == EAGAIN) {
                delay(1);
                goto again;
        }

So this looks like this path loops in goto again and out_error_or_again.

> If it's where I suspect it is, we are hitting a VFS inode that
> igrab() is failing on because I_FREEING is set and that is returning
> EAGAIN. Hence xfs_iget() sleeps for a short period and retries the
> lookup. If you've still got a system in this state, can you dump the
> xfs stats a few times about 5s apart i.e.
> 
> $ for i in `seq 0 1 5`; do echo ; date; cat /proc/fs/xfs/stat ; sleep 5 ; done
> 
> Depending on what stat is changing (i'm looking for skip vs recycle
> in the inode cache stats), that will tell us why the lookup is
> failing...

$ for i in `seq 0 1 5`; do echo ; date; cat /proc/fs/xfs/stat ; sleep 5 ; done

Mon Jul  1 09:29:57 CEST 2013
extent_alloc 1484333 2038118 1678 13182
abt 0 0 0 0
blk_map 21004635 3433178 1450438 1461372 1450017 25888309 0
bmbt 0 0 0 0
dir 1482235 1466711 7281 2529
trans 7676 6231535 1444850
ig 0 8534 299 1463749 0 1256778 262381
log 37039 2082072 414 8808 16395
push_ail 7684106 0 519016 449446 0 12401 64613 2970751 0 1036
xstrat 1441551 0
rw 1744884 1351499
attr 84933 0 0 0
icluster 130532 102985 2389817
vnodes 4293706604 0 0 0 1260692 1260692 1260692 0
buf 24539551 79603 24464366 2126 8792 75185 0 129859 9654
abtb2 1520647 1551239 12314 12331 0 0 0 0 0 0 0 0 0 0 15613
abtc2 2972473 1641548 1486215 1486232 0 0 0 0 0 0 0 0 0 0 258694
bmbt2 16968 199868 14855 0 3 0 89 0 6414 89 58 0 61 0 1800151
ibt2 4289847 39122572 22887 1 4 0 644 59 10700 0 88 0 92 0 2732985
qm 0 0 0 0 0 0 0 0
xpc 7892422656 3364392442 7942370166
debug 0

Mon Jul  1 09:30:02 CEST 2013
extent_alloc 1484362 2038147 1678 13182
abt 0 0 0 0
blk_map 21005075 3433237 1450468 1461401 1450047 25888838 0
bmbt 0 0 0 0
dir 1482265 1466741 7281 2529
trans 7676 6231652 1444880
ig 0 8534 299 1463779 0 1256778 262381
log 37039 2082072 414 8808 16395
push_ail 7684253 0 519016 449446 0 12401 64613 2970751 0 1036
xstrat 1441579 0
rw 1744914 1351499
attr 84933 0 0 0
icluster 130532 102985 2389817
vnodes 4293706604 0 0 0 1260692 1260692 1260692 0
buf 24540112 79607 24464923 2126 8792 75189 0 129863 9657
abtb2 1520676 1551268 12314 12331 0 0 0 0 0 0 0 0 0 0 15613
abtc2 2972531 1641578 1486244 1486261 0 0 0 0 0 0 0 0 0 0 258696
bmbt2 16969 199882 14856 0 3 0 89 0 6415 89 58 0 61 0 1800406
ibt2 4289937 39123472 22887 1 4 0 644 59 10700 0 88 0 92 0 2732985
qm 0 0 0 0 0 0 0 0
xpc 7892537344 3364415667 7942370166
debug 0

Mon Jul  1 09:30:07 CEST 2013
extent_alloc 1484393 2038181 1678 13182
abt 0 0 0 0
blk_map 21005515 3433297 1450498 1461431 1450077 25889368 0
bmbt 0 0 0 0
dir 1482295 1466771 7281 2529
trans 7676 6231774 1444910
ig 0 8534 299 1463809 0 1256778 262381
log 37039 2082072 414 8808 16395
push_ail 7684405 0 519016 449446 0 12401 64613 2970751 0 1036
xstrat 1441609 0
rw 1744944 1351499
attr 84933 0 0 0
icluster 130532 102985 2389817
vnodes 4293706604 0 0 0 1260692 1260692 1260692 0
buf 24540682 79609 24465491 2126 8792 75191 0 129867 9657
abtb2 1520708 1551300 12314 12331 0 0 0 0 0 0 0 0 0 0 15613
abtc2 2972593 1641609 1486275 1486292 0 0 0 0 0 0 0 0 0 0 258696
bmbt2 16969 199882 14856 0 3 0 89 0 6415 89 58 0 61 0 1800406
ibt2 4290028 39124384 22888 1 4 0 644 59 10700 0 88 0 92 0 2732985
qm 0 0 0 0 0 0 0 0
xpc 7892660224 3364438892 7942370166
debug 0

Mon Jul  1 09:30:12 CEST 2013
extent_alloc 1484424 2038215 1678 13182
abt 0 0 0 0
blk_map 21005901 3433353 1450524 1461461 1450103 25889836 0
bmbt 0 0 0 0
dir 1482321 1466797 7281 2529
trans 7677 6231889 1444936
ig 0 8534 299 1463835 0 1256778 262381
log 37045 2082361 414 8810 16398
push_ail 7684547 0 519079 449508 0 12408 64613 2971092 0 1037
xstrat 1441639 0
rw 1744970 1351499
attr 84933 0 0 0
icluster 130548 102999 2390155
vnodes 4293706604 0 0 0 1260692 1260692 1260692 0
buf 24541210 79611 24466017 2126 8792 75193 0 129871 9657
abtb2 1520740 1551332 12314 12331 0 0 0 0 0 0 0 0 0 0 15613
abtc2 2972655 1641640 1486306 1486323 0 0 0 0 0 0 0 0 0 0 258696
bmbt2 16969 199882 14856 0 3 0 89 0 6415 89 58 0 61 0 1800406
ibt2 4290107 39125176 22889 1 4 0 644 59 10700 0 88 0 92 0 2732985
qm 0 0 0 0 0 0 0 0
xpc 7892783104 3364458016 7942370166
debug 0

Mon Jul  1 09:30:17 CEST 2013
extent_alloc 1484454 2038245 1678 13182
abt 0 0 0 0
blk_map 21006341 3433413 1450554 1461491 1450133 25890366 0
bmbt 0 0 0 0
dir 1482351 1466827 7281 2529
trans 7677 6232011 1444966
ig 0 8534 299 1463865 0 1256778 262381
log 37045 2082361 414 8810 16398
push_ail 7684699 0 519175 449508 0 12408 64613 2971092 0 1037
xstrat 1441669 0
rw 1745000 1351499
attr 84933 0 0 0
icluster 130548 102999 2390155
vnodes 4293706604 0 0 0 1260692 1260692 1260692 0
buf 24541770 79611 24466577 2126 8792 75193 0 129871 9657
abtb2 1520770 1551362 12314 12331 0 0 0 0 0 0 0 0 0 0 15613
abtc2 2972715 1641670 1486336 1486353 0 0 0 0 0 0 0 0 0 0 258696
bmbt2 16969 199882 14856 0 3 0 89 0 6415 89 58 0 61 0 1800406
ibt2 4290197 39126076 22889 1 4 0 644 59 10700 0 88 0 92 0 2732985
qm 0 0 0 0 0 0 0 0
xpc 7892905984 3364481241 7942370166
debug 0

Mon Jul  1 09:30:22 CEST 2013
extent_alloc 1484486 2038280 1678 13182
abt 0 0 0 0
blk_map 21006782 3433474 1450584 1461522 1450163 25890898 0
bmbt 0 0 0 0
dir 1482381 1466857 7281 2529
trans 7677 6232134 1444996
ig 0 8534 299 1463895 0 1256778 262381
log 37045 2082361 414 8810 16398
push_ail 7684852 0 519272 449508 0 12408 64613 2971092 0 1037
xstrat 1441699 0
rw 1745030 1351499
attr 84933 0 0 0
icluster 130548 102999 2390155
vnodes 4293706604 0 0 0 1260692 1260692 1260692 0
buf 24542347 79614 24467151 2126 8792 75196 0 129876 9657
abtb2 1520803 1551395 12314 12331 0 0 0 0 0 0 0 0 0 0 15613
abtc2 2972779 1641702 1486368 1486385 0 0 0 0 0 0 0 0 0 0 258696
bmbt2 16970 199896 14857 0 3 0 89 0 6415 89 58 0 61 0 1800407
ibt2 4290288 39126988 22890 1 4 0 644 59 10700 0 88 0 92 0 2732985
qm 0 0 0 0 0 0 0 0
xpc 7893028864 3364504466 7942370166
debug 0

-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Mon, Jul 01, 2013 at 09:50:05AM +0200, Michal Hocko wrote:
> On Mon 01-07-13 11:25:58, Dave Chinner wrote:
> > On Sun, Jun 30, 2013 at 08:33:49PM +0200, Michal Hocko wrote:
> > > On Sat 29-06-13 12:55:09, Dave Chinner wrote:
> > > > On Thu, Jun 27, 2013 at 04:54:11PM +0200, Michal Hocko wrote:
> > > > > On Thu 27-06-13 09:24:26, Dave Chinner wrote:
> > > > > > On Wed, Jun 26, 2013 at 10:15:09AM +0200, Michal Hocko wrote:
> > > > > > > On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> > > > > > > > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > > > > > > > > And again, another hang. It looks like the inode deletion never
> > > > > > > > > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > > > > > > > > anymore. I am going to test with the other patch in the thread.
> > > > > > > > > 
> > > > > > > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > > > > > > > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > > > > > > > > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > > > > > > > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > > > > > > > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > > > > > > > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > > > > > > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > > > > > > > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > > > > > > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > > > > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > > > > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > > > > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > > > > > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > > > > > > > > [<ffffffffffffffff>] 0xffffffffffffffff
> > 
> > .....
> > > Do you mean sysrq+t? It is attached. 
> > > 
> > > Btw. I was able to reproduce this again. The stuck processes were
> > > sitting in the same traces for more than 28 hours without any change so
> > > I do not think this is a temporal condition.
> > > 
> > > Traces of all processes in the D state:
> > > 7561 [<ffffffffa029c03e>] xfs_iget+0xbe/0x190 [xfs]
> > > [<ffffffffa02a8e98>] xfs_lookup+0xe8/0x110 [xfs]
> > > [<ffffffffa029fad9>] xfs_vn_lookup+0x49/0x90 [xfs]
> > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > [<ffffffff8117815e>] do_last+0x2de/0x780
> > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > > [<ffffffffffffffff>] 0xffffffffffffffff
> > 
> > This looks like it may be equivalent to the ext4 trace above, though
> > I'm not totally sure on that yet. Can you get me the line of code
> > where the above code is sleeping - 'gdb> l *(xfs_iget+0xbe)' output
> > is sufficient.
> 
> OK, this is a bit tricky because I have xfs built as a module so objdump
> on xfs.ko shows nonsense
>    19039:       e8 00 00 00 00          callq  1903e <xfs_iget+0xbe>
>    1903e:       48 8b 75 c0             mov    -0x40(%rbp),%rsi
> 
> crash was more clever though and it says:
> 0xffffffffa029c034 <xfs_iget+180>:      mov    $0x1,%edi
> 0xffffffffa029c039 <xfs_iget+185>:      callq  0xffffffff815776d0
> <schedule_timeout_uninterruptible>
> /dev/shm/mhocko-build/BUILD/kernel-3.9.0mmotm+/fs/xfs/xfs_icache.c: 423
> 0xffffffffa029c03e <xfs_iget+190>:      mov    -0x40(%rbp),%rsi
> 
> which maps to:
> out_error_or_again:
>         if (error == EAGAIN) {
>                 delay(1);
>                 goto again;
>         }
> 
> So this looks like this path loops in goto again and out_error_or_again.

Yup, that's what I suspected.

> > If it's where I suspect it is, we are hitting a VFS inode that
> > igrab() is failing on because I_FREEING is set and that is returning
> > EAGAIN. Hence xfs_iget() sleeps for a short period and retries the
> > lookup. If you've still got a system in this state, can you dump the
> > xfs stats a few times about 5s apart i.e.
> > 
> > $ for i in `seq 0 1 5`; do echo ; date; cat /proc/fs/xfs/stat ; sleep 5 ; done
> > 
> > Depending on what stat is changing (i'm looking for skip vs recycle
> > in the inode cache stats), that will tell us why the lookup is
> > failing...
> 
> $ for i in `seq 0 1 5`; do echo ; date; cat /proc/fs/xfs/stat ; sleep 5 ; done
> 
> Mon Jul  1 09:29:57 CEST 2013
> extent_alloc 1484333 2038118 1678 13182
> abt 0 0 0 0
> blk_map 21004635 3433178 1450438 1461372 1450017 25888309 0
> bmbt 0 0 0 0
> dir 1482235 1466711 7281 2529
> trans 7676 6231535 1444850
> ig 0 8534 299 1463749 0 1256778 262381
            ^^^

That is the recycle stat, which indicates we've found an inode being
reclaimed. When it's found an inode that have been evicted, but not
yet reclaimed at the XFS level, that stat will increase. If the
inode is still valid at the VFS level, and igrab() fails, then we'll
get EAGAIN without that stat being increased. So, igrab() is
failing, and that means I_FREEING|I_WILL_FREE are set.

So, it looks to be the same case as the ext4 hang, and it's likely
that we have some dangling inode dispose list somewhere. So, here's
the fun part. Use tracing to grab the inode number that is stuck
(tracepoint xfs::xfs_iget_skip), and then run crash on the live
kernel on the process that is looping, and find the struct xfs_inode
and print it.  Use the inode number from the trace point to check
you've got the right inode.

Th struct inode of the VFS inode is embedded into the struct
xfs_inode, and the dispose list that it is on should be the on the
inode->i_lru_list. What that, and see how many other inodes are on
that list. Once we know if it's a single inode, and whether the
dispose list it is on is intact, empty or corrupt, we might have a
better idea of how these inodes are getting lost....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

[-- Attachment #1: Type: text/plain, Size: 8842 bytes --]

On Mon 01-07-13 18:10:56, Dave Chinner wrote:
> On Mon, Jul 01, 2013 at 09:50:05AM +0200, Michal Hocko wrote:
> > On Mon 01-07-13 11:25:58, Dave Chinner wrote:
> > > On Sun, Jun 30, 2013 at 08:33:49PM +0200, Michal Hocko wrote:
> > > > On Sat 29-06-13 12:55:09, Dave Chinner wrote:
> > > > > On Thu, Jun 27, 2013 at 04:54:11PM +0200, Michal Hocko wrote:
> > > > > > On Thu 27-06-13 09:24:26, Dave Chinner wrote:
> > > > > > > On Wed, Jun 26, 2013 at 10:15:09AM +0200, Michal Hocko wrote:
> > > > > > > > On Tue 25-06-13 12:27:54, Dave Chinner wrote:
> > > > > > > > > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote:
> > > > > > > > > > And again, another hang. It looks like the inode deletion never
> > > > > > > > > > finishes. The good thing is that I do not see any LRU related BUG_ONs
> > > > > > > > > > anymore. I am going to test with the other patch in the thread.
> > > > > > > > > > 
> > > > > > > > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0	<<< waiting for an inode to go away
> > > > > > > > > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > > > > > > > > > [<ffffffff8118525f>] iget_locked+0x4f/0x180
> > > > > > > > > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0
> > > > > > > > > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160
> > > > > > > > > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > > > > > > > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > > > > > > > > [<ffffffff8117815e>] do_last+0x2de/0x780			<<< holds i_mutex
> > > > > > > > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > > > > > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > > > > > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > > > > > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > > > > > > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b
> > > > > > > > > > [<ffffffffffffffff>] 0xffffffffffffffff
> > > 
> > > .....
> > > > Do you mean sysrq+t? It is attached. 
> > > > 
> > > > Btw. I was able to reproduce this again. The stuck processes were
> > > > sitting in the same traces for more than 28 hours without any change so
> > > > I do not think this is a temporal condition.
> > > > 
> > > > Traces of all processes in the D state:
> > > > 7561 [<ffffffffa029c03e>] xfs_iget+0xbe/0x190 [xfs]
> > > > [<ffffffffa02a8e98>] xfs_lookup+0xe8/0x110 [xfs]
> > > > [<ffffffffa029fad9>] xfs_vn_lookup+0x49/0x90 [xfs]
> > > > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > > > [<ffffffff8117815e>] do_last+0x2de/0x780
> > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > > > [<ffffffffffffffff>] 0xffffffffffffffff
> > > 
> > > This looks like it may be equivalent to the ext4 trace above, though
> > > I'm not totally sure on that yet. Can you get me the line of code
> > > where the above code is sleeping - 'gdb> l *(xfs_iget+0xbe)' output
> > > is sufficient.
> > 
> > OK, this is a bit tricky because I have xfs built as a module so objdump
> > on xfs.ko shows nonsense
> >    19039:       e8 00 00 00 00          callq  1903e <xfs_iget+0xbe>
> >    1903e:       48 8b 75 c0             mov    -0x40(%rbp),%rsi
> > 
> > crash was more clever though and it says:
> > 0xffffffffa029c034 <xfs_iget+180>:      mov    $0x1,%edi
> > 0xffffffffa029c039 <xfs_iget+185>:      callq  0xffffffff815776d0
> > <schedule_timeout_uninterruptible>
> > /dev/shm/mhocko-build/BUILD/kernel-3.9.0mmotm+/fs/xfs/xfs_icache.c: 423
> > 0xffffffffa029c03e <xfs_iget+190>:      mov    -0x40(%rbp),%rsi
> > 
> > which maps to:
> > out_error_or_again:
> >         if (error == EAGAIN) {
> >                 delay(1);
> >                 goto again;
> >         }
> > 
> > So this looks like this path loops in goto again and out_error_or_again.
> 
> Yup, that's what I suspected.
> 
> > > If it's where I suspect it is, we are hitting a VFS inode that
> > > igrab() is failing on because I_FREEING is set and that is returning
> > > EAGAIN. Hence xfs_iget() sleeps for a short period and retries the
> > > lookup. If you've still got a system in this state, can you dump the
> > > xfs stats a few times about 5s apart i.e.
> > > 
> > > $ for i in `seq 0 1 5`; do echo ; date; cat /proc/fs/xfs/stat ; sleep 5 ; done
> > > 
> > > Depending on what stat is changing (i'm looking for skip vs recycle
> > > in the inode cache stats), that will tell us why the lookup is
> > > failing...
> > 
> > $ for i in `seq 0 1 5`; do echo ; date; cat /proc/fs/xfs/stat ; sleep 5 ; done
> > 
> > Mon Jul  1 09:29:57 CEST 2013
> > extent_alloc 1484333 2038118 1678 13182
> > abt 0 0 0 0
> > blk_map 21004635 3433178 1450438 1461372 1450017 25888309 0
> > bmbt 0 0 0 0
> > dir 1482235 1466711 7281 2529
> > trans 7676 6231535 1444850
> > ig 0 8534 299 1463749 0 1256778 262381
>             ^^^
> 
> That is the recycle stat, which indicates we've found an inode being
> reclaimed. When it's found an inode that have been evicted, but not
> yet reclaimed at the XFS level, that stat will increase. If the
> inode is still valid at the VFS level, and igrab() fails, then we'll
> get EAGAIN without that stat being increased. So, igrab() is
> failing, and that means I_FREEING|I_WILL_FREE are set.
> 
> So, it looks to be the same case as the ext4 hang, and it's likely
> that we have some dangling inode dispose list somewhere. So, here's
> the fun part. Use tracing to grab the inode number that is stuck
> (tracepoint xfs::xfs_iget_skip), 

$ cat /sys/kernel/debug/tracing/trace_pipe > demon.trace.log &
$ pid=$!
$ sleep 10s ; kill $pid
$ awk '{print $1, $9}' demon.trace.log | sort -u
cc1-7561 0xf78d4f
cc1-9100 0x80b2a35

> and then run crash on the live kernel on the process that is looping,
> and find the struct xfs_inode and print it.  Use the inode number from
> the trace point to check you've got the right inode.

crash> bt -f 7561
 #4 [ffff88003744db40] xfs_iget at ffffffffa029c03e [xfs]
    ffff88003744db48: 0000000000000000 0000000000000000 
    ffff88003744db58: 0000000000013b40 ffff88003744dc30 
    ffff88003744db68: 0000000000000000 0000000000000000 
    ffff88003744db78: 0000000000f78d4f ffffffffa02dafec 
    ffff88003744db88: ffff88000c09e1c0 0000000000000008 
    ffff88003744db98: 0000000000000000 ffff88000c0a0ac0 
    ffff88003744dba8: ffff88003744dc18 0000000000000000 
    ffff88003744dbb8: ffff88003744dc08 ffffffffa02a8e98
crash> dis xfs_iget
[...]
0xffffffffa029c045 <xfs_iget+197>:      callq  0xffffffff812ca190 <radix_tree_lookup>
0xffffffffa029c04a <xfs_iget+202>:      test   %rax,%rax
0xffffffffa029c04d <xfs_iget+205>:      mov    %rax,-0x30(%rbp)

So the inode should be at -0x30(%rbp) which is
crash> struct xfs_inode.i_ino ffff88000c09e1c0
  i_ino = 16223567
crash> p /x 16223567
$15 = 0xf78d4f

> Th struct inode of the VFS inode is embedded into the struct
> xfs_inode, 

crash> struct -o xfs_inode.i_vnode ffff88000c09e1c0
struct xfs_inode {
  [ffff88000c09e2f8] struct inode i_vnode;
}

> and the dispose list that it is on should be the on the
> inode->i_lru_list. 

crash> struct inode.i_lru ffff88000c09e2f8
  i_lru = {
    next = 0xffff88000c09e3e8, 
    prev = 0xffff88000c09e3e8
  }
crash> struct inode.i_flags ffff88000c09e2f8
  i_flags = 4096

The full xfs_inode dump is attached.

> What that, and see how many other inodes are on that list. Once we
> know if it's a single inode,

The list seems to be empty. And the same is the case for the other inode:
crash> bt -f 9100
 #4 [ffff88001c8c5b40] xfs_iget at ffffffffa029c03e [xfs]
    ffff88001c8c5b48: 0000000000000000 0000000000000000 
    ffff88001c8c5b58: 0000000000013b40 ffff88001c8c5c30 
    ffff88001c8c5b68: 0000000000000000 0000000000000000 
    ffff88001c8c5b78: 00000000000b2a35 ffffffffa02dafec 
    ffff88001c8c5b88: ffff88000c09ec40 0000000000000008 
    ffff88001c8c5b98: 0000000000000000 ffff8800359e9b00 
    ffff88001c8c5ba8: ffff88001c8c5c18 0000000000000000 
    ffff88001c8c5bb8: ffff88001c8c5c08 ffffffffa02a8e98
crash> p /x 0xffff88001c8c5bb8-0x30
$16 = 0xffff88001c8c5b88
sh> struct xfs_inode.i_ino ffff88000c09ec40
  i_ino = 134949429
crash> p /x 134949429
$17 = 0x80b2a35
crash> struct -o xfs_inode.i_vnode ffff88000c09ec40
struct xfs_inode {
  [ffff88000c09ed78] struct inode i_vnode;
}
crash> struct inode.i_lru ffff88000c09ed78
  i_lru = {
    next = 0xffff88000c09ee68, 
    prev = 0xffff88000c09ee68
  }
crash> struct inode.i_flags ffff88000c09ed78
  i_flags = 4096

> and whether the dispose list it is on is intact, empty or corrupt, we
> might have a better idea of how these inodes are getting lost....
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

-- 
Michal Hocko
SUSE Labs

[-- Attachment #2: xfs_inode --]
[-- Type: text/plain, Size: 7028 bytes --]

crash> struct xfs_inode ffff88000c09e1c0
struct xfs_inode {
  i_mount = 0xffff88001ca4d000, 
  i_udquot = 0x0, 
  i_gdquot = 0x0, 
  i_ino = 16223567, 
  i_imap = {
    im_blkno = 8111776, 
    im_len = 16, 
    im_boffset = 3840
  }, 
  i_afp = 0x0, 
  i_df = {
    if_bytes = 16, 
    if_real_bytes = 0, 
    if_broot = 0x0, 
    if_broot_bytes = 0, 
    if_flags = 2 '\002', 
    if_u1 = {
      if_extents = 0xffff88000c09e218, 
      if_ext_irec = 0xffff88000c09e218, 
      if_data = 0xffff88000c09e218 ""
    }, 
    if_u2 = {
      if_inline_ext = {{
          l0 = 0, 
          l1 = 2135699750913
        }, {
          l0 = 0, 
          l1 = 0
        }}, 
      if_inline_data = "\000\000\000\000\000\000\000\000\001\000\240A\361\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000", 
      if_rdev = 0, 
      if_uuid = {
        __u_bits = "\000\000\000\000\000\000\000\000\001\000\240A\361\001\000"
      }
    }
  }, 
  i_itemp = 0x0, 
  i_lock = {
    mr_lock = {
      count = 0, 
      wait_lock = {
        raw_lock = {
          {
            head_tail = 0, 
            tickets = {
              head = 0, 
              tail = 0
            }
          }
        }
      }, 
      wait_list = {
        next = 0xffff88000c09e250, 
        prev = 0xffff88000c09e250
      }
    }
  }, 
  i_iolock = {
    mr_lock = {
      count = 0, 
      wait_lock = {
        raw_lock = {
          {
            head_tail = 0, 
            tickets = {
              head = 0, 
              tail = 0
            }
          }
        }
      }, 
      wait_list = {
        next = 0xffff88000c09e270, 
        prev = 0xffff88000c09e270
      }
    }
  }, 
  i_pincount = {
    counter = 0
  }, 
  i_flags_lock = {
    {
      rlock = {
        raw_lock = {
          {
            head_tail = 878654559, 
            tickets = {
              head = 13407, 
              tail = 13407
            }
          }
        }
      }
    }
  }, 
  i_flags = 0, 
  i_delayed_blks = 0, 
  i_d = {
    di_magic = 18766, 
    di_mode = 33204, 
    di_version = 2 '\002', 
    di_format = 2 '\002', 
    di_onlink = 0, 
    di_uid = 0, 
    di_gid = 0, 
    di_nlink = 1, 
    di_projid_lo = 0, 
    di_projid_hi = 0, 
    di_pad = "\000\000\000\000\000", 
    di_flushiter = 2, 
    di_atime = {
      t_sec = 1372433043, 
      t_nsec = 599568074
    }, 
    di_mtime = {
      t_sec = 1352637873, 
      t_nsec = 0
    }, 
    di_ctime = {
      t_sec = 1372432997, 
      t_nsec = 935341638
    }, 
    di_size = 3280, 
    di_nblocks = 1, 
    di_extsize = 0, 
    di_nextents = 1, 
    di_anextents = 0, 
    di_forkoff = 0 '\000', 
    di_aformat = 2 '\002', 
    di_dmevmask = 0, 
    di_dmstate = 0, 
    di_flags = 0, 
    di_gen = 15307555
  }, 
  i_vnode = {
    i_mode = 33204, 
    i_opflags = 5, 
    i_uid = 0, 
    i_gid = 0, 
    i_flags = 4096, 
    i_acl = 0x0, 
    i_default_acl = 0x0, 
    i_op = 0xffffffffa0301800, 
    i_sb = 0xffff88001e70c800, 
    i_mapping = 0xffff88000c09e440, 
    i_security = 0x0, 
    i_ino = 16223567, 
    {
      i_nlink = 1, 
      __i_nlink = 1
    }, 
    i_rdev = 0, 
    i_size = 3280, 
    i_atime = {
      tv_sec = 1372433043, 
      tv_nsec = 599568074
    }, 
    i_mtime = {
      tv_sec = 1352637873, 
      tv_nsec = 0
    }, 
    i_ctime = {
      tv_sec = 1372432997, 
      tv_nsec = 935341638
    }, 
    i_lock = {
      {
        rlock = {
          raw_lock = {
            {
              head_tail = 852898518, 
              tickets = {
                head = 13014, 
                tail = 13014
              }
            }
          }
        }
      }
    }, 
    i_bytes = 0, 
    i_blkbits = 12, 
    i_blocks = 0, 
    i_state = 32, 
    i_mutex = {
      count = {
        counter = 1
      }, 
      wait_lock = {
        {
          rlock = {
            raw_lock = {
              {
                head_tail = 0, 
                tickets = {
                  head = 0, 
                  tail = 0
                }
              }
            }
          }
        }
      }, 
      wait_list = {
        next = 0xffff88000c09e3a8, 
        prev = 0xffff88000c09e3a8
      }, 
      owner = 0x0
    }, 
    dirtied_when = 0, 
    i_hash = {
      next = 0x0, 
      pprev = 0xffff88000c09e3c8
    }, 
    i_wb_list = {
      next = 0xffff88000c09e3d8, 
      prev = 0xffff88000c09e3d8
    }, 
    i_lru = {
      next = 0xffff88000c09e3e8, 
      prev = 0xffff88000c09e3e8
    }, 
    i_sb_list = {
      next = 0xffff88000c0a0cf8, 
      prev = 0xffff880012423d38
    }, 
    {
      i_dentry = {
        first = 0x0
      }, 
      i_rcu = {
        next = 0x0, 
        func = 0xffffffffa029acb0 <xfs_inode_free_callback>
      }
    }, 
    i_version = 0, 
    i_count = {
      counter = 0
    }, 
    i_dio_count = {
      counter = 0
    }, 
    i_writecount = {
      counter = 0
    }, 
    i_fop = 0xffffffffa03015c0, 
    i_flock = 0x0, 
    i_data = {
      host = 0xffff88000c09e2f8, 
      page_tree = {
        height = 0, 
        gfp_mask = 32, 
        rnode = 0x0
      }, 
      tree_lock = {
        {
          rlock = {
            raw_lock = {
              {
                head_tail = 4849738, 
                tickets = {
                  head = 74, 
                  tail = 74
                }
              }
            }
          }
        }
      }, 
      i_mmap_writable = 0, 
      i_mmap = {
        rb_node = 0x0
      }, 
      i_mmap_nonlinear = {
        next = 0xffff88000c09e468, 
        prev = 0xffff88000c09e468
      }, 
      i_mmap_mutex = {
        count = {
          counter = 1
        }, 
        wait_lock = {
          {
            rlock = {
              raw_lock = {
                {
                  head_tail = 0, 
                  tickets = {
                    head = 0, 
                    tail = 0
                  }
                }
              }
            }
          }
        }, 
        wait_list = {
          next = 0xffff88000c09e480, 
          prev = 0xffff88000c09e480
        }, 
        owner = 0x0
      }, 
      nrpages = 0, 
      writeback_index = 0, 
      a_ops = 0xffffffffa0301460, 
      flags = 131290, 
      backing_dev_info = 0xffff88001c7b9950, 
      private_lock = {
        {
          rlock = {
            raw_lock = {
              {
                head_tail = 1835036, 
                tickets = {
                  head = 28, 
                  tail = 28
                }
              }
            }
          }
        }
      }, 
      private_list = {
        next = 0xffff88000c09e4c8, 
        prev = 0xffff88000c09e4c8
      }, 
      private_data = 0x0
    }, 
    i_dquot = {0x0, 0x0}, 
    i_devices = {
      next = 0xffff88000c09e4f0, 
      prev = 0xffff88000c09e4f0
    }, 
    {
      i_pipe = 0x0, 
      i_bdev = 0x0, 
      i_cdev = 0x0
    }, 
    i_generation = 15307555, 
    i_fsnotify_mask = 0, 
    i_fsnotify_marks = {
      first = 0x0
    }, 
    i_private = 0x0
  }
}

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Tue, Jul 02, 2013 at 11:22:00AM +0200, Michal Hocko wrote:
> On Mon 01-07-13 18:10:56, Dave Chinner wrote:
> > On Mon, Jul 01, 2013 at 09:50:05AM +0200, Michal Hocko wrote:
> > > On Mon 01-07-13 11:25:58, Dave Chinner wrote:
> > That is the recycle stat, which indicates we've found an inode being
> > reclaimed. When it's found an inode that have been evicted, but not
> > yet reclaimed at the XFS level, that stat will increase. If the
> > inode is still valid at the VFS level, and igrab() fails, then we'll
> > get EAGAIN without that stat being increased. So, igrab() is
> > failing, and that means I_FREEING|I_WILL_FREE are set.
> > 
> > So, it looks to be the same case as the ext4 hang, and it's likely
> > that we have some dangling inode dispose list somewhere. So, here's
> > the fun part. Use tracing to grab the inode number that is stuck
> > (tracepoint xfs::xfs_iget_skip), 
> 
> $ cat /sys/kernel/debug/tracing/trace_pipe > demon.trace.log &
> $ pid=$!
> $ sleep 10s ; kill $pid
> $ awk '{print $1, $9}' demon.trace.log | sort -u
> cc1-7561 0xf78d4f
> cc1-9100 0x80b2a35
.....
> 
> > and the dispose list that it is on should be the on the
> > inode->i_lru_list. 
> 
> crash> struct inode.i_lru ffff88000c09e2f8
>   i_lru = {
>     next = 0xffff88000c09e3e8, 
>     prev = 0xffff88000c09e3e8
>   }

Hmmm, that's empty.

> crash> struct inode.i_flags ffff88000c09e2f8
>   i_flags = 4096

I asked for the wrong field, I wanted i_state, but seeing as you:

> The full xfs_inode dump is attached.

Dumped the whole inode, I got it from below :)

>     i_state = 32, 

so, i_state = I_FREEING.

IOWs, we've got an inode marked I_FREEING that isn't on a dispose
list but hasn't passed through evict() correctly.

> crash> struct xfs_inode ffff88000c09e1c0
> struct xfs_inode {
.....
>   i_flags = 0, 

XFS doesn't see the inode as reclaimable yet, either.

Ok, so it's been leaked from a dispose list somehow. Thanks for the
info, Michal, it's time to go look at the code....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 02-07-13 22:19:47, Dave Chinner wrote:
[...]
> Ok, so it's been leaked from a dispose list somehow. Thanks for the
> info, Michal, it's time to go look at the code....

OK, just in case we will need it, I am keeping the machine in this state
for now. So we still can play with crash and check all the juicy
internals.
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Tue, Jul 02, 2013 at 02:44:27PM +0200, Michal Hocko wrote:
> On Tue 02-07-13 22:19:47, Dave Chinner wrote:
> [...]
> > Ok, so it's been leaked from a dispose list somehow. Thanks for the
> > info, Michal, it's time to go look at the code....
> 
> OK, just in case we will need it, I am keeping the machine in this state
> for now. So we still can play with crash and check all the juicy
> internals.

My current suspect is the LRU_RETRY code. I don't think what it is
doing is at all valid - list_for_each_safe() is not safe if you drop
the lock that protects the list. i.e. there is nothing that protects
the stored next pointer from being removed from the list by someone
else. Hence what I think is occurring is this:


thread 1			thread 2
lock(lru)
list_for_each_safe(lru)		lock(lru)
  isolate			......
    lock(i_lock)
    has buffers
      __iget
      unlock(i_lock)
      unlock(lru)
      .....			(gets lru lock)
      				list_for_each_safe(lru)
				  walks all the inodes
				  finds inode being isolated by other thread
				  isolate
				    i_count > 0
				      list_del_init(i_lru)
				      return LRU_REMOVED;
				   moves to next inode, inode that
				   other thread has stored as next
				   isolate
				     i_state |= I_FREEING
				     list_move(dispose_list)
				     return LRU_REMOVED
				 ....
				 unlock(lru)
      lock(lru)
      return LRU_RETRY;
  if (!first_pass)
    ....
  --nr_to_scan
  (loop again using next, which has already been removed from the
  LRU by the other thread!)
  isolate
    lock(i_lock)
    if (i_state & ~I_REFERENCED)
      list_del_init(i_lru)	<<<<< inode is on dispose list!
				<<<<< inode is now isolated, with I_FREEING set
      return LRU_REMOVED;

That fits the corpse left on your machine, Michal. One thread has
moved the inode to a dispose list, the other thread thinks it is
still on the LRU and should be removed, and removes it.

This also explains the lru item count going negative - the same item
is being removed from the lru twice. So it seems like all the
problems you've been seeing are caused by this one problem....

Patch below that should fix this.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

list_lru: fix broken LRU_RETRY behaviour

From: Dave Chinner <dchinner@redhat.com>

The LRU_RETRY code assumes that the list traversal status after we
have dropped and regained the list lock. Unfortunately, this is not
a valid assumption, and that can lead to racing traversals isolating
objects that the other traversal expects to be the next item on the
list.

This is causing problems with the inode cache shrinker isolation,
with races resulting in an inode on a dispose list being "isolated"
because a racing traversal still thinks it is on the LRU. The inode
is then never reclaimed and that causes hangs if a subsequent lookup
on that inode occurs.

Fix it by always restarting the list walk on a LRU_RETRY return from
the isolate callback. Avoid the possibility of livelocks the current
code was trying to aavoid by always decrementing the nr_to_walk
counter on retries so that even if we keep hitting the same item on
the list we'll eventually stop trying to walk and exit out of the
situation causing the problem.

Reported-by: Michal Hocko <mhocko@suse.cz>
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 mm/list_lru.c |   29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/mm/list_lru.c b/mm/list_lru.c
index dc71659..7246791 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -71,19 +71,19 @@ list_lru_walk_node(struct list_lru *lru, int nid, list_lru_walk_cb isolate,
 	struct list_lru_node	*nlru = &lru->node[nid];
 	struct list_head *item, *n;
 	unsigned long isolated = 0;
-	/*
-	 * If we don't keep state of at which pass we are, we can loop at
-	 * LRU_RETRY, since we have no guarantees that the caller will be able
-	 * to do something other than retry on the next pass. We handle this by
-	 * allowing at most one retry per object. This should not be altered
-	 * by any condition other than LRU_RETRY.
-	 */
-	bool first_pass = true;
 
 	spin_lock(&nlru->lock);
 restart:
 	list_for_each_safe(item, n, &nlru->list) {
 		enum lru_status ret;
+
+		/*
+		 * decrement nr_to_walk first so that we don't livelock if we
+		 * get stuck on large numbesr of LRU_RETRY items
+		 */
+		if (--(*nr_to_walk) == 0)
+			break;
+
 		ret = isolate(item, &nlru->lock, cb_arg);
 		switch (ret) {
 		case LRU_REMOVED:
@@ -98,19 +98,14 @@ restart:
 		case LRU_SKIP:
 			break;
 		case LRU_RETRY:
-			if (!first_pass) {
-				first_pass = true;
-				break;
-			}
-			first_pass = false;
+			/*
+			 * The lru lock has been dropped, our list traversal is
+			 * now invalid and so we have to restart from scratch.
+			 */
 			goto restart;
 		default:
 			BUG();
 		}
-
-		if ((*nr_to_walk)-- == 0)
-			break;
-
 	}
 
 	spin_unlock(&nlru->lock);

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Wed, Jul 03, 2013 at 09:24:03PM +1000, Dave Chinner wrote:
> On Tue, Jul 02, 2013 at 02:44:27PM +0200, Michal Hocko wrote:
> > On Tue 02-07-13 22:19:47, Dave Chinner wrote:
> > [...]
> > > Ok, so it's been leaked from a dispose list somehow. Thanks for the
> > > info, Michal, it's time to go look at the code....
> > 
> > OK, just in case we will need it, I am keeping the machine in this state
> > for now. So we still can play with crash and check all the juicy
> > internals.
> 
> My current suspect is the LRU_RETRY code. I don't think what it is
> doing is at all valid - list_for_each_safe() is not safe if you drop
> the lock that protects the list. i.e. there is nothing that protects
> the stored next pointer from being removed from the list by someone
> else. Hence what I think is occurring is this:
> 
> 
> thread 1			thread 2
> lock(lru)
> list_for_each_safe(lru)		lock(lru)
>   isolate			......
>     lock(i_lock)
>     has buffers
>       __iget
>       unlock(i_lock)
>       unlock(lru)
>       .....			(gets lru lock)
>       				list_for_each_safe(lru)
> 				  walks all the inodes
> 				  finds inode being isolated by other thread
> 				  isolate
> 				    i_count > 0
> 				      list_del_init(i_lru)
> 				      return LRU_REMOVED;
> 				   moves to next inode, inode that
> 				   other thread has stored as next
> 				   isolate
> 				     i_state |= I_FREEING
> 				     list_move(dispose_list)
> 				     return LRU_REMOVED
> 				 ....
> 				 unlock(lru)
>       lock(lru)
>       return LRU_RETRY;
>   if (!first_pass)
>     ....
>   --nr_to_scan
>   (loop again using next, which has already been removed from the
>   LRU by the other thread!)
>   isolate
>     lock(i_lock)
>     if (i_state & ~I_REFERENCED)
>       list_del_init(i_lru)	<<<<< inode is on dispose list!
> 				<<<<< inode is now isolated, with I_FREEING set
>       return LRU_REMOVED;
> 
> That fits the corpse left on your machine, Michal. One thread has
> moved the inode to a dispose list, the other thread thinks it is
> still on the LRU and should be removed, and removes it.
> 
> This also explains the lru item count going negative - the same item
> is being removed from the lru twice. So it seems like all the
> problems you've been seeing are caused by this one problem....
> 
> Patch below that should fix this.
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
> 
> list_lru: fix broken LRU_RETRY behaviour
> 
> From: Dave Chinner <dchinner@redhat.com>
> 
> The LRU_RETRY code assumes that the list traversal status after we
> have dropped and regained the list lock. Unfortunately, this is not
> a valid assumption, and that can lead to racing traversals isolating
> objects that the other traversal expects to be the next item on the
> list.
> 
> This is causing problems with the inode cache shrinker isolation,
> with races resulting in an inode on a dispose list being "isolated"
> because a racing traversal still thinks it is on the LRU. The inode
> is then never reclaimed and that causes hangs if a subsequent lookup
> on that inode occurs.
> 
> Fix it by always restarting the list walk on a LRU_RETRY return from
> the isolate callback. Avoid the possibility of livelocks the current
> code was trying to aavoid by always decrementing the nr_to_walk
> counter on retries so that even if we keep hitting the same item on
> the list we'll eventually stop trying to walk and exit out of the
> situation causing the problem.
> 
> Reported-by: Michal Hocko <mhocko@suse.cz>
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>  mm/list_lru.c |   29 ++++++++++++-----------------
>  1 file changed, 12 insertions(+), 17 deletions(-)
> 
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index dc71659..7246791 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -71,19 +71,19 @@ list_lru_walk_node(struct list_lru *lru, int nid, list_lru_walk_cb isolate,
>  	struct list_lru_node	*nlru = &lru->node[nid];
>  	struct list_head *item, *n;
>  	unsigned long isolated = 0;
> -	/*
> -	 * If we don't keep state of at which pass we are, we can loop at
> -	 * LRU_RETRY, since we have no guarantees that the caller will be able
> -	 * to do something other than retry on the next pass. We handle this by
> -	 * allowing at most one retry per object. This should not be altered
> -	 * by any condition other than LRU_RETRY.
> -	 */
> -	bool first_pass = true;
>  
>  	spin_lock(&nlru->lock);
>  restart:
>  	list_for_each_safe(item, n, &nlru->list) {
>  		enum lru_status ret;
> +
> +		/*
> +		 * decrement nr_to_walk first so that we don't livelock if we
> +		 * get stuck on large numbesr of LRU_RETRY items
> +		 */
> +		if (--(*nr_to_walk) == 0)
> +			break;
> +
>  		ret = isolate(item, &nlru->lock, cb_arg);
>  		switch (ret) {
>  		case LRU_REMOVED:
> @@ -98,19 +98,14 @@ restart:
>  		case LRU_SKIP:
>  			break;
>  		case LRU_RETRY:
> -			if (!first_pass) {
> -				first_pass = true;
> -				break;
> -			}
> -			first_pass = false;
> +			/*
> +			 * The lru lock has been dropped, our list traversal is
> +			 * now invalid and so we have to restart from scratch.
> +			 */
>  			goto restart;
>  		default:
>  			BUG();
>  		}
> -
> -		if ((*nr_to_walk)-- == 0)
> -			break;
> -
>  	}

This patch makes perfect sense to me, along with your description.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Wed 03-07-13 21:24:03, Dave Chinner wrote:
> On Tue, Jul 02, 2013 at 02:44:27PM +0200, Michal Hocko wrote:
> > On Tue 02-07-13 22:19:47, Dave Chinner wrote:
> > [...]
> > > Ok, so it's been leaked from a dispose list somehow. Thanks for the
> > > info, Michal, it's time to go look at the code....
> > 
> > OK, just in case we will need it, I am keeping the machine in this state
> > for now. So we still can play with crash and check all the juicy
> > internals.
> 
> My current suspect is the LRU_RETRY code. I don't think what it is
> doing is at all valid - list_for_each_safe() is not safe if you drop
> the lock that protects the list. i.e. there is nothing that protects
> the stored next pointer from being removed from the list by someone
> else. Hence what I think is occurring is this:
> 
> 
> thread 1			thread 2
> lock(lru)
> list_for_each_safe(lru)		lock(lru)
>   isolate			......
>     lock(i_lock)
>     has buffers
>       __iget
>       unlock(i_lock)
>       unlock(lru)
>       .....			(gets lru lock)
>       				list_for_each_safe(lru)
> 				  walks all the inodes
> 				  finds inode being isolated by other thread
> 				  isolate
> 				    i_count > 0
> 				      list_del_init(i_lru)
> 				      return LRU_REMOVED;
> 				   moves to next inode, inode that
> 				   other thread has stored as next
> 				   isolate
> 				     i_state |= I_FREEING
> 				     list_move(dispose_list)
> 				     return LRU_REMOVED
> 				 ....
> 				 unlock(lru)
>       lock(lru)
>       return LRU_RETRY;
>   if (!first_pass)
>     ....
>   --nr_to_scan
>   (loop again using next, which has already been removed from the
>   LRU by the other thread!)
>   isolate
>     lock(i_lock)
>     if (i_state & ~I_REFERENCED)
>       list_del_init(i_lru)	<<<<< inode is on dispose list!
> 				<<<<< inode is now isolated, with I_FREEING set
>       return LRU_REMOVED;
> 
> That fits the corpse left on your machine, Michal. One thread has
> moved the inode to a dispose list, the other thread thinks it is
> still on the LRU and should be removed, and removes it.
> 
> This also explains the lru item count going negative - the same item
> is being removed from the lru twice. So it seems like all the
> problems you've been seeing are caused by this one problem....
> 
> Patch below that should fix this.

Good news! The test was running since morning and it didn't hang nor
crashed. So this really looks like the right fix. It will run also
during weekend to be 100% sure. But I guess it is safe to say

Tested-by: Michal Hocko <mhocko@suse.cz>

Thanks a lot Dave!

> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
> 
> list_lru: fix broken LRU_RETRY behaviour
> 
> From: Dave Chinner <dchinner@redhat.com>
> 
> The LRU_RETRY code assumes that the list traversal status after we
> have dropped and regained the list lock. Unfortunately, this is not
> a valid assumption, and that can lead to racing traversals isolating
> objects that the other traversal expects to be the next item on the
> list.
> 
> This is causing problems with the inode cache shrinker isolation,
> with races resulting in an inode on a dispose list being "isolated"
> because a racing traversal still thinks it is on the LRU. The inode
> is then never reclaimed and that causes hangs if a subsequent lookup
> on that inode occurs.
> 
> Fix it by always restarting the list walk on a LRU_RETRY return from
> the isolate callback. Avoid the possibility of livelocks the current
> code was trying to aavoid by always decrementing the nr_to_walk
> counter on retries so that even if we keep hitting the same item on
> the list we'll eventually stop trying to walk and exit out of the
> situation causing the problem.
> 
> Reported-by: Michal Hocko <mhocko@suse.cz>
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>  mm/list_lru.c |   29 ++++++++++++-----------------
>  1 file changed, 12 insertions(+), 17 deletions(-)
> 
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index dc71659..7246791 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -71,19 +71,19 @@ list_lru_walk_node(struct list_lru *lru, int nid, list_lru_walk_cb isolate,
>  	struct list_lru_node	*nlru = &lru->node[nid];
>  	struct list_head *item, *n;
>  	unsigned long isolated = 0;
> -	/*
> -	 * If we don't keep state of at which pass we are, we can loop at
> -	 * LRU_RETRY, since we have no guarantees that the caller will be able
> -	 * to do something other than retry on the next pass. We handle this by
> -	 * allowing at most one retry per object. This should not be altered
> -	 * by any condition other than LRU_RETRY.
> -	 */
> -	bool first_pass = true;
>  
>  	spin_lock(&nlru->lock);
>  restart:
>  	list_for_each_safe(item, n, &nlru->list) {
>  		enum lru_status ret;
> +
> +		/*
> +		 * decrement nr_to_walk first so that we don't livelock if we
> +		 * get stuck on large numbesr of LRU_RETRY items
> +		 */
> +		if (--(*nr_to_walk) == 0)
> +			break;
> +
>  		ret = isolate(item, &nlru->lock, cb_arg);
>  		switch (ret) {
>  		case LRU_REMOVED:
> @@ -98,19 +98,14 @@ restart:
>  		case LRU_SKIP:
>  			break;
>  		case LRU_RETRY:
> -			if (!first_pass) {
> -				first_pass = true;
> -				break;
> -			}
> -			first_pass = false;
> +			/*
> +			 * The lru lock has been dropped, our list traversal is
> +			 * now invalid and so we have to restart from scratch.
> +			 */
>  			goto restart;
>  		default:
>  			BUG();
>  		}
> -
> -		if ((*nr_to_walk)-- == 0)
> -			break;
> -
>  	}
>  
>  	spin_unlock(&nlru->lock);

-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

[-- Attachment #1: Type: text/plain, Size: 12797 bytes --]

On Thu 04-07-13 18:36:43, Michal Hocko wrote:
> On Wed 03-07-13 21:24:03, Dave Chinner wrote:
> > On Tue, Jul 02, 2013 at 02:44:27PM +0200, Michal Hocko wrote:
> > > On Tue 02-07-13 22:19:47, Dave Chinner wrote:
> > > [...]
> > > > Ok, so it's been leaked from a dispose list somehow. Thanks for the
> > > > info, Michal, it's time to go look at the code....
> > > 
> > > OK, just in case we will need it, I am keeping the machine in this state
> > > for now. So we still can play with crash and check all the juicy
> > > internals.
> > 
> > My current suspect is the LRU_RETRY code. I don't think what it is
> > doing is at all valid - list_for_each_safe() is not safe if you drop
> > the lock that protects the list. i.e. there is nothing that protects
> > the stored next pointer from being removed from the list by someone
> > else. Hence what I think is occurring is this:
> > 
> > 
> > thread 1			thread 2
> > lock(lru)
> > list_for_each_safe(lru)		lock(lru)
> >   isolate			......
> >     lock(i_lock)
> >     has buffers
> >       __iget
> >       unlock(i_lock)
> >       unlock(lru)
> >       .....			(gets lru lock)
> >       				list_for_each_safe(lru)
> > 				  walks all the inodes
> > 				  finds inode being isolated by other thread
> > 				  isolate
> > 				    i_count > 0
> > 				      list_del_init(i_lru)
> > 				      return LRU_REMOVED;
> > 				   moves to next inode, inode that
> > 				   other thread has stored as next
> > 				   isolate
> > 				     i_state |= I_FREEING
> > 				     list_move(dispose_list)
> > 				     return LRU_REMOVED
> > 				 ....
> > 				 unlock(lru)
> >       lock(lru)
> >       return LRU_RETRY;
> >   if (!first_pass)
> >     ....
> >   --nr_to_scan
> >   (loop again using next, which has already been removed from the
> >   LRU by the other thread!)
> >   isolate
> >     lock(i_lock)
> >     if (i_state & ~I_REFERENCED)
> >       list_del_init(i_lru)	<<<<< inode is on dispose list!
> > 				<<<<< inode is now isolated, with I_FREEING set
> >       return LRU_REMOVED;
> > 
> > That fits the corpse left on your machine, Michal. One thread has
> > moved the inode to a dispose list, the other thread thinks it is
> > still on the LRU and should be removed, and removes it.
> > 
> > This also explains the lru item count going negative - the same item
> > is being removed from the lru twice. So it seems like all the
> > problems you've been seeing are caused by this one problem....
> > 
> > Patch below that should fix this.
> 
> Good news! The test was running since morning and it didn't hang nor
> crashed. So this really looks like the right fix. It will run also
> during weekend to be 100% sure. But I guess it is safe to say

Hmm, it seems I was too optimistic or we have yet another issue here (I
guess the later is more probable).

The weekend testing got stuck as well. 

The dmesg shows there were some hung tasks:
[275284.264312] start.sh (11025): dropped kernel caches: 3
[276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
[276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
[276962.652102]  ffff88003794d198 0000000000000046 ffff8800325f4480 0000000000000000
[276962.652113]  ffff88003794c010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[276962.652121]  0000000000012dc0 ffff88003794dfd8 ffff88003794dfd8 0000000000012dc0
[276962.652128] Call Trace:
[276962.652151]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[276962.652160]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[276962.652171]  [<ffffffff815793a4>] schedule+0x24/0x70
[276962.652178]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[276962.652187]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[276962.652194]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[276962.652200]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[276962.652206]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[276962.652215]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[276962.652224]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[276962.652231]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[276962.652239]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[276962.652246]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[276962.652254]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[276962.652262]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[276962.652274]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[276962.652282]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[276962.652291]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[276962.652298]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[276962.652305]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[276962.652313]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[276962.652323]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[276962.652332]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[276962.652343]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
[276962.652351]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
[276962.652359]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
[276962.652367]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
[276962.652490]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[276962.652540]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
[276962.652588]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[276962.652653]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
[276962.652714]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
[276962.652774]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
[276962.652834]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
[276962.652894]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
[276962.652935]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
[276962.652947]  [<ffffffff81076bd0>] ? __migrate_task+0x150/0x150
[276962.652988]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
[276962.652997]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
[276962.653004]  [<ffffffff8105f294>] rescuer_thread+0x234/0x320
[276962.653011]  [<ffffffff8105f060>] ? free_pwqs+0x30/0x30
[276962.653017]  [<ffffffff81066a86>] kthread+0xc6/0xd0
[276962.653025]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[276962.653034]  [<ffffffff8158303c>] ret_from_fork+0x7c/0xb0
[276962.653041]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70

$ dmesg | grep "blocked for more than"
[276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
[276962.653097] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
[276962.653940] INFO: task ld:14442 blocked for more than 480 seconds.
[276962.654297] INFO: task ld:14962 blocked for more than 480 seconds.
[277442.652123] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
[277442.653153] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
[277442.653997] INFO: task ld:14442 blocked for more than 480 seconds.
[277442.654353] INFO: task ld:14962 blocked for more than 480 seconds.
[277922.652069] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
[277922.653089] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.

All of them are sitting in io_schedule triggered from the memcg soft
reclaim waiting for a wake up (full dmesg is attached). I guess this has
nothing to do with the slab shrinkers directly. It is probably priority 0
reclaim which is done in the soft reclaim path.

$ uptime
 13:32pm  up 4 days  2:54,  2 users,  load average: 25.00, 24.97, 24.66

so the current timestamp should be 352854 which means that all of them
happened quite some time ago and the system obviously resurrected from
this state.

What is more important, though, is that we still have the following
tasks stuck in D state for hours:
14442 [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[<ffffffff81114e43>] shrink_page_list+0x503/0x790
[<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[<ffffffff81116418>] shrink_zones+0x108/0x220
[<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[<ffffffff8113fe93>] read_swap_cache_async+0x113/0x160
[<ffffffff8113ffe1>] swapin_readahead+0x101/0x190
[<ffffffff8112e93f>] do_swap_page+0xef/0x5e0
[<ffffffff8112f94d>] handle_pte_fault+0x1bd/0x240
[<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[<ffffffff8157b348>] page_fault+0x28/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
14962 [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[<ffffffff81114e43>] shrink_page_list+0x503/0x790
[<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[<ffffffff81116418>] shrink_zones+0x108/0x220
[<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[<ffffffff81129ebb>] do_anonymous_page+0x16b/0x350
[<ffffffff8112f9c5>] handle_pte_fault+0x235/0x240
[<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[<ffffffff8157b348>] page_fault+0x28/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
20757 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
[<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
[<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
[<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
[<ffffffffa02c62d0>] xfs_free_eofblocks+0x180/0x250 [xfs]
[<ffffffffa02c68e6>] xfs_release+0x106/0x1d0 [xfs]
[<ffffffffa02b3b20>] xfs_file_release+0x10/0x20 [xfs]
[<ffffffff8116c86d>] __fput+0xbd/0x240
[<ffffffff8116ca49>] ____fput+0x9/0x10
[<ffffffff81063221>] task_work_run+0xb1/0xe0
[<ffffffff810029e0>] do_notify_resume+0x90/0x1d0
[<ffffffff815833a2>] int_signal+0x12/0x17
[<ffffffffffffffff>] 0xffffffffffffffff
20758 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
[<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
[<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
[<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
[<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
[<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
[<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
[<ffffffff811763dd>] vfs_create+0xad/0xd0
[<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
20761 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
[<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
[<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
[<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
[<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
[<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
[<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
[<ffffffff811763dd>] vfs_create+0xad/0xd0
[<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

Hohmm, now that I am looking at pids of the stuck processes, two of them
14442 and 14962 are mentioned in the soft lockup warnings. It is really
weird that the lockups have stopped quite some time ago (~20h ago).

I am keeping the system in this state in case you want to examine
details via crash again.

Let me know whether you need any further details.

Thanks!
-- 
Michal Hocko
SUSE Labs

[-- Attachment #2: demon.dmesg --]
[-- Type: text/plain, Size: 126792 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.9.0mmotmfix+ (mhocko@noe) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1493 SMP Thu Jul 4 10:20:10 CEST 2013
[    0.000000] Command line: root=/dev/sda2 console=tty0 console=ttyS1,115200 nomodeset resume=/dev/sda1 splash=silent crashkernel= showopts vga=6 mem=1G
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000d2000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ff1ffff] usable
[    0.000000] BIOS-e820: [mem 0x000000007ff20000-0x000000007ff23fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000007ff24000-0x000000007ff7ffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000007ff80000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fff80000-0x00000000ffffffff] reserved
[    0.000000] e820: remove [mem 0x40000000-0xfffffffffffffffe] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] e820: user-defined physical RAM map:
[    0.000000] user: [mem 0x0000000000000000-0x000000000009d7ff] usable
[    0.000000] user: [mem 0x000000000009d800-0x000000000009ffff] reserved
[    0.000000] user: [mem 0x00000000000d2000-0x00000000000fffff] reserved
[    0.000000] user: [mem 0x0000000000100000-0x000000003fffffff] usable
[    0.000000] user: [mem 0x000000007ff20000-0x000000007ff23fff] ACPI data
[    0.000000] user: [mem 0x000000007ff24000-0x000000007ff7ffff] ACPI NVS
[    0.000000] user: [mem 0x000000007ff80000-0x000000007fffffff] reserved
[    0.000000] user: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
[    0.000000] user: [mem 0x00000000fff80000-0x00000000ffffffff] reserved
[    0.000000] SMBIOS 2.34 present.
[    0.000000] DMI: AMD A8440/WARTHOG, BIOS PW2A00-5 09/23/2005
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x40000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-D3FFF write-protect
[    0.000000]   D4000-E3FFF uncachable
[    0.000000]   E4000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0000000000 mask FF80000000 write-back
[    0.000000]   1 disabled
[    0.000000]   2 disabled
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] found SMP MP-table at [mem 0x000f7850-0x000f785f] mapped at [ffff8800000f7850]
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000096000] 96000 size 28672
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x01d27000, 0x01d27fff] PGTABLE
[    0.000000] BRK [0x01d28000, 0x01d28fff] PGTABLE
[    0.000000] BRK [0x01d29000, 0x01d29fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x3fe00000-0x3fffffff]
[    0.000000]  [mem 0x3fe00000-0x3fffffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x3c000000-0x3fdfffff]
[    0.000000]  [mem 0x3c000000-0x3fdfffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x00100000-0x3bffffff]
[    0.000000]  [mem 0x00100000-0x001fffff] page 4k
[    0.000000]  [mem 0x00200000-0x3bffffff] page 2M
[    0.000000] RAMDISK: [mem 0x36cbe000-0x37feffff]
[    0.000000] crashkernel: memory value expected
[    0.000000] ACPI: RSDP 00000000000f7820 00024 (v02 PTLTD )
[    0.000000] ACPI: XSDT 000000007ff2026c 00064 (v01 PTLTD  ? XSDT   06040000  LTP 00000000)
[    0.000000] ACPI: FACP 000000007ff202d0 00074 (v01 AMD    HAMMER   06040000 PTEC 000F4240)
[    0.000000] ACPI: DSDT 000000007ff20344 03107 (v01 AMD-K8  AMDACPI 06040000 MSFT 0100000E)
[    0.000000] ACPI: FACS 000000007ff24fc0 00040
[    0.000000] ACPI: SSDT 000000007ff2344b 007C4 (v01 PTLTD  POWERNOW 06040000  LTP 00000001)
[    0.000000] ACPI: SRAT 000000007ff23c0f 00178 (v01 AMD    HAMMER   06040000 AMD  00000001)
[    0.000000] ACPI: SSDT 000000007ff23d87 000E7 (v01 AMD-K8 AMD-ACPI 06040000  AMD 00000001)
[    0.000000] ACPI: HPET 000000007ff23e6e 00038 (v01 AMD    HAMMER   06040000 PTEC 00000000)
[    0.000000] ACPI: SPCR 000000007ff23ea6 00050 (v01 PTLTD  $UCRTBL$ 06040000 PTL  00000001)
[    0.000000] ACPI: APIC 000000007ff23ef6 000E2 (v01 PTLTD  ? APIC   06040000  LTP 00000000)
[    0.000000] ACPI: BOOT 000000007ff23fd8 00028 (v01 PTLTD  $SBFTBL$ 06040000  LTP 00000001)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x02 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x03 -> Node 1
[    0.000000] SRAT: PXM 2 -> APIC 0x04 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x05 -> Node 2
[    0.000000] SRAT: PXM 3 -> APIC 0x06 -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x07 -> Node 3
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0x1fffffff]
[    0.000000] SRAT: Node 1 PXM 1 [mem 0x20000000-0x3fffffff]
[    0.000000] SRAT: Node 2 PXM 2 [mem 0x40000000-0x5fffffff]
[    0.000000] SRAT: Node 3 PXM 3 [mem 0x60000000-0x7fffffff]
[    0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x1fffffff] -> [mem 0x00000000-0x1fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1fffffff]
[    0.000000]   NODE_DATA [mem 0x1ffec000-0x1fffffff]
[    0.000000] Initmem setup node 1 [mem 0x20000000-0x3fffffff]
[    0.000000]   NODE_DATA [mem 0x3ffec000-0x3fffffff]
[    0.000000] [ffffea0000700000-ffffea00007fffff] potential offnode page_structs
[    0.000000]  [ffffea0000000000-ffffea00007fffff] PMD -> [ffff88001f600000-ffff88001fdfffff] on node 0
[    0.000000]  [ffffea0000800000-ffffea0000dfffff] PMD -> [ffff88003ee00000-ffff88003f3fffff] on node 1
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009cfff]
[    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
[    0.000000]   node   1: [mem 0x20000000-0x3fffffff]
[    0.000000] On node 0 totalpages: 130972
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 22 pages reserved
[    0.000000]   DMA zone: 3996 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1736 pages used for memmap
[    0.000000]   DMA32 zone: 126976 pages, LIFO batch:31
[    0.000000] On node 1 totalpages: 131072
[    0.000000]   DMA32 zone: 1792 pages used for memmap
[    0.000000]   DMA32 zone: 131072 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0xc008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: IOAPIC (id[0x09] address[0xe8000000] gsi_base[24])
[    0.000000] IOAPIC[1]: apic_id 9, version 17, address 0xe8000000, GSI 24-27
[    0.000000] ACPI: IOAPIC (id[0x0a] address[0xe8001000] gsi_base[28])
[    0.000000] IOAPIC[2]: apic_id 10, version 17, address 0xe8001000, GSI 28-31
[    0.000000] ACPI: IOAPIC (id[0x0b] address[0xf8000000] gsi_base[32])
[    0.000000] IOAPIC[3]: apic_id 11, version 17, address 0xf8000000, GSI 32-35
[    0.000000] ACPI: IOAPIC (id[0x0c] address[0xf8001000] gsi_base[36])
[    0.000000] IOAPIC[4]: apic_id 12, version 17, address 0xf8001000, GSI 36-39
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x102282a0 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 56
[    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
[    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000d2000
[    0.000000] PM: Registered nosave memory: 00000000000d2000 - 0000000000100000
[    0.000000] e820: [mem 0x80000000-0xfebfffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:8 nr_node_ids:4
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88001f000000 s80512 r8192 d21888 u1048576
[    0.000000] pcpu-alloc: s80512 r8192 d21888 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 [0] 4 5 [0] 6 7 [1] 2 3 
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 258438
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=/dev/sda2 console=tty0 console=ttyS1,115200 nomodeset resume=/dev/sda1 splash=silent crashkernel= showopts vga=6 mem=1G
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] __ex_table already sorted, skipping sort
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ c1f6000000 size 32 MB
[    0.000000] Aperture beyond 4GB. Ignoring.
[    0.000000] Memory: 999232K/1048176K available (5659K kernel code, 671K rwdata, 2516K rodata, 1268K init, 1252K bss, 48944K reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=8.
[    0.000000] NR_IRQS:33024 nr_irqs:1016 16
[    0.000000] Console: colour VGA+ 80x60
[    0.000000] console [tty0] enabled
[    0.000000] console [ttyS1] enabled
[    0.000000] allocated 4194304 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or sysctl
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2389.785 MHz processor
[    0.000000] tsc: Marking TSC unstable due to TSCs unsynchronized
[    0.020000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4779.57 BogoMIPS (lpj=9559140)
[    0.024003] pid_max: default: 32768 minimum: 301
[    0.028077] Security Framework initialized
[    0.032033] AppArmor: AppArmor initialized
[    0.036092] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.040687] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.044375] Mount-cache hash table entries: 256
[    0.048308] Initializing cgroup subsys cpuacct
[    0.052005] Initializing cgroup subsys memory
[    0.056045] Initializing cgroup subsys devices
[    0.060005] Initializing cgroup subsys freezer
[    0.064004] Initializing cgroup subsys net_cls
[    0.068003] Initializing cgroup subsys blkio
[    0.072003] Initializing cgroup subsys perf_event
[    0.076048] tseg: 007ff80000
[    0.076052] CPU: Physical Processor ID: 0
[    0.080003] CPU: Processor Core ID: 0
[    0.084003] mce: CPU supports 5 MCE banks
[    0.088009] LVT offset 0 assigned for vector 0xf9
[    0.092011] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 4
[    0.092011] Last level dTLB entries: 4KB 512, 2MB 8, 4MB 4
[    0.092011] tlb_flushall_shift: 4
[    0.096105] Freeing SMP alternatives memory: 24K (ffffffff81be6000 - ffffffff81bec000)
[    0.100827] ACPI: Core revision 20130117
[    0.109569] ACPI: All ACPI Tables successfully acquired
[    0.113296] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.157906] smpboot: CPU0: AMD Engineering Sample (fam: 0f, model: 41, stepping: 01)
[    0.172000] Performance Events: AMD PMU driver.
[    0.172003] ... version:                0
[    0.176001] ... bit width:              48
[    0.180001] ... generic registers:      4
[    0.184001] ... value mask:             0000ffffffffffff
[    0.188001] ... max period:             00007fffffffffff
[    0.192000] ... fixed-purpose events:   0
[    0.196001] ... event mask:             000000000000000f
[    0.200717] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    0.208105] smpboot: Booting Node   0, Processors  #1 OK
[    0.305027] smpboot: Booting Node   1, Processors  #2 #3 OK
[    0.488903] smpboot: Booting Node   0, Processors  #4 #5 #6 #7 OK
[    0.852083] Brought up 8 CPUs
[    0.855288] smpboot: Total of 8 processors activated (38238.06 BogoMIPS)
[    0.860420] devtmpfs: initialized
[    0.868206] PM: Registering ACPI NVS region [mem 0x7ff24000-0x7ff7ffff] (376832 bytes)
[    0.872179] RTC time:  8:38:18, date: 07/04/13
[    0.876103] NET: Registered protocol family 16
[    0.880206] node 0 link 1: io port [0, 2fff]
[    0.880209] node 1 link 1: io port [3000, 3fff]
[    0.880212] TOM: 0000000080000000 aka 2048M
[    0.884003] node 0 link 1: mmio [e8000000, e80fffff]
[    0.884006] node 1 link 1: mmio [f8000000, f84fffff]
[    0.884008] node 0 link 1: mmio [e8200000, e85fffff]
[    0.884012] bus: [bus 00-09] on node 0 link 1
[    0.884014] bus: 00 [io  0x0000-0x2fff]
[    0.884016] bus: 00 [io  0x4000-0xffff]
[    0.884017] bus: 00 [mem 0x80000000-0xe81fffff]
[    0.884019] bus: 00 [mem 0xe8200000-0xf7ffffff]
[    0.884021] bus: 00 [mem 0xf8500000-0xfcffffffff]
[    0.884022] bus: [bus 0a-ff] on node 1 link 1
[    0.884024] bus: 0a [io  0x3000-0x3fff]
[    0.884025] bus: 0a [mem 0xf8000000-0xf84fffff]
[    0.884056] ACPI: bus type PCI registered
[    0.888073] PCI: Using configuration type 1 for base access
[    0.893297] bio: create slab <bio-0> at 0
[    0.896088] ACPI: Added _OSI(Module Device)
[    0.900002] ACPI: Added _OSI(Processor Device)
[    0.904001] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.908001] ACPI: Added _OSI(Processor Aggregator Device)
[    0.912463] ACPI: EC: Look up EC in DSDT
[    0.919157] ACPI: Interpreter enabled
[    0.920013] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130117/hwxface-568)
[    0.932002] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20130117/hwxface-568)
[    0.942418] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20130117/hwxface-568)
[    0.952005] ACPI: (supports S0 S1 S5)
[    0.956002] ACPI: Using IOAPIC for interrupt routing
[    0.960184] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
[    0.975743] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-09])
[    0.983122] acpi PNP0A03:00: host bridge window [io  0x03b0-0x03df] (ignored)
[    0.983125] acpi PNP0A03:00: host bridge window [io  0x0d00-0x2fff] (ignored)
[    0.983128] acpi PNP0A03:00: host bridge window [mem 0xfed00000-0xfed0ffff] (ignored)
[    0.983130] acpi PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff] (ignored)
[    0.983133] acpi PNP0A03:00: host bridge window [mem 0xfec00000-0xfec0ffff] (ignored)
[    0.983135] acpi PNP0A03:00: host bridge window [mem 0xf0000000-0xf7ffffff] (ignored)
[    0.983138] acpi PNP0A03:00: host bridge window [mem 0xe8000000-0xe85fffff] (ignored)
[    0.983140] acpi PNP0A03:00: host bridge window [mem 0x000c0000-0x000cafff] (ignored)
[    0.983142] acpi PNP0A03:00: host bridge window [mem 0x000d8000-0x000dbfff] (ignored)
[    0.983145] acpi PNP0A03:00: host bridge window [io  0x0000-0x03af] (ignored)
[    0.983147] acpi PNP0A03:00: host bridge window [io  0x03e0-0x0cf7] (ignored)
[    0.983150] PCI: root bus 00: hardware-probed resources
[    0.983157] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    0.984039] PCI host bridge to bus 0000:00
[    0.988003] pci_bus 0000:00: root bus resource [bus 00-09]
[    0.992003] pci_bus 0000:00: root bus resource [io  0x0000-0x2fff]
[    0.996002] pci_bus 0000:00: root bus resource [io  0x4000-0xffff]
[    1.000002] pci_bus 0000:00: root bus resource [mem 0x80000000-0xe81fffff]
[    1.004002] pci_bus 0000:00: root bus resource [mem 0xe8200000-0xf7ffffff]
[    1.008002] pci_bus 0000:00: root bus resource [mem 0xf8500000-0xfcffffffff]
[    1.012022] pci 0000:00:06.0: [1022:7460] type 01 class 0x060400
[    1.012130] pci 0000:00:06.0: System wakeup disabled by ACPI
[    1.016049] pci 0000:00:07.0: [1022:7468] type 00 class 0x060100
[    1.016157] pci 0000:00:07.1: [1022:7469] type 00 class 0x01018a
[    1.016188] pci 0000:00:07.1: reg 20: [io  0x1020-0x102f]
[    1.016266] pci 0000:00:07.2: [1022:746a] type 00 class 0x0c0500
[    1.016280] pci 0000:00:07.2: reg 10: [io  0x1000-0x101f]
[    1.016409] pci 0000:00:07.3: [1022:746b] type 00 class 0x068000
[    1.016536] pci 0000:00:0a.0: [1022:7450] type 01 class 0x060400
[    1.016601] pci 0000:00:0a.0: System wakeup disabled by ACPI
[    1.020042] pci 0000:00:0a.1: [1022:7451] type 00 class 0x080010
[    1.020051] pci 0000:00:0a.1: reg 10: [mem 0xe8000000-0xe8000fff 64bit]
[    1.020136] pci 0000:00:0b.0: [1022:7450] type 01 class 0x060400
[    1.020200] pci 0000:00:0b.0: System wakeup disabled by ACPI
[    1.024038] pci 0000:00:0b.1: [1022:7451] type 00 class 0x080010
[    1.024047] pci 0000:00:0b.1: reg 10: [mem 0xe8001000-0xe8001fff 64bit]
[    1.024144] pci 0000:00:18.0: [1022:1100] type 00 class 0x060000
[    1.024214] pci 0000:00:18.1: [1022:1101] type 00 class 0x060000
[    1.024279] pci 0000:00:18.2: [1022:1102] type 00 class 0x060000
[    1.024345] pci 0000:00:18.3: [1022:1103] type 00 class 0x060000
[    1.024420] pci 0000:00:19.0: [1022:1100] type 00 class 0x060000
[    1.024479] pci 0000:00:19.1: [1022:1101] type 00 class 0x060000
[    1.024546] pci 0000:00:19.2: [1022:1102] type 00 class 0x060000
[    1.024611] pci 0000:00:19.3: [1022:1103] type 00 class 0x060000
[    1.024683] pci 0000:00:1a.0: [1022:1100] type 00 class 0x060000
[    1.024757] pci 0000:00:1a.1: [1022:1101] type 00 class 0x060000
[    1.024824] pci 0000:00:1a.2: [1022:1102] type 00 class 0x060000
[    1.024890] pci 0000:00:1a.3: [1022:1103] type 00 class 0x060000
[    1.024963] pci 0000:00:1b.0: [1022:1100] type 00 class 0x060000
[    1.025041] pci 0000:00:1b.1: [1022:1101] type 00 class 0x060000
[    1.025108] pci 0000:00:1b.2: [1022:1102] type 00 class 0x060000
[    1.025177] pci 0000:00:1b.3: [1022:1103] type 00 class 0x060000
[    1.025307] pci 0000:01:00.0: [1022:7464] type 00 class 0x0c0310
[    1.025322] pci 0000:01:00.0: reg 10: [mem 0xe8110000-0xe8110fff]
[    1.025395] pci 0000:01:00.0: System wakeup disabled by ACPI
[    1.028050] pci 0000:01:00.1: [1022:7464] type 00 class 0x0c0310
[    1.028064] pci 0000:01:00.1: reg 10: [mem 0xe8111000-0xe8111fff]
[    1.028140] pci 0000:01:00.1: System wakeup disabled by ACPI
[    1.032062] pci 0000:01:05.0: [1095:3512] type 00 class 0x018000
[    1.032080] pci 0000:01:05.0: reg 10: [io  0x2420-0x2427]
[    1.032090] pci 0000:01:05.0: reg 14: [io  0x2414-0x2417]
[    1.032101] pci 0000:01:05.0: reg 18: [io  0x2418-0x241f]
[    1.036010] pci 0000:01:05.0: reg 1c: [io  0x2410-0x2413]
[    1.036020] pci 0000:01:05.0: reg 20: [io  0x2400-0x240f]
[    1.036031] pci 0000:01:05.0: reg 24: [mem 0xe8112000-0xe81121ff]
[    1.036042] pci 0000:01:05.0: reg 30: [mem 0x00000000-0x0007ffff pref]
[    1.036071] pci 0000:01:05.0: supports D1 D2
[    1.036126] pci 0000:01:06.0: [1002:5159] type 00 class 0x030000
[    1.036144] pci 0000:01:06.0: reg 10: [mem 0xf0000000-0xf7ffffff pref]
[    1.036155] pci 0000:01:06.0: reg 14: [io  0x2000-0x20ff]
[    1.036165] pci 0000:01:06.0: reg 18: [mem 0xe8100000-0xe810ffff]
[    1.036201] pci 0000:01:06.0: reg 30: [mem 0x00000000-0x0001ffff pref]
[    1.036231] pci 0000:01:06.0: supports D1 D2
[    1.036310] pci 0000:00:06.0: PCI bridge to [bus 01]
[    1.040005] pci 0000:00:06.0:   bridge window [io  0x2000-0x2fff]
[    1.040009] pci 0000:00:06.0:   bridge window [mem 0xe8100000-0xe81fffff]
[    1.040013] pci 0000:00:06.0:   bridge window [mem 0xf0000000-0xf7ffffff pref]
[    1.040063] pci 0000:02:01.0: [14e4:1645] type 00 class 0x020000
[    1.040074] pci 0000:02:01.0: reg 10: [mem 0xe8200000-0xe820ffff 64bit]
[    1.040117] pci 0000:02:01.0: PME# supported from D3hot D3cold
[    1.040190] pci 0000:00:0a.0: PCI bridge to [bus 02]
[    1.044005] pci 0000:00:0a.0:   bridge window [mem 0xe8200000-0xe82fffff]
[    1.044047] pci 0000:03:02.0: [14e4:1648] type 00 class 0x020000
[    1.044060] pci 0000:03:02.0: reg 10: [mem 0xe8310000-0xe831ffff 64bit]
[    1.044069] pci 0000:03:02.0: reg 18: [mem 0xe8300000-0xe830ffff 64bit]
[    1.044109] pci 0000:03:02.0: PME# supported from D3hot D3cold
[    1.044162] pci 0000:03:02.1: [14e4:1648] type 00 class 0x020000
[    1.044175] pci 0000:03:02.1: reg 10: [mem 0xe8330000-0xe833ffff 64bit]
[    1.044184] pci 0000:03:02.1: reg 18: [mem 0xe8320000-0xe832ffff 64bit]
[    1.044224] pci 0000:03:02.1: PME# supported from D3hot D3cold
[    1.044288] pci 0000:00:0b.0: PCI bridge to [bus 03]
[    1.048005] pci 0000:00:0b.0:   bridge window [mem 0xe8300000-0xe83fffff]
[    1.048020] acpi PNP0A03:00: ACPI _OSC support notification failed, disabling PCIe ASPM
[    1.052002] acpi PNP0A03:00: Unable to request _OSC control (_OSC support mask: 0x08)
[    1.056269] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 5 10 *11)
[    1.064174] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 5 *10 11)
[    1.070061] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 *5 10 11)
[    1.077462] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 5 10 *11)
[    1.086416] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 0a-ff])
[    1.092211] acpi PNP0A03:01: host bridge window [mem 0xf8000000-0xf84fffff] (ignored)
[    1.092214] acpi PNP0A03:01: host bridge window [io  0x3000-0x3fff] (ignored)
[    1.092217] PCI: root bus 0a: hardware-probed resources
[    1.092220] acpi PNP0A03:01: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    1.096039] PCI host bridge to bus 0000:0a
[    1.100002] pci_bus 0000:0a: root bus resource [bus 0a-ff]
[    1.104002] pci_bus 0000:0a: root bus resource [io  0x3000-0x3fff]
[    1.108002] pci_bus 0000:0a: root bus resource [mem 0xf8000000-0xf84fffff]
[    1.112014] pci 0000:0a:01.0: [1022:7450] type 01 class 0x060400
[    1.112074] pci 0000:0a:01.0: System wakeup disabled by ACPI
[    1.116053] pci 0000:0a:01.1: [1022:7451] type 00 class 0x080010
[    1.116063] pci 0000:0a:01.1: reg 10: [mem 0xf8000000-0xf8000fff 64bit]
[    1.116135] pci 0000:0a:02.0: [1022:7450] type 01 class 0x060400
[    1.116186] pci 0000:0a:02.0: System wakeup disabled by ACPI
[    1.120039] pci 0000:0a:02.1: [1022:7451] type 00 class 0x080010
[    1.120049] pci 0000:0a:02.1: reg 10: [mem 0xf8001000-0xf8001fff 64bit]
[    1.120168] pci 0000:0b:01.0: [8086:107c] type 00 class 0x020000
[    1.120180] pci 0000:0b:01.0: reg 10: [mem 0xf8120000-0xf813ffff]
[    1.120187] pci 0000:0b:01.0: reg 14: [mem 0xf8100000-0xf811ffff]
[    1.120193] pci 0000:0b:01.0: reg 18: [io  0x3000-0x303f]
[    1.120212] pci 0000:0b:01.0: reg 30: [mem 0x00000000-0x0001ffff pref]
[    1.120234] pci 0000:0b:01.0: PME# supported from D0 D3hot D3cold
[    1.120295] pci 0000:0a:01.0: PCI bridge to [bus 0b-0f]
[    1.124005] pci 0000:0a:01.0:   bridge window [io  0x3000-0x3fff]
[    1.124008] pci 0000:0a:01.0:   bridge window [mem 0xf8100000-0xf81fffff]
[    1.124053] pci 0000:10:02.0: [14e4:1648] type 00 class 0x020000
[    1.124066] pci 0000:10:02.0: reg 10: [mem 0xf8210000-0xf821ffff 64bit]
[    1.124075] pci 0000:10:02.0: reg 18: [mem 0xf8200000-0xf820ffff 64bit]
[    1.124118] pci 0000:10:02.0: PME# supported from D3hot D3cold
[    1.124175] pci 0000:10:02.1: [14e4:1648] type 00 class 0x020000
[    1.124188] pci 0000:10:02.1: reg 10: [mem 0xf8230000-0xf823ffff 64bit]
[    1.124198] pci 0000:10:02.1: reg 18: [mem 0xf8220000-0xf822ffff 64bit]
[    1.124241] pci 0000:10:02.1: PME# supported from D3hot D3cold
[    1.124307] pci 0000:0a:02.0: PCI bridge to [bus 10-14]
[    1.128005] pci 0000:0a:02.0:   bridge window [mem 0xf8200000-0xf82fffff]
[    1.128016] acpi PNP0A03:01: ACPI _OSC support notification failed, disabling PCIe ASPM
[    1.132002] acpi PNP0A03:01: Unable to request _OSC control (_OSC support mask: 0x08)
[    1.136084] acpi root: \_SB_.PCI0 notify handler is installed
[    1.136102] acpi root: \_SB_.PCI1 notify handler is installed
[    1.136106] Found 2 acpi root devices
[    1.136203] vgaarb: device added: PCI:0000:01:06.0,decodes=io+mem,owns=io+mem,locks=none
[    1.140003] vgaarb: loaded
[    1.144001] vgaarb: bridge control possible 0000:01:06.0
[    1.148216] SCSI subsystem initialized
[    1.152003] ACPI: bus type ATA registered
[    1.156025] libata version 3.00 loaded.
[    1.156116] PCI: Using ACPI for IRQ routing
[    1.160003] PCI: pci_cache_line_size set to 64 bytes
[    1.160076] e820: reserve RAM buffer [mem 0x0009d800-0x0009ffff]
[    1.160215] NetLabel: Initializing
[    1.164002] NetLabel:  domain hash size = 128
[    1.168001] NetLabel:  protocols = UNLABELED CIPSOv4
[    1.172017] NetLabel:  unlabeled traffic allowed by default
[    1.176037] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    1.180008] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 19
[    1.185413] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[    1.192023] Switching to clocksource hpet
[    1.200278] AppArmor: AppArmor Filesystem Enabled
[    1.205370] pnp: PnP ACPI init
[    1.208719] ACPI: bus type PNP registered
[    1.213356] pnp 00:00: [dma 4]
[    1.213415] pnp 00:00: Plug and Play ACPI device, IDs PNP0200 (active)
[    1.213487] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active)
[    1.213524] pnp 00:02: Plug and Play ACPI device, IDs PNP0800 (active)
[    1.213571] pnp 00:03: Plug and Play ACPI device, IDs PNP0c04 (active)
[    1.213769] system 00:04: [io  0x04d0-0x04d1] has been reserved
[    1.219983] system 00:04: [io  0x1100-0x117f] has been reserved
[    1.226192] system 00:04: [io  0x1180-0x11ff] has been reserved
[    1.232403] system 00:04: [io  0x0b78-0x0b7b] has been reserved
[    1.238618] system 00:04: [io  0x0190-0x0193] has been reserved
[    1.244830] system 00:04: [io  0xde00-0xdef7] has been reserved
[    1.251046] system 00:04: [io  0x0ca2-0x0ca3] has been reserved
[    1.257258] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active)
[    1.257344] pnp 00:05: Plug and Play ACPI device, IDs PNP0f13 (active)
[    1.258174] system 00:06: [mem 0x000e0000-0x000fffff] could not be reserved
[    1.265444] system 00:06: [mem 0x000c0000-0x000cafff] could not be reserved
[    1.272710] system 00:06: [mem 0xfec00000-0xfec00fff] could not be reserved
[    1.279974] system 00:06: [mem 0xffc00000-0xfff7ffff] has been reserved
[    1.286891] system 00:06: [mem 0xfee00000-0xfee00fff] has been reserved
[    1.293806] system 00:06: [mem 0xfff80000-0xffffffff] has been reserved
[    1.300721] system 00:06: [mem 0xfe000000-0xfe000fff] has been reserved
[    1.307636] system 00:06: [mem 0xfe001000-0xfe001fff] has been reserved
[    1.314553] system 00:06: [mem 0xfe002000-0xfe003fff] has been reserved
[    1.321465] system 00:06: [mem 0xfe004000-0xfe004fff] has been reserved
[    1.328380] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    1.328428] pnp 00:07: Plug and Play ACPI device, IDs PNP0303 (active)
[    1.328911] pnp 00:08: Plug and Play ACPI device, IDs PNP0501 (active)
[    1.329374] pnp 00:09: Plug and Play ACPI device, IDs PNP0501 (active)
[    1.329592] pnp 00:0a: Plug and Play ACPI device, IDs PNP0700 (disabled)
[    1.329733] pnp 00:0b: Plug and Play ACPI device, IDs PNP0501 (disabled)
[    1.329849] pnp 00:0c: Plug and Play ACPI device, IDs PNP0700 (disabled)
[    1.329916] pnp: PnP ACPI: found 13 devices
[    1.334373] ACPI: bus type PNP unregistered
[    1.347541] pci 0000:01:05.0: BAR 6: assigned [mem 0xe8180000-0xe81fffff pref]
[    1.355239] pci 0000:01:06.0: BAR 6: assigned [mem 0xe8120000-0xe813ffff pref]
[    1.362920] pci 0000:00:06.0: PCI bridge to [bus 01]
[    1.368169] pci 0000:00:06.0:   bridge window [io  0x2000-0x2fff]
[    1.374567] pci 0000:00:06.0:   bridge window [mem 0xe8100000-0xe81fffff]
[    1.381659] pci 0000:00:06.0:   bridge window [mem 0xf0000000-0xf7ffffff pref]
[    1.389344] pci 0000:00:0a.0: PCI bridge to [bus 02]
[    1.394590] pci 0000:00:0a.0:   bridge window [mem 0xe8200000-0xe82fffff]
[    1.401686] pci 0000:00:0b.0: PCI bridge to [bus 03]
[    1.406940] pci 0000:00:0b.0:   bridge window [mem 0xe8300000-0xe83fffff]
[    1.414042] pci 0000:0a:01.0: BAR 15: assigned [mem 0xf8300000-0xf83fffff pref]
[    1.421818] pci 0000:0b:01.0: BAR 6: assigned [mem 0xf8300000-0xf831ffff pref]
[    1.429501] pci 0000:0a:01.0: PCI bridge to [bus 0b-0f]
[    1.435016] pci 0000:0a:01.0:   bridge window [io  0x3000-0x3fff]
[    1.441404] pci 0000:0a:01.0:   bridge window [mem 0xf8100000-0xf81fffff]
[    1.448492] pci 0000:0a:01.0:   bridge window [mem 0xf8300000-0xf83fffff pref]
[    1.456177] pci 0000:0a:02.0: PCI bridge to [bus 10-14]
[    1.461687] pci 0000:0a:02.0:   bridge window [mem 0xf8200000-0xf82fffff]
[    1.468806] pci_bus 0000:00: resource 4 [io  0x0000-0x2fff]
[    1.468809] pci_bus 0000:00: resource 5 [io  0x4000-0xffff]
[    1.468811] pci_bus 0000:00: resource 6 [mem 0x80000000-0xe81fffff]
[    1.468814] pci_bus 0000:00: resource 7 [mem 0xe8200000-0xf7ffffff]
[    1.468816] pci_bus 0000:00: resource 8 [mem 0xf8500000-0xfcffffffff]
[    1.468819] pci_bus 0000:01: resource 0 [io  0x2000-0x2fff]
[    1.468822] pci_bus 0000:01: resource 1 [mem 0xe8100000-0xe81fffff]
[    1.468824] pci_bus 0000:01: resource 2 [mem 0xf0000000-0xf7ffffff pref]
[    1.468827] pci_bus 0000:02: resource 1 [mem 0xe8200000-0xe82fffff]
[    1.468830] pci_bus 0000:03: resource 1 [mem 0xe8300000-0xe83fffff]
[    1.468833] pci_bus 0000:0a: resource 4 [io  0x3000-0x3fff]
[    1.468836] pci_bus 0000:0a: resource 5 [mem 0xf8000000-0xf84fffff]
[    1.468839] pci_bus 0000:0b: resource 0 [io  0x3000-0x3fff]
[    1.468841] pci_bus 0000:0b: resource 1 [mem 0xf8100000-0xf81fffff]
[    1.468843] pci_bus 0000:0b: resource 2 [mem 0xf8300000-0xf83fffff pref]
[    1.468847] pci_bus 0000:10: resource 1 [mem 0xf8200000-0xf82fffff]
[    1.469098] NET: Registered protocol family 2
[    1.474226] TCP established hash table entries: 8192 (order: 5, 131072 bytes)
[    1.481786] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[    1.488714] TCP: Hash tables configured (established 8192 bind 8192)
[    1.501205] TCP: reno registered
[    1.504707] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    1.510940] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    1.517818] NET: Registered protocol family 1
[    1.522486] pci 0000:00:07.3: boot interrupts on device [1022:746b] already disabled
[    1.530705] pci 0000:00:0a.0: MSI quirk detected; subordinate MSI disabled
[    1.537886] pci 0000:00:0a.0: AMD8131 rev 12 detected; disabling PCI-X MMRBC
[    1.545244] pci 0000:00:0b.0: MSI quirk detected; subordinate MSI disabled
[    1.552422] pci 0000:00:0b.0: AMD8131 rev 12 detected; disabling PCI-X MMRBC
[    2.104095] pci 0000:01:06.0: Boot video device
[    2.104109] pci 0000:0a:01.0: MSI quirk detected; subordinate MSI disabled
[    2.111287] pci 0000:0a:01.0: AMD8131 rev 12 detected; disabling PCI-X MMRBC
[    2.118640] pci 0000:0a:02.0: MSI quirk detected; subordinate MSI disabled
[    2.125816] pci 0000:0a:02.0: AMD8131 rev 12 detected; disabling PCI-X MMRBC
[    2.133179] PCI: CLS 64 bytes, default 64
[    2.133295] Unpacking initramfs...
[    2.586419] Freeing initrd memory: 19656K (ffff880036cbe000 - ffff880037ff0000)
[    2.595778] Simple Boot Flag at 0x39 set to 0x1
[    2.601497] Scanning for low memory corruption every 60 seconds
[    2.608067] audit: initializing netlink socket (disabled)
[    2.613755] type=2000 audit(1372927099.612:1): initialized
[    2.644783] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    2.652373] VFS: Disk quotas dquot_6.5.2
[    2.656612] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.664024] msgmni has been set to 1990
[    2.668593] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    2.676506] io scheduler noop registered
[    2.680696] io scheduler deadline registered
[    2.685299] io scheduler cfq registered (default)
[    2.690638] ioapic: probe of 0000:00:0a.1 failed with error -22
[    2.696870] ioapic: probe of 0000:00:0b.1 failed with error -22
[    2.703112] ioapic: probe of 0000:0a:01.1 failed with error -22
[    2.709339] ioapic: probe of 0000:0a:02.1 failed with error -22
[    2.715710] GHES: HEST is not enabled!
[    2.719800] Serial: 8250/16550 driver, 32 ports, IRQ sharing disabled
[    2.746996] 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.773340] 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.779280] serial 00:0b: [io  0x03e8-0x03ef]
[    2.779404] serial 00:0b: [io  0x02e8-0x02ef]
[    2.779503] serial 00:0b: unable to assign resources
[    2.784738] serial: probe of 00:0b failed with error -16
[    2.792489] Non-volatile memory driver v1.3
[    2.796936] Linux agpgart interface v0.103
[    2.801533] libphy: Fixed MDIO Bus: probed
[    2.805946] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    2.817055] serio: i8042 KBD port at 0x60,0x64 irq 1
[    2.822296] serio: i8042 AUX port at 0x60,0x64 irq 12
[    2.827797] mousedev: PS/2 mouse device common for all mice
[    2.833825] rtc_cmos 00:01: RTC can wake from S4
[    2.838908] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
[    2.845346] rtc_cmos 00:01: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
[    2.853468] cpuidle: using governor ladder
[    2.857825] cpuidle: using governor menu
[    2.862011] ledtrig-cpu: registered to indicate activity on CPUs
[    2.868296] EFI Variables Facility v0.08 2004-May-17
[    2.873545] hidraw: raw HID events driver (C) Jiri Kosina
[    2.879387] TCP: cubic registered
[    2.883152] NET: Registered protocol family 10
[    2.888169] Key type dns_resolver registered
[    2.893306] PM: Checking hibernation image partition /dev/sda1
[    2.906270] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[    3.163543] PM: Hibernation image not present or could not be loaded.
[    3.163562] registered taskstats version 1
[    3.168579]   Magic number: 5:732:623
[    3.174791] Freeing unused kernel memory: 1268K (ffffffff81aa9000 - ffffffff81be6000)
[    3.183115] Write protecting the kernel read-only data: 10240k
[    3.191096] Freeing unused kernel memory: 476K (ffff880001589000 - ffff880001600000)
[    3.204841] Freeing unused kernel memory: 1580K (ffff880001875000 - ffff880001a00000)
[    3.282824] pata_amd 0000:00:07.1: version 0.4.1
[    3.283218] scsi0 : pata_amd
[    3.286618] scsi1 : pata_amd
[    3.289810] ata1: PATA max UDMA/133 cmd 0x1f0 ctl 0x3f6 bmdma 0x1020 irq 14
[    3.297063] ata2: PATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0x1028 irq 15
[    3.313835] sata_sil 0000:01:05.0: version 2.4
[    3.314338] scsi2 : sata_sil
[    3.317630] scsi3 : sata_sil
[    3.320801] ata3: SATA max UDMA/100 mmio m512@0xe8112000 tf 0xe8112080 irq 16
[    3.328228] ata4: SATA max UDMA/100 mmio m512@0xe8112000 tf 0xe81120c0 irq 16
[    3.355901] hp_sw: device handler registered
[    3.369879] rdac: device handler registered
[    3.383238] emc: device handler registered
[    3.392495] udev: starting version 147
[    3.424913] ACPI: bus type USB registered
[    3.429359] usbcore: registered new interface driver usbfs
[    3.435206] usbcore: registered new interface driver hub
[    3.440978] usbcore: registered new device driver usb
[    3.453177] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.464950] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    3.468822] ata1.00: ATAPI: DV-28E-N, 1.6A, max UDMA/33
[    3.477402] ohci_hcd 0000:01:00.0: OHCI Host Controller
[    3.482957] ohci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
[    3.484381] ata1.00: configured for UDMA/33
[    3.489297] scsi 0:0:0:0: CD-ROM            TEAC     DV-28E-N         1.6A PQ: 0 ANSI: 5
[    3.489621] ata2: port disabled--ignoring
[    3.503986] ohci_hcd 0000:01:00.0: irq 19, io mem 0xe8110000
[    3.566061] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
[    3.573139] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.580814] usb usb1: Product: OHCI Host Controller
[    3.585954] usb usb1: Manufacturer: Linux 3.9.0mmotmfix+ ohci_hcd
[    3.592330] usb usb1: SerialNumber: 0000:01:00.0
[    3.597369] hub 1-0:1.0: USB hub found
[    3.601384] hub 1-0:1.0: 3 ports detected
[    3.605897] ohci_hcd 0000:01:00.1: OHCI Host Controller
[    3.611396] ohci_hcd 0000:01:00.1: new USB bus registered, assigned bus number 2
[    3.619249] ohci_hcd 0000:01:00.1: irq 19, io mem 0xe8111000
[    3.682036] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[    3.689109] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.696771] usb usb2: Product: OHCI Host Controller
[    3.701911] usb usb2: Manufacturer: Linux 3.9.0mmotmfix+ ohci_hcd
[    3.708279] usb usb2: SerialNumber: 0000:01:00.1
[    3.713280] hub 2-0:1.0: USB hub found
[    3.717298] hub 2-0:1.0: 3 ports detected
[    4.096036] usb 2-2: new full-speed USB device number 2 using ohci_hcd
[    4.307966] usb 2-2: New USB device found, idVendor=04b4, idProduct=6560
[    4.314963] usb 2-2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.323999] hub 2-2:1.0: USB hub found
[    4.329956] hub 2-2:1.0: 4 ports detected
[    4.384061] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[    4.396478] ata3.00: ATA-7: Maxtor 6Y120M0, YAR51EW0, max UDMA/133
[    4.402937] ata3.00: 234375000 sectors, multi 0: LBA 
[    4.424464] ata3.00: configured for UDMA/100
[    4.429326] scsi 2:0:0:0: Direct-Access     ATA      Maxtor 6Y120M0   YAR5 PQ: 0 ANSI: 5
[    4.438188] sd 2:0:0:0: [sda] 234375000 512-byte logical blocks: (120 GB/111 GiB)
[    4.446318] sd 2:0:0:0: [sda] Write Protect is off
[    4.451428] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    4.451492] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    4.524038] usb 2-3: new full-speed USB device number 3 using ohci_hcd
[    4.540827]  sda: sda1 sda2 sda4 < sda5 sda6 sda7 sda8 sda9 >
[    4.547749] sd 2:0:0:0: [sda] Attached SCSI disk
[    4.731896] usb 2-3: New USB device found, idVendor=04b4, idProduct=6560
[    4.744323] usb 2-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.753928] hub 2-3:1.0: USB hub found
[    4.759883] hub 2-3:1.0: 4 ports detected
[    4.772074] ata4: SATA link down (SStatus 0 SControl 310)
[    6.003082] EXT4-fs (sda2): mounting ext3 file system using the ext4 subsystem
[    6.025866] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: acl,user_xattr
[    6.159096] EXT4-fs (sda2): re-mounted. Opts: acl,user_xattr
[    8.080592] udev: starting version 147
[    8.265123] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[    8.274118] ACPI: Power Button [PWRB]
[    8.278259] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[    8.286200] ACPI: Power Button [PWRF]
[    8.616482] input: PC Speaker as /devices/platform/pcspkr/input/input3
[    8.671970] sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
[    8.678451] cdrom: Uniform CD-ROM driver Revision: 3.20
[    8.684196] sr 0:0:0:0: Attached scsi CD-ROM sr0
[    8.740111] microcode: AMD CPU family 0xf not supported
[    8.746473] microcode: AMD CPU family 0xf not supported
[    8.757933] microcode: AMD CPU family 0xf not supported
[    8.764894] microcode: AMD CPU family 0xf not supported
[    8.771615] microcode: AMD CPU family 0xf not supported
[    8.778899] microcode: AMD CPU family 0xf not supported
[    8.785345] microcode: AMD CPU family 0xf not supported
[    8.791100] sr 0:0:0:0: Attached scsi generic sg0 type 5
[    8.796804] sd 2:0:0:0: Attached scsi generic sg1 type 0
[    8.803229] microcode: AMD CPU family 0xf not supported
[    8.975186] MCE: In-kernel MCE decoding enabled.
[    9.117064] k8temp 0000:00:18.3: Temperature readouts might be wrong - check erratum #141
[    9.125970] k8temp 0000:00:19.3: Temperature readouts might be wrong - check erratum #141
[    9.134931] k8temp 0000:00:1a.3: Temperature readouts might be wrong - check erratum #141
[    9.143797] k8temp 0000:00:1b.3: Temperature readouts might be wrong - check erratum #141
[    9.250753] pci 0000:00:07.3: AMD GPIO region 0xc0b0 already in use!
[    9.327627] AMD768 RNG detected
[    9.734197] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    9.740557] EDAC MC: Ver: 3.0.0
[    9.988372] AMD64 EDAC driver v3.4.0
[    9.992281] EDAC amd64: DRAM ECC disabled.
[    9.996793] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[    9.996793]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[    9.996793]  (Note that use of the override may cause unknown side effects.)
[   10.022838] EDAC amd64: DRAM ECC disabled.
[   10.027301] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.027301]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.027301]  (Note that use of the override may cause unknown side effects.)
[   10.053552] EDAC amd64: DRAM ECC disabled.
[   10.057976] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.057976]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.057976]  (Note that use of the override may cause unknown side effects.)
[   10.084231] EDAC amd64: DRAM ECC disabled.
[   10.088647] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.088647]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.088647]  (Note that use of the override may cause unknown side effects.)
[   10.115314] shpchp 0000:00:0a.0: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
[   10.123619] shpchp 0000:00:0a.0: Cannot reserve MMIO region
[   10.129877] shpchp 0000:00:0b.0: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
[   10.138158] shpchp 0000:00:0b.0: Cannot reserve MMIO region
[   10.144129] shpchp 0000:0a:01.0: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
[   10.152522] shpchp 0000:0a:01.0: Cannot reserve MMIO region
[   10.159498] shpchp 0000:0a:02.0: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
[   10.167858] shpchp 0000:0a:02.0: Cannot reserve MMIO region
[   10.173766] AMD64 EDAC driver v3.4.0
[   10.173850] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[   10.187096] EDAC amd64: DRAM ECC disabled.
[   10.191592] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.191592]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.191592]  (Note that use of the override may cause unknown side effects.)
[   10.218418] EDAC amd64: DRAM ECC disabled.
[   10.222891] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.222891]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.222891]  (Note that use of the override may cause unknown side effects.)
[   10.248729] EDAC amd64: DRAM ECC disabled.
[   10.253142] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.253142]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.253142]  (Note that use of the override may cause unknown side effects.)
[   10.278923] EDAC amd64: DRAM ECC disabled.
[   10.283447] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.283447]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.283447]  (Note that use of the override may cause unknown side effects.)
[   10.313313] AMD64 EDAC driver v3.4.0
[   10.317345] EDAC amd64: DRAM ECC disabled.
[   10.322145] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.322145]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.322145]  (Note that use of the override may cause unknown side effects.)
[   10.348285] EDAC amd64: DRAM ECC disabled.
[   10.352720] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.352720]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.352720]  (Note that use of the override may cause unknown side effects.)
[   10.379749] EDAC amd64: DRAM ECC disabled.
[   10.384220] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.384220]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.384220]  (Note that use of the override may cause unknown side effects.)
[   10.410363] EDAC amd64: DRAM ECC disabled.
[   10.414771] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.414771]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.414771]  (Note that use of the override may cause unknown side effects.)
[   10.445089] AMD64 EDAC driver v3.4.0
[   10.449040] EDAC amd64: DRAM ECC disabled.
[   10.453410] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.453410]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.453410]  (Note that use of the override may cause unknown side effects.)
[   10.484632] EDAC amd64: DRAM ECC disabled.
[   10.489036] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.489036]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.489036]  (Note that use of the override may cause unknown side effects.)
[   10.514589] EDAC amd64: DRAM ECC disabled.
[   10.518963] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.518963]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.518963]  (Note that use of the override may cause unknown side effects.)
[   10.544495] EDAC amd64: DRAM ECC disabled.
[   10.548858] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   10.548858]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   10.548858]  (Note that use of the override may cause unknown side effects.)
[   10.786265] pps_core: LinuxPPS API ver. 1 registered
[   10.791541] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[   10.891380] kvm: Nested Virtualization enabled
[   10.973977] PTP clock support registered
[   11.153593] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[   11.161004] e1000: Copyright (c) 1999-2006 Intel Corporation.
[   11.167467] powernow-k8: fid 0x10 (2400 MHz), vid 0x8
[   11.173090] powernow-k8: fid 0xe (2200 MHz), vid 0xa
[   11.178409] powernow-k8: fid 0xc (2000 MHz), vid 0xc
[   11.184261] powernow-k8: fid 0xa (1800 MHz), vid 0xe
[   11.189561] powernow-k8: fid 0x2 (1000 MHz), vid 0x12
[   11.194993] powernow-k8: fid 0x10 (2400 MHz), vid 0x8
[   11.200311] powernow-k8: fid 0xe (2200 MHz), vid 0xa
[   11.205714] powernow-k8: fid 0xc (2000 MHz), vid 0xc
[   11.211250] powernow-k8: fid 0xa (1800 MHz), vid 0xe
[   11.216669] powernow-k8: fid 0x2 (1000 MHz), vid 0x12
[   11.222074] powernow-k8: fid 0x10 (2400 MHz), vid 0x8
[   11.227584] powernow-k8: fid 0xe (2200 MHz), vid 0xa
[   11.232862] powernow-k8: fid 0xc (2000 MHz), vid 0xc
[   11.238097] powernow-k8: fid 0xa (1800 MHz), vid 0xe
[   11.243462] powernow-k8: fid 0x2 (1000 MHz), vid 0x12
[   11.248922] powernow-k8: fid 0x10 (2400 MHz), vid 0x8
[   11.254305] powernow-k8: fid 0xe (2200 MHz), vid 0xa
[   11.259545] powernow-k8: fid 0xc (2000 MHz), vid 0xc
[   11.264785] powernow-k8: fid 0xa (1800 MHz), vid 0xe
[   11.270232] powernow-k8: fid 0x2 (1000 MHz), vid 0x12
[   11.276179] tg3.c:v3.130 (February 14, 2013)
[   11.276800] powernow-k8: Found 2 AMD Engineering Sample (8 cpu cores) (version 2.20.00)
[   11.614401] e1000 0000:0b:01.0 eth0: (PCI:66MHz:32-bit) 00:1b:21:22:77:2e
[   11.621891] e1000 0000:0b:01.0 eth0: Intel(R) PRO/1000 Network Connection
[   12.472105] floppy0: no floppy controllers found
[   12.784966] tg3 0000:02:01.0 eth1: Tigon3 [partno(3C996B-T) rev 0105] (PCIX:133MHz:64-bit) MAC address 00:04:76:f1:0a:21
[   12.796372] tg3 0000:02:01.0 eth1: attached PHY is 5701 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   12.806600] tg3 0000:02:01.0 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[0]
[   12.814913] tg3 0000:02:01.0 eth1: dma_rwctrl[76db000f] dma_mask[64-bit]
[   12.861418] tg3 0000:03:02.0 eth2: Tigon3 [partno(BCM95704A6) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:00:1a:19:c8:e2
[   12.872987] tg3 0000:03:02.0 eth2: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   12.883224] tg3 0000:03:02.0 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   12.891531] tg3 0000:03:02.0 eth2: dma_rwctrl[769f4000] dma_mask[64-bit]
[   12.933638] tg3 0000:03:02.1 eth3: Tigon3 [partno(BCM95704A6) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:00:1a:19:c8:e1
[   12.945161] tg3 0000:03:02.1 eth3: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   12.955369] tg3 0000:03:02.1 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   12.963641] tg3 0000:03:02.1 eth3: dma_rwctrl[769f4000] dma_mask[64-bit]
[   13.009478] tg3 0000:10:02.0 eth4: Tigon3 [partno(BCM95704A6) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:00:1a:19:c9:5e
[   13.021158] tg3 0000:10:02.0 eth4: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   13.031488] tg3 0000:10:02.0 eth4: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   13.039862] tg3 0000:10:02.0 eth4: dma_rwctrl[769f4000] dma_mask[64-bit]
[   13.078155] tg3 0000:10:02.1 eth5: Tigon3 [partno(BCM95704A6) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:00:1a:19:c9:5d
[   13.089837] tg3 0000:10:02.1 eth5: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   13.100163] tg3 0000:10:02.1 eth5: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   13.108524] tg3 0000:10:02.1 eth5: dma_rwctrl[769f4000] dma_mask[64-bit]
[   15.504143] floppy0: no floppy controllers found
[   15.628978] Adding 2104476k swap on /dev/sda1.  Priority:-1 extents:1 across:2104476k 
[   16.025597] device-mapper: uevent: version 1.0.3
[   16.030896] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
[   16.579014] loop: module loaded
[   17.475596] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[   17.524049] XFS (sda9): Mounting Filesystem
[   17.950029] XFS (sda9): Starting recovery (logdev: internal)
[   18.777643] XFS (sda9): Ending recovery (logdev: internal)
[   21.090510] fuse init (API version 7.21)
[   31.600331] Bridge firewalling registered
[   31.878857] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   31.882123] device eth1 entered promiscuous mode
[   31.892116] IPv6: ADDRCONF(NETDEV_UP): br0: link is not ready
[   34.786553] tg3 0000:02:01.0 eth1: Link is up at 1000 Mbps, full duplex
[   34.786578] tg3 0000:02:01.0 eth1: Flow control is on for TX and on for RX
[   34.786612] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   34.786719] br0: port 1(eth1) entered forwarding state
[   34.786733] br0: port 1(eth1) entered forwarding state
[   34.786893] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
[   35.470290] NET: Registered protocol family 17
[   39.831645] RPC: Registered named UNIX socket transport module.
[   39.831659] RPC: Registered udp transport module.
[   39.831663] RPC: Registered tcp transport module.
[   39.831667] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   40.113821] FS-Cache: Loaded
[   40.466530] FS-Cache: Netfs 'nfs' registered for caching
[   42.180953] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found
[ 1744.821043] start.sh (5028): dropped kernel caches: 3
[ 2761.426095] run_test.sh (5050): dropped kernel caches: 3
[ 2815.442225] run_test.sh (5049): dropped kernel caches: 3
[ 2827.697507] start.sh (10101): dropped kernel caches: 3
[ 3907.055403] run_test.sh (10130): dropped kernel caches: 3
[ 3956.153741] run_test.sh (10131): dropped kernel caches: 3
[ 3967.873725] start.sh (16910): dropped kernel caches: 3
[ 5039.249150] run_test.sh (16940): dropped kernel caches: 3
[ 5047.985116] run_test.sh (16939): dropped kernel caches: 3
[ 5060.180075] start.sh (23048): dropped kernel caches: 3
[ 5879.242283] run_test.sh (23084): dropped kernel caches: 3
[ 5893.533277] run_test.sh (23083): dropped kernel caches: 3
[ 5905.577640] start.sh (26140): dropped kernel caches: 3
[ 6738.882048] run_test.sh (26169): dropped kernel caches: 3
[ 6746.648133] run_test.sh (26170): dropped kernel caches: 3
[ 6758.694415] start.sh (29415): dropped kernel caches: 3
[ 7592.056829] run_test.sh (29445): dropped kernel caches: 3
[ 7596.312199] run_test.sh (29444): dropped kernel caches: 3
[10565.621240] start.sh (892): dropped kernel caches: 3
[11535.852096] run_test.sh (922): dropped kernel caches: 3
[11635.432116] run_test.sh (923): dropped kernel caches: 3
[11647.652973] start.sh (7480): dropped kernel caches: 3
[12775.453315] run_test.sh (7510): dropped kernel caches: 3
[12792.798731] run_test.sh (7509): dropped kernel caches: 3
[12804.433827] start.sh (14720): dropped kernel caches: 3
[14032.290678] run_test.sh (14749): dropped kernel caches: 3
[14046.744583] run_test.sh (14750): dropped kernel caches: 3
[14059.287745] start.sh (22284): dropped kernel caches: 3
[14924.853945] run_test.sh (22314): dropped kernel caches: 3
[14924.863861] run_test.sh (22315): dropped kernel caches: 3
[14937.861865] start.sh (25954): dropped kernel caches: 3
[15840.230633] run_test.sh (25983): dropped kernel caches: 3
[15840.460598] run_test.sh (25984): dropped kernel caches: 3
[15858.177817] start.sh (30093): dropped kernel caches: 3
[16747.692711] run_test.sh (30128): dropped kernel caches: 3
[16754.236454] run_test.sh (30127): dropped kernel caches: 3
[16770.122066] start.sh (1675): dropped kernel caches: 3
[17894.897622] run_test.sh (1706): dropped kernel caches: 3
[17894.897647] run_test.sh (1705): dropped kernel caches: 3
[17906.602681] start.sh (9219): dropped kernel caches: 3
[19084.427238] run_test.sh (9248): dropped kernel caches: 3
[19124.504121] run_test.sh (9249): dropped kernel caches: 3
[19136.789567] start.sh (16993): dropped kernel caches: 3
[20533.841509] run_test.sh (17022): dropped kernel caches: 3
[20535.927383] run_test.sh (17023): dropped kernel caches: 3
[20547.333776] start.sh (26237): dropped kernel caches: 3
[21445.407932] run_test.sh (26267): dropped kernel caches: 3
[21446.166815] run_test.sh (26266): dropped kernel caches: 3
[21460.166389] start.sh (29985): dropped kernel caches: 3
[22354.279533] run_test.sh (30019): dropped kernel caches: 3
[22373.535224] run_test.sh (30020): dropped kernel caches: 3
[22385.815465] start.sh (2087): dropped kernel caches: 3
[23274.653688] run_test.sh (2117): dropped kernel caches: 3
[23274.653875] run_test.sh (2116): dropped kernel caches: 3
[28589.697722] start.sh (7752): dropped kernel caches: 3
[29943.626223] run_test.sh (7782): dropped kernel caches: 3
[29943.626671] run_test.sh (7781): dropped kernel caches: 3
[29955.454547] start.sh (16325): dropped kernel caches: 3
[31265.365454] run_test.sh (16355): dropped kernel caches: 3
[31265.365463] run_test.sh (16354): dropped kernel caches: 3
[31277.561806] start.sh (25020): dropped kernel caches: 3
[32649.463007] run_test.sh (25049): dropped kernel caches: 3
[32649.570650] run_test.sh (25050): dropped kernel caches: 3
[32664.115880] start.sh (1337): dropped kernel caches: 3
[33573.064679] run_test.sh (1367): dropped kernel caches: 3
[33582.338140] run_test.sh (1366): dropped kernel caches: 3
[33594.407287] start.sh (5825): dropped kernel caches: 3
[34509.102642] run_test.sh (5855): dropped kernel caches: 3
[34512.976769] run_test.sh (5854): dropped kernel caches: 3
[34525.637816] start.sh (10095): dropped kernel caches: 3
[35444.125032] run_test.sh (10125): dropped kernel caches: 3
[35444.128213] run_test.sh (10124): dropped kernel caches: 3
[35457.174141] start.sh (14289): dropped kernel caches: 3
[36808.158933] run_test.sh (14318): dropped kernel caches: 3
[36814.771629] run_test.sh (14319): dropped kernel caches: 3
[36826.320502] start.sh (23034): dropped kernel caches: 3
[38073.080136] run_test.sh (23063): dropped kernel caches: 3
[38073.080508] run_test.sh (23064): dropped kernel caches: 3
[38085.603173] start.sh (30987): dropped kernel caches: 3
[39340.373073] run_test.sh (31016): dropped kernel caches: 3
[39353.343312] run_test.sh (31017): dropped kernel caches: 3
[39364.929959] start.sh (6661): dropped kernel caches: 3
[40294.733773] run_test.sh (6691): dropped kernel caches: 3
[40296.604862] run_test.sh (6690): dropped kernel caches: 3
[40313.502875] start.sh (11182): dropped kernel caches: 3
[41265.676979] run_test.sh (11246): dropped kernel caches: 3
[41268.397822] run_test.sh (11245): dropped kernel caches: 3
[41281.146383] start.sh (15894): dropped kernel caches: 3
[42223.970711] run_test.sh (15923): dropped kernel caches: 3
[42223.986587] run_test.sh (15924): dropped kernel caches: 3
[42238.893545] start.sh (20564): dropped kernel caches: 3
[43402.502593] run_test.sh (20599): dropped kernel caches: 3
[43465.036660] run_test.sh (20598): dropped kernel caches: 3
[43476.757334] start.sh (28664): dropped kernel caches: 3
[44659.289429] run_test.sh (28694): dropped kernel caches: 3
[44678.487227] run_test.sh (28693): dropped kernel caches: 3
[44690.117820] start.sh (3628): dropped kernel caches: 3
[45971.497973] run_test.sh (3658): dropped kernel caches: 3
[46020.648218] run_test.sh (3657): dropped kernel caches: 3
[46032.489890] start.sh (12690): dropped kernel caches: 3
[46985.533535] run_test.sh (12720): dropped kernel caches: 3
[47001.385550] run_test.sh (12719): dropped kernel caches: 3
[47013.737901] start.sh (17395): dropped kernel caches: 3
[47957.262113] run_test.sh (17425): dropped kernel caches: 3
[47967.082602] run_test.sh (17424): dropped kernel caches: 3
[47979.111750] start.sh (22338): dropped kernel caches: 3
[48919.894257] run_test.sh (22367): dropped kernel caches: 3
[48919.894406] run_test.sh (22368): dropped kernel caches: 3
[48933.510810] start.sh (26651): dropped kernel caches: 3
[50266.136567] run_test.sh (26685): dropped kernel caches: 3
[50266.686401] run_test.sh (26686): dropped kernel caches: 3
[50281.446526] start.sh (2550): dropped kernel caches: 3
[51458.600907] run_test.sh (2580): dropped kernel caches: 3
[51488.945917] run_test.sh (2579): dropped kernel caches: 3
[51501.285970] start.sh (10585): dropped kernel caches: 3
[52717.389512] run_test.sh (10615): dropped kernel caches: 3
[52717.389522] run_test.sh (10614): dropped kernel caches: 3
[52730.047259] start.sh (18394): dropped kernel caches: 3
[53697.549879] run_test.sh (18424): dropped kernel caches: 3
[53697.567112] run_test.sh (18423): dropped kernel caches: 3
[53710.745912] start.sh (23106): dropped kernel caches: 3
[54682.000865] run_test.sh (23136): dropped kernel caches: 3
[54684.388645] run_test.sh (23135): dropped kernel caches: 3
[54696.694078] start.sh (27741): dropped kernel caches: 3
[55661.856076] run_test.sh (27771): dropped kernel caches: 3
[55674.920601] run_test.sh (27770): dropped kernel caches: 3
[55686.918067] start.sh (402): dropped kernel caches: 3
[57030.289106] run_test.sh (436): dropped kernel caches: 3
[57030.289202] run_test.sh (435): dropped kernel caches: 3
[57042.850406] start.sh (8795): dropped kernel caches: 3
[58289.940429] run_test.sh (8824): dropped kernel caches: 3
[58330.103350] run_test.sh (8825): dropped kernel caches: 3
[58342.308607] start.sh (16749): dropped kernel caches: 3
[59695.307881] run_test.sh (16778): dropped kernel caches: 3
[59695.450356] run_test.sh (16779): dropped kernel caches: 3
[59713.229607] start.sh (25292): dropped kernel caches: 3
[60665.211476] run_test.sh (25327): dropped kernel caches: 3
[60666.583904] run_test.sh (25326): dropped kernel caches: 3
[60683.161750] start.sh (30161): dropped kernel caches: 3
[61635.593315] run_test.sh (30195): dropped kernel caches: 3
[61639.910218] run_test.sh (30196): dropped kernel caches: 3
[61651.952558] start.sh (2606): dropped kernel caches: 3
[62630.181166] run_test.sh (2636): dropped kernel caches: 3
[62630.186838] run_test.sh (2635): dropped kernel caches: 3
[62644.173829] start.sh (7579): dropped kernel caches: 3
[63759.514246] run_test.sh (7609): dropped kernel caches: 3
[63780.868559] run_test.sh (7608): dropped kernel caches: 3
[63792.709932] start.sh (14494): dropped kernel caches: 3
[65149.818288] run_test.sh (14523): dropped kernel caches: 3
[65170.376485] run_test.sh (14524): dropped kernel caches: 3
[65182.770316] start.sh (24222): dropped kernel caches: 3
[66365.268227] run_test.sh (24256): dropped kernel caches: 3
[66392.007148] run_test.sh (24257): dropped kernel caches: 3
[66403.581820] start.sh (32183): dropped kernel caches: 3
[67355.957132] run_test.sh (32212): dropped kernel caches: 3
[67387.285364] run_test.sh (32213): dropped kernel caches: 3
[67400.041783] start.sh (5060): dropped kernel caches: 3
[68385.268815] run_test.sh (5089): dropped kernel caches: 3
[68385.449552] run_test.sh (5090): dropped kernel caches: 3
[68403.825809] start.sh (10202): dropped kernel caches: 3
[69356.260863] run_test.sh (10232): dropped kernel caches: 3
[69365.901222] run_test.sh (10231): dropped kernel caches: 3
[69378.034387] start.sh (14956): dropped kernel caches: 3
[70681.887344] run_test.sh (14986): dropped kernel caches: 3
[70734.920930] run_test.sh (14985): dropped kernel caches: 3
[70746.509502] start.sh (23993): dropped kernel caches: 3
[71939.631650] run_test.sh (24022): dropped kernel caches: 3
[72002.682784] run_test.sh (24023): dropped kernel caches: 3
[72014.718971] start.sh (32650): dropped kernel caches: 3
[73379.104611] run_test.sh (32679): dropped kernel caches: 3
[73379.777906] run_test.sh (32680): dropped kernel caches: 3
[73397.158099] start.sh (9753): dropped kernel caches: 3
[74378.872925] run_test.sh (9783): dropped kernel caches: 3
[74378.888685] run_test.sh (9782): dropped kernel caches: 3
[74393.138682] start.sh (14887): dropped kernel caches: 3
[75390.465421] run_test.sh (14921): dropped kernel caches: 3
[75390.472266] run_test.sh (14922): dropped kernel caches: 3
[75404.086033] start.sh (20107): dropped kernel caches: 3
[76409.632730] run_test.sh (20137): dropped kernel caches: 3
[76410.027730] run_test.sh (20136): dropped kernel caches: 3
[76424.077077] start.sh (25462): dropped kernel caches: 3
[77778.073863] run_test.sh (25492): dropped kernel caches: 3
[77781.630992] run_test.sh (25491): dropped kernel caches: 3
[77793.525953] start.sh (2071): dropped kernel caches: 3
[79109.632686] run_test.sh (2105): dropped kernel caches: 3
[79156.403172] run_test.sh (2106): dropped kernel caches: 3
[79168.195480] start.sh (10781): dropped kernel caches: 3
[80555.216266] run_test.sh (10810): dropped kernel caches: 3
[80555.216541] run_test.sh (10811): dropped kernel caches: 3
[80568.791191] start.sh (20040): dropped kernel caches: 3
[81558.422310] run_test.sh (20069): dropped kernel caches: 3
[81568.137960] run_test.sh (20070): dropped kernel caches: 3
[81580.637740] start.sh (25082): dropped kernel caches: 3
[82563.714165] run_test.sh (25117): dropped kernel caches: 3
[82564.414280] run_test.sh (25116): dropped kernel caches: 3
[82580.153975] start.sh (30147): dropped kernel caches: 3
[83565.827166] run_test.sh (30176): dropped kernel caches: 3
[83582.775235] run_test.sh (30177): dropped kernel caches: 3
[83596.750385] start.sh (3002): dropped kernel caches: 3
[84966.234539] run_test.sh (3041): dropped kernel caches: 3
[84966.234792] run_test.sh (3042): dropped kernel caches: 3
[84979.254451] start.sh (12258): dropped kernel caches: 3
[86424.033433] run_test.sh (12291): dropped kernel caches: 3
[86429.195606] run_test.sh (12292): dropped kernel caches: 3
[86440.671409] start.sh (21697): dropped kernel caches: 3
[87846.400516] run_test.sh (21727): dropped kernel caches: 3
[87903.868562] run_test.sh (21726): dropped kernel caches: 3
[87916.938118] start.sh (31473): dropped kernel caches: 3
[88920.952708] run_test.sh (31503): dropped kernel caches: 3
[88921.745459] run_test.sh (31502): dropped kernel caches: 3
[88935.505904] start.sh (4276): dropped kernel caches: 3
[89939.634755] run_test.sh (4309): dropped kernel caches: 3
[89939.634886] run_test.sh (4308): dropped kernel caches: 3
[89953.051421] start.sh (9642): dropped kernel caches: 3
[90972.976581] run_test.sh (9671): dropped kernel caches: 3
[90973.138177] run_test.sh (9672): dropped kernel caches: 3
[90992.453926] start.sh (15058): dropped kernel caches: 3
[92220.744922] run_test.sh (15088): dropped kernel caches: 3
[92227.401253] run_test.sh (15087): dropped kernel caches: 3
[92238.906095] start.sh (23923): dropped kernel caches: 3
[93436.878982] run_test.sh (23952): dropped kernel caches: 3
[93455.773468] run_test.sh (23953): dropped kernel caches: 3
[93468.289552] start.sh (32591): dropped kernel caches: 3
[94812.277729] run_test.sh (32620): dropped kernel caches: 3
[94823.830996] run_test.sh (32621): dropped kernel caches: 3
[94835.586039] start.sh (9245): dropped kernel caches: 3
[95841.684896] run_test.sh (9275): dropped kernel caches: 3
[95841.697956] run_test.sh (9274): dropped kernel caches: 3
[95854.658020] start.sh (14607): dropped kernel caches: 3
[96884.989131] run_test.sh (14636): dropped kernel caches: 3
[96884.989249] run_test.sh (14637): dropped kernel caches: 3
[96898.614283] start.sh (20102): dropped kernel caches: 3
[97920.642782] run_test.sh (20135): dropped kernel caches: 3
[97920.643088] run_test.sh (20134): dropped kernel caches: 3
[97934.839274] start.sh (25761): dropped kernel caches: 3
[99257.761692] run_test.sh (25790): dropped kernel caches: 3
[99341.646979] run_test.sh (25791): dropped kernel caches: 3
[99354.002311] start.sh (3268): dropped kernel caches: 3
[100800.590464] run_test.sh (3298): dropped kernel caches: 3
[100800.873909] run_test.sh (3297): dropped kernel caches: 3
[100820.903315] start.sh (12687): dropped kernel caches: 3
[102205.184912] run_test.sh (12727): dropped kernel caches: 3
[102213.310524] run_test.sh (12726): dropped kernel caches: 3
[102224.948238] start.sh (22198): dropped kernel caches: 3
[103249.654482] run_test.sh (22228): dropped kernel caches: 3
[103249.672540] run_test.sh (22227): dropped kernel caches: 3
[103262.487332] start.sh (27747): dropped kernel caches: 3
[104286.708245] run_test.sh (27776): dropped kernel caches: 3
[104286.708307] run_test.sh (27777): dropped kernel caches: 3
[104299.478042] start.sh (849): dropped kernel caches: 3
[105300.465397] run_test.sh (880): dropped kernel caches: 3
[105307.881530] run_test.sh (879): dropped kernel caches: 3
[105320.893760] start.sh (6267): dropped kernel caches: 3
[106651.940189] run_test.sh (6297): dropped kernel caches: 3
[106674.060861] run_test.sh (6296): dropped kernel caches: 3
[106688.786047] start.sh (15595): dropped kernel caches: 3
[108036.349474] run_test.sh (15625): dropped kernel caches: 3
[108117.864152] run_test.sh (15624): dropped kernel caches: 3
[108131.829728] start.sh (25463): dropped kernel caches: 3
[109585.152233] run_test.sh (25503): dropped kernel caches: 3
[109598.776067] run_test.sh (25502): dropped kernel caches: 3
[109610.261869] start.sh (2766): dropped kernel caches: 3
[110642.403124] run_test.sh (2796): dropped kernel caches: 3
[110649.498781] run_test.sh (2795): dropped kernel caches: 3
[110662.320608] start.sh (8360): dropped kernel caches: 3
[111702.106452] run_test.sh (8390): dropped kernel caches: 3
[111702.192227] run_test.sh (8389): dropped kernel caches: 3
[111719.157879] start.sh (13986): dropped kernel caches: 3
[112764.389077] run_test.sh (14015): dropped kernel caches: 3
[112764.730378] run_test.sh (14016): dropped kernel caches: 3
[112779.893688] start.sh (19616): dropped kernel caches: 3
[114144.975103] run_test.sh (19645): dropped kernel caches: 3
[114208.904219] run_test.sh (19646): dropped kernel caches: 3
[114221.134272] start.sh (29135): dropped kernel caches: 3
[115522.888860] run_test.sh (29165): dropped kernel caches: 3
[115522.889182] run_test.sh (29164): dropped kernel caches: 3
[115535.469772] start.sh (5272): dropped kernel caches: 3
[116822.380839] run_test.sh (5301): dropped kernel caches: 3
[116883.288144] run_test.sh (5302): dropped kernel caches: 3
[116895.871163] start.sh (13956): dropped kernel caches: 3
[117946.155106] run_test.sh (13986): dropped kernel caches: 3
[117960.796663] run_test.sh (13985): dropped kernel caches: 3
[117973.528215] start.sh (19632): dropped kernel caches: 3
[118989.330575] run_test.sh (19662): dropped kernel caches: 3
[118989.330839] run_test.sh (19661): dropped kernel caches: 3
[119002.770050] start.sh (25092): dropped kernel caches: 3
[120044.424328] run_test.sh (25122): dropped kernel caches: 3
[120045.213212] run_test.sh (25121): dropped kernel caches: 3
[120061.281159] start.sh (30921): dropped kernel caches: 3
[121565.202359] run_test.sh (30955): dropped kernel caches: 3
[121565.202507] run_test.sh (30956): dropped kernel caches: 3
[121577.663775] start.sh (8412): dropped kernel caches: 3
[122876.733337] run_test.sh (8441): dropped kernel caches: 3
[122889.161106] run_test.sh (8442): dropped kernel caches: 3
[122900.689041] start.sh (17195): dropped kernel caches: 3
[124423.465559] run_test.sh (17225): dropped kernel caches: 3
[124480.440672] run_test.sh (17224): dropped kernel caches: 3
[124493.461916] start.sh (28030): dropped kernel caches: 3
[125547.514213] run_test.sh (28065): dropped kernel caches: 3
[125547.722929] run_test.sh (28066): dropped kernel caches: 3
[125563.026019] start.sh (1412): dropped kernel caches: 3
[126630.041789] run_test.sh (1445): dropped kernel caches: 3
[126636.521199] run_test.sh (1444): dropped kernel caches: 3
[126649.266920] start.sh (7393): dropped kernel caches: 3
[127694.137507] run_test.sh (7422): dropped kernel caches: 3
[127694.137704] run_test.sh (7423): dropped kernel caches: 3
[127707.635121] start.sh (13355): dropped kernel caches: 3
[129001.089755] run_test.sh (13385): dropped kernel caches: 3
[129001.089946] run_test.sh (13384): dropped kernel caches: 3
[129013.676130] start.sh (22051): dropped kernel caches: 3
[130228.789373] run_test.sh (22081): dropped kernel caches: 3
[130315.903241] run_test.sh (22080): dropped kernel caches: 3
[130329.262009] start.sh (31277): dropped kernel caches: 3
[131785.156957] run_test.sh (31309): dropped kernel caches: 3
[131791.517881] run_test.sh (31310): dropped kernel caches: 3
[131803.630543] start.sh (8602): dropped kernel caches: 3
[132861.095665] run_test.sh (8632): dropped kernel caches: 3
[132861.095696] run_test.sh (8633): dropped kernel caches: 3
[132875.046985] start.sh (14506): dropped kernel caches: 3
[133940.080474] run_test.sh (14544): dropped kernel caches: 3
[133946.758541] run_test.sh (14543): dropped kernel caches: 3
[133959.733934] start.sh (20452): dropped kernel caches: 3
[135006.128904] run_test.sh (20481): dropped kernel caches: 3
[135006.154381] run_test.sh (20482): dropped kernel caches: 3
[135019.806170] start.sh (25992): dropped kernel caches: 3
[136404.069737] run_test.sh (26022): dropped kernel caches: 3
[136404.127463] run_test.sh (26021): dropped kernel caches: 3
[136420.553914] start.sh (3545): dropped kernel caches: 3
[137819.411895] run_test.sh (3575): dropped kernel caches: 3
[137823.013642] run_test.sh (3576): dropped kernel caches: 3
[137834.622029] start.sh (12876): dropped kernel caches: 3
[139292.147243] run_test.sh (12906): dropped kernel caches: 3
[139299.244901] run_test.sh (12905): dropped kernel caches: 3
[139311.541696] start.sh (23172): dropped kernel caches: 3
[140369.747771] run_test.sh (23203): dropped kernel caches: 3
[140369.748406] run_test.sh (23202): dropped kernel caches: 3
[140383.814673] start.sh (29161): dropped kernel caches: 3
[141451.945337] run_test.sh (29194): dropped kernel caches: 3
[141451.964385] run_test.sh (29193): dropped kernel caches: 3
[141464.681989] start.sh (2512): dropped kernel caches: 3
[142517.753399] run_test.sh (2542): dropped kernel caches: 3
[142519.382818] run_test.sh (2541): dropped kernel caches: 3
[142534.505885] start.sh (8659): dropped kernel caches: 3
[143889.296684] run_test.sh (8688): dropped kernel caches: 3
[143889.296741] run_test.sh (8689): dropped kernel caches: 3
[143901.904368] start.sh (18253): dropped kernel caches: 3
[145183.246550] run_test.sh (18287): dropped kernel caches: 3
[145183.246932] run_test.sh (18288): dropped kernel caches: 3
[145195.974580] start.sh (27352): dropped kernel caches: 3
[146658.565594] run_test.sh (27382): dropped kernel caches: 3
[146664.735406] run_test.sh (27381): dropped kernel caches: 3
[146678.045933] start.sh (4388): dropped kernel caches: 3
[147743.213760] run_test.sh (4426): dropped kernel caches: 3
[147743.227593] run_test.sh (4425): dropped kernel caches: 3
[147756.538377] start.sh (10237): dropped kernel caches: 3
[148835.364323] run_test.sh (10267): dropped kernel caches: 3
[148837.659236] run_test.sh (10266): dropped kernel caches: 3
[148858.133782] start.sh (16252): dropped kernel caches: 3
[149928.023317] run_test.sh (16287): dropped kernel caches: 3
[149930.358878] run_test.sh (16286): dropped kernel caches: 3
[149943.008830] start.sh (22282): dropped kernel caches: 3
[151471.561331] run_test.sh (22312): dropped kernel caches: 3
[151471.643920] run_test.sh (22311): dropped kernel caches: 3
[151491.006545] start.sh (372): dropped kernel caches: 3
[152811.219260] run_test.sh (406): dropped kernel caches: 3
[152811.227265] run_test.sh (407): dropped kernel caches: 3
[152824.816551] start.sh (10008): dropped kernel caches: 3
[154301.049094] run_test.sh (10037): dropped kernel caches: 3
[154301.535590] run_test.sh (10038): dropped kernel caches: 3
[154319.890097] start.sh (20266): dropped kernel caches: 3
[155401.818996] run_test.sh (20295): dropped kernel caches: 3
[155402.438812] run_test.sh (20296): dropped kernel caches: 3
[155421.398647] start.sh (26345): dropped kernel caches: 3
[156500.280551] run_test.sh (26374): dropped kernel caches: 3
[156507.028557] run_test.sh (26375): dropped kernel caches: 3
[156520.029019] start.sh (32431): dropped kernel caches: 3
[157609.836544] run_test.sh (32460): dropped kernel caches: 3
[157609.849909] run_test.sh (32461): dropped kernel caches: 3
[157623.797973] start.sh (6344): dropped kernel caches: 3
[159151.679187] run_test.sh (6373): dropped kernel caches: 3
[159209.562026] run_test.sh (6374): dropped kernel caches: 3
[159222.006083] start.sh (17820): dropped kernel caches: 3
[160582.073370] run_test.sh (17850): dropped kernel caches: 3
[160582.073448] run_test.sh (17849): dropped kernel caches: 3
[160594.698186] start.sh (27320): dropped kernel caches: 3
[162034.753532] run_test.sh (27350): dropped kernel caches: 3
[162041.271336] run_test.sh (27349): dropped kernel caches: 3
[162065.798111] start.sh (5216): dropped kernel caches: 3
[163159.491982] run_test.sh (5305): dropped kernel caches: 3
[163159.495754] run_test.sh (5306): dropped kernel caches: 3
[163172.770335] start.sh (11503): dropped kernel caches: 3
[164268.096965] run_test.sh (11533): dropped kernel caches: 3
[164278.643693] run_test.sh (11532): dropped kernel caches: 3
[164290.702093] start.sh (17917): dropped kernel caches: 3
[165373.085997] run_test.sh (17946): dropped kernel caches: 3
[165384.758431] run_test.sh (17947): dropped kernel caches: 3
[165398.374547] start.sh (24339): dropped kernel caches: 3
[166717.126060] run_test.sh (24368): dropped kernel caches: 3
[166784.371602] run_test.sh (24369): dropped kernel caches: 3
[166795.877796] start.sh (1645): dropped kernel caches: 3
[168311.722962] run_test.sh (1675): dropped kernel caches: 3
[168312.908954] run_test.sh (1674): dropped kernel caches: 3
[168332.314260] start.sh (13211): dropped kernel caches: 3
[169771.584652] run_test.sh (13245): dropped kernel caches: 3
[169771.584759] run_test.sh (13246): dropped kernel caches: 3
[169783.363557] start.sh (23578): dropped kernel caches: 3
[170870.300593] run_test.sh (23607): dropped kernel caches: 3
[170870.300839] run_test.sh (23608): dropped kernel caches: 3
[170884.258899] start.sh (30027): dropped kernel caches: 3
[171969.357413] run_test.sh (30065): dropped kernel caches: 3
[171969.371433] run_test.sh (30064): dropped kernel caches: 3
[171983.145997] start.sh (3824): dropped kernel caches: 3
[173066.896915] run_test.sh (3859): dropped kernel caches: 3
[173068.083913] run_test.sh (3858): dropped kernel caches: 3
[173084.710129] start.sh (9802): dropped kernel caches: 3
[174324.480275] run_test.sh (9841): dropped kernel caches: 3
[174435.277627] run_test.sh (9842): dropped kernel caches: 3
[174446.794291] start.sh (19474): dropped kernel caches: 3
[175869.392133] run_test.sh (19504): dropped kernel caches: 3
[175876.995183] run_test.sh (19503): dropped kernel caches: 3
[175889.560951] start.sh (29286): dropped kernel caches: 3
[177279.766754] run_test.sh (29320): dropped kernel caches: 3
[177293.432145] run_test.sh (29321): dropped kernel caches: 3
[177305.129115] start.sh (6220): dropped kernel caches: 3
[178382.080107] run_test.sh (6250): dropped kernel caches: 3
[178382.080285] run_test.sh (6249): dropped kernel caches: 3
[178395.565726] start.sh (12432): dropped kernel caches: 3
[179467.702291] run_test.sh (12462): dropped kernel caches: 3
[179468.909214] run_test.sh (12461): dropped kernel caches: 3
[179484.661615] start.sh (18502): dropped kernel caches: 3
[180595.731792] run_test.sh (18532): dropped kernel caches: 3
[180595.934993] run_test.sh (18531): dropped kernel caches: 3
[180613.262251] start.sh (25054): dropped kernel caches: 3
[182105.688428] run_test.sh (25088): dropped kernel caches: 3
[182134.805903] run_test.sh (25089): dropped kernel caches: 3
[182146.957388] start.sh (3753): dropped kernel caches: 3
[183517.651687] run_test.sh (3783): dropped kernel caches: 3
[183545.067241] run_test.sh (3782): dropped kernel caches: 3
[183558.328217] start.sh (13002): dropped kernel caches: 3
[184958.215274] run_test.sh (13037): dropped kernel caches: 3
[184968.609816] run_test.sh (13036): dropped kernel caches: 3
[184980.150352] start.sh (22733): dropped kernel caches: 3
[186065.714637] run_test.sh (22763): dropped kernel caches: 3
[186068.485658] run_test.sh (22762): dropped kernel caches: 3
[186082.457916] start.sh (28935): dropped kernel caches: 3
[187170.773576] run_test.sh (28975): dropped kernel caches: 3
[187171.436786] run_test.sh (28974): dropped kernel caches: 3
[187202.673902] start.sh (3113): dropped kernel caches: 3
[188304.198465] run_test.sh (3203): dropped kernel caches: 3
[188312.058524] run_test.sh (3202): dropped kernel caches: 3
[188324.133926] start.sh (9533): dropped kernel caches: 3
[189763.583430] run_test.sh (9563): dropped kernel caches: 3
[189768.906587] run_test.sh (9562): dropped kernel caches: 3
[189781.253989] start.sh (19706): dropped kernel caches: 3
[191162.628121] run_test.sh (19735): dropped kernel caches: 3
[191162.628301] run_test.sh (19736): dropped kernel caches: 3
[191176.474495] start.sh (29442): dropped kernel caches: 3
[192556.737512] run_test.sh (29472): dropped kernel caches: 3
[192559.520206] run_test.sh (29471): dropped kernel caches: 3
[192571.061558] start.sh (6107): dropped kernel caches: 3
[193675.217583] run_test.sh (6136): dropped kernel caches: 3
[193690.495735] run_test.sh (6137): dropped kernel caches: 3
[193702.838359] start.sh (12608): dropped kernel caches: 3
[194803.194251] run_test.sh (12637): dropped kernel caches: 3
[194803.205600] run_test.sh (12638): dropped kernel caches: 3
[194820.824479] start.sh (19018): dropped kernel caches: 3
[195923.875339] run_test.sh (19081): dropped kernel caches: 3
[195952.685307] run_test.sh (19080): dropped kernel caches: 3
[195965.913832] start.sh (25744): dropped kernel caches: 3
[197477.550590] run_test.sh (25773): dropped kernel caches: 3
[197477.768983] run_test.sh (25774): dropped kernel caches: 3
[197493.299276] start.sh (3760): dropped kernel caches: 3
[198936.217406] run_test.sh (3790): dropped kernel caches: 3
[198936.550862] run_test.sh (3789): dropped kernel caches: 3
[198954.864119] start.sh (13706): dropped kernel caches: 3
[200304.967112] run_test.sh (13736): dropped kernel caches: 3
[200304.967496] run_test.sh (13735): dropped kernel caches: 3
[200317.638592] start.sh (23035): dropped kernel caches: 3
[201426.923186] run_test.sh (23064): dropped kernel caches: 3
[201426.923698] run_test.sh (23065): dropped kernel caches: 3
[201439.602959] start.sh (29369): dropped kernel caches: 3
[202530.940910] run_test.sh (29399): dropped kernel caches: 3
[202531.351003] run_test.sh (29398): dropped kernel caches: 3
[202550.002082] start.sh (3264): dropped kernel caches: 3
[203660.439532] run_test.sh (3293): dropped kernel caches: 3
[203666.064045] run_test.sh (3294): dropped kernel caches: 3
[203679.341950] start.sh (9845): dropped kernel caches: 3
[205184.543569] run_test.sh (9875): dropped kernel caches: 3
[205277.438836] run_test.sh (9874): dropped kernel caches: 3
[205289.830546] start.sh (20809): dropped kernel caches: 3
[206719.577558] run_test.sh (20839): dropped kernel caches: 3
[206744.642844] run_test.sh (20838): dropped kernel caches: 3
[206757.299024] start.sh (30314): dropped kernel caches: 3
[208297.096209] run_test.sh (30343): dropped kernel caches: 3
[208299.575322] run_test.sh (30344): dropped kernel caches: 3
[208311.126118] start.sh (8322): dropped kernel caches: 3
[209422.813839] run_test.sh (8353): dropped kernel caches: 3
[209443.508452] run_test.sh (8354): dropped kernel caches: 3
[209457.076874] start.sh (14851): dropped kernel caches: 3
[210560.765080] run_test.sh (14885): dropped kernel caches: 3
[210560.946551] run_test.sh (14886): dropped kernel caches: 3
[210576.722171] start.sh (21064): dropped kernel caches: 3
[211689.652766] run_test.sh (21094): dropped kernel caches: 3
[211689.667325] run_test.sh (21093): dropped kernel caches: 3
[211702.934086] start.sh (27684): dropped kernel caches: 3
[213108.814387] run_test.sh (27714): dropped kernel caches: 3
[213109.635312] run_test.sh (27713): dropped kernel caches: 3
[213126.213782] start.sh (4844): dropped kernel caches: 3
[214513.680138] run_test.sh (4873): dropped kernel caches: 3
[214513.680201] run_test.sh (4874): dropped kernel caches: 3
[214526.190295] start.sh (14183): dropped kernel caches: 3
[215971.588432] run_test.sh (14212): dropped kernel caches: 3
[216005.115422] run_test.sh (14213): dropped kernel caches: 3
[216016.789865] start.sh (23558): dropped kernel caches: 3
[217105.145613] run_test.sh (23588): dropped kernel caches: 3
[217105.159443] run_test.sh (23587): dropped kernel caches: 3
[217118.434129] start.sh (29991): dropped kernel caches: 3
[218228.614090] run_test.sh (30020): dropped kernel caches: 3
[218228.631890] run_test.sh (30021): dropped kernel caches: 3
[218243.466150] start.sh (3980): dropped kernel caches: 3
[219331.968867] run_test.sh (4017): dropped kernel caches: 3
[219332.675081] run_test.sh (4018): dropped kernel caches: 3
[219348.351003] start.sh (10129): dropped kernel caches: 3
[220826.781998] run_test.sh (10159): dropped kernel caches: 3
[220826.782311] run_test.sh (10158): dropped kernel caches: 3
[220841.958917] start.sh (20019): dropped kernel caches: 3
[222382.486135] run_test.sh (20054): dropped kernel caches: 3
[222382.487411] run_test.sh (20053): dropped kernel caches: 3
[222395.566373] start.sh (30180): dropped kernel caches: 3
[223969.653319] run_test.sh (30210): dropped kernel caches: 3
[223998.520858] run_test.sh (30209): dropped kernel caches: 3
[224010.694063] start.sh (8366): dropped kernel caches: 3
[225111.083938] run_test.sh (8395): dropped kernel caches: 3
[225111.466991] run_test.sh (8396): dropped kernel caches: 3
[225127.138328] start.sh (14620): dropped kernel caches: 3
[226216.387030] run_test.sh (14649): dropped kernel caches: 3
[226227.741583] run_test.sh (14650): dropped kernel caches: 3
[226240.849875] start.sh (20738): dropped kernel caches: 3
[227351.819002] run_test.sh (20773): dropped kernel caches: 3
[227357.710951] run_test.sh (20772): dropped kernel caches: 3
[227371.514322] start.sh (27122): dropped kernel caches: 3
[228814.560279] run_test.sh (27151): dropped kernel caches: 3
[228874.621905] run_test.sh (27152): dropped kernel caches: 3
[228886.290780] start.sh (5754): dropped kernel caches: 3
[230284.994380] run_test.sh (5784): dropped kernel caches: 3
[230291.105378] run_test.sh (5785): dropped kernel caches: 3
[230302.722154] start.sh (15515): dropped kernel caches: 3
[231810.256545] run_test.sh (15545): dropped kernel caches: 3
[231832.737197] run_test.sh (15544): dropped kernel caches: 3
[231845.491296] start.sh (26282): dropped kernel caches: 3
[232949.986945] run_test.sh (26312): dropped kernel caches: 3
[232962.779760] run_test.sh (26311): dropped kernel caches: 3
[232975.125893] start.sh (32640): dropped kernel caches: 3
[234074.797561] run_test.sh (32670): dropped kernel caches: 3
[234075.472818] run_test.sh (32669): dropped kernel caches: 3
[234093.797875] start.sh (6395): dropped kernel caches: 3
[235200.826055] run_test.sh (6424): dropped kernel caches: 3
[235225.908461] run_test.sh (6425): dropped kernel caches: 3
[235242.949491] start.sh (13359): dropped kernel caches: 3
[236553.475567] run_test.sh (13394): dropped kernel caches: 3
[236600.924426] run_test.sh (13393): dropped kernel caches: 3
[236613.848650] start.sh (22897): dropped kernel caches: 3
[238058.072469] run_test.sh (22926): dropped kernel caches: 3
[238058.072572] run_test.sh (22927): dropped kernel caches: 3
[238072.898059] start.sh (629): dropped kernel caches: 3
[239482.334602] run_test.sh (670): dropped kernel caches: 3
[239517.814631] run_test.sh (671): dropped kernel caches: 3
[239529.820377] start.sh (11122): dropped kernel caches: 3
[240619.448652] run_test.sh (11151): dropped kernel caches: 3
[240626.388972] run_test.sh (11152): dropped kernel caches: 3
[240638.490066] start.sh (17398): dropped kernel caches: 3
[241744.113210] run_test.sh (17428): dropped kernel caches: 3
[241745.946315] run_test.sh (17427): dropped kernel caches: 3
[241762.550488] start.sh (23541): dropped kernel caches: 3
[242867.331518] run_test.sh (23570): dropped kernel caches: 3
[242867.345164] run_test.sh (23571): dropped kernel caches: 3
[242880.422146] start.sh (29831): dropped kernel caches: 3
[244318.457350] run_test.sh (29860): dropped kernel caches: 3
[244334.586792] run_test.sh (29861): dropped kernel caches: 3
[244346.021987] start.sh (8058): dropped kernel caches: 3
[245765.195062] run_test.sh (8087): dropped kernel caches: 3
[245767.559568] run_test.sh (8088): dropped kernel caches: 3
[245779.085812] start.sh (18140): dropped kernel caches: 3
[247144.679530] run_test.sh (18169): dropped kernel caches: 3
[247144.681493] run_test.sh (18170): dropped kernel caches: 3
[247157.993918] start.sh (27842): dropped kernel caches: 3
[248269.880170] run_test.sh (27876): dropped kernel caches: 3
[248270.047754] run_test.sh (27877): dropped kernel caches: 3
[248287.954475] start.sh (2056): dropped kernel caches: 3
[249392.367317] run_test.sh (2096): dropped kernel caches: 3
[249392.380440] run_test.sh (2095): dropped kernel caches: 3
[249404.972298] start.sh (8198): dropped kernel caches: 3
[250510.147648] run_test.sh (8228): dropped kernel caches: 3
[250524.049700] run_test.sh (8227): dropped kernel caches: 3
[250536.954453] start.sh (14904): dropped kernel caches: 3
[251993.289791] run_test.sh (14934): dropped kernel caches: 3
[251995.378541] run_test.sh (14933): dropped kernel caches: 3
[252011.633802] start.sh (24041): dropped kernel caches: 3
[253443.586414] run_test.sh (24070): dropped kernel caches: 3
[253525.928282] run_test.sh (24071): dropped kernel caches: 3
[253537.505873] start.sh (2631): dropped kernel caches: 3
[254882.490253] run_test.sh (2660): dropped kernel caches: 3
[254889.394961] run_test.sh (2661): dropped kernel caches: 3
[254900.785796] start.sh (12097): dropped kernel caches: 3
[256011.232547] run_test.sh (12126): dropped kernel caches: 3
[256039.701405] run_test.sh (12127): dropped kernel caches: 3
[256052.738082] start.sh (18914): dropped kernel caches: 3
[257140.270096] run_test.sh (18949): dropped kernel caches: 3
[257144.858550] run_test.sh (18948): dropped kernel caches: 3
[257175.726057] start.sh (24950): dropped kernel caches: 3
[258275.388393] run_test.sh (25059): dropped kernel caches: 3
[258275.557538] run_test.sh (25060): dropped kernel caches: 3
[258294.913890] start.sh (31043): dropped kernel caches: 3
[259786.493527] run_test.sh (31072): dropped kernel caches: 3
[259786.556141] run_test.sh (31073): dropped kernel caches: 3
[259803.877504] start.sh (9352): dropped kernel caches: 3
[261271.912278] run_test.sh (9381): dropped kernel caches: 3
[261271.912725] run_test.sh (9382): dropped kernel caches: 3
[261283.591394] start.sh (19449): dropped kernel caches: 3
[262725.014498] run_test.sh (19479): dropped kernel caches: 3
[262744.240772] run_test.sh (19478): dropped kernel caches: 3
[262757.197783] start.sh (29952): dropped kernel caches: 3
[263845.539051] run_test.sh (29981): dropped kernel caches: 3
[263854.812560] run_test.sh (29982): dropped kernel caches: 3
[263870.064340] start.sh (3916): dropped kernel caches: 3
[264983.735942] run_test.sh (3955): dropped kernel caches: 3
[264983.754280] run_test.sh (3956): dropped kernel caches: 3
[264997.446066] start.sh (10314): dropped kernel caches: 3
[266128.030868] run_test.sh (10344): dropped kernel caches: 3
[266128.030984] run_test.sh (10343): dropped kernel caches: 3
[266141.654460] start.sh (16946): dropped kernel caches: 3
[267525.910608] run_test.sh (16976): dropped kernel caches: 3
[267525.910661] run_test.sh (16975): dropped kernel caches: 3
[267537.620508] start.sh (26478): dropped kernel caches: 3
[268993.626239] run_test.sh (26507): dropped kernel caches: 3
[269009.600827] run_test.sh (26508): dropped kernel caches: 3
[269022.765919] start.sh (4954): dropped kernel caches: 3
[270475.554209] run_test.sh (4987): dropped kernel caches: 3
[270513.043821] run_test.sh (4988): dropped kernel caches: 3
[270524.838034] start.sh (15535): dropped kernel caches: 3
[271628.804566] run_test.sh (15565): dropped kernel caches: 3
[271633.287991] run_test.sh (15564): dropped kernel caches: 3
[271645.430994] start.sh (21766): dropped kernel caches: 3
[272730.714709] run_test.sh (21795): dropped kernel caches: 3
[272732.917732] run_test.sh (21796): dropped kernel caches: 3
[272748.511733] start.sh (27694): dropped kernel caches: 3
[273824.233368] run_test.sh (27724): dropped kernel caches: 3
[273829.293584] run_test.sh (27723): dropped kernel caches: 3
[273840.965968] start.sh (784): dropped kernel caches: 3
[275257.555658] run_test.sh (813): dropped kernel caches: 3
[275272.602783] run_test.sh (814): dropped kernel caches: 3
[275284.264312] start.sh (11025): dropped kernel caches: 3
[276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
[276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
[276962.652102]  ffff88003794d198 0000000000000046 ffff8800325f4480 0000000000000000
[276962.652113]  ffff88003794c010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[276962.652121]  0000000000012dc0 ffff88003794dfd8 ffff88003794dfd8 0000000000012dc0
[276962.652128] Call Trace:
[276962.652151]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[276962.652160]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[276962.652171]  [<ffffffff815793a4>] schedule+0x24/0x70
[276962.652178]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[276962.652187]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[276962.652194]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[276962.652200]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[276962.652206]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[276962.652215]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[276962.652224]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[276962.652231]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[276962.652239]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[276962.652246]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[276962.652254]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[276962.652262]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[276962.652274]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[276962.652282]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[276962.652291]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[276962.652298]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[276962.652305]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[276962.652313]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[276962.652323]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[276962.652332]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[276962.652343]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
[276962.652351]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
[276962.652359]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
[276962.652367]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
[276962.652490]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[276962.652540]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
[276962.652588]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[276962.652653]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
[276962.652714]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
[276962.652774]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
[276962.652834]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
[276962.652894]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
[276962.652935]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
[276962.652947]  [<ffffffff81076bd0>] ? __migrate_task+0x150/0x150
[276962.652988]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
[276962.652997]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
[276962.653004]  [<ffffffff8105f294>] rescuer_thread+0x234/0x320
[276962.653011]  [<ffffffff8105f060>] ? free_pwqs+0x30/0x30
[276962.653017]  [<ffffffff81066a86>] kthread+0xc6/0xd0
[276962.653025]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[276962.653034]  [<ffffffff8158303c>] ret_from_fork+0x7c/0xb0
[276962.653041]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[276962.653097] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
[276962.653100] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[276962.653104] kworker/2:2     D ffff88001ffc6448     0 17823      2 0x00000000
[276962.653111]  ffff88000b6251a8 0000000000000046 ffff8800325f4480 0000000000000000
[276962.653119]  ffff88000b624010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[276962.653126]  0000000000012dc0 ffff88000b625fd8 ffff88000b625fd8 0000000000012dc0
[276962.653134] Call Trace:
[276962.653143]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[276962.653151]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[276962.653159]  [<ffffffff815793a4>] schedule+0x24/0x70
[276962.653165]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[276962.653171]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[276962.653178]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[276962.653184]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[276962.653190]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[276962.653197]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[276962.653204]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[276962.653211]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[276962.653219]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[276962.653226]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[276962.653234]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[276962.653242]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[276962.653251]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[276962.653259]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[276962.653268]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[276962.653275]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[276962.653282]  [<ffffffff815793a4>] ? schedule+0x24/0x70
[276962.653289]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[276962.653295]  [<ffffffff815793a4>] ? schedule+0x24/0x70
[276962.653303]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[276962.653312]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[276962.653320]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[276962.653329]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
[276962.653337]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
[276962.653345]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
[276962.653353]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
[276962.653402]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[276962.653450]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
[276962.653498]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[276962.653559]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
[276962.653619]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
[276962.653679]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
[276962.653739]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
[276962.653798]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
[276962.653839]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
[276962.653880]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
[276962.653888]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
[276962.653896]  [<ffffffff81061be2>] worker_thread+0x132/0x400
[276962.653904]  [<ffffffff81061ab0>] ? manage_workers+0xf0/0xf0
[276962.653910]  [<ffffffff81066a86>] kthread+0xc6/0xd0
[276962.653917]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[276962.653924]  [<ffffffff8158303c>] ret_from_fork+0x7c/0xb0
[276962.653931]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[276962.653940] INFO: task ld:14442 blocked for more than 480 seconds.
[276962.653943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[276962.653947] ld              D ffff88001ffca3a8     0 14442  14432 0x00000000
[276962.653953]  ffff88000b66f238 0000000000000082 ffff8800325f4480 0000000000000000
[276962.653961]  ffff88000b66e010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[276962.653968]  0000000000012dc0 ffff88000b66ffd8 ffff88000b66ffd8 0000000000012dc0
[276962.653975] Call Trace:
[276962.653984]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[276962.653992]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[276962.653999]  [<ffffffff815793a4>] schedule+0x24/0x70
[276962.654006]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[276962.654012]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[276962.654019]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[276962.654025]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[276962.654031]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[276962.654038]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[276962.654045]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[276962.654052]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[276962.654060]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[276962.654067]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[276962.654074]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[276962.654083]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[276962.654092]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[276962.654100]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[276962.654109]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[276962.654116]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[276962.654123]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[276962.654131]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[276962.654140]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[276962.654148]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[276962.654157]  [<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[276962.654168]  [<ffffffff8113fe93>] read_swap_cache_async+0x113/0x160
[276962.654176]  [<ffffffff8113ffe1>] swapin_readahead+0x101/0x190
[276962.654187]  [<ffffffff8112e93f>] do_swap_page+0xef/0x5e0
[276962.654230]  [<ffffffffa02b378b>] ? xfs_rw_iunlock+0x1b/0x40 [xfs]
[276962.654239]  [<ffffffff8112f94d>] handle_pte_fault+0x1bd/0x240
[276962.654247]  [<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[276962.654257]  [<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[276962.654267]  [<ffffffff8116a8a8>] ? fsnotify_access+0x68/0x80
[276962.654274]  [<ffffffff8116b0b8>] ? vfs_read+0xd8/0x130
[276962.654283]  [<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[276962.654290]  [<ffffffff8157b348>] page_fault+0x28/0x30
[276962.654297] INFO: task ld:14962 blocked for more than 480 seconds.
[276962.654300] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[276962.654303] ld              D ffff88001ffc7a08     0 14962  14961 0x00000000
[276962.654309]  ffff880008bb9388 0000000000000086 ffff88001c157800 ffff8800325f4480
[276962.654317]  ffff880008bb8010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[276962.654323]  0000000000012dc0 ffff880008bb9fd8 ffff880008bb9fd8 0000000000012dc0
[276962.654331] Call Trace:
[276962.654340]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[276962.654348]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[276962.654355]  [<ffffffff815793a4>] schedule+0x24/0x70
[276962.654361]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[276962.654368]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[276962.654374]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[276962.654380]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[276962.654387]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[276962.654393]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[276962.654401]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[276962.654409]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[276962.654417]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[276962.654425]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[276962.654433]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[276962.654442]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[276962.654450]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[276962.654459]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[276962.654466]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[276962.654476]  [<ffffffff81009510>] ? native_sched_clock+0x20/0xa0
[276962.654483]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[276962.654491]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[276962.654499]  [<ffffffff811054ca>] ? zone_watermark_ok+0x1a/0x20
[276962.654507]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[276962.654516]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[276962.654524]  [<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[276962.654533]  [<ffffffff81129ebb>] do_anonymous_page+0x16b/0x350
[276962.654541]  [<ffffffff8112f9c5>] handle_pte_fault+0x235/0x240
[276962.654549]  [<ffffffff8115e78d>] ? do_huge_pmd_anonymous_page+0xad/0x2e0
[276962.654557]  [<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[276962.654565]  [<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[276962.654574]  [<ffffffff8111e9c9>] ? vm_mmap_pgoff+0xa9/0xd0
[276962.654581]  [<ffffffff811327a6>] ? remove_vma+0x56/0x60
[276962.654589]  [<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[276962.654596]  [<ffffffff8157b348>] page_fault+0x28/0x30
[277442.652123] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
[277442.652135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[277442.652141] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
[277442.652151]  ffff88003794d198 0000000000000046 ffff8800325f4480 0000000000000000
[277442.652162]  ffff88003794c010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[277442.652170]  0000000000012dc0 ffff88003794dfd8 ffff88003794dfd8 0000000000012dc0
[277442.652177] Call Trace:
[277442.652201]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[277442.652210]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[277442.652221]  [<ffffffff815793a4>] schedule+0x24/0x70
[277442.652228]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[277442.652237]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[277442.652244]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[277442.652250]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[277442.652256]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[277442.652265]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[277442.652274]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[277442.652281]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[277442.652289]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[277442.652296]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[277442.652305]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[277442.652313]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[277442.652324]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[277442.652333]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[277442.652341]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[277442.652349]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[277442.652356]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[277442.652363]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[277442.652374]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[277442.652382]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[277442.652394]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
[277442.652402]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
[277442.652410]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
[277442.652418]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
[277442.652543]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[277442.652593]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
[277442.652641]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[277442.652706]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
[277442.652767]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
[277442.652827]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
[277442.652887]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
[277442.652947]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
[277442.652988]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
[277442.653000]  [<ffffffff81076bd0>] ? __migrate_task+0x150/0x150
[277442.653041]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
[277442.653050]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
[277442.653058]  [<ffffffff8105f294>] rescuer_thread+0x234/0x320
[277442.653065]  [<ffffffff8105f060>] ? free_pwqs+0x30/0x30
[277442.653071]  [<ffffffff81066a86>] kthread+0xc6/0xd0
[277442.653079]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[277442.653088]  [<ffffffff8158303c>] ret_from_fork+0x7c/0xb0
[277442.653095]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[277442.653153] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
[277442.653157] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[277442.653160] kworker/2:2     D ffff88001ffc6448     0 17823      2 0x00000000
[277442.653167]  ffff88000b6251a8 0000000000000046 ffff8800325f4480 0000000000000000
[277442.653175]  ffff88000b624010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[277442.653182]  0000000000012dc0 ffff88000b625fd8 ffff88000b625fd8 0000000000012dc0
[277442.653190] Call Trace:
[277442.653199]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[277442.653207]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[277442.653215]  [<ffffffff815793a4>] schedule+0x24/0x70
[277442.653222]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[277442.653228]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[277442.653234]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[277442.653241]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[277442.653247]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[277442.653254]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[277442.653261]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[277442.653268]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[277442.653276]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[277442.653283]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[277442.653291]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[277442.653299]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[277442.653308]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[277442.653316]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[277442.653325]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[277442.653332]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[277442.653339]  [<ffffffff815793a4>] ? schedule+0x24/0x70
[277442.653346]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[277442.653353]  [<ffffffff815793a4>] ? schedule+0x24/0x70
[277442.653360]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[277442.653369]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[277442.653378]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[277442.653386]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
[277442.653394]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
[277442.653403]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
[277442.653411]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
[277442.653459]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[277442.653507]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
[277442.653555]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[277442.653616]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
[277442.653676]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
[277442.653736]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
[277442.653796]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
[277442.653855]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
[277442.653896]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
[277442.653937]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
[277442.653945]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
[277442.653953]  [<ffffffff81061be2>] worker_thread+0x132/0x400
[277442.653961]  [<ffffffff81061ab0>] ? manage_workers+0xf0/0xf0
[277442.653967]  [<ffffffff81066a86>] kthread+0xc6/0xd0
[277442.653974]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[277442.653981]  [<ffffffff8158303c>] ret_from_fork+0x7c/0xb0
[277442.653988]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[277442.653997] INFO: task ld:14442 blocked for more than 480 seconds.
[277442.654000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[277442.654003] ld              D ffff88001ffca3a8     0 14442  14432 0x00000000
[277442.654009]  ffff88000b66f238 0000000000000082 ffff8800325f4480 0000000000000000
[277442.654017]  ffff88000b66e010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[277442.654024]  0000000000012dc0 ffff88000b66ffd8 ffff88000b66ffd8 0000000000012dc0
[277442.654031] Call Trace:
[277442.654041]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[277442.654048]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[277442.654056]  [<ffffffff815793a4>] schedule+0x24/0x70
[277442.654062]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[277442.654069]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[277442.654075]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[277442.654081]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[277442.654088]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[277442.654094]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[277442.654101]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[277442.654108]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[277442.654116]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[277442.654123]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[277442.654131]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[277442.654139]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[277442.654148]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[277442.654156]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[277442.654165]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[277442.654172]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[277442.654180]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[277442.654187]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[277442.654196]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[277442.654205]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[277442.654214]  [<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[277442.654224]  [<ffffffff8113fe93>] read_swap_cache_async+0x113/0x160
[277442.654232]  [<ffffffff8113ffe1>] swapin_readahead+0x101/0x190
[277442.654243]  [<ffffffff8112e93f>] do_swap_page+0xef/0x5e0
[277442.654287]  [<ffffffffa02b378b>] ? xfs_rw_iunlock+0x1b/0x40 [xfs]
[277442.654295]  [<ffffffff8112f94d>] handle_pte_fault+0x1bd/0x240
[277442.654303]  [<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[277442.654313]  [<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[277442.654323]  [<ffffffff8116a8a8>] ? fsnotify_access+0x68/0x80
[277442.654330]  [<ffffffff8116b0b8>] ? vfs_read+0xd8/0x130
[277442.654338]  [<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[277442.654346]  [<ffffffff8157b348>] page_fault+0x28/0x30
[277442.654353] INFO: task ld:14962 blocked for more than 480 seconds.
[277442.654356] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[277442.654359] ld              D ffff88001ffc7a08     0 14962  14961 0x00000000
[277442.654365]  ffff880008bb9388 0000000000000086 ffff88001c157800 ffff8800325f4480
[277442.654373]  ffff880008bb8010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[277442.654379]  0000000000012dc0 ffff880008bb9fd8 ffff880008bb9fd8 0000000000012dc0
[277442.654387] Call Trace:
[277442.654396]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[277442.654404]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[277442.654411]  [<ffffffff815793a4>] schedule+0x24/0x70
[277442.654418]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[277442.654424]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[277442.654430]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[277442.654437]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[277442.654443]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[277442.654450]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[277442.654457]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[277442.654465]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[277442.654473]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[277442.654481]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[277442.654489]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[277442.654498]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[277442.654506]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[277442.654515]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[277442.654522]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[277442.654532]  [<ffffffff81009510>] ? native_sched_clock+0x20/0xa0
[277442.654539]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[277442.654546]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[277442.654555]  [<ffffffff811054ca>] ? zone_watermark_ok+0x1a/0x20
[277442.654563]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[277442.654572]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[277442.654580]  [<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[277442.654589]  [<ffffffff81129ebb>] do_anonymous_page+0x16b/0x350
[277442.654597]  [<ffffffff8112f9c5>] handle_pte_fault+0x235/0x240
[277442.654605]  [<ffffffff8115e78d>] ? do_huge_pmd_anonymous_page+0xad/0x2e0
[277442.654613]  [<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[277442.654621]  [<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[277442.654631]  [<ffffffff8111e9c9>] ? vm_mmap_pgoff+0xa9/0xd0
[277442.654637]  [<ffffffff811327a6>] ? remove_vma+0x56/0x60
[277442.654645]  [<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[277442.654652]  [<ffffffff8157b348>] page_fault+0x28/0x30
[277922.652069] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
[277922.652081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[277922.652087] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
[277922.652097]  ffff88003794d198 0000000000000046 ffff8800325f4480 0000000000000000
[277922.652107]  ffff88003794c010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[277922.652115]  0000000000012dc0 ffff88003794dfd8 ffff88003794dfd8 0000000000012dc0
[277922.652122] Call Trace:
[277922.652145]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[277922.652154]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[277922.652165]  [<ffffffff815793a4>] schedule+0x24/0x70
[277922.652172]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[277922.652180]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[277922.652188]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[277922.652194]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[277922.652200]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[277922.652209]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[277922.652218]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[277922.652225]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[277922.652233]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[277922.652240]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[277922.652248]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[277922.652257]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[277922.652268]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[277922.652276]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[277922.652285]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[277922.652292]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[277922.652300]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[277922.652307]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[277922.652317]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[277922.652326]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[277922.652337]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
[277922.652345]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
[277922.652354]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
[277922.652361]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
[277922.652482]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[277922.652532]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
[277922.652580]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[277922.652645]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
[277922.652706]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
[277922.652766]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
[277922.652826]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
[277922.652885]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
[277922.652927]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
[277922.652938]  [<ffffffff81076bd0>] ? __migrate_task+0x150/0x150
[277922.652979]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
[277922.652989]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
[277922.652996]  [<ffffffff8105f294>] rescuer_thread+0x234/0x320
[277922.653003]  [<ffffffff8105f060>] ? free_pwqs+0x30/0x30
[277922.653010]  [<ffffffff81066a86>] kthread+0xc6/0xd0
[277922.653017]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[277922.653027]  [<ffffffff8158303c>] ret_from_fork+0x7c/0xb0
[277922.653033]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[277922.653089] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
[277922.653092] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[277922.653096] kworker/2:2     D ffff88001ffc6448     0 17823      2 0x00000000
[277922.653102]  ffff88000b6251a8 0000000000000046 ffff8800325f4480 0000000000000000
[277922.653111]  ffff88000b624010 0000000000012dc0 0000000000012dc0 0000000000012dc0
[277922.653118]  0000000000012dc0 ffff88000b625fd8 ffff88000b625fd8 0000000000012dc0
[277922.653125] Call Trace:
[277922.653135]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
[277922.653142]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
[277922.653150]  [<ffffffff815793a4>] schedule+0x24/0x70
[277922.653156]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
[277922.653163]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[277922.653169]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
[277922.653175]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
[277922.653181]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[277922.653188]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
[277922.653195]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
[277922.653202]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
[277922.653210]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[277922.653217]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
[277922.653225]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[277922.653233]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[277922.653242]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
[277922.653250]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[277922.653258]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[277922.653266]  [<ffffffff81116418>] shrink_zones+0x108/0x220
[277922.653272]  [<ffffffff815793a4>] ? schedule+0x24/0x70
[277922.653280]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[277922.653286]  [<ffffffff815793a4>] ? schedule+0x24/0x70
[277922.653293]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[277922.653302]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[277922.653311]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[277922.653320]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
[277922.653328]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
[277922.653336]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
[277922.653344]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
[277922.653392]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[277922.653441]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
[277922.653489]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
[277922.653549]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
[277922.653610]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
[277922.653669]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
[277922.653729]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
[277922.653788]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
[277922.653829]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
[277922.653870]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
[277922.653878]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
[277922.653886]  [<ffffffff81061be2>] worker_thread+0x132/0x400
[277922.653894]  [<ffffffff81061ab0>] ? manage_workers+0xf0/0xf0
[277922.653900]  [<ffffffff81066a86>] kthread+0xc6/0xd0
[277922.653907]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
[277922.653914]  [<ffffffff8158303c>] ret_from_fork+0x7c/0xb0
[277922.653921]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Andrew Morton]

On Mon, 8 Jul 2013 14:53:52 +0200 Michal Hocko <mhocko@suse.cz> wrote:

> > Good news! The test was running since morning and it didn't hang nor
> > crashed. So this really looks like the right fix. It will run also
> > during weekend to be 100% sure. But I guess it is safe to say
> 
> Hmm, it seems I was too optimistic or we have yet another issue here (I
> guess the later is more probable).
> 
> The weekend testing got stuck as well. 
> 
> The dmesg shows there were some hung tasks:

That looks like the classic "we lost an IO completion" trace.

I think it would be prudent to defer these patches into 3.12.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Mon, Jul 08, 2013 at 02:04:19PM -0700, Andrew Morton wrote:
> On Mon, 8 Jul 2013 14:53:52 +0200 Michal Hocko <mhocko@suse.cz> wrote:
> 
> > > Good news! The test was running since morning and it didn't hang nor
> > > crashed. So this really looks like the right fix. It will run also
> > > during weekend to be 100% sure. But I guess it is safe to say
> > 
> > Hmm, it seems I was too optimistic or we have yet another issue here (I
> > guess the later is more probable).
> > 
> > The weekend testing got stuck as well. 
> > 
> > The dmesg shows there were some hung tasks:
> 
> That looks like the classic "we lost an IO completion" trace.
> 
> I think it would be prudent to defer these patches into 3.12.
Agree.

Will they still in -mm, or do I have to resend ?

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Andrew Morton]

On Tue, 9 Jul 2013 21:34:08 +0400 Glauber Costa <glommer@gmail.com> wrote:

> On Mon, Jul 08, 2013 at 02:04:19PM -0700, Andrew Morton wrote:
> > On Mon, 8 Jul 2013 14:53:52 +0200 Michal Hocko <mhocko@suse.cz> wrote:
> > 
> > > > Good news! The test was running since morning and it didn't hang nor
> > > > crashed. So this really looks like the right fix. It will run also
> > > > during weekend to be 100% sure. But I guess it is safe to say
> > > 
> > > Hmm, it seems I was too optimistic or we have yet another issue here (I
> > > guess the later is more probable).
> > > 
> > > The weekend testing got stuck as well. 
> > > 
> > > The dmesg shows there were some hung tasks:
> > 
> > That looks like the classic "we lost an IO completion" trace.
> > 
> > I think it would be prudent to defer these patches into 3.12.
> Agree.
> 
> Will they still in -mm, or do I have to resend ?

No, I don't intend to drop them from -mm.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Mon, Jul 08, 2013 at 02:53:52PM +0200, Michal Hocko wrote:
> On Thu 04-07-13 18:36:43, Michal Hocko wrote:
> > On Wed 03-07-13 21:24:03, Dave Chinner wrote:
> > > On Tue, Jul 02, 2013 at 02:44:27PM +0200, Michal Hocko wrote:
> > > > On Tue 02-07-13 22:19:47, Dave Chinner wrote:
> > > > [...]
> > > > > Ok, so it's been leaked from a dispose list somehow. Thanks for the
> > > > > info, Michal, it's time to go look at the code....
> > > > 
> > > > OK, just in case we will need it, I am keeping the machine in this state
> > > > for now. So we still can play with crash and check all the juicy
> > > > internals.
> > > 
> > > My current suspect is the LRU_RETRY code. I don't think what it is
> > > doing is at all valid - list_for_each_safe() is not safe if you drop
> > > the lock that protects the list. i.e. there is nothing that protects
> > > the stored next pointer from being removed from the list by someone
> > > else. Hence what I think is occurring is this:
> > > 
> > > 
> > > thread 1			thread 2
> > > lock(lru)
> > > list_for_each_safe(lru)		lock(lru)
> > >   isolate			......
> > >     lock(i_lock)
> > >     has buffers
> > >       __iget
> > >       unlock(i_lock)
> > >       unlock(lru)
> > >       .....			(gets lru lock)
> > >       				list_for_each_safe(lru)
> > > 				  walks all the inodes
> > > 				  finds inode being isolated by other thread
> > > 				  isolate
> > > 				    i_count > 0
> > > 				      list_del_init(i_lru)
> > > 				      return LRU_REMOVED;
> > > 				   moves to next inode, inode that
> > > 				   other thread has stored as next
> > > 				   isolate
> > > 				     i_state |= I_FREEING
> > > 				     list_move(dispose_list)
> > > 				     return LRU_REMOVED
> > > 				 ....
> > > 				 unlock(lru)
> > >       lock(lru)
> > >       return LRU_RETRY;
> > >   if (!first_pass)
> > >     ....
> > >   --nr_to_scan
> > >   (loop again using next, which has already been removed from the
> > >   LRU by the other thread!)
> > >   isolate
> > >     lock(i_lock)
> > >     if (i_state & ~I_REFERENCED)
> > >       list_del_init(i_lru)	<<<<< inode is on dispose list!
> > > 				<<<<< inode is now isolated, with I_FREEING set
> > >       return LRU_REMOVED;
> > > 
> > > That fits the corpse left on your machine, Michal. One thread has
> > > moved the inode to a dispose list, the other thread thinks it is
> > > still on the LRU and should be removed, and removes it.
> > > 
> > > This also explains the lru item count going negative - the same item
> > > is being removed from the lru twice. So it seems like all the
> > > problems you've been seeing are caused by this one problem....
> > > 
> > > Patch below that should fix this.
> > 
> > Good news! The test was running since morning and it didn't hang nor
> > crashed. So this really looks like the right fix. It will run also
> > during weekend to be 100% sure. But I guess it is safe to say
> 
> Hmm, it seems I was too optimistic or we have yet another issue here (I
> guess the later is more probable).
> 
> The weekend testing got stuck as well. 
> 
> The dmesg shows there were some hung tasks:
> [275284.264312] start.sh (11025): dropped kernel caches: 3
> [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> [276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
> [276962.652102]  ffff88003794d198 0000000000000046 ffff8800325f4480 0000000000000000
> [276962.652113]  ffff88003794c010 0000000000012dc0 0000000000012dc0 0000000000012dc0
> [276962.652121]  0000000000012dc0 ffff88003794dfd8 ffff88003794dfd8 0000000000012dc0
> [276962.652128] Call Trace:
> [276962.652151]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
> [276962.652160]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
> [276962.652171]  [<ffffffff815793a4>] schedule+0x24/0x70
> [276962.652178]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
> [276962.652187]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
> [276962.652194]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
> [276962.652200]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
> [276962.652206]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
> [276962.652215]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
> [276962.652224]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
> [276962.652231]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
> [276962.652239]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
> [276962.652246]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
> [276962.652254]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
> [276962.652262]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
> [276962.652274]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
> [276962.652282]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
> [276962.652291]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
> [276962.652298]  [<ffffffff81116418>] shrink_zones+0x108/0x220
> [276962.652305]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
> [276962.652313]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
> [276962.652323]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
> [276962.652332]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
> [276962.652343]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
> [276962.652351]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
> [276962.652359]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
> [276962.652367]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
> [276962.652490]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
> [276962.652540]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
> [276962.652588]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
> [276962.652653]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
> [276962.652714]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
> [276962.652774]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
> [276962.652834]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
> [276962.652894]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
> [276962.652935]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
> [276962.652947]  [<ffffffff81076bd0>] ? __migrate_task+0x150/0x150
> [276962.652988]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
> [276962.652997]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
> [276962.653004]  [<ffffffff8105f294>] rescuer_thread+0x234/0x320
> [276962.653011]  [<ffffffff8105f060>] ? free_pwqs+0x30/0x30
> [276962.653017]  [<ffffffff81066a86>] kthread+0xc6/0xd0
> [276962.653025]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
> [276962.653034]  [<ffffffff8158303c>] ret_from_fork+0x7c/0xb0
> [276962.653041]  [<ffffffff810669c0>] ? kthread_freezable_should_stop+0x70/0x70
> 
> $ dmesg | grep "blocked for more than"
> [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> [276962.653097] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> [276962.653940] INFO: task ld:14442 blocked for more than 480 seconds.
> [276962.654297] INFO: task ld:14962 blocked for more than 480 seconds.
> [277442.652123] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> [277442.653153] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> [277442.653997] INFO: task ld:14442 blocked for more than 480 seconds.
> [277442.654353] INFO: task ld:14962 blocked for more than 480 seconds.
> [277922.652069] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> [277922.653089] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> 

You seem to have switched to XFS. Dave posted a patch two days ago fixing some
missing conversions in the XFS side. AFAIK, Andrew hasn't yet picked the patch.

Are you running with that patch applied?

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Andrew Morton]

On Tue, 9 Jul 2013 21:32:51 +0400 Glauber Costa <glommer@gmail.com> wrote:

> > $ dmesg | grep "blocked for more than"
> > [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > [276962.653097] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> > [276962.653940] INFO: task ld:14442 blocked for more than 480 seconds.
> > [276962.654297] INFO: task ld:14962 blocked for more than 480 seconds.
> > [277442.652123] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > [277442.653153] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> > [277442.653997] INFO: task ld:14442 blocked for more than 480 seconds.
> > [277442.654353] INFO: task ld:14962 blocked for more than 480 seconds.
> > [277922.652069] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > [277922.653089] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> > 
> 
> You seem to have switched to XFS. Dave posted a patch two days ago fixing some
> missing conversions in the XFS side. AFAIK, Andrew hasn't yet picked the patch.

I can't find that patch.  Please resend?

There's also "list_lru: fix broken LRU_RETRY behaviour", which I
assume we need?

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Tue, Jul 09, 2013 at 10:50:32AM -0700, Andrew Morton wrote:
> On Tue, 9 Jul 2013 21:32:51 +0400 Glauber Costa <glommer@gmail.com> wrote:
> 
> > > $ dmesg | grep "blocked for more than"
> > > [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > > [276962.653097] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> > > [276962.653940] INFO: task ld:14442 blocked for more than 480 seconds.
> > > [276962.654297] INFO: task ld:14962 blocked for more than 480 seconds.
> > > [277442.652123] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > > [277442.653153] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> > > [277442.653997] INFO: task ld:14442 blocked for more than 480 seconds.
> > > [277442.654353] INFO: task ld:14962 blocked for more than 480 seconds.
> > > [277922.652069] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > > [277922.653089] INFO: task kworker/2:2:17823 blocked for more than 480 seconds.
> > > 
> > 
> > You seem to have switched to XFS. Dave posted a patch two days ago fixing some
> > missing conversions in the XFS side. AFAIK, Andrew hasn't yet picked the patch.
> 
> I can't find that patch.  Please resend?
> 
> There's also "list_lru: fix broken LRU_RETRY behaviour", which I
> assume we need?

Yes, we can either apply or stash that one - up to you.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 09-07-13 21:32:51, Glauber Costa wrote:
[...]
> You seem to have switched to XFS.

Yes, to make sure that the original hang is not fs specific. I can
switch to other fs if it helps. This seems to be really hard to
reproduce now so I would rather not change things if possible.

> Dave posted a patch two days ago fixing some missing conversions in
> the XFS side. AFAIK, Andrew hasn't yet picked the patch.

Could you point me to those patches, please?

> Are you running with that patch applied?

I am currently running with "list_lru: fix broken LRU_RETRY behaviour"

-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Andrew Morton]

On Tue, 9 Jul 2013 19:57:49 +0200 Michal Hocko <mhocko@suse.cz> wrote:

> On Tue 09-07-13 21:32:51, Glauber Costa wrote:
> [...]
> > You seem to have switched to XFS.
> 
> Yes, to make sure that the original hang is not fs specific. I can
> switch to other fs if it helps. This seems to be really hard to
> reproduce now so I would rather not change things if possible.
> 
> > Dave posted a patch two days ago fixing some missing conversions in
> > the XFS side. AFAIK, Andrew hasn't yet picked the patch.
> 
> Could you point me to those patches, please?

This one:

From: Dave Chinner <david@fromorbit.com>
Subject: xfs: fix dquot isolation hang

The new LRU list isolation code in xfs_qm_dquot_isolate() isn't
completely up to date.  Firstly, it needs conversion to return enum
lru_status values, not raw numbers. Secondly - most importantly - it
fails to unlock the dquot and relock the LRU in the LRU_RETRY path.
This leads to deadlocks in xfstests generic/232. Fix them.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Cc: Glauber Costa <glommer@gmail.com>
Cc: Michal Hocko <mhocko@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/xfs/xfs_qm.c |   10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff -puN fs/xfs/xfs_qm.c~xfs-convert-dquot-cache-lru-to-list_lru-fix-dquot-isolation-hang fs/xfs/xfs_qm.c
--- a/fs/xfs/xfs_qm.c~xfs-convert-dquot-cache-lru-to-list_lru-fix-dquot-isolation-hang
+++ a/fs/xfs/xfs_qm.c
@@ -659,7 +659,7 @@ xfs_qm_dquot_isolate(
 		trace_xfs_dqreclaim_want(dqp);
 		list_del_init(&dqp->q_lru);
 		XFS_STATS_DEC(xs_qm_dquot_unused);
-		return 0;
+		return LRU_REMOVED;
 	}
 
 	/*
@@ -705,17 +705,19 @@ xfs_qm_dquot_isolate(
 	XFS_STATS_DEC(xs_qm_dquot_unused);
 	trace_xfs_dqreclaim_done(dqp);
 	XFS_STATS_INC(xs_qm_dqreclaims);
-	return 0;
+	return LRU_REMOVED;
 
 out_miss_busy:
 	trace_xfs_dqreclaim_busy(dqp);
 	XFS_STATS_INC(xs_qm_dqreclaim_misses);
-	return 2;
+	return LRU_SKIP;
 
 out_unlock_dirty:
 	trace_xfs_dqreclaim_busy(dqp);
 	XFS_STATS_INC(xs_qm_dqreclaim_misses);
-	return 3;
+	xfs_dqunlock(dqp);
+	spin_lock(lru_lock);
+	return LRU_RETRY;
 }
 
 static unsigned long
_

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Mon, Jul 08, 2013 at 02:53:52PM +0200, Michal Hocko wrote:
> On Thu 04-07-13 18:36:43, Michal Hocko wrote:
> > On Wed 03-07-13 21:24:03, Dave Chinner wrote:
> > > On Tue, Jul 02, 2013 at 02:44:27PM +0200, Michal Hocko wrote:
> > > > On Tue 02-07-13 22:19:47, Dave Chinner wrote:
> > > > [...]
> > > > > Ok, so it's been leaked from a dispose list somehow. Thanks for the
> > > > > info, Michal, it's time to go look at the code....
> > > > 
> > > > OK, just in case we will need it, I am keeping the machine in this state
> > > > for now. So we still can play with crash and check all the juicy
> > > > internals.
> > > 
> > > My current suspect is the LRU_RETRY code. I don't think what it is
> > > doing is at all valid - list_for_each_safe() is not safe if you drop
> > > the lock that protects the list. i.e. there is nothing that protects
> > > the stored next pointer from being removed from the list by someone
> > > else. Hence what I think is occurring is this:
> > > 
> > > 
> > > thread 1			thread 2
> > > lock(lru)
> > > list_for_each_safe(lru)		lock(lru)
> > >   isolate			......
> > >     lock(i_lock)
> > >     has buffers
> > >       __iget
> > >       unlock(i_lock)
> > >       unlock(lru)
> > >       .....			(gets lru lock)
> > >       				list_for_each_safe(lru)
> > > 				  walks all the inodes
> > > 				  finds inode being isolated by other thread
> > > 				  isolate
> > > 				    i_count > 0
> > > 				      list_del_init(i_lru)
> > > 				      return LRU_REMOVED;
> > > 				   moves to next inode, inode that
> > > 				   other thread has stored as next
> > > 				   isolate
> > > 				     i_state |= I_FREEING
> > > 				     list_move(dispose_list)
> > > 				     return LRU_REMOVED
> > > 				 ....
> > > 				 unlock(lru)
> > >       lock(lru)
> > >       return LRU_RETRY;
> > >   if (!first_pass)
> > >     ....
> > >   --nr_to_scan
> > >   (loop again using next, which has already been removed from the
> > >   LRU by the other thread!)
> > >   isolate
> > >     lock(i_lock)
> > >     if (i_state & ~I_REFERENCED)
> > >       list_del_init(i_lru)	<<<<< inode is on dispose list!
> > > 				<<<<< inode is now isolated, with I_FREEING set
> > >       return LRU_REMOVED;
> > > 
> > > That fits the corpse left on your machine, Michal. One thread has
> > > moved the inode to a dispose list, the other thread thinks it is
> > > still on the LRU and should be removed, and removes it.
> > > 
> > > This also explains the lru item count going negative - the same item
> > > is being removed from the lru twice. So it seems like all the
> > > problems you've been seeing are caused by this one problem....
> > > 
> > > Patch below that should fix this.
> > 
> > Good news! The test was running since morning and it didn't hang nor
> > crashed. So this really looks like the right fix. It will run also
> > during weekend to be 100% sure. But I guess it is safe to say
> 
> Hmm, it seems I was too optimistic or we have yet another issue here (I
> guess the later is more probable).
> 
> The weekend testing got stuck as well. 
....
> 20761 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
> [<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
> [<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
> [<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
> [<ffffffff811763dd>] vfs_create+0xad/0xd0
> [<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
> [<ffffffff8117815e>] do_last+0x2de/0x780
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff

That's an XFS log space issue, indicating that it has run out of
space in IO the log and it is waiting for more to come free. That
requires IO completion to occur.

> [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> [276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000

Oh, that's why. This is the IO completion worker...

> [276962.652102]  ffff88003794d198 0000000000000046 ffff8800325f4480 0000000000000000
> [276962.652113]  ffff88003794c010 0000000000012dc0 0000000000012dc0 0000000000012dc0
> [276962.652121]  0000000000012dc0 ffff88003794dfd8 ffff88003794dfd8 0000000000012dc0
> [276962.652128] Call Trace:
> [276962.652151]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
> [276962.652160]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
> [276962.652171]  [<ffffffff815793a4>] schedule+0x24/0x70
> [276962.652178]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
> [276962.652187]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
> [276962.652194]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
> [276962.652200]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
> [276962.652206]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
> [276962.652215]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
> [276962.652224]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
> [276962.652231]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
> [276962.652239]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
> [276962.652246]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
> [276962.652254]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
> [276962.652262]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
> [276962.652274]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
> [276962.652282]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
> [276962.652291]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
> [276962.652298]  [<ffffffff81116418>] shrink_zones+0x108/0x220
> [276962.652305]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
> [276962.652313]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
> [276962.652323]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
> [276962.652332]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
> [276962.652343]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
> [276962.652351]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
> [276962.652359]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
> [276962.652367]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
> [276962.652490]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
> [276962.652540]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
> [276962.652588]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
> [276962.652653]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
> [276962.652714]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
> [276962.652774]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
> [276962.652834]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
> [276962.652894]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
> [276962.652935]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
> [276962.652947]  [<ffffffff81076bd0>] ? __migrate_task+0x150/0x150
> [276962.652988]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
> [276962.652997]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380

... is running IO completion work and trying to commit a transaction
that is blocked in memory allocation which is waiting for IO
completion. It's disappeared up it's own fundamental orifice.

Ok, this has absolutely nothing to do with the LRU changes - this is
a pre-existing XFS/mm interaction problem from around 3.2. The
question is now this: how the hell do I get memory allocation to not
block waiting on IO completion here? This is already being done in
GFP_NOFS allocation context here....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Wed 10-07-13 12:31:39, Dave Chinner wrote:
> On Mon, Jul 08, 2013 at 02:53:52PM +0200, Michal Hocko wrote:
[...]
> > Hmm, it seems I was too optimistic or we have yet another issue here (I
> > guess the later is more probable).
> > 
> > The weekend testing got stuck as well. 
> ....
> > 20761 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> > [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> > [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> > [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
> > [<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
> > [<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
> > [<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
> > [<ffffffff811763dd>] vfs_create+0xad/0xd0
> > [<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
> > [<ffffffff8117815e>] do_last+0x2de/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> 
> That's an XFS log space issue, indicating that it has run out of
> space in IO the log and it is waiting for more to come free. That
> requires IO completion to occur.
> 
> > [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > [276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
> 
> Oh, that's why. This is the IO completion worker...
> 
> > [276962.652102]  ffff88003794d198 0000000000000046 ffff8800325f4480 0000000000000000
> > [276962.652113]  ffff88003794c010 0000000000012dc0 0000000000012dc0 0000000000012dc0
> > [276962.652121]  0000000000012dc0 ffff88003794dfd8 ffff88003794dfd8 0000000000012dc0
> > [276962.652128] Call Trace:
> > [276962.652151]  [<ffffffff812a2c22>] ? __blk_run_queue+0x32/0x40
> > [276962.652160]  [<ffffffff812a31f8>] ? queue_unplugged+0x78/0xb0
> > [276962.652171]  [<ffffffff815793a4>] schedule+0x24/0x70
> > [276962.652178]  [<ffffffff8157948c>] io_schedule+0x9c/0xf0
> > [276962.652187]  [<ffffffff811011a9>] sleep_on_page+0x9/0x10
> > [276962.652194]  [<ffffffff815778ca>] __wait_on_bit+0x5a/0x90
> > [276962.652200]  [<ffffffff811011a0>] ? __lock_page+0x70/0x70
> > [276962.652206]  [<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
> > [276962.652215]  [<ffffffff81067190>] ? autoremove_wake_function+0x40/0x40
> > [276962.652224]  [<ffffffff81112ee1>] ? page_evictable+0x11/0x50
> > [276962.652231]  [<ffffffff81114e43>] shrink_page_list+0x503/0x790
> > [276962.652239]  [<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
> > [276962.652246]  [<ffffffff81115d5f>] ? shrink_active_list+0x29f/0x340
> > [276962.652254]  [<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
> > [276962.652262]  [<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
> > [276962.652274]  [<ffffffff81160c28>] ? mem_cgroup_reclaimable+0x108/0x150
> > [276962.652282]  [<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
> > [276962.652291]  [<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
> > [276962.652298]  [<ffffffff81116418>] shrink_zones+0x108/0x220
> > [276962.652305]  [<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
> > [276962.652313]  [<ffffffff81117d90>] try_to_free_pages+0x130/0x180
> > [276962.652323]  [<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
> > [276962.652332]  [<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
> > [276962.652343]  [<ffffffff81151c72>] kmem_getpages+0x62/0x1d0
> > [276962.652351]  [<ffffffff81153869>] fallback_alloc+0x189/0x250
> > [276962.652359]  [<ffffffff8115360d>] ____cache_alloc_node+0x8d/0x160
> > [276962.652367]  [<ffffffff81153e51>] __kmalloc+0x281/0x290
> > [276962.652490]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
> > [276962.652540]  [<ffffffffa02c6e97>] kmem_alloc+0x77/0xe0 [xfs]
> > [276962.652588]  [<ffffffffa02c6e97>] ? kmem_alloc+0x77/0xe0 [xfs]
> > [276962.652653]  [<ffffffffa030a334>] xfs_inode_item_format_extents+0x54/0x100 [xfs]
> > [276962.652714]  [<ffffffffa030a63a>] xfs_inode_item_format+0x25a/0x4f0 [xfs]
> > [276962.652774]  [<ffffffffa03081a0>] xlog_cil_prepare_log_vecs+0xa0/0x170 [xfs]
> > [276962.652834]  [<ffffffffa03082a8>] xfs_log_commit_cil+0x38/0x1c0 [xfs]
> > [276962.652894]  [<ffffffffa0303304>] xfs_trans_commit+0x74/0x260 [xfs]
> > [276962.652935]  [<ffffffffa02ac70b>] xfs_setfilesize+0x12b/0x130 [xfs]
> > [276962.652947]  [<ffffffff81076bd0>] ? __migrate_task+0x150/0x150
> > [276962.652988]  [<ffffffffa02ac985>] xfs_end_io+0x75/0xc0 [xfs]
> > [276962.652997]  [<ffffffff8105e934>] process_one_work+0x1b4/0x380
> 
> ... is running IO completion work and trying to commit a transaction
> that is blocked in memory allocation which is waiting for IO
> completion. It's disappeared up it's own fundamental orifice.
> 
> Ok, this has absolutely nothing to do with the LRU changes - this is
> a pre-existing XFS/mm interaction problem from around 3.2. 

OK. I am retesting with ext3 now.

-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Wed 10-07-13 12:31:39, Dave Chinner wrote:
[...]
> > 20761 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> > [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> > [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> > [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
> > [<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
> > [<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
> > [<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
> > [<ffffffff811763dd>] vfs_create+0xad/0xd0
> > [<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
> > [<ffffffff8117815e>] do_last+0x2de/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> 
> That's an XFS log space issue, indicating that it has run out of
> space in IO the log and it is waiting for more to come free. That
> requires IO completion to occur.
>
> > [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > [276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
> 
> Oh, that's why. This is the IO completion worker...

But that task doesn't seem to be stuck anymore (at least lockup watchdog
doesn't report it anymore and I have already rebooted to test with ext3
:/). I am sorry if the these lockups logs were more confusing than
helpful, but they happened _long_ time ago and the system obviously
recovered from them. I am pasting only the traces for processes in D
state here again for reference.

14442 [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[<ffffffff81114e43>] shrink_page_list+0x503/0x790
[<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[<ffffffff81116418>] shrink_zones+0x108/0x220
[<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[<ffffffff8113fe93>] read_swap_cache_async+0x113/0x160
[<ffffffff8113ffe1>] swapin_readahead+0x101/0x190
[<ffffffff8112e93f>] do_swap_page+0xef/0x5e0
[<ffffffff8112f94d>] handle_pte_fault+0x1bd/0x240
[<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[<ffffffff8157b348>] page_fault+0x28/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
14962 [<ffffffff811011a9>] sleep_on_page+0x9/0x10
[<ffffffff8110150f>] wait_on_page_bit+0x6f/0x80
[<ffffffff81114e43>] shrink_page_list+0x503/0x790
[<ffffffff8111570b>] shrink_inactive_list+0x1bb/0x570
[<ffffffff81115ef9>] shrink_lruvec+0xf9/0x330
[<ffffffff8111660a>] mem_cgroup_shrink_node_zone+0xda/0x140
[<ffffffff81163382>] mem_cgroup_soft_reclaim+0xb2/0x140
[<ffffffff811634af>] mem_cgroup_soft_limit_reclaim+0x9f/0x270
[<ffffffff81116418>] shrink_zones+0x108/0x220
[<ffffffff8111776a>] do_try_to_free_pages+0x8a/0x360
[<ffffffff81117d90>] try_to_free_pages+0x130/0x180
[<ffffffff8110a2fe>] __alloc_pages_slowpath+0x39e/0x790
[<ffffffff8110a8ea>] __alloc_pages_nodemask+0x1fa/0x210
[<ffffffff8114d1b0>] alloc_pages_vma+0xa0/0x120
[<ffffffff81129ebb>] do_anonymous_page+0x16b/0x350
[<ffffffff8112f9c5>] handle_pte_fault+0x235/0x240
[<ffffffff8112fcbf>] handle_mm_fault+0x2ef/0x400
[<ffffffff8157e927>] __do_page_fault+0x237/0x4f0
[<ffffffff8157ebe9>] do_page_fault+0x9/0x10
[<ffffffff8157b348>] page_fault+0x28/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
20757 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
[<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
[<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
[<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
[<ffffffffa02c62d0>] xfs_free_eofblocks+0x180/0x250 [xfs]
[<ffffffffa02c68e6>] xfs_release+0x106/0x1d0 [xfs]
[<ffffffffa02b3b20>] xfs_file_release+0x10/0x20 [xfs]
[<ffffffff8116c86d>] __fput+0xbd/0x240
[<ffffffff8116ca49>] ____fput+0x9/0x10
[<ffffffff81063221>] task_work_run+0xb1/0xe0
[<ffffffff810029e0>] do_notify_resume+0x90/0x1d0
[<ffffffff815833a2>] int_signal+0x12/0x17
[<ffffffffffffffff>] 0xffffffffffffffff
20758 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
[<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
[<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
[<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
[<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
[<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
[<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
[<ffffffff811763dd>] vfs_create+0xad/0xd0
[<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
20761 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
[<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
[<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
[<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
[<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
[<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
[<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
[<ffffffff811763dd>] vfs_create+0xad/0xd0
[<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

We are wating for page under writeback but neither of the 2 paths starts
in xfs code. So I do not think waiting for PageWriteback causes a
deadlock here.

[...]
> ... is running IO completion work and trying to commit a transaction
> that is blocked in memory allocation which is waiting for IO
> completion. It's disappeared up it's own fundamental orifice.
> 
> Ok, this has absolutely nothing to do with the LRU changes - this is
> a pre-existing XFS/mm interaction problem from around 3.2. The
> question is now this: how the hell do I get memory allocation to not
> block waiting on IO completion here? This is already being done in
> GFP_NOFS allocation context here....

Just for reference. wait_on_page_writeback is issued only for memcg
reclaim because there is no other throttling mechanism to prevent from
too many dirty pages on the list, thus pre-mature OOM killer. See
e62e384e9d (memcg: prevent OOM with too many dirty pages) for more
details. The original patch relied on may_enter_fs but that check
disappeared by later changes by c3b94f44fc (memcg: further prevent OOM
with too many dirty pages).
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Wed, Jul 10, 2013 at 10:06:05AM +0200, Michal Hocko wrote:
> On Wed 10-07-13 12:31:39, Dave Chinner wrote:
> [...]
> > > 20761 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> > > [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> > > [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> > > [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
> > > [<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
> > > [<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
> > > [<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
> > > [<ffffffff811763dd>] vfs_create+0xad/0xd0
> > > [<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
> > > [<ffffffff8117815e>] do_last+0x2de/0x780
> > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > > [<ffffffffffffffff>] 0xffffffffffffffff
> > 
> > That's an XFS log space issue, indicating that it has run out of
> > space in IO the log and it is waiting for more to come free. That
> > requires IO completion to occur.
> >
> > > [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > > [276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
> > 
> > Oh, that's why. This is the IO completion worker...
> 
> But that task doesn't seem to be stuck anymore (at least lockup watchdog
> doesn't report it anymore and I have already rebooted to test with ext3
> :/). I am sorry if the these lockups logs were more confusing than
> helpful, but they happened _long_ time ago and the system obviously
> recovered from them. I am pasting only the traces for processes in D
> state here again for reference.

Right, there are various triggers that can get XFS out of the
situation - it takes something to kick the log or metadata writeback
and that can make space in the log free up and hence things get
moving again. The problem will be that once in this low memory state
everything in the filesystem will back up on slow memory allocation
and it might take minutes to clear the backlog of IO completions....

> 20757 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]

That is the stack of a process waiting for log space to come
available.

> We are wating for page under writeback but neither of the 2 paths starts
> in xfs code. So I do not think waiting for PageWriteback causes a
> deadlock here.

The problem is this: the page that we are waiting for IO on is in
the IO completion queue, but the IO compeltion requires memory
allocation to complete the transaction. That memory allocation is
causing memcg reclaim, which then waits for IO completion on another
page, which may or may not end up in the same IO completion queue.
The CMWQ can continue to process new Io completions - up to a point
- so slow progress will be made. In the worst case, it can deadlock.

GFP_NOFS allocation is the mechanism by which filesystems are
supposed to be able to avoid this recursive deadlock...

> [...]
> > ... is running IO completion work and trying to commit a transaction
> > that is blocked in memory allocation which is waiting for IO
> > completion. It's disappeared up it's own fundamental orifice.
> > 
> > Ok, this has absolutely nothing to do with the LRU changes - this is
> > a pre-existing XFS/mm interaction problem from around 3.2. The
> > question is now this: how the hell do I get memory allocation to not
> > block waiting on IO completion here? This is already being done in
> > GFP_NOFS allocation context here....
> 
> Just for reference. wait_on_page_writeback is issued only for memcg
> reclaim because there is no other throttling mechanism to prevent from
> too many dirty pages on the list, thus pre-mature OOM killer. See
> e62e384e9d (memcg: prevent OOM with too many dirty pages) for more
> details. The original patch relied on may_enter_fs but that check
> disappeared by later changes by c3b94f44fc (memcg: further prevent OOM
> with too many dirty pages).

Aye. That's the exact code I was looking at yesterday and wondering
"how the hell is waiting on page writeback valid in GFP_NOFS
context?". It seems that memcg reclaim is intentionally ignoring
GFP_NOFS to avoid OOM issues.  That's a memcg implementation problem,
not a filesystem or LRU infrastructure problem....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Andrew Morton]

On Thu, 11 Jul 2013 12:26:34 +1000 Dave Chinner <david@fromorbit.com> wrote:

> > Just for reference. wait_on_page_writeback is issued only for memcg
> > reclaim because there is no other throttling mechanism to prevent from
> > too many dirty pages on the list, thus pre-mature OOM killer. See
> > e62e384e9d (memcg: prevent OOM with too many dirty pages) for more
> > details. The original patch relied on may_enter_fs but that check
> > disappeared by later changes by c3b94f44fc (memcg: further prevent OOM
> > with too many dirty pages).
> 
> Aye. That's the exact code I was looking at yesterday and wondering
> "how the hell is waiting on page writeback valid in GFP_NOFS
> context?". It seems that memcg reclaim is intentionally ignoring
> GFP_NOFS to avoid OOM issues.  That's a memcg implementation problem,
> not a filesystem or LRU infrastructure problem....

Yup, c3b94f44fc shouldn't have done that.

Throttling by waiting on a specific page is indeed prone to deadlocks
and has a number of efficiency problems as well: if 1,000,000 pages
came clean while you're waiting for *this* page to come clean, you're
left looking pretty stupid.

Hence congestion_wait(), which perhaps can save us here.  I'm not sure
how the wait_on_page_writeback() got back in there - I must have been
asleep at the time.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Thu 11-07-13 12:26:34, Dave Chinner wrote:
> On Wed, Jul 10, 2013 at 10:06:05AM +0200, Michal Hocko wrote:
> > On Wed 10-07-13 12:31:39, Dave Chinner wrote:
> > [...]
> > > > 20761 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> > > > [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> > > > [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> > > > [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
> > > > [<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
> > > > [<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
> > > > [<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
> > > > [<ffffffff811763dd>] vfs_create+0xad/0xd0
> > > > [<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
> > > > [<ffffffff8117815e>] do_last+0x2de/0x780
> > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > > > [<ffffffffffffffff>] 0xffffffffffffffff
> > > 
> > > That's an XFS log space issue, indicating that it has run out of
> > > space in IO the log and it is waiting for more to come free. That
> > > requires IO completion to occur.
> > >
> > > > [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > > > [276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
> > > 
> > > Oh, that's why. This is the IO completion worker...
> > 
> > But that task doesn't seem to be stuck anymore (at least lockup watchdog
> > doesn't report it anymore and I have already rebooted to test with ext3
> > :/). I am sorry if the these lockups logs were more confusing than
> > helpful, but they happened _long_ time ago and the system obviously
> > recovered from them. I am pasting only the traces for processes in D
> > state here again for reference.
> 
> Right, there are various triggers that can get XFS out of the
> situation - it takes something to kick the log or metadata writeback
> and that can make space in the log free up and hence things get
> moving again. The problem will be that once in this low memory state
> everything in the filesystem will back up on slow memory allocation
> and it might take minutes to clear the backlog of IO completions....
> 
> > 20757 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> > [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> > [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> > [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
> 
> That is the stack of a process waiting for log space to come
> available.
> 
> > We are wating for page under writeback but neither of the 2 paths starts
> > in xfs code. So I do not think waiting for PageWriteback causes a
> > deadlock here.
> 
> The problem is this: the page that we are waiting for IO on is in
> the IO completion queue, but the IO compeltion requires memory
> allocation to complete the transaction. That memory allocation is
> causing memcg reclaim, which then waits for IO completion on another
> page, which may or may not end up in the same IO completion queue.
> The CMWQ can continue to process new Io completions - up to a point
> - so slow progress will be made. In the worst case, it can deadlock.

OK, I thought something like that was going on but I just wanted to be
sure that I didn't manage to confuse you by the lockup messages.
> 
> GFP_NOFS allocation is the mechanism by which filesystems are
> supposed to be able to avoid this recursive deadlock...

Yes.

> > [...]
> > > ... is running IO completion work and trying to commit a transaction
> > > that is blocked in memory allocation which is waiting for IO
> > > completion. It's disappeared up it's own fundamental orifice.
> > > 
> > > Ok, this has absolutely nothing to do with the LRU changes - this is
> > > a pre-existing XFS/mm interaction problem from around 3.2. The
> > > question is now this: how the hell do I get memory allocation to not
> > > block waiting on IO completion here? This is already being done in
> > > GFP_NOFS allocation context here....
> > 
> > Just for reference. wait_on_page_writeback is issued only for memcg
> > reclaim because there is no other throttling mechanism to prevent from
> > too many dirty pages on the list, thus pre-mature OOM killer. See
> > e62e384e9d (memcg: prevent OOM with too many dirty pages) for more
> > details. The original patch relied on may_enter_fs but that check
> > disappeared by later changes by c3b94f44fc (memcg: further prevent OOM
> > with too many dirty pages).
> 
> Aye. That's the exact code I was looking at yesterday and wondering
> "how the hell is waiting on page writeback valid in GFP_NOFS
> context?". It seems that memcg reclaim is intentionally ignoring
> GFP_NOFS to avoid OOM issues.  That's a memcg implementation problem,
> not a filesystem or LRU infrastructure problem....

Agreed and until we have a proper per memcg dirty memory throttling we
will always be in a workaround mode. Which is sad but that is the
reality...

I am CCing Hugh (the discussion was long and started with a different
issue but the above should tell about the current xfs hang. It seems
that c3b94f44fc make xfs hang).
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Hugh Dickins]

On Thu, 11 Jul 2013, Michal Hocko wrote:
> On Thu 11-07-13 12:26:34, Dave Chinner wrote:
> > On Wed, Jul 10, 2013 at 10:06:05AM +0200, Michal Hocko wrote:
> > > On Wed 10-07-13 12:31:39, Dave Chinner wrote:
> > > [...]
> > > > > 20761 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> > > > > [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> > > > > [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> > > > > [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
> > > > > [<ffffffffa02c5999>] xfs_create+0x1a9/0x5c0 [xfs]
> > > > > [<ffffffffa02bccca>] xfs_vn_mknod+0x8a/0x1a0 [xfs]
> > > > > [<ffffffffa02bce0e>] xfs_vn_create+0xe/0x10 [xfs]
> > > > > [<ffffffff811763dd>] vfs_create+0xad/0xd0
> > > > > [<ffffffff81177e68>] lookup_open+0x1b8/0x1d0
> > > > > [<ffffffff8117815e>] do_last+0x2de/0x780
> > > > > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > > > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > > > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > > > > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > > > > [<ffffffff815830e9>] system_call_fastpath+0x16/0x1b
> > > > > [<ffffffffffffffff>] 0xffffffffffffffff
> > > > 
> > > > That's an XFS log space issue, indicating that it has run out of
> > > > space in IO the log and it is waiting for more to come free. That
> > > > requires IO completion to occur.
> > > >
> > > > > [276962.652076] INFO: task xfs-data/sda9:930 blocked for more than 480 seconds.
> > > > > [276962.652087] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > [276962.652093] xfs-data/sda9   D ffff88001ffb9cc8     0   930      2 0x00000000
> > > > 
> > > > Oh, that's why. This is the IO completion worker...
> > > 
> > > But that task doesn't seem to be stuck anymore (at least lockup watchdog
> > > doesn't report it anymore and I have already rebooted to test with ext3
> > > :/). I am sorry if the these lockups logs were more confusing than
> > > helpful, but they happened _long_ time ago and the system obviously
> > > recovered from them. I am pasting only the traces for processes in D
> > > state here again for reference.
> > 
> > Right, there are various triggers that can get XFS out of the
> > situation - it takes something to kick the log or metadata writeback
> > and that can make space in the log free up and hence things get
> > moving again. The problem will be that once in this low memory state
> > everything in the filesystem will back up on slow memory allocation
> > and it might take minutes to clear the backlog of IO completions....
> > 
> > > 20757 [<ffffffffa0305fdd>] xlog_grant_head_wait+0xdd/0x1a0 [xfs]
> > > [<ffffffffa0306166>] xlog_grant_head_check+0xc6/0xe0 [xfs]
> > > [<ffffffffa030627f>] xfs_log_reserve+0xff/0x240 [xfs]
> > > [<ffffffffa0302ac4>] xfs_trans_reserve+0x234/0x240 [xfs]
> > 
> > That is the stack of a process waiting for log space to come
> > available.
> > 
> > > We are wating for page under writeback but neither of the 2 paths starts
> > > in xfs code. So I do not think waiting for PageWriteback causes a
> > > deadlock here.
> > 
> > The problem is this: the page that we are waiting for IO on is in
> > the IO completion queue, but the IO compeltion requires memory
> > allocation to complete the transaction. That memory allocation is
> > causing memcg reclaim, which then waits for IO completion on another
> > page, which may or may not end up in the same IO completion queue.
> > The CMWQ can continue to process new Io completions - up to a point
> > - so slow progress will be made. In the worst case, it can deadlock.
> 
> OK, I thought something like that was going on but I just wanted to be
> sure that I didn't manage to confuse you by the lockup messages.
> > 
> > GFP_NOFS allocation is the mechanism by which filesystems are
> > supposed to be able to avoid this recursive deadlock...
> 
> Yes.
> 
> > > [...]
> > > > ... is running IO completion work and trying to commit a transaction
> > > > that is blocked in memory allocation which is waiting for IO
> > > > completion. It's disappeared up it's own fundamental orifice.
> > > > 
> > > > Ok, this has absolutely nothing to do with the LRU changes - this is
> > > > a pre-existing XFS/mm interaction problem from around 3.2. The
> > > > question is now this: how the hell do I get memory allocation to not
> > > > block waiting on IO completion here? This is already being done in
> > > > GFP_NOFS allocation context here....
> > > 
> > > Just for reference. wait_on_page_writeback is issued only for memcg
> > > reclaim because there is no other throttling mechanism to prevent from
> > > too many dirty pages on the list, thus pre-mature OOM killer. See
> > > e62e384e9d (memcg: prevent OOM with too many dirty pages) for more
> > > details. The original patch relied on may_enter_fs but that check
> > > disappeared by later changes by c3b94f44fc (memcg: further prevent OOM
> > > with too many dirty pages).
> > 
> > Aye. That's the exact code I was looking at yesterday and wondering
> > "how the hell is waiting on page writeback valid in GFP_NOFS
> > context?". It seems that memcg reclaim is intentionally ignoring
> > GFP_NOFS to avoid OOM issues.  That's a memcg implementation problem,
> > not a filesystem or LRU infrastructure problem....
> 
> Agreed and until we have a proper per memcg dirty memory throttling we
> will always be in a workaround mode. Which is sad but that is the
> reality...
> 
> I am CCing Hugh (the discussion was long and started with a different
> issue but the above should tell about the current xfs hang. It seems
> that c3b94f44fc make xfs hang).

The may_enter_fs test came and went several times as we prepared those
patches: one set of problems with it in, another set with it out.

When I made c3b94f44fc, I was not imagining that I/O completion might
have to wait on a further __GFP_IO allocation.  But I can see the sense
of what XFS is doing there: after writing the data, it wants to perform
(initiate?) a transaction; but if that happens to fail, wants to mark
the written data pages as bad before reaching the end_page_writeback.
I've toyed with reordering that, but its order does seem sensible.

I've always thought of GFP_NOFS as meaning "don't recurse into the
filesystem" (and wondered what that amounts to since direct reclaim
stopped doing filesystem writeback); but here XFS is expecting it
to include "and don't wait for PageWriteback to be cleared".

I've mused on this for a while, and haven't arrived at any conclusion;
but do have several mutterings on different kinds of solution.

Probably the easiest solution, but not necessarily the right solution,
would be for XFS to add a KM_NOIO akin to its KM_NOFS, and use KM_NOIO
instead of KM_NOFS in xfs_iomap_write_unwritten() (anywhere else?).
I'd find that more convincing if it were not so obviously designed
to match an assumption I'd once made over in mm/vmscan.c.

A harder solution, but one which I'd expect to have larger benefits,
would be to reinstate the may_enter_fs test there in shrink_page_list(),
but modify ext4 and xfs and gfs2 to use grab_cache_page_write_begin()
without needing AOP_FLAG_NOFS: I think it is very sad that major FS
page allocations are made with the limiting GFP_NOFS, and I hope there
might be an efficient way to make those page allocations outside of the
transaction, with __GFP_FS instead.

Another kind of solution: I did originally worry about your e62e384e9d
in rather the same way that akpm has, thinking a wait on return from
shrink_page_list() more appropriate than waiting on a single page
(with a hold on all the other pages of the page_list).  I did have a
patch I'd been playing with about the time you posted yours, but we
agreed to go ahead with yours unless problems showed up (I think mine
was not so pretty as yours).  Maybe I need to dust off my old
alternative now - though I've rather forgotten how to test it.

Hugh

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Thu, Jul 11, 2013 at 06:42:03PM -0700, Hugh Dickins wrote:
> On Thu, 11 Jul 2013, Michal Hocko wrote:
> > On Thu 11-07-13 12:26:34, Dave Chinner wrote:
> > > > We are wating for page under writeback but neither of the 2 paths starts
> > > > in xfs code. So I do not think waiting for PageWriteback causes a
> > > > deadlock here.
> > > 
> > > The problem is this: the page that we are waiting for IO on is in
> > > the IO completion queue, but the IO compeltion requires memory
> > > allocation to complete the transaction. That memory allocation is
> > > causing memcg reclaim, which then waits for IO completion on another
> > > page, which may or may not end up in the same IO completion queue.
> > > The CMWQ can continue to process new Io completions - up to a point
> > > - so slow progress will be made. In the worst case, it can deadlock.
> > 
> > OK, I thought something like that was going on but I just wanted to be
> > sure that I didn't manage to confuse you by the lockup messages.
> > > 
> > > GFP_NOFS allocation is the mechanism by which filesystems are
> > > supposed to be able to avoid this recursive deadlock...
> > 
> > Yes.
> > 
> > > > [...]
> > > > > ... is running IO completion work and trying to commit a transaction
> > > > > that is blocked in memory allocation which is waiting for IO
> > > > > completion. It's disappeared up it's own fundamental orifice.
> > > > > 
> > > > > Ok, this has absolutely nothing to do with the LRU changes - this is
> > > > > a pre-existing XFS/mm interaction problem from around 3.2. The
> > > > > question is now this: how the hell do I get memory allocation to not
> > > > > block waiting on IO completion here? This is already being done in
> > > > > GFP_NOFS allocation context here....
> > > > 
> > > > Just for reference. wait_on_page_writeback is issued only for memcg
> > > > reclaim because there is no other throttling mechanism to prevent from
> > > > too many dirty pages on the list, thus pre-mature OOM killer. See
> > > > e62e384e9d (memcg: prevent OOM with too many dirty pages) for more
> > > > details. The original patch relied on may_enter_fs but that check
> > > > disappeared by later changes by c3b94f44fc (memcg: further prevent OOM
> > > > with too many dirty pages).
> > > 
> > > Aye. That's the exact code I was looking at yesterday and wondering
> > > "how the hell is waiting on page writeback valid in GFP_NOFS
> > > context?". It seems that memcg reclaim is intentionally ignoring
> > > GFP_NOFS to avoid OOM issues.  That's a memcg implementation problem,
> > > not a filesystem or LRU infrastructure problem....
> > 
> > Agreed and until we have a proper per memcg dirty memory throttling we
> > will always be in a workaround mode. Which is sad but that is the
> > reality...
> > 
> > I am CCing Hugh (the discussion was long and started with a different
> > issue but the above should tell about the current xfs hang. It seems
> > that c3b94f44fc make xfs hang).
> 
> The may_enter_fs test came and went several times as we prepared those
> patches: one set of problems with it in, another set with it out.
> 
> When I made c3b94f44fc, I was not imagining that I/O completion might
> have to wait on a further __GFP_IO allocation.  But I can see the sense
> of what XFS is doing there: after writing the data, it wants to perform
> (initiate?) a transaction; but if that happens to fail, wants to mark
> the written data pages as bad before reaching the end_page_writeback.
> I've toyed with reordering that, but its order does seem sensible.
> 
> I've always thought of GFP_NOFS as meaning "don't recurse into the
> filesystem" (and wondered what that amounts to since direct reclaim
> stopped doing filesystem writeback); but here XFS is expecting it
> to include "and don't wait for PageWriteback to be cleared".

Well, it's more general than that - my understanding of GFP_NOFS is
that it means "don't block reclaim on anything filesystem related
because a filesystem deadlock is possible from this calling
content". Even without direct reclaim doing writeback, there is
still shrinkers that need to avoid locking filesystem objects during
direct reclaim, and the fact that waiting on writeback for specific
pages to complete may (indirectly) block a memory allocation
required to complete the writeback of that page. It's the latter
case that is the problem here...

> I've mused on this for a while, and haven't arrived at any conclusion;
> but do have several mutterings on different kinds of solution.
> 
> Probably the easiest solution, but not necessarily the right solution,
> would be for XFS to add a KM_NOIO akin to its KM_NOFS, and use KM_NOIO
> instead of KM_NOFS in xfs_iomap_write_unwritten() (anywhere else?).
> I'd find that more convincing if it were not so obviously designed
> to match an assumption I'd once made over in mm/vmscan.c.

I'd prefer not to have to start using KM_NOIO in specific places in
the filesystem layer. I can see how it may be relevant, though,
because we are in the IO completion path here, and so -technically-
we are dealing with IO layer interactions here. Hmmm - it looks like
there is already a task flag to tell memory allocation we are in IO
context without needing to pass GFP_IO: PF_MEMALLOC_NOIO.

[ As an idle thought, if we drove PF_FSTRANS into the memory
allocation to clear __GFP_FS like PF_MEMALLOC_NOIO clears __GFP_IO,
we could probably get rid of a large amount of the XFS specific
memory allocation wrappers. Hmmm, it would solve all the "we
need to do GFP_NOFS for vmalloc()" problems we have as well, which
is what DM uses PF_MEMALLOC_NOIO for.... ]

> A harder solution, but one which I'd expect to have larger benefits,
> would be to reinstate the may_enter_fs test there in shrink_page_list(),
> but modify ext4 and xfs and gfs2 to use grab_cache_page_write_begin()
> without needing AOP_FLAG_NOFS: I think it is very sad that major FS
> page allocations are made with the limiting GFP_NOFS, and I hope there
> might be an efficient way to make those page allocations outside of the
> transaction, with __GFP_FS instead.

I don't think that helps the AOP_FLAG_NOFS case - even if we aren't
in a transaction context, we're still holding (multiple) filesystem
locks when doing memory allocation...

> Another kind of solution: I did originally worry about your e62e384e9d
> in rather the same way that akpm has, thinking a wait on return from
> shrink_page_list() more appropriate than waiting on a single page
> (with a hold on all the other pages of the page_list).  I did have a
> patch I'd been playing with about the time you posted yours, but we
> agreed to go ahead with yours unless problems showed up (I think mine
> was not so pretty as yours).  Maybe I need to dust off my old
> alternative now - though I've rather forgotten how to test it.

I think that a congestion_wait()-styleange (as Andrew suggested) is
a workable interim solution but I suspect - like Michal - that we
need proper memcg awareness in balance_dirty_pages() to really solve
this problem fully....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Thu 04-07-13 18:36:43, Michal Hocko wrote:
> On Wed 03-07-13 21:24:03, Dave Chinner wrote:
> > On Tue, Jul 02, 2013 at 02:44:27PM +0200, Michal Hocko wrote:
> > > On Tue 02-07-13 22:19:47, Dave Chinner wrote:
> > > [...]
> > > > Ok, so it's been leaked from a dispose list somehow. Thanks for the
> > > > info, Michal, it's time to go look at the code....
> > > 
> > > OK, just in case we will need it, I am keeping the machine in this state
> > > for now. So we still can play with crash and check all the juicy
> > > internals.
> > 
> > My current suspect is the LRU_RETRY code. I don't think what it is
> > doing is at all valid - list_for_each_safe() is not safe if you drop
> > the lock that protects the list. i.e. there is nothing that protects
> > the stored next pointer from being removed from the list by someone
> > else. Hence what I think is occurring is this:
> > 
> > 
> > thread 1			thread 2
> > lock(lru)
> > list_for_each_safe(lru)		lock(lru)
> >   isolate			......
> >     lock(i_lock)
> >     has buffers
> >       __iget
> >       unlock(i_lock)
> >       unlock(lru)
> >       .....			(gets lru lock)
> >       				list_for_each_safe(lru)
> > 				  walks all the inodes
> > 				  finds inode being isolated by other thread
> > 				  isolate
> > 				    i_count > 0
> > 				      list_del_init(i_lru)
> > 				      return LRU_REMOVED;
> > 				   moves to next inode, inode that
> > 				   other thread has stored as next
> > 				   isolate
> > 				     i_state |= I_FREEING
> > 				     list_move(dispose_list)
> > 				     return LRU_REMOVED
> > 				 ....
> > 				 unlock(lru)
> >       lock(lru)
> >       return LRU_RETRY;
> >   if (!first_pass)
> >     ....
> >   --nr_to_scan
> >   (loop again using next, which has already been removed from the
> >   LRU by the other thread!)
> >   isolate
> >     lock(i_lock)
> >     if (i_state & ~I_REFERENCED)
> >       list_del_init(i_lru)	<<<<< inode is on dispose list!
> > 				<<<<< inode is now isolated, with I_FREEING set
> >       return LRU_REMOVED;
> > 
> > That fits the corpse left on your machine, Michal. One thread has
> > moved the inode to a dispose list, the other thread thinks it is
> > still on the LRU and should be removed, and removes it.
> > 
> > This also explains the lru item count going negative - the same item
> > is being removed from the lru twice. So it seems like all the
> > problems you've been seeing are caused by this one problem....
> > 
> > Patch below that should fix this.
> 
> Good news! The test was running since morning and it didn't hang nor
> crashed. So this really looks like the right fix. It will run also
> during weekend to be 100% sure. But I guess it is safe to say
> 
> Tested-by: Michal Hocko <mhocko@suse.cz>

And I can finally confirm this after over weekend testing on ext3.

Thanks a lot for your help Dave!
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

[-- Attachment #1: Type: text/plain, Size: 4227 bytes --]

On Tue, Jun 18, 2013 at 02:30:05AM +0400, Glauber Costa wrote:
> On Mon, Jun 17, 2013 at 02:35:08PM -0700, Andrew Morton wrote:
> > On Mon, 17 Jun 2013 19:14:12 +0400 Glauber Costa <glommer@gmail.com> wrote:
> > 
> > > > I managed to trigger:
> > > > [ 1015.776029] kernel BUG at mm/list_lru.c:92!
> > > > [ 1015.776029] invalid opcode: 0000 [#1] SMP
> > > > with Linux next (next-20130607) with https://lkml.org/lkml/2013/6/17/203
> > > > on top. 
> > > > 
> > > > This is obviously BUG_ON(nlru->nr_items < 0) and 
> > > > ffffffff81122d0b:       48 85 c0                test   %rax,%rax
> > > > ffffffff81122d0e:       49 89 44 24 18          mov    %rax,0x18(%r12)
> > > > ffffffff81122d13:       0f 84 87 00 00 00       je     ffffffff81122da0 <list_lru_walk_node+0x110>
> > > > ffffffff81122d19:       49 83 7c 24 18 00       cmpq   $0x0,0x18(%r12)
> > > > ffffffff81122d1f:       78 7b                   js     ffffffff81122d9c <list_lru_walk_node+0x10c>
> > > > [...]
> > > > ffffffff81122d9c:       0f 0b                   ud2
> > > > 
> > > > RAX is -1UL.
> > > Yes, fearing those kind of imbalances, we decided to leave the counter as a signed quantity
> > > and BUG, instead of an unsigned quantity.
> > > 
> > > > 
> > > > I assume that the current backtrace is of no use and it would most
> > > > probably be some shrinker which doesn't behave.
> > > > 
> > > There are currently 3 users of list_lru in tree: dentries, inodes and xfs.
> > > Assuming you are not using xfs, we are left with dentries and inodes.
> > > 
> > > The first thing to do is to find which one of them is misbehaving. You can try finding
> > > this out by the address of the list_lru, and where it lays in the superblock.
> > > 
> > > Once we know each of them is misbehaving, then we'll have to figure out why.
> > 
> > The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
> > it's inodes?
> > 
> Assuming there is no memory corruption of any sort going on , let's check the code.
> nr_item is only manipulated in 3 places:
> 
> 1) list_lru_add, where it is increased
> 2) list_lru_del, where it is decreased in case the user have voluntarily removed the
>    element from the list
> 3) list_lru_walk_node, where an element is removing during shrink.
> 
> All three excerpts seem to be correctly locked, so something like this indicates an imbalance.
> Either the element was never added to the list, or it was added, removed, and we didn't notice
> it. (Again, your backing storage is not XFS, is it? If it is , we have another user to look for)
> 
> I will assume that Andrew is correct and this is inode related. list_lru_del reads as follows:
>         spin_lock(&nlru->lock);
>         if (!list_empty(item)) { ... }
> 
> So one possibility is that we are manipulating this list outside this lock somewhere. Going to
> inode.c... We always manipulate the LRU inside the lock, but the element is not always in the LRU,
> if it is in a list. Could it be possible that the element is in the dispose_list, and at the same
> time someone calls list_lru_del at it, creating the imbalance ?
> 
> callers:
> iput_final, evict_inodes, invalidate_inodes.
> Both evict_inodes and invalidate_inodes will do the following pattern:
> 
>                 inode->i_state |= I_FREEING;                                            
>                 inode_lru_list_del(inode);
>                 spin_unlock(&inode->i_lock);
>                 list_add(&inode->i_lru, &dispose);
> 
> IOW, they will remove the element from the LRU, and add it to the dispose list.
> Both of them will also bail out if they see I_FREEING already set, so they are safe
> against each other - because the flag is manipulated inside the lock.
> 
> But how about iput_final? It seems to me that if we are calling iput_final at the
> same time as the other two, this *could* happen (maybe there is some extra protection
> that can be seen from Australia but not from here. Dave?)
> 
> Right now this is my best theory.
> 
> I am attaching a patch that should make a difference in case I am right.
> 
> 
> 

Which is obviously borked since I did not fix the other callers so to move I_FREEING
after lru del.

Michal, would you mind testing the following patch?


[-- Attachment #2: inode.patch --]
[-- Type: text/plain, Size: 1159 bytes --]

diff --git a/fs/inode.c b/fs/inode.c
index 00b804e..48eafa6 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -419,6 +419,8 @@ void inode_add_lru(struct inode *inode)
 
 static void inode_lru_list_del(struct inode *inode)
 {
+	if (inode->i_state & I_FREEING)
+		return;
 
 	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
 		this_cpu_dec(nr_unused);
@@ -609,8 +611,8 @@ void evict_inodes(struct super_block *sb)
 			continue;
 		}
 
-		inode->i_state |= I_FREEING;
 		inode_lru_list_del(inode);
+		inode->i_state |= I_FREEING;
 		spin_unlock(&inode->i_lock);
 		list_add(&inode->i_lru, &dispose);
 	}
@@ -653,8 +655,8 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)
 			continue;
 		}
 
-		inode->i_state |= I_FREEING;
 		inode_lru_list_del(inode);
+		inode->i_state |= I_FREEING;
 		spin_unlock(&inode->i_lock);
 		list_add(&inode->i_lru, &dispose);
 	}
@@ -1381,9 +1383,8 @@ static void iput_final(struct inode *inode)
 		inode->i_state &= ~I_WILL_FREE;
 	}
 
+	inode_lru_list_del(inode);
 	inode->i_state |= I_FREEING;
-	if (!list_empty(&inode->i_lru))
-		inode_lru_list_del(inode);
 	spin_unlock(&inode->i_lock);
 
 	evict(inode);

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 18-06-13 10:26:24, Glauber Costa wrote:
[...]
> Which is obviously borked since I did not fix the other callers so to move I_FREEING
> after lru del.
> 
> Michal, would you mind testing the following patch?

I was about to start testing with inode_lru_isolate fix. I will give it
few runs and then test this one if it is still relevant.

> diff --git a/fs/inode.c b/fs/inode.c
> index 00b804e..48eafa6 100644
> --- a/fs/inode.c
> +++ b/fs/inode.c
> @@ -419,6 +419,8 @@ void inode_add_lru(struct inode *inode)
>  
>  static void inode_lru_list_del(struct inode *inode)
>  {
> +	if (inode->i_state & I_FREEING)
> +		return;
>  
>  	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
>  		this_cpu_dec(nr_unused);
> @@ -609,8 +611,8 @@ void evict_inodes(struct super_block *sb)
>  			continue;
>  		}
>  
> -		inode->i_state |= I_FREEING;
>  		inode_lru_list_del(inode);
> +		inode->i_state |= I_FREEING;
>  		spin_unlock(&inode->i_lock);
>  		list_add(&inode->i_lru, &dispose);
>  	}
> @@ -653,8 +655,8 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)
>  			continue;
>  		}
>  
> -		inode->i_state |= I_FREEING;
>  		inode_lru_list_del(inode);
> +		inode->i_state |= I_FREEING;
>  		spin_unlock(&inode->i_lock);
>  		list_add(&inode->i_lru, &dispose);
>  	}
> @@ -1381,9 +1383,8 @@ static void iput_final(struct inode *inode)
>  		inode->i_state &= ~I_WILL_FREE;
>  	}
>  
> +	inode_lru_list_del(inode);
>  	inode->i_state |= I_FREEING;
> -	if (!list_empty(&inode->i_lru))
> -		inode_lru_list_del(inode);
>  	spin_unlock(&inode->i_lock);
>  
>  	evict(inode);


-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 18-06-13 10:26:24, Glauber Costa wrote:
[...]
> Michal, would you mind testing the following patch?
>
> diff --git a/fs/inode.c b/fs/inode.c
> index 00b804e..48eafa6 100644
> --- a/fs/inode.c
> +++ b/fs/inode.c
> @@ -419,6 +419,8 @@ void inode_add_lru(struct inode *inode)
>  
>  static void inode_lru_list_del(struct inode *inode)
>  {
> +	if (inode->i_state & I_FREEING)
> +		return;
>  
>  	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
>  		this_cpu_dec(nr_unused);
> @@ -609,8 +611,8 @@ void evict_inodes(struct super_block *sb)
>  			continue;
>  		}
>  
> -		inode->i_state |= I_FREEING;
>  		inode_lru_list_del(inode);
> +		inode->i_state |= I_FREEING;
>  		spin_unlock(&inode->i_lock);
>  		list_add(&inode->i_lru, &dispose);
>  	}
> @@ -653,8 +655,8 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)
>  			continue;
>  		}
>  
> -		inode->i_state |= I_FREEING;
>  		inode_lru_list_del(inode);
> +		inode->i_state |= I_FREEING;
>  		spin_unlock(&inode->i_lock);
>  		list_add(&inode->i_lru, &dispose);
>  	}
> @@ -1381,9 +1383,8 @@ static void iput_final(struct inode *inode)
>  		inode->i_state &= ~I_WILL_FREE;
>  	}
>  
> +	inode_lru_list_del(inode);
>  	inode->i_state |= I_FREEING;
> -	if (!list_empty(&inode->i_lru))
> -		inode_lru_list_del(inode);
>  	spin_unlock(&inode->i_lock);
>  
>  	evict(inode);

No luck. I have this on top of inode_lru_isolate one but still can see
hangs:
911 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118529f>] iget_locked+0x4f/0x180
[<ffffffff811efa23>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81175254>] __lookup_hash+0x34/0x40
[<ffffffff81179872>] path_lookupat+0x7a2/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
21409 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118529f>] iget_locked+0x4f/0x180
[<ffffffff811efa23>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
21745 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
22032 [<ffffffff81179862>] path_lookupat+0x792/0x830
[<ffffffff81179933>] filename_lookup+0x33/0xd0
[<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
[<ffffffff8117ab4c>] user_path_at+0xc/0x10
[<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
[<ffffffff81170116>] vfs_stat+0x16/0x20
[<ffffffff8117013f>] sys_newstat+0x1f/0x50
[<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
22621 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
[<ffffffff81183321>] find_inode_fast+0xa1/0xc0
[<ffffffff8118529f>] iget_locked+0x4f/0x180
[<ffffffff811efa23>] ext4_iget+0x33/0x9f0
[<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
[<ffffffff81174ad0>] lookup_real+0x20/0x60
[<ffffffff81177e25>] lookup_open+0x175/0x1d0
[<ffffffff8117815e>] do_last+0x2de/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
22711 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
22946 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
23393 [<ffffffff81178144>] do_last+0x2c4/0x780
[<ffffffff8117ae9a>] path_openat+0xda/0x400
[<ffffffff8117b303>] do_filp_open+0x43/0xa0
[<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
[<ffffffff81168f9c>] sys_open+0x1c/0x20
[<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Wed, Jun 19, 2013 at 09:13:46AM +0200, Michal Hocko wrote:
> On Tue 18-06-13 10:26:24, Glauber Costa wrote:
> [...]
> > Michal, would you mind testing the following patch?
> >
> > diff --git a/fs/inode.c b/fs/inode.c
> > index 00b804e..48eafa6 100644
> > --- a/fs/inode.c
> > +++ b/fs/inode.c
> > @@ -419,6 +419,8 @@ void inode_add_lru(struct inode *inode)
> >  
> >  static void inode_lru_list_del(struct inode *inode)
> >  {
> > +	if (inode->i_state & I_FREEING)
> > +		return;
> >  
> >  	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
> >  		this_cpu_dec(nr_unused);
> > @@ -609,8 +611,8 @@ void evict_inodes(struct super_block *sb)
> >  			continue;
> >  		}
> >  
> > -		inode->i_state |= I_FREEING;
> >  		inode_lru_list_del(inode);
> > +		inode->i_state |= I_FREEING;
> >  		spin_unlock(&inode->i_lock);
> >  		list_add(&inode->i_lru, &dispose);
> >  	}
> > @@ -653,8 +655,8 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)
> >  			continue;
> >  		}
> >  
> > -		inode->i_state |= I_FREEING;
> >  		inode_lru_list_del(inode);
> > +		inode->i_state |= I_FREEING;
> >  		spin_unlock(&inode->i_lock);
> >  		list_add(&inode->i_lru, &dispose);
> >  	}
> > @@ -1381,9 +1383,8 @@ static void iput_final(struct inode *inode)
> >  		inode->i_state &= ~I_WILL_FREE;
> >  	}
> >  
> > +	inode_lru_list_del(inode);
> >  	inode->i_state |= I_FREEING;
> > -	if (!list_empty(&inode->i_lru))
> > -		inode_lru_list_del(inode);
> >  	spin_unlock(&inode->i_lock);
> >  
> >  	evict(inode);
> 
> No luck. I have this on top of inode_lru_isolate one but still can see
> hangs:
> 911 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
> [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> [<ffffffff8118529f>] iget_locked+0x4f/0x180
> [<ffffffff811efa23>] ext4_iget+0x33/0x9f0
> [<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
> [<ffffffff81174ad0>] lookup_real+0x20/0x60
> [<ffffffff81175254>] __lookup_hash+0x34/0x40
> [<ffffffff81179872>] path_lookupat+0x7a2/0x830
> [<ffffffff81179933>] filename_lookup+0x33/0xd0
> [<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
> [<ffffffff8117ab4c>] user_path_at+0xc/0x10
> [<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
> [<ffffffff81170116>] vfs_stat+0x16/0x20
> [<ffffffff8117013f>] sys_newstat+0x1f/0x50
> [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 21409 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
> [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> [<ffffffff8118529f>] iget_locked+0x4f/0x180
> [<ffffffff811efa23>] ext4_iget+0x33/0x9f0
> [<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
> [<ffffffff81174ad0>] lookup_real+0x20/0x60
> [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> [<ffffffff8117815e>] do_last+0x2de/0x780
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 21745 [<ffffffff81179862>] path_lookupat+0x792/0x830
> [<ffffffff81179933>] filename_lookup+0x33/0xd0
> [<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
> [<ffffffff8117ab4c>] user_path_at+0xc/0x10
> [<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
> [<ffffffff81170116>] vfs_stat+0x16/0x20
> [<ffffffff8117013f>] sys_newstat+0x1f/0x50
> [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 22032 [<ffffffff81179862>] path_lookupat+0x792/0x830
> [<ffffffff81179933>] filename_lookup+0x33/0xd0
> [<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
> [<ffffffff8117ab4c>] user_path_at+0xc/0x10
> [<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
> [<ffffffff81170116>] vfs_stat+0x16/0x20
> [<ffffffff8117013f>] sys_newstat+0x1f/0x50
> [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 22621 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
> [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> [<ffffffff8118529f>] iget_locked+0x4f/0x180
> [<ffffffff811efa23>] ext4_iget+0x33/0x9f0
> [<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
> [<ffffffff81174ad0>] lookup_real+0x20/0x60
> [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> [<ffffffff8117815e>] do_last+0x2de/0x780
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 22711 [<ffffffff81178144>] do_last+0x2c4/0x780
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 22946 [<ffffffff81178144>] do_last+0x2c4/0x780
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 23393 [<ffffffff81178144>] do_last+0x2c4/0x780
> [<ffffffff8117ae9a>] path_openat+0xda/0x400
> [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> [<ffffffff81168f9c>] sys_open+0x1c/0x20
> [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> -- 
Sorry if you said that before Michal.

But given the backtrace, are you sure this is LRU-related? You mentioned you bisected
it but found nothing conclusive. I will keep looking but maybe this could benefit from
a broader fs look

In any case, the patch we suggested is obviously correct and we should apply nevertheless.
I will write it down and send it to Andrew.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Wed, Jun 19, 2013 at 11:35:27AM +0400, Glauber Costa wrote:
> On Wed, Jun 19, 2013 at 09:13:46AM +0200, Michal Hocko wrote:
> > On Tue 18-06-13 10:26:24, Glauber Costa wrote:
> > [...]
> > > Michal, would you mind testing the following patch?
> > >
> > > diff --git a/fs/inode.c b/fs/inode.c
> > > index 00b804e..48eafa6 100644
> > > --- a/fs/inode.c
> > > +++ b/fs/inode.c
> > > @@ -419,6 +419,8 @@ void inode_add_lru(struct inode *inode)
> > >  
> > >  static void inode_lru_list_del(struct inode *inode)
> > >  {
> > > +	if (inode->i_state & I_FREEING)
> > > +		return;
> > >  
> > >  	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
> > >  		this_cpu_dec(nr_unused);
> > > @@ -609,8 +611,8 @@ void evict_inodes(struct super_block *sb)
> > >  			continue;
> > >  		}
> > >  
> > > -		inode->i_state |= I_FREEING;
> > >  		inode_lru_list_del(inode);
> > > +		inode->i_state |= I_FREEING;
> > >  		spin_unlock(&inode->i_lock);
> > >  		list_add(&inode->i_lru, &dispose);
> > >  	}
> > > @@ -653,8 +655,8 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)
> > >  			continue;
> > >  		}
> > >  
> > > -		inode->i_state |= I_FREEING;
> > >  		inode_lru_list_del(inode);
> > > +		inode->i_state |= I_FREEING;
> > >  		spin_unlock(&inode->i_lock);
> > >  		list_add(&inode->i_lru, &dispose);
> > >  	}
> > > @@ -1381,9 +1383,8 @@ static void iput_final(struct inode *inode)
> > >  		inode->i_state &= ~I_WILL_FREE;
> > >  	}
> > >  
> > > +	inode_lru_list_del(inode);
> > >  	inode->i_state |= I_FREEING;
> > > -	if (!list_empty(&inode->i_lru))
> > > -		inode_lru_list_del(inode);
> > >  	spin_unlock(&inode->i_lock);
> > >  
> > >  	evict(inode);
> > 
> > No luck. I have this on top of inode_lru_isolate one but still can see
> > hangs:
> > 911 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
> > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > [<ffffffff8118529f>] iget_locked+0x4f/0x180
> > [<ffffffff811efa23>] ext4_iget+0x33/0x9f0
> > [<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
> > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > [<ffffffff81175254>] __lookup_hash+0x34/0x40
> > [<ffffffff81179872>] path_lookupat+0x7a2/0x830
> > [<ffffffff81179933>] filename_lookup+0x33/0xd0
> > [<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
> > [<ffffffff8117ab4c>] user_path_at+0xc/0x10
> > [<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
> > [<ffffffff81170116>] vfs_stat+0x16/0x20
> > [<ffffffff8117013f>] sys_newstat+0x1f/0x50
> > [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> > 21409 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
> > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > [<ffffffff8118529f>] iget_locked+0x4f/0x180
> > [<ffffffff811efa23>] ext4_iget+0x33/0x9f0
> > [<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
> > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > [<ffffffff8117815e>] do_last+0x2de/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> > 21745 [<ffffffff81179862>] path_lookupat+0x792/0x830
> > [<ffffffff81179933>] filename_lookup+0x33/0xd0
> > [<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
> > [<ffffffff8117ab4c>] user_path_at+0xc/0x10
> > [<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
> > [<ffffffff81170116>] vfs_stat+0x16/0x20
> > [<ffffffff8117013f>] sys_newstat+0x1f/0x50
> > [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> > 22032 [<ffffffff81179862>] path_lookupat+0x792/0x830
> > [<ffffffff81179933>] filename_lookup+0x33/0xd0
> > [<ffffffff8117ab0b>] user_path_at_empty+0x7b/0xb0
> > [<ffffffff8117ab4c>] user_path_at+0xc/0x10
> > [<ffffffff8116ff91>] vfs_fstatat+0x51/0xb0
> > [<ffffffff81170116>] vfs_stat+0x16/0x20
> > [<ffffffff8117013f>] sys_newstat+0x1f/0x50
> > [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> > 22621 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0
> > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0
> > [<ffffffff8118529f>] iget_locked+0x4f/0x180
> > [<ffffffff811efa23>] ext4_iget+0x33/0x9f0
> > [<ffffffff811f6a5c>] ext4_lookup+0xbc/0x160
> > [<ffffffff81174ad0>] lookup_real+0x20/0x60
> > [<ffffffff81177e25>] lookup_open+0x175/0x1d0
> > [<ffffffff8117815e>] do_last+0x2de/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> > 22711 [<ffffffff81178144>] do_last+0x2c4/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> > 22946 [<ffffffff81178144>] do_last+0x2c4/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> > 23393 [<ffffffff81178144>] do_last+0x2c4/0x780
> > [<ffffffff8117ae9a>] path_openat+0xda/0x400
> > [<ffffffff8117b303>] do_filp_open+0x43/0xa0
> > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0
> > [<ffffffff81168f9c>] sys_open+0x1c/0x20
> > [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [<ffffffffffffffff>] 0xffffffffffffffff
> > -- 
> Sorry if you said that before Michal.
> 
> But given the backtrace, are you sure this is LRU-related? You mentioned you bisected
> it but found nothing conclusive. I will keep looking but maybe this could benefit from
> a broader fs look
> 
> In any case, the patch we suggested is obviously correct and we should apply nevertheless.
> I will write it down and send it to Andrew.

My analysis of the LRU side code so far is:

* Assuming we are not hanging because of held locks, the fact that we
are hung at __wait_on_freeing_inode() means that someone who should
be waking us up is not. This would indicate that an inode is marked
as to-be-freed, but later on not freed.

* We will wait for free inodes if the state is I_FREEING or I_WILL_FREE.

* I_WILL_FREE is only set for a very short time during iput_final, and
the code path leads unconditionally to evict(), which wakes up any
waiters.

* clear_inode sets I_FREEING but it is only called from within evict,
which means we will wake up the waiters shortly.

* The LRU will not necessarily put the element into those states, but when
it does, it moves them to the dispose list. We will call evict() for all
ements in the dispose list, and that will unconditionally call wake_up_bit.
So it seems that if the LRU sets I_FREEING (we never set I_WILL_FREE)

* The same is true for evict_inodes and invalidate_inodes. They test
for the freeing bits and will skip the inodes marked as such. This seems
okay, since this means someone else marked them as freeing and it should
be their responsibility to wake up the callers.

So this shows that strangely enough, the code seems very safe and fine.
Still you are seeing hangs... Any chance we are hanging on the acquisition
of inode_hash_lock?

I need to be away for some hours, but I will be back to it soon. Meanwhile,
if Dave could take a look into it, that would be trully great.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Wed 19-06-13 11:35:27, Glauber Costa wrote:
[...]
> Sorry if you said that before Michal.
> 
> But given the backtrace, are you sure this is LRU-related?

No idea. I just know that my mm tree behaves correctly after the whole
series has been reverted (58f6e0c8fb37e8e37d5ac17a61a53ac236c15047) and
before the latest version of the patchset has been applied.

> You mentioned you bisected it but found nothing conclusive.

Yes, but I was interested in crashes and not hangs so I will try it
again.

I really hope this is not just some stupidness in my tree.

> I will keep looking but maybe this could benefit from
> a broader fs look
> 
> In any case, the patch we suggested is obviously correct and we should
> apply nevertheless.  I will write it down and send it to Andrew.

OK, feel free to stick my Tested-by there.

-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Wed, Jun 19, 2013 at 03:57:16PM +0200, Michal Hocko wrote:
> On Wed 19-06-13 11:35:27, Glauber Costa wrote:
> [...]
> > Sorry if you said that before Michal.
> > 
> > But given the backtrace, are you sure this is LRU-related?
> 
> No idea. I just know that my mm tree behaves correctly after the whole
> series has been reverted (58f6e0c8fb37e8e37d5ac17a61a53ac236c15047) and
> before the latest version of the patchset has been applied.
> 
> > You mentioned you bisected it but found nothing conclusive.
> 
> Yes, but I was interested in crashes and not hangs so I will try it
> again.
> 
> I really hope this is not just some stupidness in my tree.
> 
Okay. Just looking at the stack trace you provided me it would be hard
to implicate us. But it is not totally unreasonable either since we touch
things around this. I would right now assign more probability to a tricky
bug than to some misconfiguration on your side.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Wed 19-06-13 09:13:46, Michal Hocko wrote:
> On Tue 18-06-13 10:26:24, Glauber Costa wrote:
> [...]
> > Michal, would you mind testing the following patch?
> >
> > diff --git a/fs/inode.c b/fs/inode.c
> > index 00b804e..48eafa6 100644
> > --- a/fs/inode.c
> > +++ b/fs/inode.c
> > @@ -419,6 +419,8 @@ void inode_add_lru(struct inode *inode)
> >  
> >  static void inode_lru_list_del(struct inode *inode)
> >  {
> > +	if (inode->i_state & I_FREEING)
> > +		return;
> >  
> >  	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
> >  		this_cpu_dec(nr_unused);
> > @@ -609,8 +611,8 @@ void evict_inodes(struct super_block *sb)
> >  			continue;
> >  		}
> >  
> > -		inode->i_state |= I_FREEING;
> >  		inode_lru_list_del(inode);
> > +		inode->i_state |= I_FREEING;
> >  		spin_unlock(&inode->i_lock);
> >  		list_add(&inode->i_lru, &dispose);
> >  	}
> > @@ -653,8 +655,8 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)
> >  			continue;
> >  		}
> >  
> > -		inode->i_state |= I_FREEING;
> >  		inode_lru_list_del(inode);
> > +		inode->i_state |= I_FREEING;
> >  		spin_unlock(&inode->i_lock);
> >  		list_add(&inode->i_lru, &dispose);
> >  	}
> > @@ -1381,9 +1383,8 @@ static void iput_final(struct inode *inode)
> >  		inode->i_state &= ~I_WILL_FREE;
> >  	}
> >  
> > +	inode_lru_list_del(inode);
> >  	inode->i_state |= I_FREEING;
> > -	if (!list_empty(&inode->i_lru))
> > -		inode_lru_list_del(inode);
> >  	spin_unlock(&inode->i_lock);
> >  
> >  	evict(inode);
> 
> No luck. I have this on top of inode_lru_isolate one but still can see

And I was lucky enough to hit another BUG_ON with this kernel (the above
patch and inode_lru_isolate-fix):
[84091.219056] ------------[ cut here ]------------
[84091.220015] kernel BUG at mm/list_lru.c:42!
[84091.220015] invalid opcode: 0000 [#1] SMP 
[84091.220015] Modules linked in: edd nfsv3 nfs_acl nfs fscache lockd sunrpc af_packet bridge stp llc cpufreq_conservative cpufreq_userspace cpufreq_powersave fuse loop dm_mod powernow_k8 tg3 kvm_amd kvm ptp e1000 pps_core shpchp edac_core i2c_amd756 amd_rng pci_hotplug k8temp sg i2c_amd8111 edac_mce_amd serio_raw sr_mod pcspkr cdrom button ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[84091.220015] CPU 1 
[84091.220015] Pid: 32545, comm: rm Not tainted 3.9.0mmotmdebugging1+ #1472 AMD A8440/WARTHOG
[84091.220015] RIP: 0010:[<ffffffff81127fff>]  [<ffffffff81127fff>] list_lru_del+0xcf/0xe0
[84091.220015] RSP: 0018:ffff88001de85df8  EFLAGS: 00010286
[84091.220015] RAX: ffffffffffffffff RBX: ffff88001e1ce2c0 RCX: 0000000000000002
[84091.220015] RDX: ffff88001e1ce2c8 RSI: ffff8800087f4220 RDI: ffff88001e1ce2c0
[84091.220015] RBP: ffff88001de85e18 R08: 0000000000000000 R09: 0000000000000000
[84091.220015] R10: ffff88001d539128 R11: ffff880018234882 R12: ffff8800087f4220
[84091.220015] R13: ffff88001c68bc40 R14: 0000000000000000 R15: ffff88001de85ea8
[84091.220015] FS:  00007f43adb30700(0000) GS:ffff88001f100000(0000) knlGS:0000000000000000
[84091.220015] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[84091.220015] CR2: 0000000001ffed30 CR3: 000000001e02e000 CR4: 00000000000007e0
[84091.220015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[84091.220015] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[84091.220015] Process rm (pid: 32545, threadinfo ffff88001de84000, task ffff88001c22e5c0)
[84091.220015] Stack:
[84091.220015]  ffff8800087f4130 ffff8800087f41b8 ffff88001c68b800 0000000000000000
[84091.220015]  ffff88001de85e48 ffffffff81184357 ffff88001de85e48 ffff8800087f4130
[84091.220015]  ffff88001e005000 ffff880014e4eb40 ffff88001de85e68 ffffffff81184418
[84091.220015] Call Trace:
[84091.220015]  [<ffffffff81184357>] iput_final+0x117/0x190
[84091.220015]  [<ffffffff81184418>] iput+0x48/0x60
[84091.220015]  [<ffffffff8117a804>] do_unlinkat+0x214/0x240
[84091.220015]  [<ffffffff8117aa4d>] sys_unlinkat+0x1d/0x40
[84091.220015]  [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
[84091.220015] Code: 5c 41 5d b8 01 00 00 00 41 5e c9 c3 49 8d 45 08 f0 45 0f b3 75 08 eb db 0f 1f 40 00 66 83 03 01 5b 41 5c 41 5d 31 c0 41 5e c9 c3 <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 ba 00 00 
[84091.220015] RIP  [<ffffffff81127fff>] list_lru_del+0xcf/0xe0
[84091.220015]  RSP <ffff88001de85df8>
[84091.470390] ---[ end trace e6915e8ee0f5f079 ]---

Which is BUG_ON(nlru->nr_items < 0) from iput_final path. So it seems
that there is still a race there.
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Wed, Jun 19, 2013 at 04:28:01PM +0200, Michal Hocko wrote:
> On Wed 19-06-13 09:13:46, Michal Hocko wrote:
> > On Tue 18-06-13 10:26:24, Glauber Costa wrote:
> > [...]
> > > Michal, would you mind testing the following patch?
> > >
> > > diff --git a/fs/inode.c b/fs/inode.c
> > > index 00b804e..48eafa6 100644
> > > --- a/fs/inode.c
> > > +++ b/fs/inode.c
> > > @@ -419,6 +419,8 @@ void inode_add_lru(struct inode *inode)
> > >  
> > >  static void inode_lru_list_del(struct inode *inode)
> > >  {
> > > +	if (inode->i_state & I_FREEING)
> > > +		return;
> > >  
> > >  	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
> > >  		this_cpu_dec(nr_unused);
> > > @@ -609,8 +611,8 @@ void evict_inodes(struct super_block *sb)
> > >  			continue;
> > >  		}
> > >  
> > > -		inode->i_state |= I_FREEING;
> > >  		inode_lru_list_del(inode);
> > > +		inode->i_state |= I_FREEING;
> > >  		spin_unlock(&inode->i_lock);
> > >  		list_add(&inode->i_lru, &dispose);
> > >  	}
> > > @@ -653,8 +655,8 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)
> > >  			continue;
> > >  		}
> > >  
> > > -		inode->i_state |= I_FREEING;
> > >  		inode_lru_list_del(inode);
> > > +		inode->i_state |= I_FREEING;
> > >  		spin_unlock(&inode->i_lock);
> > >  		list_add(&inode->i_lru, &dispose);
> > >  	}
> > > @@ -1381,9 +1383,8 @@ static void iput_final(struct inode *inode)
> > >  		inode->i_state &= ~I_WILL_FREE;
> > >  	}
> > >  
> > > +	inode_lru_list_del(inode);
> > >  	inode->i_state |= I_FREEING;
> > > -	if (!list_empty(&inode->i_lru))
> > > -		inode_lru_list_del(inode);
> > >  	spin_unlock(&inode->i_lock);
> > >  
> > >  	evict(inode);
> > 
> > No luck. I have this on top of inode_lru_isolate one but still can see
> 
> And I was lucky enough to hit another BUG_ON with this kernel (the above
> patch and inode_lru_isolate-fix):
> [84091.219056] ------------[ cut here ]------------
> [84091.220015] kernel BUG at mm/list_lru.c:42!
> [84091.220015] invalid opcode: 0000 [#1] SMP 
> [84091.220015] Modules linked in: edd nfsv3 nfs_acl nfs fscache lockd sunrpc af_packet bridge stp llc cpufreq_conservative cpufreq_userspace cpufreq_powersave fuse loop dm_mod powernow_k8 tg3 kvm_amd kvm ptp e1000 pps_core shpchp edac_core i2c_amd756 amd_rng pci_hotplug k8temp sg i2c_amd8111 edac_mce_amd serio_raw sr_mod pcspkr cdrom button ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
> [84091.220015] CPU 1 
> [84091.220015] Pid: 32545, comm: rm Not tainted 3.9.0mmotmdebugging1+ #1472 AMD A8440/WARTHOG
> [84091.220015] RIP: 0010:[<ffffffff81127fff>]  [<ffffffff81127fff>] list_lru_del+0xcf/0xe0
> [84091.220015] RSP: 0018:ffff88001de85df8  EFLAGS: 00010286
> [84091.220015] RAX: ffffffffffffffff RBX: ffff88001e1ce2c0 RCX: 0000000000000002
> [84091.220015] RDX: ffff88001e1ce2c8 RSI: ffff8800087f4220 RDI: ffff88001e1ce2c0
> [84091.220015] RBP: ffff88001de85e18 R08: 0000000000000000 R09: 0000000000000000
> [84091.220015] R10: ffff88001d539128 R11: ffff880018234882 R12: ffff8800087f4220
> [84091.220015] R13: ffff88001c68bc40 R14: 0000000000000000 R15: ffff88001de85ea8
> [84091.220015] FS:  00007f43adb30700(0000) GS:ffff88001f100000(0000) knlGS:0000000000000000
> [84091.220015] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [84091.220015] CR2: 0000000001ffed30 CR3: 000000001e02e000 CR4: 00000000000007e0
> [84091.220015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [84091.220015] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [84091.220015] Process rm (pid: 32545, threadinfo ffff88001de84000, task ffff88001c22e5c0)
> [84091.220015] Stack:
> [84091.220015]  ffff8800087f4130 ffff8800087f41b8 ffff88001c68b800 0000000000000000
> [84091.220015]  ffff88001de85e48 ffffffff81184357 ffff88001de85e48 ffff8800087f4130
> [84091.220015]  ffff88001e005000 ffff880014e4eb40 ffff88001de85e68 ffffffff81184418
> [84091.220015] Call Trace:
> [84091.220015]  [<ffffffff81184357>] iput_final+0x117/0x190
> [84091.220015]  [<ffffffff81184418>] iput+0x48/0x60
> [84091.220015]  [<ffffffff8117a804>] do_unlinkat+0x214/0x240
> [84091.220015]  [<ffffffff8117aa4d>] sys_unlinkat+0x1d/0x40
> [84091.220015]  [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> [84091.220015] Code: 5c 41 5d b8 01 00 00 00 41 5e c9 c3 49 8d 45 08 f0 45 0f b3 75 08 eb db 0f 1f 40 00 66 83 03 01 5b 41 5c 41 5d 31 c0 41 5e c9 c3 <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 ba 00 00 
> [84091.220015] RIP  [<ffffffff81127fff>] list_lru_del+0xcf/0xe0
> [84091.220015]  RSP <ffff88001de85df8>
> [84091.470390] ---[ end trace e6915e8ee0f5f079 ]---
> 
> Which is BUG_ON(nlru->nr_items < 0) from iput_final path. So it seems
> that there is still a race there.

I am still looking at this - still can't reproduce, still don't know what is going
on.

Could you share with me your .config and your hardware info and dmesg? In particular, I want
to know how many nodes do you have.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

[-- Attachment #1: Type: text/plain, Size: 3604 bytes --]

On Thu 20-06-13 18:11:38, Glauber Costa wrote:
[...]
> > [84091.219056] ------------[ cut here ]------------
> > [84091.220015] kernel BUG at mm/list_lru.c:42!
> > [84091.220015] invalid opcode: 0000 [#1] SMP 
> > [84091.220015] Modules linked in: edd nfsv3 nfs_acl nfs fscache lockd sunrpc af_packet bridge stp llc cpufreq_conservative cpufreq_userspace cpufreq_powersave fuse loop dm_mod powernow_k8 tg3 kvm_amd kvm ptp e1000 pps_core shpchp edac_core i2c_amd756 amd_rng pci_hotplug k8temp sg i2c_amd8111 edac_mce_amd serio_raw sr_mod pcspkr cdrom button ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
> > [84091.220015] CPU 1 
> > [84091.220015] Pid: 32545, comm: rm Not tainted 3.9.0mmotmdebugging1+ #1472 AMD A8440/WARTHOG
> > [84091.220015] RIP: 0010:[<ffffffff81127fff>]  [<ffffffff81127fff>] list_lru_del+0xcf/0xe0
> > [84091.220015] RSP: 0018:ffff88001de85df8  EFLAGS: 00010286
> > [84091.220015] RAX: ffffffffffffffff RBX: ffff88001e1ce2c0 RCX: 0000000000000002
> > [84091.220015] RDX: ffff88001e1ce2c8 RSI: ffff8800087f4220 RDI: ffff88001e1ce2c0
> > [84091.220015] RBP: ffff88001de85e18 R08: 0000000000000000 R09: 0000000000000000
> > [84091.220015] R10: ffff88001d539128 R11: ffff880018234882 R12: ffff8800087f4220
> > [84091.220015] R13: ffff88001c68bc40 R14: 0000000000000000 R15: ffff88001de85ea8
> > [84091.220015] FS:  00007f43adb30700(0000) GS:ffff88001f100000(0000) knlGS:0000000000000000
> > [84091.220015] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [84091.220015] CR2: 0000000001ffed30 CR3: 000000001e02e000 CR4: 00000000000007e0
> > [84091.220015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [84091.220015] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [84091.220015] Process rm (pid: 32545, threadinfo ffff88001de84000, task ffff88001c22e5c0)
> > [84091.220015] Stack:
> > [84091.220015]  ffff8800087f4130 ffff8800087f41b8 ffff88001c68b800 0000000000000000
> > [84091.220015]  ffff88001de85e48 ffffffff81184357 ffff88001de85e48 ffff8800087f4130
> > [84091.220015]  ffff88001e005000 ffff880014e4eb40 ffff88001de85e68 ffffffff81184418
> > [84091.220015] Call Trace:
> > [84091.220015]  [<ffffffff81184357>] iput_final+0x117/0x190
> > [84091.220015]  [<ffffffff81184418>] iput+0x48/0x60
> > [84091.220015]  [<ffffffff8117a804>] do_unlinkat+0x214/0x240
> > [84091.220015]  [<ffffffff8117aa4d>] sys_unlinkat+0x1d/0x40
> > [84091.220015]  [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > [84091.220015] Code: 5c 41 5d b8 01 00 00 00 41 5e c9 c3 49 8d 45 08 f0 45 0f b3 75 08 eb db 0f 1f 40 00 66 83 03 01 5b 41 5c 41 5d 31 c0 41 5e c9 c3 <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 ba 00 00 
> > [84091.220015] RIP  [<ffffffff81127fff>] list_lru_del+0xcf/0xe0
> > [84091.220015]  RSP <ffff88001de85df8>
> > [84091.470390] ---[ end trace e6915e8ee0f5f079 ]---
> > 
> > Which is BUG_ON(nlru->nr_items < 0) from iput_final path. So it seems
> > that there is still a race there.
> 
> I am still looking at this - still can't reproduce, still don't know what is going
> on.

I am bisecting it again. It is quite tedious, though, because good case
is hard to be sure about.

> Could you share with me your .config and your hardware info and
> dmesg? In particular, I want to know how many nodes do you have.

Well, the machine has 4 nodes but only 2 of them are initialized because
I am booting with mem=1G to make a bigger memory pressure. dmesg and
zoneinfo is attached as well as the config.

-- 
Michal Hocko
SUSE Labs

[-- Attachment #2: demon.config --]
[-- Type: text/plain, Size: 135924 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 3.9.0 Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_GPIO=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION="bisect"
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
CONFIG_FHANDLE=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
# CONFIG_AUDIT_LOGINUID_IMMUTABLE is not set
CONFIG_HAVE_GENERIC_HARDIRQS=y

#
# IRQ subsystem
#
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_GENERIC_IRQ_CHIP=y
CONFIG_IRQ_DOMAIN=y
# CONFIG_IRQ_DOMAIN_DEBUG is not set
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_ALWAYS_USE_PERSISTENT_CLOCK=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_IRQ_TIME_ACCOUNTING is not set
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
CONFIG_RCU_STALL_COMMON=y
# CONFIG_RCU_USER_QS is not set
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_RCU_NOCB_CPU is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=18
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
CONFIG_ARCH_USES_NUMA_PROT_NONE=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_NUMA_BALANCING=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
# CONFIG_MEMCG_KMEM is not set
# CONFIG_MEMCG_DEBUG_ASYNC_DESTROY is not set
# CONFIG_CGROUP_HUGETLB is not set
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_CFS_BANDWIDTH is not set
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
# CONFIG_SCHED_AUTOGROUP is not set
CONFIG_MM_OWNER=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EXPERT is not set
CONFIG_HAVE_UID16=y
CONFIG_UID16=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
# CONFIG_COMPAT_BRK is not set
CONFIG_SLAB=y
# CONFIG_SLUB is not set
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
# CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_OPTPROBES=y
CONFIG_UPROBES=y
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
# CONFIG_MODULE_SIG is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_THROTTLING=y

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
# CONFIG_AMIGA_PARTITION is not set
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
# CONFIG_MINIX_SUBPARTITION is not set
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_SYSV68_PARTITION=y
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_ASN1=m
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_VSMP is not set
# CONFIG_X86_INTEL_LPSS is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_PARAVIRT_GUEST=y
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not set
# CONFIG_XEN is not set
# CONFIG_XEN_PRIVILEGED_GUEST is not set
CONFIG_KVM_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_SPINLOCKS is not set
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_NO_BOOTMEM=y
CONFIG_MEMTEST=y
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=512
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_MCE_INJECT=m
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_I8K=m
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_MICROCODE_INTEL_LIB=y
CONFIG_MICROCODE_INTEL_EARLY=y
CONFIG_MICROCODE_EARLY=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=m
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
CONFIG_NUMA_EMU=y
CONFIG_NODES_SHIFT=9
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_MEMORY_ISOLATION=y
CONFIG_MOVABLE_NODE=y
CONFIG_HAVE_BOOTMEM_INFO_NODE=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_X86_SMAP=y
CONFIG_EFI=y
CONFIG_EFI_STUB=y
CONFIG_SECCOMP=y
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
# CONFIG_KEXEC_JUMP is not set
CONFIG_PHYSICAL_START=0x200000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
# CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set
# CONFIG_DEBUG_HOTPLUG_CPU0 is not set
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y

#
# Power management and ACPI options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_AUTOSLEEP=y
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_WAKELOCKS_LIMIT=100
CONFIG_PM_WAKELOCKS_GC=y
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_PM_SLEEP_DEBUG=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_EC_DEBUGFS=m
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=m
# CONFIG_ACPI_DOCK is not set
CONFIG_ACPI_I2C=y
CONFIG_ACPI_PROCESSOR=m
# CONFIG_ACPI_IPMI is not set
CONFIG_ACPI_HOTPLUG_CPU=y
# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
CONFIG_ACPI_THERMAL=m
CONFIG_ACPI_NUMA=y
CONFIG_ACPI_CUSTOM_DSDT_FILE=""
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_INITRD_TABLE_OVERRIDE=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_DEBUG=y
# CONFIG_ACPI_DEBUG_FUNC_TRACE is not set
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_MEMORY=m
CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=y
# CONFIG_ACPI_CUSTOM_METHOD is not set
CONFIG_ACPI_BGRT=y
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=y
CONFIG_ACPI_APEI_PCIEAER=y
CONFIG_ACPI_APEI_MEMORY_FAILURE=y
CONFIG_ACPI_APEI_EINJ=m
# CONFIG_ACPI_APEI_ERST_DEBUG is not set
# CONFIG_SFI is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=m
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m

#
# x86 CPU frequency scaling drivers
#
# CONFIG_X86_INTEL_PSTATE is not set
CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ_CPB=y
CONFIG_X86_POWERNOW_K8=m
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set

#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_MULTIPLE_DRIVERS is not set
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
CONFIG_INTEL_IDLE=y

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
CONFIG_PCIEAER_INJECT=m
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_REALLOC_ENABLE_AUTO is not set
CONFIG_PCI_STUB=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=m
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=m
CONFIG_RAPIDIO=y
CONFIG_RAPIDIO_TSI721=y
CONFIG_RAPIDIO_DISC_TIMEOUT=30
CONFIG_RAPIDIO_ENABLE_RX_TX_PORTS=y
CONFIG_RAPIDIO_DMA_ENGINE=y
CONFIG_RAPIDIO_DEBUG=y
# CONFIG_RAPIDIO_ENUM_BASIC is not set
CONFIG_RAPIDIO_TSI57X=y
CONFIG_RAPIDIO_CPS_XX=y
CONFIG_RAPIDIO_TSI568=y
CONFIG_RAPIDIO_CPS_GEN2=y
CONFIG_RAPIDIO_TSI500=y

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=m
CONFIG_COREDUMP=y
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
# CONFIG_X86_X32 is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y

#
# Networking options
#
CONFIG_PACKET=m
CONFIG_PACKET_DIAG=m
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=m
CONFIG_XFRM_USER=m
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
# CONFIG_NET_KEY_MIGRATE is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
# CONFIG_IP_FIB_TRIE_STATS is not set
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_RARP=y
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_NET_IPVTI=m
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_INET_UDP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
# CONFIG_TCP_CONG_HSTCP is not set
# CONFIG_TCP_CONG_HYBLA is not set
# CONFIG_TCP_CONG_VEGAS is not set
# CONFIG_TCP_CONG_SCALABLE is not set
# CONFIG_TCP_CONG_LP is not set
# CONFIG_TCP_CONG_VENO is not set
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
# CONFIG_IPV6_ROUTE_INFO is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
CONFIG_IPV6_SIT=m
# CONFIG_IPV6_SIT_6RD is not set
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_GRE=m
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_TIMEOUT=y
CONFIG_NF_CONNTRACK_TIMESTAMP=y
# CONFIG_NF_CT_PROTO_DCCP is not set
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
# CONFIG_NF_CONNTRACK_SANE is not set
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NF_CT_NETLINK_TIMEOUT=m
CONFIG_NF_CT_NETLINK_HELPER=m
CONFIG_NETFILTER_NETLINK_QUEUE_CT=y
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
CONFIG_NF_NAT_TFTP=m
# CONFIG_NETFILTER_TPROXY is not set
CONFIG_NETFILTER_XTABLES=m

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_SET=m

#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_AUDIT=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_HMARK=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_LOG=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NETMAP=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_REDIRECT=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
# CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP is not set

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
# CONFIG_NETFILTER_XT_MATCH_CONNLABEL is not set
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_NFACCT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
CONFIG_IP_SET=m
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=m
CONFIG_IP_SET_BITMAP_IPMAC=m
CONFIG_IP_SET_BITMAP_PORT=m
CONFIG_IP_SET_HASH_IP=m
CONFIG_IP_SET_HASH_IPPORT=m
CONFIG_IP_SET_HASH_IPPORTIP=m
CONFIG_IP_SET_HASH_IPPORTNET=m
CONFIG_IP_SET_HASH_NET=m
CONFIG_IP_SET_HASH_NETPORT=m
CONFIG_IP_SET_HASH_NETIFACE=m
CONFIG_IP_SET_LIST_SET=m
CONFIG_IP_VS=m
CONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
# CONFIG_NF_CONNTRACK_PROC_COMPAT is not set
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_RPFILTER=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT_IPV4=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_IP_NF_MANGLE=m
# CONFIG_IP_NF_TARGET_CLUSTERIP is not set
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RPFILTER=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
CONFIG_NF_NAT_IPV6=m
CONFIG_IP6_NF_TARGET_MASQUERADE=m
CONFIG_IP6_NF_TARGET_NPT=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
# CONFIG_IP_DCCP is not set
CONFIG_IP_SCTP=m
CONFIG_NET_SCTPPROBE=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5=y
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_SHA1 is not set
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_NONE is not set
CONFIG_SCTP_COOKIE_HMAC_MD5=y
# CONFIG_SCTP_COOKIE_HMAC_SHA1 is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
# CONFIG_L2TP_V3 is not set
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_BRIDGE_VLAN_FILTERING is not set
CONFIG_HAVE_NET_DSA=y
CONFIG_NET_DSA=m
CONFIG_NET_DSA_TAG_DSA=y
CONFIG_NET_DSA_TAG_EDSA=y
CONFIG_NET_DSA_TAG_TRAILER=y
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
# CONFIG_VLAN_8021Q_MVRP is not set
# CONFIG_DECNET is not set
CONFIG_LLC=m
CONFIG_LLC2=m
CONFIG_IPX=m
CONFIG_IPX_INTERN=y
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=m
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
CONFIG_PHONET=m
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
CONFIG_NET_SCH_MQPRIO=m
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_QFQ=m
CONFIG_NET_SCH_CODEL=m
CONFIG_NET_SCH_FQ_CODEL=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_SCH_PLUG=m

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_EMATCH_IPSET=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
CONFIG_DCB=y
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_BLA=y
CONFIG_BATMAN_ADV_DAT=y
CONFIG_BATMAN_ADV_DEBUG=y
CONFIG_OPENVSWITCH=m
# CONFIG_VSOCKETS is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_NETPRIO_CGROUP=m
CONFIG_BQL=y
CONFIG_BPF_JIT=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
# CONFIG_NET_DROP_MONITOR is not set
CONFIG_HAMRADIO=y

#
# Packet Radio protocols
#
CONFIG_AX25=m
CONFIG_AX25_DAMA_SLAVE=y
CONFIG_NETROM=m
CONFIG_ROSE=m

#
# AX.25 network device drivers
#
CONFIG_MKISS=m
CONFIG_6PACK=m
CONFIG_BPQETHER=m
CONFIG_BAYCOM_SER_FDX=m
CONFIG_BAYCOM_SER_HDX=m
CONFIG_BAYCOM_PAR=m
CONFIG_YAM=m
# CONFIG_CAN is not set
CONFIG_IRDA=m

#
# IrDA protocols
#
CONFIG_IRLAN=m
CONFIG_IRNET=m
CONFIG_IRCOMM=m
CONFIG_IRDA_ULTRA=y

#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
# CONFIG_IRDA_FAST_RR is not set
# CONFIG_IRDA_DEBUG is not set

#
# Infrared-port device drivers
#

#
# SIR device drivers
#
CONFIG_IRTTY_SIR=m

#
# Dongle support
#
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
CONFIG_ACTISYS_DONGLE=m
CONFIG_TEKRAM_DONGLE=m
CONFIG_TOIM3232_DONGLE=m
CONFIG_LITELINK_DONGLE=m
# CONFIG_MA600_DONGLE is not set
# CONFIG_GIRBIL_DONGLE is not set
# CONFIG_MCP2120_DONGLE is not set
# CONFIG_OLD_BELKIN_DONGLE is not set
# CONFIG_ACT200L_DONGLE is not set
# CONFIG_KINGSUN_DONGLE is not set
# CONFIG_KSDAZZLE_DONGLE is not set
# CONFIG_KS959_DONGLE is not set

#
# FIR device drivers
#
CONFIG_USB_IRDA=m
# CONFIG_SIGMATEL_FIR is not set
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=m
CONFIG_SMC_IRCC_FIR=m
# CONFIG_ALI_FIR is not set
# CONFIG_VLSI_FIR is not set
CONFIG_VIA_FIR=m
# CONFIG_MCS_FIR is not set
CONFIG_BT=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_CMTP=m
CONFIG_BT_HIDP=m

#
# Bluetooth device drivers
#
CONFIG_BT_HCIBTUSB=m
CONFIG_BT_HCIBTSDIO=m
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_ATH3K=y
CONFIG_BT_HCIUART_LL=y
CONFIG_BT_HCIUART_3WIRE=y
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIDTL1=m
CONFIG_BT_HCIBT3C=m
CONFIG_BT_HCIBLUECARD=m
CONFIG_BT_HCIBTUART=m
CONFIG_BT_HCIVHCI=m
CONFIG_BT_MRVL=m
CONFIG_BT_MRVL_SDIO=m
CONFIG_BT_ATH3K=m
CONFIG_BT_WILINK=m
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_WIRELESS_EXT=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_WEXT_SPY=y
CONFIG_WEXT_PRIV=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_REG_DEBUG is not set
CONFIG_CFG80211_DEFAULT_PS=y
# CONFIG_CFG80211_DEBUGFS is not set
# CONFIG_CFG80211_INTERNAL_REGDB is not set
CONFIG_CFG80211_WEXT=y
CONFIG_LIB80211=m
CONFIG_LIB80211_CRYPT_WEP=m
CONFIG_LIB80211_CRYPT_CCMP=m
CONFIG_LIB80211_CRYPT_TKIP=m
# CONFIG_LIB80211_DEBUG is not set
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
# CONFIG_MAC80211_MESH is not set
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
# CONFIG_MAC80211_MESSAGE_TRACING is not set
# CONFIG_MAC80211_DEBUG_MENU is not set
CONFIG_WIMAX=m
CONFIG_WIMAX_DEBUG_LEVEL=8
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
CONFIG_NET_9P=m
# CONFIG_NET_9P_RDMA is not set
# CONFIG_NET_9P_DEBUG is not set
CONFIG_CAIF=m
# CONFIG_CAIF_DEBUG is not set
CONFIG_CAIF_NETDEV=m
CONFIG_CAIF_USB=m
CONFIG_CEPH_LIB=m
CONFIG_CEPH_LIB_PRETTYDEBUG=y
# CONFIG_CEPH_LIB_USE_DNS_RESOLVER is not set
CONFIG_NFC=m
CONFIG_NFC_NCI=m
CONFIG_NFC_HCI=m
CONFIG_NFC_SHDLC=y
CONFIG_NFC_LLCP=y

#
# Near Field Communication (NFC) devices
#
CONFIG_NFC_PN533=m
CONFIG_NFC_WILINK=m
# CONFIG_NFC_PN544 is not set
# CONFIG_NFC_MICROREAD is not set
CONFIG_HAVE_BPF_JIT=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
# CONFIG_STANDALONE is not set
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_FIRMWARE_IN_KERNEL is not set
CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_GENERIC_CPU_DEVICES is not set
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=m
CONFIG_REGMAP_MMIO=m
CONFIG_REGMAP_IRQ=y
CONFIG_DMA_SHARED_BUFFER=y

#
# Bus devices
#
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
CONFIG_MTD=m
CONFIG_MTD_TESTS=m
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
# CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set
# CONFIG_MTD_REDBOOT_PARTS_READONLY is not set
# CONFIG_MTD_CMDLINE_PARTS is not set
CONFIG_MTD_AR7_PARTS=m

#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=m
CONFIG_HAVE_MTD_OTP=y
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_MTD_BLOCK_RO=m
CONFIG_FTL=m
CONFIG_NFTL=m
CONFIG_NFTL_RW=y
CONFIG_INFTL=m
CONFIG_RFD_FTL=m
CONFIG_SSFDC=m
# CONFIG_SM_FTL is not set
CONFIG_MTD_OOPS=m
CONFIG_MTD_SWAP=m

#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
CONFIG_MTD_CFI_ADV_OPTIONS=y
CONFIG_MTD_CFI_NOSWAP=y
# CONFIG_MTD_CFI_BE_BYTE_SWAP is not set
# CONFIG_MTD_CFI_LE_BYTE_SWAP is not set
CONFIG_MTD_CFI_GEOMETRY=y
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
CONFIG_MTD_MAP_BANK_WIDTH_8=y
CONFIG_MTD_MAP_BANK_WIDTH_16=y
CONFIG_MTD_MAP_BANK_WIDTH_32=y
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
CONFIG_MTD_CFI_I4=y
CONFIG_MTD_CFI_I8=y
CONFIG_MTD_OTP=y
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
CONFIG_MTD_ROM=m
CONFIG_MTD_ABSENT=m

#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
CONFIG_MTD_PHYSMAP=m
CONFIG_MTD_PHYSMAP_COMPAT=y
CONFIG_MTD_PHYSMAP_START=0x8000000
CONFIG_MTD_PHYSMAP_LEN=0
CONFIG_MTD_PHYSMAP_BANKWIDTH=2
CONFIG_MTD_SC520CDP=m
CONFIG_MTD_NETSC520=m
CONFIG_MTD_TS5500=m
CONFIG_MTD_SBC_GXX=m
CONFIG_MTD_AMD76XROM=m
CONFIG_MTD_ICHXROM=m
CONFIG_MTD_ESB2ROM=m
CONFIG_MTD_CK804XROM=m
CONFIG_MTD_SCB2_FLASH=m
CONFIG_MTD_NETtel=m
CONFIG_MTD_L440GX=m
CONFIG_MTD_PCI=m
CONFIG_MTD_PCMCIA=m
# CONFIG_MTD_PCMCIA_ANONYMOUS is not set
CONFIG_MTD_GPIO_ADDR=m
CONFIG_MTD_INTEL_VR_NOR=m
CONFIG_MTD_PLATRAM=m
CONFIG_MTD_LATCH_ADDR=m

#
# Self-contained MTD device drivers
#
CONFIG_MTD_PMC551=m
CONFIG_MTD_PMC551_BUGFIX=y
# CONFIG_MTD_PMC551_DEBUG is not set
CONFIG_MTD_SLRAM=m
CONFIG_MTD_PHRAM=m
CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m

#
# Disk-On-Chip Device Drivers
#
CONFIG_MTD_DOC2000=m
CONFIG_MTD_DOC2001=m
CONFIG_MTD_DOC2001PLUS=m
CONFIG_MTD_DOCG3=m
CONFIG_BCH_CONST_M=14
CONFIG_BCH_CONST_T=4
CONFIG_MTD_DOCPROBE=m
CONFIG_MTD_DOCECC=m
CONFIG_MTD_DOCPROBE_ADVANCED=y
CONFIG_MTD_DOCPROBE_ADDRESS=0x0000
CONFIG_MTD_DOCPROBE_HIGH=y
CONFIG_MTD_DOCPROBE_55AA=y
CONFIG_MTD_NAND_ECC=m
CONFIG_MTD_NAND_ECC_SMC=y
CONFIG_MTD_NAND=m
CONFIG_MTD_NAND_BCH=m
CONFIG_MTD_NAND_ECC_BCH=y
CONFIG_MTD_SM_COMMON=m
CONFIG_MTD_NAND_MUSEUM_IDS=y
# CONFIG_MTD_NAND_DENALI is not set
CONFIG_MTD_NAND_IDS=m
CONFIG_MTD_NAND_RICOH=m
# CONFIG_MTD_NAND_DISKONCHIP is not set
# CONFIG_MTD_NAND_DOCG4 is not set
CONFIG_MTD_NAND_CAFE=m
CONFIG_MTD_NAND_NANDSIM=m
CONFIG_MTD_NAND_PLATFORM=m
CONFIG_MTD_ALAUDA=m
CONFIG_MTD_ONENAND=m
CONFIG_MTD_ONENAND_VERIFY_WRITE=y
CONFIG_MTD_ONENAND_GENERIC=m
CONFIG_MTD_ONENAND_OTP=y
CONFIG_MTD_ONENAND_2X_PROGRAM=y
CONFIG_MTD_ONENAND_SIM=m

#
# LPDDR flash memory drivers
#
CONFIG_MTD_LPDDR=m
CONFIG_MTD_QINFO_PROBE=m
CONFIG_MTD_UBI=m
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_LIMIT=20
# CONFIG_MTD_UBI_FASTMAP is not set
CONFIG_MTD_UBI_GLUEBI=m
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_PC_SUPERIO=y
CONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_AX88796=m
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_PARIDE=m

#
# Parallel IDE high-level drivers
#
CONFIG_PARIDE_PD=m
CONFIG_PARIDE_PCD=m
CONFIG_PARIDE_PF=m
CONFIG_PARIDE_PT=m
CONFIG_PARIDE_PG=m

#
# Parallel IDE protocol modules
#
CONFIG_PARIDE_ATEN=m
CONFIG_PARIDE_BPCK=m
CONFIG_PARIDE_COMM=m
CONFIG_PARIDE_DSTR=m
CONFIG_PARIDE_FIT2=m
CONFIG_PARIDE_FIT3=m
CONFIG_PARIDE_EPAT=m
# CONFIG_PARIDE_EPATC8 is not set
CONFIG_PARIDE_EPIA=m
CONFIG_PARIDE_FRIQ=m
CONFIG_PARIDE_FRPW=m
CONFIG_PARIDE_KBIC=m
CONFIG_PARIDE_KTTI=m
CONFIG_PARIDE_ON20=m
CONFIG_PARIDE_ON26=m
CONFIG_BLK_DEV_PCIESSD_MTIP32XX=m
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
# CONFIG_DRBD_FAULT_INJECTION is not set
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_NVME=m
CONFIG_BLK_DEV_OSD=m
CONFIG_BLK_DEV_SX8=m
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=131072
CONFIG_BLK_DEV_XIP=y
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
CONFIG_CDROM_PKTCDVD_WCACHE=y
CONFIG_ATA_OVER_ETH=m
# CONFIG_BLK_DEV_HD is not set
CONFIG_BLK_DEV_RBD=m
# CONFIG_BLK_DEV_RSXX is not set

#
# Misc devices
#
CONFIG_SENSORS_LIS3LV02D=m
CONFIG_AD525X_DPOT=m
CONFIG_AD525X_DPOT_I2C=m
CONFIG_IBM_ASM=m
CONFIG_PHANTOM=m
# CONFIG_INTEL_MID_PTI is not set
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
CONFIG_ICS932S401=m
# CONFIG_ATMEL_SSC is not set
CONFIG_ENCLOSURE_SERVICES=m
CONFIG_CS5535_MFGPT=m
CONFIG_CS5535_MFGPT_DEFAULT_IRQ=7
CONFIG_CS5535_CLOCK_EVENT_SRC=m
CONFIG_HP_ILO=m
# CONFIG_APDS9802ALS is not set
# CONFIG_ISL29003 is not set
CONFIG_ISL29020=m
CONFIG_SENSORS_TSL2550=m
CONFIG_SENSORS_BH1780=m
CONFIG_SENSORS_BH1770=m
CONFIG_SENSORS_APDS990X=m
CONFIG_HMC6352=m
CONFIG_DS1682=m
CONFIG_VMWARE_BALLOON=m
CONFIG_BMP085=y
CONFIG_BMP085_I2C=m
CONFIG_PCH_PHUB=m
CONFIG_USB_SWITCH_FSA9480=m
CONFIG_C2PORT=m
CONFIG_C2PORT_DURAMAR_2150=m

#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
CONFIG_EEPROM_LEGACY=m
CONFIG_EEPROM_MAX6875=m
CONFIG_EEPROM_93CX6=m
CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y

#
# Texas Instruments shared transport line discipline
#
CONFIG_TI_ST=m
CONFIG_SENSORS_LIS3_I2C=m

#
# Altera FPGA firmware download module
#
CONFIG_ALTERA_STAPL=m
CONFIG_INTEL_MEI=m
CONFIG_INTEL_MEI_ME=y
# CONFIG_VMWARE_VMCI is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_FC_TGT_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=m
CONFIG_SCSI_SRP_TGT_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
CONFIG_SCSI_BNX2X_FCOE=m
CONFIG_BE2ISCSI=m
CONFIG_BLK_DEV_3W_XXXX_RAID=m
CONFIG_SCSI_HPSA=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_3W_SAS=m
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=5000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
CONFIG_AIC79XX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC94XX=m
# CONFIG_AIC94XX_DEBUG is not set
CONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
CONFIG_SCSI_MVSAS_TASKLET=y
CONFIG_SCSI_MVUMI=m
CONFIG_SCSI_DPT_I2O=m
CONFIG_SCSI_ADVANSYS=m
CONFIG_SCSI_ARCMSR=m
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=m
CONFIG_MEGARAID_MAILBOX=m
CONFIG_MEGARAID_LEGACY=m
CONFIG_MEGARAID_SAS=m
CONFIG_SCSI_MPT2SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
# CONFIG_SCSI_MPT2SAS_LOGGING is not set
CONFIG_SCSI_MPT3SAS=m
CONFIG_SCSI_MPT3SAS_MAX_SGE=128
# CONFIG_SCSI_MPT3SAS_LOGGING is not set
CONFIG_SCSI_UFSHCD=m
# CONFIG_SCSI_UFSHCD_PCI is not set
CONFIG_SCSI_HPTIOP=m
CONFIG_SCSI_BUSLOGIC=m
CONFIG_VMWARE_PVSCSI=m
CONFIG_HYPERV_STORAGE=m
CONFIG_LIBFC=m
CONFIG_LIBFCOE=m
CONFIG_FCOE=m
CONFIG_FCOE_FNIC=m
CONFIG_SCSI_DMX3191D=m
CONFIG_SCSI_EATA=m
CONFIG_SCSI_EATA_TAGGED_QUEUE=y
CONFIG_SCSI_EATA_LINKED_COMMANDS=y
CONFIG_SCSI_EATA_MAX_TAGS=16
CONFIG_SCSI_FUTURE_DOMAIN=m
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_ISCI=m
CONFIG_SCSI_IPS=m
CONFIG_SCSI_INITIO=m
CONFIG_SCSI_INIA100=m
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_EPP16 is not set
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
CONFIG_SCSI_STEX=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
CONFIG_SCSI_IPR=m
CONFIG_SCSI_IPR_TRACE=y
CONFIG_SCSI_IPR_DUMP=y
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA_FC=m
CONFIG_TCM_QLA2XXX=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not set
# CONFIG_SCSI_DC395x is not set
CONFIG_SCSI_DC390T=m
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_PMCRAID=m
CONFIG_SCSI_PM8001=m
CONFIG_SCSI_SRP=m
CONFIG_SCSI_BFA_FC=m
CONFIG_SCSI_CHELSIO_FCOE=m
CONFIG_SCSI_LOWLEVEL_PCMCIA=y
CONFIG_PCMCIA_AHA152X=m
CONFIG_PCMCIA_FDOMAIN=m
CONFIG_PCMCIA_QLOGIC=m
CONFIG_PCMCIA_SYM53C500=m
CONFIG_SCSI_DH=m
CONFIG_SCSI_DH_RDAC=m
CONFIG_SCSI_DH_HP_SW=m
CONFIG_SCSI_DH_EMC=m
# CONFIG_SCSI_DH_ALUA is not set
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
# CONFIG_SCSI_OSD_DEBUG is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
# CONFIG_SATA_ZPODD is not set
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=y
CONFIG_SATA_AHCI_PLATFORM=m
CONFIG_SATA_INIC162X=m
CONFIG_SATA_ACARD_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
# CONFIG_SATA_SX4 is not set
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=m
CONFIG_SATA_HIGHBANK=m
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m

#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=m
CONFIG_PATA_ARASAN_CF=m
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
CONFIG_PATA_ATP867X=m
CONFIG_PATA_CMD64X=m
CONFIG_PATA_CS5520=m
CONFIG_PATA_CS5530=m
CONFIG_PATA_CS5536=m
# CONFIG_PATA_CYPRESS is not set
CONFIG_PATA_EFAR=m
CONFIG_PATA_HPT366=m
CONFIG_PATA_HPT37X=m
CONFIG_PATA_HPT3X2N=m
CONFIG_PATA_HPT3X3=m
# CONFIG_PATA_HPT3X3_DMA is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
CONFIG_PATA_NINJA32=m
CONFIG_PATA_NS87415=m
CONFIG_PATA_OLDPIIX=m
# CONFIG_PATA_OPTIDMA is not set
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RDC=m
CONFIG_PATA_SC1200=m
CONFIG_PATA_SCH=m
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
# CONFIG_PATA_TOSHIBA is not set
CONFIG_PATA_TRIFLEX=m
CONFIG_PATA_VIA=m
CONFIG_PATA_WINBOND=m

#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
CONFIG_PATA_MPIIX=m
CONFIG_PATA_NS87410=m
# CONFIG_PATA_OPTI is not set
CONFIG_PATA_PCMCIA=m
CONFIG_PATA_RZ1000=m

#
# Generic fallback / legacy drivers
#
CONFIG_PATA_ACPI=m
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
# CONFIG_DM_THIN_PROVISIONING is not set
# CONFIG_DM_CACHE is not set
CONFIG_DM_MIRROR=m
CONFIG_DM_RAID=m
# CONFIG_DM_LOG_USERSPACE is not set
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
# CONFIG_DM_DELAY is not set
CONFIG_DM_UEVENT=y
# CONFIG_DM_FLAKEY is not set
# CONFIG_DM_VERITY is not set
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
CONFIG_TCM_PSCSI=m
CONFIG_LOOPBACK_TARGET=m
CONFIG_TCM_FC=m
CONFIG_ISCSI_TARGET=m
CONFIG_SBP_TARGET=m
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
CONFIG_FUSION_FC=m
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
# CONFIG_FUSION_LOGGING is not set

#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_SBP2=m
CONFIG_FIREWIRE_NET=m
CONFIG_FIREWIRE_NOSY=m
CONFIG_I2O=m
CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_EXT_ADAPTEC_DMA64=y
CONFIG_I2O_CONFIG=m
CONFIG_I2O_CONFIG_OLD_IOCTL=y
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
CONFIG_EQUALIZER=m
CONFIG_NET_FC=y
CONFIG_MII=y
CONFIG_IFB=m
# CONFIG_NET_TEAM is not set
# CONFIG_MACVLAN is not set
# CONFIG_VXLAN is not set
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_RIONET=m
CONFIG_RIONET_TX_SIZE=128
CONFIG_RIONET_RX_SIZE=128
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_SUNGEM_PHY=m
# CONFIG_ARCNET is not set
CONFIG_ATM_DRIVERS=y
CONFIG_ATM_DUMMY=m
CONFIG_ATM_TCP=m
CONFIG_ATM_LANAI=m
CONFIG_ATM_ENI=m
# CONFIG_ATM_ENI_DEBUG is not set
CONFIG_ATM_ENI_TUNE_BURST=y
CONFIG_ATM_ENI_BURST_TX_16W=y
CONFIG_ATM_ENI_BURST_TX_8W=y
CONFIG_ATM_ENI_BURST_TX_4W=y
CONFIG_ATM_ENI_BURST_TX_2W=y
CONFIG_ATM_ENI_BURST_RX_16W=y
CONFIG_ATM_ENI_BURST_RX_8W=y
CONFIG_ATM_ENI_BURST_RX_4W=y
CONFIG_ATM_ENI_BURST_RX_2W=y
CONFIG_ATM_FIRESTREAM=m
CONFIG_ATM_ZATM=m
# CONFIG_ATM_ZATM_DEBUG is not set
CONFIG_ATM_NICSTAR=m
CONFIG_ATM_NICSTAR_USE_SUNI=y
CONFIG_ATM_NICSTAR_USE_IDT77105=y
CONFIG_ATM_IDT77252=m
# CONFIG_ATM_IDT77252_DEBUG is not set
# CONFIG_ATM_IDT77252_RCV_ALL is not set
CONFIG_ATM_IDT77252_USE_SUNI=y
CONFIG_ATM_AMBASSADOR=m
# CONFIG_ATM_AMBASSADOR_DEBUG is not set
CONFIG_ATM_HORIZON=m
# CONFIG_ATM_HORIZON_DEBUG is not set
CONFIG_ATM_IA=m
# CONFIG_ATM_IA_DEBUG is not set
CONFIG_ATM_FORE200E=m
CONFIG_ATM_FORE200E_USE_TASKLET=y
CONFIG_ATM_FORE200E_TX_RETRY=16
CONFIG_ATM_FORE200E_DEBUG=0
CONFIG_ATM_HE=m
CONFIG_ATM_HE_USE_SUNI=y
CONFIG_ATM_SOLOS=m

#
# CAIF transport drivers
#
CONFIG_CAIF_TTY=m
CONFIG_CAIF_SPI_SLAVE=m
# CONFIG_CAIF_SPI_SYNC is not set
CONFIG_CAIF_HSI=m

#
# Distributed Switch Architecture drivers
#
CONFIG_NET_DSA_MV88E6XXX=m
CONFIG_NET_DSA_MV88E6060=m
CONFIG_NET_DSA_MV88E6XXX_NEED_PPU=y
CONFIG_NET_DSA_MV88E6131=m
CONFIG_NET_DSA_MV88E6123_61_65=m
CONFIG_ETHERNET=y
CONFIG_MDIO=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_3C589=m
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_NET_VENDOR_ADAPTEC=y
CONFIG_ADAPTEC_STARFIRE=m
CONFIG_NET_VENDOR_ALTEON=y
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_NET_VENDOR_AMD=y
CONFIG_AMD8111_ETH=m
CONFIG_PCNET32=m
CONFIG_PCMCIA_NMCLAN=m
CONFIG_NET_VENDOR_ATHEROS=y
CONFIG_ATL2=m
CONFIG_ATL1=m
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
CONFIG_NET_CADENCE=y
CONFIG_ARM_AT91_ETHER=m
CONFIG_MACB=m
CONFIG_NET_VENDOR_BROADCOM=y
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_TIGON3=m
CONFIG_BNX2X=m
CONFIG_BNX2X_SRIOV=y
CONFIG_NET_VENDOR_BROCADE=y
CONFIG_BNA=m
CONFIG_NET_CALXEDA_XGMAC=m
CONFIG_NET_VENDOR_CHELSIO=y
CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
CONFIG_CHELSIO_T4VF=m
CONFIG_NET_VENDOR_CISCO=y
CONFIG_ENIC=m
CONFIG_DNET=m
CONFIG_NET_VENDOR_DEC=y
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
# CONFIG_TULIP_MMIO is not set
CONFIG_TULIP_NAPI=y
CONFIG_TULIP_NAPI_HW_MITIGATION=y
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
CONFIG_NET_VENDOR_DLINK=y
CONFIG_DL2K=m
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
CONFIG_NET_VENDOR_EMULEX=y
CONFIG_BE2NET=m
CONFIG_NET_VENDOR_EXAR=y
CONFIG_S2IO=m
CONFIG_VXGE=m
# CONFIG_VXGE_DEBUG_TRACE_ALL is not set
CONFIG_NET_VENDOR_FUJITSU=y
CONFIG_PCMCIA_FMVJ18X=m
CONFIG_NET_VENDOR_HP=y
CONFIG_HP100=m
CONFIG_NET_VENDOR_INTEL=y
CONFIG_E100=m
CONFIG_E1000=m
CONFIG_E1000E=m
CONFIG_IGB=m
CONFIG_IGB_HWMON=y
CONFIG_IGB_DCA=y
CONFIG_IGBVF=m
CONFIG_IXGB=m
CONFIG_IXGBE=m
CONFIG_IXGBE_HWMON=y
CONFIG_IXGBE_DCA=y
CONFIG_IXGBE_DCB=y
CONFIG_IXGBEVF=m
CONFIG_NET_VENDOR_I825XX=y
# CONFIG_IP1000 is not set
CONFIG_JME=m
CONFIG_NET_VENDOR_MARVELL=y
# CONFIG_MVMDIO is not set
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKGE_GENESIS=y
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX4_EN=m
CONFIG_MLX4_EN_DCB=y
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_NET_VENDOR_MICREL=y
CONFIG_KS8842=m
CONFIG_KS8851_MLL=m
CONFIG_KSZ884X_PCI=m
CONFIG_NET_VENDOR_MYRI=y
CONFIG_MYRI10GE=m
CONFIG_MYRI10GE_DCA=y
CONFIG_FEALNX=m
CONFIG_NET_VENDOR_NATSEMI=y
CONFIG_NATSEMI=m
CONFIG_NS83820=m
CONFIG_NET_VENDOR_8390=y
CONFIG_PCMCIA_AXNET=m
CONFIG_NE2K_PCI=m
CONFIG_PCMCIA_PCNET=m
CONFIG_NET_VENDOR_NVIDIA=y
CONFIG_FORCEDETH=m
CONFIG_NET_VENDOR_OKI=y
CONFIG_PCH_GBE=m
CONFIG_ETHOC=m
CONFIG_NET_PACKET_ENGINE=y
CONFIG_HAMACHI=m
# CONFIG_YELLOWFIN is not set
CONFIG_NET_VENDOR_QLOGIC=y
CONFIG_QLA3XXX=m
CONFIG_QLCNIC=m
CONFIG_QLGE=m
CONFIG_NETXEN_NIC=m
CONFIG_NET_VENDOR_REALTEK=y
CONFIG_ATP=m
# CONFIG_8139CP is not set
CONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_R8169=m
CONFIG_NET_VENDOR_RDC=y
CONFIG_R6040=m
CONFIG_NET_VENDOR_SEEQ=y
CONFIG_NET_VENDOR_SILAN=y
# CONFIG_SC92031 is not set
CONFIG_NET_VENDOR_SIS=y
CONFIG_SIS900=m
CONFIG_SIS190=m
CONFIG_SFC=m
CONFIG_SFC_MTD=y
CONFIG_SFC_MCDI_MON=y
CONFIG_SFC_SRIOV=y
CONFIG_NET_VENDOR_SMSC=y
CONFIG_PCMCIA_SMC91C92=m
CONFIG_EPIC100=m
CONFIG_SMSC9420=m
CONFIG_NET_VENDOR_STMICRO=y
CONFIG_STMMAC_ETH=m
CONFIG_STMMAC_PLATFORM=y
# CONFIG_STMMAC_PCI is not set
# CONFIG_STMMAC_DEBUG_FS is not set
# CONFIG_STMMAC_DA is not set
CONFIG_STMMAC_RING=y
# CONFIG_STMMAC_CHAINED is not set
CONFIG_NET_VENDOR_SUN=y
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_CASSINI=m
CONFIG_NIU=m
CONFIG_NET_VENDOR_TEHUTI=y
CONFIG_TEHUTI=m
CONFIG_NET_VENDOR_TI=y
CONFIG_TLAN=m
CONFIG_NET_VENDOR_VIA=y
CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y
CONFIG_VIA_VELOCITY=m
CONFIG_NET_VENDOR_WIZNET=y
CONFIG_WIZNET_W5100=m
CONFIG_WIZNET_W5300=m
# CONFIG_WIZNET_BUS_DIRECT is not set
# CONFIG_WIZNET_BUS_INDIRECT is not set
CONFIG_WIZNET_BUS_ANY=y
CONFIG_NET_VENDOR_XIRCOM=y
CONFIG_PCMCIA_XIRC2PS=m
CONFIG_FDDI=m
CONFIG_DEFXX=m
CONFIG_DEFXX_MMIO=y
CONFIG_SKFP=m
# CONFIG_HIPPI is not set
# CONFIG_NET_SB1000 is not set
CONFIG_PHYLIB=y

#
# MII PHY device drivers
#
CONFIG_AT803X_PHY=m
CONFIG_AMD_PHY=m
CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
CONFIG_BCM87XX_PHY=m
CONFIG_ICPLUS_PHY=m
CONFIG_REALTEK_PHY=m
CONFIG_NATIONAL_PHY=m
CONFIG_STE10XP=m
CONFIG_LSI_ET1011C_PHY=m
CONFIG_MICREL_PHY=m
CONFIG_FIXED_PHY=y
CONFIG_MDIO_BITBANG=m
CONFIG_MDIO_GPIO=m
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
# CONFIG_PPP_MPPE is not set
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPPOATM=m
# CONFIG_PPPOE is not set
# CONFIG_PPTP is not set
# CONFIG_PPPOL2TP is not set
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_SLIP=m
CONFIG_SLHC=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLIP_SMART=y
CONFIG_SLIP_MODE_SLIP6=y

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
# CONFIG_USB_RTL8150 is not set
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_AX88179_178A=m
CONFIG_USB_NET_CDCETHER=m
# CONFIG_USB_NET_CDC_EEM is not set
CONFIG_USB_NET_CDC_NCM=m
CONFIG_USB_NET_CDC_MBIM=m
CONFIG_USB_NET_DM9601=m
CONFIG_USB_NET_SMSC75XX=m
CONFIG_USB_NET_SMSC95XX=m
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
# CONFIG_USB_NET_PLUSB is not set
CONFIG_USB_NET_MCS7830=m
# CONFIG_USB_NET_RNDIS_HOST is not set
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
# CONFIG_USB_KC2190 is not set
CONFIG_USB_NET_ZAURUS=m
CONFIG_USB_NET_CX82310_ETH=m
CONFIG_USB_NET_KALMIA=m
CONFIG_USB_NET_QMI_WWAN=m
CONFIG_USB_HSO=m
CONFIG_USB_NET_INT51X1=m
CONFIG_USB_CDC_PHONET=m
CONFIG_USB_IPHETH=m
CONFIG_USB_SIERRA_NET=m
CONFIG_USB_VL600=m
CONFIG_WLAN=y
CONFIG_PCMCIA_RAYCS=m
CONFIG_LIBERTAS_THINFIRM=m
# CONFIG_LIBERTAS_THINFIRM_DEBUG is not set
CONFIG_LIBERTAS_THINFIRM_USB=m
CONFIG_AIRO=m
CONFIG_ATMEL=m
CONFIG_PCI_ATMEL=m
CONFIG_PCMCIA_ATMEL=m
CONFIG_AT76C50X_USB=m
CONFIG_AIRO_CS=m
# CONFIG_PCMCIA_WL3501 is not set
# CONFIG_PRISM54 is not set
CONFIG_USB_ZD1201=m
# CONFIG_USB_NET_RNDIS_WLAN is not set
# CONFIG_RTL8180 is not set
CONFIG_RTL8187=m
CONFIG_RTL8187_LEDS=y
# CONFIG_ADM8211 is not set
CONFIG_MAC80211_HWSIM=m
# CONFIG_MWL8K is not set
CONFIG_ATH_COMMON=m
CONFIG_ATH_CARDS=m
# CONFIG_ATH_DEBUG is not set
CONFIG_ATH5K=m
# CONFIG_ATH5K_DEBUG is not set
# CONFIG_ATH5K_TRACER is not set
CONFIG_ATH5K_PCI=y
CONFIG_ATH9K_HW=m
CONFIG_ATH9K_COMMON=m
CONFIG_ATH9K_BTCOEX_SUPPORT=y
CONFIG_ATH9K=m
CONFIG_ATH9K_PCI=y
# CONFIG_ATH9K_AHB is not set
# CONFIG_ATH9K_DEBUGFS is not set
CONFIG_ATH9K_RATE_CONTROL=y
CONFIG_ATH9K_HTC=m
# CONFIG_ATH9K_HTC_DEBUGFS is not set
# CONFIG_CARL9170 is not set
CONFIG_ATH6KL=m
CONFIG_ATH6KL_SDIO=m
# CONFIG_ATH6KL_USB is not set
# CONFIG_ATH6KL_DEBUG is not set
CONFIG_AR5523=m
CONFIG_WIL6210=m
CONFIG_WIL6210_ISR_COR=y
CONFIG_B43=m
CONFIG_B43_BCMA=y
CONFIG_B43_BCMA_EXTRA=y
CONFIG_B43_SSB=y
CONFIG_B43_PCI_AUTOSELECT=y
CONFIG_B43_PCICORE_AUTOSELECT=y
CONFIG_B43_PCMCIA=y
# CONFIG_B43_SDIO is not set
CONFIG_B43_BCMA_PIO=y
CONFIG_B43_PIO=y
# CONFIG_B43_PHY_N is not set
CONFIG_B43_PHY_LP=y
# CONFIG_B43_PHY_HT is not set
CONFIG_B43_LEDS=y
CONFIG_B43_HWRNG=y
# CONFIG_B43_DEBUG is not set
CONFIG_B43LEGACY=m
CONFIG_B43LEGACY_PCI_AUTOSELECT=y
CONFIG_B43LEGACY_PCICORE_AUTOSELECT=y
CONFIG_B43LEGACY_LEDS=y
CONFIG_B43LEGACY_HWRNG=y
# CONFIG_B43LEGACY_DEBUG is not set
CONFIG_B43LEGACY_DMA=y
CONFIG_B43LEGACY_PIO=y
CONFIG_B43LEGACY_DMA_AND_PIO_MODE=y
# CONFIG_B43LEGACY_DMA_MODE is not set
# CONFIG_B43LEGACY_PIO_MODE is not set
CONFIG_BRCMUTIL=m
CONFIG_BRCMSMAC=m
# CONFIG_BRCMFMAC is not set
# CONFIG_BRCM_TRACING is not set
# CONFIG_BRCMDBG is not set
CONFIG_HOSTAP=m
CONFIG_HOSTAP_FIRMWARE=y
CONFIG_HOSTAP_FIRMWARE_NVRAM=y
CONFIG_HOSTAP_PLX=m
CONFIG_HOSTAP_PCI=m
CONFIG_HOSTAP_CS=m
CONFIG_IPW2100=m
CONFIG_IPW2100_MONITOR=y
CONFIG_IPW2100_DEBUG=y
CONFIG_IPW2200=m
CONFIG_IPW2200_MONITOR=y
CONFIG_IPW2200_RADIOTAP=y
CONFIG_IPW2200_PROMISCUOUS=y
# CONFIG_IPW2200_QOS is not set
CONFIG_IPW2200_DEBUG=y
CONFIG_LIBIPW=m
CONFIG_LIBIPW_DEBUG=y
CONFIG_IWLWIFI=m
CONFIG_IWLDVM=m
# CONFIG_IWLMVM is not set

#
# Debugging Options
#
CONFIG_IWLWIFI_DEBUG=y
CONFIG_IWLWIFI_DEBUGFS=y
# CONFIG_IWLWIFI_DEBUG_EXPERIMENTAL_UCODE is not set
# CONFIG_IWLWIFI_DEVICE_TRACING is not set
# CONFIG_IWLWIFI_P2P is not set
CONFIG_IWLEGACY=m
CONFIG_IWL4965=m
CONFIG_IWL3945=m

#
# iwl3945 / iwl4965 Debugging Options
#
# CONFIG_IWLEGACY_DEBUG is not set
# CONFIG_IWLEGACY_DEBUGFS is not set
CONFIG_LIBERTAS=m
CONFIG_LIBERTAS_USB=m
CONFIG_LIBERTAS_CS=m
CONFIG_LIBERTAS_SDIO=m
# CONFIG_LIBERTAS_DEBUG is not set
CONFIG_LIBERTAS_MESH=y
CONFIG_HERMES=m
# CONFIG_HERMES_PRISM is not set
CONFIG_HERMES_CACHE_FW_ON_INIT=y
CONFIG_PLX_HERMES=m
CONFIG_TMD_HERMES=m
CONFIG_NORTEL_HERMES=m
CONFIG_PCMCIA_HERMES=m
CONFIG_PCMCIA_SPECTRUM=m
CONFIG_ORINOCO_USB=m
# CONFIG_P54_COMMON is not set
CONFIG_RT2X00=m
CONFIG_RT2400PCI=m
CONFIG_RT2500PCI=m
CONFIG_RT61PCI=m
CONFIG_RT2800PCI=m
CONFIG_RT2800PCI_RT33XX=y
CONFIG_RT2800PCI_RT35XX=y
CONFIG_RT2800PCI_RT53XX=y
CONFIG_RT2800PCI_RT3290=y
CONFIG_RT2500USB=m
CONFIG_RT73USB=m
CONFIG_RT2800USB=m
CONFIG_RT2800USB_RT33XX=y
CONFIG_RT2800USB_RT35XX=y
# CONFIG_RT2800USB_RT53XX is not set
CONFIG_RT2800USB_UNKNOWN=y
CONFIG_RT2800_LIB=m
CONFIG_RT2X00_LIB_MMIO=m
CONFIG_RT2X00_LIB_PCI=m
CONFIG_RT2X00_LIB_USB=m
CONFIG_RT2X00_LIB=m
CONFIG_RT2X00_LIB_FIRMWARE=y
CONFIG_RT2X00_LIB_CRYPTO=y
CONFIG_RT2X00_LIB_LEDS=y
# CONFIG_RT2X00_LIB_DEBUGFS is not set
# CONFIG_RT2X00_DEBUG is not set
CONFIG_RTLWIFI=m
CONFIG_RTLWIFI_DEBUG=y
CONFIG_RTL8192CE=m
CONFIG_RTL8192SE=m
CONFIG_RTL8192DE=m
# CONFIG_RTL8723AE is not set
CONFIG_RTL8192CU=m
CONFIG_RTL8192C_COMMON=m
CONFIG_WL_TI=y
# CONFIG_WL1251 is not set
CONFIG_WL12XX=m
CONFIG_WL18XX=m
CONFIG_WLCORE=m
CONFIG_WLCORE_SDIO=m
CONFIG_WILINK_PLATFORM_DATA=y
# CONFIG_ZD1211RW is not set
CONFIG_MWIFIEX=m
CONFIG_MWIFIEX_SDIO=m
CONFIG_MWIFIEX_PCIE=m
CONFIG_MWIFIEX_USB=m

#
# WiMAX Wireless Broadband devices
#
CONFIG_WIMAX_I2400M=m
CONFIG_WIMAX_I2400M_USB=m
CONFIG_WIMAX_I2400M_DEBUG_LEVEL=8
CONFIG_WAN=y
CONFIG_LANMEDIA=m
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
CONFIG_HDLC_RAW_ETH=m
CONFIG_HDLC_CISCO=m
CONFIG_HDLC_FR=m
CONFIG_HDLC_PPP=m

#
# X.25/LAPB support is disabled
#
CONFIG_PCI200SYN=m
CONFIG_WANXL=m
CONFIG_PC300TOO=m
CONFIG_FARSYNC=m
CONFIG_DSCC4=m
CONFIG_DSCC4_PCISYNC=y
CONFIG_DSCC4_PCI_RST=y
CONFIG_DLCI=m
CONFIG_DLCI_MAX=8
CONFIG_SBNI=m
CONFIG_SBNI_MULTILINE=y
CONFIG_VMXNET3=m
CONFIG_HYPERV_NET=m
CONFIG_ISDN=y
CONFIG_ISDN_I4L=m
CONFIG_ISDN_PPP=y
CONFIG_ISDN_PPP_VJ=y
CONFIG_ISDN_MPP=y
CONFIG_IPPP_FILTER=y
CONFIG_ISDN_PPP_BSDCOMP=m
CONFIG_ISDN_AUDIO=y
CONFIG_ISDN_TTY_FAX=y

#
# ISDN feature submodules
#
CONFIG_ISDN_DIVERSION=m

#
# ISDN4Linux hardware drivers
#

#
# Passive cards
#
CONFIG_ISDN_DRV_HISAX=m

#
# D-channel protocol features
#
CONFIG_HISAX_EURO=y
CONFIG_DE_AOC=y
# CONFIG_HISAX_NO_SENDCOMPLETE is not set
# CONFIG_HISAX_NO_LLC is not set
# CONFIG_HISAX_NO_KEYPAD is not set
CONFIG_HISAX_1TR6=y
CONFIG_HISAX_NI1=y
CONFIG_HISAX_MAX_CARDS=8

#
# HiSax supported cards
#
CONFIG_HISAX_16_3=y
CONFIG_HISAX_TELESPCI=y
CONFIG_HISAX_S0BOX=y
CONFIG_HISAX_FRITZPCI=y
CONFIG_HISAX_AVM_A1_PCMCIA=y
CONFIG_HISAX_ELSA=y
CONFIG_HISAX_DIEHLDIVA=y
CONFIG_HISAX_SEDLBAUER=y
CONFIG_HISAX_NETJET=y
CONFIG_HISAX_NETJET_U=y
CONFIG_HISAX_NICCY=y
CONFIG_HISAX_BKM_A4T=y
CONFIG_HISAX_SCT_QUADRO=y
CONFIG_HISAX_GAZEL=y
CONFIG_HISAX_HFC_PCI=y
CONFIG_HISAX_W6692=y
CONFIG_HISAX_HFC_SX=y
CONFIG_HISAX_ENTERNOW_PCI=y
# CONFIG_HISAX_DEBUG is not set

#
# HiSax PCMCIA card service modules
#
CONFIG_HISAX_SEDLBAUER_CS=m
CONFIG_HISAX_ELSA_CS=m
CONFIG_HISAX_AVM_A1_CS=m
CONFIG_HISAX_TELES_CS=m

#
# HiSax sub driver modules
#
# CONFIG_HISAX_ST5481 is not set
# CONFIG_HISAX_HFCUSB is not set
# CONFIG_HISAX_HFC4S8S is not set
# CONFIG_HISAX_FRITZ_PCIPNP is not set

#
# Active cards
#
CONFIG_ISDN_CAPI=m
CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y
CONFIG_CAPI_TRACE=y
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI_CAPIDRV=m

#
# CAPI hardware drivers
#
CONFIG_CAPI_AVM=y
CONFIG_ISDN_DRV_AVMB1_B1PCI=m
CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y
CONFIG_ISDN_DRV_AVMB1_B1PCMCIA=m
CONFIG_ISDN_DRV_AVMB1_AVM_CS=m
CONFIG_ISDN_DRV_AVMB1_T1PCI=m
CONFIG_ISDN_DRV_AVMB1_C4=m
CONFIG_CAPI_EICON=y
CONFIG_ISDN_DIVAS=m
CONFIG_ISDN_DIVAS_BRIPCI=y
CONFIG_ISDN_DIVAS_PRIPCI=y
CONFIG_ISDN_DIVAS_DIVACAPI=m
CONFIG_ISDN_DIVAS_USERIDI=m
CONFIG_ISDN_DIVAS_MAINT=m
CONFIG_ISDN_DRV_GIGASET=m
CONFIG_GIGASET_CAPI=y
# CONFIG_GIGASET_I4L is not set
# CONFIG_GIGASET_DUMMYLL is not set
CONFIG_GIGASET_BASE=m
CONFIG_GIGASET_M105=m
CONFIG_GIGASET_M101=m
# CONFIG_GIGASET_DEBUG is not set
CONFIG_HYSDN=m
CONFIG_HYSDN_CAPI=y
CONFIG_MISDN=m
CONFIG_MISDN_DSP=m
CONFIG_MISDN_L1OIP=m

#
# mISDN hardware drivers
#
CONFIG_MISDN_HFCPCI=m
CONFIG_MISDN_HFCMULTI=m
CONFIG_MISDN_HFCUSB=m
CONFIG_MISDN_AVMFRITZ=m
CONFIG_MISDN_SPEEDFAX=m
CONFIG_MISDN_INFINEON=m
CONFIG_MISDN_W6692=m
CONFIG_MISDN_NETJET=m
CONFIG_MISDN_IPAC=m
CONFIG_MISDN_ISAR=m
CONFIG_ISDN_HDLC=m

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
CONFIG_INPUT_MATRIXKMAP=m

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ADP5588=m
CONFIG_KEYBOARD_ADP5589=m
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_QT1070=m
CONFIG_KEYBOARD_QT2160=m
# CONFIG_KEYBOARD_LKKBD is not set
CONFIG_KEYBOARD_GPIO=m
CONFIG_KEYBOARD_GPIO_POLLED=m
CONFIG_KEYBOARD_TCA6416=m
CONFIG_KEYBOARD_TCA8418=m
CONFIG_KEYBOARD_MATRIX=m
CONFIG_KEYBOARD_LM8323=m
CONFIG_KEYBOARD_LM8333=m
CONFIG_KEYBOARD_MAX7359=m
CONFIG_KEYBOARD_MCS=m
CONFIG_KEYBOARD_MPR121=m
CONFIG_KEYBOARD_NEWTON=m
CONFIG_KEYBOARD_OPENCORES=m
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_XTKBD=m
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_SENTELIC=y
CONFIG_MOUSE_PS2_TOUCHKIT=y
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_APPLETOUCH=m
CONFIG_MOUSE_BCM5974=m
# CONFIG_MOUSE_CYAPA is not set
CONFIG_MOUSE_VSXXXAA=m
CONFIG_MOUSE_GPIO=m
CONFIG_MOUSE_SYNAPTICS_I2C=m
CONFIG_MOUSE_SYNAPTICS_USB=m
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=m
CONFIG_JOYSTICK_ZHENHUA=m
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
CONFIG_JOYSTICK_AS5011=m
CONFIG_JOYSTICK_JOYDUMP=m
CONFIG_JOYSTICK_XPAD=m
CONFIG_JOYSTICK_XPAD_FF=y
CONFIG_JOYSTICK_XPAD_LEDS=y
CONFIG_JOYSTICK_WALKERA0701=m
CONFIG_INPUT_TABLET=y
CONFIG_TABLET_USB_ACECAD=m
CONFIG_TABLET_USB_AIPTEK=m
CONFIG_TABLET_USB_GTCO=m
CONFIG_TABLET_USB_HANWANG=m
CONFIG_TABLET_USB_KBTAB=m
CONFIG_TABLET_USB_WACOM=m
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_AD7879=m
CONFIG_TOUCHSCREEN_AD7879_I2C=m
CONFIG_TOUCHSCREEN_ATMEL_MXT=m
CONFIG_TOUCHSCREEN_AUO_PIXCIR=m
CONFIG_TOUCHSCREEN_BU21013=m
CONFIG_TOUCHSCREEN_CY8CTMG110=m
CONFIG_TOUCHSCREEN_CYTTSP_CORE=m
CONFIG_TOUCHSCREEN_CYTTSP_I2C=m
CONFIG_TOUCHSCREEN_DYNAPRO=m
CONFIG_TOUCHSCREEN_HAMPSHIRE=m
CONFIG_TOUCHSCREEN_EETI=m
CONFIG_TOUCHSCREEN_FUJITSU=m
CONFIG_TOUCHSCREEN_ILI210X=m
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_TOUCHSCREEN_ELO=m
CONFIG_TOUCHSCREEN_WACOM_W8001=m
CONFIG_TOUCHSCREEN_WACOM_I2C=m
CONFIG_TOUCHSCREEN_MAX11801=m
CONFIG_TOUCHSCREEN_MCS5000=m
CONFIG_TOUCHSCREEN_MMS114=m
CONFIG_TOUCHSCREEN_MTOUCH=m
CONFIG_TOUCHSCREEN_INEXIO=m
CONFIG_TOUCHSCREEN_MK712=m
CONFIG_TOUCHSCREEN_PENMOUNT=m
CONFIG_TOUCHSCREEN_EDT_FT5X06=m
CONFIG_TOUCHSCREEN_TOUCHRIGHT=m
CONFIG_TOUCHSCREEN_TOUCHWIN=m
# CONFIG_TOUCHSCREEN_TI_AM335X_TSC is not set
CONFIG_TOUCHSCREEN_PIXCIR=m
CONFIG_TOUCHSCREEN_WM97XX=m
CONFIG_TOUCHSCREEN_WM9705=y
CONFIG_TOUCHSCREEN_WM9712=y
CONFIG_TOUCHSCREEN_WM9713=y
CONFIG_TOUCHSCREEN_USB_COMPOSITE=m
CONFIG_TOUCHSCREEN_USB_EGALAX=y
CONFIG_TOUCHSCREEN_USB_PANJIT=y
CONFIG_TOUCHSCREEN_USB_3M=y
CONFIG_TOUCHSCREEN_USB_ITM=y
CONFIG_TOUCHSCREEN_USB_ETURBO=y
CONFIG_TOUCHSCREEN_USB_GUNZE=y
CONFIG_TOUCHSCREEN_USB_DMC_TSC10=y
CONFIG_TOUCHSCREEN_USB_IRTOUCH=y
CONFIG_TOUCHSCREEN_USB_IDEALTEK=y
CONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=y
CONFIG_TOUCHSCREEN_USB_GOTOP=y
CONFIG_TOUCHSCREEN_USB_JASTEC=y
CONFIG_TOUCHSCREEN_USB_ELO=y
CONFIG_TOUCHSCREEN_USB_E2I=y
CONFIG_TOUCHSCREEN_USB_ZYTRONIC=y
CONFIG_TOUCHSCREEN_USB_ETT_TC45USB=y
CONFIG_TOUCHSCREEN_USB_NEXIO=y
CONFIG_TOUCHSCREEN_USB_EASYTOUCH=y
CONFIG_TOUCHSCREEN_TOUCHIT213=m
CONFIG_TOUCHSCREEN_TSC_SERIO=m
CONFIG_TOUCHSCREEN_TSC2007=m
CONFIG_TOUCHSCREEN_ST1232=m
CONFIG_TOUCHSCREEN_TPS6507X=m
CONFIG_INPUT_MISC=y
CONFIG_INPUT_88PM80X_ONKEY=m
CONFIG_INPUT_AD714X=m
CONFIG_INPUT_AD714X_I2C=m
CONFIG_INPUT_BMA150=m
CONFIG_INPUT_PCSPKR=m
CONFIG_INPUT_MMA8450=m
CONFIG_INPUT_MPU3050=m
CONFIG_INPUT_APANEL=m
CONFIG_INPUT_GP2A=m
CONFIG_INPUT_GPIO_TILT_POLLED=m
CONFIG_INPUT_ATLAS_BTNS=m
CONFIG_INPUT_ATI_REMOTE2=m
CONFIG_INPUT_KEYSPAN_REMOTE=m
CONFIG_INPUT_KXTJ9=m
# CONFIG_INPUT_KXTJ9_POLLED_MODE is not set
CONFIG_INPUT_POWERMATE=m
CONFIG_INPUT_YEALINK=m
CONFIG_INPUT_CM109=m
CONFIG_INPUT_UINPUT=m
CONFIG_INPUT_PCF8574=m
CONFIG_INPUT_PWM_BEEPER=m
CONFIG_INPUT_GPIO_ROTARY_ENCODER=m
CONFIG_INPUT_ADXL34X=m
CONFIG_INPUT_ADXL34X_I2C=m
CONFIG_INPUT_CMA3000=m
CONFIG_INPUT_CMA3000_I2C=m

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PARKBD=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_SERIO_ALTERA_PS2=m
CONFIG_SERIO_PS2MULT=m
# CONFIG_SERIO_ARC_PS2 is not set
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=0
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_ROCKETPORT=m
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
CONFIG_MOXA_INTELLIO=m
CONFIG_MOXA_SMARTIO=m
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_NOZOMI=m
CONFIG_ISI=m
CONFIG_N_HDLC=m
CONFIG_N_GSM=m
CONFIG_TRACE_ROUTER=m
CONFIG_TRACE_SINK=m
CONFIG_DEVKMEM=y
CONFIG_STALDRV=y

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_DEPRECATED_OPTIONS=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_DMA=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=32
# CONFIG_SERIAL_8250_EXTENDED is not set
# CONFIG_SERIAL_8250_DW is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MFD_HSU is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_SCCNXP is not set
CONFIG_SERIAL_TIMBERDALE=m
CONFIG_SERIAL_ALTERA_JTAGUART=m
CONFIG_SERIAL_ALTERA_UART=m
CONFIG_SERIAL_ALTERA_UART_MAXPORTS=4
CONFIG_SERIAL_ALTERA_UART_BAUDRATE=115200
CONFIG_SERIAL_PCH_UART=m
# CONFIG_SERIAL_ARC is not set
# CONFIG_SERIAL_RP2 is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_PANIC_EVENT=y
# CONFIG_IPMI_PANIC_STRING is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_HW_RANDOM_TPM=m
CONFIG_NVRAM=y
CONFIG_R3964=m
CONFIG_APPLICOM=m

#
# PCMCIA character devices
#
CONFIG_SYNCLINK_CS=m
CONFIG_CARDMAN_4000=m
CONFIG_CARDMAN_4040=m
CONFIG_IPWIRELESS=m
CONFIG_MWAVE=m
CONFIG_RAW_DRIVER=m
CONFIG_MAX_RAW_DEVS=4096
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=m
CONFIG_TCG_TIS=m
CONFIG_TCG_TIS_I2C_INFINEON=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
# CONFIG_TCG_ST33_I2C is not set
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
# CONFIG_I2C_COMPAT is not set
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_MUX=m

#
# Multiplexer I2C Chip support
#
CONFIG_I2C_MUX_GPIO=m
# CONFIG_I2C_MUX_PCA9541 is not set
# CONFIG_I2C_MUX_PCA954x is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCA=m

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
CONFIG_I2C_ALI1535=m
# CONFIG_I2C_ALI1563 is not set
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
# CONFIG_I2C_AMD756_S4882 is not set
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
# CONFIG_I2C_ISMT is not set
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
# CONFIG_I2C_NFORCE2_S4985 is not set
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
# CONFIG_I2C_VIA is not set
CONFIG_I2C_VIAPRO=m

#
# ACPI drivers
#
CONFIG_I2C_SCMI=m

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
CONFIG_I2C_CBUS_GPIO=m
CONFIG_I2C_DESIGNWARE_CORE=m
CONFIG_I2C_DESIGNWARE_PCI=m
CONFIG_I2C_EG20T=m
CONFIG_I2C_GPIO=m
# CONFIG_I2C_INTEL_MID is not set
# CONFIG_I2C_OCORES is not set
CONFIG_I2C_PCA_PLATFORM=m
# CONFIG_I2C_PXA_PCI is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_DIOLAN_U2C=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_TAOS_EVM is not set
CONFIG_I2C_TINY_USB=m
CONFIG_I2C_VIPERBOARD=m

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_SPI is not set
CONFIG_HSI=m
CONFIG_HSI_BOARDINFO=y

#
# HSI clients
#
CONFIG_HSI_CHAR=m

#
# PPS support
#
CONFIG_PPS=m
# CONFIG_PPS_DEBUG is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m
CONFIG_PPS_CLIENT_PARPORT=m
CONFIG_PPS_CLIENT_GPIO=m

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=m

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
CONFIG_PTP_1588_CLOCK_PCH=m
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_GPIO_DEVRES=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_ACPI=y
# CONFIG_DEBUG_GPIO is not set
# CONFIG_GPIO_SYSFS is not set
CONFIG_GPIO_GENERIC=m
CONFIG_GPIO_MAX730X=m

#
# Memory mapped GPIO drivers:
#
CONFIG_GPIO_GENERIC_PLATFORM=m
CONFIG_GPIO_IT8761E=m
# CONFIG_GPIO_TS5500 is not set
# CONFIG_GPIO_SCH is not set
CONFIG_GPIO_ICH=m
CONFIG_GPIO_VX855=m
# CONFIG_GPIO_LYNXPOINT is not set

#
# I2C GPIO expanders:
#
CONFIG_GPIO_MAX7300=m
CONFIG_GPIO_MAX732X=m
CONFIG_GPIO_PCA953X=m
CONFIG_GPIO_PCF857X=m
# CONFIG_GPIO_SX150X is not set
CONFIG_GPIO_ADP5588=m

#
# PCI GPIO expanders:
#
CONFIG_GPIO_CS5535=m
CONFIG_GPIO_AMD8111=m
# CONFIG_GPIO_LANGWELL is not set
# CONFIG_GPIO_PCH is not set
CONFIG_GPIO_ML_IOH=m
# CONFIG_GPIO_RDC321X is not set

#
# SPI GPIO expanders:
#
CONFIG_GPIO_MCP23S08=m

#
# AC97 GPIO expanders:
#

#
# MODULbus GPIO expanders:
#

#
# USB GPIO expanders:
#
CONFIG_GPIO_VIPERBOARD=m
CONFIG_W1=m
CONFIG_W1_CON=y

#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=m
CONFIG_W1_MASTER_DS2490=m
CONFIG_W1_MASTER_DS2482=m
CONFIG_W1_MASTER_DS1WM=m
CONFIG_W1_MASTER_GPIO=m

#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
CONFIG_W1_SLAVE_SMEM=m
CONFIG_W1_SLAVE_DS2408=m
# CONFIG_W1_SLAVE_DS2413 is not set
CONFIG_W1_SLAVE_DS2423=m
CONFIG_W1_SLAVE_DS2431=m
CONFIG_W1_SLAVE_DS2433=m
CONFIG_W1_SLAVE_DS2433_CRC=y
CONFIG_W1_SLAVE_DS2760=m
CONFIG_W1_SLAVE_DS2780=m
CONFIG_W1_SLAVE_DS2781=m
CONFIG_W1_SLAVE_DS28E04=m
CONFIG_W1_SLAVE_BQ27000=m
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
CONFIG_PDA_POWER=m
# CONFIG_TEST_POWER is not set
CONFIG_BATTERY_DS2760=m
CONFIG_BATTERY_DS2780=m
CONFIG_BATTERY_DS2781=m
CONFIG_BATTERY_DS2782=m
CONFIG_BATTERY_SBS=m
CONFIG_BATTERY_BQ27x00=m
CONFIG_BATTERY_BQ27X00_I2C=y
CONFIG_BATTERY_BQ27X00_PLATFORM=y
CONFIG_BATTERY_MAX17040=m
CONFIG_BATTERY_MAX17042=m
CONFIG_CHARGER_ISP1704=m
CONFIG_CHARGER_MAX8903=m
CONFIG_CHARGER_LP8727=m
CONFIG_CHARGER_GPIO=m
# CONFIG_CHARGER_BQ2415X is not set
CONFIG_CHARGER_SMB347=m
# CONFIG_BATTERY_GOLDFISH is not set
CONFIG_POWER_RESET=y
CONFIG_POWER_AVS=y
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7410=m
CONFIG_SENSORS_ADT7411=m
CONFIG_SENSORS_ADT7462=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7475=m
CONFIG_SENSORS_ASC7621=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_FAM15H_POWER=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS620=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHMD=m
CONFIG_SENSORS_G760A=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_GPIO_FAN=m
CONFIG_SENSORS_HIH6130=m
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_JC42=m
CONFIG_SENSORS_LINEAGE=m
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM73=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_LTC4151=m
CONFIG_SENSORS_LTC4215=m
CONFIG_SENSORS_LTC4245=m
CONFIG_SENSORS_LTC4261=m
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_LM95245=m
CONFIG_SENSORS_MAX16065=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_MAX1668=m
CONFIG_SENSORS_MAX197=m
CONFIG_SENSORS_MAX6639=m
CONFIG_SENSORS_MAX6642=m
CONFIG_SENSORS_MAX6650=m
# CONFIG_SENSORS_MAX6697 is not set
CONFIG_SENSORS_MCP3021=m
CONFIG_SENSORS_NTC_THERMISTOR=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_PCF8591=m
CONFIG_PMBUS=m
CONFIG_SENSORS_PMBUS=m
CONFIG_SENSORS_ADM1275=m
CONFIG_SENSORS_LM25066=m
CONFIG_SENSORS_LTC2978=m
CONFIG_SENSORS_MAX16064=m
CONFIG_SENSORS_MAX34440=m
CONFIG_SENSORS_MAX8688=m
CONFIG_SENSORS_UCD9000=m
CONFIG_SENSORS_UCD9200=m
CONFIG_SENSORS_ZL6100=m
CONFIG_SENSORS_SHT15=m
CONFIG_SENSORS_SHT21=m
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_SMM665=m
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_EMC1403=m
CONFIG_SENSORS_EMC2103=m
CONFIG_SENSORS_EMC6W201=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_SCH56XX_COMMON=m
CONFIG_SENSORS_SCH5627=m
CONFIG_SENSORS_SCH5636=m
CONFIG_SENSORS_ADS1015=m
CONFIG_SENSORS_ADS7828=m
CONFIG_SENSORS_AMC6821=m
# CONFIG_SENSORS_INA209 is not set
CONFIG_SENSORS_INA2XX=m
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_TMP102=m
CONFIG_SENSORS_TMP401=m
CONFIG_SENSORS_TMP421=m
CONFIG_SENSORS_VIA_CPUTEMP=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83795=m
# CONFIG_SENSORS_W83795_FANCTRL is not set
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_APPLESMC=m

#
# ACPI drivers
#
CONFIG_SENSORS_ACPI_POWER=m
CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=m
CONFIG_THERMAL_HWMON=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set
# CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set
# CONFIG_THERMAL_GOV_FAIR_SHARE is not set
CONFIG_THERMAL_GOV_STEP_WISE=y
# CONFIG_THERMAL_GOV_USER_SPACE is not set
CONFIG_CPU_THERMAL=m
# CONFIG_THERMAL_EMULATION is not set
# CONFIG_INTEL_POWERCLAMP is not set
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=m
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_F71808E_WDT=m
CONFIG_SP5100_TCO=m
CONFIG_GEODE_WDT=m
CONFIG_SC520_WDT=m
CONFIG_SBC_FITPC2_WATCHDOG=m
CONFIG_EUROTECH_WDT=m
CONFIG_IB700_WDT=m
CONFIG_IBMASR=m
CONFIG_WAFER_WDT=m
CONFIG_I6300ESB_WDT=m
# CONFIG_IE6XX_WDT is not set
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
CONFIG_HP_WATCHDOG=m
CONFIG_HPWDT_NMI_DECODING=y
CONFIG_SC1200_WDT=m
CONFIG_PC87413_WDT=m
CONFIG_NV_TCO=m
CONFIG_60XX_WDT=m
CONFIG_SBC8360_WDT=m
CONFIG_CPU5_WDT=m
CONFIG_SMSC_SCH311X_WDT=m
CONFIG_SMSC37B787_WDT=m
CONFIG_VIA_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
CONFIG_W83697UG_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
CONFIG_SBC_EPX_C3_WATCHDOG=m

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_BLOCKIO=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
CONFIG_SSB_B43_PCI_BRIDGE=y
CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
CONFIG_SSB_PCMCIAHOST=y
CONFIG_SSB_SDIOHOST_POSSIBLE=y
CONFIG_SSB_SDIOHOST=y
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_SSB_DRIVER_GPIO=y
CONFIG_BCMA_POSSIBLE=y

#
# Broadcom specific AMBA
#
CONFIG_BCMA=m
CONFIG_BCMA_BLOCKIO=y
CONFIG_BCMA_HOST_PCI_POSSIBLE=y
CONFIG_BCMA_HOST_PCI=y
CONFIG_BCMA_DRIVER_GMAC_CMN=y
CONFIG_BCMA_DRIVER_GPIO=y
# CONFIG_BCMA_DEBUG is not set

#
# Multifunction device drivers
#
CONFIG_MFD_CORE=m
# CONFIG_MFD_88PM860X is not set
CONFIG_MFD_88PM800=m
CONFIG_MFD_88PM805=m
# CONFIG_MFD_SM501 is not set
CONFIG_MFD_RTSX_PCI=m
CONFIG_MFD_TI_AM335X_TSCADC=m
CONFIG_HTC_PASIC3=m
# CONFIG_HTC_I2CPLD is not set
# CONFIG_UCB1400_CORE is not set
CONFIG_MFD_LM3533=m
# CONFIG_TPS6105X is not set
CONFIG_TPS65010=m
CONFIG_TPS6507X=m
# CONFIG_MFD_TPS65217 is not set
# CONFIG_MFD_TPS6586X is not set
# CONFIG_MFD_TPS65910 is not set
# CONFIG_MFD_TPS65912_I2C is not set
# CONFIG_MFD_TPS80031 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_TWL6040_CORE is not set
# CONFIG_MFD_STMPE is not set
# CONFIG_MFD_TC3589X is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_MFD_SMSC is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_MFD_DA9055 is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_LP8788 is not set
# CONFIG_MFD_MAX77686 is not set
# CONFIG_MFD_MAX77693 is not set
CONFIG_MFD_MAX8907=m
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_ARIZONA_I2C is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_MFD_MC13XXX_I2C is not set
# CONFIG_ABX500_CORE is not set
CONFIG_MFD_CS5535=m
# CONFIG_MFD_TIMBERDALE is not set
CONFIG_LPC_SCH=m
CONFIG_LPC_ICH=m
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
CONFIG_MFD_VX855=m
CONFIG_MFD_WL1273_CORE=m
# CONFIG_MFD_TPS65090 is not set
# CONFIG_MFD_AAT2870_CORE is not set
# CONFIG_MFD_RC5T583 is not set
# CONFIG_MFD_PALMAS is not set
CONFIG_MFD_VIPERBOARD=m
# CONFIG_MFD_RETU is not set
# CONFIG_MFD_AS3711 is not set
# CONFIG_REGULATOR is not set
CONFIG_MEDIA_SUPPORT=m

#
# Multimedia core support
#
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_ANALOG_TV_SUPPORT=y
CONFIG_MEDIA_DIGITAL_TV_SUPPORT=y
CONFIG_MEDIA_RADIO_SUPPORT=y
CONFIG_MEDIA_RC_SUPPORT=y
# CONFIG_MEDIA_CONTROLLER is not set
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2=m
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_TUNER=m
CONFIG_V4L2_MEM2MEM_DEV=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m
CONFIG_VIDEOBUF_DMA_CONTIG=m
CONFIG_VIDEOBUF_DVB=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_DMA_CONTIG=m
CONFIG_VIDEOBUF2_VMALLOC=m
# CONFIG_VIDEO_V4L2_INT_DEVICE is not set
CONFIG_DVB_CORE=m
CONFIG_DVB_NET=y
CONFIG_TTPCI_EEPROM=m
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y

#
# Media drivers
#
CONFIG_RC_CORE=m
CONFIG_RC_MAP=m
CONFIG_RC_DECODERS=y
CONFIG_LIRC=m
CONFIG_IR_LIRC_CODEC=m
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
CONFIG_IR_JVC_DECODER=m
CONFIG_IR_SONY_DECODER=m
CONFIG_IR_RC5_SZ_DECODER=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_IR_MCE_KBD_DECODER=m
CONFIG_RC_DEVICES=y
CONFIG_RC_ATI_REMOTE=m
CONFIG_IR_ENE=m
CONFIG_IR_IMON=m
CONFIG_IR_MCEUSB=m
CONFIG_IR_ITE_CIR=m
CONFIG_IR_FINTEK=m
CONFIG_IR_NUVOTON=m
CONFIG_IR_REDRAT3=m
CONFIG_IR_STREAMZAP=m
CONFIG_IR_WINBOND_CIR=m
CONFIG_IR_IGUANA=m
CONFIG_IR_TTUSBIR=m
CONFIG_RC_LOOPBACK=m
CONFIG_IR_GPIO_CIR=m
CONFIG_MEDIA_USB_SUPPORT=y

#
# Webcam devices
#
CONFIG_USB_VIDEO_CLASS=m
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_USB_GSPCA=m
CONFIG_USB_M5602=m
CONFIG_USB_STV06XX=m
CONFIG_USB_GL860=m
CONFIG_USB_GSPCA_BENQ=m
CONFIG_USB_GSPCA_CONEX=m
CONFIG_USB_GSPCA_CPIA1=m
CONFIG_USB_GSPCA_ETOMS=m
CONFIG_USB_GSPCA_FINEPIX=m
CONFIG_USB_GSPCA_JEILINJ=m
CONFIG_USB_GSPCA_JL2005BCD=m
CONFIG_USB_GSPCA_KINECT=m
CONFIG_USB_GSPCA_KONICA=m
CONFIG_USB_GSPCA_MARS=m
CONFIG_USB_GSPCA_MR97310A=m
CONFIG_USB_GSPCA_NW80X=m
CONFIG_USB_GSPCA_OV519=m
CONFIG_USB_GSPCA_OV534=m
CONFIG_USB_GSPCA_OV534_9=m
CONFIG_USB_GSPCA_PAC207=m
CONFIG_USB_GSPCA_PAC7302=m
CONFIG_USB_GSPCA_PAC7311=m
CONFIG_USB_GSPCA_SE401=m
CONFIG_USB_GSPCA_SN9C2028=m
CONFIG_USB_GSPCA_SN9C20X=m
CONFIG_USB_GSPCA_SONIXB=m
CONFIG_USB_GSPCA_SONIXJ=m
CONFIG_USB_GSPCA_SPCA500=m
CONFIG_USB_GSPCA_SPCA501=m
CONFIG_USB_GSPCA_SPCA505=m
CONFIG_USB_GSPCA_SPCA506=m
CONFIG_USB_GSPCA_SPCA508=m
CONFIG_USB_GSPCA_SPCA561=m
CONFIG_USB_GSPCA_SPCA1528=m
CONFIG_USB_GSPCA_SQ905=m
CONFIG_USB_GSPCA_SQ905C=m
CONFIG_USB_GSPCA_SQ930X=m
CONFIG_USB_GSPCA_STK014=m
CONFIG_USB_GSPCA_STV0680=m
CONFIG_USB_GSPCA_SUNPLUS=m
CONFIG_USB_GSPCA_T613=m
CONFIG_USB_GSPCA_TOPRO=m
CONFIG_USB_GSPCA_TV8532=m
CONFIG_USB_GSPCA_VC032X=m
CONFIG_USB_GSPCA_VICAM=m
CONFIG_USB_GSPCA_XIRLINK_CIT=m
CONFIG_USB_GSPCA_ZC3XX=m
CONFIG_USB_PWC=m
# CONFIG_USB_PWC_DEBUG is not set
CONFIG_USB_PWC_INPUT_EVDEV=y
CONFIG_VIDEO_CPIA2=m
CONFIG_USB_ZR364XX=m
CONFIG_USB_STKWEBCAM=m
CONFIG_USB_S2255=m
CONFIG_USB_SN9C102=m

#
# Analog TV USB devices
#
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
# CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
CONFIG_VIDEO_HDPVR=m
CONFIG_VIDEO_TLG2300=m
CONFIG_VIDEO_USBVISION=m
CONFIG_VIDEO_STK1160=m
CONFIG_VIDEO_STK1160_AC97=y

#
# Analog/digital TV USB devices
#
CONFIG_VIDEO_AU0828=m
CONFIG_VIDEO_AU0828_V4L2=y
CONFIG_VIDEO_CX231XX=m
CONFIG_VIDEO_CX231XX_RC=y
CONFIG_VIDEO_CX231XX_ALSA=m
CONFIG_VIDEO_CX231XX_DVB=m
CONFIG_VIDEO_TM6000=m
CONFIG_VIDEO_TM6000_ALSA=m
CONFIG_VIDEO_TM6000_DVB=m

#
# Digital TV USB devices
#
CONFIG_DVB_USB=m
# CONFIG_DVB_USB_DEBUG is not set
CONFIG_DVB_USB_A800=m
CONFIG_DVB_USB_DIBUSB_MB=m
# CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set
CONFIG_DVB_USB_DIBUSB_MC=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_UMT_010=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_M920X=m
CONFIG_DVB_USB_DIGITV=m
CONFIG_DVB_USB_VP7045=m
CONFIG_DVB_USB_VP702X=m
CONFIG_DVB_USB_GP8PSK=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_TTUSB2=m
CONFIG_DVB_USB_DTT200U=m
CONFIG_DVB_USB_OPERA1=m
CONFIG_DVB_USB_AF9005=m
CONFIG_DVB_USB_AF9005_REMOTE=m
CONFIG_DVB_USB_PCTV452E=m
CONFIG_DVB_USB_DW2102=m
CONFIG_DVB_USB_CINERGY_T2=m
CONFIG_DVB_USB_DTV5100=m
CONFIG_DVB_USB_FRIIO=m
CONFIG_DVB_USB_AZ6027=m
CONFIG_DVB_USB_TECHNISAT_USB2=m
CONFIG_DVB_USB_V2=m
CONFIG_DVB_USB_CYPRESS_FIRMWARE=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_USB_AF9035=m
CONFIG_DVB_USB_ANYSEE=m
CONFIG_DVB_USB_AU6610=m
CONFIG_DVB_USB_AZ6007=m
CONFIG_DVB_USB_CE6230=m
CONFIG_DVB_USB_EC168=m
CONFIG_DVB_USB_GL861=m
CONFIG_DVB_USB_IT913X=m
CONFIG_DVB_USB_LME2510=m
CONFIG_DVB_USB_MXL111SF=m
# CONFIG_DVB_USB_RTL28XXU is not set
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_SMS_USB_DRV=m
CONFIG_DVB_B2C2_FLEXCOP_USB=m
# CONFIG_DVB_B2C2_FLEXCOP_USB_DEBUG is not set

#
# Webcam, TV (analog/digital) USB devices
#
CONFIG_VIDEO_EM28XX=m
CONFIG_VIDEO_EM28XX_ALSA=m
CONFIG_VIDEO_EM28XX_DVB=m
CONFIG_VIDEO_EM28XX_RC=m
CONFIG_MEDIA_PCI_SUPPORT=y

#
# Media capture support
#
CONFIG_VIDEO_MEYE=m

#
# Media capture/analog TV support
#
CONFIG_VIDEO_IVTV=m
CONFIG_VIDEO_IVTV_ALSA=m
CONFIG_VIDEO_FB_IVTV=m
CONFIG_VIDEO_ZORAN=m
CONFIG_VIDEO_ZORAN_DC30=m
CONFIG_VIDEO_ZORAN_ZR36060=m
CONFIG_VIDEO_ZORAN_BUZ=m
CONFIG_VIDEO_ZORAN_DC10=m
CONFIG_VIDEO_ZORAN_LML33=m
CONFIG_VIDEO_ZORAN_LML33R10=m
CONFIG_VIDEO_ZORAN_AVS6EYES=m
CONFIG_VIDEO_HEXIUM_GEMINI=m
CONFIG_VIDEO_HEXIUM_ORION=m
CONFIG_VIDEO_MXB=m

#
# Media capture/analog/hybrid TV support
#
CONFIG_VIDEO_CX18=m
CONFIG_VIDEO_CX18_ALSA=m
CONFIG_VIDEO_CX23885=m
CONFIG_MEDIA_ALTERA_CI=m
CONFIG_VIDEO_CX25821=m
# CONFIG_VIDEO_CX25821_ALSA is not set
CONFIG_VIDEO_CX88=m
CONFIG_VIDEO_CX88_ALSA=m
CONFIG_VIDEO_CX88_BLACKBIRD=m
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_VP3054=m
CONFIG_VIDEO_CX88_MPEG=m
CONFIG_VIDEO_BT848=m
CONFIG_DVB_BT8XX=m
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_VIDEO_SAA7134_DVB=m
CONFIG_VIDEO_SAA7164=m

#
# Media digital TV PCI Adapters
#
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET_CORE=m
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m
CONFIG_DVB_B2C2_FLEXCOP_PCI=m
# CONFIG_DVB_B2C2_FLEXCOP_PCI_DEBUG is not set
CONFIG_DVB_PLUTO2=m
CONFIG_DVB_DM1105=m
CONFIG_DVB_PT1=m
CONFIG_MANTIS_CORE=m
CONFIG_DVB_MANTIS=m
CONFIG_DVB_HOPPER=m
CONFIG_DVB_NGENE=m
CONFIG_DVB_DDBRIDGE=m
CONFIG_V4L_PLATFORM_DRIVERS=y
CONFIG_VIDEO_CAFE_CCIC=m
CONFIG_VIDEO_VIA_CAMERA=m
CONFIG_VIDEO_TIMBERDALE=m
CONFIG_SOC_CAMERA=m
CONFIG_SOC_CAMERA_PLATFORM=m
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_MEM2MEM_DEINTERLACE=m
# CONFIG_VIDEO_SH_VEU is not set
CONFIG_V4L_TEST_DRIVERS=y
CONFIG_VIDEO_VIVI=m
# CONFIG_VIDEO_MEM2MEM_TESTDEV is not set

#
# Supported MMC/SDIO adapters
#
CONFIG_SMS_SDIO_DRV=m
CONFIG_MEDIA_PARPORT_SUPPORT=y
CONFIG_VIDEO_BWQCAM=m
CONFIG_VIDEO_CQCAM=m
CONFIG_VIDEO_W9966=m
CONFIG_RADIO_ADAPTERS=y
CONFIG_RADIO_SI470X=y
CONFIG_USB_SI470X=m
CONFIG_I2C_SI470X=m
CONFIG_USB_MR800=m
CONFIG_USB_DSBR=m
CONFIG_RADIO_MAXIRADIO=m
CONFIG_RADIO_SHARK=m
CONFIG_RADIO_SHARK2=m
CONFIG_I2C_SI4713=m
CONFIG_RADIO_SI4713=m
CONFIG_USB_KEENE=m
# CONFIG_USB_MA901 is not set
CONFIG_RADIO_TEA5764=m
CONFIG_RADIO_SAA7706H=m
CONFIG_RADIO_TEF6862=m
CONFIG_RADIO_WL1273=m

#
# Texas Instruments WL128x FM driver (ST based)
#
CONFIG_RADIO_WL128X=m

#
# Supported FireWire (IEEE 1394) Adapters
#
CONFIG_DVB_FIREDTV=m
CONFIG_DVB_FIREDTV_INPUT=y
CONFIG_MEDIA_COMMON_OPTIONS=y

#
# common driver options
#
CONFIG_VIDEO_CX2341X=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_DVB_B2C2_FLEXCOP=m
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_SMS_SIANO_MDTV=m
CONFIG_SMS_SIANO_RC=y

#
# Media ancillary drivers (tuners, sensors, i2c, frontends)
#
CONFIG_MEDIA_SUBDRV_AUTOSELECT=y
CONFIG_VIDEO_IR_I2C=m

#
# Audio decoders, processors and mixers
#
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_TDA9840=m
CONFIG_VIDEO_TEA6415C=m
CONFIG_VIDEO_TEA6420=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS5345=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_WM8739=m
CONFIG_VIDEO_VP27SMPX=m

#
# RDS decoders
#
CONFIG_VIDEO_SAA6588=m

#
# Video decoders
#
CONFIG_VIDEO_ADV7180=m
CONFIG_VIDEO_BT819=m
CONFIG_VIDEO_BT856=m
CONFIG_VIDEO_BT866=m
CONFIG_VIDEO_KS0127=m
CONFIG_VIDEO_SAA7110=m
CONFIG_VIDEO_SAA711X=m
CONFIG_VIDEO_TVP5150=m
CONFIG_VIDEO_VPX3220=m

#
# Video and audio decoders
#
CONFIG_VIDEO_SAA717X=m
CONFIG_VIDEO_CX25840=m

#
# Video encoders
#
CONFIG_VIDEO_SAA7127=m
CONFIG_VIDEO_SAA7185=m
CONFIG_VIDEO_ADV7170=m
CONFIG_VIDEO_ADV7175=m

#
# Camera sensor devices
#
CONFIG_VIDEO_OV7670=m
CONFIG_VIDEO_MT9V011=m

#
# Flash devices
#

#
# Video improvement chips
#
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m

#
# Miscelaneous helper chips
#
CONFIG_VIDEO_M52790=m

#
# Sensors used on soc_camera driver
#

#
# soc_camera sensor drivers
#
CONFIG_SOC_CAMERA_IMX074=m
CONFIG_SOC_CAMERA_MT9M001=m
CONFIG_SOC_CAMERA_MT9M111=m
CONFIG_SOC_CAMERA_MT9T031=m
CONFIG_SOC_CAMERA_MT9T112=m
CONFIG_SOC_CAMERA_MT9V022=m
CONFIG_SOC_CAMERA_OV2640=m
CONFIG_SOC_CAMERA_OV5642=m
CONFIG_SOC_CAMERA_OV6650=m
CONFIG_SOC_CAMERA_OV772X=m
CONFIG_SOC_CAMERA_OV9640=m
CONFIG_SOC_CAMERA_OV9740=m
CONFIG_SOC_CAMERA_RJ54N1=m
CONFIG_SOC_CAMERA_TW9910=m
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA827X=m
CONFIG_MEDIA_TUNER_TDA18271=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2063=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_MT2131=m
CONFIG_MEDIA_TUNER_QT1010=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_MEDIA_TUNER_XC4000=m
CONFIG_MEDIA_TUNER_MXL5005S=m
CONFIG_MEDIA_TUNER_MXL5007T=m
CONFIG_MEDIA_TUNER_MC44S803=m
CONFIG_MEDIA_TUNER_MAX2165=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_MEDIA_TUNER_FC0011=m
CONFIG_MEDIA_TUNER_TDA18212=m
CONFIG_MEDIA_TUNER_FC2580=m
CONFIG_MEDIA_TUNER_TUA9001=m

#
# Multistandard (satellite) frontends
#
CONFIG_DVB_STB0899=m
CONFIG_DVB_STB6100=m
CONFIG_DVB_STV090x=m
CONFIG_DVB_STV6110x=m

#
# Multistandard (cable + terrestrial) frontends
#
CONFIG_DVB_DRXK=m
CONFIG_DVB_TDA18271C2DD=m

#
# DVB-S (satellite) frontends
#
CONFIG_DVB_CX24110=m
CONFIG_DVB_CX24123=m
CONFIG_DVB_MT312=m
CONFIG_DVB_ZL10036=m
CONFIG_DVB_ZL10039=m
CONFIG_DVB_S5H1420=m
CONFIG_DVB_STV0288=m
CONFIG_DVB_STB6000=m
CONFIG_DVB_STV0299=m
CONFIG_DVB_STV6110=m
CONFIG_DVB_STV0900=m
CONFIG_DVB_TDA8083=m
CONFIG_DVB_TDA10086=m
CONFIG_DVB_TDA8261=m
CONFIG_DVB_VES1X93=m
CONFIG_DVB_TUNER_ITD1000=m
CONFIG_DVB_TUNER_CX24113=m
CONFIG_DVB_TDA826X=m
CONFIG_DVB_TUA6100=m
CONFIG_DVB_CX24116=m
CONFIG_DVB_SI21XX=m
CONFIG_DVB_TS2020=m
CONFIG_DVB_DS3000=m
CONFIG_DVB_MB86A16=m
CONFIG_DVB_TDA10071=m

#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_SP8870=m
CONFIG_DVB_SP887X=m
CONFIG_DVB_CX22700=m
CONFIG_DVB_CX22702=m
CONFIG_DVB_DRXD=m
CONFIG_DVB_L64781=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_MT352=m
CONFIG_DVB_ZL10353=m
CONFIG_DVB_DIB3000MB=m
CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_TDA10048=m
CONFIG_DVB_AF9013=m
CONFIG_DVB_EC100=m
CONFIG_DVB_STV0367=m
CONFIG_DVB_CXD2820R=m

#
# DVB-C (cable) frontends
#
CONFIG_DVB_VES1820=m
CONFIG_DVB_TDA10021=m
CONFIG_DVB_TDA10023=m
CONFIG_DVB_STV0297=m

#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_NXT200X=m
CONFIG_DVB_OR51211=m
CONFIG_DVB_OR51132=m
CONFIG_DVB_BCM3510=m
CONFIG_DVB_LGDT330X=m
CONFIG_DVB_LGDT3305=m
CONFIG_DVB_LG2160=m
CONFIG_DVB_S5H1409=m
CONFIG_DVB_AU8522=m
CONFIG_DVB_AU8522_DTV=m
CONFIG_DVB_AU8522_V4L=m
CONFIG_DVB_S5H1411=m

#
# ISDB-T (terrestrial) frontends
#
CONFIG_DVB_S921=m
CONFIG_DVB_DIB8000=m
CONFIG_DVB_MB86A20S=m

#
# Digital terrestrial only tuners/PLL
#
CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m
CONFIG_DVB_TUNER_DIB0090=m

#
# SEC control devices for DVB-S
#
CONFIG_DVB_LNBP21=m
CONFIG_DVB_LNBP22=m
CONFIG_DVB_ISL6405=m
CONFIG_DVB_ISL6421=m
CONFIG_DVB_ISL6423=m
CONFIG_DVB_A8293=m
CONFIG_DVB_LGS8GXX=m
CONFIG_DVB_ATBM8830=m
CONFIG_DVB_TDA665x=m
CONFIG_DVB_IX2505V=m
CONFIG_DVB_IT913X_FE=m
CONFIG_DVB_M88RS2000=m
CONFIG_DVB_AF9033=m

#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_VGA_SWITCHEROO=y
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_LOAD_EDID_FIRMWARE=y
CONFIG_DRM_TTM=m

#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=m
CONFIG_DRM_I2C_SIL164=m
# CONFIG_DRM_I2C_NXP_TDA998X is not set
CONFIG_DRM_TDFX=m
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
# CONFIG_DRM_RADEON_UMS is not set
CONFIG_DRM_NOUVEAU=m
CONFIG_NOUVEAU_DEBUG=5
CONFIG_NOUVEAU_DEBUG_DEFAULT=3
CONFIG_DRM_NOUVEAU_BACKLIGHT=y
CONFIG_DRM_I810=m
CONFIG_DRM_I915=m
CONFIG_DRM_I915_KMS=y
CONFIG_DRM_MGA=m
CONFIG_DRM_SIS=m
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
CONFIG_DRM_VMWGFX=m
CONFIG_DRM_VMWGFX_FBCON=y
# CONFIG_DRM_GMA500 is not set
# CONFIG_DRM_UDL is not set
# CONFIG_DRM_AST is not set
# CONFIG_DRM_MGAG200 is not set
# CONFIG_DRM_CIRRUS_QEMU is not set
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=y
CONFIG_HDMI=y
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_DDC=m
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=m
# CONFIG_FB_WMT_GE_ROPS is not set
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_HECUBA=m
CONFIG_FB_SVGALIB=m
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
CONFIG_FB_CIRRUS=m
CONFIG_FB_PM2=m
CONFIG_FB_PM2_FIFO_DISCONNECT=y
CONFIG_FB_CYBER2000=m
CONFIG_FB_CYBER2000_DDC=y
CONFIG_FB_ARC=m
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
CONFIG_FB_UVESA=m
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_FB_N411=m
CONFIG_FB_HGA=m
CONFIG_FB_S1D13XXX=m
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
CONFIG_FB_NVIDIA_BACKLIGHT=y
CONFIG_FB_RIVA=m
CONFIG_FB_RIVA_I2C=y
# CONFIG_FB_RIVA_DEBUG is not set
CONFIG_FB_RIVA_BACKLIGHT=y
# CONFIG_FB_I740 is not set
CONFIG_FB_LE80578=m
CONFIG_FB_CARILLO_RANCH=m
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_RADEON_BACKLIGHT=y
# CONFIG_FB_RADEON_DEBUG is not set
CONFIG_FB_ATY128=m
CONFIG_FB_ATY128_BACKLIGHT=y
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
CONFIG_FB_ATY_GENERIC_LCD=y
CONFIG_FB_ATY_GX=y
CONFIG_FB_ATY_BACKLIGHT=y
CONFIG_FB_S3=m
CONFIG_FB_S3_DDC=y
# CONFIG_FB_SAVAGE is not set
CONFIG_FB_SIS=m
CONFIG_FB_SIS_300=y
CONFIG_FB_SIS_315=y
CONFIG_FB_VIA=m
# CONFIG_FB_VIA_DIRECT_PROCFS is not set
CONFIG_FB_VIA_X_COMPATIBILITY=y
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
# CONFIG_FB_3DFX_ACCEL is not set
CONFIG_FB_3DFX_I2C=y
CONFIG_FB_VOODOO1=m
CONFIG_FB_VT8623=m
CONFIG_FB_TRIDENT=m
CONFIG_FB_ARK=m
# CONFIG_FB_PM3 is not set
CONFIG_FB_CARMINE=m
CONFIG_FB_CARMINE_DRAM_EVAL=y
# CONFIG_CARMINE_DRAM_CUSTOM is not set
# CONFIG_FB_GEODE is not set
CONFIG_FB_TMIO=m
CONFIG_FB_TMIO_ACCELL=y
CONFIG_FB_SMSCUFX=m
CONFIG_FB_UDL=m
# CONFIG_FB_GOLDFISH is not set
CONFIG_FB_VIRTUAL=m
CONFIG_FB_METRONOME=m
CONFIG_FB_MB862XX=m
CONFIG_FB_MB862XX_PCI_GDC=y
CONFIG_FB_MB862XX_I2C=y
CONFIG_FB_BROADSHEET=m
CONFIG_FB_AUO_K190X=m
CONFIG_FB_AUO_K1900=m
CONFIG_FB_AUO_K1901=m
# CONFIG_EXYNOS_VIDEO is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_PLATFORM=m
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=m
CONFIG_BACKLIGHT_LM3533=m
CONFIG_BACKLIGHT_CARILLO_RANCH=m
CONFIG_BACKLIGHT_PWM=m
CONFIG_BACKLIGHT_APPLE=m
CONFIG_BACKLIGHT_SAHARA=m
CONFIG_BACKLIGHT_ADP8860=m
CONFIG_BACKLIGHT_ADP8870=m
CONFIG_BACKLIGHT_LM3630=m
CONFIG_BACKLIGHT_LM3639=m
CONFIG_BACKLIGHT_LP855X=m
CONFIG_BACKLIGHT_OT200=m

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_LOGO is not set
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
# CONFIG_SOUND_OSS_CORE_PRECLAIM is not set
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
CONFIG_SND_VERBOSE_PRINTK=y
CONFIG_SND_DEBUG=y
# CONFIG_SND_DEBUG_VERBOSE is not set
CONFIG_SND_PCM_XRUN_DEBUG=y
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_OPL3_LIB_SEQ=m
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
CONFIG_SND_DUMMY=m
CONFIG_SND_ALOOP=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
CONFIG_SND_MTS64=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_PORTMAN2X4=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_SB_COMMON=m
CONFIG_SND_SB16_DSP=m
CONFIG_SND_TEA575X=m
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=m
CONFIG_SND_ALS300=m
CONFIG_SND_ALS4000=m
CONFIG_SND_ALI5451=m
CONFIG_SND_ASIHPI=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
CONFIG_SND_AW2=m
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_OXYGEN_LIB=m
CONFIG_SND_OXYGEN=m
CONFIG_SND_CS4281=m
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CS5530=m
CONFIG_SND_CS5535AUDIO=m
CONFIG_SND_CTXFI=m
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
CONFIG_SND_INDIGOIOX=m
CONFIG_SND_INDIGODJX=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_ES1968_INPUT=y
CONFIG_SND_ES1968_RADIO=y
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X_BOOL=y
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_PREALLOC_SIZE=1024
CONFIG_SND_HDA_HWDEP=y
CONFIG_SND_HDA_RECONFIG=y
CONFIG_SND_HDA_INPUT_BEEP=y
CONFIG_SND_HDA_INPUT_BEEP_MODE=1
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_PATCH_LOADER=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CA0132=y
# CONFIG_SND_HDA_CODEC_CA0132_DSP is not set
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
CONFIG_SND_LOLA=m
CONFIG_SND_LX6464ES=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_MAESTRO3_INPUT=y
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_PCXHR=m
CONFIG_SND_RIPTIDE=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
CONFIG_SND_VIRTUOSO=m
CONFIG_SND_VX222=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
CONFIG_SND_USB_UA101=m
CONFIG_SND_USB_USX2Y=m
CONFIG_SND_USB_CAIAQ=m
CONFIG_SND_USB_CAIAQ_INPUT=y
CONFIG_SND_USB_US122L=m
CONFIG_SND_USB_6FIRE=m
CONFIG_SND_FIREWIRE=y
CONFIG_SND_FIREWIRE_LIB=m
CONFIG_SND_FIREWIRE_SPEAKERS=m
CONFIG_SND_ISIGHT=m
CONFIG_SND_SCS1X=m
CONFIG_SND_PCMCIA=y
CONFIG_SND_VXPOCKET=m
CONFIG_SND_PDAUDIOCF=m
# CONFIG_SND_SOC is not set
CONFIG_SOUND_PRIME=m
CONFIG_SOUND_OSS=m
CONFIG_SOUND_TRACEINIT=y
CONFIG_SOUND_DMAP=y
CONFIG_SOUND_VMIDI=m
CONFIG_SOUND_TRIX=m
CONFIG_SOUND_MSS=m
CONFIG_SOUND_MPU401=m
CONFIG_SOUND_PAS=m
CONFIG_SOUND_PSS=m
CONFIG_PSS_MIXER=y
# CONFIG_PSS_HAVE_BOOT is not set
# CONFIG_SOUND_SB is not set
CONFIG_SOUND_YM3812=m
CONFIG_SOUND_UART6850=m
CONFIG_SOUND_AEDSP16=m
CONFIG_SC6600=y
CONFIG_SC6600_JOY=y
CONFIG_SC6600_CDROM=4
CONFIG_SC6600_CDROMBASE=0
CONFIG_AC97_BUS=m

#
# HID support
#
CONFIG_HID=y
# CONFIG_HID_BATTERY_STRENGTH is not set
CONFIG_HIDRAW=y
CONFIG_UHID=m
CONFIG_HID_GENERIC=m

#
# Special HID drivers
#
CONFIG_HID_A4TECH=m
CONFIG_HID_ACRUX=m
CONFIG_HID_ACRUX_FF=y
CONFIG_HID_APPLE=m
CONFIG_HID_AUREAL=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_PRODIKEYS=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_DRAGONRISE=m
CONFIG_DRAGONRISE_FF=y
CONFIG_HID_EMS_FF=m
CONFIG_HID_ELECOM=m
CONFIG_HID_EZKEY=m
CONFIG_HID_HOLTEK=m
CONFIG_HOLTEK_FF=y
CONFIG_HID_KEYTOUCH=m
CONFIG_HID_KYE=m
CONFIG_HID_UCLOGIC=m
CONFIG_HID_WALTOP=m
CONFIG_HID_GYRATION=m
CONFIG_HID_ICADE=m
CONFIG_HID_TWINHAN=m
CONFIG_HID_KENSINGTON=m
CONFIG_HID_LCPOWER=m
CONFIG_HID_LENOVO_TPKBD=m
CONFIG_HID_LOGITECH=m
CONFIG_HID_LOGITECH_DJ=m
CONFIG_LOGITECH_FF=y
CONFIG_LOGIRUMBLEPAD2_FF=y
CONFIG_LOGIG940_FF=y
CONFIG_LOGIWHEELS_FF=y
CONFIG_HID_MAGICMOUSE=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m
CONFIG_HID_MULTITOUCH=m
CONFIG_HID_NTRIG=m
CONFIG_HID_ORTEK=m
CONFIG_HID_PANTHERLORD=m
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=m
CONFIG_HID_PICOLCD=m
CONFIG_HID_PICOLCD_FB=y
CONFIG_HID_PICOLCD_BACKLIGHT=y
CONFIG_HID_PICOLCD_LCD=y
CONFIG_HID_PICOLCD_LEDS=y
CONFIG_HID_PICOLCD_CIR=y
CONFIG_HID_PRIMAX=m
CONFIG_HID_PS3REMOTE=m
CONFIG_HID_ROCCAT=m
CONFIG_HID_SAITEK=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
CONFIG_HID_SPEEDLINK=m
# CONFIG_HID_STEELSERIES is not set
CONFIG_HID_SUNPLUS=m
CONFIG_HID_GREENASIA=m
CONFIG_GREENASIA_FF=y
CONFIG_HID_HYPERV_MOUSE=m
CONFIG_HID_SMARTJOYPLUS=m
CONFIG_SMARTJOYPLUS_FF=y
CONFIG_HID_TIVO=m
CONFIG_HID_TOPSEED=m
# CONFIG_HID_THINGM is not set
CONFIG_HID_THRUSTMASTER=m
CONFIG_THRUSTMASTER_FF=y
CONFIG_HID_WACOM=m
CONFIG_HID_WIIMOTE=m
CONFIG_HID_WIIMOTE_EXT=y
CONFIG_HID_ZEROPLUS=m
CONFIG_ZEROPLUS_FF=y
CONFIG_HID_ZYDACRON=m
CONFIG_HID_SENSOR_HUB=m

#
# USB HID support
#
CONFIG_USB_HID=m
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y

#
# I2C HID support
#
CONFIG_I2C_HID=m
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB_ARCH_HAS_XHCI=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=m
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
# CONFIG_USB_DYNAMIC_MINORS is not set
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set
# CONFIG_USB_DWC3 is not set
CONFIG_USB_MON=m
CONFIG_USB_WUSB=m
CONFIG_USB_WUSB_CBAF=m
# CONFIG_USB_WUSB_CBAF_DEBUG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_C67X00_HCD=m
CONFIG_USB_XHCI_HCD=m
# CONFIG_USB_XHCI_HCD_DEBUGGING is not set
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=m
CONFIG_USB_OXU210HP_HCD=m
CONFIG_USB_ISP116X_HCD=m
CONFIG_USB_ISP1760_HCD=m
CONFIG_USB_ISP1362_HCD=m
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_HCD_SSB=y
CONFIG_USB_OHCI_HCD_PLATFORM=y
CONFIG_USB_EHCI_HCD_PLATFORM=m
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_U132_HCD=m
CONFIG_USB_SL811_HCD=m
# CONFIG_USB_SL811_HCD_ISO is not set
CONFIG_USB_SL811_CS=m
CONFIG_USB_R8A66597_HCD=m
CONFIG_USB_WHCI_HCD=m
CONFIG_USB_HWA_HCD=m
CONFIG_USB_HCD_BCMA=m
CONFIG_USB_HCD_SSB=m
CONFIG_USB_CHIPIDEA=m
# CONFIG_USB_CHIPIDEA_HOST is not set
# CONFIG_USB_CHIPIDEA_DEBUG is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_REALTEK=m
CONFIG_REALTEK_AUTOPM=y
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
CONFIG_USB_STORAGE_ONETOUCH=m
CONFIG_USB_STORAGE_KARMA=m
CONFIG_USB_STORAGE_CYPRESS_ATACB=m
CONFIG_USB_STORAGE_ENE_UB6250=m

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m

#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_CH341=m
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP210X=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_F81232=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_IUU=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_METRO=m
CONFIG_USB_SERIAL_MOS7720=m
CONFIG_USB_SERIAL_MOS7715_PARPORT=y
CONFIG_USB_SERIAL_MOS7840=m
CONFIG_USB_SERIAL_MOTOROLA=m
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_QCAUX=m
CONFIG_USB_SERIAL_QUALCOMM=m
CONFIG_USB_SERIAL_SPCP8X5=m
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
CONFIG_USB_SERIAL_SIEMENS_MPI=m
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_SYMBOL=m
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_SERIAL_OPTICON=m
CONFIG_USB_SERIAL_VIVOPAY_SERIAL=m
# CONFIG_USB_SERIAL_XSENS_MT is not set
CONFIG_USB_SERIAL_ZIO=m
CONFIG_USB_SERIAL_ZTE=m
CONFIG_USB_SERIAL_SSU100=m
CONFIG_USB_SERIAL_QT2=m
CONFIG_USB_SERIAL_DEBUG=m

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
CONFIG_USB_SEVSEG=m
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
CONFIG_USB_CYPRESS_CY7C63=m
CONFIG_USB_CYTHERM=m
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
CONFIG_USB_TRANCEVIBRATOR=m
CONFIG_USB_IOWARRIOR=m
# CONFIG_USB_TEST is not set
CONFIG_USB_ISIGHTFW=m
CONFIG_USB_YUREX=m
CONFIG_USB_EZUSB_FX2=m
# CONFIG_USB_HSIC_USB3503 is not set

#
# USB Physical Layer drivers
#
# CONFIG_OMAP_USB3 is not set
# CONFIG_OMAP_CONTROL_USB is not set
CONFIG_USB_ISP1301=m
CONFIG_USB_RCAR_PHY=m
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
# CONFIG_USB_GADGET is not set

#
# OTG and related infrastructure
#
CONFIG_USB_OTG_UTILS=y
# CONFIG_USB_GPIO_VBUS is not set
# CONFIG_NOP_USB_XCEIV is not set
CONFIG_UWB=m
CONFIG_UWB_HWA=m
CONFIG_UWB_WHCI=m
CONFIG_UWB_I1480U=m
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set
# CONFIG_MMC_CLKGATE is not set

#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y
CONFIG_SDIO_UART=m
# CONFIG_MMC_TEST is not set

#
# MMC/SD/SDIO Host Controller Drivers
#
CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_RICOH_MMC=y
CONFIG_MMC_SDHCI_ACPI=m
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_MMC_WBSD=m
# CONFIG_MMC_TIFM_SD is not set
# CONFIG_MMC_SDRICOH_CS is not set
CONFIG_MMC_CB710=m
CONFIG_MMC_VIA_SDMMC=m
CONFIG_MMC_VUB300=m
CONFIG_MMC_USHC=m
CONFIG_MMC_REALTEK_PCI=m
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set

#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=m

#
# MemoryStick Host Controller Drivers
#
# CONFIG_MEMSTICK_TIFM_MS is not set
# CONFIG_MEMSTICK_JMICRON_38X is not set
# CONFIG_MEMSTICK_R592 is not set
CONFIG_MEMSTICK_REALTEK_PCI=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

#
# LED drivers
#
CONFIG_LEDS_LM3530=m
CONFIG_LEDS_LM3533=m
CONFIG_LEDS_LM3642=m
# CONFIG_LEDS_PCA9532 is not set
CONFIG_LEDS_GPIO=m
CONFIG_LEDS_LP3944=m
CONFIG_LEDS_LP55XX_COMMON=m
CONFIG_LEDS_LP5521=m
CONFIG_LEDS_LP5523=m
CONFIG_LEDS_CLEVO_MAIL=m
CONFIG_LEDS_PCA955X=m
CONFIG_LEDS_PCA9633=m
# CONFIG_LEDS_PWM is not set
CONFIG_LEDS_BD2802=m
CONFIG_LEDS_INTEL_SS4200=m
CONFIG_LEDS_LT3593=m
CONFIG_LEDS_DELL_NETBOOKS=m
CONFIG_LEDS_TCA6507=m
CONFIG_LEDS_LM355x=m
CONFIG_LEDS_OT200=m
CONFIG_LEDS_BLINKM=m
CONFIG_LEDS_TRIGGERS=y

#
# LED Triggers
#
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_ONESHOT=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
CONFIG_LEDS_TRIGGER_BACKLIGHT=m
CONFIG_LEDS_TRIGGER_CPU=y
CONFIG_LEDS_TRIGGER_GPIO=m
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m

#
# iptables trigger is under Netfilter config (LED target)
#
CONFIG_LEDS_TRIGGER_TRANSIENT=m
# CONFIG_ACCESSIBILITY is not set
CONFIG_INFINIBAND=m
CONFIG_INFINIBAND_USER_MAD=m
CONFIG_INFINIBAND_USER_ACCESS=m
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=m
CONFIG_INFINIBAND_MTHCA_DEBUG=y
CONFIG_INFINIBAND_IPATH=m
CONFIG_INFINIBAND_QIB=m
CONFIG_INFINIBAND_AMSO1100=m
# CONFIG_INFINIBAND_AMSO1100_DEBUG is not set
CONFIG_INFINIBAND_CXGB3=m
# CONFIG_INFINIBAND_CXGB3_DEBUG is not set
CONFIG_INFINIBAND_CXGB4=m
CONFIG_MLX4_INFINIBAND=m
CONFIG_INFINIBAND_NES=m
# CONFIG_INFINIBAND_NES_DEBUG is not set
CONFIG_INFINIBAND_OCRDMA=m
CONFIG_INFINIBAND_IPOIB=m
CONFIG_INFINIBAND_IPOIB_CM=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
# CONFIG_INFINIBAND_IPOIB_DEBUG_DATA is not set
CONFIG_INFINIBAND_SRP=m
CONFIG_INFINIBAND_SRPT=m
CONFIG_INFINIBAND_ISER=m
CONFIG_EDAC=y
CONFIG_EDAC_LEGACY_SYSFS=y
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=m
CONFIG_EDAC_MCE_INJ=m
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_AMD64=m
CONFIG_EDAC_AMD64_ERROR_INJECTION=y
CONFIG_EDAC_E752X=m
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
# CONFIG_EDAC_I3200 is not set
CONFIG_EDAC_X38=m
CONFIG_EDAC_I5400=m
CONFIG_EDAC_I7CORE=m
CONFIG_EDAC_I5000=m
CONFIG_EDAC_I5100=m
CONFIG_EDAC_I7300=m
# CONFIG_EDAC_SBRIDGE is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
# CONFIG_RTC_DEBUG is not set

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
CONFIG_RTC_DRV_88PM80X=m
CONFIG_RTC_DRV_DS1307=m
CONFIG_RTC_DRV_DS1374=m
CONFIG_RTC_DRV_DS1672=m
CONFIG_RTC_DRV_DS3232=m
CONFIG_RTC_DRV_MAX6900=m
CONFIG_RTC_DRV_MAX8907=m
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_ISL12022=m
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8523=m
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
CONFIG_RTC_DRV_M41T80=m
CONFIG_RTC_DRV_M41T80_WDT=y
CONFIG_RTC_DRV_BQ32K=m
CONFIG_RTC_DRV_S35390A=m
CONFIG_RTC_DRV_FM3130=m
CONFIG_RTC_DRV_RX8581=m
CONFIG_RTC_DRV_RX8025=m
CONFIG_RTC_DRV_EM3027=m
CONFIG_RTC_DRV_RV3029C2=m

#
# SPI RTC drivers
#

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_DS1286=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_STK17TA8=m
CONFIG_RTC_DRV_M48T86=m
CONFIG_RTC_DRV_M48T35=m
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_MSM6242=m
CONFIG_RTC_DRV_BQ4802=m
CONFIG_RTC_DRV_RP5C01=m
CONFIG_RTC_DRV_V3020=m
CONFIG_RTC_DRV_DS2404=m

#
# on-CPU RTC drivers
#

#
# HID Sensor RTC drivers
#
# CONFIG_RTC_DRV_HID_SENSOR_TIME is not set
CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set

#
# DMA Devices
#
# CONFIG_INTEL_MID_DMAC is not set
CONFIG_INTEL_IOATDMA=m
# CONFIG_DW_DMAC is not set
CONFIG_TIMB_DMA=m
CONFIG_PCH_DMA=m
CONFIG_DMA_ENGINE=y

#
# DMA Clients
#
CONFIG_NET_DMA=y
CONFIG_ASYNC_TX_DMA=y
# CONFIG_DMATEST is not set
CONFIG_DCA=m
CONFIG_AUXDISPLAY=y
CONFIG_KS0108=m
CONFIG_KS0108_PORT=0x378
CONFIG_KS0108_DELAY=2
CONFIG_CFAG12864B=m
CONFIG_CFAG12864B_RATE=20
CONFIG_UIO=m
CONFIG_UIO_CIF=m
CONFIG_UIO_PDRV=m
CONFIG_UIO_PDRV_GENIRQ=m
CONFIG_UIO_DMEM_GENIRQ=m
CONFIG_UIO_AEC=m
CONFIG_UIO_SERCOS3=m
CONFIG_UIO_PCI_GENERIC=m
CONFIG_UIO_NETX=m
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
# CONFIG_VFIO_PCI_VGA is not set

#
# Virtio drivers
#
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_MMIO is not set

#
# Microsoft Hyper-V guest support
#
CONFIG_HYPERV=m
CONFIG_HYPERV_UTILS=m
CONFIG_HYPERV_BALLOON=m
CONFIG_STAGING=y
CONFIG_ET131X=m
CONFIG_SLICOSS=m
CONFIG_USBIP_CORE=m
CONFIG_USBIP_VHCI_HCD=m
CONFIG_USBIP_HOST=m
# CONFIG_USBIP_DEBUG is not set
CONFIG_W35UND=m
CONFIG_PRISM2_USB=m
CONFIG_ECHO=m
# CONFIG_COMEDI is not set
CONFIG_ASUS_OLED=m
CONFIG_PANEL=m
CONFIG_PANEL_PARPORT=0
CONFIG_PANEL_PROFILE=5
# CONFIG_PANEL_CHANGE_MESSAGE is not set
CONFIG_R8187SE=m
CONFIG_RTL8192U=m
CONFIG_RTLLIB=m
CONFIG_RTLLIB_CRYPTO_CCMP=m
CONFIG_RTLLIB_CRYPTO_TKIP=m
CONFIG_RTLLIB_CRYPTO_WEP=m
CONFIG_RTL8192E=m
CONFIG_R8712U=m
CONFIG_RTS5139=m
# CONFIG_RTS5139_DEBUG is not set
CONFIG_TRANZPORT=m
CONFIG_IDE_PHISON=m
CONFIG_LINE6_USB=m
# CONFIG_LINE6_USB_IMPULSE_RESPONSE is not set
CONFIG_USB_SERIAL_QUATECH2=m
CONFIG_VT6655=m
CONFIG_VT6656=m
CONFIG_DX_SEP=m
CONFIG_ZRAM=m
# CONFIG_ZRAM_DEBUG is not set
CONFIG_ZSMALLOC=y
CONFIG_WLAGS49_H2=m
CONFIG_WLAGS49_H25=m
CONFIG_FB_SM7XX=m
CONFIG_CRYSTALHD=m
CONFIG_CXT1E1=m
CONFIG_SBE_PMCC4_NCOMM=y
CONFIG_FB_XGI=m
CONFIG_ACPI_QUICKSTART=m
CONFIG_SBE_2T3E3=m
CONFIG_USB_ENESTORAGE=m
CONFIG_BCM_WIMAX=m
CONFIG_FT1000=m
CONFIG_FT1000_USB=m
CONFIG_FT1000_PCMCIA=m

#
# Speakup console speech
#
CONFIG_SPEAKUP=m
CONFIG_SPEAKUP_SYNTH_ACNTSA=m
CONFIG_SPEAKUP_SYNTH_ACNTPC=m
CONFIG_SPEAKUP_SYNTH_APOLLO=m
CONFIG_SPEAKUP_SYNTH_AUDPTR=m
CONFIG_SPEAKUP_SYNTH_BNS=m
CONFIG_SPEAKUP_SYNTH_DECTLK=m
CONFIG_SPEAKUP_SYNTH_DECEXT=m
CONFIG_SPEAKUP_SYNTH_DECPC=m
CONFIG_SPEAKUP_SYNTH_DTLK=m
CONFIG_SPEAKUP_SYNTH_KEYPC=m
CONFIG_SPEAKUP_SYNTH_LTLK=m
CONFIG_SPEAKUP_SYNTH_SOFT=m
CONFIG_SPEAKUP_SYNTH_SPKOUT=m
CONFIG_SPEAKUP_SYNTH_TXPRT=m
CONFIG_SPEAKUP_SYNTH_DUMMY=m
CONFIG_TOUCHSCREEN_CLEARPAD_TM1217=m
CONFIG_TOUCHSCREEN_SYNAPTICS_I2C_RMI4=m
CONFIG_STAGING_MEDIA=y
CONFIG_DVB_AS102=m
CONFIG_DVB_CXD2099=m
CONFIG_VIDEO_DT3155=m
CONFIG_DT3155_CCIR=y
CONFIG_DT3155_STREAMING=y
CONFIG_VIDEO_GO7007=m
CONFIG_VIDEO_GO7007_USB=m
CONFIG_VIDEO_GO7007_USB_S2250_BOARD=m
CONFIG_VIDEO_GO7007_OV7640=m
CONFIG_VIDEO_GO7007_SAA7113=m
CONFIG_VIDEO_GO7007_SAA7115=m
CONFIG_VIDEO_GO7007_TW9903=m
CONFIG_VIDEO_GO7007_UDA1342=m
CONFIG_VIDEO_GO7007_SONY_TUNER=m
CONFIG_VIDEO_GO7007_TW2804=m
CONFIG_SOLO6X10=m
CONFIG_LIRC_STAGING=y
CONFIG_LIRC_BT829=m
CONFIG_LIRC_IGORPLUGUSB=m
CONFIG_LIRC_IMON=m
CONFIG_LIRC_PARALLEL=m
CONFIG_LIRC_SASEM=m
CONFIG_LIRC_SERIAL=m
CONFIG_LIRC_SERIAL_TRANSMITTER=y
CONFIG_LIRC_SIR=m
CONFIG_LIRC_ZILOG=m

#
# Android
#
# CONFIG_ANDROID is not set
CONFIG_USB_WPAN_HCD=m
CONFIG_WIMAX_GDM72XX=m
# CONFIG_WIMAX_GDM72XX_QOS is not set
# CONFIG_WIMAX_GDM72XX_K_MODE is not set
CONFIG_WIMAX_GDM72XX_WIMAX2=y
CONFIG_WIMAX_GDM72XX_USB=y
# CONFIG_WIMAX_GDM72XX_SDIO is not set
CONFIG_WIMAX_GDM72XX_USB_PM=y
CONFIG_CSR_WIFI=m
CONFIG_NET_VENDOR_SILICOM=y
CONFIG_SBYPASS=m
CONFIG_BPCTL=m
CONFIG_CED1401=m
CONFIG_DGRP=m
CONFIG_FIREWIRE_SERIAL=m
CONFIG_ZCACHE=y
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_ACER_WMI=m
CONFIG_ACERHDF=m
CONFIG_ASUS_LAPTOP=m
# CONFIG_CHROMEOS_LAPTOP is not set
# CONFIG_DELL_LAPTOP is not set
CONFIG_DELL_WMI=m
CONFIG_DELL_WMI_AIO=m
CONFIG_FUJITSU_LAPTOP=m
# CONFIG_FUJITSU_LAPTOP_DEBUG is not set
CONFIG_FUJITSU_TABLET=m
CONFIG_AMILO_RFKILL=m
CONFIG_HP_ACCEL=m
CONFIG_HP_WMI=m
CONFIG_MSI_LAPTOP=m
CONFIG_PANASONIC_LAPTOP=m
CONFIG_COMPAL_LAPTOP=m
CONFIG_SONY_LAPTOP=m
CONFIG_SONYPI_COMPAT=y
CONFIG_IDEAPAD_LAPTOP=m
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
# CONFIG_THINKPAD_ACPI_DEBUGFACILITIES is not set
# CONFIG_THINKPAD_ACPI_DEBUG is not set
# CONFIG_THINKPAD_ACPI_UNSAFE_LEDS is not set
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y
CONFIG_SENSORS_HDAPS=m
CONFIG_INTEL_MENLOW=m
CONFIG_EEEPC_LAPTOP=m
CONFIG_ASUS_WMI=m
CONFIG_ASUS_NB_WMI=m
CONFIG_EEEPC_WMI=m
CONFIG_ACPI_WMI=m
CONFIG_MSI_WMI=m
CONFIG_TOPSTAR_LAPTOP=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_TOSHIBA_BT_RFKILL=m
CONFIG_ACPI_CMPC=m
CONFIG_INTEL_IPS=m
CONFIG_IBM_RTL=m
CONFIG_XO15_EBOOK=m
CONFIG_SAMSUNG_LAPTOP=m
CONFIG_MXM_WMI=m
CONFIG_INTEL_OAKTRAIL=m
CONFIG_SAMSUNG_Q10=m
CONFIG_APPLE_GMUX=m

#
# Hardware Spinlock drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
# CONFIG_MAILBOX is not set
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_AMD_IOMMU=y
# CONFIG_AMD_IOMMU_STATS is not set
# CONFIG_AMD_IOMMU_V2 is not set
CONFIG_DMAR_TABLE=y
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
# CONFIG_IRQ_REMAP is not set

#
# Remoteproc drivers
#
# CONFIG_STE_MODEM_RPROC is not set

#
# Rpmsg drivers
#
CONFIG_VIRT_DRIVERS=y
# CONFIG_PM_DEVFREQ is not set
CONFIG_EXTCON=m

#
# Extcon Device Drivers
#
CONFIG_EXTCON_GPIO=m
CONFIG_MEMORY=y
# CONFIG_IIO is not set
# CONFIG_NTB is not set
CONFIG_VME_BUS=m

#
# VME Bridge Drivers
#
CONFIG_VME_CA91CX42=m
CONFIG_VME_TSI148=m

#
# VME Board Drivers
#
CONFIG_VMIVME_7805=m

#
# VME Device Drivers
#
CONFIG_VME_USER=m
CONFIG_VME_PIO2=m
CONFIG_PWM=y
CONFIG_IPACK_BUS=m
CONFIG_BOARD_TPCI200=m
CONFIG_SERIAL_IPOCTAL=m

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=y
CONFIG_EFI_VARS_PSTORE=y
# CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE is not set
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=m
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
# CONFIG_GOOGLE_FIRMWARE is not set

#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
CONFIG_JFS_STATISTICS=y
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
# CONFIG_XFS_DEBUG is not set
CONFIG_GFS2_FS=m
# CONFIG_GFS2_FS_LOCKING_DLM is not set
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
# CONFIG_OCFS2_DEBUG_MASKLOG is not set
# CONFIG_OCFS2_DEBUG_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=m
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y

#
# Caches
#
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
CONFIG_FSCACHE_OBJECT_LIST=y
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
CONFIG_HFSPLUS_FS=m
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
# CONFIG_JFFS2_FS_WBUF_VERIFY is not set
# CONFIG_JFFS2_SUMMARY is not set
# CONFIG_JFFS2_FS_XATTR is not set
CONFIG_JFFS2_COMPRESSION_OPTIONS=y
CONFIG_JFFS2_ZLIB=y
# CONFIG_JFFS2_LZO is not set
CONFIG_JFFS2_RTIME=y
# CONFIG_JFFS2_RUBIN is not set
# CONFIG_JFFS2_CMODE_NONE is not set
CONFIG_JFFS2_CMODE_PRIORITY=y
# CONFIG_JFFS2_CMODE_SIZE is not set
# CONFIG_JFFS2_CMODE_FAVOURLZO is not set
CONFIG_UBIFS_FS=m
CONFIG_UBIFS_FS_ADVANCED_COMPR=y
CONFIG_UBIFS_FS_LZO=y
CONFIG_UBIFS_FS_ZLIB=y
# CONFIG_LOGFS is not set
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_4K_DEVBLK_SIZE is not set
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
CONFIG_VXFS_FS=m
CONFIG_MINIX_FS=m
CONFIG_OMFS_FS=m
CONFIG_HPFS_FS=m
CONFIG_QNX4FS_FS=m
CONFIG_QNX6FS_FS=m
# CONFIG_QNX6FS_DEBUG is not set
CONFIG_ROMFS_FS=m
# CONFIG_ROMFS_BACKED_BY_BLOCK is not set
# CONFIG_ROMFS_BACKED_BY_MTD is not set
CONFIG_ROMFS_BACKED_BY_BOTH=y
CONFIG_ROMFS_ON_BLOCK=y
CONFIG_ROMFS_ON_MTD=y
CONFIG_PSTORE=y
# CONFIG_PSTORE_CONSOLE is not set
CONFIG_PSTORE_RAM=m
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_UFS_DEBUG is not set
CONFIG_EXOFS_FS=m
# CONFIG_EXOFS_DEBUG is not set
CONFIG_F2FS_FS=m
CONFIG_F2FS_STAT_FS=y
CONFIG_F2FS_FS_XATTR=y
CONFIG_F2FS_FS_POSIX_ACL=y
CONFIG_ORE=m
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V2=m
CONFIG_NFS_V3=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
CONFIG_NFS_SWAP=y
CONFIG_NFS_V4_1=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_PNFS_OBJLAYOUT=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFS_DEBUG=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
# CONFIG_NFSD_V4 is not set
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_SUNRPC_XPRT_RDMA=m
CONFIG_SUNRPC_SWAP=y
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_SUNRPC_DEBUG=y
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=m
CONFIG_CIFS_STATS=y
CONFIG_CIFS_STATS2=y
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_CIFS_ACL=y
CONFIG_CIFS_DEBUG=y
# CONFIG_CIFS_DEBUG2 is not set
CONFIG_CIFS_DFS_UPCALL=y
# CONFIG_CIFS_SMB2 is not set
CONFIG_CIFS_FSCACHE=y
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
CONFIG_NCPFS_SMALLDOS=y
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
# CONFIG_AFS_FS is not set
CONFIG_9P_FS=m
# CONFIG_9P_FSCACHE is not set
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_MAC_ROMAN=m
CONFIG_NLS_MAC_CELTIC=m
CONFIG_NLS_MAC_CENTEURO=m
CONFIG_NLS_MAC_CROATIAN=m
CONFIG_NLS_MAC_CYRILLIC=m
CONFIG_NLS_MAC_GAELIC=m
CONFIG_NLS_MAC_GREEK=m
CONFIG_NLS_MAC_ICELAND=m
CONFIG_NLS_MAC_INUIT=m
CONFIG_NLS_MAC_ROMANIAN=m
CONFIG_NLS_MAC_TURKISH=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_READABLE_ASM is not set
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=1
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=480
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set

#
# RCU Debugging
#
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
# CONFIG_RCU_CPU_STALL_INFO is not set
# CONFIG_RCU_TRACE is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_LKDTM=m
CONFIG_NOTIFIER_ERROR_INJECTION=m
CONFIG_CPU_NOTIFIER_ERROR_INJECT=m
CONFIG_PM_NOTIFIER_ERROR_INJECT=m
CONFIG_MEMORY_NOTIFIER_ERROR_INJECT=m
# CONFIG_FAULT_INJECTION is not set
CONFIG_LATENCYTOP=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACE_CLOCK=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_FUNCTION_TRACER is not set
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
# CONFIG_FTRACE_SYSCALLS is not set
# CONFIG_TRACER_SNAPSHOT is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_STACK_TRACER is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_UPROBE_EVENT=y
CONFIG_PROBE_EVENTS=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
CONFIG_RING_BUFFER_BENCHMARK=m
CONFIG_RBTREE_TEST=m
CONFIG_INTERVAL_TREE_TEST=m
CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
CONFIG_FIREWIRE_OHCI_REMOTE_DMA=y
CONFIG_BUILD_DOCSRC=y
CONFIG_DYNAMIC_DEBUG=y
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_ATOMIC64_SELFTEST is not set
CONFIG_ASYNC_RAID6_TEST=m
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_KMEMCHECK is not set
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_STRICT_DEVMEM is not set
# CONFIG_X86_VERBOSE_BOOTUP is not set
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_DEBUG_RODATA_TEST is not set
CONFIG_DEBUG_SET_MODULE_RONX=y
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_DEBUG_TLBFLUSH is not set
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
# CONFIG_X86_DECODER_SELFTEST is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y
# CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
# CONFIG_DEBUG_NMI_SELFTEST is not set

#
# Security options
#
CONFIG_KEYS=y
CONFIG_TRUSTED_KEYS=m
CONFIG_ENCRYPTED_KEYS=m
# CONFIG_KEYS_DEBUG_PROC_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_SECURITY_PATH=y
# CONFIG_INTEL_TXT is not set
CONFIG_LSM_MMAP_MIN_ADDR=0
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=0
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
# CONFIG_SECURITY_SMACK is not set
CONFIG_SECURITY_TOMOYO=y
CONFIG_SECURITY_TOMOYO_MAX_ACCEPT_ENTRY=2048
CONFIG_SECURITY_TOMOYO_MAX_AUDIT_LOG=1024
# CONFIG_SECURITY_TOMOYO_OMIT_USERSPACE_LOADER is not set
CONFIG_SECURITY_TOMOYO_POLICY_LOADER="/sbin/tomoyo-init"
CONFIG_SECURITY_TOMOYO_ACTIVATION_TRIGGER="/sbin/init"
CONFIG_SECURITY_APPARMOR=y
CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1
# CONFIG_SECURITY_YAMA is not set
# CONFIG_IMA is not set
# CONFIG_DEFAULT_SECURITY_SELINUX is not set
# CONFIG_DEFAULT_SECURITY_TOMOYO is not set
CONFIG_DEFAULT_SECURITY_APPARMOR=y
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_DEFAULT_SECURITY="apparmor"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=m
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
# CONFIG_CRYPTO_PCRYPT is not set
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
# CONFIG_CRYPTO_TEST is not set
CONFIG_CRYPTO_ABLK_HELPER_X86=m
CONFIG_CRYPTO_GLUE_HELPER_X86=m

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m

#
# Block modes
#
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set
# CONFIG_CRYPTO_VMAC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_X86_64=y
CONFIG_CRYPTO_CRC32C_INTEL=m
# CONFIG_CRYPTO_CRC32 is not set
# CONFIG_CRYPTO_CRC32_PCLMUL is not set
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAMELLIA_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST5_AVX_X86_64=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_CAST6_AVX_X86_64=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX_X86_64=m
CONFIG_CRYPTO_TEA=m
# CONFIG_CRYPTO_TWOFISH is not set
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_ZLIB=m
CONFIG_CRYPTO_LZO=y

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_USER_API=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
CONFIG_ASYMMETRIC_KEY_TYPE=m
CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=m
CONFIG_PUBLIC_KEY_ALGO_RSA=m
CONFIG_X509_CERTIFICATE_PARSER=m
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
# CONFIG_VHOST_NET is not set
# CONFIG_TCM_VHOST is not set
CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_PERCPU_RWSEM=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
CONFIG_CRC7=m
CONFIG_LIBCRC32C=m
CONFIG_CRC8=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
# CONFIG_XZ_DEC_POWERPC is not set
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_ENC8=y
CONFIG_REED_SOLOMON_DEC8=y
CONFIG_REED_SOLOMON_DEC16=y
CONFIG_BCH=m
CONFIG_BCH_CONST_PARAMS=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_BTREE=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
CONFIG_LRU_CACHE=m
CONFIG_AVERAGE=y
CONFIG_CLZ_TAB=y
CONFIG_CORDIC=m
CONFIG_DDR=y
CONFIG_MPILIB=m
CONFIG_OID_REGISTRY=m
CONFIG_UCS2_STRING=y

[-- Attachment #3: demon.dmesg --]
[-- Type: text/plain, Size: 86343 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.9.0bisect+ (mhocko@noe) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1484 SMP Thu Jun 20 16:44:40 CEST 2013
[    0.000000] Command line: root=/dev/sda2 console=tty0 console=ttyS1,115200 nomodeset resume=/dev/sda1 splash=silent crashkernel= showopts vga=6 mem=1G
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000d2000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ff1ffff] usable
[    0.000000] BIOS-e820: [mem 0x000000007ff20000-0x000000007ff23fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000007ff24000-0x000000007ff7ffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000007ff80000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fff80000-0x00000000ffffffff] reserved
[    0.000000] e820: remove [mem 0x40000000-0xfffffffffffffffe] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] e820: user-defined physical RAM map:
[    0.000000] user: [mem 0x0000000000000000-0x000000000009d7ff] usable
[    0.000000] user: [mem 0x000000000009d800-0x000000000009ffff] reserved
[    0.000000] user: [mem 0x00000000000d2000-0x00000000000fffff] reserved
[    0.000000] user: [mem 0x0000000000100000-0x000000003fffffff] usable
[    0.000000] user: [mem 0x000000007ff20000-0x000000007ff23fff] ACPI data
[    0.000000] user: [mem 0x000000007ff24000-0x000000007ff7ffff] ACPI NVS
[    0.000000] user: [mem 0x000000007ff80000-0x000000007fffffff] reserved
[    0.000000] user: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
[    0.000000] user: [mem 0x00000000fff80000-0x00000000ffffffff] reserved
[    0.000000] SMBIOS 2.34 present.
[    0.000000] DMI: AMD A8440/WARTHOG, BIOS PW2A00-5 09/23/2005
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x40000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-D3FFF write-protect
[    0.000000]   D4000-E3FFF uncachable
[    0.000000]   E4000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0000000000 mask FF80000000 write-back
[    0.000000]   1 disabled
[    0.000000]   2 disabled
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] found SMP MP-table at [mem 0x000f7850-0x000f785f] mapped at [ffff8800000f7850]
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000096000] 96000 size 28672
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x01d27000, 0x01d27fff] PGTABLE
[    0.000000] BRK [0x01d28000, 0x01d28fff] PGTABLE
[    0.000000] BRK [0x01d29000, 0x01d29fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x3fe00000-0x3fffffff]
[    0.000000]  [mem 0x3fe00000-0x3fffffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x3c000000-0x3fdfffff]
[    0.000000]  [mem 0x3c000000-0x3fdfffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x00100000-0x3bffffff]
[    0.000000]  [mem 0x00100000-0x001fffff] page 4k
[    0.000000]  [mem 0x00200000-0x3bffffff] page 2M
[    0.000000] RAMDISK: [mem 0x36cc0000-0x37feffff]
[    0.000000] crashkernel: memory value expected
[    0.000000] ACPI: RSDP 00000000000f7820 00024 (v02 PTLTD )
[    0.000000] ACPI: XSDT 000000007ff2026c 00064 (v01 PTLTD  ? XSDT   06040000  LTP 00000000)
[    0.000000] ACPI: FACP 000000007ff202d0 00074 (v01 AMD    HAMMER   06040000 PTEC 000F4240)
[    0.000000] ACPI: DSDT 000000007ff20344 03107 (v01 AMD-K8  AMDACPI 06040000 MSFT 0100000E)
[    0.000000] ACPI: FACS 000000007ff24fc0 00040
[    0.000000] ACPI: SSDT 000000007ff2344b 007C4 (v01 PTLTD  POWERNOW 06040000  LTP 00000001)
[    0.000000] ACPI: SRAT 000000007ff23c0f 00178 (v01 AMD    HAMMER   06040000 AMD  00000001)
[    0.000000] ACPI: SSDT 000000007ff23d87 000E7 (v01 AMD-K8 AMD-ACPI 06040000  AMD 00000001)
[    0.000000] ACPI: HPET 000000007ff23e6e 00038 (v01 AMD    HAMMER   06040000 PTEC 00000000)
[    0.000000] ACPI: SPCR 000000007ff23ea6 00050 (v01 PTLTD  $UCRTBL$ 06040000 PTL  00000001)
[    0.000000] ACPI: APIC 000000007ff23ef6 000E2 (v01 PTLTD  ? APIC   06040000  LTP 00000000)
[    0.000000] ACPI: BOOT 000000007ff23fd8 00028 (v01 PTLTD  $SBFTBL$ 06040000  LTP 00000001)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x02 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x03 -> Node 1
[    0.000000] SRAT: PXM 2 -> APIC 0x04 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x05 -> Node 2
[    0.000000] SRAT: PXM 3 -> APIC 0x06 -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x07 -> Node 3
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0x1fffffff]
[    0.000000] SRAT: Node 1 PXM 1 [mem 0x20000000-0x3fffffff]
[    0.000000] SRAT: Node 2 PXM 2 [mem 0x40000000-0x5fffffff]
[    0.000000] SRAT: Node 3 PXM 3 [mem 0x60000000-0x7fffffff]
[    0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x1fffffff] -> [mem 0x00000000-0x1fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1fffffff]
[    0.000000]   NODE_DATA [mem 0x1ffec000-0x1fffffff]
[    0.000000] Initmem setup node 1 [mem 0x20000000-0x3fffffff]
[    0.000000]   NODE_DATA [mem 0x3ffec000-0x3fffffff]
[    0.000000] [ffffea0000700000-ffffea00007fffff] potential offnode page_structs
[    0.000000]  [ffffea0000000000-ffffea00007fffff] PMD -> [ffff88001f600000-ffff88001fdfffff] on node 0
[    0.000000]  [ffffea0000800000-ffffea0000dfffff] PMD -> [ffff88003ee00000-ffff88003f3fffff] on node 1
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009cfff]
[    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
[    0.000000]   node   1: [mem 0x20000000-0x3fffffff]
[    0.000000] On node 0 totalpages: 130972
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 22 pages reserved
[    0.000000]   DMA zone: 3996 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1736 pages used for memmap
[    0.000000]   DMA32 zone: 126976 pages, LIFO batch:31
[    0.000000] On node 1 totalpages: 131072
[    0.000000]   DMA32 zone: 1792 pages used for memmap
[    0.000000]   DMA32 zone: 131072 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0xc008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: IOAPIC (id[0x09] address[0xe8000000] gsi_base[24])
[    0.000000] IOAPIC[1]: apic_id 9, version 17, address 0xe8000000, GSI 24-27
[    0.000000] ACPI: IOAPIC (id[0x0a] address[0xe8001000] gsi_base[28])
[    0.000000] IOAPIC[2]: apic_id 10, version 17, address 0xe8001000, GSI 28-31
[    0.000000] ACPI: IOAPIC (id[0x0b] address[0xf8000000] gsi_base[32])
[    0.000000] IOAPIC[3]: apic_id 11, version 17, address 0xf8000000, GSI 32-35
[    0.000000] ACPI: IOAPIC (id[0x0c] address[0xf8001000] gsi_base[36])
[    0.000000] IOAPIC[4]: apic_id 12, version 17, address 0xf8001000, GSI 36-39
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x102282a0 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 56
[    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
[    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000d2000
[    0.000000] PM: Registered nosave memory: 00000000000d2000 - 0000000000100000
[    0.000000] e820: [mem 0x80000000-0xfebfffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:8 nr_node_ids:4
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88001f000000 s80512 r8192 d21888 u1048576
[    0.000000] pcpu-alloc: s80512 r8192 d21888 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 [0] 4 5 [0] 6 7 [1] 2 3 
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 258438
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=/dev/sda2 console=tty0 console=ttyS1,115200 nomodeset resume=/dev/sda1 splash=silent crashkernel= showopts vga=6 mem=1G
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] __ex_table already sorted, skipping sort
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ c1f6000000 size 32 MB
[    0.000000] Aperture beyond 4GB. Ignoring.
[    0.000000] Memory: 999240K/1048176K available (5657K kernel code, 671K rwdata, 2512K rodata, 1268K init, 1252K bss, 48936K reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=8.
[    0.000000] NR_IRQS:33024 nr_irqs:1016 16
[    0.000000] Console: colour VGA+ 80x60
[    0.000000] console [tty0] enabled
[    0.000000] console [ttyS1] enabled
[    0.000000] allocated 4194304 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or sysctl
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2389.643 MHz processor
[    0.000000] tsc: Marking TSC unstable due to TSCs unsynchronized
[    0.020000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4779.28 BogoMIPS (lpj=9558572)
[    0.024003] pid_max: default: 32768 minimum: 301
[    0.028077] Security Framework initialized
[    0.032034] AppArmor: AppArmor initialized
[    0.036098] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.044033] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.048371] Mount-cache hash table entries: 256
[    0.052328] Initializing cgroup subsys cpuacct
[    0.056004] Initializing cgroup subsys memory
[    0.060043] Initializing cgroup subsys devices
[    0.064004] Initializing cgroup subsys freezer
[    0.068003] Initializing cgroup subsys net_cls
[    0.072003] Initializing cgroup subsys blkio
[    0.076003] Initializing cgroup subsys perf_event
[    0.080050] tseg: 007ff80000
[    0.080053] CPU: Physical Processor ID: 0
[    0.084003] CPU: Processor Core ID: 0
[    0.088003] mce: CPU supports 5 MCE banks
[    0.092009] LVT offset 0 assigned for vector 0xf9
[    0.096011] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 4
[    0.096011] Last level dTLB entries: 4KB 512, 2MB 8, 4MB 4
[    0.096011] tlb_flushall_shift: 4
[    0.100105] Freeing SMP alternatives memory: 24K (ffffffff81be6000 - ffffffff81bec000)
[    0.104826] ACPI: Core revision 20130117
[    0.113679] ACPI: All ACPI Tables successfully acquired
[    0.117305] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.162012] smpboot: CPU0: AMD Engineering Sample (fam: 0f, model: 41, stepping: 01)
[    0.176000] Performance Events: AMD PMU driver.
[    0.176003] ... version:                0
[    0.180001] ... bit width:              48
[    0.184000] ... generic registers:      4
[    0.188001] ... value mask:             0000ffffffffffff
[    0.192001] ... max period:             00007fffffffffff
[    0.196000] ... fixed-purpose events:   0
[    0.200000] ... event mask:             000000000000000f
[    0.204710] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    0.212113] smpboot: Booting Node   0, Processors  #1 OK
[    0.309004] smpboot: Booting Node   1, Processors  #2 #3 OK
[    0.492908] smpboot: Booting Node   0, Processors  #4 #5 #6 #7 OK
[    0.856116] Brought up 8 CPUs
[    0.859329] smpboot: Total of 8 processors activated (38237.11 BogoMIPS)
[    0.864413] devtmpfs: initialized
[    0.872235] PM: Registering ACPI NVS region [mem 0x7ff24000-0x7ff7ffff] (376832 bytes)
[    0.876178] RTC time: 14:59:16, date: 06/20/13
[    0.880104] NET: Registered protocol family 16
[    0.884209] node 0 link 1: io port [0, 2fff]
[    0.884212] node 1 link 1: io port [3000, 3fff]
[    0.884215] TOM: 0000000080000000 aka 2048M
[    0.888003] node 0 link 1: mmio [e8000000, e80fffff]
[    0.888006] node 1 link 1: mmio [f8000000, f84fffff]
[    0.888009] node 0 link 1: mmio [e8200000, e85fffff]
[    0.888013] bus: [bus 00-09] on node 0 link 1
[    0.888015] bus: 00 [io  0x0000-0x2fff]
[    0.888017] bus: 00 [io  0x4000-0xffff]
[    0.888018] bus: 00 [mem 0x80000000-0xe81fffff]
[    0.888020] bus: 00 [mem 0xe8200000-0xf7ffffff]
[    0.888022] bus: 00 [mem 0xf8500000-0xfcffffffff]
[    0.888023] bus: [bus 0a-ff] on node 1 link 1
[    0.888025] bus: 0a [io  0x3000-0x3fff]
[    0.888026] bus: 0a [mem 0xf8000000-0xf84fffff]
[    0.888058] ACPI: bus type PCI registered
[    0.892075] PCI: Using configuration type 1 for base access
[    0.897330] bio: create slab <bio-0> at 0
[    0.900094] ACPI: Added _OSI(Module Device)
[    0.904003] ACPI: Added _OSI(Processor Device)
[    0.908001] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.912001] ACPI: Added _OSI(Processor Aggregator Device)
[    0.916469] ACPI: EC: Look up EC in DSDT
[    0.923193] ACPI: Interpreter enabled
[    0.924013] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130117/hwxface-568)
[    0.936002] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20130117/hwxface-568)
[    0.946424] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20130117/hwxface-568)
[    0.956005] ACPI: (supports S0 S1 S5)
[    0.960001] ACPI: Using IOAPIC for interrupt routing
[    0.964182] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
[    0.979804] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-09])
[    0.987136] acpi PNP0A03:00: host bridge window [io  0x03b0-0x03df] (ignored)
[    0.987139] acpi PNP0A03:00: host bridge window [io  0x0d00-0x2fff] (ignored)
[    0.987142] acpi PNP0A03:00: host bridge window [mem 0xfed00000-0xfed0ffff] (ignored)
[    0.987144] acpi PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff] (ignored)
[    0.987147] acpi PNP0A03:00: host bridge window [mem 0xfec00000-0xfec0ffff] (ignored)
[    0.987149] acpi PNP0A03:00: host bridge window [mem 0xf0000000-0xf7ffffff] (ignored)
[    0.987151] acpi PNP0A03:00: host bridge window [mem 0xe8000000-0xe85fffff] (ignored)
[    0.987154] acpi PNP0A03:00: host bridge window [mem 0x000c0000-0x000cafff] (ignored)
[    0.987156] acpi PNP0A03:00: host bridge window [mem 0x000d8000-0x000dbfff] (ignored)
[    0.987158] acpi PNP0A03:00: host bridge window [io  0x0000-0x03af] (ignored)
[    0.987160] acpi PNP0A03:00: host bridge window [io  0x03e0-0x0cf7] (ignored)
[    0.987164] PCI: root bus 00: hardware-probed resources
[    0.987171] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    0.988040] PCI host bridge to bus 0000:00
[    0.992004] pci_bus 0000:00: root bus resource [bus 00-09]
[    0.996002] pci_bus 0000:00: root bus resource [io  0x0000-0x2fff]
[    1.000002] pci_bus 0000:00: root bus resource [io  0x4000-0xffff]
[    1.004002] pci_bus 0000:00: root bus resource [mem 0x80000000-0xe81fffff]
[    1.008002] pci_bus 0000:00: root bus resource [mem 0xe8200000-0xf7ffffff]
[    1.012002] pci_bus 0000:00: root bus resource [mem 0xf8500000-0xfcffffffff]
[    1.016022] pci 0000:00:06.0: [1022:7460] type 01 class 0x060400
[    1.016132] pci 0000:00:06.0: System wakeup disabled by ACPI
[    1.020050] pci 0000:00:07.0: [1022:7468] type 00 class 0x060100
[    1.020161] pci 0000:00:07.1: [1022:7469] type 00 class 0x01018a
[    1.020192] pci 0000:00:07.1: reg 20: [io  0x1020-0x102f]
[    1.020275] pci 0000:00:07.2: [1022:746a] type 00 class 0x0c0500
[    1.020289] pci 0000:00:07.2: reg 10: [io  0x1000-0x101f]
[    1.020421] pci 0000:00:07.3: [1022:746b] type 00 class 0x068000
[    1.020549] pci 0000:00:0a.0: [1022:7450] type 01 class 0x060400
[    1.020615] pci 0000:00:0a.0: System wakeup disabled by ACPI
[    1.024041] pci 0000:00:0a.1: [1022:7451] type 00 class 0x080010
[    1.024050] pci 0000:00:0a.1: reg 10: [mem 0xe8000000-0xe8000fff 64bit]
[    1.024137] pci 0000:00:0b.0: [1022:7450] type 01 class 0x060400
[    1.024203] pci 0000:00:0b.0: System wakeup disabled by ACPI
[    1.028043] pci 0000:00:0b.1: [1022:7451] type 00 class 0x080010
[    1.028052] pci 0000:00:0b.1: reg 10: [mem 0xe8001000-0xe8001fff 64bit]
[    1.028151] pci 0000:00:18.0: [1022:1100] type 00 class 0x060000
[    1.028225] pci 0000:00:18.1: [1022:1101] type 00 class 0x060000
[    1.028295] pci 0000:00:18.2: [1022:1102] type 00 class 0x060000
[    1.028366] pci 0000:00:18.3: [1022:1103] type 00 class 0x060000
[    1.028444] pci 0000:00:19.0: [1022:1100] type 00 class 0x060000
[    1.028504] pci 0000:00:19.1: [1022:1101] type 00 class 0x060000
[    1.028576] pci 0000:00:19.2: [1022:1102] type 00 class 0x060000
[    1.028645] pci 0000:00:19.3: [1022:1103] type 00 class 0x060000
[    1.028721] pci 0000:00:1a.0: [1022:1100] type 00 class 0x060000
[    1.028797] pci 0000:00:1a.1: [1022:1101] type 00 class 0x060000
[    1.028867] pci 0000:00:1a.2: [1022:1102] type 00 class 0x060000
[    1.028935] pci 0000:00:1a.3: [1022:1103] type 00 class 0x060000
[    1.029015] pci 0000:00:1b.0: [1022:1100] type 00 class 0x060000
[    1.029095] pci 0000:00:1b.1: [1022:1101] type 00 class 0x060000
[    1.029165] pci 0000:00:1b.2: [1022:1102] type 00 class 0x060000
[    1.029237] pci 0000:00:1b.3: [1022:1103] type 00 class 0x060000
[    1.029371] pci 0000:01:00.0: [1022:7464] type 00 class 0x0c0310
[    1.029385] pci 0000:01:00.0: reg 10: [mem 0xe8110000-0xe8110fff]
[    1.032009] pci 0000:01:00.0: System wakeup disabled by ACPI
[    1.036051] pci 0000:01:00.1: [1022:7464] type 00 class 0x0c0310
[    1.036066] pci 0000:01:00.1: reg 10: [mem 0xe8111000-0xe8111fff]
[    1.036142] pci 0000:01:00.1: System wakeup disabled by ACPI
[    1.040065] pci 0000:01:05.0: [1095:3512] type 00 class 0x018000
[    1.040083] pci 0000:01:05.0: reg 10: [io  0x2420-0x2427]
[    1.040094] pci 0000:01:05.0: reg 14: [io  0x2414-0x2417]
[    1.040104] pci 0000:01:05.0: reg 18: [io  0x2418-0x241f]
[    1.040114] pci 0000:01:05.0: reg 1c: [io  0x2410-0x2413]
[    1.040125] pci 0000:01:05.0: reg 20: [io  0x2400-0x240f]
[    1.040135] pci 0000:01:05.0: reg 24: [mem 0xe8112000-0xe81121ff]
[    1.040146] pci 0000:01:05.0: reg 30: [mem 0x00000000-0x0007ffff pref]
[    1.040175] pci 0000:01:05.0: supports D1 D2
[    1.040233] pci 0000:01:06.0: [1002:5159] type 00 class 0x030000
[    1.040252] pci 0000:01:06.0: reg 10: [mem 0xf0000000-0xf7ffffff pref]
[    1.040262] pci 0000:01:06.0: reg 14: [io  0x2000-0x20ff]
[    1.040273] pci 0000:01:06.0: reg 18: [mem 0xe8100000-0xe810ffff]
[    1.040309] pci 0000:01:06.0: reg 30: [mem 0x00000000-0x0001ffff pref]
[    1.040339] pci 0000:01:06.0: supports D1 D2
[    1.040425] pci 0000:00:06.0: PCI bridge to [bus 01]
[    1.044005] pci 0000:00:06.0:   bridge window [io  0x2000-0x2fff]
[    1.044009] pci 0000:00:06.0:   bridge window [mem 0xe8100000-0xe81fffff]
[    1.044013] pci 0000:00:06.0:   bridge window [mem 0xf0000000-0xf7ffffff pref]
[    1.044063] pci 0000:02:01.0: [14e4:1645] type 00 class 0x020000
[    1.044075] pci 0000:02:01.0: reg 10: [mem 0xe8200000-0xe820ffff 64bit]
[    1.044117] pci 0000:02:01.0: PME# supported from D3hot D3cold
[    1.044192] pci 0000:00:0a.0: PCI bridge to [bus 02]
[    1.048005] pci 0000:00:0a.0:   bridge window [mem 0xe8200000-0xe82fffff]
[    1.048048] pci 0000:03:02.0: [14e4:1648] type 00 class 0x020000
[    1.048061] pci 0000:03:02.0: reg 10: [mem 0xe8310000-0xe831ffff 64bit]
[    1.048070] pci 0000:03:02.0: reg 18: [mem 0xe8300000-0xe830ffff 64bit]
[    1.048110] pci 0000:03:02.0: PME# supported from D3hot D3cold
[    1.048166] pci 0000:03:02.1: [14e4:1648] type 00 class 0x020000
[    1.048179] pci 0000:03:02.1: reg 10: [mem 0xe8330000-0xe833ffff 64bit]
[    1.048187] pci 0000:03:02.1: reg 18: [mem 0xe8320000-0xe832ffff 64bit]
[    1.048227] pci 0000:03:02.1: PME# supported from D3hot D3cold
[    1.048293] pci 0000:00:0b.0: PCI bridge to [bus 03]
[    1.052005] pci 0000:00:0b.0:   bridge window [mem 0xe8300000-0xe83fffff]
[    1.052021] acpi PNP0A03:00: ACPI _OSC support notification failed, disabling PCIe ASPM
[    1.056002] acpi PNP0A03:00: Unable to request _OSC control (_OSC support mask: 0x08)
[    1.060269] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 5 10 *11)
[    1.068233] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 5 *10 11)
[    1.074068] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 *5 10 11)
[    1.081465] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 5 10 *11)
[    1.090436] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 0a-ff])
[    1.096351] acpi PNP0A03:01: host bridge window [mem 0xf8000000-0xf84fffff] (ignored)
[    1.096354] acpi PNP0A03:01: host bridge window [io  0x3000-0x3fff] (ignored)
[    1.096356] PCI: root bus 0a: hardware-probed resources
[    1.096360] acpi PNP0A03:01: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    1.100040] PCI host bridge to bus 0000:0a
[    1.104002] pci_bus 0000:0a: root bus resource [bus 0a-ff]
[    1.108002] pci_bus 0000:0a: root bus resource [io  0x3000-0x3fff]
[    1.112002] pci_bus 0000:0a: root bus resource [mem 0xf8000000-0xf84fffff]
[    1.116011] pci 0000:0a:01.0: [1022:7450] type 01 class 0x060400
[    1.116071] pci 0000:0a:01.0: System wakeup disabled by ACPI
[    1.120050] pci 0000:0a:01.1: [1022:7451] type 00 class 0x080010
[    1.120060] pci 0000:0a:01.1: reg 10: [mem 0xf8000000-0xf8000fff 64bit]
[    1.120135] pci 0000:0a:02.0: [1022:7450] type 01 class 0x060400
[    1.120188] pci 0000:0a:02.0: System wakeup disabled by ACPI
[    1.124043] pci 0000:0a:02.1: [1022:7451] type 00 class 0x080010
[    1.124053] pci 0000:0a:02.1: reg 10: [mem 0xf8001000-0xf8001fff 64bit]
[    1.124174] pci 0000:0b:01.0: [8086:107c] type 00 class 0x020000
[    1.124186] pci 0000:0b:01.0: reg 10: [mem 0xf8120000-0xf813ffff]
[    1.124192] pci 0000:0b:01.0: reg 14: [mem 0xf8100000-0xf811ffff]
[    1.124199] pci 0000:0b:01.0: reg 18: [io  0x3000-0x303f]
[    1.124218] pci 0000:0b:01.0: reg 30: [mem 0x00000000-0x0001ffff pref]
[    1.124239] pci 0000:0b:01.0: PME# supported from D0 D3hot D3cold
[    1.124301] pci 0000:0a:01.0: PCI bridge to [bus 0b-0f]
[    1.128004] pci 0000:0a:01.0:   bridge window [io  0x3000-0x3fff]
[    1.128008] pci 0000:0a:01.0:   bridge window [mem 0xf8100000-0xf81fffff]
[    1.128054] pci 0000:10:02.0: [14e4:1648] type 00 class 0x020000
[    1.128068] pci 0000:10:02.0: reg 10: [mem 0xf8210000-0xf821ffff 64bit]
[    1.128077] pci 0000:10:02.0: reg 18: [mem 0xf8200000-0xf820ffff 64bit]
[    1.128120] pci 0000:10:02.0: PME# supported from D3hot D3cold
[    1.128176] pci 0000:10:02.1: [14e4:1648] type 00 class 0x020000
[    1.128189] pci 0000:10:02.1: reg 10: [mem 0xf8230000-0xf823ffff 64bit]
[    1.128199] pci 0000:10:02.1: reg 18: [mem 0xf8220000-0xf822ffff 64bit]
[    1.128242] pci 0000:10:02.1: PME# supported from D3hot D3cold
[    1.128308] pci 0000:0a:02.0: PCI bridge to [bus 10-14]
[    1.132005] pci 0000:0a:02.0:   bridge window [mem 0xf8200000-0xf82fffff]
[    1.132016] acpi PNP0A03:01: ACPI _OSC support notification failed, disabling PCIe ASPM
[    1.136002] acpi PNP0A03:01: Unable to request _OSC control (_OSC support mask: 0x08)
[    1.140085] acpi root: \_SB_.PCI0 notify handler is installed
[    1.140106] acpi root: \_SB_.PCI1 notify handler is installed
[    1.140110] Found 2 acpi root devices
[    1.140202] vgaarb: device added: PCI:0000:01:06.0,decodes=io+mem,owns=io+mem,locks=none
[    1.144004] vgaarb: loaded
[    1.148001] vgaarb: bridge control possible 0000:01:06.0
[    1.152212] SCSI subsystem initialized
[    1.156003] ACPI: bus type ATA registered
[    1.160024] libata version 3.00 loaded.
[    1.160112] PCI: Using ACPI for IRQ routing
[    1.164003] PCI: pci_cache_line_size set to 64 bytes
[    1.164074] e820: reserve RAM buffer [mem 0x0009d800-0x0009ffff]
[    1.164218] NetLabel: Initializing
[    1.168001] NetLabel:  domain hash size = 128
[    1.172000] NetLabel:  protocols = UNLABELED CIPSOv4
[    1.176016] NetLabel:  unlabeled traffic allowed by default
[    1.180037] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    1.184008] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 19
[    1.189418] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[    1.196067] Switching to clocksource hpet
[    1.204365] AppArmor: AppArmor Filesystem Enabled
[    1.209457] pnp: PnP ACPI init
[    1.212805] ACPI: bus type PNP registered
[    1.217436] pnp 00:00: [dma 4]
[    1.217494] pnp 00:00: Plug and Play ACPI device, IDs PNP0200 (active)
[    1.217563] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active)
[    1.217607] pnp 00:02: Plug and Play ACPI device, IDs PNP0800 (active)
[    1.217659] pnp 00:03: Plug and Play ACPI device, IDs PNP0c04 (active)
[    1.217862] system 00:04: [io  0x04d0-0x04d1] has been reserved
[    1.224077] system 00:04: [io  0x1100-0x117f] has been reserved
[    1.230286] system 00:04: [io  0x1180-0x11ff] has been reserved
[    1.236501] system 00:04: [io  0x0b78-0x0b7b] has been reserved
[    1.242715] system 00:04: [io  0x0190-0x0193] has been reserved
[    1.248927] system 00:04: [io  0xde00-0xdef7] has been reserved
[    1.255138] system 00:04: [io  0x0ca2-0x0ca3] has been reserved
[    1.261354] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active)
[    1.261444] pnp 00:05: Plug and Play ACPI device, IDs PNP0f13 (active)
[    1.262278] system 00:06: [mem 0x000e0000-0x000fffff] could not be reserved
[    1.269548] system 00:06: [mem 0x000c0000-0x000cafff] could not be reserved
[    1.276818] system 00:06: [mem 0xfec00000-0xfec00fff] could not be reserved
[    1.284085] system 00:06: [mem 0xffc00000-0xfff7ffff] has been reserved
[    1.291004] system 00:06: [mem 0xfee00000-0xfee00fff] has been reserved
[    1.297920] system 00:06: [mem 0xfff80000-0xffffffff] has been reserved
[    1.304836] system 00:06: [mem 0xfe000000-0xfe000fff] has been reserved
[    1.311750] system 00:06: [mem 0xfe001000-0xfe001fff] has been reserved
[    1.318665] system 00:06: [mem 0xfe002000-0xfe003fff] has been reserved
[    1.325580] system 00:06: [mem 0xfe004000-0xfe004fff] has been reserved
[    1.332495] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    1.332549] pnp 00:07: Plug and Play ACPI device, IDs PNP0303 (active)
[    1.333027] pnp 00:08: Plug and Play ACPI device, IDs PNP0501 (active)
[    1.333486] pnp 00:09: Plug and Play ACPI device, IDs PNP0501 (active)
[    1.333711] pnp 00:0a: Plug and Play ACPI device, IDs PNP0700 (disabled)
[    1.333856] pnp 00:0b: Plug and Play ACPI device, IDs PNP0501 (disabled)
[    1.333972] pnp 00:0c: Plug and Play ACPI device, IDs PNP0700 (disabled)
[    1.334043] pnp: PnP ACPI: found 13 devices
[    1.338505] ACPI: bus type PNP unregistered
[    1.351610] pci 0000:01:05.0: BAR 6: assigned [mem 0xe8180000-0xe81fffff pref]
[    1.359299] pci 0000:01:06.0: BAR 6: assigned [mem 0xe8120000-0xe813ffff pref]
[    1.366983] pci 0000:00:06.0: PCI bridge to [bus 01]
[    1.372231] pci 0000:00:06.0:   bridge window [io  0x2000-0x2fff]
[    1.378628] pci 0000:00:06.0:   bridge window [mem 0xe8100000-0xe81fffff]
[    1.385722] pci 0000:00:06.0:   bridge window [mem 0xf0000000-0xf7ffffff pref]
[    1.393415] pci 0000:00:0a.0: PCI bridge to [bus 02]
[    1.398670] pci 0000:00:0a.0:   bridge window [mem 0xe8200000-0xe82fffff]
[    1.405768] pci 0000:00:0b.0: PCI bridge to [bus 03]
[    1.411018] pci 0000:00:0b.0:   bridge window [mem 0xe8300000-0xe83fffff]
[    1.418117] pci 0000:0a:01.0: BAR 15: assigned [mem 0xf8300000-0xf83fffff pref]
[    1.425889] pci 0000:0b:01.0: BAR 6: assigned [mem 0xf8300000-0xf831ffff pref]
[    1.433572] pci 0000:0a:01.0: PCI bridge to [bus 0b-0f]
[    1.439083] pci 0000:0a:01.0:   bridge window [io  0x3000-0x3fff]
[    1.445473] pci 0000:0a:01.0:   bridge window [mem 0xf8100000-0xf81fffff]
[    1.452562] pci 0000:0a:01.0:   bridge window [mem 0xf8300000-0xf83fffff pref]
[    1.460244] pci 0000:0a:02.0: PCI bridge to [bus 10-14]
[    1.465759] pci 0000:0a:02.0:   bridge window [mem 0xf8200000-0xf82fffff]
[    1.472876] pci_bus 0000:00: resource 4 [io  0x0000-0x2fff]
[    1.472878] pci_bus 0000:00: resource 5 [io  0x4000-0xffff]
[    1.472881] pci_bus 0000:00: resource 6 [mem 0x80000000-0xe81fffff]
[    1.472884] pci_bus 0000:00: resource 7 [mem 0xe8200000-0xf7ffffff]
[    1.472886] pci_bus 0000:00: resource 8 [mem 0xf8500000-0xfcffffffff]
[    1.472889] pci_bus 0000:01: resource 0 [io  0x2000-0x2fff]
[    1.472892] pci_bus 0000:01: resource 1 [mem 0xe8100000-0xe81fffff]
[    1.472894] pci_bus 0000:01: resource 2 [mem 0xf0000000-0xf7ffffff pref]
[    1.472898] pci_bus 0000:02: resource 1 [mem 0xe8200000-0xe82fffff]
[    1.472900] pci_bus 0000:03: resource 1 [mem 0xe8300000-0xe83fffff]
[    1.472904] pci_bus 0000:0a: resource 4 [io  0x3000-0x3fff]
[    1.472906] pci_bus 0000:0a: resource 5 [mem 0xf8000000-0xf84fffff]
[    1.472909] pci_bus 0000:0b: resource 0 [io  0x3000-0x3fff]
[    1.472911] pci_bus 0000:0b: resource 1 [mem 0xf8100000-0xf81fffff]
[    1.472914] pci_bus 0000:0b: resource 2 [mem 0xf8300000-0xf83fffff pref]
[    1.472917] pci_bus 0000:10: resource 1 [mem 0xf8200000-0xf82fffff]
[    1.473163] NET: Registered protocol family 2
[    1.478287] TCP established hash table entries: 8192 (order: 5, 131072 bytes)
[    1.485845] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[    1.492770] TCP: Hash tables configured (established 8192 bind 8192)
[    1.505292] TCP: reno registered
[    1.508795] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    1.515025] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    1.521885] NET: Registered protocol family 1
[    1.526563] pci 0000:00:07.3: boot interrupts on device [1022:746b] already disabled
[    1.534777] pci 0000:00:0a.0: MSI quirk detected; subordinate MSI disabled
[    1.541954] pci 0000:00:0a.0: AMD8131 rev 12 detected; disabling PCI-X MMRBC
[    1.549314] pci 0000:00:0b.0: MSI quirk detected; subordinate MSI disabled
[    1.556492] pci 0000:00:0b.0: AMD8131 rev 12 detected; disabling PCI-X MMRBC
[    2.096126] pci 0000:01:06.0: Boot video device
[    2.096140] pci 0000:0a:01.0: MSI quirk detected; subordinate MSI disabled
[    2.103324] pci 0000:0a:01.0: AMD8131 rev 12 detected; disabling PCI-X MMRBC
[    2.110683] pci 0000:0a:02.0: MSI quirk detected; subordinate MSI disabled
[    2.117858] pci 0000:0a:02.0: AMD8131 rev 12 detected; disabling PCI-X MMRBC
[    2.125220] PCI: CLS 64 bytes, default 64
[    2.125329] Unpacking initramfs...
[    2.576094] Freeing initrd memory: 19648K (ffff880036cc0000 - ffff880037ff0000)
[    2.585458] Simple Boot Flag at 0x39 set to 0x1
[    2.591190] Scanning for low memory corruption every 60 seconds
[    2.597751] audit: initializing netlink socket (disabled)
[    2.603453] type=2000 audit(1371740356.600:1): initialized
[    2.634484] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    2.642014] VFS: Disk quotas dquot_6.5.2
[    2.646256] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.653673] msgmni has been set to 1990
[    2.658273] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    2.666177] io scheduler noop registered
[    2.670365] io scheduler deadline registered
[    2.674971] io scheduler cfq registered (default)
[    2.680445] ioapic: probe of 0000:00:0a.1 failed with error -22
[    2.686684] ioapic: probe of 0000:00:0b.1 failed with error -22
[    2.692933] ioapic: probe of 0000:0a:01.1 failed with error -22
[    2.699178] ioapic: probe of 0000:0a:02.1 failed with error -22
[    2.705568] GHES: HEST is not enabled!
[    2.709645] Serial: 8250/16550 driver, 32 ports, IRQ sharing disabled
[    2.736854] 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.763189] 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.769136] serial 00:0b: [io  0x03e8-0x03ef]
[    2.769259] serial 00:0b: [io  0x02e8-0x02ef]
[    2.769357] serial 00:0b: unable to assign resources
[    2.774587] serial: probe of 00:0b failed with error -16
[    2.782366] Non-volatile memory driver v1.3
[    2.786813] Linux agpgart interface v0.103
[    2.791423] libphy: Fixed MDIO Bus: probed
[    2.795818] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    2.806992] serio: i8042 KBD port at 0x60,0x64 irq 1
[    2.812239] serio: i8042 AUX port at 0x60,0x64 irq 12
[    2.817746] mousedev: PS/2 mouse device common for all mice
[    2.823755] rtc_cmos 00:01: RTC can wake from S4
[    2.828832] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
[    2.835266] rtc_cmos 00:01: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
[    2.843389] cpuidle: using governor ladder
[    2.847745] cpuidle: using governor menu
[    2.851934] ledtrig-cpu: registered to indicate activity on CPUs
[    2.858216] EFI Variables Facility v0.08 2004-May-17
[    2.863469] hidraw: raw HID events driver (C) Jiri Kosina
[    2.869320] TCP: cubic registered
[    2.873085] NET: Registered protocol family 10
[    2.878083] Key type dns_resolver registered
[    2.883233] PM: Checking hibernation image partition /dev/sda1
[    2.896287] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[    3.153438] PM: Hibernation image not present or could not be loaded.
[    3.153459] registered taskstats version 1
[    3.158465]   Magic number: 1:430:991
[    3.162389] machinecheck machinecheck4: hash matches
[    3.169948] Freeing unused kernel memory: 1268K (ffffffff81aa9000 - ffffffff81be6000)
[    3.178266] Write protecting the kernel read-only data: 10240k
[    3.186252] Freeing unused kernel memory: 476K (ffff880001589000 - ffff880001600000)
[    3.199991] Freeing unused kernel memory: 1584K (ffff880001874000 - ffff880001a00000)
[    3.278293] pata_amd 0000:00:07.1: version 0.4.1
[    3.278747] scsi0 : pata_amd
[    3.282131] scsi1 : pata_amd
[    3.285338] ata1: PATA max UDMA/133 cmd 0x1f0 ctl 0x3f6 bmdma 0x1020 irq 14
[    3.292593] ata2: PATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0x1028 irq 15
[    3.309264] sata_sil 0000:01:05.0: version 2.4
[    3.309801] scsi2 : sata_sil
[    3.313070] scsi3 : sata_sil
[    3.316313] ata3: SATA max UDMA/100 mmio m512@0xe8112000 tf 0xe8112080 irq 16
[    3.323744] ata4: SATA max UDMA/100 mmio m512@0xe8112000 tf 0xe81120c0 irq 16
[    3.351438] hp_sw: device handler registered
[    3.365531] rdac: device handler registered
[    3.379257] emc: device handler registered
[    3.388527] udev: starting version 147
[    3.428543] ACPI: bus type USB registered
[    3.432942] usbcore: registered new interface driver usbfs
[    3.438765] usbcore: registered new interface driver hub
[    3.447735] usbcore: registered new device driver usb
[    3.459550] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.464555] ata1.00: ATAPI: DV-28E-N, 1.6A, max UDMA/33
[    3.475720] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    3.480422] ata1.00: configured for UDMA/33
[    3.485265] scsi 0:0:0:0: CD-ROM            TEAC     DV-28E-N         1.6A PQ: 0 ANSI: 5
[    3.486009] ata2: port disabled--ignoring
[    3.495628] ohci_hcd 0000:01:00.0: OHCI Host Controller
[    3.501165] ohci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
[    3.509095] ohci_hcd 0000:01:00.0: irq 19, io mem 0xe8110000
[    3.570076] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
[    3.577155] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.584831] usb usb1: Product: OHCI Host Controller
[    3.589971] usb usb1: Manufacturer: Linux 3.9.0bisect+ ohci_hcd
[    3.596220] usb usb1: SerialNumber: 0000:01:00.0
[    3.601263] hub 1-0:1.0: USB hub found
[    3.605283] hub 1-0:1.0: 3 ports detected
[    3.609785] ohci_hcd 0000:01:00.1: OHCI Host Controller
[    3.615287] ohci_hcd 0000:01:00.1: new USB bus registered, assigned bus number 2
[    3.623146] ohci_hcd 0000:01:00.1: irq 19, io mem 0xe8111000
[    3.686051] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[    3.693121] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.700786] usb usb2: Product: OHCI Host Controller
[    3.705930] usb usb2: Manufacturer: Linux 3.9.0bisect+ ohci_hcd
[    3.712130] usb usb2: SerialNumber: 0000:01:00.1
[    3.717132] hub 2-0:1.0: USB hub found
[    3.721151] hub 2-0:1.0: 3 ports detected
[    4.100038] usb 2-2: new full-speed USB device number 2 using ohci_hcd
[    4.311978] usb 2-2: New USB device found, idVendor=04b4, idProduct=6560
[    4.318982] usb 2-2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.328035] hub 2-2:1.0: USB hub found
[    4.333965] hub 2-2:1.0: 4 ports detected
[    4.380060] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[    4.392504] ata3.00: ATA-7: Maxtor 6Y120M0, YAR51EW0, max UDMA/133
[    4.398967] ata3.00: 234375000 sectors, multi 0: LBA 
[    4.420467] ata3.00: configured for UDMA/100
[    4.425286] scsi 2:0:0:0: Direct-Access     ATA      Maxtor 6Y120M0   YAR5 PQ: 0 ANSI: 5
[    4.434199] sd 2:0:0:0: [sda] 234375000 512-byte logical blocks: (120 GB/111 GiB)
[    4.442392] sd 2:0:0:0: [sda] Write Protect is off
[    4.447458] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    4.447527] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    4.532034] usb 2-3: new full-speed USB device number 3 using ohci_hcd
[    4.534170]  sda: sda1 sda2 sda4 < sda5 sda6 sda7 sda8 sda9 >
[    4.535057] sd 2:0:0:0: [sda] Attached SCSI disk
[    4.752072] ata4: SATA link down (SStatus 0 SControl 310)
[    4.759904] usb 2-3: New USB device found, idVendor=04b4, idProduct=6560
[    4.766940] usb 2-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.775939] hub 2-3:1.0: USB hub found
[    4.781890] hub 2-3:1.0: 4 ports detected
[    6.054750] EXT4-fs (sda2): mounting ext3 file system using the ext4 subsystem
[    6.077544] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: acl,user_xattr
[    6.210756] EXT4-fs (sda2): re-mounted. Opts: acl,user_xattr
[    6.610059] BUG: Bad page state in process run-init  pfn:1e05f
[    6.616234] page:ffffea00006914c8 count:0 mapcount:0 mapping:          (null) index:0xd
[    6.624705] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    6.631677] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    6.648375] Pid: 1, comm: run-init Not tainted 3.9.0bisect+ #1484
[    6.654749] Call Trace:
[    6.657456]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    6.662781]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    6.669088]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    6.675296]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    6.681862]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    6.687716]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    6.693744]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    6.700763]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    6.707059]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    6.712257]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    6.717766]  [<ffffffff81183f48>] iput+0x48/0x60
[    6.722655]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    6.728335]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    6.734102]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    6.739523]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    6.745812] Disabling lock debugging due to kernel taint
[    6.751404] BUG: Bad page state in process run-init  pfn:3e7e6
[    6.757535] page:ffffea0000daba50 count:0 mapcount:0 mapping:          (null) index:0xc
[    6.766000] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    6.773004] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    6.789694] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    6.797045] Call Trace:
[    6.799749]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    6.805081]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    6.811376]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    6.817578]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    6.824146]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    6.830018]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    6.836053]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    6.843055]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    6.849375]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    6.854530]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    6.860050]  [<ffffffff81183f48>] iput+0x48/0x60
[    6.864945]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    6.870631]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    6.876396]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    6.881823]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    6.888150] BUG: Bad page state in process run-init  pfn:1e05e
[    6.894280] page:ffffea0000691490 count:0 mapcount:0 mapping:          (null) index:0xb
[    6.902762] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    6.909741] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    6.926459] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    6.933800] Call Trace:
[    6.936502]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    6.941844]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    6.948173]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    6.954384]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    6.960956]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    6.966815]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    6.972847]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    6.979851]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    6.986145]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    6.991302]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    6.996802]  [<ffffffff81183f48>] iput+0x48/0x60
[    7.001716]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    7.007395]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    7.013173]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    7.018595]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    7.024895] BUG: Bad page state in process run-init  pfn:3e7e5
[    7.031010] page:ffffea0000daba18 count:0 mapcount:0 mapping:          (null) index:0xa
[    7.039471] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    7.046447] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    7.063145] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    7.070495] Call Trace:
[    7.073211]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    7.078538]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    7.084850]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    7.091061]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    7.097627]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    7.103501]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    7.109535]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    7.116548]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    7.122846]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    7.127998]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    7.133511]  [<ffffffff81183f48>] iput+0x48/0x60
[    7.138400]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    7.144107]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    7.149885]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    7.155299]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    7.161598] BUG: Bad page state in process run-init  pfn:1e05d
[    7.167723] page:ffffea0000691458 count:0 mapcount:0 mapping:          (null) index:0x9
[    7.176240] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    7.183245] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    7.199960] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    7.207319] Call Trace:
[    7.210031]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    7.215370]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    7.221684]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    7.227886]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    7.234453]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    7.240349]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    7.246378]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    7.253387]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    7.259688]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    7.264846]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    7.270348]  [<ffffffff81183f48>] iput+0x48/0x60
[    7.275242]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    7.280922]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    7.286690]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    7.292105]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    7.298394] BUG: Bad page state in process run-init  pfn:3e7e4
[    7.304529] page:ffffea0000dab9e0 count:0 mapcount:0 mapping:          (null) index:0x8
[    7.312997] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    7.319957] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    7.342309] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    7.349653] Call Trace:
[    7.352357]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    7.357706]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    7.364014]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    7.370229]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    7.376804]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    7.382669]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    7.388704]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    7.395704]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    7.401997]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    7.407148]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    7.412664]  [<ffffffff81183f48>] iput+0x48/0x60
[    7.417551]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    7.423237]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    7.428999]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    7.434410]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    7.440721] BUG: Bad page state in process run-init  pfn:1e05c
[    7.446844] page:ffffea0000691420 count:0 mapcount:0 mapping:          (null) index:0x7
[    7.455324] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    7.462298] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    7.478987] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    7.486338] Call Trace:
[    7.489048]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    7.494383]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    7.500695]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    7.506896]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    7.513462]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    7.519328]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    7.525367]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    7.532385]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    7.538679]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    7.543832]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    7.549338]  [<ffffffff81183f48>] iput+0x48/0x60
[    7.554246]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    7.559933]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    7.565710]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    7.571125]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    7.577422] BUG: Bad page state in process run-init  pfn:3e7e3
[    7.583550] page:ffffea0000dab9a8 count:0 mapcount:0 mapping:          (null) index:0x6
[    7.592025] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    7.598975] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    7.615648] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    7.622990] Call Trace:
[    7.625698]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    7.631026]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    7.637336]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    7.643539]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    7.650111]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    7.655963]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    7.662001]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    7.669008]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    7.675307]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    7.680456]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    7.685962]  [<ffffffff81183f48>] iput+0x48/0x60
[    7.690854]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    7.696528]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    7.702293]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    7.707721]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    7.714014] BUG: Bad page state in process run-init  pfn:1e05b
[    7.720159] page:ffffea00006913e8 count:0 mapcount:0 mapping:          (null) index:0x5
[    7.728626] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    7.735584] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    7.752288] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    7.759630] Call Trace:
[    7.762326]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    7.767665]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    7.773972]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    7.780216]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    7.786781]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    7.792653]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    7.798681]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    7.805675]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    7.811967]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    7.817125]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    7.822632]  [<ffffffff81183f48>] iput+0x48/0x60
[    7.827520]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    7.833209]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    7.838993]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    7.844408]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    7.850707] BUG: Bad page state in process run-init  pfn:3e7e2
[    7.856841] page:ffffea0000dab970 count:0 mapcount:0 mapping:          (null) index:0x4
[    7.865310] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    7.872297] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    7.889024] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    7.896386] Call Trace:
[    7.899090]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    7.904422]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    7.910714]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    7.916929]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    7.923483]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    7.929360]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    7.935399]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    7.942413]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    7.948701]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    7.953864]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    7.959378]  [<ffffffff81183f48>] iput+0x48/0x60
[    7.964275]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    7.969960]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    7.975731]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    7.981162]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    7.987452] BUG: Bad page state in process run-init  pfn:1e05a
[    7.993586] page:ffffea00006913b0 count:0 mapcount:0 mapping:          (null) index:0x3
[    8.002051] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    8.009012] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    8.025685] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    8.033034] Call Trace:
[    8.035722]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    8.041044]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    8.047347]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    8.053559]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    8.060125]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    8.065991]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    8.072033]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    8.079036]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    8.085346]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    8.090494]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    8.096001]  [<ffffffff81183f48>] iput+0x48/0x60
[    8.100897]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    8.106584]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    8.112370]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    8.117793]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    8.124115] BUG: Bad page state in process run-init  pfn:1e059
[    8.130244] page:ffffea0000691378 count:0 mapcount:0 mapping:          (null) index:0x2
[    8.138709] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    8.145689] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    8.162381] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    8.175440] Call Trace:
[    8.178149]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    8.183473]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    8.189790]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    8.195997]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    8.202555]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    8.208439]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    8.214471]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    8.221486]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    8.227774]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    8.232935]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    8.238449]  [<ffffffff81183f48>] iput+0x48/0x60
[    8.243346]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    8.249031]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    8.254805]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    8.260258]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    8.266568] BUG: Bad page state in process run-init  pfn:3e7e1
[    8.272693] page:ffffea0000dab938 count:0 mapcount:0 mapping:          (null) index:0x1
[    8.281174] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    8.288196] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    8.304869] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    8.312254] Call Trace:
[    8.314949]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    8.320317]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    8.326619]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    8.332839]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    8.339403]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    8.345261]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    8.351302]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    8.358306]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    8.364616]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    8.369775]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    8.375274]  [<ffffffff81183f48>] iput+0x48/0x60
[    8.380214]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    8.385890]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    8.391664]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    8.397093]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    8.403393] BUG: Bad page state in process run-init  pfn:1e058
[    8.409525] page:ffffea0000691340 count:0 mapcount:0 mapping:          (null) index:0x0
[    8.417996] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    8.424968] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    8.441624] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    8.448966] Call Trace:
[    8.451662]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    8.457019]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    8.463316]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    8.469534]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    8.476134]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    8.481998]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    8.488043]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    8.495053]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    8.501344]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    8.506504]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    8.512026]  [<ffffffff81183f48>] iput+0x48/0x60
[    8.516925]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    8.522604]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    8.528390]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    8.533813]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    8.540172] BUG: Bad page state in process run-init  pfn:1e063
[    8.546318] page:ffffea00006915a8 count:0 mapcount:0 mapping:          (null) index:0x15
[    8.554874] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    8.561844] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    8.578547] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    8.585893] Call Trace:
[    8.588601]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    8.593932]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    8.600277]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    8.606492]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    8.613069]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    8.618930]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    8.624961]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    8.631957]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    8.638257]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    8.643406]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    8.648911]  [<ffffffff81183f48>] iput+0x48/0x60
[    8.653805]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    8.659497]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    8.665277]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    8.670699]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    8.677004] BUG: Bad page state in process run-init  pfn:3e7ea
[    8.683130] page:ffffea0000dabb30 count:0 mapcount:0 mapping:          (null) index:0x14
[    8.691697] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    8.698677] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    8.715368] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    8.722706] Call Trace:
[    8.725413]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    8.730745]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    8.737061]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    8.743285]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    8.749859]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    8.755725]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    8.761770]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    8.768778]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    8.775079]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    8.780273]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    8.785780]  [<ffffffff81183f48>] iput+0x48/0x60
[    8.790668]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    8.796367]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    8.802134]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    8.807553]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    8.813845] BUG: Bad page state in process run-init  pfn:1e062
[    8.819960] page:ffffea0000691570 count:0 mapcount:0 mapping:          (null) index:0x13
[    8.828543] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    8.835503] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    8.852223] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    8.859575] Call Trace:
[    8.862288]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    8.867620]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    8.873914]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    8.880136]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    8.886698]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    8.892559]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    8.898589]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    8.905595]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    8.911893]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    8.917065]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    8.922569]  [<ffffffff81183f48>] iput+0x48/0x60
[    8.927456]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    8.933142]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    8.938916]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    8.944364]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    8.950662] BUG: Bad page state in process run-init  pfn:3e7e9
[    8.956777] page:ffffea0000dabaf8 count:0 mapcount:0 mapping:          (null) index:0x12
[    8.965346] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    8.972320] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    8.989022] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    8.996395] Call Trace:
[    9.004801]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    9.010147]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    9.016439]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    9.022652]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    9.029214]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    9.035073]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    9.041116]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    9.048124]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    9.054428]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    9.059588]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    9.065111]  [<ffffffff81183f48>] iput+0x48/0x60
[    9.070009]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    9.075687]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    9.081466]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    9.086881]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    9.093186] BUG: Bad page state in process run-init  pfn:1e061
[    9.099313] page:ffffea0000691538 count:0 mapcount:0 mapping:          (null) index:0x11
[    9.107878] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    9.114850] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    9.131549] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    9.138890] Call Trace:
[    9.141582]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    9.146909]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    9.153205]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    9.159409]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    9.165962]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    9.171830]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    9.177869]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    9.184874]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    9.191174]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    9.196349]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    9.201848]  [<ffffffff81183f48>] iput+0x48/0x60
[    9.206738]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    9.212423]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    9.218186]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    9.223615]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    9.229917] BUG: Bad page state in process run-init  pfn:3e7e8
[    9.236055] page:ffffea0000dabac0 count:0 mapcount:0 mapping:          (null) index:0x10
[    9.244606] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    9.251572] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    9.268282] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    9.275629] Call Trace:
[    9.278337]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    9.283663]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    9.289965]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    9.296189]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    9.302750]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    9.308605]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    9.314632]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    9.321630]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    9.327918]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    9.333082]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    9.338595]  [<ffffffff81183f48>] iput+0x48/0x60
[    9.343491]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    9.349176]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    9.354950]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    9.360376]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    9.366677] BUG: Bad page state in process run-init  pfn:1e060
[    9.372802] page:ffffea0000691500 count:0 mapcount:0 mapping:          (null) index:0xf
[    9.381276] page flags: 0x3ff0000000004c(referenced|uptodate|active)
[    9.388275] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    9.404951] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    9.412336] Call Trace:
[    9.415027]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    9.420368]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    9.426667]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    9.432871]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    9.439437]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    9.445303]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    9.451342]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    9.458347]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    9.464650]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    9.469798]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    9.475304]  [<ffffffff81183f48>] iput+0x48/0x60
[    9.480237]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    9.485922]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    9.491687]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    9.497124]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[    9.503418] BUG: Bad page state in process run-init  pfn:3e7e7
[    9.509542] page:ffffea0000daba88 count:0 mapcount:0 mapping:          (null) index:0xe
[    9.518020] page flags: 0xbff0000000004c(referenced|uptodate|active)
[    9.525000] Modules linked in: ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
[    9.541681] Pid: 1, comm: run-init Tainted: G    B        3.9.0bisect+ #1484
[    9.549025] Call Trace:
[    9.551720]  [<ffffffff811065ce>] bad_page+0xbe/0x140
[    9.557064]  [<ffffffff81106b59>] free_pages_prepare+0x109/0x130
[    9.563355]  [<ffffffff8110a98b>] free_hot_cold_page+0x3b/0x190
[    9.569566]  [<ffffffff8110afd3>] free_hot_cold_page_list+0x73/0xa0
[    9.576150]  [<ffffffff8110f3a9>] release_pages+0x1e9/0x220
[    9.582023]  [<ffffffff8110f721>] __pagevec_release+0x21/0x30
[    9.588077]  [<ffffffff81110b84>] truncate_inode_pages_range+0x354/0x4c0
[    9.595068]  [<ffffffff81110d70>] truncate_inode_pages+0x10/0x20
[    9.601368]  [<ffffffff81183d7e>] evict+0x18e/0x1b0
[    9.606535]  [<ffffffff81183e69>] iput_final+0xc9/0x160
[    9.612047]  [<ffffffff81183f48>] iput+0x48/0x60
[    9.616941]  [<ffffffff8117a0b4>] do_unlinkat+0x214/0x240
[    9.622620]  [<ffffffff8116f941>] ? sys_newlstat+0x31/0x50
[    9.628405]  [<ffffffff8117a0f1>] sys_unlink+0x11/0x20
[    9.633819]  [<ffffffff815829e9>] system_call_fastpath+0x16/0x1b
[   10.924241] udev: starting version 147
[   11.311187] input: PC Speaker as /devices/platform/pcspkr/input/input1
[   11.445273] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input2
[   11.454543] ACPI: Power Button [PWRB]
[   11.458579] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
[   11.466873] ACPI: Power Button [PWRF]
[   11.648002] microcode: AMD CPU family 0xf not supported
[   11.654768] microcode: AMD CPU family 0xf not supported
[   11.672846] microcode: AMD CPU family 0xf not supported
[   11.679115] microcode: AMD CPU family 0xf not supported
[   11.685588] microcode: AMD CPU family 0xf not supported
[   11.691788] microcode: AMD CPU family 0xf not supported
[   11.697963] microcode: AMD CPU family 0xf not supported
[   11.712726] microcode: AMD CPU family 0xf not supported
[   11.859908] sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
[   11.866405] cdrom: Uniform CD-ROM driver Revision: 3.20
[   11.872135] sr 0:0:0:0: Attached scsi CD-ROM sr0
[   11.982964] pci 0000:00:07.3: AMD GPIO region 0xc0b0 already in use!
[   12.152854] MCE: In-kernel MCE decoding enabled.
[   12.182546] k8temp 0000:00:18.3: Temperature readouts might be wrong - check erratum #141
[   12.191376] k8temp 0000:00:19.3: Temperature readouts might be wrong - check erratum #141
[   12.200149] k8temp 0000:00:1a.3: Temperature readouts might be wrong - check erratum #141
[   12.208894] k8temp 0000:00:1b.3: Temperature readouts might be wrong - check erratum #141
[   12.286166] AMD768 RNG detected
[   12.474247] sr 0:0:0:0: Attached scsi generic sg0 type 5
[   12.479941] sd 2:0:0:0: Attached scsi generic sg1 type 0
[   12.579401] EDAC MC: Ver: 3.0.0
[   12.790107] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[   12.854706] AMD64 EDAC driver v3.4.0
[   12.858641] EDAC amd64: DRAM ECC disabled.
[   12.863057] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   12.863057]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   12.863057]  (Note that use of the override may cause unknown side effects.)
[   12.889298] EDAC amd64: DRAM ECC disabled.
[   12.893839] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   12.893839]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   12.893839]  (Note that use of the override may cause unknown side effects.)
[   12.920231] EDAC amd64: DRAM ECC disabled.
[   12.924665] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   12.924665]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   12.924665]  (Note that use of the override may cause unknown side effects.)
[   12.950677] EDAC amd64: DRAM ECC disabled.
[   12.955042] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   12.955042]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   12.955042]  (Note that use of the override may cause unknown side effects.)
[   12.984963] AMD64 EDAC driver v3.4.0
[   12.989164] EDAC amd64: DRAM ECC disabled.
[   12.993575] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   12.993575]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   12.993575]  (Note that use of the override may cause unknown side effects.)
[   13.019616] EDAC amd64: DRAM ECC disabled.
[   13.024150] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.024150]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.024150]  (Note that use of the override may cause unknown side effects.)
[   13.050483] EDAC amd64: DRAM ECC disabled.
[   13.054873] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.054873]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.054873]  (Note that use of the override may cause unknown side effects.)
[   13.080927] EDAC amd64: DRAM ECC disabled.
[   13.085419] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.085419]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.085419]  (Note that use of the override may cause unknown side effects.)
[   13.113801] AMD64 EDAC driver v3.4.0
[   13.117949] EDAC amd64: DRAM ECC disabled.
[   13.122444] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.122444]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.122444]  (Note that use of the override may cause unknown side effects.)
[   13.148559] EDAC amd64: DRAM ECC disabled.
[   13.153275] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.153275]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.153275]  (Note that use of the override may cause unknown side effects.)
[   13.180257] EDAC amd64: DRAM ECC disabled.
[   13.184714] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.184714]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.184714]  (Note that use of the override may cause unknown side effects.)
[   13.210555] EDAC amd64: DRAM ECC disabled.
[   13.214933] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.214933]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.214933]  (Note that use of the override may cause unknown side effects.)
[   13.240571] shpchp 0000:00:0a.0: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
[   13.248858] shpchp 0000:00:0a.0: Cannot reserve MMIO region
[   13.254748] shpchp 0000:00:0b.0: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
[   13.263073] shpchp 0000:00:0b.0: Cannot reserve MMIO region
[   13.268974] shpchp 0000:0a:01.0: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
[   13.277423] shpchp 0000:0a:01.0: Cannot reserve MMIO region
[   13.283564] shpchp 0000:0a:02.0: HPC vendor_id 1022 device_id 7450 ss_vid 0 ss_did 0
[   13.291805] shpchp 0000:0a:02.0: Cannot reserve MMIO region
[   13.297830] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[   13.305123] AMD64 EDAC driver v3.4.0
[   13.309310] EDAC amd64: DRAM ECC disabled.
[   13.313852] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.313852]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.313852]  (Note that use of the override may cause unknown side effects.)
[   13.339812] EDAC amd64: DRAM ECC disabled.
[   13.344350] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.344350]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.344350]  (Note that use of the override may cause unknown side effects.)
[   13.344481] kvm: Nested Virtualization enabled
[   13.374691] EDAC amd64: DRAM ECC disabled.
[   13.379524] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.379524]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.379524]  (Note that use of the override may cause unknown side effects.)
[   13.405670] EDAC amd64: DRAM ECC disabled.
[   13.410089] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
[   13.410089]  Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
[   13.410089]  (Note that use of the override may cause unknown side effects.)
[   13.695083] powernow-k8: fid 0x10 (2400 MHz), vid 0x8
[   13.700431] powernow-k8: fid 0xe (2200 MHz), vid 0xa
[   13.705670] powernow-k8: fid 0xc (2000 MHz), vid 0xc
[   13.710985] powernow-k8: fid 0xa (1800 MHz), vid 0xe
[   13.716220] powernow-k8: fid 0x2 (1000 MHz), vid 0x12
[   13.721633] powernow-k8: fid 0x10 (2400 MHz), vid 0x8
[   13.726971] powernow-k8: fid 0xe (2200 MHz), vid 0xa
[   13.732244] powernow-k8: fid 0xc (2000 MHz), vid 0xc
[   13.737472] powernow-k8: fid 0xa (1800 MHz), vid 0xe
[   13.742709] powernow-k8: fid 0x2 (1000 MHz), vid 0x12
[   13.748623] powernow-k8: fid 0x10 (2400 MHz), vid 0x8
[   13.753958] powernow-k8: fid 0xe (2200 MHz), vid 0xa
[   13.759190] powernow-k8: fid 0xc (2000 MHz), vid 0xc
[   13.764441] powernow-k8: fid 0xa (1800 MHz), vid 0xe
[   13.769684] powernow-k8: fid 0x2 (1000 MHz), vid 0x12
[   13.775887] powernow-k8: fid 0x10 (2400 MHz), vid 0x8
[   13.781212] powernow-k8: fid 0xe (2200 MHz), vid 0xa
[   13.786458] powernow-k8: fid 0xc (2000 MHz), vid 0xc
[   13.791702] powernow-k8: fid 0xa (1800 MHz), vid 0xe
[   13.796935] powernow-k8: fid 0x2 (1000 MHz), vid 0x12
[   13.803132] powernow-k8: Found 2 AMD Engineering Sample (8 cpu cores) (version 2.20.00)
[   14.061357] pps_core: LinuxPPS API ver. 1 registered
[   14.066694] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[   14.186480] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[   14.194001] e1000: Copyright (c) 1999-2006 Intel Corporation.
[   14.246463] PTP clock support registered
[   14.472111] floppy0: no floppy controllers found
[   14.650539] e1000 0000:0b:01.0 eth0: (PCI:66MHz:32-bit) 00:1b:21:22:77:2e
[   14.657874] e1000 0000:0b:01.0 eth0: Intel(R) PRO/1000 Network Connection
[   14.666374] tg3.c:v3.130 (February 14, 2013)
[   16.164980] tg3 0000:02:01.0 eth1: Tigon3 [partno(3C996B-T) rev 0105] (PCIX:133MHz:64-bit) MAC address 00:04:76:f1:0a:21
[   16.176375] tg3 0000:02:01.0 eth1: attached PHY is 5701 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   16.186809] tg3 0000:02:01.0 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[0]
[   16.195527] tg3 0000:02:01.0 eth1: dma_rwctrl[76db000f] dma_mask[64-bit]
[   16.241352] tg3 0000:03:02.0 eth2: Tigon3 [partno(BCM95704A6) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:00:1a:19:c8:e2
[   16.252904] tg3 0000:03:02.0 eth2: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   16.263137] tg3 0000:03:02.0 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   16.271437] tg3 0000:03:02.0 eth2: dma_rwctrl[769f4000] dma_mask[64-bit]
[   16.310173] tg3 0000:03:02.1 eth3: Tigon3 [partno(BCM95704A6) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:00:1a:19:c8:e1
[   16.321723] tg3 0000:03:02.1 eth3: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   16.331955] tg3 0000:03:02.1 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   16.340259] tg3 0000:03:02.1 eth3: dma_rwctrl[769f4000] dma_mask[64-bit]
[   16.389533] tg3 0000:10:02.0 eth4: Tigon3 [partno(BCM95704A6) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:00:1a:19:c9:5e
[   16.401174] tg3 0000:10:02.0 eth4: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   16.411494] tg3 0000:10:02.0 eth4: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   16.419879] tg3 0000:10:02.0 eth4: dma_rwctrl[769f4000] dma_mask[64-bit]
[   16.458182] tg3 0000:10:02.1 eth5: Tigon3 [partno(BCM95704A6) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:00:1a:19:c9:5d
[   16.469807] tg3 0000:10:02.1 eth5: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[   16.480099] tg3 0000:10:02.1 eth5: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   16.488443] tg3 0000:10:02.1 eth5: dma_rwctrl[769f4000] dma_mask[64-bit]
[   17.504134] floppy0: no floppy controllers found
[   17.597401] Adding 2104476k swap on /dev/sda1.  Priority:-1 extents:1 across:2104476k 
[   18.575678] device-mapper: uevent: version 1.0.3
[   18.580924] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
[   19.312988] loop: module loaded
[   19.387068] EXT4-fs (sda9): mounting ext3 file system using the ext4 subsystem
[   19.420221] EXT4-fs (sda9): mounted filesystem with ordered data mode. Opts: acl,user_xattr
[   21.326813] fuse init (API version 7.21)
[   31.879046] Bridge firewalling registered
[   32.158581] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   32.161746] device eth1 entered promiscuous mode
[   32.172462] IPv6: ADDRCONF(NETDEV_UP): br0: link is not ready
[   35.150013] tg3 0000:02:01.0 eth1: Link is up at 1000 Mbps, full duplex
[   35.150034] tg3 0000:02:01.0 eth1: Flow control is on for TX and on for RX
[   35.150069] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   35.150176] br0: port 1(eth1) entered forwarding state
[   35.150190] br0: port 1(eth1) entered forwarding state
[   35.150347] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
[   35.816262] NET: Registered protocol family 17
[   40.087350] RPC: Registered named UNIX socket transport module.
[   40.087362] RPC: Registered udp transport module.
[   40.087365] RPC: Registered tcp transport module.
[   40.087369] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   40.334464] FS-Cache: Loaded
[   40.525010] FS-Cache: Netfs 'nfs' registered for caching
[   42.187119] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found
[  299.416505] start.sh (4565): dropped kernel caches: 3

[-- Attachment #4: demon.zoneinfo --]
[-- Type: text/plain, Size: 5953 bytes --]

Node 0, zone      DMA
  pages free     1355
        min      197
        low      246
        high     295
        scanned  0
        spanned  4095
        present  3996
        managed  3974
    nr_free_pages 1355
    nr_inactive_anon 0
    nr_active_anon 2560
    nr_inactive_file 1
    nr_active_file 6
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 0
    nr_mapped    0
    nr_file_pages 7
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 36
    nr_slab_unreclaimable 8
    nr_page_table_pages 0
    nr_kernel_stack 0
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 4462
    nr_vmscan_immediate_reclaim 63470
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     0
    nr_dirtied   2560
    nr_written   6987
    numa_hit     7143
    numa_miss    17872
    numa_foreign 0
    numa_interleave 0
    numa_local   7142
    numa_other   17873
    nr_anon_transparent_hugepages 0
    nr_free_cma  0
        protection: (0, 473, 473, 473)
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 1
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 2
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 3
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 4
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 5
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 6
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 7
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
  all_unreclaimable: 0
  start_pfn:         1
  inactive_ratio:    1
Node 0, zone    DMA32
  pages free     25252
        min      6030
        low      7537
        high     9045
        scanned  0
        spanned  126976
        present  126976
        managed  122160
    nr_free_pages 25252
    nr_inactive_anon 2621
    nr_active_anon 39339
    nr_inactive_file 12073
    nr_active_file 13266
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 40398
    nr_mapped    2007
    nr_file_pages 25707
    nr_dirty     347
    nr_writeback 15
    nr_slab_reclaimable 2696
    nr_slab_unreclaimable 2716
    nr_page_table_pages 795
    nr_kernel_stack 191
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 41234
    nr_vmscan_immediate_reclaim 33718
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     24
    nr_dirtied   169786
    nr_written   201945
    numa_hit     3969563
    numa_miss    943832
    numa_foreign 758
    numa_interleave 7562
    numa_local   3961637
    numa_other   951758
    nr_anon_transparent_hugepages 1
    nr_free_cma  0
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 31
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 1
              count: 45
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 2
              count: 81
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 3
              count: 36
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 4
              count: 20
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 5
              count: 6
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 6
              count: 65
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 7
              count: 32
              high:  186
              batch: 31
  vm stats threshold: 24
  all_unreclaimable: 0
  start_pfn:         4096
  inactive_ratio:    1
Node 1, zone    DMA32
  pages free     8032
        min      6432
        low      8040
        high     9648
        scanned  0
        spanned  131072
        present  131072
        managed  129426
    nr_free_pages 8032
    nr_inactive_anon 3087
    nr_active_anon 40696
    nr_inactive_file 21311
    nr_active_file 24356
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 42076
    nr_mapped    2777
    nr_file_pages 47383
    nr_dirty     385
    nr_writeback 14
    nr_slab_reclaimable 3644
    nr_slab_unreclaimable 2976
    nr_page_table_pages 952
    nr_kernel_stack 99
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 17505
    nr_vmscan_immediate_reclaim 1273
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     26
    nr_dirtied   218809
    nr_written   215866
    numa_hit     8403202
    numa_miss    758
    numa_foreign 961741
    numa_interleave 7529
    numa_local   8401965
    numa_other   1995
    nr_anon_transparent_hugepages 0
    nr_free_cma  0
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 162
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 1
              count: 175
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 2
              count: 28
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 3
              count: 74
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 4
              count: 97
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 5
              count: 74
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 6
              count: 105
              high:  186
              batch: 31
  vm stats threshold: 24
    cpu: 7
              count: 46
              high:  186
              batch: 31
  vm stats threshold: 24
  all_unreclaimable: 0
  start_pfn:         131072
  inactive_ratio:    1

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Thu 20-06-13 17:12:01, Michal Hocko wrote:
> On Thu 20-06-13 18:11:38, Glauber Costa wrote:
> [...]
> > > [84091.219056] ------------[ cut here ]------------
> > > [84091.220015] kernel BUG at mm/list_lru.c:42!
> > > [84091.220015] invalid opcode: 0000 [#1] SMP 
> > > [84091.220015] Modules linked in: edd nfsv3 nfs_acl nfs fscache lockd sunrpc af_packet bridge stp llc cpufreq_conservative cpufreq_userspace cpufreq_powersave fuse loop dm_mod powernow_k8 tg3 kvm_amd kvm ptp e1000 pps_core shpchp edac_core i2c_amd756 amd_rng pci_hotplug k8temp sg i2c_amd8111 edac_mce_amd serio_raw sr_mod pcspkr cdrom button ohci_hcd ehci_hcd usbcore usb_common processor thermal_sys scsi_dh_emc scsi_dh_rdac scsi_dh_hp_sw scsi_dh ata_generic sata_sil pata_amd
> > > [84091.220015] CPU 1 
> > > [84091.220015] Pid: 32545, comm: rm Not tainted 3.9.0mmotmdebugging1+ #1472 AMD A8440/WARTHOG
> > > [84091.220015] RIP: 0010:[<ffffffff81127fff>]  [<ffffffff81127fff>] list_lru_del+0xcf/0xe0
> > > [84091.220015] RSP: 0018:ffff88001de85df8  EFLAGS: 00010286
> > > [84091.220015] RAX: ffffffffffffffff RBX: ffff88001e1ce2c0 RCX: 0000000000000002
> > > [84091.220015] RDX: ffff88001e1ce2c8 RSI: ffff8800087f4220 RDI: ffff88001e1ce2c0
> > > [84091.220015] RBP: ffff88001de85e18 R08: 0000000000000000 R09: 0000000000000000
> > > [84091.220015] R10: ffff88001d539128 R11: ffff880018234882 R12: ffff8800087f4220
> > > [84091.220015] R13: ffff88001c68bc40 R14: 0000000000000000 R15: ffff88001de85ea8
> > > [84091.220015] FS:  00007f43adb30700(0000) GS:ffff88001f100000(0000) knlGS:0000000000000000
> > > [84091.220015] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > [84091.220015] CR2: 0000000001ffed30 CR3: 000000001e02e000 CR4: 00000000000007e0
> > > [84091.220015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > [84091.220015] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > [84091.220015] Process rm (pid: 32545, threadinfo ffff88001de84000, task ffff88001c22e5c0)
> > > [84091.220015] Stack:
> > > [84091.220015]  ffff8800087f4130 ffff8800087f41b8 ffff88001c68b800 0000000000000000
> > > [84091.220015]  ffff88001de85e48 ffffffff81184357 ffff88001de85e48 ffff8800087f4130
> > > [84091.220015]  ffff88001e005000 ffff880014e4eb40 ffff88001de85e68 ffffffff81184418
> > > [84091.220015] Call Trace:
> > > [84091.220015]  [<ffffffff81184357>] iput_final+0x117/0x190
> > > [84091.220015]  [<ffffffff81184418>] iput+0x48/0x60
> > > [84091.220015]  [<ffffffff8117a804>] do_unlinkat+0x214/0x240
> > > [84091.220015]  [<ffffffff8117aa4d>] sys_unlinkat+0x1d/0x40
> > > [84091.220015]  [<ffffffff81583129>] system_call_fastpath+0x16/0x1b
> > > [84091.220015] Code: 5c 41 5d b8 01 00 00 00 41 5e c9 c3 49 8d 45 08 f0 45 0f b3 75 08 eb db 0f 1f 40 00 66 83 03 01 5b 41 5c 41 5d 31 c0 41 5e c9 c3 <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 ba 00 00 
> > > [84091.220015] RIP  [<ffffffff81127fff>] list_lru_del+0xcf/0xe0
> > > [84091.220015]  RSP <ffff88001de85df8>
> > > [84091.470390] ---[ end trace e6915e8ee0f5f079 ]---
> > > 
> > > Which is BUG_ON(nlru->nr_items < 0) from iput_final path. So it seems
> > > that there is still a race there.
> > 
> > I am still looking at this - still can't reproduce, still don't know what is going
> > on.
> 
> I am bisecting it again. It is quite tedious, though, because good case
> is hard to be sure about.

And my test case runs the following (there is very same B.run, both of
them run in its own group):
#JOBS=4
#KERNEL_CONFIG="./config"
#KERNEL_TAR="./linux-3.7-rc5.tar.bz2"
#KERNEL_OUT="build/$CGROUP/kernel"
#CGROUP=A
$ cat A.run
KERNEL_DIR="$KERNEL_OUT/${KERNEL_TAR%.tar.bz2}"
mkdir -p "$KERNEL_DIR"

tar -xf $KERNEL_TAR -C $KERNEL_OUT || fail "get the source for $KERNEL_TAR->$KERNEL_OUT"
cp "$KERNEL_CONFIG" "$KERNEL_DIR/.config" || fail "Get the config"

LOG="`pwd`/$LOG_OUT_DIR"
mkdir -p "$LOG"

old_path="`pwd`"
cd "$KERNEL_DIR"
info "$CGROUP starting build jobs:$JOBS"
TIMESTAMP=`date +%s`
( /usr/bin/time -v make -j$JOBS vmlinux >/dev/null ) > $LOG/time.$CGROUP.$TIMESTAMP 2>&1 || fail "Build the kernel at $KERNEL_DIR"
cd "$old_path"
rm -rf "$KERNEL_OUT"
sync
echo 3 > /proc/sys/vm/drop_caches
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Thu 20-06-13 17:12:01, Michal Hocko wrote:
> I am bisecting it again. It is quite tedious, though, because good case
> is hard to be sure about.

OK, so now I converged to 2d4fc052 (inode: convert inode lru list to generic lru
list code.) in my tree and I have double checked it matches what is in
the linux-next. This doesn't help much to pin point the issue I am
afraid :/

I have applied the inode_lru_isolate fix on each step.
$ git bisect log
git bisect start
# bad: [d02c11c146b626cf8e2586446773ba02999e4e2f] mm/sparse.c: put clear_hwpoisoned_pages within CONFIG_MEMORY_HOTREMOVE
git bisect bad d02c11c146b626cf8e2586446773ba02999e4e2f
# good: [58f6e0c8fb37e8e37d5ac17a61a53ac236c15047] Reverted "mm: tlb_fast_mode check missing in tlb_finish_mmu()"
git bisect good 58f6e0c8fb37e8e37d5ac17a61a53ac236c15047
# bad: [4ec7ecd30d643b12e1041226ff180da3d88918ee] include/linux/math64.h: add div64_ul()
git bisect bad 4ec7ecd30d643b12e1041226ff180da3d88918ee
# bad: [96dd4e69dc50c7ed18e407798f7f677fa5588eae] xfs: convert dquot cache lru to list_lru
git bisect bad 96dd4e69dc50c7ed18e407798f7f677fa5588eae
# bad: [2d4fc052823c2f598f03633e64bc0439cd2bfa04] inode: convert inode lru list to generic lru list code.
git bisect bad 2d4fc052823c2f598f03633e64bc0439cd2bfa04
# good: [cacbac6d9d80cc5277e8f67c7a474edb6488a5ea] dcache: remove dentries from LRU before putting on dispose list
git bisect good cacbac6d9d80cc5277e8f67c7a474edb6488a5ea
# good: [03a05514e71551bfdff1e3496a30b0fd5083f8fe] shrinker: convert superblock shrinkers to new API
git bisect good 03a05514e71551bfdff1e3496a30b0fd5083f8fe
# good: [ddc5bc7a8856e0e61ea9c2d4fcbcd3f85ecc92e7] list: add a new LRU list type
git bisect good ddc5bc7a8856e0e61ea9c2d4fcbcd3f85ecc92e7
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Fri, Jun 21, 2013 at 11:00:21AM +0200, Michal Hocko wrote:
> On Thu 20-06-13 17:12:01, Michal Hocko wrote:
> > I am bisecting it again. It is quite tedious, though, because good case
> > is hard to be sure about.
> 
> OK, so now I converged to 2d4fc052 (inode: convert inode lru list to generic lru
> list code.) in my tree and I have double checked it matches what is in
> the linux-next. This doesn't help much to pin point the issue I am
> afraid :/
> 
Can you revert this patch (easiest way ATM is to rewind your tree to a point
right before it) and apply the following patch?

As Dave has mentioned, it is very likely that this bug was already there, we
were just not ever checking imbalances. The attached patch would tell us at
least if the imbalance was there before. If this is the case, I would suggest
turning the BUG condition into a WARN_ON_ONCE since we would be officially
not introducing any regression. It is no less of a bug, though, and we should
keep looking for it.

The main change from before / after the patch is that we are now keeping things
per node. One possibility of having this BUGing would be to have an inode to be
inserted into one node-lru and removed from another. I cannot see how it could
happen, because kernel pages are stable in memory and are not moved from node
to node. We could still have some sort of weird bug in the node calculation
function. In any case, would it be possible for you to artificially restrict
your setup to a single node ? Although I have no idea how to do that, we seem
to have no parameter to disable numa. Maybe booting with less memory, enough to
fit a single node?

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

[-- Attachment #1: Type: text/plain, Size: 1773 bytes --]

On Sun, Jun 23, 2013 at 03:51:29PM +0400, Glauber Costa wrote:
> On Fri, Jun 21, 2013 at 11:00:21AM +0200, Michal Hocko wrote:
> > On Thu 20-06-13 17:12:01, Michal Hocko wrote:
> > > I am bisecting it again. It is quite tedious, though, because good case
> > > is hard to be sure about.
> > 
> > OK, so now I converged to 2d4fc052 (inode: convert inode lru list to generic lru
> > list code.) in my tree and I have double checked it matches what is in
> > the linux-next. This doesn't help much to pin point the issue I am
> > afraid :/
> > 
> Can you revert this patch (easiest way ATM is to rewind your tree to a point
> right before it) and apply the following patch?
> 
> As Dave has mentioned, it is very likely that this bug was already there, we
> were just not ever checking imbalances. The attached patch would tell us at
> least if the imbalance was there before. If this is the case, I would suggest
> turning the BUG condition into a WARN_ON_ONCE since we would be officially
> not introducing any regression. It is no less of a bug, though, and we should
> keep looking for it.
> 
> The main change from before / after the patch is that we are now keeping things
> per node. One possibility of having this BUGing would be to have an inode to be
> inserted into one node-lru and removed from another. I cannot see how it could
> happen, because kernel pages are stable in memory and are not moved from node
> to node. We could still have some sort of weird bug in the node calculation
> function. In any case, would it be possible for you to artificially restrict
> your setup to a single node ? Although I have no idea how to do that, we seem
> to have no parameter to disable numa. Maybe booting with less memory, enough to
> fit a single node?
> 
The patch:

[-- Attachment #2: BUG.patch --]
[-- Type: text/plain, Size: 919 bytes --]

diff --git a/fs/inode.c b/fs/inode.c
index 1ddaa2e..0b5c3fa 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -427,6 +427,7 @@ static void inode_lru_list_del(struct inode *inode)
 	if (!list_empty(&inode->i_lru)) {
 		list_del_init(&inode->i_lru);
 		inode->i_sb->s_nr_inodes_unused--;
+		BUG_ON(sb->s_nr_inodes_unused < 0);
 		this_cpu_dec(nr_unused);
 	}
 	spin_unlock(&inode->i_sb->s_inode_lru_lock);
@@ -739,6 +740,7 @@ long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan)
 			list_del_init(&inode->i_lru);
 			spin_unlock(&inode->i_lock);
 			sb->s_nr_inodes_unused--;
+			BUG_ON(sb->s_nr_inodes_unused < 0);
 			this_cpu_dec(nr_unused);
 			continue;
 		}
@@ -777,6 +779,7 @@ long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan)
 
 		list_move(&inode->i_lru, &freeable);
 		sb->s_nr_inodes_unused--;
+		BUG_ON(sb->s_nr_inodes_unused < 0);
 		this_cpu_dec(nr_unused);
 		freed++;
 	}

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Dave Chinner]

On Sun, Jun 23, 2013 at 03:51:29PM +0400, Glauber Costa wrote:
> On Fri, Jun 21, 2013 at 11:00:21AM +0200, Michal Hocko wrote:
> > On Thu 20-06-13 17:12:01, Michal Hocko wrote:
> > > I am bisecting it again. It is quite tedious, though, because good case
> > > is hard to be sure about.
> > 
> > OK, so now I converged to 2d4fc052 (inode: convert inode lru list to generic lru
> > list code.) in my tree and I have double checked it matches what is in
> > the linux-next. This doesn't help much to pin point the issue I am
> > afraid :/
> > 
> Can you revert this patch (easiest way ATM is to rewind your tree to a point
> right before it) and apply the following patch?
> 
> As Dave has mentioned, it is very likely that this bug was already there, we
> were just not ever checking imbalances. The attached patch would tell us at
> least if the imbalance was there before. If this is the case, I would suggest
> turning the BUG condition into a WARN_ON_ONCE since we would be officially
> not introducing any regression. It is no less of a bug, though, and we should
> keep looking for it.

We probably should do that BUG->WARN change anyway. BUG_ON is pretty
obnoxious in places where we can probably continue on without much
impact....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Sun 23-06-13 15:51:29, Glauber Costa wrote:
> On Fri, Jun 21, 2013 at 11:00:21AM +0200, Michal Hocko wrote:
> > On Thu 20-06-13 17:12:01, Michal Hocko wrote:
> > > I am bisecting it again. It is quite tedious, though, because good case
> > > is hard to be sure about.
> > 
> > OK, so now I converged to 2d4fc052 (inode: convert inode lru list to generic lru
> > list code.) in my tree and I have double checked it matches what is in
> > the linux-next. This doesn't help much to pin point the issue I am
> > afraid :/
> > 
> Can you revert this patch (easiest way ATM is to rewind your tree to a point
> right before it) and apply the following patch?

OK, I am testing it now.

> As Dave has mentioned, it is very likely that this bug was already there, we
> were just not ever checking imbalances. The attached patch would tell us at
> least if the imbalance was there before.

Maybe I wasn't clear before but I have seen mostly hangs (posted
earlier) during bisection. I do not remember BUG_ONs on imbalances after
inode_lru_isolate fix.

> If this is the case, I would suggest
> turning the BUG condition into a WARN_ON_ONCE since we would be officially
> not introducing any regression. It is no less of a bug, though, and we should
> keep looking for it.
> 
> The main change from before / after the patch is that we are now keeping things
> per node. One possibility of having this BUGing would be to have an inode to be
> inserted into one node-lru and removed from another. I cannot see how it could
> happen, because kernel pages are stable in memory and are not moved from node
> to node. We could still have some sort of weird bug in the node calculation
> function. 

> In any case, would it be possible for you to artificially restrict
> your setup to a single node ? Although I have no idea how to do that, we seem
> to have no parameter to disable numa. Maybe booting with less memory, enough to
> fit a single node?

I can play with memmap to use areas on from a single node. Let's see
whether the patch in the follow up email shows something first.
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 18-06-13 02:30:05, Glauber Costa wrote:
> On Mon, Jun 17, 2013 at 02:35:08PM -0700, Andrew Morton wrote:
[...]
> > The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
> > it's inodes?
> > 
> Assuming there is no memory corruption of any sort going on , let's
> check the code. nr_item is only manipulated in 3 places:
> 
> 1) list_lru_add, where it is increased
> 2) list_lru_del, where it is decreased in case the user have voluntarily removed the
>    element from the list
> 3) list_lru_walk_node, where an element is removing during shrink.
> 
> All three excerpts seem to be correctly locked, so something like this
> indicates an imbalance.  Either the element was never added to the
> list, or it was added, removed, and we didn't notice it. (Again, your
> backing storage is not XFS, is it? If it is , we have another user to
> look for)

No this is ext3. But I can try to test with xfs as well if it helps.
[...]
-- 
Michal Hocko
SUSE Labs

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Glauber Costa]

On Tue, Jun 18, 2013 at 10:19:31AM +0200, Michal Hocko wrote:
> On Tue 18-06-13 02:30:05, Glauber Costa wrote:
> > On Mon, Jun 17, 2013 at 02:35:08PM -0700, Andrew Morton wrote:
> [...]
> > > The trace says shrink_slab_node->super_cache_scan->prune_icache_sb.  So
> > > it's inodes?
> > > 
> > Assuming there is no memory corruption of any sort going on , let's
> > check the code. nr_item is only manipulated in 3 places:
> > 
> > 1) list_lru_add, where it is increased
> > 2) list_lru_del, where it is decreased in case the user have voluntarily removed the
> >    element from the list
> > 3) list_lru_walk_node, where an element is removing during shrink.
> > 
> > All three excerpts seem to be correctly locked, so something like this
> > indicates an imbalance.  Either the element was never added to the
> > list, or it was added, removed, and we didn't notice it. (Again, your
> > backing storage is not XFS, is it? If it is , we have another user to
> > look for)
> 
> No this is ext3. But I can try to test with xfs as well if it helps.
> [...]

XFS won't help this, on the contrary. The reason I asked is because XFS
uses list_lru for its internal structures as well. So it is actually preferred
if you are reproducing this without it, so we can at least isolate that part.

Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92

[Michal Hocko]

On Tue 18-06-13 12:21:33, Glauber Costa wrote:
> On Tue, Jun 18, 2013 at 10:19:31AM +0200, Michal Hocko wrote:
[...]
> > No this is ext3. But I can try to test with xfs as well if it helps.
> > [...]
> 
> XFS won't help this, on the contrary. The reason I asked is because XFS
> uses list_lru for its internal structures as well. So it is actually preferred
> if you are reproducing this without it, so we can at least isolate that part.

OK

-- 
Michal Hocko
SUSE Labs