* Kernel 3.11 / 3.12 OOM killer and Xen ballooning
@ 2013-12-09 17:50 James Dingwall
2013-12-09 21:48 ` Konrad Rzeszutek Wilk
2013-12-10 8:16 ` Jan Beulich
0 siblings, 2 replies; 42+ messages in thread
From: James Dingwall @ 2013-12-09 17:50 UTC (permalink / raw)
To: xen-devel
Hi,
Since 3.11 I have noticed that the OOM killer quite frequently triggers
in my Xen guest domains which use ballooning to increase/decrease their
memory allocation according to their requirements. One example domain I
have has a maximum memory setting of ~1.5Gb but it usually idles at
~300Mb, it is also configured with 2Gb swap which is almost 100% free.
# free
total used free shared buffers cached
Mem: 272080 248108 23972 0 1448 63064
-/+ buffers/cache: 183596 88484
Swap: 2097148 8 2097140
There is plenty of available free memory in the hypervisor to balloon to
the maximum size:
# xl info | grep free_mem
free_memory : 14923
An example trace (they are always the same) from the oom killer in 3.12
is added below. So far I have not been able to reproduce this at will
so it is difficult to start bisecting it to see if a particular change
introduced this. However it does seem that the behaviour is wrong
because a) ballooning could give the guest more memory, b) there is lots
of swap available which could be used as a fallback.
If other information could help or there are more tests that I could run
then please let me know.
Thanks,
James
[473233.777271] emerge invoked oom-killer: gfp_mask=0x280da, order=0,
oom_score_adj=0
[473233.777279] CPU: 0 PID: 22159 Comm: emerge Tainted: G W 3.12.0 #80
[473233.777282] ffff88000599f6f8 ffff8800117bda58 ffffffff81489a80
ffff88004760e8e8
[473233.777286] ffff88000599f1c0 ffff8800117bdaf8 ffffffff81487577
ffff8800117bdaa8
[473233.777289] ffffffff810f8c0f ffff8800117bda88 ffffffff81006dc8
ffff8800117bda98
[473233.777293] Call Trace:
[473233.777305] [<ffffffff81489a80>] dump_stack+0x46/0x58
[473233.777310] [<ffffffff81487577>] dump_header.isra.9+0x6d/0x1cc
[473233.777315] [<ffffffff810f8c0f>] ? super_cache_count+0xa8/0xb8
[473233.777321] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[473233.777324] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
[473233.777328] [<ffffffff8148f336>] ?
_raw_spin_unlock_irqrestore+0x47/0x62
[473233.777333] [<ffffffff812915d3>] ? ___ratelimit+0xcb/0xe8
[473233.777338] [<ffffffff810b2aa7>] oom_kill_process+0x70/0x2fd
[473233.777343] [<ffffffff81048775>] ? has_ns_capability_noaudit+0x12/0x19
[473233.777346] [<ffffffff8104878e>] ? has_capability_noaudit+0x12/0x14
[473233.777349] [<ffffffff810b31c6>] out_of_memory+0x31b/0x34e
[473233.777353] [<ffffffff810b72f0>] __alloc_pages_nodemask+0x65b/0x792
[473233.777358] [<ffffffff810e3c1b>] alloc_pages_vma+0xd0/0x10c
[473233.777361] [<ffffffff81003f69>] ?
__raw_callee_save_xen_pmd_val+0x11/0x1e
[473233.777365] [<ffffffff810cf685>] handle_mm_fault+0x6d4/0xd54
[473233.777371] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
[473233.777374] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[473233.777378] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[473233.777382] [<ffffffff810676c7>] ? sched_clock_local+0x12/0x75
[473233.777386] [<ffffffff810a44b4>] ? __acct_update_integrals+0xb4/0xbf
[473233.777389] [<ffffffff810a4827>] ? acct_account_cputime+0x17/0x19
[473233.777392] [<ffffffff81067bc0>] ? account_user_time+0x67/0x92
[473233.777395] [<ffffffff810680b3>] ? vtime_account_user+0x4d/0x52
[473233.777398] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
[473233.777401] [<ffffffff8148f9d8>] page_fault+0x28/0x30
[473233.777403] Mem-Info:
[473233.777405] Node 0 DMA per-cpu:
[473233.777408] CPU 0: hi: 0, btch: 1 usd: 0
[473233.777409] CPU 1: hi: 0, btch: 1 usd: 0
[473233.777411] CPU 2: hi: 0, btch: 1 usd: 0
[473233.777412] CPU 3: hi: 0, btch: 1 usd: 0
[473233.777413] Node 0 DMA32 per-cpu:
[473233.777415] CPU 0: hi: 186, btch: 31 usd: 103
[473233.777417] CPU 1: hi: 186, btch: 31 usd: 110
[473233.777419] CPU 2: hi: 186, btch: 31 usd: 175
[473233.777420] CPU 3: hi: 186, btch: 31 usd: 182
[473233.777421] Node 0 Normal per-cpu:
[473233.777423] CPU 0: hi: 0, btch: 1 usd: 0
[473233.777424] CPU 1: hi: 0, btch: 1 usd: 0
[473233.777426] CPU 2: hi: 0, btch: 1 usd: 0
[473233.777427] CPU 3: hi: 0, btch: 1 usd: 0
[473233.777433] active_anon:35740 inactive_anon:33812 isolated_anon:0
active_file:4672 inactive_file:11607 isolated_file:0
unevictable:0 dirty:4 writeback:0 unstable:0
free:2067 slab_reclaimable:3583 slab_unreclaimable:3524
mapped:3329 shmem:324 pagetables:2003 bounce:0
free_cma:0
[473233.777435] Node 0 DMA free:4200kB min:60kB low:72kB high:88kB
active_anon:264kB inactive_anon:456kB active_file:140kB
inactive_file:340kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:6176kB mlocked:0kB dirty:0kB
writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:96kB
slab_unreclaimable:112kB kernel_stack:24kB pagetables:24kB unstable:0kB
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:33270
all_unreclaimable? yes
[473233.777443] lowmem_reserve[]: 0 1036 1036 1036
[473233.777447] Node 0 DMA32 free:4060kB min:4084kB low:5104kB
high:6124kB active_anon:41256kB inactive_anon:33128kB active_file:8544kB
inactive_file:14312kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:1163264kB managed:165780kB mlocked:0kB
dirty:0kB writeback:0kB mapped:6428kB shmem:604kB
slab_reclaimable:9800kB slab_unreclaimable:12908kB kernel_stack:1832kB
pagetables:5924kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:152386 all_unreclaimable? yes
[473233.777454] lowmem_reserve[]: 0 0 0 0
[473233.777457] Node 0 Normal free:8kB min:0kB low:0kB high:0kB
active_anon:101440kB inactive_anon:101664kB active_file:10004kB
inactive_file:31776kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:393216kB managed:256412kB mlocked:0kB
dirty:16kB writeback:0kB mapped:6788kB shmem:692kB
slab_reclaimable:4436kB slab_unreclaimable:1076kB kernel_stack:136kB
pagetables:2064kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:368809 all_unreclaimable? yes
[473233.777464] lowmem_reserve[]: 0 0 0 0
[473233.777467] Node 0 DMA: 41*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (R)
1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB =
4196kB
[473233.777480] Node 0 DMA32: 1015*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4060kB
[473233.777490] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[473233.777498] 5018 total pagecache pages
[473233.777500] 16 pages in swap cache
[473233.777501] Swap cache stats: add 2829330, delete 2829314, find
344059/481859
[473233.777503] Free swap = 2096980kB
[473233.777503] Total swap = 2097148kB
[473233.794497] 557055 pages RAM
[473233.794500] 189326 pages reserved
[473233.794501] 544934 pages shared
[473233.794502] 358441 pages non-shared
[473233.794504] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[473233.794523] [ 6597] 0 6597 8156 252 20 0 -1000
udevd
[473233.794530] [ 7194] 0 7194 2232 137 10 0 0 metalog
[473233.794534] [ 7195] 0 7195 2223 31 10 3 0 metalog
[473233.794537] [ 7211] 0 7211 1064 35 8 0 0
acpid
[473233.794546] [ 7227] 702 7227 4922 183 14 0 0
dbus-daemon
[473233.794553] [ 7427] 0 7427 13630 179 29 15 0 rpcbind
[473233.794560] [ 7442] 0 7442 14743 332 32 0 0
rpc.statd
[473233.794569] [ 7472] 0 7472 6365 115 17 0 0
rpc.idmapd
[473233.794576] [ 7488] 0 7488 43602 349 40 0 0 cupsd
[473233.794583] [ 7512] 0 7512 14856 243 30 0 0
rpc.mountd
[473233.794592] [ 7552] 0 7552 148819 940 68 0 0
automount
[473233.794595] [ 7592] 0 7592 16006 233 32 0 -1000 sshd
[473233.794598] [ 7608] 0 7608 87672 2257 128 6 0
apache2
[473233.794601] [ 7633] 0 7633 521873 631 56 0 0
console-kit-dae
[473233.794604] [ 7713] 106 7713 15453 295 34 2 0 nrpe
[473233.794607] [ 7719] 986 7719 91303 798 41 0 0 polkitd
[473233.794610] [ 7757] 123 7757 7330 259 17 0 0 ntpd
[473233.794613] [ 7845] 0 7845 3583 94 12 0 0 master
[473233.794616] [ 7847] 207 7847 17745 311 38 0 0 qmgr
[473233.794619] [ 7861] 65534 7861 2101 21 9 19 0
rwhod
[473233.794622] [ 7864] 65534 7864 2101 99 9 0 0
rwhod
[473233.794625] [ 7876] 0 7876 48582 533 47 19 0 smbd
[473233.794628] [ 7881] 0 7881 44277 372 38 0 0 nmbd
[473233.794631] [ 7895] 0 7895 48646 621 45 18 0 smbd
[473233.794634] [ 7902] 2 7902 1078 39 8 4 0 slpd
[473233.794637] [ 7917] 0 7917 38452 1073 28 1 0 snmpd
[473233.794640] [ 7945] 0 7945 27552 58 9 0 0 cron
[473233.794648] [ 7993] 0 7993 201378 5432 63 39 0 nscd
[473233.794658] [ 8064] 0 8064 1060 28 7 0 0
agetty
[473233.794664] [ 8065] 0 8065 26507 29 9 0 0
agetty
[473233.794667] [ 8066] 0 8066 26507 29 9 0 0
agetty
[473233.794670] [ 8067] 0 8067 26507 28 9 0 0
agetty
[473233.794673] [ 8068] 0 8068 26507 28 8 0 0
agetty
[473233.794678] [ 8069] 0 8069 26507 30 9 0 0
agetty
[473233.794686] [ 8070] 0 8070 26507 30 9 0 0
agetty
[473233.794693] [ 8071] 0 8071 26507 30 9 0 0
agetty
[473233.794701] [ 8072] 0 8072 26507 28 9 0 0
agetty
[473233.794708] [ 8316] 0 8316 3736 83 11 6 0
ssh-agent
[473233.794712] [ 8341] 0 8341 3390 66 12 7 0
gpg-agent
[473233.794716] [ 2878] 81 2878 88431 2552 121 5 0
apache2
[473233.794718] [ 2879] 81 2879 88431 2552 121 5 0
apache2
[473233.794721] [ 2880] 81 2880 88431 2552 121 5 0
apache2
[473233.794724] [ 2881] 81 2881 88431 2552 121 5 0
apache2
[473233.794727] [ 2882] 81 2882 88431 2552 121 5 0
apache2
[473233.794734] [ 3523] 81 3523 88431 2552 121 5 0
apache2
[473233.794737] [30259] 1000 30259 3736 118 11 0 0
ssh-agent
[473233.794741] [30284] 1000 30284 3390 141 12 0 0
gpg-agent
[473233.794745] [21263] 207 21263 17703 771 39 1 0 pickup
[473233.794748] [21663] 0 21663 30743 228 16 0 0 cron
[473233.794751] [21665] 0 21665 2980 392 12 0 0
gentoosync.sh
[473233.794755] [22158] 0 22158 3181 273 12 0 0 sendmail
[473233.794757] [22159] 0 22159 77646 54920 158 0 0
emerge
[473233.794760] [22160] 0 22160 1068 85 8 0 0 tail
[473233.794764] [22161] 0 22161 3173 277 11 0 0 postdrop
[473233.794768] Out of memory: Kill process 22159 (emerge) score 57 or
sacrifice child
[473233.794771] Killed process 22159 (emerge) total-vm:310584kB,
anon-rss:215840kB, file-rss:3840kB
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-09 17:50 Kernel 3.11 / 3.12 OOM killer and Xen ballooning James Dingwall
@ 2013-12-09 21:48 ` Konrad Rzeszutek Wilk
2013-12-10 14:52 ` James Dingwall
2013-12-10 8:16 ` Jan Beulich
1 sibling, 1 reply; 42+ messages in thread
From: Konrad Rzeszutek Wilk @ 2013-12-09 21:48 UTC (permalink / raw)
To: James Dingwall, bob.liu; +Cc: xen-devel
On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
> Hi,
>
> Since 3.11 I have noticed that the OOM killer quite frequently
> triggers in my Xen guest domains which use ballooning to
> increase/decrease their memory allocation according to their
> requirements. One example domain I have has a maximum memory
> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
> configured with 2Gb swap which is almost 100% free.
>
> # free
> total used free shared buffers cached
> Mem: 272080 248108 23972 0 1448 63064
> -/+ buffers/cache: 183596 88484
> Swap: 2097148 8 2097140
>
> There is plenty of available free memory in the hypervisor to
> balloon to the maximum size:
> # xl info | grep free_mem
> free_memory : 14923
>
> An example trace (they are always the same) from the oom killer in
> 3.12 is added below. So far I have not been able to reproduce this
> at will so it is difficult to start bisecting it to see if a
> particular change introduced this. However it does seem that the
> behaviour is wrong because a) ballooning could give the guest more
> memory, b) there is lots of swap available which could be used as a
> fallback.
>
> If other information could help or there are more tests that I could
> run then please let me know.
I presume you have enabled 'tmem' both in the hypervisor and in
the guest right?
>
> Thanks,
> James
>
>
>
>
> [473233.777271] emerge invoked oom-killer: gfp_mask=0x280da,
> order=0, oom_score_adj=0
> [473233.777279] CPU: 0 PID: 22159 Comm: emerge Tainted: G W 3.12.0 #80
> [473233.777282] ffff88000599f6f8 ffff8800117bda58 ffffffff81489a80
> ffff88004760e8e8
> [473233.777286] ffff88000599f1c0 ffff8800117bdaf8 ffffffff81487577
> ffff8800117bdaa8
> [473233.777289] ffffffff810f8c0f ffff8800117bda88 ffffffff81006dc8
> ffff8800117bda98
> [473233.777293] Call Trace:
> [473233.777305] [<ffffffff81489a80>] dump_stack+0x46/0x58
> [473233.777310] [<ffffffff81487577>] dump_header.isra.9+0x6d/0x1cc
> [473233.777315] [<ffffffff810f8c0f>] ? super_cache_count+0xa8/0xb8
> [473233.777321] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
> [473233.777324] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
> [473233.777328] [<ffffffff8148f336>] ?
> _raw_spin_unlock_irqrestore+0x47/0x62
> [473233.777333] [<ffffffff812915d3>] ? ___ratelimit+0xcb/0xe8
> [473233.777338] [<ffffffff810b2aa7>] oom_kill_process+0x70/0x2fd
> [473233.777343] [<ffffffff81048775>] ? has_ns_capability_noaudit+0x12/0x19
> [473233.777346] [<ffffffff8104878e>] ? has_capability_noaudit+0x12/0x14
> [473233.777349] [<ffffffff810b31c6>] out_of_memory+0x31b/0x34e
> [473233.777353] [<ffffffff810b72f0>] __alloc_pages_nodemask+0x65b/0x792
> [473233.777358] [<ffffffff810e3c1b>] alloc_pages_vma+0xd0/0x10c
> [473233.777361] [<ffffffff81003f69>] ?
> __raw_callee_save_xen_pmd_val+0x11/0x1e
> [473233.777365] [<ffffffff810cf685>] handle_mm_fault+0x6d4/0xd54
> [473233.777371] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
> [473233.777374] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
> [473233.777378] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
> [473233.777382] [<ffffffff810676c7>] ? sched_clock_local+0x12/0x75
> [473233.777386] [<ffffffff810a44b4>] ? __acct_update_integrals+0xb4/0xbf
> [473233.777389] [<ffffffff810a4827>] ? acct_account_cputime+0x17/0x19
> [473233.777392] [<ffffffff81067bc0>] ? account_user_time+0x67/0x92
> [473233.777395] [<ffffffff810680b3>] ? vtime_account_user+0x4d/0x52
> [473233.777398] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
> [473233.777401] [<ffffffff8148f9d8>] page_fault+0x28/0x30
> [473233.777403] Mem-Info:
> [473233.777405] Node 0 DMA per-cpu:
> [473233.777408] CPU 0: hi: 0, btch: 1 usd: 0
> [473233.777409] CPU 1: hi: 0, btch: 1 usd: 0
> [473233.777411] CPU 2: hi: 0, btch: 1 usd: 0
> [473233.777412] CPU 3: hi: 0, btch: 1 usd: 0
> [473233.777413] Node 0 DMA32 per-cpu:
> [473233.777415] CPU 0: hi: 186, btch: 31 usd: 103
> [473233.777417] CPU 1: hi: 186, btch: 31 usd: 110
> [473233.777419] CPU 2: hi: 186, btch: 31 usd: 175
> [473233.777420] CPU 3: hi: 186, btch: 31 usd: 182
> [473233.777421] Node 0 Normal per-cpu:
> [473233.777423] CPU 0: hi: 0, btch: 1 usd: 0
> [473233.777424] CPU 1: hi: 0, btch: 1 usd: 0
> [473233.777426] CPU 2: hi: 0, btch: 1 usd: 0
> [473233.777427] CPU 3: hi: 0, btch: 1 usd: 0
> [473233.777433] active_anon:35740 inactive_anon:33812 isolated_anon:0
> active_file:4672 inactive_file:11607 isolated_file:0
> unevictable:0 dirty:4 writeback:0 unstable:0
> free:2067 slab_reclaimable:3583 slab_unreclaimable:3524
> mapped:3329 shmem:324 pagetables:2003 bounce:0
> free_cma:0
> [473233.777435] Node 0 DMA free:4200kB min:60kB low:72kB high:88kB
> active_anon:264kB inactive_anon:456kB active_file:140kB
> inactive_file:340kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15996kB managed:6176kB mlocked:0kB
> dirty:0kB writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:96kB
> slab_unreclaimable:112kB kernel_stack:24kB pagetables:24kB
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> pages_scanned:33270 all_unreclaimable? yes
> [473233.777443] lowmem_reserve[]: 0 1036 1036 1036
> [473233.777447] Node 0 DMA32 free:4060kB min:4084kB low:5104kB
> high:6124kB active_anon:41256kB inactive_anon:33128kB
> active_file:8544kB inactive_file:14312kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:1163264kB
> managed:165780kB mlocked:0kB dirty:0kB writeback:0kB mapped:6428kB
> shmem:604kB slab_reclaimable:9800kB slab_unreclaimable:12908kB
> kernel_stack:1832kB pagetables:5924kB unstable:0kB bounce:0kB
> free_cma:0kB writeback_tmp:0kB pages_scanned:152386
> all_unreclaimable? yes
> [473233.777454] lowmem_reserve[]: 0 0 0 0
> [473233.777457] Node 0 Normal free:8kB min:0kB low:0kB high:0kB
> active_anon:101440kB inactive_anon:101664kB active_file:10004kB
> inactive_file:31776kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:393216kB managed:256412kB mlocked:0kB
> dirty:16kB writeback:0kB mapped:6788kB shmem:692kB
> slab_reclaimable:4436kB slab_unreclaimable:1076kB kernel_stack:136kB
> pagetables:2064kB unstable:0kB bounce:0kB free_cma:0kB
> writeback_tmp:0kB pages_scanned:368809 all_unreclaimable? yes
> [473233.777464] lowmem_reserve[]: 0 0 0 0
> [473233.777467] Node 0 DMA: 41*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB
> (R) 1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R)
> 0*4096kB = 4196kB
> [473233.777480] Node 0 DMA32: 1015*4kB (U) 0*8kB 0*16kB 0*32kB
> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4060kB
> [473233.777490] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> [473233.777498] 5018 total pagecache pages
> [473233.777500] 16 pages in swap cache
> [473233.777501] Swap cache stats: add 2829330, delete 2829314, find
> 344059/481859
> [473233.777503] Free swap = 2096980kB
> [473233.777503] Total swap = 2097148kB
> [473233.794497] 557055 pages RAM
> [473233.794500] 189326 pages reserved
> [473233.794501] 544934 pages shared
> [473233.794502] 358441 pages non-shared
> [473233.794504] [ pid ] uid tgid total_vm rss nr_ptes
> swapents oom_score_adj name
> [473233.794523] [ 6597] 0 6597 8156 252 20 0
> -1000 udevd
> [473233.794530] [ 7194] 0 7194 2232 137 10 0 0 metalog
> [473233.794534] [ 7195] 0 7195 2223 31 10 3 0 metalog
> [473233.794537] [ 7211] 0 7211 1064 35 8 0
> 0 acpid
> [473233.794546] [ 7227] 702 7227 4922 183 14 0 0
> dbus-daemon
> [473233.794553] [ 7427] 0 7427 13630 179 29 15 0 rpcbind
> [473233.794560] [ 7442] 0 7442 14743 332 32 0 0
> rpc.statd
> [473233.794569] [ 7472] 0 7472 6365 115 17 0 0
> rpc.idmapd
> [473233.794576] [ 7488] 0 7488 43602 349 40 0 0 cupsd
> [473233.794583] [ 7512] 0 7512 14856 243 30 0 0
> rpc.mountd
> [473233.794592] [ 7552] 0 7552 148819 940 68 0 0
> automount
> [473233.794595] [ 7592] 0 7592 16006 233 32 0 -1000 sshd
> [473233.794598] [ 7608] 0 7608 87672 2257 128 6
> 0 apache2
> [473233.794601] [ 7633] 0 7633 521873 631 56 0 0
> console-kit-dae
> [473233.794604] [ 7713] 106 7713 15453 295 34 2 0 nrpe
> [473233.794607] [ 7719] 986 7719 91303 798 41 0 0 polkitd
> [473233.794610] [ 7757] 123 7757 7330 259 17 0 0 ntpd
> [473233.794613] [ 7845] 0 7845 3583 94 12 0 0 master
> [473233.794616] [ 7847] 207 7847 17745 311 38 0 0 qmgr
> [473233.794619] [ 7861] 65534 7861 2101 21 9 19
> 0 rwhod
> [473233.794622] [ 7864] 65534 7864 2101 99 9 0
> 0 rwhod
> [473233.794625] [ 7876] 0 7876 48582 533 47 19 0 smbd
> [473233.794628] [ 7881] 0 7881 44277 372 38 0 0 nmbd
> [473233.794631] [ 7895] 0 7895 48646 621 45 18 0 smbd
> [473233.794634] [ 7902] 2 7902 1078 39 8 4 0 slpd
> [473233.794637] [ 7917] 0 7917 38452 1073 28 1 0 snmpd
> [473233.794640] [ 7945] 0 7945 27552 58 9 0 0 cron
> [473233.794648] [ 7993] 0 7993 201378 5432 63 39 0 nscd
> [473233.794658] [ 8064] 0 8064 1060 28 7 0
> 0 agetty
> [473233.794664] [ 8065] 0 8065 26507 29 9 0
> 0 agetty
> [473233.794667] [ 8066] 0 8066 26507 29 9 0
> 0 agetty
> [473233.794670] [ 8067] 0 8067 26507 28 9 0
> 0 agetty
> [473233.794673] [ 8068] 0 8068 26507 28 8 0
> 0 agetty
> [473233.794678] [ 8069] 0 8069 26507 30 9 0
> 0 agetty
> [473233.794686] [ 8070] 0 8070 26507 30 9 0
> 0 agetty
> [473233.794693] [ 8071] 0 8071 26507 30 9 0
> 0 agetty
> [473233.794701] [ 8072] 0 8072 26507 28 9 0
> 0 agetty
> [473233.794708] [ 8316] 0 8316 3736 83 11 6 0
> ssh-agent
> [473233.794712] [ 8341] 0 8341 3390 66 12 7 0
> gpg-agent
> [473233.794716] [ 2878] 81 2878 88431 2552 121 5
> 0 apache2
> [473233.794718] [ 2879] 81 2879 88431 2552 121 5
> 0 apache2
> [473233.794721] [ 2880] 81 2880 88431 2552 121 5
> 0 apache2
> [473233.794724] [ 2881] 81 2881 88431 2552 121 5
> 0 apache2
> [473233.794727] [ 2882] 81 2882 88431 2552 121 5
> 0 apache2
> [473233.794734] [ 3523] 81 3523 88431 2552 121 5
> 0 apache2
> [473233.794737] [30259] 1000 30259 3736 118 11 0 0
> ssh-agent
> [473233.794741] [30284] 1000 30284 3390 141 12 0 0
> gpg-agent
> [473233.794745] [21263] 207 21263 17703 771 39 1 0 pickup
> [473233.794748] [21663] 0 21663 30743 228 16 0 0 cron
> [473233.794751] [21665] 0 21665 2980 392 12 0 0
> gentoosync.sh
> [473233.794755] [22158] 0 22158 3181 273 12 0 0 sendmail
> [473233.794757] [22159] 0 22159 77646 54920 158 0
> 0 emerge
> [473233.794760] [22160] 0 22160 1068 85 8 0 0 tail
> [473233.794764] [22161] 0 22161 3173 277 11 0 0 postdrop
> [473233.794768] Out of memory: Kill process 22159 (emerge) score 57
> or sacrifice child
> [473233.794771] Killed process 22159 (emerge) total-vm:310584kB,
> anon-rss:215840kB, file-rss:3840kB
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-09 17:50 Kernel 3.11 / 3.12 OOM killer and Xen ballooning James Dingwall
2013-12-09 21:48 ` Konrad Rzeszutek Wilk
@ 2013-12-10 8:16 ` Jan Beulich
2013-12-10 14:01 ` James Dingwall
1 sibling, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2013-12-10 8:16 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
>>> On 09.12.13 at 18:50, James Dingwall <james.dingwall@zynstra.com> wrote:
> Since 3.11 I have noticed that the OOM killer quite frequently triggers
> in my Xen guest domains which use ballooning to increase/decrease their
> memory allocation according to their requirements. One example domain I
> have has a maximum memory setting of ~1.5Gb but it usually idles at
> ~300Mb, it is also configured with 2Gb swap which is almost 100% free.
>
> # free
> total used free shared buffers cached
> Mem: 272080 248108 23972 0 1448 63064
> -/+ buffers/cache: 183596 88484
> Swap: 2097148 8 2097140
>
> There is plenty of available free memory in the hypervisor to balloon to
> the maximum size:
> # xl info | grep free_mem
> free_memory : 14923
But you don't tell us how you trigger the process of re-filling
memory. Yet that's the crucial point here; the fact that there is
enough memory available in the hypervisor is only a necessary
prerequisite.
Jan
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-10 8:16 ` Jan Beulich
@ 2013-12-10 14:01 ` James Dingwall
2013-12-10 14:25 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2013-12-10 14:01 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel
Jan Beulich wrote:
>>>> On 09.12.13 at 18:50, James Dingwall <james.dingwall@zynstra.com> wrote:
>> Since 3.11 I have noticed that the OOM killer quite frequently triggers
>> in my Xen guest domains which use ballooning to increase/decrease their
>> memory allocation according to their requirements. One example domain I
>> have has a maximum memory setting of ~1.5Gb but it usually idles at
>> ~300Mb, it is also configured with 2Gb swap which is almost 100% free.
>>
>> # free
>> total used free shared buffers cached
>> Mem: 272080 248108 23972 0 1448 63064
>> -/+ buffers/cache: 183596 88484
>> Swap: 2097148 8 2097140
>>
>> There is plenty of available free memory in the hypervisor to balloon to
>> the maximum size:
>> # xl info | grep free_mem
>> free_memory : 14923
> But you don't tell us how you trigger the process of re-filling
> memory. Yet that's the crucial point here; the fact that there is
> enough memory available in the hypervisor is only a necessary
> prerequisite.
My guest systems are Gentoo based and most often the problems happen
while running emerge (Python script). The example trace was taken from
an emerge command launched via a cron script which runs emerge --sync ;
emerge --deep --newuse -p -u world. Almost all the other times I have
seen the behaviour have been during emerge --deep --newuse -u world
runs, most often during the build of large packages such as kdelibs,
seamonkey or libreoffice. However occasionally I have seen it with
smaller builds or during the package merge phase where files are indexed
and copied in to the live filesystem.
I have done some testing with the program below which successfully fills
all memory and swap before being killed. One thought I had was that
perhaps there was some issue around a balloon down/up when the rate at
which memory is being requested and released is high. I will try
another program with a random pattern of malloc/free to see if I can
make a test case to help with a bisect.
Thanks,
James
#include <stdlib.h>
extern
int main(int argc, char *argv[])
{
int leak_size = 1024 * 1024;
while(1) {
malloc(leak_size);
}
return(0);
}
This is the output of vmstat 1 leading up to a kill of g++:
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
2 0 836 42564 72 35912 0 0 56 0 14049 13585 17
46 35 2
2 0 836 42564 72 35916 0 0 56 0 14214 13747 17
46 34 2
2 0 836 42616 72 35840 0 0 112 0 14007 13564 17
46 36 2
2 0 836 42428 72 36024 0 0 236 0 12453 12025 15
42 37 6
2 0 796 41672 72 36104 0 0 44 8 14709 14243 18
48 34 0
2 0 796 41932 72 36288 0 0 148 0 13987 13484 17
45 36 2
2 0 796 42020 72 36196 0 0 60 0 13749 13288 17
45 35 2
3 0 796 41924 72 36284 0 0 72 0 14390 13868 17
47 34 2
2 0 796 42168 72 36032 0 0 40 0 11211 10819 14
37 37 12
2 0 760 41460 72 36116 0 0 108 8 14427 13946 18
48 34 1
2 0 760 35900 72 41600 0 0 108 0 13773 13362 18
48 32 2
2 0 760 35940 72 41576 0 0 56 0 14452 13978 18
47 34 1
2 0 760 35956 72 41540 0 0 88 0 14415 13975 17
48 34 1
2 0 760 35824 72 41648 0 0 104 0 14370 13925 17
46 35 2
3 0 724 35112 72 41776 0 0 96 1480 14271 13794 17
46 35 2
2 0 724 35012 72 41908 0 0 104 0 14457 13972 17
47 34 1
2 0 724 34884 72 42004 0 0 184 0 13801 13347 17
45 37 2
2 0 724 34888 72 42012 0 0 72 19 14590 14150 17
48 34 1
2 0 724 34720 72 42100 0 0 152 26 10802 10579 13
35 38 13
2 0 692 34044 72 42140 0 0 20 17 14573 14097 18
47 34 1
3 0 692 34116 72 42072 0 0 20 0 14542 14052 17
47 35 1
0 0 692 38540 72 42400 0 0 220 0 11981 11613 16
42 42 1
0 1 692 32224 72 43252 0 0 1460 0 4313 4377 13 13
61 13
5 0 692 31380 72 45084 0 0 1816 0 3241 3403 3 8
28 61
3 0 660 29552 72 47012 0 0 1060 14 14843 15905 22
49 14 15
2 0 660 17460 72 55592 0 0 7856 71 14869 16513 23
51 15 12
1 1 660 16732 72 56520 0 0 892 88 10888 17232 36
53 6 6
3 0 660 16964 72 57788 0 0 640 0 15741 18390 26
57 11 6
2 1 660 14448 72 60112 0 0 1556 40 14830 16736 22
50 15 13
3 0 628 10300 72 62424 0 0 996 0 15679 17393 24
54 12 11
3 0 628 9412 72 64216 0 0 504 0 17873 19182 27
60 9 4
3 0 628 7724 72 65480 0 0 332 0 16445 18588 25
57 11 8
1 1 628 12284 72 61088 0 0 212 5144 15099 17314 30
62 5 3
2 0 628 17264 72 53488 0 0 60 0 11906 17634 40
53 2 5
3 0 600 17440 72 53996 0 0 388 34 17648 20469 27
63 6 5
2 0 600 16684 72 54224 0 0 456 182 17288 20085 27
62 8 3
3 0 600 15112 72 56100 0 0 792 8 15689 17476 24
55 11 10
3 0 600 11752 72 58464 0 0 772 0 18287 20021 27
62 7 5
0 2 600 8628 72 60744 0 0 1536 0 15871 18785 24
56 12 8
3 0 568 8120 72 61820 0 0 820 38 10480 12030 17
35 37 11
2 0 568 4548 72 49460 0 0 5408 3156 7301 7390 7 27
60 6
1 0 6792 4592 72 41252 0 0 3172 0 6733 6601 12 38
35 16
1 0 1212 135860 72 41972 0 0 3192 136 5099 5011 10 34
31 25
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-10 14:01 ` James Dingwall
@ 2013-12-10 14:25 ` Jan Beulich
2013-12-10 14:52 ` James Dingwall
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2013-12-10 14:25 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
>>> On 10.12.13 at 15:01, James Dingwall <james.dingwall@zynstra.com> wrote:
> Jan Beulich wrote:
>>>>> On 09.12.13 at 18:50, James Dingwall <james.dingwall@zynstra.com> wrote:
>>> Since 3.11 I have noticed that the OOM killer quite frequently triggers
>>> in my Xen guest domains which use ballooning to increase/decrease their
>>> memory allocation according to their requirements. One example domain I
>>> have has a maximum memory setting of ~1.5Gb but it usually idles at
>>> ~300Mb, it is also configured with 2Gb swap which is almost 100% free.
>>>
>>> # free
>>> total used free shared buffers cached
>>> Mem: 272080 248108 23972 0 1448 63064
>>> -/+ buffers/cache: 183596 88484
>>> Swap: 2097148 8 2097140
>>>
>>> There is plenty of available free memory in the hypervisor to balloon to
>>> the maximum size:
>>> # xl info | grep free_mem
>>> free_memory : 14923
>> But you don't tell us how you trigger the process of re-filling
>> memory. Yet that's the crucial point here; the fact that there is
>> enough memory available in the hypervisor is only a necessary
>> prerequisite.
> My guest systems are Gentoo based and most often the problems happen
> while running emerge (Python script). The example trace was taken from
> an emerge command launched via a cron script which runs emerge --sync ;
> emerge --deep --newuse -p -u world. Almost all the other times I have
> seen the behaviour have been during emerge --deep --newuse -u world
> runs, most often during the build of large packages such as kdelibs,
> seamonkey or libreoffice. However occasionally I have seen it with
> smaller builds or during the package merge phase where files are indexed
> and copied in to the live filesystem.
I don't think I understand what you're trying to tell me, or in what
way this answers the question.
> I have done some testing with the program below which successfully fills
> all memory and swap before being killed. One thought I had was that
> perhaps there was some issue around a balloon down/up when the rate at
> which memory is being requested and released is high. I will try
> another program with a random pattern of malloc/free to see if I can
> make a test case to help with a bisect.
Again - the question is not how to drive your system out of
memory, but what entity (if any) it is that is supposed to react on
memory becoming tight and triggering the balloon driver to re-
populate pages. One such thing could be xenballoond (which is
gone in the unstable tree, i.e. what is to become 4.4).
Jan
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-10 14:25 ` Jan Beulich
@ 2013-12-10 14:52 ` James Dingwall
2013-12-10 14:59 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2013-12-10 14:52 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel
Jan Beulich wrote:
>>>> On 10.12.13 at 15:01, James Dingwall <james.dingwall@zynstra.com> wrote:
>> Jan Beulich wrote:
>>>>>> On 09.12.13 at 18:50, James Dingwall <james.dingwall@zynstra.com> wrote:
>>>> Since 3.11 I have noticed that the OOM killer quite frequently triggers
>>>> in my Xen guest domains which use ballooning to increase/decrease their
>>>> memory allocation according to their requirements. One example domain I
>>>> have has a maximum memory setting of ~1.5Gb but it usually idles at
>>>> ~300Mb, it is also configured with 2Gb swap which is almost 100% free.
>>>>
>>>> # free
>>>> total used free shared buffers cached
>>>> Mem: 272080 248108 23972 0 1448 63064
>>>> -/+ buffers/cache: 183596 88484
>>>> Swap: 2097148 8 2097140
>>>>
>>>> There is plenty of available free memory in the hypervisor to balloon to
>>>> the maximum size:
>>>> # xl info | grep free_mem
>>>> free_memory : 14923
>>> But you don't tell us how you trigger the process of re-filling
>>> memory. Yet that's the crucial point here; the fact that there is
>>> enough memory available in the hypervisor is only a necessary
>>> prerequisite.
>> My guest systems are Gentoo based and most often the problems happen
>> while running emerge (Python script). The example trace was taken from
>> an emerge command launched via a cron script which runs emerge --sync ;
>> emerge --deep --newuse -p -u world. Almost all the other times I have
>> seen the behaviour have been during emerge --deep --newuse -u world
>> runs, most often during the build of large packages such as kdelibs,
>> seamonkey or libreoffice. However occasionally I have seen it with
>> smaller builds or during the package merge phase where files are indexed
>> and copied in to the live filesystem.
> I don't think I understand what you're trying to tell me, or in what
> way this answers the question.
>
>> I have done some testing with the program below which successfully fills
>> all memory and swap before being killed. One thought I had was that
>> perhaps there was some issue around a balloon down/up when the rate at
>> which memory is being requested and released is high. I will try
>> another program with a random pattern of malloc/free to see if I can
>> make a test case to help with a bisect.
> Again - the question is not how to drive your system out of
> memory, but what entity (if any) it is that is supposed to react on
> memory becoming tight and triggering the balloon driver to re-
> populate pages. One such thing could be xenballoond (which is
> gone in the unstable tree, i.e. what is to become 4.4).
>
> Jan
Sorry, I misunderstood the question. I'm just relying on the built in
kernel behaviour to balloon up/down as and when memory is required I
wasn't aware there were multiple ways that this could be achieved. I'm
not running anything in user space, so does tmem answer the question?
When I modprobe tmem the kernel logs:
[ 18.518714] xen:tmem: frontswap enabled, RAM provided by Xen
Transcendent Memory
[ 18.518739] xen:tmem: cleancache enabled, RAM provided by Xen
Transcendent Memory
[ 18.518741] xen_selfballoon: Initializing Xen selfballooning driver
[ 18.518742] xen_selfballoon: Initializing frontswap selfshrinking driver
The kernel command line for dom0 and domU contains tmem, the command
line for xen contains tmem tmem_dedup=on tmem_compress=on
James
>
--
*James Dingwall*
Script Monkey
zynstra-signature-logo <http://www.zynstra.com/>twitter-black
<http://www.twitter.com/zynstra>linkedin-black
<http://www.linkedin.com/company/zynstra>
Zynstra is a private limited company registered in England and Wales
(registered number 07864369). Our registered office is 5 New Street
Square, London, EC4A 3TW and our headquarters are at Bath Ventures,
Broad Quay, Bath, BA1 1UD.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-09 21:48 ` Konrad Rzeszutek Wilk
@ 2013-12-10 14:52 ` James Dingwall
2013-12-10 15:27 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2013-12-10 14:52 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, bob.liu; +Cc: xen-devel
Konrad Rzeszutek Wilk wrote:
> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>> Hi,
>>
>> Since 3.11 I have noticed that the OOM killer quite frequently
>> triggers in my Xen guest domains which use ballooning to
>> increase/decrease their memory allocation according to their
>> requirements. One example domain I have has a maximum memory
>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>> configured with 2Gb swap which is almost 100% free.
>>
>> # free
>> total used free shared buffers cached
>> Mem: 272080 248108 23972 0 1448 63064
>> -/+ buffers/cache: 183596 88484
>> Swap: 2097148 8 2097140
>>
>> There is plenty of available free memory in the hypervisor to
>> balloon to the maximum size:
>> # xl info | grep free_mem
>> free_memory : 14923
>>
>> An example trace (they are always the same) from the oom killer in
>> 3.12 is added below. So far I have not been able to reproduce this
>> at will so it is difficult to start bisecting it to see if a
>> particular change introduced this. However it does seem that the
>> behaviour is wrong because a) ballooning could give the guest more
>> memory, b) there is lots of swap available which could be used as a
>> fallback.
>>
>> If other information could help or there are more tests that I could
>> run then please let me know.
> I presume you have enabled 'tmem' both in the hypervisor and in
> the guest right?
Yes, domU and dom0 both have the tmem module loaded and tmem
tmem_dedup=on tmem_compress=on is given on the xen command line.
>> Thanks,
>> James
>>
>>
>>
>>
>> [473233.777271] emerge invoked oom-killer: gfp_mask=0x280da,
>> order=0, oom_score_adj=0
>> [473233.777279] CPU: 0 PID: 22159 Comm: emerge Tainted: G W 3.12.0 #80
>> [473233.777282] ffff88000599f6f8 ffff8800117bda58 ffffffff81489a80
>> ffff88004760e8e8
>> [473233.777286] ffff88000599f1c0 ffff8800117bdaf8 ffffffff81487577
>> ffff8800117bdaa8
>> [473233.777289] ffffffff810f8c0f ffff8800117bda88 ffffffff81006dc8
>> ffff8800117bda98
>> [473233.777293] Call Trace:
>> [473233.777305] [<ffffffff81489a80>] dump_stack+0x46/0x58
>> [473233.777310] [<ffffffff81487577>] dump_header.isra.9+0x6d/0x1cc
>> [473233.777315] [<ffffffff810f8c0f>] ? super_cache_count+0xa8/0xb8
>> [473233.777321] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
>> [473233.777324] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
>> [473233.777328] [<ffffffff8148f336>] ?
>> _raw_spin_unlock_irqrestore+0x47/0x62
>> [473233.777333] [<ffffffff812915d3>] ? ___ratelimit+0xcb/0xe8
>> [473233.777338] [<ffffffff810b2aa7>] oom_kill_process+0x70/0x2fd
>> [473233.777343] [<ffffffff81048775>] ? has_ns_capability_noaudit+0x12/0x19
>> [473233.777346] [<ffffffff8104878e>] ? has_capability_noaudit+0x12/0x14
>> [473233.777349] [<ffffffff810b31c6>] out_of_memory+0x31b/0x34e
>> [473233.777353] [<ffffffff810b72f0>] __alloc_pages_nodemask+0x65b/0x792
>> [473233.777358] [<ffffffff810e3c1b>] alloc_pages_vma+0xd0/0x10c
>> [473233.777361] [<ffffffff81003f69>] ?
>> __raw_callee_save_xen_pmd_val+0x11/0x1e
>> [473233.777365] [<ffffffff810cf685>] handle_mm_fault+0x6d4/0xd54
>> [473233.777371] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
>> [473233.777374] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
>> [473233.777378] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
>> [473233.777382] [<ffffffff810676c7>] ? sched_clock_local+0x12/0x75
>> [473233.777386] [<ffffffff810a44b4>] ? __acct_update_integrals+0xb4/0xbf
>> [473233.777389] [<ffffffff810a4827>] ? acct_account_cputime+0x17/0x19
>> [473233.777392] [<ffffffff81067bc0>] ? account_user_time+0x67/0x92
>> [473233.777395] [<ffffffff810680b3>] ? vtime_account_user+0x4d/0x52
>> [473233.777398] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
>> [473233.777401] [<ffffffff8148f9d8>] page_fault+0x28/0x30
>> [473233.777403] Mem-Info:
>> [473233.777405] Node 0 DMA per-cpu:
>> [473233.777408] CPU 0: hi: 0, btch: 1 usd: 0
>> [473233.777409] CPU 1: hi: 0, btch: 1 usd: 0
>> [473233.777411] CPU 2: hi: 0, btch: 1 usd: 0
>> [473233.777412] CPU 3: hi: 0, btch: 1 usd: 0
>> [473233.777413] Node 0 DMA32 per-cpu:
>> [473233.777415] CPU 0: hi: 186, btch: 31 usd: 103
>> [473233.777417] CPU 1: hi: 186, btch: 31 usd: 110
>> [473233.777419] CPU 2: hi: 186, btch: 31 usd: 175
>> [473233.777420] CPU 3: hi: 186, btch: 31 usd: 182
>> [473233.777421] Node 0 Normal per-cpu:
>> [473233.777423] CPU 0: hi: 0, btch: 1 usd: 0
>> [473233.777424] CPU 1: hi: 0, btch: 1 usd: 0
>> [473233.777426] CPU 2: hi: 0, btch: 1 usd: 0
>> [473233.777427] CPU 3: hi: 0, btch: 1 usd: 0
>> [473233.777433] active_anon:35740 inactive_anon:33812 isolated_anon:0
>> active_file:4672 inactive_file:11607 isolated_file:0
>> unevictable:0 dirty:4 writeback:0 unstable:0
>> free:2067 slab_reclaimable:3583 slab_unreclaimable:3524
>> mapped:3329 shmem:324 pagetables:2003 bounce:0
>> free_cma:0
>> [473233.777435] Node 0 DMA free:4200kB min:60kB low:72kB high:88kB
>> active_anon:264kB inactive_anon:456kB active_file:140kB
>> inactive_file:340kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:15996kB managed:6176kB mlocked:0kB
>> dirty:0kB writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:96kB
>> slab_unreclaimable:112kB kernel_stack:24kB pagetables:24kB
>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:33270 all_unreclaimable? yes
>> [473233.777443] lowmem_reserve[]: 0 1036 1036 1036
>> [473233.777447] Node 0 DMA32 free:4060kB min:4084kB low:5104kB
>> high:6124kB active_anon:41256kB inactive_anon:33128kB
>> active_file:8544kB inactive_file:14312kB unevictable:0kB
>> isolated(anon):0kB isolated(file):0kB present:1163264kB
>> managed:165780kB mlocked:0kB dirty:0kB writeback:0kB mapped:6428kB
>> shmem:604kB slab_reclaimable:9800kB slab_unreclaimable:12908kB
>> kernel_stack:1832kB pagetables:5924kB unstable:0kB bounce:0kB
>> free_cma:0kB writeback_tmp:0kB pages_scanned:152386
>> all_unreclaimable? yes
>> [473233.777454] lowmem_reserve[]: 0 0 0 0
>> [473233.777457] Node 0 Normal free:8kB min:0kB low:0kB high:0kB
>> active_anon:101440kB inactive_anon:101664kB active_file:10004kB
>> inactive_file:31776kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:393216kB managed:256412kB mlocked:0kB
>> dirty:16kB writeback:0kB mapped:6788kB shmem:692kB
>> slab_reclaimable:4436kB slab_unreclaimable:1076kB kernel_stack:136kB
>> pagetables:2064kB unstable:0kB bounce:0kB free_cma:0kB
>> writeback_tmp:0kB pages_scanned:368809 all_unreclaimable? yes
>> [473233.777464] lowmem_reserve[]: 0 0 0 0
>> [473233.777467] Node 0 DMA: 41*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB
>> (R) 1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R)
>> 0*4096kB = 4196kB
>> [473233.777480] Node 0 DMA32: 1015*4kB (U) 0*8kB 0*16kB 0*32kB
>> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4060kB
>> [473233.777490] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>> [473233.777498] 5018 total pagecache pages
>> [473233.777500] 16 pages in swap cache
>> [473233.777501] Swap cache stats: add 2829330, delete 2829314, find
>> 344059/481859
>> [473233.777503] Free swap = 2096980kB
>> [473233.777503] Total swap = 2097148kB
>> [473233.794497] 557055 pages RAM
>> [473233.794500] 189326 pages reserved
>> [473233.794501] 544934 pages shared
>> [473233.794502] 358441 pages non-shared
>> [473233.794504] [ pid ] uid tgid total_vm rss nr_ptes
>> swapents oom_score_adj name
>> [473233.794523] [ 6597] 0 6597 8156 252 20 0
>> -1000 udevd
>> [473233.794530] [ 7194] 0 7194 2232 137 10 0 0 metalog
>> [473233.794534] [ 7195] 0 7195 2223 31 10 3 0 metalog
>> [473233.794537] [ 7211] 0 7211 1064 35 8 0
>> 0 acpid
>> [473233.794546] [ 7227] 702 7227 4922 183 14 0 0
>> dbus-daemon
>> [473233.794553] [ 7427] 0 7427 13630 179 29 15 0 rpcbind
>> [473233.794560] [ 7442] 0 7442 14743 332 32 0 0
>> rpc.statd
>> [473233.794569] [ 7472] 0 7472 6365 115 17 0 0
>> rpc.idmapd
>> [473233.794576] [ 7488] 0 7488 43602 349 40 0 0 cupsd
>> [473233.794583] [ 7512] 0 7512 14856 243 30 0 0
>> rpc.mountd
>> [473233.794592] [ 7552] 0 7552 148819 940 68 0 0
>> automount
>> [473233.794595] [ 7592] 0 7592 16006 233 32 0 -1000 sshd
>> [473233.794598] [ 7608] 0 7608 87672 2257 128 6
>> 0 apache2
>> [473233.794601] [ 7633] 0 7633 521873 631 56 0 0
>> console-kit-dae
>> [473233.794604] [ 7713] 106 7713 15453 295 34 2 0 nrpe
>> [473233.794607] [ 7719] 986 7719 91303 798 41 0 0 polkitd
>> [473233.794610] [ 7757] 123 7757 7330 259 17 0 0 ntpd
>> [473233.794613] [ 7845] 0 7845 3583 94 12 0 0 master
>> [473233.794616] [ 7847] 207 7847 17745 311 38 0 0 qmgr
>> [473233.794619] [ 7861] 65534 7861 2101 21 9 19
>> 0 rwhod
>> [473233.794622] [ 7864] 65534 7864 2101 99 9 0
>> 0 rwhod
>> [473233.794625] [ 7876] 0 7876 48582 533 47 19 0 smbd
>> [473233.794628] [ 7881] 0 7881 44277 372 38 0 0 nmbd
>> [473233.794631] [ 7895] 0 7895 48646 621 45 18 0 smbd
>> [473233.794634] [ 7902] 2 7902 1078 39 8 4 0 slpd
>> [473233.794637] [ 7917] 0 7917 38452 1073 28 1 0 snmpd
>> [473233.794640] [ 7945] 0 7945 27552 58 9 0 0 cron
>> [473233.794648] [ 7993] 0 7993 201378 5432 63 39 0 nscd
>> [473233.794658] [ 8064] 0 8064 1060 28 7 0
>> 0 agetty
>> [473233.794664] [ 8065] 0 8065 26507 29 9 0
>> 0 agetty
>> [473233.794667] [ 8066] 0 8066 26507 29 9 0
>> 0 agetty
>> [473233.794670] [ 8067] 0 8067 26507 28 9 0
>> 0 agetty
>> [473233.794673] [ 8068] 0 8068 26507 28 8 0
>> 0 agetty
>> [473233.794678] [ 8069] 0 8069 26507 30 9 0
>> 0 agetty
>> [473233.794686] [ 8070] 0 8070 26507 30 9 0
>> 0 agetty
>> [473233.794693] [ 8071] 0 8071 26507 30 9 0
>> 0 agetty
>> [473233.794701] [ 8072] 0 8072 26507 28 9 0
>> 0 agetty
>> [473233.794708] [ 8316] 0 8316 3736 83 11 6 0
>> ssh-agent
>> [473233.794712] [ 8341] 0 8341 3390 66 12 7 0
>> gpg-agent
>> [473233.794716] [ 2878] 81 2878 88431 2552 121 5
>> 0 apache2
>> [473233.794718] [ 2879] 81 2879 88431 2552 121 5
>> 0 apache2
>> [473233.794721] [ 2880] 81 2880 88431 2552 121 5
>> 0 apache2
>> [473233.794724] [ 2881] 81 2881 88431 2552 121 5
>> 0 apache2
>> [473233.794727] [ 2882] 81 2882 88431 2552 121 5
>> 0 apache2
>> [473233.794734] [ 3523] 81 3523 88431 2552 121 5
>> 0 apache2
>> [473233.794737] [30259] 1000 30259 3736 118 11 0 0
>> ssh-agent
>> [473233.794741] [30284] 1000 30284 3390 141 12 0 0
>> gpg-agent
>> [473233.794745] [21263] 207 21263 17703 771 39 1 0 pickup
>> [473233.794748] [21663] 0 21663 30743 228 16 0 0 cron
>> [473233.794751] [21665] 0 21665 2980 392 12 0 0
>> gentoosync.sh
>> [473233.794755] [22158] 0 22158 3181 273 12 0 0 sendmail
>> [473233.794757] [22159] 0 22159 77646 54920 158 0
>> 0 emerge
>> [473233.794760] [22160] 0 22160 1068 85 8 0 0 tail
>> [473233.794764] [22161] 0 22161 3173 277 11 0 0 postdrop
>> [473233.794768] Out of memory: Kill process 22159 (emerge) score 57
>> or sacrifice child
>> [473233.794771] Killed process 22159 (emerge) total-vm:310584kB,
>> anon-rss:215840kB, file-rss:3840kB
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-10 14:52 ` James Dingwall
@ 2013-12-10 14:59 ` Jan Beulich
2013-12-10 15:16 ` James Dingwall
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2013-12-10 14:59 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
>>> On 10.12.13 at 15:52, James Dingwall <james.dingwall@zynstra.com> wrote:
> Sorry, I misunderstood the question. I'm just relying on the built in
> kernel behaviour to balloon up/down as and when memory is required I
> wasn't aware there were multiple ways that this could be achieved. I'm
> not running anything in user space, so does tmem answer the question?
Yes it does. And it means I'll defer to the tmem maintainer (who
I saw already also replied to you). You're aware that as far as
we (xenproject.org) are concerned, tmem is considered
unsupported/experimental.
Jan
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-10 14:59 ` Jan Beulich
@ 2013-12-10 15:16 ` James Dingwall
0 siblings, 0 replies; 42+ messages in thread
From: James Dingwall @ 2013-12-10 15:16 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel
Jan Beulich wrote:
>>>> On 10.12.13 at 15:52, James Dingwall <james.dingwall@zynstra.com> wrote:
>> Sorry, I misunderstood the question. I'm just relying on the built in
>> kernel behaviour to balloon up/down as and when memory is required I
>> wasn't aware there were multiple ways that this could be achieved. I'm
>> not running anything in user space, so does tmem answer the question?
> Yes it does. And it means I'll defer to the tmem maintainer (who
> I saw already also replied to you). You're aware that as far as
> we (xenproject.org) are concerned, tmem is considered
> unsupported/experimental.
I wasn't aware that it was considered experimental but I shall bear that
in mind. Having a look through the git history shows some changes in
the tmem and ballooning code which I could try experimenting with.
Thanks,
James
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-10 14:52 ` James Dingwall
@ 2013-12-10 15:27 ` Konrad Rzeszutek Wilk
2013-12-11 7:22 ` Bob Liu
0 siblings, 1 reply; 42+ messages in thread
From: Konrad Rzeszutek Wilk @ 2013-12-10 15:27 UTC (permalink / raw)
To: James Dingwall, bob.liu; +Cc: xen-devel
On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
> Konrad Rzeszutek Wilk wrote:
> >On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
> >>Hi,
> >>
> >>Since 3.11 I have noticed that the OOM killer quite frequently
> >>triggers in my Xen guest domains which use ballooning to
> >>increase/decrease their memory allocation according to their
> >>requirements. One example domain I have has a maximum memory
> >>setting of ~1.5Gb but it usually idles at ~300Mb, it is also
> >>configured with 2Gb swap which is almost 100% free.
> >>
> >># free
> >> total used free shared buffers cached
> >>Mem: 272080 248108 23972 0 1448 63064
> >>-/+ buffers/cache: 183596 88484
> >>Swap: 2097148 8 2097140
> >>
> >>There is plenty of available free memory in the hypervisor to
> >>balloon to the maximum size:
> >># xl info | grep free_mem
> >>free_memory : 14923
> >>
> >>An example trace (they are always the same) from the oom killer in
> >>3.12 is added below. So far I have not been able to reproduce this
> >>at will so it is difficult to start bisecting it to see if a
> >>particular change introduced this. However it does seem that the
> >>behaviour is wrong because a) ballooning could give the guest more
> >>memory, b) there is lots of swap available which could be used as a
> >>fallback.
Keep in mind that swap with tmem is actually no more swap. Heh, that
sounds odd -but basically pages that are destined for swap end up
going in the tmem code which pipes them up to the hypervisor.
> >>
> >>If other information could help or there are more tests that I could
> >>run then please let me know.
> >I presume you have enabled 'tmem' both in the hypervisor and in
> >the guest right?
> Yes, domU and dom0 both have the tmem module loaded and tmem
> tmem_dedup=on tmem_compress=on is given on the xen command line.
Excellent. The odd thing is that your swap is not used that much, but
it should be (as that is part of what the self-balloon is suppose to do).
Bob, you had a patch for the logic of how self-balloon is suppose
to account for the slab - would this be relevant to this problem?
Thank you.
> >>Thanks,
> >>James
> >>
> >>
> >>
> >>
> >>[473233.777271] emerge invoked oom-killer: gfp_mask=0x280da,
> >>order=0, oom_score_adj=0
> >>[473233.777279] CPU: 0 PID: 22159 Comm: emerge Tainted: G W 3.12.0 #80
> >>[473233.777282] ffff88000599f6f8 ffff8800117bda58 ffffffff81489a80
> >>ffff88004760e8e8
> >>[473233.777286] ffff88000599f1c0 ffff8800117bdaf8 ffffffff81487577
> >>ffff8800117bdaa8
> >>[473233.777289] ffffffff810f8c0f ffff8800117bda88 ffffffff81006dc8
> >>ffff8800117bda98
> >>[473233.777293] Call Trace:
> >>[473233.777305] [<ffffffff81489a80>] dump_stack+0x46/0x58
> >>[473233.777310] [<ffffffff81487577>] dump_header.isra.9+0x6d/0x1cc
> >>[473233.777315] [<ffffffff810f8c0f>] ? super_cache_count+0xa8/0xb8
> >>[473233.777321] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
> >>[473233.777324] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
> >>[473233.777328] [<ffffffff8148f336>] ?
> >>_raw_spin_unlock_irqrestore+0x47/0x62
> >>[473233.777333] [<ffffffff812915d3>] ? ___ratelimit+0xcb/0xe8
> >>[473233.777338] [<ffffffff810b2aa7>] oom_kill_process+0x70/0x2fd
> >>[473233.777343] [<ffffffff81048775>] ? has_ns_capability_noaudit+0x12/0x19
> >>[473233.777346] [<ffffffff8104878e>] ? has_capability_noaudit+0x12/0x14
> >>[473233.777349] [<ffffffff810b31c6>] out_of_memory+0x31b/0x34e
> >>[473233.777353] [<ffffffff810b72f0>] __alloc_pages_nodemask+0x65b/0x792
> >>[473233.777358] [<ffffffff810e3c1b>] alloc_pages_vma+0xd0/0x10c
> >>[473233.777361] [<ffffffff81003f69>] ?
> >>__raw_callee_save_xen_pmd_val+0x11/0x1e
> >>[473233.777365] [<ffffffff810cf685>] handle_mm_fault+0x6d4/0xd54
> >>[473233.777371] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
> >>[473233.777374] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
> >>[473233.777378] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
> >>[473233.777382] [<ffffffff810676c7>] ? sched_clock_local+0x12/0x75
> >>[473233.777386] [<ffffffff810a44b4>] ? __acct_update_integrals+0xb4/0xbf
> >>[473233.777389] [<ffffffff810a4827>] ? acct_account_cputime+0x17/0x19
> >>[473233.777392] [<ffffffff81067bc0>] ? account_user_time+0x67/0x92
> >>[473233.777395] [<ffffffff810680b3>] ? vtime_account_user+0x4d/0x52
> >>[473233.777398] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
> >>[473233.777401] [<ffffffff8148f9d8>] page_fault+0x28/0x30
> >>[473233.777403] Mem-Info:
> >>[473233.777405] Node 0 DMA per-cpu:
> >>[473233.777408] CPU 0: hi: 0, btch: 1 usd: 0
> >>[473233.777409] CPU 1: hi: 0, btch: 1 usd: 0
> >>[473233.777411] CPU 2: hi: 0, btch: 1 usd: 0
> >>[473233.777412] CPU 3: hi: 0, btch: 1 usd: 0
> >>[473233.777413] Node 0 DMA32 per-cpu:
> >>[473233.777415] CPU 0: hi: 186, btch: 31 usd: 103
> >>[473233.777417] CPU 1: hi: 186, btch: 31 usd: 110
> >>[473233.777419] CPU 2: hi: 186, btch: 31 usd: 175
> >>[473233.777420] CPU 3: hi: 186, btch: 31 usd: 182
> >>[473233.777421] Node 0 Normal per-cpu:
> >>[473233.777423] CPU 0: hi: 0, btch: 1 usd: 0
> >>[473233.777424] CPU 1: hi: 0, btch: 1 usd: 0
> >>[473233.777426] CPU 2: hi: 0, btch: 1 usd: 0
> >>[473233.777427] CPU 3: hi: 0, btch: 1 usd: 0
> >>[473233.777433] active_anon:35740 inactive_anon:33812 isolated_anon:0
> >> active_file:4672 inactive_file:11607 isolated_file:0
> >> unevictable:0 dirty:4 writeback:0 unstable:0
> >> free:2067 slab_reclaimable:3583 slab_unreclaimable:3524
> >> mapped:3329 shmem:324 pagetables:2003 bounce:0
> >> free_cma:0
> >>[473233.777435] Node 0 DMA free:4200kB min:60kB low:72kB high:88kB
> >>active_anon:264kB inactive_anon:456kB active_file:140kB
> >>inactive_file:340kB unevictable:0kB isolated(anon):0kB
> >>isolated(file):0kB present:15996kB managed:6176kB mlocked:0kB
> >>dirty:0kB writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:96kB
> >>slab_unreclaimable:112kB kernel_stack:24kB pagetables:24kB
> >>unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> >>pages_scanned:33270 all_unreclaimable? yes
> >>[473233.777443] lowmem_reserve[]: 0 1036 1036 1036
> >>[473233.777447] Node 0 DMA32 free:4060kB min:4084kB low:5104kB
> >>high:6124kB active_anon:41256kB inactive_anon:33128kB
> >>active_file:8544kB inactive_file:14312kB unevictable:0kB
> >>isolated(anon):0kB isolated(file):0kB present:1163264kB
> >>managed:165780kB mlocked:0kB dirty:0kB writeback:0kB mapped:6428kB
> >>shmem:604kB slab_reclaimable:9800kB slab_unreclaimable:12908kB
> >>kernel_stack:1832kB pagetables:5924kB unstable:0kB bounce:0kB
> >>free_cma:0kB writeback_tmp:0kB pages_scanned:152386
> >>all_unreclaimable? yes
> >>[473233.777454] lowmem_reserve[]: 0 0 0 0
> >>[473233.777457] Node 0 Normal free:8kB min:0kB low:0kB high:0kB
> >>active_anon:101440kB inactive_anon:101664kB active_file:10004kB
> >>inactive_file:31776kB unevictable:0kB isolated(anon):0kB
> >>isolated(file):0kB present:393216kB managed:256412kB mlocked:0kB
> >>dirty:16kB writeback:0kB mapped:6788kB shmem:692kB
> >>slab_reclaimable:4436kB slab_unreclaimable:1076kB kernel_stack:136kB
> >>pagetables:2064kB unstable:0kB bounce:0kB free_cma:0kB
> >>writeback_tmp:0kB pages_scanned:368809 all_unreclaimable? yes
> >>[473233.777464] lowmem_reserve[]: 0 0 0 0
> >>[473233.777467] Node 0 DMA: 41*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB
> >>(R) 1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R)
> >>0*4096kB = 4196kB
> >>[473233.777480] Node 0 DMA32: 1015*4kB (U) 0*8kB 0*16kB 0*32kB
> >>0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4060kB
> >>[473233.777490] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
> >>0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> >>[473233.777498] 5018 total pagecache pages
> >>[473233.777500] 16 pages in swap cache
> >>[473233.777501] Swap cache stats: add 2829330, delete 2829314, find
> >>344059/481859
> >>[473233.777503] Free swap = 2096980kB
> >>[473233.777503] Total swap = 2097148kB
> >>[473233.794497] 557055 pages RAM
> >>[473233.794500] 189326 pages reserved
> >>[473233.794501] 544934 pages shared
> >>[473233.794502] 358441 pages non-shared
> >>[473233.794504] [ pid ] uid tgid total_vm rss nr_ptes
> >>swapents oom_score_adj name
> >>[473233.794523] [ 6597] 0 6597 8156 252 20 0
> >>-1000 udevd
> >>[473233.794530] [ 7194] 0 7194 2232 137 10 0 0 metalog
> >>[473233.794534] [ 7195] 0 7195 2223 31 10 3 0 metalog
> >>[473233.794537] [ 7211] 0 7211 1064 35 8 0
> >>0 acpid
> >>[473233.794546] [ 7227] 702 7227 4922 183 14 0 0
> >>dbus-daemon
> >>[473233.794553] [ 7427] 0 7427 13630 179 29 15 0 rpcbind
> >>[473233.794560] [ 7442] 0 7442 14743 332 32 0 0
> >>rpc.statd
> >>[473233.794569] [ 7472] 0 7472 6365 115 17 0 0
> >>rpc.idmapd
> >>[473233.794576] [ 7488] 0 7488 43602 349 40 0 0 cupsd
> >>[473233.794583] [ 7512] 0 7512 14856 243 30 0 0
> >>rpc.mountd
> >>[473233.794592] [ 7552] 0 7552 148819 940 68 0 0
> >>automount
> >>[473233.794595] [ 7592] 0 7592 16006 233 32 0 -1000 sshd
> >>[473233.794598] [ 7608] 0 7608 87672 2257 128 6
> >>0 apache2
> >>[473233.794601] [ 7633] 0 7633 521873 631 56 0 0
> >>console-kit-dae
> >>[473233.794604] [ 7713] 106 7713 15453 295 34 2 0 nrpe
> >>[473233.794607] [ 7719] 986 7719 91303 798 41 0 0 polkitd
> >>[473233.794610] [ 7757] 123 7757 7330 259 17 0 0 ntpd
> >>[473233.794613] [ 7845] 0 7845 3583 94 12 0 0 master
> >>[473233.794616] [ 7847] 207 7847 17745 311 38 0 0 qmgr
> >>[473233.794619] [ 7861] 65534 7861 2101 21 9 19
> >>0 rwhod
> >>[473233.794622] [ 7864] 65534 7864 2101 99 9 0
> >>0 rwhod
> >>[473233.794625] [ 7876] 0 7876 48582 533 47 19 0 smbd
> >>[473233.794628] [ 7881] 0 7881 44277 372 38 0 0 nmbd
> >>[473233.794631] [ 7895] 0 7895 48646 621 45 18 0 smbd
> >>[473233.794634] [ 7902] 2 7902 1078 39 8 4 0 slpd
> >>[473233.794637] [ 7917] 0 7917 38452 1073 28 1 0 snmpd
> >>[473233.794640] [ 7945] 0 7945 27552 58 9 0 0 cron
> >>[473233.794648] [ 7993] 0 7993 201378 5432 63 39 0 nscd
> >>[473233.794658] [ 8064] 0 8064 1060 28 7 0
> >>0 agetty
> >>[473233.794664] [ 8065] 0 8065 26507 29 9 0
> >>0 agetty
> >>[473233.794667] [ 8066] 0 8066 26507 29 9 0
> >>0 agetty
> >>[473233.794670] [ 8067] 0 8067 26507 28 9 0
> >>0 agetty
> >>[473233.794673] [ 8068] 0 8068 26507 28 8 0
> >>0 agetty
> >>[473233.794678] [ 8069] 0 8069 26507 30 9 0
> >>0 agetty
> >>[473233.794686] [ 8070] 0 8070 26507 30 9 0
> >>0 agetty
> >>[473233.794693] [ 8071] 0 8071 26507 30 9 0
> >>0 agetty
> >>[473233.794701] [ 8072] 0 8072 26507 28 9 0
> >>0 agetty
> >>[473233.794708] [ 8316] 0 8316 3736 83 11 6 0
> >>ssh-agent
> >>[473233.794712] [ 8341] 0 8341 3390 66 12 7 0
> >>gpg-agent
> >>[473233.794716] [ 2878] 81 2878 88431 2552 121 5
> >>0 apache2
> >>[473233.794718] [ 2879] 81 2879 88431 2552 121 5
> >>0 apache2
> >>[473233.794721] [ 2880] 81 2880 88431 2552 121 5
> >>0 apache2
> >>[473233.794724] [ 2881] 81 2881 88431 2552 121 5
> >>0 apache2
> >>[473233.794727] [ 2882] 81 2882 88431 2552 121 5
> >>0 apache2
> >>[473233.794734] [ 3523] 81 3523 88431 2552 121 5
> >>0 apache2
> >>[473233.794737] [30259] 1000 30259 3736 118 11 0 0
> >>ssh-agent
> >>[473233.794741] [30284] 1000 30284 3390 141 12 0 0
> >>gpg-agent
> >>[473233.794745] [21263] 207 21263 17703 771 39 1 0 pickup
> >>[473233.794748] [21663] 0 21663 30743 228 16 0 0 cron
> >>[473233.794751] [21665] 0 21665 2980 392 12 0 0
> >>gentoosync.sh
> >>[473233.794755] [22158] 0 22158 3181 273 12 0 0 sendmail
> >>[473233.794757] [22159] 0 22159 77646 54920 158 0
> >>0 emerge
> >>[473233.794760] [22160] 0 22160 1068 85 8 0 0 tail
> >>[473233.794764] [22161] 0 22161 3173 277 11 0 0 postdrop
> >>[473233.794768] Out of memory: Kill process 22159 (emerge) score 57
> >>or sacrifice child
> >>[473233.794771] Killed process 22159 (emerge) total-vm:310584kB,
> >>anon-rss:215840kB, file-rss:3840kB
> >>
> >>
> >>_______________________________________________
> >>Xen-devel mailing list
> >>Xen-devel@lists.xen.org
> >>http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-10 15:27 ` Konrad Rzeszutek Wilk
@ 2013-12-11 7:22 ` Bob Liu
2013-12-11 9:25 ` James Dingwall
2013-12-11 16:30 ` James Dingwall
0 siblings, 2 replies; 42+ messages in thread
From: Bob Liu @ 2013-12-11 7:22 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk; +Cc: James Dingwall, xen-devel
[-- Attachment #1: Type: text/plain, Size: 2611 bytes --]
On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>> Konrad Rzeszutek Wilk wrote:
>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>> Hi,
>>>>
>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>> triggers in my Xen guest domains which use ballooning to
>>>> increase/decrease their memory allocation according to their
>>>> requirements. One example domain I have has a maximum memory
>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>> configured with 2Gb swap which is almost 100% free.
>>>>
>>>> # free
>>>> total used free shared buffers cached
>>>> Mem: 272080 248108 23972 0 1448 63064
>>>> -/+ buffers/cache: 183596 88484
>>>> Swap: 2097148 8 2097140
>>>>
>>>> There is plenty of available free memory in the hypervisor to
>>>> balloon to the maximum size:
>>>> # xl info | grep free_mem
>>>> free_memory : 14923
>>>>
>>>> An example trace (they are always the same) from the oom killer in
>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>> at will so it is difficult to start bisecting it to see if a
>>>> particular change introduced this. However it does seem that the
>>>> behaviour is wrong because a) ballooning could give the guest more
>>>> memory, b) there is lots of swap available which could be used as a
>>>> fallback.
>
> Keep in mind that swap with tmem is actually no more swap. Heh, that
> sounds odd -but basically pages that are destined for swap end up
> going in the tmem code which pipes them up to the hypervisor.
>
>>>>
>>>> If other information could help or there are more tests that I could
>>>> run then please let me know.
>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>> the guest right?
>> Yes, domU and dom0 both have the tmem module loaded and tmem
>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>
> Excellent. The odd thing is that your swap is not used that much, but
> it should be (as that is part of what the self-balloon is suppose to do).
>
> Bob, you had a patch for the logic of how self-balloon is suppose
> to account for the slab - would this be relevant to this problem?
>
Perhaps, I have attached the patch.
James, could you please apply it and try your application again? You
have to rebuild the guest kernel.
Oh, and also take a look at whether frontswap is in use, you can check
it by watching "cat /sys/kernel/debug/frontswap/*".
Thanks,
-Bob
[-- Attachment #2: balloon.patch --]
[-- Type: text/x-patch, Size: 1670 bytes --]
diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
index 21e18c1..4814759 100644
--- a/drivers/xen/xen-selfballoon.c
+++ b/drivers/xen/xen-selfballoon.c
@@ -191,6 +191,8 @@ static void selfballoon_process(struct work_struct *work)
tgt_pages = cur_pages; /* default is no change */
goal_pages = vm_memory_committed() +
totalreserve_pages +
+ global_page_state(NR_SLAB_RECLAIMABLE) +
+ global_page_state(NR_SLAB_UNRECLAIMABLE) +
MB2PAGES(selfballoon_reserved_mb);
#ifdef CONFIG_FRONTSWAP
/* allow space for frontswap pages to be repatriated */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 580a5f0..863b05c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -16,6 +16,7 @@
#include <linux/stddef.h>
#include <linux/mm.h>
+#include <linux/mman.h>
#include <linux/swap.h>
#include <linux/interrupt.h>
#include <linux/pagemap.h>
@@ -3075,7 +3076,7 @@ void show_free_areas(unsigned int filter)
" dirty:%lu writeback:%lu unstable:%lu\n"
" free:%lu slab_reclaimable:%lu slab_unreclaimable:%lu\n"
" mapped:%lu shmem:%lu pagetables:%lu bounce:%lu\n"
- " free_cma:%lu\n",
+ " free_cma:%lu totalram:%lu balloontarget:%lu\n",
global_page_state(NR_ACTIVE_ANON),
global_page_state(NR_INACTIVE_ANON),
global_page_state(NR_ISOLATED_ANON),
@@ -3093,7 +3094,9 @@ void show_free_areas(unsigned int filter)
global_page_state(NR_SHMEM),
global_page_state(NR_PAGETABLE),
global_page_state(NR_BOUNCE),
- global_page_state(NR_FREE_CMA_PAGES));
+ global_page_state(NR_FREE_CMA_PAGES),
+ totalram_pages,
+ vm_memory_committed() + totalreserve_pages);
for_each_populated_zone(zone) {
int i;
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-11 7:22 ` Bob Liu
@ 2013-12-11 9:25 ` James Dingwall
2013-12-11 9:54 ` Bob Liu
2013-12-11 16:30 ` James Dingwall
1 sibling, 1 reply; 42+ messages in thread
From: James Dingwall @ 2013-12-11 9:25 UTC (permalink / raw)
To: Bob Liu, Konrad Rzeszutek Wilk; +Cc: xen-devel
Bob Liu wrote:
> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>> Konrad Rzeszutek Wilk wrote:
>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>> Hi,
>>>>>
>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>> triggers in my Xen guest domains which use ballooning to
>>>>> increase/decrease their memory allocation according to their
>>>>> requirements. One example domain I have has a maximum memory
>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>
>>>>> # free
>>>>> total used free shared buffers cached
>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>> -/+ buffers/cache: 183596 88484
>>>>> Swap: 2097148 8 2097140
>>>>>
>>>>> There is plenty of available free memory in the hypervisor to
>>>>> balloon to the maximum size:
>>>>> # xl info | grep free_mem
>>>>> free_memory : 14923
>>>>>
>>>>> An example trace (they are always the same) from the oom killer in
>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>> at will so it is difficult to start bisecting it to see if a
>>>>> particular change introduced this. However it does seem that the
>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>> memory, b) there is lots of swap available which could be used as a
>>>>> fallback.
>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>> sounds odd -but basically pages that are destined for swap end up
>> going in the tmem code which pipes them up to the hypervisor.
>>
>>>>> If other information could help or there are more tests that I could
>>>>> run then please let me know.
>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>> the guest right?
>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>> Excellent. The odd thing is that your swap is not used that much, but
>> it should be (as that is part of what the self-balloon is suppose to do).
>>
>> Bob, you had a patch for the logic of how self-balloon is suppose
>> to account for the slab - would this be relevant to this problem?
> Perhaps, I have attached the patch.
> James, could you please apply it and try your application again? You
> have to rebuild the guest kernel.
> Oh, and also take a look at whether frontswap is in use, you can check
> it by watching "cat /sys/kernel/debug/frontswap/*".
I will test this patch later today and let you know how it goes. In the
meantime I have found that loading tmem with selfshrinking=0 prevents
the problem I originally reported. Frontswap is in use, these values
are from shortly after a guest reboot:
# for i in /sys/kernel/debug/frontswap/* ; do echo $i $(< $i) ; done
/sys/kernel/debug/frontswap/failed_stores 0
/sys/kernel/debug/frontswap/invalidates 5
/sys/kernel/debug/frontswap/loads 83
/sys/kernel/debug/frontswap/succ_stores 107
# for i in /sys/module/tmem/parameters/* ; do echo $i $(< $i) ; done
/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking N
James
>
> balloon.patch
>
>
> diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
> index 21e18c1..4814759 100644
> --- a/drivers/xen/xen-selfballoon.c
> +++ b/drivers/xen/xen-selfballoon.c
> @@ -191,6 +191,8 @@ static void selfballoon_process(struct work_struct *work)
> tgt_pages = cur_pages; /* default is no change */
> goal_pages = vm_memory_committed() +
> totalreserve_pages +
> + global_page_state(NR_SLAB_RECLAIMABLE) +
> + global_page_state(NR_SLAB_UNRECLAIMABLE) +
> MB2PAGES(selfballoon_reserved_mb);
> #ifdef CONFIG_FRONTSWAP
> /* allow space for frontswap pages to be repatriated */
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 580a5f0..863b05c 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -16,6 +16,7 @@
>
> #include <linux/stddef.h>
> #include <linux/mm.h>
> +#include <linux/mman.h>
> #include <linux/swap.h>
> #include <linux/interrupt.h>
> #include <linux/pagemap.h>
> @@ -3075,7 +3076,7 @@ void show_free_areas(unsigned int filter)
> " dirty:%lu writeback:%lu unstable:%lu\n"
> " free:%lu slab_reclaimable:%lu slab_unreclaimable:%lu\n"
> " mapped:%lu shmem:%lu pagetables:%lu bounce:%lu\n"
> - " free_cma:%lu\n",
> + " free_cma:%lu totalram:%lu balloontarget:%lu\n",
> global_page_state(NR_ACTIVE_ANON),
> global_page_state(NR_INACTIVE_ANON),
> global_page_state(NR_ISOLATED_ANON),
> @@ -3093,7 +3094,9 @@ void show_free_areas(unsigned int filter)
> global_page_state(NR_SHMEM),
> global_page_state(NR_PAGETABLE),
> global_page_state(NR_BOUNCE),
> - global_page_state(NR_FREE_CMA_PAGES));
> + global_page_state(NR_FREE_CMA_PAGES),
> + totalram_pages,
> + vm_memory_committed() + totalreserve_pages);
>
> for_each_populated_zone(zone) {
> int i;
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-11 9:25 ` James Dingwall
@ 2013-12-11 9:54 ` Bob Liu
2013-12-11 10:16 ` James Dingwall
0 siblings, 1 reply; 42+ messages in thread
From: Bob Liu @ 2013-12-11 9:54 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 12/11/2013 05:25 PM, James Dingwall wrote:
> Bob Liu wrote:
>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>> Konrad Rzeszutek Wilk wrote:
>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>> increase/decrease their memory allocation according to their
>>>>>> requirements. One example domain I have has a maximum memory
>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>
>>>>>> # free
>>>>>> total used free shared buffers
>>>>>> cached
>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>> -/+ buffers/cache: 183596 88484
>>>>>> Swap: 2097148 8 2097140
>>>>>>
>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>> balloon to the maximum size:
>>>>>> # xl info | grep free_mem
>>>>>> free_memory : 14923
>>>>>>
>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>> particular change introduced this. However it does seem that the
>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>> fallback.
>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>> sounds odd -but basically pages that are destined for swap end up
>>> going in the tmem code which pipes them up to the hypervisor.
>>>
>>>>>> If other information could help or there are more tests that I could
>>>>>> run then please let me know.
>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>> the guest right?
>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>> Excellent. The odd thing is that your swap is not used that much, but
>>> it should be (as that is part of what the self-balloon is suppose to
>>> do).
>>>
>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>> to account for the slab - would this be relevant to this problem?
>> Perhaps, I have attached the patch.
>> James, could you please apply it and try your application again? You
>> have to rebuild the guest kernel.
>> Oh, and also take a look at whether frontswap is in use, you can check
>> it by watching "cat /sys/kernel/debug/frontswap/*".
> I will test this patch later today and let you know how it goes. In the
Thanks!
> meantime I have found that loading tmem with selfshrinking=0 prevents
> the problem I originally reported. Frontswap is in use, these values
It's strange. I think maybe I can set up a testing environment if you
can share me your guest kernel's configuration and also the workload you
are running on the guest.
--
Regards,
-Bob
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-11 9:54 ` Bob Liu
@ 2013-12-11 10:16 ` James Dingwall
0 siblings, 0 replies; 42+ messages in thread
From: James Dingwall @ 2013-12-11 10:16 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 12/11/2013 05:25 PM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>> increase/decrease their memory allocation according to their
>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>
>>>>>>> # free
>>>>>>> total used free shared buffers
>>>>>>> cached
>>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>> Swap: 2097148 8 2097140
>>>>>>>
>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>> balloon to the maximum size:
>>>>>>> # xl info | grep free_mem
>>>>>>> free_memory : 14923
>>>>>>>
>>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>> particular change introduced this. However it does seem that the
>>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>>> fallback.
>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>> sounds odd -but basically pages that are destined for swap end up
>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>
>>>>>>> If other information could help or there are more tests that I could
>>>>>>> run then please let me know.
>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>> the guest right?
>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>> Excellent. The odd thing is that your swap is not used that much, but
>>>> it should be (as that is part of what the self-balloon is suppose to
>>>> do).
>>>>
>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>> to account for the slab - would this be relevant to this problem?
>>> Perhaps, I have attached the patch.
>>> James, could you please apply it and try your application again? You
>>> have to rebuild the guest kernel.
>>> Oh, and also take a look at whether frontswap is in use, you can check
>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>> I will test this patch later today and let you know how it goes. In the
> Thanks!
>
>> meantime I have found that loading tmem with selfshrinking=0 prevents
>> the problem I originally reported. Frontswap is in use, these values
> It's strange. I think maybe I can set up a testing environment if you
> can share me your guest kernel's configuration and also the workload you
> are running on the guest.
I am running Xen 4.3.1 with patches for XSA 7[34568] applied. The dom0
and domU kernels are 3.12.4 and from the same .config which is below. I
am carry patches in the kernel to enable cleancache for the XFS
filesystem which the guest uses and also to get xl to respect the maxmem
option in the config file.
The dom0 grub entry is:
title XEN-pv_ops 4.3.1 (3.12.4) (dom0) (VGA Console)
root (hd0,0)
kernel /boot/xen.gz console=vga,com2 com2=115200,8n1 numa=on
dom0_mem=max:4096M dom0_max_vcpus=2 dom0_vcpus_pin tmem tmem_dedup=on
tmem_compress=on
module /boot/3/12/4/kernel root=/dev/ram0 init=/linuxrc ramdisk=8192
real_root=/dev/systemvg/dom0_rootlv udev doscsi dolvm dodmraid
max_loop=32 console=hvc0 console=tty0 earlyprintk=xen tmem
module /boot/3/12/4/initramfs
The guest is booted via pv-grub with:
title Linux 3.12.4
root (hd0,0)
kernel /boot/3/12/4/kernel root=/dev/ram0 init=/linuxrc ramdisk=8192
real_root=/dev/systemvg/rootlv udev doscsi dolvm dodmraid tmem
initrd /boot/3/12/4/initramfs
from a config of:
name = "domU"
builder = "generic"
kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra = "(hd0,0)/boot/grub/grub.conf"
memory = 512
maxmem = 1024
e820_host = 1
vcpus = 2
cpus = "2-11"
disk = [ 'format=raw, vdev=xvda, access=w, target=/dev/systemvg/domU_xvda',
'format=raw, vdev=xvdb, access=w, target=/dev/systemvg/domU_xvdb',
'format=raw, vdev=xvdc1, access=w,
target=/dev/datavg/domU_datalv' ]
vif = [ 'mac=00:16:3e:00:01:32, bridge=brnet0, vifname=domU.0' ]
usb=1
usbdevice='tablet'
xen_platform_pci = "1"
The system is Gentoo based and the workload that generally triggers the
condition is an emerge (compile) of a large package such as seamonkey,
libreoffice or kdelibs.
Thanks,
James
#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 3.12.4 Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y
#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_FHANDLE=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
# CONFIG_AUDIT_LOGINUID_IMMUTABLE is not set
#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_DOMAIN=y
# CONFIG_IRQ_DOMAIN_DEBUG is not set
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y
#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
# CONFIG_NO_HZ_IDLE is not set
CONFIG_NO_HZ_FULL=y
CONFIG_NO_HZ_FULL_ALL=y
# CONFIG_NO_HZ_FULL_SYSIDLE is not set
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
#
# CPU/Task time and stats accounting
#
CONFIG_VIRT_CPU_ACCOUNTING=y
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
CONFIG_RCU_STALL_COMMON=y
CONFIG_CONTEXT_TRACKING=y
CONFIG_RCU_USER_QS=y
CONFIG_CONTEXT_TRACKING_FORCE=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_RCU_FAST_NO_HZ is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_NOCB_CPU=y
CONFIG_RCU_NOCB_CPU_ALL=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=18
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
CONFIG_ARCH_USES_NUMA_PROT_NONE=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_NUMA_BALANCING=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
# CONFIG_CPUSETS is not set
CONFIG_CGROUP_CPUACCT=y
# CONFIG_RESOURCE_COUNTERS is not set
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_CFS_BANDWIDTH is not set
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_UIDGID_STRICT_TYPE_CHECKS=y
CONFIG_SCHED_AUTOGROUP=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
# CONFIG_EXPERT is not set
CONFIG_UID16=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_PCI_QUIRKS=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y
#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
# CONFIG_COMPAT_BRK is not set
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_PROFILING is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
# CONFIG_KPROBES is not set
# CONFIG_JUMP_LABEL is not set
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y
#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_MODULE_SIG=y
CONFIG_MODULE_SIG_FORCE=y
CONFIG_MODULE_SIG_ALL=y
# CONFIG_MODULE_SIG_SHA1 is not set
# CONFIG_MODULE_SIG_SHA224 is not set
# CONFIG_MODULE_SIG_SHA256 is not set
# CONFIG_MODULE_SIG_SHA384 is not set
CONFIG_MODULE_SIG_SHA512=y
CONFIG_MODULE_SIG_HASH="sha512"
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
# CONFIG_BLK_DEV_INTEGRITY is not set
CONFIG_BLK_DEV_THROTTLING=y
# CONFIG_BLK_CMDLINE_PARSER is not set
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_AIX_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
# CONFIG_CMDLINE_PARTITION is not set
CONFIG_BLOCK_COMPAT=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
# CONFIG_IOSCHED_CFQ is not set
CONFIG_DEFAULT_DEADLINE=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_PADATA=y
CONFIG_ASN1=y
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y
#
# Processor type and features
#
CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_X2APIC=y
CONFIG_X86_MPPARSE=y
# CONFIG_X86_EXTENDED_PLATFORM is not set
# CONFIG_X86_INTEL_LPSS is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_KVM_GUEST is not set
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not set
CONFIG_PARAVIRT_CLOCK=y
CONFIG_NO_BOOTMEM=y
# CONFIG_MEMTEST is not set
CONFIG_MK8=y
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_NR_CPUS=16
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
CONFIG_X86_THERMAL_VECTOR=y
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_MICROCODE_INTEL_LIB=y
# CONFIG_MICROCODE_INTEL_EARLY is not set
# CONFIG_MICROCODE_AMD_EARLY is not set
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=6
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_MEMORY_ISOLATION=y
# CONFIG_MOVABLE_NODE is not set
CONFIG_HAVE_BOOTMEM_INFO_NODE=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_BALLOON_COMPACTION=y
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
# CONFIG_CMA is not set
CONFIG_ZBUD=y
CONFIG_ZSWAP=y
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_X86_SMAP=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
CONFIG_CC_STACKPROTECTOR=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x1000000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
# CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set
# CONFIG_DEBUG_HOTPLUG_CPU0 is not set
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
#
# Power management and ACPI options
#
# CONFIG_SUSPEND is not set
CONFIG_HIBERNATE_CALLBACKS=y
# CONFIG_HIBERNATION is not set
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
CONFIG_ACPI=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
# CONFIG_ACPI_EC_DEBUGFS is not set
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=m
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_IPMI=m
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=m
CONFIG_ACPI_NUMA=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
# CONFIG_ACPI_INITRD_TABLE_OVERRIDE is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_HOTPLUG_MEMORY is not set
CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=y
# CONFIG_ACPI_CUSTOM_METHOD is not set
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=y
CONFIG_ACPI_APEI_MEMORY_FAILURE=y
# CONFIG_ACPI_APEI_EINJ is not set
# CONFIG_ACPI_APEI_ERST_DEBUG is not set
# CONFIG_SFI is not set
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=m
CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=m
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m
#
# x86 CPU frequency scaling drivers
#
CONFIG_X86_INTEL_PSTATE=y
# CONFIG_X86_PCC_CPUFREQ is not set
CONFIG_X86_ACPI_CPUFREQ=m
# CONFIG_X86_ACPI_CPUFREQ_CPB is not set
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_AMD_FREQ_SENSITIVITY=m
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set
#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
#
# CPU Idle
#
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_MULTIPLE_DRIVERS is not set
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
CONFIG_INTEL_IDLE=y
#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_MMCONFIG is not set
CONFIG_PCI_XEN=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_PCI_MSI=y
# CONFIG_PCI_REALLOC_ENABLE_AUTO is not set
CONFIG_PCI_STUB=m
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
#
# PCI host controller drivers
#
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=m
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y
#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_PCCARD_NONSTATIC=y
# CONFIG_HOTPLUG_PCI is not set
# CONFIG_RAPIDIO is not set
CONFIG_X86_SYSFB=y
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_BINFMT_SCRIPT=y
# CONFIG_HAVE_AOUT is not set
# CONFIG_BINFMT_MISC is not set
CONFIG_COREDUMP=y
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=y
# CONFIG_X86_X32 is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_DIAG=m
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=m
CONFIG_XFRM_USER=m
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=m
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_ROUTE_CLASSID=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set
CONFIG_NET_IP_TUNNEL=m
# CONFIG_IP_MROUTE is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
CONFIG_INET_TUNNEL=m
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_INET_UDP_DIAG=m
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
# CONFIG_IPV6_ROUTER_PREF is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
CONFIG_IPV6_SIT=m
# CONFIG_IPV6_SIT_6RD is not set
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_GRE=m
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
# CONFIG_BRIDGE_NETFILTER is not set
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_ZONES=y
# CONFIG_NF_CONNTRACK_PROCFS is not set
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_TIMEOUT=y
CONFIG_NF_CONNTRACK_TIMESTAMP=y
CONFIG_NF_CONNTRACK_LABELS=y
# CONFIG_NF_CT_PROTO_DCCP is not set
# CONFIG_NF_CT_PROTO_SCTP is not set
# CONFIG_NF_CT_PROTO_UDPLITE is not set
# CONFIG_NF_CONNTRACK_AMANDA is not set
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
# CONFIG_NF_CONNTRACK_NETBIOS_NS is not set
# CONFIG_NF_CONNTRACK_SNMP is not set
# CONFIG_NF_CONNTRACK_PPTP is not set
# CONFIG_NF_CONNTRACK_SANE is not set
CONFIG_NF_CONNTRACK_SIP=m
# CONFIG_NF_CONNTRACK_TFTP is not set
CONFIG_NF_CT_NETLINK=m
CONFIG_NF_CT_NETLINK_TIMEOUT=m
CONFIG_NF_CT_NETLINK_HELPER=m
CONFIG_NETFILTER_NETLINK_QUEUE_CT=y
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
# CONFIG_NF_NAT_AMANDA is not set
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
# CONFIG_NF_NAT_TFTP is not set
CONFIG_NETFILTER_SYNPROXY=m
CONFIG_NETFILTER_XTABLES=m
#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_SET=m
#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_AUDIT=m
# CONFIG_NETFILTER_XT_TARGET_CHECKSUM is not set
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_HMARK=m
# CONFIG_NETFILTER_XT_TARGET_IDLETIMER is not set
CONFIG_NETFILTER_XT_TARGET_LOG=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NETMAP=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
# CONFIG_NETFILTER_XT_TARGET_NOTRACK is not set
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_REDIRECT=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
# CONFIG_NETFILTER_XT_MATCH_DCCP is not set
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
# CONFIG_NETFILTER_XT_MATCH_DSCP is not set
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
# CONFIG_NETFILTER_XT_MATCH_LENGTH is not set
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_NFACCT=m
# CONFIG_NETFILTER_XT_MATCH_OSF is not set
# CONFIG_NETFILTER_XT_MATCH_OWNER is not set
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
# CONFIG_NETFILTER_XT_MATCH_SCTP is not set
# CONFIG_NETFILTER_XT_MATCH_SOCKET is not set
CONFIG_NETFILTER_XT_MATCH_STATE=m
# CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set
# CONFIG_NETFILTER_XT_MATCH_STRING is not set
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
# CONFIG_NETFILTER_XT_MATCH_U32 is not set
CONFIG_IP_SET=m
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=m
CONFIG_IP_SET_BITMAP_IPMAC=m
CONFIG_IP_SET_BITMAP_PORT=m
CONFIG_IP_SET_HASH_IP=m
CONFIG_IP_SET_HASH_IPPORT=m
CONFIG_IP_SET_HASH_IPPORTIP=m
CONFIG_IP_SET_HASH_IPPORTNET=m
CONFIG_IP_SET_HASH_NET=m
CONFIG_IP_SET_HASH_NETPORT=m
CONFIG_IP_SET_HASH_NETIFACE=m
CONFIG_IP_SET_LIST_SET=m
CONFIG_IP_VS=m
CONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12
#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
# CONFIG_IP_VS_PROTO_SCTP is not set
#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8
#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m
#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_RPFILTER=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_SYNPROXY=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT_IPV4=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
# CONFIG_NF_NAT_PPTP is not set
CONFIG_NF_NAT_H323=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
#
# IPv6: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RPFILTER=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_TARGET_SYNPROXY=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
# CONFIG_NF_NAT_IPV6 is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_MRP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_HAVE_NET_DSA=y
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
CONFIG_VLAN_8021Q_MVRP=y
# CONFIG_DECNET is not set
CONFIG_LLC=m
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
CONFIG_NET_SCH_MQPRIO=m
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_QFQ=m
CONFIG_NET_SCH_CODEL=m
CONFIG_NET_SCH_FQ_CODEL=m
CONFIG_NET_SCH_FQ=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_SCH_PLUG=m
#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=m
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_EMATCH_IPSET=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
CONFIG_OPENVSWITCH=m
CONFIG_OPENVSWITCH_VXLAN=y
CONFIG_VSOCKETS=m
CONFIG_NETLINK_MMAP=y
CONFIG_NETLINK_DIAG=m
# CONFIG_NET_MPLS_GSO is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_NETPRIO_CGROUP=m
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
CONFIG_BPF_JIT=y
CONFIG_NET_FLOW_LIMIT=y
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
CONFIG_BT=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=m
#
# Bluetooth device drivers
#
CONFIG_BT_HCIBTUSB=m
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
# CONFIG_BT_HCIUART_ATH3K is not set
CONFIG_BT_HCIUART_LL=y
# CONFIG_BT_HCIUART_3WIRE is not set
CONFIG_BT_HCIBCM203X=m
# CONFIG_BT_HCIBPA10X is not set
# CONFIG_BT_HCIBFUSB is not set
# CONFIG_BT_HCIDTL1 is not set
# CONFIG_BT_HCIBT3C is not set
# CONFIG_BT_HCIBLUECARD is not set
# CONFIG_BT_HCIBTUART is not set
# CONFIG_BT_HCIVHCI is not set
# CONFIG_BT_MRVL is not set
# CONFIG_BT_ATH3K is not set
CONFIG_AF_RXRPC=m
# CONFIG_AF_RXRPC_DEBUG is not set
CONFIG_RXKAD=m
CONFIG_WIRELESS=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_REG_DEBUG is not set
CONFIG_CFG80211_DEFAULT_PS=y
# CONFIG_CFG80211_DEBUGFS is not set
# CONFIG_CFG80211_INTERNAL_REGDB is not set
# CONFIG_CFG80211_WEXT is not set
# CONFIG_LIB80211 is not set
# CONFIG_MAC80211 is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
CONFIG_CEPH_LIB=m
# CONFIG_CEPH_LIB_PRETTYDEBUG is not set
CONFIG_CEPH_LIB_USE_DNS_RESOLVER=y
# CONFIG_NFC is not set
CONFIG_HAVE_BPF_JIT=y
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_FIRMWARE_IN_KERNEL is not set
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_FW_LOADER_USER_HELPER is not set
CONFIG_SYS_HYPERVISOR=y
# CONFIG_GENERIC_CPU_DEVICES is not set
CONFIG_DMA_SHARED_BUFFER=y
#
# Bus devices
#
CONFIG_CONNECTOR=m
# CONFIG_MTD is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
# CONFIG_PARPORT_SERIAL is not set
CONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_PC_PCMCIA is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y
#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
# CONFIG_PARIDE is not set
# CONFIG_BLK_DEV_PCIESSD_MTIP32XX is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=32
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
CONFIG_BLK_DEV_DRBD=m
# CONFIG_DRBD_FAULT_INJECTION is not set
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_NVME is not set
CONFIG_BLK_DEV_OSD=m
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_VIRTIO_BLK=m
# CONFIG_BLK_DEV_HD is not set
CONFIG_BLK_DEV_RBD=m
# CONFIG_BLK_DEV_RSXX is not set
#
# Misc devices
#
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_AD525X_DPOT is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ATMEL_SSC is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_HP_ILO is not set
# CONFIG_APDS9802ALS is not set
# CONFIG_ISL29003 is not set
# CONFIG_ISL29020 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_SENSORS_BH1780 is not set
# CONFIG_SENSORS_BH1770 is not set
# CONFIG_SENSORS_APDS990X is not set
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
CONFIG_VMWARE_BALLOON=m
# CONFIG_BMP085_I2C is not set
# CONFIG_PCH_PHUB is not set
# CONFIG_USB_SWITCH_FSA9480 is not set
# CONFIG_SRAM is not set
# CONFIG_C2PORT is not set
#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
CONFIG_EEPROM_LEGACY=m
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_CB710_CORE is not set
#
# Texas Instruments shared transport line discipline
#
# CONFIG_SENSORS_LIS3_I2C is not set
#
# Altera FPGA firmware download module
#
# CONFIG_ALTERA_STAPL is not set
# CONFIG_VMWARE_VMCI is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set
#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=m
# CONFIG_CHR_DEV_SCH is not set
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
# CONFIG_SCSI_FC_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=m
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
# CONFIG_SCSI_CXGB3_ISCSI is not set
# CONFIG_SCSI_CXGB4_ISCSI is not set
CONFIG_SCSI_BNX2_ISCSI=m
# CONFIG_SCSI_BNX2X_FCOE is not set
# CONFIG_BE2ISCSI is not set
CONFIG_BLK_DEV_3W_XXXX_RAID=m
# CONFIG_SCSI_HPSA is not set
CONFIG_SCSI_3W_9XXX=y
# CONFIG_SCSI_3W_SAS is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=5000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_MVUMI is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_SCSI_ESAS2R is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_MEGARAID_SAS=m
# CONFIG_SCSI_MPT2SAS is not set
# CONFIG_SCSI_MPT3SAS is not set
# CONFIG_SCSI_UFSHCD is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_VMWARE_PVSCSI is not set
# CONFIG_LIBFC is not set
# CONFIG_LIBFCOE is not set
# CONFIG_FCOE is not set
# CONFIG_FCOE_FNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_ISCI is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
CONFIG_SCSI_QLOGIC_1280=m
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_PMCRAID is not set
# CONFIG_SCSI_PM8001 is not set
# CONFIG_SCSI_SRP is not set
# CONFIG_SCSI_BFA_FC is not set
CONFIG_SCSI_VIRTIO=m
# CONFIG_SCSI_CHELSIO_FCOE is not set
# CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
# CONFIG_SCSI_DH is not set
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
# CONFIG_SCSI_OSD_DEBUG is not set
CONFIG_ATA=m
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_ZPODD=y
CONFIG_SATA_PMP=y
#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=m
# CONFIG_SATA_AHCI_PLATFORM is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_SATA_ACARD_AHCI is not set
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
#
# SFF controllers with custom DMA interface
#
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_SX4 is not set
CONFIG_ATA_BMDMA=y
#
# SATA SFF controllers with BMDMA
#
# CONFIG_ATA_PIIX is not set
# CONFIG_SATA_HIGHBANK is not set
# CONFIG_SATA_MV is not set
CONFIG_SATA_NV=m
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_RCAR is not set
CONFIG_SATA_SIL=m
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_SVW is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
#
# PATA SFF controllers with BMDMA
#
# CONFIG_PATA_ALI is not set
CONFIG_PATA_AMD=m
# CONFIG_PATA_ARASAN_CF is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_ATP867X is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5536 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RDC is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SCH is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_TOSHIBA is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_PCMCIA is not set
# CONFIG_PATA_RZ1000 is not set
#
# Generic fallback / legacy drivers
#
# CONFIG_PATA_ACPI is not set
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=y
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
CONFIG_BCACHE=m
# CONFIG_BCACHE_DEBUG is not set
# CONFIG_BCACHE_EDEBUG is not set
# CONFIG_BCACHE_CLOSURES_DEBUG is not set
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_DEBUG is not set
CONFIG_DM_BUFIO=m
CONFIG_DM_BIO_PRISON=m
CONFIG_DM_PERSISTENT_DATA=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
# CONFIG_DM_DEBUG_BLOCK_STACK_TRACING is not set
CONFIG_DM_CACHE=m
CONFIG_DM_CACHE_MQ=m
CONFIG_DM_CACHE_CLEANER=m
CONFIG_DM_MIRROR=m
CONFIG_DM_RAID=m
# CONFIG_DM_LOG_USERSPACE is not set
CONFIG_DM_ZERO=m
# CONFIG_DM_MULTIPATH is not set
# CONFIG_DM_DELAY is not set
CONFIG_DM_UEVENT=y
# CONFIG_DM_FLAKEY is not set
CONFIG_DM_VERITY=m
# CONFIG_DM_SWITCH is not set
# CONFIG_TARGET_CORE is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_SBP2=m
CONFIG_FIREWIRE_NET=m
# CONFIG_FIREWIRE_NOSY is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_MII=m
CONFIG_NET_CORE=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
# CONFIG_EQUALIZER is not set
# CONFIG_NET_FC is not set
CONFIG_IFB=m
CONFIG_NET_TEAM=m
CONFIG_NET_TEAM_MODE_BROADCAST=m
CONFIG_NET_TEAM_MODE_ROUNDROBIN=m
CONFIG_NET_TEAM_MODE_RANDOM=m
CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=m
CONFIG_NET_TEAM_MODE_LOADBALANCE=m
# CONFIG_MACVLAN is not set
CONFIG_VXLAN=m
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
CONFIG_TUN=m
# CONFIG_VETH is not set
CONFIG_VIRTIO_NET=m
# CONFIG_NLMON is not set
# CONFIG_ARCNET is not set
#
# CAIF transport drivers
#
CONFIG_VHOST_NET=m
CONFIG_VHOST_RING=m
CONFIG_VHOST=m
#
# Distributed Switch Architecture drivers
#
# CONFIG_NET_DSA_MV88E6XXX is not set
# CONFIG_NET_DSA_MV88E6060 is not set
# CONFIG_NET_DSA_MV88E6XXX_NEED_PPU is not set
# CONFIG_NET_DSA_MV88E6131 is not set
# CONFIG_NET_DSA_MV88E6123_61_65 is not set
CONFIG_ETHERNET=y
CONFIG_NET_VENDOR_3COM=y
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_3C589=m
CONFIG_VORTEX=m
# CONFIG_TYPHOON is not set
CONFIG_NET_VENDOR_ADAPTEC=y
# CONFIG_ADAPTEC_STARFIRE is not set
CONFIG_NET_VENDOR_ALTEON=y
# CONFIG_ACENIC is not set
CONFIG_NET_VENDOR_AMD=y
# CONFIG_AMD8111_ETH is not set
# CONFIG_PCNET32 is not set
# CONFIG_PCMCIA_NMCLAN is not set
# CONFIG_NET_VENDOR_ARC is not set
CONFIG_NET_VENDOR_ATHEROS=y
# CONFIG_ATL2 is not set
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
# CONFIG_ALX is not set
# CONFIG_NET_CADENCE is not set
CONFIG_NET_VENDOR_BROADCOM=y
# CONFIG_B44 is not set
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_TIGON3=m
# CONFIG_BNX2X is not set
CONFIG_NET_VENDOR_BROCADE=y
# CONFIG_BNA is not set
# CONFIG_NET_CALXEDA_XGMAC is not set
CONFIG_NET_VENDOR_CHELSIO=y
# CONFIG_CHELSIO_T1 is not set
# CONFIG_CHELSIO_T3 is not set
# CONFIG_CHELSIO_T4 is not set
# CONFIG_CHELSIO_T4VF is not set
CONFIG_NET_VENDOR_CISCO=y
# CONFIG_ENIC is not set
# CONFIG_DNET is not set
CONFIG_NET_VENDOR_DEC=y
# CONFIG_NET_TULIP is not set
CONFIG_NET_VENDOR_DLINK=y
# CONFIG_DL2K is not set
# CONFIG_SUNDANCE is not set
CONFIG_NET_VENDOR_EMULEX=y
# CONFIG_BE2NET is not set
CONFIG_NET_VENDOR_EXAR=y
# CONFIG_S2IO is not set
# CONFIG_VXGE is not set
CONFIG_NET_VENDOR_FUJITSU=y
# CONFIG_PCMCIA_FMVJ18X is not set
CONFIG_NET_VENDOR_HP=y
# CONFIG_HP100 is not set
CONFIG_NET_VENDOR_INTEL=y
CONFIG_E100=m
CONFIG_E1000=m
CONFIG_E1000E=m
CONFIG_IGB=m
CONFIG_IGB_HWMON=y
CONFIG_IGB_DCA=y
CONFIG_IGBVF=m
# CONFIG_IXGB is not set
# CONFIG_IXGBE is not set
CONFIG_IXGBEVF=m
# CONFIG_I40E is not set
CONFIG_NET_VENDOR_I825XX=y
# CONFIG_IP1000 is not set
# CONFIG_JME is not set
CONFIG_NET_VENDOR_MARVELL=y
# CONFIG_MVMDIO is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
CONFIG_NET_VENDOR_MELLANOX=y
# CONFIG_MLX4_EN is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_MLX5_CORE is not set
CONFIG_NET_VENDOR_MICREL=y
# CONFIG_KS8842 is not set
# CONFIG_KS8851_MLL is not set
# CONFIG_KSZ884X_PCI is not set
CONFIG_NET_VENDOR_MYRI=y
# CONFIG_MYRI10GE is not set
# CONFIG_FEALNX is not set
CONFIG_NET_VENDOR_NATSEMI=y
# CONFIG_NATSEMI is not set
# CONFIG_NS83820 is not set
CONFIG_NET_VENDOR_8390=y
# CONFIG_PCMCIA_AXNET is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_PCMCIA_PCNET is not set
CONFIG_NET_VENDOR_NVIDIA=y
CONFIG_FORCEDETH=m
CONFIG_NET_VENDOR_OKI=y
# CONFIG_PCH_GBE is not set
# CONFIG_ETHOC is not set
# CONFIG_NET_PACKET_ENGINE is not set
CONFIG_NET_VENDOR_QLOGIC=y
# CONFIG_QLA3XXX is not set
# CONFIG_QLCNIC is not set
# CONFIG_QLGE is not set
# CONFIG_NETXEN_NIC is not set
CONFIG_NET_VENDOR_REALTEK=y
# CONFIG_ATP is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
CONFIG_R8169=m
# CONFIG_SH_ETH is not set
CONFIG_NET_VENDOR_RDC=y
# CONFIG_R6040 is not set
CONFIG_NET_VENDOR_SEEQ=y
CONFIG_NET_VENDOR_SILAN=y
# CONFIG_SC92031 is not set
CONFIG_NET_VENDOR_SIS=y
# CONFIG_SIS900 is not set
# CONFIG_SIS190 is not set
# CONFIG_SFC is not set
CONFIG_NET_VENDOR_SMSC=y
# CONFIG_PCMCIA_SMC91C92 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SMSC911X is not set
# CONFIG_SMSC9420 is not set
CONFIG_NET_VENDOR_STMICRO=y
# CONFIG_STMMAC_ETH is not set
CONFIG_NET_VENDOR_SUN=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NIU is not set
CONFIG_NET_VENDOR_TEHUTI=y
# CONFIG_TEHUTI is not set
CONFIG_NET_VENDOR_TI=y
# CONFIG_TLAN is not set
CONFIG_NET_VENDOR_VIA=y
# CONFIG_VIA_RHINE is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_NET_VENDOR_WIZNET is not set
CONFIG_NET_VENDOR_XIRCOM=y
# CONFIG_PCMCIA_XIRC2PS is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_NET_SB1000 is not set
CONFIG_PHYLIB=m
#
# MII PHY device drivers
#
# CONFIG_AT803X_PHY is not set
CONFIG_AMD_PHY=m
CONFIG_MARVELL_PHY=m
# CONFIG_DAVICOM_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_BROADCOM_PHY is not set
CONFIG_BCM87XX_PHY=m
# CONFIG_ICPLUS_PHY is not set
# CONFIG_REALTEK_PHY is not set
# CONFIG_NATIONAL_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_LSI_ET1011C_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_MDIO_BITBANG is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_RTL8152 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_IPHETH is not set
CONFIG_WLAN=y
# CONFIG_PCMCIA_RAYCS is not set
# CONFIG_AIRO is not set
# CONFIG_ATMEL is not set
# CONFIG_AIRO_CS is not set
# CONFIG_PCMCIA_WL3501 is not set
# CONFIG_PRISM54 is not set
# CONFIG_USB_ZD1201 is not set
# CONFIG_USB_NET_RNDIS_WLAN is not set
# CONFIG_ATH_CARDS is not set
# CONFIG_BRCMFMAC is not set
# CONFIG_HOSTAP is not set
# CONFIG_IPW2100 is not set
# CONFIG_LIBERTAS is not set
# CONFIG_WL_TI is not set
# CONFIG_MWIFIEX is not set
#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_WAN is not set
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_VMXNET3=m
# CONFIG_ISDN is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
CONFIG_INPUT_SPARSEKMAP=m
# CONFIG_INPUT_MATRIXKMAP is not set
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_LM8333 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_CYAPA is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
# CONFIG_MOUSE_SYNAPTICS_USB is not set
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=y
# CONFIG_JOYSTICK_A3D is not set
# CONFIG_JOYSTICK_ADI is not set
# CONFIG_JOYSTICK_COBRA is not set
# CONFIG_JOYSTICK_GF2K is not set
# CONFIG_JOYSTICK_GRIP is not set
# CONFIG_JOYSTICK_GRIP_MP is not set
# CONFIG_JOYSTICK_GUILLEMOT is not set
# CONFIG_JOYSTICK_INTERACT is not set
# CONFIG_JOYSTICK_SIDEWINDER is not set
# CONFIG_JOYSTICK_TMDC is not set
# CONFIG_JOYSTICK_IFORCE is not set
# CONFIG_JOYSTICK_WARRIOR is not set
# CONFIG_JOYSTICK_MAGELLAN is not set
# CONFIG_JOYSTICK_SPACEORB is not set
# CONFIG_JOYSTICK_SPACEBALL is not set
# CONFIG_JOYSTICK_STINGER is not set
# CONFIG_JOYSTICK_TWIDJOY is not set
# CONFIG_JOYSTICK_ZHENHUA is not set
# CONFIG_JOYSTICK_DB9 is not set
# CONFIG_JOYSTICK_GAMECON is not set
# CONFIG_JOYSTICK_TURBOGRAFX is not set
# CONFIG_JOYSTICK_AS5011 is not set
# CONFIG_JOYSTICK_JOYDUMP is not set
# CONFIG_JOYSTICK_XPAD is not set
# CONFIG_JOYSTICK_WALKERA0701 is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_BMA150 is not set
CONFIG_INPUT_PCSPKR=y
# CONFIG_INPUT_MMA8450 is not set
# CONFIG_INPUT_MPU3050 is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_KXTJ9 is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_PWM_BEEPER is not set
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_CMA3000 is not set
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=m
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_SERIO_ALTERA_PS2 is not set
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_SERIO_ARC_PS2 is not set
CONFIG_GAMEPORT=y
# CONFIG_GAMEPORT_NS558 is not set
# CONFIG_GAMEPORT_L4 is not set
CONFIG_GAMEPORT_EMU10K1=m
# CONFIG_GAMEPORT_FM801 is not set
#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
# CONFIG_N_GSM is not set
# CONFIG_TRACE_SINK is not set
# CONFIG_DEVKMEM is not set
#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_DMA=y
CONFIG_SERIAL_8250_PCI=y
# CONFIG_SERIAL_8250_CS is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set
# CONFIG_SERIAL_8250_DW is not set
#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MFD_HSU is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_PCH_UART is not set
# CONFIG_SERIAL_ARC is not set
# CONFIG_SERIAL_RP2 is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_ST_ASC is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_VIRTIO_CONSOLE=m
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_PANIC_EVENT=y
CONFIG_IPMI_PANIC_STRING=y
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_NVRAM=m
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
#
# PCMCIA character devices
#
# CONFIG_SYNCLINK_CS is not set
# CONFIG_CARDMAN_4000 is not set
# CONFIG_CARDMAN_4040 is not set
# CONFIG_IPWIRELESS is not set
# CONFIG_MWAVE is not set
CONFIG_RAW_DRIVER=m
CONFIG_MAX_RAW_DEVS=256
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
# CONFIG_HANGCHECK_TIMER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_MUX=m
#
# Multiplexer I2C Chip support
#
# CONFIG_I2C_MUX_PCA9541 is not set
# CONFIG_I2C_MUX_PCA954x is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=m
#
# I2C Hardware Bus support
#
#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
# CONFIG_I2C_AMD756_S4882 is not set
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
# CONFIG_I2C_ISCH is not set
CONFIG_I2C_ISMT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
# CONFIG_I2C_NFORCE2_S4985 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
#
# ACPI drivers
#
CONFIG_I2C_SCMI=m
#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_DESIGNWARE_PCI is not set
# CONFIG_I2C_EG20T is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_PXA_PCI is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set
#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
# CONFIG_I2C_PARPORT is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set
#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_SPI is not set
# CONFIG_HSI is not set
#
# PPS support
#
CONFIG_PPS=m
# CONFIG_PPS_DEBUG is not set
# CONFIG_NTP_PPS is not set
#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
# CONFIG_PPS_CLIENT_LDISC is not set
# CONFIG_PPS_CLIENT_PARPORT is not set
# CONFIG_PPS_CLIENT_GPIO is not set
#
# PPS generators support
#
#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=m
#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# CONFIG_PTP_1588_CLOCK_PCH is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_GPIO_DEVRES=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_GENERIC_ADC_BATTERY is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2780 is not set
# CONFIG_BATTERY_DS2781 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_SBS is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_MAX8903 is not set
# CONFIG_CHARGER_LP8727 is not set
# CONFIG_CHARGER_BQ2415X is not set
# CONFIG_CHARGER_SMB347 is not set
# CONFIG_POWER_RESET is not set
CONFIG_POWER_AVS=y
CONFIG_HWMON=m
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set
#
# Native drivers
#
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
CONFIG_SENSORS_ADM1021=m
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
CONFIG_SENSORS_ADM9240=m
# CONFIG_SENSORS_ADT7410 is not set
# CONFIG_SENSORS_ADT7411 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_ASC7621 is not set
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_FAM15H_POWER=m
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS620 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
# CONFIG_SENSORS_G762 is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_HIH6130 is not set
# CONFIG_SENSORS_HTU21 is not set
CONFIG_SENSORS_CORETEMP=m
# CONFIG_SENSORS_IBMAEM is not set
# CONFIG_SENSORS_IBMPEX is not set
# CONFIG_SENSORS_IIO_HWMON is not set
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_LINEAGE is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM73 is not set
CONFIG_SENSORS_LM75=m
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
CONFIG_SENSORS_LM85=m
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4151 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LTC4261 is not set
# CONFIG_SENSORS_LM95234 is not set
# CONFIG_SENSORS_LM95241 is not set
# CONFIG_SENSORS_LM95245 is not set
# CONFIG_SENSORS_MAX16065 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX1668 is not set
# CONFIG_SENSORS_MAX197 is not set
# CONFIG_SENSORS_MAX6639 is not set
# CONFIG_SENSORS_MAX6642 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_MAX6697 is not set
# CONFIG_SENSORS_MCP3021 is not set
# CONFIG_SENSORS_NCT6775 is not set
# CONFIG_SENSORS_NTC_THERMISTOR is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_PMBUS is not set
# CONFIG_SENSORS_SHT21 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMM665 is not set
CONFIG_SENSORS_DME1737=m
# CONFIG_SENSORS_EMC1403 is not set
# CONFIG_SENSORS_EMC2103 is not set
# CONFIG_SENSORS_EMC6W201 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_SCH56XX_COMMON is not set
# CONFIG_SENSORS_ADS1015 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_AMC6821 is not set
# CONFIG_SENSORS_INA209 is not set
# CONFIG_SENSORS_INA2XX is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP102 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_TMP421 is not set
# CONFIG_SENSORS_VIA_CPUTEMP is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
CONFIG_SENSORS_W83781D=m
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83795 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
CONFIG_SENSORS_W83627HF=m
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_APPLESMC is not set
#
# ACPI drivers
#
# CONFIG_SENSORS_ACPI_POWER is not set
# CONFIG_SENSORS_ATK0110 is not set
CONFIG_THERMAL=m
CONFIG_THERMAL_HWMON=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set
# CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set
CONFIG_THERMAL_GOV_FAIR_SHARE=y
CONFIG_THERMAL_GOV_STEP_WISE=y
CONFIG_THERMAL_GOV_USER_SPACE=y
CONFIG_CPU_THERMAL=y
CONFIG_THERMAL_EMULATION=y
# CONFIG_INTEL_POWERCLAMP is not set
CONFIG_X86_PKG_TEMP_THERMAL=m
#
# Texas Instruments thermal drivers
#
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y
#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y
#
# Broadcom specific AMBA
#
# CONFIG_BCMA is not set
#
# Multifunction device drivers
#
CONFIG_MFD_CORE=m
# CONFIG_MFD_CS5535 is not set
# CONFIG_MFD_CROS_EC is not set
# CONFIG_MFD_MC13XXX_I2C is not set
# CONFIG_HTC_PASIC3 is not set
CONFIG_LPC_ICH=m
# CONFIG_LPC_SCH is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_VIPERBOARD is not set
# CONFIG_MFD_RETU is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_RTSX_PCI is not set
# CONFIG_MFD_SI476X_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_TPS6105X is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TPS65217 is not set
# CONFIG_MFD_WL1273_CORE is not set
# CONFIG_MFD_LM3533 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_MFD_VX855 is not set
# CONFIG_MFD_ARIZONA_I2C is not set
# CONFIG_REGULATOR is not set
CONFIG_MEDIA_SUPPORT=m
#
# Multimedia core support
#
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_ANALOG_TV_SUPPORT=y
# CONFIG_MEDIA_DIGITAL_TV_SUPPORT is not set
# CONFIG_MEDIA_RADIO_SUPPORT is not set
CONFIG_MEDIA_RC_SUPPORT=y
# CONFIG_MEDIA_CONTROLLER is not set
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2=m
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_V4L2_MEM2MEM_DEV=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_DMA_CONTIG=m
# CONFIG_VIDEO_V4L2_INT_DEVICE is not set
# CONFIG_TTPCI_EEPROM is not set
#
# Media drivers
#
CONFIG_RC_CORE=m
CONFIG_RC_MAP=m
CONFIG_RC_DECODERS=y
CONFIG_LIRC=m
CONFIG_IR_LIRC_CODEC=m
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
CONFIG_IR_JVC_DECODER=m
CONFIG_IR_SONY_DECODER=m
CONFIG_IR_RC5_SZ_DECODER=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_IR_MCE_KBD_DECODER=m
CONFIG_RC_DEVICES=y
CONFIG_RC_ATI_REMOTE=m
# CONFIG_IR_ENE is not set
# CONFIG_IR_IMON is not set
# CONFIG_IR_MCEUSB is not set
# CONFIG_IR_ITE_CIR is not set
# CONFIG_IR_FINTEK is not set
# CONFIG_IR_NUVOTON is not set
# CONFIG_IR_REDRAT3 is not set
# CONFIG_IR_STREAMZAP is not set
# CONFIG_IR_WINBOND_CIR is not set
# CONFIG_IR_IGUANA is not set
# CONFIG_IR_TTUSBIR is not set
# CONFIG_RC_LOOPBACK is not set
CONFIG_IR_GPIO_CIR=m
# CONFIG_MEDIA_USB_SUPPORT is not set
# CONFIG_MEDIA_PCI_SUPPORT is not set
CONFIG_V4L_PLATFORM_DRIVERS=y
# CONFIG_VIDEO_CAFE_CCIC is not set
# CONFIG_VIDEO_TIMBERDALE is not set
# CONFIG_SOC_CAMERA is not set
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_MEM2MEM_DEINTERLACE=m
# CONFIG_VIDEO_SH_VEU is not set
# CONFIG_V4L_TEST_DRIVERS is not set
#
# Supported MMC/SDIO adapters
#
# CONFIG_MEDIA_PARPORT_SUPPORT is not set
# CONFIG_CYPRESS_FIRMWARE is not set
#
# Media ancillary drivers (tuners, sensors, i2c, frontends)
#
CONFIG_MEDIA_SUBDRV_AUTOSELECT=y
CONFIG_MEDIA_ATTACH=y
CONFIG_VIDEO_IR_I2C=m
#
# Audio decoders, processors and mixers
#
#
# RDS decoders
#
#
# Video decoders
#
#
# Video and audio decoders
#
#
# Video encoders
#
#
# Camera sensor devices
#
#
# Flash devices
#
#
# Video improvement chips
#
#
# Miscelaneous helper chips
#
#
# Sensors used on soc_camera driver
#
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA827X=m
CONFIG_MEDIA_TUNER_TDA18271=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_MEDIA_TUNER_XC4000=m
CONFIG_MEDIA_TUNER_MC44S803=m
#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set
#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=m
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
# CONFIG_VGA_SWITCHEROO is not set
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
# CONFIG_DRM_LOAD_EDID_FIRMWARE is not set
CONFIG_DRM_TTM=m
#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=m
CONFIG_DRM_I2C_SIL164=m
# CONFIG_DRM_I2C_NXP_TDA998X is not set
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
CONFIG_DRM_RADEON=m
# CONFIG_DRM_RADEON_UMS is not set
CONFIG_DRM_NOUVEAU=m
CONFIG_NOUVEAU_DEBUG=5
CONFIG_NOUVEAU_DEBUG_DEFAULT=3
CONFIG_DRM_NOUVEAU_BACKLIGHT=y
# CONFIG_DRM_I810 is not set
CONFIG_DRM_I915=m
CONFIG_DRM_I915_KMS=y
# CONFIG_DRM_I915_PRELIMINARY_HW_SUPPORT is not set
CONFIG_DRM_MGA=m
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
CONFIG_DRM_VMWGFX=m
# CONFIG_DRM_VMWGFX_FBCON is not set
# CONFIG_DRM_GMA500 is not set
# CONFIG_DRM_UDL is not set
# CONFIG_DRM_AST is not set
CONFIG_DRM_MGAG200=m
CONFIG_DRM_CIRRUS_QEMU=m
CONFIG_DRM_QXL=m
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_HDMI=y
CONFIG_FB=m
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_DDC=m
# CONFIG_FB_BOOT_VESA_SUPPORT is not set
CONFIG_FB_CFB_FILLRECT=m
CONFIG_FB_CFB_COPYAREA=m
CONFIG_FB_CFB_IMAGEBLIT=m
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=m
CONFIG_FB_DEFERRED_IO=y
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I740 is not set
# CONFIG_FB_LE80578 is not set
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
# CONFIG_FB_MATROX_MYSTIQUE is not set
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_RADEON_BACKLIGHT=y
# CONFIG_FB_RADEON_DEBUG is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
CONFIG_FB_TRIDENT=m
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_TMIO is not set
# CONFIG_FB_SMSCUFX is not set
# CONFIG_FB_UDL is not set
# CONFIG_FB_GOLDFISH is not set
# CONFIG_FB_VIRTUAL is not set
CONFIG_XEN_FBDEV_FRONTEND=m
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
# CONFIG_FB_AUO_K190X is not set
# CONFIG_EXYNOS_VIDEO is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_PLATFORM=m
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_BACKLIGHT_GENERIC=m
CONFIG_BACKLIGHT_PWM=m
# CONFIG_BACKLIGHT_APPLE is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_ADP8870 is not set
# CONFIG_BACKLIGHT_LM3630 is not set
# CONFIG_BACKLIGHT_LM3639 is not set
# CONFIG_BACKLIGHT_LP855X is not set
# CONFIG_BACKLIGHT_LV5207LP is not set
# CONFIG_BACKLIGHT_BD6107 is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=m
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_LOGO is not set
CONFIG_SOUND=y
# CONFIG_SOUND_OSS_CORE is not set
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
# CONFIG_SND_SEQ_DUMMY is not set
# CONFIG_SND_MIXER_OSS is not set
# CONFIG_SND_PCM_OSS is not set
# CONFIG_SND_SEQUENCER_OSS is not set
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_MAX_CARDS=32
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_OPL3_LIB_SEQ=m
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_AC97_CODEC=m
# CONFIG_SND_DRIVERS is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ASIHPI is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
CONFIG_SND_CMIPCI=m
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_CTXFI is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
CONFIG_SND_EMU10K1=m
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
CONFIG_SND_ENS1371=m
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_PREALLOC_SIZE=2048
CONFIG_SND_HDA_HWDEP=y
CONFIG_SND_HDA_RECONFIG=y
CONFIG_SND_HDA_INPUT_BEEP=y
CONFIG_SND_HDA_INPUT_BEEP_MODE=1
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_PATCH_LOADER=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_I915=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CA0132=y
# CONFIG_SND_HDA_CODEC_CA0132_DSP is not set
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=m
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_LOLA is not set
# CONFIG_SND_LX6464ES is not set
CONFIG_SND_MAESTRO3=m
# CONFIG_SND_MAESTRO3_INPUT is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
CONFIG_SND_YMFPCI=m
# CONFIG_SND_USB is not set
CONFIG_SND_FIREWIRE=y
CONFIG_SND_FIREWIRE_LIB=m
CONFIG_SND_FIREWIRE_SPEAKERS=m
# CONFIG_SND_ISIGHT is not set
# CONFIG_SND_SCS1X is not set
# CONFIG_SND_PCMCIA is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
#
# HID support
#
CONFIG_HID=y
# CONFIG_HID_BATTERY_STRENGTH is not set
CONFIG_HIDRAW=y
# CONFIG_UHID is not set
CONFIG_HID_GENERIC=y
#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
# CONFIG_HID_ACRUX is not set
CONFIG_HID_APPLE=y
# CONFIG_HID_APPLEIR is not set
# CONFIG_HID_AUREAL is not set
CONFIG_HID_BELKIN=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
# CONFIG_HID_PRODIKEYS is not set
CONFIG_HID_CYPRESS=y
CONFIG_HID_DRAGONRISE=m
# CONFIG_DRAGONRISE_FF is not set
# CONFIG_HID_EMS_FF is not set
# CONFIG_HID_ELECOM is not set
# CONFIG_HID_ELO is not set
CONFIG_HID_EZKEY=y
# CONFIG_HID_HOLTEK is not set
# CONFIG_HID_HUION is not set
# CONFIG_HID_KEYTOUCH is not set
CONFIG_HID_KYE=m
# CONFIG_HID_UCLOGIC is not set
# CONFIG_HID_WALTOP is not set
CONFIG_HID_GYRATION=m
# CONFIG_HID_ICADE is not set
CONFIG_HID_TWINHAN=m
CONFIG_HID_KENSINGTON=y
# CONFIG_HID_LCPOWER is not set
# CONFIG_HID_LENOVO_TPKBD is not set
CONFIG_HID_LOGITECH=y
CONFIG_HID_LOGITECH_DJ=m
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
# CONFIG_LOGIG940_FF is not set
# CONFIG_LOGIWHEELS_FF is not set
# CONFIG_HID_MAGICMOUSE is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
# CONFIG_HID_MULTITOUCH is not set
CONFIG_HID_NTRIG=m
CONFIG_HID_ORTEK=m
CONFIG_HID_PANTHERLORD=m
# CONFIG_PANTHERLORD_FF is not set
CONFIG_HID_PETALYNX=m
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_PRIMAX is not set
# CONFIG_HID_ROCCAT is not set
# CONFIG_HID_SAITEK is not set
CONFIG_HID_SAMSUNG=m
# CONFIG_HID_SPEEDLINK is not set
# CONFIG_HID_STEELSERIES is not set
CONFIG_HID_SUNPLUS=m
CONFIG_HID_GREENASIA=m
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_SMARTJOYPLUS=m
# CONFIG_SMARTJOYPLUS_FF is not set
# CONFIG_HID_TIVO is not set
CONFIG_HID_TOPSEED=m
CONFIG_HID_THRUSTMASTER=m
# CONFIG_THRUSTMASTER_FF is not set
# CONFIG_HID_XINMO is not set
CONFIG_HID_ZEROPLUS=m
# CONFIG_ZEROPLUS_FF is not set
# CONFIG_HID_ZYDACRON is not set
CONFIG_HID_SENSOR_HUB=m
#
# USB HID support
#
CONFIG_USB_HID=m
# CONFIG_HID_PID is not set
CONFIG_USB_HIDDEV=y
#
# I2C HID support
#
# CONFIG_I2C_HID is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
#
# Miscellaneous USB options
#
CONFIG_USB_DEFAULT_PERSIST=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_OTG is not set
# CONFIG_USB_MON is not set
# CONFIG_USB_WUSB_CBAF is not set
#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
CONFIG_USB_EHCI_PCI=m
# CONFIG_USB_EHCI_HCD_PLATFORM is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
# CONFIG_USB_FUSBH200_HCD is not set
# CONFIG_USB_FOTG210_HCD is not set
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_HCD_PCI=m
# CONFIG_USB_OHCI_HCD_PLATFORM is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_HCD_TEST_MODE is not set
#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set
#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#
#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_REALTEK is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_STORAGE_ENE_UB6250 is not set
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_DWC3 is not set
# CONFIG_USB_CHIPIDEA is not set
#
# USB port drivers
#
# CONFIG_USB_USS720 is not set
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_SIMPLE is not set
# CONFIG_USB_SERIAL_AIRCABLE is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP210X is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
CONFIG_USB_SERIAL_FTDI_SIO=m
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_F81232 is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_IUU is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_METRO is not set
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
# CONFIG_USB_SERIAL_NAVMAN is not set
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_QCAUX is not set
# CONFIG_USB_SERIAL_QUALCOMM is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_SYMBOL is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OPTION is not set
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_OPTICON is not set
# CONFIG_USB_SERIAL_XSENS_MT is not set
# CONFIG_USB_SERIAL_WISHBONE is not set
# CONFIG_USB_SERIAL_ZTE is not set
# CONFIG_USB_SERIAL_SSU100 is not set
# CONFIG_USB_SERIAL_QT2 is not set
# CONFIG_USB_SERIAL_DEBUG is not set
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_EHSET_TEST_FIXTURE is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_YUREX is not set
# CONFIG_USB_EZUSB_FX2 is not set
# CONFIG_USB_HSIC_USB3503 is not set
#
# USB Physical Layer drivers
#
# CONFIG_USB_PHY is not set
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_AM335X_PHY_USB is not set
# CONFIG_SAMSUNG_USB2PHY is not set
# CONFIG_SAMSUNG_USB3PHY is not set
# CONFIG_USB_ISP1301 is not set
# CONFIG_USB_RCAR_PHY is not set
# CONFIG_USB_GADGET is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_SYSTOHC=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
CONFIG_RTC_INTF_DEV_UIE_EMUL=y
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
CONFIG_RTC_DRV_DS1374=m
CONFIG_RTC_DRV_DS1672=m
CONFIG_RTC_DRV_DS3232=m
CONFIG_RTC_DRV_MAX6900=m
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_ISL12022=m
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF2127=m
CONFIG_RTC_DRV_PCF8523=m
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
CONFIG_RTC_DRV_M41T80=m
CONFIG_RTC_DRV_M41T80_WDT=y
CONFIG_RTC_DRV_BQ32K=m
CONFIG_RTC_DRV_S35390A=m
CONFIG_RTC_DRV_FM3130=m
CONFIG_RTC_DRV_RX8581=m
CONFIG_RTC_DRV_RX8025=m
# CONFIG_RTC_DRV_EM3027 is not set
# CONFIG_RTC_DRV_RV3029C2 is not set
#
# SPI RTC drivers
#
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=m
CONFIG_RTC_DRV_DS1286=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_STK17TA8=m
CONFIG_RTC_DRV_M48T86=m
CONFIG_RTC_DRV_M48T35=m
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_MSM6242=m
CONFIG_RTC_DRV_BQ4802=m
CONFIG_RTC_DRV_RP5C01=m
CONFIG_RTC_DRV_V3020=m
# CONFIG_RTC_DRV_DS2404 is not set
#
# on-CPU RTC drivers
#
# CONFIG_RTC_DRV_MOXART is not set
#
# HID Sensor RTC drivers
#
CONFIG_RTC_DRV_HID_SENSOR_TIME=m
CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set
#
# DMA Devices
#
# CONFIG_INTEL_MID_DMAC is not set
CONFIG_INTEL_IOATDMA=m
# CONFIG_DW_DMAC_CORE is not set
# CONFIG_DW_DMAC is not set
# CONFIG_DW_DMAC_PCI is not set
# CONFIG_TIMB_DMA is not set
# CONFIG_PCH_DMA is not set
CONFIG_DMA_ENGINE=y
CONFIG_DMA_ACPI=y
#
# DMA Clients
#
CONFIG_NET_DMA=y
CONFIG_ASYNC_TX_DMA=y
# CONFIG_DMATEST is not set
CONFIG_DCA=m
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=m
# CONFIG_UIO_CIF is not set
# CONFIG_UIO_PDRV_GENIRQ is not set
# CONFIG_UIO_DMEM_GENIRQ is not set
# CONFIG_UIO_AEC is not set
# CONFIG_UIO_SERCOS3 is not set
# CONFIG_UIO_PCI_GENERIC is not set
# CONFIG_UIO_NETX is not set
# CONFIG_UIO_MF624 is not set
# CONFIG_VFIO is not set
CONFIG_VIRT_DRIVERS=y
CONFIG_VIRTIO=m
#
# Virtio drivers
#
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_MMIO=m
# CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set
#
# Microsoft Hyper-V guest support
#
# CONFIG_HYPERV is not set
#
# Xen driver support
#
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SELFBALLOONING=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=m
CONFIG_XEN_PCIDEV_BACKEND=m
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_ACPI_PROCESSOR=m
CONFIG_XEN_MCE_LOG=y
CONFIG_XEN_HAVE_PVMMU=y
# CONFIG_STAGING is not set
CONFIG_X86_PLATFORM_DEVICES=y
# CONFIG_ACER_WMI is not set
# CONFIG_ACERHDF is not set
# CONFIG_ASUS_LAPTOP is not set
# CONFIG_CHROMEOS_LAPTOP is not set
CONFIG_DELL_WMI=m
# CONFIG_DELL_WMI_AIO is not set
# CONFIG_FUJITSU_LAPTOP is not set
# CONFIG_FUJITSU_TABLET is not set
# CONFIG_HP_ACCEL is not set
# CONFIG_HP_WMI is not set
# CONFIG_PANASONIC_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_SENSORS_HDAPS is not set
CONFIG_INTEL_MENLOW=m
CONFIG_ACPI_WMI=m
# CONFIG_MSI_WMI is not set
# CONFIG_TOPSTAR_LAPTOP is not set
# CONFIG_ACPI_TOSHIBA is not set
# CONFIG_TOSHIBA_BT_RFKILL is not set
# CONFIG_ACPI_CMPC is not set
CONFIG_INTEL_IPS=m
# CONFIG_IBM_RTL is not set
# CONFIG_XO15_EBOOK is not set
# CONFIG_SAMSUNG_LAPTOP is not set
CONFIG_MXM_WMI=m
# CONFIG_SAMSUNG_Q10 is not set
# CONFIG_APPLE_GMUX is not set
# CONFIG_INTEL_RST is not set
# CONFIG_INTEL_SMARTCONNECT is not set
CONFIG_PVPANIC=m
#
# Hardware Spinlock drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
# CONFIG_MAILBOX is not set
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_AMD_IOMMU=y
# CONFIG_AMD_IOMMU_STATS is not set
CONFIG_DMAR_TABLE=y
CONFIG_INTEL_IOMMU=y
CONFIG_INTEL_IOMMU_DEFAULT_ON=y
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
CONFIG_IRQ_REMAP=y
#
# Remoteproc drivers
#
# CONFIG_STE_MODEM_RPROC is not set
#
# Rpmsg drivers
#
CONFIG_PM_DEVFREQ=y
#
# DEVFREQ Governors
#
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
CONFIG_DEVFREQ_GOV_PERFORMANCE=y
CONFIG_DEVFREQ_GOV_POWERSAVE=y
CONFIG_DEVFREQ_GOV_USERSPACE=y
#
# DEVFREQ Drivers
#
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
CONFIG_IIO=m
# CONFIG_IIO_BUFFER is not set
# CONFIG_IIO_TRIGGER is not set
#
# Accelerometers
#
# CONFIG_BMA180 is not set
# CONFIG_HID_SENSOR_ACCEL_3D is not set
# CONFIG_IIO_ST_ACCEL_3AXIS is not set
#
# Analog to digital converters
#
# CONFIG_MAX1363 is not set
# CONFIG_NAU7802 is not set
# CONFIG_TI_ADC081C is not set
#
# Amplifiers
#
#
# Hid Sensor IIO Common
#
CONFIG_HID_SENSOR_IIO_COMMON=m
# CONFIG_HID_SENSOR_IIO_TRIGGER is not set
# CONFIG_HID_SENSOR_ENUM_BASE_QUIRKS is not set
#
# Digital to analog converters
#
# CONFIG_AD5064 is not set
# CONFIG_AD5380 is not set
# CONFIG_AD5446 is not set
# CONFIG_MAX517 is not set
# CONFIG_MCP4725 is not set
#
# Frequency Synthesizers DDS/PLL
#
#
# Clock Generator/Distribution
#
#
# Phase-Locked Loop (PLL) frequency synthesizers
#
#
# Digital gyroscope sensors
#
# CONFIG_HID_SENSOR_GYRO_3D is not set
# CONFIG_IIO_ST_GYRO_3AXIS is not set
# CONFIG_ITG3200 is not set
#
# Inertial measurement units
#
# CONFIG_INV_MPU6050_IIO is not set
#
# Light sensors
#
# CONFIG_ADJD_S311 is not set
# CONFIG_APDS9300 is not set
# CONFIG_HID_SENSOR_ALS is not set
# CONFIG_SENSORS_TSL2563 is not set
# CONFIG_VCNL4000 is not set
#
# Magnetometer sensors
#
# CONFIG_HID_SENSOR_MAGNETOMETER_3D is not set
# CONFIG_IIO_ST_MAGN_3AXIS is not set
#
# Pressure sensors
#
# CONFIG_IIO_ST_PRESS is not set
#
# Temperature sensors
#
# CONFIG_TMP006 is not set
# CONFIG_NTB is not set
# CONFIG_VME_BUS is not set
CONFIG_PWM=y
CONFIG_PWM_SYSFS=y
# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set
# CONFIG_FMC is not set
#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=m
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
# CONFIG_GOOGLE_FIRMWARE is not set
#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=m
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_XFS_FS=y
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
# CONFIG_XFS_WARN is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_GFS2_FS is not set
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_STATS=y
# CONFIG_OCFS2_DEBUG_MASKLOG is not set
# CONFIG_OCFS2_DEBUG_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
# CONFIG_BTRFS_DEBUG is not set
# CONFIG_BTRFS_ASSERT is not set
CONFIG_NILFS2_FS=m
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not set
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y
#
# Caches
#
CONFIG_FSCACHE=y
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=y
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_LOGFS is not set
# CONFIG_CRAMFS is not set
# CONFIG_SQUASHFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX6FS_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_PSTORE=y
CONFIG_PSTORE_CONSOLE=y
# CONFIG_PSTORE_RAM is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_EXOFS_FS=m
# CONFIG_EXOFS_DEBUG is not set
CONFIG_F2FS_FS=m
CONFIG_F2FS_STAT_FS=y
CONFIG_F2FS_FS_XATTR=y
CONFIG_F2FS_FS_POSIX_ACL=y
# CONFIG_F2FS_FS_SECURITY is not set
CONFIG_ORE=m
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_SWAP=y
# CONFIG_NFS_V4_1 is not set
CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_SUNRPC_SWAP=y
CONFIG_RPCSEC_GSS_KRB5=m
# CONFIG_SUNRPC_DEBUG is not set
CONFIG_CEPH_FS=m
CONFIG_CEPH_FSCACHE=y
CONFIG_CIFS=y
CONFIG_CIFS_STATS=y
# CONFIG_CIFS_STATS2 is not set
# CONFIG_CIFS_WEAK_PW_HASH is not set
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_CIFS_ACL=y
# CONFIG_CIFS_DEBUG is not set
CONFIG_CIFS_DFS_UPCALL=y
CONFIG_CIFS_SMB2=y
CONFIG_CIFS_FSCACHE=y
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
CONFIG_AFS_FS=m
# CONFIG_AFS_DEBUG is not set
CONFIG_AFS_FSCACHE=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_MAC_ROMAN=m
CONFIG_NLS_MAC_CELTIC=m
CONFIG_NLS_MAC_CENTEURO=m
CONFIG_NLS_MAC_CROATIAN=m
CONFIG_NLS_MAC_CYRILLIC=m
CONFIG_NLS_MAC_GAELIC=m
CONFIG_NLS_MAC_GREEK=m
CONFIG_NLS_MAC_ICELAND=m
CONFIG_NLS_MAC_INUIT=m
CONFIG_NLS_MAC_ROMANIAN=m
CONFIG_NLS_MAC_TURKISH=m
CONFIG_NLS_UTF8=m
# CONFIG_DLM is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
#
# printk and dmesg options
#
# CONFIG_PRINTK_TIME is not set
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
# CONFIG_DYNAMIC_DEBUG is not set
#
# Compile-time checks and compiler options
#
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_DEBUG_KERNEL is not set
#
# Memory Debugging
#
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
#
# Debug Lockups and Hangs
#
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_DEBUG_BUGVERBOSE=y
#
# RCU Debugging
#
# CONFIG_SPARSE_RCU_POINTER is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set
#
# Runtime Testing
#
# CONFIG_LKDTM is not set
# CONFIG_ATOMIC64_SELFTEST is not set
CONFIG_ASYNC_RAID6_TEST=m
# CONFIG_TEST_STRING_HELPERS is not set
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_STRICT_DEVMEM=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_SET_MODULE_RONX is not set
CONFIG_DOUBLEFAULT=y
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_OPTIMIZE_INLINING is not set
#
# Security options
#
CONFIG_KEYS=y
CONFIG_ENCRYPTED_KEYS=m
# CONFIG_KEYS_DEBUG_PROC_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_INTEL_TXT=y
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=y
CONFIG_ASYNC_MEMCPY=y
CONFIG_ASYNC_XOR=y
CONFIG_ASYNC_PQ=y
CONFIG_ASYNC_RAID6_RECOV=y
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=m
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
# CONFIG_CRYPTO_MANAGER_DISABLE_TESTS is not set
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRYPTO_ABLK_HELPER_X86=m
CONFIG_CRYPTO_GLUE_HELPER_X86=m
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m
#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m
#
# Hash modes
#
CONFIG_CRYPTO_CMAC=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m
#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=m
CONFIG_CRYPTO_CRC32=m
CONFIG_CRYPTO_CRC32_PCLMUL=m
CONFIG_CRYPTO_CRCT10DIF=y
CONFIG_CRYPTO_CRCT10DIF_PCLMUL=m
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_MD4=y
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256_SSSE3=m
CONFIG_CRYPTO_SHA512_SSSE3=m
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m
#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=y
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAMELLIA_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST5_AVX_X86_64=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_CAST6_AVX_X86_64=m
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_X86_64=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX2_X86_64=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=m
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_ZLIB=m
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_LZ4=m
CONFIG_CRYPTO_LZ4HC=m
#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_PADLOCK is not set
CONFIG_ASYMMETRIC_KEY_TYPE=y
CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y
CONFIG_PUBLIC_KEY_ALGO_RSA=y
CONFIG_X509_CERTIFICATE_PARSER=y
CONFIG_HAVE_KVM=y
CONFIG_VIRTUALIZATION=y
# CONFIG_KVM is not set
# CONFIG_BINARY_PRINTF is not set
#
# Library routines
#
CONFIG_RAID6_PQ=y
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_CMPXCHG_LOCKREF=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_CRC32_SELFTEST=y
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
CONFIG_CRC7=m
CONFIG_LIBCRC32C=y
CONFIG_CRC8=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_COMPRESS=m
CONFIG_LZ4HC_COMPRESS=m
CONFIG_LZ4_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_DECOMPRESS_LZ4=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
CONFIG_LRU_CACHE=m
# CONFIG_AVERAGE is not set
CONFIG_CLZ_TAB=y
CONFIG_CORDIC=m
# CONFIG_DDR is not set
CONFIG_MPILIB=y
CONFIG_OID_REGISTRY=y
CONFIG_FONT_SUPPORT=m
CONFIG_FONTS=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_FONT_6x11 is not set
# CONFIG_FONT_7x14 is not set
# CONFIG_FONT_PEARL_8x8 is not set
# CONFIG_FONT_ACORN_8x8 is not set
# CONFIG_FONT_MINI_4x6 is not set
# CONFIG_FONT_SUN8x16 is not set
# CONFIG_FONT_SUN12x22 is not set
# CONFIG_FONT_10x18 is not set
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-11 7:22 ` Bob Liu
2013-12-11 9:25 ` James Dingwall
@ 2013-12-11 16:30 ` James Dingwall
2013-12-12 1:03 ` Bob Liu
2013-12-18 12:04 ` Bob Liu
1 sibling, 2 replies; 42+ messages in thread
From: James Dingwall @ 2013-12-11 16:30 UTC (permalink / raw)
To: Bob Liu, Konrad Rzeszutek Wilk; +Cc: xen-devel
Bob Liu wrote:
> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>> Konrad Rzeszutek Wilk wrote:
>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>> Hi,
>>>>>
>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>> triggers in my Xen guest domains which use ballooning to
>>>>> increase/decrease their memory allocation according to their
>>>>> requirements. One example domain I have has a maximum memory
>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>
>>>>> # free
>>>>> total used free shared buffers cached
>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>> -/+ buffers/cache: 183596 88484
>>>>> Swap: 2097148 8 2097140
>>>>>
>>>>> There is plenty of available free memory in the hypervisor to
>>>>> balloon to the maximum size:
>>>>> # xl info | grep free_mem
>>>>> free_memory : 14923
>>>>>
>>>>> An example trace (they are always the same) from the oom killer in
>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>> at will so it is difficult to start bisecting it to see if a
>>>>> particular change introduced this. However it does seem that the
>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>> memory, b) there is lots of swap available which could be used as a
>>>>> fallback.
>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>> sounds odd -but basically pages that are destined for swap end up
>> going in the tmem code which pipes them up to the hypervisor.
>>
>>>>> If other information could help or there are more tests that I could
>>>>> run then please let me know.
>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>> the guest right?
>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>> Excellent. The odd thing is that your swap is not used that much, but
>> it should be (as that is part of what the self-balloon is suppose to do).
>>
>> Bob, you had a patch for the logic of how self-balloon is suppose
>> to account for the slab - would this be relevant to this problem?
>>
> Perhaps, I have attached the patch.
> James, could you please apply it and try your application again? You
> have to rebuild the guest kernel.
> Oh, and also take a look at whether frontswap is in use, you can check
> it by watching "cat /sys/kernel/debug/frontswap/*".
I have tested this patch with a workload where I have previously seen
failures and so far so good. I'll try to keep a guest with it stressed
to see if I do get any problems. I don't know if it is expected but I
did note that the system running with this patch + selfshrink has a
kswapd0 run time of ~30mins. A guest without it and selfshrink disabled
having run a similar workload has ~5mins. With the patch I also noted
the following kernel messages which I haven't seen before:
[ 8733.646820] init_memory_mapping: [mem 0x120000000-0x127ffffff]
[ 8733.646825] [mem 0x120000000-0x127ffffff] page 4k
[10506.639875] init_memory_mapping: [mem 0x128000000-0x137ffffff]
[10506.639881] [mem 0x128000000-0x137ffffff] page 4k
James
>
> balloon.patch
>
>
> diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
> index 21e18c1..4814759 100644
> --- a/drivers/xen/xen-selfballoon.c
> +++ b/drivers/xen/xen-selfballoon.c
> @@ -191,6 +191,8 @@ static void selfballoon_process(struct work_struct *work)
> tgt_pages = cur_pages; /* default is no change */
> goal_pages = vm_memory_committed() +
> totalreserve_pages +
> + global_page_state(NR_SLAB_RECLAIMABLE) +
> + global_page_state(NR_SLAB_UNRECLAIMABLE) +
> MB2PAGES(selfballoon_reserved_mb);
> #ifdef CONFIG_FRONTSWAP
> /* allow space for frontswap pages to be repatriated */
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 580a5f0..863b05c 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -16,6 +16,7 @@
>
> #include <linux/stddef.h>
> #include <linux/mm.h>
> +#include <linux/mman.h>
> #include <linux/swap.h>
> #include <linux/interrupt.h>
> #include <linux/pagemap.h>
> @@ -3075,7 +3076,7 @@ void show_free_areas(unsigned int filter)
> " dirty:%lu writeback:%lu unstable:%lu\n"
> " free:%lu slab_reclaimable:%lu slab_unreclaimable:%lu\n"
> " mapped:%lu shmem:%lu pagetables:%lu bounce:%lu\n"
> - " free_cma:%lu\n",
> + " free_cma:%lu totalram:%lu balloontarget:%lu\n",
> global_page_state(NR_ACTIVE_ANON),
> global_page_state(NR_INACTIVE_ANON),
> global_page_state(NR_ISOLATED_ANON),
> @@ -3093,7 +3094,9 @@ void show_free_areas(unsigned int filter)
> global_page_state(NR_SHMEM),
> global_page_state(NR_PAGETABLE),
> global_page_state(NR_BOUNCE),
> - global_page_state(NR_FREE_CMA_PAGES));
> + global_page_state(NR_FREE_CMA_PAGES),
> + totalram_pages,
> + vm_memory_committed() + totalreserve_pages);
>
> for_each_populated_zone(zone) {
> int i;
--
*James Dingwall*
Script Monkey
zynstra-signature-logo <http://www.zynstra.com/>twitter-black
<http://www.twitter.com/zynstra>linkedin-black
<http://www.linkedin.com/company/zynstra>
Zynstra is a private limited company registered in England and Wales
(registered number 07864369). Our registered office is 5 New Street
Square, London, EC4A 3TW and our headquarters are at Bath Ventures,
Broad Quay, Bath, BA1 1UD.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-11 16:30 ` James Dingwall
@ 2013-12-12 1:03 ` Bob Liu
2013-12-13 16:59 ` James Dingwall
2013-12-18 12:04 ` Bob Liu
1 sibling, 1 reply; 42+ messages in thread
From: Bob Liu @ 2013-12-12 1:03 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 12/12/2013 12:30 AM, James Dingwall wrote:
> Bob Liu wrote:
>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>> Konrad Rzeszutek Wilk wrote:
>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>> increase/decrease their memory allocation according to their
>>>>>> requirements. One example domain I have has a maximum memory
>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>
>>>>>> # free
>>>>>> total used free shared buffers
>>>>>> cached
>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>> -/+ buffers/cache: 183596 88484
>>>>>> Swap: 2097148 8 2097140
>>>>>>
>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>> balloon to the maximum size:
>>>>>> # xl info | grep free_mem
>>>>>> free_memory : 14923
>>>>>>
>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>> particular change introduced this. However it does seem that the
>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>> fallback.
>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>> sounds odd -but basically pages that are destined for swap end up
>>> going in the tmem code which pipes them up to the hypervisor.
>>>
>>>>>> If other information could help or there are more tests that I could
>>>>>> run then please let me know.
>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>> the guest right?
>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>> Excellent. The odd thing is that your swap is not used that much, but
>>> it should be (as that is part of what the self-balloon is suppose to
>>> do).
>>>
>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>> to account for the slab - would this be relevant to this problem?
>>>
>> Perhaps, I have attached the patch.
>> James, could you please apply it and try your application again? You
>> have to rebuild the guest kernel.
>> Oh, and also take a look at whether frontswap is in use, you can check
>> it by watching "cat /sys/kernel/debug/frontswap/*".
> I have tested this patch with a workload where I have previously seen
Thank you so much.
> failures and so far so good. I'll try to keep a guest with it stressed
> to see if I do get any problems. I don't know if it is expected but I
> did note that the system running with this patch + selfshrink has a
> kswapd0 run time of ~30mins. A guest without it and selfshrink disabled
Could you run the test again with this patch but selfshrink disabled and
compare the run time of kswapd0?
> having run a similar workload has ~5mins. With the patch I also noted
> the following kernel messages which I haven't seen before:
>
> [ 8733.646820] init_memory_mapping: [mem 0x120000000-0x127ffffff]
> [ 8733.646825] [mem 0x120000000-0x127ffffff] page 4k
> [10506.639875] init_memory_mapping: [mem 0x128000000-0x137ffffff]
> [10506.639881] [mem 0x128000000-0x137ffffff] page 4k
>
--
Regards,
-Bob
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-12 1:03 ` Bob Liu
@ 2013-12-13 16:59 ` James Dingwall
2013-12-17 6:11 ` Bob Liu
0 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2013-12-13 16:59 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 12/12/2013 12:30 AM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>> increase/decrease their memory allocation according to their
>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>
>>>>>>> # free
>>>>>>> total used free shared buffers
>>>>>>> cached
>>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>> Swap: 2097148 8 2097140
>>>>>>>
>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>> balloon to the maximum size:
>>>>>>> # xl info | grep free_mem
>>>>>>> free_memory : 14923
>>>>>>>
>>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>> particular change introduced this. However it does seem that the
>>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>>> fallback.
>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>> sounds odd -but basically pages that are destined for swap end up
>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>
>>>>>>> If other information could help or there are more tests that I could
>>>>>>> run then please let me know.
>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>> the guest right?
>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>> Excellent. The odd thing is that your swap is not used that much, but
>>>> it should be (as that is part of what the self-balloon is suppose to
>>>> do).
>>>>
>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>> to account for the slab - would this be relevant to this problem?
>>>>
>>> Perhaps, I have attached the patch.
>>> James, could you please apply it and try your application again? You
>>> have to rebuild the guest kernel.
>>> Oh, and also take a look at whether frontswap is in use, you can check
>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>> I have tested this patch with a workload where I have previously seen
> Thank you so much.
>
>> failures and so far so good. I'll try to keep a guest with it stressed
>> to see if I do get any problems. I don't know if it is expected but I
>> did note that the system running with this patch + selfshrink has a
>> kswapd0 run time of ~30mins. A guest without it and selfshrink disabled
> Could you run the test again with this patch but selfshrink disabled and
> compare the run time of kswapd0?
Here are the results against two vms with/without the patch. They are
running on the same dom0 and have comparable xen configs and were
restarted at the same point.
With patch:
# uptime ; ps -ef | grep [k]swapd0
14:58:55 up 6:32, 1 user, load average: 0.00, 0.01, 0.05
root 310 2 0 08:26 ? 00:00:01 [kswapd0]
### BUILD GLIBC
# ps -ef | grep [k]swapd0
root 310 2 0 08:26 ? 00:00:16 [kswapd0]
### BUILD KDELIBS
# ps -ef | grep [k]swapd0
root 310 2 1 08:26 ? 00:09:15 [kswapd0]
# for i in /sys/module/tmem/parameters/* ; do echo $i $(< $i) ; done
/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking N
Without patch:
# uptime ; ps -ef | grep [k]swapd0
14:59:12 up 6:32, 1 user, load average: 0.00, 0.01, 0.05
root 309 2 0 08:26 ? 00:00:01 [kswapd0]
### BUILD GLIBC
# ps -ef | grep [k]swapd0
root 309 2 0 08:26 ? 00:00:09 [kswapd0]
### BUILD KDELIBS
# ps -ef | grep [k]swapd0
root 309 2 0 08:26 ? 00:01:18 [kswapd0]
# for i in /sys/module/tmem/parameters/* ; do echo $i $(< $i) ; done
/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking N
>> having run a similar workload has ~5mins. With the patch I also noted
>> the following kernel messages which I haven't seen before:
>>
>> [ 8733.646820] init_memory_mapping: [mem 0x120000000-0x127ffffff]
>> [ 8733.646825] [mem 0x120000000-0x127ffffff] page 4k
>> [10506.639875] init_memory_mapping: [mem 0x128000000-0x137ffffff]
>> [10506.639881] [mem 0x128000000-0x137ffffff] page 4k
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-13 16:59 ` James Dingwall
@ 2013-12-17 6:11 ` Bob Liu
0 siblings, 0 replies; 42+ messages in thread
From: Bob Liu @ 2013-12-17 6:11 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 12/14/2013 12:59 AM, James Dingwall wrote:
> Bob Liu wrote:
>> On 12/12/2013 12:30 AM, James Dingwall wrote:
>>> Bob Liu wrote:
>>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>>> increase/decrease their memory allocation according to their
>>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>>
>>>>>>>> # free
>>>>>>>> total used free shared buffers
>>>>>>>> cached
>>>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>>> Swap: 2097148 8 2097140
>>>>>>>>
>>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>>> balloon to the maximum size:
>>>>>>>> # xl info | grep free_mem
>>>>>>>> free_memory : 14923
>>>>>>>>
>>>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>>> particular change introduced this. However it does seem that the
>>>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>>>> fallback.
>>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>>> sounds odd -but basically pages that are destined for swap end up
>>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>>
>>>>>>>> If other information could help or there are more tests that I
>>>>>>>> could
>>>>>>>> run then please let me know.
>>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>>> the guest right?
>>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>>> Excellent. The odd thing is that your swap is not used that much, but
>>>>> it should be (as that is part of what the self-balloon is suppose to
>>>>> do).
>>>>>
>>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>>> to account for the slab - would this be relevant to this problem?
>>>>>
>>>> Perhaps, I have attached the patch.
>>>> James, could you please apply it and try your application again? You
>>>> have to rebuild the guest kernel.
>>>> Oh, and also take a look at whether frontswap is in use, you can check
>>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>>> I have tested this patch with a workload where I have previously seen
>> Thank you so much.
>>
>>> failures and so far so good. I'll try to keep a guest with it stressed
>>> to see if I do get any problems. I don't know if it is expected but I
>>> did note that the system running with this patch + selfshrink has a
>>> kswapd0 run time of ~30mins. A guest without it and selfshrink disabled
>> Could you run the test again with this patch but selfshrink disabled and
>> compare the run time of kswapd0?
> Here are the results against two vms with/without the patch. They are
> running on the same dom0 and have comparable xen configs and were
> restarted at the same point.
>
Sorry for the later response.
> With patch:
>
> # uptime ; ps -ef | grep [k]swapd0
> 14:58:55 up 6:32, 1 user, load average: 0.00, 0.01, 0.05
> root 310 2 0 08:26 ? 00:00:01 [kswapd0]
> ### BUILD GLIBC
> # ps -ef | grep [k]swapd0
> root 310 2 0 08:26 ? 00:00:16 [kswapd0]
> ### BUILD KDELIBS
> # ps -ef | grep [k]swapd0
> root 310 2 1 08:26 ? 00:09:15 [kswapd0]
Still have a longer kswapd run time, it's strange.
In theory, with this patch more memory will be reserved to guest os and
as a result it should have shorter kswapd time.
--
Regards,
-Bob
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-11 16:30 ` James Dingwall
2013-12-12 1:03 ` Bob Liu
@ 2013-12-18 12:04 ` Bob Liu
2013-12-19 19:08 ` James Dingwall
1 sibling, 1 reply; 42+ messages in thread
From: Bob Liu @ 2013-12-18 12:04 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 12/12/2013 12:30 AM, James Dingwall wrote:
> Bob Liu wrote:
>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>> Konrad Rzeszutek Wilk wrote:
>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>> increase/decrease their memory allocation according to their
>>>>>> requirements. One example domain I have has a maximum memory
>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>
>>>>>> # free
>>>>>> total used free shared buffers
>>>>>> cached
>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>> -/+ buffers/cache: 183596 88484
>>>>>> Swap: 2097148 8 2097140
>>>>>>
>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>> balloon to the maximum size:
>>>>>> # xl info | grep free_mem
>>>>>> free_memory : 14923
>>>>>>
>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>> particular change introduced this. However it does seem that the
>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>> fallback.
>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>> sounds odd -but basically pages that are destined for swap end up
>>> going in the tmem code which pipes them up to the hypervisor.
>>>
>>>>>> If other information could help or there are more tests that I could
>>>>>> run then please let me know.
>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>> the guest right?
>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>> Excellent. The odd thing is that your swap is not used that much, but
>>> it should be (as that is part of what the self-balloon is suppose to
>>> do).
>>>
>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>> to account for the slab - would this be relevant to this problem?
>>>
>> Perhaps, I have attached the patch.
>> James, could you please apply it and try your application again? You
>> have to rebuild the guest kernel.
>> Oh, and also take a look at whether frontswap is in use, you can check
>> it by watching "cat /sys/kernel/debug/frontswap/*".
> I have tested this patch with a workload where I have previously seen
> failures and so far so good. I'll try to keep a guest with it stressed
> to see if I do get any problems. I don't know if it is expected but I
By the way, besides longer time of kswapd, is this patch work well
during your stress testing?
Have you seen the OOM killer triggered quite frequently again?(with
selfshrink=true)
Thanks,
-Bob
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-18 12:04 ` Bob Liu
@ 2013-12-19 19:08 ` James Dingwall
2013-12-20 3:17 ` Bob Liu
0 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2013-12-19 19:08 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 12/12/2013 12:30 AM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>> increase/decrease their memory allocation according to their
>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>
>>>>>>> # free
>>>>>>> total used free shared buffers
>>>>>>> cached
>>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>> Swap: 2097148 8 2097140
>>>>>>>
>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>> balloon to the maximum size:
>>>>>>> # xl info | grep free_mem
>>>>>>> free_memory : 14923
>>>>>>>
>>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>> particular change introduced this. However it does seem that the
>>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>>> fallback.
>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>> sounds odd -but basically pages that are destined for swap end up
>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>
>>>>>>> If other information could help or there are more tests that I could
>>>>>>> run then please let me know.
>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>> the guest right?
>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>> Excellent. The odd thing is that your swap is not used that much, but
>>>> it should be (as that is part of what the self-balloon is suppose to
>>>> do).
>>>>
>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>> to account for the slab - would this be relevant to this problem?
>>>>
>>> Perhaps, I have attached the patch.
>>> James, could you please apply it and try your application again? You
>>> have to rebuild the guest kernel.
>>> Oh, and also take a look at whether frontswap is in use, you can check
>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>> I have tested this patch with a workload where I have previously seen
>> failures and so far so good. I'll try to keep a guest with it stressed
>> to see if I do get any problems. I don't know if it is expected but I
> By the way, besides longer time of kswapd, is this patch work well
> during your stress testing?
>
> Have you seen the OOM killer triggered quite frequently again?(with
> selfshrink=true)
>
> Thanks,
> -Bob
It was looking good until today (selfshrink=true). The trace below is
during a compile of subversion, it looks like the memory has ballooned
to almost the maximum permissible but even under pressure the swap disk
has hardly come in to use.
James
[76253.420363] javac invoked oom-killer: gfp_mask=0x280da, order=0,
oom_score_adj=0
[76253.420371] CPU: 0 PID: 4995 Comm: javac Tainted: G W 3.12.5 #87
[76253.420374] ffff88001289fcb8 ffff880100ed1a58 ffffffff8148f1e0
ffff88001f80e8e8
[76253.420378] ffff88001289f780 ffff880100ed1af8 ffffffff8148ccd7
ffff880100ed1aa8
[76253.420381] ffffffff810f8d97 ffff880100ed1a88 ffffffff81006dc8
ffff880100ed1a98
[76253.420385] Call Trace:
[76253.420402] [<ffffffff8148f1e0>] dump_stack+0x46/0x58
[76253.420406] [<ffffffff8148ccd7>] dump_header.isra.9+0x6d/0x1cc
[76253.420412] [<ffffffff810f8d97>] ? super_cache_count+0xa8/0xb8
[76253.420417] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[76253.420421] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
[76253.420426] [<ffffffff81494a9e>] ? _raw_spin_unlock_irqrestore+0x47/0x62
[76253.420432] [<ffffffff81296b27>] ? ___ratelimit+0xcb/0xe8
[76253.420437] [<ffffffff810b2bbf>] oom_kill_process+0x70/0x2fd
[76253.420441] [<ffffffff810bca0e>] ? zone_reclaimable+0x11/0x1e
[76253.420446] [<ffffffff81048779>] ? has_ns_capability_noaudit+0x12/0x19
[76253.420449] [<ffffffff81048792>] ? has_capability_noaudit+0x12/0x14
[76253.420452] [<ffffffff810b32de>] out_of_memory+0x31b/0x34e
[76253.420456] [<ffffffff810b7438>] __alloc_pages_nodemask+0x65b/0x792
[76253.420460] [<ffffffff810e3da3>] alloc_pages_vma+0xd0/0x10c
[76253.420464] [<ffffffff81003f69>] ?
__raw_callee_save_xen_pmd_val+0x11/0x1e
[76253.420468] [<ffffffff810cf7cd>] handle_mm_fault+0x6d4/0xd54
[76253.420471] [<ffffffff810d59e7>] ? change_protection+0x4d7/0x66c
[76253.420474] [<ffffffff8149538a>] ? error_exit+0x2a/0x60
[76253.420480] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
[76253.420483] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[76253.420487] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[76253.420493] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
[76253.420497] [<ffffffff810a45cc>] ? __acct_update_integrals+0xb4/0xbf
[76253.420499] [<ffffffff810a493f>] ? acct_account_cputime+0x17/0x19
[76253.420503] [<ffffffff81067c28>] ? account_user_time+0x67/0x92
[76253.420506] [<ffffffff8106811b>] ? vtime_account_user+0x4d/0x52
[76253.420510] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
[76253.420512] [<ffffffff81495158>] page_fault+0x28/0x30
[76253.420514] Mem-Info:
[76253.420515] Node 0 DMA per-cpu:
[76253.420518] CPU 0: hi: 0, btch: 1 usd: 0
[76253.420520] CPU 1: hi: 0, btch: 1 usd: 0
[76253.420521] Node 0 DMA32 per-cpu:
[76253.420523] CPU 0: hi: 186, btch: 31 usd: 137
[76253.420525] CPU 1: hi: 186, btch: 31 usd: 110
[76253.420526] Node 0 Normal per-cpu:
[76253.420528] CPU 0: hi: 0, btch: 1 usd: 0
[76253.420529] CPU 1: hi: 0, btch: 1 usd: 0
[76253.420535] active_anon:28274 inactive_anon:35997 isolated_anon:0
active_file:10839 inactive_file:15340 isolated_file:0
unevictable:0 dirty:0 writeback:2 unstable:0
free:1142 slab_reclaimable:3001 slab_unreclaimable:3771
mapped:3635 shmem:140 pagetables:2168 bounce:0
free_cma:0 totalram:109276 balloontarget:107948
[76253.420537] Node 0 DMA free:1952kB min:88kB low:108kB high:132kB
active_anon:976kB inactive_anon:1368kB active_file:692kB
inactive_file:916kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:7576kB mlocked:0kB dirty:0kB
writeback:0kB mapped:196kB shmem:0kB slab_reclaimable:220kB
slab_unreclaimable:304kB kernel_stack:112kB pagetables:64kB unstable:0kB
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6633
all_unreclaimable? yes
[76253.420546] lowmem_reserve[]: 0 469 469 469
[76253.420549] Node 0 DMA32 free:2616kB min:2728kB low:3408kB
high:4092kB active_anon:31720kB inactive_anon:62120kB
active_file:15304kB inactive_file:32692kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:507904kB managed:205076kB
mlocked:0kB dirty:0kB writeback:0kB mapped:4784kB shmem:528kB
slab_reclaimable:7908kB slab_unreclaimable:13148kB kernel_stack:1624kB
pagetables:7500kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:215144 all_unreclaimable? yes
[76253.420557] lowmem_reserve[]: 0 0 0 0
[76253.420560] Node 0 Normal free:0kB min:0kB low:0kB high:0kB
active_anon:80400kB inactive_anon:80500kB active_file:27360kB
inactive_file:27752kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:524288kB managed:224452kB mlocked:0kB
dirty:0kB writeback:8kB mapped:9560kB shmem:32kB slab_reclaimable:3876kB
slab_unreclaimable:1632kB kernel_stack:136kB pagetables:1108kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:324394 all_unreclaimable? yes
[76253.420567] lowmem_reserve[]: 0 0 0 0
[76253.420570] Node 0 DMA: 1*4kB (U) 2*8kB (R) 1*16kB (R) 2*32kB (R)
1*64kB (R) 0*128kB 1*256kB (R) 1*512kB (R) 1*1024kB (R) 0*2048kB
0*4096kB = 1956kB
[76253.420584] Node 0 DMA32: 626*4kB (UEM) 14*8kB (UM) 0*16kB 0*32kB
0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2616kB
[76253.420594] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[76253.420603] 11432 total pagecache pages
[76253.420605] 427 pages in swap cache
[76253.420606] Swap cache stats: add 97961, delete 97534, find 10712/12435
[76253.420607] Free swap = 2065332kB
[76253.420608] Total swap = 2097148kB
[76253.434007] 425983 pages RAM
[76253.434008] 170422 pages reserved
[76253.434009] 546002 pages shared
[76253.434010] 246013 pages non-shared
<snip process list>
[76253.434243] Out of memory: Kill process 4989 (javac) score 36 or
sacrifice child
[76253.434248] Killed process 4989 (javac) total-vm:1194836kB,
anon-rss:79368kB, file-rss:9908kB
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-19 19:08 ` James Dingwall
@ 2013-12-20 3:17 ` Bob Liu
2013-12-20 12:22 ` James Dingwall
2013-12-26 8:42 ` James Dingwall
0 siblings, 2 replies; 42+ messages in thread
From: Bob Liu @ 2013-12-20 3:17 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 4021 bytes --]
On 12/20/2013 03:08 AM, James Dingwall wrote:
> Bob Liu wrote:
>> On 12/12/2013 12:30 AM, James Dingwall wrote:
>>> Bob Liu wrote:
>>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>>> increase/decrease their memory allocation according to their
>>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>>
>>>>>>>> # free
>>>>>>>> total used free shared buffers
>>>>>>>> cached
>>>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>>> Swap: 2097148 8 2097140
>>>>>>>>
>>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>>> balloon to the maximum size:
>>>>>>>> # xl info | grep free_mem
>>>>>>>> free_memory : 14923
>>>>>>>>
>>>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>>> particular change introduced this. However it does seem that the
>>>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>>>> fallback.
>>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>>> sounds odd -but basically pages that are destined for swap end up
>>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>>
>>>>>>>> If other information could help or there are more tests that I
>>>>>>>> could
>>>>>>>> run then please let me know.
>>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>>> the guest right?
>>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>>> Excellent. The odd thing is that your swap is not used that much, but
>>>>> it should be (as that is part of what the self-balloon is suppose to
>>>>> do).
>>>>>
>>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>>> to account for the slab - would this be relevant to this problem?
>>>>>
>>>> Perhaps, I have attached the patch.
>>>> James, could you please apply it and try your application again? You
>>>> have to rebuild the guest kernel.
>>>> Oh, and also take a look at whether frontswap is in use, you can check
>>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>>> I have tested this patch with a workload where I have previously seen
>>> failures and so far so good. I'll try to keep a guest with it stressed
>>> to see if I do get any problems. I don't know if it is expected but I
>> By the way, besides longer time of kswapd, is this patch work well
>> during your stress testing?
>>
>> Have you seen the OOM killer triggered quite frequently again?(with
>> selfshrink=true)
>>
>> Thanks,
>> -Bob
> It was looking good until today (selfshrink=true). The trace below is
> during a compile of subversion, it looks like the memory has ballooned
> to almost the maximum permissible but even under pressure the swap disk
> has hardly come in to use.
>
So if without selfshrink the swap disk can be used a lot?
If that's the case, I'm afraid the frontswap-selfshrink in
xen-selfballoon did something incorrect.
Could you please try this patch which make the frontswap-selfshrink
slower and add a printk for debug.
Please still keep selfshrink=true in your test but can with or without
my previous patch.
Thanks a lot!
-Bob
[-- Attachment #2: a.patch --]
[-- Type: text/x-patch, Size: 949 bytes --]
diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
index 21e18c1..6e9bf0b 100644
--- a/drivers/xen/xen-selfballoon.c
+++ b/drivers/xen/xen-selfballoon.c
@@ -133,7 +133,7 @@ static unsigned int frontswap_hysteresis __read_mostly = 20;
* frontswap selfshrinking should commence. Note that selfshrinking does
* not use a separate worker thread.
*/
-static unsigned int frontswap_inertia __read_mostly = 3;
+static unsigned int frontswap_inertia __read_mostly = 6;
/* Countdown to next invocation of frontswap_shrink() */
static unsigned long frontswap_inertia_counter;
@@ -170,6 +170,8 @@ static void frontswap_selfshrink(void)
tgt_frontswap_pages = cur_frontswap_pages -
(cur_frontswap_pages / frontswap_hysteresis);
frontswap_shrink(tgt_frontswap_pages);
+ printk("frontswap selfshrink %ld pages\n", tgt_frontswap_pages);
+ frontswap_inertia_counter = frontswap_inertia;
}
#endif /* CONFIG_FRONTSWAP */
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-20 3:17 ` Bob Liu
@ 2013-12-20 12:22 ` James Dingwall
2013-12-26 8:42 ` James Dingwall
1 sibling, 0 replies; 42+ messages in thread
From: James Dingwall @ 2013-12-20 12:22 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 12/20/2013 03:08 AM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 12/12/2013 12:30 AM, James Dingwall wrote:
>>>> Bob Liu wrote:
>>>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>>>> increase/decrease their memory allocation according to their
>>>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>>>
>>>>>>>>> # free
>>>>>>>>> total used free shared buffers
>>>>>>>>> cached
>>>>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>>>> Swap: 2097148 8 2097140
>>>>>>>>>
>>>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>>>> balloon to the maximum size:
>>>>>>>>> # xl info | grep free_mem
>>>>>>>>> free_memory : 14923
>>>>>>>>>
>>>>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>>>> particular change introduced this. However it does seem that the
>>>>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>>>>> fallback.
>>>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>>>> sounds odd -but basically pages that are destined for swap end up
>>>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>>>
>>>>>>>>> If other information could help or there are more tests that I
>>>>>>>>> could
>>>>>>>>> run then please let me know.
>>>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>>>> the guest right?
>>>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>>>> Excellent. The odd thing is that your swap is not used that much, but
>>>>>> it should be (as that is part of what the self-balloon is suppose to
>>>>>> do).
>>>>>>
>>>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>>>> to account for the slab - would this be relevant to this problem?
>>>>>>
>>>>> Perhaps, I have attached the patch.
>>>>> James, could you please apply it and try your application again? You
>>>>> have to rebuild the guest kernel.
>>>>> Oh, and also take a look at whether frontswap is in use, you can check
>>>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>>>> I have tested this patch with a workload where I have previously seen
>>>> failures and so far so good. I'll try to keep a guest with it stressed
>>>> to see if I do get any problems. I don't know if it is expected but I
>>> By the way, besides longer time of kswapd, is this patch work well
>>> during your stress testing?
>>>
>>> Have you seen the OOM killer triggered quite frequently again?(with
>>> selfshrink=true)
>>>
>>> Thanks,
>>> -Bob
>> It was looking good until today (selfshrink=true). The trace below is
>> during a compile of subversion, it looks like the memory has ballooned
>> to almost the maximum permissible but even under pressure the swap disk
>> has hardly come in to use.
>>
> So if without selfshrink the swap disk can be used a lot?
>
> If that's the case, I'm afraid the frontswap-selfshrink in
> xen-selfballoon did something incorrect.
>
> Could you please try this patch which make the frontswap-selfshrink
> slower and add a printk for debug.
> Please still keep selfshrink=true in your test but can with or without
> my previous patch.
> Thanks a lot!
>
> -Bob
I have booted a guest with this patch so I'll let you know. With my
original bit of code which deliberately leaked memory there is no
problem with getting swap space used so my thoughts were that it is to
do with the timing or race between ballooning and memory allocation.
>
> a.patch
>
>
> diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
> index 21e18c1..6e9bf0b 100644
> --- a/drivers/xen/xen-selfballoon.c
> +++ b/drivers/xen/xen-selfballoon.c
> @@ -133,7 +133,7 @@ static unsigned int frontswap_hysteresis __read_mostly = 20;
> * frontswap selfshrinking should commence. Note that selfshrinking does
> * not use a separate worker thread.
> */
> -static unsigned int frontswap_inertia __read_mostly = 3;
> +static unsigned int frontswap_inertia __read_mostly = 6;
>
> /* Countdown to next invocation of frontswap_shrink() */
> static unsigned long frontswap_inertia_counter;
> @@ -170,6 +170,8 @@ static void frontswap_selfshrink(void)
> tgt_frontswap_pages = cur_frontswap_pages -
> (cur_frontswap_pages / frontswap_hysteresis);
> frontswap_shrink(tgt_frontswap_pages);
> + printk("frontswap selfshrink %ld pages\n", tgt_frontswap_pages);
> + frontswap_inertia_counter = frontswap_inertia;
> }
>
> #endif /* CONFIG_FRONTSWAP */
--
*James Dingwall*
Script Monkey
zynstra-signature-logo <http://www.zynstra.com/>twitter-black
<http://www.twitter.com/zynstra>linkedin-black
<http://www.linkedin.com/company/zynstra>
Zynstra is a private limited company registered in England and Wales
(registered number 07864369). Our registered office is 5 New Street
Square, London, EC4A 3TW and our headquarters are at Bath Ventures,
Broad Quay, Bath, BA1 1UD.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-20 3:17 ` Bob Liu
2013-12-20 12:22 ` James Dingwall
@ 2013-12-26 8:42 ` James Dingwall
2014-01-02 6:25 ` Bob Liu
1 sibling, 1 reply; 42+ messages in thread
From: James Dingwall @ 2013-12-26 8:42 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 12/20/2013 03:08 AM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 12/12/2013 12:30 AM, James Dingwall wrote:
>>>> Bob Liu wrote:
>>>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>>>> increase/decrease their memory allocation according to their
>>>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>>>
>>>>>>>>> # free
>>>>>>>>> total used free shared buffers
>>>>>>>>> cached
>>>>>>>>> Mem: 272080 248108 23972 0 1448 63064
>>>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>>>> Swap: 2097148 8 2097140
>>>>>>>>>
>>>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>>>> balloon to the maximum size:
>>>>>>>>> # xl info | grep free_mem
>>>>>>>>> free_memory : 14923
>>>>>>>>>
>>>>>>>>> An example trace (they are always the same) from the oom killer in
>>>>>>>>> 3.12 is added below. So far I have not been able to reproduce this
>>>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>>>> particular change introduced this. However it does seem that the
>>>>>>>>> behaviour is wrong because a) ballooning could give the guest more
>>>>>>>>> memory, b) there is lots of swap available which could be used as a
>>>>>>>>> fallback.
>>>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>>>> sounds odd -but basically pages that are destined for swap end up
>>>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>>>
>>>>>>>>> If other information could help or there are more tests that I
>>>>>>>>> could
>>>>>>>>> run then please let me know.
>>>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>>>> the guest right?
>>>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>>>> Excellent. The odd thing is that your swap is not used that much, but
>>>>>> it should be (as that is part of what the self-balloon is suppose to
>>>>>> do).
>>>>>>
>>>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>>>> to account for the slab - would this be relevant to this problem?
>>>>>>
>>>>> Perhaps, I have attached the patch.
>>>>> James, could you please apply it and try your application again? You
>>>>> have to rebuild the guest kernel.
>>>>> Oh, and also take a look at whether frontswap is in use, you can check
>>>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>>>> I have tested this patch with a workload where I have previously seen
>>>> failures and so far so good. I'll try to keep a guest with it stressed
>>>> to see if I do get any problems. I don't know if it is expected but I
>>> By the way, besides longer time of kswapd, is this patch work well
>>> during your stress testing?
>>>
>>> Have you seen the OOM killer triggered quite frequently again?(with
>>> selfshrink=true)
>>>
>>> Thanks,
>>> -Bob
>> It was looking good until today (selfshrink=true). The trace below is
>> during a compile of subversion, it looks like the memory has ballooned
>> to almost the maximum permissible but even under pressure the swap disk
>> has hardly come in to use.
>>
> So if without selfshrink the swap disk can be used a lot?
>
> If that's the case, I'm afraid the frontswap-selfshrink in
> xen-selfballoon did something incorrect.
>
> Could you please try this patch which make the frontswap-selfshrink
> slower and add a printk for debug.
> Please still keep selfshrink=true in your test but can with or without
> my previous patch.
> Thanks a lot!
>
The oom trace below was triggered during a compile of gcc. I have the
full dmesg from boot which shows all the printks, please let me know if
you would like to see that.
James
[504372.929678] frontswap selfshrink 5424 pages
[504403.018185] frontswap selfshrink 5152 pages
[504433.124844] frontswap selfshrink 4894 pages
[504468.335358] frontswap selfshrink 12791 pages
[504498.536467] frontswap selfshrink 12152 pages
[504533.813484] frontswap selfshrink 19751 pages
[504589.067299] frontswap selfshrink 19043 pages
[504638.441894] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0,
oom_score_adj=0
[504638.441902] CPU: 1 PID: 21506 Comm: cc1plus Tainted: G W 3.12.5 #88
[504638.441905] ffff88001ca406f8 ffff880002c0fa58 ffffffff8148f200
ffff88001f90e8e8
[504638.441909] ffff88001ca401c0 ffff880002c0faf8 ffffffff8148ccf7
ffff880002c0faa8
[504638.441912] ffffffff810f8d97 ffff880002c0fa88 ffffffff81006dc8
ffff880002c0fa98
[504638.441917] Call Trace:
[504638.441928] [<ffffffff8148f200>] dump_stack+0x46/0x58
[504638.441932] [<ffffffff8148ccf7>] dump_header.isra.9+0x6d/0x1cc
[504638.441938] [<ffffffff810f8d97>] ? super_cache_count+0xa8/0xb8
[504638.441943] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[504638.441946] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
[504638.441951] [<ffffffff81494abe>] ?
_raw_spin_unlock_irqrestore+0x47/0x62
[504638.441957] [<ffffffff81296b27>] ? ___ratelimit+0xcb/0xe8
[504638.441962] [<ffffffff810b2bbf>] oom_kill_process+0x70/0x2fd
[504638.441966] [<ffffffff810bca0e>] ? zone_reclaimable+0x11/0x1e
[504638.441970] [<ffffffff81048779>] ? has_ns_capability_noaudit+0x12/0x19
[504638.441973] [<ffffffff81048792>] ? has_capability_noaudit+0x12/0x14
[504638.441976] [<ffffffff810b32de>] out_of_memory+0x31b/0x34e
[504638.441981] [<ffffffff810b7438>] __alloc_pages_nodemask+0x65b/0x792
[504638.441985] [<ffffffff810e3da3>] alloc_pages_vma+0xd0/0x10c
[504638.441988] [<ffffffff81003f69>] ?
__raw_callee_save_xen_pmd_val+0x11/0x1e
[504638.441993] [<ffffffff810cf7cd>] handle_mm_fault+0x6d4/0xd54
[504638.441996] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[504638.441999] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[504638.442005] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
[504638.442008] [<ffffffff8106823b>] ? arch_vtime_task_switch+0x81/0x86
[504638.442013] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
[504638.442016] [<ffffffff81062f1e>] ? finish_task_switch+0xe8/0x144
[504638.442018] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[504638.442021] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
[504638.442025] [<ffffffff810a45cc>] ? __acct_update_integrals+0xb4/0xbf
[504638.442028] [<ffffffff810a493f>] ? acct_account_cputime+0x17/0x19
[504638.442030] [<ffffffff81067c28>] ? account_user_time+0x67/0x92
[504638.442033] [<ffffffff8106811b>] ? vtime_account_user+0x4d/0x52
[504638.442036] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
[504638.442041] [<ffffffff810a065f>] ? rcu_user_enter+0xe/0x10
[504638.442044] [<ffffffff81495158>] page_fault+0x28/0x30
[504638.442046] Mem-Info:
[504638.442048] Node 0 DMA per-cpu:
[504638.442050] CPU 0: hi: 0, btch: 1 usd: 0
[504638.442052] CPU 1: hi: 0, btch: 1 usd: 0
[504638.442053] Node 0 DMA32 per-cpu:
[504638.442055] CPU 0: hi: 186, btch: 31 usd: 26
[504638.442057] CPU 1: hi: 186, btch: 31 usd: 72
[504638.442058] Node 0 Normal per-cpu:
[504638.442060] CPU 0: hi: 0, btch: 1 usd: 0
[504638.442061] CPU 1: hi: 0, btch: 1 usd: 0
[504638.442067] active_anon:103684 inactive_anon:103733 isolated_anon:0
active_file:10897 inactive_file:15059 isolated_file:0
unevictable:0 dirty:1 writeback:0 unstable:0
free:1164 slab_reclaimable:2356 slab_unreclaimable:3421
mapped:4413 shmem:200 pagetables:2699 bounce:0
free_cma:0 totalram:249264 balloontarget:315406
[504638.442069] Node 0 DMA free:1964kB min:88kB low:108kB high:132kB
active_anon:4664kB inactive_anon:4736kB active_file:628kB
inactive_file:1420kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:15084kB mlocked:0kB dirty:0kB
writeback:0kB mapped:228kB shmem:0kB slab_reclaimable:184kB
slab_unreclaimable:324kB kernel_stack:48kB pagetables:256kB unstable:0kB
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:21824
all_unreclaimable? yes
[504638.442078] lowmem_reserve[]: 0 469 469 469
[504638.442081] Node 0 DMA32 free:2692kB min:2728kB low:3408kB
high:4092kB active_anon:175172kB inactive_anon:175184kB
active_file:21244kB inactive_file:35340kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:507904kB managed:458288kB
mlocked:0kB dirty:0kB writeback:0kB mapped:7764kB shmem:676kB
slab_reclaimable:6628kB slab_unreclaimable:11496kB kernel_stack:1720kB
pagetables:8444kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:613279 all_unreclaimable? yes
[504638.442088] lowmem_reserve[]: 0 0 0 0
[504638.442091] Node 0 Normal free:0kB min:0kB low:0kB high:0kB
active_anon:234900kB inactive_anon:235012kB active_file:21716kB
inactive_file:23476kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:524288kB managed:523684kB mlocked:0kB
dirty:4kB writeback:0kB mapped:9660kB shmem:124kB
slab_reclaimable:2612kB slab_unreclaimable:1864kB kernel_stack:136kB
pagetables:2096kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:773613 all_unreclaimable? yes
[504638.442098] lowmem_reserve[]: 0 0 0 0
[504638.442101] Node 0 DMA: 1*4kB (R) 3*8kB (R) 1*16kB (R) 0*32kB 0*64kB
1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1964kB
[504638.442114] Node 0 DMA32: 673*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2692kB
[504638.442123] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[504638.442131] 22294 total pagecache pages
[504638.442133] 11197 pages in swap cache
[504638.442135] Swap cache stats: add 3449125, delete 3437928, find
590699/956067
[504638.442136] Free swap = 1868108kB
[504638.442137] Total swap = 2097148kB
[504638.452335] 262143 pages RAM
[504638.452336] 6697 pages reserved
[504638.452337] 558286 pages shared
[504638.452338] 239987 pages non-shared
[504638.452340] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
<snip process list>
[504638.452596] Out of memory: Kill process 21506 (cc1plus) score 123 or
sacrifice child
[504638.452598] Killed process 21506 (cc1plus) total-vm:543168kB,
anon-rss:350300kB, file-rss:9520kB
[504659.367289] frontswap selfshrink 18428 pages
[504689.415694] frontswap selfshrink 479 pages
[504719.462401] frontswap selfshrink 456 pages
[504749.506876] frontswap selfshrink 434 pages
[504779.558204] frontswap selfshrink 406 pages
[504809.604425] frontswap selfshrink 386 pages
[504839.654849] frontswap selfshrink 367 pages
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2013-12-26 8:42 ` James Dingwall
@ 2014-01-02 6:25 ` Bob Liu
2014-01-07 9:21 ` James Dingwall
0 siblings, 1 reply; 42+ messages in thread
From: Bob Liu @ 2014-01-02 6:25 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 12/26/2013 04:42 PM, James Dingwall wrote:
> Bob Liu wrote:
>> On 12/20/2013 03:08 AM, James Dingwall wrote:
>>> Bob Liu wrote:
>>>> On 12/12/2013 12:30 AM, James Dingwall wrote:
>>>>> Bob Liu wrote:
>>>>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>>>>> increase/decrease their memory allocation according to their
>>>>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>>>>
>>>>>>>>>> # free
>>>>>>>>>> total used free shared buffers
>>>>>>>>>> cached
>>>>>>>>>> Mem: 272080 248108 23972 0 1448
>>>>>>>>>> 63064
>>>>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>>>>> Swap: 2097148 8 2097140
>>>>>>>>>>
>>>>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>>>>> balloon to the maximum size:
>>>>>>>>>> # xl info | grep free_mem
>>>>>>>>>> free_memory : 14923
>>>>>>>>>>
>>>>>>>>>> An example trace (they are always the same) from the oom
>>>>>>>>>> killer in
>>>>>>>>>> 3.12 is added below. So far I have not been able to reproduce
>>>>>>>>>> this
>>>>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>>>>> particular change introduced this. However it does seem that the
>>>>>>>>>> behaviour is wrong because a) ballooning could give the guest
>>>>>>>>>> more
>>>>>>>>>> memory, b) there is lots of swap available which could be used
>>>>>>>>>> as a
>>>>>>>>>> fallback.
>>>>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>>>>> sounds odd -but basically pages that are destined for swap end up
>>>>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>>>>
>>>>>>>>>> If other information could help or there are more tests that I
>>>>>>>>>> could
>>>>>>>>>> run then please let me know.
>>>>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>>>>> the guest right?
>>>>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>>>>> Excellent. The odd thing is that your swap is not used that much,
>>>>>>> but
>>>>>>> it should be (as that is part of what the self-balloon is suppose to
>>>>>>> do).
>>>>>>>
>>>>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>>>>> to account for the slab - would this be relevant to this problem?
>>>>>>>
>>>>>> Perhaps, I have attached the patch.
>>>>>> James, could you please apply it and try your application again? You
>>>>>> have to rebuild the guest kernel.
>>>>>> Oh, and also take a look at whether frontswap is in use, you can
>>>>>> check
>>>>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>>>>> I have tested this patch with a workload where I have previously seen
>>>>> failures and so far so good. I'll try to keep a guest with it
>>>>> stressed
>>>>> to see if I do get any problems. I don't know if it is expected but I
>>>> By the way, besides longer time of kswapd, is this patch work well
>>>> during your stress testing?
>>>>
>>>> Have you seen the OOM killer triggered quite frequently again?(with
>>>> selfshrink=true)
>>>>
>>>> Thanks,
>>>> -Bob
>>> It was looking good until today (selfshrink=true). The trace below is
>>> during a compile of subversion, it looks like the memory has ballooned
>>> to almost the maximum permissible but even under pressure the swap disk
>>> has hardly come in to use.
>>>
>> So if without selfshrink the swap disk can be used a lot?
>>
>> If that's the case, I'm afraid the frontswap-selfshrink in
>> xen-selfballoon did something incorrect.
>>
>> Could you please try this patch which make the frontswap-selfshrink
>> slower and add a printk for debug.
>> Please still keep selfshrink=true in your test but can with or without
>> my previous patch.
>> Thanks a lot!
>>
> The oom trace below was triggered during a compile of gcc. I have the
> full dmesg from boot which shows all the printks, please let me know if
> you would like to see that.
>
Sorry for the later response.
Could you confirm that this problem doesn't exist if loading tmem with
selfshrinking=0 during compile gcc? It seems that you are compiling
difference packages during your testing.
This will help to figure out whether selfshrinking is the root cause.
Thanks,
-Bob
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-02 6:25 ` Bob Liu
@ 2014-01-07 9:21 ` James Dingwall
2014-01-09 10:48 ` Bob Liu
0 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2014-01-07 9:21 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 12/26/2013 04:42 PM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 12/20/2013 03:08 AM, James Dingwall wrote:
>>>> Bob Liu wrote:
>>>>> On 12/12/2013 12:30 AM, James Dingwall wrote:
>>>>>> Bob Liu wrote:
>>>>>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>>>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>>>>>> increase/decrease their memory allocation according to their
>>>>>>>>>>> requirements. One example domain I have has a maximum memory
>>>>>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>>>>>
>>>>>>>>>>> # free
>>>>>>>>>>> total used free shared buffers
>>>>>>>>>>> cached
>>>>>>>>>>> Mem: 272080 248108 23972 0 1448
>>>>>>>>>>> 63064
>>>>>>>>>>> -/+ buffers/cache: 183596 88484
>>>>>>>>>>> Swap: 2097148 8 2097140
>>>>>>>>>>>
>>>>>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>>>>>> balloon to the maximum size:
>>>>>>>>>>> # xl info | grep free_mem
>>>>>>>>>>> free_memory : 14923
>>>>>>>>>>>
>>>>>>>>>>> An example trace (they are always the same) from the oom
>>>>>>>>>>> killer in
>>>>>>>>>>> 3.12 is added below. So far I have not been able to reproduce
>>>>>>>>>>> this
>>>>>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>>>>>> particular change introduced this. However it does seem that the
>>>>>>>>>>> behaviour is wrong because a) ballooning could give the guest
>>>>>>>>>>> more
>>>>>>>>>>> memory, b) there is lots of swap available which could be used
>>>>>>>>>>> as a
>>>>>>>>>>> fallback.
>>>>>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>>>>>> sounds odd -but basically pages that are destined for swap end up
>>>>>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>>>>>
>>>>>>>>>>> If other information could help or there are more tests that I
>>>>>>>>>>> could
>>>>>>>>>>> run then please let me know.
>>>>>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>>>>>> the guest right?
>>>>>>>>> Yes, domU and dom0 both have the tmem module loaded and tmem
>>>>>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>>>>>> Excellent. The odd thing is that your swap is not used that much,
>>>>>>>> but
>>>>>>>> it should be (as that is part of what the self-balloon is suppose to
>>>>>>>> do).
>>>>>>>>
>>>>>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>>>>>> to account for the slab - would this be relevant to this problem?
>>>>>>>>
>>>>>>> Perhaps, I have attached the patch.
>>>>>>> James, could you please apply it and try your application again? You
>>>>>>> have to rebuild the guest kernel.
>>>>>>> Oh, and also take a look at whether frontswap is in use, you can
>>>>>>> check
>>>>>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>>>>>> I have tested this patch with a workload where I have previously seen
>>>>>> failures and so far so good. I'll try to keep a guest with it
>>>>>> stressed
>>>>>> to see if I do get any problems. I don't know if it is expected but I
>>>>> By the way, besides longer time of kswapd, is this patch work well
>>>>> during your stress testing?
>>>>>
>>>>> Have you seen the OOM killer triggered quite frequently again?(with
>>>>> selfshrink=true)
>>>>>
>>>>> Thanks,
>>>>> -Bob
>>>> It was looking good until today (selfshrink=true). The trace below is
>>>> during a compile of subversion, it looks like the memory has ballooned
>>>> to almost the maximum permissible but even under pressure the swap disk
>>>> has hardly come in to use.
>>>>
>>> So if without selfshrink the swap disk can be used a lot?
>>>
>>> If that's the case, I'm afraid the frontswap-selfshrink in
>>> xen-selfballoon did something incorrect.
>>>
>>> Could you please try this patch which make the frontswap-selfshrink
>>> slower and add a printk for debug.
>>> Please still keep selfshrink=true in your test but can with or without
>>> my previous patch.
>>> Thanks a lot!
>>>
>> The oom trace below was triggered during a compile of gcc. I have the
>> full dmesg from boot which shows all the printks, please let me know if
>> you would like to see that.
>>
> Sorry for the later response.
> Could you confirm that this problem doesn't exist if loading tmem with
> selfshrinking=0 during compile gcc? It seems that you are compiling
> difference packages during your testing.
> This will help to figure out whether selfshrinking is the root cause.
Got an oom with selfshrinking=0, again during a gcc compile.
Unfortunately I don't have a single test case which demonstrates the
problem but as I mentioned before it will generally show up under
compiles of large packages such as glibc, kdelibs, gcc etc.
I don't know if this is a separate or related issue but over the
holidays I also had a problem with six of the guests on my system where
kswapd was running at 100% and had clocked up >9000 minutes of cpu time
even though there was otherwise no load on them. Of the guests I
restarted yesterday in this state two have already got in to the same
state again, they are running a kernel with the first patch that you sent.
/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking N
James
[ 8212.940520] cc1plus invoked oom-killer: gfp_mask=0x200da, order=0,
oom_score_adj=0
[ 8212.940529] CPU: 1 PID: 23678 Comm: cc1plus Tainted: G W 3.12.5 #88
[ 8212.940532] ffff88001e38cdf8 ffff88000094f968 ffffffff8148f200
ffff88001f90e8e8
[ 8212.940536] ffff88001e38c8c0 ffff88000094fa08 ffffffff8148ccf7
ffff88000094f9b8
[ 8212.940538] ffffffff810f8d97 ffff88000094f998 ffffffff81006dc8
ffff88000094f9a8
[ 8212.940542] Call Trace:
[ 8212.940554] [<ffffffff8148f200>] dump_stack+0x46/0x58
[ 8212.940558] [<ffffffff8148ccf7>] dump_header.isra.9+0x6d/0x1cc
[ 8212.940564] [<ffffffff810f8d97>] ? super_cache_count+0xa8/0xb8
[ 8212.940569] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[ 8212.940573] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
[ 8212.940578] [<ffffffff81494abe>] ? _raw_spin_unlock_irqrestore+0x47/0x62
[ 8212.940583] [<ffffffff81296b27>] ? ___ratelimit+0xcb/0xe8
[ 8212.940588] [<ffffffff810b2bbf>] oom_kill_process+0x70/0x2fd
[ 8212.940592] [<ffffffff810bca0e>] ? zone_reclaimable+0x11/0x1e
[ 8212.940597] [<ffffffff81048779>] ? has_ns_capability_noaudit+0x12/0x19
[ 8212.940600] [<ffffffff81048792>] ? has_capability_noaudit+0x12/0x14
[ 8212.940603] [<ffffffff810b32de>] out_of_memory+0x31b/0x34e
[ 8212.940608] [<ffffffff810b7438>] __alloc_pages_nodemask+0x65b/0x792
[ 8212.940612] [<ffffffff810e3da3>] alloc_pages_vma+0xd0/0x10c
[ 8212.940617] [<ffffffff810dd5a4>] read_swap_cache_async+0x70/0x120
[ 8212.940620] [<ffffffff810dd6e4>] swapin_readahead+0x90/0xd4
[ 8212.940623] [<ffffffff81005b35>] ? pte_mfn_to_pfn+0x59/0xcb
[ 8212.940627] [<ffffffff810cf99d>] handle_mm_fault+0x8a4/0xd54
[ 8212.940630] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[ 8212.940634] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[ 8212.940638] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
[ 8212.940641] [<ffffffff8106823b>] ? arch_vtime_task_switch+0x81/0x86
[ 8212.940646] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
[ 8212.940649] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[ 8212.940652] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[ 8212.940654] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
[ 8212.940658] [<ffffffff810a45cc>] ? __acct_update_integrals+0xb4/0xbf
[ 8212.940661] [<ffffffff810a493f>] ? acct_account_cputime+0x17/0x19
[ 8212.940663] [<ffffffff81067c28>] ? account_user_time+0x67/0x92
[ 8212.940666] [<ffffffff8106811b>] ? vtime_account_user+0x4d/0x52
[ 8212.940669] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
[ 8212.940674] [<ffffffff810a065f>] ? rcu_user_enter+0xe/0x10
[ 8212.940677] [<ffffffff81495158>] page_fault+0x28/0x30
[ 8212.940679] Mem-Info:
[ 8212.940681] Node 0 DMA per-cpu:
[ 8212.940684] CPU 0: hi: 0, btch: 1 usd: 0
[ 8212.940685] CPU 1: hi: 0, btch: 1 usd: 0
[ 8212.940686] Node 0 DMA32 per-cpu:
[ 8212.940688] CPU 0: hi: 186, btch: 31 usd: 116
[ 8212.940690] CPU 1: hi: 186, btch: 31 usd: 124
[ 8212.940691] Node 0 Normal per-cpu:
[ 8212.940693] CPU 0: hi: 0, btch: 1 usd: 0
[ 8212.940694] CPU 1: hi: 0, btch: 1 usd: 0
[ 8212.940700] active_anon:105765 inactive_anon:105882 isolated_anon:0
active_file:8412 inactive_file:8612 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
free:1143 slab_reclaimable:3575 slab_unreclaimable:3464
mapped:3792 shmem:6 pagetables:2534 bounce:0
free_cma:0 totalram:246132 balloontarget:306242
[ 8212.940702] Node 0 DMA free:1964kB min:88kB low:108kB high:132kB
active_anon:5092kB inactive_anon:5328kB active_file:416kB
inactive_file:608kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:15392kB mlocked:0kB dirty:0kB
writeback:0kB mapped:320kB shmem:0kB slab_reclaimable:252kB
slab_unreclaimable:492kB kernel_stack:120kB pagetables:252kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:26951 all_unreclaimable? yes
[ 8212.940711] lowmem_reserve[]: 0 469 469 469
[ 8212.940715] Node 0 DMA32 free:2608kB min:2728kB low:3408kB
high:4092kB active_anon:181456kB inactive_anon:181528kB
active_file:22296kB inactive_file:22644kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:507904kB managed:466364kB
mlocked:0kB dirty:0kB writeback:0kB mapped:8628kB shmem:20kB
slab_reclaimable:10756kB slab_unreclaimable:12548kB kernel_stack:1688kB
pagetables:8876kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:612393 all_unreclaimable? yes
[ 8212.940722] lowmem_reserve[]: 0 0 0 0
[ 8212.940725] Node 0 Normal free:0kB min:0kB low:0kB high:0kB
active_anon:236512kB inactive_anon:236672kB active_file:10936kB
inactive_file:11196kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:524288kB managed:502772kB mlocked:0kB
dirty:0kB writeback:0kB mapped:6220kB shmem:4kB slab_reclaimable:3292kB
slab_unreclaimable:816kB kernel_stack:64kB pagetables:1008kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:745963 all_unreclaimable? yes
[ 8212.940732] lowmem_reserve[]: 0 0 0 0
[ 8212.940735] Node 0 DMA: 1*4kB (R) 0*8kB 4*16kB (R) 1*32kB (R) 1*64kB
(R) 2*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1956kB
[ 8212.940747] Node 0 DMA32: 652*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2608kB
[ 8212.940756] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[ 8212.940765] 16847 total pagecache pages
[ 8212.940766] 8381 pages in swap cache
[ 8212.940768] Swap cache stats: add 741397, delete 733016, find
250268/342284
[ 8212.940769] Free swap = 1925576kB
[ 8212.940770] Total swap = 2097148kB
[ 8212.951044] 262143 pages RAM
[ 8212.951046] 11939 pages reserved
[ 8212.951047] 540820 pages shared
[ 8212.951048] 240248 pages non-shared
[ 8212.951050] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
<snip process list>
[ 8212.951310] Out of memory: Kill process 23721 (cc1plus) score 119 or
sacrifice child
[ 8212.951313] Killed process 23721 (cc1plus) total-vm:530268kB,
anon-rss:350980kB, file-rss:9408kB
[54810.683658] kjournald starting. Commit interval 5 seconds
[54810.684381] EXT3-fs (xvda1): using internal journal
[54810.684402] EXT3-fs (xvda1): mounted filesystem with writeback data mode
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-07 9:21 ` James Dingwall
@ 2014-01-09 10:48 ` Bob Liu
2014-01-09 10:54 ` James Dingwall
` (2 more replies)
0 siblings, 3 replies; 42+ messages in thread
From: Bob Liu @ 2014-01-09 10:48 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 01/07/2014 05:21 PM, James Dingwall wrote:
> Bob Liu wrote:
>> Could you confirm that this problem doesn't exist if loading tmem with
>> selfshrinking=0 during compile gcc? It seems that you are compiling
>> difference packages during your testing.
>> This will help to figure out whether selfshrinking is the root cause.
> Got an oom with selfshrinking=0, again during a gcc compile.
> Unfortunately I don't have a single test case which demonstrates the
> problem but as I mentioned before it will generally show up under
> compiles of large packages such as glibc, kdelibs, gcc etc.
>
So the root cause is not because enabled selfshrinking.
Then what I can think of is that the xen-selfballoon driver was too
aggressive, too many pages were ballooned out which causeed heavy memory
pressure to guest OS.
And kswapd started to reclaim page until most of pages were
unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
triggered.
In theory the balloon driver should give back ballooned out pages to
guest OS, but I'm afraid this procedure is not fast enough.
My suggestion is reserve a min memory for your guest OS so that the
xen-selfballoon won't be so aggressive.
You can do it through parameters selfballoon_reserved_mb or
selfballoon_min_usable_mb.
> I don't know if this is a separate or related issue but over the
> holidays I also had a problem with six of the guests on my system where
> kswapd was running at 100% and had clocked up >9000 minutes of cpu time
> even though there was otherwise no load on them. Of the guests I
> restarted yesterday in this state two have already got in to the same
> state again, they are running a kernel with the first patch that you sent.
>
Could you get the meminfo in guest OS at that time?
cat /proc/meminfo
cat /proc/vmstat
Thanks,
-Bob
> /sys/module/tmem/parameters/cleancache Y
> /sys/module/tmem/parameters/frontswap Y
> /sys/module/tmem/parameters/selfballooning Y
> /sys/module/tmem/parameters/selfshrinking N
>
> James
>
> [ 8212.940520] cc1plus invoked oom-killer: gfp_mask=0x200da, order=0,
> oom_score_adj=0
> [ 8212.940529] CPU: 1 PID: 23678 Comm: cc1plus Tainted: G W 3.12.5 #88
> [ 8212.940532] ffff88001e38cdf8 ffff88000094f968 ffffffff8148f200
> ffff88001f90e8e8
> [ 8212.940536] ffff88001e38c8c0 ffff88000094fa08 ffffffff8148ccf7
> ffff88000094f9b8
> [ 8212.940538] ffffffff810f8d97 ffff88000094f998 ffffffff81006dc8
> ffff88000094f9a8
> [ 8212.940542] Call Trace:
> [ 8212.940554] [<ffffffff8148f200>] dump_stack+0x46/0x58
> [ 8212.940558] [<ffffffff8148ccf7>] dump_header.isra.9+0x6d/0x1cc
> [ 8212.940564] [<ffffffff810f8d97>] ? super_cache_count+0xa8/0xb8
> [ 8212.940569] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
> [ 8212.940573] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
> [ 8212.940578] [<ffffffff81494abe>] ?
> _raw_spin_unlock_irqrestore+0x47/0x62
> [ 8212.940583] [<ffffffff81296b27>] ? ___ratelimit+0xcb/0xe8
> [ 8212.940588] [<ffffffff810b2bbf>] oom_kill_process+0x70/0x2fd
> [ 8212.940592] [<ffffffff810bca0e>] ? zone_reclaimable+0x11/0x1e
> [ 8212.940597] [<ffffffff81048779>] ? has_ns_capability_noaudit+0x12/0x19
> [ 8212.940600] [<ffffffff81048792>] ? has_capability_noaudit+0x12/0x14
> [ 8212.940603] [<ffffffff810b32de>] out_of_memory+0x31b/0x34e
> [ 8212.940608] [<ffffffff810b7438>] __alloc_pages_nodemask+0x65b/0x792
> [ 8212.940612] [<ffffffff810e3da3>] alloc_pages_vma+0xd0/0x10c
> [ 8212.940617] [<ffffffff810dd5a4>] read_swap_cache_async+0x70/0x120
> [ 8212.940620] [<ffffffff810dd6e4>] swapin_readahead+0x90/0xd4
> [ 8212.940623] [<ffffffff81005b35>] ? pte_mfn_to_pfn+0x59/0xcb
> [ 8212.940627] [<ffffffff810cf99d>] handle_mm_fault+0x8a4/0xd54
> [ 8212.940630] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
> [ 8212.940634] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
> [ 8212.940638] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
> [ 8212.940641] [<ffffffff8106823b>] ? arch_vtime_task_switch+0x81/0x86
> [ 8212.940646] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
> [ 8212.940649] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
> [ 8212.940652] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
> [ 8212.940654] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
> [ 8212.940658] [<ffffffff810a45cc>] ? __acct_update_integrals+0xb4/0xbf
> [ 8212.940661] [<ffffffff810a493f>] ? acct_account_cputime+0x17/0x19
> [ 8212.940663] [<ffffffff81067c28>] ? account_user_time+0x67/0x92
> [ 8212.940666] [<ffffffff8106811b>] ? vtime_account_user+0x4d/0x52
> [ 8212.940669] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
> [ 8212.940674] [<ffffffff810a065f>] ? rcu_user_enter+0xe/0x10
> [ 8212.940677] [<ffffffff81495158>] page_fault+0x28/0x30
> [ 8212.940679] Mem-Info:
> [ 8212.940681] Node 0 DMA per-cpu:
> [ 8212.940684] CPU 0: hi: 0, btch: 1 usd: 0
> [ 8212.940685] CPU 1: hi: 0, btch: 1 usd: 0
> [ 8212.940686] Node 0 DMA32 per-cpu:
> [ 8212.940688] CPU 0: hi: 186, btch: 31 usd: 116
> [ 8212.940690] CPU 1: hi: 186, btch: 31 usd: 124
> [ 8212.940691] Node 0 Normal per-cpu:
> [ 8212.940693] CPU 0: hi: 0, btch: 1 usd: 0
> [ 8212.940694] CPU 1: hi: 0, btch: 1 usd: 0
> [ 8212.940700] active_anon:105765 inactive_anon:105882 isolated_anon:0
> active_file:8412 inactive_file:8612 isolated_file:0
> unevictable:0 dirty:0 writeback:0 unstable:0
> free:1143 slab_reclaimable:3575 slab_unreclaimable:3464
> mapped:3792 shmem:6 pagetables:2534 bounce:0
> free_cma:0 totalram:246132 balloontarget:306242
> [ 8212.940702] Node 0 DMA free:1964kB min:88kB low:108kB high:132kB
> active_anon:5092kB inactive_anon:5328kB active_file:416kB
> inactive_file:608kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15996kB managed:15392kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:320kB shmem:0kB slab_reclaimable:252kB
> slab_unreclaimable:492kB kernel_stack:120kB pagetables:252kB
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> pages_scanned:26951 all_unreclaimable? yes
> [ 8212.940711] lowmem_reserve[]: 0 469 469 469
> [ 8212.940715] Node 0 DMA32 free:2608kB min:2728kB low:3408kB
> high:4092kB active_anon:181456kB inactive_anon:181528kB
> active_file:22296kB inactive_file:22644kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:507904kB managed:466364kB
> mlocked:0kB dirty:0kB writeback:0kB mapped:8628kB shmem:20kB
> slab_reclaimable:10756kB slab_unreclaimable:12548kB kernel_stack:1688kB
> pagetables:8876kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> pages_scanned:612393 all_unreclaimable? yes
> [ 8212.940722] lowmem_reserve[]: 0 0 0 0
> [ 8212.940725] Node 0 Normal free:0kB min:0kB low:0kB high:0kB
> active_anon:236512kB inactive_anon:236672kB active_file:10936kB
> inactive_file:11196kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:524288kB managed:502772kB mlocked:0kB
> dirty:0kB writeback:0kB mapped:6220kB shmem:4kB slab_reclaimable:3292kB
> slab_unreclaimable:816kB kernel_stack:64kB pagetables:1008kB
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> pages_scanned:745963 all_unreclaimable? yes
> [ 8212.940732] lowmem_reserve[]: 0 0 0 0
> [ 8212.940735] Node 0 DMA: 1*4kB (R) 0*8kB 4*16kB (R) 1*32kB (R) 1*64kB
> (R) 2*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1956kB
> [ 8212.940747] Node 0 DMA32: 652*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2608kB
> [ 8212.940756] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
> 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> [ 8212.940765] 16847 total pagecache pages
> [ 8212.940766] 8381 pages in swap cache
> [ 8212.940768] Swap cache stats: add 741397, delete 733016, find
> 250268/342284
> [ 8212.940769] Free swap = 1925576kB
> [ 8212.940770] Total swap = 2097148kB
> [ 8212.951044] 262143 pages RAM
> [ 8212.951046] 11939 pages reserved
> [ 8212.951047] 540820 pages shared
> [ 8212.951048] 240248 pages non-shared
> [ 8212.951050] [ pid ] uid tgid total_vm rss nr_ptes swapents
> oom_score_adj name
> <snip process list>
> [ 8212.951310] Out of memory: Kill process 23721 (cc1plus) score 119 or
> sacrifice child
> [ 8212.951313] Killed process 23721 (cc1plus) total-vm:530268kB,
> anon-rss:350980kB, file-rss:9408kB
> [54810.683658] kjournald starting. Commit interval 5 seconds
> [54810.684381] EXT3-fs (xvda1): using internal journal
> [54810.684402] EXT3-fs (xvda1): mounted filesystem with writeback data mode
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-09 10:48 ` Bob Liu
@ 2014-01-09 10:54 ` James Dingwall
2014-01-09 11:04 ` James Dingwall
2014-01-15 8:49 ` James Dingwall
2 siblings, 0 replies; 42+ messages in thread
From: James Dingwall @ 2014-01-09 10:54 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 01/07/2014 05:21 PM, James Dingwall wrote:
>> Bob Liu wrote:
>>> Could you confirm that this problem doesn't exist if loading tmem with
>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>> difference packages during your testing.
>>> This will help to figure out whether selfshrinking is the root cause.
>> Got an oom with selfshrinking=0, again during a gcc compile.
>> Unfortunately I don't have a single test case which demonstrates the
>> problem but as I mentioned before it will generally show up under
>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>
> So the root cause is not because enabled selfshrinking.
> Then what I can think of is that the xen-selfballoon driver was too
> aggressive, too many pages were ballooned out which causeed heavy memory
> pressure to guest OS.
> And kswapd started to reclaim page until most of pages were
> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
> triggered.
> In theory the balloon driver should give back ballooned out pages to
> guest OS, but I'm afraid this procedure is not fast enough.
>
> My suggestion is reserve a min memory for your guest OS so that the
> xen-selfballoon won't be so aggressive.
> You can do it through parameters selfballoon_reserved_mb or
> selfballoon_min_usable_mb.
I will try your suggestions and let you know.
>
>> I don't know if this is a separate or related issue but over the
>> holidays I also had a problem with six of the guests on my system where
>> kswapd was running at 100% and had clocked up >9000 minutes of cpu time
>> even though there was otherwise no load on them. Of the guests I
>> restarted yesterday in this state two have already got in to the same
>> state again, they are running a kernel with the first patch that you sent.
>>
> Could you get the meminfo in guest OS at that time?
> cat /proc/meminfo
MemTotal: 364080 kB
MemFree: 130448 kB
Buffers: 1260 kB
Cached: 129352 kB
SwapCached: 300 kB
Active: 21412 kB
Inactive: 160888 kB
Active(anon): 7732 kB
Inactive(anon): 44676 kB
Active(file): 13680 kB
Inactive(file): 116212 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2097148 kB
SwapFree: 2096704 kB
Dirty: 44 kB
Writeback: 0 kB
AnonPages: 51532 kB
Mapped: 14172 kB
Shmem: 720 kB
Slab: 19580 kB
SReclaimable: 7732 kB
SUnreclaim: 11848 kB
KernelStack: 1824 kB
PageTables: 7968 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2279188 kB
Committed_AS: 338792 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 9020 kB
VmallocChunk: 34359716472 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
DirectMap4k: 1048576 kB
DirectMap2M: 0 kB
> cat /proc/vmstat
nr_free_pages 32775
nr_alloc_batch 0
nr_inactive_anon 11167
nr_active_anon 1904
nr_inactive_file 29054
nr_active_file 3420
nr_unevictable 0
nr_mlock 0
nr_anon_pages 12869
nr_mapped 3543
nr_file_pages 32724
nr_dirty 5
nr_writeback 0
nr_slab_reclaimable 1933
nr_slab_unreclaimable 2959
nr_page_table_pages 1988
nr_kernel_stack 228
nr_unstable 0
nr_bounce 0
nr_vmscan_write 781197
nr_vmscan_immediate_reclaim 6241
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 180
nr_dirtied 86426
nr_written 860157
numa_hit 8323637
numa_miss 0
numa_foreign 0
numa_interleave 0
numa_local 8323637
numa_other 0
nr_anon_transparent_hugepages 0
nr_free_cma 0
nr_dirty_threshold 15359
nr_dirty_background_threshold 7679
pgpgin 2044246
pgpgout 643646
pswpin 123
pswpout 153
pgalloc_dma 164528
pgalloc_dma32 7332263
pgalloc_normal 1018515
pgalloc_movable 0
pgfree 8548450
pgactivate 2011347
pgdeactivate 2274842
pgfault 7231978
pgmajfault 345038
pgrefill_dma 55260
pgrefill_dma32 2261099
pgrefill_normal 1771
pgrefill_movable 0
pgsteal_kswapd_dma 44877
pgsteal_kswapd_dma32 2586249
pgsteal_kswapd_normal 0
pgsteal_kswapd_movable 0
pgsteal_direct_dma 0
pgsteal_direct_dma32 37
pgsteal_direct_normal 0
pgsteal_direct_movable 0
pgscan_kswapd_dma 204746
pgscan_kswapd_dma32 4474736
pgscan_kswapd_normal 0
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 39
pgscan_direct_normal 0
pgscan_direct_movable 0
pgscan_direct_throttle 0
zone_reclaim_failed 0
pginodesteal 0
slabs_scanned 2713984
kswapd_inodesteal 41065
kswapd_low_wmark_hit_quickly 14894
kswapd_high_wmark_hit_quickly 115972041
pageoutrun 115992287
allocstall 1
pgrotated 8495
numa_pte_updates 0
numa_huge_pte_updates 0
numa_hint_faults 0
numa_hint_faults_local 0
numa_pages_migrated 0
pgmigrate_success 0
pgmigrate_fail 0
compact_migrate_scanned 0
compact_free_scanned 0
compact_isolated 0
compact_stall 0
compact_fail 0
compact_success 0
unevictable_pgs_culled 29364
unevictable_pgs_scanned 0
unevictable_pgs_rescued 29137
unevictable_pgs_mlocked 29542
unevictable_pgs_munlocked 29542
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
thp_fault_alloc 0
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_split 0
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
nr_tlb_remote_flush 10666
nr_tlb_remote_flush_received 21336
nr_tlb_local_flush_all 65481
nr_tlb_local_flush_one 1431260
>
> Thanks,
> -Bob
>
>> /sys/module/tmem/parameters/cleancache Y
>> /sys/module/tmem/parameters/frontswap Y
>> /sys/module/tmem/parameters/selfballooning Y
>> /sys/module/tmem/parameters/selfshrinking N
>>
>> James
>>
>> [ 8212.940520] cc1plus invoked oom-killer: gfp_mask=0x200da, order=0,
>> oom_score_adj=0
>> [ 8212.940529] CPU: 1 PID: 23678 Comm: cc1plus Tainted: G W 3.12.5 #88
>> [ 8212.940532] ffff88001e38cdf8 ffff88000094f968 ffffffff8148f200
>> ffff88001f90e8e8
>> [ 8212.940536] ffff88001e38c8c0 ffff88000094fa08 ffffffff8148ccf7
>> ffff88000094f9b8
>> [ 8212.940538] ffffffff810f8d97 ffff88000094f998 ffffffff81006dc8
>> ffff88000094f9a8
>> [ 8212.940542] Call Trace:
>> [ 8212.940554] [<ffffffff8148f200>] dump_stack+0x46/0x58
>> [ 8212.940558] [<ffffffff8148ccf7>] dump_header.isra.9+0x6d/0x1cc
>> [ 8212.940564] [<ffffffff810f8d97>] ? super_cache_count+0xa8/0xb8
>> [ 8212.940569] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
>> [ 8212.940573] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
>> [ 8212.940578] [<ffffffff81494abe>] ?
>> _raw_spin_unlock_irqrestore+0x47/0x62
>> [ 8212.940583] [<ffffffff81296b27>] ? ___ratelimit+0xcb/0xe8
>> [ 8212.940588] [<ffffffff810b2bbf>] oom_kill_process+0x70/0x2fd
>> [ 8212.940592] [<ffffffff810bca0e>] ? zone_reclaimable+0x11/0x1e
>> [ 8212.940597] [<ffffffff81048779>] ? has_ns_capability_noaudit+0x12/0x19
>> [ 8212.940600] [<ffffffff81048792>] ? has_capability_noaudit+0x12/0x14
>> [ 8212.940603] [<ffffffff810b32de>] out_of_memory+0x31b/0x34e
>> [ 8212.940608] [<ffffffff810b7438>] __alloc_pages_nodemask+0x65b/0x792
>> [ 8212.940612] [<ffffffff810e3da3>] alloc_pages_vma+0xd0/0x10c
>> [ 8212.940617] [<ffffffff810dd5a4>] read_swap_cache_async+0x70/0x120
>> [ 8212.940620] [<ffffffff810dd6e4>] swapin_readahead+0x90/0xd4
>> [ 8212.940623] [<ffffffff81005b35>] ? pte_mfn_to_pfn+0x59/0xcb
>> [ 8212.940627] [<ffffffff810cf99d>] handle_mm_fault+0x8a4/0xd54
>> [ 8212.940630] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
>> [ 8212.940634] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
>> [ 8212.940638] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
>> [ 8212.940641] [<ffffffff8106823b>] ? arch_vtime_task_switch+0x81/0x86
>> [ 8212.940646] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
>> [ 8212.940649] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
>> [ 8212.940652] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
>> [ 8212.940654] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
>> [ 8212.940658] [<ffffffff810a45cc>] ? __acct_update_integrals+0xb4/0xbf
>> [ 8212.940661] [<ffffffff810a493f>] ? acct_account_cputime+0x17/0x19
>> [ 8212.940663] [<ffffffff81067c28>] ? account_user_time+0x67/0x92
>> [ 8212.940666] [<ffffffff8106811b>] ? vtime_account_user+0x4d/0x52
>> [ 8212.940669] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
>> [ 8212.940674] [<ffffffff810a065f>] ? rcu_user_enter+0xe/0x10
>> [ 8212.940677] [<ffffffff81495158>] page_fault+0x28/0x30
>> [ 8212.940679] Mem-Info:
>> [ 8212.940681] Node 0 DMA per-cpu:
>> [ 8212.940684] CPU 0: hi: 0, btch: 1 usd: 0
>> [ 8212.940685] CPU 1: hi: 0, btch: 1 usd: 0
>> [ 8212.940686] Node 0 DMA32 per-cpu:
>> [ 8212.940688] CPU 0: hi: 186, btch: 31 usd: 116
>> [ 8212.940690] CPU 1: hi: 186, btch: 31 usd: 124
>> [ 8212.940691] Node 0 Normal per-cpu:
>> [ 8212.940693] CPU 0: hi: 0, btch: 1 usd: 0
>> [ 8212.940694] CPU 1: hi: 0, btch: 1 usd: 0
>> [ 8212.940700] active_anon:105765 inactive_anon:105882 isolated_anon:0
>> active_file:8412 inactive_file:8612 isolated_file:0
>> unevictable:0 dirty:0 writeback:0 unstable:0
>> free:1143 slab_reclaimable:3575 slab_unreclaimable:3464
>> mapped:3792 shmem:6 pagetables:2534 bounce:0
>> free_cma:0 totalram:246132 balloontarget:306242
>> [ 8212.940702] Node 0 DMA free:1964kB min:88kB low:108kB high:132kB
>> active_anon:5092kB inactive_anon:5328kB active_file:416kB
>> inactive_file:608kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:15996kB managed:15392kB mlocked:0kB dirty:0kB
>> writeback:0kB mapped:320kB shmem:0kB slab_reclaimable:252kB
>> slab_unreclaimable:492kB kernel_stack:120kB pagetables:252kB
>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:26951 all_unreclaimable? yes
>> [ 8212.940711] lowmem_reserve[]: 0 469 469 469
>> [ 8212.940715] Node 0 DMA32 free:2608kB min:2728kB low:3408kB
>> high:4092kB active_anon:181456kB inactive_anon:181528kB
>> active_file:22296kB inactive_file:22644kB unevictable:0kB
>> isolated(anon):0kB isolated(file):0kB present:507904kB managed:466364kB
>> mlocked:0kB dirty:0kB writeback:0kB mapped:8628kB shmem:20kB
>> slab_reclaimable:10756kB slab_unreclaimable:12548kB kernel_stack:1688kB
>> pagetables:8876kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:612393 all_unreclaimable? yes
>> [ 8212.940722] lowmem_reserve[]: 0 0 0 0
>> [ 8212.940725] Node 0 Normal free:0kB min:0kB low:0kB high:0kB
>> active_anon:236512kB inactive_anon:236672kB active_file:10936kB
>> inactive_file:11196kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:524288kB managed:502772kB mlocked:0kB
>> dirty:0kB writeback:0kB mapped:6220kB shmem:4kB slab_reclaimable:3292kB
>> slab_unreclaimable:816kB kernel_stack:64kB pagetables:1008kB
>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:745963 all_unreclaimable? yes
>> [ 8212.940732] lowmem_reserve[]: 0 0 0 0
>> [ 8212.940735] Node 0 DMA: 1*4kB (R) 0*8kB 4*16kB (R) 1*32kB (R) 1*64kB
>> (R) 2*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1956kB
>> [ 8212.940747] Node 0 DMA32: 652*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2608kB
>> [ 8212.940756] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
>> 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>> [ 8212.940765] 16847 total pagecache pages
>> [ 8212.940766] 8381 pages in swap cache
>> [ 8212.940768] Swap cache stats: add 741397, delete 733016, find
>> 250268/342284
>> [ 8212.940769] Free swap = 1925576kB
>> [ 8212.940770] Total swap = 2097148kB
>> [ 8212.951044] 262143 pages RAM
>> [ 8212.951046] 11939 pages reserved
>> [ 8212.951047] 540820 pages shared
>> [ 8212.951048] 240248 pages non-shared
>> [ 8212.951050] [ pid ] uid tgid total_vm rss nr_ptes swapents
>> oom_score_adj name
>> <snip process list>
>> [ 8212.951310] Out of memory: Kill process 23721 (cc1plus) score 119 or
>> sacrifice child
>> [ 8212.951313] Killed process 23721 (cc1plus) total-vm:530268kB,
>> anon-rss:350980kB, file-rss:9408kB
>> [54810.683658] kjournald starting. Commit interval 5 seconds
>> [54810.684381] EXT3-fs (xvda1): using internal journal
>> [54810.684402] EXT3-fs (xvda1): mounted filesystem with writeback data mode
>>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-09 10:48 ` Bob Liu
2014-01-09 10:54 ` James Dingwall
@ 2014-01-09 11:04 ` James Dingwall
2014-01-15 8:49 ` James Dingwall
2 siblings, 0 replies; 42+ messages in thread
From: James Dingwall @ 2014-01-09 11:04 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 01/07/2014 05:21 PM, James Dingwall wrote:
>> Bob Liu wrote:
>>> Could you confirm that this problem doesn't exist if loading tmem with
>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>> difference packages during your testing.
>>> This will help to figure out whether selfshrinking is the root cause.
>> Got an oom with selfshrinking=0, again during a gcc compile.
>> Unfortunately I don't have a single test case which demonstrates the
>> problem but as I mentioned before it will generally show up under
>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>
> So the root cause is not because enabled selfshrinking.
> Then what I can think of is that the xen-selfballoon driver was too
> aggressive, too many pages were ballooned out which causeed heavy memory
> pressure to guest OS.
> And kswapd started to reclaim page until most of pages were
> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
> triggered.
> In theory the balloon driver should give back ballooned out pages to
> guest OS, but I'm afraid this procedure is not fast enough.
>
> My suggestion is reserve a min memory for your guest OS so that the
> xen-selfballoon won't be so aggressive.
> You can do it through parameters selfballoon_reserved_mb or
> selfballoon_min_usable_mb.
>
>> I don't know if this is a separate or related issue but over the
>> holidays I also had a problem with six of the guests on my system where
>> kswapd was running at 100% and had clocked up >9000 minutes of cpu time
>> even though there was otherwise no load on them. Of the guests I
>> restarted yesterday in this state two have already got in to the same
>> state again, they are running a kernel with the first patch that you sent.
As soon as I echo 32 both (originally 0)
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_reserved_mb
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_min_usable_mb
Then the kswapd process stopped running at 100%. Unfortunately I didn't
check between the two commands to see if one by itself made a difference
but I'll look for that next time.
> Could you get the meminfo in guest OS at that time?
After
> cat /proc/meminfo
MemTotal: 397028 kB
MemFree: 163756 kB
Buffers: 1260 kB
Cached: 129284 kB
SwapCached: 132 kB
Active: 22664 kB
Inactive: 159576 kB
Active(anon): 8004 kB
Inactive(anon): 44412 kB
Active(file): 14660 kB
Inactive(file): 115164 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2097148 kB
SwapFree: 2096896 kB
Dirty: 20 kB
Writeback: 0 kB
AnonPages: 51640 kB
Mapped: 14136 kB
Shmem: 720 kB
Slab: 19492 kB
SReclaimable: 7692 kB
SUnreclaim: 11800 kB
KernelStack: 1816 kB
PageTables: 7928 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2295660 kB
Committed_AS: 338552 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 9020 kB
VmallocChunk: 34359716408 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
DirectMap4k: 1048576 kB
DirectMap2M: 0 kB
> cat /proc/vmstat
nr_free_pages 40916
nr_alloc_batch 0
nr_inactive_anon 11102
nr_active_anon 2009
nr_inactive_file 28791
nr_active_file 3665
nr_unevictable 0
nr_mlock 0
nr_anon_pages 12904
nr_mapped 3534
nr_file_pages 32669
nr_dirty 5
nr_writeback 0
nr_slab_reclaimable 1923
nr_slab_unreclaimable 2945
nr_page_table_pages 1982
nr_kernel_stack 227
nr_unstable 0
nr_bounce 0
nr_vmscan_write 781891
nr_vmscan_immediate_reclaim 6245
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 180
nr_dirtied 86609
nr_written 861010
numa_hit 8353372
numa_miss 0
numa_foreign 0
numa_interleave 0
numa_local 8353372
numa_other 0
nr_anon_transparent_hugepages 0
nr_free_cma 0
nr_dirty_threshold 16991
nr_dirty_background_threshold 8495
pgpgin 2044575
pgpgout 645866
pswpin 123
pswpout 153
pgalloc_dma 164944
pgalloc_dma32 7347917
pgalloc_normal 1032559
pgalloc_movable 0
pgfree 8586607
pgactivate 2012718
pgdeactivate 2276721
pgfault 7295414
pgmajfault 345301
pgrefill_dma 55271
pgrefill_dma32 2263007
pgrefill_normal 1771
pgrefill_movable 0
pgsteal_kswapd_dma 44880
pgsteal_kswapd_dma32 2587500
pgsteal_kswapd_normal 0
pgsteal_kswapd_movable 0
pgsteal_direct_dma 0
pgsteal_direct_dma32 37
pgsteal_direct_normal 0
pgsteal_direct_movable 0
pgscan_kswapd_dma 204749
pgscan_kswapd_dma32 4477230
pgscan_kswapd_normal 0
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 39
pgscan_direct_normal 0
pgscan_direct_movable 0
pgscan_direct_throttle 0
zone_reclaim_failed 0
pginodesteal 0
slabs_scanned 2720128
kswapd_inodesteal 41065
kswapd_low_wmark_hit_quickly 14897
kswapd_high_wmark_hit_quickly 116697740
pageoutrun 116717997
allocstall 1
pgrotated 8497
numa_pte_updates 0
numa_huge_pte_updates 0
numa_hint_faults 0
numa_hint_faults_local 0
numa_pages_migrated 0
pgmigrate_success 0
pgmigrate_fail 0
compact_migrate_scanned 0
compact_free_scanned 0
compact_isolated 0
compact_stall 0
compact_fail 0
compact_success 0
unevictable_pgs_culled 29365
unevictable_pgs_scanned 0
unevictable_pgs_rescued 29145
unevictable_pgs_mlocked 29550
unevictable_pgs_munlocked 29550
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
thp_fault_alloc 0
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_split 0
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
nr_tlb_remote_flush 10780
nr_tlb_remote_flush_received 21564
nr_tlb_local_flush_all 66247
nr_tlb_local_flush_one 1446496
>
> Thanks,
> -Bob
>
>> /sys/module/tmem/parameters/cleancache Y
>> /sys/module/tmem/parameters/frontswap Y
>> /sys/module/tmem/parameters/selfballooning Y
>> /sys/module/tmem/parameters/selfshrinking N
>>
>> James
>>
>> [ 8212.940520] cc1plus invoked oom-killer: gfp_mask=0x200da, order=0,
>> oom_score_adj=0
>> [ 8212.940529] CPU: 1 PID: 23678 Comm: cc1plus Tainted: G W 3.12.5 #88
>> [ 8212.940532] ffff88001e38cdf8 ffff88000094f968 ffffffff8148f200
>> ffff88001f90e8e8
>> [ 8212.940536] ffff88001e38c8c0 ffff88000094fa08 ffffffff8148ccf7
>> ffff88000094f9b8
>> [ 8212.940538] ffffffff810f8d97 ffff88000094f998 ffffffff81006dc8
>> ffff88000094f9a8
>> [ 8212.940542] Call Trace:
>> [ 8212.940554] [<ffffffff8148f200>] dump_stack+0x46/0x58
>> [ 8212.940558] [<ffffffff8148ccf7>] dump_header.isra.9+0x6d/0x1cc
>> [ 8212.940564] [<ffffffff810f8d97>] ? super_cache_count+0xa8/0xb8
>> [ 8212.940569] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
>> [ 8212.940573] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
>> [ 8212.940578] [<ffffffff81494abe>] ?
>> _raw_spin_unlock_irqrestore+0x47/0x62
>> [ 8212.940583] [<ffffffff81296b27>] ? ___ratelimit+0xcb/0xe8
>> [ 8212.940588] [<ffffffff810b2bbf>] oom_kill_process+0x70/0x2fd
>> [ 8212.940592] [<ffffffff810bca0e>] ? zone_reclaimable+0x11/0x1e
>> [ 8212.940597] [<ffffffff81048779>] ? has_ns_capability_noaudit+0x12/0x19
>> [ 8212.940600] [<ffffffff81048792>] ? has_capability_noaudit+0x12/0x14
>> [ 8212.940603] [<ffffffff810b32de>] out_of_memory+0x31b/0x34e
>> [ 8212.940608] [<ffffffff810b7438>] __alloc_pages_nodemask+0x65b/0x792
>> [ 8212.940612] [<ffffffff810e3da3>] alloc_pages_vma+0xd0/0x10c
>> [ 8212.940617] [<ffffffff810dd5a4>] read_swap_cache_async+0x70/0x120
>> [ 8212.940620] [<ffffffff810dd6e4>] swapin_readahead+0x90/0xd4
>> [ 8212.940623] [<ffffffff81005b35>] ? pte_mfn_to_pfn+0x59/0xcb
>> [ 8212.940627] [<ffffffff810cf99d>] handle_mm_fault+0x8a4/0xd54
>> [ 8212.940630] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
>> [ 8212.940634] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
>> [ 8212.940638] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
>> [ 8212.940641] [<ffffffff8106823b>] ? arch_vtime_task_switch+0x81/0x86
>> [ 8212.940646] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
>> [ 8212.940649] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
>> [ 8212.940652] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
>> [ 8212.940654] [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
>> [ 8212.940658] [<ffffffff810a45cc>] ? __acct_update_integrals+0xb4/0xbf
>> [ 8212.940661] [<ffffffff810a493f>] ? acct_account_cputime+0x17/0x19
>> [ 8212.940663] [<ffffffff81067c28>] ? account_user_time+0x67/0x92
>> [ 8212.940666] [<ffffffff8106811b>] ? vtime_account_user+0x4d/0x52
>> [ 8212.940669] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
>> [ 8212.940674] [<ffffffff810a065f>] ? rcu_user_enter+0xe/0x10
>> [ 8212.940677] [<ffffffff81495158>] page_fault+0x28/0x30
>> [ 8212.940679] Mem-Info:
>> [ 8212.940681] Node 0 DMA per-cpu:
>> [ 8212.940684] CPU 0: hi: 0, btch: 1 usd: 0
>> [ 8212.940685] CPU 1: hi: 0, btch: 1 usd: 0
>> [ 8212.940686] Node 0 DMA32 per-cpu:
>> [ 8212.940688] CPU 0: hi: 186, btch: 31 usd: 116
>> [ 8212.940690] CPU 1: hi: 186, btch: 31 usd: 124
>> [ 8212.940691] Node 0 Normal per-cpu:
>> [ 8212.940693] CPU 0: hi: 0, btch: 1 usd: 0
>> [ 8212.940694] CPU 1: hi: 0, btch: 1 usd: 0
>> [ 8212.940700] active_anon:105765 inactive_anon:105882 isolated_anon:0
>> active_file:8412 inactive_file:8612 isolated_file:0
>> unevictable:0 dirty:0 writeback:0 unstable:0
>> free:1143 slab_reclaimable:3575 slab_unreclaimable:3464
>> mapped:3792 shmem:6 pagetables:2534 bounce:0
>> free_cma:0 totalram:246132 balloontarget:306242
>> [ 8212.940702] Node 0 DMA free:1964kB min:88kB low:108kB high:132kB
>> active_anon:5092kB inactive_anon:5328kB active_file:416kB
>> inactive_file:608kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:15996kB managed:15392kB mlocked:0kB dirty:0kB
>> writeback:0kB mapped:320kB shmem:0kB slab_reclaimable:252kB
>> slab_unreclaimable:492kB kernel_stack:120kB pagetables:252kB
>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:26951 all_unreclaimable? yes
>> [ 8212.940711] lowmem_reserve[]: 0 469 469 469
>> [ 8212.940715] Node 0 DMA32 free:2608kB min:2728kB low:3408kB
>> high:4092kB active_anon:181456kB inactive_anon:181528kB
>> active_file:22296kB inactive_file:22644kB unevictable:0kB
>> isolated(anon):0kB isolated(file):0kB present:507904kB managed:466364kB
>> mlocked:0kB dirty:0kB writeback:0kB mapped:8628kB shmem:20kB
>> slab_reclaimable:10756kB slab_unreclaimable:12548kB kernel_stack:1688kB
>> pagetables:8876kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:612393 all_unreclaimable? yes
>> [ 8212.940722] lowmem_reserve[]: 0 0 0 0
>> [ 8212.940725] Node 0 Normal free:0kB min:0kB low:0kB high:0kB
>> active_anon:236512kB inactive_anon:236672kB active_file:10936kB
>> inactive_file:11196kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:524288kB managed:502772kB mlocked:0kB
>> dirty:0kB writeback:0kB mapped:6220kB shmem:4kB slab_reclaimable:3292kB
>> slab_unreclaimable:816kB kernel_stack:64kB pagetables:1008kB
>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:745963 all_unreclaimable? yes
>> [ 8212.940732] lowmem_reserve[]: 0 0 0 0
>> [ 8212.940735] Node 0 DMA: 1*4kB (R) 0*8kB 4*16kB (R) 1*32kB (R) 1*64kB
>> (R) 2*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1956kB
>> [ 8212.940747] Node 0 DMA32: 652*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2608kB
>> [ 8212.940756] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
>> 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>> [ 8212.940765] 16847 total pagecache pages
>> [ 8212.940766] 8381 pages in swap cache
>> [ 8212.940768] Swap cache stats: add 741397, delete 733016, find
>> 250268/342284
>> [ 8212.940769] Free swap = 1925576kB
>> [ 8212.940770] Total swap = 2097148kB
>> [ 8212.951044] 262143 pages RAM
>> [ 8212.951046] 11939 pages reserved
>> [ 8212.951047] 540820 pages shared
>> [ 8212.951048] 240248 pages non-shared
>> [ 8212.951050] [ pid ] uid tgid total_vm rss nr_ptes swapents
>> oom_score_adj name
>> <snip process list>
>> [ 8212.951310] Out of memory: Kill process 23721 (cc1plus) score 119 or
>> sacrifice child
>> [ 8212.951313] Killed process 23721 (cc1plus) total-vm:530268kB,
>> anon-rss:350980kB, file-rss:9408kB
>> [54810.683658] kjournald starting. Commit interval 5 seconds
>> [54810.684381] EXT3-fs (xvda1): using internal journal
>> [54810.684402] EXT3-fs (xvda1): mounted filesystem with writeback data mode
>>
--
*James Dingwall*
Script Monkey
zynstra-signature-logo <http://www.zynstra.com/>twitter-black
<http://www.twitter.com/zynstra>linkedin-black
<http://www.linkedin.com/company/zynstra>
Zynstra is a private limited company registered in England and Wales
(registered number 07864369). Our registered office is 5 New Street
Square, London, EC4A 3TW and our headquarters are at Bath Ventures,
Broad Quay, Bath, BA1 1UD.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-09 10:48 ` Bob Liu
2014-01-09 10:54 ` James Dingwall
2014-01-09 11:04 ` James Dingwall
@ 2014-01-15 8:49 ` James Dingwall
2014-01-15 14:41 ` Bob Liu
2 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2014-01-15 8:49 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 01/07/2014 05:21 PM, James Dingwall wrote:
>> Bob Liu wrote:
>>> Could you confirm that this problem doesn't exist if loading tmem with
>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>> difference packages during your testing.
>>> This will help to figure out whether selfshrinking is the root cause.
>> Got an oom with selfshrinking=0, again during a gcc compile.
>> Unfortunately I don't have a single test case which demonstrates the
>> problem but as I mentioned before it will generally show up under
>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>
> So the root cause is not because enabled selfshrinking.
> Then what I can think of is that the xen-selfballoon driver was too
> aggressive, too many pages were ballooned out which causeed heavy memory
> pressure to guest OS.
> And kswapd started to reclaim page until most of pages were
> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
> triggered.
> In theory the balloon driver should give back ballooned out pages to
> guest OS, but I'm afraid this procedure is not fast enough.
>
> My suggestion is reserve a min memory for your guest OS so that the
> xen-selfballoon won't be so aggressive.
> You can do it through parameters selfballoon_reserved_mb or
> selfballoon_min_usable_mb.
I am still getting OOM errors with both of these set to 32 so I'll try
another bump to 64. I think that if I do find values which prevent it
though then it is more of a work around than a fix because it still
suggests that swap is not being used when ballooning is no longer
capable of satisfying the request. I've also got an Ubuntu Saucy (3.11
kernel) guest running on the dom0 with tmem activated so I'm going to
see if I can find a comparable workload to see if I get the same issue
with a different kernel configuration.
James
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-15 8:49 ` James Dingwall
@ 2014-01-15 14:41 ` Bob Liu
2014-01-15 16:35 ` James Dingwall
0 siblings, 1 reply; 42+ messages in thread
From: Bob Liu @ 2014-01-15 14:41 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 01/15/2014 04:49 PM, James Dingwall wrote:
> Bob Liu wrote:
>> On 01/07/2014 05:21 PM, James Dingwall wrote:
>>> Bob Liu wrote:
>>>> Could you confirm that this problem doesn't exist if loading tmem with
>>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>>> difference packages during your testing.
>>>> This will help to figure out whether selfshrinking is the root cause.
>>> Got an oom with selfshrinking=0, again during a gcc compile.
>>> Unfortunately I don't have a single test case which demonstrates the
>>> problem but as I mentioned before it will generally show up under
>>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>>
>> So the root cause is not because enabled selfshrinking.
>> Then what I can think of is that the xen-selfballoon driver was too
>> aggressive, too many pages were ballooned out which causeed heavy memory
>> pressure to guest OS.
>> And kswapd started to reclaim page until most of pages were
>> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
>> triggered.
>> In theory the balloon driver should give back ballooned out pages to
>> guest OS, but I'm afraid this procedure is not fast enough.
>>
>> My suggestion is reserve a min memory for your guest OS so that the
>> xen-selfballoon won't be so aggressive.
>> You can do it through parameters selfballoon_reserved_mb or
>> selfballoon_min_usable_mb.
> I am still getting OOM errors with both of these set to 32 so I'll try
> another bump to 64. I think that if I do find values which prevent it
> though then it is more of a work around than a fix because it still
> suggests that swap is not being used when ballooning is no longer
Yes, it's like a work around. But I don't think there is a better way.
>From the recent OOM log your reported:
[ 8212.940769] Free swap = 1925576kB
[ 8212.940770] Total swap = 2097148kB
[504638.442136] Free swap = 1868108kB
[504638.442137] Total swap = 2097148kB
171572KB and 229040KB data are swapped out to swap disk, I think there
are already significantly values for guest OS has only ~300M usable memory.
guest OS can't find out pages suitable for swap any more after so many
pages are swapped out, although at that time the swap device still have
enough space.
The OOM may not be triggered if the balloon driver can give back memory
to guest OS fast enough but I think it's unrealistic.
So the best way is reserve more memory to guest OS.
> capable of satisfying the request. I've also got an Ubuntu Saucy (3.11
> kernel) guest running on the dom0 with tmem activated so I'm going to
> see if I can find a comparable workload to see if I get the same issue
> with a different kernel configuration.
>
> James
--
Regards,
-Bob
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-15 14:41 ` Bob Liu
@ 2014-01-15 16:35 ` James Dingwall
2014-01-16 1:22 ` Bob Liu
0 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2014-01-15 16:35 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 01/15/2014 04:49 PM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 01/07/2014 05:21 PM, James Dingwall wrote:
>>>> Bob Liu wrote:
>>>>> Could you confirm that this problem doesn't exist if loading tmem with
>>>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>>>> difference packages during your testing.
>>>>> This will help to figure out whether selfshrinking is the root cause.
>>>> Got an oom with selfshrinking=0, again during a gcc compile.
>>>> Unfortunately I don't have a single test case which demonstrates the
>>>> problem but as I mentioned before it will generally show up under
>>>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>>>
>>> So the root cause is not because enabled selfshrinking.
>>> Then what I can think of is that the xen-selfballoon driver was too
>>> aggressive, too many pages were ballooned out which causeed heavy memory
>>> pressure to guest OS.
>>> And kswapd started to reclaim page until most of pages were
>>> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
>>> triggered.
>>> In theory the balloon driver should give back ballooned out pages to
>>> guest OS, but I'm afraid this procedure is not fast enough.
>>>
>>> My suggestion is reserve a min memory for your guest OS so that the
>>> xen-selfballoon won't be so aggressive.
>>> You can do it through parameters selfballoon_reserved_mb or
>>> selfballoon_min_usable_mb.
>> I am still getting OOM errors with both of these set to 32 so I'll try
>> another bump to 64. I think that if I do find values which prevent it
>> though then it is more of a work around than a fix because it still
>> suggests that swap is not being used when ballooning is no longer
> Yes, it's like a work around. But I don't think there is a better way.
>
> From the recent OOM log your reported:
> [ 8212.940769] Free swap = 1925576kB
> [ 8212.940770] Total swap = 2097148kB
>
> [504638.442136] Free swap = 1868108kB
> [504638.442137] Total swap = 2097148kB
>
> 171572KB and 229040KB data are swapped out to swap disk, I think there
> are already significantly values for guest OS has only ~300M usable memory.
> guest OS can't find out pages suitable for swap any more after so many
> pages are swapped out, although at that time the swap device still have
> enough space.
>
> The OOM may not be triggered if the balloon driver can give back memory
> to guest OS fast enough but I think it's unrealistic.
> So the best way is reserve more memory to guest OS.
>
>> capable of satisfying the request. I've also got an Ubuntu Saucy (3.11
>> kernel) guest running on the dom0 with tmem activated so I'm going to
>> see if I can find a comparable workload to see if I get the same issue
>> with a different kernel configuration.
>>
I've done a bit more testing and seem to have found an extra condition
which is affecting the OOM behaviour in my guests. All my Gentoo guests
have swap space backed by a dm-crypt volume and if I remove this layer
then things seem to be behaving much more reliably. In my Ubuntu guests
I have plain swap space and so far I haven't been able to trigger an OOM
condition. Is it possible that it is the dm-crypt layer failing to get
working memory when swapping something in/out and causing the error?
James
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-15 16:35 ` James Dingwall
@ 2014-01-16 1:22 ` Bob Liu
2014-01-16 10:52 ` James Dingwall
2014-01-28 17:15 ` James Dingwall
0 siblings, 2 replies; 42+ messages in thread
From: Bob Liu @ 2014-01-16 1:22 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 3925 bytes --]
On 01/16/2014 12:35 AM, James Dingwall wrote:
> Bob Liu wrote:
>> On 01/15/2014 04:49 PM, James Dingwall wrote:
>>> Bob Liu wrote:
>>>> On 01/07/2014 05:21 PM, James Dingwall wrote:
>>>>> Bob Liu wrote:
>>>>>> Could you confirm that this problem doesn't exist if loading tmem
>>>>>> with
>>>>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>>>>> difference packages during your testing.
>>>>>> This will help to figure out whether selfshrinking is the root cause.
>>>>> Got an oom with selfshrinking=0, again during a gcc compile.
>>>>> Unfortunately I don't have a single test case which demonstrates the
>>>>> problem but as I mentioned before it will generally show up under
>>>>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>>>>
>>>> So the root cause is not because enabled selfshrinking.
>>>> Then what I can think of is that the xen-selfballoon driver was too
>>>> aggressive, too many pages were ballooned out which causeed heavy
>>>> memory
>>>> pressure to guest OS.
>>>> And kswapd started to reclaim page until most of pages were
>>>> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
>>>> triggered.
>>>> In theory the balloon driver should give back ballooned out pages to
>>>> guest OS, but I'm afraid this procedure is not fast enough.
>>>>
>>>> My suggestion is reserve a min memory for your guest OS so that the
>>>> xen-selfballoon won't be so aggressive.
>>>> You can do it through parameters selfballoon_reserved_mb or
>>>> selfballoon_min_usable_mb.
>>> I am still getting OOM errors with both of these set to 32 so I'll try
>>> another bump to 64. I think that if I do find values which prevent it
>>> though then it is more of a work around than a fix because it still
>>> suggests that swap is not being used when ballooning is no longer
>> Yes, it's like a work around. But I don't think there is a better way.
>>
>> From the recent OOM log your reported:
>> [ 8212.940769] Free swap = 1925576kB
>> [ 8212.940770] Total swap = 2097148kB
>>
>> [504638.442136] Free swap = 1868108kB
>> [504638.442137] Total swap = 2097148kB
>>
>> 171572KB and 229040KB data are swapped out to swap disk, I think there
>> are already significantly values for guest OS has only ~300M usable
>> memory.
>> guest OS can't find out pages suitable for swap any more after so many
>> pages are swapped out, although at that time the swap device still have
>> enough space.
>>
>> The OOM may not be triggered if the balloon driver can give back memory
>> to guest OS fast enough but I think it's unrealistic.
>> So the best way is reserve more memory to guest OS.
>>
>>> capable of satisfying the request. I've also got an Ubuntu Saucy (3.11
>>> kernel) guest running on the dom0 with tmem activated so I'm going to
>>> see if I can find a comparable workload to see if I get the same issue
>>> with a different kernel configuration.
>>>
> I've done a bit more testing and seem to have found an extra condition
> which is affecting the OOM behaviour in my guests. All my Gentoo guests
> have swap space backed by a dm-crypt volume and if I remove this layer
> then things seem to be behaving much more reliably. In my Ubuntu guests
> I have plain swap space and so far I haven't been able to trigger an OOM
> condition. Is it possible that it is the dm-crypt layer failing to get
> working memory when swapping something in/out and causing the error?
>
One possible reason is the dm layer and related dm target driver occupy
a significant mount of memory and there is no way for xenselfballoon to
know this. So selfballoon driver ballooned out more memory than the
system really requires.
I have made a patch by reserving extra 10% of original total memory, by
this way I think we can make the system much more reliably in all cases.
Could you please have a test? You don't need to set
selfballoon_reserved_mb by yourself any more.
--
Regards,
-Bob
[-- Attachment #2: xen_selfballoon_deaggressive.patch --]
[-- Type: text/x-patch, Size: 1727 bytes --]
diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
index 21e18c1..8f33254 100644
--- a/drivers/xen/xen-selfballoon.c
+++ b/drivers/xen/xen-selfballoon.c
@@ -175,6 +175,7 @@ static void frontswap_selfshrink(void)
#endif /* CONFIG_FRONTSWAP */
#define MB2PAGES(mb) ((mb) << (20 - PAGE_SHIFT))
+#define PAGES2MB(pages) ((pages) >> (20 - PAGE_SHIFT))
/*
* Use current balloon size, the goal (vm_committed_as), and hysteresis
@@ -525,6 +526,7 @@ EXPORT_SYMBOL(register_xen_selfballooning);
int xen_selfballoon_init(bool use_selfballooning, bool use_frontswap_selfshrink)
{
bool enable = false;
+ unsigned long reserve_pages;
if (!xen_domain())
return -ENODEV;
@@ -549,6 +551,26 @@ int xen_selfballoon_init(bool use_selfballooning, bool use_frontswap_selfshrink)
if (!enable)
return -ENODEV;
+ /*
+ * Give selfballoon_reserved_mb a default value(10% of total ram pages)
+ * to make selfballoon not so aggressive.
+ *
+ * There are two reasons:
+ * 1) The goal_page doesn't contain some pages used by kernel space,
+ * like slab cache and pages used by device drivers.
+ *
+ * 2) The balloon driver may not give back memory to guest OS fast
+ * enough when the workload suddenly aquries a lot of memory.
+ *
+ * In both cases, the guest OS will suffer from memory pressure and
+ * OOM killer may be triggered.
+ * By reserving extra 10% of total ram pages, we can keep the system
+ * much more reliably and response faster in some cases.
+ */
+ if (!selfballoon_reserved_mb) {
+ reserve_pages = totalram_pages / 10;
+ selfballoon_reserved_mb = PAGES2MB(reserve_pages);
+ }
schedule_delayed_work(&selfballoon_worker, selfballoon_interval * HZ);
return 0;
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-16 1:22 ` Bob Liu
@ 2014-01-16 10:52 ` James Dingwall
2014-01-28 17:15 ` James Dingwall
1 sibling, 0 replies; 42+ messages in thread
From: James Dingwall @ 2014-01-16 10:52 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 01/16/2014 12:35 AM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 01/15/2014 04:49 PM, James Dingwall wrote:
>>>> Bob Liu wrote:
>>>>> On 01/07/2014 05:21 PM, James Dingwall wrote:
>>>>>> Bob Liu wrote:
>>>>>>> Could you confirm that this problem doesn't exist if loading tmem
>>>>>>> with
>>>>>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>>>>>> difference packages during your testing.
>>>>>>> This will help to figure out whether selfshrinking is the root cause.
>>>>>> Got an oom with selfshrinking=0, again during a gcc compile.
>>>>>> Unfortunately I don't have a single test case which demonstrates the
>>>>>> problem but as I mentioned before it will generally show up under
>>>>>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>>>>>
>>>>> So the root cause is not because enabled selfshrinking.
>>>>> Then what I can think of is that the xen-selfballoon driver was too
>>>>> aggressive, too many pages were ballooned out which causeed heavy
>>>>> memory
>>>>> pressure to guest OS.
>>>>> And kswapd started to reclaim page until most of pages were
>>>>> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
>>>>> triggered.
>>>>> In theory the balloon driver should give back ballooned out pages to
>>>>> guest OS, but I'm afraid this procedure is not fast enough.
>>>>>
>>>>> My suggestion is reserve a min memory for your guest OS so that the
>>>>> xen-selfballoon won't be so aggressive.
>>>>> You can do it through parameters selfballoon_reserved_mb or
>>>>> selfballoon_min_usable_mb.
>>>> I am still getting OOM errors with both of these set to 32 so I'll try
>>>> another bump to 64. I think that if I do find values which prevent it
>>>> though then it is more of a work around than a fix because it still
>>>> suggests that swap is not being used when ballooning is no longer
>>> Yes, it's like a work around. But I don't think there is a better way.
>>>
>>> From the recent OOM log your reported:
>>> [ 8212.940769] Free swap = 1925576kB
>>> [ 8212.940770] Total swap = 2097148kB
>>>
>>> [504638.442136] Free swap = 1868108kB
>>> [504638.442137] Total swap = 2097148kB
>>>
>>> 171572KB and 229040KB data are swapped out to swap disk, I think there
>>> are already significantly values for guest OS has only ~300M usable
>>> memory.
>>> guest OS can't find out pages suitable for swap any more after so many
>>> pages are swapped out, although at that time the swap device still have
>>> enough space.
>>>
>>> The OOM may not be triggered if the balloon driver can give back memory
>>> to guest OS fast enough but I think it's unrealistic.
>>> So the best way is reserve more memory to guest OS.
>>>
>>>> capable of satisfying the request. I've also got an Ubuntu Saucy (3.11
>>>> kernel) guest running on the dom0 with tmem activated so I'm going to
>>>> see if I can find a comparable workload to see if I get the same issue
>>>> with a different kernel configuration.
>>>>
>> I've done a bit more testing and seem to have found an extra condition
>> which is affecting the OOM behaviour in my guests. All my Gentoo guests
>> have swap space backed by a dm-crypt volume and if I remove this layer
>> then things seem to be behaving much more reliably. In my Ubuntu guests
>> I have plain swap space and so far I haven't been able to trigger an OOM
>> condition. Is it possible that it is the dm-crypt layer failing to get
>> working memory when swapping something in/out and causing the error?
>>
> One possible reason is the dm layer and related dm target driver occupy
> a significant mount of memory and there is no way for xenselfballoon to
> know this. So selfballoon driver ballooned out more memory than the
> system really requires.
>
> I have made a patch by reserving extra 10% of original total memory, by
> this way I think we can make the system much more reliably in all cases.
> Could you please have a test? You don't need to set
> selfballoon_reserved_mb by yourself any more.
I'm running a 3.12.7 kernel with all 3 patches and original swap
configuration, so far so good. I shall keep testing and let you know
how things go in a few days.
Thanks,
James
/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking Y
/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_hysteresis
20
/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_inertia 6
/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_selfshrinking
1
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_downhysteresis
8
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballooning 1
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_interval
5
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_min_usable_mb
0
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_reserved_mb
74
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_uphysteresis
1
>
> xen_selfballoon_deaggressive.patch
>
>
> diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
> index 21e18c1..8f33254 100644
> --- a/drivers/xen/xen-selfballoon.c
> +++ b/drivers/xen/xen-selfballoon.c
> @@ -175,6 +175,7 @@ static void frontswap_selfshrink(void)
> #endif /* CONFIG_FRONTSWAP */
>
> #define MB2PAGES(mb) ((mb) << (20 - PAGE_SHIFT))
> +#define PAGES2MB(pages) ((pages) >> (20 - PAGE_SHIFT))
>
> /*
> * Use current balloon size, the goal (vm_committed_as), and hysteresis
> @@ -525,6 +526,7 @@ EXPORT_SYMBOL(register_xen_selfballooning);
> int xen_selfballoon_init(bool use_selfballooning, bool use_frontswap_selfshrink)
> {
> bool enable = false;
> + unsigned long reserve_pages;
>
> if (!xen_domain())
> return -ENODEV;
> @@ -549,6 +551,26 @@ int xen_selfballoon_init(bool use_selfballooning, bool use_frontswap_selfshrink)
> if (!enable)
> return -ENODEV;
>
> + /*
> + * Give selfballoon_reserved_mb a default value(10% of total ram pages)
> + * to make selfballoon not so aggressive.
> + *
> + * There are two reasons:
> + * 1) The goal_page doesn't contain some pages used by kernel space,
> + * like slab cache and pages used by device drivers.
> + *
> + * 2) The balloon driver may not give back memory to guest OS fast
> + * enough when the workload suddenly aquries a lot of memory.
> + *
> + * In both cases, the guest OS will suffer from memory pressure and
> + * OOM killer may be triggered.
> + * By reserving extra 10% of total ram pages, we can keep the system
> + * much more reliably and response faster in some cases.
> + */
> + if (!selfballoon_reserved_mb) {
> + reserve_pages = totalram_pages / 10;
> + selfballoon_reserved_mb = PAGES2MB(reserve_pages);
> + }
> schedule_delayed_work(&selfballoon_worker, selfballoon_interval * HZ);
>
> return 0;
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-16 1:22 ` Bob Liu
2014-01-16 10:52 ` James Dingwall
@ 2014-01-28 17:15 ` James Dingwall
2014-01-29 14:35 ` Bob Liu
1 sibling, 1 reply; 42+ messages in thread
From: James Dingwall @ 2014-01-28 17:15 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
Hi Bob,
> On 01/16/2014 12:35 AM, James Dingwall wrote:
>> Bob Liu wrote:
>>> On 01/15/2014 04:49 PM, James Dingwall wrote:
>>>> Bob Liu wrote:
>>>>> On 01/07/2014 05:21 PM, James Dingwall wrote:
>>>>>> Bob Liu wrote:
>>>>>>> Could you confirm that this problem doesn't exist if loading tmem
>>>>>>> with
>>>>>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>>>>>> difference packages during your testing.
>>>>>>> This will help to figure out whether selfshrinking is the root cause.
>>>>>> Got an oom with selfshrinking=0, again during a gcc compile.
>>>>>> Unfortunately I don't have a single test case which demonstrates the
>>>>>> problem but as I mentioned before it will generally show up under
>>>>>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>>>>>
>>>>> So the root cause is not because enabled selfshrinking.
>>>>> Then what I can think of is that the xen-selfballoon driver was too
>>>>> aggressive, too many pages were ballooned out which causeed heavy
>>>>> memory
>>>>> pressure to guest OS.
>>>>> And kswapd started to reclaim page until most of pages were
>>>>> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
>>>>> triggered.
>>>>> In theory the balloon driver should give back ballooned out pages to
>>>>> guest OS, but I'm afraid this procedure is not fast enough.
>>>>>
>>>>> My suggestion is reserve a min memory for your guest OS so that the
>>>>> xen-selfballoon won't be so aggressive.
>>>>> You can do it through parameters selfballoon_reserved_mb or
>>>>> selfballoon_min_usable_mb.
>>>> I am still getting OOM errors with both of these set to 32 so I'll try
>>>> another bump to 64. I think that if I do find values which prevent it
>>>> though then it is more of a work around than a fix because it still
>>>> suggests that swap is not being used when ballooning is no longer
>>> Yes, it's like a work around. But I don't think there is a better way.
>>>
>>> From the recent OOM log your reported:
>>> [ 8212.940769] Free swap = 1925576kB
>>> [ 8212.940770] Total swap = 2097148kB
>>>
>>> [504638.442136] Free swap = 1868108kB
>>> [504638.442137] Total swap = 2097148kB
>>>
>>> 171572KB and 229040KB data are swapped out to swap disk, I think there
>>> are already significantly values for guest OS has only ~300M usable
>>> memory.
>>> guest OS can't find out pages suitable for swap any more after so many
>>> pages are swapped out, although at that time the swap device still have
>>> enough space.
>>>
>>> The OOM may not be triggered if the balloon driver can give back memory
>>> to guest OS fast enough but I think it's unrealistic.
>>> So the best way is reserve more memory to guest OS.
>>>
>>>> capable of satisfying the request. I've also got an Ubuntu Saucy (3.11
>>>> kernel) guest running on the dom0 with tmem activated so I'm going to
>>>> see if I can find a comparable workload to see if I get the same issue
>>>> with a different kernel configuration.
>>>>
>> I've done a bit more testing and seem to have found an extra condition
>> which is affecting the OOM behaviour in my guests. All my Gentoo guests
>> have swap space backed by a dm-crypt volume and if I remove this layer
>> then things seem to be behaving much more reliably. In my Ubuntu guests
>> I have plain swap space and so far I haven't been able to trigger an OOM
>> condition. Is it possible that it is the dm-crypt layer failing to get
>> working memory when swapping something in/out and causing the error?
>>
> One possible reason is the dm layer and related dm target driver occupy
> a significant mount of memory and there is no way for xenselfballoon to
> know this. So selfballoon driver ballooned out more memory than the
> system really requires.
>
> I have made a patch by reserving extra 10% of original total memory, by
> this way I think we can make the system much more reliably in all cases.
> Could you please have a test? You don't need to set
> selfballoon_reserved_mb by yourself any more.
I have to say that with this patch the situation has definitely
improved. I have been running it with 3.12.[78] and 3.13 and pushing it
quite hard for the last 10 days or so. Unfortunately yesterday I got an
OOM during a compile (link) of webkit-gtk. I think your patch is part
of the solution but I'm not sure if the other bit is simply to be more
generous with the guest memory allocation or something else. Having
tested with memory = 512 and no tmem I get an OOM with the same
compile, with memory = 1024 and no tmem the compile completes ok (both
cases without maxmem). As my domains are usually started with memory =
512 and maxmem = 1024 it seems that there should be sufficient with my
default parameters. Also for an experiment I set memory=1024 and removed
maxmem and when tmem is activated I see "[ 3393.884105] xen:balloon:
reserve_additional_memory: add_memory() failed: -17" printed many times
in the guest kernel log.
Regards,
James
[456770.748827] Mem-Info:
[456770.748829] Node 0 DMA per-cpu:
[456770.748833] CPU 0: hi: 0, btch: 1 usd: 0
[456770.748835] CPU 1: hi: 0, btch: 1 usd: 0
[456770.748836] Node 0 DMA32 per-cpu:
[456770.748838] CPU 0: hi: 186, btch: 31 usd: 173
[456770.748840] CPU 1: hi: 186, btch: 31 usd: 120
[456770.748846] active_anon:91431 inactive_anon:96269 isolated_anon:0
active_file:13286 inactive_file:31256 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
free:1155 slab_reclaimable:7001 slab_unreclaimable:3932
mapped:2300 shmem:88 pagetables:2576 bounce:0
free_cma:0 totalram:255578 balloontarget:327320
[456770.748849] Node 0 DMA free:1956kB min:88kB low:108kB high:132kB
active_anon:3128kB inactive_anon:3328kB active_file:1888kB
inactive_file:2088kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB
writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:684kB
slab_unreclaimable:720kB kernel_stack:72kB pagetables:488kB unstable:0kB
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:17841
all_unreclaimable? yes
[456770.748863] lowmem_reserve[]: 0 469 469 469
[456770.748866] Node 0 DMA32 free:2664kB min:2728kB low:3408kB
high:4092kB active_anon:362596kB inactive_anon:381748kB
active_file:51256kB inactive_file:122936kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:1032192kB
managed:1006400kB mlocked:0kB dirty:0kB writeback:0kB mapped:9168kB
shmem:352kB slab_reclaimable:27320kB slab_unreclaimable:15008kB
kernel_stack:1784kB pagetables:9816kB unstable:0kB bounce:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:1382021 all_unreclaimable? yes
[456770.748874] lowmem_reserve[]: 0 0 0 0
[456770.748877] Node 0 DMA: 1*4kB (R) 0*8kB 0*16kB 5*32kB (R) 2*64kB (R)
1*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1956kB
[456770.748890] Node 0 DMA32: 666*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2664kB
[456770.748899] 48556 total pagecache pages
[456770.748901] 35203 pages in swap cache
[456770.748903] Swap cache stats: add 358621, delete 323418, find
206319/224002
[456770.748904] Free swap = 1671532kB
[456770.748905] Total swap = 2097148kB
[456770.748906] 262047 pages RAM
[456770.748907] 0 pages HighMem/MovableOnly
[456770.748908] 6448 pages reserved
<snip process list>
[456770.749070] Out of memory: Kill process 28271 (ld) score 110 or
sacrifice child
[456770.749073] Killed process 28271 (ld) total-vm:358488kB,
anon-rss:324588kB, file-rss:1456kB
>
>
> xen_selfballoon_deaggressive.patch
>
>
> diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
> index 21e18c1..8f33254 100644
> --- a/drivers/xen/xen-selfballoon.c
> +++ b/drivers/xen/xen-selfballoon.c
> @@ -175,6 +175,7 @@ static void frontswap_selfshrink(void)
> #endif /* CONFIG_FRONTSWAP */
>
> #define MB2PAGES(mb) ((mb) << (20 - PAGE_SHIFT))
> +#define PAGES2MB(pages) ((pages) >> (20 - PAGE_SHIFT))
>
> /*
> * Use current balloon size, the goal (vm_committed_as), and hysteresis
> @@ -525,6 +526,7 @@ EXPORT_SYMBOL(register_xen_selfballooning);
> int xen_selfballoon_init(bool use_selfballooning, bool use_frontswap_selfshrink)
> {
> bool enable = false;
> + unsigned long reserve_pages;
>
> if (!xen_domain())
> return -ENODEV;
> @@ -549,6 +551,26 @@ int xen_selfballoon_init(bool use_selfballooning, bool use_frontswap_selfshrink)
> if (!enable)
> return -ENODEV;
>
> + /*
> + * Give selfballoon_reserved_mb a default value(10% of total ram pages)
> + * to make selfballoon not so aggressive.
> + *
> + * There are two reasons:
> + * 1) The goal_page doesn't contain some pages used by kernel space,
> + * like slab cache and pages used by device drivers.
> + *
> + * 2) The balloon driver may not give back memory to guest OS fast
> + * enough when the workload suddenly aquries a lot of memory.
> + *
> + * In both cases, the guest OS will suffer from memory pressure and
> + * OOM killer may be triggered.
> + * By reserving extra 10% of total ram pages, we can keep the system
> + * much more reliably and response faster in some cases.
> + */
> + if (!selfballoon_reserved_mb) {
> + reserve_pages = totalram_pages / 10;
> + selfballoon_reserved_mb = PAGES2MB(reserve_pages);
> + }
> schedule_delayed_work(&selfballoon_worker, selfballoon_interval * HZ);
>
> return 0;
--
*James Dingwall*
Script Monkey
zynstra-signature-logo <http://www.zynstra.com/>twitter-black
<http://www.twitter.com/zynstra>linkedin-black
<http://www.linkedin.com/company/zynstra>
Zynstra is a private limited company registered in England and Wales
(registered number 07864369). Our registered office is 5 New Street
Square, London, EC4A 3TW and our headquarters are at Bath Ventures,
Broad Quay, Bath, BA1 1UD.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-28 17:15 ` James Dingwall
@ 2014-01-29 14:35 ` Bob Liu
2014-01-29 14:45 ` James Dingwall
0 siblings, 1 reply; 42+ messages in thread
From: Bob Liu @ 2014-01-29 14:35 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 01/29/2014 01:15 AM, James Dingwall wrote:
> Bob Liu wrote:
>>
>> I have made a patch by reserving extra 10% of original total memory, by
>> this way I think we can make the system much more reliably in all cases.
>> Could you please have a test? You don't need to set
>> selfballoon_reserved_mb by yourself any more.
> I have to say that with this patch the situation has definitely
> improved. I have been running it with 3.12.[78] and 3.13 and pushing it
> quite hard for the last 10 days or so. Unfortunately yesterday I got an
Good news!
> OOM during a compile (link) of webkit-gtk. I think your patch is part
> of the solution but I'm not sure if the other bit is simply to be more
> generous with the guest memory allocation or something else. Having
> tested with memory = 512 and no tmem I get an OOM with the same
> compile, with memory = 1024 and no tmem the compile completes ok (both
> cases without maxmem). As my domains are usually started with memory =
> 512 and maxmem = 1024 it seems that there should be sufficient with my
But I think from the beginning tmem/balloon driver can't expand guest
memory from size 'memory' to 'maxmem' automatically.
> default parameters. Also for an experiment I set memory=1024 and removed
> maxmem and when tmem is activated I see "[ 3393.884105] xen:balloon:
> reserve_additional_memory: add_memory() failed: -17" printed many times
> in the guest kernel log.
>
I'll take a look at it.
--
Regards,
-Bob
> Regards,
> James
>
> [456770.748827] Mem-Info:
> [456770.748829] Node 0 DMA per-cpu:
> [456770.748833] CPU 0: hi: 0, btch: 1 usd: 0
> [456770.748835] CPU 1: hi: 0, btch: 1 usd: 0
> [456770.748836] Node 0 DMA32 per-cpu:
> [456770.748838] CPU 0: hi: 186, btch: 31 usd: 173
> [456770.748840] CPU 1: hi: 186, btch: 31 usd: 120
> [456770.748846] active_anon:91431 inactive_anon:96269 isolated_anon:0
> active_file:13286 inactive_file:31256 isolated_file:0
> unevictable:0 dirty:0 writeback:0 unstable:0
> free:1155 slab_reclaimable:7001 slab_unreclaimable:3932
> mapped:2300 shmem:88 pagetables:2576 bounce:0
> free_cma:0 totalram:255578 balloontarget:327320
> [456770.748849] Node 0 DMA free:1956kB min:88kB low:108kB high:132kB
> active_anon:3128kB inactive_anon:3328kB active_file:1888kB
> inactive_file:2088kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:684kB
> slab_unreclaimable:720kB kernel_stack:72kB pagetables:488kB unstable:0kB
> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:17841
> all_unreclaimable? yes
> [456770.748863] lowmem_reserve[]: 0 469 469 469
> [456770.748866] Node 0 DMA32 free:2664kB min:2728kB low:3408kB
> high:4092kB active_anon:362596kB inactive_anon:381748kB
> active_file:51256kB inactive_file:122936kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:1032192kB
> managed:1006400kB mlocked:0kB dirty:0kB writeback:0kB mapped:9168kB
> shmem:352kB slab_reclaimable:27320kB slab_unreclaimable:15008kB
> kernel_stack:1784kB pagetables:9816kB unstable:0kB bounce:0kB
> free_cma:0kB writeback_tmp:0kB pages_scanned:1382021 all_unreclaimable? yes
> [456770.748874] lowmem_reserve[]: 0 0 0 0
> [456770.748877] Node 0 DMA: 1*4kB (R) 0*8kB 0*16kB 5*32kB (R) 2*64kB (R)
> 1*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1956kB
> [456770.748890] Node 0 DMA32: 666*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2664kB
> [456770.748899] 48556 total pagecache pages
> [456770.748901] 35203 pages in swap cache
> [456770.748903] Swap cache stats: add 358621, delete 323418, find
> 206319/224002
> [456770.748904] Free swap = 1671532kB
> [456770.748905] Total swap = 2097148kB
> [456770.748906] 262047 pages RAM
> [456770.748907] 0 pages HighMem/MovableOnly
> [456770.748908] 6448 pages reserved
> <snip process list>
> [456770.749070] Out of memory: Kill process 28271 (ld) score 110 or
> sacrifice child
> [456770.749073] Killed process 28271 (ld) total-vm:358488kB,
> anon-rss:324588kB, file-rss:1456kB
>
>>
>>
>> xen_selfballoon_deaggressive.patch
>>
>>
>> diff --git a/drivers/xen/xen-selfballoon.c
>> b/drivers/xen/xen-selfballoon.c
>> index 21e18c1..8f33254 100644
>> --- a/drivers/xen/xen-selfballoon.c
>> +++ b/drivers/xen/xen-selfballoon.c
>> @@ -175,6 +175,7 @@ static void frontswap_selfshrink(void)
>> #endif /* CONFIG_FRONTSWAP */
>> #define MB2PAGES(mb) ((mb) << (20 - PAGE_SHIFT))
>> +#define PAGES2MB(pages) ((pages) >> (20 - PAGE_SHIFT))
>> /*
>> * Use current balloon size, the goal (vm_committed_as), and hysteresis
>> @@ -525,6 +526,7 @@ EXPORT_SYMBOL(register_xen_selfballooning);
>> int xen_selfballoon_init(bool use_selfballooning, bool
>> use_frontswap_selfshrink)
>> {
>> bool enable = false;
>> + unsigned long reserve_pages;
>> if (!xen_domain())
>> return -ENODEV;
>> @@ -549,6 +551,26 @@ int xen_selfballoon_init(bool use_selfballooning,
>> bool use_frontswap_selfshrink)
>> if (!enable)
>> return -ENODEV;
>> + /*
>> + * Give selfballoon_reserved_mb a default value(10% of total ram
>> pages)
>> + * to make selfballoon not so aggressive.
>> + *
>> + * There are two reasons:
>> + * 1) The goal_page doesn't contain some pages used by kernel space,
>> + * like slab cache and pages used by device drivers.
>> + *
>> + * 2) The balloon driver may not give back memory to guest OS fast
>> + * enough when the workload suddenly aquries a lot of memory.
>> + *
>> + * In both cases, the guest OS will suffer from memory pressure and
>> + * OOM killer may be triggered.
>> + * By reserving extra 10% of total ram pages, we can keep the system
>> + * much more reliably and response faster in some cases.
>> + */
>> + if (!selfballoon_reserved_mb) {
>> + reserve_pages = totalram_pages / 10;
>> + selfballoon_reserved_mb = PAGES2MB(reserve_pages);
>> + }
>> schedule_delayed_work(&selfballoon_worker, selfballoon_interval
>> * HZ);
>> return 0;
>
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-29 14:35 ` Bob Liu
@ 2014-01-29 14:45 ` James Dingwall
2014-01-31 16:56 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 42+ messages in thread
From: James Dingwall @ 2014-01-29 14:45 UTC (permalink / raw)
To: Bob Liu; +Cc: xen-devel
Bob Liu wrote:
> On 01/29/2014 01:15 AM, James Dingwall wrote:
>> Bob Liu wrote:
>>> I have made a patch by reserving extra 10% of original total memory, by
>>> this way I think we can make the system much more reliably in all cases.
>>> Could you please have a test? You don't need to set
>>> selfballoon_reserved_mb by yourself any more.
>> I have to say that with this patch the situation has definitely
>> improved. I have been running it with 3.12.[78] and 3.13 and pushing it
>> quite hard for the last 10 days or so. Unfortunately yesterday I got an
> Good news!
>
>> OOM during a compile (link) of webkit-gtk. I think your patch is part
>> of the solution but I'm not sure if the other bit is simply to be more
>> generous with the guest memory allocation or something else. Having
>> tested with memory = 512 and no tmem I get an OOM with the same
>> compile, with memory = 1024 and no tmem the compile completes ok (both
>> cases without maxmem). As my domains are usually started with memory =
>> 512 and maxmem = 1024 it seems that there should be sufficient with my
> But I think from the beginning tmem/balloon driver can't expand guest
> memory from size 'memory' to 'maxmem' automatically.
I am carrying this patch for libxl (4.3.1) because maxmem wasn't being
honoured.
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 356f920..fb7965d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -235,7 +235,7 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap);
libxl_set_vcpuaffinity_all(ctx, domid, info->max_vcpus,
&info->cpumap);
- xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb +
LIBXL_MAXMEM_CONSTANT);
+ xc_domain_setmaxmem(ctx->xch, domid, info->max_memkb +
LIBXL_MAXMEM_CONSTANT);
xs_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenstored/domid", NULL);
state->store_domid = xs_domid ? atoi(xs_domid) : 0;
free(xs_domid);
>
>> default parameters. Also for an experiment I set memory=1024 and removed
>> maxmem and when tmem is activated I see "[ 3393.884105] xen:balloon:
>> reserve_additional_memory: add_memory() failed: -17" printed many times
>> in the guest kernel log.
>>
> I'll take a look at it.
It seems possible that this could be the same cause as for the message
being printed in dom0 and which I reported in
http://lists.xen.org/archives/html/xen-devel/2012-12/msg01607.html and
for which no fix seems to have made it to the kernel. I'm still working
around this with:
#!/bin/sh
CURRENT_KB="/sys/bus/xen_memory/devices/xen_memory0/info/current_kb"
TARGET_KB="/sys/bus/xen_memory/devices/xen_memory0/target_kb"
CKB=$(< "${CURRENT_KB}")
TKB=$(< "${TARGET_KB}")
if [ "${TKB}" -gt "${CKB}" ] ; then
echo "Resizing dom0 memory balloon target to ${CKB}"
echo "${CKB}" > "${TARGET_KB}"
fi
Thanks,
James
--
*James Dingwall*
Script Monkey
zynstra-signature-logo <http://www.zynstra.com/>twitter-black
<http://www.twitter.com/zynstra>linkedin-black
<http://www.linkedin.com/company/zynstra>
Zynstra is a private limited company registered in England and Wales
(registered number 07864369). Our registered office is 5 New Street
Square, London, EC4A 3TW and our headquarters are at Bath Ventures,
Broad Quay, Bath, BA1 1UD.
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-29 14:45 ` James Dingwall
@ 2014-01-31 16:56 ` Konrad Rzeszutek Wilk
2014-02-03 9:49 ` Daniel Kiper
0 siblings, 1 reply; 42+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-01-31 16:56 UTC (permalink / raw)
To: James Dingwall, daniel.kiper; +Cc: xen-devel
On Wed, Jan 29, 2014 at 02:45:24PM +0000, James Dingwall wrote:
> Bob Liu wrote:
> >On 01/29/2014 01:15 AM, James Dingwall wrote:
> >>Bob Liu wrote:
> >>>I have made a patch by reserving extra 10% of original total memory, by
> >>>this way I think we can make the system much more reliably in all cases.
> >>>Could you please have a test? You don't need to set
> >>>selfballoon_reserved_mb by yourself any more.
> >>I have to say that with this patch the situation has definitely
> >>improved. I have been running it with 3.12.[78] and 3.13 and pushing it
> >>quite hard for the last 10 days or so. Unfortunately yesterday I got an
> >Good news!
> >
> >>OOM during a compile (link) of webkit-gtk. I think your patch is part
> >>of the solution but I'm not sure if the other bit is simply to be more
> >>generous with the guest memory allocation or something else. Having
> >>tested with memory = 512 and no tmem I get an OOM with the same
> >>compile, with memory = 1024 and no tmem the compile completes ok (both
> >>cases without maxmem). As my domains are usually started with memory =
> >>512 and maxmem = 1024 it seems that there should be sufficient with my
> >But I think from the beginning tmem/balloon driver can't expand guest
> >memory from size 'memory' to 'maxmem' automatically.
> I am carrying this patch for libxl (4.3.1) because maxmem wasn't
> being honoured.
Daniel,
Weren't you working on a similar patch? Do you recall what happend to it?
Thanks.
>
> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> index 356f920..fb7965d 100644
> --- a/tools/libxl/libxl_dom.c
> +++ b/tools/libxl/libxl_dom.c
> @@ -235,7 +235,7 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
> libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap);
> libxl_set_vcpuaffinity_all(ctx, domid, info->max_vcpus,
> &info->cpumap);
>
> - xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb +
> LIBXL_MAXMEM_CONSTANT);
> + xc_domain_setmaxmem(ctx->xch, domid, info->max_memkb +
> LIBXL_MAXMEM_CONSTANT);
> xs_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenstored/domid", NULL);
> state->store_domid = xs_domid ? atoi(xs_domid) : 0;
> free(xs_domid);
>
> >
> >>default parameters. Also for an experiment I set memory=1024 and removed
> >>maxmem and when tmem is activated I see "[ 3393.884105] xen:balloon:
> >>reserve_additional_memory: add_memory() failed: -17" printed many times
> >>in the guest kernel log.
> >>
> >I'll take a look at it.
> It seems possible that this could be the same cause as for the
> message being printed in dom0 and which I reported in
> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01607.html
> and for which no fix seems to have made it to the kernel. I'm still
> working around this with:
>
> #!/bin/sh
>
> CURRENT_KB="/sys/bus/xen_memory/devices/xen_memory0/info/current_kb"
> TARGET_KB="/sys/bus/xen_memory/devices/xen_memory0/target_kb"
>
> CKB=$(< "${CURRENT_KB}")
> TKB=$(< "${TARGET_KB}")
>
> if [ "${TKB}" -gt "${CKB}" ] ; then
> echo "Resizing dom0 memory balloon target to ${CKB}"
> echo "${CKB}" > "${TARGET_KB}"
> fi
>
> Thanks,
> James
>
>
> --
>
> *James Dingwall*
>
> Script Monkey
>
> zynstra-signature-logo <http://www.zynstra.com/>twitter-black
> <http://www.twitter.com/zynstra>linkedin-black
> <http://www.linkedin.com/company/zynstra>
>
> Zynstra is a private limited company registered in England and Wales
> (registered number 07864369). Our registered office is 5 New Street
> Square, London, EC4A 3TW and our headquarters are at Bath Ventures,
> Broad Quay, Bath, BA1 1UD.
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-01-31 16:56 ` Konrad Rzeszutek Wilk
@ 2014-02-03 9:49 ` Daniel Kiper
2014-02-03 10:30 ` Konrad Rzeszutek Wilk
2014-02-03 11:20 ` James Dingwall
0 siblings, 2 replies; 42+ messages in thread
From: Daniel Kiper @ 2014-02-03 9:49 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk; +Cc: James Dingwall, xen-devel
Hi,
On Fri, Jan 31, 2014 at 11:56:54AM -0500, Konrad Rzeszutek Wilk wrote:
> On Wed, Jan 29, 2014 at 02:45:24PM +0000, James Dingwall wrote:
> > Bob Liu wrote:
> > >On 01/29/2014 01:15 AM, James Dingwall wrote:
> > >>Bob Liu wrote:
> > >>>I have made a patch by reserving extra 10% of original total memory, by
> > >>>this way I think we can make the system much more reliably in all cases.
> > >>>Could you please have a test? You don't need to set
> > >>>selfballoon_reserved_mb by yourself any more.
> > >>I have to say that with this patch the situation has definitely
> > >>improved. I have been running it with 3.12.[78] and 3.13 and pushing it
> > >>quite hard for the last 10 days or so. Unfortunately yesterday I got an
> > >Good news!
> > >
> > >>OOM during a compile (link) of webkit-gtk. I think your patch is part
> > >>of the solution but I'm not sure if the other bit is simply to be more
> > >>generous with the guest memory allocation or something else. Having
> > >>tested with memory = 512 and no tmem I get an OOM with the same
> > >>compile, with memory = 1024 and no tmem the compile completes ok (both
> > >>cases without maxmem). As my domains are usually started with memory =
> > >>512 and maxmem = 1024 it seems that there should be sufficient with my
Hmmm... James, how do you build webkit-gtk? Just simple "make" or "make -j"?
Could you confirm that webkit-gtk in any "subjobs" do not use "make -j"?
> > >But I think from the beginning tmem/balloon driver can't expand guest
> > >memory from size 'memory' to 'maxmem' automatically.
> > I am carrying this patch for libxl (4.3.1) because maxmem wasn't
> > being honoured.
James, what do you mean by "maxmem wasn't being honoured"?
> Daniel,
>
> Weren't you working on a similar patch? Do you recall what happend to it?
Yep, and it was not applied because it has not been so mature. Additionally,
this patch is part of bigger puzzle. I have it on my todo list but I would
like to finish EFI stuff first. However, if you wish I could back to this
work (if it is more important then EFI right now).
Daniel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-02-03 9:49 ` Daniel Kiper
@ 2014-02-03 10:30 ` Konrad Rzeszutek Wilk
2014-02-03 11:20 ` James Dingwall
1 sibling, 0 replies; 42+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-02-03 10:30 UTC (permalink / raw)
To: Daniel Kiper; +Cc: James Dingwall, xen-devel
On February 3, 2014 4:49:12 AM EST, Daniel Kiper <daniel.kiper@oracle.com> wrote:
>Hi,
>
>On Fri, Jan 31, 2014 at 11:56:54AM -0500, Konrad Rzeszutek Wilk wrote:
>> On Wed, Jan 29, 2014 at 02:45:24PM +0000, James Dingwall wrote:
>> > Bob Liu wrote:
>> > >On 01/29/2014 01:15 AM, James Dingwall wrote:
>> > >>Bob Liu wrote:
>> > >>>I have made a patch by reserving extra 10% of original total
>memory, by
>> > >>>this way I think we can make the system much more reliably in
>all cases.
>> > >>>Could you please have a test? You don't need to set
>> > >>>selfballoon_reserved_mb by yourself any more.
>> > >>I have to say that with this patch the situation has definitely
>> > >>improved. I have been running it with 3.12.[78] and 3.13 and
>pushing it
>> > >>quite hard for the last 10 days or so. Unfortunately yesterday I
>got an
>> > >Good news!
>> > >
>> > >>OOM during a compile (link) of webkit-gtk. I think your patch is
>part
>> > >>of the solution but I'm not sure if the other bit is simply to be
>more
>> > >>generous with the guest memory allocation or something else.
>Having
>> > >>tested with memory = 512 and no tmem I get an OOM with the same
>> > >>compile, with memory = 1024 and no tmem the compile completes ok
>(both
>> > >>cases without maxmem). As my domains are usually started with
>memory =
>> > >>512 and maxmem = 1024 it seems that there should be sufficient
>with my
>
>Hmmm... James, how do you build webkit-gtk? Just simple "make" or "make
>-j"?
>Could you confirm that webkit-gtk in any "subjobs" do not use "make
>-j"?
>
>> > >But I think from the beginning tmem/balloon driver can't expand
>guest
>> > >memory from size 'memory' to 'maxmem' automatically.
>> > I am carrying this patch for libxl (4.3.1) because maxmem wasn't
>> > being honoured.
>
>James, what do you mean by "maxmem wasn't being honoured"?
>
>> Daniel,
>>
>> Weren't you working on a similar patch? Do you recall what happend to
>it?
>
>Yep, and it was not applied because it has not been so mature.
>Additionally,
>this patch is part of bigger puzzle. I have it on my todo list but I
>would
>like to finish EFI stuff first. However, if you wish I could back to
>this
>work (if it is more important then EFI right now).
No - EFI is paramount. I was wondering if you had posted some patches in the past that triggered a discussion about this? Perhaps James can pick up the work and make it mature?
>
>Daniel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-02-03 9:49 ` Daniel Kiper
2014-02-03 10:30 ` Konrad Rzeszutek Wilk
@ 2014-02-03 11:20 ` James Dingwall
2014-02-03 14:00 ` Daniel Kiper
1 sibling, 1 reply; 42+ messages in thread
From: James Dingwall @ 2014-02-03 11:20 UTC (permalink / raw)
To: Daniel Kiper, Konrad Rzeszutek Wilk; +Cc: xen-devel
Daniel Kiper wrote:
> Hi,
>
> On Fri, Jan 31, 2014 at 11:56:54AM -0500, Konrad Rzeszutek Wilk wrote:
>> On Wed, Jan 29, 2014 at 02:45:24PM +0000, James Dingwall wrote:
>>> Bob Liu wrote:
>>>> On 01/29/2014 01:15 AM, James Dingwall wrote:
>>>>> Bob Liu wrote:
>>>>>> I have made a patch by reserving extra 10% of original total memory, by
>>>>>> this way I think we can make the system much more reliably in all cases.
>>>>>> Could you please have a test? You don't need to set
>>>>>> selfballoon_reserved_mb by yourself any more.
>>>>> I have to say that with this patch the situation has definitely
>>>>> improved. I have been running it with 3.12.[78] and 3.13 and pushing it
>>>>> quite hard for the last 10 days or so. Unfortunately yesterday I got an
>>>> Good news!
>>>>
>>>>> OOM during a compile (link) of webkit-gtk. I think your patch is part
>>>>> of the solution but I'm not sure if the other bit is simply to be more
>>>>> generous with the guest memory allocation or something else. Having
>>>>> tested with memory = 512 and no tmem I get an OOM with the same
>>>>> compile, with memory = 1024 and no tmem the compile completes ok (both
>>>>> cases without maxmem). As my domains are usually started with memory =
>>>>> 512 and maxmem = 1024 it seems that there should be sufficient with my
> Hmmm... James, how do you build webkit-gtk? Just simple "make" or "make -j"?
> Could you confirm that webkit-gtk in any "subjobs" do not use "make -j"?
My guest domain is Gentoo and I have MAKEOPTS="-j2" set in make.conf and
according to the build log for webkit-gtk this is used unchanged:
>>> Source configured.
>>> Compiling source in
/var/tmp/portage/net-libs/webkit-gtk-2.0.4/work/webkitgtk-2.0.4 ...
make -j2
I wouldn't read anything in particular to it being webkit as I have seen
similar from other large compiles (gcc, glibc, kdelibs, ...)
>
>>>> But I think from the beginning tmem/balloon driver can't expand guest
>>>> memory from size 'memory' to 'maxmem' automatically.
>>> I am carrying this patch for libxl (4.3.1) because maxmem wasn't
>>> being honoured.
> James, what do you mean by "maxmem wasn't being honoured"?
http://lists.xen.org/archives/html/xen-devel/2013-10/msg02228.html - the
guest would never balloon above the value of memory when maxmem was set
and was > memory in the configuration file. There were some discussions
about the correctness of this patch but the only place I could see an
impact of maxmem was the parsing of the config parameters for the setup
of the guest domain. IIRC the xl mem-max command which changes the same
parameter once the guest domain is running resulted with the balloon up
behaviour to max-mem working as expected. So the discrepancy between how
xl behaves with the maxmem in the config or the execution of xl mem-max
was what I had noted and what this patch resolved. It would be easy to
repeat those tests if necessary.
James
>
>> Daniel,
>>
>> Weren't you working on a similar patch? Do you recall what happend to it?
> Yep, and it was not applied because it has not been so mature. Additionally,
> this patch is part of bigger puzzle. I have it on my todo list but I would
> like to finish EFI stuff first. However, if you wish I could back to this
> work (if it is more important then EFI right now).
>
> Daniel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning
2014-02-03 11:20 ` James Dingwall
@ 2014-02-03 14:00 ` Daniel Kiper
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Kiper @ 2014-02-03 14:00 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On Mon, Feb 03, 2014 at 11:20:33AM +0000, James Dingwall wrote:
> Daniel Kiper wrote:
[...]
> >Hmmm... James, how do you build webkit-gtk? Just simple "make" or "make -j"?
> >Could you confirm that webkit-gtk in any "subjobs" do not use "make -j"?
> My guest domain is Gentoo and I have MAKEOPTS="-j2" set in make.conf
> and according to the build log for webkit-gtk this is used
> unchanged:
> >>> Source configured.
> >>> Compiling source in
> /var/tmp/portage/net-libs/webkit-gtk-2.0.4/work/webkitgtk-2.0.4 ...
> make -j2
>
> I wouldn't read anything in particular to it being webkit as I have
> seen similar from other large compiles (gcc, glibc, kdelibs, ...)
Thanks, it makes sens. I was afraid that somewhere "make -j" without
an argument has been called.
> >>>>But I think from the beginning tmem/balloon driver can't expand guest
> >>>>memory from size 'memory' to 'maxmem' automatically.
> >>>I am carrying this patch for libxl (4.3.1) because maxmem wasn't
> >>>being honoured.
> >James, what do you mean by "maxmem wasn't being honoured"?
> http://lists.xen.org/archives/html/xen-devel/2013-10/msg02228.html -
> the guest would never balloon above the value of memory when maxmem
> was set and was > memory in the configuration file. There were some
> discussions about the correctness of this patch but the only place I
> could see an impact of maxmem was the parsing of the config
> parameters for the setup of the guest domain. IIRC the xl mem-max
> command which changes the same parameter once the guest domain is
> running resulted with the balloon up behaviour to max-mem working as
> expected. So the discrepancy between how xl behaves with the maxmem
> in the config or the execution of xl mem-max was what I had noted
> and what this patch resolved. It would be easy to repeat those
> tests if necessary.
Please look for "[PATCH v4 0/4] libxl: memory management patches",
"[PATCH v2 0/2] xen/balloon: Extension and fix" and earlier related
threads. They contain almost hammered out and agreed solution for
this issue (you also find explanation for this xl behavior; note that
xm behavior was different). However, I was not able to finish this
patches due to other stuff on my plate. I have still this issue on
my todo list. If you would like to work on these patches go ahead.
I am happy to help but I am not able to work on them right now.
Daniel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Kernel 3.11 / 3.12 OOM killer and Xen ballooning
@ 2013-11-21 11:28 James Dingwall
0 siblings, 0 replies; 42+ messages in thread
From: James Dingwall @ 2013-11-21 11:28 UTC (permalink / raw)
To: linux-kernel
Hi,
Since 3.11 I have noticed that the OOM killer quite frequently triggers
in my Xen guest domains which use ballooning to increase/decrease their
memory allocation according to their requirements. One example domain I
have has a maximum memory setting of ~1.5Gb but it usually idles at
~300Mb, it is also configured with 2Gb swap which is almost 100% free.
# free
total used free shared buffers cached
Mem: 272080 248108 23972 0 1448 63064
-/+ buffers/cache: 183596 88484
Swap: 2097148 8 2097140
There is plenty of available free memory in the hypervisor to balloon to
the maximum size:
# xl info | grep free_mem
free_memory : 14923
An example trace from the oom killer in 3.12 is added below. So far I
have not been able to reproduce this at will so it is difficult to start
bisecting it to see if a particular change introduced this. However it
does seem that the behaviour is wrong because a) ballooning could give
the guest more memory, b) there is lots of swap available which could be
used as a fallback.
If other information could help or there are more tests that I could run
then please let me know.
Thanks,
James
[473233.777271] emerge invoked oom-killer: gfp_mask=0x280da, order=0,
oom_score_adj=0
[473233.777279] CPU: 0 PID: 22159 Comm: emerge Tainted: G W 3.12.0 #80
[473233.777282] ffff88000599f6f8 ffff8800117bda58 ffffffff81489a80
ffff88004760e8e8
[473233.777286] ffff88000599f1c0 ffff8800117bdaf8 ffffffff81487577
ffff8800117bdaa8
[473233.777289] ffffffff810f8c0f ffff8800117bda88 ffffffff81006dc8
ffff8800117bda98
[473233.777293] Call Trace:
[473233.777305] [<ffffffff81489a80>] dump_stack+0x46/0x58
[473233.777310] [<ffffffff81487577>] dump_header.isra.9+0x6d/0x1cc
[473233.777315] [<ffffffff810f8c0f>] ? super_cache_count+0xa8/0xb8
[473233.777321] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[473233.777324] [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
[473233.777328] [<ffffffff8148f336>] ?
_raw_spin_unlock_irqrestore+0x47/0x62
[473233.777333] [<ffffffff812915d3>] ? ___ratelimit+0xcb/0xe8
[473233.777338] [<ffffffff810b2aa7>] oom_kill_process+0x70/0x2fd
[473233.777343] [<ffffffff81048775>] ? has_ns_capability_noaudit+0x12/0x19
[473233.777346] [<ffffffff8104878e>] ? has_capability_noaudit+0x12/0x14
[473233.777349] [<ffffffff810b31c6>] out_of_memory+0x31b/0x34e
[473233.777353] [<ffffffff810b72f0>] __alloc_pages_nodemask+0x65b/0x792
[473233.777358] [<ffffffff810e3c1b>] alloc_pages_vma+0xd0/0x10c
[473233.777361] [<ffffffff81003f69>] ?
__raw_callee_save_xen_pmd_val+0x11/0x1e
[473233.777365] [<ffffffff810cf685>] handle_mm_fault+0x6d4/0xd54
[473233.777371] [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
[473233.777374] [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[473233.777378] [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[473233.777382] [<ffffffff810676c7>] ? sched_clock_local+0x12/0x75
[473233.777386] [<ffffffff810a44b4>] ? __acct_update_integrals+0xb4/0xbf
[473233.777389] [<ffffffff810a4827>] ? acct_account_cputime+0x17/0x19
[473233.777392] [<ffffffff81067bc0>] ? account_user_time+0x67/0x92
[473233.777395] [<ffffffff810680b3>] ? vtime_account_user+0x4d/0x52
[473233.777398] [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
[473233.777401] [<ffffffff8148f9d8>] page_fault+0x28/0x30
[473233.777403] Mem-Info:
[473233.777405] Node 0 DMA per-cpu:
[473233.777408] CPU 0: hi: 0, btch: 1 usd: 0
[473233.777409] CPU 1: hi: 0, btch: 1 usd: 0
[473233.777411] CPU 2: hi: 0, btch: 1 usd: 0
[473233.777412] CPU 3: hi: 0, btch: 1 usd: 0
[473233.777413] Node 0 DMA32 per-cpu:
[473233.777415] CPU 0: hi: 186, btch: 31 usd: 103
[473233.777417] CPU 1: hi: 186, btch: 31 usd: 110
[473233.777419] CPU 2: hi: 186, btch: 31 usd: 175
[473233.777420] CPU 3: hi: 186, btch: 31 usd: 182
[473233.777421] Node 0 Normal per-cpu:
[473233.777423] CPU 0: hi: 0, btch: 1 usd: 0
[473233.777424] CPU 1: hi: 0, btch: 1 usd: 0
[473233.777426] CPU 2: hi: 0, btch: 1 usd: 0
[473233.777427] CPU 3: hi: 0, btch: 1 usd: 0
[473233.777433] active_anon:35740 inactive_anon:33812 isolated_anon:0
active_file:4672 inactive_file:11607 isolated_file:0
unevictable:0 dirty:4 writeback:0 unstable:0
free:2067 slab_reclaimable:3583 slab_unreclaimable:3524
mapped:3329 shmem:324 pagetables:2003 bounce:0
free_cma:0
[473233.777435] Node 0 DMA free:4200kB min:60kB low:72kB high:88kB
active_anon:264kB inactive_anon:456kB active_file:140kB
inactive_file:340kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:6176kB mlocked:0kB dirty:0kB
writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:96kB
slab_unreclaimable:112kB kernel_stack:24kB pagetables:24kB unstable:0kB
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:33270
all_unreclaimable? yes
[473233.777443] lowmem_reserve[]: 0 1036 1036 1036
[473233.777447] Node 0 DMA32 free:4060kB min:4084kB low:5104kB
high:6124kB active_anon:41256kB inactive_anon:33128kB active_file:8544kB
inactive_file:14312kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:1163264kB managed:165780kB mlocked:0kB
dirty:0kB writeback:0kB mapped:6428kB shmem:604kB
slab_reclaimable:9800kB slab_unreclaimable:12908kB kernel_stack:1832kB
pagetables:5924kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:152386 all_unreclaimable? yes
[473233.777454] lowmem_reserve[]: 0 0 0 0
[473233.777457] Node 0 Normal free:8kB min:0kB low:0kB high:0kB
active_anon:101440kB inactive_anon:101664kB active_file:10004kB
inactive_file:31776kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:393216kB managed:256412kB mlocked:0kB
dirty:16kB writeback:0kB mapped:6788kB shmem:692kB
slab_reclaimable:4436kB slab_unreclaimable:1076kB kernel_stack:136kB
pagetables:2064kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:368809 all_unreclaimable? yes
[473233.777464] lowmem_reserve[]: 0 0 0 0
[473233.777467] Node 0 DMA: 41*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (R)
1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB =
4196kB
[473233.777480] Node 0 DMA32: 1015*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4060kB
[473233.777490] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[473233.777498] 5018 total pagecache pages
[473233.777500] 16 pages in swap cache
[473233.777501] Swap cache stats: add 2829330, delete 2829314, find
344059/481859
[473233.777503] Free swap = 2096980kB
[473233.777503] Total swap = 2097148kB
[473233.794497] 557055 pages RAM
[473233.794500] 189326 pages reserved
[473233.794501] 544934 pages shared
[473233.794502] 358441 pages non-shared
[473233.794504] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[473233.794523] [ 6597] 0 6597 8156 252 20 0
-1000 udevd
[473233.794530] [ 7194] 0 7194 2232 137 10 0
0 metalog
[473233.794534] [ 7195] 0 7195 2223 31 10 3
0 metalog
[473233.794537] [ 7211] 0 7211 1064 35 8 0
0 acpid
[473233.794546] [ 7227] 702 7227 4922 183 14 0
0 dbus-daemon
[473233.794553] [ 7427] 0 7427 13630 179 29 15
0 rpcbind
[473233.794560] [ 7442] 0 7442 14743 332 32 0
0 rpc.statd
[473233.794569] [ 7472] 0 7472 6365 115 17 0
0 rpc.idmapd
[473233.794576] [ 7488] 0 7488 43602 349 40 0
0 cupsd
[473233.794583] [ 7512] 0 7512 14856 243 30 0
0 rpc.mountd
[473233.794592] [ 7552] 0 7552 148819 940 68 0
0 automount
[473233.794595] [ 7592] 0 7592 16006 233 32 0
-1000 sshd
[473233.794598] [ 7608] 0 7608 87672 2257 128 6
0 apache2
[473233.794601] [ 7633] 0 7633 521873 631 56 0
0 console-kit-dae
[473233.794604] [ 7713] 106 7713 15453 295 34 2
0 nrpe
[473233.794607] [ 7719] 986 7719 91303 798 41 0
0 polkitd
[473233.794610] [ 7757] 123 7757 7330 259 17 0
0 ntpd
[473233.794613] [ 7845] 0 7845 3583 94 12 0
0 master
[473233.794616] [ 7847] 207 7847 17745 311 38 0
0 qmgr
[473233.794619] [ 7861] 65534 7861 2101 21 9 19
0 rwhod
[473233.794622] [ 7864] 65534 7864 2101 99 9 0
0 rwhod
[473233.794625] [ 7876] 0 7876 48582 533 47 19
0 smbd
[473233.794628] [ 7881] 0 7881 44277 372 38 0
0 nmbd
[473233.794631] [ 7895] 0 7895 48646 621 45 18
0 smbd
[473233.794634] [ 7902] 2 7902 1078 39 8 4
0 slpd
[473233.794637] [ 7917] 0 7917 38452 1073 28 1
0 snmpd
[473233.794640] [ 7945] 0 7945 27552 58 9 0
0 cron
[473233.794648] [ 7993] 0 7993 201378 5432 63 39
0 nscd
[473233.794658] [ 8064] 0 8064 1060 28 7 0
0 agetty
[473233.794664] [ 8065] 0 8065 26507 29 9 0
0 agetty
[473233.794667] [ 8066] 0 8066 26507 29 9 0
0 agetty
[473233.794670] [ 8067] 0 8067 26507 28 9 0
0 agetty
[473233.794673] [ 8068] 0 8068 26507 28 8 0
0 agetty
[473233.794678] [ 8069] 0 8069 26507 30 9 0
0 agetty
[473233.794686] [ 8070] 0 8070 26507 30 9 0
0 agetty
[473233.794693] [ 8071] 0 8071 26507 30 9 0
0 agetty
[473233.794701] [ 8072] 0 8072 26507 28 9 0
0 agetty
[473233.794708] [ 8316] 0 8316 3736 83 11 6
0 ssh-agent
[473233.794712] [ 8341] 0 8341 3390 66 12 7
0 gpg-agent
[473233.794716] [ 2878] 81 2878 88431 2552 121 5
0 apache2
[473233.794718] [ 2879] 81 2879 88431 2552 121 5
0 apache2
[473233.794721] [ 2880] 81 2880 88431 2552 121 5
0 apache2
[473233.794724] [ 2881] 81 2881 88431 2552 121 5
0 apache2
[473233.794727] [ 2882] 81 2882 88431 2552 121 5
0 apache2
[473233.794734] [ 3523] 81 3523 88431 2552 121 5
0 apache2
[473233.794737] [30259] 1000 30259 3736 118 11 0
0 ssh-agent
[473233.794741] [30284] 1000 30284 3390 141 12 0
0 gpg-agent
[473233.794745] [21263] 207 21263 17703 771 39 1
0 pickup
[473233.794748] [21663] 0 21663 30743 228 16 0
0 cron
[473233.794751] [21665] 0 21665 2980 392 12 0
0 gentoosync.sh
[473233.794755] [22158] 0 22158 3181 273 12 0
0 sendmail
[473233.794757] [22159] 0 22159 77646 54920 158 0
0 emerge
[473233.794760] [22160] 0 22160 1068 85 8 0
0 tail
[473233.794764] [22161] 0 22161 3173 277 11 0
0 postdrop
[473233.794768] Out of memory: Kill process 22159 (emerge) score 57 or
sacrifice child
[473233.794771] Killed process 22159 (emerge) total-vm:310584kB,
anon-rss:215840kB, file-rss:3840kB
^ permalink raw reply [flat|nested] 42+ messages in thread
end of thread, other threads:[~2014-02-03 14:00 UTC | newest]
Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-12-09 17:50 Kernel 3.11 / 3.12 OOM killer and Xen ballooning James Dingwall
2013-12-09 21:48 ` Konrad Rzeszutek Wilk
2013-12-10 14:52 ` James Dingwall
2013-12-10 15:27 ` Konrad Rzeszutek Wilk
2013-12-11 7:22 ` Bob Liu
2013-12-11 9:25 ` James Dingwall
2013-12-11 9:54 ` Bob Liu
2013-12-11 10:16 ` James Dingwall
2013-12-11 16:30 ` James Dingwall
2013-12-12 1:03 ` Bob Liu
2013-12-13 16:59 ` James Dingwall
2013-12-17 6:11 ` Bob Liu
2013-12-18 12:04 ` Bob Liu
2013-12-19 19:08 ` James Dingwall
2013-12-20 3:17 ` Bob Liu
2013-12-20 12:22 ` James Dingwall
2013-12-26 8:42 ` James Dingwall
2014-01-02 6:25 ` Bob Liu
2014-01-07 9:21 ` James Dingwall
2014-01-09 10:48 ` Bob Liu
2014-01-09 10:54 ` James Dingwall
2014-01-09 11:04 ` James Dingwall
2014-01-15 8:49 ` James Dingwall
2014-01-15 14:41 ` Bob Liu
2014-01-15 16:35 ` James Dingwall
2014-01-16 1:22 ` Bob Liu
2014-01-16 10:52 ` James Dingwall
2014-01-28 17:15 ` James Dingwall
2014-01-29 14:35 ` Bob Liu
2014-01-29 14:45 ` James Dingwall
2014-01-31 16:56 ` Konrad Rzeszutek Wilk
2014-02-03 9:49 ` Daniel Kiper
2014-02-03 10:30 ` Konrad Rzeszutek Wilk
2014-02-03 11:20 ` James Dingwall
2014-02-03 14:00 ` Daniel Kiper
2013-12-10 8:16 ` Jan Beulich
2013-12-10 14:01 ` James Dingwall
2013-12-10 14:25 ` Jan Beulich
2013-12-10 14:52 ` James Dingwall
2013-12-10 14:59 ` Jan Beulich
2013-12-10 15:16 ` James Dingwall
-- strict thread matches above, loose matches on Subject: below --
2013-11-21 11:28 James Dingwall
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.