3.4-rc4 oom killer out of control.

* 3.4-rc4 oom killer out of control.
@ 2012-04-26 19:35 ` Dave Jones
  0 siblings, 0 replies; 36+ messages in thread
From: Dave Jones @ 2012-04-26 19:35 UTC (permalink / raw)
  To: linux-mm; +Cc: Linux Kernel

On a test machine that was running my system call fuzzer, I just saw
the oom killer take out everything but the process that was doing all
the memory exhausting.

Partial logs below. The machine locked up completely (even capslock wouldn't work).
The console had logs up to 5041.xxxxxx before the wedge, but they never made it to disk.
It was just more of the same below..

Note that the trinity processes have largest RSS, yet seem immune to getting killed.

	Dave

Apr 26 13:56:45 dhcp-189-232 kernel: [ 4959.198003] modprobe invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.198349] modprobe cpuset=/ mems_allowed=0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.198705] Pid: 14950, comm: modprobe Not tainted 3.4.0-rc4+ #55
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.199406] Call Trace:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.200214]  [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.201084]  [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.201928]  [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.202814]  [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.203687]  [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.204539]  [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.205416]  [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.206277]  [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.207142]  [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.207999]  [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.208892]  [<ffffffff8117df8f>] __do_fault+0x6f/0x540
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.209752]  [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.210633]  [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.211509]  [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.212384]  [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.213475]  [<ffffffff81335a1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.214137]  [<ffffffff816af019>] ? retint_swapgs+0x13/0x1b
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.215018]  [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.215886]  [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.216722] Mem-Info:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.217608] Node 0 DMA per-cpu:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.218500] CPU    0: hi:    0, btch:   1 usd:   0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.219400] CPU    1: hi:    0, btch:   1 usd:   0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.220292] CPU    2: hi:    0, btch:   1 usd:   0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.221186] CPU    3: hi:    0, btch:   1 usd:   0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.222067] Node 0 DMA32 per-cpu:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.222937] CPU    0: hi:  186, btch:  31 usd:  16
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.223833] CPU    1: hi:  186, btch:  31 usd:   7
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.224692] CPU    2: hi:  186, btch:  31 usd:   1
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.225554] CPU    3: hi:  186, btch:  31 usd:   9
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.226408] Node 0 Normal per-cpu:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.227253] CPU    0: hi:  186, btch:  31 usd:   2
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.228095] CPU    1: hi:  186, btch:  31 usd:  65
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.228918] CPU    2: hi:  186, btch:  31 usd:  33
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.229767] CPU    3: hi:  186, btch:  31 usd:  21
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230600] active_anon:1421942 inactive_anon:313560 isolated_anon:0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230604]  active_file:88 inactive_file:120 isolated_file:70
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230608]  unevictable:3038 dirty:7 writeback:0 unstable:0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230610]  free:25574 slab_reclaimable:14238 slab_unreclaimable:101544
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230611]  mapped:232 shmem:86 pagetables:89711 bounce:0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.234745] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.237548] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.238522] Node 0 DMA32 free:46340kB min:27252kB low:34064kB high:40876kB active_anon:2330464kB inactive_anon:582676kB active_file:16kB inactive_file:44kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:112kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132232kB kernel_stack:2272kB pagetables:133664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? yes
Apr 26 13:56:50 dhcp-189-232 kernel: [ 4959.241635] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:09 dhcp-189-232 kernel: [ 4959.242701] Node 0 Normal free:40080kB min:40196kB low:50244kB high:60292kB active_anon:3357304kB inactive_anon:671564kB active_file:336kB inactive_file:436kB unevictable:7404kB isolated(anon):0kB isolated(file):280kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:816kB shmem:240kB slab_reclaimable:54120kB slab_unreclaimable:273912kB kernel_stack:4528kB pagetables:225180kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:970 all_unreclaimable? yes
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.246216] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.247423] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.248684] Node 0 DMA32: 217*4kB 152*8kB 158*16kB 232*32kB 134*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46340kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.249873] Node 0 Normal: 475*4kB 405*8kB 272*16kB 352*32kB 144*64kB 40*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 39956kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.251173] 2037 total pagecache pages
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.252403] 1692 pages in swap cache
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.253630] Swap cache stats: add 1828870, delete 1827178, find 493465/493919
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.254887] Free swap  = 0kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.256126] Total swap = 1023996kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.347525] 2097136 pages RAM
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.348055] 59776 pages reserved
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.348865] 2554149 pages shared
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.349991] 2007673 pages non-shared
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.351168] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.352382] [  353]     0   353     4425       23   2     -17         -1000 udevd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.353575] [  355]     0   355     5802        1   2       0             0 systemd-stdout-
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.354796] [  737]     0   737     4766        1   3       0             0 smartd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.356016] [  751]     0   751    29591       24   0       0             0 crond
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.357246] [  755]     0   755     1616        1   1       0             0 acpid
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.358447] [  756]     0   756    67222      107   0       0             0 NetworkManager
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.359656] [  770]     0   770     7083        1   0       0             0 systemd-logind
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.360879] [  773]    70   773     7005       50   3       0             0 avahi-daemon
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.362082] [  776]    70   776     6972        4   0       0             0 avahi-daemon
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.363301] [  778]     0   778     1612        1   0       0             0 mcelog
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.364526] [  783]   994   783     4893       24   0       0             0 chronyd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.365748] [  787]    81   787     5472       69   0     -13          -900 dbus-daemon
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.366862] [  790]     0   790    29064       39   0       0             0 ksmtuned
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.368085] [  797]     0   797    46610       31   1     -13          -900 polkitd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.369305] [  802]     0   802    60782       52   1       0             0 rsyslogd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.370527] [  815]     0   815    18861        1   0     -17         -1000 sshd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.371756] [  816]     0   816     8512        1   2       0             0 rpc.idmapd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.372995] [  819]     0   819     4792       17   2       0             0 rpcbind
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.374229] [  824]    29   824     6938        1   0       0             0 rpc.statd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.375462] [  837]     0   837    27232        1   3       0             0 agetty
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.376695] [  842]     0   842    21549        1   2       0             0 dhclient
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.377938] [ 1170]  1000  1170     6134       81   0       0             0 tmux
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.379192] [ 1171]  1000  1171    29914        1   2       0             0 bash
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.380455] [ 1262]  1000  1262     4022      125   3       0             0 trinity
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.381709] [ 1325]     0  1325     4424       57   2     -17         -1000 udevd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.382928] [15065]     0 15065     3725       87   3       0             0 anacron
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.384164] [19925]     0 19925     4424        1   3     -17         -1000 udevd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.385449] [13786]     0 13786    26704       68   2       0             0 sleep
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.386748] [14941]     0 14941     1629       87   0       0             0 modprobe
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.388042] [14944]     0 14944     1628       63   0       0             0 modprobe
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.389325] [14945]  1000 14945     4418      639   3       0             0 trinity
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.390618] [14950]     0 14950      106        3   2       0             0 modprobe
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.391913] [14952]  1000 14952     4022      203   0       0             0 trinity
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.393252] [14955]  1000 14955     4022      124   0       0             0 trinity
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.394560] Out of memory: Kill process 355 (systemd-stdout-) score 1 or sacrifice child
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.395847] Killed process 355 (systemd-stdout-) total-vm:23208kB, anon-rss:0kB, file-rss:4kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.358425] systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.358956] systemd cpuset=/ mems_allowed=0
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.359956] Pid: 1, comm: systemd Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.361271] Call Trace:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.362557]  [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.363827]  [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.365052]  [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.366208]  [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.367393]  [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.368532]  [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.369673]  [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.370780]  [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.371855]  [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.372910]  [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.373935]  [<ffffffff8117df8f>] __do_fault+0x6f/0x540
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.374939]  [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.375917]  [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.376883]  [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.377826]  [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.378765]  [<ffffffff8120f046>] ? sys_epoll_wait+0x96/0x470
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.379700]  [<ffffffff816b6b15>] ? sysret_check+0x22/0x5d
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.380620]  [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.381549]  [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.382409] Mem-Info:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.383329] Node 0 DMA per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.384259] CPU    0: hi:    0, btch:   1 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.385200] CPU    1: hi:    0, btch:   1 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.386123] CPU    2: hi:    0, btch:   1 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.387024] CPU    3: hi:    0, btch:   1 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.387898] Node 0 DMA32 per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.388802] CPU    0: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.389691] CPU    1: hi:  186, btch:  31 usd:  30
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.390587] CPU    2: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.391466] CPU    3: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.392334] Node 0 Normal per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.393198] CPU    0: hi:  186, btch:  31 usd:  75
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.394078] CPU    1: hi:  186, btch:  31 usd:  30
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.394934] CPU    2: hi:  186, btch:  31 usd:  22
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.395810] CPU    3: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396663] active_anon:1421870 inactive_anon:313545 isolated_anon:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396664]  active_file:103 inactive_file:50 isolated_file:64
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396665]  unevictable:3038 dirty:7 writeback:0 unstable:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396666]  free:25649 slab_reclaimable:14238 slab_unreclaimable:101538
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396668]  mapped:237 shmem:86 pagetables:89658 bounce:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.400859] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.403671] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.404653] Node 0 DMA32 free:46192kB min:27252kB low:34064kB high:40876kB active_anon:2330460kB inactive_anon:582672kB active_file:0kB inactive_file:96kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:108kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132232kB kernel_stack:2272kB pagetables:133664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.407764] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.408824] Node 0 Normal free:40528kB min:40196kB low:50244kB high:60292kB active_anon:3357020kB inactive_anon:671508kB active_file:412kB inactive_file:104kB unevictable:7404kB isolated(anon):0kB isolated(file):256kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:840kB shmem:240kB slab_reclaimable:54120kB slab_unreclaimable:273888kB kernel_stack:4528kB pagetables:224968kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:23 all_unreclaimable? no
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.412250] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.413420] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.414664] Node 0 DMA32: 226*4kB 155*8kB 160*16kB 229*32kB 134*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46336kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.415839] Node 0 Normal: 542*4kB 428*8kB 273*16kB 353*32kB 145*64kB 40*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40520kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.417112] 2026 total pagecache pages
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.418352] 1685 pages in swap cache
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.419591] Swap cache stats: add 1828918, delete 1827233, find 493469/493923
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.420839] Free swap  = 0kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.422068] Total swap = 1023996kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.502902] 2097136 pages RAM
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.503469] 59776 pages reserved
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.504253] 2553433 pages shared
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.505446] 2007548 pages non-shared
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.506614] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.507848] [  353]     0   353     4425       23   2     -17         -1000 udevd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.509055] [  737]     0   737     4766        1   3       0             0 smartd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.510247] [  751]     0   751    29591       24   0       0             0 crond
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.511462] [  755]     0   755     1616        1   1       0             0 acpid
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.512665] [  756]     0   756    67222      107   1       0             0 NetworkManager
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.513879] [  770]     0   770     7083        1   0       0             0 systemd-logind
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.515077] [  773]    70   773     7005       50   3       0             0 avahi-daemon
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.516211] [  776]    70   776     6972        4   0       0             0 avahi-daemon
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.517431] [  778]     0   778     1612        1   0       0             0 mcelog
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.518630] [  783]   994   783     4893       24   0       0             0 chronyd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.519848] [  787]    81   787     5472       69   0     -13          -900 dbus-daemon
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.521071] [  790]     0   790    29064       39   0       0             0 ksmtuned
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.522285] [  797]     0   797    46610       31   2     -13          -900 polkitd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.523503] [  802]     0   802    60782       52   1       0             0 rsyslogd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.524745] [  815]     0   815    18861        1   0     -17         -1000 sshd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.525972] [  816]     0   816     8512        1   2       0             0 rpc.idmapd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.527201] [  819]     0   819     4792       17   2       0             0 rpcbind
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.528428] [  824]    29   824     6938        1   0       0             0 rpc.statd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.529659] [  837]     0   837    27232        1   3       0             0 agetty
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.530899] [  842]     0   842    21549        1   2       0             0 dhclient
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.532097] [ 1170]  1000  1170     6134       78   2       0             0 tmux
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.533306] [ 1171]  1000  1171    29914        1   2       0             0 bash
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.534547] [ 1262]  1000  1262     4022      125   1       0             0 trinity
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.535816] [ 1325]     0  1325     4424       57   2     -17         -1000 udevd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.537076] [15065]     0 15065     3725       87   3       0             0 anacron
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.538337] [19925]     0 19925     4424        1   3     -17         -1000 udevd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.539605] [13786]     0 13786    26704       68   2       0             0 sleep
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.540888] [14941]     0 14941     1629       87   1       0             0 modprobe
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.542179] [14944]     0 14944     1628       63   1       0             0 modprobe
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.543473] [14945]  1000 14945     4418      652   2       0             0 trinity
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.544756] [14950]     0 14950      106        7   3       0             0 modprobe
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.546051] [14955]  1000 14955     4022      124   2       0             0 trinity
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.547353] Out of memory: Kill process 737 (smartd) score 1 or sacrifice child
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.548644] Killed process 737 (smartd) total-vm:19064kB, anon-rss:0kB, file-rss:4kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.791375] trinity invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.791871] trinity cpuset=/ mems_allowed=0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.792948] Pid: 14972, comm: trinity Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.794230] Call Trace:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.795559]  [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.796866]  [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.798155]  [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.799432]  [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.800679]  [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.801892]  [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.803092]  [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.804244]  [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.805393]  [<ffffffff81181300>] handle_pte_fault+0x6a0/0xa10
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.806510]  [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807590]  [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807596]  [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807602]  [<ffffffff811b4f68>] do_huge_pmd_anonymous_page+0xc8/0x380
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807607]  [<ffffffff81181976>] handle_mm_fault+0x146/0x2f0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807610]  [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807614]  [<ffffffff811871d6>] ? do_brk+0x246/0x360
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807619]  [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807623]  [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807627] Mem-Info:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807630] Node 0 DMA per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807633] CPU    0: hi:    0, btch:   1 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807637] CPU    1: hi:    0, btch:   1 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807639] CPU    2: hi:    0, btch:   1 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807641] CPU    3: hi:    0, btch:   1 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807643] Node 0 DMA32 per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807646] CPU    0: hi:  186, btch:  31 usd:   1
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807648] CPU    1: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807651] CPU    2: hi:  186, btch:  31 usd:   3
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807653] CPU    3: hi:  186, btch:  31 usd:   1
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807655] Node 0 Normal per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807658] CPU    0: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807660] CPU    1: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807662] CPU    2: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807664] CPU    3: hi:  186, btch:  31 usd:   0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807670] active_anon:1421827 inactive_anon:313613 isolated_anon:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807672]  active_file:63 inactive_file:73 isolated_file:110
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807673]  unevictable:3038 dirty:7 writeback:0 unstable:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807674]  free:25681 slab_reclaimable:14174 slab_unreclaimable:101572
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807675]  mapped:229 shmem:86 pagetables:89725 bounce:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807677] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807687] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807692] Node 0 DMA32 free:46508kB min:27252kB low:34064kB high:40876kB active_anon:2330216kB inactive_anon:582636kB active_file:24kB inactive_file:28kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2296kB pagetables:133680kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:83 all_unreclaimable? yes
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807702] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:14 dhcp-189-232 systemd[1]: systemd-stdout-syslog-bridge.service: main process exited, code=killed, status=9
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807706] Node 0 Normal free:40340kB min:40196kB low:50244kB high:60292kB active_anon:3357092kB inactive_anon:671816kB active_file:228kB inactive_file:264kB unevictable:7404kB isolated(anon):0kB isolated(file):440kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4576kB pagetables:225220kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:770 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807717] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807720] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807731] Node 0 DMA32: 219*4kB 156*8kB 166*16kB 230*32kB 135*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46508kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807742] Node 0 Normal: 517*4kB 412*8kB 266*16kB 350*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40340kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807752] 1526 total pagecache pages
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807754] 1249 pages in swap cache
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807757] Swap cache stats: add 1829703, delete 1828454, find 493950/494458
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807759] Free swap  = 0kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807760] Total swap = 1023996kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.924517] 2097136 pages RAM
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.925073] 59776 pages reserved
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.925829] 2551230 pages shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.927017] 2007729 pages non-shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.928103] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.929299] [  353]     0   353     4425       23   2     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.930485] [  751]     0   751    29591       24   0       0             0 crond
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.931669] [  755]     0   755     1616        1   1       0             0 acpid
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.932860] [  756]     0   756    67222      107   1       0             0 NetworkManager
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.934065] [  770]     0   770     7083        1   0       0             0 systemd-logind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.935251] [  773]    70   773     7005       50   3       0             0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.936454] [  776]    70   776     6972        4   0       0             0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.937669] [  778]     0   778     1612        1   0       0             0 mcelog
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.938838] [  783]   994   783     4893       24   0       0             0 chronyd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.940037] [  787]    81   787     5472       69   0     -13          -900 dbus-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.941223] [  790]     0   790    29064       39   0       0             0 ksmtuned
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.942428] [  797]     0   797    46610       31   0     -13          -900 polkitd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.943620] [  802]     0   802    60782       59   1       0             0 rsyslogd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.944740] [  815]     0   815    18861        1   0     -17         -1000 sshd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.945960] [  816]     0   816     8512        1   2       0             0 rpc.idmapd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.947173] [  819]     0   819     4792       17   3       0             0 rpcbind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.948373] [  824]    29   824     6938        1   0       0             0 rpc.statd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.949594] [  837]     0   837    27232        1   3       0             0 agetty
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.950802] [  842]     0   842    21549        1   2       0             0 dhclient
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.952021] [ 1170]  1000  1170     6133       77   0       0             0 tmux
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.953241] [ 1171]  1000  1171    29914        1   2       0             0 bash
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.954555] [ 1262]  1000  1262     4022      125   2       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.955715] [ 1325]     0  1325     4424       57   2     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.956971] [15065]     0 15065     3725       87   3       0             0 anacron
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.958231] [19925]     0 19925     4424        1   3     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.959476] [14944]     0 14944     1629       86   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.960698] [14950]     0 14950     1595       57   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.961949] [14954]     0 14954     1595       36   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.963230] [14958]     0 14958     2810       19   2       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.964494] [14961]  1000 14961     4286      459   1       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.965773] [14965]     0 14965     1618       15   0       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.967069] [14966]     0 14966     2290       19   1       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.968363] [14967]     0 14967     1621       35   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.969669] [14970]     0 14970     1595       58   0       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.970972] [14972]  1000 14972     4220      456   3       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.972269] [14975]     0 14975      104        1   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.973563] Out of memory: Kill process 751 (crond) score 1 or sacrifice child
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.974863] Killed process 751 (crond) total-vm:118364kB, anon-rss:92kB, file-rss:4kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.036332] systemd-cgroups invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.036811] systemd-cgroups cpuset=/ mems_allowed=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.037772] Pid: 14958, comm: systemd-cgroups Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.039032] Call Trace:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.040280]  [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.041507]  [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.042610]  [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.043793]  [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.044947]  [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.046063]  [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.047156]  [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.048226]  [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.049259]  [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.050282]  [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.051270]  [<ffffffff8117df8f>] __do_fault+0x6f/0x540
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.052234]  [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.053175]  [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.054132]  [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.055081]  [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.056015]  [<ffffffff811c1453>] ? sys_close+0x43/0x1a0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.056940]  [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.057895]  [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.058855] Mem-Info:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.059727] Node 0 DMA per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.060674] CPU    0: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.061619] CPU    1: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.062547] CPU    2: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.063456] CPU    3: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.064373] Node 0 DMA32 per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.065269] CPU    0: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.066178] CPU    1: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.067069] CPU    2: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.067947] CPU    3: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.068854] Node 0 Normal per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.069730] CPU    0: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.070587] CPU    1: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.071458] CPU    2: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.072321] CPU    3: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073165] active_anon:1421858 inactive_anon:313540 isolated_anon:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073166]  active_file:94 inactive_file:65 isolated_file:151
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073167]  unevictable:3038 dirty:9 writeback:0 unstable:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073170]  free:25544 slab_reclaimable:14174 slab_unreclaimable:101572
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073172]  mapped:229 shmem:86 pagetables:89708 bounce:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.077353] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.080149] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.081123] Node 0 DMA32 free:46240kB min:27252kB low:34064kB high:40876kB active_anon:2330228kB inactive_anon:582636kB active_file:28kB inactive_file:36kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:8kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2288kB pagetables:133684kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5 all_unreclaimable? no
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.084201] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.085253] Node 0 Normal free:40060kB min:40196kB low:50244kB high:60292kB active_anon:3357204kB inactive_anon:671524kB active_file:348kB inactive_file:224kB unevictable:7404kB isolated(anon):0kB isolated(file):604kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4600kB pagetables:225148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? no
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.088669] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.089845] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.091098] Node 0 DMA32: 204*4kB 143*8kB 161*16kB 231*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46360kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.092319] Node 0 Normal: 565*4kB 415*8kB 265*16kB 350*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40540kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.093613] 1616 total pagecache pages
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.094829] 1336 pages in swap cache
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.096071] Swap cache stats: add 1829817, delete 1828481, find 493950/494458
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.097332] Free swap  = 0kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.098596] Total swap = 1023996kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.205067] 2097136 pages RAM
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.205470] 59776 pages reserved
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206413] 2549841 pages shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206415] 2007690 pages non-shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206418] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206444] [  353]     0   353     4425       23   2     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206459] [  755]     0   755     1616        1   1       0             0 acpid
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206465] [  756]     0   756    67222      107   1       0             0 NetworkManager
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206470] [  770]     0   770     7083        1   0       0             0 systemd-logind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206475] [  773]    70   773     7005       50   3       0             0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206480] [  776]    70   776     6972        4   0       0             0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206485] [  778]     0   778     1612        1   0       0             0 mcelog
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206490] [  783]   994   783     4893       24   0       0             0 chronyd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206495] [  787]    81   787     5472       69   0     -13          -900 dbus-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206500] [  790]     0   790    29064       39   2       0             0 ksmtuned
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206505] [  797]     0   797    46610       31   1     -13          -900 polkitd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206510] [  802]     0   802    60782       59   1       0             0 rsyslogd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206515] [  815]     0   815    18861        1   0     -17         -1000 sshd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206520] [  816]     0   816     8512        1   2       0             0 rpc.idmapd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206524] [  819]     0   819     4792       17   3       0             0 rpcbind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206529] [  824]    29   824     6938        1   0       0             0 rpc.statd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206534] [  837]     0   837    27232        1   3       0             0 agetty
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206539] [  842]     0   842    21549        1   2       0             0 dhclient
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206544] [ 1170]  1000  1170     6133       77   2       0             0 tmux
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206549] [ 1171]  1000  1171    29914        1   2       0             0 bash
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206554] [ 1262]  1000  1262     4022      125   3       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206559] [ 1325]     0  1325     4424       57   2     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206565] [15065]     0 15065     3725       87   3       0             0 anacron
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206573] [19925]     0 19925     4424        1   3     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206581] [14944]     0 14944     1629       86   3       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206585] [14950]     0 14950     1628       63   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206590] [14954]     0 14954     1595       36   3       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206596] [14958]     0 14958     2810       20   0       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206600] [14961]  1000 14961     4286      459   1       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206606] [14965]     0 14965     1618       15   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206610] [14966]     0 14966     2290       19   3       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206615] [14967]     0 14967     1621       35   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206620] [14970]     0 14970     1628       62   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206625] [14972]  1000 14972     4286      529   3       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206631] [14975]     0 14975      106        8   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206636] Out of memory: Kill process 755 (acpid) score 1 or sacrifice child
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206640] Killed process 755 (acpid) total-vm:6464kB, anon-rss:0kB, file-rss:4kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.882053] trinity invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.882827] trinity cpuset=/ mems_allowed=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.883513] Pid: 14972, comm: trinity Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.884768] Call Trace:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.886025]  [<ffffffff816ae775>] ? _raw_spin_unlock+0x55/0x60
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.887232]  [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.888433]  [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.889579]  [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.890703]  [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.891787]  [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.892873]  [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.893940]  [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.894969]  [<ffffffff81181300>] handle_pte_fault+0x6a0/0xa10
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.895959]  [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.896952]  [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.897917]  [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.898874]  [<ffffffff811871d6>] ? do_brk+0x246/0x360
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.899797]  [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.900733]  [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.901645] Mem-Info:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.902542] Node 0 DMA per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.903447] CPU    0: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.904352] CPU    1: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.905264] CPU    2: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.906166] CPU    3: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.907061] Node 0 DMA32 per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.907881] CPU    0: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.908779] CPU    1: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.909655] CPU    2: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.910525] CPU    3: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.911379] Node 0 Normal per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.912233] CPU    0: hi:  186, btch:  31 usd:   1
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.913091] CPU    1: hi:  186, btch:  31 usd:   4
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.913930] CPU    2: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.914792] CPU    3: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915639] active_anon:1421866 inactive_anon:313517 isolated_anon:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915641]  active_file:86 inactive_file:127 isolated_file:114
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915642]  unevictable:3038 dirty:9 writeback:0 unstable:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915643]  free:25566 slab_reclaimable:14174 slab_unreclaimable:101572
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915644]  mapped:229 shmem:86 pagetables:89704 bounce:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.919846] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.922609] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.923570] Node 0 DMA32 free:46384kB min:27252kB low:34064kB high:40876kB active_anon:2330352kB inactive_anon:582628kB active_file:0kB inactive_file:64kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:8kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2280kB pagetables:133684kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:228 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.926571] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.927617] Node 0 Normal free:40004kB min:40196kB low:50244kB high:60292kB active_anon:3357112kB inactive_anon:671440kB active_file:344kB inactive_file:444kB unevictable:7404kB isolated(anon):0kB isolated(file):456kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4600kB pagetables:225132kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.931017] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.932183] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.933420] Node 0 DMA32: 206*4kB 145*8kB 161*16kB 231*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46384kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.934698] Node 0 Normal: 506*4kB 408*8kB 263*16kB 351*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40248kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.935988] 1630 total pagecache pages
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.937219] 1286 pages in swap cache
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.938456] Swap cache stats: add 1829855, delete 1828569, find 493952/494460
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.939701] Free swap  = 0kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.940881] Total swap = 1023996kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.024043] 2097136 pages RAM
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.024445] 59776 pages reserved
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.025432] 2548364 pages shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.026651] 2007726 pages non-shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.027872] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.029123] [  353]     0   353     4425       23   2     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.030346] [  756]     0   756    67222      111   0       0             0 NetworkManager
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.031539] [  770]     0   770     7083        1   0       0             0 systemd-logind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.032769] [  773]    70   773     7005       50   3       0             0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.034004] [  776]    70   776     6972        4   0       0             0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.035225] [  778]     0   778     1612        1   0       0             0 mcelog
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.036452] [  783]   994   783     4893       24   0       0             0 chronyd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.037654] [  787]    81   787     5472       69   0     -13          -900 dbus-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.038878] [  790]     0   790    29064       39   1       0             0 ksmtuned
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.040105] [  797]     0   797    46610       39   2     -13          -900 polkitd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.041252] [  802]     0   802    60782       59   1       0             0 rsyslogd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.042478] [  815]     0   815    18861        1   0     -17         -1000 sshd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.043706] [  816]     0   816     8512        1   2       0             0 rpc.idmapd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.044954] [  819]     0   819     4792       17   3       0             0 rpcbind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.046187] [  824]    29   824     6938        1   0       0             0 rpc.statd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.047426] [  837]     0   837    27232        1   3       0             0 agetty
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.048669] [  842]     0   842    21549        1   2       0             0 dhclient
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.049899] [ 1170]  1000  1170     6133       77   2       0             0 tmux
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.051132] [ 1171]  1000  1171    29914        1   2       0             0 bash
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.052362] [ 1262]  1000  1262     4022      125   2       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.053596] [ 1325]     0  1325     4424       57   2     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.054842] [15065]     0 15065     3725       87   3       0             0 anacron
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.056115] [19925]     0 19925     4424        1   3     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.057359] [14944]     0 14944     1629       86   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.058602] [14950]     0 14950     1628       65   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.059853] [14954]     0 14954     1595       36   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.061136] [14958]     0 14958     2810       20   1       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.062414] [14961]  1000 14961     4286      459   0       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.063716] [14965]     0 14965     1618       16   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.065015] [14966]     0 14966     2290       19   2       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.066295] [14967]     0 14967     1595       37   0       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.067591] [14970]     0 14970     1628       65   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.068897] [14972]  1000 14972     4286      521   3       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.070217] [14975]     0 14975      106        8   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.071527] [14979]  1000 14979     4022      125   1       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.072832] Out of memory: Kill process 756 (NetworkManager) score 1 or sacrifice child
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.074110] Killed process 842 (dhclient) total-vm:86196kB, anon-rss:0kB, file-rss:4kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.242452] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.242915] systemd-cgroups cpuset=/ mems_allowed=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.243952] Pid: 14958, comm: systemd-cgroups Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.245244] Call Trace:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.246511]  [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.247766]  [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.248996]  [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.250204]  [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.251350]  [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.252505]  [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.253627]  [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.254720]  [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.255793]  [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.256780]  [<ffffffff8117e248>] __do_fault+0x328/0x540
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.257795]  [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.258789]  [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.259769]  [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.260734]  [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.261697]  [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.262650]  [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.263586] Mem-Info:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.264522] Node 0 DMA per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.265473] CPU    0: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.266423] CPU    1: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.267372] CPU    2: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.268309] CPU    3: hi:    0, btch:   1 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.269234] Node 0 DMA32 per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.270149] CPU    0: hi:  186, btch:  31 usd:   2
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.271066] CPU    1: hi:  186, btch:  31 usd:   1
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.271957] CPU    2: hi:  186, btch:  31 usd:   0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.272856] CPU    3: hi:  186, btch:  31 usd:  30
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.273681] Node 0 Normal per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.274571] CPU    0: hi:  186, btch:  31 usd: 100
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.275466] CPU    1: hi:  186, btch:  31 usd:  45
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.276351] CPU    2: hi:  186, btch:  31 usd:  87
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.277222] CPU    3: hi:  186, btch:  31 usd:  41
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278087] active_anon:1421461 inactive_anon:313478 isolated_anon:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278088]  active_file:185 inactive_file:290 isolated_file:69
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278090]  unevictable:3038 dirty:0 writeback:0 unstable:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278091]  free:25550 slab_reclaimable:14134 slab_unreclaimable:101600
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278092]  mapped:305 shmem:86 pagetables:89707 bounce:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.282445] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.285312] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.286303] Node 0 DMA32 free:46272kB min:27252kB low:34064kB high:40876kB active_anon:2330240kB inactive_anon:582708kB active_file:24kB inactive_file:20kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2800kB slab_unreclaimable:132256kB kernel_stack:2296kB pagetables:133744kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:296 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.289453] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.290460] Node 0 Normal free:40052kB min:40196kB low:50244kB high:60292kB active_anon:3355604kB inactive_anon:671204kB active_file:716kB inactive_file:1140kB unevictable:7404kB isolated(anon):0kB isolated(file):276kB present:4902912kB mlocked:7404kB dirty:0kB writeback:0kB mapped:1116kB shmem:240kB slab_reclaimable:53736kB slab_unreclaimable:274112kB kernel_stack:4616kB pagetables:225084kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4068 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.295125] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.296327] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.297596] Node 0 DMA32: 179*4kB 143*8kB 158*16kB 229*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46148kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.298883] Node 0 Normal: 449*4kB 360*8kB 263*16kB 353*32kB 150*64kB 42*128kB 10*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 39764kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.300186] 2587 total pagecache pages
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.301448] 1924 pages in swap cache
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.302721] Swap cache stats: add 1830963, delete 1829039, find 494366/494878
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.303994] Free swap  = 0kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.305280] Total swap = 1023996kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.371694] 2097136 pages RAM
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.372170] 59776 pages reserved
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.373070] 2548580 pages shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.374252] 2007488 pages non-shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.375461] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.376713] [  353]     0   353     4425       23   2     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.377924] [  756]     0   756    67222      108   1       0             0 NetworkManager
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.379160] [  770]     0   770     7083        1   0       0             0 systemd-logind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.380398] [  773]    70   773     7005       50   3       0             0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.381640] [  776]    70   776     6972        4   0       0             0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.382883] [  778]     0   778     1612        1   0       0             0 mcelog
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.384116] [  783]   994   783     4893       24   0       0             0 chronyd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.385370] [  787]    81   787     5472       69   0     -13          -900 dbus-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.386595] [  790]     0   790    29064       41   1       0             0 ksmtuned
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.387830] [  797]     0   797    46610       34   2     -13          -900 polkitd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.389066] [  802]     0   802    60782       63   1       0             0 rsyslogd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.390244] [  815]     0   815    18861        1   0     -17         -1000 sshd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.391487] [  816]     0   816     8512        1   2       0             0 rpc.idmapd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.392739] [  819]     0   819     4792       17   3       0             0 rpcbind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.393980] [  824]    29   824     6938        1   0       0             0 rpc.statd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.395227] [  837]     0   837    27232        1   3       0             0 agetty
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.396486] [ 1170]  1000  1170     6133      112   3       0             0 tmux
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.397727] [ 1171]  1000  1171    29914        1   2       0             0 bash
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.398971] [ 1262]  1000  1262     4022      131   1       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.400219] [ 1325]     0  1325     4424       57   2     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.401493] [15065]     0 15065     3725       87   3       0             0 anacron
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.402759] [19925]     0 19925     4424        1   3     -17         -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.404053] [14950]     0 14950     1628       89   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.405370] [14954]     0 14954     1595       56   3       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.406583] [14958]     0 14958     3763       32   2       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.407859] [14965]     0 14965     1621       42   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.409294] [14966]     0 14966     2810       43   3       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.410460] [14967]     0 14967     1595       59   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.411770] [14970]     0 14970     1628       89   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.413072] [14975]     0 14975     1621       55   1       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.414379] [14977]     0 14977     1750       36   2       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.415685] [14979]  1000 14979     4022      271   1       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.417022] [14980]     0 14980     1750       36   0       0             0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.418337] [14982]     0 14982      666       31   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.419660] [14986]  1000 14986     4319      557   2       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.420971] [14987]  1000 14987     4022      269   0       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.422278] [14988]  1000 14988     4187      402   3       0             0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.423520] [14990]     0 14990      666       35   0       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.424793] [14992]     0 14992      104        1   2       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.426033] [14994]     0 14994      104        1   0       0             0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.427247] Out of memory: Kill process 756 (NetworkManager) score 1 or sacrifice child
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.428451] Killed process 756 (NetworkManager) total-vm:268888kB, anon-rss:424kB, file-rss:8kB

^ permalink raw reply	[flat|nested] 36+ messages in thread