* Still OOM problems with 4.9er kernels @ 2016-11-30 7:10 Gerhard Wiesinger 2016-11-30 7:20 ` Gerhard Wiesinger 0 siblings, 1 reply; 88+ messages in thread From: Gerhard Wiesinger @ 2016-11-30 7:10 UTC (permalink / raw) To: linux-kernel; +Cc: Linus Torvalds torvalds Hello, I'm having out of memory situations with my "low memory" VMs in KVM under Fedora (Kernel 4.7, 4.8 and also before). They started to get more and more sensitive to OOM. I recently found the following info: https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ https://www.spinics.net/lists/linux-mm/msg113661.html Therefore I tried the latest Fedora kernels: 4.9.0-0.rc6.git2.1.fc26.x86_64 But OOM situation is still very easy to reproduce: 1.) VM with 128-384MB under Fedora 25 2.) Having some processes run without any load (e.g. Apache) 3.) run an update with: dnf clean all; dnf update 4.) dnf python process get's killed Please make the VM system working again in Kernel 4.9 and to use swap again correctly. Thnx. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-11-30 7:10 Still OOM problems with 4.9er kernels Gerhard Wiesinger @ 2016-11-30 7:20 ` Gerhard Wiesinger 2016-12-09 7:06 ` Gerhard Wiesinger 0 siblings, 1 reply; 88+ messages in thread From: Gerhard Wiesinger @ 2016-11-30 7:20 UTC (permalink / raw) To: linux-kernel; +Cc: Linus Torvalds Hello, See also: Bug 1314697 - Kernel 4.4.3-300.fc23.x86_64 is not stable inside a KVM VM https://bugzilla.redhat.com/show_bug.cgi?id=1314697 Ciao, Gerhard On 30.11.2016 08:10, Gerhard Wiesinger wrote: > Hello, > > I'm having out of memory situations with my "low memory" VMs in KVM > under Fedora (Kernel 4.7, 4.8 and also before). They started to get > more and more sensitive to OOM. I recently found the following info: > > https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ > > https://www.spinics.net/lists/linux-mm/msg113661.html > > Therefore I tried the latest Fedora kernels: > 4.9.0-0.rc6.git2.1.fc26.x86_64 > > But OOM situation is still very easy to reproduce: > > 1.) VM with 128-384MB under Fedora 25 > > 2.) Having some processes run without any load (e.g. Apache) > > 3.) run an update with: dnf clean all; dnf update > > 4.) dnf python process get's killed > > > Please make the VM system working again in Kernel 4.9 and to use swap > again correctly. > > Thnx. > > Ciao, > > Gerhard > > ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-11-30 7:20 ` Gerhard Wiesinger @ 2016-12-09 7:06 ` Gerhard Wiesinger 2016-12-09 13:40 ` Michal Hocko 0 siblings, 1 reply; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 7:06 UTC (permalink / raw) To: linux-kernel, linux-mm; +Cc: Linus Torvalds Hello, same with latest kernel rc, dnf still killed with OOM (but sometimes better). ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} (does dnf clean all;dnf update) Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Updated bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1314697 Any chance to get it fixed in 4.9.0 release? Ciao, Gerhard On 30.11.2016 08:20, Gerhard Wiesinger wrote: > Hello, > > See also: > Bug 1314697 - Kernel 4.4.3-300.fc23.x86_64 is not stable inside a KVM VM > https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > > Ciao, > Gerhard > > > On 30.11.2016 08:10, Gerhard Wiesinger wrote: >> Hello, >> >> I'm having out of memory situations with my "low memory" VMs in KVM >> under Fedora (Kernel 4.7, 4.8 and also before). They started to get >> more and more sensitive to OOM. I recently found the following info: >> >> https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ >> >> https://www.spinics.net/lists/linux-mm/msg113661.html >> >> Therefore I tried the latest Fedora kernels: >> 4.9.0-0.rc6.git2.1.fc26.x86_64 >> >> But OOM situation is still very easy to reproduce: >> >> 1.) VM with 128-384MB under Fedora 25 >> >> 2.) Having some processes run without any load (e.g. Apache) >> >> 3.) run an update with: dnf clean all; dnf update >> >> 4.) dnf python process get's killed >> >> >> Please make the VM system working again in Kernel 4.9 and to use swap >> again correctly. >> >> Thnx. >> >> Ciao, >> >> Gerhard >> >> > ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 7:06 ` Gerhard Wiesinger @ 2016-12-09 13:40 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2016-12-09 13:40 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: linux-kernel, linux-mm, Linus Torvalds On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: > Hello, > > same with latest kernel rc, dnf still killed with OOM (but sometimes > better). > > ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} > (does dnf clean all;dnf update) > Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 > 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > > Updated bug report: > https://bugzilla.redhat.com/show_bug.cgi?id=1314697 Could you post your oom report please? -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-09 13:40 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2016-12-09 13:40 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: linux-kernel, linux-mm, Linus Torvalds On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: > Hello, > > same with latest kernel rc, dnf still killed with OOM (but sometimes > better). > > ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} > (does dnf clean all;dnf update) > Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 > 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > > Updated bug report: > https://bugzilla.redhat.com/show_bug.cgi?id=1314697 Could you post your oom report please? -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 13:40 ` Michal Hocko @ 2016-12-09 15:52 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 15:52 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 14:40, Michal Hocko wrote: > On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >> Hello, >> >> same with latest kernel rc, dnf still killed with OOM (but sometimes >> better). >> >> ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} >> (does dnf clean all;dnf update) >> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >> >> Updated bug report: >> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > Could you post your oom report please? E.g. a new one with more than one included, first one after boot ... Just setup a low mem VM under KVM and it is easily triggerable. Still enough virtual memory available ... 4.9.0-0.rc8.git2.1.fc26.x86_64 [ 624.862777] ksoftirqd/0: page allocation failure: order:0, mode:0x2080020(GFP_ATOMIC) [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 0000000000000000 [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 ffffffff90c7dd98 [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 ffffaa62c007f9a0 [ 624.864998] Call Trace: [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 [virtio_net] [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 [virtio_net] [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 bytes left [95895.765570] kworker/1:1H: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [95895.766060] Workqueue: kblockd blk_mq_run_work_fn [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 0000000000000000 [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 ffffffff90c7dd98 [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 ffffaa62c0257670 [95895.766417] Call Trace: [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 [95895.773406] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 [95895.775212] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 [95895.781502] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 [virtio_ring] [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 [virtio_blk] [95895.785419] [<ffffffff901312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 [virtio_blk] [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 [virtio_blk] [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 [95895.794212] [<ffffffff90111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 [95895.796408] Mem-Info: [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 active_file:263 inactive_file:238 isolated_file:0 unevictable:0 dirty:0 writeback:330 unstable:0 slab_reclaimable:5241 slab_unreclaimable:9538 mapped:470 shmem:9 pagetables:2200 bounce:0 free:690 free_pcp:68 free_cma:0 [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB active_file:1052kB inactive_file:952kB unevictable:0kB isolated(anon):128kB isolated(file):0kB mapped:1880kB dirty:0kB writeback:1320kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB unstable:0kB pages_scanned:179 all_unreclaimable? no [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB active_anon:3544kB inactive_anon:3944kB active_file:84kB inactive_file:140kB unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:1728kB slab_unreclaimable:2964kB kernel_stack:84kB pagetables:1396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [95895.805936] lowmem_reserve[]: 0 117 117 117 117 [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB high:1944kB active_anon:31636kB inactive_anon:32164kB active_file:968kB inactive_file:804kB unevictable:0kB writepending:1288kB present:180080kB managed:139012kB mlocked:0kB slab_reclaimable:19236kB slab_unreclaimable:35188kB kernel_stack:1852kB pagetables:7404kB bounce:0kB free_pcp:272kB local_pcp:156kB free_cma:0kB [95895.809223] lowmem_reserve[]: 0 0 0 0 0 [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB (H) 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1836kB [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [95895.813981] 1804 total pagecache pages [95895.814931] 1289 pages in swap cache [95895.815849] Swap cache stats: add 5288014, delete 5286725, find 11568655/13881082 [95895.816792] Free swap = 1791816kB [95895.817706] Total swap = 2064380kB [95895.819222] 49018 pages RAM [95895.820145] 0 pages HighMem/MovableOnly [95895.821039] 10288 pages reserved [95895.823325] 0 pages cma reserved [95895.824244] 0 pages hwpoisoned [95895.825237] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [95895.826140] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 [97883.838418] httpd invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0 [97883.843507] httpd cpuset=/ mems_allowed=0 [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 ffff96b98b8b3100 [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 0000000000000000 [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d ffffaa62c395f9b0 [97883.849945] Call Trace: [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 [97883.882611] Mem-Info: [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 active_file:3902 inactive_file:3639 isolated_file:0 unevictable:0 dirty:205 writeback:0 unstable:0 slab_reclaimable:9856 slab_unreclaimable:9682 mapped:3722 shmem:59 pagetables:2080 bounce:0 free:748 free_pcp:15 free_cma:0 [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB unstable:0kB pages_scanned:168352 all_unreclaimable? yes [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB active_anon:1716kB inactive_anon:912kB active_file:2292kB inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:4652kB slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [97883.898799] lowmem_reserve[]: 0 117 117 117 117 [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB high:1944kB active_anon:9944kB inactive_anon:12572kB active_file:13316kB inactive_file:13680kB unevictable:0kB writepending:768kB present:180080kB managed:139012kB mlocked:0kB slab_reclaimable:34772kB slab_unreclaimable:35876kB kernel_stack:1828kB pagetables:7824kB bounce:0kB free_pcp:60kB local_pcp:52kB free_cma:0kB [97883.903033] lowmem_reserve[]: 0 0 0 0 0 [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB (H) 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1464kB [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB (H) 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1508kB [97883.908598] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [97883.908600] 10093 total pagecache pages [97883.909623] 2486 pages in swap cache [97883.910981] Swap cache stats: add 5936545, delete 5934059, find 11939248/14560523 [97883.911932] Free swap = 1924040kB [97883.912844] Total swap = 2064380kB [97883.913799] 49018 pages RAM [97883.915101] 0 pages HighMem/MovableOnly [97883.916149] 10288 pages reserved [97883.917465] 0 pages cma reserved [97883.918528] 0 pages hwpoisoned [97883.919635] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [97883.919656] [ 442] 0 442 15414 276 31 3 89 0 systemd-journal [97883.919661] [ 479] 0 479 11179 0 22 3 323 -1000 systemd-udevd [97883.919666] [ 535] 0 535 13888 0 28 3 114 -1000 auditd [97883.919671] [ 559] 81 559 11565 85 26 3 88 -900 dbus-daemon [97883.919675] [ 562] 70 562 12026 0 28 3 108 0 avahi-daemon [97883.919680] [ 578] 0 578 182739 11 279 4 263 0 rsyslogd [97883.919684] [ 579] 0 579 11890 26 27 3 105 0 systemd-logind [97883.919688] [ 580] 0 580 123902 265 223 3 1889 0 httpd [97883.919693] [ 581] 70 581 12026 0 26 3 83 0 avahi-daemon [97883.919697] [ 596] 0 596 1095 0 8 3 35 0 acpid [97883.919701] [ 614] 0 614 33234 1 18 4 155 0 crond [97883.919706] [ 630] 0 630 29260 0 9 3 32 0 agetty [97883.919710] [ 639] 0 639 25089 70 49 3 430 0 sendmail [97883.919715] [ 643] 0 643 20785 0 43 3 231 -1000 sshd [97883.919719] [ 674] 51 674 23428 0 46 3 415 0 sendmail [97883.919724] [ 723] 0 723 44394 148 38 3 2134 0 munin-node [97883.919728] [ 2513] 38 2513 6971 23 18 3 130 0 ntpd [97883.919733] [13908] 0 13908 36690 0 74 3 320 0 sshd [97883.919738] [13910] 0 13910 15956 1 35 3 235 0 systemd [97883.919742] [13913] 0 13913 60419 0 49 4 463 0 (sd-pam) [97883.919747] [13918] 0 13918 36690 147 71 3 280 0 sshd [97883.919751] [13930] 0 13930 30730 1 13 4 285 0 bash [97883.919756] [13958] 0 13958 29971 0 13 3 55 0 update.sh [97883.919761] [13985] 0 13985 159639 4589 192 3 26191 0 dnf [97883.919766] [17393] 48 17393 177835 704 238 3 1711 0 httpd [97883.919771] [19018] 48 19018 128626 1434 222 3 1342 0 httpd [97883.919774] [19043] 48 19043 128067 2010 219 3 1492 0 httpd [97883.919776] Out of memory: Kill process 13985 (dnf) score 54 or sacrifice child [97883.920895] Killed process 13985 (dnf) total-vm:638556kB, anon-rss:12840kB, file-rss:5516kB, shmem-rss:0kB [97883.986808] oom_reaper: reaped process 13985 (dnf), now anon-rss:0kB, file-rss:1552kB, shmem-rss:0kB Thank you. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-09 15:52 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 15:52 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 14:40, Michal Hocko wrote: > On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >> Hello, >> >> same with latest kernel rc, dnf still killed with OOM (but sometimes >> better). >> >> ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} >> (does dnf clean all;dnf update) >> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >> >> Updated bug report: >> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > Could you post your oom report please? E.g. a new one with more than one included, first one after boot ... Just setup a low mem VM under KVM and it is easily triggerable. Still enough virtual memory available ... 4.9.0-0.rc8.git2.1.fc26.x86_64 [ 624.862777] ksoftirqd/0: page allocation failure: order:0, mode:0x2080020(GFP_ATOMIC) [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 0000000000000000 [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 ffffffff90c7dd98 [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 ffffaa62c007f9a0 [ 624.864998] Call Trace: [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 [virtio_net] [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 [virtio_net] [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 bytes left [95895.765570] kworker/1:1H: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [95895.766060] Workqueue: kblockd blk_mq_run_work_fn [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 0000000000000000 [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 ffffffff90c7dd98 [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 ffffaa62c0257670 [95895.766417] Call Trace: [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 [95895.773406] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 [95895.775212] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 [95895.781502] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 [virtio_ring] [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 [virtio_blk] [95895.785419] [<ffffffff901312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 [virtio_blk] [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 [virtio_blk] [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 [95895.794212] [<ffffffff90111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 [95895.796408] Mem-Info: [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 active_file:263 inactive_file:238 isolated_file:0 unevictable:0 dirty:0 writeback:330 unstable:0 slab_reclaimable:5241 slab_unreclaimable:9538 mapped:470 shmem:9 pagetables:2200 bounce:0 free:690 free_pcp:68 free_cma:0 [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB active_file:1052kB inactive_file:952kB unevictable:0kB isolated(anon):128kB isolated(file):0kB mapped:1880kB dirty:0kB writeback:1320kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB unstable:0kB pages_scanned:179 all_unreclaimable? no [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB active_anon:3544kB inactive_anon:3944kB active_file:84kB inactive_file:140kB unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:1728kB slab_unreclaimable:2964kB kernel_stack:84kB pagetables:1396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [95895.805936] lowmem_reserve[]: 0 117 117 117 117 [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB high:1944kB active_anon:31636kB inactive_anon:32164kB active_file:968kB inactive_file:804kB unevictable:0kB writepending:1288kB present:180080kB managed:139012kB mlocked:0kB slab_reclaimable:19236kB slab_unreclaimable:35188kB kernel_stack:1852kB pagetables:7404kB bounce:0kB free_pcp:272kB local_pcp:156kB free_cma:0kB [95895.809223] lowmem_reserve[]: 0 0 0 0 0 [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB (H) 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1836kB [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [95895.813981] 1804 total pagecache pages [95895.814931] 1289 pages in swap cache [95895.815849] Swap cache stats: add 5288014, delete 5286725, find 11568655/13881082 [95895.816792] Free swap = 1791816kB [95895.817706] Total swap = 2064380kB [95895.819222] 49018 pages RAM [95895.820145] 0 pages HighMem/MovableOnly [95895.821039] 10288 pages reserved [95895.823325] 0 pages cma reserved [95895.824244] 0 pages hwpoisoned [95895.825237] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [95895.826140] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 [97883.838418] httpd invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0 [97883.843507] httpd cpuset=/ mems_allowed=0 [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 ffff96b98b8b3100 [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 0000000000000000 [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d ffffaa62c395f9b0 [97883.849945] Call Trace: [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 [97883.882611] Mem-Info: [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 active_file:3902 inactive_file:3639 isolated_file:0 unevictable:0 dirty:205 writeback:0 unstable:0 slab_reclaimable:9856 slab_unreclaimable:9682 mapped:3722 shmem:59 pagetables:2080 bounce:0 free:748 free_pcp:15 free_cma:0 [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB unstable:0kB pages_scanned:168352 all_unreclaimable? yes [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB active_anon:1716kB inactive_anon:912kB active_file:2292kB inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:4652kB slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [97883.898799] lowmem_reserve[]: 0 117 117 117 117 [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB high:1944kB active_anon:9944kB inactive_anon:12572kB active_file:13316kB inactive_file:13680kB unevictable:0kB writepending:768kB present:180080kB managed:139012kB mlocked:0kB slab_reclaimable:34772kB slab_unreclaimable:35876kB kernel_stack:1828kB pagetables:7824kB bounce:0kB free_pcp:60kB local_pcp:52kB free_cma:0kB [97883.903033] lowmem_reserve[]: 0 0 0 0 0 [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB (H) 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1464kB [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB (H) 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1508kB [97883.908598] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [97883.908600] 10093 total pagecache pages [97883.909623] 2486 pages in swap cache [97883.910981] Swap cache stats: add 5936545, delete 5934059, find 11939248/14560523 [97883.911932] Free swap = 1924040kB [97883.912844] Total swap = 2064380kB [97883.913799] 49018 pages RAM [97883.915101] 0 pages HighMem/MovableOnly [97883.916149] 10288 pages reserved [97883.917465] 0 pages cma reserved [97883.918528] 0 pages hwpoisoned [97883.919635] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [97883.919656] [ 442] 0 442 15414 276 31 3 89 0 systemd-journal [97883.919661] [ 479] 0 479 11179 0 22 3 323 -1000 systemd-udevd [97883.919666] [ 535] 0 535 13888 0 28 3 114 -1000 auditd [97883.919671] [ 559] 81 559 11565 85 26 3 88 -900 dbus-daemon [97883.919675] [ 562] 70 562 12026 0 28 3 108 0 avahi-daemon [97883.919680] [ 578] 0 578 182739 11 279 4 263 0 rsyslogd [97883.919684] [ 579] 0 579 11890 26 27 3 105 0 systemd-logind [97883.919688] [ 580] 0 580 123902 265 223 3 1889 0 httpd [97883.919693] [ 581] 70 581 12026 0 26 3 83 0 avahi-daemon [97883.919697] [ 596] 0 596 1095 0 8 3 35 0 acpid [97883.919701] [ 614] 0 614 33234 1 18 4 155 0 crond [97883.919706] [ 630] 0 630 29260 0 9 3 32 0 agetty [97883.919710] [ 639] 0 639 25089 70 49 3 430 0 sendmail [97883.919715] [ 643] 0 643 20785 0 43 3 231 -1000 sshd [97883.919719] [ 674] 51 674 23428 0 46 3 415 0 sendmail [97883.919724] [ 723] 0 723 44394 148 38 3 2134 0 munin-node [97883.919728] [ 2513] 38 2513 6971 23 18 3 130 0 ntpd [97883.919733] [13908] 0 13908 36690 0 74 3 320 0 sshd [97883.919738] [13910] 0 13910 15956 1 35 3 235 0 systemd [97883.919742] [13913] 0 13913 60419 0 49 4 463 0 (sd-pam) [97883.919747] [13918] 0 13918 36690 147 71 3 280 0 sshd [97883.919751] [13930] 0 13930 30730 1 13 4 285 0 bash [97883.919756] [13958] 0 13958 29971 0 13 3 55 0 update.sh [97883.919761] [13985] 0 13985 159639 4589 192 3 26191 0 dnf [97883.919766] [17393] 48 17393 177835 704 238 3 1711 0 httpd [97883.919771] [19018] 48 19018 128626 1434 222 3 1342 0 httpd [97883.919774] [19043] 48 19043 128067 2010 219 3 1492 0 httpd [97883.919776] Out of memory: Kill process 13985 (dnf) score 54 or sacrifice child [97883.920895] Killed process 13985 (dnf) total-vm:638556kB, anon-rss:12840kB, file-rss:5516kB, shmem-rss:0kB [97883.986808] oom_reaper: reaped process 13985 (dnf), now anon-rss:0kB, file-rss:1552kB, shmem-rss:0kB Thank you. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 15:52 ` Gerhard Wiesinger @ 2016-12-09 15:58 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 15:58 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 16:52, Gerhard Wiesinger wrote: > On 09.12.2016 14:40, Michal Hocko wrote: >> On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >>> Hello, >>> >>> same with latest kernel rc, dnf still killed with OOM (but sometimes >>> better). >>> >>> ./update.sh: line 40: 1591 Killed ${EXE} update >>> ${PARAMS} >>> (does dnf clean all;dnf update) >>> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >>> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >>> >>> Updated bug report: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 >> Could you post your oom report please? > > E.g. a new one with more than one included, first one after boot ... > > Just setup a low mem VM under KVM and it is easily triggerable. > > Still enough virtual memory available ... > > 4.9.0-0.rc8.git2.1.fc26.x86_64 > > [ 624.862777] ksoftirqd/0: page allocation failure: order:0, > mode:0x2080020(GFP_ATOMIC) > [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.9.3 > [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 > 0000000000000000 > [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 > ffffffff90c7dd98 > [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 > ffffaa62c007f9a0 > [ 624.864998] Call Trace: > [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 > [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 > [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 > [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 > [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 > [virtio_net] > [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 > [virtio_net] > [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] > [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 > [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 > [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 > [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 > [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 > [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 > [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 > [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 > [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 > [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 > [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 > [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left > [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left > [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left > [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 > bytes left > [95895.765570] kworker/1:1H: page allocation failure: order:0, > mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) > [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.9.3 > [95895.766060] Workqueue: kblockd blk_mq_run_work_fn > [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 > 0000000000000000 > [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 > ffffffff90c7dd98 > [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 > ffffaa62c0257670 > [95895.766417] Call Trace: > [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 > [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 > [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 > [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 > [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 > [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 > [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 > [95895.773406] [<ffffffffc005850d>] ? > alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] > [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 > [95895.775212] [<ffffffffc005850d>] ? > alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] > [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 > [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 > [95895.781502] [<ffffffffc005850d>] ? > alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] > [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 > [virtio_ring] > [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 > [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 > [virtio_blk] > [95895.785419] [<ffffffff901312fd>] ? > debug_lockdep_rcu_enabled+0x1d/0x20 > [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 > [virtio_blk] > [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 > [virtio_blk] > [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 > [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 > [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 > [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 > [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 > [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 > [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 > [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 > [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 > [95895.794212] [<ffffffff90111775>] ? > trace_hardirqs_on_caller+0xf5/0x1b0 > [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 > [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 > [95895.796408] Mem-Info: > [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 > active_file:263 inactive_file:238 isolated_file:0 > unevictable:0 dirty:0 writeback:330 unstable:0 > slab_reclaimable:5241 slab_unreclaimable:9538 > mapped:470 shmem:9 pagetables:2200 bounce:0 > free:690 free_pcp:68 free_cma:0 > [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB > active_file:1052kB inactive_file:952kB unevictable:0kB > isolated(anon):128kB isolated(file):0kB mapped:1880kB dirty:0kB > writeback:1320kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB > anon_thp: 36kB writeback_tmp:0kB unstable:0kB pages_scanned:179 > all_unreclaimable? no > [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB > active_anon:3544kB inactive_anon:3944kB active_file:84kB > inactive_file:140kB unevictable:0kB writepending:4kB present:15992kB > managed:15908kB mlocked:0kB slab_reclaimable:1728kB > slab_unreclaimable:2964kB kernel_stack:84kB pagetables:1396kB > bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [95895.805936] lowmem_reserve[]: 0 117 117 117 117 > [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB > high:1944kB active_anon:31636kB inactive_anon:32164kB > active_file:968kB inactive_file:804kB unevictable:0kB > writepending:1288kB present:180080kB managed:139012kB mlocked:0kB > slab_reclaimable:19236kB slab_unreclaimable:35188kB > kernel_stack:1852kB pagetables:7404kB bounce:0kB free_pcp:272kB > local_pcp:156kB free_cma:0kB > [95895.809223] lowmem_reserve[]: 0 0 0 0 0 > [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB > (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB > [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB > (H) 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 1836kB > [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 > hugepages_surp=0 hugepages_size=2048kB > [95895.813981] 1804 total pagecache pages > [95895.814931] 1289 pages in swap cache > [95895.815849] Swap cache stats: add 5288014, delete 5286725, find > 11568655/13881082 > [95895.816792] Free swap = 1791816kB > [95895.817706] Total swap = 2064380kB > [95895.819222] 49018 pages RAM > [95895.820145] 0 pages HighMem/MovableOnly > [95895.821039] 10288 pages reserved > [95895.823325] 0 pages cma reserved > [95895.824244] 0 pages hwpoisoned > [95895.825237] SLUB: Unable to allocate memory on node -1, > gfp=0x2080020(GFP_ATOMIC) > [95895.826140] cache: kmalloc-256, object size: 256, buffer size: > 256, default order: 0, min order: 0 > [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 > [97883.838418] httpd invoked oom-killer: > gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, > order=0, oom_score_adj=0 > [97883.843507] httpd cpuset=/ mems_allowed=0 > [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.9.3 > [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 > ffff96b98b8b3100 > [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 > 0000000000000000 > [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d > ffffaa62c395f9b0 > [97883.849945] Call Trace: > [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f > [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 > [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 > [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 > [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 > [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 > [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 > [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 > [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 > [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 > [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 > [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 > [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 > [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 > [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 > [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 > [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 > [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 > [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 > [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 > [97883.882611] Mem-Info: > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > active_file:3902 inactive_file:3639 isolated_file:0 > unevictable:0 dirty:205 writeback:0 unstable:0 > slab_reclaimable:9856 slab_unreclaimable:9682 > mapped:3722 shmem:59 pagetables:2080 bounce:0 > free:748 free_pcp:15 free_cma:0 > [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB > active_file:15608kB inactive_file:14556kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB mapped:14888kB dirty:820kB > writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: > 236kB writeback_tmp:0kB unstable:0kB pages_scanned:168352 > all_unreclaimable? yes > [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB > active_anon:1716kB inactive_anon:912kB active_file:2292kB > inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB > managed:15908kB mlocked:0kB slab_reclaimable:4652kB > slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB > bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [97883.898799] lowmem_reserve[]: 0 117 117 117 117 > [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB > high:1944kB active_anon:9944kB inactive_anon:12572kB > active_file:13316kB inactive_file:13680kB unevictable:0kB > writepending:768kB present:180080kB managed:139012kB mlocked:0kB > slab_reclaimable:34772kB slab_unreclaimable:35876kB > kernel_stack:1828kB pagetables:7824kB bounce:0kB free_pcp:60kB > local_pcp:52kB free_cma:0kB > [97883.903033] lowmem_reserve[]: 0 0 0 0 0 > [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB > (H) 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB > = 1464kB > [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB > (H) 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB > 0*4096kB = 1508kB > [97883.908598] Node 0 hugepages_total=0 hugepages_free=0 > hugepages_surp=0 hugepages_size=2048kB > [97883.908600] 10093 total pagecache pages > [97883.909623] 2486 pages in swap cache > [97883.910981] Swap cache stats: add 5936545, delete 5934059, find > 11939248/14560523 > [97883.911932] Free swap = 1924040kB > [97883.912844] Total swap = 2064380kB > [97883.913799] 49018 pages RAM > [97883.915101] 0 pages HighMem/MovableOnly > [97883.916149] 10288 pages reserved > [97883.917465] 0 pages cma reserved > [97883.918528] 0 pages hwpoisoned > [97883.919635] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds > swapents oom_score_adj name > [97883.919656] [ 442] 0 442 15414 276 31 3 > 89 0 systemd-journal > [97883.919661] [ 479] 0 479 11179 0 22 3 > 323 -1000 systemd-udevd > [97883.919666] [ 535] 0 535 13888 0 28 3 > 114 -1000 auditd > [97883.919671] [ 559] 81 559 11565 85 26 3 > 88 -900 dbus-daemon > [97883.919675] [ 562] 70 562 12026 0 28 3 > 108 0 avahi-daemon > [97883.919680] [ 578] 0 578 182739 11 279 4 > 263 0 rsyslogd > [97883.919684] [ 579] 0 579 11890 26 27 3 > 105 0 systemd-logind > [97883.919688] [ 580] 0 580 123902 265 223 3 > 1889 0 httpd > [97883.919693] [ 581] 70 581 12026 0 26 3 > 83 0 avahi-daemon > [97883.919697] [ 596] 0 596 1095 0 8 3 > 35 0 acpid > [97883.919701] [ 614] 0 614 33234 1 18 4 > 155 0 crond > [97883.919706] [ 630] 0 630 29260 0 9 3 > 32 0 agetty > [97883.919710] [ 639] 0 639 25089 70 49 3 > 430 0 sendmail > [97883.919715] [ 643] 0 643 20785 0 43 3 > 231 -1000 sshd > [97883.919719] [ 674] 51 674 23428 0 46 3 > 415 0 sendmail > [97883.919724] [ 723] 0 723 44394 148 38 3 > 2134 0 munin-node > [97883.919728] [ 2513] 38 2513 6971 23 18 3 > 130 0 ntpd > [97883.919733] [13908] 0 13908 36690 0 74 3 > 320 0 sshd > [97883.919738] [13910] 0 13910 15956 1 35 3 > 235 0 systemd > [97883.919742] [13913] 0 13913 60419 0 49 4 > 463 0 (sd-pam) > [97883.919747] [13918] 0 13918 36690 147 71 3 > 280 0 sshd > [97883.919751] [13930] 0 13930 30730 1 13 4 > 285 0 bash > [97883.919756] [13958] 0 13958 29971 0 13 3 > 55 0 update.sh > [97883.919761] [13985] 0 13985 159639 4589 192 3 > 26191 0 dnf > [97883.919766] [17393] 48 17393 177835 704 238 3 > 1711 0 httpd > [97883.919771] [19018] 48 19018 128626 1434 222 3 > 1342 0 httpd > [97883.919774] [19043] 48 19043 128067 2010 219 3 > 1492 0 httpd > [97883.919776] Out of memory: Kill process 13985 (dnf) score 54 or > sacrifice child > [97883.920895] Killed process 13985 (dnf) total-vm:638556kB, > anon-rss:12840kB, file-rss:5516kB, shmem-rss:0kB > [97883.986808] oom_reaper: reaped process 13985 (dnf), now > anon-rss:0kB, file-rss:1552kB, shmem-rss:0kB > Forgot to mention: rpm DB is also mostly not usable afterwards, needs reboot: error: rpmdb: BDB0113 Thread/process 13985/140089649317632 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm Error: Error: rpmdb open failed Thank you. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-09 15:58 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 15:58 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 16:52, Gerhard Wiesinger wrote: > On 09.12.2016 14:40, Michal Hocko wrote: >> On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >>> Hello, >>> >>> same with latest kernel rc, dnf still killed with OOM (but sometimes >>> better). >>> >>> ./update.sh: line 40: 1591 Killed ${EXE} update >>> ${PARAMS} >>> (does dnf clean all;dnf update) >>> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >>> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >>> >>> Updated bug report: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 >> Could you post your oom report please? > > E.g. a new one with more than one included, first one after boot ... > > Just setup a low mem VM under KVM and it is easily triggerable. > > Still enough virtual memory available ... > > 4.9.0-0.rc8.git2.1.fc26.x86_64 > > [ 624.862777] ksoftirqd/0: page allocation failure: order:0, > mode:0x2080020(GFP_ATOMIC) > [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.9.3 > [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 > 0000000000000000 > [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 > ffffffff90c7dd98 > [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 > ffffaa62c007f9a0 > [ 624.864998] Call Trace: > [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 > [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 > [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 > [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 > [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 > [virtio_net] > [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 > [virtio_net] > [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] > [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 > [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 > [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 > [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 > [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 > [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 > [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 > [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 > [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 > [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 > [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 > [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left > [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left > [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left > [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 > bytes left > [95895.765570] kworker/1:1H: page allocation failure: order:0, > mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) > [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.9.3 > [95895.766060] Workqueue: kblockd blk_mq_run_work_fn > [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 > 0000000000000000 > [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 > ffffffff90c7dd98 > [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 > ffffaa62c0257670 > [95895.766417] Call Trace: > [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 > [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 > [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 > [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 > [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 > [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 > [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 > [95895.773406] [<ffffffffc005850d>] ? > alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] > [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 > [95895.775212] [<ffffffffc005850d>] ? > alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] > [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 > [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 > [95895.781502] [<ffffffffc005850d>] ? > alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] > [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 > [virtio_ring] > [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 > [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 > [virtio_blk] > [95895.785419] [<ffffffff901312fd>] ? > debug_lockdep_rcu_enabled+0x1d/0x20 > [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 > [virtio_blk] > [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 > [virtio_blk] > [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 > [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 > [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 > [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 > [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 > [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 > [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 > [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 > [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 > [95895.794212] [<ffffffff90111775>] ? > trace_hardirqs_on_caller+0xf5/0x1b0 > [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 > [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 > [95895.796408] Mem-Info: > [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 > active_file:263 inactive_file:238 isolated_file:0 > unevictable:0 dirty:0 writeback:330 unstable:0 > slab_reclaimable:5241 slab_unreclaimable:9538 > mapped:470 shmem:9 pagetables:2200 bounce:0 > free:690 free_pcp:68 free_cma:0 > [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB > active_file:1052kB inactive_file:952kB unevictable:0kB > isolated(anon):128kB isolated(file):0kB mapped:1880kB dirty:0kB > writeback:1320kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB > anon_thp: 36kB writeback_tmp:0kB unstable:0kB pages_scanned:179 > all_unreclaimable? no > [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB > active_anon:3544kB inactive_anon:3944kB active_file:84kB > inactive_file:140kB unevictable:0kB writepending:4kB present:15992kB > managed:15908kB mlocked:0kB slab_reclaimable:1728kB > slab_unreclaimable:2964kB kernel_stack:84kB pagetables:1396kB > bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [95895.805936] lowmem_reserve[]: 0 117 117 117 117 > [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB > high:1944kB active_anon:31636kB inactive_anon:32164kB > active_file:968kB inactive_file:804kB unevictable:0kB > writepending:1288kB present:180080kB managed:139012kB mlocked:0kB > slab_reclaimable:19236kB slab_unreclaimable:35188kB > kernel_stack:1852kB pagetables:7404kB bounce:0kB free_pcp:272kB > local_pcp:156kB free_cma:0kB > [95895.809223] lowmem_reserve[]: 0 0 0 0 0 > [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB > (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB > [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB > (H) 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 1836kB > [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 > hugepages_surp=0 hugepages_size=2048kB > [95895.813981] 1804 total pagecache pages > [95895.814931] 1289 pages in swap cache > [95895.815849] Swap cache stats: add 5288014, delete 5286725, find > 11568655/13881082 > [95895.816792] Free swap = 1791816kB > [95895.817706] Total swap = 2064380kB > [95895.819222] 49018 pages RAM > [95895.820145] 0 pages HighMem/MovableOnly > [95895.821039] 10288 pages reserved > [95895.823325] 0 pages cma reserved > [95895.824244] 0 pages hwpoisoned > [95895.825237] SLUB: Unable to allocate memory on node -1, > gfp=0x2080020(GFP_ATOMIC) > [95895.826140] cache: kmalloc-256, object size: 256, buffer size: > 256, default order: 0, min order: 0 > [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 > [97883.838418] httpd invoked oom-killer: > gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, > order=0, oom_score_adj=0 > [97883.843507] httpd cpuset=/ mems_allowed=0 > [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.9.3 > [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 > ffff96b98b8b3100 > [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 > 0000000000000000 > [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d > ffffaa62c395f9b0 > [97883.849945] Call Trace: > [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f > [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 > [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 > [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 > [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 > [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 > [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 > [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 > [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 > [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 > [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 > [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 > [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 > [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 > [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 > [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 > [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 > [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 > [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 > [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 > [97883.882611] Mem-Info: > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > active_file:3902 inactive_file:3639 isolated_file:0 > unevictable:0 dirty:205 writeback:0 unstable:0 > slab_reclaimable:9856 slab_unreclaimable:9682 > mapped:3722 shmem:59 pagetables:2080 bounce:0 > free:748 free_pcp:15 free_cma:0 > [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB > active_file:15608kB inactive_file:14556kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB mapped:14888kB dirty:820kB > writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: > 236kB writeback_tmp:0kB unstable:0kB pages_scanned:168352 > all_unreclaimable? yes > [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB > active_anon:1716kB inactive_anon:912kB active_file:2292kB > inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB > managed:15908kB mlocked:0kB slab_reclaimable:4652kB > slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB > bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [97883.898799] lowmem_reserve[]: 0 117 117 117 117 > [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB > high:1944kB active_anon:9944kB inactive_anon:12572kB > active_file:13316kB inactive_file:13680kB unevictable:0kB > writepending:768kB present:180080kB managed:139012kB mlocked:0kB > slab_reclaimable:34772kB slab_unreclaimable:35876kB > kernel_stack:1828kB pagetables:7824kB bounce:0kB free_pcp:60kB > local_pcp:52kB free_cma:0kB > [97883.903033] lowmem_reserve[]: 0 0 0 0 0 > [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB > (H) 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB > = 1464kB > [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB > (H) 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB > 0*4096kB = 1508kB > [97883.908598] Node 0 hugepages_total=0 hugepages_free=0 > hugepages_surp=0 hugepages_size=2048kB > [97883.908600] 10093 total pagecache pages > [97883.909623] 2486 pages in swap cache > [97883.910981] Swap cache stats: add 5936545, delete 5934059, find > 11939248/14560523 > [97883.911932] Free swap = 1924040kB > [97883.912844] Total swap = 2064380kB > [97883.913799] 49018 pages RAM > [97883.915101] 0 pages HighMem/MovableOnly > [97883.916149] 10288 pages reserved > [97883.917465] 0 pages cma reserved > [97883.918528] 0 pages hwpoisoned > [97883.919635] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds > swapents oom_score_adj name > [97883.919656] [ 442] 0 442 15414 276 31 3 > 89 0 systemd-journal > [97883.919661] [ 479] 0 479 11179 0 22 3 > 323 -1000 systemd-udevd > [97883.919666] [ 535] 0 535 13888 0 28 3 > 114 -1000 auditd > [97883.919671] [ 559] 81 559 11565 85 26 3 > 88 -900 dbus-daemon > [97883.919675] [ 562] 70 562 12026 0 28 3 > 108 0 avahi-daemon > [97883.919680] [ 578] 0 578 182739 11 279 4 > 263 0 rsyslogd > [97883.919684] [ 579] 0 579 11890 26 27 3 > 105 0 systemd-logind > [97883.919688] [ 580] 0 580 123902 265 223 3 > 1889 0 httpd > [97883.919693] [ 581] 70 581 12026 0 26 3 > 83 0 avahi-daemon > [97883.919697] [ 596] 0 596 1095 0 8 3 > 35 0 acpid > [97883.919701] [ 614] 0 614 33234 1 18 4 > 155 0 crond > [97883.919706] [ 630] 0 630 29260 0 9 3 > 32 0 agetty > [97883.919710] [ 639] 0 639 25089 70 49 3 > 430 0 sendmail > [97883.919715] [ 643] 0 643 20785 0 43 3 > 231 -1000 sshd > [97883.919719] [ 674] 51 674 23428 0 46 3 > 415 0 sendmail > [97883.919724] [ 723] 0 723 44394 148 38 3 > 2134 0 munin-node > [97883.919728] [ 2513] 38 2513 6971 23 18 3 > 130 0 ntpd > [97883.919733] [13908] 0 13908 36690 0 74 3 > 320 0 sshd > [97883.919738] [13910] 0 13910 15956 1 35 3 > 235 0 systemd > [97883.919742] [13913] 0 13913 60419 0 49 4 > 463 0 (sd-pam) > [97883.919747] [13918] 0 13918 36690 147 71 3 > 280 0 sshd > [97883.919751] [13930] 0 13930 30730 1 13 4 > 285 0 bash > [97883.919756] [13958] 0 13958 29971 0 13 3 > 55 0 update.sh > [97883.919761] [13985] 0 13985 159639 4589 192 3 > 26191 0 dnf > [97883.919766] [17393] 48 17393 177835 704 238 3 > 1711 0 httpd > [97883.919771] [19018] 48 19018 128626 1434 222 3 > 1342 0 httpd > [97883.919774] [19043] 48 19043 128067 2010 219 3 > 1492 0 httpd > [97883.919776] Out of memory: Kill process 13985 (dnf) score 54 or > sacrifice child > [97883.920895] Killed process 13985 (dnf) total-vm:638556kB, > anon-rss:12840kB, file-rss:5516kB, shmem-rss:0kB > [97883.986808] oom_reaper: reaped process 13985 (dnf), now > anon-rss:0kB, file-rss:1552kB, shmem-rss:0kB > Forgot to mention: rpm DB is also mostly not usable afterwards, needs reboot: error: rpmdb: BDB0113 Thread/process 13985/140089649317632 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm Error: Error: rpmdb open failed Thank you. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 15:52 ` Gerhard Wiesinger @ 2016-12-09 16:09 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2016-12-09 16:09 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: linux-kernel, linux-mm, Linus Torvalds On Fri 09-12-16 16:52:07, Gerhard Wiesinger wrote: > On 09.12.2016 14:40, Michal Hocko wrote: > > On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: > > > Hello, > > > > > > same with latest kernel rc, dnf still killed with OOM (but sometimes > > > better). > > > > > > ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} > > > (does dnf clean all;dnf update) > > > Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 > > > 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > > > > > > Updated bug report: > > > https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > > Could you post your oom report please? > > E.g. a new one with more than one included, first one after boot ... > > Just setup a low mem VM under KVM and it is easily triggerable. What is the workload? > Still enough virtual memory available ... Well, you will always have a lot of virtual memory... > 4.9.0-0.rc8.git2.1.fc26.x86_64 > > [ 624.862777] ksoftirqd/0: page allocation failure: order:0, mode:0x2080020(GFP_ATOMIC) [...] > [95895.765570] kworker/1:1H: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) These are atomic allocation failures and should be recoverable. [...] > [97883.838418] httpd invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0 But this is a real OOM killer invocation because a single page allocation cannot proceed. [...] > [97883.882611] Mem-Info: > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > active_file:3902 inactive_file:3639 isolated_file:0 > unevictable:0 dirty:205 writeback:0 unstable:0 > slab_reclaimable:9856 slab_unreclaimable:9682 > mapped:3722 shmem:59 pagetables:2080 bounce:0 > free:748 free_pcp:15 free_cma:0 there is still some page cache which doesn't seem to be neither dirty nor under writeback. So it should be theoretically reclaimable but for some reason we cannot seem to reclaim that memory. There is still some anonymous memory and free swap so we could reclaim it as well but it all seems pretty down and the memory pressure is really large > [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB > active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB > unstable:0kB pages_scanned:168352 all_unreclaimable? yes all_unreclaimable also agrees that basically nothing is reclaimable. That was one of the criterion to hit the OOM killer prior to the rewrite in 4.6 kernel. So I suspect that older kernels would OOM under your memory pressure as well. -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-09 16:09 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2016-12-09 16:09 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: linux-kernel, linux-mm, Linus Torvalds On Fri 09-12-16 16:52:07, Gerhard Wiesinger wrote: > On 09.12.2016 14:40, Michal Hocko wrote: > > On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: > > > Hello, > > > > > > same with latest kernel rc, dnf still killed with OOM (but sometimes > > > better). > > > > > > ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} > > > (does dnf clean all;dnf update) > > > Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 > > > 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > > > > > > Updated bug report: > > > https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > > Could you post your oom report please? > > E.g. a new one with more than one included, first one after boot ... > > Just setup a low mem VM under KVM and it is easily triggerable. What is the workload? > Still enough virtual memory available ... Well, you will always have a lot of virtual memory... > 4.9.0-0.rc8.git2.1.fc26.x86_64 > > [ 624.862777] ksoftirqd/0: page allocation failure: order:0, mode:0x2080020(GFP_ATOMIC) [...] > [95895.765570] kworker/1:1H: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) These are atomic allocation failures and should be recoverable. [...] > [97883.838418] httpd invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0 But this is a real OOM killer invocation because a single page allocation cannot proceed. [...] > [97883.882611] Mem-Info: > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > active_file:3902 inactive_file:3639 isolated_file:0 > unevictable:0 dirty:205 writeback:0 unstable:0 > slab_reclaimable:9856 slab_unreclaimable:9682 > mapped:3722 shmem:59 pagetables:2080 bounce:0 > free:748 free_pcp:15 free_cma:0 there is still some page cache which doesn't seem to be neither dirty nor under writeback. So it should be theoretically reclaimable but for some reason we cannot seem to reclaim that memory. There is still some anonymous memory and free swap so we could reclaim it as well but it all seems pretty down and the memory pressure is really large > [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB > active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB > unstable:0kB pages_scanned:168352 all_unreclaimable? yes all_unreclaimable also agrees that basically nothing is reclaimable. That was one of the criterion to hit the OOM killer prior to the rewrite in 4.6 kernel. So I suspect that older kernels would OOM under your memory pressure as well. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 16:09 ` Michal Hocko (?) @ 2016-12-09 16:58 ` Gerhard Wiesinger 2016-12-09 17:30 ` Michal Hocko -1 siblings, 1 reply; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 16:58 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 17:09, Michal Hocko wrote: > On Fri 09-12-16 16:52:07, Gerhard Wiesinger wrote: >> On 09.12.2016 14:40, Michal Hocko wrote: >>> On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >>>> Hello, >>>> >>>> same with latest kernel rc, dnf still killed with OOM (but sometimes >>>> better). >>>> >>>> ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} >>>> (does dnf clean all;dnf update) >>>> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >>>> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >>>> >>>> Updated bug report: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 >>> Could you post your oom report please? >> E.g. a new one with more than one included, first one after boot ... >> >> Just setup a low mem VM under KVM and it is easily triggerable. > What is the workload? just run dnf clean all;dnf update (and the other tasks running on those machine. The normal load on most of these machines is pretty VERY LOW, e.g. running just an apache httpd doing nothing or e.g. running samba domain controller doing nothing) So my setups are low mem VMs so that KVM host has most of the caching effects shared. I'm running this setup since Fedora 17 under kernel-3.3.4-5.fc17.x86_64 and had NO problems. Problems started with 4.4.3-300.fc23.x86_64 and got worser in each major kernel versions (for upgrades I had even give the VMs temporarilly more memory for the upgrade situation). (from my bug report at https://bugzilla.redhat.com/show_bug.cgi?id=1314697 Previous kernel version on guest/host was rocket stable. Revert to kernel-4.3.5-300.fc23.x86_64 also solved it.) For completeness the actual kernel parameters on all hosts and VMs. vm.dirty_background_ratio=3 vm.dirty_ratio=15 vm.overcommit_memory=2 vm.overcommit_ratio=80 vm.swappiness=10 With kernel 4.9.0rc7 or rc8 it was getting better. But still not there where it should be (and was already). > >> Still enough virtual memory available ... > Well, you will always have a lot of virtual memory... And why is it not used, e.g. swapped and gets into an OOM situation? > >> 4.9.0-0.rc8.git2.1.fc26.x86_64 >> >> [ 624.862777] ksoftirqd/0: page allocation failure: order:0, mode:0x2080020(GFP_ATOMIC) > [...] >> [95895.765570] kworker/1:1H: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) > These are atomic allocation failures and should be recoverable. > [...] > >> [97883.838418] httpd invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0 > But this is a real OOM killer invocation because a single page allocation > cannot proceed. > > [...] >> [97883.882611] Mem-Info: >> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >> active_file:3902 inactive_file:3639 isolated_file:0 >> unevictable:0 dirty:205 writeback:0 unstable:0 >> slab_reclaimable:9856 slab_unreclaimable:9682 >> mapped:3722 shmem:59 pagetables:2080 bounce:0 >> free:748 free_pcp:15 free_cma:0 > there is still some page cache which doesn't seem to be neither dirty > nor under writeback. So it should be theoretically reclaimable but for > some reason we cannot seem to reclaim that memory. > There is still some anonymous memory and free swap so we could reclaim > it as well but it all seems pretty down and the memory pressure is > really large Yes, it might be large on the update situation, but that should be handled by a virtual memory system by the kernel, right? > >> [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB >> active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB >> isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB >> shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB >> unstable:0kB pages_scanned:168352 all_unreclaimable? yes > all_unreclaimable also agrees that basically nothing is reclaimable. > That was one of the criterion to hit the OOM killer prior to the rewrite > in 4.6 kernel. So I suspect that older kernels would OOM under your > memory pressure as well. See comments above. Thnx. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 16:58 ` Gerhard Wiesinger @ 2016-12-09 17:30 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2016-12-09 17:30 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: linux-kernel, linux-mm, Linus Torvalds On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: > On 09.12.2016 17:09, Michal Hocko wrote: [...] > > > [97883.882611] Mem-Info: > > > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > > > active_file:3902 inactive_file:3639 isolated_file:0 > > > unevictable:0 dirty:205 writeback:0 unstable:0 > > > slab_reclaimable:9856 slab_unreclaimable:9682 > > > mapped:3722 shmem:59 pagetables:2080 bounce:0 > > > free:748 free_pcp:15 free_cma:0 > > there is still some page cache which doesn't seem to be neither dirty > > nor under writeback. So it should be theoretically reclaimable but for > > some reason we cannot seem to reclaim that memory. > > There is still some anonymous memory and free swap so we could reclaim > > it as well but it all seems pretty down and the memory pressure is > > really large > > Yes, it might be large on the update situation, but that should be handled > by a virtual memory system by the kernel, right? Well this is what we try and call it memory reclaim. But if we are not able to reclaim anything then we eventually have to give up and trigger the OOM killer. Now the information that 4.4 made a difference is interesting. I do not really see any major differences in the reclaim between 4.3 and 4.4 kernels. The reason might be somewhere else as well. E.g. some of the subsystem consumes much more memory than before. Just curious, what kind of filesystem are you using? Could you try some additional debugging. Enabling reclaim related tracepoints might tell us more. The following should tell us more mount -t tracefs none /trace echo 1 > /trace/events/vmscan/enable echo 1 > /trace/events/writeback/writeback_congestion_wait/enable cat /trace/trace_pipe > trace.log Collecting /proc/vmstat over time might be helpful as well mkdir logs while true do cp /proc/vmstat vmstat.$(date +%s) sleep 1s done -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-09 17:30 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2016-12-09 17:30 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: linux-kernel, linux-mm, Linus Torvalds On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: > On 09.12.2016 17:09, Michal Hocko wrote: [...] > > > [97883.882611] Mem-Info: > > > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > > > active_file:3902 inactive_file:3639 isolated_file:0 > > > unevictable:0 dirty:205 writeback:0 unstable:0 > > > slab_reclaimable:9856 slab_unreclaimable:9682 > > > mapped:3722 shmem:59 pagetables:2080 bounce:0 > > > free:748 free_pcp:15 free_cma:0 > > there is still some page cache which doesn't seem to be neither dirty > > nor under writeback. So it should be theoretically reclaimable but for > > some reason we cannot seem to reclaim that memory. > > There is still some anonymous memory and free swap so we could reclaim > > it as well but it all seems pretty down and the memory pressure is > > really large > > Yes, it might be large on the update situation, but that should be handled > by a virtual memory system by the kernel, right? Well this is what we try and call it memory reclaim. But if we are not able to reclaim anything then we eventually have to give up and trigger the OOM killer. Now the information that 4.4 made a difference is interesting. I do not really see any major differences in the reclaim between 4.3 and 4.4 kernels. The reason might be somewhere else as well. E.g. some of the subsystem consumes much more memory than before. Just curious, what kind of filesystem are you using? Could you try some additional debugging. Enabling reclaim related tracepoints might tell us more. The following should tell us more mount -t tracefs none /trace echo 1 > /trace/events/vmscan/enable echo 1 > /trace/events/writeback/writeback_congestion_wait/enable cat /trace/trace_pipe > trace.log Collecting /proc/vmstat over time might be helpful as well mkdir logs while true do cp /proc/vmstat vmstat.$(date +%s) sleep 1s done -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 17:30 ` Michal Hocko @ 2016-12-09 18:01 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 18:01 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 18:30, Michal Hocko wrote: > On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: >> On 09.12.2016 17:09, Michal Hocko wrote: > [...] >>>> [97883.882611] Mem-Info: >>>> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >>>> active_file:3902 inactive_file:3639 isolated_file:0 >>>> unevictable:0 dirty:205 writeback:0 unstable:0 >>>> slab_reclaimable:9856 slab_unreclaimable:9682 >>>> mapped:3722 shmem:59 pagetables:2080 bounce:0 >>>> free:748 free_pcp:15 free_cma:0 >>> there is still some page cache which doesn't seem to be neither dirty >>> nor under writeback. So it should be theoretically reclaimable but for >>> some reason we cannot seem to reclaim that memory. >>> There is still some anonymous memory and free swap so we could reclaim >>> it as well but it all seems pretty down and the memory pressure is >>> really large >> Yes, it might be large on the update situation, but that should be handled >> by a virtual memory system by the kernel, right? > Well this is what we try and call it memory reclaim. But if we are not > able to reclaim anything then we eventually have to give up and trigger > the OOM killer. I'm not familiar with the Linux implementation of the VM system in detail. But can't you reserve as much memory for the kernel (non pageable) at least that you can swap everything out (even without killing a process at least as long there is enough swap available, which should be in all of my cases)? > Now the information that 4.4 made a difference is > interesting. I do not really see any major differences in the reclaim > between 4.3 and 4.4 kernels. The reason might be somewhere else as well. > E.g. some of the subsystem consumes much more memory than before. > > Just curious, what kind of filesystem are you using? I'm using ext4 only with virt-* drivers (storage, network). But it is definitly a virtual memory allocation/swap usage issue. > Could you try some > additional debugging. Enabling reclaim related tracepoints might tell us > more. The following should tell us more > mount -t tracefs none /trace > echo 1 > /trace/events/vmscan/enable > echo 1 > /trace/events/writeback/writeback_congestion_wait/enable > cat /trace/trace_pipe > trace.log > > Collecting /proc/vmstat over time might be helpful as well > mkdir logs > while true > do > cp /proc/vmstat vmstat.$(date +%s) > sleep 1s > done Activated it. But I think it should be very easy to trigger also on your side. A very small configured VM with a program running RAM allocations/writes (I guess you have some testing programs already) should be sufficient to trigger it. You can also use the attached program which I used to trigger such situations some years ago. If it doesn't help try to reduce the available CPU for the VM and also I/O (e.g. use all CPU/IO on the host or other VMs). BTW: Don't know if you have seen also my original message on the kernel mailinglist only: Linus had also OOM problems with 1kB RAM requests and a lot of free RAM (use a translation service for the german page): https://lkml.org/lkml/2016/11/30/64 https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ https://www.spinics.net/lists/linux-mm/msg113661.html Thnx. Ciao, Gerhard // mallocsleep.c #include <stdlib.h> #include <stdio.h> #include <unistd.h> typedef unsigned int BOOL; typedef char* PCHAR; typedef unsigned int DWORD; typedef unsigned long DDWORD; #define FALSE 0 #define TRUE 1 BOOL getlong(PCHAR s, DDWORD* retvalue) { char *eptr; long value; value=strtoll(s,&eptr,0); if ((eptr == s)||(*eptr != '\0')) return FALSE; if (value < 0) return FALSE; *retvalue = value; return TRUE; } int main(int argc, char* argv[]) { unsigned long* p; unsigned long size = 16*1024*1024; unsigned long size_of = sizeof(*p); unsigned long i; unsigned long sleep_allocated = 3600; unsigned long sleep_freed = 3600; if (argc > 1) { if (!getlong(argv[1], &size)) { printf("Wrong memsize!\n"); exit(1); } } if (argc > 2) { if (!getlong(argv[2], &sleep_allocated)) { printf("Wrong sleep_allocated time!\n"); exit(1); } } if (argc > 3) { if (!getlong(argv[3], &sleep_freed)) { printf("Wrong sleep_freed time!\n"); exit(1); } } printf("size=%lu, size_of=%lu\n", size, size_of); fflush(stdout); p = malloc(size); if (!p) { printf("Could not allocate memory!\n"); exit(2); } printf("malloc done, writing to memory, p=%p ...\n", (void*)p); fflush(stdout); for(i = 0;i < (size/size_of);i++) p[i]=i; printf("writing to memory done, sleeping for %lu seconds ...\n", sleep_allocated); fflush(stdout); sleep(sleep_allocated); printf("sleeping done, freeing ...\n"); fflush(stdout); free(p); printf("freeing done, sleeping for %lu seconds ...\n", sleep_freed); fflush(stdout); sleep(sleep_freed); printf("sleeping done, exitiing ...\n"); fflush(stdout); exit(0); return 0; } ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-09 18:01 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 18:01 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 18:30, Michal Hocko wrote: > On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: >> On 09.12.2016 17:09, Michal Hocko wrote: > [...] >>>> [97883.882611] Mem-Info: >>>> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >>>> active_file:3902 inactive_file:3639 isolated_file:0 >>>> unevictable:0 dirty:205 writeback:0 unstable:0 >>>> slab_reclaimable:9856 slab_unreclaimable:9682 >>>> mapped:3722 shmem:59 pagetables:2080 bounce:0 >>>> free:748 free_pcp:15 free_cma:0 >>> there is still some page cache which doesn't seem to be neither dirty >>> nor under writeback. So it should be theoretically reclaimable but for >>> some reason we cannot seem to reclaim that memory. >>> There is still some anonymous memory and free swap so we could reclaim >>> it as well but it all seems pretty down and the memory pressure is >>> really large >> Yes, it might be large on the update situation, but that should be handled >> by a virtual memory system by the kernel, right? > Well this is what we try and call it memory reclaim. But if we are not > able to reclaim anything then we eventually have to give up and trigger > the OOM killer. I'm not familiar with the Linux implementation of the VM system in detail. But can't you reserve as much memory for the kernel (non pageable) at least that you can swap everything out (even without killing a process at least as long there is enough swap available, which should be in all of my cases)? > Now the information that 4.4 made a difference is > interesting. I do not really see any major differences in the reclaim > between 4.3 and 4.4 kernels. The reason might be somewhere else as well. > E.g. some of the subsystem consumes much more memory than before. > > Just curious, what kind of filesystem are you using? I'm using ext4 only with virt-* drivers (storage, network). But it is definitly a virtual memory allocation/swap usage issue. > Could you try some > additional debugging. Enabling reclaim related tracepoints might tell us > more. The following should tell us more > mount -t tracefs none /trace > echo 1 > /trace/events/vmscan/enable > echo 1 > /trace/events/writeback/writeback_congestion_wait/enable > cat /trace/trace_pipe > trace.log > > Collecting /proc/vmstat over time might be helpful as well > mkdir logs > while true > do > cp /proc/vmstat vmstat.$(date +%s) > sleep 1s > done Activated it. But I think it should be very easy to trigger also on your side. A very small configured VM with a program running RAM allocations/writes (I guess you have some testing programs already) should be sufficient to trigger it. You can also use the attached program which I used to trigger such situations some years ago. If it doesn't help try to reduce the available CPU for the VM and also I/O (e.g. use all CPU/IO on the host or other VMs). BTW: Don't know if you have seen also my original message on the kernel mailinglist only: Linus had also OOM problems with 1kB RAM requests and a lot of free RAM (use a translation service for the german page): https://lkml.org/lkml/2016/11/30/64 https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ https://www.spinics.net/lists/linux-mm/msg113661.html Thnx. Ciao, Gerhard // mallocsleep.c #include <stdlib.h> #include <stdio.h> #include <unistd.h> typedef unsigned int BOOL; typedef char* PCHAR; typedef unsigned int DWORD; typedef unsigned long DDWORD; #define FALSE 0 #define TRUE 1 BOOL getlong(PCHAR s, DDWORD* retvalue) { char *eptr; long value; value=strtoll(s,&eptr,0); if ((eptr == s)||(*eptr != '\0')) return FALSE; if (value < 0) return FALSE; *retvalue = value; return TRUE; } int main(int argc, char* argv[]) { unsigned long* p; unsigned long size = 16*1024*1024; unsigned long size_of = sizeof(*p); unsigned long i; unsigned long sleep_allocated = 3600; unsigned long sleep_freed = 3600; if (argc > 1) { if (!getlong(argv[1], &size)) { printf("Wrong memsize!\n"); exit(1); } } if (argc > 2) { if (!getlong(argv[2], &sleep_allocated)) { printf("Wrong sleep_allocated time!\n"); exit(1); } } if (argc > 3) { if (!getlong(argv[3], &sleep_freed)) { printf("Wrong sleep_freed time!\n"); exit(1); } } printf("size=%lu, size_of=%lu\n", size, size_of); fflush(stdout); p = malloc(size); if (!p) { printf("Could not allocate memory!\n"); exit(2); } printf("malloc done, writing to memory, p=%p ...\n", (void*)p); fflush(stdout); for(i = 0;i < (size/size_of);i++) p[i]=i; printf("writing to memory done, sleeping for %lu seconds ...\n", sleep_allocated); fflush(stdout); sleep(sleep_allocated); printf("sleeping done, freeing ...\n"); fflush(stdout); free(p); printf("freeing done, sleeping for %lu seconds ...\n", sleep_freed); fflush(stdout); sleep(sleep_freed); printf("sleeping done, exitiing ...\n"); fflush(stdout); exit(0); return 0; } -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 18:01 ` Gerhard Wiesinger @ 2016-12-09 21:42 ` Vlastimil Babka -1 siblings, 0 replies; 88+ messages in thread From: Vlastimil Babka @ 2016-12-09 21:42 UTC (permalink / raw) To: Gerhard Wiesinger, Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 12/09/2016 07:01 PM, Gerhard Wiesinger wrote: > On 09.12.2016 18:30, Michal Hocko wrote: >> On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: >>> On 09.12.2016 17:09, Michal Hocko wrote: >> [...] >>>>> [97883.882611] Mem-Info: >>>>> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >>>>> active_file:3902 inactive_file:3639 isolated_file:0 >>>>> unevictable:0 dirty:205 writeback:0 unstable:0 >>>>> slab_reclaimable:9856 slab_unreclaimable:9682 >>>>> mapped:3722 shmem:59 pagetables:2080 bounce:0 >>>>> free:748 free_pcp:15 free_cma:0 >>>> there is still some page cache which doesn't seem to be neither dirty >>>> nor under writeback. So it should be theoretically reclaimable but for >>>> some reason we cannot seem to reclaim that memory. >>>> There is still some anonymous memory and free swap so we could reclaim >>>> it as well but it all seems pretty down and the memory pressure is >>>> really large >>> Yes, it might be large on the update situation, but that should be handled >>> by a virtual memory system by the kernel, right? >> Well this is what we try and call it memory reclaim. But if we are not >> able to reclaim anything then we eventually have to give up and trigger >> the OOM killer. > > I'm not familiar with the Linux implementation of the VM system in > detail. But can't you reserve as much memory for the kernel (non > pageable) at least that you can swap everything out (even without > killing a process at least as long there is enough swap available, which > should be in all of my cases)? We don't have such bulletproof reserves. In this case the amount of anonymous memory that can be swapped out is relatively low, and either something is pinning it in memory, or it's being swapped back in quickly. >> Now the information that 4.4 made a difference is >> interesting. I do not really see any major differences in the reclaim >> between 4.3 and 4.4 kernels. The reason might be somewhere else as well. >> E.g. some of the subsystem consumes much more memory than before. >> >> Just curious, what kind of filesystem are you using? > > I'm using ext4 only with virt-* drivers (storage, network). But it is > definitly a virtual memory allocation/swap usage issue. > >> Could you try some >> additional debugging. Enabling reclaim related tracepoints might tell us >> more. The following should tell us more >> mount -t tracefs none /trace >> echo 1 > /trace/events/vmscan/enable >> echo 1 > /trace/events/writeback/writeback_congestion_wait/enable >> cat /trace/trace_pipe > trace.log >> >> Collecting /proc/vmstat over time might be helpful as well >> mkdir logs >> while true >> do >> cp /proc/vmstat vmstat.$(date +%s) >> sleep 1s >> done > > Activated it. But I think it should be very easy to trigger also on your > side. A very small configured VM with a program running RAM > allocations/writes (I guess you have some testing programs already) > should be sufficient to trigger it. You can also use the attached > program which I used to trigger such situations some years ago. If it > doesn't help try to reduce the available CPU for the VM and also I/O > (e.g. use all CPU/IO on the host or other VMs). Well it's not really a surprise that if the VM is small enough and workload large enough, OOM killer will kick in. The exact threshold might have changed between kernel versions for a number of possible reasons. > > BTW: Don't know if you have seen also my original message on the kernel > mailinglist only: > > Linus had also OOM problems with 1kB RAM requests and a lot of free RAM > (use a translation service for the german page): > https://lkml.org/lkml/2016/11/30/64 > https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ > https://www.spinics.net/lists/linux-mm/msg113661.html Yeah we were involved in the last one. The regressions were about high-order allocations though (the 1kB premise turned out to be misinterpretation) and there were regressions for those in 4.7/4.8. But yours are order-0. ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-09 21:42 ` Vlastimil Babka 0 siblings, 0 replies; 88+ messages in thread From: Vlastimil Babka @ 2016-12-09 21:42 UTC (permalink / raw) To: Gerhard Wiesinger, Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 12/09/2016 07:01 PM, Gerhard Wiesinger wrote: > On 09.12.2016 18:30, Michal Hocko wrote: >> On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: >>> On 09.12.2016 17:09, Michal Hocko wrote: >> [...] >>>>> [97883.882611] Mem-Info: >>>>> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >>>>> active_file:3902 inactive_file:3639 isolated_file:0 >>>>> unevictable:0 dirty:205 writeback:0 unstable:0 >>>>> slab_reclaimable:9856 slab_unreclaimable:9682 >>>>> mapped:3722 shmem:59 pagetables:2080 bounce:0 >>>>> free:748 free_pcp:15 free_cma:0 >>>> there is still some page cache which doesn't seem to be neither dirty >>>> nor under writeback. So it should be theoretically reclaimable but for >>>> some reason we cannot seem to reclaim that memory. >>>> There is still some anonymous memory and free swap so we could reclaim >>>> it as well but it all seems pretty down and the memory pressure is >>>> really large >>> Yes, it might be large on the update situation, but that should be handled >>> by a virtual memory system by the kernel, right? >> Well this is what we try and call it memory reclaim. But if we are not >> able to reclaim anything then we eventually have to give up and trigger >> the OOM killer. > > I'm not familiar with the Linux implementation of the VM system in > detail. But can't you reserve as much memory for the kernel (non > pageable) at least that you can swap everything out (even without > killing a process at least as long there is enough swap available, which > should be in all of my cases)? We don't have such bulletproof reserves. In this case the amount of anonymous memory that can be swapped out is relatively low, and either something is pinning it in memory, or it's being swapped back in quickly. >> Now the information that 4.4 made a difference is >> interesting. I do not really see any major differences in the reclaim >> between 4.3 and 4.4 kernels. The reason might be somewhere else as well. >> E.g. some of the subsystem consumes much more memory than before. >> >> Just curious, what kind of filesystem are you using? > > I'm using ext4 only with virt-* drivers (storage, network). But it is > definitly a virtual memory allocation/swap usage issue. > >> Could you try some >> additional debugging. Enabling reclaim related tracepoints might tell us >> more. The following should tell us more >> mount -t tracefs none /trace >> echo 1 > /trace/events/vmscan/enable >> echo 1 > /trace/events/writeback/writeback_congestion_wait/enable >> cat /trace/trace_pipe > trace.log >> >> Collecting /proc/vmstat over time might be helpful as well >> mkdir logs >> while true >> do >> cp /proc/vmstat vmstat.$(date +%s) >> sleep 1s >> done > > Activated it. But I think it should be very easy to trigger also on your > side. A very small configured VM with a program running RAM > allocations/writes (I guess you have some testing programs already) > should be sufficient to trigger it. You can also use the attached > program which I used to trigger such situations some years ago. If it > doesn't help try to reduce the available CPU for the VM and also I/O > (e.g. use all CPU/IO on the host or other VMs). Well it's not really a surprise that if the VM is small enough and workload large enough, OOM killer will kick in. The exact threshold might have changed between kernel versions for a number of possible reasons. > > BTW: Don't know if you have seen also my original message on the kernel > mailinglist only: > > Linus had also OOM problems with 1kB RAM requests and a lot of free RAM > (use a translation service for the german page): > https://lkml.org/lkml/2016/11/30/64 > https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ > https://www.spinics.net/lists/linux-mm/msg113661.html Yeah we were involved in the last one. The regressions were about high-order allocations though (the 1kB premise turned out to be misinterpretation) and there were regressions for those in 4.7/4.8. But yours are order-0. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 21:42 ` Vlastimil Babka @ 2016-12-10 13:50 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-10 13:50 UTC (permalink / raw) To: Vlastimil Babka, Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 22:42, Vlastimil Babka wrote: > On 12/09/2016 07:01 PM, Gerhard Wiesinger wrote: >> On 09.12.2016 18:30, Michal Hocko wrote: >>> On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: >>>> On 09.12.2016 17:09, Michal Hocko wrote: >>> [...] >>>>>> [97883.882611] Mem-Info: >>>>>> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >>>>>> active_file:3902 inactive_file:3639 isolated_file:0 >>>>>> unevictable:0 dirty:205 writeback:0 unstable:0 >>>>>> slab_reclaimable:9856 slab_unreclaimable:9682 >>>>>> mapped:3722 shmem:59 pagetables:2080 bounce:0 >>>>>> free:748 free_pcp:15 free_cma:0 >>>>> there is still some page cache which doesn't seem to be neither dirty >>>>> nor under writeback. So it should be theoretically reclaimable but for >>>>> some reason we cannot seem to reclaim that memory. >>>>> There is still some anonymous memory and free swap so we could reclaim >>>>> it as well but it all seems pretty down and the memory pressure is >>>>> really large >>>> Yes, it might be large on the update situation, but that should be handled >>>> by a virtual memory system by the kernel, right? >>> Well this is what we try and call it memory reclaim. But if we are not >>> able to reclaim anything then we eventually have to give up and trigger >>> the OOM killer. >> I'm not familiar with the Linux implementation of the VM system in >> detail. But can't you reserve as much memory for the kernel (non >> pageable) at least that you can swap everything out (even without >> killing a process at least as long there is enough swap available, which >> should be in all of my cases)? > We don't have such bulletproof reserves. In this case the amount of > anonymous memory that can be swapped out is relatively low, and either > something is pinning it in memory, or it's being swapped back in quickly. > >>> Now the information that 4.4 made a difference is >>> interesting. I do not really see any major differences in the reclaim >>> between 4.3 and 4.4 kernels. The reason might be somewhere else as well. >>> E.g. some of the subsystem consumes much more memory than before. >>> >>> Just curious, what kind of filesystem are you using? >> I'm using ext4 only with virt-* drivers (storage, network). But it is >> definitly a virtual memory allocation/swap usage issue. >> >>> Could you try some >>> additional debugging. Enabling reclaim related tracepoints might tell us >>> more. The following should tell us more >>> mount -t tracefs none /trace >>> echo 1 > /trace/events/vmscan/enable >>> echo 1 > /trace/events/writeback/writeback_congestion_wait/enable >>> cat /trace/trace_pipe > trace.log >>> >>> Collecting /proc/vmstat over time might be helpful as well >>> mkdir logs >>> while true >>> do >>> cp /proc/vmstat vmstat.$(date +%s) >>> sleep 1s >>> done >> Activated it. But I think it should be very easy to trigger also on your >> side. A very small configured VM with a program running RAM >> allocations/writes (I guess you have some testing programs already) >> should be sufficient to trigger it. You can also use the attached >> program which I used to trigger such situations some years ago. If it >> doesn't help try to reduce the available CPU for the VM and also I/O >> (e.g. use all CPU/IO on the host or other VMs). > Well it's not really a surprise that if the VM is small enough and > workload large enough, OOM killer will kick in. The exact threshold > might have changed between kernel versions for a number of possible reasons. IMHO: The OOM killer should NOT kick in even on the highest workloads if there is swap available. https://www.spinics.net/lists/linux-mm/msg113665.html Yeah, but I do think that "oom when you have 156MB free and 7GB reclaimable, and haven't even tried swapping" counts as obviously wrong. So Linus also thinks that trying swapping is a must have. And there always was enough swap available in my cases. Then it should swap out/swapin all the time (which worked well in kernel 2.4/2.6 times). Another topic: Why does the kernel prefer to swap in/swap out instead of use cache pages/buffers (see vmstat 1 output below)? > >> BTW: Don't know if you have seen also my original message on the kernel >> mailinglist only: >> >> Linus had also OOM problems with 1kB RAM requests and a lot of free RAM >> (use a translation service for the german page): >> https://lkml.org/lkml/2016/11/30/64 >> https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ >> https://www.spinics.net/lists/linux-mm/msg113661.html > Yeah we were involved in the last one. The regressions were about > high-order allocations > though (the 1kB premise turned out to be misinterpretation) and there > were regressions > for those in 4.7/4.8. But yours are order-0. > With kernel 4.7./4.8 it was really reaproduceable at every dnf update. With 4.9rc8 it has been much much better. So something must have changed, too. As far as I understood it the order is 2^order kB pagesize. I don't think it makes a difference when swap is not used which order the memory allocation request is. BTW: What were the commit that introduced the regression anf fixed it in 4.9? Thnx. Ciao, Gerhard procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 45232 9252 1956 109644 428 232 3536 416 4310 4228 38 36 14 7 6 2 0 45124 10524 1960 110192 124 0 528 96 2478 2243 45 29 20 5 1 4 1 45136 3896 1968 114388 84 64 4824 260 2689 2655 38 31 15 12 4 1 1 45484 10648 288 114032 88 356 20028 1132 5078 5122 24 45 4 21 5 2 0 44700 8092 1240 115204 728 0 2624 536 4204 4413 38 38 18 3 4 2 0 44852 10272 1240 111324 52 212 2736 1548 3311 2970 41 36 12 9 2 4 0 44844 10716 1240 111216 8 0 8 72 3067 3287 42 30 18 7 3 3 0 44828 10268 1248 111280 16 0 16 60 2139 1610 43 29 11 1 17 1 0 44828 11644 1248 111192 0 0 0 0 2367 1911 50 32 14 0 3 4 0 44820 9004 1248 111284 8 0 8 0 2207 1867 55 31 14 0 1 7 0 45664 6360 1816 109264 20 868 3076 968 4122 3783 43 37 17 0 3 4 4 46880 6732 1092 101960 244 1332 7968 3352 5836 6431 17 51 1 27 4 10 2 47064 6940 1364 96340 20 196 25708 1720 7346 6447 13 70 0 18 1 15 3 47572 3672 2156 92604 68 580 29244 1692 5640 5102 5 57 0 37 2 12 4 48300 6740 352 87924 80 948 36208 2948 7287 7955 7 73 0 18 2 12 9 50796 4832 584 88372 0 2496 16064 3312 3425 4185 2 30 0 66 1 10 9 52636 3608 2068 90132 56 1840 24552 2836 4123 4099 3 43 0 52 1 7 11 56740 10376 424 86204 184 4152 33116 5628 7949 7952 4 67 0 23 6 10 4 61384 8000 776 86956 644 4784 28380 5484 7965 9935 7 64 0 26 2 11 4 68052 5260 1028 87268 1244 7164 23380 8684 10715 10863 8 71 0 20 1 11 2 72244 3924 1052 85160 980 4264 23756 4940 7231 7930 8 62 0 29 1 6 1 76388 5352 4948 86204 1292 4640 27380 5244 7816 8714 10 63 0 22 5 8 5 77376 4168 1944 86528 3064 3684 19876 4104 9325 9076 9 64 1 22 4 5 4 75464 7272 1240 81684 3912 3188 25656 4100 9973 10515 11 65 0 20 4 5 2 77364 4440 1852 84744 528 2304 28588 3304 6605 6311 7 61 8 18 4 9 2 81648 3760 3188 86012 440 4588 17928 5368 6377 6320 8 48 2 40 4 6 2 82404 6608 668 86092 2016 2084 24396 3564 7440 7510 8 66 1 20 4 4 4 81728 3796 2260 87764 1392 984 18512 1684 5196 4652 6 48 0 42 4 8 4 84700 6436 1428 85744 1188 3708 20256 4364 6405 5998 9 63 0 24 4 3 1 86360 4836 924 87700 1388 2692 19460 3504 5498 6117 8 48 0 34 9 4 4 87916 3768 176 86592 2788 3220 19664 4032 7285 8342 19 63 0 10 9 4 4 89612 4952 180 88076 1516 2988 17560 3936 5737 5794 7 46 0 37 10 7 5 87768 12244 196 87856 3344 2544 22248 3348 6934 7497 8 59 0 22 10 10 1 83436 4768 840 96452 4096 836 20100 1160 6191 6614 21 52 0 13 14 0 6 82868 6972 348 91020 1108 520 4896 568 3274 4214 11 26 29 30 4 ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-10 13:50 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-10 13:50 UTC (permalink / raw) To: Vlastimil Babka, Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 22:42, Vlastimil Babka wrote: > On 12/09/2016 07:01 PM, Gerhard Wiesinger wrote: >> On 09.12.2016 18:30, Michal Hocko wrote: >>> On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: >>>> On 09.12.2016 17:09, Michal Hocko wrote: >>> [...] >>>>>> [97883.882611] Mem-Info: >>>>>> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >>>>>> active_file:3902 inactive_file:3639 isolated_file:0 >>>>>> unevictable:0 dirty:205 writeback:0 unstable:0 >>>>>> slab_reclaimable:9856 slab_unreclaimable:9682 >>>>>> mapped:3722 shmem:59 pagetables:2080 bounce:0 >>>>>> free:748 free_pcp:15 free_cma:0 >>>>> there is still some page cache which doesn't seem to be neither dirty >>>>> nor under writeback. So it should be theoretically reclaimable but for >>>>> some reason we cannot seem to reclaim that memory. >>>>> There is still some anonymous memory and free swap so we could reclaim >>>>> it as well but it all seems pretty down and the memory pressure is >>>>> really large >>>> Yes, it might be large on the update situation, but that should be handled >>>> by a virtual memory system by the kernel, right? >>> Well this is what we try and call it memory reclaim. But if we are not >>> able to reclaim anything then we eventually have to give up and trigger >>> the OOM killer. >> I'm not familiar with the Linux implementation of the VM system in >> detail. But can't you reserve as much memory for the kernel (non >> pageable) at least that you can swap everything out (even without >> killing a process at least as long there is enough swap available, which >> should be in all of my cases)? > We don't have such bulletproof reserves. In this case the amount of > anonymous memory that can be swapped out is relatively low, and either > something is pinning it in memory, or it's being swapped back in quickly. > >>> Now the information that 4.4 made a difference is >>> interesting. I do not really see any major differences in the reclaim >>> between 4.3 and 4.4 kernels. The reason might be somewhere else as well. >>> E.g. some of the subsystem consumes much more memory than before. >>> >>> Just curious, what kind of filesystem are you using? >> I'm using ext4 only with virt-* drivers (storage, network). But it is >> definitly a virtual memory allocation/swap usage issue. >> >>> Could you try some >>> additional debugging. Enabling reclaim related tracepoints might tell us >>> more. The following should tell us more >>> mount -t tracefs none /trace >>> echo 1 > /trace/events/vmscan/enable >>> echo 1 > /trace/events/writeback/writeback_congestion_wait/enable >>> cat /trace/trace_pipe > trace.log >>> >>> Collecting /proc/vmstat over time might be helpful as well >>> mkdir logs >>> while true >>> do >>> cp /proc/vmstat vmstat.$(date +%s) >>> sleep 1s >>> done >> Activated it. But I think it should be very easy to trigger also on your >> side. A very small configured VM with a program running RAM >> allocations/writes (I guess you have some testing programs already) >> should be sufficient to trigger it. You can also use the attached >> program which I used to trigger such situations some years ago. If it >> doesn't help try to reduce the available CPU for the VM and also I/O >> (e.g. use all CPU/IO on the host or other VMs). > Well it's not really a surprise that if the VM is small enough and > workload large enough, OOM killer will kick in. The exact threshold > might have changed between kernel versions for a number of possible reasons. IMHO: The OOM killer should NOT kick in even on the highest workloads if there is swap available. https://www.spinics.net/lists/linux-mm/msg113665.html Yeah, but I do think that "oom when you have 156MB free and 7GB reclaimable, and haven't even tried swapping" counts as obviously wrong. So Linus also thinks that trying swapping is a must have. And there always was enough swap available in my cases. Then it should swap out/swapin all the time (which worked well in kernel 2.4/2.6 times). Another topic: Why does the kernel prefer to swap in/swap out instead of use cache pages/buffers (see vmstat 1 output below)? > >> BTW: Don't know if you have seen also my original message on the kernel >> mailinglist only: >> >> Linus had also OOM problems with 1kB RAM requests and a lot of free RAM >> (use a translation service for the german page): >> https://lkml.org/lkml/2016/11/30/64 >> https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/ >> https://www.spinics.net/lists/linux-mm/msg113661.html > Yeah we were involved in the last one. The regressions were about > high-order allocations > though (the 1kB premise turned out to be misinterpretation) and there > were regressions > for those in 4.7/4.8. But yours are order-0. > With kernel 4.7./4.8 it was really reaproduceable at every dnf update. With 4.9rc8 it has been much much better. So something must have changed, too. As far as I understood it the order is 2^order kB pagesize. I don't think it makes a difference when swap is not used which order the memory allocation request is. BTW: What were the commit that introduced the regression anf fixed it in 4.9? Thnx. Ciao, Gerhard procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 45232 9252 1956 109644 428 232 3536 416 4310 4228 38 36 14 7 6 2 0 45124 10524 1960 110192 124 0 528 96 2478 2243 45 29 20 5 1 4 1 45136 3896 1968 114388 84 64 4824 260 2689 2655 38 31 15 12 4 1 1 45484 10648 288 114032 88 356 20028 1132 5078 5122 24 45 4 21 5 2 0 44700 8092 1240 115204 728 0 2624 536 4204 4413 38 38 18 3 4 2 0 44852 10272 1240 111324 52 212 2736 1548 3311 2970 41 36 12 9 2 4 0 44844 10716 1240 111216 8 0 8 72 3067 3287 42 30 18 7 3 3 0 44828 10268 1248 111280 16 0 16 60 2139 1610 43 29 11 1 17 1 0 44828 11644 1248 111192 0 0 0 0 2367 1911 50 32 14 0 3 4 0 44820 9004 1248 111284 8 0 8 0 2207 1867 55 31 14 0 1 7 0 45664 6360 1816 109264 20 868 3076 968 4122 3783 43 37 17 0 3 4 4 46880 6732 1092 101960 244 1332 7968 3352 5836 6431 17 51 1 27 4 10 2 47064 6940 1364 96340 20 196 25708 1720 7346 6447 13 70 0 18 1 15 3 47572 3672 2156 92604 68 580 29244 1692 5640 5102 5 57 0 37 2 12 4 48300 6740 352 87924 80 948 36208 2948 7287 7955 7 73 0 18 2 12 9 50796 4832 584 88372 0 2496 16064 3312 3425 4185 2 30 0 66 1 10 9 52636 3608 2068 90132 56 1840 24552 2836 4123 4099 3 43 0 52 1 7 11 56740 10376 424 86204 184 4152 33116 5628 7949 7952 4 67 0 23 6 10 4 61384 8000 776 86956 644 4784 28380 5484 7965 9935 7 64 0 26 2 11 4 68052 5260 1028 87268 1244 7164 23380 8684 10715 10863 8 71 0 20 1 11 2 72244 3924 1052 85160 980 4264 23756 4940 7231 7930 8 62 0 29 1 6 1 76388 5352 4948 86204 1292 4640 27380 5244 7816 8714 10 63 0 22 5 8 5 77376 4168 1944 86528 3064 3684 19876 4104 9325 9076 9 64 1 22 4 5 4 75464 7272 1240 81684 3912 3188 25656 4100 9973 10515 11 65 0 20 4 5 2 77364 4440 1852 84744 528 2304 28588 3304 6605 6311 7 61 8 18 4 9 2 81648 3760 3188 86012 440 4588 17928 5368 6377 6320 8 48 2 40 4 6 2 82404 6608 668 86092 2016 2084 24396 3564 7440 7510 8 66 1 20 4 4 4 81728 3796 2260 87764 1392 984 18512 1684 5196 4652 6 48 0 42 4 8 4 84700 6436 1428 85744 1188 3708 20256 4364 6405 5998 9 63 0 24 4 3 1 86360 4836 924 87700 1388 2692 19460 3504 5498 6117 8 48 0 34 9 4 4 87916 3768 176 86592 2788 3220 19664 4032 7285 8342 19 63 0 10 9 4 4 89612 4952 180 88076 1516 2988 17560 3936 5737 5794 7 46 0 37 10 7 5 87768 12244 196 87856 3344 2544 22248 3348 6934 7497 8 59 0 22 10 10 1 83436 4768 840 96452 4096 836 20100 1160 6191 6614 21 52 0 13 14 0 6 82868 6972 348 91020 1108 520 4896 568 3274 4214 11 26 29 30 4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-10 13:50 ` Gerhard Wiesinger @ 2016-12-12 8:24 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2016-12-12 8:24 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Vlastimil Babka, linux-kernel, linux-mm, Linus Torvalds On Sat 10-12-16 14:50:34, Gerhard Wiesinger wrote: [...] > IMHO: The OOM killer should NOT kick in even on the highest workloads if > there is swap available. This is not so simple. Take a heavy swap trashing situation as an example. You still have a lot of swap space and an anonymous memory which can be swapped out but if the workload simply keeps refaulting the swapped out memory all the time then you can barely make any further progress and end up in the swap IO hell. Invoking the OOM killer in such a situation would be a relief to make the system usable again. > https://www.spinics.net/lists/linux-mm/msg113665.html > > Yeah, but I do think that "oom when you have 156MB free and 7GB > reclaimable, and haven't even tried swapping" counts as obviously > wrong. No question about this part of course. > So Linus also thinks that trying swapping is a must have. And there > always was enough swap available in my cases. Then it should swap > out/swapin all the time (which worked well in kernel 2.4/2.6 times). > > Another topic: Why does the kernel prefer to swap in/swap out instead of use > cache pages/buffers (see vmstat 1 output below)? In the vast majority cases it is quite contrary. We heavily bias page cache reclaim in favor of the anonymous memory. Have a look at get_scan_count function which determines the balance. I would need to see /proc/vmstat collected during this time period to tell you more about why the particular balance was used. [...] > With kernel 4.7./4.8 it was really reaproduceable at every dnf update. With > 4.9rc8 it has been much much better. So something must have changed, too. that is good to hear but I it would be much better to collect reclaim related data so that we can analyse what is actually going on here. > As far as I understood it the order is 2^order kB pagesize. I don't think it > makes a difference when swap is not used which order the memory allocation > request is. > > BTW: What were the commit that introduced the regression anf fixed it in > 4.9? I cannot really answer this without actually understanding what is going on here and for that I do not have the relevant data. -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-12 8:24 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2016-12-12 8:24 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Vlastimil Babka, linux-kernel, linux-mm, Linus Torvalds On Sat 10-12-16 14:50:34, Gerhard Wiesinger wrote: [...] > IMHO: The OOM killer should NOT kick in even on the highest workloads if > there is swap available. This is not so simple. Take a heavy swap trashing situation as an example. You still have a lot of swap space and an anonymous memory which can be swapped out but if the workload simply keeps refaulting the swapped out memory all the time then you can barely make any further progress and end up in the swap IO hell. Invoking the OOM killer in such a situation would be a relief to make the system usable again. > https://www.spinics.net/lists/linux-mm/msg113665.html > > Yeah, but I do think that "oom when you have 156MB free and 7GB > reclaimable, and haven't even tried swapping" counts as obviously > wrong. No question about this part of course. > So Linus also thinks that trying swapping is a must have. And there > always was enough swap available in my cases. Then it should swap > out/swapin all the time (which worked well in kernel 2.4/2.6 times). > > Another topic: Why does the kernel prefer to swap in/swap out instead of use > cache pages/buffers (see vmstat 1 output below)? In the vast majority cases it is quite contrary. We heavily bias page cache reclaim in favor of the anonymous memory. Have a look at get_scan_count function which determines the balance. I would need to see /proc/vmstat collected during this time period to tell you more about why the particular balance was used. [...] > With kernel 4.7./4.8 it was really reaproduceable at every dnf update. With > 4.9rc8 it has been much much better. So something must have changed, too. that is good to hear but I it would be much better to collect reclaim related data so that we can analyse what is actually going on here. > As far as I understood it the order is 2^order kB pagesize. I don't think it > makes a difference when swap is not used which order the memory allocation > request is. > > BTW: What were the commit that introduced the regression anf fixed it in > 4.9? I cannot really answer this without actually understanding what is going on here and for that I do not have the relevant data. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 15:52 ` Gerhard Wiesinger @ 2016-12-23 2:55 ` Minchan Kim -1 siblings, 0 replies; 88+ messages in thread From: Minchan Kim @ 2016-12-23 2:55 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Michal Hocko, linux-kernel, linux-mm, Linus Torvalds On Fri, Dec 09, 2016 at 04:52:07PM +0100, Gerhard Wiesinger wrote: > On 09.12.2016 14:40, Michal Hocko wrote: > >On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: > >>Hello, > >> > >>same with latest kernel rc, dnf still killed with OOM (but sometimes > >>better). > >> > >>./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} > >>(does dnf clean all;dnf update) > >>Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 > >>17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > >> > >>Updated bug report: > >>https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > >Could you post your oom report please? > > E.g. a new one with more than one included, first one after boot ... > > Just setup a low mem VM under KVM and it is easily triggerable. > > Still enough virtual memory available ... > > 4.9.0-0.rc8.git2.1.fc26.x86_64 > > [ 624.862777] ksoftirqd/0: page allocation failure: order:0, > mode:0x2080020(GFP_ATOMIC) > [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > 1.9.3 > [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 > 0000000000000000 > [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 > ffffffff90c7dd98 > [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 > ffffaa62c007f9a0 > [ 624.864998] Call Trace: > [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 > [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 > [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 > [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 > [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 > [virtio_net] > [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 > [virtio_net] > [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] > [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 > [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 > [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 > [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 > [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 > [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 > [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 > [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 > [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 > [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 > [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 > [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left > [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left > [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left > [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 bytes > left > [95895.765570] kworker/1:1H: page allocation failure: order:0, > mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) > [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > 1.9.3 > [95895.766060] Workqueue: kblockd blk_mq_run_work_fn > [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 > 0000000000000000 > [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 > ffffffff90c7dd98 > [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 > ffffaa62c0257670 > [95895.766417] Call Trace: > [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 > [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 > [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 > [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 > [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 > [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 > [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 > [95895.773406] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 > [95895.775212] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 > [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 > [95895.781502] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 > [virtio_ring] > [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 > [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 > [virtio_blk] > [95895.785419] [<ffffffff901312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 > [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 > [virtio_blk] > [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 > [virtio_blk] > [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 > [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 > [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 > [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 > [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 > [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 > [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 > [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 > [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 > [95895.794212] [<ffffffff90111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 > [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 > [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 > [95895.796408] Mem-Info: > [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 > active_file:263 inactive_file:238 isolated_file:0 > unevictable:0 dirty:0 writeback:330 unstable:0 > slab_reclaimable:5241 slab_unreclaimable:9538 > mapped:470 shmem:9 pagetables:2200 bounce:0 > free:690 free_pcp:68 free_cma:0 > [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB > active_file:1052kB inactive_file:952kB unevictable:0kB isolated(anon):128kB > isolated(file):0kB mapped:1880kB dirty:0kB writeback:1320kB shmem:0kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB > unstable:0kB pages_scanned:179 all_unreclaimable? no > [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB > active_anon:3544kB inactive_anon:3944kB active_file:84kB inactive_file:140kB > unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB > slab_reclaimable:1728kB slab_unreclaimable:2964kB kernel_stack:84kB > pagetables:1396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [95895.805936] lowmem_reserve[]: 0 117 117 117 117 > [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB high:1944kB > active_anon:31636kB inactive_anon:32164kB active_file:968kB > inactive_file:804kB unevictable:0kB writepending:1288kB present:180080kB > managed:139012kB mlocked:0kB slab_reclaimable:19236kB > slab_unreclaimable:35188kB kernel_stack:1852kB pagetables:7404kB bounce:0kB > free_pcp:272kB local_pcp:156kB free_cma:0kB > [95895.809223] lowmem_reserve[]: 0 0 0 0 0 > [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB (H) > 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB > [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB (H) > 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1836kB > [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 > hugepages_size=2048kB > [95895.813981] 1804 total pagecache pages > [95895.814931] 1289 pages in swap cache > [95895.815849] Swap cache stats: add 5288014, delete 5286725, find > 11568655/13881082 > [95895.816792] Free swap = 1791816kB > [95895.817706] Total swap = 2064380kB > [95895.819222] 49018 pages RAM > [95895.820145] 0 pages HighMem/MovableOnly > [95895.821039] 10288 pages reserved > [95895.823325] 0 pages cma reserved > [95895.824244] 0 pages hwpoisoned > [95895.825237] SLUB: Unable to allocate memory on node -1, > gfp=0x2080020(GFP_ATOMIC) > [95895.826140] cache: kmalloc-256, object size: 256, buffer size: 256, > default order: 0, min order: 0 > [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 > [97883.838418] httpd invoked oom-killer: > gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, > oom_score_adj=0 > [97883.843507] httpd cpuset=/ mems_allowed=0 > [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > 1.9.3 > [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 > ffff96b98b8b3100 > [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 > 0000000000000000 > [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d > ffffaa62c395f9b0 > [97883.849945] Call Trace: > [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f > [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 > [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 > [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 > [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 > [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 > [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 > [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 > [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 > [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 > [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 > [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 > [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 > [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 > [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 > [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 > [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 > [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 > [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 > [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 > [97883.882611] Mem-Info: > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > active_file:3902 inactive_file:3639 isolated_file:0 > unevictable:0 dirty:205 writeback:0 unstable:0 > slab_reclaimable:9856 slab_unreclaimable:9682 > mapped:3722 shmem:59 pagetables:2080 bounce:0 > free:748 free_pcp:15 free_cma:0 > [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB > active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB > unstable:0kB pages_scanned:168352 all_unreclaimable? yes > [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB > active_anon:1716kB inactive_anon:912kB active_file:2292kB > inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB > managed:15908kB mlocked:0kB slab_reclaimable:4652kB > slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB bounce:0kB > free_pcp:0kB local_pcp:0kB free_cma:0kB > [97883.898799] lowmem_reserve[]: 0 117 117 117 117 > [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB high:1944kB > active_anon:9944kB inactive_anon:12572kB active_file:13316kB > inactive_file:13680kB unevictable:0kB writepending:768kB present:180080kB > managed:139012kB mlocked:0kB slab_reclaimable:34772kB > slab_unreclaimable:35876kB kernel_stack:1828kB pagetables:7824kB bounce:0kB > free_pcp:60kB local_pcp:52kB free_cma:0kB > [97883.903033] lowmem_reserve[]: 0 0 0 0 0 > [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB (H) > 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1464kB > [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB (H) > 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 1508kB (H) mean highorder atomic reserved which was introduced since v4.4 and some patches to use up that reserved memory went to linux-next recently via mmotm tree. It doesn't land to 4.9 so it might help to test recent linux-next tree. It should include [1]. [1] 04c8716f7b00, mm: try to exhaust highatomic reserve before the OOM Thanks. ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-23 2:55 ` Minchan Kim 0 siblings, 0 replies; 88+ messages in thread From: Minchan Kim @ 2016-12-23 2:55 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Michal Hocko, linux-kernel, linux-mm, Linus Torvalds On Fri, Dec 09, 2016 at 04:52:07PM +0100, Gerhard Wiesinger wrote: > On 09.12.2016 14:40, Michal Hocko wrote: > >On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: > >>Hello, > >> > >>same with latest kernel rc, dnf still killed with OOM (but sometimes > >>better). > >> > >>./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} > >>(does dnf clean all;dnf update) > >>Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 > >>17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > >> > >>Updated bug report: > >>https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > >Could you post your oom report please? > > E.g. a new one with more than one included, first one after boot ... > > Just setup a low mem VM under KVM and it is easily triggerable. > > Still enough virtual memory available ... > > 4.9.0-0.rc8.git2.1.fc26.x86_64 > > [ 624.862777] ksoftirqd/0: page allocation failure: order:0, > mode:0x2080020(GFP_ATOMIC) > [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > 1.9.3 > [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 > 0000000000000000 > [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 > ffffffff90c7dd98 > [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 > ffffaa62c007f9a0 > [ 624.864998] Call Trace: > [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 > [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 > [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 > [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 > [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 > [virtio_net] > [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 > [virtio_net] > [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] > [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 > [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 > [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 > [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 > [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 > [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 > [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 > [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 > [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 > [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 > [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 > [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left > [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left > [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left > [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 bytes > left > [95895.765570] kworker/1:1H: page allocation failure: order:0, > mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) > [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > 1.9.3 > [95895.766060] Workqueue: kblockd blk_mq_run_work_fn > [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 > 0000000000000000 > [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 > ffffffff90c7dd98 > [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 > ffffaa62c0257670 > [95895.766417] Call Trace: > [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 > [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 > [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 > [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 > [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 > [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 > [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 > [95895.773406] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 > [95895.775212] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 > [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 > [95895.781502] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 > [virtio_ring] > [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 > [virtio_ring] > [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 > [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 > [virtio_blk] > [95895.785419] [<ffffffff901312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 > [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 > [virtio_blk] > [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 > [virtio_blk] > [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 > [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 > [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 > [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 > [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 > [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 > [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 > [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 > [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 > [95895.794212] [<ffffffff90111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 > [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 > [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 > [95895.796408] Mem-Info: > [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 > active_file:263 inactive_file:238 isolated_file:0 > unevictable:0 dirty:0 writeback:330 unstable:0 > slab_reclaimable:5241 slab_unreclaimable:9538 > mapped:470 shmem:9 pagetables:2200 bounce:0 > free:690 free_pcp:68 free_cma:0 > [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB > active_file:1052kB inactive_file:952kB unevictable:0kB isolated(anon):128kB > isolated(file):0kB mapped:1880kB dirty:0kB writeback:1320kB shmem:0kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB > unstable:0kB pages_scanned:179 all_unreclaimable? no > [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB > active_anon:3544kB inactive_anon:3944kB active_file:84kB inactive_file:140kB > unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB > slab_reclaimable:1728kB slab_unreclaimable:2964kB kernel_stack:84kB > pagetables:1396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [95895.805936] lowmem_reserve[]: 0 117 117 117 117 > [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB high:1944kB > active_anon:31636kB inactive_anon:32164kB active_file:968kB > inactive_file:804kB unevictable:0kB writepending:1288kB present:180080kB > managed:139012kB mlocked:0kB slab_reclaimable:19236kB > slab_unreclaimable:35188kB kernel_stack:1852kB pagetables:7404kB bounce:0kB > free_pcp:272kB local_pcp:156kB free_cma:0kB > [95895.809223] lowmem_reserve[]: 0 0 0 0 0 > [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB (H) > 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB > [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB (H) > 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1836kB > [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 > hugepages_size=2048kB > [95895.813981] 1804 total pagecache pages > [95895.814931] 1289 pages in swap cache > [95895.815849] Swap cache stats: add 5288014, delete 5286725, find > 11568655/13881082 > [95895.816792] Free swap = 1791816kB > [95895.817706] Total swap = 2064380kB > [95895.819222] 49018 pages RAM > [95895.820145] 0 pages HighMem/MovableOnly > [95895.821039] 10288 pages reserved > [95895.823325] 0 pages cma reserved > [95895.824244] 0 pages hwpoisoned > [95895.825237] SLUB: Unable to allocate memory on node -1, > gfp=0x2080020(GFP_ATOMIC) > [95895.826140] cache: kmalloc-256, object size: 256, buffer size: 256, > default order: 0, min order: 0 > [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 > [97883.838418] httpd invoked oom-killer: > gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, > oom_score_adj=0 > [97883.843507] httpd cpuset=/ mems_allowed=0 > [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted > 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 > [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > 1.9.3 > [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 > ffff96b98b8b3100 > [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 > 0000000000000000 > [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d > ffffaa62c395f9b0 > [97883.849945] Call Trace: > [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 > [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f > [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 > [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 > [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 > [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 > [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 > [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 > [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 > [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 > [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 > [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 > [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 > [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 > [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 > [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 > [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 > [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 > [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 > [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 > [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 > [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 > [97883.882611] Mem-Info: > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > active_file:3902 inactive_file:3639 isolated_file:0 > unevictable:0 dirty:205 writeback:0 unstable:0 > slab_reclaimable:9856 slab_unreclaimable:9682 > mapped:3722 shmem:59 pagetables:2080 bounce:0 > free:748 free_pcp:15 free_cma:0 > [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB > active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB > unstable:0kB pages_scanned:168352 all_unreclaimable? yes > [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB > active_anon:1716kB inactive_anon:912kB active_file:2292kB > inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB > managed:15908kB mlocked:0kB slab_reclaimable:4652kB > slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB bounce:0kB > free_pcp:0kB local_pcp:0kB free_cma:0kB > [97883.898799] lowmem_reserve[]: 0 117 117 117 117 > [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB high:1944kB > active_anon:9944kB inactive_anon:12572kB active_file:13316kB > inactive_file:13680kB unevictable:0kB writepending:768kB present:180080kB > managed:139012kB mlocked:0kB slab_reclaimable:34772kB > slab_unreclaimable:35876kB kernel_stack:1828kB pagetables:7824kB bounce:0kB > free_pcp:60kB local_pcp:52kB free_cma:0kB > [97883.903033] lowmem_reserve[]: 0 0 0 0 0 > [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB (H) > 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1464kB > [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB (H) > 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 1508kB (H) mean highorder atomic reserved which was introduced since v4.4 and some patches to use up that reserved memory went to linux-next recently via mmotm tree. It doesn't land to 4.9 so it might help to test recent linux-next tree. It should include [1]. [1] 04c8716f7b00, mm: try to exhaust highatomic reserve before the OOM Thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-23 2:55 ` Minchan Kim @ 2017-01-01 17:20 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-01-01 17:20 UTC (permalink / raw) To: Minchan Kim; +Cc: Michal Hocko, linux-kernel, linux-mm, Linus Torvalds On 23.12.2016 03:55, Minchan Kim wrote: > On Fri, Dec 09, 2016 at 04:52:07PM +0100, Gerhard Wiesinger wrote: >> On 09.12.2016 14:40, Michal Hocko wrote: >>> On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >>>> Hello, >>>> >>>> same with latest kernel rc, dnf still killed with OOM (but sometimes >>>> better). >>>> >>>> ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} >>>> (does dnf clean all;dnf update) >>>> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >>>> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >>>> >>>> Updated bug report: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 >>> Could you post your oom report please? >> E.g. a new one with more than one included, first one after boot ... >> >> Just setup a low mem VM under KVM and it is easily triggerable. >> >> Still enough virtual memory available ... >> >> 4.9.0-0.rc8.git2.1.fc26.x86_64 >> >> [ 624.862777] ksoftirqd/0: page allocation failure: order:0, >> mode:0x2080020(GFP_ATOMIC) >> [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 >> 0000000000000000 >> [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 >> ffffffff90c7dd98 >> [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 >> ffffaa62c007f9a0 >> [ 624.864998] Call Trace: >> [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 >> [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 >> [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 >> [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 >> [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 >> [virtio_net] >> [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 >> [virtio_net] >> [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] >> [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 >> [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 >> [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 >> [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 >> [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 >> [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 >> [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 >> [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 >> [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 >> [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 >> [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 >> [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left >> [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left >> [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left >> [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 bytes >> left >> [95895.765570] kworker/1:1H: page allocation failure: order:0, >> mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) >> [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [95895.766060] Workqueue: kblockd blk_mq_run_work_fn >> [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 >> 0000000000000000 >> [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 >> ffffffff90c7dd98 >> [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 >> ffffaa62c0257670 >> [95895.766417] Call Trace: >> [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 >> [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 >> [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 >> [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 >> [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 >> [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 >> [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 >> [95895.773406] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 >> [95895.775212] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 >> [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 >> [95895.781502] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 >> [virtio_ring] >> [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 >> [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 >> [virtio_blk] >> [95895.785419] [<ffffffff901312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 >> [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 >> [virtio_blk] >> [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 >> [virtio_blk] >> [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 >> [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 >> [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 >> [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 >> [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 >> [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 >> [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 >> [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 >> [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 >> [95895.794212] [<ffffffff90111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 >> [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 >> [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 >> [95895.796408] Mem-Info: >> [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 >> active_file:263 inactive_file:238 isolated_file:0 >> unevictable:0 dirty:0 writeback:330 unstable:0 >> slab_reclaimable:5241 slab_unreclaimable:9538 >> mapped:470 shmem:9 pagetables:2200 bounce:0 >> free:690 free_pcp:68 free_cma:0 >> [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB >> active_file:1052kB inactive_file:952kB unevictable:0kB isolated(anon):128kB >> isolated(file):0kB mapped:1880kB dirty:0kB writeback:1320kB shmem:0kB >> shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB >> unstable:0kB pages_scanned:179 all_unreclaimable? no >> [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB >> active_anon:3544kB inactive_anon:3944kB active_file:84kB inactive_file:140kB >> unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB >> slab_reclaimable:1728kB slab_unreclaimable:2964kB kernel_stack:84kB >> pagetables:1396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB >> [95895.805936] lowmem_reserve[]: 0 117 117 117 117 >> [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB high:1944kB >> active_anon:31636kB inactive_anon:32164kB active_file:968kB >> inactive_file:804kB unevictable:0kB writepending:1288kB present:180080kB >> managed:139012kB mlocked:0kB slab_reclaimable:19236kB >> slab_unreclaimable:35188kB kernel_stack:1852kB pagetables:7404kB bounce:0kB >> free_pcp:272kB local_pcp:156kB free_cma:0kB >> [95895.809223] lowmem_reserve[]: 0 0 0 0 0 >> [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB (H) >> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB >> [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB (H) >> 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1836kB >> [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 >> hugepages_size=2048kB >> [95895.813981] 1804 total pagecache pages >> [95895.814931] 1289 pages in swap cache >> [95895.815849] Swap cache stats: add 5288014, delete 5286725, find >> 11568655/13881082 >> [95895.816792] Free swap = 1791816kB >> [95895.817706] Total swap = 2064380kB >> [95895.819222] 49018 pages RAM >> [95895.820145] 0 pages HighMem/MovableOnly >> [95895.821039] 10288 pages reserved >> [95895.823325] 0 pages cma reserved >> [95895.824244] 0 pages hwpoisoned >> [95895.825237] SLUB: Unable to allocate memory on node -1, >> gfp=0x2080020(GFP_ATOMIC) >> [95895.826140] cache: kmalloc-256, object size: 256, buffer size: 256, >> default order: 0, min order: 0 >> [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 >> [97883.838418] httpd invoked oom-killer: >> gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, >> oom_score_adj=0 >> [97883.843507] httpd cpuset=/ mems_allowed=0 >> [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 >> ffff96b98b8b3100 >> [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 >> 0000000000000000 >> [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d >> ffffaa62c395f9b0 >> [97883.849945] Call Trace: >> [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f >> [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 >> [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 >> [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 >> [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 >> [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 >> [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 >> [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 >> [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 >> [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 >> [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 >> [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 >> [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 >> [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 >> [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 >> [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 >> [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 >> [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 >> [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 >> [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 >> [97883.882611] Mem-Info: >> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >> active_file:3902 inactive_file:3639 isolated_file:0 >> unevictable:0 dirty:205 writeback:0 unstable:0 >> slab_reclaimable:9856 slab_unreclaimable:9682 >> mapped:3722 shmem:59 pagetables:2080 bounce:0 >> free:748 free_pcp:15 free_cma:0 >> [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB >> active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB >> isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB >> shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB >> unstable:0kB pages_scanned:168352 all_unreclaimable? yes >> [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB >> active_anon:1716kB inactive_anon:912kB active_file:2292kB >> inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB >> managed:15908kB mlocked:0kB slab_reclaimable:4652kB >> slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB bounce:0kB >> free_pcp:0kB local_pcp:0kB free_cma:0kB >> [97883.898799] lowmem_reserve[]: 0 117 117 117 117 >> [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB high:1944kB >> active_anon:9944kB inactive_anon:12572kB active_file:13316kB >> inactive_file:13680kB unevictable:0kB writepending:768kB present:180080kB >> managed:139012kB mlocked:0kB slab_reclaimable:34772kB >> slab_unreclaimable:35876kB kernel_stack:1828kB pagetables:7824kB bounce:0kB >> free_pcp:60kB local_pcp:52kB free_cma:0kB >> [97883.903033] lowmem_reserve[]: 0 0 0 0 0 >> [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB (H) >> 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1464kB >> [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB (H) >> 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = >> 1508kB > (H) mean highorder atomic reserved which was introduced since v4.4 and some > patches to use up that reserved memory went to linux-next recently via mmotm > tree. > It doesn't land to 4.9 so it might help to test recent linux-next tree. > It should include [1]. > > [1] 04c8716f7b00, mm: try to exhaust highatomic reserve before the OOM > Hello Minchan, 4.10.0-0.rc0.git9.1.fc26.x86_64 No still OOMs. I'll send you the infos per private mail. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2017-01-01 17:20 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-01-01 17:20 UTC (permalink / raw) To: Minchan Kim; +Cc: Michal Hocko, linux-kernel, linux-mm, Linus Torvalds On 23.12.2016 03:55, Minchan Kim wrote: > On Fri, Dec 09, 2016 at 04:52:07PM +0100, Gerhard Wiesinger wrote: >> On 09.12.2016 14:40, Michal Hocko wrote: >>> On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >>>> Hello, >>>> >>>> same with latest kernel rc, dnf still killed with OOM (but sometimes >>>> better). >>>> >>>> ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} >>>> (does dnf clean all;dnf update) >>>> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >>>> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >>>> >>>> Updated bug report: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 >>> Could you post your oom report please? >> E.g. a new one with more than one included, first one after boot ... >> >> Just setup a low mem VM under KVM and it is easily triggerable. >> >> Still enough virtual memory available ... >> >> 4.9.0-0.rc8.git2.1.fc26.x86_64 >> >> [ 624.862777] ksoftirqd/0: page allocation failure: order:0, >> mode:0x2080020(GFP_ATOMIC) >> [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 >> 0000000000000000 >> [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 >> ffffffff90c7dd98 >> [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 >> ffffaa62c007f9a0 >> [ 624.864998] Call Trace: >> [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 >> [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 >> [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 >> [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 >> [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 >> [virtio_net] >> [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 >> [virtio_net] >> [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] >> [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 >> [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 >> [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 >> [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 >> [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 >> [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 >> [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 >> [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 >> [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 >> [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 >> [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 >> [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left >> [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left >> [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left >> [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 bytes >> left >> [95895.765570] kworker/1:1H: page allocation failure: order:0, >> mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) >> [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [95895.766060] Workqueue: kblockd blk_mq_run_work_fn >> [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 >> 0000000000000000 >> [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 >> ffffffff90c7dd98 >> [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 >> ffffaa62c0257670 >> [95895.766417] Call Trace: >> [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 >> [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 >> [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 >> [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 >> [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 >> [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 >> [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 >> [95895.773406] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 >> [95895.775212] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 >> [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 >> [95895.781502] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 >> [virtio_ring] >> [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 >> [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 >> [virtio_blk] >> [95895.785419] [<ffffffff901312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 >> [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 >> [virtio_blk] >> [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 >> [virtio_blk] >> [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 >> [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 >> [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 >> [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 >> [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 >> [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 >> [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 >> [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 >> [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 >> [95895.794212] [<ffffffff90111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 >> [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 >> [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 >> [95895.796408] Mem-Info: >> [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 >> active_file:263 inactive_file:238 isolated_file:0 >> unevictable:0 dirty:0 writeback:330 unstable:0 >> slab_reclaimable:5241 slab_unreclaimable:9538 >> mapped:470 shmem:9 pagetables:2200 bounce:0 >> free:690 free_pcp:68 free_cma:0 >> [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB >> active_file:1052kB inactive_file:952kB unevictable:0kB isolated(anon):128kB >> isolated(file):0kB mapped:1880kB dirty:0kB writeback:1320kB shmem:0kB >> shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB >> unstable:0kB pages_scanned:179 all_unreclaimable? no >> [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB >> active_anon:3544kB inactive_anon:3944kB active_file:84kB inactive_file:140kB >> unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB >> slab_reclaimable:1728kB slab_unreclaimable:2964kB kernel_stack:84kB >> pagetables:1396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB >> [95895.805936] lowmem_reserve[]: 0 117 117 117 117 >> [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB high:1944kB >> active_anon:31636kB inactive_anon:32164kB active_file:968kB >> inactive_file:804kB unevictable:0kB writepending:1288kB present:180080kB >> managed:139012kB mlocked:0kB slab_reclaimable:19236kB >> slab_unreclaimable:35188kB kernel_stack:1852kB pagetables:7404kB bounce:0kB >> free_pcp:272kB local_pcp:156kB free_cma:0kB >> [95895.809223] lowmem_reserve[]: 0 0 0 0 0 >> [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB (H) >> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB >> [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB (H) >> 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1836kB >> [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 >> hugepages_size=2048kB >> [95895.813981] 1804 total pagecache pages >> [95895.814931] 1289 pages in swap cache >> [95895.815849] Swap cache stats: add 5288014, delete 5286725, find >> 11568655/13881082 >> [95895.816792] Free swap = 1791816kB >> [95895.817706] Total swap = 2064380kB >> [95895.819222] 49018 pages RAM >> [95895.820145] 0 pages HighMem/MovableOnly >> [95895.821039] 10288 pages reserved >> [95895.823325] 0 pages cma reserved >> [95895.824244] 0 pages hwpoisoned >> [95895.825237] SLUB: Unable to allocate memory on node -1, >> gfp=0x2080020(GFP_ATOMIC) >> [95895.826140] cache: kmalloc-256, object size: 256, buffer size: 256, >> default order: 0, min order: 0 >> [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 >> [97883.838418] httpd invoked oom-killer: >> gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, >> oom_score_adj=0 >> [97883.843507] httpd cpuset=/ mems_allowed=0 >> [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 >> ffff96b98b8b3100 >> [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 >> 0000000000000000 >> [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d >> ffffaa62c395f9b0 >> [97883.849945] Call Trace: >> [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f >> [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 >> [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 >> [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 >> [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 >> [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 >> [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 >> [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 >> [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 >> [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 >> [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 >> [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 >> [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 >> [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 >> [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 >> [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 >> [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 >> [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 >> [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 >> [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 >> [97883.882611] Mem-Info: >> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >> active_file:3902 inactive_file:3639 isolated_file:0 >> unevictable:0 dirty:205 writeback:0 unstable:0 >> slab_reclaimable:9856 slab_unreclaimable:9682 >> mapped:3722 shmem:59 pagetables:2080 bounce:0 >> free:748 free_pcp:15 free_cma:0 >> [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB >> active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB >> isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB >> shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB >> unstable:0kB pages_scanned:168352 all_unreclaimable? yes >> [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB >> active_anon:1716kB inactive_anon:912kB active_file:2292kB >> inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB >> managed:15908kB mlocked:0kB slab_reclaimable:4652kB >> slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB bounce:0kB >> free_pcp:0kB local_pcp:0kB free_cma:0kB >> [97883.898799] lowmem_reserve[]: 0 117 117 117 117 >> [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB high:1944kB >> active_anon:9944kB inactive_anon:12572kB active_file:13316kB >> inactive_file:13680kB unevictable:0kB writepending:768kB present:180080kB >> managed:139012kB mlocked:0kB slab_reclaimable:34772kB >> slab_unreclaimable:35876kB kernel_stack:1828kB pagetables:7824kB bounce:0kB >> free_pcp:60kB local_pcp:52kB free_cma:0kB >> [97883.903033] lowmem_reserve[]: 0 0 0 0 0 >> [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB (H) >> 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1464kB >> [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB (H) >> 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = >> 1508kB > (H) mean highorder atomic reserved which was introduced since v4.4 and some > patches to use up that reserved memory went to linux-next recently via mmotm > tree. > It doesn't land to 4.9 so it might help to test recent linux-next tree. > It should include [1]. > > [1] 04c8716f7b00, mm: try to exhaust highatomic reserve before the OOM > Hello Minchan, 4.10.0-0.rc0.git9.1.fc26.x86_64 No still OOMs. I'll send you the infos per private mail. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-23 2:55 ` Minchan Kim (?) (?) @ 2017-01-04 8:40 ` Gerhard Wiesinger 2017-01-04 9:11 ` Michal Hocko -1 siblings, 1 reply; 88+ messages in thread From: Gerhard Wiesinger @ 2017-01-04 8:40 UTC (permalink / raw) To: Minchan Kim; +Cc: Michal Hocko, linux-kernel, linux-mm, Linus Torvalds On 23.12.2016 03:55, Minchan Kim wrote: > On Fri, Dec 09, 2016 at 04:52:07PM +0100, Gerhard Wiesinger wrote: >> On 09.12.2016 14:40, Michal Hocko wrote: >>> On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >>>> Hello, >>>> >>>> same with latest kernel rc, dnf still killed with OOM (but sometimes >>>> better). >>>> >>>> ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} >>>> (does dnf clean all;dnf update) >>>> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >>>> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >>>> >>>> Updated bug report: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 >>> Could you post your oom report please? >> E.g. a new one with more than one included, first one after boot ... >> >> Just setup a low mem VM under KVM and it is easily triggerable. >> >> Still enough virtual memory available ... >> >> 4.9.0-0.rc8.git2.1.fc26.x86_64 >> >> [ 624.862777] ksoftirqd/0: page allocation failure: order:0, >> mode:0x2080020(GFP_ATOMIC) >> [ 624.863319] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [ 624.863410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [ 624.863510] ffffaa62c007f958 ffffffff904774e3 ffffffff90c7dd98 >> 0000000000000000 >> [ 624.863923] ffffaa62c007f9e0 ffffffff9020e6ea 0208002000000246 >> ffffffff90c7dd98 >> [ 624.864019] ffffaa62c007f980 ffff96b900000010 ffffaa62c007f9f0 >> ffffaa62c007f9a0 >> [ 624.864998] Call Trace: >> [ 624.865149] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [ 624.865347] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 >> [ 624.865432] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 >> [ 624.865563] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [ 624.865675] [<ffffffff9020f983>] __alloc_page_frag+0x193/0x200 >> [ 624.866024] [<ffffffff907a1d7e>] __napi_alloc_skb+0x8e/0xf0 >> [ 624.866113] [<ffffffffc017777d>] page_to_skb.isra.28+0x5d/0x310 >> [virtio_net] >> [ 624.866201] [<ffffffffc01794cb>] virtnet_receive+0x2db/0x9a0 >> [virtio_net] >> [ 624.867378] [<ffffffffc0179bad>] virtnet_poll+0x1d/0x80 [virtio_net] >> [ 624.867494] [<ffffffff907b501e>] net_rx_action+0x23e/0x470 >> [ 624.867612] [<ffffffff9091a8cd>] __do_softirq+0xcd/0x4b9 >> [ 624.867704] [<ffffffff900dd164>] ? smpboot_thread_fn+0x34/0x1f0 >> [ 624.867833] [<ffffffff900dd25d>] ? smpboot_thread_fn+0x12d/0x1f0 >> [ 624.867924] [<ffffffff900b7c95>] run_ksoftirqd+0x25/0x80 >> [ 624.868109] [<ffffffff900dd258>] smpboot_thread_fn+0x128/0x1f0 >> [ 624.868197] [<ffffffff900dd130>] ? sort_range+0x30/0x30 >> [ 624.868596] [<ffffffff900d82c2>] kthread+0x102/0x120 >> [ 624.868679] [<ffffffff909117a0>] ? wait_for_completion+0x110/0x140 >> [ 624.868768] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 >> [ 624.868850] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 >> [ 843.528656] httpd (2490) used greatest stack depth: 10304 bytes left >> [ 878.077750] httpd (2976) used greatest stack depth: 10096 bytes left >> [93918.861109] netstat (14579) used greatest stack depth: 9488 bytes left >> [94050.874669] kworker/dying (6253) used greatest stack depth: 9008 bytes >> left >> [95895.765570] kworker/1:1H: page allocation failure: order:0, >> mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) >> [95895.765819] CPU: 1 PID: 440 Comm: kworker/1:1H Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [95895.765911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [95895.766060] Workqueue: kblockd blk_mq_run_work_fn >> [95895.766143] ffffaa62c0257628 ffffffff904774e3 ffffffff90c7dd98 >> 0000000000000000 >> [95895.766235] ffffaa62c02576b0 ffffffff9020e6ea 0228002000000046 >> ffffffff90c7dd98 >> [95895.766325] ffffaa62c0257650 ffff96b900000010 ffffaa62c02576c0 >> ffffaa62c0257670 >> [95895.766417] Call Trace: >> [95895.766502] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [95895.766596] [<ffffffff9020e6ea>] warn_alloc+0x13a/0x170 >> [95895.766681] [<ffffffff9020e9e2>] __alloc_pages_slowpath+0x252/0xbb0 >> [95895.766767] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [95895.766866] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 >> [95895.766971] [<ffffffff90916f17>] ? _raw_spin_unlock+0x27/0x40 >> [95895.767073] [<ffffffff90278956>] new_slab+0x316/0x7c0 >> [95895.767160] [<ffffffff9027ae8b>] ___slab_alloc+0x3fb/0x5c0 >> [95895.772611] [<ffffffff9010b042>] ? cpuacct_charge+0xf2/0x1f0 >> [95895.773406] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.774327] [<ffffffff901319d5>] ? rcu_read_lock_sched_held+0x45/0x80 >> [95895.775212] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.776155] [<ffffffff9027b0a1>] __slab_alloc+0x51/0x90 >> [95895.777090] [<ffffffff9027d141>] __kmalloc+0x251/0x320 >> [95895.781502] [<ffffffffc005850d>] ? alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.782309] [<ffffffffc005850d>] alloc_indirect.isra.11+0x1d/0x50 >> [virtio_ring] >> [95895.783334] [<ffffffffc0059193>] virtqueue_add_sgs+0x1c3/0x4a0 >> [virtio_ring] >> [95895.784059] [<ffffffff90068475>] ? kvm_sched_clock_read+0x25/0x40 >> [95895.784742] [<ffffffffc006665c>] __virtblk_add_req+0xbc/0x220 >> [virtio_blk] >> [95895.785419] [<ffffffff901312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 >> [95895.786086] [<ffffffffc0066935>] ? virtio_queue_rq+0x105/0x290 >> [virtio_blk] >> [95895.786750] [<ffffffffc006695d>] virtio_queue_rq+0x12d/0x290 >> [virtio_blk] >> [95895.787427] [<ffffffff9045015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 >> [95895.788106] [<ffffffff904502e2>] blk_mq_run_work_fn+0x12/0x20 >> [95895.789065] [<ffffffff900d097e>] process_one_work+0x23e/0x6f0 >> [95895.789741] [<ffffffff900d08fa>] ? process_one_work+0x1ba/0x6f0 >> [95895.790444] [<ffffffff900d0e7e>] worker_thread+0x4e/0x490 >> [95895.791178] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 >> [95895.791911] [<ffffffff900d0e30>] ? process_one_work+0x6f0/0x6f0 >> [95895.792653] [<ffffffff90003eec>] ? do_syscall_64+0x6c/0x1f0 >> [95895.793397] [<ffffffff900d82c2>] kthread+0x102/0x120 >> [95895.794212] [<ffffffff90111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 >> [95895.794942] [<ffffffff900d81c0>] ? kthread_park+0x60/0x60 >> [95895.795689] [<ffffffff90917afa>] ret_from_fork+0x2a/0x40 >> [95895.796408] Mem-Info: >> [95895.797110] active_anon:8800 inactive_anon:9030 isolated_anon:32 >> active_file:263 inactive_file:238 isolated_file:0 >> unevictable:0 dirty:0 writeback:330 unstable:0 >> slab_reclaimable:5241 slab_unreclaimable:9538 >> mapped:470 shmem:9 pagetables:2200 bounce:0 >> free:690 free_pcp:68 free_cma:0 >> [95895.801218] Node 0 active_anon:35200kB inactive_anon:36120kB >> active_file:1052kB inactive_file:952kB unevictable:0kB isolated(anon):128kB >> isolated(file):0kB mapped:1880kB dirty:0kB writeback:1320kB shmem:0kB >> shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB >> unstable:0kB pages_scanned:179 all_unreclaimable? no >> [95895.803264] Node 0 DMA free:924kB min:172kB low:212kB high:252kB >> active_anon:3544kB inactive_anon:3944kB active_file:84kB inactive_file:140kB >> unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB >> slab_reclaimable:1728kB slab_unreclaimable:2964kB kernel_stack:84kB >> pagetables:1396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB >> [95895.805936] lowmem_reserve[]: 0 117 117 117 117 >> [95895.806751] Node 0 DMA32 free:1836kB min:1296kB low:1620kB high:1944kB >> active_anon:31636kB inactive_anon:32164kB active_file:968kB >> inactive_file:804kB unevictable:0kB writepending:1288kB present:180080kB >> managed:139012kB mlocked:0kB slab_reclaimable:19236kB >> slab_unreclaimable:35188kB kernel_stack:1852kB pagetables:7404kB bounce:0kB >> free_pcp:272kB local_pcp:156kB free_cma:0kB >> [95895.809223] lowmem_reserve[]: 0 0 0 0 0 >> [95895.810071] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 6*32kB (H) >> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920kB >> [95895.812089] Node 0 DMA32: 77*4kB (H) 71*8kB (H) 28*16kB (H) 8*32kB (H) >> 4*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1836kB >> [95895.813979] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 >> hugepages_size=2048kB >> [95895.813981] 1804 total pagecache pages >> [95895.814931] 1289 pages in swap cache >> [95895.815849] Swap cache stats: add 5288014, delete 5286725, find >> 11568655/13881082 >> [95895.816792] Free swap = 1791816kB >> [95895.817706] Total swap = 2064380kB >> [95895.819222] 49018 pages RAM >> [95895.820145] 0 pages HighMem/MovableOnly >> [95895.821039] 10288 pages reserved >> [95895.823325] 0 pages cma reserved >> [95895.824244] 0 pages hwpoisoned >> [95895.825237] SLUB: Unable to allocate memory on node -1, >> gfp=0x2080020(GFP_ATOMIC) >> [95895.826140] cache: kmalloc-256, object size: 256, buffer size: 256, >> default order: 0, min order: 0 >> [95895.827034] node 0: slabs: 113, objs: 1808, free: 0 >> [97883.838418] httpd invoked oom-killer: >> gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, >> oom_score_adj=0 >> [97883.843507] httpd cpuset=/ mems_allowed=0 >> [97883.843601] CPU: 1 PID: 19043 Comm: httpd Not tainted >> 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 >> [97883.844628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> 1.9.3 >> [97883.845839] ffffaa62c395f958 ffffffff904774e3 ffffaa62c395fb20 >> ffff96b98b8b3100 >> [97883.846970] ffffaa62c395f9e0 ffffffff902a8c41 0000000000000000 >> 0000000000000000 >> [97883.848388] ffffffff90ec6840 ffffaa62c395f990 ffffffff9011183d >> ffffaa62c395f9b0 >> [97883.849945] Call Trace: >> [97883.851366] [<ffffffff904774e3>] dump_stack+0x86/0xc3 >> [97883.852535] [<ffffffff902a8c41>] dump_header+0x7b/0x24f >> [97883.853718] [<ffffffff9011183d>] ? trace_hardirqs_on+0xd/0x10 >> [97883.854857] [<ffffffff902085d3>] oom_kill_process+0x203/0x3e0 >> [97883.856192] [<ffffffff90208afb>] out_of_memory+0x13b/0x580 >> [97883.857334] [<ffffffff90208bea>] ? out_of_memory+0x22a/0x580 >> [97883.858590] [<ffffffff9020f31a>] __alloc_pages_slowpath+0xb8a/0xbb0 >> [97883.859706] [<ffffffff9020f74d>] __alloc_pages_nodemask+0x40d/0x4b0 >> [97883.860854] [<ffffffff90037de9>] ? sched_clock+0x9/0x10 >> [97883.862120] [<ffffffff9026db51>] alloc_pages_current+0xa1/0x1f0 >> [97883.863251] [<ffffffff90201d96>] __page_cache_alloc+0x146/0x190 >> [97883.864449] [<ffffffff9020366c>] ? pagecache_get_page+0x2c/0x300 >> [97883.865602] [<ffffffff90206055>] filemap_fault+0x345/0x790 >> [97883.866661] [<ffffffff90206238>] ? filemap_fault+0x528/0x790 >> [97883.867795] [<ffffffff903639d9>] ext4_filemap_fault+0x39/0x50 >> [97883.869289] [<ffffffff90241ca3>] __do_fault+0x83/0x1d0 >> [97883.870301] [<ffffffff90246642>] handle_mm_fault+0x11e2/0x17a0 >> [97883.871304] [<ffffffff902454ba>] ? handle_mm_fault+0x5a/0x17a0 >> [97883.872491] [<ffffffff9006de16>] __do_page_fault+0x266/0x520 >> [97883.873406] [<ffffffff9006e1a8>] trace_do_page_fault+0x58/0x2a0 >> [97883.874262] [<ffffffff90067f3a>] do_async_page_fault+0x1a/0xa0 >> [97883.875168] [<ffffffff90918e28>] async_page_fault+0x28/0x30 >> [97883.882611] Mem-Info: >> [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 >> active_file:3902 inactive_file:3639 isolated_file:0 >> unevictable:0 dirty:205 writeback:0 unstable:0 >> slab_reclaimable:9856 slab_unreclaimable:9682 >> mapped:3722 shmem:59 pagetables:2080 bounce:0 >> free:748 free_pcp:15 free_cma:0 >> [97883.890766] Node 0 active_anon:11660kB inactive_anon:13504kB >> active_file:15608kB inactive_file:14556kB unevictable:0kB isolated(anon):0kB >> isolated(file):0kB mapped:14888kB dirty:820kB writeback:0kB shmem:0kB >> shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 236kB writeback_tmp:0kB >> unstable:0kB pages_scanned:168352 all_unreclaimable? yes >> [97883.893210] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB >> active_anon:1716kB inactive_anon:912kB active_file:2292kB >> inactive_file:876kB unevictable:0kB writepending:24kB present:15992kB >> managed:15908kB mlocked:0kB slab_reclaimable:4652kB >> slab_unreclaimable:2852kB kernel_stack:76kB pagetables:496kB bounce:0kB >> free_pcp:0kB local_pcp:0kB free_cma:0kB >> [97883.898799] lowmem_reserve[]: 0 117 117 117 117 >> [97883.899735] Node 0 DMA32 free:1524kB min:1296kB low:1620kB high:1944kB >> active_anon:9944kB inactive_anon:12572kB active_file:13316kB >> inactive_file:13680kB unevictable:0kB writepending:768kB present:180080kB >> managed:139012kB mlocked:0kB slab_reclaimable:34772kB >> slab_unreclaimable:35876kB kernel_stack:1828kB pagetables:7824kB bounce:0kB >> free_pcp:60kB local_pcp:52kB free_cma:0kB >> [97883.903033] lowmem_reserve[]: 0 0 0 0 0 >> [97883.904371] Node 0 DMA: 36*4kB (H) 29*8kB (H) 22*16kB (H) 9*32kB (H) >> 3*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1464kB >> [97883.906442] Node 0 DMA32: 13*4kB (H) 4*8kB (H) 13*16kB (H) 8*32kB (H) >> 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = >> 1508kB > (H) mean highorder atomic reserved which was introduced since v4.4 and some > patches to use up that reserved memory went to linux-next recently via mmotm > tree. > It doesn't land to 4.9 so it might help to test recent linux-next tree. > It should include [1]. > > [1] 04c8716f7b00, mm: try to exhaust highatomic reserve before the OOM > Ok, 4.10.0-0.rc2.git0.1.fc26.x86_64 is not stable (4.9.0-1.fc26.x86_64 was). The VM stops working (e.g. not pingable) after around 8h (will be restarted automatically), happened serveral times. Had also further OOMs which I sent to Mincham. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2017-01-04 8:40 ` Gerhard Wiesinger @ 2017-01-04 9:11 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-01-04 9:11 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Wed 04-01-17 09:40:25, Gerhard Wiesinger wrote: [...] > Ok, 4.10.0-0.rc2.git0.1.fc26.x86_64 is not stable (4.9.0-1.fc26.x86_64 was). Does the same happen with the vanilla kernels? > The VM stops working (e.g. not pingable) after around 8h (will be restarted > automatically), happened serveral times. > > Had also further OOMs which I sent to Mincham. Could you post them to the mailing list as well, please? -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2017-01-04 9:11 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-01-04 9:11 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Wed 04-01-17 09:40:25, Gerhard Wiesinger wrote: [...] > Ok, 4.10.0-0.rc2.git0.1.fc26.x86_64 is not stable (4.9.0-1.fc26.x86_64 was). Does the same happen with the vanilla kernels? > The VM stops working (e.g. not pingable) after around 8h (will be restarted > automatically), happened serveral times. > > Had also further OOMs which I sent to Mincham. Could you post them to the mailing list as well, please? -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-01-04 9:11 ` Michal Hocko (?) @ 2017-02-26 8:40 ` Gerhard Wiesinger 2017-02-27 8:27 ` Michal Hocko 2017-02-27 9:02 ` Minchan Kim -1 siblings, 2 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-02-26 8:40 UTC (permalink / raw) To: Michal Hocko; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 04.01.2017 10:11, Michal Hocko wrote: >> The VM stops working (e.g. not pingable) after around 8h (will be restarted >> automatically), happened serveral times. >> >> Had also further OOMs which I sent to Mincham. > Could you post them to the mailing list as well, please? Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as well on 4.9.9-200.fc25.x86_64 On 4.10er kernels: Free swap = 1137532kB cat /etc/sysctl.d/* | grep ^vm vm.dirty_background_ratio = 3 vm.dirty_ratio = 15 vm.overcommit_memory = 2 vm.overcommit_ratio = 80 vm.swappiness=10 kernel: python invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0 kernel: python cpuset=/ mems_allowed=0 kernel: CPU: 1 PID: 813 Comm: python Not tainted 4.10.0-1.fc26.x86_64 #1 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 04/01/2014 kernel: Call Trace: kernel: dump_stack+0x63/0x84 kernel: dump_header+0x7b/0x1f6 kernel: ? do_try_to_free_pages+0x2c5/0x340 kernel: oom_kill_process+0x202/0x3d0 kernel: out_of_memory+0x2b7/0x4e0 kernel: __alloc_pages_slowpath+0x915/0xb80 kernel: __alloc_pages_nodemask+0x218/0x2d0 kernel: alloc_pages_current+0x93/0x150 kernel: __page_cache_alloc+0xcf/0x100 kernel: filemap_fault+0x39d/0x800 kernel: ? page_add_file_rmap+0xe5/0x200 kernel: ? filemap_map_pages+0x2e1/0x4e0 kernel: ext4_filemap_fault+0x36/0x50 kernel: __do_fault+0x21/0x110 kernel: handle_mm_fault+0xdd1/0x1410 kernel: ? swake_up+0x42/0x50 kernel: __do_page_fault+0x23f/0x4c0 kernel: trace_do_page_fault+0x41/0x120 kernel: do_async_page_fault+0x51/0xa0 kernel: async_page_fault+0x28/0x30 kernel: RIP: 0033:0x7f0681ad6350 kernel: RSP: 002b:00007ffcbdd238d8 EFLAGS: 00010246 kernel: RAX: 00007f0681b0f960 RBX: 0000000000000000 RCX: 7fffffffffffffff kernel: RDX: 0000000000000000 RSI: 3ff0000000000000 RDI: 3ff0000000000000 kernel: RBP: 00007f067461ab40 R08: 0000000000000000 R09: 3ff0000000000000 kernel: R10: 0000556f1c6d8a80 R11: 0000000000000001 R12: 00007f0676d1a8d0 kernel: R13: 0000000000000000 R14: 00007f06746168bc R15: 00007f0674385910 kernel: Mem-Info: kernel: active_anon:37423 inactive_anon:37512 isolated_anon:0 active_file:462 inactive_file:603 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:3538 slab_unreclaimable:4818 mapped:859 shmem:9 pagetables:3370 bounce:0 free:1650 free_pcp:103 free_cma:0 kernel: Node 0 active_anon:149380kB inactive_anon:149704kB active_file:1848kB inactive_file:3660kB unevictable:0kB isolated(anon):128kB isolated(file):0kB mapped:4580kB dirty:0kB writeback:380kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB unstable:0kB pages_scanned:352 all_unreclaimable? no kernel: Node 0 DMA free:1484kB min:104kB low:128kB high:152kB active_anon:5660kB inactive_anon:6156kB active_file:56kB inactive_file:64kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:444kB slab_unreclaimable:1208kB kernel_stack:32kB pagetables:592kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB kernel: lowmem_reserve[]: 0 327 327 327 327 kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB active_anon:143580kB inactive_anon:143300kB active_file:2576kB inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB managed:353968kB mlocked:0kB slab_reclaimable:13708kB slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB free_pcp:412kB local_pcp:88kB free_cma:0kB kernel: lowmem_reserve[]: 0 0 0 0 0 kernel: Node 0 DMA: 70*4kB (UMEH) 20*8kB (UMEH) 13*16kB (MH) 5*32kB (H) 4*64kB (H) 2*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1576kB kernel: Node 0 DMA32: 1134*4kB (UMEH) 25*8kB (UMEH) 13*16kB (MH) 7*32kB (H) 3*64kB (H) 0*128kB 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5616kB kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB kernel: 6561 total pagecache pages kernel: 5240 pages in swap cache kernel: Swap cache stats: add 100078658, delete 100073419, find 199458343/238460223 kernel: Free swap = 1137532kB kernel: Total swap = 2064380kB kernel: 98170 pages RAM kernel: 0 pages HighMem/MovableOnly kernel: 5701 pages reserved kernel: 0 pages cma reserved kernel: 0 pages hwpoisoned kernel: Out of memory: Kill process 11968 (clamscan) score 170 or sacrifice child kernel: Killed process 11968 (clamscan) total-vm:538120kB, anon-rss:182220kB, file-rss:464kB, shmem-rss:0kB On 4.9er kernels: Free swap = 1826688kB cat /etc/sysctl.d/* | grep ^vm vm.dirty_background_ratio=3 vm.dirty_ratio=15 vm.overcommit_memory=2 vm.overcommit_ratio=80 vm.swappiness=10 kernel: dnf invoked oom-killer: gfp_mask=0x24280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0, order=0, oom_score_adj=0 kernel: dnf cpuset=/ mems_allowed=0 kernel: CPU: 0 PID: 20049 Comm: dnf Not tainted 4.9.9-200.fc25.x86_64 #1 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 04/01/2014 kernel: ffffa764434d7ac0 ffffffff933f467d ffffa764434d7c90 ffff9569999057c0 kernel: ffffa764434d7b38 ffffffff932555c1 0000000000000000 0000000000000000 kernel: ffffffffc02b2afa 00000000ffffffff 0000000000000000 ffffa764434d7b28 kernel: Call Trace: kernel: [<ffffffff933f467d>] dump_stack+0x63/0x86 kernel: [<ffffffff932555c1>] dump_header+0x7b/0x1f6 kernel: [<ffffffffc02b2afa>] ? virtballoon_oom_notify+0x2a/0x80 [virtio_balloon] kernel: [<ffffffff931c711f>] oom_kill_process+0x1ff/0x3d0 kernel: [<ffffffff931c7623>] out_of_memory+0x143/0x4e0 kernel: [<ffffffff931cd3cd>] __alloc_pages_slowpath+0xb2d/0xc40 kernel: [<ffffffff931cd736>] __alloc_pages_nodemask+0x256/0x2c0 kernel: [<ffffffff9322567b>] alloc_pages_vma+0xab/0x290 kernel: [<ffffffff931fe2ed>] handle_mm_fault+0x13bd/0x1610 kernel: [<ffffffff9306283e>] __do_page_fault+0x23e/0x4e0 kernel: [<ffffffff93062ba1>] trace_do_page_fault+0x41/0x120 kernel: [<ffffffff9305caba>] do_async_page_fault+0x1a/0xa0 kernel: [<ffffffff9381f3b8>] async_page_fault+0x28/0x30 kernel: Mem-Info: kernel: active_anon:31057 inactive_anon:30347 isolated_anon:0 active_file:20333 inactive_file:25749 isolated_file:32 unevictable:0 dirty:1444 writeback:0 unstable:0 slab_reclaimable:4537 slab_unreclaimable:5448 mapped:15224 shmem:120 pagetables:2537 bounce:0 free:1133 free_pcp:196 free_cma:0 kernel: Node 0 active_anon:122692kB inactive_anon:122948kB active_file:81332kB inactive_file:102996kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:60896kB dirty:5776kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 480kB writeback_tmp:0kB unstable:0kB pages_scanned:962974 all_unreclaimable? yes kernel: Node 0 DMA free:1892kB min:92kB low:112kB high:132kB active_anon:544kB inactive_anon:10872kB active_file:16kB inactive_file:996kB unevictable:0kB writepending:1128kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:488kB slab_unreclaimable:388kB kernel_stack:0kB pagetables:24kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB kernel: lowmem_reserve[]: 0 451 451 451 451 kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB active_anon:122148kB inactive_anon:112068kB active_file:81324kB inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB managed:484384kB mlocked:0kB slab_reclaimable:17660kB slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB kernel: lowmem_reserve[]: 0 0 0 0 0 kernel: Node 0 DMA: 81*4kB (UEH) 26*8kB (UMEH) 15*16kB (UH) 7*32kB (H) 8*64kB (H) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1892kB kernel: Node 0 DMA32: 375*4kB (UMH) 18*8kB (MH) 7*16kB (MH) 6*32kB (MH) 6*64kB (H) 4*128kB (H) 2*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3356kB kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB kernel: 49304 total pagecache pages kernel: 3085 pages in swap cache kernel: Swap cache stats: add 129095, delete 126010, find 24611247/24627536 kernel: Free swap = 1826688kB kernel: Total swap = 2064380kB kernel: 130938 pages RAM kernel: 0 pages HighMem/MovableOnly kernel: 5865 pages reserved kernel: 0 pages cma reserved kernel: 0 pages hwpoisoned kernel: Out of memory: Kill process 995 (named) score 87 or sacrifice child kernel: Killed process 995 (named) total-vm:399260kB, anon-rss:86516kB, file-rss:1288kB, shmem-rss:0kB kernel: oom_reaper: reaped process 995 (named), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM with ext4 and Fedora 25 and some memory load and updating the VM. Any further progress? Thnx. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-02-26 8:40 ` Still OOM problems with 4.9er/4.10er kernels Gerhard Wiesinger @ 2017-02-27 8:27 ` Michal Hocko 2017-02-27 9:02 ` Minchan Kim 1 sibling, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-02-27 8:27 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote: > On 04.01.2017 10:11, Michal Hocko wrote: > >>The VM stops working (e.g. not pingable) after around 8h (will be restarted > >>automatically), happened serveral times. > >> > >>Had also further OOMs which I sent to Mincham. > >Could you post them to the mailing list as well, please? > > Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as > well on 4.9.9-200.fc25.x86_64 > > On 4.10er kernels: [...] > kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB > active_anon:143580kB inactive_anon:143300kB active_file:2576kB > inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB > managed:353968kB mlocked:0kB slab_reclaimable:13708kB > slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB > free_pcp:412kB local_pcp:88kB free_cma:0kB [...] > On 4.9er kernels: [...] > kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB > active_anon:122148kB inactive_anon:112068kB active_file:81324kB > inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB > managed:484384kB mlocked:0kB slab_reclaimable:17660kB > slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB > free_pcp:120kB local_pcp:0kB free_cma:0kB In both cases the amount if free memory is above the min watermark, so we shouldn't be hitting the oom. We might have somebody freeing memory after the last attempt, though... [...] > Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM > with ext4 and Fedora 25 and some memory load and updating the VM. > > Any further progress? The linux-next (resp. mmotm tree) has new tracepoints which should help to tell us more about what is going on here. Could you try to enable oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end} -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-02-27 8:27 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-02-27 8:27 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote: > On 04.01.2017 10:11, Michal Hocko wrote: > >>The VM stops working (e.g. not pingable) after around 8h (will be restarted > >>automatically), happened serveral times. > >> > >>Had also further OOMs which I sent to Mincham. > >Could you post them to the mailing list as well, please? > > Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as > well on 4.9.9-200.fc25.x86_64 > > On 4.10er kernels: [...] > kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB > active_anon:143580kB inactive_anon:143300kB active_file:2576kB > inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB > managed:353968kB mlocked:0kB slab_reclaimable:13708kB > slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB > free_pcp:412kB local_pcp:88kB free_cma:0kB [...] > On 4.9er kernels: [...] > kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB > active_anon:122148kB inactive_anon:112068kB active_file:81324kB > inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB > managed:484384kB mlocked:0kB slab_reclaimable:17660kB > slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB > free_pcp:120kB local_pcp:0kB free_cma:0kB In both cases the amount if free memory is above the min watermark, so we shouldn't be hitting the oom. We might have somebody freeing memory after the last attempt, though... [...] > Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM > with ext4 and Fedora 25 and some memory load and updating the VM. > > Any further progress? The linux-next (resp. mmotm tree) has new tracepoints which should help to tell us more about what is going on here. Could you try to enable oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end} -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-02-27 8:27 ` Michal Hocko @ 2017-02-28 6:06 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-02-28 6:06 UTC (permalink / raw) To: Michal Hocko; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 27.02.2017 09:27, Michal Hocko wrote: > On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote: >> On 04.01.2017 10:11, Michal Hocko wrote: >>>> The VM stops working (e.g. not pingable) after around 8h (will be restarted >>>> automatically), happened serveral times. >>>> >>>> Had also further OOMs which I sent to Mincham. >>> Could you post them to the mailing list as well, please? >> Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as >> well on 4.9.9-200.fc25.x86_64 >> >> On 4.10er kernels: > [...] >> kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB >> active_anon:143580kB inactive_anon:143300kB active_file:2576kB >> inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB >> managed:353968kB mlocked:0kB slab_reclaimable:13708kB >> slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB >> free_pcp:412kB local_pcp:88kB free_cma:0kB > [...] > >> On 4.9er kernels: > [...] >> kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB >> active_anon:122148kB inactive_anon:112068kB active_file:81324kB >> inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB >> managed:484384kB mlocked:0kB slab_reclaimable:17660kB >> slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB >> free_pcp:120kB local_pcp:0kB free_cma:0kB > In both cases the amount if free memory is above the min watermark, so > we shouldn't be hitting the oom. We might have somebody freeing memory > after the last attempt, though... > > [...] >> Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM >> with ext4 and Fedora 25 and some memory load and updating the VM. >> >> Any further progress? > The linux-next (resp. mmotm tree) has new tracepoints which should help > to tell us more about what is going on here. Could you try to enable > oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end} Is this available in this version? https://koji.fedoraproject.org/koji/buildinfo?buildID=862775 kernel-4.11.0-0.rc0.git5.1.fc26 How to enable? Thnx. Ciao, gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-02-28 6:06 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-02-28 6:06 UTC (permalink / raw) To: Michal Hocko; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 27.02.2017 09:27, Michal Hocko wrote: > On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote: >> On 04.01.2017 10:11, Michal Hocko wrote: >>>> The VM stops working (e.g. not pingable) after around 8h (will be restarted >>>> automatically), happened serveral times. >>>> >>>> Had also further OOMs which I sent to Mincham. >>> Could you post them to the mailing list as well, please? >> Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as >> well on 4.9.9-200.fc25.x86_64 >> >> On 4.10er kernels: > [...] >> kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB >> active_anon:143580kB inactive_anon:143300kB active_file:2576kB >> inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB >> managed:353968kB mlocked:0kB slab_reclaimable:13708kB >> slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB >> free_pcp:412kB local_pcp:88kB free_cma:0kB > [...] > >> On 4.9er kernels: > [...] >> kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB >> active_anon:122148kB inactive_anon:112068kB active_file:81324kB >> inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB >> managed:484384kB mlocked:0kB slab_reclaimable:17660kB >> slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB >> free_pcp:120kB local_pcp:0kB free_cma:0kB > In both cases the amount if free memory is above the min watermark, so > we shouldn't be hitting the oom. We might have somebody freeing memory > after the last attempt, though... > > [...] >> Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM >> with ext4 and Fedora 25 and some memory load and updating the VM. >> >> Any further progress? > The linux-next (resp. mmotm tree) has new tracepoints which should help > to tell us more about what is going on here. Could you try to enable > oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end} Is this available in this version? https://koji.fedoraproject.org/koji/buildinfo?buildID=862775 kernel-4.11.0-0.rc0.git5.1.fc26 How to enable? Thnx. Ciao, gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-02-28 6:06 ` Gerhard Wiesinger @ 2017-02-28 8:14 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-02-28 8:14 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Tue 28-02-17 07:06:41, Gerhard Wiesinger wrote: > On 27.02.2017 09:27, Michal Hocko wrote: > >On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote: > >>On 04.01.2017 10:11, Michal Hocko wrote: > >>>>The VM stops working (e.g. not pingable) after around 8h (will be restarted > >>>>automatically), happened serveral times. > >>>> > >>>>Had also further OOMs which I sent to Mincham. > >>>Could you post them to the mailing list as well, please? > >>Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as > >>well on 4.9.9-200.fc25.x86_64 > >> > >>On 4.10er kernels: > >[...] > >>kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB > >>active_anon:143580kB inactive_anon:143300kB active_file:2576kB > >>inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB > >>managed:353968kB mlocked:0kB slab_reclaimable:13708kB > >>slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB > >>free_pcp:412kB local_pcp:88kB free_cma:0kB > >[...] > > > >>On 4.9er kernels: > >[...] > >>kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB > >>active_anon:122148kB inactive_anon:112068kB active_file:81324kB > >>inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB > >>managed:484384kB mlocked:0kB slab_reclaimable:17660kB > >>slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB > >>free_pcp:120kB local_pcp:0kB free_cma:0kB > >In both cases the amount if free memory is above the min watermark, so > >we shouldn't be hitting the oom. We might have somebody freeing memory > >after the last attempt, though... > > > >[...] > >>Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM > >>with ext4 and Fedora 25 and some memory load and updating the VM. > >> > >>Any further progress? > >The linux-next (resp. mmotm tree) has new tracepoints which should help > >to tell us more about what is going on here. Could you try to enable > >oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end} > > Is this available in this version? > > https://koji.fedoraproject.org/koji/buildinfo?buildID=862775 > > kernel-4.11.0-0.rc0.git5.1.fc26 no idea. > > How to enable? mount -t tracefs none /trace echo 1 > /trace/events/oom/reclaim_retry_zone/enabled echo 1 > /trace/events/vmscan/mm_vmscan_direct_reclaim_begin echo 1 > /trace/events/vmscan/mm_vmscan_direct_reclaim_end -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-02-28 8:14 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-02-28 8:14 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Tue 28-02-17 07:06:41, Gerhard Wiesinger wrote: > On 27.02.2017 09:27, Michal Hocko wrote: > >On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote: > >>On 04.01.2017 10:11, Michal Hocko wrote: > >>>>The VM stops working (e.g. not pingable) after around 8h (will be restarted > >>>>automatically), happened serveral times. > >>>> > >>>>Had also further OOMs which I sent to Mincham. > >>>Could you post them to the mailing list as well, please? > >>Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as > >>well on 4.9.9-200.fc25.x86_64 > >> > >>On 4.10er kernels: > >[...] > >>kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB > >>active_anon:143580kB inactive_anon:143300kB active_file:2576kB > >>inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB > >>managed:353968kB mlocked:0kB slab_reclaimable:13708kB > >>slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB > >>free_pcp:412kB local_pcp:88kB free_cma:0kB > >[...] > > > >>On 4.9er kernels: > >[...] > >>kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB > >>active_anon:122148kB inactive_anon:112068kB active_file:81324kB > >>inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB > >>managed:484384kB mlocked:0kB slab_reclaimable:17660kB > >>slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB > >>free_pcp:120kB local_pcp:0kB free_cma:0kB > >In both cases the amount if free memory is above the min watermark, so > >we shouldn't be hitting the oom. We might have somebody freeing memory > >after the last attempt, though... > > > >[...] > >>Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM > >>with ext4 and Fedora 25 and some memory load and updating the VM. > >> > >>Any further progress? > >The linux-next (resp. mmotm tree) has new tracepoints which should help > >to tell us more about what is going on here. Could you try to enable > >oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end} > > Is this available in this version? > > https://koji.fedoraproject.org/koji/buildinfo?buildID=862775 > > kernel-4.11.0-0.rc0.git5.1.fc26 no idea. > > How to enable? mount -t tracefs none /trace echo 1 > /trace/events/oom/reclaim_retry_zone/enabled echo 1 > /trace/events/vmscan/mm_vmscan_direct_reclaim_begin echo 1 > /trace/events/vmscan/mm_vmscan_direct_reclaim_end -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-02-26 8:40 ` Still OOM problems with 4.9er/4.10er kernels Gerhard Wiesinger @ 2017-02-27 9:02 ` Minchan Kim 2017-02-27 9:02 ` Minchan Kim 1 sibling, 0 replies; 88+ messages in thread From: Minchan Kim @ 2017-02-27 9:02 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Michal Hocko, linux-kernel, linux-mm, Linus Torvalds On Sun, Feb 26, 2017 at 09:40:42AM +0100, Gerhard Wiesinger wrote: > On 04.01.2017 10:11, Michal Hocko wrote: > >>The VM stops working (e.g. not pingable) after around 8h (will be restarted > >>automatically), happened serveral times. > >> > >>Had also further OOMs which I sent to Mincham. > >Could you post them to the mailing list as well, please? > > Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as > well on 4.9.9-200.fc25.x86_64 > > On 4.10er kernels: > > Free swap = 1137532kB > > cat /etc/sysctl.d/* | grep ^vm > vm.dirty_background_ratio = 3 > vm.dirty_ratio = 15 > vm.overcommit_memory = 2 > vm.overcommit_ratio = 80 > vm.swappiness=10 > > kernel: python invoked oom-killer: > gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, > oom_score_adj=0 > kernel: python cpuset=/ mems_allowed=0 > kernel: CPU: 1 PID: 813 Comm: python Not tainted 4.10.0-1.fc26.x86_64 #1 > kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 > 04/01/2014 > kernel: Call Trace: > kernel: dump_stack+0x63/0x84 > kernel: dump_header+0x7b/0x1f6 > kernel: ? do_try_to_free_pages+0x2c5/0x340 > kernel: oom_kill_process+0x202/0x3d0 > kernel: out_of_memory+0x2b7/0x4e0 > kernel: __alloc_pages_slowpath+0x915/0xb80 > kernel: __alloc_pages_nodemask+0x218/0x2d0 > kernel: alloc_pages_current+0x93/0x150 > kernel: __page_cache_alloc+0xcf/0x100 > kernel: filemap_fault+0x39d/0x800 > kernel: ? page_add_file_rmap+0xe5/0x200 > kernel: ? filemap_map_pages+0x2e1/0x4e0 > kernel: ext4_filemap_fault+0x36/0x50 > kernel: __do_fault+0x21/0x110 > kernel: handle_mm_fault+0xdd1/0x1410 > kernel: ? swake_up+0x42/0x50 > kernel: __do_page_fault+0x23f/0x4c0 > kernel: trace_do_page_fault+0x41/0x120 > kernel: do_async_page_fault+0x51/0xa0 > kernel: async_page_fault+0x28/0x30 > kernel: RIP: 0033:0x7f0681ad6350 > kernel: RSP: 002b:00007ffcbdd238d8 EFLAGS: 00010246 > kernel: RAX: 00007f0681b0f960 RBX: 0000000000000000 RCX: 7fffffffffffffff > kernel: RDX: 0000000000000000 RSI: 3ff0000000000000 RDI: 3ff0000000000000 > kernel: RBP: 00007f067461ab40 R08: 0000000000000000 R09: 3ff0000000000000 > kernel: R10: 0000556f1c6d8a80 R11: 0000000000000001 R12: 00007f0676d1a8d0 > kernel: R13: 0000000000000000 R14: 00007f06746168bc R15: 00007f0674385910 > kernel: Mem-Info: > kernel: active_anon:37423 inactive_anon:37512 isolated_anon:0 > active_file:462 inactive_file:603 isolated_file:0 > unevictable:0 dirty:0 writeback:0 unstable:0 > slab_reclaimable:3538 slab_unreclaimable:4818 > mapped:859 shmem:9 pagetables:3370 bounce:0 > free:1650 free_pcp:103 free_cma:0 > kernel: Node 0 active_anon:149380kB inactive_anon:149704kB > active_file:1848kB inactive_file:3660kB unevictable:0kB isolated(anon):128kB > isolated(file):0kB mapped:4580kB dirty:0kB writeback:380kB shmem:0kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB > unstable:0kB pages_scanned:352 all_unreclaimable? no > kernel: Node 0 DMA free:1484kB min:104kB low:128kB high:152kB > active_anon:5660kB inactive_anon:6156kB active_file:56kB inactive_file:64kB > unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB > slab_reclaimable:444kB slab_unreclaimable:1208kB kernel_stack:32kB > pagetables:592kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > kernel: lowmem_reserve[]: 0 327 327 327 327 > kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB > active_anon:143580kB inactive_anon:143300kB active_file:2576kB > inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB > managed:353968kB mlocked:0kB slab_reclaimable:13708kB > slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB > free_pcp:412kB local_pcp:88kB free_cma:0kB > kernel: lowmem_reserve[]: 0 0 0 0 0 > kernel: Node 0 DMA: 70*4kB (UMEH) 20*8kB (UMEH) 13*16kB (MH) 5*32kB (H) > 4*64kB (H) 2*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 1576kB > kernel: Node 0 DMA32: 1134*4kB (UMEH) 25*8kB (UMEH) 13*16kB (MH) 7*32kB (H) > 3*64kB (H) 0*128kB 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5616kB Althogh DMA32 zone has enough free memory, free memory includes H pageblock which is reserved memory for high-order atomic allocation. That might be a reason you cannot succeed watermark check for the allocation. I tried to solve the issue in 4.9 time to use up the reserved memory before the OOM and merged into 4.10 but I think there is a hole so could you apply this patch on top of your 4.10? (To be clear, cannot apply it to 4.9) >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 From: Minchan Kim <minchan@kernel.org> Date: Mon, 27 Feb 2017 17:39:06 +0900 Subject: [PATCH] mm: use up highatomic before OOM kill Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> --- mm/page_alloc.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 614cd0397ce3..e073cca4969e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, *no_progress_loops = 0; else (*no_progress_loops)++; - - /* - * Make sure we converge to OOM if we cannot make any progress - * several times in the row. - */ - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { - /* Before OOM, exhaust highatomic_reserve */ - return unreserve_highatomic_pageblock(ac, true); - } - /* * Keep reclaiming pages while there is a chance this will lead * somewhere. If none of the target zones can satisfy our allocation @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, if (read_mems_allowed_retry(cpuset_mems_cookie)) goto retry_cpuset; + /* Before OOM, exhaust highatomic_reserve */ + if (unreserve_highatomic_pageblock(ac, true)) + goto retry; + /* Reclaim has failed us, start killing things */ page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); if (page) -- 2.7.4 ^ permalink raw reply related [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-02-27 9:02 ` Minchan Kim 0 siblings, 0 replies; 88+ messages in thread From: Minchan Kim @ 2017-02-27 9:02 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Michal Hocko, linux-kernel, linux-mm, Linus Torvalds On Sun, Feb 26, 2017 at 09:40:42AM +0100, Gerhard Wiesinger wrote: > On 04.01.2017 10:11, Michal Hocko wrote: > >>The VM stops working (e.g. not pingable) after around 8h (will be restarted > >>automatically), happened serveral times. > >> > >>Had also further OOMs which I sent to Mincham. > >Could you post them to the mailing list as well, please? > > Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as > well on 4.9.9-200.fc25.x86_64 > > On 4.10er kernels: > > Free swap = 1137532kB > > cat /etc/sysctl.d/* | grep ^vm > vm.dirty_background_ratio = 3 > vm.dirty_ratio = 15 > vm.overcommit_memory = 2 > vm.overcommit_ratio = 80 > vm.swappiness=10 > > kernel: python invoked oom-killer: > gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, > oom_score_adj=0 > kernel: python cpuset=/ mems_allowed=0 > kernel: CPU: 1 PID: 813 Comm: python Not tainted 4.10.0-1.fc26.x86_64 #1 > kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 > 04/01/2014 > kernel: Call Trace: > kernel: dump_stack+0x63/0x84 > kernel: dump_header+0x7b/0x1f6 > kernel: ? do_try_to_free_pages+0x2c5/0x340 > kernel: oom_kill_process+0x202/0x3d0 > kernel: out_of_memory+0x2b7/0x4e0 > kernel: __alloc_pages_slowpath+0x915/0xb80 > kernel: __alloc_pages_nodemask+0x218/0x2d0 > kernel: alloc_pages_current+0x93/0x150 > kernel: __page_cache_alloc+0xcf/0x100 > kernel: filemap_fault+0x39d/0x800 > kernel: ? page_add_file_rmap+0xe5/0x200 > kernel: ? filemap_map_pages+0x2e1/0x4e0 > kernel: ext4_filemap_fault+0x36/0x50 > kernel: __do_fault+0x21/0x110 > kernel: handle_mm_fault+0xdd1/0x1410 > kernel: ? swake_up+0x42/0x50 > kernel: __do_page_fault+0x23f/0x4c0 > kernel: trace_do_page_fault+0x41/0x120 > kernel: do_async_page_fault+0x51/0xa0 > kernel: async_page_fault+0x28/0x30 > kernel: RIP: 0033:0x7f0681ad6350 > kernel: RSP: 002b:00007ffcbdd238d8 EFLAGS: 00010246 > kernel: RAX: 00007f0681b0f960 RBX: 0000000000000000 RCX: 7fffffffffffffff > kernel: RDX: 0000000000000000 RSI: 3ff0000000000000 RDI: 3ff0000000000000 > kernel: RBP: 00007f067461ab40 R08: 0000000000000000 R09: 3ff0000000000000 > kernel: R10: 0000556f1c6d8a80 R11: 0000000000000001 R12: 00007f0676d1a8d0 > kernel: R13: 0000000000000000 R14: 00007f06746168bc R15: 00007f0674385910 > kernel: Mem-Info: > kernel: active_anon:37423 inactive_anon:37512 isolated_anon:0 > active_file:462 inactive_file:603 isolated_file:0 > unevictable:0 dirty:0 writeback:0 unstable:0 > slab_reclaimable:3538 slab_unreclaimable:4818 > mapped:859 shmem:9 pagetables:3370 bounce:0 > free:1650 free_pcp:103 free_cma:0 > kernel: Node 0 active_anon:149380kB inactive_anon:149704kB > active_file:1848kB inactive_file:3660kB unevictable:0kB isolated(anon):128kB > isolated(file):0kB mapped:4580kB dirty:0kB writeback:380kB shmem:0kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB > unstable:0kB pages_scanned:352 all_unreclaimable? no > kernel: Node 0 DMA free:1484kB min:104kB low:128kB high:152kB > active_anon:5660kB inactive_anon:6156kB active_file:56kB inactive_file:64kB > unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB > slab_reclaimable:444kB slab_unreclaimable:1208kB kernel_stack:32kB > pagetables:592kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > kernel: lowmem_reserve[]: 0 327 327 327 327 > kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB > active_anon:143580kB inactive_anon:143300kB active_file:2576kB > inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB > managed:353968kB mlocked:0kB slab_reclaimable:13708kB > slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB > free_pcp:412kB local_pcp:88kB free_cma:0kB > kernel: lowmem_reserve[]: 0 0 0 0 0 > kernel: Node 0 DMA: 70*4kB (UMEH) 20*8kB (UMEH) 13*16kB (MH) 5*32kB (H) > 4*64kB (H) 2*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 1576kB > kernel: Node 0 DMA32: 1134*4kB (UMEH) 25*8kB (UMEH) 13*16kB (MH) 7*32kB (H) > 3*64kB (H) 0*128kB 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5616kB Althogh DMA32 zone has enough free memory, free memory includes H pageblock which is reserved memory for high-order atomic allocation. That might be a reason you cannot succeed watermark check for the allocation. I tried to solve the issue in 4.9 time to use up the reserved memory before the OOM and merged into 4.10 but I think there is a hole so could you apply this patch on top of your 4.10? (To be clear, cannot apply it to 4.9) ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-02-27 9:02 ` Minchan Kim @ 2017-02-27 9:44 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-02-27 9:44 UTC (permalink / raw) To: Minchan Kim; +Cc: Gerhard Wiesinger, linux-kernel, linux-mm, Linus Torvalds On Mon 27-02-17 18:02:36, Minchan Kim wrote: [...] > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 > From: Minchan Kim <minchan@kernel.org> > Date: Mon, 27 Feb 2017 17:39:06 +0900 > Subject: [PATCH] mm: use up highatomic before OOM kill > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> > --- > mm/page_alloc.c | 14 ++++---------- > 1 file changed, 4 insertions(+), 10 deletions(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 614cd0397ce3..e073cca4969e 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, > *no_progress_loops = 0; > else > (*no_progress_loops)++; > - > - /* > - * Make sure we converge to OOM if we cannot make any progress > - * several times in the row. > - */ > - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { > - /* Before OOM, exhaust highatomic_reserve */ > - return unreserve_highatomic_pageblock(ac, true); > - } > - > /* > * Keep reclaiming pages while there is a chance this will lead > * somewhere. If none of the target zones can satisfy our allocation > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > if (read_mems_allowed_retry(cpuset_mems_cookie)) > goto retry_cpuset; > > + /* Before OOM, exhaust highatomic_reserve */ > + if (unreserve_highatomic_pageblock(ac, true)) > + goto retry; > + OK, this can help for higher order requests when we do not exhaust all the retries and fail on compaction but I fail to see how can this help for order-0 requets which was what happened in this case. I am not saying this is wrong, though. > /* Reclaim has failed us, start killing things */ > page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); > if (page) > -- > 2.7.4 -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-02-27 9:44 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-02-27 9:44 UTC (permalink / raw) To: Minchan Kim; +Cc: Gerhard Wiesinger, linux-kernel, linux-mm, Linus Torvalds On Mon 27-02-17 18:02:36, Minchan Kim wrote: [...] > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 > From: Minchan Kim <minchan@kernel.org> > Date: Mon, 27 Feb 2017 17:39:06 +0900 > Subject: [PATCH] mm: use up highatomic before OOM kill > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> > --- > mm/page_alloc.c | 14 ++++---------- > 1 file changed, 4 insertions(+), 10 deletions(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 614cd0397ce3..e073cca4969e 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, > *no_progress_loops = 0; > else > (*no_progress_loops)++; > - > - /* > - * Make sure we converge to OOM if we cannot make any progress > - * several times in the row. > - */ > - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { > - /* Before OOM, exhaust highatomic_reserve */ > - return unreserve_highatomic_pageblock(ac, true); > - } > - > /* > * Keep reclaiming pages while there is a chance this will lead > * somewhere. If none of the target zones can satisfy our allocation > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > if (read_mems_allowed_retry(cpuset_mems_cookie)) > goto retry_cpuset; > > + /* Before OOM, exhaust highatomic_reserve */ > + if (unreserve_highatomic_pageblock(ac, true)) > + goto retry; > + OK, this can help for higher order requests when we do not exhaust all the retries and fail on compaction but I fail to see how can this help for order-0 requets which was what happened in this case. I am not saying this is wrong, though. > /* Reclaim has failed us, start killing things */ > page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); > if (page) > -- > 2.7.4 -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-02-27 9:44 ` Michal Hocko @ 2017-02-28 5:17 ` Minchan Kim -1 siblings, 0 replies; 88+ messages in thread From: Minchan Kim @ 2017-02-28 5:17 UTC (permalink / raw) To: Michal Hocko; +Cc: Gerhard Wiesinger, linux-kernel, linux-mm, Linus Torvalds On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote: > On Mon 27-02-17 18:02:36, Minchan Kim wrote: > [...] > > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 > > From: Minchan Kim <minchan@kernel.org> > > Date: Mon, 27 Feb 2017 17:39:06 +0900 > > Subject: [PATCH] mm: use up highatomic before OOM kill > > > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> > > --- > > mm/page_alloc.c | 14 ++++---------- > > 1 file changed, 4 insertions(+), 10 deletions(-) > > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > index 614cd0397ce3..e073cca4969e 100644 > > --- a/mm/page_alloc.c > > +++ b/mm/page_alloc.c > > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, > > *no_progress_loops = 0; > > else > > (*no_progress_loops)++; > > - > > - /* > > - * Make sure we converge to OOM if we cannot make any progress > > - * several times in the row. > > - */ > > - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { > > - /* Before OOM, exhaust highatomic_reserve */ > > - return unreserve_highatomic_pageblock(ac, true); > > - } > > - > > /* > > * Keep reclaiming pages while there is a chance this will lead > > * somewhere. If none of the target zones can satisfy our allocation > > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > > if (read_mems_allowed_retry(cpuset_mems_cookie)) > > goto retry_cpuset; > > > > + /* Before OOM, exhaust highatomic_reserve */ > > + if (unreserve_highatomic_pageblock(ac, true)) > > + goto retry; > > + > > OK, this can help for higher order requests when we do not exhaust all > the retries and fail on compaction but I fail to see how can this help > for order-0 requets which was what happened in this case. I am not > saying this is wrong, though. The should_reclaim_retry can return false although no_progress_loop is less than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages by the progress_loop. In that case, unreserve_highatomic_pageblock cannot be called so that VM can keep a pageblock(e.g., 2M) for highatomic reserve. Then, zone_watermark_ok subtracts nr_reserved_highatomic pages for the pass/fail decision whichs is very conservative but no choice for the hot path performance. With that, order-0 allocation can be failed. Thanks. ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-02-28 5:17 ` Minchan Kim 0 siblings, 0 replies; 88+ messages in thread From: Minchan Kim @ 2017-02-28 5:17 UTC (permalink / raw) To: Michal Hocko; +Cc: Gerhard Wiesinger, linux-kernel, linux-mm, Linus Torvalds On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote: > On Mon 27-02-17 18:02:36, Minchan Kim wrote: > [...] > > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 > > From: Minchan Kim <minchan@kernel.org> > > Date: Mon, 27 Feb 2017 17:39:06 +0900 > > Subject: [PATCH] mm: use up highatomic before OOM kill > > > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> > > --- > > mm/page_alloc.c | 14 ++++---------- > > 1 file changed, 4 insertions(+), 10 deletions(-) > > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > index 614cd0397ce3..e073cca4969e 100644 > > --- a/mm/page_alloc.c > > +++ b/mm/page_alloc.c > > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, > > *no_progress_loops = 0; > > else > > (*no_progress_loops)++; > > - > > - /* > > - * Make sure we converge to OOM if we cannot make any progress > > - * several times in the row. > > - */ > > - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { > > - /* Before OOM, exhaust highatomic_reserve */ > > - return unreserve_highatomic_pageblock(ac, true); > > - } > > - > > /* > > * Keep reclaiming pages while there is a chance this will lead > > * somewhere. If none of the target zones can satisfy our allocation > > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > > if (read_mems_allowed_retry(cpuset_mems_cookie)) > > goto retry_cpuset; > > > > + /* Before OOM, exhaust highatomic_reserve */ > > + if (unreserve_highatomic_pageblock(ac, true)) > > + goto retry; > > + > > OK, this can help for higher order requests when we do not exhaust all > the retries and fail on compaction but I fail to see how can this help > for order-0 requets which was what happened in this case. I am not > saying this is wrong, though. The should_reclaim_retry can return false although no_progress_loop is less than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages by the progress_loop. In that case, unreserve_highatomic_pageblock cannot be called so that VM can keep a pageblock(e.g., 2M) for highatomic reserve. Then, zone_watermark_ok subtracts nr_reserved_highatomic pages for the pass/fail decision whichs is very conservative but no choice for the hot path performance. With that, order-0 allocation can be failed. Thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-02-28 5:17 ` Minchan Kim @ 2017-02-28 8:12 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-02-28 8:12 UTC (permalink / raw) To: Minchan Kim; +Cc: Gerhard Wiesinger, linux-kernel, linux-mm, Linus Torvalds On Tue 28-02-17 14:17:23, Minchan Kim wrote: > On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote: > > On Mon 27-02-17 18:02:36, Minchan Kim wrote: > > [...] > > > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 > > > From: Minchan Kim <minchan@kernel.org> > > > Date: Mon, 27 Feb 2017 17:39:06 +0900 > > > Subject: [PATCH] mm: use up highatomic before OOM kill > > > > > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> > > > --- > > > mm/page_alloc.c | 14 ++++---------- > > > 1 file changed, 4 insertions(+), 10 deletions(-) > > > > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > > index 614cd0397ce3..e073cca4969e 100644 > > > --- a/mm/page_alloc.c > > > +++ b/mm/page_alloc.c > > > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, > > > *no_progress_loops = 0; > > > else > > > (*no_progress_loops)++; > > > - > > > - /* > > > - * Make sure we converge to OOM if we cannot make any progress > > > - * several times in the row. > > > - */ > > > - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { > > > - /* Before OOM, exhaust highatomic_reserve */ > > > - return unreserve_highatomic_pageblock(ac, true); > > > - } > > > - > > > /* > > > * Keep reclaiming pages while there is a chance this will lead > > > * somewhere. If none of the target zones can satisfy our allocation > > > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > > > if (read_mems_allowed_retry(cpuset_mems_cookie)) > > > goto retry_cpuset; > > > > > > + /* Before OOM, exhaust highatomic_reserve */ > > > + if (unreserve_highatomic_pageblock(ac, true)) > > > + goto retry; > > > + > > > > OK, this can help for higher order requests when we do not exhaust all > > the retries and fail on compaction but I fail to see how can this help > > for order-0 requets which was what happened in this case. I am not > > saying this is wrong, though. > > The should_reclaim_retry can return false although no_progress_loop is less > than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages > by the progress_loop. Yes, sorry I should have been more clear. I was talking about this particular case where we had a lot of reclaimable pages (a lot of anonymous with the swap available). -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-02-28 8:12 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-02-28 8:12 UTC (permalink / raw) To: Minchan Kim; +Cc: Gerhard Wiesinger, linux-kernel, linux-mm, Linus Torvalds On Tue 28-02-17 14:17:23, Minchan Kim wrote: > On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote: > > On Mon 27-02-17 18:02:36, Minchan Kim wrote: > > [...] > > > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 > > > From: Minchan Kim <minchan@kernel.org> > > > Date: Mon, 27 Feb 2017 17:39:06 +0900 > > > Subject: [PATCH] mm: use up highatomic before OOM kill > > > > > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> > > > --- > > > mm/page_alloc.c | 14 ++++---------- > > > 1 file changed, 4 insertions(+), 10 deletions(-) > > > > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > > index 614cd0397ce3..e073cca4969e 100644 > > > --- a/mm/page_alloc.c > > > +++ b/mm/page_alloc.c > > > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, > > > *no_progress_loops = 0; > > > else > > > (*no_progress_loops)++; > > > - > > > - /* > > > - * Make sure we converge to OOM if we cannot make any progress > > > - * several times in the row. > > > - */ > > > - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { > > > - /* Before OOM, exhaust highatomic_reserve */ > > > - return unreserve_highatomic_pageblock(ac, true); > > > - } > > > - > > > /* > > > * Keep reclaiming pages while there is a chance this will lead > > > * somewhere. If none of the target zones can satisfy our allocation > > > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > > > if (read_mems_allowed_retry(cpuset_mems_cookie)) > > > goto retry_cpuset; > > > > > > + /* Before OOM, exhaust highatomic_reserve */ > > > + if (unreserve_highatomic_pageblock(ac, true)) > > > + goto retry; > > > + > > > > OK, this can help for higher order requests when we do not exhaust all > > the retries and fail on compaction but I fail to see how can this help > > for order-0 requets which was what happened in this case. I am not > > saying this is wrong, though. > > The should_reclaim_retry can return false although no_progress_loop is less > than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages > by the progress_loop. Yes, sorry I should have been more clear. I was talking about this particular case where we had a lot of reclaimable pages (a lot of anonymous with the swap available). -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-02-28 8:12 ` Michal Hocko @ 2017-03-02 7:17 ` Minchan Kim -1 siblings, 0 replies; 88+ messages in thread From: Minchan Kim @ 2017-03-02 7:17 UTC (permalink / raw) To: Michal Hocko; +Cc: Gerhard Wiesinger, linux-kernel, linux-mm, Linus Torvalds Hi Michal, On Tue, Feb 28, 2017 at 09:12:24AM +0100, Michal Hocko wrote: > On Tue 28-02-17 14:17:23, Minchan Kim wrote: > > On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote: > > > On Mon 27-02-17 18:02:36, Minchan Kim wrote: > > > [...] > > > > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 > > > > From: Minchan Kim <minchan@kernel.org> > > > > Date: Mon, 27 Feb 2017 17:39:06 +0900 > > > > Subject: [PATCH] mm: use up highatomic before OOM kill > > > > > > > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> > > > > --- > > > > mm/page_alloc.c | 14 ++++---------- > > > > 1 file changed, 4 insertions(+), 10 deletions(-) > > > > > > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > > > index 614cd0397ce3..e073cca4969e 100644 > > > > --- a/mm/page_alloc.c > > > > +++ b/mm/page_alloc.c > > > > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, > > > > *no_progress_loops = 0; > > > > else > > > > (*no_progress_loops)++; > > > > - > > > > - /* > > > > - * Make sure we converge to OOM if we cannot make any progress > > > > - * several times in the row. > > > > - */ > > > > - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { > > > > - /* Before OOM, exhaust highatomic_reserve */ > > > > - return unreserve_highatomic_pageblock(ac, true); > > > > - } > > > > - > > > > /* > > > > * Keep reclaiming pages while there is a chance this will lead > > > > * somewhere. If none of the target zones can satisfy our allocation > > > > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > > > > if (read_mems_allowed_retry(cpuset_mems_cookie)) > > > > goto retry_cpuset; > > > > > > > > + /* Before OOM, exhaust highatomic_reserve */ > > > > + if (unreserve_highatomic_pageblock(ac, true)) > > > > + goto retry; > > > > + > > > > > > OK, this can help for higher order requests when we do not exhaust all > > > the retries and fail on compaction but I fail to see how can this help > > > for order-0 requets which was what happened in this case. I am not > > > saying this is wrong, though. > > > > The should_reclaim_retry can return false although no_progress_loop is less > > than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages > > by the progress_loop. > > Yes, sorry I should have been more clear. I was talking about this > particular case where we had a lot of reclaimable pages (a lot of > anonymous with the swap available). This reports shows two problems. Why we see OOM 1) enough *free* pages and 2) enough *freeable* pages. I just pointed out 1) and sent the patch to solve it. About 2), one of my imaginary scenario is inactive anon list is full of pinned pages so VM can unmap them successfully in shrink_page_list but fail to free due to increased page refcount. In that case, the page will be added to inactive anonymous LRU list again without activating so inactive_list_is_low on anonymous LRU is always false. IOW, there is no deactivation from active list. It's just my picture without no clue. ;-) ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-02 7:17 ` Minchan Kim 0 siblings, 0 replies; 88+ messages in thread From: Minchan Kim @ 2017-03-02 7:17 UTC (permalink / raw) To: Michal Hocko; +Cc: Gerhard Wiesinger, linux-kernel, linux-mm, Linus Torvalds Hi Michal, On Tue, Feb 28, 2017 at 09:12:24AM +0100, Michal Hocko wrote: > On Tue 28-02-17 14:17:23, Minchan Kim wrote: > > On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote: > > > On Mon 27-02-17 18:02:36, Minchan Kim wrote: > > > [...] > > > > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 > > > > From: Minchan Kim <minchan@kernel.org> > > > > Date: Mon, 27 Feb 2017 17:39:06 +0900 > > > > Subject: [PATCH] mm: use up highatomic before OOM kill > > > > > > > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> > > > > --- > > > > mm/page_alloc.c | 14 ++++---------- > > > > 1 file changed, 4 insertions(+), 10 deletions(-) > > > > > > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > > > index 614cd0397ce3..e073cca4969e 100644 > > > > --- a/mm/page_alloc.c > > > > +++ b/mm/page_alloc.c > > > > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, > > > > *no_progress_loops = 0; > > > > else > > > > (*no_progress_loops)++; > > > > - > > > > - /* > > > > - * Make sure we converge to OOM if we cannot make any progress > > > > - * several times in the row. > > > > - */ > > > > - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { > > > > - /* Before OOM, exhaust highatomic_reserve */ > > > > - return unreserve_highatomic_pageblock(ac, true); > > > > - } > > > > - > > > > /* > > > > * Keep reclaiming pages while there is a chance this will lead > > > > * somewhere. If none of the target zones can satisfy our allocation > > > > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > > > > if (read_mems_allowed_retry(cpuset_mems_cookie)) > > > > goto retry_cpuset; > > > > > > > > + /* Before OOM, exhaust highatomic_reserve */ > > > > + if (unreserve_highatomic_pageblock(ac, true)) > > > > + goto retry; > > > > + > > > > > > OK, this can help for higher order requests when we do not exhaust all > > > the retries and fail on compaction but I fail to see how can this help > > > for order-0 requets which was what happened in this case. I am not > > > saying this is wrong, though. > > > > The should_reclaim_retry can return false although no_progress_loop is less > > than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages > > by the progress_loop. > > Yes, sorry I should have been more clear. I was talking about this > particular case where we had a lot of reclaimable pages (a lot of > anonymous with the swap available). This reports shows two problems. Why we see OOM 1) enough *free* pages and 2) enough *freeable* pages. I just pointed out 1) and sent the patch to solve it. About 2), one of my imaginary scenario is inactive anon list is full of pinned pages so VM can unmap them successfully in shrink_page_list but fail to free due to increased page refcount. In that case, the page will be added to inactive anonymous LRU list again without activating so inactive_list_is_low on anonymous LRU is always false. IOW, there is no deactivation from active list. It's just my picture without no clue. ;-) -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-02 7:17 ` Minchan Kim @ 2017-03-16 6:38 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-16 6:38 UTC (permalink / raw) To: Minchan Kim, Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 02.03.2017 08:17, Minchan Kim wrote: > Hi Michal, > > On Tue, Feb 28, 2017 at 09:12:24AM +0100, Michal Hocko wrote: >> On Tue 28-02-17 14:17:23, Minchan Kim wrote: >>> On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote: >>>> On Mon 27-02-17 18:02:36, Minchan Kim wrote: >>>> [...] >>>>> >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 >>>>> From: Minchan Kim <minchan@kernel.org> >>>>> Date: Mon, 27 Feb 2017 17:39:06 +0900 >>>>> Subject: [PATCH] mm: use up highatomic before OOM kill >>>>> >>>>> Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> >>>>> --- >>>>> mm/page_alloc.c | 14 ++++---------- >>>>> 1 file changed, 4 insertions(+), 10 deletions(-) >>>>> >>>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c >>>>> index 614cd0397ce3..e073cca4969e 100644 >>>>> --- a/mm/page_alloc.c >>>>> +++ b/mm/page_alloc.c >>>>> @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, >>>>> *no_progress_loops = 0; >>>>> else >>>>> (*no_progress_loops)++; >>>>> - >>>>> - /* >>>>> - * Make sure we converge to OOM if we cannot make any progress >>>>> - * several times in the row. >>>>> - */ >>>>> - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { >>>>> - /* Before OOM, exhaust highatomic_reserve */ >>>>> - return unreserve_highatomic_pageblock(ac, true); >>>>> - } >>>>> - >>>>> /* >>>>> * Keep reclaiming pages while there is a chance this will lead >>>>> * somewhere. If none of the target zones can satisfy our allocation >>>>> @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, >>>>> if (read_mems_allowed_retry(cpuset_mems_cookie)) >>>>> goto retry_cpuset; >>>>> >>>>> + /* Before OOM, exhaust highatomic_reserve */ >>>>> + if (unreserve_highatomic_pageblock(ac, true)) >>>>> + goto retry; >>>>> + >>>> OK, this can help for higher order requests when we do not exhaust all >>>> the retries and fail on compaction but I fail to see how can this help >>>> for order-0 requets which was what happened in this case. I am not >>>> saying this is wrong, though. >>> The should_reclaim_retry can return false although no_progress_loop is less >>> than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages >>> by the progress_loop. >> Yes, sorry I should have been more clear. I was talking about this >> particular case where we had a lot of reclaimable pages (a lot of >> anonymous with the swap available). > This reports shows two problems. Why we see OOM 1) enough *free* pages and > 2) enough *freeable* pages. > > I just pointed out 1) and sent the patch to solve it. > > About 2), one of my imaginary scenario is inactive anon list is full of > pinned pages so VM can unmap them successfully in shrink_page_list but fail > to free due to increased page refcount. In that case, the page will be added > to inactive anonymous LRU list again without activating so inactive_list_is_low > on anonymous LRU is always false. IOW, there is no deactivation from active list. > > It's just my picture without no clue. ;-) With latest kernels (4.11.0-0.rc2.git0.2.fc26.x86_64) I'm having the issue that swapping is active all the time after some runtime (~1day). top - 07:30:17 up 1 day, 19:42, 1 user, load average: 13.71, 16.98, 15.36 Tasks: 130 total, 2 running, 128 sleeping, 0 stopped, 0 zombie %Cpu(s): 15.8 us, 33.5 sy, 0.0 ni, 3.9 id, 34.5 wa, 4.9 hi, 1.0 si, 6.4 st KiB Mem : 369700 total, 5484 free, 311556 used, 52660 buff/cache KiB Swap: 2064380 total, 1187684 free, 876696 used. 20340 avail Mem [root@smtp ~]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 1 876280 7132 16536 64840 238 226 1027 258 80 97 2 3 83 11 1 0 4 876140 3812 10520 64552 3676 168 11840 1100 2255 2582 7 13 8 70 3 0 3 875372 3628 4024 56160 5424 64 10004 476 2157 2580 2 14 0 83 2 0 4 875560 24056 2208 56296 9032 2180 39928 2388 4111 4549 10 32 0 55 3 2 2 875660 7540 5256 58220 5536 1604 48756 1864 4505 4196 12 23 5 58 3 0 3 875264 3664 2120 57596 2304 116 17904 560 2223 1825 15 15 0 67 3 0 2 875564 3800 588 57856 1340 1068 14780 1184 1390 1364 12 10 0 77 3 1 2 875724 3740 372 53988 3104 928 16884 1068 1560 1527 3 12 0 83 3 0 3 881096 3708 532 52220 4604 5872 21004 6104 2752 2259 7 18 5 67 2 The following commit is included in that version: commit 710531320af876192d76b2c1f68190a1df941b02 Author: Michal Hocko <mhocko@suse.com> Date: Wed Feb 22 15:45:58 2017 -0800 mm, vmscan: cleanup lru size claculations commit fd538803731e50367b7c59ce4ad3454426a3d671 upstream. But still OOMs: [157048.030760] clamscan: page allocation stalls for 19405ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) [157048.031985] clamscan cpuset=/ mems_allowed=0 [157048.031993] CPU: 1 PID: 9597 Comm: clamscan Not tainted 4.11.0-0.rc2.git0.2.fc26.x86_64 #1 [157048.033197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 04/01/2014 [157048.034382] Call Trace: [157048.035532] dump_stack+0x63/0x84 [157048.036735] warn_alloc+0x10c/0x1b0 [157048.037768] __alloc_pages_slowpath+0x93d/0xe60 [157048.038873] ? dd_dispatch_request+0x2b/0x1a0 [157048.041033] ? get_page_from_freelist+0x122/0xbf0 [157048.042435] __alloc_pages_nodemask+0x290/0x2b0 [157048.043662] alloc_pages_vma+0xa0/0x2b0 [157048.044796] __read_swap_cache_async+0x146/0x210 [157048.045841] read_swap_cache_async+0x26/0x60 [157048.046858] swapin_readahead+0x186/0x230 [157048.047854] ? radix_tree_lookup_slot+0x22/0x50 [157048.049006] ? find_get_entry+0x20/0x140 [157048.053109] ? pagecache_get_page+0x2c/0x2e0 [157048.054179] do_swap_page+0x276/0x7b0 [157048.055138] __handle_mm_fault+0x6fd/0x1160 [157048.057571] ? pick_next_task_fair+0x48c/0x560 [157048.058608] handle_mm_fault+0xb3/0x250 [157048.059622] __do_page_fault+0x23f/0x4c0 [157048.068926] trace_do_page_fault+0x41/0x120 [157048.070143] do_async_page_fault+0x51/0xa0 [157048.071254] async_page_fault+0x28/0x30 [157048.072606] RIP: 0033:0x7f78659eb675 [157048.073858] RSP: 002b:00007ffcaba111b8 EFLAGS: 00010202 [157048.075192] RAX: 0000000000000941 RBX: 00007f785957e8d0 RCX: 00007f784e968b48 [157048.076609] RDX: 00007f784f87bce8 RSI: 00007f7851fdb0cb RDI: 00007f7866726000 [157048.077809] RBP: 00007f785957e910 R08: 0000000000040000 R09: 0000000000000000 [157048.078935] R10: ffffffffffffff48 R11: 0000000000000246 R12: 00007f78600c81c0 [157048.080028] R13: 00007f785957e970 R14: 00007f78594ffba8 R15: 0000000003406237 [157048.081827] Mem-Info: [157048.083005] active_anon:19902 inactive_anon:19920 isolated_anon:383 active_file:816 inactive_file:529 isolated_file:0 unevictable:0 dirty:0 writeback:19 unstable:0 slab_reclaimable:4225 slab_unreclaimable:6483 mapped:942 shmem:3 pagetables:3553 bounce:0 free:944 free_pcp:87 free_cma:0 [157048.089470] Node 0 active_anon:79552kB inactive_anon:79588kB active_file:3108kB inactive_file:2144kB unevictable:0kB isolated(anon):1624kB isolated(file):0kB mapped:3612kB dirty:0kB writeback:76kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12kB writeback_tmp:0kB unstable:0kB pages_scanned:247 all_unreclaimable? no [157048.092318] Node 0 DMA free:1408kB min:104kB low:128kB high:152kB active_anon:664kB inactive_anon:3124kB active_file:48kB inactive_file:40kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:564kB slab_unreclaimable:2148kB kernel_stack:92kB pagetables:1328kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [157048.096008] lowmem_reserve[]: 0 327 327 327 327 [157048.097234] Node 0 DMA32 free:2576kB min:2264kB low:2828kB high:3392kB active_anon:78844kB inactive_anon:76612kB active_file:2840kB inactive_file:1896kB unevictable:0kB writepending:76kB present:376688kB managed:353792kB mlocked:0kB slab_reclaimable:16336kB slab_unreclaimable:23784kB kernel_stack:2388kB pagetables:12884kB bounce:0kB free_pcp:644kB local_pcp:312kB free_cma:0kB [157048.101118] lowmem_reserve[]: 0 0 0 0 0 [157048.102190] Node 0 DMA: 37*4kB (UEH) 12*8kB (H) 13*16kB (H) 10*32kB (H) 4*64kB (H) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1412kB [157048.104989] Node 0 DMA32: 79*4kB (UMEH) 199*8kB (UMEH) 18*16kB (UMH) 5*32kB (H) 2*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2484kB [157048.107789] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [157048.107790] 2027 total pagecache pages [157048.109125] 710 pages in swap cache [157048.115088] Swap cache stats: add 36179491, delete 36179123, find 86964755/101977142 [157048.116934] Free swap = 808064kB [157048.118466] Total swap = 2064380kB [157048.122828] 98170 pages RAM [157048.124039] 0 pages HighMem/MovableOnly [157048.125051] 5745 pages reserved [157048.125997] 0 pages cma reserved [157048.127008] 0 pages hwpoisoned Thnx. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-16 6:38 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-16 6:38 UTC (permalink / raw) To: Minchan Kim, Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 02.03.2017 08:17, Minchan Kim wrote: > Hi Michal, > > On Tue, Feb 28, 2017 at 09:12:24AM +0100, Michal Hocko wrote: >> On Tue 28-02-17 14:17:23, Minchan Kim wrote: >>> On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote: >>>> On Mon 27-02-17 18:02:36, Minchan Kim wrote: >>>> [...] >>>>> >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001 >>>>> From: Minchan Kim <minchan@kernel.org> >>>>> Date: Mon, 27 Feb 2017 17:39:06 +0900 >>>>> Subject: [PATCH] mm: use up highatomic before OOM kill >>>>> >>>>> Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org> >>>>> --- >>>>> mm/page_alloc.c | 14 ++++---------- >>>>> 1 file changed, 4 insertions(+), 10 deletions(-) >>>>> >>>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c >>>>> index 614cd0397ce3..e073cca4969e 100644 >>>>> --- a/mm/page_alloc.c >>>>> +++ b/mm/page_alloc.c >>>>> @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, >>>>> *no_progress_loops = 0; >>>>> else >>>>> (*no_progress_loops)++; >>>>> - >>>>> - /* >>>>> - * Make sure we converge to OOM if we cannot make any progress >>>>> - * several times in the row. >>>>> - */ >>>>> - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { >>>>> - /* Before OOM, exhaust highatomic_reserve */ >>>>> - return unreserve_highatomic_pageblock(ac, true); >>>>> - } >>>>> - >>>>> /* >>>>> * Keep reclaiming pages while there is a chance this will lead >>>>> * somewhere. If none of the target zones can satisfy our allocation >>>>> @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, >>>>> if (read_mems_allowed_retry(cpuset_mems_cookie)) >>>>> goto retry_cpuset; >>>>> >>>>> + /* Before OOM, exhaust highatomic_reserve */ >>>>> + if (unreserve_highatomic_pageblock(ac, true)) >>>>> + goto retry; >>>>> + >>>> OK, this can help for higher order requests when we do not exhaust all >>>> the retries and fail on compaction but I fail to see how can this help >>>> for order-0 requets which was what happened in this case. I am not >>>> saying this is wrong, though. >>> The should_reclaim_retry can return false although no_progress_loop is less >>> than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages >>> by the progress_loop. >> Yes, sorry I should have been more clear. I was talking about this >> particular case where we had a lot of reclaimable pages (a lot of >> anonymous with the swap available). > This reports shows two problems. Why we see OOM 1) enough *free* pages and > 2) enough *freeable* pages. > > I just pointed out 1) and sent the patch to solve it. > > About 2), one of my imaginary scenario is inactive anon list is full of > pinned pages so VM can unmap them successfully in shrink_page_list but fail > to free due to increased page refcount. In that case, the page will be added > to inactive anonymous LRU list again without activating so inactive_list_is_low > on anonymous LRU is always false. IOW, there is no deactivation from active list. > > It's just my picture without no clue. ;-) With latest kernels (4.11.0-0.rc2.git0.2.fc26.x86_64) I'm having the issue that swapping is active all the time after some runtime (~1day). top - 07:30:17 up 1 day, 19:42, 1 user, load average: 13.71, 16.98, 15.36 Tasks: 130 total, 2 running, 128 sleeping, 0 stopped, 0 zombie %Cpu(s): 15.8 us, 33.5 sy, 0.0 ni, 3.9 id, 34.5 wa, 4.9 hi, 1.0 si, 6.4 st KiB Mem : 369700 total, 5484 free, 311556 used, 52660 buff/cache KiB Swap: 2064380 total, 1187684 free, 876696 used. 20340 avail Mem [root@smtp ~]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 1 876280 7132 16536 64840 238 226 1027 258 80 97 2 3 83 11 1 0 4 876140 3812 10520 64552 3676 168 11840 1100 2255 2582 7 13 8 70 3 0 3 875372 3628 4024 56160 5424 64 10004 476 2157 2580 2 14 0 83 2 0 4 875560 24056 2208 56296 9032 2180 39928 2388 4111 4549 10 32 0 55 3 2 2 875660 7540 5256 58220 5536 1604 48756 1864 4505 4196 12 23 5 58 3 0 3 875264 3664 2120 57596 2304 116 17904 560 2223 1825 15 15 0 67 3 0 2 875564 3800 588 57856 1340 1068 14780 1184 1390 1364 12 10 0 77 3 1 2 875724 3740 372 53988 3104 928 16884 1068 1560 1527 3 12 0 83 3 0 3 881096 3708 532 52220 4604 5872 21004 6104 2752 2259 7 18 5 67 2 The following commit is included in that version: commit 710531320af876192d76b2c1f68190a1df941b02 Author: Michal Hocko <mhocko@suse.com> Date: Wed Feb 22 15:45:58 2017 -0800 mm, vmscan: cleanup lru size claculations commit fd538803731e50367b7c59ce4ad3454426a3d671 upstream. But still OOMs: [157048.030760] clamscan: page allocation stalls for 19405ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) [157048.031985] clamscan cpuset=/ mems_allowed=0 [157048.031993] CPU: 1 PID: 9597 Comm: clamscan Not tainted 4.11.0-0.rc2.git0.2.fc26.x86_64 #1 [157048.033197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 04/01/2014 [157048.034382] Call Trace: [157048.035532] dump_stack+0x63/0x84 [157048.036735] warn_alloc+0x10c/0x1b0 [157048.037768] __alloc_pages_slowpath+0x93d/0xe60 [157048.038873] ? dd_dispatch_request+0x2b/0x1a0 [157048.041033] ? get_page_from_freelist+0x122/0xbf0 [157048.042435] __alloc_pages_nodemask+0x290/0x2b0 [157048.043662] alloc_pages_vma+0xa0/0x2b0 [157048.044796] __read_swap_cache_async+0x146/0x210 [157048.045841] read_swap_cache_async+0x26/0x60 [157048.046858] swapin_readahead+0x186/0x230 [157048.047854] ? radix_tree_lookup_slot+0x22/0x50 [157048.049006] ? find_get_entry+0x20/0x140 [157048.053109] ? pagecache_get_page+0x2c/0x2e0 [157048.054179] do_swap_page+0x276/0x7b0 [157048.055138] __handle_mm_fault+0x6fd/0x1160 [157048.057571] ? pick_next_task_fair+0x48c/0x560 [157048.058608] handle_mm_fault+0xb3/0x250 [157048.059622] __do_page_fault+0x23f/0x4c0 [157048.068926] trace_do_page_fault+0x41/0x120 [157048.070143] do_async_page_fault+0x51/0xa0 [157048.071254] async_page_fault+0x28/0x30 [157048.072606] RIP: 0033:0x7f78659eb675 [157048.073858] RSP: 002b:00007ffcaba111b8 EFLAGS: 00010202 [157048.075192] RAX: 0000000000000941 RBX: 00007f785957e8d0 RCX: 00007f784e968b48 [157048.076609] RDX: 00007f784f87bce8 RSI: 00007f7851fdb0cb RDI: 00007f7866726000 [157048.077809] RBP: 00007f785957e910 R08: 0000000000040000 R09: 0000000000000000 [157048.078935] R10: ffffffffffffff48 R11: 0000000000000246 R12: 00007f78600c81c0 [157048.080028] R13: 00007f785957e970 R14: 00007f78594ffba8 R15: 0000000003406237 [157048.081827] Mem-Info: [157048.083005] active_anon:19902 inactive_anon:19920 isolated_anon:383 active_file:816 inactive_file:529 isolated_file:0 unevictable:0 dirty:0 writeback:19 unstable:0 slab_reclaimable:4225 slab_unreclaimable:6483 mapped:942 shmem:3 pagetables:3553 bounce:0 free:944 free_pcp:87 free_cma:0 [157048.089470] Node 0 active_anon:79552kB inactive_anon:79588kB active_file:3108kB inactive_file:2144kB unevictable:0kB isolated(anon):1624kB isolated(file):0kB mapped:3612kB dirty:0kB writeback:76kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12kB writeback_tmp:0kB unstable:0kB pages_scanned:247 all_unreclaimable? no [157048.092318] Node 0 DMA free:1408kB min:104kB low:128kB high:152kB active_anon:664kB inactive_anon:3124kB active_file:48kB inactive_file:40kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:564kB slab_unreclaimable:2148kB kernel_stack:92kB pagetables:1328kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [157048.096008] lowmem_reserve[]: 0 327 327 327 327 [157048.097234] Node 0 DMA32 free:2576kB min:2264kB low:2828kB high:3392kB active_anon:78844kB inactive_anon:76612kB active_file:2840kB inactive_file:1896kB unevictable:0kB writepending:76kB present:376688kB managed:353792kB mlocked:0kB slab_reclaimable:16336kB slab_unreclaimable:23784kB kernel_stack:2388kB pagetables:12884kB bounce:0kB free_pcp:644kB local_pcp:312kB free_cma:0kB [157048.101118] lowmem_reserve[]: 0 0 0 0 0 [157048.102190] Node 0 DMA: 37*4kB (UEH) 12*8kB (H) 13*16kB (H) 10*32kB (H) 4*64kB (H) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1412kB [157048.104989] Node 0 DMA32: 79*4kB (UMEH) 199*8kB (UMEH) 18*16kB (UMH) 5*32kB (H) 2*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2484kB [157048.107789] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [157048.107790] 2027 total pagecache pages [157048.109125] 710 pages in swap cache [157048.115088] Swap cache stats: add 36179491, delete 36179123, find 86964755/101977142 [157048.116934] Free swap = 808064kB [157048.118466] Total swap = 2064380kB [157048.122828] 98170 pages RAM [157048.124039] 0 pages HighMem/MovableOnly [157048.125051] 5745 pages reserved [157048.125997] 0 pages cma reserved [157048.127008] 0 pages hwpoisoned Thnx. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-16 6:38 ` Gerhard Wiesinger @ 2017-03-16 8:27 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-16 8:27 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu 16-03-17 07:38:08, Gerhard Wiesinger wrote: [...] > The following commit is included in that version: > commit 710531320af876192d76b2c1f68190a1df941b02 > Author: Michal Hocko <mhocko@suse.com> > Date: Wed Feb 22 15:45:58 2017 -0800 > > mm, vmscan: cleanup lru size claculations > > commit fd538803731e50367b7c59ce4ad3454426a3d671 upstream. This patch shouldn't make any difference. It is a cleanup patch. I guess you meant 71ab6cfe88dc ("mm, vmscan: consider eligible zones in get_scan_count") but even that one shouldn't make any difference for 64b systems. > But still OOMs: > [157048.030760] clamscan: page allocation stalls for 19405ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) This is not OOM it is an allocation stall. The allocation request cannot simply make forward progress for more than 10s. This alone is bad but considering this is GFP_HIGHUSER_MOVABLE which has the full reclaim capabilities I would suspect your workload overcommits the available memory too much. You only have ~380MB of RAM with ~160MB sitting in the anonymous memory, almost nothing in the page cache so I am not wondering that you see a constant swap activity. There seems to be only 40M in the slab so we are still missing ~180MB which is neither on the LRU lists nor allocated by slab. This means that some kernel subsystem allocates from the page allocator directly. That being said, I believe that what you are seeing is not a bug in the MM subsystem but rather some susbsytem using more memory than it used to before so your workload doesn't fit into the amount of memory you have anymore. [...] > [157048.081827] Mem-Info: > [157048.083005] active_anon:19902 inactive_anon:19920 isolated_anon:383 > active_file:816 inactive_file:529 isolated_file:0 > unevictable:0 dirty:0 writeback:19 unstable:0 > slab_reclaimable:4225 slab_unreclaimable:6483 > mapped:942 shmem:3 pagetables:3553 bounce:0 > free:944 free_pcp:87 free_cma:0 > [157048.089470] Node 0 active_anon:79552kB inactive_anon:79588kB > active_file:3108kB inactive_file:2144kB unevictable:0kB > isolated(anon):1624kB isolated(file):0kB mapped:3612kB dirty:0kB > writeback:76kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12kB > writeback_tmp:0kB unstable:0kB pages_scanned:247 all_unreclaimable? no > [157048.092318] Node 0 DMA free:1408kB min:104kB low:128kB high:152kB > active_anon:664kB inactive_anon:3124kB active_file:48kB inactive_file:40kB > unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB > slab_reclaimable:564kB slab_unreclaimable:2148kB kernel_stack:92kB > pagetables:1328kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [157048.096008] lowmem_reserve[]: 0 327 327 327 327 > [157048.097234] Node 0 DMA32 free:2576kB min:2264kB low:2828kB high:3392kB > active_anon:78844kB inactive_anon:76612kB active_file:2840kB > inactive_file:1896kB unevictable:0kB writepending:76kB present:376688kB > managed:353792kB mlocked:0kB slab_reclaimable:16336kB > slab_unreclaimable:23784kB kernel_stack:2388kB pagetables:12884kB bounce:0kB > free_pcp:644kB local_pcp:312kB free_cma:0kB > [157048.101118] lowmem_reserve[]: 0 0 0 0 0 > [157048.102190] Node 0 DMA: 37*4kB (UEH) 12*8kB (H) 13*16kB (H) 10*32kB (H) > 4*64kB (H) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1412kB > [157048.104989] Node 0 DMA32: 79*4kB (UMEH) 199*8kB (UMEH) 18*16kB (UMH) > 5*32kB (H) 2*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 2484kB > [157048.107789] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 > hugepages_size=2048kB > [157048.107790] 2027 total pagecache pages > [157048.109125] 710 pages in swap cache > [157048.115088] Swap cache stats: add 36179491, delete 36179123, find > 86964755/101977142 > [157048.116934] Free swap = 808064kB > [157048.118466] Total swap = 2064380kB > [157048.122828] 98170 pages RAM > [157048.124039] 0 pages HighMem/MovableOnly > [157048.125051] 5745 pages reserved > [157048.125997] 0 pages cma reserved > [157048.127008] 0 pages hwpoisoned > > > Thnx. > > Ciao, > Gerhard -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-16 8:27 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-16 8:27 UTC (permalink / raw) To: Gerhard Wiesinger; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu 16-03-17 07:38:08, Gerhard Wiesinger wrote: [...] > The following commit is included in that version: > commit 710531320af876192d76b2c1f68190a1df941b02 > Author: Michal Hocko <mhocko@suse.com> > Date: Wed Feb 22 15:45:58 2017 -0800 > > mm, vmscan: cleanup lru size claculations > > commit fd538803731e50367b7c59ce4ad3454426a3d671 upstream. This patch shouldn't make any difference. It is a cleanup patch. I guess you meant 71ab6cfe88dc ("mm, vmscan: consider eligible zones in get_scan_count") but even that one shouldn't make any difference for 64b systems. > But still OOMs: > [157048.030760] clamscan: page allocation stalls for 19405ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) This is not OOM it is an allocation stall. The allocation request cannot simply make forward progress for more than 10s. This alone is bad but considering this is GFP_HIGHUSER_MOVABLE which has the full reclaim capabilities I would suspect your workload overcommits the available memory too much. You only have ~380MB of RAM with ~160MB sitting in the anonymous memory, almost nothing in the page cache so I am not wondering that you see a constant swap activity. There seems to be only 40M in the slab so we are still missing ~180MB which is neither on the LRU lists nor allocated by slab. This means that some kernel subsystem allocates from the page allocator directly. That being said, I believe that what you are seeing is not a bug in the MM subsystem but rather some susbsytem using more memory than it used to before so your workload doesn't fit into the amount of memory you have anymore. [...] > [157048.081827] Mem-Info: > [157048.083005] active_anon:19902 inactive_anon:19920 isolated_anon:383 > active_file:816 inactive_file:529 isolated_file:0 > unevictable:0 dirty:0 writeback:19 unstable:0 > slab_reclaimable:4225 slab_unreclaimable:6483 > mapped:942 shmem:3 pagetables:3553 bounce:0 > free:944 free_pcp:87 free_cma:0 > [157048.089470] Node 0 active_anon:79552kB inactive_anon:79588kB > active_file:3108kB inactive_file:2144kB unevictable:0kB > isolated(anon):1624kB isolated(file):0kB mapped:3612kB dirty:0kB > writeback:76kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12kB > writeback_tmp:0kB unstable:0kB pages_scanned:247 all_unreclaimable? no > [157048.092318] Node 0 DMA free:1408kB min:104kB low:128kB high:152kB > active_anon:664kB inactive_anon:3124kB active_file:48kB inactive_file:40kB > unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB > slab_reclaimable:564kB slab_unreclaimable:2148kB kernel_stack:92kB > pagetables:1328kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [157048.096008] lowmem_reserve[]: 0 327 327 327 327 > [157048.097234] Node 0 DMA32 free:2576kB min:2264kB low:2828kB high:3392kB > active_anon:78844kB inactive_anon:76612kB active_file:2840kB > inactive_file:1896kB unevictable:0kB writepending:76kB present:376688kB > managed:353792kB mlocked:0kB slab_reclaimable:16336kB > slab_unreclaimable:23784kB kernel_stack:2388kB pagetables:12884kB bounce:0kB > free_pcp:644kB local_pcp:312kB free_cma:0kB > [157048.101118] lowmem_reserve[]: 0 0 0 0 0 > [157048.102190] Node 0 DMA: 37*4kB (UEH) 12*8kB (H) 13*16kB (H) 10*32kB (H) > 4*64kB (H) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1412kB > [157048.104989] Node 0 DMA32: 79*4kB (UMEH) 199*8kB (UMEH) 18*16kB (UMH) > 5*32kB (H) 2*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 2484kB > [157048.107789] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 > hugepages_size=2048kB > [157048.107790] 2027 total pagecache pages > [157048.109125] 710 pages in swap cache > [157048.115088] Swap cache stats: add 36179491, delete 36179123, find > 86964755/101977142 > [157048.116934] Free swap = 808064kB > [157048.118466] Total swap = 2064380kB > [157048.122828] 98170 pages RAM > [157048.124039] 0 pages HighMem/MovableOnly > [157048.125051] 5745 pages reserved > [157048.125997] 0 pages cma reserved > [157048.127008] 0 pages hwpoisoned > > > Thnx. > > Ciao, > Gerhard -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-16 8:27 ` Michal Hocko @ 2017-03-16 8:47 ` lkml -1 siblings, 0 replies; 88+ messages in thread From: lkml @ 2017-03-16 8:47 UTC (permalink / raw) To: Michal Hocko Cc: Gerhard Wiesinger, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu, Mar 16, 2017 at 09:27:14AM +0100, Michal Hocko wrote: > On Thu 16-03-17 07:38:08, Gerhard Wiesinger wrote: > [...] > > The following commit is included in that version: > > commit 710531320af876192d76b2c1f68190a1df941b02 > > Author: Michal Hocko <mhocko@suse.com> > > Date: Wed Feb 22 15:45:58 2017 -0800 > > > > mm, vmscan: cleanup lru size claculations > > > > commit fd538803731e50367b7c59ce4ad3454426a3d671 upstream. > > This patch shouldn't make any difference. It is a cleanup patch. > I guess you meant 71ab6cfe88dc ("mm, vmscan: consider eligible zones in > get_scan_count") but even that one shouldn't make any difference for 64b > systems. > > > But still OOMs: > > [157048.030760] clamscan: page allocation stalls for 19405ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) > > This is not OOM it is an allocation stall. The allocation request cannot > simply make forward progress for more than 10s. This alone is bad but > considering this is GFP_HIGHUSER_MOVABLE which has the full reclaim > capabilities I would suspect your workload overcommits the available > memory too much. You only have ~380MB of RAM with ~160MB sitting in the > anonymous memory, almost nothing in the page cache so I am not wondering > that you see a constant swap activity. There seems to be only 40M in the > slab so we are still missing ~180MB which is neither on the LRU lists > nor allocated by slab. This means that some kernel subsystem allocates > from the page allocator directly. > > That being said, I believe that what you are seeing is not a bug in the > MM subsystem but rather some susbsytem using more memory than it used to > before so your workload doesn't fit into the amount of memory you have > anymore. > While on the topic of understanding allocation stalls, Philip Freeman recently mailed linux-kernel with a similar report, and in his case there are plenty of page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. I'm no MM expert, but it appears a bit broken for such a low-order allocation to stall on the order of 10 seconds when there's plenty of reclaimable pages, in addition to mostly unused and abundant swap space on SSD. Regards, Vito Caputo ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-16 8:47 ` lkml 0 siblings, 0 replies; 88+ messages in thread From: lkml @ 2017-03-16 8:47 UTC (permalink / raw) To: Michal Hocko Cc: Gerhard Wiesinger, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu, Mar 16, 2017 at 09:27:14AM +0100, Michal Hocko wrote: > On Thu 16-03-17 07:38:08, Gerhard Wiesinger wrote: > [...] > > The following commit is included in that version: > > commit 710531320af876192d76b2c1f68190a1df941b02 > > Author: Michal Hocko <mhocko@suse.com> > > Date: Wed Feb 22 15:45:58 2017 -0800 > > > > mm, vmscan: cleanup lru size claculations > > > > commit fd538803731e50367b7c59ce4ad3454426a3d671 upstream. > > This patch shouldn't make any difference. It is a cleanup patch. > I guess you meant 71ab6cfe88dc ("mm, vmscan: consider eligible zones in > get_scan_count") but even that one shouldn't make any difference for 64b > systems. > > > But still OOMs: > > [157048.030760] clamscan: page allocation stalls for 19405ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) > > This is not OOM it is an allocation stall. The allocation request cannot > simply make forward progress for more than 10s. This alone is bad but > considering this is GFP_HIGHUSER_MOVABLE which has the full reclaim > capabilities I would suspect your workload overcommits the available > memory too much. You only have ~380MB of RAM with ~160MB sitting in the > anonymous memory, almost nothing in the page cache so I am not wondering > that you see a constant swap activity. There seems to be only 40M in the > slab so we are still missing ~180MB which is neither on the LRU lists > nor allocated by slab. This means that some kernel subsystem allocates > from the page allocator directly. > > That being said, I believe that what you are seeing is not a bug in the > MM subsystem but rather some susbsytem using more memory than it used to > before so your workload doesn't fit into the amount of memory you have > anymore. > While on the topic of understanding allocation stalls, Philip Freeman recently mailed linux-kernel with a similar report, and in his case there are plenty of page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. I'm no MM expert, but it appears a bit broken for such a low-order allocation to stall on the order of 10 seconds when there's plenty of reclaimable pages, in addition to mostly unused and abundant swap space on SSD. Regards, Vito Caputo -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-16 8:47 ` lkml @ 2017-03-16 9:08 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-16 9:08 UTC (permalink / raw) To: lkml Cc: Gerhard Wiesinger, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu 16-03-17 01:47:33, lkml@pengaru.com wrote: [...] > While on the topic of understanding allocation stalls, Philip Freeman recently > mailed linux-kernel with a similar report, and in his case there are plenty of > page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. care to point me to the report? > I'm no MM expert, but it appears a bit broken for such a low-order allocation > to stall on the order of 10 seconds when there's plenty of reclaimable pages, > in addition to mostly unused and abundant swap space on SSD. yes this might indeed signal a problem. -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-16 9:08 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-16 9:08 UTC (permalink / raw) To: lkml Cc: Gerhard Wiesinger, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu 16-03-17 01:47:33, lkml@pengaru.com wrote: [...] > While on the topic of understanding allocation stalls, Philip Freeman recently > mailed linux-kernel with a similar report, and in his case there are plenty of > page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. care to point me to the report? > I'm no MM expert, but it appears a bit broken for such a low-order allocation > to stall on the order of 10 seconds when there's plenty of reclaimable pages, > in addition to mostly unused and abundant swap space on SSD. yes this might indeed signal a problem. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-16 9:08 ` Michal Hocko @ 2017-03-16 9:23 ` lkml -1 siblings, 0 replies; 88+ messages in thread From: lkml @ 2017-03-16 9:23 UTC (permalink / raw) To: Michal Hocko Cc: lkml, Gerhard Wiesinger, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu, Mar 16, 2017 at 10:08:44AM +0100, Michal Hocko wrote: > On Thu 16-03-17 01:47:33, lkml@pengaru.com wrote: > [...] > > While on the topic of understanding allocation stalls, Philip Freeman recently > > mailed linux-kernel with a similar report, and in his case there are plenty of > > page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. > > care to point me to the report? http://lkml.iu.edu/hypermail/linux/kernel/1703.1/06360.html > > > I'm no MM expert, but it appears a bit broken for such a low-order allocation > > to stall on the order of 10 seconds when there's plenty of reclaimable pages, > > in addition to mostly unused and abundant swap space on SSD. > > yes this might indeed signal a problem. Well maybe I missed something obvious that a better informed eye will catch. Regards, Vito Caputo ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-16 9:23 ` lkml 0 siblings, 0 replies; 88+ messages in thread From: lkml @ 2017-03-16 9:23 UTC (permalink / raw) To: Michal Hocko Cc: lkml, Gerhard Wiesinger, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu, Mar 16, 2017 at 10:08:44AM +0100, Michal Hocko wrote: > On Thu 16-03-17 01:47:33, lkml@pengaru.com wrote: > [...] > > While on the topic of understanding allocation stalls, Philip Freeman recently > > mailed linux-kernel with a similar report, and in his case there are plenty of > > page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. > > care to point me to the report? http://lkml.iu.edu/hypermail/linux/kernel/1703.1/06360.html > > > I'm no MM expert, but it appears a bit broken for such a low-order allocation > > to stall on the order of 10 seconds when there's plenty of reclaimable pages, > > in addition to mostly unused and abundant swap space on SSD. > > yes this might indeed signal a problem. Well maybe I missed something obvious that a better informed eye will catch. Regards, Vito Caputo -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-16 9:23 ` lkml @ 2017-03-16 9:39 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-16 9:39 UTC (permalink / raw) To: lkml Cc: Gerhard Wiesinger, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu 16-03-17 02:23:18, lkml@pengaru.com wrote: > On Thu, Mar 16, 2017 at 10:08:44AM +0100, Michal Hocko wrote: > > On Thu 16-03-17 01:47:33, lkml@pengaru.com wrote: > > [...] > > > While on the topic of understanding allocation stalls, Philip Freeman recently > > > mailed linux-kernel with a similar report, and in his case there are plenty of > > > page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. > > > > care to point me to the report? > > http://lkml.iu.edu/hypermail/linux/kernel/1703.1/06360.html Thanks. It is gone from my lkml mailbox. Could you CC me (and linux-mm) please? > > > > > I'm no MM expert, but it appears a bit broken for such a low-order allocation > > > to stall on the order of 10 seconds when there's plenty of reclaimable pages, > > > in addition to mostly unused and abundant swap space on SSD. > > > > yes this might indeed signal a problem. > > Well maybe I missed something obvious that a better informed eye will catch. Nothing really obvious. There is indeed a lot of anonymous memory to swap out. Almost no pages on file LRU lists (active_file:759 inactive_file:749) but 158783 total pagecache pages so we have to have a lot of pages in the swap cache. I would probably have to see more data to make a full picture. -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-16 9:39 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-16 9:39 UTC (permalink / raw) To: lkml Cc: Gerhard Wiesinger, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu 16-03-17 02:23:18, lkml@pengaru.com wrote: > On Thu, Mar 16, 2017 at 10:08:44AM +0100, Michal Hocko wrote: > > On Thu 16-03-17 01:47:33, lkml@pengaru.com wrote: > > [...] > > > While on the topic of understanding allocation stalls, Philip Freeman recently > > > mailed linux-kernel with a similar report, and in his case there are plenty of > > > page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. > > > > care to point me to the report? > > http://lkml.iu.edu/hypermail/linux/kernel/1703.1/06360.html Thanks. It is gone from my lkml mailbox. Could you CC me (and linux-mm) please? > > > > > I'm no MM expert, but it appears a bit broken for such a low-order allocation > > > to stall on the order of 10 seconds when there's plenty of reclaimable pages, > > > in addition to mostly unused and abundant swap space on SSD. > > > > yes this might indeed signal a problem. > > Well maybe I missed something obvious that a better informed eye will catch. Nothing really obvious. There is indeed a lot of anonymous memory to swap out. Almost no pages on file LRU lists (active_file:759 inactive_file:749) but 158783 total pagecache pages so we have to have a lot of pages in the swap cache. I would probably have to see more data to make a full picture. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-16 9:39 ` Michal Hocko @ 2017-03-17 16:37 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-17 16:37 UTC (permalink / raw) To: Michal Hocko, lkml; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 16.03.2017 10:39, Michal Hocko wrote: > On Thu 16-03-17 02:23:18, lkml@pengaru.com wrote: >> On Thu, Mar 16, 2017 at 10:08:44AM +0100, Michal Hocko wrote: >>> On Thu 16-03-17 01:47:33, lkml@pengaru.com wrote: >>> [...] >>>> While on the topic of understanding allocation stalls, Philip Freeman recently >>>> mailed linux-kernel with a similar report, and in his case there are plenty of >>>> page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. >>> care to point me to the report? >> http://lkml.iu.edu/hypermail/linux/kernel/1703.1/06360.html > Thanks. It is gone from my lkml mailbox. Could you CC me (and linux-mm) please? > >>> >>>> I'm no MM expert, but it appears a bit broken for such a low-order allocation >>>> to stall on the order of 10 seconds when there's plenty of reclaimable pages, >>>> in addition to mostly unused and abundant swap space on SSD. >>> yes this might indeed signal a problem. >> Well maybe I missed something obvious that a better informed eye will catch. > Nothing really obvious. There is indeed a lot of anonymous memory to > swap out. Almost no pages on file LRU lists (active_file:759 > inactive_file:749) but 158783 total pagecache pages so we have to have a > lot of pages in the swap cache. I would probably have to see more data > to make a full picture. > Why does the kernel prefer to swapin/out and not use a.) the free memory? b.) the buffer/cache? There is ~100M memory available but kernel swaps all the time ... Any ideas? Kernel: 4.9.14-200.fc25.x86_64 top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie %Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 st KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 5 4 292852 52904 756 58584 19600 448 48780 540 8088 10528 18 61 1 7 13 3 3 288792 49052 1152 65924 4856 576 9824 1100 4324 5720 7 18 2 64 8 2 2 283676 54160 716 67604 6332 344 31740 964 3879 5055 12 34 10 37 7 3 3 286852 66712 216 53136 28064 4832 56532 4920 9175 12625 10 55 12 14 10 2 0 299680 62428 196 53316 36312 13164 54728 13212 16820 25283 7 56 18 12 7 1 1 300756 63220 624 58160 17944 1260 24528 1304 5804 9302 3 22 38 34 3 Thnx. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-17 16:37 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-17 16:37 UTC (permalink / raw) To: Michal Hocko, lkml; +Cc: Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 16.03.2017 10:39, Michal Hocko wrote: > On Thu 16-03-17 02:23:18, lkml@pengaru.com wrote: >> On Thu, Mar 16, 2017 at 10:08:44AM +0100, Michal Hocko wrote: >>> On Thu 16-03-17 01:47:33, lkml@pengaru.com wrote: >>> [...] >>>> While on the topic of understanding allocation stalls, Philip Freeman recently >>>> mailed linux-kernel with a similar report, and in his case there are plenty of >>>> page cache pages. It was also a GFP_HIGHUSER_MOVABLE 0-order allocation. >>> care to point me to the report? >> http://lkml.iu.edu/hypermail/linux/kernel/1703.1/06360.html > Thanks. It is gone from my lkml mailbox. Could you CC me (and linux-mm) please? > >>> >>>> I'm no MM expert, but it appears a bit broken for such a low-order allocation >>>> to stall on the order of 10 seconds when there's plenty of reclaimable pages, >>>> in addition to mostly unused and abundant swap space on SSD. >>> yes this might indeed signal a problem. >> Well maybe I missed something obvious that a better informed eye will catch. > Nothing really obvious. There is indeed a lot of anonymous memory to > swap out. Almost no pages on file LRU lists (active_file:759 > inactive_file:749) but 158783 total pagecache pages so we have to have a > lot of pages in the swap cache. I would probably have to see more data > to make a full picture. > Why does the kernel prefer to swapin/out and not use a.) the free memory? b.) the buffer/cache? There is ~100M memory available but kernel swaps all the time ... Any ideas? Kernel: 4.9.14-200.fc25.x86_64 top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie %Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 st KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 5 4 292852 52904 756 58584 19600 448 48780 540 8088 10528 18 61 1 7 13 3 3 288792 49052 1152 65924 4856 576 9824 1100 4324 5720 7 18 2 64 8 2 2 283676 54160 716 67604 6332 344 31740 964 3879 5055 12 34 10 37 7 3 3 286852 66712 216 53136 28064 4832 56532 4920 9175 12625 10 55 12 14 10 2 0 299680 62428 196 53316 36312 13164 54728 13212 16820 25283 7 56 18 12 7 1 1 300756 63220 624 58160 17944 1260 24528 1304 5804 9302 3 22 38 34 3 Thnx. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-17 16:37 ` Gerhard Wiesinger @ 2017-03-17 17:13 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-17 17:13 UTC (permalink / raw) To: Gerhard Wiesinger Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: [...] > Why does the kernel prefer to swapin/out and not use > > a.) the free memory? It will use all the free memory up to min watermark which is set up based on min_free_kbytes. > b.) the buffer/cache? the memory reclaim is strongly biased towards page cache and we try to avoid swapout as much as possible (see get_scan_count). > There is ~100M memory available but kernel swaps all the time ... > > Any ideas? > > Kernel: 4.9.14-200.fc25.x86_64 > > top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 > Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie > %Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 > st > KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache > > procs -----------memory---------- ---swap-- -----io---- -system-- > ------cpu----- > r b swpd free buff cache si so bi bo in cs us sy id wa st > 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 I am really surprised to see any reclaim at all. 26% of free memory doesn't sound as if we should do a reclaim at all. Do you have an unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there anything running inside a memory cgroup with a small limit? -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-17 17:13 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-17 17:13 UTC (permalink / raw) To: Gerhard Wiesinger Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: [...] > Why does the kernel prefer to swapin/out and not use > > a.) the free memory? It will use all the free memory up to min watermark which is set up based on min_free_kbytes. > b.) the buffer/cache? the memory reclaim is strongly biased towards page cache and we try to avoid swapout as much as possible (see get_scan_count). > There is ~100M memory available but kernel swaps all the time ... > > Any ideas? > > Kernel: 4.9.14-200.fc25.x86_64 > > top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 > Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie > %Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 > st > KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache > > procs -----------memory---------- ---swap-- -----io---- -system-- > ------cpu----- > r b swpd free buff cache si so bi bo in cs us sy id wa st > 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 I am really surprised to see any reclaim at all. 26% of free memory doesn't sound as if we should do a reclaim at all. Do you have an unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there anything running inside a memory cgroup with a small limit? -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-17 17:13 ` Michal Hocko @ 2017-03-17 20:08 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-17 20:08 UTC (permalink / raw) To: Michal Hocko; +Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 17.03.2017 18:13, Michal Hocko wrote: > On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: > [...] >> Why does the kernel prefer to swapin/out and not use >> >> a.) the free memory? > It will use all the free memory up to min watermark which is set up > based on min_free_kbytes. Makes sense, how is /proc/sys/vm/min_free_kbytes default value calculated? > >> b.) the buffer/cache? > the memory reclaim is strongly biased towards page cache and we try to > avoid swapout as much as possible (see get_scan_count). If I understand it correctly, swapping is preferred over dropping the cache, right. Can this behaviour be changed to prefer dropping the cache to some minimum amount? Is this also configurable in a way? (As far as I remember e.g. kernel 2.4 dropped the caches well). > >> There is ~100M memory available but kernel swaps all the time ... >> >> Any ideas? >> >> Kernel: 4.9.14-200.fc25.x86_64 >> >> top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 >> Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 >> st >> KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache >> >> procs -----------memory---------- ---swap-- -----io---- -system-- >> ------cpu----- >> r b swpd free buff cache si so bi bo in cs us sy id wa st >> 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 > I am really surprised to see any reclaim at all. 26% of free memory > doesn't sound as if we should do a reclaim at all. Do you have an > unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there > anything running inside a memory cgroup with a small limit? nothing special set regarding /proc/sys/vm/min_free_kbytes (default values), detailed config below. Regarding cgroups, none of I know. How to check (I guess nothing is set because cg* commands are not available)? cat /etc/sysctl.d/* | grep "^vm" vm.dirty_background_ratio = 3 vm.dirty_ratio = 15 vm.overcommit_memory = 2 vm.overcommit_ratio = 80 vm.swappiness=10 find /proc/sys/vm -type f -exec echo {} \; -exec cat {} \; /proc/sys/vm/admin_reserve_kbytes 8192 /proc/sys/vm/block_dump 0 /proc/sys/vm/compact_memory cat: /proc/sys/vm/compact_memory: Permission denied /proc/sys/vm/compact_unevictable_allowed 1 /proc/sys/vm/dirty_background_bytes 0 /proc/sys/vm/dirty_background_ratio 3 /proc/sys/vm/dirty_bytes 0 /proc/sys/vm/dirty_expire_centisecs 3000 /proc/sys/vm/dirty_ratio 15 /proc/sys/vm/dirty_writeback_centisecs 500 /proc/sys/vm/dirtytime_expire_seconds 43200 /proc/sys/vm/drop_caches 0 /proc/sys/vm/extfrag_threshold 500 /proc/sys/vm/hugepages_treat_as_movable 0 /proc/sys/vm/hugetlb_shm_group 0 /proc/sys/vm/laptop_mode 0 /proc/sys/vm/legacy_va_layout 0 /proc/sys/vm/lowmem_reserve_ratio 256 256 32 1 /proc/sys/vm/max_map_count 65530 /proc/sys/vm/memory_failure_early_kill 0 /proc/sys/vm/memory_failure_recovery 1 /proc/sys/vm/min_free_kbytes 45056 /proc/sys/vm/min_slab_ratio 5 /proc/sys/vm/min_unmapped_ratio 1 /proc/sys/vm/mmap_min_addr 65536 /proc/sys/vm/mmap_rnd_bits 28 /proc/sys/vm/mmap_rnd_compat_bits 8 /proc/sys/vm/nr_hugepages 0 /proc/sys/vm/nr_hugepages_mempolicy 0 /proc/sys/vm/nr_overcommit_hugepages 0 /proc/sys/vm/nr_pdflush_threads 0 /proc/sys/vm/numa_zonelist_order default /proc/sys/vm/oom_dump_tasks 1 /proc/sys/vm/oom_kill_allocating_task 0 /proc/sys/vm/overcommit_kbytes 0 /proc/sys/vm/overcommit_memory 2 /proc/sys/vm/overcommit_ratio 80 /proc/sys/vm/page-cluster 3 /proc/sys/vm/panic_on_oom 0 /proc/sys/vm/percpu_pagelist_fraction 0 /proc/sys/vm/stat_interval 1 /proc/sys/vm/stat_refresh /proc/sys/vm/swappiness 10 /proc/sys/vm/user_reserve_kbytes 31036 /proc/sys/vm/vfs_cache_pressure 100 /proc/sys/vm/watermark_scale_factor 10 /proc/sys/vm/zone_reclaim_mode 0 Thnx. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-17 20:08 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-17 20:08 UTC (permalink / raw) To: Michal Hocko; +Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 17.03.2017 18:13, Michal Hocko wrote: > On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: > [...] >> Why does the kernel prefer to swapin/out and not use >> >> a.) the free memory? > It will use all the free memory up to min watermark which is set up > based on min_free_kbytes. Makes sense, how is /proc/sys/vm/min_free_kbytes default value calculated? > >> b.) the buffer/cache? > the memory reclaim is strongly biased towards page cache and we try to > avoid swapout as much as possible (see get_scan_count). If I understand it correctly, swapping is preferred over dropping the cache, right. Can this behaviour be changed to prefer dropping the cache to some minimum amount? Is this also configurable in a way? (As far as I remember e.g. kernel 2.4 dropped the caches well). > >> There is ~100M memory available but kernel swaps all the time ... >> >> Any ideas? >> >> Kernel: 4.9.14-200.fc25.x86_64 >> >> top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 >> Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 >> st >> KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache >> >> procs -----------memory---------- ---swap-- -----io---- -system-- >> ------cpu----- >> r b swpd free buff cache si so bi bo in cs us sy id wa st >> 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 > I am really surprised to see any reclaim at all. 26% of free memory > doesn't sound as if we should do a reclaim at all. Do you have an > unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there > anything running inside a memory cgroup with a small limit? nothing special set regarding /proc/sys/vm/min_free_kbytes (default values), detailed config below. Regarding cgroups, none of I know. How to check (I guess nothing is set because cg* commands are not available)? cat /etc/sysctl.d/* | grep "^vm" vm.dirty_background_ratio = 3 vm.dirty_ratio = 15 vm.overcommit_memory = 2 vm.overcommit_ratio = 80 vm.swappiness=10 find /proc/sys/vm -type f -exec echo {} \; -exec cat {} \; /proc/sys/vm/admin_reserve_kbytes 8192 /proc/sys/vm/block_dump 0 /proc/sys/vm/compact_memory cat: /proc/sys/vm/compact_memory: Permission denied /proc/sys/vm/compact_unevictable_allowed 1 /proc/sys/vm/dirty_background_bytes 0 /proc/sys/vm/dirty_background_ratio 3 /proc/sys/vm/dirty_bytes 0 /proc/sys/vm/dirty_expire_centisecs 3000 /proc/sys/vm/dirty_ratio 15 /proc/sys/vm/dirty_writeback_centisecs 500 /proc/sys/vm/dirtytime_expire_seconds 43200 /proc/sys/vm/drop_caches 0 /proc/sys/vm/extfrag_threshold 500 /proc/sys/vm/hugepages_treat_as_movable 0 /proc/sys/vm/hugetlb_shm_group 0 /proc/sys/vm/laptop_mode 0 /proc/sys/vm/legacy_va_layout 0 /proc/sys/vm/lowmem_reserve_ratio 256 256 32 1 /proc/sys/vm/max_map_count 65530 /proc/sys/vm/memory_failure_early_kill 0 /proc/sys/vm/memory_failure_recovery 1 /proc/sys/vm/min_free_kbytes 45056 /proc/sys/vm/min_slab_ratio 5 /proc/sys/vm/min_unmapped_ratio 1 /proc/sys/vm/mmap_min_addr 65536 /proc/sys/vm/mmap_rnd_bits 28 /proc/sys/vm/mmap_rnd_compat_bits 8 /proc/sys/vm/nr_hugepages 0 /proc/sys/vm/nr_hugepages_mempolicy 0 /proc/sys/vm/nr_overcommit_hugepages 0 /proc/sys/vm/nr_pdflush_threads 0 /proc/sys/vm/numa_zonelist_order default /proc/sys/vm/oom_dump_tasks 1 /proc/sys/vm/oom_kill_allocating_task 0 /proc/sys/vm/overcommit_kbytes 0 /proc/sys/vm/overcommit_memory 2 /proc/sys/vm/overcommit_ratio 80 /proc/sys/vm/page-cluster 3 /proc/sys/vm/panic_on_oom 0 /proc/sys/vm/percpu_pagelist_fraction 0 /proc/sys/vm/stat_interval 1 /proc/sys/vm/stat_refresh /proc/sys/vm/swappiness 10 /proc/sys/vm/user_reserve_kbytes 31036 /proc/sys/vm/vfs_cache_pressure 100 /proc/sys/vm/watermark_scale_factor 10 /proc/sys/vm/zone_reclaim_mode 0 Thnx. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-17 20:08 ` Gerhard Wiesinger @ 2017-03-19 8:17 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-19 8:17 UTC (permalink / raw) To: Michal Hocko; +Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 17.03.2017 21:08, Gerhard Wiesinger wrote: > On 17.03.2017 18:13, Michal Hocko wrote: >> On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: >> [...] 4.11.0-0.rc2.git4.1.fc27.x86_64 There are also lockups after some runtime hours to 1 day: Message from syslogd@myserver Mar 19 08:22:33 ... kernel:BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 18717s! Message from syslogd@myserver at Mar 19 08:22:33 ... kernel:BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 18078s! repeated a lot of times .... Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-19 8:17 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-19 8:17 UTC (permalink / raw) To: Michal Hocko; +Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 17.03.2017 21:08, Gerhard Wiesinger wrote: > On 17.03.2017 18:13, Michal Hocko wrote: >> On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: >> [...] 4.11.0-0.rc2.git4.1.fc27.x86_64 There are also lockups after some runtime hours to 1 day: Message from syslogd@myserver Mar 19 08:22:33 ... kernel:BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 18717s! Message from syslogd@myserver at Mar 19 08:22:33 ... kernel:BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 18078s! repeated a lot of times .... Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-19 8:17 ` Gerhard Wiesinger @ 2017-03-20 1:54 ` Tetsuo Handa -1 siblings, 0 replies; 88+ messages in thread From: Tetsuo Handa @ 2017-03-20 1:54 UTC (permalink / raw) To: Gerhard Wiesinger Cc: Michal Hocko, lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 2017/03/19 17:17, Gerhard Wiesinger wrote: > On 17.03.2017 21:08, Gerhard Wiesinger wrote: >> On 17.03.2017 18:13, Michal Hocko wrote: >>> On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: >>> [...] > > 4.11.0-0.rc2.git4.1.fc27.x86_64 > > There are also lockups after some runtime hours to 1 day: > Message from syslogd@myserver Mar 19 08:22:33 ... > kernel:BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 18717s! > > Message from syslogd@myserver at Mar 19 08:22:33 ... > kernel:BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 18078s! > > repeated a lot of times .... > > Ciao, > Gerhard "kernel:BUG: workqueue lockup" lines alone do not help. It does not tell what work is stalling. Maybe stalling due to constant swapping while doing memory allocation when processing some work, but relevant lines are needed in order to know what is happening. You can try SysRq-t to dump what workqueue threads are doing when you encounter such lines. You might want to try kmallocwd at http://lkml.kernel.org/r/1489578541-81526-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp . ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-20 1:54 ` Tetsuo Handa 0 siblings, 0 replies; 88+ messages in thread From: Tetsuo Handa @ 2017-03-20 1:54 UTC (permalink / raw) To: Gerhard Wiesinger Cc: Michal Hocko, lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 2017/03/19 17:17, Gerhard Wiesinger wrote: > On 17.03.2017 21:08, Gerhard Wiesinger wrote: >> On 17.03.2017 18:13, Michal Hocko wrote: >>> On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: >>> [...] > > 4.11.0-0.rc2.git4.1.fc27.x86_64 > > There are also lockups after some runtime hours to 1 day: > Message from syslogd@myserver Mar 19 08:22:33 ... > kernel:BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 18717s! > > Message from syslogd@myserver at Mar 19 08:22:33 ... > kernel:BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 18078s! > > repeated a lot of times .... > > Ciao, > Gerhard "kernel:BUG: workqueue lockup" lines alone do not help. It does not tell what work is stalling. Maybe stalling due to constant swapping while doing memory allocation when processing some work, but relevant lines are needed in order to know what is happening. You can try SysRq-t to dump what workqueue threads are doing when you encounter such lines. You might want to try kmallocwd at http://lkml.kernel.org/r/1489578541-81526-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp . -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-17 20:08 ` Gerhard Wiesinger @ 2017-03-19 15:18 ` Michal Hocko -1 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-19 15:18 UTC (permalink / raw) To: Gerhard Wiesinger Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Fri 17-03-17 21:08:31, Gerhard Wiesinger wrote: > On 17.03.2017 18:13, Michal Hocko wrote: > >On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: > >[...] > >>Why does the kernel prefer to swapin/out and not use > >> > >>a.) the free memory? > >It will use all the free memory up to min watermark which is set up > >based on min_free_kbytes. > > Makes sense, how is /proc/sys/vm/min_free_kbytes default value calculated? See init_per_zone_wmark_min > >>b.) the buffer/cache? > >the memory reclaim is strongly biased towards page cache and we try to > >avoid swapout as much as possible (see get_scan_count). > > If I understand it correctly, swapping is preferred over dropping the > cache, right. Can this behaviour be changed to prefer dropping the > cache to some minimum amount? Is this also configurable in a way? No, we enforce swapping if the amount of free + file pages are below the cumulative high watermark. > (As far as I remember e.g. kernel 2.4 dropped the caches well). > > >>There is ~100M memory available but kernel swaps all the time ... > >> > >>Any ideas? > >> > >>Kernel: 4.9.14-200.fc25.x86_64 > >> > >>top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 > >>Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie > >>%Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 > >>st > >>KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache > >> > >>procs -----------memory---------- ---swap-- -----io---- -system-- > >>------cpu----- > >> r b swpd free buff cache si so bi bo in cs us sy id wa st > >> 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 > >I am really surprised to see any reclaim at all. 26% of free memory > >doesn't sound as if we should do a reclaim at all. Do you have an > >unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there > >anything running inside a memory cgroup with a small limit? > > nothing special set regarding /proc/sys/vm/min_free_kbytes (default values), > detailed config below. Regarding cgroups, none of I know. How to check (I > guess nothing is set because cg* commands are not available)? be careful because systemd started to use some controllers. You can easily check cgroup mount points. > /proc/sys/vm/min_free_kbytes > 45056 So at least 45M will be kept reserved for the system. Your data indicated you had more memory. How does /proc/zoneinfo look like? Btw. you seem to be using fc kernel, are there any patches applied on top of Linus tree? Could you try to retest vanilla kernel? -- Michal Hocko SUSE Labs ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-19 15:18 ` Michal Hocko 0 siblings, 0 replies; 88+ messages in thread From: Michal Hocko @ 2017-03-19 15:18 UTC (permalink / raw) To: Gerhard Wiesinger Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Fri 17-03-17 21:08:31, Gerhard Wiesinger wrote: > On 17.03.2017 18:13, Michal Hocko wrote: > >On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: > >[...] > >>Why does the kernel prefer to swapin/out and not use > >> > >>a.) the free memory? > >It will use all the free memory up to min watermark which is set up > >based on min_free_kbytes. > > Makes sense, how is /proc/sys/vm/min_free_kbytes default value calculated? See init_per_zone_wmark_min > >>b.) the buffer/cache? > >the memory reclaim is strongly biased towards page cache and we try to > >avoid swapout as much as possible (see get_scan_count). > > If I understand it correctly, swapping is preferred over dropping the > cache, right. Can this behaviour be changed to prefer dropping the > cache to some minimum amount? Is this also configurable in a way? No, we enforce swapping if the amount of free + file pages are below the cumulative high watermark. > (As far as I remember e.g. kernel 2.4 dropped the caches well). > > >>There is ~100M memory available but kernel swaps all the time ... > >> > >>Any ideas? > >> > >>Kernel: 4.9.14-200.fc25.x86_64 > >> > >>top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 > >>Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie > >>%Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 > >>st > >>KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache > >> > >>procs -----------memory---------- ---swap-- -----io---- -system-- > >>------cpu----- > >> r b swpd free buff cache si so bi bo in cs us sy id wa st > >> 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 > >I am really surprised to see any reclaim at all. 26% of free memory > >doesn't sound as if we should do a reclaim at all. Do you have an > >unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there > >anything running inside a memory cgroup with a small limit? > > nothing special set regarding /proc/sys/vm/min_free_kbytes (default values), > detailed config below. Regarding cgroups, none of I know. How to check (I > guess nothing is set because cg* commands are not available)? be careful because systemd started to use some controllers. You can easily check cgroup mount points. > /proc/sys/vm/min_free_kbytes > 45056 So at least 45M will be kept reserved for the system. Your data indicated you had more memory. How does /proc/zoneinfo look like? Btw. you seem to be using fc kernel, are there any patches applied on top of Linus tree? Could you try to retest vanilla kernel? -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-19 15:18 ` Michal Hocko @ 2017-03-19 16:02 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-19 16:02 UTC (permalink / raw) To: Michal Hocko; +Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 19.03.2017 16:18, Michal Hocko wrote: > On Fri 17-03-17 21:08:31, Gerhard Wiesinger wrote: >> On 17.03.2017 18:13, Michal Hocko wrote: >>> On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: >>> [...] >>>> Why does the kernel prefer to swapin/out and not use >>>> >>>> a.) the free memory? >>> It will use all the free memory up to min watermark which is set up >>> based on min_free_kbytes. >> Makes sense, how is /proc/sys/vm/min_free_kbytes default value calculated? > See init_per_zone_wmark_min > >>>> b.) the buffer/cache? >>> the memory reclaim is strongly biased towards page cache and we try to >>> avoid swapout as much as possible (see get_scan_count). >> If I understand it correctly, swapping is preferred over dropping the >> cache, right. Can this behaviour be changed to prefer dropping the >> cache to some minimum amount? Is this also configurable in a way? > No, we enforce swapping if the amount of free + file pages are below the > cumulative high watermark. > >> (As far as I remember e.g. kernel 2.4 dropped the caches well). >> >>>> There is ~100M memory available but kernel swaps all the time ... >>>> >>>> Any ideas? >>>> >>>> Kernel: 4.9.14-200.fc25.x86_64 >>>> >>>> top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 >>>> Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie >>>> %Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 >>>> st >>>> KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache >>>> >>>> procs -----------memory---------- ---swap-- -----io---- -system-- >>>> ------cpu----- >>>> r b swpd free buff cache si so bi bo in cs us sy id wa st >>>> 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 >>> I am really surprised to see any reclaim at all. 26% of free memory >>> doesn't sound as if we should do a reclaim at all. Do you have an >>> unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there >>> anything running inside a memory cgroup with a small limit? >> nothing special set regarding /proc/sys/vm/min_free_kbytes (default values), >> detailed config below. Regarding cgroups, none of I know. How to check (I >> guess nothing is set because cg* commands are not available)? > be careful because systemd started to use some controllers. You can > easily check cgroup mount points. See below. > >> /proc/sys/vm/min_free_kbytes >> 45056 > So at least 45M will be kept reserved for the system. Your data > indicated you had more memory. How does /proc/zoneinfo look like? > Btw. you seem to be using fc kernel, are there any patches applied on > top of Linus tree? Could you try to retest vanilla kernel? System looks normally now, FYI (e.g. now permanent swapping) free total used free shared buff/cache available Mem: 349076 154112 41560 184 153404 148716 Swap: 2064380 831844 1232536 cat /proc/zoneinfo Node 0, zone DMA per-node stats nr_inactive_anon 9543 nr_active_anon 22105 nr_inactive_file 9877 nr_active_file 13416 nr_unevictable 0 nr_isolated_anon 0 nr_isolated_file 0 nr_pages_scanned 0 workingset_refault 1926013 workingset_activate 707166 workingset_nodereclaim 187276 nr_anon_pages 11429 nr_mapped 6852 nr_file_pages 46772 nr_dirty 1 nr_writeback 0 nr_writeback_temp 0 nr_shmem 46 nr_shmem_hugepages 0 nr_shmem_pmdmapped 0 nr_anon_transparent_hugepages 0 nr_unstable 0 nr_vmscan_write 3319047 nr_vmscan_immediate_reclaim 32363 nr_dirtied 222115 nr_written 3537529 pages free 3110 min 27 low 33 high 39 node_scanned 0 spanned 4095 present 3998 managed 3977 nr_free_pages 3110 nr_zone_inactive_anon 18 nr_zone_active_anon 3 nr_zone_inactive_file 51 nr_zone_active_file 75 nr_zone_unevictable 0 nr_zone_write_pending 0 nr_mlock 0 nr_slab_reclaimable 214 nr_slab_unreclaimable 289 nr_page_table_pages 185 nr_kernel_stack 16 nr_bounce 0 nr_zspages 0 numa_hit 1214071 numa_miss 0 numa_foreign 0 numa_interleave 0 numa_local 1214071 numa_other 0 nr_free_cma 0 protection: (0, 306, 306, 306, 306) pagesets cpu: 0 count: 0 high: 0 batch: 1 vm stats threshold: 4 cpu: 1 count: 0 high: 0 batch: 1 vm stats threshold: 4 node_unreclaimable: 0 start_pfn: 1 node_inactive_ratio: 0 Node 0, zone DMA32 pages free 7921 min 546 low 682 high 818 node_scanned 0 spanned 94172 present 94172 managed 83292 nr_free_pages 7921 nr_zone_inactive_anon 9525 nr_zone_active_anon 22102 nr_zone_inactive_file 9826 nr_zone_active_file 13341 nr_zone_unevictable 0 nr_zone_write_pending 1 nr_mlock 0 nr_slab_reclaimable 5829 nr_slab_unreclaimable 8622 nr_page_table_pages 2638 nr_kernel_stack 2208 nr_bounce 0 nr_zspages 0 numa_hit 23125334 numa_miss 0 numa_foreign 0 numa_interleave 14307 numa_local 23125334 numa_other 0 nr_free_cma 0 protection: (0, 0, 0, 0, 0) pagesets cpu: 0 count: 17 high: 90 batch: 15 vm stats threshold: 12 cpu: 1 count: 55 high: 90 batch: 15 vm stats threshold: 12 node_unreclaimable: 0 start_pfn: 4096 node_inactive_ratio: 0 mount | grep cgroup tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) There are patches (see below), but as far as I saw nothing regarding the issues which happen. BTW: Does it make sense to reduce lower limit for low mem VMs? e.g. echo "10000" > /proc/sys/vm/min_free_kbytes Thnx. Ciao, Gerhard https://koji.fedoraproject.org/koji/buildinfo?buildID=870215 ## Patches needed for building this package # build tweak for build ID magic, even for -vanilla Patch001: kbuild-AFTER_LINK.patch ## compile fixes # ongoing complaint, full discussion delayed until ksummit/plumbers Patch002: 0001-iio-Use-event-header-from-kernel-tree.patch %if !%{nopatches} # Git trees. # Standalone patches # a tempory patch for QCOM hardware enablement. Will be gone by end of 2016/F-26 GA Patch420: qcom-QDF2432-tmp-errata.patch # http://www.spinics.net/lists/arm-kernel/msg490981.html Patch421: geekbox-v4-device-tree-support.patch # http://www.spinics.net/lists/linux-tegra/msg26029.html Patch422: usb-phy-tegra-Add-38.4MHz-clock-table-entry.patch # Fix OMAP4 (pandaboard) Patch423: arm-revert-mmc-omap_hsmmc-Use-dma_request_chan-for-reque.patch # Not particularly happy we don't yet have a proper upstream resolution this is the right direction # https://www.spinics.net/lists/arm-kernel/msg535191.html Patch424: arm64-mm-Fix-memmap-to-be-initialized-for-the-entire-section.patch # http://patchwork.ozlabs.org/patch/587554/ Patch425: ARM-tegra-usb-no-reset.patch Patch426: AllWinner-net-emac.patch # http://www.spinics.net/lists/devicetree/msg163238.html Patch430: bcm2837-initial-support.patch # http://www.spinics.net/lists/dri-devel/msg132235.html Patch433: drm-vc4-Fix-OOPSes-from-trying-to-cache-a-partially-constructed-BO..patch # bcm283x mmc for wifi http://www.spinics.net/lists/arm-kernel/msg567077.html Patch434: bcm283x-mmc-bcm2835.patch # Upstream fixes for i2c/serial/ethernet MAC addresses Patch435: bcm283x-fixes.patch # https://lists.freedesktop.org/archives/dri-devel/2017-February/133823.html Patch436: vc4-fix-vblank-cursor-update-issue.patch # http://www.spinics.net/lists/arm-kernel/msg552554.html Patch438: arm-imx6-hummingboard2.patch Patch460: lib-cpumask-Make-CPUMASK_OFFSTACK-usable-without-deb.patch Patch466: input-kill-stupid-messages.patch Patch467: die-floppy-die.patch Patch468: no-pcspkr-modalias.patch Patch470: silence-fbcon-logo.patch Patch471: Kbuild-Add-an-option-to-enable-GCC-VTA.patch Patch472: crash-driver.patch Patch473: efi-lockdown.patch Patch487: Add-EFI-signature-data-types.patch Patch488: Add-an-EFI-signature-blob-parser-and-key-loader.patch # This doesn't apply. It seems like it could be replaced by # https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=5ac7eace2d00eab5ae0e9fdee63e38aee6001f7c # which has an explicit line about blacklisting Patch489: KEYS-Add-a-system-blacklist-keyring.patch Patch490: MODSIGN-Import-certificates-from-UEFI-Secure-Boot.patch Patch491: MODSIGN-Support-not-importing-certs-from-db.patch Patch493: drm-i915-hush-check-crtc-state.patch Patch494: disable-i8042-check-on-apple-mac.patch Patch495: lis3-improve-handling-of-null-rate.patch Patch497: scsi-sd_revalidate_disk-prevent-NULL-ptr-deref.patch Patch498: criu-no-expert.patch Patch499: ath9k-rx-dma-stop-check.patch Patch500: xen-pciback-Don-t-disable-PCI_COMMAND-on-PCI-device-.patch Patch501: Input-synaptics-pin-3-touches-when-the-firmware-repo.patch Patch502: firmware-Drop-WARN-from-usermodehelper_read_trylock-.patch # Patch503: drm-i915-turn-off-wc-mmaps.patch Patch509: MODSIGN-Don-t-try-secure-boot-if-EFI-runtime-is-disa.patch #CVE-2016-3134 rhbz 1317383 1317384 Patch665: netfilter-x_tables-deal-with-bogus-nextoffset-values.patch # grabbed from mailing list Patch667: v3-Revert-tty-serial-pl011-add-ttyAMA-for-matching-pl011-console.patch # END OF PATCH DEFINITIONS ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-19 16:02 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-19 16:02 UTC (permalink / raw) To: Michal Hocko; +Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 19.03.2017 16:18, Michal Hocko wrote: > On Fri 17-03-17 21:08:31, Gerhard Wiesinger wrote: >> On 17.03.2017 18:13, Michal Hocko wrote: >>> On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: >>> [...] >>>> Why does the kernel prefer to swapin/out and not use >>>> >>>> a.) the free memory? >>> It will use all the free memory up to min watermark which is set up >>> based on min_free_kbytes. >> Makes sense, how is /proc/sys/vm/min_free_kbytes default value calculated? > See init_per_zone_wmark_min > >>>> b.) the buffer/cache? >>> the memory reclaim is strongly biased towards page cache and we try to >>> avoid swapout as much as possible (see get_scan_count). >> If I understand it correctly, swapping is preferred over dropping the >> cache, right. Can this behaviour be changed to prefer dropping the >> cache to some minimum amount? Is this also configurable in a way? > No, we enforce swapping if the amount of free + file pages are below the > cumulative high watermark. > >> (As far as I remember e.g. kernel 2.4 dropped the caches well). >> >>>> There is ~100M memory available but kernel swaps all the time ... >>>> >>>> Any ideas? >>>> >>>> Kernel: 4.9.14-200.fc25.x86_64 >>>> >>>> top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 >>>> Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie >>>> %Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 >>>> st >>>> KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache >>>> >>>> procs -----------memory---------- ---swap-- -----io---- -system-- >>>> ------cpu----- >>>> r b swpd free buff cache si so bi bo in cs us sy id wa st >>>> 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 >>> I am really surprised to see any reclaim at all. 26% of free memory >>> doesn't sound as if we should do a reclaim at all. Do you have an >>> unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there >>> anything running inside a memory cgroup with a small limit? >> nothing special set regarding /proc/sys/vm/min_free_kbytes (default values), >> detailed config below. Regarding cgroups, none of I know. How to check (I >> guess nothing is set because cg* commands are not available)? > be careful because systemd started to use some controllers. You can > easily check cgroup mount points. See below. > >> /proc/sys/vm/min_free_kbytes >> 45056 > So at least 45M will be kept reserved for the system. Your data > indicated you had more memory. How does /proc/zoneinfo look like? > Btw. you seem to be using fc kernel, are there any patches applied on > top of Linus tree? Could you try to retest vanilla kernel? System looks normally now, FYI (e.g. now permanent swapping) free total used free shared buff/cache available Mem: 349076 154112 41560 184 153404 148716 Swap: 2064380 831844 1232536 cat /proc/zoneinfo Node 0, zone DMA per-node stats nr_inactive_anon 9543 nr_active_anon 22105 nr_inactive_file 9877 nr_active_file 13416 nr_unevictable 0 nr_isolated_anon 0 nr_isolated_file 0 nr_pages_scanned 0 workingset_refault 1926013 workingset_activate 707166 workingset_nodereclaim 187276 nr_anon_pages 11429 nr_mapped 6852 nr_file_pages 46772 nr_dirty 1 nr_writeback 0 nr_writeback_temp 0 nr_shmem 46 nr_shmem_hugepages 0 nr_shmem_pmdmapped 0 nr_anon_transparent_hugepages 0 nr_unstable 0 nr_vmscan_write 3319047 nr_vmscan_immediate_reclaim 32363 nr_dirtied 222115 nr_written 3537529 pages free 3110 min 27 low 33 high 39 node_scanned 0 spanned 4095 present 3998 managed 3977 nr_free_pages 3110 nr_zone_inactive_anon 18 nr_zone_active_anon 3 nr_zone_inactive_file 51 nr_zone_active_file 75 nr_zone_unevictable 0 nr_zone_write_pending 0 nr_mlock 0 nr_slab_reclaimable 214 nr_slab_unreclaimable 289 nr_page_table_pages 185 nr_kernel_stack 16 nr_bounce 0 nr_zspages 0 numa_hit 1214071 numa_miss 0 numa_foreign 0 numa_interleave 0 numa_local 1214071 numa_other 0 nr_free_cma 0 protection: (0, 306, 306, 306, 306) pagesets cpu: 0 count: 0 high: 0 batch: 1 vm stats threshold: 4 cpu: 1 count: 0 high: 0 batch: 1 vm stats threshold: 4 node_unreclaimable: 0 start_pfn: 1 node_inactive_ratio: 0 Node 0, zone DMA32 pages free 7921 min 546 low 682 high 818 node_scanned 0 spanned 94172 present 94172 managed 83292 nr_free_pages 7921 nr_zone_inactive_anon 9525 nr_zone_active_anon 22102 nr_zone_inactive_file 9826 nr_zone_active_file 13341 nr_zone_unevictable 0 nr_zone_write_pending 1 nr_mlock 0 nr_slab_reclaimable 5829 nr_slab_unreclaimable 8622 nr_page_table_pages 2638 nr_kernel_stack 2208 nr_bounce 0 nr_zspages 0 numa_hit 23125334 numa_miss 0 numa_foreign 0 numa_interleave 14307 numa_local 23125334 numa_other 0 nr_free_cma 0 protection: (0, 0, 0, 0, 0) pagesets cpu: 0 count: 17 high: 90 batch: 15 vm stats threshold: 12 cpu: 1 count: 55 high: 90 batch: 15 vm stats threshold: 12 node_unreclaimable: 0 start_pfn: 4096 node_inactive_ratio: 0 mount | grep cgroup tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) There are patches (see below), but as far as I saw nothing regarding the issues which happen. BTW: Does it make sense to reduce lower limit for low mem VMs? e.g. echo "10000" > /proc/sys/vm/min_free_kbytes Thnx. Ciao, Gerhard https://koji.fedoraproject.org/koji/buildinfo?buildID=870215 ## Patches needed for building this package # build tweak for build ID magic, even for -vanilla Patch001: kbuild-AFTER_LINK.patch ## compile fixes # ongoing complaint, full discussion delayed until ksummit/plumbers Patch002: 0001-iio-Use-event-header-from-kernel-tree.patch %if !%{nopatches} # Git trees. # Standalone patches # a tempory patch for QCOM hardware enablement. Will be gone by end of 2016/F-26 GA Patch420: qcom-QDF2432-tmp-errata.patch # http://www.spinics.net/lists/arm-kernel/msg490981.html Patch421: geekbox-v4-device-tree-support.patch # http://www.spinics.net/lists/linux-tegra/msg26029.html Patch422: usb-phy-tegra-Add-38.4MHz-clock-table-entry.patch # Fix OMAP4 (pandaboard) Patch423: arm-revert-mmc-omap_hsmmc-Use-dma_request_chan-for-reque.patch # Not particularly happy we don't yet have a proper upstream resolution this is the right direction # https://www.spinics.net/lists/arm-kernel/msg535191.html Patch424: arm64-mm-Fix-memmap-to-be-initialized-for-the-entire-section.patch # http://patchwork.ozlabs.org/patch/587554/ Patch425: ARM-tegra-usb-no-reset.patch Patch426: AllWinner-net-emac.patch # http://www.spinics.net/lists/devicetree/msg163238.html Patch430: bcm2837-initial-support.patch # http://www.spinics.net/lists/dri-devel/msg132235.html Patch433: drm-vc4-Fix-OOPSes-from-trying-to-cache-a-partially-constructed-BO..patch # bcm283x mmc for wifi http://www.spinics.net/lists/arm-kernel/msg567077.html Patch434: bcm283x-mmc-bcm2835.patch # Upstream fixes for i2c/serial/ethernet MAC addresses Patch435: bcm283x-fixes.patch # https://lists.freedesktop.org/archives/dri-devel/2017-February/133823.html Patch436: vc4-fix-vblank-cursor-update-issue.patch # http://www.spinics.net/lists/arm-kernel/msg552554.html Patch438: arm-imx6-hummingboard2.patch Patch460: lib-cpumask-Make-CPUMASK_OFFSTACK-usable-without-deb.patch Patch466: input-kill-stupid-messages.patch Patch467: die-floppy-die.patch Patch468: no-pcspkr-modalias.patch Patch470: silence-fbcon-logo.patch Patch471: Kbuild-Add-an-option-to-enable-GCC-VTA.patch Patch472: crash-driver.patch Patch473: efi-lockdown.patch Patch487: Add-EFI-signature-data-types.patch Patch488: Add-an-EFI-signature-blob-parser-and-key-loader.patch # This doesn't apply. It seems like it could be replaced by # https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=5ac7eace2d00eab5ae0e9fdee63e38aee6001f7c # which has an explicit line about blacklisting Patch489: KEYS-Add-a-system-blacklist-keyring.patch Patch490: MODSIGN-Import-certificates-from-UEFI-Secure-Boot.patch Patch491: MODSIGN-Support-not-importing-certs-from-db.patch Patch493: drm-i915-hush-check-crtc-state.patch Patch494: disable-i8042-check-on-apple-mac.patch Patch495: lis3-improve-handling-of-null-rate.patch Patch497: scsi-sd_revalidate_disk-prevent-NULL-ptr-deref.patch Patch498: criu-no-expert.patch Patch499: ath9k-rx-dma-stop-check.patch Patch500: xen-pciback-Don-t-disable-PCI_COMMAND-on-PCI-device-.patch Patch501: Input-synaptics-pin-3-touches-when-the-firmware-repo.patch Patch502: firmware-Drop-WARN-from-usermodehelper_read_trylock-.patch # Patch503: drm-i915-turn-off-wc-mmaps.patch Patch509: MODSIGN-Don-t-try-secure-boot-if-EFI-runtime-is-disa.patch #CVE-2016-3134 rhbz 1317383 1317384 Patch665: netfilter-x_tables-deal-with-bogus-nextoffset-values.patch # grabbed from mailing list Patch667: v3-Revert-tty-serial-pl011-add-ttyAMA-for-matching-pl011-console.patch # END OF PATCH DEFINITIONS -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-19 16:02 ` Gerhard Wiesinger @ 2017-03-20 3:05 ` Mike Galbraith -1 siblings, 0 replies; 88+ messages in thread From: Mike Galbraith @ 2017-03-20 3:05 UTC (permalink / raw) To: Gerhard Wiesinger, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Sun, 2017-03-19 at 17:02 +0100, Gerhard Wiesinger wrote: > mount | grep cgroup Just because controllers are mounted doesn't mean they're populated. To check that, you want to look for directories under the mount points with a non-empty 'tasks'. You will find some, but memory cgroup assignments would likely be most interesting for this thread. You can eliminate any diddling there by booting with cgroup_disable=memory. -Mike ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-20 3:05 ` Mike Galbraith 0 siblings, 0 replies; 88+ messages in thread From: Mike Galbraith @ 2017-03-20 3:05 UTC (permalink / raw) To: Gerhard Wiesinger, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Sun, 2017-03-19 at 17:02 +0100, Gerhard Wiesinger wrote: > mount | grep cgroup Just because controllers are mounted doesn't mean they're populated. To check that, you want to look for directories under the mount points with a non-empty 'tasks'. You will find some, but memory cgroup assignments would likely be most interesting for this thread. You can eliminate any diddling there by booting with cgroup_disable=memory. -Mike -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-20 3:05 ` Mike Galbraith @ 2017-03-21 5:59 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-21 5:59 UTC (permalink / raw) To: Mike Galbraith, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 20.03.2017 04:05, Mike Galbraith wrote: > On Sun, 2017-03-19 at 17:02 +0100, Gerhard Wiesinger wrote: > >> mount | grep cgroup > Just because controllers are mounted doesn't mean they're populated. To > check that, you want to look for directories under the mount points > with a non-empty 'tasks'. You will find some, but memory cgroup > assignments would likely be most interesting for this thread. You can > eliminate any diddling there by booting with cgroup_disable=memory. > Is this the correct information? mount | grep "type cgroup" | cut -f 3 -d ' ' | while read LINE; do echo "================================================================================================================================================================";echo ${LINE};ls -l ${LINE}; done ================================================================================================================================================================ /sys/fs/cgroup/systemd total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 60 root root 0 Mar 21 06:50 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/net_cls,net_prio total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 net_cls.classid -rw-r--r-- 1 root root 0 Mar 20 14:31 net_prio.ifpriomap -r--r--r-- 1 root root 0 Mar 20 14:31 net_prio.prioidx -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/cpu,cpuacct total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.stat -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_all -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_percpu -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_percpu_sys -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_percpu_user -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_sys -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_user -rw-r--r-- 1 root root 0 Mar 20 14:31 cpu.cfs_period_us -rw-r--r-- 1 root root 0 Mar 20 14:31 cpu.cfs_quota_us -rw-r--r-- 1 root root 0 Mar 20 14:31 cpu.shares -r--r--r-- 1 root root 0 Mar 20 14:31 cpu.stat drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 2 root root 0 Mar 20 14:31 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/devices total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior --w------- 1 root root 0 Mar 20 14:31 devices.allow --w------- 1 root root 0 Mar 20 14:31 devices.deny -r--r--r-- 1 root root 0 Mar 20 14:31 devices.list drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 60 root root 0 Mar 21 06:50 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/freezer total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/perf_event total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/cpuset total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.cpu_exclusive -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.cpus -r--r--r-- 1 root root 0 Mar 20 14:31 cpuset.effective_cpus -r--r--r-- 1 root root 0 Mar 20 14:31 cpuset.effective_mems -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.mem_exclusive -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.mem_hardwall -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_migrate -r--r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_pressure -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_pressure_enabled -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_spread_page -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_spread_slab -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.mems -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.sched_load_balance -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.sched_relax_domain_level -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/memory total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children --w--w--w- 1 root root 0 Mar 20 14:31 cgroup.event_control -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.failcnt --w------- 1 root root 0 Mar 20 14:31 memory.force_empty -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.failcnt -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.max_usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.slabinfo -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.tcp.failcnt -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.tcp.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.tcp.max_usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.tcp.usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.max_usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.memsw.failcnt -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.memsw.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.memsw.max_usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.memsw.usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.move_charge_at_immigrate -r--r--r-- 1 root root 0 Mar 20 14:31 memory.numa_stat -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.oom_control ---------- 1 root root 0 Mar 20 14:31 memory.pressure_level -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.soft_limit_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.stat -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.swappiness -r--r--r-- 1 root root 0 Mar 20 14:31 memory.usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.use_hierarchy -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 2 root root 0 Mar 20 14:31 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/pids total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 60 root root 0 Mar 21 06:50 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/hugetlb total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 hugetlb.2MB.failcnt -rw-r--r-- 1 root root 0 Mar 20 14:31 hugetlb.2MB.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 hugetlb.2MB.max_usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 hugetlb.2MB.usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/blkio total 0 -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.avg_queue_size -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.dequeue -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.empty_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.group_wait_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.idle_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_merged -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_merged_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_queued -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_queued_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_service_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_service_bytes_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_serviced -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_serviced_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_service_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_service_time_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_wait_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_wait_time_recursive -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.leaf_weight -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.leaf_weight_device --w------- 1 root root 0 Mar 20 14:31 blkio.reset_stats -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.sectors -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.sectors_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.io_service_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.io_serviced -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.read_bps_device -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.read_iops_device -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.write_bps_device -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.write_iops_device -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.time_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.unaccounted_time -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.weight -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.weight_device -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 2 root root 0 Mar 20 14:31 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice Thnx. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-21 5:59 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-21 5:59 UTC (permalink / raw) To: Mike Galbraith, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 20.03.2017 04:05, Mike Galbraith wrote: > On Sun, 2017-03-19 at 17:02 +0100, Gerhard Wiesinger wrote: > >> mount | grep cgroup > Just because controllers are mounted doesn't mean they're populated. To > check that, you want to look for directories under the mount points > with a non-empty 'tasks'. You will find some, but memory cgroup > assignments would likely be most interesting for this thread. You can > eliminate any diddling there by booting with cgroup_disable=memory. > Is this the correct information? mount | grep "type cgroup" | cut -f 3 -d ' ' | while read LINE; do echo "================================================================================================================================================================";echo ${LINE};ls -l ${LINE}; done ================================================================================================================================================================ /sys/fs/cgroup/systemd total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 60 root root 0 Mar 21 06:50 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/net_cls,net_prio total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 net_cls.classid -rw-r--r-- 1 root root 0 Mar 20 14:31 net_prio.ifpriomap -r--r--r-- 1 root root 0 Mar 20 14:31 net_prio.prioidx -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/cpu,cpuacct total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.stat -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_all -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_percpu -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_percpu_sys -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_percpu_user -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_sys -r--r--r-- 1 root root 0 Mar 20 14:31 cpuacct.usage_user -rw-r--r-- 1 root root 0 Mar 20 14:31 cpu.cfs_period_us -rw-r--r-- 1 root root 0 Mar 20 14:31 cpu.cfs_quota_us -rw-r--r-- 1 root root 0 Mar 20 14:31 cpu.shares -r--r--r-- 1 root root 0 Mar 20 14:31 cpu.stat drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 2 root root 0 Mar 20 14:31 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/devices total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior --w------- 1 root root 0 Mar 20 14:31 devices.allow --w------- 1 root root 0 Mar 20 14:31 devices.deny -r--r--r-- 1 root root 0 Mar 20 14:31 devices.list drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 60 root root 0 Mar 21 06:50 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/freezer total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/perf_event total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/cpuset total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.cpu_exclusive -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.cpus -r--r--r-- 1 root root 0 Mar 20 14:31 cpuset.effective_cpus -r--r--r-- 1 root root 0 Mar 20 14:31 cpuset.effective_mems -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.mem_exclusive -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.mem_hardwall -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_migrate -r--r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_pressure -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_pressure_enabled -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_spread_page -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.memory_spread_slab -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.mems -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.sched_load_balance -rw-r--r-- 1 root root 0 Mar 20 14:31 cpuset.sched_relax_domain_level -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/memory total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children --w--w--w- 1 root root 0 Mar 20 14:31 cgroup.event_control -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.failcnt --w------- 1 root root 0 Mar 20 14:31 memory.force_empty -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.failcnt -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.max_usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.slabinfo -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.tcp.failcnt -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.tcp.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.tcp.max_usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.tcp.usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.kmem.usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.max_usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.memsw.failcnt -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.memsw.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.memsw.max_usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.memsw.usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.move_charge_at_immigrate -r--r--r-- 1 root root 0 Mar 20 14:31 memory.numa_stat -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.oom_control ---------- 1 root root 0 Mar 20 14:31 memory.pressure_level -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.soft_limit_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 memory.stat -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.swappiness -r--r--r-- 1 root root 0 Mar 20 14:31 memory.usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 memory.use_hierarchy -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 2 root root 0 Mar 20 14:31 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/pids total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 60 root root 0 Mar 21 06:50 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice ================================================================================================================================================================ /sys/fs/cgroup/hugetlb total 0 -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior -rw-r--r-- 1 root root 0 Mar 20 14:31 hugetlb.2MB.failcnt -rw-r--r-- 1 root root 0 Mar 20 14:31 hugetlb.2MB.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 hugetlb.2MB.max_usage_in_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 hugetlb.2MB.usage_in_bytes -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks ================================================================================================================================================================ /sys/fs/cgroup/blkio total 0 -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.avg_queue_size -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.dequeue -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.empty_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.group_wait_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.idle_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_merged -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_merged_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_queued -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_queued_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_service_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_service_bytes_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_serviced -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_serviced_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_service_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_service_time_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_wait_time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.io_wait_time_recursive -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.leaf_weight -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.leaf_weight_device --w------- 1 root root 0 Mar 20 14:31 blkio.reset_stats -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.sectors -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.sectors_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.io_service_bytes -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.io_serviced -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.read_bps_device -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.read_iops_device -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.write_bps_device -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.throttle.write_iops_device -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.time -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.time_recursive -r--r--r-- 1 root root 0 Mar 20 14:31 blkio.unaccounted_time -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.weight -rw-r--r-- 1 root root 0 Mar 20 14:31 blkio.weight_device -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.clone_children -rw-r--r-- 1 root root 0 Mar 20 14:31 cgroup.procs -r--r--r-- 1 root root 0 Mar 20 14:31 cgroup.sane_behavior drwxr-xr-x 2 root root 0 Mar 20 14:31 init.scope -rw-r--r-- 1 root root 0 Mar 20 14:31 notify_on_release -rw-r--r-- 1 root root 0 Mar 20 14:31 release_agent drwxr-xr-x 2 root root 0 Mar 20 14:31 system.slice -rw-r--r-- 1 root root 0 Mar 20 14:31 tasks drwxr-xr-x 4 root root 0 Mar 21 06:55 user.slice Thnx. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-21 5:59 ` Gerhard Wiesinger @ 2017-03-21 7:13 ` Mike Galbraith -1 siblings, 0 replies; 88+ messages in thread From: Mike Galbraith @ 2017-03-21 7:13 UTC (permalink / raw) To: Gerhard Wiesinger, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: > Is this the correct information? Incomplete, but enough to reiterate cgroup_disable=memory suggestion. -Mike ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-21 7:13 ` Mike Galbraith 0 siblings, 0 replies; 88+ messages in thread From: Mike Galbraith @ 2017-03-21 7:13 UTC (permalink / raw) To: Gerhard Wiesinger, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: > Is this the correct information? Incomplete, but enough to reiterate cgroup_disable=memory suggestion. -Mike -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-21 7:13 ` Mike Galbraith @ 2017-03-23 7:16 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-23 7:16 UTC (permalink / raw) To: Mike Galbraith, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 21.03.2017 08:13, Mike Galbraith wrote: > On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: > >> Is this the correct information? > Incomplete, but enough to reiterate cgroup_disable=memory suggestion. > How to collect complete information? Thnx. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-23 7:16 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-23 7:16 UTC (permalink / raw) To: Mike Galbraith, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 21.03.2017 08:13, Mike Galbraith wrote: > On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: > >> Is this the correct information? > Incomplete, but enough to reiterate cgroup_disable=memory suggestion. > How to collect complete information? Thnx. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-23 7:16 ` Gerhard Wiesinger @ 2017-03-23 8:38 ` Mike Galbraith -1 siblings, 0 replies; 88+ messages in thread From: Mike Galbraith @ 2017-03-23 8:38 UTC (permalink / raw) To: Gerhard Wiesinger, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu, 2017-03-23 at 08:16 +0100, Gerhard Wiesinger wrote: > On 21.03.2017 08:13, Mike Galbraith wrote: > > On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: > > > > > Is this the correct information? > > Incomplete, but enough to reiterate cgroup_disable=memory > > suggestion. > > > > How to collect complete information? If Michal wants specifics, I suspect he'll ask. I posted only to pass along a speck of information, and offer a test suggestion.. twice. -Mike ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-23 8:38 ` Mike Galbraith 0 siblings, 0 replies; 88+ messages in thread From: Mike Galbraith @ 2017-03-23 8:38 UTC (permalink / raw) To: Gerhard Wiesinger, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On Thu, 2017-03-23 at 08:16 +0100, Gerhard Wiesinger wrote: > On 21.03.2017 08:13, Mike Galbraith wrote: > > On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: > > > > > Is this the correct information? > > Incomplete, but enough to reiterate cgroup_disable=memory > > suggestion. > > > > How to collect complete information? If Michal wants specifics, I suspect he'll ask. I posted only to pass along a speck of information, and offer a test suggestion.. twice. -Mike -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-23 8:38 ` Mike Galbraith @ 2017-03-23 14:46 ` Tetsuo Handa -1 siblings, 0 replies; 88+ messages in thread From: Tetsuo Handa @ 2017-03-23 14:46 UTC (permalink / raw) To: Mike Galbraith, Gerhard Wiesinger, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 2017/03/23 17:38, Mike Galbraith wrote: > On Thu, 2017-03-23 at 08:16 +0100, Gerhard Wiesinger wrote: >> On 21.03.2017 08:13, Mike Galbraith wrote: >>> On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: >>> >>>> Is this the correct information? >>> Incomplete, but enough to reiterate cgroup_disable=memory >>> suggestion. >>> >> >> How to collect complete information? > > If Michal wants specifics, I suspect he'll ask. I posted only to pass > along a speck of information, and offer a test suggestion.. twice. > > -Mike Isn't information Mike asked something like output from below command for i in `find /sys/fs/cgroup/memory/ -type f`; do echo ========== $i ==========; cat $i | xargs echo; done and check which cgroups stalling tasks belong to? Also, Mike suggested to reproduce your problem with cgroup_disable=memory kernel command line option in order to bisect whether memory cgroups are related to your problem. I don't know whether Michal already knows specific information to collect. I think memory allocation watchdog might give us some clue. It will give us output like http://I-love.SAKURA.ne.jp/tmp/serial-20170321.txt.xz . Can you afford building kernels with watchdog patch applied? Steps I tried for building kernels are shown below. (If you can't afford building but can afford trying binary rpms, I can upload them.) ---------------------------------------- yum -y install yum-utils wget https://dl.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/SRPMS/kernel-4.11.0-0.rc3.git0.1.fc27.src.rpm yum-builddep -y kernel-4.11.0-0.rc3.git0.1.fc27.src.rpm rpm -ivh kernel-4.11.0-0.rc3.git0.1.fc27.src.rpm yum-builddep -y ~/rpmbuild/SPECS/kernel.spec patch -p1 -d ~/rpmbuild/SPECS/ << "EOF" --- a/kernel.spec +++ b/kernel.spec @@ -24,7 +24,7 @@ %global zipsed -e 's/\.ko$/\.ko.xz/' %endif -# define buildid .local +%define buildid .kmallocwd # baserelease defines which build revision of this kernel version we're # building. We used to call this fedora_build, but the magical name @@ -1207,6 +1207,8 @@ git am %{patches} +patch -p1 < $RPM_SOURCE_DIR/kmallocwd.patch + # END OF PATCH APPLICATIONS # Any further pre-build tree manipulations happen here. @@ -1243,6 +1245,8 @@ do cat $i > temp-$i mv $i .config + echo 'CONFIG_DETECT_MEMALLOC_STALL_TASK=y' >> .config + echo 'CONFIG_DEFAULT_MEMALLOC_TASK_TIMEOUT=30' >> .config Arch=`head -1 .config | cut -b 3-` make ARCH=$Arch listnewconfig | grep -E '^CONFIG_' >.newoptions || true %if %{listnewconfig_fail} EOF wget -O ~/rpmbuild/SOURCES/kmallocwd.patch 'https://marc.info/?l=linux-mm&m=148957858821214&q=raw' rpmbuild -bb ~/rpmbuild/SPECS/kernel.spec ---------------------------------------- ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-23 14:46 ` Tetsuo Handa 0 siblings, 0 replies; 88+ messages in thread From: Tetsuo Handa @ 2017-03-23 14:46 UTC (permalink / raw) To: Mike Galbraith, Gerhard Wiesinger, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 2017/03/23 17:38, Mike Galbraith wrote: > On Thu, 2017-03-23 at 08:16 +0100, Gerhard Wiesinger wrote: >> On 21.03.2017 08:13, Mike Galbraith wrote: >>> On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: >>> >>>> Is this the correct information? >>> Incomplete, but enough to reiterate cgroup_disable=memory >>> suggestion. >>> >> >> How to collect complete information? > > If Michal wants specifics, I suspect he'll ask. I posted only to pass > along a speck of information, and offer a test suggestion.. twice. > > -Mike Isn't information Mike asked something like output from below command for i in `find /sys/fs/cgroup/memory/ -type f`; do echo ========== $i ==========; cat $i | xargs echo; done and check which cgroups stalling tasks belong to? Also, Mike suggested to reproduce your problem with cgroup_disable=memory kernel command line option in order to bisect whether memory cgroups are related to your problem. I don't know whether Michal already knows specific information to collect. I think memory allocation watchdog might give us some clue. It will give us output like http://I-love.SAKURA.ne.jp/tmp/serial-20170321.txt.xz . Can you afford building kernels with watchdog patch applied? Steps I tried for building kernels are shown below. (If you can't afford building but can afford trying binary rpms, I can upload them.) ---------------------------------------- yum -y install yum-utils wget https://dl.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/SRPMS/kernel-4.11.0-0.rc3.git0.1.fc27.src.rpm yum-builddep -y kernel-4.11.0-0.rc3.git0.1.fc27.src.rpm rpm -ivh kernel-4.11.0-0.rc3.git0.1.fc27.src.rpm yum-builddep -y ~/rpmbuild/SPECS/kernel.spec patch -p1 -d ~/rpmbuild/SPECS/ << "EOF" --- a/kernel.spec +++ b/kernel.spec @@ -24,7 +24,7 @@ %global zipsed -e 's/\.ko$/\.ko.xz/' %endif -# define buildid .local +%define buildid .kmallocwd # baserelease defines which build revision of this kernel version we're # building. We used to call this fedora_build, but the magical name @@ -1207,6 +1207,8 @@ git am %{patches} +patch -p1 < $RPM_SOURCE_DIR/kmallocwd.patch + # END OF PATCH APPLICATIONS # Any further pre-build tree manipulations happen here. @@ -1243,6 +1245,8 @@ do cat $i > temp-$i mv $i .config + echo 'CONFIG_DETECT_MEMALLOC_STALL_TASK=y' >> .config + echo 'CONFIG_DEFAULT_MEMALLOC_TASK_TIMEOUT=30' >> .config Arch=`head -1 .config | cut -b 3-` make ARCH=$Arch listnewconfig | grep -E '^CONFIG_' >.newoptions || true %if %{listnewconfig_fail} EOF wget -O ~/rpmbuild/SOURCES/kmallocwd.patch 'https://marc.info/?l=linux-mm&m=148957858821214&q=raw' rpmbuild -bb ~/rpmbuild/SPECS/kernel.spec ---------------------------------------- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels 2017-03-23 8:38 ` Mike Galbraith @ 2017-03-26 8:36 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-26 8:36 UTC (permalink / raw) To: Mike Galbraith, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 23.03.2017 09:38, Mike Galbraith wrote: > On Thu, 2017-03-23 at 08:16 +0100, Gerhard Wiesinger wrote: >> On 21.03.2017 08:13, Mike Galbraith wrote: >>> On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: >>> >>>> Is this the correct information? >>> Incomplete, but enough to reiterate cgroup_disable=memory >>> suggestion. >>> >> How to collect complete information? > If Michal wants specifics, I suspect he'll ask. I posted only to pass > along a speck of information, and offer a test suggestion.. twice. Still OOM with cgroup_disable=memory, kernel 4.11.0-0.rc3.git0.2.fc26.x86_64,I set vm.min_free_kbytes = 10240 in these tests. # Full config grep "vm\." /etc/sysctl.d/* /etc/sysctl.d/00-dirty_background_ratio.conf:vm.dirty_background_ratio = 3 /etc/sysctl.d/00-dirty_ratio.conf:vm.dirty_ratio = 15 /etc/sysctl.d/00-kernel-vm-min-free-kbyzes.conf:vm.min_free_kbytes = 10240 /etc/sysctl.d/00-overcommit_memory.conf:vm.overcommit_memory = 2 /etc/sysctl.d/00-overcommit_ratio.conf:vm.overcommit_ratio = 80 /etc/sysctl.d/00-swappiness.conf:vm.swappiness=10 [31880.623557] sa1: page allocation stalls for 10942ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) [31880.623623] sa1 cpuset=/ mems_allowed=0 [31880.623630] CPU: 1 PID: 17112 Comm: sa1 Not tainted 4.11.0-0.rc3.git0.2.fc26.x86_64 #1 [31880.623724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 04/01/2014 [31880.623819] Call Trace: [31880.623893] dump_stack+0x63/0x84 [31880.623971] warn_alloc+0x10c/0x1b0 [31880.624046] __alloc_pages_slowpath+0x93d/0xe60 [31880.624142] ? get_page_from_freelist+0x122/0xbf0 [31880.624225] ? unmap_region+0xf7/0x130 [31880.624312] __alloc_pages_nodemask+0x290/0x2b0 [31880.624388] alloc_pages_vma+0xa0/0x2b0 [31880.624463] __handle_mm_fault+0x4d0/0x1160 [31880.624550] handle_mm_fault+0xb3/0x250 [31880.624628] __do_page_fault+0x23f/0x4c0 [31880.624701] trace_do_page_fault+0x41/0x120 [31880.624781] do_async_page_fault+0x51/0xa0 [31880.624866] async_page_fault+0x28/0x30 [31880.624941] RIP: 0033:0x7f9218d4914f [31880.625032] RSP: 002b:00007ffe0d1376a8 EFLAGS: 00010206 [31880.625153] RAX: 00007f9218d2a314 RBX: 00007f9218f4e658 RCX: 00007f9218d2a354 [31880.625235] RDX: 00000000000005ec RSI: 0000000000000000 RDI: 00007f9218d2a314 [31880.625324] RBP: 00007ffe0d137950 R08: 00007f9218d2a900 R09: 0000000000027000 [31880.625423] R10: 00007ffe0d1376e0 R11: 00007f9218d2a900 R12: 0000000000000003 [31880.625505] R13: 00007ffe0d137a38 R14: 000000000000fd01 R15: 0000000000000002 [31880.625688] Mem-Info: [31880.625762] active_anon:36671 inactive_anon:36711 isolated_anon:88 active_file:1399 inactive_file:1410 isolated_file:0 unevictable:0 dirty:5 writeback:15 unstable:0 slab_reclaimable:3099 slab_unreclaimable:3558 mapped:2037 shmem:3 pagetables:3340 bounce:0 free:2972 free_pcp:102 free_cma:0 [31880.627334] Node 0 active_anon:146684kB inactive_anon:146816kB active_file:5596kB inactive_file:5572kB unevictable:0kB isolated(anon):368kB isolated(file):0kB mapped:8044kB dirty:20kB writeback:136kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12kB writeback_tmp:0kB unstable:0kB pages_scanned:82 all_unreclaimable? no [31880.627606] Node 0 DMA free:1816kB min:440kB low:548kB high:656kB active_anon:5636kB inactive_anon:6844kB active_file:132kB inactive_file:148kB unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:284kB slab_unreclaimable:532kB kernel_stack:0kB pagetables:188kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [31880.627883] lowmem_reserve[]: 0 327 327 327 327 [31880.627959] Node 0 DMA32 free:10072kB min:9796kB low:12244kB high:14692kB active_anon:141048kB inactive_anon:140000kB active_file:5432kB inactive_file:5444kB unevictable:0kB writepending:152kB present:376688kB managed:353760kB mlocked:0kB slab_reclaimable:12112kB slab_unreclaimable:13700kB kernel_stack:2464kB pagetables:13172kB bounce:0kB free_pcp:504kB local_pcp:272kB free_cma:0kB [31880.628334] lowmem_reserve[]: 0 0 0 0 0 [31880.629882] Node 0 DMA: 33*4kB (UME) 24*8kB (UM) 26*16kB (UME) 4*32kB (UME) 5*64kB (UME) 1*128kB (E) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1828kB [31880.632255] Node 0 DMA32: 174*4kB (UMEH) 107*8kB (UMEH) 96*16kB (UMEH) 59*32kB (UME) 30*64kB (UMEH) 8*128kB (UEH) 8*256kB (UMEH) 1*512kB (E) 0*1024kB 0*2048kB 0*4096kB = 10480kB [31880.634344] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [31880.634346] 7276 total pagecache pages [31880.635277] 4367 pages in swap cache [31880.636206] Swap cache stats: add 5639999, delete 5635551, find 6573228/8496821 [31880.637145] Free swap = 973736kB [31880.638038] Total swap = 2064380kB [31880.638988] 98170 pages RAM [31880.640309] 0 pages HighMem/MovableOnly [31880.641791] 5753 pages reserved [31880.642908] 0 pages cma reserved [31880.643978] 0 pages hwpoisoned Will try your suggestions with the custom build kernel. Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er/4.10er kernels @ 2017-03-26 8:36 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2017-03-26 8:36 UTC (permalink / raw) To: Mike Galbraith, Michal Hocko Cc: lkml, Minchan Kim, linux-kernel, linux-mm, Linus Torvalds On 23.03.2017 09:38, Mike Galbraith wrote: > On Thu, 2017-03-23 at 08:16 +0100, Gerhard Wiesinger wrote: >> On 21.03.2017 08:13, Mike Galbraith wrote: >>> On Tue, 2017-03-21 at 06:59 +0100, Gerhard Wiesinger wrote: >>> >>>> Is this the correct information? >>> Incomplete, but enough to reiterate cgroup_disable=memory >>> suggestion. >>> >> How to collect complete information? > If Michal wants specifics, I suspect he'll ask. I posted only to pass > along a speck of information, and offer a test suggestion.. twice. Still OOM with cgroup_disable=memory, kernel 4.11.0-0.rc3.git0.2.fc26.x86_64,I set vm.min_free_kbytes = 10240 in these tests. # Full config grep "vm\." /etc/sysctl.d/* /etc/sysctl.d/00-dirty_background_ratio.conf:vm.dirty_background_ratio = 3 /etc/sysctl.d/00-dirty_ratio.conf:vm.dirty_ratio = 15 /etc/sysctl.d/00-kernel-vm-min-free-kbyzes.conf:vm.min_free_kbytes = 10240 /etc/sysctl.d/00-overcommit_memory.conf:vm.overcommit_memory = 2 /etc/sysctl.d/00-overcommit_ratio.conf:vm.overcommit_ratio = 80 /etc/sysctl.d/00-swappiness.conf:vm.swappiness=10 [31880.623557] sa1: page allocation stalls for 10942ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) [31880.623623] sa1 cpuset=/ mems_allowed=0 [31880.623630] CPU: 1 PID: 17112 Comm: sa1 Not tainted 4.11.0-0.rc3.git0.2.fc26.x86_64 #1 [31880.623724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 04/01/2014 [31880.623819] Call Trace: [31880.623893] dump_stack+0x63/0x84 [31880.623971] warn_alloc+0x10c/0x1b0 [31880.624046] __alloc_pages_slowpath+0x93d/0xe60 [31880.624142] ? get_page_from_freelist+0x122/0xbf0 [31880.624225] ? unmap_region+0xf7/0x130 [31880.624312] __alloc_pages_nodemask+0x290/0x2b0 [31880.624388] alloc_pages_vma+0xa0/0x2b0 [31880.624463] __handle_mm_fault+0x4d0/0x1160 [31880.624550] handle_mm_fault+0xb3/0x250 [31880.624628] __do_page_fault+0x23f/0x4c0 [31880.624701] trace_do_page_fault+0x41/0x120 [31880.624781] do_async_page_fault+0x51/0xa0 [31880.624866] async_page_fault+0x28/0x30 [31880.624941] RIP: 0033:0x7f9218d4914f [31880.625032] RSP: 002b:00007ffe0d1376a8 EFLAGS: 00010206 [31880.625153] RAX: 00007f9218d2a314 RBX: 00007f9218f4e658 RCX: 00007f9218d2a354 [31880.625235] RDX: 00000000000005ec RSI: 0000000000000000 RDI: 00007f9218d2a314 [31880.625324] RBP: 00007ffe0d137950 R08: 00007f9218d2a900 R09: 0000000000027000 [31880.625423] R10: 00007ffe0d1376e0 R11: 00007f9218d2a900 R12: 0000000000000003 [31880.625505] R13: 00007ffe0d137a38 R14: 000000000000fd01 R15: 0000000000000002 [31880.625688] Mem-Info: [31880.625762] active_anon:36671 inactive_anon:36711 isolated_anon:88 active_file:1399 inactive_file:1410 isolated_file:0 unevictable:0 dirty:5 writeback:15 unstable:0 slab_reclaimable:3099 slab_unreclaimable:3558 mapped:2037 shmem:3 pagetables:3340 bounce:0 free:2972 free_pcp:102 free_cma:0 [31880.627334] Node 0 active_anon:146684kB inactive_anon:146816kB active_file:5596kB inactive_file:5572kB unevictable:0kB isolated(anon):368kB isolated(file):0kB mapped:8044kB dirty:20kB writeback:136kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12kB writeback_tmp:0kB unstable:0kB pages_scanned:82 all_unreclaimable? no [31880.627606] Node 0 DMA free:1816kB min:440kB low:548kB high:656kB active_anon:5636kB inactive_anon:6844kB active_file:132kB inactive_file:148kB unevictable:0kB writepending:4kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:284kB slab_unreclaimable:532kB kernel_stack:0kB pagetables:188kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [31880.627883] lowmem_reserve[]: 0 327 327 327 327 [31880.627959] Node 0 DMA32 free:10072kB min:9796kB low:12244kB high:14692kB active_anon:141048kB inactive_anon:140000kB active_file:5432kB inactive_file:5444kB unevictable:0kB writepending:152kB present:376688kB managed:353760kB mlocked:0kB slab_reclaimable:12112kB slab_unreclaimable:13700kB kernel_stack:2464kB pagetables:13172kB bounce:0kB free_pcp:504kB local_pcp:272kB free_cma:0kB [31880.628334] lowmem_reserve[]: 0 0 0 0 0 [31880.629882] Node 0 DMA: 33*4kB (UME) 24*8kB (UM) 26*16kB (UME) 4*32kB (UME) 5*64kB (UME) 1*128kB (E) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1828kB [31880.632255] Node 0 DMA32: 174*4kB (UMEH) 107*8kB (UMEH) 96*16kB (UMEH) 59*32kB (UME) 30*64kB (UMEH) 8*128kB (UEH) 8*256kB (UMEH) 1*512kB (E) 0*1024kB 0*2048kB 0*4096kB = 10480kB [31880.634344] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [31880.634346] 7276 total pagecache pages [31880.635277] 4367 pages in swap cache [31880.636206] Swap cache stats: add 5639999, delete 5635551, find 6573228/8496821 [31880.637145] Free swap = 973736kB [31880.638038] Total swap = 2064380kB [31880.638988] 98170 pages RAM [31880.640309] 0 pages HighMem/MovableOnly [31880.641791] 5753 pages reserved [31880.642908] 0 pages cma reserved [31880.643978] 0 pages hwpoisoned Will try your suggestions with the custom build kernel. Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels 2016-12-09 13:40 ` Michal Hocko @ 2016-12-09 16:03 ` Gerhard Wiesinger -1 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 16:03 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 14:40, Michal Hocko wrote: > On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >> Hello, >> >> same with latest kernel rc, dnf still killed with OOM (but sometimes >> better). >> >> ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} >> (does dnf clean all;dnf update) >> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >> >> Updated bug report: >> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > Could you post your oom report please? And another one which ended in a native_safe_halt .... [73366.837826] nmbd: page allocation failure: order:0, mode:0x2280030(GFP_ATOMIC|__GFP_RECLAIMABLE|__GFP_NOTRACK) [73366.837985] CPU: 1 PID: 2005 Comm: nmbd Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [73366.838075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [73366.838175] ffffaa4ac059f548 ffffffff8d4774e3 ffffffff8dc7dd98 0000000000000000 [73366.838272] ffffaa4ac059f5d0 ffffffff8d20e6ea 0228003000000046 ffffffff8dc7dd98 [73366.838364] ffffaa4ac059f570 ffff9c3700000010 ffffaa4ac059f5e0 ffffaa4ac059f590 [73366.838458] Call Trace: [73366.838590] [<ffffffff8d4774e3>] dump_stack+0x86/0xc3 [73366.838680] [<ffffffff8d20e6ea>] warn_alloc+0x13a/0x170 [73366.838762] [<ffffffff8d20e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [73366.838846] [<ffffffff8d0e09a0>] ? finish_task_switch+0xb0/0x260 [73366.838926] [<ffffffff8d20f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [73366.839007] [<ffffffff8d26db51>] alloc_pages_current+0xa1/0x1f0 [73366.839088] [<ffffffff8d068475>] ? kvm_sched_clock_read+0x25/0x40 [73366.839170] [<ffffffff8d278956>] new_slab+0x316/0x7c0 [73366.839245] [<ffffffff8d27ae8b>] ___slab_alloc+0x3fb/0x5c0 [73366.839325] [<ffffffff8d068475>] ? kvm_sched_clock_read+0x25/0x40 [73366.839409] [<ffffffff8d3a4503>] ? __es_insert_extent+0xb3/0x330 [73366.839501] [<ffffffff8d3a4503>] ? __es_insert_extent+0xb3/0x330 [73366.839583] [<ffffffff8d27b0a1>] __slab_alloc+0x51/0x90 [73366.839662] [<ffffffff8d3a4503>] ? __es_insert_extent+0xb3/0x330 [73366.839743] [<ffffffff8d27b326>] kmem_cache_alloc+0x246/0x2d0 [73366.839822] [<ffffffff8d3a5066>] ? __es_remove_extent+0x56/0x2d0 [73366.839906] [<ffffffff8d3a4503>] __es_insert_extent+0xb3/0x330 [73366.839985] [<ffffffff8d3a573e>] ext4_es_insert_extent+0xee/0x280 [73366.840067] [<ffffffff8d35a704>] ? ext4_map_blocks+0x2b4/0x5f0 [73366.840147] [<ffffffff8d35a773>] ext4_map_blocks+0x323/0x5f0 [73366.840225] [<ffffffff8d23dfda>] ? workingset_refault+0x10a/0x220 [73366.840314] [<ffffffff8d3ad7d3>] ext4_mpage_readpages+0x413/0xa60 [73366.840397] [<ffffffff8d201d96>] ? __page_cache_alloc+0x146/0x190 [73366.840487] [<ffffffff8d358235>] ext4_readpages+0x35/0x40 [73366.840569] [<ffffffff8d216d3f>] __do_page_cache_readahead+0x2bf/0x390 [73366.840651] [<ffffffff8d216bea>] ? __do_page_cache_readahead+0x16a/0x390 [73366.840735] [<ffffffff8d20622b>] filemap_fault+0x51b/0x790 [73366.840814] [<ffffffff8d3639ce>] ? ext4_filemap_fault+0x2e/0x50 [73366.840896] [<ffffffff8d3639d9>] ext4_filemap_fault+0x39/0x50 [73366.840976] [<ffffffff8d241ca3>] __do_fault+0x83/0x1d0 [73366.841056] [<ffffffff8d246642>] handle_mm_fault+0x11e2/0x17a0 [73366.841138] [<ffffffff8d2454ba>] ? handle_mm_fault+0x5a/0x17a0 [73366.841220] [<ffffffff8d06de16>] __do_page_fault+0x266/0x520 [73366.841300] [<ffffffff8d06e1a8>] trace_do_page_fault+0x58/0x2a0 [73366.841382] [<ffffffff8d067f3a>] do_async_page_fault+0x1a/0xa0 [73366.841464] [<ffffffff8d918e28>] async_page_fault+0x28/0x30 [73366.842500] Mem-Info: [73366.843149] active_anon:8677 inactive_anon:8798 isolated_anon:0 active_file:328 inactive_file:317 isolated_file:32 unevictable:0 dirty:0 writeback:2 unstable:0 slab_reclaimable:4968 slab_unreclaimable:9242 mapped:365 shmem:1 pagetables:2690 bounce:0 free:764 free_pcp:41 free_cma:0 [73366.846832] Node 0 active_anon:34708kB inactive_anon:35192kB active_file:1312kB inactive_file:1268kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:1460kB dirty:0kB writeback:8kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 4kB writeback_tmp:0kB unstable:0kB pages_scanned:32 all_unreclaimable? no [73366.848711] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB active_anon:3216kB inactive_anon:3448kB active_file:40kB inactive_file:228kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:2064kB slab_unreclaimable:2960kB kernel_stack:100kB pagetables:1536kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [73366.850769] lowmem_reserve[]: 0 116 116 116 116 [73366.851479] Node 0 DMA32 free:1588kB min:1296kB low:1620kB high:1944kB active_anon:31464kB inactive_anon:31740kB active_file:1236kB inactive_file:1056kB unevictable:0kB writepending:0kB present:180080kB managed:139012kB mlocked:0kB slab_reclaimable:17808kB slab_unreclaimable:34008kB kernel_stack:1676kB pagetables:9224kB bounce:0kB free_pcp:164kB local_pcp:12kB free_cma:0kB [73366.853757] lowmem_reserve[]: 0 0 0 0 0 [73366.854544] Node 0 DMA: 13*4kB (H) 13*8kB (H) 17*16kB (H) 12*32kB (H) 8*64kB (H) 1*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1452kB [73366.856200] Node 0 DMA32: 70*4kB (UMH) 12*8kB (MH) 12*16kB (H) 2*32kB (H) 5*64kB (H) 5*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1592kB [73366.857955] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [73366.857956] 2401 total pagecache pages [73366.858829] 1741 pages in swap cache [73366.859721] Swap cache stats: add 1230889, delete 1229148, find 3509739/3747264 [73366.860616] Free swap = 2059496kB [73366.861500] Total swap = 2097148kB [73366.862578] 49018 pages RAM [73366.863560] 0 pages HighMem/MovableOnly [73366.864531] 10288 pages reserved [73366.865436] 0 pages cma reserved [73366.866395] 0 pages hwpoisoned [73366.867503] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [73366.868507] cache: ext4_extent_status, object size: 40, buffer size: 40, default order: 0, min order: 0 [73366.869508] node 0: slabs: 13, objs: 1326, free: 0 [96351.012045] dmcrypt_write: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) [96351.024364] CPU: 0 PID: 1593 Comm: dmcrypt_write Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [96351.027132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [96351.029537] ffffaa4ac025b508 ffffffff8d4774e3 ffffffff8dc7dd98 0000000000000000 [96351.033023] ffffaa4ac025b590 ffffffff8d20e6ea 0228002000000046 ffffffff8dc7dd98 [96351.036214] ffffaa4ac025b530 ffff9c3700000010 ffffaa4ac025b5a0 ffffaa4ac025b550 [96351.039266] Call Trace: [96351.041426] [<ffffffff8d4774e3>] dump_stack+0x86/0xc3 [96351.044511] [<ffffffff8d20e6ea>] warn_alloc+0x13a/0x170 [96351.046698] [<ffffffff8d20e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [96351.048852] [<ffffffff8d20f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [96351.052025] [<ffffffff8d26db51>] alloc_pages_current+0xa1/0x1f0 [96351.053975] [<ffffffff8d916f17>] ? _raw_spin_unlock+0x27/0x40 [96351.055066] [<ffffffff8d278956>] new_slab+0x316/0x7c0 [96351.056134] [<ffffffff8d0ed4c7>] ? sched_clock_cpu+0xa7/0xc0 [96351.057236] [<ffffffff8d27ae8b>] ___slab_alloc+0x3fb/0x5c0 [96351.058287] [<ffffffff8d10b042>] ? cpuacct_charge+0xf2/0x1f0 [96351.059374] [<ffffffffc03a650d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [96351.060439] [<ffffffffc03a650d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [96351.061484] [<ffffffff8d27b0a1>] __slab_alloc+0x51/0x90 [96351.062505] [<ffffffff8d27d141>] __kmalloc+0x251/0x320 [96351.063619] [<ffffffffc03a650d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [96351.064624] [<ffffffffc03a650d>] alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [96351.065631] [<ffffffffc03a7193>] virtqueue_add_sgs+0x1c3/0x4a0 [virtio_ring] [96351.066564] [<ffffffff8d068475>] ? kvm_sched_clock_read+0x25/0x40 [96351.067504] [<ffffffffc041e65c>] __virtblk_add_req+0xbc/0x220 [virtio_blk] [96351.068425] [<ffffffff8d1312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 [96351.069351] [<ffffffffc041e935>] ? virtio_queue_rq+0x105/0x290 [virtio_blk] [96351.070229] [<ffffffffc041e95d>] virtio_queue_rq+0x12d/0x290 [virtio_blk] [96351.071059] [<ffffffff8d45015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 [96351.071904] [<ffffffff8d44fecd>] blk_mq_run_hw_queue+0xad/0xd0 [96351.072727] [<ffffffff8d450fca>] blk_mq_insert_requests+0x24a/0x320 [96351.073607] [<ffffffff8d452419>] blk_mq_flush_plug_list+0x139/0x160 [96351.074397] [<ffffffff8d4451d6>] blk_flush_plug_list+0xb6/0x250 [96351.075111] [<ffffffff8d44586c>] blk_finish_plug+0x2c/0x40 [96351.075869] [<ffffffffc03adef0>] dmcrypt_write+0x210/0x220 [dm_crypt] [96351.076613] [<ffffffff8d0e6590>] ? wake_up_q+0x80/0x80 [96351.077376] [<ffffffffc03adce0>] ? crypt_iv_essiv_dtr+0x70/0x70 [dm_crypt] [96351.078086] [<ffffffffc03adce0>] ? crypt_iv_essiv_dtr+0x70/0x70 [dm_crypt] [96351.078892] [<ffffffff8d0d82c2>] kthread+0x102/0x120 [96351.079635] [<ffffffff8d111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 [96351.080403] [<ffffffff8d0d81c0>] ? kthread_park+0x60/0x60 [96351.081115] [<ffffffff8d917afa>] ret_from_fork+0x2a/0x40 [96351.081882] Mem-Info: [96351.082616] active_anon:8390 inactive_anon:8478 isolated_anon:32 active_file:25 inactive_file:30 isolated_file:0 unevictable:0 dirty:0 writeback:151 unstable:0 slab_reclaimable:5304 slab_unreclaimable:9678 mapped:24 shmem:0 pagetables:3012 bounce:0 free:715 free_pcp:77 free_cma:0 [96351.086909] Node 0 active_anon:33560kB inactive_anon:33912kB active_file:100kB inactive_file:120kB unevictable:0kB isolated(anon):128kB isolated(file):0kB mapped:96kB dirty:0kB writeback:604kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:395 all_unreclaimable? no [96351.089052] Node 0 DMA free:1452kB min:172kB low:212kB high:252kB active_anon:3296kB inactive_anon:3768kB active_file:12kB inactive_file:0kB unevictable:0kB writepending:24kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:2508kB slab_unreclaimable:3108kB kernel_stack:100kB pagetables:1024kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [96351.091383] lowmem_reserve[]: 0 116 116 116 116 [96351.092155] Node 0 DMA32 free:1408kB min:1296kB low:1620kB high:1944kB active_anon:30264kB inactive_anon:30140kB active_file:100kB inactive_file:124kB unevictable:0kB writepending:580kB present:180080kB managed:139012kB mlocked:0kB slab_reclaimable:18708kB slab_unreclaimable:35604kB kernel_stack:1820kB pagetables:11024kB bounce:0kB free_pcp:308kB local_pcp:152kB free_cma:0kB [96351.094895] lowmem_reserve[]: 0 0 0 0 0 [96351.095803] Node 0 DMA: 13*4kB (H) 13*8kB (H) 17*16kB (H) 12*32kB (H) 8*64kB (H) 1*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1452kB [96351.097708] Node 0 DMA32: 16*4kB (H) 6*8kB (H) 5*16kB (H) 8*32kB (H) 5*64kB (H) 5*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB [96351.099672] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [96351.099673] 5020 total pagecache pages [96351.100659] 4961 pages in swap cache [96351.101635] Swap cache stats: add 1422746, delete 1417785, find 4506321/4784232 [96351.102630] Free swap = 1987544kB [96351.103666] Total swap = 2097148kB [96351.104730] 49018 pages RAM [96351.105716] 0 pages HighMem/MovableOnly [96351.106687] 10288 pages reserved [96351.107642] 0 pages cma reserved [96351.108584] 0 pages hwpoisoned [96351.109523] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [96351.110479] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [96351.111428] node 0: slabs: 102, objs: 1632, free: 0 [96361.915109] swapper/0: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) [96361.916235] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [96361.917276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [96361.918313] ffff9c3749e03628 ffffffff8d4774e3 ffffffff8dc7dd98 0000000000000000 [96361.919381] ffff9c3749e036b0 ffffffff8d20e6ea 0228002000000046 ffffffff8dc7dd98 [96361.920511] ffff9c3749e03650 ffff9c3700000010 ffff9c3749e036c0 ffff9c3749e03670 [96361.921551] Call Trace: [96361.922549] <IRQ> [96361.922566] [<ffffffff8d4774e3>] dump_stack+0x86/0xc3 [96361.923684] [<ffffffff8d20e6ea>] warn_alloc+0x13a/0x170 [96361.924735] [<ffffffff8d20e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [96361.925757] [<ffffffff8d20f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [96361.926772] [<ffffffff8d26db51>] alloc_pages_current+0xa1/0x1f0 [96361.927844] [<ffffffff8d278956>] new_slab+0x316/0x7c0 [96361.928884] [<ffffffff8d27ae8b>] ___slab_alloc+0x3fb/0x5c0 [96361.929874] [<ffffffff8d79b00b>] ? __alloc_skb+0x5b/0x1e0 [96361.930854] [<ffffffff8d037de9>] ? sched_clock+0x9/0x10 [96361.931829] [<ffffffff8d0ed4c7>] ? sched_clock_cpu+0xa7/0xc0 [96361.932796] [<ffffffff8d79b00b>] ? __alloc_skb+0x5b/0x1e0 [96361.933778] [<ffffffff8d27b0a1>] __slab_alloc+0x51/0x90 [96361.934764] [<ffffffff8d27b7c2>] kmem_cache_alloc_node+0xb2/0x310 [96361.935713] [<ffffffff8d79b00b>] ? __alloc_skb+0x5b/0x1e0 [96361.936637] [<ffffffff8d79b00b>] __alloc_skb+0x5b/0x1e0 [96361.937534] [<ffffffff8d7c2a9f>] __neigh_notify+0x3f/0xd0 [96361.938407] [<ffffffff8d7c64b9>] neigh_update+0x379/0x8b0 [96361.939251] [<ffffffff8d0b7000>] ? __local_bh_enable_ip+0x70/0xc0 [96361.940091] [<ffffffff8d844702>] ? udp4_ufo_fragment+0x122/0x1a0 [96361.940915] [<ffffffff8d845ee8>] arp_process+0x2e8/0x9f0 [96361.941708] [<ffffffff8d846745>] arp_rcv+0x135/0x300 [96361.942473] [<ffffffff8d111ed6>] ? __lock_acquire+0x346/0x1290 [96361.943211] [<ffffffff8d7b42cd>] ? netif_receive_skb_internal+0x6d/0x200 [96361.943942] [<ffffffff8d7b36da>] __netif_receive_skb_core+0x23a/0xd60 [96361.944664] [<ffffffff8d7b42d3>] ? netif_receive_skb_internal+0x73/0x200 [96361.945360] [<ffffffff8d7b4218>] __netif_receive_skb+0x18/0x60 [96361.946045] [<ffffffff8d7b4320>] netif_receive_skb_internal+0xc0/0x200 [96361.946693] [<ffffffff8d7b42d3>] ? netif_receive_skb_internal+0x73/0x200 [96361.947329] [<ffffffff8d7b628c>] napi_gro_receive+0x13c/0x200 [96361.948002] [<ffffffffc04cb667>] virtnet_receive+0x477/0x9a0 [virtio_net] [96361.948690] [<ffffffffc04cbbad>] virtnet_poll+0x1d/0x80 [virtio_net] [96361.949323] [<ffffffff8d7b501e>] net_rx_action+0x23e/0x470 [96361.949994] [<ffffffff8d91a8cd>] __do_softirq+0xcd/0x4b9 [96361.950639] [<ffffffff8d0b7e98>] irq_exit+0x108/0x110 [96361.951275] [<ffffffff8d91a49a>] do_IRQ+0x6a/0x120 [96361.951925] [<ffffffff8d918256>] common_interrupt+0x96/0x96 [96361.952578] <EOI> [96361.952589] [<ffffffff8d916676>] ? native_safe_halt+0x6/0x10 [96361.953245] [<ffffffff8d916235>] default_idle+0x25/0x190 [96361.953912] [<ffffffff8d03918f>] arch_cpu_idle+0xf/0x20 [96361.954562] [<ffffffff8d916873>] default_idle_call+0x23/0x40 [96361.955192] [<ffffffff8d104695>] cpu_startup_entry+0x1d5/0x250 [96361.955864] [<ffffffff8d9066b5>] rest_init+0x135/0x140 [96361.956485] [<ffffffff8e1ca01a>] start_kernel+0x48e/0x4af [96361.957065] [<ffffffff8e1c9120>] ? early_idt_handler_array+0x120/0x120 [96361.957644] [<ffffffff8e1c92ca>] x86_64_start_reservations+0x24/0x26 [96361.958212] [<ffffffff8e1c9419>] x86_64_start_kernel+0x14d/0x170 [96361.958880] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [96361.959482] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [96361.960073] node 0: slabs: 124, objs: 1984, free: 0 [99301.847630] kworker/dying (27373) used greatest stack depth: 8976 bytes left Ciao, Gerhard ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: Still OOM problems with 4.9er kernels @ 2016-12-09 16:03 ` Gerhard Wiesinger 0 siblings, 0 replies; 88+ messages in thread From: Gerhard Wiesinger @ 2016-12-09 16:03 UTC (permalink / raw) To: Michal Hocko; +Cc: linux-kernel, linux-mm, Linus Torvalds On 09.12.2016 14:40, Michal Hocko wrote: > On Fri 09-12-16 08:06:25, Gerhard Wiesinger wrote: >> Hello, >> >> same with latest kernel rc, dnf still killed with OOM (but sometimes >> better). >> >> ./update.sh: line 40: 1591 Killed ${EXE} update ${PARAMS} >> (does dnf clean all;dnf update) >> Linux database.intern 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 SMP Wed Dec 7 >> 17:53:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >> >> Updated bug report: >> https://bugzilla.redhat.com/show_bug.cgi?id=1314697 > Could you post your oom report please? And another one which ended in a native_safe_halt .... [73366.837826] nmbd: page allocation failure: order:0, mode:0x2280030(GFP_ATOMIC|__GFP_RECLAIMABLE|__GFP_NOTRACK) [73366.837985] CPU: 1 PID: 2005 Comm: nmbd Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [73366.838075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [73366.838175] ffffaa4ac059f548 ffffffff8d4774e3 ffffffff8dc7dd98 0000000000000000 [73366.838272] ffffaa4ac059f5d0 ffffffff8d20e6ea 0228003000000046 ffffffff8dc7dd98 [73366.838364] ffffaa4ac059f570 ffff9c3700000010 ffffaa4ac059f5e0 ffffaa4ac059f590 [73366.838458] Call Trace: [73366.838590] [<ffffffff8d4774e3>] dump_stack+0x86/0xc3 [73366.838680] [<ffffffff8d20e6ea>] warn_alloc+0x13a/0x170 [73366.838762] [<ffffffff8d20e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [73366.838846] [<ffffffff8d0e09a0>] ? finish_task_switch+0xb0/0x260 [73366.838926] [<ffffffff8d20f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [73366.839007] [<ffffffff8d26db51>] alloc_pages_current+0xa1/0x1f0 [73366.839088] [<ffffffff8d068475>] ? kvm_sched_clock_read+0x25/0x40 [73366.839170] [<ffffffff8d278956>] new_slab+0x316/0x7c0 [73366.839245] [<ffffffff8d27ae8b>] ___slab_alloc+0x3fb/0x5c0 [73366.839325] [<ffffffff8d068475>] ? kvm_sched_clock_read+0x25/0x40 [73366.839409] [<ffffffff8d3a4503>] ? __es_insert_extent+0xb3/0x330 [73366.839501] [<ffffffff8d3a4503>] ? __es_insert_extent+0xb3/0x330 [73366.839583] [<ffffffff8d27b0a1>] __slab_alloc+0x51/0x90 [73366.839662] [<ffffffff8d3a4503>] ? __es_insert_extent+0xb3/0x330 [73366.839743] [<ffffffff8d27b326>] kmem_cache_alloc+0x246/0x2d0 [73366.839822] [<ffffffff8d3a5066>] ? __es_remove_extent+0x56/0x2d0 [73366.839906] [<ffffffff8d3a4503>] __es_insert_extent+0xb3/0x330 [73366.839985] [<ffffffff8d3a573e>] ext4_es_insert_extent+0xee/0x280 [73366.840067] [<ffffffff8d35a704>] ? ext4_map_blocks+0x2b4/0x5f0 [73366.840147] [<ffffffff8d35a773>] ext4_map_blocks+0x323/0x5f0 [73366.840225] [<ffffffff8d23dfda>] ? workingset_refault+0x10a/0x220 [73366.840314] [<ffffffff8d3ad7d3>] ext4_mpage_readpages+0x413/0xa60 [73366.840397] [<ffffffff8d201d96>] ? __page_cache_alloc+0x146/0x190 [73366.840487] [<ffffffff8d358235>] ext4_readpages+0x35/0x40 [73366.840569] [<ffffffff8d216d3f>] __do_page_cache_readahead+0x2bf/0x390 [73366.840651] [<ffffffff8d216bea>] ? __do_page_cache_readahead+0x16a/0x390 [73366.840735] [<ffffffff8d20622b>] filemap_fault+0x51b/0x790 [73366.840814] [<ffffffff8d3639ce>] ? ext4_filemap_fault+0x2e/0x50 [73366.840896] [<ffffffff8d3639d9>] ext4_filemap_fault+0x39/0x50 [73366.840976] [<ffffffff8d241ca3>] __do_fault+0x83/0x1d0 [73366.841056] [<ffffffff8d246642>] handle_mm_fault+0x11e2/0x17a0 [73366.841138] [<ffffffff8d2454ba>] ? handle_mm_fault+0x5a/0x17a0 [73366.841220] [<ffffffff8d06de16>] __do_page_fault+0x266/0x520 [73366.841300] [<ffffffff8d06e1a8>] trace_do_page_fault+0x58/0x2a0 [73366.841382] [<ffffffff8d067f3a>] do_async_page_fault+0x1a/0xa0 [73366.841464] [<ffffffff8d918e28>] async_page_fault+0x28/0x30 [73366.842500] Mem-Info: [73366.843149] active_anon:8677 inactive_anon:8798 isolated_anon:0 active_file:328 inactive_file:317 isolated_file:32 unevictable:0 dirty:0 writeback:2 unstable:0 slab_reclaimable:4968 slab_unreclaimable:9242 mapped:365 shmem:1 pagetables:2690 bounce:0 free:764 free_pcp:41 free_cma:0 [73366.846832] Node 0 active_anon:34708kB inactive_anon:35192kB active_file:1312kB inactive_file:1268kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:1460kB dirty:0kB writeback:8kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 4kB writeback_tmp:0kB unstable:0kB pages_scanned:32 all_unreclaimable? no [73366.848711] Node 0 DMA free:1468kB min:172kB low:212kB high:252kB active_anon:3216kB inactive_anon:3448kB active_file:40kB inactive_file:228kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:2064kB slab_unreclaimable:2960kB kernel_stack:100kB pagetables:1536kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [73366.850769] lowmem_reserve[]: 0 116 116 116 116 [73366.851479] Node 0 DMA32 free:1588kB min:1296kB low:1620kB high:1944kB active_anon:31464kB inactive_anon:31740kB active_file:1236kB inactive_file:1056kB unevictable:0kB writepending:0kB present:180080kB managed:139012kB mlocked:0kB slab_reclaimable:17808kB slab_unreclaimable:34008kB kernel_stack:1676kB pagetables:9224kB bounce:0kB free_pcp:164kB local_pcp:12kB free_cma:0kB [73366.853757] lowmem_reserve[]: 0 0 0 0 0 [73366.854544] Node 0 DMA: 13*4kB (H) 13*8kB (H) 17*16kB (H) 12*32kB (H) 8*64kB (H) 1*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1452kB [73366.856200] Node 0 DMA32: 70*4kB (UMH) 12*8kB (MH) 12*16kB (H) 2*32kB (H) 5*64kB (H) 5*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1592kB [73366.857955] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [73366.857956] 2401 total pagecache pages [73366.858829] 1741 pages in swap cache [73366.859721] Swap cache stats: add 1230889, delete 1229148, find 3509739/3747264 [73366.860616] Free swap = 2059496kB [73366.861500] Total swap = 2097148kB [73366.862578] 49018 pages RAM [73366.863560] 0 pages HighMem/MovableOnly [73366.864531] 10288 pages reserved [73366.865436] 0 pages cma reserved [73366.866395] 0 pages hwpoisoned [73366.867503] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [73366.868507] cache: ext4_extent_status, object size: 40, buffer size: 40, default order: 0, min order: 0 [73366.869508] node 0: slabs: 13, objs: 1326, free: 0 [96351.012045] dmcrypt_write: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) [96351.024364] CPU: 0 PID: 1593 Comm: dmcrypt_write Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [96351.027132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [96351.029537] ffffaa4ac025b508 ffffffff8d4774e3 ffffffff8dc7dd98 0000000000000000 [96351.033023] ffffaa4ac025b590 ffffffff8d20e6ea 0228002000000046 ffffffff8dc7dd98 [96351.036214] ffffaa4ac025b530 ffff9c3700000010 ffffaa4ac025b5a0 ffffaa4ac025b550 [96351.039266] Call Trace: [96351.041426] [<ffffffff8d4774e3>] dump_stack+0x86/0xc3 [96351.044511] [<ffffffff8d20e6ea>] warn_alloc+0x13a/0x170 [96351.046698] [<ffffffff8d20e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [96351.048852] [<ffffffff8d20f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [96351.052025] [<ffffffff8d26db51>] alloc_pages_current+0xa1/0x1f0 [96351.053975] [<ffffffff8d916f17>] ? _raw_spin_unlock+0x27/0x40 [96351.055066] [<ffffffff8d278956>] new_slab+0x316/0x7c0 [96351.056134] [<ffffffff8d0ed4c7>] ? sched_clock_cpu+0xa7/0xc0 [96351.057236] [<ffffffff8d27ae8b>] ___slab_alloc+0x3fb/0x5c0 [96351.058287] [<ffffffff8d10b042>] ? cpuacct_charge+0xf2/0x1f0 [96351.059374] [<ffffffffc03a650d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [96351.060439] [<ffffffffc03a650d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [96351.061484] [<ffffffff8d27b0a1>] __slab_alloc+0x51/0x90 [96351.062505] [<ffffffff8d27d141>] __kmalloc+0x251/0x320 [96351.063619] [<ffffffffc03a650d>] ? alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [96351.064624] [<ffffffffc03a650d>] alloc_indirect.isra.11+0x1d/0x50 [virtio_ring] [96351.065631] [<ffffffffc03a7193>] virtqueue_add_sgs+0x1c3/0x4a0 [virtio_ring] [96351.066564] [<ffffffff8d068475>] ? kvm_sched_clock_read+0x25/0x40 [96351.067504] [<ffffffffc041e65c>] __virtblk_add_req+0xbc/0x220 [virtio_blk] [96351.068425] [<ffffffff8d1312fd>] ? debug_lockdep_rcu_enabled+0x1d/0x20 [96351.069351] [<ffffffffc041e935>] ? virtio_queue_rq+0x105/0x290 [virtio_blk] [96351.070229] [<ffffffffc041e95d>] virtio_queue_rq+0x12d/0x290 [virtio_blk] [96351.071059] [<ffffffff8d45015d>] __blk_mq_run_hw_queue+0x26d/0x3b0 [96351.071904] [<ffffffff8d44fecd>] blk_mq_run_hw_queue+0xad/0xd0 [96351.072727] [<ffffffff8d450fca>] blk_mq_insert_requests+0x24a/0x320 [96351.073607] [<ffffffff8d452419>] blk_mq_flush_plug_list+0x139/0x160 [96351.074397] [<ffffffff8d4451d6>] blk_flush_plug_list+0xb6/0x250 [96351.075111] [<ffffffff8d44586c>] blk_finish_plug+0x2c/0x40 [96351.075869] [<ffffffffc03adef0>] dmcrypt_write+0x210/0x220 [dm_crypt] [96351.076613] [<ffffffff8d0e6590>] ? wake_up_q+0x80/0x80 [96351.077376] [<ffffffffc03adce0>] ? crypt_iv_essiv_dtr+0x70/0x70 [dm_crypt] [96351.078086] [<ffffffffc03adce0>] ? crypt_iv_essiv_dtr+0x70/0x70 [dm_crypt] [96351.078892] [<ffffffff8d0d82c2>] kthread+0x102/0x120 [96351.079635] [<ffffffff8d111775>] ? trace_hardirqs_on_caller+0xf5/0x1b0 [96351.080403] [<ffffffff8d0d81c0>] ? kthread_park+0x60/0x60 [96351.081115] [<ffffffff8d917afa>] ret_from_fork+0x2a/0x40 [96351.081882] Mem-Info: [96351.082616] active_anon:8390 inactive_anon:8478 isolated_anon:32 active_file:25 inactive_file:30 isolated_file:0 unevictable:0 dirty:0 writeback:151 unstable:0 slab_reclaimable:5304 slab_unreclaimable:9678 mapped:24 shmem:0 pagetables:3012 bounce:0 free:715 free_pcp:77 free_cma:0 [96351.086909] Node 0 active_anon:33560kB inactive_anon:33912kB active_file:100kB inactive_file:120kB unevictable:0kB isolated(anon):128kB isolated(file):0kB mapped:96kB dirty:0kB writeback:604kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:395 all_unreclaimable? no [96351.089052] Node 0 DMA free:1452kB min:172kB low:212kB high:252kB active_anon:3296kB inactive_anon:3768kB active_file:12kB inactive_file:0kB unevictable:0kB writepending:24kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:2508kB slab_unreclaimable:3108kB kernel_stack:100kB pagetables:1024kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [96351.091383] lowmem_reserve[]: 0 116 116 116 116 [96351.092155] Node 0 DMA32 free:1408kB min:1296kB low:1620kB high:1944kB active_anon:30264kB inactive_anon:30140kB active_file:100kB inactive_file:124kB unevictable:0kB writepending:580kB present:180080kB managed:139012kB mlocked:0kB slab_reclaimable:18708kB slab_unreclaimable:35604kB kernel_stack:1820kB pagetables:11024kB bounce:0kB free_pcp:308kB local_pcp:152kB free_cma:0kB [96351.094895] lowmem_reserve[]: 0 0 0 0 0 [96351.095803] Node 0 DMA: 13*4kB (H) 13*8kB (H) 17*16kB (H) 12*32kB (H) 8*64kB (H) 1*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1452kB [96351.097708] Node 0 DMA32: 16*4kB (H) 6*8kB (H) 5*16kB (H) 8*32kB (H) 5*64kB (H) 5*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB [96351.099672] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [96351.099673] 5020 total pagecache pages [96351.100659] 4961 pages in swap cache [96351.101635] Swap cache stats: add 1422746, delete 1417785, find 4506321/4784232 [96351.102630] Free swap = 1987544kB [96351.103666] Total swap = 2097148kB [96351.104730] 49018 pages RAM [96351.105716] 0 pages HighMem/MovableOnly [96351.106687] 10288 pages reserved [96351.107642] 0 pages cma reserved [96351.108584] 0 pages hwpoisoned [96351.109523] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [96351.110479] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [96351.111428] node 0: slabs: 102, objs: 1632, free: 0 [96361.915109] swapper/0: page allocation failure: order:0, mode:0x2280020(GFP_ATOMIC|__GFP_NOTRACK) [96361.916235] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.0-0.rc8.git2.1.fc26.x86_64 #1 [96361.917276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 [96361.918313] ffff9c3749e03628 ffffffff8d4774e3 ffffffff8dc7dd98 0000000000000000 [96361.919381] ffff9c3749e036b0 ffffffff8d20e6ea 0228002000000046 ffffffff8dc7dd98 [96361.920511] ffff9c3749e03650 ffff9c3700000010 ffff9c3749e036c0 ffff9c3749e03670 [96361.921551] Call Trace: [96361.922549] <IRQ> [96361.922566] [<ffffffff8d4774e3>] dump_stack+0x86/0xc3 [96361.923684] [<ffffffff8d20e6ea>] warn_alloc+0x13a/0x170 [96361.924735] [<ffffffff8d20e9e2>] __alloc_pages_slowpath+0x252/0xbb0 [96361.925757] [<ffffffff8d20f74d>] __alloc_pages_nodemask+0x40d/0x4b0 [96361.926772] [<ffffffff8d26db51>] alloc_pages_current+0xa1/0x1f0 [96361.927844] [<ffffffff8d278956>] new_slab+0x316/0x7c0 [96361.928884] [<ffffffff8d27ae8b>] ___slab_alloc+0x3fb/0x5c0 [96361.929874] [<ffffffff8d79b00b>] ? __alloc_skb+0x5b/0x1e0 [96361.930854] [<ffffffff8d037de9>] ? sched_clock+0x9/0x10 [96361.931829] [<ffffffff8d0ed4c7>] ? sched_clock_cpu+0xa7/0xc0 [96361.932796] [<ffffffff8d79b00b>] ? __alloc_skb+0x5b/0x1e0 [96361.933778] [<ffffffff8d27b0a1>] __slab_alloc+0x51/0x90 [96361.934764] [<ffffffff8d27b7c2>] kmem_cache_alloc_node+0xb2/0x310 [96361.935713] [<ffffffff8d79b00b>] ? __alloc_skb+0x5b/0x1e0 [96361.936637] [<ffffffff8d79b00b>] __alloc_skb+0x5b/0x1e0 [96361.937534] [<ffffffff8d7c2a9f>] __neigh_notify+0x3f/0xd0 [96361.938407] [<ffffffff8d7c64b9>] neigh_update+0x379/0x8b0 [96361.939251] [<ffffffff8d0b7000>] ? __local_bh_enable_ip+0x70/0xc0 [96361.940091] [<ffffffff8d844702>] ? udp4_ufo_fragment+0x122/0x1a0 [96361.940915] [<ffffffff8d845ee8>] arp_process+0x2e8/0x9f0 [96361.941708] [<ffffffff8d846745>] arp_rcv+0x135/0x300 [96361.942473] [<ffffffff8d111ed6>] ? __lock_acquire+0x346/0x1290 [96361.943211] [<ffffffff8d7b42cd>] ? netif_receive_skb_internal+0x6d/0x200 [96361.943942] [<ffffffff8d7b36da>] __netif_receive_skb_core+0x23a/0xd60 [96361.944664] [<ffffffff8d7b42d3>] ? netif_receive_skb_internal+0x73/0x200 [96361.945360] [<ffffffff8d7b4218>] __netif_receive_skb+0x18/0x60 [96361.946045] [<ffffffff8d7b4320>] netif_receive_skb_internal+0xc0/0x200 [96361.946693] [<ffffffff8d7b42d3>] ? netif_receive_skb_internal+0x73/0x200 [96361.947329] [<ffffffff8d7b628c>] napi_gro_receive+0x13c/0x200 [96361.948002] [<ffffffffc04cb667>] virtnet_receive+0x477/0x9a0 [virtio_net] [96361.948690] [<ffffffffc04cbbad>] virtnet_poll+0x1d/0x80 [virtio_net] [96361.949323] [<ffffffff8d7b501e>] net_rx_action+0x23e/0x470 [96361.949994] [<ffffffff8d91a8cd>] __do_softirq+0xcd/0x4b9 [96361.950639] [<ffffffff8d0b7e98>] irq_exit+0x108/0x110 [96361.951275] [<ffffffff8d91a49a>] do_IRQ+0x6a/0x120 [96361.951925] [<ffffffff8d918256>] common_interrupt+0x96/0x96 [96361.952578] <EOI> [96361.952589] [<ffffffff8d916676>] ? native_safe_halt+0x6/0x10 [96361.953245] [<ffffffff8d916235>] default_idle+0x25/0x190 [96361.953912] [<ffffffff8d03918f>] arch_cpu_idle+0xf/0x20 [96361.954562] [<ffffffff8d916873>] default_idle_call+0x23/0x40 [96361.955192] [<ffffffff8d104695>] cpu_startup_entry+0x1d5/0x250 [96361.955864] [<ffffffff8d9066b5>] rest_init+0x135/0x140 [96361.956485] [<ffffffff8e1ca01a>] start_kernel+0x48e/0x4af [96361.957065] [<ffffffff8e1c9120>] ? early_idt_handler_array+0x120/0x120 [96361.957644] [<ffffffff8e1c92ca>] x86_64_start_reservations+0x24/0x26 [96361.958212] [<ffffffff8e1c9419>] x86_64_start_kernel+0x14d/0x170 [96361.958880] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [96361.959482] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [96361.960073] node 0: slabs: 124, objs: 1984, free: 0 [99301.847630] kworker/dying (27373) used greatest stack depth: 8976 bytes left Ciao, Gerhard -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 88+ messages in thread
end of thread, other threads:[~2017-03-26 8:37 UTC | newest] Thread overview: 88+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-11-30 7:10 Still OOM problems with 4.9er kernels Gerhard Wiesinger 2016-11-30 7:20 ` Gerhard Wiesinger 2016-12-09 7:06 ` Gerhard Wiesinger 2016-12-09 13:40 ` Michal Hocko 2016-12-09 13:40 ` Michal Hocko 2016-12-09 15:52 ` Gerhard Wiesinger 2016-12-09 15:52 ` Gerhard Wiesinger 2016-12-09 15:58 ` Gerhard Wiesinger 2016-12-09 15:58 ` Gerhard Wiesinger 2016-12-09 16:09 ` Michal Hocko 2016-12-09 16:09 ` Michal Hocko 2016-12-09 16:58 ` Gerhard Wiesinger 2016-12-09 17:30 ` Michal Hocko 2016-12-09 17:30 ` Michal Hocko 2016-12-09 18:01 ` Gerhard Wiesinger 2016-12-09 18:01 ` Gerhard Wiesinger 2016-12-09 21:42 ` Vlastimil Babka 2016-12-09 21:42 ` Vlastimil Babka 2016-12-10 13:50 ` Gerhard Wiesinger 2016-12-10 13:50 ` Gerhard Wiesinger 2016-12-12 8:24 ` Michal Hocko 2016-12-12 8:24 ` Michal Hocko 2016-12-23 2:55 ` Minchan Kim 2016-12-23 2:55 ` Minchan Kim 2017-01-01 17:20 ` Gerhard Wiesinger 2017-01-01 17:20 ` Gerhard Wiesinger 2017-01-04 8:40 ` Gerhard Wiesinger 2017-01-04 9:11 ` Michal Hocko 2017-01-04 9:11 ` Michal Hocko 2017-02-26 8:40 ` Still OOM problems with 4.9er/4.10er kernels Gerhard Wiesinger 2017-02-27 8:27 ` Michal Hocko 2017-02-27 8:27 ` Michal Hocko 2017-02-28 6:06 ` Gerhard Wiesinger 2017-02-28 6:06 ` Gerhard Wiesinger 2017-02-28 8:14 ` Michal Hocko 2017-02-28 8:14 ` Michal Hocko 2017-02-27 9:02 ` Minchan Kim 2017-02-27 9:02 ` Minchan Kim 2017-02-27 9:44 ` Michal Hocko 2017-02-27 9:44 ` Michal Hocko 2017-02-28 5:17 ` Minchan Kim 2017-02-28 5:17 ` Minchan Kim 2017-02-28 8:12 ` Michal Hocko 2017-02-28 8:12 ` Michal Hocko 2017-03-02 7:17 ` Minchan Kim 2017-03-02 7:17 ` Minchan Kim 2017-03-16 6:38 ` Gerhard Wiesinger 2017-03-16 6:38 ` Gerhard Wiesinger 2017-03-16 8:27 ` Michal Hocko 2017-03-16 8:27 ` Michal Hocko 2017-03-16 8:47 ` lkml 2017-03-16 8:47 ` lkml 2017-03-16 9:08 ` Michal Hocko 2017-03-16 9:08 ` Michal Hocko 2017-03-16 9:23 ` lkml 2017-03-16 9:23 ` lkml 2017-03-16 9:39 ` Michal Hocko 2017-03-16 9:39 ` Michal Hocko 2017-03-17 16:37 ` Gerhard Wiesinger 2017-03-17 16:37 ` Gerhard Wiesinger 2017-03-17 17:13 ` Michal Hocko 2017-03-17 17:13 ` Michal Hocko 2017-03-17 20:08 ` Gerhard Wiesinger 2017-03-17 20:08 ` Gerhard Wiesinger 2017-03-19 8:17 ` Gerhard Wiesinger 2017-03-19 8:17 ` Gerhard Wiesinger 2017-03-20 1:54 ` Tetsuo Handa 2017-03-20 1:54 ` Tetsuo Handa 2017-03-19 15:18 ` Michal Hocko 2017-03-19 15:18 ` Michal Hocko 2017-03-19 16:02 ` Gerhard Wiesinger 2017-03-19 16:02 ` Gerhard Wiesinger 2017-03-20 3:05 ` Mike Galbraith 2017-03-20 3:05 ` Mike Galbraith 2017-03-21 5:59 ` Gerhard Wiesinger 2017-03-21 5:59 ` Gerhard Wiesinger 2017-03-21 7:13 ` Mike Galbraith 2017-03-21 7:13 ` Mike Galbraith 2017-03-23 7:16 ` Gerhard Wiesinger 2017-03-23 7:16 ` Gerhard Wiesinger 2017-03-23 8:38 ` Mike Galbraith 2017-03-23 8:38 ` Mike Galbraith 2017-03-23 14:46 ` Tetsuo Handa 2017-03-23 14:46 ` Tetsuo Handa 2017-03-26 8:36 ` Gerhard Wiesinger 2017-03-26 8:36 ` Gerhard Wiesinger 2016-12-09 16:03 ` Still OOM problems with 4.9er kernels Gerhard Wiesinger 2016-12-09 16:03 ` Gerhard Wiesinger
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.