From: "Odzioba, Lukasz" <lukasz.odzioba@intel.com> To: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org> Cc: "Shutemov, Kirill" <kirill.shutemov@intel.com>, "Hansen, Dave" <dave.hansen@intel.com>, "Anaczkowski, Lukasz" <lukasz.anaczkowski@intel.com> Subject: mm: pages are not freed from lru_add_pvecs after process termination Date: Wed, 27 Apr 2016 17:01:57 +0000 [thread overview] Message-ID: <D6EDEBF1F91015459DB866AC4EE162CC023AEF26@IRSMSX103.ger.corp.intel.com> (raw) Hi, I encounter a problem which I'd like to discuss here (tested on 3.10 and 4.5). While running some workloads we noticed that in case of "improper" application exit (like SIGTERM) quite a bit (a few GBs) of memory is not being reclaimed after process termination. Executing echo 1 > /proc/sys/vm/compact_memory makes the memory available again. This memory is not reclaimed so OOM will kill process trying to allocate memory which technically should be available. Such behavior is present only when THP are [always] enabled. Disabling it makes the issue not visible to the naked eye. An important information is that it is visible mostly due to large amount of CPUs in the system (>200) and amount of missing memory varies with the number of CPUs. This memory seems to not be accounted anywhere, but I was able to found it on per cpu lru_add_pvec lists thanks to Dave Hansen's suggestion. Knowing that I am able to reproduce this problem with much simpler code: //compile with: gcc repro.c -o repro -fopenmp #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/mman.h> #include "omp.h" int main() { #pragma omp parallel { size_t size = 55*1000*1000; // tweaked for 288cpus, "leaks" ~3.5GB unsigned long nodemask = 1; void *p = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS , -1, 0); if(p) memset(p, 0, size); //munmap(p, size); // uncomment to make the problem go away } return 0; } Exemplary execution: $ numactl -H | grep "node 1" | grep MB node 1 size: 16122 MB node 1 free: 16026 MB $ ./repro $ numactl -H | grep "node 1" | grep MB node 1 size: 16122 MB node 1 free: 13527 MB After a couple of minutes on idle system some of this memory is reclaimed, but never all unless I run tasks on every CPU: node 1 size: 16122 MB node 1 free: 14823 MB Pieces of the puzzle: A) after process termination memory is not getting freed nor accounted as free B) memory cannot be allocated by other processes (unless it is allocated by all CPUs) I am not sure whether it is expected behavior or a side effect of something else not going as it should. Temporarily I added lru_add_drain_all() to try_to_free_pages() which sort of hammers B case, but A is still present. I am not familiar with this code, but I feel like draining lru_add work should be split into smaller pieces and done by kswapd to fix A and drain only as much pages as needed in try_to_free_pages to fix B. Any comments/ideas/patches for a proper fix are welcome. Thanks, Lukas
WARNING: multiple messages have this Message-ID (diff)
From: "Odzioba, Lukasz" <lukasz.odzioba@intel.com> To: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org> Cc: "Shutemov, Kirill" <kirill.shutemov@intel.com>, "Hansen, Dave" <dave.hansen@intel.com>, "Anaczkowski, Lukasz" <lukasz.anaczkowski@intel.com> Subject: mm: pages are not freed from lru_add_pvecs after process termination Date: Wed, 27 Apr 2016 17:01:57 +0000 [thread overview] Message-ID: <D6EDEBF1F91015459DB866AC4EE162CC023AEF26@IRSMSX103.ger.corp.intel.com> (raw) Hi, I encounter a problem which I'd like to discuss here (tested on 3.10 and 4.5). While running some workloads we noticed that in case of "improper" application exit (like SIGTERM) quite a bit (a few GBs) of memory is not being reclaimed after process termination. Executing echo 1 > /proc/sys/vm/compact_memory makes the memory available again. This memory is not reclaimed so OOM will kill process trying to allocate memory which technically should be available. Such behavior is present only when THP are [always] enabled. Disabling it makes the issue not visible to the naked eye. An important information is that it is visible mostly due to large amount of CPUs in the system (>200) and amount of missing memory varies with the number of CPUs. This memory seems to not be accounted anywhere, but I was able to found it on per cpu lru_add_pvec lists thanks to Dave Hansen's suggestion. Knowing that I am able to reproduce this problem with much simpler code: //compile with: gcc repro.c -o repro -fopenmp #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/mman.h> #include "omp.h" int main() { #pragma omp parallel { size_t size = 55*1000*1000; // tweaked for 288cpus, "leaks" ~3.5GB unsigned long nodemask = 1; void *p = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS , -1, 0); if(p) memset(p, 0, size); //munmap(p, size); // uncomment to make the problem go away } return 0; } Exemplary execution: $ numactl -H | grep "node 1" | grep MB node 1 size: 16122 MB node 1 free: 16026 MB $ ./repro $ numactl -H | grep "node 1" | grep MB node 1 size: 16122 MB node 1 free: 13527 MB After a couple of minutes on idle system some of this memory is reclaimed, but never all unless I run tasks on every CPU: node 1 size: 16122 MB node 1 free: 14823 MB Pieces of the puzzle: A) after process termination memory is not getting freed nor accounted as free B) memory cannot be allocated by other processes (unless it is allocated by all CPUs) I am not sure whether it is expected behavior or a side effect of something else not going as it should. Temporarily I added lru_add_drain_all() to try_to_free_pages() which sort of hammers B case, but A is still present. I am not familiar with this code, but I feel like draining lru_add work should be split into smaller pieces and done by kswapd to fix A and drain only as much pages as needed in try_to_free_pages to fix B. Any comments/ideas/patches for a proper fix are welcome. Thanks, Lukas -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next reply other threads:[~2016-04-27 17:02 UTC|newest] Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-04-27 17:01 Odzioba, Lukasz [this message] 2016-04-27 17:01 ` mm: pages are not freed from lru_add_pvecs after process termination Odzioba, Lukasz 2016-04-27 17:11 ` Dave Hansen 2016-04-27 17:11 ` Dave Hansen 2016-04-28 14:37 ` Michal Hocko 2016-04-28 14:37 ` Michal Hocko 2016-05-02 13:00 ` Michal Hocko 2016-05-02 13:00 ` Michal Hocko 2016-05-04 19:41 ` Odzioba, Lukasz 2016-05-04 19:41 ` Odzioba, Lukasz 2016-05-04 20:16 ` Dave Hansen 2016-05-04 20:16 ` Dave Hansen 2016-05-04 20:36 ` Michal Hocko 2016-05-04 20:36 ` Michal Hocko 2016-05-05 7:21 ` Michal Hocko 2016-05-05 7:21 ` Michal Hocko 2016-05-05 17:25 ` Odzioba, Lukasz 2016-05-05 17:25 ` Odzioba, Lukasz 2016-05-11 7:38 ` Michal Hocko 2016-05-11 7:38 ` Michal Hocko 2016-05-06 15:10 ` Odzioba, Lukasz 2016-05-06 15:10 ` Odzioba, Lukasz 2016-05-06 16:04 ` Dave Hansen 2016-05-06 16:04 ` Dave Hansen 2016-05-11 7:53 ` Michal Hocko 2016-05-11 7:53 ` Michal Hocko 2016-05-13 11:29 ` Vlastimil Babka 2016-05-13 11:29 ` Vlastimil Babka 2016-05-13 12:05 ` Odzioba, Lukasz 2016-05-13 12:05 ` Odzioba, Lukasz 2016-06-07 9:02 ` Odzioba, Lukasz 2016-06-07 9:02 ` Odzioba, Lukasz 2016-06-07 11:19 ` Michal Hocko 2016-06-07 11:19 ` Michal Hocko 2016-06-08 8:51 ` Odzioba, Lukasz 2016-06-08 8:51 ` Odzioba, Lukasz 2016-05-02 14:39 ` Vlastimil Babka 2016-05-02 14:39 ` Vlastimil Babka 2016-05-02 15:01 ` Kirill A. Shutemov 2016-05-02 15:01 ` Kirill A. Shutemov 2016-05-02 15:13 ` Vlastimil Babka 2016-05-02 15:13 ` Vlastimil Babka 2016-05-02 15:49 ` Dave Hansen 2016-05-02 15:49 ` Dave Hansen 2016-05-02 16:02 ` Kirill A. Shutemov 2016-05-02 16:02 ` Kirill A. Shutemov 2016-05-03 7:37 ` Michal Hocko 2016-05-03 7:37 ` Michal Hocko 2016-05-03 10:07 ` Kirill A. Shutemov 2016-05-03 10:07 ` Kirill A. Shutemov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=D6EDEBF1F91015459DB866AC4EE162CC023AEF26@IRSMSX103.ger.corp.intel.com \ --to=lukasz.odzioba@intel.com \ --cc=dave.hansen@intel.com \ --cc=kirill.shutemov@intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=lukasz.anaczkowski@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.