* [PATCH] tracing/ring_buffer: Try harder to allocate
@ 2017-07-11 6:05 Joel Fernandes
2017-07-11 6:12 ` Michal Hocko
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Joel Fernandes @ 2017-07-11 6:05 UTC (permalink / raw)
To: linux-kernel
Cc: kernel-team, linux-mm, Joel Fernandes, Alexander Duyck,
Mel Gorman, Hao Lee, Vladimir Davydov, Johannes Weiner,
Joonsoo Kim, Michal Hocko, Tim Murray, Ingo Molnar,
Steven Rostedt, stable
ftrace can fail to allocate per-CPU ring buffer on systems with a large
number of CPUs coupled while large amounts of cache happening in the
page cache. Currently the ring buffer allocation doesn't retry in the VM
implementation even if direct-reclaim made some progress but still
wasn't able to find a free page. On retrying I see that the allocations
almost always succeed. The retry doesn't happen because __GFP_NORETRY is
used in the tracer to prevent the case where we might OOM, however if we
drop __GFP_NORETRY, we risk destabilizing the system if OOM killer is
triggered. To prevent this situation, use the __GFP_RETRY_MAYFAIL flag
introduced recently [1].
Tested the following still succeeds without destabilizing a system with
1GB memory.
echo 300000 > /sys/kernel/debug/tracing/buffer_size_kb
[1] https://marc.info/?l=linux-mm&m=149820805124906&w=2
Cc: Alexander Duyck <alexander.h.duyck@intel.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Hao Lee <haolee.swjtu@gmail.com>
Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Tim Murray <timmurray@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: stable@vger.kernel.org
Signed-off-by: Joel Fernandes <joelaf@google.com>
---
kernel/trace/ring_buffer.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 4ae268e687fe..529cc50d7243 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -1136,12 +1136,12 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)
for (i = 0; i < nr_pages; i++) {
struct page *page;
/*
- * __GFP_NORETRY flag makes sure that the allocation fails
- * gracefully without invoking oom-killer and the system is
- * not destabilized.
+ * __GFP_RETRY_MAYFAIL flag makes sure that the allocation fails
+ * gracefully without invoking oom-killer and the system is not
+ * destabilized.
*/
bpage = kzalloc_node(ALIGN(sizeof(*bpage), cache_line_size()),
- GFP_KERNEL | __GFP_NORETRY,
+ GFP_KERNEL | __GFP_RETRY_MAYFAIL,
cpu_to_node(cpu));
if (!bpage)
goto free_pages;
@@ -1149,7 +1149,7 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)
list_add(&bpage->list, pages);
page = alloc_pages_node(cpu_to_node(cpu),
- GFP_KERNEL | __GFP_NORETRY, 0);
+ GFP_KERNEL | __GFP_RETRY_MAYFAIL, 0);
if (!page)
goto free_pages;
bpage->page = page_address(page);
--
2.13.2.725.g09c95d1e9-goog
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] tracing/ring_buffer: Try harder to allocate
2017-07-11 6:05 [PATCH] tracing/ring_buffer: Try harder to allocate Joel Fernandes
@ 2017-07-11 6:12 ` Michal Hocko
2017-07-11 6:17 ` Vlastimil Babka
2017-07-11 17:22 ` Johannes Weiner
2 siblings, 0 replies; 4+ messages in thread
From: Michal Hocko @ 2017-07-11 6:12 UTC (permalink / raw)
To: Joel Fernandes
Cc: linux-kernel, kernel-team, linux-mm, Alexander Duyck, Mel Gorman,
Hao Lee, Vladimir Davydov, Johannes Weiner, Joonsoo Kim,
Tim Murray, Ingo Molnar, Steven Rostedt, stable
On Mon 10-07-17 23:05:00, Joel Fernandes wrote:
> ftrace can fail to allocate per-CPU ring buffer on systems with a large
> number of CPUs coupled while large amounts of cache happening in the
> page cache. Currently the ring buffer allocation doesn't retry in the VM
> implementation even if direct-reclaim made some progress but still
> wasn't able to find a free page. On retrying I see that the allocations
> almost always succeed. The retry doesn't happen because __GFP_NORETRY is
> used in the tracer to prevent the case where we might OOM, however if we
> drop __GFP_NORETRY, we risk destabilizing the system if OOM killer is
> triggered. To prevent this situation, use the __GFP_RETRY_MAYFAIL flag
> introduced recently [1].
>
> Tested the following still succeeds without destabilizing a system with
> 1GB memory.
> echo 300000 > /sys/kernel/debug/tracing/buffer_size_kb
>
> [1] https://marc.info/?l=linux-mm&m=149820805124906&w=2
Yes this is the correct usage of the new flag.
> Cc: Alexander Duyck <alexander.h.duyck@intel.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Hao Lee <haolee.swjtu@gmail.com>
> Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Tim Murray <timmurray@google.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: stable@vger.kernel.org
I do not think stable tag is appropriate. The new flag hasn't been
merged yet and it is not a stable material.
> Signed-off-by: Joel Fernandes <joelaf@google.com>
Feel free to add
Acked-by: Michal Hocko <mhocko@suse.com>
> ---
> kernel/trace/ring_buffer.c | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
> index 4ae268e687fe..529cc50d7243 100644
> --- a/kernel/trace/ring_buffer.c
> +++ b/kernel/trace/ring_buffer.c
> @@ -1136,12 +1136,12 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)
> for (i = 0; i < nr_pages; i++) {
> struct page *page;
> /*
> - * __GFP_NORETRY flag makes sure that the allocation fails
> - * gracefully without invoking oom-killer and the system is
> - * not destabilized.
> + * __GFP_RETRY_MAYFAIL flag makes sure that the allocation fails
> + * gracefully without invoking oom-killer and the system is not
> + * destabilized.
> */
> bpage = kzalloc_node(ALIGN(sizeof(*bpage), cache_line_size()),
> - GFP_KERNEL | __GFP_NORETRY,
> + GFP_KERNEL | __GFP_RETRY_MAYFAIL,
> cpu_to_node(cpu));
> if (!bpage)
> goto free_pages;
> @@ -1149,7 +1149,7 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)
> list_add(&bpage->list, pages);
>
> page = alloc_pages_node(cpu_to_node(cpu),
> - GFP_KERNEL | __GFP_NORETRY, 0);
> + GFP_KERNEL | __GFP_RETRY_MAYFAIL, 0);
> if (!page)
> goto free_pages;
> bpage->page = page_address(page);
> --
> 2.13.2.725.g09c95d1e9-goog
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] tracing/ring_buffer: Try harder to allocate
2017-07-11 6:05 [PATCH] tracing/ring_buffer: Try harder to allocate Joel Fernandes
2017-07-11 6:12 ` Michal Hocko
@ 2017-07-11 6:17 ` Vlastimil Babka
2017-07-11 17:22 ` Johannes Weiner
2 siblings, 0 replies; 4+ messages in thread
From: Vlastimil Babka @ 2017-07-11 6:17 UTC (permalink / raw)
To: Joel Fernandes, linux-kernel
Cc: kernel-team, linux-mm, Alexander Duyck, Mel Gorman, Hao Lee,
Vladimir Davydov, Johannes Weiner, Joonsoo Kim, Michal Hocko,
Tim Murray, Ingo Molnar, Steven Rostedt, stable
On 07/11/2017 08:05 AM, Joel Fernandes wrote:
> ftrace can fail to allocate per-CPU ring buffer on systems with a large
> number of CPUs coupled while large amounts of cache happening in the
> page cache. Currently the ring buffer allocation doesn't retry in the VM
> implementation even if direct-reclaim made some progress but still
> wasn't able to find a free page. On retrying I see that the allocations
> almost always succeed. The retry doesn't happen because __GFP_NORETRY is
> used in the tracer to prevent the case where we might OOM, however if we
> drop __GFP_NORETRY, we risk destabilizing the system if OOM killer is
> triggered. To prevent this situation, use the __GFP_RETRY_MAYFAIL flag
> introduced recently [1].
>
> Tested the following still succeeds without destabilizing a system with
> 1GB memory.
> echo 300000 > /sys/kernel/debug/tracing/buffer_size_kb
>
> [1] https://marc.info/?l=linux-mm&m=149820805124906&w=2
>
> Cc: Alexander Duyck <alexander.h.duyck@intel.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Hao Lee <haolee.swjtu@gmail.com>
> Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Tim Murray <timmurray@google.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: stable@vger.kernel.org
Not stable, as Michal mentioned.
Acked-by: Vlastimil Babka <vbabka@suse.cz>
> Signed-off-by: Joel Fernandes <joelaf@google.com>
> ---
> kernel/trace/ring_buffer.c | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
> index 4ae268e687fe..529cc50d7243 100644
> --- a/kernel/trace/ring_buffer.c
> +++ b/kernel/trace/ring_buffer.c
> @@ -1136,12 +1136,12 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)
> for (i = 0; i < nr_pages; i++) {
> struct page *page;
> /*
> - * __GFP_NORETRY flag makes sure that the allocation fails
> - * gracefully without invoking oom-killer and the system is
> - * not destabilized.
> + * __GFP_RETRY_MAYFAIL flag makes sure that the allocation fails
> + * gracefully without invoking oom-killer and the system is not
> + * destabilized.
> */
> bpage = kzalloc_node(ALIGN(sizeof(*bpage), cache_line_size()),
> - GFP_KERNEL | __GFP_NORETRY,
> + GFP_KERNEL | __GFP_RETRY_MAYFAIL,
> cpu_to_node(cpu));
> if (!bpage)
> goto free_pages;
> @@ -1149,7 +1149,7 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)
> list_add(&bpage->list, pages);
>
> page = alloc_pages_node(cpu_to_node(cpu),
> - GFP_KERNEL | __GFP_NORETRY, 0);
> + GFP_KERNEL | __GFP_RETRY_MAYFAIL, 0);
> if (!page)
> goto free_pages;
> bpage->page = page_address(page);
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] tracing/ring_buffer: Try harder to allocate
2017-07-11 6:05 [PATCH] tracing/ring_buffer: Try harder to allocate Joel Fernandes
2017-07-11 6:12 ` Michal Hocko
2017-07-11 6:17 ` Vlastimil Babka
@ 2017-07-11 17:22 ` Johannes Weiner
2 siblings, 0 replies; 4+ messages in thread
From: Johannes Weiner @ 2017-07-11 17:22 UTC (permalink / raw)
To: Joel Fernandes
Cc: linux-kernel, kernel-team, linux-mm, Alexander Duyck, Mel Gorman,
Hao Lee, Vladimir Davydov, Joonsoo Kim, Michal Hocko, Tim Murray,
Ingo Molnar, Steven Rostedt, stable
On Mon, Jul 10, 2017 at 11:05:00PM -0700, Joel Fernandes wrote:
> ftrace can fail to allocate per-CPU ring buffer on systems with a large
> number of CPUs coupled while large amounts of cache happening in the
> page cache. Currently the ring buffer allocation doesn't retry in the VM
> implementation even if direct-reclaim made some progress but still
> wasn't able to find a free page. On retrying I see that the allocations
> almost always succeed. The retry doesn't happen because __GFP_NORETRY is
> used in the tracer to prevent the case where we might OOM, however if we
> drop __GFP_NORETRY, we risk destabilizing the system if OOM killer is
> triggered. To prevent this situation, use the __GFP_RETRY_MAYFAIL flag
> introduced recently [1].
>
> Tested the following still succeeds without destabilizing a system with
> 1GB memory.
> echo 300000 > /sys/kernel/debug/tracing/buffer_size_kb
>
> [1] https://marc.info/?l=linux-mm&m=149820805124906&w=2
>
> Cc: Alexander Duyck <alexander.h.duyck@intel.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Hao Lee <haolee.swjtu@gmail.com>
> Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Tim Murray <timmurray@google.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: stable@vger.kernel.org
> Signed-off-by: Joel Fernandes <joelaf@google.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-07-11 17:22 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-11 6:05 [PATCH] tracing/ring_buffer: Try harder to allocate Joel Fernandes
2017-07-11 6:12 ` Michal Hocko
2017-07-11 6:17 ` Vlastimil Babka
2017-07-11 17:22 ` Johannes Weiner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).