* [RELEASE] LTTng-modules 2.9.11, 2.10.8, 2.11.0-rc2 (Linux kernel tracer)
@ 2018-11-01 22:56 Mathieu Desnoyers
2018-11-01 23:33 ` [diamon-discuss] " Joel Fernandes
0 siblings, 1 reply; 6+ messages in thread
From: Mathieu Desnoyers @ 2018-11-01 22:56 UTC (permalink / raw)
To: lttng-dev, diamon-discuss; +Cc: linux-kernel
Hi,
This is a set of bugfix releases of the LTTng modules kernel tracer.
It covers the three currently active lttng-modules branches: the
2.9 and 2.10 stable branches, as well as the 2.11 branch in release
candidate cycle.
Those releases add support for kernel 4.19.
One important improvement is to prevent allocation of buffers larger
than the available memory, which can cause the OOM killer to trigger.
Even if the OOM killer end up having to trigger, the current OOM kill
target is set to the current thread while allocating buffers.
Enjoy!
Mathieu
Project website: http://lttng.org
Documentation: http://lttng.org/docs
Download link: http://lttng.org/download
Changelog:
2018-11-01 (National Men Make Dinner Day) LTTng modules 2.11.0-rc2
* Fix: update kvm instrumentation for SLES12 SP2 LTSS >= 4.4.121-92.92
* Fix: Add missing const to lttng_tracepoint_ptr_deref prototype
* Fix: adapt to kernel relative references
* Fix: implicit declarations caused by buffer size checks.
* Prevent allocation of buffers if exceeding available memory
* Fix: btrfs instrumentation namespacing
* Fix: Convert rcu tracepointis to gp_seq (v4.19)
* Fix: tracing: Centralize preemptirq tracepoints (4.19)
* Fix: net: expose sk wmem in sock_exceed_buf_limit tracepoint (4.19)
* Fix: access migrate_disable field directly
* Fix: out of memory error handling
2018-11-01 (National Men Make Dinner Day) LTTng modules 2.10.8
* Fix: update kvm instrumentation for SLES12 SP2 LTSS >= 4.4.121-92.92
* Fix: Add missing const to lttng_tracepoint_ptr_deref prototype
* Fix: adapt to kernel relative references
* Fix: implicit declarations caused by buffer size checks.
* Prevent allocation of buffers if exceeding available memory
* Fix: Convert rcu tracepointis to gp_seq (v4.19)
* Fix: net: expose sk wmem in sock_exceed_buf_limit tracepoint (4.19)
* Fix: access migrate_disable field directly
* Fix: out of memory error handling
2018-11-01 (National Men Make Dinner Day) LTTng modules 2.9.11
* Fix: update kvm instrumentation for SLES12 SP2 LTSS >= 4.4.121-92.92
* Fix: Add missing const to lttng_tracepoint_ptr_deref prototype
* Fix: adapt to kernel relative references
* Fix: implicit declarations caused by buffer size checks.
* Prevent allocation of buffers if exceeding available memory
* Fix: Convert rcu tracepointis to gp_seq (v4.19)
* Fix: net: expose sk wmem in sock_exceed_buf_limit tracepoint (4.19)
* Fix: access migrate_disable field directly
* Fix: out of memory error handling
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [diamon-discuss] [RELEASE] LTTng-modules 2.9.11, 2.10.8, 2.11.0-rc2 (Linux kernel tracer) 2018-11-01 22:56 [RELEASE] LTTng-modules 2.9.11, 2.10.8, 2.11.0-rc2 (Linux kernel tracer) Mathieu Desnoyers @ 2018-11-01 23:33 ` Joel Fernandes 2019-03-19 17:34 ` Mathieu Desnoyers 0 siblings, 1 reply; 6+ messages in thread From: Joel Fernandes @ 2018-11-01 23:33 UTC (permalink / raw) To: Mathieu Desnoyers; +Cc: lttng-dev, diamon-discuss, LKML On Thu, Nov 1, 2018 at 3:56 PM Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: > > Hi, > > This is a set of bugfix releases of the LTTng modules kernel tracer. > It covers the three currently active lttng-modules branches: the > 2.9 and 2.10 stable branches, as well as the 2.11 branch in release > candidate cycle. > > Those releases add support for kernel 4.19. > > One important improvement is to prevent allocation of buffers larger > than the available memory, which can cause the OOM killer to trigger. > Even if the OOM killer end up having to trigger, the current OOM kill > target is set to the current thread while allocating buffers. This is interesting. Me and Steve were looking at exactly this issue with the ftrace ring buffer a few months ago. Turns out that even setting the OOM kill target may not be enough to prevent all OOMs. I don't remember the reason why not, I'll have to dig out those threads but that's what the -mm folks said at the time. I did remember vaguely that I tested it and the kill target doesn't always get killed.. its possible that something *other* parallel allocation can be victimized AFAIR, even though the culprit is the kill target. - Joel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] [RELEASE] LTTng-modules 2.9.11, 2.10.8, 2.11.0-rc2 (Linux kernel tracer) 2018-11-01 23:33 ` [diamon-discuss] " Joel Fernandes @ 2019-03-19 17:34 ` Mathieu Desnoyers 2019-03-21 12:41 ` Joel Fernandes 0 siblings, 1 reply; 6+ messages in thread From: Mathieu Desnoyers @ 2019-03-19 17:34 UTC (permalink / raw) To: Joel Fernandes; +Cc: diamon-discuss, lttng-dev, linux-kernel ----- On Nov 1, 2018, at 7:33 PM, Joel Fernandes via diamon-discuss diamon-discuss@lists.linuxfoundation.org wrote: > On Thu, Nov 1, 2018 at 3:56 PM Mathieu Desnoyers > <mathieu.desnoyers@efficios.com> wrote: >> >> Hi, >> >> This is a set of bugfix releases of the LTTng modules kernel tracer. >> It covers the three currently active lttng-modules branches: the >> 2.9 and 2.10 stable branches, as well as the 2.11 branch in release >> candidate cycle. >> >> Those releases add support for kernel 4.19. >> >> One important improvement is to prevent allocation of buffers larger >> than the available memory, which can cause the OOM killer to trigger. >> Even if the OOM killer end up having to trigger, the current OOM kill >> target is set to the current thread while allocating buffers. > > This is interesting. Me and Steve were looking at exactly this issue > with the ftrace ring buffer a few months ago. Turns out that even > setting the OOM kill target may not be enough to prevent all OOMs. I > don't remember the reason why not, I'll have to dig out those threads > but that's what the -mm folks said at the time. I did remember vaguely > that I tested it and the kill target doesn't always get killed.. its > possible that something *other* parallel allocation can be victimized > AFAIR, even though the culprit is the kill target. > Hi Joel, Sorry for the late reply. Thanks for your input! Here is a description of the solution we implemented: " Get an estimate of the number of available pages and return ENOMEM if there are not enough pages to cover the needs of the caller. Also, mark the calling user thread as the first target for the OOM killer in case the estimate of available pages was wrong. This greatly reduces the attack surface of this issue as well as reducing its potential impact. This approach is inspired by the one taken by the Linux kernel trace ring buffer[1]." This is implemented in commit 1f0ab1eb040 "Prevent allocation of buffers if exceeding available memory" within lttng-modules. Are you aware of another way to achieve this that would prevent the incorrect OOM victimization scenario you describe above ? Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] [RELEASE] LTTng-modules 2.9.11, 2.10.8, 2.11.0-rc2 (Linux kernel tracer) 2019-03-19 17:34 ` Mathieu Desnoyers @ 2019-03-21 12:41 ` Joel Fernandes 2019-03-21 13:13 ` Steven Rostedt 0 siblings, 1 reply; 6+ messages in thread From: Joel Fernandes @ 2019-03-21 12:41 UTC (permalink / raw) To: Mathieu Desnoyers; +Cc: diamon-discuss, lttng-dev, linux-kernel, rostedt On Tue, Mar 19, 2019 at 01:34:34PM -0400, Mathieu Desnoyers wrote: > ----- On Nov 1, 2018, at 7:33 PM, Joel Fernandes via diamon-discuss diamon-discuss@lists.linuxfoundation.org wrote: > > > On Thu, Nov 1, 2018 at 3:56 PM Mathieu Desnoyers > > <mathieu.desnoyers@efficios.com> wrote: > >> > >> Hi, > >> > >> This is a set of bugfix releases of the LTTng modules kernel tracer. > >> It covers the three currently active lttng-modules branches: the > >> 2.9 and 2.10 stable branches, as well as the 2.11 branch in release > >> candidate cycle. > >> > >> Those releases add support for kernel 4.19. > >> > >> One important improvement is to prevent allocation of buffers larger > >> than the available memory, which can cause the OOM killer to trigger. > >> Even if the OOM killer end up having to trigger, the current OOM kill > >> target is set to the current thread while allocating buffers. > > > > This is interesting. Me and Steve were looking at exactly this issue > > with the ftrace ring buffer a few months ago. Turns out that even > > setting the OOM kill target may not be enough to prevent all OOMs. I > > don't remember the reason why not, I'll have to dig out those threads > > but that's what the -mm folks said at the time. I did remember vaguely > > that I tested it and the kill target doesn't always get killed.. its > > possible that something *other* parallel allocation can be victimized > > AFAIR, even though the culprit is the kill target. > > > > Hi Joel, > > Sorry for the late reply. Thanks for your input! No problem, thanks for the reply :) > Here is a description of the solution we implemented: > > " Get an estimate of the number of available pages and return ENOMEM if > there are not enough pages to cover the needs of the caller. Also, mark > the calling user thread as the first target for the OOM killer in case > the estimate of available pages was wrong. > > This greatly reduces the attack surface of this issue as well as reducing > its potential impact. > > This approach is inspired by the one taken by the Linux kernel > trace ring buffer[1]." > > This is implemented in commit 1f0ab1eb040 "Prevent allocation of buffers if exceeding available memory" > within lttng-modules. > > Are you aware of another way to achieve this that would prevent the incorrect > OOM victimization scenario you describe above ? Adding Steve as well. As far as I can see, lttng does exactly the same thing ftrace does, however ftrace also does allocations with __GFP_RETRY_MAYFAIL. I think you want to do that too. That will prevent the ring buffer allocation from being the source of the OOM trigger. However, if OOM is triggered due to parallel allocations from other sources, then the ring-buffer allocation will still be killed even though it is not the only source of the large allocation. That is the best we came up with. Also does it also make sense for lttng ring buffer to use the ftrace code for ring buffer, or make the ftrace ring buffer better and have lttng use it? Or is the lttng ring buffer design too radically different? thanks, - Joel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] [RELEASE] LTTng-modules 2.9.11, 2.10.8, 2.11.0-rc2 (Linux kernel tracer) 2019-03-21 12:41 ` Joel Fernandes @ 2019-03-21 13:13 ` Steven Rostedt 2019-03-22 13:48 ` Joel Fernandes 0 siblings, 1 reply; 6+ messages in thread From: Steven Rostedt @ 2019-03-21 13:13 UTC (permalink / raw) To: Joel Fernandes; +Cc: Mathieu Desnoyers, diamon-discuss, lttng-dev, linux-kernel On Thu, 21 Mar 2019 08:41:22 -0400 Joel Fernandes <joel@joelfernandes.org> wrote: > Also does it also make sense for lttng ring buffer to use the ftrace code for > ring buffer, or make the ftrace ring buffer better and have lttng use it? Or > is the lttng ring buffer design too radically different? We tried in the past and never actually got something that we both could agree on. Now I believe the user space tools depend on the way each one is, so to change one will break the tools that read it. -- Steve ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] [RELEASE] LTTng-modules 2.9.11, 2.10.8, 2.11.0-rc2 (Linux kernel tracer) 2019-03-21 13:13 ` Steven Rostedt @ 2019-03-22 13:48 ` Joel Fernandes 0 siblings, 0 replies; 6+ messages in thread From: Joel Fernandes @ 2019-03-22 13:48 UTC (permalink / raw) To: Steven Rostedt; +Cc: Mathieu Desnoyers, diamon-discuss, lttng-dev, linux-kernel On Thu, Mar 21, 2019 at 09:13:56AM -0400, Steven Rostedt wrote: > On Thu, 21 Mar 2019 08:41:22 -0400 > Joel Fernandes <joel@joelfernandes.org> wrote: > > > Also does it also make sense for lttng ring buffer to use the ftrace code for > > ring buffer, or make the ftrace ring buffer better and have lttng use it? Or > > is the lttng ring buffer design too radically different? > > We tried in the past and never actually got something that we both > could agree on. Now I believe the user space tools depend on the way > each one is, so to change one will break the tools that read it. I understand :-\ thanks, - Joel ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-03-22 13:48 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-11-01 22:56 [RELEASE] LTTng-modules 2.9.11, 2.10.8, 2.11.0-rc2 (Linux kernel tracer) Mathieu Desnoyers 2018-11-01 23:33 ` [diamon-discuss] " Joel Fernandes 2019-03-19 17:34 ` Mathieu Desnoyers 2019-03-21 12:41 ` Joel Fernandes 2019-03-21 13:13 ` Steven Rostedt 2019-03-22 13:48 ` Joel Fernandes
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).