Re: Regarding LTTng Support for ARM NO-HZ

Re: Regarding LTTng Support for ARM NO-HZ

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 3970 bytes --]

----- Original Message -----

> From: "Hongbo Zhang" <hongbo.zhang@linaro.org>
> To: "mathieu desnoyers" <mathieu.desnoyers@efficios.com>,
> lttng-dev@lists.lttng.org
> Sent: Tuesday, March 18, 2014 4:07:19 AM
> Subject: Regarding LTTng Support for ARM NO-HZ

> Hi Mathieu Desnoyers and all LTTng developers,

Hi, 

> This is Hongbo Zhang, working for Linaro. We have to enable the NO-HZ feature
> for some special case, and this leads to the question whether the LTTng can
> work without kernel ticks on ARM or not.

> I found out some clue that LTTng only supports x86 NO-HZ in an old link:
> http://git.lttng.org/?p=lttv.git;a=blob_plain;f=doc/developer/lttng-lttv-roadmap.html

This is an old LTTng 0.x link, and does not apply to LTTng 2.x. 

> I think the the LTTng doesn't support ARM NO-HZ, right?

While lib ring buffer within lttng-modules has configuration modes to allow 
minimizing the impact on NO_HZ systems, the current lttng-modules 2.x 
is not setup to use those configuration options, so lttng-modules will wakeup 
cpus periodically to flush buffers (when kernel tracing is active). 

> And I am going to add the ARM NO-HZ support to it now, so could you please
> give me some instructions for quick start?

You will probably want to start by looking at the "read timer" option of 
channels, and see how it is handled within lttng-modules, and how it impacts 
the lib ring buffer client configuration with lttng-modules. 

what you probably want is lttng-modules: 

lib/ringbuffer/config.h 

RING_BUFFER_WAKEUP_BY_TIMER 
RING_BUFFER_WAKEUP_BY_WRITER 
(and compare the various wakeup modes) 

However, if you use wakeup by writer, this has impacts on the instrumentation 
coverage (e.g. you then can trigger issues when instrumenting some core scheduler 
code paths, NMIs, etc). So wakeup by timer is really the mode that allows being 
called from the entire kernel code base without risking having side-effects due to 
interaction of the wakeup mechanism with the rest of the kernel. 

> 1. Which part of LTTng relies on kernel ticks? currently I only find the
> ring-bugger by "grep NO*HZ -i -r" command.

ring_buffer_tick_nohz_callback() was a callback meant to be called by the 
in-kernel code that manages no hz. This was implemented when lib ring buffer 
was an in-kernel library (a patchset to the Linux kernel) rather than sitting within 
lttng-modules. 

When it was a in-kernel patchset, we could modify the nohz notifier code from 
the Linux kernel to notify the ring buffer that it enters or exits nohz mode. Unfortunately, 
there is no such notifier in the kernel today (and Thomas Gleixner refused to add one). 

So we need to add this notifier to the Linux kernel if we want to make the "timer-based" 
mode of the lib ring buffer work well on nohz kernels (meaning: don't wake up the CPUs 
that entered idle state needlessly). 

A possible work-around to try is to change the per-cpu timers within lib ring buffer into 
deferrable timers. This should be attempted too. 

Another possible solution would be to implement a global system wide timer rather than 
a per-cpu timer. This would lessen the impact of the timers on idle CPUs, at the expense 
of scalability. The loss of scalability is the reason why I did not implement this solution. 

> 2. How does this part(ring-buffer and maybe others) relies on kernel ticks?
> can you give some explanations or do you have any document of it/them?

> I will look through the codes, but I think some clear explanations can make
> me go to the right way and speed up before reading so many codes without any
> dedicated target.

lttng-modules lib ring buffer is the only part of lttng-modules that depends on timers, 
and therefore is the only part that should interact with NO HZ. 

Hopefully the above explanations shed some light on the topic. 

Please let me know if you have more questions, 

Thanks, 

Mathieu 

> Thanks.

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 8650 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Regarding LTTng Support for ARM NO-HZ

[Hongbo Zhang]

[-- Attachment #1.1: Type: text/plain, Size: 4659 bytes --]

On 21 March 2014 04:21, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>wrote:

> ------------------------------
>
> *From: *"Hongbo Zhang" <hongbo.zhang@linaro.org>
> *To: *"mathieu desnoyers" <mathieu.desnoyers@efficios.com>,
> lttng-dev@lists.lttng.org
> *Sent: *Tuesday, March 18, 2014 4:07:19 AM
> *Subject: *Regarding LTTng Support for ARM NO-HZ
>
>
> Hi Mathieu Desnoyers and all LTTng developers,
>
>
> Hi,
>
> This is Hongbo Zhang, working for Linaro. We have to enable the NO-HZ
> feature for some special case, and this leads to the question whether the
> LTTng can work without kernel ticks on ARM or not.
>
> I found out some clue that LTTng only supports x86 NO-HZ in an old link:
>
> http://git.lttng.org/?p=lttv.git;a=blob_plain;f=doc/developer/lttng-lttv-roadmap.html
>
>
> This is an old LTTng 0.x link, and does not apply to LTTng 2.x.
>
> I think the the LTTng doesn't support ARM NO-HZ, right?
>
>
> While lib ring buffer within lttng-modules has configuration modes to allow
> minimizing the impact on NO_HZ systems, the current lttng-modules 2.x
> is not setup to use those configuration options, so lttng-modules will
> wakeup
> cpus periodically to flush buffers (when kernel tracing is active).
>
> And I am going to add the ARM NO-HZ support to it now, so could you please
> give me some instructions for quick start?
>
>
> You will probably want to start by looking at the "read timer" option of
> channels, and see how it is handled within lttng-modules, and how it
> impacts
> the lib ring buffer client configuration with lttng-modules.
>
> what you probably want is lttng-modules:
>
> lib/ringbuffer/config.h
>
> RING_BUFFER_WAKEUP_BY_TIMER
> RING_BUFFER_WAKEUP_BY_WRITER
> (and compare the various wakeup modes)
>
> However, if you use wakeup by writer, this has impacts on the
> instrumentation
> coverage (e.g. you then can trigger issues when instrumenting some core
> scheduler
> code paths, NMIs, etc). So wakeup by timer is really the mode that allows
> being
> called from the entire kernel code base without risking having
> side-effects due to
> interaction of the wakeup mechanism with the rest of the kernel.
>
>
> 1. Which part of LTTng relies on kernel ticks? currently I only find the
> ring-bugger by "grep NO*HZ -i -r" command.
>
> ring_buffer_tick_nohz_callback() was a callback meant to be called by the
> in-kernel code that manages no hz. This was implemented when lib ring
> buffer
> was an in-kernel library (a patchset to the Linux kernel) rather than
> sitting within
> lttng-modules.
>
> When it was a in-kernel patchset, we could modify the nohz notifier code
> from
> the Linux kernel to notify the ring buffer that it enters or exits nohz
> mode. Unfortunately,
> there is no such notifier in the kernel today (and Thomas Gleixner refused
> to add one).
>
> So we need to add this notifier to the Linux kernel if we want to make the
> "timer-based"
> mode of the lib ring buffer work well on nohz kernels (meaning: don't wake
> up the CPUs
> that entered idle state needlessly).
>
> A possible work-around to try is to change the per-cpu timers within lib
> ring buffer into
> deferrable timers. This should be attempted too.
>
> Another possible solution would be to implement a global system wide timer
> rather than
> a per-cpu timer. This would lessen the impact of the timers on idle CPUs,
> at the expense
> of scalability. The loss of scalability is the reason why I did not
> implement this solution.
>
> 2. How does this part(ring-buffer and maybe others) relies on kernel
> ticks? can you give some explanations or do you have any document of
> it/them?
>
> I will look through the codes, but I think some clear explanations can
> make me go to the right way and speed up before reading so many codes
> without any dedicated target.
>
>
> lttng-modules lib ring buffer is the only part of lttng-modules that
> depends on timers,
> and therefore is the only part that should interact with NO HZ.
>
> Hopefully the above explanations shed some light on the topic.
>
> Please let me know if you have more questions,
>
>
Thank you Mathieu very much, I will look into the codes and raise questions
about further details/solutions if(likely) I have.

But quick questions for now:
I see there was an attempt to unify ring buffer for lttng, ftrace and perf,
but nowadays they are still using their own ring buffer respectively, am I
right? no further effort to unify it any more?

And by the way, do you have any idea about ftrace and perf, do they relies
on kernel ticks?
Thanks.


Thanks,
>
> Mathieu
>
>
> Thanks.
>
>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 8595 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Regarding LTTng Support for ARM NO-HZ

[Hongbo Zhang]

[-- Attachment #1.1: Type: text/plain, Size: 1022 bytes --]

Hi Mathieu Desnoyers and all LTTng developers,
This is Hongbo Zhang, working for Linaro. We have to enable the NO-HZ
feature for some special case, and this leads to the question whether the
LTTng can work without kernel ticks on ARM or not.

I found out some clue that LTTng only supports x86 NO-HZ in an old link:
http://git.lttng.org/?p=lttv.git;a=blob_plain;f=doc/developer/lttng-lttv-roadmap.html

I think the the LTTng doesn't support ARM NO-HZ, right?
And I am going to add the ARM NO-HZ support to it now, so could you please
give me some instructions for quick start?

1. Which part of LTTng relies on kernel ticks? currently I only find the
ring-bugger by "grep NO*HZ -i -r" command.
2. How does this part(ring-buffer and maybe others) relies on kernel ticks?
can you give some explanations or do you have any document of it/them?

I will look through the codes, but I think some clear explanations can make
me go to the right way and speed up before reading so many codes without
any dedicated target.

Thanks.

[-- Attachment #1.2: Type: text/html, Size: 1332 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev