RFC: How to handle function tracing, frame pointers and -mfentry?

RFC: How to handle function tracing, frame pointers and -mfentry?

[Steven Rostedt]

I have code that implements -mfentry for the function tracer.

For those that are unfamiliar with this, it was introduced into gcc in
4.6.0 for x86 only. Used with the -pg option, it replaces mcount with
fentry, and is placed at the first instruction of a function (instead of
after the frame has been built, as mcount does).

So what's my problem?  With -mfentry, function tracer does not depend on
frame pointers. As frame pointers may add overhead, some distros
(Debian) has disabled frame pointers from their kernels and with that,
has also disabled function tracing.

Currently, function tracing selects CONFIG_FRAME_POINTER for various
archs (including x86), as the kernel will not compile without it, if
function tracing is enabled. But if -mfentry is available with the
compiler, it does not have this dependency. The kernel will compile fine
with -pg -mfentry and without frame pointers.

My question is, how do I remove the dependency in kconfig based on the
compiler?

I've tried a few things and here's some ideas:

1) Have kconfig detect if -mfentry is supported with the current
compiler. If it is, then enable a "auto" config called
CONFIG_CC_HAS_FENTRY, and allow function tracer be able to select
FRAME_POINTER if that's not defined.

I actually got this to work, but it only works if the host compiler is
the same as the compiler building the kernel. Which in lots of cases is
not (my default setup does not have this).

2) Remove the select entirely. On build, if the compiler fails because
it does not support -pg running without frame pointers, have a big error
message telling the user they can't have function tracing without frame
pointers and that they either need to enable frame pointers or disable
function tracing.

3) Add frame pointers silently if gcc fails to build with gcc -pg.

I personally do not really care which approach is taken (or a new
approach that I didn't list). I just want to be able to build a kernel
with function tracing and without frame pointers on x86.

Thanks,

-- Steve

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[Steven Rostedt]

On Fri, 2012-04-27 at 14:46 -0400, Steven Rostedt wrote:

> Currently, function tracing selects CONFIG_FRAME_POINTER for various
> archs (including x86), as the kernel will not compile without it, if
> function tracing is enabled. But if -mfentry is available with the
> compiler, it does not have this dependency. The kernel will compile fine
> with -pg -mfentry and without frame pointers.

Just to show the difference. I ran several iterations of hackbench, on
an Intel Quad Core2 2.6GHz.

Here's the fentry code with frame pointers (tracing disabled):

Time: 2.006
Time: 2.028
Time: 2.028
Time: 1.999
Time: 2.035
Time: 2.037
Time: 2.006
Time: 1.996
Time: 2.049
Time: 1.991
Time: 2.038
Time: 2.047
Time: 2.039
Time: 2.000
Time: 2.021
Time: 2.011
Time: 2.007
Time: 2.024
Time: 2.033
Time: 2.027
Time: 2.044


And the fentry code with frame pointers disabled:

Time: 1.870
Time: 1.861
Time: 1.865
Time: 1.884
Time: 1.867
Time: 1.867
Time: 1.875
Time: 1.883
Time: 1.863
Time: 1.877
Time: 1.865
Time: 1.885
Time: 1.842
Time: 1.863
Time: 1.899
Time: 1.877
Time: 1.837
Time: 1.900
Time: 1.897
Time: 1.877
Time: 1.853
Time: 1.856

That's about a 8% difference.

-- Steve

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[Sam Ravnborg]

On Fri, Apr 27, 2012 at 02:46:39PM -0400, Steven Rostedt wrote:
> I have code that implements -mfentry for the function tracer.
> 
> For those that are unfamiliar with this, it was introduced into gcc in
> 4.6.0 for x86 only. Used with the -pg option, it replaces mcount with
> fentry, and is placed at the first instruction of a function (instead of
> after the frame has been built, as mcount does).
> 
> So what's my problem?  With -mfentry, function tracer does not depend on
> frame pointers. As frame pointers may add overhead, some distros
> (Debian) has disabled frame pointers from their kernels and with that,
> has also disabled function tracing.
> 
> Currently, function tracing selects CONFIG_FRAME_POINTER for various
> archs (including x86), as the kernel will not compile without it, if
> function tracing is enabled. But if -mfentry is available with the
> compiler, it does not have this dependency. The kernel will compile fine
> with -pg -mfentry and without frame pointers.
> 
> My question is, how do I remove the dependency in kconfig based on the
> compiler?

Allow the user to select one of:
1) function tracer
2) frame pointer
3) none of the above

This should be trivial to do in Kconfig language.

If "function tracer" is selected then use -mfentry if supported,
with a fallback to frame pointers.

This looks simple IMO and should be easy to implement too.

	Sam

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[H. Peter Anvin]

On 04/27/2012 01:27 PM, Sam Ravnborg wrote:
>>
>> My question is, how do I remove the dependency in kconfig based on the
>> compiler?
> 
> Allow the user to select one of:
> 1) function tracer
> 2) frame pointer
> 3) none of the above
> 
> This should be trivial to do in Kconfig language.
> 
> If "function tracer" is selected then use -mfentry if supported,
> with a fallback to frame pointers.
> 
> This looks simple IMO and should be easy to implement too.
> 

Except it is the wrong thing.  This is not the only user of frame
pointer.  What I think you mean is remove the dependency in Kconfig, but
force the frame pointer enabled if -mfentry is not supported.

	-hpa

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[Steven Rostedt]

On Fri, 2012-04-27 at 13:31 -0700, H. Peter Anvin wrote:
> On 04/27/2012 01:27 PM, Sam Ravnborg wrote:
> >>
> >> My question is, how do I remove the dependency in kconfig based on the
> >> compiler?
> > 
> > Allow the user to select one of:
> > 1) function tracer
> > 2) frame pointer
> > 3) none of the above
> > 
> > This should be trivial to do in Kconfig language.
> > 
> > If "function tracer" is selected then use -mfentry if supported,
> > with a fallback to frame pointers.
> > 
> > This looks simple IMO and should be easy to implement too.
> > 
> 
> Except it is the wrong thing.  This is not the only user of frame
> pointer.  What I think you mean is remove the dependency in Kconfig, but
> force the frame pointer enabled if -mfentry is not supported.

You mean option 3?

3) Add frame pointers silently if gcc fails to build with gcc -pg.

Which I should have added, relies on the same thing as option 2:

2) Remove the select entirely.


-- Steve

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[H. Peter Anvin]

On 04/27/2012 01:57 PM, Steven Rostedt wrote:
>>
>> Except it is the wrong thing.  This is not the only user of frame
>> pointer.  What I think you mean is remove the dependency in Kconfig, but
>> force the frame pointer enabled if -mfentry is not supported.
> 
> You mean option 3?
> 
> 3) Add frame pointers silently if gcc fails to build with gcc -pg.
> 

Yes, your option 3.

	-hypa

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[Steven Rostedt]

On Fri, 2012-04-27 at 14:11 -0700, H. Peter Anvin wrote:
> On 04/27/2012 01:57 PM, Steven Rostedt wrote:
> >>
> >> Except it is the wrong thing.  This is not the only user of frame
> >> pointer.  What I think you mean is remove the dependency in Kconfig, but
> >> force the frame pointer enabled if -mfentry is not supported.
> > 
> > You mean option 3?
> > 
> > 3) Add frame pointers silently if gcc fails to build with gcc -pg.
> > 
> 
> Yes, your option 3.
> 

Actually, I just noticed this in the Makefile:

ifdef CONFIG_FRAME_POINTER
KBUILD_CFLAGS   += -fno-omit-frame-pointer -fno-optimize-sibling-calls
else
# Some targets (ARM with Thumb2, for example), can't be built with frame
# pointers.  For those, we don't have FUNCTION_TRACER automatically
# select FRAME_POINTER.  However, FUNCTION_TRACER adds -pg, and this is
# incompatible with -fomit-frame-pointer with current GCC, so we don't use
# -fomit-frame-pointer with FUNCTION_TRACER.
ifndef CONFIG_FUNCTION_TRACER
KBUILD_CFLAGS   += -fomit-frame-pointer
endif
endif


Because of ARM thumb2, -fomit-frame-pointer is not added if
CONFIG_FRAME_POINTER is not set and FUNCTION_TRACER is. As it is fine to
just use '-pg', which gcc will add frame pointers if needed, and it is
only a problem if both -pg and -fomit-frame-pointer is set. We only need
to remove the select without doing anything else.

If -mfentry is added with -pg, and -fno-omit-frame-pointer is not set,
gcc will not add frame pointers by default.

So it seems that gcc took care of this problem for me :-)

I'll go and make a patch that simply removes the select.

Sorry for the noise ;-)

-- Steve

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[Andi Kleen]

> 1) Have kconfig detect if -mfentry is supported with the current
> compiler. If it is, then enable a "auto" config called
> CONFIG_CC_HAS_FENTRY, and allow function tracer be able to select
> FRAME_POINTER if that's not defined.
> 
> I actually got this to work, but it only works if the host compiler is
> the same as the compiler building the kernel. Which in lots of cases is
> not (my default setup does not have this).

Kconfig just needs to learn how to run the target compiler

I think that's the right direction. Right now our main
Makefiles get polluted more and more with "test compiles", each
of which makes a "null make" slower and slower. 

I just measured and a null compile (nothing changes) of a current
tree calls "gcc" 141 times.

All this stuff should be cached in the Kconfig instead.

It may break some obscure setups (that can be probably fixed without
too much effort), but the development turnaround improvement
for everyone else would be worth it.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[Sam Ravnborg]

> 
> I just measured and a null compile (nothing changes) of a current
> tree calls "gcc" 141 times.
Something is wrong then which must be fixed.
There are 55 uses of cc-option in */x86/*
And only 7 uses of cc-option in the top-level Makefile.

So it looks like we evalute the same assignmet twice or more.

	Sam

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[Sam Ravnborg]

On Sat, Apr 28, 2012 at 10:50:28AM +0200, Sam Ravnborg wrote:
> > 
> > I just measured and a null compile (nothing changes) of a current
> > tree calls "gcc" 141 times.
> Something is wrong then which must be fixed.
> There are 55 uses of cc-option in */x86/*
> And only 7 uses of cc-option in the top-level Makefile.
> 
> So it looks like we evalute the same assignmet twice or more.

"make defconfig" in my box calls gcc 50 times.
This looks much closer to the actual usage of cc-option in the
above mentioned files.
How did you manage to achieve 144 calls to gcc?

Any specific configuration that triggers this?

	Sam

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[Andi Kleen]

> "make defconfig" in my box calls gcc 50 times.
> This looks much closer to the actual usage of cc-option in the
> above mentioned files.
> How did you manage to achieve 144 calls to gcc?
> 
> Any specific configuration that triggers this?

Hmm, thanks for double checking. I suspect I misgrepped
(included the children of the gcc from the strace log). 
If it calls two children that would be roughly your number.

But even 50 is quite bad.

-Andi

Re: RFC: How to handle function tracing, frame pointers and -mfentry?

[H. Peter Anvin]

On 04/28/2012 01:36 AM, Andi Kleen wrote:
> 
> Kconfig just needs to learn how to run the target compiler
> 
> I think that's the right direction. Right now our main
> Makefiles get polluted more and more with "test compiles", each
> of which makes a "null make" slower and slower. 
> 
> I just measured and a null compile (nothing changes) of a current
> tree calls "gcc" 141 times.
> 
> All this stuff should be cached in the Kconfig instead.
> 
> It may break some obscure setups (that can be probably fixed without
> too much effort), but the development turnaround improvement
> for everyone else would be worth it.
> 

I don't think it is a performance issue as much as a utility issue.
When Kconfig can't actually reflect what is built into the resulting
kernel, that is a problem.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.