Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

[Marc MERLIN]

Hi Josh,

I'll start with the story as to why.
i've lost more hours than I care to list, because I was unable to build
the virtualbox kernel driver with newer kernels.
Sadly, it gives no useful debug info outside of
make[1]: *** No rule to make target '/tmp/vbox.0/linux/SUPDrv-linux.o', needed by '/tmp/vbox.0/vboxdrv.o'.  Stop.

It took some pretty deep debugging to finally see this:
     Trying rule prerequisite 'tools/objtool/objtool'.
     Looking for a rule with intermediate file 'tools/objtool/objtool'.
      Avoiding implicit rule recursion.
which look quite inoccuous and don't look as errors at all.
When I filed a bug with the vbox folks, they were unable to find out why
the module refused to build on my kernel, and I was stuck with older
kernels as a result.

Then, I had another module, bbswitch, to turn off the nvidia chip on my
laptop to save battery. That one also failed to build with newer
kernels, but thankfully made it more clear that the problem was related
to tools/objtool/objtool missing.

But why was it missing? No idea...
I trace that down to CONFIG_STACK_VALIDATION which there seems to be no
menu option for, so I manually disable it in .config, rebuild, and it's
automatically re-enabled. Gah.

More hair pulling, and finally I make a typo
saruman:/usr/src/linux-block# make xonfig
Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
scripts/kconfig/conf  --silentoldconfig Kconfig
Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
make: *** No rule to make target 'xonfig'.  Stop.

Sure enough, this was my problem, but I never saw the error message
because I build kernels with 
make-kpkg --revision 1gandalf kernel-image
which does other stuff and hid that warning, which really should have
been a fatal error in my opinion.

Given that 
1) CONFIG_STACK_VALIDATION seems silently auto enabled.
2) without libelf-dev, the kernel will build but will leave a tree
missing objtool, which in turn causes (all?) 3rd party modules to fail
building.
3) and that it's kind of non trivial to find out why if that happens, 

Would you consider making
"Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
a build error as opposed to a warning?
This sure would have saved me countless errors of debugging the wrong
things.

Thank you
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

[Josh Poimboeuf]

On Mon, Feb 13, 2017 at 12:07:09AM -0800, Marc MERLIN wrote:
> Hi Josh,
> 
> I'll start with the story as to why.
> i've lost more hours than I care to list, because I was unable to build
> the virtualbox kernel driver with newer kernels.
> Sadly, it gives no useful debug info outside of
> make[1]: *** No rule to make target '/tmp/vbox.0/linux/SUPDrv-linux.o', needed by '/tmp/vbox.0/vboxdrv.o'.  Stop.
> 
> It took some pretty deep debugging to finally see this:
>      Trying rule prerequisite 'tools/objtool/objtool'.
>      Looking for a rule with intermediate file 'tools/objtool/objtool'.
>       Avoiding implicit rule recursion.
> which look quite inoccuous and don't look as errors at all.
> When I filed a bug with the vbox folks, they were unable to find out why
> the module refused to build on my kernel, and I was stuck with older
> kernels as a result.
> 
> Then, I had another module, bbswitch, to turn off the nvidia chip on my
> laptop to save battery. That one also failed to build with newer
> kernels, but thankfully made it more clear that the problem was related
> to tools/objtool/objtool missing.
> 
> But why was it missing? No idea...
> I trace that down to CONFIG_STACK_VALIDATION which there seems to be no
> menu option for, so I manually disable it in .config, rebuild, and it's
> automatically re-enabled. Gah.

Hm, that doesn't sound right.  Nothing automatically enables
CONFIG_STACK_VALIDATION.  It should be disabled unless manually enabled.
Maybe you got it confused with CONFIG_HAVE_STACK_VALIDATION, which is
always enabled?

BTW, there is a config option for it in the menu:

  Kernel hacking
    Compile-time checks and compiler options
      Compile-time stack metadata validation

> More hair pulling, and finally I make a typo
> saruman:/usr/src/linux-block# make xonfig
> Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> scripts/kconfig/conf  --silentoldconfig Kconfig
> Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> make: *** No rule to make target 'xonfig'.  Stop.
> 
> Sure enough, this was my problem, but I never saw the error message
> because I build kernels with 
> make-kpkg --revision 1gandalf kernel-image
> which does other stuff and hid that warning, which really should have
> been a fatal error in my opinion.
>
> Given that 
> 1) CONFIG_STACK_VALIDATION seems silently auto enabled.
> 2) without libelf-dev, the kernel will build but will leave a tree
> missing objtool, which in turn causes (all?) 3rd party modules to fail
> building.

Yes, this is a bug.

> 3) and that it's kind of non trivial to find out why if that happens, 
> 
> Would you consider making
> "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> a build error as opposed to a warning?
> This sure would have saved me countless errors of debugging the wrong
> things.

Correct me if I'm wrong, but it sounds like make-kpkg suppressed stderr?
If so, that should be fixed.

When I try to build an OOT module with CONFIG_STACK_VALIDATION enabled
and elfutils-libelf-devel missing (on Fedora), I get:

  make: Entering directory '/home/jpoimboe/git/linux'
  make[1]: Entering directory '/home/jpoimboe/ktest/output'
    CC [M]  /home/jpoimboe/livepatch-test/1/livepatch2.o
  /bin/sh: ./tools/objtool/objtool: No such file or directory
  /home/jpoimboe/git/linux/scripts/Makefile.build:300: recipe for target '/home/jpoimboe/livepatch-test/1/livepatch2.o' failed
  make[2]: *** [/home/jpoimboe/livepatch-test/1/livepatch2.o] Error 1
  /home/jpoimboe/git/linux/Makefile:1490: recipe for target '_module_/home/jpoimboe/livepatch-test/1' failed
  make[1]: *** [_module_/home/jpoimboe/livepatch-test/1] Error 2
  make[1]: Leaving directory '/home/jpoimboe/ktest/output'
  Makefile:150: recipe for target 'sub-make' failed
  make: *** [sub-make] Error 2
  make: Leaving directory '/home/jpoimboe/git/linux'

It's not a perfect error message, but the

  '/bin/sh: ./tools/objtool/objtool: No such file or directory'

is at least a big clue.  I'm curious why you didn't see that.

Anyway, the above libelf-dev warning is just a warning and not a build
error because CONFIG_STACK_VALIDATION is enabled for allyesconfig, and
it's not a severe enough problem to warrant breaking the build.

Ideally the same warning should be printed when building OOT modules.
I'll try to figure out if there's a way to do that it.

-- 
Josh

Re: Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

[Marc MERLIN]

On Mon, Feb 13, 2017 at 12:41:06PM -0600, Josh Poimboeuf wrote:
> Hm, that doesn't sound right.  Nothing automatically enables
> CONFIG_STACK_VALIDATION.  It should be disabled unless manually enabled.
> Maybe you got it confused with CONFIG_HAVE_STACK_VALIDATION, which is
> always enabled?
 
I did mean CONFIG_STACK_VALIDATION, which is what requires objtool.
It's very possible I enabled it myself during a make oldconfig some time
back, what I meant is that disabling it from the config file doesn't make it
go away, it comes back on its own (see below)

> BTW, there is a config option for it in the menu:
> 
>   Kernel hacking
>     Compile-time checks and compiler options
>       Compile-time stack metadata validation
 
Thanks, I had a hard time finding it since it was not in the same place than
the other options around it. To be honest, I never quite know how to find
where a .config option is located in an xconfig menu, so I looked around
other ones above and below it in .config, and turns out it was the wrong
place.

Anyway, after not finding it in xconfig, I editted .config, and did:
# CONFIG_STACK_VALIDATION is not set
save .config
and the next build re-enabled the option.
That's what caught me by surprise. Did I do something wrong, or is there an
issue there?

> > 2) without libelf-dev, the kernel will build but will leave a tree
> > missing objtool, which in turn causes (all?) 3rd party modules to fail
> > building.
> 
> Yes, this is a bug.
 
Obviously the fix is to make sure objtool builds, but is there a way to make
things better if it doesn't build?
(apparently yes, as you replied below)

> Correct me if I'm wrong, but it sounds like make-kpkg suppressed stderr?
> If so, that should be fixed.
 
It does not, but it adds lines of output before the build starts, and since
the error with libelf-dev missing is not colorized, it was effectively
invisible (one line amongst hundreds scrolling on the screen).
Now that I know what the error is and how to look for it, I can see it, but
as a diagnosis that things were wrong and that things should be fixed, or
3rd party modules would fail to build in weird ways, it was unfortunately
useless.

> When I try to build an OOT module with CONFIG_STACK_VALIDATION enabled
> and elfutils-libelf-devel missing (on Fedora), I get:
> 
>   make: Entering directory '/home/jpoimboe/git/linux'
>   make[1]: Entering directory '/home/jpoimboe/ktest/output'
>     CC [M]  /home/jpoimboe/livepatch-test/1/livepatch2.o
>   /bin/sh: ./tools/objtool/objtool: No such file or directory
>   /home/jpoimboe/git/linux/scripts/Makefile.build:300: recipe for target '/home/jpoimboe/livepatch-test/1/livepatch2.o' failed
>   make[2]: *** [/home/jpoimboe/livepatch-test/1/livepatch2.o] Error 1
>   /home/jpoimboe/git/linux/Makefile:1490: recipe for target '_module_/home/jpoimboe/livepatch-test/1' failed
>   make[1]: *** [_module_/home/jpoimboe/livepatch-test/1] Error 2
>   make[1]: Leaving directory '/home/jpoimboe/ktest/output'
>   Makefile:150: recipe for target 'sub-make' failed
>   make: *** [sub-make] Error 2
>   make: Leaving directory '/home/jpoimboe/git/linux'
> 
> It's not a perfect error message, but the
>   '/bin/sh: ./tools/objtool/objtool: No such file or directory'
> is at least a big clue.  I'm curious why you didn't see that.
 
In the virtualbox build, it just doesn't show up at all, even in the debug
log :(
It's only after spending many many hours trying to find why virtualbox was
not working, that I realized that my bbswitch module wasn't building either,
and that one did point to objtool as a culprit.
But even after I found this, it was non trivial to link this to libelf-dev
missing, given that the message wasn't that visible in a kernel build.

> Anyway, the above libelf-dev warning is just a warning and not a build
> error because CONFIG_STACK_VALIDATION is enabled for allyesconfig, and
> it's not a severe enough problem to warrant breaking the build.

Understood.

> Ideally the same warning should be printed when building OOT modules.
> I'll try to figure out if there's a way to do that it.

This would help, although in that case you can even make the warning an
error since objtool missing seems to be fatal?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

[Josh Poimboeuf]

On Mon, Feb 13, 2017 at 01:31:32PM -0800, Marc MERLIN wrote:
> Anyway, after not finding it in xconfig, I editted .config, and did:
> # CONFIG_STACK_VALIDATION is not set
> save .config
> and the next build re-enabled the option.
> That's what caught me by surprise. Did I do something wrong, or is there an
> issue there?

I really don't see how it would be possible for it to come back by
itself, as it's disabled by default, and no other options select it.
When I remove it, it stays disabled.

> > Ideally the same warning should be printed when building OOT modules.
> > I'll try to figure out if there's a way to do that it.
> 
> This would help, although in that case you can even make the warning an
> error since objtool missing seems to be fatal?

It doesn't need to be fatal though.  It should just be a warning and the
build should succeed, like it does when building the kernel.

-- 
Josh

Re: Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

[Marc MERLIN]

On Mon, Feb 13, 2017 at 04:00:02PM -0600, Josh Poimboeuf wrote:
> On Mon, Feb 13, 2017 at 01:31:32PM -0800, Marc MERLIN wrote:
> > Anyway, after not finding it in xconfig, I editted .config, and did:
> > # CONFIG_STACK_VALIDATION is not set
> > save .config
> > and the next build re-enabled the option.
> > That's what caught me by surprise. Did I do something wrong, or is there an
> > issue there?
> 
> I really don't see how it would be possible for it to come back by
> itself, as it's disabled by default, and no other options select it.
> When I remove it, it stays disabled.
 
Mmmh, you are correct.
I have no idea why/how it got re-enabled yesterday. I'm not seeing this
again today.

> > This would help, although in that case you can even make the warning an
> > error since objtool missing seems to be fatal?
> 
> It doesn't need to be fatal though.  It should just be a warning and the
> build should succeed, like it does when building the kernel.

Agreed, that would be even better.

Thanks for looking at that.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

[Jessica Yu]

+++ Josh Poimboeuf [13/02/17 12:41 -0600]:
>On Mon, Feb 13, 2017 at 12:07:09AM -0800, Marc MERLIN wrote:
>> Hi Josh,
>>
>> I'll start with the story as to why.
>> i've lost more hours than I care to list, because I was unable to build
>> the virtualbox kernel driver with newer kernels.
>> Sadly, it gives no useful debug info outside of
>> make[1]: *** No rule to make target '/tmp/vbox.0/linux/SUPDrv-linux.o', needed by '/tmp/vbox.0/vboxdrv.o'.  Stop.
>>
>> It took some pretty deep debugging to finally see this:
>>      Trying rule prerequisite 'tools/objtool/objtool'.
>>      Looking for a rule with intermediate file 'tools/objtool/objtool'.
>>       Avoiding implicit rule recursion.
>> which look quite inoccuous and don't look as errors at all.
>> When I filed a bug with the vbox folks, they were unable to find out why
>> the module refused to build on my kernel, and I was stuck with older
>> kernels as a result.
>>
>> Then, I had another module, bbswitch, to turn off the nvidia chip on my
>> laptop to save battery. That one also failed to build with newer
>> kernels, but thankfully made it more clear that the problem was related
>> to tools/objtool/objtool missing.
>>
>> But why was it missing? No idea...
>> I trace that down to CONFIG_STACK_VALIDATION which there seems to be no
>> menu option for, so I manually disable it in .config, rebuild, and it's
>> automatically re-enabled. Gah.
>
>Hm, that doesn't sound right.  Nothing automatically enables
>CONFIG_STACK_VALIDATION.  It should be disabled unless manually enabled.
>Maybe you got it confused with CONFIG_HAVE_STACK_VALIDATION, which is
>always enabled?
>
>BTW, there is a config option for it in the menu:
>
>  Kernel hacking
>    Compile-time checks and compiler options
>      Compile-time stack metadata validation
>
>> More hair pulling, and finally I make a typo
>> saruman:/usr/src/linux-block# make xonfig
>> Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
>> scripts/kconfig/conf  --silentoldconfig Kconfig
>> Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
>> make: *** No rule to make target 'xonfig'.  Stop.
>>
>> Sure enough, this was my problem, but I never saw the error message
>> because I build kernels with
>> make-kpkg --revision 1gandalf kernel-image
>> which does other stuff and hid that warning, which really should have
>> been a fatal error in my opinion.
>>
>> Given that
>> 1) CONFIG_STACK_VALIDATION seems silently auto enabled.
>> 2) without libelf-dev, the kernel will build but will leave a tree
>> missing objtool, which in turn causes (all?) 3rd party modules to fail
>> building.
>
>Yes, this is a bug.
>
>> 3) and that it's kind of non trivial to find out why if that happens,
>>
>> Would you consider making
>> "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
>> a build error as opposed to a warning?
>> This sure would have saved me countless errors of debugging the wrong
>> things.
>
>Correct me if I'm wrong, but it sounds like make-kpkg suppressed stderr?
>If so, that should be fixed.
>
>When I try to build an OOT module with CONFIG_STACK_VALIDATION enabled
>and elfutils-libelf-devel missing (on Fedora), I get:
>
>  make: Entering directory '/home/jpoimboe/git/linux'
>  make[1]: Entering directory '/home/jpoimboe/ktest/output'
>    CC [M]  /home/jpoimboe/livepatch-test/1/livepatch2.o
>  /bin/sh: ./tools/objtool/objtool: No such file or directory
>  /home/jpoimboe/git/linux/scripts/Makefile.build:300: recipe for target '/home/jpoimboe/livepatch-test/1/livepatch2.o' failed
>  make[2]: *** [/home/jpoimboe/livepatch-test/1/livepatch2.o] Error 1
>  /home/jpoimboe/git/linux/Makefile:1490: recipe for target '_module_/home/jpoimboe/livepatch-test/1' failed
>  make[1]: *** [_module_/home/jpoimboe/livepatch-test/1] Error 2
>  make[1]: Leaving directory '/home/jpoimboe/ktest/output'
>  Makefile:150: recipe for target 'sub-make' failed
>  make: *** [sub-make] Error 2
>  make: Leaving directory '/home/jpoimboe/git/linux'
>
>It's not a perfect error message, but the
>
>  '/bin/sh: ./tools/objtool/objtool: No such file or directory'
>
>is at least a big clue.  I'm curious why you didn't see that.
>
>Anyway, the above libelf-dev warning is just a warning and not a build
>error because CONFIG_STACK_VALIDATION is enabled for allyesconfig, and
>it's not a severe enough problem to warrant breaking the build.
>
>Ideally the same warning should be printed when building OOT modules.
>I'll try to figure out if there's a way to do that it.

Btw, it looks like the libelf warning is inside an `ifeq ($(KBUILD_EXTMOD),)`
block, so it does not get applied to OOT modules. It would be
possible to add the warning to the corresponding KBUILD_EXTMOD (else)
block or somewhere else in the Makefile common to both cases (probably
more preferable). I tried the latter case and the warning prints for
all cases (vmlinux, intree, external modules).

Jessica

Re: Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

[Josh Poimboeuf]

On Mon, Feb 13, 2017 at 06:08:52PM -0500, Jessica Yu wrote:
> +++ Josh Poimboeuf [13/02/17 12:41 -0600]:
> > On Mon, Feb 13, 2017 at 12:07:09AM -0800, Marc MERLIN wrote:
> > > Hi Josh,
> > > 
> > > I'll start with the story as to why.
> > > i've lost more hours than I care to list, because I was unable to build
> > > the virtualbox kernel driver with newer kernels.
> > > Sadly, it gives no useful debug info outside of
> > > make[1]: *** No rule to make target '/tmp/vbox.0/linux/SUPDrv-linux.o', needed by '/tmp/vbox.0/vboxdrv.o'.  Stop.
> > > 
> > > It took some pretty deep debugging to finally see this:
> > >      Trying rule prerequisite 'tools/objtool/objtool'.
> > >      Looking for a rule with intermediate file 'tools/objtool/objtool'.
> > >       Avoiding implicit rule recursion.
> > > which look quite inoccuous and don't look as errors at all.
> > > When I filed a bug with the vbox folks, they were unable to find out why
> > > the module refused to build on my kernel, and I was stuck with older
> > > kernels as a result.
> > > 
> > > Then, I had another module, bbswitch, to turn off the nvidia chip on my
> > > laptop to save battery. That one also failed to build with newer
> > > kernels, but thankfully made it more clear that the problem was related
> > > to tools/objtool/objtool missing.
> > > 
> > > But why was it missing? No idea...
> > > I trace that down to CONFIG_STACK_VALIDATION which there seems to be no
> > > menu option for, so I manually disable it in .config, rebuild, and it's
> > > automatically re-enabled. Gah.
> > 
> > Hm, that doesn't sound right.  Nothing automatically enables
> > CONFIG_STACK_VALIDATION.  It should be disabled unless manually enabled.
> > Maybe you got it confused with CONFIG_HAVE_STACK_VALIDATION, which is
> > always enabled?
> > 
> > BTW, there is a config option for it in the menu:
> > 
> >  Kernel hacking
> >    Compile-time checks and compiler options
> >      Compile-time stack metadata validation
> > 
> > > More hair pulling, and finally I make a typo
> > > saruman:/usr/src/linux-block# make xonfig
> > > Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> > > scripts/kconfig/conf  --silentoldconfig Kconfig
> > > Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> > > make: *** No rule to make target 'xonfig'.  Stop.
> > > 
> > > Sure enough, this was my problem, but I never saw the error message
> > > because I build kernels with
> > > make-kpkg --revision 1gandalf kernel-image
> > > which does other stuff and hid that warning, which really should have
> > > been a fatal error in my opinion.
> > > 
> > > Given that
> > > 1) CONFIG_STACK_VALIDATION seems silently auto enabled.
> > > 2) without libelf-dev, the kernel will build but will leave a tree
> > > missing objtool, which in turn causes (all?) 3rd party modules to fail
> > > building.
> > 
> > Yes, this is a bug.
> > 
> > > 3) and that it's kind of non trivial to find out why if that happens,
> > > 
> > > Would you consider making
> > > "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> > > a build error as opposed to a warning?
> > > This sure would have saved me countless errors of debugging the wrong
> > > things.
> > 
> > Correct me if I'm wrong, but it sounds like make-kpkg suppressed stderr?
> > If so, that should be fixed.
> > 
> > When I try to build an OOT module with CONFIG_STACK_VALIDATION enabled
> > and elfutils-libelf-devel missing (on Fedora), I get:
> > 
> >  make: Entering directory '/home/jpoimboe/git/linux'
> >  make[1]: Entering directory '/home/jpoimboe/ktest/output'
> >    CC [M]  /home/jpoimboe/livepatch-test/1/livepatch2.o
> >  /bin/sh: ./tools/objtool/objtool: No such file or directory
> >  /home/jpoimboe/git/linux/scripts/Makefile.build:300: recipe for target '/home/jpoimboe/livepatch-test/1/livepatch2.o' failed
> >  make[2]: *** [/home/jpoimboe/livepatch-test/1/livepatch2.o] Error 1
> >  /home/jpoimboe/git/linux/Makefile:1490: recipe for target '_module_/home/jpoimboe/livepatch-test/1' failed
> >  make[1]: *** [_module_/home/jpoimboe/livepatch-test/1] Error 2
> >  make[1]: Leaving directory '/home/jpoimboe/ktest/output'
> >  Makefile:150: recipe for target 'sub-make' failed
> >  make: *** [sub-make] Error 2
> >  make: Leaving directory '/home/jpoimboe/git/linux'
> > 
> > It's not a perfect error message, but the
> > 
> >  '/bin/sh: ./tools/objtool/objtool: No such file or directory'
> > 
> > is at least a big clue.  I'm curious why you didn't see that.
> > 
> > Anyway, the above libelf-dev warning is just a warning and not a build
> > error because CONFIG_STACK_VALIDATION is enabled for allyesconfig, and
> > it's not a severe enough problem to warrant breaking the build.
> > 
> > Ideally the same warning should be printed when building OOT modules.
> > I'll try to figure out if there's a way to do that it.
> 
> Btw, it looks like the libelf warning is inside an `ifeq ($(KBUILD_EXTMOD),)`
> block, so it does not get applied to OOT modules. It would be
> possible to add the warning to the corresponding KBUILD_EXTMOD (else)
> block or somewhere else in the Makefile common to both cases (probably
> more preferable). I tried the latter case and the warning prints for
> all cases (vmlinux, intree, external modules).

Aha, that explains it, thanks.  Will look into fixing it.

-- 
Josh

Re: Please turn "Cannot use CONFIG_STACK_VALIDATION" into build error

[Josh Poimboeuf]

On Tue, Feb 14, 2017 at 07:59:38AM -0600, Josh Poimboeuf wrote:
> On Mon, Feb 13, 2017 at 06:08:52PM -0500, Jessica Yu wrote:
> > +++ Josh Poimboeuf [13/02/17 12:41 -0600]:
> > > On Mon, Feb 13, 2017 at 12:07:09AM -0800, Marc MERLIN wrote:
> > > > Hi Josh,
> > > > 
> > > > I'll start with the story as to why.
> > > > i've lost more hours than I care to list, because I was unable to build
> > > > the virtualbox kernel driver with newer kernels.
> > > > Sadly, it gives no useful debug info outside of
> > > > make[1]: *** No rule to make target '/tmp/vbox.0/linux/SUPDrv-linux.o', needed by '/tmp/vbox.0/vboxdrv.o'.  Stop.
> > > > 
> > > > It took some pretty deep debugging to finally see this:
> > > >      Trying rule prerequisite 'tools/objtool/objtool'.
> > > >      Looking for a rule with intermediate file 'tools/objtool/objtool'.
> > > >       Avoiding implicit rule recursion.
> > > > which look quite inoccuous and don't look as errors at all.
> > > > When I filed a bug with the vbox folks, they were unable to find out why
> > > > the module refused to build on my kernel, and I was stuck with older
> > > > kernels as a result.
> > > > 
> > > > Then, I had another module, bbswitch, to turn off the nvidia chip on my
> > > > laptop to save battery. That one also failed to build with newer
> > > > kernels, but thankfully made it more clear that the problem was related
> > > > to tools/objtool/objtool missing.
> > > > 
> > > > But why was it missing? No idea...
> > > > I trace that down to CONFIG_STACK_VALIDATION which there seems to be no
> > > > menu option for, so I manually disable it in .config, rebuild, and it's
> > > > automatically re-enabled. Gah.
> > > 
> > > Hm, that doesn't sound right.  Nothing automatically enables
> > > CONFIG_STACK_VALIDATION.  It should be disabled unless manually enabled.
> > > Maybe you got it confused with CONFIG_HAVE_STACK_VALIDATION, which is
> > > always enabled?
> > > 
> > > BTW, there is a config option for it in the menu:
> > > 
> > >  Kernel hacking
> > >    Compile-time checks and compiler options
> > >      Compile-time stack metadata validation
> > > 
> > > > More hair pulling, and finally I make a typo
> > > > saruman:/usr/src/linux-block# make xonfig
> > > > Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> > > > scripts/kconfig/conf  --silentoldconfig Kconfig
> > > > Makefile:1044: "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> > > > make: *** No rule to make target 'xonfig'.  Stop.
> > > > 
> > > > Sure enough, this was my problem, but I never saw the error message
> > > > because I build kernels with
> > > > make-kpkg --revision 1gandalf kernel-image
> > > > which does other stuff and hid that warning, which really should have
> > > > been a fatal error in my opinion.
> > > > 
> > > > Given that
> > > > 1) CONFIG_STACK_VALIDATION seems silently auto enabled.
> > > > 2) without libelf-dev, the kernel will build but will leave a tree
> > > > missing objtool, which in turn causes (all?) 3rd party modules to fail
> > > > building.
> > > 
> > > Yes, this is a bug.
> > > 
> > > > 3) and that it's kind of non trivial to find out why if that happens,
> > > > 
> > > > Would you consider making
> > > > "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
> > > > a build error as opposed to a warning?
> > > > This sure would have saved me countless errors of debugging the wrong
> > > > things.
> > > 
> > > Correct me if I'm wrong, but it sounds like make-kpkg suppressed stderr?
> > > If so, that should be fixed.
> > > 
> > > When I try to build an OOT module with CONFIG_STACK_VALIDATION enabled
> > > and elfutils-libelf-devel missing (on Fedora), I get:
> > > 
> > >  make: Entering directory '/home/jpoimboe/git/linux'
> > >  make[1]: Entering directory '/home/jpoimboe/ktest/output'
> > >    CC [M]  /home/jpoimboe/livepatch-test/1/livepatch2.o
> > >  /bin/sh: ./tools/objtool/objtool: No such file or directory
> > >  /home/jpoimboe/git/linux/scripts/Makefile.build:300: recipe for target '/home/jpoimboe/livepatch-test/1/livepatch2.o' failed
> > >  make[2]: *** [/home/jpoimboe/livepatch-test/1/livepatch2.o] Error 1
> > >  /home/jpoimboe/git/linux/Makefile:1490: recipe for target '_module_/home/jpoimboe/livepatch-test/1' failed
> > >  make[1]: *** [_module_/home/jpoimboe/livepatch-test/1] Error 2
> > >  make[1]: Leaving directory '/home/jpoimboe/ktest/output'
> > >  Makefile:150: recipe for target 'sub-make' failed
> > >  make: *** [sub-make] Error 2
> > >  make: Leaving directory '/home/jpoimboe/git/linux'
> > > 
> > > It's not a perfect error message, but the
> > > 
> > >  '/bin/sh: ./tools/objtool/objtool: No such file or directory'
> > > 
> > > is at least a big clue.  I'm curious why you didn't see that.
> > > 
> > > Anyway, the above libelf-dev warning is just a warning and not a build
> > > error because CONFIG_STACK_VALIDATION is enabled for allyesconfig, and
> > > it's not a severe enough problem to warrant breaking the build.
> > > 
> > > Ideally the same warning should be printed when building OOT modules.
> > > I'll try to figure out if there's a way to do that it.
> > 
> > Btw, it looks like the libelf warning is inside an `ifeq ($(KBUILD_EXTMOD),)`
> > block, so it does not get applied to OOT modules. It would be
> > possible to add the warning to the corresponding KBUILD_EXTMOD (else)
> > block or somewhere else in the Makefile common to both cases (probably
> > more preferable). I tried the latter case and the warning prints for
> > all cases (vmlinux, intree, external modules).
> 
> Aha, that explains it, thanks.  Will look into fixing it.

Yep, that was an easy fix.  Marc, can you test the following patch?


diff --git a/Makefile b/Makefile
index 8e223e0..3e794d4 100644
--- a/Makefile
+++ b/Makefile
@@ -908,6 +908,17 @@ mod_sign_cmd = true
 endif
 export mod_sign_cmd
 
+ifdef CONFIG_STACK_VALIDATION
+  has_libelf := $(call try-run,\
+		echo "int main() {}" | $(HOSTCC) -xc -o /dev/null -lelf -,1,0)
+  ifeq ($(has_libelf),1)
+    objtool_target := tools/objtool FORCE
+  else
+    $(warning "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel")
+    SKIP_STACK_VALIDATION := 1
+    export SKIP_STACK_VALIDATION
+  endif
+endif
 
 ifeq ($(KBUILD_EXTMOD),)
 core-y		+= kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/
@@ -1035,18 +1046,6 @@ prepare0: archprepare gcc-plugins
 # All the preparing..
 prepare: prepare0 prepare-objtool
 
-ifdef CONFIG_STACK_VALIDATION
-  has_libelf := $(call try-run,\
-		echo "int main() {}" | $(HOSTCC) -xc -o /dev/null -lelf -,1,0)
-  ifeq ($(has_libelf),1)
-    objtool_target := tools/objtool FORCE
-  else
-    $(warning "Cannot use CONFIG_STACK_VALIDATION, please install libelf-dev, libelf-devel or elfutils-libelf-devel")
-    SKIP_STACK_VALIDATION := 1
-    export SKIP_STACK_VALIDATION
-  endif
-endif
-
 PHONY += prepare-objtool
 prepare-objtool: $(objtool_target)