[regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[Sergey Senozhatsky]

Hello,

after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
enabled") all of my build scripts build the kernel without the modules.

shouldn't that 'KBUILD_MODULES := 1' happen before we `export KBUILD_MODULES'?


this silly patch fixes the build for me:
======================================================================

From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Subject: [PATCH] Kbuild: set KBUILD_MODULES before we export it

Tweak the KBUILD_MODULES flag for TRIM_UNUSED_KSYMS configs
before we `export' it.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 Makefile | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/Makefile b/Makefile
index 369099d..035206e 100644
--- a/Makefile
+++ b/Makefile
@@ -336,6 +336,13 @@ ifeq ($(MAKECMDGOALS),)
   KBUILD_MODULES := 1
 endif
 
+# For the kernel to actually contain only the needed exported symbols,
+# we have to build modules as well to determine what those symbols are.
+# (this can be evaluated only once include/config/auto.conf has been included)
+ifdef CONFIG_TRIM_UNUSED_KSYMS
+  KBUILD_MODULES := 1
+endif
+
 export KBUILD_MODULES KBUILD_BUILTIN
 export KBUILD_CHECKSRC KBUILD_SRC KBUILD_EXTMOD
 
@@ -607,13 +614,6 @@ else
 include/config/auto.conf: ;
 endif # $(dot-config)
 
-# For the kernel to actually contain only the needed exported symbols,
-# we have to build modules as well to determine what those symbols are.
-# (this can be evaluated only once include/config/auto.conf has been included)
-ifdef CONFIG_TRIM_UNUSED_KSYMS
-  KBUILD_MODULES := 1
-endif
-
 # The all: target is the default when no target is given on the
 # command line.
 # This allow a user to issue only 'make' to build a kernel including modules
-- 
2.10.2

Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[Sergey Senozhatsky]

On (12/09/16 11:43), Sergey Senozhatsky wrote:
> 
> after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
> enabled") all of my build scripts build the kernel without the modules.

[..]
> +# For the kernel to actually contain only the needed exported symbols,
> +# we have to build modules as well to determine what those symbols are.
> +# (this can be evaluated only once include/config/auto.conf has been included)
					^^^^^

ah, the config. so the patch is wrong.

	-ss

Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[Nicolas Pitre]

On Fri, 9 Dec 2016, Sergey Senozhatsky wrote:

> On (12/09/16 11:43), Sergey Senozhatsky wrote:
> > 
> > after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
> > enabled") all of my build scripts build the kernel without the modules.
> 
> [..]
> > +# For the kernel to actually contain only the needed exported symbols,
> > +# we have to build modules as well to determine what those symbols are.
> > +# (this can be evaluated only once include/config/auto.conf has been included)
> 					^^^^^
> 
> ah, the config. so the patch is wrong.

Furthermore, the export statement exports the variable not its value. 
Even if the value is changed afterwards, referencing the exported 
variable will see the new value.

You can try out this little Makefile to prove it:

FOOBAR := 0
export FOOBAR
FOOBAR := 1
all:
	echo FOOBAR is $$FOOBAR


Nicolas

Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[Sergey Senozhatsky]

On (12/08/16 22:40), Nicolas Pitre wrote:
> > > after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
> > > enabled") all of my build scripts build the kernel without the modules.
> > 
> > [..]
> > > +# For the kernel to actually contain only the needed exported symbols,
> > > +# we have to build modules as well to determine what those symbols are.
> > > +# (this can be evaluated only once include/config/auto.conf has been included)
> > 					^^^^^
> > 
> > ah, the config. so the patch is wrong.
> 
> Furthermore, the export statement exports the variable not its value. 
> Even if the value is changed afterwards, referencing the exported 
> variable will see the new value.

yeah, need to investigate more. for some, unknown, reason
modules_install gives me empty lib/modules.

the build script (archlinux) builds it in 2 steps: build() and package().
basically:

build:
make -j4 > build_log 2>&1

package:
make -j4 INSTALL_MOD_PATH="${pkgdir}" modules_install >> build_log 2>&1



makepkg
ls -la
pkg/kernel-4.9.0-rc8-dbg-00085-ga37102d-dirty/lib/modules/4.9.0-rc8-dbg-00085-ga37102d-dirty/kernel/
drwxr-xr-x 2 ss ss 4096 Dec  9 13:51 .
drwxr-xr-x 3 ss ss 4096 Dec  9 13:51 ..


git revert 865563924022d8
makepkg
ls -la pkg/kernel-4.9.0-rc8-dbg-00086-g7ea3980-dirty/lib/modules/4.9.0-rc8-dbg-00086-g7ea3980-dirty/kernel/
drwxr-xr-x 10 ss ss 4096 Dec  9 13:55 .
drwxr-xr-x  3 ss ss 4096 Dec  9 13:55 ..
drwxr-xr-x  3 ss ss 4096 Dec  9 13:55 arch
drwxr-xr-x  2 ss ss 4096 Dec  9 13:55 crypto
drwxr-xr-x 14 ss ss 4096 Dec  9 13:55 drivers
drwxr-xr-x  4 ss ss 4096 Dec  9 13:55 fs
drwxr-xr-x  4 ss ss 4096 Dec  9 13:55 lib
drwxr-xr-x  2 ss ss 4096 Dec  9 13:55 mm
drwxr-xr-x  3 ss ss 4096 Dec  9 13:55 net
drwxr-xr-x  5 ss ss 4096 Dec  9 13:55 sound


revert-the-revert

git revert 7ea39801035a23
[master 7b90890] Revert "Revert "kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled""


makepkg
ls -la
pkg/kernel-4.9.0-rc8-dbg-00087-g7b90890-dirty/lib/modules/4.9.0-rc8-dbg-00087-g7b90890-dirty/kernel/
drwxr-xr-x 2 ss ss 4096 Dec  9 13:56 .
drwxr-xr-x 3 ss ss 4096 Dec  9 13:56 ..


not sure I see why...

	-ss

Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[Nicolas Pitre]

On Fri, 9 Dec 2016, Sergey Senozhatsky wrote:

> On (12/08/16 22:40), Nicolas Pitre wrote:
> > > > after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
> > > > enabled") all of my build scripts build the kernel without the modules.
> > > 
> > > [..]
> > > > +# For the kernel to actually contain only the needed exported symbols,
> > > > +# we have to build modules as well to determine what those symbols are.
> > > > +# (this can be evaluated only once include/config/auto.conf has been included)
> > > 					^^^^^
> > > 
> > > ah, the config. so the patch is wrong.
> > 
> > Furthermore, the export statement exports the variable not its value. 
> > Even if the value is changed afterwards, referencing the exported 
> > variable will see the new value.
> 
> yeah, need to investigate more. for some, unknown, reason
> modules_install gives me empty lib/modules.
> 
> the build script (archlinux) builds it in 2 steps: build() and package().
> basically:
> 
> build:
> make -j4 > build_log 2>&1
> 
> package:
> make -j4 INSTALL_MOD_PATH="${pkgdir}" modules_install >> build_log 2>&1
> 

Weird. I basically did that and I can't reproduce your problem.

> makepkg
> ls -la
> pkg/kernel-4.9.0-rc8-dbg-00085-ga37102d-dirty/lib/modules/4.9.0-rc8-dbg-00085-ga37102d-dirty/kernel/
> drwxr-xr-x 2 ss ss 4096 Dec  9 13:51 .
> drwxr-xr-x 3 ss ss 4096 Dec  9 13:51 ..
> 
> 
> git revert 865563924022d8
> makepkg
> ls -la pkg/kernel-4.9.0-rc8-dbg-00086-g7ea3980-dirty/lib/modules/4.9.0-rc8-dbg-00086-g7ea3980-dirty/kernel/
> drwxr-xr-x 10 ss ss 4096 Dec  9 13:55 .
> drwxr-xr-x  3 ss ss 4096 Dec  9 13:55 ..
> drwxr-xr-x  3 ss ss 4096 Dec  9 13:55 arch
> drwxr-xr-x  2 ss ss 4096 Dec  9 13:55 crypto
> drwxr-xr-x 14 ss ss 4096 Dec  9 13:55 drivers
> drwxr-xr-x  4 ss ss 4096 Dec  9 13:55 fs
> drwxr-xr-x  4 ss ss 4096 Dec  9 13:55 lib
> drwxr-xr-x  2 ss ss 4096 Dec  9 13:55 mm
> drwxr-xr-x  3 ss ss 4096 Dec  9 13:55 net
> drwxr-xr-x  5 ss ss 4096 Dec  9 13:55 sound

You must have CONFIG_TRIM_UNUSED_KSYMS=y in your .config, right?

What if you set it to n instead without reverting 865563924022d8. Do you 
still have the same issue?

In any case, could you give me your .config to help me reproduce?


Nicolas

Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[Sergey Senozhatsky]

On (12/09/16 13:07), Nicolas Pitre wrote:
[..]
> > build:
> > make -j4 > build_log 2>&1
> > 
> > package:
> > make -j4 INSTALL_MOD_PATH="${pkgdir}" modules_install >> build_log 2>&1
> 
> Weird.

it is. sorry for long reply, it took me some time to track it down.
turned out, the script also does `prepare' and `kernelrelease'. so
the sequence of commands in my build script is

	make prepare
	make kernelrelease
# functon build
	make -j4
# finction package
	make -j4 INSTALL_MOD_PATH=XXXX modules_install


now. the problem here is that, apparently, and I didn't know that,
"make prepare" and "make kernelrelease" are executed twice.

- first time when I build the kernel
 make prepare
 make kernelrelease
 make -j4

- second time when I install the modules
 make prepare
 make kernelrelease
 make -j4 INSTALL_MOD_PATH=XXXX modules_install


so this will not install modules:
 make prepare; make kernelrelease; make -j4; make prepare; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install

and this will:
 make prepare; make kernelrelease; make -j4; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install


> You must have CONFIG_TRIM_UNUSED_KSYMS=y in your .config, right?

yes.

> What if you set it to n instead without reverting 865563924022d8. Do you 
> still have the same issue?

!CONFIG_TRIM_UNUSED_KSYMS or !865563924022d8 builds just fine with extra
`make prepare'.
no `extra make prepare' builds ok regardless the state of Makefile/config
files.

I guess you don't need my .config any more.

	-ss

Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[Nicolas Pitre]

On Sat, 10 Dec 2016, Sergey Senozhatsky wrote:

> On (12/09/16 13:07), Nicolas Pitre wrote:
> [..]
> > > build:
> > > make -j4 > build_log 2>&1
> > > 
> > > package:
> > > make -j4 INSTALL_MOD_PATH="${pkgdir}" modules_install >> build_log 2>&1
> > 
> > Weird.
> 
> it is. sorry for long reply, it took me some time to track it down.
> turned out, the script also does `prepare' and `kernelrelease'. so
> the sequence of commands in my build script is
> 
> 	make prepare
> 	make kernelrelease
> # functon build
> 	make -j4
> # finction package
> 	make -j4 INSTALL_MOD_PATH=XXXX modules_install
> 
> 
> now. the problem here is that, apparently, and I didn't know that,
> "make prepare" and "make kernelrelease" are executed twice.
> 
> - first time when I build the kernel
>  make prepare
>  make kernelrelease
>  make -j4
> 
> - second time when I install the modules
>  make prepare
>  make kernelrelease
>  make -j4 INSTALL_MOD_PATH=XXXX modules_install
> 
> 
> so this will not install modules:
>  make prepare; make kernelrelease; make -j4; make prepare; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
> 
> and this will:
>  make prepare; make kernelrelease; make -j4; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install

Right. And this is because of this in the main Makefile:

# Create temporary dir for module support files
# clean it up only when building all modules
cmd_crmodverdir = $(Q)mkdir -p $(MODVERDIR) \
                  $(if $(KBUILD_MODULES),; rm -f $(MODVERDIR)/*)

The list of modules to install is created from files found in 
$(MODVERDIR)/. This is cleared when KBUILD_MODULES is set.  Oddly 
enough, KBUILD_MODULES is _not_ globally set when building individual 
modules probably not to clear MODVERDIR. This requires explicit override 
like in this rule:

%.ko: prepare scripts FORCE
	$(cmd_crmodverdir)
	$(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1)  \
	  $(build)=$(build-dir) $(@:.ko=.o)

One could wonder why $(cmd_crmodverdir) is executed again here given 
that it is already part of the "prepare" target, but that's an 
orthogonal issue.

Another question is whether or not KBUILD_MODULES is the right criteria 
for clearing MODVERDIR. My first reaction is to say it is not, but I 
can't come up with anything better at the moment.

And KBUILD_MODULES must be set for any target that results in 
vmlinux being built (and there are many of them including arch specific) 
whenever CONFIG_TRIM_UNUSED_KSYMS=y. Can this be enforced elsewhere in 
the Makefile, like in the recipe for $(vmlinux-dirs)? I don't know. IMHO 
this will only make things even less pretty than they are now.

In the mean time, though, I'm wondering why you have to do "make 
prepare" twice, or even at all.  Semantically, we could think of 
"prepare" as meaning to set things up for the build.  That could imply 
the erasing of some temporary files or even product files. Therefore 
that shouldn't be appropriate before a "modules_install" IMHO. 
Furthermore the "prepare" target is not listed amongst the documented 
make targets neither in the README file nor in the "make help" output.

So, given all the above considerations, would it be possible for you to 
"fix" your build script instead?


Nicolas

Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled

[Sergey Senozhatsky]

On (12/10/16 16:04), Nicolas Pitre wrote:
> On Sat, 10 Dec 2016, Sergey Senozhatsky wrote:
[..]
> > so this will not install modules:
> >  make prepare; make kernelrelease; make -j4; make prepare; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
> > 
> > and this will:
> >  make prepare; make kernelrelease; make -j4; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
> 
> Right. And this is because of this in the main Makefile:
> 
> # Create temporary dir for module support files
> # clean it up only when building all modules
> cmd_crmodverdir = $(Q)mkdir -p $(MODVERDIR) \
>                   $(if $(KBUILD_MODULES),; rm -f $(MODVERDIR)/*)
> 
> The list of modules to install is created from files found in 
> $(MODVERDIR)/. This is cleared when KBUILD_MODULES is set.  Oddly 
> enough, KBUILD_MODULES is _not_ globally set when building individual 
> modules probably not to clear MODVERDIR. This requires explicit override 
> like in this rule:
> 
> %.ko: prepare scripts FORCE
> 	$(cmd_crmodverdir)
> 	$(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1)  \
> 	  $(build)=$(build-dir) $(@:.ko=.o)
> 
> One could wonder why $(cmd_crmodverdir) is executed again here given 
> that it is already part of the "prepare" target, but that's an 
> orthogonal issue.
> 
> Another question is whether or not KBUILD_MODULES is the right criteria 
> for clearing MODVERDIR. My first reaction is to say it is not, but I 
> can't come up with anything better at the moment.
> 
> And KBUILD_MODULES must be set for any target that results in 
> vmlinux being built (and there are many of them including arch specific) 
> whenever CONFIG_TRIM_UNUSED_KSYMS=y. Can this be enforced elsewhere in 
> the Makefile, like in the recipe for $(vmlinux-dirs)? I don't know. IMHO 
> this will only make things even less pretty than they are now.
> 
> In the mean time, though, I'm wondering why you have to do "make 
> prepare" twice, or even at all.

well, not that I really wanted to execute it twice, it just was at
the beginning of the build/packaging script that I touched some 6-7
years ago.

> So, given all the above considerations, would it be possible for you to 
> "fix" your build script instead?

sure.

thanks.

	-ss