* [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled
@ 2016-12-09 2:43 Sergey Senozhatsky
2016-12-09 2:56 ` Sergey Senozhatsky
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Senozhatsky @ 2016-12-09 2:43 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Jarod Wilson, Michal Marek, linux-kbuild, linux-kernel
Hello,
after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
enabled") all of my build scripts build the kernel without the modules.
shouldn't that 'KBUILD_MODULES := 1' happen before we `export KBUILD_MODULES'?
this silly patch fixes the build for me:
======================================================================
From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Subject: [PATCH] Kbuild: set KBUILD_MODULES before we export it
Tweak the KBUILD_MODULES flag for TRIM_UNUSED_KSYMS configs
before we `export' it.
Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
Makefile | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/Makefile b/Makefile
index 369099d..035206e 100644
--- a/Makefile
+++ b/Makefile
@@ -336,6 +336,13 @@ ifeq ($(MAKECMDGOALS),)
KBUILD_MODULES := 1
endif
+# For the kernel to actually contain only the needed exported symbols,
+# we have to build modules as well to determine what those symbols are.
+# (this can be evaluated only once include/config/auto.conf has been included)
+ifdef CONFIG_TRIM_UNUSED_KSYMS
+ KBUILD_MODULES := 1
+endif
+
export KBUILD_MODULES KBUILD_BUILTIN
export KBUILD_CHECKSRC KBUILD_SRC KBUILD_EXTMOD
@@ -607,13 +614,6 @@ else
include/config/auto.conf: ;
endif # $(dot-config)
-# For the kernel to actually contain only the needed exported symbols,
-# we have to build modules as well to determine what those symbols are.
-# (this can be evaluated only once include/config/auto.conf has been included)
-ifdef CONFIG_TRIM_UNUSED_KSYMS
- KBUILD_MODULES := 1
-endif
-
# The all: target is the default when no target is given on the
# command line.
# This allow a user to issue only 'make' to build a kernel including modules
--
2.10.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled
2016-12-09 2:43 [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled Sergey Senozhatsky
@ 2016-12-09 2:56 ` Sergey Senozhatsky
2016-12-09 3:40 ` Nicolas Pitre
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Senozhatsky @ 2016-12-09 2:56 UTC (permalink / raw)
To: Sergey Senozhatsky
Cc: Nicolas Pitre, Jarod Wilson, Michal Marek, linux-kbuild, linux-kernel
On (12/09/16 11:43), Sergey Senozhatsky wrote:
>
> after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
> enabled") all of my build scripts build the kernel without the modules.
[..]
> +# For the kernel to actually contain only the needed exported symbols,
> +# we have to build modules as well to determine what those symbols are.
> +# (this can be evaluated only once include/config/auto.conf has been included)
^^^^^
ah, the config. so the patch is wrong.
-ss
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled
2016-12-09 2:56 ` Sergey Senozhatsky
@ 2016-12-09 3:40 ` Nicolas Pitre
2016-12-09 4:58 ` Sergey Senozhatsky
0 siblings, 1 reply; 8+ messages in thread
From: Nicolas Pitre @ 2016-12-09 3:40 UTC (permalink / raw)
To: Sergey Senozhatsky; +Cc: Jarod Wilson, Michal Marek, linux-kbuild, linux-kernel
On Fri, 9 Dec 2016, Sergey Senozhatsky wrote:
> On (12/09/16 11:43), Sergey Senozhatsky wrote:
> >
> > after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
> > enabled") all of my build scripts build the kernel without the modules.
>
> [..]
> > +# For the kernel to actually contain only the needed exported symbols,
> > +# we have to build modules as well to determine what those symbols are.
> > +# (this can be evaluated only once include/config/auto.conf has been included)
> ^^^^^
>
> ah, the config. so the patch is wrong.
Furthermore, the export statement exports the variable not its value.
Even if the value is changed afterwards, referencing the exported
variable will see the new value.
You can try out this little Makefile to prove it:
FOOBAR := 0
export FOOBAR
FOOBAR := 1
all:
echo FOOBAR is $$FOOBAR
Nicolas
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled
2016-12-09 3:40 ` Nicolas Pitre
@ 2016-12-09 4:58 ` Sergey Senozhatsky
2016-12-09 18:07 ` Nicolas Pitre
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Senozhatsky @ 2016-12-09 4:58 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Sergey Senozhatsky, Jarod Wilson, Michal Marek, linux-kbuild,
linux-kernel
On (12/08/16 22:40), Nicolas Pitre wrote:
> > > after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
> > > enabled") all of my build scripts build the kernel without the modules.
> >
> > [..]
> > > +# For the kernel to actually contain only the needed exported symbols,
> > > +# we have to build modules as well to determine what those symbols are.
> > > +# (this can be evaluated only once include/config/auto.conf has been included)
> > ^^^^^
> >
> > ah, the config. so the patch is wrong.
>
> Furthermore, the export statement exports the variable not its value.
> Even if the value is changed afterwards, referencing the exported
> variable will see the new value.
yeah, need to investigate more. for some, unknown, reason
modules_install gives me empty lib/modules.
the build script (archlinux) builds it in 2 steps: build() and package().
basically:
build:
make -j4 > build_log 2>&1
package:
make -j4 INSTALL_MOD_PATH="${pkgdir}" modules_install >> build_log 2>&1
makepkg
ls -la
pkg/kernel-4.9.0-rc8-dbg-00085-ga37102d-dirty/lib/modules/4.9.0-rc8-dbg-00085-ga37102d-dirty/kernel/
drwxr-xr-x 2 ss ss 4096 Dec 9 13:51 .
drwxr-xr-x 3 ss ss 4096 Dec 9 13:51 ..
git revert 865563924022d8
makepkg
ls -la pkg/kernel-4.9.0-rc8-dbg-00086-g7ea3980-dirty/lib/modules/4.9.0-rc8-dbg-00086-g7ea3980-dirty/kernel/
drwxr-xr-x 10 ss ss 4096 Dec 9 13:55 .
drwxr-xr-x 3 ss ss 4096 Dec 9 13:55 ..
drwxr-xr-x 3 ss ss 4096 Dec 9 13:55 arch
drwxr-xr-x 2 ss ss 4096 Dec 9 13:55 crypto
drwxr-xr-x 14 ss ss 4096 Dec 9 13:55 drivers
drwxr-xr-x 4 ss ss 4096 Dec 9 13:55 fs
drwxr-xr-x 4 ss ss 4096 Dec 9 13:55 lib
drwxr-xr-x 2 ss ss 4096 Dec 9 13:55 mm
drwxr-xr-x 3 ss ss 4096 Dec 9 13:55 net
drwxr-xr-x 5 ss ss 4096 Dec 9 13:55 sound
revert-the-revert
git revert 7ea39801035a23
[master 7b90890] Revert "Revert "kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled""
makepkg
ls -la
pkg/kernel-4.9.0-rc8-dbg-00087-g7b90890-dirty/lib/modules/4.9.0-rc8-dbg-00087-g7b90890-dirty/kernel/
drwxr-xr-x 2 ss ss 4096 Dec 9 13:56 .
drwxr-xr-x 3 ss ss 4096 Dec 9 13:56 ..
not sure I see why...
-ss
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled
2016-12-09 4:58 ` Sergey Senozhatsky
@ 2016-12-09 18:07 ` Nicolas Pitre
2016-12-10 7:23 ` Sergey Senozhatsky
0 siblings, 1 reply; 8+ messages in thread
From: Nicolas Pitre @ 2016-12-09 18:07 UTC (permalink / raw)
To: Sergey Senozhatsky; +Cc: Jarod Wilson, Michal Marek, linux-kbuild, linux-kernel
On Fri, 9 Dec 2016, Sergey Senozhatsky wrote:
> On (12/08/16 22:40), Nicolas Pitre wrote:
> > > > after 53924022d8a ("kbuild: fix building bzImage withCONFIG_TRIM_UNUSED_KSYMS
> > > > enabled") all of my build scripts build the kernel without the modules.
> > >
> > > [..]
> > > > +# For the kernel to actually contain only the needed exported symbols,
> > > > +# we have to build modules as well to determine what those symbols are.
> > > > +# (this can be evaluated only once include/config/auto.conf has been included)
> > > ^^^^^
> > >
> > > ah, the config. so the patch is wrong.
> >
> > Furthermore, the export statement exports the variable not its value.
> > Even if the value is changed afterwards, referencing the exported
> > variable will see the new value.
>
> yeah, need to investigate more. for some, unknown, reason
> modules_install gives me empty lib/modules.
>
> the build script (archlinux) builds it in 2 steps: build() and package().
> basically:
>
> build:
> make -j4 > build_log 2>&1
>
> package:
> make -j4 INSTALL_MOD_PATH="${pkgdir}" modules_install >> build_log 2>&1
>
Weird. I basically did that and I can't reproduce your problem.
> makepkg
> ls -la
> pkg/kernel-4.9.0-rc8-dbg-00085-ga37102d-dirty/lib/modules/4.9.0-rc8-dbg-00085-ga37102d-dirty/kernel/
> drwxr-xr-x 2 ss ss 4096 Dec 9 13:51 .
> drwxr-xr-x 3 ss ss 4096 Dec 9 13:51 ..
>
>
> git revert 865563924022d8
> makepkg
> ls -la pkg/kernel-4.9.0-rc8-dbg-00086-g7ea3980-dirty/lib/modules/4.9.0-rc8-dbg-00086-g7ea3980-dirty/kernel/
> drwxr-xr-x 10 ss ss 4096 Dec 9 13:55 .
> drwxr-xr-x 3 ss ss 4096 Dec 9 13:55 ..
> drwxr-xr-x 3 ss ss 4096 Dec 9 13:55 arch
> drwxr-xr-x 2 ss ss 4096 Dec 9 13:55 crypto
> drwxr-xr-x 14 ss ss 4096 Dec 9 13:55 drivers
> drwxr-xr-x 4 ss ss 4096 Dec 9 13:55 fs
> drwxr-xr-x 4 ss ss 4096 Dec 9 13:55 lib
> drwxr-xr-x 2 ss ss 4096 Dec 9 13:55 mm
> drwxr-xr-x 3 ss ss 4096 Dec 9 13:55 net
> drwxr-xr-x 5 ss ss 4096 Dec 9 13:55 sound
You must have CONFIG_TRIM_UNUSED_KSYMS=y in your .config, right?
What if you set it to n instead without reverting 865563924022d8. Do you
still have the same issue?
In any case, could you give me your .config to help me reproduce?
Nicolas
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled
2016-12-09 18:07 ` Nicolas Pitre
@ 2016-12-10 7:23 ` Sergey Senozhatsky
2016-12-10 21:04 ` Nicolas Pitre
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Senozhatsky @ 2016-12-10 7:23 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Sergey Senozhatsky, Jarod Wilson, Michal Marek, linux-kbuild,
linux-kernel, Sergey Senozhatsky
On (12/09/16 13:07), Nicolas Pitre wrote:
[..]
> > build:
> > make -j4 > build_log 2>&1
> >
> > package:
> > make -j4 INSTALL_MOD_PATH="${pkgdir}" modules_install >> build_log 2>&1
>
> Weird.
it is. sorry for long reply, it took me some time to track it down.
turned out, the script also does `prepare' and `kernelrelease'. so
the sequence of commands in my build script is
make prepare
make kernelrelease
# functon build
make -j4
# finction package
make -j4 INSTALL_MOD_PATH=XXXX modules_install
now. the problem here is that, apparently, and I didn't know that,
"make prepare" and "make kernelrelease" are executed twice.
- first time when I build the kernel
make prepare
make kernelrelease
make -j4
- second time when I install the modules
make prepare
make kernelrelease
make -j4 INSTALL_MOD_PATH=XXXX modules_install
so this will not install modules:
make prepare; make kernelrelease; make -j4; make prepare; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
and this will:
make prepare; make kernelrelease; make -j4; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
> You must have CONFIG_TRIM_UNUSED_KSYMS=y in your .config, right?
yes.
> What if you set it to n instead without reverting 865563924022d8. Do you
> still have the same issue?
!CONFIG_TRIM_UNUSED_KSYMS or !865563924022d8 builds just fine with extra
`make prepare'.
no `extra make prepare' builds ok regardless the state of Makefile/config
files.
I guess you don't need my .config any more.
-ss
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled
2016-12-10 7:23 ` Sergey Senozhatsky
@ 2016-12-10 21:04 ` Nicolas Pitre
2016-12-11 2:19 ` Sergey Senozhatsky
0 siblings, 1 reply; 8+ messages in thread
From: Nicolas Pitre @ 2016-12-10 21:04 UTC (permalink / raw)
To: Sergey Senozhatsky
Cc: Jarod Wilson, Michal Marek, linux-kbuild, linux-kernel,
Sergey Senozhatsky
On Sat, 10 Dec 2016, Sergey Senozhatsky wrote:
> On (12/09/16 13:07), Nicolas Pitre wrote:
> [..]
> > > build:
> > > make -j4 > build_log 2>&1
> > >
> > > package:
> > > make -j4 INSTALL_MOD_PATH="${pkgdir}" modules_install >> build_log 2>&1
> >
> > Weird.
>
> it is. sorry for long reply, it took me some time to track it down.
> turned out, the script also does `prepare' and `kernelrelease'. so
> the sequence of commands in my build script is
>
> make prepare
> make kernelrelease
> # functon build
> make -j4
> # finction package
> make -j4 INSTALL_MOD_PATH=XXXX modules_install
>
>
> now. the problem here is that, apparently, and I didn't know that,
> "make prepare" and "make kernelrelease" are executed twice.
>
> - first time when I build the kernel
> make prepare
> make kernelrelease
> make -j4
>
> - second time when I install the modules
> make prepare
> make kernelrelease
> make -j4 INSTALL_MOD_PATH=XXXX modules_install
>
>
> so this will not install modules:
> make prepare; make kernelrelease; make -j4; make prepare; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
>
> and this will:
> make prepare; make kernelrelease; make -j4; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
Right. And this is because of this in the main Makefile:
# Create temporary dir for module support files
# clean it up only when building all modules
cmd_crmodverdir = $(Q)mkdir -p $(MODVERDIR) \
$(if $(KBUILD_MODULES),; rm -f $(MODVERDIR)/*)
The list of modules to install is created from files found in
$(MODVERDIR)/. This is cleared when KBUILD_MODULES is set. Oddly
enough, KBUILD_MODULES is _not_ globally set when building individual
modules probably not to clear MODVERDIR. This requires explicit override
like in this rule:
%.ko: prepare scripts FORCE
$(cmd_crmodverdir)
$(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1) \
$(build)=$(build-dir) $(@:.ko=.o)
One could wonder why $(cmd_crmodverdir) is executed again here given
that it is already part of the "prepare" target, but that's an
orthogonal issue.
Another question is whether or not KBUILD_MODULES is the right criteria
for clearing MODVERDIR. My first reaction is to say it is not, but I
can't come up with anything better at the moment.
And KBUILD_MODULES must be set for any target that results in
vmlinux being built (and there are many of them including arch specific)
whenever CONFIG_TRIM_UNUSED_KSYMS=y. Can this be enforced elsewhere in
the Makefile, like in the recipe for $(vmlinux-dirs)? I don't know. IMHO
this will only make things even less pretty than they are now.
In the mean time, though, I'm wondering why you have to do "make
prepare" twice, or even at all. Semantically, we could think of
"prepare" as meaning to set things up for the build. That could imply
the erasing of some temporary files or even product files. Therefore
that shouldn't be appropriate before a "modules_install" IMHO.
Furthermore the "prepare" target is not listed amongst the documented
make targets neither in the README file nor in the "make help" output.
So, given all the above considerations, would it be possible for you to
"fix" your build script instead?
Nicolas
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled
2016-12-10 21:04 ` Nicolas Pitre
@ 2016-12-11 2:19 ` Sergey Senozhatsky
0 siblings, 0 replies; 8+ messages in thread
From: Sergey Senozhatsky @ 2016-12-11 2:19 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Sergey Senozhatsky, Jarod Wilson, Michal Marek, linux-kbuild,
linux-kernel, Sergey Senozhatsky
On (12/10/16 16:04), Nicolas Pitre wrote:
> On Sat, 10 Dec 2016, Sergey Senozhatsky wrote:
[..]
> > so this will not install modules:
> > make prepare; make kernelrelease; make -j4; make prepare; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
> >
> > and this will:
> > make prepare; make kernelrelease; make -j4; make kernelrelease; make -j4 INSTALL_MOD_PATH=/tmp/MODULES modules_install
>
> Right. And this is because of this in the main Makefile:
>
> # Create temporary dir for module support files
> # clean it up only when building all modules
> cmd_crmodverdir = $(Q)mkdir -p $(MODVERDIR) \
> $(if $(KBUILD_MODULES),; rm -f $(MODVERDIR)/*)
>
> The list of modules to install is created from files found in
> $(MODVERDIR)/. This is cleared when KBUILD_MODULES is set. Oddly
> enough, KBUILD_MODULES is _not_ globally set when building individual
> modules probably not to clear MODVERDIR. This requires explicit override
> like in this rule:
>
> %.ko: prepare scripts FORCE
> $(cmd_crmodverdir)
> $(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1) \
> $(build)=$(build-dir) $(@:.ko=.o)
>
> One could wonder why $(cmd_crmodverdir) is executed again here given
> that it is already part of the "prepare" target, but that's an
> orthogonal issue.
>
> Another question is whether or not KBUILD_MODULES is the right criteria
> for clearing MODVERDIR. My first reaction is to say it is not, but I
> can't come up with anything better at the moment.
>
> And KBUILD_MODULES must be set for any target that results in
> vmlinux being built (and there are many of them including arch specific)
> whenever CONFIG_TRIM_UNUSED_KSYMS=y. Can this be enforced elsewhere in
> the Makefile, like in the recipe for $(vmlinux-dirs)? I don't know. IMHO
> this will only make things even less pretty than they are now.
>
> In the mean time, though, I'm wondering why you have to do "make
> prepare" twice, or even at all.
well, not that I really wanted to execute it twice, it just was at
the beginning of the build/packaging script that I touched some 6-7
years ago.
> So, given all the above considerations, would it be possible for you to
> "fix" your build script instead?
sure.
thanks.
-ss
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-12-11 2:19 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-12-09 2:43 [regression ?] kbuild: fix building bzImage with CONFIG_TRIM_UNUSED_KSYMS enabled Sergey Senozhatsky
2016-12-09 2:56 ` Sergey Senozhatsky
2016-12-09 3:40 ` Nicolas Pitre
2016-12-09 4:58 ` Sergey Senozhatsky
2016-12-09 18:07 ` Nicolas Pitre
2016-12-10 7:23 ` Sergey Senozhatsky
2016-12-10 21:04 ` Nicolas Pitre
2016-12-11 2:19 ` Sergey Senozhatsky
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).