* [PATCH] kbuild: Fail the build early when no lz4 present
@ 2018-10-10 22:23 Borislav Petkov
2018-10-10 22:34 ` Randy Dunlap
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Borislav Petkov @ 2018-10-10 22:23 UTC (permalink / raw)
To: LKML; +Cc: Masahiro Yamada, Michal Marek, linux-kbuild
From: Borislav Petkov <bp@suse.de>
When building randconfigs, the build fails at kernel compression stage
due to missing lz4 on the system but CONFIG_KERNEL_LZ4 has been selected
by randconfig. The result looks somethins like this:
(cat arch/x86/boot/compressed/vmlinux.bin arch/x86/boot/compressed/vmlinux.relocs | lz4c -l -c1 stdin stdout && printf \334\141\301\001) > arch/x86/boot/compressed/vmlinux.bin.lz4 || (rm -f arch/x86/boot/compressed/vmlinux.bin.lz4 ; false)
/bin/sh: 1: lz4c: not found
make[2]: *** [arch/x86/boot/compressed/Makefile:143: arch/x86/boot/compressed/vmlinux.bin.lz4] Error 1
make[1]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2
make: *** [arch/x86/Makefile:290: bzImage] Error 2
Fail the build much earlier by checking for lz4c presence before doing
anything else.
Signed-off-by: Borislav Petkov <bp@suse.de>
Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: linux-kbuild@vger.kernel.org
---
Makefile | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/Makefile b/Makefile
index a0c32de80482..f91de649234b 100644
--- a/Makefile
+++ b/Makefile
@@ -788,6 +788,12 @@ KBUILD_CFLAGS_KERNEL += -ffunction-sections -fdata-sections
LDFLAGS_vmlinux += --gc-sections
endif
+ifdef CONFIG_KERNEL_LZ4
+ifeq ($(CONFIG_SHELL which lz4c),)
+$(error "lz4 tool not found on this system but CONFIG_KERNEL_LZ4 enabled")
+endif
+endif
+
# arch Makefile may override CC so keep this after arch Makefile is included
NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)
--
2.19.0.271.gfe8321ec057f
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH] kbuild: Fail the build early when no lz4 present 2018-10-10 22:23 [PATCH] kbuild: Fail the build early when no lz4 present Borislav Petkov @ 2018-10-10 22:34 ` Randy Dunlap 2018-10-11 4:26 ` Masahiro Yamada 2018-10-11 20:33 ` kbuild test robot 2 siblings, 0 replies; 8+ messages in thread From: Randy Dunlap @ 2018-10-10 22:34 UTC (permalink / raw) To: Borislav Petkov, LKML; +Cc: Masahiro Yamada, Michal Marek, linux-kbuild On 10/10/18 3:23 PM, Borislav Petkov wrote: > From: Borislav Petkov <bp@suse.de> > > When building randconfigs, the build fails at kernel compression stage > due to missing lz4 on the system but CONFIG_KERNEL_LZ4 has been selected > by randconfig. The result looks somethins like this: > > (cat arch/x86/boot/compressed/vmlinux.bin arch/x86/boot/compressed/vmlinux.relocs | lz4c -l -c1 stdin stdout && printf \334\141\301\001) > arch/x86/boot/compressed/vmlinux.bin.lz4 || (rm -f arch/x86/boot/compressed/vmlinux.bin.lz4 ; false) > /bin/sh: 1: lz4c: not found > make[2]: *** [arch/x86/boot/compressed/Makefile:143: arch/x86/boot/compressed/vmlinux.bin.lz4] Error 1 > make[1]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2 > make: *** [arch/x86/Makefile:290: bzImage] Error 2 > > Fail the build much earlier by checking for lz4c presence before doing > anything else. > > Signed-off-by: Borislav Petkov <bp@suse.de> > Cc: Masahiro Yamada <yamada.masahiro@socionext.com> > Cc: Michal Marek <michal.lkml@markovi.net> > Cc: linux-kbuild@vger.kernel.org > --- > Makefile | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/Makefile b/Makefile > index a0c32de80482..f91de649234b 100644 > --- a/Makefile > +++ b/Makefile > @@ -788,6 +788,12 @@ KBUILD_CFLAGS_KERNEL += -ffunction-sections -fdata-sections > LDFLAGS_vmlinux += --gc-sections > endif > > +ifdef CONFIG_KERNEL_LZ4 > +ifeq ($(CONFIG_SHELL which lz4c),) Hi, I have been informed in the past the POSIX spells "which" as "command -v" and hence requested to do s/which/command -v/ > +$(error "lz4 tool not found on this system but CONFIG_KERNEL_LZ4 enabled") > +endif > +endif > + > # arch Makefile may override CC so keep this after arch Makefile is included > NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include) > > cheers, -- ~Randy ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] kbuild: Fail the build early when no lz4 present 2018-10-10 22:23 [PATCH] kbuild: Fail the build early when no lz4 present Borislav Petkov 2018-10-10 22:34 ` Randy Dunlap @ 2018-10-11 4:26 ` Masahiro Yamada 2018-10-11 8:52 ` Borislav Petkov 2018-10-11 20:33 ` kbuild test robot 2 siblings, 1 reply; 8+ messages in thread From: Masahiro Yamada @ 2018-10-11 4:26 UTC (permalink / raw) To: Borislav Petkov Cc: Linux Kernel Mailing List, Michal Marek, Linux Kbuild mailing list Hi Borislav, On Thu, Oct 11, 2018 at 7:23 AM Borislav Petkov <bp@alien8.de> wrote: > > From: Borislav Petkov <bp@suse.de> > > When building randconfigs, the build fails at kernel compression stage > due to missing lz4 on the system but CONFIG_KERNEL_LZ4 has been selected > by randconfig. The result looks somethins like this: > > (cat arch/x86/boot/compressed/vmlinux.bin arch/x86/boot/compressed/vmlinux.relocs | lz4c -l -c1 stdin stdout && printf \334\141\301\001) > arch/x86/boot/compressed/vmlinux.bin.lz4 || (rm -f arch/x86/boot/compressed/vmlinux.bin.lz4 ; false) > /bin/sh: 1: lz4c: not found So, the cause of the failure is clear enough from the build log. It is weird to check only lz4c. If CONFIG_KERNEL_LZO is enabled, but lzop is not installed, I see this log LZO arch/x86/boot/compressed/vmlinux.bin.lzo /bin/sh: 1: lzop: not found It is still clear what to do, though. > make[2]: *** [arch/x86/boot/compressed/Makefile:143: arch/x86/boot/compressed/vmlinux.bin.lz4] Error 1 > make[1]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2 > make: *** [arch/x86/Makefile:290: bzImage] Error 2 > > Fail the build much earlier by checking for lz4c presence before doing > anything else. Is it necessary to check this earlier? If you get this error, you just need to install the tool. Then, you can re-run the incremental build. BTW, this patch has a drawback. [1] Enable CONFIG_KERNEL_LZ4 on the system without lz4c installed [2] Run 'make' and you will get the error "lz4 tool not found on this system but CONFIG_KERNEL_LZ4 enabled" [3] Run 'make menuconfig' and switch from CONFIG_KERNEL_LZ4 to CONFIG_KERNEL_GZIP [4] Run 'make' and you will still get the same error even after you have chosen to use GZIP instead of LZ4. > Signed-off-by: Borislav Petkov <bp@suse.de> > Cc: Masahiro Yamada <yamada.masahiro@socionext.com> > Cc: Michal Marek <michal.lkml@markovi.net> > Cc: linux-kbuild@vger.kernel.org > --- > Makefile | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/Makefile b/Makefile > index a0c32de80482..f91de649234b 100644 > --- a/Makefile > +++ b/Makefile > @@ -788,6 +788,12 @@ KBUILD_CFLAGS_KERNEL += -ffunction-sections -fdata-sections > LDFLAGS_vmlinux += --gc-sections > endif > > +ifdef CONFIG_KERNEL_LZ4 > +ifeq ($(CONFIG_SHELL which lz4c),) > +$(error "lz4 tool not found on this system but CONFIG_KERNEL_LZ4 enabled") > +endif > +endif > + > # arch Makefile may override CC so keep this after arch Makefile is included > NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include) > > -- > 2.19.0.271.gfe8321ec057f > -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] kbuild: Fail the build early when no lz4 present 2018-10-11 4:26 ` Masahiro Yamada @ 2018-10-11 8:52 ` Borislav Petkov 2018-10-11 15:08 ` Masahiro Yamada 0 siblings, 1 reply; 8+ messages in thread From: Borislav Petkov @ 2018-10-11 8:52 UTC (permalink / raw) To: Masahiro Yamada Cc: Linux Kernel Mailing List, Michal Marek, Linux Kbuild mailing list On Thu, Oct 11, 2018 at 01:26:23PM +0900, Masahiro Yamada wrote: > So, the cause of the failure is clear enough > from the build log. Sure but it wasts unnecessary build cycles and appears only after all the compilation units have been produced. > It is weird to check only lz4c. > If CONFIG_KERNEL_LZO is enabled, but lzop is not installed, > I see this log > > LZO arch/x86/boot/compressed/vmlinux.bin.lzo > /bin/sh: 1: lzop: not found > > It is still clear what to do, though. Right, we should test for that too. Unless distros start installing it by default. It is not installed by default on a debian guest here, at least. SLES15 either: $ lzop If 'lzop' is not a typo you can use command-not-found to lookup the package that contains it, like this: cnf lzop > Is it necessary to check this earlier? Yes, see above. > If you get this error, you just need to install the tool. > Then, you can re-run the incremental build. Actually, I'd prefer the build to fail early so that machine can continue with the next build. > BTW, this patch has a drawback. > > [1] Enable CONFIG_KERNEL_LZ4 on the system > without lz4c installed > > [2] Run 'make' and you will get the error > "lz4 tool not found on this system but CONFIG_KERNEL_LZ4 enabled" > > [3] Run 'make menuconfig' and > switch from CONFIG_KERNEL_LZ4 to CONFIG_KERNEL_GZIP > > [4] Run 'make' and you will still get the same error > even after you have chosen to use GZIP instead of LZ4. Well, there's code in Makefile to refresh include/config/auto.conf but for whatever reason that make in step 4 doesn't run it. And *that* looks like a bug - if a user runs "make menuconfig" and .config gets updated, then include/config/auto.conf better get updated too. Right? Thx. -- Regards/Gruss, Boris. Good mailing practices for 400: avoid top-posting and trim the reply. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] kbuild: Fail the build early when no lz4 present 2018-10-11 8:52 ` Borislav Petkov @ 2018-10-11 15:08 ` Masahiro Yamada 2018-10-11 15:32 ` Borislav Petkov 0 siblings, 1 reply; 8+ messages in thread From: Masahiro Yamada @ 2018-10-11 15:08 UTC (permalink / raw) To: Borislav Petkov Cc: Linux Kernel Mailing List, Michal Marek, Linux Kbuild mailing list On Thu, Oct 11, 2018 at 5:54 PM Borislav Petkov <bp@alien8.de> wrote: > > On Thu, Oct 11, 2018 at 01:26:23PM +0900, Masahiro Yamada wrote: > > So, the cause of the failure is clear enough > > from the build log. > > Sure but it wasts unnecessary build cycles and appears only after all > the compilation units have been produced. Not a waste of time. Install lz4c, and run 'make' again. Almost all objects have been built there, so you will finish it soon. If you are building up a compile-test machine, you will probably need to install various tools anyway. I do not want to add ugly checks in random places. > > It is weird to check only lz4c. > > If CONFIG_KERNEL_LZO is enabled, but lzop is not installed, > > I see this log > > > > LZO arch/x86/boot/compressed/vmlinux.bin.lzo > > /bin/sh: 1: lzop: not found > > > > It is still clear what to do, though. > > Right, we should test for that too. Unless distros start installing it > by default. It is not installed by default on a debian guest here, at > least. > > SLES15 either: > > $ lzop > If 'lzop' is not a typo you can use command-not-found to lookup the package that contains it, like this: > cnf lzop > > > Is it necessary to check this earlier? > > Yes, see above. > > > If you get this error, you just need to install the tool. > > Then, you can re-run the incremental build. > > Actually, I'd prefer the build to fail early so that machine can > continue with the next build. > > > BTW, this patch has a drawback. > > > > [1] Enable CONFIG_KERNEL_LZ4 on the system > > without lz4c installed > > > > [2] Run 'make' and you will get the error > > "lz4 tool not found on this system but CONFIG_KERNEL_LZ4 enabled" > > > > [3] Run 'make menuconfig' and > > switch from CONFIG_KERNEL_LZ4 to CONFIG_KERNEL_GZIP > > > > [4] Run 'make' and you will still get the same error > > even after you have chosen to use GZIP instead of LZ4. > > Well, there's code in Makefile to refresh include/config/auto.conf but > for whatever reason that make in step 4 doesn't run it. > > And *that* looks like a bug - if a user runs "make menuconfig" and > .config gets updated, then include/config/auto.conf better get updated > too. Right? Not a bug. include/config/auto.conf gets updates. But the top Makefile is parsed twice with a stale auto.conf, and with a fresh auto.conf It is wrong to add $(error ...) in the plain context in the top Makefile. > Thx. > > -- > Regards/Gruss, > Boris. > > Good mailing practices for 400: avoid top-posting and trim the reply. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] kbuild: Fail the build early when no lz4 present 2018-10-11 15:08 ` Masahiro Yamada @ 2018-10-11 15:32 ` Borislav Petkov 2018-10-16 9:17 ` Masahiro Yamada 0 siblings, 1 reply; 8+ messages in thread From: Borislav Petkov @ 2018-10-11 15:32 UTC (permalink / raw) To: Masahiro Yamada Cc: Linux Kernel Mailing List, Michal Marek, Linux Kbuild mailing list On Fri, Oct 12, 2018 at 12:08:07AM +0900, Masahiro Yamada wrote: > Install lz4c, and run 'make' again. > Almost all objects have been built there, > so you will finish it soon. So when this LZ4 thing got added at the time, the lz4 package had to be downloaded and built and installed and it wasn't as easy as zypper in lz4 or apt-get install lz4. And I *might* install it if there were a human readable error message which would tell me so. And we should strive to be as user-friendly as possible. And no, this: /bin/sh: 1: lz4c: not found make[2]: *** [arch/x86/boot/compressed/Makefile:145: arch/x86/boot/compressed/vmlinux.bin.lz4] Error 1 make[2]: *** Waiting for unfinished jobs.... make[1]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2 make: *** [arch/x86/Makefile:290: bzImage] Error 2 make: *** Waiting for unfinished jobs.... is not user-friendly. > If you are building up a compile-test machine, > you will probably need to install various tools anyway. Yes, and look how perf tool solves this. Much much better. > I do not want to add ugly checks in random places. That's fair. What would be a fitting place to add such checks and be able to issue a human-readable error message to people? Btw, we fail the same cryptic way when there's no openSSL headers installed on the system: --- scripts/sign-file.c:25:10: fatal error: openssl/opensslv.h: No such file or directory #include <openssl/opensslv.h> ^~~~~~~~~~~~~~~~~~~~ compilation terminated. make[1]: *** [scripts/Makefile.host:90: scripts/sign-file] Error 1 make[1]: *** Waiting for unfinished jobs.... scripts/extract-cert.c:21:10: fatal error: openssl/bio.h: No such file or directory #include <openssl/bio.h> ^~~~~~~~~~~~~~~ compilation terminated. make[1]: *** [scripts/Makefile.host:90: scripts/extract-cert] Error 1 make: *** [Makefile:1065: scripts] Error 2 make: *** Waiting for unfinished jobs.... --- and all those beginners who are trying to build the kernel for the first time would have hard time figuring out what's expected of them. Now look at perf tool: Makefile.config:445: No sys/sdt.h found, no SDT events are defined, please install systemtap-sdt-devel or systemtap-sdt-dev Makefile.config:491: No libunwind found. Please install libunwind-dev[el] >= 1.1 and/or set LIBUNWIND_DIR Makefile.config:721: No bfd.h/libbfd found, please install binutils-dev[el]/zlib-static/libiberty-dev to gain symbol demangling Much better IMO. Thx. -- Regards/Gruss, Boris. Good mailing practices for 400: avoid top-posting and trim the reply. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] kbuild: Fail the build early when no lz4 present 2018-10-11 15:32 ` Borislav Petkov @ 2018-10-16 9:17 ` Masahiro Yamada 0 siblings, 0 replies; 8+ messages in thread From: Masahiro Yamada @ 2018-10-16 9:17 UTC (permalink / raw) To: Borislav Petkov Cc: Linux Kernel Mailing List, Michal Marek, Linux Kbuild mailing list On Fri, Oct 12, 2018 at 12:33 AM Borislav Petkov <bp@alien8.de> wrote: > > On Fri, Oct 12, 2018 at 12:08:07AM +0900, Masahiro Yamada wrote: > > Install lz4c, and run 'make' again. > > Almost all objects have been built there, > > so you will finish it soon. > > So when this LZ4 thing got added at the time, the lz4 package had to be > downloaded and built and installed and it wasn't as easy as zypper in > lz4 or apt-get install lz4. > > And I *might* install it if there were a human readable error message > which would tell me so. > > And we should strive to be as user-friendly as possible. And no, this: > > /bin/sh: 1: lz4c: not found > make[2]: *** [arch/x86/boot/compressed/Makefile:145: arch/x86/boot/compressed/vmlinux.bin.lz4] Error 1 > make[2]: *** Waiting for unfinished jobs.... > make[1]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2 > make: *** [arch/x86/Makefile:290: bzImage] Error 2 > make: *** Waiting for unfinished jobs.... > > is not user-friendly. > > > If you are building up a compile-test machine, > > you will probably need to install various tools anyway. > > Yes, and look how perf tool solves this. Much much better. > > > I do not want to add ugly checks in random places. > > That's fair. > > What would be a fitting place to add such checks and be able to issue a > human-readable error message to people? Sorry for delay. I have been thinking of this because I have a motivation to move libelf check in the top Makefile to somewhere else. I just made a trial. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] kbuild: Fail the build early when no lz4 present 2018-10-10 22:23 [PATCH] kbuild: Fail the build early when no lz4 present Borislav Petkov 2018-10-10 22:34 ` Randy Dunlap 2018-10-11 4:26 ` Masahiro Yamada @ 2018-10-11 20:33 ` kbuild test robot 2 siblings, 0 replies; 8+ messages in thread From: kbuild test robot @ 2018-10-11 20:33 UTC (permalink / raw) To: Borislav Petkov Cc: kbuild-all, LKML, Masahiro Yamada, Michal Marek, linux-kbuild [-- Attachment #1: Type: text/plain, Size: 1115 bytes --] Hi Borislav, I love your patch! Yet something to improve: [auto build test ERROR on linus/master] [also build test ERROR on v4.19-rc7 next-20181011] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system] url: https://github.com/0day-ci/linux/commits/Borislav-Petkov/kbuild-Fail-the-build-early-when-no-lz4-present/20181012-013752 config: i386-randconfig-b0-10112221 (attached as .config) compiler: gcc-4.9 (Debian 4.9.4-2) 4.9.4 reproduce: # save the attached .config to linux build tree make ARCH=i386 All errors (new ones prefixed by >>): >> Makefile:793: *** "lz4 tool not found on this system but CONFIG_KERNEL_LZ4 enabled". Stop. make: *** [sub-make] Error 2 vim +793 Makefile 790 791 ifdef CONFIG_KERNEL_LZ4 792 ifeq ($(CONFIG_SHELL which lz4c),) > 793 $(error "lz4 tool not found on this system but CONFIG_KERNEL_LZ4 enabled") 794 endif 795 endif 796 --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/kbuild-all Intel Corporation [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 31850 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-10-16 9:18 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-10 22:23 [PATCH] kbuild: Fail the build early when no lz4 present Borislav Petkov 2018-10-10 22:34 ` Randy Dunlap 2018-10-11 4:26 ` Masahiro Yamada 2018-10-11 8:52 ` Borislav Petkov 2018-10-11 15:08 ` Masahiro Yamada 2018-10-11 15:32 ` Borislav Petkov 2018-10-16 9:17 ` Masahiro Yamada 2018-10-11 20:33 ` kbuild test robot
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).