[PATCH] kbuild: Run syncconfig with -s

[PATCH] kbuild: Run syncconfig with -s

[Ingo Molnar]

On every kernel build that runs --syncconfig, there's an output of the following line:

  scripts/kconfig/conf  --syncconfig Kconfig

This is the only non-platform build message the kbuild system emits that isn't
prefixed by at least a space, or is a build warning.

Run it under -s - if there's any problem it will emit messages anyway.

With this change the following simple grep filter will show all build warnings
and errors of a kernel build:

   make | grep -v '^ '

Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 9cac6fde3479..2057c92a6205 100644
--- a/Makefile
+++ b/Makefile
@@ -709,7 +709,7 @@ $(KCONFIG_CONFIG):
 # The syncconfig should be executed only once to make all the targets.
 # (Note: use the grouped target '&:' when we bump to GNU Make 4.3)
 %/config/auto.conf %/config/auto.conf.cmd %/generated/autoconf.h: $(KCONFIG_CONFIG)
-	$(Q)$(MAKE) -f $(srctree)/Makefile syncconfig
+	$(Q)$(MAKE) -sf $(srctree)/Makefile syncconfig
 else # !may-sync-config
 # External modules and some install targets need include/generated/autoconf.h
 # and include/config/auto.conf but do not care if they are up-to-date.

Re: [PATCH] kbuild: Run syncconfig with -s

[Masahiro Yamada]

On Thu, Aug 20, 2020 at 3:35 PM Ingo Molnar <mingo@kernel.org> wrote:
>
> On every kernel build that runs --syncconfig, there's an output of the following line:
>
>   scripts/kconfig/conf  --syncconfig Kconfig
>
> This is the only non-platform build message the kbuild system emits that isn't
> prefixed by at least a space, or is a build warning.
>
> Run it under -s - if there's any problem it will emit messages anyway.
>
> With this change the following simple grep filter will show all build warnings
> and errors of a kernel build:
>
>    make | grep -v '^ '



I do want to see something when syncconfig is invoked.

I will apply this instead:
https://patchwork.kernel.org/patch/11727445/




> Signed-off-by: Ingo Molnar <mingo@kernel.org>
> ---
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index 9cac6fde3479..2057c92a6205 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -709,7 +709,7 @@ $(KCONFIG_CONFIG):
>  # The syncconfig should be executed only once to make all the targets.
>  # (Note: use the grouped target '&:' when we bump to GNU Make 4.3)
>  %/config/auto.conf %/config/auto.conf.cmd %/generated/autoconf.h: $(KCONFIG_CONFIG)
> -       $(Q)$(MAKE) -f $(srctree)/Makefile syncconfig
> +       $(Q)$(MAKE) -sf $(srctree)/Makefile syncconfig
>  else # !may-sync-config
>  # External modules and some install targets need include/generated/autoconf.h
>  # and include/config/auto.conf but do not care if they are up-to-date.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH] kbuild: Run syncconfig with -s

[Ingo Molnar]

* Masahiro Yamada <masahiroy@kernel.org> wrote:

> On Thu, Aug 20, 2020 at 3:35 PM Ingo Molnar <mingo@kernel.org> wrote:
> >
> > On every kernel build that runs --syncconfig, there's an output of the following line:
> >
> >   scripts/kconfig/conf  --syncconfig Kconfig
> >
> > This is the only non-platform build message the kbuild system emits that isn't
> > prefixed by at least a space, or is a build warning.
> >
> > Run it under -s - if there's any problem it will emit messages anyway.
> >
> > With this change the following simple grep filter will show all build warnings
> > and errors of a kernel build:
> >
> >    make | grep -v '^ '
> 
> 
> 
> I do want to see something when syncconfig is invoked.
> 
> I will apply this instead:
> https://patchwork.kernel.org/patch/11727445/

That's perfect, thank you very much!

	Ingo

Re: [PATCH] kbuild: Run syncconfig with -s

[Ingo Molnar]

* Masahiro Yamada <masahiroy@kernel.org> wrote:

> On Thu, Aug 20, 2020 at 3:35 PM Ingo Molnar <mingo@kernel.org> wrote:
> >
> > On every kernel build that runs --syncconfig, there's an output of the following line:
> >
> >   scripts/kconfig/conf  --syncconfig Kconfig
> >
> > This is the only non-platform build message the kbuild system emits that isn't
> > prefixed by at least a space, or is a build warning.
> >
> > Run it under -s - if there's any problem it will emit messages anyway.
> >
> > With this change the following simple grep filter will show all build warnings
> > and errors of a kernel build:
> >
> >    make | grep -v '^ '
> 
> 
> 
> I do want to see something when syncconfig is invoked.
> 
> I will apply this instead:
> https://patchwork.kernel.org/patch/11727445/

BTW., there's another, rather spurious bug I recently triggered in kbuild.

Occasionally when I Ctrl-C a kernel build on a system with a lot of CPUs, 
the .o.cmd file gets corrupted:

  mm/.pgtable-generic.o.cmd:5: *** unterminated call to function 'wildcard': missing ')'.  Stop.
  make: *** [Makefile:1788: mm] Error 2
  make: *** Waiting for unfinished jobs....

The .o.cmd file is half-finished:

    $(wildcard include/config/shmem.h) \
    $(wildcard include/config/hugetlb/page.h) \
    $(wildcard include/config/zone/device.h) \
    $(wildcard include/config/dev/pagemap/ops.h) \
    $(wildcard include/config/device/private.h) \
    $(wildcard include/config/pci/p2pdma.h) \
    $(wildcard include/config/sparsemem.h) \
    $(wildcard include/config/sparsemem/vmemmap.h) \
    $(wildcard include/config/numa/balancing.h) \
    $(wildcard i
    [premature EOF]

Instead of the regular rules that end in:

    $(wildcard include/config/memory/hotplug/sparse.h) \

    mm/pgtable-generic.o: $(deps_mm/pgtable-generic.o)

    $(deps_mm/pgtable-generic.o):
    [regular EOF]

Manually removing the corrupted .o.cmd dot file solves the bug.

There's no reproducer other than Ctrl-C-ing large build jobs a couple of times.

Thanks,

	Ingo

Re: [PATCH] kbuild: Run syncconfig with -s

[Masahiro Yamada]

On Mon, Sep 14, 2020 at 9:37 PM Ingo Molnar <mingo@kernel.org> wrote:
>
>
> * Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> > On Thu, Aug 20, 2020 at 3:35 PM Ingo Molnar <mingo@kernel.org> wrote:
> > >
> > > On every kernel build that runs --syncconfig, there's an output of the following line:
> > >
> > >   scripts/kconfig/conf  --syncconfig Kconfig
> > >
> > > This is the only non-platform build message the kbuild system emits that isn't
> > > prefixed by at least a space, or is a build warning.
> > >
> > > Run it under -s - if there's any problem it will emit messages anyway.
> > >
> > > With this change the following simple grep filter will show all build warnings
> > > and errors of a kernel build:
> > >
> > >    make | grep -v '^ '
> >
> >
> >
> > I do want to see something when syncconfig is invoked.
> >
> > I will apply this instead:
> > https://patchwork.kernel.org/patch/11727445/
>
> BTW., there's another, rather spurious bug I recently triggered in kbuild.
>
> Occasionally when I Ctrl-C a kernel build on a system with a lot of CPUs,
> the .o.cmd file gets corrupted:
>
>   mm/.pgtable-generic.o.cmd:5: *** unterminated call to function 'wildcard': missing ')'.  Stop.
>   make: *** [Makefile:1788: mm] Error 2
>   make: *** Waiting for unfinished jobs....
>
> The .o.cmd file is half-finished:
>
>     $(wildcard include/config/shmem.h) \
>     $(wildcard include/config/hugetlb/page.h) \
>     $(wildcard include/config/zone/device.h) \
>     $(wildcard include/config/dev/pagemap/ops.h) \
>     $(wildcard include/config/device/private.h) \
>     $(wildcard include/config/pci/p2pdma.h) \
>     $(wildcard include/config/sparsemem.h) \
>     $(wildcard include/config/sparsemem/vmemmap.h) \
>     $(wildcard include/config/numa/balancing.h) \
>     $(wildcard i
>     [premature EOF]


Hmm, previously fixdep was not robust against
errors or interruption.
So, I occasionally observed the same problem as you saw.


I fixed it in
6f9ac9f4427ec0470ccffbf852cfaf326677cc21


Since then, I have not seen
the broken .cmd file issue.


I repeated 'make -j256' and Ctrl-C,
but I could not reproduce it...



If GNU Make was interrupted during the build rule
of mm/pgtable-generic.o, GNU Make should automatically
have deleted mm/pgtable-generic.o because
partially updated (i.e. corrupted) files are
the source of all the troubles.

Isn't this the case for you?


If I interrupt Kbuild, I see several
Deleting file '.....'
messages, like follows:



  CC      fs/seq_file.o
  CC      drivers/acpi/dock.o
  CC      drivers/scsi/scsi_devinfo.o
  CC      fs/xattr.o
^Cmake[1]: *** [scripts/Makefile.build:283: fs/seq_file.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: net/ipv6/af_inet6.o] Interrupt
make[3]: *** [scripts/Makefile.build:283:
drivers/net/phy/mdio-boardinfo.o] Interrupt
make[4]: *** [scripts/Makefile.build:283:
drivers/tty/serial/8250/8250_pnp.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: security/selinux/status.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: arch/x86/kernel/dumpstack.o] Interrupt
make[2]: *** Deleting file 'drivers/acpi/resource.o'
make[2]: *** [scripts/Makefile.build:500: drivers/net/phy] Interrupt
make[2]: *** [scripts/Makefile.build:283: net/sched/act_api.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: drivers/scsi/scsi_devinfo.o] Interrupt
make[1]: *** Deleting file 'arch/x86/pci/mmconfig_64.o'
make[1]: *** [scripts/Makefile.build:500: net/ipv6] Interrupt
make[2]: *** [scripts/Makefile.build:283: net/ethtool/wol.o] Interrupt
make[1]: *** Deleting file 'arch/x86/pci/init.o'
make[1]: *** [scripts/Makefile.build:283: arch/x86/pci/mmconfig_64.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: net/sunrpc/socklib.o] Interrupt
make[1]: *** [scripts/Makefile.build:283: fs/xattr.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: sound/hda/hdac_sysfs.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: net/mac80211/main.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: drivers/input/input-mt.o] Interrupt
make[2]: *** [scripts/Makefile.build:283: net/rfkill/core.o] Interrupt
make[1]: *** [scripts/Makefile.build:283: lib/bug.o] Interrupt
make: *** [Makefile:1792: lib] Interrupt
make[2]: *** Deleting file 'drivers/rtc/systohc.o'




> Instead of the regular rules that end in:
>
>     $(wildcard include/config/memory/hotplug/sparse.h) \
>
>     mm/pgtable-generic.o: $(deps_mm/pgtable-generic.o)
>
>     $(deps_mm/pgtable-generic.o):
>     [regular EOF]
>
> Manually removing the corrupted .o.cmd dot file solves the bug.
>
> There's no reproducer other than Ctrl-C-ing large build jobs a couple of times.
>
> Thanks,
>
>         Ingo


--
Best Regards
Masahiro Yamada

Re: [PATCH] kbuild: Run syncconfig with -s

[Alexey Dobriyan]

> BTW., there's another, rather spurious bug I recently triggered in kbuild.
> 
> Occasionally when I Ctrl-C a kernel build on a system with a lot of CPUs, 
> the .o.cmd file gets corrupted:

Those are temporary files, truncated at page boundary.

	$ stat -c %s XXX.pata_sil680.mod.o.cmd
	12288

I tried to fix this by inserting shell 'trap' directive but it failed
somewhere else.

	cmd_and_fixdep =                                                     \
        $(cmd);                                                              \
        scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).cmd;\
        rm -f $(depfile)

Re: [PATCH] kbuild: Run syncconfig with -s

[Masahiro Yamada]

On Mon, Sep 14, 2020 at 10:36 PM Alexey Dobriyan <adobriyan@gmail.com> wrote:
>
> > BTW., there's another, rather spurious bug I recently triggered in kbuild.
> >
> > Occasionally when I Ctrl-C a kernel build on a system with a lot of CPUs,
> > the .o.cmd file gets corrupted:
>
> Those are temporary files, truncated at page boundary.
>
>         $ stat -c %s XXX.pata_sil680.mod.o.cmd
>         12288
>
> I tried to fix this by inserting shell 'trap' directive but it failed
> somewhere else.
>
>         cmd_and_fixdep =                                                     \
>         $(cmd);                                                              \
>         scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).cmd;\
>         rm -f $(depfile)



No. It is *OK* to have broken .o.cmd files
as long as .o files are deleted on error or interruption
(and this should be automatically handled by GNU Make).


.o.cmd is included only when the corresponding .o exists.


This is explained in the case [2] of commit
392885ee82d35d515ba2af7b72c5e357c3002113



-- 
Best Regards
Masahiro Yamada