[GIT PULL 1/2] Kbuild updates for v5.6-rc1

[GIT PULL 1/2] Kbuild updates for v5.6-rc1

[Masahiro Yamada]

Hi Linus,

Please pull Kbuild updates for v5.6-rc1.

Thanks!


The following changes since commit c79f46a282390e0f5b306007bf7b11a46d529538:

  Linux 5.5-rc5 (2020-01-05 14:23:27 -0800)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
tags/kbuild-v5.6

for you to fetch changes up to 451dff37f0752cc8ad6f1bb82081a98d7b760310:

  builddeb: split libc headers deployment out into a function
(2020-01-29 01:51:30 +0900)

----------------------------------------------------------------
Kbuild updates for v5.6

 - detect missing include guard in UAPI headers

 - do not create orphan built-in.a or obj-y objects

 - generate modules.builtin more simply, and drop tristate.conf

 - simplify built-in initramfs creation

 - make linux-headers deb package thinner

 - optimize the deb package build script

 - misc cleanups

----------------------------------------------------------------
Masahiro Yamada (32):
      kbuild: detect missing include guard for exported headers
      kbuild: rename header-test- to no-header-test in usr/include/Makefile
      kbuild: do not create orphan built-in.a or obj-y objects
      kbuild: use pattern rule for building built-in.a in sub-directories
      kbuild: add stringify helper to quote a string passed to C files
      kbuild: pass KBUILD_MODFILE when compiling builtin objects
      kbuild: create modules.builtin without Makefile.modbuiltin or
tristate.conf
      initramfs: replace klibcdirs in Makefile with FORCE
      gen_initramfs_list.sh: remove unused variable 'default_list'
      gen_initramfs_list.sh: fix the tool name in the comment
      initramfs: rename gen_initramfs_list.sh to gen_initramfs.sh
      initramfs: remove redundant dependency on BLK_DEV_INITRD
      initramfs: make compression options not depend on INITRAMFS_SOURCE
      initramfs: make initramfs compression choice non-optional
      initramfs: specify $(src)/gen_initramfs.sh as a prerequisite in Makefile
      initramfs: generate dependency list and cpio at the same time
      initramfs: add default_cpio_list, and delete -d option support
      gen_initramfs.sh: always output cpio even without -o option
      initramfs: refactor the initramfs build rules
      gen_initramfs.sh: remove intermediate cpio_list on errors
      modpost: assume STT_SPARC_REGISTER is defined
      kbuild: remove PYTHON2 variable
      kbuild: remove *.tmp file when filechk fails
      kbuild: remove 'Building modules, stage 2.' log
      kbuild: use -S instead of -E for precise cc-option test in Kconfig
      builddeb: remove unneeded files in hdrobjfiles for headers package
      builddeb: match temporary directory name to the package name
      builddeb: remove redundant $objtree/
      builddeb: avoid invoking sub-shells where possible
      builddeb: remove redundant make for ARCH=um
      builddeb: split kernel headers deployment out into a function
      builddeb: split libc headers deployment out into a function

Michał Mirosław (2):
      builddeb: make headers package thinner
      builddeb: allow selection of .deb compressor

 Documentation/kbuild/kconfig.rst                |   5 -
 Makefile                                        |  24 +---
 include/linux/module.h                          |  12 +-
 scripts/Kbuild.include                          |  25 ++--
 scripts/Kconfig.include                         |   2 +-
 scripts/Makefile.build                          |   4 +-
 scripts/Makefile.lib                            |  21 ++-
 scripts/Makefile.modbuiltin                     |  57 --------
 scripts/Makefile.modpost                        |   1 -
 scripts/kconfig/confdata.c                      |  45 +-----
 scripts/link-vmlinux.sh                         |   4 +
 scripts/mod/modpost.c                           |   8 +-
 scripts/package/builddeb                        | 106 ++++++++------
 usr/.gitignore                                  |   8 +-
 usr/Kconfig                                     |  26 ----
 usr/Makefile                                    |  97 ++++++++-----
 usr/default_cpio_list                           |   6 +
 usr/{gen_initramfs_list.sh => gen_initramfs.sh} | 167 ++++++----------------
 usr/include/Makefile                            | 107 +++++++-------
 usr/initramfs_data.S                            |   5 +-
 20 files changed, 281 insertions(+), 449 deletions(-)
 delete mode 100644 scripts/Makefile.modbuiltin
 create mode 100644 usr/default_cpio_list
 rename usr/{gen_initramfs_list.sh => gen_initramfs.sh} (53%)


-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v5.6-rc1

[pr-tracker-bot]

The pull request you sent on Sat, 1 Feb 2020 13:05:32 +0900:

> git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git tags/kbuild-v5.6

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/368d060bb497775e42e323b9f676c45e2011da27

Thank you!

-- 
Deet-doot-dot, I am a bot.
https://korg.wiki.kernel.org/userdoc/prtracker

Re: [GIT PULL 1/2] Kbuild updates for v5.6-rc1

[Linus Torvalds]

On Fri, Jan 31, 2020 at 8:06 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
>  - simplify built-in initramfs creation

Hmm.

This may simplify it from a _technical_ angle, but it seems to be a
fairly annoying step backwards from a UI perspective.

Now Kconfig asks a completely pointless question that most people have
absolutely zero interest in. The old situation was better, I feel.

Basically, I feel that from a "get normal users to test development
kernels", our Kconfig pain ends up being the biggest hurdle by far.

The kernel is easy to build and doesn't really require all that much
infrastructure, but generating the config - particularly when it
changes over time and you can't just say "just use the distro config"
- is a big step for people.

So honestly, while I've pulled this, I feel that this kind of change
is going _exactly_ the wrong way when it asks people questions that
they don't care one whit about.

If I as a kernel developer can't find it in myself to care and go "why
does it ask this new question", then that should tell you something.

Why do we have this choice in the first place? And no, it's not a
"simplification" to make life more complex for users.

                   Linus

Re: [GIT PULL 1/2] Kbuild updates for v5.6-rc1

[Masahiro Yamada]

Hi Linus,

On Sun, Feb 2, 2020 at 3:45 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Fri, Jan 31, 2020 at 8:06 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> >  - simplify built-in initramfs creation
>
> Hmm.
>
> This may simplify it from a _technical_ angle, but it seems to be a
> fairly annoying step backwards from a UI perspective.
>
> Now Kconfig asks a completely pointless question that most people have
> absolutely zero interest in. The old situation was better, I feel.
>
> Basically, I feel that from a "get normal users to test development
> kernels", our Kconfig pain ends up being the biggest hurdle by far.
>
> The kernel is easy to build and doesn't really require all that much
> infrastructure, but generating the config - particularly when it
> changes over time and you can't just say "just use the distro config"
> - is a big step for people.
>
> So honestly, while I've pulled this, I feel that this kind of change
> is going _exactly_ the wrong way when it asks people questions that
> they don't care one whit about.
>
> If I as a kernel developer can't find it in myself to care and go "why
> does it ask this new question", then that should tell you something.
>
> Why do we have this choice in the first place?



Generally, initramfs is passed from a boot-loader,
but some architectures embed initramfs into vmlinux
(perhaps due to poor boot-loader support??)

arch/arc/configs/tb10x_defconfig:CONFIG_INITRAMFS_SOURCE="../tb10x-rootfs.cpio"
arch/unicore32/configs/defconfig:#CONFIG_INITRAMFS_SOURCE="arch/unicore/ramfs/ramfs_config"
arch/xtensa/configs/cadence_csp_defconfig:CONFIG_INITRAMFS_SOURCE="$$KERNEL_INITRAMFS_SOURCE"

So, data-compression is useful - that's is what I understand.


For major architectures, vmlinux embeds a tiny initramfs,
which is generated based on usr/default_cpio_list.

We do not need data-compression for such a small cpio,
but handling it in a consistent way is sensible.
This is annoying from the users' PoV, I admit.



> And no, it's not a
> "simplification" to make life more complex for users.
>
>                    Linus



-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v5.6-rc1

[Geert Uytterhoeven]

Hi Yamada-san,

On Mon, Feb 3, 2020 at 2:28 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> On Sun, Feb 2, 2020 at 3:45 AM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> > On Fri, Jan 31, 2020 at 8:06 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> > >
> > >  - simplify built-in initramfs creation
> >
> > Hmm.
> >
> > This may simplify it from a _technical_ angle, but it seems to be a
> > fairly annoying step backwards from a UI perspective.
> >
> > Now Kconfig asks a completely pointless question that most people have
> > absolutely zero interest in. The old situation was better, I feel.
> >
> > Basically, I feel that from a "get normal users to test development
> > kernels", our Kconfig pain ends up being the biggest hurdle by far.
> >
> > The kernel is easy to build and doesn't really require all that much
> > infrastructure, but generating the config - particularly when it
> > changes over time and you can't just say "just use the distro config"
> > - is a big step for people.
> >
> > So honestly, while I've pulled this, I feel that this kind of change
> > is going _exactly_ the wrong way when it asks people questions that
> > they don't care one whit about.
> >
> > If I as a kernel developer can't find it in myself to care and go "why
> > does it ask this new question", then that should tell you something.
> >
> > Why do we have this choice in the first place?
>
> Generally, initramfs is passed from a boot-loader,
> but some architectures embed initramfs into vmlinux
> (perhaps due to poor boot-loader support??)
>
> arch/arc/configs/tb10x_defconfig:CONFIG_INITRAMFS_SOURCE="../tb10x-rootfs.cpio"
> arch/unicore32/configs/defconfig:#CONFIG_INITRAMFS_SOURCE="arch/unicore/ramfs/ramfs_config"
> arch/xtensa/configs/cadence_csp_defconfig:CONFIG_INITRAMFS_SOURCE="$$KERNEL_INITRAMFS_SOURCE"

Note that the above are examples that do not actually work, as the files
referred to are not present in mainline (read below[1] why I have just
checked that ;-).

> So, data-compression is useful - that's is what I understand.

Yes it is, depending on your config.

> For major architectures, vmlinux embeds a tiny initramfs,
> which is generated based on usr/default_cpio_list.
>
> We do not need data-compression for such a small cpio,
> but handling it in a consistent way is sensible.
> This is annoying from the users' PoV, I admit.

I was also confused by this question, as by default you have
    CONFIG_RD_GZIP=y
    CONFIG_RD_BZIP2=y
    CONFIG_RD_LZMA=y
    CONFIG_RD_XZ=y
    CONFIG_RD_LZO=y
    CONFIG_RD_LZ4=y
so the old 'default ".gz" if RD_GZIP' looked like it would use gzip.
However, the tiny default initramfs ended up being uncompressed anyway
before, as until commit ddd09bcc899fd374 ("initramfs: make compression
options not depend on INITRAMFS_SOURCE"), INITRAMFS_COMPRESSION wasn't
taken into account for the default tiny initramfs...

So INITRAMFS_COMPRESSION_NONE is the right answer to retain the
old behavior? One might question why not to use gzip anyway, as
CONFIG_RD_GZIP=y is enabled by default, and would give a (small)
improvement of ca. 350 bytes ;-)
Hence there is some area for improvement...

Thanks!

[1] I'm still carrying a local patch for handling relative initramfs
    paths with O=, but it's been a while I actually used it.  Due to your
    recent changes, it no longer applies, and needs to be updated.
    But of course that is only useful if there are some real users...
    https://lore.kernel.org/lkml/1384467283-14806-1-git-send-email-geert@linux-m68k.org/

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [GIT PULL 1/2] Kbuild updates for v5.6-rc1

[Masahiro Yamada]

Hi Geert,

On Mon, Feb 3, 2020 at 8:00 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>
> Hi Yamada-san,
>
> On Mon, Feb 3, 2020 at 2:28 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> > On Sun, Feb 2, 2020 at 3:45 AM Linus Torvalds
> > <torvalds@linux-foundation.org> wrote:
> > > On Fri, Jan 31, 2020 at 8:06 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> > > >
> > > >  - simplify built-in initramfs creation
> > >
> > > Hmm.
> > >
> > > This may simplify it from a _technical_ angle, but it seems to be a
> > > fairly annoying step backwards from a UI perspective.
> > >
> > > Now Kconfig asks a completely pointless question that most people have
> > > absolutely zero interest in. The old situation was better, I feel.
> > >
> > > Basically, I feel that from a "get normal users to test development
> > > kernels", our Kconfig pain ends up being the biggest hurdle by far.
> > >
> > > The kernel is easy to build and doesn't really require all that much
> > > infrastructure, but generating the config - particularly when it
> > > changes over time and you can't just say "just use the distro config"
> > > - is a big step for people.
> > >
> > > So honestly, while I've pulled this, I feel that this kind of change
> > > is going _exactly_ the wrong way when it asks people questions that
> > > they don't care one whit about.
> > >
> > > If I as a kernel developer can't find it in myself to care and go "why
> > > does it ask this new question", then that should tell you something.
> > >
> > > Why do we have this choice in the first place?
> >
> > Generally, initramfs is passed from a boot-loader,
> > but some architectures embed initramfs into vmlinux
> > (perhaps due to poor boot-loader support??)
> >
> > arch/arc/configs/tb10x_defconfig:CONFIG_INITRAMFS_SOURCE="../tb10x-rootfs.cpio"
> > arch/unicore32/configs/defconfig:#CONFIG_INITRAMFS_SOURCE="arch/unicore/ramfs/ramfs_config"
> > arch/xtensa/configs/cadence_csp_defconfig:CONFIG_INITRAMFS_SOURCE="$$KERNEL_INITRAMFS_SOURCE"
>
> Note that the above are examples that do not actually work, as the files
> referred to are not present in mainline (read below[1] why I have just
> checked that ;-).
>
> > So, data-compression is useful - that's is what I understand.
>
> Yes it is, depending on your config.
>
> > For major architectures, vmlinux embeds a tiny initramfs,
> > which is generated based on usr/default_cpio_list.
> >
> > We do not need data-compression for such a small cpio,
> > but handling it in a consistent way is sensible.
> > This is annoying from the users' PoV, I admit.
>
> I was also confused by this question, as by default you have
>     CONFIG_RD_GZIP=y
>     CONFIG_RD_BZIP2=y
>     CONFIG_RD_LZMA=y
>     CONFIG_RD_XZ=y
>     CONFIG_RD_LZO=y
>     CONFIG_RD_LZ4=y
> so the old 'default ".gz" if RD_GZIP' looked like it would use gzip.
> However, the tiny default initramfs ended up being uncompressed anyway
> before, as until commit ddd09bcc899fd374 ("initramfs: make compression
> options not depend on INITRAMFS_SOURCE"), INITRAMFS_COMPRESSION wasn't
> taken into account for the default tiny initramfs...


Because there was a bug in old usr/Kconfig.


See the following dependency in the old code:

config INITRAMFS_COMPRESSION
        depends on INITRAMFS_SOURCE!=""


In the default .config, INITRAMFS_SOURCE is "".
Hence, CONFIG_INITRAMFS_COMPRESSION is undefined.
So, suffix_y gets empty in usr/Makefile.

That is why the cpio was not compressed
even though CONFIG_RD_GZIP=y



> So INITRAMFS_COMPRESSION_NONE is the right answer to retain the
> old behavior?

Yes, INITRAMFS_COMPRESSION_NONE retains the previous behavior.

But, as far as he I understood what Linus said,
"we do not care".


> One might question why not to use gzip anyway, as
> CONFIG_RD_GZIP=y is enabled by default, and would give a (small)
> improvement of ca. 350 bytes ;-)
> Hence there is some area for improvement...


GZIP is not the only compression algorithm.

Somebody may want to disable RD_GZIP,
then enable RD_XZ.
If we allow the data compression,
Kconfig must ask "which compression algorithm".


So, if Kconfig would siltently choose something
as default, INITRAMFS_COMPRESSION_NONE would be the best.




>
> Thanks!
>
> [1] I'm still carrying a local patch for handling relative initramfs
>     paths with O=, but it's been a while I actually used it.  Due to your
>     recent changes, it no longer applies, and needs to be updated.
>     But of course that is only useful if there are some real users...
>     https://lore.kernel.org/lkml/1384467283-14806-1-git-send-email-geert@linux-m68k.org/
>
> Gr{oetje,eeting}s,
>
>                         Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds



-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v5.6-rc1

[Geert Uytterhoeven]

Hi Yamada-san,

On Mon, Feb 3, 2020 at 12:38 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> On Mon, Feb 3, 2020 at 8:00 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> > On Mon, Feb 3, 2020 at 2:28 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> > > On Sun, Feb 2, 2020 at 3:45 AM Linus Torvalds
> > > <torvalds@linux-foundation.org> wrote:
> > > > On Fri, Jan 31, 2020 at 8:06 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> > > > >
> > > > >  - simplify built-in initramfs creation
> > > >
> > > > Hmm.
> > > >
> > > > This may simplify it from a _technical_ angle, but it seems to be a
> > > > fairly annoying step backwards from a UI perspective.
> > > >
> > > > Now Kconfig asks a completely pointless question that most people have
> > > > absolutely zero interest in. The old situation was better, I feel.
> > > >
> > > > Basically, I feel that from a "get normal users to test development
> > > > kernels", our Kconfig pain ends up being the biggest hurdle by far.
> > > >
> > > > The kernel is easy to build and doesn't really require all that much
> > > > infrastructure, but generating the config - particularly when it
> > > > changes over time and you can't just say "just use the distro config"
> > > > - is a big step for people.
> > > >
> > > > So honestly, while I've pulled this, I feel that this kind of change
> > > > is going _exactly_ the wrong way when it asks people questions that
> > > > they don't care one whit about.
> > > >
> > > > If I as a kernel developer can't find it in myself to care and go "why
> > > > does it ask this new question", then that should tell you something.
> > > >
> > > > Why do we have this choice in the first place?
> > >
> > > Generally, initramfs is passed from a boot-loader,
> > > but some architectures embed initramfs into vmlinux
> > > (perhaps due to poor boot-loader support??)
> > >
> > > arch/arc/configs/tb10x_defconfig:CONFIG_INITRAMFS_SOURCE="../tb10x-rootfs.cpio"
> > > arch/unicore32/configs/defconfig:#CONFIG_INITRAMFS_SOURCE="arch/unicore/ramfs/ramfs_config"
> > > arch/xtensa/configs/cadence_csp_defconfig:CONFIG_INITRAMFS_SOURCE="$$KERNEL_INITRAMFS_SOURCE"
> >
> > Note that the above are examples that do not actually work, as the files
> > referred to are not present in mainline (read below[1] why I have just
> > checked that ;-).
> >
> > > So, data-compression is useful - that's is what I understand.
> >
> > Yes it is, depending on your config.
> >
> > > For major architectures, vmlinux embeds a tiny initramfs,
> > > which is generated based on usr/default_cpio_list.
> > >
> > > We do not need data-compression for such a small cpio,
> > > but handling it in a consistent way is sensible.
> > > This is annoying from the users' PoV, I admit.

> > So INITRAMFS_COMPRESSION_NONE is the right answer to retain the
> > old behavior?
>
> Yes, INITRAMFS_COMPRESSION_NONE retains the previous behavior.
>
> But, as far as he I understood what Linus said,
> "we do not care".
>
> > One might question why not to use gzip anyway, as
> > CONFIG_RD_GZIP=y is enabled by default, and would give a (small)
> > improvement of ca. 350 bytes ;-)
> > Hence there is some area for improvement...
>
> GZIP is not the only compression algorithm.
>
> Somebody may want to disable RD_GZIP,
> then enable RD_XZ.

Sure, it can only be used when available.

> If we allow the data compression,
> Kconfig must ask "which compression algorithm".

Or default to one of the available ones.

> So, if Kconfig would siltently choose something
> as default, INITRAMFS_COMPRESSION_NONE would be the best.

Yeah, probably it's good enough.


Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [GIT PULL 1/2] Kbuild updates for v5.6-rc1

[Masahiro Yamada]

Hi Linus,


On Mon, Feb 3, 2020 at 8:07 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
>
>
> On Mon, Feb 3, 2020, 01:16 Masahiro Yamada <masahiroy@kernel.org> wrote:
>>
>>
>> Generally, initramfs is passed from a boot-loader,
>> but some architectures embed initramfs into vmlinux
>> (perhaps due to poor boot-loader support??)
>
>
> You didn't answer my real question.
>
> Why do we give the user the choice, when it doesn't matter, and the user doesn't care?

I do not want that commit simply reverted.

Please let me clarify what you want to see:

[1] Remove this choice completely ?
    The build system will choose the best one.
    For example, CONFIG_RD_XZ  is enabled,
    '.xz' is _always_ preferred choice over '.gz'


[2] Hide this choice unless INITRAMFS_SOURCE!=""

    As Geert mentioned, we still could save
    a little more data size, but we assume
    people would not care about hundreds bytes.


Which one ?


[2] was the previous behavior.
I think you are complaining
because you noticed a new prompt showed up.


> The argument for the commit was "it's simpler".
>
> But that is simply not *true*.
>
> It's simpler only technically. It's more complexity for the only party that matters: the user.
>
> So I'm likely going to just revert that commit as incorrect and misleading. It's not simpler at all. It's more complex.
>
> The configuration code should care about the user interface more than it seems to do. Some complexity in order to make for less pointless questions is food m good.
>
>        Linus



--
Best Regards
Masahiro Yamada