From: Masahiro Yamada <yamada.masahiro@socionext.com>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>,
"Joel Fernandes (Google)" <joel@joelfernandes.org>
Cc: Networking <netdev@vger.kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Alexei Starovoitov <ast@kernel.org>,
atish patra <atishp04@gmail.com>,
Daniel Colascione <dancol@google.com>,
Dan Williams <dan.j.williams@intel.com>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Jonathan Corbet <corbet@lwn.net>,
Karim Yaghmour <karim.yaghmour@opersys.com>,
Kees Cook <keescook@chromium.org>,
kernel-team@android.com,
"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
"open list:KERNEL SELFTEST FRAMEWORK"
<linux-kselftest@vger.kernel.org>,
Manoj Rao <linux@manojrajarao.com>,
Paul McKenney <paulmck@linux.vnet.ibm.com>,
"Peter Zijlstra (Intel)" <peterz@infradead.org>,
Randy Dunlap <rdunlap@infradead.org>,
Steven Rostedt <rostedt@goodmis.org>,
Shuah Khan <shuah@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>, Yonghong Song <yhs@fb.com>
Subject: Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
Date: Tue, 19 Feb 2019 13:42:13 +0900 [thread overview]
Message-ID: <CAK7LNATf4adEEqWko2YO46wHHi-Vvc+_LBjQGNnM7rt8GSOgZQ@mail.gmail.com> (raw)
In-Reply-To: <CAK7LNAThSm1n1B_Ss0gRDZ3H-DWj2bvpZ+6Ui2jEo4Pt9YynEA@mail.gmail.com>
On Tue, Feb 19, 2019 at 1:14 PM Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>
> On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > Introduce in-kernel headers and other artifacts which are made available
> > > as an archive through proc (/proc/kheaders.txz file).
>
>
> The extension '.txz' is not used in kernel code.
>
>
> '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc.
>
>
> $ git grep '\.txz'
> $ git grep '\.tar\.xz'
> Documentation/admin-guide/README.rst: xz -cd linux-4.X.tar.xz | tar xvf -
> arch/x86/crypto/camellia-aesni-avx-asm_64.S: *
> http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz
> crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> scripts/package/Makefile: @echo ' perf-tarxz-src-pkg - Build
> $(perf-tar).tar.xz source tarball'
> tools/testing/selftests/gen_kselftest_tar.sh:
> ext=".tar.xz"
>
>
>
> I prefer '.tar.xz' for consistency.
>
>
>
>
>
>
> BTW, have you ever looked at scripts/extract-ikconfig?
>
> You added IKHD_ST and IKHD_ED
> just to mimic kernel/configs.c
>
> It is currently pointless without the extracting tool,
> but you might think it is useful to extract headers
> from vmlinux or the module without mounting procfs.
>
>
>
>
> > > This archive makes
> > > it possible to build kernel modules, run eBPF programs, and other
> > > tracing programs that need to extend the kernel for tracing purposes
> > > without any dependency on the file system having headers and build
> > > artifacts.
> > >
> > > On Android and embedded systems, it is common to switch kernels but not
> > > have kernel headers available on the file system. Raw kernel headers
> > > also cannot be copied into the filesystem like they can be on other
> > > distros, due to licensing and other issues. There's no linux-headers
> > > package on Android. Further once a different kernel is booted, any
> > > headers stored on the file system will no longer be useful. By storing
> > > the headers as a compressed archive within the kernel, we can avoid these
> > > issues that have been a hindrance for a long time.
> >
> > The set looks good to me and since the main use case is building bpf progs
> > I can route it via bpf-next tree if there are no objections.
> > Masahiro, could you please ack it?
>
>
> Honestly, I was not tracking this thread
> since I did not know I was responsible for this.
>
>
> I just started to take a closer look, then immediately got scared.
>
> This version is not mature enough for the merge.
>
>
>
> First of all, this patch cannot be compiled out-of-tree (O= option).
>
>
> I do not know why 0-day bot did not catch this apparent breakage.
>
>
> $ make -j8 O=hoge
> make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge'
> GEN Makefile
> Using .. as source for kernel
> DESCEND objtool
> CALL ../scripts/checksyscalls.sh
> CHK include/generated/compile.h
> make[2]: *** No rule to make target 'Module.symvers', needed by
> 'kernel/kheaders_data.txz'. Stop.
> make[2]: *** Waiting for unfinished jobs....
> /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target
> 'kernel' failed
> make[1]: *** [kernel] Error 2
> make[1]: *** Waiting for unfinished jobs....
> make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge'
> Makefile:152: recipe for target 'sub-make' failed
> make: *** [sub-make] Error 2
I saw this build error for in-tree building as well.
We cannot build this from a pristine source tree.
For example, I observed the build error in the following procedure.
$ make mrproper
$ make defconfig
Set CONFIG_IKHEADERS_PROC=y
$ make
Module.symvers is generated in the modpost stage
(the very last stage of build).
But, vmlinux depends on kernel/kheaders_data.txz,
which includes Module.symvers.
So, this is not so simple since it is a circular dependency...
--
Best Regards
Masahiro Yamada
next prev parent reply other threads:[~2019-02-19 4:43 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-02-11 14:35 [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel Joel Fernandes (Google)
2019-02-11 14:36 ` [PATCH v2 2/2] Add selftests for module build using in-kernel headers Joel Fernandes (Google)
2019-02-13 22:50 ` [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel Karim Yaghmour
2019-02-15 3:19 ` Alexei Starovoitov
2019-02-15 3:47 ` Joel Fernandes
2019-02-16 19:10 ` Manoj
2019-02-19 4:14 ` Masahiro Yamada
2019-02-19 4:28 ` Alexei Starovoitov
2019-02-19 4:34 ` Joel Fernandes
2019-02-19 4:42 ` Masahiro Yamada [this message]
2019-02-19 5:12 ` Joel Fernandes
2019-02-19 15:16 ` Joel Fernandes
2019-02-21 14:34 ` Masahiro Yamada
2019-02-21 15:29 ` Joel Fernandes
2019-03-25 13:49 ` Joel Fernandes
2019-03-27 17:31 ` Joel Fernandes
2019-04-03 7:48 ` Masahiro Yamada
2019-04-03 17:20 ` Joel Fernandes
2019-04-03 17:46 ` Daniel Colascione
2019-04-03 17:56 ` Joel Fernandes
2019-04-04 3:54 ` Masahiro Yamada
2019-02-19 6:05 Alexey Dobriyan
2019-02-19 17:25 ` Joel Fernandes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAK7LNATf4adEEqWko2YO46wHHi-Vvc+_LBjQGNnM7rt8GSOgZQ@mail.gmail.com \
--to=yamada.masahiro@socionext.com \
--cc=akpm@linux-foundation.org \
--cc=alexei.starovoitov@gmail.com \
--cc=ast@kernel.org \
--cc=atishp04@gmail.com \
--cc=corbet@lwn.net \
--cc=dan.j.williams@intel.com \
--cc=dancol@google.com \
--cc=gregkh@linuxfoundation.org \
--cc=joel@joelfernandes.org \
--cc=karim.yaghmour@opersys.com \
--cc=keescook@chromium.org \
--cc=kernel-team@android.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux@manojrajarao.com \
--cc=netdev@vger.kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=rdunlap@infradead.org \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
--cc=tglx@linutronix.de \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).