[RFC] Provide in-kernel headers for making it easy to extend the kernel

[RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

From: "Joel Fernandes (Google)" <joel@joelfernandes.org>

Introduce in-kernel headers and other artifacts which are made available
as an archive through proc (/proc/kheaders.tgz file). This archive makes
it possible to build kernel modules, run eBPF programs, and other
tracing programs that need to extend the kernel for tracing purposes
without any dependency on the file system having headers and build
artifacts.

On Android and embedded systems, it is common to switch kernels but not
have kernel headers available on the file system. Raw kernel headers
also cannot be copied into the filesystem like they can be on other
distros, due to licensing and other issues. There's no linux-headers
package on Android. Further once a different kernel is booted, any
headers stored on the file system will no longer be useful. By storing
the headers as a compressed archive within the kernel, we can avoid these
issues that have been a hindrance for a long time.

The feature is buildable as a module thus making it possible to load and
unload it on demand. A tracing program, or a kernel module builder can
then load the module, do its thing, and then unload the module to save
the kernel memory. At the moment the total memory needed is 7MB.
However, this can be brought down by stripping comments from the header
files, which brings it down to 4MB. Since many of the headers have
copyright information, I have avoided stripping the comments in this
first pass, and can provide future patches to bring the size down if
needed. A further reduction is possible using bzip compression instead
of gzip, however I have stuck to gzip to avoid any dependency on less
popular compression tools.

The code to read the headers is based on /proc/config.gz code and uses
the same technique to embed the headers.

To build a module, the below steps have been tested on an x86 machine:
modprobe kheaders    # If building as IKHEADERS as a module
rm -rf $HOME/headers
mkdir -p $HOME/headers
tar -xvf /proc/kheaders.tgz -C $HOME/headers >/dev/null
cd my-kernel-module
make -C $HOME/headers M=$(PWD) modules
rmmod kheaders

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 Documentation/dontdiff  |  1 +
 arch/x86/Makefile       |  2 ++
 init/Kconfig            | 10 ++++++
 kernel/.gitignore       |  2 ++
 kernel/Makefile         | 29 ++++++++++++++++
 kernel/kheaders.c       | 74 +++++++++++++++++++++++++++++++++++++++++
 scripts/gen_ikh_data.sh | 13 ++++++++
 7 files changed, 131 insertions(+)
 create mode 100644 kernel/kheaders.c
 create mode 100755 scripts/gen_ikh_data.sh

diff --git a/Documentation/dontdiff b/Documentation/dontdiff
index 2228fcc8e29f..05a2319ee2a2 100644
--- a/Documentation/dontdiff
+++ b/Documentation/dontdiff
@@ -151,6 +151,7 @@ int8.c
 kallsyms
 kconfig
 keywords.c
+kheaders_data.h*
 ksym.c*
 ksym.h*
 kxgettext
diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 88398fdf8129..ad176d669da4 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -240,6 +240,8 @@ archmacros:
 ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s
 export ASM_MACRO_FLAGS
 KBUILD_CFLAGS += $(ASM_MACRO_FLAGS)
+IKH_EXTRA += arch/x86/kernel/macros.s
+export IKH_EXTRA
 
 ###
 # Kernel objects
diff --git a/init/Kconfig b/init/Kconfig
index a4112e95724a..0594df0b33ac 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -549,6 +549,16 @@ config IKCONFIG_PROC
 	  This option enables access to the kernel configuration file
 	  through /proc/config.gz.
 
+config IKHEADERS_PROC
+	tristate "Enable kernel header artifacts through /proc/kheaders.tgz"
+	select BUILD_BIN2C
+	depends on PROC_FS
+	help
+	  This option enables access to the kernel header and other artifacts that
+          are generated during the build process. These can be used to build kernel
+          modules, and other in-kernel programs such as those generated by eBPF
+          and systemtap tools.
+
 config LOG_BUF_SHIFT
 	int "Kernel log buffer size (16 => 64KB, 17 => 128KB)"
 	range 12 25
diff --git a/kernel/.gitignore b/kernel/.gitignore
index b3097bde4e9c..cacf6b7f6d32 100644
--- a/kernel/.gitignore
+++ b/kernel/.gitignore
@@ -3,5 +3,7 @@
 #
 config_data.h
 config_data.gz
+kheaders_data.h
+kheaders_data.tgz
 timeconst.h
 hz.bc
diff --git a/kernel/Makefile b/kernel/Makefile
index 7343b3a9bff0..d5968305d716 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -73,6 +73,7 @@ obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_USER_NS) += user_namespace.o
 obj-$(CONFIG_PID_NS) += pid_namespace.o
 obj-$(CONFIG_IKCONFIG) += configs.o
+obj-$(CONFIG_IKHEADERS_PROC) += kheaders.o
 obj-$(CONFIG_SMP) += stop_machine.o
 obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o
 obj-$(CONFIG_AUDIT) += audit.o auditfilter.o
@@ -131,3 +132,31 @@ $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
 targets += config_data.h
 $(obj)/config_data.h: $(obj)/config_data.gz FORCE
 	$(call filechk,ikconfiggz)
+
+# Build a list of in-kernel headers for building kernel modules
+# Any other files will be stored in IKH_EXTRA variable.
+ikh_file_list := include/
+ikh_file_list += arch/$(ARCH)/Makefile
+ikh_file_list += arch/$(ARCH)/include/
+ikh_file_list += $(IKH_EXTRA)
+ikh_file_list += scripts/
+ikh_file_list += Makefile
+ikh_file_list += Module.symvers
+ifeq ($(CONFIG_STACK_VALIDATION), y)
+ikh_file_list += $(objtree)/tools/objtool/objtool
+endif
+
+$(obj)/kheaders.o: $(obj)/kheaders_data.h
+
+targets += kheaders_data.tgz
+
+quiet_cmd_genikh = GEN     $(obj)/kheaders_data.tgz
+cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^ >/dev/null 2>&1
+$(obj)/kheaders_data.tgz: $(ikh_file_list) FORCE
+	$(call cmd,genikh)
+
+filechk_ikheadersgz = (echo "static const char kernel_headers_data[] __used = KH_MAGIC_START"; cat $< | scripts/bin2c; echo "KH_MAGIC_END;")
+
+targets += kheaders_data.h
+$(obj)/kheaders_data.h: $(obj)/kheaders_data.tgz FORCE
+	$(call filechk,ikheadersgz)
diff --git a/kernel/kheaders.c b/kernel/kheaders.c
new file mode 100644
index 000000000000..ac4ca0248b8b
--- /dev/null
+++ b/kernel/kheaders.c
@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * kernel/kheaders.c
+ * Provide headers and artifacts needed to build kernel modules.
+ * (Borrowed code from kernel/configs.c)
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+#include <linux/init.h>
+#include <linux/uaccess.h>
+
+/*
+ * Define kernel_headers_data and kernel_headers_data_size, which contains the
+ * compressed kernel headers.  The file is first compressed with gzip and then
+ * bounded by two eight byte magic numbers to allow extraction from a binary
+ * kernel image:
+ *
+ *   IKHD_ST
+ *   <image>
+ *   IKHD_ED
+ */
+#define KH_MAGIC_START	"IKHD_ST"
+#define KH_MAGIC_END	"IKHD_ED"
+#include "kheaders_data.h"
+
+
+#define KH_MAGIC_SIZE (sizeof(KH_MAGIC_START) - 1)
+#define kernel_headers_data_size \
+	(sizeof(kernel_headers_data) - 1 - KH_MAGIC_SIZE * 2)
+
+static ssize_t
+ikheaders_read_current(struct file *file, char __user *buf,
+		      size_t len, loff_t *offset)
+{
+	return simple_read_from_buffer(buf, len, offset,
+				       kernel_headers_data + KH_MAGIC_SIZE,
+				       kernel_headers_data_size);
+}
+
+static const struct file_operations ikheaders_file_ops = {
+	.owner = THIS_MODULE,
+	.read = ikheaders_read_current,
+	.llseek = default_llseek,
+};
+
+static int __init ikheaders_init(void)
+{
+	struct proc_dir_entry *entry;
+
+	/* create the current headers file */
+	entry = proc_create("kheaders.tgz", S_IFREG | S_IRUGO, NULL,
+			    &ikheaders_file_ops);
+	if (!entry)
+		return -ENOMEM;
+
+	proc_set_size(entry, kernel_headers_data_size);
+
+	return 0;
+}
+
+static void __exit ikheaders_cleanup(void)
+{
+	remove_proc_entry("kheaders.tgz", NULL);
+}
+
+module_init(ikheaders_init);
+module_exit(ikheaders_cleanup);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Joel Fernandes");
+MODULE_DESCRIPTION("Echo the kernel header artifacts used to build the kernel");
diff --git a/scripts/gen_ikh_data.sh b/scripts/gen_ikh_data.sh
new file mode 100755
index 000000000000..e87a0f0c13e5
--- /dev/null
+++ b/scripts/gen_ikh_data.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+echo "${@:2}" > $1.list
+
+rm -rf $1.tmp
+mkdir $1.tmp
+
+for f in "${@:2}";
+	do find "$f" ! -name "*.c" ! -name "*.o" ! -name "*.cmd" ! -name ".*";
+done | cpio -pd $1.tmp
+
+tar -jcf $1 -C $1.tmp/ . > /dev/null
+rm -rf $1.tmp
-- 
2.20.1.321.g9e740568ce-goog

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Greg KH]

On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> --- /dev/null
> +++ b/kernel/kheaders.c
> @@ -0,0 +1,74 @@
> +// SPDX-License-Identifier: GPL-2.0

Nice, but:

<snip>

> +MODULE_LICENSE("GPL");

That means "GPL2+" (yeah, horrible, I know, we all get it wrong, look at
include/linux/module.h for details)

So you should change that line to:
	MODULE_LICENSE("GPLv2");

Other than that, nice work.

greg k-h

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Sat, Jan 19, 2019 at 09:25:32AM +0100, Greg KH wrote:
> On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> > --- /dev/null
> > +++ b/kernel/kheaders.c
> > @@ -0,0 +1,74 @@
> > +// SPDX-License-Identifier: GPL-2.0
> 
> Nice, but:
> 
> <snip>
> 
> > +MODULE_LICENSE("GPL");
> 
> That means "GPL2+" (yeah, horrible, I know, we all get it wrong, look at
> include/linux/module.h for details)
> 
> So you should change that line to:
> 	MODULE_LICENSE("GPLv2");

Ok, will fix.

> Other than that, nice work.

thanks!

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Sat, Jan 19, 2019 at 11:27 AM Joel Fernandes <joel@joelfernandes.org> wrote:
>
> On Sat, Jan 19, 2019 at 09:25:32AM +0100, Greg KH wrote:
> > On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> > > --- /dev/null
> > > +++ b/kernel/kheaders.c

Thanks a ton for this work. It'll make it much easier to do cool
things with BPF. One question: I can imagine wanting to probe
structures that are defined, not in headers, but in random
implementation files. Would it be possible to optionally include *all*
kernel source files? If not, what about a hash, so we could at least
do precise correlation between a candidate local tree and what's
actually on device?

BTW, I'm not sure that the magic constants you've defined are long
enough.  I'd feel more comfortable with two UUIDs (16 bytes each).

I'd also strongly consider LZMA compression: xz -9 on the kernel
headers (with comments) brings the size down to 5MB, compared to the
7MB I get for gzip -9. Considering that this feature is optional, I
think it's okay to introduce a dependency on widespread modern
compression tools. (For comparison, bzip2 -9 gets us 6MB.)

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Sat, Jan 19, 2019 at 12:43:35PM -0500, Daniel Colascione wrote:
> On Sat, Jan 19, 2019 at 11:27 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> >
> > On Sat, Jan 19, 2019 at 09:25:32AM +0100, Greg KH wrote:
> > > On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> > > > --- /dev/null
> > > > +++ b/kernel/kheaders.c
> 
> Thanks a ton for this work. It'll make it much easier to do cool
> things with BPF.

You're welcome, thanks.

> One question: I can imagine wanting to probe
> structures that are defined, not in headers, but in random
> implementation files. Would it be possible to optionally include *all*
> kernel source files?

That would be prohibitively too large to justify keeping it in memory, even
compressed. Arguably, such structures should be moved into include/ if
modules or whatever is extending the kernel like eBPF needs them.

> If not, what about a hash, so we could at least
> do precise correlation between a candidate local tree and what's
> actually on device?

That would make a tool too difficult to write wouldn't it, since they you have to
correlate every possible hash and keep updating eBPF tools with new hashes -
probably not scalable. I think what you want is to use the kernel version to
assume what such internal structures look like although that's still not
robust.

> BTW, I'm not sure that the magic constants you've defined are long
> enough.  I'd feel more comfortable with two UUIDs (16 bytes each).

Ok, I'll expand it.

> I'd also strongly consider LZMA compression: xz -9 on the kernel
> headers (with comments) brings the size down to 5MB, compared to the
> 7MB I get for gzip -9. Considering that this feature is optional, I
> think it's okay to introduce a dependency on widespread modern
> compression tools. (For comparison, bzip2 -9 gets us 6MB.)

Ok, I'll look into LZMA. Thanks for checking the compression sizes.

- Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[hpa]

On January 19, 2019 3:25:03 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
>On Sat, Jan 19, 2019 at 12:43:35PM -0500, Daniel Colascione wrote:
>> On Sat, Jan 19, 2019 at 11:27 AM Joel Fernandes
><joel@joelfernandes.org> wrote:
>> >
>> > On Sat, Jan 19, 2019 at 09:25:32AM +0100, Greg KH wrote:
>> > > On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
>> > > > --- /dev/null
>> > > > +++ b/kernel/kheaders.c
>> 
>> Thanks a ton for this work. It'll make it much easier to do cool
>> things with BPF.
>
>You're welcome, thanks.
>
>> One question: I can imagine wanting to probe
>> structures that are defined, not in headers, but in random
>> implementation files. Would it be possible to optionally include
>*all*
>> kernel source files?
>
>That would be prohibitively too large to justify keeping it in memory,
>even
>compressed. Arguably, such structures should be moved into include/ if
>modules or whatever is extending the kernel like eBPF needs them.
>
>> If not, what about a hash, so we could at least
>> do precise correlation between a candidate local tree and what's
>> actually on device?
>
>That would make a tool too difficult to write wouldn't it, since they
>you have to
>correlate every possible hash and keep updating eBPF tools with new
>hashes -
>probably not scalable. I think what you want is to use the kernel
>version to
>assume what such internal structures look like although that's still
>not
>robust.
>
>> BTW, I'm not sure that the magic constants you've defined are long
>> enough.  I'd feel more comfortable with two UUIDs (16 bytes each).
>
>Ok, I'll expand it.
>
>> I'd also strongly consider LZMA compression: xz -9 on the kernel
>> headers (with comments) brings the size down to 5MB, compared to the
>> 7MB I get for gzip -9. Considering that this feature is optional, I
>> think it's okay to introduce a dependency on widespread modern
>> compression tools. (For comparison, bzip2 -9 gets us 6MB.)
>
>Ok, I'll look into LZMA. Thanks for checking the compression sizes.
>
>- Joel

Don't use lzma, use xz if you are going to do something.

However, it seems unlikely to me that someone not willing to spend the space in the filesystem will spend unswappable kernel memory.

It would seem that a far saner way to do this is to use inittmpfs or perhaps an auxiliary "ktmpfs" so it can at least be swapped out if you have swap.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Sat, Jan 19, 2019 at 03:44:48PM -0800, hpa@zytor.com wrote:
> On January 19, 2019 3:25:03 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
> >On Sat, Jan 19, 2019 at 12:43:35PM -0500, Daniel Colascione wrote:
> >> On Sat, Jan 19, 2019 at 11:27 AM Joel Fernandes
> ><joel@joelfernandes.org> wrote:
> >> >
> >> > On Sat, Jan 19, 2019 at 09:25:32AM +0100, Greg KH wrote:
> >> > > On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> >> > > > --- /dev/null
> >> > > > +++ b/kernel/kheaders.c
> >> 
> >> Thanks a ton for this work. It'll make it much easier to do cool
> >> things with BPF.
> >
> >You're welcome, thanks.
> >
> >> One question: I can imagine wanting to probe
> >> structures that are defined, not in headers, but in random
> >> implementation files. Would it be possible to optionally include
> >*all*
> >> kernel source files?
> >
> >That would be prohibitively too large to justify keeping it in memory,
> >even
> >compressed. Arguably, such structures should be moved into include/ if
> >modules or whatever is extending the kernel like eBPF needs them.
> >
> >> If not, what about a hash, so we could at least
> >> do precise correlation between a candidate local tree and what's
> >> actually on device?
> >
> >That would make a tool too difficult to write wouldn't it, since they
> >you have to
> >correlate every possible hash and keep updating eBPF tools with new
> >hashes -
> >probably not scalable. I think what you want is to use the kernel
> >version to
> >assume what such internal structures look like although that's still
> >not
> >robust.
> >
> >> BTW, I'm not sure that the magic constants you've defined are long
> >> enough.  I'd feel more comfortable with two UUIDs (16 bytes each).
> >
> >Ok, I'll expand it.
> >
> >> I'd also strongly consider LZMA compression: xz -9 on the kernel
> >> headers (with comments) brings the size down to 5MB, compared to the
> >> 7MB I get for gzip -9. Considering that this feature is optional, I
> >> think it's okay to introduce a dependency on widespread modern
> >> compression tools. (For comparison, bzip2 -9 gets us 6MB.)
> >
> >Ok, I'll look into LZMA. Thanks for checking the compression sizes.
> >
> >- Joel
> 
> Don't use lzma, use xz if you are going to do something.

Ok, sounds good.

> However, it seems unlikely to me that someone not willing to spend the space in the filesystem will spend unswappable kernel memory.
> 
> It would seem that a far saner way to do this is to use inittmpfs or perhaps an auxiliary "ktmpfs" so it can at least be swapped out if you have swap.

But this is already possible with the proposed solution, you would load the
module, extract it into a tmpfs, and unload the module. TMPFS pages can
already be swapped.

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]


> > >Ok, I'll look into LZMA. Thanks for checking the compression sizes.
> > >
> > >- Joel
> > 
> > Don't use lzma, use xz if you are going to do something.
> 
> Ok, sounds good.
> 
> > However, it seems unlikely to me that someone not willing to spend the space in the filesystem will spend unswappable kernel memory.
> > 
> > It would seem that a far saner way to do this is to use inittmpfs or perhaps an auxiliary "ktmpfs" so it can at least be swapped out if you have swap.
> 
> But this is already possible with the proposed solution, you would load the
> module, extract it into a tmpfs, and unload the module. TMPFS pages can
> already be swapped.

So your licensing requirements prevent you from having headers in the
filesystem, but allow module with the headers hidden inside on the
filesystem?

Looks like you should just tar xvzf
this-is-a-kernel-module-I-promise.ko /usr/src/linux/include :-).
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Wed, Mar 6, 2019 at 3:09 PM Pavel Machek <pavel@ucw.cz> wrote:
>
>
> > > >Ok, I'll look into LZMA. Thanks for checking the compression sizes.
> > > >
> > > >- Joel
> > >
> > > Don't use lzma, use xz if you are going to do something.
> >
> > Ok, sounds good.

XZ is a file format for LZMA2. Everyone's right. :-)

> >
> > > However, it seems unlikely to me that someone not willing to spend the space in the filesystem will spend unswappable kernel memory.
> > >
> > > It would seem that a far saner way to do this is to use inittmpfs or perhaps an auxiliary "ktmpfs" so it can at least be swapped out if you have swap.
> >
> > But this is already possible with the proposed solution, you would load the
> > module, extract it into a tmpfs, and unload the module. TMPFS pages can
> > already be swapped.
>
> So your licensing requirements prevent you from having headers in the
> filesystem, but allow module with the headers hidden inside on the
> filesystem?
>
> Looks like you should just tar xvzf
> this-is-a-kernel-module-I-promise.ko /usr/src/linux/include :-).

I just don't get the opposition to Joel's work. The rest of the thread
already goes into detail about the problems with pure-filesystem
solutions, and you and others are just totally ignoring those
well-thought-out rationales for the module approach and doing
inflooping on "lol just use a tarball". That's not productive.

Look; here's the bottom line: without this work, doing certain kinds
of system tracing is a nightmare, and with this patch, it Just Works.
You're arguing that various tools should do a better job of keeping
the filesystem in sync with the kernel. Maybe you're right. But we
don't live in a world where they will, because if this coherence were
going to happen, it'd work already. But this work solves the problem:
by necessity, anything that changes a kernel image *must* update
modules coherently, whether the kernel image and module come from the
filesystem, network boot, some kind of SQL database, or carrier
pigeon. There's nothing wrong with work that very cheaply makes the
kernel self-describing (introspection is elegant) and that takes
advantage of *existing* kernel tooling infrastructure to transparently
do a new thing.

You don't have to use this patch if you don't want to. Please stop
trying to block it.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[H. Peter Anvin]

On 3/6/19 3:37 PM, Daniel Colascione wrote:
> 
> I just don't get the opposition to Joel's work. The rest of the thread
> already goes into detail about the problems with pure-filesystem
> solutions, and you and others are just totally ignoring those
> well-thought-out rationales for the module approach and doing
> inflooping on "lol just use a tarball". That's not productive.
> 
> Look; here's the bottom line: without this work, doing certain kinds
> of system tracing is a nightmare, and with this patch, it Just Works.
> You're arguing that various tools should do a better job of keeping
> the filesystem in sync with the kernel. Maybe you're right. But we
> don't live in a world where they will, because if this coherence were
> going to happen, it'd work already. But this work solves the problem:
> by necessity, anything that changes a kernel image *must* update
> modules coherently, whether the kernel image and module come from the
> filesystem, network boot, some kind of SQL database, or carrier
> pigeon. There's nothing wrong with work that very cheaply makes the
> kernel self-describing (introspection is elegant) and that takes
> advantage of *existing* kernel tooling infrastructure to transparently
> do a new thing.
> 
> You don't have to use this patch if you don't want to. Please stop
> trying to block it.
> 

No, that's not how this works. It is far worse to do something the wrong
way than not doing it at all, when it affects the kernel-user space
interactions.

Experience -- and we have almost 30 years of it -- has shown that hacks
of this nature become engrained and all of a sudden is "mandatory". At
the *very least* it needs to comply with the zero-one-infinity rule
rather than being yet another ad hoc hack.

More fundamentally, we already have multiple ways to handle objects that
need to go into the filesystem: they can be installed with (or as)
modules, they can use the firmware interface, and so on.

Saying "it can be a module" is worse than a copout: even if dynamically
loaded -- and many setups lock out post-boot module loadings for
security reasons -- there is nothing to cause it to unload.

The bottom line is that in the end there is no difference between making
this an archive of some sort and a module, except that to use the module
you need privilege that you otherwise may not need. If your argument is
that someone may not be providing the whole set of items provided by
"make modules_install", what is there to say that they would include
your specific module?

Here are some better ways of implementation I can see:

1. Include an archive in "make modules_install". Most trivial; no kernel
   changes at all.
2. Generalize the initramfs code to be able to create a pre-populated
   tmpfs at any time, that can be populated from an archive provided by
   the firmware loading mechanism; like all firmware allows it to either
   be built in or fetched from the filesystem. This allows it to be
   built in to the kernel image if that becomes necessary; using tmpfs
   means that it can be pushed out to swap rather than permanently
   stored in kernel memory, and this filesystem can be unmounted freeing
   its memory.
3. Use a squashfs image instead of an archive.

	-hpa

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Wed, Mar 6, 2019 at 4:07 PM H. Peter Anvin <hpa@zytor.com> wrote:
>
> On 3/6/19 3:37 PM, Daniel Colascione wrote:
> >
> > I just don't get the opposition to Joel's work. The rest of the thread
> > already goes into detail about the problems with pure-filesystem
> > solutions, and you and others are just totally ignoring those
> > well-thought-out rationales for the module approach and doing
> > inflooping on "lol just use a tarball". That's not productive.
> >
> > Look; here's the bottom line: without this work, doing certain kinds
> > of system tracing is a nightmare, and with this patch, it Just Works.
> > You're arguing that various tools should do a better job of keeping
> > the filesystem in sync with the kernel. Maybe you're right. But we
> > don't live in a world where they will, because if this coherence were
> > going to happen, it'd work already. But this work solves the problem:
> > by necessity, anything that changes a kernel image *must* update
> > modules coherently, whether the kernel image and module come from the
> > filesystem, network boot, some kind of SQL database, or carrier
> > pigeon. There's nothing wrong with work that very cheaply makes the
> > kernel self-describing (introspection is elegant) and that takes
> > advantage of *existing* kernel tooling infrastructure to transparently
> > do a new thing.
> >
> > You don't have to use this patch if you don't want to. Please stop
> > trying to block it.
> >
>
> No, that's not how this works. It is far worse to do something the wrong
> way than not doing it at all, when it affects the kernel-user space
> interactions.

And what are the supposedly disastrous consequences of this change?
It's basically a souped-up version /proc/config.gz. Tell me more about
the trail of destruction and regret behind /proc/config.gz.

> Experience -- and we have almost 30 years of it -- has shown that hacks
> of this nature become engrained and all of a sudden is "mandatory". At
> the *very least* it needs to comply with the zero-one-infinity rule
> rather than being yet another ad hoc hack.

It already satisfies the zero-one-infinity rule by virtue of not being
a system for encoding an arbitrary number of random kernel header
blobs for some reason in a single kernel.

> More fundamentally, we already have multiple ways to handle objects that
> need to go into the filesystem: they can be installed with (or as)
> modules, they can use the firmware interface, and so on.

*There* *may* *be* *no* *filesystem*. Or the filesystem may be
read-only. The only thing the kernel can really guarantee is its own
existence --- it should be entire in itself. If I'm hacking on an
Android kernel and say "fastboot boot mykernel" without making any
changes to the device's boot filesystem, I should still be able to use
tracing tools that rely on knowing the headers for the kernel with
which the device happened to boot. Any approach that requires
coordinated kernel and filesystem changes to make this usecase work is
inferior to what Joel's proposed.

> Saying "it can be a module" is worse than a copout: even if dynamically
> loaded -- and many setups lock out post-boot module loadings for
> security reasons -- there is nothing to cause it to unload.

Those setups can ship kernel headers as they do today. Or a
tmpfs-based approach may be workable.

> The bottom line is that in the end there is no difference between making
> this an archive of some sort and a module, except that to use the module
> you need privilege that you otherwise may not need. If your argument is
> that someone may not be providing the whole set of items provided by
> "make modules_install", what is there to say that they would include
> your specific module?
>
> Here are some better ways of implementation I can see:
>
> 1. Include an archive in "make modules_install". Most trivial; no kernel
>    changes at all.

No. See above.

> 2. Generalize the initramfs code to be able to create a pre-populated
>    tmpfs at any time, that can be populated from an archive provided by
>    the firmware loading mechanism; like all firmware allows it to either
>    be built in or fetched from the filesystem. This allows it to be
>    built in to the kernel image if that becomes necessary; using tmpfs
>    means that it can be pushed out to swap rather than permanently
>    stored in kernel memory, and this filesystem can be unmounted freeing
>    its memory.

Backing the blob storage with tmpfs is a reasonable tweak to Joel's
existing model. We can mark the header blob discardable and memcpy it
into some tmpfs-backed storage. This way, it can swap, and you can
release the memory with rm(1) as well as unmount. You might as well
expose the facility as a new just-like-tmpfs filesystem that init
scripts can mount anywhere --- once. Making the thing a firmware blob
sounds fine too, although I know less about that subsystem. But
blocking this work as a whole in favor of some yet-to-be-designed
general-purpose initramfs-tmpfs conversion thingamajig really is
perfect-is-the-enemy-of-the-good-ism, and I don't think tmpfs storage
is necessary for the initial version of this work.

> 3. Use a squashfs image instead of an archive.

Why?

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Enrico Weigelt, metux IT consult]

On 07.03.19 01:33, Daniel Colascione wrote:

> *There* *may* *be* *no* *filesystem*. 

A Linux system w/o any filesystem at all ? Well, that's interesting.

> The only thing the kernel can really guarantee is its own> existence --- it should be entire in itself.

I vaguely recall some option for linking in an initrd ... does this
still exist ?

> If I'm hacking on an> Android kernel and say "fastboot boot mykernel" without making any>
changes to the device's boot filesystem, I should still be able to use>
tracing tools that rely on knowing the headers for the kernel with

Fix fastboot to support initrd or use a remote filesystem ?

I'm doing embedded development all the day, and one of the first things
I usually set up for a project is an fully automatic netboot (or at
least usb boot). Shouldn't be so hard, and is a more generic solution.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Wed, Mar 6, 2019 at 5:23 PM Enrico Weigelt, metux IT consult
<lkml@metux.net> wrote:
>
> On 07.03.19 01:33, Daniel Colascione wrote:
>
> > *There* *may* *be* *no* *filesystem*.
>
> A Linux system w/o any filesystem at all ? Well, that's interesting.

Entirely FS-less operation is uncommon, granted. :-) I guess I've just
spent too much time debugging emulators that refuse to mount their
root filesystems. :-) There are legitimate reasons why a device's
filesystems might not be rw-mountable though, and I can imagine a
world where I want to attach tracing tools *very* early.

> > The only thing the kernel can really guarantee is its own> existence --- it should be entire in itself.
>
> I vaguely recall some option for linking in an initrd ... does this
> still exist ?
>
> > If I'm hacking on an> Android kernel and say "fastboot boot mykernel" without making any>
> changes to the device's boot filesystem, I should still be able to use>
> tracing tools that rely on knowing the headers for the kernel with
>
> Fix fastboot to support initrd or use a remote filesystem ?

Sure. There's some support for a ramdisk already in fastboot. But just
including a blob in initrd doesn't *automatically* make it available
to userspace in any uniform way. With Joel's approach --- which
defines both a propagation mechanism and an access interface --- we
have a chance to make very useful tools work transparently everywhere,
and without additional work across a fragmented and uncoordinated
ecosystem. That's not nothing! While I appreciate the purity merits of
a file-based approach, insisting on it will lead to a world where
it'll be many more years before we can apply these powerful analysis
tools universally. Human factors are just as important as technical
ones if you want to actually get anything done, and in this case, a
consideration of the human factors points toward the kernel as
coordination point for kernel metadata. It'd be different if we were
working in a more-coordinated system NT or FreeBSD, but this is Linux,
where fragmentation starts as soon as you exit ring zero.

That said, I think it's fair to want the configuration blob to be
swappable and removable even in a monolothic, no-module system. There
are lots of ways of meeting these requirements.

> I'm doing embedded development all the day, and one of the first things
> I usually set up for a project is an fully automatic netboot (or at
> least usb boot). Shouldn't be so hard, and is a more generic solution.

Yeah. I think every embedded developer quickly developers a rich set
of shell scripts for this stuff. That's part of my point: if we can
pull it off, existing toolsets, which we can't possibly change all at
once, should properly propagate this kernel metadata without having to
even realize that they're doing it. Practically speaking, the only
universal mechanism is to bake something into the kernel image or one
of its modules.

[RFC] Provide in-kernel headers for making it easy to extend the kernel

[Enrico Weigelt, metux IT consult]

On 07.03.19 02:49, Daniel Colascione wrote:

> Entirely FS-less operation is uncommon, granted. :-) I guess I've just
> spent too much time debugging emulators that refuse to mount their> root filesystems. :-)

Fix these emulators ?

> There are legitimate reasons why a device's> filesystems might not be rw-mountable though, and I can imagine a
> world where I want to attach tracing tools *very* early.

Ok, but the filesystem where the modules live is mountable, right ?

> Sure. There's some support for a ramdisk already in fastboot. But just
> including a blob in initrd doesn't *automatically* make it available
> to userspace in any uniform way. With Joel's approach --- which> defines both a propagation mechanism and an access interface --- we

I can define such an interface with a few words:

* the kernel headers lives in a (compressed) squashfs image in the
  module directory for the corresponding kernel version:
      /lib/modules/<version>-<flavour>/headers.img"
* this image shall be mounted at a canonical mountpoint, eg:
      /usr/src/linux-headers-<version>-<flavour>
* the kernel needs to support squashfs (may be module or built-in)

That's it. I don't need to touch a single line of kernel code for that,
just a very simple and small convention.

> have a chance to make very useful tools work transparently everywhere,
> and without additional work across a fragmented and uncoordinated
> ecosystem. 

Why not instead starting to clean up your fragmented and uncoordinated
ecosystem ? :o

> That's not nothing!

It's nothing more that we already have built-in for aeons.

> While I appreciate the purity merits of a file-based approach, insisting 
> on it will lead to a world where it'll be many more years before we can> apply these powerful analysis tools universally.

So, adding a few LOC in Android build machinery for just creating an
archive/squashfs is such a complicated things that takes many years ?

> Human factors are just as important as technical ones,

Human factors like people just not willing to learn the fundamental
basics of the operating they're customizing ?

> ones if you want to actually get anything done, and in this case, a
> consideration of the human factors points toward the kernel as
> coordination point for kernel metadata. 

So, if userland folks are incapable of doing pretty simple things,
put a complex machinery into the kernel ?

> It'd be different if we were
> working in a more-coordinated system NT or FreeBSD, 

Ah, NT. Windows. The platform where everybody rolls his own monstreaus
drivers stacks (eg. several megabytes for trivialities like a CAN
adapter or about a gigabyte for a bunch of adc cards), usually all with
their own private userland api, because the OS vendor just doesn't
manage to provide generic subsystems for common stuff.

The one where applications are usually 10 times bigger and all come
with their own, private installer, just because there's no coordinated
package management and distros.

> but this is Linux,

No, this is *Android* we're talking about.

In GNU/Linux world, the problem you're trying to solve here already
*is* solved since aeons. Entirely in userland.

> where fragmentation starts as soon as you exit ring zero.

Such fragmentation like virtually any device vendor forking it's own
private kernel for each model, doing a huge bunch of pretty crude
hacks and dropping all maintenance as soon as the next model coming
up on the horizon ?

> Practically speaking, the only universal mechanism is to bake something
> into the kernel image or one of its modules.

Aha, and when do we start moving widget toolkits and html renderers
into the kernel ? :o


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Greg KH]

On Thu, Mar 07, 2019 at 09:41:24PM +0100, Enrico Weigelt, metux IT consult wrote:
> On 07.03.19 02:49, Daniel Colascione wrote:
> 
> > Entirely FS-less operation is uncommon, granted. :-) I guess I've just
> > spent too much time debugging emulators that refuse to mount their> root filesystems. :-)
> 
> Fix these emulators ?
> 
> > There are legitimate reasons why a device's> filesystems might not be rw-mountable though, and I can imagine a
> > world where I want to attach tracing tools *very* early.
> 
> Ok, but the filesystem where the modules live is mountable, right ?
> 
> > Sure. There's some support for a ramdisk already in fastboot. But just
> > including a blob in initrd doesn't *automatically* make it available
> > to userspace in any uniform way. With Joel's approach --- which> defines both a propagation mechanism and an access interface --- we
> 
> I can define such an interface with a few words:
> 
> * the kernel headers lives in a (compressed) squashfs image in the
>   module directory for the corresponding kernel version:
>       /lib/modules/<version>-<flavour>/headers.img"
> * this image shall be mounted at a canonical mountpoint, eg:
>       /usr/src/linux-headers-<version>-<flavour>
> * the kernel needs to support squashfs (may be module or built-in)
> 
> That's it. I don't need to touch a single line of kernel code for that,
> just a very simple and small convention.

Ick, no, no more squashfs please, let's just kill that mess once and for
all :)

Again, putting this in a simple compressed tar image allows anyone to do
whatever they need to with this.  If they want a full filesystem,
uncompress it and use it there.  If they just want it in-memory where
they can uncompress it and then discard it, that works too.

And if no one wants this, just don't select the config option and do not
load the module with this data in it.  Just like /proc/config.gz is
today.

thanks,

greg k-h

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Enrico Weigelt, metux IT consult]

On 07.03.19 21:55, Greg KH wrote:

> Ick, no, no more squashfs please, let's just kill that mess once and for
> all :)

okay, then: s/squashfs/whatever_fs_image_or_archive_you_like/;

> Again, putting this in a simple compressed tar image allows anyone to do
> whatever they need to with this.  If they want a full filesystem,
> uncompress it and use it there.  If they just want it in-memory where
> they can uncompress it and then discard it, that works too.

And let me stress the point: doesn't need any kernel changes at all,
when it's just a file in the same place where the .ko's live.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

Enrico,

On Thu, Mar 07, 2019 at 11:11:22PM +0100, Enrico Weigelt, metux IT consult wrote:
> On 07.03.19 21:55, Greg KH wrote:
> 
> > Ick, no, no more squashfs please, let's just kill that mess once and for
> > all :)
> 
> okay, then: s/squashfs/whatever_fs_image_or_archive_you_like/;
> 
> > Again, putting this in a simple compressed tar image allows anyone to do
> > whatever they need to with this.  If they want a full filesystem,
> > uncompress it and use it there.  If they just want it in-memory where
> > they can uncompress it and then discard it, that works too.
> 
> And let me stress the point: doesn't need any kernel changes at all,
> when it's just a file in the same place where the .ko's live.

Yes, but you're missing the point that some people would also opt to build it
into the kernel during their development/debugging (Config=y). For such
folks, they don't want to update the FS with anything during debug runs
either. Your "whole same place where the .ko lives" doesn't address Daniel's
usecase. You may say "initrd", but this is a much cleaner solution to that
IMO. There is no initrd needed and the path to the header files will be at a
standard location that is already pre-decided by the kernel.

As Greg said, you are welcome to keep it disabled for yourself if you don't
want it. This doesn't affect anyone else who doesn't use it.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[hpa]

On March 7, 2019 3:12:07 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
>Enrico,
>
>On Thu, Mar 07, 2019 at 11:11:22PM +0100, Enrico Weigelt, metux IT
>consult wrote:
>> On 07.03.19 21:55, Greg KH wrote:
>> 
>> > Ick, no, no more squashfs please, let's just kill that mess once
>and for
>> > all :)
>> 
>> okay, then: s/squashfs/whatever_fs_image_or_archive_you_like/;
>> 
>> > Again, putting this in a simple compressed tar image allows anyone
>to do
>> > whatever they need to with this.  If they want a full filesystem,
>> > uncompress it and use it there.  If they just want it in-memory
>where
>> > they can uncompress it and then discard it, that works too.
>> 
>> And let me stress the point: doesn't need any kernel changes at all,
>> when it's just a file in the same place where the .ko's live.
>
>Yes, but you're missing the point that some people would also opt to
>build it
>into the kernel during their development/debugging (Config=y). For such
>folks, they don't want to update the FS with anything during debug runs
>either. Your "whole same place where the .ko lives" doesn't address
>Daniel's
>usecase. You may say "initrd", but this is a much cleaner solution to
>that
>IMO. There is no initrd needed and the path to the header files will be
>at a
>standard location that is already pre-decided by the kernel.
>
>As Greg said, you are welcome to keep it disabled for yourself if you
>don't
>want it. This doesn't affect anyone else who doesn't use it.

You do know that initrd can be built into the kernel, right?
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Thu, Mar 07, 2019 at 03:40:37PM -0800, hpa@zytor.com wrote:
> On March 7, 2019 3:12:07 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
> >Enrico,
> >
> >On Thu, Mar 07, 2019 at 11:11:22PM +0100, Enrico Weigelt, metux IT
> >consult wrote:
> >> On 07.03.19 21:55, Greg KH wrote:
> >> 
> >> > Ick, no, no more squashfs please, let's just kill that mess once
> >and for
> >> > all :)
> >> 
> >> okay, then: s/squashfs/whatever_fs_image_or_archive_you_like/;
> >> 
> >> > Again, putting this in a simple compressed tar image allows anyone
> >to do
> >> > whatever they need to with this.  If they want a full filesystem,
> >> > uncompress it and use it there.  If they just want it in-memory
> >where
> >> > they can uncompress it and then discard it, that works too.
> >> 
> >> And let me stress the point: doesn't need any kernel changes at all,
> >> when it's just a file in the same place where the .ko's live.
> >
> >Yes, but you're missing the point that some people would also opt to
> >build it
> >into the kernel during their development/debugging (Config=y). For such
> >folks, they don't want to update the FS with anything during debug runs
> >either. Your "whole same place where the .ko lives" doesn't address
> >Daniel's
> >usecase. You may say "initrd", but this is a much cleaner solution to
> >that
> >IMO. There is no initrd needed and the path to the header files will be
> >at a
> >standard location that is already pre-decided by the kernel.
> >
> >As Greg said, you are welcome to keep it disabled for yourself if you
> >don't
> >want it. This doesn't affect anyone else who doesn't use it.
> 
> You do know that initrd can be built into the kernel, right?

Yes of course, Hans. I meant the proposed solution is cleaner than the initrd
(built-in or otherwise). The proc file location is fixed and all tools can
just refer to it than worrying about where in the initrd are the headers
located.

thanks!

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Wed, Mar 06, 2019 at 04:07:13PM -0800, H. Peter Anvin wrote:
> On 3/6/19 3:37 PM, Daniel Colascione wrote:
> > 
> > I just don't get the opposition to Joel's work. The rest of the thread
> > already goes into detail about the problems with pure-filesystem
> > solutions, and you and others are just totally ignoring those
> > well-thought-out rationales for the module approach and doing
> > inflooping on "lol just use a tarball". That's not productive.
> > 
> > Look; here's the bottom line: without this work, doing certain kinds
> > of system tracing is a nightmare, and with this patch, it Just Works.
> > You're arguing that various tools should do a better job of keeping
> > the filesystem in sync with the kernel. Maybe you're right. But we
> > don't live in a world where they will, because if this coherence were
> > going to happen, it'd work already. But this work solves the problem:
> > by necessity, anything that changes a kernel image *must* update
> > modules coherently, whether the kernel image and module come from the
> > filesystem, network boot, some kind of SQL database, or carrier
> > pigeon. There's nothing wrong with work that very cheaply makes the
> > kernel self-describing (introspection is elegant) and that takes
> > advantage of *existing* kernel tooling infrastructure to transparently
> > do a new thing.
> > 
> > You don't have to use this patch if you don't want to. Please stop
> > trying to block it.
> > 
> 
> No, that's not how this works. It is far worse to do something the wrong
> way than not doing it at all, when it affects the kernel-user space
> interactions.
> 
> Experience -- and we have almost 30 years of it -- has shown that hacks
> of this nature become engrained and all of a sudden is "mandatory". At
> the *very least* it needs to comply with the zero-one-infinity rule
> rather than being yet another ad hoc hack.
> 
> More fundamentally, we already have multiple ways to handle objects that
> need to go into the filesystem: they can be installed with (or as)
> modules, they can use the firmware interface, and so on.
> 
> Saying "it can be a module" is worse than a copout: even if dynamically
> loaded -- and many setups lock out post-boot module loadings for
> security reasons -- there is nothing to cause it to unload.
> 
> The bottom line is that in the end there is no difference between making
> this an archive of some sort and a module, except that to use the module
> you need privilege that you otherwise may not need. If your argument is
> that someone may not be providing the whole set of items provided by
> "make modules_install", what is there to say that they would include
> your specific module?

They would include the module because we can enforce that certain
config options in the kernel need to be enabled for Android features to work.
People can obviously mess up their kernel config in any number of ways and
misconfigure their kernel such that user space may even fail to boot. So how
is it relevant that people would include the module or not? If they want a
feature to work, their kernel has to enable the feature, right?

About disabling post-boot module loading, we don't do it on Android and
that's really the target system here. Can we not deal with such post-boot
module load disabling issue at a later time if such user really wants this?
If there is a CONFIG option that disables post-boot module loading, then we
can make this feature depend on that CONFIG being disabled.

> Here are some better ways of implementation I can see:
> 
> 1. Include an archive in "make modules_install". Most trivial; no kernel
>    changes at all.

As I said in previous posts, some people want to boot a kernel image without
any update to FS, for debug purposes. In the Android/embedded world,
developers want to build and boot a single binary blob (for debugging). For
such usecases, hacking modules_install would not help.

Once they are done debugging, they can switch the CONFIG option back to
module instead of building it in and consuming memory, so that it is not
loaded into memory until needed.

> 2. Generalize the initramfs code to be able to create a pre-populated
>    tmpfs at any time, that can be populated from an archive provided by
>    the firmware loading mechanism; like all firmware allows it to either
>    be built in or fetched from the filesystem. This allows it to be
>    built in to the kernel image if that becomes necessary; using tmpfs
>    means that it can be pushed out to swap rather than permanently
>    stored in kernel memory, and this filesystem can be unmounted freeing
>    its memory.

I see your points about how building it in can make it permanently lost
memory, but would that really be an issue if it is built-in by a user ONLY
for debugging?

Anyway, I'll try to think more about how I can make it memory that can be
unloaded/removed even if built-in into the kernel.

About making it a firmware built-in, and then populating tmpfs with the
firmware's contents, isn't that pointless if the firmware is built-in into
the kernel, and the memory that the firmware blob itself consumes is
permanently lost? Whether the tmpfs copy is swapped out or not, would not
help save the built-in firmware's memory.

> 3. Use a squashfs image instead of an archive.

This point number 3 sounds like a non-starter because it will make the kernel
build system fail if squashfs-tools is not available. So it is not an option
I think. I also don't see how it would solve the above swapping issue you
mentioned.  Whether it is an archive or squashfs image doesn't matter as much
as where the image lives.

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Enrico Weigelt, metux IT consult]

On 07.03.19 02:42, Joel Fernandes wrote:


Speaking as an embedded linux architect, who's daily job for a large
part is to getting bootloader, kernel, userland and customer
applications play nice together, in a stable, secure and easily
maintainable way:


> They would include the module because we can enforce that certain> config options in the kernel need to be enabled for Android features
to work.

Ok, but then you could also enforce them to simply put a tarball to the
module directory.

The modules need to built and deployed together with the kernel image
itself anyways, as well as they need to live on some filesystem that
has to be mounted at runtime. So, putting a tarball there is obvious
and trivial. No need to invent extra tools to build and use that stuff,
don't even need to touch the kernel source for that.

> As I said in previous posts, some people want to boot a kernel image without> any update to FS, for debug purposes.

So, it's just a workaround for your weird build/deployment mechanisms ?
(sorry, but I've got a hard time containing myself from starting a
really big rant against the Android architecture and build/deployment)

> In the Android/embedded world,> developers want to build and boot a single binary blob (for debugging).

Maybe in Android world (I didn't touch this for aeons - I'm just too
lazy add another hdd for it and wait days for everything pulled and
compiled), but in embedded world (speaking of: industrial machinery,
power plants, freighter ships, locomotives, medical devices or even
TV sets - that's the stuff I'm doing all the day), we usually either
build and deploy full system images (including the whole userland)
or use package management (eg. dpkg, ipkg, etc).

For me, as an embedded linux architect, the whole idea of having the
kernel have a completely different lifecycle process (even from a
completely different vendor) than the userland, is a really weird idea.

[ Yes: I've sometimes got customers with similar weird ideas (like BSP
coming from the board vendor, as a binary image, and they "just" put
their application ontop), but sooner or later this heavily crashes
against the wall, and then I can teach them how to do it right. ]

> Once they are done debugging, they can switch the CONFIG option back to> module instead of building it in and consuming memory, so that it is
not> loaded into memory until needed.
Okay, if it's just for debugging (in the lab, not in the field), then
why not fixing your development environment ? Why do you need that very
android specific stuff mainlined ?

Quite frankly: the whole thing really reminds me to the folks who wanna
push desktop bus into the kernel, just because their strange init system
is entirely based on that and they've created some nasty circular
dependencies. Please not yet another flamewar of that kind.

IMHO, you've got a serious architectural userland and development
environment problem (the massive problem of many millions completely
unmaintained and insecure devices in the field comes from exactly
that. Begins with the same fundamental mistakes which MS also still
refused learn, over decades - lack of a decent package management).

I doubt that workarounds in the kernel are good solution to that.

>> 3. Use a squashfs image instead of an archive.>
> This point number 3 sounds like a non-starter because it will make the kernel
> build system fail if squashfs-tools is not available. 

apt-get install squashfs-tools ?

Sorry, but the whole Android build machinery is so *extremly* huge,
that adding yet another tiny tool, which is in all relevant distros
for  aeons, really shouldn't be a problem worth even talking about.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[H. Peter Anvin]

On 3/6/19 3:37 PM, Daniel Colascione wrote:
> 
> I just don't get the opposition to Joel's work. The rest of the thread
> already goes into detail about the problems with pure-filesystem
> solutions, and you and others are just totally ignoring those
> well-thought-out rationales for the module approach and doing
> inflooping on "lol just use a tarball". That's not productive.
> 

You might think they are well thought out, but at least from what I can
tell they seem completely spurious.

	-hpa

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Wed, Mar 6, 2019 at 4:33 PM H. Peter Anvin <hpa@zytor.com> wrote:
>
> On 3/6/19 3:37 PM, Daniel Colascione wrote:
> >
> > I just don't get the opposition to Joel's work. The rest of the thread
> > already goes into detail about the problems with pure-filesystem
> > solutions, and you and others are just totally ignoring those
> > well-thought-out rationales for the module approach and doing
> > inflooping on "lol just use a tarball". That's not productive.
> >
>
> You might think they are well thought out, but at least from what I can
> tell they seem completely spurious.

That sentence is a general-purpose objection to literally anything.
Anything so general is useless. If you want to claim that the
rationale behind the work is inadequate, you have to explain why the
use cases that it enables are either illegitimate or amenable to other
solutions, not just call them spurious.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Enrico Weigelt, metux IT consult]

On 07.03.19 00:09, Pavel Machek wrote:

> So your licensing requirements prevent you from having headers in the
> filesystem, but allow module with the headers hidden inside on the
> filesystem?

Maybe it's just because I've missed most of the thread, but which
license requirements exactly could that be ?

The kernel still is GPL, isn't it ? So, any device vendor needs to hand
out the exact kernel source he hat built the image from, anyways.
(by the way: we should start a flash mob action of demanding sources
from the pocket-computers-with-3g vendors, but that's another story ;-))

So, what kind of license issue could it be, which prevents putting the
files (that you'd have to hand out anyways) directly into filesystem,
while allowing them in a .ko in the same filesystem ?

I'm confused.

> Looks like you should just tar xvzf
> this-is-a-kernel-module-I-promise.ko /usr/src/linux/include :-).

We should ask coreutils folks to add an ELF encapsulation to tar ;-)


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Thu, Mar 07, 2019 at 01:42:45AM +0100, Enrico Weigelt, metux IT consult wrote:
> On 07.03.19 00:09, Pavel Machek wrote:
> 
> > So your licensing requirements prevent you from having headers in the
> > filesystem, but allow module with the headers hidden inside on the
> > filesystem?
> 
> Maybe it's just because I've missed most of the thread, but which
> license requirements exactly could that be ?
> 
> The kernel still is GPL, isn't it ? So, any device vendor needs to hand
> out the exact kernel source he hat built the image from, anyways.
> (by the way: we should start a flash mob action of demanding sources
> from the pocket-computers-with-3g vendors, but that's another story ;-))
> 
> So, what kind of license issue could it be, which prevents putting the
> files (that you'd have to hand out anyways) directly into filesystem,
> while allowing them in a .ko in the same filesystem ?
> 
> I'm confused.

Take a look at this thread: https://lkml.org/lkml/2019/2/28/634

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Enrico Weigelt, metux IT consult]

On 07.03.19 02:48, Joel Fernandes wrote:
>> I'm confused.
> 
> Take a look at this thread: https://lkml.org/lkml/2019/2/28/634

Okay, replying to that mail:

> > > There's no linux-headers

That's the first fundamental problem. Actually, there's not even any
decent package management at all (no, apk seriously doesn't count as
that, not if you're used to apt+frieds for over 20 years)

> I have already been down that road. In the Android ecosystem, the
> Android teams only provide a "userspace system image" which goes on
> the system partition of the flash (and a couple other images are also
> provided but system is the main one).

These Android teams should learn how GNU/Linux distros and package
management works for over 20 years. Really, this is pretty trivial.

For such kind of general purpose devices, where users can install
arbitrary applications, I'd never come to the strange idea of deploying
whole operating system and applications in one image. (most likely not
even initially in the factory)

> The system image cannot contain GPL source code.

Why exactly ? Beacuse Big Manitu said so ?

Don't these folks that GPL doesn't prohibit shipping binaries on
disk/flash images ?

> It is also not possible to put kernel headers for every kernel version
> on the system images that ship and is not practical.

Of course not. They should only be deployed when needed, for the
versions needed.

> Android boots on 1000s of forked kernels.

Next big fundamental problem. Why all these forks in the first place ?

I'm not talking about the small patch queue ontop of mainline (or maybe
Andoid kernel baseline) - that's something we do all the day in embedded
world (of course we try to mainline as much as we can).

But what I usually see from the Andoid vendors are pretty much full
forks (full of horrible hacks) that quickly aren't maintained anymore.

If I'd be the responsible manager @google, that'd be one of the very
first things I'd care for: mainline first. (and, of course, ban all
proprietary binary-only kernel modules).

> Now for kernel modules, there's another image called the "vendor
> image" which is flashed onto the vendor parition, this is where kernel
> modules go.

Okay, then why not just putting a tarball there ?


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Greg KH]

On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -549,6 +549,16 @@ config IKCONFIG_PROC
>  	  This option enables access to the kernel configuration file
>  	  through /proc/config.gz.
>  
> +config IKHEADERS_PROC
> +	tristate "Enable kernel header artifacts through /proc/kheaders.tgz"
> +	select BUILD_BIN2C
> +	depends on PROC_FS
> +	help
> +	  This option enables access to the kernel header and other artifacts that
> +          are generated during the build process. These can be used to build kernel
> +          modules, and other in-kernel programs such as those generated by eBPF
> +          and systemtap tools.
> +

Minor nit, say what the module name here is if you pick the option as a
module?

thanks,

greg k-h

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Sat, Jan 19, 2019 at 09:26:04AM +0100, Greg KH wrote:
> On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> > --- a/init/Kconfig
> > +++ b/init/Kconfig
> > @@ -549,6 +549,16 @@ config IKCONFIG_PROC
> >  	  This option enables access to the kernel configuration file
> >  	  through /proc/config.gz.
> >  
> > +config IKHEADERS_PROC
> > +	tristate "Enable kernel header artifacts through /proc/kheaders.tgz"
> > +	select BUILD_BIN2C
> > +	depends on PROC_FS
> > +	help
> > +	  This option enables access to the kernel header and other artifacts that
> > +          are generated during the build process. These can be used to build kernel
> > +          modules, and other in-kernel programs such as those generated by eBPF
> > +          and systemtap tools.
> > +
> 
> Minor nit, say what the module name here is if you pick the option as a
> module?

Sure, will add the name of the module in the help text. thanks!

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Christoph Hellwig]

This seems like a pretty horrible idea and waste of kernel memory.
Just add support to kbuild to store a compressed archive in initramfs
and unpack it in the right place.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Greg KH]

On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig wrote:
> This seems like a pretty horrible idea and waste of kernel memory.

It's only a waste if you want it to be a waste, i.e. if you load the
kernel module.

This really isn't any different from how /proc/config.gz works.

> Just add support to kbuild to store a compressed archive in initramfs
> and unpack it in the right place.

I think the issue is that some devices do not use initramfs, or switch
away from it after init happens or something like that.  Joel has all of
the looney details that he can provide.

thanks,

greg k-h

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Sat, Jan 19, 2019 at 11:36:06AM +0100, Greg KH wrote:
> On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig wrote:
> > This seems like a pretty horrible idea and waste of kernel memory.
> 
> It's only a waste if you want it to be a waste, i.e. if you load the
> kernel module.
> 
> This really isn't any different from how /proc/config.gz works.

Yes, that's right, it can be built as a module and loaded and unloaded on
demand to free the kernel memory. Also such a module can be made a part of
the initrd image itself (like other kernel modules that are built into an
initrd), so this solution is already initrd-ready if anyone really wants it
that way.

> > Just add support to kbuild to store a compressed archive in initramfs
> > and unpack it in the right place.
> 
> I think the issue is that some devices do not use initramfs, or switch
> away from it after init happens or something like that.  Joel has all of
> the looney details that he can provide.

Initially I had written this as an image embedded into an init section,
similar to the initramfs. On boot if a boot argument was passed, then the
archive embedded was allocated additional non-init kernel memory to store it for
retrieval through /proc. After that, the image that's embedded would be
dropped entirely as its init memory. If no boot argument was passed, then no
additional memory is used at all. Even though I got that working, I decided
not to go with it because it was clunky because: 1. It required modification
of the linker scripts in the kernel to embed the image into the init section,
and also required a boot argument to be passed which makes the decision on
whether to have headers as a boot-time decision than a post-boot decision.
For these reasons, it is much cleaner to just have a loadable module and make
it on-demand, as I am doing in this patch similar to what /proc/config.gz
does.

thanks!

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[hpa]

On January 19, 2019 2:36:06 AM PST, Greg KH <gregkh@linuxfoundation.org> wrote:
>On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig wrote:
>> This seems like a pretty horrible idea and waste of kernel memory.
>
>It's only a waste if you want it to be a waste, i.e. if you load the
>kernel module.
>
>This really isn't any different from how /proc/config.gz works.
>
>> Just add support to kbuild to store a compressed archive in initramfs
>> and unpack it in the right place.
>
>I think the issue is that some devices do not use initramfs, or switch
>away from it after init happens or something like that.  Joel has all
>of
>the looney details that he can provide.
>
>thanks,
>
>greg k-h

Yeah, well... but it is kind of a losing game... the more in-kernel stuff there is the less smiley are things to actually be supported.

Modularizing is it in some ways even crazier in the sense that at that point you are relying on the filesystem containing the module, which has to be loaded into the kernel by a root user. One could even wonder if a better way to do this would be to have "make modules_install" park an archive file – or even a directory as opposed to a symlink – with this stuff in /lib/modules. We could even provide a tmpfs shim which autoloads such an archive via the firmware loader; this might even be generically useful, who knows.

Note also that initramfs contents can be built into the kernel. Extracting such content into a single-instance tmpfs would again be a possibility.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Sat, Jan 19, 2019 at 11:01:13PM -0800, hpa@zytor.com wrote:
> On January 19, 2019 2:36:06 AM PST, Greg KH <gregkh@linuxfoundation.org> wrote:
> >On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig wrote:
> >> This seems like a pretty horrible idea and waste of kernel memory.
> >
> >It's only a waste if you want it to be a waste, i.e. if you load the
> >kernel module.
> >
> >This really isn't any different from how /proc/config.gz works.
> >
> >> Just add support to kbuild to store a compressed archive in initramfs
> >> and unpack it in the right place.
> >
> >I think the issue is that some devices do not use initramfs, or switch
> >away from it after init happens or something like that.  Joel has all
> >of
> >the looney details that he can provide.
> >
> >thanks,
> >
> >greg k-h
> 
> Yeah, well... but it is kind of a losing game... the more in-kernel stuff there is the less smiley are things to actually be supported.

It is better than nothing, and if this makes things a bit easier and solves
real-world issues people have been having, and is optional, then I don't see
why not.

> Modularizing is it in some ways even crazier in the sense that at that point you are relying on the filesystem containing the module, which has to be loaded into the kernel by a root user. One could even wonder if a better way to do this would be to have "make modules_install" park an archive file – or even a directory as opposed to a symlink – with this stuff in /lib/modules. We could even provide a tmpfs shim which autoloads such an archive via the firmware loader; this might even be generically useful, who knows.

All this seems to assume where the modules are located. In Android, we don't
have /lib/modules. This patch generically fits into the grand scheme things
and I think is just better made a part of the kernel since it is not that
huge once compressed, as Dan also pointed. The more complex, and the more
assumptions we make, the less likely people writing tools will get it right
and be able to easily use it.

> 
> Note also that initramfs contents can be built into the kernel. Extracting such content into a single-instance tmpfs would again be a possibility

Such an approach would bloat the kernel image size though, which may not work
for everyone. The module based approach, on the other hand, gives an option
to the user to enable the feature, but not have it loaded into memory or used
until it is really needed.

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[hpa]

On January 20, 2019 8:10:03 AM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
>On Sat, Jan 19, 2019 at 11:01:13PM -0800, hpa@zytor.com wrote:
>> On January 19, 2019 2:36:06 AM PST, Greg KH
><gregkh@linuxfoundation.org> wrote:
>> >On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig wrote:
>> >> This seems like a pretty horrible idea and waste of kernel memory.
>> >
>> >It's only a waste if you want it to be a waste, i.e. if you load the
>> >kernel module.
>> >
>> >This really isn't any different from how /proc/config.gz works.
>> >
>> >> Just add support to kbuild to store a compressed archive in
>initramfs
>> >> and unpack it in the right place.
>> >
>> >I think the issue is that some devices do not use initramfs, or
>switch
>> >away from it after init happens or something like that.  Joel has
>all
>> >of
>> >the looney details that he can provide.
>> >
>> >thanks,
>> >
>> >greg k-h
>> 
>> Yeah, well... but it is kind of a losing game... the more in-kernel
>stuff there is the less smiley are things to actually be supported.
>
>It is better than nothing, and if this makes things a bit easier and
>solves
>real-world issues people have been having, and is optional, then I
>don't see
>why not.
>
>> Modularizing is it in some ways even crazier in the sense that at
>that point you are relying on the filesystem containing the module,
>which has to be loaded into the kernel by a root user. One could even
>wonder if a better way to do this would be to have "make
>modules_install" park an archive file – or even a directory as opposed
>to a symlink – with this stuff in /lib/modules. We could even provide a
>tmpfs shim which autoloads such an archive via the firmware loader;
>this might even be generically useful, who knows.
>
>All this seems to assume where the modules are located. In Android, we
>don't
>have /lib/modules. This patch generically fits into the grand scheme
>things
>and I think is just better made a part of the kernel since it is not
>that
>huge once compressed, as Dan also pointed. The more complex, and the
>more
>assumptions we make, the less likely people writing tools will get it
>right
>and be able to easily use it.
>
>> 
>> Note also that initramfs contents can be built into the kernel.
>Extracting such content into a single-instance tmpfs would again be a
>possibility
>
>Such an approach would bloat the kernel image size though, which may
>not work
>for everyone. The module based approach, on the other hand, gives an
>option
>to the user to enable the feature, but not have it loaded into memory
>or used
>until it is really needed.
>
>thanks,
>
> - Joel

Well, where are the modules? They must exist in the filesystem.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Sun, Jan 20, 2019 at 01:58:15PM -0800, hpa@zytor.com wrote:
> On January 20, 2019 8:10:03 AM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
> >On Sat, Jan 19, 2019 at 11:01:13PM -0800, hpa@zytor.com wrote:
> >> On January 19, 2019 2:36:06 AM PST, Greg KH
> ><gregkh@linuxfoundation.org> wrote:
> >> >On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig wrote:
> >> >> This seems like a pretty horrible idea and waste of kernel memory.
> >> >
> >> >It's only a waste if you want it to be a waste, i.e. if you load the
> >> >kernel module.
> >> >
> >> >This really isn't any different from how /proc/config.gz works.
> >> >
> >> >> Just add support to kbuild to store a compressed archive in
> >initramfs
> >> >> and unpack it in the right place.
> >> >
> >> >I think the issue is that some devices do not use initramfs, or
> >switch
> >> >away from it after init happens or something like that.  Joel has
> >all
> >> >of
> >> >the looney details that he can provide.
> >> >
> >> >thanks,
> >> >
> >> >greg k-h
> >> 
> >> Yeah, well... but it is kind of a losing game... the more in-kernel
> >stuff there is the less smiley are things to actually be supported.
> >
> >It is better than nothing, and if this makes things a bit easier and
> >solves
> >real-world issues people have been having, and is optional, then I
> >don't see
> >why not.
> >
> >> Modularizing is it in some ways even crazier in the sense that at
> >that point you are relying on the filesystem containing the module,
> >which has to be loaded into the kernel by a root user. One could even
> >wonder if a better way to do this would be to have "make
> >modules_install" park an archive file – or even a directory as opposed
> >to a symlink – with this stuff in /lib/modules. We could even provide a
> >tmpfs shim which autoloads such an archive via the firmware loader;
> >this might even be generically useful, who knows.
> >
> >All this seems to assume where the modules are located. In Android, we
> >don't
> >have /lib/modules. This patch generically fits into the grand scheme
> >things
> >and I think is just better made a part of the kernel since it is not
> >that
> >huge once compressed, as Dan also pointed. The more complex, and the
> >more
> >assumptions we make, the less likely people writing tools will get it
> >right
> >and be able to easily use it.
> >
> >> 
> >> Note also that initramfs contents can be built into the kernel.
> >Extracting such content into a single-instance tmpfs would again be a
> >possibility
> >
> >Such an approach would bloat the kernel image size though, which may
> >not work
> >for everyone. The module based approach, on the other hand, gives an
> >option
> >to the user to enable the feature, but not have it loaded into memory
> >or used
> >until it is really needed.
> >
> >thanks,
> >
> > - Joel
> 
> Well, where are the modules? They must exist in the filesystem.

The scheme of loading a module doesn't depend on _where_ the module is on the
filesystem. As long as a distro knows how to load a module in its own way (by
looking into whichever paths it cares about), that's all that matters.  And
the module contains compressed headers which saves space, vs storing it
uncompressed on the file system.

To remove complete reliance on the filesystem, there is an option of not
building it as a module, and making it as a built-in.

I think I see your point now - you're saying if its built-in, then it
becomes kernel memory that is lost and unswappable. Did I get that right?
I am saying that if that's a major concern, then:
1. Don't make it a built-in, make it a module.
2. Don't enable it at for your distro, and use a linux-headers package or
whatever else you have been using so far that works for you.

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[hpa]

On January 20, 2019 5:45:53 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
>On Sun, Jan 20, 2019 at 01:58:15PM -0800, hpa@zytor.com wrote:
>> On January 20, 2019 8:10:03 AM PST, Joel Fernandes
><joel@joelfernandes.org> wrote:
>> >On Sat, Jan 19, 2019 at 11:01:13PM -0800, hpa@zytor.com wrote:
>> >> On January 19, 2019 2:36:06 AM PST, Greg KH
>> ><gregkh@linuxfoundation.org> wrote:
>> >> >On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig
>wrote:
>> >> >> This seems like a pretty horrible idea and waste of kernel
>memory.
>> >> >
>> >> >It's only a waste if you want it to be a waste, i.e. if you load
>the
>> >> >kernel module.
>> >> >
>> >> >This really isn't any different from how /proc/config.gz works.
>> >> >
>> >> >> Just add support to kbuild to store a compressed archive in
>> >initramfs
>> >> >> and unpack it in the right place.
>> >> >
>> >> >I think the issue is that some devices do not use initramfs, or
>> >switch
>> >> >away from it after init happens or something like that.  Joel has
>> >all
>> >> >of
>> >> >the looney details that he can provide.
>> >> >
>> >> >thanks,
>> >> >
>> >> >greg k-h
>> >> 
>> >> Yeah, well... but it is kind of a losing game... the more
>in-kernel
>> >stuff there is the less smiley are things to actually be supported.
>> >
>> >It is better than nothing, and if this makes things a bit easier and
>> >solves
>> >real-world issues people have been having, and is optional, then I
>> >don't see
>> >why not.
>> >
>> >> Modularizing is it in some ways even crazier in the sense that at
>> >that point you are relying on the filesystem containing the module,
>> >which has to be loaded into the kernel by a root user. One could
>even
>> >wonder if a better way to do this would be to have "make
>> >modules_install" park an archive file – or even a directory as
>opposed
>> >to a symlink – with this stuff in /lib/modules. We could even
>provide a
>> >tmpfs shim which autoloads such an archive via the firmware loader;
>> >this might even be generically useful, who knows.
>> >
>> >All this seems to assume where the modules are located. In Android,
>we
>> >don't
>> >have /lib/modules. This patch generically fits into the grand scheme
>> >things
>> >and I think is just better made a part of the kernel since it is not
>> >that
>> >huge once compressed, as Dan also pointed. The more complex, and the
>> >more
>> >assumptions we make, the less likely people writing tools will get
>it
>> >right
>> >and be able to easily use it.
>> >
>> >> 
>> >> Note also that initramfs contents can be built into the kernel.
>> >Extracting such content into a single-instance tmpfs would again be
>a
>> >possibility
>> >
>> >Such an approach would bloat the kernel image size though, which may
>> >not work
>> >for everyone. The module based approach, on the other hand, gives an
>> >option
>> >to the user to enable the feature, but not have it loaded into
>memory
>> >or used
>> >until it is really needed.
>> >
>> >thanks,
>> >
>> > - Joel
>> 
>> Well, where are the modules? They must exist in the filesystem.
>
>The scheme of loading a module doesn't depend on _where_ the module is
>on the
>filesystem. As long as a distro knows how to load a module in its own
>way (by
>looking into whichever paths it cares about), that's all that matters. 
>And
>the module contains compressed headers which saves space, vs storing it
>uncompressed on the file system.
>
>To remove complete reliance on the filesystem, there is an option of
>not
>building it as a module, and making it as a built-in.
>
>I think I see your point now - you're saying if its built-in, then it
>becomes kernel memory that is lost and unswappable. Did I get that
>right?
>I am saying that if that's a major concern, then:
>1. Don't make it a built-in, make it a module.
>2. Don't enable it at for your distro, and use a linux-headers package
>or
>whatever else you have been using so far that works for you.
>
>thanks,
>
> - Joel

My point is that if we're going to actually solve a problem, we need to make it so that the distro won't just disable it anyway, and it ought to be something scalable; otherwise nothing is gained.

I am *not* disagreeing with the problem statement!

Now, /proc isn't something that will autoload modules. A filesystem *will*, although you need to be able to mount it; furthermore, it makes it trivially to extend it (and the firmware interface provides an . easy way to feed the data to such a filesystem without having to muck with anything magic.)

Heck, we could even make it a squashfs image that can just be mounted.

So, first of all, where does Android keep its modules, and what is actually included? Is /sbin/modprobe used to load the modules, as is normal? We might even be able to address this with some fairly trivial enhancements to modprobe; specifically to search in the module paths for something that isn't a module per se.

The best scenario would be if we could simply have the tools find the location equivalent of /lib/modules/$version/source...
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Sandeep Patil]

On Sun, Jan 20, 2019 at 06:49:56PM -0800, hpa@zytor.com wrote:
> On January 20, 2019 5:45:53 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
> >On Sun, Jan 20, 2019 at 01:58:15PM -0800, hpa@zytor.com wrote:
> >> On January 20, 2019 8:10:03 AM PST, Joel Fernandes
> ><joel@joelfernandes.org> wrote:
> >> >On Sat, Jan 19, 2019 at 11:01:13PM -0800, hpa@zytor.com wrote:
> >> >> On January 19, 2019 2:36:06 AM PST, Greg KH
> >> ><gregkh@linuxfoundation.org> wrote:
> >> >> >On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig
> >wrote:
> >> >> >> This seems like a pretty horrible idea and waste of kernel
> >memory.
> >> >> >
> >> >> >It's only a waste if you want it to be a waste, i.e. if you load
> >the
> >> >> >kernel module.
> >> >> >
> >> >> >This really isn't any different from how /proc/config.gz works.
> >> >> >
> >> >> >> Just add support to kbuild to store a compressed archive in
> >> >initramfs
> >> >> >> and unpack it in the right place.
> >> >> >
> >> >> >I think the issue is that some devices do not use initramfs, or
> >> >switch
> >> >> >away from it after init happens or something like that.  Joel has
> >> >all
> >> >> >of
> >> >> >the looney details that he can provide.
> >> >> >
> >> >> >thanks,
> >> >> >
> >> >> >greg k-h
> >> >> 
> >> >> Yeah, well... but it is kind of a losing game... the more
> >in-kernel
> >> >stuff there is the less smiley are things to actually be supported.
> >> >
> >> >It is better than nothing, and if this makes things a bit easier and
> >> >solves
> >> >real-world issues people have been having, and is optional, then I
> >> >don't see
> >> >why not.
> >> >
> >> >> Modularizing is it in some ways even crazier in the sense that at
> >> >that point you are relying on the filesystem containing the module,
> >> >which has to be loaded into the kernel by a root user. One could
> >even
> >> >wonder if a better way to do this would be to have "make
> >> >modules_install" park an archive file – or even a directory as
> >opposed
> >> >to a symlink – with this stuff in /lib/modules. We could even
> >provide a
> >> >tmpfs shim which autoloads such an archive via the firmware loader;
> >> >this might even be generically useful, who knows.
> >> >
> >> >All this seems to assume where the modules are located. In Android,
> >we
> >> >don't
> >> >have /lib/modules. This patch generically fits into the grand scheme
> >> >things
> >> >and I think is just better made a part of the kernel since it is not
> >> >that
> >> >huge once compressed, as Dan also pointed. The more complex, and the
> >> >more
> >> >assumptions we make, the less likely people writing tools will get
> >it
> >> >right
> >> >and be able to easily use it.
> >> >
> >> >> 
> >> >> Note also that initramfs contents can be built into the kernel.
> >> >Extracting such content into a single-instance tmpfs would again be
> >a
> >> >possibility
> >> >
> >> >Such an approach would bloat the kernel image size though, which may
> >> >not work
> >> >for everyone. The module based approach, on the other hand, gives an
> >> >option
> >> >to the user to enable the feature, but not have it loaded into
> >memory
> >> >or used
> >> >until it is really needed.
> >> >
> >> >thanks,
> >> >
> >> > - Joel
> >> 
> >> Well, where are the modules? They must exist in the filesystem.
> >
> >The scheme of loading a module doesn't depend on _where_ the module is
> >on the
> >filesystem. As long as a distro knows how to load a module in its own
> >way (by
> >looking into whichever paths it cares about), that's all that matters. 
> >And
> >the module contains compressed headers which saves space, vs storing it
> >uncompressed on the file system.
> >
> >To remove complete reliance on the filesystem, there is an option of
> >not
> >building it as a module, and making it as a built-in.
> >
> >I think I see your point now - you're saying if its built-in, then it
> >becomes kernel memory that is lost and unswappable. Did I get that
> >right?
> >I am saying that if that's a major concern, then:
> >1. Don't make it a built-in, make it a module.
> >2. Don't enable it at for your distro, and use a linux-headers package
> >or
> >whatever else you have been using so far that works for you.
> >
> >thanks,
> >
> > - Joel
> 
> My point is that if we're going to actually solve a problem, we need to make it so that the distro won't just disable it anyway, and it ought to be something scalable; otherwise nothing is gained.
> 
> I am *not* disagreeing with the problem statement!
> 
> Now, /proc isn't something that will autoload modules. A filesystem *will*, although you need to be able to mount it; furthermore, it makes it trivially to extend it (and the firmware interface provides an . easy way to feed the data to such a filesystem without having to muck with anything magic.)
> 
> Heck, we could even make it a squashfs image that can just be mounted.
> 
> So, first of all, where does Android keep its modules, and what is actually included? Is /sbin/modprobe used to load the modules, as is normal? We might even be able to address this with some fairly trivial enhancements to modprobe; specifically to search in the module paths for something that isn't a module per se.
> 

FWIW, 'modprobe' does exist on Android. Although most of the times,
Init's builtin insmod function[1] gets used.

The module locations are /{system, vendor,odm}/lib/modules and yes,
modprobe can be made to load modules from there too. [2]

- ssp

> The best scenario would be if we could simply have the tools find the location equivalent of /lib/modules/$version/source...


[1] https://android.googlesource.com/platform/system/core/+/master/init/builtins.cpp#230
[2] https://source.android.com/devices/architecture/kernel/modular-kernels

> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> 
> 

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

Hi Hans,

Thanks for the discussion and sorry for my late reply due to the holidays.
I replied inline below:

On Sun, Jan 20, 2019 at 06:49:56PM -0800, hpa@zytor.com wrote:
> On January 20, 2019 5:45:53 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
> >On Sun, Jan 20, 2019 at 01:58:15PM -0800, hpa@zytor.com wrote:
> >> On January 20, 2019 8:10:03 AM PST, Joel Fernandes
> ><joel@joelfernandes.org> wrote:
> >> >On Sat, Jan 19, 2019 at 11:01:13PM -0800, hpa@zytor.com wrote:
> >> >> On January 19, 2019 2:36:06 AM PST, Greg KH
> >> ><gregkh@linuxfoundation.org> wrote:
> >> >> >On Sat, Jan 19, 2019 at 02:28:00AM -0800, Christoph Hellwig
> >wrote:
> >> >> >> This seems like a pretty horrible idea and waste of kernel
> >memory.
> >> >> >
> >> >> >It's only a waste if you want it to be a waste, i.e. if you load
> >the
> >> >> >kernel module.
> >> >> >
> >> >> >This really isn't any different from how /proc/config.gz works.
> >> >> >
> >> >> >> Just add support to kbuild to store a compressed archive in
> >> >initramfs
> >> >> >> and unpack it in the right place.
> >> >> >
> >> >> >I think the issue is that some devices do not use initramfs, or
> >> >switch
> >> >> >away from it after init happens or something like that.  Joel has
> >> >all
> >> >> >of
> >> >> >the looney details that he can provide.
> >> >> >
> >> >> >thanks,
> >> >> >
> >> >> >greg k-h
> >> >> 
> >> >> Yeah, well... but it is kind of a losing game... the more
> >in-kernel
> >> >stuff there is the less smiley are things to actually be supported.
> >> >
> >> >It is better than nothing, and if this makes things a bit easier and
> >> >solves
> >> >real-world issues people have been having, and is optional, then I
> >> >don't see
> >> >why not.
> >> >
> >> >> Modularizing is it in some ways even crazier in the sense that at
> >> >that point you are relying on the filesystem containing the module,
> >> >which has to be loaded into the kernel by a root user. One could
> >even
> >> >wonder if a better way to do this would be to have "make
> >> >modules_install" park an archive file – or even a directory as
> >opposed
> >> >to a symlink – with this stuff in /lib/modules. We could even
> >provide a
> >> >tmpfs shim which autoloads such an archive via the firmware loader;
> >> >this might even be generically useful, who knows.
> >> >
> >> >All this seems to assume where the modules are located. In Android,
> >we
> >> >don't
> >> >have /lib/modules. This patch generically fits into the grand scheme
> >> >things
> >> >and I think is just better made a part of the kernel since it is not
> >> >that
> >> >huge once compressed, as Dan also pointed. The more complex, and the
> >> >more
> >> >assumptions we make, the less likely people writing tools will get
> >it
> >> >right
> >> >and be able to easily use it.
> >> >
> >> >> 
> >> >> Note also that initramfs contents can be built into the kernel.
> >> >Extracting such content into a single-instance tmpfs would again be
> >a
> >> >possibility
> >> >
> >> >Such an approach would bloat the kernel image size though, which may
> >> >not work
> >> >for everyone. The module based approach, on the other hand, gives an
> >> >option
> >> >to the user to enable the feature, but not have it loaded into
> >memory
> >> >or used
> >> >until it is really needed.
> >> >
> >> >thanks,
> >> >
> >> > - Joel
> >> 
> >> Well, where are the modules? They must exist in the filesystem.
> >
> >The scheme of loading a module doesn't depend on _where_ the module is
> >on the
> >filesystem. As long as a distro knows how to load a module in its own
> >way (by
> >looking into whichever paths it cares about), that's all that matters. 
> >And
> >the module contains compressed headers which saves space, vs storing it
> >uncompressed on the file system.
> >
> >To remove complete reliance on the filesystem, there is an option of
> >not
> >building it as a module, and making it as a built-in.
> >
> >I think I see your point now - you're saying if its built-in, then it
> >becomes kernel memory that is lost and unswappable. Did I get that
> >right?
> >I am saying that if that's a major concern, then:
> >1. Don't make it a built-in, make it a module.
> >2. Don't enable it at for your distro, and use a linux-headers package
> >or
> >whatever else you have been using so far that works for you.
> >
> >thanks,
> >
> > - Joel
> 
> My point is that if we're going to actually solve a problem, we need to make it so that the distro won't just disable it anyway, and it ought to be something scalable; otherwise nothing is gained.

Agreed. Although there are many ways distros can misconfigure a kernel and
shoot themselves in the foot.

> I am *not* disagreeing with the problem statement!

Ok, that's great to know. Thanks for all the discussions!

> Now, /proc isn't something that will autoload modules. A filesystem *will*, although you need to be able to mount it; furthermore, it makes it trivially to extend it (and the firmware interface provides an . easy way to feed the data to such a filesystem without having to muck with anything magic.)

My thought is for the tools needing the module to try to load them, if they
need the headers. And then unload the module once they are done with it.

> Heck, we could even make it a squashfs image that can just be mounted.

That would add another dependency on a kernel configuration though.

> So, first of all, where does Android keep its modules, and what is actually included? Is /sbin/modprobe used to load the modules, as is normal? We might even be able to address this with some fairly trivial enhancements to modprobe; specifically to search in the module paths for something that isn't a module per se.
> The best scenario would be if we could simply have the tools find the location equivalent of /lib/modules/$version/source...

Sandeep answered this in the other thread, modprobe is there in Android.
About storing sources on the filesystem, I already went down this route (this
was my first effort) and there is quite some resistance to ship sources on
the filesystem of an Android device due to Licensing reasons. A kernel module
though is something that's a binary artifact and is not something
"distributed" by Google.

The other issue is with enforcing different OEM vendors that they should
store kernel-header sources on their Android products. That's much harder to
enforce, however we can easily enforce kernel config options be enabled,
since we already enforce a kernel configuration options that are needed for
Android to work correctly:
https://source.android.com/devices/architecture/kernel/config. So this config
option can just be another one.

By the way, we can easily write a script to just extract the .ko directly -
if the whole "load it as a module" thing bothers you. The kheaders.ko can
just be thought of as a tarball. There's already a script to extract
/proc/config.gz in the same/similar way: scripts/extract-ikconfig

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Karim Yaghmour]

On 1/22/19 2:39 PM, Joel Fernandes wrote:
[snip]
> On Sun, Jan 20, 2019 at 06:49:56PM -0800, hpa@zytor.com wrote:
[snip]
>> My point is that if we're going to actually solve a problem, we need to make it so that the distro won't just disable it anyway, and it ought to be something scalable; otherwise nothing is gained.
> 
> Agreed. Although there are many ways distros can misconfigure a kernel and
> shoot themselves in the foot.
> 
>> I am *not* disagreeing with the problem statement!
> 
> Ok, that's great to know. Thanks for all the discussions!
> 
>> Now, /proc isn't something that will autoload modules. A filesystem *will*, although you need to be able to mount it; furthermore, it makes it trivially to extend it (and the firmware interface provides an . easy way to feed the data to such a filesystem without having to muck with anything magic.)
> 
> My thought is for the tools needing the module to try to load them, if they
> need the headers. And then unload the module once they are done with it.
> 
>> Heck, we could even make it a squashfs image that can just be mounted.
> 
> That would add another dependency on a kernel configuration though.
> 
>> So, first of all, where does Android keep its modules, and what is actually included? Is /sbin/modprobe used to load the modules, as is normal? We might even be able to address this with some fairly trivial enhancements to modprobe; specifically to search in the module paths for something that isn't a module per se.
>> The best scenario would be if we could simply have the tools find the location equivalent of /lib/modules/$version/source...
> 
> Sandeep answered this in the other thread, modprobe is there in Android.
> About storing sources on the filesystem, I already went down this route (this
> was my first effort) and there is quite some resistance to ship sources on
> the filesystem of an Android device due to Licensing reasons. A kernel module
> though is something that's a binary artifact and is not something
> "distributed" by Google.
> 
> The other issue is with enforcing different OEM vendors that they should
> store kernel-header sources on their Android products. That's much harder to
> enforce, however we can easily enforce kernel config options be enabled,
> since we already enforce a kernel configuration options that are needed for
> Android to work correctly:
> https://source.android.com/devices/architecture/kernel/config. So this config
> option can just be another one.
> 
> By the way, we can easily write a script to just extract the .ko directly -
> if the whole "load it as a module" thing bothers you. The kheaders.ko can
> just be thought of as a tarball. There's already a script to extract
> /proc/config.gz in the same/similar way: scripts/extract-ikconfig

If I may add a few more thoughts here ... in no specific order:

I've been helping customers put Android in all sorts of weird devices 
over the past decade, handsets included, and have had a front-row seat 
to how this ecosystem works/has-evolved (Google, SoC vendors, 
manufacturers, etc.) One thing that stands out is that while the outside 
perception might be that Google has a stronghold on what "partners" do, 
the reality is that almost any rule that can be broken will be broken by 
someone in the ecosystem. And yet, still, Google somehow has to manage 
and provide app developers and, ultimately, consumers with a coherent 
experience, while still herding everyone to more-or-less go in the same 
direction ...

 From that point of view, if something comes from or is rooted in 
mainline, instead of being mandated, it's usually easier to find it 
across the board. A perfect example of this is ftrace. The fact that 
it's in mainline has enabled google to: a) instrument their entire stack 
to log events to it (see 
http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/82 
and 
http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/83), 
and b) provide app-developer-facing tools (see 
https://developer.android.com/studio/command-line/systrace). Since this 
tracing functionality is now integrated into Android Studio (look for 
"System Trace" here: 
https://developer.android.com/studio/profile/cpu-profiler), it's very 
much "standard android" and additional proof, if any was needed, that 
tracing is useful to everyone.

A few years back I was asked by a customer to put together some class 
material for internal Android debugging and performance analysis 
(commercial disclaimer applies, but slides/exercises are under 
"courseware": 
http://www.opersys.com/training/android-debug-and-performance). ftrace 
was very much in those early versions and it was great to show people 
that they could use it "out of the box". Recently I wanted to update 
this class material to cover eBPF and its applicability in Android. Holy 
cow. That turned out to be less obvious than necessary and somewhat 
peculiar to pull off. In this specific case, Joel tried a few things 
(see 
http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/111) 
before eventually settling on loading a Debian chroot jail into a live 
Android (https://github.com/joelagnel/adeb) ... all of which require a 
proper set of kernel headers to properly function. Don't get me wrong, 
Joel's Androdeb makes this easy, but it's still outside the standard 
Android MO.

In short, let's just say that, contrary to ftrace, I don't see the path 
for eBPF to being part of the standard toolset used by app developers 
any time soon. The recently-released bpftrace might help in that regard, 
but the kernel headers aren't too far in that regard either.

Personally I advocated a more aggressive approach with Joel in private: 
just put the darn headers straight into the kernel image, it's the 
*only* artifact we're sure will follow the Android device whatever 
happens to it (like built-in ftrace). To that end, I even had some crazy 
ideas on how to compress the headers even further than with std 
compression algorithms -- here's a snippet from an email I sent Joel 
some time back detailing such a hack:
> Since C headers have fairly constrained semantics and since the types of semantics generally used to name structs, etc. in the Linux kernel are well established, we can likely devise a very customized compression algorithm for the purpose.
> 
> We know, for example, that there are quite a few ascii characters that can't be found in a normal header. So we can play some games. Say, replace all the "#" dictionary with 2 bytes, the first being a magic (illegal) ascii character and the 2nd with a constant. As such, we could replace all #foo entries with 2 bytes:
> #define goes from 7 bytes to 2
> #include from 8 to 2
> #ifdef from 6 to 2
> etc.
> 
> The same goes for several keywords of the C language and words very often found in Linux kernel headers:
> struct from 6 to 2
> linux from 5 to 2
> CONFIG from 6 to 2
> "unsigned long" from 13 to 2
> etc.
> 
> And if we run out of space with one magic character (256), we can start using a next one.
> 
> We'd have a script that runs at build time that does this replacement, and we could even package the counter-script as part of the kernel image the to convert the "cooked" headers back to their original format; ex: Steven has a README in the ftrace directory.
> 
> Such a replacement strategy combined with stripping comments and gzipping the result may provide some very interesting compression results. 

Whether such craziness makes sense or is adopted or not isn't mine to 
chart, but I certainly can't see eBPF reaching the same mass deployment 
ftrace has within the Android ecosystem until there's a way to use it 
without having to chase kernel headers independently of kernel images. 
There are "too many clicks" involved and someone somewhere will drop the 
ball if it's not glued to the kernel in some way shape or form. Any 
solution that solves this is one I'd love to hear about.

My $0.02

-- 
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Wed, Jan 23, 2019 at 1:29 PM Karim Yaghmour
<karim.yaghmour@opersys.com> wrote:
> > By the way, we can easily write a script to just extract the .ko directly -
> > if the whole "load it as a module" thing bothers you. The kheaders.ko can
> > just be thought of as a tarball. There's already a script to extract
> > /proc/config.gz in the same/similar way: scripts/extract-ikconfig
>
> If I may add a few more thoughts here ... in no specific order:
>
>  From that point of view, if something comes from or is rooted in
> mainline, instead of being mandated, it's usually easier to find it
> across the board. A perfect example of this is ftrace. The fact that
> it's in mainline has enabled google to: a) instrument their entire stack
> to log events to it (see
> http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/82
> and
> http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/83),
> and b) provide app-developer-facing tools (see
> https://developer.android.com/studio/command-line/systrace). Since this
> tracing functionality is now integrated into Android Studio (look for
> "System Trace" here:
> https://developer.android.com/studio/profile/cpu-profiler), it's very
> much "standard android" and additional proof, if any was needed, that
> tracing is useful to everyone.
>
> A few years back I was asked by a customer to put together some class
> material for internal Android debugging and performance analysis
> (commercial disclaimer applies, but slides/exercises are under
> "courseware":
> http://www.opersys.com/training/android-debug-and-performance). ftrace
> was very much in those early versions and it was great to show people
> that they could use it "out of the box". Recently I wanted to update
> this class material to cover eBPF and its applicability in Android. Holy
> cow. That turned out to be less obvious than necessary and somewhat
> peculiar to pull off. In this specific case, Joel tried a few things
> (see
> http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/111)
> before eventually settling on loading a Debian chroot jail into a live
> Android (https://github.com/joelagnel/adeb) ... all of which require a
> proper set of kernel headers to properly function. Don't get me wrong,
> Joel's Androdeb makes this easy, but it's still outside the standard
> Android MO.
>
> In short, let's just say that, contrary to ftrace, I don't see the path
> for eBPF to being part of the standard toolset used by app developers
> any time soon. The recently-released bpftrace might help in that regard,
> but the kernel headers aren't too far in that regard either.

While I think there's definitely a place for eBPF as part of the
Android performance toolkit, I think most users will end up using it
through rich front-end performance collection and analysis tools (of
the sort I'm working on) rather than directly as a first-line window
into the operation of the system. Below this level is probably
something like bpftrace, and below that, raw eBPF and ftrace
manipulation. It's also worth noting that much of the time, system
analysis is retrospection, not inspection (e.g., investigating the
causes of rare and hard-to-reproduce bad behavior), and so iteration
via interactive specification of eBPF programs isn't a practical path
forward. It's still useful, even in this scenario, to be able (as part
of higher-level tools) attach "canned" eBPF programs to the kernel to
extract certain generally-useful bits of information, and in this
capacity, Joel's header module would be useful.

> Personally I advocated a more aggressive approach with Joel in private:
> just put the darn headers straight into the kernel image, it's the
> *only* artifact we're sure will follow the Android device whatever
> happens to it (like built-in ftrace).

I was thinking along similar lines. Ordinarily, we make loadable
kernel modules. What we kind of want here is a non-loadable kernel
module --- or a non-loadable section in the kernel image proper. I'm
not familiar with early-stage kernel loader operation: I know it's
possible to crease discardable sections in the kernel image, but can
we create sections that are never slurped into memory in the first
place? If not, maybe loading and immediately discarding the header
section is good enough.

> To that end, I even had some crazy
> ideas on how to compress the headers even further than with std
> compression algorithms -- here's a snippet from an email I sent Joel
> some time back detailing such a hack:
> > Since C headers have fairly constrained semantics and since the types of semantics generally used to name structs, etc. in the Linux kernel are well established, we can likely devise a very customized compression algorithm for the purpose.

Would such a thing really do better than LZMA? LZMA already has very
clever techniques for eliminating long-range redundancies in
compressible text, including redundancies at the sub-byte level. I can
certainly understand the benefit of stripping comments, since removing
comments really does decrease the total amount of information the
compressor has to preserve, but I'm not sure how much the encoding
scheme you propose below would help, since it reminds me of the
encoding scheme that LZMA would discover automatically.

> Whether such craziness makes sense or is adopted or not isn't mine to
> chart, but I certainly can't see eBPF reaching the same mass deployment
> ftrace has within the Android ecosystem until there's a way to use it
> without having to chase kernel headers independently of kernel images.
> There are "too many clicks" involved and someone somewhere will drop the
> ball if it's not glued to the kernel in some way shape or form. Any
> solution that solves this is one I'd love to hear about.

I agree. There definitely needs to be a "just collect a damn trace"
button that works on any device, and for this button to work and
incorporate eBPF, the system needs to be able to describe itself.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Wed, Jan 23, 2019 at 02:37:47PM -0800, Daniel Colascione wrote:
> On Wed, Jan 23, 2019 at 1:29 PM Karim Yaghmour
> <karim.yaghmour@opersys.com> wrote:
[...]
> > Personally I advocated a more aggressive approach with Joel in private:
> > just put the darn headers straight into the kernel image, it's the
> > *only* artifact we're sure will follow the Android device whatever
> > happens to it (like built-in ftrace).
> 
> I was thinking along similar lines. Ordinarily, we make loadable
> kernel modules. What we kind of want here is a non-loadable kernel
> module --- or a non-loadable section in the kernel image proper. I'm
> not familiar with early-stage kernel loader operation: I know it's
> possible to crease discardable sections in the kernel image, but can
> we create sections that are never slurped into memory in the first
> place? If not, maybe loading and immediately discarding the header
> section is good enough.

I am happy to see if I can shrink it down further. Especially using xz and
stripping all comments period. I am optimistic this can be brought down
further to a point where it would make sense to everyone to build it into the
kernel. Lets see.

Last time I stripped comments, it went down by ~40%. What I haven't tried is
doing this *with* xz compression. I am also open to brainstorming what else
can be stripped.

OTOH the reason I didn't focus much on size is: modules are pretty much
universal and I'm confident of wide spread use of this feature for Android-based
products and not needing of "chasing headers" if we modularize it, since
Android's project treble has modularized things and modules are now default
enabled. Putting headers into a module lets us enjoy the ride there.

I am quite hosed for the next week or so to work on this, but I should be
able to get back to it after.

cheers,

 - Joel


> 
> > To that end, I even had some crazy
> > ideas on how to compress the headers even further than with std
> > compression algorithms -- here's a snippet from an email I sent Joel
> > some time back detailing such a hack:
> > > Since C headers have fairly constrained semantics and since the types of semantics generally used to name structs, etc. in the Linux kernel are well established, we can likely devise a very customized compression algorithm for the purpose.
> 
> Would such a thing really do better than LZMA? LZMA already has very
> clever techniques for eliminating long-range redundancies in
> compressible text, including redundancies at the sub-byte level. I can
> certainly understand the benefit of stripping comments, since removing
> comments really does decrease the total amount of information the
> compressor has to preserve, but I'm not sure how much the encoding
> scheme you propose below would help, since it reminds me of the
> encoding scheme that LZMA would discover automatically.
> 
> > Whether such craziness makes sense or is adopted or not isn't mine to
> > chart, but I certainly can't see eBPF reaching the same mass deployment
> > ftrace has within the Android ecosystem until there's a way to use it
> > without having to chase kernel headers independently of kernel images.
> > There are "too many clicks" involved and someone somewhere will drop the
> > ball if it's not glued to the kernel in some way shape or form. Any
> > solution that solves this is one I'd love to hear about.
> 
> I agree. There definitely needs to be a "just collect a damn trace"
> button that works on any device, and for this button to work and
> incorporate eBPF, the system needs to be able to describe itself.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Wed, Jan 23, 2019 at 09:32:16PM -0500, Joel Fernandes wrote:
> On Wed, Jan 23, 2019 at 02:37:47PM -0800, Daniel Colascione wrote:
> > On Wed, Jan 23, 2019 at 1:29 PM Karim Yaghmour
> > <karim.yaghmour@opersys.com> wrote:
> [...]
> > > Personally I advocated a more aggressive approach with Joel in private:
> > > just put the darn headers straight into the kernel image, it's the
> > > *only* artifact we're sure will follow the Android device whatever
> > > happens to it (like built-in ftrace).
> > 
> > I was thinking along similar lines. Ordinarily, we make loadable
> > kernel modules. What we kind of want here is a non-loadable kernel
> > module --- or a non-loadable section in the kernel image proper. I'm
> > not familiar with early-stage kernel loader operation: I know it's
> > possible to crease discardable sections in the kernel image, but can
> > we create sections that are never slurped into memory in the first
> > place? If not, maybe loading and immediately discarding the header
> > section is good enough.
> 
> I am happy to see if I can shrink it down further. Especially using xz and
> stripping all comments period. I am optimistic this can be brought down
> further to a point where it would make sense to everyone to build it into the
> kernel. Lets see.
> 
> Last time I stripped comments, it went down by ~40%. What I haven't tried is
> doing this *with* xz compression. I am also open to brainstorming what else
> can be stripped.

Removing comments (/* */) with xz compression brings it down to 3.3MB.

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Karim Yaghmour]

On 1/23/19 11:37 PM, Daniel Colascione wrote:
> While I think there's definitely a place for eBPF as part of the
> Android performance toolkit, I think most users will end up using it
> through rich front-end performance collection and analysis tools (of
> the sort I'm working on) rather than directly as a first-line window
> into the operation of the system.

Sure, I don't disagree.

> Below this level is probably
> something like bpftrace, and below that, raw eBPF and ftrace
> manipulation. It's also worth noting that much of the time, system
> analysis is retrospection, not inspection (e.g., investigating the
> causes of rare and hard-to-reproduce bad behavior), and so iteration
> via interactive specification of eBPF programs isn't a practical path
> forward. It's still useful, even in this scenario, to be able (as part
> of higher-level tools) attach "canned" eBPF programs to the kernel to
> extract certain generally-useful bits of information, and in this
> capacity, Joel's header module would be useful.

Hmm. Not sure I agree about that. There's no reason I can't use Android 
Studio to "right-click" on a line of code or even a span of code and 
select a "trace this in detail for me" option, where "in detail" could 
mean different things depending on the code that's highlighted. Doing 
I/O calls? Then automatically measure some I/O benchmarks for that 
portion of code. Doing graphics calls? Do the same for the graphics 
stack, etc.

>> Personally I advocated a more aggressive approach with Joel in private:
>> just put the darn headers straight into the kernel image, it's the
>> *only* artifact we're sure will follow the Android device whatever
>> happens to it (like built-in ftrace).
> 
> I was thinking along similar lines. Ordinarily, we make loadable
> kernel modules. What we kind of want here is a non-loadable kernel
> module --- or a non-loadable section in the kernel image proper. I'm
> not familiar with early-stage kernel loader operation: I know it's
> possible to crease discardable sections in the kernel image, but can
> we create sections that are never slurped into memory in the first
> place? If not, maybe loading and immediately discarding the header
> section is good enough.

Interesting. Maybe just append it to the image but have it not loaded 
and have a kernel parameter than enables a "/proc/kheaders" driver to 
know where the fetch the appended headers from storage at runtime. There 
would be no RAM loading whatsoever of the headers, just some sort of 
"kheaders=/dev/foobar:offset:size" parameter. If you turn the option on, 
you get a fatter kernel image size to store on permanent storage, but no 
impact on what's loaded at boot time.

> Would such a thing really do better than LZMA? LZMA already has very
> clever techniques for eliminating long-range redundancies in
> compressible text, including redundancies at the sub-byte level. I can
> certainly understand the benefit of stripping comments, since removing
> comments really does decrease the total amount of information the
> compressor has to preserve, but I'm not sure how much the encoding
> scheme you propose below would help, since it reminds me of the
> encoding scheme that LZMA would discover automatically.

I'm no compression algorithm expert. If you say LZMA would do the 
same/better than what I suggested then I have no reason to contest that. 
My goal is to see the headers as part of the kernel image that's 
distributed on devices so that they don't have to be chased around. I'm 
just trying to make it as palatable as possible.

>> Whether such craziness makes sense or is adopted or not isn't mine to
>> chart, but I certainly can't see eBPF reaching the same mass deployment
>> ftrace has within the Android ecosystem until there's a way to use it
>> without having to chase kernel headers independently of kernel images.
>> There are "too many clicks" involved and someone somewhere will drop the
>> ball if it's not glued to the kernel in some way shape or form. Any
>> solution that solves this is one I'd love to hear about.
> 
> I agree. There definitely needs to be a "just collect a damn trace"
> button that works on any device, and for this button to work and
> incorporate eBPF, the system needs to be able to describe itself.

I like that: "the system needs to be able to describe itself". True.

Cheers,

-- 
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Thu, Jan 24, 2019 at 07:57:26PM +0100, Karim Yaghmour wrote:
> 
> On 1/23/19 11:37 PM, Daniel Colascione wrote:
[..]
> > > Personally I advocated a more aggressive approach with Joel in private:
> > > just put the darn headers straight into the kernel image, it's the
> > > *only* artifact we're sure will follow the Android device whatever
> > > happens to it (like built-in ftrace).
> > 
> > I was thinking along similar lines. Ordinarily, we make loadable
> > kernel modules. What we kind of want here is a non-loadable kernel
> > module --- or a non-loadable section in the kernel image proper. I'm
> > not familiar with early-stage kernel loader operation: I know it's
> > possible to crease discardable sections in the kernel image, but can
> > we create sections that are never slurped into memory in the first
> > place? If not, maybe loading and immediately discarding the header
> > section is good enough.
> 
> Interesting. Maybe just append it to the image but have it not loaded and
> have a kernel parameter than enables a "/proc/kheaders" driver to know where
> the fetch the appended headers from storage at runtime. There would be no
> RAM loading whatsoever of the headers, just some sort of
> "kheaders=/dev/foobar:offset:size" parameter. If you turn the option on, you
> get a fatter kernel image size to store on permanent storage, but no impact
> on what's loaded at boot time.

Embedding anything into the kernel image does impact boot time though because
it increase the time spent by bootloader. A module OTOH would not have such
overhead.

Also a kernel can be booted in any number of ways other than mass storage so
it is not a generic Linux-wide solution to have a kheaders= option like that.
If the option is forgotten, then the running system can't use the feature.
The other issue is it requires a kernel command line option / bootloader
changes for that which adds more configuration burden, which not be needed
with a module.

> > Would such a thing really do better than LZMA? LZMA already has very
> > clever techniques for eliminating long-range redundancies in
> > compressible text, including redundancies at the sub-byte level. I can
> > certainly understand the benefit of stripping comments, since removing
> > comments really does decrease the total amount of information the
> > compressor has to preserve, but I'm not sure how much the encoding
> > scheme you propose below would help, since it reminds me of the
> > encoding scheme that LZMA would discover automatically.
> 
> I'm no compression algorithm expert. If you say LZMA would do the
> same/better than what I suggested then I have no reason to contest that. My
> goal is to see the headers as part of the kernel image that's distributed on
> devices so that they don't have to be chased around. I'm just trying to make
> it as palatable as possible.

I believe LZMA is really good at that sort of thing too.

Also at 3.3MB of module size, I think we are really good size-wise. But Dan
is helping look at possibly reducing further if he gets time. Many modules in
my experience are much bigger. amdgpu.ko on my Linux machine is 6.1MB.

I really think making it a module is the best way to make sure this is
bundled with the kernel on the widest number of Android and other Linux
systems, without incurring boot time overhead, or any other command line
configuration burden.

I spoke to so many people at LPC personally with other kernel contributors,
and many folks told me one word - MODULE :D.  Even though I hesitated at
first, now it seems the right solution.

If no one seriously objects, I'll clean this up and post a v2 and with the
RFC tag taken off. Thank you!

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[hpa]

On January 24, 2019 12:59:29 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
>On Thu, Jan 24, 2019 at 07:57:26PM +0100, Karim Yaghmour wrote:
>> 
>> On 1/23/19 11:37 PM, Daniel Colascione wrote:
>[..]
>> > > Personally I advocated a more aggressive approach with Joel in
>private:
>> > > just put the darn headers straight into the kernel image, it's
>the
>> > > *only* artifact we're sure will follow the Android device
>whatever
>> > > happens to it (like built-in ftrace).
>> > 
>> > I was thinking along similar lines. Ordinarily, we make loadable
>> > kernel modules. What we kind of want here is a non-loadable kernel
>> > module --- or a non-loadable section in the kernel image proper.
>I'm
>> > not familiar with early-stage kernel loader operation: I know it's
>> > possible to crease discardable sections in the kernel image, but
>can
>> > we create sections that are never slurped into memory in the first
>> > place? If not, maybe loading and immediately discarding the header
>> > section is good enough.
>> 
>> Interesting. Maybe just append it to the image but have it not loaded
>and
>> have a kernel parameter than enables a "/proc/kheaders" driver to
>know where
>> the fetch the appended headers from storage at runtime. There would
>be no
>> RAM loading whatsoever of the headers, just some sort of
>> "kheaders=/dev/foobar:offset:size" parameter. If you turn the option
>on, you
>> get a fatter kernel image size to store on permanent storage, but no
>impact
>> on what's loaded at boot time.
>
>Embedding anything into the kernel image does impact boot time though
>because
>it increase the time spent by bootloader. A module OTOH would not have
>such
>overhead.
>
>Also a kernel can be booted in any number of ways other than mass
>storage so
>it is not a generic Linux-wide solution to have a kheaders= option like
>that.
>If the option is forgotten, then the running system can't use the
>feature.
>The other issue is it requires a kernel command line option /
>bootloader
>changes for that which adds more configuration burden, which not be
>needed
>with a module.
>
>> > Would such a thing really do better than LZMA? LZMA already has
>very
>> > clever techniques for eliminating long-range redundancies in
>> > compressible text, including redundancies at the sub-byte level. I
>can
>> > certainly understand the benefit of stripping comments, since
>removing
>> > comments really does decrease the total amount of information the
>> > compressor has to preserve, but I'm not sure how much the encoding
>> > scheme you propose below would help, since it reminds me of the
>> > encoding scheme that LZMA would discover automatically.
>> 
>> I'm no compression algorithm expert. If you say LZMA would do the
>> same/better than what I suggested then I have no reason to contest
>that. My
>> goal is to see the headers as part of the kernel image that's
>distributed on
>> devices so that they don't have to be chased around. I'm just trying
>to make
>> it as palatable as possible.
>
>I believe LZMA is really good at that sort of thing too.
>
>Also at 3.3MB of module size, I think we are really good size-wise. But
>Dan
>is helping look at possibly reducing further if he gets time. Many
>modules in
>my experience are much bigger. amdgpu.ko on my Linux machine is 6.1MB.
>
>I really think making it a module is the best way to make sure this is
>bundled with the kernel on the widest number of Android and other Linux
>systems, without incurring boot time overhead, or any other command
>line
>configuration burden.
>
>I spoke to so many people at LPC personally with other kernel
>contributors,
>and many folks told me one word - MODULE :D.  Even though I hesitated
>at
>first, now it seems the right solution.
>
>If no one seriously objects, I'll clean this up and post a v2 and with
>the
>RFC tag taken off. Thank you!
>
> - Joel

So let me throw in a different notion.

A kernel module really is nothing other than a kernel build system artifact stored in the filesystem.

I really don't at any reason whatsoever why this is direct from just producing an archive and putting it in the module directory, except that the latter is far simpler.

I see literally *no* problem, social or technical, you are solvin by actually making it a kernel ELF object.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Fri, Jan 25, 2019 at 11:01 AM <hpa@zytor.com> wrote:
>
> On January 24, 2019 12:59:29 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
> >On Thu, Jan 24, 2019 at 07:57:26PM +0100, Karim Yaghmour wrote:
> >>
> >> On 1/23/19 11:37 PM, Daniel Colascione wrote:
> >[..]
> >> > > Personally I advocated a more aggressive approach with Joel in
> >private:
> >> > > just put the darn headers straight into the kernel image, it's
> >the
> >> > > *only* artifact we're sure will follow the Android device
> >whatever
> >> > > happens to it (like built-in ftrace).
> >> >
> >> > I was thinking along similar lines. Ordinarily, we make loadable
> >> > kernel modules. What we kind of want here is a non-loadable kernel
> >> > module --- or a non-loadable section in the kernel image proper.
> >I'm
> >> > not familiar with early-stage kernel loader operation: I know it's
> >> > possible to crease discardable sections in the kernel image, but
> >can
> >> > we create sections that are never slurped into memory in the first
> >> > place? If not, maybe loading and immediately discarding the header
> >> > section is good enough.
> >>
> >> Interesting. Maybe just append it to the image but have it not loaded
> >and
> >> have a kernel parameter than enables a "/proc/kheaders" driver to
> >know where
> >> the fetch the appended headers from storage at runtime. There would
> >be no
> >> RAM loading whatsoever of the headers, just some sort of
> >> "kheaders=/dev/foobar:offset:size" parameter. If you turn the option
> >on, you
> >> get a fatter kernel image size to store on permanent storage, but no
> >impact
> >> on what's loaded at boot time.
> >
> >Embedding anything into the kernel image does impact boot time though
> >because
> >it increase the time spent by bootloader. A module OTOH would not have
> >such
> >overhead.
> >
> >Also a kernel can be booted in any number of ways other than mass
> >storage so
> >it is not a generic Linux-wide solution to have a kheaders= option like
> >that.
> >If the option is forgotten, then the running system can't use the
> >feature.
> >The other issue is it requires a kernel command line option /
> >bootloader
> >changes for that which adds more configuration burden, which not be
> >needed
> >with a module.
> >
> >> > Would such a thing really do better than LZMA? LZMA already has
> >very
> >> > clever techniques for eliminating long-range redundancies in
> >> > compressible text, including redundancies at the sub-byte level. I
> >can
> >> > certainly understand the benefit of stripping comments, since
> >removing
> >> > comments really does decrease the total amount of information the
> >> > compressor has to preserve, but I'm not sure how much the encoding
> >> > scheme you propose below would help, since it reminds me of the
> >> > encoding scheme that LZMA would discover automatically.
> >>
> >> I'm no compression algorithm expert. If you say LZMA would do the
> >> same/better than what I suggested then I have no reason to contest
> >that. My
> >> goal is to see the headers as part of the kernel image that's
> >distributed on
> >> devices so that they don't have to be chased around. I'm just trying
> >to make
> >> it as palatable as possible.
> >
> >I believe LZMA is really good at that sort of thing too.
> >
> >Also at 3.3MB of module size, I think we are really good size-wise. But
> >Dan
> >is helping look at possibly reducing further if he gets time. Many
> >modules in
> >my experience are much bigger. amdgpu.ko on my Linux machine is 6.1MB.
> >
> >I really think making it a module is the best way to make sure this is
> >bundled with the kernel on the widest number of Android and other Linux
> >systems, without incurring boot time overhead, or any other command
> >line
> >configuration burden.
> >
> >I spoke to so many people at LPC personally with other kernel
> >contributors,
> >and many folks told me one word - MODULE :D.  Even though I hesitated
> >at
> >first, now it seems the right solution.
> >
> >If no one seriously objects, I'll clean this up and post a v2 and with
> >the
> >RFC tag taken off. Thank you!
> >
> > - Joel
>
> So let me throw in a different notion.
>
> A kernel module really is nothing other than a kernel build system artifact stored in the filesystem.
>
> I really don't at any reason whatsoever why this is direct from just producing an archive and putting it in the module directory, except that the latter is far simpler.
>
> I see literally *no* problem, social or technical, you are solvin by actually making it a kernel ELF object.

Joel does have a point. Suppose we're on Android and we're testing a
random local kernel we've built. We can command the device to boot
into the bootloader, then send our locally-built kernel to the device
with "fastboot boot mykernelz". Having booted the device this way,
there's no on-disk artifact corresponding to mykernelz: we just sent
the boot kernel directly to the device's memory. Now, suppose I want
to attach DCTV or some other fancy ftrace-based analysis tool to the
device in order to study how mykernelz acts in some scenario I care
about. With Joel's approach, DCTV would be able to grab the kernel
headers from the running kernel, compile whatever kprobe or BPF
incantations needed, and have everything Just Work. If we put the
headers only on disk without any way to retrieve them at runtime, we'd
need a different path for kernel self-description in the case where a
running kernel doesn't have an on-disk representation, and that adds a
lot of complexity for everyone everywhere. By providing
guaranteed-correct kernel headers via some runtime interface, we make
a lot of things Just Work, and that has value.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[hpa]

On January 25, 2019 11:15:52 AM PST, Daniel Colascione <dancol@google.com> wrote:
>On Fri, Jan 25, 2019 at 11:01 AM <hpa@zytor.com> wrote:
>>
>> On January 24, 2019 12:59:29 PM PST, Joel Fernandes
><joel@joelfernandes.org> wrote:
>> >On Thu, Jan 24, 2019 at 07:57:26PM +0100, Karim Yaghmour wrote:
>> >>
>> >> On 1/23/19 11:37 PM, Daniel Colascione wrote:
>> >[..]
>> >> > > Personally I advocated a more aggressive approach with Joel in
>> >private:
>> >> > > just put the darn headers straight into the kernel image, it's
>> >the
>> >> > > *only* artifact we're sure will follow the Android device
>> >whatever
>> >> > > happens to it (like built-in ftrace).
>> >> >
>> >> > I was thinking along similar lines. Ordinarily, we make loadable
>> >> > kernel modules. What we kind of want here is a non-loadable
>kernel
>> >> > module --- or a non-loadable section in the kernel image proper.
>> >I'm
>> >> > not familiar with early-stage kernel loader operation: I know
>it's
>> >> > possible to crease discardable sections in the kernel image, but
>> >can
>> >> > we create sections that are never slurped into memory in the
>first
>> >> > place? If not, maybe loading and immediately discarding the
>header
>> >> > section is good enough.
>> >>
>> >> Interesting. Maybe just append it to the image but have it not
>loaded
>> >and
>> >> have a kernel parameter than enables a "/proc/kheaders" driver to
>> >know where
>> >> the fetch the appended headers from storage at runtime. There
>would
>> >be no
>> >> RAM loading whatsoever of the headers, just some sort of
>> >> "kheaders=/dev/foobar:offset:size" parameter. If you turn the
>option
>> >on, you
>> >> get a fatter kernel image size to store on permanent storage, but
>no
>> >impact
>> >> on what's loaded at boot time.
>> >
>> >Embedding anything into the kernel image does impact boot time
>though
>> >because
>> >it increase the time spent by bootloader. A module OTOH would not
>have
>> >such
>> >overhead.
>> >
>> >Also a kernel can be booted in any number of ways other than mass
>> >storage so
>> >it is not a generic Linux-wide solution to have a kheaders= option
>like
>> >that.
>> >If the option is forgotten, then the running system can't use the
>> >feature.
>> >The other issue is it requires a kernel command line option /
>> >bootloader
>> >changes for that which adds more configuration burden, which not be
>> >needed
>> >with a module.
>> >
>> >> > Would such a thing really do better than LZMA? LZMA already has
>> >very
>> >> > clever techniques for eliminating long-range redundancies in
>> >> > compressible text, including redundancies at the sub-byte level.
>I
>> >can
>> >> > certainly understand the benefit of stripping comments, since
>> >removing
>> >> > comments really does decrease the total amount of information
>the
>> >> > compressor has to preserve, but I'm not sure how much the
>encoding
>> >> > scheme you propose below would help, since it reminds me of the
>> >> > encoding scheme that LZMA would discover automatically.
>> >>
>> >> I'm no compression algorithm expert. If you say LZMA would do the
>> >> same/better than what I suggested then I have no reason to contest
>> >that. My
>> >> goal is to see the headers as part of the kernel image that's
>> >distributed on
>> >> devices so that they don't have to be chased around. I'm just
>trying
>> >to make
>> >> it as palatable as possible.
>> >
>> >I believe LZMA is really good at that sort of thing too.
>> >
>> >Also at 3.3MB of module size, I think we are really good size-wise.
>But
>> >Dan
>> >is helping look at possibly reducing further if he gets time. Many
>> >modules in
>> >my experience are much bigger. amdgpu.ko on my Linux machine is
>6.1MB.
>> >
>> >I really think making it a module is the best way to make sure this
>is
>> >bundled with the kernel on the widest number of Android and other
>Linux
>> >systems, without incurring boot time overhead, or any other command
>> >line
>> >configuration burden.
>> >
>> >I spoke to so many people at LPC personally with other kernel
>> >contributors,
>> >and many folks told me one word - MODULE :D.  Even though I
>hesitated
>> >at
>> >first, now it seems the right solution.
>> >
>> >If no one seriously objects, I'll clean this up and post a v2 and
>with
>> >the
>> >RFC tag taken off. Thank you!
>> >
>> > - Joel
>>
>> So let me throw in a different notion.
>>
>> A kernel module really is nothing other than a kernel build system
>artifact stored in the filesystem.
>>
>> I really don't at any reason whatsoever why this is direct from just
>producing an archive and putting it in the module directory, except
>that the latter is far simpler.
>>
>> I see literally *no* problem, social or technical, you are solvin by
>actually making it a kernel ELF object.
>
>Joel does have a point. Suppose we're on Android and we're testing a
>random local kernel we've built. We can command the device to boot
>into the bootloader, then send our locally-built kernel to the device
>with "fastboot boot mykernelz". Having booted the device this way,
>there's no on-disk artifact corresponding to mykernelz: we just sent
>the boot kernel directly to the device's memory. Now, suppose I want
>to attach DCTV or some other fancy ftrace-based analysis tool to the
>device in order to study how mykernelz acts in some scenario I care
>about. With Joel's approach, DCTV would be able to grab the kernel
>headers from the running kernel, compile whatever kprobe or BPF
>incantations needed, and have everything Just Work. If we put the
>headers only on disk without any way to retrieve them at runtime, we'd
>need a different path for kernel self-description in the case where a
>running kernel doesn't have an on-disk representation, and that adds a
>lot of complexity for everyone everywhere. By providing
>guaranteed-correct kernel headers via some runtime interface, we make
>a lot of things Just Work, and that has value.

Joel specifically is talking about using a module, which *does* have to be in the filesystem.

You can't have it both ways, unfortunately.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Fri, Jan 25, 2019 at 12:28 PM <hpa@zytor.com> wrote:
>
> On January 25, 2019 11:15:52 AM PST, Daniel Colascione <dancol@google.com> wrote:
> >On Fri, Jan 25, 2019 at 11:01 AM <hpa@zytor.com> wrote:
> >>
> >> On January 24, 2019 12:59:29 PM PST, Joel Fernandes
> ><joel@joelfernandes.org> wrote:
> >> >On Thu, Jan 24, 2019 at 07:57:26PM +0100, Karim Yaghmour wrote:
> >> >>
> >> >> On 1/23/19 11:37 PM, Daniel Colascione wrote:
> >> >[..]
> >> >> > > Personally I advocated a more aggressive approach with Joel in
> >> >private:
> >> >> > > just put the darn headers straight into the kernel image, it's
> >> >the
> >> >> > > *only* artifact we're sure will follow the Android device
> >> >whatever
> >> >> > > happens to it (like built-in ftrace).
> >> >> >
> >> >> > I was thinking along similar lines. Ordinarily, we make loadable
> >> >> > kernel modules. What we kind of want here is a non-loadable
> >kernel
> >> >> > module --- or a non-loadable section in the kernel image proper.
> >> >I'm
> >> >> > not familiar with early-stage kernel loader operation: I know
> >it's
> >> >> > possible to crease discardable sections in the kernel image, but
> >> >can
> >> >> > we create sections that are never slurped into memory in the
> >first
> >> >> > place? If not, maybe loading and immediately discarding the
> >header
> >> >> > section is good enough.
> >> >>
> >> >> Interesting. Maybe just append it to the image but have it not
> >loaded
> >> >and
> >> >> have a kernel parameter than enables a "/proc/kheaders" driver to
> >> >know where
> >> >> the fetch the appended headers from storage at runtime. There
> >would
> >> >be no
> >> >> RAM loading whatsoever of the headers, just some sort of
> >> >> "kheaders=/dev/foobar:offset:size" parameter. If you turn the
> >option
> >> >on, you
> >> >> get a fatter kernel image size to store on permanent storage, but
> >no
> >> >impact
> >> >> on what's loaded at boot time.
> >> >
> >> >Embedding anything into the kernel image does impact boot time
> >though
> >> >because
> >> >it increase the time spent by bootloader. A module OTOH would not
> >have
> >> >such
> >> >overhead.
> >> >
> >> >Also a kernel can be booted in any number of ways other than mass
> >> >storage so
> >> >it is not a generic Linux-wide solution to have a kheaders= option
> >like
> >> >that.
> >> >If the option is forgotten, then the running system can't use the
> >> >feature.
> >> >The other issue is it requires a kernel command line option /
> >> >bootloader
> >> >changes for that which adds more configuration burden, which not be
> >> >needed
> >> >with a module.
> >> >
> >> >> > Would such a thing really do better than LZMA? LZMA already has
> >> >very
> >> >> > clever techniques for eliminating long-range redundancies in
> >> >> > compressible text, including redundancies at the sub-byte level.
> >I
> >> >can
> >> >> > certainly understand the benefit of stripping comments, since
> >> >removing
> >> >> > comments really does decrease the total amount of information
> >the
> >> >> > compressor has to preserve, but I'm not sure how much the
> >encoding
> >> >> > scheme you propose below would help, since it reminds me of the
> >> >> > encoding scheme that LZMA would discover automatically.
> >> >>
> >> >> I'm no compression algorithm expert. If you say LZMA would do the
> >> >> same/better than what I suggested then I have no reason to contest
> >> >that. My
> >> >> goal is to see the headers as part of the kernel image that's
> >> >distributed on
> >> >> devices so that they don't have to be chased around. I'm just
> >trying
> >> >to make
> >> >> it as palatable as possible.
> >> >
> >> >I believe LZMA is really good at that sort of thing too.
> >> >
> >> >Also at 3.3MB of module size, I think we are really good size-wise.
> >But
> >> >Dan
> >> >is helping look at possibly reducing further if he gets time. Many
> >> >modules in
> >> >my experience are much bigger. amdgpu.ko on my Linux machine is
> >6.1MB.
> >> >
> >> >I really think making it a module is the best way to make sure this
> >is
> >> >bundled with the kernel on the widest number of Android and other
> >Linux
> >> >systems, without incurring boot time overhead, or any other command
> >> >line
> >> >configuration burden.
> >> >
> >> >I spoke to so many people at LPC personally with other kernel
> >> >contributors,
> >> >and many folks told me one word - MODULE :D.  Even though I
> >hesitated
> >> >at
> >> >first, now it seems the right solution.
> >> >
> >> >If no one seriously objects, I'll clean this up and post a v2 and
> >with
> >> >the
> >> >RFC tag taken off. Thank you!
> >> >
> >> > - Joel
> >>
> >> So let me throw in a different notion.
> >>
> >> A kernel module really is nothing other than a kernel build system
> >artifact stored in the filesystem.
> >>
> >> I really don't at any reason whatsoever why this is direct from just
> >producing an archive and putting it in the module directory, except
> >that the latter is far simpler.
> >>
> >> I see literally *no* problem, social or technical, you are solvin by
> >actually making it a kernel ELF object.
> >
> >Joel does have a point. Suppose we're on Android and we're testing a
> >random local kernel we've built. We can command the device to boot
> >into the bootloader, then send our locally-built kernel to the device
> >with "fastboot boot mykernelz". Having booted the device this way,
> >there's no on-disk artifact corresponding to mykernelz: we just sent
> >the boot kernel directly to the device's memory. Now, suppose I want
> >to attach DCTV or some other fancy ftrace-based analysis tool to the
> >device in order to study how mykernelz acts in some scenario I care
> >about. With Joel's approach, DCTV would be able to grab the kernel
> >headers from the running kernel, compile whatever kprobe or BPF
> >incantations needed, and have everything Just Work. If we put the
> >headers only on disk without any way to retrieve them at runtime, we'd
> >need a different path for kernel self-description in the case where a
> >running kernel doesn't have an on-disk representation, and that adds a
> >lot of complexity for everyone everywhere. By providing
> >guaranteed-correct kernel headers via some runtime interface, we make
> >a lot of things Just Work, and that has value.
>
> Joel specifically is talking about using a module, which *does* have to be in the filesystem.
>
> You can't have it both ways, unfortunately.

In general, whatever we support in module form, we also support as
part of the kernel image itself.

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Fri, Jan 25, 2019 at 12:34:52PM -0800, Daniel Colascione wrote:
> On Fri, Jan 25, 2019 at 12:28 PM <hpa@zytor.com> wrote:
> >
> > On January 25, 2019 11:15:52 AM PST, Daniel Colascione <dancol@google.com> wrote:
> > >On Fri, Jan 25, 2019 at 11:01 AM <hpa@zytor.com> wrote:
> > >>
> > >> On January 24, 2019 12:59:29 PM PST, Joel Fernandes
> > ><joel@joelfernandes.org> wrote:
> > >> >On Thu, Jan 24, 2019 at 07:57:26PM +0100, Karim Yaghmour wrote:
> > >> >>
> > >> >> On 1/23/19 11:37 PM, Daniel Colascione wrote:
> > >> >[..]
> > >> >> > > Personally I advocated a more aggressive approach with Joel in
> > >> >private:
> > >> >> > > just put the darn headers straight into the kernel image, it's
> > >> >the
> > >> >> > > *only* artifact we're sure will follow the Android device
> > >> >whatever
> > >> >> > > happens to it (like built-in ftrace).
> > >> >> >
> > >> >> > I was thinking along similar lines. Ordinarily, we make loadable
> > >> >> > kernel modules. What we kind of want here is a non-loadable
> > >kernel
> > >> >> > module --- or a non-loadable section in the kernel image proper.
> > >> >I'm
> > >> >> > not familiar with early-stage kernel loader operation: I know
> > >it's
> > >> >> > possible to crease discardable sections in the kernel image, but
> > >> >can
> > >> >> > we create sections that are never slurped into memory in the
> > >first
> > >> >> > place? If not, maybe loading and immediately discarding the
> > >header
> > >> >> > section is good enough.
> > >> >>
> > >> >> Interesting. Maybe just append it to the image but have it not
> > >loaded
> > >> >and
> > >> >> have a kernel parameter than enables a "/proc/kheaders" driver to
> > >> >know where
> > >> >> the fetch the appended headers from storage at runtime. There
> > >would
> > >> >be no
> > >> >> RAM loading whatsoever of the headers, just some sort of
> > >> >> "kheaders=/dev/foobar:offset:size" parameter. If you turn the
> > >option
> > >> >on, you
> > >> >> get a fatter kernel image size to store on permanent storage, but
> > >no
> > >> >impact
> > >> >> on what's loaded at boot time.
> > >> >
> > >> >Embedding anything into the kernel image does impact boot time
> > >though
> > >> >because
> > >> >it increase the time spent by bootloader. A module OTOH would not
> > >have
> > >> >such
> > >> >overhead.
> > >> >
> > >> >Also a kernel can be booted in any number of ways other than mass
> > >> >storage so
> > >> >it is not a generic Linux-wide solution to have a kheaders= option
> > >like
> > >> >that.
> > >> >If the option is forgotten, then the running system can't use the
> > >> >feature.
> > >> >The other issue is it requires a kernel command line option /
> > >> >bootloader
> > >> >changes for that which adds more configuration burden, which not be
> > >> >needed
> > >> >with a module.
> > >> >
> > >> >> > Would such a thing really do better than LZMA? LZMA already has
> > >> >very
> > >> >> > clever techniques for eliminating long-range redundancies in
> > >> >> > compressible text, including redundancies at the sub-byte level.
> > >I
> > >> >can
> > >> >> > certainly understand the benefit of stripping comments, since
> > >> >removing
> > >> >> > comments really does decrease the total amount of information
> > >the
> > >> >> > compressor has to preserve, but I'm not sure how much the
> > >encoding
> > >> >> > scheme you propose below would help, since it reminds me of the
> > >> >> > encoding scheme that LZMA would discover automatically.
> > >> >>
> > >> >> I'm no compression algorithm expert. If you say LZMA would do the
> > >> >> same/better than what I suggested then I have no reason to contest
> > >> >that. My
> > >> >> goal is to see the headers as part of the kernel image that's
> > >> >distributed on
> > >> >> devices so that they don't have to be chased around. I'm just
> > >trying
> > >> >to make
> > >> >> it as palatable as possible.
> > >> >
> > >> >I believe LZMA is really good at that sort of thing too.
> > >> >
> > >> >Also at 3.3MB of module size, I think we are really good size-wise.
> > >But
> > >> >Dan
> > >> >is helping look at possibly reducing further if he gets time. Many
> > >> >modules in
> > >> >my experience are much bigger. amdgpu.ko on my Linux machine is
> > >6.1MB.
> > >> >
> > >> >I really think making it a module is the best way to make sure this
> > >is
> > >> >bundled with the kernel on the widest number of Android and other
> > >Linux
> > >> >systems, without incurring boot time overhead, or any other command
> > >> >line
> > >> >configuration burden.
> > >> >
> > >> >I spoke to so many people at LPC personally with other kernel
> > >> >contributors,
> > >> >and many folks told me one word - MODULE :D.  Even though I
> > >hesitated
> > >> >at
> > >> >first, now it seems the right solution.
> > >> >
> > >> >If no one seriously objects, I'll clean this up and post a v2 and
> > >with
> > >> >the
> > >> >RFC tag taken off. Thank you!
> > >> >
> > >> > - Joel
> > >>
> > >> So let me throw in a different notion.
> > >>
> > >> A kernel module really is nothing other than a kernel build system
> > >artifact stored in the filesystem.
> > >>
> > >> I really don't at any reason whatsoever why this is direct from just
> > >producing an archive and putting it in the module directory, except
> > >that the latter is far simpler.
> > >>
> > >> I see literally *no* problem, social or technical, you are solvin by
> > >actually making it a kernel ELF object.
> > >
> > >Joel does have a point. Suppose we're on Android and we're testing a
> > >random local kernel we've built. We can command the device to boot
> > >into the bootloader, then send our locally-built kernel to the device
> > >with "fastboot boot mykernelz". Having booted the device this way,
> > >there's no on-disk artifact corresponding to mykernelz: we just sent
> > >the boot kernel directly to the device's memory. Now, suppose I want
> > >to attach DCTV or some other fancy ftrace-based analysis tool to the
> > >device in order to study how mykernelz acts in some scenario I care
> > >about. With Joel's approach, DCTV would be able to grab the kernel
> > >headers from the running kernel, compile whatever kprobe or BPF
> > >incantations needed, and have everything Just Work. If we put the
> > >headers only on disk without any way to retrieve them at runtime, we'd
> > >need a different path for kernel self-description in the case where a
> > >running kernel doesn't have an on-disk representation, and that adds a
> > >lot of complexity for everyone everywhere. By providing
> > >guaranteed-correct kernel headers via some runtime interface, we make
> > >a lot of things Just Work, and that has value.
> >
> > Joel specifically is talking about using a module, which *does* have to be in the filesystem.
> >
> > You can't have it both ways, unfortunately.

You can have it whichever way is convenient to you, do all Linux users have
the same CONFIG options?

For me its module, for Dan its built-in. So what's your point? :)

> In general, whatever we support in module form, we also support as
> part of the kernel image itself.

Exactly.

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Fri, Jan 25, 2019 at 11:00:25AM -0800, hpa@zytor.com wrote:
> On January 24, 2019 12:59:29 PM PST, Joel Fernandes <joel@joelfernandes.org> wrote:
> >On Thu, Jan 24, 2019 at 07:57:26PM +0100, Karim Yaghmour wrote:
> >> 
> >> On 1/23/19 11:37 PM, Daniel Colascione wrote:
> >[..]
> >> > > Personally I advocated a more aggressive approach with Joel in
> >private:
> >> > > just put the darn headers straight into the kernel image, it's
> >the
> >> > > *only* artifact we're sure will follow the Android device
> >whatever
> >> > > happens to it (like built-in ftrace).
> >> > 
> >> > I was thinking along similar lines. Ordinarily, we make loadable
> >> > kernel modules. What we kind of want here is a non-loadable kernel
> >> > module --- or a non-loadable section in the kernel image proper.
> >I'm
> >> > not familiar with early-stage kernel loader operation: I know it's
> >> > possible to crease discardable sections in the kernel image, but
> >can
> >> > we create sections that are never slurped into memory in the first
> >> > place? If not, maybe loading and immediately discarding the header
> >> > section is good enough.
> >> 
> >> Interesting. Maybe just append it to the image but have it not loaded
> >and
> >> have a kernel parameter than enables a "/proc/kheaders" driver to
> >know where
> >> the fetch the appended headers from storage at runtime. There would
> >be no
> >> RAM loading whatsoever of the headers, just some sort of
> >> "kheaders=/dev/foobar:offset:size" parameter. If you turn the option
> >on, you
> >> get a fatter kernel image size to store on permanent storage, but no
> >impact
> >> on what's loaded at boot time.
> >
> >Embedding anything into the kernel image does impact boot time though
> >because
> >it increase the time spent by bootloader. A module OTOH would not have
> >such
> >overhead.
> >
> >Also a kernel can be booted in any number of ways other than mass
> >storage so
> >it is not a generic Linux-wide solution to have a kheaders= option like
> >that.
> >If the option is forgotten, then the running system can't use the
> >feature.
> >The other issue is it requires a kernel command line option /
> >bootloader
> >changes for that which adds more configuration burden, which not be
> >needed
> >with a module.
> >
> >> > Would such a thing really do better than LZMA? LZMA already has
> >very
> >> > clever techniques for eliminating long-range redundancies in
> >> > compressible text, including redundancies at the sub-byte level. I
> >can
> >> > certainly understand the benefit of stripping comments, since
> >removing
> >> > comments really does decrease the total amount of information the
> >> > compressor has to preserve, but I'm not sure how much the encoding
> >> > scheme you propose below would help, since it reminds me of the
> >> > encoding scheme that LZMA would discover automatically.
> >> 
> >> I'm no compression algorithm expert. If you say LZMA would do the
> >> same/better than what I suggested then I have no reason to contest
> >that. My
> >> goal is to see the headers as part of the kernel image that's
> >distributed on
> >> devices so that they don't have to be chased around. I'm just trying
> >to make
> >> it as palatable as possible.
> >
> >I believe LZMA is really good at that sort of thing too.
> >
> >Also at 3.3MB of module size, I think we are really good size-wise. But
> >Dan
> >is helping look at possibly reducing further if he gets time. Many
> >modules in
> >my experience are much bigger. amdgpu.ko on my Linux machine is 6.1MB.
> >
> >I really think making it a module is the best way to make sure this is
> >bundled with the kernel on the widest number of Android and other Linux
> >systems, without incurring boot time overhead, or any other command
> >line
> >configuration burden.
> >
> >I spoke to so many people at LPC personally with other kernel
> >contributors,
> >and many folks told me one word - MODULE :D.  Even though I hesitated
> >at
> >first, now it seems the right solution.
> >
> >If no one seriously objects, I'll clean this up and post a v2 and with
> >the
> >RFC tag taken off. Thank you!
> >
> > - Joel
> 
> So let me throw in a different notion.
> 
> A kernel module really is nothing other than a kernel build system artifact stored in the filesystem.
> 
> I really don't at any reason whatsoever why this is direct from just producing an archive and putting it in the module directory, except that the latter is far simpler.
> 
> I see literally *no* problem, social or technical, you are solvin by actually making it a kernel ELF object.

I think you may not be seeing the point being made here, Hans. A module can
be in any number of locations. The work to do the search of the modules is
already done by modprobe. Diverse userspaces already know how to look for and
load modules - whether its Android or some other user space. Modules are
universal and integrated well within the kernel's ecosystem even though the
Linux systems are so diverse.

Also, as I was saying before - there is an option to build it into the kernel
too with this proposal, if you hate modules so much.

This is exactly why /proc/config.gz is so popular and used widely by people
debugging the kernel. Same with /proc/kallsysms. So if we had it your way,
then you might as well make those non-ELF artifacts right?

You can look at it as a convenince option. Making things more convenient for
users is solving problems, I'm sorry you don't see it that way.

thanks,

 - Joel

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1117 bytes --]

On Fri 2019-01-18 17:55:43, Joel Fernandes wrote:
> From: "Joel Fernandes (Google)" <joel@joelfernandes.org>
> 
> Introduce in-kernel headers and other artifacts which are made available
> as an archive through proc (/proc/kheaders.tgz file). This archive makes
> it possible to build kernel modules, run eBPF programs, and other
> tracing programs that need to extend the kernel for tracing purposes
> without any dependency on the file system having headers and build
> artifacts.
> 
> On Android and embedded systems, it is common to switch kernels but not
> have kernel headers available on the file system. Raw kernel headers
> also cannot be copied into the filesystem like they can be on other
> distros, due to licensing and other issues. There's no linux-headers

If your licensing prevents you from having headers on the
filesystem... then I guess you should fix the licensing.

I agree with Christoph, this looks pretty horrible.
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[H. Peter Anvin]

On 3/6/19 3:09 PM, Pavel Machek wrote:
> On Fri 2019-01-18 17:55:43, Joel Fernandes wrote:
>> From: "Joel Fernandes (Google)" <joel@joelfernandes.org>
>>
>> Introduce in-kernel headers and other artifacts which are made available
>> as an archive through proc (/proc/kheaders.tgz file). This archive makes
>> it possible to build kernel modules, run eBPF programs, and other
>> tracing programs that need to extend the kernel for tracing purposes
>> without any dependency on the file system having headers and build
>> artifacts.
>>
>> On Android and embedded systems, it is common to switch kernels but not
>> have kernel headers available on the file system. Raw kernel headers
>> also cannot be copied into the filesystem like they can be on other
>> distros, due to licensing and other issues. There's no linux-headers
> 
> If your licensing prevents you from having headers on the
> filesystem... then I guess you should fix the licensing.
> 
> I agree with Christoph, this looks pretty horrible.
> 									Pavel
> 

The argument that "it can be a module" is basically an admission of
failure - if it isn't part of the kernel image itself there is no
benefit over where the modules are stored, which will be *somewhere* in
the filesystem.

What I *do* think makes sense is to create an archive with this
information and stuff it in the same place as the modules. It reduces
the amount it is possible to muck it up.

	-hpa

Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Norbert Lange]

I have a similar problem, which is caused by an attempt to separate
the kernel installation
from the rootfs. updates of the kernel should not affect the
(read-only) rootfs or initramfs.
For technical reasons I am unable to built all modules static.

-   have multiple kernels #K and rootfs installations #R without needing
    #R times #N module subdirectories. (think of fallback kernels/installations)
-   be able to use readonly rootfs, and need to regenerate it for new kernels.
-   ideally be as space efficient as possible

I ended up embedding an initramfs in the kernel, but this has the
annoying drawback of disabling the builtin init of the kernel so I
needed to further add busybox and shellscripts just to get it to parse
the cmdline and boot.
I used a compressed squashfs for the kernel modules, further requiring
a loop device.

So the point I am trying to make, that there could be a better
approach for my problem,
like having the kernel expose a blockdevice in /proc/bundle.img (no
need for a loopdevice).
Maybe some more generic approach would cover both this issue as well
as the problem with kernel headers on android. The /proc/kheaders.tgz
file is very specific.

(Well, I would be happy if there is a way to keep the kernels builtin
init enabled, even if a custon initramfs is builtin)