Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Karim Yaghmour <karim.yaghmour@opersys.com>]

On 1/22/19 2:39 PM, Joel Fernandes wrote:
[snip]
> On Sun, Jan 20, 2019 at 06:49:56PM -0800, hpa@zytor.com wrote:
[snip]
>> My point is that if we're going to actually solve a problem, we need to make it so that the distro won't just disable it anyway, and it ought to be something scalable; otherwise nothing is gained.
> 
> Agreed. Although there are many ways distros can misconfigure a kernel and
> shoot themselves in the foot.
> 
>> I am *not* disagreeing with the problem statement!
> 
> Ok, that's great to know. Thanks for all the discussions!
> 
>> Now, /proc isn't something that will autoload modules. A filesystem *will*, although you need to be able to mount it; furthermore, it makes it trivially to extend it (and the firmware interface provides an . easy way to feed the data to such a filesystem without having to muck with anything magic.)
> 
> My thought is for the tools needing the module to try to load them, if they
> need the headers. And then unload the module once they are done with it.
> 
>> Heck, we could even make it a squashfs image that can just be mounted.
> 
> That would add another dependency on a kernel configuration though.
> 
>> So, first of all, where does Android keep its modules, and what is actually included? Is /sbin/modprobe used to load the modules, as is normal? We might even be able to address this with some fairly trivial enhancements to modprobe; specifically to search in the module paths for something that isn't a module per se.
>> The best scenario would be if we could simply have the tools find the location equivalent of /lib/modules/$version/source...
> 
> Sandeep answered this in the other thread, modprobe is there in Android.
> About storing sources on the filesystem, I already went down this route (this
> was my first effort) and there is quite some resistance to ship sources on
> the filesystem of an Android device due to Licensing reasons. A kernel module
> though is something that's a binary artifact and is not something
> "distributed" by Google.
> 
> The other issue is with enforcing different OEM vendors that they should
> store kernel-header sources on their Android products. That's much harder to
> enforce, however we can easily enforce kernel config options be enabled,
> since we already enforce a kernel configuration options that are needed for
> Android to work correctly:
> https://source.android.com/devices/architecture/kernel/config. So this config
> option can just be another one.
> 
> By the way, we can easily write a script to just extract the .ko directly -
> if the whole "load it as a module" thing bothers you. The kheaders.ko can
> just be thought of as a tarball. There's already a script to extract
> /proc/config.gz in the same/similar way: scripts/extract-ikconfig

If I may add a few more thoughts here ... in no specific order:

I've been helping customers put Android in all sorts of weird devices 
over the past decade, handsets included, and have had a front-row seat 
to how this ecosystem works/has-evolved (Google, SoC vendors, 
manufacturers, etc.) One thing that stands out is that while the outside 
perception might be that Google has a stronghold on what "partners" do, 
the reality is that almost any rule that can be broken will be broken by 
someone in the ecosystem. And yet, still, Google somehow has to manage 
and provide app developers and, ultimately, consumers with a coherent 
experience, while still herding everyone to more-or-less go in the same 
direction ...

 From that point of view, if something comes from or is rooted in 
mainline, instead of being mandated, it's usually easier to find it 
across the board. A perfect example of this is ftrace. The fact that 
it's in mainline has enabled google to: a) instrument their entire stack 
to log events to it (see 
http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/82 
and 
http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/83), 
and b) provide app-developer-facing tools (see 
https://developer.android.com/studio/command-line/systrace). Since this 
tracing functionality is now integrated into Android Studio (look for 
"System Trace" here: 
https://developer.android.com/studio/profile/cpu-profiler), it's very 
much "standard android" and additional proof, if any was needed, that 
tracing is useful to everyone.

A few years back I was asked by a customer to put together some class 
material for internal Android debugging and performance analysis 
(commercial disclaimer applies, but slides/exercises are under 
"courseware": 
http://www.opersys.com/training/android-debug-and-performance). ftrace 
was very much in those early versions and it was great to show people 
that they could use it "out of the box". Recently I wanted to update 
this class material to cover eBPF and its applicability in Android. Holy 
cow. That turned out to be less obvious than necessary and somewhat 
peculiar to pull off. In this specific case, Joel tried a few things 
(see 
http://www.opersys.com/downloads/cc-slides/android-debug/slides-main-181012.html#/111) 
before eventually settling on loading a Debian chroot jail into a live 
Android (https://github.com/joelagnel/adeb) ... all of which require a 
proper set of kernel headers to properly function. Don't get me wrong, 
Joel's Androdeb makes this easy, but it's still outside the standard 
Android MO.

In short, let's just say that, contrary to ftrace, I don't see the path 
for eBPF to being part of the standard toolset used by app developers 
any time soon. The recently-released bpftrace might help in that regard, 
but the kernel headers aren't too far in that regard either.

Personally I advocated a more aggressive approach with Joel in private: 
just put the darn headers straight into the kernel image, it's the 
*only* artifact we're sure will follow the Android device whatever 
happens to it (like built-in ftrace). To that end, I even had some crazy 
ideas on how to compress the headers even further than with std 
compression algorithms -- here's a snippet from an email I sent Joel 
some time back detailing such a hack:
> Since C headers have fairly constrained semantics and since the types of semantics generally used to name structs, etc. in the Linux kernel are well established, we can likely devise a very customized compression algorithm for the purpose.
> 
> We know, for example, that there are quite a few ascii characters that can't be found in a normal header. So we can play some games. Say, replace all the "#" dictionary with 2 bytes, the first being a magic (illegal) ascii character and the 2nd with a constant. As such, we could replace all #foo entries with 2 bytes:
> #define goes from 7 bytes to 2
> #include from 8 to 2
> #ifdef from 6 to 2
> etc.
> 
> The same goes for several keywords of the C language and words very often found in Linux kernel headers:
> struct from 6 to 2
> linux from 5 to 2
> CONFIG from 6 to 2
> "unsigned long" from 13 to 2
> etc.
> 
> And if we run out of space with one magic character (256), we can start using a next one.
> 
> We'd have a script that runs at build time that does this replacement, and we could even package the counter-script as part of the kernel image the to convert the "cooked" headers back to their original format; ex: Steven has a README in the ftrace directory.
> 
> Such a replacement strategy combined with stripping comments and gzipping the result may provide some very interesting compression results. 

Whether such craziness makes sense or is adopted or not isn't mine to 
chart, but I certainly can't see eBPF reaching the same mass deployment 
ftrace has within the Android ecosystem until there's a way to use it 
without having to chase kernel headers independently of kernel images. 
There are "too many clicks" involved and someone somewhere will drop the 
ball if it's not glued to the kernel in some way shape or form. Any 
solution that solves this is one I'd love to hear about.

My $0.02

-- 
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour