Re: [PATCH 0/3] build linux-next without perl

Re: [PATCH 0/3] build linux-next without perl

[John Spencer]

Rob Landley wrote:
>  Before 2.6.25 building Linux never required perl. This patch series 
removes
>  the requirement from basic kernel builds (tested on i686, x86_64, arm, 
mips,
>  powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1.

using this patch series since 2.6.32 in sabotage linux, it is highly 
useful and it's about time to get it merged.
tested against linux 3.8, works perfectly.

Acked-By: John Spencer <maillist-linux@barfooze.de>

Re: [PATCH 0/3] build linux-next without perl

[Sam Ravnborg]

On Wed, Feb 27, 2013 at 05:55:31PM +0100, John Spencer wrote:
> Rob Landley wrote:
>>  Before 2.6.25 building Linux never required perl. This patch series 
> removes
>>  the requirement from basic kernel builds (tested on i686, x86_64, arm, 
>> 
> mips,
>>  powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1.
>
> using this patch series since 2.6.32 in sabotage linux, it is highly  
> useful and it's about time to get it merged.
> tested against linux 3.8, works perfectly.

Why do you apply this patch serie to sabotage linux?

	Sam

Re: [PATCH 0/3] build linux-next without perl

[John Spencer]

On 02/27/2013 07:53 PM, Sam Ravnborg wrote:
> On Wed, Feb 27, 2013 at 05:55:31PM +0100, John Spencer wrote:
>> Rob Landley wrote:
>>>   Before 2.6.25 building Linux never required perl. This patch series
>> removes
>>>   the requirement from basic kernel builds (tested on i686, x86_64, arm,
>>>
>> mips,
>>>   powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1.
>>
>> using this patch series since 2.6.32 in sabotage linux, it is highly
>> useful and it's about time to get it merged.
>> tested against linux 3.8, works perfectly.
>
> Why do you apply this patch serie to sabotage linux?
>
> 	Sam
>

it allows me to build the kernel as soon as the minimal base system is 
bootstrapped. building perl takes about as long as the whole base system 
(busybox,gcc,binutils,make) together, so having that dependency removed 
is a huge relief.

btw, bootstrap-linux [1] uses this patch series as well.

[1] https://github.com/pikhq/bootstrap-linux

Re: [PATCH 0/3] build linux-next without perl

[Michal Marek]

On 27.2.2013 22:51, Andrew Morton wrote:
> It'll need to be reasonably good motivation, too.  Because not only do
> we need to patch the kernel, we also need to *maintain* its
> perl-freeness and fix up perlisms as they later get added by others.
> 
> (Perhaps one way of doing this would be to disable perl in regular
> builds, so even if a developer has perl installed on his machine, his
> build will still fail when he invokes it.  Add "PERL=/dev/null" to some
> build targets in some manner.)

I don't think we need to go this far. Apparently, there are people who
want to be able to build the kernel without perl, so let's rely on them
to report if perl appears again as a hard build dependency. If these
people lose interest, then after some time we might end up with perl
being required again, but then this by definition won't be a problem for
anyone who cares.

It is like bashisms in shell scripts and Makefile commands. They should
be avoided, but it wasn't until Debian and Ubuntu switched to /bin/sh ->
dash when we started to fix them. Nowadays, the kernel should build file
with a minimal POSIX shell, but we did not need any
SHELL=$(objtree)/scripts/dash to achieve this.

Michal

Re: [PATCH 0/3] build linux-next without perl

[Geert Uytterhoeven]

On Wed, Feb 27, 2013 at 10:51 PM, Andrew Morton
<akpm@linux-foundation.org> wrote:
> It'll need to be reasonably good motivation, too.  Because not only do
> we need to patch the kernel, we also need to *maintain* its
> perl-freeness and fix up perlisms as they later get added by others.

That's actually a backwards argument: as long as there are scripts in the kernel
source tree using perl, people will add more of them, since they see perl is an
accepted scripting language within the Linux kernel build system.

It's the same as refactoring existing code: if we don't, people will
copy the bad
examples. So we fix them, to avoid this.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH 0/3] build linux-next without perl

[Rob Landley]

On 02/27/2013 03:51:55 PM, Andrew Morton wrote:
> On Tue, 26 Feb 2013 21:57:52 -0800 (PST)
> Rob Landley <rob@landley.net> wrote:
> 
> > Before 2.6.25 building Linux never required perl. This patch series  
> removes
> > the requirement from basic kernel builds (tested on i686, x86_64,  
> arm, mips,
> > powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1.
> >
> > Note, this removes perl from the _build_ environment, not from the  
> _development_
> > environment. This is approximately the same logic behind "make  
> menuconfig"
> > requiring curses but "make oldconfig" not requiring curses.  
> Including
> > zconf.lex.c_shipped in kconfig and then requiring perl makes no  
> sense.
> >
> > ...
> >
> > Mostly people just copy the patches into their local projects (ala
> > https://github.com/rofl0r/sabotage/tree/master/KEEP ) but I'm  
> reposting
> > them to linux-kernel after Gentoo considered using these patches,  
> but didn't
> > because they weren't upstream:
> >   https://bugs.gentoo.org/show_bug.cgi?id=421483
> 
> Sitting here scratching head wondering why you-need-perl is a problem
> for anyone.
> 
> That gentoo bug report provides some explanation: "perl was removed
> from @system".  But I expect other people have different reasons.
> 
> IOW, please better describe the motivation for this patchset.

P.S. A much shorter reply than mine, from David Anders:

https://plus.google.com/101339419642360856354/posts/AyRnz2REq8C

Rob

Re: [PATCH 0/3] build linux-next without perl

[Rob Landley]

On 02/27/2013 03:51:55 PM, Andrew Morton wrote:
> On Tue, 26 Feb 2013 21:57:52 -0800 (PST)
> Rob Landley <rob@landley.net> wrote:
> 
> > Before 2.6.25 building Linux never required perl. This patch series  
> removes
> > the requirement from basic kernel builds (tested on i686, x86_64,  
> arm, mips,
> > powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1.
> >
> > Note, this removes perl from the _build_ environment, not from the  
> _development_
> > environment. This is approximately the same logic behind "make  
> menuconfig"
> > requiring curses but "make oldconfig" not requiring curses.  
> Including
> > zconf.lex.c_shipped in kconfig and then requiring perl makes no  
> sense.
> >
> > ...
> >
> > Mostly people just copy the patches into their local projects (ala
> > https://github.com/rofl0r/sabotage/tree/master/KEEP ) but I'm  
> reposting
> > them to linux-kernel after Gentoo considered using these patches,  
> but didn't
> > because they weren't upstream:
> >   https://bugs.gentoo.org/show_bug.cgi?id=421483
> 
> Sitting here scratching head wondering why you-need-perl is a problem
> for anyone.

I'm scratching my head that people basically keep doing the "you go  
girl!" thing at me about this patch series _off_ the list (even people  
I'd expect to see here, like  
https://twitter.com/jonmasters/status/301166688852901888 ) but this is  
something like the dozenth time I've posted it and nobody seems to  
notice. Oh well.

Can we start with the fact it's a completely gratuitous build  
environment dependency, and the kernel has a history of removing those?  
(I mentioned two in the message you're replying to, ncurses and lex in  
oldconfig.)

This isn't even a "workaround", this is an alternate implementation  
that is as simple or simpler. (Sam Ravnborg acked one of the scripts  
not because he cared about perl, but because it simplified the kernel  
build.)

> That gentoo bug report provides some explanation: "perl was removed
> from @system".  But I expect other people have different reasons.

Actually, removing perl from the build environment is common in cross  
compiling situations. Removing everything you _can_ from the build  
environment is normal when cross compiling. This is because cross  
compiling sucks:

   http://landley.net/writing/docs/cross-compiling.html

Cross compiling has inherent combinatorial complexity. Native compiling  
build complexity is "number of packages times number of package  
versions", with an addendum that things like the compiler and libc  
count as packages with different versions.

When cross compiling, you basically multiply the number of targets  
you're supporting times the number of package versions you're building  
times the number of hosts you're building from. I've installed distros  
I'd never even _heard_ of under kvm because some bug only happened  
there, but of course all the big ones break too:

   http://landley.net/hg/aboriginal/rev/1532
   http://landley.net/hg/aboriginal/rev/1518
   http://landley.net/hg/aboriginal/rev/1318
   http://landley.net/hg/aboriginal/rev/1160

It's not just the combinatorial complexity, it's also less testing in  
general (most people natively compile), plus the entire configure step  
is wrong at the design level for cross compiling: it asks questions  
about the machine you're building on and applies those answers to the  
program you're building. When host and target aren't the same, this is  
at _best_ useless.

So if you're cross compiling in any remotely portable way, you need an  
"airlock step", as described on pages 98-100 the slides for the old  
talk I gave at Ohio LinuxFest, Flourish, and Celf, which is apparently  
making the rounds again:

   https://twitter.com/solardiz/status/306575964064866305

It's the same general idea as Linux From Scratch chapter 5 (you  
populate a directory with just the binaries you need, and restrict the  
$PATH to that) but with a minimalist twist: everything you add is a  
sharp edge some package can catch on. If not now, then after the next  
version upgrade.

And perl is a GIANT HAIRBALL of sharp edges in this regard. There is no  
perl standard, just a single perl implementation that may or may not  
have the whole of CPAN installed. (In fact the "canned values" logic in  
kernel/timeconst.pl uses a giant array of precomputed values because  
the installed perl may or may not have Math::BigInt might not be  
available on the target. Way back in 2008 I thought this meant we had  
to be able to run without that and the Math::BigInt stuff was just for  
regenerating the table, but Peter said  
https://lkml.org/lkml/2008/2/15/548 and didn't mind letting the user  
figure out what the dependencies were when the build broke. Now apply  
that to lots of other packages and guess why letting ./configure not  
find perl is appealing to cross compile environments.)

I'm surprised perl doesn't get dinged more for the single  
implementation. All the shell scripts in the kernel are supposed to  
work with #!/bin/sh pointing to dash instead of bash, people freak when  
Microsoft Word or Excel are whatever some random program parses rather  
than an actual file format. But perl? Everybody remember when perl was  
going to be reimplemented on top of the "parrot" engine?  
(http://www.perlmonks.org/?node_id=272641) You know why that didn't  
happen? Because after several years of effort they couldn't quite make  
it work reliably. Getting a fresh from-scratch engine implementation to  
run the existing corpus of perl code turned out to be _really_hard_.  
Python's got http://wiki.python.org/moin/PythonImplementations and  
there's even an embedded implementation of php (http://ph7.symisc.net/)  
but perl is this one _specific_ giant hairball. If you have trouble  
getting that hairball to work on a new target? Tough. (When I did  
bootstrap work on Hexagon back in 2010, and built linux from scratch  
natively on the result, "will perl work" was one of the big unknowns.  
Luckily it only took about a week of poking and prodding to get it to  
build. Didn't particularly stress it to see how _well_ it worked,  
mostly because I'd carefully arranged the build to need it as little as  
possible. One of x11's dependencies needed it though, off in Beyond  
Linux From Scratch, and wouldn't ./configure it out the way libiconv  
and such did. Don't remember which one.)

By the way, I'm not saying restricting the $PATH by itself is a  
_sufficient_ airlock step. When Wolfgang Denk (the u-boot maintainer)  
tried my cross compiling build environment it immediately broke in 3  
different and strangely fascinating ways for him, one of which  
(http://landley.net/hg/aboriginal/rev/997) evolved into an entire  
environment variable whitelisting step  
(http://landley.net/hg/aboriginal/rev/1175) because once it works well  
enough more people try it and break it in new ways...

And no, my build system isn't special, I just use it as an example  
because it's what I'm most familiar with. What an awful lot of distros  
do is set up a chroot and then "env -i chroot" into it (I.E. the  
Linux From Scratch approach). My build system goes to extra effort so  
that no part of it requires root access on the host (which is why I  
don't chroot, I run the target system under qemu instead, hence the  
title of the above giant slide deck from 2008).

> IOW, please better describe the motivation for this patchset.

You want more? Ok. (You asked.)

In addition to all the above, last week I gave a talk at the Linux  
Foundation Embedded Linux Conference (used to be called CELF before  
they ate it) on turning Android into a real build environment. I didn't  
do slides this time but the outline's here:

   http://landley.net/talks/celf-2013.txt

It would be really convenient if the video of that talk was up so I  
could just point you at it, but you'll have to ask the Linux Foundation  
when that'll be. The outline was just "notes to self" for me, lemme see  
if I can summarize an hour talk in a couple paragraphs.

What I'm doing there is trying to expand Android into a full  
self-hosting development environment so it can get on with being a  
disruptive technology and kicking the PC up into the server space like  
the minicomputer and mainframe before it. I would _very_ much like  
Android to do this before iPhone does because when the S-curve of  
adoptions flattens out (somewhere between 1 and 3.5 billion unit  
installed base I'd guess) and the positive feedback loop of network  
effects kick in, being locked out of _another_ generation of technology  
by an actually COMPETENT monopolist would really suck.

The hardware to use a smartphone as a workstation is just a USB hub  
with keyboard, mouse, and video adapter plugged into it; that's here  
today (although USB3 makes it easier). The rest is software. But  
there's a LOT of software.

This software has 4 basic parts: kernel (which works now because they  
just added stuff to linux without removing anything), a command line  
(I'm writing a new BSD-licensed posix command line; same general reason  
I did years of work in busybox only this time it's old hat), a C  
library (musl-libc.org is the leading contender), and a toolchain  
(looks like llvm at the moment, I'd like to do http://landley.net/qcc  
but my plate's full and there's no time. Why no time? Who is sponsoring  
llvm? Who did "airplay" to put a phone display on an HDTV? You think  
Steve Jobs didn't _notice_ that 8->16->32->64 bits is sustaining  
technologies but mainframe->minicomputer->microcomputer->smartphone is  
disruptive yet _inevitable_?)

This 4-package thing is even more simplified than my Aboriginal Linux  
build (which got it down to 7 packages, but the licensing of those is  
wrong -- no GPL in userspace -- so preinstalling any of it is a  
violation of the Android licensing guidelines and the trademark grows  
teeth so your ads have to be really horribly phrased.)

The reason you _want_ to simplify it is that Google is shipping a  
billion unadministered unix systems with broadband access, which is  
TERRIFYING from a security standpoint. The reason bionic and toolbox  
are stubs even though uClibc and busybox both predate android isn't  
_just_ licensing issues, it's that Google intentionally shipped the  
minimum environment necessary to boot dalvik and get into the java  
sandbox, and is minimizing the attack surface if you can manage to  
escape that.

But Dalvik is this generation's version of ROM Basic: it's something  
the platform has to outgrow in order to wean itself off of the previous  
generation it's cross-compiled from. Once the PC became a self-hosting  
development environment there was an explosion of software for it,  
because you no longer needed a PDP-10 to develop for the PC, having a  
PC was enough to be a developer. This is not currently the case for  
phones, but it should _become_ the case.

If you then say "and to be a self-hosting system, you must add perl".  
And to preinstall perl, you must audit perl for security concerns...  
Can we please, please, please just remove the need for perl as part of  
a self-hosting development environment instead?

P.S. These are just _my_ reasons. Dave Anders is the one who first  
complained to _me_ that perl had been added as a build requirement back  
in 2.6.25. I finally met him in person at the BeagleBone tutorial at  
CELF last week after knowing him for years on freenode (prplague), but  
like all the embedded guys he doesn't hang out here. (I just cc'd him.)  
Nor do the 3 of the 4 other people who congratulated me on getting this  
posted on freenode today (I convinced _one_ of them to come here and  
ack the darn patch).

> It'll need to be reasonably good motivation, too.  Because not only do
> we need to patch the kernel, we also need to *maintain* its
> perl-freeness and fix up perlisms as they later get added by others.

I've already been maintaining it (and submitting it here) for 5 years  
now:

   https://lkml.org/lkml/2008/2/15/541

Sam Ravnborg acked the headers_install change not because of perl but  
because the replacement was simpler than what it replaced. It would be  
difficult to make the timeconst thing with the giant blob of  
pregenerated values _worse_.

As for fixing up existing perlisms, people who use my scripts already  
send you patches to remove perl dependencies in things that I don't  
enable in my builds:

   http://lkml.indiana.edu/hypermail/linux/kernel/0910.0/01896.html

Honestly, not the only one who does this. Sing along:

   http://barb.velvet.com/humor/lurkers.html

(Which is really, really annoying. But that's embedded linux for you.  
They've all written upstream off as ignoring them. You think I'm typing  
an epic tl;dr at _you_ guys, it's _harder_ to get through the other  
way...)

> (Perhaps one way of doing this would be to disable perl in regular
> builds, so even if a developer has perl installed on his machine, his
> build will still fail when he invokes it.  Add "PERL=/dev/null" to  
> some
> build targets in some manner.)

   http://landley.net/aboriginal/about.html

(And yes, I need to get a release out that uses 3.8 but they screwed up  
interrupt routing on QEMU's arm versatile board emulation again and I  
haven't had time to track it down because I've been trying to get  
_this_ pushed upstream this merge window. Again. Plus I need to figure  
out what I broke in powerpc userspace, and run the automated Linux From  
Scratch build under qemu on all targets to make sure I haven't missed  
anything else...)

Rob

Re: [PATCH 0/3] build linux-next without perl

[Andrew Morton]

On Tue, 26 Feb 2013 21:57:52 -0800 (PST)
Rob Landley <rob@landley.net> wrote:

> Before 2.6.25 building Linux never required perl. This patch series removes
> the requirement from basic kernel builds (tested on i686, x86_64, arm, mips,
> powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1.
> 
> Note, this removes perl from the _build_ environment, not from the _development_
> environment. This is approximately the same logic behind "make menuconfig"
> requiring curses but "make oldconfig" not requiring curses. Including
> zconf.lex.c_shipped in kconfig and then requiring perl makes no sense.
>
> ...
>
> Mostly people just copy the patches into their local projects (ala
> https://github.com/rofl0r/sabotage/tree/master/KEEP ) but I'm reposting
> them to linux-kernel after Gentoo considered using these patches, but didn't 
> because they weren't upstream:
>   https://bugs.gentoo.org/show_bug.cgi?id=421483

Sitting here scratching head wondering why you-need-perl is a problem
for anyone.

That gentoo bug report provides some explanation: "perl was removed
from @system".  But I expect other people have different reasons.

IOW, please better describe the motivation for this patchset.

It'll need to be reasonably good motivation, too.  Because not only do
we need to patch the kernel, we also need to *maintain* its
perl-freeness and fix up perlisms as they later get added by others.

(Perhaps one way of doing this would be to disable perl in regular
builds, so even if a developer has perl installed on his machine, his
build will still fail when he invokes it.  Add "PERL=/dev/null" to some
build targets in some manner.)

[PATCH 0/3] build linux-next without perl

[Rob Landley]

Before 2.6.25 building Linux never required perl. This patch series removes
the requirement from basic kernel builds (tested on i686, x86_64, arm, mips,
powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1.

Note, this removes perl from the _build_ environment, not from the _development_
environment. This is approximately the same logic behind "make menuconfig"
requiring curses but "make oldconfig" not requiring curses. Including
zconf.lex.c_shipped in kconfig and then requiring perl makes no sense.

Earlier versions have been posted to the mailing list over, and over, and over:

http://lkml.indiana.edu/hypermail/linux/kernel/1212.2/00734.html
http://lkml.indiana.edu/hypermail/linux/kernel/1201.2/02849.html
http://lkml.indiana.edu/hypermail/linux/kernel/0912.1/00093.htm
http://lkml.indiana.edu/hypermail/linux/kernel/0909.2/01661.html
http://lkml.indiana.edu/hypermail/linux/kernel/0901.0/00772.html
http://lkml.indiana.edu/hypermail/linux/kernel/0901.0/00148.html
http://lkml.indiana.edu/hypermail/linux/kernel/0802.1/4393.html

They've also been posted individually rather than as a series:
http://lkml.indiana.edu/hypermail/linux/kernel/1101.2/00650.html
http://lkml.indiana.edu/hypermail/linux/kernel/1101.2/00115.html

Mentioned during related perl removal patches from other people:
http://lkml.indiana.edu/hypermail/linux/kernel/0910.0/02145.html

And so on. So here it is again. The most up to date version of these patches
is always available at http://landley.net/hg/aboriginal in the sources/patches
directory. Specifically, check:

http://landley.net/hg/aboriginal/log/tip/sources/patches/linux-noperl-capflags.patch
http://landley.net/hg/aboriginal/log/tip/sources/patches/linux-noperl-headers.patch
http://landley.net/hg/aboriginal/log/tip/sources/patches/linux-noperl-timeconst.patch

(Earlier versions had the kernel version number in the filename, so you'll have
to dig in the repository a bit to find them if you want the version for an
older kernel.)

Mostly people just copy the patches into their local projects (ala
https://github.com/rofl0r/sabotage/tree/master/KEEP ) but I'm reposting
them to linux-kernel after Gentoo considered using these patches, but didn't 
because they weren't upstream:
  https://bugs.gentoo.org/show_bug.cgi?id=421483

Here's either Peter Anvin or Greg KH (not quite sure which) suggesting removing
timeconst.pl:
http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00506.html

Here's Sam Ravnborg acking the headers_install patch:
https://lkml.org/lkml/2013/2/15/121

Here's Michal Marek reviewing the headers_install patch (his suggestions
are incorporated in this version):
http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00078.html

Here's somebody else reminding me to repost the patch series:
http://lkml.indiana.edu/hypermail/linux/kernel/1302.0/00607.html

Plus various people keep poking me and asking about it. I'll see who I can
remember to ping.

(Sorry for the delay. Balsa is such a horrible email client I wrote a python
script to send this email. Took a while to debug. Probably still likely to
screw something up...)