* [PATCH 0/3] build linux-next without perl @ 2013-02-27 5:57 Rob Landley 2013-02-27 5:57 ` [PATCH 1/3] convert mkcapflags.pl to mkcapflags.sh Rob Landley ` (3 more replies) 0 siblings, 4 replies; 17+ messages in thread From: Rob Landley @ 2013-02-27 5:57 UTC (permalink / raw) To: linux-kernel Cc: Al Viro, Andrew Morton, Arnd Bergmann, Cyrill Gorcunov, Dave Jones, David Howells, David Howells, Greg Kroah-Hartman, H. Peter Anvin, H. Peter Anvin, Ingo Molnar, Josh Boyer, Michal Marek, Paul E. McKenney, Peter Zijlstra, Rusty Russell, Sam Ravnborg, Thomas Gleixner, Vince Weaver, x86, Zheng Yan Before 2.6.25 building Linux never required perl. This patch series removes the requirement from basic kernel builds (tested on i686, x86_64, arm, mips, powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1. Note, this removes perl from the _build_ environment, not from the _development_ environment. This is approximately the same logic behind "make menuconfig" requiring curses but "make oldconfig" not requiring curses. Including zconf.lex.c_shipped in kconfig and then requiring perl makes no sense. Earlier versions have been posted to the mailing list over, and over, and over: http://lkml.indiana.edu/hypermail/linux/kernel/1212.2/00734.html http://lkml.indiana.edu/hypermail/linux/kernel/1201.2/02849.html http://lkml.indiana.edu/hypermail/linux/kernel/0912.1/00093.htm http://lkml.indiana.edu/hypermail/linux/kernel/0909.2/01661.html http://lkml.indiana.edu/hypermail/linux/kernel/0901.0/00772.html http://lkml.indiana.edu/hypermail/linux/kernel/0901.0/00148.html http://lkml.indiana.edu/hypermail/linux/kernel/0802.1/4393.html They've also been posted individually rather than as a series: http://lkml.indiana.edu/hypermail/linux/kernel/1101.2/00650.html http://lkml.indiana.edu/hypermail/linux/kernel/1101.2/00115.html Mentioned during related perl removal patches from other people: http://lkml.indiana.edu/hypermail/linux/kernel/0910.0/02145.html And so on. So here it is again. The most up to date version of these patches is always available at http://landley.net/hg/aboriginal in the sources/patches directory. Specifically, check: http://landley.net/hg/aboriginal/log/tip/sources/patches/linux-noperl-capflags.patch http://landley.net/hg/aboriginal/log/tip/sources/patches/linux-noperl-headers.patch http://landley.net/hg/aboriginal/log/tip/sources/patches/linux-noperl-timeconst.patch (Earlier versions had the kernel version number in the filename, so you'll have to dig in the repository a bit to find them if you want the version for an older kernel.) Mostly people just copy the patches into their local projects (ala https://github.com/rofl0r/sabotage/tree/master/KEEP ) but I'm reposting them to linux-kernel after Gentoo considered using these patches, but didn't because they weren't upstream: https://bugs.gentoo.org/show_bug.cgi?id=421483 Here's either Peter Anvin or Greg KH (not quite sure which) suggesting removing timeconst.pl: http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00506.html Here's Sam Ravnborg acking the headers_install patch: https://lkml.org/lkml/2013/2/15/121 Here's Michal Marek reviewing the headers_install patch (his suggestions are incorporated in this version): http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00078.html Here's somebody else reminding me to repost the patch series: http://lkml.indiana.edu/hypermail/linux/kernel/1302.0/00607.html Plus various people keep poking me and asking about it. I'll see who I can remember to ping. (Sorry for the delay. Balsa is such a horrible email client I wrote a python script to send this email. Took a while to debug. Probably still likely to screw something up...) ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 1/3] convert mkcapflags.pl to mkcapflags.sh 2013-02-27 5:57 [PATCH 0/3] build linux-next without perl Rob Landley @ 2013-02-27 5:57 ` Rob Landley 2013-02-27 5:58 ` [PATCH 2/3] convert timeconst.pl to mktimeconst.c Rob Landley ` (2 subsequent siblings) 3 siblings, 0 replies; 17+ messages in thread From: Rob Landley @ 2013-02-27 5:57 UTC (permalink / raw) To: linux-kernel Cc: Arnd Bergmann, Dave Jones, David Howells, Paul E. McKenney, Cyrill Gorcunov, Vince Weaver, Zheng Yan, Peter Zijlstra, x86, H. Peter Anvin, Ingo Molnar, Thomas Gleixner From: Rob Landley <rob@landley.net> Generate asm-x86/cpufeature.h with posix-2008 commands instead of perl. Signed-off-by: Rob Landley <rob@landley.net> --- arch/x86/kernel/cpu/Makefile | 4 +- arch/x86/kernel/cpu/mkcapflags.pl | 48 ---------------------------- arch/x86/kernel/cpu/mkcapflags.sh | 41 +++++++++++++++++++++++ 3 files changed, 43 insertions(+), 50 deletions(-) --- linux/arch/x86/kernel/cpu/mkcapflags.pl +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/perl -w -# -# Generate the x86_cap_flags[] array from include/asm-x86/cpufeature.h -# - -($in, $out) = @ARGV; - -open(IN, "< $in\0") or die "$0: cannot open: $in: $!\n"; -open(OUT, "> $out\0") or die "$0: cannot create: $out: $!\n"; - -print OUT "#ifndef _ASM_X86_CPUFEATURE_H\n"; -print OUT "#include <asm/cpufeature.h>\n"; -print OUT "#endif\n"; -print OUT "\n"; -print OUT "const char * const x86_cap_flags[NCAPINTS*32] = {\n"; - -%features = (); -$err = 0; - -while (defined($line = <IN>)) { - if ($line =~ /^\s*\#\s*define\s+(X86_FEATURE_(\S+))\s+(.*)$/) { - $macro = $1; - $feature = "\L$2"; - $tail = $3; - if ($tail =~ /\/\*\s*\"([^"]*)\".*\*\//) { - $feature = "\L$1"; - } - - next if ($feature eq ''); - - if ($features{$feature}++) { - print STDERR "$in: duplicate feature name: $feature\n"; - $err++; - } - printf OUT "\t%-32s = \"%s\",\n", "[$macro]", $feature; - } -} -print OUT "};\n"; - -close(IN); -close(OUT); - -if ($err) { - unlink($out); - exit(1); -} - -exit(0); --- /dev/null 2012-07-11 05:54:12.790418331 -0500 +++ linux/arch/x86/kernel/cpu/mkcapflags.sh 2012-07-22 13:13:57.700823094 -0500 @@ -0,0 +1,41 @@ +#!/bin/sh +# +# Generate the x86_cap_flags[] array from include/asm/cpufeature.h +# + +IN=$1 +OUT=$2 + +TABS="$(printf '\t\t\t\t\t')" +trap 'rm "$OUT"' EXIT + +( + echo "#ifndef _ASM_X86_CPUFEATURE_H" + echo "#include <asm/cpufeature.h>" + echo "#endif" + echo "" + echo "const char * const x86_cap_flags[NCAPINTS*32] = {" + + # Iterate through any input lines starting with #define X86_FEATURE_ + sed -n -e 's/\t/ /g' -e 's/^ *# *define *X86_FEATURE_//p' $IN | + while read i + do + # Name is everything up to the first whitespace + NAME="$(echo "$i" | sed 's/ .*//')" + + # If the /* comment */ starts with a quote string, grab that. + VALUE="$(echo "$i" | sed -n 's@.*/\* *\("[^"]*"\).*\*/@\1@p')" + [ -z "$VALUE" ] && VALUE="\"$NAME\"" + [ "$VALUE" == '""' ] && continue + + # Name is uppercase, VALUE is all lowercase + VALUE="$(echo "$VALUE" | tr A-Z a-z)" + + TABCOUNT=$(( ( 5*8 - 14 - $(echo "$NAME" | wc -c) ) / 8 )) + printf "\t[%s]%.*s = %s,\n" \ + "X86_FEATURE_$NAME" "$TABCOUNT" "$TABS" "$VALUE" + done + echo "};" +) > $OUT + +trap - EXIT diff --git a/arch/x86/kernel/cpu/Makefile b/arch/x86/kernel/cpu/Makefile index 25f24dc..9edf7e7 100644 --- a/arch/x86/kernel/cpu/Makefile +++ b/arch/x86/kernel/cpu/Makefile @@ -43,10 +43,10 @@ obj-$(CONFIG_MTRR) += mtrr/ obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o perf_event_amd_ibs.o quiet_cmd_mkcapflags = MKCAP $@ - cmd_mkcapflags = $(PERL) $(srctree)/$(src)/mkcapflags.pl $< $@ + cmd_mkcapflags = $(CONFIG_SHELL) $(srctree)/$(src)/mkcapflags.sh $< $@ cpufeature = $(src)/../../include/asm/cpufeature.h targets += capflags.c -$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.pl FORCE +$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.sh FORCE $(call if_changed,mkcapflags) ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 2/3] convert timeconst.pl to mktimeconst.c 2013-02-27 5:57 [PATCH 0/3] build linux-next without perl Rob Landley 2013-02-27 5:57 ` [PATCH 1/3] convert mkcapflags.pl to mkcapflags.sh Rob Landley @ 2013-02-27 5:58 ` Rob Landley 2013-02-27 5:58 ` [PATCH 3/3] convert headers_install.pl to headers_install.sh Rob Landley 2013-02-27 21:51 ` [PATCH 0/3] build linux-next without perl Andrew Morton 3 siblings, 0 replies; 17+ messages in thread From: Rob Landley @ 2013-02-27 5:58 UTC (permalink / raw) To: linux-kernel Cc: H. Peter Anvin, Al Viro, Paul E. McKenney, Andrew Morton, David Howells, Rusty Russell, Greg Kroah-Hartman From: Rob Landley <rob@landley.net> Replace perl header file generator with smaller/faster/simpler C version. Hasn't changed in several years, see: Message-ID: <4D35FEF3.4070001@parallels.com> Subject: Re: [PATCH] Use sed instead of perl to generate x86/kernel/cpu/capflags.c. Note, the version I submitted 2 years ago included a test program to show it was producing the same output for HZ 1 through 5000: https://lkml.org/lkml/2011/1/18/257 Signed-off-by: Rob Landley <rob@landley.net> --- kernel/Makefile | 10 - kernel/mktimeconst.c | 110 +++++++++++ kernel/timeconst.pl | 376 ----------------------------------------- 3 files changed, 116 insertions(+), 380 deletions(-) diff --git a/kernel/Makefile b/kernel/Makefile index eceac38..f00be6d 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -127,11 +127,13 @@ $(obj)/config_data.h: $(obj)/config_data.gz FORCE $(obj)/time.o: $(obj)/timeconst.h -quiet_cmd_timeconst = TIMEC $@ - cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@ +hostprogs-y += mktimeconst +quiet_cmd_mktimeconst = TIMEC $@ + cmd_mktimeconst = $(obj)/mktimeconst $(CONFIG_HZ) $@ || ( rm -f $@ && exit 1 ) + targets += timeconst.h -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE - $(call if_changed,timeconst) +$(obj)/timeconst.h: $(obj)/mktimeconst FORCE + $(call if_changed,mktimeconst) ifeq ($(CONFIG_MODULE_SIG),y) # --- /dev/null +++ linux/kernel/mktimeconst.c @@ -0,0 +1,110 @@ +/* Copyright 2010 Parallels Inc, licensed under GPLv2 */ +/* Copyright 2010-2013 Rob Landley <rob@landley.net> */ + +#include <inttypes.h> +#include <stdio.h> +#include <stdlib.h> + +int main(int argc, char *argv[]) +{ + uint64_t hz, periods[] = {1000, 1000000}; + char *names[] = {"MSEC", "USEC"}; + FILE *file; + int i, j; + + if (argc != 3 || (hz = atol(argv[1])) < 1 + || !(file = fopen(argv[2], "w"))) + { + fprintf(stderr, "Usage: mktimeconst HZ FILENAME\n\n"); + fprintf(stderr, "Generate a header file with constants to convert between\n"); + fprintf(stderr, "decimal HZ timer ticks and milisecond or microsecond delays,\n"); + fprintf(stderr, "using reciprocal multiplication to avoid 64 bit division.\n"); + exit(1); + } + + fprintf(file, + "/* Automatically generated by kernel/mktimeconst */\n" + "/* Conversion constants for HZ == %"PRIu64" */\n\n" + "#ifndef __KERNEL_TIMECONST_H\n" + "#define __KERNEL_TIMECONST_H\n\n" + "#include <linux/param.h>\n" + "#include <linux/types.h>\n\n" + "#if HZ != %"PRIu64"\n" + "#error \"kernel/timeconst.h has the wrong HZ value!\"\n" + "#endif\n\n", hz, hz); + + /* Repeat for MSEC and USEC */ + + for (i = 0; i < 2; i++) { + uint64_t gcd, period; + + /* Find greatest common denominator using Euclid's algorithm. */ + + gcd = hz; + period = periods[i]; + while (period) { + uint64_t temp = gcd % period; + gcd = period; + period = temp; + } + + /* Output both directions (HZ_TO_PERIOD and PERIOD_TO_HZ) */ + + for (j = 0; j < 2; j++) { + char name[16]; + uint64_t from = j ? periods[i] : hz; + uint64_t to = j ? hz : periods[i]; + uint64_t mul32 = 0, adj32 = 0, shift = 0; + + sprintf(name, j ? "%s_TO_HZ" : "HZ_TO_%s", names[i]); + + /* Figure out what shift value gives 32 significant + bits of MUL32 data. (Worst case to=1 from=1000000 + uses 52 bits, to<<shift won't overflow 64 bit math.) + */ + + for (;;) { + mul32 = ((to << shift) + from - 1) / from; + if (mul32 >= (1UL<<31)) + break; + shift++; + } + + /* ADJ32 is is just (((FROM/GCD)-1)<<SHIFT)/(FROM/GCD) + but this can overflow 64 bit math (examples, HZ=24 + or HZ=122). Worst case scenario uses 32+20+20=72 + bits. Workaround: split off bottom 32 bits and + reassemble after calculation (32+64=96 bits). */ + + adj32 = from / gcd; + + if (shift > 32) { + uint64_t upper, lower; + + upper = (adj32 - 1) << (shift - 32); + lower = (upper % adj32) << 32; + adj32 = ((upper/adj32) << 32) + (lower/adj32); + } else + adj32 = ((adj32 - 1) << shift) / adj32; + + /* Emit the constants into the header file. */ + + fprintf(file, "#define %s_MUL32\tU64_C(0x%"PRIx64")\n", + name, mul32); + fprintf(file, "#define %s_ADJ32\tU64_C(0x%"PRIx64")\n", + name, adj32); + fprintf(file, "#define %s_SHR32\t%"PRIu64"\n", + name, shift); + fprintf(file, "#define %s_NUM\t\tU64_C(%"PRIu64")\n", + name, to/gcd); + fprintf(file, "#define %s_DEN\t\tU64_C(%"PRIu64")\n\n", + name, from/gcd); + } + } + fprintf(file, "#endif /* __KERNEL_TIMECONST_H */\n"); + + /* Notice if the disk fills up. */ + + fflush(stdout); + return ferror(stdout); +} --- linux/kernel/timeconst.pl +++ /dev/null @@ -1,376 +0,0 @@ -#!/usr/bin/perl -# ----------------------------------------------------------------------- -# -# Copyright 2007-2008 rPath, Inc. - All Rights Reserved -# -# This file is part of the Linux kernel, and is made available under -# the terms of the GNU General Public License version 2 or (at your -# option) any later version; incorporated herein by reference. -# -# ----------------------------------------------------------------------- -# - -# -# Usage: timeconst.pl HZ > timeconst.h -# - -# Precomputed values for systems without Math::BigInt -# Generated by: -# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 1200 -%canned_values = ( - 24 => [ - '0xa6aaaaab','0x2aaaaaa',26, - 125,3, - '0xc49ba5e4','0x1fbe76c8b4',37, - 3,125, - '0xa2c2aaab','0xaaaa',16, - 125000,3, - '0xc9539b89','0x7fffbce4217d',47, - 3,125000, - ], 32 => [ - '0xfa000000','0x6000000',27, - 125,4, - '0x83126e98','0xfdf3b645a',36, - 4,125, - '0xf4240000','0x0',17, - 31250,1, - '0x8637bd06','0x3fff79c842fa',46, - 1,31250, - ], 48 => [ - '0xa6aaaaab','0x6aaaaaa',27, - 125,6, - '0xc49ba5e4','0xfdf3b645a',36, - 6,125, - '0xa2c2aaab','0x15555',17, - 62500,3, - '0xc9539b89','0x3fffbce4217d',46, - 3,62500, - ], 64 => [ - '0xfa000000','0xe000000',28, - 125,8, - '0x83126e98','0x7ef9db22d',35, - 8,125, - '0xf4240000','0x0',18, - 15625,1, - '0x8637bd06','0x1fff79c842fa',45, - 1,15625, - ], 100 => [ - '0xa0000000','0x0',28, - 10,1, - '0xcccccccd','0x733333333',35, - 1,10, - '0x9c400000','0x0',18, - 10000,1, - '0xd1b71759','0x1fff2e48e8a7',45, - 1,10000, - ], 122 => [ - '0x8325c53f','0xfbcda3a',28, - 500,61, - '0xf9db22d1','0x7fbe76c8b',35, - 61,500, - '0x8012e2a0','0x3ef36',18, - 500000,61, - '0xffda4053','0x1ffffbce4217',45, - 61,500000, - ], 128 => [ - '0xfa000000','0x1e000000',29, - 125,16, - '0x83126e98','0x3f7ced916',34, - 16,125, - '0xf4240000','0x40000',19, - 15625,2, - '0x8637bd06','0xfffbce4217d',44, - 2,15625, - ], 200 => [ - '0xa0000000','0x0',29, - 5,1, - '0xcccccccd','0x333333333',34, - 1,5, - '0x9c400000','0x0',19, - 5000,1, - '0xd1b71759','0xfff2e48e8a7',44, - 1,5000, - ], 250 => [ - '0x80000000','0x0',29, - 4,1, - '0x80000000','0x180000000',33, - 1,4, - '0xfa000000','0x0',20, - 4000,1, - '0x83126e98','0x7ff7ced9168',43, - 1,4000, - ], 256 => [ - '0xfa000000','0x3e000000',30, - 125,32, - '0x83126e98','0x1fbe76c8b',33, - 32,125, - '0xf4240000','0xc0000',20, - 15625,4, - '0x8637bd06','0x7ffde7210be',43, - 4,15625, - ], 300 => [ - '0xd5555556','0x2aaaaaaa',30, - 10,3, - '0x9999999a','0x1cccccccc',33, - 3,10, - '0xd0555556','0xaaaaa',20, - 10000,3, - '0x9d495183','0x7ffcb923a29',43, - 3,10000, - ], 512 => [ - '0xfa000000','0x7e000000',31, - 125,64, - '0x83126e98','0xfdf3b645',32, - 64,125, - '0xf4240000','0x1c0000',21, - 15625,8, - '0x8637bd06','0x3ffef39085f',42, - 8,15625, - ], 1000 => [ - '0x80000000','0x0',31, - 1,1, - '0x80000000','0x0',31, - 1,1, - '0xfa000000','0x0',22, - 1000,1, - '0x83126e98','0x1ff7ced9168',41, - 1,1000, - ], 1024 => [ - '0xfa000000','0xfe000000',32, - 125,128, - '0x83126e98','0x7ef9db22',31, - 128,125, - '0xf4240000','0x3c0000',22, - 15625,16, - '0x8637bd06','0x1fff79c842f',41, - 16,15625, - ], 1200 => [ - '0xd5555556','0xd5555555',32, - 5,6, - '0x9999999a','0x66666666',31, - 6,5, - '0xd0555556','0x2aaaaa',22, - 2500,3, - '0x9d495183','0x1ffcb923a29',41, - 3,2500, - ] -); - -$has_bigint = eval 'use Math::BigInt qw(bgcd); 1;'; - -sub bint($) -{ - my($x) = @_; - return Math::BigInt->new($x); -} - -# -# Constants for division by reciprocal multiplication. -# (bits, numerator, denominator) -# -sub fmul($$$) -{ - my ($b,$n,$d) = @_; - - $n = bint($n); - $d = bint($d); - - return scalar (($n << $b)+$d-bint(1))/$d; -} - -sub fadj($$$) -{ - my($b,$n,$d) = @_; - - $n = bint($n); - $d = bint($d); - - $d = $d/bgcd($n, $d); - return scalar (($d-bint(1)) << $b)/$d; -} - -sub fmuls($$$) { - my($b,$n,$d) = @_; - my($s,$m); - my($thres) = bint(1) << ($b-1); - - $n = bint($n); - $d = bint($d); - - for ($s = 0; 1; $s++) { - $m = fmul($s,$n,$d); - return $s if ($m >= $thres); - } - return 0; -} - -# Generate a hex value if the result fits in 64 bits; -# otherwise skip. -sub bignum_hex($) { - my($x) = @_; - my $s = $x->as_hex(); - - return (length($s) > 18) ? undef : $s; -} - -# Provides mul, adj, and shr factors for a specific -# (bit, time, hz) combination -sub muladj($$$) { - my($b, $t, $hz) = @_; - my $s = fmuls($b, $t, $hz); - my $m = fmul($s, $t, $hz); - my $a = fadj($s, $t, $hz); - return (bignum_hex($m), bignum_hex($a), $s); -} - -# Provides numerator, denominator values -sub numden($$) { - my($n, $d) = @_; - my $g = bgcd($n, $d); - return ($n/$g, $d/$g); -} - -# All values for a specific (time, hz) combo -sub conversions($$) { - my ($t, $hz) = @_; - my @val = (); - - # HZ_TO_xx - push(@val, muladj(32, $t, $hz)); - push(@val, numden($t, $hz)); - - # xx_TO_HZ - push(@val, muladj(32, $hz, $t)); - push(@val, numden($hz, $t)); - - return @val; -} - -sub compute_values($) { - my($hz) = @_; - my @val = (); - my $s, $m, $a, $g; - - if (!$has_bigint) { - die "$0: HZ == $hz not canned and ". - "Math::BigInt not available\n"; - } - - # MSEC conversions - push(@val, conversions(1000, $hz)); - - # USEC conversions - push(@val, conversions(1000000, $hz)); - - return @val; -} - -sub outputval($$) -{ - my($name, $val) = @_; - my $csuf; - - if (defined($val)) { - if ($name !~ /SHR/) { - $val = "U64_C($val)"; - } - printf "#define %-23s %s\n", $name.$csuf, $val.$csuf; - } -} - -sub output($@) -{ - my($hz, @val) = @_; - my $pfx, $bit, $suf, $s, $m, $a; - - print "/* Automatically generated by kernel/timeconst.pl */\n"; - print "/* Conversion constants for HZ == $hz */\n"; - print "\n"; - print "#ifndef KERNEL_TIMECONST_H\n"; - print "#define KERNEL_TIMECONST_H\n"; - print "\n"; - - print "#include <linux/param.h>\n"; - print "#include <linux/types.h>\n"; - - print "\n"; - print "#if HZ != $hz\n"; - print "#error \"kernel/timeconst.h has the wrong HZ value!\"\n"; - print "#endif\n"; - print "\n"; - - foreach $pfx ('HZ_TO_MSEC','MSEC_TO_HZ', - 'HZ_TO_USEC','USEC_TO_HZ') { - foreach $bit (32) { - foreach $suf ('MUL', 'ADJ', 'SHR') { - outputval("${pfx}_$suf$bit", shift(@val)); - } - } - foreach $suf ('NUM', 'DEN') { - outputval("${pfx}_$suf", shift(@val)); - } - } - - print "\n"; - print "#endif /* KERNEL_TIMECONST_H */\n"; -} - -# Pretty-print Perl values -sub perlvals(@) { - my $v; - my @l = (); - - foreach $v (@_) { - if (!defined($v)) { - push(@l, 'undef'); - } elsif ($v =~ /^0x/) { - push(@l, "\'".$v."\'"); - } else { - push(@l, $v.''); - } - } - return join(',', @l); -} - -($hz) = @ARGV; - -# Use this to generate the %canned_values structure -if ($hz eq '--can') { - shift(@ARGV); - @hzlist = sort {$a <=> $b} (@ARGV); - - print "# Precomputed values for systems without Math::BigInt\n"; - print "# Generated by:\n"; - print "# timeconst.pl --can ", join(' ', @hzlist), "\n"; - print "\%canned_values = (\n"; - my $pf = "\t"; - foreach $hz (@hzlist) { - my @values = compute_values($hz); - print "$pf$hz => [\n"; - while (scalar(@values)) { - my $bit; - foreach $bit (32) { - my $m = shift(@values); - my $a = shift(@values); - my $s = shift(@values); - print "\t\t", perlvals($m,$a,$s), ",\n"; - } - my $n = shift(@values); - my $d = shift(@values); - print "\t\t", perlvals($n,$d), ",\n"; - } - print "\t]"; - $pf = ', '; - } - print "\n);\n"; -} else { - $hz += 0; # Force to number - if ($hz < 1) { - die "Usage: $0 HZ\n"; - } - - $cv = $canned_values{$hz}; - @val = defined($cv) ? @$cv : compute_values($hz); - output($hz, @val); -} -exit 0; ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 3/3] convert headers_install.pl to headers_install.sh 2013-02-27 5:57 [PATCH 0/3] build linux-next without perl Rob Landley 2013-02-27 5:57 ` [PATCH 1/3] convert mkcapflags.pl to mkcapflags.sh Rob Landley 2013-02-27 5:58 ` [PATCH 2/3] convert timeconst.pl to mktimeconst.c Rob Landley @ 2013-02-27 5:58 ` Rob Landley 2013-02-28 8:19 ` Michal Marek 2013-02-27 21:51 ` [PATCH 0/3] build linux-next without perl Andrew Morton 3 siblings, 1 reply; 17+ messages in thread From: Rob Landley @ 2013-02-27 5:58 UTC (permalink / raw) To: linux-kbuild Cc: linux-kernel, Thomas Gleixner, Josh Boyer, Paul E. McKenney, Andrew Morton, David Howells, Michal Marek, Sam Ravnborg From: Rob Landley <rob@landley.net> Remove perl from make headers_install by replacing a perl script (doing a simple regex search and replace) with a smaller, faster, simpler, POSIX-2008 shell script implementation. The new shell script is a single for loop calling sed and piping its output through unifdef to produce the target file. Same as last time except for minor tweak to deal with code review from here: http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00078.html (Note that this drops the "arch" argument, which isn't used. Kbuild already points to the right input files on the command line.) Signed-off-by: Rob Landley <rob@landley.net> --- scripts/Makefile.headersinst | 4 +- scripts/headers_install.pl | 63 --------------------------------- scripts/headers_install.sh | 43 ++++++++++++++++++++++ 3 files changed, 45 insertions(+), 65 deletions(-) diff -ruN linux-3.1/scripts/headers_install.sh linux-2.6.30/scripts/headers_install.sh --- linux-3.1/scripts/headers_install.sh +++ linux/scripts/headers_install.sh @@ -0,0 +1,43 @@ +#!/bin/sh + +if [ $# -lt 1 ] +then + echo "Usage: headers_install.sh OUTDIR [FILES...] + echo + echo "Prepares kernel header files for use by user space, by removing" + echo "all compiler.h definitions and #includes, removing any" + echo "#ifdef __KERNEL__ sections, and putting __underscores__ around" + echo "asm/inline/volatile keywords." + echo + echo "OUTDIR: directory to write each userspace header FILE to." + echo "FILES: list of header files to operate on." + + exit 1 +fi + +# Grab arguments + +OUTDIR="$1" +shift + +# Iterate through files listed on command line + +FILE= +trap 'rm -f "$OUTDIR/$FILE" "$OUTDIR/$FILE.sed"' EXIT +for i in "$@" +do + FILE="$(basename "$i")" + sed -r \ + -e 's/([ \t(])(__user|__force|__iomem)[ \t]/\1/g' \ + -e 's/__attribute_const__([ \t]|$)/\1/g' \ + -e 's@^#include <linux/compiler.h>@@' \ + -e 's/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g' \ + -e 's/(^|[ \t(])(inline|asm|volatile)([ \t(]|$)/\1__\2__\3/g' \ + -e 's@#(ifndef|define|endif[ \t]*/[*])[ \t]*_UAPI@#\1 @' \ + "$i" > "$OUTDIR/$FILE.sed" || exit 1 + scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ "$OUTDIR/$FILE.sed" \ + > "$OUTDIR/$FILE" + [ $? -gt 1 ] && exit 1 + rm -f "$OUTDIR/$FILE.sed" +done +trap - EXIT diff -ruN linux-3.1/scripts/Makefile.headersinst --- linux-3.1/scripts/Makefile.headersinst +++ linux/scripts/Makefile.headersinst @@ -71,7 +71,7 @@ quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ file$(if $(word 2, $(all-files)),s)) cmd_install = \ - $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \ + $(CONFIG_SHELL) $< $(installdir) $(input-files); \ for F in $(wrapper-files); do \ echo "\#include <asm-generic/$$F>" > $(installdir)/$$F; \ done; \ @@ -97,7 +97,7 @@ @: targets += $(install-file) -$(install-file): scripts/headers_install.pl $(input-files) FORCE +$(install-file): scripts/headers_install.sh $(input-files) FORCE $(if $(unwanted),$(call cmd,remove),) $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) $(call if_changed,install) --- a/scripts/headers_install.pl +++ /dev/null @@ -1,63 +0,0 @@ -#!/usr/bin/perl -w -# -# headers_install prepare the listed header files for use in -# user space and copy the files to their destination. -# -# Usage: headers_install.pl readdir installdir arch [files...] -# installdir: dir to install the files to -# arch: current architecture -# arch is used to force a reinstallation when the arch -# changes because kbuild then detect a command line change. -# files: list of files to check -# -# Step in preparation for users space: -# 1) Drop all use of compiler.h definitions -# 2) Drop include of compiler.h -# 3) Drop all sections defined out by __KERNEL__ (using unifdef) - -use strict; - -my ($installdir, $arch, @files) = @ARGV; - -my $unifdef = "scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__"; - -foreach my $filename (@files) { - my $file = $filename; - $file =~ s!^.*/!!; - - my $tmpfile = "$installdir/$file.tmp"; - - open(my $in, '<', $filename) - or die "$filename: $!\n"; - open(my $out, '>', $tmpfile) - or die "$tmpfile: $!\n"; - while (my $line = <$in>) { - $line =~ s/([\s(])__user\s/$1/g; - $line =~ s/([\s(])__force\s/$1/g; - $line =~ s/([\s(])__iomem\s/$1/g; - $line =~ s/\s__attribute_const__\s/ /g; - $line =~ s/\s__attribute_const__$//g; - $line =~ s/\b__packed\b/__attribute__((packed))/g; - $line =~ s/^#include <linux\/compiler.h>//; - $line =~ s/(^|\s)(inline)\b/$1__$2__/g; - $line =~ s/(^|\s)(asm)\b(\s|[(]|$)/$1__$2__$3/g; - $line =~ s/(^|\s|[(])(volatile)\b(\s|[(]|$)/$1__$2__$3/g; - $line =~ s/#ifndef\s+_UAPI/#ifndef /; - $line =~ s/#define\s+_UAPI/#define /; - $line =~ s!#endif\s+/[*]\s*_UAPI!#endif /* !; - printf {$out} "%s", $line; - } - close $out; - close $in; - - system $unifdef . " $tmpfile > $installdir/$file"; - # unifdef will exit 0 on success, and will exit 1 when the - # file was processed successfully but no changes were made, - # so abort only when it's higher than that. - my $e = $? >> 8; - if ($e > 1) { - die "$tmpfile: $!\n"; - } - unlink $tmpfile; -} -exit 0; ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/3] convert headers_install.pl to headers_install.sh 2013-02-27 5:58 ` [PATCH 3/3] convert headers_install.pl to headers_install.sh Rob Landley @ 2013-02-28 8:19 ` Michal Marek 2013-03-01 5:07 ` Rob Landley 2013-04-09 10:52 ` Michal Marek 0 siblings, 2 replies; 17+ messages in thread From: Michal Marek @ 2013-02-28 8:19 UTC (permalink / raw) To: Rob Landley Cc: linux-kbuild, linux-kernel, Thomas Gleixner, Josh Boyer, Paul E. McKenney, Andrew Morton, David Howells, Sam Ravnborg On 27.2.2013 06:58, Rob Landley wrote: > From: Rob Landley <rob@landley.net> > > Remove perl from make headers_install by replacing a perl script (doing > a simple regex search and replace) with a smaller, faster, simpler, > POSIX-2008 shell script implementation. The new shell script is a single > for loop calling sed and piping its output through unifdef to produce the > target file. > > Same as last time except for minor tweak to deal with code review from here: > http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00078.html > > (Note that this drops the "arch" argument, which isn't used. Kbuild > already points to the right input files on the command line.) > > Signed-off-by: Rob Landley <rob@landley.net> Looks good, I will apply it after v3.9-rc1 is out. Michal ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/3] convert headers_install.pl to headers_install.sh 2013-02-28 8:19 ` Michal Marek @ 2013-03-01 5:07 ` Rob Landley 2013-04-09 10:52 ` Michal Marek 1 sibling, 0 replies; 17+ messages in thread From: Rob Landley @ 2013-03-01 5:07 UTC (permalink / raw) To: Michal Marek Cc: linux-kbuild, linux-kernel, Thomas Gleixner, Josh Boyer, Paul E. McKenney, Andrew Morton, David Howells, Sam Ravnborg On 02/28/2013 02:19:24 AM, Michal Marek wrote: > On 27.2.2013 06:58, Rob Landley wrote: > > From: Rob Landley <rob@landley.net> > > > > Remove perl from make headers_install by replacing a perl script > (doing > > a simple regex search and replace) with a smaller, faster, simpler, > > POSIX-2008 shell script implementation. The new shell script is a > single > > for loop calling sed and piping its output through unifdef to > produce the > > target file. > > > > Same as last time except for minor tweak to deal with code review > from here: > > http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00078.html > > > > (Note that this drops the "arch" argument, which isn't used. Kbuild > > already points to the right input files on the command line.) > > > > Signed-off-by: Rob Landley <rob@landley.net> > > Looks good, I will apply it after v3.9-rc1 is out. Yay! Thank you! (Andrew took the other two! Thank you!) Rob ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/3] convert headers_install.pl to headers_install.sh 2013-02-28 8:19 ` Michal Marek 2013-03-01 5:07 ` Rob Landley @ 2013-04-09 10:52 ` Michal Marek 2013-04-11 16:25 ` Rob Landley 1 sibling, 1 reply; 17+ messages in thread From: Michal Marek @ 2013-04-09 10:52 UTC (permalink / raw) To: Rob Landley Cc: linux-kbuild, linux-kernel, Thomas Gleixner, Josh Boyer, Paul E. McKenney, Andrew Morton, David Howells, Sam Ravnborg On 28.2.2013 09:19, Michal Marek wrote: > On 27.2.2013 06:58, Rob Landley wrote: >> From: Rob Landley <rob@landley.net> >> >> Remove perl from make headers_install by replacing a perl script (doing >> a simple regex search and replace) with a smaller, faster, simpler, >> POSIX-2008 shell script implementation. The new shell script is a single >> for loop calling sed and piping its output through unifdef to produce the >> target file. >> >> Same as last time except for minor tweak to deal with code review from here: >> http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00078.html >> >> (Note that this drops the "arch" argument, which isn't used. Kbuild >> already points to the right input files on the command line.) >> >> Signed-off-by: Rob Landley <rob@landley.net> > > Looks good, I will apply it after v3.9-rc1 is out. I forgot to do it right after rc1, but I took the patch from the mm tree now. I also added Sam's Ack from the earlier submission. Michal ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/3] convert headers_install.pl to headers_install.sh 2013-04-09 10:52 ` Michal Marek @ 2013-04-11 16:25 ` Rob Landley 2013-04-11 19:20 ` Michal Marek 0 siblings, 1 reply; 17+ messages in thread From: Rob Landley @ 2013-04-11 16:25 UTC (permalink / raw) To: Michal Marek Cc: linux-kbuild, linux-kernel, Thomas Gleixner, Josh Boyer, Paul E. McKenney, Andrew Morton, David Howells, Sam Ravnborg On 04/09/2013 05:52:32 AM, Michal Marek wrote: > On 28.2.2013 09:19, Michal Marek wrote: > > On 27.2.2013 06:58, Rob Landley wrote: > >> From: Rob Landley <rob@landley.net> > >> > >> Remove perl from make headers_install by replacing a perl script > (doing > >> a simple regex search and replace) with a smaller, faster, simpler, > >> POSIX-2008 shell script implementation. The new shell script is a > single > >> for loop calling sed and piping its output through unifdef to > produce the > >> target file. > >> > >> Same as last time except for minor tweak to deal with code review > from here: > >> http://lkml.indiana.edu/hypermail/linux/kernel/1302.3/00078.html > >> > >> (Note that this drops the "arch" argument, which isn't used. Kbuild > >> already points to the right input files on the command line.) > >> > >> Signed-off-by: Rob Landley <rob@landley.net> > > > > Looks good, I will apply it after v3.9-rc1 is out. > > I forgot to do it right after rc1, but I took the patch from the mm > tree > now. I also added Sam's Ack from the earlier submission. > > Michal Yay! Is this for 3.10? (Not seeing it in Linus's tree now, and -rc7 seems a bit late for anything to go in...) Thanks, Rob ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/3] convert headers_install.pl to headers_install.sh 2013-04-11 16:25 ` Rob Landley @ 2013-04-11 19:20 ` Michal Marek 0 siblings, 0 replies; 17+ messages in thread From: Michal Marek @ 2013-04-11 19:20 UTC (permalink / raw) To: Rob Landley Cc: linux-kbuild, linux-kernel, Thomas Gleixner, Josh Boyer, Paul E. McKenney, Andrew Morton, David Howells, Sam Ravnborg Dne 11.4.2013 18:25, Rob Landley napsal(a): > On 04/09/2013 05:52:32 AM, Michal Marek wrote: >> I forgot to do it right after rc1, but I took the patch from the mm tree >> now. I also added Sam's Ack from the earlier submission. > > Yay! > > Is this for 3.10? Yes. The kbuild, kconfig, misc and trivial branches of kbuild.git contain changes for the next merge window (next -rc1). The rc-fixes branch is for the current rc phase. Michal ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/3] build linux-next without perl 2013-02-27 5:57 [PATCH 0/3] build linux-next without perl Rob Landley ` (2 preceding siblings ...) 2013-02-27 5:58 ` [PATCH 3/3] convert headers_install.pl to headers_install.sh Rob Landley @ 2013-02-27 21:51 ` Andrew Morton 2013-02-28 4:01 ` Rob Landley ` (3 more replies) 3 siblings, 4 replies; 17+ messages in thread From: Andrew Morton @ 2013-02-27 21:51 UTC (permalink / raw) To: Rob Landley Cc: linux-kernel, Al Viro, Arnd Bergmann, Cyrill Gorcunov, Dave Jones, David Howells, David Howells, Greg Kroah-Hartman, H. Peter Anvin, H. Peter Anvin, Ingo Molnar, Josh Boyer, Michal Marek, Paul E. McKenney, Peter Zijlstra, Rusty Russell, Sam Ravnborg, Thomas Gleixner, Vince Weaver, x86, Zheng Yan On Tue, 26 Feb 2013 21:57:52 -0800 (PST) Rob Landley <rob@landley.net> wrote: > Before 2.6.25 building Linux never required perl. This patch series removes > the requirement from basic kernel builds (tested on i686, x86_64, arm, mips, > powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1. > > Note, this removes perl from the _build_ environment, not from the _development_ > environment. This is approximately the same logic behind "make menuconfig" > requiring curses but "make oldconfig" not requiring curses. Including > zconf.lex.c_shipped in kconfig and then requiring perl makes no sense. > > ... > > Mostly people just copy the patches into their local projects (ala > https://github.com/rofl0r/sabotage/tree/master/KEEP ) but I'm reposting > them to linux-kernel after Gentoo considered using these patches, but didn't > because they weren't upstream: > https://bugs.gentoo.org/show_bug.cgi?id=421483 Sitting here scratching head wondering why you-need-perl is a problem for anyone. That gentoo bug report provides some explanation: "perl was removed from @system". But I expect other people have different reasons. IOW, please better describe the motivation for this patchset. It'll need to be reasonably good motivation, too. Because not only do we need to patch the kernel, we also need to *maintain* its perl-freeness and fix up perlisms as they later get added by others. (Perhaps one way of doing this would be to disable perl in regular builds, so even if a developer has perl installed on his machine, his build will still fail when he invokes it. Add "PERL=/dev/null" to some build targets in some manner.) ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/3] build linux-next without perl 2013-02-27 21:51 ` [PATCH 0/3] build linux-next without perl Andrew Morton @ 2013-02-28 4:01 ` Rob Landley 2013-02-28 4:48 ` Rob Landley ` (2 subsequent siblings) 3 siblings, 0 replies; 17+ messages in thread From: Rob Landley @ 2013-02-28 4:01 UTC (permalink / raw) To: Andrew Morton Cc: linux-kernel, Al Viro, Arnd Bergmann, Cyrill Gorcunov, Dave Jones, David Howells, David Howells, Greg Kroah-Hartman, H. Peter Anvin, H. Peter Anvin, Ingo Molnar, Josh Boyer, Michal Marek, Paul E. McKenney, Peter Zijlstra, Rusty Russell, Sam Ravnborg, Thomas Gleixner, Vince Weaver, x86, Zheng Yan, David Anders On 02/27/2013 03:51:55 PM, Andrew Morton wrote: > On Tue, 26 Feb 2013 21:57:52 -0800 (PST) > Rob Landley <rob@landley.net> wrote: > > > Before 2.6.25 building Linux never required perl. This patch series > removes > > the requirement from basic kernel builds (tested on i686, x86_64, > arm, mips, > > powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1. > > > > Note, this removes perl from the _build_ environment, not from the > _development_ > > environment. This is approximately the same logic behind "make > menuconfig" > > requiring curses but "make oldconfig" not requiring curses. > Including > > zconf.lex.c_shipped in kconfig and then requiring perl makes no > sense. > > > > ... > > > > Mostly people just copy the patches into their local projects (ala > > https://github.com/rofl0r/sabotage/tree/master/KEEP ) but I'm > reposting > > them to linux-kernel after Gentoo considered using these patches, > but didn't > > because they weren't upstream: > > https://bugs.gentoo.org/show_bug.cgi?id=421483 > > Sitting here scratching head wondering why you-need-perl is a problem > for anyone. I'm scratching my head that people basically keep doing the "you go girl!" thing at me about this patch series _off_ the list (even people I'd expect to see here, like https://twitter.com/jonmasters/status/301166688852901888 ) but this is something like the dozenth time I've posted it and nobody seems to notice. Oh well. Can we start with the fact it's a completely gratuitous build environment dependency, and the kernel has a history of removing those? (I mentioned two in the message you're replying to, ncurses and lex in oldconfig.) This isn't even a "workaround", this is an alternate implementation that is as simple or simpler. (Sam Ravnborg acked one of the scripts not because he cared about perl, but because it simplified the kernel build.) > That gentoo bug report provides some explanation: "perl was removed > from @system". But I expect other people have different reasons. Actually, removing perl from the build environment is common in cross compiling situations. Removing everything you _can_ from the build environment is normal when cross compiling. This is because cross compiling sucks: http://landley.net/writing/docs/cross-compiling.html Cross compiling has inherent combinatorial complexity. Native compiling build complexity is "number of packages times number of package versions", with an addendum that things like the compiler and libc count as packages with different versions. When cross compiling, you basically multiply the number of targets you're supporting times the number of package versions you're building times the number of hosts you're building from. I've installed distros I'd never even _heard_ of under kvm because some bug only happened there, but of course all the big ones break too: http://landley.net/hg/aboriginal/rev/1532 http://landley.net/hg/aboriginal/rev/1518 http://landley.net/hg/aboriginal/rev/1318 http://landley.net/hg/aboriginal/rev/1160 It's not just the combinatorial complexity, it's also less testing in general (most people natively compile), plus the entire configure step is wrong at the design level for cross compiling: it asks questions about the machine you're building on and applies those answers to the program you're building. When host and target aren't the same, this is at _best_ useless. So if you're cross compiling in any remotely portable way, you need an "airlock step", as described on pages 98-100 the slides for the old talk I gave at Ohio LinuxFest, Flourish, and Celf, which is apparently making the rounds again: https://twitter.com/solardiz/status/306575964064866305 It's the same general idea as Linux From Scratch chapter 5 (you populate a directory with just the binaries you need, and restrict the $PATH to that) but with a minimalist twist: everything you add is a sharp edge some package can catch on. If not now, then after the next version upgrade. And perl is a GIANT HAIRBALL of sharp edges in this regard. There is no perl standard, just a single perl implementation that may or may not have the whole of CPAN installed. (In fact the "canned values" logic in kernel/timeconst.pl uses a giant array of precomputed values because the installed perl may or may not have Math::BigInt might not be available on the target. Way back in 2008 I thought this meant we had to be able to run without that and the Math::BigInt stuff was just for regenerating the table, but Peter said https://lkml.org/lkml/2008/2/15/548 and didn't mind letting the user figure out what the dependencies were when the build broke. Now apply that to lots of other packages and guess why letting ./configure not find perl is appealing to cross compile environments.) I'm surprised perl doesn't get dinged more for the single implementation. All the shell scripts in the kernel are supposed to work with #!/bin/sh pointing to dash instead of bash, people freak when Microsoft Word or Excel are whatever some random program parses rather than an actual file format. But perl? Everybody remember when perl was going to be reimplemented on top of the "parrot" engine? (http://www.perlmonks.org/?node_id=272641) You know why that didn't happen? Because after several years of effort they couldn't quite make it work reliably. Getting a fresh from-scratch engine implementation to run the existing corpus of perl code turned out to be _really_hard_. Python's got http://wiki.python.org/moin/PythonImplementations and there's even an embedded implementation of php (http://ph7.symisc.net/) but perl is this one _specific_ giant hairball. If you have trouble getting that hairball to work on a new target? Tough. (When I did bootstrap work on Hexagon back in 2010, and built linux from scratch natively on the result, "will perl work" was one of the big unknowns. Luckily it only took about a week of poking and prodding to get it to build. Didn't particularly stress it to see how _well_ it worked, mostly because I'd carefully arranged the build to need it as little as possible. One of x11's dependencies needed it though, off in Beyond Linux From Scratch, and wouldn't ./configure it out the way libiconv and such did. Don't remember which one.) By the way, I'm not saying restricting the $PATH by itself is a _sufficient_ airlock step. When Wolfgang Denk (the u-boot maintainer) tried my cross compiling build environment it immediately broke in 3 different and strangely fascinating ways for him, one of which (http://landley.net/hg/aboriginal/rev/997) evolved into an entire environment variable whitelisting step (http://landley.net/hg/aboriginal/rev/1175) because once it works well enough more people try it and break it in new ways... And no, my build system isn't special, I just use it as an example because it's what I'm most familiar with. What an awful lot of distros do is set up a chroot and then "env -i chroot" into it (I.E. the Linux From Scratch approach). My build system goes to extra effort so that no part of it requires root access on the host (which is why I don't chroot, I run the target system under qemu instead, hence the title of the above giant slide deck from 2008). > IOW, please better describe the motivation for this patchset. You want more? Ok. (You asked.) In addition to all the above, last week I gave a talk at the Linux Foundation Embedded Linux Conference (used to be called CELF before they ate it) on turning Android into a real build environment. I didn't do slides this time but the outline's here: http://landley.net/talks/celf-2013.txt It would be really convenient if the video of that talk was up so I could just point you at it, but you'll have to ask the Linux Foundation when that'll be. The outline was just "notes to self" for me, lemme see if I can summarize an hour talk in a couple paragraphs. What I'm doing there is trying to expand Android into a full self-hosting development environment so it can get on with being a disruptive technology and kicking the PC up into the server space like the minicomputer and mainframe before it. I would _very_ much like Android to do this before iPhone does because when the S-curve of adoptions flattens out (somewhere between 1 and 3.5 billion unit installed base I'd guess) and the positive feedback loop of network effects kick in, being locked out of _another_ generation of technology by an actually COMPETENT monopolist would really suck. The hardware to use a smartphone as a workstation is just a USB hub with keyboard, mouse, and video adapter plugged into it; that's here today (although USB3 makes it easier). The rest is software. But there's a LOT of software. This software has 4 basic parts: kernel (which works now because they just added stuff to linux without removing anything), a command line (I'm writing a new BSD-licensed posix command line; same general reason I did years of work in busybox only this time it's old hat), a C library (musl-libc.org is the leading contender), and a toolchain (looks like llvm at the moment, I'd like to do http://landley.net/qcc but my plate's full and there's no time. Why no time? Who is sponsoring llvm? Who did "airplay" to put a phone display on an HDTV? You think Steve Jobs didn't _notice_ that 8->16->32->64 bits is sustaining technologies but mainframe->minicomputer->microcomputer->smartphone is disruptive yet _inevitable_?) This 4-package thing is even more simplified than my Aboriginal Linux build (which got it down to 7 packages, but the licensing of those is wrong -- no GPL in userspace -- so preinstalling any of it is a violation of the Android licensing guidelines and the trademark grows teeth so your ads have to be really horribly phrased.) The reason you _want_ to simplify it is that Google is shipping a billion unadministered unix systems with broadband access, which is TERRIFYING from a security standpoint. The reason bionic and toolbox are stubs even though uClibc and busybox both predate android isn't _just_ licensing issues, it's that Google intentionally shipped the minimum environment necessary to boot dalvik and get into the java sandbox, and is minimizing the attack surface if you can manage to escape that. But Dalvik is this generation's version of ROM Basic: it's something the platform has to outgrow in order to wean itself off of the previous generation it's cross-compiled from. Once the PC became a self-hosting development environment there was an explosion of software for it, because you no longer needed a PDP-10 to develop for the PC, having a PC was enough to be a developer. This is not currently the case for phones, but it should _become_ the case. If you then say "and to be a self-hosting system, you must add perl". And to preinstall perl, you must audit perl for security concerns... Can we please, please, please just remove the need for perl as part of a self-hosting development environment instead? P.S. These are just _my_ reasons. Dave Anders is the one who first complained to _me_ that perl had been added as a build requirement back in 2.6.25. I finally met him in person at the BeagleBone tutorial at CELF last week after knowing him for years on freenode (prplague), but like all the embedded guys he doesn't hang out here. (I just cc'd him.) Nor do the 3 of the 4 other people who congratulated me on getting this posted on freenode today (I convinced _one_ of them to come here and ack the darn patch). > It'll need to be reasonably good motivation, too. Because not only do > we need to patch the kernel, we also need to *maintain* its > perl-freeness and fix up perlisms as they later get added by others. I've already been maintaining it (and submitting it here) for 5 years now: https://lkml.org/lkml/2008/2/15/541 Sam Ravnborg acked the headers_install change not because of perl but because the replacement was simpler than what it replaced. It would be difficult to make the timeconst thing with the giant blob of pregenerated values _worse_. As for fixing up existing perlisms, people who use my scripts already send you patches to remove perl dependencies in things that I don't enable in my builds: http://lkml.indiana.edu/hypermail/linux/kernel/0910.0/01896.html Honestly, not the only one who does this. Sing along: http://barb.velvet.com/humor/lurkers.html (Which is really, really annoying. But that's embedded linux for you. They've all written upstream off as ignoring them. You think I'm typing an epic tl;dr at _you_ guys, it's _harder_ to get through the other way...) > (Perhaps one way of doing this would be to disable perl in regular > builds, so even if a developer has perl installed on his machine, his > build will still fail when he invokes it. Add "PERL=/dev/null" to > some > build targets in some manner.) http://landley.net/aboriginal/about.html (And yes, I need to get a release out that uses 3.8 but they screwed up interrupt routing on QEMU's arm versatile board emulation again and I haven't had time to track it down because I've been trying to get _this_ pushed upstream this merge window. Again. Plus I need to figure out what I broke in powerpc userspace, and run the automated Linux From Scratch build under qemu on all targets to make sure I haven't missed anything else...) Rob ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/3] build linux-next without perl 2013-02-27 21:51 ` [PATCH 0/3] build linux-next without perl Andrew Morton 2013-02-28 4:01 ` Rob Landley @ 2013-02-28 4:48 ` Rob Landley 2013-02-28 6:47 ` Geert Uytterhoeven 2013-02-28 8:44 ` Michal Marek 3 siblings, 0 replies; 17+ messages in thread From: Rob Landley @ 2013-02-28 4:48 UTC (permalink / raw) To: Andrew Morton Cc: linux-kernel, Al Viro, Arnd Bergmann, Cyrill Gorcunov, Dave Jones, David Howells, David Howells, Greg Kroah-Hartman, H. Peter Anvin, H. Peter Anvin, Ingo Molnar, Josh Boyer, Michal Marek, Paul E. McKenney, Peter Zijlstra, Rusty Russell, Sam Ravnborg, Thomas Gleixner, Vince Weaver, x86, Zheng Yan On 02/27/2013 03:51:55 PM, Andrew Morton wrote: > On Tue, 26 Feb 2013 21:57:52 -0800 (PST) > Rob Landley <rob@landley.net> wrote: > > > Before 2.6.25 building Linux never required perl. This patch series > removes > > the requirement from basic kernel builds (tested on i686, x86_64, > arm, mips, > > powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1. > > > > Note, this removes perl from the _build_ environment, not from the > _development_ > > environment. This is approximately the same logic behind "make > menuconfig" > > requiring curses but "make oldconfig" not requiring curses. > Including > > zconf.lex.c_shipped in kconfig and then requiring perl makes no > sense. > > > > ... > > > > Mostly people just copy the patches into their local projects (ala > > https://github.com/rofl0r/sabotage/tree/master/KEEP ) but I'm > reposting > > them to linux-kernel after Gentoo considered using these patches, > but didn't > > because they weren't upstream: > > https://bugs.gentoo.org/show_bug.cgi?id=421483 > > Sitting here scratching head wondering why you-need-perl is a problem > for anyone. > > That gentoo bug report provides some explanation: "perl was removed > from @system". But I expect other people have different reasons. > > IOW, please better describe the motivation for this patchset. P.S. A much shorter reply than mine, from David Anders: https://plus.google.com/101339419642360856354/posts/AyRnz2REq8C Rob ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/3] build linux-next without perl 2013-02-27 21:51 ` [PATCH 0/3] build linux-next without perl Andrew Morton 2013-02-28 4:01 ` Rob Landley 2013-02-28 4:48 ` Rob Landley @ 2013-02-28 6:47 ` Geert Uytterhoeven 2013-02-28 8:44 ` Michal Marek 3 siblings, 0 replies; 17+ messages in thread From: Geert Uytterhoeven @ 2013-02-28 6:47 UTC (permalink / raw) To: Andrew Morton Cc: Rob Landley, linux-kernel, Al Viro, Arnd Bergmann, Cyrill Gorcunov, Dave Jones, David Howells, David Howells, Greg Kroah-Hartman, H. Peter Anvin, H. Peter Anvin, Ingo Molnar, Josh Boyer, Michal Marek, Paul E. McKenney, Peter Zijlstra, Rusty Russell, Sam Ravnborg, Thomas Gleixner, Vince Weaver, x86, Zheng Yan On Wed, Feb 27, 2013 at 10:51 PM, Andrew Morton <akpm@linux-foundation.org> wrote: > It'll need to be reasonably good motivation, too. Because not only do > we need to patch the kernel, we also need to *maintain* its > perl-freeness and fix up perlisms as they later get added by others. That's actually a backwards argument: as long as there are scripts in the kernel source tree using perl, people will add more of them, since they see perl is an accepted scripting language within the Linux kernel build system. It's the same as refactoring existing code: if we don't, people will copy the bad examples. So we fix them, to avoid this. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/3] build linux-next without perl 2013-02-27 21:51 ` [PATCH 0/3] build linux-next without perl Andrew Morton ` (2 preceding siblings ...) 2013-02-28 6:47 ` Geert Uytterhoeven @ 2013-02-28 8:44 ` Michal Marek 3 siblings, 0 replies; 17+ messages in thread From: Michal Marek @ 2013-02-28 8:44 UTC (permalink / raw) To: Andrew Morton Cc: Rob Landley, linux-kernel, Al Viro, Arnd Bergmann, Cyrill Gorcunov, Dave Jones, David Howells, David Howells, Greg Kroah-Hartman, H. Peter Anvin, H. Peter Anvin, Ingo Molnar, Josh Boyer, Paul E. McKenney, Peter Zijlstra, Rusty Russell, Sam Ravnborg, Thomas Gleixner, Vince Weaver, x86, Zheng Yan On 27.2.2013 22:51, Andrew Morton wrote: > It'll need to be reasonably good motivation, too. Because not only do > we need to patch the kernel, we also need to *maintain* its > perl-freeness and fix up perlisms as they later get added by others. > > (Perhaps one way of doing this would be to disable perl in regular > builds, so even if a developer has perl installed on his machine, his > build will still fail when he invokes it. Add "PERL=/dev/null" to some > build targets in some manner.) I don't think we need to go this far. Apparently, there are people who want to be able to build the kernel without perl, so let's rely on them to report if perl appears again as a hard build dependency. If these people lose interest, then after some time we might end up with perl being required again, but then this by definition won't be a problem for anyone who cares. It is like bashisms in shell scripts and Makefile commands. They should be avoided, but it wasn't until Debian and Ubuntu switched to /bin/sh -> dash when we started to fix them. Nowadays, the kernel should build file with a minimal POSIX shell, but we did not need any SHELL=$(objtree)/scripts/dash to achieve this. Michal ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/3] build linux-next without perl @ 2013-02-27 16:55 John Spencer 2013-02-27 18:53 ` Sam Ravnborg 0 siblings, 1 reply; 17+ messages in thread From: John Spencer @ 2013-02-27 16:55 UTC (permalink / raw) To: linux-kernel; +Cc: Rob Landley Rob Landley wrote: > Before 2.6.25 building Linux never required perl. This patch series removes > the requirement from basic kernel builds (tested on i686, x86_64, arm, mips, > powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1. using this patch series since 2.6.32 in sabotage linux, it is highly useful and it's about time to get it merged. tested against linux 3.8, works perfectly. Acked-By: John Spencer <maillist-linux@barfooze.de> ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/3] build linux-next without perl 2013-02-27 16:55 John Spencer @ 2013-02-27 18:53 ` Sam Ravnborg 2013-02-27 19:50 ` John Spencer 0 siblings, 1 reply; 17+ messages in thread From: Sam Ravnborg @ 2013-02-27 18:53 UTC (permalink / raw) To: John Spencer; +Cc: linux-kernel, Rob Landley On Wed, Feb 27, 2013 at 05:55:31PM +0100, John Spencer wrote: > Rob Landley wrote: >> Before 2.6.25 building Linux never required perl. This patch series > removes >> the requirement from basic kernel builds (tested on i686, x86_64, arm, >> > mips, >> powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1. > > using this patch series since 2.6.32 in sabotage linux, it is highly > useful and it's about time to get it merged. > tested against linux 3.8, works perfectly. Why do you apply this patch serie to sabotage linux? Sam ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/3] build linux-next without perl 2013-02-27 18:53 ` Sam Ravnborg @ 2013-02-27 19:50 ` John Spencer 0 siblings, 0 replies; 17+ messages in thread From: John Spencer @ 2013-02-27 19:50 UTC (permalink / raw) To: Sam Ravnborg; +Cc: linux-kernel, Rob Landley On 02/27/2013 07:53 PM, Sam Ravnborg wrote: > On Wed, Feb 27, 2013 at 05:55:31PM +0100, John Spencer wrote: >> Rob Landley wrote: >>> Before 2.6.25 building Linux never required perl. This patch series >> removes >>> the requirement from basic kernel builds (tested on i686, x86_64, arm, >>> >> mips, >>> powerpc, sparc, sh4, and m68k). Now updated to 3.8-rc1. >> >> using this patch series since 2.6.32 in sabotage linux, it is highly >> useful and it's about time to get it merged. >> tested against linux 3.8, works perfectly. > > Why do you apply this patch serie to sabotage linux? > > Sam > it allows me to build the kernel as soon as the minimal base system is bootstrapped. building perl takes about as long as the whole base system (busybox,gcc,binutils,make) together, so having that dependency removed is a huge relief. btw, bootstrap-linux [1] uses this patch series as well. [1] https://github.com/pikhq/bootstrap-linux ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2013-04-11 19:20 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-02-27 5:57 [PATCH 0/3] build linux-next without perl Rob Landley 2013-02-27 5:57 ` [PATCH 1/3] convert mkcapflags.pl to mkcapflags.sh Rob Landley 2013-02-27 5:58 ` [PATCH 2/3] convert timeconst.pl to mktimeconst.c Rob Landley 2013-02-27 5:58 ` [PATCH 3/3] convert headers_install.pl to headers_install.sh Rob Landley 2013-02-28 8:19 ` Michal Marek 2013-03-01 5:07 ` Rob Landley 2013-04-09 10:52 ` Michal Marek 2013-04-11 16:25 ` Rob Landley 2013-04-11 19:20 ` Michal Marek 2013-02-27 21:51 ` [PATCH 0/3] build linux-next without perl Andrew Morton 2013-02-28 4:01 ` Rob Landley 2013-02-28 4:48 ` Rob Landley 2013-02-28 6:47 ` Geert Uytterhoeven 2013-02-28 8:44 ` Michal Marek 2013-02-27 16:55 John Spencer 2013-02-27 18:53 ` Sam Ravnborg 2013-02-27 19:50 ` John Spencer
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).