* PATCH [0/3]: Simplify the kernel build by removing perl. @ 2009-01-02 8:07 Rob Landley 2009-01-02 8:13 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley ` (6 more replies) 0 siblings, 7 replies; 135+ messages in thread From: Rob Landley @ 2009-01-02 8:07 UTC (permalink / raw) To: Embedded Linux mailing list Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kernel never required perl to be installed on the build system. (Various development and debugging scripts were written in perl and python and such, but they weren't involved in actually building a kernel.) Building a kernel before 2.6.25 could be done with a minimal system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing else. (Embedded developers creating clean cross compile environments that won't leak bits of the host system into the target, or bootstrapping native development environments to run on target hardware or under emulators, tend to care about this sort of thing.) The perl checkin for 2.6.25 was the camel's nose under the tent flap, and since then two more instances of perl have shown up in the core kernel build. This patch series removes the three required to build a basic kernel for qemu for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4, and of course x86 and x86-64), replacing them with shell scripts. Historically the kernel has gone out of its way to minimize environmental dependencies for the build. For example, the plethora of *_shipped files mean we don't need lex and yacc to build the kernel. The kconfig infrastructure once required curses to run "make oldconfig", but that requirement was restricted to just menuconfig so the kernel could build on systems without ncurses. That was a very nice policy. Kernel development already requires an in-depth knowledge of C, make, and shell, plus things like the kconfig language. A similarly in-depth knowledge of perl is bigger than all that combined (even Larry Wall probably doesn't know all of Perl anymore), and then what's the excuse _not_ to add Python to the core build? And after that why not java, or lua? Where does it end? What's the criteria to say "no" here? This patch series saves time and says no now. ^ permalink raw reply [flat|nested] 135+ messages in thread
* [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-02 8:07 PATCH [0/3]: Simplify the kernel build by removing perl Rob Landley @ 2009-01-02 8:13 ` Rob Landley 2009-01-02 9:04 ` Sam Ravnborg ` (2 more replies) 2009-01-02 8:14 ` [PATCH 2/3]: Remove perl from make headers_install Rob Landley ` (5 subsequent siblings) 6 siblings, 3 replies; 135+ messages in thread From: Rob Landley @ 2009-01-02 8:13 UTC (permalink / raw) To: Embedded Linux mailing list Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg From: Rob Landley <rob@landley.net> Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell scriptis much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003. Peter Anvin added this perl to 2.6.25. Before that, the kernel had neverrequired perl to build. Signed-off-by: Rob Landley <rob@landley.net>--- kernel/Makefile | 4 kernel/timeconst.pl | 378 ------------------------------------------ kernel/timeconst.sh | 91 ++++++++++ 3 files changed, 93 insertions(+), 380 deletions(-) diff -r d0c7611dcfd6 kernel/Makefile--- a/kernel/Makefile Tue Dec 30 17:48:25 2008 -0800+++ b/kernel/Makefile Fri Jan 02 00:10:44 2009 -0600@@ -115,7 +115,7 @@ $(obj)/time.o: $(obj)/timeconst.h quiet_cmd_timeconst = TIMEC $@- cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@+ cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) > $@ targets += timeconst.h-$(obj)/timeconst.h: $(src)/timeconst.pl FORCE+$(obj)/timeconst.h: $(src)/timeconst.sh FORCE $(call if_changed,timeconst)--- /dev/null 1969-12-31 00:00:00.000000000 -0600+++ hg/kernel/timeconst.sh 2009-01-01 23:53:17.000000000 -0600@@ -0,0 +1,91 @@+#!/bin/bash++if [ $# -ne 1 ]+then+ echo "Usage: timeconst.sh HZ"+ exit 1+fi++HZ=$1++# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math.++[ $((1<<32)) -lt 0 ] && exit 1++# Output start of header file++cat << EOF+/* Automatically generated by kernel/timeconst.sh */+/* Conversion constants for HZ == $HZ */++#ifndef KERNEL_TIMECONST_H+#define KERNEL_TIMECONST_H++#include <linux/param.h>+#include <linux/types.h>++#if HZ != $HZ+#error "kernel/timeconst.h has the wrong HZ value!"+#endif++EOF++# For both Miliseconds and Microseconds++for i in "MSEC 1000" "USEC 1000000"+do+ NAME=$(echo $i | awk '{print $1}')+ PERIOD=$(echo $i | awk '{print $2}')++ # Find greatest common denominator (using Euclid's algorithm)++ A=$HZ+ B=$PERIOD++ while [ $B -ne 0 ]+ do+ C=$(($A%$B))+ A=$B+ B=$C+ done++ GCD=$A++ # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ)++ for DIRECTION in 0 1+ do+ if [ $DIRECTION -eq 0 ]+ then+ CONVERT="HZ_TO_${NAME}"+ FROM=$HZ+ TO=$PERIOD+ else+ CONVERT="${NAME}_TO_HZ"+ FROM=$PERIOD+ TO=$HZ+ fi++ # How many shift bits give 32 bits of significant data?++ SHIFT=0+ while [ $(( (($TO<<$SHIFT)+$FROM-1)/$FROM )) -lt $((1<<31)) ]+ do+ SHIFT=$(( $SHIFT+1 ))+ done++ MUL32=$(( (($TO<<$SHIFT)+$FROM-1)/$FROM ))+ MUL32=$(printf %x $MUL32)+ echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)"+ ADJ32=$(($FROM/$GCD))+ ADJ32=$(( (($ADJ32-1)<<$SHIFT)/$ADJ32 ))+ ADJ32=$(printf %x $ADJ32)+ echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)"+ echo "#define ${CONVERT}_SHR32 $SHIFT"+ echo "#define ${CONVERT}_NUM U64_C($(($TO/$GCD)))"+ echo "#define ${CONVERT}_DEN U64_C($(($FROM/$GCD)))"+ done+done++echo+echo "#endif /* KERNEL_TIMECHONST_H */"--- hg/kernel/timeconst.pl 2008-11-22 19:09:13.000000000 -0600+++ /dev/null 1969-12-31 00:00:00.000000000 -0600@@ -1,378 +0,0 @@-#!/usr/bin/perl-# ------------------------------------------------------------------------#-# Copyright 2007-2008 rPath, Inc. - All Rights Reserved-#-# This file is part of the Linux kernel, and is made available under-# the terms of the GNU General Public License version 2 or (at your-# option) any later version; incorporated herein by reference.-#-# ------------------------------------------------------------------------#--#-# Usage: timeconst.pl HZ > timeconst.h-#--# Precomputed values for systems without Math::BigInt-# Generated by:-# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 1200-%canned_values = (- 24 => [- '0xa6aaaaab','0x2aaaaaa',26,- 125,3,- '0xc49ba5e4','0x1fbe76c8b4',37,- 3,125,- '0xa2c2aaab','0xaaaa',16,- 125000,3,- '0xc9539b89','0x7fffbce4217d',47,- 3,125000,- ], 32 => [- '0xfa000000','0x6000000',27,- 125,4,- '0x83126e98','0xfdf3b645a',36,- 4,125,- '0xf4240000','0x0',17,- 31250,1,- '0x8637bd06','0x3fff79c842fa',46,- 1,31250,- ], 48 => [- '0xa6aaaaab','0x6aaaaaa',27,- 125,6,- '0xc49ba5e4','0xfdf3b645a',36,- 6,125,- '0xa2c2aaab','0x15555',17,- 62500,3,- '0xc9539b89','0x3fffbce4217d',46,- 3,62500,- ], 64 => [- '0xfa000000','0xe000000',28,- 125,8,- '0x83126e98','0x7ef9db22d',35,- 8,125,- '0xf4240000','0x0',18,- 15625,1,- '0x8637bd06','0x1fff79c842fa',45,- 1,15625,- ], 100 => [- '0xa0000000','0x0',28,- 10,1,- '0xcccccccd','0x733333333',35,- 1,10,- '0x9c400000','0x0',18,- 10000,1,- '0xd1b71759','0x1fff2e48e8a7',45,- 1,10000,- ], 122 => [- '0x8325c53f','0xfbcda3a',28,- 500,61,- '0xf9db22d1','0x7fbe76c8b',35,- 61,500,- '0x8012e2a0','0x3ef36',18,- 500000,61,- '0xffda4053','0x1ffffbce4217',45,- 61,500000,- ], 128 => [- '0xfa000000','0x1e000000',29,- 125,16,- '0x83126e98','0x3f7ced916',34,- 16,125,- '0xf4240000','0x40000',19,- 15625,2,- '0x8637bd06','0xfffbce4217d',44,- 2,15625,- ], 200 => [- '0xa0000000','0x0',29,- 5,1,- '0xcccccccd','0x333333333',34,- 1,5,- '0x9c400000','0x0',19,- 5000,1,- '0xd1b71759','0xfff2e48e8a7',44,- 1,5000,- ], 250 => [- '0x80000000','0x0',29,- 4,1,- '0x80000000','0x180000000',33,- 1,4,- '0xfa000000','0x0',20,- 4000,1,- '0x83126e98','0x7ff7ced9168',43,- 1,4000,- ], 256 => [- '0xfa000000','0x3e000000',30,- 125,32,- '0x83126e98','0x1fbe76c8b',33,- 32,125,- '0xf4240000','0xc0000',20,- 15625,4,- '0x8637bd06','0x7ffde7210be',43,- 4,15625,- ], 300 => [- '0xd5555556','0x2aaaaaaa',30,- 10,3,- '0x9999999a','0x1cccccccc',33,- 3,10,- '0xd0555556','0xaaaaa',20,- 10000,3,- '0x9d495183','0x7ffcb923a29',43,- 3,10000,- ], 512 => [- '0xfa000000','0x7e000000',31,- 125,64,- '0x83126e98','0xfdf3b645',32,- 64,125,- '0xf4240000','0x1c0000',21,- 15625,8,- '0x8637bd06','0x3ffef39085f',42,- 8,15625,- ], 1000 => [- '0x80000000','0x0',31,- 1,1,- '0x80000000','0x0',31,- 1,1,- '0xfa000000','0x0',22,- 1000,1,- '0x83126e98','0x1ff7ced9168',41,- 1,1000,- ], 1024 => [- '0xfa000000','0xfe000000',32,- 125,128,- '0x83126e98','0x7ef9db22',31,- 128,125,- '0xf4240000','0x3c0000',22,- 15625,16,- '0x8637bd06','0x1fff79c842f',41,- 16,15625,- ], 1200 => [- '0xd5555556','0xd5555555',32,- 5,6,- '0x9999999a','0x66666666',31,- 6,5,- '0xd0555556','0x2aaaaa',22,- 2500,3,- '0x9d495183','0x1ffcb923a29',41,- 3,2500,- ]-);--$has_bigint = eval 'use Math::BigInt qw(bgcd); 1;';--sub bint($)-{- my($x) = @_;- return Math::BigInt->new($x);-}--#-# Constants for division by reciprocal multiplication.-# (bits, numerator, denominator)-#-sub fmul($$$)-{- my ($b,$n,$d) = @_;-- $n = bint($n);- $d = bint($d);-- return scalar (($n << $b)+$d-bint(1))/$d;-}--sub fadj($$$)-{- my($b,$n,$d) = @_;-- $n = bint($n);- $d = bint($d);-- $d = $d/bgcd($n, $d);- return scalar (($d-bint(1)) << $b)/$d;-}--sub fmuls($$$) {- my($b,$n,$d) = @_;- my($s,$m);- my($thres) = bint(1) << ($b-1);-- $n = bint($n);- $d = bint($d);-- for ($s = 0; 1; $s++) {- $m = fmul($s,$n,$d);- return $s if ($m >= $thres);- }- return 0;-}--# Generate a hex value if the result fits in 64 bits;-# otherwise skip.-sub bignum_hex($) {- my($x) = @_;- my $s = $x->as_hex();-- return (length($s) > 18) ? undef : $s;-}--# Provides mul, adj, and shr factors for a specific-# (bit, time, hz) combination-sub muladj($$$) {- my($b, $t, $hz) = @_;- my $s = fmuls($b, $t, $hz);- my $m = fmul($s, $t, $hz);- my $a = fadj($s, $t, $hz);- return (bignum_hex($m), bignum_hex($a), $s);-}--# Provides numerator, denominator values-sub numden($$) {- my($n, $d) = @_;- my $g = bgcd($n, $d);- return ($n/$g, $d/$g);-}--# All values for a specific (time, hz) combo-sub conversions($$) {- my ($t, $hz) = @_;- my @val = ();-- # HZ_TO_xx- push(@val, muladj(32, $t, $hz));- push(@val, numden($t, $hz));-- # xx_TO_HZ- push(@val, muladj(32, $hz, $t));- push(@val, numden($hz, $t));-- return @val;-}--sub compute_values($) {- my($hz) = @_;- my @val = ();- my $s, $m, $a, $g;-- if (!$has_bigint) {- die "$0: HZ == $hz not canned and ".- "Math::BigInt not available\n";- }-- # MSEC conversions- push(@val, conversions(1000, $hz));-- # USEC conversions- push(@val, conversions(1000000, $hz));-- return @val;-}--sub outputval($$)-{- my($name, $val) = @_;- my $csuf;-- if (defined($val)) {- if ($name !~ /SHR/) {- $val = "U64_C($val)";- }- printf "#define %-23s %s\n", $name.$csuf, $val.$csuf;- }-}--sub output($@)-{- my($hz, @val) = @_;- my $pfx, $bit, $suf, $s, $m, $a;-- print "/* Automatically generated by kernel/timeconst.pl */\n";- print "/* Conversion constants for HZ == $hz */\n";- print "\n";- print "#ifndef KERNEL_TIMECONST_H\n";- print "#define KERNEL_TIMECONST_H\n";- print "\n";-- print "#include <linux/param.h>\n";- print "#include <linux/types.h>\n";-- print "\n";- print "#if HZ != $hz\n";- print "#error \"kernel/timeconst.h has the wrong HZ value!\"\n";- print "#endif\n";- print "\n";-- foreach $pfx ('HZ_TO_MSEC','MSEC_TO_HZ',- 'HZ_TO_USEC','USEC_TO_HZ') {- foreach $bit (32) {- foreach $suf ('MUL', 'ADJ', 'SHR') {- outputval("${pfx}_$suf$bit", shift(@val));- }- }- foreach $suf ('NUM', 'DEN') {- outputval("${pfx}_$suf", shift(@val));- }- }-- print "\n";- print "#endif /* KERNEL_TIMECONST_H */\n";-}--# Pretty-print Perl values-sub perlvals(@) {- my $v;- my @l = ();-- foreach $v (@_) {- if (!defined($v)) {- push(@l, 'undef');- } elsif ($v =~ /^0x/) {- push(@l, "\'".$v."\'");- } else {- push(@l, $v.'');- }- }- return join(',', @l);-}--($hz) = @ARGV;--# Use this to generate the %canned_values structure-if ($hz eq '--can') {- shift(@ARGV);- @hzlist = sort {$a <=> $b} (@ARGV);-- print "# Precomputed values for systems without Math::BigInt\n";- print "# Generated by:\n";- print "# timeconst.pl --can ", join(' ', @hzlist), "\n";- print "\%canned_values = (\n";- my $pf = "\t";- foreach $hz (@hzlist) {- my @values = compute_values($hz);- print "$pf$hz => [\n";- while (scalar(@values)) {- my $bit;- foreach $bit (32) {- my $m = shift(@values);- my $a = shift(@values);- my $s = shift(@values);- print "\t\t", perlvals($m,$a,$s), ",\n";- }- my $n = shift(@values);- my $d = shift(@values);- print "\t\t", perlvals($n,$d), ",\n";- }- print "\t]";- $pf = ', ';- }- print "\n);\n";-} else {- $hz += 0; # Force to number- if ($hz < 1) {- die "Usage: $0 HZ\n";- }-- @val = @{$canned_values{$hz}};- if (!defined(@val)) {- @val = compute_values($hz);- }- output($hz, @val);-}-exit 0;\0ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥ ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-02 8:13 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley @ 2009-01-02 9:04 ` Sam Ravnborg 2009-01-02 12:00 ` Rob Landley 2009-01-03 12:28 ` Ingo Oeser 2009-01-05 0:41 ` Ray Lee 2 siblings, 1 reply; 135+ messages in thread From: Sam Ravnborg @ 2009-01-02 9:04 UTC (permalink / raw) To: Rob Landley Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin Hi Rob. On Fri, Jan 02, 2009 at 02:13:30AM -0600, Rob Landley wrote: > From: Rob Landley <rob@landley.net> > > Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script > is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003. This part of the changelog is OK except that is fails to address why we want to get away from perl. Please drop the remining part of the changelog (but not the s-o-b). > Peter Anvin added this perl to 2.6.25. Before that, the kernel had never > required perl to build. > > Signed-off-by: Rob Landley <rob@landley.net> > --- > > kernel/Makefile | 4 > kernel/timeconst.pl | 378 ------------------------------------------ > kernel/timeconst.sh | 91 ++++++++++ > 3 files changed, 93 insertions(+), 380 deletions(-) > > diff -r d0c7611dcfd6 kernel/Makefile > --- a/kernel/Makefile Tue Dec 30 17:48:25 2008 -0800 > +++ b/kernel/Makefile Fri Jan 02 00:10:44 2009 -0600 > @@ -115,7 +115,7 @@ > $(obj)/time.o: $(obj)/timeconst.h > > quiet_cmd_timeconst = TIMEC $@ > - cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@ > + cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) > $@ > targets += timeconst.h > -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE > +$(obj)/timeconst.h: $(src)/timeconst.sh FORCE > $(call if_changed,timeconst) OK > --- /dev/null 1969-12-31 00:00:00.000000000 -0600 > +++ hg/kernel/timeconst.sh 2009-01-01 23:53:17.000000000 -0600 > @@ -0,0 +1,91 @@ > +#!/bin/bash > + > +if [ $# -ne 1 ] > +then > + echo "Usage: timeconst.sh HZ" > + exit 1 > +fi That usage is useless. Either extend it or spend a few lines in the shell script explainign what the shell script is supposed to do. > + > +HZ=$1 > + > +# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math. > + > +[ $((1<<32)) -lt 0 ] && exit 1 > + If it fails then add an error message explaining why. So if we get reports that this fails then we at least can see something like: "timeconst noticed that the shell did not support 64 bit math - stop" > +# Output start of header file > + > +cat << EOF > +/* Automatically generated by kernel/timeconst.sh */ > +/* Conversion constants for HZ == $HZ */ > + > +#ifndef KERNEL_TIMECONST_H > +#define KERNEL_TIMECONST_H Please use __KERNEL_TIMECONST_H. The two underscores are standard. > + > +#include <linux/param.h> > +#include <linux/types.h> > + > +#if HZ != $HZ > +#error "kernel/timeconst.h has the wrong HZ value!" > +#endif > + > +EOF > + > +# For both Miliseconds and Microseconds > + > +for i in "MSEC 1000" "USEC 1000000" > +do > + NAME=$(echo $i | awk '{print $1}') > + PERIOD=$(echo $i | awk '{print $2}') > + > + # Find greatest common denominator (using Euclid's algorithm) > + > + A=$HZ > + B=$PERIOD > + > + while [ $B -ne 0 ] > + do > + C=$(($A%$B)) > + A=$B > + B=$C > + done > + > + GCD=$A > + > + # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ) > + > + for DIRECTION in 0 1 > + do > + if [ $DIRECTION -eq 0 ] > + then > + CONVERT="HZ_TO_${NAME}" > + FROM=$HZ > + TO=$PERIOD > + else > + CONVERT="${NAME}_TO_HZ" > + FROM=$PERIOD > + TO=$HZ > + fi > + > + # How many shift bits give 32 bits of significant data? > + > + SHIFT=0 > + while [ $(( (($TO<<$SHIFT)+$FROM-1)/$FROM )) -lt $((1<<31)) ] > + do > + SHIFT=$(( $SHIFT+1 )) > + done > + > + MUL32=$(( (($TO<<$SHIFT)+$FROM-1)/$FROM )) > + MUL32=$(printf %x $MUL32) > + echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)" > + ADJ32=$(($FROM/$GCD)) > + ADJ32=$(( (($ADJ32-1)<<$SHIFT)/$ADJ32 )) > + ADJ32=$(printf %x $ADJ32) > + echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)" > + echo "#define ${CONVERT}_SHR32 $SHIFT" > + echo "#define ${CONVERT}_NUM U64_C($(($TO/$GCD)))" > + echo "#define ${CONVERT}_DEN U64_C($(($FROM/$GCD)))" > + done > +done Is it a shell limitation that all spaces around operators are missing? Makes it hard to read.. You should use trap in your shell script so we do not end up with a partially generated file. Or you should change the rule in the Makefile to use a temperary file and then rename it. We have similar bugs in other places - but no need to add in more places. Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-02 9:04 ` Sam Ravnborg @ 2009-01-02 12:00 ` Rob Landley 2009-01-02 19:33 ` H. Peter Anvin 2009-01-03 6:28 ` Harvey Harrison 0 siblings, 2 replies; 135+ messages in thread From: Rob Landley @ 2009-01-02 12:00 UTC (permalink / raw) To: Sam Ravnborg Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin On Friday 02 January 2009 03:04:39 Sam Ravnborg wrote: > Hi Rob. > > On Fri, Jan 02, 2009 at 02:13:30AM -0600, Rob Landley wrote: > > From: Rob Landley <rob@landley.net> > > > > Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell > > script is much simpler, about 1/4 the size, and runs on Red Hat 9 from > > 2003. > > This part of the changelog is OK except that is fails to > address why we want to get away from perl. You mean "The new shell script is much simpler, about 1/4 the size, runs on Red Hat 9 from 2003, and isn't perl?" :) > Please drop the remining part of the changelog (but not the s-o-b). ok. > > Peter Anvin added this perl to 2.6.25. Before that, the kernel had never > > required perl to build. > > > > Signed-off-by: Rob Landley <rob@landley.net> > > --- > > > > kernel/Makefile | 4 > > kernel/timeconst.pl | 378 ------------------------------------------ > > kernel/timeconst.sh | 91 ++++++++++ > > 3 files changed, 93 insertions(+), 380 deletions(-) > > > > diff -r d0c7611dcfd6 kernel/Makefile > > --- a/kernel/Makefile Tue Dec 30 17:48:25 2008 -0800 > > +++ b/kernel/Makefile Fri Jan 02 00:10:44 2009 -0600 > > @@ -115,7 +115,7 @@ > > $(obj)/time.o: $(obj)/timeconst.h > > > > quiet_cmd_timeconst = TIMEC $@ > > - cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@ > > + cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) > $@ > > targets += timeconst.h > > -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE > > +$(obj)/timeconst.h: $(src)/timeconst.sh FORCE > > $(call if_changed,timeconst) > > OK > > > --- /dev/null 1969-12-31 00:00:00.000000000 -0600 > > +++ hg/kernel/timeconst.sh 2009-01-01 23:53:17.000000000 -0600 > > @@ -0,0 +1,91 @@ > > +#!/bin/bash > > + > > +if [ $# -ne 1 ] > > +then > > + echo "Usage: timeconst.sh HZ" > > + exit 1 > > +fi > > That usage is useless. Either extend it or spend a few lines > in the shell script explainign what the shell script is supposed to do. Do you mean something more like: echo "Usage: timeconst.sh HZ" echo echo "Generates a header file of constants for converting between decimal" echo "HZ timer ticks and milisecond or microsecond delays" I'm happy turning it into a comment instead, I just found a quick check that I'd remembered to type an argument useful during debugging. :) > > + > > +HZ=$1 > > + > > +# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit > > math. + > > +[ $((1<<32)) -lt 0 ] && exit 1 > > + > > If it fails then add an error message explaining why. So if we get reports > that this fails then we at least can see something like: > "timeconst noticed that the shell did not support 64 bit math - stop" Ok. > > +# Output start of header file > > + > > +cat << EOF > > +/* Automatically generated by kernel/timeconst.sh */ > > +/* Conversion constants for HZ == $HZ */ > > + > > +#ifndef KERNEL_TIMECONST_H > > +#define KERNEL_TIMECONST_H > > Please use __KERNEL_TIMECONST_H. The two underscores are standard. Sure thing. (I was just copying the perl there, I'll post an updated patch after I get some sleep.) > > + > > +#include <linux/param.h> > > +#include <linux/types.h> > > + > > +#if HZ != $HZ > > +#error "kernel/timeconst.h has the wrong HZ value!" > > +#endif > > + > > +EOF > > + > > +# For both Miliseconds and Microseconds > > + > > +for i in "MSEC 1000" "USEC 1000000" > > +do > > + NAME=$(echo $i | awk '{print $1}') > > + PERIOD=$(echo $i | awk '{print $2}') > > + > > + # Find greatest common denominator (using Euclid's algorithm) > > + > > + A=$HZ > > + B=$PERIOD > > + > > + while [ $B -ne 0 ] > > + do > > + C=$(($A%$B)) > > + A=$B > > + B=$C > > + done > > + > > + GCD=$A > > + > > + # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ) > > + > > + for DIRECTION in 0 1 > > + do > > + if [ $DIRECTION -eq 0 ] > > + then > > + CONVERT="HZ_TO_${NAME}" > > + FROM=$HZ > > + TO=$PERIOD > > + else > > + CONVERT="${NAME}_TO_HZ" > > + FROM=$PERIOD > > + TO=$HZ > > + fi > > + > > + # How many shift bits give 32 bits of significant data? > > + > > + SHIFT=0 > > + while [ $(( (($TO<<$SHIFT)+$FROM-1)/$FROM )) -lt $((1<<31)) ] > > + do > > + SHIFT=$(( $SHIFT+1 )) > > + done > > + > > + MUL32=$(( (($TO<<$SHIFT)+$FROM-1)/$FROM )) > > + MUL32=$(printf %x $MUL32) > > + echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)" > > + ADJ32=$(($FROM/$GCD)) > > + ADJ32=$(( (($ADJ32-1)<<$SHIFT)/$ADJ32 )) > > + ADJ32=$(printf %x $ADJ32) > > + echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)" > > + echo "#define ${CONVERT}_SHR32 $SHIFT" > > + echo "#define ${CONVERT}_NUM U64_C($(($TO/$GCD)))" > > + echo "#define ${CONVERT}_DEN U64_C($(($FROM/$GCD)))" > > + done > > +done > > Is it a shell limitation that all spaces around operators are missing? > Makes it hard to read.. No, I was just trying to make sure I didn't go over the 80 char limit. (Several temporary variables are there primarily because the style guide says indentation should be via 8 space tabs.) I'll add some more spaces on the respin. > > You should use trap in your shell script so we do not end up with a > partially generated file. > Or you should change the rule in the Makefile to use > a temperary file and then rename it. > > We have similar bugs in other places - but no need to add > in more places. Well, preserve in this case. :) I can add an "EXIT" trap and then disarm it later, but the output is redirected in the makefile, so generating an error won't remove the output if you run again. Perhaps the file to create should be fed in as a second argument to the script, so the trap can "rm" the output file if it triggers? I'll poke at it in the morning. Thanks for the feedback, Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-02 12:00 ` Rob Landley @ 2009-01-02 19:33 ` H. Peter Anvin 2009-01-04 1:32 ` Rob Landley 2009-01-03 6:28 ` Harvey Harrison 1 sibling, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-01-02 19:33 UTC (permalink / raw) To: Rob Landley Cc: Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton Rob Landley wrote: > > You mean "The new shell script is much simpler, about 1/4 the size, runs on > Red Hat 9 from 2003, and isn't perl?" :) > And introduces unclear environment dependencies depending on how external utilities are implemented. The whole point of why that script was written in Perl was to have access to arbitrary-precision arithmetic -- after it was shown that bc would simply lock up on some systems. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-02 19:33 ` H. Peter Anvin @ 2009-01-04 1:32 ` Rob Landley 2009-01-04 1:35 ` H. Peter Anvin 2009-01-04 12:07 ` Alan Cox 0 siblings, 2 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:32 UTC (permalink / raw) To: H. Peter Anvin Cc: Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton On Friday 02 January 2009 13:33:02 H. Peter Anvin wrote: > Rob Landley wrote: > > You mean "The new shell script is much simpler, about 1/4 the size, runs > > on Red Hat 9 from 2003, and isn't perl?" :) > > And introduces unclear environment dependencies depending on how > external utilities are implemented. I note that sed and printf and such are all susv3. I have an explicit test for 32 bit math in the script that cares, and this worked in Red Hat 9 circa 2003. I consider this a step up from code with an implicit dependency on a CPAN library. > The whole point of why that script was written in Perl was to have > access to arbitrary-precision arithmetic -- after it was shown that bc > would simply lock up on some systems. A) I'm not using bc. B) You don't need arbitrary precision arithmetic, you need around 72 bits worth of arithmetic for the pathological case. C) Your definition of "access to arbitrary-precision arithmetic" includes the following, cut and paste verbatim from your old script: # Precomputed values for systems without Math::BigInt # Generated by: # timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 1200 %canned_values = ( 24 => [ '0xa6aaaaab','0x2aaaaaa',26, 125,3, '0xc49ba5e4','0x1fbe76c8b4',37, 3,125, '0xa2c2aaab','0xaaaa',16, 125000,3, '0xc9539b89','0x7fffbce4217d',47, 3,125000, ], 32 => [ '0xfa000000','0x6000000',27, 125,4, '0x83126e98','0xfdf3b645a',36, 4,125, '0xf4240000','0x0',17, 31250,1, '0x8637bd06','0x3fff79c842fa',46, 1,31250, ], 48 => [ '0xa6aaaaab','0x6aaaaaa',27, 125,6, '0xc49ba5e4','0xfdf3b645a',36, 6,125, '0xa2c2aaab','0x15555',17, 62500,3, '0xc9539b89','0x3fffbce4217d',46, 3,62500, ], 64 => [ '0xfa000000','0xe000000',28, 125,8, '0x83126e98','0x7ef9db22d',35, 8,125, '0xf4240000','0x0',18, 15625,1, '0x8637bd06','0x1fff79c842fa',45, 1,15625, ], 100 => [ '0xa0000000','0x0',28, 10,1, '0xcccccccd','0x733333333',35, 1,10, '0x9c400000','0x0',18, 10000,1, '0xd1b71759','0x1fff2e48e8a7',45, 1,10000, ], 122 => [ '0x8325c53f','0xfbcda3a',28, 500,61, '0xf9db22d1','0x7fbe76c8b',35, 61,500, '0x8012e2a0','0x3ef36',18, 500000,61, '0xffda4053','0x1ffffbce4217',45, 61,500000, ], 128 => [ '0xfa000000','0x1e000000',29, 125,16, '0x83126e98','0x3f7ced916',34, 16,125, '0xf4240000','0x40000',19, 15625,2, '0x8637bd06','0xfffbce4217d',44, 2,15625, ], 200 => [ '0xa0000000','0x0',29, 5,1, '0xcccccccd','0x333333333',34, 1,5, '0x9c400000','0x0',19, 5000,1, '0xd1b71759','0xfff2e48e8a7',44, 1,5000, ], 250 => [ '0x80000000','0x0',29, 4,1, '0x80000000','0x180000000',33, 1,4, '0xfa000000','0x0',20, 4000,1, '0x83126e98','0x7ff7ced9168',43, 1,4000, ], 256 => [ '0xfa000000','0x3e000000',30, 125,32, '0x83126e98','0x1fbe76c8b',33, 32,125, '0xf4240000','0xc0000',20, 15625,4, '0x8637bd06','0x7ffde7210be',43, 4,15625, ], 300 => [ '0xd5555556','0x2aaaaaaa',30, 10,3, '0x9999999a','0x1cccccccc',33, 3,10, '0xd0555556','0xaaaaa',20, 10000,3, '0x9d495183','0x7ffcb923a29',43, 3,10000, ], 512 => [ '0xfa000000','0x7e000000',31, 125,64, '0x83126e98','0xfdf3b645',32, 64,125, '0xf4240000','0x1c0000',21, 15625,8, '0x8637bd06','0x3ffef39085f',42, 8,15625, ], 1000 => [ '0x80000000','0x0',31, 1,1, '0x80000000','0x0',31, 1,1, '0xfa000000','0x0',22, 1000,1, '0x83126e98','0x1ff7ced9168',41, 1,1000, ], 1024 => [ '0xfa000000','0xfe000000',32, 125,128, '0x83126e98','0x7ef9db22',31, 128,125, '0xf4240000','0x3c0000',22, 15625,16, '0x8637bd06','0x1fff79c842f',41, 16,15625, ], 1200 => [ '0xd5555556','0xd5555555',32, 5,6, '0x9999999a','0x66666666',31, 6,5, '0xd0555556','0x2aaaaa',22, 2500,3, '0x9d495183','0x1ffcb923a29',41, 3,2500, ] ); Plus a decent chunk of the remaining logic was code to regenerate that table, and to figure out when to use the table and when to compute new values. (And erroring out if the system wasn't capable of doing so.) I don't understand why you didn't just precompute the actual header file output instead or precomputing perl source, but that's a side issue. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 1:32 ` Rob Landley @ 2009-01-04 1:35 ` H. Peter Anvin 2009-01-04 12:07 ` Alan Cox 1 sibling, 0 replies; 135+ messages in thread From: H. Peter Anvin @ 2009-01-04 1:35 UTC (permalink / raw) To: Rob Landley Cc: Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton Rob Landley wrote: > > I consider this a step up from code with an implicit dependency on a CPAN > library. > There is no CPAN library in use. Math::BigInt is a standard part of Perl, and the canned values is there only to support extremely old versions of Perl, or weird system configurations, as a belt-and-suspenders measure. -hpa ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 1:32 ` Rob Landley 2009-01-04 1:35 ` H. Peter Anvin @ 2009-01-04 12:07 ` Alan Cox 2009-01-04 18:36 ` H. Peter Anvin 2009-01-04 19:03 ` Rob Landley 1 sibling, 2 replies; 135+ messages in thread From: Alan Cox @ 2009-01-04 12:07 UTC (permalink / raw) To: Rob Landley Cc: H. Peter Anvin, Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton > I note that sed and printf and such are all susv3. I have an explicit test > for 32 bit math in the script that cares, and this worked in Red Hat 9 circa > 2003. If you are trying to do arbitary precision maths using standard posix tools just use dc. That way the standard is explicit about what you will get. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 12:07 ` Alan Cox @ 2009-01-04 18:36 ` H. Peter Anvin 2009-01-04 19:03 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: H. Peter Anvin @ 2009-01-04 18:36 UTC (permalink / raw) To: Alan Cox Cc: Rob Landley, Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton Alan Cox wrote: >> I note that sed and printf and such are all susv3. I have an explicit test >> for 32 bit math in the script that cares, and this worked in Red Hat 9 circa >> 2003. > > If you are trying to do arbitary precision maths using standard posix > tools just use dc. That way the standard is explicit about what you will > get. The original patch used bc. Unfortunately, it ran into bc bugs -- on some platforms like some of akpm's older test machines it would just hang. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 12:07 ` Alan Cox 2009-01-04 18:36 ` H. Peter Anvin @ 2009-01-04 19:03 ` Rob Landley 2009-01-04 20:39 ` H. Peter Anvin 1 sibling, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-04 19:03 UTC (permalink / raw) To: Alan Cox Cc: H. Peter Anvin, Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton On Sunday 04 January 2009 06:07:35 Alan Cox wrote: > > I note that sed and printf and such are all susv3. I have an explicit > > test for 32 bit math in the script that cares, and this worked in Red Hat > > 9 circa 2003. > > If you are trying to do arbitary precision maths using standard posix > tools just use dc. That way the standard is explicit about what you will > get. I looked at that, but: A) the Open Group Base Specifications (which I normally go by, since they're roughly synonymous with SUSv3 and Posix and available free on the web; they just released version 7 a week or so back) don't list dc as one of their utilities. (They mention bc, but not dc.) B) The busybox implementation of dc is crap. I got 'em to fix the bug where the output defaulted to binary instead of decimal, but the math is all done as floating point rather than arbitrary precision, and they don't implement the << operator. C) The only calculation which can overflow 64 bits (the ADJ32 one) turns out not to need arbitrary precision math, just 72 bits, and if it ever uses more than 32 then bottom 32 are all zero before the divide so you can do it in three lines. Essentially, the ADJ32 calculation is "(($NUMBER-1)<<$SHIFT)/$NUMBER". $SHIFT maxes out at 51 and the largest interesting $NUMBER is 1000000. (That's the pathological case of HZ=1, calculating the USEC_TO_HZ direction. A larger $HZ results in a smaller $SHIFT by the number of bits needed to store $HZ, by the way, so a $HZ of 17 would have a shift of 47. So even a HZ bigger than a million should have a small enough $SHIFT not to cause trouble here, although that's probably an _insane_ input to this script.) 1 million needs 20 bits to store, so the above calculation has to cope with an intermediate value of 999999<<51 which takes a little over 70 bits to store, hence the potential to overflow 63 bits of signed math. But this calculation has two special properties: 1) The number you start with before the shift is divided back out at the end (more or less), so the _result_ has to be less than 1<<$SHIFT and thus only takes $SHIFT bits to store. With a maximum $SHIFT of 51 it has to fit in a 64 bit result with about a dozen bits to spare. 2) The bottom $SHIFT many bits are all zero before the divide. We can use these two properties to easily break the math into chunks that can't overflow by: a) Chopping off the bottom X bits and dividing what's left by $NUMBER, keeping both the dividend and the remainder. Choose any X that's big enough that this step won't overflow. (I chose X=32, leaving at most 40-ish bits here). b) Shift that dividend X bits to the left. This can't overflow because of special property 1 above. c) Shift the remainder X bits to the left. The remainder can't be larger than the $NUMBER you started with, so if X+bits($NUMBER)<64 this has to fit too. With X=32 and bits=20 we again have a dozen bits to spare. d) Add the results of (b) and (c) together. Since the bottom X bits were all zero, this is equivalent to having done the full divide. (Easy enough to mask those bottom bits off and add them to the remainder before the divide if they weren't, but we didn't need to do that because we know they were zero.) So no arbitrary precision math is actually required here, and yes there's a comment in the source about this. :) Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 19:03 ` Rob Landley @ 2009-01-04 20:39 ` H. Peter Anvin 2009-01-05 0:59 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-01-04 20:39 UTC (permalink / raw) To: Rob Landley Cc: Alan Cox, Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton Rob Landley wrote: > > C) The only calculation which can overflow 64 bits (the ADJ32 one) turns out > not to need arbitrary precision math, just 72 bits, and if it ever uses more > than 32 then bottom 32 are all zero before the divide so you can do it in > three lines. > ... for the current code (32 bits). When we get an overflow-less 64-bit implementation, this code will have to be redone, which is not true for a properly done implementation. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 20:39 ` H. Peter Anvin @ 2009-01-05 0:59 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-05 0:59 UTC (permalink / raw) To: H. Peter Anvin Cc: Alan Cox, Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton On Sunday 04 January 2009 14:39:36 H. Peter Anvin wrote: > Rob Landley wrote: > > C) The only calculation which can overflow 64 bits (the ADJ32 one) turns > > out not to need arbitrary precision math, just 72 bits, and if it ever > > uses more than 32 then bottom 32 are all zero before the divide so you > > can do it in three lines. > > ... for the current code (32 bits). When we get an overflow-less 64-bit > implementation, this code will have to be redone, which is not true for > a properly done implementation. One extra mask and add is a strange definition of "redone", but I can add it now if you like. (I'd personally prefer to wait for something to actually need it, but...) > -hpa Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-02 12:00 ` Rob Landley 2009-01-02 19:33 ` H. Peter Anvin @ 2009-01-03 6:28 ` Harvey Harrison 1 sibling, 0 replies; 135+ messages in thread From: Harvey Harrison @ 2009-01-03 6:28 UTC (permalink / raw) To: Rob Landley Cc: Sam Ravnborg, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin On Fri, 2009-01-02 at 06:00 -0600, Rob Landley wrote: > On Friday 02 January 2009 03:04:39 Sam Ravnborg wrote: > > > +# Output start of header file > > > + > > > +cat << EOF > > > +/* Automatically generated by kernel/timeconst.sh */ > > > +/* Conversion constants for HZ == $HZ */ > > > + > > > +#ifndef KERNEL_TIMECONST_H > > > +#define KERNEL_TIMECONST_H > > > > Please use __KERNEL_TIMECONST_H. The two underscores are standard. > > Sure thing. (I was just copying the perl there, I'll post an updated patch > after I get some sleep.) In the x86 discussions recently regarding header guards, I though a single underscore was eventually decided on as 'standard'. Harvey ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-02 8:13 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley 2009-01-02 9:04 ` Sam Ravnborg @ 2009-01-03 12:28 ` Ingo Oeser 2009-01-04 1:36 ` Rob Landley 2009-01-05 0:41 ` Ray Lee 2 siblings, 1 reply; 135+ messages in thread From: Ingo Oeser @ 2009-01-03 12:28 UTC (permalink / raw) To: Rob Landley Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Hi Robert, I fully support your argument here: Requiring lots of hacks in various languages to build a system core sucks. But now sth. more productive: On Friday 02 January 2009, Rob Landley wrote: > diff -r d0c7611dcfd6 kernel/Makefile > --- a/kernel/Makefile Tue Dec 30 17:48:25 2008 -0800 > +++ b/kernel/Makefile Fri Jan 02 00:10:44 2009 -0600 > @@ -115,7 +115,7 @@ > $(obj)/time.o: $(obj)/timeconst.h > > quiet_cmd_timeconst = TIMEC $@ > - cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@ > + cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) > $@ > targets += timeconst.h > -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE > +$(obj)/timeconst.h: $(src)/timeconst.sh FORCE > $(call if_changed,timeconst) > --- /dev/null 1969-12-31 00:00:00.000000000 -0600 > +++ hg/kernel/timeconst.sh 2009-01-01 23:53:17.000000000 -0600 > @@ -0,0 +1,91 @@ > +#!/bin/bash > + > +if [ $# -ne 1 ] > +then > + echo "Usage: timeconst.sh HZ" > + exit 1 > +fi > + > +HZ=$1 > + > +# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math. > + > +[ $((1<<32)) -lt 0 ] && exit 1 > + > +# Output start of header file > + > +cat << EOF > +/* Automatically generated by kernel/timeconst.sh */ > +/* Conversion constants for HZ == $HZ */ > + > +#ifndef KERNEL_TIMECONST_H > +#define KERNEL_TIMECONST_H > + > +#include <linux/param.h> > +#include <linux/types.h> > + > +#if HZ != $HZ > +#error "kernel/timeconst.h has the wrong HZ value!" > +#endif > + > +EOF > + > +# For both Miliseconds and Microseconds > + > +for i in "MSEC 1000" "USEC 1000000" > +do > + NAME=$(echo $i | awk '{print $1}') cut -d' ' -f1 does the same > + PERIOD=$(echo $i | awk '{print $2}') cut -d' ' -f2 does the same Best Regards Ingo Oeser ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-03 12:28 ` Ingo Oeser @ 2009-01-04 1:36 ` Rob Landley 2009-01-04 5:07 ` Valdis.Kletnieks 2009-01-04 7:15 ` Michal Jaegermann 0 siblings, 2 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:36 UTC (permalink / raw) To: Ingo Oeser Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote: > > +for i in "MSEC 1000" "USEC 1000000" > > +do > > + NAME=$(echo $i | awk '{print $1}') > > cut -d' ' -f1 does the same > > > + PERIOD=$(echo $i | awk '{print $2}') > > cut -d' ' -f2 does the same >From a standards perspective http://www.opengroup.org/onlinepubs/9699919799/utilities/cut.html vs http://www.opengroup.org/onlinepubs/9699919799/utilities/awk.html is probably a wash, but from a simplicity perspective using the tool that _isn't_ its own programming language is probably a win. :) I made the change in the second round of patches I just posted. Thanks, Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 1:36 ` Rob Landley @ 2009-01-04 5:07 ` Valdis.Kletnieks 2009-01-04 6:43 ` Rob Landley 2009-01-04 21:51 ` Alejandro Mery 2009-01-04 7:15 ` Michal Jaegermann 1 sibling, 2 replies; 135+ messages in thread From: Valdis.Kletnieks @ 2009-01-04 5:07 UTC (permalink / raw) To: Rob Landley Cc: Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg [-- Attachment #1: Type: text/plain, Size: 769 bytes --] On Sat, 03 Jan 2009 19:36:04 CST, Rob Landley said: > On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote: > > > +for i in "MSEC 1000" "USEC 1000000" > > > +do > > > + NAME=$(echo $i | awk '{print $1}') > > > > cut -d' ' -f1 does the same > > > > > + PERIOD=$(echo $i | awk '{print $2}') > > > > cut -d' ' -f2 does the same Close, but no cee-gar. cut does something counter-intuitive with multiple blanks: % echo 'a b' | awk '{print $2}' b % echo 'a b' | cut -d' ' -f2 % echo 'a b' | sed -r 's/[ ]+/ /g' | cut -d' ' -f2 b Unfortunately, 'sed -r' isn't in the opengroup.org list of required options, and sed 's/ / /g' doesn't DTRT for 3 or more blanks (as it won't recursively apply the change to a *new* double blank formed by the previous change). [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 5:07 ` Valdis.Kletnieks @ 2009-01-04 6:43 ` Rob Landley 2009-01-04 22:13 ` Jamie Lokier 2009-01-04 21:51 ` Alejandro Mery 1 sibling, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-04 6:43 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Saturday 03 January 2009 23:07:55 Valdis.Kletnieks@vt.edu wrote: > On Sat, 03 Jan 2009 19:36:04 CST, Rob Landley said: > > On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote: > > > > +for i in "MSEC 1000" "USEC 1000000" > > > > +do > > > > + NAME=$(echo $i | awk '{print $1}') > > > > > > cut -d' ' -f1 does the same > > > > > > > + PERIOD=$(echo $i | awk '{print $2}') > > > > > > cut -d' ' -f2 does the same > > Close, but no cee-gar. cut does something counter-intuitive with multiple > blanks: Yes, but in this case I'm the one passing in the data so I know there aren't multiple blanks. (Or tabs instead of spaces.) In a private email, Bernd Petrovitsch suggested "set -- $i" and then using NAME=$1; PERIOD=$2. (I keep getting private email responses to these sort of threads, and then getting dismissed as the only one who cares about the issue. Less so this time around, but still...) This apparently works all the way back to the bourne shell. Not worth rolling another patch for, but if I do wind up rolling another patch for other reasons I might switch over to that. Both "cut" and "awk" are susv3, though. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 6:43 ` Rob Landley @ 2009-01-04 22:13 ` Jamie Lokier 2009-01-05 0:15 ` Bernd Petrovitsch 0 siblings, 1 reply; 135+ messages in thread From: Jamie Lokier @ 2009-01-04 22:13 UTC (permalink / raw) To: Rob Landley Cc: Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Rob Landley wrote: > In a private email, Bernd Petrovitsch suggested "set -- $i" and then > using NAME=$1; PERIOD=$2. (I keep getting private email responses > to these sort of threads, and then getting dismissed as the only one > who cares about the issue. Less so this time around, but still...) > This apparently works all the way back to the bourne shell. If you're going "all the way back to the bourne shell", don't use "set -- $i"; use "set x $i" instead, and don't expect to do any arithmetic in the shell; use "expr" or "awk" for arithmetic. (Not relevant to kernel scripts, imho, since you can always assume something a bit more modern and not too stripped down). (I have 850 Linux boxes on my network with a bourne shell which doesn't do $((...)). I won't be building kernels on them though :-) -- Jamie ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 22:13 ` Jamie Lokier @ 2009-01-05 0:15 ` Bernd Petrovitsch 2009-01-05 2:23 ` Jamie Lokier 2009-01-05 4:50 ` Rob Landley 0 siblings, 2 replies; 135+ messages in thread From: Bernd Petrovitsch @ 2009-01-05 0:15 UTC (permalink / raw) To: Jamie Lokier Cc: Rob Landley, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Son, 2009-01-04 at 22:13 +0000, Jamie Lokier wrote: > Rob Landley wrote: > > In a private email, Bernd Petrovitsch suggested "set -- $i" and then > > using NAME=$1; PERIOD=$2. (I keep getting private email responses > > to these sort of threads, and then getting dismissed as the only one > > who cares about the issue. Less so this time around, but still...) > > This apparently works all the way back to the bourne shell. > > If you're going "all the way back to the bourne shell", don't use "set "Going back to a Bourne shell" was neither the intention nor makes it sense IMHO. I mentioned it to point out that the `set -- ' (or `set x `) is nothing new or even a bash-ism. > -- $i"; use "set x $i" instead, and don't expect to do any arithmetic > in the shell; use "expr" or "awk" for arithmetic. > > (Not relevant to kernel scripts, imho, since you can always assume > something a bit more modern and not too stripped down). ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat too extreme. > (I have 850 Linux boxes on my network with a bourne shell which > doesn't do $((...)). I won't be building kernels on them though :-) Believe it or not, but there are folks out there who build the firmware on ARM 200 MHz NFS-mounted systems natively (and not simply cross-compile it on a 2GHz PC .....). Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 0:15 ` Bernd Petrovitsch @ 2009-01-05 2:23 ` Jamie Lokier 2009-01-05 10:46 ` Bernd Petrovitsch 2009-01-05 4:50 ` Rob Landley 1 sibling, 1 reply; 135+ messages in thread From: Jamie Lokier @ 2009-01-05 2:23 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Rob Landley, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Bernd Petrovitsch wrote: > > (I have 850 Linux boxes on my network with a bourne shell which > > doesn't do $((...)). I won't be building kernels on them though :-) > > Believe it or not, but there are folks out there who build the firmware > on ARM 200 MHz NFS-mounted systems natively (and not simply > cross-compile it on a 2GHz PC .....). Really? My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted. Their /bin/sh does not do $((...)), and Bash is not there at all. If I were installing GCC natively on them, I'd install GNU Make and a proper shell while I were at it. But I don't know if Bash works properly without fork()* - or even if GCC does :-) Perl might be hard, as shared libraries aren't supported by the toolchain which targets my ARMs* and Perl likes its loadable modules. I'm not sure why I would want to build a kernel on these devices. But I see why people with mobile ARM devices like gphones might want to, when they're out travelling. -- Jamie (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add proper shared libs. Feel free to fund this :-) ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 2:23 ` Jamie Lokier @ 2009-01-05 10:46 ` Bernd Petrovitsch 2009-01-05 15:01 ` Jamie Lokier 2009-01-05 21:07 ` Rob Landley 0 siblings, 2 replies; 135+ messages in thread From: Bernd Petrovitsch @ 2009-01-05 10:46 UTC (permalink / raw) To: Jamie Lokier Cc: Rob Landley, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Mon, 2009-01-05 at 02:23 +0000, Jamie Lokier wrote: > Bernd Petrovitsch wrote: > > > (I have 850 Linux boxes on my network with a bourne shell which > > > doesn't do $((...)). I won't be building kernels on them though :-) > > > > Believe it or not, but there are folks out there who build the firmware > > on ARM 200 MHz NFS-mounted systems natively (and not simply > > cross-compile it on a 2GHz PC .....). > > Really? > > My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted. > Their /bin/sh does not do $((...)), and Bash is not there at all. I assume that the NFS-mounted root filesystem is a real distribution. And on the local flash is a usual busybox based firmware. > If I were installing GCC natively on them, I'd install GNU Make and a > proper shell while I were at it. But I don't know if Bash works ACK. > properly without fork()* - or even if GCC does :-) > > Perl might be hard, as shared libraries aren't supported by the > toolchain which targets my ARMs* and Perl likes its loadable modules. The simplest way to go is probably to use CentOS or Debian or another ready binary distribution on ARM (or MIPS or PPC or whatever core the embedded system has) possibly on a custom build kernel (if necessary). [...] > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > proper shared libs. Feel free to fund this :-) The above mentioned ARMs have a MMU. Without MMU, it would be truly insane IMHO. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 10:46 ` Bernd Petrovitsch @ 2009-01-05 15:01 ` Jamie Lokier 2009-01-05 16:18 ` Bernd Petrovitsch 2009-01-06 0:06 ` Rob Landley 2009-01-05 21:07 ` Rob Landley 1 sibling, 2 replies; 135+ messages in thread From: Jamie Lokier @ 2009-01-05 15:01 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Rob Landley, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Bernd Petrovitsch wrote: > I assume that the NFS-mounted root filesystem is a real distribution. Not unless you call uClinux (MMU-less) a real distribution, no. > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > > proper shared libs. Feel free to fund this :-) > > The above mentioned ARMs have a MMU. Without MMU, it would be truly > insane IMHO. We have similar cross-build issues without MMUs... I.e. that a lot of useful packages don't cross-build properly (including many which use Autoconf), and it might be easier to make a native build environment than to debug and patch all the broken-for-cross-build packages. Especially as sometimes they build, but fail at run-time in some conditions. But you're right it's probably insane to try. I haven't dared as I suspect GCC and/or Binutils would break too :-) I'm sticking instead with "oh well cross-build a few packages by hand and just don't even _try_ to use most of the handy software out there". You mentioned ARM Debian. According to http://wiki.debian.org/ArmEabiPort one recommended method of bootstrapping it is building natively on an emulated ARM, because cross-building is fragile. -- Jamie ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 15:01 ` Jamie Lokier @ 2009-01-05 16:18 ` Bernd Petrovitsch 2009-01-06 0:06 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: Bernd Petrovitsch @ 2009-01-05 16:18 UTC (permalink / raw) To: Jamie Lokier Cc: Rob Landley, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Mon, 2009-01-05 at 15:01 +0000, Jamie Lokier wrote: > Bernd Petrovitsch wrote: > > I assume that the NFS-mounted root filesystem is a real distribution. > > Not unless you call uClinux (MMU-less) a real distribution, no. Not really. > > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > > > proper shared libs. Feel free to fund this :-) > > > > The above mentioned ARMs have a MMU. Without MMU, it would be truly > > insane IMHO. > > We have similar cross-build issues without MMUs... I.e. that a lot of Of course. > useful packages don't cross-build properly (including many which use > Autoconf), and it might be easier to make a native build environment Tell me about it - AC_TRY_RUN() is the culprit. And `pkg-config` supports cross-compilation only since 18 months or so. Before one had to rewrite the generated .pc files. [...] > You mentioned ARM Debian. According to > http://wiki.debian.org/ArmEabiPort one recommended method of > bootstrapping it is building natively on an emulated ARM, because > cross-building is fragile. That's of course the other solution - if qemu supports your $EMBEDDED_CPU good enough. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 15:01 ` Jamie Lokier 2009-01-05 16:18 ` Bernd Petrovitsch @ 2009-01-06 0:06 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-06 0:06 UTC (permalink / raw) To: Jamie Lokier Cc: Bernd Petrovitsch, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Monday 05 January 2009 09:01:56 Jamie Lokier wrote: > Bernd Petrovitsch wrote: > > I assume that the NFS-mounted root filesystem is a real distribution. > > Not unless you call uClinux (MMU-less) a real distribution, no. I want things to be orthogonal. The following should be completely separate steps: 1) Creating a cross compiler 2) building a native development environment 3) booting a native development environment (on real hardware or under and emulator) 4) natively building your target system. You should be able to mix and match. Crosstool for #1, go download "fedora for arm" instead of #2, qemu or real hardware is your choice for #3, and then you should be able to natively build gentoo under an ubuntu host or vice versa. (How is not currently properly documented, but I'm working on that.) My objection to build systems like buildroot or uClinux is that they bundle all this together into a big hairball. They create their own cross compiler, build their own root file system, use their own packaging system, and you have to take it all or nothing. My build system is ruthlessly orthogonal. I try not to make it depend on other bits of _itself_ more than necessary. > > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > > > proper shared libs. Feel free to fund this :-) > > > > The above mentioned ARMs have a MMU. Without MMU, it would be truly > > insane IMHO. > > We have similar cross-build issues without MMUs... I.e. that a lot of > useful packages don't cross-build properly (including many which use > Autoconf), and it might be easier to make a native build environment > than to debug and patch all the broken-for-cross-build packages. > Especially as sometimes they build, but fail at run-time in some > conditions. If you can get a version of the same architecture with an mmu you can actually build natively on that. It's not ideal (it's a bit like trying to build i486 code on an i686; the fact it runs on the host is no guarantee it'll run on the target), but it's better than cross compiling. And most things have a broad enough compatible "base architecture" that you can mostly get away with it. > But you're right it's probably insane to try. I haven't dared as I > suspect GCC and/or Binutils would break too :-) Oh it does, but you can fix it. :) > I'm sticking instead with "oh well cross-build a few packages by hand > and just don't even _try_ to use most of the handy software out there". Cross compiling doesn't scale, and it bit-rots insanely quickly. > You mentioned ARM Debian. According to > http://wiki.debian.org/ArmEabiPort one recommended method of > bootstrapping it is building natively on an emulated ARM, because > cross-building is fragile. That's what my firmware linux project does too. (I believe I was one of the first doing this back in 2006, but there are three or four others out there doing it now.) Native compiling under emulation is an idea whose time has come. Emulators on cheap x86-64 laptops today are about as powerful as high end tricked out build servers circa 2001, and Moore's Law continues to advance. More memory, more CPU (maybe via SMP but distcc can take advantage of that today and qemu will develop threading someday). You can throw engineering time at the problem (making cross compiling work) or you can throw hardware at the problem (build natively and buy fast native or emulator-hosting hardware). The balance used to be in favor of the former; not so much anymore. That said, my drive for reproducibility and orthogonality says that your native development environment must be something you can reproduce entirely from source on an arbitrary host. You can't make cross compiling go away entirely, the best you can do is limit it to bootstrapping the native environment. But I want to keep the parts I have to cross compile as small and simple as possible, and then run a native build script to get a richer environment. For the past 5+ years my definition has been "an environment that can rebuild itself under itself is powerful enough, that's all I need to cross compile", and from the first time I tried this (late 2002) up until 2.6.25 that was 7 packages. That's why I responded to the addition of perl as a regression, because for my use case it was. > -- Jamie Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 10:46 ` Bernd Petrovitsch 2009-01-05 15:01 ` Jamie Lokier @ 2009-01-05 21:07 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-05 21:07 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Jamie Lokier, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Monday 05 January 2009 04:46:18 Bernd Petrovitsch wrote: > > My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted. > > Their /bin/sh does not do $((...)), and Bash is not there at all. > > I assume that the NFS-mounted root filesystem is a real distribution. > And on the local flash is a usual busybox based firmware. Building on an nfs mount is evil. Make cares greatly about timestamp accuracy, and NFS's dentry cacheing doesn't really, especially when it discards cached copies and re-fetches them, and the server and client's clocks are a half-second off from each other. Sometimes you haven't got a choice, but I hate having to debug the build problems this intermittently causes. If you never do anything except "make all" it should suck less. > > If I were installing GCC natively on them, I'd install GNU Make and a > > proper shell while I were at it. But I don't know if Bash works > > ACK. > > > properly without fork()* - or even if GCC does :-) > > > > Perl might be hard, as shared libraries aren't supported by the > > toolchain which targets my ARMs* and Perl likes its loadable modules. > > The simplest way to go is probably to use CentOS or Debian or another > ready binary distribution on ARM (or MIPS or PPC or whatever core the > embedded system has) possibly on a custom build kernel (if necessary). Building natively on target hardware or under QEMU is growing in popularity. That's how the non-x86 versions of major distros build, and they even have policy documents about it. Here's Fedora's: http://fedoraproject.org/wiki/Architectures/ARM#Native_Compilation And here are the guys who opened the door for Ubuntu's official Arm port: http://mojo.handhelds.org/files/HandheldsMojo_ELC2008.pdf Of course hobbyists like myself haven't got the budget to buy a cluster of high-end arm systems and they're not always even _available_ for things like cris, and for new architectures (Xylinx microblaze anyone?) you'll always have to cross compile to bootstrap the first development environment on 'em anyway, and it's nice for your environment to be _reproducible_... So a more flexible approach is to cross compile just enough to get a working native development environment on the target, and then continue the build natively (whether it's under qemu or on a sufficiently powerful piece of target hardware). That's what my "art piece" Firmware Linux project does, and there's a scratchbox rewrite (sbox2, http://www.freedesktop.org/wiki/Software/sbox2 ) that does the same sort of thing, and there are others out there in various states of development. With x86 hardware so cheap and powerful, building under emulation for less powerful targets starts to make sense. Building natively under emulation (QEMU) is available to hobbyists like me and avoids most of the fun cross compiling issues you don't find out about until after you've shipped the system and somebody tries to do something with it you didn't test. So far the record for diagnosing one of these is the two full- time weeks my friend Garrett spent back at TimeSys tracking down why perl signal handling wasn't working on mips; turned out it was using x86 signal numbers rather which don't match the mips ones. The BSP had been shipping for over a year at that point, but nobody had ever tried to do signal handling in perl on mips before, and since the perl ./configure step is written in perl finding the broken part took some doing. This was back in the mists of early 2007 so it's ancient history by now, of course... If you have set up a cross compiler, you can configure QEMU to use distcc to call out through its virtual network to the cross compiler running on the host, which gives you a speed boost without reintroducing most of the horrible cross compiling issues: there's still only a native toolchain so your build doesn't have to keep two contexts (hostcc/targetcc) straight, ./configure still runs natively so any binaries it builds can run and any questions it asks about the host it's building on should give the right answers for the target it's building for (including uname -m and friends), headers are #included natively and libraries are linked natively (that's just how distcc works, preprocessing and linking happen on the local machine) and there's only one set so they can't accidentally mix and the cross compiler isn't even _involved_ in that, make runs natively so it won't get confused by strange environment variables (yeah, seen that one)...) Only the heavy lifting of compiling preprocessed .c files to .o files gets exported, which is the one thing the cross compiler can't really screw up. But bootstraping a native build environment to run under the emulator is something you want to keep down to as few packages as possible, because if you're trying to get the same behavior across half a dozen boards, cross compiling breaks every time you upgrade _anything_. > [...] > > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > > proper shared libs. Feel free to fund this :-) > > The above mentioned ARMs have a MMU. Without MMU, it would be truly > insane IMHO. Without an mmu you have a restricted set of packages that run anyway. No variable length stacks, you have to use vfork() instead of fork() (no copy on write), memory fragmentation is a big problem so malloc() fails way more often... So toolchain problems aren't a "hump" to get past on nommu systems: the area past that it isn't necessarily any easier. > Bernd Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 0:15 ` Bernd Petrovitsch 2009-01-05 2:23 ` Jamie Lokier @ 2009-01-05 4:50 ` Rob Landley 2009-01-05 12:29 ` Bernd Petrovitsch 1 sibling, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-05 4:50 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Jamie Lokier, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Sunday 04 January 2009 18:15:30 Bernd Petrovitsch wrote: > On Son, 2009-01-04 at 22:13 +0000, Jamie Lokier wrote: > > Rob Landley wrote: > > > In a private email, Bernd Petrovitsch suggested "set -- $i" and then > > > using NAME=$1; PERIOD=$2. (I keep getting private email responses > > > to these sort of threads, and then getting dismissed as the only one > > > who cares about the issue. Less so this time around, but still...) > > > This apparently works all the way back to the bourne shell. > > > > If you're going "all the way back to the bourne shell", don't use "set > > "Going back to a Bourne shell" was neither the intention nor makes it > sense IMHO. > I mentioned it to point out that the `set -- ' (or `set x `) is nothing > new or even a bash-ism. > > > -- $i"; use "set x $i" instead, and don't expect to do any arithmetic > > in the shell; use "expr" or "awk" for arithmetic. > > > > (Not relevant to kernel scripts, imho, since you can always assume > > something a bit more modern and not too stripped down). > > ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat > too extreme. I have yet to encounter a system that uses dash _without_ bash. (All ubuntu variants, even jeos, install bash by default. They moved the /bin/sh symlink but they didn't stop installing bash, and the kernel will preferentially use bash if it finds it.) People keep telling me they exist. I suppose you could uninstall bash. You could also uninstall gcc. Not sure what that proves. (And nobody's shown me this mythical second implementation of perl that all these perl scripts are supposed to be portable to...) Busybox ash is a more interesting case, but that implements lots of bash extensions. That said, it's easy enough the scripts to work with current versions of dash. The whole shell portability issue mostly seems to be a stand-in for other objections (Peter Anvin didn't change syslinux and klibc to require perl to build this year because of dash), but it's easy enough to just address the proxy objection and move on rather than arguing about it... > > (I have 850 Linux boxes on my network with a bourne shell which > > doesn't do $((...)). I won't be building kernels on them though :-) > > Believe it or not, but there are folks out there who build the firmware > on ARM 200 MHz NFS-mounted systems natively (and not simply > cross-compile it on a 2GHz PC .....). Yeah, but according to Changes if they do it with the current kernel they do it with at least gcc 3.2 (August 2002) and make 3.79.1 (June 2000), so trying to make it work on software released pre-Y2K probably isn't that high a priority. :) > Bernd Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 4:50 ` Rob Landley @ 2009-01-05 12:29 ` Bernd Petrovitsch 0 siblings, 0 replies; 135+ messages in thread From: Bernd Petrovitsch @ 2009-01-05 12:29 UTC (permalink / raw) To: Rob Landley Cc: Jamie Lokier, Valdis.Kletnieks, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Son, 2009-01-04 at 22:50 -0600, Rob Landley wrote: > On Sunday 04 January 2009 18:15:30 Bernd Petrovitsch wrote: [...] > > ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat > > too extreme. > > I have yet to encounter a system that uses dash _without_ bash. (All ubuntu Hmm, should be doable with a chroot environment quite cheap and simple. > variants, even jeos, install bash by default. They moved the /bin/sh symlink Yes, I know (small) embedded systems that have a bash (and not "only" one of busybox shells). It eases writing somewhat fast shell scripts without the need for lots of fork()s+exec()s too ..... Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 5:07 ` Valdis.Kletnieks 2009-01-04 6:43 ` Rob Landley @ 2009-01-04 21:51 ` Alejandro Mery 1 sibling, 0 replies; 135+ messages in thread From: Alejandro Mery @ 2009-01-04 21:51 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Rob Landley, Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg [-- Attachment #1: Type: text/plain, Size: 579 bytes --] Valdis.Kletnieks@vt.edu wrote: > Close, but no cee-gar. cut does something counter-intuitive with multiple > blanks: > > % echo 'a b' | awk '{print $2}' > b > % echo 'a b' | cut -d' ' -f2 > > % echo 'a b' | sed -r 's/[ ]+/ /g' | cut -d' ' -f2 > b > > Unfortunately, 'sed -r' isn't in the opengroup.org list of required options, > and sed 's/ / /g' doesn't DTRT for 3 or more blanks (as it won't recursively > apply the change to a *new* double blank formed by the previous change). echo 'a b' | tr -s ' ' | cut -d' ' -f2 b that is the light way ;-) Alejandro Mery [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/x-pkcs7-signature, Size: 5013 bytes --] ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-04 1:36 ` Rob Landley 2009-01-04 5:07 ` Valdis.Kletnieks @ 2009-01-04 7:15 ` Michal Jaegermann 1 sibling, 0 replies; 135+ messages in thread From: Michal Jaegermann @ 2009-01-04 7:15 UTC (permalink / raw) To: Rob Landley Cc: Ingo Oeser, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Sat, Jan 03, 2009 at 07:36:04PM -0600, Rob Landley wrote: > On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote: > > > +for i in "MSEC 1000" "USEC 1000000" > > > +do > > > + NAME=$(echo $i | awk '{print $1}') > > > > cut -d' ' -f1 does the same > > > > > + PERIOD=$(echo $i | awk '{print $2}') > > > > cut -d' ' -f2 does the same > > From a standards perspective > http://www.opengroup.org/onlinepubs/9699919799/utilities/cut.html vs > http://www.opengroup.org/onlinepubs/9699919799/utilities/awk.html is probably > a wash, but from a simplicity perspective using the tool that _isn't_ its own > programming language is probably a win. :) Vagaries of 'cut' aside you can limit yourself here to just shell: set_name_period () { NAME=$1 ; PERIOD=$2 } for i in "MSEC 1000" "USEC 1000000" do set_name_period $i .... done or you may skip a shell function and do 'set $i' within a loop plus assignments of $1 and $2 to NAME and PERIOD but that overwrites original positional parameters (which may be already not important). Michał ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-02 8:13 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley 2009-01-02 9:04 ` Sam Ravnborg 2009-01-03 12:28 ` Ingo Oeser @ 2009-01-05 0:41 ` Ray Lee 2009-01-05 5:08 ` Rob Landley 2 siblings, 1 reply; 135+ messages in thread From: Ray Lee @ 2009-01-05 0:41 UTC (permalink / raw) To: Rob Landley Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Fri, Jan 2, 2009 at 12:13 AM, Rob Landley <rob@landley.net> wrote: > Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script > is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003. > > Peter Anvin added this perl to 2.6.25. Before that, the kernel had never > required perl to build. Nice work. As the computations can all be done in 64-bit precision now, and there have been concerns expressed about some shells not supporting 64 bit integers, is there any reason this can't be done using long longs in C? Other than ruining a good bike shed argument, anyway. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-01-05 0:41 ` Ray Lee @ 2009-01-05 5:08 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-05 5:08 UTC (permalink / raw) To: Ray Lee Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Sunday 04 January 2009 18:41:15 Ray Lee wrote: > On Fri, Jan 2, 2009 at 12:13 AM, Rob Landley <rob@landley.net> wrote: > > Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell > > script is much simpler, about 1/4 the size, and runs on Red Hat 9 from > > 2003. > > > > Peter Anvin added this perl to 2.6.25. Before that, the kernel had never > > required perl to build. > > Nice work. Thanks. You'll definitely want to look at the _second_ version of that patch rather than the first, though. :) > As the computations can all be done in 64-bit precision > now, and there have been concerns expressed about some shells not > supporting 64 bit integers, is there any reason this can't be done > using long longs in C? Nope. Any of this could be done in C. (And that's the approach Sam Ravnborg prefers to take for the second patch in the series, upgrading unifdef.c to do everything itself.) I tend to lean towards scripts that create header files rather than programs that create header files, but as long as you remember to use HOSTCC it's fairly straightforward. :) > Other than ruining a good bike shed argument, anyway. Oh pile on. It beats being dismissed as the only one on the planet who cares about the issue (again). :) Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* [PATCH 2/3]: Remove perl from make headers_install 2009-01-02 8:07 PATCH [0/3]: Simplify the kernel build by removing perl Rob Landley 2009-01-02 8:13 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley @ 2009-01-02 8:14 ` Rob Landley 2009-01-02 9:09 ` Sam Ravnborg 2009-01-02 8:15 ` [PATCH 3/3]: Convert mkcapflags.pl to mkcapflags.sh Rob Landley ` (4 subsequent siblings) 6 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-02 8:14 UTC (permalink / raw) To: Embedded Linux mailing list Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg From: Rob Landley <rob@landley.net> Remove perl from make headers_install by replacing a perl script (doinga simple regex search and replace) with a smaller and faster shell scriptimplementation. The new shell script is a single for loop calling sed andpiping its output through unifdef to produce the target file. Sam Ravnborg added this perl to 2.6.27. Signed-off-by: Rob Landley <rob@landley.net>--- scripts/Makefile.headersinst | 6 ++-- scripts/headers_install.pl | 46 --------------------------------- scripts/headers_install.sh | 23 ++++++++++++++++ 3 files changed, 26 insertions(+), 49 deletions(-) --- /dev/null 2008-11-21 04:46:41.000000000 -0600+++ b/scripts/headers_install.sh 2008-12-15 22:09:45.000000000 -0600@@ -0,0 +1,23 @@+#!/bin/bash++# Grab arguments++INDIR="$1"+shift+OUTDIR="$1"+shift+ARCH="$1"+shift++# Iterate through files listed on command line++for i in "$@"+do+ sed -r \+ -e 's/([ \t(])(__user|__force|__iomem)[ \t]/\1/g' \+ -e 's/__attribute_const__([ \t]|$)/\1/g' \+ -e 's@^#include <linux/compiler.h>@@' "$INDIR/$i" |+ scripts/unifdef -U__KERNEL__ - > "$OUTDIR/$i"+done++exit 0diff -r d9b501c91442 scripts/Makefile.headersinst--- a/scripts/Makefile.headersinst Sun Dec 14 16:25:19 2008 -0800+++ b/scripts/Makefile.headersinst Mon Dec 15 23:30:15 2008 -0600@@ -44,8 +44,8 @@ quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ file$(if $(word 2, $(all-files)),s)) cmd_install = \- $(PERL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \- $(PERL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \+ $(CONFIG_SHELL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \+ $(CONFIG_SHELL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \ touch $@ quiet_cmd_remove = REMOVE $(unwanted)@@ -64,7 +64,7 @@ @: targets += $(install-file)-$(install-file): scripts/headers_install.pl $(input-files) FORCE+$(install-file): scripts/headers_install.sh $(input-files) FORCE $(if $(unwanted),$(call cmd,remove),) $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) $(call if_changed,install)--- hg/scripts/headers_install.pl 2008-11-22 19:09:21.000000000 -0600+++ /dev/null 1970-01-01 00:00:00 -0600@@ -1,46 +0,0 @@-#!/usr/bin/perl -w-#-# headers_install prepare the listed header files for use in-# user space and copy the files to their destination.-#-# Usage: headers_install.pl readdir installdir arch [files...]-# readdir: dir to open files-# installdir: dir to install the files-# arch: current architecture-# arch is used to force a reinstallation when the arch-# changes because kbuild then detect a command line change.-# files: list of files to check-#-# Step in preparation for users space:-# 1) Drop all use of compiler.h definitions-# 2) Drop include of compiler.h-# 3) Drop all sections defined out by __KERNEL__ (using unifdef)--use strict;--my ($readdir, $installdir, $arch, @files) = @ARGV;--my $unifdef = "scripts/unifdef -U__KERNEL__";--foreach my $file (@files) {- local *INFILE;- local *OUTFILE;- my $tmpfile = "$installdir/$file.tmp";- open(INFILE, "<$readdir/$file")- or die "$readdir/$file: $!\n";- open(OUTFILE, ">$tmpfile") or die "$tmpfile: $!\n";- while (my $line = <INFILE>) {- $line =~ s/([\s(])__user\s/$1/g;- $line =~ s/([\s(])__force\s/$1/g;- $line =~ s/([\s(])__iomem\s/$1/g;- $line =~ s/\s__attribute_const__\s/ /g;- $line =~ s/\s__attribute_const__$//g;- $line =~ s/^#include <linux\/compiler.h>//;- printf OUTFILE "%s", $line;- }- close OUTFILE;- close INFILE;- system $unifdef . " $tmpfile > $installdir/$file";- unlink $tmpfile;-}-exit 0;\0ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥ ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 2/3]: Remove perl from make headers_install 2009-01-02 8:14 ` [PATCH 2/3]: Remove perl from make headers_install Rob Landley @ 2009-01-02 9:09 ` Sam Ravnborg 0 siblings, 0 replies; 135+ messages in thread From: Sam Ravnborg @ 2009-01-02 9:09 UTC (permalink / raw) To: Rob Landley Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin On Fri, Jan 02, 2009 at 02:14:32AM -0600, Rob Landley wrote: > From: Rob Landley <rob@landley.net> > > Remove perl from make headers_install by replacing a perl script (doing > a simple regex search and replace) with a smaller and faster shell script > implementation. The new shell script is a single for loop calling sed and > piping its output through unifdef to produce the target file. OK > > Sam Ravnborg added this perl to 2.6.27. Drop this part - this is just to make you happy and for no use for others. > > Signed-off-by: Rob Landley <rob@landley.net> > --- > > scripts/Makefile.headersinst | 6 ++-- > scripts/headers_install.pl | 46 --------------------------------- > scripts/headers_install.sh | 23 ++++++++++++++++ > 3 files changed, 26 insertions(+), 49 deletions(-) > > --- /dev/null 2008-11-21 04:46:41.000000000 -0600 > +++ b/scripts/headers_install.sh 2008-12-15 22:09:45.000000000 -0600 > @@ -0,0 +1,23 @@ > +#!/bin/bash > + > +# Grab arguments > + > +INDIR="$1" > +shift > +OUTDIR="$1" > +shift > +ARCH="$1" > +shift Please add a short explanation what this file is used for and what it does. You can take more or less a direct copy from headers_install.pl > + > +# Iterate through files listed on command line > + > +for i in "$@" > +do > + sed -r \ > + -e 's/([ \t(])(__user|__force|__iomem)[ \t]/\1/g' \ > + -e 's/__attribute_const__([ \t]|$)/\1/g' \ > + -e 's@^#include <linux/compiler.h>@@' "$INDIR/$i" | > + scripts/unifdef -U__KERNEL__ - > "$OUTDIR/$i" > +done > + > +exit 0 > diff -r d9b501c91442 scripts/Makefile.headersinst > --- a/scripts/Makefile.headersinst Sun Dec 14 16:25:19 2008 -0800 > +++ b/scripts/Makefile.headersinst Mon Dec 15 23:30:15 2008 -0600 > @@ -44,8 +44,8 @@ > quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ > file$(if $(word 2, $(all-files)),s)) > cmd_install = \ > - $(PERL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \ > - $(PERL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \ > + $(CONFIG_SHELL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \ > + $(CONFIG_SHELL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \ Stick to the coding style i the file and do not add your own. Makefile.headersinst uses tabs for commands and not for indent. Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* [PATCH 3/3]: Convert mkcapflags.pl to mkcapflags.sh 2009-01-02 8:07 PATCH [0/3]: Simplify the kernel build by removing perl Rob Landley 2009-01-02 8:13 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley 2009-01-02 8:14 ` [PATCH 2/3]: Remove perl from make headers_install Rob Landley @ 2009-01-02 8:15 ` Rob Landley 2009-01-02 9:12 ` Sam Ravnborg 2009-01-02 9:26 ` PATCH [0/3]: Simplify the kernel build by removing perl Arkadiusz Miskiewicz ` (3 subsequent siblings) 6 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-02 8:15 UTC (permalink / raw) To: Embedded Linux mailing list Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg From: Rob Landley <rob@landley.net> Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh. This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h. Peter Anvin added this perl to 2.6.28. Signed-off-by: Rob Landley <rob@landley.net>--- arch/x86/kernel/cpu/Makefile | 4 +-- arch/x86/kernel/cpu/mkcapflags.pl | 32 ---------------------------- arch/x86/kernel/cpu/mkcapflags.sh | 28 ++++++++++++++++++++++++ 3 files changed, 30 insertions(+), 34 deletions(-) diff -ruN alt-linux/arch/x86/kernel/cpu/Makefile alt-linux2/arch/x86/kernel/cpu/Makefile--- alt-linux/arch/x86/kernel/cpu/Makefile 2008-12-24 17:26:37.000000000 -0600+++ alt-linux2/arch/x86/kernel/cpu/Makefile 2008-12-28 18:10:51.000000000 -0600@@ -23,10 +23,10 @@ obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o quiet_cmd_mkcapflags = MKCAP $@- cmd_mkcapflags = $(PERL) $(srctree)/$(src)/mkcapflags.pl $< $@+ cmd_mkcapflags = $(CONFIG_SHELL) $(srctree)/$(src)/mkcapflags.sh $< $@ cpufeature = $(src)/../../include/asm/cpufeature.h targets += capflags.c-$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.pl FORCE+$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.sh FORCE $(call if_changed,mkcapflags)diff -ruN alt-linux/arch/x86/kernel/cpu/mkcapflags.pl alt-linux2/arch/x86/kernel/cpu/mkcapflags.pl--- alt-linux/arch/x86/kernel/cpu/mkcapflags.pl 2008-12-24 17:26:37.000000000 -0600+++ alt-linux2/arch/x86/kernel/cpu/mkcapflags.pl 1969-12-31 18:00:00.000000000 -0600@@ -1,32 +0,0 @@-#!/usr/bin/perl-#-# Generate the x86_cap_flags[] array from include/asm-x86/cpufeature.h-#--($in, $out) = @ARGV;--open(IN, "< $in\0") or die "$0: cannot open: $in: $!\n";-open(OUT, "> $out\0") or die "$0: cannot create: $out: $!\n";--print OUT "#include <asm/cpufeature.h>\n\n";-print OUT "const char * const x86_cap_flags[NCAPINTS*32] = {\n";--while (defined($line = <IN>)) {- if ($line =~ /^\s*\#\s*define\s+(X86_FEATURE_(\S+))\s+(.*)$/) {- $macro = $1;- $feature = $2;- $tail = $3;- if ($tail =~ /\/\*\s*\"([^"]*)\".*\*\//) {- $feature = $1;- }-- if ($feature ne '') {- printf OUT "\t%-32s = \"%s\",\n",- "[$macro]", "\L$feature";- }- }-}-print OUT "};\n";--close(IN);-close(OUT);diff -ruN alt-linux/arch/x86/kernel/cpu/mkcapflags.sh alt-linux2/arch/x86/kernel/cpu/mkcapflags.sh--- alt-linux/arch/x86/kernel/cpu/mkcapflags.sh 1969-12-31 18:00:00.000000000 -0600+++ alt-linux2/arch/x86/kernel/cpu/mkcapflags.sh 2008-12-28 18:08:50.000000000 -0600@@ -0,0 +1,28 @@+#!/bin/bash+#+# Generate the x86_cap_flags[] array from include/asm/cpufeature.h+#++IN=$1+OUT=$2++(+ echo "#include <asm/cpufeature.h>"+ echo ""+ echo "const char * const x86_cap_flags[NCAPINTS*32] = {"++ # Iterate through any input lines starting with #define X86_FEATURE_+ sed -n -e 's/\t/ /g' -e 's/^ *# *define *X86_FEATURE_//p' $IN |+ while read i+ do+ # Name is everything up to the first whitespace+ NAME="$(echo "$i" | sed 's/ .*//')"++ # If the /* comment */ starts with a quote string, grab that.+ VALUE="$(echo "$i" | sed -n 's@.*/\* *\("[^"]*"\).*\*/@\1@p')"+ [ -z "$VALUE" ] && VALUE="\"$(echo "$NAME" | tr A-Z a-z)\""++ [ "$VALUE" != '""' ] && echo " [X86_FEATURE_$NAME] = $VALUE,"+ done+ echo "};"+) > $OUT\0ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥ ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 3/3]: Convert mkcapflags.pl to mkcapflags.sh 2009-01-02 8:15 ` [PATCH 3/3]: Convert mkcapflags.pl to mkcapflags.sh Rob Landley @ 2009-01-02 9:12 ` Sam Ravnborg 0 siblings, 0 replies; 135+ messages in thread From: Sam Ravnborg @ 2009-01-02 9:12 UTC (permalink / raw) To: Rob Landley Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin On Fri, Jan 02, 2009 at 02:15:36AM -0600, Rob Landley wrote: > From: Rob Landley <rob@landley.net> > > Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh. > > This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h. OK > Peter Anvin added this perl to 2.6.28. Drop it. Implementation looks OK. Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 8:07 PATCH [0/3]: Simplify the kernel build by removing perl Rob Landley ` (2 preceding siblings ...) 2009-01-02 8:15 ` [PATCH 3/3]: Convert mkcapflags.pl to mkcapflags.sh Rob Landley @ 2009-01-02 9:26 ` Arkadiusz Miskiewicz 2009-01-02 9:49 ` Christoph Hellwig ` (2 more replies) 2009-01-02 9:50 ` Paul Mundt ` (2 subsequent siblings) 6 siblings, 3 replies; 135+ messages in thread From: Arkadiusz Miskiewicz @ 2009-01-02 9:26 UTC (permalink / raw) To: linux-kernel Cc: Rob Landley, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Friday 02 of January 2009, Rob Landley wrote: > Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) > building a Linux kernel never required perl to be installed on the build > system. (Various development and debugging scripts were written in perl > and python and such, but they weren't involved in actually building a > kernel.) Building a kernel before 2.6.25 could be done with a minimal > system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux > kernel, and nothing else. And now bash is going to be required... while some distros don't need/have bash. /bin/sh should be enough. Heh, -- Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:26 ` PATCH [0/3]: Simplify the kernel build by removing perl Arkadiusz Miskiewicz @ 2009-01-02 9:49 ` Christoph Hellwig 2009-01-02 10:16 ` Alejandro Mery 2009-01-02 12:56 ` Rob Landley 2009-01-02 10:02 ` Mark Miller 2009-01-02 11:13 ` Rob Landley 2 siblings, 2 replies; 135+ messages in thread From: Christoph Hellwig @ 2009-01-02 9:49 UTC (permalink / raw) To: Arkadiusz Miskiewicz Cc: linux-kernel, Rob Landley, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote: > On Friday 02 of January 2009, Rob Landley wrote: > > Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) > > building a Linux kernel never required perl to be installed on the build > > system. (Various development and debugging scripts were written in perl > > and python and such, but they weren't involved in actually building a > > kernel.) Building a kernel before 2.6.25 could be done with a minimal > > system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux > > kernel, and nothing else. > > And now bash is going to be required... while some distros don't need/have > bash. /bin/sh should be enough. *nod* bash is in many ways a worse requirement than perl. strict posix /bin/sh + awk + sed would be nicest, but if that's too much work perl seems reasonable. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:49 ` Christoph Hellwig @ 2009-01-02 10:16 ` Alejandro Mery 2009-01-02 10:30 ` Mark Miller ` (2 more replies) 2009-01-02 12:56 ` Rob Landley 1 sibling, 3 replies; 135+ messages in thread From: Alejandro Mery @ 2009-01-02 10:16 UTC (permalink / raw) To: Christoph Hellwig Cc: Arkadiusz Miskiewicz, linux-kernel, Rob Landley, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg [-- Attachment #1: Type: text/plain, Size: 1276 bytes --] Christoph Hellwig escribió: > On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote: > >> On Friday 02 of January 2009, Rob Landley wrote: >> >>> Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) >>> building a Linux kernel never required perl to be installed on the build >>> system. (Various development and debugging scripts were written in perl >>> and python and such, but they weren't involved in actually building a >>> kernel.) Building a kernel before 2.6.25 could be done with a minimal >>> system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux >>> kernel, and nothing else. >>> >> And now bash is going to be required... while some distros don't need/have >> bash. /bin/sh should be enough. >> > > *nod* bash is in many ways a worse requirement than perl. strict posix > /bin/sh + awk + sed would be nicest, but if that's too much work perl > seems reasonable. well, bash is not worse as bash is trivial to cross-compile to run on a constrained sandbox and perl is a nightmare, but I agree bash should be avoided too. I think the $(( ... )) bash-ism can be replaced with a simple .c helper toy. Thank Rob for reopening the topic. Alejandro Mery [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/x-pkcs7-signature, Size: 5013 bytes --] ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 10:16 ` Alejandro Mery @ 2009-01-02 10:30 ` Mark Miller 2009-01-02 11:18 ` Matt Keenan 2009-01-02 10:41 ` Måns Rullgård 2009-01-02 11:15 ` Rob Landley 2 siblings, 1 reply; 135+ messages in thread From: Mark Miller @ 2009-01-02 10:30 UTC (permalink / raw) To: Alejandro Mery Cc: Christoph Hellwig, Arkadiusz Miskiewicz, linux-kernel, Rob Landley, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Jan 2, 2009, at 4:16 AM, Alejandro Mery wrote: > Christoph Hellwig escribió: >> On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote: >> >>> On Friday 02 of January 2009, Rob Landley wrote: >>> >>>> Before 2.6.25 (specifically git >>>> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) >>>> building a Linux kernel never required perl to be installed on >>>> the build >>>> system. (Various development and debugging scripts were written >>>> in perl >>>> and python and such, but they weren't involved in actually >>>> building a >>>> kernel.) Building a kernel before 2.6.25 could be done with a >>>> minimal >>>> system built from gcc, binutils, bash, make, busybox, uClibc, and >>>> the Linux >>>> kernel, and nothing else. >>>> >>> And now bash is going to be required... while some distros don't >>> need/have >>> bash. /bin/sh should be enough. >>> >> >> *nod* bash is in many ways a worse requirement than perl. strict >> posix >> /bin/sh + awk + sed would be nicest, but if that's too much work perl >> seems reasonable. >> > well, bash is not worse as bash is trivial to cross-compile to run > on a > constrained sandbox and perl is a nightmare, but I agree bash should > be > avoided too. > > I think the $(( ... )) bash-ism can be replaced with a simple .c > helper toy. > > Thank Rob for reopening the topic. > > Alejandro Mery And actually, one of the things that I just recalled, is that several of the Perl configure scripts in order to actually build itself, rely on Bourne shell calls. So the argument to require a strict POSIX+sed +awk implementation rather than Perl to build the kernel, fails, since you already require some variant of shell greater than strict POSIX / bin/sh to build Perl. So this is one less dependency. Also, attempting to cross-compile Perl, is indeed a nightmare. -- Mark Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 10:30 ` Mark Miller @ 2009-01-02 11:18 ` Matt Keenan 0 siblings, 0 replies; 135+ messages in thread From: Matt Keenan @ 2009-01-02 11:18 UTC (permalink / raw) To: Mark Miller Cc: Alejandro Mery, Christoph Hellwig, Arkadiusz Miskiewicz, linux-kernel, Rob Landley, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Fri, 2009-01-02 at 04:30 -0600, Mark Miller wrote: > On Jan 2, 2009, at 4:16 AM, Alejandro Mery wrote: > > > Christoph Hellwig escribió: > >> On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote: > >> > >>> On Friday 02 of January 2009, Rob Landley wrote: > >>> > >>>> Before 2.6.25 (specifically git > >>>> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) > >>>> building a Linux kernel never required perl to be installed on > >>>> the build > >>>> system. (Various development and debugging scripts were written > >>>> in perl > >>>> and python and such, but they weren't involved in actually > >>>> building a > >>>> kernel.) Building a kernel before 2.6.25 could be done with a > >>>> minimal > >>>> system built from gcc, binutils, bash, make, busybox, uClibc, and > >>>> the Linux > >>>> kernel, and nothing else. > >>>> > >>> And now bash is going to be required... while some distros don't > >>> need/have > >>> bash. /bin/sh should be enough. > >>> > >> > >> *nod* bash is in many ways a worse requirement than perl. strict > >> posix > >> /bin/sh + awk + sed would be nicest, but if that's too much work perl > >> seems reasonable. > > >> > > well, bash is not worse as bash is trivial to cross-compile to run > > on a > > constrained sandbox and perl is a nightmare, but I agree bash should > > be > > avoided too. > > > > I think the $(( ... )) bash-ism can be replaced with a simple .c > > helper toy. > > > > Thank Rob for reopening the topic. > > > > Alejandro Mery > > And actually, one of the things that I just recalled, is that several > of the Perl configure scripts in order to actually build itself, rely > on Bourne shell calls. So the argument to require a strict POSIX+sed > +awk implementation rather than Perl to build the kernel, fails, since > you already require some variant of shell greater than strict POSIX / > bin/sh to build Perl. So this is one less dependency. > > Also, attempting to cross-compile Perl, is indeed a nightmare. > Having cross compiled Perl, on to a Unix with a brain dead third party TCP/IP no less, only 4 years after starting to use Unix / Linux I can attest that it is not that difficult to cross compile. Heck it even runs on those weird Crays that don't know which byte sex they are. Matt ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 10:16 ` Alejandro Mery 2009-01-02 10:30 ` Mark Miller @ 2009-01-02 10:41 ` Måns Rullgård 2009-01-15 12:59 ` Pádraig Brady 2009-01-02 11:15 ` Rob Landley 2 siblings, 1 reply; 135+ messages in thread From: Måns Rullgård @ 2009-01-02 10:41 UTC (permalink / raw) To: linux-kernel; +Cc: linux-embedded Alejandro Mery <amery@opensde.org> writes: > Christoph Hellwig escribió: >> On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote: >> >>> On Friday 02 of January 2009, Rob Landley wrote: >>> >>>> Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) >>>> building a Linux kernel never required perl to be installed on the build >>>> system. (Various development and debugging scripts were written in perl >>>> and python and such, but they weren't involved in actually building a >>>> kernel.) Building a kernel before 2.6.25 could be done with a minimal >>>> system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux >>>> kernel, and nothing else. >>>> >>> And now bash is going to be required... while some distros don't need/have >>> bash. /bin/sh should be enough. >>> >> >> *nod* bash is in many ways a worse requirement than perl. strict posix >> /bin/sh + awk + sed would be nicest, but if that's too much work perl >> seems reasonable. > well, bash is not worse as bash is trivial to cross-compile to run on a > constrained sandbox and perl is a nightmare, but I agree bash should be > avoided too. > > I think the $(( ... )) bash-ism can be replaced with a simple .c helper toy. The $(( ... )) construct is standard POSIX shell syntax, see http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_04 Bash supports $[ ... ] as an alternate syntax for the same thing. Perhaps you were thinking of that. -- Måns Rullgård mans@mansr.com ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 10:41 ` Måns Rullgård @ 2009-01-15 12:59 ` Pádraig Brady 2009-01-15 18:52 ` Jamie Lokier 0 siblings, 1 reply; 135+ messages in thread From: Pádraig Brady @ 2009-01-15 12:59 UTC (permalink / raw) To: Måns Rullgård; +Cc: linux-kernel, linux-embedded Måns Rullgård wrote: > Alejandro Mery <amery@opensde.org> writes: > >> I think the $(( ... )) bash-ism can be replaced with a simple .c helper toy. > > The $(( ... )) construct is standard POSIX shell syntax, see > http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_04 > > Bash supports $[ ... ] as an alternate syntax for the same thing. > Perhaps you were thinking of that. I think the misconception that $(( ... )) is a bashism is caused by the wrong highlighting defaults chosen by vim. To fix this add this to ~/.vimrc let g:is_posix = 1 That will also allow you to use the $(command) POSIX construct. BTW, the vim syntax maintainers don't agree with changing this default: http://groups.google.com/group/vim_dev/browse_thread/thread/41139a32772b2f5f cheers, Pádraig. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-15 12:59 ` Pádraig Brady @ 2009-01-15 18:52 ` Jamie Lokier 2009-01-15 19:45 ` Måns Rullgård 0 siblings, 1 reply; 135+ messages in thread From: Jamie Lokier @ 2009-01-15 18:52 UTC (permalink / raw) To: Pádraig Brady; +Cc: Måns Rullgård, linux-kernel, linux-embedded Pádraig Brady wrote: > > The $(( ... )) construct is standard POSIX shell syntax, see > > http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_04 > > > > Bash supports $[ ... ] as an alternate syntax for the same thing. > > Perhaps you were thinking of that. > > I think the misconception that $(( ... )) is a bashism is caused by > the wrong highlighting defaults chosen by vim. I think the misconception is because traditional unix bourne shells don't implement that construct. I just tried it on a few machines, and it failed on 4 of them. Admittedly, the only up to date one is running Solaris 10; the others are older unixes that you're unlikely to build Linux on. -- Jamie ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-15 18:52 ` Jamie Lokier @ 2009-01-15 19:45 ` Måns Rullgård 0 siblings, 0 replies; 135+ messages in thread From: Måns Rullgård @ 2009-01-15 19:45 UTC (permalink / raw) To: Jamie Lokier Cc: Pádraig Brady, Måns Rullgård, linux-kernel, linux-embedded Jamie Lokier <jamie@shareable.org> writes: > Pádraig Brady wrote: >> > The $(( ... )) construct is standard POSIX shell syntax, see >> > http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_04 >> > >> > Bash supports $[ ... ] as an alternate syntax for the same thing. >> > Perhaps you were thinking of that. >> >> I think the misconception that $(( ... )) is a bashism is caused by >> the wrong highlighting defaults chosen by vim. > > I think the misconception is because traditional unix bourne shells > don't implement that construct. I just tried it on a few machines, > and it failed on 4 of them. Admittedly, the only up to date one is > running Solaris 10; the others are older unixes that you're unlikely > to build Linux on. On Solaris, /usr/xpg4/bin/sh is usually a POSIX-compliant shell. I don't know how long it has been there. -- Måns Rullgård mans@mansr.com ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 10:16 ` Alejandro Mery 2009-01-02 10:30 ` Mark Miller 2009-01-02 10:41 ` Måns Rullgård @ 2009-01-02 11:15 ` Rob Landley 2009-01-02 11:44 ` Sam Ravnborg 2 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-02 11:15 UTC (permalink / raw) To: Alejandro Mery Cc: Christoph Hellwig, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Friday 02 January 2009 04:16:53 Alejandro Mery wrote: > Christoph Hellwig escribió: > > On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote: > >> On Friday 02 of January 2009, Rob Landley wrote: > >>> Before 2.6.25 (specifically git > >>> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kernel > >>> never required perl to be installed on the build system. (Various > >>> development and debugging scripts were written in perl and python and > >>> such, but they weren't involved in actually building a kernel.) > >>> Building a kernel before 2.6.25 could be done with a minimal system > >>> built from gcc, binutils, bash, make, busybox, uClibc, and the Linux > >>> kernel, and nothing else. > >> > >> And now bash is going to be required... while some distros don't > >> need/have bash. /bin/sh should be enough. > > > > *nod* bash is in many ways a worse requirement than perl. strict posix > > /bin/sh + awk + sed would be nicest, but if that's too much work perl > > seems reasonable. > > well, bash is not worse as bash is trivial to cross-compile to run on a > constrained sandbox and perl is a nightmare, but I agree bash should be > avoided too. > > I think the $(( ... )) bash-ism can be replaced with a simple .c helper > toy. No, $[ ] is the bashism, $(( )) is susv3: http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04 I intentionally switched from $[ ] to $(( )) to make dash work. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 11:15 ` Rob Landley @ 2009-01-02 11:44 ` Sam Ravnborg 0 siblings, 0 replies; 135+ messages in thread From: Sam Ravnborg @ 2009-01-02 11:44 UTC (permalink / raw) To: Rob Landley Cc: Alejandro Mery, Christoph Hellwig, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin On Fri, Jan 02, 2009 at 05:15:32AM -0600, Rob Landley wrote: > On Friday 02 January 2009 04:16:53 Alejandro Mery wrote: > > Christoph Hellwig escribió: > > > On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote: > > >> On Friday 02 of January 2009, Rob Landley wrote: > > >>> Before 2.6.25 (specifically git > > >>> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kernel > > >>> never required perl to be installed on the build system. (Various > > >>> development and debugging scripts were written in perl and python and > > >>> such, but they weren't involved in actually building a kernel.) > > >>> Building a kernel before 2.6.25 could be done with a minimal system > > >>> built from gcc, binutils, bash, make, busybox, uClibc, and the Linux > > >>> kernel, and nothing else. > > >> > > >> And now bash is going to be required... while some distros don't > > >> need/have bash. /bin/sh should be enough. > > > > > > *nod* bash is in many ways a worse requirement than perl. strict posix > > > /bin/sh + awk + sed would be nicest, but if that's too much work perl > > > seems reasonable. > > > > well, bash is not worse as bash is trivial to cross-compile to run on a > > constrained sandbox and perl is a nightmare, but I agree bash should be > > avoided too. > > > > I think the $(( ... )) bash-ism can be replaced with a simple .c helper > > toy. > > No, $[ ] is the bashism, $(( )) is susv3: > http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04 > > I intentionally switched from $[ ] to $(( )) to make dash work. The focus on this patch is to create a minimal set of dependencies so please document what dependencies / compatibility this patch introduces. It is not obvious for me for example if the script requires sh, bash or dash or whatever. The shebang is so often wrong that this is not docuemnting such things. Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:49 ` Christoph Hellwig 2009-01-02 10:16 ` Alejandro Mery @ 2009-01-02 12:56 ` Rob Landley 2009-01-02 14:04 ` Theodore Tso 1 sibling, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-02 12:56 UTC (permalink / raw) To: Christoph Hellwig Cc: Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Friday 02 January 2009 03:49:34 Christoph Hellwig wrote: > On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote: > > On Friday 02 of January 2009, Rob Landley wrote: > > > Before 2.6.25 (specifically git > > > bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kernel > > > never required perl to be installed on the build system. (Various > > > development and debugging scripts were written in perl and python and > > > such, but they weren't involved in actually building a kernel.) > > > Building a kernel before 2.6.25 could be done with a minimal system > > > built from gcc, binutils, bash, make, busybox, uClibc, and the Linux > > > kernel, and nothing else. > > > > And now bash is going to be required... while some distros don't > > need/have bash. /bin/sh should be enough. > > *nod* bash is in many ways a worse requirement than perl. strict posix > /bin/sh + awk + sed would be nicest, but if that's too much work perl > seems reasonable. The scripts should work with dash (modulo the one that needs 64 bit math, which dash only provides on a 64 bit host), or with busybox ash (which can provide 64 bit math on 32 bit hosts just like bash can). I'll explicitly retest both of those when I respin the patches in the <strike>morning</strike> afternoon. (And yes I thought about writing my own arbitrary precision arithmetic shell functions, but it really didn't seem worth the complexity since the only 32 bit Linux distros I've seen that install dash also install bash by default. I just put in a test for 32 bit math so it can spot it and fail, on the off chance you're running a 32 bit host with dash after explicitly uninstalling bash. All the embedded 32 bit ones that try to act as development environments use busybox ash, or more often just install bash.) That said, how is bash _worse_ than perl? (Where's the second implementation of perl? Even Python had jython, but perl has... what? The attempt to rebase on Parrot went down in flames...) If the argument is that "depending on a specific shell implementation is as bad as depending on the one and only implementation of perl", that argument I can at least follow, even if it doesn't actually apply in this case. But where does "worse" come in? Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 12:56 ` Rob Landley @ 2009-01-02 14:04 ` Theodore Tso 2009-01-03 3:22 ` Jamie Lokier 2009-01-04 2:23 ` Rob Landley 0 siblings, 2 replies; 135+ messages in thread From: Theodore Tso @ 2009-01-02 14:04 UTC (permalink / raw) To: Rob Landley Cc: Christoph Hellwig, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Fri, Jan 02, 2009 at 06:56:31AM -0600, Rob Landley wrote: > That said, how is bash _worse_ than perl? (Where's the second > implementation of perl? Even Python had jython, but perl > has... what? The attempt to rebase on Parrot went down in > flames...) (1) bash implies POSIX extensions; perl is actually quite portable. (2) There are distributions that install with perl by default but not bash; they use dash for speed reasons. Sounds like though modulo dealing with 64-bit arithmetic, your patches are mostly dash/POSIX.2 comformant, so you're probably mostly good on that front once you address the 32/64-bit issues. I'd also suggest explicitly add a reminder to the shell scripts' comments to avoid bashisms for maximum portability, to remind developers in the future who might try to change the shell scripts to watch out for portability issues. - Ted ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 14:04 ` Theodore Tso @ 2009-01-03 3:22 ` Jamie Lokier 2009-01-04 2:23 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: Jamie Lokier @ 2009-01-03 3:22 UTC (permalink / raw) To: Theodore Tso, Rob Landley, Christoph Hellwig, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg Theodore Tso wrote: > perl is actually quite portable. Portability aside, Perl has another fun issue. The number of times I've had a Perl script break when copied to a newer system which had a newer version of Perl is... noticable. > I'd also suggest explicitly add a reminder to the shell scripts' > comments to avoid bashisms for maximum portability, to remind > developers in the future who might try to change the shell scripts > to watch out for portability issues. You can force Bash into POSIX mode if that's helpful. -- Jamie ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 14:04 ` Theodore Tso 2009-01-03 3:22 ` Jamie Lokier @ 2009-01-04 2:23 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 2:23 UTC (permalink / raw) To: Theodore Tso Cc: Christoph Hellwig, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Friday 02 January 2009 08:04:09 Theodore Tso wrote: > Sounds like though modulo dealing with 64-bit arithmetic, your patches > are mostly dash/POSIX.2 comformant, so you're probably mostly good on > that front once you address the 32/64-bit issues. I'd also suggest > explicitly add a reminder to the shell scripts' comments to avoid > bashisms for maximum portability, to remind developers in the future > who might try to change the shell scripts to watch out for portability > issues. I changed the scripts to start with #!/bin/sh and retested under dash. If scripts say #!/bin/sh when they actually need bash, or say #!/bin/bash when they work with dash, that should probably be fixed. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:26 ` PATCH [0/3]: Simplify the kernel build by removing perl Arkadiusz Miskiewicz 2009-01-02 9:49 ` Christoph Hellwig @ 2009-01-02 10:02 ` Mark Miller 2009-01-02 10:03 ` Mark Miller 2009-01-02 11:13 ` Rob Landley 2 siblings, 1 reply; 135+ messages in thread From: Mark Miller @ 2009-01-02 10:02 UTC (permalink / raw) To: Arkadiusz Miskiewicz Cc: linux-kernel, Rob Landley, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Jan 2, 2009, at 3:26 AM, Arkadiusz Miskiewicz wrote: > On Friday 02 of January 2009, Rob Landley wrote: >> Before 2.6.25 (specifically git >> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) >> building a Linux kernel never required perl to be installed on the >> build >> system. (Various development and debugging scripts were written in >> perl >> and python and such, but they weren't involved in actually building a >> kernel.) Building a kernel before 2.6.25 could be done with a minimal >> system built from gcc, binutils, bash, make, busybox, uClibc, and >> the Linux >> kernel, and nothing else. > > And now bash is going to be required... while some distros don't > need/have > bash. /bin/sh should be enough. Which distros only have /bin/sh which do not have Perl? I'm honestly curious. > -- > Arkadiusz Miśkiewicz PLD/Linux Team > arekm / maven.pl http://ftp.pld-linux.org/ -- Mark Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 10:02 ` Mark Miller @ 2009-01-02 10:03 ` Mark Miller 0 siblings, 0 replies; 135+ messages in thread From: Mark Miller @ 2009-01-02 10:03 UTC (permalink / raw) To: Mark Miller Cc: Arkadiusz Miskiewicz, linux-kernel, Rob Landley, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Jan 2, 2009, at 4:02 AM, Mark Miller wrote: > > On Jan 2, 2009, at 3:26 AM, Arkadiusz Miskiewicz wrote: > >> On Friday 02 of January 2009, Rob Landley wrote: >>> Before 2.6.25 (specifically git >>> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) >>> building a Linux kernel never required perl to be installed on the >>> build >>> system. (Various development and debugging scripts were written >>> in perl >>> and python and such, but they weren't involved in actually >>> building a >>> kernel.) Building a kernel before 2.6.25 could be done with a >>> minimal >>> system built from gcc, binutils, bash, make, busybox, uClibc, and >>> the Linux >>> kernel, and nothing else. >> >> And now bash is going to be required... while some distros don't >> need/have >> bash. /bin/sh should be enough. > > Which distros only have /bin/sh which do not have Perl? I'm honestly > curious. That is, *do* have Perl. Typo there. > > >> -- >> Arkadiusz Miśkiewicz PLD/Linux Team >> arekm / maven.pl http://ftp.pld-linux.org/ -- Mark Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:26 ` PATCH [0/3]: Simplify the kernel build by removing perl Arkadiusz Miskiewicz 2009-01-02 9:49 ` Christoph Hellwig 2009-01-02 10:02 ` Mark Miller @ 2009-01-02 11:13 ` Rob Landley 2009-01-02 16:04 ` Matthieu CASTET 2 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-02 11:13 UTC (permalink / raw) To: Arkadiusz Miskiewicz Cc: linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote: > On Friday 02 of January 2009, Rob Landley wrote: > > Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 > > ) building a Linux kernel never required perl to be installed on the > > build system. (Various development and debugging scripts were written in > > perl and python and such, but they weren't involved in actually building > > a kernel.) Building a kernel before 2.6.25 could be done with a minimal > > system built from gcc, binutils, bash, make, busybox, uClibc, and the > > Linux kernel, and nothing else. > > And now bash is going to be required... while some distros don't need/have > bash. /bin/sh should be enough. > > Heh, I believe all three scripts run under dash and busybox ash. (The timeconst.sh one needs 64 bit math which dash only provides on 64 bit hosts, which is a regression from Red Hat 9 in 2003 by the way. Busybox ash can also provide 64 bit math on 32 bit hosts, and the script should run with that just fine if you haven't got bash and that's what your "sh" in the path is.) The makefiles execute those scripts via CONFIG_SHELL, not via the #!/blah line at the start, so it's largely irrelevant what gets put there anyway. If you haven't got bash installed it'll use "sh", which should work with dash on a 64 bit host or with busybox ash. (That's why that one file has a test to make sure 64 bit math _does_ work. The only Linux development environment I'm aware of where that test would trigger is if you use a 32 bit ubuntu and go out of your way to _uninstall_ bash. Even Cygwin uses bash.) Beyond that, "find linux -type f | xargs grep bin/bash | wc" comes up with 38 instances (admittedly half of 'em in Documentation, but lots in scripts, and makefiles, and defconfigs, at least one hardwired in C code.) So this would not a _new_ dependency. By the way, what Linux distros install a compiler toolchain but not bash? I'm curious. (Even after Ubuntu moved #!/bin/sh to point to dash, it still installs bash as part of the default environment, even if you don't install development tools.) You've built the kernel on this system before? Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 11:13 ` Rob Landley @ 2009-01-02 16:04 ` Matthieu CASTET 2009-01-03 19:46 ` Rob Landley 2009-01-04 1:32 ` Rob Landley 0 siblings, 2 replies; 135+ messages in thread From: Matthieu CASTET @ 2009-01-02 16:04 UTC (permalink / raw) To: Rob Landley Cc: Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg Rob Landley a écrit : > On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote: >> On Friday 02 of January 2009, Rob Landley wrote: >> >> Heh, > > I believe all three scripts run under dash and busybox ash. (The timeconst.sh > one needs 64 bit math which dash only provides on 64 bit hosts, which is a > regression from Red Hat 9 in 2003 by the way. With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for 32 bit hosts : $ uname -m i686 $ dash -c 'echo $((1<<32))' 4294967296 Matthieu ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 16:04 ` Matthieu CASTET @ 2009-01-03 19:46 ` Rob Landley 2009-01-03 20:10 ` Sam Ravnborg 2009-01-04 0:44 ` Robert Hancock 2009-01-04 1:32 ` Rob Landley 1 sibling, 2 replies; 135+ messages in thread From: Rob Landley @ 2009-01-03 19:46 UTC (permalink / raw) To: Matthieu CASTET Cc: Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Friday 02 January 2009 10:04:08 Matthieu CASTET wrote: > Rob Landley a écrit : > > On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote: > >> On Friday 02 of January 2009, Rob Landley wrote: > >> > >> Heh, > > > > I believe all three scripts run under dash and busybox ash. (The > > timeconst.sh one needs 64 bit math which dash only provides on 64 bit > > hosts, which is a regression from Red Hat 9 in 2003 by the way. > > With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for > 32 bit hosts : > $ uname -m > i686 > $ dash -c 'echo $((1<<32))' > 4294967296 Cool. The "relatively recent" 32 bit image I have lying around for testing purposes is xubuntu 7.10, and when dash was first introduced into ubuntu it had _buckets_ of bugs. (If you backgrounded a task with & and then hit ctrl-z on the command line, it would suspend the backgrounded task. It was Not Ready for Prime Time in a big way.) Lack of 64 bit math could easily be one more. (It _is_ a regression vs Red Hat 9.) The dash in ubuntu 8.10 seems to have a lot of the more obvious problems worked out. Good to know. :) That said, while testing the new round of patches against various shells and making it reproduce the full set of time constants that the old perl script kept cached values for (24, 32, 48, 64, 100, 122, 128, 200, 250, 256, 300, 512, 1000, 1024, and 1200 hz), I found a bug. The first patch is miscalculating USEC_TO_HZ_ADJ32 for 24 HZ and 122 HZ. All the other values are fine.) It's an integer overflow. The GCD of 24 and 1000000 is 8, so that's 17 significant bits with a shift of 47... which is exactly 64 bits, but the math is _signed_, so it goes boing. For the record, the reason I can't just pregenerate all these suckers on a system that's got an arbitrary precision calculator (ala dc) and then just ship the resulting header files (more or less the what the first version of that first patch did) is that some architectures (arm omap and and arm at91) allow you to enter arbitrary HZ values in kconfig. (Their help text says that in many cases values that aren't powers of two won't work, but nothing enforces this.) So if we didn't have the capability to dynamically generate these, you could enter a .config value that would break the build. I'd be willing to use dc in the script if A) it was mentioned in SUSv3 (it's not, although bc is), B) the version of dc in busybox wasn't crap (it's floating point rather than arbitrary precision, and doesn't implement the left shift operator). The reason I'm not looking more closely at what SUSv3 has to say about bc is that A) it claims to be an entire programming language, which is definitely overkill for this B) busybox hasn't bothered to implement it so it can't be all that widely used anymore. I'll fix this and resubmit, it just wasn't ready last night. (If the merge window is closing soon I could resubmit the other two with Sam's suggestions and resubmit this one next time 'round, but it was only a couple days to write in the first place, once I finally figured out what the perl version was trying to _do_...) I believe ADJ32 is the only operation with any potential to overflow. The pathological case for SHIFT is HZ 1, which for USEC conversions would give a shift around 52 (32 significant bits plus 20 bits to divide by 1000000), but MUL32 still wouldn't overflow because the shift loop stops when it finds 32 significant bits, and any larger HZ value would produce a smaller shift. The problem with ADJ32 is it uses the MUL32 shift value, so a small $TO (24 HZ) with a big $FROM (1000000 USEC, 19 signficant bits) and a small Greatest Common Denominator (in this case 8) can overflow 64 bits. Pathological case is still HZ 1. (Or any other smallish prime number.) If I make that work, everything else has to. So anyway, it's not _arbitrary_ precision math. It's more like 32+20+20=72 bits, and I can probably fake that pretty easily by breaking a couple of operations into two stages... Fallen a bit behind on the thread since I noticed this and went off to code, I'll try to catch up later today. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-03 19:46 ` Rob Landley @ 2009-01-03 20:10 ` Sam Ravnborg 2009-01-03 20:50 ` H. Peter Anvin 2009-01-04 1:45 ` Rob Landley 2009-01-04 0:44 ` Robert Hancock 1 sibling, 2 replies; 135+ messages in thread From: Sam Ravnborg @ 2009-01-03 20:10 UTC (permalink / raw) To: Rob Landley Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin > > I'll fix this and resubmit, it just wasn't ready last night. (If the merge > window is closing soon I could resubmit the other two with Sam's suggestions > and resubmit this one next time 'round, but it was only a couple days to write > in the first place, once I finally figured out what the perl version was > trying to _do_...) For kbuild only fixes and trivial stuff will be merged until next merge window. Neither of the three patches fall into that category. With respect to your three patches the plan is to: - add the updated timeconst patch to kbuild-next - add the updated cpu-feature patch to kbuild-next - the patch touching headers_install will not be merged. The way forward for headers_install is to combine the unifdef feature and the header modifications. And this must be in a single program that can process all headers in one go so the install process becomes so fast that we do not worry about if it was done before or not. Then we can avoid all the .* files in the directory where we isntall the headers. The program is a prime candidate for a small C program and I hope someone can take the challenge to write it. Migrating from perl to shell does not help us here and the shell version was less readable than the perl version. Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-03 20:10 ` Sam Ravnborg @ 2009-01-03 20:50 ` H. Peter Anvin 2009-01-04 1:47 ` Rob Landley 2009-01-04 1:45 ` Rob Landley 1 sibling, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-01-03 20:50 UTC (permalink / raw) To: Sam Ravnborg Cc: Rob Landley, Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton Sam Ravnborg wrote: > With respect to your three patches the plan is to: > - add the updated timeconst patch to kbuild-next If you add this, you take the responsibility for the breakages that will occur. The reason his patch is "simpler" is because he removes the arbitrary-precision arithmetic, and simply hopes that the system utilities that he uses uses an integer size which happens to be big enough. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-03 20:50 ` H. Peter Anvin @ 2009-01-04 1:47 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:47 UTC (permalink / raw) To: H. Peter Anvin Cc: Sam Ravnborg, Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton On Saturday 03 January 2009 14:50:43 H. Peter Anvin wrote: > Sam Ravnborg wrote: > > With respect to your three patches the plan is to: > > - add the updated timeconst patch to kbuild-next > > If you add this, you take the responsibility for the breakages that will > occur. The reason his patch is "simpler" is because he removes the > arbitrary-precision arithmetic, and simply hopes that the system > utilities that he uses uses an integer size which happens to be big enough. Actually once I _noticed_ that ADJ32 could overflow (none of the other values can), it was fairly easy to prevent it from doing so. I added some comments analyzing the pathological case (HZ 1, assuming nobody's interested in HZ>1000000). > -hpa Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-03 20:10 ` Sam Ravnborg 2009-01-03 20:50 ` H. Peter Anvin @ 2009-01-04 1:45 ` Rob Landley 2009-01-04 8:09 ` Sam Ravnborg 1 sibling, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:45 UTC (permalink / raw) To: Sam Ravnborg Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin On Saturday 03 January 2009 14:10:59 Sam Ravnborg wrote: > > I'll fix this and resubmit, it just wasn't ready last night. (If the > > merge window is closing soon I could resubmit the other two with Sam's > > suggestions and resubmit this one next time 'round, but it was only a > > couple days to write in the first place, once I finally figured out what > > the perl version was trying to _do_...) > > For kbuild only fixes and trivial stuff will be merged until next merge > window. Neither of the three patches fall into that category. *shrug* I poke my head into kernel development every few months, and have just enough familiarity with it to remember that "changes go in during the merge window". Seemed a good time to post 'em. > With respect to your three patches the plan is to: > - add the updated timeconst patch to kbuild-next > - add the updated cpu-feature patch to kbuild-next > > - the patch touching headers_install will not be merged. > The way forward for headers_install is to combine the > unifdef feature and the header modifications. Since you're turning down an existing patch in favor of a theoretical patch, I assume you have plans to do this yourself? > And this must be in a single program that can process > all headers in one go so the install process becomes so fast > that we do not worry about if it was done before or not. > Then we can avoid all the .* files in the directory > where we isntall the headers. What if they run out of disk space halfway through writing a file and thus it creates a short file (or a 0 length file where the dentry was created but no blocks could be allocated for the write)? I expected headers_install to overwrite destination files and create directories with -p, so if you interrupt it you can just re-run it again with the same arguments and it could install everything again cleanly over existing partial output. I take it this isn't what's happening, or that isn't sufficient somehow? I didn't look too closely at what the makefile was doing (it makes my eyes bleed), I just rewrote the perl script and changed the call. I could try to upgrade the script to not need the makefile to tell it what files to work on, and just take the appropriate top level include directory and the output directory and figure out which files it needs to operate on by itself so it _does_ work that way. Figuring out where the make file is getting this info from now shouldn't be too much harder than reading perl scripts and figuring out what they're doing. Not during this merge window, though. > The program is a prime candidate for a small C program > and I hope someone can take the challenge to write it. Good luck with that. Having written most of the busybox sed implementation, and before that having written my own regex implementation back under OS/2, I've pretty much gotten my fill of doing regexes in C, at least without a good reason. I suppose if I was feeling really bored I could try to implement unifdef as a sed script, but the only way to get a single invocation of sed to work on a bunch of individual files coherently is to use -i mode, which A) ain't in susv3 (although busybox sed supports it), B) would involve a cp of all the files to the destination first, which is kind of ugly. > Migrating from perl to shell does not help us here > and the shell version was less readable than the perl version. The shell version is less readable but you never noticed that the perl version isn't using its $ARCH argument? *shrug* Ok. I can try to make the shell version more readable, and more powerful. It's already noticeably faster than the perl version. I have no objections to making unifdef do all of this, I just haven't got any interest either. > Sam Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 1:45 ` Rob Landley @ 2009-01-04 8:09 ` Sam Ravnborg 2009-01-04 20:19 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: Sam Ravnborg @ 2009-01-04 8:09 UTC (permalink / raw) To: Rob Landley Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin On Sat, Jan 03, 2009 at 07:45:34PM -0600, Rob Landley wrote: > > With respect to your three patches the plan is to: > > - add the updated timeconst patch to kbuild-next > > - add the updated cpu-feature patch to kbuild-next > > > > - the patch touching headers_install will not be merged. > > The way forward for headers_install is to combine the > > unifdef feature and the header modifications. > > Since you're turning down an existing patch in favor of a theoretical patch, I > assume you have plans to do this yourself? If noone else beats me I will do so - yes. > > > And this must be in a single program that can process > > all headers in one go so the install process becomes so fast > > that we do not worry about if it was done before or not. > > Then we can avoid all the .* files in the directory > > where we isntall the headers. > > What if they run out of disk space halfway through writing a file and thus it > creates a short file (or a 0 length file where the dentry was created but no > blocks could be allocated for the write)? Then they fail and make will know. Then may leave a file or 100 but it still failed. At next run everything will be done right assuming the culprint has been fixed. > I can try to make the shell version more readable, and more powerful. It's > already noticeably faster than the perl version. I have no objections to > making unifdef do all of this, I just haven't got any interest either. I have no interest in merging a shell version. I clearly expressed above that we need a _single_ program doing all of the preparations and we do not need a reimplmentatio of the current headers_install. I also explained why. Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 8:09 ` Sam Ravnborg @ 2009-01-04 20:19 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 20:19 UTC (permalink / raw) To: Sam Ravnborg Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin On Sunday 04 January 2009 02:09:31 Sam Ravnborg wrote: > On Sat, Jan 03, 2009 at 07:45:34PM -0600, Rob Landley wrote: > > Since you're turning down an existing patch in favor of a theoretical > > patch, I assume you have plans to do this yourself? > > If noone else beats me I will do so - yes. Ok. > > > And this must be in a single program that can process > > > all headers in one go so the install process becomes so fast > > > that we do not worry about if it was done before or not. > > > Then we can avoid all the .* files in the directory > > > where we isntall the headers. > > > > What if they run out of disk space halfway through writing a file and > > thus it creates a short file (or a 0 length file where the dentry was > > created but no blocks could be allocated for the write)? > > Then they fail and make will know. Then may leave a file or 100 > but it still failed. At next run everything will be done right > assuming the culprint has been fixed. Ok, so the important thing is propagating failures up to the exit code, then? When making this patch I hit a problem that the exit code of "unifdef" seems to depend on whether it found anything to remove within the file it was processing, so when I changed the caller to actually care about its exit code it spontaneously aborted. Fixing that probably does require changing unifdef.c. > > I can try to make the shell version more readable, and more powerful. > > It's already noticeably faster than the perl version. I have no > > objections to making unifdef do all of this, I just haven't got any > > interest either. > > I have no interest in merging a shell version. *shrug* Ok. I await your C version, and have a workable patch meeting my own needs in the meantime. Thanks, Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-03 19:46 ` Rob Landley 2009-01-03 20:10 ` Sam Ravnborg @ 2009-01-04 0:44 ` Robert Hancock 2009-01-04 1:39 ` David Brownell 2009-01-04 3:05 ` Rob Landley 1 sibling, 2 replies; 135+ messages in thread From: Robert Hancock @ 2009-01-04 0:44 UTC (permalink / raw) To: Rob Landley Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg Rob Landley wrote: > For the record, the reason I can't just pregenerate all these suckers on a > system that's got an arbitrary precision calculator (ala dc) and then just > ship the resulting header files (more or less the what the first version of > that first patch did) is that some architectures (arm omap and and arm at91) > allow you to enter arbitrary HZ values in kconfig. (Their help text says that > in many cases values that aren't powers of two won't work, but nothing > enforces this.) So if we didn't have the capability to dynamically generate > these, you could enter a .config value that would break the build. Is there a good reason that these archs allow you enter arbitrary HZ values? The use case for using custom HZ values at all nowadays seems fairly low now that dynticks is around (if that arch supports it anyway), let alone being able to specify wierd obscure values for it. Especially if nothing can ensure that all values it allows will actually result in a functional kernel.. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 0:44 ` Robert Hancock @ 2009-01-04 1:39 ` David Brownell 2009-01-04 3:05 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: David Brownell @ 2009-01-04 1:39 UTC (permalink / raw) To: Robert Hancock Cc: Rob Landley, Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Saturday 03 January 2009, Robert Hancock wrote: > Rob Landley wrote: > > ... some architectures (arm omap and and arm at91) > > allow you to enter arbitrary HZ values in kconfig. (Their help text says that > > in many cases values that aren't powers of two won't work, but nothing > > enforces this.) > > Is there a good reason that these archs allow you enter arbitrary HZ > values? Power-of-two can be desirable when using a 32 KiHz oscillator, because other values accumulate rounding errors ... you can't make 100 Hz, or 250 Hz, or 300 Hz, or 1000 Hz, by a binary division of 32 KiHz. Other values were supported to help work around stupid software making bad assumptions about HZ. IMO, enforcing power-of-two would be better; that software breaks with dyntick anyway, and needs fixing. > The use case for using custom HZ values at all nowadays seems > fairly low now that dynticks is around (if that arch supports it > anyway), A better argument would be that GENERIC_TIME exists (and works on OMAP and AT91), which avoids some flavors of rounding error. ISTR those CONFIG_HZ options predate GENERIC_TIME support. However, the issue remains that most kernel times are measured in jiffies not ktime_t -- they're easier and more efficient, all those 64-bit multiplies can hurt on ARM (32-bit, non-GHz) -- so it's still good to be able to ensure that jiffies-centric logic won't always be inserting easily avoidable errors. - Dave ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 0:44 ` Robert Hancock 2009-01-04 1:39 ` David Brownell @ 2009-01-04 3:05 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 3:05 UTC (permalink / raw) To: Robert Hancock Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Saturday 03 January 2009 18:44:58 Robert Hancock wrote: > Rob Landley wrote: > > For the record, the reason I can't just pregenerate all these suckers on > > a system that's got an arbitrary precision calculator (ala dc) and then > > just ship the resulting header files (more or less the what the first > > version of that first patch did) is that some architectures (arm omap and > > and arm at91) allow you to enter arbitrary HZ values in kconfig. (Their > > help text says that in many cases values that aren't powers of two won't > > work, but nothing enforces this.) So if we didn't have the capability to > > dynamically generate these, you could enter a .config value that would > > break the build. > > Is there a good reason that these archs allow you enter arbitrary HZ > values? Not that I've noticed, no. But you should ask Thomas Gleixner about that about that, I'm not a domain expert... > The use case for using custom HZ values at all nowadays seems > fairly low now that dynticks is around (if that arch supports it > anyway), let alone being able to specify wierd obscure values for it. And high performance event timers. A kernel can have more than one time source these days... Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 16:04 ` Matthieu CASTET 2009-01-03 19:46 ` Rob Landley @ 2009-01-04 1:32 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:32 UTC (permalink / raw) To: Matthieu CASTET Cc: Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Friday 02 January 2009 10:04:08 Matthieu CASTET wrote: > Rob Landley a écrit : > > On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote: > >> On Friday 02 of January 2009, Rob Landley wrote: > >> > >> Heh, > > > > I believe all three scripts run under dash and busybox ash. (The > > timeconst.sh one needs 64 bit math which dash only provides on 64 bit > > hosts, which is a regression from Red Hat 9 in 2003 by the way. > > With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for > 32 bit hosts : > $ uname -m > i686 > $ dash -c 'echo $((1<<32))' > 4294967296 > > > Matthieu Alas, my attempt to install a 32 bit version of xubuntu 8.10 under qemu hung at "Scanning files: 15%", and has been there for an hour now. I'll have to take your word for it. (All three scripts work fine under 64 bit dash.) I encountered one bug in busybox, which I pinged that list about, but otherwise busybox ash works on 'em all too. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 8:07 PATCH [0/3]: Simplify the kernel build by removing perl Rob Landley ` (3 preceding siblings ...) 2009-01-02 9:26 ` PATCH [0/3]: Simplify the kernel build by removing perl Arkadiusz Miskiewicz @ 2009-01-02 9:50 ` Paul Mundt 2009-01-02 10:32 ` Mark Miller ` (3 more replies) 2009-01-03 22:54 ` Leon Woestenberg 2009-01-04 1:24 ` PATCH [0/3]: Simplify the kernel build by removing perl v2 Rob Landley 6 siblings, 4 replies; 135+ messages in thread From: Paul Mundt @ 2009-01-02 9:50 UTC (permalink / raw) To: Rob Landley Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Fri, Jan 02, 2009 at 02:07:28AM -0600, Rob Landley wrote: > Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) > building a Linux kernel never required perl to be installed on the build > system. (Various development and debugging scripts were written in perl and > python and such, but they weren't involved in actually building a kernel.) > Building a kernel before 2.6.25 could be done with a minimal system built from > gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing > else. (Embedded developers creating clean cross compile environments that > won't leak bits of the host system into the target, or bootstrapping native > development environments to run on target hardware or under emulators, tend to > care about this sort of thing.) > > The perl checkin for 2.6.25 was the camel's nose under the tent flap, and > since then two more instances of perl have shown up in the core kernel build. > This patch series removes the three required to build a basic kernel for qemu > for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4, and of > course x86 and x86-64), replacing them with shell scripts. Misguided rhetoric aside, what does this actually accomplish? If folks add meaningful tools in to the kernel that require python, and it is generally regarded as being fairly ubiquitous, I can't imagine there being any substantiated objections against it. Your main reasons against inclusion of perl seem to be that there is no realistic expectation for target systems that will be self-hosting will have perl included, or the inherent complexity in maintaining a coherent cross compiling environment. Both of these things are issues with your own environment, and in no way are these representative of the embedded development community at large. Now with that out of the way, this entire series fundamentally fails to convert half of the perl scripts shipped with the kernel today, some that are required for build depending on Kconfig options, and others that are simply useful tools for self-hosted environments. Simply converting a couple of scripts over you find objectionable is certainly fine if there is a real benefit in doing so, but this doesn't actually accomplish your goal of removing the perl dependency. Ignoring the compile-time dependencies that you overlooked, what you define as "development and debugging" scripts are still an integral part of the system, unless you are trying to argue that embedded developers have no interest in things like checkstack due to the trouble of trying to get perl built. Until you can post a series that converts all of scripts/*.pl in its entirety, you have failed to address the fundamental reason why perl is used in the first place. Trying to toss bizarre policy statements around regarding things you personally find objectionable without any coherent technical argument to the contrary is of no constructive use whatsoever. The kernel is and always has been about using the right tool for the job, not a matter of dictating what tools you must use in order to accomplish a task you are interested in. Common sense does apply here though, so this might be a more daunting task for some than others. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:50 ` Paul Mundt @ 2009-01-02 10:32 ` Mark Miller 2009-01-02 10:57 ` Paul Mundt 2009-01-02 12:44 ` Rob Landley ` (2 subsequent siblings) 3 siblings, 1 reply; 135+ messages in thread From: Mark Miller @ 2009-01-02 10:32 UTC (permalink / raw) To: Paul Mundt Cc: Rob Landley, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Jan 2, 2009, at 3:50 AM, Paul Mundt wrote: > On Fri, Jan 02, 2009 at 02:07:28AM -0600, Rob Landley wrote: >> Before 2.6.25 (specifically git >> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) >> building a Linux kernel never required perl to be installed on the >> build >> system. (Various development and debugging scripts were written in >> perl and >> python and such, but they weren't involved in actually building a >> kernel.) >> Building a kernel before 2.6.25 could be done with a minimal system >> built from >> gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, >> and nothing >> else. (Embedded developers creating clean cross compile >> environments that >> won't leak bits of the host system into the target, or >> bootstrapping native >> development environments to run on target hardware or under >> emulators, tend to >> care about this sort of thing.) >> >> The perl checkin for 2.6.25 was the camel's nose under the tent >> flap, and >> since then two more instances of perl have shown up in the core >> kernel build. >> This patch series removes the three required to build a basic >> kernel for qemu >> for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4, >> and of >> course x86 and x86-64), replacing them with shell scripts. > > Misguided rhetoric aside, what does this actually accomplish? If folks > add meaningful tools in to the kernel that require python, and it is > generally regarded as being fairly ubiquitous, I can't imagine there > being any substantiated objections against it. And if the said meaningful tools introduce complex dependencies, then there should be an open discussion as to why exactly we need those tools as opposed to a simpler implementation. > Your main reasons against inclusion of perl seem to be that there is > no > realistic expectation for target systems that will be self-hosting > will > have perl included, or the inherent complexity in maintaining a > coherent > cross compiling environment. Both of these things are issues with your > own environment, and in no way are these representative of the > embedded > development community at large. I feel that if you attempted to look for discussions on "cross- compiling perl", you will meet with a variety of complaints on what a nightmare it is to do so in a sandboxed environment. > Now with that out of the way, this entire series fundamentally fails > to > convert half of the perl scripts shipped with the kernel today, some > that > are required for build depending on Kconfig options, and others that > are > simply useful tools for self-hosted environments. Simply converting a > couple of scripts over you find objectionable is certainly fine if > there > is a real benefit in doing so, but this doesn't actually accomplish > your > goal of removing the perl dependency. From what I can tell, it allows one to fully build the Linux kernel without Perl. Without these series of patches, to actually *build* the kernel, one requires Perl. Unless there is an alternate way to do "make headers_install" that I am not seeing, that appears to now be done in Perl. All the other Perl scripts were merely nice to have, not necessary to build a Linux kernel. > Ignoring the compile-time dependencies that you overlooked, what you > define as "development and debugging" scripts are still an integral > part > of the system, unless you are trying to argue that embedded developers > have no interest in things like checkstack due to the trouble of > trying > to get perl built. Do I need that to compile a kernel? No. > Until you can post a series that converts all of scripts/*.pl in its > entirety, you have failed to address the fundamental reason why perl > is > used in the first place. Trying to toss bizarre policy statements > around > regarding things you personally find objectionable without any > coherent > technical argument to the contrary is of no constructive use > whatsoever. Perl was not required to build the Linux kernel. Now it is. It adds another dependency to the Linux kernel. Requiring bash is far less a dependency that Perl is. > The kernel is and always has been about using the right tool for the > job, > not a matter of dictating what tools you must use in order to > accomplish > a task you are interested in. Common sense does apply here though, so > this might be a more daunting task for some than others. How is Perl a better tool for the job than what currently bash and other standard utilities already offer? -- Mark Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 10:32 ` Mark Miller @ 2009-01-02 10:57 ` Paul Mundt 2009-01-02 12:11 ` Mark Miller 0 siblings, 1 reply; 135+ messages in thread From: Paul Mundt @ 2009-01-02 10:57 UTC (permalink / raw) To: Mark Miller Cc: Rob Landley, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Fri, Jan 02, 2009 at 04:32:42AM -0600, Mark Miller wrote: > On Jan 2, 2009, at 3:50 AM, Paul Mundt wrote: > >Misguided rhetoric aside, what does this actually accomplish? If folks > >add meaningful tools in to the kernel that require python, and it is > >generally regarded as being fairly ubiquitous, I can't imagine there > >being any substantiated objections against it. > > And if the said meaningful tools introduce complex dependencies, then > there should be an open discussion as to why exactly we need those > tools as opposed to a simpler implementation. > Complex is relative, something that is fairly ubiquitious can hardly be labelled as complex, regardless of whether historically people have had issues with that dependency in certain spaces. In any event, simplifying things is always good, but this open discussion thing is pure fancy, since when was the kernel a democracy? > >Your main reasons against inclusion of perl seem to be that there is > >no realistic expectation for target systems that will be self-hosting > >will have perl included, or the inherent complexity in maintaining a > >coherent cross compiling environment. Both of these things are issues > >with your own environment, and in no way are these representative of > >the embedded development community at large. > > I feel that if you attempted to look for discussions on "cross- > compiling perl", you will meet with a variety of complaints on what a > nightmare it is to do so in a sandboxed environment. > I've had to deal with cross compiling perl for over a decade, in all of its various forms, in all manner of embedded applications, so please tell someone who cares. There are various options around this headache today, as there have been for ages (though the options these days are rather less painful), and if you felt like attempting to look for discussions on those rather than trying to push this silly matter perhaps you might come up with something. The key thing you hit on is that there are a variety of complaints, and that is all they have ever been. If your system is so constrained, you shouldn't be doing builds on it in the first place. You certainly won't be able to build anything in your crippled environment outside of some simple applications anyways, this isn't a valid technical reason for keeping the kernel a policy stranglehold. > >Now with that out of the way, this entire series fundamentally fails > >to convert half of the perl scripts shipped with the kernel today, > >some that are required for build depending on Kconfig options, > [snip] > From what I can tell, it allows one to fully build the Linux kernel > without Perl. Wrong, re-read what I said and try again. > >Ignoring the compile-time dependencies that you overlooked, what you > >define as "development and debugging" scripts are still an integral > >part of the system, unless you are trying to argue that embedded > >developers have no interest in things like checkstack due to the > >trouble of trying to get perl built. > > Do I need that to compile a kernel? No. > Compile-time dependencies you do, yes. > Perl was not required to build the Linux kernel. Now it is. It adds > another dependency to the Linux kernel. Requiring bash is far less a > dependency that Perl is. > Perhaps so, but that is largely irrelevant. Moving off of the bash dependency is a lot more trivial than moving off of the perl one. The kernel bashisms are fairly marginal, and if someone were to do the leg work for that, it might even be accepted. Getting rid of perl is an uphill battle for no visible gain. People will continue to write and add scripts both in bash and perl on a routine basis regardless. > >The kernel is and always has been about using the right tool for the > >job, not a matter of dictating what tools you must use in order to > >accomplish a task you are interested in. Common sense does apply here > >though, so this might be a more daunting task for some than others. > > How is Perl a better tool for the job than what currently bash and > other standard utilities already offer? > How is bash a better tool for than job than what sed and posix shell already offer? Yes, we can reduce our dependencies to the bare minimum, but that is not constructive for the folks that are actually writing the scripts in the first place. Likewise, this is not even a real problem in the embedded developer demographic, the only place this is a problem is in specially stripped distributions or people that don't want to go through the pain of cross compiling perl. None of which is of any concern. If people are going to write useful things that are reasonably expected to be standard on build machines, there is no reason to restrict what tools they are permitted to use. If you have a personal vendetta against something that is fairly standard, that is entirely your own personal problem, you can choose to deal with it or patch out of tree for your own crippled environment (assuming patch isn't too much of an obscure dependency). ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 10:57 ` Paul Mundt @ 2009-01-02 12:11 ` Mark Miller 0 siblings, 0 replies; 135+ messages in thread From: Mark Miller @ 2009-01-02 12:11 UTC (permalink / raw) To: Paul Mundt Cc: Rob Landley, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Jan 2, 2009, at 4:57 AM, Paul Mundt wrote: > On Fri, Jan 02, 2009 at 04:32:42AM -0600, Mark Miller wrote: >> On Jan 2, 2009, at 3:50 AM, Paul Mundt wrote: >>> Misguided rhetoric aside, what does this actually accomplish? If >>> folks >>> add meaningful tools in to the kernel that require python, and it is >>> generally regarded as being fairly ubiquitous, I can't imagine there >>> being any substantiated objections against it. >> >> And if the said meaningful tools introduce complex dependencies, then >> there should be an open discussion as to why exactly we need those >> tools as opposed to a simpler implementation. >> > Complex is relative, something that is fairly ubiquitious can hardly > be > labelled as complex, regardless of whether historically people have > had > issues with that dependency in certain spaces. In any event, > simplifying > things is always good, but this open discussion thing is pure fancy, > since when was the kernel a democracy? I'm ignoring the bait. >>> Your main reasons against inclusion of perl seem to be that there is >>> no realistic expectation for target systems that will be self- >>> hosting >>> will have perl included, or the inherent complexity in maintaining a >>> coherent cross compiling environment. Both of these things are >>> issues >>> with your own environment, and in no way are these representative of >>> the embedded development community at large. >> >> I feel that if you attempted to look for discussions on "cross- >> compiling perl", you will meet with a variety of complaints on what a >> nightmare it is to do so in a sandboxed environment. >> > I've had to deal with cross compiling perl for over a decade, in all > of > its various forms, in all manner of embedded applications, so please > tell > someone who cares. Ignoring this as well. > There are various options around this headache today, > as there have been for ages (though the options these days are rather > less painful), and if you felt like attempting to look for > discussions on > those rather than trying to push this silly matter perhaps you might > come > up with something. And ignoring this too. > The key thing you hit on is that there are a variety of complaints, > and > that is all they have ever been. If your system is so constrained, you > shouldn't be doing builds on it in the first place. You certainly > won't > be able to build anything in your crippled environment outside of some > simple applications anyways, this isn't a valid technical reason for > keeping the kernel a policy stranglehold. I merely don't like seeing another dependency added when there's no logical reason to do it, other than "why not". And there have been reasons given for the "not". Your reply, seems merely to be, "Because." >>> Now with that out of the way, this entire series fundamentally fails >>> to convert half of the perl scripts shipped with the kernel today, >>> some that are required for build depending on Kconfig options, >> > [snip] > >> From what I can tell, it allows one to fully build the Linux kernel >> without Perl. > > Wrong, re-read what I said and try again. I have done so. And I restate, without these patches, Perl is required *for any install*. Your logic seems to be, if Perl is required for any part of doing something with the Linux kernel, then there is no reason it should not be allowed in all parts. With this logic, soon the Linux kernel will require so many dependencies, that even a minimal system to rebuild itself under itself will comprise 200MB for stripped binaries. Right now, it's down to about 20ish. With that, you also require more expertise in maintaing these variety of tools. You have given no logical reason on why we need to add more complexity to the kernel, when previous tools have managed to deal with this task for over ten years without requiring Perl. >>> Ignoring the compile-time dependencies that you overlooked, what you >>> define as "development and debugging" scripts are still an integral >>> part of the system, unless you are trying to argue that embedded >>> developers have no interest in things like checkstack due to the >>> trouble of trying to get perl built. >> >> Do I need that to compile a kernel? No. >> > Compile-time dependencies you do, yes. > >> Perl was not required to build the Linux kernel. Now it is. It adds >> another dependency to the Linux kernel. Requiring bash is far less a >> dependency that Perl is. >> > Perhaps so, but that is largely irrelevant. Moving off of the bash > dependency is a lot more trivial than moving off of the perl one. The > kernel bashisms are fairly marginal, and if someone were to do the leg > work for that, it might even be accepted. Getting rid of perl is an > uphill battle for no visible gain. People will continue to write and > add > scripts both in bash and perl on a routine basis regardless. > >>> The kernel is and always has been about using the right tool for the >>> job, not a matter of dictating what tools you must use in order to >>> accomplish a task you are interested in. Common sense does apply >>> here >>> though, so this might be a more daunting task for some than others. >> >> How is Perl a better tool for the job than what currently bash and >> other standard utilities already offer? >> > How is bash a better tool for than job than what sed and posix shell > already offer? Yes, we can reduce our dependencies to the bare > minimum, > but that is not constructive for the folks that are actually writing > the > scripts in the first place. > > Likewise, this is not even a real problem in the embedded developer > demographic, the only place this is a problem is in specially stripped > distributions or people that don't want to go through the pain of > cross > compiling perl. None of which is of any concern. Yet again, you have a habit of writing off a large segment of the embedded development population that you seem to think doesn't exist. > If people are going to write useful things that are reasonably > expected > to be standard on build machines, there is no reason to restrict what > tools they are permitted to use. > If you have a personal vendetta against > something that is fairly standard, that is entirely your own personal > problem, you can choose to deal with it or patch out of tree for > your own > crippled environment (assuming patch isn't too much of an obscure > dependency). Last I checked, patch was a part of busybox. Or is that yet another realm of embedded development that you are casting off into the depths of nowhere? Honestly, it seems you have a personal vendetta against those not wanting to add another dependency. -- Mark Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:50 ` Paul Mundt 2009-01-02 10:32 ` Mark Miller @ 2009-01-02 12:44 ` Rob Landley 2009-01-02 17:25 ` Wookey 2009-01-03 14:59 ` Wolfgang Denk 3 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-02 12:44 UTC (permalink / raw) To: Paul Mundt Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Friday 02 January 2009 03:50:23 Paul Mundt wrote: > On Fri, Jan 02, 2009 at 02:07:28AM -0600, Rob Landley wrote: > > The perl checkin for 2.6.25 was the camel's nose under the tent flap, and > > since then two more instances of perl have shown up in the core kernel > > build. This patch series removes the three required to build a basic > > kernel for qemu for the targets I've tested (arm, mips, powerpc, sparc, > > m68k, sh4, and of course x86 and x86-64), replacing them with shell > > scripts. > > Misguided rhetoric aside, what does this actually accomplish? If folks > add meaningful tools in to the kernel that require python, and it is > generally regarded as being fairly ubiquitous, I can't imagine there > being any substantiated objections against it. I think bloat-o-meter is a marvelous tool, and I'm a big fan of python. But I don't think you shouldn't have to run that to compile a kernel either, largely because not needing it for the first 17 years or so implied living without this requirement could be done, sustainably even. There's a difference between a development workstation and a dedicated build system. Requiring you to install X11 plus qt on the headless build server cranking out nightly snapshots in order to run the configure stage of the kernel build would be silly. But this is not an argument for ripping out "make xconfig" from the kernel. Spot the difference? > Your main reasons against inclusion of perl seem to be that there is no > realistic expectation for target systems that will be self-hosting will > have perl included, or the inherent complexity in maintaining a coherent > cross compiling environment. I'm saying it's a major new environmental dependency that went in fairly recently and largely without comment, and it causes real world headaches for real people, of which I am only one. If you don't think environmental dependencies are a problem, I welcome you to attempt to build open office. (Even the gentoo guys gave up on that one and just started shipping a prebuilt binary.) I think large amounts of complexity start with small amounts of complexity that grow. Complexity is inevitable, but there should be a _reason_ for increases in it. > Both of these things are issues with your > own environment, and in no way are these representative of the embedded > development community at large. > > Now with that out of the way, this entire series fundamentally fails to > convert half of the perl scripts shipped with the kernel today, some that > are required for build depending on Kconfig options, and others that are > simply useful tools for self-hosted environments. I didn't say the job was finished. These are just the ones I've already personally hit, and thus A) needed to rewrite to build the kernel in my build environment, B) have a handy test case for. > Simply converting a > couple of scripts over you find objectionable is certainly fine if there > is a real benefit in doing so, but this doesn't actually accomplish your > goal of removing the perl dependency. A) It's a start. B) It works for me, and builds the .configs I've personally needed so far. > Ignoring the compile-time dependencies that you overlooked, what you > define as "development and debugging" scripts are still an integral part > of the system, unless you are trying to argue that embedded developers > have no interest in things like checkstack due to the trouble of trying > to get perl built. Coming up with new patches and modifying the source is a different use for source code than going "./configure; make; make install". This is true for most open source software, I'd expect. Or are you implying that eclipse or Emacs are such great IDEs that being able to build outside of a GUI isn't interesting? The ability to build within an IDE should be allowed to preclude the ability to build without one? > Until you can post a series that converts all of scripts/*.pl in its > entirety, you have failed to address the fundamental reason why perl is > used in the first place. Never start anything unless you can finish it all in one go, eh? Last I heard the kernel guys tend to frown on people who wander off in their own corner for a year and then dump a massive rewrite on them. They seem to prefer the incremental "trail of breadcrumbs" approach. Release early, release often, incorporate feedback, keep at it. Or am I wrong? > Trying to toss bizarre policy statements around > regarding things you personally find objectionable without any coherent > technical argument to the contrary is of no constructive use whatsoever. Complexity is a cost, environmental dependencies are a form of complexity, if the benefit isn't worth the cost (or you can get the benefit without the cost) then you shouldn't pay the cost. I was unaware this was a controversial attitude? > The kernel is and always has been about using the right tool for the job, > not a matter of dictating what tools you must use in order to accomplish > a task you are interested in. And yet despite "unifdef" being an existing tool, the kernel developers didn't require you to install it on your host, but instead built their own version from source. (They didn't even require you to install a version of cpio on the host that understands -H newc, but instead built their own initramfs packer in C.) More recently, they integrated dtc (the device tree compiler). In each case, they made a balancing decision, and decided that integrating and maintaining a small C tool was better than having an environmental dependency. > Common sense does apply here though, so > this might be a more daunting task for some than others. Apparently so. I don't care if checkstack.pl is written in perl for the same reason I don't care if eclipse is written in java. They're not build dependencies. If I had to install gdb as a precondition of compiling the Linux kernel I'd think something was funny with that, which isn't saying I object to the existence of gdb or the ability to hook it up to kgdb or qemu or UML to poke at kernels. It's two different categories. It seems clear to _me_, anyway. But it is coming up on 7am and I haven't gotten to bed yet, so take that for what it's worth... Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:50 ` Paul Mundt 2009-01-02 10:32 ` Mark Miller 2009-01-02 12:44 ` Rob Landley @ 2009-01-02 17:25 ` Wookey 2009-01-02 18:01 ` Sam Ravnborg 2009-01-03 14:59 ` Wolfgang Denk 3 siblings, 1 reply; 135+ messages in thread From: Wookey @ 2009-01-02 17:25 UTC (permalink / raw) To: Embedded Linux mailing list Cc: Paul Mundt, Rob Landley, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On 2009-01-02 18:50 +0900, Paul Mundt wrote: > Your main reasons against inclusion of perl seem to be that there is no > realistic expectation for target systems that will be self-hosting will > have perl included, or the inherent complexity in maintaining a coherent > cross compiling environment. Both of these things are issues with your > own environment, and in no way are these representative of the embedded > development community at large. It may well be true that most embedded people cross-build kernels and use native perl on a fat build box, but there are plenty of situations where being able to build kernels without perl is useful. Given the simplicitly of these patches I can't see any reason not to put them in, and appreciate Rob's work on this. And if cross-building perl is really easy, as some in this thread claim, can someone fix the Debian packages to do it, because that would be really useful (it appears to be of similar complexity to cross-building gcc, requiring a 2-stage self-referential build). Wookey -- Principal hats: Balloonz - Toby Churchill - Aleph One - Debian http://wookware.org/ ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 17:25 ` Wookey @ 2009-01-02 18:01 ` Sam Ravnborg 2009-01-02 19:27 ` H. Peter Anvin 2009-01-03 19:48 ` Rob Landley 0 siblings, 2 replies; 135+ messages in thread From: Sam Ravnborg @ 2009-01-02 18:01 UTC (permalink / raw) To: Embedded Linux mailing list, Paul Mundt, Rob Landley, linux-kernel, Andrew Morton, H. Peter Anvin Hi Wookey. > Given the > simplicitly of these patches I can't see any reason not to put them > in Please do NOT do the mistake and think this the same thing. Rob's patch simplyfy the timecost stuff - and will be applied on this merit alone assuming comments will be addressed. But the serie rased anohter topic: shall we ban use of perl for generating a kernel. And this is what primary is discussed and the outcome of that discussion will not prevent patches that stands on their own to be applied. Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 18:01 ` Sam Ravnborg @ 2009-01-02 19:27 ` H. Peter Anvin 2009-01-04 1:35 ` Rob Landley 2009-01-03 19:48 ` Rob Landley 1 sibling, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-01-02 19:27 UTC (permalink / raw) To: Sam Ravnborg Cc: Embedded Linux mailing list, Paul Mundt, Rob Landley, linux-kernel, Andrew Morton Sam Ravnborg wrote: > Hi Wookey. > >> Given the >> simplicitly of these patches I can't see any reason not to put them >> in > > Please do NOT do the mistake and think this the same thing. > > Rob's patch simplyfy the timecost stuff - and will be applied on > this merit alone assuming comments will be addressed. > > But the serie rased anohter topic: shall we ban use of perl > for generating a kernel. > And this is what primary is discussed and the outcome of > that discussion will not prevent patches that stands on their > own to be applied. > My personal opinion on this is that this is ridiculous. Given that you need gcc, binutils, make etc. to build the kernel, and this is more than inherent, you have to have a pretty bloody strangely constrained system to disallow Perl, which is as close to a standard Unix utility you can get without making it into SuS. The only *real* motivation I have seen for this is a system that as far I can tell is nothing other than a toy, specifically designed to show off how little you need to build the kernel. In other words, it's not a practical application, it's a show-off art piece. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 19:27 ` H. Peter Anvin @ 2009-01-04 1:35 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:35 UTC (permalink / raw) To: H. Peter Anvin Cc: Sam Ravnborg, Embedded Linux mailing list, Paul Mundt, linux-kernel, Andrew Morton On Friday 02 January 2009 13:27:45 H. Peter Anvin wrote: > Sam Ravnborg wrote: > > Hi Wookey. > > > >> Given the > >> simplicitly of these patches I can't see any reason not to put them > >> in > > > > Please do NOT do the mistake and think this the same thing. > > > > Rob's patch simplyfy the timecost stuff - and will be applied on > > this merit alone assuming comments will be addressed. > > > > But the serie rased anohter topic: shall we ban use of perl > > for generating a kernel. > > And this is what primary is discussed and the outcome of > > that discussion will not prevent patches that stands on their > > own to be applied. > > My personal opinion on this is that this is ridiculous. Given that you > need gcc, binutils, make etc. to build the kernel, I believe Intel's icc builds the kernel, and tinycc previously built a subset of the kernel. The pcc and llvm/clang projects are getting close to being able to build the kernel. Ever since c99 came out, lots of gcc-isms with c99 equivalents have been swapped over, most of the rest is testing. > and this is more than > inherent, you have to have a pretty bloody strangely constrained system > to disallow Perl, which is as close to a standard Unix utility you can > get without making it into SuS. Please show me A) the standard perl implements, B) the second implementation of that standard ala IETF guidelines. > The only *real* motivation I have seen for this is a system that as far > I can tell is nothing other than a toy, specifically designed to show > off how little you need to build the kernel. In other words, it's not a > practical application, it's a show-off art piece. I'm glad you think my Firmware Linux project is a work of art, but if you'd like to hear directly from my users I can ask them to complain at you in person, if you like. I'm not sure what that would prove, though. When cross compiling, it's good to have as constrained an environment as possible, because otherwise bits of the host system leak into the target system. If you don't tightly control your cross compiling environment, it won't work. That's just about an axiom in embedded development. I know every single dependency my system has. I can list them, explicitly. I did this because it's very _USEFUL_ in this context. Add perl scripts that call cpan, and this is no longer true. > -hpa Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 18:01 ` Sam Ravnborg 2009-01-02 19:27 ` H. Peter Anvin @ 2009-01-03 19:48 ` Rob Landley 1 sibling, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-03 19:48 UTC (permalink / raw) To: Sam Ravnborg Cc: Embedded Linux mailing list, Paul Mundt, linux-kernel, Andrew Morton, H. Peter Anvin On Friday 02 January 2009 12:01:34 Sam Ravnborg wrote: > But the serie rased anohter topic: shall we ban use of perl > for generating a kernel. I dunno about "ban", but every time somebody adds perl to the "hot path" of the kernel build it breaks my build system, and I write a removal patch anyway. I have to maintain them anyway, so I might as well try to push 'em upstream. (I posted the first patch in this series twice before, once for 25 and then an updated version to the linux-embedded list for .26.) I didn't discover this topic independently. Somebody pinged me about it on freenode back in February, and several other people sent me private email about it, and it's been previously raised on several other mailing lists (such as busybox and uclibc ones). Unfortunately, most of the embedded developers I know aren't subscribed to linux-kernel. (You either do kernel development, or you do everything else. It's not really feasible to keep up with the guts of the kernel and uClibc and busybox and gcc and qemu and the current offerings of three different hardware vendors and whatever userspace application the board's supposed to run and your build system and what INSANE things your EEdiot electrical engineer decided to miswire this time and fighting off marketing's desire to switch everything over to WinCE because they can get their entire advertising budget subsidized and there's a trade show next week we're not READY for... Not only can none of 'em read a meaningful subset of linux-kernel anymore, but if you disappear into your own little niche for nine months, when you pop back up the kernel's all different and sometimes even the patch submission policy's migrated a bit. Heck, I'm three months behind reading the LWN kernel page myself, and that's no substitute for kernel-traffic, RIP...) Hopefully the cc: to linux-embedded is helping get more embedded guys involved in the discussion than just me. :) > And this is what primary is discussed and the outcome of > that discussion will not prevent patches that stands on their > own to be applied. The best way to get a patch applied is always for that patch to stand on its own merits. Larger agendas are secondary. Whether or not the kernel decides on a policy of keeping perl out of the kernel build's "hot path", I still need these patches for my own use, and plan to keep coming up with them and submitting them. I haven't removed ones that haven't broken my build yet, but just because I'm not using md today doesn't mean I won't _start_. (And if enough other people keep poking me about the kernel build I can tackle 'em to please them. I actually _do_ know some embedded developers using raid for network attached storage and video servers and such...) > Sam Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 9:50 ` Paul Mundt ` (2 preceding siblings ...) 2009-01-02 17:25 ` Wookey @ 2009-01-03 14:59 ` Wolfgang Denk 3 siblings, 0 replies; 135+ messages in thread From: Wolfgang Denk @ 2009-01-03 14:59 UTC (permalink / raw) To: Paul Mundt Cc: Rob Landley, Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Dear Paul Mundt, In message <20090102095023.GA28078@linux-sh.org> you wrote: > > Your main reasons against inclusion of perl seem to be that there is no > realistic expectation for target systems that will be self-hosting will > have perl included, or the inherent complexity in maintaining a coherent > cross compiling environment. Both of these things are issues with your > own environment, and in no way are these representative of the embedded > development community at large. I'm not sure how representative for the "embedded development community at large" your statement is. Just to add a data point to the statistice, I do agree with Rob's opinion about needing Per to build the Linux kernel. All other arguments aside, natively compiling the Linux kernel on a target board, especially with root file system mounted over NFS, has always been an excellent stress test (you may even call it regression test) for Linux running on some target hardware. Losing this ability just because some scripts are implemented in language FOO when plain shell will do as well is something we should try to avoid. Of course, YMMV. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de Include the success of others in your dreams for your own success. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-02 8:07 PATCH [0/3]: Simplify the kernel build by removing perl Rob Landley ` (4 preceding siblings ...) 2009-01-02 9:50 ` Paul Mundt @ 2009-01-03 22:54 ` Leon Woestenberg 2009-01-03 23:03 ` H. Peter Anvin 2009-01-04 1:24 ` PATCH [0/3]: Simplify the kernel build by removing perl v2 Rob Landley 6 siblings, 1 reply; 135+ messages in thread From: Leon Woestenberg @ 2009-01-03 22:54 UTC (permalink / raw) To: Rob Landley Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Hello all, On Fri, Jan 2, 2009 at 9:07 AM, Rob Landley <rob@landley.net> wrote: > Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) > building a Linux kernel never required perl to be installed on the build > system. (Various development and debugging scripts were written in perl and > python and such, but they weren't involved in actually building a kernel.) > Building a kernel before 2.6.25 could be done with a minimal system built from > gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing > else. (Embedded developers creating clean cross compile environments that > I agree with Rob that the amount of required dependencies should be kept to a minimum. If we only use 0.5% of a certain language (or: dependent package), then rather implement that 0.5% in the existing language. Dependencies very quickly become dependency hell. If A requires B, then A also inherits all (future) requirements of B, etc. etc. In my daily software development work with Linux and GNU software in general, 10% of it is spent fighting/removing these extremely "thin" or false depencies, so that it is usuable in embedded devices. Regards, -- Leon ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-03 22:54 ` Leon Woestenberg @ 2009-01-03 23:03 ` H. Peter Anvin 2009-01-04 0:37 ` Leon Woestenberg 2009-01-04 2:06 ` Rob Landley 0 siblings, 2 replies; 135+ messages in thread From: H. Peter Anvin @ 2009-01-03 23:03 UTC (permalink / raw) To: Leon Woestenberg Cc: Rob Landley, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg Leon Woestenberg wrote: > > I agree with Rob that the amount of required dependencies should be > kept to a minimum. > > If we only use 0.5% of a certain language (or: dependent package), > then rather implement that 0.5% in the existing language. > > Dependencies very quickly become dependency hell. If A requires B, > then A also inherits all (future) requirements of B, etc. etc. > > In my daily software development work with Linux and GNU software in > general, 10% of it is spent fighting/removing these extremely "thin" > or false depencies, so that it is usuable in embedded devices. > First of all, I largely consider this a joke. All real-life embedded kernel builds take place on hosted platforms; anything else seems to be done "just because it can be done", as a kind of show-off art project. Cute, but hardly worth impeding the rest of the kernel community for. We're not talking about general platform dependencies here, but build dependencies for the kernel. A platform that can build the kernel is not a small platform. Second of all, these patches are not fullworthy replacements. The original patch using bc had less dependencies, but bc failed on some platforms, mysteriously. The new patches have *more* environmental dependencies than that ever did. Third, if someone actually cares to do it right, I have a smallish bignum library at http://git.zytor.com/?p=lib/pbn.git;a=summary that might be a starting point. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-03 23:03 ` H. Peter Anvin @ 2009-01-04 0:37 ` Leon Woestenberg 2009-01-04 2:53 ` Rob Landley 2009-01-04 2:06 ` Rob Landley 1 sibling, 1 reply; 135+ messages in thread From: Leon Woestenberg @ 2009-01-04 0:37 UTC (permalink / raw) To: H. Peter Anvin Cc: Rob Landley, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg Hello, On Sun, Jan 4, 2009 at 12:03 AM, H. Peter Anvin <hpa@zytor.com> wrote: >> Dependencies very quickly become dependency hell. If A requires B, >> then A also inherits all (future) requirements of B, etc. etc. >> >> In my daily software development work with Linux and GNU software in >> general, 10% of it is spent fighting/removing these extremely "thin" >> or false depencies, so that it is usuable in embedded devices. >> > > First of all, I largely consider this a joke. All real-life embedded > kernel builds take place on hosted platforms; anything else seems to be > done "just because it can be done", as a kind of show-off art project. > Cute, but hardly worth impeding the rest of the kernel community for. > Let me explain why it is not a joke for me, although yes I agree it's a funny way of how software engineering works. My argument on thin dependencies indeed mostly holds for run-time dependencies (to reduce size) but also for build dependency (to reduce complexity)*. In general the right version of the right tool is not available on the build host. If I cross-build 30 packages all of which need a build-host-native perl during their build, consider the chance of these packages building with the one version of perl that lives on the system. It's near 0% for the average mix of packages. Yes, the host is fat enough so that we can build (say three) different versions of perl to accomodate the build. The truth often is, of the 30 build dependencies 25 of them are thin (build) dependencies. So yes, the say extra 10% of build tool dependency needed to make 0.5% of the build system work, can be solved on the fat build platform, but I do not always consider it a well engineered solution. For a (dangerous) metaphor: In mechanical engineering, if there are 30 joints using nuts and bolts, and a 31st is designed in, it is strongly prefered to make it a nut and bolt as well, even if using a screw would work equally well or even better. The overall costs (over complexity, tool dependency, maintainability) would be lower. Regards, -- Leon * Since Intel, since recently, pays another team of excellent people maintaining a system dealing with the (cross) build tool dependency hell, I rest my case. :-) ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 0:37 ` Leon Woestenberg @ 2009-01-04 2:53 ` Rob Landley 2009-01-04 3:38 ` Markus Heidelberg 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-04 2:53 UTC (permalink / raw) To: Leon Woestenberg Cc: H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Saturday 03 January 2009 18:37:12 Leon Woestenberg wrote: > Hello, > > On Sun, Jan 4, 2009 at 12:03 AM, H. Peter Anvin <hpa@zytor.com> wrote: > >> Dependencies very quickly become dependency hell. If A requires B, > >> then A also inherits all (future) requirements of B, etc. etc. > >> > >> In my daily software development work with Linux and GNU software in > >> general, 10% of it is spent fighting/removing these extremely "thin" > >> or false depencies, so that it is usuable in embedded devices. > > > > First of all, I largely consider this a joke. All real-life embedded > > kernel builds take place on hosted platforms; anything else seems to be > > done "just because it can be done", as a kind of show-off art project. > > Cute, but hardly worth impeding the rest of the kernel community for. > > Let me explain why it is not a joke for me, although yes I agree it's > a funny way of how software engineering works. > > My argument on thin dependencies indeed mostly holds for run-time > dependencies (to reduce size) but also for build dependency (to reduce > complexity)*. I usually just point to the gnucash 1.6 release as where this sort of thing leads if you ignore it long enough: http://lwn.net/2001/0614/ These days, a more modern example is the way that after even the gentoo folks gave up on trying to build openoffice (and shipped prebuilt binaries of it in their "build everything from source code" OS), Open Office's own developers described that project "profoundly sick" and "stagnating" ( http://developers.slashdot.org/article.pl?sid=08/12/28/0124230 ). Neither project _started_ with an inbred development community that presented a brick wall to new developers. Complexity grew because they didn't fight against it, and because they didn't have good rules by which they could say "no" to any. Environmental dependencies in your build environment are a cost, and as with all costs it's ok if you get enough in return for it. The Linux kernel has historically been extremely lean in this regard, and discarding that strength should at the very least come with commensurate concrete benefits. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 2:53 ` Rob Landley @ 2009-01-04 3:38 ` Markus Heidelberg 2009-01-04 4:57 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: Markus Heidelberg @ 2009-01-04 3:38 UTC (permalink / raw) To: Rob Landley Cc: Leon Woestenberg, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg Rob Landley, 04.01.2009: > On Saturday 03 January 2009 18:37:12 Leon Woestenberg wrote: > > My argument on thin dependencies indeed mostly holds for run-time > > dependencies (to reduce size) but also for build dependency (to reduce > > complexity)*. > > I usually just point to the gnucash 1.6 release as where this sort of thing > leads if you ignore it long enough: > http://lwn.net/2001/0614/ > > These days, a more modern example is the way that after even the gentoo folks > gave up on trying to build openoffice (and shipped prebuilt binaries of it in > their "build everything from source code" OS), Open Office's own developers > described that project "profoundly sick" and "stagnating" ( > http://developers.slashdot.org/article.pl?sid=08/12/28/0124230 ). Now that you mention this the second time, I have to ask where you have this information from. Since I use Gentoo, I was always able to compile OpenOffice (version 1, 2 and now 3) myself. At the same time, it was always possible to use prebuilt packages as an alternative - the same way as it is possible for a few other packages (Firefox, Thunderbird, Seamonkey, maybe more). But AFAIK compiling from source is still the preferred method. Markus ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 3:38 ` Markus Heidelberg @ 2009-01-04 4:57 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 4:57 UTC (permalink / raw) To: markus.heidelberg Cc: Leon Woestenberg, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Saturday 03 January 2009 21:38:13 Markus Heidelberg wrote: > Rob Landley, 04.01.2009: > Now that you mention this the second time, I have to ask where you have > this information from. Since I use Gentoo, I was always able to compile > OpenOffice (version 1, 2 and now 3) myself. The gentoo panel last OLS. (Either a BOF or a tutorial, I don't remember which.) It's still _possible_ to build it from source, but they created a separate "openoffice-bin" package for the sole purpose of _not_ compiling it from source, and it's what they recommend installing. > At the same time, it was always possible to use prebuilt packages as an > alternative - the same way as it is possible for a few other packages > (Firefox, Thunderbird, Seamonkey, maybe more). But AFAIK compiling from > source is still the preferred method. Apparently not for Open Office. First hit googling "gentoo openoffice install": http://grokdoc.net/index.php/Gentoo-OpenOffice.org The next two hits are bug tracker entries, and the one after that: http://www.linuxforums.org/forum/gentoo-linux-help/71086-installing- openoffice-question.html Contains this cut and paste from emerge output: These are the packages that would be merged, in order: Calculating dependencies !!! All ebuilds that could satisfy "openoffice" have been masked. !!! One of the following masked packages is required to complete your request: - app-office/openoffice-2.0.4_rc1-r1 (masked by: package.mask, package.mask, ~am d64 keyword) # 2005/10/24 Simon Stelling <blubb@gentoo.org> # Don't even try to compile openoffice-2.x, it won't work. But not to go too far down this rathole, I'm just using openoffice as an example here. If you want to talk about it more, take it off list please. > Markus Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-03 23:03 ` H. Peter Anvin 2009-01-04 0:37 ` Leon Woestenberg @ 2009-01-04 2:06 ` Rob Landley 2009-01-04 2:14 ` H. Peter Anvin ` (2 more replies) 1 sibling, 3 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 2:06 UTC (permalink / raw) To: H. Peter Anvin Cc: Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Saturday 03 January 2009 17:03:11 H. Peter Anvin wrote: > Leon Woestenberg wrote: > > I agree with Rob that the amount of required dependencies should be > > kept to a minimum. > > > > If we only use 0.5% of a certain language (or: dependent package), > > then rather implement that 0.5% in the existing language. > > > > Dependencies very quickly become dependency hell. If A requires B, > > then A also inherits all (future) requirements of B, etc. etc. > > > > In my daily software development work with Linux and GNU software in > > general, 10% of it is spent fighting/removing these extremely "thin" > > or false depencies, so that it is usuable in embedded devices. > > First of all, I largely consider this a joke. Yes, I've noticed this. The fact multiple other people do _not_ consider this a joke doesn't seem to have sunk in yet. > All real-life embedded > kernel builds take place on hosted platforms; anything else seems to be > done "just because it can be done", as a kind of show-off art project. > Cute, but hardly worth impeding the rest of the kernel community for. > > We're not talking about general platform dependencies here, but build > dependencies for the kernel. A platform that can build the kernel is > not a small platform. The kernel didn't need perl to build until 2.6.25. For 17 years, this dependency was not required. You added it, in a way that affected even "allnoconfig", for no obvious gain. > Second of all, these patches are not fullworthy replacements. The > original patch using bc had less dependencies, but bc failed on some > platforms, mysteriously. So rather than debugging it, you rewrote it in perl. Much less potential mysterious behavior there. > The new patches have *more* environmental > dependencies than that ever did. Could you please be a little more specific? > Third, if someone actually cares to do it right, I have a smallish > bignum library at http://git.zytor.com/?p=lib/pbn.git;a=summary that > might be a starting point. This doesn't _need_ bignum support. It maxes out around 72 bits and the _result_ can't use more than about $SHIFT bits because you're dividing by the amount you shifted, so just chop off the bottom 32 bits, do a normal 64 bit division on the top (it has to fit), and then do the same division on the appropriate shifted remainder, and combine the results. This is easy because when the shift _is_ 32 bits or more, the bottom 32 bits all have to be zeroes so you don't even have to mask and add, just shift the remainder left 32 bits so you can continue the divide. Pulling out perl isn't always a good alternative to thinking about the problem. > -hpa Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 2:06 ` Rob Landley @ 2009-01-04 2:14 ` H. Peter Anvin 2009-01-04 6:29 ` Rob Landley 2009-01-04 2:36 ` Jamie Lokier 2009-01-04 3:06 ` Paul Mundt 2 siblings, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-01-04 2:14 UTC (permalink / raw) To: Rob Landley Cc: Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg Rob Landley wrote: > >> The new patches have *more* environmental >> dependencies than that ever did. > > Could you please be a little more specific? > In this case, you're assuming that every version of every shell this is going to get involved with is going to do math correctly with the requisite precision, which is nowhere guaranteed, I'm pretty sure. >> Third, if someone actually cares to do it right, I have a smallish >> bignum library at http://git.zytor.com/?p=lib/pbn.git;a=summary that >> might be a starting point. > > This doesn't _need_ bignum support. It maxes out around 72 bits and the > _result_ can't use more than about $SHIFT bits because you're dividing by the > amount you shifted, so just chop off the bottom 32 bits, do a normal 64 bit > division on the top (it has to fit), and then do the same division on the > appropriate shifted remainder, and combine the results. This is easy because > when the shift _is_ 32 bits or more, the bottom 32 bits all have to be zeroes > so you don't even have to mask and add, just shift the remainder left 32 bits > so you can continue the divide. > > Pulling out perl isn't always a good alternative to thinking about the > problem. > Neither is open-coding a bignum operation instead of relying on an existing, validated implementation. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 2:14 ` H. Peter Anvin @ 2009-01-04 6:29 ` Rob Landley 2009-01-15 14:32 ` Pádraig Brady 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-04 6:29 UTC (permalink / raw) To: H. Peter Anvin Cc: Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Saturday 03 January 2009 20:14:44 H. Peter Anvin wrote: > Rob Landley wrote: > >> The new patches have *more* environmental > >> dependencies than that ever did. > > > > Could you please be a little more specific? > > In this case, you're assuming that every version of every shell this is > going to get involved with is going to do math correctly with the > requisite precision, which is nowhere guaranteed, I'm pretty sure. Well, SUSv3 requires that the shell support signed long arithmetic: http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04 And the LP64 standard says that on 64 bit systems, long must be 64 bit: http://www.unix.org/whitepapers/64bit.html Now the potential weakness there is that on 32 bit systems, shells might only support 32 bit math instead of 64 bit math. (You'll notice I have a test for that.) However, bash has supported 64 bit math on 32 bit systems since at least 2003. (I keep a Red Hat 9 test image around because it had 50% market share when it shipped, so the majority of "old" Linux systems still in use _are_ RH9 or similar. It also has the oldest gcc version the kernel still claims to build under.) Busybox ash can also support 64 bit math on 32 bit hosts, and I just confirmed that the dash in the 32 bit xubuntu 8.10 supports 64 bit math as well. (It would also be possible to do a similar overflow avoiding trick to do the lot entirely in 32 bit math, but given that the three main shells in use on Linux _do_ support 64 bit math on 32 bit hosts and are _required_ to support it on 64 bit hosts according to SUSv3, the extra complexity really doesn't seem worth it.) > >> Third, if someone actually cares to do it right, I have a smallish > >> bignum library at http://git.zytor.com/?p=lib/pbn.git;a=summary that > >> might be a starting point. > > > > This doesn't _need_ bignum support. It maxes out around 72 bits and the > > _result_ can't use more than about $SHIFT bits because you're dividing by > > the amount you shifted, so just chop off the bottom 32 bits, do a normal > > 64 bit division on the top (it has to fit), and then do the same division > > on the appropriate shifted remainder, and combine the results. This is > > easy because when the shift _is_ 32 bits or more, the bottom 32 bits all > > have to be zeroes so you don't even have to mask and add, just shift the > > remainder left 32 bits so you can continue the divide. > > > > Pulling out perl isn't always a good alternative to thinking about the > > problem. > > Neither is open-coding a bignum operation instead of relying on an > existing, validated implementation. Implementing something by hand isn't _always_ a good alternative, sure. That would be the "thinking about the problem" part. In this instance, avoiding overflow is trivial. (If 1<<-1 didn't wrap around, it wouldn't even need the if statement.) I'm curious, would the "existing, validated implementation" in this instance be the perl implementation, or the library you wrote and pointed me to above as a potential replacement for it? I suppose there's a certain amount of style in accusing me of reinventing the wheel while pointing me at your reinvention of the wheel. (Are you aiming to replace Gnu's gmplib.org, the perhaps the BSD licensed one in openssh? Dropbear uses Libtommath. A quick google for C open source big number libraries also found Libimath, MPI, NTL, BigDigits, decNumber, and MPI. The last time I personally wrote my own arbitrary precision math package from scratch was in 1998, and that was in Java, so I'm a little rusty...) But I don't personally consider avoiding the need for an arbitrary precision math library to be the same as reimplementing one. > -hpa Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 6:29 ` Rob Landley @ 2009-01-15 14:32 ` Pádraig Brady 0 siblings, 0 replies; 135+ messages in thread From: Pádraig Brady @ 2009-01-15 14:32 UTC (permalink / raw) To: Rob Landley Cc: H. Peter Anvin, Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Sam Ravnborg, austin-group-l Rob Landley wrote: > Implementing something by hand isn't _always_ a good alternative, sure. That > would be the "thinking about the problem" part. In this instance, avoiding > overflow is trivial. (If 1<<-1 didn't wrap around, it wouldn't even need the > if statement.) I don't think this affects your script but it's worth noting that both bash and ksh use arithmetic rather than logical shift for the >> operator. Now arithmetic shift is not useful on 2's compliment machines, and moreover it's compiler dependent as to whether arithmetic or logical shift is done for >>. Therefore to increase usefulness and decrease ambiguity, shells really should only shift unsigned variables internally. I know the opengroup spec says to use signed ints, but I think that is intended to disambiguate input and output, rather than defining internal operations. This is correct I think since the POSIX spec says you can even use floating point internally if you like. I asked the bash maintainer who said he would need clarification from the austin group (CC'd) before changing anything. cheers, Pádraig. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 2:06 ` Rob Landley 2009-01-04 2:14 ` H. Peter Anvin @ 2009-01-04 2:36 ` Jamie Lokier 2009-01-04 2:39 ` H. Peter Anvin 2009-01-04 3:06 ` Paul Mundt 2 siblings, 1 reply; 135+ messages in thread From: Jamie Lokier @ 2009-01-04 2:36 UTC (permalink / raw) To: Rob Landley Cc: H. Peter Anvin, Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg Rob Landley wrote: > This doesn't _need_ bignum support. It maxes out around 72 bits and > the _result_ can't use more than about $SHIFT bits because you're > dividing by the amount you shifted, so just chop off the bottom 32 > bits, do a normal 64 bit division on the top (it has to fit), and > then do the same division on the appropriate shifted remainder, and > combine the results. This is easy because when the shift _is_ 32 > bits or more, the bottom 32 bits all have to be zeroes so you don't > even have to mask and add, just shift the remainder left 32 bits so > you can continue the divide. > > Pulling out perl isn't always a good alternative to thinking about > the problem. Related query: Does the Perl script being replaced use 64-bit arithmetic? Because many Perl installations only do 32-bit arithmetic. If the Perl version works in 32-bit arithmetic, why does the shell version not do the same thing? -- Jamie ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 2:36 ` Jamie Lokier @ 2009-01-04 2:39 ` H. Peter Anvin 2009-01-04 2:43 ` H. Peter Anvin 0 siblings, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-01-04 2:39 UTC (permalink / raw) To: Jamie Lokier Cc: Rob Landley, Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg Jamie Lokier wrote: > > Related query: > > Does the Perl script being replaced use 64-bit arithmetic? Because > many Perl installations only do 32-bit arithmetic. > > If the Perl version works in 32-bit arithmetic, why does the shell > version not do the same thing? > The Perl version uses Math::BigInt, a Perl standard module (with a canned-values fallback for ancient or minimal Perl installations) to do arbitrary precision arithmetic. The original version also produced constants that could be used with 64-bit values, but since gcc doesn't support 128-bit arithmetic on 32-bit platforms (gcc *does* support 128-bit arithmetic on 64-bit platforms) we didn't end up using it and removed them, although the code to generate them can still be activated. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 2:39 ` H. Peter Anvin @ 2009-01-04 2:43 ` H. Peter Anvin 0 siblings, 0 replies; 135+ messages in thread From: H. Peter Anvin @ 2009-01-04 2:43 UTC (permalink / raw) To: Jamie Lokier Cc: Rob Landley, Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg H. Peter Anvin wrote: > Jamie Lokier wrote: >> Related query: >> >> Does the Perl script being replaced use 64-bit arithmetic? Because >> many Perl installations only do 32-bit arithmetic. >> >> If the Perl version works in 32-bit arithmetic, why does the shell >> version not do the same thing? >> > > The Perl version uses Math::BigInt, a Perl standard module (with a > canned-values fallback for ancient or minimal Perl installations) to do > arbitrary precision arithmetic. > > The original version also produced constants that could be used with > 64-bit values, but since gcc doesn't support 128-bit arithmetic on > 32-bit platforms (gcc *does* support 128-bit arithmetic on 64-bit > platforms) we didn't end up using it and removed them, although the code > to generate them can still be activated. > I should point out that we really *should* use same kind of techniques on 64 bits as well. Even though the likelihood of overflow is much less there (and the use of the LCD reduces it further) it is nonzero. However, the places that were most seriously affected were all operating on 32-bit input (int), and therefore the overflow-free 64-bit code never got written. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 2:06 ` Rob Landley 2009-01-04 2:14 ` H. Peter Anvin 2009-01-04 2:36 ` Jamie Lokier @ 2009-01-04 3:06 ` Paul Mundt 2009-01-04 10:23 ` Leon Woestenberg 2009-01-04 16:22 ` Vladimir Dronnikov 2 siblings, 2 replies; 135+ messages in thread From: Paul Mundt @ 2009-01-04 3:06 UTC (permalink / raw) To: Rob Landley Cc: H. Peter Anvin, Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Sat, Jan 03, 2009 at 08:06:47PM -0600, Rob Landley wrote: > On Saturday 03 January 2009 17:03:11 H. Peter Anvin wrote: > > Leon Woestenberg wrote: > > > I agree with Rob that the amount of required dependencies should be > > > kept to a minimum. > > > > > > If we only use 0.5% of a certain language (or: dependent package), > > > then rather implement that 0.5% in the existing language. > > > > > > Dependencies very quickly become dependency hell. If A requires B, > > > then A also inherits all (future) requirements of B, etc. etc. > > > > > > In my daily software development work with Linux and GNU software in > > > general, 10% of it is spent fighting/removing these extremely "thin" > > > or false depencies, so that it is usuable in embedded devices. > > > > First of all, I largely consider this a joke. > > Yes, I've noticed this. The fact multiple other people do _not_ consider this > a joke doesn't seem to have sunk in yet. > Let's look at the rationale presented so far in this thread: 1 - Being able to build the kernel natively on a constrained target is useful, regardless of whether it is being used for regression/stress testing or for headers installation or whatever else. 2 - Cross-compiling perl is hard. 3 - Some oddly constrained target distributions manage to ship with an entire toolchain yet fail to provide any implementation of perl. 4 - It wasn't required before. If there is anything I missed, feel free to add it to the list. It was difficult to extract even those 4 from the ranting. #1 is a logical fallacy. If you have enough space and CPU power and complete build environment to crunch away at the kernel for stress testing natively, you can do the same with building perl and negating point #2. This is especially true for NFS root filesystems where one is not space constrained during the development phase. #2 is another byproduct of your environment and generally a non-issue. There are plenty of options around having to cross-compile perl, and for those that still insist on doing so, people have been doing it long enough to be aware of the pitfalls involved. It is not a pleasant experience, but that is again entire your problem and entirely constrained to your environment. #3 seems to have come up a surprising number of times, and again seems to originate from the fact that no one wants to be bothered with #2 whilst putting together their oddly constrained rootfs. So far no one has actually posted any coherent rationale as to why these distributions are shipping with a full gcc/binutils/etc. environment yet are unable to supply perl. Obviously size is not a factor if it ships with a full build environment otherwise, so this suggests that the only logical objection to fixing up the distributions stems from #4. As far as #4 goes, I have a hard time seeing why this should be anyone's problem. Progress is not made by restricting people to the way things were, progress is made by adapting to new things as they come along. In the case of the perl scripts provided, perl was picked by the developer in question as the right tool for the job, and things generally went along pretty smoothly. Given that one has a reasonable expectation of perl being available on the vast majority of systems today, this is hardly an unrealistic tool to leverage for use within the kernel scripts. The perl dependency has never been an issue for me on any of the platforms I routinely work on, ranging from tiny microcontrollers to multi-node NUMA and SMP configurations and everything in between. In places where the target is capable of building natively, I have no qualms with building a reasonable development environment. And in places where builds are unrealistic, I will do them on the host instead. This has always been the way things were, and I find the implication that the majority of the embedded development community sits fixedly on #3 to be completely ridiculous. I will repeat again that no one has provided a _single_ reason for why they are unable to provide perl within their constrained environment. Until that happens, this entire thread is a joke. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 3:06 ` Paul Mundt @ 2009-01-04 10:23 ` Leon Woestenberg 2009-01-08 13:29 ` Mike Frysinger 2009-01-11 12:45 ` Bernd Petrovitsch 2009-01-04 16:22 ` Vladimir Dronnikov 1 sibling, 2 replies; 135+ messages in thread From: Leon Woestenberg @ 2009-01-04 10:23 UTC (permalink / raw) To: Paul Mundt, Rob Landley, H. Peter Anvin, Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg Hello, On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal@linux-sh.org> wrote: > Let's look at the rationale presented so far in this thread: > > 1 - Being able to build the kernel natively on a constrained > target is useful, regardless of whether it is being used for > regression/stress testing or for headers installation or whatever > else. > > 2 - Cross-compiling perl is hard. > > 3 - Some oddly constrained target distributions manage to ship > with an entire toolchain yet fail to provide any implementation > of perl. > > 4 - It wasn't required before. > > If there is anything I missed, feel free to add it to the list. It was > difficult to extract even those 4 from the ranting. > 2 is not hard. 5. Tool *version* dependency is hard to get right. When cross-building 30 software packages all requiring native perl, we probably need to build a few versions of perl (native), one for each set of packages. Regards, -- Leon ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 10:23 ` Leon Woestenberg @ 2009-01-08 13:29 ` Mike Frysinger 2009-01-11 12:45 ` Bernd Petrovitsch 1 sibling, 0 replies; 135+ messages in thread From: Mike Frysinger @ 2009-01-08 13:29 UTC (permalink / raw) To: Leon Woestenberg Cc: Paul Mundt, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Sun, Jan 4, 2009 at 05:23, Leon Woestenberg wrote: > On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal@linux-sh.org> wrote: >> Let's look at the rationale presented so far in this thread: >> >> 2 - Cross-compiling perl is hard. > > 2 is not hard. i dont know where you're getting this mythical version of perl, but anything beyond micro-perl is a pita. and micro-perl is practically worthless. maybe for kernel/ -mike ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 10:23 ` Leon Woestenberg 2009-01-08 13:29 ` Mike Frysinger @ 2009-01-11 12:45 ` Bernd Petrovitsch 2009-01-12 3:36 ` Mark A. Miller [not found] ` <31014a580901111928u586e2246uccf370ff941c8a01@mail.gmail.com> 1 sibling, 2 replies; 135+ messages in thread From: Bernd Petrovitsch @ 2009-01-11 12:45 UTC (permalink / raw) To: Leon Woestenberg Cc: Paul Mundt, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Son, 2009-01-04 at 11:23 +0100, Leon Woestenberg wrote: [...] > On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal@linux-sh.org> wrote: [...] I'm ignoring the "cross-compile perl" issue - haven't tried it for years. > 5. Tool *version* dependency is hard to get right. When cross-building > 30 software packages all requiring native perl, we probably need to > build a few versions of perl (native), one for each set of packages. perl is IMHO special (and quite different to others - including especially autotools): perl5 is used widely enough so that "one somewhat recent version" should cover all of 30 software packages. The hard part are the CPAN modules and their versions which are really a PITA. As long as you don't use modules from CPAN, "perl5" should be specific enough. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-11 12:45 ` Bernd Petrovitsch @ 2009-01-12 3:36 ` Mark A. Miller 2009-01-12 5:11 ` H. Peter Anvin ` (2 more replies) [not found] ` <31014a580901111928u586e2246uccf370ff941c8a01@mail.gmail.com> 1 sibling, 3 replies; 135+ messages in thread From: Mark A. Miller @ 2009-01-12 3:36 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Leon Woestenberg, Paul Mundt, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Sun, Jan 11, 2009 at 6:45 AM, Bernd Petrovitsch <bernd@firmix.at> wrote: > > On Son, 2009-01-04 at 11:23 +0100, Leon Woestenberg wrote: > [...] > > On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal@linux-sh.org> wrote: > [...] > > I'm ignoring the "cross-compile perl" issue - haven't tried it for > years. > > > 5. Tool *version* dependency is hard to get right. When cross-building > > 30 software packages all requiring native perl, we probably need to > > build a few versions of perl (native), one for each set of packages. > > perl is IMHO special (and quite different to others - including > especially autotools): perl5 is used widely enough so that "one somewhat > recent version" should cover all of 30 software packages. > The hard part are the CPAN modules and their versions which are really a > PITA. > As long as you don't use modules from CPAN, "perl5" should be specific > enough. > > Bernd > -- > Firmix Software GmbH http://www.firmix.at/ > mobil: +43 664 4416156 fax: +43 1 7890849-55 > Embedded Linux Development and Services Actually, something that has amused me during this discussion, is that right now, the latest stable Perl (5.8.8) does not compile correctly on a uclibc host, which is typically what you want for embedded systems, which is why you'd bother to cross compile. (Albeit I was doing this natively under QEMU with a gcc/uclibc toolchain). I'll have a patch submitted upstream shortly, but basically the hints/linux.sh assumes *obviously* you're linking to glibc. I've made that less libc dependent, looking for either glibc or uclibc. So without patching Perl, by adding it to the kernel configuration, it's broken being able to compile the kernel on most embedded architectures. And for H. Peter Anvin, before you refer to such uses as compiling the kernel under a native environment as a "piece of art", please be aware that the mainstream embedded development environment, buildroot, is also attempting to utilize QEMU for a "sanity check" on the environment. There are several other packages which are broken for embedded architectures, which I will hopefully attempt to fix by submitting patches upstream. But this is why we should be cautious about including new tools for compiling the kernel. Sam Ravnborg was correct in that a C program to do the work would be the proper way. But by not addressing a currently existing problem with an adequate replacement with something that does not exist currently, seems faulty. -- Mark A. Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 3:36 ` Mark A. Miller @ 2009-01-12 5:11 ` H. Peter Anvin 2009-01-12 5:23 ` Mark A. Miller 2009-01-12 8:20 ` Paul Mundt 2009-01-12 8:27 ` Peter Korsgaard 2 siblings, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-01-12 5:11 UTC (permalink / raw) To: Mark A. Miller Cc: Bernd Petrovitsch, Leon Woestenberg, Paul Mundt, Rob Landley, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg Mark A. Miller wrote: > > Actually, something that has amused me during this discussion, is that > right now, the latest stable Perl (5.8.8) does not compile correctly > on a uclibc host... > The latest stable Perl is 5.10.0, and the latest of the 5.8 series is 5.8.9. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 5:11 ` H. Peter Anvin @ 2009-01-12 5:23 ` Mark A. Miller 0 siblings, 0 replies; 135+ messages in thread From: Mark A. Miller @ 2009-01-12 5:23 UTC (permalink / raw) To: H. Peter Anvin Cc: Bernd Petrovitsch, Leon Woestenberg, Paul Mundt, Rob Landley, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Sun, Jan 11, 2009 at 11:11 PM, H. Peter Anvin <hpa@zytor.com> wrote: > Mark A. Miller wrote: >> >> Actually, something that has amused me during this discussion, is that >> right now, the latest stable Perl (5.8.8) does not compile correctly >> on a uclibc host... >> > > The latest stable Perl is 5.10.0, and the latest of the 5.8 series is 5.8.9. > > -hpa > > -- > H. Peter Anvin, Intel Open Source Technology Center > I work for Intel. I don't speak on their behalf. My mistake. However, http://perl5.git.perl.org/perl.git/blob_plain/HEAD:/hints/linux.sh still has the issue, specifically: if test -L /lib/libc.so.6; then libc=`ls -l /lib/libc.so.6 | awk '{print $NF}'` libc=/lib/$libc fi So, my version was incorrect, yet the problem still exists. I've got a patch, need to submit it, yet Perl *does not compile* on a uclibc target *as is*. And this is why we should avoid adding new tools to build the kernel, because they introduce yet more break points, as such. Thanks. -- Mark A. Miller mark@mirell.org "My greatest strength, I guess it would be my humility. My greatest weakness, it's possible that I'm a little too awesome" - Barack Obama ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 3:36 ` Mark A. Miller 2009-01-12 5:11 ` H. Peter Anvin @ 2009-01-12 8:20 ` Paul Mundt 2009-01-12 9:18 ` Mark A. Miller 2009-01-12 8:27 ` Peter Korsgaard 2 siblings, 1 reply; 135+ messages in thread From: Paul Mundt @ 2009-01-12 8:20 UTC (permalink / raw) To: Mark A. Miller Cc: Bernd Petrovitsch, Leon Woestenberg, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Sun, Jan 11, 2009 at 09:36:58PM -0600, Mark A. Miller wrote: > Actually, something that has amused me during this discussion, is that > right now, the latest stable Perl (5.8.8) does not compile correctly > on a uclibc host, which is typically what you want for embedded > systems, which is why you'd bother to cross compile. (Albeit I was > doing this natively under QEMU with a gcc/uclibc toolchain). > > I'll have a patch submitted upstream shortly, but basically the > hints/linux.sh assumes *obviously* you're linking to glibc. I've made > that less libc dependent, looking for either glibc or uclibc. > There are plenty that ship with glibc, too. What you "want" for embedded systems depends entirely on the application for the device, not general hand-wavy assertions. We (Renesas) ship BSPs on both precisely because of this reason, and eglibc will probably factor in at some point later on too. > So without patching Perl, by adding it to the kernel configuration, > it's broken being able to compile the kernel on most embedded > architectures. > This again has nothing to do with the kernel and everything to do with your distribution. I use perl on uclibc natively just fine, it is possible there are patches that have not been merged upstream, but this is again an entirely separate issue. You seem to be confusing the fact that people who build distributions and people who use them are one and the same, whereas "most" embedded developers are going to be using pre-built distributions provided with their reference hardware, and locking it down during productization. The fact you are doing a distribution aimed at embedded devices is nice, but do not try to push off problems you run in to that have a reasonable expectation of working everywhere else on to the kernel community as something we ought to care about. If you need to use a different libc on your platform, yes, you will have to update packages for this. This used to be true for gcc and other packages as well, but those were all fixed over time. The fact perl still stands out despite there being patches available is simply an indicator that folks working in that area haven't been very proactive in getting their changes merged upstream. Tough. This is now entirely off-topic and has nothing to do with the kernel any more. Please take this to the uclibc or perl lists instead. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 8:20 ` Paul Mundt @ 2009-01-12 9:18 ` Mark A. Miller 2009-01-12 9:41 ` Paul Mundt 0 siblings, 1 reply; 135+ messages in thread From: Mark A. Miller @ 2009-01-12 9:18 UTC (permalink / raw) To: Paul Mundt, Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Mon, Jan 12, 2009 at 2:20 AM, Paul Mundt <lethal@linux-sh.org> wrote: > On Sun, Jan 11, 2009 at 09:36:58PM -0600, Mark A. Miller wrote: >> Actually, something that has amused me during this discussion, is that >> right now, the latest stable Perl (5.8.8) does not compile correctly >> on a uclibc host, which is typically what you want for embedded >> systems, which is why you'd bother to cross compile. (Albeit I was >> doing this natively under QEMU with a gcc/uclibc toolchain). >> >> I'll have a patch submitted upstream shortly, but basically the >> hints/linux.sh assumes *obviously* you're linking to glibc. I've made >> that less libc dependent, looking for either glibc or uclibc. >> > There are plenty that ship with glibc, too. What you "want" for embedded > systems depends entirely on the application for the device, not general > hand-wavy assertions. We (Renesas) ship BSPs on both precisely because of > this reason, and eglibc will probably factor in at some point later on > too. > >> So without patching Perl, by adding it to the kernel configuration, >> it's broken being able to compile the kernel on most embedded >> architectures. >> > This again has nothing to do with the kernel and everything to do with > your distribution. I use perl on uclibc natively just fine, it is > possible there are patches that have not been merged upstream, but this > is again an entirely separate issue. > > You seem to be confusing the fact that people who build distributions and > people who use them are one and the same, whereas "most" embedded > developers are going to be using pre-built distributions provided with > their reference hardware, and locking it down during productization. The > fact you are doing a distribution aimed at embedded devices is nice, but > do not try to push off problems you run in to that have a reasonable > expectation of working everywhere else on to the kernel community as > something we ought to care about. > > If you need to use a different libc on your platform, yes, you will have > to update packages for this. This used to be true for gcc and other > packages as well, but those were all fixed over time. The fact perl still > stands out despite there being patches available is simply an indicator > that folks working in that area haven't been very proactive in getting > their changes merged upstream. Tough. > > This is now entirely off-topic and has nothing to do with the kernel any > more. Please take this to the uclibc or perl lists instead. > Paul: I initially wrote a rather details response to your e-mail. But instead, I shall quote a previous e-mail of yours: > I will repeat again that no one has provided a > _single_ reason for why they are unable to provide perl within their > constrained environment. Until that happens, this entire thread is a > joke. And I did so. And you have disregarded it. That makes me question the logic of your fervent vehemence against such "Perl is perhaps not a good idea in the kernel build infrastructure" people like myself. Thanks. -- Mark A. Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 9:18 ` Mark A. Miller @ 2009-01-12 9:41 ` Paul Mundt 2009-01-12 10:03 ` Mark A. Miller 2009-01-12 17:56 ` Rob Landley 0 siblings, 2 replies; 135+ messages in thread From: Paul Mundt @ 2009-01-12 9:41 UTC (permalink / raw) To: Mark A. Miller Cc: Bernd Petrovitsch, Leon Woestenberg, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Mon, Jan 12, 2009 at 03:18:53AM -0600, Mark A. Miller wrote: > On Mon, Jan 12, 2009 at 2:20 AM, Paul Mundt <lethal@linux-sh.org> wrote: > > Paul: > I initially wrote a rather details response to your e-mail. But > instead, I shall quote a previous e-mail of yours: > > > I will repeat again that no one has provided a > > _single_ reason for why they are unable to provide perl within their > > constrained environment. Until that happens, this entire thread is a > > joke. > > And I did so. And you have disregarded it. That makes me question the > logic of your fervent vehemence against such "Perl is perhaps not a > good idea in the kernel build infrastructure" people like myself. > You have done no such thing. You have provided an example as to why you personally find perl objectionable, and in your previous mail you even noted that you have patches for perl to fix the build issues, so there is no fundamental reason why you are _unable_ to provide perl in your environment. It all comes down to the fact you don't want to be bothered to put the effort in to getting perl setup in your environment, which quite frankly is no one's problem but your own. Personally I think perl (and python for that matter) is a terrible language and I wouldn't use it for anything of consequence, but again, that is my personal opinion and has nothing to do with regards to the build system and whether it was the better tool for the job as perceived by the people that elected to implemented infrastructure using it. I choose not to use it for my own projects, but I have no qualms with those that do. The kernel does not need to provide justification for every change that goes on as long as there is a reasonable attempt not to break things for other people. The onus is (and always has been) on you to demonstrate why perl is an unreasonable dependency to push on embedded developers, and you have failed utterly at demonstrating this in any sort of coherent fashion. I will repeat, there has not been a single coherent argument against what makes perl inherently incapable of being supported. Every single thing you have presented as a rebuttal has been your personal preference, which in this case is simply irrelevant. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 9:41 ` Paul Mundt @ 2009-01-12 10:03 ` Mark A. Miller 2009-01-12 10:34 ` Paul Mundt 2009-01-12 17:56 ` Rob Landley 1 sibling, 1 reply; 135+ messages in thread From: Mark A. Miller @ 2009-01-12 10:03 UTC (permalink / raw) To: Paul Mundt, Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Mon, Jan 12, 2009 at 3:41 AM, Paul Mundt <lethal@linux-sh.org> wrote: > I will repeat, there has not been a single coherent argument against what > makes perl inherently incapable of being supported. You're right, this thread is worthless with that particular mindset, Paul. Not, perhaps that the tool in question is brittle, and prone to potentially break on more architectures than the current make and C code infrastructure, no, your stance is, unless Perl *cannot* run on that particular architecture and environment, it has a valid place in the kernel because it was chosen by certain developers. And you're right, I did patch around Perl in order to get it to build under a MIPSEL uclibc environment. But yes, this particular thread with you *is* worthless, because it's an argument who's stance is not worth fighting against because of it's flawed premise. -- Mark A. Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 10:03 ` Mark A. Miller @ 2009-01-12 10:34 ` Paul Mundt 0 siblings, 0 replies; 135+ messages in thread From: Paul Mundt @ 2009-01-12 10:34 UTC (permalink / raw) To: Mark A. Miller Cc: Bernd Petrovitsch, Leon Woestenberg, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Mon, Jan 12, 2009 at 04:03:32AM -0600, Mark A. Miller wrote: > On Mon, Jan 12, 2009 at 3:41 AM, Paul Mundt <lethal@linux-sh.org> wrote: > > I will repeat, there has not been a single coherent argument against what > > makes perl inherently incapable of being supported. > > You're right, this thread is worthless with that particular mindset, > Paul. Not, perhaps that the tool in question is brittle, and prone to > potentially break on more architectures than the current make and C > code infrastructure, no, your stance is, unless Perl *cannot* run on > that particular architecture and environment, it has a valid place in > the kernel because it was chosen by certain developers. > Nonsense. I singled out that point because that was the one you were replying to in the first place. I itemized the objections in this thread earlier on and attempted to indicate why they were not applicable in this context, and asked people to add to it if anything had been overlooked. If you want to play semantics, do it somewhere else. If the tool is brittle and constantly breaking, we will see bug reports, and re-evaluate the support position. This hasn't happened to date in the context of the kernel build system, so there is no point in even bringing this up. Anyways, given that you haven't contributed anything to the kernel and are therefore perhaps unfamiliar with how things work, I attempted to show you why the kernel made the decision it did and what it would take to change that. You have from the beginning only wanted to focus on idle semantics and refused to re-evaluate your own position on what precisely it is you find to be problematic in the first place. So, with that, I am done with this thread, and it seems the key takeaways from this entire thing has only been a few new lines in my killfile. It's regrettable you didn't get anything else out of this thread, though I think both the kernel and embedded linux will survive. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 9:41 ` Paul Mundt 2009-01-12 10:03 ` Mark A. Miller @ 2009-01-12 17:56 ` Rob Landley 2009-01-12 18:04 ` Alan Cox 1 sibling, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-12 17:56 UTC (permalink / raw) To: Paul Mundt Cc: Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Monday 12 January 2009 03:41:22 Paul Mundt wrote: > Personally I think perl (and python for that matter) is a terrible > language and I wouldn't use it for anything of consequence, but again, > that is my personal opinion and has nothing to do with regards to the > build system and whether it was the better tool for the job as perceived > by the people that elected to implemented infrastructure using it. I > choose not to use it for my own projects, but I have no qualms with those > that do. Apparently you have qualms with people who chose to reimplement the perl bits in one of the languages kernel developers already needed to know this time last year (shell, C, make). > The kernel does not need to provide justification for every change that > goes on as long as there is a reasonable attempt not to break things for > other people. I submitted a change, you insisted that I justify it to your satisfaction. That pretty much summarizes your participation in this thread. > The onus is (and always has been) on you to demonstrate why > perl is an unreasonable dependency to push on embedded developers, and > you have failed utterly at demonstrating this in any sort of coherent > fashion. Large additional dependencies without benefit are unreasonable. My primary objection to perl is that it happens to be an additional large dependency without a correspondingly large benefit. Your position is not internally consistent. There was no need to scrutinize it when it went in, but there is a need to scrutinize patches reimplementing those bits without it. You don't need the word "perl" in that sentence for your position to be a touch unbalanced. > I will repeat, there has not been a single coherent argument against what > makes perl inherently incapable of being supported. I didn't say it was incapable of being supported. We're _capable_ of reimplementing the entire kernel in perl except for a microkernel interpreter running on the bare metal. Or cobol. Sun spent some time trying to do one in Java a few years back. "It can be done" and "It's a good idea" are two completely different criteria. > Every single thing > you have presented as a rebuttal has been your personal preference, which > in this case is simply irrelevant. Stop getting so hung up on the word "perl". Did you ever notice the _shipped files in the kernel so you don't have to have lex or yacc installed? That's been kernel policy for how many years now? The arguments about "dash vs bash" when reviewing the shell versions of these scripts are a similar impulse: trying to reduce unnecessary dependencies. My first version of the timeconst patch didn't remove the perl script that generated the file, it simply shipped the pregenerated .h file so it was possible to _build_ without perl. That was not sufficient for technical reasons (due to the two architectures that allow you to enter arbitrary values), so I redid the patch. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 17:56 ` Rob Landley @ 2009-01-12 18:04 ` Alan Cox 0 siblings, 0 replies; 135+ messages in thread From: Alan Cox @ 2009-01-12 18:04 UTC (permalink / raw) To: Rob Landley Cc: Paul Mundt, Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg > I didn't say it was incapable of being supported. We're _capable_ of > reimplementing the entire kernel in perl Which perl. What minor release, what day of the week syntax. Ask anyone in the distribution business about the joy of perl and you can listen to the screams for hours. Perl5 has no formal grammar and you cannot tell what perl of the week does and perl of last week doesn't do. That makes it a bad candidate for our toolchain dependencies. Alan -- "I don't want world domination if it means I have to deal with more people like this" - Mike Wangsmo ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 3:36 ` Mark A. Miller 2009-01-12 5:11 ` H. Peter Anvin 2009-01-12 8:20 ` Paul Mundt @ 2009-01-12 8:27 ` Peter Korsgaard 2009-01-12 17:45 ` Rob Landley 2 siblings, 1 reply; 135+ messages in thread From: Peter Korsgaard @ 2009-01-12 8:27 UTC (permalink / raw) To: Mark A. Miller Cc: Bernd Petrovitsch, Leon Woestenberg, Paul Mundt, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg >>>>> "Mark" == Mark A Miller <mark@mirell.org> writes: Mark> And for H. Peter Anvin, before you refer to such uses as compiling the Mark> kernel under a native environment as a "piece of art", please be aware Mark> that the mainstream embedded development environment, buildroot, is Mark> also attempting to utilize QEMU for a "sanity check" on the Mark> environment. That's for verifying that the rootfs'es actually work, not for building stuff. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 8:27 ` Peter Korsgaard @ 2009-01-12 17:45 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-12 17:45 UTC (permalink / raw) To: Peter Korsgaard Cc: Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, Paul Mundt, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg On Monday 12 January 2009 02:27:32 Peter Korsgaard wrote: > >>>>> "Mark" == Mark A Miller <mark@mirell.org> writes: > > Mark> And for H. Peter Anvin, before you refer to such uses as compiling > the Mark> kernel under a native environment as a "piece of art", please be > aware Mark> that the mainstream embedded development environment, > buildroot, is Mark> also attempting to utilize QEMU for a "sanity check" on > the Mark> environment. > > That's for verifying that the rootfs'es actually work, not for > building stuff. Not in my case. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
[parent not found: <31014a580901111928u586e2246uccf370ff941c8a01@mail.gmail.com>]
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. [not found] ` <31014a580901111928u586e2246uccf370ff941c8a01@mail.gmail.com> @ 2009-01-12 5:35 ` Sam Ravnborg 2009-01-12 5:50 ` Mark A. Miller 0 siblings, 1 reply; 135+ messages in thread From: Sam Ravnborg @ 2009-01-12 5:35 UTC (permalink / raw) To: Mark A. Miller Cc: Bernd Petrovitsch, Leon Woestenberg, Paul Mundt, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton > There are several other packages which are broken for embedded > architectures, which I will hopefully attempt to fix by submitting patches > upstream. But this is why we should be cautious about including new tools > for compiling the kernel. Sam Ravnborg was correct in that a C program to do > the work would be the proper way. But by not addressing a currently existing > problem with an adequate replacement with something that does not exist > currently, seems faulty. Why are "make headers_install" such a crucial thing for your embedded environmnet? I would assume that if this of such improtance then there is also someone to step up and contribute a C version of it. Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 5:35 ` Sam Ravnborg @ 2009-01-12 5:50 ` Mark A. Miller 2009-01-12 10:18 ` Sam Ravnborg 0 siblings, 1 reply; 135+ messages in thread From: Mark A. Miller @ 2009-01-12 5:50 UTC (permalink / raw) To: Sam Ravnborg Cc: Bernd Petrovitsch, Leon Woestenberg, Paul Mundt, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton On Sun, Jan 11, 2009 at 11:35 PM, Sam Ravnborg <sam@ravnborg.org> wrote: >> There are several other packages which are broken for embedded >> architectures, which I will hopefully attempt to fix by submitting patches >> upstream. But this is why we should be cautious about including new tools >> for compiling the kernel. Sam Ravnborg was correct in that a C program to do >> the work would be the proper way. But by not addressing a currently existing >> problem with an adequate replacement with something that does not exist >> currently, seems faulty. > > Why are "make headers_install" such a crucial thing for your > embedded environmnet? Sanity check. If the environment cannot replicate itself, then something has been faulty in the cross-compiling stage, that was used to propagate a native environment for the target architecture. > I would assume that if this of such improtance then there is also > someone to step up and contribute a C version of it. You've already dismissed a shell version to correct the issue, in hopes of a "possible" C version. It would be nice, I'm not capable of doing it personally, but a solution already exists to "unbreak" the kernel build. If you're unwilling to merge the current patches, then feel free to claim that this doesn't break anything on current architectures, but it's incorrect, due to Perl not even compiling as is currently on a native uclibc environment. I look forward to what other tools will be introduced to break yet more architectures until the kernel cannot be compiled unless on an i686+glibc architecture. > Sam -- Mark A. Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 5:50 ` Mark A. Miller @ 2009-01-12 10:18 ` Sam Ravnborg 2009-01-12 10:22 ` Mark A. Miller 2009-01-12 10:38 ` Paul Mundt 0 siblings, 2 replies; 135+ messages in thread From: Sam Ravnborg @ 2009-01-12 10:18 UTC (permalink / raw) To: Mark A. Miller Cc: Bernd Petrovitsch, Leon Woestenberg, Paul Mundt, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton On Sun, Jan 11, 2009 at 11:50:31PM -0600, Mark A. Miller wrote: > On Sun, Jan 11, 2009 at 11:35 PM, Sam Ravnborg <sam@ravnborg.org> wrote: > >> There are several other packages which are broken for embedded > >> architectures, which I will hopefully attempt to fix by submitting patches > >> upstream. But this is why we should be cautious about including new tools > >> for compiling the kernel. Sam Ravnborg was correct in that a C program to do > >> the work would be the proper way. But by not addressing a currently existing > >> problem with an adequate replacement with something that does not exist > >> currently, seems faulty. > > > > Why are "make headers_install" such a crucial thing for your > > embedded environmnet? > > Sanity check. If the environment cannot replicate itself, then > something has been faulty in the cross-compiling stage, that was used > to propagate a native environment for the target architecture. So you actually build your target toolchain on your target? Sam ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 10:18 ` Sam Ravnborg @ 2009-01-12 10:22 ` Mark A. Miller 2009-01-12 10:38 ` Paul Mundt 1 sibling, 0 replies; 135+ messages in thread From: Mark A. Miller @ 2009-01-12 10:22 UTC (permalink / raw) To: Sam Ravnborg Cc: Bernd Petrovitsch, Leon Woestenberg, Paul Mundt, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton On Mon, Jan 12, 2009 at 4:18 AM, Sam Ravnborg <sam@ravnborg.org> wrote: > On Sun, Jan 11, 2009 at 11:50:31PM -0600, Mark A. Miller wrote: >> On Sun, Jan 11, 2009 at 11:35 PM, Sam Ravnborg <sam@ravnborg.org> wrote: >> >> There are several other packages which are broken for embedded >> >> architectures, which I will hopefully attempt to fix by submitting patches >> >> upstream. But this is why we should be cautious about including new tools >> >> for compiling the kernel. Sam Ravnborg was correct in that a C program to do >> >> the work would be the proper way. But by not addressing a currently existing >> >> problem with an adequate replacement with something that does not exist >> >> currently, seems faulty. >> > >> > Why are "make headers_install" such a crucial thing for your >> > embedded environmnet? >> >> Sanity check. If the environment cannot replicate itself, then >> something has been faulty in the cross-compiling stage, that was used >> to propagate a native environment for the target architecture. > > So you actually build your target toolchain on your target? > > Sam Correct, albeit under emulation, such as QEMU. Obviously the target architecture, such as an embedded MIPSEL device with only 8MB of flash and 64MB of RAM, is not going to (particularly well) re-compile its entire environment, QEMU allows it nicely with distcc at a reasonable speed. (Albeit there is no distconfigure, but that's entirely an unrelated tanget of muck and despair and rants against configure, but we're not going there...) -- Mark A. Miller mark@mirell.org ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 10:18 ` Sam Ravnborg 2009-01-12 10:22 ` Mark A. Miller @ 2009-01-12 10:38 ` Paul Mundt 2009-01-14 2:51 ` Jamie Lokier 1 sibling, 1 reply; 135+ messages in thread From: Paul Mundt @ 2009-01-12 10:38 UTC (permalink / raw) To: Sam Ravnborg Cc: Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton On Mon, Jan 12, 2009 at 11:18:03AM +0100, Sam Ravnborg wrote: > On Sun, Jan 11, 2009 at 11:50:31PM -0600, Mark A. Miller wrote: > > On Sun, Jan 11, 2009 at 11:35 PM, Sam Ravnborg <sam@ravnborg.org> wrote: > > >> There are several other packages which are broken for embedded > > >> architectures, which I will hopefully attempt to fix by submitting patches > > >> upstream. But this is why we should be cautious about including new tools > > >> for compiling the kernel. Sam Ravnborg was correct in that a C program to do > > >> the work would be the proper way. But by not addressing a currently existing > > >> problem with an adequate replacement with something that does not exist > > >> currently, seems faulty. > > > > > > Why are "make headers_install" such a crucial thing for your > > > embedded environmnet? > > > > Sanity check. If the environment cannot replicate itself, then > > something has been faulty in the cross-compiling stage, that was used > > to propagate a native environment for the target architecture. > > So you actually build your target toolchain on your target? > This happens in a lot of places, like embedded gentoo ports, where almost all of the work is sent across distcc to a cross-compilation machine. In systems that use package management, it is done on the host through emulation, or painfully cross-compiled. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-12 10:38 ` Paul Mundt @ 2009-01-14 2:51 ` Jamie Lokier 2009-01-16 6:11 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: Jamie Lokier @ 2009-01-14 2:51 UTC (permalink / raw) To: Paul Mundt, Sam Ravnborg, Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, Rob Landley, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton Paul Mundt wrote: > This happens in a lot of places, like embedded gentoo ports, where almost > all of the work is sent across distcc to a cross-compilation machine. In > systems that use package management, it is done on the host through > emulation, or painfully cross-compiled. Ah yes, I remember using embedded Gentoo. 95% of the time in ./configure scripts, 5% in compilations. And this is on x86! I dread to think how slow it gets on something slow. -- Jamie ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-14 2:51 ` Jamie Lokier @ 2009-01-16 6:11 ` Rob Landley 2009-01-16 14:54 ` Valdis.Kletnieks 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-16 6:11 UTC (permalink / raw) To: Jamie Lokier Cc: Paul Mundt, Sam Ravnborg, Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton On Tuesday 13 January 2009 20:51:16 Jamie Lokier wrote: > Paul Mundt wrote: > > This happens in a lot of places, like embedded gentoo ports, where almost > > all of the work is sent across distcc to a cross-compilation machine. In > > systems that use package management, it is done on the host through > > emulation, or painfully cross-compiled. > > Ah yes, I remember using embedded Gentoo. > > 95% of the time in ./configure scripts, 5% in compilations. With SMP becoming commonplace, expect this to become the norm everywhere. Once you get to around quad processor, any C program with a ./configure step is probably going to take longer to configure than to compile. (Of course C++ manages to remain slow enough that autoconf isn't so obvious a bottleneck.) > And this is on x86! I dread to think how slow it gets on something > slow. My friend Mark's been experimenting with the amazon "cloud" thing, feeding in an image with a qemu instance and distcc+cross-compiler, and running builds under that. Renting an 8-way ~2.5 ghz server with 7 gigabytes of ram and 1.6 terabytes of disk is 80 cents/hour through them plus another few cents/day for bandwidth and persistent storage and such. That's likely to get cheaper as time goes on. We're still planning to buy a build server of our own to have something in- house, but for running nightly builds it's almost to the point where depreciation on the hardware is more than buying time from a server farm. Just _one_ of those 8-way servers is enough hardware to build an entire distro in an hour or so. What this really allows us to do is experiment with "how parallel can we get our build"? Because renting ten 8-way servers in a cluster is $8/hour, and distcc already scales trivially over that. Down the road what Firmware Linux is working towards is multiple qemu instances running in parallel with a central instance distributing builds to each one, so each can do its own ./configure in parallel, distribute compilation to the distccd instances as it has stuff to compile, and then package up the resulting binary into one of those portage tarballs and send it back to the central node to install on a network mount that the lot of 'em can mount as build context, so the packages can get their dependencies right. (You don't want your build taking place in a network mount, but your OS being on one you never write to isn't so bad as long as you have local storage to build in.) We'll probably leverage the heck out of Portage for this, and might wind up modifying it heavily. Dunno yet. (We can even force dependencies on portage so it doesn't need to calculate 'em, the central node can do that and then say "you have these packages, _build_"...) But yeah, hobbyists with a laptop, network access, and a monthly budget of $20 can do cluster builds these days. Rob P.S. I still hope autoconf dies off and the world wakes up and moves away from that. And from makefiles for that matter. But in the meantime, I can work around it with enough effort. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-16 6:11 ` Rob Landley @ 2009-01-16 14:54 ` Valdis.Kletnieks 2009-01-16 21:54 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: Valdis.Kletnieks @ 2009-01-16 14:54 UTC (permalink / raw) To: Rob Landley Cc: Jamie Lokier, Paul Mundt, Sam Ravnborg, Mark A. Miller, Bernd Petrovitsch, Leon Woestenberg, H. Peter Anvin, Embedded Linux mailing list, linux-kernel, Andrew Morton [-- Attachment #1: Type: text/plain, Size: 658 bytes --] On Fri, 16 Jan 2009 00:11:09 CST, Rob Landley said: > P.S. I still hope autoconf dies off and the world wakes up and moves away > from that. And from makefiles for that matter. But in the meantime, I can > work around it with enough effort. What do you propose autoconf and makefiles get replaced by? % wc pidgin/configure* 34287 118303 1004074 pidgin/configure 2499 7684 81532 pidgin/configure.ac Which you rather code, 2.5K lines of autoconf or 35K lines of configure script? As long as there's enough diversity to require configure scripts, there's going to be a demand for an autoconf-ish feature to ease writing them. [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-16 14:54 ` Valdis.Kletnieks @ 2009-01-16 21:54 ` Rob Landley 2009-01-17 9:51 ` Jamie Lokier 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-16 21:54 UTC (permalink / raw) To: Valdis.Kletnieks, Embedded Linux mailing list; +Cc: Sam Ravnborg, linux-kernel On Friday 16 January 2009 08:54:42 Valdis.Kletnieks@vt.edu wrote: > On Fri, 16 Jan 2009 00:11:09 CST, Rob Landley said: > > P.S. I still hope autoconf dies off and the world wakes up and moves > > away from that. And from makefiles for that matter. But in the > > meantime, I can work around it with enough effort. > > What do you propose autoconf and makefiles get replaced by? At a first guess, meaningful standards and an acknowledgement that Linux is a platform in its own right? Between the two of them, that's 90% of your problem right there. Not a new issue, here's a link from 2002: http://news.cnet.com/2100-1001-950180.html And here's a multi-vendor effort at a standards group (the 86open project to definite a Unix binary standard for x86 processors) dissolving itself way back in 1999 with a declaration that the Linux binary format already was an open standard and other unixes should just use things like lxrun to support that: http://www.telly.org/86open/ Also, it would be nice if configure steps could stop entangling 1) users providing preferences (command line options like --prefix or most of the -- enable and --disable flags) for which things like kconfig might make more sense, 2) probe for installed packages like zlib and enabling optional support if found, 3) questions like "does my build environment provide strlcpy()". Turning that into a big hairball does not help keep things simple. > % wc pidgin/configure* > 34287 118303 1004074 pidgin/configure > 2499 7684 81532 pidgin/configure.ac > > Which you rather code, 2.5K lines of autoconf or 35K lines of configure > script? I consider it a false dichotomy. I prefer "neither", and have seen it done successfully many times. I've never built pidgin from source, but I've got the output of the binutils build in a log file. How many of these tests are actually necessary on any Linux system: checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking for suffix of object files... o checking whether gcc accepts -g... yes checking for library containing strerror... none required checking how to run the C preprocessor... gcc -E checking whether build environment is sane... yes checking whether gcc and cc understand -c and -o together... yes checking for an ANSI C-conforming const... yes checking for inline... inline It just goes on and on and on like this. Tests like "checking whether byte ordering is bigendian... no" means "Either I didn't know endian.h existed, or I don't trust it to be there". How about the long stretches checking for the existence of header files specified by posix? Or this gem: checking for gawk... no checking for mawk... no checking for nawk... no checking for awk... awk (If you can use awk, when is it _not_ there? Probable answer: there was some broken version on irix back in 1992 with zero remaining seats today, and they never went back to clean anything out of the makefile because "simplifying the build" and "autoconf" do not go together, and because you can't sanely regression test against build environments nobody has anymore.) This argument is a can of worms, and the linux-kernel list probably isn't the best place for it. However, "install mingw on windows instead of trying to get your thing to build under Visual Studio" is a decent support strategy. Tinycc and Intel's icc both implemented gcc extensions to build the kernel, and the kernel's been removing such extensions where c99 versions are available since then. Things like uClibc implement standards and copy glibc extensions that are actually used. In the era of open source, it's now a viable strategy to specify, document, and require a standardized build environment. The way to make that build environment portable is to keep it simple and standardized, not to create a tower of #ifdefs testing the output of a huge environment probing shell script. Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-16 21:54 ` Rob Landley @ 2009-01-17 9:51 ` Jamie Lokier 2009-01-18 1:44 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: Jamie Lokier @ 2009-01-17 9:51 UTC (permalink / raw) To: Rob Landley Cc: Valdis.Kletnieks, Embedded Linux mailing list, Sam Ravnborg, linux-kernel Rob Landley wrote: > On Friday 16 January 2009 08:54:42 Valdis.Kletnieks@vt.edu wrote: > > On Fri, 16 Jan 2009 00:11:09 CST, Rob Landley said: > > > P.S. I still hope autoconf dies off and the world wakes up and moves > > > away from that. And from makefiles for that matter. But in the > > > meantime, I can work around it with enough effort. > > > > What do you propose autoconf and makefiles get replaced by? > > I've never built pidgin from source, but I've got the output of the binutils > build in a log file. > How many of these tests are actually necessary on an Linux system: None, but then it's not a Linux-only program that you're compiling. (Nor is it Linux-in-2009-only). If you _know_ you're running on Linux from a particular era, you can provide a config.cache file with the correct answers already filled in. I agree that Autoconf sucks (I've written enough sucking Autoconf macros myself, I hate it), but the tough part is providing a suitable replacement when you still want portable source code. > It just goes on and on and on like this. Tests like "checking > whether byte ordering is bigendian... no" means "Either I didn't > know endian.h existed, or I don't trust it to be there". How about > the long stretches checking for the existence of header files > specified by posix? You seem to be arguing for "let's make all our programs Linux-specific (and Glibc-specific in many cases)". Given all the problems you've seen with cross-compiling, let alone compiling for different OS platforms, that seems a little odd. -- Jamie ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-17 9:51 ` Jamie Lokier @ 2009-01-18 1:44 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-18 1:44 UTC (permalink / raw) To: Jamie Lokier Cc: Valdis.Kletnieks, Embedded Linux mailing list, Sam Ravnborg, linux-kernel On Saturday 17 January 2009 03:51:43 Jamie Lokier wrote: > Rob Landley wrote: > > On Friday 16 January 2009 08:54:42 Valdis.Kletnieks@vt.edu wrote: > > > On Fri, 16 Jan 2009 00:11:09 CST, Rob Landley said: > > > > P.S. I still hope autoconf dies off and the world wakes up and moves > > > > away from that. And from makefiles for that matter. But in the > > > > meantime, I can work around it with enough effort. > > > > > > What do you propose autoconf and makefiles get replaced by? > > > > I've never built pidgin from source, but I've got the output of the > > binutils build in a log file. > > How many of these tests are actually necessary on an Linux system: > > None, but then it's not a Linux-only program that you're compiling. > (Nor is it Linux-in-2009-only). Yeah, I noticed. It's not quite as bad as OpenSSL (where Linux support is intentionally an afterthought), but things like "Libtool" are supposed to be a NOP on ELF Linux and yet regularly screw up builds. (It's supposed to do nothing, and can't manage to do it correctly. That's sad.) > If you _know_ you're running on Linux from a particular era, you can > provide a config.cache file with the correct answers already filled in. And yet very few projects actually do. As for "from a particular era", just for fun I fired up the Red Hat 9 qemu image I keep around for this sort of thing, downloaded glibc 2.7 (the most recent one they bothered to cut a tarball for on ftp.gnu.org and one of the big autoconf offenders), and ran its ./configure. It died with: configure: error: *** These critical programs are missing or too old: gcc *** Check the INSTALL file for required versions. So you can't build a 2 year old version of glibc under a 6 year old version of Linux (which was the most popular Linux version in the world when it shipped, with about 50% market share among Linux seats). And yet glibc (one of the FSF's flagship projects) bothers doing extensive autoconf probes. Why? Autoconf isn't really _helping_ here... The bottom line is that if your assumption is that you have an open source application targeting an open source operating system, lots of the hoops you used to have to jump through just aren't very interesting anymore. > I agree that Autoconf sucks (I've written enough sucking Autoconf > macros myself, I hate it), but the tough part is providing a suitable > replacement when you still want portable source code. Depends on your definition of "portable". The unix wars of the 80's are over; they're pretty much all dead. Even the surviving legacy deployments of Solaris and AIX provide Linux emulation environments. And of course FreeBSD's done so for years: http://www.onlamp.com/pub/a/bsd/2006/01/12/Big_Scary_Daemons.html MacOS X and windows are still very much alive, but if you want to target those you can either A) treat them as Posix/SUSv3 (which both _claim_ to support), B) use cross platform libraries like SDL and opengl and program to their APIs, C) bother to do a proper port of the thing ala http://porting.openoffice.org/mac/ or http://www.kju-app.org/ or the way khtml wound up in Safari. For Windows there's Cygwin, running windows programs on Linux has Wine. Or just qemu/kvm in either direction. Basically, pick a standard to write to. If you want to write to posix and SUSv4, do it. If you want to add in LSB and the Linux man pages, go for it. But autoconf was designed for portability between Irix and HP-UX, which just doesn't come up much anymore. > > It just goes on and on and on like this. Tests like "checking > > whether byte ordering is bigendian... no" means "Either I didn't > > know endian.h existed, or I don't trust it to be there". How about > > the long stretches checking for the existence of header files > > specified by posix? > > You seem to be arguing for "let's make all our programs Linux-specific > (and Glibc-specific in many cases)". Checking for the existence of posix header files is Linux-specific? I'm saying there are many standards, and you can choose to adhere to standards like LP64 (which MacOSX, Linux, and the BSDs all support) and _say_ you do so and achieve portability that way. You also have "#if defined __linux__" and "#if defined __APPLE__" and so on, so header files can do a lot of the tests that people wind up doing with autoconf for some reason. And there will always be platforms you're NOT portable to. (Game consoles come to mind: your average autoconf recipe isn't going to make your program run on a PS3, XBox 360, or Wii. Unless you load Linux on those systems first and program for Linux.) > Given all the problems you've > seen with cross-compiling, let alone compiling for different OS > platforms, that seems a little odd. If I can't get Linux running on the hardware (which is seldom an interesting case anymore; it's on everything from cell phones to the S390), and I can't get a Linux emulation environment like Solaris' lxrun, aix5L, or cygwin, then I probably want a rigidly posix environment. (Heck, even wince has celib and gnuwince, not that I really care.) The extra "portability" isn't even going to buy you 5% more seats these days. It's really not that interesting. It's not that the world is homogenous now, it's that between open source and open standards we don't need giant if/else ladders with hundreds of tests to cover the interesting cases. And trying to achieve portability by relying on standards is a superior approach to trying to achieve portability via an infinite series of special case checks. > -- Jamie Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [0/3]: Simplify the kernel build by removing perl. 2009-01-04 3:06 ` Paul Mundt 2009-01-04 10:23 ` Leon Woestenberg @ 2009-01-04 16:22 ` Vladimir Dronnikov 1 sibling, 0 replies; 135+ messages in thread From: Vladimir Dronnikov @ 2009-01-04 16:22 UTC (permalink / raw) To: Paul Mundt, Rob Landley, H. Peter Anvin, Leon Woestenberg, Embedded Linux mailing list, linux-kernel, Andrew Morton, Sam Ravnborg > Let's look at the rationale presented so far in this thread: > > 1 - Being able to build the kernel natively on a constrained > target is useful, regardless of whether it is being used for > regression/stress testing or for headers installation or whatever > else. > > 2 - Cross-compiling perl is hard. > > 3 - Some oddly constrained target distributions manage to ship > with an entire toolchain yet fail to provide any implementation > of perl. > > 4 - It wasn't required before. OK, let's see. > If you have enough space and CPU power and > complete build environment to crunch away at the kernel for stress > testing natively, you can do the same with building perl and negating > point #2. It seems you meant "If you have enough space ... for stress testing natively, you _MUST ALSO_ do the same with building perl _TO JUST_ negate point #2". From this point of view your further arguments are obvious. > > #2 is another byproduct of your environment and generally a non-issue. > So you put a constraint on environment to build kernel. Not on a specific version of shell or gcc. But require SANE environment in rhetoric sence. In the absence of clear specification of sane environment it _IS_ an issue. Or simply: "sane" now reads "perlful enough"? > > If there is anything I missed, feel free to add it to the list. > The rationale of changes being proposed is to keep people able to make their choice on how to build and use their environment. If the code, which now has to be generated by perl scripts, was shipped with linux*.tar.bz2, I'd nullify the majority of my objections. Please, DO NOT require this code to be BUILT, and many would again be happy. You see, the total question is then reduces to some changes in makefiles (eliminating FORCE prerequisites). Otherwise you just force the number of people to invent and maintain "regress" patches, which is counter-progressive in all sences. Best regards, -- Vladimir V. Dronnikov ^ permalink raw reply [flat|nested] 135+ messages in thread
* PATCH [0/3]: Simplify the kernel build by removing perl v2 2009-01-02 8:07 PATCH [0/3]: Simplify the kernel build by removing perl Rob Landley ` (5 preceding siblings ...) 2009-01-03 22:54 ` Leon Woestenberg @ 2009-01-04 1:24 ` Rob Landley 2009-01-04 1:27 ` PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2) Rob Landley ` (2 more replies) 6 siblings, 3 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:24 UTC (permalink / raw) To: Embedded Linux mailing list Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Here's an updated set of patches to remove use of perl from the kernel build's "hot path" (roughly defined as "make allnoconfig; make; make headers_install"). This update incorporates feedback from Sam Ravnborg, Ted Tso, Joe Perches, Ingo Oeser, and others. It also fixes an integer overflow error in the first patch, and all three scripts have been retested under dash. ^ permalink raw reply [flat|nested] 135+ messages in thread
* PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2) 2009-01-04 1:24 ` PATCH [0/3]: Simplify the kernel build by removing perl v2 Rob Landley @ 2009-01-04 1:27 ` Rob Landley 2009-01-04 2:48 ` David Vrabel 2009-01-04 1:28 ` PATCH [2/3]: Remove perl from make headers_install Rob Landley 2009-01-04 1:29 ` PATCH [3/3]: Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh Rob Landley 2 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:27 UTC (permalink / raw) To: Embedded Linux mailing list Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg From: Rob Landley <rob@landley.net> Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell scriptis much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003. It requires a shell which can do 64 bit math, such as bash, busybox ash,or dash running on a 64 bit host. Changes from previous version: Redo ADJ32 math to avoid integer overflow for small HZ sizes (such as 24 or122). In the pathological case (HZ=1) both versions now produceUSEC_TO_HZ_ADJ32 of 0x7ffff79c842fa. (See source comments for details.) Also expand usage message, add error message when no 64 bit math available inshell (and suggest some shells that support it), add whitespace aroundoperators in equations, added underscores before __KERNEL_TIMECONST_H, changemakefile so script is responsible for creating output file, make script delete output file on error, change shebang to #!/bin/sh and test with dash and bash. Signed-off-by: Rob Landley <rob@landley.net>--- kernel/Makefile | 4 kernel/timeconst.pl | 378 ------------------------------------------ kernel/timeconst.sh | 149 ++++++++++++++++ 3 files changed, 151 insertions(+), 380 deletions(-) --- linux-2.6.28/kernel/timeconst.pl 2008-12-24 17:26:37.000000000 -0600+++ /dev/null 2008-11-21 04:46:41.000000000 -0600@@ -1,378 +0,0 @@-#!/usr/bin/perl-# ------------------------------------------------------------------------#-# Copyright 2007-2008 rPath, Inc. - All Rights Reserved-#-# This file is part of the Linux kernel, and is made available under-# the terms of the GNU General Public License version 2 or (at your-# option) any later version; incorporated herein by reference.-#-# ------------------------------------------------------------------------#--#-# Usage: timeconst.pl HZ > timeconst.h-#--# Precomputed values for systems without Math::BigInt-# Generated by:-# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 1200-%canned_values = (- 24 => [- '0xa6aaaaab','0x2aaaaaa',26,- 125,3,- '0xc49ba5e4','0x1fbe76c8b4',37,- 3,125,- '0xa2c2aaab','0xaaaa',16,- 125000,3,- '0xc9539b89','0x7fffbce4217d',47,- 3,125000,- ], 32 => [- '0xfa000000','0x6000000',27,- 125,4,- '0x83126e98','0xfdf3b645a',36,- 4,125,- '0xf4240000','0x0',17,- 31250,1,- '0x8637bd06','0x3fff79c842fa',46,- 1,31250,- ], 48 => [- '0xa6aaaaab','0x6aaaaaa',27,- 125,6,- '0xc49ba5e4','0xfdf3b645a',36,- 6,125,- '0xa2c2aaab','0x15555',17,- 62500,3,- '0xc9539b89','0x3fffbce4217d',46,- 3,62500,- ], 64 => [- '0xfa000000','0xe000000',28,- 125,8,- '0x83126e98','0x7ef9db22d',35,- 8,125,- '0xf4240000','0x0',18,- 15625,1,- '0x8637bd06','0x1fff79c842fa',45,- 1,15625,- ], 100 => [- '0xa0000000','0x0',28,- 10,1,- '0xcccccccd','0x733333333',35,- 1,10,- '0x9c400000','0x0',18,- 10000,1,- '0xd1b71759','0x1fff2e48e8a7',45,- 1,10000,- ], 122 => [- '0x8325c53f','0xfbcda3a',28,- 500,61,- '0xf9db22d1','0x7fbe76c8b',35,- 61,500,- '0x8012e2a0','0x3ef36',18,- 500000,61,- '0xffda4053','0x1ffffbce4217',45,- 61,500000,- ], 128 => [- '0xfa000000','0x1e000000',29,- 125,16,- '0x83126e98','0x3f7ced916',34,- 16,125,- '0xf4240000','0x40000',19,- 15625,2,- '0x8637bd06','0xfffbce4217d',44,- 2,15625,- ], 200 => [- '0xa0000000','0x0',29,- 5,1,- '0xcccccccd','0x333333333',34,- 1,5,- '0x9c400000','0x0',19,- 5000,1,- '0xd1b71759','0xfff2e48e8a7',44,- 1,5000,- ], 250 => [- '0x80000000','0x0',29,- 4,1,- '0x80000000','0x180000000',33,- 1,4,- '0xfa000000','0x0',20,- 4000,1,- '0x83126e98','0x7ff7ced9168',43,- 1,4000,- ], 256 => [- '0xfa000000','0x3e000000',30,- 125,32,- '0x83126e98','0x1fbe76c8b',33,- 32,125,- '0xf4240000','0xc0000',20,- 15625,4,- '0x8637bd06','0x7ffde7210be',43,- 4,15625,- ], 300 => [- '0xd5555556','0x2aaaaaaa',30,- 10,3,- '0x9999999a','0x1cccccccc',33,- 3,10,- '0xd0555556','0xaaaaa',20,- 10000,3,- '0x9d495183','0x7ffcb923a29',43,- 3,10000,- ], 512 => [- '0xfa000000','0x7e000000',31,- 125,64,- '0x83126e98','0xfdf3b645',32,- 64,125,- '0xf4240000','0x1c0000',21,- 15625,8,- '0x8637bd06','0x3ffef39085f',42,- 8,15625,- ], 1000 => [- '0x80000000','0x0',31,- 1,1,- '0x80000000','0x0',31,- 1,1,- '0xfa000000','0x0',22,- 1000,1,- '0x83126e98','0x1ff7ced9168',41,- 1,1000,- ], 1024 => [- '0xfa000000','0xfe000000',32,- 125,128,- '0x83126e98','0x7ef9db22',31,- 128,125,- '0xf4240000','0x3c0000',22,- 15625,16,- '0x8637bd06','0x1fff79c842f',41,- 16,15625,- ], 1200 => [- '0xd5555556','0xd5555555',32,- 5,6,- '0x9999999a','0x66666666',31,- 6,5,- '0xd0555556','0x2aaaaa',22,- 2500,3,- '0x9d495183','0x1ffcb923a29',41,- 3,2500,- ]-);--$has_bigint = eval 'use Math::BigInt qw(bgcd); 1;';--sub bint($)-{- my($x) = @_;- return Math::BigInt->new($x);-}--#-# Constants for division by reciprocal multiplication.-# (bits, numerator, denominator)-#-sub fmul($$$)-{- my ($b,$n,$d) = @_;-- $n = bint($n);- $d = bint($d);-- return scalar (($n << $b)+$d-bint(1))/$d;-}--sub fadj($$$)-{- my($b,$n,$d) = @_;-- $n = bint($n);- $d = bint($d);-- $d = $d/bgcd($n, $d);- return scalar (($d-bint(1)) << $b)/$d;-}--sub fmuls($$$) {- my($b,$n,$d) = @_;- my($s,$m);- my($thres) = bint(1) << ($b-1);-- $n = bint($n);- $d = bint($d);-- for ($s = 0; 1; $s++) {- $m = fmul($s,$n,$d);- return $s if ($m >= $thres);- }- return 0;-}--# Generate a hex value if the result fits in 64 bits;-# otherwise skip.-sub bignum_hex($) {- my($x) = @_;- my $s = $x->as_hex();-- return (length($s) > 18) ? undef : $s;-}--# Provides mul, adj, and shr factors for a specific-# (bit, time, hz) combination-sub muladj($$$) {- my($b, $t, $hz) = @_;- my $s = fmuls($b, $t, $hz);- my $m = fmul($s, $t, $hz);- my $a = fadj($s, $t, $hz);- return (bignum_hex($m), bignum_hex($a), $s);-}--# Provides numerator, denominator values-sub numden($$) {- my($n, $d) = @_;- my $g = bgcd($n, $d);- return ($n/$g, $d/$g);-}--# All values for a specific (time, hz) combo-sub conversions($$) {- my ($t, $hz) = @_;- my @val = ();-- # HZ_TO_xx- push(@val, muladj(32, $t, $hz));- push(@val, numden($t, $hz));-- # xx_TO_HZ- push(@val, muladj(32, $hz, $t));- push(@val, numden($hz, $t));-- return @val;-}--sub compute_values($) {- my($hz) = @_;- my @val = ();- my $s, $m, $a, $g;-- if (!$has_bigint) {- die "$0: HZ == $hz not canned and ".- "Math::BigInt not available\n";- }-- # MSEC conversions- push(@val, conversions(1000, $hz));-- # USEC conversions- push(@val, conversions(1000000, $hz));-- return @val;-}--sub outputval($$)-{- my($name, $val) = @_;- my $csuf;-- if (defined($val)) {- if ($name !~ /SHR/) {- $val = "U64_C($val)";- }- printf "#define %-23s %s\n", $name.$csuf, $val.$csuf;- }-}--sub output($@)-{- my($hz, @val) = @_;- my $pfx, $bit, $suf, $s, $m, $a;-- print "/* Automatically generated by kernel/timeconst.pl */\n";- print "/* Conversion constants for HZ == $hz */\n";- print "\n";- print "#ifndef KERNEL_TIMECONST_H\n";- print "#define KERNEL_TIMECONST_H\n";- print "\n";-- print "#include <linux/param.h>\n";- print "#include <linux/types.h>\n";-- print "\n";- print "#if HZ != $hz\n";- print "#error \"kernel/timeconst.h has the wrong HZ value!\"\n";- print "#endif\n";- print "\n";-- foreach $pfx ('HZ_TO_MSEC','MSEC_TO_HZ',- 'HZ_TO_USEC','USEC_TO_HZ') {- foreach $bit (32) {- foreach $suf ('MUL', 'ADJ', 'SHR') {- outputval("${pfx}_$suf$bit", shift(@val));- }- }- foreach $suf ('NUM', 'DEN') {- outputval("${pfx}_$suf", shift(@val));- }- }-- print "\n";- print "#endif /* KERNEL_TIMECONST_H */\n";-}--# Pretty-print Perl values-sub perlvals(@) {- my $v;- my @l = ();-- foreach $v (@_) {- if (!defined($v)) {- push(@l, 'undef');- } elsif ($v =~ /^0x/) {- push(@l, "\'".$v."\'");- } else {- push(@l, $v.'');- }- }- return join(',', @l);-}--($hz) = @ARGV;--# Use this to generate the %canned_values structure-if ($hz eq '--can') {- shift(@ARGV);- @hzlist = sort {$a <=> $b} (@ARGV);-- print "# Precomputed values for systems without Math::BigInt\n";- print "# Generated by:\n";- print "# timeconst.pl --can ", join(' ', @hzlist), "\n";- print "\%canned_values = (\n";- my $pf = "\t";- foreach $hz (@hzlist) {- my @values = compute_values($hz);- print "$pf$hz => [\n";- while (scalar(@values)) {- my $bit;- foreach $bit (32) {- my $m = shift(@values);- my $a = shift(@values);- my $s = shift(@values);- print "\t\t", perlvals($m,$a,$s), ",\n";- }- my $n = shift(@values);- my $d = shift(@values);- print "\t\t", perlvals($n,$d), ",\n";- }- print "\t]";- $pf = ', ';- }- print "\n);\n";-} else {- $hz += 0; # Force to number- if ($hz < 1) {- die "Usage: $0 HZ\n";- }-- @val = @{$canned_values{$hz}};- if (!defined(@val)) {- @val = compute_values($hz);- }- output($hz, @val);-}-exit 0;--- /dev/null 2008-11-21 04:46:41.000000000 -0600+++ linux-2.6.28-new/kernel/timeconst.sh 2009-01-03 15:24:01.000000000 -0600@@ -0,0 +1,147 @@+#!/bin/sh++if [ $# -ne 2 ]+then+ echo "Usage: timeconst.sh HZ FILENAME"+ echo+ echo "Generate a header file with constants for coverting between"+ echo "decimal HZ timer ticks and milisecond or microsecond delays."+ echo+ exit 1+fi++HZ=$1+shift+FILENAME=$1++# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math.++if [ $((1 << 32)) -lt 0 ]+then+ echo "timeconst.sh needs a shell with 64 bit math, such as bash,"+ echo "busybox ash, or dash running on a 64 bit host."+ exit 1+fi++# If this script exits for any reason before this trap is removed,+# delete the output file so a partial file won't confuse the build.++trap "rm $FILENAME" EXIT++# Output start of header file++cat > $FILENAME << EOF || exit 1+/* Automatically generated by kernel/timeconst.sh */+/* Conversion constants for HZ == $HZ */++#ifndef __KERNEL_TIMECONST_H+#define __KERNEL_TIMECONST_H++#include <linux/param.h>+#include <linux/types.h>++#if HZ != $HZ+#error "kernel/timeconst.h has the wrong HZ value!"+#endif++EOF++# For both Milliseconds and Microseconds++for i in "MSEC 1000" "USEC 1000000"+do+ NAME=$(echo $i | cut -d ' ' -f 1)+ PERIOD=$(echo $i | cut -d ' ' -f 2)++ # Find greatest common denominator (using Euclid's algorithm)++ A=$HZ+ B=$PERIOD++ while [ $B -ne 0 ]+ do+ C=$(( $A % $B ))+ A=$B+ B=$C+ done++ GCD=$A++ # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ)++ for DIRECTION in 0 1+ do+ if [ $DIRECTION -eq 0 ]+ then+ CONVERT="HZ_TO_${NAME}"+ FROM=$HZ+ TO=$PERIOD+ else+ CONVERT="${NAME}_TO_HZ"+ FROM=$PERIOD+ TO=$HZ+ fi++ # Calculate 32 significant bits of MUL32 data.++ SHIFT=0+ while true+ do+ # This can't overflow 64 bit math. Pathological case+ # (TO=1, FROM=1000000) uses around 32+20=52 bits.++ MUL32=$(( ( ( $TO << $SHIFT ) + $FROM - 1 ) / $FROM ))++ # Keep increasing $SHIFT until we've got 32 bits.++ [ $MUL32 -gt $(( 1 << 31 )) ] && break+ SHIFT=$(( $SHIFT + 1 ))+ done+ MUL32=$( printf %x $MUL32 )++ # ADJ32 is just (((FROM/GCD)-1)<<SHIFT)/(FROM/GCD) but this+ # can overflow 64 bit math (examples, HZ=24 or HZ=122).+ # Pathological case could use 32+20+20=72 bits. (And this is+ # the pathological case because a larger $HZ results in a+ # smaller $SHIFT, so even insane HZ>USEC cases should be ok.)++ # To get around this, we chop the bottom 32 bits off the+ # calculation and then reassemble it to avoid overflow:+ # 32+64=96, which is > 72.++ ADJ32=$(( $FROM / $GCD ))+ if [ $SHIFT -gt 32 ]+ then+ UPPER=$(( ( $ADJ32 - 1 ) << ( $SHIFT - 32 ) ))+ LOWER=$(( ( $UPPER % $ADJ32 ) << 32 ))+ ADJ32=$(( ( ( $UPPER / $ADJ32 ) << 32 ) + ( $LOWER / $ADJ32 )))+ else+ ADJ32=$(( ( ( $ADJ32 - 1 ) << $SHIFT) / $ADJ32 ))+ fi+ ADJ32=$( printf %x $ADJ32 )++ NUM=$(( $TO / $GCD ))+ DEN=$(( $FROM / $GCD ))++ # Output next chunk of header data to file++ (+ echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)" &&+ echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)" &&+ echo "#define ${CONVERT}_SHR32 $SHIFT" &&+ echo "#define ${CONVERT}_NUM U64_C($NUM)" &&+ echo "#define ${CONVERT}_DEN U64_C($DEN)"+ ) >> $FILENAME || exit 1+ done+done++(+ echo+ echo "#endif /* __KERNEL_TIMECHONST_H */"+) >> $FILENAME || exit 1++# Don't rm $FILENAME on exit anymore.++trap "" EXIT++exit 0--- linux-2.6.28/kernel/Makefile 2008-12-24 17:26:37.000000000 -0600+++ linux-2.6.28-new/kernel/Makefile 2009-01-03 00:40:21.000000000 -0600@@ -116,7 +116,7 @@ $(obj)/time.o: $(obj)/timeconst.h quiet_cmd_timeconst = TIMEC $@- cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@+ cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) $@ targets += timeconst.h-$(obj)/timeconst.h: $(src)/timeconst.pl FORCE+$(obj)/timeconst.h: $(src)/timeconst.sh FORCE $(call if_changed,timeconst)\0ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥ ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2) 2009-01-04 1:27 ` PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2) Rob Landley @ 2009-01-04 2:48 ` David Vrabel 2009-01-04 20:21 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: David Vrabel @ 2009-01-04 2:48 UTC (permalink / raw) To: Rob Landley Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg Rob Landley wrote: > From: Rob Landley <rob@landley.net> > > Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script > is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003. > > It requires a shell which can do 64 bit math, such as bash, busybox ash, > or dash running on a 64 bit host. I use Ubuntu (hence dash) on 32 bit systems so I think this needs to work with dash on 32 bit hosts. David ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2) 2009-01-04 2:48 ` David Vrabel @ 2009-01-04 20:21 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 20:21 UTC (permalink / raw) To: David Vrabel Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg On Saturday 03 January 2009 20:48:21 David Vrabel wrote: > Rob Landley wrote: > > From: Rob Landley <rob@landley.net> > > > > Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell > > script is much simpler, about 1/4 the size, and runs on Red Hat 9 from > > 2003. > > > > It requires a shell which can do 64 bit math, such as bash, busybox ash, > > or dash running on a 64 bit host. > > I use Ubuntu (hence dash) on 32 bit systems so I think this needs to > work with dash on 32 bit hosts. I have a qemu/images directory full of various OS images for testing purposes. I just fired up my jeos 7.10 image to make sure that even the most stripped- down version of Ubuntu ("just enough operating system) still installs bash by default, and it does. (It doesn't install a development toolchain, but it does install bash.) I also installed a 32 bit xubuntu 8.10 image (which took 4 hours for some reason, and which also has bash), and explicitly tested its 32-bit "/bin/dash", and that did 64-bit math too. So current versions of dash do offer 64 bit math on 32 bit platforms. > David Rob ^ permalink raw reply [flat|nested] 135+ messages in thread
* PATCH [2/3]: Remove perl from make headers_install. 2009-01-04 1:24 ` PATCH [0/3]: Simplify the kernel build by removing perl v2 Rob Landley 2009-01-04 1:27 ` PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2) Rob Landley @ 2009-01-04 1:28 ` Rob Landley 2009-01-04 1:29 ` PATCH [3/3]: Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh Rob Landley 2 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:28 UTC (permalink / raw) To: Embedded Linux mailing list Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg From: Rob Landley <rob@landley.net> Remove perl from make headers_install by replacing a perl script (doinga simple regex search and replace) with a smaller and faster shell scriptimplementation. The new shell script is a single for loop calling sed andpiping its output through unifdef to produce the target file. Changes from previous version: Added help text and a check for the rightnumber of arguments. Removed unused ARCH input from script and makefile(the makefile incorporates ARCH into INDIR, so the script doesn't care),fixed a whitespace mistake in the makefile pointed out by Sam Ravnborg,changed the shebang to #!/bin/sh and tested under bash and dash. put_changelog_here Signed-off-by: Rob Landley <rob@landley.net>--- scripts/Makefile.headersinst | 6 ++-- scripts/headers_install.pl | 46 --------------------------------- scripts/headers_install.sh | 36 +++++++++++++++++++++++++ 3 files changed, 39 insertions(+), 49 deletions(-) diff -ruN linux-2.6.28/scripts/headers_install.sh linux-2.6.28-new/scripts/headers_install.sh--- linux-2.6.28/scripts/headers_install.sh 1969-12-31 18:00:00.000000000 -0600+++ linux-2.6.28-new/scripts/headers_install.sh 2009-01-02 22:35:17.000000000 -0600@@ -0,0 +1,36 @@+#!/bin/sh++if [ $# -lt 2 ]+then+ echo "Usage: headers_install.sh INDIR OUTDIR [FILES...]+ echo+ echo "Prepares kernel header files for use by user space, by removing"+ echo "all compiler.h definitions and #includes, and removing any"+ echo "#ifdef __KERNEL__ sections."+ echo+ echo "INDIR: directory to read each kernel header FILE from."+ echo "OUTDIR: directory to write each userspace header FILE to."+ echo "FILES: list of header files to operate on."++ exit 1+fi++# Grab arguments++INDIR="$1"+shift+OUTDIR="$1"+shift++# Iterate through files listed on command line++for i in "$@"+do+ sed -r \+ -e 's/([ \t(])(__user|__force|__iomem)[ \t]/\1/g' \+ -e 's/__attribute_const__([ \t]|$)/\1/g' \+ -e 's@^#include <linux/compiler.h>@@' "$INDIR/$i" |+ scripts/unifdef -U__KERNEL__ - > "$OUTDIR/$i"+done++exit 0diff -ruN linux-2.6.28/scripts/Makefile.headersinst linux-2.6.28-new/scripts/Makefile.headersinst--- linux-2.6.28/scripts/Makefile.headersinst 2008-12-24 17:26:37.000000000 -0600+++ linux-2.6.28-new/scripts/Makefile.headersinst 2009-01-02 22:36:42.000000000 -0600@@ -44,8 +44,8 @@ quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ file$(if $(word 2, $(all-files)),s)) cmd_install = \- $(PERL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \- $(PERL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \+ $(CONFIG_SHELL) $< $(srctree)/$(obj) $(install) $(header-y); \+ $(CONFIG_SHELL) $< $(objtree)/$(obj) $(install) $(objhdr-y); \ touch $@ quiet_cmd_remove = REMOVE $(unwanted)@@ -64,7 +64,7 @@ @: targets += $(install-file)-$(install-file): scripts/headers_install.pl $(input-files) FORCE+$(install-file): scripts/headers_install.sh $(input-files) FORCE $(if $(unwanted),$(call cmd,remove),) $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) $(call if_changed,install)diff -ruN linux-2.6.28/scripts/headers_install.pl linux-2.6.28-new/scripts/headers_install.pl--- linux-2.6.28/scripts/headers_install.pl 2008-12-24 17:26:37.000000000 -0600+++ linux-2.6.28-new/scripts/headers_install.pl 1969-12-31 18:00:00.000000000 -0600@@ -1,46 +0,0 @@-#!/usr/bin/perl -w-#-# headers_install prepare the listed header files for use in-# user space and copy the files to their destination.-#-# Usage: headers_install.pl readdir installdir arch [files...]-# readdir: dir to open files-# installdir: dir to install the files-# arch: current architecture-# arch is used to force a reinstallation when the arch-# changes because kbuild then detect a command line change.-# files: list of files to check-#-# Step in preparation for users space:-# 1) Drop all use of compiler.h definitions-# 2) Drop include of compiler.h-# 3) Drop all sections defined out by __KERNEL__ (using unifdef)--use strict;--my ($readdir, $installdir, $arch, @files) = @ARGV;--my $unifdef = "scripts/unifdef -U__KERNEL__";--foreach my $file (@files) {- local *INFILE;- local *OUTFILE;- my $tmpfile = "$installdir/$file.tmp";- open(INFILE, "<$readdir/$file")- or die "$readdir/$file: $!\n";- open(OUTFILE, ">$tmpfile") or die "$tmpfile: $!\n";- while (my $line = <INFILE>) {- $line =~ s/([\s(])__user\s/$1/g;- $line =~ s/([\s(])__force\s/$1/g;- $line =~ s/([\s(])__iomem\s/$1/g;- $line =~ s/\s__attribute_const__\s/ /g;- $line =~ s/\s__attribute_const__$//g;- $line =~ s/^#include <linux\/compiler.h>//;- printf OUTFILE "%s", $line;- }- close OUTFILE;- close INFILE;- system $unifdef . " $tmpfile > $installdir/$file";- unlink $tmpfile;-}-exit 0;\0ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥ ^ permalink raw reply [flat|nested] 135+ messages in thread
* PATCH [3/3]: Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh 2009-01-04 1:24 ` PATCH [0/3]: Simplify the kernel build by removing perl v2 Rob Landley 2009-01-04 1:27 ` PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2) Rob Landley 2009-01-04 1:28 ` PATCH [2/3]: Remove perl from make headers_install Rob Landley @ 2009-01-04 1:29 ` Rob Landley 2 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-01-04 1:29 UTC (permalink / raw) To: Embedded Linux mailing list Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg From: Rob Landley <rob@landley.net> Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh. This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h. Changes from last time: changed shebang to #!/bin/sh and tested under bashand dash. Signed-off-by: Rob Landley <rob@landley.net>--- arch/x86/kernel/cpu/Makefile | 4 +-- arch/x86/kernel/cpu/mkcapflags.pl | 32 ---------------------------- arch/x86/kernel/cpu/mkcapflags.sh | 28 ++++++++++++++++++++++++ 3 files changed, 30 insertions(+), 34 deletions(-) diff -ruN linux-2.6.28/arch/x86/kernel/cpu/Makefile linux-2.6.28-new/arch/x86/kernel/cpu/Makefile--- linux-2.6.28/arch/x86/kernel/cpu/Makefile 2008-12-24 17:26:37.000000000 -0600+++ linux-2.6.28-new/arch/x86/kernel/cpu/Makefile 2009-01-02 01:10:00.000000000 -0600@@ -23,10 +23,10 @@ obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o quiet_cmd_mkcapflags = MKCAP $@- cmd_mkcapflags = $(PERL) $(srctree)/$(src)/mkcapflags.pl $< $@+ cmd_mkcapflags = $(CONFIG_SHELL) $(srctree)/$(src)/mkcapflags.sh $< $@ cpufeature = $(src)/../../include/asm/cpufeature.h targets += capflags.c-$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.pl FORCE+$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.sh FORCE $(call if_changed,mkcapflags)diff -ruN linux-2.6.28/arch/x86/kernel/cpu/mkcapflags.pl linux-2.6.28-new/arch/x86/kernel/cpu/mkcapflags.pl--- linux-2.6.28/arch/x86/kernel/cpu/mkcapflags.pl 2008-12-24 17:26:37.000000000 -0600+++ linux-2.6.28-new/arch/x86/kernel/cpu/mkcapflags.pl 1969-12-31 18:00:00.000000000 -0600@@ -1,32 +0,0 @@-#!/usr/bin/perl-#-# Generate the x86_cap_flags[] array from include/asm-x86/cpufeature.h-#--($in, $out) = @ARGV;--open(IN, "< $in\0") or die "$0: cannot open: $in: $!\n";-open(OUT, "> $out\0") or die "$0: cannot create: $out: $!\n";--print OUT "#include <asm/cpufeature.h>\n\n";-print OUT "const char * const x86_cap_flags[NCAPINTS*32] = {\n";--while (defined($line = <IN>)) {- if ($line =~ /^\s*\#\s*define\s+(X86_FEATURE_(\S+))\s+(.*)$/) {- $macro = $1;- $feature = $2;- $tail = $3;- if ($tail =~ /\/\*\s*\"([^"]*)\".*\*\//) {- $feature = $1;- }-- if ($feature ne '') {- printf OUT "\t%-32s = \"%s\",\n",- "[$macro]", "\L$feature";- }- }-}-print OUT "};\n";--close(IN);-close(OUT);diff -ruN linux-2.6.28/arch/x86/kernel/cpu/mkcapflags.sh linux-2.6.28-new/arch/x86/kernel/cpu/mkcapflags.sh--- linux-2.6.28/arch/x86/kernel/cpu/mkcapflags.sh 1969-12-31 18:00:00.000000000 -0600+++ linux-2.6.28-new/arch/x86/kernel/cpu/mkcapflags.sh 2009-01-02 01:10:00.000000000 -0600@@ -0,0 +1,28 @@+#!/bin/sh+#+# Generate the x86_cap_flags[] array from include/asm/cpufeature.h+#++IN=$1+OUT=$2++(+ echo "#include <asm/cpufeature.h>"+ echo ""+ echo "const char * const x86_cap_flags[NCAPINTS*32] = {"++ # Iterate through any input lines starting with #define X86_FEATURE_+ sed -n -e 's/\t/ /g' -e 's/^ *# *define *X86_FEATURE_//p' $IN |+ while read i+ do+ # Name is everything up to the first whitespace+ NAME="$(echo "$i" | sed 's/ .*//')"++ # If the /* comment */ starts with a quote string, grab that.+ VALUE="$(echo "$i" | sed -n 's@.*/\* *\("[^"]*"\).*\*/@\1@p')"+ [ -z "$VALUE" ] && VALUE="\"$(echo "$NAME" | tr A-Z a-z)\""++ [ "$VALUE" != '""' ] && echo " [X86_FEATURE_$NAME] = $VALUE,"+ done+ echo "};"+) > $OUT\0ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥ ^ permalink raw reply [flat|nested] 135+ messages in thread
* [PATCH 0/3] Perl removal patches for 2.6.31 @ 2009-09-19 1:25 Rob Landley 2009-09-19 1:33 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-09-19 1:25 UTC (permalink / raw) To: kernel list Perl dependencies were introduced into the build system in 2.6.25. This removes the central ones preventing a bootable kernel from building without perl. For the record, the perl removal patches I submitted for 2.6.30, still work fine (they apply with offsets). Here are versions that apply without offsets to 2.6.31. Rob -- Latency is more important than throughput. It's that simple. - Linus Torvalds ^ permalink raw reply [flat|nested] 135+ messages in thread
* [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-09-19 1:25 [PATCH 0/3] Perl removal patches for 2.6.31 Rob Landley @ 2009-09-19 1:33 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-09-19 1:33 UTC (permalink / raw) To: kernel list From: Rob Landley <rob@xxxxxxxxxxx> Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003. Peter Anvin added this perl to 2.6.25. Before that, the kernel had never required perl to build. Signed-off-by: Rob Landley <rob@xxxxxxxxxxx> -- kernel/Makefile | 4 kernel/timeconst.pl | 378 ------------------------------------------ kernel/timeconst.sh | 91 ++++++++++ 3 files changed, 93 insertions(+), 380 deletions(-) diff -ruN linux-2.6.30/kernel/Makefile linux-2.6.30.new/kernel/Makefile --- linux-2.6.30/kernel/Makefile 2009-06-09 22:05:27.000000000 -0500 +++ linux-2.6.30.new/kernel/Makefile 2009-06-22 14:29:16.000000000 -0500 @@ -127,7 +127,7 @@ $(obj)/time.o: $(obj)/timeconst.h quiet_cmd_timeconst = TIMEC $@ - cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@ + cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) $@ targets += timeconst.h -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE +$(obj)/timeconst.h: $(src)/timeconst.sh FORCE $(call if_changed,timeconst) diff -ruN linux-2.6.30/kernel/timeconst.pl linux-2.6.30.new/kernel/timeconst.pl --- linux-2.6.30/kernel/timeconst.pl 2009-06-09 22:05:27.000000000 -0500 +++ linux-2.6.30.new/kernel/timeconst.pl 1969-12-31 18:00:00.000000000 -0600 @@ -1,378 +0,0 @@ -#!/usr/bin/perl -# ----------------------------------------------------------------------- -# -# Copyright 2007-2008 rPath, Inc. - All Rights Reserved -# -# This file is part of the Linux kernel, and is made available under -# the terms of the GNU General Public License version 2 or (at your -# option) any later version; incorporated herein by reference. -# -# ----------------------------------------------------------------------- -# - -# -# Usage: timeconst.pl HZ > timeconst.h -# - -# Precomputed values for systems without Math::BigInt -# Generated by: -# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 1200 -%canned_values = ( - 24 => [ - '0xa6aaaaab','0x2aaaaaa',26, - 125,3, - '0xc49ba5e4','0x1fbe76c8b4',37, - 3,125, - '0xa2c2aaab','0xaaaa',16, - 125000,3, - '0xc9539b89','0x7fffbce4217d',47, - 3,125000, - ], 32 => [ - '0xfa000000','0x6000000',27, - 125,4, - '0x83126e98','0xfdf3b645a',36, - 4,125, - '0xf4240000','0x0',17, - 31250,1, - '0x8637bd06','0x3fff79c842fa',46, - 1,31250, - ], 48 => [ - '0xa6aaaaab','0x6aaaaaa',27, - 125,6, - '0xc49ba5e4','0xfdf3b645a',36, - 6,125, - '0xa2c2aaab','0x15555',17, - 62500,3, - '0xc9539b89','0x3fffbce4217d',46, - 3,62500, - ], 64 => [ - '0xfa000000','0xe000000',28, - 125,8, - '0x83126e98','0x7ef9db22d',35, - 8,125, - '0xf4240000','0x0',18, - 15625,1, - '0x8637bd06','0x1fff79c842fa',45, - 1,15625, - ], 100 => [ - '0xa0000000','0x0',28, - 10,1, - '0xcccccccd','0x733333333',35, - 1,10, - '0x9c400000','0x0',18, - 10000,1, - '0xd1b71759','0x1fff2e48e8a7',45, - 1,10000, - ], 122 => [ - '0x8325c53f','0xfbcda3a',28, - 500,61, - '0xf9db22d1','0x7fbe76c8b',35, - 61,500, - '0x8012e2a0','0x3ef36',18, - 500000,61, - '0xffda4053','0x1ffffbce4217',45, - 61,500000, - ], 128 => [ - '0xfa000000','0x1e000000',29, - 125,16, - '0x83126e98','0x3f7ced916',34, - 16,125, - '0xf4240000','0x40000',19, - 15625,2, - '0x8637bd06','0xfffbce4217d',44, - 2,15625, - ], 200 => [ - '0xa0000000','0x0',29, - 5,1, - '0xcccccccd','0x333333333',34, - 1,5, - '0x9c400000','0x0',19, - 5000,1, - '0xd1b71759','0xfff2e48e8a7',44, - 1,5000, - ], 250 => [ - '0x80000000','0x0',29, - 4,1, - '0x80000000','0x180000000',33, - 1,4, - '0xfa000000','0x0',20, - 4000,1, - '0x83126e98','0x7ff7ced9168',43, - 1,4000, - ], 256 => [ - '0xfa000000','0x3e000000',30, - 125,32, - '0x83126e98','0x1fbe76c8b',33, - 32,125, - '0xf4240000','0xc0000',20, - 15625,4, - '0x8637bd06','0x7ffde7210be',43, - 4,15625, - ], 300 => [ - '0xd5555556','0x2aaaaaaa',30, - 10,3, - '0x9999999a','0x1cccccccc',33, - 3,10, - '0xd0555556','0xaaaaa',20, - 10000,3, - '0x9d495183','0x7ffcb923a29',43, - 3,10000, - ], 512 => [ - '0xfa000000','0x7e000000',31, - 125,64, - '0x83126e98','0xfdf3b645',32, - 64,125, - '0xf4240000','0x1c0000',21, - 15625,8, - '0x8637bd06','0x3ffef39085f',42, - 8,15625, - ], 1000 => [ - '0x80000000','0x0',31, - 1,1, - '0x80000000','0x0',31, - 1,1, - '0xfa000000','0x0',22, - 1000,1, - '0x83126e98','0x1ff7ced9168',41, - 1,1000, - ], 1024 => [ - '0xfa000000','0xfe000000',32, - 125,128, - '0x83126e98','0x7ef9db22',31, - 128,125, - '0xf4240000','0x3c0000',22, - 15625,16, - '0x8637bd06','0x1fff79c842f',41, - 16,15625, - ], 1200 => [ - '0xd5555556','0xd5555555',32, - 5,6, - '0x9999999a','0x66666666',31, - 6,5, - '0xd0555556','0x2aaaaa',22, - 2500,3, - '0x9d495183','0x1ffcb923a29',41, - 3,2500, - ] -); - -$has_bigint = eval 'use Math::BigInt qw(bgcd); 1;'; - -sub bint($) -{ - my($x) = @_; - return Math::BigInt->new($x); -} - -# -# Constants for division by reciprocal multiplication. -# (bits, numerator, denominator) -# -sub fmul($$$) -{ - my ($b,$n,$d) = @_; - - $n = bint($n); - $d = bint($d); - - return scalar (($n << $b)+$d-bint(1))/$d; -} - -sub fadj($$$) -{ - my($b,$n,$d) = @_; - - $n = bint($n); - $d = bint($d); - - $d = $d/bgcd($n, $d); - return scalar (($d-bint(1)) << $b)/$d; -} - -sub fmuls($$$) { - my($b,$n,$d) = @_; - my($s,$m); - my($thres) = bint(1) << ($b-1); - - $n = bint($n); - $d = bint($d); - - for ($s = 0; 1; $s++) { - $m = fmul($s,$n,$d); - return $s if ($m >= $thres); - } - return 0; -} - -# Generate a hex value if the result fits in 64 bits; -# otherwise skip. -sub bignum_hex($) { - my($x) = @_; - my $s = $x->as_hex(); - - return (length($s) > 18) ? undef : $s; -} - -# Provides mul, adj, and shr factors for a specific -# (bit, time, hz) combination -sub muladj($$$) { - my($b, $t, $hz) = @_; - my $s = fmuls($b, $t, $hz); - my $m = fmul($s, $t, $hz); - my $a = fadj($s, $t, $hz); - return (bignum_hex($m), bignum_hex($a), $s); -} - -# Provides numerator, denominator values -sub numden($$) { - my($n, $d) = @_; - my $g = bgcd($n, $d); - return ($n/$g, $d/$g); -} - -# All values for a specific (time, hz) combo -sub conversions($$) { - my ($t, $hz) = @_; - my @val = (); - - # HZ_TO_xx - push(@val, muladj(32, $t, $hz)); - push(@val, numden($t, $hz)); - - # xx_TO_HZ - push(@val, muladj(32, $hz, $t)); - push(@val, numden($hz, $t)); - - return @val; -} - -sub compute_values($) { - my($hz) = @_; - my @val = (); - my $s, $m, $a, $g; - - if (!$has_bigint) { - die "$0: HZ == $hz not canned and ". - "Math::BigInt not available\n"; - } - - # MSEC conversions - push(@val, conversions(1000, $hz)); - - # USEC conversions - push(@val, conversions(1000000, $hz)); - - return @val; -} - -sub outputval($$) -{ - my($name, $val) = @_; - my $csuf; - - if (defined($val)) { - if ($name !~ /SHR/) { - $val = "U64_C($val)"; - } - printf "#define %-23s %s\n", $name.$csuf, $val.$csuf; - } -} - -sub output($@) -{ - my($hz, @val) = @_; - my $pfx, $bit, $suf, $s, $m, $a; - - print "/* Automatically generated by kernel/timeconst.pl */\n"; - print "/* Conversion constants for HZ == $hz */\n"; - print "\n"; - print "#ifndef KERNEL_TIMECONST_H\n"; - print "#define KERNEL_TIMECONST_H\n"; - print "\n"; - - print "#include <linux/param.h>\n"; - print "#include <linux/types.h>\n"; - - print "\n"; - print "#if HZ != $hz\n"; - print "#error \"kernel/timeconst.h has the wrong HZ value!\"\n"; - print "#endif\n"; - print "\n"; - - foreach $pfx ('HZ_TO_MSEC','MSEC_TO_HZ', - 'HZ_TO_USEC','USEC_TO_HZ') { - foreach $bit (32) { - foreach $suf ('MUL', 'ADJ', 'SHR') { - outputval("${pfx}_$suf$bit", shift(@val)); - } - } - foreach $suf ('NUM', 'DEN') { - outputval("${pfx}_$suf", shift(@val)); - } - } - - print "\n"; - print "#endif /* KERNEL_TIMECONST_H */\n"; -} - -# Pretty-print Perl values -sub perlvals(@) { - my $v; - my @l = (); - - foreach $v (@_) { - if (!defined($v)) { - push(@l, 'undef'); - } elsif ($v =~ /^0x/) { - push(@l, "\'".$v."\'"); - } else { - push(@l, $v.''); - } - } - return join(',', @l); -} - -($hz) = @ARGV; - -# Use this to generate the %canned_values structure -if ($hz eq '--can') { - shift(@ARGV); - @hzlist = sort {$a <=> $b} (@ARGV); - - print "# Precomputed values for systems without Math::BigInt\n"; - print "# Generated by:\n"; - print "# timeconst.pl --can ", join(' ', @hzlist), "\n"; - print "\%canned_values = (\n"; - my $pf = "\t"; - foreach $hz (@hzlist) { - my @values = compute_values($hz); - print "$pf$hz => [\n"; - while (scalar(@values)) { - my $bit; - foreach $bit (32) { - my $m = shift(@values); - my $a = shift(@values); - my $s = shift(@values); - print "\t\t", perlvals($m,$a,$s), ",\n"; - } - my $n = shift(@values); - my $d = shift(@values); - print "\t\t", perlvals($n,$d), ",\n"; - } - print "\t]"; - $pf = ', '; - } - print "\n);\n"; -} else { - $hz += 0; # Force to number - if ($hz < 1) { - die "Usage: $0 HZ\n"; - } - - @val = @{$canned_values{$hz}}; - if (!defined(@val)) { - @val = compute_values($hz); - } - output($hz, @val); -} -exit 0; diff -ruN linux-2.6.30/kernel/timeconst.sh linux-2.6.30.new/kernel/timeconst.sh --- linux-2.6.30/kernel/timeconst.sh 1969-12-31 18:00:00.000000000 -0600 +++ linux-2.6.30.new/kernel/timeconst.sh 2009-06-22 14:29:16.000000000 -0500 @@ -0,0 +1,148 @@ +#!/bin/sh + +if [ $# -ne 2 ] +then + echo "Usage: timeconst.sh HZ FILENAME" + echo + echo "Generate a header file with constants for coverting between" + echo "decimal HZ timer ticks and milisecond or microsecond delays." + echo + exit 1 +fi + +HZ=$1 +shift +FILENAME=$1 + +# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math. + +if [ $((1 << 32)) -lt 0 ] +then + echo "timeconst.sh needs a shell with 64 bit math, such as bash," + echo "busybox ash, or dash running on a 64 bit host." + exit 1 +fi + +# If this script exits for any reason before this trap is removed, +# delete the output file so a partial file won't confuse the build. + +trap "rm $FILENAME" EXIT + +# Output start of header file + +cat > $FILENAME << EOF || exit 1 +/* Automatically generated by kernel/timeconst.sh */ +/* Conversion constants for HZ == $HZ */ + +#ifndef __KERNEL_TIMECONST_H +#define __KERNEL_TIMECONST_H + +#include <linux/param.h> +#include <linux/types.h> + +#if HZ != $HZ +#error "kernel/timeconst.h has the wrong HZ value!" +#endif + +EOF + +# For both Milliseconds and Microseconds + +cat << EOF | +MSEC 1000 +USEC 1000000 +EOF +while read NAME PERIOD +do + # Find greatest common denominator (using Euclid's algorithm) + + A=$HZ + B=$PERIOD + + while [ $B -ne 0 ] + do + C=$(( $A % $B )) + A=$B + B=$C + done + + GCD=$A + + # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ) + + for DIRECTION in 0 1 + do + if [ $DIRECTION -eq 0 ] + then + CONVERT="HZ_TO_${NAME}" + FROM=$HZ + TO=$PERIOD + else + CONVERT="${NAME}_TO_HZ" + FROM=$PERIOD + TO=$HZ + fi + + # Calculate 32 significant bits of MUL32 data. + + SHIFT=0 + while true + do + # This can't overflow 64 bit math. Pathological case + # (TO=1, FROM=1000000) uses around 32+20=52 bits. + + MUL32=$(( ( ( $TO << $SHIFT ) + $FROM - 1 ) / $FROM )) + + # Keep increasing $SHIFT until we've got 32 bits. + + [ $MUL32 -gt $(( 1 << 31 )) ] && break + SHIFT=$(( $SHIFT + 1 )) + done + MUL32=$( printf %x $MUL32 ) + + # ADJ32 is just (((FROM/GCD)-1)<<SHIFT)/(FROM/GCD) but this + # can overflow 64 bit math (examples, HZ=24 or HZ=122). + # Pathological case could use 32+20+20=72 bits. (And this is + # the pathological case because a larger $HZ results in a + # smaller $SHIFT, so even insane HZ>USEC cases should be ok.) + + # To get around this, we chop the bottom 32 bits off the + # calculation and then reassemble it to avoid overflow: + # 32+64=96, which is > 72. + + ADJ32=$(( $FROM / $GCD )) + if [ $SHIFT -gt 32 ] + then + UPPER=$(( ( $ADJ32 - 1 ) << ( $SHIFT - 32 ) )) + LOWER=$(( ( $UPPER % $ADJ32 ) << 32 )) + ADJ32=$(( ( ( $UPPER / $ADJ32 ) << 32 ) + ( $LOWER / $ADJ32 ))) + else + ADJ32=$(( ( ( $ADJ32 - 1 ) << $SHIFT) / $ADJ32 )) + fi + ADJ32=$( printf %x $ADJ32 ) + + NUM=$(( $TO / $GCD )) + DEN=$(( $FROM / $GCD )) + + # Output next chunk of header data to file + + ( + echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)" && + echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)" && + echo "#define ${CONVERT}_SHR32 $SHIFT" && + echo "#define ${CONVERT}_NUM U64_C($NUM)" && + echo "#define ${CONVERT}_DEN U64_C($DEN)" + ) >> $FILENAME || exit 1 + done +done + +( + echo + echo "#endif /* __KERNEL_TIMECHONST_H */" +) >> $FILENAME || exit 1 + +# Don't rm $FILENAME on exit anymore. + +trap "" EXIT + +exit 0 -- Latency is more important than throughput. It's that simple. - Linus Torvalds ^ permalink raw reply [flat|nested] 135+ messages in thread
* [PATCH 0/3] clean out Perl from build system. @ 2009-12-08 9:17 Rob Landley 2009-12-08 9:19 ` [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-12-08 9:17 UTC (permalink / raw) To: linux-kernel Perl was introduced as a build dependency in 2.6.25. My cross-compile environment does not use perl, so ever since then I've patched it back out. With these patches I've personally built kernels for x86, x86_64, arm, mips, powerpc, sparc, sh4, and m68k. -- Latency is more important than throughput. It's that simple. - Linus Torvalds ^ permalink raw reply [flat|nested] 135+ messages in thread
* [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-08 9:17 [PATCH 0/3] clean out Perl from build system Rob Landley @ 2009-12-08 9:19 ` Rob Landley 2009-12-09 15:45 ` Michal Marek 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-12-08 9:19 UTC (permalink / raw) To: linux-kernel From: Rob Landley <rob@landley.net> Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003. Signed-off-by: Rob Landley <rob@landley.net> -- kernel/Makefile | 4 kernel/timeconst.pl | 378 ------------------------------------------ kernel/timeconst.sh | 91 ++++++++++ 3 files changed, 93 insertions(+), 380 deletions(-) diff -ruN linux-2.6.30/kernel/Makefile linux-2.6.30.new/kernel/Makefile --- linux-2.6.30/kernel/Makefile 2009-06-09 22:05:27.000000000 -0500 +++ linux-2.6.30.new/kernel/Makefile 2009-06-22 14:29:16.000000000 -0500 @@ -123,7 +123,7 @@ $(obj)/time.o: $(obj)/timeconst.h quiet_cmd_timeconst = TIMEC $@ - cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@ + cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) $@ targets += timeconst.h -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE +$(obj)/timeconst.h: $(src)/timeconst.sh FORCE $(call if_changed,timeconst) diff -ruN linux-2.6.30/kernel/timeconst.pl linux-2.6.30.new/kernel/timeconst.pl --- linux-2.6.30/kernel/timeconst.pl 2009-06-09 22:05:27.000000000 -0500 +++ linux-2.6.30.new/kernel/timeconst.pl 1969-12-31 18:00:00.000000000 -0600 @@ -1,378 +0,0 @@ -#!/usr/bin/perl -# ----------------------------------------------------------------------- -# -# Copyright 2007-2008 rPath, Inc. - All Rights Reserved -# -# This file is part of the Linux kernel, and is made available under -# the terms of the GNU General Public License version 2 or (at your -# option) any later version; incorporated herein by reference. -# -# ----------------------------------------------------------------------- -# - -# -# Usage: timeconst.pl HZ > timeconst.h -# - -# Precomputed values for systems without Math::BigInt -# Generated by: -# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 1200 -%canned_values = ( - 24 => [ - '0xa6aaaaab','0x2aaaaaa',26, - 125,3, - '0xc49ba5e4','0x1fbe76c8b4',37, - 3,125, - '0xa2c2aaab','0xaaaa',16, - 125000,3, - '0xc9539b89','0x7fffbce4217d',47, - 3,125000, - ], 32 => [ - '0xfa000000','0x6000000',27, - 125,4, - '0x83126e98','0xfdf3b645a',36, - 4,125, - '0xf4240000','0x0',17, - 31250,1, - '0x8637bd06','0x3fff79c842fa',46, - 1,31250, - ], 48 => [ - '0xa6aaaaab','0x6aaaaaa',27, - 125,6, - '0xc49ba5e4','0xfdf3b645a',36, - 6,125, - '0xa2c2aaab','0x15555',17, - 62500,3, - '0xc9539b89','0x3fffbce4217d',46, - 3,62500, - ], 64 => [ - '0xfa000000','0xe000000',28, - 125,8, - '0x83126e98','0x7ef9db22d',35, - 8,125, - '0xf4240000','0x0',18, - 15625,1, - '0x8637bd06','0x1fff79c842fa',45, - 1,15625, - ], 100 => [ - '0xa0000000','0x0',28, - 10,1, - '0xcccccccd','0x733333333',35, - 1,10, - '0x9c400000','0x0',18, - 10000,1, - '0xd1b71759','0x1fff2e48e8a7',45, - 1,10000, - ], 122 => [ - '0x8325c53f','0xfbcda3a',28, - 500,61, - '0xf9db22d1','0x7fbe76c8b',35, - 61,500, - '0x8012e2a0','0x3ef36',18, - 500000,61, - '0xffda4053','0x1ffffbce4217',45, - 61,500000, - ], 128 => [ - '0xfa000000','0x1e000000',29, - 125,16, - '0x83126e98','0x3f7ced916',34, - 16,125, - '0xf4240000','0x40000',19, - 15625,2, - '0x8637bd06','0xfffbce4217d',44, - 2,15625, - ], 200 => [ - '0xa0000000','0x0',29, - 5,1, - '0xcccccccd','0x333333333',34, - 1,5, - '0x9c400000','0x0',19, - 5000,1, - '0xd1b71759','0xfff2e48e8a7',44, - 1,5000, - ], 250 => [ - '0x80000000','0x0',29, - 4,1, - '0x80000000','0x180000000',33, - 1,4, - '0xfa000000','0x0',20, - 4000,1, - '0x83126e98','0x7ff7ced9168',43, - 1,4000, - ], 256 => [ - '0xfa000000','0x3e000000',30, - 125,32, - '0x83126e98','0x1fbe76c8b',33, - 32,125, - '0xf4240000','0xc0000',20, - 15625,4, - '0x8637bd06','0x7ffde7210be',43, - 4,15625, - ], 300 => [ - '0xd5555556','0x2aaaaaaa',30, - 10,3, - '0x9999999a','0x1cccccccc',33, - 3,10, - '0xd0555556','0xaaaaa',20, - 10000,3, - '0x9d495183','0x7ffcb923a29',43, - 3,10000, - ], 512 => [ - '0xfa000000','0x7e000000',31, - 125,64, - '0x83126e98','0xfdf3b645',32, - 64,125, - '0xf4240000','0x1c0000',21, - 15625,8, - '0x8637bd06','0x3ffef39085f',42, - 8,15625, - ], 1000 => [ - '0x80000000','0x0',31, - 1,1, - '0x80000000','0x0',31, - 1,1, - '0xfa000000','0x0',22, - 1000,1, - '0x83126e98','0x1ff7ced9168',41, - 1,1000, - ], 1024 => [ - '0xfa000000','0xfe000000',32, - 125,128, - '0x83126e98','0x7ef9db22',31, - 128,125, - '0xf4240000','0x3c0000',22, - 15625,16, - '0x8637bd06','0x1fff79c842f',41, - 16,15625, - ], 1200 => [ - '0xd5555556','0xd5555555',32, - 5,6, - '0x9999999a','0x66666666',31, - 6,5, - '0xd0555556','0x2aaaaa',22, - 2500,3, - '0x9d495183','0x1ffcb923a29',41, - 3,2500, - ] -); - -$has_bigint = eval 'use Math::BigInt qw(bgcd); 1;'; - -sub bint($) -{ - my($x) = @_; - return Math::BigInt->new($x); -} - -# -# Constants for division by reciprocal multiplication. -# (bits, numerator, denominator) -# -sub fmul($$$) -{ - my ($b,$n,$d) = @_; - - $n = bint($n); - $d = bint($d); - - return scalar (($n << $b)+$d-bint(1))/$d; -} - -sub fadj($$$) -{ - my($b,$n,$d) = @_; - - $n = bint($n); - $d = bint($d); - - $d = $d/bgcd($n, $d); - return scalar (($d-bint(1)) << $b)/$d; -} - -sub fmuls($$$) { - my($b,$n,$d) = @_; - my($s,$m); - my($thres) = bint(1) << ($b-1); - - $n = bint($n); - $d = bint($d); - - for ($s = 0; 1; $s++) { - $m = fmul($s,$n,$d); - return $s if ($m >= $thres); - } - return 0; -} - -# Generate a hex value if the result fits in 64 bits; -# otherwise skip. -sub bignum_hex($) { - my($x) = @_; - my $s = $x->as_hex(); - - return (length($s) > 18) ? undef : $s; -} - -# Provides mul, adj, and shr factors for a specific -# (bit, time, hz) combination -sub muladj($$$) { - my($b, $t, $hz) = @_; - my $s = fmuls($b, $t, $hz); - my $m = fmul($s, $t, $hz); - my $a = fadj($s, $t, $hz); - return (bignum_hex($m), bignum_hex($a), $s); -} - -# Provides numerator, denominator values -sub numden($$) { - my($n, $d) = @_; - my $g = bgcd($n, $d); - return ($n/$g, $d/$g); -} - -# All values for a specific (time, hz) combo -sub conversions($$) { - my ($t, $hz) = @_; - my @val = (); - - # HZ_TO_xx - push(@val, muladj(32, $t, $hz)); - push(@val, numden($t, $hz)); - - # xx_TO_HZ - push(@val, muladj(32, $hz, $t)); - push(@val, numden($hz, $t)); - - return @val; -} - -sub compute_values($) { - my($hz) = @_; - my @val = (); - my $s, $m, $a, $g; - - if (!$has_bigint) { - die "$0: HZ == $hz not canned and ". - "Math::BigInt not available\n"; - } - - # MSEC conversions - push(@val, conversions(1000, $hz)); - - # USEC conversions - push(@val, conversions(1000000, $hz)); - - return @val; -} - -sub outputval($$) -{ - my($name, $val) = @_; - my $csuf; - - if (defined($val)) { - if ($name !~ /SHR/) { - $val = "U64_C($val)"; - } - printf "#define %-23s %s\n", $name.$csuf, $val.$csuf; - } -} - -sub output($@) -{ - my($hz, @val) = @_; - my $pfx, $bit, $suf, $s, $m, $a; - - print "/* Automatically generated by kernel/timeconst.pl */\n"; - print "/* Conversion constants for HZ == $hz */\n"; - print "\n"; - print "#ifndef KERNEL_TIMECONST_H\n"; - print "#define KERNEL_TIMECONST_H\n"; - print "\n"; - - print "#include <linux/param.h>\n"; - print "#include <linux/types.h>\n"; - - print "\n"; - print "#if HZ != $hz\n"; - print "#error \"kernel/timeconst.h has the wrong HZ value!\"\n"; - print "#endif\n"; - print "\n"; - - foreach $pfx ('HZ_TO_MSEC','MSEC_TO_HZ', - 'HZ_TO_USEC','USEC_TO_HZ') { - foreach $bit (32) { - foreach $suf ('MUL', 'ADJ', 'SHR') { - outputval("${pfx}_$suf$bit", shift(@val)); - } - } - foreach $suf ('NUM', 'DEN') { - outputval("${pfx}_$suf", shift(@val)); - } - } - - print "\n"; - print "#endif /* KERNEL_TIMECONST_H */\n"; -} - -# Pretty-print Perl values -sub perlvals(@) { - my $v; - my @l = (); - - foreach $v (@_) { - if (!defined($v)) { - push(@l, 'undef'); - } elsif ($v =~ /^0x/) { - push(@l, "\'".$v."\'"); - } else { - push(@l, $v.''); - } - } - return join(',', @l); -} - -($hz) = @ARGV; - -# Use this to generate the %canned_values structure -if ($hz eq '--can') { - shift(@ARGV); - @hzlist = sort {$a <=> $b} (@ARGV); - - print "# Precomputed values for systems without Math::BigInt\n"; - print "# Generated by:\n"; - print "# timeconst.pl --can ", join(' ', @hzlist), "\n"; - print "\%canned_values = (\n"; - my $pf = "\t"; - foreach $hz (@hzlist) { - my @values = compute_values($hz); - print "$pf$hz => [\n"; - while (scalar(@values)) { - my $bit; - foreach $bit (32) { - my $m = shift(@values); - my $a = shift(@values); - my $s = shift(@values); - print "\t\t", perlvals($m,$a,$s), ",\n"; - } - my $n = shift(@values); - my $d = shift(@values); - print "\t\t", perlvals($n,$d), ",\n"; - } - print "\t]"; - $pf = ', '; - } - print "\n);\n"; -} else { - $hz += 0; # Force to number - if ($hz < 1) { - die "Usage: $0 HZ\n"; - } - - @val = @{$canned_values{$hz}}; - if (!defined(@val)) { - @val = compute_values($hz); - } - output($hz, @val); -} -exit 0; diff -ruN linux-2.6.30/kernel/timeconst.sh linux-2.6.30.new/kernel/timeconst.sh --- linux-2.6.30/kernel/timeconst.sh 1969-12-31 18:00:00.000000000 -0600 +++ linux-2.6.30.new/kernel/timeconst.sh 2009-06-22 14:29:16.000000000 -0500 @@ -0,0 +1,148 @@ +#!/bin/sh + +if [ $# -ne 2 ] +then + echo "Usage: timeconst.sh HZ FILENAME" + echo + echo "Generate a header file with constants for coverting between" + echo "decimal HZ timer ticks and milisecond or microsecond delays." + echo + exit 1 +fi + +HZ=$1 +shift +FILENAME=$1 + +# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math. + +if [ $((1 << 32)) -lt 0 ] +then + echo "timeconst.sh needs a shell with 64 bit math, such as bash," + echo "busybox ash, or dash running on a 64 bit host." + exit 1 +fi + +# If this script exits for any reason before this trap is removed, +# delete the output file so a partial file won't confuse the build. + +trap "rm $FILENAME" EXIT + +# Output start of header file + +cat > $FILENAME << EOF || exit 1 +/* Automatically generated by kernel/timeconst.sh */ +/* Conversion constants for HZ == $HZ */ + +#ifndef __KERNEL_TIMECONST_H +#define __KERNEL_TIMECONST_H + +#include <linux/param.h> +#include <linux/types.h> + +#if HZ != $HZ +#error "kernel/timeconst.h has the wrong HZ value!" +#endif + +EOF + +# For both Milliseconds and Microseconds + +cat << EOF | +MSEC 1000 +USEC 1000000 +EOF +while read NAME PERIOD +do + # Find greatest common denominator (using Euclid's algorithm) + + A=$HZ + B=$PERIOD + + while [ $B -ne 0 ] + do + C=$(( $A % $B )) + A=$B + B=$C + done + + GCD=$A + + # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ) + + for DIRECTION in 0 1 + do + if [ $DIRECTION -eq 0 ] + then + CONVERT="HZ_TO_${NAME}" + FROM=$HZ + TO=$PERIOD + else + CONVERT="${NAME}_TO_HZ" + FROM=$PERIOD + TO=$HZ + fi + + # Calculate 32 significant bits of MUL32 data. + + SHIFT=0 + while true + do + # This can't overflow 64 bit math. Pathological case + # (TO=1, FROM=1000000) uses around 32+20=52 bits. + + MUL32=$(( ( ( $TO << $SHIFT ) + $FROM - 1 ) / $FROM )) + + # Keep increasing $SHIFT until we've got 32 bits. + + [ $MUL32 -gt $(( 1 << 31 )) ] && break + SHIFT=$(( $SHIFT + 1 )) + done + MUL32=$( printf %x $MUL32 ) + + # ADJ32 is just (((FROM/GCD)-1)<<SHIFT)/(FROM/GCD) but this + # can overflow 64 bit math (examples, HZ=24 or HZ=122). + # Pathological case could use 32+20+20=72 bits. (And this is + # the pathological case because a larger $HZ results in a + # smaller $SHIFT, so even insane HZ>USEC cases should be ok.) + + # To get around this, we chop the bottom 32 bits off the + # calculation and then reassemble it to avoid overflow: + # 32+64=96, which is > 72. + + ADJ32=$(( $FROM / $GCD )) + if [ $SHIFT -gt 32 ] + then + UPPER=$(( ( $ADJ32 - 1 ) << ( $SHIFT - 32 ) )) + LOWER=$(( ( $UPPER % $ADJ32 ) << 32 )) + ADJ32=$(( ( ( $UPPER / $ADJ32 ) << 32 ) + ( $LOWER / $ADJ32 ))) + else + ADJ32=$(( ( ( $ADJ32 - 1 ) << $SHIFT) / $ADJ32 )) + fi + ADJ32=$( printf %x $ADJ32 ) + + NUM=$(( $TO / $GCD )) + DEN=$(( $FROM / $GCD )) + + # Output next chunk of header data to file + + ( + echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)" && + echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)" && + echo "#define ${CONVERT}_SHR32 $SHIFT" && + echo "#define ${CONVERT}_NUM U64_C($NUM)" && + echo "#define ${CONVERT}_DEN U64_C($DEN)" + ) >> $FILENAME || exit 1 + done +done + +( + echo + echo "#endif /* __KERNEL_TIMECHONST_H */" +) >> $FILENAME || exit 1 + +# Don't rm $FILENAME on exit anymore. + +trap "" EXIT + +exit 0 -- Latency is more important than throughput. It's that simple. - Linus Torvalds ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-08 9:19 ` [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley @ 2009-12-09 15:45 ` Michal Marek 2009-12-09 23:40 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: Michal Marek @ 2009-12-09 15:45 UTC (permalink / raw) To: Rob Landley; +Cc: linux-kernel, H. Peter Anvin, linux-kbuild [CC hpa who wrote the timeconst.pl script] On 8.12.2009 10:19, Rob Landley wrote: > From: Rob Landley <rob@landley.net> > > Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script > is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003. I tried the shell script with the precomputed values in timeconst.pl and it gave me different results than the perl version for 250 and 1000: with HZ=250: --- perl +++ bash @@ -8,14 +8,14 @@ #error "kernel/timeconst.h has the wrong HZ value!" #endif -#define HZ_TO_MSEC_MUL32 U64_C(0x80000000) +#define HZ_TO_MSEC_MUL32 U64_C(0x100000000) #define HZ_TO_MSEC_ADJ32 U64_C(0x0) -#define HZ_TO_MSEC_SHR32 29 +#define HZ_TO_MSEC_SHR32 30 #define HZ_TO_MSEC_NUM U64_C(4) #define HZ_TO_MSEC_DEN U64_C(1) -#define MSEC_TO_HZ_MUL32 U64_C(0x80000000) -#define MSEC_TO_HZ_ADJ32 U64_C(0x180000000) -#define MSEC_TO_HZ_SHR32 33 +#define MSEC_TO_HZ_MUL32 U64_C(0x100000000) +#define MSEC_TO_HZ_ADJ32 U64_C(0x300000000) +#define MSEC_TO_HZ_SHR32 34 #define MSEC_TO_HZ_NUM U64_C(1) #define MSEC_TO_HZ_DEN U64_C(4) #define HZ_TO_USEC_MUL32 U64_C(0xfa000000) and with HZ=1000: --- perl +++ bash @@ -8,14 +8,14 @@ #error "kernel/timeconst.h has the wrong HZ value!" #endif -#define HZ_TO_MSEC_MUL32 U64_C(0x80000000) +#define HZ_TO_MSEC_MUL32 U64_C(0x100000000) #define HZ_TO_MSEC_ADJ32 U64_C(0x0) -#define HZ_TO_MSEC_SHR32 31 +#define HZ_TO_MSEC_SHR32 32 #define HZ_TO_MSEC_NUM U64_C(1) #define HZ_TO_MSEC_DEN U64_C(1) -#define MSEC_TO_HZ_MUL32 U64_C(0x80000000) +#define MSEC_TO_HZ_MUL32 U64_C(0x100000000) #define MSEC_TO_HZ_ADJ32 U64_C(0x0) -#define MSEC_TO_HZ_SHR32 31 +#define MSEC_TO_HZ_SHR32 32 #define MSEC_TO_HZ_NUM U64_C(1) #define MSEC_TO_HZ_DEN U64_C(1) #define HZ_TO_USEC_MUL32 U64_C(0xfa000000) $ bash --version GNU bash, version 4.0.33(1)-release (x86_64-suse-linux-gnu) ... You're trying to avoid the build dependency on Perl. What about adding a timeconst.h_shipped with the precomputed values from timeconst.pl: #if HZ == 24 #define ... ... #endif #if HZ == 32 ... #endif ... #ifndef HZ_TO_MSEC_MUL32 # error "Unknown HZ value, please update kernel/timeconst.h using kernel/timeconst.pl" #endif plus some makefile automagic to run the script iff the HZ value isn't precomputed. Then you would only need Perl for exotic HZ configurations. > > Signed-off-by: Rob Landley <rob@landley.net> > -- > > kernel/Makefile | 4 > kernel/timeconst.pl | 378 ------------------------------------------ > kernel/timeconst.sh | 91 ++++++++++ > 3 files changed, 93 insertions(+), 380 deletions(-) > > diff -ruN linux-2.6.30/kernel/Makefile linux-2.6.30.new/kernel/Makefile > --- linux-2.6.30/kernel/Makefile 2009-06-09 22:05:27.000000000 -0500 > +++ linux-2.6.30.new/kernel/Makefile 2009-06-22 14:29:16.000000000 -0500 > @@ -123,7 +123,7 @@ > $(obj)/time.o: $(obj)/timeconst.h > > quiet_cmd_timeconst = TIMEC $@ > - cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@ > + cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) $@ > targets += timeconst.h > -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE > +$(obj)/timeconst.h: $(src)/timeconst.sh FORCE > $(call if_changed,timeconst) > diff -ruN linux-2.6.30/kernel/timeconst.pl linux-2.6.30.new/kernel/timeconst.pl > --- linux-2.6.30/kernel/timeconst.pl 2009-06-09 22:05:27.000000000 -0500 > +++ linux-2.6.30.new/kernel/timeconst.pl 1969-12-31 18:00:00.000000000 -0600 > @@ -1,378 +0,0 @@ > -#!/usr/bin/perl > -# ----------------------------------------------------------------------- > -# > -# Copyright 2007-2008 rPath, Inc. - All Rights Reserved > -# > -# This file is part of the Linux kernel, and is made available under > -# the terms of the GNU General Public License version 2 or (at your > -# option) any later version; incorporated herein by reference. > -# > -# ----------------------------------------------------------------------- > -# > - > -# > -# Usage: timeconst.pl HZ > timeconst.h [snip] > diff -ruN linux-2.6.30/kernel/timeconst.sh linux-2.6.30.new/kernel/timeconst.sh > --- linux-2.6.30/kernel/timeconst.sh 1969-12-31 18:00:00.000000000 -0600 > +++ linux-2.6.30.new/kernel/timeconst.sh 2009-06-22 14:29:16.000000000 -0500 > @@ -0,0 +1,148 @@ > +#!/bin/sh > + > +if [ $# -ne 2 ] > +then > + echo "Usage: timeconst.sh HZ FILENAME" > + echo > + echo "Generate a header file with constants for coverting between" > + echo "decimal HZ timer ticks and milisecond or microsecond delays." > + echo > + exit 1 > +fi > + > +HZ=$1 > +shift > +FILENAME=$1 > + > +# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math. > + > +if [ $((1 << 32)) -lt 0 ] > +then > + echo "timeconst.sh needs a shell with 64 bit math, such as bash," > + echo "busybox ash, or dash running on a 64 bit host." > + exit 1 > +fi > + > +# If this script exits for any reason before this trap is removed, > +# delete the output file so a partial file won't confuse the build. > + > +trap "rm $FILENAME" EXIT > + > +# Output start of header file > + > +cat > $FILENAME << EOF || exit 1 > +/* Automatically generated by kernel/timeconst.sh */ > +/* Conversion constants for HZ == $HZ */ > + > +#ifndef __KERNEL_TIMECONST_H > +#define __KERNEL_TIMECONST_H > + > +#include <linux/param.h> > +#include <linux/types.h> > + > +#if HZ != $HZ > +#error "kernel/timeconst.h has the wrong HZ value!" > +#endif > + > +EOF > + > +# For both Milliseconds and Microseconds > + > +cat << EOF | > +MSEC 1000 > +USEC 1000000 > +EOF > +while read NAME PERIOD > +do > + # Find greatest common denominator (using Euclid's algorithm) > + > + A=$HZ > + B=$PERIOD > + > + while [ $B -ne 0 ] > + do > + C=$(( $A % $B )) > + A=$B > + B=$C > + done > + > + GCD=$A > + > + # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ) > + > + for DIRECTION in 0 1 > + do > + if [ $DIRECTION -eq 0 ] > + then > + CONVERT="HZ_TO_${NAME}" > + FROM=$HZ > + TO=$PERIOD > + else > + CONVERT="${NAME}_TO_HZ" > + FROM=$PERIOD > + TO=$HZ > + fi > + > + # Calculate 32 significant bits of MUL32 data. > + > + SHIFT=0 > + while true > + do > + # This can't overflow 64 bit math. Pathological case > + # (TO=1, FROM=1000000) uses around 32+20=52 bits. > + > + MUL32=$(( ( ( $TO << $SHIFT ) + $FROM - 1 ) / $FROM )) > + > + # Keep increasing $SHIFT until we've got 32 bits. > + > + [ $MUL32 -gt $(( 1 << 31 )) ] && break > + SHIFT=$(( $SHIFT + 1 )) > + done > + MUL32=$( printf %x $MUL32 ) > + > + # ADJ32 is just (((FROM/GCD)-1)<<SHIFT)/(FROM/GCD) but this > + # can overflow 64 bit math (examples, HZ=24 or HZ=122). > + # Pathological case could use 32+20+20=72 bits. (And this is > + # the pathological case because a larger $HZ results in a > + # smaller $SHIFT, so even insane HZ>USEC cases should be ok.) > + > + # To get around this, we chop the bottom 32 bits off the > + # calculation and then reassemble it to avoid overflow: > + # 32+64=96, which is > 72. > + > + ADJ32=$(( $FROM / $GCD )) > + if [ $SHIFT -gt 32 ] > + then > + UPPER=$(( ( $ADJ32 - 1 ) << ( $SHIFT - 32 ) )) > + LOWER=$(( ( $UPPER % $ADJ32 ) << 32 )) > + ADJ32=$(( ( ( $UPPER / $ADJ32 ) << 32 ) + ( $LOWER / $ADJ32 ))) > + else > + ADJ32=$(( ( ( $ADJ32 - 1 ) << $SHIFT) / $ADJ32 )) > + fi > + ADJ32=$( printf %x $ADJ32 ) > + > + NUM=$(( $TO / $GCD )) > + DEN=$(( $FROM / $GCD )) > + > + # Output next chunk of header data to file > + > + ( > + echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)" && > + echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)" && > + echo "#define ${CONVERT}_SHR32 $SHIFT" && > + echo "#define ${CONVERT}_NUM U64_C($NUM)" && > + echo "#define ${CONVERT}_DEN U64_C($DEN)" > + ) >> $FILENAME || exit 1 > + done > +done > + > +( > + echo > + echo "#endif /* __KERNEL_TIMECHONST_H */" ^ Should be "__KERNEL_TIMECONST_H". > +) >> $FILENAME || exit 1 > + > +# Don't rm $FILENAME on exit anymore. > + > +trap "" EXIT > + > +exit 0 > Michal ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-09 15:45 ` Michal Marek @ 2009-12-09 23:40 ` Rob Landley 2009-12-09 23:45 ` H. Peter Anvin 2009-12-10 13:52 ` Michal Marek 0 siblings, 2 replies; 135+ messages in thread From: Rob Landley @ 2009-12-09 23:40 UTC (permalink / raw) To: Michal Marek; +Cc: linux-kernel, H. Peter Anvin, linux-kbuild On Wednesday 09 December 2009 09:45:37 Michal Marek wrote: > [CC hpa who wrote the timeconst.pl script] > > On 8.12.2009 10:19, Rob Landley wrote: > > From: Rob Landley <rob@landley.net> > > > > Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell > > script is much simpler, about 1/4 the size, and runs on Red Hat 9 from > > 2003. > > I tried the shell script with the precomputed values in timeconst.pl and > it gave me different results than the perl version for 250 and 1000: You're right. They're functionally equivalent (due to the relationship between MUL32 and SHR32), which is why this code has worked for me and other people for a year now. The difference is the possibility of an integer overflow if you try to convert a time period of "0xffffffff". Trivial fix: --- a/sources/patches/linux-2.6.25-rc1-noperl.patch Tue Dec 08 20:22:53 2009 -0600 +++ b/sources/patches/linux-2.6.25-rc1-noperl.patch Wed Dec 09 17:28:26 2009 -0600 @@ -507,7 +507,7 @@ + + # Keep increasing $SHIFT until we've got 32 bits. + -+ [ $MUL32 -gt $(( 1 << 31 )) ] && break ++ [ $MUL32 -ge $(( 1 << 31 )) ] && break + SHIFT=$(( $SHIFT + 1 )) + done + MUL32=$( printf %x $MUL32 ) As long as MUL32 fits in 32 bits than you can multiply it by another 32 bit number without overflow, and that's probably all the kernel's enforcing. > #define HZ_TO_MSEC_NUM U64_C(4) > #define HZ_TO_MSEC_DEN U64_C(1) > -#define MSEC_TO_HZ_MUL32 U64_C(0x80000000) > -#define MSEC_TO_HZ_ADJ32 U64_C(0x180000000) > -#define MSEC_TO_HZ_SHR32 33 > +#define MSEC_TO_HZ_MUL32 U64_C(0x100000000) > +#define MSEC_TO_HZ_ADJ32 U64_C(0x300000000) > +#define MSEC_TO_HZ_SHR32 34 > #define MSEC_TO_HZ_NUM U64_C(1) > #define MSEC_TO_HZ_DEN U64_C(4) > #define HZ_TO_USEC_MUL32 U64_C(0xfa000000) The same. > and with HZ=1000: > --- perl > +++ bash > @@ -8,14 +8,14 @@ > #error "kernel/timeconst.h has the wrong HZ value!" > #endif > > -#define HZ_TO_MSEC_MUL32 U64_C(0x80000000) > +#define HZ_TO_MSEC_MUL32 U64_C(0x100000000) > #define HZ_TO_MSEC_ADJ32 U64_C(0x0) > -#define HZ_TO_MSEC_SHR32 31 > +#define HZ_TO_MSEC_SHR32 32 And again. > #define HZ_TO_MSEC_NUM U64_C(1) > #define HZ_TO_MSEC_DEN U64_C(1) > -#define MSEC_TO_HZ_MUL32 U64_C(0x80000000) > +#define MSEC_TO_HZ_MUL32 U64_C(0x100000000) > #define MSEC_TO_HZ_ADJ32 U64_C(0x0) > -#define MSEC_TO_HZ_SHR32 31 > +#define MSEC_TO_HZ_SHR32 32 Same thing. > You're trying to avoid the build dependency on Perl. What about adding a > timeconst.h_shipped with the precomputed values from timeconst.pl: Been there, done that. My first patch (way back for 2.6.25) took that approach: http://landley.net/hg/hgwebdir.cgi/firmware/file/a791ca629d9c/sources/patches/linux-2.6.25- rc1-noperl.patch But it turns out various non-x86 targets (such as ARM OMAP) allow HZ to be specified by an entry field in the config file, into which the user can type a range of numbers. See this post from last year for details: http://lists.impactlinux.com/pipermail/firmware-impactlinux.com/2008- December/000022.html This is why reducing the perl version to just the precomputed constants wouldn't work either. (They're there so that you only need to install a random cpan library when surprised by a build break on non-x86 machines.) > > + echo > > + echo "#endif /* __KERNEL_TIMECHONST_H */" > > ^ > > Should be "__KERNEL_TIMECONST_H". Yeah, Joe Perches pointed that out to me off-list. It's just a comment so I didn't resubmit the patch for that, but I've fixed my local version already. Rob -- Latency is more important than throughput. It's that simple. - Linus Torvalds ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-09 23:40 ` Rob Landley @ 2009-12-09 23:45 ` H. Peter Anvin 2009-12-10 1:50 ` Rob Landley 2009-12-10 13:52 ` Michal Marek 1 sibling, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-12-09 23:45 UTC (permalink / raw) To: Rob Landley; +Cc: Michal Marek, linux-kernel, linux-kbuild On 12/09/2009 03:40 PM, Rob Landley wrote: > > This is why reducing the perl version to just the precomputed constants > wouldn't work either. (They're there so that you only need to install a > random cpan library when surprised by a build break on non-x86 machines.) > It's not "a random CPAN library" - it's a core module. -hpa ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-09 23:45 ` H. Peter Anvin @ 2009-12-10 1:50 ` Rob Landley 2009-12-10 1:53 ` H. Peter Anvin 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-12-10 1:50 UTC (permalink / raw) To: H. Peter Anvin; +Cc: Michal Marek, linux-kernel, linux-kbuild On Wednesday 09 December 2009 17:45:21 H. Peter Anvin wrote: > On 12/09/2009 03:40 PM, Rob Landley wrote: > > This is why reducing the perl version to just the precomputed constants > > wouldn't work either. (They're there so that you only need to install a > > random cpan library when surprised by a build break on non-x86 machines.) > > It's not "a random CPAN library" - it's a core module. Then why cache large quantities of its output and test for its existence? Rob -- Latency is more important than throughput. It's that simple. - Linus Torvalds ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-10 1:50 ` Rob Landley @ 2009-12-10 1:53 ` H. Peter Anvin 0 siblings, 0 replies; 135+ messages in thread From: H. Peter Anvin @ 2009-12-10 1:53 UTC (permalink / raw) To: Rob Landley; +Cc: Michal Marek, linux-kernel, linux-kbuild On 12/09/2009 05:50 PM, Rob Landley wrote: > On Wednesday 09 December 2009 17:45:21 H. Peter Anvin wrote: >> On 12/09/2009 03:40 PM, Rob Landley wrote: >>> This is why reducing the perl version to just the precomputed constants >>> wouldn't work either. (They're there so that you only need to install a >>> random cpan library when surprised by a build break on non-x86 machines.) >> >> It's not "a random CPAN library" - it's a core module. > > Then why cache large quantities of its output and test for its existence? > To support archaic Perl versions and Perl binaries without libraries. -hpa ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-09 23:40 ` Rob Landley 2009-12-09 23:45 ` H. Peter Anvin @ 2009-12-10 13:52 ` Michal Marek 2009-12-10 23:16 ` Rob Landley 1 sibling, 1 reply; 135+ messages in thread From: Michal Marek @ 2009-12-10 13:52 UTC (permalink / raw) To: Rob Landley; +Cc: linux-kernel, H. Peter Anvin, linux-kbuild On 10.12.2009 00:40, Rob Landley wrote: > On Wednesday 09 December 2009 09:45:37 Michal Marek wrote: >> You're trying to avoid the build dependency on Perl. What about adding a >> timeconst.h_shipped with the precomputed values from timeconst.pl: > > Been there, done that. My first patch (way back for 2.6.25) took that > approach: > > http://landley.net/hg/hgwebdir.cgi/firmware/file/a791ca629d9c/sources/patches/linux-2.6.25- > rc1-noperl.patch > > But it turns out various non-x86 targets (such as ARM OMAP) allow HZ to be > specified by an entry field in the config file, into which the user can type a > range of numbers. See this post from last year for details: > > http://lists.impactlinux.com/pipermail/firmware-impactlinux.com/2008- > December/000022.html > > This is why reducing the perl version to just the precomputed constants > wouldn't work either. (They're there so that you only need to install a > random cpan library when surprised by a build break on non-x86 machines.) That's why I wrote >> plus some makefile automagic to run the script iff the HZ value isn't >> precomputed. Then you would only need Perl for exotic HZ configurations. E.g. make it ... #elif HZ == 1200 ... #else #include "timeconst_custom.h" #endif and the makefile would run timeconst.pl to generate timeconst_custom.h iff HZ is set to something arbitrary. I don't have a patch for that, but I don't see a fundamental problem with such approach. Michal ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-10 13:52 ` Michal Marek @ 2009-12-10 23:16 ` Rob Landley 2009-12-11 15:31 ` Michal Marek 0 siblings, 1 reply; 135+ messages in thread From: Rob Landley @ 2009-12-10 23:16 UTC (permalink / raw) To: Michal Marek; +Cc: linux-kernel, H. Peter Anvin, linux-kbuild On Thursday 10 December 2009 07:52:44 Michal Marek wrote: > On 10.12.2009 00:40, Rob Landley wrote: > > On Wednesday 09 December 2009 09:45:37 Michal Marek wrote: > >> You're trying to avoid the build dependency on Perl. What about adding a > >> timeconst.h_shipped with the precomputed values from timeconst.pl: > > > > Been there, done that. My first patch (way back for 2.6.25) took that > > approach: > > > > http://landley.net/hg/hgwebdir.cgi/firmware/file/a791ca629d9c/sources/pat > >ches/linux-2.6.25- rc1-noperl.patch > > > > But it turns out various non-x86 targets (such as ARM OMAP) allow HZ to > > be specified by an entry field in the config file, into which the user > > can type a range of numbers. See this post from last year for details: > > > > http://lists.impactlinux.com/pipermail/firmware-impactlinux.com/2008- > > December/000022.html > > > > This is why reducing the perl version to just the precomputed constants > > wouldn't work either. (They're there so that you only need to install a > > random cpan library when surprised by a build break on non-x86 machines.) > > That's why I wrote > > >> plus some makefile automagic to run the script iff the HZ value isn't > >> precomputed. Then you would only need Perl for exotic HZ configurations. > > E.g. make it > ... > #elif HZ == 1200 > ... > #else > #include "timeconst_custom.h" > #endif > > and the makefile would run timeconst.pl to generate timeconst_custom.h > iff HZ is set to something arbitrary. I don't have a patch for that, but > I don't see a fundamental problem with such approach. I looked at that when I was attempting to update (rather than replace) my original patch. There's a reason I didn't go there. You're suggesting there be two code paths, one maintained by hand and the other just about never tested against bit-rot. You want to add logic to the makefile so it knows when to run the perl script and when not to, meaning the makefile needs its own list of the cached HZ values, which needs to be kept in sync with the header of cached values, and thus the same information has to live in two places. Meanwhile the shell script exists, works, and takes a fraction of a second to run. Using it every build provides a single consistent code path which works the same for everybody. I'm not seeing any advantage in cacheing values that are easy to calculate at compile time. It's just as easy to completely eliminate the perl dependency as to merely mitigate it. > Michal Rob -- Latency is more important than throughput. It's that simple. - Linus Torvalds ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-10 23:16 ` Rob Landley @ 2009-12-11 15:31 ` Michal Marek 2009-12-11 19:13 ` H. Peter Anvin 0 siblings, 1 reply; 135+ messages in thread From: Michal Marek @ 2009-12-11 15:31 UTC (permalink / raw) To: Rob Landley; +Cc: linux-kernel, H. Peter Anvin, linux-kbuild Rob Landley wrote: > On Thursday 10 December 2009 07:52:44 Michal Marek wrote: >> and the makefile would run timeconst.pl to generate timeconst_custom.h >> iff HZ is set to something arbitrary. I don't have a patch for that, but >> I don't see a fundamental problem with such approach. > > I looked at that when I was attempting to update (rather than replace) my > original patch. There's a reason I didn't go there. You're suggesting there > be two code paths, one maintained by hand and the other just about never > tested against bit-rot. OK, that's valid point, indeed. Peter, would you ack Rob's patch with the oneline fix added (http://lkml.org/lkml/2009/12/8/94 plus http://lkml.org/lkml/2009/12/9/435)? Michal ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-11 15:31 ` Michal Marek @ 2009-12-11 19:13 ` H. Peter Anvin 2009-12-12 0:49 ` Rob Landley 0 siblings, 1 reply; 135+ messages in thread From: H. Peter Anvin @ 2009-12-11 19:13 UTC (permalink / raw) To: Michal Marek; +Cc: Rob Landley, linux-kernel, linux-kbuild On 12/11/2009 07:31 AM, Michal Marek wrote: > > OK, that's valid point, indeed. Peter, would you ack Rob's patch with > the oneline fix added (http://lkml.org/lkml/2009/12/8/94 plus > http://lkml.org/lkml/2009/12/9/435)? > I strongly dislike his patch, as he open-codes specific multiprecision arithmetic. This makes it hard for other people to maintain, and makes it prone to errors -- as evidenced by the fact that it didn't even replicate the known-good results. I have made my position clear on this and other patches several times before: I consider it a fool's errand, and a result of a completely pointless crusade to make a particular science fair-type project a wee bit easier. We have already seen real damage caused by it, since people have used awk instead, and have gotten bitten by incompatibilities between awk implementations. As such, no, I will not ack this patch, and will consider myself released of any obligation to maintain the code if this goes in anyway. I would consider acking a C program which does proper multiprecision arithmetic, but I'm also not going to spend my time on it. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. ^ permalink raw reply [flat|nested] 135+ messages in thread
* Re: [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh 2009-12-11 19:13 ` H. Peter Anvin @ 2009-12-12 0:49 ` Rob Landley 0 siblings, 0 replies; 135+ messages in thread From: Rob Landley @ 2009-12-12 0:49 UTC (permalink / raw) To: H. Peter Anvin; +Cc: Michal Marek, linux-kernel, linux-kbuild On Friday 11 December 2009 13:13:39 H. Peter Anvin wrote: > On 12/11/2009 07:31 AM, Michal Marek wrote: > > OK, that's valid point, indeed. Peter, would you ack Rob's patch with > > the oneline fix added (http://lkml.org/lkml/2009/12/8/94 plus > > http://lkml.org/lkml/2009/12/9/435)? > > I strongly dislike his patch, as he open-codes specific multiprecision > arithmetic. This makes it hard for other people to maintain, and makes > it prone to errors It's always easier to understand code that you wrote, doubly so with perl. The perl code this is replacing has comments, function names like fmul() and fmuls() with no way to distinguish what they do except by reading their implementation, and half of it is devoting to cacheing the output of the other half for reasons which are never explained in the code. The few comments the perl code does have are sometimes actually misleading. For example, bignum_hex() has the comment "generate a hex value if the result fits in 64 bits; otherwise skip". What would skpping mean in this context? Would it silently emit a broken header, or would it break the build? (My first guess is that "undef" without an expr would be a build break, but maybe this is perl's way of saying "None"? I do know that whatever happened the result wouldn't be a usable kernel, so "skip" is disingenous.) That said, I read through and managed to understand the perl code. But it took me more than a day, and I was motivated by the desire to replace it. > -- as evidenced by the fact that it didn't even > replicate the known-good results. Actually I had noticed the difference back at the time I wrote the script, but the value plus shift were functionally equivalent, and at the time (a year ago) I vaguely recall working through the math (and looking at the kernel code using the output) convinced me that that integer overflow couldn't actually happen for real inputs. But I don't remember why (perhaps the value to be converted is range checked somewhere so ffffffff couldn't actually make it to this code, I don't remember) and thus my version would give more precision to the result (which seemed to be the whole point of the exercise). No point in trying to reconstruct my reasoning from a year ago when it's a one line change to make it non-controversial, and the "value always fits in 32 bits so you can safely multiply by an unsigned int due to LP64" is easier to explain/document anyway. (On the other end of things, I do remember that the admittedly insane HZ of ffffffff would have a GCD of 5 with 1000 and thus not be saved as that. That came up because I was worried an ffffffff value times ffffffff might _round_ up to overflow when adj32 was added. I did attempt to document some of the edge cases in the shell script comments...) > I have made my position clear on this and other patches several times > before: I consider it a fool's errand, I.E. your fundamental objection is to removing perl form the build. You are the one who added this perl to the build system. You're also the one who added perl to all the other projects you maintain (klibc and syslinux I noticed) at approximately the same time, after years of those not requiring perl. I don't know why you suddenly decided perl needed to be everywhere, but the Linux kernel is not the only project you simultaneously applied this philosophy to. I strongly suspect the other objections you've raised in the past two years are a proxy for the objection to removing perl. However, your central objection to removing perl hasn't convinced people like Alan Cox, who called Perl "a bad candidate for our toolchain dependencies" here: http://lkml.indiana.edu/hypermail/linux/kernel/0901.1/02108.html Thus you raise other issues, which seem to be proxies for that fundamental objection. I've tried to address the ones that look like they were actually intended to be addressed, but I note that your current objection is to the something like the seventh submission of this patch, and I believe this is the first time you've raised this particular issue. I believe I first submitted an ancestor of this patch on February 15, 2008. Just this year I've submitted previous versions of this patch (more or less in its current form) on January 2, 2009 (with a resubmit on January 3 in response to feedback), on June 22, and on September 18. And again now. The patch hasn't changed much, your objections to it have. I don't even remember you previously complaining about the output differing. (At a guess, you never actually tried my patch?) But it has been a while, I could easily be forgetting details by now. I'm posting these here because they're useful to other people, not because I expect you to ever stop objecting to them no matter what changes I make. You've made it clear you object to the _goal_ of the patch. The implementation seems to be a side issue. > and a result of a completely > pointless crusade to make a particular science fair-type project a wee > bit easier. Embedded developers' desire to restrict their build environment to something controllable seems to strike you as weird, yes. Would it help to explain that it's the same general theory behind Linux From Scratch making a temporary system, chrooting into it, and then building in there in Chapter 5? Leaking random crap from the host is bad, often in subtle ways you don't immediately catch. Some programs (perl, libtool, syscfg) are champions at leaking context, and to be avoided if at all possible. (I suppose you could always suggest setting up a different build environment for the kernel as for userspace, the way Red Hat used to do with kgcc. Doesn't strike me as a good idea, though.) > We have already seen real damage caused by it, since people > have used awk instead, and have gotten bitten by incompatibilities > between awk implementations. I'm not using AWK, and I'm sticking with POSIX shell constructs that were there in susv3, let alone susv4. Since I wasn't the one proposing any patches extensively using awk (I need the man page open to make awk do anything more complicated than '{print $2}'), and this patch isn't using awk at all, I guess you're once again objecting to the removal of perl in general, not to the patch in front of you. And in doing so, you're providing evidence that I'm not the only one motivated to submit perl removal patches, which might undermine your "particular science fair project" argument a bit. I am motivated to follow up on this. I've been maintaining perl removal patches for a couple years now because I personally need them, but I'm not the only one using these patches, and I've been thanked for tackling this problem by a dozen embedded developers. I wasn't the one who first noticed the perl dependency when it went in back in .25, David Anders notified me of the issue before I'd tested the -rc that introduced it. The most recent guy to say, and I quote, "Thanks for eliminating the perl dependency." to me in private email was Joe Perches (on Tuesday). Embedded developers tend to do stuff off-list. This is because every new kernel breaks random stuff on non-x86 targets. For example, I finally got powerpc's pmac_zilog fixed a few days ago (allowing me to use the serial console on qemu's mac99 target), after a thread stretching back two months (it broke after 2.6.28). The "works for me" fix is here: http://lists.ozlabs.org/pipermail/linuxppc-dev/2009-December/078700.html The most recent arm breakage turned out to be simple (a new CONFIG_MMU symbol showed up between 2.6.31 and 2.6.32 and needed to be switched on), but diagnosing "it doesn't boot, no console messages" is always a pain. The most recent mips breakage was fixed by commit d71789b6fa37c21ce5eb588d279f57904a62e7e2 which I asked to be backported to 2.6.31 in the stable series and Greg KH acknowledged on November 5th. (I'd link to my submission to stable@kernel.org but there's no entry for that in vger lists and googling doesn't seem to come up with a public archive.) That's just in the past month or so. Trying to git bisect each of those problems found huge ranges of "it doesn't even compile" in the repository, because that's the _norm_ for non-x86 targets during development. Most embedded developers don't want to deal with this, so they stay a year or three behind the bleeding edge and only upgrade after crazy people like me have gotten it working. Since even the ones who _don't_ start their messages by apologizing about their English know they'll be flamed or ignored if they raise an issue about an 18 month old kernel here, they develop a fear of posting on linux-kernel at all because it's an "unfriendly" place. (I try to talk them out of it, but it's a cultural thing at this point. The new kernel embedded list hasn't really helped more than 10% of this problem.) I regularly tackle these issues anyway, am generally friendly to newbies on the lists I follow, and apparently have no shame, so people sometimes forward issues to me and hope I'll push them upstream so they don't have to. (And then go away again once they've managed to upgrade to whatever snapshot they'll be using for the next 3 years, leaving me to follow up.) *shrug* I'm used to it. You feed stray issues patches and they wind up adopting you... > As such, no, I will not ack this patch, and will consider myself > released of any obligation to maintain the code if this goes in anyway. I've been maintaining this patch series for 2 years already. I would be happy to maintain my changed version in future if it gets in. I expected that. > I would consider acking a C program which does proper multiprecision > arithmetic, but I'm also not going to spend my time on it. Hmmm, "would consider acking" (not "would ack", not even "could ack"), with the additional caveat that the multiprecision arithmetic must be an unspecified "proper", whatever that means. The "proxy objections" thesis stated above would label this an obvious delaying tactic, but if I promise to call your bluff, then the current patch gets dropped on the floor for another development cycle... Well played, sir. I must plead the 5th for now. > -hpa Rob -- Latency is more important than throughput. It's that simple. - Linus Torvalds ^ permalink raw reply [flat|nested] 135+ messages in thread
end of thread, other threads:[~2009-12-12 0:49 UTC | newest] Thread overview: 135+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-01-02 8:07 PATCH [0/3]: Simplify the kernel build by removing perl Rob Landley 2009-01-02 8:13 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley 2009-01-02 9:04 ` Sam Ravnborg 2009-01-02 12:00 ` Rob Landley 2009-01-02 19:33 ` H. Peter Anvin 2009-01-04 1:32 ` Rob Landley 2009-01-04 1:35 ` H. Peter Anvin 2009-01-04 12:07 ` Alan Cox 2009-01-04 18:36 ` H. Peter Anvin 2009-01-04 19:03 ` Rob Landley 2009-01-04 20:39 ` H. Peter Anvin 2009-01-05 0:59 ` Rob Landley 2009-01-03 6:28 ` Harvey Harrison 2009-01-03 12:28 ` Ingo Oeser 2009-01-04 1:36 ` Rob Landley 2009-01-04 5:07 ` Valdis.Kletnieks 2009-01-04 6:43 ` Rob Landley 2009-01-04 22:13 ` Jamie Lokier 2009-01-05 0:15 ` Bernd Petrovitsch 2009-01-05 2:23 ` Jamie Lokier 2009-01-05 10:46 ` Bernd Petrovitsch 2009-01-05 15:01 ` Jamie Lokier 2009-01-05 16:18 ` Bernd Petrovitsch 2009-01-06 0:06 ` Rob Landley 2009-01-05 21:07 ` Rob Landley 2009-01-05 4:50 ` Rob Landley 2009-01-05 12:29 ` Bernd Petrovitsch 2009-01-04 21:51 ` Alejandro Mery 2009-01-04 7:15 ` Michal Jaegermann 2009-01-05 0:41 ` Ray Lee 2009-01-05 5:08 ` Rob Landley 2009-01-02 8:14 ` [PATCH 2/3]: Remove perl from make headers_install Rob Landley 2009-01-02 9:09 ` Sam Ravnborg 2009-01-02 8:15 ` [PATCH 3/3]: Convert mkcapflags.pl to mkcapflags.sh Rob Landley 2009-01-02 9:12 ` Sam Ravnborg 2009-01-02 9:26 ` PATCH [0/3]: Simplify the kernel build by removing perl Arkadiusz Miskiewicz 2009-01-02 9:49 ` Christoph Hellwig 2009-01-02 10:16 ` Alejandro Mery 2009-01-02 10:30 ` Mark Miller 2009-01-02 11:18 ` Matt Keenan 2009-01-02 10:41 ` Måns Rullgård 2009-01-15 12:59 ` Pádraig Brady 2009-01-15 18:52 ` Jamie Lokier 2009-01-15 19:45 ` Måns Rullgård 2009-01-02 11:15 ` Rob Landley 2009-01-02 11:44 ` Sam Ravnborg 2009-01-02 12:56 ` Rob Landley 2009-01-02 14:04 ` Theodore Tso 2009-01-03 3:22 ` Jamie Lokier 2009-01-04 2:23 ` Rob Landley 2009-01-02 10:02 ` Mark Miller 2009-01-02 10:03 ` Mark Miller 2009-01-02 11:13 ` Rob Landley 2009-01-02 16:04 ` Matthieu CASTET 2009-01-03 19:46 ` Rob Landley 2009-01-03 20:10 ` Sam Ravnborg 2009-01-03 20:50 ` H. Peter Anvin 2009-01-04 1:47 ` Rob Landley 2009-01-04 1:45 ` Rob Landley 2009-01-04 8:09 ` Sam Ravnborg 2009-01-04 20:19 ` Rob Landley 2009-01-04 0:44 ` Robert Hancock 2009-01-04 1:39 ` David Brownell 2009-01-04 3:05 ` Rob Landley 2009-01-04 1:32 ` Rob Landley 2009-01-02 9:50 ` Paul Mundt 2009-01-02 10:32 ` Mark Miller 2009-01-02 10:57 ` Paul Mundt 2009-01-02 12:11 ` Mark Miller 2009-01-02 12:44 ` Rob Landley 2009-01-02 17:25 ` Wookey 2009-01-02 18:01 ` Sam Ravnborg 2009-01-02 19:27 ` H. Peter Anvin 2009-01-04 1:35 ` Rob Landley 2009-01-03 19:48 ` Rob Landley 2009-01-03 14:59 ` Wolfgang Denk 2009-01-03 22:54 ` Leon Woestenberg 2009-01-03 23:03 ` H. Peter Anvin 2009-01-04 0:37 ` Leon Woestenberg 2009-01-04 2:53 ` Rob Landley 2009-01-04 3:38 ` Markus Heidelberg 2009-01-04 4:57 ` Rob Landley 2009-01-04 2:06 ` Rob Landley 2009-01-04 2:14 ` H. Peter Anvin 2009-01-04 6:29 ` Rob Landley 2009-01-15 14:32 ` Pádraig Brady 2009-01-04 2:36 ` Jamie Lokier 2009-01-04 2:39 ` H. Peter Anvin 2009-01-04 2:43 ` H. Peter Anvin 2009-01-04 3:06 ` Paul Mundt 2009-01-04 10:23 ` Leon Woestenberg 2009-01-08 13:29 ` Mike Frysinger 2009-01-11 12:45 ` Bernd Petrovitsch 2009-01-12 3:36 ` Mark A. Miller 2009-01-12 5:11 ` H. Peter Anvin 2009-01-12 5:23 ` Mark A. Miller 2009-01-12 8:20 ` Paul Mundt 2009-01-12 9:18 ` Mark A. Miller 2009-01-12 9:41 ` Paul Mundt 2009-01-12 10:03 ` Mark A. Miller 2009-01-12 10:34 ` Paul Mundt 2009-01-12 17:56 ` Rob Landley 2009-01-12 18:04 ` Alan Cox 2009-01-12 8:27 ` Peter Korsgaard 2009-01-12 17:45 ` Rob Landley [not found] ` <31014a580901111928u586e2246uccf370ff941c8a01@mail.gmail.com> 2009-01-12 5:35 ` Sam Ravnborg 2009-01-12 5:50 ` Mark A. Miller 2009-01-12 10:18 ` Sam Ravnborg 2009-01-12 10:22 ` Mark A. Miller 2009-01-12 10:38 ` Paul Mundt 2009-01-14 2:51 ` Jamie Lokier 2009-01-16 6:11 ` Rob Landley 2009-01-16 14:54 ` Valdis.Kletnieks 2009-01-16 21:54 ` Rob Landley 2009-01-17 9:51 ` Jamie Lokier 2009-01-18 1:44 ` Rob Landley 2009-01-04 16:22 ` Vladimir Dronnikov 2009-01-04 1:24 ` PATCH [0/3]: Simplify the kernel build by removing perl v2 Rob Landley 2009-01-04 1:27 ` PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2) Rob Landley 2009-01-04 2:48 ` David Vrabel 2009-01-04 20:21 ` Rob Landley 2009-01-04 1:28 ` PATCH [2/3]: Remove perl from make headers_install Rob Landley 2009-01-04 1:29 ` PATCH [3/3]: Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh Rob Landley 2009-09-19 1:25 [PATCH 0/3] Perl removal patches for 2.6.31 Rob Landley 2009-09-19 1:33 ` [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley 2009-12-08 9:17 [PATCH 0/3] clean out Perl from build system Rob Landley 2009-12-08 9:19 ` [PATCH 1/3] Replace kernel/timeconst.pl with kernel/timeconst.sh Rob Landley 2009-12-09 15:45 ` Michal Marek 2009-12-09 23:40 ` Rob Landley 2009-12-09 23:45 ` H. Peter Anvin 2009-12-10 1:50 ` Rob Landley 2009-12-10 1:53 ` H. Peter Anvin 2009-12-10 13:52 ` Michal Marek 2009-12-10 23:16 ` Rob Landley 2009-12-11 15:31 ` Michal Marek 2009-12-11 19:13 ` H. Peter Anvin 2009-12-12 0:49 ` Rob Landley
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).