Kernel source tree splitting

Kernel source tree splitting

[rmoser]

Eh, Linus won't be happy making a bunch of tarballs.
I've made it less work if you read the message here...

The message mirrored at:

http://marc.theaimsgroup.com/?l=linux-kernel&m=105173077417526&w=2

Shows my pre-thought on this subject.  I thought a bit more,
and began to come up with a simple sketch to lead the
way in case anyone becomes interested.

First off, the kernel tarballs would be built by a script
that splits the source tree apart appropriately and tar's it
up.  How this is done is explained.

Second off, there's always a script to download that runs
wget and gets the source tree from which it was downloaded.
The whole thing.  As in, every tarball is downloaded and
untar'd for the user, assembling the full kernel source
tree (as it would be if you untar'd it now).

Now, I explained LOD's in that message above in small
detail.  But, for clarity, LOD's are files which explain
which pieces of source in the kernel tree belong to the
LOD; what gets added to the config; where their makefiles
are; what config options depend on other linux options;
and what groups these LOD's are in.

A command such as `make disttree` should read the LOD's,
split apart each linux option, tar 'em together, and
then compress the tar's.  Then Linus could just scp the
new directory of tar's and a script up.

As for download, the script that goes up can be
downloaded (duh), and then run (... why do I bother?).
Now this script would run in "dumb mode" (unless the
user tells it not to maybe?) and rip down the whole
tree, untar it, and rebuild the original source tree.
I think.  I'm not sure, I really haven't tried yet.
I'll tell you how it works after it's implimented, if
ever that happens.  This would likely require wget.

Of course there's always the ftp method.  Go download
the pieces you want, untar 'em, copy 'em to the same
directory, and the build system adjusts.  but newbies
and developers, for completely opposite reasons, will
want to use the script in dumb mode.

For experienced users, this will make configuration
somewhat easier, as the user can avoid being prompted
for irrelavent drivers.  This is just a concept idea,
not a fully thought-out idea.  What do you think?

--Bluefox Icy

Re: Kernel source tree splitting

[Randy.Dunlap]

Hi,

I'm probably misreading this...but,

Have you tried this yet?  Does it modify/customize all Kconfig
and Makefiles for the selected tree splits?

A few days ago, in one tree, I rm-ed arch/{all that I don't need}
and drivers/{all that I don't need}.
After that I couldn't run "make *config" because it wants all of
those files, even if I don't want them.

So there are many edits that needed to be done in lots of
Kconfig and Makefiles if one selectively pulls or omits certain
sub-directories.



On Wed, 30 Apr 2003 19:46:13 -0400 rmoser <mlmoser@comcast.net> wrote:

| Eh, Linus won't be happy making a bunch of tarballs.
| I've made it less work if you read the message here...
| 
| The message mirrored at:
| 
| http://marc.theaimsgroup.com/?l=linux-kernel&m=105173077417526&w=2
| 
| Shows my pre-thought on this subject.  I thought a bit more,
| and began to come up with a simple sketch to lead the
| way in case anyone becomes interested.
| 
| First off, the kernel tarballs would be built by a script
| that splits the source tree apart appropriately and tar's it
| up.  How this is done is explained.
| 
| Second off, there's always a script to download that runs
| wget and gets the source tree from which it was downloaded.
| The whole thing.  As in, every tarball is downloaded and
| untar'd for the user, assembling the full kernel source
| tree (as it would be if you untar'd it now).
| 
| Now, I explained LOD's in that message above in small
| detail.  But, for clarity, LOD's are files which explain
| which pieces of source in the kernel tree belong to the
| LOD; what gets added to the config; where their makefiles
| are; what config options depend on other linux options;
| and what groups these LOD's are in.
| 
| A command such as `make disttree` should read the LOD's,
| split apart each linux option, tar 'em together, and
| then compress the tar's.  Then Linus could just scp the
| new directory of tar's and a script up.
| 
| As for download, the script that goes up can be
| downloaded (duh), and then run (... why do I bother?).
| Now this script would run in "dumb mode" (unless the
| user tells it not to maybe?) and rip down the whole
| tree, untar it, and rebuild the original source tree.
| I think.  I'm not sure, I really haven't tried yet.
| I'll tell you how it works after it's implimented, if
| ever that happens.  This would likely require wget.
| 
| Of course there's always the ftp method.  Go download
| the pieces you want, untar 'em, copy 'em to the same
| directory, and the build system adjusts.  but newbies
| and developers, for completely opposite reasons, will
| want to use the script in dumb mode.
| 
| For experienced users, this will make configuration
| somewhat easier, as the user can avoid being prompted
| for irrelavent drivers.  This is just a concept idea,
| not a fully thought-out idea.  What do you think?
| 
| --Bluefox Icy

--
~Randy

Re: Kernel source tree splitting

[rmoser]

*********** REPLY SEPARATOR  ***********

On 4/30/2003 at 5:21 PM Randy.Dunlap wrote:

>Hi,
>
>I'm probably misreading this...but,
>
>Have you tried this yet?  Does it modify/customize all Kconfig
>and Makefiles for the selected tree splits?
>

I didn't try it.  It would require knowledge of all interdependencies
between modules (i.e. ide-scsi is part of ide.  ide-scsi depends
on scsi), also all source files that belong to each config option
would likely need to be understood by the persons working to
do this, and also the entire build system semantics would need
to be designed to work in pieces.  Assuming this is ever done.

It goes like this:

make *config reads kernel-tree/options/foo.lod
make *config gets to configuration baz in bar.lod
make *config sees baz needs foo
make *config lists baz.
make *config sees biz needs data
make *config refuses to show biz
make missing-depends shows a list of unselectable options and
-----------selecting one tells which kernel option is needed.
make bzImage reads through kernel-tree/options/ and finds which
-----------makefiles to call (current makes have these embedded)
make bzImage builds a kernel.
make modules reads through kernel-tree/options/ and finds which
-----------makefiles to call.
make modules builds a kernel.

>A few days ago, in one tree, I rm-ed arch/{all that I don't need}
>and drivers/{all that I don't need}.
>After that I couldn't run "make *config" because it wants all of
>those files, even if I don't want them.
>

That WILL break something.

>So there are many edits that needed to be done in lots of
>Kconfig and Makefiles if one selectively pulls or omits certain
>sub-directories.
>

The main Makefile will have to be redone.  So would the kconfig
things (make config/menuconfig/xconfig).

The extra plus to this is that other people can steal the build
system for other projects lol.

>
>

Re: Kernel source tree splitting

[Randy.Dunlap]

> On 4/30/2003 at 5:21 PM Randy.Dunlap wrote:
>
>>Hi,
>>
>>I'm probably misreading this...but,
>>
>>Have you tried this yet?  Does it modify/customize all Kconfig
>>and Makefiles for the selected tree splits?
>>
>
> I didn't try it.  It would require knowledge of all interdependencies
> between modules (i.e. ide-scsi is part of ide.  ide-scsi depends
> on scsi), also all source files that belong to each config option
> would likely need to be understood by the persons working to
> do this, and also the entire build system semantics would need
> to be designed to work in pieces.  Assuming this is ever done.
>
> It goes like this:
>
> make *config reads kernel-tree/options/foo.lod
> make *config gets to configuration baz in bar.lod
> make *config sees baz needs foo
> make *config lists baz.
> make *config sees biz needs data
> make *config refuses to show biz
> make missing-depends shows a list of unselectable options and
> -----------selecting one tells which kernel option is needed.
> make bzImage reads through kernel-tree/options/ and finds which
> -----------makefiles to call (current makes have these embedded)
> make bzImage builds a kernel.
> make modules reads through kernel-tree/options/ and finds which
> -----------makefiles to call.
> make modules builds a kernel.
>
>>A few days ago, in one tree, I rm-ed arch/{all that I don't need} and
>> drivers/{all that I don't need}.
>>After that I couldn't run "make *config" because it wants all of
>>those files, even if I don't want them.
>>
>
> That WILL break something.

or "That does break something"  (when I tried it).

>>So there are many edits that needed to be done in lots of
>>Kconfig and Makefiles if one selectively pulls or omits certain
>>sub-directories.
>>
>
> The main Makefile will have to be redone.  So would the kconfig
> things (make config/menuconfig/xconfig).
>
> The extra plus to this is that other people can steal the build
> system for other projects lol.

I seem to try for simple solutions when possible and feasible.

In this case, if I were doing it, I would try changing (e.g.) in
arch/i386/kernel/Kconfig, this line:
source "drivers/eisa/Kconfig"
to
optsource "drivers/eisa/Kconfig"
where optsource means that the file is optional -- if not found,
ignore it.  And then see what happens, how far it can go,
what the next problem is....

If this could be made to work, then entire subdirectories/subsystems
could be optional.

~Randy

Re: Kernel source tree splitting

[rmoser]

*********** REPLY SEPARATOR  ***********

On 4/30/2003 at 7:51 PM Randy.Dunlap wrote:

>> On 4/30/2003 at 5:21 PM Randy.Dunlap wrote:
>>
>>>Hi,
>>>
>>>I'm probably misreading this...but,
>>>
>>>Have you tried this yet?  Does it modify/customize all Kconfig
>>>and Makefiles for the selected tree splits?
>>>
>>
>> I didn't try it.  It would require knowledge of all interdependencies
>> between modules (i.e. ide-scsi is part of ide.  ide-scsi depends
>> on scsi), also all source files that belong to each config option
>> would likely need to be understood by the persons working to
>> do this, and also the entire build system semantics would need
>> to be designed to work in pieces.  Assuming this is ever done.
>>
>> It goes like this:
>>
>> make *config reads kernel-tree/options/foo.lod
>> make *config gets to configuration baz in bar.lod
>> make *config sees baz needs foo
>> make *config lists baz.
>> make *config sees biz needs data
>> make *config refuses to show biz
>> make missing-depends shows a list of unselectable options and
>> -----------selecting one tells which kernel option is needed.
>> make bzImage reads through kernel-tree/options/ and finds which
>> -----------makefiles to call (current makes have these embedded)
>> make bzImage builds a kernel.
>> make modules reads through kernel-tree/options/ and finds which
>> -----------makefiles to call.
>> make modules builds a kernel.
>>
>>>A few days ago, in one tree, I rm-ed arch/{all that I don't need} and
>>> drivers/{all that I don't need}.
>>>After that I couldn't run "make *config" because it wants all of
>>>those files, even if I don't want them.
>>>
>>
>> That WILL break something.
>
>or "That does break something"  (when I tried it).
>
>>>So there are many edits that needed to be done in lots of
>>>Kconfig and Makefiles if one selectively pulls or omits certain
>>>sub-directories.
>>>
>>
>> The main Makefile will have to be redone.  So would the kconfig
>> things (make config/menuconfig/xconfig).
>>
>> The extra plus to this is that other people can steal the build
>> system for other projects lol.
>
>I seem to try for simple solutions when possible and feasible.
>
>In this case, if I were doing it, I would try changing (e.g.) in
>arch/i386/kernel/Kconfig, this line:
>source "drivers/eisa/Kconfig"
>to
>optsource "drivers/eisa/Kconfig"
>where optsource means that the file is optional -- if not found,
>ignore it.  And then see what happens, how far it can go,
>what the next problem is....
>
>If this could be made to work, then entire subdirectories/subsystems
>could be optional.
>

Yeah but hat about things that came out of nowhere, i.e. a piece of
hardware compes with a linux driver source?

>~Randy
>
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: Kernel source tree splitting

[Larry McVoy]

It would be *really* cool if the Makefile dependencies actually worked.
It's a very little known fact but if you are in an RCS or SCCS (and BK
looks like SCCS to make) source tree and the files are not checked out,
you can just say

	make

and make will look for a makefile, if there isn't one but there is a 
SCCS/s.[Mm]akefile it will check it out, look at the dependencies and start
checking those out and keep doing it to satisfy the target.

It's a really pleasant way to work, the "make clobber" target "cleans"
all the source so it isn't checked out, the directory is nice and empty.
This makes it easy to see stuff you still need to check in or think about.
It's definitely an old timer way of working, I'm pretty sure that the
original Unix was done this way but just because it is old doesn't mean
it is bad.  Opinions differ on that :)

Here's a make in a cleaned BK source tree:

$ lf
Notes/  gnu/  ide/   port/  tmp/       utils/  win32/
SCCS/   gui/  mdbm/  t/     tomcrypt/  web/    zlib/
$ make p
get   SCCS/s.Makefile
Makefile 1.326: 684 lines
make CFLAGS="-g -O2 -Wall -Wno-parentheses -Wno-char-subscripts -Wno-format-y2k -Wstrict-prototypes " all
get   SCCS/s.abort.c
abort.c 1.11: 157 lines
get   SCCS/s.unix.h
unix.h 1.48: 79 lines
get   SCCS/s.bkd.h
bkd.h 1.68: 157 lines
get   SCCS/s.lib_tcp.h
lib_tcp.h 1.12: 21 lines
get   SCCS/s.logging.h
logging.h 1.10: 80 lines
get   mdbm/SCCS/s.common.h
mdbm/common.h 1.6: 111 lines
get   mdbm/SCCS/s.mdbm.h
mdbm/mdbm.h 1.11: 426 lines
get   mdbm/SCCS/s.tune.h
mdbm/tune.h 1.3: 66 lines
get   SCCS/s.mmap.h
mmap.h 1.5: 30 lines
get   SCCS/s.purify.h
purify.h 1.31: 114 lines
get   SCCS/s.range.h
range.h 1.18: 42 lines
get   SCCS/s.rcs.h
rcs.h 1.6: 60 lines
get   SCCS/s.liblines.h
liblines.h 1.5: 81 lines
get   SCCS/s.resolve.h
resolve.h 1.36: 178 lines
get   SCCS/s.sccs.h
sccs.h 1.497: 1171 lines
get   SCCS/s.system.h
system.h 1.55: 157 lines
get   tomcrypt/SCCS/s.mpi-config.h
tomcrypt/mpi-config.h 1.1: 87 lines
get   tomcrypt/SCCS/s.mpi.h
tomcrypt/mpi.h 1.1: 225 lines
get   tomcrypt/SCCS/s.mycrypt_cfg.h
tomcrypt/mycrypt_cfg.h 1.4: 174 lines
get   tomcrypt/SCCS/s.mpi-types.h
tomcrypt/mpi-types.h 1.1: 19 lines
get   tomcrypt/SCCS/s.mycrypt.h
tomcrypt/mycrypt.h 1.2: 913 lines
get   SCCS/s.zgets.h
zgets.h 1.5: 23 lines
get   zlib/SCCS/s.deflate.h
zlib/deflate.h 1.4: 318 lines
get   zlib/SCCS/s.infblock.h
zlib/infblock.h 1.4: 39 lines
get   zlib/SCCS/s.infcodes.h
zlib/infcodes.h 1.4: 27 lines
get   zlib/SCCS/s.inffast.h
zlib/inffast.h 1.4: 17 lines
get   zlib/SCCS/s.inffixed.h
zlib/inffixed.h 1.2: 151 lines
get   zlib/SCCS/s.inftrees.h
zlib/inftrees.h 1.4: 58 lines
get   zlib/SCCS/s.infutil.h
zlib/infutil.h 1.4: 98 lines
get   zlib/SCCS/s.trees.h
zlib/trees.h 1.2: 128 lines
get   zlib/SCCS/s.zconf.h
zlib/zconf.h 1.4: 279 lines
get   zlib/SCCS/s.zlib.h
zlib/zlib.h 1.4: 893 lines
get   zlib/SCCS/s.zutil.h
zlib/zutil.h 1.3: 220 lines
cc -g -O2 -Wall -Wno-parentheses -Wno-char-subscripts -Wno-format-y2k -Wstrict-prototypes  -Izlib -Imdbm -c abort.c -o abort.o
get   SCCS/s.adler32.c
adler32.c 1.15: 144 lines
cc -g -O2 -Wall -Wno-parentheses -Wno-char-subscripts -Wno-format-y2k -Wstrict-prototypes  -Izlib -Imdbm -c adler32.c -o adler32.o
get   SCCS/s.admin.c
admin.c 1.122: 453 lines
cc -g -O2 -Wall -Wno-parentheses -Wno-char-subscripts -Wno-format-y2k -Wstrict-prototypes  -Izlib -Imdbm -c admin.c -o admin.o
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: Kernel source tree splitting

[rmoser]

*********** REPLY SEPARATOR  ***********

On 4/30/2003 at 5:52 PM Larry McVoy wrote:

>It would be *really* cool if the Makefile dependencies actually worked.
>It's a very little known fact but if you are in an RCS or SCCS (and BK
>looks like SCCS to make) source tree and the files are not checked out,
>you can just say
>
>	make
>
>and make will look for a makefile, if there isn't one but there is a
>SCCS/s.[Mm]akefile it will check it out, look at the dependencies and start
>checking those out and keep doing it to satisfy the target.
>
>It's a really pleasant way to work, the "make clobber" target "cleans"
>all the source so it isn't checked out, the directory is nice and empty.
>This makes it easy to see stuff you still need to check in or think about.
>It's definitely an old timer way of working, I'm pretty sure that the
>original Unix was done this way but just because it is old doesn't mean
>it is bad.  Opinions differ on that :)
>
>Here's a make in a cleaned BK source tree:
[cut]

....

Larry... what the heck are you talking about?  No really you lost me
:)

Are you talking about hitting a button and flushing all the code but
what's new, or about actually making different tarballs cause the
source tree to auto-adapt to what's in it?

What is RCS/SCCS

--Bluefox Icy

Re: Kernel source tree splitting

[James H. Cloos Jr.]

>>>>> "Larry" == Larry McVoy <lm@bitmover.com> writes:

Larry> It's a very little known fact but if you are in an RCS
Larry> or SCCS (and BK looks like SCCS to make) source tree and the
Larry> files are not checked out, you can just say

Larry> 	make

Larry> and make will look for a makefile, if there isn't one but there
Larry> is a SCCS/s.[Mm]akefile it will check it out, look at the
Larry> dependencies and start checking those out and keep doing it to
Larry> satisfy the target.

That has never worked for me with Linus' or Marcello's trees.  It
always falls down at some point or other.  

Usually it fails looking for something under include, 2.5 also dies
in make oldconfig.

-JimC

Re: Kernel source tree splitting

[Martin J. Bligh]

> I'm probably misreading this...but,
> 
> Have you tried this yet?  Does it modify/customize all Kconfig
> and Makefiles for the selected tree splits?
> 
> A few days ago, in one tree, I rm-ed arch/{all that I don't need}
> and drivers/{all that I don't need}.
> After that I couldn't run "make *config" because it wants all of
> those files, even if I don't want them.
> 
> So there are many edits that needed to be done in lots of
> Kconfig and Makefiles if one selectively pulls or omits certain
> sub-directories.

Indeed, I ran across the same thing a while back. Would be *really* nice to
fix, if only so some poor sod over a modem can download a smaller tarball,
or save some diskspace.

M.

Re: Kernel source tree splitting

[Ed Sweetman]

Martin J. Bligh wrote:
>>I'm probably misreading this...but,
>>
>>Have you tried this yet?  Does it modify/customize all Kconfig
>>and Makefiles for the selected tree splits?
>>
>>A few days ago, in one tree, I rm-ed arch/{all that I don't need}
>>and drivers/{all that I don't need}.
>>After that I couldn't run "make *config" because it wants all of
>>those files, even if I don't want them.
>>
>>So there are many edits that needed to be done in lots of
>>Kconfig and Makefiles if one selectively pulls or omits certain
>>sub-directories.
> 
> 
> Indeed, I ran across the same thing a while back. Would be *really* nice to
> fix, if only so some poor sod over a modem can download a smaller tarball,
> or save some diskspace.
> 
> M.
> 

Tried doing this a while ago with my own cvs server. I'd download the 
tarball untar it into the cvs dump. I made a modules file tha that 
described all kinds of possible modules and then modules that utilized 
those and so on until you could just do co -z9 linux-i386 to get a all 
the files required to make any i386 linux ...it was mostly to just 
separate the archs, not unneeded drivers. Of course i quickly ran into 
the make not working without all the other files from the other archs 
and i came to the conclusion you couldn't do this without editing 
certain build scripts and files (and since I didn't want the files to be 
altered in case people would eventually want to call security into 
question) i killed the whole idea.

This idea comes and goes like many others.  As far as modular 
programming goes the kernel isn't written to be very modular, either 
that or it is and the build system just looks over that fact.  It would 
be interesting to see how hard it would be to have the current config to 
not only generate its .config but also a list of commands that an 
autoconf created ./configure can read and use to generate the makefiles 
we then use to build the kernel.  it could also help in enforcing 
minimum versions instead of hoping the user does it before complaining 
to people who could be doing better things with their time.  Anyways, 
i'm sure this idea has been coined and dropped before as well.  So it's 
back to the way things are done like normal.

Re: Kernel source tree splitting

[Peter Riocreux]

This might also help highlight files that are not really in a natural
place in the tree. If people building for x86 platforms start finding
they need a file that is in the sparc part of the tree then that
suggests that said file is probably at least partly non-arch
specific. This would also apply to, say, the audio and video parts of
the drivers section of the tree.

While these things do get reported on here occasionally, and fixed,
making it possible to wholesale remove parts of the tree that people
naturally assume you *shouldn't* need for a particular config ought to
show these up more easily.

Of course it adds another variable into the mix when people report
build problems....

Peter

Re: Kernel source tree splitting

[Randy.Dunlap]

[-- Attachment #1: Type: text/plain, Size: 4256 bytes --]

On Wed, 30 Apr 2003 17:21:02 -0700 "Randy.Dunlap" <rddunlap@osdl.org> wrote:

| Hi,
| 
| I'm probably misreading this...but,
| 
| Have you tried this yet?  Does it modify/customize all Kconfig
| and Makefiles for the selected tree splits?
| 
| A few days ago, in one tree, I rm-ed arch/{all that I don't need}
| and drivers/{all that I don't need}.
| After that I couldn't run "make *config" because it wants all of
| those files, even if I don't want them.
| 
| So there are many edits that needed to be done in lots of
| Kconfig and Makefiles if one selectively pulls or omits certain
| sub-directories.

and on 2003-04-30 rddunlap wrote:
| I seem to try for simple solutions when possible and feasible.
| 
| In this case, if I were doing it, I would try changing (e.g.) in
| arch/i386/kernel/Kconfig, this line:
| source "drivers/eisa/Kconfig"
| to
| optsource "drivers/eisa/Kconfig"
| where optsource means that the file is optional -- if not found,
| ignore it.  And then see what happens, how far it can go,
| what the next problem is....
| 
| If this could be made to work, then entire subdirectories/subsystems
| could be optional.

So I did a proof-of-concept version of this, without modifying any
source code.  I rm-ed arch/<many>, drivers/<many>, and fs/<many>
and then wrote a shell script that looks for missing dirs, Kconfigs,
and Makefile.lib files and puts empty ones back in their places.
The shell script only works for what I rm-ed, but it could be made
smarter if anyone wants to pursue this.  (attached)

After doing that I was able to build and boot that kernel, so it
(concept) did work.

For a kernel source tree that hadn't been built/compiled in, the size
was reduced from roughly 200 MB down to roughly 133 MB.

~Randy


| On Wed, 30 Apr 2003 19:46:13 -0400 rmoser <mlmoser@comcast.net> wrote:
| 
| | Eh, Linus won't be happy making a bunch of tarballs.
| | I've made it less work if you read the message here...
| | 
| | The message mirrored at:
| | 
| | http://marc.theaimsgroup.com/?l=linux-kernel&m=105173077417526&w=2
| | 
| | Shows my pre-thought on this subject.  I thought a bit more,
| | and began to come up with a simple sketch to lead the
| | way in case anyone becomes interested.
| | 
| | First off, the kernel tarballs would be built by a script
| | that splits the source tree apart appropriately and tar's it
| | up.  How this is done is explained.
| | 
| | Second off, there's always a script to download that runs
| | wget and gets the source tree from which it was downloaded.
| | The whole thing.  As in, every tarball is downloaded and
| | untar'd for the user, assembling the full kernel source
| | tree (as it would be if you untar'd it now).
| | 
| | Now, I explained LOD's in that message above in small
| | detail.  But, for clarity, LOD's are files which explain
| | which pieces of source in the kernel tree belong to the
| | LOD; what gets added to the config; where their makefiles
| | are; what config options depend on other linux options;
| | and what groups these LOD's are in.
| | 
| | A command such as `make disttree` should read the LOD's,
| | split apart each linux option, tar 'em together, and
| | then compress the tar's.  Then Linus could just scp the
| | new directory of tar's and a script up.
| | 
| | As for download, the script that goes up can be
| | downloaded (duh), and then run (... why do I bother?).
| | Now this script would run in "dumb mode" (unless the
| | user tells it not to maybe?) and rip down the whole
| | tree, untar it, and rebuild the original source tree.
| | I think.  I'm not sure, I really haven't tried yet.
| | I'll tell you how it works after it's implimented, if
| | ever that happens.  This would likely require wget.
| | 
| | Of course there's always the ftp method.  Go download
| | the pieces you want, untar 'em, copy 'em to the same
| | directory, and the build system adjusts.  but newbies
| | and developers, for completely opposite reasons, will
| | want to use the script in dumb mode.
| | 
| | For experienced users, this will make configuration
| | somewhat easier, as the user can avoid being prompted
| | for irrelavent drivers.  This is just a concept idea,
| | not a fully thought-out idea.  What do you think?
| | 
| | --Bluefox Icy

[-- Attachment #2: builder --]
[-- Type: application/octet-stream, Size: 1021 bytes --]

#! /bin/sh
# idea and script by Randy.Dunlap <rddunlap@osdl.org>

# optsource test:  any Kconfig "source <filename>" files that are missing
#	are touched to produce/provide empty files for them;
# make clean ---> make -k clean
#   to ignore missing Makefiles

DIRS="drivers/eisa drivers/mca drivers/mtd drivers/acorn drivers/acorn/scsi drivers/message drivers/message/fusion drivers/message/i2o net/ipx net/decnet drivers/acorn/net drivers/atm drivers/s390 drivers/s390/net net/ax25 net/irda drivers/isdn drivers/telephony net/bluetooth"

LIBDIRS="net/bluetooth net/bluetooth/bnep"
LIBFILES="net/bluetooth/bnep"

# mkdir for each DIRS and touch dir/Kconfig
for dir in $DIRS ; do
	if [ ! -d $dir ]; then
		mkdir $dir
	fi
	if [ ! -f $dir/Kconfig ]; then
		touch $dir/Kconfig
	fi
done

# mkdir for each LIBDIRS
for dir in $LIBDIRS ; do
	if [ ! -d $dir ]; then
		mkdir $dir
	fi
done

# touch Makefile.lib for each LIBFILES
for dir in $LIBDIRS ; do
	if [ ! -f $dir/Makefile.lib ]; then
		touch $dir/Makefile.lib
	fi
done

###

Re: Kernel source tree splitting

[rmoser]

*********** REPLY SEPARATOR  ***********

On 5/1/2003 at 5:09 PM Randy.Dunlap wrote:

>On Wed, 30 Apr 2003 17:21:02 -0700 "Randy.Dunlap" <rddunlap@osdl.org>
>wrote:
>
>| Hi,
>|
>| I'm probably misreading this...but,
>|
>| Have you tried this yet?  Does it modify/customize all Kconfig
>| and Makefiles for the selected tree splits?
>|
>| A few days ago, in one tree, I rm-ed arch/{all that I don't need}
>| and drivers/{all that I don't need}.
>| After that I couldn't run "make *config" because it wants all of
>| those files, even if I don't want them.
>|
>| So there are many edits that needed to be done in lots of
>| Kconfig and Makefiles if one selectively pulls or omits certain
>| sub-directories.
>
>and on 2003-04-30 rddunlap wrote:
>| I seem to try for simple solutions when possible and feasible.
>|
>| In this case, if I were doing it, I would try changing (e.g.) in
>| arch/i386/kernel/Kconfig, this line:
>| source "drivers/eisa/Kconfig"
>| to
>| optsource "drivers/eisa/Kconfig"
>| where optsource means that the file is optional -- if not found,
>| ignore it.  And then see what happens, how far it can go,
>| what the next problem is....
>|
>| If this could be made to work, then entire subdirectories/subsystems
>| could be optional.
>
>So I did a proof-of-concept version of this, without modifying any
>source code.  I rm-ed arch/<many>, drivers/<many>, and fs/<many>
>and then wrote a shell script that looks for missing dirs, Kconfigs,
>and Makefile.lib files and puts empty ones back in their places.
>The shell script only works for what I rm-ed, but it could be made
>smarter if anyone wants to pursue this.  (attached)
>

Yes.  Well the build system (kernel configuration) would be modified
to instead of having a list of Kconfigs and dir's and Makefile.libs and
such, be able to scan a directory of files which tell it about those
things.  At least, in my design it would.

>After doing that I was able to build and boot that kernel, so it
>(concept) did work.
>

Well that's good.

>For a kernel source tree that hadn't been built/compiled in, the size
>was reduced from roughly 200 MB down to roughly 133 MB.
>

...............   You know.  Maybe it's not enough to split into categories.
Maybe it should be categories and categorical breakdowns.  I can see
3.0 or something (maybe even 2.8) reaching 1 gig.

--Bluefox Icy
>~Randy
>
>

Re: Kernel source tree splitting

[Chuck Ebbert]

Martin J. Bligh wrote:

>> So there are many edits that needed to be done in lots of
>> Kconfig and Makefiles if one selectively pulls or omits certain
>> sub-directories.
>
> Indeed, I ran across the same thing a while back. Would be *really* nice to
> fix, if only so some poor sod over a modem can download a smaller tarball,
> or save some diskspace.

 I have seven source trees on disk right now.  Getting rid off all
the archs but i386 would not only save tons of space, it would also
make 'grep -r' go faster and stop spewing irrelevant hits for archs
that I couldn't care less about.


------
 Chuck

Re: Kernel source tree splitting

[Martin J. Bligh]

>>> So there are many edits that needed to be done in lots of
>>> Kconfig and Makefiles if one selectively pulls or omits certain
>>> sub-directories.
>> 
>> Indeed, I ran across the same thing a while back. Would be *really* nice
>> to fix, if only so some poor sod over a modem can download a smaller
>> tarball, or save some diskspace.
> 
>  I have seven source trees on disk right now.  Getting rid off all
> the archs but i386 would not only save tons of space, it would also
> make 'grep -r' go faster and stop spewing irrelevant hits for archs
> that I couldn't care less about.

Indeed. But whilst you're waiting, hardlink everything together, and 
patch the differences (patch knows how to break hardlinks). Make a 
script that cp -lR's the tree to another copy (normally takes < 1s), and
then remove the other arches. grep that.

cscope with prebuilt indeces on a filtered subset of the files may well do
better than grep, depending on exactly what you're doing (does 99% of it
for me). Don't use the cscope in Debian Woody, it's broken.

M.

Re: Kernel source tree splitting

[Willy TARREAU]

On Thu, May 01, 2003 at 07:06:37AM -0700, Martin J. Bligh wrote:
 
> Indeed. But whilst you're waiting, hardlink everything together, and 
> patch the differences (patch knows how to break hardlinks). Make a 
> script that cp -lR's the tree to another copy (normally takes < 1s), and
> then remove the other arches. grep that.

I agree with Martin here, I always use hardlinks, and when I have too many
kernel trees, I even recompact them by diff/rm/cp -l/patch to get as small
differences as possible. You can have tens of kernels in less than 400 MB,
and tools such as diff and grep are really fast because it's easy to keep
several kernels in the cache.

The only danger is to modify several files at once with stupid operations such
as "cat $file.help >> Documentation/Configure.help" which are sometimes
included in some scripts. It would be cool to be able to lock the source, but
I never found how (perhaps I should try chattr+i ?). And I don't know how to
force vi and emacs to unlink before saving, so I have to be careful before
certain operations. But all in all, it's extremely useful.

Cheers,
Willy

Re: Kernel source tree splitting

[Martin J. Bligh]

>> Indeed. But whilst you're waiting, hardlink everything together, and 
>> patch the differences (patch knows how to break hardlinks). Make a 
>> script that cp -lR's the tree to another copy (normally takes < 1s), and
>> then remove the other arches. grep that.
> 
> I agree with Martin here, I always use hardlinks, and when I have too many
> kernel trees, I even recompact them by diff/rm/cp -l/patch to get as small
> differences as possible. You can have tens of kernels in less than 400 MB,
> and tools such as diff and grep are really fast because it's easy to keep
> several kernels in the cache.
> 
> The only danger is to modify several files at once with stupid operations
> such as "cat $file.help >> Documentation/Configure.help" which are
> sometimes included in some scripts. It would be cool to be able to lock
> the source, but I never found how (perhaps I should try chattr+i ?). And
> I don't know how to force vi and emacs to unlink before saving, so I have
> to be careful before certain operations. But all in all, it's extremely
> useful.

find -type f | xargs chmod ugo-w 

whenever you make a new copy seems to work pretty well to me. 
Then you use "dupvi" to edit the files, which is just a little wrapper that
breaks the link, and edits the file. 

For added paranoia, I suppose you could make your "main" views (eg the
unpatched ones) owned by another user. But I've never had a problem with
just chmod, and I have a lot of views ... 1689 all linked together ;-)

-r--r--r--  1689 fletch   fletch      18691 Nov 17 20:29 COPYING

Oh, and diff of views takes < 1s (diff understands hardlinks too, it seems).
Any SCM can kiss my ass ;-)

M.

Re: Kernel source tree splitting

[Willy TARREAU]

On Thu, May 01, 2003 at 07:35:48AM -0700, Martin J. Bligh wrote:
> >> Indeed. But whilst you're waiting, hardlink everything together, and 
> >> patch the differences (patch knows how to break hardlinks). Make a 
> >> script that cp -lR's the tree to another copy (normally takes < 1s), and
> >> then remove the other arches. grep that.
> > 
> > I agree with Martin here, I always use hardlinks, and when I have too many
> > kernel trees, I even recompact them by diff/rm/cp -l/patch to get as small
> > differences as possible. You can have tens of kernels in less than 400 MB,
> > and tools such as diff and grep are really fast because it's easy to keep
> > several kernels in the cache.
> > 
> > The only danger is to modify several files at once with stupid operations
> > such as "cat $file.help >> Documentation/Configure.help" which are
> > sometimes included in some scripts. It would be cool to be able to lock
> > the source, but I never found how (perhaps I should try chattr+i ?). And
> > I don't know how to force vi and emacs to unlink before saving, so I have
> > to be careful before certain operations. But all in all, it's extremely
> > useful.
> 
> find -type f | xargs chmod ugo-w 

that's also what I do when I don't trust a script ;-)

> whenever you make a new copy seems to work pretty well to me. 
> Then you use "dupvi" to edit the files, which is just a little wrapper that
> breaks the link, and edits the file. 

I didn't know about dupvi. But I admit that it's really easy to break the link
before starting vi, to ensure there's no problem.

> For added paranoia, I suppose you could make your "main" views (eg the
> unpatched ones) owned by another user.

this could help emacs because it tries every possibility to save a file, even
changing its rights if there's no other way.

> But I've never had a problem with just chmod, and I have a lot of views ...
> 1689 all linked together ;-)
> 
> -r--r--r--  1689 fletch   fletch      18691 Nov 17 20:29 COPYING

and I thought it was dirty when I began to reach 50 links.... :-)

Cheers,
Willy

Re: Kernel source tree splitting

[Larry McVoy]

On Thu, May 01, 2003 at 07:35:48AM -0700, Martin J. Bligh wrote:
> just chmod, and I have a lot of views ... 1689 all linked together ;-)
> 
> -r--r--r--  1689 fletch   fletch      18691 Nov 17 20:29 COPYING

That's a bunch.  Who's fletch?  

And more importantly, how do you keep track of what is in each of those?
I can see having 20, 100, whatever, and keeping it straight in your head
but 1600?  

> Oh, and diff of views takes < 1s (diff understands hardlinks too, it seems).
> Any SCM can kiss my ass ;-)

Kiss, kiss :)

Ted T'so made us support hard links for the revision control files for the
same reasons and it works pretty well.  We haven't extended that to the
checked out files because I'm nervous about tools which don't break the
links.

On the other hand, we could hard link the checked out files if they
were checked out read-only which mimics what you are doing with the
chmod...  That's a thought.

We'll still never be as fast as a pure hardlinked tree, that's balls to
the wall as fast as you can go as far as I can tell.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: Kernel source tree splitting

[Martin J. Bligh]

>> -r--r--r--  1689 fletch   fletch      18691 Nov 17 20:29 COPYING
> 
> That's a bunch.  Who's fletch?  
> 
> And more importantly, how do you keep track of what is in each of those?
> I can see having 20, 100, whatever, and keeping it straight in your head
> but 1600?  

I have one view for every patch, and a bunch of scripts to manage them,
tear them down, build them up, and create patches from all of them in one
dir (they're a numbered sequence). A decent tree structure helps ;-)

It helps me to keep all the patches separated out - I want to be able to
carry forward 100 patches (at least) in sequence against the mainline tree,
and keep them all separate. Totally different problem from the one Linus
has, IMHO. 

I guess I have a view for what you call a changeset ... AFAICS, if you just
take 100 stacked patches, and do a merge forward through 30 versions, you
just end up with a big mess that you can't extract the real "changes"
you're making back out from the main view. But I've never really tried, it
might work out in BK or something I suppose. If that worked (in ~1 minute
for 100 patches), I'd be very tempted to try it (I hate learning curves for
new tools, half the time they're just burnt time).

> We'll still never be as fast as a pure hardlinked tree, that's balls to
> the wall as fast as you can go as far as I can tell.

Ow ;-)

M.

Re: Kernel source tree splitting

[Balram Adlakha]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

On Thu, May 01, 2003 at 07:54:03AM -0400, Chuck Ebbert wrote:
> Martin J. Bligh wrote:
> 
> >> So there are many edits that needed to be done in lots of
> >> Kconfig and Makefiles if one selectively pulls or omits certain
> >> sub-directories.
> >
> > Indeed, I ran across the same thing a while back. Would be *really* nice to
> > fix, if only so some poor sod over a modem can download a smaller tarball,
> > or save some diskspace.
> 
>  I have seven source trees on disk right now.  Getting rid off all
> the archs but i386 would not only save tons of space, it would also
> make 'grep -r' go faster and stop spewing irrelevant hits for archs
> that I couldn't care less about.
> 
> 
> ------
>  Chuck
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

I agree with you. Making different trees for different archs will make the tarball much smaller. Usually people only use one architecture and the other code lies waste. I think this has been discussed many times but It really is worth doing.
-- 

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Kernel source tree splitting

[Ben Greear]

Balram Adlakha wrote:
> On Thu, May 01, 2003 at 07:54:03AM -0400, Chuck Ebbert wrote:

>> I have seven source trees on disk right now.  Getting rid off all
>>the archs but i386 would not only save tons of space, it would also
>>make 'grep -r' go faster and stop spewing irrelevant hits for archs
>>that I couldn't care less about.

> 
> I agree with you. Making different trees for different archs will make the tarball much smaller. Usually people only use one architecture and the other code lies waste. I think this has been discussed many times but It really is worth doing.

How about a script to just prune it once you download it.  That will at least fix your
disk space & grep issue, and will not affect those of us who like to see it all.

If you want to save download bandwidth, just use incremental diffs and/or something
like bk or one of the cvs exports.

Ben


> -- 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: Kernel source tree splitting

[John Bradford]

> >> I have seven source trees on disk right now.  Getting rid off all
> >>the archs but i386 would not only save tons of space, it would also
> >>make 'grep -r' go faster and stop spewing irrelevant hits for archs
> >>that I couldn't care less about.
> 
> > 
> > I agree with you. Making different trees for different archs will make the tarball much smaller. Usually people only use one architecture and the other code lies waste. I think this has been discussed many times but It really is worth doing.
> 
> How about a script to just prune it once you download it.  That will at least fix your
> disk space & grep issue, and will not affect those of us who like to see it all.

Agreed - we don't want to obfuscate getting the whole kernel tree for
anybody who wants to - some of us have loads of bandwidth, and do
compile for multiple architectures :-).

> If you want to save download bandwidth, just use incremental diffs and/or something
> like bk or one of the cvs exports.

Even the current 2.5 source tree is downloadable during a, (long),
lunch break over ISDN, (or a very long lunch break over a 56K modem
:-) ).

John.

Re: Kernel source tree splitting

[Chuck Ebbert]

> But whilst you're waiting, hardlink everything together, and 
> patch the differences (patch knows how to break hardlinks).

  I don't trust that approach -- too easy to screw up.

> Make a script that cp -lR's the tree to another copy (normally
> takes < 1s), and then remove the other arches. grep that.

  That could work for the reference tree.

> cscope with prebuilt indeces on a filtered subset of the files may well do
> better than grep, depending on exactly what you're doing (does 99% of it
> for me).

  Have you tried lxr?  The website is cool but you really need a local
copy for speed.

------
 Chuck