XZ Migration discussion

XZ Migration discussion

[J.H.]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey Everyone,

So as the subject states this is more a centralized discussion on
migration plans to using and providing xz for content on kernel.org.
Currently we provide gz and bz2, with gz acting as the original content
and kernel.org itself generating the resulting bz2 files.  There are a
couple of possible proposals and wanted to toss them out there, and get
feedback from everyone: the kernel community, the mirrors of kernel.org
and the direct users of kernel.org.

========================================================================

Option 1)

Leave gz as the master, and migrate bz2 to xz.  This will happen in
stages obviously. with bz2 ultimately being phased out.

	Migration option 1)

		All new content would be provided in .bz2 and .xz with
		an ultimate date set that the .bz2 files would stop
		being generated with new content.  This would leave all
		existing content alone and it would not be a migration
		of the current .bz2 files to xz

	Migration option 2)

		At some point there would be a mass conversion of all
		existing content to include .bz2 and .xz.  These would
		be run in parallel for a time period until it was
		determined that .bz2 was no longer needed and it would
		be removed from the servers leaving .gz and .xz

Option 2)

Convert the master data from gz to bz2 and use xz as the new file
format.  This has the downside of causing more tool churn as it means
the kernel developers will have to eventually convert from gz to bz2,
which means for a time there will be nag e-mails if you upload gz
instead of bz2 and such.  It would also mean that we (kernel.org) would
need to be able to support .gz and .bz2 as master data for a time.

Migration options are identical to Option 1 more or less, with either
just new content getting converted, or all content getting converted.

========================================================================

I'm personally leaning towards option 1, though personally don't really
have a preference on the migration options, as both obviously offer
different advantages, and again this e-mail is more to spur on the
discussion and come to some general consensus across all of the groups
concerned before moving forward with a more specific plan.

So I'm inviting discussion, questions and comments on this so we know
which way to ultimately go.

- - John 'Warthog9' Hawley
Chief Kernel.org Administrator
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkt0ThMACgkQ/E3kyWU9dicQIwCggCACEHuEViVvnIiv42McbCQh
SmUAn049beEHPucEc7X+0hBQkW6A5oTt
=ugaX
-----END PGP SIGNATURE-----

Re: XZ Migration discussion

[david]

On Thu, 11 Feb 2010, J.H. wrote:

> Hey Everyone,
>
> So as the subject states this is more a centralized discussion on
> migration plans to using and providing xz for content on kernel.org.
> Currently we provide gz and bz2, with gz acting as the original content
> and kernel.org itself generating the resulting bz2 files.  There are a
> couple of possible proposals and wanted to toss them out there, and get
> feedback from everyone: the kernel community, the mirrors of kernel.org
> and the direct users of kernel.org.
>
> ========================================================================
>
> Option 1)
>
> Leave gz as the master, and migrate bz2 to xz.  This will happen in
> stages obviously. with bz2 ultimately being phased out.
>
> 	Migration option 1)
>
> 		All new content would be provided in .bz2 and .xz with
> 		an ultimate date set that the .bz2 files would stop
> 		being generated with new content.  This would leave all
> 		existing content alone and it would not be a migration
> 		of the current .bz2 files to xz
>
> 	Migration option 2)
>
> 		At some point there would be a mass conversion of all
> 		existing content to include .bz2 and .xz.  These would
> 		be run in parallel for a time period until it was
> 		determined that .bz2 was no longer needed and it would
> 		be removed from the servers leaving .gz and .xz
>
> Option 2)
>
> Convert the master data from gz to bz2 and use xz as the new file
> format.  This has the downside of causing more tool churn as it means
> the kernel developers will have to eventually convert from gz to bz2,
> which means for a time there will be nag e-mails if you upload gz
> instead of bz2 and such.  It would also mean that we (kernel.org) would
> need to be able to support .gz and .bz2 as master data for a time.
>
> Migration options are identical to Option 1 more or less, with either
> just new content getting converted, or all content getting converted.

I thought that the result of the poll that Linus did a few days ago was 
that there were advantages in having .gz, but that .xz was both smaller 
and faster than .bz2 and therefor the best thing to do would be to end up 
with .gz and .xz.

As such your option 2 doesn't seem like a reasonable path to go through.

David Lang

Re: XZ Migration discussion

[H. Peter Anvin]

On 02/11/2010 10:36 AM, J.H. wrote:
> 
> Option 1)
> 
> Leave gz as the master, and migrate bz2 to xz.  This will happen in
> stages obviously. with bz2 ultimately being phased out.
> 
> 	Migration option 1)
> 
> 		All new content would be provided in .bz2 and .xz with
> 		an ultimate date set that the .bz2 files would stop
> 		being generated with new content.  This would leave all
> 		existing content alone and it would not be a migration
> 		of the current .bz2 files to xz
> 
> 	Migration option 2)
> 
> 		At some point there would be a mass conversion of all
> 		existing content to include .bz2 and .xz.  These would
> 		be run in parallel for a time period until it was
> 		determined that .bz2 was no longer needed and it would
> 		be removed from the servers leaving .gz and .xz
> Option 2)
> 
> Convert the master data from gz to bz2 and use xz as the new file
> format.  This has the downside of causing more tool churn as it means
> the kernel developers will have to eventually convert from gz to bz2,
> which means for a time there will be nag e-mails if you upload gz
> instead of bz2 and such.  It would also mean that we (kernel.org) would
> need to be able to support .gz and .bz2 as master data for a time.
> 
> Migration options are identical to Option 1 more or less, with either
> just new content getting converted, or all content getting converted.
> 
> ========================================================================
> 
> I'm personally leaning towards option 1, though personally don't really
> have a preference on the migration options, as both obviously offer
> different advantages, and again this e-mail is more to spur on the
> discussion and come to some general consensus across all of the groups
> concerned before moving forward with a more specific plan.
> 
> So I'm inviting discussion, questions and comments on this so we know
> which way to ultimately go.

My personal recommendation would be for Option 1, Migration option 2.

I think the idea of having two "best" file formats (Migration Option 1)
in use indefinitely is rather pointless.

As for the motivations for Option 1:

a) Currently, .gz contents is original, whereas .bz2 contents is
automatically generated.  Flushing the .gz files would mean flushing
original content.

b) .gz is one of the most widely supported compression formats ever
created, plus it is very fast, especially on the decompression side.
.xz is reasonably fast but very memory intensive; .bz2 is moderately
fast and still fairly memory intensive (but not as much as .xz).  In
other words, .gz provides an option for small, slow or old systems, or
systems running inferior operating systems.  .xz provides the best
compression.  .bz2 is in between, but it doesn't serve either purpose as
well as the other two.

Realistically speaking, kernel.org itself will carry all three formats
for an extended transition time (at least a year); we will probably
discontinue the victim format only when we start running shy on disk
space.  However, we obviously don't want to push this burden onto all
the mirrors.  Therefore, I would really appreciate feedback from mirror
admins as to how you would prefer to see the transition -- either
transition -- happen.

	-hpa

Re: [kernel.org mirrors] XZ Migration discussion

[Carlos Carvalho]

H. Peter Anvin (hpa@zytor.com) wrote on 11 February 2010 11:48:
 >On 02/11/2010 10:36 AM, J.H. wrote:
 >> 
 >> Option 1)
 >> 
 >> Leave gz as the master, and migrate bz2 to xz.  This will happen in
 >> stages obviously. with bz2 ultimately being phased out.
 >> 
 >> 	Migration option 1)
 >> 
 >> 		All new content would be provided in .bz2 and .xz with
 >> 		an ultimate date set that the .bz2 files would stop
 >> 		being generated with new content.  This would leave all
 >> 		existing content alone and it would not be a migration
 >> 		of the current .bz2 files to xz
 >> 
 >> 	Migration option 2)
 >> 
 >> 		At some point there would be a mass conversion of all
 >> 		existing content to include .bz2 and .xz.  These would
 >> 		be run in parallel for a time period until it was
 >> 		determined that .bz2 was no longer needed and it would
 >> 		be removed from the servers leaving .gz and .xz
 >> Option 2)
 >> 
 >> Convert the master data from gz to bz2 and use xz as the new file
 >> format.  This has the downside of causing more tool churn as it means
 >> the kernel developers will have to eventually convert from gz to bz2,
 >> which means for a time there will be nag e-mails if you upload gz
 >> instead of bz2 and such.  It would also mean that we (kernel.org) would
 >> need to be able to support .gz and .bz2 as master data for a time.
 >> 
 >> Migration options are identical to Option 1 more or less, with either
 >> just new content getting converted, or all content getting converted.
 >
 >My personal recommendation would be for Option 1, Migration option 2.

Agreed (speaking for the ftp.br.kernel.org mirror).

Re: [kernel.org users] XZ Migration discussion

[Willy Tarreau]

Hi John,

On Thu, Feb 11, 2010 at 10:36:03AM -0800, J.H. wrote:
(...)
> I'm personally leaning towards option 1, though personally don't really
> have a preference on the migration options, as both obviously offer
> different advantages, and again this e-mail is more to spur on the
> discussion and come to some general consensus across all of the groups
> concerned before moving forward with a more specific plan.

I too think option 1 is better for various reasons, one of them being
that gzip is present on every platform and OS right now, while bzip2
is orders of magnitude less common. Of course on Linux we all have both
of them, but I find it important to continue to provide the sources in
a standard format. Also, on the platforms where bzip2 is not installed
by default, switching to xz should not be that hard, because either bz2
is currently not used and that changes nothing, or the users managed to
install it themselves, and they'll simply have to do the same with xz.

I don't have much preference for any of the two migration options though.

Just my 2 cents,
Willy

Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

Hi!

> Option 1)
> 
> Leave gz as the master, and migrate bz2 to xz.  This will happen in
> stages obviously. with bz2 ultimately being phased out.
> 
> 	Migration option 1)
> 
> 		All new content would be provided in .bz2 and .xz with
> 		an ultimate date set that the .bz2 files would stop
> 		being generated with new content.  This would leave all
> 		existing content alone and it would not be a migration
> 		of the current .bz2 files to xz

I believe this is cleanest. gzip has performance advantages, and old
files suddenly disappearing would be just weird.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/11/2010 12:51 PM, Pavel Machek wrote:
> Hi!
> 
>> Option 1)
>>
>> Leave gz as the master, and migrate bz2 to xz.  This will happen in
>> stages obviously. with bz2 ultimately being phased out.
>>
>> 	Migration option 1)
>>
>> 		All new content would be provided in .bz2 and .xz with
>> 		an ultimate date set that the .bz2 files would stop
>> 		being generated with new content.  This would leave all
>> 		existing content alone and it would not be a migration
>> 		of the current .bz2 files to xz
> 
> I believe this is cleanest. gzip has performance advantages, and old
> files suddenly disappearing would be just weird.
> 

It would also be "just weird" to:

a) require that the end user knows the particular compression format
used by a particular legacy file.
b) having to have the mirror system deal with a mix of .bz2 and .xz files.

	-hpa

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Thu, 11 Feb 2010 14:22:35 -0800, H. Peter Anvin wrote:
> It would also be "just weird" to:
> 
> a) require that the end user knows the particular compression format
> used by a particular legacy file.

While we're here... I think it is weird that the testing files for the
latest kernel are in testing/ directly, while the older ones have their
own subdirectory. I have a script which is greatly confused by this,
and I can imagine that other tools (for example ketchup) appreciate
this inconsistency moderately too. It is probably not ideal for mirrors
either... I don't know if the mirroring system is smart enough to deal
with file moves? If not then it generates useless traffic.

So I would like to propose that testing files are placed directly in
their final location. This would be both more consistent and easier
for everyone.

Thanks,
-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

Hi Pavel, all,

On Thu, 11 Feb 2010 21:51:29 +0100, Pavel Machek wrote:
> Hi!
> 
> > Option 1)
> > 
> > Leave gz as the master, and migrate bz2 to xz.  This will happen in
> > stages obviously. with bz2 ultimately being phased out.
> > 
> > 	Migration option 1)
> > 
> > 		All new content would be provided in .bz2 and .xz with
> > 		an ultimate date set that the .bz2 files would stop
> > 		being generated with new content.  This would leave all
> > 		existing content alone and it would not be a migration
> > 		of the current .bz2 files to xz
> 
> I believe this is cleanest. gzip has performance advantages, (...)

I have been investigating this issue and would like to share my
findings as an additional data point for this discussion.

For my testing, I have been using the slowest machine I still have
available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard
disk drive. I've been writing down the duration of each task it took to
boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2
formats.

Raw results are as follow (format=min:s):

downloading linux-2.6.27.tar.bz2	  5:01
downloading patch-2.6.27.45.bz2		  0:02
unpacking linux-2.6.27.tar.bz2		  7:28
applying patch-2.6.27.45.bz2		  1:21
----------------------------------------------
total for bz2				 13:52

downloading linux-2.6.27.tar.gz		  6:23
downloading patch-2.6.27.45.gz		  0:02
unpacking linux-2.6.27.tar.gz		  3:20
applying patch-2.6.27.45.gz		  1:10
----------------------------------------------
total for gz				 10:55

So the gz option is unsurprisingly faster, setting up the source tree
takes almost 3 minutes less (-21%).

Then the (common) build and installation times:

building				117:26
installing modules			  0:12
----------------------------------------------
total					117:38

This is a customized kernel, as small as I could do, with almost no
features and the minimal set of drivers. As you can see, the build time
is one order of magnitude greater than the tree setup time. Comparing
the total times from download to install between bz2 and gz:

bz2: 13:52 + 117:38 = 131:30
gz:  10:55 + 117:38 = 128:33

Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I
think we can plain discard the argument "I need .gz because my machine
is slow" from now on. It simply doesn't hold.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Geert Uytterhoeven]

On Sat, Feb 13, 2010 at 18:10, Jean Delvare <khali@linux-fr.org> wrote:
> On Thu, 11 Feb 2010 21:51:29 +0100, Pavel Machek wrote:
>> I believe this is cleanest. gzip has performance advantages, (...)
>
> I have been investigating this issue and would like to share my
> findings as an additional data point for this discussion.
>
> For my testing, I have been using the slowest machine I still have
> available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard
> disk drive. I've been writing down the duration of each task it took to
> boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2
> formats.
>
> Raw results are as follow (format=min:s):
>
> downloading linux-2.6.27.tar.bz2          5:01
> downloading patch-2.6.27.45.bz2           0:02
> unpacking linux-2.6.27.tar.bz2            7:28
> applying patch-2.6.27.45.bz2              1:21
> ----------------------------------------------
> total for bz2                            13:52
>
> downloading linux-2.6.27.tar.gz           6:23
> downloading patch-2.6.27.45.gz            0:02
> unpacking linux-2.6.27.tar.gz             3:20
> applying patch-2.6.27.45.gz               1:10
> ----------------------------------------------
> total for gz                             10:55
>
> So the gz option is unsurprisingly faster, setting up the source tree
> takes almost 3 minutes less (-21%).
>
> Then the (common) build and installation times:
>
> building                                117:26
> installing modules                        0:12
> ----------------------------------------------
> total                                   117:38
>
> This is a customized kernel, as small as I could do, with almost no
> features and the minimal set of drivers. As you can see, the build time
> is one order of magnitude greater than the tree setup time. Comparing
> the total times from download to install between bz2 and gz:
>
> bz2: 13:52 + 117:38 = 131:30
> gz:  10:55 + 117:38 = 128:33
>
> Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I
> think we can plain discard the argument "I need .gz because my machine
> is slow" from now on. It simply doesn't hold.

166 MHz and 64 MB of RAM is still an order of magnitude more than some other
machines Linux is capable of running on.

Of course we no longer build kernels on those machines natively, we
cross-compile
on much faster machines (and use git to fetch the kernel sources, FWIW ;-).

BTW, who still downloads full kernel tarballs? From my experience, that's mostly
distro people who have scripts to build kernels from tarballs + a
bunch of patches?
I guess actual developers use git nowadays, even if they're stuck on
an old 2.6.x kernel?
Backporting critical fixes is sooo much easier using git cherry-pick...

Do we have statistics about tarball downloads vs. git clone / git pull?

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: [kernel.org users] XZ Migration discussion

[Randy Dunlap]

On 02/13/10 10:49, Geert Uytterhoeven wrote:
> On Sat, Feb 13, 2010 at 18:10, Jean Delvare <khali@linux-fr.org> wrote:
>> On Thu, 11 Feb 2010 21:51:29 +0100, Pavel Machek wrote:
>>> I believe this is cleanest. gzip has performance advantages, (...)
>>
>> I have been investigating this issue and would like to share my
>> findings as an additional data point for this discussion.
>>
>> For my testing, I have been using the slowest machine I still have
>> available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard
>> disk drive. I've been writing down the duration of each task it took to
>> boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2
>> formats.
>>
>> Raw results are as follow (format=min:s):
>>
>> downloading linux-2.6.27.tar.bz2          5:01
>> downloading patch-2.6.27.45.bz2           0:02
>> unpacking linux-2.6.27.tar.bz2            7:28
>> applying patch-2.6.27.45.bz2              1:21
>> ----------------------------------------------
>> total for bz2                            13:52
>>
>> downloading linux-2.6.27.tar.gz           6:23
>> downloading patch-2.6.27.45.gz            0:02
>> unpacking linux-2.6.27.tar.gz             3:20
>> applying patch-2.6.27.45.gz               1:10
>> ----------------------------------------------
>> total for gz                             10:55
>>
>> So the gz option is unsurprisingly faster, setting up the source tree
>> takes almost 3 minutes less (-21%).
>>
>> Then the (common) build and installation times:
>>
>> building                                117:26
>> installing modules                        0:12
>> ----------------------------------------------
>> total                                   117:38
>>
>> This is a customized kernel, as small as I could do, with almost no
>> features and the minimal set of drivers. As you can see, the build time
>> is one order of magnitude greater than the tree setup time. Comparing
>> the total times from download to install between bz2 and gz:
>>
>> bz2: 13:52 + 117:38 = 131:30
>> gz:  10:55 + 117:38 = 128:33
>>
>> Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I
>> think we can plain discard the argument "I need .gz because my machine
>> is slow" from now on. It simply doesn't hold.
> 
> 166 MHz and 64 MB of RAM is still an order of magnitude more than some other
> machines Linux is capable of running on.
> 
> Of course we no longer build kernels on those machines natively, we
> cross-compile
> on much faster machines (and use git to fetch the kernel sources, FWIW ;-).
> 
> BTW, who still downloads full kernel tarballs? From my experience, that's mostly
> distro people who have scripts to build kernels from tarballs + a
> bunch of patches?

All of my daily build & boot testing downloads tarballs.
They use full linux-x.y.z for "releases" (like 2.6.32)
and they use patch-x.y.z.gz/bz2 for intermediate versions.

I don't mind updating the scripts to use .xz tarballs, but I would
really prefer to stick with tarballs instead of git trees.

> I guess actual developers use git nowadays, even if they're stuck on
> an old 2.6.x kernel?
> Backporting critical fixes is sooo much easier using git cherry-pick...
> 
> Do we have statistics about tarball downloads vs. git clone / git pull?

That would be good to see.

-- 
~Randy

Re: [kernel.org users] XZ Migration discussion

[Steven Rostedt]

On Sat, 2010-02-13 at 19:49 +0100, Geert Uytterhoeven wrote:

> BTW, who still downloads full kernel tarballs? From my experience, that's mostly
> distro people who have scripts to build kernels from tarballs + a
> bunch of patches?

When I download to develop, I use git. When I just want to install the
latest kernel on a box (not used for development) I use ketchup
(downloads tarballs).

For those that just want the latest kernel release, git is too bloated.

-- Steve

Re: [kernel.org users] XZ Migration discussion

[Stefan Richter]

Jean Delvare wrote:
> For my testing, I have been using the slowest machine I still have
> available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard
> disk drive. I've been writing down the duration of each task it took to
> boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2
> formats.
> 
> Raw results are as follow (format=min:s):
> 
> downloading linux-2.6.27.tar.bz2	  5:01
> downloading patch-2.6.27.45.bz2		  0:02
> unpacking linux-2.6.27.tar.bz2		  7:28
> applying patch-2.6.27.45.bz2		  1:21
> ----------------------------------------------
> total for bz2				 13:52
> 
> downloading linux-2.6.27.tar.gz		  6:23
> downloading patch-2.6.27.45.gz		  0:02
> unpacking linux-2.6.27.tar.gz		  3:20
> applying patch-2.6.27.45.gz		  1:10
> ----------------------------------------------
> total for gz				 10:55
> 
> So the gz option is unsurprisingly faster, setting up the source tree
> takes almost 3 minutes less (-21%).

If the download link had been slower than about 75 kB/s, the bz2 option
would have been faster even on this old machine.

With xz, download would be faster than bz2 and decompression would be
somewhere between bz2 and gz --- at least on machines without notable
memory constraints.  xz's decompressor is more memory hungry than
bzip2's one as far as I understand their manual pages.  But at the
default xz compressor setting of -6, the decompressor will still use
just 10 MB and should therefore not cause even your 64 MB machine to
swap all the time during decompression.

> Then the (common) build and installation times:
> 
> building				117:26
> installing modules			  0:12
> ----------------------------------------------
> total					117:38
> 
> This is a customized kernel, as small as I could do, with almost no
> features and the minimal set of drivers. As you can see, the build time
> is one order of magnitude greater than the tree setup time. Comparing
> the total times from download to install between bz2 and gz:
> 
> bz2: 13:52 + 117:38 = 131:30
> gz:  10:55 + 117:38 = 128:33
> 
> Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I
> think we can plain discard the argument "I need .gz because my machine
> is slow" from now on. It simply doesn't hold.

Yep, whether the target machine is meant to compile the kernel or to be
used to browse the source code (with a bare minimum of comfort), the
hardware resources required for either task mean that there isn't such a
great difference between gz, bz2, xz WRT resource requirements at the
receivers, except that xz goes easiest on network bandwidth and disk
utilization.
-- 
Stefan Richter
-=====-==-=- --=- -==-=
http://arcgraph.de/sr/

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Sun, 14 Feb 2010 00:28:39 +0100, Stefan Richter wrote:
> Jean Delvare wrote:
> > So the gz option is unsurprisingly faster, setting up the source tree
> > takes almost 3 minutes less (-21%).
> 
> If the download link had been slower than about 75 kB/s, the bz2 option
> would have been faster even on this old machine.
> 
> With xz, download would be faster than bz2 and decompression would be
> somewhere between bz2 and gz --- at least on machines without notable
> memory constraints.  xz's decompressor is more memory hungry than
> bzip2's one as far as I understand their manual pages.  But at the
> default xz compressor setting of -6, the decompressor will still use
> just 10 MB and should therefore not cause even your 64 MB machine to
> swap all the time during decompression.

Note that, if memory consumption is really a concern on either end, we
could use xz -5, which still achieves much better compression than bz2
but doesn't require more memory for decompression.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Phillip Lougher]

Jean Delvare wrote:

> 
> Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I
> think we can plain discard the argument "I need .gz because my machine
> is slow" from now on. It simply doesn't hold.
> 

I agree, but, IMHO the main argument for keeping .gz is cross-platform
availability and wide language support, not hardware limitations.  Doing
a quick google brings up .gz interfaces for every language you can think
of (C, Java, Perl, Python, TCL etc.), not to mention complete separate
implementations in Java and Pascal (not just wrappers on top of the zlib
library), and probably more.

With xz you have just one C/C++ implementation with a single library with
an undocumented API for C/C++ programmers.

It may be a slight stretch of the imagination, but with with .gz you can
conceive programmers writing programs to download a .gz from kernel.org and
decompressing/searching it, in almost any language of choice.  With the JAVA
implementation .gz is genuinely cross platform and you don't need glibc/
C++ compilers, just a Java VM.  Contrast with xz, where if the xz utility
isn't available, or doesn't do what you want, you're stuck with programming
in C/C++ with all the baggage that entails.

Phillip

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Sat, 13 Feb 2010 23:52:17 +0000, Phillip Lougher wrote:
> Jean Delvare wrote:
> 
> > 
> > Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I
> > think we can plain discard the argument "I need .gz because my machine
> > is slow" from now on. It simply doesn't hold.
> > 
> 
> I agree, but, IMHO the main argument for keeping .gz is cross-platform
> availability and wide language support, not hardware limitations.  Doing
> a quick google brings up .gz interfaces for every language you can think
> of (C, Java, Perl, Python, TCL etc.), not to mention complete separate
> implementations in Java and Pascal (not just wrappers on top of the zlib
> library), and probably more.
> 
> With xz you have just one C/C++ implementation with a single library with
> an undocumented API for C/C++ programmers.

This can probably be easily explained. gz is very fast decompressing so
it is a very good choice for transparent decompression of files which
must be accessible fast but aren't used frequently. Manual pages or
printer drivers come to mind. bz2 and lzma, OTOH, are meant for longer
term archiving. Their compression ratio benefit is only worth it for
larger files that you don't access that frequently.

I am not claiming that gzip is dead. It is very useful and it is there
to stay for the years to come, no doubt about that. What I'm saying is
that it isn't the best choice for large files to be downloaded from a
remote server.

> It may be a slight stretch of the imagination, but with with .gz you can
> conceive programmers writing programs to download a .gz from kernel.org and
> decompressing/searching it, in almost any language of choice.  With the JAVA
> implementation .gz is genuinely cross platform and you don't need glibc/
> C++ compilers, just a Java VM.  Contrast with xz, where if the xz utility
> isn't available, or doesn't do what you want, you're stuck with programming
> in C/C++ with all the baggage that entails.

Honestly, I don't think we care at all when it comes to the kernel.org
files. Accessing individual files inside a compressed kernel tarball
without first expanding it entirely would be horribly slow and
unpractical, no matter which compression format was used. I can't think
of any case where you won't unpack the tarball first, and for this task
an external tool will do just fine.

And, once again, there are several public instances of gitweb and LXR
available if you only want to browse the code.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Justin P. Mattock]

On 02/14/10 01:23, Jean Delvare wrote:
> On Sat, 13 Feb 2010 23:52:17 +0000, Phillip Lougher wrote:
>> Jean Delvare wrote:
>>
>>>
>>> Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I
>>> think we can plain discard the argument "I need .gz because my machine
>>> is slow" from now on. It simply doesn't hold.
>>>
>>
>> I agree, but, IMHO the main argument for keeping .gz is cross-platform
>> availability and wide language support, not hardware limitations.  Doing
>> a quick google brings up .gz interfaces for every language you can think
>> of (C, Java, Perl, Python, TCL etc.), not to mention complete separate
>> implementations in Java and Pascal (not just wrappers on top of the zlib
>> library), and probably more.
>>
>> With xz you have just one C/C++ implementation with a single library with
>> an undocumented API for C/C++ programmers.
>
> This can probably be easily explained. gz is very fast decompressing so
> it is a very good choice for transparent decompression of files which
> must be accessible fast but aren't used frequently. Manual pages or
> printer drivers come to mind. bz2 and lzma, OTOH, are meant for longer
> term archiving. Their compression ratio benefit is only worth it for
> larger files that you don't access that frequently.
>
> I am not claiming that gzip is dead. It is very useful and it is there
> to stay for the years to come, no doubt about that. What I'm saying is
> that it isn't the best choice for large files to be downloaded from a
> remote server.
>
>> It may be a slight stretch of the imagination, but with with .gz you can
>> conceive programmers writing programs to download a .gz from kernel.org and
>> decompressing/searching it, in almost any language of choice.  With the JAVA
>> implementation .gz is genuinely cross platform and you don't need glibc/
>> C++ compilers, just a Java VM.  Contrast with xz, where if the xz utility
>> isn't available, or doesn't do what you want, you're stuck with programming
>> in C/C++ with all the baggage that entails.
>
> Honestly, I don't think we care at all when it comes to the kernel.org
> files. Accessing individual files inside a compressed kernel tarball
> without first expanding it entirely would be horribly slow and
> unpractical, no matter which compression format was used. I can't think
> of any case where you won't unpack the tarball first, and for this task
> an external tool will do just fine.
>
> And, once again, there are several public instances of gitweb and LXR
> available if you only want to browse the code.
>

just out of curiosity what would happen if by say
I take a file and turn it into .gz then turn the .gz
into .xz or vice versa?

so at the end of the day you have a list of .gz's(or whatever),
then expending on the type(.gz,.bz2,etc..) unpackage and voila either a 
tree or some other compressed file(.bz2,xz, or .gz).

just thinking out loud(so don't shoot me please).

Justin P. Mattock

Re: [kernel.org users] XZ Migration discussion

[Willy Tarreau]

Hi Jean,

On Sun, Feb 14, 2010 at 10:23:08AM +0100, Jean Delvare wrote:
> I am not claiming that gzip is dead. It is very useful and it is there
> to stay for the years to come, no doubt about that. What I'm saying is
> that it isn't the best choice for large files to be downloaded from a
> remote server.

Well, I personally like to be able to simply run "less patch-2.6.27.45.gz"
and have it transparently uncompressed and dumped on my terminal. It
doesn't do that on bz2. We could find multiple examples.

Another thing comes to mind, because I've been beaten by this in the
past. People working in enterprises where the internet access passes
via mandatory proxies coupled with anti-virus can often download many
things but not binaries that can't be analysed. At this time, I could
only download tar.gz but not .bz2. And please don't tell me I have to
go to the admin to tell him to change his proxy's configuration, you
can't do that when you work as a consultant for a 20000 persons group
where products are selected after 6-12 months of testing and managed
by different people from those who qualify them.

In my opinion, .xz is a very good option to replace .bz2 as it will
bring advantages without downsides. But .gz should stay as it has
been available from day 1, at least for all people who may have
trouble with .xz for whatever reason. If it has not been a problem
for the last 16 years, I don't see why it would suddenly become one.

Regards,
Willy

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

Salut Willy,

On Sun, 14 Feb 2010 10:49:40 +0100, Willy Tarreau wrote:
> On Sun, Feb 14, 2010 at 10:23:08AM +0100, Jean Delvare wrote:
> > I am not claiming that gzip is dead. It is very useful and it is there
> > to stay for the years to come, no doubt about that. What I'm saying is
> > that it isn't the best choice for large files to be downloaded from a
> > remote server.
> 
> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz"
> and have it transparently uncompressed and dumped on my terminal. It
> doesn't do that on bz2. We could find multiple examples.

This only underlines what I just wrote: the purpose of gzip is
different from the purpose of bzip2 or lzma. Transparent decompression
makes sense for the former but little for the latter two. But this
doesn't mean everyone must make all their files available in .gz format.
Not every file, and not everyone, needs transparent decompression. It
makes sense on patch files, but not on tarballs. You have the need, but
I don't. Furthermore, the fact that your local usage of patch files is
more convenient if they come in .gz format doesn't imply that this is
the format in which you have to download them. You are free to repack
files after downloading them. This can even be easily automated.

hpa seems to be very reluctant to the idea, but one thing I had in mind
was that patches could be in .gz format and tarballs in .xz, for
example. Having to stick to a common strategy for all the files seems
suboptimal, given their different natures and uses.

(And for completeness: on my distribution, the bzip2 package comes with
a little helper script named bzless, which does exactly what you want
for bzip2-compressed patches. But I wouldn't recommend using it...)

> Another thing comes to mind, because I've been beaten by this in the
> past. People working in enterprises where the internet access passes
> via mandatory proxies coupled with anti-virus can often download many
> things but not binaries that can't be analysed. At this time, I could
> only download tar.gz but not .bz2. And please don't tell me I have to
> go to the admin to tell him to change his proxy's configuration, you
> can't do that when you work as a consultant for a 20000 persons group
> where products are selected after 6-12 months of testing and managed
> by different people from those who qualify them.

I've been working in such environments in the past, so I see what you
mean. And I do not disagree that the format in which projects make
their files available can be more or less convenient in such
situations. For example, I remember being deeply annoyed by projects
not releasing tarballs, because I simply couldn't access CVS
repositories.

That being said, I also don't think that you can put all the burden of
bad company policies on the shoulder of open source software providers.
If someone worked in a company where even .gz compressed files can't be
downloaded, we wouldn't ask kernel.org to provide uncompressed files on
top of .gz and .bz2, would we? There is a trade-off to be found between
flexibility of access and disk and network usage.

It should be possible to retrieve the kernel sources using git over
HTTP these days anyway, right? Or are firewalls frequently blocking
this as well?

> In my opinion, .xz is a very good option to replace .bz2 as it will
> bring advantages without downsides. But .gz should stay as it has
> been available from day 1, at least for all people who may have
> trouble with .xz for whatever reason. If it has not been a problem
> for the last 16 years, I don't see why it would suddenly become one.

People have been using Windows for the last 16 years, it has not been a
problem, so I don't see why it would suddenly become one. Let's stop
working on an alternative. That way we don't have to decide which
compression format to use! Problem solved ;)

Seriously: things can become a problem over time even if they were not
one initially, and needs may evolve as well. The Linux kernel releases
have grown very big compared to the early days, and we release more
often, and new compression technologies exist and are getting widely
adopted. You don't have to stand in front of the wall to declare that
there's a problem. Just improving the user experience is worth the
change sometimes.

That being all said, I don't fundamentally disagree with you: just
replacing .bz2 with .xz would be fine with me. Really, my main concern
is the file count in directory v2.6/, much more than the compression
formats being used.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[James Cloos]

>>>>> "W" == Willy Tarreau <w@1wt.eu> writes:

W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz"
W> and have it transparently uncompressed and dumped on my terminal. It
W> doesn't do that on bz2. We could find multiple examples.

It does here, and lzma & xz, too.  And has since just days after Lasse
annouced that the new name would be xz.

Your LESSOPEN controls that, and can be easily coded to support any archive.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: [kernel.org users] XZ Migration discussion

[Willy Tarreau]

On Mon, Feb 15, 2010 at 04:31:59PM -0500, James Cloos wrote:
> >>>>> "W" == Willy Tarreau <w@1wt.eu> writes:
> 
> W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz"
> W> and have it transparently uncompressed and dumped on my terminal. It
> W> doesn't do that on bz2. We could find multiple examples.
> 
> It does here, and lzma & xz, too.  And has since just days after Lasse
> annouced that the new name would be xz.
> 
> Your LESSOPEN controls that, and can be easily coded to support any archive.

Just checked and I found it funny to see that patch-2.6.1.bz2 is not
correctly opened while 2.6.27.45.bz2 is. Maybe some things have slightly
changed in the tools by that time and the output differs slightly. However
bzcat opens them both so the file is not corrupted.

This raises the point of the maturity of the tools BTW. GZ is mature,
BZ2 has become mature over the years, XZ is very recent and may still
be buggy at times. We'll only know that when people will complain that
they cannot open one file from time to time.

Willy

Re: [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/16/2010 09:40 PM, Willy Tarreau wrote:
> 
> This raises the point of the maturity of the tools BTW. GZ is mature,
> BZ2 has become mature over the years, XZ is very recent and may still
> be buggy at times. We'll only know that when people will complain that
> they cannot open one file from time to time.
> 

This isn't a huge problem.  Since it is generated content we can, if
necessary, run a robot over the archive and verify and regenerate files
that have problems.

	-hpa

Re: [kernel.org users] XZ Migration discussion

[Petri Kaukasoina]

On Wed, Feb 17, 2010 at 06:40:47AM +0100, Willy Tarreau wrote:
> On Mon, Feb 15, 2010 at 04:31:59PM -0500, James Cloos wrote:
> > >>>>> "W" == Willy Tarreau <w@1wt.eu> writes:
> > 
> > W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz"
> > W> and have it transparently uncompressed and dumped on my terminal. It
> > W> doesn't do that on bz2. We could find multiple examples.
> > 
> > It does here, and lzma & xz, too.  And has since just days after Lasse
> > annouced that the new name would be xz.
> > 
> > Your LESSOPEN controls that, and can be easily coded to support any archive.
> 
> Just checked and I found it funny to see that patch-2.6.1.bz2 is not
> correctly opened while 2.6.27.45.bz2 is. Maybe some things have slightly
> changed in the tools by that time and the output differs slightly. However
> bzcat opens them both so the file is not corrupted.
> 
> This raises the point of the maturity of the tools BTW. GZ is mature,
> BZ2 has become mature over the years, XZ is very recent and may still
> be buggy at times. We'll only know that when people will complain that
> they cannot open one file from time to time.
> 
> Willy
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [kernel.org users] XZ Migration discussion

[Petri Kaukasoina]

On Wed, Feb 17, 2010 at 06:40:47AM +0100, Willy Tarreau wrote:
> On Mon, Feb 15, 2010 at 04:31:59PM -0500, James Cloos wrote:
> > >>>>> "W" == Willy Tarreau <w@1wt.eu> writes:
> > 
> > W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz"
> > W> and have it transparently uncompressed and dumped on my terminal. It
> > W> doesn't do that on bz2. We could find multiple examples.
> > 
> > It does here, and lzma & xz, too.  And has since just days after Lasse
> > annouced that the new name would be xz.
> > 
> > Your LESSOPEN controls that, and can be easily coded to support any archive.
> 
> Just checked and I found it funny to see that patch-2.6.1.bz2 is not
> correctly opened while 2.6.27.45.bz2 is.


There is a bug in lesspipe.sh, at least here. It confuses it with compresses
man pages.

--- lesspipe.sh~        2009-11-06 02:14:58.000000000 +0200
+++ lesspipe.sh 2010-02-17 12:19:09.000000000 +0200
@@ -50,6 +50,7 @@

*.1.bz2|*.2.bz2|*.3.bz2|*.4.bz2|*.5.bz2|*.6.bz2|*.7.bz2|*.8.bz2|*.9.bz2|*.n.bz2|*.man.bz2)
# compressed *roff src?
     if bzip2 -dc "$1" | file - | grep roff 1> /dev/null ; then
       bzip2 -dc "$1" | nroff -S -mandoc -
+    else bzip2 -dc "$1"  2>/dev/null
     fi ;;
   *.gz) gzip -dc "$1"  2>/dev/null ;;
   *.bz2) bzip2 -dc "$1" 2>/dev/null ;;

Re: [kernel.org users] XZ Migration discussion

[Andi Kleen]

> With xz you have just one C/C++ implementation with a single library with
> an undocumented API for C/C++ programmers.

... which is not even widely deployed. I did a quick survey and none of 
my systems (none of which terrible old and not particularly embedded) 
have it installed and for most them there's only "lzma-utils" in the 
distribution package repositories which I understand is not compatible.

I would basically need to download the source by hand and install
it like back in the bad old "unix with all useful commands missing"
HP-UX/Solaris/etc. days.

Please keep .gz and better .bz2

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [kernel.org users] XZ Migration discussion

[Jan Engelhardt]

On Sunday 2010-02-14 10:56, Andi Kleen wrote:

>> With xz you have just one C/C++ implementation with a single library with
>> an undocumented API for C/C++ programmers.
>
>... which is not even widely deployed. I did a quick survey and none of 
>my systems (none of which terrible old and not particularly embedded) 
>have it installed and for most them there's only "lzma-utils" in the 
>distribution package repositories which I understand is not compatible.
>
>I would basically need to download the source by hand and install
>it like back in the bad old "unix with all useful commands missing"
>HP-UX/Solaris/etc. days.

When was the last time you compiled a Linux kernel on HP-UX or Solaris?
I think others did that first (me being along the party), and quite
frankly, the plain Solaris without CSW has quickly-reachable limits
even for non-kernels.

Re: [kernel.org users] XZ Migration discussion

[Lasse Collin]

On 2010-02-14 Phillip Lougher wrote:
> With xz you have just one C/C++ implementation with a single library
> with an undocumented API for C/C++ programmers.

I completely agree that language support is bad compared to the .gz 
format, but I think the above sentence makes it sound a bit too bad.

There are three C (no C++ needed) libraries that support the .xz format:
  - LZMA SDK (7-zip.org)
  - XZ Utils has liblzma (tukaani.org)
  - XZ Embedded (tukaani.org, limited support only)

The latter two are more or less based on LZMA SDK, although the code 
looks very different (different coding style, different APIs etc.).

The liblzma API has reference documentation as Doxygen tags in the API 
headers. They aren't the best docs and there's no tutorial or example 
programs yet, but the API certainly isn't undocumented. It has 
similarities to the zlib API, so those used to zlib shouldn't have 
trouble learning the basic features of liblzma, which are enough for 
most people.

There are liblzma bindings for Perl and Python, but I don't know how 
good they are.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Thu, 11 Feb 2010 10:36:03 -0800, J.H. wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hey Everyone,
> 
> So as the subject states this is more a centralized discussion on
> migration plans to using and providing xz for content on kernel.org.
> Currently we provide gz and bz2, with gz acting as the original content
> and kernel.org itself generating the resulting bz2 files.  There are a
> couple of possible proposals and wanted to toss them out there, and get
> feedback from everyone: the kernel community, the mirrors of kernel.org
> and the direct users of kernel.org.

Don't you have download statistics available? If we knew which
compression format is preferred, an by which margin, it would help make
an educated decision.

> ========================================================================
> 
> Option 1)
> 
> Leave gz as the master, and migrate bz2 to xz.  This will happen in
> stages obviously. with bz2 ultimately being phased out.
> 
> 	Migration option 1)
> 
> 		All new content would be provided in .bz2 and .xz with
> 		an ultimate date set that the .bz2 files would stop
> 		being generated with new content.  This would leave all
> 		existing content alone and it would not be a migration
> 		of the current .bz2 files to xz
> 
> 	Migration option 2)
> 
> 		At some point there would be a mass conversion of all
> 		existing content to include .bz2 and .xz.  These would
> 		be run in parallel for a time period until it was
> 		determined that .bz2 was no longer needed and it would
> 		be removed from the servers leaving .gz and .xz
> 
> Option 2)
> 
> Convert the master data from gz to bz2 and use xz as the new file
> format.  This has the downside of causing more tool churn as it means
> the kernel developers will have to eventually convert from gz to bz2,
> which means for a time there will be nag e-mails if you upload gz
> instead of bz2 and such.  It would also mean that we (kernel.org) would
> need to be able to support .gz and .bz2 as master data for a time.
> 
> Migration options are identical to Option 1 more or less, with either
> just new content getting converted, or all content getting converted.
> 
> ========================================================================
> 
> I'm personally leaning towards option 1, though personally don't really
> have a preference on the migration options, as both obviously offer
> different advantages, and again this e-mail is more to spur on the
> discussion and come to some general consensus across all of the groups
> concerned before moving forward with a more specific plan.
> 
> So I'm inviting discussion, questions and comments on this so we know
> which way to ultimately go.

Maybe that's just me, but my main concern is neither download times nor
decompression times. My main concern is the access time to directory
indexes when browsing the kernel archive, because there are 5 entries
for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign.
This is horribly slow. The main directory for 2.6 kernels has an index
weighting over 300 kB raw, turning into a ~600 kB document when
HTML-ized. Just fetching it takes 3 seconds and then my browser takes a
long time to format it. There are 3881 entries in that directory today,
and it keeps growing!

So, once we have settled for a compression strategy, I think it would
be the right time to discuss the directory structure. With the advent
of the stable branches and the new development model - which pretty
much implies that we'll live with main version 2.6 forever - the file
count is much higher than it used to me.

I can think of several ways to improve the situation here, some of
which could be combined.

1* Keep a single compression format. This saves almost 40% of the
files.

2* Move one of the compression formats somewhere else, so that it
doesn't get in the way but is still available if needed.

3* Create a new subdirectory for every 2.6.x kernel, and move all the
related files there. This would shrink the main index drastically, and
each subdirectory would have a reasonable size (except maybe 2.6.16 and
2.6.27.) Oddly enough this has been done for the files under testing/
already, so I am curious why we don't do it for the release files (and
the testing/incr/ files, while we're at it.)

4* Get rid of the LATEST-IS-* files. This is a small count, won't save
much, but these files seem totally useless to me these days. Depending
on what you want exactly, there are many versions which can be
considered the latest, and there are better ways to know which they are
(for example http://www.eu.kernel.org/kdist/finger_banner ). And these
files tend to get stuck so you can't rely on them anyway.

I wouldn't worry too much about breaking the current locations. Just
give some time for software authors (ketchup comes to mind) to update
their code and it shouldn't be a big problem.

Thanks,
-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Linus Torvalds]

On Fri, 12 Feb 2010, Jean Delvare wrote:
> 
> Maybe that's just me, but my main concern is neither download times nor
> decompression times. My main concern is the access time to directory
> indexes when browsing the kernel archive, because there are 5 entries
> for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign.
> This is horribly slow.

This was actually the main reason for me personally to ask about just 
dropping support for .gz files - not because I care deeply about how much 
disk space kernel.org wastes, but because the long directory listings make 
it slower for me to mentally index the directory.

> 1* Keep a single compression format. This saves almost 40% of the
> files.
> 
> 2* Move one of the compression formats somewhere else, so that it
> doesn't get in the way but is still available if needed.
> 
> 3* Create a new subdirectory for every 2.6.x kernel, and move all the
> related files there.

I did 3* for the testing kernels (exactly because the directory listing 
got to be unreadable), and you just complained about it ;)

Of course, your complaint was that the subdirectory wasn't done 
immediately, and that the old files get moved to their own subdirectory 
later as a "archival" thing.

I just didn't want to change the location for the latest kernel.

> 4* Get rid of the LATEST-IS-* files. This is a small count, won't save
> much, but these files seem totally useless to me these days.

Yeah, they also end up continually being stale.

			Linus

Re: [kernel.org users] XZ Migration discussion

[J.H.]

On 02/12/2010 07:21 AM, Linus Torvalds wrote:
> 
> 
> On Fri, 12 Feb 2010, Jean Delvare wrote:
>>
>> Maybe that's just me, but my main concern is neither download times nor
>> decompression times. My main concern is the access time to directory
>> indexes when browsing the kernel archive, because there are 5 entries
>> for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign.
>> This is horribly slow.
> 
> This was actually the main reason for me personally to ask about just 
> dropping support for .gz files - not because I care deeply about how much 
> disk space kernel.org wastes, but because the long directory listings make 
> it slower for me to mentally index the directory.

It's probably worth keeping things like the .gz files around, if nothing
else for older distros, systems, etc that don't have xz yet (since it's
still relatively new)

Breaking things out into directories or such might be the easiest way
with that

v2.6/
v2.6/2.6.23
v2.6/2.6.32.6

etc

Would clean up the v2.6 directory a lot.

>> 1* Keep a single compression format. This saves almost 40% of the
>> files.
>>
>> 2* Move one of the compression formats somewhere else, so that it
>> doesn't get in the way but is still available if needed.
>>
>> 3* Create a new subdirectory for every 2.6.x kernel, and move all the
>> related files there.
> 
> I did 3* for the testing kernels (exactly because the directory listing 
> got to be unreadable), and you just complained about it ;)
> 
> Of course, your complaint was that the subdirectory wasn't done 
> immediately, and that the old files get moved to their own subdirectory 
> later as a "archival" thing.
> 
> I just didn't want to change the location for the latest kernel.
> 
>> 4* Get rid of the LATEST-IS-* files. This is a small count, won't save
>> much, but these files seem totally useless to me these days.
> 
> Yeah, they also end up continually being stale.

Only thoughts there are that there seem to be a lot of automated
processes that rely on LATEST-IS-*.  Personally I'd rather see them snag
the RSS feed and figure out what they want from there, but that may not
be completely feasible.  It also gives a quick indication as to what's
latest in the directory and a quick search on the page usually gets me
what I'm looking for when I do it.

- John 'Warthog9' Hawley

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Fri, 12 Feb 2010 11:02:52 -0800, J.H. wrote:
> On 02/12/2010 07:21 AM, Linus Torvalds wrote:
> > 
> > 
> > On Fri, 12 Feb 2010, Jean Delvare wrote:
> >>
> >> Maybe that's just me, but my main concern is neither download times nor
> >> decompression times. My main concern is the access time to directory
> >> indexes when browsing the kernel archive, because there are 5 entries
> >> for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign.
> >> This is horribly slow.
> > 
> > This was actually the main reason for me personally to ask about just 
> > dropping support for .gz files - not because I care deeply about how much 
> > disk space kernel.org wastes, but because the long directory listings make 
> > it slower for me to mentally index the directory.
> 
> It's probably worth keeping things like the .gz files around, if nothing
> else for older distros, systems, etc that don't have xz yet (since it's
> still relatively new)

Hardly a good reason IMHO. xz can be installed on these systems. When
we switched to git, nobody had it and it did not stop us.

> 
> Breaking things out into directories or such might be the easiest way
> with that
> 
> v2.6/
> v2.6/2.6.23

Yes.

> v2.6/2.6.32.6

I'd rather group all 2.6.32.* files together, so that the main index is
as small as possible. We're adding one indirection step, so it should
be fast.

> 
> etc
> 
> Would clean up the v2.6 directory a lot.

> >> (...)
> >> 4* Get rid of the LATEST-IS-* files. This is a small count, won't save
> >> much, but these files seem totally useless to me these days.
> > 
> > Yeah, they also end up continually being stale.
> 
> Only thoughts there are that there seem to be a lot of automated
> processes that rely on LATEST-IS-*.

Care to give details? Given how often the old files get stuck, I am
surprised any process can really rely on them. And I also can't really
think of any automated process that would care.

>  Personally I'd rather see them snag
> the RSS feed and figure out what they want from there, but that may not
> be completely feasible.

There's RSS, there's a mailing list and there's a web page. Certainly
one of these 3 methods would work.

>  It also gives a quick indication as to what's
> latest in the directory

Sorting by time works just as well.

> and a quick search on the page usually gets me
> what I'm looking for when I do it.

What's your workflow? I normally go to the download directory after
either reading the kernel.org main page or some post on the announce
mailing list. So I already know which version I am looking for. Having
to search for "LATEST-IS" and then again for the version doesn't look
terribly efficient.

If we really want a helper to locate the latest version, I'd rather
have a "latest" symbolic link pointing to the most recent v2.6.x
subdirectory. Or maybe two, "latest-stable" and "latest-devel". Can be
discussed...

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Willy Tarreau]

On Fri, Feb 12, 2010 at 08:23:57PM +0100, Jean Delvare wrote:
> > It's probably worth keeping things like the .gz files around, if nothing
> > else for older distros, systems, etc that don't have xz yet (since it's
> > still relatively new)
> 
> Hardly a good reason IMHO. xz can be installed on these systems. When
> we switched to git, nobody had it and it did not stop us.

I don't agree, it's different. Git is only used by developers, and even
not all of them. Sources are a reference. Anyone can download them to
look for anything. Switching to a specific format which really is not
common at all on older distros nor on any system looks a bit like
proprietary encoding eventhough it's not the case. But it's a way to
tell people that if they want the sources in clear text form, they
first have to find a tool capable of decompressing them. Gzip is well
defined as a standard, it's even described in an RFC and is present
on almost any system (unix or not) now. Any student who wants to take
a look at the kernel will have access to gunzip, even from an old
Solaris 8 workstation or a Windows XP desktop PC. XZ if far from
being there, and the student will not necessarily be able to install
it. And Peter raised some valid points about the hardware requirements
to run such tools ; I'm not sure the guys running Linux on their old
Sparc-2 would like XZ only a lot.

Regards,
Willy

Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

Hi!

> > > It's probably worth keeping things like the .gz files around, if nothing
> > > else for older distros, systems, etc that don't have xz yet (since it's
> > > still relatively new)
> > 
> > Hardly a good reason IMHO. xz can be installed on these systems. When
> > we switched to git, nobody had it and it did not stop us.
> 
> I don't agree, it's different. Git is only used by developers, and even
> not all of them. Sources are a reference. Anyone can download them to
> look for anything. Switching to a specific format which really is not
> common at all on older distros nor on any system looks a bit like
> proprietary encoding eventhough it's not the case. But it's a way to
> tell people that if they want the sources in clear text form, they
> first have to find a tool capable of decompressing them. Gzip is well
> defined as a standard, it's even described in an RFC and is present
> on almost any system (unix or not) now. Any student who wants to take
> a look at the kernel will have access to gunzip, even from an old
> Solaris 8 workstation or a Windows XP desktop PC. XZ if far from
> being there, and the student will not necessarily be able to install
> it. And Peter raised some valid points about the hardware requirements
> to run such tools ; I'm not sure the guys running Linux on their old
> Sparc-2 would like XZ only a lot.

It is not just student on old workstation. I'm trying to keep Linux
working on spitz PDA and kohjinsha subnotebook. Especially zaurus has
about power of old sparc two... otoh it runs 4 hours on cellphone
battery.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [kernel.org users] XZ Migration discussion

[Stefan Richter]

Pavel Machek wrote:
>> And Peter raised some valid points about the hardware requirements
>> to run such tools ; I'm not sure the guys running Linux on their old
>> Sparc-2 would like XZ only a lot.
> 
> It is not just student on old workstation. I'm trying to keep Linux
> working on spitz PDA and kohjinsha subnotebook. Especially zaurus has
> about power of old sparc two...

Do you actually require to download and unpack (and configure, build)
the kernel sources from kernel.org to the PDA directly?  As Jean wrote,
how does unpack time matter compared to build time?
-- 
Stefan Richter
-=====-==-=- --=- -==-=
http://arcgraph.de/sr/

Re: [kernel.org users] XZ Migration discussion

[Stefan Richter]

Stefan Richter wrote:
> Pavel Machek wrote:
>>> And Peter raised some valid points about the hardware requirements
>>> to run such tools ; I'm not sure the guys running Linux on their old
>>> Sparc-2 would like XZ only a lot.
>> It is not just student on old workstation. I'm trying to keep Linux
>> working on spitz PDA and kohjinsha subnotebook. Especially zaurus has
>> about power of old sparc two...
> 
> Do you actually require to download and unpack (and configure, build)
> the kernel sources from kernel.org to the PDA directly?  As Jean wrote,
> how does unpack time matter compared to build time?

PS:  It boils down to CPU time requirements.  For small memory systems,
there is the XZ Embedded decompressor.
-- 
Stefan Richter
-=====-==-=- --=- -==-=
http://arcgraph.de/sr/

Re: [kernel.org users] XZ Migration discussion

[Lasse Collin]

On 2010-02-13 Stefan Richter wrote:
> PS:  It boils down to CPU time requirements.  For small memory
> systems, there is the XZ Embedded decompressor.

XZ Embedded saves only code size. If the file was compressed with 
settings that make the decompressor require e.g. 65 MiB RAM (xz -9), 
using XZ Embedded won't help you.

XZ Embedded is also very limited. It cannot decompress all .xz files. It 
should still be quite useful, since on some architectures it can 
compress the kernel image better than the current LZMA support in the 
kernel, and the API should be nice to use e.g. by Squashfs.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

Salut Willy,

On Sat, 13 Feb 2010 00:07:02 +0100, Willy Tarreau wrote:
> On Fri, Feb 12, 2010 at 08:23:57PM +0100, Jean Delvare wrote:
> > > It's probably worth keeping things like the .gz files around, if nothing
> > > else for older distros, systems, etc that don't have xz yet (since it's
> > > still relatively new)
> > 
> > Hardly a good reason IMHO. xz can be installed on these systems. When
> > we switched to git, nobody had it and it did not stop us.
> 
> I don't agree, it's different. Git is only used by developers, and even
> not all of them. Sources are a reference. Anyone can download them to
> look for anything. Switching to a specific format which really is not
> common at all on older distros nor on any system looks a bit like
> proprietary encoding eventhough it's not the case. But it's a way to
> tell people that if they want the sources in clear text form, they
> first have to find a tool capable of decompressing them.

Just like switching to git was a way to tell people that if they wanted
to contribute to the kernel, they first had to install the right, at
the time uncommon tool. Of course the audience isn't the same, but the
principle is similar. And the audience is still fairly limited in both
cases. My parents aren't downloading kernel tarballs. I would assume
that anyone willing and able to download, read and understand the linux
kernel source code wouldn't be frightened by having to install a small
tool to extract these sources. And they may not even have to do this:
sources can be read with just a web browser: we have the gitweb
interface, and several public LXR installations are deployed as well.
These days, web browsers are much more popular than ftp clients and
tarballs.

As a matter of fact, I am advocating the use of xz while I don't have
it installed on most of my machines. I really don't see this as a
blocker.

> Gzip is well
> defined as a standard, it's even described in an RFC and is present
> on almost any system (unix or not) now. Any student who wants to take
> a look at the kernel will have access to gunzip, even from an old
> Solaris 8 workstation or a Windows XP desktop PC.

Really? I have a Windows XP laptop at hand and it can't read .gz files.
If I ask it to try, it tells me I should install WinZip. I also seem to
recall that I had to install GNU gzip myself back when I was working on
a Solaris workstation (but I might remember badly.)

> XZ if far from
> being there, and the student will not necessarily be able to install
> it. And Peter raised some valid points about the hardware requirements
> to run such tools ; I'm not sure the guys running Linux on their old
> Sparc-2 would like XZ only a lot.

I don't quite buy this argument either. I suspect this is a very
limited count of users, and these users have access to other, more
powerful machines where they can easily achieve any format conversion
they need.

I have an old, slow machine here which I am going to use to perform
some real world testing, and I'll post the results when I'm done. But I
suspect that building a kernel on this machine, even a small one with
just the drivers it needs, will take much longer than unpacking the
sources. So anyone worrying about performance would rather rely on
cross-compilation, and in turn can afford whatever decompression tool
is needed.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Stefan Richter]

Jean Delvare wrote:
> On Sat, 13 Feb 2010 00:07:02 +0100, Willy Tarreau wrote:
>> Gzip is well
>> defined as a standard, it's even described in an RFC and is present
>> on almost any system (unix or not) now. Any student who wants to take
>> a look at the kernel will have access to gunzip, even from an old
>> Solaris 8 workstation or a Windows XP desktop PC.
> 
> Really? I have a Windows XP laptop at hand and it can't read .gz files.
> If I ask it to try, it tells me I should install WinZip. I also seem to
> recall that I had to install GNU gzip myself back when I was working on
> a Solaris workstation (but I might remember badly.)

As a side note:  7zip, a very popular and Free archiving tool for
Windows, supports xz since its version 9 which is currently available as
a beta.  So, WRT the need to get an extra unarchiver, xz is just as
accessible to Windows users as gz is.
-- 
Stefan Richter
-=====-==-=- --=- -==-=
http://arcgraph.de/sr/

Re: [kernel.org users] XZ Migration discussion

[Avi Kivity]

On 02/13/2010 11:59 AM, Stefan Richter wrote:
> Jean Delvare wrote:
>    
>> On Sat, 13 Feb 2010 00:07:02 +0100, Willy Tarreau wrote:
>>      
>>> Gzip is well
>>> defined as a standard, it's even described in an RFC and is present
>>> on almost any system (unix or not) now. Any student who wants to take
>>> a look at the kernel will have access to gunzip, even from an old
>>> Solaris 8 workstation or a Windows XP desktop PC.
>>>        
>> Really? I have a Windows XP laptop at hand and it can't read .gz files.
>> If I ask it to try, it tells me I should install WinZip. I also seem to
>> recall that I had to install GNU gzip myself back when I was working on
>> a Solaris workstation (but I might remember badly.)
>>      
> As a side note:  7zip, a very popular and Free archiving tool for
> Windows, supports xz since its version 9 which is currently available as
> a beta.  So, WRT the need to get an extra unarchiver, xz is just as
> accessible to Windows users as gz is.
>    

Can you even unpack a kernel tree on Windows?  There are some files 
(e.g. net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a 
case-preserving filesystem.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [kernel] Re: [kernel.org users] XZ Migration discussion

[Mr. James W. Laferriere]

On Sat, 13 Feb 2010, Avi Kivity wrote:

> On 02/13/2010 11:59 AM, Stefan Richter wrote:
>> Jean Delvare wrote:
...snip...
>> As a side note:  7zip, a very popular and Free archiving tool for
>> Windows, supports xz since its version 9 which is currently available as
>> a beta.  So, WRT the need to get an extra unarchiver, xz is just as
>> accessible to Windows users as gz is.
>
> Can you even unpack a kernel tree on Windows?  There are some files (e.g. 
> net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a case-preserving 
> filesystem.
 	What filesystem version are you talking about ?
 	ntfs as of recent windows is not case insensitive .
 	What do you mean by , "a case-preserving filesystem" ?

 	Now whether or not the tools available are case insensitive is another 
matter .

 		Hth ,  JimL
-- 
+------------------------------------------------------------------+
| James   W.   Laferriere | System    Techniques | Give me VMS     |
| Network&System Engineer | 3237     Holden Road |  Give me Linux  |
| babydr@baby-dragons.com | Fairbanks, AK. 99709 |   only  on  AXP |
+------------------------------------------------------------------+

Re: [kernel] Re: [kernel.org users] XZ Migration discussion

[Bernd Petrovitsch]

On Sam, 2010-02-13 at 12:37 -0900, Mr. James W. Laferriere wrote:
> On Sat, 13 Feb 2010, Avi Kivity wrote:
[...]
> > Can you even unpack a kernel tree on Windows?  There are some files (e.g. 
Given a sane filesystem, yes. If (the) native filesystem(s) are sane, is
more a flamebait than anything else.

> > net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a case-preserving 
> > filesystem.
>  	What filesystem version are you talking about ?
All of NTFS (and basically FAT*). Are there are different versions? If
yes, how do I tell?

>  	ntfs as of recent windows is not case insensitive .
It *is* case insensitive (as it compares filesystem names
case-insensitive on the equivalent of an open() syscall).

>  	What do you mean by , "a case-preserving filesystem" ?
Apart from "please google that, thank you": The filesystem (driver))
preserves the case of a filename if you create it (TTBOMK). But it
ignores the case of the filename for other file operations like open(),
stat(), etc.

>  	Now whether or not the tools available are case insensitive is another 
> matter .
ACK. But if you have a case-insensitive or case-preserving filesystem,
the tools can´t really do that much.
And given the "understanding" of the monopolist (as such) in that area,
most of the "tools" are actually inherent part of the OS (and not just
another 3rd party app). SCNR ....

	Bernd
-- 
Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
                     LUGA : http://www.luga.at

Re: [kernel.org users] [kernel] Re: XZ Migration discussion

[Steve French]

On Sat, Feb 13, 2010 at 4:39 PM, Bernd Petrovitsch
<bernd@petrovitsch.priv.at> wrote:
> On Sam, 2010-02-13 at 12:37 -0900, Mr. James W. Laferriere wrote:
>> On Sat, 13 Feb 2010, Avi Kivity wrote:
> [...]
>> > Can you even unpack a kernel tree on Windows?  There are some files (e.g.
> Given a sane filesystem, yes. If (the) native filesystem(s) are sane, is
> more a flamebait than anything else.
>
>> > net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a case-preserving
>> > filesystem.
>>       What filesystem version are you talking about ?
> All of NTFS (and basically FAT*). Are there are different versions? If
> yes, how do I tell?
>
>>       ntfs as of recent windows is not case insensitive .
> It *is* case insensitive (as it compares filesystem names
> case-insensitive on the equivalent of an open() syscall).

There are case sensitive Windows subsystems (e.g. SFU/SUA
at least when running over NTFS), and the default behavior for
Win32 apps even can be changed to be case sensitive
via a registry key: ObCaseInsensitive).

More important is how easy it is to install - since XZ is
not even available via apt-get install on recent
distros (e.g. April 2009 Ubuntu 9.04), this discussion
seems about a year premature.

-- 
Thanks,

Steve

Re: [kernel.org users] [kernel] Re: XZ Migration discussion

[Bernd Petrovitsch]

On Mon, 2010-02-15 at 13:33 -0600, Steve French wrote:
> On Sat, Feb 13, 2010 at 4:39 PM, Bernd Petrovitsch
> <bernd@petrovitsch.priv.at> wrote:
[...]
> >>       ntfs as of recent windows is not case insensitive .
> > It *is* case insensitive (as it compares filesystem names
> > case-insensitive on the equivalent of an open() syscall).
> 
> There are case sensitive Windows subsystems (e.g. SFU/SUA
> at least when running over NTFS), and the default behavior for

> Win32 apps even can be changed to be case sensitive
> via a registry key: ObCaseInsensitive).
It's somewhat - ähemm - strange IMHO that the casing is an app-specific
feature (and not filesystem specific).
And it is really that implemented that the app can choose in what way
the filesystem below - given that it supports that feature - compares
two filenames?

> More important is how easy it is to install - since XZ is
> not even available via apt-get install on recent
> distros (e.g. April 2009 Ubuntu 9.04), this discussion
> seems about a year premature.
It's in recent Fedoras so Ubuntu is perhaps just late. And FWIW, it' 2
packages too as the second one is the .lzma support (so probably Debian
should be able to fix the clash with whatever the current lzma package
is).

	Bernd
-- 
Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
                     LUGA : http://www.luga.at

Re: [kernel.org users] [kernel] Re: XZ Migration discussion

[J.H.]

On 02/16/2010 01:16 AM, Bernd Petrovitsch wrote:
> On Mon, 2010-02-15 at 13:33 -0600, Steve French wrote:
>> On Sat, Feb 13, 2010 at 4:39 PM, Bernd Petrovitsch
>> <bernd@petrovitsch.priv.at> wrote:
> [...]
>>>>       ntfs as of recent windows is not case insensitive .
>>> It *is* case insensitive (as it compares filesystem names
>>> case-insensitive on the equivalent of an open() syscall).
>>
>> There are case sensitive Windows subsystems (e.g. SFU/SUA
>> at least when running over NTFS), and the default behavior for
> 
>> Win32 apps even can be changed to be case sensitive
>> via a registry key: ObCaseInsensitive).
> It's somewhat - ähemm - strange IMHO that the casing is an app-specific
> feature (and not filesystem specific).
> And it is really that implemented that the app can choose in what way
> the filesystem below - given that it supports that feature - compares
> two filenames?
> 
>> More important is how easy it is to install - since XZ is
>> not even available via apt-get install on recent
>> distros (e.g. April 2009 Ubuntu 9.04), this discussion
>> seems about a year premature.
> It's in recent Fedoras so Ubuntu is perhaps just late. And FWIW, it' 2
> packages too as the second one is the .lzma support (so probably Debian
> should be able to fix the clash with whatever the current lzma package
> is).
> 
> 	Bernd

There's a package on Fedora called 'xz' that provides it on my Fedora 11
& 12 boxes.

- John 'Warthog9' Hawley

Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

On Sat 2010-02-13 10:59:44, Stefan Richter wrote:
> Jean Delvare wrote:
> > On Sat, 13 Feb 2010 00:07:02 +0100, Willy Tarreau wrote:
> >> Gzip is well
> >> defined as a standard, it's even described in an RFC and is present
> >> on almost any system (unix or not) now. Any student who wants to take
> >> a look at the kernel will have access to gunzip, even from an old
> >> Solaris 8 workstation or a Windows XP desktop PC.
> > 
> > Really? I have a Windows XP laptop at hand and it can't read .gz files.
> > If I ask it to try, it tells me I should install WinZip. I also seem to
> > recall that I had to install GNU gzip myself back when I was working on
> > a Solaris workstation (but I might remember badly.)
> 
> As a side note:  7zip, a very popular and Free archiving tool for
> Windows, supports xz since its version 9 which is currently available as
> a beta.  So, WRT the need to get an extra unarchiver, xz is just as
> accessible to Windows users as gz is.

Wow, what a stretch.

There's about milion tools supporting gz for Windows, and many of them
are out of beta... xz is not even supported on my zaurus, and I'll
need to write some emails to get it installed on school servers...

Anyway, gz+xz should be reasonable combination.
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [kernel.org users] XZ Migration discussion

[Stefan Richter]

Pavel Machek wrote:
> On Sat 2010-02-13 10:59:44, Stefan Richter wrote:
>> As a side note:  7zip, a very popular and Free archiving tool for
>> Windows, supports xz since its version 9 which is currently available as
>> a beta.  So, WRT the need to get an extra unarchiver, xz is just as
>> accessible to Windows users as gz is.
> 
> Wow, what a stretch.
> 
> There's about milion tools supporting gz for Windows, and many of them
> are out of beta...

And roughly half of this million archiving/ compression tools use 7zip's
backend library, which is just one way of getting xz support in a
Windows application.  Besides, what one project calls beta is called
dot-zero release elsewhere. :-)  Seriously, getting xz support on
Windows is just as easy or as difficult as getting gz support.
-- 
Stefan Richter
-=====-==-=- --=- -===-
http://arcgraph.de/sr/

Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

On Sun 2010-02-14 18:33:49, Stefan Richter wrote:
> Pavel Machek wrote:
> > On Sat 2010-02-13 10:59:44, Stefan Richter wrote:
> >> As a side note:  7zip, a very popular and Free archiving tool for
> >> Windows, supports xz since its version 9 which is currently available as
> >> a beta.  So, WRT the need to get an extra unarchiver, xz is just as
> >> accessible to Windows users as gz is.
> > 
> > Wow, what a stretch.
> > 
> > There's about milion tools supporting gz for Windows, and many of them
> > are out of beta...
> 
> And roughly half of this million archiving/ compression tools use 7zip's
> backend library, which is just one way of getting xz support in a
> Windows application.  Besides, what one project calls beta is called
> dot-zero release elsewhere. :-)  Seriously, getting xz support on
> Windows is just as easy or as difficult as getting gz support.

Sorry, but I just don't think that's true. No more windows systems
here. On android, gzip is supported, xzip is not. Debian machine
supports both, but gzip was preinstalled and I had to pull xzip. (This
does not help either:

root@amd:~# apt-cache search xzip
xzip - Interpreter of Infocom-format story-files

)									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [kernel.org users] XZ Migration discussion

[Stefan Richter]

Pavel Machek wrote:
> On Sun 2010-02-14 18:33:49, Stefan Richter wrote:
>> Seriously, getting xz support on
>> Windows is just as easy or as difficult as getting gz support.
> 
> Sorry, but I just don't think that's true.

Although it is. :-)

> No more windows systems
> here. On android, gzip is supported, xzip is not. Debian machine
> supports both, but gzip was preinstalled and I had to pull xzip. (This
> does not help either:
> 
> root@amd:~# apt-cache search xzip
> xzip - Interpreter of Infocom-format story-files

Considering that xz support is available even on niche systems like
Amiga OS and BeOS (via p7zip if not by other means) and xz-utils proper
build even on DOS, OpenVMS and other systems, how hard can it be to
obtain an xz decompressor on Android, Debian, or Ubuntu¹?  (¹About which
there was a note somewhere else in this thread that there is a conflict
between xz-utils and lzma-utils... That's basically because the former
supersede the latter.)

The name confusion between xz-utils and xzip can be avoided if you
search for the package in a package manager which shows package
categories (archivers vs. games).
-- 
Stefan Richter
-=====-==-=- --=- -====
http://arcgraph.de/sr/

Re: [kernel.org users] XZ Migration discussion

[Tom Rini]

On Mon, Feb 15, 2010 at 09:48:21AM +0100, Stefan Richter wrote:
> Pavel Machek wrote:
[snip]
> > No more windows systems
> > here. On android, gzip is supported, xzip is not. Debian machine
> > supports both, but gzip was preinstalled and I had to pull xzip. (This
> > does not help either:
> > 
> > root@amd:~# apt-cache search xzip
> > xzip - Interpreter of Infocom-format story-files
> 
> Considering that xz support is available even on niche systems like
> Amiga OS and BeOS (via p7zip if not by other means) and xz-utils proper
> build even on DOS, OpenVMS and other systems, how hard can it be to
> obtain an xz decompressor on Android, Debian, or Ubuntu¹?  (¹About which
> there was a note somewhere else in this thread that there is a conflict
> between xz-utils and lzma-utils... That's basically because the former
> supersede the latter.)

This, I think gets to one of the problems.  You're telling me that the
p7zip thing I installed for work is this .xz thing?  And it's really all
this LZMA algo?  That's part of the transition problem, folks quite
likely have access, easily, to a decompressor but don't know what it is
(.gz->gzip, .bz2 -> bzip2, .xz -> {p7zip, p7zip-full, xz-utils, ???}).

In fact, why not .lzma?  I'm assuming we're talking about the format,
and xz-utils nad p7zip and others all implement the same format and it's
all compatible and it's just a "how do we get there quickest / fastest"
kind of thing between the utils.

> The name confusion between xz-utils and xzip can be avoided if you
> search for the package in a package manager which shows package
> categories (archivers vs. games).

Actually, no, there's no 'xz-utils' in Debian/Lenny or Ubuntu 9.04, but
there is in 9.10.  But in 9.10, searching on xzip (a good, but
incorrect guess) only gives that.  Searching on xz shows xz-utils too.
At least with apt-cache, and since the desc doesn't list xzip (since
it's not xzip, it's an incorrect but not illogical guess), that
wouldn't help.

-- 
Tom Rini

Re: [kernel.org users] XZ Migration discussion

[Stefan Richter]

Tom Rini wrote:
> That's part of the transition problem, folks quite
> likely have access, easily, to a decompressor but don't know what it is

Naturally.  Information on it is quickly found online though.
-- 
Stefan Richter
-=====-==-=- --=- =----
http://arcgraph.de/sr/

Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

Hi!

> As a matter of fact, I am advocating the use of xz while I don't have
> it installed on most of my machines. I really don't see this as a
> blocker.

Eh?

Making many people around the world install uncommon tool is not
something that should be done lightly.

> I have an old, slow machine here which I am going to use to perform
> some real world testing, and I'll post the results when I'm done. But I
> suspect that building a kernel on this machine, even a small one with
> just the drivers it needs, will take much longer than unpacking the
> sources. So anyone worrying about performance would rather rely on
> cross-compilation, and in turn can afford whatever decompression tool
> is needed.

On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So
that one is ... well ... done overnight.

Untar is something I normally wait for, since you need to run
(interactive) oldconfig after that.
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [kernel.org users] XZ Migration discussion

[Stefan Richter]

Pavel Machek wrote:
[Jean Delvare wrote]
>> I have an old, slow machine here which I am going to use to perform
>> some real world testing, and I'll post the results when I'm done. But I
>> suspect that building a kernel on this machine, even a small one with
>> just the drivers it needs, will take much longer than unpacking the
>> sources. So anyone worrying about performance would rather rely on
>> cross-compilation, and in turn can afford whatever decompression tool
>> is needed.
> 
> On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So
> that one is ... well ... done overnight.
> 
> Untar is something I normally wait for, since you need to run
> (interactive) oldconfig after that.

If the download takes m minutes and unpacking and unarchiving tar.gz
takes another n minutes, what difference does it make for this workflow
when instead m + p minutes are spent for download + unpacking and
unarchiving tar.bz2 or tar.xz?
-- 
Stefan Richter
-=====-==-=- --=- -===-
http://arcgraph.de/sr/

Re: [kernel.org users] XZ Migration discussion

[david]

On Sun, 14 Feb 2010, Stefan Richter wrote:

> Pavel Machek wrote:
> [Jean Delvare wrote]
>>> I have an old, slow machine here which I am going to use to perform
>>> some real world testing, and I'll post the results when I'm done. But I
>>> suspect that building a kernel on this machine, even a small one with
>>> just the drivers it needs, will take much longer than unpacking the
>>> sources. So anyone worrying about performance would rather rely on
>>> cross-compilation, and in turn can afford whatever decompression tool
>>> is needed.
>>
>> On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So
>> that one is ... well ... done overnight.
>>
>> Untar is something I normally wait for, since you need to run
>> (interactive) oldconfig after that.
>
> If the download takes m minutes and unpacking and unarchiving tar.gz
> takes another n minutes, what difference does it make for this workflow
> when instead m + p minutes are spent for download + unpacking and
> unarchiving tar.bz2 or tar.xz?

The difference is that the user is waiting for the n or m minutes, but 
goes off and does something else for the p minutes.

So it doesn't really matter if the compile takes 1 hour or 6 hours. As 
Pavel noted, this is done overnight and it doesn't matter if it gets done 
at 2am or 6am.

But the uncompression has a user waiting for it (to do the config step), 
so here it makes a big difference if it takes 6 minutes ot 8 minutes.

David Lang

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Sun, 14 Feb 2010 13:04:45 -0800 (PST), david@lang.hm wrote:
> On Sun, 14 Feb 2010, Stefan Richter wrote:
> 
> > Pavel Machek wrote:
> >> On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So
> >> that one is ... well ... done overnight.
> >>
> >> Untar is something I normally wait for, since you need to run
> >> (interactive) oldconfig after that.
> >
> > If the download takes m minutes and unpacking and unarchiving tar.gz
> > takes another n minutes, what difference does it make for this workflow
> > when instead m + p minutes are spent for download + unpacking and
> > unarchiving tar.bz2 or tar.xz?
> 
> The difference is that the user is waiting for the n or m minutes, but 
> goes off and does something else for the p minutes.
> 
> So it doesn't really matter if the compile takes 1 hour or 6 hours. As 
> Pavel noted, this is done overnight and it doesn't matter if it gets done 
> at 2am or 6am.
> 
> But the uncompression has a user waiting for it (to do the config step), 
> so here it makes a big difference if it takes 6 minutes ot 8 minutes.

You totally missed Stefan's point. Read again.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/14/2010 09:07 AM, Pavel Machek wrote:
> Hi!
> 
>> As a matter of fact, I am advocating the use of xz while I don't have
>> it installed on most of my machines. I really don't see this as a
>> blocker.
> 
> Eh?
> 
> Making many people around the world install uncommon tool is not
> something that should be done lightly.
> 

Indeed.

Let me make this straight before we waste more breath on this: we're not
going to go to xz only at this time.  Heck, we were getting pushback on
the idea of going to bz2 only *ten years* after it was deployed.

	-hpa

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Sun, 14 Feb 2010 18:07:24 +0100, Pavel Machek wrote:
> Hi!
> 
> > As a matter of fact, I am advocating the use of xz while I don't have
> > it installed on most of my machines. I really don't see this as a
> > blocker.
> 
> Eh?
> 
> Making many people around the world install uncommon tool is not
> something that should be done lightly.

It's pretty obvious that xz will become popular quickly, at least on
Linux and BSD systems, much like bz2 is today. I'm not asking people to
start using ClearCase ;) xz will supersede bz2, it's only a matter of
time. I see no problem in being one of the early adopters.

> > I have an old, slow machine here which I am going to use to perform
> > some real world testing, and I'll post the results when I'm done. But I
> > suspect that building a kernel on this machine, even a small one with
> > just the drivers it needs, will take much longer than unpacking the
> > sources. So anyone worrying about performance would rather rely on
> > cross-compilation, and in turn can afford whatever decompression tool
> > is needed.
> 
> On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So
> that one is ... well ... done overnight.

Out of curiosity, if it takes that long, why don't you use a
cross-compiler?

> Untar is something I normally wait for, since you need to run
> (interactive) oldconfig after that.

You'll have to wait, no matter what compression format you use (and
even if you don't compress the tarball). Judging by the duration of the
build on your machine, I'd estimate the decompression time to 7 minutes
for gz vs. 15 minutes for bz2 maybe? I doubt you sit in front of the
machine for 7 minutes waiting for tar.gz to decompress, right? So I
fail to see what difference it makes. You'll just do something else for
15 minutes instead of doing something else for 7 minutes.

Anyway, as I have been saying several times already, nothing prevents
you from repacking tarballs to gz before uploading it to your slow
system if such is your desire. I can understand the portability
argument, but the decompression time, no way.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[tytso]

On Sun, Feb 14, 2010 at 08:08:03PM +0100, Jean Delvare wrote:
> On Sun, 14 Feb 2010 18:07:24 +0100, Pavel Machek wrote:
> > Hi!
> > 
> > > As a matter of fact, I am advocating the use of xz while I don't have
> > > it installed on most of my machines. I really don't see this as a
> > > blocker.
> > 
> > Eh?
> > 
> > Making many people around the world install uncommon tool is not
> > something that should be done lightly.
> 
> It's pretty obvious that xz will become popular quickly, at least on
> Linux and BSD systems, much like bz2 is today. I'm not asking people to
> start using ClearCase ;) xz will supersede bz2, it's only a matter of
> time. I see no problem in being one of the early adopters.

If by "quickly" you mean 'ten years', sure, maybe.  Keep in mind that
there are people where who are still using RHEL 3, and some of them
might want to download from ftp.kernel.org.  So those people who are
suggesting that we replace .gz files with .xz on kernel.org are
*really* smoking something good.

People who think xz are good should be working to get it installed by
default into the community and then enterprise distro's, first....

						- Ted

Re: [kernel.org users] XZ Migration discussion

[J.H.]

On 02/15/2010 07:15 AM, tytso@mit.edu wrote:
> On Sun, Feb 14, 2010 at 08:08:03PM +0100, Jean Delvare wrote:
>> On Sun, 14 Feb 2010 18:07:24 +0100, Pavel Machek wrote:
>>> Hi!
>>>
>>>> As a matter of fact, I am advocating the use of xz while I don't have
>>>> it installed on most of my machines. I really don't see this as a
>>>> blocker.
>>>
>>> Eh?
>>>
>>> Making many people around the world install uncommon tool is not
>>> something that should be done lightly.
>>
>> It's pretty obvious that xz will become popular quickly, at least on
>> Linux and BSD systems, much like bz2 is today. I'm not asking people to
>> start using ClearCase ;) xz will supersede bz2, it's only a matter of
>> time. I see no problem in being one of the early adopters.
> 
> If by "quickly" you mean 'ten years', sure, maybe.  Keep in mind that
> there are people where who are still using RHEL 3, and some of them
> might want to download from ftp.kernel.org.  So those people who are
> suggesting that we replace .gz files with .xz on kernel.org are
> *really* smoking something good.
> 
> People who think xz are good should be working to get it installed by
> default into the community and then enterprise distro's, first....
> 
> 						- Ted

As a note xz is available via EPEL for Redhat Enterprice Linux 5 and
anything that's derived from that.  Doesn't seem to be available,
directly, for 4 or lower.  Not sure on Suse, and Debian's already been
mentioned.  Just trying to do a quick survey of what's out there already.

- John 'Warthog9' Hawley

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Tue, 16 Feb 2010 07:29:04 -0800, J.H. wrote:
> As a note xz is available via EPEL for Redhat Enterprice Linux 5 and
> anything that's derived from that.  Doesn't seem to be available,
> directly, for 4 or lower.  Not sure on Suse, and Debian's already been
> mentioned.  Just trying to do a quick survey of what's out there already.

xz is available since openSUSE 11.2, not before.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[David Rees]

On Tue, Feb 16, 2010 at 7:29 AM, J.H. <warthog9@kernel.org> wrote:
> On 02/15/2010 07:15 AM, tytso@mit.edu wrote:
>> People who think xz are good should be working to get it installed by
>> default into the community and then enterprise distro's, first....
>
> As a note xz is available via EPEL for Redhat Enterprice Linux 5 and
> anything that's derived from that.  Doesn't seem to be available,
> directly, for 4 or lower.  Not sure on Suse, and Debian's already been
> mentioned.  Just trying to do a quick survey of what's out there already.

Something I noticed, too.  Hopefully going to work on getting set up
as a packager and get xz pushed out to EPEL EL-4 over the next couple
weeks unless someone else beats me to it as I still have a handful of
EL-4 machines grinding away. :-)

-Dave

Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

> > > I have an old, slow machine here which I am going to use to perform
> > > some real world testing, and I'll post the results when I'm done. But I
> > > suspect that building a kernel on this machine, even a small one with
> > > just the drivers it needs, will take much longer than unpacking the
> > > sources. So anyone worrying about performance would rather rely on
> > > cross-compilation, and in turn can afford whatever decompression tool
> > > is needed.
> > 
> > On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So
> > that one is ... well ... done overnight.
> 
> Out of curiosity, if it takes that long, why don't you use a
> cross-compiler?

Because I hack it on the go. First build is long and ugly, but
subsequent builds only take 5 minutes or so, so development is
possible.

(If I crosscompiled it, I'd have to crosscompile even the subsequent
builds, which is impossible -- no powerful machine nearby).

> > Untar is something I normally wait for, since you need to run
> > (interactive) oldconfig after that.
> 
> You'll have to wait, no matter what compression format you use (and
> even if you don't compress the tarball). Judging by the duration of the
> build on your machine, I'd estimate the decompression time to 7 minutes
> for gz vs. 15 minutes for bz2 maybe? I doubt you sit in front of the
> machine for 7 minutes waiting for tar.gz to decompress, right? So I
> fail to see what difference it makes. You'll just do something else for
> 15 minutes instead of doing something else for 7 minutes.

Actually I do sit in front of the machine, reading mails while it
decompresses.

[I'll get some numbers.]

sh-3.2$ time bzip2 -d < ~/.ketchup/l^Ginux-2.6.31.tar.bz2 > delme.tar
485.73user 137.35system 683.32 (11m23.320s) elapsed 91.18%CPU
sh-3.2$ df -h^H^H^H^H^Htime cat delme.tar > /usr/src/delme.tar
0.57user 109.03system 381.13 (6m21.133s) elapsed 28.75%CPU
sh-3.2$ time zca^G^Gt delme.tar > /u ^H^H ^H^H ^H^H ^H^Hdelme.tar.gz
^H^H ^H^H ^H^H
+^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H.gz > delme.tar
43.97user 106.22system 223.26 (3m43.261s) elapsed 67.27%CPU

So... gzip is actually _faster_ than uncompressed data, while bzip is
twice slower. Don't know about xzip.

Anyway, gzip just makes sense. It is both smaller and faster than
alternatives, and nearly as portable.

> Anyway, as I have been saying several times already, nothing prevents
> you from repacking tarballs to gz before uploading it to your slow
> system if such is your desire. I can understand the portability
> argument, but the decompression time, no way.

Ok, so lets go by the portability argument.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [kernel.org users] XZ Migration discussion

[Jan Engelhardt]

On Friday 2010-02-12 16:21, Linus Torvalds wrote:
>On Fri, 12 Feb 2010, Jean Delvare wrote:
>> 
>> Maybe that's just me, but my main concern is neither download times nor
>> decompression times. My main concern is the access time to directory
>> indexes when browsing the kernel archive, because there are 5 entries
>> for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign.
>> This is horribly slow.
>
>This was actually the main reason for me personally to ask about just 
>dropping support for .gz files - not because I care deeply about how much 
>disk space kernel.org wastes, but because the long directory listings make 
>it slower for me to mentally index the directory.

Can I feature-request that someone reduces the git.kernel.org frontpage?
It's almost twice as large as the v2.6 dir index in http-delivered form,
so you can already experience what it's like when v2.6/ grows bigger.

Re: [kernel.org users] XZ Migration discussion

[J.H.]

On 02/18/2010 04:08 PM, Jan Engelhardt wrote:
> 
> On Friday 2010-02-12 16:21, Linus Torvalds wrote:
>> On Fri, 12 Feb 2010, Jean Delvare wrote:
>>>
>>> Maybe that's just me, but my main concern is neither download times nor
>>> decompression times. My main concern is the access time to directory
>>> indexes when browsing the kernel archive, because there are 5 entries
>>> for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign.
>>> This is horribly slow.
>>
>> This was actually the main reason for me personally to ask about just 
>> dropping support for .gz files - not because I care deeply about how much 
>> disk space kernel.org wastes, but because the long directory listings make 
>> it slower for me to mentally index the directory.
> 
> Can I feature-request that someone reduces the git.kernel.org frontpage?
> It's almost twice as large as the v2.6 dir index in http-delivered form,
> so you can already experience what it's like when v2.6/ grows bigger.

That's a topic for a completely different thread / audience than this,
lets try to keep this a bit more focused since we have the mirrors on
this discussion.

- John 'Warthog9' Hawley

Re: [kernel.org users] XZ Migration discussion

[Steven Rostedt]

On Fri, 2010-02-12 at 15:01 +0100, Jean Delvare wrote:

> I wouldn't worry too much about breaking the current locations. Just
> give some time for software authors (ketchup comes to mind) to update
> their code and it shouldn't be a big problem.

As the new maintainer of ketchup:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ketchup.git

I could write a fix for the new format real quick. My concern is that
users of ketchup may not see this update for a long time.

-- Steve

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Fri, 12 Feb 2010 10:25:18 -0500, Steven Rostedt wrote:
> On Fri, 2010-02-12 at 15:01 +0100, Jean Delvare wrote:
> 
> > I wouldn't worry too much about breaking the current locations. Just
> > give some time for software authors (ketchup comes to mind) to update
> > their code and it shouldn't be a big problem.
> 
> As the new maintainer of ketchup:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ketchup.git
> 
> I could write a fix for the new format real quick. My concern is that
> users of ketchup may not see this update for a long time.

What would you consider a reasonable turnaround time? 3 month? 6 month?
More?

We could make a decision now, you update ketchup immediately, and the
actual change happens at the specified date in the future.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Steven Rostedt]

On Fri, 2010-02-12 at 17:11 +0100, Jean Delvare wrote:

> What would you consider a reasonable turnaround time? 3 month? 6 month?
> More?
> 
> We could make a decision now, you update ketchup immediately, and the
> actual change happens at the specified date in the future.
> 

Usually users don't update ketchup until it breaks ;)

But sure, 3 months probably would be fine. Then Matt Mackall will start
getting emails about it not working, and he will forward them to me.
Where I can then point them to the new repository.

-- Steve

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Fri, 12 Feb 2010 11:30:39 -0500, Steven Rostedt wrote:
> On Fri, 2010-02-12 at 17:11 +0100, Jean Delvare wrote:
> 
> > What would you consider a reasonable turnaround time? 3 month? 6 month?
> > More?
> > 
> > We could make a decision now, you update ketchup immediately, and the
> > actual change happens at the specified date in the future.
> > 
> 
> Usually users don't update ketchup until it breaks ;)
> 
> But sure, 3 months probably would be fine. Then Matt Mackall will start
> getting emails about it not working, and he will forward them to me.
> Where I can then point them to the new repository.

You can probably anticipate this a bit by warning the maintainers of
the ketchup package in openSUSE, Fedora etc.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Junio C Hamano]

Steven Rostedt <rostedt@goodmis.org> writes:

> On Fri, 2010-02-12 at 17:11 +0100, Jean Delvare wrote:
>
>> What would you consider a reasonable turnaround time? 3 month? 6 month?
>> More?
>> 
>> We could make a decision now, you update ketchup immediately, and the
>> actual change happens at the specified date in the future.
>> 
>
> Usually users don't update ketchup until it breaks ;)

Then the archive location can change immediately after you issue an update
;-)

Would it be an option for a newer version of ketchup to have a fallback
codepath to read a machine-readable description of how the archive is
structured from a known location?  E.g. instead of appending a hard-coded
string '/people/akpm/patches/2.6/' when computing latest_mm(), the script
would read that "relative path to mm" from such a description file.  The
logic of doing 'url = "%s/v%s/%s" % (kernel_url, t, f)' in install_nearest
would similarly be customizable, without too much hassle, I think.

Re: [kernel.org users] XZ Migration discussion

[Steven Rostedt]

On Fri, 2010-02-12 at 12:34 -0800, Junio C Hamano wrote:
> Steven Rostedt <rostedt@goodmis.org> writes:
> 
> > On Fri, 2010-02-12 at 17:11 +0100, Jean Delvare wrote:
> >
> >> What would you consider a reasonable turnaround time? 3 month? 6 month?
> >> More?
> >> 
> >> We could make a decision now, you update ketchup immediately, and the
> >> actual change happens at the specified date in the future.
> >> 
> >
> > Usually users don't update ketchup until it breaks ;)
> 
> Then the archive location can change immediately after you issue an update
> ;-)
> 
> Would it be an option for a newer version of ketchup to have a fallback
> codepath to read a machine-readable description of how the archive is
> structured from a known location?  E.g. instead of appending a hard-coded
> string '/people/akpm/patches/2.6/' when computing latest_mm(), the script
> would read that "relative path to mm" from such a description file.  The
> logic of doing 'url = "%s/v%s/%s" % (kernel_url, t, f)' in install_nearest
> would similarly be customizable, without too much hassle, I think.

Are you suggesting that the path be stored at a known location on
kernel.org? Such that it can be read and the tools like ketchup can find
where a particular version resides?

-- Steve

ketchup was Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

Hi!

> > I wouldn't worry too much about breaking the current locations. Just
> > give some time for software authors (ketchup comes to mind) to update
> > their code and it shouldn't be a big problem.
> 
> As the new maintainer of ketchup:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ketchup.git
> 
> I could write a fix for the new format real quick. My concern is that
> users of ketchup may not see this update for a long time.

I'd really like to fix ketchup to behave sanely in late -rc stages:
going from -rc6 to -rc8 is possible using two downloads through --rc7
(like 100KB?)  instead of two big ones through full release (like
10MB).

But first step is --only-dl option -- when you want to cache the
needed patches but not apply anything yet.

Please apply,
									Pavel

diff --git a/ketchup b/ketchup
index 0728aec..3249cbc 100755
--- a/ketchup
+++ b/ketchup
@@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
         r = " -R"
 
     qprint("Applying %s%s" % (os.path.basename(p), r))
-    if options["dry-run"]:
+    if options["dry-run"] or options["only-dl"]:
         return ver
 
     def cmd(patch, reverse, dry):
@@ -484,7 +484,7 @@ def install_nearest(ver):
         ver = list[0][2]
 
     qprint("Unpacking %s" % os.path.basename(f))
-    if options["dry-run"]: return ver
+    if options["dry-run"] or options["only-dl"]: return ver
     untar(f)
 
     return ver
@@ -658,6 +658,7 @@ opts = [
     ('l', 'list-trees', None, 'list supported trees'),
     ('m', 'show-makefile', None, 'output version in makefile <arg>'),
     ('n', 'dry-run', None, 'don\'t download or apply patches'),
+    ('o', 'only-dl', None, 'don\'t apply patches'),
     ('p', 'show-previous', None, 'output version previous to <arg>'),
     ('q', 'quiet', None, 'reduce output'),
     ('r', 'rename-directory', None, 'rename updated directory to %s<v>'
@@ -750,7 +751,7 @@ if not a and os.listdir("."):
 b = find_ver(args[0])
 qprint("%s -> %s" % (a, b))
 transform(a, b)
-if options["rename-directory"] and not options["dry-run"]:
+if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
     rename_dir(b)
 
 if postcommand and os.system(postcommand):

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

[Steven Rostedt]

On Tue, 2010-02-16 at 16:39 +0100, Pavel Machek wrote:

> I'd really like to fix ketchup to behave sanely in late -rc stages:
> going from -rc6 to -rc8 is possible using two downloads through --rc7
> (like 100KB?)  instead of two big ones through full release (like
> 10MB).
> 
> But first step is --only-dl option -- when you want to cache the
> needed patches but not apply anything yet.
> 
> Please apply,
> 									Pavel

Can you send this to me as an official patch. With SoB and all.

Thanks,

-- Steve

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

Add --only-dl option -- when you want to cache the needed patches but
not apply anything yet.

Signed-off-by: Pavel Machek <pavel@ucw.cz>

diff --git a/ketchup b/ketchup
index 0728aec..3249cbc 100755
--- a/ketchup
+++ b/ketchup
@@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
         r = " -R"
 
     qprint("Applying %s%s" % (os.path.basename(p), r))
-    if options["dry-run"]:
+    if options["dry-run"] or options["only-dl"]:
         return ver
 
     def cmd(patch, reverse, dry):
@@ -484,7 +484,7 @@ def install_nearest(ver):
         ver = list[0][2]
 
     qprint("Unpacking %s" % os.path.basename(f))
-    if options["dry-run"]: return ver
+    if options["dry-run"] or options["only-dl"]: return ver
     untar(f)
 
     return ver
@@ -658,6 +658,7 @@ opts = [
     ('l', 'list-trees', None, 'list supported trees'),
     ('m', 'show-makefile', None, 'output version in makefile <arg>'),
     ('n', 'dry-run', None, 'don\'t download or apply patches'),
+    ('o', 'only-dl', None, 'don\'t apply patches'),
     ('p', 'show-previous', None, 'output version previous to <arg>'),
     ('q', 'quiet', None, 'reduce output'),
     ('r', 'rename-directory', None, 'rename updated directory to %s<v>'
@@ -750,7 +751,7 @@ if not a and os.listdir("."):
 b = find_ver(args[0])
 qprint("%s -> %s" % (a, b))
 transform(a, b)
-if options["rename-directory"] and not options["dry-run"]:
+if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
     rename_dir(b)
 
 if postcommand and os.system(postcommand):


diff --git a/ketchup.1 b/ketchup.1
index 0a313ee..9e5a385 100644
--- a/ketchup.1
+++ b/ketchup.1
@@ -1,5 +1,5 @@
 .\"                                      Hey, EMACS: -*- nroff -*-
-.TH KETCHUP 1 "April 12, 2006"
+.TH KETCHUP 1 "February 16, 2010"
 .\" Please adjust this date whenever revising the manpage.
 .\"
 .\" Some roff macros, for reference:
@@ -74,6 +74,11 @@ output version in makefile <arg>
 .IP
 don't download or apply patches
 .HP
+.B \-o
+.B \-\-only\-dl
+.IP
+don't apply patches
+.HP
 .B \-p
 .B \-\-show\-previous
 .IP


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

On Tue 2010-02-16 17:27:45, Pavel Machek wrote:
> Add --only-dl option -- when you want to cache the needed patches but
> not apply anything yet.

Actually, you'll probably get a better patch if you relace 'only-dl'
with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

> Signed-off-by: Pavel Machek <pavel@ucw.cz>
> 
> diff --git a/ketchup b/ketchup
> index 0728aec..3249cbc 100755
> --- a/ketchup
> +++ b/ketchup
> @@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
>          r = " -R"
>  
>      qprint("Applying %s%s" % (os.path.basename(p), r))
> -    if options["dry-run"]:
> +    if options["dry-run"] or options["only-dl"]:
>          return ver
>  
>      def cmd(patch, reverse, dry):
> @@ -484,7 +484,7 @@ def install_nearest(ver):
>          ver = list[0][2]
>  
>      qprint("Unpacking %s" % os.path.basename(f))
> -    if options["dry-run"]: return ver
> +    if options["dry-run"] or options["only-dl"]: return ver
>      untar(f)
>  
>      return ver
> @@ -658,6 +658,7 @@ opts = [
>      ('l', 'list-trees', None, 'list supported trees'),
>      ('m', 'show-makefile', None, 'output version in makefile <arg>'),
>      ('n', 'dry-run', None, 'don\'t download or apply patches'),
> +    ('o', 'only-dl', None, 'don\'t apply patches'),
>      ('p', 'show-previous', None, 'output version previous to <arg>'),
>      ('q', 'quiet', None, 'reduce output'),
>      ('r', 'rename-directory', None, 'rename updated directory to %s<v>'
> @@ -750,7 +751,7 @@ if not a and os.listdir("."):
>  b = find_ver(args[0])
>  qprint("%s -> %s" % (a, b))
>  transform(a, b)
> -if options["rename-directory"] and not options["dry-run"]:
> +if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
>      rename_dir(b)
>  
>  if postcommand and os.system(postcommand):
> 
> 
> diff --git a/ketchup.1 b/ketchup.1
> index 0a313ee..9e5a385 100644
> --- a/ketchup.1
> +++ b/ketchup.1
> @@ -1,5 +1,5 @@
>  .\"                                      Hey, EMACS: -*- nroff -*-
> -.TH KETCHUP 1 "April 12, 2006"
> +.TH KETCHUP 1 "February 16, 2010"
>  .\" Please adjust this date whenever revising the manpage.
>  .\"
>  .\" Some roff macros, for reference:
> @@ -74,6 +74,11 @@ output version in makefile <arg>
>  .IP
>  don't download or apply patches
>  .HP
> +.B \-o
> +.B \-\-only\-dl
> +.IP
> +don't apply patches
> +.HP
>  .B \-p
>  .B \-\-show\-previous
>  .IP
> 
> 

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Sun, 21 Feb 2010 14:53:41 +0100, Pavel Machek wrote:
> On Tue 2010-02-16 17:27:45, Pavel Machek wrote:
> > Add --only-dl option -- when you want to cache the needed patches but
> > not apply anything yet.
> 
> Actually, you'll probably get a better patch if you relace 'only-dl'
> with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

Or --download. Acronyms just suck.

> 
> > Signed-off-by: Pavel Machek <pavel@ucw.cz>
> > 
> > diff --git a/ketchup b/ketchup
> > index 0728aec..3249cbc 100755
> > --- a/ketchup
> > +++ b/ketchup
> > @@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
> >          r = " -R"
> >  
> >      qprint("Applying %s%s" % (os.path.basename(p), r))
> > -    if options["dry-run"]:
> > +    if options["dry-run"] or options["only-dl"]:
> >          return ver
> >  
> >      def cmd(patch, reverse, dry):
> > @@ -484,7 +484,7 @@ def install_nearest(ver):
> >          ver = list[0][2]
> >  
> >      qprint("Unpacking %s" % os.path.basename(f))
> > -    if options["dry-run"]: return ver
> > +    if options["dry-run"] or options["only-dl"]: return ver
> >      untar(f)
> >  
> >      return ver
> > @@ -658,6 +658,7 @@ opts = [
> >      ('l', 'list-trees', None, 'list supported trees'),
> >      ('m', 'show-makefile', None, 'output version in makefile <arg>'),
> >      ('n', 'dry-run', None, 'don\'t download or apply patches'),
> > +    ('o', 'only-dl', None, 'don\'t apply patches'),
> >      ('p', 'show-previous', None, 'output version previous to <arg>'),
> >      ('q', 'quiet', None, 'reduce output'),
> >      ('r', 'rename-directory', None, 'rename updated directory to %s<v>'
> > @@ -750,7 +751,7 @@ if not a and os.listdir("."):
> >  b = find_ver(args[0])
> >  qprint("%s -> %s" % (a, b))
> >  transform(a, b)
> > -if options["rename-directory"] and not options["dry-run"]:
> > +if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
> >      rename_dir(b)
> >  
> >  if postcommand and os.system(postcommand):
> > 
> > 
> > diff --git a/ketchup.1 b/ketchup.1
> > index 0a313ee..9e5a385 100644
> > --- a/ketchup.1
> > +++ b/ketchup.1
> > @@ -1,5 +1,5 @@
> >  .\"                                      Hey, EMACS: -*- nroff -*-
> > -.TH KETCHUP 1 "April 12, 2006"
> > +.TH KETCHUP 1 "February 16, 2010"
> >  .\" Please adjust this date whenever revising the manpage.
> >  .\"
> >  .\" Some roff macros, for reference:
> > @@ -74,6 +74,11 @@ output version in makefile <arg>
> >  .IP
> >  don't download or apply patches
> >  .HP
> > +.B \-o
> > +.B \-\-only\-dl
> > +.IP
> > +don't apply patches
> > +.HP
> >  .B \-p
> >  .B \-\-show\-previous
> >  .IP
> > 
> > 
> 


-- 
Jean Delvare

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

On Sun 2010-02-21 15:26:21, Jean Delvare wrote:
> On Sun, 21 Feb 2010 14:53:41 +0100, Pavel Machek wrote:
> > On Tue 2010-02-16 17:27:45, Pavel Machek wrote:
> > > Add --only-dl option -- when you want to cache the needed patches but
> > > not apply anything yet.
> > 
> > Actually, you'll probably get a better patch if you relace 'only-dl'
> > with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.
> 
> Or --download. Acronyms just suck.

Works for me.

And here's quick patch to show what I'm doing: it teaches ketchup
about 2.6.33-rc1-rc2 patches, thus saving lot of bandwidth. Of course,
I'll need to get it to work for more than 2.6.A->2.6.B-rcC case,
but...

If there are some comments, or maybe better approach, let me know...
diff --git a/ketchup b/ketchup
index 3249cbc..dc1bbf8 100755
--- a/ketchup
+++ b/ketchup
@@ -107,19 +107,28 @@ local_trees = {}
 # Functions to parse version strings
 
 def tree(ver):
+    """returns 2.6"""
     return float(re.match(r'(\d+\.\d+)', ver).group(1))
 
 def rev(ver):
+    """given 2.6.31 or 2.6.32-rc1 returns 31"""
     p = pre(ver)
     r = int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
     if p: r = r - 1
     return r
 
 def pre(ver):
+    """returns rc5"""
     try: return re.match(r'\d+\.\d+\.\d+(\.\d+)?-((rc|pre)\d+)', ver).group(2)
     except: return None
 
+def next_pre(ver):
+    s = pre(ver)
+    i = int(re.match(r'rc(\d+)', s).group(1))
+    return "rc%d" % (i+1)
+
 def post(ver):
+    """given 2.6.27.1 returns 1"""
     try: return re.match(r'\d+\.\d+\.\d+\.(\d+)', ver).group(1)
     except: return None
 
@@ -132,9 +141,15 @@ def prenum(ver):
     except: return None
 
 def prebase(ver):
+    """returns 2.6.13-rc1"""
     return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.)\d+)?)', ver).group(1)
 
+def preincr(ver):
+    """only use when incremental patches are requested"""
+    return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.).+)?)', ver).group(1)
+
 def revbase(ver):
+    """returns 2.6.23 for 2.6.23.15 or 2.6.24-rc5"""
     return "%s.%s" % (tree(ver), rev(ver))
 
 def base(ver):
@@ -283,8 +298,12 @@ def find_info(ver):
     f = forkname(ver)
     p = pre(ver)
 
+    print "find_info (ver) ", ver, "f=", f
+
     s = b
-    if f:
+    if re.match(".*rc.*rc.*", ver):
+        s = "%s-incrc" %b
+    elif f:
         s = "%s-%s" % (b, f)
     elif p:
         s = "%s-pre" % b
@@ -297,11 +316,14 @@ def version_urls(ver):
     if type(i) != type([]):
         i = [i]
 
+    print "version urls i = ", i    
+
     v = {
         'full': ver,
         'tree': tree(ver),
         'base': base(ver),
-        'prebase': prebase(ver)
+        'prebase': prebase(ver),
+        'preincr': preincr(ver),
         }
 
     l = []
@@ -399,6 +421,7 @@ def get_patch(ver):
 def apply_patch(ver, reverse = 0):
     """Find the patch to upgrade from the predecessor of ver to ver and
     apply or reverse it."""
+    print 'apply patch ', ver, ' reverse = ', reverse
     p = get_patch(ver)
     r = ""
     if reverse:
@@ -501,6 +524,15 @@ def find_ver(ver):
     else:
         return ver
 
+def remove_pre(a):
+    print 'remove -pre ' + a
+    apply_patch(a, 1)
+
+def goto_pre(a):
+    print 'goto -pre ' + a
+    apply_patch(base(a)+'-rc1', 0)
+    return base(a)+'-rc1'
+
 def transform(a, b):
     if a == b:
         qprint("Nothing to do!")
@@ -514,9 +546,16 @@ def transform(a, b):
     if fork(a):
         apply_patch(a, 1)
         a = prebase(a)
+
+    if base(a) == base(b):
+        if pre(a) and pre(b) and pre(a) < pre(b):
+            print a, ' to ', b, ' possible small steps'
+
     if prebase(a) != prebase(b):
+        print 'prebase ', a, ' is not prebase ', b
         if pre(a):
-            apply_patch(a, 1)
+            print 'pre (a)', pre(a)
+            remove_pre(a)
             a = base(a)
 
         if post(a) and (post(a) != post(b) or rev(a) != rev(b)):
@@ -536,8 +575,21 @@ def transform(a, b):
             a = base(b)
 
         if pre(b):
-            apply_patch(prebase(b))
-            a = prebase(b)
+            print "Now at ", a, " should go to ", b
+            rb = rev(a)
+            rc1base = "%s.%s" % (t, rb+1)
+            print "... ", rc1base
+            a = goto_pre(rc1base)
+
+            print "Now at ", a
+            
+            while pre(a) < pre(b):
+                s = ("%s-%s-%s" % (rc1base, pre(a), next_pre(a)))
+                print "Applying ", s
+                apply_patch(s, 0)
+                a = "%s-%s" % (rc1base, next_pre(a))
+                print "Should be at ", a
+            
 
     if fork(b):
         a = apply_patch(b)
@@ -573,6 +625,10 @@ version_info = {
             kernel_url + "/v2.6" + "/patch-%(prebase)s.bz2",
             r'patch-(.*?).bz2',
             1, "current stable kernel series"),
+    '2.6-incrc': (latest_dir,
+                kernel_url + "/v2.6" + "/testing/incr/patch-%(preincr)s.bz2",
+                r'patch-(.*?).bz2',
+                1, "current stable kernel series prereleases -- incremental"),
     '2.6-rc': (latest_dir,
                 kernel_url + "/v2.6" + "/testing/patch-%(prebase)s.bz2",
                 r'patch-(.*?).bz2',


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

On Sun 2010-02-21 14:53:41, Pavel Machek wrote:
> On Tue 2010-02-16 17:27:45, Pavel Machek wrote:
> > Add --only-dl option -- when you want to cache the needed patches but
> > not apply anything yet.
> 
> Actually, you'll probably get a better patch if you relace 'only-dl'
> with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

This should work; save time&bandwidth by using incremental patches
between -rcX.

Is there testsuite for ketchup? It would be useful at this point...

   	 	       		   	    	      	   Pavel

--- ketchup.orig	2010-02-16 16:36:51.000000000 +0100
+++ ketchup	2010-02-22 19:53:56.000000000 +0100
@@ -107,19 +107,32 @@
 # Functions to parse version strings
 
 def tree(ver):
+    """returns 2.6"""
     return float(re.match(r'(\d+\.\d+)', ver).group(1))
 
+def rawrev(ver):
+    """given 2.6.31 or 2.6.31-rc1 returns 31"""
+    return int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
+
 def rev(ver):
+    """given 2.6.31 or 2.6.32-rc1 returns 31"""
     p = pre(ver)
-    r = int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
+    r = rawrev(ver)
     if p: r = r - 1
     return r
 
 def pre(ver):
+    """returns rc5"""
     try: return re.match(r'\d+\.\d+\.\d+(\.\d+)?-((rc|pre)\d+)', ver).group(2)
     except: return None
 
+def next_pre(ver, inc = 1):
+    s = pre(ver)
+    i = int(re.match(r'rc(\d+)', s).group(1))
+    return "rc%d" % (i+inc)
+
 def post(ver):
+    """given 2.6.27.1 returns 1"""
     try: return re.match(r'\d+\.\d+\.\d+\.(\d+)', ver).group(1)
     except: return None
 
@@ -132,9 +145,15 @@
     except: return None
 
 def prebase(ver):
+    """returns 2.6.13-rc1"""
     return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.)\d+)?)', ver).group(1)
 
+def preincr(ver):
+    """only use when incremental patches are requested"""
+    return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.).+)?)', ver).group(1)
+
 def revbase(ver):
+    """returns 2.6.23 for 2.6.23.15 or 2.6.24-rc5"""
     return "%s.%s" % (tree(ver), rev(ver))
 
 def base(ver):
@@ -283,8 +302,12 @@
     f = forkname(ver)
     p = pre(ver)
 
+    print "find_info (ver) ", ver, "f=", f
+
     s = b
-    if f:
+    if re.match(".*rc.*rc.*", ver):
+        s = "%s-incrc" %b
+    elif f:
         s = "%s-%s" % (b, f)
     elif p:
         s = "%s-pre" % b
@@ -297,11 +320,14 @@
     if type(i) != type([]):
         i = [i]
 
+    print "version urls i = ", i    
+
     v = {
         'full': ver,
         'tree': tree(ver),
         'base': base(ver),
-        'prebase': prebase(ver)
+        'prebase': prebase(ver),
+        'preincr': preincr(ver),
         }
 
     l = []
@@ -399,6 +425,7 @@
 def apply_patch(ver, reverse = 0):
     """Find the patch to upgrade from the predecessor of ver to ver and
     apply or reverse it."""
+    print 'apply patch ', ver, ' reverse = ', reverse
     p = get_patch(ver)
     r = ""
     if reverse:
@@ -501,6 +528,24 @@
     else:
         return ver
 
+def step_rc(a, b):
+    rc1base = "%s.%s" % (tree(b), rawrev(b))    
+    print 'stepping from ', a, ' to ', b 
+    while pre(a) != pre(b):
+        if pre(a) < pre(b):
+            next = next_pre(a, 1)
+            s = ("%s-%s-%s" % (rc1base, pre(a), next))
+            apply_patch(s)
+        else:
+            next = next_pre(a, -1)
+            s = ("%s-%s-%s" % (rc1base, next, pre(a)))
+            apply_patch(s, 1)
+
+        print "Applying ", s
+
+        a = "%s-%s" % (rc1base, next)
+        print "Should be at ", a
+
 def transform(a, b):
     if a == b:
         qprint("Nothing to do!")
@@ -514,10 +559,25 @@
     if fork(a):
         apply_patch(a, 1)
         a = prebase(a)
+
+    if base(a) == base(b):
+        if pre(a) and pre(b):
+            print a, ' to ', b, ' possible small steps'
+            step_rc(prebase(a), prebase(b))
+            a = prebase(b)
+
     if prebase(a) != prebase(b):
-        if pre(a):
-            apply_patch(a, 1)
-            a = base(a)
+        print 'prebase ', a, ' is not prebase ', b
+        if (rev(a) != rev(b)) and pre(a):
+            print 'stepping to release, first'
+            rc1base = "%s.%s" % (tree(a), rawrev(a))
+            rc1 = rc1base + '-rc1'
+            print "Base is ", rc1base, "now at ", a, " should go to ", rc1
+            step_rc(a, rc1)
+            print "Should now be on -rc1"
+            apply_patch(rc1, 1)
+            a = base(rc1)
+            print "Should now be at", a
 
         if post(a) and (post(a) != post(b) or rev(a) != rev(b)):
             apply_patch(prebase(a), 1)
@@ -536,9 +596,12 @@
             a = base(b)
 
         if pre(b):
-            apply_patch(prebase(b))
-            a = prebase(b)
-
+            rc1base = "%s.%s" % (tree(b), rawrev(b))
+            rc1 = rc1base + '-rc1'
+            print "Base is ", rc1base, "now at ", a, " should go to ", b
+            apply_patch(rc1)
+            step_rc(rc1, b)
+            
     if fork(b):
         a = apply_patch(b)
 
@@ -573,6 +636,10 @@
             kernel_url + "/v2.6" + "/patch-%(prebase)s.bz2",
             r'patch-(.*?).bz2',
             1, "current stable kernel series"),
+    '2.6-incrc': (latest_dir,
+                kernel_url + "/v2.6" + "/testing/incr/patch-%(preincr)s.bz2",
+                r'patch-(.*?).bz2',
+                1, "current stable kernel series prereleases -- incremental"),
     '2.6-rc': (latest_dir,
                 kernel_url + "/v2.6" + "/testing/patch-%(prebase)s.bz2",
                 r'patch-(.*?).bz2',




-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

[Steven Rostedt]

On Mon, 2010-02-22 at 19:59 +0100, Pavel Machek wrote:
> On Sun 2010-02-21 14:53:41, Pavel Machek wrote:
> > On Tue 2010-02-16 17:27:45, Pavel Machek wrote:
> > > Add --only-dl option -- when you want to cache the needed patches but
> > > not apply anything yet.
> > 
> > Actually, you'll probably get a better patch if you relace 'only-dl'
> > with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.
> 
> This should work; save time&bandwidth by using incremental patches
> between -rcX.
> 
> Is there testsuite for ketchup? It would be useful at this point...

I don't have one, but that would be helpful.

Would you like to take over maintaining ketchup? I took it over because
Matt did not want it anymore and I was the last to send him patches.

I've been quite busy working on Ftrace, trace-cmd and kconfig stuff,
that I'm willing to hand this over to you, if you want it. My last
release of ketchup is my git tree on kernel.org.

-- Steve

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

[Pavel Machek]

Hi!

> > > > Add --only-dl option -- when you want to cache the needed patches but
> > > > not apply anything yet.
> > > 
> > > Actually, you'll probably get a better patch if you relace 'only-dl'
> > > with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.
> > 
> > This should work; save time&bandwidth by using incremental patches
> > between -rcX.
> > 
> > Is there testsuite for ketchup? It would be useful at this point...
> 
> I don't have one, but that would be helpful.
> 
> Would you like to take over maintaining ketchup? I took it over because
> Matt did not want it anymore and I was the last to send him patches.
> 
> I've been quite busy working on Ftrace, trace-cmd and kconfig stuff,
> that I'm willing to hand this over to you, if you want it. My last
> release of ketchup is my git tree on kernel.org.

I'd really prefer not to maintain it -- I'm using it on my zaurus,
because git is just too slow there -- but I don't really have time
just now.

(Plus, I'm not good enough with git, I'm afraid).

But, if you think the patches are good enough, I can just prepare
nice, easy to apply series? (But I'd really like someone to check that
the basic idea is right...)

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [kernel.org users] XZ Migration discussion

[Phillip Lougher]

Jean Delvare wrote:

> There are 3881 entries in that directory today,
> and it keeps growing!
> 
> 
> I can think of several ways to improve the situation here, some of
> which could be combined.
> 
> 1* Keep a single compression format. This saves almost 40% of the
> files.
> 
> 2* Move one of the compression formats somewhere else, so that it
> doesn't get in the way but is still available if needed.
> 
> 3* Create a new subdirectory for every 2.6.x kernel, and move all the
> related files there. This would shrink the main index drastically, and
> each subdirectory would have a reasonable size (except maybe 2.6.16 and
> 2.6.27.) Oddly enough this has been done for the files under testing/
> already, so I am curious why we don't do it for the release files (and
> the testing/incr/ files, while we're at it.)
> 
> 4* Get rid of the LATEST-IS-* files. This is a small count, won't save
> much, but these files seem totally useless to me these days. Depending
> on what you want exactly, there are many versions which can be
> considered the latest, and there are better ways to know which they are
> (for example http://www.eu.kernel.org/kdist/finger_banner ). And these
> files tend to get stuck so you can't rely on them anyway.
> 

5* Archive all the older 2.6.x files and move them into a separate
    directory (e.g. v2.6-pre20).  Moving all the pre 2.6.20 files
    saves 42% of the file listing.

This seems an obvious solution, what am I missing?

Thanks

Phillip

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Fri, 12 Feb 2010 19:02:39 +0000, Phillip Lougher wrote:
> 5* Archive all the older 2.6.x files and move them into a separate
>     directory (e.g. v2.6-pre20).  Moving all the pre 2.6.20 files
>     saves 42% of the file listing.
> 
> This seems an obvious solution, what am I missing?

This is confusing, inconsistent and unstable. Confusing because 2.6-pre
referred so far to the releases immediately preceding 2.6.0.
Inconsistent because it requires the downloader to have preliminary
knowledge about what the break point is. Unstable because, while you
consider pre20 to qualify as "old" today, in 5 years you will want
pre30 to qualify as "old" instead, meaning that tools such as ketchup
would have to be updated once again.

I think we want to come up with a directory structure which won't change
in the future.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Phillip Lougher]

Jean Delvare wrote:
> On Fri, 12 Feb 2010 19:02:39 +0000, Phillip Lougher wrote:
>> 5* Archive all the older 2.6.x files and move them into a separate
>>     directory (e.g. v2.6-pre20).  Moving all the pre 2.6.20 files
>>     saves 42% of the file listing.
>>
>> This seems an obvious solution, what am I missing?
> 
> This is confusing, inconsistent and unstable. Confusing because 2.6-pre
> referred so far to the releases immediately preceding 2.6.0.

I didn't say "2.6-pre", anyway it could be called something different,
like 'older-releases'.

> Inconsistent because it requires the downloader to have preliminary
> knowledge about what the break point is. Unstable because, while you
> consider pre20 to qualify as "old" today, in 5 years you will want
> pre30 to qualify as "old" instead, meaning that tools such as ketchup
> would have to be updated once again.

You yourself said "I wouldn't worry too much about breaking the current locations.
Just give some time for software authors (ketchup comes to mind) to update
their code and it shouldn't be a big problem."

The major advantage with my suggestion is for the majority of users/tools
interested in "recent" kernels, nothing changes at all.  Your suggestions
break everything for everyone.

> 
> I think we want to come up with a directory structure which won't change
> in the future.
> 

I think trying to do that is utterly futile.

Phillip

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Fri, 12 Feb 2010 19:57:26 +0000, Phillip Lougher wrote:
> Jean Delvare wrote:
> > On Fri, 12 Feb 2010 19:02:39 +0000, Phillip Lougher wrote:
> >> 5* Archive all the older 2.6.x files and move them into a separate
> >>     directory (e.g. v2.6-pre20).  Moving all the pre 2.6.20 files
> >>     saves 42% of the file listing.
> >>
> >> This seems an obvious solution, what am I missing?
> > 
> > This is confusing, inconsistent and unstable. Confusing because 2.6-pre
> > referred so far to the releases immediately preceding 2.6.0.
> 
> I didn't say "2.6-pre", anyway it could be called something different,
> like 'older-releases'.
> 
> > Inconsistent because it requires the downloader to have preliminary
> > knowledge about what the break point is. Unstable because, while you
> > consider pre20 to qualify as "old" today, in 5 years you will want
> > pre30 to qualify as "old" instead, meaning that tools such as ketchup
> > would have to be updated once again.
> 
> You yourself said "I wouldn't worry too much about breaking the current locations.
> Just give some time for software authors (ketchup comes to mind) to update
> their code and it shouldn't be a big problem."

Yes, I did say that, and I can repeat it if needed. There's a big
difference between breaking an old scheme which used to be valid and
happens to no longer be, and designing a new scheme where breakages are
bound to happen by design. Even if technically that's the same
breakage, its reception by the affected users will be very different,
because in the latter case you have no excuse.

> The major advantage with my suggestion is for the majority of users/tools
> interested in "recent" kernels, nothing changes at all.

Prove it. Many people out there are still working on older trees. I am
working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or
other tools break for these trees only and not more recent ones, that
won't help me at all, I will still have to update them.

> Your suggestions break everything for everyone.

This doesn't make me happy, but at least it is consistent and durable.
When you change something for everyone, it has the advantage that the
solutions are general, come quickly and are widely documented. Using
quirks to limit the effects is a burden for the future, and may not
even help that much in practice.

> > I think we want to come up with a directory structure which won't change
> > in the future.
> 
> I think trying to do that is utterly futile.

You didn't have to join this discussion in the first place.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Phillip Lougher]

Jean Delvare wrote:

>>> I think we want to come up with a directory structure which won't change
>>> in the future.
>> I think trying to do that is utterly futile.
> 
> You didn't have to join this discussion in the first place.
> 

And who are you to tell me I shouldn't have?  I was trying to be
constructive, and it seems as if I've been insulted just because I
dared to disagree with you.  Grow up.

My comment regarding futility is that things rarely turn out as
planned even with the best intentions, and it's rarely worth getting
hung up about designing the perfect system for something you
can't predict.

Phillip

Re: [kernel.org users] XZ Migration discussion

[Phillip Lougher]

Jean Delvare wrote:

> Prove it. Many people out there are still working on older trees. I am
> working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or
> other tools break for these trees only and not more recent ones, that
> won't help me at all, I will still have to update them.

Why are you still working on 2.6.5 and 2.6.16 kernels on a weekly
basis?  This could quite useful to know.  I could be pedantic and simply
turn your "prove it" around, and ask you to prove you're not an isolated
case, but that wouldn't be constructive, but irritating, like you.


Phillip

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Fri, 12 Feb 2010 23:39:51 +0000, Phillip Lougher wrote:
> Jean Delvare wrote:
> > Prove it. Many people out there are still working on older trees. I am
> > working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or
> > other tools break for these trees only and not more recent ones, that
> > won't help me at all, I will still have to update them.
> 
> Why are you still working on 2.6.5 and 2.6.16 kernels on a weekly
> basis?

Because I am doing support for enterprise customers who are using
distributions based on these kernel versions. These are SLE 9 and SLE
10, to name them, but RHEL supporters are in the same situation. And
I've heard embedded developers report many times that they had to stick
to older 2.6.x kernels too for various reasons. Not everyone is using a
recent 2.6.x kernel, which makes it hard to draw a line between what
should be considered old ones and what should be considered new ones.

-- 
Jean Delvare
"It is not necessary to hope in order to undertake, nor to succeed in
order to persevere" -- Charles The Bold

Re: [kernel.org users] XZ Migration discussion

[Phillip Lougher]

Jean Delvare wrote:
> On Fri, 12 Feb 2010 23:39:51 +0000, Phillip Lougher wrote:
>> Jean Delvare wrote:
>>> Prove it. Many people out there are still working on older trees. I am
>>> working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or
>>> other tools break for these trees only and not more recent ones, that
>>> won't help me at all, I will still have to update them.
>> Why are you still working on 2.6.5 and 2.6.16 kernels on a weekly
>> basis?
> 
> Because I am doing support for enterprise customers who are using
> distributions based on these kernel versions. These are SLE 9 and SLE
> 10, to name them, but RHEL supporters are in the same situation. And
> I've heard embedded developers report many times that they had to stick
> to older 2.6.x kernels too for various reasons. Not everyone is using a
> recent 2.6.x kernel, which makes it hard to draw a line between what
> should be considered old ones and what should be considered new ones.
> 

Embedded and enterprise distro users are usually stuck on ancient kernels that
were downloaded from kernel.org and patched *years ago*.  The reason they're
stuck on them is due to local modifications, and so they're not going to be
downloading ancient vanilla kernels from kernel.org now.

Phillip

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

Hi Phillip,

On Sat, 13 Feb 2010 22:37:41 +0000, Phillip Lougher wrote:
> Embedded and enterprise distro users are usually stuck on ancient kernels that
> were downloaded from kernel.org and patched *years ago*.  The reason they're
> stuck on them is due to local modifications, and so they're not going to be
> downloading ancient vanilla kernels from kernel.org now.

They perfectly could. This is exactly what we're doing at Suse and I can
easily imagine other companies follow the same model. We store our
local changes as patches on top of the old kernel version. When a new
developer joins the team and needs to setup a working tree, our setup
script gets the patches from our internal repository, fetches the
relevant kernel tarball from kernel.org, unpacks it and applies all the
patches.

This is one of the reasons why others have been claiming in this
discussion: it would be weird if files which were previously available
would suddenly disappear. We can discuss the cost and benefits of any
change done to the tree structure, compression formats etc. but please
do not assume that nobody is downloading the old files from kernel.org.

Personally I wouldn't mind at all if old files would disappear and our
tools have to be adjusted accordingly, as long as it happens only once
in a long while and not on a regular basis by (broken) design.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[Justin P. Mattock]

On 02/14/10 01:32, Jean Delvare wrote:
> Hi Phillip,
>
> On Sat, 13 Feb 2010 22:37:41 +0000, Phillip Lougher wrote:
>> Embedded and enterprise distro users are usually stuck on ancient kernels that
>> were downloaded from kernel.org and patched *years ago*.  The reason they're
>> stuck on them is due to local modifications, and so they're not going to be
>> downloading ancient vanilla kernels from kernel.org now.
>
> They perfectly could. This is exactly what we're doing at Suse and I can
> easily imagine other companies follow the same model. We store our
> local changes as patches on top of the old kernel version. When a new
> developer joins the team and needs to setup a working tree, our setup
> script gets the patches from our internal repository, fetches the
> relevant kernel tarball from kernel.org, unpacks it and applies all the
> patches.
>
> This is one of the reasons why others have been claiming in this
> discussion: it would be weird if files which were previously available
> would suddenly disappear. We can discuss the cost and benefits of any
> change done to the tree structure, compression formats etc. but please
> do not assume that nobody is downloading the old files from kernel.org.
>
> Personally I wouldn't mind at all if old files would disappear and our
> tools have to be adjusted accordingly, as long as it happens only once
> in a long while and not on a regular basis by (broken) design.
>


not trying to cut in, but the best example I can see for this
(hopefully), or a good example of just changing everything
(cut the middle man per say)is libc there is no libc-2.11.90.so 
.tar.gz(etc..)only through git(but could be wrong).

My system that I built is only handling everything(meaning every package 
as much as possible)through git.


Justin P. Mattock

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Fri, 12 Feb 2010 14:35:47 -0800, Luis R. Rodriguez wrote:
> On Fri, Feb 12, 2010 at 01:59:23PM -0800, Jean Delvare wrote:
> > Prove it. Many people out there are still working on older trees. I am
> > working on 2.6.5 and 2.6.16 kernels on a weekly basis.
> 
> I have to ask, I've read about the reasons why people still using 2.4 [1],
> why are you using 2.5? Same thing?
> 
> /me gets popcorn

Maybe try getting glasses instead? ;)

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/12/2010 06:01 AM, Jean Delvare wrote:
> 1* Keep a single compression format. This saves almost 40% of the
> files.
> 
> 2* Move one of the compression formats somewhere else, so that it
> doesn't get in the way but is still available if needed.

Neither of these are feasible, sorry.

> 3* Create a new subdirectory for every 2.6.x kernel, and move all the
> related files there. This would shrink the main index drastically, and
> each subdirectory would have a reasonable size (except maybe 2.6.16 and
> 2.6.27.) Oddly enough this has been done for the files under testing/
> already, so I am curious why we don't do it for the release files (and
> the testing/incr/ files, while we're at it.)

Well, part of the reason why is that we're functionally "stuck" on 2.6;
a prefix which really has lost all meaning.

It might open up the question if we shouldn't just do a Solaris and drop
the leading 2 (so the next kernel would be 6.33) or call the kernel
after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

	-hpa

Re: [kernel.org users] XZ Migration discussion

[Matthew Wilcox]

On Fri, Feb 12, 2010 at 11:03:26AM -0800, H. Peter Anvin wrote:
> > 3* Create a new subdirectory for every 2.6.x kernel, and move all the
> > related files there. This would shrink the main index drastically, and
> > each subdirectory would have a reasonable size (except maybe 2.6.16 and
> > 2.6.27.) Oddly enough this has been done for the files under testing/
> > already, so I am curious why we don't do it for the release files (and
> > the testing/incr/ files, while we're at it.)
> 
> Well, part of the reason why is that we're functionally "stuck" on 2.6;
> a prefix which really has lost all meaning.
> 
> It might open up the question if we shouldn't just do a Solaris and drop
> the leading 2 (so the next kernel would be 6.33) or call the kernel
> after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

Damn, we forgot to have that fight at Kernel Summit last year.

I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for
3.0.1, 3.0.2, 3.1.1, etc.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [kernel.org users] XZ Migration discussion

[Greg KH]

On Fri, Feb 12, 2010 at 01:25:34PM -0700, Matthew Wilcox wrote:
> On Fri, Feb 12, 2010 at 11:03:26AM -0800, H. Peter Anvin wrote:
> > > 3* Create a new subdirectory for every 2.6.x kernel, and move all the
> > > related files there. This would shrink the main index drastically, and
> > > each subdirectory would have a reasonable size (except maybe 2.6.16 and
> > > 2.6.27.) Oddly enough this has been done for the files under testing/
> > > already, so I am curious why we don't do it for the release files (and
> > > the testing/incr/ files, while we're at it.)
> > 
> > Well, part of the reason why is that we're functionally "stuck" on 2.6;
> > a prefix which really has lost all meaning.
> > 
> > It might open up the question if we shouldn't just do a Solaris and drop
> > the leading 2 (so the next kernel would be 6.33) or call the kernel
> > after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.
> 
> Damn, we forgot to have that fight at Kernel Summit last year.

No one wanted to take it on :(

> I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for
> 3.0.1, 3.0.2, 3.1.1, etc.

I'm in favor of almost _anything_ new, the current numbering scheme
drives me crazy, but then, I'm the one having to deal with it more than
anyone these days...

thanks,

greg k-h

Re: [kernel.org users] XZ Migration discussion

[Steven Rostedt]

On Fri, 2010-02-12 at 13:25 -0700, Matthew Wilcox wrote:

> > It might open up the question if we shouldn't just do a Solaris and drop
> > the leading 2 (so the next kernel would be 6.33) or call the kernel
> > after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.
> 
> Damn, we forgot to have that fight at Kernel Summit last year.
> 
> I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for
> 3.0.1, 3.0.2, 3.1.1, etc.
> 

At last years LinuxCon conference, I was practically bouncing out of my
chair wanting to ask Linus about 3.0 in front of everyone while he was
on the panel discussion. But James Bottomley refused to take my
question.

-- Steve

Re: [kernel.org users] XZ Migration discussion

[Harald Arnesen]

Matthew Wilcox <matthew@wil.cx> writes:

> On Fri, Feb 12, 2010 at 11:03:26AM -0800, H. Peter Anvin wrote:
>> > 3* Create a new subdirectory for every 2.6.x kernel, and move all the
>> > related files there. This would shrink the main index drastically, and
>> > each subdirectory would have a reasonable size (except maybe 2.6.16 and
>> > 2.6.27.) Oddly enough this has been done for the files under testing/
>> > already, so I am curious why we don't do it for the release files (and
>> > the testing/incr/ files, while we're at it.)
>> 
>> Well, part of the reason why is that we're functionally "stuck" on 2.6;
>> a prefix which really has lost all meaning.
>> 
>> It might open up the question if we shouldn't just do a Solaris and drop
>> the leading 2 (so the next kernel would be 6.33) or call the kernel
>> after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.
>
> Damn, we forgot to have that fight at Kernel Summit last year.
>
> I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for
> 3.0.1, 3.0.2, 3.1.1, etc.

Like I suggested in October 2008, but it would have been more natural at
that time:

<http://marc.info/?l=linux-kernel&m=122418454113793&w=2>
-- 
Hilsen Harald.

Re: [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/14/2010 06:49 AM, Harald Arnesen wrote:
>>
>> I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for
>> 3.0.1, 3.0.2, 3.1.1, etc.
> 
> Like I suggested in October 2008, but it would have been more natural at
> that time:
> 
> <http://marc.info/?l=linux-kernel&m=122418454113793&w=2>

You and several other people.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [kernel.org mirrors] [kernel.org users] XZ Migration discussion

[Sleddens, J.P.G.]

>>> On 12-2-2010 at 20:03, "H. Peter Anvin" <hpa@zytor.com> wrote:
> On 02/12/2010 06:01 AM, Jean Delvare wrote:
>> 3* Create a new subdirectory for every 2.6.x kernel, and move all the
>> related files there. This would shrink the main index drastically, and
>> each subdirectory would have a reasonable size (except maybe 2.6.16 and
>> 2.6.27.) Oddly enough this has been done for the files under testing/
>> already, so I am curious why we don't do it for the release files (and
>> the testing/incr/ files, while we're at it.)
> 
> Well, part of the reason why is that we're functionally "stuck" on 2.6;
> a prefix which really has lost all meaning.
> 
> It might open up the question if we shouldn't just do a Solaris and drop
> the leading 2 (so the next kernel would be 6.33) or call the kernel
> after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

I remember the whole LKML discussion about this a few years back:
http://lkml.org/lkml/2008/10/15/377 

The whole year.version   or year/month versioning Greg HK proposed
made a lot of sense to me.  It would also solve our problem with the 2.6
directory just growing and growing as the year versioning would make a
natural hierarchy which keeps going no matter what.

--
Jeffry Sleddens
Rotterdam University

Re: [kernel.org mirrors] [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/12/2010 12:31 PM, Sleddens, J.P.G. wrote:
>>>> On 12-2-2010 at 20:03, "H. Peter Anvin" <hpa@zytor.com> wrote:
>> On 02/12/2010 06:01 AM, Jean Delvare wrote:
>>> 3* Create a new subdirectory for every 2.6.x kernel, and move all the
>>> related files there. This would shrink the main index drastically, and
>>> each subdirectory would have a reasonable size (except maybe 2.6.16 and
>>> 2.6.27.) Oddly enough this has been done for the files under testing/
>>> already, so I am curious why we don't do it for the release files (and
>>> the testing/incr/ files, while we're at it.)
>>
>> Well, part of the reason why is that we're functionally "stuck" on 2.6;
>> a prefix which really has lost all meaning.
>>
>> It might open up the question if we shouldn't just do a Solaris and drop
>> the leading 2 (so the next kernel would be 6.33) or call the kernel
>> after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.
> 
> I remember the whole LKML discussion about this a few years back:
> http://lkml.org/lkml/2008/10/15/377 
> 
> The whole year.version   or year/month versioning Greg HK proposed
> made a lot of sense to me.  It would also solve our problem with the 2.6
> directory just growing and growing as the year versioning would make a
> natural hierarchy which keeps going no matter what.

Note also that every time this conversation happens it starts to pull
away in different directions, and as a result nothing happens.

I'm going to stick my foot in it and state the following: I think
incremental numbers work well, and everyone are used to them.  It
doesn't seem to be the major issue with the current scheme; the issue
with the current scheme is that we have one or two levels too much.

	-hpa

Re: [kernel.org users] [kernel.org mirrors] XZ Migration discussion

[Willy Tarreau]

On Fri, Feb 12, 2010 at 02:11:05PM -0800, H. Peter Anvin wrote:
> On 02/12/2010 12:31 PM, Sleddens, J.P.G. wrote:
> >>>> On 12-2-2010 at 20:03, "H. Peter Anvin" <hpa@zytor.com> wrote:
> >> On 02/12/2010 06:01 AM, Jean Delvare wrote:
> >>> 3* Create a new subdirectory for every 2.6.x kernel, and move all the
> >>> related files there. This would shrink the main index drastically, and
> >>> each subdirectory would have a reasonable size (except maybe 2.6.16 and
> >>> 2.6.27.) Oddly enough this has been done for the files under testing/
> >>> already, so I am curious why we don't do it for the release files (and
> >>> the testing/incr/ files, while we're at it.)
> >>
> >> Well, part of the reason why is that we're functionally "stuck" on 2.6;
> >> a prefix which really has lost all meaning.
> >>
> >> It might open up the question if we shouldn't just do a Solaris and drop
> >> the leading 2 (so the next kernel would be 6.33) or call the kernel
> >> after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.
> > 
> > I remember the whole LKML discussion about this a few years back:
> > http://lkml.org/lkml/2008/10/15/377 
> > 
> > The whole year.version   or year/month versioning Greg HK proposed
> > made a lot of sense to me.  It would also solve our problem with the 2.6
> > directory just growing and growing as the year versioning would make a
> > natural hierarchy which keeps going no matter what.
> 
> Note also that every time this conversation happens it starts to pull
> away in different directions, and as a result nothing happens.
> 
> I'm going to stick my foot in it and state the following: I think
> incremental numbers work well, and everyone are used to them.  It
> doesn't seem to be the major issue with the current scheme; the issue
> with the current scheme is that we have one or two levels too much.

Exactly. And for having worked with projects using dates, it's a hell
complicated to handle delayed releases... I'd like 3.0, 3.1, ... too,
and am using that scheme for my projects, which people easily understand
very well, as opposed to x.y.z.t.

But now I think that Linus has a "grep -v 3.0" filter on his inbox, I
never got a reply from him on the subject and I know I'm not the only
one :-(

Willy

Re: [kernel.org users] XZ Migration discussion

[Tony Luck]

On Fri, Feb 12, 2010 at 11:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> It might open up the question if we shouldn't just do a Solaris and drop
> the leading 2 (so the next kernel would be 6.33) or call the kernel
> after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

This sounds like a good plan (and since we have so
far failed to come up with some new feature in Linux
that is so awesome that it warrants bumping the
version number to 3.0 ... we might as well manufacture
one, and "The HTML for the archive directory on
kernel.org has gotten too big" seems a pretty good
reason to me).

So the plan could be:
1) Declare 2.6.35 (or so) to be the last in the 2.6 series.
2) Define a better archive directory structure for the 3.x
releases (scripts will have to be changed anyway to
handle the 3.x change)
3) Keep all the .gz and .bz2 in the old 2.x hierarchy
4) Only have .xz in the new 3.x directories.

This should give time for people to update scripts
and for xz compression tools to become widely
available.

People on the bleeding edge versions are the most
likely ones to update their tools promptly. People
still using 2.x series kernels can keep using their
old tools forever.

-Tony

Re: [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/12/2010 11:42 PM, Tony Luck wrote:
> 
> 3) Keep all the .gz and .bz2 in the old 2.x hierarchy
> 4) Only have .xz in the new 3.x directories.
> 

I really dislike this for reasons already stated.  Keep in mind that the
compression management is global to the entire archive, not just to the
kernel, too.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Sat, 13 Feb 2010 00:14:21 -0800, H. Peter Anvin wrote:
> On 02/12/2010 11:42 PM, Tony Luck wrote:
> > 
> > 3) Keep all the .gz and .bz2 in the old 2.x hierarchy
> > 4) Only have .xz in the new 3.x directories.
> > 
> 
> I really dislike this for reasons already stated.  Keep in mind that the
> compression management is global to the entire archive, not just to the
> kernel, too.

How hard a requirement is this? It might be the right time to
reconsider... Tony's plan may not be perfect but overall I think I like
it.

-- 
Jean Delvare

Re: [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/13/2010 12:53 AM, Jean Delvare wrote:
>>
>> I really dislike this for reasons already stated.  Keep in mind that the
>> compression management is global to the entire archive, not just to the
>> kernel, too.
>
> How hard a requirement is this? It might be the right time to
> reconsider... Tony's plan may not be perfect but overall I think I like
> it.
>

It's pretty darn hard.  It makes it particularly messy for the mirrors 
to ignore it.

	-hpa

Re: [kernel.org users] XZ Migration discussion

[H. Peter Anvin]

On 02/16/2010 09:55 AM, Luck, Tony wrote:
> 
>>> How hard a requirement is this? It might be the right time to
>>> reconsider... Tony's plan may not be perfect but overall I think I like
>>> it.
>>>
>>
>> It's pretty darn hard.  It makes it particularly messy for the mirrors 
>> to ignore it.
> 
> Sometimes hard things are worth the effort though.  I would think that
> there would be considerable advantage to a mirror system that had a
> mechanism to mark entire subtrees as "archive only - no further changes".
> 

There are a couple of reasons to not do that.

a) the moment we do that, some mirrors will start purging those areas.
We have tried to keep our mirror system universally usable by keeping a
"whole mirrors only, please" policy, and it has worked really well so far.

b) it would prevent us from *ever* reorganizing those areas, as many be
required.

c) it complicates a lot of the machinery, including the bits that allow
mirrors to only carry some of the compression formats.  That includes
mirror-side configuration, which I'd really loathe to make anything more
than very simple; mirror admins want things to work as hands-off and
automatic as possible, with a minimum of setup.  Having to learn the
intricacies of each mirror target is just plain hell.

	-hpa

Re: [kernel.org users] XZ Migration discussion

[Eric W. Biederman]

"J.H." <warthog9@kernel.org> writes:

> Hey Everyone,
>
> So as the subject states this is more a centralized discussion on
> migration plans to using and providing xz for content on kernel.org.
> Currently we provide gz and bz2, with gz acting as the original content
> and kernel.org itself generating the resulting bz2 files.  There are a
> couple of possible proposals and wanted to toss them out there, and get
> feedback from everyone: the kernel community, the mirrors of kernel.org
> and the direct users of kernel.org.
>
> ========================================================================
>
> Option 1)
>
> Leave gz as the master, and migrate bz2 to xz.  This will happen in
> stages obviously. with bz2 ultimately being phased out.
>
> 	Migration option 1)
>
> 		All new content would be provided in .bz2 and .xz with
> 		an ultimate date set that the .bz2 files would stop
> 		being generated with new content.  This would leave all
> 		existing content alone and it would not be a migration
> 		of the current .bz2 files to xz
>
> 	Migration option 2)
>
> 		At some point there would be a mass conversion of all
> 		existing content to include .bz2 and .xz.  These would
> 		be run in parallel for a time period until it was
> 		determined that .bz2 was no longer needed and it would
> 		be removed from the servers leaving .gz and .xz
>
> Option 2)
>
> Convert the master data from gz to bz2 and use xz as the new file
> format.  This has the downside of causing more tool churn as it means
> the kernel developers will have to eventually convert from gz to bz2,
> which means for a time there will be nag e-mails if you upload gz
> instead of bz2 and such.  It would also mean that we (kernel.org) would
> need to be able to support .gz and .bz2 as master data for a time.
>
> Migration options are identical to Option 1 more or less, with either
> just new content getting converted, or all content getting converted.
>
> ========================================================================
>
> I'm personally leaning towards option 1, though personally don't really
> have a preference on the migration options, as both obviously offer
> different advantages, and again this e-mail is more to spur on the
> discussion and come to some general consensus across all of the groups
> concerned before moving forward with a more specific plan.
>
> So I'm inviting discussion, questions and comments on this so we know
> which way to ultimately go.
>
> - John 'Warthog9' Hawley
> Chief Kernel.org Administrator

My would go for Option 1.

With respect to migration Option 2.  The last time we had a
succsessful change of the default compression format from compress to
gzip, it required a change from a encumbered format to an unencombered
format, and gunzip was able to decompress files created with compress.

Neither xz nor bzip2 can decompress gzip'd files, and appear to be
heavy enough that for modest file sizes they give little gain.  bzip2
fills a niche but does not act as a generally available, general
purpose compressor today.  It appears that xz is coming up fast in
bzip2's niche, and provides better compression, so would appear to be
a good replacement for bzip2.

The role of gzip is to be compatible with the everything, and I would
be very sad to see it gzip disappear as that would make files on
kernel.org much less accessible.  


Migration option 2 sounds like I would could just replace bzip2 support
in scripts with xz support so it sounds handy, but I will leave the
details to those who have to run the servers.


Right now xz suffers from limited availability.  I just checked the
systems I have handy and of the 4 distro's I have installed on various
machines only 1 provides xz, so I expect it will be a while before
xz achieves ubiquity.  At the same time the fact that xz reduces
the linux-2.6.32 tarball by 10Megs is a welcome benefit, and seems
to make the replacement of bzip2 with xz worth doing.

Eric

Re: [kernel.org users] XZ Migration discussion

[Stephen Hemminger]

On Sun, 14 Feb 2010 10:03:31 -0800
ebiederm@xmission.com (Eric W. Biederman) wrote:

> Right now xz suffers from limited availability.  I just checked the
> systems I have handy and of the 4 distro's I have installed on various
> machines only 1 provides xz, so I expect it will be a while before
> xz achieves ubiquity.  At the same time the fact that xz reduces
> the linux-2.6.32 tarball by 10Megs is a welcome benefit, and seems
> to make the replacement of bzip2 with xz worth doing.

xv is not standard on Ubuntu.
Installing xv-utils on Ubuntu 9.10 produces big scary warning
about package conflict with lzma.

This seems like a premature optimization at this point

-- 

Re: [kernel.org users] XZ Migration discussion

[Jean Delvare]

On Sun, 14 Feb 2010 14:27:29 -0800, Stephen Hemminger wrote:
> On Sun, 14 Feb 2010 10:03:31 -0800
> ebiederm@xmission.com (Eric W. Biederman) wrote:
> 
> > Right now xz suffers from limited availability.  I just checked the
> > systems I have handy and of the 4 distro's I have installed on various
> > machines only 1 provides xz, so I expect it will be a while before
> > xz achieves ubiquity.  At the same time the fact that xz reduces
> > the linux-2.6.32 tarball by 10Megs is a welcome benefit, and seems
> > to make the replacement of bzip2 with xz worth doing.
> 
> xv is not standard on Ubuntu.
> Installing xv-utils on Ubuntu 9.10 produces big scary warning
> about package conflict with lzma.
> 
> This seems like a premature optimization at this point

That would be a packaging bug, xz should transparently supersede lzma.

-- 
Jean Delvare