Re: RFC: Starting a stable kernel series off the 2.6 kernel

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[John Kelly]

On 12/3/05, Adrian Bunk <bunk@stusta.de> wrote:

> Since Andrew and Linus do AFAIK not plan to change the development 
> model, what about the following for getting a stable kernel series 
> without leaving the current development model:

> Kernel 2.6.16 will be the base for a stable series.

Or just wait for a "good" one, whatever number that happens to be.

I believe Linus current development model is better than the old way,
because it keeps the kernel moving ahead.  But like you say, it's hard
on users.

My "distro" can't help me because I don't use a "distro."  There are
plenty of users rolling their own, via linuxfromscratch, diy-linux, or
some other alternative.

We would benefit from your proposal.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lars Marowsky-Bree]

On 2005-12-05T23:18:04, John Kelly <jakelly@shtc.net> wrote:

> My "distro" can't help me because I don't use a "distro."  There are
> plenty of users rolling their own, via linuxfromscratch, diy-linux, or
> some other alternative.
> 
> We would benefit from your proposal.

So stop whining and go fucking do it, one of you.

Who, or what, is stopping you?

The model works for the people currently doing it. You can't expect them
to change because it would work better for _you_. This is an Open Source
world, so go eat your own dogfood.


Sincerely,
    Lars Marowsky-Brée

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[John Kelly]

On Tue, 06 Dec 2005 14:27:58 +0100, Lars Marowsky-Bree <lmb@suse.de>
wrote:

>On 2005-12-05T23:18:04, John Kelly <jakelly@shtc.net> wrote:

>> My "distro" can't help me because I don't use a "distro."  There are
>> plenty of users rolling their own, via linuxfromscratch, diy-linux, or
>> some other alternative.
 
>> We would benefit from your proposal.

>So stop whining and go fucking do it, one of you.

Does it annoy you that I don't use a "distro" Lars?


>Who, or what, is stopping you?

Adrian asked for comments.  He can probably do it better than I can.


>The model works for the people currently doing it. You can't expect them
>to change because it would work better for _you_.

How did you infer that from my post?


>This is an Open Source world, so go eat your own dogfood.

I wonder why you have such a hostile attitude.  Maybe the atmosphere
is not too happy at SUSE nowadays.  It makes me glad I don't use your
"distro."


>
>Sincerely,
>    Lars Marowsky-Brée

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lars Marowsky-Bree]

On 2005-12-06T13:21:23, John Kelly <jakelly@shtc.net> wrote:

> >So stop whining and go fucking do it, one of you.
> Does it annoy you that I don't use a "distro" Lars?

I couldn't care less ;-) Technical savvy users create way too much work,
anyway. ;-)

No, I'm just annoyed by the explosion in this thread where so many
people comment, but very few who say they want to do the work, yet many
complain that the current model doesn't meet their needs. And yes, I
call that whining, and someone should just go fucking do it and get it
over with.

> I wonder why you have such a hostile attitude.  Maybe the atmosphere
> is not too happy at SUSE nowadays.  It makes me glad I don't use your
> "distro."

You're free not to. This is the beauty of Linux.


Sincerely,
    Lars Marowsky-Brée

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Felix Oxley]

On 13 Dec 2005, at 13:17, Horst von Brand wrote:

> Felix Oxley <lkml@oxley.org> wrote:
>> On 12 Dec 2005, at 17:17, Horst von Brand wrote:
>>> Felix Oxley <lkml@oxley.org> wrote:
>>>
>>> [...]
>>>
>>>> What if ...
>>>>
>>>> 1. When people make a patch set, if they have encountered any  
>>>> 'bugs'
>>>> they split them out as separate items.
>
>>> No need. Patches are either (a) bug fixes, or (b) infrastructure
>>> changes, or (c) additions (mostly drivers). You only need to pick  
>>> (a)
>>> items. Check.
>
> [...]
>
>>>> 3. When the patch is posted to LKML, it is tagged [PATCH][FIX]  
>>>> in the
>>>> subject line.
>>>>      In the body of the fix would be noted each kernel to which the
>>>>      fix applied e.g [FIX 2.6.11][FIX 2.6.12][FIX 2.6.13][FIX  
>>>> 2.6.14]
>
>>> No do. Problem are the (b) and (c) patches above, they change
>>> whatever the patch applies to and make it not apply anymore. The  
>>> effort
>>> of finding out if the patch is (a) or (c) class, seeing if it is  
>>> really
>>> needed, and modifying it so it applies to your source base is called
>>> "backporting". And it remains hard, thankless work.
>>
>> If this was done for 'trivial' patches of type (a):
>> 	1. Would that make it simple enough for people to actually do it?
>> 	2. Would it be worthwhile? (Are there enough 'trivial fixes'?)
>
> Not all important fixes are "trivial", far from it; so this is rather
> suspect in any case. Changes to the underlying source make even  
> "trivial"
> patches soon not apply anymore. And there still is the job of  
> finding out
> if some patch is or is not necessary...
>
>> I envisaged something like the current Stable series, just for longer
>> than a single release cycle.
>
> Go right ahead. If enough people get interested and work on it, it  
> might
> turn out useful. I rather doubt it, as the current development  
> model is
> exactly geared towards keeping people up to date, not running ancient
> kernels and then jumping a few versions ahead. The problem with  
> doing that
> is that instead of one problem at a time you see a dozen, and then  
> it is
> hard to pin down /when/ it broke (and thus what change is  
> responsible).
> Plus the drift from backported patches, where you can't be sure it / 
> seemed/
> to work because of some random patch.
>
> Again, this development model was tried /hard/ for some 12 years by  
> the
> distributions, and found sorely lacking (and essentially unfixable).

Thank you for your explanation.
I will retire to lurk quietly in my armchair.  :-)

regards,
Felix

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Felix Oxley <lkml@oxley.org> wrote:
> On 12 Dec 2005, at 17:17, Horst von Brand wrote:
> > Felix Oxley <lkml@oxley.org> wrote:
> >
> > [...]
> >
> >> What if ...
> >>
> >> 1. When people make a patch set, if they have encountered any 'bugs'
> >> they split them out as separate items.

> > No need. Patches are either (a) bug fixes, or (b) infrastructure
> > changes, or (c) additions (mostly drivers). You only need to pick (a)
> > items. Check.

[...]

> >> 3. When the patch is posted to LKML, it is tagged [PATCH][FIX] in the
> >> subject line.
> >>      In the body of the fix would be noted each kernel to which the
> >>      fix applied e.g [FIX 2.6.11][FIX 2.6.12][FIX 2.6.13][FIX 2.6.14]

> > No do. Problem are the (b) and (c) patches above, they change
> > whatever the patch applies to and make it not apply anymore. The effort
> > of finding out if the patch is (a) or (c) class, seeing if it is really
> > needed, and modifying it so it applies to your source base is called
> > "backporting". And it remains hard, thankless work.
> 
> If this was done for 'trivial' patches of type (a):
> 	1. Would that make it simple enough for people to actually do it?
> 	2. Would it be worthwhile? (Are there enough 'trivial fixes'?)

Not all important fixes are "trivial", far from it; so this is rather
suspect in any case. Changes to the underlying source make even "trivial"
patches soon not apply anymore. And there still is the job of finding out
if some patch is or is not necessary...

> I envisaged something like the current Stable series, just for longer
> than a single release cycle.

Go right ahead. If enough people get interested and work on it, it might
turn out useful. I rather doubt it, as the current development model is
exactly geared towards keeping people up to date, not running ancient
kernels and then jumping a few versions ahead. The problem with doing that
is that instead of one problem at a time you see a dozen, and then it is
hard to pin down /when/ it broke (and thus what change is responsible).
Plus the drift from backported patches, where you can't be sure it /seemed/
to work because of some random patch.

Again, this development model was tried /hard/ for some 12 years by the
distributions, and found sorely lacking (and essentially unfixable).
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Felix Oxley]

On 12 Dec 2005, at 17:17, Horst von Brand wrote:

> Felix Oxley <lkml@oxley.org> wrote:
>
> [...]
>
>> What if ...
>>
>> 1. When people make a patch set, if they have encountered any 'bugs'
>> they split them out as separate items.
>
> No need. Patches are either (a) bug fixes, or (b) infrastructure  
> changes,
> or (c) additions (mostly drivers). You only need to pick (a) items.  
> Check.
>
>> 2. The submitter would identify through GIT when the error had been
>> introduced
>
> Hard to find out. Nobody will do so.
>
>>            so that the the person responsible could be CC'ed, also
>> anybody who had worked on the code recently would be CCed, therefore
>> the programmers who were most familiar with that section of code
>> would be made aware of it.
>
> Cc:ing them is part of the development anyway (in reality, Cc:ing  
> anybody
> interested in the area). Check.
>
>> 3. When the patch is posted to LKML, it is tagged [PATCH][FIX] in the
>> subject line.
>>      In the body of the fix would be noted each kernel to which the
>>      fix applied e.g [FIX 2.6.11][FIX 2.6.12][FIX 2.6.13][FIX 2.6.14]
>
> No do. Problem are the (b) and (c) patches above, they change  
> whatever the
> patch applies to and make it not apply anymore. The effort of  
> finding out
> if the patch is (a) or (c) class, seeing if it is really needed, and
> modifying it so it applies to your source base is called  
> "backporting". And
> it remains hard, thankless work.

If this was done for 'trivial' patches of type (a):
	1. Would that make it simple enough for people to actually do it?
	2. Would it be worthwhile? (Are there enough 'trivial fixes'?)

I envisaged something like the current Stable series, just for longer  
than a single release cycle.

>> 4. The programmers mentioned in (2) would ACK the patch which would
>> then become part of an 'official' fixes list.
>
> Won't happen.
>
>> 5. If a volunteer wanted to maintain, say, 2.6.14 + fixes, they could
>> build and test it and be a point of contact regarding any problems.
>> These could hopefully be tracked down and submitted as a new fix  
>> patch.
>
> Go right ahead. Just be warned that distributions hired a small  
> army of
> kernel specialists to do exactly this, and got tired of doing so.  
> Among
> others because the patches deemed necessary were different from one
> distributuion to the next, and then usually incompatible with one  
> another
> and with what turned out to be the standard solution. This gave  
> rise to the
> current development model...
>
> Armchair software engineering is much like armchair $SPORT.

I am guilty :-)

Thanks for your reply.

regards,
Felix

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Felix Oxley <lkml@oxley.org> wrote:

[...]

> What if ...
> 
> 1. When people make a patch set, if they have encountered any 'bugs'
> they split them out as separate items.

No need. Patches are either (a) bug fixes, or (b) infrastructure changes,
or (c) additions (mostly drivers). You only need to pick (a) items. Check.

> 2. The submitter would identify through GIT when the error had been
> introduced

Hard to find out. Nobody will do so.

>            so that the the person responsible could be CC'ed, also
> anybody who had worked on the code recently would be CCed, therefore
> the programmers who were most familiar with that section of code
> would be made aware of it.

Cc:ing them is part of the development anyway (in reality, Cc:ing anybody
interested in the area). Check.

> 3. When the patch is posted to LKML, it is tagged [PATCH][FIX] in the
> subject line.
>      In the body of the fix would be noted each kernel to which the
>      fix applied e.g [FIX 2.6.11][FIX 2.6.12][FIX 2.6.13][FIX 2.6.14]

No do. Problem are the (b) and (c) patches above, they change whatever the
patch applies to and make it not apply anymore. The effort of finding out
if the patch is (a) or (c) class, seeing if it is really needed, and
modifying it so it applies to your source base is called "backporting". And
it remains hard, thankless work.

> 4. The programmers mentioned in (2) would ACK the patch which would
> then become part of an 'official' fixes list.

Won't happen.

> 5. If a volunteer wanted to maintain, say, 2.6.14 + fixes, they could
> build and test it and be a point of contact regarding any problems.
> These could hopefully be tracked down and submitted as a new fix patch.

Go right ahead. Just be warned that distributions hired a small army of
kernel specialists to do exactly this, and got tired of doing so. Among
others because the patches deemed necessary were different from one
distributuion to the next, and then usually incompatible with one another
and with what turned out to be the standard solution. This gave rise to the
current development model...

Armchair software engineering is much like armchair $SPORT.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Felix Oxley]

On 3 Dec 2005, at 13:56, Adrian Bunk wrote:

> The current kernel development model is pretty good for people who
> always want to use or offer their costumers the maximum amount of the
> latest bugs^Wfeatures without having to resort on additional  
> patches for
> them.
>
> Problems of the current development model from a user's point of view
> are:
> - many regressions in every new release
> - kernel updates often require updates for the kernel-related  
> userspace
>   (e.g. for udev or the pcmcia tools switch)
>
> One problem following from this is that people continue to use older
> kernels with known security holes because the amount of work for  
> kernel
> upgrades is too high.
>
> These problems follow from the development model.
>
> The latest stable kernel series without these problems is 2.4, but 2.4
> is becoming more and more obsolete and might e.g. lack driver support
> for some recent hardware you want to use.
>
> Since Andrew and Linus do AFAIK not plan to change the development
> model, what about the following for getting a stable kernel series
> without leaving the current development model:
>
>
> Kernel 2.6.16 will be the base for a stable series.
>
> After 2.6.16, there will be a 2.6.16.y series with the usual stable
> rules.
>
> After the release of 2.6.17, this 2.6.16.y series will be continued  
> with
> more relaxed rules similar to the rules in kernel 2.4 since the  
> release
> of kernel 2.6.0 (e.g. driver updates will be allowed).
>
>
> Q:
> What is the target audience for this 2.6.16 series?
>
> A:
> The target audience are users still using 2.4 (or who'd still use  
> kernel
> 2.4 if they weren't forced to upgrade to 2.6 for some reason) who  
> want a
> stable kernel series including security fixes but excluding many
> regressions.
> It might also be interesting for distributions that prefer stability
> over always using the latest stuff.
>
>
> Q:
> Does this proposal imply anything for the development between  
> 2.6.15 and
> 2.6.16?
>
> A:
> In theory not.
> In practice, it would be a big advantage if some of the bigger
> changes that might go into 2.6.16 would be postponed to 2.6.17.
>
>
> Q:
> Why not start with the more relaxed rules before the release of  
> 2.6.17?
>
> A:
> After 2.6.16.y following the usual stable rules, the kernel should be
> relatively stable and well-tested giving the best possible basis for a
> long-living series.
>
>
> Q:
> How long should this 2.6.16 series be maintained?
>
> A:
> Time will tell, but if people use it I'd expect 2 or 3 years.
>
>
> Q:
> Stable API/ABI for external modules?
>
> A:
> No.
>
>
> Q:
> Who will maintain this branch?
>
> A:
> I could do it, but if someone more experienced wants to do it that  
> would
> be even better.
> -
> To unsubscribe from this list: send the line "unsubscribe linux- 
> kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


What if ...

1. When people make a patch set, if they have encountered any 'bugs'  
they split them out as separate items.

2. The submitter would identify through GIT when the error had been  
introduced so that the the person responsible could be CC'ed, also  
anybody who had worked on the code recently would be CCed, therefore  
the programmers who were most familiar with that section of code  
would be made aware of it.

3. When the patch is posted to LKML, it is tagged [PATCH][FIX] in the  
subject line.
     In the body of the fix would be noted each kernel to which the  
fix applied e.g [FIX 2.6.11][FIX 2.6.12][FIX 2.6.13][FIX 2.6.14]

4. The programmers mentioned in (2) would ACK the patch which would  
then become part of an 'official' fixes list.

5. If a volunteer wanted to maintain, say, 2.6.14 + fixes, they could  
build and test it and be a point of contact regarding any problems.  
These could hopefully be tracked down and submitted as a new fix patch.

regards,
Felix

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Douglas McNaught wrote:

>Bill Davidsen <davidsen@tmr.com> writes:
>
>  
>
>>Rob Landley wrote:
>>
>>    
>>
>>>Re-raising the same objections over and over again when they've
>>>already been aired, considered, and rejections is called "whining".
>>>
>>>      
>>>
>>Repeating the same information over and over until it sinks in is
>>called "rote learning." The question is not if cryptoloop is perfect,
>>every crypto seems to fail eventually, recently md5 was cracked,
>>etc. But people have used cryptoloop now, and removing it from the
>>kernel will lock them out of their own data, or prevent them from
>>moving forward. There's no replacement for cryptoloop, so I can't just
>>reconfigure X and still read my 147 DVDs full of business data, or
>>access the current data on 34 laptops around the country.
>>    
>>
>
>Bill, I still don't think your complaints are justified.
>  
>
I never expected anyone to admit they were wrong, so that doesn't 
surprise me...

>You're only "locked out of your own data" if you knowingly upgrade to
>a kernel that doesn't support cryptoloop.  Nobody's forcing you to do
>that. 
>  
>

Are you endorsing ignoring security fixes? Of course you're forced to if 
you are trying to be secure. If there were a replacement for cryptoloop 
that wouldn't be a problem. But saying that CL must go because it isn't 
perfect is like saying that you shouldn't lock your window because 
someone could still break it and get in.

>The kernel developers owe *nothing* to J. Random User.  They are
>either doing what they do for their own reasons (the "fun" of it), or
>being paid by an organization with specific objectives (even if, in
>Linus' case, the objective is just "make the best kernel possible,
>based on your judgement and that of people you trust").
>  
>

A little later you say that most of the developers are paid for working 
on Linux. Just who is the ultimate source of that funding if not the 
random user? Almost all of it comes from people who lack the ability to 
maintain the kernel, one way or the other. If kernel features are left 
to the vendors you encourage fragmentation, and all you have to do is 
look at BSD to see what a success that is.

What Linux has going over Windows is choice... the ability to configure 
WITHOUT having to depend on the judgement of someone else. And when that 
judgement is to remove a feature which has no replacement, in which uses 
have made an investment, then the choice is gone.

>That said, of course none of them want to break things unnecessarily.
>But they make technical decisions, with the goal of having the best
>kernel, that do sometimes have painful consequences.  You're free, of
>course, to disagree with those decisions and maintain your own kernel.
>  
>

Why waste electrons on statments like that. Yes, I could do that, but 
the average users can't, and after maintaining GECOS, and MULTICS, and 
supporting BSD installations and writing a realtime control o/s, I 
certainly don't have the slightest interest in spending my time doing 
that. Effectively anything not in a kernel.org kernel is going to die.

>They don't owe you security fixes either.  Sorry, but that's the way
>it is.  We're all lucky that they take security very seriously and
>respond quickly to problems.
>
>  
>
>>In most cases CL is not expected to protect against goverment agencies
>>but rather stolen laptops in airports (yes the pros have added MacOS
>>and Linux to their business model) or the occasional lost DVD in the
>>mail. Removing CL is not a hell of a lot better morally than these
>>viruses which encrypt your data and then hold it for ransom, with the
>>price being doing without security fixes in future kernels.
>>    
>>
>
>That last sentence is crap.
>
>You're free to backport security fixes to cryptoloop-supporting
>kernels forever, or pay someone to do so.  Or maintain a cryptoloop
>patch against current kernels, or pay someone to do so.  Or write a
>converter for cryptoloop data to whatever's currently in the kernel,
>or pay someone to do so.
>  
>
Given that CL has minimal (essentially no) maintenence cost, I wish

>>the ivory tower developers could understand that real people have
>>invested real money in it, and real data in the technology. Since
>>there is no alternative solution offered, CL is far better than no
>>crypto at all, and I wish there were a few more developers who had
>>experience working in the real word.
>>    
>>
>
>If you include a crypto solution in the mainstream kernel, you're in
>some sense endorsing its security.  If that solution has known
>weaknesses, I can understand wanting to either fix it or rip it out.
>Crypto is hard enough to get right as it is.
>
>Your "ivory tower" statement is really condescending.  Linux is way
>past the stage where college students were the main contributors (if
>it ever was so after Linus graduated). and a great majority of
>developers now are paid to work on the kernel.  There are probably
>very few of them that don't have at least a little sysadmin
>experience.
>  
>
I wonder... running servers is relatively easy, supporting end user 
systems (admin, not help desk) is hard.

>If you've invested money and put important data in a system, and you
>haven't contracted with anyone to support that system, supply security
>fixes, and make sure it does what you want it to do, who's the fool?
>  
>
The advantage of using dynamic systems rather than locking in with 
something like RHEL was attractive. Trusting a third party was not.

>Basically, you're complaining about something you get *for free* that
>represents millions of hours of work, because it doesn't work quite
>the way you want it to, when you have perfect freedom to make it meet
>your needs by putting in your own time, effort and/or money.
>

Most users have no such ability, but people jumped abord Linux when 
2.6.0 came out, and it WAS called a "new stable release." Redefining 
what stable means after people have used the software is not something I 
would feel comfortable doing.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Saturday 10 December 2005 11:05, Douglas McNaught wrote:
> The kernel developers owe *nothing* to J. Random User.  They are
> either doing what they do for their own reasons (the "fun" of it), or
> being paid by an organization with specific objectives (even if, in
> Linus' case, the objective is just "make the best kernel possible,
> based on your judgement and that of people you trust").

That's not a good argument.  I don't think Bill has a good argument either, 
but this isn't one.

The kernel developers have very good reasons for what they're doing, and 
ultimately they believe that what they're doing is in the best long-term 
interests of the users.  Bill has objections based on the short-term 
interests of users.  The kernel developers are saying that looking after the 
short-term interests of the users is the distros' problems, while they focus 
on the long-term and the big picture.

Both are doing what they believe to be in the best interests of the users.  
Bill believes (and keeps uselessly repeating) that the kernel developers 
should pay more attention to short-term interests.  The whole "stable series" 
vs "continuous development" is about short-term interests vs long-term 
interests.

Overall, development of new technology (and adoption of new technology) goes 
faster when it's continuous than when you have large discontinuous jumps.  
You find problems faster, and you find them one at a time when it's easy to 
isolate what's wrong rather than receiving three years of development in one 
gulp and experiencing fifteen different failures all at once.  Problems are 
more frequent, but they're smaller and simpler and easier to diagnose and 
easier to fix.  User can _cope_ with a higher rate of change when it comes in 
smaller chunks.  The learning curve isn't a cliff.

> They don't owe you security fixes either.

Actually, they seem to think they do.  They're quite dilligent about providing 
security fixes.  But the security fixes they give are part of the ongoing 
development process.  Separating them out and backporting them to previous 
versions is your problem, and if you don't feel up to it they point you to 
distributions, which will do that for you.

So there are two different ways you can get the fixes.  (Upgrade, or use a 
distro maintained kernel.)  Some people think they're owed a third way, and 
want somebody else to provide it for them.

> Your "ivory tower" statement is really condescending.

*shrug*  This whole thread is basically people bitching about what other 
people should do, instead of banding together and giving it a try themselves.  
It started condescending.  In the absence of constructive suggestions or 
anyone actually doing real work rather than merely bitching about the state 
of the world, I think most of the developers have written off the whiners 
with a silent "sucks to be you" and moved on...

(I do hope we get the "buffer flush" dot-releases though.  That would be 
nice.)

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Saturday 10 December 2005 07:41, Bill Davidsen wrote:
> Given that CL has minimal (essentially no) maintenence cost,

then by your own admission maintaining it as an external patch for people who 
need it for legacy reasons is trivial, and removing it to discourage new 
users from picking it up remains a good idea.

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Ryan Anderson]

On Sat, Dec 03, 2005 at 04:23:39PM +0100, Adrian Bunk wrote:
> On Sat, Dec 03, 2005 at 03:36:38PM +0100, Arjan van de Ven wrote:
> 
> > If the current model doesn't work as you claim it doesn't, then maybe
> > the model needs finetuning. Right now the biggest pain is the userland
> > ABI changes that need new packages; sometimes (often) for no real hard
> > reason. Maybe we should just stop doing those bits, they're not in any
> > fundamental way blocking general progress (sure there's some code bloat
> > due to it, but I guess we'll just have to live with that).
> 
> IOW, we should e.g. ensure that today's udev will still work flawlessly 
> with kernel 2.6.30 (sic)?
> 
> This could work, but it should be officially announced that e.g. a 
> userspace running kernel 2.6.15 must work flawlessly with _any_ future 
> 2.6 kernel.
> 
> For how many years do you think we will be able to ensure that this will 
> stay true?

I'd rather see the statement that if a kernel 2.6.N requires a userspace
update (udev, alsa, whatever), that the new userspace works correctly on
2.6.(N-10).

I think that is the bit of the problem that has been really frustrating
to the people that have run into it.

(I think that is the complaint Dave Jones made during his OLS keynote,
and I've seen a similar complaint about udev, though the udev issue
may have been Debian specific.)

-- 

Ryan Anderson
  sometimes Pug Majere

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Douglas McNaught]

Bill Davidsen <davidsen@tmr.com> writes:

> Rob Landley wrote:
>
>> Re-raising the same objections over and over again when they've
>> already been aired, considered, and rejections is called "whining".
>>
> Repeating the same information over and over until it sinks in is
> called "rote learning." The question is not if cryptoloop is perfect,
> every crypto seems to fail eventually, recently md5 was cracked,
> etc. But people have used cryptoloop now, and removing it from the
> kernel will lock them out of their own data, or prevent them from
> moving forward. There's no replacement for cryptoloop, so I can't just
> reconfigure X and still read my 147 DVDs full of business data, or
> access the current data on 34 laptops around the country.

Bill, I still don't think your complaints are justified.

You're only "locked out of your own data" if you knowingly upgrade to
a kernel that doesn't support cryptoloop.  Nobody's forcing you to do
that. 

The kernel developers owe *nothing* to J. Random User.  They are
either doing what they do for their own reasons (the "fun" of it), or
being paid by an organization with specific objectives (even if, in
Linus' case, the objective is just "make the best kernel possible,
based on your judgement and that of people you trust").

That said, of course none of them want to break things unnecessarily.
But they make technical decisions, with the goal of having the best
kernel, that do sometimes have painful consequences.  You're free, of
course, to disagree with those decisions and maintain your own kernel.

They don't owe you security fixes either.  Sorry, but that's the way
it is.  We're all lucky that they take security very seriously and
respond quickly to problems.

> In most cases CL is not expected to protect against goverment agencies
> but rather stolen laptops in airports (yes the pros have added MacOS
> and Linux to their business model) or the occasional lost DVD in the
> mail. Removing CL is not a hell of a lot better morally than these
> viruses which encrypt your data and then hold it for ransom, with the
> price being doing without security fixes in future kernels.

That last sentence is crap.

You're free to backport security fixes to cryptoloop-supporting
kernels forever, or pay someone to do so.  Or maintain a cryptoloop
patch against current kernels, or pay someone to do so.  Or write a
converter for cryptoloop data to whatever's currently in the kernel,
or pay someone to do so.

> Given that CL has minimal (essentially no) maintenence cost, I wish
> the ivory tower developers could understand that real people have
> invested real money in it, and real data in the technology. Since
> there is no alternative solution offered, CL is far better than no
> crypto at all, and I wish there were a few more developers who had
> experience working in the real word.

If you include a crypto solution in the mainstream kernel, you're in
some sense endorsing its security.  If that solution has known
weaknesses, I can understand wanting to either fix it or rip it out.
Crypto is hard enough to get right as it is.

Your "ivory tower" statement is really condescending.  Linux is way
past the stage where college students were the main contributors (if
it ever was so after Linus graduated). and a great majority of
developers now are paid to work on the kernel.  There are probably
very few of them that don't have at least a little sysadmin
experience.

If you've invested money and put important data in a system, and you
haven't contracted with anyone to support that system, supply security
fixes, and make sure it does what you want it to do, who's the fool?

Basically, you're complaining about something you get *for free* that
represents millions of hours of work, because it doesn't work quite
the way you want it to, when you have perfect freedom to make it meet
your needs by putting in your own time, effort and/or money.

-Doug

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Rob Landley wrote:

>
>I'm under the impression the problem with cryptoloop is bad cryptography:
>http://lwn.net/Articles/67216/
>
>Anybody actually using cryptography with an expoitable weakness needs all the 
>wake-up calls they can get.  This is _not_ a case where you want to support 
>old broken crap, that defeats the whole purpose of using cryptography in the 
>first place.
>
>Especially the cryptoloop removal was an intentional decision that the kernel 
>developers made.  People raised their objections at the time, and these were 
>taken into consideration when making the decision:
>http://kerneltrap.org/node/2433
>
>Re-raising the same objections over and over again when they've already been 
>aired, considered, and rejections is called "whining".
>
Repeating the same information over and over until it sinks in is called 
"rote learning." The question is not if cryptoloop is perfect, every 
crypto seems to fail eventually, recently md5 was cracked, etc. But 
people have used cryptoloop now, and removing it from the kernel will 
lock them out of their own data, or prevent them from moving forward. 
There's no replacement for cryptoloop, so I can't just reconfigure X and 
still read my 147 DVDs full of business data, or access the current data 
on 34 laptops around the country.

In most cases CL is not expected to protect against goverment agencies 
but rather stolen laptops in airports (yes the pros have added MacOS and 
Linux to their business model) or the occasional lost DVD in the mail. 
Removing CL is not a hell of a lot better morally than these viruses 
which encrypt your data and then hold it for ransom, with the price 
being doing without security fixes in future kernels.

Given that CL has minimal (essentially no) maintenence cost, I wish the 
ivory tower developers could understand that real people have invested 
real money in it, and real data in the technology. Since there is no 
alternative solution offered, CL is far better than no crypto at all, 
and I wish there were a few more developers who had experience working 
in the real word.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Fri, Dec 09, 2005 at 08:16:42PM -0600, Rob Landley wrote:
> As far as I can tell, what broke with udev was their embedded version of 
> "libsysfs", which is an abstraction layer I've _never_ understood the point 
> of.  (Because opening single value files in /sys is just too hard.  Nobody 
> needed a "libproc", the parsing of which is actual work, but they felt a need 
> a libsysfs.  Uh-huh...)

The original goal of libsysfs was to have a library that handled all of
the direct sysfs calls, and have it create structures that looked
something like what sysfs exported (devices, busses, etc.)

Then, if things changed in the sysfs layout or structure, only libsysfs
would need to be changed, and all apps that used it would continue to
work just fine.

But in reality, it was only used by udev, and was very fragile.  So
fragile udev is thinking of ripping it out as it has had a lot of
problems in the past.

Hope this explains things better.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Tuesday 06 December 2005 08:58, Bill Davidsen wrote:
> Greg KH wrote:
> > On Sun, Dec 04, 2005 at 04:46:31AM +0000, Luke-Jr wrote:
> >>Well, devfs does have some abilities udev doesn't: hotplug/udev
> >>doesn't detect everything, and can result in rarer or non-PnP devices
> >>not being automatically available;
> >
> > Are you sure about that today?  And udev wasn't created to do everything
> > that devfs does.  And devfs can't do everything that udev can (by
> > far...)
> >
> >>devfs has the effect of trying to load a module when a program looks
> >>for the devices it provides-- while it can cause problems, it does
> >>have a possibility to work better.
> >
> > Sorry, but that model of loading modules is very broken and it is good
> > that we don't do that anymore (as you allude to.)
> >
> >>Interesting effects of switching my desktop from devfs to udev:
> >>1. my DVD burners are left uninitialized until I manually modprobe ide-cd
> >> or (more recently) ide-scsi
> >
> > Sounds like a broken distro configuration :)
> >
> >>2. my sound card is autodetected and the drivers loaded, but the OSS
> >> emulation modules are omitted; with devfs, they would be autoloaded when
> >> an app tried to use OSS
> >
> > Again, broken distro configuration :)
>
> If a new udev config is needed with every new kernel, why isn't it in
> the kernel tarball?

Why isn't inittab in the kernel tarball?

I have a shell script that initializes /dev.  (I've posted it here a few 
times, somebody ported it to C, and a micro-udev replacement will go into 
busybox in 1.2.)

Why isn't there a command shell in the kernel tarball?  Kinda hard to use your 
system without a shell...

As far as I can tell, what broke with udev was their embedded version of 
"libsysfs", which is an abstraction layer I've _never_ understood the point 
of.  (Because opening single value files in /sys is just too hard.  Nobody 
needed a "libproc", the parsing of which is actual work, but they felt a need 
a libsysfs.  Uh-huh...)

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Tuesday 06 December 2005 05:21, Matthias Andree wrote:
> On Tue, 06 Dec 2005, Florian Weimer wrote:
> > From a vendor POV, the lack of official kernel.org advisories may be a
> > feature.  I find it rather disturbing, and I'm puzzled that the kernel
> > developer community doesn't view this a problem.  I know I'm alone,
>
> You're not alone in viewing this as a problem, but QA is a burden kernel
> developers are not interested in. But it is necessary.

If you want to run a big automated regression test against the kernel, 
exercising the full API and immediately catching any regressions, go right 
ahead.  Nobody's stopping you and you don't need our permission anyway.  The 
Linux Test Project is working on something like this already, and ODSL does 
some of this to.  (It's not like QA is being ignored.)

The problem is that the bulk of the kernel code is device drivers, and nobody 
has all the strange and esoteric hardware that the drivers push.  Nope, not 
even IBM.  I doubt any one organization anywhere on the planet has 
_everything_ the kernel has been used to drive.
.

> QA has to happen at all levels if it is supposed to be affordable or
> scalable. The development process was scaled up, but QA wasn't.
>
> How about the Signed-off-by: lines? Those people who pass on the changes
> also pass on the bugs, and they are responsible for the code - not only
> license-wise, but also quality-wise. That's the latest point where
> regression tests MUST happen.

I can't test your setup for you.  I haven't got your setup.  All I can tell 
you is that it worked for me.

I spent most of a week last month fighting to get User Mode Linux 2.6.15-rc1 
through rc4 to compile and run on both x86 ubuntu and x86-64 PLD.  Different 
versions of GCC compiled the darn interface code differently (there's a 
section where it switches stacks and gcc kept trying to touch the stack in 
the middle of this, and segfaulting).  Worked fine for Jeff Dike and 
Blaisorblade, because they weren't using a semi-obsolete version of ubuntu.

Over on PLD, I had a fight just to get it to _compile_, because the header 
files were all different (PLD uses Mazur's cleaned up 2.6 headers which 
uClibc systems also use, while most things use the glibc package, and at one 
point they had userspace and kernel space headers reversed and it worked fine 
with the glibc kernel headers but Mazur's headers really are cleaned up and 
don't leak nearly so much kernel stuff into userspace).  And then /lib wasn't 
a symlink to /lib64 (it is on Fedora and Debian, but on PLD they're separate 
directories) so the link path had to be adjusted (/lib64 was the correct 
directory for a 64-bit build and should be checked first).  Then getting it 
to run had another half-dozen problems with various interface code: for some 
reason on PLD page_size was linked as a function call when they expected it 
to be a constant...

Another fun little thing is just a performance issue: UML gets its "physical 
memory" from an mmap file (easy to share between processes), but if that file 
isn't on tmpfs then every page UML dirties gets scheduled for writeout, over 
and over again, keeping the hard drive constantly busy and slowing the system 
to a crawl.   Of course it _works_, but so it's hard to pin down what the 
problem is.  (UML isn't slowed down, the rest of the system is by the 
unnecessary I/O.)  Again, on Jeff's system /tmp is a tmpfs mount.  On most 
systems, /dev/shm is a tmpfs mount and /tmp inherits /.  (Meaning on knoppix 
it's tmpfs, but on Fedora or Ubuntu or Gentoo, it isn't by default.  Unless 
the sysadmin has changed it, which many sysadmins will.)  And strangely, on 
the PLD system I'm borrowing /dev/shm isn't tmpfs, so changing the default is 
the right thing but it needed an improved error message.

It all worked just _fine_ for the people who wrote it.  (And continues to.)  
And all of this is why there was an -rc1, so people like me could try it and 
report that it didn't work the same way for us and spend a week figuring out 
all the various different _ways_ it didn't work.

This isn't the full set of bugs I plowed through.  I had a version at one 
point that ran fine, gave me a command shell (init=/bin/sh) and I reported 
success and then came back the next day with "nope, fork segfaults".  
(Actually it was exec segfaulting.)  The shell _did_ come up fine.  (And echo 
$USER hadn't actually had to exec anything...)  But that wasn't the end of 
it.

The thing is, me spending all this time making sure it worked _for_me_ was 
something that I did on my own time, voluntarily.  I'm not really a UML 
developer, I have too much to do elsewhere.  If I hadn't done this, would it 
work on ubuntu and PLD right now?  Maybe.  I don't know.  But it already 
worked for Jeff Dike when he checked it in.  Worked just fine.  Because he 
didn't have the environment I had.  He could find _none_ of these problems 
because the bugs only manifest in an environment he doesn't have.

And all this is a _rounding_error_ compared to the kernel as a whole.  This is 
just one little corner of it, in one little release, where one person spent 
one week debugging on just two systems.

And this wasn't even hardware dependent!  (Or an intermittent problem that you 
_think_ is fixed because you haven't seen it, or something requiring a 
particularly arduous reproduction sequence like a 40 hour calculation, or 
access to a machine that's only available thursdays from 2-4 am...)

You seem to _deeply_ misunderstand nature of the problem.

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Wed, 07 Dec 2005, Dmitry Torokhov wrote:

> On Wednesday 07 December 2005 06:29, Matthias Andree wrote:
> > What I'm saying is that people (maintainer) should have a selected
> > number of people (users) test the patches before they are merged.
> 
> And we try. Take for example psmouse_resync patch that is now in -mm.
> I got about 30 reports that it worked and fixed people's problems before
> I got it to Andrew. And still as soon as it got to -mm I got a complaint
> that it failed on one of boxes ;(

The important thing is to get these failing boxes into the regular test
set. I know that's not always easy, because end users tend to go away as
soon as it works again for them.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Dmitry Torokhov]

On Wednesday 07 December 2005 06:29, Matthias Andree wrote:
> On Tue, 06 Dec 2005, Dmitry Torokhov wrote:
> 
> > On 12/6/05, Matthias Andree <matthias.andree@gmx.de> wrote:
> > >
> > > QA has to happen at all levels if it is supposed to be affordable or
> > > scalable. The development process was scaled up, but QA wasn't.
> > >
> > > How about the Signed-off-by: lines? Those people who pass on the changes
> > > also pass on the bugs, and they are responsible for the code - not only
> > > license-wise, but also quality-wise. That's the latest point where
> > > regression tests MUST happen.
> > 
> > People who pass the changes can only test ones they have hardware for.
> > For the rest they can try to validate the code by reading patches but
> > have to rely on the submitter wrt to the patch actually working.
> 
> What I'm saying is that people (maintainer) should have a selected
> number of people (users) test the patches before they are merged.
> 

And we try. Take for example psmouse_resync patch that is now in -mm.
I got about 30 reports that it worked and fixed people's problems before
I got it to Andrew. And still as soon as it got to -mm I got a complaint
that it failed on one of boxes ;(

-- 
Dmitry

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Nix]

On 6 Dec 2005, Rob Landley moaned:
> On Saturday 03 December 2005 17:35, Chris Wright wrote:
>> relevant.  About the only thing I think is helpful in this case is perhaps
>> one extra -stable cycle on the last branch when newest branch is released
>> (basically flush the queue).  That much I'm willing to do in -stable.
> 
> Yay rah cool!

Seconded (thirded?), this is a very good idea (and as it's just a queue
flush is probably quite easy to do).

That way those of us who are paranoid can upgrade our experimental boxes
immediately, apply the latest -stable to the non-experimental boxes, and
then cautiously upgrade those boxes when the experimental ones seem to
be working OK. Currently whenever there's a non-stable kernel rev I'm
filled with trepidation: do I upgrade the stable boxes and risk
instability, or leave them as they are and risk insecurity?

-- 
`Y'know, London's nice at this time of year. If you like your cities
 freezing cold and full of surly gits.' --- David Damerell

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Arjan van de Ven <arjan@infradead.org> wrote:

> > The other group is the people who use and depend on some feature, be it 
> > cryptoloop, 8k stacks, ndiswrapper, ipchains, whatever... which is 
> > scheduled for extinction. 

Come on, most of those were scheduled for deletion a /long/ time ago.

> these are actually 2 groups
> 
> 1) people who depend on an in-kernel features
> 
> 2) people who depend on out of kernel / binary modules
> 
> treating them as one is not correct or fair to this thread.

Right.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Tuesday 06 December 2005 12:51, Bill Davidsen wrote:
> Just so we're all on the same page, I think there are two sets of
> unhappy people here... one is the group who want new stuff fast and
> stable. For the most part that's not me, although I was in the "if
> you're going to add ipw2200 support, why not something that works?"
> group. But new stuff is going in faster than most people can assimilate
> it if they have a real job, so I don't see too much problem there.

My laptop has an ipw2200 but I can't get it to work in any kernel I built 
because the kernels I build aren't modular.  I hope to be able to work around 
this someday with a clever enough initramfs (if necessary, moving the 
initramfs initialization earlier in the boot sequence), but it hasn't made it 
far enough up my todo list yet.

So whether or not the driver actually works if I could get it initialized is, 
for me, a moot point.

> The other group is the people who use and depend on some feature, be it
> cryptoloop, 8k stacks, ndiswrapper, ipchains, whatever... which is

Ndiswrapper isn't a kernel feature.  Don't confuse the shark with the remoras.

The only complaint about 8k stacks I've heard is the ndiswrapper people, and 
8k isn't actually sufficient for them anyway (_their_ ad-hoc spec guarantees 
12k), so they should have been swapping to their own stack all along.  They 
should probably even statically allocate the sucker at boot time and 
serialize all drivers using it with a semaphore, because I _really_ doubt 
it's ever been preempt-safe.

Isn't ipchains obsolete since 2.2?  it was already deprecated back in 2.4.  
There has been quite adequate warning on that one, thanks very much.

I'm under the impression the problem with cryptoloop is bad cryptography:
http://lwn.net/Articles/67216/

Anybody actually using cryptography with an expoitable weakness needs all the 
wake-up calls they can get.  This is _not_ a case where you want to support 
old broken crap, that defeats the whole purpose of using cryptography in the 
first place.

Especially the cryptoloop removal was an intentional decision that the kernel 
developers made.  People raised their objections at the time, and these were 
taken into consideration when making the decision:
http://kerneltrap.org/node/2433

Re-raising the same objections over and over again when they've already been 
aired, considered, and rejections is called "whining".

> scheduled for extinction. That's a departure from the way 2.{0,2,4} were
> done, where adds happened regularly, but features were only deleted in 
> development trees.

If features were really were deleted in development trees, devfs and ipchains 
never would have made it through 2.4.  So you're talking about an idealized 
version fo the past that doesn't match what people actually did back then. 

> Deleting features leaves anyone who can't keep their 
> own tree without security fixes.

Security fixes are a separate issue.

I asked for one more security fix to flush the pending fixes queue a while 
ago:
http://seclists.org/lists/linux-kernel/2005/Nov/4187.html

On Saturday, stable series co-maintainer Chris Wright decided it might be 
worth a try:
http://seclists.org/lists/linux-kernel/2005/Dec/0740.html

  > About the only thing I think is helpful in this case is perhaps 
  >  one extra -stable cycle on the last branch when newest branch is released 
  >  (basically flush the queue). That much I'm willing to do in -stable. 

To which I replied (and again I quote), "Yay, rah, cool!".

This gives you a trail of breadcrumbs to follow.  There are a series of 
incremental patches (which may have to be fixed up to apply) that address the 
known security-specific issues.

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Massimiliano Hofer]

On Wednesday 7 December 2005 5:05 pm, Horst von Brand wrote:

> You can certainly keep 2.6.x.y for a while when 2.6.(x+1) shows up, and
> even wait for 2.6.(x+1).1. Note that the stable series maintainers are
> sypmathetic to the idea of doing a last 2.6.x.(y+1), flushing the queued
> patches when 2.6.(x+1) shows up. Is this enough for you?

If a 2.6.x.1 is released and a vulnerability is discovered with the wrong 
timing, this leaves us with a kernel that has had little or no testing.

We already had a 2.6.x that didn't even boot on half my servers. When 2.6.x.1 
is the first bootable version and a security patch arrives, this leaves me 
with an uncomfortable choice between an old, stable and vulnerable version 
and a new, shiny and untested one.

Having 2.6.x-1.y and 2.6.x.y would avoid this situation.

-- 
Bye,
   Massimiliano Hofer

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Massimiliano Hofer <max@bbs.cc.uniud.it> wrote:

[...]

> I maintain a number of servers and don't like to depend on a distro for
> the kernel. I do my tests before deployment and can live with some
> problems in a specific release (noone is perfect), but I'd like to have a
> plan B without creating my own branch.

Reasonable.

> Having security patches in a 2.6.(x-1).y would allow me to test the
> latest vanilla AND have stable production servers without the rush that
> usually accompanies a new release followed by a vulnerability.

You can certainly keep 2.6.x.y for a while when 2.6.(x+1) shows up, and
even wait for 2.6.(x+1).1. Note that the stable series maintainers are
sypmathetic to the idea of doing a last 2.6.x.(y+1), flushing the queued
patches when 2.6.(x+1) shows up. Is this enough for you?
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> The other group is the people who use and depend on some feature, be it 
> cryptoloop, 8k stacks, ndiswrapper, ipchains, whatever... which is 
> scheduled for extinction. 

these are actually 2 groups

1) people who depend on an in-kernel features

2) people who depend on out of kernel / binary modules

treating them as one is not correct or fair to this thread.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Massimiliano Hofer]

On Tuesday 6 December 2005 6:44 pm, Greg KH wrote:

> > Now what? Do I as a user upgrade my production environment to the latest
> > and greatest kernel experiment, hope that the problems can be fixed
> > quickly, and hope that I don't lose too much data in the process?
>
> No, if you rely on a production environment for your stuff, stick with a
> disto kernel which has been tested and is backed up by a company that
> will maintain it over time.

If the purpose of not having a 2.7 branch or longer RCs is to have people test 
the latest vanilla, you can't simultaneously send users away.

I maintain a number of servers and don't like to depend on a distro for the 
kernel. I do my tests before deployment and can live with some problems in a 
specific release (noone is perfect), but I'd like to have a plan B without 
creating my own branch.

Having security patches in a 2.6.(x-1).y would allow me to test the latest 
vanilla AND have stable production servers without the rush that usually 
accompanies a new release followed by a vulnerability.

-- 
Bye,
   Massimiliano Hofer

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Matthias Andree <matthias.andree@gmx.de> wrote:
> On Tue, 06 Dec 2005, Dmitry Torokhov wrote:
> > On 12/6/05, Matthias Andree <matthias.andree@gmx.de> wrote:
> > > QA has to happen at all levels if it is supposed to be affordable or
> > > scalable. The development process was scaled up, but QA wasn't.

> > > How about the Signed-off-by: lines? Those people who pass on the changes
> > > also pass on the bugs, and they are responsible for the code - not only
> > > license-wise, but also quality-wise. That's the latest point where
> > > regression tests MUST happen.

> > People who pass the changes can only test ones they have hardware for.
> > For the rest they can try to validate the code by reading patches but
> > have to rely on the submitter wrt to the patch actually working.

> What I'm saying is that people (maintainer) should have a selected
> number of people (users) test the patches before they are merged.

Each one gets the patches from people who tested them on their machines
(presumably), and the maintainer signs them off for code decency and (if he
has the hardware) working on his machine too. It is very hard to get much
more than that in terms of early testers. Again, one of the standard
complaints here is that very few test the -rcX kernels, and then scream
murder when the -final breaks. Now consider the case of proposed patches...

Yet, the solution is /very/ simple: Instead of complaining, set up a
machine with the hardware that particularly interests you, get on the
relevant lists, find out where the proposed patches are kept and test
them. You could even add a Signed-off-by: line to the patches you check
out. And again, "somebody should do..." won't get you anywhere, "here I
do..."  has more of a chance of success.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Tue, 06 Dec 2005, Dmitry Torokhov wrote:

> On 12/6/05, Matthias Andree <matthias.andree@gmx.de> wrote:
> >
> > QA has to happen at all levels if it is supposed to be affordable or
> > scalable. The development process was scaled up, but QA wasn't.
> >
> > How about the Signed-off-by: lines? Those people who pass on the changes
> > also pass on the bugs, and they are responsible for the code - not only
> > license-wise, but also quality-wise. That's the latest point where
> > regression tests MUST happen.
> 
> People who pass the changes can only test ones they have hardware for.
> For the rest they can try to validate the code by reading patches but
> have to rely on the submitter wrt to the patch actually working.

What I'm saying is that people (maintainer) should have a selected
number of people (users) test the patches before they are merged.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[David S. Miller]

From: Greg KH <greg@kroah.com>
Date: Tue, 6 Dec 2005 09:47:14 -0800

> On Tue, Dec 06, 2005 at 05:55:42PM +0100, Felipe Alfaro Solana wrote:
> > > There might be some subtle changes in the netfilter/routing
> > > interaction which break user configurations, but this still being
> > > tracked down (and maybe the any behavior is fine because it's
> > > unspecified; hard to tell).
> > 
> > Yeah! For example, the first datagram triggering an IPSec SA is always
> > lost (instead of being queued until the IPSec SA has been
> > established).
> > 
> > For example, try pinging the IPSec SA peer for the very first time and
> > the first ICMP datagram will always return "resource currently
> > unavailable" and, of course, will get lost.
> > 
> > BTW this works perfectly under *BSD and Mac OS X.
> 
> Do the network kernel developers know about this issue?  And if so, what
> have they said about it?

It's on the TODO list, known problem with not an easy solution.

BTW, BSD doesn't do any better, the KAME BSD ipsec stack drops the
initial datagram just like we do.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[John Kelly]

On Tue, 06 Dec 2005 22:55:26 +0100, Adrian Bunk <bunk@stusta.de>
wrote:

>> So if Adrian wants to begin where -stable ends, there is no reason for
>> people to oppose his efforts.

>I've read the whole thread, and I haven't seen anyone opposing my idea.

>Most people in this thread who did or do maintain some kernel branch 
>simply expressed that in their opinion my idea would be too much work 
>for too few users...

If you build it, will they come?

No one really knows.  There is only one way to find out.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Tue, Dec 06, 2005 at 01:57:55PM -0500, John Kelly wrote:
> On Tue, 06 Dec 2005 09:54:22 -0800, Greg KH <greg@kroah.com> wrote:
> 
> >I don't think people realize that the -stable series only contains
> >patches that other people send us.  For the most part, Chris and I
> >aren't going out there and activly writing up fixes for some of these
> >issues, as we both don't have the time and energy to do this.
> 
> The -stable series was a good idea, and thanks for doing it.  Adrian's
> idea is a natural extension of what you started.
> 
> 
> >And also, anyone else can easily take over maintaining these kernel
> >branches.  The git trees are public, as is our stable patch queue.  So
> >if anyone wants to maintain older kernels, it is quite easy to start the
> >process.
> 
> So if Adrian wants to begin where -stable ends, there is no reason for
> people to oppose his efforts.

I've read the whole thread, and I haven't seen anyone opposing my idea.

Most people in this thread who did or do maintain some kernel branch 
simply expressed that in their opinion my idea would be too much work 
for too few users...

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Kay Sievers]

On Tue, Dec 06, 2005 at 04:10:12PM -0500, Bill Davidsen wrote:
> Greg KH wrote:
> >On Tue, Dec 06, 2005 at 09:58:54AM -0500, Bill Davidsen wrote:
> >
> >>If a new udev config is needed with every new kernel, why isn't it in
> >>the kernel tarball? Is that what you mean by "broken distro
> >>configuration?" The info should be in /proc or /sys and not in an
> >>external config file, particularly if a different versions per-kernel is
> >>needed and people are trying new kernels and perhaps falling back to the
> >>old.
> >
> >Every distro has different needs for its device naming and groups and
> >other intergration into the boot process.  To force all of them to unify
> >on one-grand-way-of-doing-things would just not work out at all.
> 
> Did I say that. No, I said it would be desirable to provide a working 
> config with the kernel, to which something could be symlinked. This no 
> more "forces" distributions to do anything than LSB. It would provide a 
> default, it would provide something working, and if I didn't like it I 
> could change it. But I wouldn't have to try and change thing way up in 
> initrd so I can boot one kernel or another...

That already works today. All distros I know are capable to run
kernel.org kernels, if you care yourself, that the rootfs is accessible.

> >Look at all of the variations in the udev tarball between the different
> >vendor configurations (we put them in there for other people to base
> >their distro off of, if they want to.)
> >
> >So providing this config in the kernel will just not work, sorry.
> 
> We have standard libraries, header files, system calls, why is a 
> standard in this case a bad thing? Actually not even a standard, 
> perhaps, a default. It wouldn't make it one bit harder to have custom 
> names, for those who believe different is better.

Just give it its time. It will happen without anybody claiming to have
or to be a standard. We are already much more similar than we've ever
been across the current devel releases of all major distros.
The complete replacement of hotplug by udev rules, kernel uevents and
kernel event replay triggers made the situation so much simpler and better
than it ever was.

It's, as in most cases, not about someone defining some standard, talk a
lot about it, create an interest group, write a noisy paper... - it's the
whole lot of real work to come up with a solution that is convincing enough
for the involved parties, and to manage all the different interests people
have and still get things done at the same time. It has almost nothing
to do with a "standard".

Convergence will just happen, cause it makes sense and there is a reasonable
way to do it. And there was no such thing in that area in the past that
made this kind of sense.

And yeah, the never ending discussion about stupid options like devfs
does not help anything here. It should have been removed a long time
ago, so that the the confused people start to walk in the right direction
instead of delaying the needed convergence progress again and again.

Thanks,
Kay

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Greg KH wrote:
> On Mon, Dec 05, 2005 at 04:17:53PM +0100, Jakob Oestergaard wrote:
> 
>>On Sun, Dec 04, 2005 at 02:39:31PM -0800, Greg KH wrote:
>>
>>>On Sun, Dec 04, 2005 at 06:00:49PM +0100, Jakob Oestergaard wrote:
>>>
>>>>In the real world, however, admins currently need to pick out specific
>>>>versions of the kernel for specific workloads (try running a large
>>>>fileserver on anything but 2.6.11.11 for example - any earlier or later
>>>>kernel will barf reliably.
>>>
>>>Have you filed a but at bugzilla.kernel.org about this?  If not, how do
>>>you expect it to get fixed?
>>
>>I don't expect to get it fixed. It's futile. It can get fixed in one
>>version and broken two days later, and it seems the attitude is that
>>that is just fine.
> 
> 
> Huh?  That is just not true at all.  Please give us a bit more credit
> than that.
> 
> 
>>After a long long back-and-forth, 2.6.11 was fixed to the point where it
>>could reliably serve files (at least on uniprocessor configurations -
>>and in my setup I don't see problems on NUMA either, but as far as I
>>know that's just me being lucky).
>>
>>Right after that, someone thought it was a great idea to pry out the PCI
>>subsystem and shovel in something else.  Find, that's great for a
>>development kernel, but for a kernel that's supposed to be stable it's
>>just not something you can realistically do and expect things to work
>>afterwards.  And things broke - try mounting 10-20 XFS filesystems
>>simultaneously on 2.6.14.  Boom - PCI errors.
> 
> 
> What PCI errors are you speaking of?  We did that PCI work to fix a lot
> of other machines that were having problems.  And yes, this did break
> some working machines, and we are very sorry about this.  But in the
> future, changes to this area will not cause this to happen due to the
> changes made.

I don't think it's reasonable to get overly upset about *accidental* 
breakages. People make mistakes, otherwise you don't get progress.

Note that I haven't changed my mind about deliberately removing features 
for which there is no practical alternative.
-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Greg KH wrote:
> On Tue, Dec 06, 2005 at 09:58:54AM -0500, Bill Davidsen wrote:
> 
>>If a new udev config is needed with every new kernel, why isn't it in
>>the kernel tarball? Is that what you mean by "broken distro
>>configuration?" The info should be in /proc or /sys and not in an
>>external config file, particularly if a different versions per-kernel is
>>needed and people are trying new kernels and perhaps falling back to the
>>old.
> 
> 
> Every distro has different needs for its device naming and groups and
> other intergration into the boot process.  To force all of them to unify
> on one-grand-way-of-doing-things would just not work out at all.

Did I say that. No, I said it would be desirable to provide a working 
config with the kernel, to which something could be symlinked. This no 
more "forces" distributions to do anything than LSB. It would provide a 
default, it would provide something working, and if I didn't like it I 
could change it. But I wouldn't have to try and change thing way up in 
initrd so I can boot one kernel or another...
> 
> Look at all of the variations in the udev tarball between the different
> vendor configurations (we put them in there for other people to base
> their distro off of, if they want to.)
> 
> So providing this config in the kernel will just not work, sorry.

We have standard libraries, header files, system calls, why is a 
standard in this case a bad thing? Actually not even a standard, 
perhaps, a default. It wouldn't make it one bit harder to have custom 
names, for those who believe different is better.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Lars Marowsky-Bree wrote:
> On 2005-12-05T14:30:09, Bill Davidsen <davidsen@tmr.com> wrote:
> 
> 
>>Actually I would be happy with the stability of this series if people 
>>would stop trying to take working features OUT of it!
> 
> 
> Features are removed when they are no longer features, but design
> irritations in a new and improved design, and usually, equivalent or
> better (or at least thought to be) functionality is available still in
> the big picture (which includes user-space), hopefully in a cleaner
> place.
> 
> Now, design is often a holy war, and people disagree. That's fine and to
> be expected. And sometimes, the whole solution takes a while to
> materialize and be implemented from the kernel up to all user-space and
> even longer until it has been implemented in the brains of the admins.
> This, too, is fine and expected. It's called "innovation" and
> "development", sometimes iterative.

Removing features because there are better solutions is one thing, 
although it has been done at kernel tree changes for a decade. Removing 
features for reasons of religion is rather a case of developers removing 
a useful and unbroken feature for which there is no replacement purely 
because someone doesn't like it, or it saves a dozen lines of code.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matan Peled]

Richard Knutsson wrote:
> But I do wonder how copyright and GPL can co-exist. Do the copyright 
> holder own the changes anybody else does to the code?
> Anyone care to explain?

IANAL, but GPL is a copyright license. Copyright is the right to make copies of 
somethings, to distribute it to be precise.

So if I write foo.c and release it under the GPL, and JR Hacker takes it and 
writes foo++.c but doesn't give his super duper sekrit version to anyone, then 
he isn't bound by copyright laws (he isn't making copies) and therefore the GPL 
doesn't hold for him.

The moment he wants to give a copy to his best friend, the GPL does kick in, 
though, and he has to abide by the GPL and distribute the whole piece (as it is 
a "derivative work") with the source code included (or an offer, or whatever. 
Read the GPL sometime, its not legalese at all).

For more information, ask your friendly (*cough*) neighborhood lawyer.

-- 
[Name      ]   ::  [Matan I. Peled    ]
[Location  ]   ::  [Israel            ]
[Public Key]   ::  [0xD6F42CA5        ]
[Keyserver ]   ::  [keyserver.kjsl.com]
encrypted/signed  plain text  preferred

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Alan Cox]

On Maw, 2005-12-06 at 01:43 +0100, Florian Weimer wrote:
> As far as I know, many of the recent CVE assignments for kernel
> vulnerabilities have been done by MITRE, requested by individuals
> which are neither known as kernel developers, nor vendor security
> folks (for "vendor" as in "we have our own legal department with real
> lawyers").

Most of them will be because vendors employ security professionals to
handle security CVE work and do all the tedious and terribly important
tracking of bugs v releases and what needs to be fixed by whom and when
- and developers to write code.

> Maybe the source of CVE assignments paints a wrong picture.  But if
> the CVE picture is correct, vendor-paid kernel developers help behind
> the scenes, but there is little interest in openly documenting
> security issues, so that users (and what kernel.org considers fringe
> distros) can apply the relevant patches if they use kernel.org
> kernels.

The 2.6.x.y maintainers are directly involved in security@kernel.org
last time I checked.

> database.  But the only answers we get is that everything is fine,
> vendors handle the situation, security@kernel.org actually does this
> already, etc.

Having someone doing that on kernel.org sounds a good plan

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Alan Cox]

On Sul, 2005-12-04 at 19:09 -0800, Joel Becker wrote:
> On Sun, Dec 04, 2005 at 05:17:09PM +0100, Matthias Andree wrote:
> > There are things that old Sun Workshop versions bitch about that GCC
> > deals with without complaining, and I'm not talking about C99/C++-style
> > comments. C standard issue? I believe not.
> 
> 	I have seen many a code like so:
> 
>     char buf[4];
>     memcpy(buf, source, 5);
> 
> accepted by the Sun compilers and run just fine.  When the application
> was ported to Linux/GCC, the developers complained their program
> segfaulted, and "it must be something broken on Linux!"
> 	Just because Sun's compiler does something doesn't mean it's

It isnt the compiler quite often. The usual case is

	char buf[4];
	strcpy(buf, "bits");

And those cases usually work because its a big endian box and the \00
ends up overwriting the \00 in the return address.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Horst von Brand wrote:
> Matthias Andree <matthias.andree@gmx.de> wrote:
> 
>>On Sat, 03 Dec 2005, David Ranson wrote:
>>
>>>Adrian Bunk wrote:
>>>
>>>
>>>>- support for ipfwadm and ipchains was removed during 2.6
> 
> 
>>>Surely this one had loads of notice though? I was using iptables with
>>>2.4 kernels.
> 
> 
> Sure had. They were scheduled for removal in march, 2005 a long time ago.
> 
> 
>>So was I. And now what? ipfwadm and ipchains should have been removed
>>from 2.6.0 if 2.6.0 was not to support these.
> 
> 
> Or in 2.6.10, or 2.6.27, or whatever.
> 
> 
>>                                              That opportunity was
>>missed, the removal wasn't made up for in 2.6.1, so the stuff has to
>>stick until 2.8.0.
> 
> 
> Sorry, but the new development model is that there is no "uneven" series
> anymore. Sure, it /might/ open for worldshattering changes, but nothing of
> that sort is remotely in sight right now, so...
> 
> 
>>>>- devfs support was removed during 2.6
>>>
>>>Did this affect many 'real' users?
> 
> 
>>This doesn't matter. A kernel that calls itself stable CAN NOT remove
>>features unless they had been critically broken from the beginning. And
>>this level of breakage is a moot point, so removal is not justified.
> 
> 
> devfs was broken, and very little used.

Perhaps there is a cause and effect relationship? If devfs worked I 
don't see the need for every distro to have it's own udev (or mdev or 
sdev or whatever the flavor is this month).
-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Jeff Garzik wrote:
> Bill Davidsen wrote:
> 
>> I do think the old model was better; by holding down major changes for 
>> six months or so after a new even release came out, people had a 
>> chance to polich the stable release, and developers had time to 
>> recharge their batteries so to speak, and to sit and think about what 
>> they wanted to do, without feeling the pressure to write code and 
>> submit it right away. Knowing that there's no place to send code for 
>> six months is a great aid to generating GOOD code.
> 
> 
> It never worked that way, which is why the model changed.
> 
> Like it or not, developers would only focus on one release.  In the old 
> model, unstable things would get shoved into the stable kernel, because 
> people didn't want to wait six months.  And for the unstable kernel, it 
> would often be so horribly broken that even developers couldn't use it 
> for development (think 2.5.x IDE).

I was actually thinking of Rusty's module code... I do every time I have 
to build an initrd file by hand "Although the syntax is similar to the 
older /etc/modules.conf, there are many features missing."

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Tuesday 06 December 2005 04:34, Luke-Jr wrote:
> > > Nope, but I don't see how udev can possibly detect something that
> > > doesn't let the OS know it's there-- except, of course, loading the
> > > driver for it and seeing if it works.
> >
> > Stuff shows up in /sys whether or not Linux has a driver loaded for it.
>
> Only if Linux is aware it exists. I'm thinking of those old ISA cards and
> such.

A) This is only true for obsolete hardware.  Can you name an example that's 
currently being manufactured?  (I'm trying to figure out if serial mice 
count, not that you really need kernel support to detect those.)

B) You can insmod the module from userspace to actively probe for stuff.  If 
the kernel doesn't know either until it probes, and you can trigger the probe 
at will, what additional kernel support do you need?

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lee Revell]

On Tue, 2005-12-06 at 15:32 +0100, Florian Weimer wrote:
> * Greg KH:
> 
> > What are we breaking that people are complaining so much about?
> > Specifics please.
> 
> Drastic performance changes in certain pipe usage patterns.  This was
> probably too early in the 2.6 series to count, though.

Where's the bug report?

Lee

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[John Kelly]

On Tue, 06 Dec 2005 09:54:22 -0800, Greg KH <greg@kroah.com> wrote:

>I don't think people realize that the -stable series only contains
>patches that other people send us.  For the most part, Chris and I
>aren't going out there and activly writing up fixes for some of these
>issues, as we both don't have the time and energy to do this.

The -stable series was a good idea, and thanks for doing it.  Adrian's
idea is a natural extension of what you started.


>And also, anyone else can easily take over maintaining these kernel
>branches.  The git trees are public, as is our stable patch queue.  So
>if anyone wants to maintain older kernels, it is quite easy to start the
>process.

So if Adrian wants to begin where -stable ends, there is no reason for
people to oppose his efforts.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Rob Landley wrote:
> On Monday 05 December 2005 10:28, Lee Revell wrote:
> 
>>>Things like this are all too regular an occurance.
>>
>>The distro should have solved this problem by making sure that the
>>kernel upgrade depends on a new wpa_supplicant package.  Don't they
>>bother to test this stuff before they ship it?!?
> 
> 
> I've broken stuff by upgrading glibc, upgrading X, upgrading KDE...
> 
> Upgrading the kernel is safer?  (Anybody remember 2.4.11-dontuse?)
> 
> Yay, modular component-based design.  We have standard interfaces so that 
> stuff mostly works when you swap out different versions (or entire different 
> components).  This is cool.
> 
> But if such interfaces were actually sufficient to specify all the 
> functionality you actually want to use, why would you ever need to upgrade a 
> component implementing that interface to a new version?
> 
> The real problem people are seeing is that the rate of change has increased.  
> Linus used to be a hell of a bottleneck, and this stopped being the case when 
> source control systems took over a lot of the patch tracking, ordering, 
> batching, and integration burden.
> 
> Automating the patch flow allowed entire subsystems to be effectively 
> delegated (and thus the "lieutenants" layer formed and was formalized), and 
> each of _them_ is now doing as much work as Linus used to do.
> 
> And _that_ is why there is no longer any point in -devel forks, because Linus 
> is now fielding as many patches in a month as he used to in a year.  That 
> means in every 3 months the Linux kernel undergoes as much development (in 
> terms of patches merged and lines of code changed) as an entire -devel series 
> used to do.  (Somebody confirm the numbers, these are approximations.)
> 
> There's no point in launching a fork that's only expected to last three 
> months.  Hence no -devel fork.

Just so we're all on the same page, I think there are two sets of 
unhappy people here... one is the group who want new stuff fast and 
stable. For the most part that's not me, although I was in the "if 
you're going to add ipw2200 support, why not something that works?" 
group. But new stuff is going in faster than most people can assimilate 
it if they have a real job, so I don't see too much problem there.

The other group is the people who use and depend on some feature, be it 
cryptoloop, 8k stacks, ndiswrapper, ipchains, whatever... which is 
scheduled for extinction. That's a departure from the way 2.{0,2,4} were 
done, where adds happened regularly, but features were only deleted in 
development trees. Deleting features leaves anyone who can't keep their 
own tree without security fixes. I see that as bed, and a far more 
important departure from the old model than the speed of new adds.
-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3767 bytes --]

Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 6 Dec 2005, Florian Weimer wrote:
> > > Actually, they are already maintaining 2.6.x.y, (x => 11, 12, ...)

> > I think the 2.6.x.y series is no longer maintained once 2.6.x+1 has
> > been released for some time (surely after 2.6.x+2).

> The same can still go for this, but instead of stopping at 2.6.x+2 we could
> stop at 2.6.x+6 (or +5), and just not care about 2.6.x+[1-4].  But that
> would be strong enough for those that would like the stable branch to
> maintain it themselves.  Currently it'l hard to pick a 2.6.x that you want
> to stay with since the 2.6.x.y is stopped right after 2.6.x+1 is out.  But
> if not all 2.6.x has a .y, then that would focus more distrobutions or
> whatever to pick the same one to support.

OK, let's step back and...

- People work on Linux (or whatever other stuff) because it is /fun/ and
  /exciting/. 
- People who don't actively work on a piece of code won't know it
  intimately, so they'll make mistakes when looking for bugs/fixes
- There is little excitement in just fixing bugs in frozen code, developers
  will just migrate elsewhere if there is no fun here

- "Experimental versions" are only run by masochists and bored people who
  need fireworks now and then to know they are still alive. End users don't
  even consider them, when the software is stable enough for the crazier
  ones to unleash on the world as "stable" is when the bugs surface
  (Remember the disaster of the early 2.4s? Ever heard of "Let's wait for
  X.<even>.10 or so, by then it will be stable enough for everyday use"?).
- End users /don't/ test "prereleases", they deem them too risky... so the
  "releases" usually ship with lethal bugs for some people. Decreeing that
  each 5th (or whatever) release will be "golden" will just get people to
  skip all the others, resulting in /much more/ serious bugs in the end

- Backporting new features into a different setting is almost as hard, or
  perhaps much harder, than developing said features in the first place.
  Backpòrting bug fixes is a thankless job.
- Distributions /do/ have the infrastructure in place to collect bug
  reports and correlate them with hardware and software configurations,
  moreover they work with /one/ (or at most a few) kernel configuration,
  not with the almost random assortment of kernel configurations you'll
  find with self-built ones. They also have the (paid) manpower to extract
  conclusions from the above data. 
- If all distributions work from approximately the same base, much
  duplicate/wasted work (yes, backporting is mostly wasted effort IMHO) is
  saved.

- Tools for kernel work are /much/ better now, it is reasonable to maintain
  patches out-of-vanilla and keep the base syncronized with the standard
  kernel source. Lots of people do so now, when it previously was a
  full-time job for the incredibly productive gnome community known
  collectively as "Alan Cox" to do so. There is thus much less need for
  "odd" series in which to integrate new stuff.

All of the above led to the current kernel development model. Users might
not be too happy about it, but the key developers are. And the important
people, the ones you /have/ to keep happy in the OSS development model,
aren't the users. Besides, everybody moaning about the new development
model want something impossible: Fast development, timely integration of
support for newest hardware, while bug free all the time. Won't ever
happen.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Tue, Dec 06, 2005 at 09:58:54AM -0500, Bill Davidsen wrote:
> 
> If a new udev config is needed with every new kernel, why isn't it in
> the kernel tarball? Is that what you mean by "broken distro
> configuration?" The info should be in /proc or /sys and not in an
> external config file, particularly if a different versions per-kernel is
> needed and people are trying new kernels and perhaps falling back to the
> old.

Every distro has different needs for its device naming and groups and
other intergration into the boot process.  To force all of them to unify
on one-grand-way-of-doing-things would just not work out at all.

Look at all of the variations in the udev tarball between the different
vendor configurations (we put them in there for other people to base
their distro off of, if they want to.)

So providing this config in the kernel will just not work, sorry.

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Tue, Dec 06, 2005 at 02:19:29AM +0100, Florian Weimer wrote:
> * Greg KH:
> 
> >> Yes but not home users with relatively new/bleeding edge hardware or
> >> small projects writing for example a wifi driver or a security patch
> >> or whatever without full time commitment to tracking kernel changes.
> >
> > If you are a user that wants this kind of support, then use a distro
> > that can handle this.  Obvious examples that come to mind are both
> > Debian and Gentoo and Fedora and OpenSuSE, and I'm sure there are
> > others.
> 
> IIRC, Gentoo ignores some kinds of security bugs so that the task
> remains manageable.

Do you have specific details about this?  I know the Gentoo kernel team
is currently revamping the way they handle their security updates, as it
is known to need some work.  I'm sure they would be glad for input into
this process.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Mon, Dec 05, 2005 at 07:26:09AM +0100, Willy Tarreau wrote:
> Maybe you should just join forces, eg Chris and you to catch
> new patches, and Adrian to merge them to older kernels ? Every
> software maker always supports a few older releases for the
> people who need to stay on something stable, and it is clearly
> what is missing now in 2.6.

I don't think people realize that the -stable series only contains
patches that other people send us.  For the most part, Chris and I
aren't going out there and activly writing up fixes for some of these
issues, as we both don't have the time and energy to do this.

But if someone wants to start sending us more patches that do this, we
will be glad to incorporate them.  And as Chris said, we will be glad to
release an extra release for the last kernel if we have pending patches.

And also, anyone else can easily take over maintaining these kernel
branches.  The git trees are public, as is our stable patch queue.  So
if anyone wants to maintain older kernels, it is quite easy to start the
process.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Mon, Dec 05, 2005 at 07:51:10PM +0100, Adrian Bunk wrote:
> On Sun, Dec 04, 2005 at 03:24:54PM -0800, Greg KH wrote:
> > On Sun, Dec 04, 2005 at 12:56:50PM +0100, Matthias Andree wrote:
> > > The problem is the upstream breaking backwards compatibility for no good
> > > reason. This can sometimes be a genuine unintended regression (aka.
> > > bug), but quite often this is deliberate breakage because someone wants
> > > to get rid of cruft. While the motivation is sound, breaking between
> > > 2.6.N and 2.6.M must stop.
> > 
> > What are we breaking that people are complaining so much about?
> > Specifics please.
> > 
> > And if you bring up udev, please see my previous comments in this thread
> > about that issue.
> > 
> > It isn't userspace stuff that is breaking, as applications built on 2.2
> > still work just fine here on 2.6 for me.
> > 
> > Yes we break in-kernel apis, all the time, that's fine.  See
> > Documentation/stable-api-nonsense.txt for details about why we do that.
> > 
> > So again, specifics please?
> 
> It's the kernel-related userspace that is the problem (besides
> regressions that are simply bugs).

Again, specifics please?

> Be it the devfs removal, the requirement for a more recent
> wpa_supplicant package or my pending removal of the obsolete raw
> driver.

devfs was documented for over a year that it would be removed.  This
after 2 year notice that it was going to be removed some time in the
future.  So for over 3 years people have known about this.

You have also notified people about the raw driver going away, what more
can we do about this?

And there will always be a need for new package upgrades for some small
subset of programs that are tightly tied to the kernel (like
wpa_supplicant or alsa-libs, or even udev).  But "normal" userspace
applicatations should not break, and if they do, we want to know about
it so we can fix it.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Tue, Dec 06, 2005 at 05:55:42PM +0100, Felipe Alfaro Solana wrote:
> > There might be some subtle changes in the netfilter/routing
> > interaction which break user configurations, but this still being
> > tracked down (and maybe the any behavior is fine because it's
> > unspecified; hard to tell).
> 
> Yeah! For example, the first datagram triggering an IPSec SA is always
> lost (instead of being queued until the IPSec SA has been
> established).
> 
> For example, try pinging the IPSec SA peer for the very first time and
> the first ICMP datagram will always return "resource currently
> unavailable" and, of course, will get lost.
> 
> BTW this works perfectly under *BSD and Mac OS X.

Do the network kernel developers know about this issue?  And if so, what
have they said about it?

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Mon, Dec 05, 2005 at 03:48:06PM +0100, Florian Weimer wrote:
> * Greg KH:
> 
> > On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> >> 
> >> Why can't this be done by distributors/vendors?
> >
> > It already is done by these people, look at the "enterprise" Linux
> > distributions and their 5 years of maintance (or whatever the number
> > is.)
> >
> > If people/customers want stability, they already have this option.
> 
> It seems that vendor kernels lack most DoS-related fixes.  I'm only
> aware of a single vendor which tracks them to the point that CVE names
> are assigned.

If those DoS-related problems are only related to local user DoS's, yes,
that is understandable.  If you have questions about this, ask the
vendor, they usually have a good reason for not including those fixes.

> Vendor kernels are not a panacea, either.  With some of the basic
> support contracts (in the four-figure range per year and CPU), the
> vendor won't look extensively at random kernel crashes which could (in
> theory) be attributed to faulty hardware, *and* you don't get
> community support for these heavily patched kernel collages.

Yes, the lack of community support is one thing you do give up with
them.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Mon, Dec 05, 2005 at 04:17:53PM +0100, Jakob Oestergaard wrote:
> On Sun, Dec 04, 2005 at 02:39:31PM -0800, Greg KH wrote:
> > On Sun, Dec 04, 2005 at 06:00:49PM +0100, Jakob Oestergaard wrote:
> > > In the real world, however, admins currently need to pick out specific
> > > versions of the kernel for specific workloads (try running a large
> > > fileserver on anything but 2.6.11.11 for example - any earlier or later
> > > kernel will barf reliably.
> > 
> > Have you filed a but at bugzilla.kernel.org about this?  If not, how do
> > you expect it to get fixed?
> 
> I don't expect to get it fixed. It's futile. It can get fixed in one
> version and broken two days later, and it seems the attitude is that
> that is just fine.

Huh?  That is just not true at all.  Please give us a bit more credit
than that.

> After a long long back-and-forth, 2.6.11 was fixed to the point where it
> could reliably serve files (at least on uniprocessor configurations -
> and in my setup I don't see problems on NUMA either, but as far as I
> know that's just me being lucky).
> 
> Right after that, someone thought it was a great idea to pry out the PCI
> subsystem and shovel in something else.  Find, that's great for a
> development kernel, but for a kernel that's supposed to be stable it's
> just not something you can realistically do and expect things to work
> afterwards.  And things broke - try mounting 10-20 XFS filesystems
> simultaneously on 2.6.14.  Boom - PCI errors.

What PCI errors are you speaking of?  We did that PCI work to fix a lot
of other machines that were having problems.  And yes, this did break
some working machines, and we are very sorry about this.  But in the
future, changes to this area will not cause this to happen due to the
changes made.

Please file a bug in bugzilla.kernel.org and let us know you are having
problems.  Otherwise we have no idea.

> Now what? Do I as a user upgrade my production environment to the latest
> and greatest kernel experiment, hope that the problems can be fixed
> quickly, and hope that I don't lose too much data in the process?

No, if you rely on a production environment for your stuff, stick with a
disto kernel which has been tested and is backed up by a company that
will maintain it over time.

> (remember I will have people unable to do their jobs whenever the file
> server is down).   Or do I stay on 2.6.11.11 which works on this
> particular server?
> 
> I think I stay.

As you wish.

> > > There are very very good reasons for offering a 'stable series' in plain
> > > source-tree form - lots of admins of real-world systems need this.
> > 
> > But it sounds like you will want different stable series depending on
> > what kind of server you are running.  And that will be even more work...
> 
> The idea would be to fix the actual bugs. After a while, one could have
> a kernel of higher quality with fewer bugs, making it a lot more likely
> that the *same* kernel tree could be used for both workloads A and B.
> 
> It's really very simple :)

If you can point out the "actual bugs" yes.  It sounds so simple from a
theoritical view, but we all know how well theory works in real life
sometimes...

We don't try to create new bugs they are the side affect of fixing other
bugs, or adding new features that other people need.  And again, if
those bugs aren't reported, we have no idea they are present.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Mon, Dec 05, 2005 at 05:59:33AM +0000, Luke-Jr wrote:
> On Sunday 04 December 2005 23:22, Greg KH wrote:
> > On Sun, Dec 04, 2005 at 04:46:31AM +0000, Luke-Jr wrote:
> > > Well, devfs does have some abilities udev doesn't: hotplug/udev
> > > doesn't detect everything, and can result in rarer or non-PnP devices
> > > not being automatically available;
> >
> > Are you sure about that today?
> 
> Nope, but I don't see how udev can possibly detect something that
> doesn't let the OS know it's there-- except, of course, loading the
> driver for it and seeing if it works.
> 
> > And udev wasn't created to do everything that devfs does.
> 
> Which might be a case for leaving devfs in. *shrug*

Heh, no.  Please go look up why devfs was deleted, this one broken
feature is not a good enough reason to keep it around.

> > And devfs can't do everything that udev can (by far...)
> 
> Didn't say it could...
> 
> > > Interesting effects of switching my desktop from devfs to udev:
> > > 1. my DVD burners are left uninitialized until I manually modprobe ide-cd
> > > or (more recently) ide-scsi
> >
> > Sounds like a broken distro configuration :)
> 
> Well, I was assuming you kept Gentoo's udev packages up to date. ;)
> [ebuild   R   ] sys-fs/udev-070-r1  (-selinux) -static 429 kB

That's the latest stable udev release in Gentoo, yes.  Do you have
problems with that one?  If so, please file them in the proper Gentoo
bugzilla.

And yes, Gentoo is lagging a bit on the latest udev support (073 is in
the tree, but 076 has been released.)  That's totally my fault, and you
can blame my lack of free time to do what is needed to fully intregrate
it (due to some very good snow in our local mountains...)

> > > devfs also has the advantage of keeping the module info all in one
> > > place-- the kernel or the module.
> > > In particular, with udev the detection and /dev info is scattered into
> > > different locations of the filesystem. This can probably be fixed
> > > easily simply by having udev read such info from modules or via a /sys
> > > entry, though.
> >
> > What information are you talking about here?
> 
> I'm assuming everything in /etc/udev/rules.d/50-udev.rules used to be
> in the kernel for devfs-- perhaps it was PAM though, I'm not sure.

That is not true, it was not.

> Other than that, I don't expect that simply installing a new kernel
> module will allow the device to be detected automatically, but that
> some hotplug or udev configurations will need to be updated also.

Have you tried this?  What "new kernel module" are you speaking of?

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Michael Frank]

On Tuesday 06 December 2005 01:54, Horst von Brand wrote:
> Michael Frank <mhf@users.berlios.de> wrote:
>
> [...]
>
> > As to security, most vulnerabilities are hard to
> > exploit remotely
>
> Right.
>
> >          and practical security can be much more
> > improved by hiding detailed software versions from
> > clients.
>
> Ever heard of nmap <http://www.nmap.org>? 

I wrote my original post with nmap in mind.

I use nmap all the time, it is a excellent tool. At times I 
used nmap to scan those IP's who scan my IP and have 
unleashed at times a barrage of counter-scans to the point 
of bringing my connection pretty much down. Some of those 
scanners must have big pipes.

> Or perhaps 
> noticed all kinds of attacks against Linux using old
> exploits or Windows specific ones? 

100s to 1000s of unsolicited packets mainly to windows 
specific service ports day and also several burst attacks a 
day. These days I just leave the firewall logging off in 
order to play less with nmap ;)

> Hiding versions is 
> /not/ secure. At most marginally so, 

Sorry, do not concur. Unlike windows and such, linux is a 
_fast_ moving target. The best bet to crack it a _inside_  
job by a trusted perpetrator (for example the debian case 
or the corruption of the kernels bk derived cvs repo ), 
which still ring bells... Remote exploitation of the odd 
random vulnerability in the absence of detailed version 
info is as likely as winning a jackpot.

> and the pain for 
> whoever needs the version for legitimate reasons just
> isn't worth it.

Oh well, If I have a legitimate requirement for the detailed 
versions someone is running, I can ask. 

Again, most violations are  made possible by:

a) access to hardware or admin/root passwords (what was  the 
kernel.org case tracked to?)

b)  access to version info and utilizing a exploit short 
term (the debian case)

IMHO, to have good security, 1) use open source and 2)  
sound implementation of common sense security procedures 
beginning with the basic doctrine who does not have to know 
won't know.  Yes, some things seem never to change :-(

> >                                                    
> > Apache 2 on linux 2.6 will do instead of providing full
> > vendor specific package versions!
> >
> > As to drivers, in case 3 month driver delay matters, HW
> > vendor can improve situation substantially  by not
> > waiting 6+ months before (if at all) releasing
> > drivers/docs for linux!
>
> For /server/ type workloads, where you /need/ stability,
> you carefully pick the hardware and then run a selected
> "enterprise" distro on it. The distro people do the hard
> work of keeping your kernel up to date and secure. And
> even worry about a smooth upgrade to the next version.
> For a price, sure. But either you really need it (and
> gladly pay the price) 

sure

> or you don't (in which case you 
> have nothing to complain about).

kernel.org kernels and gentoo linux will do for me.

	Thank you
	Michael

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Gene Heskett]

On Tuesday 06 December 2005 09:01, Florian Weimer wrote:
>* Horst von Brand:
>>> You mentioned security issues in your initial post.  I think it
>>> would help immensely if security bugs would be documented properly
>>> (affected versions, configuration requirements, attack range, loss
>>> type etc.) when the bug is fixed, by someone who is familiar with
>>> the code. (Currently, this information is scraped together mostly by
>>> security folks, sometimes after considerable time has passed.) 
>>> Having a central repository with this kind of information would
>>> enable vendors and not-quite-vendors (people who have their own set
>>> of kernels for their machines) to address more vulnerabilties
>>> promptly, including less critical ones.
>>
>> I've fixed bugs which turned out to be security vulnerabilities. And
>> I didn't know (or even care much) at the time. Finding out if some
>> random bug has security implications, and exactly which ones/how much
>> of a risk they pose is normally /much/ harder than to fix the bugs.
>
>I know, it happens all the time: vulnerabilities are fixed because
>they are bugs, and not because they are vulnerabilities.  It's
>unfortunate if people are unnecessarily exposed to the vulnerability
>(because they don't know about it and don't apply the fix as a result),
>but it's better than carrying around the bug indefinitely.
>
>But if there's considerable evidence that you might have fixed a
>security bug, preserving this information (and other bits that are
>immediately obvious to you as a developer, but not necessarily who
>reviews the issue) seems worthwhile.  Maybe you don't want to put it
>into the public commit message, but forwarding what you have to some
>trusted group of volunteers would make sense.  The volunteers would
>distill the information, add more data and assign a CVE if necessary,
>and declassify the information as soon as the public is ready (in the
>form of a short security advisory, like the ones you see for most
>applications).
>
>Does this sound too far-fetched?  Why don't you think this would be a
>valuable service to all users, vanilla kernels or not?

But, as you'll recall, ALan Cox fixed a couple of major security things 
about 2 years ago, and because of the DMCA, those patches weren't 
commented at all.  Apparently the fix could only be determined to be 
correct by violating the DMCA, and he was very pointed in his comments 
re the DMCA at the time.

So keeping good records might/could be a double edged sword.

>> And rather pointless, after the fix is in.
>
>It doesn't matter much if the fix is in the kernel.org tree, when I'm
>supposed to use vendor kernels. 8-)
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.36% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Dmitry Torokhov]

On 12/6/05, Matthias Andree <matthias.andree@gmx.de> wrote:
>
> QA has to happen at all levels if it is supposed to be affordable or
> scalable. The development process was scaled up, but QA wasn't.
>
> How about the Signed-off-by: lines? Those people who pass on the changes
> also pass on the bugs, and they are responsible for the code - not only
> license-wise, but also quality-wise. That's the latest point where
> regression tests MUST happen.

People who pass the changes can only test ones they have hardware for.
For the rest they can try to validate the code by reading patches but
have to rely on the submitter wrt to the patch actually working.

--
Dmitry

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Matthias Andree:

> On Tue, 06 Dec 2005, Florian Weimer wrote:
>
>> From a vendor POV, the lack of official kernel.org advisories may be a
>> feature.  I find it rather disturbing, and I'm puzzled that the kernel
>> developer community doesn't view this a problem.  I know I'm alone,
>
> You're not alone in viewing this as a problem, 

I know, it's a typo.

> How about the Signed-off-by: lines? Those people who pass on the changes
> also pass on the bugs, and they are responsible for the code - not only
> license-wise, but also quality-wise. That's the latest point where
> regression tests MUST happen.

There are critical kernel parts for which automated regression testing
is very hard.  In some twisted sense, regression testing ist best done
by those who run real applications, i.e. end users.  The interesting
thing is that you end up with reasonably stable software this way,
except in a few corner cases.

The main point of debate seems to be how relevant the corner cases
are, and how much general kernel development should care about them.
(And no, not everyone in such a corner has $$,$$$ to spend.)

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Lee Revell wrote:
> On Mon, 2005-12-05 at 11:17 -0500, Mark Lord wrote:
> 
>>>>>Ahh OK .. I don't use it, so wouldn't have been affected. That's one
>>>>>userspace interface broken during the series, does anyone have any more?
>>
>>Ah.. another one, that I was just reminded of again
>>by the umpteenth person posting that their wireless
>>no longer is WPA capable after upgrading from 2.6.12.
>>
>>Of course, the known solution for that issue is to
>>upgrade to the recently "fixed" latest wpa_supplicant
>>daemon in userspace, since the old one no longer works.
>>
>>Things like this are all too regular an occurance.
> 
> 
> The distro should have solved this problem by making sure that the
> kernel upgrade depends on a new wpa_supplicant package.  Don't they
> bother to test this stuff before they ship it?!?

Could you provide a little detail on the technology by which a distro
checks for functionality against a kernel which wasn't necessarily
released when the distro shipped. My udev doesn't generate /dev/timewarp.

Going to a new kernel in the same series shouldn't have to be treated as
if it were a change to a whole new operating system, and shouldn't
require completely replacing existing utilities with new ones which
aren't backware compatible to allow fallback to the original kernel.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

David Ranson wrote:
> Matthias Andree wrote:
> 
> 
>>So was I. And now what? ipfwadm and ipchains should have been removed
> 
>>from 2.6.0 if 2.6.0 was not to support these. That opportunity was
> 
>>missed, the removal wasn't made up for in 2.6.1, so the stuff has to
>>stick until 2.8.0.
>>
>>
> 
> I'm not aware of that policy... maybe I overlooked something?

Until 2.6, a stable series did not remove existing features, so someone
building an rpm or deb package could release it for "2.4.12 or later"
and expect it to work.

As a for instance there are people who went to 2.6 and kept their old
firwall rules written in ipchains, because they still worked. Now if
ipchains are deleted a full rewrite of firewall rules is needed, and
that just shouldn't be don't in haste. My personal opinion is that
ipfwadm and ipchains should have followed some other features into the
night before 2.6.0 ever came out. No one would have gotten a nasty
surprise later. I also think that reiser4 is 2.7 material, if there were
a 2.7.

I didn't see all that much wrong with the old odd/even model to tell the
truth, it wasn't perfect but you knew what you got.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Greg KH wrote:
> On Sun, Dec 04, 2005 at 04:46:31AM +0000, Luke-Jr wrote:
> 
>>Well, devfs does have some abilities udev doesn't: hotplug/udev
>>doesn't detect everything, and can result in rarer or non-PnP devices
>>not being automatically available;
> 
> 
> Are you sure about that today?  And udev wasn't created to do everything
> that devfs does.  And devfs can't do everything that udev can (by
> far...)
> 
> 
>>devfs has the effect of trying to load a module when a program looks
>>for the devices it provides-- while it can cause problems, it does
>>have a possibility to work better.
> 
> 
> Sorry, but that model of loading modules is very broken and it is good
> that we don't do that anymore (as you allude to.)
> 
> 
>>Interesting effects of switching my desktop from devfs to udev:
>>1. my DVD burners are left uninitialized until I manually modprobe ide-cd or 
>>(more recently) ide-scsi
> 
> 
> Sounds like a broken distro configuration :)
> 
> 
>>2. my sound card is autodetected and the drivers loaded, but the OSS emulation 
>>modules are omitted; with devfs, they would be autoloaded when an app tried 
>>to use OSS
> 
> 
> Again, broken distro configuration :)

If a new udev config is needed with every new kernel, why isn't it in
the kernel tarball? Is that what you mean by "broken distro
configuration?" The info should be in /proc or /sys and not in an
external config file, particularly if a different versions per-kernel is
needed and people are trying new kernels and perhaps falling back to the
old.

Have "make install" drop the udev config in /boot like the initrd file.
The the boot could create an slink to "/boot/$(uname -r)-udev" or some such.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Greg KH:

> What are we breaking that people are complaining so much about?
> Specifics please.

Drastic performance changes in certain pipe usage patterns.  This was
probably too early in the 2.6 series to count, though.

There might be some subtle changes in the netfilter/routing
interaction which break user configurations, but this still being
tracked down (and maybe the any behavior is fine because it's
unspecified; hard to tell).

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Horst von Brand:

>> You mentioned security issues in your initial post.  I think it would
>> help immensely if security bugs would be documented properly (affected
>> versions, configuration requirements, attack range, loss type etc.)
>> when the bug is fixed, by someone who is familiar with the code.
>> (Currently, this information is scraped together mostly by security
>> folks, sometimes after considerable time has passed.)  Having a
>> central repository with this kind of information would enable vendors
>> and not-quite-vendors (people who have their own set of kernels for
>> their machines) to address more vulnerabilties promptly, including
>> less critical ones.
>
> I've fixed bugs which turned out to be security vulnerabilities. And I
> didn't know (or even care much) at the time. Finding out if some random bug
> has security implications, and exactly which ones/how much of a risk they
> pose is normally /much/ harder than to fix the bugs.

I know, it happens all the time: vulnerabilities are fixed because
they are bugs, and not because they are vulnerabilities.  It's
unfortunate if people are unnecessarily exposed to the vulnerability
(because they don't know about it and don't apply the fix as a result),
but it's better than carrying around the bug indefinitely.

But if there's considerable evidence that you might have fixed a
security bug, preserving this information (and other bits that are
immediately obvious to you as a developer, but not necessarily who
reviews the issue) seems worthwhile.  Maybe you don't want to put it
into the public commit message, but forwarding what you have to some
trusted group of volunteers would make sense.  The volunteers would
distill the information, add more data and assign a CVE if necessary,
and declassify the information as soon as the public is ready (in the
form of a short security advisory, like the ones you see for most
applications).

Does this sound too far-fetched?  Why don't you think this would be a
valuable service to all users, vanilla kernels or not?

> And rather pointless, after the fix is in.

It doesn't matter much if the fix is in the kernel.org tree, when I'm
supposed to use vendor kernels. 8-)

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Lars Marowsky-Bree:

> On 2005-12-06T01:14:23, Florian Weimer <fw@deneb.enyo.de> wrote:
>
>> > The right way to address this is to work with the distribution of your
>> > choice to make these updates available faster.
>> Working with a distribution benefits that distribution alone.  Working
>> on (e.g.) kernel security advisories would benefit everyone.  It's not
>> a speed issue, it's more about coverage.  And full coverage is very
>> hard to get without support from the real developers.
>
> The distributions differ from another in their sync and branch points
> from the main kernel, and there's no way before hell freezes over this
> is going to change.
>
> So, you essentially need to maintain the kernel your distribution
> branched from. This means: backport fixes/features relevant to your
> release, and make sure the rest of the system stays in sync. This puts
> the effort there where it belongs: to those people benefitting from it.

Lars, please read again what I wrote.  It's not about branch
maintenance ("here's the patch"), it's about providing basic
information which can guide branch maintenance ("here's an issue you
need to look at if you use code from the IPv6 stack in version 2.6.10
to 2.6.12").

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lars Marowsky-Bree]

On 2005-12-05T14:30:09, Bill Davidsen <davidsen@tmr.com> wrote:

> Actually I would be happy with the stability of this series if people 
> would stop trying to take working features OUT of it!

Features are removed when they are no longer features, but design
irritations in a new and improved design, and usually, equivalent or
better (or at least thought to be) functionality is available still in
the big picture (which includes user-space), hopefully in a cleaner
place.

Now, design is often a holy war, and people disagree. That's fine and to
be expected. And sometimes, the whole solution takes a while to
materialize and be implemented from the kernel up to all user-space and
even longer until it has been implemented in the brains of the admins.
This, too, is fine and expected. It's called "innovation" and
"development", sometimes iterative.

> working all that well in any case. But if existing features suddenly 
> drop out from beneath the user, then you will find people doing what you 
> mentioned, staying with old kernels with holes rather than moving to 
> kernels which are simply no longer functional.

You're assuming the kernel is both "static" design-wise as well as
independent (or at least basically eternally backwards compatible) from
user-space. Both assumptions are no longer true. Get over it.


Sincerely,
    Lars Marowsky-Brée

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lars Marowsky-Bree]

On 2005-12-06T01:14:23, Florian Weimer <fw@deneb.enyo.de> wrote:

> > The right way to address this is to work with the distribution of your
> > choice to make these updates available faster.
> Working with a distribution benefits that distribution alone.  Working
> on (e.g.) kernel security advisories would benefit everyone.  It's not
> a speed issue, it's more about coverage.  And full coverage is very
> hard to get without support from the real developers.

The distributions differ from another in their sync and branch points
from the main kernel, and there's no way before hell freezes over this
is going to change.

So, you essentially need to maintain the kernel your distribution
branched from. This means: backport fixes/features relevant to your
release, and make sure the rest of the system stays in sync. This puts
the effort there where it belongs: to those people benefitting from it.

The current model actually works _better_ for the existing
distributions, because they get to choose their branchpoint - with all
the features up to that point - instead of having, say, 2.6.x as the
stable base and then already starting out with having to backport major
features from 2.7 (because of user demand).

A single stable branch beneficial to all users means frozen innovation
for the distributions, and they still have to significant QA on the
releases and the updates to that kernel (to stay on that issue, it
applies to other major components too). Even with 2.4.x, a distribution
couldn't simply stick in newer 2.4.x+n releases instead of 2.4.x,
because, as someone already so well said, one users bugfix is another's
regression. And all the distributors would have to agree on the same
policy for kernel changes and sync even updates!

Thus, more effort for less gain.

The truth is that right now we have _several_ stable branches maintained
by the distributors (be they commercial or free) together with the
kernel-related user-land.

I daresay this is a feature and works pretty well.

If someone wants to maintain a stable 2.6.x release, nobody will stop
them from maintaining 2.6.x.y until y overflows, or until the 6 months
are full and then they can release their new major update and plot a
transition path with the updates to all required user-land.

The fact people are complaining about stems from the fact that the Linux
kernel itself is useless; it is intimately tied to various components
which reside in user-space, and so it is inherent to the process that a
major kernel update very likely maps to a distribution update. The
components are developed separately, but they do not have a stable
modular interface, at essentially no level but the POSIX/system call
interfaces and sometimes, glibc or what is specified in the LSB.

This is a _feature_! It allows us to more quickly move and adapt. The
BSD model is, as Dave pointed out, even further along this road, and
every distribution basically does exactly that, because our user
community is big enough to sustain it.


Sincerely,
    Lars Marowsky-Brée

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Mon, 05 Dec 2005, Bill Davidsen wrote:

> If a firm policy of not removing supported features until 2.7 was 
> adopted I don't see a problem. The bulk of the instability (not 

I do - the problem is someone will let it bit-rot for a few releases and
then declare it broken. Remember ATAPI CD writing vs. DMA?

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Mon, 05 Dec 2005, Jeff Garzik wrote:

> Bill Davidsen wrote:
> >I do think the old model was better; by holding down major changes for 
> >six months or so after a new even release came out, people had a chance 
> >to polich the stable release, and developers had time to recharge their 
> >batteries so to speak, and to sit and think about what they wanted to 
> >do, without feeling the pressure to write code and submit it right away. 
> >Knowing that there's no place to send code for six months is a great aid 
> >to generating GOOD code.
> 
> It never worked that way, which is why the model changed.
> 
> Like it or not, developers would only focus on one release.  In the old 
> model, unstable things would get shoved into the stable kernel, because 
> people didn't want to wait six months.  And for the unstable kernel, it 
> would often be so horribly broken that even developers couldn't use it 
> for development (think 2.5.x IDE).

So why haven't the broken patches (yes, TCQ and all that, too) been
backed out at the time?

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Tue, 06 Dec 2005, Florian Weimer wrote:

> From a vendor POV, the lack of official kernel.org advisories may be a
> feature.  I find it rather disturbing, and I'm puzzled that the kernel
> developer community doesn't view this a problem.  I know I'm alone,

You're not alone in viewing this as a problem, but QA is a burden kernel
developers are not interested in. But it is necessary.

QA has to happen at all levels if it is supposed to be affordable or
scalable. The development process was scaled up, but QA wasn't.

How about the Signed-off-by: lines? Those people who pass on the changes
also pass on the bugs, and they are responsible for the code - not only
license-wise, but also quality-wise. That's the latest point where
regression tests MUST happen.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Mon, 05 Dec 2005, Florian Weimer wrote:

> * Matthias Andree:
> 
> > Basically, no-one should have permission to touch any core parts, except
> > for fixes, until 2.7. Yes, that means going back to older models. Yes,
> > that means that the discussions will start all over. And yes, that means
> > that the cool stuff has to wait. Solution: release more often.
> 
> Would this alone change much?  I think what we really want is that our
> favorite branch (whatever it is) gets critical fixes forever (well,
> maybe one or two years, but this is forever).  This is a bit
> unrealistic because everyone has a slightly different branchpoint.
> Releasing more often doesn't change that, really.

Releasing minor releases more often and enforcing "don't touch unless
you must" policy would create such synchronization point and a branch
where everyone could safely hop between releases.

> In the security area, I think there is enough experience out there to
> collect data which would help each local branch maintainer to install
> the relevant fixes.  But for general development, this seems to be
> infeasible, unless you focus your software architecture on this
> purpose (which is probably a terrible idea to do for kernel
> development).

I don't think focusing the kernel on code quality and security is wrong
though. The actual problem we've seen from postings by Lee and others is
that the burden of test is shifted to the distros and their QA teams so
that effectively everyone is free to break things at will, downstream QA
will fix it anyways.

This however doesn't work, and the problem here is the propagation
delay. At the time the end user sees a problem with his kernel, the
upstream has already abandoned the 2.6.X.Y stable branch the distro was
based on, and upstream is at 2.6.X+2 or even farther ahead. What is
actually needed is to enclose this end user system in the tests run
before further changes in the same area. And as udev etc. need to
change, a simple test if the current kernel works means updating some
user space packages, hotplug, modutils (OK this was 2.5), udev,
whatever, and what's even worse, if that doesn't help or breaks other
things. Going back may not even work through the packaging system
because the old kernel version may not have a "udev <= N" dependency
either...

So before this can work, the actual package maintenance systems such as
yum, yast, dpkg, apt and rpm will need to support what Emacs Lisp calls
excursions. It means, snapshot the packages and revert to the same set
later.

Even if this were solved and excursions were cheap, it would still not
solve the time skew bug report and upstream fixes...

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Mon, 05 Dec 2005, Benjamin LaHaise wrote:

> On Mon, Dec 05, 2005 at 04:47:55PM -0600, Rob Landley wrote:
> > So no in-kernel filesystem can get this right without help from userspace 
> > (even devfs had devfsd), and as soon as you've got a userspace daemon to tell 
> > the kernel who is who you might as well do the whole thing there, now that 
> > the kernel is exporting everyting _else_ we need to know via /sys 
> > and /sbin/hotplug.
> 
> /sbin/hotplug is suboptimal.  Even a pretty fast machine is slowed down 
> pretty significantly by the ~thousand fork and exec that take place during 
> startup.  For the most common devices -- common tty, pty, floppy, etc that 
> every system has, this is a plain waste of resources -- otherwise known as 
> bloat.

You mean that distro boot scripts now need to wait for 20 seconds until
hotplug has finally handled all the coldplug events so the script can
finally slap the IPv4 address onto the interface or start dhcpcd. :-)

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Mon, 05 Dec 2005, Horst von Brand wrote:

> > You mentioned security issues in your initial post.  I think it would
> > help immensely if security bugs would be documented properly (affected
> > versions, configuration requirements, attack range, loss type etc.)
> > when the bug is fixed, by someone who is familiar with the code.
> > (Currently, this information is scraped together mostly by security
> > folks, sometimes after considerable time has passed.)  Having a
> > central repository with this kind of information would enable vendors
> > and not-quite-vendors (people who have their own set of kernels for
> > their machines) to address more vulnerabilties promptly, including
> > less critical ones.
> 
> I've fixed bugs which turned out to be security vulnerabilities. And I
> didn't know (or even care much) at the time. Finding out if some random bug
> has security implications, and exactly which ones/how much of a risk they
> pose is normally /much/ harder than to fix the bugs.  And rather pointless,
> after the fix is in.

I believe everyone who maintains a nontrivial piece of software has
experienced a situation where a bug fix addressed a bug that could
actually be exploited and that wasn't clear at the time.

Calling this "pointless" after the fix is in leaves people in danger
unaware, unless it happens on a branch where every user can be expected
to update because only tested fixes are merged. As this isn't the case
for the kernel, but everyone moves on at will, doesn't care if a
previous bug fix is exploitable and whatnot, the Linux kernel's security
is essentially nonexistent, and expecting downstream QA teams to handle
this is just ridiculous for many reasons already mentioned.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Luke-Jr]

On Tuesday 06 December 2005 00:34, Rob Landley wrote:
> On Sunday 04 December 2005 23:59, Luke-Jr wrote:
> > On Sunday 04 December 2005 23:22, Greg KH wrote:
> > > On Sun, Dec 04, 2005 at 04:46:31AM +0000, Luke-Jr wrote:
> > > > Well, devfs does have some abilities udev doesn't: hotplug/udev
> > > > doesn't detect everything, and can result in rarer or non-PnP devices
> > > > not being automatically available;
> > >
> > > Are you sure about that today?
> >
> > Nope, but I don't see how udev can possibly detect something that doesn't
> > let the OS know it's there-- except, of course, loading the driver for it
> > and seeing if it works.
>
> Stuff shows up in /sys whether or not Linux has a driver loaded for it.

Only if Linux is aware it exists. I'm thinking of those old ISA cards and 
such.
-- 
Luke-Jr
Developer, Utopios
http://utopios.org/

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bernd Petrovitsch]

On Mon, 2005-12-05 at 21:41 -0300, Horst von Brand wrote:
> Bernd Petrovitsch <bernd@firmix.at> wrote:
[...]
[...]
> > > whole libc -> glib switchover.

Ah, that should have read "libc.5(*) to glibc" switchover"?
(*) IIRC.

> > glib has AFAIK next to nothing to do with a libc AFAICT (i.e. it is
> > using standard libc functions but that's all).
> 
> He refers to the a.out to ELF switchover. Yes, it was painful. But not as

Was it? And it was ages ago (i can't even remember since when I disable
a.out in the kernel completely and never had a problem with it).

> much as he makes out. The Win98 --> WinNT change was worse, IMHO.

Of course. Especially if you started to use the permission system and
not let the NT installation stay in the default mode where every user
may do everything everywhere (and they are hiding the contents of
certain directories in the file browser instead of simply letting the
administrator change it's contents so that folks really learn it).

[....]
> > >                                                                 The
> > > reason - infighting and lack of backwards 
> 
> > Yes, probably - MSFT is spreading the same story since ages.
> 
> Gandhi-con 3 ;-)

???? Sorry, what do you mean?

[...]
> > As other told there never was a stable kernel module interface. Of
> > course there is probably enough willing manpower out there who will work
> > on that once you pay them. Or you can provide such support on your own.
> 
> Right.
> 
> > Or do you (or anybody else) has drivers which should be maintained for
> > vanilla-kernel and/or vendor kernels and/or other kernels (to fix the
> > breakage in a cosntructive way), we can provide you with an offer to do
> > that.
> 
> Constructive criticism? Even of the sort that contributes something? What

No, since we interface here with the commercial world , it is a
commercial offer (well, sort of - at least a an offer to provide an
offer it the details and requirements are defined/clear).

> are you thinking about?!

$COMPANY wants a maintained "open" driver (probably GPL but that's not
the point)?
$COMPANY gives us money (a to be defined amount of money for a to be
defined time, for to be defined distributions and/or kernel trees, to be
defined QA with respect to the hardware driven by the driver, etc.) and
we do that for you.

Feel free to ignore it ....
	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Monday 05 December 2005 21:32, Benjamin LaHaise wrote:
> On Mon, Dec 05, 2005 at 09:19:28PM -0600, Rob Landley wrote:
> > > /sbin/hotplug is suboptimal.  Even a pretty fast machine is slowed down
> > > pretty significantly by the ~thousand fork and exec that take place
> > > during startup.
> >
> > Why do you need hotplug events on startup?  Can't you just scan /sys for
> > "dev" entries do the initial populate of /dev from that?
>
> That's my point: I don't.  Yet the kernel tries to exec /sbin/hotplug on
> startup around 1000 times.
>
>   -ben

At what stage?  If it's initramfs, then don't have one on initramfs.  (Not by 
default anyway, add a symlink when you're ready to start caring, or write the 
correct path to /proc/sys/heeeeeeere's_hotplog.)

Failure to exec 1000 times shouldn't take too long.  I have shell scripts that 
fork and exec 1000 times in under a second, and they're actually doing 
something.

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Benjamin LaHaise]

On Mon, Dec 05, 2005 at 09:19:28PM -0600, Rob Landley wrote:
> > /sbin/hotplug is suboptimal.  Even a pretty fast machine is slowed down
> > pretty significantly by the ~thousand fork and exec that take place during
> > startup.
> 
> Why do you need hotplug events on startup?  Can't you just scan /sys for "dev" 
> entries do the initial populate of /dev from that?

That's my point: I don't.  Yet the kernel tries to exec /sbin/hotplug on 
startup around 1000 times.

		-ben
-- 
"You know, I've seen some crystals do some pretty trippy shit, man."
Don't Email: <dont@kvack.org>.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Monday 05 December 2005 21:15, Greg KH wrote:
> On Mon, Dec 05, 2005 at 04:47:55PM -0600, Rob Landley wrote:
> > On the busybox list we're currently working out a design for mdev, the
> > micro-udev that'll go into busybox 1.2.  So we're thinking about this
> > issue pretty carefully, as we speak.  What's the minimal amount of work
> > we can't get away with not doing?
>
> I suggest you take this discussion to the linux-hotplug-devel mailing
> list, which is the proper place for it.  Not this thread about stable
> kernel series :)

Didn't know there was one.  Will do, but probably not today...

> thanks,
>
> greg k-h

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Monday 05 December 2005 18:54, Steven Rostedt wrote:

> > Hint:  Any plan in a volunteer community that starts with "$BUSY_PEOPLE
> > should do $THIS" fails.  Any plan that starts with "I could do $THIS" at
> > least has a chance.
>
> Actually, they are already maintaining 2.6.x.y, (x => 11, 12, ...) I was
> trying to get them to only maintain 2.6.x.y (x => 11, 12, 13, 14, 20, 25,
> ...)

That's still "trying to get them" rather than "I could"...

> So maybe it would actually be easier.  But I'm sure they wouldn't be
> fooled, since the longer you maintain a fork, the harder it becomes.

And the number exponentially increases (2.6.x+1.y, 2.6.x+2.y, all at the same 
time...)

No, I pestered them a while back about possibly doing a 2.6.x.y+1 to flush 
their patch queue before doing a 2.6.x+1.1, and they seem more receptive to 
the idea now.  But then backporting 2.6.x+1.y to 2.6.x becomes your job...

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Monday 05 December 2005 17:05, Benjamin LaHaise wrote:
> On Mon, Dec 05, 2005 at 04:47:55PM -0600, Rob Landley wrote:
> > So no in-kernel filesystem can get this right without help from userspace
> > (even devfs had devfsd), and as soon as you've got a userspace daemon to
> > tell the kernel who is who you might as well do the whole thing there,
> > now that the kernel is exporting everyting _else_ we need to know via
> > /sys and /sbin/hotplug.
>
> /sbin/hotplug is suboptimal.  Even a pretty fast machine is slowed down
> pretty significantly by the ~thousand fork and exec that take place during
> startup.

Why do you need hotplug events on startup?  Can't you just scan /sys for "dev" 
entries do the initial populate of /dev from that?

> For the most common devices -- common tty, pty, floppy, etc that 
> every system has, this is a plain waste of resources -- otherwise known as
> bloat.

I get those from a scan of /sys, and only care about hotplug events that come 
in after that.  (Could just be me...)

>   -ben

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Mon, Dec 05, 2005 at 04:47:55PM -0600, Rob Landley wrote:
> On the busybox list we're currently working out a design for mdev, the 
> micro-udev that'll go into busybox 1.2.  So we're thinking about this issue 
> pretty carefully, as we speak.  What's the minimal amount of work we can't 
> get away with not doing?

I suggest you take this discussion to the linux-hotplug-devel mailing
list, which is the proper place for it.  Not this thread about stable
kernel series :)

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Bill Davidsen <davidsen@tmr.com> wrote:
> Ben Collins wrote:

> > What you're suggesting sounds just like going back to the old style of
> > development where 2.<even>.x is stable, and 2.<odd>.x is development.
> > You might as well just suggest that after 2.6.16, we fork to 2.7.0, and
> > 2.6.17+ will be stable increments like we always used to do.

> I'll let him speak to what he intended, but my idea of stable is to
> keep the features of 2.6.0 in 2.6.N for any value of N.

That works iff N == 0.

>                                                         Adding new
> stuff rapidly hasn't been nearly the problem people feared,

... because they had the leeway to change broken/unsuitable things to fit,
and because the tools today are so much better...

>                                                             but that's
> largely due to the efforts of akpm to act as throttle, and somehow get
> more people to try his versions and knock the corners off the new code
> before it goes mainline.

Heroic efforts, sure.

> I do think the old model was better;

It just /didn't work/. You don't remember the pain of jumping from 2.2 to
2.4, do you? Sure, while 2.2 lasted it was stable, but everybody screamed
that the latest&greatest whatever-card didn't work, and either jumped to
2.3.x du jour (good luck! had to futz around with lots of matching userland
changes /without/ distro support for that) or choose a distro which shipped
a patched 2.3 kernel (which was totally incompatible when 2.4 showed up) or
(tried to) backport features to 2.2 (ditto, resulting from a /huge/ amount
of wasted effort).

>                                      by holding down major changes for
> six months or so after a new even release came out, people had a
> chance to polich the stable release,

No chance. The people who would have been doing so just got bored and
looked elsewhere for challenges. Do enough of that, and you'll be left
without any volunteers at all.

>                                      and developers had time to
> recharge their batteries so to speak, and to sit and think about what
> they wanted to do, without feeling the pressure to write code and
> submit it right away.

This assumes kernel development is uniform movement. Far from it, at any
moment there are pieces that haven't been touched in ages, others in active
turnover, others just finished being worked over.

>                       Knowing that there's no place to send code for
> six months is a great aid to generating GOOD code.

For something else, sure.

> The other advantage of a development tree was that features could be
> added and removed without the argument that it would break this or
> that. It was development, no one was supposed to use it for
> production, no one could claim that there was even an implied promise
> of things working or even existing.

With the current tools, that development can be done outside the vanilla
tree, and integrated with not too much pain. The /reason/ for "wild
development" phases is not there anymore.

>                                     ipchains could have gone out of
> 2.6 with no more fuss than xiafs departing. The people who really want
> it stay with the old kernel.

Come on, it has been announced for a /long/ time. It is not like anybody
could have been caught unaware. More like people thinking it would /never/
be done as it was called so long before...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Jeff Garzik]

Bill Davidsen wrote:
> I do think the old model was better; by holding down major changes for 
> six months or so after a new even release came out, people had a chance 
> to polich the stable release, and developers had time to recharge their 
> batteries so to speak, and to sit and think about what they wanted to 
> do, without feeling the pressure to write code and submit it right away. 
> Knowing that there's no place to send code for six months is a great aid 
> to generating GOOD code.

It never worked that way, which is why the model changed.

Like it or not, developers would only focus on one release.  In the old 
model, unstable things would get shoved into the stable kernel, because 
people didn't want to wait six months.  And for the unstable kernel, it 
would often be so horribly broken that even developers couldn't use it 
for development (think 2.5.x IDE).

	Jeff

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Steven Rostedt]

On Tue, 6 Dec 2005, Florian Weimer wrote:
> >
> > Actually, they are already maintaining 2.6.x.y, (x => 11, 12, ...)
>
> I think the 2.6.x.y series is no longer maintained once 2.6.x+1 has
> been released for some time (surely after 2.6.x+2).
>

The same can still go for this, but instead of stopping at 2.6.x+2 we could
stop at 2.6.x+6 (or +5), and just not care about 2.6.x+[1-4].  But that
would be strong enough for those that would like the stable branch to
maintain it themselves.  Currently it'l hard to pick a 2.6.x that you want
to stay with since the 2.6.x.y is stopped right after 2.6.x+1 is out.  But
if not all 2.6.x has a .y, then that would focus more distrobutions or
whatever to pick the same one to support.

Oh well, I'm just spitting out a bunch of lip service here. It actually
seems interesting to try, and if I actually had a need to do this, I
would. But right now my focus is elsewhere.

Cheers,

-- Steve

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Greg KH:

>> Yes but not home users with relatively new/bleeding edge hardware or
>> small projects writing for example a wifi driver or a security patch
>> or whatever without full time commitment to tracking kernel changes.
>
> If you are a user that wants this kind of support, then use a distro
> that can handle this.  Obvious examples that come to mind are both
> Debian and Gentoo and Fedora and OpenSuSE, and I'm sure there are
> others.

IIRC, Gentoo ignores some kinds of security bugs so that the task
remains manageable.  Debian, in contrast, hasn't released a kernel
update for its stable distribution since June (but unstable and even
testing is in surprisingly good shape).

Maybe the real vendor kernels are better.  Without CVE-based bug
tracking on their part, it is hard to tell, though. 8-)

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Florian Weimer <fw@deneb.enyo.de> wrote:

[...]

> You mentioned security issues in your initial post.  I think it would
> help immensely if security bugs would be documented properly (affected
> versions, configuration requirements, attack range, loss type etc.)
> when the bug is fixed, by someone who is familiar with the code.
> (Currently, this information is scraped together mostly by security
> folks, sometimes after considerable time has passed.)  Having a
> central repository with this kind of information would enable vendors
> and not-quite-vendors (people who have their own set of kernels for
> their machines) to address more vulnerabilties promptly, including
> less critical ones.

I've fixed bugs which turned out to be security vulnerabilities. And I
didn't know (or even care much) at the time. Finding out if some random bug
has security implications, and exactly which ones/how much of a risk they
pose is normally /much/ harder than to fix the bugs.  And rather pointless,
after the fix is in.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Steven Rostedt:

>> Hint:  Any plan in a volunteer community that starts with "$BUSY_PEOPLE should
>> do $THIS" fails.  Any plan that starts with "I could do $THIS" at least has a
>> chance.
>
> Actually, they are already maintaining 2.6.x.y, (x => 11, 12, ...)

I think the 2.6.x.y series is no longer maintained once 2.6.x+1 has
been released for some time (surely after 2.6.x+2).

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Steven Rostedt:

>> Would this alone change much?  I think what we really want is that our
>> favorite branch (whatever it is) gets critical fixes forever (well,
>> maybe one or two years, but this is forever).  This is a bit
>> unrealistic because everyone has a slightly different branchpoint.
>> Releasing more often doesn't change that, really.
>
> Maybe that is what is needed.  A branch that all can use.

There isn't a single one.  Even for Debian, it was a hard struggle to
get sown to just two (or three?).  Now try that across distributions,
or for people who own choosy hardware. (I once had to deal with a box
which didn't like anything else except 2.6.0-test9.  I believe it's
still running this version, maybe slightly patched.)

> Have every 5 or so 2.6.x become a "stable" branch.  Where
> distributions and users can work together on keeping it stable.  The
> rules to modifying such a branch would pretty much stay with what it
> already takes to modify the current 2.6.x.y branch.  If you want a
> feature, you must either take the latest "unstable" 2.6.x branch or
> wait for the next "stable" 2.6.x branch to merge.

In essence, this is just a slower version of the current model.  It
won't change that much, unless the speed of the development cycle (and
its phase) matches your needs, which is unlikely.  Security bugs would
still be discovered at about the same rate.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Michael Frank <mhf@users.berlios.de> wrote:

[...]

> As to security, most vulnerabilities are hard to exploit 
> remotely

Right.

>          and practical security can be much more improved 
> by hiding detailed software versions from clients.

Ever heard of nmap <http://www.nmap.org>? Or perhaps noticed all kinds of
attacks against Linux using old exploits or Windows specific ones? Hiding
versions is /not/ secure. At most marginally so, and the pain for whoever
needs the version for legitimate reasons just isn't worth it.

>                                                     Apache 
> 2 on linux 2.6 will do instead of providing full vendor 
> specific package versions!
> 
> As to drivers, in case 3 month driver delay matters, HW 
> vendor can improve situation substantially  by not waiting 
> 6+ months before (if at all) releasing drivers/docs for 
> linux! 

For /server/ type workloads, where you /need/ stability, you carefully pick
the hardware and then run a selected "enterprise" distro on it. The distro
people do the hard work of keeping your kernel up to date and secure. And
even worry about a smooth upgrade to the next version. For a price, sure.
But either you really need it (and gladly pay the price) or you don't (in
which case you have nothing to complain about).
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Steven Rostedt]

On Mon, 5 Dec 2005, Rob Landley wrote:
> On Monday 05 December 2005 15:21, Steven Rostedt wrote:
> > Perhaps, we could start out having Greg and Chris just concentrate on
> > every fifth branch instead of every one, and that way the stability will
> > last much longer.
>
> Ah, belling the cat.

:)

>
> Hint:  Any plan in a volunteer community that starts with "$BUSY_PEOPLE should
> do $THIS" fails.  Any plan that starts with "I could do $THIS" at least has a
> chance.

Actually, they are already maintaining 2.6.x.y, (x => 11, 12, ...) I was
trying to get them to only maintain 2.6.x.y (x => 11, 12, 13, 14, 20, 25, ...)

So maybe it would actually be easier.  But I'm sure they wouldn't be
fooled, since the longer you maintain a fork, the harder it becomes.

I was just making a suggestion, so that if someone else thought it was a
good idea, they could do it.  I personally don't need such a beast, since
I would just stay with the latest 2.6.x anyway.  Since I have that luxury.

>
> This is not limited to open source, by the way...

Yep, I know that.

-- Steve

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Arjan van de Ven:

>> Well, if there's a CVE name, the proper patch isn't *that* far away
>> (someone has already done a bit of work to isolate the fix).  The real
>> issue seems to be how to make sure that CVE names are assigned during
>> the kernel development process (and not just as an afterthought by the
>> security folks).
>
> security@kernel.org works that way already in a way.

As far as I know, many of the recent CVE assignments for kernel
vulnerabilities have been done by MITRE, requested by individuals
which are neither known as kernel developers, nor vendor security
folks (for "vendor" as in "we have our own legal department with real
lawyers").

Maybe the source of CVE assignments paints a wrong picture.  But if
the CVE picture is correct, vendor-paid kernel developers help behind
the scenes, but there is little interest in openly documenting
security issues, so that users (and what kernel.org considers fringe
distros) can apply the relevant patches if they use kernel.org
kernels.

>From a vendor POV, the lack of official kernel.org advisories may be a
feature.  I find it rather disturbing, and I'm puzzled that the kernel
developer community doesn't view this a problem.  I know I'm alone,
and there are certainly part-time security guys who would be willing
join forces to create something like a kernel.org security bug
database.  But the only answers we get is that everything is fine,
vendors handle the situation, security@kernel.org actually does this
already, etc.

> The hardest part is actually knowing which versions are affected,

First, you need to know that the patch plugs a security hole. 8-) This
isn't always obvious based on the patch, even if the fact is known to
the comitter.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Bernd Petrovitsch <bernd@firmix.at> wrote:
> [ Minimized quoted part ]
> On Sun, 2005-12-04 at 17:43 -0700, Jeff V. Merkey wrote:
> > Bernd Petrovitsch wrote:
> > >On Sat, 2005-12-03 at 17:52 -0700, Jeff V. Merkey wrote:
> [...]
> > >>of this code. I have apps written for Windows in 1990 and 1998 that 
> > >                      ^^^^
> > >>still run on Windows XP today. Linux has no such concept of

> > >But this not even holds for nearly all apps.

> > >>backwards compatiblity. Every company who has embraced it outside of 

> > >The same holds (probably) for Linux apps (given that your kernel can
> > >start a.out). And AFAIBT by Win* driver developers even in the Win*
> > >world you have to change your driver because of a new Win* version now
> > >and then.
> [...]
> > whole libc -> glib switchover.

> glib has AFAIK next to nothing to do with a libc AFAICT (i.e. it is
> using standard libc functions but that's all).

He refers to the a.out to ELF switchover. Yes, it was painful. But not as
much as he makes out. The Win98 --> WinNT change was worse, IMHO.

> > It's hilarious that BSD had to create a Linux app compat lib,

And Solaris forever had a BSD compatibility suite, including libraries and
tools. So what?

> >                                                               and the 
> > RedHat shipped compat libs for 3 releases

So legacy stuff continued working. And that is bad how?

> Here you have your backwards compatibility.

Right.

> > as well.   Not even close.  Windows has won.  M$ has won.  Linux lost 
> > the desktop wars

First of all, Linux isn't about "winning a war". And the desktop wars
haven't really started...

> >                  and will soon loose the server wars as well.

Sorry, but that one is almost over, and Linux has won.

> >                                                                 The
> > reason - infighting and lack of backwards 

> Yes, probably - MSFT is spreading the same story since ages.

Gandhi-con 3 ;-)

> >                                           compatibility.  Binary only
> > module breakage kernel to kernel will continue. 

So what? Binary modules are mostly bad and break the kernel, so...

> As other told there never was a stable kernel module interface. Of
> course there is probably enough willing manpower out there who will work
> on that once you pay them. Or you can provide such support on your own.

Right.

> Or do you (or anybody else) has drivers which should be maintained for
> vanilla-kernel and/or vendor kernels and/or other kernels (to fix the
> breakage in a cosntructive way), we can provide you with an offer to do
> that.

Constructive criticism? Even of the sort that contributes something? What
are you thinking about?!
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Saturday 03 December 2005 17:35, Chris Wright wrote:
> relevant.  About the only thing I think is helpful in this case is perhaps
> one extra -stable cycle on the last branch when newest branch is released
> (basically flush the queue).  That much I'm willing to do in -stable.

Yay rah cool!

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Sunday 04 December 2005 23:59, Luke-Jr wrote:
> On Sunday 04 December 2005 23:22, Greg KH wrote:
> > On Sun, Dec 04, 2005 at 04:46:31AM +0000, Luke-Jr wrote:
> > > Well, devfs does have some abilities udev doesn't: hotplug/udev
> > > doesn't detect everything, and can result in rarer or non-PnP devices
> > > not being automatically available;
> >
> > Are you sure about that today?
>
> Nope, but I don't see how udev can possibly detect something that doesn't
> let the OS know it's there-- except, of course, loading the driver for it
> and seeing if it works.

Stuff shows up in /sys whether or not Linux has a driver loaded for it.

ls -l /sys/bus/*/devices

Hotplug insertion events are for when the _device_ shows up, not for when the 
driver is loaded.

> > And udev wasn't created to do everything that devfs does.
>
> Which might be a case for leaving devfs in. *shrug*

I think it was a polite way of saying "udev doesn't suck, have unfixable 
races, randomly crash your system..."

> > What information are you talking about here?
>
> I'm assuming everything in /etc/udev/rules.d/50-udev.rules used to be in
> the kernel for devfs-- perhaps it was PAM though, I'm not sure.
> Other than that, I don't expect that simply installing a new kernel module
> will allow the device to be detected automatically, but that some hotplug
> or udev configurations will need to be updated also.

On an unrelated note, the proposed file format for busybox's mdev looks 
something like:

hd[a-z]  0:3 700
hd[a-z][0-9]* 0:3 740
.* 0:0 700

There's a little more to it than that, but really, specifying ownership and 
permissions is all we _really_ care about.  (We're not trying to obsolete 
udev.)

The point is, 90% of the complexity of udev is optional.  This _can_ be a lot 
simpler if you're not trying to tackle strange persistent naming issues 
resulting in dynamically generated symlinks, and similar fun.  (Which we may 
add the ability to do as compile-time config options later, but perhaps not 
until somebody actually misses them...)

We really don't forsee having to update mdev for deal with new kernel 
versions...  ever, if we can help it.

And a dynamic module loader hanging off of /sbin/hotplug is probably 
(conceptually) a different tool from mdev...

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Lars Marowsky-Bree:

> The right way to address this is to work with the distribution of your
> choice to make these updates available faster.

Working with a distribution benefits that distribution alone.  Working
on (e.g.) kernel security advisories would benefit everyone.  It's not
a speed issue, it's more about coverage.  And full coverage is very
hard to get without support from the real developers.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Bernd Petrovitsch:

> On Tue, 2005-12-06 at 00:00 +0100, Florian Weimer wrote:
> [...]
>> fixes (and other critical bug fixes).  For picking functionality, I
>> agree, but critical bug fixes which basically affect everone are a
>> different matter.  It doesn't make sense to redo the same analysis
>> over and over again, at each vendor.
>
> Then vendors should cooperate/collaborate. Where's the problem?

Usually, publicly visisble security bug handling is not separated from
the main development effort, especially if there is already a
centralized team for that purpose.

It's also a waste of resources if someone with no detailed knowledge
of the first analysis (which was made when the bug was fixed) or the
source code in question has to redo the whole analysis, just to pick
up the correct patches and classify the vulnerability.  If you
duplicate the work just once, things are a bit better, but it's still
a waste of resources, and people not familiar with the code tend to
make more mistakes.

It's not that there isn't any cooperation, either.  As far as I can
tell, it's possible to get most insider know-how on vulnerabilities
once it is published.  It's just more time-consuming than necessary.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Sunday 04 December 2005 09:25, Richard Knutsson wrote:
> But I do wonder how copyright and GPL can co-exist. Do the copyright
> holder own the changes anybody else does to the code?
> Anyone care to explain?

The GPL is a copyright license.  A license is a permission statement, ala "you 
can pitch a tent on my lawn as long as you don't leave trash all over it".  
Doesn't change ownership of the lawn, just says what you can do with the lawn 
somebody else owns, and it can have strings attached.

> Thanks
> Richard Knutsson

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Sunday 04 December 2005 06:56, Indrek Kruusa wrote:
> After reading "Linux 2.6.15-rc5: off-line for a week" from Torvalds it
> seems like this:.
>
> a) Torvalds thinks that nobody cares about kernel testing
> b) other gurus (they are also only "on-line" testers nowadays) doesn't
> feel good with development model (or at least they have no resources to
> do testing [Torvalds])
> c) end-users (or those who are not kernel maintainers) are directed
> permanently to distros kernels and "stay away from kernel.org you
> wanna-bees!"

Yeah, normally this would mean it's a tuesday.  We're running a little early.

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Mon, Dec 05, 2005 at 02:30:09PM -0500, Bill Davidsen wrote:
> 
> Actually I would be happy with the stability of this series if people 
> would stop trying to take working features OUT of it! That's the largest 
> problem I see, not that the existing features are unstable, and we have 
> a -stable branch to cover that, but that I can't count on features I use 
> and which are required for useful work.
> 
> If a firm policy of not removing supported features until 2.7 was 
> adopted I don't see a problem. The bulk of the instability (not 
> absolutely all, I grant), is in new features, or features which aren't 
> working all that well in any case. But if existing features suddenly 
> drop out from beneath the user, then you will find people doing what you 
> mentioned, staying with old kernels with holes rather than moving to 
> kernels which are simply no longer functional.

You are thinking in terms of the old development model.

This is not an option since the current development model says that 
there might never be a 2.7 kernel series.

We might like it or not, but this is the current development model and 
Andrew and Linus don't seem to want to change it.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Monday 05 December 2005 15:21, Steven Rostedt wrote:
> Perhaps, we could start out having Greg and Chris just concentrate on
> every fifth branch instead of every one, and that way the stability will
> last much longer.

Ah, belling the cat.

Hint:  Any plan in a volunteer community that starts with "$BUSY_PEOPLE should 
do $THIS" fails.  Any plan that starts with "I could do $THIS" at least has a 
chance.

This is not limited to open source, by the way...

-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bernd Petrovitsch]

On Tue, 2005-12-06 at 00:00 +0100, Florian Weimer wrote:
[...]
> fixes (and other critical bug fixes).  For picking functionality, I
> agree, but critical bug fixes which basically affect everone are a
> different matter.  It doesn't make sense to redo the same analysis
> over and over again, at each vendor.

Then vendors should cooperate/collaborate. Where's the problem?

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Benjamin LaHaise]

On Mon, Dec 05, 2005 at 04:47:55PM -0600, Rob Landley wrote:
> So no in-kernel filesystem can get this right without help from userspace 
> (even devfs had devfsd), and as soon as you've got a userspace daemon to tell 
> the kernel who is who you might as well do the whole thing there, now that 
> the kernel is exporting everyting _else_ we need to know via /sys 
> and /sbin/hotplug.

/sbin/hotplug is suboptimal.  Even a pretty fast machine is slowed down 
pretty significantly by the ~thousand fork and exec that take place during 
startup.  For the most common devices -- common tty, pty, floppy, etc that 
every system has, this is a plain waste of resources -- otherwise known as 
bloat.

		-ben
-- 
"You know, I've seen some crystals do some pretty trippy shit, man."
Don't Email: <dont@kvack.org>.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Ben Collins wrote:

> What you're suggesting sounds just like going back to the old style of
> development where 2.<even>.x is stable, and 2.<odd>.x is development.
> You might as well just suggest that after 2.6.16, we fork to 2.7.0, and
> 2.6.17+ will be stable increments like we always used to do.

I'll let him speak to what he intended, but my idea of stable is to keep 
the features of 2.6.0 in 2.6.N for any value of N. Adding new stuff 
rapidly hasn't been nearly the problem people feared, but that's largely 
due to the efforts of akpm to act as throttle, and somehow get more 
people to try his versions and knock the corners off the new code before 
it goes mainline.

I do think the old model was better; by holding down major changes for 
six months or so after a new even release came out, people had a chance 
to polich the stable release, and developers had time to recharge their 
batteries so to speak, and to sit and think about what they wanted to 
do, without feeling the pressure to write code and submit it right away. 
Knowing that there's no place to send code for six months is a great aid 
to generating GOOD code.

The other advantage of a development tree was that features could be 
added and removed without the argument that it would break this or that. 
It was development, no one was supposed to use it for production, no one 
could claim that there was even an implied promise of things working or 
even existing. ipchains could have gone out of 2.6 with no more fuss 
than xiafs departing. The people who really want it stay with the old 
kernel.

To a large extent -mm has become the development kernel, and as neat as 
that is, a development model which depends on a small number of 
dedicated and talented people to make it work is fragile.

Just my thoughts, I think we had it right before, I think it's less 
inherently stable now.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Lee Revell:

> On Mon, 2005-12-05 at 22:05 +0100, Florian Weimer wrote:
>> * Lee Revell:
>> 
>> >> The point that just escaped you as the motivation for this thread was
>> >> the availability of security (or other critical) fixes for older
>> >> kernels. It would all be fine if, say, the fix for CVE-2004-2492 were
>> >> available for those who find 2.6.8 works for them (the fix went into
>> >> 2.6.14 BTW), and the concern is the development model isn't fit to
>> >> accomodate needs like this.
>> >> 
>> >
>> > If you want security fixes backported then you can get a distro kernel.
>> 
>> And these distro kernels appear magically from nowhere?
>> 
>
> No you get them from Red Hat or SuSE or whoever.

"Whoever"?  Debian?  Slackware?  Gentoo?  Even companies like SGI
might have difficulties providing security support for their custom
kernels, not to speak of tons of embedded developers.

Can I buy security support for my custom MIPS kernel, like I can buy
GCC support for the platform?  Is there a similar market?

> One of the core assumptions of the new development model is that
> distros whose business model involves paying people to do QA and
> regression testing and have access to bug reports from zillions of
> users are better positioned than kernel developers to decide what a
> "stable" kernel is.

But they aren't more qualified when it comes to extracting security
fixes (and other critical bug fixes).  For picking functionality, I
agree, but critical bug fixes which basically affect everone are a
different matter.  It doesn't make sense to redo the same analysis
over and over again, at each vendor.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Saturday 03 December 2005 18:20, Greg KH wrote:
> On Sat, Dec 03, 2005 at 11:50:20PM +0100, Matthias Andree wrote:
> > The point is, removing something that has worked well enough that some
> > people had a reason to use it, is not "stable".
>
> Please remember, no one is calling 2.6 "stable" anymore than they are
> calling it "development".  The current development model is different
> than what we used to do pre 2.6.  See the archives for details about
> this if you want more information.
>
> > Third, IF udev is so sexy but OTOH a real kernel-space devfs can be done
> > in 200 LoC as has been claimed so often,
>
> 282 LoC:
...
> > why in hell is this not happening?
>
> Because it's not the correct solution.

More detail on this:

On the busybox list we're currently working out a design for mdev, the 
micro-udev that'll go into busybox 1.2.  So we're thinking about this issue 
pretty carefully, as we speak.  What's the minimal amount of work we can't 
get away with not doing?

And much as we'd like to, we can't eliminate the config file.  In mdev we can 
accept the kernel's suggested names for devices, throw everything into a 
single directory with no subdirectories, even configure out hotplugging 
support (since not all embedded devices need that).  But nowhere in sys is 
there any hint about the correct ownership and permissions for a device, and 
you can't create a device node without specifying that.

The fundamental problem is that the kernel _can't_ tell us this through /sys 
because the kernel has no idea what users and groups are on a given system.  
It can't, and it shouldn't.  That's not it's job.  (It deals with uid and gid 
and never looks at /etc/passwd or /etc/groups.  And if it doesn't know who 
should own what, it can't know what permissions they should have either.)

So no in-kernel filesystem can get this right without help from userspace 
(even devfs had devfsd), and as soon as you've got a userspace daemon to tell 
the kernel who is who you might as well do the whole thing there, now that 
the kernel is exporting everyting _else_ we need to know via /sys 
and /sbin/hotplug.

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lee Revell]

On Mon, 2005-12-05 at 22:05 +0100, Florian Weimer wrote:
> * Lee Revell:
> 
> >> The point that just escaped you as the motivation for this thread was
> >> the availability of security (or other critical) fixes for older
> >> kernels. It would all be fine if, say, the fix for CVE-2004-2492 were
> >> available for those who find 2.6.8 works for them (the fix went into
> >> 2.6.14 BTW), and the concern is the development model isn't fit to
> >> accomodate needs like this.
> >> 
> >
> > If you want security fixes backported then you can get a distro kernel.
> 
> And these distro kernels appear magically from nowhere?
> 

No you get them from Red Hat or SuSE or whoever.  One of the core
assumptions of the new development model is that distros whose business
model involves paying people to do QA and regression testing and have
access to bug reports from zillions of users are better positioned than
kernel developers to decide what a "stable" kernel is.

Lee

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Lee Revell wrote:
> On Mon, 2005-12-05 at 11:17 -0500, Mark Lord wrote:
> 
>>>>>Ahh OK .. I don't use it, so wouldn't have been affected. That's one
>>>>>userspace interface broken during the series, does anyone have any more?
>>
>>Ah.. another one, that I was just reminded of again
>>by the umpteenth person posting that their wireless
>>no longer is WPA capable after upgrading from 2.6.12.
>>
>>Of course, the known solution for that issue is to
>>upgrade to the recently "fixed" latest wpa_supplicant
>>daemon in userspace, since the old one no longer works.
>>
>>Things like this are all too regular an occurance.
> 
> 
> The distro should have solved this problem by making sure that the
> kernel upgrade depends on a new wpa_supplicant package.  Don't they
> bother to test this stuff before they ship it?!?

Could you provide a little detail on the technology by which a distro 
checks for functionality against a kernel which wasn't necessarily 
released when the distro shipped. My udev doesn't generate /dev/timewarp.

Going to a new kernel in the same series shouldn't have to be treated as 
if it were a change to a whole new operating system, and shouldn't 
require completely replacing existing utilities with new ones which 
aren't backware compatible to allow fallback to the original kernel.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Steven Rostedt]

On Mon, 2005-12-05 at 21:52 +0100, Florian Weimer wrote:
> * Matthias Andree:
> 
> > Basically, no-one should have permission to touch any core parts, except
> > for fixes, until 2.7. Yes, that means going back to older models. Yes,
> > that means that the discussions will start all over. And yes, that means
> > that the cool stuff has to wait. Solution: release more often.
> 
> Would this alone change much?  I think what we really want is that our
> favorite branch (whatever it is) gets critical fixes forever (well,
> maybe one or two years, but this is forever).  This is a bit
> unrealistic because everyone has a slightly different branchpoint.
> Releasing more often doesn't change that, really.

Maybe that is what is needed.  A branch that all can use.  Have every 5
or so 2.6.x become a "stable" branch.  Where distributions and users can
work together on keeping it stable.  The rules to modifying such a
branch would pretty much stay with what it already takes to modify the
current 2.6.x.y branch.  If you want a feature, you must either take the
latest "unstable" 2.6.x branch or wait for the next "stable" 2.6.x
branch to merge.

Now who should chose which version the "stable" branch should be?  Well,
we could just say ever 5 branches will become one, or if we have a
"F*cked up" branch (really bad bug made it in), then we can skip it and
go to the 6th to branch.

Perhaps, we could start out having Greg and Chris just concentrate on
every fifth branch instead of every one, and that way the stability will
last much longer.  Again, if you want the latest functionality, you go
with the latest "unstable" release, or wait for the next stable.  Since
these releases come out about every month or two, waiting 5 releases
will last for almost a year.

For this to work, the normal releases would just continue like normal.
And just the marked branch will become stable.  This may be similar to
what Linus formally proposed.  Where he had every odd revision be
unstable, and every even stable.  What I'm suggesting would not make the
stable branch stable by what goes into it. It's just that those are the
branches that would have the .y version.  And then we could ignore the
other branches instead.

This idea combines pretty much the idea of the 2.7 with the current
2.6.x.y.  Actually it is more like the 2.7 approach, but it's hidden :-)
The problem with 2.7 is that nobody tests it, and it takes too long to
go from 2.6 to 2.8.  My method here hides that fact.  You just basically
say "here's the stable version" and let it fork.  Continue on with the
2.6.x and when you think too many people are using the last stable
version, and are not testing the current branch, just release the new
"stable", and pull everyone back in.


-- Steve

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

On Mon, 2005-12-05 at 22:00 +0100, Florian Weimer wrote:
> * Matthias Andree:
> 
> > The point that just escaped you as the motivation for this thread was
> > the availability of security (or other critical) fixes for older
> > kernels. It would all be fine if, say, the fix for CVE-2004-2492 were
> > available for those who find 2.6.8 works for them (the fix went into
> > 2.6.14 BTW), and the concern is the development model isn't fit to
> > accomodate needs like this.
> 
> Well, if there's a CVE name, the proper patch isn't *that* far away
> (someone has already done a bit of work to isolate the fix).  The real
> issue seems to be how to make sure that CVE names are assigned during
> the kernel development process (and not just as an afterthought by the
> security folks).

security@kernel.org works that way already in a way. Of course it'd be
nice to add a cve name while coding the security hole into the kernel,
but nobody is that clearvoyant ;) The hardest part is actually knowing
which versions are affected, especially when the code no longer quite is
the same, the locking rules got cleaned up in later versions (which
means the older kernel was a mess, so you're always looking at more
messy code than the "new" kernel). For some stuff this is easy. For
other stuff it for sure isn't, especially if you want to keep api/abi
compatibility, or at least low impact. Some security fixes just are
invasive. Those are rare, maybe 2 or 3 times a year or so. But they do
exist... unfortunate as it is. The irony is that doing a "hacky" less
invasive risk actually may introduce more risk to stability than doing
the full invasive fix. Nasty trade-offs there....
 

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Lee Revell:

>> The point that just escaped you as the motivation for this thread was
>> the availability of security (or other critical) fixes for older
>> kernels. It would all be fine if, say, the fix for CVE-2004-2492 were
>> available for those who find 2.6.8 works for them (the fix went into
>> 2.6.14 BTW), and the concern is the development model isn't fit to
>> accomodate needs like this.
>> 
>
> If you want security fixes backported then you can get a distro kernel.

And these distro kernels appear magically from nowhere?

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Matthias Andree:

> The point that just escaped you as the motivation for this thread was
> the availability of security (or other critical) fixes for older
> kernels. It would all be fine if, say, the fix for CVE-2004-2492 were
> available for those who find 2.6.8 works for them (the fix went into
> 2.6.14 BTW), and the concern is the development model isn't fit to
> accomodate needs like this.

Well, if there's a CVE name, the proper patch isn't *that* far away
(someone has already done a bit of work to isolate the fix).  The real
issue seems to be how to make sure that CVE names are assigned during
the kernel development process (and not just as an afterthought by the
security folks).

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Matthias Andree:

> Basically, no-one should have permission to touch any core parts, except
> for fixes, until 2.7. Yes, that means going back to older models. Yes,
> that means that the discussions will start all over. And yes, that means
> that the cool stuff has to wait. Solution: release more often.

Would this alone change much?  I think what we really want is that our
favorite branch (whatever it is) gets critical fixes forever (well,
maybe one or two years, but this is forever).  This is a bit
unrealistic because everyone has a slightly different branchpoint.
Releasing more often doesn't change that, really.

In the security area, I think there is enough experience out there to
collect data which would help each local branch maintainer to install
the relevant fixes.  But for general development, this seems to be
infeasible, unless you focus your software architecture on this
purpose (which is probably a terrible idea to do for kernel
development).

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

David Ranson wrote:
> Adrian Bunk wrote:
> 
> 
>>- support for ipfwadm and ipchains was removed during 2.6
>>
>>
> 
> Surely this one had loads of notice though? I was using iptables with
> 2.4 kernels.
> 
> 
>>- devfs support was removed during 2.6
>>
>>
> 
> Did this affect many 'real' users?
> 
> 
>>- removal of kernel support for pcmcia-cs is pending
>>- ip{,6}_queue removal is pending
>>- removal of the RAW driver is pending
>>
>>
> 
> I don't use any of these. I guess pcmcia-cs may be disruptive for laptop
> users.

You don't seem to grasp that thousands of people DO use these features, 
and by removing the features those users are blocked from security, 
reliability, and performance related changes. And there are a number of 
other features mentioned
> 
> So far I don't see evidence to suggest huge repeated userspace breakages
> between Kernel versions that were implied earlier in this thread.
> Whatever, we aren't going to see any more stable branches without
> volunteers to do the spadework. As has been pointed out, this won't
> always be an easy task.

To a large extent I don't think it's a needed task. If new stuff doesn't 
work that doesn't hurt established uses, it's only when changes like the 
PCI rethink go in that existing users are impacted. As long as things 
aren't taken OUT, the current kernel is usefully stable.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Adrian Bunk:

> I don't get the point where the advantage is when every distribution 
> creates it's own stable branches.

Different vendors have different needs WRT proprietary drivers,
experimental file systems, network stack tweaks.  Their release cycles
aren't synchronized, so it's not clear at which point you can make
sorely needed architectural changes in a stable kernel series (for
example, to fix some egregious performance issues, or complicated
security issues).

> What's wrong with offering an unified branch with few regressions for 
> both users and distributions?

One user's regression is another's bug fix.  And where do those
regressions come from if you don't make any changes in functionality? 8-)

> It's not that every distribution will use 
> it, but as soon as one or two distributions are using it the amount of 
> extra work for maintaining the branch should become pretty low.

Maybe, but I don't see why a vendor should give up its kernel
branding.

You mentioned security issues in your initial post.  I think it would
help immensely if security bugs would be documented properly (affected
versions, configuration requirements, attack range, loss type etc.)
when the bug is fixed, by someone who is familiar with the code.
(Currently, this information is scraped together mostly by security
folks, sometimes after considerable time has passed.)  Having a
central repository with this kind of information would enable vendors
and not-quite-vendors (people who have their own set of kernels for
their machines) to address more vulnerabilties promptly, including
less critical ones.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Arjan van de Ven wrote:
> On Sun, 2005-12-04 at 15:57 +0100, M. wrote:
> 
> 
>>if distros would align on those 6months versions those less
>>experienced users would get 5 years support on those kernels. 
> 
> 
> no distro gives 5 years of support for a kernel done every 6 months;
> they start such projects more like every 18 to 24 months (SuSE used to
> do it a bit more frequently but it seems they also slowed this down).
> 
> 
>>example: redhat, suse and mandriva are releasing their new product
>>using the latest 6months (or whatever) kernel; they are not going to
>>patch it except for new filesystems or bugfixes because of the new dev
> 
> 
> "except for" is a slipperly slope. And "except for bugfixes" would be
> wrong... those would be the ones that need to be in the kernel.org
> kernel. As well as new hardware support. At which point.. what is the
> difference? Where do 'features' stop and where do 'only needed bugfixes'
> begin?

Given the examples of 2.2 and 2.4 ongoing low level maintenence, I think 
that's a poor objection, a stable series (in the old sense) needs one 
maintainer to make the decisions on what goings in, and typically people 
will do the actualy work cooperating with the primary maintainer.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bill Davidsen]

Adrian Bunk wrote:
> The current kernel development model is pretty good for people who 
> always want to use or offer their costumers the maximum amount of the 
> latest bugs^Wfeatures without having to resort on additional patches for 
> them.
> 
> Problems of the current development model from a user's point of view 
> are:
> - many regressions in every new release
> - kernel updates often require updates for the kernel-related userspace 
>   (e.g. for udev or the pcmcia tools switch)
> 
> One problem following from this is that people continue to use older 
> kernels with known security holes because the amount of work for kernel 
> upgrades is too high.

Depending on where you work, "not working" may be acceptable vs. 
"working with a known security hole."
> 
> These problems follow from the development model.
> 
> The latest stable kernel series without these problems is 2.4, but 2.4 
> is becoming more and more obsolete and might e.g. lack driver support 
> for some recent hardware you want to use.
> 
> Since Andrew and Linus do AFAIK not plan to change the development 
> model, what about the following for getting a stable kernel series 
> without leaving the current development model:
> 
> 
> Kernel 2.6.16 will be the base for a stable series.
> 
> After 2.6.16, there will be a 2.6.16.y series with the usual stable 
> rules.
> 
> After the release of 2.6.17, this 2.6.16.y series will be continued with 
> more relaxed rules similar to the rules in kernel 2.4 since the release 
> of kernel 2.6.0 (e.g. driver updates will be allowed).

Actually I would be happy with the stability of this series if people 
would stop trying to take working features OUT of it! That's the largest 
problem I see, not that the existing features are unstable, and we have 
a -stable branch to cover that, but that I can't count on features I use 
and which are required for useful work.

If a firm policy of not removing supported features until 2.7 was 
adopted I don't see a problem. The bulk of the instability (not 
absolutely all, I grant), is in new features, or features which aren't 
working all that well in any case. But if existing features suddenly 
drop out from beneath the user, then you will find people doing what you 
mentioned, staying with old kernels with holes rather than moving to 
kernels which are simply no longer functional.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Sun, Dec 04, 2005 at 03:24:54PM -0800, Greg KH wrote:
> On Sun, Dec 04, 2005 at 12:56:50PM +0100, Matthias Andree wrote:
> > The problem is the upstream breaking backwards compatibility for no good
> > reason. This can sometimes be a genuine unintended regression (aka.
> > bug), but quite often this is deliberate breakage because someone wants
> > to get rid of cruft. While the motivation is sound, breaking between
> > 2.6.N and 2.6.M must stop.
> 
> What are we breaking that people are complaining so much about?
> Specifics please.
> 
> And if you bring up udev, please see my previous comments in this thread
> about that issue.
> 
> It isn't userspace stuff that is breaking, as applications built on 2.2
> still work just fine here on 2.6 for me.
> 
> Yes we break in-kernel apis, all the time, that's fine.  See
> Documentation/stable-api-nonsense.txt for details about why we do that.
> 
> So again, specifics please?

It's the kernel-related userspace that is the problem (besides 
regressions that are simply bugs).

Be it the devfs removal, the requirement for a more recent
wpa_supplicant package or my pending removal of the obsolete
raw driver.

> thanks,
> 
> greg k-h

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Sun, Dec 04, 2005 at 09:31:12PM -0600, Rob Landley wrote:
> On Saturday 03 December 2005 11:53, Adrian Bunk wrote:
> > On Sat, Dec 03, 2005 at 05:17:41PM +0000, David Ranson wrote:
> > > Steven Rostedt wrote:
> > > >udev ;)
> > > >
> > > >http://seclists.org/lists/linux-kernel/2005/Dec/0180.html
> > >
> > > Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> > > userspace interface broken during the series, does anyone have any more?
> >
> > - support for ipfwadm and ipchains was removed during 2.6
> > - devfs support was removed during 2.6
> > - removal of kernel support for pcmcia-cs is pending
> > - ip{,6}_queue removal is pending
> > - removal of the RAW driver is pending
> 
> So what you're upset about is the feature removal scheduling mechanism, which 
> usually gives a full year's warning, and the removal patch can be reversed 
> into a feature addition patch you can maintain outside the tree if you really 
> care?

I'm not upset about is the feature removal scheduling mechanism.

Please check who has most entries in 
Documentation/feature-removal-schedule.txt ...

The removal of features within the 2.6 series is an essential part of 
the current development model.

The problem is the lack of a long-living relatively stable series within 
the development model.

> Rob

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Monday 05 December 2005 10:28, Lee Revell wrote:
> >
> > Things like this are all too regular an occurance.
>
> The distro should have solved this problem by making sure that the
> kernel upgrade depends on a new wpa_supplicant package.  Don't they
> bother to test this stuff before they ship it?!?

I've broken stuff by upgrading glibc, upgrading X, upgrading KDE...

Upgrading the kernel is safer?  (Anybody remember 2.4.11-dontuse?)

Yay, modular component-based design.  We have standard interfaces so that 
stuff mostly works when you swap out different versions (or entire different 
components).  This is cool.

But if such interfaces were actually sufficient to specify all the 
functionality you actually want to use, why would you ever need to upgrade a 
component implementing that interface to a new version?

The real problem people are seeing is that the rate of change has increased.  
Linus used to be a hell of a bottleneck, and this stopped being the case when 
source control systems took over a lot of the patch tracking, ordering, 
batching, and integration burden.

Automating the patch flow allowed entire subsystems to be effectively 
delegated (and thus the "lieutenants" layer formed and was formalized), and 
each of _them_ is now doing as much work as Linus used to do.

And _that_ is why there is no longer any point in -devel forks, because Linus 
is now fielding as many patches in a month as he used to in a year.  That 
means in every 3 months the Linux kernel undergoes as much development (in 
terms of patches merged and lines of code changed) as an entire -devel series 
used to do.  (Somebody confirm the numbers, these are approximations.)

There's no point in launching a fork that's only expected to last three 
months.  Hence no -devel fork.

Also, forks are cheaper now.  The new source control tools (not just git but 
quilt and ketchup and so on) allow multiple parallel trees to be trivially 
integrated.  It used to take somebody like Alan Cox all his spare time to 
maintain a tree and merge with linus, and back then the -ac tree was very 
special.  Now Con Colivas can maintain a tree in his spare time with a day 
job as an anesthesiologist, and this is _normal_.  There are dozens of trees 
out there feeding into each other, and anybody who wants to can grab the 
relevant subsystem tree and try it out to make sure that the issue they care 
about is fixed, and be assured that it'll all flow in to Linus's tree.

What's special about Linus's tree is that it's the point to the wedge.  This 
is the farthest we've advanced, this is your best bet.  There's always 
something wrong with any piece of software, but half the complaints have 
always been that something is fixed but not merged yet.  (Orinoco scanning, 
ISDN, ALSA, examples are legion.)   That's getting way better.

Now the _new_ class of complaints is that the IPW2200 driver that first got 
merged was too old.  (I noticed this because that's the wireless card in my 
laptop.)  Stop and think about that for a bit.  People were used to IPW2200 
not being there at all, so they could easily add an external patch.  Then 
2.6.14 grew partial IPW2200 support, but with an older driver, and people 
were mad because the external patch they had to add support only applied when 
the driver wasn't there at all, and it didn't apply over the older version.  
They were mad that _insufficient_ support showed up.

This is being fixed in 2.6.15.  It didn't last long.

The new model is that if the kernel has half what you need, you need to come 
up with an incremental patch to get it the rest of the way, and submit that.  
And the up-side is that it'll go in pretty fast now.  Yes, the kernel is 
changing rapidly enough that external patches probably need to be fixed up 
with every new version.  (And if you're using the nvidia driver, this sucks 
rocks.  You were warned.)

The other thing people are complaining about is the deprecation schedule.  
Notice is posted prominently up to a YEAR before a feature gets yanked.  
Apparently, they want to upgrade to the new version when it comes out, but 
don't want to read the instructions for this version (X went away), or the 
warnings about upcoming versions...

Possibly if we had a CHANGES file at the root level of the source mentioning

A) What's new in this version.

B) What's slated for next version (DEPRECATION COMING).

C) What was new in the last version, in case you missed it.

A combination of Documentation/feature-removal-schedule.txt and 
http://wiki.kernelnewbies.org/LinuxChanges, with emphasis on userspace tools 
known to be impacted.

> Lee

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Mon, 05 Dec 2005, Lee Revell wrote:

> On Mon, 2005-12-05 at 17:44 +0100, Matthias Andree wrote:
> > This constant shifting the blame on someone else is becoming
> > offensive.
> > 
> > Diligent maintainers put "INCOMPATIBLE CHANGES" sections up front in
> > their release announcements or notes, that is the upstream
> > maintainer's chance to state "wpa_supplicant version >= 1.2.3
> > required" and really pass the buck on to the distros. Without such
> > upgrade-required-for: notes, it's just rude. "We break everything but
> > you need to find out for
> > yourself which..."
> > 
> 
> I'm not trying to shift blame, I am just saying that with their access
> to a larger hardware and user base the distros are in a much better
> position to regression test changes than the kernel developers.

You just described what shifting burden or blame means.

Are you seriously saying it's the distributors' fault for not trying the
random monkey patch on end users machines?

Heck, SUSE 9.2 ate a complete server because SUSE (they take the blame)
didn't manage to (1) notice in time, (2) therefore package a CRITICAL
(as in causes data corruption) MegaRAID bugfix. Do you really want such
things to happen as intrinsic part of the kernel development? Do the
upstream gurus such as Linus and Andrew want that? If so, they can say
so and we'll see the companies running for their sheer lives and putting
their money into other kernels.

BSD makes it only easier to provide binary modules, because you don't
even have to discuss with anyone if it's derived work or not, you can
just embrace, extend and lock the beast up and everyone in.

> And I didn't even mention the cases where the distros just don't do
> their homework.  For example in order to insulate users from internal
> changes ALSA has a kernel and userspace (alsa-lib) component and both
> must be upgraded in sync to properly support new hardware.  This is
> common knowledge.  But many distros keep shipping kernel upgrades that
> introduce new ALSA drivers but don't bother to make the kernel upgrade
> depend on an alsa-lib upgrade, or even to make a newer alsa-lib
> available.

Major distros usually aim for small and well-audited changes in order
not to make things worse, at least where end-user support is concerned.

Given the development pace and the ridiculous policy which is
effectively "you may break everything in the two weeks after release,
and we'll collect those fixes that come in until Linus's machine works
and ship without the rest".

Basically, no-one should have permission to touch any core parts, except
for fixes, until 2.7. Yes, that means going back to older models. Yes,
that means that the discussions will start all over. And yes, that means
that the cool stuff has to wait. Solution: release more often.

I'm so bold as to claim that a new minor release every 6 months with a
tight "only fixes allowed as core changes" policy would satisfy many. Of
course you cannot break the binary driver of the day that way, but it
also means fewer chances to break NFS, XFS, MegaRAID and whatnot.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lee Revell]

On Mon, 2005-12-05 at 17:44 +0100, Matthias Andree wrote:
> This constant shifting the blame on someone else is becoming
> offensive.
> 
> Diligent maintainers put "INCOMPATIBLE CHANGES" sections up front in
> their release announcements or notes, that is the upstream
> maintainer's chance to state "wpa_supplicant version >= 1.2.3
> required" and really pass the buck on to the distros. Without such
> upgrade-required-for: notes, it's just rude. "We break everything but
> you need to find out for
> yourself which..."
> 

I'm not trying to shift blame, I am just saying that with their access
to a larger hardware and user base the distros are in a much better
position to regression test changes than the kernel developers.

And I didn't even mention the cases where the distros just don't do
their homework.  For example in order to insulate users from internal
changes ALSA has a kernel and userspace (alsa-lib) component and both
must be upgraded in sync to properly support new hardware.  This is
common knowledge.  But many distros keep shipping kernel upgrades that
introduce new ALSA drivers but don't bother to make the kernel upgrade
depend on an alsa-lib upgrade, or even to make a newer alsa-lib
available.

Lee

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Jakob Oestergaard]

On Mon, Dec 05, 2005 at 05:44:08PM +0200, Pekka Enberg wrote:
> Hi,
> 
> On Sun, Dec 04, 2005 at 02:39:31PM -0800, Greg KH wrote:
> > > Have you filed a but at bugzilla.kernel.org about this?  If not, how do
> > > you expect it to get fixed?
> 
> On 12/5/05, Jakob Oestergaard <jakob@unthought.net> wrote:
> > I don't expect to get it fixed. It's futile. It can get fixed in one
> > version and broken two days later, and it seems the attitude is that
> > that is just fine.
> 
> I don't think anyone breaks things on purpose.

Of course not, silly :)

But there's a difference between having a tree where you fix bugs and
having a tree where you are very lax about major changes. By including
major changes you (knowingly or not) risk breaking things, and things do
break regularly.

> Please feel free to
> report the bug as many times as necessary to get it fixed.

Thank you :)

If it did not occur to you, then I was never in doubt of this. I tried
to point out a problem in this approach - namely, that you will never
end up with a stable tree if major changes (read; new bugs) go in as
fast as old bugs are squashed.

You do get a tree that is evolving very quickly, and which is somewhat
stable for most purposes. Whether or not that is good enough for
everyone, was pretty much the topic of this thread as I understand it.

> You
> shouldn't be complaining if you're not doing your part.

I'm not complaining.  I'm voicing my oppinion in a thread that discusses
whether or not it would be a good idea to try and produce a stable tree.

I think it would be a good idea, and you're free to disagree.

Please read the thread if you're in doubt of the context of my comments  :)

-- 

 / jakob

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Mon, 05 Dec 2005, Lee Revell wrote:

> On Mon, 2005-12-05 at 11:17 -0500, Mark Lord wrote:
> > >>>Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> > >>>userspace interface broken during the series, does anyone have any more?
> > 
> > Ah.. another one, that I was just reminded of again
> > by the umpteenth person posting that their wireless
> > no longer is WPA capable after upgrading from 2.6.12.
> > 
> > Of course, the known solution for that issue is to
> > upgrade to the recently "fixed" latest wpa_supplicant
> > daemon in userspace, since the old one no longer works.
> > 
> > Things like this are all too regular an occurance.
> 
> The distro should have solved this problem by making sure that the
> kernel upgrade depends on a new wpa_supplicant package.  Don't they
> bother to test this stuff before they ship it?!?

This constant shifting the blame on someone else is becoming
offensive.

Diligent maintainers put "INCOMPATIBLE CHANGES" sections up front in
their release announcements or notes, that is the upstream maintainer's
chance to state "wpa_supplicant version >= 1.2.3 required" and really
pass the buck on to the distros. Without such upgrade-required-for:
notes, it's just rude. "We break everything but you need to find out for
yourself which..."

Let's not mention that section 2, 7 and 9 manual pages should be
maintained by the kernel developers rather than an external maintainer.

If you need a luminous example how release management works, look at
Postfix. I don't suggest taking Postfix's development model of printing
the diffs and using text marker and pencil, but the "Incompatible
changes", "Major changes" in Release Notes, with all the details in a
separate changelog, works rather well for distros and direct users.

My suggestion is to build upon the signed-off-by: stuff and require that
every incompatible change carry such a RFC2822-header-like line if it is
to be merged into baseline, and unconditionally back out all
incompatibilities that are later found but not documented.

Perhaps just making people actually write such notes can cut the number
of these shipwrecks.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lee Revell]

On Mon, 2005-12-05 at 11:17 -0500, Mark Lord wrote:
> >>>Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> >>>userspace interface broken during the series, does anyone have any more?
> 
> Ah.. another one, that I was just reminded of again
> by the umpteenth person posting that their wireless
> no longer is WPA capable after upgrading from 2.6.12.
> 
> Of course, the known solution for that issue is to
> upgrade to the recently "fixed" latest wpa_supplicant
> daemon in userspace, since the old one no longer works.
> 
> Things like this are all too regular an occurance.

The distro should have solved this problem by making sure that the
kernel upgrade depends on a new wpa_supplicant package.  Don't they
bother to test this stuff before they ship it?!?

Lee

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Mark Lord]

>>>Ahh OK .. I don't use it, so wouldn't have been affected. That's one
>>>userspace interface broken during the series, does anyone have any more?

Ah.. another one, that I was just reminded of again
by the umpteenth person posting that their wireless
no longer is WPA capable after upgrading from 2.6.12.

Of course, the known solution for that issue is to
upgrade to the recently "fixed" latest wpa_supplicant
daemon in userspace, since the old one no longer works.

Things like this are all too regular an occurance.

Cheers

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Pekka Enberg]

Hi,

On Sun, Dec 04, 2005 at 02:39:31PM -0800, Greg KH wrote:
> > Have you filed a but at bugzilla.kernel.org about this?  If not, how do
> > you expect it to get fixed?

On 12/5/05, Jakob Oestergaard <jakob@unthought.net> wrote:
> I don't expect to get it fixed. It's futile. It can get fixed in one
> version and broken two days later, and it seems the attitude is that
> that is just fine.

I don't think anyone breaks things on purpose. Please feel free to
report the bug as many times as necessary to get it fixed. You
shouldn't be complaining if you're not doing your part.

                                          Pekka

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Jakob Oestergaard]

On Sun, Dec 04, 2005 at 02:39:31PM -0800, Greg KH wrote:
> On Sun, Dec 04, 2005 at 06:00:49PM +0100, Jakob Oestergaard wrote:
> > 
> > If the kernel was stable (reliability wise - as in "not crashing") then
> > you'd be perfectly right.
> 
> But isn't it? :)

I like your sense of humor :)

> > In the real world, however, admins currently need to pick out specific
> > versions of the kernel for specific workloads (try running a large
> > fileserver on anything but 2.6.11.11 for example - any earlier or later
> > kernel will barf reliably.
> 
> Have you filed a but at bugzilla.kernel.org about this?  If not, how do
> you expect it to get fixed?

I don't expect to get it fixed. It's futile. It can get fixed in one
version and broken two days later, and it seems the attitude is that
that is just fine.

After a long long back-and-forth, 2.6.11 was fixed to the point where it
could reliably serve files (at least on uniprocessor configurations -
and in my setup I don't see problems on NUMA either, but as far as I
know that's just me being lucky).

Right after that, someone thought it was a great idea to pry out the PCI
subsystem and shovel in something else.  Find, that's great for a
development kernel, but for a kernel that's supposed to be stable it's
just not something you can realistically do and expect things to work
afterwards.  And things broke - try mounting 10-20 XFS filesystems
simultaneously on 2.6.14.  Boom - PCI errors.

Now what? Do I as a user upgrade my production environment to the latest
and greatest kernel experiment, hope that the problems can be fixed
quickly, and hope that I don't lose too much data in the process?
(remember I will have people unable to do their jobs whenever the file
server is down).   Or do I stay on 2.6.11.11 which works on this
particular server?

I think I stay.

> 
> > For web serving it's another kernel that's golden, I forgot which).
> 
> That sounds very strange, the same kernel version should work just as
> well for all workloads.  If not, it's a bug and should be fixed.

Well...  You have bugs in different places in different kernels. It's
perfectly understandable that kernel A works for workload p and fails on
workload q, where kernel B works for workload q and fails on p.

> 
> > There are very very good reasons for offering a 'stable series' in plain
> > source-tree form - lots of admins of real-world systems need this.
> 
> But it sounds like you will want different stable series depending on
> what kind of server you are running.  And that will be even more work...

The idea would be to fix the actual bugs. After a while, one could have
a kernel of higher quality with fewer bugs, making it a lot more likely
that the *same* kernel tree could be used for both workloads A and B.

It's really very simple :)

Now, I'm just giving my oppinion as a user, and my advise as a developer
- I know how much it sucks to postpone new great cleanups or features,
just because some policy says the current branch has to be 'stable'. But
I also know how much it sucks to have users complain that a new feature
broke their existing setup. That's not a problem for a kernel developer
of course, because users don't pay for the service and the "if it breaks
you get to keep the pieces" attitude can be defended. But as a user, it
really really sucks, even if you get to keep the pieces.

I don't mean to be entirely negative - sure there are great things about
the new development model. But there is a very significant downside for
at least a group of users too.

My 0.02 Euro, for what it's worth.

-- 

 / jakob

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Florian Weimer]

* Greg KH:

> On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
>> 
>> Why can't this be done by distributors/vendors?
>
> It already is done by these people, look at the "enterprise" Linux
> distributions and their 5 years of maintance (or whatever the number
> is.)
>
> If people/customers want stability, they already have this option.

It seems that vendor kernels lack most DoS-related fixes.  I'm only
aware of a single vendor which tracks them to the point that CVE names
are assigned.

Vendor kernels are not a panacea, either.  With some of the basic
support contracts (in the four-figure range per year and CPU), the
vendor won't look extensively at random kernel crashes which could (in
theory) be attributed to faulty hardware, *and* you don't get
community support for these heavily patched kernel collages.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> We used to have this back in the 2.4 days with longer release cycles.
> However I am not sure if that was better or worse.

2.4 also had its fair share of regressions. Just that people by now have
forgotten about those ;)

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bernd Eckenfels]

In article <20051205114028.GD5148@marowsky-bree.de> you wrote:
> As I said, please, go on maintaining a release for a longer period of
> time.

On the other hand: 

Since it is such a ugly big boring and complicated task, why do you still
think it can be done by volunteers? (or why will commercial distributions
have the power to pay for this in the long run)

I think this is exactly the reason why it cannot be done in parallel by
volunteers without support from all of the contributors. With an official
stable trunk it attrackts more backports. It is clear where the security
fixes must happen, etc.

We used to have this back in the 2.4 days with longer release cycles.
However I am not sure if that was better or worse.

Gruss
Bernd

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Willy Tarreau]

On Mon, Dec 05, 2005 at 12:40:28PM +0100, Lars Marowsky-Bree wrote:
> On 2005-12-05T12:34:20, Willy Tarreau <willy@w.ods.org> wrote:
> 
> > > Anyway, good luck to you.
> > > 
> > > The current 2.6.x.y-stable series is quite sane, because they are
> > > essentially just fixing very critical bugs in very recent kernels, with
> > > little back porting effort.
> > 
> > I agree it is sane. The problem is that it does not exist for long enough.
> > When you have 2.6.14.X working perfectly and you need a fix for a newly
> > discovered security fix which only exists in 2.6.15.Y, then you have to
> > leave 2.6.14 and enter 2.6.15. That is the problem, because for just a
> > fix, you change megabytes of source code which will bring their equivalent
> > in bugs.
> 
> As I said, please, go on maintaining a release for a longer period of
> time.

As I said, I know this is difficult, I already do this for 2.4 and 2.4 is
not moving fast. But what Adrian wants to do might be far more difficult.
That's why I suggest him to do "only" this, he will have less work and get
a lot of happy users.

Regards,
Willy

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lars Marowsky-Bree]

On 2005-12-05T12:34:20, Willy Tarreau <willy@w.ods.org> wrote:

> > Anyway, good luck to you.
> > 
> > The current 2.6.x.y-stable series is quite sane, because they are
> > essentially just fixing very critical bugs in very recent kernels, with
> > little back porting effort.
> 
> I agree it is sane. The problem is that it does not exist for long enough.
> When you have 2.6.14.X working perfectly and you need a fix for a newly
> discovered security fix which only exists in 2.6.15.Y, then you have to
> leave 2.6.14 and enter 2.6.15. That is the problem, because for just a
> fix, you change megabytes of source code which will bring their equivalent
> in bugs.

As I said, please, go on maintaining a release for a longer period of
time.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Willy Tarreau]

On Mon, Dec 05, 2005 at 11:55:36AM +0100, Lars Marowsky-Bree wrote:
> On 2005-12-05T07:26:09, Willy Tarreau <willy@w.ods.org> wrote:
> 
> > What I think should be done is to still maintain older 2.6
> > (eg: 2, 3 or 4 previous releases) so that people will have
> > the time to switch to a new one. And I think that what Adrian
> > wants to do would be useful *only* if he proceeds that way.
> >  
> > Maybe you should just join forces, eg Chris and you to catch
> > new patches, and Adrian to merge them to older kernels ? Every
> > software maker always supports a few older releases for the
> > people who need to stay on something stable, and it is clearly
> > what is missing now in 2.6.
> 
> Well, this is probably the most useful suggestion so far. The kernel is
> free land; if you or someone else wants to maintain the upcoming 2.6.16
> "forever", and backport fixes or selected features, by all means, do it.
> Define your policy, set up a tree, and off you go.
> 
> If Adrian will maintain it, it'll for sure be the most static kernel
> ever.
> 
> This won't impact the Linux kernel, which will just continue to run its
> course. The kernel process as a whole doesn't need to change; just
> someone needs to do the grunt work.
> 
> If your kernel is wildly successful and adopted by users as well as
> distributions, you'll be very happy and tell us 'told ya so!'. If not,
> no harm will be done either, and you'll have the kernel you want for
> your own purposes.
> 
> Be aware however that this is a very painful job. Trust me, I've been
> involved with the receiving end of maintaining such a kernel for SLES
> for a couple of releases. ;-)
> 
> Which is exactly the point: it's so painful that for this, people want
> to be paid, and don't like doing it in their spare time. You may
> maintain it for 6 months, sure, which will be less painful than
> maintaining it for 5, 7 years, but when you rebase, you'll still put
> your users into the dependency hell, and they won't have tested the
> intermediate releases... Ouch. Not to mention that not every backported
> fix is trivial to do.
> 
> Anyway, good luck to you.
> 
> The current 2.6.x.y-stable series is quite sane, because they are
> essentially just fixing very critical bugs in very recent kernels, with
> little back porting effort.

I agree it is sane. The problem is that it does not exist for long enough.
When you have 2.6.14.X working perfectly and you need a fix for a newly
discovered security fix which only exists in 2.6.15.Y, then you have to
leave 2.6.14 and enter 2.6.15. That is the problem, because for just a
fix, you change megabytes of source code which will bring their equivalent
in bugs.

Regards,
willy

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Marcus Meissner]

In article <1133714480.5188.62.camel@laptopd505.fenrus.org.suse.lists.linux.kernel> you wrote:
> On Sun, 2005-12-04 at 17:11 +0100, Matthias Andree wrote:
>> On Sun, 04 Dec 2005, Arjan van de Ven wrote:
>> 
>> > On Sun, 2005-12-04 at 15:57 +0100, M. wrote:
>> > 
>> > > 
>> > > if distros would align on those 6months versions those less
>> > > experienced users would get 5 years support on those kernels. 
>> > 
>> > no distro gives 5 years of support for a kernel done every 6 months;
>> > they start such projects more like every 18 to 24 months (SuSE used to
>> > do it a bit more frequently but it seems they also slowed this down).
>> 
>> SUSE end-user distros (SUSE LINUX <version>) are released every 6 months
>> or so, and are supported for 24 months. Their "enterprise server" is
>> supported for 60 months though, SLES 9 forked off 9.1.
> 
> sure.. but they don't add new hw support really, and I'd not be
> surprised if they rebase to a newer upstream kernel after a while. I
> know we did that for RHL, eg RHL 7.(Y-1) got the kernel of RHL7.Y after
> RHL7.Y was released, not only to keep the maintenance down, but more so
> to get all the bugfixes and hardware support out to customers. 

We tried rebasing the kernel from 2.4.19 to 2.4.21 in a SLES 8 service
pack. It was like doing a new product.

Currently for SLES our policy is to add new driver support at Service Pack
releases.... Currently this list includes major updates to e1000 and other
Gigabit network drivers for instance (and likely other things, I am not
really keeping track).

So:

- SUSE Linux distro every 6 months with then current mainline, 
  24 months lifetime, kernel gets only critical bugfixes and security fixes.


- SUSE Linux Enterprise Line (every 18 - 24 months) with then current mainline,
  kernel version stays stable.

  Non Service Pack kernel updates:
  	- security and critical bugfixes only.

  Service Packs kernel updates:
  	- new features (but only with backing business case/ partner request)
	- driver updates for enterprise critical hardware

Ciao, Marcus (only responsible for the security part luckily)

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lars Marowsky-Bree]

On 2005-12-05T07:26:09, Willy Tarreau <willy@w.ods.org> wrote:

> What I think should be done is to still maintain older 2.6
> (eg: 2, 3 or 4 previous releases) so that people will have
> the time to switch to a new one. And I think that what Adrian
> wants to do would be useful *only* if he proceeds that way.
>  
> Maybe you should just join forces, eg Chris and you to catch
> new patches, and Adrian to merge them to older kernels ? Every
> software maker always supports a few older releases for the
> people who need to stay on something stable, and it is clearly
> what is missing now in 2.6.

Well, this is probably the most useful suggestion so far. The kernel is
free land; if you or someone else wants to maintain the upcoming 2.6.16
"forever", and backport fixes or selected features, by all means, do it.
Define your policy, set up a tree, and off you go.

If Adrian will maintain it, it'll for sure be the most static kernel
ever.

This won't impact the Linux kernel, which will just continue to run its
course. The kernel process as a whole doesn't need to change; just
someone needs to do the grunt work.

If your kernel is wildly successful and adopted by users as well as
distributions, you'll be very happy and tell us 'told ya so!'. If not,
no harm will be done either, and you'll have the kernel you want for
your own purposes.

Be aware however that this is a very painful job. Trust me, I've been
involved with the receiving end of maintaining such a kernel for SLES
for a couple of releases. ;-)

Which is exactly the point: it's so painful that for this, people want
to be paid, and don't like doing it in their spare time. You may
maintain it for 6 months, sure, which will be less painful than
maintaining it for 5, 7 years, but when you rebase, you'll still put
your users into the dependency hell, and they won't have tested the
intermediate releases... Ouch. Not to mention that not every backported
fix is trivial to do.

Anyway, good luck to you.

The current 2.6.x.y-stable series is quite sane, because they are
essentially just fixing very critical bugs in very recent kernels, with
little back porting effort.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Mon, 05 Dec 2005, Willy Tarreau wrote:

> However, the problem is that you stop maintaining old versions
> and quickly switch to new ones when a new kernel comes up. I know
> it's not easy, and may be terribly more difficult to maintain
> several versions in a development kernel than in a stable kernel.
> But imagine the users' position : they run 2.6.14.3 which finally
> fixes all their problems. Then they get a new problem, but 2.6.15
> comes out. There will not be any other 2.6.14, so they have the
> choice of staying to 2.6.14.3 or jumping to fresh new and barely
> tested 2.6.15.

"Regression" as the threat of updating. That was the starting point :)

I believe the reason to abandon the previous "stable" branchlet was the
assumption that the new kernel had all fixes from the previous "stable"
merged, i. e. every patch between 2.6.14 and 2.6.14.3 would become part
of 2.6.15 (or the underlying problem addressed by a patch were resolved
some other way).

> What I think should be done is to still maintain older 2.6
> (eg: 2, 3 or 4 previous releases) so that people will have
> the time to switch to a new one. And I think that what Adrian
> wants to do would be useful *only* if he proceeds that way.

Perhaps a fixed number of releases doesn't cut it. Perhaps a fixed time
neither. If the changes betweeen two subsequent releases is low, one or
more extra versions come for free, but if lots of changes go in all over
the map, it's going to be a royal pain.

It ultimately boils down to the question: how far^Wfast the upstream
wants to run away^W^Wprogress from its previous release.

> Also, I think differently from Adrian. He wants to backport
> all new drivers and new features, while I think that they are
> the most sensible parts and the one which bring the more
> changes to the kernel. In fact, we should *only* maintain
> security and critical fixes on older releases. People in the
> need of a new driver must upgrade for this. This is the

I think there is no such thing as The Single One New Driver[tm]. Some
are quite intrusive, some aren't. Sometimes the new driver works with
older kernels (if the driver is self-contained, for instance just a
dozen new lines of code in an existing driver), sometimes not (because
midlayer or core changes are required to support the new driver).

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Michael Frank]

On Saturday 03 December 2005 16:39, Arjan van de Ven wrote:
> > > this is a contradiction. You can't eat a cake and
> > > have it; either you're really low churn (like
> > > existing -stable) or you start adding new features
> > > like hardware support. the problem with hardware
> > > support is that it's not just a tiny driver update.
> > > If involves midlayer updates as well usually, and
> > > especially if those midlayers diverge between your
> > > stable and mainline, the "backports" are getting
> > > increasingly unsafe and hard.
> >
> > In the beginning, backporting hardware support is
> > relatively easy, and therefore cherry-picking from
> > mainline 2.6 should be relatively safe.
>
> and then there's reality. At least in my experience as
> distro kernel maintainer... you can do this for a few
> months, but it gets exponentially more expensive after 4
> to 5 months. And "safe" is just not true. API, midlayer
> and locking changes in the newer kernels just void that
> concept entirely. And then there's the testing dillema;
> the people who'd run such a branch are EXACTLY the ones
> who wouldn't test prereleases of such branch (and yes 2.4
> suffers from this as well).
>
> I doubt many distros would go for it as well for longer
> than a few months, simply because the hardware support
> and other features are going to be needed for them.
>
> > Things will change as time passes by, but then there's
> > the possibility to open the next branch and bring the
> > older branch into a security-fixes only mode.
>
> if you end up with 5 such branches it's no longer fun,
> trust me on that. Especially if the security fix is in a
> tricky area or a high flux area, then it's just not a
> matter of a simple backport anymore, even knowing if
> you're vulnerable or not is going to be a pain. And then
> there are the holes that happened to have gone away by
> later changes... what are you going to do then... put
> those changes in? that won't work longer term.
>
> > > If the current model doesn't work as you claim it
> > > doesn't, then maybe the model needs finetuning. Right
> > > now the biggest pain is the userland ABI changes that
> > > need new packages; sometimes (often) for no real hard
> > > reason. Maybe we should just stop doing those bits,
> > > they're not in any fundamental way blocking general
> > > progress (sure there's some code bloat due to it, but
> > > I guess we'll just have to live with that).
> >
> > IOW, we should e.g. ensure that today's udev will still
> > work flawlessly with kernel 2.6.30 (sic)?
>
> I'd say yes. It doesn't need to support all new
> functionality, but at least what it does today it should
> be able to do then. If that really isn't possible maybe
> udev should be part of the kernel build and per kernel
> version.

Most problems are avoided when packages closely linked to 
the kernel like udev and  pcmcia will be updated by the 
distro together with the kernel by way of package version 
dependencies matching for example 2.6.14 to udev 065-069
and kernel 2.6.15 to udev 070. udev 070 and later could 
block kernels <=2.6.14.

udev bit me recently when a scanner stopped working after 
updating to udev 070. In that way I had a chance to figure 
out how udev  works and how to make some rules. Neat! ;)

Perhaps extra care should be taken by the distro to not 
break the 50-udev.rules configuration file.  

>
> > This could work, but it should be officially announced
> > that e.g. a userspace running kernel 2.6.15 must work
> > flawlessly with _any_ future 2.6 kernel.
>
> I would argue that this in theory already is the current
> policy. Now "any" is pretty wide, but still. Maybe any
> such changes need to be scheduled to specific kernel
> releases only. Eg only do it every 4th or 5th kernel
> release.

My 2 cents:

Test drive some rc's (2.6.15-rc)
Use current -stable kernel as much as possible (2.6.14.3)
Critical apps use -stable -1 or -2 (2.6.13.x or 2.6.12.x)

Using -stable to -stable -2 one is about 3 months behind the 
latest and greatest instability ;).

As to security, most vulnerabilities are hard to exploit 
remotely and practical security can be much more improved 
by hiding detailed software versions from clients.  Apache 
2 on linux 2.6 will do instead of providing full vendor 
specific package versions!

As to drivers, in case 3 month driver delay matters, HW 
vendor can improve situation substantially  by not waiting 
6+ months before (if at all) releasing drivers/docs for 
linux! 

	Michael

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bernd Petrovitsch]

[ Minimized quoted part ]
On Sun, 2005-12-04 at 17:43 -0700, Jeff V. Merkey wrote:
> Bernd Petrovitsch wrote:
> >On Sat, 2005-12-03 at 17:52 -0700, Jeff V. Merkey wrote:
[...]
> >>of this code. I have apps written for Windows in 1990 and 1998 that 
> >                       ^^^^
> >>still run on Windows XP today. Linux has no such concept of
> >
> >But this not even holds for nearly all apps.
> >
> >>backwards compatiblity. Every company who has embraced it outside of 
> >
> >The same holds (probably) for Linux apps (given that your kernel can
> >start a.out). And AFAIBT by Win* driver developers even in the Win*
> >world you have to change your driver because of a new Win* version now
> >and then.
[...]
> No.  BIND was has been busted between 2.4 and 2.6.  Not to mention the

Hmmm, URL? Details? I can't remember anything about such issues.

> whole libc -> glib switchover.

glib has AFAIK next to nothing to do with a libc AFAICT (i.e. it is
using standard libc functions but that's all).

> It's hilarious that BSD had to create a Linux app compat lib, and the 
> RedHat shipped compat libs for 3 releases

Here you have your backwards compatibility.

> as well.   Not even close.  Windows has won.  M$ has won.  Linux lost 
> the desktop wars and will soon loose
> the server wars as well.   The reason - infighting and lack of backwards 

Yes, probably - MSFT is spreading the same story since ages.

> compatibility.  Binary only module
> breakage kernel to kernel will continue. 

As other told there never was a stable kernel module interface. Of
course there is probably enough willing manpower out there who will work
on that once you pay them. Or you can provide such support on your own.

Or do you (or anybody else) has drivers which should be maintained for
vanilla-kernel and/or vendor kernels and/or other kernels (to fix the
breakage in a cosntructive way), we can provide you with an offer to do
that.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Willy Tarreau]

Hi Greg,

I've been following most of this thread but did hot take the time to
jump in.

On Sun, Dec 04, 2005 at 03:24:54PM -0800, Greg KH wrote:
> On Sun, Dec 04, 2005 at 12:56:50PM +0100, Matthias Andree wrote:
> > The problem is the upstream breaking backwards compatibility for no good
> > reason. This can sometimes be a genuine unintended regression (aka.
> > bug), but quite often this is deliberate breakage because someone wants
> > to get rid of cruft. While the motivation is sound, breaking between
> > 2.6.N and 2.6.M must stop.
> 
> What are we breaking that people are complaining so much about?
> Specifics please.

Speaking for myself, I will not be complaining about specific things
which break, but I'll explain my point of view on the real problem.

The kernel has entered a permanent development phase, with some
versions more stable/usable than others. You and Chris do a
wonderful job at producing stable versions. I know how difficult
it is, for doing the same in 2.4 (and 2.4 moves slower than 2.6).

However, the problem is that you stop maintaining old versions
and quickly switch to new ones when a new kernel comes up. I know
it's not easy, and may be terribly more difficult to maintain
several versions in a development kernel than in a stable kernel.
But imagine the users' position : they run 2.6.14.3 which finally
fixes all their problems. Then they get a new problem, but 2.6.15
comes out. There will not be any other 2.6.14, so they have the
choice of staying to 2.6.14.3 or jumping to fresh new and barely
tested 2.6.15.

People have been surprized that I still maintain several old
versions of 2.4 at once, but I've received lots of "thank you"
emails from people who still relied on them for a particular
tree which does not evolve as fast as mainline. And 2.4 is
easier to follow than 2.6.

What I think should be done is to still maintain older 2.6
(eg: 2, 3 or 4 previous releases) so that people will have
the time to switch to a new one. And I think that what Adrian
wants to do would be useful *only* if he proceeds that way.
 
Maybe you should just join forces, eg Chris and you to catch
new patches, and Adrian to merge them to older kernels ? Every
software maker always supports a few older releases for the
people who need to stay on something stable, and it is clearly
what is missing now in 2.6.

Also, I think differently from Adrian. He wants to backport
all new drivers and new features, while I think that they are
the most sensible parts and the one which bring the more
changes to the kernel. In fact, we should *only* maintain
security and critical fixes on older releases. People in the
need of a new driver must upgrade for this. This is the
difference between staying on an old thing because you don't
need to upgrade and switching to a new one because you need
this new shiny feature. It follows the "if it ain't broke,
don't fix it" rule. Users will never excuse you for breaking
their working kernels by adding something they don't use.

I would have liked to help in this area (I even discussed
about maintaining a 2.6-stable a long time ago) but I don't
have enough time for this. 2.4 is already time-consuming.

Regards,
Willy

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Luke-Jr]

On Sunday 04 December 2005 23:22, Greg KH wrote:
> On Sun, Dec 04, 2005 at 04:46:31AM +0000, Luke-Jr wrote:
> > Well, devfs does have some abilities udev doesn't: hotplug/udev
> > doesn't detect everything, and can result in rarer or non-PnP devices
> > not being automatically available;
>
> Are you sure about that today?

Nope, but I don't see how udev can possibly detect something that doesn't let 
the OS know it's there-- except, of course, loading the driver for it and 
seeing if it works.

> And udev wasn't created to do everything that devfs does.

Which might be a case for leaving devfs in. *shrug*

> And devfs can't do everything that udev can (by far...)

Didn't say it could...

> > Interesting effects of switching my desktop from devfs to udev:
> > 1. my DVD burners are left uninitialized until I manually modprobe ide-cd
> > or (more recently) ide-scsi
>
> Sounds like a broken distro configuration :)

Well, I was assuming you kept Gentoo's udev packages up to date. ;)
[ebuild   R   ] sys-fs/udev-070-r1  (-selinux) -static 429 kB

> > devfs also has the advantage of keeping the module info all in one
> > place-- the kernel or the module.
> > In particular, with udev the detection and /dev info is scattered into
> > different locations of the filesystem. This can probably be fixed
> > easily simply by having udev read such info from modules or via a /sys
> > entry, though.
>
> What information are you talking about here?

I'm assuming everything in /etc/udev/rules.d/50-udev.rules used to be in the 
kernel for devfs-- perhaps it was PAM though, I'm not sure.
Other than that, I don't expect that simply installing a new kernel module 
will allow the device to be detected automatically, but that some hotplug or 
udev configurations will need to be updated also.
-- 
Luke-Jr
Developer, Utopios
http://utopios.org/

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

On Saturday 03 December 2005 11:53, Adrian Bunk wrote:
> On Sat, Dec 03, 2005 at 05:17:41PM +0000, David Ranson wrote:
> > Steven Rostedt wrote:
> > >udev ;)
> > >
> > >http://seclists.org/lists/linux-kernel/2005/Dec/0180.html
> >
> > Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> > userspace interface broken during the series, does anyone have any more?
>
> - support for ipfwadm and ipchains was removed during 2.6
> - devfs support was removed during 2.6
> - removal of kernel support for pcmcia-cs is pending
> - ip{,6}_queue removal is pending
> - removal of the RAW driver is pending

So what you're upset about is the feature removal scheduling mechanism, which 
usually gives a full year's warning, and the removal patch can be reversed 
into a feature addition patch you can maintain outside the tree if you really 
care?

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Rob Landley]

[-- Attachment #1: Type: text/plain, Size: 2559 bytes --]

On Saturday 03 December 2005 09:23, Adrian Bunk wrote:
> IOW, we should e.g. ensure that today's udev will still work flawlessly
> with kernel 2.6.30 (sic)?

A) udev changing its interface every three months isn't the kernel's fault, 
it's udev's.  Heck,  udev doesn't even accept a dependency on an external 
libsys but instead bundles its own copy in there because obviously the proper 
way to use a shared library is to block copy it into the source tree of every 
potential user.  Its config file format keeps changing.  What commands you 
run to invoke it keeps changing.  What does that have to do with the kernel?

Attached is a shell script that does the basics of what udev does.  (No, it 
doesn't handle permissions or persistent naming.  But it also doesn't show a 
lot of version dependencies, does it?)

B) When I install new packages I have to update dependencies like SDL or zlib 
all the time, yet you believe the kernel isn't allowed to have dependencies?  
Not even on things like modprobe or glibc that interface to the kernel not 
just graphically but "with tongue", as it were?  (Despite that, they're 
pretty darn good at staying compatible anyway.)

> This could work, but it should be officially announced that e.g. a
> userspace running kernel 2.6.15 must work flawlessly with _any_ future
> 2.6 kernel.

Oh don't start throwing around "must" and "officially announced" as if those 
terms actually mean something.

If you can come up with a compelling proposal and implement it and attract 
followers, fine.  You don't need to grab a flag and get blessed by somebody 
else to do anything.  (Whose flag and blessing did Linus get way back when?  
And where the heck did Ubuntu or Gentoo come from?)

The _right_ way to do this would have been to announce that you are 
maintaining a new tree, a -stable fork of whatever release, and give a URL to 
where it can be downloaded.  Announce code, not intentions.  (Announcing 
intentions never works.  Code attracts code and talk attracts talk.)  And that 
way the difficulty and sustainability of your task becomes self-apparent.

By the way, I'll guarantee you I can configure a 2.6.15 kernel that your 
userspace won't work with, with no code changes.  (To start with, I'd yank 
elf and force you to use a.out executable format...)

> For how many years do you think we will be able to ensure that this will
> stay true?

Who is "we", kemosabe?

> cu
> Adrian

Rob
-- 
Steve Ballmer: Innovation!  Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

[-- Attachment #2: setupdev.sh --]
[-- Type: application/x-shellscript, Size: 413 bytes --]

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Joel Becker]

On Sun, Dec 04, 2005 at 05:17:09PM +0100, Matthias Andree wrote:
> There are things that old Sun Workshop versions bitch about that GCC
> deals with without complaining, and I'm not talking about C99/C++-style
> comments. C standard issue? I believe not.

	I have seen many a code like so:

    char buf[4];
    memcpy(buf, source, 5);

accepted by the Sun compilers and run just fine.  When the application
was ported to Linux/GCC, the developers complained their program
segfaulted, and "it must be something broken on Linux!"
	Just because Sun's compiler does something doesn't mean it's
right :-)

Joel

-- 

Life's Little Instruction Book #510

	"Count your blessings."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Greg KH <greg@kroah.com> wrote:
> On Sun, Dec 04, 2005 at 04:24:36PM +0100, M. wrote:

[...]

> > yeah but I would mean if there's a 6months release cycle like GNOME & co.
> > there would be more opportunities in different distros using the same
> > kernel like those distros do with GNOME & co. If they use the same
> > 'current' 6months kernel available in the 18/24 time window this will
> > lead to unified base kernel for every distro and those distro could
> > mantain it for years

> The kernel is unlike GNOME in so many different ways, there's just no
> way to compare their development cycles.

Gnome is a /collection/ of (mostly independent) programs, after a while
what program+version survives a stress test is decreed to be part of
version N + 1 to be released at the 6-month point; lather, rinse,
repeat. In that sense it is much more like a distribution (which also have
similar release cycles). The kernel is /one/ program, and large and complex
to boot.

> People remember, the kernel evolves organically.  We don't know what's
> going to be in the next 2 kernel releases just because we don't know
> what's going to show up, and what hardware is going to be released, and
> what kind of problems people are going to have, and what kind of
> proposed patches are going to work out.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Jeff V. Merkey]

Bernd Petrovitsch wrote:

>On Sat, 2005-12-03 at 17:52 -0700, Jeff V. Merkey wrote:
>[...]
>  
>
>>of this code. I have apps written for Windows in 1990 and 1998 that 
>>    
>>
>                       ^^^^
>  
>
>>still run on Windows XP today. Linux has no such concept of
>>    
>>
>
>But this not even holds for nearly all apps.
>
>  
>
>>backwards compatiblity. Every company who has embraced it outside of 
>>    
>>
>
>The same holds (probably) for Linux apps (given that your kernel can
>start a.out). And AFAIBT by Win* driver developers even in the Win*
>world you have to change your driver because of a new Win* version now
>and then.
>
>	Bernd
>  
>
No.  BIND was has been busted between 2.4 and 2.6.  Not to mention the 
whole libc -> glib switchover.
It's hilarious that BSD had to create a Linux app compat lib, and the 
RedHat shipped compat libs for 3 releases
as well.   Not even close.  Windows has won.  M$ has won.  Linux lost 
the desktop wars and will soon loose
the server wars as well.   The reason - infighting and lack of backwards 
compatibility.  Binary only module
breakage kernel to kernel will continue. 

Jeff

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sun, Dec 04, 2005 at 12:56:50PM +0100, Matthias Andree wrote:
> The problem is the upstream breaking backwards compatibility for no good
> reason. This can sometimes be a genuine unintended regression (aka.
> bug), but quite often this is deliberate breakage because someone wants
> to get rid of cruft. While the motivation is sound, breaking between
> 2.6.N and 2.6.M must stop.

What are we breaking that people are complaining so much about?
Specifics please.

And if you bring up udev, please see my previous comments in this thread
about that issue.

It isn't userspace stuff that is breaking, as applications built on 2.2
still work just fine here on 2.6 for me.

Yes we break in-kernel apis, all the time, that's fine.  See
Documentation/stable-api-nonsense.txt for details about why we do that.

So again, specifics please?

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sun, Dec 04, 2005 at 04:46:31AM +0000, Luke-Jr wrote:
> Well, devfs does have some abilities udev doesn't: hotplug/udev
> doesn't detect everything, and can result in rarer or non-PnP devices
> not being automatically available;

Are you sure about that today?  And udev wasn't created to do everything
that devfs does.  And devfs can't do everything that udev can (by
far...)

> devfs has the effect of trying to load a module when a program looks
> for the devices it provides-- while it can cause problems, it does
> have a possibility to work better.

Sorry, but that model of loading modules is very broken and it is good
that we don't do that anymore (as you allude to.)

> Interesting effects of switching my desktop from devfs to udev:
> 1. my DVD burners are left uninitialized until I manually modprobe ide-cd or 
> (more recently) ide-scsi

Sounds like a broken distro configuration :)

> 2. my sound card is autodetected and the drivers loaded, but the OSS emulation 
> modules are omitted; with devfs, they would be autoloaded when an app tried 
> to use OSS

Again, broken distro configuration :)

> devfs also has the advantage of keeping the module info all in one place-- the 
> kernel or the module.

What?

> In particular, with udev the detection and /dev info is scattered into
> different locations of the filesystem. This can probably be fixed
> easily simply by having udev read such info from modules or via a /sys
> entry, though.

What information are you talking about here?

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Mon, Dec 05, 2005 at 12:03:20AM +0100, M. wrote:
> > The way the kernel is developed is _so_ different from any traditional
> > software development process is defined.  So for people to try to put
> > traditional requirements on the kernel (6 month cycles, etc.) is just
> > not realistic.
> 
> Mhhh BSDs and MacOSX kernel are developed without taking "the unknown" into
> account: planned releases and bla blah and they dont miss features. Yeah
> linux is faster, but there could be a middle point between strict release
> cycles and "ok let's put it in cause it's going to make something run
> faster"

MacOSX is developed this way?  I think you will find a lot of Apple
engineers disagree with you...

And BSD is also quite different than Linux in many different ways, the
development community being one of these differences.  And that one
difference makes a lot of difference.

> > And please for everyone wanting to go with a stable series like is being
> > proposed, go read the thread a while ago on this list that caused the
> > creation of the -stable tree.  In it lots of people who know what they
> > are talking about discuss the difficulties of doing a "bug fix only"
> > tree, and other such things.  Out of that discussion came the very
> > restrictive guidelines that are described in
> > Documentation/stable_kernel_rules.txt.  To try to do more than what is
> > defined there, without lots of money and man-power behind you, is a
> > quick trip to madness...
> 
> 
> The proposal was in fact to come out with a 2.6.X release every 6 months
> trying to align every distro on it and to "get" their man-power and money as
> a side effect. Maybe i'm not sufficiently good with english to let you all
> understand clearly but the proposal was to do 2/3/4 releases the
> normal/current style, even adding new and previously unknown
> features/patches/whatever and to do the last release before the next
> 2.6.Xwith only bugfix and stabilization in mind; If you think over it
> it's the
> same approach of every release:
> 
> - patches get in until -rc;
> - 2 weeks bugfix only;
> - release;
> 
> but applied to a 6months timeline.

That will not work.  Again, go back and read that thread.  We seeing
this shorter development cycle get backed up even when it streaches to 2
months.  For it to go to 6 months would be madness.

If you don't understand why I say this, please go look at the different
developer trees that start accumilating patches during this "bugfix
only" timeframe.

> It's not a -stable/strict/specifics-based tree but a way to align
> everyone to the same kernel version and to get stabilization and
> maintenance as a side effect.

And you believe that the enterprise distros will some how flock to this
kernel release instead?  Why would they?  What would it gain them?

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sun, Dec 04, 2005 at 04:24:36PM +0100, M. wrote:
> On 12/4/05, Arjan van de Ven <arjan@infradead.org> wrote:
> > On Sun, 2005-12-04 at 15:57 +0100, M. wrote:
> > > if distros would align on those 6months versions those less
> > > experienced users would get 5 years support on those kernels.
> >
> > no distro gives 5 years of support for a kernel done every 6 months;
> > they start such projects more like every 18 to 24 months (SuSE used to
> > do it a bit more frequently but it seems they also slowed this down).
> 
> 
> yeah but I would mean if there's a 6months release cycle like GNOME & co.
> there would be more opportunities in different distros using the same kernel
> like those distros do with GNOME & co. If they use the same 'current'
> 6months kernel available in the 18/24 time window this will lead to unified
> base kernel for every distro and those distro could mantain it for years

The kernel is unlike GNOME in so many different ways, there's just no
way to compare their development cycles.

People remember, the kernel evolves organically.  We don't know what's
going to be in the next 2 kernel releases just because we don't know
what's going to show up, and what hardware is going to be released, and
what kind of problems people are going to have, and what kind of
proposed patches are going to work out.

The way the kernel is developed is _so_ different from any traditional
software development process is defined.  So for people to try to put
traditional requirements on the kernel (6 month cycles, etc.) is just
not realistic.

And please for everyone wanting to go with a stable series like is being
proposed, go read the thread a while ago on this list that caused the
creation of the -stable tree.  In it lots of people who know what they
are talking about discuss the difficulties of doing a "bug fix only"
tree, and other such things.  Out of that discussion came the very
restrictive guidelines that are described in
Documentation/stable_kernel_rules.txt.  To try to do more than what is
defined there, without lots of money and man-power behind you, is a
quick trip to madness...

Best of luck to you all,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sun, Dec 04, 2005 at 06:00:49PM +0100, Jakob Oestergaard wrote:
> 
> If the kernel was stable (reliability wise - as in "not crashing") then
> you'd be perfectly right.

But isn't it? :)

> In the real world, however, admins currently need to pick out specific
> versions of the kernel for specific workloads (try running a large
> fileserver on anything but 2.6.11.11 for example - any earlier or later
> kernel will barf reliably.

Have you filed a but at bugzilla.kernel.org about this?  If not, how do
you expect it to get fixed?

> For web serving it's another kernel that's golden, I forgot which).

That sounds very strange, the same kernel version should work just as
well for all workloads.  If not, it's a bug and should be fixed.

> There are very very good reasons for offering a 'stable series' in plain
> source-tree form - lots of admins of real-world systems need this.

But it sounds like you will want different stable series depending on
what kind of server you are running.  And that will be even more work...

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Bernd Petrovitsch]

On Sat, 2005-12-03 at 17:52 -0700, Jeff V. Merkey wrote:
[...]
> of this code. I have apps written for Windows in 1990 and 1998 that 
                       ^^^^
> still run on Windows XP today. Linux has no such concept of

But this not even holds for nearly all apps.

> backwards compatiblity. Every company who has embraced it outside of 

The same holds (probably) for Linux apps (given that your kernel can
start a.out). And AFAIBT by Win* driver developers even in the Win*
world you have to change your driver because of a new Win* version now
and then.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Paul Jackson]

Arjan, responding to Matthias
> > SUSE end-user distros (SUSE LINUX <version>) are released every 6 months
> > or so, and are supported for 24 months. Their "enterprise server" is
> > supported for 60 months though, SLES 9 forked off 9.1.
> 
> sure.. but they don't add new hw support really, and I'd not be
> surprised if they rebase to a newer upstream kernel after a while.

They will start a new enterprise release, off a new base, every so
often, and call the next one SLES 10.  But SLES 9 will continue to be
supported, off its current kernel.org base, for an extended period of
time.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Jeff V. Merkey <jmerkey@wolfmountaingroup.com> wrote:
> Matthias Andree wrote:
> >On Sat, 03 Dec 2005, Arjan van de Ven wrote:

[...]

> These folks have nothing new to innovate here. The memory manager and
> VM gets revamped every other release.

That is a sign of movement. Sure, one can argue if it is random movement of
definite progress, but change /is/ a part of innovation.

>                                       Exports get broken, binary only
> module compatibility busted every rev of the kernel.

In-kernel API was /never/ guaranteed, so you have nothing to complain here.

>                                                      I spend weeks on
> each kernel fixing the breakage. These people don't get it,

They do get it, and have their reasons to act this way. Reasons you don't
get, it seems.

>                                                             don't care,

They do care. But not about the same thing you care about.

> and to be honest, you are wasting your time here trying to convince
> them.

It is /their/ code, with which they are free to do as they wish. If you
contribute enough, you get a say in it too; until that time, just be
grateful for what you get given freely.

>       It's never stable because they don't want it to be. This is how
> they maintain control of this code.

Licensing under GPL while "maintaining control" is a contradiction in
terms. If you don't like where the development is going, you are free to
fork it. Just the developers won't follow you, as the consensus between
them is against your wishes.

>                                     I have apps written for Windows in
> 1990 and 1998 that still run on Windows XP today.

I have programs written in the early 80s that still can be compiled and run
on Linux. And AFAIK the very first a.out binaries for Linux still work (I'd
assume that setting up the shared libraries for them is a royal pain).

I have programs by /Microsoft/ which don't run after Win98, even though
they are supposed to run on later versions. 

Besides, this backward compatibility is exactly one of the major reasons
for Windows breakage, they just can't get a clean cut from the past.

>                                                   Linux has no such
> concept of backwards compatiblity.

For user programs? Sure is.

>                                    Every company who has embraced it
> outside of hardware based solutions is dying or has died.

So?

>                                                           IBM is secretly
> forking it as we speak and using it to get out of paying for Unix
> licenses.

Could you please give some kind of evidence for this misbehaviour by IBM?

> As annoying as it is, accept it and live with it. These people have no
> sense of loyalty to you or your customers. They don't even care about
> each other.

Right. Their only loyalty is to a better kernel (system). Nothing
else. Nobody /ever/ promised anything else, in fact, they have /always/
stated that that is their only objective. You definitely haven't gotten
shafted.

[...]

> You are standing on a battlefield. Quietly fork each release, make your
> mods, post patches somewhere for the poor civilians who are "collateral
> damage" in the war with constantly busted software, and you might help
> some folks.

This is what distributions do routinely... and the current development
model is precisely to /disminish/ the cost of keeping up to date by third
parties. In that sense the kernel community /does/ care for them, just that
they don't make any promises. Take it or leave it. There are alternatives,
like the various BSD flavors. Or even Solaris, or closed Unix variants.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Matthias Andree <matthias.andree@gmx.de> wrote:
> On Sat, 03 Dec 2005, Horst von Brand wrote:
> > Matthias Andree <matthias.andree@gmx.de> wrote:
> > > On Sat, 03 Dec 2005, David Ranson wrote:
> > > > Adrian Bunk wrote:
> > > > 
> > > > >- support for ipfwadm and ipchains was removed during 2.6
> > 
> > > > Surely this one had loads of notice though? I was using iptables with
> > > > 2.4 kernels.

> > Sure had. They were scheduled for removal in march, 2005 a long time ago.
> > 
> > > So was I. And now what? ipfwadm and ipchains should have been removed
> > > from 2.6.0 if 2.6.0 was not to support these.
> > 
> > Or in 2.6.10, or 2.6.27, or whatever.

> No. If you need to remove major components, it is only diligent to bump
> the minor revision and call the beast 2.7.0.

In the case of ipfwadm and ipchains, that was precisely the changeover to
2.6... ipfwadm is from 2.2, obsoleted by ipchains in 2.4, and both axed in
the timespan promised.

devfs was never widely used, and was also announced to be killed by 2.6,
IIRC.

>                                              At that time, not only one
> or two subsystems, but all that were marked deprecated for 6 months or
> so, should be dropped.

That /is/ what is happening, modulo needless changes in version number.

> > > This doesn't matter. A kernel that calls itself stable CAN NOT remove
> > > features unless they had been critically broken from the beginning. And
> > > this level of breakage is a moot point, so removal is not justified.
> > 
> > devfs was broken, and very little used.
> 
> OK. This however doesn't hold for ipfwadm (which should probably never
> have made it into 2.6.0 in the first place) or ipchains.

They were carried along exactly to keep the promise of killing them off by a
certain date.

It seems you are going by what is called around here "Palos porque bogas,
palos porque no bogas" (roughly translated, "get whipped for rowing, and
get whipped for not rowing").

> > > Linux 2.6 is not "stable" in this regard.

> > Right. The idea of "stable series" had to go. And went.

> So what is the point in using Linux anyhow if the kernel developers
> don't care for the outside world, one might ask?

Sorry, but again:

- Userland API is remarkably stable, only some stuff with intimate kernel
  connection have ever changed
- In-kernel API has /never/ been stable. At all. Sure, changes are much
  faster now (the community is larger, the tools are better, ....).

>                                                  What is in the way of
> reflecting feature removals in the minor version of the project, say,
> remove devfs, ipfwadm, ipchains and whatnot in one go and call the new
> release without this legacies 2.7.0?

It /was/ called 2.6...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Jakob Oestergaard]

On Sat, Dec 03, 2005 at 12:19:45PM -0800, Greg KH wrote:
> On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> > 
> > Why can't this be done by distributors/vendors?
> 
> It already is done by these people, look at the "enterprise" Linux
> distributions and their 5 years of maintance (or whatever the number
> is.)
> 
> If people/customers want stability, they already have this option.

If the kernel was stable (reliability wise - as in "not crashing") then
you'd be perfectly right.

In the real world, however, admins currently need to pick out specific
versions of the kernel for specific workloads (try running a large
fileserver on anything but 2.6.11.11 for example - any earlier or later
kernel will barf reliably. For web serving it's another kernel that's
golden, I forgot which).

There are very very good reasons for offering a 'stable series' in plain
source-tree form - lots of admins of real-world systems need this.

Adrian, I like the idea :)

-- 

 / jakob

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

On Sun, 2005-12-04 at 17:11 +0100, Matthias Andree wrote:
> On Sun, 04 Dec 2005, Arjan van de Ven wrote:
> 
> > On Sun, 2005-12-04 at 15:57 +0100, M. wrote:
> > 
> > > 
> > > if distros would align on those 6months versions those less
> > > experienced users would get 5 years support on those kernels. 
> > 
> > no distro gives 5 years of support for a kernel done every 6 months;
> > they start such projects more like every 18 to 24 months (SuSE used to
> > do it a bit more frequently but it seems they also slowed this down).
> 
> SUSE end-user distros (SUSE LINUX <version>) are released every 6 months
> or so, and are supported for 24 months. Their "enterprise server" is
> supported for 60 months though, SLES 9 forked off 9.1.

sure.. but they don't add new hw support really, and I'd not be
surprised if they rebase to a newer upstream kernel after a while. I
know we did that for RHL, eg RHL 7.(Y-1) got the kernel of RHL7.Y after
RHL7.Y was released, not only to keep the maintenance down, but more so
to get all the bugfixes and hardware support out to customers. 

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sun, 04 Dec 2005, Arjan van de Ven wrote:

> >  (C) Copyright notice and "All rights reserved."
> > 
> > > > These use inter_module_get() 
> > > 
> > > which is still there even in the latest 2.6.15-rc. It should be going
> > > out but hasn't yet. And that is the case for at least a year (eg they
> > > are __deprecated but still there).
> > 
> > No, they aren't - at least not anywhere declared below include/ and thus
> > uncompilable with GCC4.
> 
> # pwd
> /mnt/raid/linux/linux-2.6.15-rc4/include/linux
> [root@jackhammer linux]# grep inter_mod *
> module.h:extern void __deprecated inter_module_register(const char *,
> module.h:extern void __deprecated inter_module_unregister(const char *);
> module.h:extern const void * __deprecated inter_module_get_request(const
> char *,
> module.h:extern void __deprecated inter_module_put(const char *);

Same story with -rc5. As you can see, there is no declaration for
inter_module_get(), just the _request() variant.  So what now?  :-P

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sun, 04 Dec 2005, Andreas Schwab wrote:

> Matthias Andree <matthias.andree@gmx.de> writes:
> 
> > Yes. "extern type foo; static type foo;" is way stupid, but 10% of the
> > blame can be shifted on the GCC guys for being much too tolerant.
> 
> You should rather blame the C standard.

There are things that old Sun Workshop versions bitch about that GCC
deals with without complaining, and I'm not talking about C99/C++-style
comments. C standard issue? I believe not.

Anyways, this is getting off-topic and ultimately the author of broken
code is responsible, of course. But it's still nice if the tools help
produce good code.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sun, 04 Dec 2005, Arjan van de Ven wrote:

> On Sun, 2005-12-04 at 15:57 +0100, M. wrote:
> 
> > 
> > if distros would align on those 6months versions those less
> > experienced users would get 5 years support on those kernels. 
> 
> no distro gives 5 years of support for a kernel done every 6 months;
> they start such projects more like every 18 to 24 months (SuSE used to
> do it a bit more frequently but it seems they also slowed this down).

SUSE end-user distros (SUSE LINUX <version>) are released every 6 months
or so, and are supported for 24 months. Their "enterprise server" is
supported for 60 months though, SLES 9 forked off 9.1.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Andreas Schwab]

Matthias Andree <matthias.andree@gmx.de> writes:

> Yes. "extern type foo; static type foo;" is way stupid, but 10% of the
> blame can be shifted on the GCC guys for being much too tolerant.

You should rather blame the C standard.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Indrek Kruusa]

Arjan van de Ven wrote:

>>nt model (or at least they have no resources to 
>>do testing [Torvalds])
>>c) end-users (or those who are not kernel maintainers) are directed 
>>permanently to distros kernels and "stay away from kernel.org you 
>>wanna-bees!
>>    
>>
>
>this is not what is being said. What is being said is that if you can't
>deal with occasional breakage, you're better off using vendor kernels.
>But.. if you can't deal with occasional breakage, you wouldn't test test
>kernels EITHER. If you can deal with an occasional breakage, I hope you
>and everyone else who can, will run and test kernel.org kernels,
>especially the -rc ones. 
>
>Most of the "instability" people complain about with the new 2.6 model
>is caused by people not testing the -rc kernels before they are
>released, so that they end up being released with regressions.
>

I think I have seen special live-cd distribution for KDE beta testers. 
Kernel is not a KDE but such a very broken distribution with -rc kernel 
could be more easily maintained than "udev forever!". Live-cd (or 
live-usb) wouldn't be too flexible - you can't say there "can you give a 
whirl to this patch, please" but I bet you will have more testers.

thanks,
Indrek

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Richard Knutsson]

Matthias Andree wrote:

>As I say, these aren't licensed for inclusion into the kernel, they bear
>a (C) Copyright notice and "All rights reserved."
>  
>
In the 2.6.15-rc5 kernel we find:
data@amazon linux-2.6.15-rc5]$ find . -name *.[chS] | xargs grep -n "All 
rights reserved" | wc -l
932
[data@amazon linux-2.6.15-rc5]$ find . -name *.[chS] | xargs grep -n 
"Copyright" | wc -l
15083

But I do wonder how copyright and GPL can co-exist. Do the copyright 
holder own the changes anybody else does to the code?
Anyone care to explain?

Thanks
Richard Knutsson

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> But I do wonder how copyright and GPL can co-exist. Do the copyright 
> holder own the changes anybody else does to the code?
> Anyone care to explain?

The GPL *is* copyright. You and I as copyright holders reserve all
rights, and then grant selected rights; the rights and the conditions
they are granted under are described in the COPYING file. It's a
misunderstanding to think that GPL means "no copyright" or "can do
anything I want"; in a way the GPL is quite a restrictive license.
(while it allows you to distribute, copy and make derived works, it only
does allow that under the condition that the result is made available
under the GPL as well in full source form, and there's some additional
conditions as well)

so GPL can copyright are not in conflict, the GPL can exist BECAUSE of
copyright actually.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

>  (C) Copyright notice and "All rights reserved."
> 
> > > These use inter_module_get() 
> > 
> > which is still there even in the latest 2.6.15-rc. It should be going
> > out but hasn't yet. And that is the case for at least a year (eg they
> > are __deprecated but still there).
> 
> No, they aren't - at least not anywhere declared below include/ and thus
> uncompilable with GCC4.

# pwd
/mnt/raid/linux/linux-2.6.15-rc4/include/linux
[root@jackhammer linux]# grep inter_mod *
module.h:extern void __deprecated inter_module_register(const char *,
module.h:extern void __deprecated inter_module_unregister(const char *);
module.h:extern const void * __deprecated inter_module_get_request(const
char *,
module.h:extern void __deprecated inter_module_put(const char *);


sounds you need to invoke the warranty on your grep binary ;)

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> Perhaps the dates give a clue. Since when has Linux had IPMI in the
> baseline code?

2.4.21 already had it, so it's been quite a while

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

On Sun, 2005-12-04 at 15:57 +0100, M. wrote:

> 
> if distros would align on those 6months versions those less
> experienced users would get 5 years support on those kernels. 

no distro gives 5 years of support for a kernel done every 6 months;
they start such projects more like every 18 to 24 months (SuSE used to
do it a bit more frequently but it seems they also slowed this down).

> example: redhat, suse and mandriva are releasing their new product
> using the latest 6months (or whatever) kernel; they are not going to
> patch it except for new filesystems or bugfixes because of the new dev

"except for" is a slipperly slope. And "except for bugfixes" would be
wrong... those would be the ones that need to be in the kernel.org
kernel. As well as new hardware support. At which point.. what is the
difference? Where do 'features' stop and where do 'only needed bugfixes'
begin?

>  model granting them all the needed new features; then, they start to
> mantain this kernel for their customers (and they could do it in a
> collaborative way, thus mantaining the kernel.org kernel plus their
> separate patches) and every user of redhat, suse, mandriva and the
> kernel.org 6months kernel they are using would benefit from this and
> would get 5 years support on this kernel.

that's not practical though. And it's still no better from the
regression point of view; those enterprise kernels undergo quite a bit
of churn as well, but just very directed churn to the point that I doubt
it would satisfy your target audience....

> 

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sun, 04 Dec 2005, Arjan van de Ven wrote:

> > As I say, these aren't licensed for inclusion into the kernel, they bear
> > a (C) Copyright notice and "All rights reserved."
> 
> and
> MODULE_LICENSE("GPL");
> 
> so it *IS* gpl licensed!
> 
> the code is a bit horrible though and no surprise it breaks ;)

Yes. "extern type foo; static type foo;" is way stupid, but 10% of the
blame can be shifted on the GCC guys for being much too tolerant.

> you can always make drivers broken enough to break at the slightest
> change ;)
> 
> (it also seems to contain an entire ipmi layer, linux already has one so
> I wonder why they're not just using that as basis)

Perhaps the dates give a clue. Since when has Linux had IPMI in the
baseline code?

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Michael Frank]

On Sunday 04 December 2005 05:46, Luke-Jr wrote:
> On Sunday 04 December 2005 00:20, Greg KH wrote:
> > > Switch "broken bloaty bulky devfs" to "lean & clean
> > > devfs"?  This ship would have been flying the
> > > "clean-up nation" flags.
> >
> > Again, because an in-kernel devfs is not the correct
> > thing to do.  devfs has been disabled for a few months
> > now, and I don't think anyone has missed it yet :)
>
> Well, devfs does have some abilities udev doesn't:
> hotplug/udev doesn't detect everything, and can result in
> rarer or non-PnP devices not being automatically
> available; devfs has the effect of trying to load a
> module when a program looks for the devices it provides--
> while it can cause problems, it does have a possibility
> to work better. Interesting effects of switching my
> desktop from devfs to udev:
> 1. my DVD burners are left uninitialized until I manually
> modprobe ide-cd or (more recently) ide-scsi
> 2. my sound card is autodetected and the drivers loaded,
> but the OSS emulation modules are omitted; with devfs,
> they would be autoloaded when an app tried to use OSS
> devfs also has the advantage of keeping the module info
> all in one place-- the kernel or the module. In
> particular, with udev the detection and /dev info is
> scattered into different locations of the filesystem.
> This can probably be fixed easily simply by having udev
> read such info from modules or via a /sys entry, though.

Seems your complaints are related to missing/broken 
configuration (of your distribution). 

Here are some references which may be of help:

http://webpages.charter.net/decibelshelp/LinuxHelp_UDEVPrimer.html
http://www.gentoo.org/doc/en/udev-guide.xml
http://gentoo-wiki.com/HOWTO_Customizing_UDEV
http://ubuntuforums.org/archive/index.php/t-11718.html

IME, udev is easier to manage than devfs because there are 
tools such as udevtest.

Please see also manuals for udevstart udevtest, udevinfo, 
udevcontrol

Also strace udevstart is interesting...

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

On Sun, 2005-12-04 at 15:25 +0100, Matthias Andree wrote:
> On Sun, 04 Dec 2005, Arjan van de Ven wrote:
> 
> > On Sun, 2005-12-04 at 14:28 +0100, Matthias Andree wrote:
> > 
> > > I meant the ipmi, smbus and copa modules by Fujitsu-Siemens.
> > > 
> > > They are provided in source form, but I just found out (reading the
> > > headers and not just the lines that broke the compile) they are not open
> > > source. Perhaps one should prod them to slap a modified-BSD or perhaps
> > > GPL label onto their modules.
> > 
> > is there an URL to these?
> 
> http://download.fujitsu-siemens.com/prim_supportcd/Programs/General/ServView/Linux/agents/srvmagt-mods_src.suse.rpm
> 
> > > It seems you'd then maintain them after their submission? :-)
> > 
> > usually such modules are extremely low maintenance once merged.... There
> > are many many drivers without a maintainer, and they still get fixed.
> 
> As I say, these aren't licensed for inclusion into the kernel, they bear
> a (C) Copyright notice and "All rights reserved."

and
MODULE_LICENSE("GPL");

so it *IS* gpl licensed!

the code is a bit horrible though and no surprise it breaks ;)

you can always make drivers broken enough to break at the slightest
change ;)

(it also seems to contain an entire ipmi layer, linux already has one so
I wonder why they're not just using that as basis)

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sun, 04 Dec 2005, Arjan van de Ven wrote:

> On Sun, 2005-12-04 at 14:28 +0100, Matthias Andree wrote:
> 
> > I meant the ipmi, smbus and copa modules by Fujitsu-Siemens.
> > 
> > They are provided in source form, but I just found out (reading the
> > headers and not just the lines that broke the compile) they are not open
> > source. Perhaps one should prod them to slap a modified-BSD or perhaps
> > GPL label onto their modules.
> 
> is there an URL to these?

http://download.fujitsu-siemens.com/prim_supportcd/Programs/General/ServView/Linux/agents/srvmagt-mods_src.suse.rpm

> > It seems you'd then maintain them after their submission? :-)
> 
> usually such modules are extremely low maintenance once merged.... There
> are many many drivers without a maintainer, and they still get fixed.

As I say, these aren't licensed for inclusion into the kernel, they bear
a (C) Copyright notice and "All rights reserved."

> > These use inter_module_get() 
> 
> which is still there even in the latest 2.6.15-rc. It should be going
> out but hasn't yet. And that is the case for at least a year (eg they
> are __deprecated but still there).

No, they aren't - at least not anywhere declared below include/ and thus
uncompilable with GCC4.

> Most of those were already in the final place with a temporary compat
> header in the old one I guess. But if this is all.... that really isn't
> a big deal. I suspect some of these headers aren't even used by the
> driver (sometimes people just include the world for no reason)..

The reason would be convenience, or maintainer efficiency as the Camel
book ("Programming Perl") words it.

If not including the right header file (such as ioctl32.h on x86_64)
breaks the compile, I presume they are needed.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Denis Vlasenko]

On Saturday 03 December 2005 17:39, Arjan van de Ven wrote:
> > IOW, we should e.g. ensure that today's udev will still work flawlessly 
> > with kernel 2.6.30 (sic)?
> 
> I'd say yes. It doesn't need to support all new functionality, but at
> least what it does today it should be able to do then. If that really
> isn't possible maybe udev should be part of the kernel build and per
> kernel version.

I agree with this idea.
--
vda

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

On Sun, 2005-12-04 at 14:28 +0100, Matthias Andree wrote:

> I meant the ipmi, smbus and copa modules by Fujitsu-Siemens.
> 
> They are provided in source form, but I just found out (reading the
> headers and not just the lines that broke the compile) they are not open
> source. Perhaps one should prod them to slap a modified-BSD or perhaps
> GPL label onto their modules.

is there an URL to these?

> 
> It seems you'd then maintain them after their submission? :-)

usually such modules are extremely low maintenance once merged.... There
are many many drivers without a maintainer, and they still get fixed.


> > It's rare even in the 2.6 tree to mass-break well written drivers. Just
> > because it's a lot of work to fix all in kernel drivers up. But a fully
> > stable API is also not good. My guess is that the drivers that break
> > most are the ones using the not-right APIs (eg internals and such). 
> 
> These use inter_module_get() 

which is still there even in the latest 2.6.15-rc. It should be going
out but hasn't yet. And that is the case for at least a year (eg they
are __deprecated but still there).

>  and some #include headers that have moved around between
> linux and asm directories.

Most of those were already in the final place with a temporary compat
header in the old one I guess. But if this is all.... that really isn't
a big deal. I suspect some of these headers aren't even used by the
driver (sometimes people just include the world for no reason)..

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sun, 04 Dec 2005, Arjan van de Ven wrote:

> On Sun, 2005-12-04 at 13:12 +0100, Matthias Andree wrote:
> > On Sat, 03 Dec 2005, Jeff V. Merkey wrote:
> > 
> > > These folks have nothing new to innovate here. The memory manager and VM 
> > > gets revamped every other release. Exports get broken, binary only 
> > > module compatibility busted every rev of the kernel. I spend weeks on 
> > 
> > Who cares for binary modules?
> > 
> > It hurts however if external OSS modules are broken.
> 
> then those modules should be submitted realistically. That's just best
> for everyone involved. Which modules in particular do you mean btw?

I meant the ipmi, smbus and copa modules by Fujitsu-Siemens.

They are provided in source form, but I just found out (reading the
headers and not just the lines that broke the compile) they are not open
source. Perhaps one should prod them to slap a modified-BSD or perhaps
GPL label onto their modules.

It seems you'd then maintain them after their submission? :-)

> It's rare even in the 2.6 tree to mass-break well written drivers. Just
> because it's a lot of work to fix all in kernel drivers up. But a fully
> stable API is also not good. My guess is that the drivers that break
> most are the ones using the not-right APIs (eg internals and such). 

These use inter_module_get() (ok, inter_module_get_request isn't
difficult) and some #include headers that have moved around between
linux and asm directories.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> nt model (or at least they have no resources to 
> do testing [Torvalds])
> c) end-users (or those who are not kernel maintainers) are directed 
> permanently to distros kernels and "stay away from kernel.org you 
> wanna-bees!

this is not what is being said. What is being said is that if you can't
deal with occasional breakage, you're better off using vendor kernels.
But.. if you can't deal with occasional breakage, you wouldn't test test
kernels EITHER. If you can deal with an occasional breakage, I hope you
and everyone else who can, will run and test kernel.org kernels,
especially the -rc ones. 

Most of the "instability" people complain about with the new 2.6 model
is caused by people not testing the -rc kernels before they are
released, so that they end up being released with regressions. But...
that will happen no matter what model you use actually. Before july
there was a problem where -rc kernels kept changing bigtime, so it was
hard to know which one to test if you only had time to test one, but
that is now changed... 

Maybe it's a bit extremist, but I'd like to even state it like this:
"If you can't be bothered to test -rc kernels, you have no right to
complain about the final ones", sort of as analogy to the "if you don't
go voting you have no right to complain about the politicians".

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Indrek Kruusa]

>These problems follow from the development model.
>  
>

Hi!
I am not evil or unsatisfied end-user by any means, it is just quite 
interesting to follow today's lkml posts.

After reading "Linux 2.6.15-rc5: off-line for a week" from Torvalds it 
seems like this:.

a) Torvalds thinks that nobody cares about kernel testing
b) other gurus (they are also only "on-line" testers nowadays) doesn't 
feel good with development model (or at least they have no resources to 
do testing [Torvalds])
c) end-users (or those who are not kernel maintainers) are directed 
permanently to distros kernels and "stay away from kernel.org you 
wanna-bees!"

I can't say is it good or bad but this is interesting at least. I hope 
you guys keep going!

Have  a nice day!
Indrek

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

On Sun, 2005-12-04 at 13:12 +0100, Matthias Andree wrote:
> On Sat, 03 Dec 2005, Jeff V. Merkey wrote:
> 
> > These folks have nothing new to innovate here. The memory manager and VM 
> > gets revamped every other release. Exports get broken, binary only 
> > module compatibility busted every rev of the kernel. I spend weeks on 
> 
> Who cares for binary modules?
> 
> It hurts however if external OSS modules are broken.

then those modules should be submitted realistically. That's just best
for everyone involved. Which modules in particular do you mean btw?

It's rare even in the 2.6 tree to mass-break well written drivers. Just
because it's a lot of work to fix all in kernel drivers up. But a fully
stable API is also not good. My guess is that the drivers that break
most are the ones using the not-right APIs (eg internals and such). 

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, Jeff V. Merkey wrote:

> These folks have nothing new to innovate here. The memory manager and VM 
> gets revamped every other release. Exports get broken, binary only 
> module compatibility busted every rev of the kernel. I spend weeks on 

Who cares for binary modules?

It hurts however if external OSS modules are broken.

> of this code. I have apps written for Windows in 1990 and 1998 that 
> still run on Windows XP today.

Sure, you're loading Windows 3.1 drivers into XP...  You can tell us
more of that crap later, but not here.

Properly written 1995 software usually still works on Linux as long as
it doesn't need to care about kernel or devices.

[rest of Merkey rantings removed]

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, Horst von Brand wrote:

> Matthias Andree <matthias.andree@gmx.de> wrote:
> > On Sat, 03 Dec 2005, David Ranson wrote:
> > > Adrian Bunk wrote:
> > > 
> > > >- support for ipfwadm and ipchains was removed during 2.6
> 
> > > Surely this one had loads of notice though? I was using iptables with
> > > 2.4 kernels.
> 
> Sure had. They were scheduled for removal in march, 2005 a long time ago.
> 
> > So was I. And now what? ipfwadm and ipchains should have been removed
> > from 2.6.0 if 2.6.0 was not to support these.
> 
> Or in 2.6.10, or 2.6.27, or whatever.

No. If you need to remove major components, it is only diligent to bump
the minor revision and call the beast 2.7.0. At that time, not only one
or two subsystems, but all that were marked deprecated for 6 months or
so, should be dropped.

> > This doesn't matter. A kernel that calls itself stable CAN NOT remove
> > features unless they had been critically broken from the beginning. And
> > this level of breakage is a moot point, so removal is not justified.
> 
> devfs was broken, and very little used.

OK. This however doesn't hold for ipfwadm (which should probably never
have made it into 2.6.0 in the first place) or ipchains.

> > Linux 2.6 is not "stable" in this regard.
> 
> Right. The idea of "stable series" had to go. And went.

So what is the point in using Linux anyhow if the kernel developers
don't care for the outside world, one might ask? What is in the way of
reflecting feature removals in the minor version of the project, say,
remove devfs, ipfwadm, ipchains and whatnot in one go and call the new
release without this legacies 2.7.0?

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sun, 04 Dec 2005, Jesper Juhl wrote:

> On 12/3/05, Greg KH <greg@kroah.com> wrote:
> > On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> > >
> > > Why can't this be done by distributors/vendors?
> >
> > It already is done by these people, look at the "enterprise" Linux
> > distributions and their 5 years of maintance (or whatever the number
> > is.)
> >
> > If people/customers want stability, they already have this option.
> >
> 
> Yes, I know this is what's done with the "enterprise" distro kernels.
> Perhaps I should have phrased it "Why can't this job just stay with
> vendors".

Because this is just shifting the blame for and work to make up for the
upstream not providing a stable tree on somebody else and prescinds from
the fact that many people are apparently unhappy with 2.6.X policies.

I cannot see a project issuing "stable releases" if every other
developer bleats "let the distro snapshot and backport fixes on their
own". This is exactly the point that turns away half of those who hadn't
been scared away by the "Linux has no uniform userland" problem yet.

2.6.0 is now nearly two years old, perhaps the current discussions mean
that 2.7/2.8 are long overdue - some people feel the need for more
radical code changes, which are 2.7 stuff.

The problem is the upstream breaking backwards compatibility for no good
reason. This can sometimes be a genuine unintended regression (aka.
bug), but quite often this is deliberate breakage because someone wants
to get rid of cruft. While the motivation is sound, breaking between
2.6.N and 2.6.M must stop.

One of the ideas of the new development style and versioning scheme was
to have 2.6 progress faster than 2.3 or 2.5, and to have shorter release
cycles.  It was found to introduce way too much breakage. Linus's
relatively new policy "merge new stuff only during the fortnight after
release, then fix up" is a concession to these observations that too
many things break if there is a constant influx of feature changes.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Xose Vazquez Perez]

Jeff V. Merkey wrote:

> These folks have nothing new to innovate here. The memory manager and VM 
> gets revamped every other release. Exports get broken, binary only 
> module compatibility busted every rev of the kernel. I spend weeks on 
> each kernel fixing the breakage. These people don't get it, don't care, 
> and to be honest, you are wasting your time here trying to convince 
> them. It's never stable because they don't want it to be. This is how 
> they maintain control
> of this code. I have apps written for Windows in 1990 and 1998 that 
> still run on Windows XP today. Linux has no such concept of
> backwards compatiblity. Every company who has embraced it outside of 
> hardware based solutions is dying or has died. IBM is secretly
> forking it as we speak and using it to get out of paying for Unix licenses.
> As annoying as it is, accept it and live with it. These people have no 
> sense of loyalty to you or your customers. They don't even care about 
> each other.

Linux is _only_ a kernel, not a complete OS. And in a very big development
process [1].

If you want a complete OS get Fedora, openSUSE, Debian, etc. And if you need
longer life time, support, certifications get SLES or RHEL.


Btw, latest Coverity reports [2] shows things are getting better and the
main root of bugs are _drivers_ (53%), and far away filesystems(18%) and
inside net(15%).


[1] http://www.zip.com.au/~akpm/x.jpg
[2] http://www.coverity.com/forms/register.php?continue[]=open_source
-- 
Romanes eunt domus

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> What's wrong with offering an unified branch with few regressions for 
> both users and distributions? It's not that every distribution will use 
> it, but as soon as one or two distributions are using it the amount of 
> extra work for maintaining the branch should become pretty low.

the problem is simple: distributions will NOT use it. They can't, they
need the newer HW support and the new features. The only difference is
that you basically added just another distro branch. If you knew how
many people it takes a distro to run such an old tree you'd be scared.
(you need to include the QA people as well in this)

And distros hardly add hw support (the only hw support the enterprise
distros add is contained to like 5 or 10 drivers of "enterprise" stuff,
well testable and often validated by the vendor of the hw; and even then
there are regressions regularly)... for your branch you target a
different audience that does want a lot broader new HW support.

And then there's the bugfixes.. those tend to have regressions too,
especially if you move them to a different context than they originally
came from.

I'm not saying that you couldn't or shouldn't do this, I'm just saying
that I think it'll be a LOT more work than you think, and that the gain
is relatively minor. Especially since the main branch needs to sort the
compatibility item ANYWAY.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> >
> > > . Another
> > > advantage would be to benefit external projects and hardware producers
> > > writing open drivers, enlowering the effort in writing and mantaining
> > > a driver.
> >
> > there is an even better model for those: Get it merged into kernel.org!
> >
> >
> > There is an even bigger deal here: even if you're not ready to get
> > merged yet, staying on the same old version for 6 months is NOT going to
> > help you. In fact it's worse: it is 10x easier to deal with 6 small
> > steps of 1 month than to deal with 1 big step of 6 months.
> >
> >
> from the kernel.org point of view it does make sense but from users
> pov i think no. Users stuck with old drivers not actively mantained
> would benefit from this.

actually no. since that only buys them a few months at most, and then
you're entirely stuck again. And patching it up after 6 months is a lot
harder than doing smaller steps of 1 month, especially for less
experienced people.


> There are some open drivers wrote by hardware mantainers which will
> never get into kernel.org cause of code not following kernel style
> guides and so on. 

which ones did you have in mind?

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Luke-Jr]

On Sunday 04 December 2005 00:20, Greg KH wrote:
> > Switch "broken bloaty bulky devfs" to "lean & clean devfs"?  This ship
> > would have been flying the "clean-up nation" flags.
>
> Again, because an in-kernel devfs is not the correct thing to do.  devfs
> has been disabled for a few months now, and I don't think anyone has
> missed it yet :)

Well, devfs does have some abilities udev doesn't: hotplug/udev doesn't detect 
everything, and can result in rarer or non-PnP devices not being 
automatically available; devfs has the effect of trying to load a module when 
a program looks for the devices it provides-- while it can cause problems, it 
does have a possibility to work better. Interesting effects of switching my 
desktop from devfs to udev:
1. my DVD burners are left uninitialized until I manually modprobe ide-cd or 
(more recently) ide-scsi
2. my sound card is autodetected and the drivers loaded, but the OSS emulation 
modules are omitted; with devfs, they would be autoloaded when an app tried 
to use OSS
devfs also has the advantage of keeping the module info all in one place-- the 
kernel or the module. In particular, with udev the detection and /dev info is 
scattered into different locations of the filesystem. This can probably be 
fixed easily simply by having udev read such info from modules or via a /sys 
entry, though.
-- 
Luke-Jr
Developer, Utopios
http://utopios.org/

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Jesper Juhl]

On 12/3/05, Greg KH <greg@kroah.com> wrote:
> On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> >
> > Why can't this be done by distributors/vendors?
>
> It already is done by these people, look at the "enterprise" Linux
> distributions and their 5 years of maintance (or whatever the number
> is.)
>
> If people/customers want stability, they already have this option.
>

Yes, I know this is what's done with the "enterprise" distro kernels.
Perhaps I should have phrased it "Why can't this job just stay with
vendors".


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Mark Lord]

Linus Torvalds wrote:
> 
> I don't think vbetool has been "broken", it should work fine again. It was 
> just temporarily (and unintentionally) broken for a while.
> 
> But if it's still broken in 2.6.15-rc4, please do holler

Yup, working fine again now with rc4.

Thanks Linus!

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Horst von Brand]

Matthias Andree <matthias.andree@gmx.de> wrote:
> On Sat, 03 Dec 2005, David Ranson wrote:
> > Adrian Bunk wrote:
> > 
> > >- support for ipfwadm and ipchains was removed during 2.6

> > Surely this one had loads of notice though? I was using iptables with
> > 2.4 kernels.

Sure had. They were scheduled for removal in march, 2005 a long time ago.

> So was I. And now what? ipfwadm and ipchains should have been removed
> from 2.6.0 if 2.6.0 was not to support these.

Or in 2.6.10, or 2.6.27, or whatever.

>                                               That opportunity was
> missed, the removal wasn't made up for in 2.6.1, so the stuff has to
> stick until 2.8.0.

Sorry, but the new development model is that there is no "uneven" series
anymore. Sure, it /might/ open for worldshattering changes, but nothing of
that sort is remotely in sight right now, so...

> > >- devfs support was removed during 2.6
> >
> > Did this affect many 'real' users?

> This doesn't matter. A kernel that calls itself stable CAN NOT remove
> features unless they had been critically broken from the beginning. And
> this level of breakage is a moot point, so removal is not justified.

devfs was broken, and very little used.

> > >- removal of kernel support for pcmcia-cs is pending
> > >- ip{,6}_queue removal is pending
> > >- removal of the RAW driver is pending
> 
> > I don't use any of these. I guess pcmcia-cs may be disruptive for laptop
> > users.

> Who cares what you or I use? It's a commonly acknowledged policy that
> "stable" releases do not remove features that are good enough for some.

Right, for a suitably large set of "some". If the set is too small...

> Linux 2.6 is not "stable" in this regard.

Right. The idea of "stable series" had to go. And went.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Jeff V. Merkey]

Matthias Andree wrote:

>On Sat, 03 Dec 2005, Arjan van de Ven wrote:
>
>  
>
>>>Exactly that, and kernel interfaces going away just to annoy binary
>>>module providers also hurts third-party OSS modules, such as
>>>Fujitsu-Siemens's ServerView agents.
>>>      
>>>
>>in kernel API never was and never will be stable, that's entirely
>>irrelevant and independent of the proposal at hand.
>>    
>>
>
>It's still an annoying side effect - is there a list of kernel functions
>removed, with version removed, and with replacement? I know of none, but
>then again I don't hack the kernel very often.
>
>  
>
These folks have nothing new to innovate here. The memory manager and VM 
gets revamped every other release. Exports get broken, binary only 
module compatibility busted every rev of the kernel. I spend weeks on 
each kernel fixing the breakage. These people don't get it, don't care, 
and to be honest, you are wasting your time here trying to convince 
them. It's never stable because they don't want it to be. This is how 
they maintain control
of this code. I have apps written for Windows in 1990 and 1998 that 
still run on Windows XP today. Linux has no such concept of
backwards compatiblity. Every company who has embraced it outside of 
hardware based solutions is dying or has died. IBM is secretly
forking it as we speak and using it to get out of paying for Unix licenses.

As annoying as it is, accept it and live with it. These people have no 
sense of loyalty to you or your customers. They don't even care about 
each other.
Linux is not a "family" in any sense. I wanted very much to believe this 
and I was loyal to these folks for 10 years, invested millions of dollars
in development of my and others money in development to support it, 
crippled Novell and pushed them to embrace Linux and my reward was
smearing and expulsion from the community. They have no direction and 
the whole thing is stagnant now. All the development is incremental
bug fixes and anti-competitive mods to break each others distros.

You are standing on a battlefield. Quietly fork each release, make your 
mods, post patches somewhere for the poor civilians who are
"collateral damage" in the war with constantly busted software, and you 
might help some folks.

Jeff

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sat, Dec 03, 2005 at 11:50:20PM +0100, Matthias Andree wrote:
> The point is, removing something that has worked well enough that some
> people had a reason to use it, is not "stable".

Please remember, no one is calling 2.6 "stable" anymore than they are
calling it "development".  The current development model is different
than what we used to do pre 2.6.  See the archives for details about
this if you want more information.

> Third, IF udev is so sexy but OTOH a real kernel-space devfs can be done
> in 200 LoC as has been claimed so often,

282 LoC:
 drivers/base/class.c   |    7 +
 fs/Kconfig             |    3 
 fs/Makefile            |    1 
 fs/ndevfs/Makefile     |    4 
 fs/ndevfs/inode.c      |  249 +++++++++++++++++++++++++++++++++++++++++++++++++
 fs/partitions/check.c  |    6 +
 include/linux/ndevfs.h |   13 ++
 7 files changed, 282 insertions(+), 1 deletion(-)

> why in hell is this not happening?

Because it's not the correct solution.

> Switch "broken bloaty bulky devfs" to "lean & clean devfs"?  This ship
> would have been flying the "clean-up nation" flags.

Again, because an in-kernel devfs is not the correct thing to do.  devfs
has been disabled for a few months now, and I don't think anyone has
missed it yet :)

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lee Revell]

On Sat, 2005-12-03 at 23:58 +0100, Matthias Andree wrote:
> On Sat, 03 Dec 2005, Lee Revell wrote:
> 
> > You seem to be saying that the current development model is unacceptable
> > for users for whom older kernel work just fine, and the main risk in
> > upgrading is regression.  But the new development model is clearly
> > needed for those users whose needs are not met by the old kernel, say
> > due to unacceptable soft RT performance or unsupported hardware.
> > 
> > But it's wrong to try to evenly balance the needs of these two classes
> > of users, because the first class has another option - they can stick
> > with the old kernel that works for them.  The second class of users has
> 
> The point that just escaped you as the motivation for this thread was
> the availability of security (or other critical) fixes for older
> kernels. It would all be fine if, say, the fix for CVE-2004-2492 were
> available for those who find 2.6.8 works for them (the fix went into
> 2.6.14 BTW), and the concern is the development model isn't fit to
> accomodate needs like this.
> 

If you want security fixes backported then you can get a distro kernel.

Lee

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sun, Dec 04, 2005 at 12:05:20AM +0100, Matthias Andree wrote:
> On Sat, 03 Dec 2005, Arjan van de Ven wrote:
> 
> > 
> > > Exactly that, and kernel interfaces going away just to annoy binary
> > > module providers also hurts third-party OSS modules, such as
> > > Fujitsu-Siemens's ServerView agents.
> > 
> > in kernel API never was and never will be stable, that's entirely
> > irrelevant and independent of the proposal at hand.
> 
> It's still an annoying side effect - is there a list of kernel functions
> removed, with version removed, and with replacement? I know of none, but
> then again I don't hack the kernel very often.

Both lwn.net and the kernelnewbies wiki are trying to track this.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Chris Wright]

* Adrian Bunk (bunk@stusta.de) wrote:
> I don't get the point where the advantage is when every distribution 
> creates it's own stable branches.

They often start from a different base.

> AFAIR one of the reasons for the current 2.6 development model was to 
> reduce the amount of feature patches distributions ship by offering an 
> ftp.kernel.org kernel that gets new features early.
> 
> What's wrong with offering an unified branch with few regressions for 
> both users and distributions? It's not that every distribution will use 
> it, but as soon as one or two distributions are using it the amount of 
> extra work for maintaining the branch should become pretty low.

Backporting is real work, and can introduce instabilities of its own.
If the goal is to stop breaking userspace ABI, there's no guarantee this
will be done via backporting w/out careful inspection, esp. for sysfs and
proc entries.  More to the point, breaking userspace (I'm not talking about
deprecated features) should stop upstream, not only in some frozen branch.
If the goal is to make sure old kernels have security fixes, which old
kernel branch do you follow, the numbers will only grow.  Distros are in a
position to look at current -stable updates and see if security fixes are
relevant.  About the only thing I think is helpful in this case is perhaps
one extra -stable cycle on the last branch when newest branch is released
(basically flush the queue).  That much I'm willing to do in -stable.

thanks,
-chris

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sat, Dec 03, 2005 at 11:51:05PM +0100, Adrian Bunk wrote:
> On Sat, Dec 03, 2005 at 12:19:45PM -0800, Greg KH wrote:
> > On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> > > 
> > > Why can't this be done by distributors/vendors?
> > 
> > It already is done by these people, look at the "enterprise" Linux
> > distributions and their 5 years of maintance (or whatever the number
> > is.)
> > 
> > If people/customers want stability, they already have this option.
> 
> I don't get the point where the advantage is when every distribution 
> creates it's own stable branches.

They only do so because they are on different release cycles.

> AFAIR one of the reasons for the current 2.6 development model was to 
> reduce the amount of feature patches distributions ship by offering an 
> ftp.kernel.org kernel that gets new features early.

Sure, that's one of the reasons.  But not the only one by far (I have a
whole list somewhere, I'm sure it's in the archives...)

> What's wrong with offering an unified branch with few regressions for 
> both users and distributions? It's not that every distribution will use 
> it, but as soon as one or two distributions are using it the amount of 
> extra work for maintaining the branch should become pretty low.

"pretty low"?  hahahahaha.  As Arjan has said, the time and energy to do
this for a long period of time is huge.  If I were you, I would listen
to the people who have and do maintain these kinds of kernels, it's not
a simple job by any means.

But by all means, if you want to try to do this, please do.  I wish you
good luck.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Dave Jones]

On Sun, Dec 04, 2005 at 12:02:54AM +0100, Matthias Andree wrote:
 > On Sat, 03 Dec 2005, Dave Jones wrote:
 > 
 > > In many cases, submitters of changes know that things are going
 > > to break. Maybe we need a policy that says changes requiring userspace updates
 > > need to be clearly documented in the mails Linus gets (Especially if its
 > > a git pull request), so that when the next point release gets released,
 > > Linus can put a section in the announcement detailing what bits
 > > of userspace are needed to be updated.
 > 
 > This isn't acceptable in stable kernels. FreeBSD has a very tight
 > policy, newer kernels off the same branch support older user space.

The BSDs have an advantage in that their userspace & kernels are closely
coupled. When kernel changes need a userspace change, it gets done at the
same time in the same repository.

		Dave

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, Arjan van de Ven wrote:

> 
> > Exactly that, and kernel interfaces going away just to annoy binary
> > module providers also hurts third-party OSS modules, such as
> > Fujitsu-Siemens's ServerView agents.
> 
> in kernel API never was and never will be stable, that's entirely
> irrelevant and independent of the proposal at hand.

It's still an annoying side effect - is there a list of kernel functions
removed, with version removed, and with replacement? I know of none, but
then again I don't hack the kernel very often.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, Dave Jones wrote:

> In many cases, submitters of changes know that things are going
> to break. Maybe we need a policy that says changes requiring userspace updates
> need to be clearly documented in the mails Linus gets (Especially if its
> a git pull request), so that when the next point release gets released,
> Linus can put a section in the announcement detailing what bits
> of userspace are needed to be updated.

This isn't acceptable in stable kernels. FreeBSD has a very tight
policy, newer kernels off the same branch support older user space. The
upgrade path is clear, reboot into new kernel, have it spit a few
reminders that your userspace needs update (Linux also has this, for
instance, with SG_IO and its predecessors) but still everything works.

Requiring new userspace at a patchlevel kernel upgrade is nothing but
impertinent, unless that updated userspace ships as part of the kernel.

> It still isn't to solve the problem of regressions in drivers, but
> that's a problem that's not easily solvable.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, Lee Revell wrote:

> You seem to be saying that the current development model is unacceptable
> for users for whom older kernel work just fine, and the main risk in
> upgrading is regression.  But the new development model is clearly
> needed for those users whose needs are not met by the old kernel, say
> due to unacceptable soft RT performance or unsupported hardware.
> 
> But it's wrong to try to evenly balance the needs of these two classes
> of users, because the first class has another option - they can stick
> with the old kernel that works for them.  The second class of users has

The point that just escaped you as the motivation for this thread was
the availability of security (or other critical) fixes for older
kernels. It would all be fine if, say, the fix for CVE-2004-2492 were
available for those who find 2.6.8 works for them (the fix went into
2.6.14 BTW), and the concern is the development model isn't fit to
accomodate needs like this.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Sat, Dec 03, 2005 at 12:19:45PM -0800, Greg KH wrote:
> On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> > 
> > Why can't this be done by distributors/vendors?
> 
> It already is done by these people, look at the "enterprise" Linux
> distributions and their 5 years of maintance (or whatever the number
> is.)
> 
> If people/customers want stability, they already have this option.

I don't get the point where the advantage is when every distribution 
creates it's own stable branches.

AFAIR one of the reasons for the current 2.6 development model was to 
reduce the amount of feature patches distributions ship by offering an 
ftp.kernel.org kernel that gets new features early.

What's wrong with offering an unified branch with few regressions for 
both users and distributions? It's not that every distribution will use 
it, but as soon as one or two distributions are using it the amount of 
extra work for maintaining the branch should become pretty low.

> thanks,
> 
> greg k-h

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, David Ranson wrote:

> Matthias Andree wrote:
> 
> >So was I. And now what? ipfwadm and ipchains should have been removed
> >from 2.6.0 if 2.6.0 was not to support these. That opportunity was
> >missed, the removal wasn't made up for in 2.6.1, so the stuff has to
> >stick until 2.8.0.
> >
> >
> I'm not aware of that policy... maybe I overlooked something?

Everyone does it, except Linux and GDBM.

> I guess our definitions of stable (and the degree of stability
> acceptable) differ.

No need to guess, this is quite obvious.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, Lee Revell wrote:

> On Sat, 2005-12-03 at 23:27 +0100, Matthias Andree wrote:
> > A kernel that calls itself stable CAN NOT remove
> > features unless they had been critically broken from the beginning. 
> 
> So in your opinion we can't add support for new hardware to a stable
> kernel either because there's a chance of breaking something that worked
> before, which brings us right back to "stable" meaning "no progress
> allowed", which begs the question of why you want to upgrade at all.

Perhaps I did not word not clearly enough, please bear with me as I'm
not a native speaker.

There's a gray area between these two extremes. I don't mind if new
drivers are added, not even if they touch other parts of the code if
these changes are thoroughly (and I mean thoroughly, not a smoke test of
the "works for me" kind) examined.

If devfs had been marked "DEPRECATED, WILL BE REMOVED FROM 2.6.0", all
would have been fine. (I don't recall the exact version, 2.6.12? It
wasn't known beforehand), I certainly do not expect new drivers that are
added to support it.

First step, make a note "this driver has been added in Linux 2.6.14" so
that everyone is aware the driver doesn't need to support devfs as that
one was already deprecated then the driver moved in. Even better, mark
which deprecated subsystems are unsupported by the driver.

Second, schedule for removal such subsystems well ahead of time, for
instance, "DEPRECATED, WILL BE REMOVED JUST BEFORE 2.8.0", and only use
minor releases to that extent.

The point is, removing something that has worked well enough that some
people had a reason to use it, is not "stable".

Third, IF udev is so sexy but OTOH a real kernel-space devfs can be done
in 200 LoC as has been claimed so often, why in hell is this not
happening? Switch "broken bloaty bulky devfs" to "lean & clean devfs"?
This ship would have been flying the "clean-up nation" flags.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Steven Rostedt]

On Sat, 2005-12-03 at 14:29 -0800, Greg KH wrote:
> On Sat, Dec 03, 2005 at 11:21:38PM +0100, Matthias Andree wrote:
> > On Sat, 03 Dec 2005, David Ranson wrote:
> > 
> > > Steven Rostedt wrote:
> > > 
> > > >udev ;)
> > > >
> > > >http://seclists.org/lists/linux-kernel/2005/Dec/0180.html
> > > >
> > > >
> > > Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> > > userspace interface broken during the series, does anyone have any more?
> > 
> > Not only that, udev is default for instance in recent SUSE Linux
> > releases, so whether to use or not to use it is a major effort.
> 
> And if you use SUSE releases, use OpenSuSE to keep up to date with all
> of the needed kernel programs for the latest kernels.  Same with Fedora,
> Debian, or Gentoo.  They all keep up to date quite well.

And if you use Debian, make sure you remember to do an update after
changing a kernel and finding out that something in userland doesn't
work.  As Greg mentioned to me in the above mentioned thread.
Debian/unstable had the proper udev.  I just haven't done an update
recently, but I download kernels practically every day.

-- Steve

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, Greg KH wrote:

> And if you use SUSE releases, use OpenSuSE to keep up to date with all
> of the needed kernel programs for the latest kernels.  Same with Fedora,
> Debian, or Gentoo.  They all keep up to date quite well.

Well, particular problem I've had: netfilter-enabled machines were
incapable to download large files, for instance newer ICC8 releases,
(major annoyance, their IIS crap doesn't support "Bytes" ranges, so you
start all over if one packet went down the wrong throat). I was told
"try newer kernels first", there had been fixes with out-of-window ACKs
and whatnot. I do not intend to upgrade all of my distro to the bleeding
OpenSUSE release just to find out if the new kernel fixes it and to see
new regressions. I have more interesting things to do with my time than
chase the userspace change of the day.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Sat, Dec 03, 2005 at 10:18:10PM +0100, Lars Marowsky-Bree wrote:
> On 2005-12-03T16:13:29, Dave Jones <davej@redhat.com> wrote:
> 
> > The big problem is though that we don't typically find out that
> > we've regressed until after a kernel update is in the end-users hands.
> > 
> > In many cases, submitters of changes know that things are going
> > to break. Maybe we need a policy that says changes requiring userspace updates
> > need to be clearly documented in the mails Linus gets (Especially if its
> > a git pull request), so that when the next point release gets released,
> > Linus can put a section in the announcement detailing what bits
> > of userspace are needed to be updated.
> 
> True, but this first block doesn't really qualify as a "regression".
> Yes, a clearer-than-crystal documentation of "this kernel requires
> user-space component foo to be at least x.y.z if feature bar is used"
> would go a long way.
> 
> And if then user-space itself was tolerant of at least version N and
> N-1, then users could even roll back one kernel version if problems
> arise.
> 
> Both of these are documentation and user-space issues, and don't much
> depend on changes to kernel development model.
>...

One effect that you mustn't underestimate is that if people had to got 
to N-1 for two or three different N due to different regressions in 
different kernels, they might decide to stay with kernel version M even 
though kernel M+10 might have been released and version M lacks tons of 
security fixes.

> Sincerely,
>     Lars Marowsky-Brée <lmb@suse.de>

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[David Ranson]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

Matthias Andree wrote:

>So was I. And now what? ipfwadm and ipchains should have been removed
>from 2.6.0 if 2.6.0 was not to support these. That opportunity was
>missed, the removal wasn't made up for in 2.6.1, so the stuff has to
>stick until 2.8.0.
>
>
I'm not aware of that policy... maybe I overlooked something?

>This doesn't matter. A kernel that calls itself stable CAN NOT remove
>features unless they had been critically broken from the beginning. And
>this level of breakage is a moot point, so removal is not justified.
>
>

>Who cares what you or I use? It's a commonly acknowledged policy that
>"stable" releases do not remove features that are good enough for some.
>Linux 2.6 is not "stable" in this regard.
>
>
I guess our definitions of stable (and the degree of stability
acceptable) differ.

David



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 187 bytes --]

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lee Revell]

On Sat, 2005-12-03 at 23:27 +0100, Matthias Andree wrote:
> A kernel that calls itself stable CAN NOT remove
> features unless they had been critically broken from the beginning. 

So in your opinion we can't add support for new hardware to a stable
kernel either because there's a chance of breaking something that worked
before, which brings us right back to "stable" meaning "no progress
allowed", which begs the question of why you want to upgrade at all.

Lee

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sat, Dec 03, 2005 at 11:21:38PM +0100, Matthias Andree wrote:
> On Sat, 03 Dec 2005, David Ranson wrote:
> 
> > Steven Rostedt wrote:
> > 
> > >udev ;)
> > >
> > >http://seclists.org/lists/linux-kernel/2005/Dec/0180.html
> > >
> > >
> > Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> > userspace interface broken during the series, does anyone have any more?
> 
> Not only that, udev is default for instance in recent SUSE Linux
> releases, so whether to use or not to use it is a major effort.

And if you use SUSE releases, use OpenSuSE to keep up to date with all
of the needed kernel programs for the latest kernels.  Same with Fedora,
Debian, or Gentoo.  They all keep up to date quite well.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, David Ranson wrote:

> Adrian Bunk wrote:
> 
> >- support for ipfwadm and ipchains was removed during 2.6
> >
> >
> Surely this one had loads of notice though? I was using iptables with
> 2.4 kernels.

So was I. And now what? ipfwadm and ipchains should have been removed
from 2.6.0 if 2.6.0 was not to support these. That opportunity was
missed, the removal wasn't made up for in 2.6.1, so the stuff has to
stick until 2.8.0.

> >- devfs support was removed during 2.6
>
> Did this affect many 'real' users?

This doesn't matter. A kernel that calls itself stable CAN NOT remove
features unless they had been critically broken from the beginning. And
this level of breakage is a moot point, so removal is not justified.

> >- removal of kernel support for pcmcia-cs is pending
> >- ip{,6}_queue removal is pending
> >- removal of the RAW driver is pending

> I don't use any of these. I guess pcmcia-cs may be disruptive for laptop
> users.

Who cares what you or I use? It's a commonly acknowledged policy that
"stable" releases do not remove features that are good enough for some.
Linux 2.6 is not "stable" in this regard.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sat, Dec 03, 2005 at 10:53:18PM +0100, M. wrote:
> from the kernel.org point of view it does make sense but from users
> pov i think no. Users stuck with old drivers not actively mantained
> would benefit from this.

Any specific examples of this?

> There are some open drivers wrote by hardware mantainers which will
> never get into kernel.org cause of code not following kernel style
> guides and so on. Yeah, you should not buy poorly supported hardware
> and use bad drivers but a lot of new users have poorly supported
> hardware and a "more stable than usual and at fixed dates" release
> could enlower the skills barrier in approaching linux.

The skills barrier has _nothing_ to do with the release cycles.

And if you want to get a driver into the main tree, that isn't being
maintained, just get a piece of the hardware to a kernel developer,
along with the driver and it will be maintained.  I know I've made that
offer many times in the past, and so has others.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, David Ranson wrote:

> Steven Rostedt wrote:
> 
> >udev ;)
> >
> >http://seclists.org/lists/linux-kernel/2005/Dec/0180.html
> >
> >
> Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> userspace interface broken during the series, does anyone have any more?

Not only that, udev is default for instance in recent SUSE Linux
releases, so whether to use or not to use it is a major effort.

-- 
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sat, Dec 03, 2005 at 10:31:03PM +0100, M. wrote:
> makes sense, but are you sure having distros like Debian, enterprise
> products from redhat etc using the same 6months release for their
> stable versions does not translate in minor fragmentation on kernel
> development and in benefits for every user?

It hasn't so far from my viewpoint.  Do you think it has?

> Under this light i think a 6months cycle starts to mean something when
> stable distros gets older and older (debian and redhat enterprise are
> released every 18/24 months) and developers and who wants bleeding
> edge features can always use Fedora, openSUSE, Gentoo etc. Another
> advantage would be to benefit external projects and hardware producers
> writing open drivers, enlowering the effort in writing and mantaining
> a driver.

Drivers belong in the main kernel.org kernel tree.  See
Documentation/stable-api-nonsense.txt for why this is so, and
Documentation/HOWTO for how to get this accomplished.  It is explained
in great detail there.  If you have further questions about this, please
let us know.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[M.]

On 12/3/05, Arjan van de Ven <arjan@infradead.org> wrote:
>
> > <sorry for the direct reply>
> >
> > makes sense, but are you sure having distros like Debian, enterprise
> > products from redhat etc using the same 6months release for their
> > stable versions does not translate in minor fragmentation on kernel
> > development
>
> I'm quite sure that there isn't significant fragmentation; all those
> distros in their maintenance generally only take patches that are
> already upstream (or they send them upstream during the maintenance)
> just to make sure that their long term costs don't go insane
> (eg for the $nextversion, the distros can just start clean because they
> know all bugfixes from maintenance versions are already in the new
> kernel.org kernel they get; not doing that is REALLY expensive so
> distros like to avoid that)
>
>
> > and in benefits for every user?
>
> you can't have it both ways; you can't be "new" and "old stable" at the
> same time.
>
> > . Another
> > advantage would be to benefit external projects and hardware producers
> > writing open drivers, enlowering the effort in writing and mantaining
> > a driver.
>
> there is an even better model for those: Get it merged into kernel.org!
>
>
> There is an even bigger deal here: even if you're not ready to get
> merged yet, staying on the same old version for 6 months is NOT going to
> help you. In fact it's worse: it is 10x easier to deal with 6 small
> steps of 1 month than to deal with 1 big step of 6 months.
>
>
from the kernel.org point of view it does make sense but from users
pov i think no. Users stuck with old drivers not actively mantained
would benefit from this.

There are some open drivers wrote by hardware mantainers which will
never get into kernel.org cause of code not following kernel style
guides and so on. Yeah, you should not buy poorly supported hardware
and use bad drivers but a lot of new users have poorly supported
hardware and a "more stable than usual and at fixed dates" release
could enlower the skills barrier in approaching linux.

Michele

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> <sorry for the direct reply>
> 
> makes sense, but are you sure having distros like Debian, enterprise
> products from redhat etc using the same 6months release for their
> stable versions does not translate in minor fragmentation on kernel
> development 

I'm quite sure that there isn't significant fragmentation; all those
distros in their maintenance generally only take patches that are
already upstream (or they send them upstream during the maintenance)
just to make sure that their long term costs don't go insane
(eg for the $nextversion, the distros can just start clean because they
know all bugfixes from maintenance versions are already in the new
kernel.org kernel they get; not doing that is REALLY expensive so
distros like to avoid that)


> and in benefits for every user?

you can't have it both ways; you can't be "new" and "old stable" at the
same time. 

> . Another
> advantage would be to benefit external projects and hardware producers
> writing open drivers, enlowering the effort in writing and mantaining
> a driver.

there is an even better model for those: Get it merged into kernel.org!


There is an even bigger deal here: even if you're not ready to get
merged yet, staying on the same old version for 6 months is NOT going to
help you. In fact it's worse: it is 10x easier to deal with 6 small
steps of 1 month than to deal with 1 big step of 6 months. 

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[James Courtier-Dutton]

M. wrote:
> 
> Yes but not home users with relatively new/bleeding edge hardware or
> small projects writing for example a wifi driver or a security patch
> or whatever without full time commitment to tracking kernel changes.
> 

If there are "small projects writing" their own wifi driver, they should 
try to get it included in the kernel ASAP. Then they won't have to track 
the changes, as the person making the changes will automatically change 
their little driver to keep it working after the changes.
Drivers very rarely impact the stability of the rest of the kernel.
It initially gets added as "EXPERIMENTAL" so the user can choose whether 
to even use it or not.

All it takes is for the "small project" to build their own git tree, and 
then ask the Linus or Andrew to pull it. It should get added pretty 
easily, so long as the code looks pretty. :-)
It is really that simple. There is no logical reason for any external 
driver not to be added into the main kernel, so why do people not want 
to add them?

James

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[M.]

On 12/3/05, Greg KH <greg@kroah.com> wrote:
> <dragging the converstation back to lkml, where it belongs...>
>
> On Sat, Dec 03, 2005 at 09:54:35PM +0100, M. wrote:
> > On 12/3/05, Greg KH <greg@kroah.com> wrote:
> > > On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> > > >
> > > > Why can't this be done by distributors/vendors?
> > >
> > > It already is done by these people, look at the "enterprise" Linux
> > > distributions and their 5 years of maintance (or whatever the number
> > > is.)
> > >
> > > If people/customers want stability, they already have this option.
> > >
> > > thanks,
> > >
> > > greg k-h
> >
> > Yes but not home users with relatively new/bleeding edge hardware or
> > small projects writing for example a wifi driver or a security patch
> > or whatever without full time commitment to tracking kernel changes.
>
> If you are a user that wants this kind of support, then use a distro
> that can handle this.  Obvious examples that come to mind are both
> Debian and Gentoo and Fedora and OpenSuSE, and I'm sure there are
> others.
>
> But if you are a developer, you can usually stay up to date by tracking
> the main releases, which should be about once a month.  If you have
> problems porting your stuff to the latest kernel when you need to submit
> it for inclusion, there are lots of people to help point you in the
> proper direction for what is needed to be done.
>
> > Enterprise products are suited for production servers,
> > school/government/companies desktops and not for "enthusiasts" or for
> > small kernel projects (they obviously cant write drivers or patches
> > for custom distro kernels). Those enthusiasts have to get mad with
> > performance regressions, new incompatibilities, new crashes etc.
>
> Sure, then use a different distro for them.  That's why Linux has so
> many different ones, they all are targeted at different users.
>
> thanks,
>
> greg k-h
>
<sorry for the direct reply>

makes sense, but are you sure having distros like Debian, enterprise
products from redhat etc using the same 6months release for their
stable versions does not translate in minor fragmentation on kernel
development and in benefits for every user?

Under this light i think a 6months cycle starts to mean something when
stable distros gets older and older (debian and redhat enterprise are
released every 18/24 months) and developers and who wants bleeding
edge features can always use Fedora, openSUSE, Gentoo etc. Another
advantage would be to benefit external projects and hardware producers
writing open drivers, enlowering the effort in writing and mantaining
a driver.

Michele

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lars Marowsky-Bree]

On 2005-12-03T16:13:29, Dave Jones <davej@redhat.com> wrote:

> The big problem is though that we don't typically find out that
> we've regressed until after a kernel update is in the end-users hands.
> 
> In many cases, submitters of changes know that things are going
> to break. Maybe we need a policy that says changes requiring userspace updates
> need to be clearly documented in the mails Linus gets (Especially if its
> a git pull request), so that when the next point release gets released,
> Linus can put a section in the announcement detailing what bits
> of userspace are needed to be updated.

True, but this first block doesn't really qualify as a "regression".
Yes, a clearer-than-crystal documentation of "this kernel requires
user-space component foo to be at least x.y.z if feature bar is used"
would go a long way.

And if then user-space itself was tolerant of at least version N and
N-1, then users could even roll back one kernel version if problems
arise.

Both of these are documentation and user-space issues, and don't much
depend on changes to kernel development model.

> It still isn't to solve the problem of regressions in drivers, but
> that's a problem that's not easily solvable.

True. Regressions will always occur when driver updates happen. There'll
always be the next bug. I don't think anyone introduces these on purpose
;-)


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Dave Jones]

On Sat, Dec 03, 2005 at 09:59:11PM +0100, Lars Marowsky-Bree wrote:
 > On 2005-12-03T14:56:08, Adrian Bunk <bunk@stusta.de> wrote:
 > 
 > > The current kernel development model is pretty good for people who 
 > > always want to use or offer their costumers the maximum amount of the 
 > > latest bugs^Wfeatures without having to resort on additional patches for 
 > > them.
 > > 
 > > Problems of the current development model from a user's point of view 
 > > are:
 > > - many regressions in every new release
 > > - kernel updates often require updates for the kernel-related userspace 
 > >   (e.g. for udev or the pcmcia tools switch)
 > 
 > Your problem statement is correct, but you're fixing the symptoms, not
 > the cause.
 > 
 > What we need is an easier way for users to pull in kernel updates with
 > the matching kernel-related user-space.
 > 
 > This is provided, though with some lag, by Fedora, openSUSE, Debian
 > testing, dare I say gentoo and others.
 > 
 > The right way to address this is to work with the distribution of your
 > choice to make these updates available faster.

The big problem is though that we don't typically find out that
we've regressed until after a kernel update is in the end-users hands.

In many cases, submitters of changes know that things are going
to break. Maybe we need a policy that says changes requiring userspace updates
need to be clearly documented in the mails Linus gets (Especially if its
a git pull request), so that when the next point release gets released,
Linus can put a section in the announcement detailing what bits
of userspace are needed to be updated.

It still isn't to solve the problem of regressions in drivers, but
that's a problem that's not easily solvable.

		Dave

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

<dragging the converstation back to lkml, where it belongs...>

On Sat, Dec 03, 2005 at 09:54:35PM +0100, M. wrote:
> On 12/3/05, Greg KH <greg@kroah.com> wrote:
> > On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> > >
> > > Why can't this be done by distributors/vendors?
> >
> > It already is done by these people, look at the "enterprise" Linux
> > distributions and their 5 years of maintance (or whatever the number
> > is.)
> >
> > If people/customers want stability, they already have this option.
> >
> > thanks,
> >
> > greg k-h
> 
> Yes but not home users with relatively new/bleeding edge hardware or
> small projects writing for example a wifi driver or a security patch
> or whatever without full time commitment to tracking kernel changes.

If you are a user that wants this kind of support, then use a distro
that can handle this.  Obvious examples that come to mind are both
Debian and Gentoo and Fedora and OpenSuSE, and I'm sure there are
others.

But if you are a developer, you can usually stay up to date by tracking
the main releases, which should be about once a month.  If you have
problems porting your stuff to the latest kernel when you need to submit
it for inclusion, there are lots of people to help point you in the
proper direction for what is needed to be done.

> Enterprise products are suited for production servers,
> school/government/companies desktops and not for "enthusiasts" or for
> small kernel projects (they obviously cant write drivers or patches
> for custom distro kernels). Those enthusiasts have to get mad with
> performance regressions, new incompatibilities, new crashes etc.

Sure, then use a different distro for them.  That's why Linux has so
many different ones, they all are targeted at different users.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[M.]

On 12/3/05, Greg KH <greg@kroah.com> wrote:
> On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> >
> > Why can't this be done by distributors/vendors?
>
> It already is done by these people, look at the "enterprise" Linux
> distributions and their 5 years of maintance (or whatever the number
> is.)
>
> If people/customers want stability, they already have this option.
>
> thanks,
>
> greg k-h
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Yes but not home users with relatively new/bleeding edge hardware or
small projects writing for example a wifi driver or a security patch
or whatever without full time commitment to tracking kernel changes.

Enterprise products are suited for production servers,
school/government/companies desktops and not for "enthusiasts" or for
small kernel projects (they obviously cant write drivers or patches
for custom distro kernels). Those enthusiasts have to get mad with
performance regressions, new incompatibilities, new crashes etc.

My suggestion was to release a 2.6.X kernel every 6months reducing
kernel development fragmentation between different distros and giving
away stabler stuff.

Michele

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[M.]

On 12/3/05, Lee Revell <rlrevell@joe-job.com> wrote:
> On Sat, 2005-12-03 at 20:35 +0100, Adrian Bunk wrote:
> >
> > But yes, what I suggest is partially a step back in a way that it
> > doesn't conflict with the current 2.6.17+ development model.
> >
> > Well, if taking the best from the old style development is improving
> > things that isn't something bad.
>
> You seem to be saying that the current development model is unacceptable
> for users for whom older kernel work just fine, and the main risk in
> upgrading is regression.  But the new development model is clearly
> needed for those users whose needs are not met by the old kernel, say
> due to unacceptable soft RT performance or unsupported hardware.
>
> But it's wrong to try to evenly balance the needs of these two classes
> of users, because the first class has another option - they can stick
> with the old kernel that works for them.  The second class of users has
> no other option, so their needs should be given greater weight.
>
> Lee
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Mhhh something much simpler:

Use the current development model between 2.6.N and 2.6.N+1. This
means the current dev model would be used like NOW, BUT changing the
versioning scheme in a way which reflects the 6months release plans.
TRYING TO EXPLAIN THINGS BETTER: You could skip the version scheme
change and do something like:

EXAMPLE:

2.6.14 - ok let's plan 3/4/whatever releases in the next 6 months with
those major new features:
 - A
 - B
 - C

then: .15, .16 like every 2.6 kernel, same dev model, same everything
(erhm not every just like the latest releases, ok)
.18 would be MAINLY stabilization: no device model changes, no
networking change, and so on (but i think it would be ok new drivers
since they cant be disabled or simply not used)

and .19 would come out  months after the .14 release

it's not a stable/devel or 2.6/2.7 or better 2.4/2.5 scheme (even a
2.6/2.7 scheme almost exists with main tree/-mm tree atm) or old stuff
under a different point of view BUT it's the SAME current dev model
applied to every release EXCEPT for the last one BEFORE the 6months
timeline.

what does it means: no one says nothing is unacceptable, it means
things could be better for distros/users/external kernel projects by
doing something that could be called a two-layers model:

--------------------
Linus & co. "hey, let's write down the major features for the next 6
months and plan things a little"
--------------------
Other kernel developers "I would like to get in X, Y and Z. Let's do
it 2.6.x style."
--------------------

The changed versioning scheme just reflects the 6months release ciclye
idea. fullstop.

trying to clarify again:

> What you're suggesting sounds just like going back to the old style of
> development where 2.<even>.x is stable, and 2.<odd>.x is development.
> You might as well just suggest that after 2.6.16, we fork to 2.7.0, and
> 2.6.17+ will be stable increments like we always used to do.

every 2.6.N.x release would be the SAME as a 2.6.x release. The entire
6months cycle puts in some time/features/stability boundaries on top
of the existent model. Finally, those stability boundaries would be
planned every 6months assuring the best fit of the model with
developers, distros and users needs.


hope I clarified,
Michele

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lars Marowsky-Bree]

On 2005-12-03T14:56:08, Adrian Bunk <bunk@stusta.de> wrote:

> The current kernel development model is pretty good for people who 
> always want to use or offer their costumers the maximum amount of the 
> latest bugs^Wfeatures without having to resort on additional patches for 
> them.
> 
> Problems of the current development model from a user's point of view 
> are:
> - many regressions in every new release
> - kernel updates often require updates for the kernel-related userspace 
>   (e.g. for udev or the pcmcia tools switch)

Your problem statement is correct, but you're fixing the symptoms, not
the cause.

What we need is an easier way for users to pull in kernel updates with
the matching kernel-related user-space.

This is provided, though with some lag, by Fedora, openSUSE, Debian
testing, dare I say gentoo and others.

The right way to address this is to work with the distribution of your
choice to make these updates available faster.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Greg KH]

On Sat, Dec 03, 2005 at 03:29:54PM +0100, Jesper Juhl wrote:
> 
> Why can't this be done by distributors/vendors?

It already is done by these people, look at the "enterprise" Linux
distributions and their 5 years of maintance (or whatever the number
is.)

If people/customers want stability, they already have this option.

thanks,

greg k-h

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Lee Revell]

On Sat, 2005-12-03 at 20:35 +0100, Adrian Bunk wrote:
> 
> But yes, what I suggest is partially a step back in a way that it 
> doesn't conflict with the current 2.6.17+ development model.
> 
> Well, if taking the best from the old style development is improving 
> things that isn't something bad. 

You seem to be saying that the current development model is unacceptable
for users for whom older kernel work just fine, and the main risk in
upgrading is regression.  But the new development model is clearly
needed for those users whose needs are not met by the old kernel, say
due to unacceptable soft RT performance or unsupported hardware.

But it's wrong to try to evenly balance the needs of these two classes
of users, because the first class has another option - they can stick
with the old kernel that works for them.  The second class of users has
no other option, so their needs should be given greater weight.

Lee

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Sat, Dec 03, 2005 at 09:31:03AM -0500, Ben Collins wrote:
> On Sat, 2005-12-03 at 14:56 +0100, Adrian Bunk wrote:
> > The current kernel development model is pretty good for people who 
> > always want to use or offer their costumers the maximum amount of the 
> > latest bugs^Wfeatures without having to resort on additional patches for 
> > them.
> > 
> > Problems of the current development model from a user's point of view 
> > are:
> > - many regressions in every new release
> > - kernel updates often require updates for the kernel-related userspace 
> >   (e.g. for udev or the pcmcia tools switch)
> > 
> > One problem following from this is that people continue to use older 
> > kernels with known security holes because the amount of work for kernel 
> > upgrades is too high.
> 
> What you're suggesting sounds just like going back to the old style of
> development where 2.<even>.x is stable, and 2.<odd>.x is development.
> You might as well just suggest that after 2.6.16, we fork to 2.7.0, and
> 2.6.17+ will be stable increments like we always used to do.
> 
> You're just munging the version scheme :)

The 2.6.17+ development model is different from a traditional 2.7 
development model in the sense that 2.6.17+ contains regular relatively 
stable releases.

But yes, what I suggest is partially a step back in a way that it 
doesn't conflict with the current 2.6.17+ development model.

Well, if taking the best from the old style development is improving 
things that isn't something bad.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Linus Torvalds]

On Sat, 3 Dec 2005, Mark Lord wrote:

> David Ranson wrote:
> > 
> > Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> > userspace interface broken during the series, does anyone have any more?
> 
> vbetool.

I don't think vbetool has been "broken", it should work fine again. It was 
just temporarily (and unintentionally) broken for a while.

But if it's still broken in 2.6.15-rc4, please do holler (with as many 
details as you can)

		Linus

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Dr. David Alan Gilbert]

Hi Adrian,
  I would really appreciate such a move to a stable series.
I've had some really bad luck with instability of 2.6.x - in
particular with NFS.

  Would such a stable kernel keep up to date on basic drivers?
I ask since I got into a messy situation on a series of production
servers;  they were really new Dell servers using standard Intel
chipsets but needed SATA stuff that went in recently.
Does 2.6.16 have the basic infrastructure for all the current
hardware so that if you branch now you aren't going to have
to do any really heavy backports to be able to run on
'current' hardware?

I hit the situation where I have a 2.6.5 kernel I use on everything
else and whose NFS works fine; and 2.6.11 or newer which supports
the hardware - but whose NFS is giving me broken locking to some
obscure systems.

IMHO we've also got into a real mess where it is vendor
kernels that have stability fixes in for many things (NFS in
particular) - but the lkml doesn't want to know about vendor
kernels, but at the same time they aren't up for stabilisation.

Good luck with such a branch!

Dave
--
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[David Ranson]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

Adrian Bunk wrote:

>- support for ipfwadm and ipchains was removed during 2.6
>
>
Surely this one had loads of notice though? I was using iptables with
2.4 kernels.

>- devfs support was removed during 2.6
>
>
Did this affect many 'real' users?

>- removal of kernel support for pcmcia-cs is pending
>- ip{,6}_queue removal is pending
>- removal of the RAW driver is pending
>
>
I don't use any of these. I guess pcmcia-cs may be disruptive for laptop
users.

So far I don't see evidence to suggest huge repeated userspace breakages
between Kernel versions that were implied earlier in this thread.
Whatever, we aren't going to see any more stable branches without
volunteers to do the spadework. As has been pointed out, this won't
always be an easy task.

David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 187 bytes --]

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Mark Lord]

David Ranson wrote:
>
> Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> userspace interface broken during the series, does anyone have any more?

vbetool.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Sat, Dec 03, 2005 at 05:17:41PM +0000, David Ranson wrote:
> Steven Rostedt wrote:
> 
> >udev ;)
> >
> >http://seclists.org/lists/linux-kernel/2005/Dec/0180.html
> >
> >
> Ahh OK .. I don't use it, so wouldn't have been affected. That's one
> userspace interface broken during the series, does anyone have any more?

- support for ipfwadm and ipchains was removed during 2.6
- devfs support was removed during 2.6
- removal of kernel support for pcmcia-cs is pending
- ip{,6}_queue removal is pending
- removal of the RAW driver is pending

> David

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[M.]

Hi everyone.

Here to expose my point of view, hope u'll enjoy :)

Starting stable series off the 2.6 simply wouldn't work: there are no
paid QA guys ready to test, fix, stabilize those series, kernel
developers wants to hang on new exciting stuff.
This tendence leads to faster innovations in kernel core and features
set but leaving the "forking" effort to distributions leads to
fragmentation too: almost every distro has a different base kernel on
which doing testing and fixing and this, in my opinion, is not
positive for the kernel.org kernels. Another problem of the current
development model is that fast changes are difficult to track for
small external projects (those whitout big $$ behind), the small
projects which made linux so great in the past.
Maybe a way to reduce those problems is to release the kernel like
GNOME, KDE, fedora & co. do for their stuff: one stable release every
6 months but build on top of the current way of doing things. Example:

2.6.14 released on 27 October, then:
  2.6.14.1-gitN until 2.6.14.1-rcN -> 2.6.14.1
  2.6.14.2-gitN until 2.6.14.2-rcN -> 2.6.14.2
  ...
  (maybe last 2.6.14-N, which could be called 2.6.15-gitN ->
2.6.15-rcN, only bugfixes and small changes, main development in -mm
or in a new -something tree during this last phase)
2.6.15 release on March

those middle releases would be handled with the current development
model except for the last one. So, the largest part of developers
would continue to think and to work using the current development
model and some guys would be able to plan a list of features and
functionalities every 6 months and, based on this list, handle the
6-months release giving guide lines (like Linus and friends already do
but, i repeat, focusing on a 6 months time window)

Doing things this way would lead to distributions aligning to the same
kernel and open up a possible scenario of distros collaborating to
mantain the latest stable release. This should make small projects and
users who want to run bleeding edge stuff lifes easier too.

of couse the time window could be larger or smaller but doing things
6months-based should align kernel development to some other big
projects too.


cheers,
Michele

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> Exactly that, and kernel interfaces going away just to annoy binary
> module providers also hurts third-party OSS modules, such as
> Fujitsu-Siemens's ServerView agents.

in kernel API never was and never will be stable, that's entirely
irrelevant and independent of the proposal at hand.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[David Ranson]

[-- Attachment #1: Type: text/plain, Size: 246 bytes --]

Steven Rostedt wrote:

>udev ;)
>
>http://seclists.org/lists/linux-kernel/2005/Dec/0180.html
>
>
Ahh OK .. I don't use it, so wouldn't have been affected. That's one
userspace interface broken during the series, does anyone have any more?

David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 187 bytes --]

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Steven Rostedt]

On Sat, 2005-12-03 at 16:58 +0000, David Ranson wrote:
> Matthias Andree wrote:
> 
> >>>the model needs finetuning. Right now the biggest pain is the userland
> >>>ABI changes that need new packages; sometimes (often) for no real hard
> >>>reason. Maybe we should just stop doing those bits, they're not in any
> >>>fundamental way blocking general progress (sure there's some code bloat
> >>>due to it, but I guess we'll just have to live with that)
> >>>
> >>>
> Well as a mildly technical 'user' who's been tracking the 2.6 series I
> can't recall having to update _any_ userland packages due to kernel
> changes. An example of this would be helpful.

udev ;)

http://seclists.org/lists/linux-kernel/2005/Dec/0180.html

-- Steve

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[David Ranson]

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

Matthias Andree wrote:

>>>the model needs finetuning. Right now the biggest pain is the userland
>>>ABI changes that need new packages; sometimes (often) for no real hard
>>>reason. Maybe we should just stop doing those bits, they're not in any
>>>fundamental way blocking general progress (sure there's some code bloat
>>>due to it, but I guess we'll just have to live with that)
>>>
>>>
Well as a mildly technical 'user' who's been tracking the 2.6 series I
can't recall having to update _any_ userland packages due to kernel
changes. An example of this would be helpful.

>Exactly that, and kernel interfaces going away just to annoy binary
>module providers also hurts third-party OSS modules, such as
>Fujitsu-Siemens's ServerView agents.
>
>
As many here say. Too bad, we really don't care. Hardware providers
should have the requisite relationships with the distros and target
their management utilities to those. Surely no business using this sort
of high end hardware is running anything other than RHEL or SLES etc ?

>I found myself chasing some genuine bugs in that code to please GCC 4
>(which doesn't tolerate extern blah declarations before static blah
>definitions), and some idiotic breakage when inter_module_get was
>dropped somewhen between 2.6.8 (where the modules compiled just fine)
>and 2.6.13 (where they didn't as functions had been removed).  And
>there's no point arguing about deprecation warnings being around,
>interfaces can be removed between 2.6 and 2.8 but not between
>
>
See above.

>Well, the current model, since it has been in effect, is just "let's
>break the interfaces at will, but let's not hint anyone, so let's always
>
>
Only kernel interfaces. And that's too bad for out of kernel modules.

>Linux is in fact light years away from being "stable", and while it was
>
>
If you want stable you want a distro 'enterprise' version.

>and some early 2.6.X release, moving between 2.6.X and 2.6.Y means
>switching user-space, too, and chances are the new user-space doesn't
>work with the old kernel.
>
>
In my experience this isn't the case.

Cheers
David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 187 bytes --]

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Otavio Salvador]

Matthias Andree <matthias.andree@gmx.de> writes:

>> This could work, but it should be officially announced that e.g. a 
>> userspace running kernel 2.6.15 must work flawlessly with _any_ future 
>> 2.6 kernel.
>
> Right. This effectively means to call the beast 2.8.0 if you feel you
> must break the applications.

Looks like things are leaving clear the need of 2.7 for those
developments. This is my point of view.

-- 
        O T A V I O    S A L V A D O R
---------------------------------------------
 E-mail: otavio@debian.org      UIN: 5906116
 GNU/Linux User: 239058     GPG ID: 49A5F855
 Home Page: http://www.freedom.ind.br/otavio
---------------------------------------------
"Microsoft gives you Windows ... Linux gives
 you the whole house."

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Matthias Andree]

On Sat, 03 Dec 2005, Adrian Bunk wrote:

> Things will change as time passes by, but then there's the possibility 
> to open the next branch and bring the older branch into a security-fixes 
> only mode.
> 
> > If the current model doesn't work as you claim it doesn't, then maybe
> > the model needs finetuning. Right now the biggest pain is the userland
> > ABI changes that need new packages; sometimes (often) for no real hard
> > reason. Maybe we should just stop doing those bits, they're not in any
> > fundamental way blocking general progress (sure there's some code bloat
> > due to it, but I guess we'll just have to live with that).
> 
> IOW, we should e.g. ensure that today's udev will still work flawlessly 
> with kernel 2.6.30 (sic)?

Exactly that, and kernel interfaces going away just to annoy binary
module providers also hurts third-party OSS modules, such as
Fujitsu-Siemens's ServerView agents.

I found myself chasing some genuine bugs in that code to please GCC 4
(which doesn't tolerate extern blah declarations before static blah
definitions), and some idiotic breakage when inter_module_get was
dropped somewhen between 2.6.8 (where the modules compiled just fine)
and 2.6.13 (where they didn't as functions had been removed).  And
there's no point arguing about deprecation warnings being around,
interfaces can be removed between 2.6 and 2.8 but not between
2.6.foo.bar and 2.6.foo.baz. Why not use

#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,8,0)

or similar to automatically disable such interfaces at the given
version?

> This could work, but it should be officially announced that e.g. a 
> userspace running kernel 2.6.15 must work flawlessly with _any_ future 
> 2.6 kernel.

Right. This effectively means to call the beast 2.8.0 if you feel you
must break the applications.

> For how many years do you think we will be able to ensure that this will 
> stay true?

Well, the current model, since it has been in effect, is just "let's
break the interfaces at will, but let's not hint anyone, so let's always
just bump the patchlevel no matter how intrusive the change is", and the
bandaid 2.6.M.N releases cannot cover the fact that the M -> M+1
transition causes breakage.

Effectively, 2.6.X behaves like a bleeding edge development ("everything
changes") kernel such as 2.3 or 2.5; it isn't stable because some code
monkeys claim it is. You cannot declare the can of pea soup in front of
you "open" either.

Linux is in fact light years away from being "stable", and while it was
relatively easy to move forward between 2.4.X releases and even 2.4.X
and some early 2.6.X release, moving between 2.6.X and 2.6.Y means
switching user-space, too, and chances are the new user-space doesn't
work with the old kernel.

--
Matthias Andree

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> > this is a contradiction. You can't eat a cake and have it; either you're
> > really low churn (like existing -stable) or you start adding new
> > features like hardware support. the problem with hardware support is
> > that it's not just a tiny driver update. If involves midlayer updates as
> > well usually, and especially if those midlayers diverge between your
> > stable and mainline, the "backports" are getting increasingly unsafe and
> > hard.
> 
> In the beginning, backporting hardware support is relatively easy, and 
> therefore cherry-picking from mainline 2.6 should be relatively safe.

and then there's reality. At least in my experience as distro kernel
maintainer... you can do this for a few months, but it gets
exponentially more expensive after 4 to 5 months. And "safe" is just not
true. API, midlayer and locking changes in the newer kernels just void
that concept entirely. And then there's the testing dillema; the people
who'd run such a branch are EXACTLY the ones who wouldn't test
prereleases of such branch (and yes 2.4 suffers from this as well). 

I doubt many distros would go for it as well for longer than a few
months, simply because the hardware support and other features are going
to be needed for them.


> Things will change as time passes by, but then there's the possibility 
> to open the next branch and bring the older branch into a security-fixes 
> only mode.

if you end up with 5 such branches it's no longer fun, trust me on that.
Especially if the security fix is in a tricky area or a high flux area,
then it's just not a matter of a simple backport anymore, even knowing
if you're vulnerable or not is going to be a pain. And then there are
the holes that happened to have gone away by later changes... what are
you going to do then... put those changes in? that won't work longer
term.

> 
> > If the current model doesn't work as you claim it doesn't, then maybe
> > the model needs finetuning. Right now the biggest pain is the userland
> > ABI changes that need new packages; sometimes (often) for no real hard
> > reason. Maybe we should just stop doing those bits, they're not in any
> > fundamental way blocking general progress (sure there's some code bloat
> > due to it, but I guess we'll just have to live with that).
> 
> IOW, we should e.g. ensure that today's udev will still work flawlessly 
> with kernel 2.6.30 (sic)?

I'd say yes. It doesn't need to support all new functionality, but at
least what it does today it should be able to do then. If that really
isn't possible maybe udev should be part of the kernel build and per
kernel version.


> This could work, but it should be officially announced that e.g. a 
> userspace running kernel 2.6.15 must work flawlessly with _any_ future 
> 2.6 kernel.

I would argue that this in theory already is the current policy. Now
"any" is pretty wide, but still. Maybe any such changes need to be
scheduled to specific kernel releases only. Eg only do it every 4th or
5th kernel release.

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

On Sat, Dec 03, 2005 at 03:36:38PM +0100, Arjan van de Ven wrote:
> > ase for a stable series.
> > 
> > After 2.6.16, there will be a 2.6.16.y series with the usual stable 
> > rules.
> > 
> > After the release of 2.6.17, this 2.6.16.y series will be continued with 
> > more relaxed rules similar to the rules in kernel 2.4 since the release 
> > of kernel 2.6.0 (e.g. driver updates will be allowed).
> > 
> 
> 
> this is a contradiction. You can't eat a cake and have it; either you're
> really low churn (like existing -stable) or you start adding new
> features like hardware support. the problem with hardware support is
> that it's not just a tiny driver update. If involves midlayer updates as
> well usually, and especially if those midlayers diverge between your
> stable and mainline, the "backports" are getting increasingly unsafe and
> hard.

In the beginning, backporting hardware support is relatively easy, and 
therefore cherry-picking from mainline 2.6 should be relatively safe.

Things will change as time passes by, but then there's the possibility 
to open the next branch and bring the older branch into a security-fixes 
only mode.

> If the current model doesn't work as you claim it doesn't, then maybe
> the model needs finetuning. Right now the biggest pain is the userland
> ABI changes that need new packages; sometimes (often) for no real hard
> reason. Maybe we should just stop doing those bits, they're not in any
> fundamental way blocking general progress (sure there's some code bloat
> due to it, but I guess we'll just have to live with that).

IOW, we should e.g. ensure that today's udev will still work flawlessly 
with kernel 2.6.30 (sic)?

This could work, but it should be officially announced that e.g. a 
userspace running kernel 2.6.15 must work flawlessly with _any_ future 
2.6 kernel.

For how many years do you think we will be able to ensure that this will 
stay true?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Arjan van de Ven]

> ase for a stable series.
> 
> After 2.6.16, there will be a 2.6.16.y series with the usual stable 
> rules.
> 
> After the release of 2.6.17, this 2.6.16.y series will be continued with 
> more relaxed rules similar to the rules in kernel 2.4 since the release 
> of kernel 2.6.0 (e.g. driver updates will be allowed).
> 


this is a contradiction. You can't eat a cake and have it; either you're
really low churn (like existing -stable) or you start adding new
features like hardware support. the problem with hardware support is
that it's not just a tiny driver update. If involves midlayer updates as
well usually, and especially if those midlayers diverge between your
stable and mainline, the "backports" are getting increasingly unsafe and
hard.

If the current model doesn't work as you claim it doesn't, then maybe
the model needs finetuning. Right now the biggest pain is the userland
ABI changes that need new packages; sometimes (often) for no real hard
reason. Maybe we should just stop doing those bits, they're not in any
fundamental way blocking general progress (sure there's some code bloat
due to it, but I guess we'll just have to live with that).

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Ben Collins]

On Sat, 2005-12-03 at 14:56 +0100, Adrian Bunk wrote:
> The current kernel development model is pretty good for people who 
> always want to use or offer their costumers the maximum amount of the 
> latest bugs^Wfeatures without having to resort on additional patches for 
> them.
> 
> Problems of the current development model from a user's point of view 
> are:
> - many regressions in every new release
> - kernel updates often require updates for the kernel-related userspace 
>   (e.g. for udev or the pcmcia tools switch)
> 
> One problem following from this is that people continue to use older 
> kernels with known security holes because the amount of work for kernel 
> upgrades is too high.

What you're suggesting sounds just like going back to the old style of
development where 2.<even>.x is stable, and 2.<odd>.x is development.
You might as well just suggest that after 2.6.16, we fork to 2.7.0, and
2.6.17+ will be stable increments like we always used to do.

You're just munging the version scheme :)

-- 
   Ben Collins <ben.collins@ubuntu.com>
   Developer
   Ubuntu Linux

Re: RFC: Starting a stable kernel series off the 2.6 kernel

[Jesper Juhl]

On 12/3/05, Adrian Bunk <bunk@stusta.de> wrote:
> The current kernel development model is pretty good for people who
> always want to use or offer their costumers the maximum amount of the
> latest bugs^Wfeatures without having to resort on additional patches for
> them.
>
> Problems of the current development model from a user's point of view
> are:
> - many regressions in every new release
> - kernel updates often require updates for the kernel-related userspace
>   (e.g. for udev or the pcmcia tools switch)
>
> One problem following from this is that people continue to use older
> kernels with known security holes because the amount of work for kernel
> upgrades is too high.
>
> These problems follow from the development model.
>
> The latest stable kernel series without these problems is 2.4, but 2.4
> is becoming more and more obsolete and might e.g. lack driver support
> for some recent hardware you want to use.
>
> Since Andrew and Linus do AFAIK not plan to change the development
> model, what about the following for getting a stable kernel series
> without leaving the current development model:
>
>
> Kernel 2.6.16 will be the base for a stable series.
>
[snip]

Why can't this be done by distributors/vendors?

Any vendor is free to branch off at 2.6.<whatever> and then maintain
that indefinately.  Why create yet-another-stable-branch?

In effect you'd be making 2.6.17+ into a 2.7.x tree and 2.6.16.y would
become a 2.6 tree in 2.4.x style, with all the backporting problems
and vendor skew that 2.4.x suffered from.

Personally I don't like this proposal. In my oppinion 2.6 + the
-stable branch as we have it now works well and people who want
userspace & kernel in sync are perfectly free to use vendor kernels
(which is also the recommended thing to do for most users as far as I
know).

Just my 0.02euro.

--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

RFC: Starting a stable kernel series off the 2.6 kernel

[Adrian Bunk]

The current kernel development model is pretty good for people who 
always want to use or offer their costumers the maximum amount of the 
latest bugs^Wfeatures without having to resort on additional patches for 
them.

Problems of the current development model from a user's point of view 
are:
- many regressions in every new release
- kernel updates often require updates for the kernel-related userspace 
  (e.g. for udev or the pcmcia tools switch)

One problem following from this is that people continue to use older 
kernels with known security holes because the amount of work for kernel 
upgrades is too high.

These problems follow from the development model.

The latest stable kernel series without these problems is 2.4, but 2.4 
is becoming more and more obsolete and might e.g. lack driver support 
for some recent hardware you want to use.

Since Andrew and Linus do AFAIK not plan to change the development 
model, what about the following for getting a stable kernel series 
without leaving the current development model:


Kernel 2.6.16 will be the base for a stable series.

After 2.6.16, there will be a 2.6.16.y series with the usual stable 
rules.

After the release of 2.6.17, this 2.6.16.y series will be continued with 
more relaxed rules similar to the rules in kernel 2.4 since the release 
of kernel 2.6.0 (e.g. driver updates will be allowed).


Q:
What is the target audience for this 2.6.16 series?

A:
The target audience are users still using 2.4 (or who'd still use kernel 
2.4 if they weren't forced to upgrade to 2.6 for some reason) who want a 
stable kernel series including security fixes but excluding many 
regressions.
It might also be interesting for distributions that prefer stability 
over always using the latest stuff.


Q:
Does this proposal imply anything for the development between 2.6.15 and 
2.6.16?

A:
In theory not.
In practice, it would be a big advantage if some of the bigger 
changes that might go into 2.6.16 would be postponed to 2.6.17.


Q:
Why not start with the more relaxed rules before the release of 2.6.17?

A:
After 2.6.16.y following the usual stable rules, the kernel should be 
relatively stable and well-tested giving the best possible basis for a 
long-living series.


Q:
How long should this 2.6.16 series be maintained?

A:
Time will tell, but if people use it I'd expect 2 or 3 years.


Q:
Stable API/ABI for external modules?

A:
No.


Q:
Who will maintain this branch?

A:
I could do it, but if someone more experienced wants to do it that would 
be even better.