XFS problem in 2.6.32

XFS problem in 2.6.32

[Stefan Priebe - Profihost AG]

Hi List,

i've massive problems with a machine using 2.6.32 kernel and XFS: I've 
desribed the problem here:
http://www.spinics.net/lists/xfs/msg04984.html

The reply on the XFS mailinglist was:
http://www.spinics.net/lists/xfs/msg04994.html

So who keeps track on which patches needs to get backported or not? And 
who will backport XFS fixes back to 2.6.32?

Thanks!

Please CC me i'm NOT on list.

Regards,
Stefan

Re: XFS problem in 2.6.32

[Theodore Tso]

On Jun 7, 2011, at 8:46 AM, Stefan Priebe - Profihost AG wrote:

> Hi List,
> 
> i've massive problems with a machine using 2.6.32 kernel and XFS: I've desribed the problem here:
> http://www.spinics.net/lists/xfs/msg04984.html
> 
> The reply on the XFS mailinglist was:
> http://www.spinics.net/lists/xfs/msg04994.html
> 
> So who keeps track on which patches needs to get backported or not? And who will backport XFS fixes back to 2.6.32?

An interested kernel developer.   They can become interested because they personally have the time or interest, or because someone pays them to become interested.  Support of the stable kernel series is not something that happens magically, or which is funded by a charity, you know.  That's why some companies pay $$$ for a supported distribution kernel.

-- Ted

Re: XFS problem in 2.6.32

[Stefan Priebe - Profihost AG]

>> So who keeps track on which patches needs to get backported or not? And who will backport XFS fixes back to 2.6.32?
>
> An interested kernel developer.   They can become interested because they personally have the time or interest, or because someone pays them to become interested.  Support of the stable kernel series is not something that happens magically, or which is funded by a charity, you know.  That's why some companies pay $$$ for a supported distribution kernel.
OK so my thought was totally wrong. I thought the longterm stable 
releases will still get bugfixed by SGI or whoever wrote the stuff. 
Sorry for that then. But what is then the idea of a longterm stable?

Stefan

Re: XFS problem in 2.6.32

[david]

On Tue, 7 Jun 2011, Stefan Priebe - Profihost AG wrote:

> Date: Tue, 07 Jun 2011 15:49:36 +0200
>>> So who keeps track on which patches needs to get backported or not? And 
>>> who will backport XFS fixes back to 2.6.32?
>> 
>> An interested kernel developer.   They can become interested because they 
>> personally have the time or interest, or because someone pays them to 
>> become interested.  Support of the stable kernel series is not something 
>> that happens magically, or which is funded by a charity, you know.  That's 
>> why some companies pay $$$ for a supported distribution kernel.

> OK so my thought was totally wrong. I thought the longterm stable releases 
> will still get bugfixed by SGI or whoever wrote the stuff. Sorry for that 
> then. But what is then the idea of a longterm stable?

development and bugfixes are done on the latest kernel, if the problem is 
known to affect old kernels the developers sometimes remember to notify 
the -stable list that this patch is important and needs to be applied to 
older kernels.

whoever the maintainer of the -stable/-longterm tree is (be it an 
individual or a team employeed by some comapny) then looks at the patch 
and considers backporting it (if it's too hard, or to intrusive, they may 
decide not to).

the idea of the lonterm kernels is that organizations need to maintain a 
kernel for a long time due to commitments that they have made (Debian 
doesn't want to change the kernel it ships in a stable version, RedHat 
doesn't want to change the kernel version in a RHEL release, etc), and so 
they publicly announce this so that anyone else wanting to use the same 
kernel version can share in the work (and therefor everyone can benifit 
from each other's work)

David Lang

Re: XFS problem in 2.6.32

[Stefan Priebe - Profihost AG]

> whoever the maintainer of the -stable/-longterm tree is (be it an
> individual or a team employeed by some comapny) then looks at the patch
> and considers backporting it (if it's too hard, or to intrusive, they
> may decide not to).
So i have to contact Greg Kohan from SUSE directly?

> the idea of the lonterm kernels is that organizations need to maintain a
> kernel for a long time due to commitments that they have made (Debian
> doesn't want to change the kernel it ships in a stable version, RedHat
> doesn't want to change the kernel version in a RHEL release, etc), and
> so they publicly announce this so that anyone else wanting to use the
> same kernel version can share in the work (and therefor everyone can
> benifit from each other's work)
That was what i thoght. So a bug like this should get fixed right? 
Otherwise this makes no sense. Sadly Redhat has ported the fix back in 
his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / 
vanilla team.

Stefan

Re: XFS problem in 2.6.32

[david]

On Tue, 7 Jun 2011, Stefan Priebe - Profihost AG wrote:

>> whoever the maintainer of the -stable/-longterm tree is (be it an
>> individual or a team employeed by some comapny) then looks at the patch
>> and considers backporting it (if it's too hard, or to intrusive, they
>> may decide not to).
> So i have to contact Greg Kohan from SUSE directly?

if he's the maintainer of the longterm tree, then yes, you would contact 
him or the stable mailing list.

>> the idea of the lonterm kernels is that organizations need to maintain a
>> kernel for a long time due to commitments that they have made (Debian
>> doesn't want to change the kernel it ships in a stable version, RedHat
>> doesn't want to change the kernel version in a RHEL release, etc), and
>> so they publicly announce this so that anyone else wanting to use the
>> same kernel version can share in the work (and therefor everyone can
>> benifit from each other's work)
> That was what i thoght. So a bug like this should get fixed right? Otherwise 
> this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 
> 2.6.32 kernel but they haven't send the patch to stable / vanilla team.

it depends on how intrusive/disruptive the patch is.

David Lang

Re: XFS problem in 2.6.32

[Theodore Tso]

On Jun 7, 2011, at 5:57 PM, Stefan Priebe - Profihost AG wrote:

>> whoever the maintainer of the -stable/-longterm tree is (be it an
>> individual or a team employeed by some comapny) then looks at the patch
>> and considers backporting it (if it's too hard, or to intrusive, they
>> may decide not to).
> So i have to contact Greg Kohan from SUSE directly?

Only if you can identity a specific patch you want backported to 2.6.32.

It's not the responsibility of the long-term stable tree maintainer to go looking through potentially tens of thousands of commits to find the one that should be backported.  And if the code has changed too much since .2.6.32, and requires detailed reworking before the patch can get integrated into 2.6.32, then a subsystem developer will have to do that work.

> 
>> the idea of the lonterm kernels is that organizations need to maintain a
>> kernel for a long time due to commitments that they have made (Debian
>> doesn't want to change the kernel it ships in a stable version, RedHat
>> doesn't want to change the kernel version in a RHEL release, etc), and
>> so they publicly announce this so that anyone else wanting to use the
>> same kernel version can share in the work (and therefor everyone can
>> benifit from each other's work)
> That was what i thoght. So a bug like this should get fixed right? Otherwise this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / vanilla team.

Not all distributions will participate in the maintenance stable tree.  Red Hat for example is probably worried about people  (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them.  So it doesn't surprise me that they aren't submitting patches to the stable tree.  After all, they would like you to purchase a support contract if you want to get high quality, supported kernel.   Why should they give that work away for free?  After all, their salaried developers have to get paid somehow.  Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.

The thing that you have to understand that this is all a volunteer effort, and a few of your messages sound like you are expecting people to do the work for you.   It doesn't work that way.  If you ask nicely, maybe one of the XFS developers will help you.  But please don't expect free support.  That will just annoy people.

-- Ted

Re: XFS problem in 2.6.32

[Stefan Priebe - Profihost AG]

Hi Ted,

>
> Not all distributions will participate in the maintenance stable tree.  Red Hat for example is probably worried about people  (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them.  So it doesn't surprise me that they aren't submitting patches to the stable tree.  After all, they would like you to purchase a support contract if you want to get high quality, supported kernel.   Why should they give that work away for free?  After all, their salaried developers have to get paid somehow.  Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.
>
> The thing that you have to understand that this is all a volunteer effort, and a few of your messages sound like you are expecting people to do the work for you.   It doesn't work that way.  If you ask nicely, maybe one of the XFS developers will help you.  But please don't expect free support.  That will just annoy people.
i'm sorry this was never my intention. I'm no native speaker so it might 
be that my word selection was not perfect.

I quite understand that alle people are doing this for free or got paid 
by a company who have their own interests.

I just thought it would be important to make it public that there is a 
problem with XFS in the longterm stable kernel and that it would be 
really nice if someone can take a look at it.

Greets Stefan

Re: XFS problem in 2.6.32

[John Kacur]

On Wed, Jun 8, 2011 at 5:28 AM, Theodore Tso <tytso@mit.edu> wrote:
>
> On Jun 7, 2011, at 5:57 PM, Stefan Priebe - Profihost AG wrote:
>
>>> whoever the maintainer of the -stable/-longterm tree is (be it an
>>> individual or a team employeed by some comapny) then looks at the patch
>>> and considers backporting it (if it's too hard, or to intrusive, they
>>> may decide not to).
>> So i have to contact Greg Kohan from SUSE directly?
>
> Only if you can identity a specific patch you want backported to 2.6.32.
>
> It's not the responsibility of the long-term stable tree maintainer to go looking through potentially tens of thousands of commits to find the one that should be backported.  And if the code has changed too much since .2.6.32, and requires detailed reworking before the patch can get integrated into 2.6.32, then a subsystem developer will have to do that work.
>
>>
>>> the idea of the lonterm kernels is that organizations need to maintain a
>>> kernel for a long time due to commitments that they have made (Debian
>>> doesn't want to change the kernel it ships in a stable version, RedHat
>>> doesn't want to change the kernel version in a RHEL release, etc), and
>>> so they publicly announce this so that anyone else wanting to use the
>>> same kernel version can share in the work (and therefor everyone can
>>> benifit from each other's work)
>> That was what i thoght. So a bug like this should get fixed right? Otherwise this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / vanilla team.
>
> Not all distributions will participate in the maintenance stable tree.  Red Hat for example is probably worried about people  (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them.  So it doesn't surprise me that they aren't submitting patches to the stable tree.  After all, they would like you to purchase a support contract if you want to get high quality, supported kernel.   Why should they give that work away for free?  After all, their salaried developers have to get paid somehow.  Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.
>
> The thing that you have to understand that this is all a volunteer effort, and a few of your messages sound like you are expecting people to do the work for you.   It doesn't work that way.  If you ask nicely, maybe one of the XFS developers will help you.  But please don't expect free support.  That will just annoy people.
>

Ok, I don't speak for my company, and your point about not expecting
people to do this work for you is valid, however I don't see why you
need to take potshots at Red Hat, - they are quite active in the
stable effort.

wget http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.38.8
grep Author ChangeLog-2.6.38.8 | grep -i redhat | wc -l
9

grep Author ChangeLog-2.6.38.8 | grep -i suse | wc -l
10

grep Author ChangeLog-2.6.38.8 | grep -i canonical | wc -l
5

I'm not even claiming that these are typical stats, but as just a
quick check on your statement, the contributions for one stable
release are in the same ballpark as everyone else.
Don't know if lwn has stable stats.

Thanks

John

Re: XFS problem in 2.6.32

[Alan Cox]

> > That was what i thoght. So a bug like this should get fixed right? Otherwise this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / vanilla team.

It's often not that simple. As vendor trees diverge you may find it's not
a case of 'sending patch X upstream' because the patches you need may
well depend upon other things in the vendor tree not all of which will be
in the 'stable' tree. So it's a base at best.

> Not all distributions will participate in the maintenance stable tree.  Red Hat for example is probably worried about people  (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them.

Judging from the commit log that isn't the case.

Besides which I hear a rumour that oracle employees have learned how to
download files from other websites and look at the content so it won't
make any odds 8)

Alan

Re: XFS problem in 2.6.32

[Steven Rostedt]

On Tue, Jun 07, 2011 at 11:28:59PM -0400, Theodore Tso wrote:

Ted, you need a new MUA, as it reads horribly in mutt.

> 
> Not all distributions will participate in the maintenance stable tree.  Red Hat for example is probably worried about people  (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them.  So it doesn't surprise me that they aren't submitting patches to the stable tree.  After all, they would like you to purchase a support contract if you want to get high quality, supported kernel.   Why should they give that work away for free?  After all, their salaried developers have to get paid somehow.  Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.
> 

As a Red Hat employee, I must speak against this. I have *never* been
told to keep something from stable. In fact, I've always been encouraged
to push to stable. But it is usually the individual engineer's
responsibility to get a patch into stable. If something was missed, it
was more the fault of that individual engineer that made the fix than
Red Hat.

Ask Greg, I'm constantly sending stable patches to him.

-- Steve

Re: XFS problem in 2.6.32

[Ted Ts'o]

On Wed, Jun 08, 2011 at 10:00:39AM +0200, John Kacur wrote:

> Ok, I don't speak for my company, and your point about not expecting
> people to do this work for you is valid, however I don't see why you
> need to take potshots at Red Hat

It wasn't a potshot; if it is true, it is a completely rational
economic argument that is completely within the bounds of the
requirements of the GPL, and with LKML community standards.

The reason why I say it is because of (a) http://lwn.net/Articles/430098/,
and (b) a few months ago, when I quietly floated starting a new
long-term stable kernel series that a number of companies would
maintain cooperatively, I was told, privately, that such a proposal
would not likely be received positively by Red Hat management because
of the reasons behind the policy instituted by (a).

Which is fine, I don't consider that a potshot, no more than I
consider the fact that IBM forked the OpenOffice before it was
relicensed to the LGPL and made changes which they didn't give back,
and seems to be actively supporting Oracle's attempt to get Apache to
adopt OpenOffice.org inspite of the currently healthy LibreOffice
LGPL3 branch which already has an active community to be a potshot.

Both are completely legal things set by the ground rules of the
copyright licenses involvled, and someone no less than Linus has said
that the ability to fork is healthy because it keeps people honest.

> - they are quite active in the stable effort.
> 
> wget http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.38.8
> grep Author ChangeLog-2.6.38.8 | grep -i redhat | wc -l
> 9
> 
> grep Author ChangeLog-2.6.38.8 | grep -i suse | wc -l
> 10
> 
> grep Author ChangeLog-2.6.38.8 | grep -i canonical | wc -l
> 5
> 
> I'm not even claiming that these are typical stats, but as just a
> quick check on your statement, the contributions for one stable
> release are in the same ballpark as everyone else.

Nah, that just means that commits which are labelled with "CC:
stable@vger.kernel.org" are automatically accepted into stable kernel
series.

If you can point efforts where painful backports of ext4 and xfs bug
fixes into RHEL 6.x are making it back into 2.6.32.y, even though in
some cases it takes tens of hours of engineering and QA efforts, we
can talk.  But please note that I wasn't calling out Red Hat as being
bad or evil by doing what they are doing; it is completely
economically rational and allowed by the GPL rules for them to be
doing what they are doing.  (Just as what Android has been doing with
their constant forward porting of the Wakelocks API is completely
within the GPL rules.)

	       		     		   - Ted

Re: XFS problem in 2.6.32

[Mike Snitzer]

Ted,

On Wed, Jun 8, 2011 at 9:38 AM, Ted Ts'o <tytso@mit.edu> wrote:
> On Wed, Jun 08, 2011 at 10:00:39AM +0200, John Kacur wrote:
>
>> Ok, I don't speak for my company, and your point about not expecting
>> people to do this work for you is valid, however I don't see why you
>> need to take potshots at Red Hat
>
> It wasn't a potshot; if it is true, it is a completely rational
> economic argument that is completely within the bounds of the
> requirements of the GPL, and with LKML community standards.
>
> The reason why I say it is because of (a) http://lwn.net/Articles/430098/,
> and (b) a few months ago, when I quietly floated starting a new
> long-term stable kernel series that a number of companies would
> maintain cooperatively, I was told, privately, that such a proposal
> would not likely be received positively by Red Hat management because
> of the reasons behind the policy instituted by (a).

OK, you seem to think you have it all figured out and that your Red
Hat contact knows everything there is to the Red Hat kernel
engineering development process.

However, it doesn't jive with what I know or how I (and my coworkers)
work on a daily basis.

Since we're referencing lwn.net articles please have a look at these
comments that I previously made on this subject of RHEL and stable@:
http://lwn.net/Articles/431317/
http://lwn.net/Articles/431420/

<snip irrelevant tangent about GPL>

>> I'm not even claiming that these are typical stats, but as just a
>> quick check on your statement, the contributions for one stable
>> release are in the same ballpark as everyone else.
>
> Nah, that just means that commits which are labelled with "CC:
> stable@vger.kernel.org" are automatically accepted into stable kernel
> series.
>
> If you can point efforts where painful backports of ext4 and xfs bug
> fixes into RHEL 6.x are making it back into 2.6.32.y, even though in
> some cases it takes tens of hours of engineering and QA efforts, we
> can talk.  But please note that I wasn't calling out Red Hat as being
> bad or evil by doing what they are doing; it is completely
> economically rational and allowed by the GPL rules for them to be
> doing what they are doing.  (Just as what Android has been doing with
> their constant forward porting of the Wakelocks API is completely
> within the GPL rules.)

And what _exactly_ is Red Hat (not) doing?  Red Hat isn't going crazy
backporting its upstream > 2.6.32 fixes to 2.6.32.y even when Red Hat
doesn't consume 2.6.32.y?  *gasp*

Mike

Re: XFS problem in 2.6.32

[Ted Ts'o]

On Wed, Jun 08, 2011 at 10:16:59AM -0400, Mike Snitzer wrote:
> 
> And what _exactly_ is Red Hat (not) doing?  Red Hat isn't going crazy
> backporting its upstream > 2.6.32 fixes to 2.6.32.y even when Red Hat
> doesn't consume 2.6.32.y?  *gasp*

Well, the original poster was expecting that (unspecified people)
would be doing this regularly (in fact he was complaining about how an
XFS bug fixed in RHEL wasn't fixed in 2.6.32.y).  I was explaining how
it wasn't happening, and it was perfectly acceptable for that to be
the case.

So I was actually *defending* Red Hat....

							- Ted

Re: XFS problem in 2.6.32

[Christoph Hellwig]

On Wed, Jun 08, 2011 at 02:50:16PM -0400, Ted Ts'o wrote:
> On Wed, Jun 08, 2011 at 10:16:59AM -0400, Mike Snitzer wrote:
> > 
> > And what _exactly_ is Red Hat (not) doing?  Red Hat isn't going crazy
> > backporting its upstream > 2.6.32 fixes to 2.6.32.y even when Red Hat
> > doesn't consume 2.6.32.y?  *gasp*
> 
> Well, the original poster was expecting that (unspecified people)
> would be doing this regularly (in fact he was complaining about how an
> XFS bug fixed in RHEL wasn't fixed in 2.6.32.y).  I was explaining how
> it wasn't happening, and it was perfectly acceptable for that to be
> the case.
> 
> So I was actually *defending* Red Hat....

Btw, Redhat has a completely different XFS codebase than mainline
2.6.32, here's the diffstat summary between 2.6.32 and rhel6.1:

 96 files changed, 5083 insertions(+), 5211 deletions(-)

Which roughly equals the diff between 2.6.32 and 2.6.34.  So there's
a fairly large chance things simply won't apply as-is anyway.  The same
is also true for ext4, btrfs and nfs.

Re: XFS problem in 2.6.32

[Dave Chinner]

On Wed, Jun 08, 2011 at 09:30:37AM -0400, Steven Rostedt wrote:
> On Tue, Jun 07, 2011 at 11:28:59PM -0400, Theodore Tso wrote:
> 
> Ted, you need a new MUA, as it reads horribly in mutt.

Seconded. ;)

> > Not all distributions will participate in the maintenance stable
> > tree.  Red Hat for example is probably worried about people
> > (specifically, Oracle) taking their kernel expertise "for free"
> > and bidding it against them.  So it doesn't surprise me that
> > they aren't submitting patches to the stable tree.  After all,
> > they would like you to purchase a support contract if you want
> > to get high quality, supported kernel.   Why should they give
> > that work away for free?  After all, their salaried developers
> > have to get paid somehow.  Others will contribute work in the
> > hopes that other people will also contribute fixes back, but of
> > course nothing forces Red Hat to do this.
> 
> As a Red Hat employee, I must speak against this. I have *never* been
> told to keep something from stable. In fact, I've always been encouraged
> to push to stable. But it is usually the individual engineer's
> responsibility to get a patch into stable. If something was missed, it
> was more the fault of that individual engineer that made the fix than
> Red Hat.

If you want someone to blame, then point the fingers at me, not
RedHat or RedHat's processes - RedHat is not involved in the
processes and decisions as to what fixes
get backported into the community stable trees. How that is done
is mainly dictated by available resources, which are generally
scarce. We push bug fixes into the lastest kernel release, and if
known to be needed for stable series they get pushed back via the
stable queues.

This particular case is clearly an oversight - when we fixed the
problem I pushed it into the current stable release, but forgot
that we'd also backported fixes to .32 that meant we also needed to
backport this fix to .32 as well.

If you look at a 2.6.32 kernel the fix is not needed for backport,
but somewhere in the 2.6.32.y series, other fixes were backported
which introduce the bug that make this fix necessary. That history
is not in the mainline git tree and so it's really quite easy to
make such a mistake.  So the real cause of this oversight is the
fact that there are simply too many disjoint community trees and
that makes it difficult to determine what fixes need to go where.

As a result of the difficulty tracking stable trees, I don't even
bother trying. Instead I use bug reports as triggers for "we need to
backport this fix" to a stable kernel because I do not have the time
to waste backporting fixes for bugs no-one is actually hitting...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS problem in 2.6.32

[Stefan Priebe - Profihost AG]

> If you want someone to blame, then point the fingers at me, not
> RedHat or RedHat's processes - RedHat is not involved in the
> processes and decisions as to what fixes
> get backported into the community stable trees. How that is done
> is mainly dictated by available resources, which are generally
> scarce. We push bug fixes into the lastest kernel release, and if
> known to be needed for stable series they get pushed back via the
> stable queues.
I'm really sorry that my post goes that way. It was never my intention 
to blame anybody for anything.

I just didn't know the things work and was just wondering that Redhat 
had a fix which wasn't pushed back to vanilla 2.6.32. But i also didn't 
know that the xfs codebase is totally different due to other backports.

I'm sorry.

Stefan