Re: regression: 100% io-wait with 2.6.24-rcX

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

----- Original Message ----
> From: Alasdair G Kergon <agk@redhat.com>
> To: Martin Knoblauch <spamtrap@knobisoft.de>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>; Mel Gorman <mel@csn.ul.ie>; Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; James.Bottomley@steeleye.com; Jens Axboe <jens.axboe@oracle.com>; Milan Broz <mbroz@redhat.com>; Neil Brown <neilb@suse.de>
> Sent: Tuesday, January 22, 2008 3:39:33 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> On Fri, Jan 18, 2008 at 11:01:11AM -0800, Martin Knoblauch wrote:
> >  At least, rc1-rc5 have shown that the CCISS system can do well. Now
> > the question is which part of the system does not cope well with the
> > larger IO sizes? Is it the CCISS controller, LVM or both. I am
> open
> 
 to
> > suggestions on how to debug that. 
> 
> What is your LVM device configuration?
>   E.g. 'dmsetup table' and 'dmsetup info -c' output.
> Some configurations lead to large IOs getting split up on the
> way
> 
 through
> device-mapper.
>
Hi Alastair,

 here is the output, the filesystem in question is on LogVol02:

  [root@lpsdm52 ~]# dmsetup table
VolGroup00-LogVol02: 0 350945280 linear 104:2 67109248
VolGroup00-LogVol01: 0 8388608 linear 104:2 418054528
VolGroup00-LogVol00: 0 67108864 linear 104:2 384
[root@lpsdm52 ~]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
VolGroup00-LogVol02 253   1 L--w    1    1      0 LVM-IV4PeE8cdxA3piC1qk79GY9PE9OC4OgmOZ4OzOgGQIdF3qDx6fJmlZukXXLIy39R
VolGroup00-LogVol01 253   2 L--w    1    1      0 LVM-IV4PeE8cdxA3piC1qk79GY9PE9OC4Ogmfn2CcAd2Fh7i48twe8PZc2XK5bSOe1Fq
VolGroup00-LogVol00 253   0 L--w    1    1      0 LVM-IV4PeE8cdxA3piC1qk79GY9PE9OC4OgmfYjxQKFP3zw2fGsezJN7ypSrfmP7oSvE

> See if these patches make any difference:
> 
  http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/
> 
>     dm-md-merge_bvec_fn-with-separate-bdev-and-sector.patch
>     dm-introduce-merge_bvec_fn.patch
>     dm-linear-add-merge.patch
>     dm-table-remove-merge_bvec-sector-restriction.patch
>  

 thanks for the suggestion. Are they supposed to apply to mainline?

Cheers
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Alasdair G Kergon]

On Tue, Jan 22, 2008 at 07:25:15AM -0800, Martin Knoblauch wrote:
>   [root@lpsdm52 ~]# dmsetup table
> VolGroup00-LogVol02: 0 350945280 linear 104:2 67109248
> VolGroup00-LogVol01: 0 8388608 linear 104:2 418054528
> VolGroup00-LogVol00: 0 67108864 linear 104:2 384

The IO should pass straight through simple linear targets like that without
needing to get broken up, so I wouldn't expect those patches to make any
difference in this particular case.

Alasdair
-- 
agk@redhat.com

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Alasdair G Kergon <agk@redhat.com>
> To: Martin Knoblauch <spamtrap@knobisoft.de>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>; Mel Gorman <mel@csn.ul.ie>; Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; James.Bottomley@steeleye.com; Jens Axboe <jens.axboe@oracle.com>; Milan Broz <mbroz@redhat.com>; Neil Brown <neilb@suse.de>
> Sent: Wednesday, January 23, 2008 12:40:52 AM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> On Tue, Jan 22, 2008 at 07:25:15AM -0800, Martin Knoblauch wrote:
> >   [root@lpsdm52 ~]# dmsetup table
> > VolGroup00-LogVol02: 0 350945280 linear 104:2 67109248
> > VolGroup00-LogVol01: 0 8388608 linear 104:2 418054528
> > VolGroup00-LogVol00: 0 67108864 linear 104:2 384
> 
> The IO should pass straight through simple linear targets like
> that without needing to get broken up, so I wouldn't expect those patches to
> make any difference in this particular case.
> 

Alasdair,

 LVM/DM are off the hook :-) I converted one box to direct using partitions and the performance is the same disappointment as with LVM/DM. Thanks anyway for looking at my problem.

 I will move the discussion now to a new thread, targetting CCISS directly. 

Cheers
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Alasdair G Kergon <agk@redhat.com>
> To: Martin Knoblauch <spamtrap@knobisoft.de>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>; Mel Gorman <mel@csn.ul.ie>; Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; James.Bottomley@steeleye.com; Jens Axboe <jens.axboe@oracle.com>; Milan Broz <mbroz@redhat.com>; Neil Brown <neilb@suse.de>
> Sent: Tuesday, January 22, 2008 3:39:33 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> 
> See if these patches make any difference:
> 
  http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/
> 
>     dm-md-merge_bvec_fn-with-separate-bdev-and-sector.patch
>     dm-introduce-merge_bvec_fn.patch
>     dm-linear-add-merge.patch
>     dm-table-remove-merge_bvec-sector-restriction.patch
>  


 nope. Exactely the same poor results. To rule out LVM/DM I really have to see what happens if I setup a system with filesystems directly on partitions. Might take some time though.

Cheers
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Alasdair G Kergon]

On Fri, Jan 18, 2008 at 11:01:11AM -0800, Martin Knoblauch wrote:
>  At least, rc1-rc5 have shown that the CCISS system can do well. Now
> the question is which part of the system does not cope well with the
> larger IO sizes? Is it the CCISS controller, LVM or both. I am open to
> suggestions on how to debug that. 

What is your LVM device configuration?
  E.g. 'dmsetup table' and 'dmsetup info -c' output.
Some configurations lead to large IOs getting split up on the way through
device-mapper.

See if these patches make any difference:
  http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/

    dm-md-merge_bvec_fn-with-separate-bdev-and-sector.patch
    dm-introduce-merge_bvec_fn.patch
    dm-linear-add-merge.patch
    dm-table-remove-merge_bvec-sector-restriction.patch
 
Alasdair
-- 
agk@redhat.com

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Mike Snitzer <snitzer@gmail.com>
> To: Linus Torvalds <torvalds@linux-foundation.org>
> Cc: Mel Gorman <mel@csn.ul.ie>; Martin Knoblauch <spamtrap@knobisoft.de>; Fengguang Wu <wfg@mail.ustc.edu.cn>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; James.Bottomley@steeleye.com
> Sent: Friday, January 18, 2008 11:47:02 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> > I can fire up 2.6.24-rc8 in short order to see if things are vastly
> > improved (as Martin seems to indicate that he is happy with
> > AACRAID on 2.6.24-rc8).  Although even Martin's AACRAID
> > numbers from 2.6.19.2
> 
 > are still quite good (relative to mine).  Martin can you share any tuning
> > you may have done to get AACRAID to where it is for you right now?
Mike,

 I have always been happy with the AACRAID box compared to the CCISS system. Even with the "regression" in 2.6.24-rc1..rc5 it was more than acceptable to me. For me the differences between 2.6.19  and 2.6.24-rc8 on the AACRAID setup are:

- 11% (single stream) to 25% (dual/triple stream) regression in DIO. Something I do not care much about. I just measure it for reference.
+ the very nice behaviour when writing to different targets (mix3), which I attribute to Peter's per-dbi stuff.

 And until -rc6 I was extremely pleased with the cool speedup I saw on my CCISS boxes. This would have been the next "production" kernel for me. But lets discuss this under a seperate topic. It has nothing to do with the original wait-io issue.

 Oh, before I forget. There has been no tuning for the AACRAID. The system is an IBM x3650 with built in AACRAID and battery backed write cache. The disks are 6x142GB/15krpm in a RAID5 setup. I see one big difference between your an my tests. I do 1MB writes to simulate the behaviour of the real applications, while yours seem to be much smaller.
 
Cheers
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Mike Snitzer]

On Jan 18, 2008 3:00 PM, Mike Snitzer <snitzer@gmail.com> wrote:
>
> On Jan 18, 2008 12:46 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >
> >
> > On Fri, 18 Jan 2008, Mel Gorman wrote:
> > >
> > > Right, and this is consistent with other complaints about the PFN of the
> > > page mattering to some hardware.
> >
> > I don't think it's actually the PFN per se.
> >
> > I think it's simply that some controllers (quite probably affected by both
> > driver and hardware limits) have some subtle interactions with the size of
> > the IO commands.
> >
> > For example, let's say that you have a controller that has some limit X on
> > the size of IO in flight (whether due to hardware or driver issues doesn't
> > really matter) in addition to a limit on the size of the scatter-gather
> > size. They all tend to have limits, and they differ.
> >
> > Now, the PFN doesn't matter per se, but the allocation pattern definitely
> > matters for whether the IO's are physically contiguous, and thus matters
> > for the size of the scatter-gather thing.
> >
> > Now, generally the rule-of-thumb is that you want big commands, so
> > physical merging is good for you, but I could well imagine that the IO
> > limits interact, and end up hurting each other. Let's say that a better
> > allocation order allows for bigger contiguous physical areas, and thus
> > fewer scatter-gather entries.
> >
> > What does that result in? The obvious answer is
> >
> >   "Better performance obviously, because the controller needs to do fewer
> >    scatter-gather lookups, and the requests are bigger, because there are
> >    fewer IO's that hit scatter-gather limits!"
> >
> > Agreed?
> >
> > Except maybe the *real* answer for some controllers end up being
> >
> >   "Worse performance, because individual commands grow because they don't
> >    hit the per-command limits, but now we hit the global size-in-flight
> >    limits and have many fewer of these good commands in flight. And while
> >    the commands are larger, it means that there are fewer outstanding
> >    commands, which can mean that the disk cannot scheduling things
> >    as well, or makes high latency of command generation by the controller
> >    much more visible because there aren't enough concurrent requests
> >    queued up to hide it"
> >
> > Is this the reason? I have no idea. But somebody who knows the AACRAID
> > hardware and driver limits might think about interactions like that.
> > Sometimes you actually might want to have smaller individual commands if
> > there is some other limit that means that it can be more advantageous to
> > have many small requests over a few big onees.
> >
> > RAID might well make it worse. Maybe small requests work better because
> > they are simpler to schedule because they only hit one disk (eg if you
> > have simple striping)! So that's another reason why one *large* request
> > may actually be slower than two requests half the size, even if it's
> > against the "normal rule".
> >
> > And it may be that that AACRAID box takes a big hit on DIO exactly because
> > DIO has been optimized almost purely for making one command as big as
> > possible.
> >
> > Just a theory.
>
> Oddly enough, I'm seeing the opposite here with 2.6.22.16 w/ AACRAID
> configured with 5 LUNS (each 2disk HW RAID0, 1024k stripesz).  That
> is, with dd the avgrqsiz (from iostat) shows DIO to be ~130k whereas
> non-DIO is a mere ~13k! (NOTE: with aacraid, max_hw_sectors_kb=192)
...
> I can fire up 2.6.24-rc8 in short order to see if things are vastly
> improved (as Martin seems to indicate that he is happy with AACRAID on
> 2.6.24-rc8).  Although even Martin's AACRAID numbers from 2.6.19.2 are
> still quite good (relative to mine).  Martin can you share any tuning
> you may have done to get AACRAID to where it is for you right now?

I can confirm 2.6.24-rc8 behaves like Martin has posted for the
AACRAID.  Slower DIO with smaller avgreqsiz.  Much faster buffered IO
(for my config anyway) with a much larger avgreqsiz (180K).

I have no idea why 2.6.22.16's request size on non-DIO is _so_ small...

Mike

Re: regression: 100% io-wait with 2.6.24-rcX

[Mike Snitzer]

On Jan 18, 2008 12:46 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Fri, 18 Jan 2008, Mel Gorman wrote:
> >
> > Right, and this is consistent with other complaints about the PFN of the
> > page mattering to some hardware.
>
> I don't think it's actually the PFN per se.
>
> I think it's simply that some controllers (quite probably affected by both
> driver and hardware limits) have some subtle interactions with the size of
> the IO commands.
>
> For example, let's say that you have a controller that has some limit X on
> the size of IO in flight (whether due to hardware or driver issues doesn't
> really matter) in addition to a limit on the size of the scatter-gather
> size. They all tend to have limits, and they differ.
>
> Now, the PFN doesn't matter per se, but the allocation pattern definitely
> matters for whether the IO's are physically contiguous, and thus matters
> for the size of the scatter-gather thing.
>
> Now, generally the rule-of-thumb is that you want big commands, so
> physical merging is good for you, but I could well imagine that the IO
> limits interact, and end up hurting each other. Let's say that a better
> allocation order allows for bigger contiguous physical areas, and thus
> fewer scatter-gather entries.
>
> What does that result in? The obvious answer is
>
>   "Better performance obviously, because the controller needs to do fewer
>    scatter-gather lookups, and the requests are bigger, because there are
>    fewer IO's that hit scatter-gather limits!"
>
> Agreed?
>
> Except maybe the *real* answer for some controllers end up being
>
>   "Worse performance, because individual commands grow because they don't
>    hit the per-command limits, but now we hit the global size-in-flight
>    limits and have many fewer of these good commands in flight. And while
>    the commands are larger, it means that there are fewer outstanding
>    commands, which can mean that the disk cannot scheduling things
>    as well, or makes high latency of command generation by the controller
>    much more visible because there aren't enough concurrent requests
>    queued up to hide it"
>
> Is this the reason? I have no idea. But somebody who knows the AACRAID
> hardware and driver limits might think about interactions like that.
> Sometimes you actually might want to have smaller individual commands if
> there is some other limit that means that it can be more advantageous to
> have many small requests over a few big onees.
>
> RAID might well make it worse. Maybe small requests work better because
> they are simpler to schedule because they only hit one disk (eg if you
> have simple striping)! So that's another reason why one *large* request
> may actually be slower than two requests half the size, even if it's
> against the "normal rule".
>
> And it may be that that AACRAID box takes a big hit on DIO exactly because
> DIO has been optimized almost purely for making one command as big as
> possible.
>
> Just a theory.

Oddly enough, I'm seeing the opposite here with 2.6.22.16 w/ AACRAID
configured with 5 LUNS (each 2disk HW RAID0, 1024k stripesz).  That
is, with dd the avgrqsiz (from iostat) shows DIO to be ~130k whereas
non-DIO is a mere ~13k! (NOTE: with aacraid, max_hw_sectors_kb=192)

DIO cmdline:  dd if=/dev/zero of=/dev/sdX bs=8192k count=1k
non-DIO cmdline: dd if=/dev/zero of=/dev/sdX bs=8192k count=1k

DIO is ~80MB/s on all 5 LUNs for a total of ~400MB/s
non-DIO is only ~12MB on all 5 LUNs for a mere ~70MB/s aggregate
(deadline w/ nr_requests=32)

Calls into question the theory of small requests being beneficial for
AACRAID.  Martin, what are you seeing for the avg request size when
you're conducting your AACRAID tests?

I can fire up 2.6.24-rc8 in short order to see if things are vastly
improved (as Martin seems to indicate that he is happy with AACRAID on
2.6.24-rc8).  Although even Martin's AACRAID numbers from 2.6.19.2 are
still quite good (relative to mine).  Martin can you share any tuning
you may have done to get AACRAID to where it is for you right now?

regards,
Mike

Re: regression: 100% io-wait with 2.6.24-rcX

[Linus Torvalds]

On Fri, 18 Jan 2008, Martin Knoblauch wrote:
>
>  just to make one thing clear - I am not so much concerned about the
> performance of AACRAID. It is OK with or without Mel's patch. It is
> better with Mel's patch. The regression in DIO compared to 2.6.19.2 is
> completely independent of Mel's stuff.
> 
>  What interests me much more is the behaviour of the CCISS+LVM based
> system. Here I see a huge benefit of reverting Mel's patch.

Ok, I just got your usage cases confused.

The argument stays the same: some controllers/drivers may have subtle 
behavioural differences that come from the IO limits themselves.

So it wasn't AACRAID, it was CCISS+LVM. The argument is the same: it may 
well be that the *bigger* IO sizes are actually what hurts, even if the 
conventional wisdom is traditionally that bigger submissions are better.

>  At least, rc1-rc5 have shown that the CCISS system can do well. Now
> the question is which part of the system does not cope well with the
> larger IO sizes? Is it the CCISS controller, LVM or both. I am open to
> suggestions on how to debug that. 

I think you need to ask the MD/DM people for suggestions.. Who aren't cc'd 
here.

		Linus

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

--- Linus Torvalds <torvalds@linux-foundation.org> wrote:

> 
> 
> On Fri, 18 Jan 2008, Mel Gorman wrote:
> > 
> > Right, and this is consistent with other complaints about the PFN
> > of the page mattering to some hardware.
> 
> I don't think it's actually the PFN per se.
> 
> I think it's simply that some controllers (quite probably affected by
> both  driver and hardware limits) have some subtle interactions with
> the size of  the IO commands.
> 
> For example, let's say that you have a controller that has some limit
> X on  the size of IO in flight (whether due to hardware or driver
> issues doesn't  really matter) in addition to a limit on the size
> of the scatter-gather  size. They all tend to have limits, and
> they differ.
> 
> Now, the PFN doesn't matter per se, but the allocation pattern
> definitely  matters for whether the IO's are physically
> contiguous, and thus matters  for the size of the scatter-gather
> thing.
> 
> Now, generally the rule-of-thumb is that you want big commands, so 
> physical merging is good for you, but I could well imagine that the
> IO  limits interact, and end up hurting each other. Let's say that a
> better  allocation order allows for bigger contiguous physical areas,
> and thus  fewer scatter-gather entries.
> 
> What does that result in? The obvious answer is
> 
>   "Better performance obviously, because the controller needs to do
> fewer scatter-gather lookups, and the requests are bigger, because
> there are     fewer IO's that hit scatter-gather limits!"
> 
> Agreed?
> 
> Except maybe the *real* answer for some controllers end up being
> 
>   "Worse performance, because individual commands grow because they
> don't  hit the per-command limits, but now we hit the global
> size-in-flight limits and have many fewer of these good commands in
> flight. And while the commands are larger, it means that there
> are fewer outstanding commands, which can mean that the disk
> cannot scheduling things as well, or makes high latency of command
> generation by the controller much more visible because there aren't
> enough concurrent requests queued up to hide it"
> 
> Is this the reason? I have no idea. But somebody who knows the
> AACRAID hardware and driver limits might think about interactions
> like that. Sometimes you actually might want to have smaller 
> individual commands if there is some other limit that means that
> it can be more advantageous to have many small requests over a
> few big onees.
> 
> RAID might well make it worse. Maybe small requests work better
> because they are simpler to schedule because they only hit one
> disk (eg if you have simple striping)! So that's another reason
> why one *large* request may actually be slower than two requests
> half the size, even if it's against the "normal rule".
> 
> And it may be that that AACRAID box takes a big hit on DIO
> exactly because DIO has been optimized almost purely for making
> one command as big as possible.
> 
> Just a theory.
> 
> 		Linus

 just to make one thing clear - I am not so much concerned about the
performance of AACRAID. It is OK with or without Mel's patch. It is
better with Mel's patch. The regression in DIO compared to 2.6.19.2 is
completely independent of Mel's stuff.

 What interests me much more is the behaviour of the CCISS+LVM based
system. Here I see a huge benefit of reverting Mel's patch.

 I dirtied the system after reboot as Mel suggested (24 parallel kernel
build) and repeated the tests. The dirtying did not make any
difference. Here are the results:

Test      -rc8    -rc8-without-Mels-Patch
dd1       57      94
dd1-dir   87      86
dd2       2x8.5   2x45
dd2-dir   2x43    2x43
dd3       3x7     3x30
dd3-dir   3x28.5  3x28.5
mix3      59,2x25 98,2x24

 The big IO size with Mel's patch really has a devastating effect on
the parallel write. Nowhere near the value one would expect, while the
numbers are perfect without Mel's patch as in rc1-rc5. To bad I did not
see this earlier. Maybe we could have found a solution for .24.

 At least, rc1-rc5 have shown that the CCISS system can do well. Now
the question is which part of the system does not cope well with the
larger IO sizes? Is it the CCISS controller, LVM or both. I am open to
suggestions on how to debug that. 

Cheers
Martin

------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

Re: regression: 100% io-wait with 2.6.24-rcX

[Linus Torvalds]

On Fri, 18 Jan 2008, Mel Gorman wrote:
> 
> Right, and this is consistent with other complaints about the PFN of the
> page mattering to some hardware.

I don't think it's actually the PFN per se.

I think it's simply that some controllers (quite probably affected by both 
driver and hardware limits) have some subtle interactions with the size of 
the IO commands.

For example, let's say that you have a controller that has some limit X on 
the size of IO in flight (whether due to hardware or driver issues doesn't 
really matter) in addition to a limit on the size of the scatter-gather 
size. They all tend to have limits, and they differ.

Now, the PFN doesn't matter per se, but the allocation pattern definitely 
matters for whether the IO's are physically contiguous, and thus matters 
for the size of the scatter-gather thing.

Now, generally the rule-of-thumb is that you want big commands, so 
physical merging is good for you, but I could well imagine that the IO 
limits interact, and end up hurting each other. Let's say that a better 
allocation order allows for bigger contiguous physical areas, and thus 
fewer scatter-gather entries.

What does that result in? The obvious answer is

  "Better performance obviously, because the controller needs to do fewer 
   scatter-gather lookups, and the requests are bigger, because there are 
   fewer IO's that hit scatter-gather limits!"

Agreed?

Except maybe the *real* answer for some controllers end up being

  "Worse performance, because individual commands grow because they don't 
   hit the per-command limits, but now we hit the global size-in-flight 
   limits and have many fewer of these good commands in flight. And while 
   the commands are larger, it means that there are fewer outstanding 
   commands, which can mean that the disk cannot scheduling things 
   as well, or makes high latency of command generation by the controller 
   much more visible because there aren't enough concurrent requests 
   queued up to hide it"

Is this the reason? I have no idea. But somebody who knows the AACRAID 
hardware and driver limits might think about interactions like that. 
Sometimes you actually might want to have smaller individual commands if 
there is some other limit that means that it can be more advantageous to 
have many small requests over a few big onees.

RAID might well make it worse. Maybe small requests work better because 
they are simpler to schedule because they only hit one disk (eg if you 
have simple striping)! So that's another reason why one *large* request 
may actually be slower than two requests half the size, even if it's 
against the "normal rule".

And it may be that that AACRAID box takes a big hit on DIO exactly because 
DIO has been optimized almost purely for making one command as big as 
possible.

Just a theory.

		Linus

Re: regression: 100% io-wait with 2.6.24-rcX

[Mel Gorman]

On (18/01/08 00:19), Martin Knoblauch didst pronounce:
> > > The effect  is  defintely  depending on  the  IO  hardware.
> > > performed the same tests
> > > on a different box with an AACRAID controller and there things
> > > look different.
> > 
> > I take it different also means it does not show this odd performance
> > behaviour and is similar whether the patch is applied or not?
> >
> 
> Here are the numbers (MB/s) from the AACRAID box, after a fresh boot:
> 
> Test            2.6.19.2  2.6.24-rc6  2.6.24-rc6-81eabcbe0b991ddef5216f30ae91c4b226d54b6d
> dd1             325       350         290
> dd1-dir         180       160         160
> dd2           2x 90     2x113       2x110
> dd2-dir       2x120     2x 92       2x 93
> dd3           3x 54     3x 70       3x 70
> dd3-dir       3x 83     3x 64       3x 64
> mix3       55,2x 30 400,2x 25   310,2x 25
> 
>  What we are seing here is that:
> 
> a) DIRECT IO takes a much bigger hit (2.6.19 vs. 2.6.24) on this IO system compared to the CCISS box
> b) Reverting your patch hurts single stream

Right, and this is consistent with other complaints about the PFN of the
page mattering to some hardware.

> c) dual/triple stream are not affected by your patch and are improved over 2.6.19

I am not very surprised. The callers to the page allocator are probably
making no special effort to get a batch of pages in PFN-order. They are just
assuming that subsequent calls give contiguous pages. With two or more
threads involved, there will not be a correlation between physical pages
and what is on disk any more.

> d) the mix3 performance is improved compared to 2.6.19.
> d1) reverting your patch hurts the local-disk part of mix3
> e) the AACRAID setup is definitely faster than the CCISS.
> 
>  So, on this box your patch is definitely needed to get the pre-2.6.24 performance
> when writing a single big file.
> 
>  Actually things on the CCISS box might be even more complicated. I forgot the fact
> that on that box we have ext2/LVM/DM/Hardware, while on the AACRAID box we have
> ext2/Hardware. Do you think that the LVM/MD are sensitive to the page order/coloring?
> 

I don't have enough experience with LVM setups to make an intelligent
guess.

>  Anyway: does your patch only address this performance issue, or are there also
> data integrity concerns without it?

Performance issue only. There are no data integrity concerns with that
patch.

> I may consider reverting the patch for my
> production environment. It really helps two thirds of my boxes big time, while it does
> not hurt the other third that much :-)
> 

That is certainly an option.

> > > 
> > >  I can certainly stress the box before doing the tests. Please
> > > define "many" for the kernel compiles :-)
> > > 
> > 
> > With 8GiB of RAM, try making 24 copies of the kernel and compiling them
> > all simultaneously. Running that for for 20-30 minutes should be enough
> > 
>  to randomise the freelists affecting what color of page is used for the
> > dd  test.
> > 
> 
>  ouch :-) OK, I will try that.
> 

Thanks.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Mel Gorman <mel@csn.ul.ie>
> To: Martin Knoblauch <spamtrap@knobisoft.de>
> Cc: Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>; James.Bottomley@steeleye.com
> Sent: Thursday, January 17, 2008 11:12:21 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> On (17/01/08 13:50), Martin Knoblauch didst pronounce:
> > > 
> > 
> > The effect  is  defintely  depending on  the  IO  hardware.
> > 
 performed the same tests
> > on a different box with an AACRAID controller and there things
> > look different.
> 
> I take it different also means it does not show this odd performance
> behaviour and is similar whether the patch is applied or not?
>

Here are the numbers (MB/s) from the AACRAID box, after a fresh boot:

Test           2.6.19.2   2.6.24-rc6  2.6.24-rc6-81eabcbe0b991ddef5216f30ae91c4b226d54b6d
dd1             325       350             290
dd1-dir       180       160             160
dd2             2x90     2x113         2x110
dd2-dir       2x120   2x92            2x93
dd3            3x54      3x70           3x70
dd3-dir      3x83      3x64           3x64
mix3          55,2x30  400,2x25   310,2x25

 What we are seing here is that:

a) DIRECT IO takes a much bigger hit (2.6.19 vs. 2.6.24) on this IO system compared to the CCISS box
b) Reverting your patch hurts single stream
c) dual/triple stream are not affected by your patch and are improved over 2.6.19
d) the mix3 performance is improved compared to 2.6.19.
d1) reverting your patch hurts the local-disk part of mix3
e) the AACRAID setup is definitely faster than the CCISS.

 So, on this box your patch is definitely needed to get the pre-2.6.24 performance
when writing a single big file.

 Actually things on the CCISS box might be even more complicated. I forgot the fact
that on that box we have ext2/LVM/DM/Hardware, while on the AACRAID box we have
ext2/Hardware. Do you think that the LVM/MD are sensitive to the page order/coloring?

 Anyway: does your patch only address this performance issue, or are there also
data integrity concerns without it? I may consider reverting the patch for my
production environment. It really helps two thirds of my boxes big time, while it does
not hurt the other third that much :-)

> > 
> >  I can certainly stress the box before doing the tests. Please
> > define "many" for the kernel compiles :-)
> > 
> 
> With 8GiB of RAM, try making 24 copies of the kernel and compiling them
> all simultaneously. Running that for for 20-30 minutes should be enough
> 
 to randomise the freelists affecting what color of page is used for the
> dd  test.
> 

 ouch :-) OK, I will try that.

Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Mel Gorman]

On (17/01/08 13:50), Martin Knoblauch didst pronounce:
> > <mail manglement snipped>
> 
> The effect  is  defintely  depending on  the  IO  hardware. I performed the same tests
> on a different box with an AACRAID controller and there things look different.

I take it different also means it does not show this odd performance
behaviour and is similar whether the patch is applied or not?

> Basically
> the "offending" commit helps seingle stream performance on that box, while dual/triple
> stream are not affected. So I suspect that the CCISS is just not behaving well.
> 
> And yes, the tests are usually done on a freshly booted box. Of course, I repeat them
> a few times. On the CCISS box the numbers are very constant. On the AACRAID box
> they vary quite a bit.
> 
>  I can certainly stress the box before doing the tests. Please define "many" for the kernel
> compiles :-)
> 

With 8GiB of RAM, try making 24 copies of the kernel and compiling them
all simultaneously. Running that for for 20-30 minutes should be enough to
randomise the freelists affecting what color of page is used for the dd test.

> > >  OK, the change happened between rc5 and rc6. Just following a
> > > gut feeling, I reverted
> > > 
> > > #commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d
> > > #Author: Mel Gorman 
> > > #Date:   Mon Dec 17 16:20:05 2007 -0800
> > > #
> 
> > > 
> > > This has brought back the good results I observed and reported.
> > > I do not know what to make out of this. At least on the systems
> > > I care about (HP/DL380g4, dual CPUs, HT-enabled, 8 GB Memory,
> > > SmartaArray6i controller with 4x72GB SCSI disks as RAID5 (battery
> > > protected writeback cache enabled) and gigabit networking (tg3)) this
> > > optimisation is a dissaster.
> > > 
> > 
> > That patch was not an optimisation, it was a regression fix
> > against 2.6.23 and I don't believe reverting it is an option. Other IO
> > hardware benefits from having the allocator supply pages in PFN order.
> 
>  I think this late in the 2.6.24 game we just should leave things as they are. But
> we should try to find a way to make CCISS faster, as it apparently can be faster.
> 
> > Your controller would seem to suffer when presented with the same situation
> > but I don't know why that is. I've added James to the cc in case he has seen this
> > sort of situation before.
> > 
> > > On the other hand, it is not a regression against 2.6.22/23. Those
> > 
>  > had bad IO scaling to. It would just be a shame to loose an apparently
> > 
>  > great performance win.
> > 
> > Could you try running your tests again when the system has been
> > stressed with some other workload first?
> > 
> 
>  Will do.
> 

Thanks

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Mel Gorman <mel@csn.ul.ie>
> To: Martin Knoblauch <spamtrap@knobisoft.de>
> Cc: Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>; James.Bottomley@steeleye.com
> Sent: Thursday, January 17, 2008 9:23:57 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> On (17/01/08 09:44), Martin Knoblauch didst pronounce:
> > > > > > > > On Wed, Jan 16, 2008 at 01:26:41AM -0800,
> Martin
> 
 Knoblauch wrote:
> > > > > > > For those interested in using your writeback
> improvements
> 
 in
> > > > > > > production sooner rather than later (primarily with
> ext3);
> 
 what
> > > > > > > recommendations do you have?  Just heavily test our
> own
> 
 2.6.24
> > > > > > > evolving "close, but not ready for merge" -mm
> writeback
> 
 patchset?
> > > > > > > 
> > > > > > 
> > > > > >  I can add myself to Mikes question. It would be good to
> know
> 
 a
> > > > > 
> > > > > "roadmap" for the writeback changes. Testing 2.6.24-rcX so
> far
> 
 has
> > > > > been showing quite nice improvement of the overall
> writeback
> 
 situation and
> > > > > it would be sad to see this [partially] gone in 2.6.24-final.
> > > > > Linus apparently already has reverted  "...2250b". I
> will
> 
 definitely
> > > > > repeat my tests  with -rc8. and report.
> > > > > 
> > > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7?
> > > > Maybe we can push it to 2.6.24 after your testing.
> > > > 
> > > Hi Fengguang,
> > > 
> > > something really bad has happened between -rc3 and -rc6.
> > > Embarrassingly I did not catch that earlier :-(
> > > Compared to the numbers I posted in
> > > http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec
> > > (slight plus), while dd2/dd3 suck the same way as in pre 2.6.24.
> > > The only test that is still good is mix3, which I attribute to
> > > the per-BDI stuff.
> 
> I suspect that the IO hardware you have is very sensitive to the
> color of the physical page. I wonder, do you boot the system cleanly
> and then run these tests? If so, it would be interesting to know what
> happens if you stress the system first (many kernel compiles for example,
> basically anything that would use a lot of memory in different ways for some
> time) to randomise the free lists a bit and then run your test. You'd need to run
> the test three times for 2.6.23, 2.6.24-rc8 and 2.6.24-rc8 with the patch you
> identified reverted.
>

 The effect  is  defintely  depending on  the  IO  hardware. I performed the same tests
on a different box with an AACRAID controller and there things look different. Basically
the "offending" commit helps seingle stream performance on that box, while dual/triple
stream are not affected. So I suspect that the CCISS is just not behaving well.

 And yes, the tests are usually done on a freshly booted box. Of course, I repeat them
a few times. On the CCISS box the numbers are very constant. On the AACRAID box
they vary quite a bit.

 I can certainly stress the box before doing the tests. Please define "many" for the kernel
compiles :-)

> > 
> >  OK, the change happened between rc5 and rc6. Just following a
> > gut feeling, I reverted
> > 
> > #commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d
> > #Author: Mel Gorman 
> > #Date:   Mon Dec 17 16:20:05 2007 -0800
> > #

> > 
> > This has brought back the good results I observed and reported.
> > I do not know what to make out of this. At least on the systems
> > I care about (HP/DL380g4, dual CPUs, HT-enabled, 8 GB Memory,
> > SmartaArray6i controller with 4x72GB SCSI disks as RAID5 (battery
> > protected writeback cache enabled) and gigabit networking (tg3)) this
> > optimisation is a dissaster.
> > 
> 
> That patch was not an optimisation, it was a regression fix
> against 2.6.23 and I don't believe reverting it is an option. Other IO
> hardware benefits from having the allocator supply pages in PFN order.

 I think this late in the 2.6.24 game we just should leave things as they are. But
we should try to find a way to make CCISS faster, as it apparently can be faster.

> Your controller would seem to suffer when presented with the same situation
> but I don't know why that is. I've added James to the cc in case he has seen this
> sort of situation before.
> 
> > On the other hand, it is not a regression against 2.6.22/23. Those
> 
 > had bad IO scaling to. It would just be a shame to loose an apparently
> 
 > great performance win.
> 
> Could you try running your tests again when the system has been
> stressed with some other workload first?
> 

 Will do.

Cheers
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Mel Gorman]

On (17/01/08 09:44), Martin Knoblauch didst pronounce:
> > > > > > > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote:
> > > > > > For those interested in using your writeback improvements in
> > > > > > production sooner rather than later (primarily with ext3); what
> > > > > > recommendations do you have?  Just heavily test our own 2.6.24
> > > > > > evolving "close, but not ready for merge" -mm writeback patchset?
> > > > > > 
> > > > > 
> > > > >  I can add myself to Mikes question. It would be good to know a
> > > > 
> > > > "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has
> > > > been showing quite nice improvement of the overall writeback situation and
> > > > it would be sad to see this [partially] gone in 2.6.24-final.
> > > > Linus apparently already has reverted  "...2250b". I will definitely
> > > > repeat my tests  with -rc8. and report.
> > > > 
> > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7?
> > > Maybe we can push it to 2.6.24 after your testing.
> > > 
> > Hi Fengguang,
> > 
> > something really bad has happened between -rc3 and -rc6.
> > Embarrassingly I did not catch that earlier :-(
> > Compared to the numbers I posted in
> > http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec
> > (slight plus), while dd2/dd3 suck the same way as in pre 2.6.24.
> > The only test that is still good is mix3, which I attribute to
> > the per-BDI stuff.

I suspect that the IO hardware you have is very sensitive to the color of the
physical page. I wonder, do you boot the system cleanly and then run these
tests? If so, it would be interesting to know what happens if you stress
the system first (many kernel compiles for example, basically anything that
would use a lot of memory in different ways for some time) to randomise the
free lists a bit and then run your test. You'd need to run the test
three times for 2.6.23, 2.6.24-rc8 and 2.6.24-rc8 with the patch you
identified reverted.

> > At the moment I am frantically trying to find when things went down. I
> > did run -rc8 and rc8+yourpatch. No difference to what I see with -rc6.
> > 
> > Sorry that I cannot provide any input to your patch.
> 
>  OK, the change happened between rc5 and rc6. Just following a gut feeling, I reverted
> 
> #commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d
> #Author: Mel Gorman <mel@csn.ul.ie>
> #Date:   Mon Dec 17 16:20:05 2007 -0800
> #
> #    mm: fix page allocation for larger I/O segments
> #    
> #    In some cases the IO subsystem is able to merge requests if the pages are
> #    adjacent in physical memory.  This was achieved in the allocator by having
> #    expand() return pages in physically contiguous order in situations were a
> #    large buddy was split.  However, list-based anti-fragmentation changed the
> #    order pages were returned in to avoid searching in buffered_rmqueue() for a
> #    page of the appropriate migrate type.
> #    
> #    This patch restores behaviour of rmqueue_bulk() preserving the physical
> #    order of pages returned by the allocator without incurring increased search
> #    costs for anti-fragmentation.
> #    
> #    Signed-off-by: Mel Gorman <mel@csn.ul.ie>
> #    Cc: James Bottomley <James.Bottomley@steeleye.com>
> #    Cc: Jens Axboe <jens.axboe@oracle.com>
> #    Cc: Mark Lord <mlord@pobox.com
> #    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> #    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
> diff -urN linux-2.6.24-rc5/mm/page_alloc.c linux-2.6.24-rc6/mm/page_alloc.c
> --- linux-2.6.24-rc5/mm/page_alloc.c    2007-12-21 04:14:11.305633890 +0000
> +++ linux-2.6.24-rc6/mm/page_alloc.c    2007-12-21 04:14:17.746985697 +0000
> @@ -847,8 +847,19 @@
>                 struct page *page = __rmqueue(zone, order, migratetype);
>                 if (unlikely(page == NULL))
>                         break;
> +
> +               /*
> +                * Split buddy pages returned by expand() are received here
> +                * in physical page order. The page is added to the callers and
> +                * list and the list head then moves forward. From the callers
> +                * perspective, the linked list is ordered by page number in
> +                * some conditions. This is useful for IO devices that can
> +                * merge IO requests if the physical pages are ordered
> +                * properly.
> +                */
>                 list_add(&page->lru, list);
>                 set_page_private(page, migratetype);
> +               list = &page->lru;
>         }
>         spin_unlock(&zone->lock);
>         return i;
> 
> This has brought back the good results I observed and reported.
> I do not know what to make out of this. At least on the systems I care
> about (HP/DL380g4, dual CPUs, HT-enabled, 8 GB Memory, SmartaArray6i
> controller with 4x72GB SCSI disks as RAID5 (battery protected writeback
> cache enabled) and gigabit networking (tg3)) this optimisation is a dissaster.
> 

That patch was not an optimisation, it was a regression fix against 2.6.23
and I don't believe reverting it is an option. Other IO hardware benefits
from having the allocator supply pages in PFN order. Your controller would
seem to suffer when presented with the same situation but I don't know why
that is. I've added James to the cc in case he has seen this sort of
situation before.

> On the other hand, it is not a regression against 2.6.22/23. Those had
> bad IO scaling to. It would just be a shame to loose an apparently great
> performance win.
> is

Could you try running your tests again when the system has been stressed
with some other workload first?

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Mike Snitzer <snitzer@gmail.com>
> To: Martin Knoblauch <spamtrap@knobisoft.de>
> Cc: Fengguang Wu <wfg@mail.ustc.edu.cn>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>
> Sent: Thursday, January 17, 2008 5:11:50 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> 
> I've backported Peter's perbdi patchset to 2.6.22.x.  I can share it
> with anyone who might be interested.
> 
> As expected, it has yielded 2.6.24-rcX level scaling.  Given the test
> result matrix you previously posted, 2.6.22.x+perbdi might give you
> what you're looking for (sans improved writeback that 2.6.24 was
> thought to be providing).  That is, much improved scaling with better
> O_DIRECT and network throughput.  Just a thought...
> 
> Unfortunately, my priorities (and computing resources) have shifted
> and I won't be able to thoroughly test Fengguang's new writeback patch
> on 2.6.24-rc8... whereby missing out on providing
> justification/testing to others on _some_ improved writeback being
> included in 2.6.24 final.
> 
> Not to mention the window for writeback improvement is all but closed
> considering the 2.6.24-rc8 announcement's 2.6.24 final release
> timetable.
> 
Mike,

 thanks for the offer, but the improved throughput is my #1 priority nowadays.
And while the better scaling for different targets is nothing to frown upon, the
much better scaling when writing to the same target would have been the big
winner for me.

 Anyway, I located the "offending" commit. Lets see what the experts say.


Cheers
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Martin Knoblauch <spamtrap@knobisoft.de>
> To: Fengguang Wu <wfg@mail.ustc.edu.cn>
> Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>
> Sent: Thursday, January 17, 2008 2:52:58 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> ----- Original Message ----
> > From: Fengguang Wu 
> > To: Martin Knoblauch 
> > Cc: Mike Snitzer ; Peter
> Zijlstra
> 
 ; jplatte@naasa.net; Ingo Molnar
> ;
> 
 linux-kernel@vger.kernel.org;
> "linux-ext4@vger.kernel.org"
> 
 ; Linus
> Torvalds
> 
 
> > Sent: Wednesday, January 16, 2008 1:00:04 PM
> > Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> > 
> > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote:
> > > > For those interested in using your writeback improvements in
> > > > production sooner rather than later (primarily with ext3); what
> > > > recommendations do you have?  Just heavily test our own 2.6.24
> > +
> > 
>  your
> > > > evolving "close, but not ready for merge" -mm writeback patchset?
> > > > 
> > > Hi Fengguang, Mike,
> > > 
> > >  I can add myself to Mikes question. It would be good to know
> > a
> > 
>  "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has
> > been
> > 
>  showing quite nice improvement of the overall writeback situation and
> > it
> > 
>  would be sad to see this [partially] gone in 2.6.24-final.
> > Linus
> > 
>  apparently already has reverted  "...2250b". I will definitely
> repeat
> 
 my
> > tests
> > 
>  with -rc8. and report.
> > 
> > Thank you, Martin. Can you help test this patch on 2.6.24-rc7?
> > Maybe we can push it to 2.6.24 after your testing.
> > 
> Hi Fengguang,
> 
>  something really bad has happened between -rc3 and
> -rc6.
> 
 Embarrassingly I did not catch that earlier :-(
> 
>  Compared to the numbers I posted
> in
> 
 http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec
> (slight
> 
 plus), while dd2/dd3 suck the same way as in pre 2.6.24. The only
> test
> 
 that is still good is mix3, which I attribute to the per-BDI stuff.
> 
>  At the moment I am frantically trying to find when things went down.
> I
> 
 did run -rc8 and rc8+yourpatch. No difference to what I see with
> -rc6.
> 
 Sorry that I cannot provide any input to your patch.
> 

 OK, the change happened between rc5 and rc6. Just following a gut feeling, I reverted

#commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d
#Author: Mel Gorman <mel@csn.ul.ie>
#Date:   Mon Dec 17 16:20:05 2007 -0800
#
#    mm: fix page allocation for larger I/O segments
#    
#    In some cases the IO subsystem is able to merge requests if the pages are
#    adjacent in physical memory.  This was achieved in the allocator by having
#    expand() return pages in physically contiguous order in situations were a
#    large buddy was split.  However, list-based anti-fragmentation changed the
#    order pages were returned in to avoid searching in buffered_rmqueue() for a
#    page of the appropriate migrate type.
#    
#    This patch restores behaviour of rmqueue_bulk() preserving the physical
#    order of pages returned by the allocator without incurring increased search
#    costs for anti-fragmentation.
#    
#    Signed-off-by: Mel Gorman <mel@csn.ul.ie>
#    Cc: James Bottomley <James.Bottomley@steeleye.com>
#    Cc: Jens Axboe <jens.axboe@oracle.com>
#    Cc: Mark Lord <mlord@pobox.com
#    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
#    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
diff -urN linux-2.6.24-rc5/mm/page_alloc.c linux-2.6.24-rc6/mm/page_alloc.c
--- linux-2.6.24-rc5/mm/page_alloc.c    2007-12-21 04:14:11.305633890 +0000
+++ linux-2.6.24-rc6/mm/page_alloc.c    2007-12-21 04:14:17.746985697 +0000
@@ -847,8 +847,19 @@
                struct page *page = __rmqueue(zone, order, migratetype);
                if (unlikely(page == NULL))
                        break;
+
+               /*
+                * Split buddy pages returned by expand() are received here
+                * in physical page order. The page is added to the callers and
+                * list and the list head then moves forward. From the callers
+                * perspective, the linked list is ordered by page number in
+                * some conditions. This is useful for IO devices that can
+                * merge IO requests if the physical pages are ordered
+                * properly.
+                */
                list_add(&page->lru, list);
                set_page_private(page, migratetype);
+               list = &page->lru;
        }
        spin_unlock(&zone->lock);
        return i;



This has brought back the good results I observed and reported.
I do not know what to make out of this. At least on the systems I care
about (HP/DL380g4, dual CPUs, HT-enabled, 8 GB Memory, SmartaArray6i
controller with 4x72GB SCSI disks as RAID5 (battery protected writeback
cache enabled) and gigabit networking (tg3)) this optimisation is a dissaster.

 On the other hand, it is not a regression against 2.6.22/23. Those had
bad IO scaling to. It would just be a shame to loose an apparently great
performance win.
is

Cheers
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Mike Snitzer]

On Jan 17, 2008 8:52 AM, Martin Knoblauch <spamtrap@knobisoft.de> wrote:
>
> ----- Original Message ----
> > From: Fengguang Wu <wfg@mail.ustc.edu.cn>
> > To: Martin Knoblauch <knobi@knobisoft.de>
> > Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>
> > Sent: Wednesday, January 16, 2008 1:00:04 PM
> > Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> >
> > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote:
> > > > For those interested in using your writeback improvements in
> > > > production sooner rather than later (primarily with ext3); what
> > > > recommendations do you have?  Just heavily test our own 2.6.24
> > +
> >
>  your
> > > > evolving "close, but not ready for merge" -mm writeback patchset?
> > > >
> > > Hi Fengguang, Mike,
> > >
> > >  I can add myself to Mikes question. It would be good to know
> > a
> >
>  "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has
> > been
> >
>  showing quite nice improvement of the overall writeback situation and
> > it
> >
>  would be sad to see this [partially] gone in 2.6.24-final.
> > Linus
> >
>  apparently already has reverted  "...2250b". I will definitely repeat my
> > tests
> >
>  with -rc8. and report.
> >
> > Thank you, Martin. Can you help test this patch on 2.6.24-rc7?
> > Maybe we can push it to 2.6.24 after your testing.
> >
> Hi Fengguang,
>
>  something really bad has happened between -rc3 and -rc6. Embarrassingly I did not catch that earlier :-(
>
>  Compared to the numbers I posted in http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec (slight plus), while dd2/dd3 suck the same way as in pre 2.6.24. The only test that is still good is mix3, which I attribute to the per-BDI stuff.
>
>  At the moment I am frantically trying to find when things went down. I did run -rc8 and rc8+yourpatch. No difference to what I see with -rc6. Sorry that I cannot provide any input to your patch.
>
> Depressed
> Martin

Martin,

I've backported Peter's perbdi patchset to 2.6.22.x.  I can share it
with anyone who might be interested.

As expected, it has yielded 2.6.24-rcX level scaling.  Given the test
result matrix you previously posted, 2.6.22.x+perbdi might give you
what you're looking for (sans improved writeback that 2.6.24 was
thought to be providing).  That is, much improved scaling with better
O_DIRECT and network throughput.  Just a thought...

Unfortunately, my priorities (and computing resources) have shifted
and I won't be able to thoroughly test Fengguang's new writeback patch
on 2.6.24-rc8... whereby missing out on providing
justification/testing to others on _some_ improved writeback being
included in 2.6.24 final.

Not to mention the window for writeback improvement is all but closed
considering the 2.6.24-rc8 announcement's 2.6.24 final release
timetable.

regards,
Mike

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Fengguang Wu <wfg@mail.ustc.edu.cn>
> To: Martin Knoblauch <knobi@knobisoft.de>
> Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>
> Sent: Wednesday, January 16, 2008 1:00:04 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote:
> > > For those interested in using your writeback improvements in
> > > production sooner rather than later (primarily with ext3); what
> > > recommendations do you have?  Just heavily test our own 2.6.24
> +
> 
 your
> > > evolving "close, but not ready for merge" -mm writeback patchset?
> > > 
> > Hi Fengguang, Mike,
> > 
> >  I can add myself to Mikes question. It would be good to know
> a
> 
 "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has
> been
> 
 showing quite nice improvement of the overall writeback situation and
> it
> 
 would be sad to see this [partially] gone in 2.6.24-final.
> Linus
> 
 apparently already has reverted  "...2250b". I will definitely repeat my
> tests
> 
 with -rc8. and report.
> 
> Thank you, Martin. Can you help test this patch on 2.6.24-rc7?
> Maybe we can push it to 2.6.24 after your testing.
> 
Hi Fengguang,

 something really bad has happened between -rc3 and -rc6. Embarrassingly I did not catch that earlier :-(

 Compared to the numbers I posted in http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec (slight plus), while dd2/dd3 suck the same way as in pre 2.6.24. The only test that is still good is mix3, which I attribute to the per-BDI stuff.

 At the moment I am frantically trying to find when things went down. I did run -rc8 and rc8+yourpatch. No difference to what I see with -rc6. Sorry that I cannot provide any input to your patch.

Depressed
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Mike Snitzer]

On Jan 16, 2008 9:15 AM, Martin Knoblauch <spamtrap@knobisoft.de> wrote:
> ----- Original Message ----
> > From: Fengguang Wu <wfg@mail.ustc.edu.cn>
> > To: Martin Knoblauch <knobi@knobisoft.de>
> > Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>
> > Sent: Wednesday, January 16, 2008 1:00:04 PM
> > Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> >
>
> > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote:
> > > > For those interested in using your writeback improvements in
> > > > production sooner rather than later (primarily with ext3); what
> > > > recommendations do you have?  Just heavily test our own 2.6.24
> > +
> >
>  your
> > > > evolving "close, but not ready for merge" -mm writeback patchset?
> > > >
> > > Hi Fengguang, Mike,
> > >
> > >  I can add myself to Mikes question. It would be good to know
> > a
> >
>  "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has
> > been
> >
>  showing quite nice improvement of the overall writeback situation and
> > it
> >
>  would be sad to see this [partially] gone in 2.6.24-final.
> > Linus
> >
>  apparently already has reverted  "...2250b". I will definitely repeat my
> > tests
> >
>  with -rc8. and report.
> >
> > Thank you, Martin. Can you help test this patch on 2.6.24-rc7?
> > Maybe we can push it to 2.6.24 after your testing.
> >
>
>  Will do tomorrow or friday. Actually a patch against -rc8 would be nicer for me, as I have not looked at -rc7 due to holidays and some of the reported problems with it.

Fengguang's latest writeback patch applies cleanly, builds, boots on 2.6.24-rc8.

I'll be able to share ext3 performance results (relative to 2.6.24-rc7) shortly.

Mike
>
>
> > Fengguang
> > ---
> >  fs/fs-writeback.c         |   17 +++++++++++++++--
> >  include/linux/writeback.h |    1 +
> >  mm/page-writeback.c       |    9 ++++++---
> >  3 files changed, 22 insertions(+), 5 deletions(-)
> >
> > --- linux.orig/fs/fs-writeback.c
> > +++ linux/fs/fs-writeback.c
> > @@ -284,7 +284,16 @@ __sync_single_inode(struct inode *inode,
> >                   * soon as the queue becomes uncongested.
> >                   */
> >                  inode->i_state |= I_DIRTY_PAGES;
> > -                requeue_io(inode);
> > +                if (wbc->nr_to_write <= 0)
> > +                    /*
> > +                     * slice used up: queue for next turn
> > +                     */
> > +                    requeue_io(inode);
> > +                else
> > +                    /*
> > +                     * somehow blocked: retry later
> > +                     */
> > +                    redirty_tail(inode);
> >              } else {
> >                  /*
> >                   * Otherwise fully redirty the inode so that
> > @@ -479,8 +488,12 @@ sync_sb_inodes(struct super_block *sb, s
> >          iput(inode);
> >          cond_resched();
> >          spin_lock(&inode_lock);
> > -        if (wbc->nr_to_write <= 0)
> > +        if (wbc->nr_to_write <= 0) {
> > +            wbc->more_io = 1;
> >              break;
> > +        }
> > +        if (!list_empty(&sb->s_more_io))
> > +            wbc->more_io = 1;
> >      }
> >      return;        /* Leave any unwritten inodes on s_io */
> >  }
> > --- linux.orig/include/linux/writeback.h
> > +++ linux/include/linux/writeback.h
> > @@ -62,6 +62,7 @@ struct writeback_control {
> >      unsigned for_reclaim:1;        /* Invoked from the page
> > allocator
> >
>  */
> >      unsigned for_writepages:1;    /* This is a writepages() call */
> >      unsigned range_cyclic:1;    /* range_start is cyclic */
> > +    unsigned more_io:1;        /* more io to be dispatched */
> >  };
> >
> >  /*
> > --- linux.orig/mm/page-writeback.c
> > +++ linux/mm/page-writeback.c
> > @@ -558,6 +558,7 @@ static void background_writeout(unsigned
> >              global_page_state(NR_UNSTABLE_NFS) < background_thresh
> >                  && min_pages <= 0)
> >              break;
> > +        wbc.more_io = 0;
> >          wbc.encountered_congestion = 0;
> >          wbc.nr_to_write = MAX_WRITEBACK_PAGES;
> >          wbc.pages_skipped = 0;
> > @@ -565,8 +566,9 @@ static void background_writeout(unsigned
> >          min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
> >          if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
> >              /* Wrote less than expected */
> > -            congestion_wait(WRITE, HZ/10);
> > -            if (!wbc.encountered_congestion)
> > +            if (wbc.encountered_congestion || wbc.more_io)
> > +                congestion_wait(WRITE, HZ/10);
> > +            else
> >                  break;
> >          }
> >      }
> > @@ -631,11 +633,12 @@ static void wb_kupdate(unsigned long arg
> >              global_page_state(NR_UNSTABLE_NFS) +
> >              (inodes_stat.nr_inodes - inodes_stat.nr_unused);
> >      while (nr_to_write > 0) {
> > +        wbc.more_io = 0;
> >          wbc.encountered_congestion = 0;
> >          wbc.nr_to_write = MAX_WRITEBACK_PAGES;
> >          writeback_inodes(&wbc);
> >          if (wbc.nr_to_write > 0) {
> > -            if (wbc.encountered_congestion)
> > +            if (wbc.encountered_congestion || wbc.more_io)
> >                  congestion_wait(WRITE, HZ/10);
> >              else
> >                  break;    /* All the old data is written */
> >
> >
> >
>
>
>

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Fengguang Wu <wfg@mail.ustc.edu.cn>
> To: Martin Knoblauch <knobi@knobisoft.de>
> Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>
> Sent: Wednesday, January 16, 2008 1:00:04 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote:
> > > For those interested in using your writeback improvements in
> > > production sooner rather than later (primarily with ext3); what
> > > recommendations do you have?  Just heavily test our own 2.6.24
> +
> 
 your
> > > evolving "close, but not ready for merge" -mm writeback patchset?
> > > 
> > Hi Fengguang, Mike,
> > 
> >  I can add myself to Mikes question. It would be good to know
> a
> 
 "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has
> been
> 
 showing quite nice improvement of the overall writeback situation and
> it
> 
 would be sad to see this [partially] gone in 2.6.24-final.
> Linus
> 
 apparently already has reverted  "...2250b". I will definitely repeat my
> tests
> 
 with -rc8. and report.
> 
> Thank you, Martin. Can you help test this patch on 2.6.24-rc7?
> Maybe we can push it to 2.6.24 after your testing.
> 

 Will do tomorrow or friday. Actually a patch against -rc8 would be nicer for me, as I have not looked at -rc7 due to holidays and some of the reported problems with it.

Cheers
Martin

> Fengguang
> ---
>  fs/fs-writeback.c         |   17 +++++++++++++++--
>  include/linux/writeback.h |    1 +
>  mm/page-writeback.c       |    9 ++++++---
>  3 files changed, 22 insertions(+), 5 deletions(-)
> 
> --- linux.orig/fs/fs-writeback.c
> +++ linux/fs/fs-writeback.c
> @@ -284,7 +284,16 @@ __sync_single_inode(struct inode *inode,
>                   * soon as the queue becomes uncongested.
>                   */
>                  inode->i_state |= I_DIRTY_PAGES;
> -                requeue_io(inode);
> +                if (wbc->nr_to_write <= 0)
> +                    /*
> +                     * slice used up: queue for next turn
> +                     */
> +                    requeue_io(inode);
> +                else
> +                    /*
> +                     * somehow blocked: retry later
> +                     */
> +                    redirty_tail(inode);
>              } else {
>                  /*
>                   * Otherwise fully redirty the inode so that
> @@ -479,8 +488,12 @@ sync_sb_inodes(struct super_block *sb, s
>          iput(inode);
>          cond_resched();
>          spin_lock(&inode_lock);
> -        if (wbc->nr_to_write <= 0)
> +        if (wbc->nr_to_write <= 0) {
> +            wbc->more_io = 1;
>              break;
> +        }
> +        if (!list_empty(&sb->s_more_io))
> +            wbc->more_io = 1;
>      }
>      return;        /* Leave any unwritten inodes on s_io */
>  }
> --- linux.orig/include/linux/writeback.h
> +++ linux/include/linux/writeback.h
> @@ -62,6 +62,7 @@ struct writeback_control {
>      unsigned for_reclaim:1;        /* Invoked from the page
> allocator
> 
 */
>      unsigned for_writepages:1;    /* This is a writepages() call */
>      unsigned range_cyclic:1;    /* range_start is cyclic */
> +    unsigned more_io:1;        /* more io to be dispatched */
>  };
>  
>  /*
> --- linux.orig/mm/page-writeback.c
> +++ linux/mm/page-writeback.c
> @@ -558,6 +558,7 @@ static void background_writeout(unsigned
>              global_page_state(NR_UNSTABLE_NFS) < background_thresh
>                  && min_pages <= 0)
>              break;
> +        wbc.more_io = 0;
>          wbc.encountered_congestion = 0;
>          wbc.nr_to_write = MAX_WRITEBACK_PAGES;
>          wbc.pages_skipped = 0;
> @@ -565,8 +566,9 @@ static void background_writeout(unsigned
>          min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
>          if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
>              /* Wrote less than expected */
> -            congestion_wait(WRITE, HZ/10);
> -            if (!wbc.encountered_congestion)
> +            if (wbc.encountered_congestion || wbc.more_io)
> +                congestion_wait(WRITE, HZ/10);
> +            else
>                  break;
>          }
>      }
> @@ -631,11 +633,12 @@ static void wb_kupdate(unsigned long arg
>              global_page_state(NR_UNSTABLE_NFS) +
>              (inodes_stat.nr_inodes - inodes_stat.nr_unused);
>      while (nr_to_write > 0) {
> +        wbc.more_io = 0;
>          wbc.encountered_congestion = 0;
>          wbc.nr_to_write = MAX_WRITEBACK_PAGES;
>          writeback_inodes(&wbc);
>          if (wbc.nr_to_write > 0) {
> -            if (wbc.encountered_congestion)
> +            if (wbc.encountered_congestion || wbc.more_io)
>                  congestion_wait(WRITE, HZ/10);
>              else
>                  break;    /* All the old data is written */
> 
> 
> 

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote:
> > For those interested in using your writeback improvements in
> > production sooner rather than later (primarily with ext3); what
> > recommendations do you have?  Just heavily test our own 2.6.24 + your
> > evolving "close, but not ready for merge" -mm writeback patchset?
> > 
> Hi Fengguang, Mike,
> 
>  I can add myself to Mikes question. It would be good to know a "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has been showing quite nice improvement of the overall writeback situation and it would be sad to see this [partially] gone in 2.6.24-final. Linus apparently already has reverted  "...2250b". I will definitely repeat my tests with -rc8. and report.

Thank you, Martin. Can you help test this patch on 2.6.24-rc7?
Maybe we can push it to 2.6.24 after your testing.

Fengguang
---
 fs/fs-writeback.c         |   17 +++++++++++++++--
 include/linux/writeback.h |    1 +
 mm/page-writeback.c       |    9 ++++++---
 3 files changed, 22 insertions(+), 5 deletions(-)

--- linux.orig/fs/fs-writeback.c
+++ linux/fs/fs-writeback.c
@@ -284,7 +284,16 @@ __sync_single_inode(struct inode *inode,
 				 * soon as the queue becomes uncongested.
 				 */
 				inode->i_state |= I_DIRTY_PAGES;
-				requeue_io(inode);
+				if (wbc->nr_to_write <= 0)
+					/*
+					 * slice used up: queue for next turn
+					 */
+					requeue_io(inode);
+				else
+					/*
+					 * somehow blocked: retry later
+					 */
+					redirty_tail(inode);
 			} else {
 				/*
 				 * Otherwise fully redirty the inode so that
@@ -479,8 +488,12 @@ sync_sb_inodes(struct super_block *sb, s
 		iput(inode);
 		cond_resched();
 		spin_lock(&inode_lock);
-		if (wbc->nr_to_write <= 0)
+		if (wbc->nr_to_write <= 0) {
+			wbc->more_io = 1;
 			break;
+		}
+		if (!list_empty(&sb->s_more_io))
+			wbc->more_io = 1;
 	}
 	return;		/* Leave any unwritten inodes on s_io */
 }
--- linux.orig/include/linux/writeback.h
+++ linux/include/linux/writeback.h
@@ -62,6 +62,7 @@ struct writeback_control {
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned for_writepages:1;	/* This is a writepages() call */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
+	unsigned more_io:1;		/* more io to be dispatched */
 };
 
 /*
--- linux.orig/mm/page-writeback.c
+++ linux/mm/page-writeback.c
@@ -558,6 +558,7 @@ static void background_writeout(unsigned
 			global_page_state(NR_UNSTABLE_NFS) < background_thresh
 				&& min_pages <= 0)
 			break;
+		wbc.more_io = 0;
 		wbc.encountered_congestion = 0;
 		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		wbc.pages_skipped = 0;
@@ -565,8 +566,9 @@ static void background_writeout(unsigned
 		min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
 		if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
 			/* Wrote less than expected */
-			congestion_wait(WRITE, HZ/10);
-			if (!wbc.encountered_congestion)
+			if (wbc.encountered_congestion || wbc.more_io)
+				congestion_wait(WRITE, HZ/10);
+			else
 				break;
 		}
 	}
@@ -631,11 +633,12 @@ static void wb_kupdate(unsigned long arg
 			global_page_state(NR_UNSTABLE_NFS) +
 			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
 	while (nr_to_write > 0) {
+		wbc.more_io = 0;
 		wbc.encountered_congestion = 0;
 		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		writeback_inodes(&wbc);
 		if (wbc.nr_to_write > 0) {
-			if (wbc.encountered_congestion)
+			if (wbc.encountered_congestion || wbc.more_io)
 				congestion_wait(WRITE, HZ/10);
 			else
 				break;	/* All the old data is written */

Re: regression: 100% io-wait with 2.6.24-rcX

[Martin Knoblauch]

----- Original Message ----
> From: Mike Snitzer <snitzer@gmail.com>
> To: Fengguang Wu <wfg@mail.ustc.edu.cn>
> Cc: Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>; Andrew Morton <akpm@li>
> Sent: Tuesday, January 15, 2008 10:13:22 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
> 
> On Jan 14, 2008 7:50 AM, Fengguang Wu  wrote:
> > On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote:
> > >
> > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote:
> > > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu:
> > > >
> > > > > Joerg, this patch fixed the bug for me :-)
> > > >
> > > > Fengguang, congratulations, I can confirm that your patch
> fixed
> 
 the bug! With
> > > > previous kernels the bug showed up after each reboot. Now,
> when
> 
 booting the
> > > > patched kernel everything is fine and there is no longer
> any
> 
 suspicious
> > > > iowait!
> > > >
> > > > Do you have an idea why this problem appeared in 2.6.24?
> Did
> 
 somebody change
> > > > the ext2 code or is it related to the changes in the scheduler?
> > >
> > > It was Fengguang who changed the inode writeback code, and I
> guess
> 
 the
> > > new and improved code was less able do deal with these funny corner
> > > cases. But he has been very good in tracking them down and
> solving
> 
 them,
> > > kudos to him for that work!
> >
> > Thank you.
> >
> > In particular the bug is triggered by the patch named:
> >         "writeback: introduce writeback_control.more_io to
> indicate
> 
 more io"
> > That patch means to speed up writeback, but unfortunately its
> > aggressiveness has disclosed bugs in reiserfs, jfs and now ext2.
> >
> > Linus, given the number of bugs it triggered, I'd recommend revert
> > this patch(git commit
> 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b).
> 
 Let's
> > push it back to -mm tree for more testings?
> 
> Fengguang,
> 
> I'd like to better understand where your writeback work stands
> relative to 2.6.24-rcX and -mm.  To be clear, your changes in
> 2.6.24-rc7 have been benchmarked to provide a ~33% sequential write
> performance improvement with ext3 (as compared to 2.6.22, CFS could be
> helping, etc but...).  Very impressive!
> 
> Given this improvement it is unfortunate to see your request to revert
> 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b but it is understandable if
> you're not confident in it for 2.6.24.
> 
> That said, you recently posted an -mm patchset that first reverts
> 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b and then goes on to address
> the "slow writes for concurrent large and small file writes" bug:
> http://lkml.org/lkml/2008/1/15/132
> 
> For those interested in using your writeback improvements in
> production sooner rather than later (primarily with ext3); what
> recommendations do you have?  Just heavily test our own 2.6.24 + your
> evolving "close, but not ready for merge" -mm writeback patchset?
> 
Hi Fengguang, Mike,

 I can add myself to Mikes question. It would be good to know a "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has been showing quite nice improvement of the overall writeback situation and it would be sad to see this [partially] gone in 2.6.24-final. Linus apparently already has reverted  "...2250b". I will definitely repeat my tests with -rc8. and report.

 Cheers
Martin

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Tue, Jan 15, 2008 at 04:13:22PM -0500, Mike Snitzer wrote:
> On Jan 14, 2008 7:50 AM, Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
> > On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote:
> > >
> > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote:
> > > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu:
> > > >
> > > > > Joerg, this patch fixed the bug for me :-)
> > > >
> > > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With
> > > > previous kernels the bug showed up after each reboot. Now, when booting the
> > > > patched kernel everything is fine and there is no longer any suspicious
> > > > iowait!
> > > >
> > > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change
> > > > the ext2 code or is it related to the changes in the scheduler?
> > >
> > > It was Fengguang who changed the inode writeback code, and I guess the
> > > new and improved code was less able do deal with these funny corner
> > > cases. But he has been very good in tracking them down and solving them,
> > > kudos to him for that work!
> >
> > Thank you.
> >
> > In particular the bug is triggered by the patch named:
> >         "writeback: introduce writeback_control.more_io to indicate more io"
> > That patch means to speed up writeback, but unfortunately its
> > aggressiveness has disclosed bugs in reiserfs, jfs and now ext2.
> >
> > Linus, given the number of bugs it triggered, I'd recommend revert
> > this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's
> > push it back to -mm tree for more testings?
> 
> Fengguang,
> 
> I'd like to better understand where your writeback work stands
> relative to 2.6.24-rcX and -mm.  To be clear, your changes in
> 2.6.24-rc7 have been benchmarked to provide a ~33% sequential write
> performance improvement with ext3 (as compared to 2.6.22, CFS could be
> helping, etc but...).  Very impressive!

Wow, glad to hear that.

> Given this improvement it is unfortunate to see your request to revert
> 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b but it is understandable if
> you're not confident in it for 2.6.24.
> 
> That said, you recently posted an -mm patchset that first reverts
> 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b and then goes on to address
> the "slow writes for concurrent large and small file writes" bug:
> http://lkml.org/lkml/2008/1/15/132
> 
> For those interested in using your writeback improvements in
> production sooner rather than later (primarily with ext3); what
> recommendations do you have?  Just heavily test our own 2.6.24 + your
> evolving "close, but not ready for merge" -mm writeback patchset?

It's not ready mainly because it is fresh made and need more
feedbacks. It's doing OK on my desktop :-)

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Tue, Jan 15, 2008 at 10:42:13PM +0100, Ingo Molnar wrote:
> 
> * Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
> 
> > On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote:
> > > 
> > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote:
> > > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu:
> > > > 
> > > > > Joerg, this patch fixed the bug for me :-)
> > > > 
> > > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With 
> > > > previous kernels the bug showed up after each reboot. Now, when booting the 
> > > > patched kernel everything is fine and there is no longer any suspicious 
> > > > iowait!
> > > > 
> > > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change 
> > > > the ext2 code or is it related to the changes in the scheduler?
> > > 
> > > It was Fengguang who changed the inode writeback code, and I guess the
> > > new and improved code was less able do deal with these funny corner
> > > cases. But he has been very good in tracking them down and solving them,
> > > kudos to him for that work!
> > 
> > Thank you.
> > 
> > In particular the bug is triggered by the patch named:
> >         "writeback: introduce writeback_control.more_io to indicate more io"
> > That patch means to speed up writeback, but unfortunately its
> > aggressiveness has disclosed bugs in reiserfs, jfs and now ext2.
> > 
> > Linus, given the number of bugs it triggered, I'd recommend revert 
> > this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's 
> > push it back to -mm tree for more testings?
> 
> i dont think a revert at this stage is a good idea and i'm not sure 
> pushing it back into -mm would really expose more of these bugs. And 
> these are real bugs in filesystems - bugs which we want to see fixed 
> anyway. You are also tracking down those bugs very fast.
> 
> [ perhaps, if it's possible technically (and if it is clean enough), you
>   might want to offer a runtime debug tunable that can be used to switch
>   off the new aspects of your code. That would speed up testing, in case
>   anyone suspects the new writeback code. ]

The patch is too aggressive in itself. We'd better not risk on it.
The iowait is only unpleasant not destructive. But it will hurt if
many users complaints. Comment says that "nfs_writepages() sometimes
bales out without doing anything."

However I have an improved and more safe patch now. It won't iowait
when nfs_writepages() bale out without increasing pages_skipped, or
even when some buggy filesystem forget to clear PAGECACHE_TAG_DIRTY.
(The magic lies in the first chunk below.)

Mike, you can use this one on 2.6.24.


---
 fs/fs-writeback.c         |   17 +++++++++++++++--
 include/linux/writeback.h |    1 +
 mm/page-writeback.c       |    9 ++++++---
 3 files changed, 22 insertions(+), 5 deletions(-)

--- linux.orig/fs/fs-writeback.c
+++ linux/fs/fs-writeback.c
@@ -284,7 +284,16 @@ __sync_single_inode(struct inode *inode,
 				 * soon as the queue becomes uncongested.
 				 */
 				inode->i_state |= I_DIRTY_PAGES;
-				requeue_io(inode);
+				if (wbc->nr_to_write <= 0)
+					/*
+					 * slice used up: queue for next turn
+					 */
+					requeue_io(inode);
+				else
+					/*
+					 * somehow blocked: retry later
+					 */
+					redirty_tail(inode);
 			} else {
 				/*
 				 * Otherwise fully redirty the inode so that
@@ -479,8 +488,12 @@ sync_sb_inodes(struct super_block *sb, s
 		iput(inode);
 		cond_resched();
 		spin_lock(&inode_lock);
-		if (wbc->nr_to_write <= 0)
+		if (wbc->nr_to_write <= 0) {
+			wbc->more_io = 1;
 			break;
+		}
+		if (!list_empty(&sb->s_more_io))
+			wbc->more_io = 1;
 	}
 	return;		/* Leave any unwritten inodes on s_io */
 }
--- linux.orig/include/linux/writeback.h
+++ linux/include/linux/writeback.h
@@ -62,6 +62,7 @@ struct writeback_control {
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned for_writepages:1;	/* This is a writepages() call */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
+	unsigned more_io:1;		/* more io to be dispatched */
 };
 
 /*
--- linux.orig/mm/page-writeback.c
+++ linux/mm/page-writeback.c
@@ -558,6 +558,7 @@ static void background_writeout(unsigned
 			global_page_state(NR_UNSTABLE_NFS) < background_thresh
 				&& min_pages <= 0)
 			break;
+		wbc.more_io = 0;
 		wbc.encountered_congestion = 0;
 		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		wbc.pages_skipped = 0;
@@ -565,8 +566,9 @@ static void background_writeout(unsigned
 		min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
 		if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
 			/* Wrote less than expected */
-			congestion_wait(WRITE, HZ/10);
-			if (!wbc.encountered_congestion)
+			if (wbc.encountered_congestion || wbc.more_io)
+				congestion_wait(WRITE, HZ/10);
+			else
 				break;
 		}
 	}
@@ -631,11 +633,12 @@ static void wb_kupdate(unsigned long arg
 			global_page_state(NR_UNSTABLE_NFS) +
 			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
 	while (nr_to_write > 0) {
+		wbc.more_io = 0;
 		wbc.encountered_congestion = 0;
 		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		writeback_inodes(&wbc);
 		if (wbc.nr_to_write > 0) {
-			if (wbc.encountered_congestion)
+			if (wbc.encountered_congestion || wbc.more_io)
 				congestion_wait(WRITE, HZ/10);
 			else
 				break;	/* All the old data is written */

Re: regression: 100% io-wait with 2.6.24-rcX

[Ingo Molnar]

* Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:

> On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote:
> > 
> > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote:
> > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu:
> > > 
> > > > Joerg, this patch fixed the bug for me :-)
> > > 
> > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With 
> > > previous kernels the bug showed up after each reboot. Now, when booting the 
> > > patched kernel everything is fine and there is no longer any suspicious 
> > > iowait!
> > > 
> > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change 
> > > the ext2 code or is it related to the changes in the scheduler?
> > 
> > It was Fengguang who changed the inode writeback code, and I guess the
> > new and improved code was less able do deal with these funny corner
> > cases. But he has been very good in tracking them down and solving them,
> > kudos to him for that work!
> 
> Thank you.
> 
> In particular the bug is triggered by the patch named:
>         "writeback: introduce writeback_control.more_io to indicate more io"
> That patch means to speed up writeback, but unfortunately its
> aggressiveness has disclosed bugs in reiserfs, jfs and now ext2.
> 
> Linus, given the number of bugs it triggered, I'd recommend revert 
> this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's 
> push it back to -mm tree for more testings?

i dont think a revert at this stage is a good idea and i'm not sure 
pushing it back into -mm would really expose more of these bugs. And 
these are real bugs in filesystems - bugs which we want to see fixed 
anyway. You are also tracking down those bugs very fast.

[ perhaps, if it's possible technically (and if it is clean enough), you
  might want to offer a runtime debug tunable that can be used to switch
  off the new aspects of your code. That would speed up testing, in case
  anyone suspects the new writeback code. ]

	Ingo

Re: regression: 100% io-wait with 2.6.24-rcX

[Mike Snitzer]

On Jan 14, 2008 7:50 AM, Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
> On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote:
> >
> > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote:
> > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu:
> > >
> > > > Joerg, this patch fixed the bug for me :-)
> > >
> > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With
> > > previous kernels the bug showed up after each reboot. Now, when booting the
> > > patched kernel everything is fine and there is no longer any suspicious
> > > iowait!
> > >
> > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change
> > > the ext2 code or is it related to the changes in the scheduler?
> >
> > It was Fengguang who changed the inode writeback code, and I guess the
> > new and improved code was less able do deal with these funny corner
> > cases. But he has been very good in tracking them down and solving them,
> > kudos to him for that work!
>
> Thank you.
>
> In particular the bug is triggered by the patch named:
>         "writeback: introduce writeback_control.more_io to indicate more io"
> That patch means to speed up writeback, but unfortunately its
> aggressiveness has disclosed bugs in reiserfs, jfs and now ext2.
>
> Linus, given the number of bugs it triggered, I'd recommend revert
> this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's
> push it back to -mm tree for more testings?

Fengguang,

I'd like to better understand where your writeback work stands
relative to 2.6.24-rcX and -mm.  To be clear, your changes in
2.6.24-rc7 have been benchmarked to provide a ~33% sequential write
performance improvement with ext3 (as compared to 2.6.22, CFS could be
helping, etc but...).  Very impressive!

Given this improvement it is unfortunate to see your request to revert
2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b but it is understandable if
you're not confident in it for 2.6.24.

That said, you recently posted an -mm patchset that first reverts
2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b and then goes on to address
the "slow writes for concurrent large and small file writes" bug:
http://lkml.org/lkml/2008/1/15/132

For those interested in using your writeback improvements in
production sooner rather than later (primarily with ext3); what
recommendations do you have?  Just heavily test our own 2.6.24 + your
evolving "close, but not ready for merge" -mm writeback patchset?

regards,
Mike

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote:
> 
> On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote:
> > Am Montag, 14. Januar 2008 schrieb Fengguang Wu:
> > 
> > > Joerg, this patch fixed the bug for me :-)
> > 
> > Fengguang, congratulations, I can confirm that your patch fixed the bug! With 
> > previous kernels the bug showed up after each reboot. Now, when booting the 
> > patched kernel everything is fine and there is no longer any suspicious 
> > iowait!
> > 
> > Do you have an idea why this problem appeared in 2.6.24? Did somebody change 
> > the ext2 code or is it related to the changes in the scheduler?
> 
> It was Fengguang who changed the inode writeback code, and I guess the
> new and improved code was less able do deal with these funny corner
> cases. But he has been very good in tracking them down and solving them,
> kudos to him for that work!

Thank you.

In particular the bug is triggered by the patch named:
        "writeback: introduce writeback_control.more_io to indicate more io"
That patch means to speed up writeback, but unfortunately its
aggressiveness has disclosed bugs in reiserfs, jfs and now ext2.

Linus, given the number of bugs it triggered, I'd recommend revert
this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's
push it back to -mm tree for more testings?

Regards,
Fengguang

Re: regression: 100% io-wait with 2.6.24-rcX

[Peter Zijlstra]

On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote:
> Am Montag, 14. Januar 2008 schrieb Fengguang Wu:
> 
> > Joerg, this patch fixed the bug for me :-)
> 
> Fengguang, congratulations, I can confirm that your patch fixed the bug! With 
> previous kernels the bug showed up after each reboot. Now, when booting the 
> patched kernel everything is fine and there is no longer any suspicious 
> iowait!
> 
> Do you have an idea why this problem appeared in 2.6.24? Did somebody change 
> the ext2 code or is it related to the changes in the scheduler?

It was Fengguang who changed the inode writeback code, and I guess the
new and improved code was less able do deal with these funny corner
cases. But he has been very good in tracking them down and solving them,
kudos to him for that work!

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Montag, 14. Januar 2008 schrieb Fengguang Wu:

> Joerg, this patch fixed the bug for me :-)

Fengguang, congratulations, I can confirm that your patch fixed the bug! With 
previous kernels the bug showed up after each reboot. Now, when booting the 
patched kernel everything is fine and there is no longer any suspicious 
iowait!

Do you have an idea why this problem appeared in 2.6.24? Did somebody change 
the ext2 code or is it related to the changes in the scheduler?

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Mon, Jan 14, 2008 at 11:54:39AM +0800, Fengguang Wu wrote:
> > particular this bug is triggered because the dir mapping page has
> > PAGECACHE_TAG_DIRTY set and PG_dirty cleared, staying in an
> > inconsistent state.
> 
> Just found that a deleted dir will enter that inconsistent state when
> someone still have reference to it...

Joerg, this patch fixed the bug for me :-)

Fengguang
---

clear PAGECACHE_TAG_DIRTY for truncated page in block_write_full_page()

The `truncated' page in block_write_full_page() may stick for a long time.
E.g. ext2_rmdir() will set i_size to 0. The dir may still be referenced by
someone, and have dirty pages in it.

So clear PAGECACHE_TAG_DIRTY to prevent pdflush from retrying and iowaiting on
it.

Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
fs/buffer.c |    2 ++
 1 files changed, 2 insertions(+)

Index: linux/fs/buffer.c
===================================================================
--- linux.orig/fs/buffer.c
+++ linux/fs/buffer.c
@@ -2820,7 +2820,9 @@ int block_write_full_page(struct page *p
 		 * freeable here, so the page does not leak.
 		 */
 		do_invalidatepage(page, 0);
+		set_page_writeback(page);
 		unlock_page(page);
+		end_page_writeback(page);
 		return 0; /* don't care */
 	}
 

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Sun, Jan 13, 2008 at 07:59:33PM +0800, Fengguang Wu wrote:
> On Sun, Jan 13, 2008 at 10:49:31AM +0100, Joerg Platte wrote:
> > register_jprobe(ext2_writepage) = 0
> > register_jprobe(requeue_io) = 0
> > register_kprobe(submit_bio) = 0
> > requeue_io:
> > inode 114019(sda7/.kde) count 2,2 size 0 pages 1
> > 0	2	0	U____
> > requeue_io:
> > inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
> > 0	2	0	U____
> > requeue_io:
> > inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
> > 0	2	0	U____
> > requeue_io:
> > inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
> > 0	2	0	U____
> 
> It helps. Thank you, Joerg!
> 
> The .kde/cache-ibm/socket-ibm/0266584877 above are directories.
> It's weird that dirs would have their own mappings in ext2. In

Oh, ext2 dirs have their own mapping pages. Not the same with ext3.

> particular this bug is triggered because the dir mapping page has
> PAGECACHE_TAG_DIRTY set and PG_dirty cleared, staying in an
> inconsistent state.

Just found that a deleted dir will enter that inconsistent state when
someone still have reference to it...

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Sun, Jan 13, 2008 at 10:49:31AM +0100, Joerg Platte wrote:
> register_jprobe(ext2_writepage) = 0
> register_jprobe(requeue_io) = 0
> register_kprobe(submit_bio) = 0
> requeue_io:
> inode 114019(sda7/.kde) count 2,2 size 0 pages 1
> 0	2	0	U____
> requeue_io:
> inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
> 0	2	0	U____
> requeue_io:
> inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
> 0	2	0	U____
> requeue_io:
> inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
> 0	2	0	U____

It helps. Thank you, Joerg!

The .kde/cache-ibm/socket-ibm/0266584877 above are directories.
It's weird that dirs would have their own mappings in ext2. In
particular this bug is triggered because the dir mapping page has
PAGECACHE_TAG_DIRTY set and PG_dirty cleared, staying in an
inconsistent state.

Fengguang

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

Am Sonntag, 13. Januar 2008 schrieb Fengguang Wu:
> On Sun, Jan 13, 2008 at 09:05:43AM +0100, Joerg Platte wrote:
> > Am Sonntag, 13. Januar 2008 schrieb Fengguang Wu:
> > > No idea yet :-/ I'm afraid I have to trouble you again - the bug just
> > > refused to appear in my system. I prepared a kernel module for you to
> > > gather more information:
> > >
> > > make && insmod ext2-writeback-debug.ko && sleep 1s && rmmod
> > > ext2-writeback-debug dmesg > ext2-writeback-debug.dmesg
> >
> > Unfortunately, I unable to compile the module:
> >
> > make -C /lib/modules/2.6.24-rc7-ext2/build SUBDIRS=/export/src modules
> > make[1]: Entering directory `/export/src/linux-2.6.24-rc7'
> >   CC [M]  /export/src/ext2-writeback-debug.o
> > /export/src/ext2-writeback-debug.c:89: error: variable ‘my_kprobe’ has
> > initializer but incomplete type
>
> Do you have CONFIG_KPROBES=y?

Now I have :)

Here is the result.

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605


[-- Attachment #2: ext2-writeback-debug.dmesg --]
[-- Type: text/plain, Size: 49111 bytes --]

Linux version 2.6.24-rc7-ext2 (root@ibm) (gcc version 4.2.3 20080102 (prerelease) (Debian 4.2.2-5)) #1 PREEMPT Sun Jan 13 09:48:55 CET 2008
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000d2000 - 00000000000d4000 (reserved)
 BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000005ff50000 (usable)
 BIOS-e820: 000000005ff50000 - 000000005ff67000 (ACPI data)
 BIOS-e820: 000000005ff67000 - 000000005ff79000 (ACPI NVS)
 BIOS-e820: 000000005ff80000 - 0000000060000000 (reserved)
 BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
639MB HIGHMEM available.
896MB LOWMEM available.
Entering add_active_range(0, 0, 393040) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   229376
  HighMem    229376 ->   393040
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0:        0 ->   393040
On node 0 totalpages: 393040
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 1760 pages used for memmap
  Normal zone: 223520 pages, LIFO batch:31
  HighMem zone: 1278 pages used for memmap
  HighMem zone: 162386 pages, LIFO batch:31
  Movable zone: 0 pages used for memmap
DMI present.
ACPI: RSDP 000F6D70, 0024 (r2 IBM   )
ACPI: XSDT 5FF5A672, 004C (r1 IBM    TP-1R        3230  LTP        0)
ACPI: FACP 5FF5A700, 00F4 (r3 IBM    TP-1R        3230 IBM         1)
ACPI Warning (tbfadt-0442): Optional field "Gpe1Block" has zero address or length: 000000000000102C/0 [20070126]
ACPI: DSDT 5FF5A8E7, C530 (r1 IBM    TP-1R        3230 MSFT  100000E)
ACPI: FACS 5FF78000, 0040
ACPI: SSDT 5FF5A8B4, 0033 (r1 IBM    TP-1R        3230 MSFT  100000E)
ACPI: ECDT 5FF66E17, 0052 (r1 IBM    TP-1R        3230 IBM         1)
ACPI: TCPA 5FF66E69, 0032 (r1 IBM    TP-1R        3230 PTL         1)
ACPI: BOOT 5FF66FD8, 0028 (r1 IBM    TP-1R        3230  LTP        1)
ACPI: PM-Timer IO Port: 0x1008
Allocating PCI resources starting at 70000000 (gap: 60000000:9f800000)
swsusp: Registered nosave memory region: 000000000009f000 - 00000000000a0000
swsusp: Registered nosave memory region: 00000000000a0000 - 00000000000d2000
swsusp: Registered nosave memory region: 00000000000d2000 - 00000000000d4000
swsusp: Registered nosave memory region: 00000000000d4000 - 00000000000dc000
swsusp: Registered nosave memory region: 00000000000dc000 - 0000000000100000
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 389970
Kernel command line: root=LABEL=ROOT ro resume=LABEL=SWAP panic=10 usbcore.autosuspend=1 hpet=force 
Unknown boot option `usbcore.autosuspend=1': ignoring
Local APIC disabled by BIOS -- you can enable it with "lapic"
mapped APIC to ffffb000 (01c06000)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0343000 soft=c0342000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Detected 1598.683 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 1547564k/1572160k available (1494k kernel code, 23352k reserved, 597k data, 204k init, 654656k highmem)
virtual kernel memory layout:
    fixmap  : 0xfffa8000 - 0xfffff000   ( 348 kB)
    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
    vmalloc : 0xf8800000 - 0xff7fe000   ( 111 MB)
    lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
      .init : 0xc030c000 - 0xc033f000   ( 204 kB)
      .data : 0xc0275a5f - 0xc030aef0   ( 597 kB)
      .text : 0xc0100000 - 0xc0275a5f   (1494 kB)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
SLUB: Genslabs=11, HWalign=64, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Calibrating delay using timer specific routine.. 3199.89 BogoMIPS (lpj=5331036)
Security Framework initialized
Capability LSM initialized
Mount-cache hash table entries: 512
CPU: After generic identify, caps: a7e9f9bf 00000000 00000000 00000000 00000180 00000000 00000000 00000000
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 1024K
CPU: After all inits, caps: a7e9f9bf 00000000 00000000 00002040 00000180 00000000 00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
Compat vDSO mapped to ffffe000.
CPU: Intel(R) Pentium(R) M processor 1600MHz stepping 05
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 0k freed
ACPI: Core revision 20070126
ACPI: setting ELCR to 0200 (from 0800)
net_namespace: 64 bytes
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfd8d6, last bus=8
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: EC: EC description table is found, configuring boot EC
ACPI: EC: non-query interrupt received, switching to interrupt mode
ACPI: Interpreter enabled
ACPI: (supports S0 S3 S4 S5)
ACPI: Using PIC for interrupt routing
ACPI: EC: GPE = 0x1c, I/O: command/status = 0x66, data = 0x62
ACPI: EC: driver started in poll mode
ACPI: PCI Root Bridge [PCI0] (0000:00)
Force enabled HPET at base address 0xfed00000
PCI quirk: region 1000-107f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 1180-11bf claimed by ICH4 GPIO
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: EC: non-query interrupt received, switching to interrupt mode
ACPI: Power Resource [PUBS] (on)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnpacpi: exceeded the max number of mem resources: 12
pnpacpi: exceeded the max number of mem resources: 12
pnpacpi: exceeded the max number of mem resources: 12
pnpacpi: exceeded the max number of mem resources: 12
pnpacpi: exceeded the max number of IO resources: 24 
pnp: PnP ACPI: found 13 devices
ACPI: ACPI bus type pnp unregistered
PnPBIOS: Disabled by ACPI PNP
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
hpet clockevent registered
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
hpet0: 3 64-bit timers, 14318180 Hz
ACPI: RTC can wake from S4
Time: tsc clocksource has been installed.
system 00:00: iomem range 0x0-0x9ffff could not be reserved
system 00:00: iomem range 0xc0000-0xc3fff could not be reserved
system 00:00: iomem range 0xc4000-0xc7fff could not be reserved
system 00:00: iomem range 0xc8000-0xcbfff could not be reserved
system 00:00: iomem range 0xcc000-0xcffff could not be reserved
system 00:00: iomem range 0xd0000-0xd3fff could not be reserved
system 00:00: iomem range 0x0-0x0 could not be reserved
system 00:00: iomem range 0x0-0x0 could not be reserved
system 00:00: iomem range 0xdc000-0xdffff has been reserved
system 00:00: iomem range 0xe0000-0xe3fff could not be reserved
system 00:00: iomem range 0xe4000-0xe7fff could not be reserved
system 00:00: iomem range 0xe8000-0xebfff could not be reserved
system 00:02: ioport range 0x1000-0x107f has been reserved
system 00:02: ioport range 0x1180-0x11bf has been reserved
system 00:02: ioport range 0x15e0-0x15ef has been reserved
system 00:02: ioport range 0x1600-0x162f has been reserved
system 00:02: ioport range 0x1632-0x167f has been reserved
PCI: Bridge: 0000:00:01.0
  IO window: 3000-3fff
  MEM window: c0100000-c01fffff
  PREFETCH window: e0000000-e7ffffff
PCI: Bus 3, cardbus bridge: 0000:02:00.0
  IO window: 00004000-000040ff
  IO window: 00004400-000044ff
  PREFETCH window: e8000000-ebffffff
  MEM window: c4000000-c7ffffff
PCI: Bus 7, cardbus bridge: 0000:02:00.1
  IO window: 00004800-000048ff
  IO window: 00004c00-00004cff
  PREFETCH window: ec000000-efffffff
  MEM window: c8000000-cbffffff
PCI: Bridge: 0000:00:1e.0
  IO window: 4000-8fff
  MEM window: c0200000-cfffffff
  PREFETCH window: e8000000-efffffff
PCI: Setting latency timer of device 0000:00:1e.0 to 64
ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
PCI: setting IRQ 11 as level-triggered
ACPI: PCI Interrupt 0000:02:00.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
ACPI: PCI Interrupt 0000:02:00.1[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
checking if image is initramfs... it is
Switched to high resolution mode on CPU 0
Freeing initrd memory: 6682k freed
Simple Boot Flag at 0x35 set to 0x1
highmem bounce pool size: 64 pages
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered
io scheduler cfq registered (default)
Boot video device is 0000:01:00.0
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Real Time Clock Driver v1.12ac
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
cpuidle: using governor ladder
cpuidle: using governor menu
Using IPI Shortcut mode
registered taskstats version 1
Freeing unused kernel memory: 204k freed
input: AT Translated Set 2 keyboard as /class/input/input0
ACPI: PCI Interrupt 0000:01:00.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11
radeonfb: Retrieved PLL infos from BIOS
radeonfb: Reference=27.00 MHz (RefDiv=12) Memory=252.00 Mhz, System=200.00 MHz
radeonfb: PLL min 20000 max 35000
i2c-adapter i2c-1: unable to read EDID block.
i2c-adapter i2c-1: unable to read EDID block.
i2c-adapter i2c-1: unable to read EDID block.
i2c-adapter i2c-3: unable to read EDID block.
i2c-adapter i2c-3: unable to read EDID block.
i2c-adapter i2c-3: unable to read EDID block.
Non-DDC laptop panel detected
i2c-adapter i2c-2: unable to read EDID block.
i2c-adapter i2c-2: unable to read EDID block.
i2c-adapter i2c-2: unable to read EDID block.
i2c-adapter i2c-3: unable to read EDID block.
i2c-adapter i2c-3: unable to read EDID block.
i2c-adapter i2c-3: unable to read EDID block.
radeonfb: Monitor 1 type LCD found
radeonfb: Monitor 2 type no found
radeonfb: panel ID string: SXGA+ Single (85MHz)    
radeonfb: detected LVDS panel size from BIOS: 1400x1050
radeondb: BIOS provided dividers will be used
radeonfb: Dynamic Clock Power Management enabled
radeonfb: IBM Thinkpad T40p detected, enabling workaround
radeonfb (0000:01:00.0): ATI Radeon Lf 
Console: switching to colour frame buffer device 175x65
Non-volatile memory driver v1.2
thinkpad_acpi: ThinkPad ACPI Extras v0.17
thinkpad_acpi: http://ibm-acpi.sf.net/
thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
thinkpad_acpi: IBM ThinkPad T40p
input: ThinkPad Extra Buttons as /class/input/input1
NET: Registered protocol family 1
ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: Processor [CPU] (supports 8 throttling states)
Marking TSC unstable due to: TSC halts in idle.
Time: hpet clocksource has been installed.
ACPI: Thermal Zone [THM0] (52 C)
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 11, io base 0x00001800
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:1d.1[B] -> Link [LNKD] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 11, io base 0x00001820
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
SCSI subsystem initialized
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:1d.2[C] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.2: irq 11, io base 0x00001840
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
Intel(R) PRO/1000 Network Driver - version 7.3.20-k2
Copyright (c) 1999-2006 Intel Corporation.
libata version 3.00 loaded.
ACPI: PCI Interrupt Link [LNKH] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:1d.7[D] -> Link [LNKH] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 4
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 11, io mem 0xc0000000
usb 1-1: new full speed USB device using uhci_hcd and address 2
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 6 ports detected
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
ACPI: PCI Interrupt 0000:02:01.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11
usb 4-1: new high speed USB device using ehci_hcd and address 2
usb 4-1: configuration #1 chosen from 1 choice
hub 4-1:1.0: USB hub found
hub 4-1:1.0: 4 ports detected
e1000: 0000:02:01.0: e1000_validate_option: PHY Smart Power Down Enabled
e1000: 0000:02:01.0: e1000_probe: (PCI:33MHz:32-bit) 00:16:41:10:19:11
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ata_piix 0000:00:1f.1: version 2.12
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI Interrupt 0000:00:1f.1[A] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1f.1 to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x1860 irq 14
ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x1868 irq 15
usb 4-1.1: new high speed USB device using ehci_hcd and address 3
usb 4-1.1: configuration #1 chosen from 1 choice
ata1.00: ATA-7: SAMSUNG HM160JC, AP100-16, max UDMA/100
ata1.00: 312581808 sectors, multi 16: LBA48 
ata1.00: configured for UDMA/100
usb 4-1.4: new low speed USB device using ehci_hcd and address 4
usb 4-1.4: configuration #1 chosen from 1 choice
usbcore: registered new interface driver hiddev
input: Logitech USB-PS/2 Optical Mouse as /class/input/input2
input: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:1d.7-1.4
usbcore: registered new interface driver usbhid
drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver
ata2.00: ATAPI: UJDA745 DVD/CDRW, 1.06, max UDMA/33
ata2.00: configured for UDMA/33
scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG HM160JC  AP10 PQ: 0 ANSI: 5
scsi 1:0:0:0: CD-ROM            MATSHITA UJDA745 DVD/CDRW 1.06 PQ: 0 ANSI: 5
Driver 'sd' needs updating - please use bus_type methods
sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda:<4>Driver 'sr' needs updating - please use bus_type methods
 sda1 sda2 < sda5 sda6 sda7 sda8 >
sd 0:0:0:0: [sda] Attached SCSI disk
sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
sd 0:0:0:0: Attached scsi generic sg0 type 0
sr 1:0:0:0: Attached scsi generic sg1 type 5
Attempting manual resume
swsusp: Marking nosave pages: 000000000009f000 - 0000000000100000
swsusp: Basic memory bitmaps created
swsusp: Basic memory bitmaps freed
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Linux agpgart interface v0.102
agpgart: Detected an Intel 855PM Chipset.
agpgart: AGP aperture is 256M @ 0xd0000000
input: Power Button (FF) as /class/input/input3
ACPI: Power Button (FF) [PWRF]
input: Lid Switch as /class/input/input4
ACPI: Lid Switch [LID]
input: Sleep Button (CM) as /class/input/input5
ACPI: Sleep Button (CM) [SLPB]
input: Video Bus as /class/input/input6
ACPI: Video Device [VID] (multi-head: yes  rom: no  post: no)
ACPI: AC Adapter [AC] (on-line)
ACPI: Battery Slot [BAT0] (battery present)
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.02 (26-Jul-2007)
iTCO_wdt: Found a ICH4-M TCO device (Version=1, TCOBASE=0x1060)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
intel_rng: FWH not detected
ACPI: PCI Interrupt 0000:00:1f.3[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
DVB: registering new adapter (TerraTec/qanu USB2.0 Highspeed DVB-T Receiver)
input: TerraTec/qanu USB2.0 Highspeed DVB-T Receiver remote control as /class/input/input7
usbcore: registered new interface driver cinergyT2
input: PC Speaker as /class/input/input8
Yenta: CardBus bridge found at 0000:02:00.0 [1014:0512]
Yenta: Using INTVAL to route CSC interrupts to PCI
Yenta: Routing CardBus interrupts to PCI
Yenta TI: socket 0000:02:00.0, mfunc 0x01d21022, devctl 0x64
Serial: 8250/16550 driver $Revision: 1.90 $ 2 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
Yenta: ISA IRQ mask 0x04b8, PCI irq 11
Socket status: 30000006
pcmcia: parent PCI bridge I/O window: 0x4000 - 0x8fff
cs: IO port probe 0x4000-0x8fff: clean.
pcmcia: parent PCI bridge Memory window: 0xc0200000 - 0xcfffffff
pcmcia: parent PCI bridge Memory window: 0xe8000000 - 0xefffffff
00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
NET: Registered protocol family 23
Yenta: CardBus bridge found at 0000:02:00.1 [1014:0512]
Yenta: Using INTVAL to route CSC interrupts to PCI
Yenta: Routing CardBus interrupts to PCI
Yenta TI: socket 0000:02:00.1, mfunc 0x01d21022, devctl 0x64
Synaptics Touchpad, model: 1, fw: 5.9, id: 0x2c6ab1, caps: 0x884793/0x0
serio: Synaptics pass-through port at isa0060/serio1/input0
input: SynPS/2 Synaptics TouchPad as /class/input/input9
Yenta: ISA IRQ mask 0x04b8, PCI irq 11
Socket status: 30000006
pcmcia: parent PCI bridge I/O window: 0x4000 - 0x8fff
cs: IO port probe 0x4000-0x8fff: clean.
pcmcia: parent PCI bridge Memory window: 0xc0200000 - 0xcfffffff
pcmcia: parent PCI bridge Memory window: 0xe8000000 - 0xefffffff
parport_pc 00:0b: reported by Plug and Play ACPI
parport0: PC-style at 0x3bc (0x7bc), irq 7, dma 0 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
nsc-ircc 00:0c: activated
nsc-ircc, chip->init
nsc-ircc, Found chip at base=0x02e
nsc-ircc, driver loaded (Dag Brattli)
IrDA: Registered device irda0
nsc-ircc, Using dongle: IBM31T1100 or Temic TFDS6000/TFDS6500
ACPI: PCI Interrupt 0000:00:1f.6[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
ACPI: PCI interrupt for device 0000:00:1f.6 disabled
ACPI: PCI Interrupt 0000:00:1f.5[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1f.5 to 64
cs: IO port probe 0xc00-0xcff: clean.
cs: IO port probe 0x800-0x8ff: clean.
cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7
cs: IO port probe 0xa00-0xaff: clean.
cs: IO port probe 0xc00-0xcff: clean.
cs: IO port probe 0x800-0x8ff: clean.
cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7
cs: IO port probe 0xa00-0xaff: clean.
intel8x0_measure_ac97_clock: measured 53254 usecs
intel8x0: measured clock 206 rejected
intel8x0: clocking to 48000
ACPI: PCI Interrupt 0000:00:1f.6[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1f.6 to 64
Adding 1951856k swap on /dev/sda5.  Priority:-1 extents:1 across:1951856k
EXT3 FS on sda6, internal journal
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
IrCOMM protocol (Dag Brattli)
io scheduler anticipatory registered
io scheduler deadline registered
Ebtables v2.0 registered
Bridge firewalling registered
fuse init (API version 7.9)
EXT2-fs warning (device sda7): ext2_fill_super: mounting ext3 filesystem as ext2
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda8, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
NTFS driver 2.1.29 [Flags: R/O MODULE].
NTFS volume version 3.1.
IBM TrackPoint firmware: 0x0e, buttons: 3/3
input: TPPS/2 IBM TrackPoint as /class/input/input10
NET: Registered protocol family 15
padlock: VIA PadLock Hash Engine not detected.
padlock: VIA PadLock Hash Engine not detected.
padlock: VIA PadLock not detected.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
Clocksource tsc unstable (delta = -197922093 ns)
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
NET: Registered protocol family 17
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
Bluetooth: Core ver 2.11
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.9
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM TTY layer initialized
Bluetooth: RFCOMM ver 1.8
vmmon: disagrees about version of symbol struct_module
vmnet: disagrees about version of symbol struct_module
[drm] Initialized drm 1.1.0 20060810
[drm] Initialized radeon 1.28.0 20060524 on minor 0
agpgart: Found an AGP 2.0 compliant device at 0000:00:00.0.
agpgart: Putting AGP V2 device at 0000:00:00.0 into 4x mode
agpgart: Putting AGP V2 device at 0000:01:00.0 into 4x mode
[drm] Setting GART location based on new memory map
[drm] Loading R200 Microcode
[drm] writeback test succeeded in 2 usecs
register_jprobe(ext2_writepage) = 0
register_jprobe(requeue_io) = 0
register_kprobe(submit_bio) = 0
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	2	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	2	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	2	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	2	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	3	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	3	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	3	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	3	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	4	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	4	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	4	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	4	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	5	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	5	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	5	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	5	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	6	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	6	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	6	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	6	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	7	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	7	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	7	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	7	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	8	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	8	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	8	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	8	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	9	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	9	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	9	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	9	0	U____
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd]
 [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd]
 [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd]
 [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd]
 [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	10	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	10	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	10	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	10	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	11	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	11	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	11	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	11	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	12	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	12	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	12	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	12	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	13	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	13	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	13	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	13	0	U____
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c014007b>] devm_free_irq+0x4f/0x61
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd]
 [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c014007b>] devm_free_irq+0x4f/0x61
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd]
 [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c014007b>] devm_free_irq+0x4f/0x61
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd]
 [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c014007b>] devm_free_irq+0x4f/0x61
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd]
 [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	14	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	14	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	14	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	14	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	15	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	15	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	15	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	15	0	U____
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<f99e1140>] __journal_file_buffer+0x8d/0x10d [jbd]
 [<c0272baf>] int3+0x27/0x2c
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e3083>] journal_commit_transaction+0x791/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1
 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug]
 [<c02737f2>] kprobe_exceptions_notify+0x151/0x368
 [<c0274276>] notifier_call_chain+0x2a/0x52
 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41
 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a
 [<c012e29a>] notify_die+0x30/0x34
 [<c0272ca9>] do_int3+0x2f/0x6e
 [<c0272baf>] int3+0x27/0x2c
 [<c01acf12>] submit_bio+0x1/0xfb
 [<c017d77c>] submit_bh+0xb9/0xd4
 [<f99e3083>] journal_commit_transaction+0x791/0xcca [jbd]
 [<c01152a4>] update_curr+0x52/0xc8
 [<c011534d>] set_next_entity+0x11/0x38
 [<c02714a1>] schedule+0x2f9/0x33f
 [<f99e5cf0>] kjournald+0xbc/0x223 [jbd]
 [<c012a91b>] autoremove_wake_function+0x0/0x33
 [<f99e5c34>] kjournald+0x0/0x223 [jbd]
 [<c012a861>] kthread+0x36/0x5d
 [<c012a82b>] kthread+0x0/0x5d
 [<c010493b>] kernel_thread_helper+0x7/0x10
 =======================
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	16	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	16	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	16	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	16	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	17	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	17	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	17	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	17	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	18	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	18	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	18	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	18	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	19	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	19	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	19	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	19	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	20	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	20	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	20	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	20	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	21	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	21	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	21	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	21	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	22	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	22	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	22	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	22	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	23	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	23	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	23	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	23	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	24	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	24	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	24	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	24	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	25	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	25	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	25	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	25	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	26	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	26	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	26	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	26	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	27	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	27	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	27	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	27	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	28	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	28	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	28	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	28	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	29	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	29	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	29	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	29	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	30	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	30	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	30	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	30	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	31	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	31	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	31	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	31	0	U____
register_jprobe(ext2_writepage) = 0
register_jprobe(requeue_io) = 0
register_kprobe(submit_bio) = 0
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	32	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	32	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	32	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	32	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	33	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	33	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	33	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	33	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	34	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	34	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	34	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	34	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	35	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	35	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	35	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	35	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	36	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	36	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	36	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	36	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	37	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	37	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	37	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	37	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	38	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	38	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	38	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	38	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	39	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	39	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	39	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	39	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	40	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	40	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	40	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	40	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	41	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	41	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	41	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	41	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	42	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	42	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	42	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	42	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	43	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	43	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	43	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	43	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	44	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	44	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	44	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	44	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	45	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	45	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	45	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	45	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	46	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	46	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	46	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	46	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	47	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	47	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	47	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	47	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	48	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	48	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	48	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	48	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	49	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	49	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	49	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	49	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	50	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	50	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	50	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	50	0	U____
requeue_io:
inode 114019(sda7/.kde) count 2,2 size 0 pages 1
0	51	0	U____
requeue_io:
inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1
0	51	0	U____
requeue_io:
inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1
0	51	0	U____
requeue_io:
inode 114017(sda7/0266584877) count 3,6 size 0 pages 1
0	51	0	U____

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Sun, Jan 13, 2008 at 09:05:43AM +0100, Joerg Platte wrote:
> Am Sonntag, 13. Januar 2008 schrieb Fengguang Wu:
> 
> > No idea yet :-/ I'm afraid I have to trouble you again - the bug just
> > refused to appear in my system. I prepared a kernel module for you to
> > gather more information:
> 
> > make && insmod ext2-writeback-debug.ko && sleep 1s && rmmod
> > ext2-writeback-debug dmesg > ext2-writeback-debug.dmesg
> 
> Unfortunately, I unable to compile the module:
 
> make -C /lib/modules/2.6.24-rc7-ext2/build SUBDIRS=/export/src modules
> make[1]: Entering directory `/export/src/linux-2.6.24-rc7'
>   CC [M]  /export/src/ext2-writeback-debug.o
> /export/src/ext2-writeback-debug.c:89: error: variable ‘my_kprobe’ has 
> initializer but incomplete type

Do you have CONFIG_KPROBES=y?

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Sonntag, 13. Januar 2008 schrieb Fengguang Wu:

> No idea yet :-/ I'm afraid I have to trouble you again - the bug just
> refused to appear in my system. I prepared a kernel module for you to
> gather more information:

> make && insmod ext2-writeback-debug.ko && sleep 1s && rmmod
> ext2-writeback-debug dmesg > ext2-writeback-debug.dmesg

Unfortunately, I unable to compile the module:

make -C /lib/modules/2.6.24-rc7-ext2/build SUBDIRS=/export/src modules
make[1]: Entering directory `/export/src/linux-2.6.24-rc7'
  CC [M]  /export/src/ext2-writeback-debug.o
/export/src/ext2-writeback-debug.c:89: error: variable ‘my_kprobe’ has 
initializer but incomplete type
/export/src/ext2-writeback-debug.c:90: error: unknown field ‘pre_handler’ 
specified in initializer
/export/src/ext2-writeback-debug.c:90: warning: excess elements in struct 
initializer
/export/src/ext2-writeback-debug.c:90: warning: (near initialization 
for ‘my_kprobe’)
/export/src/ext2-writeback-debug.c:91: error: unknown field ‘post_handler’ 
specified in initializer
/export/src/ext2-writeback-debug.c:91: warning: excess elements in struct 
initializer
/export/src/ext2-writeback-debug.c:91: warning: (near initialization 
for ‘my_kprobe’)
/export/src/ext2-writeback-debug.c:92: error: unknown field ‘symbol_name’ 
specified in initializer
/export/src/ext2-writeback-debug.c:93: warning: excess elements in struct 
initializer
/export/src/ext2-writeback-debug.c:93: warning: (near initialization 
for ‘my_kprobe’)
/export/src/ext2-writeback-debug.c:109: error: 
variable ‘jprobe_ext2_writepage’ has initializer but incomplete type
/export/src/ext2-writeback-debug.c:110: error: unknown field ‘entry’ specified 
in initializer
/export/src/ext2-writeback-debug.c:110: warning: excess elements in struct 
initializer
/export/src/ext2-writeback-debug.c:110: warning: (near initialization 
for ‘jprobe_ext2_writepage’)
/export/src/ext2-writeback-debug.c:111: error: unknown field ‘kp’ specified in 
initializer
/export/src/ext2-writeback-debug.c:112: warning: excess elements in struct 
initializer
/export/src/ext2-writeback-debug.c:112: warning: (near initialization 
for ‘jprobe_ext2_writepage’)
/export/src/ext2-writeback-debug.c:124: error: variable ‘jprobe_requeue_io’ 
has initializer but incomplete type
/export/src/ext2-writeback-debug.c:125: error: unknown field ‘entry’ specified 
in initializer
/export/src/ext2-writeback-debug.c:125: warning: excess elements in struct 
initializer
/export/src/ext2-writeback-debug.c:125: warning: (near initialization 
for ‘jprobe_requeue_io’)
/export/src/ext2-writeback-debug.c:126: error: unknown field ‘kp’ specified in 
initializer
/export/src/ext2-writeback-debug.c:127: warning: excess elements in struct 
initializer
/export/src/ext2-writeback-debug.c:127: warning: (near initialization 
for ‘jprobe_requeue_io’)
/export/src/ext2-writeback-debug.c: In function ‘setup_kprobe’:
/export/src/ext2-writeback-debug.c:132: error: dereferencing pointer to 
incomplete type
/export/src/ext2-writeback-debug.c: In function ‘setup_jprobe’:
/export/src/ext2-writeback-debug.c:139: error: dereferencing pointer to 
incomplete type
make[2]: *** [/export/src/ext2-writeback-debug.o] Error 1
make[1]: *** [_module_/export/src] Error 2
make[1]: Leaving directory `/export/src/linux-2.6.24-rc7'
make: *** [default] Error 2

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

[-- Attachment #1: Type: text/plain, Size: 2118 bytes --]

On Sun, Jan 13, 2008 at 12:32:30AM +0100, Joerg Platte wrote:
> Am Freitag, 11. Januar 2008 schrieb Fengguang Wu:
> > On Thu, Jan 10, 2008 at 11:03:05AM +0100, Joerg Platte wrote:
> > > Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu:
> > > > > problem, because the iowait problem disappeared today after the
> > > > > regular Debian update. I'll try to install the old package versions
> > > > > to make it show up again. Maybe that helps to debug it.
> > > >
> > > > Thank you. I'm running sid, ext2 as rootfs now ;-)
> > >
> > > The error is back and I'm getting thousands of messages like this with
> > > the patched kernel:
> > >
> > > mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc
> > > _M > tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7)
> > > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc
> > > _M > tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7)
> > > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc
> > > _M > tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7)
> >
> > Joerg, what's the output of `dumpe2fs /dev/sda7` and `lsof|grep /tmp`?
> 
> After another reboot I tried to get more information about the konqueror 
> process possibly causing the iowait load by using strace -p. Here is the 
> output:
> 
> gettimeofday({1200180588, 878508}, NULL) = 0
> setitimer(ITIMER_VIRTUAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
> rt_sigaction(SIGVTALRM, {SIG_DFL}, {0xb5cffed0, [VTALRM], SA_RESTART}, 8) = 0
> gettimeofday({1200180588, 879942}, NULL) = 0
> time(NULL)                              = 1200180588
> gettimeofday({1200180588, 880838}, NULL) = 0
> gettimeofday({1200180588, 881284}, NULL) = 0

No idea yet :-/ I'm afraid I have to trouble you again - the bug just
refused to appear in my system. I prepared a kernel module for you to
gather more information:

make && insmod ext2-writeback-debug.ko && sleep 1s && rmmod ext2-writeback-debug
dmesg > ext2-writeback-debug.dmesg

Please do it when 100% iowait appears, and send the dmesg file.

Thank you,
Fengguang

[-- Attachment #2: Makefile --]
[-- Type: text/plain, Size: 187 bytes --]

obj-m := ext2-writeback-debug.o
KDIR  := /lib/modules/$(shell uname -r)/build
PWD   := $(shell pwd)

default:
	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
clean:  
	rm -f *.mod.c *.ko *.o


[-- Attachment #3: ext2-writeback-debug.c --]
[-- Type: text/x-csrc, Size: 3607 bytes --]

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/fs.h>
#include <linux/writeback.h>
#include <linux/kprobes.h>
#include <linux/pagevec.h>

void print_page(struct page *page)
{
	printk(KERN_DEBUG "%lu\t%u\t%u\t%c%c%c%c%c\n",
			page->index,
			page_count(page),
			page_mapcount(page),
			PageUptodate(page)      ? 'U' : '_',
			PageDirty(page)         ? 'D' : '_',
			PageWriteback(page)     ? 'W' : '_',
			PagePrivate(page)       ? 'P' : '_',
			PageLocked(page)        ? 'L' : '_');
}

void print_writeback_control(struct writeback_control *wbc)
{
	printk(KERN_DEBUG
			"global dirty %lu writeback %lu nfs %lu\n"
			"wbc flags %c%c towrite %ld skipped %ld\n",
			global_page_state(NR_FILE_DIRTY),
			global_page_state(NR_WRITEBACK),
			global_page_state(NR_UNSTABLE_NFS),
			wbc->encountered_congestion ? 'C':'_',
			wbc->more_io ? 'M':'_',
			wbc->nr_to_write,
			wbc->pages_skipped);
}

void print_inode_pages(struct inode *inode)
{
	struct address_space *mapping = inode->i_mapping;
	struct pagevec pvec;
	int nr_pages;
	int i;
	pgoff_t index = 0;
	struct dentry *dentry;
	int dcount;
	char *dname;

	nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
			PAGECACHE_TAG_DIRTY,
			(pgoff_t)PAGEVEC_SIZE);

	if (list_empty(&inode->i_dentry)) {
		dname = "";
		dcount = 0;
	} else {
		dentry = list_entry(inode->i_dentry.next,
					struct dentry, d_alias);
		dname = dentry->d_iname;
		dcount = atomic_read(&dentry->d_count);
	}

	printk(KERN_DEBUG "inode %lu(%s/%s) count %d,%d size %llu pages %lu\n",
			inode->i_ino,
			inode->i_sb->s_id,
			dname,
			atomic_read(&inode->i_count),
			dcount,
			i_size_read(inode),
			mapping->nrpages
	      );

	for (i = 0; i < nr_pages; i++)
		print_page(pvec.pages[i]);
}

int handler_pre(struct kprobe *p, struct pt_regs *regs)
{
	static int count = 0;

	if (count++ < 10)
		dump_stack();

	return 0;
}

void handler_post(struct kprobe *p, struct pt_regs *regs, unsigned long flags)
{
}

static struct kprobe my_kprobe = {
	.pre_handler = handler_pre,
	.post_handler = handler_post,
	.symbol_name = "submit_bio"
};


static int jdo_ext2_writepage(struct page *page, struct writeback_control *wbc)
{
	struct inode * const inode = page->mapping->host;

	if (!i_size_read(inode)) {
		printk(KERN_DEBUG "ext2_writepage:\n");
		print_page(page);
		print_writeback_control(wbc);
	}

	jprobe_return();
	return 0;
}
static struct jprobe jprobe_ext2_writepage = {
	.entry = jdo_ext2_writepage,
	.kp.symbol_name = "ext2_writepage"
};


static void jdo_requeue_io(struct inode *inode)
{
	if (!i_size_read(inode)) {
		printk(KERN_DEBUG "requeue_io:\n");
		print_inode_pages(inode);
	}

	jprobe_return();
}
static struct jprobe jprobe_requeue_io = {
	.entry = jdo_requeue_io,
	.kp.symbol_name = "requeue_io"
};

static int setup_kprobe(struct kprobe *kprobe)
{
	int ret = register_kprobe(kprobe);
	printk("register_kprobe(%s) = %d\n", kprobe->symbol_name, ret);
	return ret;
}

static int setup_jprobe(struct jprobe *jprobe)
{
	int ret = register_jprobe(jprobe);
	printk("register_jprobe(%s) = %d\n", jprobe->kp.symbol_name, ret);
	return ret;
}

static int __init jprobe_init(void)
{
	int ret;

	ret = setup_jprobe(&jprobe_ext2_writepage);
	if (ret)
		return -1;

	ret = setup_jprobe(&jprobe_requeue_io);
	if (ret)
		return -1;

	ret = setup_kprobe(&my_kprobe);
	if (ret)
		return -1;

	return 0;
}

static void __exit jprobe_exit(void)
{
	unregister_kprobe(&my_kprobe);
	unregister_jprobe(&jprobe_requeue_io);
	unregister_jprobe(&jprobe_ext2_writepage);
}

module_init(jprobe_init)
module_exit(jprobe_exit)
MODULE_LICENSE("GPL");

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Freitag, 11. Januar 2008 schrieb Fengguang Wu:
> On Thu, Jan 10, 2008 at 11:03:05AM +0100, Joerg Platte wrote:
> > Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu:
> > > > problem, because the iowait problem disappeared today after the
> > > > regular Debian update. I'll try to install the old package versions
> > > > to make it show up again. Maybe that helps to debug it.
> > >
> > > Thank you. I'm running sid, ext2 as rootfs now ;-)
> >
> > The error is back and I'm getting thousands of messages like this with
> > the patched kernel:
> >
> > mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc
> > _M > tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7)
> > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc
> > _M > tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7)
> > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc
> > _M > tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7)
>
> Joerg, what's the output of `dumpe2fs /dev/sda7` and `lsof|grep /tmp`?

After another reboot I tried to get more information about the konqueror 
process possibly causing the iowait load by using strace -p. Here is the 
output:

gettimeofday({1200180588, 878508}, NULL) = 0
setitimer(ITIMER_VIRTUAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
rt_sigaction(SIGVTALRM, {SIG_DFL}, {0xb5cffed0, [VTALRM], SA_RESTART}, 8) = 0
gettimeofday({1200180588, 879942}, NULL) = 0
time(NULL)                              = 1200180588
gettimeofday({1200180588, 880838}, NULL) = 0
gettimeofday({1200180588, 881284}, NULL) = 0
time(NULL)                              = 1200180588
gettimeofday({1200180588, 882131}, NULL) = 0
gettimeofday({1200180588, 882572}, NULL) = 0
ioctl(5, FIONREAD, [0])                 = 0
gettimeofday({1200180588, 883477}, NULL) = 0
select(16, [5 6 7 9 11 14 15], [], [], {0, 150095}) = 0 (Timeout)
gettimeofday({1200180589, 34269}, NULL) = 0
gettimeofday({1200180589, 34885}, NULL) = 0
time(NULL)                              = 1200180589
gettimeofday({1200180589, 36672}, NULL) = 0
rt_sigaction(SIGVTALRM, {0xb5cffed0, [VTALRM], SA_RESTART}, {SIG_DFL}, 8) = 0
setitimer(ITIMER_VIRTUAL, {it_interval={10, 0}, it_value={5, 0}}, 
{it_interval={0, 0}, it_value={0, 0}}) = 0
gettimeofday({1200180589, 38555}, NULL) = 0
time(NULL)                              = 1200180589
gettimeofday({1200180589, 39802}, NULL) = 0
setitimer(ITIMER_VIRTUAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
rt_sigaction(SIGVTALRM, {SIG_DFL}, {0xb5cffed0, [VTALRM], SA_RESTART}, 8) = 0
gettimeofday({1200180589, 40912}, NULL) = 0
time(NULL)                              = 1200180589
gettimeofday({1200180589, 42019}, NULL) = 0
gettimeofday({1200180589, 42458}, NULL) = 0
time(NULL)                              = 1200180589
gettimeofday({1200180589, 43303}, NULL) = 0
gettimeofday({1200180589, 43747}, NULL) = 0
ioctl(5, FIONREAD, [0])                 = 0
gettimeofday({1200180589, 45834}, NULL) = 0
select(16, [5 6 7 9 11 14 15], [], [], {0, 149913}) = 0 (Timeout)
gettimeofday({1200180589, 194815}, NULL) = 0
ioctl(5, FIONREAD, [0])                 = 0
gettimeofday({1200180589, 195730}, NULL) = 0
select(16, [5 6 7 9 11 14 15], [], [], {0, 17}) = 0 (Timeout)
gettimeofday({1200180589, 197555}, NULL) = 0
gettimeofday({1200180589, 198020}, NULL) = 0
time(NULL)                              = 1200180589
gettimeofday({1200180589, 198884}, NULL) = 0
rt_sigaction(SIGVTALRM, {0xb5cffed0, [VTALRM], SA_RESTART}, {SIG_DFL}, 8) = 0
setitimer(ITIMER_VIRTUAL, {it_interval={10, 0}, it_value={5, 0}}, 
{it_interval={0, 0}, it_value={0, 0}}) = 0
gettimeofday({1200180589, 200702}, NULL) = 0
time(NULL)                              = 1200180589
gettimeofday({1200180589, 200806}, NULL) = 0
setitimer(ITIMER_VIRTUAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
rt_sigaction(SIGVTALRM, {SIG_DFL}, {0xb5cffed0, [VTALRM], SA_RESTART}, 8) = 0
gettimeofday({1200180589, 202975}, NULL) = 0
time(NULL)                              = 1200180589
gettimeofday({1200180589, 203837}, NULL) = 0
gettimeofday({1200180589, 204319}, NULL) = 0
time(NULL)                              = 1200180589
gettimeofday({1200180589, 205169}, NULL) = 0
gettimeofday({1200180589, 205613}, NULL) = 0
ioctl(5, FIONREAD, [0])                 = 0
gettimeofday({1200180589, 206515}, NULL) = 0
select(16, [5 6 7 9 11 14 15], [], [], {0, 149098} <unfinished ...>

Fengguang, do you have any idea what's going wrong here?

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Freitag, 11. Januar 2008 schrieb Joerg Platte:

> konqueror   987    jplatte  mem       REG        8,6  2590525
> 1336104 /var/tmp/kdecache-jplatte/ksycoca
> konqueror   987    jplatte   12r      REG        8,6  2590525
> 1336104 /var/tmp/kdecache-jplatte/ksycoca
> konqueror   987    jplatte   13u      REG        8,7      579
> 97731 /tmp/kde-jplatte/konqueror-crash-Jv2u8a.log
> konqueror   987    jplatte   14u      REG        8,7   128289
> 97734 /tmp/kde-jplatte/khtmlcacheA7VjAa.tmp (deleted)
> konqueror   987    jplatte   16u      REG        8,7    43334
> 97750 /tmp/kde-jplatte/khtmlcacheZUNlsc.tmp (deleted)
> konqueror   987    jplatte   22u      REG        8,7      797
> 97751 /tmp/kde-jplatte/khtmlcache76bZYa.tmp (deleted)
> konqueror   987    jplatte   27u      REG        8,7   102347
> 97735 /tmp/kde-jplatte/khtmlcacheMhlDJb.tmp (deleted)
> konqueror   987    jplatte   31u      REG        8,7      354
> 97738 /tmp/kde-jplatte/khtmlcacheq21uoc.tmp (deleted)
> konqueror   987    jplatte   32u      REG        8,7     1360
> 97742 /tmp/kde-jplatte/khtmlcacheo2Ms2a.tmp (deleted)
> konqueror   987    jplatte   34u      REG        8,7     6378
> 97745 /tmp/kde-jplatte/khtmlcacheLETtgc.tmp (deleted)
> konqueror   987    jplatte   35u      REG        8,7    97350
> 97746 /tmp/kde-jplatte/khtmlcache5jit8a.tmp (deleted)
> konqueror   987    jplatte   36u      REG        8,7      354
> 97747 /tmp/kde-jplatte/khtmlcache7VBSNa.tmp (deleted)
> konqueror   987    jplatte   37u      REG        8,7     1360
> 97748 /tmp/kde-jplatte/khtmlcachetSNbub.tmp (deleted)
> konqueror   987    jplatte   38u      REG        8,7     6073
> 97749 /tmp/kde-jplatte/khtmlcacheKAKhuc.tmp (deleted)

Fengguang, a few minutes after killing this konqueror process the high iowait 
load has disappeared. Lets see if it'll come back...

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Freitag, 11. Januar 2008 schrieb Fengguang Wu:

> Joerg, what's the output of `dumpe2fs /dev/sda7` and `lsof|grep /tmp`?

Fengang, here is the output (kernel 2.6.24-rc7 without your patches):

Filesystem volume name:   TMP
Last mounted on:          <not available>
Filesystem UUID:          e23ae961-bbdc-44bc-b662-f28f7314939b
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype 
sparse_super large_file
Filesystem flags:         signed directory hash
Default mount options:    (none)
Filesystem state:         not clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              244320
Block count:              487966
Reserved block count:     24398
Free blocks:              468728
Free inodes:              244162
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      119
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16288
Inode blocks per group:   509
Filesystem created:       Thu Feb  8 18:46:19 2007
Last mount time:          Thu Jan 10 11:09:35 2008
Last write time:          Fri Jan 11 06:12:58 2008
Mount count:              25
Maximum mount count:      39
Last checked:             Tue Dec 11 22:07:03 2007
Check interval:           15552000 (6 months)
Next check after:         Sun Jun  8 23:07:03 2008
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      79e271e9-b874-4e11-92b0-cdb36d07e1c1
Journal backup:           inode blocks
Journal size:             32M


Group 0: (Blocks 0-32767)
  Primary superblock at 0, Group descriptors at 1-1
  Reserved GDT blocks at 2-120
  Block bitmap at 121 (+121), Inode bitmap at 122 (+122)
  Inode table at 123-631 (+123)
  23926 free blocks, 16275 free inodes, 2 directories
  Free blocks: 638-10239, 16384, 18444-30719, 30721-32767
  Free inodes: 14-16288
Group 1: (Blocks 32768-65535)
  Backup superblock at 32768, Group descriptors at 32769-32769
  Reserved GDT blocks at 32770-32888
  Block bitmap at 32889 (+121), Inode bitmap at 32890 (+122)
  Inode table at 32891-33399 (+123)
  32135 free blocks, 16287 free inodes, 1 directories
  Free blocks: 33400-57343, 57345-65535
  Free inodes: 16290-32576
Group 2: (Blocks 65536-98303)
  Block bitmap at 65536 (+0), Inode bitmap at 65537 (+1)
  Inode table at 65538-66046 (+2)
  32256 free blocks, 16286 free inodes, 1 directories
  Free blocks: 66047-96255, 96257-98303
  Free inodes: 32579-48864
Group 3: (Blocks 98304-131071)
  Backup superblock at 98304, Group descriptors at 98305-98305
  Reserved GDT blocks at 98306-98424
  Block bitmap at 98425 (+121), Inode bitmap at 98426 (+122)
  Inode table at 98427-98935 (+123)
  32135 free blocks, 16286 free inodes, 1 directories
  Free blocks: 98936-120831, 120833-131071
  Free inodes: 48867-65152
Group 4: (Blocks 131072-163839)
  Block bitmap at 131072 (+0), Inode bitmap at 131073 (+1)
  Inode table at 131074-131582 (+2)
  32256 free blocks, 16287 free inodes, 1 directories
  Free blocks: 131583-133119, 133121-163839
  Free inodes: 65154-81440
Group 5: (Blocks 163840-196607)
  Backup superblock at 163840, Group descriptors at 163841-163841
  Reserved GDT blocks at 163842-163960
  Block bitmap at 163961 (+121), Inode bitmap at 163962 (+122)
  Inode table at 163963-164471 (+123)
  32135 free blocks, 16286 free inodes, 1 directories
  Free blocks: 164472-182271, 182273-196607
  Free inodes: 81443-97728
Group 6: (Blocks 196608-229375)
  Block bitmap at 196608 (+0), Inode bitmap at 196609 (+1)
  Inode table at 196610-197118 (+2)
  32087 free blocks, 16265 free inodes, 1 directories
  Free blocks: 197119-200703, 200705-210943, 210945-210951, 210970-210975, 
210988-215039, 215041-215047, 215058-215071, 215090-217087, 217089-219135, 
219137-219143, 219177-219199, 219226-219255, 219257-219263, 219265-219271, 
219274-219279, 219305-219335, 219337-219343, 219345-219351, 219354-219359, 
219371-219383, 219385-227327, 227331-229375
  Free inodes: 97752-114016
Group 7: (Blocks 229376-262143)
  Backup superblock at 229376, Group descriptors at 229377-229377
  Reserved GDT blocks at 229378-229496
  Block bitmap at 229497 (+121), Inode bitmap at 229498 (+122)
  Inode table at 229499-230007 (+123)
  32134 free blocks, 16278 free inodes, 1 directories
  Free blocks: 230008-253951, 253953-260095, 260097-262143
  Free inodes: 114022, 114025-114027, 114031-130304
Group 8: (Blocks 262144-294911)
  Block bitmap at 262144 (+0), Inode bitmap at 262145 (+1)
  Inode table at 262146-262654 (+2)
  29809 free blocks, 16193 free inodes, 17 directories
  Free blocks: 262661-264191, 264244, 264250, 264263, 264272, 264281, 
264288-266239, 266870-266871, 266877-268287, 268289-270335, 270965-274431, 
274434-282623, 282625-290815, 291897-292863, 292866-294911
  Free inodes: 130316, 130319, 130388, 130402-130406, 130408-146592
Group 9: (Blocks 294912-327679)
  Backup superblock at 294912, Group descriptors at 294913-294913
  Reserved GDT blocks at 294914-295032
  Block bitmap at 295033 (+121), Inode bitmap at 295034 (+122)
  Inode table at 295035-295543 (+123)
  32135 free blocks, 16286 free inodes, 1 directories
  Free blocks: 295544-313343, 313345-327679
  Free inodes: 146595-162880
Group 10: (Blocks 327680-360447)
  Block bitmap at 327680 (+0), Inode bitmap at 327681 (+1)
  Inode table at 327682-328190 (+2)
  32256 free blocks, 16283 free inodes, 1 directories
  Free blocks: 328191-342015, 342017-360447
  Free inodes: 162886-179168
Group 11: (Blocks 360448-393215)
  Block bitmap at 360448 (+0), Inode bitmap at 360449 (+1)
  Inode table at 360450-360958 (+2)
  32248 free blocks, 16286 free inodes, 1 directories
  Free blocks: 360959-372735, 372744-380927, 380929-393215
  Free inodes: 179171-195456
Group 12: (Blocks 393216-425983)
  Block bitmap at 393216 (+0), Inode bitmap at 393217 (+1)
  Inode table at 393218-393726 (+2)
  32257 free blocks, 16288 free inodes, 0 directories
  Free blocks: 393727-425983
  Free inodes: 195457-211744
Group 13: (Blocks 425984-458751)
  Block bitmap at 425984 (+0), Inode bitmap at 425985 (+1)
  Inode table at 425986-426494 (+2)
  32256 free blocks, 16287 free inodes, 1 directories
  Free blocks: 426495-436223, 436225-458751
  Free inodes: 211746-228032
Group 14: (Blocks 458752-487965)
  Block bitmap at 458752 (+0), Inode bitmap at 458753 (+1)
  Inode table at 458754-459262 (+2)
  28703 free blocks, 16288 free inodes, 0 directories
  Free blocks: 459263-487965
  Free inodes: 228033-244320

konqueror   987    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
konqueror   987    jplatte   12r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
konqueror   987    jplatte   13u      REG        8,7      579      
97731 /tmp/kde-jplatte/konqueror-crash-Jv2u8a.log
konqueror   987    jplatte   14u      REG        8,7   128289      
97734 /tmp/kde-jplatte/khtmlcacheA7VjAa.tmp (deleted)
konqueror   987    jplatte   16u      REG        8,7    43334      
97750 /tmp/kde-jplatte/khtmlcacheZUNlsc.tmp (deleted)
konqueror   987    jplatte   22u      REG        8,7      797      
97751 /tmp/kde-jplatte/khtmlcache76bZYa.tmp (deleted)
konqueror   987    jplatte   27u      REG        8,7   102347      
97735 /tmp/kde-jplatte/khtmlcacheMhlDJb.tmp (deleted)
konqueror   987    jplatte   31u      REG        8,7      354      
97738 /tmp/kde-jplatte/khtmlcacheq21uoc.tmp (deleted)
konqueror   987    jplatte   32u      REG        8,7     1360      
97742 /tmp/kde-jplatte/khtmlcacheo2Ms2a.tmp (deleted)
konqueror   987    jplatte   34u      REG        8,7     6378      
97745 /tmp/kde-jplatte/khtmlcacheLETtgc.tmp (deleted)
konqueror   987    jplatte   35u      REG        8,7    97350      
97746 /tmp/kde-jplatte/khtmlcache5jit8a.tmp (deleted)
konqueror   987    jplatte   36u      REG        8,7      354      
97747 /tmp/kde-jplatte/khtmlcache7VBSNa.tmp (deleted)
konqueror   987    jplatte   37u      REG        8,7     1360      
97748 /tmp/kde-jplatte/khtmlcachetSNbub.tmp (deleted)
konqueror   987    jplatte   38u      REG        8,7     6073      
97749 /tmp/kde-jplatte/khtmlcacheKAKhuc.tmp (deleted)
xfs        5503       root    3u     unix 0xdfdd7e00               
15884 /tmp/.font-unix/fs7100
xfs        5503       root    4u     unix 0xf724ee00               
18389 /tmp/.font-unix/fs7100
Xorg       6031       root    1u     unix 0xf7151000               
17447 /tmp/.X11-unix/X0
Xorg       6031       root    3u     unix 0xf40f5540              
108479 /tmp/.X11-unix/X0
Xorg       6031       root   10u     unix 0xf400ec40              
107863 /tmp/.X11-unix/X0
Xorg       6031       root   13u     unix 0xf724ea80               
18532 /tmp/.X11-unix/X0
Xorg       6031       root   14u     unix 0xf6702000               
18880 /tmp/.X11-unix/X0
Xorg       6031       root   15u     unix 0xf5d6ac40               
19298 /tmp/.X11-unix/X0
Xorg       6031       root   16u     unix 0xf5297c40               
25334 /tmp/.X11-unix/X0
Xorg       6031       root   17u     unix 0xf410b1c0               
31056 /tmp/.X11-unix/X0
Xorg       6031       root   18u     unix 0xf67021c0               
19002 /tmp/.X11-unix/X0
Xorg       6031       root   19u     unix 0xf5c7f700               
19014 /tmp/.X11-unix/X0
Xorg       6031       root   20u     unix 0xf5051380               
20514 /tmp/.X11-unix/X0
Xorg       6031       root   21u     unix 0xf5ccc540               
19100 /tmp/.X11-unix/X0
Xorg       6031       root   22u     unix 0xf5c7fa80               
19086 /tmp/.X11-unix/X0
Xorg       6031       root   23u     unix 0xf725c380               
19123 /tmp/.X11-unix/X0
Xorg       6031       root   24u     unix 0xf5d29380               
19147 /tmp/.X11-unix/X0
Xorg       6031       root   25u     unix 0xf5174700               
31277 /tmp/.X11-unix/X0
Xorg       6031       root   26u     unix 0xf5db3380               
19252 /tmp/.X11-unix/X0
Xorg       6031       root   27u     unix 0xf51ce8c0               
41610 /tmp/.X11-unix/X0
Xorg       6031       root   28u     unix 0xf5de8e00               
19379 /tmp/.X11-unix/X0
Xorg       6031       root   29u     unix 0xf5e0b8c0               
19401 /tmp/.X11-unix/X0
Xorg       6031       root   30u     unix 0xf5e5c540               
19419 /tmp/.X11-unix/X0
Xorg       6031       root   31u     unix 0xf5e9ba80               
19500 /tmp/.X11-unix/X0
Xorg       6031       root   32u     unix 0xf5e0ba80               
19515 /tmp/.X11-unix/X0
Xorg       6031       root   33u     unix 0xf5e7fa80               
19531 /tmp/.X11-unix/X0
Xorg       6031       root   34u     unix 0xf5ea61c0               
19982 /tmp/.X11-unix/X0
Xorg       6031       root   35u     unix 0xf5ea6a80               
19991 /tmp/.X11-unix/X0
Xorg       6031       root   36u     unix 0xf7220000               
20286 /tmp/.X11-unix/X0
Xorg       6031       root   37u     unix 0xf5187700               
54906 /tmp/.X11-unix/X0
Xorg       6031       root   39u     unix 0xf5187e00               
43577 /tmp/.X11-unix/X0
Xorg       6031       root   40u     unix 0xf5ec9a80               
20377 /tmp/.X11-unix/X0
Xorg       6031       root   41u     unix 0xf51b2380               
20809 /tmp/.X11-unix/X0
Xorg       6031       root   42u     unix 0xf71921c0               
20455 /tmp/.X11-unix/X0
Xorg       6031       root   43u     unix 0xf528a1c0               
99356 /tmp/.X11-unix/X0
Xorg       6031       root   44u     unix 0xf5297540               
21298 /tmp/.X11-unix/X0
Xorg       6031       root   45u     unix 0xf5099c40               
20549 /tmp/.X11-unix/X0
Xorg       6031       root   46u     unix 0xf50ff380               
20600 /tmp/.X11-unix/X0
Xorg       6031       root   47u     unix 0xf510c8c0               
20655 /tmp/.X11-unix/X0
Xorg       6031       root   48u     unix 0xf510cc40               
20657 /tmp/.X11-unix/X0
Xorg       6031       root   49u     unix 0xf5138540               
20663 /tmp/.X11-unix/X0
Xorg       6031       root   50u     unix 0xf50cb380               
20683 /tmp/.X11-unix/X0
Xorg       6031       root   51u     unix 0xf51b2e00               
20883 /tmp/.X11-unix/X0
Xorg       6031       root   52u     unix 0xf52c3a80               
21316 /tmp/.X11-unix/X0
Xorg       6031       root   53u     unix 0xf52f3a80               
21373 /tmp/.X11-unix/X0
Xorg       6031       root   54u     unix 0xf528ac40               
21871 /tmp/.X11-unix/X0
Xorg       6031       root   55u     unix 0xdfd301c0               
25107 /tmp/.X11-unix/X0
Xorg       6031       root   56u     unix 0xf67ab380               
25380 /tmp/.X11-unix/X0
Xorg       6031       root   58u     unix 0xf512c700               
49503 /tmp/.X11-unix/X0
Xorg       6031       root   59u     unix 0xf51cee00               
55093 /tmp/.X11-unix/X0
Xorg       6031       root   60u     unix 0xf5187a80               
54949 /tmp/.X11-unix/X0
kontact    6669    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kontact    6669    jplatte   12r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kontact    6669    jplatte   20u     unix 0xf5005000              
109900 /tmp/ksocket-jplatte/kontactb5GSMb.slave-socket
kontact    6669    jplatte   33u     unix 0xf712a700              
108568 /tmp/ksocket-jplatte/kontact7somnc.slave-socket
kontact    6669    jplatte   34u     unix 0xf50d78c0              
108570 /tmp/ksocket-jplatte/kontactQdchTb.slave-socket
kontact    6669    jplatte   36u     unix 0xf50d7000              
108569 /tmp/ksocket-jplatte/kontactdlqZ9b.slave-socket
kio_imap4  6691    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kio_imap4  6691    jplatte   10r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kio_imap4  6698    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kio_imap4  6698    jplatte   10r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kio_imap4  6699    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kio_imap4  6699    jplatte   10r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
gam_serve  6972       root  cwd       DIR        8,7        0     
211745 /tmp/1878472609 (deleted)
ssh-agent  7097    jplatte    3u     unix 0xf724f1c0               
18873 /tmp/ssh-gdxKKv7049/agent.7049
gpg-agent  7098    jplatte    5u     unix 0xf724f540               
18877 /tmp/gpg-cKJSSb/S.gpg-agent
kdeinit    7159    jplatte    9u     unix 0xf5c2ba80               
18966 /tmp/ksocket-jplatte/kdeinit__0
kdeinit    7159    jplatte   10u     unix 0xf5c2bc40               
18968 /tmp/ksocket-jplatte/kdeinit-
dcopserve  7162    jplatte    6u     unix 0xf5c7f380               
19011 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte    8u     unix 0xf5c891c0               
18977 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   11u     unix 0xf5c89e00               
18993 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   12u     unix 0xf5cd6700               
20495 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   13u     unix 0xf5c2b1c0               
19092 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   14u     unix 0xf5d1d540               
19107 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   15u     unix 0xf5dd5e00               
19295 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   16u     unix 0xf725c000               
19118 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   17u     unix 0xf725ce00               
19142 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   18u     unix 0xf5ee9000              
108474 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   20u     unix 0xf5db3000               
19249 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   21u     unix 0xf5db3540               
19264 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   22u     unix 0xf51cea80              
108608 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   23u     unix 0xf5de81c0               
19376 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   24u     unix 0xf7161700               
20283 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   25u     unix 0xf400e000              
108491 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   26u     unix 0xf5e0b540               
19398 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   27u     unix 0xf5e9b700               
19497 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   28u     unix 0xf5e5c1c0               
19416 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   29u     unix 0xf5de8c40               
19541 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   30u     unix 0xf400e700              
108493 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   31u     unix 0xf5ea4a80               
20007 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   32u     unix 0xf5e7f700               
19526 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   33u     unix 0xf5ea6e00               
19993 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   34u     unix 0xf400e380              
108495 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   35u     unix 0xf50ff540              
108497 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   36u     unix 0xf5cb0000              
108499 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   37u     unix 0xf5ec9380               
20374 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   38u     unix 0xf5cb0700              
108501 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   39u     unix 0xf5cb01c0              
108503 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   40u     unix 0xf514b8c0               
20914 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   41u     unix 0xf506be00               
21019 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   42u     unix 0xf5cd6a80               
20506 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   43u     unix 0xf51ce540              
108610 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   44u     unix 0xf5099380               
20542 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   45u     unix 0xf50998c0               
20546 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   46u     unix 0xf5138c40               
20680 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   47u     unix 0xdfd4fc40               
99363 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   49u     unix 0xf50ff000               
20597 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   50u     unix 0xf724f700               
99470 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   51u     unix 0xf50ffe00               
20616 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   54u     unix 0xf50fd8c0               
20623 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   56u     unix 0xf510c380               
20649 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   58u     unix 0xf6702c40              
108612 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   60u     unix 0xf50d7e00               
20764 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   74u     unix 0xf5255540               
43572 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   88u     unix 0xf407da80               
25329 /tmp/.ICE-unix/dcop7162-1199959838
dcopserve  7162    jplatte   89u     unix 0xf6702540               
25375 /tmp/.ICE-unix/dcop7162-1199959838
klauncher  7164    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
klauncher  7164    jplatte   10r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
klauncher  7164    jplatte   11u     unix 0xf5c89540               
18999 /tmp/ksocket-jplatte/klauncherDCSyUa.slave-socket
klauncher  7164    jplatte   12u     unix 0xf5255a80              
110308 /tmp/ksocket-jplatte/klauncherDCSyUa.slave-socket
kded       7166    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kded       7166    jplatte   14r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
ksmserver  7183    jplatte    7u     unix 0xf5ccc8c0               
19076 /tmp/ksocket-jplatte/kdeinit__0
ksmserver  7183    jplatte   12u     unix 0xf5c2b8c0               
19093 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   13u     unix 0xf5d1d1c0               
19102 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   14u     unix 0xf5d1d8c0               
19110 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   15u     unix 0xf725ca80               
19127 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   16u     unix 0xf5d298c0               
19151 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   17u     unix 0xf5db3a80               
19256 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   18u     unix 0xf5c2b000               
19306 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   19u     unix 0xdfdd7700               
19472 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   20u     unix 0xf5c7fc40               
19384 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   21u     unix 0xf5dddc40               
19405 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   22u     unix 0xf712a380               
20290 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   23u     unix 0xf5e5ca80               
19504 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   24u     unix 0xf5e7f1c0               
19517 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   25u     unix 0xf5e7fe00               
19533 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   26u     unix 0xf5ea6700               
19984 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   27u     unix 0xf5ea4380               
19995 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   28u     unix 0xf5d6aa80               
43581 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   29u     unix 0xf40f5700              
108483 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   30u     unix 0xdfd4f1c0               
99358 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   31u     unix 0xf5ee9e00               
20381 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   32u     unix 0xf5005700               
20467 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   34u     unix 0xf512c1c0               
21389 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   35u     unix 0xf50cbc40               
20568 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   36u     unix 0xf50e4000               
20571 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   37u     unix 0xf400e540               
25338 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   38u     unix 0xf50e4700               
20575 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   39u     unix 0xf50ffa80               
20604 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   40u     unix 0xf51381c0               
20665 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   41u     unix 0xf514b380               
20693 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   42u     unix 0xf514b540               
20851 /tmp/.ICE-unix/7183
ksmserver  7183    jplatte   43u     unix 0xf52c3700               
25384 /tmp/.ICE-unix/7183
kicker     7192    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kicker     7192    jplatte   12r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kxkb       7215    jplatte   12r      REG        8,7    11912      
97730 /tmp/kde-jplatte/de..xkm
konqueror  7255    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
konqueror  7255    jplatte   12r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
konqueror  7255    jplatte   13u      REG        8,7      283      
97732 /tmp/kde-jplatte/konqueror-crash-LPnWVb.log
konqueror  7255    jplatte   18u      REG        8,7    67736      
97740 /tmp/kde-jplatte/khtmlcacheJfHsua.tmp (deleted)
konqueror  7255    jplatte   21u      REG        8,7    48368      
97741 /tmp/kde-jplatte/khtmlcache8KkbLa.tmp (deleted)
konqueror  7337    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
konqueror  7337    jplatte   12r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
konqueror  7337    jplatte   13u      REG        8,7      102      
97733 /tmp/kde-jplatte/konqueror-crash-YtoSma.log
konqueror  7337    jplatte   16u      REG        8,7    37907      
97743 /tmp/kde-jplatte/khtmlcacheemjbkb.tmp (deleted)
konqueror  7337    jplatte   22u      REG        8,7    66887      
97744 /tmp/kde-jplatte/khtmlcache1u4D4b.tmp (deleted)
amarokapp  7351    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
amarokapp  7351    jplatte   18r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kopete     7447    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kopete     7447    jplatte   12r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kdeinit    7647       root  cwd       DIR        8,7     4096     
130305 /tmp/1878472609
kdeinit    7647       root    7u     unix 0xf51ce1c0               
21255 /tmp/1878472609/.kde/socket-ibm/kdeinit__0
kdeinit    7647       root    8u     unix 0xf67ab8c0               
21257 /tmp/1878472609/.kde/socket-ibm/kdeinit-
dcopserve  7651       root  cwd       DIR        8,7     4096     
130305 /tmp/1878472609
dcopserve  7651       root    5u     unix 0xf5297000               
21279 /tmp/.ICE-unix/dcop7651-1199959912
dcopserve  7651       root    7u     unix 0xf67abc40               
21265 /tmp/.ICE-unix/dcop7651-1199959912
dcopserve  7651       root   10u     unix 0xf52c3000               
21308 /tmp/.ICE-unix/dcop7651-1199959912
dcopserve  7651       root   11u     unix 0xdfd30c40               
25104 /tmp/.ICE-unix/dcop7651-1199959912
klauncher  7653       root  cwd       DIR        8,7     4096     
130305 /tmp/1878472609
klauncher  7653       root  mem       REG        8,7              
130321 /tmp/1878472609/.kde/cache-ibm/ksycoca (path inode=130326)
klauncher  7653       root    9r      REG        8,7  2564654     
130321 /tmp/1878472609/.kde/cache-ibm/ksycoca
klauncher  7653       root   10u     unix 0xf5297a80               
21295 /tmp/1878472609/.kde/socket-ibm/klauncher1FOdCb.slave-socket
kded       7662       root  cwd       DIR        8,7     4096     
130305 /tmp/1878472609
kded       7662       root  mem       REG        8,7  2564654     
130326 /tmp/1878472609/.kde/cache-ibm/ksycoca
kded       7662       root   12r      REG        8,7  2564654     
130326 /tmp/1878472609/.kde/cache-ibm/ksycoca
kio_smtp   7710    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kio_smtp   7710    jplatte    9r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kio_file   7786    jplatte  mem       REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
kio_file   7786    jplatte    9r      REG        8,6  2590525    
1336104 /var/tmp/kdecache-jplatte/ksycoca
java      10922    jplatte  mem       REG        8,7    32768     
179170 /tmp/hsperfdata_jplatte/10922

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Thu, Jan 10, 2008 at 11:03:05AM +0100, Joerg Platte wrote:
> Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu:
> > > problem, because the iowait problem disappeared today after the regular
> > > Debian update. I'll try to install the old package versions to make it
> > > show up again. Maybe that helps to debug it.
> >
> > Thank you. I'm running sid, ext2 as rootfs now ;-)
> 
> The error is back and I'm getting thousands of messages like this with the 
> patched kernel:
> 
> mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M > tw 1024 sk 0
> requeue_io 301: inode 81441 size 0 at 08:07(sda7)
> mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M > tw 1024 sk 2
> requeue_io 301: inode 81441 size 0 at 08:07(sda7)
> mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M > tw 1024 sk 2
> requeue_io 301: inode 81441 size 0 at 08:07(sda7)

Joerg, what's the output of `dumpe2fs /dev/sda7` and `lsof|grep /tmp`?

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu:
> > problem, because the iowait problem disappeared today after the regular
> > Debian update. I'll try to install the old package versions to make it
> > show up again. Maybe that helps to debug it.
>
> Thank you. I'm running sid, ext2 as rootfs now ;-)

The error is back and I'm getting thousands of messages like this with the 
patched kernel:

mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3937 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3937 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3937 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3937 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3937 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3937 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3937 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3937 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3938 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3938 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3939 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3939 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3939 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3939 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3939 0 0 wc _M 
tw 1024 sk 2
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3939 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3939 0 0 wc _M 
tw 1024 sk 0
requeue_io 301: inode 81441 size 0 at 08:07(sda7)
mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3939 0 0 wc _M 
tw 1024 sk 2

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Thu, Jan 10, 2008 at 09:37:53AM +0100, Joerg Platte wrote:
> Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu:
> > On Thu, Jan 10, 2008 at 03:30:46PM +0800, Fengguang Wu wrote:
> > > Joerg,
> > >
> > > Can you try the attached patches? Thank you.
> > > I cannot reliably reproduce the bug yet.
> >
> > Please ignore the first patch and only apply the two debugging
> > patches. They will produce many printk messages. The output of
> > `dmesg` will be enough for me.
> 
> Too late, I'm already compiling the kernel. But I have currently another 

It won't hurt, as you don't use XIP.

> problem, because the iowait problem disappeared today after the regular 
> Debian update. I'll try to install the old package versions to make it show 
> up again. Maybe that helps to debug it.

Thank you. I'm running sid, ext2 as rootfs now ;-)

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu:
> On Thu, Jan 10, 2008 at 03:30:46PM +0800, Fengguang Wu wrote:
> > Joerg,
> >
> > Can you try the attached patches? Thank you.
> > I cannot reliably reproduce the bug yet.
>
> Please ignore the first patch and only apply the two debugging
> patches. They will produce many printk messages. The output of
> `dmesg` will be enough for me.

Too late, I'm already compiling the kernel. But I have currently another 
problem, because the iowait problem disappeared today after the regular 
Debian update. I'll try to install the old package versions to make it show 
up again. Maybe that helps to debug it.

Grüße,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Thu, Jan 10, 2008 at 03:30:46PM +0800, Fengguang Wu wrote:
> Joerg,
> 
> Can you try the attached patches? Thank you.
> I cannot reliably reproduce the bug yet.

Please ignore the first patch and only apply the two debugging 
patches. They will produce many printk messages. The output of 
`dmesg` will be enough for me.

Thank you,
Fengguang

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

[-- Attachment #1: Type: text/plain, Size: 105 bytes --]

Joerg,

Can you try the attached patches? Thank you.
I cannot reliably reproduce the bug yet.

Fengguang

[-- Attachment #2: writeback-ext2-fix.patch --]
[-- Type: text/plain, Size: 437 bytes --]

 mm/filemap_xip.c |    1 +
 1 files changed, 1 insertion(+)

Index: linux/mm/filemap_xip.c
===================================================================
--- linux.orig/mm/filemap_xip.c
+++ linux/mm/filemap_xip.c
@@ -431,6 +431,7 @@ xip_truncate_page(struct address_space *
 			return PTR_ERR(page);
 	}
 	zero_user_page(page, offset, length, KM_USER0);
+	set_page_dirty(page);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(xip_truncate_page);

[-- Attachment #3: writeback-debug.patch --]
[-- Type: text/plain, Size: 2677 bytes --]

---
 mm/page-writeback.c |   29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

--- linux-2.6.24-git17.orig/mm/page-writeback.c
+++ linux-2.6.24-git17/mm/page-writeback.c
@@ -98,6 +98,26 @@ EXPORT_SYMBOL(laptop_mode);
 
 /* End of sysctl-exported parameters */
 
+#define writeback_debug_report(n, wbc) do {                               \
+	__writeback_debug_report(n, wbc, __FILE__, __LINE__, __FUNCTION__); \
+} while (0)
+
+void __writeback_debug_report(long n, struct writeback_control *wbc,
+		const char *file, int line, const char *func)
+{
+	printk(KERN_DEBUG "%s %d %s: %s(%d) %ld "
+			"global %lu %lu %lu "
+			"wc %c%c tw %ld sk %ld\n",
+			file, line, func,
+			current->comm, current->pid, n,
+			global_page_state(NR_FILE_DIRTY),
+			global_page_state(NR_WRITEBACK),
+			global_page_state(NR_UNSTABLE_NFS),
+			wbc->encountered_congestion ? 'C':'_',
+			wbc->more_io ? 'M':'_',
+			wbc->nr_to_write,
+			wbc->pages_skipped);
+}
 
 static void background_writeout(unsigned long _min_pages);
 
@@ -395,6 +415,7 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;
 			get_dirty_limits(&background_thresh, &dirty_thresh,
 				       &bdi_thresh, bdi);
+			writeback_debug_report(pages_written, &wbc);
 		}
 
 		/*
@@ -421,6 +442,7 @@ static void balance_dirty_pages(struct a
 			break;		/* We've done our duty */
 
 		congestion_wait(WRITE, HZ/10);
+		writeback_debug_report(-pages_written, &wbc);
 	}
 
 	if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
@@ -515,6 +537,11 @@ void throttle_vm_writeout(gfp_t gfp_mask
 			global_page_state(NR_WRITEBACK) <= dirty_thresh)
                         	break;
                 congestion_wait(WRITE, HZ/10);
+		printk(KERN_DEBUG "throttle_vm_writeout: "
+				"congestion_wait on %lu+%lu > %lu\n",
+				global_page_state(NR_UNSTABLE_NFS),
+				global_page_state(NR_WRITEBACK),
+				dirty_thresh);
 
 		/*
 		 * The caller might hold locks which can prevent IO completion
@@ -557,6 +584,7 @@ static void background_writeout(unsigned
 		wbc.pages_skipped = 0;
 		writeback_inodes(&wbc);
 		min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+		writeback_debug_report(min_pages, &wbc);
 		if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
 			/* Wrote less than expected */
 			if (wbc.encountered_congestion || wbc.more_io)
@@ -630,6 +658,7 @@ static void wb_kupdate(unsigned long arg
 		wbc.encountered_congestion = 0;
 		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		writeback_inodes(&wbc);
+		writeback_debug_report(nr_to_write, &wbc);
 		if (wbc.nr_to_write > 0) {
 			if (wbc.encountered_congestion || wbc.more_io)
 				congestion_wait(WRITE, HZ/10);

[-- Attachment #4: requeue_io-debug.patch --]
[-- Type: text/plain, Size: 1140 bytes --]

Subject: track redirty_tail() calls

It helps a lot to know how redirty_tail() are called.

Cc: Ken Chen <kenchen@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |   16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

--- linux-2.6.24-git17.orig/fs/fs-writeback.c
+++ linux-2.6.24-git17/fs/fs-writeback.c
@@ -164,12 +164,26 @@ static void redirty_tail(struct inode *i
 	list_move(&inode->i_list, &sb->s_dirty);
 }
 
+#define requeue_io(inode)						\
+	do {								\
+		__requeue_io(inode, __LINE__);				\
+	} while (0)
+
 /*
  * requeue inode for re-scanning after sb->s_io list is exhausted.
  */
-static void requeue_io(struct inode *inode)
+static void __requeue_io(struct inode *inode, int line)
 {
 	list_move(&inode->i_list, &inode->i_sb->s_more_io);
+
+	printk(KERN_DEBUG "requeue_io %d: inode %lu size %llu at %02x:%02x(%s)\n",
+			line,
+			inode->i_ino,
+			i_size_read(inode),
+			MAJOR(inode->i_sb->s_dev),
+			MINOR(inode->i_sb->s_dev),
+			inode->i_sb->s_id
+			);
 }
 
 static void inode_sync_complete(struct inode *inode)

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Wed, Jan 09, 2008 at 02:04:29PM +0100, Joerg Platte wrote:
> Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu:
> > On Wed, Jan 09, 2008 at 01:22:33PM +0100, Joerg Platte wrote:
> > > Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu:
> > >
> > > Thank your for the hint with the filesystems!
> > >
> > > > Thank you for the clue. However I cannot reproduce the bug on
> > > > ext2/2.6.24-rc7. Can you provide more details? Thank you.
> > >
> > > I attached some more information. I'm using the ata_piix driver for my
> > > PATA disk and cdrom drive and booted with hpet=force. Kernel 2.6.23.X has
> > > been
> >
> >                                                           ~~~~~~~~
> > 							  not 2.6.24-rc7?
> >
> No, 2.6.23.X was the last working kernel without this problem, the bug shows 
> up with 2.6.24-rcX. I just wanted to emphasis, that I don't think that it has 
> something to do with the hpet stuff. Kernel 2.6.24-rcX is unpatched, because 
> the hrt stuff has been merged.
> 
> > > patched with the -hrt patches to enable the hidden hpet time on the
> > > ICH4-M chipset. I just rebooted the notebook and mounted /tmp again as
> > > ext2 and now the iowait problem is back. Seems to be reproducible on my
> > > computer. What additional information do you need?
> >
> > I mounted an ext2 as tmp and find no problem. My config options are:
> >
> > CONFIG_EXT2_FS=y
> > CONFIG_EXT2_FS_XATTR=y
> > CONFIG_EXT2_FS_POSIX_ACL=y
> > CONFIG_EXT2_FS_SECURITY=y
> > # CONFIG_EXT2_FS_XIP is not set
> >
> > Fengguang
> 
> CONFIG_EXT2_FS=m
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT2_FS_POSIX_ACL=y
> CONFIG_EXT2_FS_SECURITY=y
> CONFIG_EXT2_FS_XIP=y
> 
> Here it is modular and I enabled CONFIG_EXT2_FS_XIP, but the same is enabled 
> on 2.6.23.X. Maybe it is related to libata? I additionally discovered, that 
> the problem disappears for a few seconds when I press the eject button for 
> the ultra bay of my thinkpad. Pressing the button unregisters the cdrom drive 
> to be able to replace it with a hard drive or a battery. Maybe this bug is 
> thinkpad relared?

At last I caught it :-)

[ 1862.219189] requeue_io 301: inode 50948 size 0 at 03:02(hda2)
[ 1862.219199] requeue_io 301: inode 51616 size 0 at 03:02(hda2)
[ 1862.219204] requeue_io 301: inode 51656 size 0 at 03:02(hda2)
[ 1862.219208] requeue_io 301: inode 51655 size 0 at 03:02(hda2)
[ 1862.219216] mm/page-writeback.c 668 wb_kupdate: pdflush(182) 10768 global 3100 0 0 wc _M tw 1024 sk 1
[ 1862.319039] requeue_io 301: inode 50948 size 0 at 03:02(hda2)
[ 1862.319050] requeue_io 301: inode 51616 size 0 at 03:02(hda2)
[ 1862.319055] requeue_io 301: inode 51656 size 0 at 03:02(hda2)
[ 1862.319059] requeue_io 301: inode 51655 size 0 at 03:02(hda2)
[ 1862.319068] mm/page-writeback.c 668 wb_kupdate: pdflush(182) 10768 global 3100 0 0 wc _M tw 1024 sk 1

They are some zero sized files, maybe something goes wrong with the truncate code.

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu:
> On Wed, Jan 09, 2008 at 01:22:33PM +0100, Joerg Platte wrote:
> > Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu:
> >
> > Thank your for the hint with the filesystems!
> >
> > > Thank you for the clue. However I cannot reproduce the bug on
> > > ext2/2.6.24-rc7. Can you provide more details? Thank you.
> >
> > I attached some more information. I'm using the ata_piix driver for my
> > PATA disk and cdrom drive and booted with hpet=force. Kernel 2.6.23.X has
> > been
>
>                                                           ~~~~~~~~
> 							  not 2.6.24-rc7?
>
No, 2.6.23.X was the last working kernel without this problem, the bug shows 
up with 2.6.24-rcX. I just wanted to emphasis, that I don't think that it has 
something to do with the hpet stuff. Kernel 2.6.24-rcX is unpatched, because 
the hrt stuff has been merged.

> > patched with the -hrt patches to enable the hidden hpet time on the
> > ICH4-M chipset. I just rebooted the notebook and mounted /tmp again as
> > ext2 and now the iowait problem is back. Seems to be reproducible on my
> > computer. What additional information do you need?
>
> I mounted an ext2 as tmp and find no problem. My config options are:
>
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT2_FS_POSIX_ACL=y
> CONFIG_EXT2_FS_SECURITY=y
> # CONFIG_EXT2_FS_XIP is not set
>
> Fengguang

CONFIG_EXT2_FS=m
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT2_FS_XIP=y

Here it is modular and I enabled CONFIG_EXT2_FS_XIP, but the same is enabled 
on 2.6.23.X. Maybe it is related to libata? I additionally discovered, that 
the problem disappears for a few seconds when I press the eject button for 
the ultra bay of my thinkpad. Pressing the button unregisters the cdrom drive 
to be able to replace it with a hard drive or a battery. Maybe this bug is 
thinkpad relared?

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Wed, Jan 09, 2008 at 01:22:33PM +0100, Joerg Platte wrote:
> Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu:
> 
> Thank your for the hint with the filesystems!
> 
> > Thank you for the clue. However I cannot reproduce the bug on
> > ext2/2.6.24-rc7. Can you provide more details? Thank you.
> 
> I attached some more information. I'm using the ata_piix driver for my PATA 
> disk and cdrom drive and booted with hpet=force. Kernel 2.6.23.X has been 
                                                          ~~~~~~~~
							  not 2.6.24-rc7?

> patched with the -hrt patches to enable the hidden hpet time on the ICH4-M 
> chipset. I just rebooted the notebook and mounted /tmp again as ext2 and now 
> the iowait problem is back. Seems to be reproducible on my computer. What 
> additional information do you need?

I mounted an ext2 as tmp and find no problem. My config options are:

CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set

Fengguang

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu:

Thank your for the hint with the filesystems!

> Thank you for the clue. However I cannot reproduce the bug on
> ext2/2.6.24-rc7. Can you provide more details? Thank you.

I attached some more information. I'm using the ata_piix driver for my PATA 
disk and cdrom drive and booted with hpet=force. Kernel 2.6.23.X has been 
patched with the -hrt patches to enable the hidden hpet time on the ICH4-M 
chipset. I just rebooted the notebook and mounted /tmp again as ext2 and now 
the iowait problem is back. Seems to be reproducible on my computer. What 
additional information do you need?

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605


[-- Attachment #2: lspci --]
[-- Type: text/plain, Size: 1620 bytes --]

00:00.0 Host bridge: Intel Corporation 82855PM Processor to I/O Controller (rev 03)
00:01.0 PCI bridge: Intel Corporation 82855PM Processor to AGP Controller (rev 03)
00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 81)
00:1f.0 ISA bridge: Intel Corporation 82801DBM (ICH4-M) LPC Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801DBM (ICH4-M) IDE Controller (rev 01)
00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) SMBus Controller (rev 01)
00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 01)
00:1f.6 Modem: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Modem Controller (rev 01)
01:00.0 VGA compatible controller: ATI Technologies Inc Radeon R250 [Mobility FireGL 9000] (rev 02)
02:00.0 CardBus bridge: Texas Instruments PCI1520 PC card Cardbus Controller (rev 01)
02:00.1 CardBus bridge: Texas Instruments PCI1520 PC card Cardbus Controller (rev 01)
02:01.0 Ethernet controller: Intel Corporation 82540EP Gigabit Ethernet Controller (Mobile) (rev 03)
02:02.0 Ethernet controller: Atheros Communications, Inc. AR5211 802.11ab NIC (rev 01)

[-- Attachment #3: lsmod --]
[-- Type: text/plain, Size: 7475 bytes --]

Module                  Size  Used by
radeon                112644  2 
drm                    73684  3 radeon
binfmt_misc            10568  1 
rfcomm                 35924  1 
l2cap                  22660  5 rfcomm
bluetooth              51872  4 rfcomm,l2cap
snd_rtctimer            3616  1 
nfsd                  201652  13 
auth_rpcgss            39492  1 nfsd
exportfs                4544  1 nfsd
af_packet              20292  4 
autofs4                20356  2 
ipt_MASQUERADE          3328  3 
iptable_nat             6660  1 
nf_nat                 17948  2 ipt_MASQUERADE,iptable_nat
nf_conntrack_ipv4      16328  11 iptable_nat
xt_state                2304  9 
ipt_LOG                 5632  9 
xt_limit                2432  9 
ipt_REJECT              4288  2 
xt_mark                 1728  2 
xt_tcpudp               2944  12 
xt_mac                  1728  38 
iptable_filter          2752  1 
xt_MARK                 2048  3 
xt_multiport            2880  8 
iptable_mangle          2560  1 
ip_tables              12168  3 iptable_nat,iptable_filter,iptable_mangle
x_tables               13828  12 ipt_MASQUERADE,iptable_nat,xt_state,ipt_LOG,xt_limit,ipt_REJECT,xt_mark,xt_tcpudp,xt_mac,xt_MARK,xt_multiport,ip_tables
nf_conntrack_ftp        8224  0 
nf_conntrack           61200  6 ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4,xt_state,nf_conntrack_ftp
cpufreq_userspace       3800  0 
cpufreq_stats           4868  0 
cpufreq_powersave       1600  0 
nfs                   227136  0 
lockd                  59576  3 nfsd,nfs
nfs_acl                 3264  2 nfsd,nfs
sunrpc                169308  11 nfsd,auth_rpcgss,nfs,lockd,nfs_acl
deflate                 3520  0 
zlib_deflate           17960  1 deflate
zlib_inflate           15488  1 deflate
twofish                 6592  0 
twofish_common         36416  1 twofish
camellia               28736  0 
serpent                18368  0 
blowfish                8128  0 
des_generic            16128  0 
cbc                     4288  0 
ecb                     3328  0 
aes_generic            26536  0 
aes_i586               32436  0 
geode_aes               5700  0 
blkcipher               6404  3 cbc,ecb,geode_aes
xcbc                    5384  0 
sha256_generic         11008  0 
sha1_generic            2432  0 
md5                     3840  0 
crypto_null             2368  0 
hmac                    4352  0 
crypto_hash             2048  2 xcbc,hmac
cryptomgr               3520  0 
crypto_algapi          16256  19 deflate,twofish,camellia,serpent,blowfish,des_generic,cbc,ecb,aes_generic,aes_i586,geode_aes,blkcipher,xcbc,sha256_generic,sha1_generic,md5,crypto_null,hmac,cryptomgr
af_key                 35536  0 
nls_utf8                1920  1 
ntfs                   92900  1 
nls_base                7040  2 nls_utf8,ntfs
ext2                   65160  1 
fuse                   44948  1 
ebtable_broute          1984  0 
bridge                 49112  1 ebtable_broute
llc                     7128  1 bridge
ebtable_nat             2176  0 
ebtable_filter          2176  0 
ebtables               17024  3 ebtable_broute,ebtable_nat,ebtable_filter
deadline_iosched        5376  0 
as_iosched             12488  0 
ircomm_tty             22728  0 
ircomm                 12868  1 ircomm_tty
tun                    10816  0 
acpi_cpufreq            6732  1 
sbs                    14024  0 
sbshc                   6656  1 sbs
joydev                 11008  0 
snd_intel8x0           31260  2 
snd_intel8x0m          16588  4 
snd_ac97_codec         92644  2 snd_intel8x0,snd_intel8x0m
ac97_bus                1920  1 snd_ac97_codec
snd_pcm_oss            43040  0 
snd_pcm                75976  6 snd_intel8x0,snd_intel8x0m,snd_ac97_codec,snd_pcm_oss
snd_mixer_oss          15552  1 snd_pcm_oss
irtty_sir               5952  0 
sir_dev                13508  1 irtty_sir
nsc_ircc               17040  0 
irda                  108680  4 ircomm_tty,ircomm,sir_dev,nsc_ircc
crc_ccitt               1984  1 irda
snd_seq_oss            29460  0 
parport_pc             36092  0 
parport                33544  1 parport_pc
snd_seq_midi            8800  0 
snd_rawmidi            23200  1 snd_seq_midi
8250_pnp                9472  0 
snd_seq_midi_event      6656  2 snd_seq_oss,snd_seq_midi
snd_seq                48732  6 snd_seq_oss,snd_seq_midi,snd_seq_midi_event
snd_timer              21572  3 snd_rtctimer,snd_pcm,snd_seq
snd_seq_device          7820  4 snd_seq_oss,snd_seq_midi,snd_rawmidi,snd_seq
yenta_socket           24268  2 
rsrc_nonstatic         11584  1 yenta_socket
pcmcia                 36660  0 
firmware_class          9216  1 pcmcia
snd                    50644  22 snd_intel8x0,snd_intel8x0m,snd_ac97_codec,snd_pcm_oss,snd_pcm,snd_mixer_oss,snd_seq_oss,snd_rawmidi,snd_seq,snd_timer,snd_seq_device
pcmcia_core            36688  3 yenta_socket,rsrc_nonstatic,pcmcia
serio_raw               6404  0 
pcspkr                  2624  0 
psmouse                36176  0 
soundcore               7364  1 snd
snd_page_alloc         10184  3 snd_intel8x0,snd_intel8x0m,snd_pcm
8250_pci               22912  0 
8250                   21880  2 8250_pnp,8250_pci
serial_core            20480  1 8250
i2c_i801                8272  0 
rng_core                4804  0 
iTCO_wdt               11616  0 
iTCO_vendor_support     3652  1 iTCO_wdt
battery                13252  0 
ac                      5892  0 
power_supply            9604  3 sbs,battery,ac
video                  18192  0 
output                  3456  1 video
button                  8080  0 
intel_agp              22868  1 
agpgart                30896  2 drm,intel_agp
evdev                  10752  7 
ext3                  121032  2 
jbd                    44116  1 ext3
mbcache                 8192  2 ext2,ext3
sg                     32800  0 
sr_mod                 15396  0 
cdrom                  32604  1 sr_mod
sd_mod                 25680  6 
ata_generic             7172  0 
usbhid                 40100  0 
hid                    26628  1 usbhid
ff_memless              5128  1 usbhid
ata_piix               17476  5 
floppy                 51632  0 
pata_acpi               6976  0 
e1000                 109824  0 
libata                144016  3 ata_generic,ata_piix,pata_acpi
scsi_mod              140948  4 sg,sr_mod,sd_mod,libata
ehci_hcd               31372  0 
uhci_hcd               22668  0 
usbcore               131256  4 usbhid,ehci_hcd,uhci_hcd
thermal                15580  0 
processor              28016  3 acpi_cpufreq,thermal
fan                     4356  0 
unix                   27220  1085 
cpufreq_conservative     6752  0 
cpufreq_ondemand        7608  1 
freq_table              4100  3 cpufreq_stats,acpi_cpufreq,cpufreq_ondemand
thinkpad_acpi          49296  0 
hwmon                   3092  1 thinkpad_acpi
nvram                   8136  1 thinkpad_acpi
fbcon                  37976  72 
tileblit                2368  1 fbcon
font                    8320  1 fbcon
bitblit                 5184  1 fbcon
softcursor              2112  1 bitblit
radeonfb               97392  1 
fb                     45768  5 fbcon,tileblit,bitblit,softcursor,radeonfb
fb_ddc                  2432  1 radeonfb
backlight               4932  3 video,thinkpad_acpi,radeonfb
i2c_algo_bit            5828  1 radeonfb
cfbcopyarea             3264  1 radeonfb
i2c_core               22224  4 i2c_i801,radeonfb,fb_ddc,i2c_algo_bit
cfbimgblt               2624  1 radeonfb
cfbfillrect             3392  1 radeonfb

[-- Attachment #4: dmesg --]
[-- Type: text/plain, Size: 15481 bytes --]

river
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for hkey
thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle HKEY for hkey
thinkpad_acpi: wan_init: wan is not supported, status 0xf7ed8e5c
thinkpad_acpi: ibm_init: probing for video
thinkpad_acpi: video_init: initializing video subdriver
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for vid
thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle \_SB.PCI0.AGP.VID for vid
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for vid2
thinkpad_acpi: drv_acpi_handle_init: ACPI handle for vid2 not found
thinkpad_acpi: video_init: video is supported, mode 3
thinkpad_acpi: ibm_init: video installed
thinkpad_acpi: ibm_init: probing for light
thinkpad_acpi: light_init: initializing light subdriver
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for ledb
thinkpad_acpi: drv_acpi_handle_init: ACPI handle for ledb not found
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for lght
thinkpad_acpi: drv_acpi_handle_init: ACPI handle for lght not found
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for cmos
thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle \UCMS for cmos
thinkpad_acpi: light_init: light is supported
thinkpad_acpi: ibm_init: light installed
thinkpad_acpi: ibm_init: probing for bay
thinkpad_acpi: bay_init: initializing bay subdriver
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for bay
thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle \_SB.PCI0.IDE0.SCND.MSTR for bay
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for bay_ej
thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle _EJ0 for bay_ej
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for bay2
thinkpad_acpi: drv_acpi_handle_init: ACPI handle for bay2 not found
thinkpad_acpi: bay_init: bay 1: status supported, eject supported; bay 2: status not supported, eject not supported
thinkpad_acpi: setup_acpi_notify: setting up ACPI notify for bay
thinkpad_acpi: ibm_init: bay installed
thinkpad_acpi: ibm_init: probing for cmos
thinkpad_acpi: cmos_init: initializing cmos commands subdriver
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for cmos
thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle \UCMS for cmos
thinkpad_acpi: cmos_init: cmos commands are supported
thinkpad_acpi: ibm_init: cmos installed
thinkpad_acpi: ibm_init: probing for led
thinkpad_acpi: led_init: initializing LED subdriver
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for led
thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle LED for led
thinkpad_acpi: led_init: LED commands are supported, mode 3
thinkpad_acpi: ibm_init: led installed
thinkpad_acpi: ibm_init: probing for beep
thinkpad_acpi: beep_init: initializing beep subdriver
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for beep
thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle BEEP for beep
thinkpad_acpi: beep_init: beep is supported
thinkpad_acpi: ibm_init: beep installed
thinkpad_acpi: ibm_init: probing for thermal
thinkpad_acpi: thermal_init: initializing thermal subdriver
thinkpad_acpi: thermal_init: thermal is supported, mode 3
thinkpad_acpi: ibm_init: thermal installed
thinkpad_acpi: ibm_init: probing for ecdump
thinkpad_acpi: ibm_init: ecdump installed
thinkpad_acpi: ibm_init: probing for brightness
thinkpad_acpi: brightness_init: initializing brightness subdriver
thinkpad_acpi: brightness_init: selected brightness_mode=3
thinkpad_acpi: brightness_init: brightness is supported
thinkpad_acpi: ibm_init: brightness installed
thinkpad_acpi: ibm_init: probing for volume
thinkpad_acpi: ibm_init: volume installed
thinkpad_acpi: ibm_init: probing for fan
thinkpad_acpi: fan_init: initializing fan subdriver
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for fans
thinkpad_acpi: drv_acpi_handle_init: ACPI handle for fans not found
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for gfan
thinkpad_acpi: drv_acpi_handle_init: ACPI handle for gfan not found
thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for sfan
thinkpad_acpi: drv_acpi_handle_init: ACPI handle for sfan not found
thinkpad_acpi: fan_init: fan is supported, modes 2, 2
thinkpad_acpi: ibm_init: fan installed
input: ThinkPad Extra Buttons as /class/input/input1
NET: Registered protocol family 1
ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: Processor [CPU] (supports 8 throttling states)
Marking TSC unstable due to: TSC halts in idle.
Time: hpet clocksource has been installed.
ACPI: Thermal Zone [THM0] (50 C)
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 11, io base 0x00001800
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:1d.1[B] -> Link [LNKD] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 11, io base 0x00001820
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
SCSI subsystem initialized
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:1d.2[C] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.2: irq 11, io base 0x00001840
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
Intel(R) PRO/1000 Network Driver - version 7.3.20-k2
Copyright (c) 1999-2006 Intel Corporation.
libata version 3.00 loaded.
ACPI: PCI Interrupt Link [LNKH] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:1d.7[D] -> Link [LNKH] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 4
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 11, io mem 0xc0000000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 6 ports detected
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
ACPI: PCI Interrupt 0000:02:01.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11
e1000: 0000:02:01.0: e1000_validate_option: PHY Smart Power Down Enabled
e1000: 0000:02:01.0: e1000_probe: (PCI:33MHz:32-bit) 00:16:41:10:19:11
usb 2-1: new low speed USB device using uhci_hcd and address 2
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI Interrupt 0000:00:1f.1[A] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1f.1 to 64
ACPI: PCI interrupt for device 0000:00:1f.1 disabled
ata_piix 0000:00:1f.1: version 2.12
ACPI: PCI Interrupt 0000:00:1f.1[A] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1f.1 to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x1860 irq 14
ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x1868 irq 15
usb 2-1: configuration #1 chosen from 1 choice
usbcore: registered new interface driver hiddev
input: Logitech Optical USB Mouse as /class/input/input2
input: USB HID v1.10 Mouse [Logitech Optical USB Mouse] on usb-0000:00:1d.1-1
usbcore: registered new interface driver usbhid
drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver
ata1.00: ATA-7: SAMSUNG HM160JC, AP100-16, max UDMA/100
ata1.00: 312581808 sectors, multi 16: LBA48 
ata1.00: configured for UDMA/100
ata2.00: ATAPI: UJDA745 DVD/CDRW, 1.06, max UDMA/33
ata2.00: configured for UDMA/33
scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG HM160JC  AP10 PQ: 0 ANSI: 5
scsi 1:0:0:0: CD-ROM            MATSHITA UJDA745 DVD/CDRW 1.06 PQ: 0 ANSI: 5
Driver 'sd' needs updating - please use bus_type methods
sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Driver 'sr' needs updating - please use bus_type methods
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 < sda5 sda6 sda7 sda8 >
sd 0:0:0:0: [sda] Attached SCSI disk
sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
sd 0:0:0:0: Attached scsi generic sg0 type 0
sr 1:0:0:0: Attached scsi generic sg1 type 5
Attempting manual resume
swsusp: Marking nosave pages: 000000000009f000 - 0000000000100000
swsusp: Basic memory bitmaps created
swsusp: Basic memory bitmaps freed
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Linux agpgart interface v0.102
agpgart: Detected an Intel 855PM Chipset.
agpgart: AGP aperture is 256M @ 0xd0000000
input: Power Button (FF) as /class/input/input3
ACPI: Power Button (FF) [PWRF]
input: Lid Switch as /class/input/input4
ACPI: Lid Switch [LID]
input: Sleep Button (CM) as /class/input/input5
ACPI: Sleep Button (CM) [SLPB]
input: Video Bus as /class/input/input6
ACPI: Video Device [VID] (multi-head: yes  rom: no  post: no)
ACPI: AC Adapter [AC] (on-line)
ACPI: Battery Slot [BAT0] (battery present)
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.02 (26-Jul-2007)
iTCO_wdt: Found a ICH4-M TCO device (Version=1, TCOBASE=0x1060)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
intel_rng: FWH not detected
ACPI: PCI Interrupt 0000:00:1f.3[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
Serial: 8250/16550 driver $Revision: 1.90 $ 2 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
ACPI: PCI Interrupt 0000:00:1f.6[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
ACPI: PCI interrupt for device 0000:00:1f.6 disabled
input: PC Speaker as /class/input/input7
Yenta: CardBus bridge found at 0000:02:00.0 [1014:0512]
Yenta: Using INTVAL to route CSC interrupts to PCI
Yenta: Routing CardBus interrupts to PCI
Yenta TI: socket 0000:02:00.0, mfunc 0x01d21022, devctl 0x64
Yenta: ISA IRQ mask 0x04b8, PCI irq 11
Socket status: 30000006
pcmcia: parent PCI bridge I/O window: 0x4000 - 0x8fff
cs: IO port probe 0x4000-0x8fff: clean.
pcmcia: parent PCI bridge Memory window: 0xc0200000 - 0xcfffffff
pcmcia: parent PCI bridge Memory window: 0xe8000000 - 0xefffffff
00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
Synaptics Touchpad, model: 1, fw: 5.9, id: 0x2c6ab1, caps: 0x884793/0x0
serio: Synaptics pass-through port at isa0060/serio1/input0
Yenta: CardBus bridge found at 0000:02:00.1 [1014:0512]
Yenta: Using INTVAL to route CSC interrupts to PCI
Yenta: Routing CardBus interrupts to PCI
Yenta TI: socket 0000:02:00.1, mfunc 0x01d21022, devctl 0x64
input: SynPS/2 Synaptics TouchPad as /class/input/input8
parport_pc 00:0b: reported by Plug and Play ACPI
parport0: PC-style at 0x3bc (0x7bc), irq 7, dma 0 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
NET: Registered protocol family 23
nsc-ircc 00:0c: activated
nsc-ircc, chip->init
nsc-ircc, Found chip at base=0x02e
nsc-ircc, driver loaded (Dag Brattli)
IrDA: Registered device irda0
nsc-ircc, Using dongle: IBM31T1100 or Temic TFDS6000/TFDS6500
Yenta: ISA IRQ mask 0x0430, PCI irq 11
Socket status: 30000006
pcmcia: parent PCI bridge I/O window: 0x4000 - 0x8fff
cs: IO port probe 0x4000-0x8fff: clean.
pcmcia: parent PCI bridge Memory window: 0xc0200000 - 0xcfffffff
pcmcia: parent PCI bridge Memory window: 0xe8000000 - 0xefffffff
ACPI: PCI Interrupt 0000:00:1f.6[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1f.6 to 64
ACPI: PCI Interrupt 0000:00:1f.5[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11
PCI: Setting latency timer of device 0000:00:1f.5 to 64
cs: IO port probe 0xc00-0xcff: clean.
cs: IO port probe 0x800-0x8ff: clean.
cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7
cs: IO port probe 0xa00-0xaff: clean.
cs: IO port probe 0xc00-0xcff: clean.
cs: IO port probe 0x800-0x8ff: clean.
cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7
cs: IO port probe 0xa00-0xaff: clean.
intel8x0_measure_ac97_clock: measured 52955 usecs
intel8x0: measured clock 207 rejected
intel8x0: clocking to 48000
Adding 1951856k swap on /dev/sda5.  Priority:-1 extents:1 across:1951856k
EXT3 FS on sda6, internal journal
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
IrCOMM protocol (Dag Brattli)
io scheduler anticipatory registered
io scheduler deadline registered
Ebtables v2.0 registered
Bridge firewalling registered
fuse init (API version 7.9)
EXT2-fs warning (device sda7): ext2_fill_super: mounting ext3 filesystem as ext2
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda8, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
NTFS driver 2.1.29 [Flags: R/O MODULE].
NTFS volume version 3.1.
IBM TrackPoint firmware: 0x0e, buttons: 3/3
input: TPPS/2 IBM TrackPoint as /class/input/input9
NET: Registered protocol family 15
padlock: VIA PadLock Hash Engine not detected.
padlock: VIA PadLock Hash Engine not detected.
padlock: VIA PadLock not detected.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
Clocksource tsc unstable (delta = -197916262 ns)
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
NET: Registered protocol family 17
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
Bluetooth: Core ver 2.11
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.9
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM TTY layer initialized
Bluetooth: RFCOMM ver 1.8
[drm] Initialized drm 1.1.0 20060810
[drm] Initialized radeon 1.28.0 20060524 on minor 0
agpgart: Found an AGP 2.0 compliant device at 0000:00:00.0.
agpgart: Putting AGP V2 device at 0000:00:00.0 into 4x mode
agpgart: Putting AGP V2 device at 0000:01:00.0 into 4x mode
[drm] Setting GART location based on new memory map
[drm] Loading R200 Microcode
[drm] writeback test succeeded in 2 usecs

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Wed, Jan 09, 2008 at 07:13:14AM +0100, Joerg Platte wrote:
> Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu:
> > > /dev/sda6 on / type ext3 (rw,noatime,errors=remount-ro,acl)
> > > tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
> > > proc on /proc type proc (rw,noexec,nosuid,nodev)
> > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> > > procbususb on /proc/bus/usb type usbfs (rw)
> > > udev on /dev type tmpfs (rw,mode=0755)
> > > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
> > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
> > > fusectl on /sys/fs/fuse/connections type fusectl (rw)
> > > /dev/sda7 on /tmp type ext2 (rw,noatime,errors=remount-ro,acl)
> > > /dev/sda8 on /export type ext3 (rw,noatime,errors=remount-ro,acl)
> > > /dev/sda1 on /winxp type ntfs (rw,umask=002,gid=10000,nls=utf8)
> >
> > So they are ext3/ext2/ntfs.  What if you umount ntfs? and ext2 if possible?
> 
> Unmounting ntfs doesn't help, hence I converted the remaining ext2 filesystem 
> to ext3, modified the fstab entry accordingly and rebooted. Now everything 
> seems to be fine! Top reports an idle system and there is no abnormal iowait 
> any longer! Seems to be ext2 was causing this! Later today I can try to 
> remount the filesystem as ext2 to be sure the bug shows up again.

Thank you for the clue. However I cannot reproduce the bug on
ext2/2.6.24-rc7. Can you provide more details? Thank you.

Fengguang

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu:
> > /dev/sda6 on / type ext3 (rw,noatime,errors=remount-ro,acl)
> > tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
> > proc on /proc type proc (rw,noexec,nosuid,nodev)
> > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> > procbususb on /proc/bus/usb type usbfs (rw)
> > udev on /dev type tmpfs (rw,mode=0755)
> > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
> > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
> > fusectl on /sys/fs/fuse/connections type fusectl (rw)
> > /dev/sda7 on /tmp type ext2 (rw,noatime,errors=remount-ro,acl)
> > /dev/sda8 on /export type ext3 (rw,noatime,errors=remount-ro,acl)
> > /dev/sda1 on /winxp type ntfs (rw,umask=002,gid=10000,nls=utf8)
>
> So they are ext3/ext2/ntfs.  What if you umount ntfs? and ext2 if possible?

Unmounting ntfs doesn't help, hence I converted the remaining ext2 filesystem 
to ext3, modified the fstab entry accordingly and rebooted. Now everything 
seems to be fine! Top reports an idle system and there is no abnormal iowait 
any longer! Seems to be ext2 was causing this! Later today I can try to 
remount the filesystem as ext2 to be sure the bug shows up again.

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Fengguang Wu]

On Mon, Jan 07, 2008 at 02:40:13PM +0100, Joerg Platte wrote:
> Am Montag, 7. Januar 2008 schrieb Peter Zijlstra:
> > On Mon, 2008-01-07 at 14:24 +0100, Joerg Platte wrote:
> >
> > This is from: 2.6.24-rc7
> >
> > > kernel: pdflush       D f41c2f14     0 18822      2
> > > kernel:        f673f000 00000046 00000286 f41c2f14 f5194ce0 00000286
> > > 00000286 f41c2f14 kernel:        00175279 f41c2f6c 00000000 c0271f6c
> > > f5ff363c f5ff3644 c0354a90 c0354a90 kernel:        00175279 c0123251
> > > f5194b80 c03546c0 c0271f67 6c666470 00687375 00000000 kernel: Call Trace:
> > > kernel:  [<c0271f6c>] schedule_timeout+0x6e/0x8b
> > > kernel:  [<c0123251>] process_timeout+0x0/0x5
> > > kernel:  [<c0271f67>] schedule_timeout+0x69/0x8b
> > > kernel:  [<c027179a>] __sched_text_start+0x3a/0x70
> > > kernel:  [<c014d34b>] congestion_wait+0x4e/0x62
> > > kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
> > > kernel:  [<c014971e>] pdflush+0x0/0x1bf
> > > kernel:  [<c01493c6>] wb_kupdate+0x8c/0xd1
> > > kernel:  [<c014971e>] pdflush+0x0/0x1bf
> > > kernel:  [<c0149839>] pdflush+0x11b/0x1bf
> > > kernel:  [<c014933a>] wb_kupdate+0x0/0xd1
> > > kernel:  [<c012bd71>] kthread+0x36/0x5d
> > > kernel:  [<c012bd3b>] kthread+0x0/0x5d
> > > kernel:  [<c010493b>] kernel_thread_helper+0x7/0x10
> > > kernel:  =======================
> >
> > What filesystem are you using?
> 
> Here you can see all currently mounted filesystems:
> 
> /dev/sda6 on / type ext3 (rw,noatime,errors=remount-ro,acl)
> tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> procbususb on /proc/bus/usb type usbfs (rw)
> udev on /dev type tmpfs (rw,mode=0755)
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
> fusectl on /sys/fs/fuse/connections type fusectl (rw)
> /dev/sda7 on /tmp type ext2 (rw,noatime,errors=remount-ro,acl)
> /dev/sda8 on /export type ext3 (rw,noatime,errors=remount-ro,acl)
> /dev/sda1 on /winxp type ntfs (rw,umask=002,gid=10000,nls=utf8)

So they are ext3/ext2/ntfs.  What if you umount ntfs? and ext2 if possible?

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Am Montag, 7. Januar 2008 schrieb Peter Zijlstra:
> On Mon, 2008-01-07 at 14:24 +0100, Joerg Platte wrote:
>
> This is from: 2.6.24-rc7
>
> > kernel: pdflush       D f41c2f14     0 18822      2
> > kernel:        f673f000 00000046 00000286 f41c2f14 f5194ce0 00000286
> > 00000286 f41c2f14 kernel:        00175279 f41c2f6c 00000000 c0271f6c
> > f5ff363c f5ff3644 c0354a90 c0354a90 kernel:        00175279 c0123251
> > f5194b80 c03546c0 c0271f67 6c666470 00687375 00000000 kernel: Call Trace:
> > kernel:  [<c0271f6c>] schedule_timeout+0x6e/0x8b
> > kernel:  [<c0123251>] process_timeout+0x0/0x5
> > kernel:  [<c0271f67>] schedule_timeout+0x69/0x8b
> > kernel:  [<c027179a>] __sched_text_start+0x3a/0x70
> > kernel:  [<c014d34b>] congestion_wait+0x4e/0x62
> > kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
> > kernel:  [<c014971e>] pdflush+0x0/0x1bf
> > kernel:  [<c01493c6>] wb_kupdate+0x8c/0xd1
> > kernel:  [<c014971e>] pdflush+0x0/0x1bf
> > kernel:  [<c0149839>] pdflush+0x11b/0x1bf
> > kernel:  [<c014933a>] wb_kupdate+0x0/0xd1
> > kernel:  [<c012bd71>] kthread+0x36/0x5d
> > kernel:  [<c012bd3b>] kthread+0x0/0x5d
> > kernel:  [<c010493b>] kernel_thread_helper+0x7/0x10
> > kernel:  =======================
>
> What filesystem are you using?

Here you can see all currently mounted filesystems:

/dev/sda6 on / type ext3 (rw,noatime,errors=remount-ro,acl)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/sda7 on /tmp type ext2 (rw,noatime,errors=remount-ro,acl)
/dev/sda8 on /export type ext3 (rw,noatime,errors=remount-ro,acl)
/dev/sda1 on /winxp type ntfs (rw,umask=002,gid=10000,nls=utf8)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
automount(pid4936) on /amnt type autofs 
(rw,fd=4,pgrp=4936,minproto=2,maxproto=4)
automount(pid4965) on /home type autofs 
(rw,fd=4,pgrp=4965,minproto=2,maxproto=4)
nfsd on /proc/fs/nfsd type nfsd (rw)
/export/home/jplatte on /home/jplatte type none (rw,bind)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc 
(rw,noexec,nosuid,nodev)

regards,
Jörg

-- 
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

Re: regression: 100% io-wait with 2.6.24-rcX

[Peter Zijlstra]

On Mon, 2008-01-07 at 14:24 +0100, Joerg Platte wrote:

This is from: 2.6.24-rc7

> kernel: pdflush       D f41c2f14     0 18822      2
> kernel:        f673f000 00000046 00000286 f41c2f14 f5194ce0 00000286 00000286 f41c2f14 
> kernel:        00175279 f41c2f6c 00000000 c0271f6c f5ff363c f5ff3644 c0354a90 c0354a90 
> kernel:        00175279 c0123251 f5194b80 c03546c0 c0271f67 6c666470 00687375 00000000 
> kernel: Call Trace:
> kernel:  [<c0271f6c>] schedule_timeout+0x6e/0x8b
> kernel:  [<c0123251>] process_timeout+0x0/0x5
> kernel:  [<c0271f67>] schedule_timeout+0x69/0x8b
> kernel:  [<c027179a>] __sched_text_start+0x3a/0x70
> kernel:  [<c014d34b>] congestion_wait+0x4e/0x62
> kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
> kernel:  [<c014971e>] pdflush+0x0/0x1bf
> kernel:  [<c01493c6>] wb_kupdate+0x8c/0xd1
> kernel:  [<c014971e>] pdflush+0x0/0x1bf
> kernel:  [<c0149839>] pdflush+0x11b/0x1bf
> kernel:  [<c014933a>] wb_kupdate+0x0/0xd1
> kernel:  [<c012bd71>] kthread+0x36/0x5d
> kernel:  [<c012bd3b>] kthread+0x0/0x5d
> kernel:  [<c010493b>] kernel_thread_helper+0x7/0x10
> kernel:  =======================

What filesystem are you using?

Re: regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

Am Montag, 7. Januar 2008 schrieb Ingo Molnar:

> do:
>
>   echo t > /proc/sysrq-trigger
>
> and send us the dmesg output. If the dmesg output does not include the
> bootup bits then increase CONFIG_LOG_BUF_SHIFT to 20 or so:
>
>   CONFIG_LOG_BUF_SHIFT=20
>
> to have a large enough kernel messages buffer.

The buffer was too small, hence I copied the relevant parts 
of /var/log/kern.log, I hope it contains all required information as well.

regards,
Jörg

[-- Attachment #2: dmesg --]
[-- Type: text/plain, Size: 17530 bytes --]

kernel:  f509eb84 f50dc700 f50f3be4 00000246 f50f3be4 f5c37b80 c01672ef 
kernel: Call Trace:
kernel:  [<c0271f11>] schedule_timeout+0x13/0x8b
kernel:  [<c01672ef>] pipe_poll+0x24/0x7f
kernel:  [<c016d53f>] do_select+0x370/0x3b9
kernel:  [<c016daf0>] __pollwait+0x0/0xa9
kernel:  [<c0118a51>] default_wake_function+0x0/0x5
last message repeated 4 times
kernel:  [<c0146ba7>] __rmqueue+0x14/0x196
kernel:  [<c01169ce>] enqueue_entity+0x2b/0x3d
kernel:  [<c01169f6>] enqueue_task_fair+0x16/0x24
kernel:  [<c011642e>] enqueue_task+0x3f/0x4a
kernel:  [<c01162b8>] __wake_up_common+0x31/0x56
kernel:  [<c015e1ef>] __slab_free+0x5b/0x279
kernel:  [<c0215e3a>] __kfree_skb+0x8/0x63
kernel:  [<c0214a2c>] sock_wfree+0x20/0x34
kernel:  [<c0216521>] skb_release_all+0xa1/0xf6
kernel:  [<f8892813>] unix_stream_recvmsg+0x3e9/0x4d6 [unix]
kernel:  [<c016d774>] core_sys_select+0x1ec/0x282
kernel:  [<c0210d34>] sock_aio_read+0xf2/0xfa
kernel:  [<c0123d9a>] recalc_sigpending+0xa/0x2d
kernel:  [<c0162077>] do_sync_read+0xc6/0x109
kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
kernel:  [<f88929a4>] unix_ioctl+0xa4/0xac [unix]
kernel:  [<c0210adb>] sock_ioctl+0x1b2/0x1d7
kernel:  [<c016dc39>] sys_select+0xa0/0x167
kernel:  [<c0103d7a>] sysenter_past_esp+0x5f/0x85
kernel:  [<c0270000>] netlbl_unlabel_list+0x9f/0x1bf
kernel:  =======================
kernel: bash          R running      0  8034   7712
kernel: bash          S 000000fd     0  8062   7713
kernel:        dfcc0a80 00000082 12210264 000000fd f537ace0 01366da4 00000000 7fffffff 
kernel:        dfc77000 00000001 bf9130cf c0271f11 f78127b0 00000000 f535ffb8 f535ff90 
kernel:        f537b034 00000000 bf9130cf f535ffb8 f535f000 c010348a 00000000 080c5c60 
kernel: Call Trace:
kernel:  [<c0271f11>] schedule_timeout+0x13/0x8b
kernel:  [<c010348a>] do_notify_resume+0x81/0x5f6
kernel:  [<c01f014b>] read_chan+0x2fc/0x521
kernel:  [<c0118a51>] default_wake_function+0x0/0x5
kernel:  [<c01efe4f>] read_chan+0x0/0x521
kernel:  [<c01ed351>] tty_read+0x6f/0xad
kernel:  [<c01ed2e2>] tty_read+0x0/0xad
kernel:  [<c01628e5>] vfs_read+0x9f/0x149
kernel:  [<c0162c74>] sys_read+0x41/0x67
kernel:  [<c0103d7a>] sysenter_past_esp+0x5f/0x85
kernel:  =======================
kernel: kwalletmanage S 00000648     0  8371      1
kernel:        f4128600 00200086 fcf08470 00000648 f52fd860 0006eb74 00000000 7fffffff 
kernel:        f407a500 00000400 f4176f4c c0271f11 f414b200 00000000 00000000 00200200 
kernel:        0016e496 c0123251 f4123e00 f4176be4 00200246 f4176be4 f407a500 001672ef 
kernel: Call Trace:
kernel:  [<c0271f11>] schedule_timeout+0x13/0x8b
kernel:  [<c0123251>] process_timeout+0x0/0x5
kernel:  [<c016d53f>] do_select+0x370/0x3b9
kernel:  [<c016daf0>] __pollwait+0x0/0xa9
kernel:  [<c0118a51>] default_wake_function+0x0/0x5
last message repeated 4 times
kernel:  [<c01abd85>] elv_insert+0xa6/0x146
kernel:  [<c01169ce>] enqueue_entity+0x2b/0x3d
kernel:  [<c01169f6>] enqueue_task_fair+0x16/0x24
kernel:  [<c011642e>] enqueue_task+0x3f/0x4a
kernel:  [<c01169ce>] enqueue_entity+0x2b/0x3d
kernel:  [<c01169f6>] enqueue_task_fair+0x16/0x24
kernel:  [<c011642e>] enqueue_task+0x3f/0x4a
kernel:  [<c01162b8>] __wake_up_common+0x31/0x56
kernel:  [<c02131ab>] sock_alloc_send_skb+0x7c/0x193
kernel:  [<c01162b8>] __wake_up_common+0x31/0x56
kernel:  [<c021474d>] sock_def_readable+0x31/0x6b
kernel:  [<f8892c62>] unix_stream_sendmsg+0x263/0x32c [unix]
kernel:  [<c016d774>] core_sys_select+0x1ec/0x282
kernel:  [<c0210c3a>] sock_aio_write+0xe0/0xe8
kernel:  [<c01681c4>] pipe_read+0x32c/0x338
kernel:  [<c0161f6e>] do_sync_write+0xc6/0x109
kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
kernel:  [<c01508b4>] free_pgtables+0x90/0xa0
kernel:  [<f88929a4>] unix_ioctl+0xa4/0xac [unix]
kernel:  [<c0210adb>] sock_ioctl+0x1b2/0x1d7
kernel:  [<c016dc39>] sys_select+0xa0/0x167
kernel:  [<c0103d7a>] sysenter_past_esp+0x5f/0x85
kernel:  =======================
kernel: kttsd         S 00000655     0  8384      1
kernel:        dfcc0900 00200086 af7cbb2c 00000655 f52bcce0 00044e4d 00000000 7fffffff 
kernel:        f4131180 00000400 f4050f4c c0271f11 c010478b f889407c 00000000 f4ba7000 
kernel:        f416a600 00000020 f4075c40 f4050be4 00200246 f4050be4 f4131180 c01672ef 
kernel: Call Trace:
kernel:  [<c0271f11>] schedule_timeout+0x13/0x8b
kernel:  [<c010478b>] common_interrupt+0x23/0x28
kernel:  [<c01672ef>] pipe_poll+0x24/0x7f
kernel:  [<c016d53f>] do_select+0x370/0x3b9
kernel:  [<c016daf0>] __pollwait+0x0/0xa9
kernel:  [<c0118a51>] default_wake_function+0x0/0x5
last message repeated 4 times
kernel:  [<c017e48d>] __getblk+0x14/0x1cd
kernel:  [<c0160960>] get_unused_fd_flags+0x55/0xcb
kernel:  [<f9a154ff>] ext3_getblk+0xb9/0x173 [ext3]
kernel:  [<c0218cb7>] scm_detach_fds+0xec/0x125
kernel:  [<c0218982>] __scm_destroy+0x23/0x38
kernel:  [<c01169ce>] enqueue_entity+0x2b/0x3d
kernel:  [<c01169f6>] enqueue_task_fair+0x16/0x24
kernel:  [<c011642e>] enqueue_task+0x3f/0x4a
kernel:  [<c01162b8>] __wake_up_common+0x31/0x56
kernel:  [<f889357c>] unix_write_space+0x32/0x6a [unix]
kernel:  [<c0214a2c>] sock_wfree+0x20/0x34
kernel:  [<c0216521>] skb_release_all+0xa1/0xf6
kernel:  [<f8892813>] unix_stream_recvmsg+0x3e9/0x4d6 [unix]
kernel:  [<c016d774>] core_sys_select+0x1ec/0x282
kernel:  [<c0210d34>] sock_aio_read+0xf2/0xfa
kernel:  [<c0123d9a>] recalc_sigpending+0xa/0x2d
kernel:  [<c0162077>] do_sync_read+0xc6/0x109
kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
kernel:  [<f88929a4>] unix_ioctl+0xa4/0xac [unix]
kernel:  [<c0210adb>] sock_ioctl+0x1b2/0x1d7
kernel:  [<c016dc39>] sys_select+0xa0/0x167
kernel:  [<c0103d7a>] sysenter_past_esp+0x5f/0x85
kernel:  [<c0270000>] netlbl_unlabel_list+0x9f/0x1bf
kernel:  =======================
kernel: konqueror     S 00000648     0  8665   7259
kernel:        f40fb000 00000086 fa0dc86a 00000648 f418d2a0 00003d78 00000000 7fffffff 
kernel:        f5049c80 00000800 f41b5f4c c0271f11 00000200 f41b5f4c 00000000 00200200 
kernel:        0016a41f c0123251 f5cf31c0 00000246 f5cf31c0 f41b5be4 f889106f 001672ef 
kernel: Call Trace:
kernel:  [<c0271f11>] schedule_timeout+0x13/0x8b
kernel:  [<c0123251>] process_timeout+0x0/0x5
kernel:  [<f889106f>] unix_poll+0x17/0x8b [unix]
kernel:  [<c016d53f>] do_select+0x370/0x3b9
kernel:  [<c016daf0>] __pollwait+0x0/0xa9
kernel:  [<c0118a51>] default_wake_function+0x0/0x5
last message repeated 4 times
kernel:  [<c017e46f>] __find_get_block+0x13d/0x147
kernel:  [<c01169ce>] enqueue_entity+0x2b/0x3d
kernel:  [<c017e76b>] __set_page_dirty+0x125/0x132
kernel:  [<c01169ce>] enqueue_entity+0x2b/0x3d
kernel:  [<c01169f6>] enqueue_task_fair+0x16/0x24
kernel:  [<c011642e>] enqueue_task+0x3f/0x4a
kernel:  [<c01162b8>] __wake_up_common+0x31/0x56
kernel:  [<f88927ca>] unix_stream_recvmsg+0x3a0/0x4d6 [unix]
kernel:  [<c0214a2c>] sock_wfree+0x20/0x34
kernel:  [<f88927ca>] unix_stream_recvmsg+0x3a0/0x4d6 [unix]
kernel:  [<f88927ca>] unix_stream_recvmsg+0x3a0/0x4d6 [unix]
kernel:  [<f8892813>] unix_stream_recvmsg+0x3e9/0x4d6 [unix]
kernel:  [<c016d774>] core_sys_select+0x1ec/0x282
kernel:  [<c0210d34>] sock_aio_read+0xf2/0xfa
kernel:  [<c0162077>] do_sync_read+0xc6/0x109
kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
kernel:  [<c0150079>] handle_mm_fault+0x268/0x57d
kernel:  [<f88929a4>] unix_ioctl+0xa4/0xac [unix]
kernel:  [<c0210adb>] sock_ioctl+0x1b2/0x1d7
kernel:  [<c016dc39>] sys_select+0xa0/0x167
kernel:  [<c0103d7a>] sysenter_past_esp+0x5f/0x85
kernel:  [<c0270000>] netlbl_unlabel_list+0x9f/0x1bf
kernel:  =======================
kernel: kio_file      S 00000195     0 10548   7259
kernel:        f52e0d80 00000082 97e3a48e 00000195 f5032160 00000000 00000000 7fffffff 
kernel:        f4099380 00000020 f509af4c c0271f11 00000000 00000000 00000000 00000000 
kernel:        00000000 00000000 f66b1a80 00000246 f66b1a80 f509abe4 f889106f 0009abe4 
kernel: Call Trace:
kernel:  [<c0271f11>] schedule_timeout+0x13/0x8b
kernel:  [<f889106f>] unix_poll+0x17/0x8b [unix]
kernel:  [<c016d53f>] do_select+0x370/0x3b9
kernel:  [<c016daf0>] __pollwait+0x0/0xa9
kernel:  [<c0118a51>] default_wake_function+0x0/0x5
kernel:  [<c0147e3f>] __alloc_pages+0x5d/0x2d9
kernel:  [<c0146ba7>] __rmqueue+0x14/0x196
kernel:  [<c01169ce>] enqueue_entity+0x2b/0x3d
kernel:  [<c01169f6>] enqueue_task_fair+0x16/0x24
kernel:  [<c011642e>] enqueue_task+0x3f/0x4a
kernel:  [<c01162b8>] __wake_up_common+0x31/0x56
kernel:  [<c02131ab>] sock_alloc_send_skb+0x7c/0x193
kernel:  [<c01162b8>] __wake_up_common+0x31/0x56
kernel:  [<c021474d>] sock_def_readable+0x31/0x6b
kernel:  [<f8892c62>] unix_stream_sendmsg+0x263/0x32c [unix]
kernel:  [<c016d774>] core_sys_select+0x1ec/0x282
kernel:  [<c0210c3a>] sock_aio_write+0xe0/0xe8
kernel:  [<c0161f6e>] do_sync_write+0xc6/0x109
kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
kernel:  [<c014fffc>] handle_mm_fault+0x1eb/0x57d
kernel:  [<c016dc39>] sys_select+0xa0/0x167
kernel:  [<c0162cdb>] sys_write+0x41/0x67
kernel:  [<c0103d7a>] sysenter_past_esp+0x5f/0x85
kernel:  =======================
kernel: pdflush       D f41c2f14     0 18822      2
kernel:        f673f000 00000046 00000286 f41c2f14 f5194ce0 00000286 00000286 f41c2f14 
kernel:        00175279 f41c2f6c 00000000 c0271f6c f5ff363c f5ff3644 c0354a90 c0354a90 
kernel:        00175279 c0123251 f5194b80 c03546c0 c0271f67 6c666470 00687375 00000000 
kernel: Call Trace:
kernel:  [<c0271f6c>] schedule_timeout+0x6e/0x8b
kernel:  [<c0123251>] process_timeout+0x0/0x5
kernel:  [<c0271f67>] schedule_timeout+0x69/0x8b
kernel:  [<c027179a>] __sched_text_start+0x3a/0x70
kernel:  [<c014d34b>] congestion_wait+0x4e/0x62
kernel:  [<c012be2b>] autoremove_wake_function+0x0/0x33
kernel:  [<c014971e>] pdflush+0x0/0x1bf
kernel:  [<c01493c6>] wb_kupdate+0x8c/0xd1
kernel:  [<c014971e>] pdflush+0x0/0x1bf
kernel:  [<c0149839>] pdflush+0x11b/0x1bf
kernel:  [<c014933a>] wb_kupdate+0x0/0xd1
kernel:  [<c012bd71>] kthread+0x36/0x5d
kernel:  [<c012bd3b>] kthread+0x0/0x5d
kernel:  [<c010493b>] kernel_thread_helper+0x7/0x10
kernel:  =======================
kernel: Sched Debug Version: v0.07, 2.6.24-rc7 #1
kernel: now at 5394978.742321 msecs
kernel:   .sysctl_sched_latency                    : 20.000000
kernel:   .sysctl_sched_min_granularity            : 4.000000
kernel:   .sysctl_sched_wakeup_granularity         : 10.000000
kernel:   .sysctl_sched_batch_wakeup_granularity   : 10.000000
kernel:   .sysctl_sched_child_runs_first           : 0.000001
kernel:   .sysctl_sched_features                   : 7
kernel: 
kernel: cpu#0, 599.501 MHz
kernel:   .nr_running                    : 3
kernel:   .load                          : 11596
kernel:   .nr_switches                   : 3472897
kernel:   .nr_load_updates               : 789297
kernel:   .nr_uninterruptible            : 1
kernel:   .jiffies                       : 1528423
kernel:   .next_balance                  : 0.000000
kernel:   .curr->pid                     : 8034
kernel:   .clock                         : 6967153.202929
kernel:   .idle_clock                    : 4447303.677171
kernel:   .prev_clock_raw                : 4676606.867330
kernel:   .clock_warps                   : 103
kernel:   .clock_overflows               : 4982682
kernel:   .clock_deep_idle_events        : 944576
kernel:   .clock_max_delta               : 3.333116
kernel:   .cpu_load[0]                   : 11596
kernel:   .cpu_load[1]                   : 5798
kernel:   .cpu_load[2]                   : 2899
kernel:   .cpu_load[3]                   : 1450
kernel:   .cpu_load[4]                   : 744
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 1
kernel:   .load                          : 2048
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 0.000001
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 0.000001
kernel:   .spread                        : 0.000000
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 0
kernel:   .load                          : 0
kernel:   .nr_spread_over                : 0
kernel: 
kernel: cfs_rq
kernel:   .exec_clock                    : 0.000000
kernel:   .MIN_vruntime                  : 214548.087264
kernel:   .min_vruntime                  : 616382.544882
kernel:   .max_vruntime                  : 214549.007330
kernel:   .spread                        : 0.920066
kernel:   .spread0                       : 0.000000
kernel:   .nr_running                    : 3
kernel:   .load                          : 11596
kernel:   .nr_spread_over                : 0
kernel: 
kernel: runnable tasks:
kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
kernel: ----------------------------------------------------------------------------------------------------------
kernel:            klogd  4715    214548.087264      4266   120               0               0               0.000000               0.000000               0.000000
kernel:             Xorg  5872    214549.007330    983644   110               0               0               0.000000               0.000000               0.000000
kernel: R           bash  8034    214548.926777      1418   120               0               0               0.000000               0.000000               0.000000
kernel: 

Re: regression: 100% io-wait with 2.6.24-rcX

[Ingo Molnar]

* Joerg Platte <lists@naasa.net> wrote:

> Hi,
> 
> when booting kernel 2.6.24-rc{4,5,6,7} top reports up to 100% iowait, 
> even if no program accesses the disc on my Thinkpad T40p. Kernel 
> 2.6.23.12 does not suffer from this. Is there anything I can do to 
> find out which process or which part of the kernel is responsible for 
> this? I can try to bisect it, but maybe there are other possibilities 
> to debug this, since I cannot boot this computer frequently. I 
> discovered, that there is no iowait within the first few seconds after 
> waking up from suspend to ram...

do:

  echo t > /proc/sysrq-trigger

and send us the dmesg output. If the dmesg output does not include the 
bootup bits then increase CONFIG_LOG_BUF_SHIFT to 20 or so:

  CONFIG_LOG_BUF_SHIFT=20

to have a large enough kernel messages buffer.

	Ingo

regression: 100% io-wait with 2.6.24-rcX

[Joerg Platte]

Hi,

when booting kernel 2.6.24-rc{4,5,6,7} top reports up to 100% iowait, even if 
no program accesses the disc on my Thinkpad T40p. Kernel 2.6.23.12 does not 
suffer from this. Is there anything I can do to find out which process or 
which part of the kernel is responsible for this? I can try to bisect it, but 
maybe there are other possibilities to debug this, since I cannot boot this 
computer frequently. I discovered, that there is no iowait within the first 
few seconds after waking up from suspend to ram...

regards,
Jörg