Mysterious CFQ crash and RCU

Mysterious CFQ crash and RCU

[Vivek Goyal]

Hi Paul,

Few people have reported hard to reproduce CFQ crash time to time. Most of
the people could not reproduce the issue consistently hence we could not
debug it further. Now for the first time, Paul Bolle, can reproduce this issue
reliably on his machine, so I am hopeful that we can get to bottom of this
long time pending mysterious issue.

There might be many bz tracking this issue but one of the bzs is 577968 in
bugzilla.redhat.com

BUG: unable to handle kernel paging request at 6b6b6b6b
IP: [<c05e859d>] call_for_each_cic+0x29/0x44
*pde = 00000000 
Oops: 0000 [#1] SMP 
last sysfs file:
/sys/devices/LNXSYSTM:00/device:00/PNP0A03:00/device:01/PNP0C09:00/PNP0C0A:00/power_supply/BAT0/energy_full
Modules linked in: bnep bluetooth fuse cpufreq_ondemand acpi_cpufreq mperf
ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_netbios_ns
ip6table_filter nf_conntrack_broadcast ip6_tables snd_intel8x0m
snd_intel8x0
arc4 snd_ac97_codec ath5k ac97_bus snd_seq ath snd_seq_device mac80211
snd_pcm
ppdev thinkpad_acpi cfg80211 microcode snd_timer parport_pc e1000 snd
rfkill
parport iTCO_wdt soundcore snd_page_alloc iTCO_vendor_support joydev
i2c_i801
pcspkr uinput ipv6 yenta_socket video radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]

Pid: 11397, comm: mandb Not tainted 2.6.39-0.rc5.git0.0.local.fc16.i686 #1
IBM 
      /       
EIP: 0060:[<c05e859d>] EFLAGS: 00010202 CPU: 0
EIP is at call_for_each_cic+0x29/0x44
EAX: 00000001 EBX: 6b6b6b6b ECX: 00000246 EDX: c0ad4a98
ESI: ecfd0980 EDI: c05e88a4 EBP: c670bf18 ESP: c670bf0c
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process mandb (pid: 11397, ti=c670a000 task=d40769f0 task.ti=c670a000)
Stack:
 ecfd0980 d40769f0 d4076dc0 c670bf20 c05e85ca c670bf2c c05ddc86 ecfd0980
 c670bf40 c05ddd16 d40769f0 00000012 c670bf74 c670bf8c c0440813 f2a182e8
 d401ac08 c670bf70 c04f48da 00000001 d4076bf4 d40769e8 d4076bf4 00000001
Call Trace:
 [<c05e85ca>] cfq_free_io_context+0x12/0x14
 [<c05ddc86>] put_io_context+0x34/0x5c
 [<c05ddd16>] exit_io_context+0x68/0x6d
 [<c0440813>] do_exit+0x63e/0x661
 [<c04f48da>] ? fsnotify_modify+0x64/0x6c
 [<c0440a4d>] do_group_exit+0x63/0x86
 [<c0440a88>] sys_exit_group+0x18/0x18
 [<c082551f>] sysenter_do_call+0x12/0x38
Code: 5d c3 55 89 e5 57 56 53 3e 8d 74 26 00 89 c6 89 d7 e8 3a ec ff ff 8b
5e
4c e8 00 b1 e6 ff 85 c0 74 05 e8 b7 ff ff ff 85 db 74 11 <8b> 03 0f 18 00
90 8d
53 d8 89 f0 ff d7 8b 1b eb dd e8 49 ec ff 
EIP: [<c05e859d>] call_for_each_cic+0x29/0x44 SS:ESP 0068:c670bf0c
CR2: 000000006b6b6b6b
---[ end trace 70dea08ef41fda48 ]---

In summary once in a while people notice CFQ crash. Debugging shows that
we have a rcu protected hlist of elements of type cfq_io_context. Head of
the list is at ioc->cic_list. We crash while traversing ioc->cic_list
under rcu.

Looks like an element which we are trying to fetch the next pointer from got
freed to slab, and the object got poisoned with 0x6b6b6b6b.. and then we
tried to fetch the next object pointer and ended up dereferencing a
freed object and CFQ crashes.

The function in question here is call_for_each_cic() in block/cfq-iosched.c

We free the cfq_io_context object using call_rcu(). So on the surface
it looks like that we decoupled a cfq_io_context object from the hash
list and scheduled a call_rcu() so that it is freed after rcu grace
period but somehow object got freed earlier and got released to slab
and got poisoned.

Is it possible? We have looked at the code many a times and we think
that rcu locking around it is fine. Is it possible that a call_rcu()
can fire before rcu grace period is over.

I had put a debug patch in CFQ (details are in bugzilla) and I can
see that after decoupling the object from the hash list, it got
freed while we were still under rcu_read_lock().

Is there any known issue or is there any quick tip on how can I 
go about debugging it further from rcu point of view.

Thanks
Vivek

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Thu, May 19, 2011 at 06:24:04PM -0400, Vivek Goyal wrote:
> Hi Paul,
> 
> Few people have reported hard to reproduce CFQ crash time to time. Most of
> the people could not reproduce the issue consistently hence we could not
> debug it further. Now for the first time, Paul Bolle, can reproduce this issue
> reliably on his machine, so I am hopeful that we can get to bottom of this
> long time pending mysterious issue.
> 
> There might be many bz tracking this issue but one of the bzs is 577968 in
> bugzilla.redhat.com

It does look like a tough one!

> BUG: unable to handle kernel paging request at 6b6b6b6b
> IP: [<c05e859d>] call_for_each_cic+0x29/0x44
> *pde = 00000000 
> Oops: 0000 [#1] SMP 
> last sysfs file:
> /sys/devices/LNXSYSTM:00/device:00/PNP0A03:00/device:01/PNP0C09:00/PNP0C0A:00/power_supply/BAT0/energy_full
> Modules linked in: bnep bluetooth fuse cpufreq_ondemand acpi_cpufreq mperf
> ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_netbios_ns
> ip6table_filter nf_conntrack_broadcast ip6_tables snd_intel8x0m
> snd_intel8x0
> arc4 snd_ac97_codec ath5k ac97_bus snd_seq ath snd_seq_device mac80211
> snd_pcm
> ppdev thinkpad_acpi cfg80211 microcode snd_timer parport_pc e1000 snd
> rfkill
> parport iTCO_wdt soundcore snd_page_alloc iTCO_vendor_support joydev
> i2c_i801
> pcspkr uinput ipv6 yenta_socket video radeon ttm drm_kms_helper drm
> i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
> 
> Pid: 11397, comm: mandb Not tainted 2.6.39-0.rc5.git0.0.local.fc16.i686 #1
> IBM 
>       /       
> EIP: 0060:[<c05e859d>] EFLAGS: 00010202 CPU: 0
> EIP is at call_for_each_cic+0x29/0x44
> EAX: 00000001 EBX: 6b6b6b6b ECX: 00000246 EDX: c0ad4a98
> ESI: ecfd0980 EDI: c05e88a4 EBP: c670bf18 ESP: c670bf0c
>  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> Process mandb (pid: 11397, ti=c670a000 task=d40769f0 task.ti=c670a000)
> Stack:
>  ecfd0980 d40769f0 d4076dc0 c670bf20 c05e85ca c670bf2c c05ddc86 ecfd0980
>  c670bf40 c05ddd16 d40769f0 00000012 c670bf74 c670bf8c c0440813 f2a182e8
>  d401ac08 c670bf70 c04f48da 00000001 d4076bf4 d40769e8 d4076bf4 00000001
> Call Trace:
>  [<c05e85ca>] cfq_free_io_context+0x12/0x14
>  [<c05ddc86>] put_io_context+0x34/0x5c
>  [<c05ddd16>] exit_io_context+0x68/0x6d
>  [<c0440813>] do_exit+0x63e/0x661
>  [<c04f48da>] ? fsnotify_modify+0x64/0x6c
>  [<c0440a4d>] do_group_exit+0x63/0x86
>  [<c0440a88>] sys_exit_group+0x18/0x18
>  [<c082551f>] sysenter_do_call+0x12/0x38
> Code: 5d c3 55 89 e5 57 56 53 3e 8d 74 26 00 89 c6 89 d7 e8 3a ec ff ff 8b
> 5e
> 4c e8 00 b1 e6 ff 85 c0 74 05 e8 b7 ff ff ff 85 db 74 11 <8b> 03 0f 18 00
> 90 8d
> 53 d8 89 f0 ff d7 8b 1b eb dd e8 49 ec ff 
> EIP: [<c05e859d>] call_for_each_cic+0x29/0x44 SS:ESP 0068:c670bf0c
> CR2: 000000006b6b6b6b
> ---[ end trace 70dea08ef41fda48 ]---
> 
> In summary once in a while people notice CFQ crash. Debugging shows that
> we have a rcu protected hlist of elements of type cfq_io_context. Head of
> the list is at ioc->cic_list. We crash while traversing ioc->cic_list
> under rcu.
> 
> Looks like an element which we are trying to fetch the next pointer from got
> freed to slab, and the object got poisoned with 0x6b6b6b6b.. and then we
> tried to fetch the next object pointer and ended up dereferencing a
> freed object and CFQ crashes.
> 
> The function in question here is call_for_each_cic() in block/cfq-iosched.c
> 
> We free the cfq_io_context object using call_rcu(). So on the surface
> it looks like that we decoupled a cfq_io_context object from the hash
> list and scheduled a call_rcu() so that it is freed after rcu grace
> period but somehow object got freed earlier and got released to slab
> and got poisoned.
> 
> Is it possible? We have looked at the code many a times and we think
> that rcu locking around it is fine. Is it possible that a call_rcu()
> can fire before rcu grace period is over.

If it does, that would be a bug in RCU.

> I had put a debug patch in CFQ (details are in bugzilla) and I can
> see that after decoupling the object from the hash list, it got
> freed while we were still under rcu_read_lock().
> 
> Is there any known issue or is there any quick tip on how can I 
> go about debugging it further from rcu point of view.

First for uses of RCU:

o	One thing to try would be CONFIG_PROVE_RCU, which could help
	find missing rcu_read_lock()s and similar.  Some years back, it
	used to be the case that spin_lock() implied rcu_read_lock(),
	but it no longer does.	There might still be some cases where
	spin_lock() needs to have an rcu_read_lock() added.

o	There are a few entries in the bugzilla mentioning that elements
	are being removed more often than expected.  There is a config
	option CONFIG_DEBUG_OBJECTS_RCU_HEAD that complains if the same
	object is passed to call_rcu() before the grace period ends for
	the first round.

o	Try switching between CONFIG_TREE_RCU and CONFIG_TREE_PREEMPT_RCU.
	These two settings are each sensitive to different forms of abuse.
	For example, if you have CONFIG_PREEMPT=n and CONFIG_TREE_RCU=y,
	illegally placing a synchronize_rcu() -- or anything else that
	blocks -- in an RCU read-side critical section will silently
	partition that RCU read-side critical section.  In contrast,
	CONFIG_TREE_PREEMPT_RCU=y will complain about this.

Second, for RCU itself, CONFIG_RCU_TRACE enables counter-based tracing
in RCU.  Sampling each of the files in the debugfs directory "rcu"
before and after the badness (if possible) could help me see if anything
untoward is happening.

							Thanx, Paul

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

Paul,

On Sat, 2011-05-21 at 14:00 -0700, Paul E. McKenney wrote:
> On Thu, May 19, 2011 at 06:24:04PM -0400, Vivek Goyal wrote:
> It does look like a tough one!

Thank you!

> > Is it possible? We have looked at the code many a times and we think
> > that rcu locking around it is fine. Is it possible that a call_rcu()
> > can fire before rcu grace period is over.
> 
> If it does, that would be a bug in RCU.
> 
> > I had put a debug patch in CFQ (details are in bugzilla) and I can
> > see that after decoupling the object from the hash list, it got
> > freed while we were still under rcu_read_lock().
> > 
> > Is there any known issue or is there any quick tip on how can I 
> > go about debugging it further from rcu point of view.
> 
> First for uses of RCU:
> 
> o	One thing to try would be CONFIG_PROVE_RCU, which could help
> 	find missing rcu_read_lock()s and similar.  Some years back, it
> 	used to be the case that spin_lock() implied rcu_read_lock(),
> 	but it no longer does.	There might still be some cases where
> 	spin_lock() needs to have an rcu_read_lock() added.
> 
> o	There are a few entries in the bugzilla mentioning that elements
> 	are being removed more often than expected.  There is a config
> 	option CONFIG_DEBUG_OBJECTS_RCU_HEAD that complains if the same
> 	object is passed to call_rcu() before the grace period ends for
> 	the first round.
> 
> o	Try switching between CONFIG_TREE_RCU and CONFIG_TREE_PREEMPT_RCU.
> 	These two settings are each sensitive to different forms of abuse.
> 	For example, if you have CONFIG_PREEMPT=n and CONFIG_TREE_RCU=y,
> 	illegally placing a synchronize_rcu() -- or anything else that
> 	blocks -- in an RCU read-side critical section will silently
> 	partition that RCU read-side critical section.  In contrast,
> 	CONFIG_TREE_PREEMPT_RCU=y will complain about this.
> 
> Second, for RCU itself, CONFIG_RCU_TRACE enables counter-based tracing
> in RCU.  Sampling each of the files in the debugfs directory "rcu"
> before and after the badness (if possible) could help me see if anything
> untoward is happening.

Before we go down that route, I'd like to note that I seem to be unable
to reproduce this Oops under v2.6.39 (either using the first v2.6.39 rpm
for i686 shipped for Fedora Rawhide, or two versions of that rpm I built
locally).

Is anyone able to spot one or more commits in v2.6.39-rc7..v2.6.39 that
might have fixed this Oops? Or did my chance of hitting this Oops,
somehow, just got a lot smaller in v.2.6.39?

Please note that I have tried to reproduce this Oops very often, using
quite a number of kernels, so there's a non-zero chance I tricked myself
in seeing a pattern where there actually is none.


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Sun, May 22, 2011 at 12:23:50AM +0200, Paul Bolle wrote:
> Paul,
> 
> On Sat, 2011-05-21 at 14:00 -0700, Paul E. McKenney wrote:
> > On Thu, May 19, 2011 at 06:24:04PM -0400, Vivek Goyal wrote:
> > It does look like a tough one!
> 
> Thank you!
> 
> > > Is it possible? We have looked at the code many a times and we think
> > > that rcu locking around it is fine. Is it possible that a call_rcu()
> > > can fire before rcu grace period is over.
> > 
> > If it does, that would be a bug in RCU.
> > 
> > > I had put a debug patch in CFQ (details are in bugzilla) and I can
> > > see that after decoupling the object from the hash list, it got
> > > freed while we were still under rcu_read_lock().
> > > 
> > > Is there any known issue or is there any quick tip on how can I 
> > > go about debugging it further from rcu point of view.
> > 
> > First for uses of RCU:
> > 
> > o	One thing to try would be CONFIG_PROVE_RCU, which could help
> > 	find missing rcu_read_lock()s and similar.  Some years back, it
> > 	used to be the case that spin_lock() implied rcu_read_lock(),
> > 	but it no longer does.	There might still be some cases where
> > 	spin_lock() needs to have an rcu_read_lock() added.
> > 
> > o	There are a few entries in the bugzilla mentioning that elements
> > 	are being removed more often than expected.  There is a config
> > 	option CONFIG_DEBUG_OBJECTS_RCU_HEAD that complains if the same
> > 	object is passed to call_rcu() before the grace period ends for
> > 	the first round.
> > 
> > o	Try switching between CONFIG_TREE_RCU and CONFIG_TREE_PREEMPT_RCU.
> > 	These two settings are each sensitive to different forms of abuse.
> > 	For example, if you have CONFIG_PREEMPT=n and CONFIG_TREE_RCU=y,
> > 	illegally placing a synchronize_rcu() -- or anything else that
> > 	blocks -- in an RCU read-side critical section will silently
> > 	partition that RCU read-side critical section.  In contrast,
> > 	CONFIG_TREE_PREEMPT_RCU=y will complain about this.
> > 
> > Second, for RCU itself, CONFIG_RCU_TRACE enables counter-based tracing
> > in RCU.  Sampling each of the files in the debugfs directory "rcu"
> > before and after the badness (if possible) could help me see if anything
> > untoward is happening.
> 
> Before we go down that route, I'd like to note that I seem to be unable
> to reproduce this Oops under v2.6.39 (either using the first v2.6.39 rpm
> for i686 shipped for Fedora Rawhide, or two versions of that rpm I built
> locally).
> 
> Is anyone able to spot one or more commits in v2.6.39-rc7..v2.6.39 that
> might have fixed this Oops? Or did my chance of hitting this Oops,
> somehow, just got a lot smaller in v.2.6.39?

5f45c69589b7d ("read_lock() does not always imply rcu_read_lock()") might
well be a fix.

> Please note that I have tried to reproduce this Oops very often, using
> quite a number of kernels, so there's a non-zero chance I tricked myself
> in seeing a pattern where there actually is none.

Understood -- races can be a bit frustrating.  How long should you run
before you conclude that you fixed it?  ;-)

							Thanx, Paul

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Sat, 2011-05-21 at 16:54 -0700, Paul E. McKenney wrote:
> On Sun, May 22, 2011 at 12:23:50AM +0200, Paul Bolle wrote: 
> > Is anyone able to spot one or more commits in v2.6.39-rc7..v2.6.39 that
> > might have fixed this Oops? Or did my chance of hitting this Oops,
> > somehow, just got a lot smaller in v.2.6.39?

As I just discovered, Fedora Rawhide ships release candidates configured
with a lot more debugging options set than it ships final releases. So,
sadly, this didn't get fixed in v2.6.39. It just didn't show up because
one or more configuration options weren't set while they used to be set
in the preceding release candidates kernels.

> 5f45c69589b7d ("read_lock() does not always imply rcu_read_lock()") might
> well be a fix.

It doesn't really matter anymore, but that one was actually already
included in v2.6.39-rc5.


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Sun, May 22, 2011 at 09:30:57PM +0200, Paul Bolle wrote:
> On Sat, 2011-05-21 at 16:54 -0700, Paul E. McKenney wrote:
> > On Sun, May 22, 2011 at 12:23:50AM +0200, Paul Bolle wrote: 
> > > Is anyone able to spot one or more commits in v2.6.39-rc7..v2.6.39 that
> > > might have fixed this Oops? Or did my chance of hitting this Oops,
> > > somehow, just got a lot smaller in v.2.6.39?
> 
> As I just discovered, Fedora Rawhide ships release candidates configured
> with a lot more debugging options set than it ships final releases. So,
> sadly, this didn't get fixed in v2.6.39. It just didn't show up because
> one or more configuration options weren't set while they used to be set
> in the preceding release candidates kernels.
> 
> > 5f45c69589b7d ("read_lock() does not always imply rcu_read_lock()") might
> > well be a fix.
> 
> It doesn't really matter anymore, but that one was actually already
> included in v2.6.39-rc5.

OK, so maybe time to try out the debug approaches I called out earlier?

							Thanx, Paul

Re: Mysterious CFQ crash and RCU

[Vivek Goyal]

On Sat, May 21, 2011 at 02:00:13PM -0700, Paul E. McKenney wrote:

[..]
> > In summary once in a while people notice CFQ crash. Debugging shows that
> > we have a rcu protected hlist of elements of type cfq_io_context. Head of
> > the list is at ioc->cic_list. We crash while traversing ioc->cic_list
> > under rcu.
> > 
> > Looks like an element which we are trying to fetch the next pointer from got
> > freed to slab, and the object got poisoned with 0x6b6b6b6b.. and then we
> > tried to fetch the next object pointer and ended up dereferencing a
> > freed object and CFQ crashes.
> > 
> > The function in question here is call_for_each_cic() in block/cfq-iosched.c
> > 
> > We free the cfq_io_context object using call_rcu(). So on the surface
> > it looks like that we decoupled a cfq_io_context object from the hash
> > list and scheduled a call_rcu() so that it is freed after rcu grace
> > period but somehow object got freed earlier and got released to slab
> > and got poisoned.
> > 
> > Is it possible? We have looked at the code many a times and we think
> > that rcu locking around it is fine. Is it possible that a call_rcu()
> > can fire before rcu grace period is over.
> 
> If it does, that would be a bug in RCU.
> 
> > I had put a debug patch in CFQ (details are in bugzilla) and I can
> > see that after decoupling the object from the hash list, it got
> > freed while we were still under rcu_read_lock().
> > 
> > Is there any known issue or is there any quick tip on how can I 
> > go about debugging it further from rcu point of view.
> 

Thanks for the response paul.

> First for uses of RCU:
> 
> o	One thing to try would be CONFIG_PROVE_RCU, which could help
> 	find missing rcu_read_lock()s and similar.  Some years back, it
> 	used to be the case that spin_lock() implied rcu_read_lock(),
> 	but it no longer does.	There might still be some cases where
> 	spin_lock() needs to have an rcu_read_lock() added.
> 

I believe that PaulB already had CONFIG_PROVE_RCU=y for his kernels. I
also built a kernel CONFIG_PROVE_RCU=y and no warning popped up. In
fact it looks like (comment 113 in bz 577968) that with 2.6.39 if PaulB
takes fedora kernel release config andn enabled CONFIG_PROVE_RCU=y, he
can reproduce the problem.

I am wondering if CONFIG_PROVE_RCU has some side affects.

> o	There are a few entries in the bugzilla mentioning that elements
> 	are being removed more often than expected.  There is a config
> 	option CONFIG_DEBUG_OBJECTS_RCU_HEAD that complains if the same
> 	object is passed to call_rcu() before the grace period ends for
> 	the first round.

I noticed that CONFIG_DEBUG_OBJECTS_RCU_HEAD gets enabled only if
PREEMPT is enabled. In Paul's fedora config preemption is not enabled
and I see following.

# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set

So are you suggesting that we should explicitly enable preemption
and set CONFIG_PREEMPT=y and CONFIG_DEBUG_OBJECTS_RCU_HEAD=y and try
to reproduce the problem again?

> 
> o	Try switching between CONFIG_TREE_RCU and CONFIG_TREE_PREEMPT_RCU.
> 	These two settings are each sensitive to different forms of abuse.
> 	For example, if you have CONFIG_PREEMPT=n and CONFIG_TREE_RCU=y,
> 	illegally placing a synchronize_rcu() -- or anything else that
> 	blocks -- in an RCU read-side critical section will silently
> 	partition that RCU read-side critical section.  In contrast,
> 	CONFIG_TREE_PREEMPT_RCU=y will complain about this.

Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
the issue?
> 
> Second, for RCU itself, CONFIG_RCU_TRACE enables counter-based tracing
> in RCU.  Sampling each of the files in the debugfs directory "rcu"
> before and after the badness (if possible) could help me see if anything
> untoward is happening.

This sounds doable. So you don't want periodic polling of these rcu
files? I am assuming that this reading of rcu files is happening in
user space. How do I do polling at specific events (before and after
badness). Any suggestions ?

After badness we try to capture the crash dump. So hopefully appropriate
files we should be able to read from crash dump. So the key quesiton 
would be what's the easiest way to let a user space process poll these
files before badness and display on console.

Thanks
Vivek

Re: Mysterious CFQ crash and RCU

[Vivek Goyal]

On Sat, May 21, 2011 at 02:00:13PM -0700, Paul E. McKenney wrote:

[..]
> > Is there any known issue or is there any quick tip on how can I 
> > go about debugging it further from rcu point of view.
> 
> First for uses of RCU:
> 
> o	One thing to try would be CONFIG_PROVE_RCU, which could help
> 	find missing rcu_read_lock()s and similar.  Some years back, it
> 	used to be the case that spin_lock() implied rcu_read_lock(),
> 	but it no longer does.	There might still be some cases where
> 	spin_lock() needs to have an rcu_read_lock() added.

In this case we take explicit rcu_read_lock() in call_for_each_cic()
and do not rely on that spin_lock() also means rcu_read_lock().

call_for_each_cic() {
	rcu_read_lock();
	 hlist_for_each_entry_rcu(cic, n, &ioc->cic_list, cic_list)
                func(ioc, cic);
        rcu_read_unlock();
}

> 
> o	There are a few entries in the bugzilla mentioning that elements
> 	are being removed more often than expected.

Are you referring to my comments about additional messages of cgroup
changed. If yes, that issue has now been identified and I have posted
a fix to get rid of thos unnecessary calls.

https://lkml.org/lkml/2011/5/23/201

Is still there a need to enable CONFIG_DEBUG_OBJECTS_RCU_HEAD as the
kernel in question is non-preemptible one.

Thanks
Vivek

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

[-- Attachment #1: Type: text/plain, Size: 5128 bytes --]

On Mon, May 23, 2011 at 11:21:41AM -0400, Vivek Goyal wrote:
> On Sat, May 21, 2011 at 02:00:13PM -0700, Paul E. McKenney wrote:
> 
> [..]
> > > In summary once in a while people notice CFQ crash. Debugging shows that
> > > we have a rcu protected hlist of elements of type cfq_io_context. Head of
> > > the list is at ioc->cic_list. We crash while traversing ioc->cic_list
> > > under rcu.
> > > 
> > > Looks like an element which we are trying to fetch the next pointer from got
> > > freed to slab, and the object got poisoned with 0x6b6b6b6b.. and then we
> > > tried to fetch the next object pointer and ended up dereferencing a
> > > freed object and CFQ crashes.
> > > 
> > > The function in question here is call_for_each_cic() in block/cfq-iosched.c
> > > 
> > > We free the cfq_io_context object using call_rcu(). So on the surface
> > > it looks like that we decoupled a cfq_io_context object from the hash
> > > list and scheduled a call_rcu() so that it is freed after rcu grace
> > > period but somehow object got freed earlier and got released to slab
> > > and got poisoned.
> > > 
> > > Is it possible? We have looked at the code many a times and we think
> > > that rcu locking around it is fine. Is it possible that a call_rcu()
> > > can fire before rcu grace period is over.
> > 
> > If it does, that would be a bug in RCU.
> > 
> > > I had put a debug patch in CFQ (details are in bugzilla) and I can
> > > see that after decoupling the object from the hash list, it got
> > > freed while we were still under rcu_read_lock().
> > > 
> > > Is there any known issue or is there any quick tip on how can I 
> > > go about debugging it further from rcu point of view.
> > 
> 
> Thanks for the response paul.
> 
> > First for uses of RCU:
> > 
> > o	One thing to try would be CONFIG_PROVE_RCU, which could help
> > 	find missing rcu_read_lock()s and similar.  Some years back, it
> > 	used to be the case that spin_lock() implied rcu_read_lock(),
> > 	but it no longer does.	There might still be some cases where
> > 	spin_lock() needs to have an rcu_read_lock() added.
> > 
> 
> I believe that PaulB already had CONFIG_PROVE_RCU=y for his kernels. I
> also built a kernel CONFIG_PROVE_RCU=y and no warning popped up. In
> fact it looks like (comment 113 in bz 577968) that with 2.6.39 if PaulB
> takes fedora kernel release config andn enabled CONFIG_PROVE_RCU=y, he
> can reproduce the problem.
> 
> I am wondering if CONFIG_PROVE_RCU has some side affects.
> 
> > o	There are a few entries in the bugzilla mentioning that elements
> > 	are being removed more often than expected.  There is a config
> > 	option CONFIG_DEBUG_OBJECTS_RCU_HEAD that complains if the same
> > 	object is passed to call_rcu() before the grace period ends for
> > 	the first round.
> 
> I noticed that CONFIG_DEBUG_OBJECTS_RCU_HEAD gets enabled only if
> PREEMPT is enabled. In Paul's fedora config preemption is not enabled
> and I see following.
> 
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> 
> So are you suggesting that we should explicitly enable preemption
> and set CONFIG_PREEMPT=y and CONFIG_DEBUG_OBJECTS_RCU_HEAD=y and try
> to reproduce the problem again?

Running under CONFIG_PREEMPT=y (along with CONFIG_TREE_PREEMPT_RCU=y)
could be very helpful in and of itself.  CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
can also be helpful.  In post-2.6.39 mainline, it should be possible
to set CONFIG_DEBUG_OBJECTS_RCU_HEAD=y without CONFIG_PREEMPT=y, but
again, CONFIG_PREEMPT=y can help find problems.

> > o	Try switching between CONFIG_TREE_RCU and CONFIG_TREE_PREEMPT_RCU.
> > 	These two settings are each sensitive to different forms of abuse.
> > 	For example, if you have CONFIG_PREEMPT=n and CONFIG_TREE_RCU=y,
> > 	illegally placing a synchronize_rcu() -- or anything else that
> > 	blocks -- in an RCU read-side critical section will silently
> > 	partition that RCU read-side critical section.  In contrast,
> > 	CONFIG_TREE_PREEMPT_RCU=y will complain about this.
> 
> Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
> we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
> the issue?

Please!

> > Second, for RCU itself, CONFIG_RCU_TRACE enables counter-based tracing
> > in RCU.  Sampling each of the files in the debugfs directory "rcu"
> > before and after the badness (if possible) could help me see if anything
> > untoward is happening.
> 
> This sounds doable. So you don't want periodic polling of these rcu
> files? I am assuming that this reading of rcu files is happening in
> user space. How do I do polling at specific events (before and after
> badness). Any suggestions ?
> 
> After badness we try to capture the crash dump. So hopefully appropriate
> files we should be able to read from crash dump. So the key quesiton 
> would be what's the easiest way to let a user space process poll these
> files before badness and display on console.

Polling is fine.  Please see attached for a script to poll at 15-second
intervals.  Please also feel free to adjust, just tell me what you
adjusted.

							Thanx, Paul

[-- Attachment #2: collectdebugfs.sh --]
[-- Type: application/x-sh, Size: 1715 bytes --]

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
> Running under CONFIG_PREEMPT=y (along with CONFIG_TREE_PREEMPT_RCU=y)
> could be very helpful in and of itself.  CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> can also be helpful.  In post-2.6.39 mainline, it should be possible
> to set CONFIG_DEBUG_OBJECTS_RCU_HEAD=y without CONFIG_PREEMPT=y, but
> again, CONFIG_PREEMPT=y can help find problems.

0) The first thing I tried (from your suggestions) was
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y. Given its dependencies (and, well, the
build system I used) I ended up with:

    $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local3.fc16.i686 |
    grep -v "^#"
    CONFIG_TREE_PREEMPT_RCU=y
    CONFIG_PREEMPT_RCU=y
    CONFIG_RCU_FANOUT=32
    CONFIG_PREEMPT_NOTIFIERS=y
    CONFIG_PREEMPT=y
    CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
    CONFIG_DEBUG_PREEMPT=y
    CONFIG_PROVE_RCU=y
    CONFIG_SPARSE_RCU_POINTER=y

It looks like I am unable to trigger the issue we're talking about here
when using that config.

1) For reference, the config of a kernel that does trigger it had:

    $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local2.fc16.i686 |
    grep -v "^#"
    CONFIG_TREE_RCU=y
    CONFIG_RCU_FANOUT=32
    CONFIG_RCU_FAST_NO_HZ=y
    CONFIG_PREEMPT_NOTIFIERS=y
    CONFIG_PREEMPT_VOLUNTARY=y
    CONFIG_PROVE_RCU=y
    CONFIG_SPARSE_RCU_POINTER=y

> > Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
> > we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
> > the issue?
> 
> Please!

2) It appears I can't reproduce with those options enabled (see above).

> Polling is fine.  Please see attached for a script to poll at 15-second
> intervals.  Please also feel free to adjust, just tell me what you
> adjusted.

And should I now try to run that script on a config that triggers this
issue (such as the config under 1) above)?


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Tue, May 24, 2011 at 12:20:40AM +0200, Paul Bolle wrote:
> On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
> > Running under CONFIG_PREEMPT=y (along with CONFIG_TREE_PREEMPT_RCU=y)
> > could be very helpful in and of itself.  CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> > can also be helpful.  In post-2.6.39 mainline, it should be possible
> > to set CONFIG_DEBUG_OBJECTS_RCU_HEAD=y without CONFIG_PREEMPT=y, but
> > again, CONFIG_PREEMPT=y can help find problems.
> 
> 0) The first thing I tried (from your suggestions) was
> CONFIG_DEBUG_OBJECTS_RCU_HEAD=y. Given its dependencies (and, well, the
> build system I used) I ended up with:
> 
>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local3.fc16.i686 |
>     grep -v "^#"
>     CONFIG_TREE_PREEMPT_RCU=y
>     CONFIG_PREEMPT_RCU=y
>     CONFIG_RCU_FANOUT=32
>     CONFIG_PREEMPT_NOTIFIERS=y
>     CONFIG_PREEMPT=y
>     CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
>     CONFIG_DEBUG_PREEMPT=y
>     CONFIG_PROVE_RCU=y
>     CONFIG_SPARSE_RCU_POINTER=y
> 
> It looks like I am unable to trigger the issue we're talking about here
> when using that config.

Interesting.  One way for this to happen is to block inside an RCU
read-side critical section.  I need to think about appropriate
diagnostics for this.

> 1) For reference, the config of a kernel that does trigger it had:
> 
>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local2.fc16.i686 |
>     grep -v "^#"
>     CONFIG_TREE_RCU=y
>     CONFIG_RCU_FANOUT=32
>     CONFIG_RCU_FAST_NO_HZ=y
>     CONFIG_PREEMPT_NOTIFIERS=y
>     CONFIG_PREEMPT_VOLUNTARY=y
>     CONFIG_PROVE_RCU=y
>     CONFIG_SPARSE_RCU_POINTER=y
> 
> > > Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
> > > we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
> > > the issue?
> > 
> > Please!
> 
> 2) It appears I can't reproduce with those options enabled (see above).
> 
> > Polling is fine.  Please see attached for a script to poll at 15-second
> > intervals.  Please also feel free to adjust, just tell me what you
> > adjusted.
> 
> And should I now try to run that script on a config that triggers this
> issue (such as the config under 1) above)?

It might help while I am working out more targetted diagnostics.

							Thanx, Paul

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-05-24 00:20, Paul Bolle wrote:
> On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
>> Running under CONFIG_PREEMPT=y (along with CONFIG_TREE_PREEMPT_RCU=y)
>> could be very helpful in and of itself.  CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
>> can also be helpful.  In post-2.6.39 mainline, it should be possible
>> to set CONFIG_DEBUG_OBJECTS_RCU_HEAD=y without CONFIG_PREEMPT=y, but
>> again, CONFIG_PREEMPT=y can help find problems.
> 
> 0) The first thing I tried (from your suggestions) was
> CONFIG_DEBUG_OBJECTS_RCU_HEAD=y. Given its dependencies (and, well, the
> build system I used) I ended up with:
> 
>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local3.fc16.i686 |
>     grep -v "^#"
>     CONFIG_TREE_PREEMPT_RCU=y
>     CONFIG_PREEMPT_RCU=y
>     CONFIG_RCU_FANOUT=32
>     CONFIG_PREEMPT_NOTIFIERS=y
>     CONFIG_PREEMPT=y
>     CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
>     CONFIG_DEBUG_PREEMPT=y
>     CONFIG_PROVE_RCU=y
>     CONFIG_SPARSE_RCU_POINTER=y
> 
> It looks like I am unable to trigger the issue we're talking about here
> when using that config.
> 
> 1) For reference, the config of a kernel that does trigger it had:
> 
>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local2.fc16.i686 |
>     grep -v "^#"
>     CONFIG_TREE_RCU=y
>     CONFIG_RCU_FANOUT=32
>     CONFIG_RCU_FAST_NO_HZ=y
>     CONFIG_PREEMPT_NOTIFIERS=y
>     CONFIG_PREEMPT_VOLUNTARY=y
>     CONFIG_PROVE_RCU=y
>     CONFIG_SPARSE_RCU_POINTER=y
> 
>>> Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
>>> we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
>>> the issue?
>>
>> Please!
> 
> 2) It appears I can't reproduce with those options enabled (see above).
> 
>> Polling is fine.  Please see attached for a script to poll at 15-second
>> intervals.  Please also feel free to adjust, just tell me what you
>> adjusted.
> 
> And should I now try to run that script on a config that triggers this
> issue (such as the config under 1) above)?

Paul, can we see a dmesg from your running system? Perhaps there's some
dependency on a particular driver or device that makes this easier to
reproduce.

-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

[-- Attachment #1: Type: text/plain, Size: 2449 bytes --]

On Tue, May 24, 2011 at 11:41:10AM +0200, Jens Axboe wrote:
> On 2011-05-24 00:20, Paul Bolle wrote:
> > On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
> >> Running under CONFIG_PREEMPT=y (along with CONFIG_TREE_PREEMPT_RCU=y)
> >> could be very helpful in and of itself.  CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> >> can also be helpful.  In post-2.6.39 mainline, it should be possible
> >> to set CONFIG_DEBUG_OBJECTS_RCU_HEAD=y without CONFIG_PREEMPT=y, but
> >> again, CONFIG_PREEMPT=y can help find problems.
> > 
> > 0) The first thing I tried (from your suggestions) was
> > CONFIG_DEBUG_OBJECTS_RCU_HEAD=y. Given its dependencies (and, well, the
> > build system I used) I ended up with:
> > 
> >     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local3.fc16.i686 |
> >     grep -v "^#"
> >     CONFIG_TREE_PREEMPT_RCU=y
> >     CONFIG_PREEMPT_RCU=y
> >     CONFIG_RCU_FANOUT=32
> >     CONFIG_PREEMPT_NOTIFIERS=y
> >     CONFIG_PREEMPT=y
> >     CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> >     CONFIG_DEBUG_PREEMPT=y
> >     CONFIG_PROVE_RCU=y
> >     CONFIG_SPARSE_RCU_POINTER=y
> > 
> > It looks like I am unable to trigger the issue we're talking about here
> > when using that config.
> > 
> > 1) For reference, the config of a kernel that does trigger it had:
> > 
> >     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local2.fc16.i686 |
> >     grep -v "^#"
> >     CONFIG_TREE_RCU=y
> >     CONFIG_RCU_FANOUT=32
> >     CONFIG_RCU_FAST_NO_HZ=y
> >     CONFIG_PREEMPT_NOTIFIERS=y
> >     CONFIG_PREEMPT_VOLUNTARY=y
> >     CONFIG_PROVE_RCU=y
> >     CONFIG_SPARSE_RCU_POINTER=y
> > 
> >>> Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
> >>> we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
> >>> the issue?
> >>
> >> Please!
> > 
> > 2) It appears I can't reproduce with those options enabled (see above).
> > 
> >> Polling is fine.  Please see attached for a script to poll at 15-second
> >> intervals.  Please also feel free to adjust, just tell me what you
> >> adjusted.
> > 
> > And should I now try to run that script on a config that triggers this
> > issue (such as the config under 1) above)?
> 
> Paul, can we see a dmesg from your running system? Perhaps there's some
> dependency on a particular driver or device that makes this easier to
> reproduce.

Here you go, please see attached.

I should have some additional diagnostics later today Pacific time.

							Thanx, Paul

[-- Attachment #2: console.log --]
[-- Type: text/plain, Size: 100661 bytes --]

Connected to vogons console [channel connected] (~$quit to exit)
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Shutting down Avahi daemon: [  OK  ]
Running guests on default URI: no running guests.
Stopping libvirtd daemon: [  OK  ]
Stopping atd: [  OK  ]
Stopping abrt daemon: [  OK  ]
Stopping sshd: [  OK  ]
Shutting down postfix: [  OK  ]
Stopping crond: [  OK  ]
Stopping automount: [  OK  ]
Stopping HAL daemon: [  OK  ]
Stopping iprdump: [  OK  ]
Stopping iprinit: [  OK  ]Stopping iprupdate: [  OK  ]Stopping NetworkManager daemon: ehea: eth1: Logical port down
ehea: eth1: Physical port up
ehea: External switch port is backup port
[  OK  ]
Killing mdmonitor: [  OK  ]
Stopping system message bus: [  OK  ]
Stopping rpcbind: [  OK  ]
Stopping auditd: type=1305 audit(1306123304.459:167): audit_pid=0 old=983 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[  OK  ]
Shutting down interface eth0:  ehea: eth0: Logical port down
ehea: eth0: Physical port up
ehea: External switch port is backup port
[  OK  ]
Shutting down loopback interface:  [  OK  ]
IPv6 over IPv4 tunneling driver
sit0: Disabled Privacy Extensions
ip6tables: Flushing firewall rules: [  OK  ]
ip6tables: Setting chains to policy ACCEPT: filter [  OK  ]
ip6tables: Unloading modules: [  OK  ]
iptables: Flushing firewall rules: [  OK  ]
iptables: Setting chains to policy ACCEPT: mangle nat filter [  OK  ]
iptables: Unloading modules: [  OK  ]
Stopping cgconfig service: [  OK  ]
Sending all processes the TERM signal... [  OK  ]
Sending all processes the KILL signal... [  OK  ]
Saving random seed:  [  OK  ]
Syncing hardware clock to system time type=1111 audit(1306123314.500:168): user pid=15173 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:hwclock_t:s0 msg='changing system time: exe="/sbin/hwclock" hostname=? addr=? terminal=? res=success'
[  OK  ]
Turning off quotas:  [  OK  ]
Unmounting pipe file systems:  [  OK  ]
Unmounting file systems:  [  OK  ]
init: Re-executing /sbin/init
Please stand by while rebooting the system...
type=1128 audit(1306123315.030:169): user pid=14476 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:initrc_t:s0 msg='init: exe="/sbin/reboot" hostname=? addr=? terminal=console res=success'
md: stopping all md devices.
ehea: Reboot: freeing all eHEA resources
Restarting system.
























IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 

          1 = SMS Menu                          5 = Default Boot List
          8 = Open Firmware Prompt              6 = Stored Boot List


     Memory      Keyboard     Network     SCSI     Speaker 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM                             IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM     STARTING SOFTWARE       IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM        PLEASE WAIT...       IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM                             IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
-\|/-\|/-\|
Elapsed time since release of system processors: 2247 mins 32 secs

Config file read, 2048 bytes
Welcome to Red Hat Enterprise Linux!
Hit <TAB> for boot options
Welcome to yaboot version 1.3.16 (Red Hat 1.3.16-0.2.el6)
Enter "help" to get some basic usage information
boot: default
Please wait, loading kernel...
   Elf64 kernel loaded...
Loading ramdisk...
ramdisk loaded at 03500000, size: 13620 Kbytes
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 2.6.32-71.el6.ppc64 (mockbuild@js20-bc2-12.build.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Wed Sep 1 02:56:55 EDT 2010
Max number of cores passed to firmware: 0x0000000000000200
Calling ibm,client-architecture-support... done
command line: root=/dev/sda5 console=hvc0 multipath=off  
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000004250000
  alloc_top    : 0000000010000000
  alloc_top_hi : 0000000010000000
  rmo_top      : 0000000010000000
  ram_top      : 0000000010000000
found display   : /pci@800000020000202/display@1, opening... done
instantiating rtas at 0x000000000f3e0000... done
boot cpu hw idx 0000000000000000
starting cpu hw idx 0000000000000002... done
starting cpu hw idx 0000000000000004... done
starting cpu hw idx 0000000000000006... done
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000004260000 -> 0x00000000042617da
Device tree struct  0x0000000004270000 -> 0x0000000004290000
Calling quiesce...
returning from prom_init
Using pSeries machine description
Using 1TB segments
Found initrd at 0xc000000003500000:0xc00000000424d000
bootconsole [udbg0] enabled
Partition configured for 8 cpus.
CPU maps initialized for 2 threads per core
Starting Linux PPC64 #1 SMP Wed Sep 1 02:56:55 EDT 2010
-----------------------------------------------------
ppc64_pft_size                = 0x1c
physicalMemorySize            = 0x3d4000000
htab_hash_mask                = 0x1fffff
-----------------------------------------------------
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-71.el6.ppc64 (mockbuild@js20-bc2-12.build.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Wed Sep 1 02:56:55 EDT 2010
[boot]0012 Setup Arch
PCI host bridge /pci@800000020000200  ranges:
  IO 0x00003dffe01f0000..0x00003dffe01fffff -> 0x00000000000f0000
 MEM 0x00003c0000000000..0x00003c007fffffff -> 0x0000000080000000 
PCI host bridge /pci@800000020000201  ranges:
  IO 0x00003dffe02f0000..0x00003dffe02fffff -> 0x00000000000f0000
 MEM 0x00003c0080000000..0x00003c00ffffffff -> 0x0000000080000000 
PCI host bridge /pci@800000020000202  ranges:
  IO 0x00003dffe03f0000..0x00003dffe03fffff -> 0x00000000000f0000
 MEM 0x00003c0100000000..0x00003c017fffffff -> 0x0000000080000000 
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 15360 bytes
Zone PFN ranges:
  DMA      0x00000000 -> 0x0003d400
  Normal   0x0003d400 -> 0x0003d400
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000000 -> 0x0001e000
    1: 0x0001e000 -> 0x0003d400
[boot]0015 Setup Done
PERCPU: Embedded 2 pages/cpu @c000000001400000 s93240 r0 d37832 u131072
pcpu-alloc: s93240 r0 d37832 u131072 alloc=1*1048576
pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
Built 2 zonelists in Node order, mobility grouping on.  Total pages: 250665
Policy zone: DMA
Kernel command line: root=/dev/sda5 console=hvc0 multipath=off  
PID hash table entries: 4096 (order: -1, 32768 bytes)
freeing bootmem node 0
freeing bootmem node 1
Memory: 15945856k/16056320k available (12928k kernel code, 110464k reserved, 2752k data, 4414k bss, 4800k init)
Hierarchical RCU implementation.
NR_IRQS:512
[boot]0020 XICS Init
[boot]0021 XICS Done
clocksource: timebase mult[7d0000] shift[22] registered
Console: colour dummy device 80x25
console [hvc0] enabled, bootconsole disabled
console [hvc0] enabled, bootconsole disabled
allocated 10035200 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 2097152 (order: 8, 16777216 bytes)
Inode-cache hash table entries: 1048576 (order: 7, 8388608 bytes)
Mount-cache hash table entries: 4096
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Processor 1 found.
Processor 2 found.
Processor 3 found.
Processor 4 found.
Processor 5 found.
Processor 6 found.
Processor 7 found.
Brought up 8 CPUs
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
IBM eBus Device Driver
POWER6 performance monitor hardware support registered
PCI: Probing PCI hardware
IOMMU table initialized, virtual merging enabled
pci 0001:00:01.0: PME# supported from D0 D1 D2 D3hot
pci 0001:00:01.0: PME# disabled
pci 0001:00:01.1: PME# supported from D0 D1 D2 D3hot
pci 0001:00:01.1: PME# disabled
pci 0001:00:01.2: PME# supported from D0 D1 D2 D3hot
pci 0001:00:01.2: PME# disabled
bio: create slab <bio-0> at 0
vgaarb: device added: PCI:0002:00:01.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource timebase
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 6, 4194304 bytes)
TCP established hash table entries: 524288 (order: 7, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 13620k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1306123375.390:1): initialized
HugeTLB registered 16 MB page size, pre-allocated 0 pages
HugeTLB registered 16 GB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
msgmni has been set to 31168
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key FCF3FD17E3D277B8
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
rpaphp: RPA HOT Plug PCI Controller Driver version: 0.1
pci-stub: invalid id string ""
Using unsupported 640x480 display at 3c0178000000, depth=8, pitch=640
Console: switching to colour frame buffer device 80x30
fb0: Open Firmware frame buffer device on /pci@800000020000202/display@1
Linux agpgart interface v0.103
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0001:00:01.2: enabling device (0140 -> 0142)
ehci_hcd 0001:00:01.2: EHCI Host Controller
ehci_hcd 0001:00:01.2: new USB bus registered, assigned bus number 1
ehci_hcd 0001:00:01.2: irq 307, io mem 0x3c00ffffd000
ehci_hcd 0001:00:01.2: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-71.el6.ppc64 ehci_hcd
usb usb1: SerialNumber: 0001:00:01.2
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 5 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci_hcd 0001:00:01.0: OHCI Host Controller
ohci_hcd 0001:00:01.0: new USB bus registered, assigned bus number 2
ohci_hcd 0001:00:01.0: irq 305, io mem 0x3c00ffffe000
usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: OHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.32-71.el6.ppc64 ohci_hcd
usb usb2: SerialNumber: 0001:00:01.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
ohci_hcd 0001:00:01.1: OHCI Host Controller
ohci_hcd 0001:00:01.1: new USB bus registered, assigned bus number 3
ohci_hcd 0001:00:01.1: irq 306, io mem 0x3c00fffff000
usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: OHCI Host Controller
usb usb3: Manufacturer: Linux 2.6.32-71.el6.ppc64 ohci_hcd
usb usb3: SerialNumber: 0001:00:01.1
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
uhci_hcd: USB Universal Host Controller Interface driver
mice: PS/2 mouse device common for all mice
rtc-generic rtc-generic: rtc core: registered rtc-generic as rtc0
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc-generic rtc-generic: setting system clock to 2011-05-23 04:02:56 UTC (1306123376)
Initalizing network drop monitor service
Freeing unused kernel memory: 4800k freed
dracut: dracut-004-32.el6
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.17.0-ioctl (2010-03-05) initialised: dm-devel@redhat.com
udev: starting version 147
[drm] Initialized drm 1.1.0 20060810
[drm] radeon defaulting to kernel modesetting.
[drm] radeon kernel modesetting enabled.
[drm] initializing kernel modesetting (RV100 0x1002:0x515E).
[drm] register mmio base: 0x77FD0000
[drm] register mmio size: 65536
radeon 0002:00:01.0: Invalid ROM contents
radeon 0002:00:01.0: Invalid ROM contents
[drm:radeon_get_bios] *ERROR* Unable to locate a BIOS ROM
[drm] Using generic clock info
radeon 0002:00:01.0: VRAM: 64M 0x3C0178000000 - 0x3C017BFFFFFF (16M used)
radeon 0002:00:01.0: GTT: 512M 0x3C017C000000 - 0x3C019BFFFFFF
[drm] radeon: irq initialized.
[drm] Detected VRAM RAM=64M, BAR=128M
[drm] RAM width 16bits DDR
[TTM] Zone  kernel: Available graphics memory: 7982144 kiB.
[TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
[drm] radeon: 16M of VRAM memory ready
[drm] radeon: 512M of GTT memory ready.
[drm] GART: num cpu pages 8192, num gpu pages 131072
[drm] Loading R100 Microcode
platform radeon_cp.0: firmware: requesting radeon/R100_cp.bin
[drm] radeon: ring at 0x00003C017C000000
[drm] ring test succeeded in 1 usecs
[drm] radeon: ib pool ready.
[drm] ib test succeeded in 0 usecs
[drm] Connector Table: 10 (rn50-power)
[drm] No TV DAC info found in BIOS
[drm] Radeon Display Connectors
[drm] Connector 0:
[drm]   VGA
[drm]   DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
[drm]   Encoders:
[drm]     CRT1: INTERNAL_DAC1
[drm] Connector 1:
[drm]   VGA
[drm]   DDC: 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c
[drm]   Encoders:
[drm]     CRT2: INTERNAL_DAC2
[drm] fb mappable at 0x3C0178040000
[drm] vram apper at 0x3C0178000000
[drm] size 786432
[drm] fb depth is 8
[drm]    pitch is 1024
fb: conflicting fb hw usage radeondrmfb vs OFfb display - removing generic driver
Console: switching to colour dummy device 80x25
Trying to free nonexistent resource <00003c0178000000-00003c017804afff>
Console: switching to colour frame buffer device 128x48
fb0: radeondrmfb frame buffer device
drm: registered panic notifier
Slow work thread pool: Starting up
Slow work thread pool: Ready
[drm] Initialized radeon 2.1.0 20080528 for 0002:00:01.0 on minor 0
dracut: Starting plymouth daemon
could not read byte from child: Success
IBM eHEA ethernet device driver (Release EHEA_0103-02)
ehea: eth0: Jumbo frames are enabled
ehea: eth0 -> logical port id #1
ehea: eth1: Jumbo frames are enabled
ehea: eth1 -> logical port id #2
ipr: IBM Power RAID SCSI Device Driver version: 2.4.3 (June 10, 2009)
ipr 0000:00:01.0: Found IOA with IRQ: 289
ipr 0000:00:01.0: Starting IOA initialization sequence.
ipr 0000:00:01.0: Adapter firmware version: 0220004A
ipr 0000:00:01.0: IOA initialized.
scsi0 : IBM 572C Storage Adapter
scsi 0:3:0:0: Direct-Access     IBM-ESXS ST9146802SS      B526 PQ: 0 ANSI: 5
scsi 0:8:0:0: Enclosure         IBM      VSBPD1BB   SAS     01 PQ: 0 ANSI: 2
scsi: unknown device type 31
scsi 0:255:255:255: No Device         IBM      572C001SISIOA    0150 PQ: 0 ANSI: 0
sd 0:3:0:0: [sda] 286748000 512-byte logical blocks: (146 GB/136 GiB)
sd 0:3:0:0: [sda] Write Protect is off
sd 0:3:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
 sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 >
sd 0:3:0:0: [sda] Attached SCSI disk
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
dracut: Remounting /dev/sda5 with -o errors=remount-ro,ro
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
dracut: Mounted root filesystem /dev/sda5
dracut: Loading SELinux policy
SELinux:  Disabled at runtime.
type=1404 audit(1306123379.672:2): selinux=0 auid=4294967295 ses=4294967295
dracut: /sbin/load_policy: Can't load policy: No such file or directory
dracut: Switching root
readahead-collector: starting
                Welcome to Red Hat Enterprise Linux Server
Starting udev: udev: starting version 147
sd 0:3:0:0: Attached scsi generic sg0 type 0
scsi 0:8:0:0: Attached scsi generic sg1 type 13
scsi 0:255:255:255: Attached scsi generic sg2 type 31
ses 0:8:0:0: Attached Enclosure device
[  OK  ]
Setting hostname vogons.ltc.austin.ibm.com:  [  OK  ]
Setting up Logical Volume Management:   No volume groups found
[  OK  ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda5 
deploy_root: clean, 52117/1281120 files, 414664/5120000 blocks
[  OK  ]
Remounting root filesystem in read-write mode:  EXT3 FS on sda5, internal journal
[  OK  ]
Mounting local filesystems:  [  OK  ]
Enabling local filesystem quotas:  [  OK  ]
Enabling /etc/fstab swaps:  Adding 4095872k swap on /dev/sda7.  Priority:-1 extents:1 across:4095872k 
[  OK  ]
readahead-disable-service: delaying service auditd
Entering non-interactive startup
Calling the system activity data collector (sadc): 
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  ehea: eth0: Physical port up
ehea: External switch port is backup port
[  OK  ]
Bringing up interface eth1:  ehea: eth1: Physical port up
ehea: External switch port is backup port
[  OK  ]
Starting system logger: [  OK  ]
Starting irqbalance: [  OK  ]
Starting mdmonitor: [  OK  ]
Starting kdump:[FAILED]
Starting system message bus: [  OK  ]
Mounting other filesystems:  [  OK  ]
Starting HAL daemon: [  OK  ]
Starting iprinit: [  OK  ]
Starting iprupdate: [  OK  ]
Retrigger failed udev events[  OK  ]
Adding udev persistent rules[  OK  ]
Starting iprdump: [  OK  ]
Starting sshd: [  OK  ]
Starting postfix: [  OK  ]
Starting abrt daemon: [  OK  ]
Starting crond: [  OK  ][  OK  ]
Starting atd: [  OK  ][  OK  ]

Red Hat Enterprise Linux Server release 6.0 (Santiago)
Kernel 2.6.32-71.el6.ppc64 on an ppc64

vogons.ltc.austin.ibm.com login: readahead-collector: starting delayed service auditd
readahead-collector: sorting
readahead-collector: finished
(bot:conmon-payload) connected
-- 0:conmux-control -- time-stamp -- May/22/11 21:05:39 --
-- 0:conmux-control -- time-stamp -- May/22/11 21:11:02 --
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
could not read byte from child: Success
initctl: Event failed
Stopping atd: [  OK  ]
Stopping abrt daemon: [  OK  ]
Stopping sshd: [  OK  ]
Shutting down postfix: [  OK  ]
Stopping crond: [  OK  ]
Stopping HAL daemon: [  OK  ]
Stopping iprdump: [  OK  ]
Stopping iprinit: [  OK  ]Stopping iprupdate: [  OK  ]Killing mdmonitor: [  OK  ]
Stopping system message bus: [  OK  ]
Stopping auditd: type=1305 audit(1306123950.060:15436): audit_pid=0 old=1342 auid=4294967295 ses=4294967295 res=1
[  OK  ]
Shutting down interface eth0:  ehea: eth0: Logical port down
ehea: eth0: Physical port up
ehea: External switch port is backup port
[  OK  ]
Shutting down interface eth1:  ehea: eth1: Logical port down
ehea: eth1: Physical port up
ehea: External switch port is backup port
[  OK  ]
Shutting down loopback interface:  [  OK  ]
IPv6 over IPv4 tunneling driver
sit0: Disabled Privacy Extensions
Sending all processes the TERM signal... [  OK  ]
Sending all processes the KILL signal... [  OK  ]
Saving random seed:  [  OK  ]
Syncing hardware clock to system time type=1111 audit(1306123954.500:15437): user pid=11921 uid=0 auid=4294967295 ses=4294967295 msg='changing system time: exe="/sbin/hwclock" hostname=? addr=? terminal=console res=success'
[  OK  ]
Turning off swap:  [  OK  ]
Turning off quotas:  [  OK  ]
init: Re-executing /sbin/init
Please stand by while rebooting the system...
type=1128 audit(1306123955.068:15438): user pid=11582 uid=0 auid=4294967295 ses=4294967295 msg='init: exe="/sbin/reboot" hostname=? addr=? terminal=console res=success'
md: stopping all md devices.
ehea: Reboot: freeing all eHEA resources
Restarting system.
























IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 

          1 = SMS Menu                          5 = Default Boot List
          8 = Open Firmware Prompt              6 = Stored Boot List


     Memory      Keyboard     Network     SCSI     Speaker 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM                             IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM     STARTING SOFTWARE       IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM        PLEASE WAIT...       IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM                             IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
-\|/-\|/-\|
Elapsed time since release of system processors: 2258 mins 11 secs

Config file read, 2048 bytes
Welcome to Red Hat Enterprise Linux!
Hit <TAB> for boot options
Welcome to yaboot version 1.3.16 (Red Hat 1.3.16-0.2.el6)
Enter "help" to get some basic usage information
boot: autobench
Please wait, loading kernel...
   Elf64 kernel loaded...
Loading ramdisk...
ramdisk loaded at 03500000, size: 6629 Kbytes
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 2.6.39-rc6-autokern1 (root@vogons.ltc.austin.ibm.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon May 23 00:09:33 EDT 2011
Max number of cores passed to firmware: 4 (NR_CPUS = 8)
Calling ibm,client-architecture-support... done
command line: console=hvc0 autobench_args: root=/dev/sda5 ABAT:1306123860 
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000003b80000
  alloc_top    : 0000000010000000
  alloc_top_hi : 0000000010000000
  rmo_top      : 0000000010000000
  ram_top      : 0000000010000000
found display   : /pci@800000020000202/display@1, opening... done
instantiating rtas at 0x000000000f3e0000... done
boot cpu hw idx 0
starting cpu hw idx 2... done
starting cpu hw idx 4... done
starting cpu hw idx 6... done
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000003b90000 -> 0x0000000003b917da
Device tree struct  0x0000000003ba0000 -> 0x0000000003bc0000
Calling quiesce...
returning from prom_init
[    0.000000] Using pSeries machine description
[    0.000000] Using 1TB segments
[    0.000000] Found initrd at 0xc000000003500000:0xc000000003b79400
[    0.000000] bootconsole [udbg0] enabled
[    0.000000] Partition configured for 8 cpus.
[    0.000000] CPU maps initialized for 2 threads per core
[    0.000000] Starting Linux PPC64 #1 SMP Mon May 23 00:09:33 EDT 2011
[    0.000000] -----------------------------------------------------
[    0.000000] ppc64_pft_size                = 0x1c
[    0.000000] physicalMemorySize            = 0x3d4000000
[    0.000000] htab_hash_mask                = 0x1fffff
[    0.000000] -----------------------------------------------------
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.39-rc6-autokern1 (root@vogons.ltc.austin.ibm.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon May 23 00:09:33 EDT 2011
[    0.000000] [boot]0012 Setup Arch
[    0.000000] PCI host bridge /pci@800000020000200  ranges:
[    0.000000]   IO 0x00003dffe01f0000..0x00003dffe01fffff -> 0x00000000000f0000
[    0.000000]  MEM 0x00003c0000000000..0x00003c007fffffff -> 0x0000000080000000 
[    0.000000] PCI host bridge /pci@800000020000201  ranges:
[    0.000000]   IO 0x00003dffe02f0000..0x00003dffe02fffff -> 0x00000000000f0000
[    0.000000]  MEM 0x00003c0080000000..0x00003c00ffffffff -> 0x0000000080000000 
[    0.000000] PCI host bridge /pci@800000020000202  ranges:
[    0.000000]   IO 0x00003dffe03f0000..0x00003dffe03fffff -> 0x00000000000f0000
[    0.000000]  MEM 0x00003c0100000000..0x00003c017fffffff -> 0x0000000080000000 
[    0.000000] EEH: PCI Enhanced I/O Error Handling Enabled
[    0.000000] PPC64 nvram contains 15360 bytes
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000000 -> 0x0003d400
[    0.000000]   Normal   empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[1] active PFN ranges
[    0.000000]     0: 0x00000000 -> 0x0003d400
[    0.000000] [boot]0015 Setup Done
[    0.000000] PERCPU: Embedded 1 pages/cpu @c000000001a00000 s30336 r0 d35200 u131072
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 250665
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line: console=hvc0 autobench_args: root=/dev/sda5 ABAT:1306123860 
[    0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[    0.000000] freeing bootmem node 0
[    0.000000] Memory: 15947776k/16056320k available (20928k kernel code, 108544k reserved, 1792k data, 2626k bss, 6720k init)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU debugfs-based tracing is enabled.
[    0.000000]  CONFIG_RCU_FANOUT set to non-default value of 8
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] NR_IRQS:512
[    0.000000] [boot]0020 XICS Init
[    0.000000] [boot]0021 XICS Done
[    0.000000] clocksource: timebase mult[7d0000] shift[22] registered
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [hvc0] enabled, bootconsole disabled
[    0.000000] console [hvc0] enabled, bootconsole disabled
[    0.000000] allocated 8028160 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.003261] pid_max: default: 32768 minimum: 301
[    0.003388] Security Framework initialized
[    0.003397] SELinux:  Initializing.
[    0.003689] Dentry cache hash table entries: 2097152 (order: 8, 16777216 bytes)
[    0.008685] Inode-cache hash table entries: 1048576 (order: 7, 8388608 bytes)
[    0.011228] Mount-cache hash table entries: 4096
[    0.013381] Initializing cgroup subsys ns
[    0.013388] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.013395] Initializing cgroup subsys cpuacct
[    0.013411] Initializing cgroup subsys memory
[    0.013537] Initializing cgroup subsys devices
[    0.013542] Initializing cgroup subsys freezer
[    0.013546] Initializing cgroup subsys net_cls
[    0.013551] Initializing cgroup subsys blkio
[    0.013684] POWER6 performance monitor hardware support registered
[    0.020123] Brought up 8 CPUs
[    0.021265] devtmpfs: initialized
[    0.032354] print_constraints: dummy: 
[    0.032500] NET: Registered protocol family 16
[    0.032514] IBM eBus Device Driver
[    0.043433] PCI: Probing PCI hardware
[    0.074166] IOMMU table initialized, virtual merging enabled
[    0.157679] bio: create slab <bio-0> at 0
[    0.158128] vgaarb: device added: PCI:0002:00:01.0,decodes=io+mem,owns=io+mem,locks=none
[    0.158134] vgaarb: loaded
[    0.158404] SCSI subsystem initialized
[    0.158550] usbcore: registered new interface driver usbfs
[    0.158574] usbcore: registered new interface driver hub
[    0.158637] usbcore: registered new device driver usb
[    0.158758] Advanced Linux Sound Architecture Driver Version 1.0.24.
[    0.158892] NetLabel: Initializing
[    0.158896] NetLabel:  domain hash size = 128
[    0.158899] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.158916] NetLabel:  unlabeled traffic allowed by default
[    0.158939] Switching to clocksource timebase
[    0.158999] Switched to NOHz mode on CPU #0
[    0.159244] Switched to NOHz mode on CPU #6
[    0.160122] Switched to NOHz mode on CPU #7
[    0.164467] NET: Registered protocol family 2
[    0.164620] IP route cache hash table entries: 524288 (order: 6, 4194304 bytes)
[    0.166027] TCP established hash table entries: 524288 (order: 7, 8388608 bytes)
[    0.168911] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
[    0.169027] Switched to NOHz mode on CPU #1
[    0.169280] TCP: Hash tables configured (established 524288 bind 65536)
[    0.169286] TCP reno registered
[    0.169297] UDP hash table entries: 8192 (order: 2, 262144 bytes)
[    0.169388] UDP-Lite hash table entries: 8192 (order: 2, 262144 bytes)
[    0.169737] NET: Registered protocol family 1
[    0.170515] Trying to unpack rootfs image as initramfs...
[    0.185745] Switched to NOHz mode on CPU #2
[    0.186632] Switched to NOHz mode on CPU #3
[    0.187525] Switched to NOHz mode on CPU #4
[    0.188386] Switched to NOHz mode on CPU #5
[    0.329630] Freeing initrd memory: 6629k freed
[    0.330778] vio 30000000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[    0.330805] vio 4000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[    0.330838] vio 4001: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[    0.330862] vio 4002: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[    0.330887] vio 4004: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[    0.330911] vio 4003: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[    0.331081] Hypercall H_BEST_ENERGY not supported
[    0.331375] audit: initializing netlink socket (disabled)
[    0.331393] type=2000 audit(1306109616.240:1): initialized
[    0.463067] HugeTLB registered 16 MB page size, pre-allocated 0 pages
[    0.463076] HugeTLB registered 16 GB page size, pre-allocated 0 pages
[    0.463639] VFS: Disk quotas dquot_6.5.2
[    0.463732] Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    0.464062] msgmni has been set to 31160
[    0.464551] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.464559] io scheduler noop registered
[    0.464564] io scheduler deadline registered
[    0.464615] io scheduler cfq registered (default)
[    0.464716] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.464736] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.464742] rpaphp: RPA HOT Plug PCI Controller Driver version: 0.1
[    0.665251] radeonfb 0002:00:01.0: Invalid ROM contents
[    0.679003] radeonfb: No ATY,RefCLK property !
[    0.769002] xtal calculation failed: 26880
[    0.769009] radeonfb: Used default PLL infos
[    0.769015] radeonfb: Reference=27.00 MHz (RefDiv=12) Memory=166.00 Mhz, System=166.00 MHz
[    0.769022] radeonfb: PLL min 12000 max 35000
[    0.869005] radeonfb: Monitor 1 type CRT found
[    0.869013] radeonfb: Monitor 2 type no found
[    1.464714] Console: switching to colour frame buffer device 80x30
[    1.500072] radeonfb (0002:00:01.0): ATI Radeon 515e "Q^"
[    1.500528] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    1.501512] Linux agpgart interface v0.103
[    1.501533] [drm] Initialized drm 1.1.0 20060810
[    1.501564] [drm] radeon defaulting to kernel modesetting.
[    1.501569] [drm] radeon kernel modesetting enabled.
[    1.503327] brd: module loaded
[    1.504180] loop: module loaded
[    1.504217] Loading iSCSI transport class v2.0-870.
[    1.504306] ipr: IBM Power RAID SCSI Device Driver version: 2.5.1 (August 10, 2010)
[    1.504486] ipr 0000:00:01.0: Found IOA with IRQ: 289
[    1.504629] ipr 0000:00:01.0: Using 32-bit DMA via iommu
[    1.508131] ipr 0000:00:01.0: Starting IOA initialization sequence.
[    1.508863] ipr 0000:00:01.0: Adapter firmware version: 0220004A
[    1.512230] ipr 0000:00:01.0: IOA initialized.
[    1.512246] scsi0 : IBM 572C Storage Adapter
[    1.551067] scsi 0:3:0:0: Direct-Access     IBM-ESXS ST9146802SS      B526 PQ: 0 ANSI: 5
[    1.625369] scsi 0:8:0:0: Enclosure         IBM      VSBPD1BB   SAS     01 PQ: 0 ANSI: 2
[    2.365360] scsi: unknown device type 31
[    2.365370] scsi 0:255:255:255: No Device         IBM      572C001SISIOA    0150 PQ: 0 ANSI: 0
[    2.365538] libcxgbi:libcxgbi_init_module: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.
[    2.365546] libcxgbi:ddp_setup_host_page_size: system PAGE 65536, ddp idx 3.
[    2.365552] Chelsio T3 iSCSI Driver cxgb3i v2.0.0 (Jun. 2010)
[    2.365584] iscsi: registered transport (cxgb3i)
[    2.365590] Broadcom NetXtreme II iSCSI Driver bnx2i v2.6.2.3 (Dec 31, 2010)
[    2.365613] iscsi: registered transport (bnx2i)
[    2.366018] sd 0:3:0:0: Attached scsi generic sg0 type 0
[    2.366152] scsi 0:8:0:0: Attached scsi generic sg1 type 13
[    2.366286] scsi 0:255:255:255: Attached scsi generic sg2 type 31
[    2.368840] sd 0:3:0:0: [sda] 286748000 512-byte logical blocks: (146 GB/136 GiB)
[    2.371772] ses 0:8:0:0: Attached Enclosure device
[    2.371904] sd 0:3:0:0: [sda] Write Protect is off
[    2.372773] Fixed MDIO Bus: probed
[    2.372801] ehea: IBM eHEA ethernet device driver (Release EHEA_0107)
[    2.373650] sd 0:3:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    2.428501]  sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 >
[    2.434859] sd 0:3:0:0: [sda] Attached SCSI disk
[    2.721794] ibmebus port0: eth0: Jumbo frames are enabled
[    2.721810] ibmebus port0: eth0: logical port id #1
[    2.722064] ibmebus port1: eth1: Jumbo frames are enabled
[    2.722070] ibmebus port1: eth1: logical port id #2
[    2.722119] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.2.13 (Jan 31, 2011)
[    2.722218] Solarflare NET driver v3.1
[    2.722377] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.722593] ehci_hcd 0001:00:01.2: enabling device (0140 -> 0142)
[    2.722855] ehci_hcd 0001:00:01.2: EHCI Host Controller
[    2.723110] ehci_hcd 0001:00:01.2: new USB bus registered, assigned bus number 1
[    2.859084] ehci_hcd 0001:00:01.2: irq 307, io mem 0x3c00ffffd000
[    2.879005] ehci_hcd 0001:00:01.2: USB 2.0 started, EHCI 1.00
[    2.879042] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    2.879049] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.879056] usb usb1: Product: EHCI Host Controller
[    2.879061] usb usb1: Manufacturer: Linux 2.6.39-rc6-autokern1 ehci_hcd
[    2.879068] usb usb1: SerialNumber: 0001:00:01.2
[    2.879206] hub 1-0:1.0: USB hub found
[    2.879214] hub 1-0:1.0: 5 ports detected
[    2.879317] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.879588] ohci_hcd 0001:00:01.0: OHCI Host Controller
[    2.879642] ohci_hcd 0001:00:01.0: new USB bus registered, assigned bus number 2
[    2.889109] ohci_hcd 0001:00:01.0: irq 305, io mem 0x3c00ffffe000
[    2.979039] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[    2.979049] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.979055] usb usb2: Product: OHCI Host Controller
[    2.979061] usb usb2: Manufacturer: Linux 2.6.39-rc6-autokern1 ohci_hcd
[    2.979066] usb usb2: SerialNumber: 0001:00:01.0
[    2.979186] hub 2-0:1.0: USB hub found
[    2.979196] hub 2-0:1.0: 3 ports detected
[    2.979512] ohci_hcd 0001:00:01.1: OHCI Host Controller
[    2.979565] ohci_hcd 0001:00:01.1: new USB bus registered, assigned bus number 3
[    2.989098] ohci_hcd 0001:00:01.1: irq 306, io mem 0x3c00fffff000
[    3.079035] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[    3.079045] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.079052] usb usb3: Product: OHCI Host Controller
[    3.079057] usb usb3: Manufacturer: Linux 2.6.39-rc6-autokern1 ohci_hcd
[    3.079064] usb usb3: SerialNumber: 0001:00:01.1
[    3.079195] hub 3-0:1.0: USB hub found
[    3.079205] hub 3-0:1.0: 2 ports detected
[    3.079290] uhci_hcd: USB Universal Host Controller Interface driver
[    3.079456] mousedev: PS/2 mouse device common for all mice
[    3.079668] rtc-generic rtc-generic: rtc core: registered rtc-generic as rtc0
[    3.079721] lirc_dev: IR Remote Control driver registered, major 249 
[    3.079727] IR NEC protocol handler initialized
[    3.079732] IR RC5(x) protocol handler initialized
[    3.079736] IR RC6 protocol handler initialized
[    3.079741] IR JVC protocol handler initialized
[    3.079746] IR Sony protocol handler initialized
[    3.079750] IR RC5 (streamzap) protocol handler initialized
[    3.079755] IR LIRC bridge handler initialized
[    3.079760] Linux video capture interface: v2.00
[    3.079780] i2c-core: driver [tuner] using legacy suspend method
[    3.079786] i2c-core: driver [tuner] using legacy resume method
[    3.079852] i2c-core: driver [msp3400] using legacy suspend method
[    3.079858] i2c-core: driver [msp3400] using legacy resume method
[    3.079940] bttv: driver version 0.9.18 loaded
[    3.079946] bttv: using 8 buffers with 2112k (33 pages) each for capture
[    3.079986] cx88/0: cx2388x v4l2 driver version 0.0.8 loaded
[    3.080016] ivtv: Start initialization, version 1.4.2
[    3.080038] ivtv: End initialization
[    3.080043] cx18:  Start initialization, version 1.4.0
[    3.080064] cx18:  End initialization
[    3.080070] cx23885 driver version 0.0.2 loaded
[    3.080093] au0828 driver loaded
[    3.080117] usbcore: registered new interface driver au0828
[    3.080122] saa7164 driver loaded
[    3.080165] bt878: AUDIO driver version 0.0.0 loaded
[    3.080334] device-mapper: uevent: version 1.0.3
[    3.080441] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) initialised: dm-devel@redhat.com
[    3.080805] usbcore: registered new interface driver usbhid
[    3.080811] usbhid: USB HID core driver
[    3.080871] ALSA device list:
[    3.080876]   No soundcards found.
[    3.080927] TCP cubic registered
[    3.081073] NET: Registered protocol family 10
[    3.084277] NET: Registered protocol family 17
[    3.086348] registered taskstats version 1
[    3.088974] rtc-generic rtc-generic: setting system clock to 2011-05-23 00:13:40 UTC (1306109620)
[    3.088988] Initializing network drop monitor service
[    3.090833] Freeing unused kernel memory: 6720k freed
[    3.109731] dracut: dracut-004-32.el6
[    3.125026] udev: starting version 147
[    3.125106] udevd (103): /proc/103/oom_adj is deprecated, please use /proc/103/oom_score_adj instead.
[    3.200846] dracut: Starting plymouth daemon
[    4.915583] EXT3-fs: barriers not enabled
[    4.920290] kjournald starting.  Commit interval 5 seconds
[    4.920589] EXT3-fs (sda5): mounted filesystem with ordered data mode
[    5.069390] dracut: Remounting /dev/sda5 with -o errors=remount-ro,ro
[    5.072976] EXT3-fs: barriers not enabled
[    5.073241] kjournald starting.  Commit interval 5 seconds
[    5.073538] EXT3-fs (sda5): mounted filesystem with ordered data mode
[    5.082228] dracut: Mounted root filesystem /dev/sda5
[    5.135981] dracut: Loading SELinux policy
[    5.345407] SELinux:  Disabled at runtime.
[    5.729055] type=1404 audit(1306109623.140:2): selinux=0 auid=4294967295 ses=4294967295
[    5.729314] dracut: /sbin/load_policy: Can't load policy: No such file or directory
[    6.096298] dracut: Switching root
[    6.892524] readahead: starting
[    7.329092] udev: starting version 147
[    7.449252] WARNING! power/level is deprecated; use power/control instead
[    8.854267] EXT3-fs (sda5): using internal journal
[    9.210822] Adding 4095936k swap on /dev/sda7.  Priority:-1 extents:1 across:4095936k 
[    9.963675] ibmebus port0: eth0: Physical port up
[    9.963688] ibmebus port0: eth0: External switch port is backup port
[   11.450988] ibmebus port1: eth1: Physical port up
[   11.451001] ibmebus port1: eth1: External switch port is backup port

Red Hat Enterprise Linux Server release 6.0 (Santiago)
Kernel 2.6.39-rc6-autokern1 on an ppc64

vogons.ltc.austin.ibm.com login: [   57.308347] rcu-torture:--- Start of test: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[   62.840044] cpu 7 (hwid 7) Ready to die...
[   65.969954] cpu 1 (hwid 1) Ready to die...
[   69.099908] cpu 4 (hwid 4) Ready to die...
[   72.209933] cpu 2 (hwid 2) Ready to die...
[   75.239002] Switched to NOHz mode on CPU #2
[   78.719945] cpu 2 (hwid 2) Ready to die...
[   81.830108] cpu 3 (hwid 3) Ready to die...
[   84.859001] Switched to NOHz mode on CPU #6
[   87.309028] rcu-torture: rtc: d000000002b07c58 ver: 1339 tfle: 0 rta: 1339 rtaf: 107 rtf: 1338 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 22508
[   87.309031] rcu-torture: Reader Pipe:  57820449 276 0 0 0 0 0 0 0 0 0
[   87.309032] rcu-torture: Reader Batch:  57820497 236 0 0 0 0 0 0 0 0 0
[   87.309034] rcu-torture: Free-Block Circulation:  1338 1338 1338 1338 1338 1338 1338 1338 1338 1338 0
[   88.218997] Switched to NOHz mode on CPU #1
[   91.710244] cpu 6 (hwid 6) Ready to die...
[   94.809595] cpu 1 (hwid 1) Ready to die...
[   97.849004] Switched to NOHz mode on CPU #7
[  101.229000] Switched to NOHz mode on CPU #3
[  104.749768] cpu 5 (hwid 5) Ready to die...
[  107.778997] Switched to NOHz mode on CPU #1
[  111.218998] Switched to NOHz mode on CPU #5
[  114.749843] cpu 1 (hwid 1) Ready to die...
[  117.309040] rcu-torture: rtc: d000000002b08618 ver: 2692 tfle: 0 rta: 2692 rtaf: 195 rtf: 2691 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 45732
[  117.309043] rcu-torture: Reader Pipe:  99475710 468 0 0 0 0 0 0 0 0 0
[  117.309044] rcu-torture: Reader Batch:  99475855 374 0 0 0 0 0 0 0 0 0
[  117.309046] rcu-torture: Free-Block Circulation:  2691 2691 2691 2691 2691 2691 2691 2691 2691 2691 0
[  117.849778] cpu 0 (hwid 0) Ready to die...
[  120.888999] Switched to NOHz mode on CPU #1
[  124.299006] Switched to NOHz mode on CPU #2
[  127.709957] cpu 7 (hwid 7) Ready to die...
[  130.799924] cpu 5 (hwid 5) Ready to die...
[  133.829004] Switched to NOHz mode on CPU #7
[  137.239754] cpu 1 (hwid 1) Ready to die...
[  140.340108] cpu 7 (hwid 7) Ready to die...
[  143.419551] cpu 3 (hwid 3) Ready to die...
[  146.459002] Switched to NOHz mode on CPU #5
[  147.309031] rcu-torture: rtc: d000000002b07928 ver: 4033 tfle: 0 rta: 4033 rtaf: 326 rtf: 4032 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 69222
[  147.309033] rcu-torture: Reader Pipe:  131520986 597 0 0 0 0 0 0 0 0 0
[  147.309034] rcu-torture: Reader Batch:  131521161 498 0 0 0 0 0 0 0 0 0
[  147.309036] rcu-torture: Free-Block Circulation:  4032 4032 4032 4032 4032 4032 4032 4032 4032 4032 0
[  149.509005] Switched to NOHz mode on CPU #7
[  152.909630] cpu 5 (hwid 5) Ready to die...
[  155.999535] cpu 7 (hwid 7) Ready to die...
[  159.059000] Switched to NOHz mode on CPU #6
[  162.399003] Switched to NOHz mode on CPU #7
[  165.759002] Switched to NOHz mode on CPU #4
[  169.178999] Switched to NOHz mode on CPU #1
[  172.689824] cpu 4 (hwid 4) Ready to die...
[  175.719003] Switched to NOHz mode on CPU #5
[  177.309035] rcu-torture: rtc: d000000002b08378 ver: 5477 tfle: 0 rta: 5477 rtaf: 326 rtf: 5476 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 92739
[  177.309037] rcu-torture: Reader Pipe:  167861749 756 0 0 0 0 0 0 0 0 0
[  177.309039] rcu-torture: Reader Batch:  167861963 629 0 0 0 0 0 0 0 0 0
[  177.309040] rcu-torture: Free-Block Circulation:  5476 5476 5476 5476 5476 5476 5476 5476 5476 5476 0
[  179.199961] cpu 7 (hwid 7) Ready to die...
[  182.309899] cpu 5 (hwid 5) Ready to die...
[  185.419635] cpu 2 (hwid 2) Ready to die...
[  188.510514] cpu 6 (hwid 6) Ready to die...
[  191.578998] Switched to NOHz mode on CPU #2
[  194.919002] Switched to NOHz mode on CPU #5
[  198.279004] Switched to NOHz mode on CPU #7
[  201.679717] cpu 7 (hwid 7) Ready to die...
[  204.709003] Switched to NOHz mode on CPU #7
[  207.309028] rcu-torture: rtc: d000000002b07d78 ver: 6845 tfle: 0 rta: 6845 rtaf: 365 rtf: 6844 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 115995
[  207.309030] rcu-torture: Reader Pipe:  204020855 899 0 0 0 0 0 0 0 0 0
[  207.309032] rcu-torture: Reader Batch:  204021133 737 0 0 0 0 0 0 0 0 0
[  207.309033] rcu-torture: Free-Block Circulation:  6844 6844 6844 6844 6844 6844 6844 6844 6844 6844 0
[  208.129993] cpu 5 (hwid 5) Ready to die...
[  211.158995] Switched to NOHz mode on CPU #0
[  214.369765] cpu 1 (hwid 1) Ready to die...
[  217.399000] Switched to NOHz mode on CPU #6
[  220.749003] Switched to NOHz mode on CPU #5
[  224.249799] cpu 2 (hwid 2) Ready to die...
[  227.369723] cpu 0 (hwid 0) Ready to die...
[  230.399008] Switched to NOHz mode on CPU #1
[  233.849749] cpu 1 (hwid 1) Ready to die...
[  236.879001] Switched to NOHz mode on CPU #2
[  237.309023] rcu-torture: rtc: d000000002b078c8 ver: 8219 tfle: 0 rta: 8219 rtaf: 465 rtf: 8218 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 139180
[  237.309026] rcu-torture: Reader Pipe:  243079329 1068 0 0 0 0 0 0 0 0 0
[  237.309027] rcu-torture: Reader Batch:  243079691 847 0 0 0 0 0 0 0 0 0
[  237.309028] rcu-torture: Free-Block Circulation:  8218 8218 8218 8218 8218 8218 8218 8218 8218 8218 0
[  240.238999] Switched to NOHz mode on CPU #1
[  243.769811] cpu 2 (hwid 2) Ready to die...
[  246.879712] cpu 6 (hwid 6) Ready to die...
[  249.990137] cpu 7 (hwid 7) Ready to die...
[  253.019003] Switched to NOHz mode on CPU #0
[  256.439578] cpu 1 (hwid 1) Ready to die...
[  259.469001] Switched to NOHz mode on CPU #3
[  262.819006] Switched to NOHz mode on CPU #4
[  266.229679] cpu 4 (hwid 4) Ready to die...
[  267.309028] rcu-torture: rtc: d000000002b07958 ver: 9634 tfle: 0 rta: 9634 rtaf: 518 rtf: 9633 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 162647
[  267.309030] rcu-torture: Reader Pipe:  280679035 1221 0 0 0 0 0 0 0 0 0
[  267.309032] rcu-torture: Reader Batch:  280679461 978 0 0 0 0 0 0 0 0 0
[  267.309033] rcu-torture: Free-Block Circulation:  9633 9633 9633 9633 9633 9633 9633 9633 9633 9633 0
[  269.349753] cpu 5 (hwid 5) Ready to die...
[  272.379003] Switched to NOHz mode on CPU #4
[  275.729005] Switched to NOHz mode on CPU #1
[  279.199785] cpu 0 (hwid 0) Ready to die...
[  282.229000] Switched to NOHz mode on CPU #5
[  285.579005] Switched to NOHz mode on CPU #7
[  289.010033] cpu 1 (hwid 1) Ready to die...
[  292.130185] cpu 3 (hwid 3) Ready to die...
[  295.159004] Switched to NOHz mode on CPU #1
[  297.309035] rcu-torture: rtc: d000000002b08288 ver: 10971 tfle: 0 rta: 10971 rtaf: 640 rtf: 10970 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 185887
[  297.309037] rcu-torture: Reader Pipe:  317303546 1363 0 0 0 0 0 0 0 0 0
[  297.309038] rcu-torture: Reader Batch:  317304050 1056 0 0 0 0 0 0 0 0 0
[  297.309040] rcu-torture: Free-Block Circulation:  10970 10970 10970 10970 10970 10970 10970 10970 10970 10970 0
[  298.599848] cpu 7 (hwid 7) Ready to die...
[  301.711055] cpu 4 (hwid 4) Ready to die...
[  304.739003] Switched to NOHz mode on CPU #0
[  308.240254] cpu 5 (hwid 5) Ready to die...
[  311.278998] Switched to NOHz mode on CPU #3
[  314.629004] Switched to NOHz mode on CPU #5
[  318.069753] cpu 3 (hwid 3) Ready to die...
[  321.099002] Switched to NOHz mode on CPU #6
[  324.519730] cpu 6 (hwid 6) Ready to die...
[  327.309030] rcu-torture: rtc: d000000002b07718 ver: 12216 tfle: 0 rta: 12216 rtaf: 827 rtf: 12215 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 208961
[  327.309033] rcu-torture: Reader Pipe:  342130752 1442 0 0 0 0 0 0 0 0 0
[  327.309034] rcu-torture: Reader Batch:  342131299 1133 0 0 0 0 0 0 0 0 0
[  327.309036] rcu-torture: Free-Block Circulation:  12215 12215 12215 12215 12215 12215 12215 12215 12215 12215 0
[  327.619973] cpu 0 (hwid 0) Ready to die...
[  330.658998] Switched to NOHz mode on CPU #4
[  334.059635] cpu 1 (hwid 1) Ready to die...
[  337.089002] Switched to NOHz mode on CPU #1
[  340.449000] Switched to NOHz mode on CPU #7
[  343.789006] Switched to NOHz mode on CPU #6
[  347.129002] Switched to NOHz mode on CPU #0
[  350.580175] cpu 0 (hwid 0) Ready to die...
[  353.689868] cpu 6 (hwid 6) Ready to die...
[  356.809676] cpu 1 (hwid 1) Ready to die...
[  357.309030] rcu-torture: rtc: d000000002b08738 ver: 13597 tfle: 0 rta: 13597 rtaf: 904 rtf: 13596 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 232248
[  357.309033] rcu-torture: Reader Pipe:  385077127 1595 0 0 0 0 0 0 0 0 0
[  357.309034] rcu-torture: Reader Batch:  385077683 1305 0 0 0 0 0 0 0 0 0
[  357.309035] rcu-torture: Free-Block Circulation:  13596 13596 13596 13596 13596 13596 13596 13596 13596 13596 0
[  359.910136] cpu 4 (hwid 4) Ready to die...
[  362.939003] Switched to NOHz mode on CPU #0
[  366.299007] Switched to NOHz mode on CPU #2
[  369.709002] Switched to NOHz mode on CPU #1
[  373.069005] Switched to NOHz mode on CPU #6
[  376.659990] cpu 0 (hwid 0) Ready to die...
[  379.780047] cpu 7 (hwid 7) Ready to die...
[  382.809004] Switched to NOHz mode on CPU #7
[  386.159004] Switched to NOHz mode on CPU #0
[  387.309021] rcu-torture: rtc: d000000002b08108 ver: 14862 tfle: 0 rta: 14862 rtaf: 1075 rtf: 14861 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 255431
[  387.309023] rcu-torture: Reader Pipe:  427667186 1735 0 0 0 0 0 0 0 0 0
[  387.309025] rcu-torture: Reader Batch:  427667732 1458 0 0 0 0 0 0 0 0 0
[  387.309026] rcu-torture: Free-Block Circulation:  14861 14861 14861 14861 14861 14861 14861 14861 14861 14861 0
[  389.590417] cpu 6 (hwid 6) Ready to die...
[  392.869847] cpu 2 (hwid 2) Ready to die...
[  395.899005] Switched to NOHz mode on CPU #4
[  399.359863] cpu 0 (hwid 0) Ready to die...
[  402.389005] Switched to NOHz mode on CPU #3
[  405.919891] cpu 5 (hwid 5) Ready to die...
[  408.948999] Switched to NOHz mode on CPU #6
[  412.409831] cpu 6 (hwid 6) Ready to die...
[  415.539770] cpu 1 (hwid 1) Ready to die...
[  417.309029] rcu-torture: rtc: d000000002b07b98 ver: 16235 tfle: 0 rta: 16235 rtaf: 1173 rtf: 16234 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 278878
[  417.309031] rcu-torture: Reader Pipe:  482365168 1945 0 0 0 0 0 0 0 0 0
[  417.309033] rcu-torture: Reader Batch:  482365703 1704 0 0 0 0 0 0 0 0 0
[  417.309034] rcu-torture: Free-Block Circulation:  16234 16234 16234 16234 16234 16234 16234 16234 16234 16234 0
[  418.569001] Switched to NOHz mode on CPU #2
[  422.010132] cpu 7 (hwid 7) Ready to die...
[  425.099654] cpu 4 (hwid 4) Ready to die...
[  428.139001] Switched to NOHz mode on CPU #6
[  431.508998] Switched to NOHz mode on CPU #1
[  434.929765] cpu 3 (hwid 3) Ready to die...
[  438.029881] cpu 6 (hwid 6) Ready to die...
[  441.149489] cpu 2 (hwid 2) Ready to die...
[  444.189001] Switched to NOHz mode on CPU #3
[  447.239002] Switched to NOHz mode on CPU #2
[  447.309032] rcu-torture: rtc: d000000002b07898 ver: 17541 tfle: 0 rta: 17541 rtaf: 1281 rtf: 17540 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 302172
[  447.309035] rcu-torture: Reader Pipe:  510233529 2039 0 0 0 0 0 0 0 0 0
[  447.309037] rcu-torture: Reader Batch:  510234137 1769 0 0 0 0 0 0 0 0 0
[  447.309038] rcu-torture: Free-Block Circulation:  17540 17540 17540 17540 17540 17540 17540 17540 17540 17540 0
[  450.589000] Switched to NOHz mode on CPU #6
[  454.009715] cpu 2 (hwid 2) Ready to die...
[  457.109603] cpu 6 (hwid 6) Ready to die...
[  460.220150] cpu 1 (hwid 1) Ready to die...
[  463.259001] Switched to NOHz mode on CPU #1
[  466.309001] Switched to NOHz mode on CPU #4
[  469.648999] Switched to NOHz mode on CPU #5
[  473.079697] cpu 4 (hwid 4) Ready to die...
[  476.179633] cpu 1 (hwid 1) Ready to die...
[  477.309037] rcu-torture: rtc: d000000002b08348 ver: 18971 tfle: 0 rta: 18971 rtaf: 1322 rtf: 18970 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 325896
[  477.309040] rcu-torture: Reader Pipe:  541015434 2137 0 0 0 0 0 0 0 0 0
[  477.309041] rcu-torture: Reader Batch:  541016074 1857 0 0 0 0 0 0 0 0 0
[  477.309043] rcu-torture: Free-Block Circulation:  18970 18970 18970 18970 18970 18970 18970 18970 18970 18970 0
[  479.218996] Switched to NOHz mode on CPU #2
[  482.579002] Switched to NOHz mode on CPU #4
[  486.079718] cpu 2 (hwid 2) Ready to die...
[  489.109004] Switched to NOHz mode on CPU #7
[  492.459000] Switched to NOHz mode on CPU #1
[  495.869007] Switched to NOHz mode on CPU #2
[  499.410306] cpu 4 (hwid 4) Ready to die...
[  502.539862] cpu 5 (hwid 5) Ready to die...
[  505.569003] Switched to NOHz mode on CPU #6
[  507.309034] rcu-torture: rtc: d000000002b08228 ver: 20365 tfle: 0 rta: 20365 rtaf: 1408 rtf: 20364 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 349493
[  507.309037] rcu-torture: Reader Pipe:  584666871 2320 0 0 0 0 0 0 0 0 0
[  507.309038] rcu-torture: Reader Batch:  584667596 1989 0 0 0 0 0 0 0 0 0
[  507.309039] rcu-torture: Free-Block Circulation:  20364 20364 20364 20364 20364 20364 20364 20364 20364 20364 0
[  509.059867] cpu 2 (hwid 2) Ready to die...
[  512.089001] Switched to NOHz mode on CPU #5
[  515.609877] cpu 7 (hwid 7) Ready to die...
[  518.639000] Switched to NOHz mode on CPU #0
[  521.989000] Switched to NOHz mode on CPU #7
[  525.679978] cpu 0 (hwid 0) Ready to die...
[  528.799901] cpu 3 (hwid 3) Ready to die...
[  531.828999] Switched to NOHz mode on CPU #4
[  535.299830] cpu 6 (hwid 6) Ready to die...
[  537.309029] rcu-torture: rtc: d000000002b08288 ver: 21670 tfle: 0 rta: 21670 rtaf: 1549 rtf: 21669 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 372686
[  537.309031] rcu-torture: Reader Pipe:  630754044 2509 0 0 0 0 0 0 0 0 0
[  537.309033] rcu-torture: Reader Batch:  630754852 2126 0 0 0 0 0 0 0 0 0
[  537.309034] rcu-torture: Free-Block Circulation:  21669 21669 21669 21669 21669 21669 21669 21669 21669 21669 0
[  538.449755] cpu 5 (hwid 5) Ready to die...
[  541.479005] Switched to NOHz mode on CPU #2
[  545.099797] cpu 1 (hwid 1) Ready to die...
[  548.199786] cpu 4 (hwid 4) Ready to die...
[  551.239001] Switched to NOHz mode on CPU #1
[  554.669001] Switched to NOHz mode on CPU #4
[  558.100519] cpu 1 (hwid 1) Ready to die...
[  561.129002] Switched to NOHz mode on CPU #1
[  564.439007] Switched to NOHz mode on CPU #6
[  567.309035] rcu-torture: rtc: d000000002b086d8 ver: 23095 tfle: 0 rta: 23095 rtaf: 1561 rtf: 23094 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 395967
[  567.309038] rcu-torture: Reader Pipe:  666570812 2664 0 0 0 0 0 0 0 0 0
[  567.309040] rcu-torture: Reader Batch:  666571679 2230 0 0 0 0 0 0 0 0 0
[  567.309041] rcu-torture: Free-Block Circulation:  23094 23094 23094 23094 23094 23094 23094 23094 23094 23094 0
[  567.840052] cpu 2 (hwid 2) Ready to die...
[  570.868998] Switched to NOHz mode on CPU #2
[  574.379835] cpu 6 (hwid 6) Ready to die...
[  577.480272] cpu 7 (hwid 7) Ready to die...
[  580.610357] cpu 1 (hwid 1) Ready to die...
[  583.659002] Switched to NOHz mode on CPU #7
[  587.049635] cpu 4 (hwid 4) Ready to die...
[  590.129551] cpu 2 (hwid 2) Ready to die...
[  593.169001] Switched to NOHz mode on CPU #6
[  596.219001] Switched to NOHz mode on CPU #2
[  597.309031] rcu-torture: rtc: d000000002b081f8 ver: 24468 tfle: 0 rta: 24468 rtaf: 1624 rtf: 24467 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 419247
[  597.309034] rcu-torture: Reader Pipe:  696464023 2759 0 0 0 0 0 0 0 0 0
[  597.309035] rcu-torture: Reader Batch:  696464935 2339 0 0 0 0 0 0 0 0 0
[  597.309036] rcu-torture: Free-Block Circulation:  24467 24467 24467 24467 24467 24467 24467 24467 24467 24467 0
[  599.649703] cpu 6 (hwid 6) Ready to die...
[  602.679005] Switched to NOHz mode on CPU #3
[  606.099614] cpu 3 (hwid 3) Ready to die...
[  609.129001] Switched to NOHz mode on CPU #4
[  612.479003] Switched to NOHz mode on CPU #1
[  615.899695] cpu 2 (hwid 2) Ready to die...
[  618.928997] Switched to NOHz mode on CPU #2
[  622.349010] Switched to NOHz mode on CPU #6
[  625.879836] cpu 6 (hwid 6) Ready to die...
[  627.309023] rcu-torture: rtc: d000000002b07538 ver: 25802 tfle: 0 rta: 25802 rtaf: 1755 rtf: 25801 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 442689
[  627.309025] rcu-torture: Reader Pipe:  727706425 2869 0 0 0 0 0 0 0 0 0
[  627.309027] rcu-torture: Reader Batch:  727707412 2425 0 0 0 0 0 0 0 0 0
[  627.309028] rcu-torture: Free-Block Circulation:  25801 25801 25801 25801 25801 25801 25801 25801 25801 25801 0
[  628.908997] Switched to NOHz mode on CPU #0
[  632.339812] cpu 4 (hwid 4) Ready to die...
[  635.369003] Switched to NOHz mode on CPU #6
[  638.729007] Switched to NOHz mode on CPU #5
[  642.088999] Switched to NOHz mode on CPU #4
[  645.650036] cpu 5 (hwid 5) Ready to die...
[  648.679000] Switched to NOHz mode on CPU #3
[  652.160172] cpu 2 (hwid 2) Ready to die...
[  655.319933] cpu 6 (hwid 6) Ready to die...
[  657.309019] rcu-torture: rtc: d000000002b08318 ver: 27170 tfle: 0 rta: 27170 rtaf: 1838 rtf: 27169 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 466080
[  657.309022] rcu-torture: Reader Pipe:  793342101 3161 0 0 0 0 0 0 0 0 0
[  657.309023] rcu-torture: Reader Batch:  793343120 2718 0 0 0 0 0 0 0 0 0
[  657.309024] rcu-torture: Free-Block Circulation:  27169 27169 27169 27169 27169 27169 27169 27169 27169 27169 0
[  658.439917] cpu 7 (hwid 7) Ready to die...
[  661.561098] cpu 3 (hwid 3) Ready to die...
[  664.589005] Switched to NOHz mode on CPU #6
[  668.029000] Switched to NOHz mode on CPU #3
[  671.450205] cpu 0 (hwid 0) Ready to die...
[  674.479007] Switched to NOHz mode on CPU #5
[  677.940774] cpu 4 (hwid 4) Ready to die...
[  680.969001] Switched to NOHz mode on CPU #7
[  684.549816] cpu 5 (hwid 5) Ready to die...
[  687.309027] rcu-torture: rtc: d000000002b08558 ver: 28593 tfle: 0 rta: 28593 rtaf: 1878 rtf: 28592 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 489425
[  687.309029] rcu-torture: Reader Pipe:  843852580 3389 0 0 0 0 0 0 0 0 0
[  687.309031] rcu-torture: Reader Batch:  843853648 2924 0 0 0 0 0 0 0 0 0
[  687.309032] rcu-torture: Free-Block Circulation:  28592 28592 28592 28592 28592 28592 28592 28592 28592 28592 0
[  687.599001] Switched to NOHz mode on CPU #5
[  690.958998] Switched to NOHz mode on CPU #4
[  694.560023] cpu 1 (hwid 1) Ready to die...
[  697.719866] cpu 5 (hwid 5) Ready to die...
[  700.749000] Switched to NOHz mode on CPU #0
[  704.169835] cpu 6 (hwid 6) Ready to die...
[  707.199002] Switched to NOHz mode on CPU #1
[  710.699826] cpu 4 (hwid 4) Ready to die...
[  713.729005] Switched to NOHz mode on CPU #6
[  717.089004] Switched to NOHz mode on CPU #4
[  717.309024] rcu-torture: rtc: d000000002b07538 ver: 29896 tfle: 0 rta: 29896 rtaf: 2020 rtf: 29895 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 512297
[  717.309027] rcu-torture: Reader Pipe:  894409233 3591 0 0 0 0 0 0 0 0 0
[  717.309028] rcu-torture: Reader Batch:  894410295 3168 0 0 0 0 0 0 0 0 0
[  717.309030] rcu-torture: Free-Block Circulation:  29895 29895 29895 29895 29895 29895 29895 29895 29895 29895 0
[  720.530062] cpu 6 (hwid 6) Ready to die...
[  723.649923] cpu 3 (hwid 3) Ready to die...
[  726.679004] Switched to NOHz mode on CPU #6
[  730.160738] cpu 4 (hwid 4) Ready to die...
[  733.259696] cpu 6 (hwid 6) Ready to die...
[  736.289002] Switched to NOHz mode on CPU #4
[  739.770004] cpu 4 (hwid 4) Ready to die...
[  742.799002] Switched to NOHz mode on CPU #3
[  746.149006] Switched to NOHz mode on CPU #2
[  747.309027] rcu-torture: rtc: d000000002b08318 ver: 31173 tfle: 0 rta: 31173 rtaf: 2192 rtf: 31172 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 535439
[  747.309030] rcu-torture: Reader Pipe:  944813051 3772 0 0 0 0 0 0 0 0 0
[  747.309031] rcu-torture: Reader Batch:  944814136 3358 0 0 0 0 0 0 0 0 0
[  747.309033] rcu-torture: Free-Block Circulation:  31172 31172 31172 31172 31172 31172 31172 31172 31172 31172 0
[  749.508997] Switched to NOHz mode on CPU #6
[  752.950016] cpu 0 (hwid 0) Ready to die...
[  756.079819] cpu 2 (hwid 2) Ready to die...
[  759.109003] Switched to NOHz mode on CPU #4
[  762.529006] Switched to NOHz mode on CPU #0
[  766.059903] cpu 3 (hwid 3) Ready to die...
[  769.088998] Switched to NOHz mode on CPU #3
[  772.559929] cpu 0 (hwid 0) Ready to die...
[  775.679836] cpu 7 (hwid 7) Ready to die...
[  777.309022] rcu-torture: rtc: d000000002b07bf8 ver: 32345 tfle: 0 rta: 32345 rtaf: 2493 rtf: 32344 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 558730
[  777.309025] rcu-torture: Reader Pipe:  994426665 3953 0 0 0 0 0 0 0 0 0
[  777.309026] rcu-torture: Reader Batch:  994427719 3609 0 0 0 0 0 0 0 0 0
[  777.309028] rcu-torture: Free-Block Circulation:  32344 32344 32344 32344 32344 32344 32344 32344 32344 32344 0
[  778.799712] cpu 6 (hwid 6) Ready to die...
[  781.829001] Switched to NOHz mode on CPU #7
[  785.179006] Switched to NOHz mode on CPU #2
[  788.599005] Switched to NOHz mode on CPU #0
[  792.120068] cpu 1 (hwid 1) Ready to die...
[  795.149008] Switched to NOHz mode on CPU #5
[  798.610316] cpu 5 (hwid 5) Ready to die...
[  801.739934] cpu 0 (hwid 0) Ready to die...
[  804.769003] Switched to NOHz mode on CPU #5
[  807.309028] rcu-torture: rtc: d000000002b08468 ver: 33489 tfle: 0 rta: 33489 rtaf: 2739 rtf: 33488 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 580911
[  807.309030] rcu-torture: Reader Pipe:  1040624638 4102 0 0 0 0 0 0 0 0 0
[  807.309032] rcu-torture: Reader Batch:  1040625661 3777 0 0 0 0 0 0 0 0 0
[  807.309034] rcu-torture: Free-Block Circulation:  33488 33488 33488 33488 33488 33488 33488 33488 33488 33488 0
[  808.310290] cpu 5 (hwid 5) Ready to die...
[  811.470945] cpu 7 (hwid 7) Ready to die...
[  814.569621] cpu 3 (hwid 3) Ready to die...
[  817.599003] Switched to NOHz mode on CPU #6
[  821.040106] cpu 4 (hwid 4) Ready to die...
[  824.129530] cpu 6 (hwid 6) Ready to die...
[  833.199001] Switched to NOHz mode on CPU #6
[  836.309535] cpu 6 (hwid 6) Ready to die...
[  837.309027] rcu-torture: rtc: d000000002b07ce8 ver: 34862 tfle: 0 rta: 34862 rtaf: 2795 rtf: 34861 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 604269
[  837.309029] rcu-torture: Reader Pipe:  1067776459 4202 0 0 0 0 0 0 0 0 0
[  837.309030] rcu-torture: Reader Batch:  1067777527 3839 0 0 0 0 0 0 0 0 0
[  837.309032] rcu-torture: Free-Block Circulation:  34861 34861 34861 34861 34861 34861 34861 34861 34861 34861 0
[  839.378993] Switched to NOHz mode on CPU #3
[  842.719003] Switched to NOHz mode on CPU #1
[  846.069003] Switched to NOHz mode on CPU #5
[  849.418999] Switched to NOHz mode on CPU #6
[  852.959872] cpu 2 (hwid 2) Ready to die...
[  855.989006] Switched to NOHz mode on CPU #4
[  859.539885] cpu 5 (hwid 5) Ready to die...
[  862.569007] Switched to NOHz mode on CPU #7
[  866.119836] cpu 7 (hwid 7) Ready to die...
[  867.309034] rcu-torture: rtc: d000000002b07da8 ver: 36221 tfle: 0 rta: 36221 rtaf: 2863 rtf: 36220 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 627435
[  867.309037] rcu-torture: Reader Pipe:  1106674713 4355 0 0 0 0 0 0 0 0 0
[  867.309038] rcu-torture: Reader Batch:  1106675808 4009 0 0 0 0 0 0 0 0 0
[  867.309040] rcu-torture: Free-Block Circulation:  36220 36220 36220 36220 36220 36220 36220 36220 36220 36220 0
[  869.260001] cpu 1 (hwid 1) Ready to die...
[  872.288999] Switched to NOHz mode on CPU #5
[  875.699752] cpu 6 (hwid 6) Ready to die...
[  878.738995] Switched to NOHz mode on CPU #2
[  882.079005] Switched to NOHz mode on CPU #7
[  885.519835] cpu 4 (hwid 4) Ready to die...
[  888.620588] cpu 7 (hwid 7) Ready to die...
[  891.709002] Switched to NOHz mode on CPU #6
[  895.109750] cpu 5 (hwid 5) Ready to die...
[  897.309028] rcu-torture: rtc: d000000002b07e98 ver: 37557 tfle: 0 rta: 37557 rtaf: 2973 rtf: 37556 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 650766
[  897.309031] rcu-torture: Reader Pipe:  1144454313 4496 0 0 0 0 0 0 0 0 0
[  897.309033] rcu-torture: Reader Batch:  1144455536 4095 0 0 0 0 0 0 0 0 0
[  897.309034] rcu-torture: Free-Block Circulation:  37556 37556 37556 37556 37556 37556 37556 37556 37556 37556 0
[  898.220080] cpu 6 (hwid 6) Ready to die...
[  901.258995] Switched to NOHz mode on CPU #6
[  904.669609] cpu 2 (hwid 2) Ready to die...
[  907.699002] Switched to NOHz mode on CPU #0
[  911.058999] Switched to NOHz mode on CPU #4
[  914.449734] cpu 4 (hwid 4) Ready to die...
[  917.489001] Switched to NOHz mode on CPU #4
[  920.838998] Switched to NOHz mode on CPU #7
[  924.439845] cpu 4 (hwid 4) Ready to die...
[  927.309032] rcu-torture: rtc: d000000002b07598 ver: 39017 tfle: 0 rta: 39017 rtaf: 2973 rtf: 39016 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 674310
[  927.309034] rcu-torture: Reader Pipe:  1183799858 4643 0 0 0 0 0 0 0 0 0
[  927.309036] rcu-torture: Reader Batch:  1183801142 4225 0 0 0 0 0 0 0 0 0
[  927.309037] rcu-torture: Free-Block Circulation:  39016 39016 39016 39016 39016 39016 39016 39016 39016 39016 0
[  927.619737] cpu 6 (hwid 6) Ready to die...
[  930.719573] cpu 7 (hwid 7) Ready to die...
[  933.749003] Switched to NOHz mode on CPU #6
[  937.179571] cpu 6 (hwid 6) Ready to die...
[  940.208998] Switched to NOHz mode on CPU #2
[  943.579001] Switched to NOHz mode on CPU #6
[  946.919007] Switched to NOHz mode on CPU #1
[  950.269002] Switched to NOHz mode on CPU #4
[  953.749961] cpu 0 (hwid 0) Ready to die...
[  956.779006] Switched to NOHz mode on CPU #5
[  957.309025] rcu-torture: rtc: d000000002b07b38 ver: 40198 tfle: 0 rta: 40198 rtaf: 3250 rtf: 40197 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 697556
[  957.309028] rcu-torture: Reader Pipe:  1217872657 4754 0 0 0 0 0 0 0 0 0
[  957.309030] rcu-torture: Reader Batch:  1217873965 4341 0 0 0 0 0 0 0 0 0
[  957.309031] rcu-torture: Free-Block Circulation:  40197 40197 40197 40197 40197 40197 40197 40197 40197 40197 0
[  960.209004] Switched to NOHz mode on CPU #7
[  963.840061] cpu 1 (hwid 1) Ready to die...
[  966.869001] Switched to NOHz mode on CPU #0
[  970.310245] cpu 6 (hwid 6) Ready to die...
[  973.459939] cpu 2 (hwid 2) Ready to die...
[  976.569845] cpu 3 (hwid 3) Ready to die...
[  979.680271] cpu 0 (hwid 0) Ready to die...
[  982.779706] cpu 4 (hwid 4) Ready to die...
[  985.809002] Switched to NOHz mode on CPU #6
[  987.309024] rcu-torture: rtc: d000000002b07898 ver: 41351 tfle: 0 rta: 41351 rtaf: 3438 rtf: 41350 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 719350
[  987.309026] rcu-torture: Reader Pipe:  1271744824 4972 0 0 0 0 0 0 0 0 0
[  987.309028] rcu-torture: Reader Batch:  1271746112 4611 0 0 0 0 0 0 0 0 0
[  987.309029] rcu-torture: Free-Block Circulation:  41350 41350 41350 41350 41350 41350 41350 41350 41350 41350 0
[  989.169002] Switched to NOHz mode on CPU #0
[  992.519002] Switched to NOHz mode on CPU #3
[  995.849008] Switched to NOHz mode on CPU #2
[  999.699914] cpu 0 (hwid 0) Ready to die...
[ 1002.989831] cpu 3 (hwid 3) Ready to die...
[ 1006.099726] cpu 6 (hwid 6) Ready to die...
[ 1009.128997] Switched to NOHz mode on CPU #4
[ 1012.569767] cpu 7 (hwid 7) Ready to die...
[ 1015.599003] Switched to NOHz mode on CPU #7
[ 1017.309024] rcu-torture: rtc: d000000002b07568 ver: 42692 tfle: 0 rta: 42692 rtaf: 3561 rtf: 42691 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 742625
[ 1017.309026] rcu-torture: Reader Pipe:  1319106169 5144 0 0 0 0 0 0 0 0 0
[ 1017.309028] rcu-torture: Reader Batch:  1319107478 4774 0 0 0 0 0 0 0 0 0
[ 1017.309029] rcu-torture: Free-Block Circulation:  42691 42691 42691 42691 42691 42691 42691 42691 42691 42691 0
[ 1019.050343] cpu 4 (hwid 4) Ready to die...
[ 1022.210194] cpu 7 (hwid 7) Ready to die...
[ 1025.239003] Switched to NOHz mode on CPU #3
[ 1028.639581] cpu 3 (hwid 3) Ready to die...
[ 1031.669002] Switched to NOHz mode on CPU #3
[ 1035.018995] Switched to NOHz mode on CPU #6
[ 1038.368997] Switched to NOHz mode on CPU #7
[ 1041.790002] cpu 2 (hwid 2) Ready to die...
[ 1044.909713] cpu 3 (hwid 3) Ready to die...
[ 1047.309031] rcu-torture: rtc: d000000002b08348 ver: 43973 tfle: 0 rta: 43973 rtaf: 3731 rtf: 43972 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 766181
[ 1047.309033] rcu-torture: Reader Pipe:  1352432576 5235 0 0 0 0 0 0 0 0 0
[ 1047.309035] rcu-torture: Reader Batch:  1352433893 4890 0 0 0 0 0 0 0 0 0
[ 1047.309036] rcu-torture: Free-Block Circulation:  43972 43972 43972 43972 43972 43972 43972 43972 43972 43972 0
[ 1047.949003] Switched to NOHz mode on CPU #3
[ 1051.490702] cpu 6 (hwid 6) Ready to die...
[ 1054.519004] Switched to NOHz mode on CPU #2
[ 1058.039695] cpu 2 (hwid 2) Ready to die...
[ 1061.068997] Switched to NOHz mode on CPU #2
[ 1064.489685] cpu 2 (hwid 2) Ready to die...
[ 1067.610080] cpu 5 (hwid 5) Ready to die...
[ 1070.648999] Switched to NOHz mode on CPU #5
[ 1074.129669] cpu 3 (hwid 3) Ready to die...
[ 1077.159001] Switched to NOHz mode on CPU #3
[ 1077.309030] rcu-torture: rtc: d000000002b078c8 ver: 45443 tfle: 0 rta: 45443 rtaf: 3731 rtf: 45442 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 789548
[ 1077.309033] rcu-torture: Reader Pipe:  1389751161 5367 0 0 0 0 0 0 0 0 0
[ 1077.309034] rcu-torture: Reader Batch:  1389752516 4993 0 0 0 0 0 0 0 0 0
[ 1077.309036] rcu-torture: Free-Block Circulation:  45442 45442 45442 45442 45442 45442 45442 45442 45442 45442 0
[ 1080.539004] Switched to NOHz mode on CPU #1
[ 1084.049819] cpu 5 (hwid 5) Ready to die...
[ 1087.079004] Switched to NOHz mode on CPU #2
[ 1090.439000] Switched to NOHz mode on CPU #5
[ 1093.789010] Switched to NOHz mode on CPU #4
[ 1097.400727] cpu 4 (hwid 4) Ready to die...
[ 1100.519820] cpu 3 (hwid 3) Ready to die...
[ 1103.549005] Switched to NOHz mode on CPU #0
[ 1106.969003] Switched to NOHz mode on CPU #3
[ 1107.309027] rcu-torture: rtc: d000000002b08018 ver: 46783 tfle: 0 rta: 46783 rtaf: 3842 rtf: 46782 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 812800
[ 1107.309030] rcu-torture: Reader Pipe:  1436860431 5541 0 0 0 0 0 0 0 0 0
[ 1107.309031] rcu-torture: Reader Batch:  1436861819 5148 0 0 0 0 0 0 0 0 0
[ 1107.309033] rcu-torture: Free-Block Circulation:  46782 46782 46782 46782 46782 46782 46782 46782 46782 46782 0
[ 1110.500218] cpu 0 (hwid 0) Ready to die...
[ 1113.629808] cpu 5 (hwid 5) Ready to die...
[ 1116.739677] cpu 7 (hwid 7) Ready to die...
[ 1119.769004] Switched to NOHz mode on CPU #6
[ 1123.199778] cpu 1 (hwid 1) Ready to die...
[ 1126.319602] cpu 6 (hwid 6) Ready to die...
[ 1129.359001] Switched to NOHz mode on CPU #4
[ 1132.709001] Switched to NOHz mode on CPU #6
[ 1136.129002] Switched to NOHz mode on CPU #0
[ 1137.309028] rcu-torture: rtc: d000000002b07b98 ver: 48052 tfle: 0 rta: 48052 rtaf: 4020 rtf: 48051 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 836334
[ 1137.309030] rcu-torture: Reader Pipe:  1478456758 5700 0 0 0 0 0 0 0 0 0
[ 1137.309032] rcu-torture: Reader Batch:  1478458162 5314 0 0 0 0 0 0 0 0 0
[ 1137.309034] rcu-torture: Free-Block Circulation:  48051 48051 48051 48051 48051 48051 48051 48051 48051 48051 0
[ 1139.640536] cpu 0 (hwid 0) Ready to die...
[ 1142.739808] cpu 2 (hwid 2) Ready to die...
[ 1145.769000] Switched to NOHz mode on CPU #0
[ 1149.270555] cpu 3 (hwid 3) Ready to die...
[ 1152.369626] cpu 6 (hwid 6) Ready to die...
[ 1155.399000] Switched to NOHz mode on CPU #2
[ 1158.899571] cpu 4 (hwid 4) Ready to die...
[ 1161.929001] Switched to NOHz mode on CPU #6
[ 1165.279001] Switched to NOHz mode on CPU #4
[ 1167.309027] rcu-torture: rtc: d000000002b085e8 ver: 49374 tfle: 0 rta: 49374 rtaf: 4138 rtf: 49373 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 859363
[ 1167.309030] rcu-torture: Reader Pipe:  1515282107 5897 0 0 0 0 0 0 0 0 0
[ 1167.309032] rcu-torture: Reader Batch:  1515283536 5500 0 0 0 0 0 0 0 0 0
[ 1167.309033] rcu-torture: Free-Block Circulation:  49373 49373 49373 49373 49373 49373 49373 49373 49373 49373 0
[ 1168.708998] Switched to NOHz mode on CPU #7
[ 1172.148999] Switched to NOHz mode on CPU #3
[ 1175.479005] Switched to NOHz mode on CPU #1
[ 1178.980028] cpu 2 (hwid 2) Ready to die...
[ 1182.099964] cpu 1 (hwid 1) Ready to die...
[ 1185.129004] Switched to NOHz mode on CPU #1
[ 1188.560076] cpu 4 (hwid 4) Ready to die...
[ 1191.729902] cpu 6 (hwid 6) Ready to die...
[ 1194.839719] cpu 3 (hwid 3) Ready to die...
[ 1197.309019] rcu-torture: rtc: d000000002b07bc8 ver: 50600 tfle: 0 rta: 50600 rtaf: 4340 rtf: 50599 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 882296
[ 1197.309022] rcu-torture: Reader Pipe:  1572568082 6118 0 0 0 0 0 0 0 0 0
[ 1197.309023] rcu-torture: Reader Batch:  1572569513 5756 0 0 0 0 0 0 0 0 0
[ 1197.309024] rcu-torture: Free-Block Circulation:  50599 50599 50599 50599 50599 50599 50599 50599 50599 50599 0
[ 1197.960142] cpu 1 (hwid 1) Ready to die...
[ 1200.998998] Switched to NOHz mode on CPU #3
[ 1204.379004] Switched to NOHz mode on CPU #6
[ 1207.809788] cpu 7 (hwid 7) Ready to die...
[ 1210.839001] Switched to NOHz mode on CPU #5
[ 1214.339820] cpu 0 (hwid 0) Ready to die...
[ 1217.368997] Switched to NOHz mode on CPU #2
[ 1220.729001] Switched to NOHz mode on CPU #1
[ 1224.149839] cpu 6 (hwid 6) Ready to die...
[ 1227.269673] cpu 5 (hwid 5) Ready to die...
[ 1227.309044] rcu-torture: rtc: d000000002b07988 ver: 51932 tfle: 0 rta: 51932 rtaf: 4460 rtf: 51931 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 905659
[ 1227.309047] rcu-torture: Reader Pipe:  1608378751 6264 0 0 0 0 0 0 0 0 0
[ 1227.309048] rcu-torture: Reader Batch:  1608380217 5873 0 0 0 0 0 0 0 0 0
[ 1227.309050] rcu-torture: Free-Block Circulation:  51931 51931 51931 51931 51931 51931 51931 51931 51931 51931 0
[ 1230.380012] cpu 2 (hwid 2) Ready to die...
[ 1233.409002] Switched to NOHz mode on CPU #6
[ 1236.809698] cpu 3 (hwid 3) Ready to die...
[ 1239.910277] cpu 1 (hwid 1) Ready to die...
[ 1242.949002] Switched to NOHz mode on CPU #4
[ 1245.999003] Switched to NOHz mode on CPU #3
[ 1249.390601] cpu 6 (hwid 6) Ready to die...
[ 1252.469487] cpu 4 (hwid 4) Ready to die...
[ 1255.509000] Switched to NOHz mode on CPU #2
[ 1257.309028] rcu-torture: rtc: d000000002b07628 ver: 53327 tfle: 0 rta: 53327 rtaf: 4509 rtf: 53326 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 929221
[ 1257.309031] rcu-torture: Reader Pipe:  1634440053 6343 0 0 0 0 0 0 0 0 0
[ 1257.309032] rcu-torture: Reader Batch:  1634441503 5969 0 0 0 0 0 0 0 0 0
[ 1257.309034] rcu-torture: Free-Block Circulation:  53326 53326 53326 53326 53326 53326 53326 53326 53326 53326 0
[ 1258.568998] Switched to NOHz mode on CPU #1
[ 1261.928999] Switched to NOHz mode on CPU #6
[ 1265.289004] Switched to NOHz mode on CPU #5
[ 1268.809960] cpu 3 (hwid 3) Ready to die...
[ 1271.920251] cpu 1 (hwid 1) Ready to die...
[ 1274.949007] Switched to NOHz mode on CPU #4
[ 1278.328999] Switched to NOHz mode on CPU #1
[ 1281.769999] cpu 5 (hwid 5) Ready to die...
[ 1284.869788] cpu 2 (hwid 2) Ready to die...
[ 1287.309027] rcu-torture: rtc: d000000002b075f8 ver: 54616 tfle: 0 rta: 54616 rtaf: 4682 rtf: 54615 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 952306
[ 1287.309029] rcu-torture: Reader Pipe:  1674571436 6495 0 0 0 0 0 0 0 0 0
[ 1287.309031] rcu-torture: Reader Batch:  1674572898 6139 0 0 0 0 0 0 0 0 0
[ 1287.309032] rcu-torture: Free-Block Circulation:  54615 54615 54615 54615 54615 54615 54615 54615 54615 54615 0
[ 1287.898998] Switched to NOHz mode on CPU #5
[ 1291.328999] Switched to NOHz mode on CPU #2
[ 1294.749895] cpu 1 (hwid 1) Ready to die...
[ 1297.778994] Switched to NOHz mode on CPU #3
[ 1301.199915] cpu 2 (hwid 2) Ready to die...
[ 1304.299776] cpu 3 (hwid 3) Ready to die...
[ 1307.409851] cpu 6 (hwid 6) Ready to die...
[ 1310.539692] cpu 4 (hwid 4) Ready to die...
[ 1313.579005] Switched to NOHz mode on CPU #7
[ 1316.629000] Switched to NOHz mode on CPU #2
[ 1317.309026] rcu-torture: rtc: d000000002b07748 ver: 56039 tfle: 0 rta: 56039 rtaf: 4682 rtf: 56038 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 975281
[ 1317.309028] rcu-torture: Reader Pipe:  1711711038 6628 0 0 0 0 0 0 0 0 0
[ 1317.309030] rcu-torture: Reader Batch:  1711712535 6287 0 0 0 0 0 0 0 0 0
[ 1317.309031] rcu-torture: Free-Block Circulation:  56038 56038 56038 56038 56038 56038 56038 56038 56038 56038 0
[ 1320.129890] cpu 7 (hwid 7) Ready to die...
[ 1323.229602] cpu 5 (hwid 5) Ready to die...
[ 1326.269002] Switched to NOHz mode on CPU #4
[ 1329.420121] cpu 4 (hwid 4) Ready to die...
[ 1332.459003] Switched to NOHz mode on CPU #6
[ 1335.509004] Switched to NOHz mode on CPU #7
[ 1338.848999] Switched to NOHz mode on CPU #1
[ 1342.209003] Switched to NOHz mode on CPU #5
[ 1345.559007] Switched to NOHz mode on CPU #0
[ 1347.309029] rcu-torture: rtc: d000000002b07658 ver: 57504 tfle: 0 rta: 57504 rtaf: 4682 rtf: 57503 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 998737
[ 1347.309032] rcu-torture: Reader Pipe:  1741502304 6739 0 0 0 0 0 0 0 0 0
[ 1347.309033] rcu-torture: Reader Batch:  1741503826 6379 0 0 0 0 0 0 0 0 0
[ 1347.309035] rcu-torture: Free-Block Circulation:  57503 57503 57503 57503 57503 57503 57503 57503 57503 57503 0
[ 1349.029936] cpu 7 (hwid 7) Ready to die...
[ 1352.059000] Switched to NOHz mode on CPU #4
[ 1355.659940] cpu 5 (hwid 5) Ready to die...
[ 1358.790147] cpu 4 (hwid 4) Ready to die...
[ 1361.839001] Switched to NOHz mode on CPU #4
[ 1365.259857] cpu 2 (hwid 2) Ready to die...
[ 1368.288997] Switched to NOHz mode on CPU #5
[ 1371.780046] cpu 0 (hwid 0) Ready to die...
[ 1374.859001] Switched to NOHz mode on CPU #0
[ 1377.309030] rcu-torture: rtc: d000000002b08408 ver: 58788 tfle: 0 rta: 58788 rtaf: 4868 rtf: 58787 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1022291
[ 1377.309032] rcu-torture: Reader Pipe:  1794636737 6945 0 0 0 0 0 0 0 0 0
[ 1377.309034] rcu-torture: Reader Batch:  1794638221 6651 0 0 0 0 0 0 0 0 0
[ 1377.309035] rcu-torture: Free-Block Circulation:  58787 58787 58787 58787 58787 58787 58787 58787 58787 58787 0
[ 1378.310733] cpu 6 (hwid 6) Ready to die...
[ 1381.348998] Switched to NOHz mode on CPU #6
[ 1384.689005] Switched to NOHz mode on CPU #7
[ 1388.199993] cpu 4 (hwid 4) Ready to die...
[ 1391.229001] Switched to NOHz mode on CPU #4
[ 1394.669942] cpu 7 (hwid 7) Ready to die...
[ 1397.789999] cpu 4 (hwid 4) Ready to die...
[ 1400.819000] Switched to NOHz mode on CPU #7
[ 1404.249833] cpu 7 (hwid 7) Ready to die...
[ 1407.309026] rcu-torture: rtc: d000000002b078f8 ver: 60027 tfle: 0 rta: 60027 rtaf: 5099 rtf: 60026 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1045552
[ 1407.309028] rcu-torture: Reader Pipe:  1842757013 7135 0 0 0 0 0 0 0 0 0
[ 1407.309030] rcu-torture: Reader Batch:  1842758568 6836 0 0 0 0 0 0 0 0 0
[ 1407.309031] rcu-torture: Free-Block Circulation:  60026 60026 60026 60026 60026 60026 60026 60026 60026 60026 0
[ 1407.369729] cpu 6 (hwid 6) Ready to die...
[ 1410.408996] Switched to NOHz mode on CPU #4
[ 1413.859686] cpu 4 (hwid 4) Ready to die...
[ 1416.959616] cpu 0 (hwid 0) Ready to die...
[ 1419.998995] Switched to NOHz mode on CPU #4
[ 1423.419003] Switched to NOHz mode on CPU #0
[ 1426.919762] cpu 1 (hwid 1) Ready to die...
[ 1430.050161] cpu 0 (hwid 0) Ready to die...
[ 1433.139485] cpu 5 (hwid 5) Ready to die...
[ 1436.179004] Switched to NOHz mode on CPU #1
[ 1437.309024] rcu-torture: rtc: d000000002b083d8 ver: 61377 tfle: 0 rta: 61377 rtaf: 5226 rtf: 61376 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1068844
[ 1437.309026] rcu-torture: Reader Pipe:  1871408577 7257 0 0 0 0 0 0 0 0 0
[ 1437.309028] rcu-torture: Reader Batch:  1871410264 6899 0 0 0 0 0 0 0 0 0
[ 1437.309029] rcu-torture: Free-Block Circulation:  61376 61376 61376 61376 61376 61376 61376 61376 61376 61376 0
[ 1439.229001] Switched to NOHz mode on CPU #0
[ 1442.609002] Switched to NOHz mode on CPU #6
[ 1446.029730] cpu 1 (hwid 1) Ready to die...
[ 1449.058998] Switched to NOHz mode on CPU #5
[ 1452.509747] cpu 5 (hwid 5) Ready to die...
[ 1455.599698] cpu 4 (hwid 4) Ready to die...
[ 1458.628999] Switched to NOHz mode on CPU #7
[ 1462.059003] Switched to NOHz mode on CPU #2
[ 1465.489005] Switched to NOHz mode on CPU #1
[ 1467.309026] rcu-torture: rtc: d000000002b07f88 ver: 62750 tfle: 0 rta: 62750 rtaf: 5311 rtf: 62749 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1092378
[ 1467.309029] rcu-torture: Reader Pipe:  1904431875 7407 0 0 0 0 0 0 0 0 0
[ 1467.309030] rcu-torture: Reader Batch:  1904433633 6996 0 0 0 0 0 0 0 0 0
[ 1467.309032] rcu-torture: Free-Block Circulation:  62749 62749 62749 62749 62749 62749 62749 62749 62749 62749 0
[ 1468.909871] cpu 7 (hwid 7) Ready to die...
[ 1472.054584] cpu 2 (hwid 2) Ready to die...
[ 1475.098996] Switched to NOHz mode on CPU #7
[ 1478.519961] cpu 6 (hwid 6) Ready to die...
[ 1481.558998] Switched to NOHz mode on CPU #2
[ 1485.009740] cpu 7 (hwid 7) Ready to die...
[ 1488.039004] Switched to NOHz mode on CPU #5
[ 1491.549690] cpu 2 (hwid 2) Ready to die...
[ 1494.679625] cpu 0 (hwid 0) Ready to die...
[ 1497.309031] rcu-torture: rtc: d000000002b07ad8 ver: 64015 tfle: 0 rta: 64015 rtaf: 5506 rtf: 64014 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1115873
[ 1497.309034] rcu-torture: Reader Pipe:  1942059632 7542 0 0 0 0 0 0 0 0 0
[ 1497.309035] rcu-torture: Reader Batch:  1942061398 7173 0 0 0 0 0 0 0 0 0
[ 1497.309037] rcu-torture: Free-Block Circulation:  64014 64014 64014 64014 64014 64014 64014 64014 64014 64014 0
[ 1497.719001] Switched to NOHz mode on CPU #7
[ 1501.170548] cpu 5 (hwid 5) Ready to die...
[ 1504.219006] Switched to NOHz mode on CPU #3
[ 1507.579001] Switched to NOHz mode on CPU #4
[ 1510.998997] Switched to NOHz mode on CPU #0
[ 1514.609871] cpu 0 (hwid 0) Ready to die...
[ 1517.639003] Switched to NOHz mode on CPU #0
[ 1521.070247] cpu 3 (hwid 3) Ready to die...
[ 1524.099001] Switched to NOHz mode on CPU #3
[ 1527.309024] rcu-torture: rtc: d000000002b07f58 ver: 65445 tfle: 0 rta: 65445 rtaf: 5527 rtf: 65444 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1139039
[ 1527.309026] rcu-torture: Reader Pipe:  1987317115 7720 0 0 0 0 0 0 0 0 0
[ 1527.309028] rcu-torture: Reader Batch:  1987318931 7321 0 0 0 0 0 0 0 0 0
[ 1527.309029] rcu-torture: Free-Block Circulation:  65444 65444 65444 65444 65444 65444 65444 65444 65444 65444 0
[ 1527.449000] Switched to NOHz mode on CPU #5
[ 1530.940022] cpu 1 (hwid 1) Ready to die...
[ 1534.079856] cpu 7 (hwid 7) Ready to die...
[ 1537.169766] cpu 3 (hwid 3) Ready to die...
[ 1540.208998] Switched to NOHz mode on CPU #3
[ 1543.639004] Switched to NOHz mode on CPU #7
[ 1547.109840] cpu 7 (hwid 7) Ready to die...
[ 1550.138996] Switched to NOHz mode on CPU #1
[ 1553.499006] Switched to NOHz mode on CPU #6
[ 1556.939922] cpu 1 (hwid 1) Ready to die...
[ 1557.309033] rcu-torture: rtc: d000000002b08528 ver: 66648 tfle: 0 rta: 66648 rtaf: 5759 rtf: 66647 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1162083
[ 1557.309035] rcu-torture: Reader Pipe:  2034033213 7869 0 0 0 0 0 0 0 0 0
[ 1557.309037] rcu-torture: Reader Batch:  2034034983 7544 0 0 0 0 0 0 0 0 0
[ 1557.309038] rcu-torture: Free-Block Circulation:  66647 66647 66647 66647 66647 66647 66647 66647 66647 66647 0
[ 1559.969002] Switched to NOHz mode on CPU #1
[ 1563.579933] cpu 1 (hwid 1) Ready to die...
[ 1566.869843] cpu 4 (hwid 4) Ready to die...
[ 1569.990409] cpu 5 (hwid 5) Ready to die...
[ 1573.019003] Switched to NOHz mode on CPU #7
[ 1576.449771] cpu 0 (hwid 0) Ready to die...
[ 1579.488997] Switched to NOHz mode on CPU #2
[ 1583.039673] cpu 2 (hwid 2) Ready to die...
[ 1586.069001] Switched to NOHz mode on CPU #1
[ 1587.309029] rcu-torture: rtc: d000000002b07fb8 ver: 68040 tfle: 0 rta: 68040 rtaf: 5822 rtf: 68039 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1185427
[ 1587.309032] rcu-torture: Reader Pipe:  2081200988 8077 0 0 0 0 0 0 0 0 0
[ 1587.309033] rcu-torture: Reader Batch:  2081202862 7696 0 0 0 0 0 0 0 0 0
[ 1587.309035] rcu-torture: Free-Block Circulation:  68039 68039 68039 68039 68039 68039 68039 68039 68039 68039 0
[ 1589.499695] cpu 6 (hwid 6) Ready to die...
[ 1592.529002] Switched to NOHz mode on CPU #2
[ 1595.879003] Switched to NOHz mode on CPU #4
[ 1599.459865] cpu 4 (hwid 4) Ready to die...
[ 1602.559749] cpu 1 (hwid 1) Ready to die...
[ 1605.669642] cpu 7 (hwid 7) Ready to die...
[ 1608.708999] Switched to NOHz mode on CPU #4
[ 1612.069000] Switched to NOHz mode on CPU #7
[ 1615.409003] Switched to NOHz mode on CPU #1
[ 1617.309024] rcu-torture: rtc: d000000002b08588 ver: 69354 tfle: 0 rta: 69354 rtaf: 5968 rtf: 69353 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1208647
[ 1617.309027] rcu-torture: Reader Pipe:  2115106458 8221 0 0 0 0 0 0 0 0 0
[ 1617.309029] rcu-torture: Reader Batch:  2115108400 7829 0 0 0 0 0 0 0 0 0
[ 1617.309030] rcu-torture: Free-Block Circulation:  69353 69353 69353 69353 69353 69353 69353 69353 69353 69353 0
[ 1618.829802] cpu 1 (hwid 1) Ready to die...
[ 1621.859001] Switched to NOHz mode on CPU #5
[ 1625.219001] Switched to NOHz mode on CPU #1
[ 1628.649000] Switched to NOHz mode on CPU #6
[ 1632.150333] cpu 3 (hwid 3) Ready to die...
[ 1635.179006] Switched to NOHz mode on CPU #0
[ 1638.680084] cpu 7 (hwid 7) Ready to die...
[ 1641.810173] cpu 5 (hwid 5) Ready to die...
[ 1644.839006] Switched to NOHz mode on CPU #7
[ 1647.309025] rcu-torture: rtc: d000000002b07778 ver: 70591 tfle: 0 rta: 70591 rtaf: 6188 rtf: 70590 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1232046
[ 1647.309027] rcu-torture: Reader Pipe:  2173994787 8453 0 0 0 0 0 0 0 0 0
[ 1647.309029] rcu-torture: Reader Batch:  2173996731 8099 0 0 0 0 0 0 0 0 0
[ 1647.309030] rcu-torture: Free-Block Circulation:  70590 70590 70590 70590 70590 70590 70590 70590 70590 70590 0
[ 1648.319962] cpu 6 (hwid 6) Ready to die...
[ 1651.348997] Switched to NOHz mode on CPU #3
[ 1654.739010] Switched to NOHz mode on CPU #5
[ 1658.294997] cpu 7 (hwid 7) Ready to die...
[ 1661.470017] cpu 1 (hwid 1) Ready to die...
[ 1664.589836] cpu 0 (hwid 0) Ready to die...
[ 1667.699865] cpu 2 (hwid 2) Ready to die...
[ 1670.799632] cpu 3 (hwid 3) Ready to die...
[ 1673.889515] cpu 5 (hwid 5) Ready to die...
[ 1676.929002] Switched to NOHz mode on CPU #2
[ 1677.309028] rcu-torture: rtc: d000000002b084c8 ver: 71858 tfle: 0 rta: 71858 rtaf: 6384 rtf: 71857 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1255032
[ 1677.309030] rcu-torture: Reader Pipe:  2222621877 8657 0 0 0 0 0 0 0 0 0
[ 1677.309032] rcu-torture: Reader Batch:  2222623907 8285 0 0 0 0 0 0 0 0 0
[ 1677.309033] rcu-torture: Free-Block Circulation:  71857 71857 71857 71857 71857 71857 71857 71857 71857 71857 0
[ 1679.978995] Switched to NOHz mode on CPU #3
[ 1683.379593] cpu 3 (hwid 3) Ready to die...
[ 1686.409003] Switched to NOHz mode on CPU #6
[ 1689.869899] cpu 4 (hwid 4) Ready to die...
[ 1692.899001] Switched to NOHz mode on CPU #1
[ 1696.239604] cpu 6 (hwid 6) Ready to die...
[ 1699.319912] cpu 1 (hwid 1) Ready to die...
[ 1702.379001] Switched to NOHz mode on CPU #1
[ 1705.429004] Switched to NOHz mode on CPU #5
[ 1707.309037] rcu-torture: rtc: d000000002b07778 ver: 73193 tfle: 0 rta: 73193 rtaf: 6460 rtf: 73192 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1278258
[ 1707.309040] rcu-torture: Reader Pipe:  2244080094 8734 0 0 0 0 0 0 0 0 0
[ 1707.309041] rcu-torture: Reader Batch:  2244082164 8319 0 0 0 0 0 0 0 0 0
[ 1707.309043] rcu-torture: Free-Block Circulation:  73192 73192 73192 73192 73192 73192 73192 73192 73192 73192 0
[ 1708.848996] Switched to NOHz mode on CPU #3
[ 1712.369723] cpu 3 (hwid 3) Ready to die...
[ 1715.469676] cpu 2 (hwid 2) Ready to die...
[ 1718.569549] cpu 1 (hwid 1) Ready to die...
[ 1721.639001] Switched to NOHz mode on CPU #0
[ 1724.979001] Switched to NOHz mode on CPU #3
[ 1728.369593] cpu 3 (hwid 3) Ready to die...
[ 1731.450271] cpu 5 (hwid 5) Ready to die...
[ 1734.489002] Switched to NOHz mode on CPU #1
[ 1737.309029] rcu-torture: rtc: d000000002b07b68 ver: 74568 tfle: 0 rta: 74568 rtaf: 6524 rtf: 74567 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1301711
[ 1737.309032] rcu-torture: Reader Pipe:  2271004395 8826 0 0 0 0 0 0 0 0 0
[ 1737.309033] rcu-torture: Reader Batch:  2271006492 8393 0 0 0 0 0 0 0 0 0
[ 1737.309034] rcu-torture: Free-Block Circulation:  74567 74567 74567 74567 74567 74567 74567 74567 74567 74567 0
[ 1737.549002] Switched to NOHz mode on CPU #7
[ 1740.889005] Switched to NOHz mode on CPU #4
[ 1744.229006] Switched to NOHz mode on CPU #6
[ 1747.732099] cpu 6 (hwid 6) Ready to die...
[ 1750.850145] cpu 1 (hwid 1) Ready to die...
[ 1753.879001] Switched to NOHz mode on CPU #3
[ 1757.309719] cpu 4 (hwid 4) Ready to die...
[ 1760.430036] cpu 3 (hwid 3) Ready to die...
[ 1763.529574] cpu 0 (hwid 0) Ready to die...
[ 1766.569000] Switched to NOHz mode on CPU #4
[ 1767.309026] rcu-torture: rtc: d000000002b07fb8 ver: 75940 tfle: 0 rta: 75940 rtaf: 6611 rtf: 75939 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1324929
[ 1767.309028] rcu-torture: Reader Pipe:  2306992809 8968 0 0 0 0 0 0 0 0 0
[ 1767.309030] rcu-torture: Reader Batch:  2306994915 8533 0 0 0 0 0 0 0 0 0
[ 1767.309031] rcu-torture: Free-Block Circulation:  75939 75939 75939 75939 75939 75939 75939 75939 75939 75939 0
[ 1769.619004] Switched to NOHz mode on CPU #1
[ 1772.969002] Switched to NOHz mode on CPU #2
[ 1776.329005] Switched to NOHz mode on CPU #5
[ 1779.759941] cpu 7 (hwid 7) Ready to die...
[ 1782.849789] cpu 1 (hwid 1) Ready to die...
[ 1785.949598] cpu 5 (hwid 5) Ready to die...
[ 1788.978999] Switched to NOHz mode on CPU #1
[ 1792.339000] Switched to NOHz mode on CPU #3
[ 1795.759003] Switched to NOHz mode on CPU #5
[ 1797.309018] rcu-torture: rtc: d000000002b08618 ver: 77194 tfle: 0 rta: 77194 rtaf: 6810 rtf: 77193 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1348090
[ 1797.309020] rcu-torture: Reader Pipe:  2336638413 9069 0 0 0 0 0 0 0 0 0
[ 1797.309021] rcu-torture: Reader Batch:  2336640610 8595 0 0 0 0 0 0 0 0 0
[ 1797.309023] rcu-torture: Free-Block Circulation:  77193 77193 77193 77193 77193 77193 77193 77193 77193 77193 0
[ 1799.189003] Switched to NOHz mode on CPU #0
[ 1802.689936] cpu 3 (hwid 3) Ready to die...
[ 1805.809794] cpu 5 (hwid 5) Ready to die...
[ 1808.959800] cpu 2 (hwid 2) Ready to die...
[ 1811.989000] Switched to NOHz mode on CPU #3
[ 1815.339004] Switched to NOHz mode on CPU #6
[ 1818.819848] cpu 0 (hwid 0) Ready to die...
[ 1821.848998] Switched to NOHz mode on CPU #0
[ 1825.389883] cpu 1 (hwid 1) Ready to die...
[ 1827.309023] rcu-torture: rtc: d000000002b08318 ver: 78356 tfle: 0 rta: 78356 rtaf: 7072 rtf: 78355 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1371257
[ 1827.309026] rcu-torture: Reader Pipe:  2378621042 9229 0 0 0 0 0 0 0 0 0
[ 1827.309027] rcu-torture: Reader Batch:  2378623246 8768 0 0 0 0 0 0 0 0 0
[ 1827.309028] rcu-torture: Free-Block Circulation:  78355 78355 78355 78355 78355 78355 78355 78355 78355 78355 0
[ 1828.419002] Switched to NOHz mode on CPU #7
[ 1831.860049] cpu 3 (hwid 3) Ready to die...
[ 1834.939002] Switched to NOHz mode on CPU #1
[ 1838.288998] Switched to NOHz mode on CPU #5
[ 1841.820168] cpu 4 (hwid 4) Ready to die...
[ 1844.939853] cpu 0 (hwid 0) Ready to die...
[ 1848.040477] cpu 7 (hwid 7) Ready to die...
[ 1851.069001] Switched to NOHz mode on CPU #2
[ 1854.519715] cpu 6 (hwid 6) Ready to die...
[ 1857.309034] rcu-torture: rtc: d000000002b08528 ver: 79746 tfle: 0 rta: 79746 rtaf: 7141 rtf: 79745 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1394468
[ 1857.309037] rcu-torture: Reader Pipe:  2429087479 9439 0 0 0 0 0 0 0 0 0
[ 1857.309039] rcu-torture: Reader Batch:  2429089713 8978 0 0 0 0 0 0 0 0 0
[ 1857.309040] rcu-torture: Free-Block Circulation:  79745 79745 79745 79745 79745 79745 79745 79745 79745 79745 0
[ 1857.315808] rcu-torture: rtc:           (null) ver: 79746 tfle: 0 rta: 79746 rtaf: 7141 rtf: 79745 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1394468
[ 1857.315810] rcu-torture: Reader Pipe:  2429087479 9439 0 0 0 0 0 0 0 0 0
[ 1857.315812] rcu-torture: Reader Batch:  2429089713 8978 0 0 0 0 0 0 0 0 0
[ 1857.315813] rcu-torture: Free-Block Circulation:  79745 79745 79745 79745 79745 79745 79745 79745 79745 79745 0
[ 1857.369027] rcu-torture: rtc:           (null) ver: 79746 tfle: 0 rta: 79746 rtaf: 7141 rtf: 79745 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 1394468
[ 1857.369030] rcu-torture: Reader Pipe:  2429087479 9439 0 0 0 0 0 0 0 0 0
[ 1857.369031] rcu-torture: Reader Batch:  2429089713 8978 0 0 0 0 0 0 0 0 0
[ 1857.369033] rcu-torture: Free-Block Circulation:  79745 79745 79745 79745 79745 79745 79745 79745 79745 79745 0
[ 1857.369064] rcu-torture:--- End of test: SUCCESS: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[ 1857.629688] cpu 2 (hwid 2) Ready to die...
(bot:conmon-payload) disconnected

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-05-24 16:35, Paul E. McKenney wrote:
> On Tue, May 24, 2011 at 11:41:10AM +0200, Jens Axboe wrote:
>> On 2011-05-24 00:20, Paul Bolle wrote:
>>> On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
>>>> Running under CONFIG_PREEMPT=y (along with CONFIG_TREE_PREEMPT_RCU=y)
>>>> could be very helpful in and of itself.  CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
>>>> can also be helpful.  In post-2.6.39 mainline, it should be possible
>>>> to set CONFIG_DEBUG_OBJECTS_RCU_HEAD=y without CONFIG_PREEMPT=y, but
>>>> again, CONFIG_PREEMPT=y can help find problems.
>>>
>>> 0) The first thing I tried (from your suggestions) was
>>> CONFIG_DEBUG_OBJECTS_RCU_HEAD=y. Given its dependencies (and, well, the
>>> build system I used) I ended up with:
>>>
>>>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local3.fc16.i686 |
>>>     grep -v "^#"
>>>     CONFIG_TREE_PREEMPT_RCU=y
>>>     CONFIG_PREEMPT_RCU=y
>>>     CONFIG_RCU_FANOUT=32
>>>     CONFIG_PREEMPT_NOTIFIERS=y
>>>     CONFIG_PREEMPT=y
>>>     CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
>>>     CONFIG_DEBUG_PREEMPT=y
>>>     CONFIG_PROVE_RCU=y
>>>     CONFIG_SPARSE_RCU_POINTER=y
>>>
>>> It looks like I am unable to trigger the issue we're talking about here
>>> when using that config.
>>>
>>> 1) For reference, the config of a kernel that does trigger it had:
>>>
>>>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local2.fc16.i686 |
>>>     grep -v "^#"
>>>     CONFIG_TREE_RCU=y
>>>     CONFIG_RCU_FANOUT=32
>>>     CONFIG_RCU_FAST_NO_HZ=y
>>>     CONFIG_PREEMPT_NOTIFIERS=y
>>>     CONFIG_PREEMPT_VOLUNTARY=y
>>>     CONFIG_PROVE_RCU=y
>>>     CONFIG_SPARSE_RCU_POINTER=y
>>>
>>>>> Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
>>>>> we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
>>>>> the issue?
>>>>
>>>> Please!
>>>
>>> 2) It appears I can't reproduce with those options enabled (see above).
>>>
>>>> Polling is fine.  Please see attached for a script to poll at 15-second
>>>> intervals.  Please also feel free to adjust, just tell me what you
>>>> adjusted.
>>>
>>> And should I now try to run that script on a config that triggers this
>>> issue (such as the config under 1) above)?
>>
>> Paul, can we see a dmesg from your running system? Perhaps there's some
>> dependency on a particular driver or device that makes this easier to
>> reproduce.
> 
> Here you go, please see attached.
> 
> I should have some additional diagnostics later today Pacific time.

Heh sorry, _other_ Paul :-)

You are not seeing this issue, are you?

As per your earlier comment on sleeping under rcu_read_lock(), I checked
everything again and it seems sane. Would that not trigger an
immediately schedule-while-atomic in any case, regardless of RCU config?

-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Tue, May 24, 2011 at 04:51:44PM +0200, Jens Axboe wrote:
> On 2011-05-24 16:35, Paul E. McKenney wrote:
> > On Tue, May 24, 2011 at 11:41:10AM +0200, Jens Axboe wrote:
> >> On 2011-05-24 00:20, Paul Bolle wrote:
> >>> On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
> >>>> Running under CONFIG_PREEMPT=y (along with CONFIG_TREE_PREEMPT_RCU=y)
> >>>> could be very helpful in and of itself.  CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> >>>> can also be helpful.  In post-2.6.39 mainline, it should be possible
> >>>> to set CONFIG_DEBUG_OBJECTS_RCU_HEAD=y without CONFIG_PREEMPT=y, but
> >>>> again, CONFIG_PREEMPT=y can help find problems.
> >>>
> >>> 0) The first thing I tried (from your suggestions) was
> >>> CONFIG_DEBUG_OBJECTS_RCU_HEAD=y. Given its dependencies (and, well, the
> >>> build system I used) I ended up with:
> >>>
> >>>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local3.fc16.i686 |
> >>>     grep -v "^#"
> >>>     CONFIG_TREE_PREEMPT_RCU=y
> >>>     CONFIG_PREEMPT_RCU=y
> >>>     CONFIG_RCU_FANOUT=32
> >>>     CONFIG_PREEMPT_NOTIFIERS=y
> >>>     CONFIG_PREEMPT=y
> >>>     CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> >>>     CONFIG_DEBUG_PREEMPT=y
> >>>     CONFIG_PROVE_RCU=y
> >>>     CONFIG_SPARSE_RCU_POINTER=y
> >>>
> >>> It looks like I am unable to trigger the issue we're talking about here
> >>> when using that config.
> >>>
> >>> 1) For reference, the config of a kernel that does trigger it had:
> >>>
> >>>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local2.fc16.i686 |
> >>>     grep -v "^#"
> >>>     CONFIG_TREE_RCU=y
> >>>     CONFIG_RCU_FANOUT=32
> >>>     CONFIG_RCU_FAST_NO_HZ=y
> >>>     CONFIG_PREEMPT_NOTIFIERS=y
> >>>     CONFIG_PREEMPT_VOLUNTARY=y
> >>>     CONFIG_PROVE_RCU=y
> >>>     CONFIG_SPARSE_RCU_POINTER=y
> >>>
> >>>>> Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
> >>>>> we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
> >>>>> the issue?
> >>>>
> >>>> Please!
> >>>
> >>> 2) It appears I can't reproduce with those options enabled (see above).
> >>>
> >>>> Polling is fine.  Please see attached for a script to poll at 15-second
> >>>> intervals.  Please also feel free to adjust, just tell me what you
> >>>> adjusted.
> >>>
> >>> And should I now try to run that script on a config that triggers this
> >>> issue (such as the config under 1) above)?
> >>
> >> Paul, can we see a dmesg from your running system? Perhaps there's some
> >> dependency on a particular driver or device that makes this easier to
> >> reproduce.
> > 
> > Here you go, please see attached.
> > 
> > I should have some additional diagnostics later today Pacific time.
> 
> Heh sorry, _other_ Paul :-)

;-)

> You are not seeing this issue, are you?

No, I am not.

> As per your earlier comment on sleeping under rcu_read_lock(), I checked
> everything again and it seems sane. Would that not trigger an
> immediately schedule-while-atomic in any case, regardless of RCU config?

This used to be the case, but not anymore.  Hence my additional
diagnostics.

The way that this used to happen is that CONFIG_PREEMPT=y and
CONFIG_TREE_RCU=y would disable preemption across RCU read-side sections.
Then you would see schedule-while-atomic.

However, I recently changed the config setup so that CONFIG_PREEMPT=y
and CONFIG_SMP=y forces CONFIG_TREE_PREEMPT_RCU=y, which avoids that
diagnostic.  So does CONFIG_SMP=n and CONFIG_TREE_RCU=y, in which case
preemption is always disabled, so there is no way for the scheduler to
see that there was anything atomic going on.

So I am adding a similar diagnostic under CONFIG_PROVE_RCU.

Rather embarrassing -- I totally missed the fact that this diagnostic
would be lost.  :-/

Please see below for a sneak preview of the diagnostic patch.

							Thanx, Paul

------------------------------------------------------------------------

rcu: Restore checks for blocking in RCU read-side critical sections

Long ago, using TREE_RCU with PREEMPT would result in "scheduling
while atomic" diagnostics if you blocked in an RCU read-side critical
section.  However, PREEMPT now implies TREE_PREEMPT_RCU, which defeats
this diagnostic.  This commit therefore adds a replacement diagnostic
based on PROVE_RCU.

Because rcu_lockdep_assert() and lockdep_rcu_dereference() are now being
used for things that have nothing to do with rcu_dereference(), rename
lockdep_rcu_dereference() to lockdep_rcu_suspicious() and add a third
argument that is a string indicating what is suspicious.  This third
argument is passed in from a new third argument to rcu_lockdep_assert().
Update all calls to rcu_lockdep_assert() to add an informative third
argument.

Finally, add a pair of rcu_lockdep_assert() calls from within
rcu_note_context_switch(), one complaining if a context switch occurs
in an RCU-bh read-side critical section and another complaining if a
context switch occurs in an RCU-sched read-side critical section.
These are present only if the PROVE_RCU kernel parameter is enabled.

Again, you must enable PROVE_RCU to see these new diagnostics.  But you
are enabling PROVE_RCU to check out new RCU uses in any case, aren't you?

Not-yet-signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h
index 4aef1dd..bfa66e7 100644
--- a/include/linux/lockdep.h
+++ b/include/linux/lockdep.h
@@ -545,7 +545,7 @@ do {									\
 #endif
 
 #ifdef CONFIG_PROVE_RCU
-extern void lockdep_rcu_dereference(const char *file, const int line);
+void lockdep_rcu_suspicious(const char *file, const int line, const char *s);
 #endif
 
 #endif /* __LINUX_LOCKDEP_H */
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 58b13f1..fb2933d 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -297,19 +297,20 @@ extern int rcu_my_thread_group_empty(void);
 /**
  * rcu_lockdep_assert - emit lockdep splat if specified condition not met
  * @c: condition to check
+ * @s: informative message
  */
-#define rcu_lockdep_assert(c)						\
+#define rcu_lockdep_assert(c, s)					\
 	do {								\
 		static bool __warned;					\
 		if (debug_lockdep_rcu_enabled() && !__warned && !(c)) {	\
 			__warned = true;				\
-			lockdep_rcu_dereference(__FILE__, __LINE__);	\
+			lockdep_rcu_suspicious(__FILE__, __LINE__, s);	\
 		}							\
 	} while (0)
 
 #else /* #ifdef CONFIG_PROVE_RCU */
 
-#define rcu_lockdep_assert(c) do { } while (0)
+#define rcu_lockdep_assert(c, s) do { } while (0)
 
 #endif /* #else #ifdef CONFIG_PROVE_RCU */
 
@@ -338,14 +339,16 @@ extern int rcu_my_thread_group_empty(void);
 #define __rcu_dereference_check(p, c, space) \
 	({ \
 		typeof(*p) *_________p1 = (typeof(*p)*__force )ACCESS_ONCE(p); \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, "suspicious rcu_dereference_check()" \
+				      " usage"); \
 		rcu_dereference_sparse(p, space); \
 		smp_read_barrier_depends(); \
 		((typeof(*p) __force __kernel *)(_________p1)); \
 	})
 #define __rcu_dereference_protected(p, c, space) \
 	({ \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, "suspicious rcu_dereference_protected()" \
+				      " usage"); \
 		rcu_dereference_sparse(p, space); \
 		((typeof(*p) __force __kernel *)(p)); \
 	})
@@ -359,7 +362,9 @@ extern int rcu_my_thread_group_empty(void);
 #define __rcu_dereference_index_check(p, c) \
 	({ \
 		typeof(p) _________p1 = ACCESS_ONCE(p); \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, \
+				   "suspicious rcu_dereference_index_check()" \
+				   " usage"); \
 		smp_read_barrier_depends(); \
 		(_________p1); \
 	})
diff --git a/kernel/lockdep.c b/kernel/lockdep.c
index 53a6895..f3e05c1 100644
--- a/kernel/lockdep.c
+++ b/kernel/lockdep.c
@@ -3798,7 +3798,7 @@ void lockdep_sys_exit(void)
 	}
 }
 
-void lockdep_rcu_dereference(const char *file, const int line)
+void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
 {
 	struct task_struct *curr = current;
 
@@ -3807,9 +3807,10 @@ void lockdep_rcu_dereference(const char *file, const int line)
 		return;
 #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
 	/* Note: the following can be executed concurrently, so be careful. */
-	printk("\n===================================================\n");
-	printk(  "[ INFO: suspicious rcu_dereference_check() usage. ]\n");
-	printk(  "---------------------------------------------------\n");
+	printk("\n===============================\n");
+	printk(  "[ INFO: suspicious RCU usage. ]\n");
+	printk(  "-------------------------------\n");
+	printk(  "%s\n", s);
 	printk("%s:%d invoked rcu_dereference_check() without protection!\n",
 			file, line);
 	printk("\nother info that might help us debug this:\n\n");
diff --git a/kernel/pid.c b/kernel/pid.c
index 57a8346..a7577b3 100644
--- a/kernel/pid.c
+++ b/kernel/pid.c
@@ -419,7 +419,9 @@ EXPORT_SYMBOL(pid_task);
  */
 struct task_struct *find_task_by_pid_ns(pid_t nr, struct pid_namespace *ns)
 {
-	rcu_lockdep_assert(rcu_read_lock_held());
+	rcu_lockdep_assert(rcu_read_lock_held(),
+			   "find_task_by_pid_ns() needs rcu_read_lock()"
+			   " protection");
 	return pid_task(find_pid_ns(nr, ns), PIDTYPE_PID);
 }
 
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b2868ea..ec851db 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -153,6 +153,12 @@ void rcu_bh_qs(int cpu)
  */
 void rcu_note_context_switch(int cpu)
 {
+	rcu_lockdep_assert(!rcu_read_lock_bh_held(),
+			   "Illegal context switch in RCU-bh"
+			   " read-side critical section");
+	rcu_lockdep_assert(!rcu_read_lock_sched_held(),
+			   "Illegal context switch in RCU-sched"
+			   " read-side critical section");
 	rcu_sched_qs(cpu);
 	rcu_preempt_note_context_switch(cpu);
 }
@@ -1675,11 +1681,15 @@ static int rcu_node_kthread(void *arg)
 
 	for (;;) {
 		rnp->node_kthread_status = RCU_KTHREAD_WAITING;
+		printk(KERN_INFO "rcun %p starting wait for work.\n", rnp);
 		rcu_wait(atomic_read(&rnp->wakemask) != 0);
+		printk(KERN_INFO "rcun %p completed wait for work.\n", rnp);
 		rnp->node_kthread_status = RCU_KTHREAD_RUNNING;
 		raw_spin_lock_irqsave(&rnp->lock, flags);
 		mask = atomic_xchg(&rnp->wakemask, 0);
+		printk(KERN_INFO "rcun %p initiating boost.\n", rnp);
 		rcu_initiate_boost(rnp, flags); /* releases rnp->lock. */
+		printk(KERN_INFO "rcun %p completed boost.\n", rnp);
 		for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask >>= 1) {
 			if ((mask & 0x1) == 0)
 				continue;
@@ -1689,10 +1699,12 @@ static int rcu_node_kthread(void *arg)
 				preempt_enable();
 				continue;
 			}
+			printk(KERN_INFO "rcun %p awaking rcuc%d.\n", rnp, cpu);
 			per_cpu(rcu_cpu_has_work, cpu) = 1;
 			sp.sched_priority = RCU_KTHREAD_PRIO;
 			sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
 			preempt_enable();
+			printk(KERN_INFO "rcun %p awakened rcuc%d.\n", rnp, cpu);
 		}
 	}
 	/* NOTREACHED */

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Tue, May 24, 2011 at 08:42:38AM -0700, Paul E. McKenney wrote:
> On Tue, May 24, 2011 at 04:51:44PM +0200, Jens Axboe wrote:
> > On 2011-05-24 16:35, Paul E. McKenney wrote:
> > > On Tue, May 24, 2011 at 11:41:10AM +0200, Jens Axboe wrote:
> > >> On 2011-05-24 00:20, Paul Bolle wrote:
> > >>> On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
> > >>>> Running under CONFIG_PREEMPT=y (along with CONFIG_TREE_PREEMPT_RCU=y)
> > >>>> could be very helpful in and of itself.  CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> > >>>> can also be helpful.  In post-2.6.39 mainline, it should be possible
> > >>>> to set CONFIG_DEBUG_OBJECTS_RCU_HEAD=y without CONFIG_PREEMPT=y, but
> > >>>> again, CONFIG_PREEMPT=y can help find problems.
> > >>>
> > >>> 0) The first thing I tried (from your suggestions) was
> > >>> CONFIG_DEBUG_OBJECTS_RCU_HEAD=y. Given its dependencies (and, well, the
> > >>> build system I used) I ended up with:
> > >>>
> > >>>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local3.fc16.i686 |
> > >>>     grep -v "^#"
> > >>>     CONFIG_TREE_PREEMPT_RCU=y
> > >>>     CONFIG_PREEMPT_RCU=y
> > >>>     CONFIG_RCU_FANOUT=32
> > >>>     CONFIG_PREEMPT_NOTIFIERS=y
> > >>>     CONFIG_PREEMPT=y
> > >>>     CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> > >>>     CONFIG_DEBUG_PREEMPT=y
> > >>>     CONFIG_PROVE_RCU=y
> > >>>     CONFIG_SPARSE_RCU_POINTER=y
> > >>>
> > >>> It looks like I am unable to trigger the issue we're talking about here
> > >>> when using that config.
> > >>>
> > >>> 1) For reference, the config of a kernel that does trigger it had:
> > >>>
> > >>>     $ grep -e PREEMPT -e RCU /boot/config-2.6.39-0.local2.fc16.i686 |
> > >>>     grep -v "^#"
> > >>>     CONFIG_TREE_RCU=y
> > >>>     CONFIG_RCU_FANOUT=32
> > >>>     CONFIG_RCU_FAST_NO_HZ=y
> > >>>     CONFIG_PREEMPT_NOTIFIERS=y
> > >>>     CONFIG_PREEMPT_VOLUNTARY=y
> > >>>     CONFIG_PROVE_RCU=y
> > >>>     CONFIG_SPARSE_RCU_POINTER=y
> > >>>
> > >>>>> Again CONFIG_TREE_PREEMPT_RCU is available only if PREEMPT=y. So should
> > >>>>> we enable preemtion and CONFIG_TREE_PREEMPT_RCU=y and try to reproduce
> > >>>>> the issue?
> > >>>>
> > >>>> Please!
> > >>>
> > >>> 2) It appears I can't reproduce with those options enabled (see above).
> > >>>
> > >>>> Polling is fine.  Please see attached for a script to poll at 15-second
> > >>>> intervals.  Please also feel free to adjust, just tell me what you
> > >>>> adjusted.
> > >>>
> > >>> And should I now try to run that script on a config that triggers this
> > >>> issue (such as the config under 1) above)?
> > >>
> > >> Paul, can we see a dmesg from your running system? Perhaps there's some
> > >> dependency on a particular driver or device that makes this easier to
> > >> reproduce.
> > > 
> > > Here you go, please see attached.
> > > 
> > > I should have some additional diagnostics later today Pacific time.
> > 
> > Heh sorry, _other_ Paul :-)
> 
> ;-)
> 
> > You are not seeing this issue, are you?
> 
> No, I am not.
> 
> > As per your earlier comment on sleeping under rcu_read_lock(), I checked
> > everything again and it seems sane. Would that not trigger an
> > immediately schedule-while-atomic in any case, regardless of RCU config?
> 
> This used to be the case, but not anymore.  Hence my additional
> diagnostics.
> 
> The way that this used to happen is that CONFIG_PREEMPT=y and
> CONFIG_TREE_RCU=y would disable preemption across RCU read-side sections.
> Then you would see schedule-while-atomic.
> 
> However, I recently changed the config setup so that CONFIG_PREEMPT=y
> and CONFIG_SMP=y forces CONFIG_TREE_PREEMPT_RCU=y, which avoids that
> diagnostic.  So does CONFIG_SMP=n and CONFIG_TREE_RCU=y, in which case
> preemption is always disabled, so there is no way for the scheduler to
> see that there was anything atomic going on.
> 
> So I am adding a similar diagnostic under CONFIG_PROVE_RCU.
> 
> Rather embarrassing -- I totally missed the fact that this diagnostic
> would be lost.  :-/
> 
> Please see below for a sneak preview of the diagnostic patch.

And see below for one that might have a better chance of working
in modules.

							Thanx, Paul

------------------------------------------------------------------------

rcu: Restore checks for blocking in RCU read-side critical sections

Long ago, using TREE_RCU with PREEMPT would result in "scheduling
while atomic" diagnostics if you blocked in an RCU read-side critical
section.  However, PREEMPT now implies TREE_PREEMPT_RCU, which defeats
this diagnostic.  This commit therefore adds a replacement diagnostic
based on PROVE_RCU.

Because rcu_lockdep_assert() and lockdep_rcu_dereference() are now being
used for things that have nothing to do with rcu_dereference(), rename
lockdep_rcu_dereference() to lockdep_rcu_suspicious() and add a third
argument that is a string indicating what is suspicious.  This third
argument is passed in from a new third argument to rcu_lockdep_assert().
Update all calls to rcu_lockdep_assert() to add an informative third
argument.

Finally, add a pair of rcu_lockdep_assert() calls from within
rcu_note_context_switch(), one complaining if a context switch occurs
in an RCU-bh read-side critical section and another complaining if a
context switch occurs in an RCU-sched read-side critical section.
These are present only if the PROVE_RCU kernel parameter is enabled.

Again, you must enable PROVE_RCU to see these new diagnostics.  But you
are enabling PROVE_RCU to check out new RCU uses in any case, aren't you?

Still-not-yet-signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h
index 4aef1dd..bfa66e7 100644
--- a/include/linux/lockdep.h
+++ b/include/linux/lockdep.h
@@ -545,7 +545,7 @@ do {									\
 #endif
 
 #ifdef CONFIG_PROVE_RCU
-extern void lockdep_rcu_dereference(const char *file, const int line);
+void lockdep_rcu_suspicious(const char *file, const int line, const char *s);
 #endif
 
 #endif /* __LINUX_LOCKDEP_H */
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 58b13f1..fb2933d 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -297,19 +297,20 @@ extern int rcu_my_thread_group_empty(void);
 /**
  * rcu_lockdep_assert - emit lockdep splat if specified condition not met
  * @c: condition to check
+ * @s: informative message
  */
-#define rcu_lockdep_assert(c)						\
+#define rcu_lockdep_assert(c, s)					\
 	do {								\
 		static bool __warned;					\
 		if (debug_lockdep_rcu_enabled() && !__warned && !(c)) {	\
 			__warned = true;				\
-			lockdep_rcu_dereference(__FILE__, __LINE__);	\
+			lockdep_rcu_suspicious(__FILE__, __LINE__, s);	\
 		}							\
 	} while (0)
 
 #else /* #ifdef CONFIG_PROVE_RCU */
 
-#define rcu_lockdep_assert(c) do { } while (0)
+#define rcu_lockdep_assert(c, s) do { } while (0)
 
 #endif /* #else #ifdef CONFIG_PROVE_RCU */
 
@@ -338,14 +339,16 @@ extern int rcu_my_thread_group_empty(void);
 #define __rcu_dereference_check(p, c, space) \
 	({ \
 		typeof(*p) *_________p1 = (typeof(*p)*__force )ACCESS_ONCE(p); \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, "suspicious rcu_dereference_check()" \
+				      " usage"); \
 		rcu_dereference_sparse(p, space); \
 		smp_read_barrier_depends(); \
 		((typeof(*p) __force __kernel *)(_________p1)); \
 	})
 #define __rcu_dereference_protected(p, c, space) \
 	({ \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, "suspicious rcu_dereference_protected()" \
+				      " usage"); \
 		rcu_dereference_sparse(p, space); \
 		((typeof(*p) __force __kernel *)(p)); \
 	})
@@ -359,7 +362,9 @@ extern int rcu_my_thread_group_empty(void);
 #define __rcu_dereference_index_check(p, c) \
 	({ \
 		typeof(p) _________p1 = ACCESS_ONCE(p); \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, \
+				   "suspicious rcu_dereference_index_check()" \
+				   " usage"); \
 		smp_read_barrier_depends(); \
 		(_________p1); \
 	})
diff --git a/kernel/lockdep.c b/kernel/lockdep.c
index 53a6895..9789028 100644
--- a/kernel/lockdep.c
+++ b/kernel/lockdep.c
@@ -3798,7 +3798,7 @@ void lockdep_sys_exit(void)
 	}
 }
 
-void lockdep_rcu_dereference(const char *file, const int line)
+void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
 {
 	struct task_struct *curr = current;
 
@@ -3807,9 +3807,10 @@ void lockdep_rcu_dereference(const char *file, const int line)
 		return;
 #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
 	/* Note: the following can be executed concurrently, so be careful. */
-	printk("\n===================================================\n");
-	printk(  "[ INFO: suspicious rcu_dereference_check() usage. ]\n");
-	printk(  "---------------------------------------------------\n");
+	printk("\n===============================\n");
+	printk(  "[ INFO: suspicious RCU usage. ]\n");
+	printk(  "-------------------------------\n");
+	printk(  "%s\n", s);
 	printk("%s:%d invoked rcu_dereference_check() without protection!\n",
 			file, line);
 	printk("\nother info that might help us debug this:\n\n");
@@ -3818,4 +3819,4 @@ void lockdep_rcu_dereference(const char *file, const int line)
 	printk("\nstack backtrace:\n");
 	dump_stack();
 }
-EXPORT_SYMBOL_GPL(lockdep_rcu_dereference);
+EXPORT_SYMBOL_GPL(lockdep_rcu_suspicious);
diff --git a/kernel/pid.c b/kernel/pid.c
index 57a8346..a7577b3 100644
--- a/kernel/pid.c
+++ b/kernel/pid.c
@@ -419,7 +419,9 @@ EXPORT_SYMBOL(pid_task);
  */
 struct task_struct *find_task_by_pid_ns(pid_t nr, struct pid_namespace *ns)
 {
-	rcu_lockdep_assert(rcu_read_lock_held());
+	rcu_lockdep_assert(rcu_read_lock_held(),
+			   "find_task_by_pid_ns() needs rcu_read_lock()"
+			   " protection");
 	return pid_task(find_pid_ns(nr, ns), PIDTYPE_PID);
 }
 
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b2868ea..ec851db 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -153,6 +153,12 @@ void rcu_bh_qs(int cpu)
  */
 void rcu_note_context_switch(int cpu)
 {
+	rcu_lockdep_assert(!rcu_read_lock_bh_held(),
+			   "Illegal context switch in RCU-bh"
+			   " read-side critical section");
+	rcu_lockdep_assert(!rcu_read_lock_sched_held(),
+			   "Illegal context switch in RCU-sched"
+			   " read-side critical section");
 	rcu_sched_qs(cpu);
 	rcu_preempt_note_context_switch(cpu);
 }
@@ -1675,11 +1681,15 @@ static int rcu_node_kthread(void *arg)
 
 	for (;;) {
 		rnp->node_kthread_status = RCU_KTHREAD_WAITING;
+		printk(KERN_INFO "rcun %p starting wait for work.\n", rnp);
 		rcu_wait(atomic_read(&rnp->wakemask) != 0);
+		printk(KERN_INFO "rcun %p completed wait for work.\n", rnp);
 		rnp->node_kthread_status = RCU_KTHREAD_RUNNING;
 		raw_spin_lock_irqsave(&rnp->lock, flags);
 		mask = atomic_xchg(&rnp->wakemask, 0);
+		printk(KERN_INFO "rcun %p initiating boost.\n", rnp);
 		rcu_initiate_boost(rnp, flags); /* releases rnp->lock. */
+		printk(KERN_INFO "rcun %p completed boost.\n", rnp);
 		for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask >>= 1) {
 			if ((mask & 0x1) == 0)
 				continue;
@@ -1689,10 +1699,12 @@ static int rcu_node_kthread(void *arg)
 				preempt_enable();
 				continue;
 			}
+			printk(KERN_INFO "rcun %p awaking rcuc%d.\n", rnp, cpu);
 			per_cpu(rcu_cpu_has_work, cpu) = 1;
 			sp.sched_priority = RCU_KTHREAD_PRIO;
 			sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
 			preempt_enable();
+			printk(KERN_INFO "rcun %p awakened rcuc%d.\n", rnp, cpu);
 		}
 	}
 	/* NOTREACHED */

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Tue, 2011-05-24 at 11:41 +0200, Jens Axboe wrote:
> Paul, can we see a dmesg from your running system? Perhaps there's some
> dependency on a particular driver or device that makes this easier to
> reproduce.

Sure. What follows is (1K+ lines output of) the dmesg of a session in
which I managed to trigger this issue (three times) with my favorite
testing tool (ie, "mandb -c"). Feel free to prod me for the details of
my system that are not apparent from the flood of information in this
log.


Paul Bolle

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.39-0.local2.fc16.i686 ([...]@t41.thuisdomein) (gcc version 4.6.0 20110509 (Red Hat 4.6.0-7) (GCC) ) #1 SMP Sun May 22 18:11:44 CEST 2011
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
[    0.000000]  BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000d2000 - 00000000000d4000 (reserved)
[    0.000000]  BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 000000007ff60000 (usable)
[    0.000000]  BIOS-e820: 000000007ff60000 - 000000007ff77000 (ACPI data)
[    0.000000]  BIOS-e820: 000000007ff77000 - 000000007ff79000 (ACPI NVS)
[    0.000000]  BIOS-e820: 000000007ff80000 - 0000000080000000 (reserved)
[    0.000000]  BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
[    0.000000] Notice: NX (Execute Disable) protection missing in CPU!
[    0.000000] DMI present.
[    0.000000] DMI: IBM        /       , BIOS 1RETDIWW (3.14 ) 01/20/2005
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[    0.000000] last_pfn = 0x7ff60 max_arch_pfn = 0x100000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-CFFFF write-protect
[    0.000000]   D0000-DBFFF uncachable
[    0.000000]   DC000-DFFFF write-back
[    0.000000]   E0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000 mask F80000000 write-back
[    0.000000]   1 base 07FF80000 mask FFFF80000 uncachable
[    0.000000]   2 disabled
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] PAT not supported by CPU.
[    0.000000] original variable MTRRs
[    0.000000] reg 0, base: 0GB, range: 2GB, type WB
[    0.000000] reg 1, base: 2096640KB, range: 512KB, type UC
[    0.000000] total RAM covered: 2047M
[    0.000000] Found optimal setting for mtrr clean up
[    0.000000]  gran_size: 64K 	chunk_size: 1M 	num_reg: 2  	lose cover RAM: 0G
[    0.000000] New variable MTRRs
[    0.000000] reg 0, base: 0GB, range: 2GB, type WB
[    0.000000] reg 1, base: 2096640KB, range: 512KB, type UC
[    0.000000] initial memory mapped : 0 - 01800000
[    0.000000] Base memory trampoline at [c009b000] 9b000 size 16384
[    0.000000] init_memory_mapping: 0000000000000000-0000000036ffe000
[    0.000000]  0000000000 - 0000400000 page 4k
[    0.000000]  0000400000 - 0036c00000 page 2M
[    0.000000]  0036c00000 - 0036ffe000 page 4k
[    0.000000] kernel direct mapping tables up to 36ffe000 @ 17f9000-1800000
[    0.000000] RAMDISK: 371c8000 - 37ff0000
[    0.000000] Allocated new RAMDISK: 361d6000 - 36ffd59e
[    0.000000] Move RAMDISK from 00000000371c8000 - 0000000037fef59d to 361d6000 - 36ffd59d
[    0.000000] ACPI: RSDP 000f6df0 00024 (v02 IBM   )
[    0.000000] ACPI: XSDT 7ff6a6cd 0004C (v01 IBM    TP-1R    00003140  LTP 00000000)
[    0.000000] ACPI: FACP 7ff6a800 000F4 (v03 IBM    TP-1R    00003140 IBM  00000001)
[    0.000000] ACPI Warning: 32/64X length mismatch in Gpe1Block: 0/32 (20110316/tbfadt-529)
[    0.000000] ACPI Warning: Optional field Gpe1Block has zero address or length: 0x000000000000102C/0x0 (20110316/tbfadt-560)
[    0.000000] ACPI: DSDT 7ff6a9e7 0C4D5 (v01 IBM    TP-1R    00003140 MSFT 0100000E)
[    0.000000] ACPI: FACS 7ff78000 00040
[    0.000000] ACPI: SSDT 7ff6a9b4 00033 (v01 IBM    TP-1R    00003140 MSFT 0100000E)
[    0.000000] ACPI: ECDT 7ff76ebc 00052 (v01 IBM    TP-1R    00003140 IBM  00000001)
[    0.000000] ACPI: TCPA 7ff76f0e 00032 (v01 IBM    TP-1R    00003140 PTL  00000001)
[    0.000000] ACPI: BOOT 7ff76fd8 00028 (v01 IBM    TP-1R    00003140  LTP 00000001)
[    0.000000] 1167MB HIGHMEM available.
[    0.000000] 879MB LOWMEM available.
[    0.000000]   mapped low ram: 0 - 36ffe000
[    0.000000]   low ram: 0 - 36ffe000
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   Normal   0x00001000 -> 0x00036ffe
[    0.000000]   HighMem  0x00036ffe -> 0x0007ff60
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x0007ff60
[    0.000000] On node 0 totalpages: 524015
[    0.000000] free_area_init_node: node 0, pgdat c0aefc40, node_mem_map f51d5200
[    0.000000]   DMA zone: 32 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 3951 pages, LIFO batch:0
[    0.000000]   Normal zone: 1728 pages used for memmap
[    0.000000]   Normal zone: 219454 pages, LIFO batch:31
[    0.000000]   HighMem zone: 2335 pages used for memmap
[    0.000000]   HighMem zone: 296515 pages, LIFO batch:31
[    0.000000] Using APIC driver default
[    0.000000] ACPI: PM-Timer IO Port: 0x1008
[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000d2000
[    0.000000] PM: Registered nosave memory: 00000000000d2000 - 00000000000d4000
[    0.000000] PM: Registered nosave memory: 00000000000d4000 - 00000000000dc000
[    0.000000] PM: Registered nosave memory: 00000000000dc000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7f800000)
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Embedded 332 pages/cpu @f4c00000 s1339200 r0 d20672 u4194304
[    0.000000] pcpu-alloc: s1339200 r0 d20672 u4194304 alloc=1*4194304
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 519920
[    0.000000] Kernel command line: ro root=UUID=283c5a40-0759-4b13-83df-deae686190d1 SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=us-acentos radeon.agpmode=1
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Initializing CPU#0
[    0.000000] allocated 8385792 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] Initializing HighMem for node 0 (00036ffe:0007ff60)
[    0.000000] Memory: 2038648k/2096512k available (4230k kernel code, 57412k reserved, 2929k data, 1880k init, 1195400k highmem)
[    0.000000] virtual kernel memory layout:
[    0.000000]     fixmap  : 0xff575000 - 0xfffff000   (10792 kB)
[    0.000000]     pkmap   : 0xff000000 - 0xff400000   (4096 kB)
[    0.000000]     vmalloc : 0xf77fe000 - 0xfeffe000   ( 120 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xf6ffe000   ( 879 MB)
[    0.000000]       .init : 0xc0afe000 - 0xc0cd4000   (1880 kB)
[    0.000000]       .data : 0xc082193a - 0xc0afddc0   (2929 kB)
[    0.000000]       .text : 0xc0400000 - 0xc082193a   (4230 kB)
[    0.000000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	RCU lockdep checking is enabled.
[    0.000000] 	RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:2304
[    0.000000] CPU 0 irqstacks, hard=f40a0000 soft=f40a2000
[    0.000000] Extended CMOS year: 2000
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled
[    0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.000000] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.000000] ... MAX_LOCK_DEPTH:          48
[    0.000000] ... MAX_LOCKDEP_KEYS:        8191
[    0.000000] ... CLASSHASH_SIZE:          4096
[    0.000000] ... MAX_LOCKDEP_ENTRIES:     16384
[    0.000000] ... MAX_LOCKDEP_CHAINS:      32768
[    0.000000] ... CHAINHASH_SIZE:          16384
[    0.000000]  memory used by lock dependency info: 3823 kB
[    0.000000]  per task-struct memory footprint: 1920 bytes
[    0.000000] ODEBUG: 10 of 10 active objects replaced
[    0.000000] Fast TSC calibration using PIT
[    0.000000] Detected 1694.254 MHz processor.
[    0.002005] Calibrating delay loop (skipped), value calculated using timer frequency.. 3388.50 BogoMIPS (lpj=1694254)
[    0.002242] pid_max: default: 32768 minimum: 301
[    0.003238] Security Framework initialized
[    0.003374] SELinux:  Initializing.
[    0.003655] SELinux:  Starting in permissive mode
[    0.004122] Mount-cache hash table entries: 512
[    0.007412] Initializing cgroup subsys ns
[    0.007544] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.007755] Initializing cgroup subsys cpuacct
[    0.007958] Initializing cgroup subsys memory
[    0.008144] Initializing cgroup subsys devices
[    0.008264] Initializing cgroup subsys freezer
[    0.008382] Initializing cgroup subsys net_cls
[    0.008499] Initializing cgroup subsys blkio
[    0.008656] Initializing cgroup subsys perf_event
[    0.009184] mce: CPU supports 5 MCE banks
[    0.009606] SMP alternatives: switching to UP code
[    0.013398] Freeing SMP alternatives: 12k freed
[    0.013527] ACPI: Core revision 20110316
[    0.080388] ACPI: setting ELCR to 0200 (from 0800)
[    0.081027] ftrace: allocating 23687 entries in 47 pages
[    0.083562] weird, boot CPU (#0) not listed by the BIOS.
[    0.083689] SMP motherboard not detected.
[    0.083803] Local APIC not detected. Using dummy APIC emulation.
[    0.083921] SMP disabled
[    0.084036] Performance Events: 
[    0.084065] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.084291] no hardware sampling interrupt available.
[    0.084408] p6 PMU driver.
[    0.084551] ... version:                0
[    0.084665] ... bit width:              32
[    0.084779] ... generic registers:      2
[    0.084894] ... value mask:             00000000ffffffff
[    0.085002] ... max period:             000000007fffffff
[    0.085118] ... fixed-purpose events:   0
[    0.085233] ... event mask:             0000000000000003
[    0.087267] NMI watchdog disabled (cpu0): not supported (no LAPIC?)
[    0.087560] Brought up 1 CPUs
[    0.087677] Total of 1 processors activated (3388.50 BogoMIPS).
[    0.089372] devtmpfs: initialized
[    0.091358] PM: Registering ACPI NVS region at 7ff77000 (8192 bytes)
[    0.093963] atomic64 test passed for i586+ platform with CX8 and with SSE
[    0.094097] Time: 14:40:00  Date: 05/24/11
[    0.095025] NET: Registered protocol family 16
[    0.097262] ACPI: bus type pci registered
[    0.098299] PCI: PCI BIOS revision 2.10 entry at 0xfd8d6, last bus=8
[    0.098419] PCI: Using configuration type 1 for base access
[    0.117281] bio: create slab <bio-0> at 0
[    0.138249] ACPI: EC: EC description table is found, configuring boot EC
[    0.269290] ACPI: Interpreter enabled
[    0.269412] ACPI: (supports S0 S3 S4 S5)
[    0.269845] ACPI: Using PIC for interrupt routing
[    0.299170] ACPI: Power Resource [PUBS] (on)
[    0.323053] ACPI: EC: GPE = 0x1c, I/O: command/status = 0x66, data = 0x62
[    0.335346] ACPI: ACPI Dock Station Driver: 3 docks/bays found
[    0.335466] HEST: Table not found.
[    0.335583] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
[    0.335907] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.336597] pci_root PNP0A03:00: host bridge window [io  0x0000-0x0cf7] (ignored)
[    0.336602] pci_root PNP0A03:00: host bridge window [io  0x0d00-0xffff] (ignored)
[    0.336606] pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff] (ignored)
[    0.336611] pci_root PNP0A03:00: host bridge window [mem 0x000d4000-0x000d7fff] (ignored)
[    0.336615] pci_root PNP0A03:00: host bridge window [mem 0x000d8000-0x000dbfff] (ignored)
[    0.336620] pci_root PNP0A03:00: host bridge window [mem 0x80000000-0xfebfffff] (ignored)
[    0.336710] pci 0000:00:00.0: [8086:3340] type 0 class 0x000600
[    0.336729] pci 0000:00:00.0: reg 10: [mem 0xd0000000-0xdfffffff pref]
[    0.337033] pci 0000:00:01.0: [8086:3341] type 1 class 0x000604
[    0.337226] pci 0000:00:1d.0: [8086:24c2] type 0 class 0x000c03
[    0.337335] pci 0000:00:1d.0: reg 20: [io  0x1800-0x181f]
[    0.337428] pci 0000:00:1d.1: [8086:24c4] type 0 class 0x000c03
[    0.337536] pci 0000:00:1d.1: reg 20: [io  0x1820-0x183f]
[    0.337628] pci 0000:00:1d.2: [8086:24c7] type 0 class 0x000c03
[    0.337737] pci 0000:00:1d.2: reg 20: [io  0x1840-0x185f]
[    0.337857] pci 0000:00:1d.7: [8086:24cd] type 0 class 0x000c03
[    0.337904] pci 0000:00:1d.7: reg 10: [mem 0xc0000000-0xc00003ff]
[    0.338108] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[    0.338118] pci 0000:00:1d.7: PME# disabled
[    0.338163] pci 0000:00:1e.0: [8086:2448] type 1 class 0x000604
[    0.338277] pci 0000:00:1f.0: [8086:24cc] type 0 class 0x000601
[    0.338444] pci 0000:00:1f.0: quirk: [io  0x1000-0x107f] claimed by ICH4 ACPI/GPIO/TCO
[    0.338659] pci 0000:00:1f.0: quirk: [io  0x1180-0x11bf] claimed by ICH4 GPIO
[    0.338818] pci 0000:00:1f.1: [8086:24ca] type 0 class 0x000101
[    0.338850] pci 0000:00:1f.1: reg 10: [io  0x0000-0x0007]
[    0.338873] pci 0000:00:1f.1: reg 14: [io  0x0000-0x0003]
[    0.338896] pci 0000:00:1f.1: reg 18: [io  0x0000-0x0007]
[    0.338919] pci 0000:00:1f.1: reg 1c: [io  0x0000-0x0003]
[    0.338941] pci 0000:00:1f.1: reg 20: [io  0x1860-0x186f]
[    0.338982] pci 0000:00:1f.1: reg 24: [mem 0x00000000-0x000003ff]
[    0.339062] pci 0000:00:1f.3: [8086:24c3] type 0 class 0x000c05
[    0.339170] pci 0000:00:1f.3: reg 20: [io  0x1880-0x189f]
[    0.339272] pci 0000:00:1f.5: [8086:24c5] type 0 class 0x000401
[    0.339310] pci 0000:00:1f.5: reg 10: [io  0x1c00-0x1cff]
[    0.339331] pci 0000:00:1f.5: reg 14: [io  0x18c0-0x18ff]
[    0.339353] pci 0000:00:1f.5: reg 18: [mem 0xc0000c00-0xc0000dff]
[    0.339375] pci 0000:00:1f.5: reg 1c: [mem 0xc0000800-0xc00008ff]
[    0.339478] pci 0000:00:1f.5: PME# supported from D0 D3hot D3cold
[    0.339486] pci 0000:00:1f.5: PME# disabled
[    0.339531] pci 0000:00:1f.6: [8086:24c6] type 0 class 0x000703
[    0.339569] pci 0000:00:1f.6: reg 10: [io  0x2400-0x24ff]
[    0.339590] pci 0000:00:1f.6: reg 14: [io  0x2000-0x207f]
[    0.339728] pci 0000:00:1f.6: PME# supported from D0 D3hot D3cold
[    0.339737] pci 0000:00:1f.6: PME# disabled
[    0.339813] pci 0000:01:00.0: [1002:4c66] type 0 class 0x000300
[    0.339853] pci 0000:01:00.0: reg 10: [mem 0xe0000000-0xe7ffffff pref]
[    0.339874] pci 0000:01:00.0: reg 14: [io  0x3000-0x30ff]
[    0.339894] pci 0000:01:00.0: reg 18: [mem 0xc0100000-0xc010ffff]
[    0.339984] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
[    0.340044] pci 0000:01:00.0: supports D1 D2
[    0.340162] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    0.340285] pci 0000:00:01.0:   bridge window [io  0x3000-0x3fff]
[    0.340293] pci 0000:00:01.0:   bridge window [mem 0xc0100000-0xc01fffff]
[    0.340301] pci 0000:00:01.0:   bridge window [mem 0xe0000000-0xe7ffffff pref]
[    0.340419] pci 0000:02:00.0: [104c:ac46] type 2 class 0x000607
[    0.340459] pci 0000:02:00.0: reg 10: [mem 0xb0000000-0xb0000fff]
[    0.340511] pci 0000:02:00.0: supports D1 D2
[    0.340515] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.340594] pci 0000:02:00.0: PME# disabled
[    0.340641] pci 0000:02:00.1: [104c:ac46] type 2 class 0x000607
[    0.340682] pci 0000:02:00.1: reg 10: [mem 0xb1000000-0xb1000fff]
[    0.340733] pci 0000:02:00.1: supports D1 D2
[    0.340737] pci 0000:02:00.1: PME# supported from D0 D1 D2 D3hot D3cold
[    0.340747] pci 0000:02:00.1: PME# disabled
[    0.340815] pci 0000:02:01.0: [8086:101e] type 0 class 0x000200
[    0.340855] pci 0000:02:01.0: reg 10: [mem 0xc0220000-0xc023ffff]
[    0.340878] pci 0000:02:01.0: reg 14: [mem 0xc0200000-0xc020ffff]
[    0.340901] pci 0000:02:01.0: reg 18: [io  0x8000-0x803f]
[    0.340998] pci 0000:02:01.0: reg 30: [mem 0x00000000-0x0000ffff pref]
[    0.341050] pci 0000:02:01.0: PME# supported from D0 D3hot D3cold
[    0.341060] pci 0000:02:01.0: PME# disabled
[    0.341107] pci 0000:02:02.0: [168c:1014] type 0 class 0x000200
[    0.341147] pci 0000:02:02.0: reg 10: [mem 0xc0210000-0xc021ffff]
[    0.341426] pci 0000:00:1e.0: PCI bridge to [bus 02-08] (subtractive decode)
[    0.341553] pci 0000:00:1e.0:   bridge window [io  0x4000-0x8fff]
[    0.341563] pci 0000:00:1e.0:   bridge window [mem 0xc0200000-0xcfffffff]
[    0.341572] pci 0000:00:1e.0:   bridge window [mem 0xe8000000-0xefffffff pref]
[    0.341578] pci 0000:00:1e.0:   bridge window [io  0x0000-0xffff] (subtractive decode)
[    0.341583] pci 0000:00:1e.0:   bridge window [mem 0x00000000-0xffffffff] (subtractive decode)
[    0.341804] pci_bus 0000:00: on NUMA node 0
[    0.341818] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    0.342449] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT]
[    0.342720] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
[    0.379526] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11)
[    0.380779] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11)
[    0.382062] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11)
[    0.383308] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11)
[    0.384334] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[    0.385487] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[    0.386631] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[    0.388038] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11)
[    0.389603] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.389998] vgaarb: loaded
[    0.391709] SCSI subsystem initialized
[    0.392252] libata version 3.00 loaded.
[    0.392798] usbcore: registered new interface driver usbfs
[    0.393057] usbcore: registered new interface driver hub
[    0.393351] usbcore: registered new device driver usb
[    0.394155] PCI: Using ACPI for IRQ routing
[    0.394525] PCI: pci_cache_line_size set to 64 bytes
[    0.394920] reserve RAM buffer: 000000000009f000 - 000000000009ffff 
[    0.394939] reserve RAM buffer: 000000007ff60000 - 000000007fffffff 
[    0.396249] NetLabel: Initializing
[    0.396365] NetLabel:  domain hash size = 128
[    0.396479] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.396796] NetLabel:  unlabeled traffic allowed by default
[    0.397086] Switching to clocksource pit
[    0.504412] pnp: PnP ACPI init
[    0.504707] ACPI: bus type pnp registered
[    0.510166] pnp 00:00: [mem 0x00000000-0x0009ffff]
[    0.510171] pnp 00:00: [mem 0x000c0000-0x000c3fff]
[    0.510176] pnp 00:00: [mem 0x000c4000-0x000c7fff]
[    0.510181] pnp 00:00: [mem 0x000c8000-0x000cbfff]
[    0.510185] pnp 00:00: [mem 0x000cc000-0x000cffff]
[    0.510190] pnp 00:00: [mem 0x000d0000-0x000d3fff]
[    0.510195] pnp 00:00: [mem 0x000d4000-0x000d3fff disabled]
[    0.510199] pnp 00:00: [mem 0x000d8000-0x000d7fff disabled]
[    0.510204] pnp 00:00: [mem 0x000dc000-0x000dffff]
[    0.510209] pnp 00:00: [mem 0x000e0000-0x000e3fff]
[    0.510213] pnp 00:00: [mem 0x000e4000-0x000e7fff]
[    0.510218] pnp 00:00: [mem 0x000e8000-0x000ebfff]
[    0.510222] pnp 00:00: [mem 0x000ec000-0x000effff]
[    0.510227] pnp 00:00: [mem 0x000f0000-0x000fffff]
[    0.510231] pnp 00:00: [mem 0x00100000-0x7fffffff]
[    0.510251] pnp 00:00: [mem 0xfec00000-0xffffffff]
[    0.510727] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
[    0.510860] system 00:00: [mem 0x000c0000-0x000c3fff] could not be reserved
[    0.510990] system 00:00: [mem 0x000c4000-0x000c7fff] could not be reserved
[    0.511119] system 00:00: [mem 0x000c8000-0x000cbfff] could not be reserved
[    0.511249] system 00:00: [mem 0x000cc000-0x000cffff] could not be reserved
[    0.511378] system 00:00: [mem 0x000d0000-0x000d3fff] could not be reserved
[    0.511508] system 00:00: [mem 0x000dc000-0x000dffff] could not be reserved
[    0.511669] system 00:00: [mem 0x000e0000-0x000e3fff] could not be reserved
[    0.511799] system 00:00: [mem 0x000e4000-0x000e7fff] could not be reserved
[    0.511929] system 00:00: [mem 0x000e8000-0x000ebfff] could not be reserved
[    0.512058] system 00:00: [mem 0x000ec000-0x000effff] could not be reserved
[    0.512188] system 00:00: [mem 0x000f0000-0x000fffff] could not be reserved
[    0.512317] system 00:00: [mem 0x00100000-0x7fffffff] could not be reserved
[    0.512447] system 00:00: [mem 0xfec00000-0xffffffff] could not be reserved
[    0.512576] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active)
[    0.512743] pnp 00:01: [bus 00-ff]
[    0.512748] pnp 00:01: [io  0x0cf8-0x0cff]
[    0.512753] pnp 00:01: [io  0x0000-0x0cf7 window]
[    0.512758] pnp 00:01: [io  0x0d00-0xffff window]
[    0.512763] pnp 00:01: [mem 0x000a0000-0x000bffff window]
[    0.512768] pnp 00:01: [mem 0x000c0000-0x000c3fff window]
[    0.512772] pnp 00:01: [mem 0x000c4000-0x000c7fff window]
[    0.512777] pnp 00:01: [mem 0x000c8000-0x000cbfff window]
[    0.512782] pnp 00:01: [mem 0x000cc000-0x000cffff window]
[    0.512787] pnp 00:01: [mem 0x000d0000-0x000d3fff window]
[    0.512792] pnp 00:01: [mem 0x000d4000-0x000d7fff window]
[    0.512797] pnp 00:01: [mem 0x000d8000-0x000dbfff window]
[    0.512801] pnp 00:01: [mem 0x000dc000-0x000dffff window]
[    0.512806] pnp 00:01: [mem 0x000e0000-0x000e3fff window]
[    0.512811] pnp 00:01: [mem 0x000e4000-0x000e7fff window]
[    0.512816] pnp 00:01: [mem 0x000e8000-0x000ebfff window]
[    0.512821] pnp 00:01: [mem 0x000ec000-0x000effff window]
[    0.512826] pnp 00:01: [mem 0x80000000-0xfebfffff window]
[    0.513170] pnp 00:01: Plug and Play ACPI device, IDs PNP0a03 (active)
[    0.513953] pnp 00:02: [io  0x0010-0x001f]
[    0.513958] pnp 00:02: [io  0x0090-0x009f]
[    0.513963] pnp 00:02: [io  0x0024-0x0025]
[    0.513967] pnp 00:02: [io  0x0028-0x0029]
[    0.513971] pnp 00:02: [io  0x002c-0x002d]
[    0.513976] pnp 00:02: [io  0x0030-0x0031]
[    0.513980] pnp 00:02: [io  0x0034-0x0035]
[    0.513984] pnp 00:02: [io  0x0038-0x0039]
[    0.513988] pnp 00:02: [io  0x003c-0x003d]
[    0.513993] pnp 00:02: [io  0x00a4-0x00a5]
[    0.513997] pnp 00:02: [io  0x00a8-0x00a9]
[    0.514001] pnp 00:02: [io  0x00ac-0x00ad]
[    0.514006] pnp 00:02: [io  0x00b0-0x00b5]
[    0.514010] pnp 00:02: [io  0x00b8-0x00b9]
[    0.514014] pnp 00:02: [io  0x00bc-0x00bd]
[    0.514018] pnp 00:02: [io  0x0050-0x0053]
[    0.514023] pnp 00:02: [io  0x0072-0x0077]
[    0.514039] pnp 00:02: [io  0x002e-0x002f]
[    0.514044] pnp 00:02: [io  0x1000-0x107f]
[    0.514048] pnp 00:02: [io  0x1180-0x11bf]
[    0.514052] pnp 00:02: [io  0x15e0-0x15ef]
[    0.514057] pnp 00:02: [io  0x1600-0x162f]
[    0.514061] pnp 00:02: [io  0x1632-0x167f]
[    0.514065] pnp 00:02: [io  0x004e-0x004f]
[    0.514069] pnp 00:02: [io  0x1630-0x1631]
[    0.514524] system 00:02: [io  0x1000-0x107f] has been reserved
[    0.514680] system 00:02: [io  0x1180-0x11bf] has been reserved
[    0.514802] system 00:02: [io  0x15e0-0x15ef] has been reserved
[    0.514923] system 00:02: [io  0x1600-0x162f] has been reserved
[    0.515045] system 00:02: [io  0x1632-0x167f] has been reserved
[    0.515166] system 00:02: [io  0x1630-0x1631] has been reserved
[    0.515291] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.515427] pnp 00:03: [io  0x0000-0x000f]
[    0.515432] pnp 00:03: [io  0x0080-0x008f]
[    0.515436] pnp 00:03: [io  0x00c0-0x00df]
[    0.515441] pnp 00:03: [dma 4]
[    0.515691] pnp 00:03: Plug and Play ACPI device, IDs PNP0200 (active)
[    0.515781] pnp 00:04: [io  0x0061]
[    0.516013] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[    0.516091] pnp 00:05: [io  0x00f0]
[    0.516098] pnp 00:05: [irq 13]
[    0.516319] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
[    0.516396] pnp 00:06: [io  0x0070-0x0071]
[    0.516401] pnp 00:06: [irq 8]
[    0.516668] pnp 00:06: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.516745] pnp 00:07: [io  0x0060]
[    0.516750] pnp 00:07: [io  0x0064]
[    0.516754] pnp 00:07: [irq 1]
[    0.516979] pnp 00:07: Plug and Play ACPI device, IDs PNP0303 (active)
[    0.517055] pnp 00:08: [irq 12]
[    0.517303] pnp 00:08: Plug and Play ACPI device, IDs IBM0057 PNP0f13 (active)
[    0.517555] pnp 00:09: [io  0x03f0-0x03f5]
[    0.517560] pnp 00:09: [io  0x03f7]
[    0.517565] pnp 00:09: [irq 6]
[    0.517569] pnp 00:09: [dma 2]
[    0.517878] pnp 00:09: Plug and Play ACPI device, IDs PNP0700 (active)
[    0.518561] pnp 00:0a: [io  0x03f8-0x03ff]
[    0.518566] pnp 00:0a: [irq 4]
[    0.518989] pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.519932] pnp 00:0b: [io  0x03bc-0x03be]
[    0.519938] pnp 00:0b: [irq 7]
[    0.520378] pnp 00:0b: Plug and Play ACPI device, IDs PNP0400 (active)
[    0.521390] pnp 00:0c: [io  0x02f8-0x02ff]
[    0.521395] pnp 00:0c: [irq 3]
[    0.521399] pnp 00:0c: [dma 3]
[    0.521814] pnp 00:0c: Plug and Play ACPI device, IDs IBM0071 PNP0511 (active)
[    0.525763] pnp: PnP ACPI: found 13 devices
[    0.525883] ACPI: ACPI bus type pnp unregistered
[    0.577342] Switching to clocksource acpi_pm
[    0.577667] Switched to NOHz mode on CPU #0
[    0.577905] pci 0000:00:1f.1: BAR 5: assigned [mem 0x80000000-0x800003ff]
[    0.578077] pci 0000:00:1f.1: BAR 5: set to [mem 0x80000000-0x800003ff] (PCI address [0x80000000-0x800003ff])
[    0.578296] pci 0000:01:00.0: BAR 6: assigned [mem 0xc0120000-0xc013ffff pref]
[    0.578505] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    0.578625] pci 0000:00:01.0:   bridge window [io  0x3000-0x3fff]
[    0.578750] pci 0000:00:01.0:   bridge window [mem 0xc0100000-0xc01fffff]
[    0.578875] pci 0000:00:01.0:   bridge window [mem 0xe0000000-0xe7ffffff pref]
[    0.579131] pci 0000:02:00.0: BAR 15: assigned [mem 0xe8000000-0xebffffff pref]
[    0.579339] pci 0000:02:00.0: BAR 16: assigned [mem 0xc4000000-0xc7ffffff]
[    0.579461] pci 0000:02:00.1: BAR 15: assigned [mem 0xec000000-0xefffffff pref]
[    0.579669] pci 0000:02:00.1: BAR 16: assigned [mem 0xc8000000-0xcbffffff]
[    0.579792] pci 0000:02:01.0: BAR 6: assigned [mem 0xc0240000-0xc024ffff pref]
[    0.579999] pci 0000:02:00.0: BAR 13: assigned [io  0x4000-0x40ff]
[    0.580148] pci 0000:02:00.0: BAR 14: assigned [io  0x4400-0x44ff]
[    0.580269] pci 0000:02:00.1: BAR 13: assigned [io  0x4800-0x48ff]
[    0.580389] pci 0000:02:00.1: BAR 14: assigned [io  0x4c00-0x4cff]
[    0.580524] pci 0000:02:00.0: CardBus bridge to [bus 03-06]
[    0.580642] pci 0000:02:00.0:   bridge window [io  0x4000-0x40ff]
[    0.580766] pci 0000:02:00.0:   bridge window [io  0x4400-0x44ff]
[    0.580890] pci 0000:02:00.0:   bridge window [mem 0xe8000000-0xebffffff pref]
[    0.581130] pci 0000:02:00.0:   bridge window [mem 0xc4000000-0xc7ffffff]
[    0.581256] pci 0000:02:00.1: CardBus bridge to [bus 07-07]
[    0.581373] pci 0000:02:00.1:   bridge window [io  0x4800-0x48ff]
[    0.581497] pci 0000:02:00.1:   bridge window [io  0x4c00-0x4cff]
[    0.581621] pci 0000:02:00.1:   bridge window [mem 0xec000000-0xefffffff pref]
[    0.581832] pci 0000:02:00.1:   bridge window [mem 0xc8000000-0xcbffffff]
[    0.581957] pci 0000:00:1e.0: PCI bridge to [bus 02-08]
[    0.582106] pci 0000:00:1e.0:   bridge window [io  0x4000-0x8fff]
[    0.582233] pci 0000:00:1e.0:   bridge window [mem 0xc0200000-0xcfffffff]
[    0.582359] pci 0000:00:1e.0:   bridge window [mem 0xe8000000-0xefffffff pref]
[    0.582608] pci 0000:00:1e.0: setting latency timer to 64
[    0.584509] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
[    0.584630] PCI: setting IRQ 11 as level-triggered
[    0.584638] pci 0000:02:00.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[    0.586586] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[    0.586707] pci 0000:02:00.1: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[    0.586922] pci_bus 0000:00: resource 0 [io  0x0000-0xffff]
[    0.586927] pci_bus 0000:00: resource 1 [mem 0x00000000-0xffffffff]
[    0.586931] pci_bus 0000:01: resource 0 [io  0x3000-0x3fff]
[    0.586935] pci_bus 0000:01: resource 1 [mem 0xc0100000-0xc01fffff]
[    0.586939] pci_bus 0000:01: resource 2 [mem 0xe0000000-0xe7ffffff pref]
[    0.586944] pci_bus 0000:02: resource 0 [io  0x4000-0x8fff]
[    0.586948] pci_bus 0000:02: resource 1 [mem 0xc0200000-0xcfffffff]
[    0.586952] pci_bus 0000:02: resource 2 [mem 0xe8000000-0xefffffff pref]
[    0.586956] pci_bus 0000:02: resource 4 [io  0x0000-0xffff]
[    0.586960] pci_bus 0000:02: resource 5 [mem 0x00000000-0xffffffff]
[    0.586964] pci_bus 0000:03: resource 0 [io  0x4000-0x40ff]
[    0.586968] pci_bus 0000:03: resource 1 [io  0x4400-0x44ff]
[    0.586972] pci_bus 0000:03: resource 2 [mem 0xe8000000-0xebffffff pref]
[    0.586976] pci_bus 0000:03: resource 3 [mem 0xc4000000-0xc7ffffff]
[    0.586980] pci_bus 0000:07: resource 0 [io  0x4800-0x48ff]
[    0.586984] pci_bus 0000:07: resource 1 [io  0x4c00-0x4cff]
[    0.586988] pci_bus 0000:07: resource 2 [mem 0xec000000-0xefffffff pref]
[    0.586992] pci_bus 0000:07: resource 3 [mem 0xc8000000-0xcbffffff]
[    0.587389] NET: Registered protocol family 2
[    0.588259] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.590770] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.593214] TCP bind hash table entries: 65536 (order: 9, 2883584 bytes)
[    0.603717] TCP: Hash tables configured (established 131072 bind 65536)
[    0.603937] TCP reno registered
[    0.604134] UDP hash table entries: 512 (order: 3, 49152 bytes)
[    0.604429] UDP-Lite hash table entries: 512 (order: 3, 49152 bytes)
[    0.605629] NET: Registered protocol family 1
[    0.606087] pci 0000:01:00.0: Boot video device
[    0.606147] PCI: CLS 32 bytes, default 64
[    0.607273] Trying to unpack rootfs image as initramfs...
[    1.548160] Freeing initrd memory: 14496k freed
[    1.634058] DMA-API: preallocated 32768 debug entries
[    1.634185] DMA-API: debugging enabled by kernel config
[    1.634320] Simple Boot Flag at 0x35 set to 0x1
[    1.636370] IBM machine detected. Enabling interrupts during APM calls.
[    1.636498] apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
[    1.636617] apm: overridden by ACPI.
[    1.637805] audit: initializing netlink socket (disabled)
[    1.638089] type=2000 audit(1306248001.636:1): initialized
[    1.677402] highmem bounce pool size: 64 pages
[    1.677539] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[    1.703700] VFS: Disk quotas dquot_6.5.2
[    1.704488] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    1.711362] msgmni has been set to 1675
[    1.712363] SELinux:  Registering netfilter hooks
[    1.714383] cryptomgr_test used greatest stack depth: 6840 bytes left
[    1.714925] cryptomgr_test used greatest stack depth: 6804 bytes left
[    1.715650] cryptomgr_test used greatest stack depth: 6788 bytes left
[    1.715923] alg: No test for stdrng (krng)
[    1.716140] NET: Registered protocol family 38
[    1.716668] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    1.717215] io scheduler noop registered
[    1.717331] io scheduler deadline registered
[    1.717906] io scheduler cfq registered (default)
[    1.718071] start plist test
[    1.720226] end plist test
[    1.720339] list_sort_test: start testing list_sort()
[    1.724905] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    1.725509] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    1.725629] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    1.727784] acpiphp: Slot [1] registered
[    1.730699] ACPI: Deprecated procfs I/F for AC is loaded, please retry with CONFIG_ACPI_PROCFS_POWER cleared
[    1.731632] ACPI: AC Adapter [AC] (on-line)
[    1.733148] input: Lid Switch as /devices/LNXSYSTM:00/device:00/PNP0C0D:00/input/input0
[    1.733754] ACPI: Lid Switch [LID]
[    1.734216] input: Sleep Button as /devices/LNXSYSTM:00/device:00/PNP0C0E:00/input/input1
[    1.734504] ACPI: Sleep Button [SLPB]
[    1.734971] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[    1.735220] ACPI: Power Button [PWRF]
[    1.736834] ACPI: acpi_idle registered with cpuidle
[    1.738044] Marking TSC unstable due to TSC halts in idle
[    1.762581] thermal LNXTHERM:00: registered as thermal_zone0
[    1.762702] ACPI: Thermal Zone [THM0] (56 C)
[    1.762930] ERST: Table is not found!
[    1.763305] isapnp: Scanning for PnP cards...
[    2.122542] isapnp: No Plug & Play device found
[    2.124291] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.126835] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
[    2.127659] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a NS16550A
[    2.135185] 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
[    2.137330] serial 0000:00:1f.6: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[    2.137611] serial 0000:00:1f.6: PCI INT B disabled
[    2.144332] Non-volatile memory driver v1.3
[    2.144450] Linux agpgart interface v0.103
[    2.146339] agpgart-intel 0000:00:00.0: Intel 855PM Chipset
[    2.161965] agpgart-intel 0000:00:00.0: AGP aperture is 256M @ 0xd0000000
[    2.164910] ACPI: Deprecated procfs I/F for battery is loaded, please retry with CONFIG_ACPI_PROCFS_POWER cleared
[    2.165214] ACPI: Battery Slot [BAT0] (battery present)
[    2.178304] brd: module loaded
[    2.184588] loop: module loaded
[    2.185886] ata_piix 0000:00:1f.1: version 2.13
[    2.185913] ata_piix 0000:00:1f.1: enabling device (0005 -> 0007)
[    2.187875] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[    2.187998] ata_piix 0000:00:1f.1: PCI INT A -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[    2.188478] ata_piix 0000:00:1f.1: setting latency timer to 64
[    2.191735] scsi0 : ata_piix
[    2.193217] scsi1 : ata_piix
[    2.199377] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x1860 irq 14
[    2.199499] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x1868 irq 15
[    2.200816] Fixed MDIO Bus: probed
[    2.201643] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.201843] ehci_hcd 0000:00:1d.7: power state changed by ACPI to D0
[    2.201977] ehci_hcd 0000:00:1d.7: power state changed by ACPI to D0
[    2.203890] ACPI: PCI Interrupt Link [LNKH] enabled at IRQ 11
[    2.204043] ehci_hcd 0000:00:1d.7: PCI INT D -> Link[LNKH] -> GSI 11 (level, low) -> IRQ 11
[    2.204494] ehci_hcd 0000:00:1d.7: setting latency timer to 64
[    2.204501] ehci_hcd 0000:00:1d.7: EHCI Host Controller
[    2.206411] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
[    2.206867] ehci_hcd 0000:00:1d.7: debug port 1
[    2.210883] ehci_hcd 0000:00:1d.7: cache line size of 32 is not supported
[    2.210916] ehci_hcd 0000:00:1d.7: irq 11, io mem 0xc0000000
[    2.220071] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
[    2.220604] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    2.220725] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.223660] usb usb1: Product: EHCI Host Controller
[    2.223776] usb usb1: Manufacturer: Linux 2.6.39-0.local2.fc16.i686 ehci_hcd
[    2.223897] usb usb1: SerialNumber: 0000:00:1d.7
[    2.225759] hub 1-0:1.0: USB hub found
[    2.226053] hub 1-0:1.0: 6 ports detected
[    2.227342] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.227576] uhci_hcd: USB Universal Host Controller Interface driver
[    2.228078] uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
[    2.228210] uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
[    2.228343] uhci_hcd 0000:00:1d.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[    2.228578] uhci_hcd 0000:00:1d.0: setting latency timer to 64
[    2.228585] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[    2.229130] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
[    2.229442] uhci_hcd 0000:00:1d.0: irq 11, io base 0x00001800
[    2.229887] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[    2.230039] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.230245] usb usb2: Product: UHCI Host Controller
[    2.230361] usb usb2: Manufacturer: Linux 2.6.39-0.local2.fc16.i686 uhci_hcd
[    2.230481] usb usb2: SerialNumber: 0000:00:1d.0
[    2.231392] hub 2-0:1.0: USB hub found
[    2.231559] hub 2-0:1.0: 2 ports detected
[    2.232285] uhci_hcd 0000:00:1d.1: power state changed by ACPI to D0
[    2.232416] uhci_hcd 0000:00:1d.1: power state changed by ACPI to D0
[    2.234364] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[    2.234486] uhci_hcd 0000:00:1d.1: PCI INT B -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
[    2.234717] uhci_hcd 0000:00:1d.1: setting latency timer to 64
[    2.234724] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[    2.235273] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
[    2.235550] uhci_hcd 0000:00:1d.1: irq 11, io base 0x00001820
[    2.235926] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[    2.236078] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.236283] usb usb3: Product: UHCI Host Controller
[    2.236400] usb usb3: Manufacturer: Linux 2.6.39-0.local2.fc16.i686 uhci_hcd
[    2.236520] usb usb3: SerialNumber: 0000:00:1d.1
[    2.237436] hub 3-0:1.0: USB hub found
[    2.237595] hub 3-0:1.0: 2 ports detected
[    2.238285] uhci_hcd 0000:00:1d.2: PCI INT C -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[    2.238516] uhci_hcd 0000:00:1d.2: setting latency timer to 64
[    2.238523] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[    2.239089] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
[    2.239365] uhci_hcd 0000:00:1d.2: irq 11, io base 0x00001840
[    2.239780] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[    2.239901] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.240138] usb usb4: Product: UHCI Host Controller
[    2.240255] usb usb4: Manufacturer: Linux 2.6.39-0.local2.fc16.i686 uhci_hcd
[    2.240375] usb usb4: SerialNumber: 0000:00:1d.2
[    2.241265] hub 4-0:1.0: USB hub found
[    2.241422] hub 4-0:1.0: 2 ports detected
[    2.242440] usbcore: registered new interface driver usbserial
[    2.242789] USB Serial support registered for generic
[    2.243060] usbcore: registered new interface driver usbserial_generic
[    2.243179] usbserial: USB Serial Driver core
[    2.243368] USB Serial support registered for GSM modem (1-port)
[    2.243629] usbcore: registered new interface driver option
[    2.243746] option: v0.7.2:USB Driver for GSM modems
[    2.244146] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    2.251592] serio: i8042 KBD port at 0x60,0x64 irq 1
[    2.251894] serio: i8042 AUX port at 0x60,0x64 irq 12
[    2.253224] mousedev: PS/2 mouse device common for all mice
[    2.255644] rtc_cmos 00:06: RTC can wake from S4
[    2.256781] rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
[    2.257001] rtc0: alarms up to one month, y3k, 114 bytes nvram
[    2.258848] device-mapper: uevent: version 1.0.3
[    2.260516] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input3
[    2.261654] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) initialised: dm-devel@redhat.com
[    2.263404] cpuidle: using governor ladder
[    2.264705] cpuidle: using governor menu
[    2.268400] EFI Variables Facility v0.08 2004-May-17
[    2.270439] usbcore: registered new interface driver usbhid
[    2.270557] usbhid: USB HID core driver
[    2.273656] ip_tables: (C) 2000-2006 Netfilter Core Team
[    2.273882] TCP cubic registered
[    2.273996] Initializing XFRM netlink socket
[    2.274220] NET: Registered protocol family 17
[    2.274489] Registering the dns_resolver key type
[    2.274976] Using IPI No-Shortcut mode
[    2.275940] PM: Hibernation image not present or could not be loaded.
[    2.276083] registered taskstats version 1
[    2.277207] IMA: No TPM chip found, activating TPM-bypass!
[    2.279915]   Magic number: 11:414:688
[    2.280246] rtc_cmos 00:06: setting system clock to 2011-05-24 14:40:02 UTC (1306248002)
[    2.280634] Initializing network drop monitor service
[    2.354664] ata1.00: HPA detected: current 149285282, native 156301488
[    2.354790] ata1.00: ATA-7: Hitachi HTS541680J9AT00, SB2OA70H, max UDMA/100
[    2.354912] ata1.00: 149285282 sectors, multi 16: LBA48 
[    2.355443] ata2.01: NODEV after polling detection
[    2.358635] ata2.00: ATAPI: UJDA755zDVD/CDRW, 1.20, max UDMA/33
[    2.359392] ata1.00: configured for UDMA/100
[    2.361814] ata2.00: configured for UDMA/33
[    2.362198] scsi 0:0:0:0: Direct-Access     ATA      Hitachi HTS54168 SB2O PQ: 0 ANSI: 5
[    2.365717] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    2.367700] sd 0:0:0:0: [sda] 149285282 512-byte logical blocks: (76.4 GB/71.1 GiB)
[    2.368460] sd 0:0:0:0: [sda] Write Protect is off
[    2.368661] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.369005] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.370164] scsi 1:0:0:0: CD-ROM            MATSHITA UJDA755zDVD/CDRW 1.20 PQ: 0 ANSI: 5
[    2.373211] sda: detected capacity change from 0 to 76434064384
[    2.374182] sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray
[    2.374309] cdrom: Uniform CD-ROM driver Revision: 3.20
[    2.375776] sr 1:0:0:0: Attached scsi CD-ROM sr0
[    2.376285] sr 1:0:0:0: Attached scsi generic sg1 type 5
[    2.393254]  sda: sda1 sda2 sda3
[    2.395836] sd 0:0:0:0: [sda] Attached SCSI disk
[    2.396278] Freeing unused kernel memory: 1880k freed
[    2.399481] Write protecting the kernel text: 4232k
[    2.400329] Write protecting the kernel read-only data: 2072k
[    2.410347] mknod used greatest stack depth: 6604 bytes left
[    2.552893] dracut: dracut-010-1.fc16
[    2.622619] udev[102]: starting version 167
[    2.939484] Synaptics Touchpad, model: 1, fw: 5.9, id: 0x2c6ab1, caps: 0x884793/0x0/0x0
[    2.939714] serio: Synaptics pass-through port at isa0060/serio1/input0
[    2.983922] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input4
[    3.083562] [drm] Initialized drm 1.1.0 20060810
[    3.147914] console_init used greatest stack depth: 6396 bytes left
[    3.221309] [drm] radeon defaulting to kernel modesetting.
[    3.221435] [drm] radeon kernel modesetting enabled.
[    3.222063] radeon 0000:01:00.0: power state changed by ACPI to D0
[    3.222200] radeon 0000:01:00.0: power state changed by ACPI to D0
[    3.222337] radeon 0000:01:00.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[    3.226654] [drm] initializing kernel modesetting (RV250 0x1002:0x4C66).
[    3.226822] [drm] register mmio base: 0xC0100000
[    3.226937] [drm] register mmio size: 65536
[    3.227633] [drm] AGP mode requested: 1
[    3.227754] agpgart-intel 0000:00:00.0: AGP 2.0 bridge
[    3.227925] agpgart-intel 0000:00:00.0: putting AGP V2 device into 1x mode
[    3.228249] radeon 0000:01:00.0: putting AGP V2 device into 1x mode
[    3.228458] radeon 0000:01:00.0: GTT: 256M 0xD0000000 - 0xDFFFFFFF
[    3.228586] radeon 0000:01:00.0: VRAM: 128M 0x00000000E0000000 - 0x00000000E7FFFFFF (32M used)
[    3.228960] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    3.229111] [drm] Driver supports precise vblank timestamp query.
[    3.229286] [drm] radeon: irq initialized.
[    3.229683] [drm] Detected VRAM RAM=128M, BAR=128M
[    3.229806] [drm] RAM width 64bits DDR
[    3.231188] [TTM] Zone  kernel: Available graphics memory: 429818 kiB.
[    3.231308] [TTM] Zone highmem: Available graphics memory: 1027518 kiB.
[    3.231428] [TTM] Initializing pool allocator.
[    3.232788] [drm] radeon: 32M of VRAM memory ready
[    3.232910] [drm] radeon: 256M of GTT memory ready.
[    3.237327] radeon 0000:01:00.0: WB disabled
[    3.239671] [drm] Loading R200 Microcode
[    3.258943] [drm] radeon: ring at 0x00000000D0001000
[    3.259248] [drm] ring test succeeded in 0 usecs
[    3.261058] [drm] radeon: ib pool ready.
[    3.264470] [drm] ib test succeeded in 0 usecs
[    3.274521] [drm] Panel ID String: SXGA+ Single (85MHz)    
[    3.274644] [drm] Panel Size 1400x1050
[    3.286937] [drm] radeon legacy LVDS backlight initialized
[    3.287280] [drm] No TV DAC info found in BIOS
[    3.287738] [drm] Radeon Display Connectors
[    3.287854] [drm] Connector 0:
[    3.287966] [drm]   VGA
[    3.288110] [drm]   DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
[    3.288228] [drm]   Encoders:
[    3.288340] [drm]     CRT1: INTERNAL_DAC1
[    3.288455] [drm] Connector 1:
[    3.288567] [drm]   DVI-D
[    3.288679] [drm]   HPD1
[    3.288792] [drm]   DDC: 0x64 0x64 0x64 0x64 0x64 0x64 0x64 0x64
[    3.288909] [drm]   Encoders:
[    3.289049] [drm]     DFP1: INTERNAL_TMDS1
[    3.289163] [drm] Connector 2:
[    3.289276] [drm]   LVDS
[    3.289387] [drm]   Encoders:
[    3.289499] [drm]     LCD1: INTERNAL_LVDS
[    3.289614] [drm] Connector 3:
[    3.292469] [drm]   S-video
[    3.292581] [drm]   Encoders:
[    3.292694] [drm]     TV1: INTERNAL_DAC2
[    3.314790] [drm] radeon: power management initialized
[    3.355134] [drm] fb mappable at 0xE0040000
[    3.355250] [drm] vram apper at 0xE0000000
[    3.355365] [drm] size 1478656
[    3.355478] [drm] fb depth is 8
[    3.355590] [drm]    pitch is 1408
[    3.357288] fbcon: radeondrmfb (fb0) is primary device
[    3.404378] Console: switching to colour frame buffer device 175x65
[    3.414137] fb0: radeondrmfb frame buffer device
[    3.414188] drm: registered panic notifier
[    3.414396] [drm] Initialized radeon 2.9.0 20080528 for 0000:01:00.0 on minor 0
[    3.416817] modprobe used greatest stack depth: 5184 bytes left
[    3.453474] dracut: Starting plymouth daemon
[    4.029601] input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A03:00/device:03/LNXVIDEO:00/input/input5
[    4.058647] ACPI: Video Device [VID] (multi-head: yes  rom: no  post: no)
[    4.354184] yenta_cardbus 0000:02:00.0: CardBus bridge found [1014:0552]
[    4.354301] yenta_cardbus 0000:02:00.0: Using INTVAL to route CSC interrupts to PCI
[    4.354387] yenta_cardbus 0000:02:00.0: Routing CardBus interrupts to PCI
[    4.354467] yenta_cardbus 0000:02:00.0: TI: mfunc 0x01d21b22, devctl 0x64
[    4.625794] yenta_cardbus 0000:02:00.0: ISA IRQ mask 0x04b8, PCI irq 11
[    4.625879] yenta_cardbus 0000:02:00.0: Socket status: 30000086
[    4.706450] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [io  0x4000-0x8fff]
[    4.706554] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x4000-0x8fff: excluding 0x4000-0x40ff 0x4400-0x44ff 0x4800-0x48ff 0x4c00-0x4cff 0x8000-0x803f
[    4.966404] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [mem 0xc0200000-0xcfffffff]
[    4.969975] pcmcia_socket pcmcia_socket0: cs: memory probe 0xc0200000-0xcfffffff: excluding 0xc0200000-0xc09fffff 0xc3a00000-0xcc1fffff 0xcfa00000-0xd01fffff
[    4.970561] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [mem 0xe8000000-0xefffffff pref]
[    4.970667] pcmcia_socket pcmcia_socket0: cs: memory probe 0xe8000000-0xefffffff: excluding 0xe8000000-0xefffffff
[    4.985854] yenta_cardbus 0000:02:00.1: CardBus bridge found [1014:0552]
[    4.985971] yenta_cardbus 0000:02:00.1: Using INTVAL to route CSC interrupts to PCI
[    4.986197] yenta_cardbus 0000:02:00.1: Routing CardBus interrupts to PCI
[    4.986278] yenta_cardbus 0000:02:00.1: TI: mfunc 0x01d21b22, devctl 0x64
[    5.220457] yenta_cardbus 0000:02:00.1: ISA IRQ mask 0x04b8, PCI irq 11
[    5.220538] yenta_cardbus 0000:02:00.1: Socket status: 30000086
[    5.220760] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [io  0x4000-0x8fff]
[    5.220852] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x4000-0x8fff: excluding 0x4000-0x40ff 0x4400-0x44ff 0x4800-0x48ff 0x4c00-0x4cff 0x8000-0x803f
[    5.254214] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [mem 0xc0200000-0xcfffffff]
[    5.254311] pcmcia_socket pcmcia_socket1: cs: memory probe 0xc0200000-0xcfffffff: excluding 0xc0200000-0xc09fffff 0xc3a00000-0xcc1fffff 0xcfa00000-0xd01fffff
[    5.254842] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [mem 0xe8000000-0xefffffff pref]
[    5.254946] pcmcia_socket pcmcia_socket1: cs: memory probe 0xe8000000-0xefffffff: excluding 0xe8000000-0xefffffff
[    5.399475] dracut: Scanning devices sda2  for LVM volume groups 
[    5.533375] dracut: Reading all physical volumes. This may take a while...
[    5.533777] dracut: Found volume group "VolGroup00" using metadata type lvm2
[    5.710083] psmouse serio2: ID: 10 00 64
[    6.423185] dracut: 3 logical volume(s) in volume group "VolGroup00" now active
[    6.560615] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[    6.657922] dracut: Checking filesystems
[    6.658189] dracut: fsck -T -t noopts=_netdev -A -a
[    6.880526] dracut: F10-PR-i686-Live: clean, 211137/524288 files, 1490030/2097152 blocks
[    6.881332] dracut: Remounting /dev/disk/by-uuid/283c5a40-0759-4b13-83df-deae686190d1 with -o ro,
[    6.908515] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[    6.941847] dracut: Mounted root filesystem /dev/mapper/VolGroup00-rootlv
[    7.193870] dracut: Switching root
[    8.164938] type=1404 audit(1306248008.383:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
[    9.252385] IBM TrackPoint firmware: 0x0e, buttons: 3/3
[    9.493896] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/serio2/input/input6
[   10.617489] SELinux: 2048 avtab hash slots, 218519 rules.
[   10.905161] SELinux: 2048 avtab hash slots, 218519 rules.
[   12.927104] SELinux:  9 users, 13 roles, 3619 types, 191 bools, 1 sens, 1024 cats
[   12.927113] SELinux:  81 classes, 218519 rules
[   12.961633] SELinux:  Completing initialization.
[   12.961641] SELinux:  Setting up existing superblocks.
[   12.962132] SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
[   12.962386] SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
[   12.962700] SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
[   12.962824] SELinux: initialized (dev proc, type proc), uses genfs_contexts
[   12.963372] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[   12.963620] SELinux: initialized (dev devtmpfs, type devtmpfs), uses transition SIDs
[   12.966416] SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
[   12.966642] SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
[   12.983390] SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
[   12.983643] SELinux: initialized (dev anon_inodefs, type anon_inodefs), uses genfs_contexts
[   12.983795] SELinux: initialized (dev devpts, type devpts), uses transition SIDs
[   12.984178] SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses transition SIDs
[   12.984488] SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
[   12.984708] SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
[   12.985159] SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
[   12.985406] SELinux: initialized (dev securityfs, type securityfs), uses genfs_contexts
[   12.985491] SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
[   12.992042] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[   12.992086] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[   12.993947] SELinux: initialized (dev dm-0, type ext4), uses xattr
[   13.029291] type=1403 audit(1306248013.248:3): policy loaded auid=4294967295 ses=4294967295
[   13.788944] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[   13.794161] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.386570] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.390167] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.393724] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.397495] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.401296] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.405062] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.408548] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.412102] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.415915] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.419457] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts
[   14.500400] systemd[1]: systemd 26 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +SYSVINIT +LIBCRYPTSETUP; fedora)
[   14.782292] NET: Registered protocol family 10
[   14.808105] systemd[1]: Set hostname to <t41.thuisdomein>.
[   16.674360] SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
[   16.751478] SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
[   16.805558] SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
[   16.944303] SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
[   16.949482] SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
[   17.747826] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[   17.762901] SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
[   17.873511] EXT4-fs (dm-0): re-mounted. Opts: (null)
[   18.038362] udevd[368]: starting version 170
[   19.922864] intel_rng: FWH not detected
[   20.124069] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x100-0x3af: excluding 0x170-0x177 0x1f0-0x1f7 0x2f8-0x2ff
[   20.151457] iTCO_vendor_support: vendor-support=0
[   20.178246]  0x370-0x377
[   20.243529] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x3e0-0x4ff: excluding 0x3f0-0x3ff 0x4d0-0x4d7
[   20.261958] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x100-0x3af: excluding 0x170-0x177 0x1f0-0x1f7 0x2f8-0x2ff 0x370-0x377
[   20.312882] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.06
[   20.343618] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x820-0x8ff: clean.
[   20.393678] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x3e0-0x4ff: excluding 0x3f0-0x3ff 0x4d0-0x4d7
[   20.443462] iTCO_wdt: Found a ICH4-M TCO device (Version=1, TCOBASE=0x1060)
[   20.477644] pcmcia_socket pcmcia_socket0: cs: IO port probe 0xc00-0xcf7:
[   20.489667] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[   20.497117] e1000: Copyright (c) 1999-2006 Intel Corporation.
[   20.513526]  clean.
[   20.540609] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x820-0x8ff: clean.
[   20.611627] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[   20.643883] pcmcia_socket pcmcia_socket0: cs: memory probe 0x0c0000-0x0fffff: excluding 0xc0000-0xd3fff 0xdc000-0xfffff
[   20.686738] parport_pc 00:0b: reported by Plug and Play ACPI
[   20.692598] pcmcia_socket pcmcia_socket1: cs: IO port probe 0xc00-0xcf7: clean.
[   20.777808] pcmcia_socket pcmcia_socket0: cs: memory probe 0xa0000000-0xa0ffffff: clean.
[   20.803900] parport0: PC-style at 0x3bc, irq 7 [PCSPP,TRISTATE]
[   20.815943] e1000 0000:02:01.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[   20.823595] pcmcia_socket pcmcia_socket1: cs: memory probe 0x0c0000-0x0fffff: excluding 0xc0000-0xd3fff 0xdc000-0xfffff
[   20.852999] input: PC Speaker as /devices/platform/pcspkr/input/input7
[   20.858471] i801_smbus 0000:00:1f.3: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[   20.882768] pcmcia_socket pcmcia_socket0: cs: memory probe 0x60000000-0x60ffffff: excluding 0x60000000-0x60ffffff
[   21.153644] pcmcia_socket pcmcia_socket1: cs: memory probe 0xa0000000-0xa0ffffff: clean.
[   21.215598] pcmcia_socket pcmcia_socket0: cs: IO port probe 0xa00-0xaff: clean.
[   21.249960] pcmcia_socket pcmcia_socket1: cs: memory probe 0x60000000-0x60ffffff: excluding 0x60000000-0x60ffffff
[   21.350644] pcmcia_socket pcmcia_socket1: cs: IO port probe 0xa00-0xaff:
[   21.356905] thinkpad_acpi: ThinkPad ACPI Extras v0.24
[   21.364432] thinkpad_acpi: http://ibm-acpi.sf.net/
[   21.368226] thinkpad_acpi: ThinkPad BIOS 1RETDIWW (3.14 ), EC 1RHT71WW-3.04
[   21.372058] thinkpad_acpi: IBM ThinkPad T41 , model        
[   21.375888] thinkpad_acpi: WARNING: Outdated ThinkPad BIOS/EC firmware
[   21.379744] thinkpad_acpi: WARNING: This firmware may be missing critical bug fixes and/or important features
[   21.399469]  clean.
[   21.415465] cfg80211: Calling CRDA to update world regulatory domain
[   21.653464] thinkpad_acpi: detected a 8-level brightness capable ThinkPad
[   22.161783] microcode: CPU0 sig=0x695, pf=0x20, revision=0x5
[   22.394510] thinkpad_acpi: rfkill switch tpacpi_bluetooth_sw: radio is blocked
[   22.590201] Registered led device: tpacpi::thinklight
[   22.590530] Registered led device: tpacpi::power
[   22.590679] Registered led device: tpacpi::standby
[   22.640170] e1000 0000:02:01.0: eth0: (PCI:33MHz:32-bit) [...]
[   22.644695] e1000 0000:02:01.0: eth0: Intel(R) PRO/1000 Network Connection
[   22.990725] thinkpad_acpi: Console audio control enabled, mode: monitor (read only)
[   23.033475] Intel ICH Modem 0000:00:1f.6: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[   23.121539] Intel ICH Modem 0000:00:1f.6: setting latency timer to 64
[   23.160806] ath5k 0000:02:02.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[   23.263177] ath5k 0000:02:02.0: registered as 'phy0'
[   23.300599] Intel ICH 0000:00:1f.5: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[   23.309178] ppdev: user-space parallel port driver
[   23.335163] Intel ICH 0000:00:1f.5: setting latency timer to 64
[   23.651541] input: ThinkPad Extra Buttons as /devices/platform/thinkpad_acpi/input/input8
[   23.821270] ath: EEPROM regdomain: 0x61
[   23.821275] ath: EEPROM indicates we should expect a direct regpair map
[   23.821281] ath: Country alpha2 being used: 00
[   23.821283] ath: Regpair used: 0x61
[   23.997683] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   24.000371] ath5k 0000:02:02.0: wlan0: Features changed: 0x00004800 -> 0x00004000
[   24.012824] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[   24.049336] Registered led device: ath5k-phy0::rx
[   24.052254] Registered led device: ath5k-phy0::tx
[   24.052331] ath5k phy0: Atheros AR5212 chip found (MAC: 0x56, PHY: 0x41)
[   24.057224] ath5k phy0: RF5111 5GHz radio found (0x17)
[   24.061868] ath5k phy0: RF2111 2GHz radio found (0x23)
[   24.358075] intel8x0_measure_ac97_clock: measured 50429 usecs (2430 samples)
[   24.363103] intel8x0: clocking to 48000
[   24.952191] cfg80211: World regulatory domain updated:
[   24.956926] cfg80211:     (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[   24.961727] cfg80211:     (2402000 KHz - 2472000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[   24.966528] cfg80211:     (2457000 KHz - 2482000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
[   24.971325] cfg80211:     (2474000 KHz - 2494000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
[   24.976096] cfg80211:     (5170000 KHz - 5250000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[   24.980840] cfg80211:     (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[   25.522625] Adding 2097148k swap on /dev/mapper/VolGroup00-swaplv.  Priority:0 extents:1 across:2097148k 
[   25.588427] cfg80211: Calling CRDA for country: NL
[   25.684123] cfg80211: Regulatory domain changed to country: NL
[   25.688822] cfg80211:     (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[   25.693827] cfg80211:     (2402000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm)
[   25.698598] cfg80211:     (5170000 KHz - 5250000 KHz @ 40000 KHz), (N/A, 2000 mBm)
[   25.703256] cfg80211:     (5250000 KHz - 5330000 KHz @ 40000 KHz), (N/A, 2000 mBm)
[   25.707808] cfg80211:     (5490000 KHz - 5710000 KHz @ 40000 KHz), (N/A, 2700 mBm)
[   25.716708] microcode: CPU0 updated to revision 0x7, date = 2004-11-09
[   27.198250] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[   27.251943] SELinux: initialized (dev sda1, type ext4), uses xattr
[   27.353636] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[   27.375528] SELinux: initialized (dev dm-1, type ext4), uses xattr
[   29.765238] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.797718] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.806493] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.824003] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_run_t:s0 for /run: Permission denied
[   29.834244] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_run_t:s0 for /run: Permission denied
[   29.842133] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_run_t:s0 for /run: Permission denied
[   29.850136] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.858158] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.870159] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.878589] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.886355] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.894170] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.902325] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.910164] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.917133] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.922680] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.926326] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.929878] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_run_t:s0 for /run: Permission denied
[   29.971472] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.976732] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_t:s0 for /var: Permission denied
[   29.995363] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:var_run_t:s0 for /run: Permission denied
[   30.005896] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:tmp_t:s0 for /tmp: Permission denied
[   30.026122] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:tmp_t:s0 for /tmp: Permission denied
[   30.030485] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:tmp_t:s0 for /tmp: Permission denied
[   30.043108] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:tmp_t:s0 for /tmp: Permission denied
[   30.052164] systemd-tmpfiles[689]: Failed to set security context system_u:object_r:tmp_t:s0 for /tmp: Permission denied
[   30.479979] /usr/sbin/gpm[703]: *** info [daemon/startup.c(136)]:
[   30.494204] /usr/sbin/gpm[703]: Started gpm successfully. Entered daemon mode.
[   30.568271] avahi-daemon[697]: Found user 'avahi' (UID 497) and group 'avahi' (GID 494).
[   30.577982] avahi-daemon[697]: Successfully dropped root privileges.
[   30.585135] avahi-daemon[697]: avahi-daemon 0.6.30 starting up.
[   31.092713] abrtd[710]: Unrecognized variable 'Kerneloops' in '/etc/abrt/abrt.conf'
[   31.104699] abrtd[710]: Unrecognized variable 'EnabledPlugins' in '/etc/abrt/abrt.conf'
[   31.151115] abrtd[710]: Unrecognized variable 'Python' in '/etc/abrt/abrt.conf'
[   31.175620] abrtd[710]: Unrecognized variable 'Database' in '/etc/abrt/abrt.conf'
[   31.196120] abrtd[710]: Unrecognized variable 'CCpp' in '/etc/abrt/abrt.conf'
[   31.213133] abrtd[710]: Unrecognized variable '120' in '/etc/abrt/abrt.conf'
[   31.256126] abrtd[710]: Unrecognized variable 'OpenGPGPublicKeys' in '/etc/abrt/abrt.conf'
[   32.507682] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   32.569561] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   37.423178] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   37.475333] ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   37.922380] Bluetooth: Core ver 2.16
[   37.929831] NET: Registered protocol family 31
[   37.932724] Bluetooth: HCI device and connection manager initialized
[   37.936139] Bluetooth: HCI socket layer initialized
[   37.938964] Bluetooth: L2CAP socket layer initialized
[   37.950769] Bluetooth: SCO socket layer initialized
[   38.067227] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[   38.070042] Bluetooth: BNEP filters: protocol multicast
[   43.610662] wlan0: direct probe to [...] (try 1/3)
[   43.616220] wlan0: direct probe responded
[   43.616351] wlan0: authenticate with [...] (try 1)
[   43.617952] wlan0: authenticated
[   43.632115] wlan0: associate with [...] (try 1)
[   43.634128] wlan0: RX AssocResp from [...] (capab=0x411 status=0 aid=1)
[   43.634134] wlan0: associated
[   43.645331] ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   46.543500] auditd (1042): /proc/1042/oom_adj is deprecated, please use /proc/1042/oom_score_adj instead.
[   47.643237] P-state transition latency capped at 20 uS
[   54.562091] wlan0: no IPv6 routers present
[  236.276757] cfg80211: Calling CRDA to update world regulatory domain
[  236.332681] cfg80211: World regulatory domain updated:
[  236.336002] cfg80211:     (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[  236.339264] cfg80211:     (2402000 KHz - 2472000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[  236.342505] cfg80211:     (2457000 KHz - 2482000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
[  236.345710] cfg80211:     (2474000 KHz - 2494000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
[  236.348870] cfg80211:     (5170000 KHz - 5250000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[  236.352006] cfg80211:     (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[  236.364384] cfg80211: Calling CRDA for country: NL
[  236.383648] cfg80211: Regulatory domain changed to country: NL
[  236.386816] cfg80211:     (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[  236.390041] cfg80211:     (2402000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm)
[  236.393316] cfg80211:     (5170000 KHz - 5250000 KHz @ 40000 KHz), (N/A, 2000 mBm)
[  236.396524] cfg80211:     (5250000 KHz - 5330000 KHz @ 40000 KHz), (N/A, 2000 mBm)
[  236.399652] cfg80211:     (5490000 KHz - 5710000 KHz @ 40000 KHz), (N/A, 2700 mBm)
[  240.817188] wlan0: authenticate with [...] (try 1)
[  240.819068] wlan0: authenticated
[  240.821645] wlan0: associate with [...] (try 1)
[  240.823698] wlan0: RX ReassocResp from [...] (capab=0x411 status=0 aid=1)
[  240.823704] wlan0: associated
[  462.828090] BUG: unable to handle kernel paging request at 6b6b6b6b
[  462.829020] IP: [<c05ea6a4>] call_for_each_cic+0x29/0x44
[  462.829020] *pde = 00000000 
[  462.829020] Oops: 0000 [#1] SMP 
[  462.829020] last sysfs file: /sys/devices/platform/regulatory.0/uevent
[  462.829020] Modules linked in: cpufreq_ondemand acpi_cpufreq mperf bnep bluetooth ip6t_REJECT nf_conntrack_ipv6 nf_conntrack_netbios_ns nf_conntrack_broadcast nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state ip6table_filter nf_conntrack ip6_tables arc4 ppdev ath5k snd_intel8x0 snd_intel8x0m snd_ac97_codec ath ac97_bus snd_seq mac80211 snd_seq_device microcode snd_pcm thinkpad_acpi snd_timer joydev i2c_i801 pcspkr cfg80211 parport_pc snd_page_alloc snd e1000 soundcore iTCO_wdt parport rfkill iTCO_vendor_support uinput ipv6 yenta_socket video radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
[  462.829020] 
[  462.829020] Pid: 2983, comm: mandb Not tainted 2.6.39-0.local2.fc16.i686 #1 IBM        /       
[  462.829020] EIP: 0060:[<c05ea6a4>] EFLAGS: 00010202 CPU: 0
[  462.829020] EIP is at call_for_each_cic+0x29/0x44
[  462.829020] EAX: 00000001 EBX: 6b6b6b6b ECX: 00000246 EDX: c0aa8a98
[  462.829020] ESI: f6baab00 EDI: c05ea9ab EBP: eda59f18 ESP: eda59f0c
[  462.829020]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  462.829020] Process mandb (pid: 2983, ti=eda58000 task=f2e29530 task.ti=eda58000)
[  462.829020] Stack:
[  462.829020]  f6baab00 f2e29530 f2e29900 eda59f20 c05ea6d1 eda59f2c c05dfd0e f6baab00
[  462.829020]  eda59f40 c05dfd9e f2e29530 00000012 eda59f74 eda59f8c c044149f ef1825dc
[  462.829020]  ef182580 00000009 f2e29530 00000007 f2e29734 f2e29528 f2e29734 00000001
[  462.829020] Call Trace:
[  462.829020]  [<c05ea6d1>] cfq_free_io_context+0x12/0x14
[  462.829020]  [<c05dfd0e>] put_io_context+0x34/0x5c
[  462.829020]  [<c05dfd9e>] exit_io_context+0x68/0x6d
[  462.829020]  [<c044149f>] do_exit+0x63e/0x661
[  462.829020]  [<c04416d9>] do_group_exit+0x63/0x86
[  462.829020]  [<c0441714>] sys_exit_group+0x18/0x18
[  462.829020]  [<c082095f>] sysenter_do_call+0x12/0x38
[  462.829020] Code: 5d c3 55 89 e5 57 56 53 3e 8d 74 26 00 89 c6 89 d7 e8 f3 eb ff ff 8b 5e 4c e8 d5 9d e6 ff 85 c0 74 05 e8 b7 ff ff ff 85 db 74 11 <8b> 03 0f 18 00 90 8d 53 d8 89 f0 ff d7 8b 1b eb dd e8 02 ec ff 
[  462.829020] EIP: [<c05ea6a4>] call_for_each_cic+0x29/0x44 SS:ESP 0068:eda59f0c
[  462.829020] CR2: 000000006b6b6b6b
[  462.963259] ---[ end trace 3109e82224f9ab0c ]---
[  462.967829] Fixing recursive fault but reboot is needed!
[  501.079105] BUG: unable to handle kernel paging request at 6b6b6b6b
[  501.080022] IP: [<c05ea6a4>] call_for_each_cic+0x29/0x44
[  501.080022] *pde = 00000000 
[  501.080022] Oops: 0000 [#2] SMP 
[  501.080022] last sysfs file: /sys/devices/platform/regulatory.0/uevent
[  501.080022] Modules linked in: cpufreq_ondemand acpi_cpufreq mperf bnep bluetooth ip6t_REJECT nf_conntrack_ipv6 nf_conntrack_netbios_ns nf_conntrack_broadcast nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state ip6table_filter nf_conntrack ip6_tables arc4 ppdev ath5k snd_intel8x0 snd_intel8x0m snd_ac97_codec ath ac97_bus snd_seq mac80211 snd_seq_device microcode snd_pcm thinkpad_acpi snd_timer joydev i2c_i801 pcspkr cfg80211 parport_pc snd_page_alloc snd e1000 soundcore iTCO_wdt parport rfkill iTCO_vendor_support uinput ipv6 yenta_socket video radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
[  501.080022] 
[  501.080022] Pid: 9769, comm: mandb Tainted: G      D     2.6.39-0.local2.fc16.i686 #1 IBM        /       
[  501.080022] EIP: 0060:[<c05ea6a4>] EFLAGS: 00010202 CPU: 0
[  501.080022] EIP is at call_for_each_cic+0x29/0x44
[  501.080022] EAX: 00000000 EBX: 6b6b6b6b ECX: 00000286 EDX: 00031179
[  501.080022] ESI: f6baaf00 EDI: c05ea9ab EBP: ef31bf18 ESP: ef31bf0c
[  501.080022]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  501.080022] Process mandb (pid: 9769, ti=ef31a000 task=f65369f0 task.ti=ef31a000)
[  501.080022] Stack:
[  501.080022]  f6baaf00 f65369f0 f6536dc0 ef31bf20 c05ea6d1 ef31bf2c c05dfd0e f6baaf00
[  501.080022]  ef31bf40 c05dfd9e f65369f0 00000012 ef31bf74 ef31bf8c c044149f ef183bdc
[  501.080022]  ef183b80 00000009 f65369f0 00000007 f6536bf4 f65369e8 f6536bf4 00000001
[  501.080022] Call Trace:
[  501.080022]  [<c05ea6d1>] cfq_free_io_context+0x12/0x14
[  501.080022]  [<c05dfd0e>] put_io_context+0x34/0x5c
[  501.080022]  [<c05dfd9e>] exit_io_context+0x68/0x6d
[  501.080022]  [<c044149f>] do_exit+0x63e/0x661
[  501.080022]  [<c04feb69>] ? path_put+0x1a/0x1d
[  501.080022]  [<c04416d9>] do_group_exit+0x63/0x86
[  501.080022]  [<c0441714>] sys_exit_group+0x18/0x18
[  501.080022]  [<c082095f>] sysenter_do_call+0x12/0x38
[  501.080022] Code: 5d c3 55 89 e5 57 56 53 3e 8d 74 26 00 89 c6 89 d7 e8 f3 eb ff ff 8b 5e 4c e8 d5 9d e6 ff 85 c0 74 05 e8 b7 ff ff ff 85 db 74 11 <8b> 03 0f 18 00 90 8d 53 d8 89 f0 ff d7 8b 1b eb dd e8 02 ec ff 
[  501.080022] EIP: [<c05ea6a4>] call_for_each_cic+0x29/0x44 SS:ESP 0068:ef31bf0c
[  501.080022] CR2: 000000006b6b6b6b
[  501.287309] ---[ end trace 3109e82224f9ab0d ]---
[  501.293414] Fixing recursive fault but reboot is needed!
[  519.041105] BUG: unable to handle kernel paging request at 6b6b6b6b
[  519.042023] IP: [<c05ea6a4>] call_for_each_cic+0x29/0x44
[  519.042023] *pde = 00000000 
[  519.042023] Oops: 0000 [#3] SMP 
[  519.042023] last sysfs file: /sys/devices/platform/regulatory.0/uevent
[  519.042023] Modules linked in: cpufreq_ondemand acpi_cpufreq mperf bnep bluetooth ip6t_REJECT nf_conntrack_ipv6 nf_conntrack_netbios_ns nf_conntrack_broadcast nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state ip6table_filter nf_conntrack ip6_tables arc4 ppdev ath5k snd_intel8x0 snd_intel8x0m snd_ac97_codec ath ac97_bus snd_seq mac80211 snd_seq_device microcode snd_pcm thinkpad_acpi snd_timer joydev i2c_i801 pcspkr cfg80211 parport_pc snd_page_alloc snd e1000 soundcore iTCO_wdt parport rfkill iTCO_vendor_support uinput ipv6 yenta_socket video radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
[  519.042023] 
[  519.042023] Pid: 13689, comm: mandb Tainted: G      D     2.6.39-0.local2.fc16.i686 #1 IBM        /       
[  519.042023] EIP: 0060:[<c05ea6a4>] EFLAGS: 00010202 CPU: 0
[  519.042023] EIP is at call_for_each_cic+0x29/0x44
[  519.042023] EAX: 00000000 EBX: 6b6b6b6b ECX: 00000286 EDX: 000357a3
[  519.042023] ESI: edb1d080 EDI: c05ea9ab EBP: eda4bf18 ESP: eda4bf0c
[  519.042023]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  519.042023] Process mandb (pid: 13689, ti=eda4a000 task=ef5369f0 task.ti=eda4a000)
[  519.042023] Stack:
[  519.042023]  edb1d080 ef5369f0 ef536dc0 eda4bf20 c05ea6d1 eda4bf2c c05dfd0e edb1d080
[  519.042023]  eda4bf40 c05dfd9e ef5369f0 00000012 eda4bf74 eda4bf8c c044149f ef1825dc
[  519.042023]  ef182580 00000009 ef5369f0 00000007 ef536bf4 ef5369e8 ef536bf4 00000001
[  519.042023] Call Trace:
[  519.042023]  [<c05ea6d1>] cfq_free_io_context+0x12/0x14
[  519.042023]  [<c05dfd0e>] put_io_context+0x34/0x5c
[  519.042023]  [<c05dfd9e>] exit_io_context+0x68/0x6d
[  519.042023]  [<c044149f>] do_exit+0x63e/0x661
[  519.042023]  [<c04feb69>] ? path_put+0x1a/0x1d
[  519.042023]  [<c04416d9>] do_group_exit+0x63/0x86
[  519.042023]  [<c0441714>] sys_exit_group+0x18/0x18
[  519.042023]  [<c082095f>] sysenter_do_call+0x12/0x38
[  519.042023] Code: 5d c3 55 89 e5 57 56 53 3e 8d 74 26 00 89 c6 89 d7 e8 f3 eb ff ff 8b 5e 4c e8 d5 9d e6 ff 85 c0 74 05 e8 b7 ff ff ff 85 db 74 11 <8b> 03 0f 18 00 90 8d 53 d8 89 f0 ff d7 8b 1b eb dd e8 02 ec ff 
[  519.042023] EIP: [<c05ea6a4>] call_for_each_cic+0x29/0x44 SS:ESP 0068:eda4bf0c
[  519.042023] CR2: 000000006b6b6b6b
[  519.251453] ---[ end trace 3109e82224f9ab0e ]---
[  519.257648] Fixing recursive fault but reboot is needed!
[  648.160192] kworker/u:0 used greatest stack depth: 4708 bytes left

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-05-25 10:28, Paul Bolle wrote:
> On Tue, 2011-05-24 at 11:41 +0200, Jens Axboe wrote:
>> Paul, can we see a dmesg from your running system? Perhaps there's some
>> dependency on a particular driver or device that makes this easier to
>> reproduce.
> 
> Sure. What follows is (1K+ lines output of) the dmesg of a session in
> which I managed to trigger this issue (three times) with my favorite
> testing tool (ie, "mandb -c"). Feel free to prod me for the details of
> my system that are not apparent from the flood of information in this
> log.

Great, thanks Paul. Are you easily able to test a patch out? Having a
test case on your machine that is as easily reproducible as it seems to
be, would be a great place to test ideas out.

I don't think we are dealing with bad RCU usage in CFQ. My gut tells me
that this is related to the merging of cooperating queues. It fits
roughly with the time frame of when this issue started occuring, and
some of that reference logic looks fragile/racy.

So if you _can_ test a patch easily, please try this one. It'll disable
that logic.

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index ab7a9e6..f37e3df 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1827,6 +1827,7 @@ static struct cfq_queue *cfqq_close(struct cfq_data *cfqd,
 static struct cfq_queue *cfq_close_cooperator(struct cfq_data *cfqd,
 					      struct cfq_queue *cur_cfqq)
 {
+#if 0
 	struct cfq_queue *cfqq;
 
 	if (cfq_class_idle(cur_cfqq))
@@ -1870,6 +1871,9 @@ static struct cfq_queue *cfq_close_cooperator(struct cfq_data *cfqd,
 		return NULL;
 
 	return cfqq;
+#else
+	return cfqq;
+#endif
 }
 
 /*

-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Wed, 2011-05-25 at 10:46 +0200, Jens Axboe wrote:
> Are you easily able to test a patch out?

Yes. 

> Having a
> test case on your machine that is as easily reproducible as it seems to
> be, would be a great place to test ideas out.

Maybe I should now also try to build CFQ (and the other schedulers, I
guess) as a module (instead of Fedora's default of builtin). That might
make it both easier and quicker to iterate over a number of ideas.

Does using the schedulers as modules require any handholding (on your
part, I'm afraid) or is it not really different from using any other
kernel module?


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-05-25 11:13, Paul Bolle wrote:
> On Wed, 2011-05-25 at 10:46 +0200, Jens Axboe wrote:
>> Are you easily able to test a patch out?
> 
> Yes. 

Super!

>> Having a
>> test case on your machine that is as easily reproducible as it seems to
>> be, would be a great place to test ideas out.
> 
> Maybe I should now also try to build CFQ (and the other schedulers, I
> guess) as a module (instead of Fedora's default of builtin). That might
> make it both easier and quicker to iterate over a number of ideas.

As most debug patches will likely end up in cfq, yes that makes sense.

> Does using the schedulers as modules require any handholding (on your
> part, I'm afraid) or is it not really different from using any other
> kernel module?

If you build CFQ modular, then all you have to do is switch to using CFQ
before starting your testing:

# echo cfq > /sys/block/sda/queue/scheduler

and switch sda to eg noop before you can remove cfq-iosched again. As
long as things crash, then you wont be able to remove cfq anyway and you
have to reboot to test something new anyway. So the modular nature of
cfq-iosched may not help you all that much in the end.


-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Wed, 2011-05-25 at 11:30 +0200, Jens Axboe wrote:
> If you build CFQ modular, then all you have to do is switch to using CFQ
> before starting your testing:
> 
> # echo cfq > /sys/block/sda/queue/scheduler
> 
> and switch sda to eg noop before you can remove cfq-iosched again. As
> long as things crash, then you wont be able to remove cfq anyway and you
> have to reboot to test something new anyway. So the modular nature of
> cfq-iosched may not help you all that much in the end.

The main advantage I hope for is to reduce build and install time (it's
much quicker to rebuild a module, copy it into /lib/modules, perhaps
upgrade the initramfs, etc., than it is to rebuild an entire kernel, all
its modules, etc.).


Paul

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

> Polling is fine.  Please see attached for a script to poll at 15-second
> intervals.  Please also feel free to adjust, just tell me what you
> adjusted.

0) I have not yet tried this script, but I did have a look at it (to see
what's needed to run it).

1) All debugfs files it polls come from kernel/rcutree_trace.c. That
file is written as a module (and will only be built if
CONFIG_TREE_RCU_TRACE isn't "n"). But in init/Kconfig, the
TREE_RCU_TRACE config entry is set to "def_bool".

2) So shouldn't either the config entry be set to "tristate" or the
module support removed from kernel/rcutree_trace.c?


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Wed, 2011-05-25 at 10:46 +0200, Jens Axboe wrote:
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index ab7a9e6..f37e3df 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -1827,6 +1827,7 @@ static struct cfq_queue *cfqq_close(struct cfq_data *cfqd,
>  static struct cfq_queue *cfq_close_cooperator(struct cfq_data *cfqd,
>  					      struct cfq_queue *cur_cfqq)
>  {
> +#if 0
>  	struct cfq_queue *cfqq;
>  
>  	if (cfq_class_idle(cur_cfqq))
> @@ -1870,6 +1871,9 @@ static struct cfq_queue *cfq_close_cooperator(struct cfq_data *cfqd,
>  		return NULL;
>  
>  	return cfqq;
> +#else
> +	return cfqq;

This won't compile, obviously. Did you perhaps mean

	return NULL;

here?

> +#endif
>  }
>  
>  /*


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-05-25 14:48, Paul Bolle wrote:
> On Wed, 2011-05-25 at 10:46 +0200, Jens Axboe wrote:
>> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
>> index ab7a9e6..f37e3df 100644
>> --- a/block/cfq-iosched.c
>> +++ b/block/cfq-iosched.c
>> @@ -1827,6 +1827,7 @@ static struct cfq_queue *cfqq_close(struct cfq_data *cfqd,
>>  static struct cfq_queue *cfq_close_cooperator(struct cfq_data *cfqd,
>>  					      struct cfq_queue *cur_cfqq)
>>  {
>> +#if 0
>>  	struct cfq_queue *cfqq;
>>  
>>  	if (cfq_class_idle(cur_cfqq))
>> @@ -1870,6 +1871,9 @@ static struct cfq_queue *cfq_close_cooperator(struct cfq_data *cfqd,
>>  		return NULL;
>>  
>>  	return cfqq;
>> +#else
>> +	return cfqq;
> 
> This won't compile, obviously. Did you perhaps mean
> 
> 	return NULL;
> 
> here?

Eh yes, of course, sorry about that. A bit too quick on the send button.

-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Wed, May 25, 2011 at 12:17:16PM +0200, Paul Bolle wrote:
> > Polling is fine.  Please see attached for a script to poll at 15-second
> > intervals.  Please also feel free to adjust, just tell me what you
> > adjusted.
> 
> 0) I have not yet tried this script, but I did have a look at it (to see
> what's needed to run it).
> 
> 1) All debugfs files it polls come from kernel/rcutree_trace.c. That
> file is written as a module (and will only be built if
> CONFIG_TREE_RCU_TRACE isn't "n"). But in init/Kconfig, the
> TREE_RCU_TRACE config entry is set to "def_bool".
> 
> 2) So shouldn't either the config entry be set to "tristate" or the
> module support removed from kernel/rcutree_trace.c?

No.

Just set CONFIG_RCU_TRACE=y and you will have RCU tracing.  Here is why:

CONFIG_TREE_RCU_TRACE is defined as follows:

	config TREE_RCU_TRACE
		def_bool RCU_TRACE && ( TREE_RCU || TREE_PREEMPT_RCU )
		select DEBUG_FS
		help
		  This option provides tracing for the TREE_RCU and
		  TREE_PREEMPT_RCU implementations, permitting Makefile to
		  trivially select kernel/rcutree_trace.c.

In other words, CONFIG_TREE_RCU_TRACE will be set to "y" if CONFIG_RCU_TRACE=y
and if either CONFIG_TREE_RCU=y or CONFIG_TREE_PREEMPT_RCU=y.  If you
are running CONFIG_SMP=y, then we are guaranteed that either
CONFIG_TREE_RCU=y or CONFIG_TREE_PREEMPT_RCU=y, which leaves
CONFIG_RCU_TRACE.

CONFIG_RCU_TRACE is defined as follows:

	config RCU_TRACE
		bool "Enable tracing for RCU"
		help
		  This option provides tracing in RCU which presents stats
		  in debugfs for debugging RCU implementation.

		  Say Y here if you want to enable RCU tracing
		  Say N if you are unsure.

So you can set CONFIG_RCU_TRACE, and if you are running CONFIG_SMP=y,
you will have RCU tracing in your kernel.

							Thanx, Paul

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Wed, 2011-05-25 at 10:46 +0200, Jens Axboe wrote:
> I don't think we are dealing with bad RCU usage in CFQ. My gut tells me
> that this is related to the merging of cooperating queues. It fits
> roughly with the time frame of when this issue started occuring, and
> some of that reference logic looks fragile/racy.
> 
> So if you _can_ test a patch easily, please try this one. It'll disable
> that logic.

I'm sorry, but with that patch (adapted to out previous discussion, so
simply returning NULL) applied I still hit the same Oops:

[  417.526021] Oops: 0000 [#1] SMP 
[  417.526021] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.1/host0/target0:0:0/0:0:0:0/block/sda/queue/scheduler
[  417.526021] Modules linked in: cfq_iosched cpufreq_ondemand acpi_cpufreq mperf bnep bluetooth nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 ip6t_REJECT nf_defrag_ipv4 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ppdev ath5k snd_intel8x0m snd_intel8x0 ath snd_ac97_codec mac80211 microcode ac97_bus snd_seq snd_seq_device snd_pcm cfg80211 joydev pcspkr thinkpad_acpi parport_pc e1000 rfkill parport snd_timer snd iTCO_wdt soundcore snd_page_alloc i2c_i801 iTCO_vendor_support uinput ipv6 yenta_socket video radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
[  417.526021] 
[  417.526021] Pid: 30030, comm: mandb Not tainted 2.6.39-0.local5.fc16.i686 #1 IBM        /       
[  417.526021] EIP: 0060:[<f7efe929>] EFLAGS: 00010202 CPU: 0
[  417.526021] EIP is at call_for_each_cic+0x29/0x44 [cfq_iosched]
[  417.526021] EAX: 00000001 EBX: 6b6b6b6b ECX: 00000246 EDX: c0aa4a98
[  417.526021] ESI: f2f53580 EDI: f7efec18 EBP: edda5f18 ESP: edda5f0c
[  417.526021]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  417.526021] Process mandb (pid: 30030, ti=edda4000 task=f6a1d4c0 task.ti=edda4000)
[  417.526021] Stack:
[  417.526021]  f2f53580 f6a1d4c0 f6a1d890 edda5f20 f7efe956 edda5f2c c05e0506 f2f53580
[  417.526021]  edda5f40 c05e0596 f6a1d4c0 00000012 edda5f74 edda5f8c c044149f f646631c
[  417.526021]  f64662c0 00000009 f6a1d4c0 00000007 f6a1d6c4 f6a1d4b8 f6a1d6c4 00000001
[  417.526021] Call Trace:
[  417.526021]  [<f7efe956>] cfq_free_io_context+0x12/0x14 [cfq_iosched]
[  417.526021]  [<c05e0506>] put_io_context+0x34/0x5c
[  417.526021]  [<c05e0596>] exit_io_context+0x68/0x6d
[  417.526021]  [<c044149f>] do_exit+0x63e/0x661
[  417.526021]  [<c04416d9>] do_group_exit+0x63/0x86
[  417.526021]  [<c0441714>] sys_exit_group+0x18/0x18
[  417.526021]  [<c081cc9f>] sysenter_do_call+0x12/0x38
[  417.526021] Code: 5d c3 55 89 e5 57 56 53 3e 8d 74 26 00 89 c6 89 d7 e8 01 db ff ff 8b 5e 4c e8 50 5b 55 c8 85 c0 74 05 e8 b7 ff ff ff 85 db 74 11 <8b> 03 0f 18 00 90 8d 53 d8 89 f0 ff d7 8b 1b eb dd e8 10 db ff 
[  417.526021] EIP: [<f7efe929>] call_for_each_cic+0x29/0x44 [cfq_iosched] SS:ESP 0068:edda5f0c
[  417.526021] CR2: 000000006b6b6b6b
[  417.717510] ---[ end trace 24344cc07101e5e5 ]---

(That last sysfs file apparently was because I now had to switch to from
deadline to cfq manually.)



Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Wed, 2011-05-25 at 08:33 -0700, Paul E. McKenney wrote:
> On Wed, May 25, 2011 at 12:17:16PM +0200, Paul Bolle wrote:
> > 2) So shouldn't either the config entry be set to "tristate" or the
> > module support removed from kernel/rcutree_trace.c?
> 
> No.
> 
> Just set CONFIG_RCU_TRACE=y and you will have RCU tracing.  Here is why:
> 
>[...]
>
> So you can set CONFIG_RCU_TRACE, and if you are running CONFIG_SMP=y,
> you will have RCU tracing in your kernel.

The point - which I must have expressed poorly - is that there's a
mismatch between the code in kernel/rcutree_trace.c (which allows for
that file to be built as a module) and its kconfig setup (which requires
it to be either builtin or not built at all, since it's boolean).
Anyhow, your explanation makes clear that this kconfig setup is actually
correct and that the module support in kernel/rcutree_trace.c might as
well be removed, since that file can never be part of a module.


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-05-25 19:28, Paul Bolle wrote:
> On Wed, 2011-05-25 at 10:46 +0200, Jens Axboe wrote:
>> I don't think we are dealing with bad RCU usage in CFQ. My gut tells me
>> that this is related to the merging of cooperating queues. It fits
>> roughly with the time frame of when this issue started occuring, and
>> some of that reference logic looks fragile/racy.
>>
>> So if you _can_ test a patch easily, please try this one. It'll disable
>> that logic.
> 
> I'm sorry, but with that patch (adapted to out previous discussion, so
> simply returning NULL) applied I still hit the same Oops:

No worries, it's still a data point. Back to the drawing board...

-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Wed, May 25, 2011 at 07:44:51PM +0200, Paul Bolle wrote:
> On Wed, 2011-05-25 at 08:33 -0700, Paul E. McKenney wrote:
> > On Wed, May 25, 2011 at 12:17:16PM +0200, Paul Bolle wrote:
> > > 2) So shouldn't either the config entry be set to "tristate" or the
> > > module support removed from kernel/rcutree_trace.c?
> > 
> > No.
> > 
> > Just set CONFIG_RCU_TRACE=y and you will have RCU tracing.  Here is why:
> > 
> >[...]
> >
> > So you can set CONFIG_RCU_TRACE, and if you are running CONFIG_SMP=y,
> > you will have RCU tracing in your kernel.
> 
> The point - which I must have expressed poorly - is that there's a
> mismatch between the code in kernel/rcutree_trace.c (which allows for
> that file to be built as a module) and its kconfig setup (which requires
> it to be either builtin or not built at all, since it's boolean).
> Anyhow, your explanation makes clear that this kconfig setup is actually
> correct and that the module support in kernel/rcutree_trace.c might as
> well be removed, since that file can never be part of a module.

Ah -- this is using (perhaps abusing) the module capabilities to
get initialization and cleanup done.

Interesting point, though -- might be worth allowing this to be built
as a module.

Regardless, apologies for missing your point.

							Thanx, Paul

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
> Please see attached for a script to poll at 15-second intervals.

0) The output from collectdebugfs.sh is pasted below. In this case the "CFQ
crash" occurred at "May 26 10:47:07".

1) Big friendly warning: the "CFQ crash" that occurred while running
your script didn't happen in a clean session. Not at all! It actually
happened after (summarized a bit):
- two "CFQ crashes" with the patch for Jens' first idea;
- switching to deadline
- removing cfq_iosched
- recompiling cfq-iosched.ko (to revert Jens' patch)
- installing cfq_iosched.ko
- inserting cfq_iosched
- switching back to cfq again

So if this makes you doubt the validity of the data I collected, which I
guess it could, I'd be happy to try again in a clean session (ie, after
a reboot).

2) Output:
Thu May 26 10:43:29 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=554580  gpnum=554581
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=554582 g=554583 s=2 jfq=2 j=2580 nfqs=246266/nfqsng=0(246266) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6869485 j=2580 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=554585 g=554586 pq=0 pqc=554584 qp=1 dt=7848133/1 dn=0 df=40 of=0
ri=12566 ql=6 b=10 ci=1329766 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7848133/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1818464 qsp=166998 rpq=396272 cbr=691 cng=93496 gpc=94571 gps=0
nf=13093 nn=1220341
rcu_bh:
  0 np=1220341 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1203345
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:43:44 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=559354  gpnum=559355
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=559358 g=559359 s=2 jfq=3 j=6161 nfqs=248249/nfqsng=0(248249) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6884814 j=6161 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=559362 g=559363 pq=0 pqc=559361 qp=1 dt=7854289/1 dn=0 df=40 of=0
ri=12596 ql=4 b=10 ci=1342505 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7854289/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1824426 qsp=167950 rpq=399419 cbr=694 cng=93931 gpc=94850 gps=0
nf=13137 nn=1222395
rcu_bh:
  0 np=1222395 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1205399
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:43:59 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=570906  gpnum=570907
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=570916 g=570917 s=2 jfq=3 j=9d4e nfqs=254774/nfqsng=0(254774) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6900155 j=9d4e nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=570921 g=570922 pq=0 pqc=570920 qp=1 dt=7859875/1 dn=0 df=40 of=0
ri=12606 ql=5 b=10 ci=1368036 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7859875/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1833555 qsp=169800 rpq=405228 cbr=715 cng=94910 gpc=95245 gps=0
nf=13149 nn=1224308
rcu_bh:
  0 np=1224308 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1207312
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:44:15 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=581243  gpnum=581244
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=581249 g=581250 s=2 jfq=2 j=d997 nfqs=260206/nfqsng=0(260206) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6915588 j=d997 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=581252 g=581253 pq=0 pqc=581251 qp=1 dt=7865933/1 dn=0 df=41 of=0
ri=12617 ql=6 b=10 ci=1390017 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7865933/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1842728 qsp=171719 rpq=410839 cbr=725 cng=95777 gpc=95677 gps=0
nf=13165 nn=1226545
rcu_bh:
  0 np=1226545 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1209549
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:44:30 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=593539  gpnum=593540
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=593552 g=593553 s=2 jfq=3 j=1597 nfqs=266219/nfqsng=0(266219) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6930948 j=1597 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=593557 g=593558 pq=0 pqc=593556 qp=1 dt=7870313/1 dn=0 df=42 of=0
ri=12639 ql=6 b=10 ci=1422737 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7870313/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1852849 qsp=173694 rpq=417772 cbr=734 cng=96590 gpc=95945 gps=0
nf=13190 nn=1228618
rcu_bh:
  0 np=1228618 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1211622
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:44:46 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=601780  gpnum=601781
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=601783 g=601784 s=2 jfq=3 j=51b6 nfqs=270338/nfqsng=0(270338) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6946339 j=51b6 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=601786 g=601787 pq=0 pqc=601785 qp=1 dt=7874647/1 dn=0 df=42 of=0
ri=12971 ql=10 b=10 ci=1443173 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7874647/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1863333 qsp=177584 rpq=422912 cbr=749 cng=97218 gpc=96255 gps=0
nf=13619 nn=1232580
rcu_bh:
  0 np=1232580 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1215584
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:45:01 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=609841  gpnum=609842
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=609850 g=609851 s=2 jfq=2 j=8dd5 nfqs=273748/nfqsng=0(273748) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6961730 j=8dd5 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=609853 g=609854 pq=0 pqc=609852 qp=1 dt=7879015/1 dn=0 df=42 of=0
ri=13281 ql=10 b=10 ci=1463867 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7879015/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1874562 qsp=181352 rpq=428551 cbr=761 cng=97850 gpc=96610 gps=0
nf=14023 nn=1236767
rcu_bh:
  0 np=1236767 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1219771
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:45:16 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=617798  gpnum=617799
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=617801 g=617802 s=2 jfq=3 j=c9f3 nfqs=277130/nfqsng=0(277130) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6977120 j=c9f3 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=617812 g=617813 pq=0 pqc=617811 qp=1 dt=7884079/1 dn=0 df=42 of=0
ri=13494 ql=3 b=10 ci=1483693 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7884079/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1886008 qsp=184720 rpq=433990 cbr=770 cng=98567 gpc=97047 gps=0
nf=14341 nn=1241293
rcu_bh:
  0 np=1241293 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1224297
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:45:32 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=625561  gpnum=625562
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=625567 g=625568 s=2 jfq=3 j=5d3 nfqs=281021/nfqsng=0(281021) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-6992448 j=5d3 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=625578 g=625579 pq=0 pqc=625577 qp=1 dt=7888495/1 dn=0 df=42 of=0
ri=13864 ql=6 b=10 ci=1503026 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7888495/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1896278 qsp=188883 rpq=438949 cbr=777 cng=99121 gpc=97317 gps=0
nf=14796 nn=1245318
rcu_bh:
  0 np=1245318 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1228322
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:45:47 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=633312  gpnum=633313
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=633324 g=633325 s=2 jfq=1 j=41a7 nfqs=284960/nfqsng=0(284960) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7007764 j=41a7 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=633336 g=633337 pq=0 pqc=633335 qp=1 dt=7892507/1 dn=0 df=42 of=0
ri=14241 ql=5 b=10 ci=1522479 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7892507/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1906914 qsp=193302 rpq=443888 cbr=783 cng=99667 gpc=97585 gps=0
nf=15282 nn=1249709
rcu_bh:
  0 np=1249709 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1232713
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:46:02 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=641266  gpnum=641267
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=641276 g=641277 s=2 jfq=3 j=7d7d nfqs=288841/nfqsng=0(288841) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7023082 j=7d7d nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=641284 g=641285 pq=0 pqc=641283 qp=1 dt=7896725/1 dn=0 df=42 of=0
ri=14601 ql=1 b=10 ci=1542099 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7896725/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1917497 qsp=197518 rpq=449040 cbr=789 cng=100209 gpc=97845 gps=0
nf=15738 nn=1253876
rcu_bh:
  0 np=1253876 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1236880
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:46:18 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=649448  gpnum=649448
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=649453 g=649454 s=2 jfq=3 j=b955 nfqs=292503/nfqsng=0(292503) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7038402 j=b955 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=649468 g=649469 pq=1 pqc=649468 qp=1 dt=7901627/1 dn=0 df=42 of=0
ri=14855 ql=1 b=10 ci=1562352 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7901627/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1928207 qsp=201146 rpq=454475 cbr=797 cng=100805 gpc=98171 gps=0
nf=16080 nn=1257879
rcu_bh:
  0 np=1257879 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1240883
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:46:33 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=657553  gpnum=657554
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=657559 g=657560 s=2 jfq=3 j=f5e5 nfqs=295995/nfqsng=0(295995) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7053906 j=f5e5 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=657564 g=657565 pq=0 pqc=657563 qp=1 dt=7907237/1 dn=0 df=42 of=0
ri=15111 ql=4 b=10 ci=1582002 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7907237/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1939294 qsp=204834 rpq=459994 cbr=804 cng=101419 gpc=98518 gps=0
nf=16431 nn=1262128
rcu_bh:
  0 np=1262128 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1245132
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:46:49 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=665689  gpnum=665690
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=665692 g=665693 s=2 jfq=3 j=32a6 nfqs=299855/nfqsng=0(299855) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7069459 j=32a6 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=665700 g=665701 pq=0 pqc=665699 qp=1 dt=7911759/1 dn=0 df=42 of=0
ri=15459 ql=5 b=10 ci=1601915 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7911759/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1950196 qsp=209090 rpq=465319 cbr=814 cng=101986 gpc=98794 gps=0
nf=16887 nn=1266396
rcu_bh:
  0 np=1266396 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1249400
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:47:04 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=674282  gpnum=674283
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=674286 g=674287 s=2 jfq=1 j=6ef9 nfqs=303532/nfqsng=0(303532) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7084902 j=6ef9 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=674300 g=674301 pq=0 pqc=674299 qp=1 dt=7916737/1 dn=0 df=42 of=0
ri=15670 ql=5 b=10 ci=1622776 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7916737/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1961236 qsp=212684 rpq=471110 cbr=822 cng=102605 gpc=99130 gps=0
nf=17168 nn=1270401
rcu_bh:
  0 np=1270401 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1253405
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:47:20 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=682249  gpnum=682250
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=682257 g=682258 s=2 jfq=3 j=aad8 nfqs=307404/nfqsng=0(307404) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7100229 j=aad8 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=682260 g=682261 pq=0 pqc=682259 qp=1 dt=7921175/1 dn=0 df=42 of=0
ri=16003 ql=4 b=10 ci=1642246 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7921175/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1971567 qsp=216816 rpq=476234 cbr=830 cng=103139 gpc=99383 gps=0
nf=17588 nn=1274393
rcu_bh:
  0 np=1274393 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1257397
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:47:35 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=689950  gpnum=689951
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=689955 g=689956 s=2 jfq=2 j=e69e nfqs=311206/nfqsng=0(311206) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7115531 j=e69e nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=689964 g=689965 pq=0 pqc=689963 qp=1 dt=7925441/1 dn=0 df=42 of=0
ri=16397 ql=5 b=10 ci=1661655 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7925441/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1982355 qsp=221269 rpq=481281 cbr=835 cng=103658 gpc=99618 gps=0
nf=18095 nn=1278868
rcu_bh:
  0 np=1278868 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1261872
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:47:50 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=698992  gpnum=698993
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=698998 g=698999 s=2 jfq=3 j=2298 nfqs=313944/nfqsng=0(313944) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7130885 j=2298 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=699006 g=699007 pq=0 pqc=699005 qp=1 dt=7928625/1 dn=0 df=42 of=0
ri=16821 ql=4 b=10 ci=1693458 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7928625/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=1994830 qsp=225627 rpq=488590 cbr=837 cng=103889 gpc=99736 gps=0
nf=18667 nn=1283111
rcu_bh:
  0 np=1283111 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1266115
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:48:06 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=711432  gpnum=711433
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=711441 g=711441 s=0 jfq=1 j=5eaf nfqs=314929/nfqsng=0(314929) fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7146268 j=5eaf nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=711443 g=711444 pq=0 pqc=711442 qp=1 dt=7930563/1 dn=0 df=42 of=0
ri=16851 ql=9 b=10 ci=1748424 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7930563/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=2008491 qsp=226909 rpq=500253 cbr=837 cng=104041 gpc=99813 gps=0
nf=18710 nn=1284837
rcu_bh:
  0 np=1284837 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1267841
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost

Thu May 26 10:48:21 CEST 2011
/sys/kernel/debug/rcu/rcugp:
rcu_sched: completed=719902  gpnum=719903
rcu_bh: completed=-114  gpnum=4294967182
/sys/kernel/debug/rcu/rcuhier:
rcu_sched:
c=719907 g=719908 s=2 jfq=2 j=9aa3 nfqs=315817/nfqsng=0(315817) fqlh=0
1/1 ..>.. 0:63 ^0    
1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
rcu_bh:
c=4294967182 g=4294967182 s=0 jfq=-7161616 j=9aa3 nfqs=50/nfqsng=0(50)
fqlh=0
0/1 ..>.. 0:63 ^0    
0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2
0/0 ..>.. 48:63 ^3    
/sys/kernel/debug/rcu/rcudata:
rcu_sched:
  0 c=719915 g=719916 pq=0 pqc=719914 qp=1 dt=7934501/1 dn=0 df=42 of=0
ri=16854 ql=4 b=10 ci=1780327 co=0 ca=0
rcu_bh:
  0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7934501/1 dn=0
df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
/sys/kernel/debug/rcu/rcu_pending:
rcu_sched:
  0 np=2020635 qsp=227848 rpq=507995 cbr=838 cng=104418 gpc=100105 gps=0
nf=18720 nn=1288559
rcu_bh:
  0 np=1288559 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1271563
/sys/kernel/debug/rcu/rcutorture:
no /sys/kernel/debug/rcu/rcutorture
/sys/kernel/debug/rcu/rcuboost:
no /sys/kernel/debug/rcu/rcuboost


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Thu, May 26, 2011 at 11:15:21AM +0200, Paul Bolle wrote:
> On Mon, 2011-05-23 at 08:38 -0700, Paul E. McKenney wrote:
> > Please see attached for a script to poll at 15-second intervals.
> 
> 0) The output from collectdebugfs.sh is pasted below. In this case the "CFQ
> crash" occurred at "May 26 10:47:07".

Thank you for the info!

And please accept my apologies for being so slow to get to it.

> 1) Big friendly warning: the "CFQ crash" that occurred while running
> your script didn't happen in a clean session. Not at all! It actually
> happened after (summarized a bit):
> - two "CFQ crashes" with the patch for Jens' first idea;
> - switching to deadline
> - removing cfq_iosched
> - recompiling cfq-iosched.ko (to revert Jens' patch)
> - installing cfq_iosched.ko
> - inserting cfq_iosched
> - switching back to cfq again
> 
> So if this makes you doubt the validity of the data I collected, which I
> guess it could, I'd be happy to try again in a clean session (ie, after
> a reboot).

Well, given that I cannot make it happen myself, your data is all I
have to go on.

> 2) Output:
> Thu May 26 10:43:29 CEST 2011
> /sys/kernel/debug/rcu/rcugp:
> rcu_sched: completed=554580  gpnum=554581
> rcu_bh: completed=-114  gpnum=4294967182
> /sys/kernel/debug/rcu/rcuhier:
> rcu_sched:
> c=554582 g=554583 s=2 jfq=2 j=2580 nfqs=246266/nfqsng=0(246266) fqlh=0
> 1/1 ..>.. 0:63 ^0    
> 1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    

OK, so this system is running a kernel built with CONFIG_SMP=y, but
with only one CPU online.  You are also running with CONFIG_TREE_RCU=y,
rather than with CONFIG_TREE_PREEMPT=y.  Because you were using plain
RCU (rather than RCU-bh, RCU-sched, or SRCU), the "rcu_sched" entries
are the ones that are pertinent.

In this situation, synchronize_rcu() is by definition a grace period
all by itself -- just the fact that you invoked it suffices.

If you are instead using call_rcu(), things are still pretty simple:
the first context switch is both a quiescent state for the single
CPU, and, by definition a grace period for the whole system.

Jumping ahead to 10:47:

[ . . . ]

> Thu May 26 10:46:49 CEST 2011
> /sys/kernel/debug/rcu/rcugp:
> rcu_sched: completed=665689  gpnum=665690

Here we are waiting for a grace period to complete.  No surprise.

> rcu_bh: completed=-114  gpnum=4294967182
> /sys/kernel/debug/rcu/rcuhier:
> rcu_sched:
> c=665692 g=665693 s=2 jfq=3 j=32a6 nfqs=299855/nfqsng=0(299855) fqlh=0
> 1/1 ..>.. 0:63 ^0    
> 1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    

We have advanced a couple of grace periods in the time it took to print
out the intervening messages, still no surprise.  We are waiting on
CPU 0 to pass through a quiescent state, which is no surprise because
there is no other online CPU.

RCU has gotten to the point of checking for dyntick-idle, and will
set_need_resched() in 3 jiffies if the grace period doesn't end by then.
The jiffies counter is advancing (don't laugh, I have seen cases where it
wasn't advancing).  RCU has felt the need to kick the grace-period process
not quite 300K times since boot, and has succeeded every time it tried.
The former is not too surprising, as it can take more than one kick to
get a grace period going, and this system has passed through more than
600K grace periods since boot.  The latter is not surprising either,
given that there is no other CPU to contend for the lock or to finish
the grace period asynchronously.

> rcu_bh:
> c=4294967182 g=4294967182 s=0 jfq=-7069459 j=32a6 nfqs=50/nfqsng=0(50)
> fqlh=0
> 0/1 ..>.. 0:63 ^0    
> 0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> /sys/kernel/debug/rcu/rcudata:
> rcu_sched:
>   0 c=665700 g=665701 pq=0 pqc=665699 qp=1 dt=7911759/1 dn=0 df=42 of=0 ri=15459 ql=5 b=10 ci=1601915 co=0 ca=0

Eight more grace periods have elapsed.  This CPU has not passed a
quiescent state for this grace period (otherwise RCU would be idle),
and the last time we did pass a quiescent state, it was for the
previous grace period.  RCU is waiting for this CPU to pass through
a quiescent state, and the CPU is not in dyntick-idle mode -- and
has transitioned to and from dyntick-idle mode a -lot-.  It is not
current in an NMI handler, and the grace period has ended due to the
CPU being in dyntick-idle mode 42 times.  (No, the CPU cannot possibly
have been in dyntick-idle mode when RCU was trying to force a quiescent
state, but it could well have visited dyntick-idle mode between two
successive attempts to force a quiescent state for the same grace
period.)

RCU has never intuited a quiescent state due to the CPU being offline
(running with zero CPUs being problematic), and has sent itself
a resched IPI (really just set_need_resched() in the single-CPU case)
about 15K times.  There are five callbacks queued, and there have
been more than 1.6M RCU callbacks invoked since boot, or an average
of 2.4 RCU callbacks per grace period.

> rcu_bh:
>   0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7911759/1 dn=0 df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
> /sys/kernel/debug/rcu/rcu_pending:
> rcu_sched:
>   0 np=1950196 qsp=209090 rpq=465319 cbr=814 cng=101986 gpc=98794 gps=0 nf=16887 nn=1266396

RCU has checked to see if there is work to be done not quite 2M times,
and 1.2M of those times, nothing was needed.  The numbers in between are
the reasons there was something to do: RCU was waiting for a quiescent
state from the CPU, the CPU had passed through a quiescent state and
needed to report this to the RCU core, there were callbacks ready to
invoke, the CPU needed a new grace period to be started, the CPU needed
to be informed of a newly completed grace period, a grace period had
been started by some other CPU (yep, that is why it is zero), and RCU
needed to force a quiescent state.

> rcu_bh:
>   0 np=1266396 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1249400

The stuff below isn't compiled in, so you have CONFIG_RCU_TORTURE_TEST=n.

> /sys/kernel/debug/rcu/rcutorture:
> no /sys/kernel/debug/rcu/rcutorture

Also, because you don't have CONFIG_TREE_PREEMPT_RCU=y, you can't have
CONFIG_RCU_BOOST=y, so the below stuff isn't compiled in either.

> /sys/kernel/debug/rcu/rcuboost:
> no /sys/kernel/debug/rcu/rcuboost
> 
> Thu May 26 10:47:04 CEST 2011
> /sys/kernel/debug/rcu/rcugp:
> rcu_sched: completed=674282  gpnum=674283

15 more seconds, lots more grace periods.

> rcu_bh: completed=-114  gpnum=4294967182
> /sys/kernel/debug/rcu/rcuhier:
> rcu_sched:
> c=674286 g=674287 s=2 jfq=1 j=6ef9 nfqs=303532/nfqsng=0(303532) fqlh=0
> 1/1 ..>.. 0:63 ^0    
> 1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    

Four more grace periods, again waiting on CPU 0 (no one else to be waiting
on), only one jiffy from forcing a quiescent state, and several more
attempts to force have been recorded.

> rcu_bh:
> c=4294967182 g=4294967182 s=0 jfq=-7084902 j=6ef9 nfqs=50/nfqsng=0(50)
> fqlh=0
> 0/1 ..>.. 0:63 ^0    
> 0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> /sys/kernel/debug/rcu/rcudata:
> rcu_sched:
>   0 c=674300 g=674301 pq=0 pqc=674299 qp=1 dt=7916737/1 dn=0 df=42 of=0 ri=15670 ql=5 b=10 ci=1622776 co=0 ca=0

Another four grace periods down.  Again waiting on CPU 0, a few thousand
more dyntick-idle transitions, still not in an NMI handler, and a few
hundred more set_need_resched() calls.

> rcu_bh:
>   0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7916737/1 dn=0 df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
> /sys/kernel/debug/rcu/rcu_pending:
> rcu_sched:
>   0 np=1961236 qsp=212684 rpq=471110 cbr=822 cng=102605 gpc=99130 gps=0 nf=17168 nn=1270401

Still no grace periods started by nonexistent other CPUs, but a bit more
work in every other category over the past 15 seconds.  (Other than responding
to grace periods started by nonexistent other CPUs.)

> rcu_bh:
>   0 np=1270401 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1253405
> /sys/kernel/debug/rcu/rcutorture:
> no /sys/kernel/debug/rcu/rcutorture
> /sys/kernel/debug/rcu/rcuboost:
> no /sys/kernel/debug/rcu/rcuboost
> 
> Thu May 26 10:47:20 CEST 2011
> /sys/kernel/debug/rcu/rcugp:
> rcu_sched: completed=682249  gpnum=682250

15 more seconds, a few thousand more grace periods.  About 500 grace
periods per second, which is quite reasonable on a single-CPU system.

> rcu_bh: completed=-114  gpnum=4294967182
> /sys/kernel/debug/rcu/rcuhier:
> rcu_sched:
> c=682257 g=682258 s=2 jfq=3 j=aad8 nfqs=307404/nfqsng=0(307404) fqlh=0
> 1/1 ..>.. 0:63 ^0    
> 1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    

Eight more grace periods, force quiescent state in three jiffies, and
about 4K attempts to force quiescent states over the past 15 seconds.
This is consistent with HZ=1000.  As always, waiting on CPU 0.

> rcu_bh:
> c=4294967182 g=4294967182 s=0 jfq=-7100229 j=aad8 nfqs=50/nfqsng=0(50)
> fqlh=0
> 0/1 ..>.. 0:63 ^0    
> 0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> /sys/kernel/debug/rcu/rcudata:
> rcu_sched:
>   0 c=682260 g=682261 pq=0 pqc=682259 qp=1 dt=7921175/1 dn=0 df=42 of=0 ri=16003 ql=4 b=10 ci=1642246 co=0 ca=0

Three more grace periods, several thousand more dyntick-idle transitions
over the past 15 seconds.  A few hundred more set_need_resched() calls.
About 1300 RCU callbacks invoked per second over the past 15 seconds.

> rcu_bh:
>   0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7921175/1 dn=0 df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
> /sys/kernel/debug/rcu/rcu_pending:
> rcu_sched:
>   0 np=1971567 qsp=216816 rpq=476234 cbr=830 cng=103139 gpc=99383 gps=0 nf=17588 nn=1274393

Seven more grace periods.  A bit more work in every other category other
than gps.

Looks healthy to me...

							Thanx, Paul

> rcu_bh:
>   0 np=1274393 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1257397
> /sys/kernel/debug/rcu/rcutorture:
> no /sys/kernel/debug/rcu/rcutorture
> /sys/kernel/debug/rcu/rcuboost:
> no /sys/kernel/debug/rcu/rcuboost
> 
> Thu May 26 10:47:35 CEST 2011
> /sys/kernel/debug/rcu/rcugp:
> rcu_sched: completed=689950  gpnum=689951
> rcu_bh: completed=-114  gpnum=4294967182
> /sys/kernel/debug/rcu/rcuhier:
> rcu_sched:
> c=689955 g=689956 s=2 jfq=2 j=e69e nfqs=311206/nfqsng=0(311206) fqlh=0
> 1/1 ..>.. 0:63 ^0    
> 1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> rcu_bh:
> c=4294967182 g=4294967182 s=0 jfq=-7115531 j=e69e nfqs=50/nfqsng=0(50)
> fqlh=0
> 0/1 ..>.. 0:63 ^0    
> 0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> /sys/kernel/debug/rcu/rcudata:
> rcu_sched:
>   0 c=689964 g=689965 pq=0 pqc=689963 qp=1 dt=7925441/1 dn=0 df=42 of=0 ri=16397 ql=5 b=10 ci=1661655 co=0 ca=0
> rcu_bh:
>   0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7925441/1 dn=0 df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
> /sys/kernel/debug/rcu/rcu_pending:
> rcu_sched:
>   0 np=1982355 qsp=221269 rpq=481281 cbr=835 cng=103658 gpc=99618 gps=0 nf=18095 nn=1278868
> rcu_bh:
>   0 np=1278868 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1261872
> /sys/kernel/debug/rcu/rcutorture:
> no /sys/kernel/debug/rcu/rcutorture
> /sys/kernel/debug/rcu/rcuboost:
> no /sys/kernel/debug/rcu/rcuboost
> 
> Thu May 26 10:47:50 CEST 2011
> /sys/kernel/debug/rcu/rcugp:
> rcu_sched: completed=698992  gpnum=698993
> rcu_bh: completed=-114  gpnum=4294967182
> /sys/kernel/debug/rcu/rcuhier:
> rcu_sched:
> c=698998 g=698999 s=2 jfq=3 j=2298 nfqs=313944/nfqsng=0(313944) fqlh=0
> 1/1 ..>.. 0:63 ^0    
> 1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> rcu_bh:
> c=4294967182 g=4294967182 s=0 jfq=-7130885 j=2298 nfqs=50/nfqsng=0(50)
> fqlh=0
> 0/1 ..>.. 0:63 ^0    
> 0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> /sys/kernel/debug/rcu/rcudata:
> rcu_sched:
>   0 c=699006 g=699007 pq=0 pqc=699005 qp=1 dt=7928625/1 dn=0 df=42 of=0 ri=16821 ql=4 b=10 ci=1693458 co=0 ca=0
> rcu_bh:
>   0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7928625/1 dn=0 df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
> /sys/kernel/debug/rcu/rcu_pending:
> rcu_sched:
>   0 np=1994830 qsp=225627 rpq=488590 cbr=837 cng=103889 gpc=99736 gps=0 nf=18667 nn=1283111
> rcu_bh:
>   0 np=1283111 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1266115
> /sys/kernel/debug/rcu/rcutorture:
> no /sys/kernel/debug/rcu/rcutorture
> /sys/kernel/debug/rcu/rcuboost:
> no /sys/kernel/debug/rcu/rcuboost
> 
> Thu May 26 10:48:06 CEST 2011
> /sys/kernel/debug/rcu/rcugp:
> rcu_sched: completed=711432  gpnum=711433
> rcu_bh: completed=-114  gpnum=4294967182
> /sys/kernel/debug/rcu/rcuhier:
> rcu_sched:
> c=711441 g=711441 s=0 jfq=1 j=5eaf nfqs=314929/nfqsng=0(314929) fqlh=0
> 0/1 ..>.. 0:63 ^0    
> 0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> rcu_bh:
> c=4294967182 g=4294967182 s=0 jfq=-7146268 j=5eaf nfqs=50/nfqsng=0(50)
> fqlh=0
> 0/1 ..>.. 0:63 ^0    
> 0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> /sys/kernel/debug/rcu/rcudata:
> rcu_sched:
>   0 c=711443 g=711444 pq=0 pqc=711442 qp=1 dt=7930563/1 dn=0 df=42 of=0 ri=16851 ql=9 b=10 ci=1748424 co=0 ca=0
> rcu_bh:
>   0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7930563/1 dn=0 df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
> /sys/kernel/debug/rcu/rcu_pending:
> rcu_sched:
>   0 np=2008491 qsp=226909 rpq=500253 cbr=837 cng=104041 gpc=99813 gps=0 nf=18710 nn=1284837
> rcu_bh:
>   0 np=1284837 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1267841
> /sys/kernel/debug/rcu/rcutorture:
> no /sys/kernel/debug/rcu/rcutorture
> /sys/kernel/debug/rcu/rcuboost:
> no /sys/kernel/debug/rcu/rcuboost
> 
> Thu May 26 10:48:21 CEST 2011
> /sys/kernel/debug/rcu/rcugp:
> rcu_sched: completed=719902  gpnum=719903
> rcu_bh: completed=-114  gpnum=4294967182
> /sys/kernel/debug/rcu/rcuhier:
> rcu_sched:
> c=719907 g=719908 s=2 jfq=2 j=9aa3 nfqs=315817/nfqsng=0(315817) fqlh=0
> 1/1 ..>.. 0:63 ^0    
> 1/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> rcu_bh:
> c=4294967182 g=4294967182 s=0 jfq=-7161616 j=9aa3 nfqs=50/nfqsng=0(50) fqlh=0
> 0/1 ..>.. 0:63 ^0    
> 0/1 ..>.. 0:15 ^0    0/0 ..>.. 16:31 ^1    0/0 ..>.. 32:47 ^2 0/0 ..>.. 48:63 ^3    
> /sys/kernel/debug/rcu/rcudata:
> rcu_sched:
>   0 c=719915 g=719916 pq=0 pqc=719914 qp=1 dt=7934501/1 dn=0 df=42 of=0 ri=16854 ql=4 b=10 ci=1780327 co=0 ca=0
> rcu_bh:
>   0 c=4294967182 g=4294967182 pq=1 pqc=4294967181 qp=0 dt=7934501/1 dn=0 df=1 of=0 ri=0 ql=0 b=10 ci=1925 co=0 ca=0
> /sys/kernel/debug/rcu/rcu_pending:
> rcu_sched:
>   0 np=2020635 qsp=227848 rpq=507995 cbr=838 cng=104418 gpc=100105 gps=0 nf=18720 nn=1288559
> rcu_bh:
>   0 np=1288559 qsp=0 rpq=16939 cbr=0 cng=33 gpc=24 gps=0 nf=0 nn=1271563
> /sys/kernel/debug/rcu/rcutorture:
> no /sys/kernel/debug/rcu/rcutorture
> /sys/kernel/debug/rcu/rcuboost:
> no /sys/kernel/debug/rcu/rcuboost
> 
> 
> Paul Bolle
> 

Re: Mysterious CFQ crash and RCU

[Vivek Goyal]

On Thu, Jun 02, 2011 at 10:07:24PM -0700, Paul E. McKenney wrote:

[..]
> > Thu May 26 10:47:20 CEST 2011
> > /sys/kernel/debug/rcu/rcugp:
> > rcu_sched: completed=682249  gpnum=682250
> 
> 15 more seconds, a few thousand more grace periods.  About 500 grace
> periods per second, which is quite reasonable on a single-CPU system.

PaulB mentioned that crash happened at May 26 10:47:07. I am wondering
how are we able to sample the data after the crash. I am assuming
that above data gives information only before crash and does not
tell us anything about what happened just before crash. What am I missing.

PaulM, in one of the mails you had mentioned that one could print
context switch id to make sure we did not block in rcu section. Would
you have quick pointer where is context switch id stored. May be
I can write a small patch for PaulB.

Thanks
Vivek

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Fri, Jun 03, 2011 at 09:45:14AM -0400, Vivek Goyal wrote:
> On Thu, Jun 02, 2011 at 10:07:24PM -0700, Paul E. McKenney wrote:
> 
> [..]
> > > Thu May 26 10:47:20 CEST 2011
> > > /sys/kernel/debug/rcu/rcugp:
> > > rcu_sched: completed=682249  gpnum=682250
> > 
> > 15 more seconds, a few thousand more grace periods.  About 500 grace
> > periods per second, which is quite reasonable on a single-CPU system.
> 
> PaulB mentioned that crash happened at May 26 10:47:07. I am wondering
> how are we able to sample the data after the crash. I am assuming
> that above data gives information only before crash and does not
> tell us anything about what happened just before crash. What am I missing.
> 
> PaulM, in one of the mails you had mentioned that one could print
> context switch id to make sure we did not block in rcu section. Would
> you have quick pointer where is context switch id stored. May be
> I can write a small patch for PaulB.

>From what I can see, the task_struct nvcsw and nivcsw fields should do
it, though I am not seeing where these are incremented.

So if these don't do what you need, the following (untested but trivial)
patch will provide an rcu_switch_count in the task structure.

							Thanx, Paul

------------------------------------------------------------------------

rcu: add diagnostic per-task context-switch count

Note that this is also incremented by softirqs.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 2a8621c..5ef22e2 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1261,6 +1261,7 @@ struct task_struct {
 #ifdef CONFIG_RCU_BOOST
 	struct rt_mutex *rcu_boost_mutex;
 #endif /* #ifdef CONFIG_RCU_BOOST */
+	unsigned long rcu_switch_count;
 
 #if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
 	struct sched_info sched_info;
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 89419ff..080c6eb 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -154,6 +154,7 @@ void rcu_bh_qs(int cpu)
  */
 void rcu_note_context_switch(int cpu)
 {
+	current->rcu_switch_count++;
 	rcu_sched_qs(cpu);
 	rcu_preempt_note_context_switch(cpu);
 }

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Fri, Jun 03, 2011 at 08:33:44AM -0700, Paul E. McKenney wrote:
> On Fri, Jun 03, 2011 at 09:45:14AM -0400, Vivek Goyal wrote:
> > On Thu, Jun 02, 2011 at 10:07:24PM -0700, Paul E. McKenney wrote:
> > 
> > [..]
> > > > Thu May 26 10:47:20 CEST 2011
> > > > /sys/kernel/debug/rcu/rcugp:
> > > > rcu_sched: completed=682249  gpnum=682250
> > > 
> > > 15 more seconds, a few thousand more grace periods.  About 500 grace
> > > periods per second, which is quite reasonable on a single-CPU system.
> > 
> > PaulB mentioned that crash happened at May 26 10:47:07. I am wondering
> > how are we able to sample the data after the crash. I am assuming
> > that above data gives information only before crash and does not
> > tell us anything about what happened just before crash. What am I missing.
> > 
> > PaulM, in one of the mails you had mentioned that one could print
> > context switch id to make sure we did not block in rcu section. Would
> > you have quick pointer where is context switch id stored. May be
> > I can write a small patch for PaulB.
> 
> From what I can see, the task_struct nvcsw and nivcsw fields should do
> it, though I am not seeing where these are incremented.

And Milton Miller pointed out that schedule() takes the address of
these fields in a local pointer "switch_count" and then increments
through this pointer.  So there you have it.

> So if these don't do what you need, the following (untested but trivial)
> patch will provide an rcu_switch_count in the task structure.

And he also noted that (nvcsw+nivcsw) is incremented only when context
switches from one task to another, while my patch below would count
every call to schedule(), whether or not a context switch occurs.

Either approach should work.

							Thanx, Paul

> ------------------------------------------------------------------------
> 
> rcu: add diagnostic per-task context-switch count
> 
> Note that this is also incremented by softirqs.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 2a8621c..5ef22e2 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -1261,6 +1261,7 @@ struct task_struct {
>  #ifdef CONFIG_RCU_BOOST
>  	struct rt_mutex *rcu_boost_mutex;
>  #endif /* #ifdef CONFIG_RCU_BOOST */
> +	unsigned long rcu_switch_count;
>  
>  #if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
>  	struct sched_info sched_info;
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 89419ff..080c6eb 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -154,6 +154,7 @@ void rcu_bh_qs(int cpu)
>   */
>  void rcu_note_context_switch(int cpu)
>  {
> +	current->rcu_switch_count++;
>  	rcu_sched_qs(cpu);
>  	rcu_preempt_note_context_switch(cpu);
>  }

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Fri, 2011-06-03 at 09:45 -0400, Vivek Goyal wrote:
> PaulB mentioned that crash happened at May 26 10:47:07. I am wondering
> how are we able to sample the data after the crash. I am assuming
> that above data gives information only before crash and does not
> tell us anything about what happened just before crash. What am I missing.

Well, what you called a "CFQ crash" is an Oops (apparently generated by
arch/x86/mm/fault.c:show_fault_oops()). But the traces I posted at the
bugzilla.redhat.com issue for this always end with: "Fixing recursive
fault but reboot is needed" (see kernel/exit.c:do_exit()). At that point
the system is still running.

Perhaps you run with panic_on_oops on by default (rumor has it that's an
RHEL default) which might make the result of this Oops surprising.
Anyhow, it turns out that my system is suspiciously happy after the
process(es) causing this Oops has (have) finished. See the big friendly
warning I put on top of the message in which I pasted the output of
Paul's script:

> 1) Big friendly warning: the "CFQ crash" that occurred while running
> your script didn't happen in a clean session. Not at all! It actually
> happened after (summarized a bit):
> - two "CFQ crashes" with the patch for Jens' first idea;
> - switching to deadline
> - removing cfq_iosched
> - recompiling cfq-iosched.ko (to revert Jens' patch)
> - installing cfq_iosched.ko
> - inserting cfq_iosched
> - switching back to cfq again

(Yes, putting "CFQ crash" in quotes there was a bit of legalese on my
part.)


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Thu, 2011-06-02 at 22:07 -0700, Paul E. McKenney wrote:
> And please accept my apologies for being so slow to get to it.

Thanks, but it was just a week (ie, quite a quick response by my
standards).

> Looks healthy to me...

How should I understand that? Something like: "As far as this hlist is
used with RCU everything seems OK. Perhaps something is messing with the
entries of this hlist outside of RCU. Perhaps additional locking is
needed."


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Sat, Jun 04, 2011 at 02:50:17PM +0200, Paul Bolle wrote:
> On Thu, 2011-06-02 at 22:07 -0700, Paul E. McKenney wrote:
> > And please accept my apologies for being so slow to get to it.
> 
> Thanks, but it was just a week (ie, quite a quick response by my
> standards).
> 
> > Looks healthy to me...
> 
> How should I understand that? Something like: "As far as this hlist is
> used with RCU everything seems OK. Perhaps something is messing with the
> entries of this hlist outside of RCU. Perhaps additional locking is
> needed."

More like "based on these diagnostics, I see no evidence of the RCU
implementation misbehaving."  Which is of course different than "I can
prove that the RCU implementation is not misbehaving".  That said, the
fact that you are running on a single CPU makes it hard for me to see
any latitude for RCU-implementation misbehavior.

Clearly something is wrong somewhere.  Given the fact that on a single-CPU
system, synchronize_rcu() is a no-op, and given that you weren't able
to reproduce with CONFIG_TREE_PREEMPT_RCU=y, my guess is that there is
a synchronize_rcu() that occasionally (illegally) gets executed within
an RCU read-side critical section.

							Thanx, Paul

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Sat, 2011-06-04 at 09:03 -0700, Paul E. McKenney wrote:
> More like "based on these diagnostics, I see no evidence of the RCU
> implementation misbehaving."  Which is of course different than "I can
> prove that the RCU implementation is not misbehaving".  That said, the
> fact that you are running on a single CPU makes it hard for me to see
> any latitude for RCU-implementation misbehavior.
> 
> Clearly something is wrong somewhere.

Yes.

> Given the fact that on a single-CPU
> system, synchronize_rcu() is a no-op, and given that you weren't able
> to reproduce with CONFIG_TREE_PREEMPT_RCU=y, my guess is that there is
> a synchronize_rcu() that occasionally (illegally) gets executed within
> an RCU read-side critical section.

I think I finally found it!

The culprit seems to be io_context.ioc_data (not the most clear of
names!). It seems to be a single entry "last-hit cache" of an hlist
called cic_list. (There are three, subtly different, cic_lists in the
CFQ code!) It is not entirely clear, but that last-hit cache can get out
of sync with the hlist it is supposed to cache. My guess it that every
now and then a member of the hlist gets deleted while it's still in that
(single entry) cache. If it then gets retrieved from that cache it
already points to poisoned memory. For some strange reason this only
results in an Oops if one or more debugging options are set (as are set
in the Fedora Rawhide non-stable kernels that I ran into this). I have
no clue whatsoever, why that is ...

Anyhow, after ripping out ioc_data this bug seems to have disappeared!
Jens, Vivek, could you please have a look at this? In the mean time I
hope to pinpoint this issue and draft a small patch to really solve it
(ie, not by simply ripping out ioc_data).


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul E. McKenney]

On Sun, Jun 05, 2011 at 12:48:05AM +0200, Paul Bolle wrote:
> On Sat, 2011-06-04 at 09:03 -0700, Paul E. McKenney wrote:
> > More like "based on these diagnostics, I see no evidence of the RCU
> > implementation misbehaving."  Which is of course different than "I can
> > prove that the RCU implementation is not misbehaving".  That said, the
> > fact that you are running on a single CPU makes it hard for me to see
> > any latitude for RCU-implementation misbehavior.
> > 
> > Clearly something is wrong somewhere.
> 
> Yes.
> 
> > Given the fact that on a single-CPU
> > system, synchronize_rcu() is a no-op, and given that you weren't able
> > to reproduce with CONFIG_TREE_PREEMPT_RCU=y, my guess is that there is
> > a synchronize_rcu() that occasionally (illegally) gets executed within
> > an RCU read-side critical section.
> 
> I think I finally found it!

Very cool!!!

> The culprit seems to be io_context.ioc_data (not the most clear of
> names!). It seems to be a single entry "last-hit cache" of an hlist
> called cic_list. (There are three, subtly different, cic_lists in the
> CFQ code!) It is not entirely clear, but that last-hit cache can get out
> of sync with the hlist it is supposed to cache. My guess it that every
> now and then a member of the hlist gets deleted while it's still in that
> (single entry) cache. If it then gets retrieved from that cache it
> already points to poisoned memory. For some strange reason this only
> results in an Oops if one or more debugging options are set (as are set
> in the Fedora Rawhide non-stable kernels that I ran into this). I have
> no clue whatsoever, why that is ...

Oooh...  If I understand you correctly, this is exactly analogous to my
first-ever RCU bug way back in the early 90s.  This was a hashed version
of a BSD-style routing table, where each hash bucket maintained a pointer
to the last routing entry used.  When I deleted a routing entry, I forgot
to check the last-entry pointer.

The amazing thing is that the guy who found and fixed this bug generated
a correct fix despite having absolutely no idea what RCU was or how it
worked.  All he knew was that if he waited for the sun to reach Oregon,
his users were going to be way beyond upset.  ;-)

> Anyhow, after ripping out ioc_data this bug seems to have disappeared!
> Jens, Vivek, could you please have a look at this? In the mean time I
> hope to pinpoint this issue and draft a small patch to really solve it
> (ie, not by simply ripping out ioc_data).

Again, very cool!!!

							Thanx, Paul

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-06-05 00:48, Paul Bolle wrote:
> I think I finally found it!
> 
> The culprit seems to be io_context.ioc_data (not the most clear of
> names!). It seems to be a single entry "last-hit cache" of an hlist
> called cic_list. (There are three, subtly different, cic_lists in the
> CFQ code!) It is not entirely clear, but that last-hit cache can get out
> of sync with the hlist it is supposed to cache. My guess it that every
> now and then a member of the hlist gets deleted while it's still in that
> (single entry) cache. If it then gets retrieved from that cache it
> already points to poisoned memory. For some strange reason this only
> results in an Oops if one or more debugging options are set (as are set
> in the Fedora Rawhide non-stable kernels that I ran into this). I have
> no clue whatsoever, why that is ...
> 
> Anyhow, after ripping out ioc_data this bug seems to have disappeared!
> Jens, Vivek, could you please have a look at this? In the mean time I
> hope to pinpoint this issue and draft a small patch to really solve it
> (ie, not by simply ripping out ioc_data).

Does this fix it? It will introduce a hierarchy that is queue -> ioc
lock, but as far as I can remember (and tell from a quick look), we
don't have any dependencies on that order of locking at this moment. So
should be OK.

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 3c7b537..fa7ef54 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -2772,8 +2772,11 @@ static void __cfq_exit_single_io_context(struct cfq_data *cfqd,
 	smp_wmb();
 	cic->key = cfqd_dead_key(cfqd);
 
-	if (ioc->ioc_data == cic)
+	if (ioc->ioc_data == cic) {
+		spin_lock(&ioc->lock);
 		rcu_assign_pointer(ioc->ioc_data, NULL);
+		spin_unlock(&ioc->lock);
+	}
 
 	if (cic->cfqq[BLK_RW_ASYNC]) {
 		cfq_exit_cfqq(cfqd, cic->cfqq[BLK_RW_ASYNC]);

-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Sun, 2011-06-05 at 08:56 +0200, Jens Axboe wrote:
> Does this fix it? It will introduce a hierarchy that is queue -> ioc
> lock, but as far as I can remember (and tell from a quick look), we
> don't have any dependencies on that order of locking at this moment. So
> should be OK.
> 
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index 3c7b537..fa7ef54 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -2772,8 +2772,11 @@ static void __cfq_exit_single_io_context(struct cfq_data *cfqd,
>  	smp_wmb();
>  	cic->key = cfqd_dead_key(cfqd);
>  
> -	if (ioc->ioc_data == cic)
> +	if (ioc->ioc_data == cic) {
> +		spin_lock(&ioc->lock);
>  		rcu_assign_pointer(ioc->ioc_data, NULL);
> +		spin_unlock(&ioc->lock);
> +	}
>  
>  	if (cic->cfqq[BLK_RW_ASYNC]) {
>  		cfq_exit_cfqq(cfqd, cic->cfqq[BLK_RW_ASYNC]);
> 

0) I'd guess not, as the last thing I tried before simply ripping
io_context.ioc_data out, was:

        spin_lock_irqsave(&ioc->lock, flags);
        rcu_read_lock();
        ioc_data = rcu_dereference(ioc->ioc_data);
        rcu_read_unlock();
        if (ioc_data == cic)
                rcu_assign_pointer(ioc->ioc_data, NULL);
        spin_unlock_irqrestore(&ioc->lock, flags);

(By this time I had already wrapped all access to io_context.ioc_data in
rcu_read_lock(), rcu_dereference(), and rcu_read_unlock() voodoo. I also
wrapped all access of io_context members - other than refcount and
nr_tasks - in a spin_lock_irqsave()/spin_unlock_irqrestore() on
io_context.lock. This gave no warnings, nor lockups, but the code just
kept crashing in the exact same location it always did!)

1) Of course, by now I have forgotten what I had in mind when I stopped
working on this last night. My first bet currently is to change the core
of cic_free_func() into something like:

        spin_lock_irqsave(&ioc->lock, flags);
        radix_tree_delete(&ioc->radix_root, dead_key >>
CIC_DEAD_INDEX_SHIFT);

        rcu_read_lock();
        ioc_data = rcu_dereference(ioc->ioc_data);
        rcu_read_unlock();
        if (ioc_data == cic)
		rcu_assign_pointer(ioc->ioc_data, NULL);

        hlist_del_rcu(&cic->cic_list);
        spin_unlock_irqrestore(&ioc->lock, flags);

2) But, I must admit I'm not yet at full speed today.


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Paul Bolle]

On Sun, 2011-06-05 at 10:39 +0200, Paul Bolle wrote:
> 0) I'd guess not, as the last thing I tried before simply ripping
> io_context.ioc_data out, was:
> 
>         spin_lock_irqsave(&ioc->lock, flags);
>         rcu_read_lock();
>         ioc_data = rcu_dereference(ioc->ioc_data);
>         rcu_read_unlock();
>         if (ioc_data == cic)
>                 rcu_assign_pointer(ioc->ioc_data, NULL);
>         spin_unlock_irqrestore(&ioc->lock, flags);

0) Strike that. It does seem to fix it. I must have done something silly
while doing yet another test of this stuff (perhaps by forgetting to
remove an old, crashing version of the cfq_iosched module before
installing a new version and switching back to cfq).

1) Jens, I suggest I'll send in a small series of patches, along these
lines:
- cleanup: make two functions static in block/cfq-iosched.c
- cleanup: rename io_context.cic_list -> io_context.cic_hlist
           rename io_context.ioc_data -> io_context.last_cic
           change io_context.last_cic from void* to cfq_io_context* 
           rename cfq_io_context.cic_list -> cfq_io_context.cic_node
- cleanup: rename *ret -> *ioc in two functions
- locking: make sure io_context.last_cic is accessed under
           io_context.lock and using the required RCU voodoo

2) Any objections beforehand?


Paul Bolle

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-06-05 12:38, Paul Bolle wrote:
> On Sun, 2011-06-05 at 10:39 +0200, Paul Bolle wrote:
>> 0) I'd guess not, as the last thing I tried before simply ripping
>> io_context.ioc_data out, was:
>>
>>         spin_lock_irqsave(&ioc->lock, flags);
>>         rcu_read_lock();
>>         ioc_data = rcu_dereference(ioc->ioc_data);
>>         rcu_read_unlock();
>>         if (ioc_data == cic)
>>                 rcu_assign_pointer(ioc->ioc_data, NULL);
>>         spin_unlock_irqrestore(&ioc->lock, flags);

You don't need to nest a rcu_read_lock/unlock() inside that write lock.

> 0) Strike that. It does seem to fix it. I must have done something silly
> while doing yet another test of this stuff (perhaps by forgetting to
> remove an old, crashing version of the cfq_iosched module before
> installing a new version and switching back to cfq).

Good, I thought it should!

> 1) Jens, I suggest I'll send in a small series of patches, along these
> lines:
> - cleanup: make two functions static in block/cfq-iosched.c
> - cleanup: rename io_context.cic_list -> io_context.cic_hlist
>            rename io_context.ioc_data -> io_context.last_cic
>            change io_context.last_cic from void* to cfq_io_context* 
>            rename cfq_io_context.cic_list -> cfq_io_context.cic_node
> - cleanup: rename *ret -> *ioc in two functions
> - locking: make sure io_context.last_cic is accessed under
>            io_context.lock and using the required RCU voodoo
> 
> 2) Any objections beforehand?

I've already applied the fix I sent to you, since it was definitely a
bug. So please send anything around that. Was out today, so not pushed
out until now and just now catching up on the email.

And thanks a ton for being persistent and doing a ton of work trying to
figure out what was wrong. Fixing is the easy part, illuminating where
the problem is or may be is the hard part!

-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Vivek Goyal]

On Sun, Jun 05, 2011 at 08:56:33AM +0200, Jens Axboe wrote:
> On 2011-06-05 00:48, Paul Bolle wrote:
> > I think I finally found it!
> > 
> > The culprit seems to be io_context.ioc_data (not the most clear of
> > names!). It seems to be a single entry "last-hit cache" of an hlist
> > called cic_list. (There are three, subtly different, cic_lists in the
> > CFQ code!) It is not entirely clear, but that last-hit cache can get out
> > of sync with the hlist it is supposed to cache. My guess it that every
> > now and then a member of the hlist gets deleted while it's still in that
> > (single entry) cache. If it then gets retrieved from that cache it
> > already points to poisoned memory. For some strange reason this only
> > results in an Oops if one or more debugging options are set (as are set
> > in the Fedora Rawhide non-stable kernels that I ran into this). I have
> > no clue whatsoever, why that is ...
> > 
> > Anyhow, after ripping out ioc_data this bug seems to have disappeared!
> > Jens, Vivek, could you please have a look at this? In the mean time I
> > hope to pinpoint this issue and draft a small patch to really solve it
> > (ie, not by simply ripping out ioc_data).
> 
> Does this fix it? It will introduce a hierarchy that is queue -> ioc
> lock, but as far as I can remember (and tell from a quick look), we
> don't have any dependencies on that order of locking at this moment. So
> should be OK.

Excellent debugging PaulB. This bug was troubling us for a very long
time now.

While we are looking at races, I am wondering if we still have more races
in process exit vs queue exit path. That's a different thing that it is
not easy to hit the races.

So process exit path we do following.

cfq_exit_single_io_context {
	cfqd = cic_to_cfqd();
	q = cfqd->queue; <----- Racy
	spin_lock_irqsave(q->queue_lock, flags); <--- Racy
}

And in queue exit path we do following.

cfq_exit_queue()
{
	spin_lock_irq(q->queue_lock);
		__cfq_exit_single_io_context {
			cic->key = cfqd_dead_key(cfqd);	
		}
	spin_unlock_irq(q->queue_lock);
	kfree(cfqd);	
}

In process exit path, we retrieve cfqd without any locks (Except rcu read
lock), and it might race with queue exit path. Resulting in the fact that
by the time we fetch queue pointer from cfqd, cfqd might have already been
freed.

I think issue here seems to be that we fetch cfqd = cic->key under rcu
but free up cfqd without waiting for rcu grace period.

If we introduce an unconditional synchronize_rcu() in cfq_exit_queue()
then it takes a long time for drivers which create lots of queues
and tear them down during boot.

If we use call_rcu() to free cfqd(), then there is no gurantee that
request queue object is still around. (cfqd->queue).

Even if we take request queue reference to keep request queue object,
then there is no gurantee that queue->lock is around as driver might
have freed it up.

Not sure how to fix this cleanly. Just thought of raising the concern
though while we are at it.

Thanks
Vivek

Re: Mysterious CFQ crash and RCU

[Vivek Goyal]

On Sat, Jun 04, 2011 at 04:06:33PM -0700, Paul E. McKenney wrote:
> On Sun, Jun 05, 2011 at 12:48:05AM +0200, Paul Bolle wrote:
> > On Sat, 2011-06-04 at 09:03 -0700, Paul E. McKenney wrote:
> > > More like "based on these diagnostics, I see no evidence of the RCU
> > > implementation misbehaving."  Which is of course different than "I can
> > > prove that the RCU implementation is not misbehaving".  That said, the
> > > fact that you are running on a single CPU makes it hard for me to see
> > > any latitude for RCU-implementation misbehavior.
> > > 
> > > Clearly something is wrong somewhere.
> > 
> > Yes.
> > 
> > > Given the fact that on a single-CPU
> > > system, synchronize_rcu() is a no-op, and given that you weren't able
> > > to reproduce with CONFIG_TREE_PREEMPT_RCU=y, my guess is that there is
> > > a synchronize_rcu() that occasionally (illegally) gets executed within
> > > an RCU read-side critical section.
> > 
> > I think I finally found it!
> 
> Very cool!!!
> 
> > The culprit seems to be io_context.ioc_data (not the most clear of
> > names!). It seems to be a single entry "last-hit cache" of an hlist
> > called cic_list. (There are three, subtly different, cic_lists in the
> > CFQ code!) It is not entirely clear, but that last-hit cache can get out
> > of sync with the hlist it is supposed to cache. My guess it that every
> > now and then a member of the hlist gets deleted while it's still in that
> > (single entry) cache. If it then gets retrieved from that cache it
> > already points to poisoned memory. For some strange reason this only
> > results in an Oops if one or more debugging options are set (as are set
> > in the Fedora Rawhide non-stable kernels that I ran into this). I have
> > no clue whatsoever, why that is ...
> 
> Oooh...  If I understand you correctly, this is exactly analogous to my
> first-ever RCU bug way back in the early 90s.  This was a hashed version
> of a BSD-style routing table, where each hash bucket maintained a pointer
> to the last routing entry used.  When I deleted a routing entry, I forgot
> to check the last-entry pointer.
> 
> The amazing thing is that the guy who found and fixed this bug generated
> a correct fix despite having absolutely no idea what RCU was or how it
> worked.  All he knew was that if he waited for the sun to reach Oregon,
> his users were going to be way beyond upset.  ;-)
> 
> > Anyhow, after ripping out ioc_data this bug seems to have disappeared!
> > Jens, Vivek, could you please have a look at this? In the mean time I
> > hope to pinpoint this issue and draft a small patch to really solve it
> > (ie, not by simply ripping out ioc_data).
> 
> Again, very cool!!!

Hi Paul,

Remember this CFQ issue where we suspected that we are freeing up an
object before rcu period is over. We thought we fixed it. Guess what,
it seems be to be back.

**************************************************************************
[21265.163144] BUG: unable to handle kernel paging request at 6b6b6b43
[21265.164022] IP: [<f7d55fe0>] cic_free_func+0x10/0x6c [cfq_iosched]
[21265.164022] *pde = 00000000 
[21265.164022] Oops: 0000 [#1] SMP 
[21265.164022] Modules linked in: cfq_iosched tcp_lp fuse arc4 ppdev
snd_intel8x0m snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
microcode
 snd_pcm joydev snd_timer ath5k pcspkr snd_page_alloc thinkpad_acpi
i2c_i801
ath mac80211 snd soundcore cfg80211 rfkill parport_pc parport e1000
iTCO_wd
t iTCO_vendor_support binfmt_misc uinput yenta_socket video radeon ttm
drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
[21265.164022] 
[21265.164022] Pid: 18514, comm: mandb Not tainted
3.0.0-1.local2.fc17.i686 #1
IBM        /       
[21265.164022] EIP: 0060:[<f7d55fe0>] EFLAGS: 00010286 CPU: 0
[21265.164022] EIP is at cic_free_func+0x10/0x6c [cfq_iosched]
[21265.164022] EAX: f242e400 EBX: 6b6b6b43 ECX: 00000246 EDX: 6b6b6b43
[21265.164022] ESI: f242e400 EDI: f7d55fd0 EBP: df92df04 ESP: df92def0
[21265.164022]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[21265.164022] Process mandb (pid: 18514, ti=df92c000 task=ed1554c0
task.ti=df92c000)
[21265.164022] Stack:
[21265.164022]  c0468df7 c0af0cb0 6b6b6b6b f242e400 f7d55fd0 df92df18
f7d54510
f242e400
[21265.164022]  ed1554c0 ed155898 df92df20 f7d54530 df92df2c c05cc8ba
f242e400
df92df40
[21265.164022]  c05cc94a ed1554c0 00000012 df92df74 df92df8c c04423e2
eedf2b5c
eedf2b00
[21265.164022] Call Trace:
[21265.164022]  [<c0468df7>] ? lock_is_held+0x73/0x7b
[21265.164022]  [<f7d55fd0>] ? cfqq_process_refs+0x18/0x18 [cfq_iosched]
[21265.164022]  [<f7d54510>] call_for_each_cic+0x3e/0x4c [cfq_iosched]
[21265.164022]  [<f7d54530>] cfq_free_io_context+0x12/0x14 [cfq_iosched]
[21265.164022]  [<c05cc8ba>] put_io_context+0x34/0x5c
[21265.164022]  [<c05cc94a>] exit_io_context+0x68/0x6d
[21265.164022]  [<c04423e2>] do_exit+0x62f/0x66a
[21265.164022]  [<c0442634>] do_group_exit+0x63/0x86
[21265.164022]  [<c044266f>] sys_exit_group+0x18/0x18
[21265.164022]  [<c083b0df>] sysenter_do_call+0x12/0x38
[21265.164022] Code: 5f 5d c3 55 89 e5 3e 8d 74 26 00 8b 50 3c 03 50 40 8b
00
29 d0 79 02 0f 0b 5d c3 55 89 e5 57 56 53 83 ec 08 3e 8d 74 26 00 89 d3 <8
b> 12 89 c6 f6 c2 01 75 02 0f 0b 8d 78 08 89 f8 89 55 ec e8 c8 
[21265.164022] EIP: [<f7d55fe0>] cic_free_func+0x10/0x6c [cfq_iosched]
SS:ESP
0068:df92def0
[21265.164022] CR2: 000000006b6b6b43
[21265.363749] ---[ end trace a779435da6d0a152 ]---

**************************************************************************
Again details have been added to old bz.

https://bugzilla.redhat.com/show_bug.cgi?id=577968

Now two users have run into this issue. PaulB, again can reliably
reproduce this issue even with 3.0 kernels.  He  just needs to run
"mandb -c" on this machine.

He has again run the CFQ debug V2 patch which seems to be telling that
cfq_io_context object was freed while in rcu read section.

**************************************************************************
[  562.598881] debug: Will exit io context cic=0xf0469230, Next pointer is  
(null)
[  562.598881] debug: cic=0xf0469230 exit io context done. Next pointer is  
(null)
[  562.598881] debug: Will decouple-free cic cic=0xf0469230, Next pointer
is  (null)
[  562.633403] debug: cic=0xf0469230 decouple-free cic done. Next pointer
is  (null)
[  562.644949] debug: freeing up cic=f0469230
[  562.652971] cic=f0469230 freed while in RCU read lock

*****************************************************************************

I am hoping that I have written the debug patch right. As usual i am
running out of ideas on how to debug it further. Anything debug
information you would like to have a look at?

Thanks
Vivek

Re: Mysterious CFQ crash and RCU

[Jens Axboe]

On 2011-08-04 17:05, Vivek Goyal wrote:
> On Sat, Jun 04, 2011 at 04:06:33PM -0700, Paul E. McKenney wrote:
>> On Sun, Jun 05, 2011 at 12:48:05AM +0200, Paul Bolle wrote:
>>> On Sat, 2011-06-04 at 09:03 -0700, Paul E. McKenney wrote:
>>>> More like "based on these diagnostics, I see no evidence of the RCU
>>>> implementation misbehaving."  Which is of course different than "I can
>>>> prove that the RCU implementation is not misbehaving".  That said, the
>>>> fact that you are running on a single CPU makes it hard for me to see
>>>> any latitude for RCU-implementation misbehavior.
>>>>
>>>> Clearly something is wrong somewhere.
>>>
>>> Yes.
>>>
>>>> Given the fact that on a single-CPU
>>>> system, synchronize_rcu() is a no-op, and given that you weren't able
>>>> to reproduce with CONFIG_TREE_PREEMPT_RCU=y, my guess is that there is
>>>> a synchronize_rcu() that occasionally (illegally) gets executed within
>>>> an RCU read-side critical section.
>>>
>>> I think I finally found it!
>>
>> Very cool!!!
>>
>>> The culprit seems to be io_context.ioc_data (not the most clear of
>>> names!). It seems to be a single entry "last-hit cache" of an hlist
>>> called cic_list. (There are three, subtly different, cic_lists in the
>>> CFQ code!) It is not entirely clear, but that last-hit cache can get out
>>> of sync with the hlist it is supposed to cache. My guess it that every
>>> now and then a member of the hlist gets deleted while it's still in that
>>> (single entry) cache. If it then gets retrieved from that cache it
>>> already points to poisoned memory. For some strange reason this only
>>> results in an Oops if one or more debugging options are set (as are set
>>> in the Fedora Rawhide non-stable kernels that I ran into this). I have
>>> no clue whatsoever, why that is ...
>>
>> Oooh...  If I understand you correctly, this is exactly analogous to my
>> first-ever RCU bug way back in the early 90s.  This was a hashed version
>> of a BSD-style routing table, where each hash bucket maintained a pointer
>> to the last routing entry used.  When I deleted a routing entry, I forgot
>> to check the last-entry pointer.
>>
>> The amazing thing is that the guy who found and fixed this bug generated
>> a correct fix despite having absolutely no idea what RCU was or how it
>> worked.  All he knew was that if he waited for the sun to reach Oregon,
>> his users were going to be way beyond upset.  ;-)
>>
>>> Anyhow, after ripping out ioc_data this bug seems to have disappeared!
>>> Jens, Vivek, could you please have a look at this? In the mean time I
>>> hope to pinpoint this issue and draft a small patch to really solve it
>>> (ie, not by simply ripping out ioc_data).
>>
>> Again, very cool!!!
> 
> Hi Paul,
> 
> Remember this CFQ issue where we suspected that we are freeing up an
> object before rcu period is over. We thought we fixed it. Guess what,
> it seems be to be back.
> 
> **************************************************************************
> [21265.163144] BUG: unable to handle kernel paging request at 6b6b6b43
> [21265.164022] IP: [<f7d55fe0>] cic_free_func+0x10/0x6c [cfq_iosched]
> [21265.164022] *pde = 00000000 
> [21265.164022] Oops: 0000 [#1] SMP 
> [21265.164022] Modules linked in: cfq_iosched tcp_lp fuse arc4 ppdev
> snd_intel8x0m snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
> microcode
>  snd_pcm joydev snd_timer ath5k pcspkr snd_page_alloc thinkpad_acpi
> i2c_i801
> ath mac80211 snd soundcore cfg80211 rfkill parport_pc parport e1000
> iTCO_wd
> t iTCO_vendor_support binfmt_misc uinput yenta_socket video radeon ttm
> drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
> [21265.164022] 
> [21265.164022] Pid: 18514, comm: mandb Not tainted
> 3.0.0-1.local2.fc17.i686 #1
> IBM        /       
> [21265.164022] EIP: 0060:[<f7d55fe0>] EFLAGS: 00010286 CPU: 0
> [21265.164022] EIP is at cic_free_func+0x10/0x6c [cfq_iosched]
> [21265.164022] EAX: f242e400 EBX: 6b6b6b43 ECX: 00000246 EDX: 6b6b6b43
> [21265.164022] ESI: f242e400 EDI: f7d55fd0 EBP: df92df04 ESP: df92def0
> [21265.164022]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> [21265.164022] Process mandb (pid: 18514, ti=df92c000 task=ed1554c0
> task.ti=df92c000)
> [21265.164022] Stack:
> [21265.164022]  c0468df7 c0af0cb0 6b6b6b6b f242e400 f7d55fd0 df92df18
> f7d54510
> f242e400
> [21265.164022]  ed1554c0 ed155898 df92df20 f7d54530 df92df2c c05cc8ba
> f242e400
> df92df40
> [21265.164022]  c05cc94a ed1554c0 00000012 df92df74 df92df8c c04423e2
> eedf2b5c
> eedf2b00
> [21265.164022] Call Trace:
> [21265.164022]  [<c0468df7>] ? lock_is_held+0x73/0x7b
> [21265.164022]  [<f7d55fd0>] ? cfqq_process_refs+0x18/0x18 [cfq_iosched]
> [21265.164022]  [<f7d54510>] call_for_each_cic+0x3e/0x4c [cfq_iosched]
> [21265.164022]  [<f7d54530>] cfq_free_io_context+0x12/0x14 [cfq_iosched]
> [21265.164022]  [<c05cc8ba>] put_io_context+0x34/0x5c
> [21265.164022]  [<c05cc94a>] exit_io_context+0x68/0x6d
> [21265.164022]  [<c04423e2>] do_exit+0x62f/0x66a
> [21265.164022]  [<c0442634>] do_group_exit+0x63/0x86
> [21265.164022]  [<c044266f>] sys_exit_group+0x18/0x18
> [21265.164022]  [<c083b0df>] sysenter_do_call+0x12/0x38
> [21265.164022] Code: 5f 5d c3 55 89 e5 3e 8d 74 26 00 8b 50 3c 03 50 40 8b
> 00
> 29 d0 79 02 0f 0b 5d c3 55 89 e5 57 56 53 83 ec 08 3e 8d 74 26 00 89 d3 <8
> b> 12 89 c6 f6 c2 01 75 02 0f 0b 8d 78 08 89 f8 89 55 ec e8 c8 
> [21265.164022] EIP: [<f7d55fe0>] cic_free_func+0x10/0x6c [cfq_iosched]
> SS:ESP
> 0068:df92def0
> [21265.164022] CR2: 000000006b6b6b43
> [21265.363749] ---[ end trace a779435da6d0a152 ]---
> 
> **************************************************************************
> Again details have been added to old bz.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=577968
> 
> Now two users have run into this issue. PaulB, again can reliably
> reproduce this issue even with 3.0 kernels.  He  just needs to run
> "mandb -c" on this machine.
> 
> He has again run the CFQ debug V2 patch which seems to be telling that
> cfq_io_context object was freed while in rcu read section.
> 
> **************************************************************************
> [  562.598881] debug: Will exit io context cic=0xf0469230, Next pointer is  
> (null)
> [  562.598881] debug: cic=0xf0469230 exit io context done. Next pointer is  
> (null)
> [  562.598881] debug: Will decouple-free cic cic=0xf0469230, Next pointer
> is  (null)
> [  562.633403] debug: cic=0xf0469230 decouple-free cic done. Next pointer
> is  (null)
> [  562.644949] debug: freeing up cic=f0469230
> [  562.652971] cic=f0469230 freed while in RCU read lock
> 
> *****************************************************************************
> 
> I am hoping that I have written the debug patch right. As usual i am
> running out of ideas on how to debug it further. Anything debug
> information you would like to have a look at?

Darn! Vivek, can you re-send the debug patch for inspection?

-- 
Jens Axboe

Re: Mysterious CFQ crash and RCU

[Vivek Goyal]

On Thu, Aug 04, 2011 at 09:43:27PM +0200, Jens Axboe wrote:
> On 2011-08-04 17:05, Vivek Goyal wrote:
> > On Sat, Jun 04, 2011 at 04:06:33PM -0700, Paul E. McKenney wrote:
> >> On Sun, Jun 05, 2011 at 12:48:05AM +0200, Paul Bolle wrote:
> >>> On Sat, 2011-06-04 at 09:03 -0700, Paul E. McKenney wrote:
> >>>> More like "based on these diagnostics, I see no evidence of the RCU
> >>>> implementation misbehaving."  Which is of course different than "I can
> >>>> prove that the RCU implementation is not misbehaving".  That said, the
> >>>> fact that you are running on a single CPU makes it hard for me to see
> >>>> any latitude for RCU-implementation misbehavior.
> >>>>
> >>>> Clearly something is wrong somewhere.
> >>>
> >>> Yes.
> >>>
> >>>> Given the fact that on a single-CPU
> >>>> system, synchronize_rcu() is a no-op, and given that you weren't able
> >>>> to reproduce with CONFIG_TREE_PREEMPT_RCU=y, my guess is that there is
> >>>> a synchronize_rcu() that occasionally (illegally) gets executed within
> >>>> an RCU read-side critical section.
> >>>
> >>> I think I finally found it!
> >>
> >> Very cool!!!
> >>
> >>> The culprit seems to be io_context.ioc_data (not the most clear of
> >>> names!). It seems to be a single entry "last-hit cache" of an hlist
> >>> called cic_list. (There are three, subtly different, cic_lists in the
> >>> CFQ code!) It is not entirely clear, but that last-hit cache can get out
> >>> of sync with the hlist it is supposed to cache. My guess it that every
> >>> now and then a member of the hlist gets deleted while it's still in that
> >>> (single entry) cache. If it then gets retrieved from that cache it
> >>> already points to poisoned memory. For some strange reason this only
> >>> results in an Oops if one or more debugging options are set (as are set
> >>> in the Fedora Rawhide non-stable kernels that I ran into this). I have
> >>> no clue whatsoever, why that is ...
> >>
> >> Oooh...  If I understand you correctly, this is exactly analogous to my
> >> first-ever RCU bug way back in the early 90s.  This was a hashed version
> >> of a BSD-style routing table, where each hash bucket maintained a pointer
> >> to the last routing entry used.  When I deleted a routing entry, I forgot
> >> to check the last-entry pointer.
> >>
> >> The amazing thing is that the guy who found and fixed this bug generated
> >> a correct fix despite having absolutely no idea what RCU was or how it
> >> worked.  All he knew was that if he waited for the sun to reach Oregon,
> >> his users were going to be way beyond upset.  ;-)
> >>
> >>> Anyhow, after ripping out ioc_data this bug seems to have disappeared!
> >>> Jens, Vivek, could you please have a look at this? In the mean time I
> >>> hope to pinpoint this issue and draft a small patch to really solve it
> >>> (ie, not by simply ripping out ioc_data).
> >>
> >> Again, very cool!!!
> > 
> > Hi Paul,
> > 
> > Remember this CFQ issue where we suspected that we are freeing up an
> > object before rcu period is over. We thought we fixed it. Guess what,
> > it seems be to be back.
> > 
> > **************************************************************************
> > [21265.163144] BUG: unable to handle kernel paging request at 6b6b6b43
> > [21265.164022] IP: [<f7d55fe0>] cic_free_func+0x10/0x6c [cfq_iosched]
> > [21265.164022] *pde = 00000000 
> > [21265.164022] Oops: 0000 [#1] SMP 
> > [21265.164022] Modules linked in: cfq_iosched tcp_lp fuse arc4 ppdev
> > snd_intel8x0m snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
> > microcode
> >  snd_pcm joydev snd_timer ath5k pcspkr snd_page_alloc thinkpad_acpi
> > i2c_i801
> > ath mac80211 snd soundcore cfg80211 rfkill parport_pc parport e1000
> > iTCO_wd
> > t iTCO_vendor_support binfmt_misc uinput yenta_socket video radeon ttm
> > drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
> > [21265.164022] 
> > [21265.164022] Pid: 18514, comm: mandb Not tainted
> > 3.0.0-1.local2.fc17.i686 #1
> > IBM        /       
> > [21265.164022] EIP: 0060:[<f7d55fe0>] EFLAGS: 00010286 CPU: 0
> > [21265.164022] EIP is at cic_free_func+0x10/0x6c [cfq_iosched]
> > [21265.164022] EAX: f242e400 EBX: 6b6b6b43 ECX: 00000246 EDX: 6b6b6b43
> > [21265.164022] ESI: f242e400 EDI: f7d55fd0 EBP: df92df04 ESP: df92def0
> > [21265.164022]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> > [21265.164022] Process mandb (pid: 18514, ti=df92c000 task=ed1554c0
> > task.ti=df92c000)
> > [21265.164022] Stack:
> > [21265.164022]  c0468df7 c0af0cb0 6b6b6b6b f242e400 f7d55fd0 df92df18
> > f7d54510
> > f242e400
> > [21265.164022]  ed1554c0 ed155898 df92df20 f7d54530 df92df2c c05cc8ba
> > f242e400
> > df92df40
> > [21265.164022]  c05cc94a ed1554c0 00000012 df92df74 df92df8c c04423e2
> > eedf2b5c
> > eedf2b00
> > [21265.164022] Call Trace:
> > [21265.164022]  [<c0468df7>] ? lock_is_held+0x73/0x7b
> > [21265.164022]  [<f7d55fd0>] ? cfqq_process_refs+0x18/0x18 [cfq_iosched]
> > [21265.164022]  [<f7d54510>] call_for_each_cic+0x3e/0x4c [cfq_iosched]
> > [21265.164022]  [<f7d54530>] cfq_free_io_context+0x12/0x14 [cfq_iosched]
> > [21265.164022]  [<c05cc8ba>] put_io_context+0x34/0x5c
> > [21265.164022]  [<c05cc94a>] exit_io_context+0x68/0x6d
> > [21265.164022]  [<c04423e2>] do_exit+0x62f/0x66a
> > [21265.164022]  [<c0442634>] do_group_exit+0x63/0x86
> > [21265.164022]  [<c044266f>] sys_exit_group+0x18/0x18
> > [21265.164022]  [<c083b0df>] sysenter_do_call+0x12/0x38
> > [21265.164022] Code: 5f 5d c3 55 89 e5 3e 8d 74 26 00 8b 50 3c 03 50 40 8b
> > 00
> > 29 d0 79 02 0f 0b 5d c3 55 89 e5 57 56 53 83 ec 08 3e 8d 74 26 00 89 d3 <8
> > b> 12 89 c6 f6 c2 01 75 02 0f 0b 8d 78 08 89 f8 89 55 ec e8 c8 
> > [21265.164022] EIP: [<f7d55fe0>] cic_free_func+0x10/0x6c [cfq_iosched]
> > SS:ESP
> > 0068:df92def0
> > [21265.164022] CR2: 000000006b6b6b43
> > [21265.363749] ---[ end trace a779435da6d0a152 ]---
> > 
> > **************************************************************************
> > Again details have been added to old bz.
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=577968
> > 
> > Now two users have run into this issue. PaulB, again can reliably
> > reproduce this issue even with 3.0 kernels.  He  just needs to run
> > "mandb -c" on this machine.
> > 
> > He has again run the CFQ debug V2 patch which seems to be telling that
> > cfq_io_context object was freed while in rcu read section.
> > 
> > **************************************************************************
> > [  562.598881] debug: Will exit io context cic=0xf0469230, Next pointer is  
> > (null)
> > [  562.598881] debug: cic=0xf0469230 exit io context done. Next pointer is  
> > (null)
> > [  562.598881] debug: Will decouple-free cic cic=0xf0469230, Next pointer
> > is  (null)
> > [  562.633403] debug: cic=0xf0469230 decouple-free cic done. Next pointer
> > is  (null)
> > [  562.644949] debug: freeing up cic=f0469230
> > [  562.652971] cic=f0469230 freed while in RCU read lock
> > 
> > *****************************************************************************
> > 
> > I am hoping that I have written the debug patch right. As usual i am
> > running out of ideas on how to debug it further. Anything debug
> > information you would like to have a look at?
> 
> Darn! Vivek, can you re-send the debug patch for inspection?

Sure Jens. Here it is.

Thanks
Vivek


---
 block/cfq-iosched.c |   43 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 40 insertions(+), 3 deletions(-)

Index: linux-2.6/block/cfq-iosched.c
===================================================================
--- linux-2.6.orig/block/cfq-iosched.c	2011-05-18 18:06:11.499007946 -0400
+++ linux-2.6/block/cfq-iosched.c	2011-05-18 18:54:12.896660278 -0400
@@ -2581,6 +2581,17 @@ static void cfq_put_queue(struct cfq_que
 	cfq_put_cfqg(cfqg);
 }
 
+static void cfq_free_io_context(struct io_context *ioc);
+static void cfq_exit_io_context(struct io_context *ioc);
+static void cfq_ioc_set_ioprio(struct io_context *ioc);
+static void cfq_ioc_set_cgroup(struct io_context *ioc);
+
+static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic);
+static void cfq_exit_single_io_context(struct io_context *ioc,
+				       struct cfq_io_context *cic);
+static void changed_ioprio(struct io_context *ioc, struct cfq_io_context *cic);
+static void changed_cgroup(struct io_context *ioc, struct cfq_io_context *cic);
+
 /*
  * Call func for each cic attached to this ioc.
  */
@@ -2589,13 +2600,37 @@ call_for_each_cic(struct io_context *ioc
 		  void (*func)(struct io_context *, struct cfq_io_context *))
 {
 	struct cfq_io_context *cic;
-	struct hlist_node *n;
+	struct hlist_node *n, *next;
+
+	char *action;
+
+
+	if ((void *) func == (void *) cic_free_func)
+		action = "decouple-free cic";
+	else if((void *) func == (void *) cfq_exit_single_io_context)
+		action = "exit io context";
+	else if((void *) func == (void *) changed_ioprio)
+		action = "change ioprio";
+	else if((void *)func == (void *)changed_cgroup)
+		action = "change cgroup";
+	else
+		action = "unknown";
 
 	rcu_read_lock();
 
-	hlist_for_each_entry_rcu(cic, n, &ioc->cic_list, cic_list)
+	hlist_for_each_entry_rcu(cic, n, &ioc->cic_list, cic_list) {
+		printk("debug: Will %s cic=0x%p, Next pointer is %p\n",
+			action, cic, cic->cic_list.next);
 		func(ioc, cic);
-
+		printk("debug: cic=0x%p %s done. Next pointer is %p\n", cic,
+			action, cic->cic_list.next);
+		next = rcu_dereference_raw(hlist_next_rcu(&cic->cic_list));
+
+		if (next == (void *) 0x6b6b6b6b) {
+			printk("cic=%p freed while in RCU read lock\n", cic);
+			BUG();
+		}
+	}
 	rcu_read_unlock();
 }
 
@@ -2605,6 +2640,7 @@ static void cfq_cic_free_rcu(struct rcu_
 
 	cic = container_of(head, struct cfq_io_context, rcu_head);
 
+	printk("debug: freeing up cic=%p\n", cic);
 	kmem_cache_free(cfq_ioc_pool, cic);
 	elv_ioc_count_dec(cfq_ioc_count);
 
@@ -3019,6 +3055,7 @@ cfq_drop_dead_cic(struct cfq_data *cfqd,
 	hlist_del_rcu(&cic->cic_list);
 	spin_unlock_irqrestore(&ioc->lock, flags);
 
+	printk("debug: Dropping dead cic=%p\n", cic);
 	cfq_cic_free(cic);
 }