* ia64 won't boot because of rcu_sched self-detected stall
@ 2012-08-21 22:45 Tony Luck
2012-08-21 23:20 ` Paul E. McKenney
0 siblings, 1 reply; 11+ messages in thread
From: Tony Luck @ 2012-08-21 22:45 UTC (permalink / raw)
To: paulmck; +Cc: linux-next
last few linux-next builds (at least from next-20120806 to
next-20120821) fail to boot on ia64 with these messages:
...
Total of 2 processors activated (3891.20 BogoMIPS).
SMBIOS 2.3 present.
NET: Registered protocol family 16
ACPI: bus type pci registered
bio: create slab <bio-0> at 0
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
INFO: rcu_sched self-detected stall on CPUINFO: rcu_sched detected
stalls on CPUs/tasks: { 1} (detected by 0, t=15002 jiffies)
Call Trace:
[<a0000001000154e0>] show_stack+0x80/0xa0
sp=a000000101247a20 bsp=a0000001012410b0
[<a000000100d76df0>] dump_stack+0x30/0x50
sp=a000000101247bf0 bsp=a000000101241098
[<a0000001001311e0>] print_other_cpu_stall+0x440/0x540
sp=a000000101247bf0 bsp=a000000101241038
[<a000000100131860>] rcu_check_callbacks+0x580/0x780
sp=a000000101247bf0 bsp=a000000101240fd8
[<a0000001000967d0>] update_process_times+0x50/0xa0
sp=a000000101247bf0 bsp=a000000101240fa8
[<a00000010003f7f0>] timer_interrupt+0x130/0x3c0
sp=a000000101247bf0 bsp=a000000101240f58
[<a00000010011ee00>] handle_irq_event_percpu+0x80/0x460
sp=a000000101247bf0 bsp=a000000101240ef0
[<a0000001001266c0>] handle_percpu_irq+0x120/0x1a0
sp=a000000101247bf0 bsp=a000000101240ec0
[<a00000010011e0e0>] generic_handle_irq+0xa0/0xe0
sp=a000000101247bf0 bsp=a000000101240ea0
[<a0000001000117b0>] ia64_handle_irq+0x110/0x440
sp=a000000101247bf0 bsp=a000000101240e18
[<a00000010000c140>] ia64_native_leave_kernel+0x0/0x270
sp=a000000101247bf0 bsp=a000000101240e18
[<a000000100013bf0>] default_idle+0x130/0x180
sp=a000000101247dc0 bsp=a000000101240dd0
[<a000000100014ee0>] cpu_idle+0x1c0/0x3c0
sp=a000000101247e20 bsp=a000000101240d68
[<a000000100d379e0>] rest_init+0xe0/0x100
sp=a000000101247e20 bsp=a000000101240d50
[<a0000001010f9360>] start_kernel+0x760/0x870
sp=a000000101247e20 bsp=a000000101240cd0
[<a000000100d378c0>] start_ap+0x760/0x780
sp=a000000101247e30 bsp=a000000101240bc0
{ 1} (t=15613 jiffies)
Call Trace:
... stack traces keep coming every 64 seconds.
Any clues on where I should look to debug this?
Thanks
-Tony
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ia64 won't boot because of rcu_sched self-detected stall
2012-08-21 22:45 ia64 won't boot because of rcu_sched self-detected stall Tony Luck
@ 2012-08-21 23:20 ` Paul E. McKenney
2012-08-21 23:53 ` Luck, Tony
0 siblings, 1 reply; 11+ messages in thread
From: Paul E. McKenney @ 2012-08-21 23:20 UTC (permalink / raw)
To: Tony Luck; +Cc: linux-next
On Tue, Aug 21, 2012 at 03:45:52PM -0700, Tony Luck wrote:
> last few linux-next builds (at least from next-20120806 to
> next-20120821) fail to boot on ia64 with these messages:
>
> ...
> Total of 2 processors activated (3891.20 BogoMIPS).
> SMBIOS 2.3 present.
> NET: Registered protocol family 16
> ACPI: bus type pci registered
> bio: create slab <bio-0> at 0
> ACPI: Added _OSI(Module Device)
> ACPI: Added _OSI(Processor Device)
> ACPI: Added _OSI(3.0 _SCP Extensions)
> ACPI: Added _OSI(Processor Aggregator Device)
> INFO: rcu_sched self-detected stall on CPUINFO: rcu_sched detected
> stalls on CPUs/tasks: { 1} (detected by 0, t=15002 jiffies)
Two of them simultaneously, it appears. Looks like CPU 1 detected itself
about the same time that CPU 0 detected it. Which is a bit strange,
given that the code gives the CPU a couple of jiffies to complain about
itself before other CPUs are allowed to complain. Was this a guest OS?
Is the system heavily loaded during boot?
> Call Trace:
> [<a0000001000154e0>] show_stack+0x80/0xa0
> sp=a000000101247a20 bsp=a0000001012410b0
> [<a000000100d76df0>] dump_stack+0x30/0x50
> sp=a000000101247bf0 bsp=a000000101241098
> [<a0000001001311e0>] print_other_cpu_stall+0x440/0x540
> sp=a000000101247bf0 bsp=a000000101241038
> [<a000000100131860>] rcu_check_callbacks+0x580/0x780
> sp=a000000101247bf0 bsp=a000000101240fd8
> [<a0000001000967d0>] update_process_times+0x50/0xa0
> sp=a000000101247bf0 bsp=a000000101240fa8
> [<a00000010003f7f0>] timer_interrupt+0x130/0x3c0
> sp=a000000101247bf0 bsp=a000000101240f58
> [<a00000010011ee00>] handle_irq_event_percpu+0x80/0x460
> sp=a000000101247bf0 bsp=a000000101240ef0
> [<a0000001001266c0>] handle_percpu_irq+0x120/0x1a0
> sp=a000000101247bf0 bsp=a000000101240ec0
> [<a00000010011e0e0>] generic_handle_irq+0xa0/0xe0
> sp=a000000101247bf0 bsp=a000000101240ea0
> [<a0000001000117b0>] ia64_handle_irq+0x110/0x440
> sp=a000000101247bf0 bsp=a000000101240e18
> [<a00000010000c140>] ia64_native_leave_kernel+0x0/0x270
> sp=a000000101247bf0 bsp=a000000101240e18
> [<a000000100013bf0>] default_idle+0x130/0x180
> sp=a000000101247dc0 bsp=a000000101240dd0
> [<a000000100014ee0>] cpu_idle+0x1c0/0x3c0
> sp=a000000101247e20 bsp=a000000101240d68
> [<a000000100d379e0>] rest_init+0xe0/0x100
> sp=a000000101247e20 bsp=a000000101240d50
> [<a0000001010f9360>] start_kernel+0x760/0x870
> sp=a000000101247e20 bsp=a000000101240cd0
> [<a000000100d378c0>] start_ap+0x760/0x780
> sp=a000000101247e30 bsp=a000000101240bc0
And it looks like the stall was from the idle loop. Except isn't this
CPU 0's stack, rather than that of the stalled CPU 1?
Having CPU 1's stack would of course be much more helpful.
> { 1} (t=15613 jiffies)
This is the tail end of CPU 1's stall-warning message.
> Call Trace:
>
> ... stack traces keep coming every 64 seconds.
Hmmm... Is the immediately following stack trace that of CPU 1?
That would likely be quite helpful. If not, please see below.
> Any clues on where I should look to debug this?
CONFIG_RCU_CPU_STALL_INFO=y should give more information on why RCU
thought a stall warning should be printed. In addition, you might
bump up the value of RCU_STALL_RAT_DELAY (defined in kernel/rcutree.h)
in order to give CPU 0 more time to complain about itself -- hopefully
getting its stack rather than that of an innocent bystander. It should
be no problem setting RCU_STALL_RAT_DELAY to HZ or perhaps even higher.
Thanx, Paul
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: ia64 won't boot because of rcu_sched self-detected stall
2012-08-21 23:20 ` Paul E. McKenney
@ 2012-08-21 23:53 ` Luck, Tony
2012-08-22 0:46 ` Paul E. McKenney
0 siblings, 1 reply; 11+ messages in thread
From: Luck, Tony @ 2012-08-21 23:53 UTC (permalink / raw)
To: paulmck; +Cc: linux-next
Thanks for the pointers.
I turned on CONFIG_RCU_CPU_STALL_INFO=y and bumped RCU_STALL_RAT_DELAY
from 2 to 20
This is the new console log. There is a minute of hang before the first
pair of stack traces. Then hang for a minute and the second pair show
up.
Linux version 3.6.0-rc2-zx1-smp-next-20120821 (aegl@linux-bxb1) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #2 SMP Tue Aug 21 16:44:17 PDT 2012
EFI v1.10 by HP: SALsystab=0x3fefa000 ACPI 2.0=0x3fd5e000 SMBIOS=0x3fefc000 HCDP=0x3fd5c000
Early serial console at MMIO 0xff5e0000 (options '9600')
bootconsole [uart0] enabled
PCDP: v0 at 0x3fd5c000
Explicit "console="; ignoring PCDP
ACPI: RSDP 000000003fd5e000 00028 (v02 HP)
ACPI: XSDT 000000003fd5e02c 00094 (v01 HP rx2620 00000000 HP 00000000)
ACPI: FACP 000000003fd67390 000F4 (v03 HP rx2620 00000000 HP 00000000)
ACPI BIOS Bug: Warning: 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20120711/tbfadt-567)
ACPI BIOS Bug: Warning: 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20120711/tbfadt-567)
ACPI: DSDT 000000003fd5e100 05F3C (v01 HP rx2620 00000007 INTL 02012044)
ACPI: FACS 000000003fd67488 00040
ACPI: SPCR 000000003fd674c8 00050 (v01 HP rx2620 00000000 HP 00000000)
ACPI: DBGP 000000003fd67518 00034 (v01 HP rx2620 00000000 HP 00000000)
ACPI: APIC 000000003fd67610 000B0 (v01 HP rx2620 00000000 HP 00000000)
ACPI: SPMI 000000003fd67550 00050 (v04 HP rx2620 00000000 HP 00000000)
ACPI: CPEP 000000003fd675a0 00034 (v01 HP rx2620 00000000 HP 00000000)
ACPI: SSDT 000000003fd64040 001D6 (v01 HP rx2620 00000006 INTL 02012044)
ACPI: SSDT 000000003fd64220 00702 (v01 HP rx2620 00000006 INTL 02012044)
ACPI: SSDT 000000003fd64930 00A16 (v01 HP rx2620 00000006 INTL 02012044)
ACPI: SSDT 000000003fd65350 00A16 (v01 HP rx2620 00000006 INTL 02012044)
ACPI: SSDT 000000003fd65d70 00A16 (v01 HP rx2620 00000006 INTL 02012044)
ACPI: SSDT 000000003fd66790 00A16 (v01 HP rx2620 00000006 INTL 02012044)
ACPI: SSDT 000000003fd671b0 000EB (v01 HP rx2620 00000006 INTL 02012044)
ACPI: SSDT 000000003fd672a0 000EF (v01 HP rx2620 00000006 INTL 02012044)
ACPI: Local APIC address c0000000fee00000
2 CPUs available, 2 CPUs total
warning: skipping physical page 0
Initial ramdisk at: 0xe00000407e9bb000 (6071698 bytes)
SAL 3.1: HP version 3.15
SAL Platform features: None
SAL: AP wakeup using external interrupt vector 0xff
MCA related initialization done
warning: skipping physical page 0
Zone ranges:
DMA [mem 0x00004000-0xffffffff]
Normal [mem 0x100000000-0x407ffc7fff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x00004000-0x3f4ebfff]
node 0: [mem 0x3fc00000-0x3fd5bfff]
node 0: [mem 0x4040000000-0x407fd2bfff]
node 0: [mem 0x407fd98000-0x407fe07fff]
node 0: [mem 0x407fe80000-0x407ffc7fff]
Virtual mem_map starts at 0xa0007fffc7900000
Built 1 zonelists in Zone order, mobility grouping off. Total pages: 72586
Kernel command line: BOOT_IMAGE=scsi0:\efi\SuSE\l-zx1-smp.gz root=/dev/disk/by-id/scsi-200000e1100a5d5f2-part2 console=uart,mmio,0xff5e0000
PID hash table entries: 4096 (order: 1, 32768 bytes)
Dentry cache hash table entries: 262144 (order: 7, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 6, 1048576 bytes)
Memory: 2048432k/2086064k available (13698k code, 37632k reserved, 5791k data, 816k init)
SLUB: Genslabs=17, HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Hierarchical RCU implementation.
Additional per-CPU info printed with stalls.
RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=2.
NR_IRQS:768
ACPI: Local APIC address c0000000fee00000
GSI 36 (level, low) -> CPU 0 (0x0000) vector 48
Console: colour dummy device 80x25
Calibrating delay loop... 1945.60 BogoMIPS (lpj=3891200)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024
ACPI: Core revision 20120711
Boot processor id 0x0/0x0
Fixed BSP b0 value from CPU 1
CPU 1: synchronized ITC with CPU 0 (last diff -3 cycles, maxerr 579 cycles)
Brought up 2 CPUs
Total of 2 processors activated (3891.20 BogoMIPS).
SMBIOS 2.3 present.
NET: Registered protocol family 16
ACPI: bus type pci registered
bio: create slab <bio-0> at 0
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
INFO: rcu_sched self-detected stall on CPU
1: (15000 ticks this GP) idle=001/140000000000001/0
INFO: rcu_sched detected stalls on CPUs/tasks:
1: (15000 ticks this GP) idle=001/140000000000001/0
(detected by 0, t=15020 jiffies)
Call Trace:
[<a000000100015420>] show_stack+0x80/0xa0
sp=a0000001012079d0 bsp=a0000001012010d0
[<a000000100d4d0e0>] dump_stack+0x30/0x50
sp=a000000101207ba0 bsp=a0000001012010b8
[<a000000100130fb0>] rcu_check_callbacks+0xf70/0x10a0
sp=a000000101207ba0 bsp=a000000101200fd8
[<a000000100099b70>] update_process_times+0x50/0xa0
sp=a000000101207bf0 bsp=a000000101200fa8
[<a00000010003ef30>] timer_interrupt+0x130/0x3c0
sp=a000000101207bf0 bsp=a000000101200f58
[<a00000010011ee80>] handle_irq_event_percpu+0x80/0x460
sp=a000000101207bf0 bsp=a000000101200ef0
[<a000000100126740>] handle_percpu_irq+0x120/0x1a0
sp=a000000101207bf0 bsp=a000000101200ec0
[<a00000010011e160>] generic_handle_irq+0xa0/0xe0
sp=a000000101207bf0 bsp=a000000101200ea0
[<a000000100011750>] ia64_handle_irq+0x110/0x440
sp=a000000101207bf0 bsp=a000000101200e18
[<a00000010000c0e0>] ia64_native_leave_kernel+0x0/0x270
sp=a000000101207bf0 bsp=a000000101200e18
[<a000000100013b90>] default_idle+0x130/0x180
sp=a000000101207dc0 bsp=a000000101200dd0
[<a000000100014e20>] cpu_idle+0x1c0/0x3c0
sp=a000000101207e20 bsp=a000000101200d68
[<a000000100d09240>] rest_init+0xe0/0x100
sp=a000000101207e20 bsp=a000000101200d50
[<a0000001010b5310>] start_kernel+0x760/0x870
sp=a000000101207e20 bsp=a000000101200cd0
[<a000000100d09120>] start_ap+0x760/0x780
sp=a000000101207e30 bsp=a000000101200bc0
(t=15605 jiffies)
Call Trace:
[<a000000100015420>] show_stack+0x80/0xa0
sp=e0000040601179e0 bsp=e000004060110f78
[<a000000100d4d0e0>] dump_stack+0x30/0x50
sp=e000004060117bb0 bsp=e000004060110f60
[<a0000001001304f0>] rcu_check_callbacks+0x4b0/0x10a0
sp=e000004060117bb0 bsp=e000004060110e88
[<a000000100099b70>] update_process_times+0x50/0xa0
sp=e000004060117c00 bsp=e000004060110e58
[<a00000010003ef30>] timer_interrupt+0x130/0x3c0
sp=e000004060117c00 bsp=e000004060110e08
[<a00000010011ee80>] handle_irq_event_percpu+0x80/0x460
sp=e000004060117c00 bsp=e000004060110d98
[<a000000100126740>] handle_percpu_irq+0x120/0x1a0
sp=e000004060117c00 bsp=e000004060110d68
[<a00000010011e160>] generic_handle_irq+0xa0/0xe0
sp=e000004060117c00 bsp=e000004060110d48
[<a000000100011750>] ia64_handle_irq+0x110/0x440
sp=e000004060117c00 bsp=e000004060110cc0
[<a00000010000c0e0>] ia64_native_leave_kernel+0x0/0x270
sp=e000004060117c00 bsp=e000004060110cc0
[<a000000100013b90>] default_idle+0x130/0x180
sp=e000004060117dd0 bsp=e000004060110c80
[<a000000100014e20>] cpu_idle+0x1c0/0x3c0
sp=e000004060117e30 bsp=e000004060110c18
[<a000000100d46840>] start_secondary+0x390/0x530
sp=e000004060117e30 bsp=e000004060110bc0
[<a000000100d09110>] start_ap+0x750/0x780
sp=e000004060117e30 bsp=e000004060110bc0
INFO: rcu_sched self-detected stall on CPU
1: (15000 ticks this GP) idle=001/140000000000001/0
INFO: rcu_sched detected stalls on CPUs/tasks:
1: (15000 ticks this GP) idle=001/140000000000001/0
(detected by 0, t=15020 jiffies)
Call Trace:
[<a000000100015420>] show_stack+0x80/0xa0
sp=a0000001012079d0 bsp=a0000001012010d0
[<a000000100d4d0e0>] dump_stack+0x30/0x50
sp=a000000101207ba0 bsp=a0000001012010b8
[<a000000100130fb0>] rcu_check_callbacks+0xf70/0x10a0
sp=a000000101207ba0 bsp=a000000101200fd8
[<a000000100099b70>] update_process_times+0x50/0xa0
sp=a000000101207bf0 bsp=a000000101200fa8
[<a00000010003ef30>] timer_interrupt+0x130/0x3c0
sp=a000000101207bf0 bsp=a000000101200f58
[<a00000010011ee80>] handle_irq_event_percpu+0x80/0x460
sp=a000000101207bf0 bsp=a000000101200ef0
[<a000000100126740>] handle_percpu_irq+0x120/0x1a0
sp=a000000101207bf0 bsp=a000000101200ec0
[<a00000010011e160>] generic_handle_irq+0xa0/0xe0
sp=a000000101207bf0 bsp=a000000101200ea0
[<a000000100011750>] ia64_handle_irq+0x110/0x440
sp=a000000101207bf0 bsp=a000000101200e18
[<a00000010000c0e0>] ia64_native_leave_kernel+0x0/0x270
sp=a000000101207bf0 bsp=a000000101200e18
[<a000000100013b90>] default_idle+0x130/0x180
sp=a000000101207dc0 bsp=a000000101200dd0
[<a000000100014e20>] cpu_idle+0x1c0/0x3c0
sp=a000000101207e20 bsp=a000000101200d68
[<a000000100d09240>] rest_init+0xe0/0x100
sp=a000000101207e20 bsp=a000000101200d50
[<a0000001010b5310>] start_kernel+0x760/0x870
sp=a000000101207e20 bsp=a000000101200cd0
[<a000000100d09120>] start_ap+0x760/0x780
sp=a000000101207e30 bsp=a000000101200bc0
(t=15605 jiffies)
Call Trace:
[<a000000100015420>] show_stack+0x80/0xa0
sp=e0000040601179e0 bsp=e000004060110f78
[<a000000100d4d0e0>] dump_stack+0x30/0x50
sp=e000004060117bb0 bsp=e000004060110f60
[<a0000001001304f0>] rcu_check_callbacks+0x4b0/0x10a0
sp=e000004060117bb0 bsp=e000004060110e88
[<a000000100099b70>] update_process_times+0x50/0xa0
sp=e000004060117c00 bsp=e000004060110e58
[<a00000010003ef30>] timer_interrupt+0x130/0x3c0
sp=e000004060117c00 bsp=e000004060110e08
[<a00000010011ee80>] handle_irq_event_percpu+0x80/0x460
sp=e000004060117c00 bsp=e000004060110d98
[<a000000100126740>] handle_percpu_irq+0x120/0x1a0
sp=e000004060117c00 bsp=e000004060110d68
[<a00000010011e160>] generic_handle_irq+0xa0/0xe0
sp=e000004060117c00 bsp=e000004060110d48
[<a000000100011750>] ia64_handle_irq+0x110/0x440
sp=e000004060117c00 bsp=e000004060110cc0
[<a00000010000c0e0>] ia64_native_leave_kernel+0x0/0x270
sp=e000004060117c00 bsp=e000004060110cc0
[<a000000100013b90>] default_idle+0x130/0x180
sp=e000004060117dd0 bsp=e000004060110c80
[<a000000100014e20>] cpu_idle+0x1c0/0x3c0
sp=e000004060117e30 bsp=e000004060110c18
[<a000000100d46840>] start_secondary+0x390/0x530
sp=e000004060117e30 bsp=e000004060110bc0
[<a000000100d09110>] start_ap+0x750/0x780
sp=e000004060117e30 bsp=e000004060110bc0
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ia64 won't boot because of rcu_sched self-detected stall
2012-08-21 23:53 ` Luck, Tony
@ 2012-08-22 0:46 ` Paul E. McKenney
2012-08-22 13:42 ` Luck, Tony
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Paul E. McKenney @ 2012-08-22 0:46 UTC (permalink / raw)
To: Luck, Tony; +Cc: linux-next, fweisbec
On Tue, Aug 21, 2012 at 11:53:50PM +0000, Luck, Tony wrote:
> Thanks for the pointers.
>
> I turned on CONFIG_RCU_CPU_STALL_INFO=y and bumped RCU_STALL_RAT_DELAY
> from 2 to 20
>
> This is the new console log. There is a minute of hang before the first
> pair of stack traces. Then hang for a minute and the second pair show
> up.
>
> Linux version 3.6.0-rc2-zx1-smp-next-20120821 (aegl@linux-bxb1) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #2 SMP Tue Aug 21 16:44:17 PDT 2012
> EFI v1.10 by HP: SALsystab=0x3fefa000 ACPI 2.0=0x3fd5e000 SMBIOS=0x3fefc000 HCDP=0x3fd5c000
> Early serial console at MMIO 0xff5e0000 (options '9600')
> bootconsole [uart0] enabled
> PCDP: v0 at 0x3fd5c000
> Explicit "console="; ignoring PCDP
> ACPI: RSDP 000000003fd5e000 00028 (v02 HP)
> ACPI: XSDT 000000003fd5e02c 00094 (v01 HP rx2620 00000000 HP 00000000)
> ACPI: FACP 000000003fd67390 000F4 (v03 HP rx2620 00000000 HP 00000000)
> ACPI BIOS Bug: Warning: 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20120711/tbfadt-567)
> ACPI BIOS Bug: Warning: 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20120711/tbfadt-567)
> ACPI: DSDT 000000003fd5e100 05F3C (v01 HP rx2620 00000007 INTL 02012044)
> ACPI: FACS 000000003fd67488 00040
> ACPI: SPCR 000000003fd674c8 00050 (v01 HP rx2620 00000000 HP 00000000)
> ACPI: DBGP 000000003fd67518 00034 (v01 HP rx2620 00000000 HP 00000000)
> ACPI: APIC 000000003fd67610 000B0 (v01 HP rx2620 00000000 HP 00000000)
> ACPI: SPMI 000000003fd67550 00050 (v04 HP rx2620 00000000 HP 00000000)
> ACPI: CPEP 000000003fd675a0 00034 (v01 HP rx2620 00000000 HP 00000000)
> ACPI: SSDT 000000003fd64040 001D6 (v01 HP rx2620 00000006 INTL 02012044)
> ACPI: SSDT 000000003fd64220 00702 (v01 HP rx2620 00000006 INTL 02012044)
> ACPI: SSDT 000000003fd64930 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> ACPI: SSDT 000000003fd65350 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> ACPI: SSDT 000000003fd65d70 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> ACPI: SSDT 000000003fd66790 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> ACPI: SSDT 000000003fd671b0 000EB (v01 HP rx2620 00000006 INTL 02012044)
> ACPI: SSDT 000000003fd672a0 000EF (v01 HP rx2620 00000006 INTL 02012044)
> ACPI: Local APIC address c0000000fee00000
> 2 CPUs available, 2 CPUs total
> warning: skipping physical page 0
> Initial ramdisk at: 0xe00000407e9bb000 (6071698 bytes)
> SAL 3.1: HP version 3.15
> SAL Platform features: None
> SAL: AP wakeup using external interrupt vector 0xff
> MCA related initialization done
> warning: skipping physical page 0
> Zone ranges:
> DMA [mem 0x00004000-0xffffffff]
> Normal [mem 0x100000000-0x407ffc7fff]
> Movable zone start for each node
> Early memory node ranges
> node 0: [mem 0x00004000-0x3f4ebfff]
> node 0: [mem 0x3fc00000-0x3fd5bfff]
> node 0: [mem 0x4040000000-0x407fd2bfff]
> node 0: [mem 0x407fd98000-0x407fe07fff]
> node 0: [mem 0x407fe80000-0x407ffc7fff]
> Virtual mem_map starts at 0xa0007fffc7900000
> Built 1 zonelists in Zone order, mobility grouping off. Total pages: 72586
> Kernel command line: BOOT_IMAGE=scsi0:\efi\SuSE\l-zx1-smp.gz root=/dev/disk/by-id/scsi-200000e1100a5d5f2-part2 console=uart,mmio,0xff5e0000
> PID hash table entries: 4096 (order: 1, 32768 bytes)
> Dentry cache hash table entries: 262144 (order: 7, 2097152 bytes)
> Inode-cache hash table entries: 131072 (order: 6, 1048576 bytes)
> Memory: 2048432k/2086064k available (13698k code, 37632k reserved, 5791k data, 816k init)
> SLUB: Genslabs=17, HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
> Hierarchical RCU implementation.
> Additional per-CPU info printed with stalls.
> RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=2.
> NR_IRQS:768
> ACPI: Local APIC address c0000000fee00000
> GSI 36 (level, low) -> CPU 0 (0x0000) vector 48
> Console: colour dummy device 80x25
> Calibrating delay loop... 1945.60 BogoMIPS (lpj=3891200)
> pid_max: default: 32768 minimum: 301
> Mount-cache hash table entries: 1024
> ACPI: Core revision 20120711
> Boot processor id 0x0/0x0
> Fixed BSP b0 value from CPU 1
> CPU 1: synchronized ITC with CPU 0 (last diff -3 cycles, maxerr 579 cycles)
> Brought up 2 CPUs
> Total of 2 processors activated (3891.20 BogoMIPS).
> SMBIOS 2.3 present.
> NET: Registered protocol family 16
> ACPI: bus type pci registered
> bio: create slab <bio-0> at 0
> ACPI: Added _OSI(Module Device)
> ACPI: Added _OSI(Processor Device)
> ACPI: Added _OSI(3.0 _SCP Extensions)
> ACPI: Added _OSI(Processor Aggregator Device)
> INFO: rcu_sched self-detected stall on CPU
> 1: (15000 ticks this GP) idle=001/140000000000001/0
OK, this is strange. The stacks below would lead me to believe that
the CPUs are idle. But the idle= value above says that RCU believes
that this CPU was executing in non-idle process context when the
interrupt occurred.
OK, time to take a look at the IA64 idle loop. And I don't see any
calls to rcu_idle_enter()... Please see below for my best guess as
to where to place it and rcu_idle_exit() -- the rule is that there must
be no use of RCU read-side critical sections between the call to the
rcu_idle_enter() and the rcu_idle_exit(), so you probably know better
than I where to put them.
void __attribute__((noreturn))
cpu_idle (void)
{
void (*mark_idle)(int) = ia64_mark_idle;
int cpu = smp_processor_id();
/* endless idle loop with no priority at all */
while (1) {
rcu_idle_enter(); /* HERE */
if (can_do_pal_halt) {
current_thread_info()->status &= ~TS_POLLING;
/*
* TS_POLLING-cleared state must be visible before we
* test NEED_RESCHED:
*/
smp_mb();
} else {
current_thread_info()->status |= TS_POLLING;
}
if (!need_resched()) {
void (*idle)(void);
#ifdef CONFIG_SMP
min_xtp();
#endif
rmb();
if (mark_idle)
(*mark_idle)(1);
idle = pm_idle;
if (!idle)
idle = default_idle;
(*idle)();
if (mark_idle)
(*mark_idle)(0);
#ifdef CONFIG_SMP
normal_xtp();
#endif
}
rcu_idle_exit(); /* AND HERE */
schedule_preempt_disabled();
check_pgt_cache();
if (cpu_is_offline(cpu))
play_dead();
}
}
Without the calls to rcu_idle_enter() and rcu_idle_exit(), RCU has no
way of knowing that the CPU is idle, so waits forever for a context
switch.
Ah, I bet I know what happened... I don't see tick_nohz_idle_enter(),
so I would guess that there is no dyntick-idle, so the recent changes in
dyntick-idle didn't cause rcu_idle_enter() to be added.
I wonder how many other architectures don't do dyntick-idle?
Looks like about 12 more. Probably need fixing as well...
In the meantime, please let me know if the above additions fixes the
problem for IA64.
Thanx, Paul
> INFO: rcu_sched detected stalls on CPUs/tasks:
> 1: (15000 ticks this GP) idle=001/140000000000001/0
> (detected by 0, t=15020 jiffies)
>
> Call Trace:
> [<a000000100015420>] show_stack+0x80/0xa0
> sp=a0000001012079d0 bsp=a0000001012010d0
> [<a000000100d4d0e0>] dump_stack+0x30/0x50
> sp=a000000101207ba0 bsp=a0000001012010b8
> [<a000000100130fb0>] rcu_check_callbacks+0xf70/0x10a0
> sp=a000000101207ba0 bsp=a000000101200fd8
> [<a000000100099b70>] update_process_times+0x50/0xa0
> sp=a000000101207bf0 bsp=a000000101200fa8
> [<a00000010003ef30>] timer_interrupt+0x130/0x3c0
> sp=a000000101207bf0 bsp=a000000101200f58
> [<a00000010011ee80>] handle_irq_event_percpu+0x80/0x460
> sp=a000000101207bf0 bsp=a000000101200ef0
> [<a000000100126740>] handle_percpu_irq+0x120/0x1a0
> sp=a000000101207bf0 bsp=a000000101200ec0
> [<a00000010011e160>] generic_handle_irq+0xa0/0xe0
> sp=a000000101207bf0 bsp=a000000101200ea0
> [<a000000100011750>] ia64_handle_irq+0x110/0x440
> sp=a000000101207bf0 bsp=a000000101200e18
> [<a00000010000c0e0>] ia64_native_leave_kernel+0x0/0x270
> sp=a000000101207bf0 bsp=a000000101200e18
> [<a000000100013b90>] default_idle+0x130/0x180
> sp=a000000101207dc0 bsp=a000000101200dd0
> [<a000000100014e20>] cpu_idle+0x1c0/0x3c0
> sp=a000000101207e20 bsp=a000000101200d68
> [<a000000100d09240>] rest_init+0xe0/0x100
> sp=a000000101207e20 bsp=a000000101200d50
> [<a0000001010b5310>] start_kernel+0x760/0x870
> sp=a000000101207e20 bsp=a000000101200cd0
> [<a000000100d09120>] start_ap+0x760/0x780
> sp=a000000101207e30 bsp=a000000101200bc0
> (t=15605 jiffies)
>
> Call Trace:
> [<a000000100015420>] show_stack+0x80/0xa0
> sp=e0000040601179e0 bsp=e000004060110f78
> [<a000000100d4d0e0>] dump_stack+0x30/0x50
> sp=e000004060117bb0 bsp=e000004060110f60
> [<a0000001001304f0>] rcu_check_callbacks+0x4b0/0x10a0
> sp=e000004060117bb0 bsp=e000004060110e88
> [<a000000100099b70>] update_process_times+0x50/0xa0
> sp=e000004060117c00 bsp=e000004060110e58
> [<a00000010003ef30>] timer_interrupt+0x130/0x3c0
> sp=e000004060117c00 bsp=e000004060110e08
> [<a00000010011ee80>] handle_irq_event_percpu+0x80/0x460
> sp=e000004060117c00 bsp=e000004060110d98
> [<a000000100126740>] handle_percpu_irq+0x120/0x1a0
> sp=e000004060117c00 bsp=e000004060110d68
> [<a00000010011e160>] generic_handle_irq+0xa0/0xe0
> sp=e000004060117c00 bsp=e000004060110d48
> [<a000000100011750>] ia64_handle_irq+0x110/0x440
> sp=e000004060117c00 bsp=e000004060110cc0
> [<a00000010000c0e0>] ia64_native_leave_kernel+0x0/0x270
> sp=e000004060117c00 bsp=e000004060110cc0
> [<a000000100013b90>] default_idle+0x130/0x180
> sp=e000004060117dd0 bsp=e000004060110c80
> [<a000000100014e20>] cpu_idle+0x1c0/0x3c0
> sp=e000004060117e30 bsp=e000004060110c18
> [<a000000100d46840>] start_secondary+0x390/0x530
> sp=e000004060117e30 bsp=e000004060110bc0
> [<a000000100d09110>] start_ap+0x750/0x780
> sp=e000004060117e30 bsp=e000004060110bc0
> INFO: rcu_sched self-detected stall on CPU
> 1: (15000 ticks this GP) idle=001/140000000000001/0
> INFO: rcu_sched detected stalls on CPUs/tasks:
> 1: (15000 ticks this GP) idle=001/140000000000001/0
> (detected by 0, t=15020 jiffies)
>
> Call Trace:
> [<a000000100015420>] show_stack+0x80/0xa0
> sp=a0000001012079d0 bsp=a0000001012010d0
> [<a000000100d4d0e0>] dump_stack+0x30/0x50
> sp=a000000101207ba0 bsp=a0000001012010b8
> [<a000000100130fb0>] rcu_check_callbacks+0xf70/0x10a0
> sp=a000000101207ba0 bsp=a000000101200fd8
> [<a000000100099b70>] update_process_times+0x50/0xa0
> sp=a000000101207bf0 bsp=a000000101200fa8
> [<a00000010003ef30>] timer_interrupt+0x130/0x3c0
> sp=a000000101207bf0 bsp=a000000101200f58
> [<a00000010011ee80>] handle_irq_event_percpu+0x80/0x460
> sp=a000000101207bf0 bsp=a000000101200ef0
> [<a000000100126740>] handle_percpu_irq+0x120/0x1a0
> sp=a000000101207bf0 bsp=a000000101200ec0
> [<a00000010011e160>] generic_handle_irq+0xa0/0xe0
> sp=a000000101207bf0 bsp=a000000101200ea0
> [<a000000100011750>] ia64_handle_irq+0x110/0x440
> sp=a000000101207bf0 bsp=a000000101200e18
> [<a00000010000c0e0>] ia64_native_leave_kernel+0x0/0x270
> sp=a000000101207bf0 bsp=a000000101200e18
> [<a000000100013b90>] default_idle+0x130/0x180
> sp=a000000101207dc0 bsp=a000000101200dd0
> [<a000000100014e20>] cpu_idle+0x1c0/0x3c0
> sp=a000000101207e20 bsp=a000000101200d68
> [<a000000100d09240>] rest_init+0xe0/0x100
> sp=a000000101207e20 bsp=a000000101200d50
> [<a0000001010b5310>] start_kernel+0x760/0x870
> sp=a000000101207e20 bsp=a000000101200cd0
> [<a000000100d09120>] start_ap+0x760/0x780
> sp=a000000101207e30 bsp=a000000101200bc0
> (t=15605 jiffies)
>
> Call Trace:
> [<a000000100015420>] show_stack+0x80/0xa0
> sp=e0000040601179e0 bsp=e000004060110f78
> [<a000000100d4d0e0>] dump_stack+0x30/0x50
> sp=e000004060117bb0 bsp=e000004060110f60
> [<a0000001001304f0>] rcu_check_callbacks+0x4b0/0x10a0
> sp=e000004060117bb0 bsp=e000004060110e88
> [<a000000100099b70>] update_process_times+0x50/0xa0
> sp=e000004060117c00 bsp=e000004060110e58
> [<a00000010003ef30>] timer_interrupt+0x130/0x3c0
> sp=e000004060117c00 bsp=e000004060110e08
> [<a00000010011ee80>] handle_irq_event_percpu+0x80/0x460
> sp=e000004060117c00 bsp=e000004060110d98
> [<a000000100126740>] handle_percpu_irq+0x120/0x1a0
> sp=e000004060117c00 bsp=e000004060110d68
> [<a00000010011e160>] generic_handle_irq+0xa0/0xe0
> sp=e000004060117c00 bsp=e000004060110d48
> [<a000000100011750>] ia64_handle_irq+0x110/0x440
> sp=e000004060117c00 bsp=e000004060110cc0
> [<a00000010000c0e0>] ia64_native_leave_kernel+0x0/0x270
> sp=e000004060117c00 bsp=e000004060110cc0
> [<a000000100013b90>] default_idle+0x130/0x180
> sp=e000004060117dd0 bsp=e000004060110c80
> [<a000000100014e20>] cpu_idle+0x1c0/0x3c0
> sp=e000004060117e30 bsp=e000004060110c18
> [<a000000100d46840>] start_secondary+0x390/0x530
> sp=e000004060117e30 bsp=e000004060110bc0
> [<a000000100d09110>] start_ap+0x750/0x780
> sp=e000004060117e30 bsp=e000004060110bc0
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: ia64 won't boot because of rcu_sched self-detected stall
2012-08-22 0:46 ` Paul E. McKenney
@ 2012-08-22 13:42 ` Luck, Tony
2012-08-22 15:12 ` Frederic Weisbecker
2012-08-23 19:54 ` Luck, Tony
2 siblings, 0 replies; 11+ messages in thread
From: Luck, Tony @ 2012-08-22 13:42 UTC (permalink / raw)
To: paulmck; +Cc: linux-next, fweisbec
> In the meantime, please let me know if the above additions fixes the
> problem for IA64.
I'm on a plane today ... so it will be Thursday before I can try this. Thanks
for the quick response.
-Tony
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ia64 won't boot because of rcu_sched self-detected stall
2012-08-22 0:46 ` Paul E. McKenney
2012-08-22 13:42 ` Luck, Tony
@ 2012-08-22 15:12 ` Frederic Weisbecker
2012-08-22 15:29 ` Paul E. McKenney
2012-08-23 19:54 ` Luck, Tony
2 siblings, 1 reply; 11+ messages in thread
From: Frederic Weisbecker @ 2012-08-22 15:12 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Luck, Tony, linux-next
On Tue, Aug 21, 2012 at 05:46:08PM -0700, Paul E. McKenney wrote:
> On Tue, Aug 21, 2012 at 11:53:50PM +0000, Luck, Tony wrote:
> > Thanks for the pointers.
> >
> > I turned on CONFIG_RCU_CPU_STALL_INFO=y and bumped RCU_STALL_RAT_DELAY
> > from 2 to 20
> >
> > This is the new console log. There is a minute of hang before the first
> > pair of stack traces. Then hang for a minute and the second pair show
> > up.
> >
> > Linux version 3.6.0-rc2-zx1-smp-next-20120821 (aegl@linux-bxb1) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #2 SMP Tue Aug 21 16:44:17 PDT 2012
> > EFI v1.10 by HP: SALsystab=0x3fefa000 ACPI 2.0=0x3fd5e000 SMBIOS=0x3fefc000 HCDP=0x3fd5c000
> > Early serial console at MMIO 0xff5e0000 (options '9600')
> > bootconsole [uart0] enabled
> > PCDP: v0 at 0x3fd5c000
> > Explicit "console="; ignoring PCDP
> > ACPI: RSDP 000000003fd5e000 00028 (v02 HP)
> > ACPI: XSDT 000000003fd5e02c 00094 (v01 HP rx2620 00000000 HP 00000000)
> > ACPI: FACP 000000003fd67390 000F4 (v03 HP rx2620 00000000 HP 00000000)
> > ACPI BIOS Bug: Warning: 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20120711/tbfadt-567)
> > ACPI BIOS Bug: Warning: 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20120711/tbfadt-567)
> > ACPI: DSDT 000000003fd5e100 05F3C (v01 HP rx2620 00000007 INTL 02012044)
> > ACPI: FACS 000000003fd67488 00040
> > ACPI: SPCR 000000003fd674c8 00050 (v01 HP rx2620 00000000 HP 00000000)
> > ACPI: DBGP 000000003fd67518 00034 (v01 HP rx2620 00000000 HP 00000000)
> > ACPI: APIC 000000003fd67610 000B0 (v01 HP rx2620 00000000 HP 00000000)
> > ACPI: SPMI 000000003fd67550 00050 (v04 HP rx2620 00000000 HP 00000000)
> > ACPI: CPEP 000000003fd675a0 00034 (v01 HP rx2620 00000000 HP 00000000)
> > ACPI: SSDT 000000003fd64040 001D6 (v01 HP rx2620 00000006 INTL 02012044)
> > ACPI: SSDT 000000003fd64220 00702 (v01 HP rx2620 00000006 INTL 02012044)
> > ACPI: SSDT 000000003fd64930 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> > ACPI: SSDT 000000003fd65350 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> > ACPI: SSDT 000000003fd65d70 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> > ACPI: SSDT 000000003fd66790 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> > ACPI: SSDT 000000003fd671b0 000EB (v01 HP rx2620 00000006 INTL 02012044)
> > ACPI: SSDT 000000003fd672a0 000EF (v01 HP rx2620 00000006 INTL 02012044)
> > ACPI: Local APIC address c0000000fee00000
> > 2 CPUs available, 2 CPUs total
> > warning: skipping physical page 0
> > Initial ramdisk at: 0xe00000407e9bb000 (6071698 bytes)
> > SAL 3.1: HP version 3.15
> > SAL Platform features: None
> > SAL: AP wakeup using external interrupt vector 0xff
> > MCA related initialization done
> > warning: skipping physical page 0
> > Zone ranges:
> > DMA [mem 0x00004000-0xffffffff]
> > Normal [mem 0x100000000-0x407ffc7fff]
> > Movable zone start for each node
> > Early memory node ranges
> > node 0: [mem 0x00004000-0x3f4ebfff]
> > node 0: [mem 0x3fc00000-0x3fd5bfff]
> > node 0: [mem 0x4040000000-0x407fd2bfff]
> > node 0: [mem 0x407fd98000-0x407fe07fff]
> > node 0: [mem 0x407fe80000-0x407ffc7fff]
> > Virtual mem_map starts at 0xa0007fffc7900000
> > Built 1 zonelists in Zone order, mobility grouping off. Total pages: 72586
> > Kernel command line: BOOT_IMAGE=scsi0:\efi\SuSE\l-zx1-smp.gz root=/dev/disk/by-id/scsi-200000e1100a5d5f2-part2 console=uart,mmio,0xff5e0000
> > PID hash table entries: 4096 (order: 1, 32768 bytes)
> > Dentry cache hash table entries: 262144 (order: 7, 2097152 bytes)
> > Inode-cache hash table entries: 131072 (order: 6, 1048576 bytes)
> > Memory: 2048432k/2086064k available (13698k code, 37632k reserved, 5791k data, 816k init)
> > SLUB: Genslabs=17, HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
> > Hierarchical RCU implementation.
> > Additional per-CPU info printed with stalls.
> > RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=2.
> > NR_IRQS:768
> > ACPI: Local APIC address c0000000fee00000
> > GSI 36 (level, low) -> CPU 0 (0x0000) vector 48
> > Console: colour dummy device 80x25
> > Calibrating delay loop... 1945.60 BogoMIPS (lpj=3891200)
> > pid_max: default: 32768 minimum: 301
> > Mount-cache hash table entries: 1024
> > ACPI: Core revision 20120711
> > Boot processor id 0x0/0x0
> > Fixed BSP b0 value from CPU 1
> > CPU 1: synchronized ITC with CPU 0 (last diff -3 cycles, maxerr 579 cycles)
> > Brought up 2 CPUs
> > Total of 2 processors activated (3891.20 BogoMIPS).
> > SMBIOS 2.3 present.
> > NET: Registered protocol family 16
> > ACPI: bus type pci registered
> > bio: create slab <bio-0> at 0
> > ACPI: Added _OSI(Module Device)
> > ACPI: Added _OSI(Processor Device)
> > ACPI: Added _OSI(3.0 _SCP Extensions)
> > ACPI: Added _OSI(Processor Aggregator Device)
> > INFO: rcu_sched self-detected stall on CPU
> > 1: (15000 ticks this GP) idle=001/140000000000001/0
>
> OK, this is strange. The stacks below would lead me to believe that
> the CPUs are idle. But the idle= value above says that RCU believes
> that this CPU was executing in non-idle process context when the
> interrupt occurred.
>
> OK, time to take a look at the IA64 idle loop. And I don't see any
> calls to rcu_idle_enter()... Please see below for my best guess as
> to where to place it and rcu_idle_exit() -- the rule is that there must
> be no use of RCU read-side critical sections between the call to the
> rcu_idle_enter() and the rcu_idle_exit(), so you probably know better
> than I where to put them.
>
> void __attribute__((noreturn))
> cpu_idle (void)
> {
> void (*mark_idle)(int) = ia64_mark_idle;
> int cpu = smp_processor_id();
>
> /* endless idle loop with no priority at all */
> while (1) {
> rcu_idle_enter(); /* HERE */
> if (can_do_pal_halt) {
> current_thread_info()->status &= ~TS_POLLING;
> /*
> * TS_POLLING-cleared state must be visible before we
> * test NEED_RESCHED:
> */
> smp_mb();
> } else {
> current_thread_info()->status |= TS_POLLING;
> }
>
> if (!need_resched()) {
> void (*idle)(void);
> #ifdef CONFIG_SMP
> min_xtp();
> #endif
> rmb();
> if (mark_idle)
> (*mark_idle)(1);
>
> idle = pm_idle;
> if (!idle)
> idle = default_idle;
> (*idle)();
> if (mark_idle)
> (*mark_idle)(0);
> #ifdef CONFIG_SMP
> normal_xtp();
> #endif
> }
> rcu_idle_exit(); /* AND HERE */
> schedule_preempt_disabled();
> check_pgt_cache();
> if (cpu_is_offline(cpu))
> play_dead();
> }
> }
>
> Without the calls to rcu_idle_enter() and rcu_idle_exit(), RCU has no
> way of knowing that the CPU is idle, so waits forever for a context
> switch.
>
> Ah, I bet I know what happened... I don't see tick_nohz_idle_enter(),
> so I would guess that there is no dyntick-idle, so the recent changes in
> dyntick-idle didn't cause rcu_idle_enter() to be added.
>
> I wonder how many other architectures don't do dyntick-idle?
>
> Looks like about 12 more. Probably need fixing as well...
Ouch, that's bad. Ok see below for the conversion of other architectures.
While doing this, I realized that most of these archs just use the same
cpu_idle() function, basically:
void cpu_idle(void)
{
while (1) {
+ rcu_idle_enter();
while (!need_resched())
do_arch_thing();
+ rcu_idle_exit();
schedule_preempt_disabled();
}
}
So I think it may be worth creating a "simple idle loop" generic function
for those archs that they can call. This way there is less conversion to do.
Now this is all a regression, so IMO we should first fix the things locally and
do that generic idle loop later, since it's rather a feature.
Hmm?
I'm cooking the patches.
diff --git a/arch/alpha/kernel/process.c b/arch/alpha/kernel/process.c
index 153d3fc..2ebf7b5 100644
--- a/arch/alpha/kernel/process.c
+++ b/arch/alpha/kernel/process.c
@@ -28,6 +28,7 @@
#include <linux/tty.h>
#include <linux/console.h>
#include <linux/slab.h>
+#include <linux/rcupdate.h>
#include <asm/reg.h>
#include <asm/uaccess.h>
@@ -50,13 +51,16 @@ cpu_idle(void)
{
set_thread_flag(TIF_POLLING_NRFLAG);
+ preempt_disable();
while (1) {
/* FIXME -- EV6 and LCA45 know how to power down
the CPU. */
+ rcu_idle_enter();
while (!need_resched())
cpu_relax();
- schedule();
+ rcu_idle_exit();
+ schedule_preempt_disabled();
}
}
diff --git a/arch/cris/kernel/process.c b/arch/cris/kernel/process.c
index 66fd017..7f65be6 100644
--- a/arch/cris/kernel/process.c
+++ b/arch/cris/kernel/process.c
@@ -25,6 +25,7 @@
#include <linux/elfcore.h>
#include <linux/mqueue.h>
#include <linux/reboot.h>
+#include <linux/rcupdate.h>
//#define DEBUG
@@ -74,6 +75,7 @@ void cpu_idle (void)
{
/* endless idle loop with no priority at all */
while (1) {
+ rcu_idle_enter();
while (!need_resched()) {
void (*idle)(void);
/*
@@ -86,6 +88,7 @@ void cpu_idle (void)
idle = default_idle;
idle();
}
+ rcu_idle_exit();
schedule_preempt_disabled();
}
}
diff --git a/arch/frv/kernel/process.c b/arch/frv/kernel/process.c
index ff95f50..2eb7fa5 100644
--- a/arch/frv/kernel/process.c
+++ b/arch/frv/kernel/process.c
@@ -25,6 +25,7 @@
#include <linux/reboot.h>
#include <linux/interrupt.h>
#include <linux/pagemap.h>
+#include <linux/rcupdate.h>
#include <asm/asm-offsets.h>
#include <asm/uaccess.h>
@@ -69,12 +70,14 @@ void cpu_idle(void)
{
/* endless idle loop with no priority at all */
while (1) {
+ rcu_idle_enter();
while (!need_resched()) {
check_pgt_cache();
if (!frv_dma_inprogress && idle)
idle();
}
+ rcu_idle_exit();
schedule_preempt_disabled();
}
diff --git a/arch/h8300/kernel/process.c b/arch/h8300/kernel/process.c
index 0e9c315..f153ed1 100644
--- a/arch/h8300/kernel/process.c
+++ b/arch/h8300/kernel/process.c
@@ -36,6 +36,7 @@
#include <linux/reboot.h>
#include <linux/fs.h>
#include <linux/slab.h>
+#include <linux/rcupdate.h>
#include <asm/uaccess.h>
#include <asm/traps.h>
@@ -78,8 +79,10 @@ void (*idle)(void) = default_idle;
void cpu_idle(void)
{
while (1) {
+ rcu_idle_enter();
while (!need_resched())
idle();
+ rcu_idle_exit();
schedule_preempt_disabled();
}
}
diff --git a/arch/m32r/kernel/process.c b/arch/m32r/kernel/process.c
index 3a4a32b2..384e63f 100644
--- a/arch/m32r/kernel/process.c
+++ b/arch/m32r/kernel/process.c
@@ -26,6 +26,7 @@
#include <linux/ptrace.h>
#include <linux/unistd.h>
#include <linux/hardirq.h>
+#include <linux/rcupdate.h>
#include <asm/io.h>
#include <asm/uaccess.h>
@@ -82,6 +83,7 @@ void cpu_idle (void)
{
/* endless idle loop with no priority at all */
while (1) {
+ rcu_idle_enter();
while (!need_resched()) {
void (*idle)(void) = pm_idle;
@@ -90,6 +92,7 @@ void cpu_idle (void)
idle();
}
+ rcu_idle_exit();
schedule_preempt_disabled();
}
}
diff --git a/arch/m68k/kernel/process.c b/arch/m68k/kernel/process.c
index c488e3c..ac2892e 100644
--- a/arch/m68k/kernel/process.c
+++ b/arch/m68k/kernel/process.c
@@ -25,6 +25,7 @@
#include <linux/reboot.h>
#include <linux/init_task.h>
#include <linux/mqueue.h>
+#include <linux/rcupdate.h>
#include <asm/uaccess.h>
#include <asm/traps.h>
@@ -75,8 +76,10 @@ void cpu_idle(void)
{
/* endless idle loop with no priority at all */
while (1) {
+ rcu_idle_enter();
while (!need_resched())
idle();
+ rcu_idle_exit();
schedule_preempt_disabled();
}
}
diff --git a/arch/mn10300/kernel/process.c b/arch/mn10300/kernel/process.c
index 7dab0cd..e9cceba 100644
--- a/arch/mn10300/kernel/process.c
+++ b/arch/mn10300/kernel/process.c
@@ -25,6 +25,7 @@
#include <linux/err.h>
#include <linux/fs.h>
#include <linux/slab.h>
+#include <linux/rcupdate.h>
#include <asm/uaccess.h>
#include <asm/pgtable.h>
#include <asm/io.h>
@@ -107,6 +108,7 @@ void cpu_idle(void)
{
/* endless idle loop with no priority at all */
for (;;) {
+ rcu_idle_enter();
while (!need_resched()) {
void (*idle)(void);
@@ -121,6 +123,7 @@ void cpu_idle(void)
}
idle();
}
+ rcu_idle_exit();
schedule_preempt_disabled();
}
diff --git a/arch/parisc/kernel/process.c b/arch/parisc/kernel/process.c
index d4b94b3..c54a4db 100644
--- a/arch/parisc/kernel/process.c
+++ b/arch/parisc/kernel/process.c
@@ -48,6 +48,7 @@
#include <linux/unistd.h>
#include <linux/kallsyms.h>
#include <linux/uaccess.h>
+#include <linux/rcupdate.h>
#include <asm/io.h>
#include <asm/asm-offsets.h>
@@ -69,8 +70,10 @@ void cpu_idle(void)
/* endless idle loop with no priority at all */
while (1) {
+ rcu_idle_enter();
while (!need_resched())
barrier();
+ rcu_idle_exit();
schedule_preempt_disabled();
check_pgt_cache();
}
diff --git a/arch/score/kernel/process.c b/arch/score/kernel/process.c
index 2707023..637970c 100644
--- a/arch/score/kernel/process.c
+++ b/arch/score/kernel/process.c
@@ -27,6 +27,7 @@
#include <linux/reboot.h>
#include <linux/elfcore.h>
#include <linux/pm.h>
+#include <linux/rcupdate.h>
void (*pm_power_off)(void);
EXPORT_SYMBOL(pm_power_off);
@@ -50,9 +51,10 @@ void __noreturn cpu_idle(void)
{
/* endless idle loop with no priority at all */
while (1) {
+ rcu_idle_enter();
while (!need_resched())
barrier();
-
+ rcu_idle_exit();
schedule_preempt_disabled();
}
}
diff --git a/arch/xtensa/kernel/process.c b/arch/xtensa/kernel/process.c
index 2c8d6a3..bc44311 100644
--- a/arch/xtensa/kernel/process.c
+++ b/arch/xtensa/kernel/process.c
@@ -31,6 +31,7 @@
#include <linux/mqueue.h>
#include <linux/fs.h>
#include <linux/slab.h>
+#include <linux/rcupdate.h>
#include <asm/pgtable.h>
#include <asm/uaccess.h>
@@ -110,8 +111,10 @@ void cpu_idle(void)
/* endless idle loop with no priority at all */
while (1) {
+ rcu_idle_enter();
while (!need_resched())
platform_idle();
+ rcu_idle_exit();
schedule_preempt_disabled();
}
}
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: ia64 won't boot because of rcu_sched self-detected stall
2012-08-22 15:12 ` Frederic Weisbecker
@ 2012-08-22 15:29 ` Paul E. McKenney
0 siblings, 0 replies; 11+ messages in thread
From: Paul E. McKenney @ 2012-08-22 15:29 UTC (permalink / raw)
To: Frederic Weisbecker; +Cc: Luck, Tony, linux-next
On Wed, Aug 22, 2012 at 05:12:03PM +0200, Frederic Weisbecker wrote:
> On Tue, Aug 21, 2012 at 05:46:08PM -0700, Paul E. McKenney wrote:
> > On Tue, Aug 21, 2012 at 11:53:50PM +0000, Luck, Tony wrote:
> > > Thanks for the pointers.
> > >
> > > I turned on CONFIG_RCU_CPU_STALL_INFO=y and bumped RCU_STALL_RAT_DELAY
> > > from 2 to 20
> > >
> > > This is the new console log. There is a minute of hang before the first
> > > pair of stack traces. Then hang for a minute and the second pair show
> > > up.
> > >
> > > Linux version 3.6.0-rc2-zx1-smp-next-20120821 (aegl@linux-bxb1) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #2 SMP Tue Aug 21 16:44:17 PDT 2012
> > > EFI v1.10 by HP: SALsystab=0x3fefa000 ACPI 2.0=0x3fd5e000 SMBIOS=0x3fefc000 HCDP=0x3fd5c000
> > > Early serial console at MMIO 0xff5e0000 (options '9600')
> > > bootconsole [uart0] enabled
> > > PCDP: v0 at 0x3fd5c000
> > > Explicit "console="; ignoring PCDP
> > > ACPI: RSDP 000000003fd5e000 00028 (v02 HP)
> > > ACPI: XSDT 000000003fd5e02c 00094 (v01 HP rx2620 00000000 HP 00000000)
> > > ACPI: FACP 000000003fd67390 000F4 (v03 HP rx2620 00000000 HP 00000000)
> > > ACPI BIOS Bug: Warning: 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20120711/tbfadt-567)
> > > ACPI BIOS Bug: Warning: 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20120711/tbfadt-567)
> > > ACPI: DSDT 000000003fd5e100 05F3C (v01 HP rx2620 00000007 INTL 02012044)
> > > ACPI: FACS 000000003fd67488 00040
> > > ACPI: SPCR 000000003fd674c8 00050 (v01 HP rx2620 00000000 HP 00000000)
> > > ACPI: DBGP 000000003fd67518 00034 (v01 HP rx2620 00000000 HP 00000000)
> > > ACPI: APIC 000000003fd67610 000B0 (v01 HP rx2620 00000000 HP 00000000)
> > > ACPI: SPMI 000000003fd67550 00050 (v04 HP rx2620 00000000 HP 00000000)
> > > ACPI: CPEP 000000003fd675a0 00034 (v01 HP rx2620 00000000 HP 00000000)
> > > ACPI: SSDT 000000003fd64040 001D6 (v01 HP rx2620 00000006 INTL 02012044)
> > > ACPI: SSDT 000000003fd64220 00702 (v01 HP rx2620 00000006 INTL 02012044)
> > > ACPI: SSDT 000000003fd64930 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> > > ACPI: SSDT 000000003fd65350 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> > > ACPI: SSDT 000000003fd65d70 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> > > ACPI: SSDT 000000003fd66790 00A16 (v01 HP rx2620 00000006 INTL 02012044)
> > > ACPI: SSDT 000000003fd671b0 000EB (v01 HP rx2620 00000006 INTL 02012044)
> > > ACPI: SSDT 000000003fd672a0 000EF (v01 HP rx2620 00000006 INTL 02012044)
> > > ACPI: Local APIC address c0000000fee00000
> > > 2 CPUs available, 2 CPUs total
> > > warning: skipping physical page 0
> > > Initial ramdisk at: 0xe00000407e9bb000 (6071698 bytes)
> > > SAL 3.1: HP version 3.15
> > > SAL Platform features: None
> > > SAL: AP wakeup using external interrupt vector 0xff
> > > MCA related initialization done
> > > warning: skipping physical page 0
> > > Zone ranges:
> > > DMA [mem 0x00004000-0xffffffff]
> > > Normal [mem 0x100000000-0x407ffc7fff]
> > > Movable zone start for each node
> > > Early memory node ranges
> > > node 0: [mem 0x00004000-0x3f4ebfff]
> > > node 0: [mem 0x3fc00000-0x3fd5bfff]
> > > node 0: [mem 0x4040000000-0x407fd2bfff]
> > > node 0: [mem 0x407fd98000-0x407fe07fff]
> > > node 0: [mem 0x407fe80000-0x407ffc7fff]
> > > Virtual mem_map starts at 0xa0007fffc7900000
> > > Built 1 zonelists in Zone order, mobility grouping off. Total pages: 72586
> > > Kernel command line: BOOT_IMAGE=scsi0:\efi\SuSE\l-zx1-smp.gz root=/dev/disk/by-id/scsi-200000e1100a5d5f2-part2 console=uart,mmio,0xff5e0000
> > > PID hash table entries: 4096 (order: 1, 32768 bytes)
> > > Dentry cache hash table entries: 262144 (order: 7, 2097152 bytes)
> > > Inode-cache hash table entries: 131072 (order: 6, 1048576 bytes)
> > > Memory: 2048432k/2086064k available (13698k code, 37632k reserved, 5791k data, 816k init)
> > > SLUB: Genslabs=17, HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
> > > Hierarchical RCU implementation.
> > > Additional per-CPU info printed with stalls.
> > > RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=2.
> > > NR_IRQS:768
> > > ACPI: Local APIC address c0000000fee00000
> > > GSI 36 (level, low) -> CPU 0 (0x0000) vector 48
> > > Console: colour dummy device 80x25
> > > Calibrating delay loop... 1945.60 BogoMIPS (lpj=3891200)
> > > pid_max: default: 32768 minimum: 301
> > > Mount-cache hash table entries: 1024
> > > ACPI: Core revision 20120711
> > > Boot processor id 0x0/0x0
> > > Fixed BSP b0 value from CPU 1
> > > CPU 1: synchronized ITC with CPU 0 (last diff -3 cycles, maxerr 579 cycles)
> > > Brought up 2 CPUs
> > > Total of 2 processors activated (3891.20 BogoMIPS).
> > > SMBIOS 2.3 present.
> > > NET: Registered protocol family 16
> > > ACPI: bus type pci registered
> > > bio: create slab <bio-0> at 0
> > > ACPI: Added _OSI(Module Device)
> > > ACPI: Added _OSI(Processor Device)
> > > ACPI: Added _OSI(3.0 _SCP Extensions)
> > > ACPI: Added _OSI(Processor Aggregator Device)
> > > INFO: rcu_sched self-detected stall on CPU
> > > 1: (15000 ticks this GP) idle=001/140000000000001/0
> >
> > OK, this is strange. The stacks below would lead me to believe that
> > the CPUs are idle. But the idle= value above says that RCU believes
> > that this CPU was executing in non-idle process context when the
> > interrupt occurred.
> >
> > OK, time to take a look at the IA64 idle loop. And I don't see any
> > calls to rcu_idle_enter()... Please see below for my best guess as
> > to where to place it and rcu_idle_exit() -- the rule is that there must
> > be no use of RCU read-side critical sections between the call to the
> > rcu_idle_enter() and the rcu_idle_exit(), so you probably know better
> > than I where to put them.
> >
> > void __attribute__((noreturn))
> > cpu_idle (void)
> > {
> > void (*mark_idle)(int) = ia64_mark_idle;
> > int cpu = smp_processor_id();
> >
> > /* endless idle loop with no priority at all */
> > while (1) {
> > rcu_idle_enter(); /* HERE */
> > if (can_do_pal_halt) {
> > current_thread_info()->status &= ~TS_POLLING;
> > /*
> > * TS_POLLING-cleared state must be visible before we
> > * test NEED_RESCHED:
> > */
> > smp_mb();
> > } else {
> > current_thread_info()->status |= TS_POLLING;
> > }
> >
> > if (!need_resched()) {
> > void (*idle)(void);
> > #ifdef CONFIG_SMP
> > min_xtp();
> > #endif
> > rmb();
> > if (mark_idle)
> > (*mark_idle)(1);
> >
> > idle = pm_idle;
> > if (!idle)
> > idle = default_idle;
> > (*idle)();
> > if (mark_idle)
> > (*mark_idle)(0);
> > #ifdef CONFIG_SMP
> > normal_xtp();
> > #endif
> > }
> > rcu_idle_exit(); /* AND HERE */
> > schedule_preempt_disabled();
> > check_pgt_cache();
> > if (cpu_is_offline(cpu))
> > play_dead();
> > }
> > }
> >
> > Without the calls to rcu_idle_enter() and rcu_idle_exit(), RCU has no
> > way of knowing that the CPU is idle, so waits forever for a context
> > switch.
> >
> > Ah, I bet I know what happened... I don't see tick_nohz_idle_enter(),
> > so I would guess that there is no dyntick-idle, so the recent changes in
> > dyntick-idle didn't cause rcu_idle_enter() to be added.
> >
> > I wonder how many other architectures don't do dyntick-idle?
> >
> > Looks like about 12 more. Probably need fixing as well...
>
> Ouch, that's bad. Ok see below for the conversion of other architectures.
>
> While doing this, I realized that most of these archs just use the same
> cpu_idle() function, basically:
>
> void cpu_idle(void)
> {
> while (1) {
> + rcu_idle_enter();
> while (!need_resched())
> do_arch_thing();
> + rcu_idle_exit();
> schedule_preempt_disabled();
> }
> }
>
> So I think it may be worth creating a "simple idle loop" generic function
> for those archs that they can call. This way there is less conversion to do.
>
> Now this is all a regression, so IMO we should first fix the things locally and
> do that generic idle loop later, since it's rather a feature.
>
> Hmm?
Makes sense to me! And the patches look sane, but I must defer to the
arch maintainers.
Thanx, Paul
> I'm cooking the patches.
>
> diff --git a/arch/alpha/kernel/process.c b/arch/alpha/kernel/process.c
> index 153d3fc..2ebf7b5 100644
> --- a/arch/alpha/kernel/process.c
> +++ b/arch/alpha/kernel/process.c
> @@ -28,6 +28,7 @@
> #include <linux/tty.h>
> #include <linux/console.h>
> #include <linux/slab.h>
> +#include <linux/rcupdate.h>
>
> #include <asm/reg.h>
> #include <asm/uaccess.h>
> @@ -50,13 +51,16 @@ cpu_idle(void)
> {
> set_thread_flag(TIF_POLLING_NRFLAG);
>
> + preempt_disable();
> while (1) {
> /* FIXME -- EV6 and LCA45 know how to power down
> the CPU. */
>
> + rcu_idle_enter();
> while (!need_resched())
> cpu_relax();
> - schedule();
> + rcu_idle_exit();
> + schedule_preempt_disabled();
> }
> }
>
> diff --git a/arch/cris/kernel/process.c b/arch/cris/kernel/process.c
> index 66fd017..7f65be6 100644
> --- a/arch/cris/kernel/process.c
> +++ b/arch/cris/kernel/process.c
> @@ -25,6 +25,7 @@
> #include <linux/elfcore.h>
> #include <linux/mqueue.h>
> #include <linux/reboot.h>
> +#include <linux/rcupdate.h>
>
> //#define DEBUG
>
> @@ -74,6 +75,7 @@ void cpu_idle (void)
> {
> /* endless idle loop with no priority at all */
> while (1) {
> + rcu_idle_enter();
> while (!need_resched()) {
> void (*idle)(void);
> /*
> @@ -86,6 +88,7 @@ void cpu_idle (void)
> idle = default_idle;
> idle();
> }
> + rcu_idle_exit();
> schedule_preempt_disabled();
> }
> }
> diff --git a/arch/frv/kernel/process.c b/arch/frv/kernel/process.c
> index ff95f50..2eb7fa5 100644
> --- a/arch/frv/kernel/process.c
> +++ b/arch/frv/kernel/process.c
> @@ -25,6 +25,7 @@
> #include <linux/reboot.h>
> #include <linux/interrupt.h>
> #include <linux/pagemap.h>
> +#include <linux/rcupdate.h>
>
> #include <asm/asm-offsets.h>
> #include <asm/uaccess.h>
> @@ -69,12 +70,14 @@ void cpu_idle(void)
> {
> /* endless idle loop with no priority at all */
> while (1) {
> + rcu_idle_enter();
> while (!need_resched()) {
> check_pgt_cache();
>
> if (!frv_dma_inprogress && idle)
> idle();
> }
> + rcu_idle_exit();
>
> schedule_preempt_disabled();
> }
> diff --git a/arch/h8300/kernel/process.c b/arch/h8300/kernel/process.c
> index 0e9c315..f153ed1 100644
> --- a/arch/h8300/kernel/process.c
> +++ b/arch/h8300/kernel/process.c
> @@ -36,6 +36,7 @@
> #include <linux/reboot.h>
> #include <linux/fs.h>
> #include <linux/slab.h>
> +#include <linux/rcupdate.h>
>
> #include <asm/uaccess.h>
> #include <asm/traps.h>
> @@ -78,8 +79,10 @@ void (*idle)(void) = default_idle;
> void cpu_idle(void)
> {
> while (1) {
> + rcu_idle_enter();
> while (!need_resched())
> idle();
> + rcu_idle_exit();
> schedule_preempt_disabled();
> }
> }
> diff --git a/arch/m32r/kernel/process.c b/arch/m32r/kernel/process.c
> index 3a4a32b2..384e63f 100644
> --- a/arch/m32r/kernel/process.c
> +++ b/arch/m32r/kernel/process.c
> @@ -26,6 +26,7 @@
> #include <linux/ptrace.h>
> #include <linux/unistd.h>
> #include <linux/hardirq.h>
> +#include <linux/rcupdate.h>
>
> #include <asm/io.h>
> #include <asm/uaccess.h>
> @@ -82,6 +83,7 @@ void cpu_idle (void)
> {
> /* endless idle loop with no priority at all */
> while (1) {
> + rcu_idle_enter();
> while (!need_resched()) {
> void (*idle)(void) = pm_idle;
>
> @@ -90,6 +92,7 @@ void cpu_idle (void)
>
> idle();
> }
> + rcu_idle_exit();
> schedule_preempt_disabled();
> }
> }
> diff --git a/arch/m68k/kernel/process.c b/arch/m68k/kernel/process.c
> index c488e3c..ac2892e 100644
> --- a/arch/m68k/kernel/process.c
> +++ b/arch/m68k/kernel/process.c
> @@ -25,6 +25,7 @@
> #include <linux/reboot.h>
> #include <linux/init_task.h>
> #include <linux/mqueue.h>
> +#include <linux/rcupdate.h>
>
> #include <asm/uaccess.h>
> #include <asm/traps.h>
> @@ -75,8 +76,10 @@ void cpu_idle(void)
> {
> /* endless idle loop with no priority at all */
> while (1) {
> + rcu_idle_enter();
> while (!need_resched())
> idle();
> + rcu_idle_exit();
> schedule_preempt_disabled();
> }
> }
> diff --git a/arch/mn10300/kernel/process.c b/arch/mn10300/kernel/process.c
> index 7dab0cd..e9cceba 100644
> --- a/arch/mn10300/kernel/process.c
> +++ b/arch/mn10300/kernel/process.c
> @@ -25,6 +25,7 @@
> #include <linux/err.h>
> #include <linux/fs.h>
> #include <linux/slab.h>
> +#include <linux/rcupdate.h>
> #include <asm/uaccess.h>
> #include <asm/pgtable.h>
> #include <asm/io.h>
> @@ -107,6 +108,7 @@ void cpu_idle(void)
> {
> /* endless idle loop with no priority at all */
> for (;;) {
> + rcu_idle_enter();
> while (!need_resched()) {
> void (*idle)(void);
>
> @@ -121,6 +123,7 @@ void cpu_idle(void)
> }
> idle();
> }
> + rcu_idle_exit();
>
> schedule_preempt_disabled();
> }
> diff --git a/arch/parisc/kernel/process.c b/arch/parisc/kernel/process.c
> index d4b94b3..c54a4db 100644
> --- a/arch/parisc/kernel/process.c
> +++ b/arch/parisc/kernel/process.c
> @@ -48,6 +48,7 @@
> #include <linux/unistd.h>
> #include <linux/kallsyms.h>
> #include <linux/uaccess.h>
> +#include <linux/rcupdate.h>
>
> #include <asm/io.h>
> #include <asm/asm-offsets.h>
> @@ -69,8 +70,10 @@ void cpu_idle(void)
>
> /* endless idle loop with no priority at all */
> while (1) {
> + rcu_idle_enter();
> while (!need_resched())
> barrier();
> + rcu_idle_exit();
> schedule_preempt_disabled();
> check_pgt_cache();
> }
> diff --git a/arch/score/kernel/process.c b/arch/score/kernel/process.c
> index 2707023..637970c 100644
> --- a/arch/score/kernel/process.c
> +++ b/arch/score/kernel/process.c
> @@ -27,6 +27,7 @@
> #include <linux/reboot.h>
> #include <linux/elfcore.h>
> #include <linux/pm.h>
> +#include <linux/rcupdate.h>
>
> void (*pm_power_off)(void);
> EXPORT_SYMBOL(pm_power_off);
> @@ -50,9 +51,10 @@ void __noreturn cpu_idle(void)
> {
> /* endless idle loop with no priority at all */
> while (1) {
> + rcu_idle_enter();
> while (!need_resched())
> barrier();
> -
> + rcu_idle_exit();
> schedule_preempt_disabled();
> }
> }
> diff --git a/arch/xtensa/kernel/process.c b/arch/xtensa/kernel/process.c
> index 2c8d6a3..bc44311 100644
> --- a/arch/xtensa/kernel/process.c
> +++ b/arch/xtensa/kernel/process.c
> @@ -31,6 +31,7 @@
> #include <linux/mqueue.h>
> #include <linux/fs.h>
> #include <linux/slab.h>
> +#include <linux/rcupdate.h>
>
> #include <asm/pgtable.h>
> #include <asm/uaccess.h>
> @@ -110,8 +111,10 @@ void cpu_idle(void)
>
> /* endless idle loop with no priority at all */
> while (1) {
> + rcu_idle_enter();
> while (!need_resched())
> platform_idle();
> + rcu_idle_exit();
> schedule_preempt_disabled();
> }
> }
>
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: ia64 won't boot because of rcu_sched self-detected stall
2012-08-22 0:46 ` Paul E. McKenney
2012-08-22 13:42 ` Luck, Tony
2012-08-22 15:12 ` Frederic Weisbecker
@ 2012-08-23 19:54 ` Luck, Tony
2012-08-24 20:37 ` Paul E. McKenney
2 siblings, 1 reply; 11+ messages in thread
From: Luck, Tony @ 2012-08-23 19:54 UTC (permalink / raw)
To: paulmck; +Cc: linux-next, fweisbec
> Without the calls to rcu_idle_enter() and rcu_idle_exit(), RCU has no
> way of knowing that the CPU is idle, so waits forever for a context
> switch.
Adding the calls at the places you suggested solves the problem. Thanks.
Which tree is feeding these changes to linux-next? How do I get
this ia64 fix into that tree so it will go to Linus in the same merge
that the changes that required this will be in?
Do you want me to create a patch (I can do that, but I'm not sure
that I can write a good commit message). If someone else does,
then it can be marked:
Tested-by: Tony Luck <tony.luck@intel.com>
-Tony
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ia64 won't boot because of rcu_sched self-detected stall
2012-08-23 19:54 ` Luck, Tony
@ 2012-08-24 20:37 ` Paul E. McKenney
2012-08-24 21:44 ` Luck, Tony
0 siblings, 1 reply; 11+ messages in thread
From: Paul E. McKenney @ 2012-08-24 20:37 UTC (permalink / raw)
To: Luck, Tony; +Cc: linux-next, fweisbec
On Thu, Aug 23, 2012 at 07:54:37PM +0000, Luck, Tony wrote:
> > Without the calls to rcu_idle_enter() and rcu_idle_exit(), RCU has no
> > way of knowing that the CPU is idle, so waits forever for a context
> > switch.
>
> Adding the calls at the places you suggested solves the problem. Thanks.
>
> Which tree is feeding these changes to linux-next? How do I get
> this ia64 fix into that tree so it will go to Linus in the same merge
> that the changes that required this will be in?
>
> Do you want me to create a patch (I can do that, but I'm not sure
> that I can write a good commit message). If someone else does,
> then it can be marked:
>
> Tested-by: Tony Luck <tony.luck@intel.com>
Does the following match what you tested? I optimistically assumed
that it was, but figured I should check. ;-)
Thanx, Paul
------------------------------------------------------------------------
ia64: Add missing RCU idle APIs on idle loop
Traditionally, the entire idle task served as an RCU quiescent state.
But when RCU read side critical sections started appearing within the
idle loop, this traditional strategy became untenable. The fix was to
create new RCU APIs named rcu_idle_enter() and rcu_idle_exit(), which
must be called by each architecture's idle loop so that RCU can tell
when it is safe to ignore a given idle CPU.
Unfortunately, this fix was never applied to ia64, a shortcoming remedied
by this commit.
Reported by: Tony Luck <tony.luck@intel.com>
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested by: Tony Luck <tony.luck@intel.com>
diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c
index dd6fc14..3e316ec 100644
--- a/arch/ia64/kernel/process.c
+++ b/arch/ia64/kernel/process.c
@@ -29,6 +29,7 @@
#include <linux/kdebug.h>
#include <linux/utsname.h>
#include <linux/tracehook.h>
+#include <linux/rcupdate.h>
#include <asm/cpu.h>
#include <asm/delay.h>
@@ -279,6 +280,7 @@ cpu_idle (void)
/* endless idle loop with no priority at all */
while (1) {
+ rcu_idle_enter();
if (can_do_pal_halt) {
current_thread_info()->status &= ~TS_POLLING;
/*
@@ -309,6 +311,7 @@ cpu_idle (void)
normal_xtp();
#endif
}
+ rcu_idle_exit();
schedule_preempt_disabled();
check_pgt_cache();
if (cpu_is_offline(cpu))
^ permalink raw reply related [flat|nested] 11+ messages in thread
* RE: ia64 won't boot because of rcu_sched self-detected stall
2012-08-24 20:37 ` Paul E. McKenney
@ 2012-08-24 21:44 ` Luck, Tony
2012-08-24 22:06 ` Paul E. McKenney
0 siblings, 1 reply; 11+ messages in thread
From: Luck, Tony @ 2012-08-24 21:44 UTC (permalink / raw)
To: paulmck; +Cc: linux-next, fweisbec
> Does the following match what you tested? I optimistically assumed
> that it was, but figured I should check. ;-)
Yes. That' what I tested.
> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Split personality disorder?
You finally managed to clone yourself to help cover RCU issues?
Cunning way to get double credit for each patch?
The lawyers made you do it?
-Tony
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ia64 won't boot because of rcu_sched self-detected stall
2012-08-24 21:44 ` Luck, Tony
@ 2012-08-24 22:06 ` Paul E. McKenney
0 siblings, 0 replies; 11+ messages in thread
From: Paul E. McKenney @ 2012-08-24 22:06 UTC (permalink / raw)
To: Luck, Tony; +Cc: linux-next, fweisbec
On Fri, Aug 24, 2012 at 09:44:39PM +0000, Luck, Tony wrote:
> > Does the following match what you tested? I optimistically assumed
> > that it was, but figured I should check. ;-)
>
> Yes. That' what I tested.
Very good, thank you!
> > Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
> Split personality disorder?
> You finally managed to clone yourself to help cover RCU issues?
> Cunning way to get double credit for each patch?
> The lawyers made you do it?
And just why does everyone insist on calling split personality a disorder,
anyway???
No cloning, at best, clowning. Jon and Greg spiked the two-email dodge
long before I started using two, so currently each of my patches,
regardless of how it is signed off, is counted but once and credited to
Linaro. And for once, no lawyers were involved.
Hey, you asked!!! ;-)
Thanx, Paul
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-08-24 22:07 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-08-21 22:45 ia64 won't boot because of rcu_sched self-detected stall Tony Luck
2012-08-21 23:20 ` Paul E. McKenney
2012-08-21 23:53 ` Luck, Tony
2012-08-22 0:46 ` Paul E. McKenney
2012-08-22 13:42 ` Luck, Tony
2012-08-22 15:12 ` Frederic Weisbecker
2012-08-22 15:29 ` Paul E. McKenney
2012-08-23 19:54 ` Luck, Tony
2012-08-24 20:37 ` Paul E. McKenney
2012-08-24 21:44 ` Luck, Tony
2012-08-24 22:06 ` Paul E. McKenney
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).