All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant
@ 2012-10-19  9:56 Linus Walleij
  2012-10-19 16:15 ` Shawn Guo
  2012-10-22  5:13 ` Shawn Guo
  0 siblings, 2 replies; 6+ messages in thread
From: Linus Walleij @ 2012-10-19  9:56 UTC (permalink / raw)
  To: linux-arm-kernel

This makes the SMP_TWD clock .setup()/.stop() pair reentrant by
not re-fetching the clk and re-registering the clock event every
time .setup() is called. We also make sure to call the
clk_enable()/clk_disable() pair on subsequent calls.

As it has been brought to my knowledge that this pair is going
to be called from atomic contexts for CPU clusters coming and
going, the clk_prepare()/clk_unprepare() calls cannot be called
on subsequent .setup()/.stop() iterations.

The patch assumes that the code will make sure that
twd_set_mode() is called through .set_mode() on the clock
event *after* the .setup() call, so that the timer registers
are fully re-programmed after a new .setup() cycle.

Cc: Shawn Guo <shawn.guo@linaro.org>
Reported-by: Peter Chen <peter.chen@freescale.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
---
Peter/Shawn: can you please respond with a Tested-by from your
system(s) to indicate if this works as expected?
---
 arch/arm/kernel/smp_twd.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/arch/arm/kernel/smp_twd.c b/arch/arm/kernel/smp_twd.c
index b92d524..229231a 100644
--- a/arch/arm/kernel/smp_twd.c
+++ b/arch/arm/kernel/smp_twd.c
@@ -31,6 +31,7 @@ static void __iomem *twd_base;
 
 static struct clk *twd_clk;
 static unsigned long twd_timer_rate;
+static bool initial_setup_called;
 
 static struct clock_event_device __percpu **twd_evt;
 static int twd_ppi;
@@ -93,6 +94,8 @@ static void twd_timer_stop(struct clock_event_device *clk)
 {
 	twd_set_mode(CLOCK_EVT_MODE_UNUSED, clk);
 	disable_percpu_irq(clk->irq);
+	if (!IS_ERR(twd_clk))
+		clk_disable(twd_clk);
 }
 
 #ifdef CONFIG_COMMON_CLK
@@ -265,8 +268,21 @@ static int __cpuinit twd_timer_setup(struct clock_event_device *clk)
 {
 	struct clock_event_device **this_cpu_clk;
 
-	if (!twd_clk)
-		twd_clk = twd_get_clock();
+	/*
+	 * If the basic setup has been done before, don't bother
+	 * with yet again looking up the clock and register the clock
+	 * source.
+	 */
+	if (initial_setup_called) {
+		if (!IS_ERR(twd_clk))
+			clk_enable(twd_clk);
+		__raw_writel(0, twd_base + TWD_TIMER_CONTROL);
+		enable_percpu_irq(clk->irq, 0);
+		return 0;
+	}
+	initial_setup_called = true;
+
+	twd_clk = twd_get_clock();
 
 	if (!IS_ERR_OR_NULL(twd_clk))
 		twd_timer_rate = clk_get_rate(twd_clk);
-- 
1.7.11.7

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant
  2012-10-19  9:56 [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant Linus Walleij
@ 2012-10-19 16:15 ` Shawn Guo
  2012-10-22  5:13 ` Shawn Guo
  1 sibling, 0 replies; 6+ messages in thread
From: Shawn Guo @ 2012-10-19 16:15 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Oct 19, 2012 at 11:56:29AM +0200, Linus Walleij wrote:
> This makes the SMP_TWD clock .setup()/.stop() pair reentrant by
> not re-fetching the clk and re-registering the clock event every
> time .setup() is called. We also make sure to call the
> clk_enable()/clk_disable() pair on subsequent calls.
> 
> As it has been brought to my knowledge that this pair is going
> to be called from atomic contexts for CPU clusters coming and
> going, the clk_prepare()/clk_unprepare() calls cannot be called
> on subsequent .setup()/.stop() iterations.
> 
> The patch assumes that the code will make sure that
> twd_set_mode() is called through .set_mode() on the clock
> event *after* the .setup() call, so that the timer registers
> are fully re-programmed after a new .setup() cycle.
> 
> Cc: Shawn Guo <shawn.guo@linaro.org>
> Reported-by: Peter Chen <peter.chen@freescale.com>
> Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
> ---
> Peter/Shawn: can you please respond with a Tested-by from your
> system(s) to indicate if this works as expected?

Yes, it fixes the sleep-in-atomic warning.  However I'm having some
issue with the resume function now.  See below for details.

It's getting late here.  I will investigate it tomorrow if I do not
anything back from you.

Shawn

$ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
PM: Preparing system for mem sleep
mmc1: card e1c3 removed
Freezing user space processes ... (elapsed 0.01 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.03 seconds) done.
PM: Entering mem sleep
PM: suspend of devices complete after 4.651 msecs
PM: suspend devices took 0.010 seconds
PM: late suspend of devices complete after 1.437 msecs
PM: noirq suspend of devices complete after 2.167 msecs
Disabling non-boot CPUs ...
CPU1: shutdown
CPU2: shutdown
CPU3: shutdown
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up
PM: noirq resume of devices complete after 1.030 msecs
PM: early resume of devices complete after 1.553 msecs
PM: resume of devices complete after 4.179 msecs
PM: resume devices took 0.010 seconds
PM: Finishing wakeup.
Restarting tasks ... done.
mmc1: new high speed SDHC card at address e1c3
mmcblk0: mmc1:e1c3 SD04G 3.69 GiB
 mmcblk0: p1 p2 p3
libphy: 2188000.ethernet:01 - Link is Down
libphy: 2188000.ethernet:01 - Link is Up - 100/Full
INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=6002 jiffies)
Backtrace:
[<80011e74>] (dump_backtrace+0x0/0x10c) from [<804d8d84>] (dump_stack+0x18/0x1c)
 r6:8069db00 r5:806b4800 r4:8144bb00 r3:00000000
[<804d8d6c>] (dump_stack+0x0/0x1c) from [<80080d78>] (rcu_check_callbacks+0x598/0x624)
[<800807e0>] (rcu_check_callbacks+0x0/0x624) from [<8002e31c>] (update_process_times+0x40/0x54)
[<8002e2dc>] (update_process_times+0x0/0x54) from [<8005f928>] (tick_sched_timer+0x88/0xec)
 r6:8144ba18 r5:00000014 r4:9c341e08 r3:20000013
[<8005f8a0>] (tick_sched_timer+0x0/0xec) from [<80043214>] (__run_hrtimer.isra.18+0x4c/0xe0)
 r8:9c341767 r7:8005f8a0 r6:8144b880 r5:8144b8d0 r4:8144ba18
[<800431c8>] (__run_hrtimer.isra.18+0x0/0xe0) from [<80043c68>] (hrtimer_interrupt+0x120/0x2c0)
 r7:00000001 r6:8144b880 r5:00000000 r4:8144b8d0
[<80043b48>] (hrtimer_interrupt+0x0/0x2c0) from [<8001401c>] (twd_handler+0x34/0x48)
[<80013fe8>] (twd_handler+0x0/0x48) from [<8007b5e8>] (handle_percpu_devid_irq+0x88/0xa8)
 r4:bf807600 r3:80013fe8
[<8007b560>] (handle_percpu_devid_irq+0x0/0xa8) from [<80077cb8>] (generic_handle_irq+0x28/0x38)
 r8:00000000 r7:0000001d r6:806a0000 r5:8069e02c r4:0000001d
r3:8007b560
[<80077c90>] (generic_handle_irq+0x0/0x38) from [<8000ee94>] (handle_IRQ+0x54/0xb4)
 r4:806a9258 r3:00000180
[<8000ee40>] (handle_IRQ+0x0/0xb4) from [<80008504>] (gic_handle_irq+0x30/0x64)
 r8:806ad048 r7:f4000100 r6:806a1f00 r5:806a8970 r4:f400010c
r3:00000000
[<800084d4>] (gic_handle_irq+0x0/0x64) from [<8000e124>] (__irq_svc+0x44/0x5c)
Exception stack(0x806a1f00 to 0x806a1f48)
1f00: 00000001 00000001 00000000 806abd48 806a0000 806a0000 806e9088 804e2aa4
1f20: 806ad048 806a0000 00000000 806a1f54 806a1f18 806a1f48 8006629c 8000f1c0
1f40: 20000013 ffffffff
 r7:806a1f34 r6:ffffffff r5:20000013 r4:8000f1c0
[<8000f198>] (default_idle+0x0/0x44) from [<8000f338>] (cpu_idle+0x54/0x104)
[<8000f2e4>] (cpu_idle+0x0/0x104) from [<804c8534>] (rest_init+0xc4/0xec)
[<804c8470>] (rest_init+0x0/0xec) from [<8066183c>] (start_kernel+0x2bc/0x30c)
 r7:80691f50 r6:806e8fc0 r5:ffffffff r4:806a8f90
[<80661580>] (start_kernel+0x0/0x30c) from [<10008044>] (0x10008044)

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant
  2012-10-19  9:56 [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant Linus Walleij
  2012-10-19 16:15 ` Shawn Guo
@ 2012-10-22  5:13 ` Shawn Guo
  2012-10-22  6:51   ` Santosh Shilimkar
  2012-10-22 10:14   ` Linus Walleij
  1 sibling, 2 replies; 6+ messages in thread
From: Shawn Guo @ 2012-10-22  5:13 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Oct 19, 2012 at 11:56:29AM +0200, Linus Walleij wrote:
> This makes the SMP_TWD clock .setup()/.stop() pair reentrant by
> not re-fetching the clk and re-registering the clock event every
> time .setup() is called. We also make sure to call the
> clk_enable()/clk_disable() pair on subsequent calls.
> 
> As it has been brought to my knowledge that this pair is going
> to be called from atomic contexts for CPU clusters coming and
> going, the clk_prepare()/clk_unprepare() calls cannot be called
> on subsequent .setup()/.stop() iterations.
> 
> The patch assumes that the code will make sure that
> twd_set_mode() is called through .set_mode() on the clock
> event *after* the .setup() call, so that the timer registers
> are fully re-programmed after a new .setup() cycle.
> 
> Cc: Shawn Guo <shawn.guo@linaro.org>
> Reported-by: Peter Chen <peter.chen@freescale.com>
> Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
> ---
> Peter/Shawn: can you please respond with a Tested-by from your
> system(s) to indicate if this works as expected?

I have seen two problems with the patch.

1. twd_timer_setup() is called on per-cpu path, and initial_setup_called
   should be a per-cpu variable.

2. When secondary cores get off-line, the clockevent devices for
   the cores will be released.  So when they become online, the
   clockevent devices should be registered again.

I can only have my system work as expected with the following changes.

Shawn

8<----

diff --git a/arch/arm/kernel/smp_twd.c b/arch/arm/kernel/smp_twd.c
index 5c2d85b..b826bf0 100644
--- a/arch/arm/kernel/smp_twd.c
+++ b/arch/arm/kernel/smp_twd.c
@@ -31,7 +31,7 @@ static void __iomem *twd_base;

 static struct clk *twd_clk;
 static unsigned long twd_timer_rate;
-static bool initial_setup_called;
+static DEFINE_PER_CPU(bool, initial_setup_called);

 static struct clock_event_device __percpu **twd_evt;
 static int twd_ppi;
@@ -275,20 +275,22 @@ static struct clk *twd_get_clock(void)
 static int __cpuinit twd_timer_setup(struct clock_event_device *clk)
 {
        struct clock_event_device **this_cpu_clk;
+       int cpu = smp_processor_id();

        /*
         * If the basic setup has been done before, don't bother
         * with yet again looking up the clock and register the clock
         * source.
         */
-       if (initial_setup_called) {
+       if (per_cpu(initial_setup_called, cpu)) {
                if (!IS_ERR(twd_clk))
                        clk_enable(twd_clk);
                __raw_writel(0, twd_base + TWD_TIMER_CONTROL);
+               clockevents_register_device(*__this_cpu_ptr(twd_evt));
                enable_percpu_irq(clk->irq, 0);
                return 0;
        }
-       initial_setup_called = true;
+       per_cpu(initial_setup_called, cpu) = true;

        twd_clk = twd_get_clock();

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant
  2012-10-22  5:13 ` Shawn Guo
@ 2012-10-22  6:51   ` Santosh Shilimkar
  2012-10-22 10:14   ` Linus Walleij
  1 sibling, 0 replies; 6+ messages in thread
From: Santosh Shilimkar @ 2012-10-22  6:51 UTC (permalink / raw)
  To: linux-arm-kernel

Linus,

On Monday 22 October 2012 10:43 AM, Shawn Guo wrote:
> On Fri, Oct 19, 2012 at 11:56:29AM +0200, Linus Walleij wrote:
>> This makes the SMP_TWD clock .setup()/.stop() pair reentrant by
>> not re-fetching the clk and re-registering the clock event every
>> time .setup() is called. We also make sure to call the
>> clk_enable()/clk_disable() pair on subsequent calls.
>>
>> As it has been brought to my knowledge that this pair is going
>> to be called from atomic contexts for CPU clusters coming and
>> going, the clk_prepare()/clk_unprepare() calls cannot be called
>> on subsequent .setup()/.stop() iterations.
>>
>> The patch assumes that the code will make sure that
>> twd_set_mode() is called through .set_mode() on the clock
>> event *after* the .setup() call, so that the timer registers
>> are fully re-programmed after a new .setup() cycle.
>>
>> Cc: Shawn Guo <shawn.guo@linaro.org>
>> Reported-by: Peter Chen <peter.chen@freescale.com>
>> Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
>> ---
>> Peter/Shawn: can you please respond with a Tested-by from your
>> system(s) to indicate if this works as expected?
>
> I have seen two problems with the patch.
>
> 1. twd_timer_setup() is called on per-cpu path, and initial_setup_called
>     should be a per-cpu variable.
>
> 2. When secondary cores get off-line, the clockevent devices for
>     the cores will be released.  So when they become online, the
>     clockevent devices should be registered again.
>
> I can only have my system work as expected with the following changes.
>

How about moving twd_get_clock() outside setup and do that in onetime
init code like register etc. Its is pointless to keep doing
re-registration, clock_enable/disable() since you can not really
disable this clock. Remember we added this clock node for CPUFREQ
and it is just CPU clock with an additional fixed divider.

With above, you neither see the sleep while atomic warning nor
there is any need to fiddle with clock node after boot.

Let me know if I have missed any other consideration here.

Regards,
Santosh

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant
  2012-10-22  5:13 ` Shawn Guo
  2012-10-22  6:51   ` Santosh Shilimkar
@ 2012-10-22 10:14   ` Linus Walleij
  2012-10-22 14:14     ` Shawn Guo
  1 sibling, 1 reply; 6+ messages in thread
From: Linus Walleij @ 2012-10-22 10:14 UTC (permalink / raw)
  To: linux-arm-kernel

On Mon, Oct 22, 2012 at 7:13 AM, Shawn Guo <shawn.guo@linaro.org> wrote:

> I have seen two problems with the patch.
>
> 1. twd_timer_setup() is called on per-cpu path, and initial_setup_called
>    should be a per-cpu variable.

OK folded this into my v3 patch. I was not properly multithreaded
in my head...

> 2. When secondary cores get off-line, the clockevent devices for
>    the cores will be released.  So when they become online, the
>    clockevent devices should be registered again.

So that is how it happens. Where is the code releasing all
clock events? I was thinking this would more properly be done
in the .stop() path of the SMP TWD driver?

Right now it's not looking good because it "just so happens"
that the clockevent silently goes away in the cycle where
.stop() is called. :-(

Yours,
Linus Walleij

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant
  2012-10-22 10:14   ` Linus Walleij
@ 2012-10-22 14:14     ` Shawn Guo
  0 siblings, 0 replies; 6+ messages in thread
From: Shawn Guo @ 2012-10-22 14:14 UTC (permalink / raw)
  To: linux-arm-kernel

On Mon, Oct 22, 2012 at 12:14:03PM +0200, Linus Walleij wrote:
> > 2. When secondary cores get off-line, the clockevent devices for
> >    the cores will be released.  So when they become online, the
> >    clockevent devices should be registered again.
> 
> So that is how it happens. Where is the code releasing all
> clock events? I was thinking this would more properly be done
> in the .stop() path of the SMP TWD driver?
> 
tick_notify() will receive CLOCK_EVT_NOTIFY_CPU_DEAD when a cpu goes
away.  It results in tick_shutdown() invoking which will release the
clockevent device for the cpu.

Shawn

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2012-10-22 14:14 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-10-19  9:56 [PATCH v2] ARM: SMP_TWD: make setup()/stop() reentrant Linus Walleij
2012-10-19 16:15 ` Shawn Guo
2012-10-22  5:13 ` Shawn Guo
2012-10-22  6:51   ` Santosh Shilimkar
2012-10-22 10:14   ` Linus Walleij
2012-10-22 14:14     ` Shawn Guo

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.