Delays, clocks, timers, hrtimers, etc

Delays, clocks, timers, hrtimers, etc

[Mason]

Hello,

I am swimming in a sea of confusion, and am hoping someone would toss
me a life-jacket (of enlightenment). Please forgive me if some of my
questions are poorly asked or appear in seemingly random order.

Working on a Cortex A9 based SoC, I set out to "clean up" the platform
specific timer code, by using as much generic framework as possible.
(Right now, there's a lot of "redundant" code in the mach dir.)


Q1. the {n,u,m}delay function family

arch/arm/include/asm/delay.h mentions
"Delay routines, using a pre-computed "loops_per_second" value."
*BUT* if the frequency changes dynamically (thanks to cpufreq)
the "loops_per_second" value cannot be pre-computed, as it would
change dynamically too, right?

Looking at arch/arm/lib/delay.c it seems the default implementation
is a busy loop (in delay-loop.S) which looks up "loops_per_jiffy"
in the prolog to determine the number of times to loop, right?

http://lxr.free-electrons.com/source/arch/arm/lib/delay-loop.S

(Side issue, why is the loop unrolled in __loop_delay? What is the
point of unrolling a busy loop? This is commented code however.)

What happens if loops_per_jiffy changes while one core is in the
busy loop? It seems we might exit the loop too early, which could
break some drivers with some weird heisenbug, no?

Also, is the update of loops_per_jiffy atomic? Is it possible that
if one core reads it while another updates it, we get garbage?

I suppose this is one reason why the default functions are overridden
by register_current_timer_delay(&arch_delay_timer) right? I think the
property of a timer is that its frequency doesn't change, even if the
CPU's frequency changes? So we are still busy looping, but we are
checking the actual time spent in the loop, whatever the cpufreq?

Reference
https://www.kernel.org/doc/Documentation/timers/timers-howto.txt


Q2. Cortex A9 global and private timers

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html

(What are private timers used for?)

In my platform-specific code, there is a config option to choose between

1) the ARM global timer
2) a platform-specific timer (timer0)

I noticed that there is generic code to support the global timer in
drivers/clocksource/arm_global_timer.c

config ARM_GLOBAL_TIMER
	bool
	select CLKSRC_OF if OF
	help
	  This options enables support for the ARM global timer unit

config CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
	bool
	depends on ARM_GLOBAL_TIMER
	default y
	help
	 Use ARM global timer clock source as sched_clock

I was thinking it would be better to use the "standard" option (ARM global timer)
as it is "officially" supported in the vanilla kernel. So less code to write and
to debug, and it has likely received more testing. Why would one rely on
platform-specific timers then?

Are high-resolution timers supported with the global timer?


Q3. Using the generic global timer implementation

So, how do I use that implementation?
(Is someone other than STMicro using it?)

I see:

static void __init global_timer_of_register(struct device_node *np)
CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer", global_timer_of_register);

OF stands for open firmware, yes?
So is this related to device tree?

http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/arm/global_timer.txt

This file makes no sense to me.

- interrupts : One interrupt to each core
interrupts = <1 13 0xf01>;
what are 1 13 0xf01 ??

- clocks : Should be phandle to a clock.
clocks = <&arm_periph_clk>;

For my (old) 3.14 kernel, I found this:

     /*
      * ARM Peripheral clock for timers
      */
     arm_periph_clk: arm_periph_clk {
       #clock-cells = <0>;
       compatible = "fixed-clock";
       clock-frequency = <600000000>;
     };

But it looks like the definitions have moved around since then?

This device tree concept is too much to swallow in a single serving.
Please tell me if I'm going down the correct rabbit hole, and I'll
do some LWN readings to try to wrap my mind around the concept.


Anyway, if anyone can help me out on some of these topics, I'd be
eternally grateful.

Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Hello,

I am swimming in a sea of confusion, and am hoping someone would toss
me a life-jacket (of enlightenment). Please forgive me if some of my
questions are poorly asked or appear in seemingly random order.

Working on a Cortex A9 based SoC, I set out to "clean up" the platform
specific timer code, by using as much generic framework as possible.
(Right now, there's a lot of "redundant" code in the mach dir.)


Q1. the {n,u,m}delay function family

arch/arm/include/asm/delay.h mentions
"Delay routines, using a pre-computed "loops_per_second" value."
*BUT* if the frequency changes dynamically (thanks to cpufreq)
the "loops_per_second" value cannot be pre-computed, as it would
change dynamically too, right?

Looking at arch/arm/lib/delay.c it seems the default implementation
is a busy loop (in delay-loop.S) which looks up "loops_per_jiffy"
in the prolog to determine the number of times to loop, right?

http://lxr.free-electrons.com/source/arch/arm/lib/delay-loop.S

(Side issue, why is the loop unrolled in __loop_delay? What is the
point of unrolling a busy loop? This is commented code however.)

What happens if loops_per_jiffy changes while one core is in the
busy loop? It seems we might exit the loop too early, which could
break some drivers with some weird heisenbug, no?

Also, is the update of loops_per_jiffy atomic? Is it possible that
if one core reads it while another updates it, we get garbage?

I suppose this is one reason why the default functions are overridden
by register_current_timer_delay(&arch_delay_timer) right? I think the
property of a timer is that its frequency doesn't change, even if the
CPU's frequency changes? So we are still busy looping, but we are
checking the actual time spent in the loop, whatever the cpufreq?

Reference
https://www.kernel.org/doc/Documentation/timers/timers-howto.txt


Q2. Cortex A9 global and private timers

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html

(What are private timers used for?)

In my platform-specific code, there is a config option to choose between

1) the ARM global timer
2) a platform-specific timer (timer0)

I noticed that there is generic code to support the global timer in
drivers/clocksource/arm_global_timer.c

config ARM_GLOBAL_TIMER
	bool
	select CLKSRC_OF if OF
	help
	  This options enables support for the ARM global timer unit

config CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
	bool
	depends on ARM_GLOBAL_TIMER
	default y
	help
	 Use ARM global timer clock source as sched_clock

I was thinking it would be better to use the "standard" option (ARM global timer)
as it is "officially" supported in the vanilla kernel. So less code to write and
to debug, and it has likely received more testing. Why would one rely on
platform-specific timers then?

Are high-resolution timers supported with the global timer?


Q3. Using the generic global timer implementation

So, how do I use that implementation?
(Is someone other than STMicro using it?)

I see:

static void __init global_timer_of_register(struct device_node *np)
CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer", global_timer_of_register);

OF stands for open firmware, yes?
So is this related to device tree?

http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/arm/global_timer.txt

This file makes no sense to me.

- interrupts : One interrupt to each core
interrupts = <1 13 0xf01>;
what are 1 13 0xf01 ??

- clocks : Should be phandle to a clock.
clocks = <&arm_periph_clk>;

For my (old) 3.14 kernel, I found this:

     /*
      * ARM Peripheral clock for timers
      */
     arm_periph_clk: arm_periph_clk {
       #clock-cells = <0>;
       compatible = "fixed-clock";
       clock-frequency = <600000000>;
     };

But it looks like the definitions have moved around since then?

This device tree concept is too much to swallow in a single serving.
Please tell me if I'm going down the correct rabbit hole, and I'll
do some LWN readings to try to wrap my mind around the concept.


Anyway, if anyone can help me out on some of these topics, I'd be
eternally grateful.

Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

[ I am aware that my message is way too long, and that few people would have
the time to answer all these questions. So maybe, if someone feels inclined
to answer just one or two, that might kickstart some discussion, and I might
learn something along the way. Regards. ]

FTR, I've been reading about DeviceTree:

http://lwn.net/Articles/573409/
http://www.carbondesignsystems.com/virtual-prototype-blog/bid/195122/Running-the-Latest-Linux-Kernel-on-a-Minimal-ARM-Cortex-A15-System
http://devicetree.org/Device_Tree_Usage

And I am resisting the urge to pile on a few more questions ;-/

Regards.

On 28/01/2015 14:16, Mason wrote:
> Hello,
>
> I am swimming in a sea of confusion, and am hoping someone would toss
> me a life-jacket (of enlightenment). Please forgive me if some of my
> questions are poorly asked or appear in seemingly random order.
>
> Working on a Cortex A9 based SoC, I set out to "clean up" the platform
> specific timer code, by using as much generic framework as possible.
> (Right now, there's a lot of "redundant" code in the mach dir.)
>
>
> Q1. the {n,u,m}delay function family
>
> arch/arm/include/asm/delay.h mentions
> "Delay routines, using a pre-computed "loops_per_second" value."
> *BUT* if the frequency changes dynamically (thanks to cpufreq)
> the "loops_per_second" value cannot be pre-computed, as it would
> change dynamically too, right?
>
> Looking at arch/arm/lib/delay.c it seems the default implementation
> is a busy loop (in delay-loop.S) which looks up "loops_per_jiffy"
> in the prolog to determine the number of times to loop, right?
>
> http://lxr.free-electrons.com/source/arch/arm/lib/delay-loop.S
>
> (Side issue, why is the loop unrolled in __loop_delay? What is the
> point of unrolling a busy loop? This is commented code however.)
>
> What happens if loops_per_jiffy changes while one core is in the
> busy loop? It seems we might exit the loop too early, which could
> break some drivers with some weird heisenbug, no?
>
> Also, is the update of loops_per_jiffy atomic? Is it possible that
> if one core reads it while another updates it, we get garbage?
>
> I suppose this is one reason why the default functions are overridden
> by register_current_timer_delay(&arch_delay_timer) right? I think the
> property of a timer is that its frequency doesn't change, even if the
> CPU's frequency changes? So we are still busy looping, but we are
> checking the actual time spent in the loop, whatever the cpufreq?
>
> Reference
> https://www.kernel.org/doc/Documentation/timers/timers-howto.txt
>
>
> Q2. Cortex A9 global and private timers
>
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html
>
> (What are private timers used for?)
>
> In my platform-specific code, there is a config option to choose between
>
> 1) the ARM global timer
> 2) a platform-specific timer (timer0)
>
> I noticed that there is generic code to support the global timer in
> drivers/clocksource/arm_global_timer.c
>
> config ARM_GLOBAL_TIMER
>      bool
>      select CLKSRC_OF if OF
>      help
>        This options enables support for the ARM global timer unit
>
> config CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
>      bool
>      depends on ARM_GLOBAL_TIMER
>      default y
>      help
>       Use ARM global timer clock source as sched_clock
>
> I was thinking it would be better to use the "standard" option (ARM global timer)
> as it is "officially" supported in the vanilla kernel. So less code to write and
> to debug, and it has likely received more testing. Why would one rely on
> platform-specific timers then?
>
> Are high-resolution timers supported with the global timer?
>
>
> Q3. Using the generic global timer implementation
>
> So, how do I use that implementation?
> (Is someone other than STMicro using it?)
>
> I see:
>
> static void __init global_timer_of_register(struct device_node *np)
> CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer", global_timer_of_register);
>
> OF stands for open firmware, yes?
> So is this related to device tree?
>
> http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/arm/global_timer.txt
>
> This file makes no sense to me.
>
> - interrupts : One interrupt to each core
> interrupts = <1 13 0xf01>;
> what are 1 13 0xf01 ??
>
> - clocks : Should be phandle to a clock.
> clocks = <&arm_periph_clk>;
>
> For my (old) 3.14 kernel, I found this:
>
>      /*
>       * ARM Peripheral clock for timers
>       */
>      arm_periph_clk: arm_periph_clk {
>        #clock-cells = <0>;
>        compatible = "fixed-clock";
>        clock-frequency = <600000000>;
>      };
>
> But it looks like the definitions have moved around since then?
>
> This device tree concept is too much to swallow in a single serving.
> Please tell me if I'm going down the correct rabbit hole, and I'll
> do some LWN readings to try to wrap my mind around the concept.
>
>
> Anyway, if anyone can help me out on some of these topics, I'd be
> eternally grateful.
>
> Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

[ I am aware that my message is way too long, and that few people would have
the time to answer all these questions. So maybe, if someone feels inclined
to answer just one or two, that might kickstart some discussion, and I might
learn something along the way. Regards. ]

FTR, I've been reading about DeviceTree:

http://lwn.net/Articles/573409/
http://www.carbondesignsystems.com/virtual-prototype-blog/bid/195122/Running-the-Latest-Linux-Kernel-on-a-Minimal-ARM-Cortex-A15-System
http://devicetree.org/Device_Tree_Usage

And I am resisting the urge to pile on a few more questions ;-/

Regards.

On 28/01/2015 14:16, Mason wrote:
> Hello,
>
> I am swimming in a sea of confusion, and am hoping someone would toss
> me a life-jacket (of enlightenment). Please forgive me if some of my
> questions are poorly asked or appear in seemingly random order.
>
> Working on a Cortex A9 based SoC, I set out to "clean up" the platform
> specific timer code, by using as much generic framework as possible.
> (Right now, there's a lot of "redundant" code in the mach dir.)
>
>
> Q1. the {n,u,m}delay function family
>
> arch/arm/include/asm/delay.h mentions
> "Delay routines, using a pre-computed "loops_per_second" value."
> *BUT* if the frequency changes dynamically (thanks to cpufreq)
> the "loops_per_second" value cannot be pre-computed, as it would
> change dynamically too, right?
>
> Looking at arch/arm/lib/delay.c it seems the default implementation
> is a busy loop (in delay-loop.S) which looks up "loops_per_jiffy"
> in the prolog to determine the number of times to loop, right?
>
> http://lxr.free-electrons.com/source/arch/arm/lib/delay-loop.S
>
> (Side issue, why is the loop unrolled in __loop_delay? What is the
> point of unrolling a busy loop? This is commented code however.)
>
> What happens if loops_per_jiffy changes while one core is in the
> busy loop? It seems we might exit the loop too early, which could
> break some drivers with some weird heisenbug, no?
>
> Also, is the update of loops_per_jiffy atomic? Is it possible that
> if one core reads it while another updates it, we get garbage?
>
> I suppose this is one reason why the default functions are overridden
> by register_current_timer_delay(&arch_delay_timer) right? I think the
> property of a timer is that its frequency doesn't change, even if the
> CPU's frequency changes? So we are still busy looping, but we are
> checking the actual time spent in the loop, whatever the cpufreq?
>
> Reference
> https://www.kernel.org/doc/Documentation/timers/timers-howto.txt
>
>
> Q2. Cortex A9 global and private timers
>
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html
>
> (What are private timers used for?)
>
> In my platform-specific code, there is a config option to choose between
>
> 1) the ARM global timer
> 2) a platform-specific timer (timer0)
>
> I noticed that there is generic code to support the global timer in
> drivers/clocksource/arm_global_timer.c
>
> config ARM_GLOBAL_TIMER
>      bool
>      select CLKSRC_OF if OF
>      help
>        This options enables support for the ARM global timer unit
>
> config CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
>      bool
>      depends on ARM_GLOBAL_TIMER
>      default y
>      help
>       Use ARM global timer clock source as sched_clock
>
> I was thinking it would be better to use the "standard" option (ARM global timer)
> as it is "officially" supported in the vanilla kernel. So less code to write and
> to debug, and it has likely received more testing. Why would one rely on
> platform-specific timers then?
>
> Are high-resolution timers supported with the global timer?
>
>
> Q3. Using the generic global timer implementation
>
> So, how do I use that implementation?
> (Is someone other than STMicro using it?)
>
> I see:
>
> static void __init global_timer_of_register(struct device_node *np)
> CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer", global_timer_of_register);
>
> OF stands for open firmware, yes?
> So is this related to device tree?
>
> http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/arm/global_timer.txt
>
> This file makes no sense to me.
>
> - interrupts : One interrupt to each core
> interrupts = <1 13 0xf01>;
> what are 1 13 0xf01 ??
>
> - clocks : Should be phandle to a clock.
> clocks = <&arm_periph_clk>;
>
> For my (old) 3.14 kernel, I found this:
>
>      /*
>       * ARM Peripheral clock for timers
>       */
>      arm_periph_clk: arm_periph_clk {
>        #clock-cells = <0>;
>        compatible = "fixed-clock";
>        clock-frequency = <600000000>;
>      };
>
> But it looks like the definitions have moved around since then?
>
> This device tree concept is too much to swallow in a single serving.
> Please tell me if I'm going down the correct rabbit hole, and I'll
> do some LWN readings to try to wrap my mind around the concept.
>
>
> Anyway, if anyone can help me out on some of these topics, I'd be
> eternally grateful.
>
> Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Russell King - ARM Linux]

On Wed, Jan 28, 2015 at 02:16:21PM +0100, Mason wrote:
> Q1. the {n,u,m}delay function family
> 
> arch/arm/include/asm/delay.h mentions
> "Delay routines, using a pre-computed "loops_per_second" value."
> *BUT* if the frequency changes dynamically (thanks to cpufreq)
> the "loops_per_second" value cannot be pre-computed, as it would
> change dynamically too, right?

cpufreq changes the loops_per_second value, but an already in-progress
delay doesn't see that (new delays will see the update though.)

> Also, is the update of loops_per_jiffy atomic? Is it possible that
> if one core reads it while another updates it, we get garbage?

32-bit reads and writes are atomic.  You read either the old value or
the new value.  There's no inbetween.

> I suppose this is one reason why the default functions are overridden
> by register_current_timer_delay(&arch_delay_timer) right? I think the
> property of a timer is that its frequency doesn't change, even if the
> CPU's frequency changes? So we are still busy looping, but we are
> checking the actual time spent in the loop, whatever the cpufreq?

Timers are preferred because of the problems with the software delay loop.

Note that it has always been the case that the software delay loop is
"approximate" - even without cpufreq etc, the loops_per_jiffy is slightly
on the small side because of the way the calibration works.  It's about
98% of the actual value, and depends on the workload of the timer
interrupt.  It's obvious when you think about it - it's counting the
number of cycles between two timer interrupts, and the timer interrupt
consumes some of the cycles.

This means that even if you ask for a 10us delay, you'll probably get a
delay of 9.8us instead.

> Q2. Cortex A9 global and private timers
> 
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html
> 
> (What are private timers used for?)

The per-cpu private timers are mostly scheduling of threads.

> In my platform-specific code, there is a config option to choose between
> 
> 1) the ARM global timer
> 2) a platform-specific timer (timer0)

Most platforms implement their own timer, because its really sexy for
hardware engineers to create yet another different timer implementation
which is soo much better than every other timer implementation that has
already been created.  You wouldn't believe how many different ways that
there are to create a timer - and we still have people coming up with
new novel implementations!

> Q3. Using the generic global timer implementation
> 
> So, how do I use that implementation?
> (Is someone other than STMicro using it?)
> 
> I see:
> 
> static void __init global_timer_of_register(struct device_node *np)
> CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer", global_timer_of_register);
> 
> OF stands for open firmware, yes?
> So is this related to device tree?

Yes.

> http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/arm/global_timer.txt
> 
> This file makes no sense to me.
> 
> - interrupts : One interrupt to each core
> interrupts = <1 13 0xf01>;
> what are 1 13 0xf01 ??

For this see Documentation/devicetree/bindings/arm/gic.txt, the
#interrupt-cells property defines the number of values between the <>,
and it goes on to define what each means.

The interrupts= property depends on your interrupt controller.

> - clocks : Should be phandle to a clock.
> clocks = <&arm_periph_clk>;
> 
> For my (old) 3.14 kernel, I found this:
> 
>     /*
>      * ARM Peripheral clock for timers
>      */
>     arm_periph_clk: arm_periph_clk {
>       #clock-cells = <0>;
>       compatible = "fixed-clock";
>       clock-frequency = <600000000>;
>     };
> 
> But it looks like the definitions have moved around since then?

No idea.  You do need to tell it where the global timer gets its clock
from so that it knows how fast it ticks, and whether there's anything
that needs to be enabled for that clock to be supplied.

> This device tree concept is too much to swallow in a single serving.
> Please tell me if I'm going down the correct rabbit hole, and I'll
> do some LWN readings to try to wrap my mind around the concept.

Yes, DT has made stuff more complicated; unfortunately, that's life now.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Delays, clocks, timers, hrtimers, etc

[Russell King - ARM Linux]

On Wed, Jan 28, 2015 at 02:16:21PM +0100, Mason wrote:
> Q1. the {n,u,m}delay function family
> 
> arch/arm/include/asm/delay.h mentions
> "Delay routines, using a pre-computed "loops_per_second" value."
> *BUT* if the frequency changes dynamically (thanks to cpufreq)
> the "loops_per_second" value cannot be pre-computed, as it would
> change dynamically too, right?

cpufreq changes the loops_per_second value, but an already in-progress
delay doesn't see that (new delays will see the update though.)

> Also, is the update of loops_per_jiffy atomic? Is it possible that
> if one core reads it while another updates it, we get garbage?

32-bit reads and writes are atomic.  You read either the old value or
the new value.  There's no inbetween.

> I suppose this is one reason why the default functions are overridden
> by register_current_timer_delay(&arch_delay_timer) right? I think the
> property of a timer is that its frequency doesn't change, even if the
> CPU's frequency changes? So we are still busy looping, but we are
> checking the actual time spent in the loop, whatever the cpufreq?

Timers are preferred because of the problems with the software delay loop.

Note that it has always been the case that the software delay loop is
"approximate" - even without cpufreq etc, the loops_per_jiffy is slightly
on the small side because of the way the calibration works.  It's about
98% of the actual value, and depends on the workload of the timer
interrupt.  It's obvious when you think about it - it's counting the
number of cycles between two timer interrupts, and the timer interrupt
consumes some of the cycles.

This means that even if you ask for a 10us delay, you'll probably get a
delay of 9.8us instead.

> Q2. Cortex A9 global and private timers
> 
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html
> 
> (What are private timers used for?)

The per-cpu private timers are mostly scheduling of threads.

> In my platform-specific code, there is a config option to choose between
> 
> 1) the ARM global timer
> 2) a platform-specific timer (timer0)

Most platforms implement their own timer, because its really sexy for
hardware engineers to create yet another different timer implementation
which is soo much better than every other timer implementation that has
already been created.  You wouldn't believe how many different ways that
there are to create a timer - and we still have people coming up with
new novel implementations!

> Q3. Using the generic global timer implementation
> 
> So, how do I use that implementation?
> (Is someone other than STMicro using it?)
> 
> I see:
> 
> static void __init global_timer_of_register(struct device_node *np)
> CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer", global_timer_of_register);
> 
> OF stands for open firmware, yes?
> So is this related to device tree?

Yes.

> http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/arm/global_timer.txt
> 
> This file makes no sense to me.
> 
> - interrupts : One interrupt to each core
> interrupts = <1 13 0xf01>;
> what are 1 13 0xf01 ??

For this see Documentation/devicetree/bindings/arm/gic.txt, the
#interrupt-cells property defines the number of values between the <>,
and it goes on to define what each means.

The interrupts= property depends on your interrupt controller.

> - clocks : Should be phandle to a clock.
> clocks = <&arm_periph_clk>;
> 
> For my (old) 3.14 kernel, I found this:
> 
>     /*
>      * ARM Peripheral clock for timers
>      */
>     arm_periph_clk: arm_periph_clk {
>       #clock-cells = <0>;
>       compatible = "fixed-clock";
>       clock-frequency = <600000000>;
>     };
> 
> But it looks like the definitions have moved around since then?

No idea.  You do need to tell it where the global timer gets its clock
from so that it knows how fast it ticks, and whether there's anything
that needs to be enabled for that clock to be supplied.

> This device tree concept is too much to swallow in a single serving.
> Please tell me if I'm going down the correct rabbit hole, and I'll
> do some LWN readings to try to wrap my mind around the concept.

Yes, DT has made stuff more complicated; unfortunately, that's life now.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Hello Russell,

First of all, thanks for the insight you've been providing. It helps a lot.
(And thanks to everyone who's been answering some of my questions.)

Russell King - ARM Linux wrote:

> On Wed, Jan 28, 2015 at 02:16:21PM +0100, Mason wrote:
>
>> [snip udelay discussion]
>>
>> Q2. Cortex A9 global and private timers
>>
>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html
>>
>> (What are private timers used for?)
>
> The per-cpu private timers are mostly scheduling of threads.

Hmmm, I'm confused (again).

CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK means "use the global timer
for the scheduler clock", right? In that case, are the local timers
unused by Linux?

#ifdef CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
   sched_clock_register(gt_sched_clock_read, 64, gt_clk_rate);
#endif

Is there generic code to set up local timers? (If so, where?)
sched_clock_register only seems to appear in machine-specific code.

What are the pros/cons of global timer vs local timers?
Or is such a question irrelevant?
(Because they are used for different purposes?)

>> In my platform-specific code, there is a config option to choose between
>>
>> 1) the ARM global timer
>> 2) a platform-specific timer (timer0)
>
> Most platforms implement their own timer, because its really sexy for
> hardware engineers to create yet another different timer implementation
> which is soo much better than every other timer implementation that has
> already been created.  You wouldn't believe how many different ways that
> there are to create a timer - and we still have people coming up with
> new novel implementations!

There is trouble in paradise. I was planning to give the global timer a try,
instead of the platform-specific timer, until I noticed: "The global timer
is clocked by PERIPHCLK." And it turns PERIPHCLK is connected to the SoC's
CPU_CLK (the clock that drives the CPU cores); the same clock that is managed
by cpufreq. I imagine it's double plus ungood for precise time-keeping to
have frequency changes of the clocksource input?

>> Q3. Using the generic global timer implementation
>>
>> So, how do I use that implementation?
>> (Is someone other than STMicro using it?)

To answer that question:

$ grep -rn ARM_GLOBAL_TIMER | grep -v SCHED_CLOCK
arch/arm/mach-vexpress/Kconfig:7:	select ARM_GLOBAL_TIMER
arch/arm/mach-bcm/Kconfig:87:	select ARM_GLOBAL_TIMER
arch/arm/mach-bcm/Kconfig:108:	select ARM_GLOBAL_TIMER
arch/arm/mach-rockchip/Kconfig:14:	select ARM_GLOBAL_TIMER
arch/arm/mach-sti/Kconfig:4:	select ARM_GLOBAL_TIMER
arch/arm/mach-zynq/Kconfig:6:	select ARM_GLOBAL_TIMER if !CPU_FREQ

There are indeed a few users; also I wonder why mach-zynq enables
ARM_GLOBAL_TIMER only if !CPU_FREQ. (Could it be that they have
the CPU CLOCK driving PERIPHCLK?)

Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Hello Russell,

First of all, thanks for the insight you've been providing. It helps a lot.
(And thanks to everyone who's been answering some of my questions.)

Russell King - ARM Linux wrote:

> On Wed, Jan 28, 2015 at 02:16:21PM +0100, Mason wrote:
>
>> [snip udelay discussion]
>>
>> Q2. Cortex A9 global and private timers
>>
>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html
>>
>> (What are private timers used for?)
>
> The per-cpu private timers are mostly scheduling of threads.

Hmmm, I'm confused (again).

CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK means "use the global timer
for the scheduler clock", right? In that case, are the local timers
unused by Linux?

#ifdef CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
   sched_clock_register(gt_sched_clock_read, 64, gt_clk_rate);
#endif

Is there generic code to set up local timers? (If so, where?)
sched_clock_register only seems to appear in machine-specific code.

What are the pros/cons of global timer vs local timers?
Or is such a question irrelevant?
(Because they are used for different purposes?)

>> In my platform-specific code, there is a config option to choose between
>>
>> 1) the ARM global timer
>> 2) a platform-specific timer (timer0)
>
> Most platforms implement their own timer, because its really sexy for
> hardware engineers to create yet another different timer implementation
> which is soo much better than every other timer implementation that has
> already been created.  You wouldn't believe how many different ways that
> there are to create a timer - and we still have people coming up with
> new novel implementations!

There is trouble in paradise. I was planning to give the global timer a try,
instead of the platform-specific timer, until I noticed: "The global timer
is clocked by PERIPHCLK." And it turns PERIPHCLK is connected to the SoC's
CPU_CLK (the clock that drives the CPU cores); the same clock that is managed
by cpufreq. I imagine it's double plus ungood for precise time-keeping to
have frequency changes of the clocksource input?

>> Q3. Using the generic global timer implementation
>>
>> So, how do I use that implementation?
>> (Is someone other than STMicro using it?)

To answer that question:

$ grep -rn ARM_GLOBAL_TIMER | grep -v SCHED_CLOCK
arch/arm/mach-vexpress/Kconfig:7:	select ARM_GLOBAL_TIMER
arch/arm/mach-bcm/Kconfig:87:	select ARM_GLOBAL_TIMER
arch/arm/mach-bcm/Kconfig:108:	select ARM_GLOBAL_TIMER
arch/arm/mach-rockchip/Kconfig:14:	select ARM_GLOBAL_TIMER
arch/arm/mach-sti/Kconfig:4:	select ARM_GLOBAL_TIMER
arch/arm/mach-zynq/Kconfig:6:	select ARM_GLOBAL_TIMER if !CPU_FREQ

There are indeed a few users; also I wonder why mach-zynq enables
ARM_GLOBAL_TIMER only if !CPU_FREQ. (Could it be that they have
the CPU CLOCK driving PERIPHCLK?)

Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Russell King - ARM Linux]

On Fri, Feb 06, 2015 at 10:37:32AM -0800, Mason wrote:
> Hmmm, I'm confused (again).
> 
> CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK means "use the global timer
> for the scheduler clock", right? In that case, are the local timers
> unused by Linux?
> 
> #ifdef CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
>   sched_clock_register(gt_sched_clock_read, 64, gt_clk_rate);
> #endif
> 
> Is there generic code to set up local timers? (If so, where?)
> sched_clock_register only seems to appear in machine-specific code.
> 
> What are the pros/cons of global timer vs local timers?
> Or is such a question irrelevant?
> (Because they are used for different purposes?)

Correct.

The sched_clock itself is about providing the scheduler with a stable,
monotonically increasing 64-bit nanosecond value which it can use to
account for the passing of time (so it can accurately measure how long
a thread is running for, and make a decision when to pre-empt it.)

Local timers are used to set "alarms" to cause an interrupt at a certain
point in time to do something (like, run the scheduler to switch from
the current thread to another thread.)  It can also be used to update
the current time of day as well.

Global timers are used as a fallback when local timers are not available,
and are less efficient - the CPU receiving the global timer interrupt
has to broadcast the interrupt to other CPUs, and it also has to take
account of the earliest event across all CPUs.

There is a fourth "timer" which is used as a monotonically incrementing
counter - this is called the "clocksource".  This is used to maintain
the kernel's time-of-day.  This may be the same as the sched_clock.

> There is trouble in paradise. I was planning to give the global timer a try,
> instead of the platform-specific timer, until I noticed: "The global timer
> is clocked by PERIPHCLK." And it turns PERIPHCLK is connected to the SoC's
> CPU_CLK (the clock that drives the CPU cores); the same clock that is managed
> by cpufreq. I imagine it's double plus ungood for precise time-keeping to
> have frequency changes of the clocksource input?

Yes and no.  You can use that for local timers and/or a global timer, but
you really don't want to use that for the sched_clock nor clocksource as
these are really the fundamentals of time keeping.  I'd advise against
using a timer derived from the CPU clock in general though, but if you
have no other possible timers, then it has to do.

> $ grep -rn ARM_GLOBAL_TIMER | grep -v SCHED_CLOCK
> arch/arm/mach-vexpress/Kconfig:7:	select ARM_GLOBAL_TIMER
> arch/arm/mach-bcm/Kconfig:87:	select ARM_GLOBAL_TIMER
> arch/arm/mach-bcm/Kconfig:108:	select ARM_GLOBAL_TIMER
> arch/arm/mach-rockchip/Kconfig:14:	select ARM_GLOBAL_TIMER
> arch/arm/mach-sti/Kconfig:4:	select ARM_GLOBAL_TIMER
> arch/arm/mach-zynq/Kconfig:6:	select ARM_GLOBAL_TIMER if !CPU_FREQ
> 
> There are indeed a few users; also I wonder why mach-zynq enables
> ARM_GLOBAL_TIMER only if !CPU_FREQ. (Could it be that they have
> the CPU CLOCK driving PERIPHCLK?)

I'm afraid I don't know.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Delays, clocks, timers, hrtimers, etc

[Russell King - ARM Linux]

On Fri, Feb 06, 2015 at 10:37:32AM -0800, Mason wrote:
> Hmmm, I'm confused (again).
> 
> CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK means "use the global timer
> for the scheduler clock", right? In that case, are the local timers
> unused by Linux?
> 
> #ifdef CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
>   sched_clock_register(gt_sched_clock_read, 64, gt_clk_rate);
> #endif
> 
> Is there generic code to set up local timers? (If so, where?)
> sched_clock_register only seems to appear in machine-specific code.
> 
> What are the pros/cons of global timer vs local timers?
> Or is such a question irrelevant?
> (Because they are used for different purposes?)

Correct.

The sched_clock itself is about providing the scheduler with a stable,
monotonically increasing 64-bit nanosecond value which it can use to
account for the passing of time (so it can accurately measure how long
a thread is running for, and make a decision when to pre-empt it.)

Local timers are used to set "alarms" to cause an interrupt at a certain
point in time to do something (like, run the scheduler to switch from
the current thread to another thread.)  It can also be used to update
the current time of day as well.

Global timers are used as a fallback when local timers are not available,
and are less efficient - the CPU receiving the global timer interrupt
has to broadcast the interrupt to other CPUs, and it also has to take
account of the earliest event across all CPUs.

There is a fourth "timer" which is used as a monotonically incrementing
counter - this is called the "clocksource".  This is used to maintain
the kernel's time-of-day.  This may be the same as the sched_clock.

> There is trouble in paradise. I was planning to give the global timer a try,
> instead of the platform-specific timer, until I noticed: "The global timer
> is clocked by PERIPHCLK." And it turns PERIPHCLK is connected to the SoC's
> CPU_CLK (the clock that drives the CPU cores); the same clock that is managed
> by cpufreq. I imagine it's double plus ungood for precise time-keeping to
> have frequency changes of the clocksource input?

Yes and no.  You can use that for local timers and/or a global timer, but
you really don't want to use that for the sched_clock nor clocksource as
these are really the fundamentals of time keeping.  I'd advise against
using a timer derived from the CPU clock in general though, but if you
have no other possible timers, then it has to do.

> $ grep -rn ARM_GLOBAL_TIMER | grep -v SCHED_CLOCK
> arch/arm/mach-vexpress/Kconfig:7:	select ARM_GLOBAL_TIMER
> arch/arm/mach-bcm/Kconfig:87:	select ARM_GLOBAL_TIMER
> arch/arm/mach-bcm/Kconfig:108:	select ARM_GLOBAL_TIMER
> arch/arm/mach-rockchip/Kconfig:14:	select ARM_GLOBAL_TIMER
> arch/arm/mach-sti/Kconfig:4:	select ARM_GLOBAL_TIMER
> arch/arm/mach-zynq/Kconfig:6:	select ARM_GLOBAL_TIMER if !CPU_FREQ
> 
> There are indeed a few users; also I wonder why mach-zynq enables
> ARM_GLOBAL_TIMER only if !CPU_FREQ. (Could it be that they have
> the CPU CLOCK driving PERIPHCLK?)

I'm afraid I don't know.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: Delays, clocks, timers, hrtimers, etc

[Stefan Agner]

On 2015-01-28 14:16, Mason wrote:
> Hello,
> 
> I am swimming in a sea of confusion, and am hoping someone would toss
> me a life-jacket (of enlightenment). Please forgive me if some of my
> questions are poorly asked or appear in seemingly random order.
> 
> Working on a Cortex A9 based SoC, I set out to "clean up" the platform
> specific timer code, by using as much generic framework as possible.
> (Right now, there's a lot of "redundant" code in the mach dir.)
> 
> 
> Q1. the {n,u,m}delay function family
> 
> arch/arm/include/asm/delay.h mentions
> "Delay routines, using a pre-computed "loops_per_second" value."
> *BUT* if the frequency changes dynamically (thanks to cpufreq)
> the "loops_per_second" value cannot be pre-computed, as it would
> change dynamically too, right?
> 
> Looking at arch/arm/lib/delay.c it seems the default implementation
> is a busy loop (in delay-loop.S) which looks up "loops_per_jiffy"
> in the prolog to determine the number of times to loop, right?
> 
> http://lxr.free-electrons.com/source/arch/arm/lib/delay-loop.S
> 
> (Side issue, why is the loop unrolled in __loop_delay? What is the
> point of unrolling a busy loop? This is commented code however.)
> 
> What happens if loops_per_jiffy changes while one core is in the
> busy loop? It seems we might exit the loop too early, which could
> break some drivers with some weird heisenbug, no?
> 
> Also, is the update of loops_per_jiffy atomic? Is it possible that
> if one core reads it while another updates it, we get garbage?
> 
> I suppose this is one reason why the default functions are overridden
> by register_current_timer_delay(&arch_delay_timer) right? I think the
> property of a timer is that its frequency doesn't change, even if the
> CPU's frequency changes? So we are still busy looping, but we are
> checking the actual time spent in the loop, whatever the cpufreq?
> 
> Reference
> https://www.kernel.org/doc/Documentation/timers/timers-howto.txt
> 
> 
> Q2. Cortex A9 global and private timers
> 
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html
> 
> (What are private timers used for?)
> 
> In my platform-specific code, there is a config option to choose between
> 
> 1) the ARM global timer
> 2) a platform-specific timer (timer0)
> 
> I noticed that there is generic code to support the global timer in
> drivers/clocksource/arm_global_timer.c
> 
> config ARM_GLOBAL_TIMER
> bool
> select CLKSRC_OF if OF
> help
>   This options enables support for the ARM global timer unit
> 
> config CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
> bool
> depends on ARM_GLOBAL_TIMER
> default y
> help
>  Use ARM global timer clock source as sched_clock
> 
> I was thinking it would be better to use the "standard" option (ARM
> global timer)
> as it is "officially" supported in the vanilla kernel. So less code to write and
> to debug, and it has likely received more testing. Why would one rely on
> platform-specific timers then?
> 
> Are high-resolution timers supported with the global timer?
> 
> 
> Q3. Using the generic global timer implementation
> 
> So, how do I use that implementation?
> (Is someone other than STMicro using it?)
> 
> I see:
> 
> static void __init global_timer_of_register(struct device_node *np)
> CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer",
> global_timer_of_register);
> 
> OF stands for open firmware, yes?
> So is this related to device tree?
> 
> http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/arm/global_timer.txt
> 
> This file makes no sense to me.
> 
> - interrupts : One interrupt to each core
> interrupts = <1 13 0xf01>;
> what are 1 13 0xf01 ??
> 
> - clocks : Should be phandle to a clock.
> clocks = <&arm_periph_clk>;
> 
> For my (old) 3.14 kernel, I found this:
> 
>     /*
>      * ARM Peripheral clock for timers
>      */
>     arm_periph_clk: arm_periph_clk {
>       #clock-cells = <0>;
>       compatible = "fixed-clock";
>       clock-frequency = <600000000>;
>     };
> 
> But it looks like the definitions have moved around since then?

Hi Mason,

Just recently I added support of ARM global timer as clocksource for
Vybrid SoC. This SoC doesn't use cpufreq, hence it is safe to use the
ARM global timer. The nice thing of device tree is, the patch to add
support for that did not change a single line of code:

http://thread.gmane.org/gmane.linux.kernel/1794460

--
Stefan

> 
> This device tree concept is too much to swallow in a single serving.
> Please tell me if I'm going down the correct rabbit hole, and I'll
> do some LWN readings to try to wrap my mind around the concept.
> 
> 
> Anyway, if anyone can help me out on some of these topics, I'd be
> eternally grateful.
> 
> Regards.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Delays, clocks, timers, hrtimers, etc

[Stefan Agner]

On 2015-01-28 14:16, Mason wrote:
> Hello,
> 
> I am swimming in a sea of confusion, and am hoping someone would toss
> me a life-jacket (of enlightenment). Please forgive me if some of my
> questions are poorly asked or appear in seemingly random order.
> 
> Working on a Cortex A9 based SoC, I set out to "clean up" the platform
> specific timer code, by using as much generic framework as possible.
> (Right now, there's a lot of "redundant" code in the mach dir.)
> 
> 
> Q1. the {n,u,m}delay function family
> 
> arch/arm/include/asm/delay.h mentions
> "Delay routines, using a pre-computed "loops_per_second" value."
> *BUT* if the frequency changes dynamically (thanks to cpufreq)
> the "loops_per_second" value cannot be pre-computed, as it would
> change dynamically too, right?
> 
> Looking at arch/arm/lib/delay.c it seems the default implementation
> is a busy loop (in delay-loop.S) which looks up "loops_per_jiffy"
> in the prolog to determine the number of times to loop, right?
> 
> http://lxr.free-electrons.com/source/arch/arm/lib/delay-loop.S
> 
> (Side issue, why is the loop unrolled in __loop_delay? What is the
> point of unrolling a busy loop? This is commented code however.)
> 
> What happens if loops_per_jiffy changes while one core is in the
> busy loop? It seems we might exit the loop too early, which could
> break some drivers with some weird heisenbug, no?
> 
> Also, is the update of loops_per_jiffy atomic? Is it possible that
> if one core reads it while another updates it, we get garbage?
> 
> I suppose this is one reason why the default functions are overridden
> by register_current_timer_delay(&arch_delay_timer) right? I think the
> property of a timer is that its frequency doesn't change, even if the
> CPU's frequency changes? So we are still busy looping, but we are
> checking the actual time spent in the loop, whatever the cpufreq?
> 
> Reference
> https://www.kernel.org/doc/Documentation/timers/timers-howto.txt
> 
> 
> Q2. Cortex A9 global and private timers
> 
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0407f/CIHGECHJ.html
> 
> (What are private timers used for?)
> 
> In my platform-specific code, there is a config option to choose between
> 
> 1) the ARM global timer
> 2) a platform-specific timer (timer0)
> 
> I noticed that there is generic code to support the global timer in
> drivers/clocksource/arm_global_timer.c
> 
> config ARM_GLOBAL_TIMER
> bool
> select CLKSRC_OF if OF
> help
>   This options enables support for the ARM global timer unit
> 
> config CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
> bool
> depends on ARM_GLOBAL_TIMER
> default y
> help
>  Use ARM global timer clock source as sched_clock
> 
> I was thinking it would be better to use the "standard" option (ARM
> global timer)
> as it is "officially" supported in the vanilla kernel. So less code to write and
> to debug, and it has likely received more testing. Why would one rely on
> platform-specific timers then?
> 
> Are high-resolution timers supported with the global timer?
> 
> 
> Q3. Using the generic global timer implementation
> 
> So, how do I use that implementation?
> (Is someone other than STMicro using it?)
> 
> I see:
> 
> static void __init global_timer_of_register(struct device_node *np)
> CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer",
> global_timer_of_register);
> 
> OF stands for open firmware, yes?
> So is this related to device tree?
> 
> http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/arm/global_timer.txt
> 
> This file makes no sense to me.
> 
> - interrupts : One interrupt to each core
> interrupts = <1 13 0xf01>;
> what are 1 13 0xf01 ??
> 
> - clocks : Should be phandle to a clock.
> clocks = <&arm_periph_clk>;
> 
> For my (old) 3.14 kernel, I found this:
> 
>     /*
>      * ARM Peripheral clock for timers
>      */
>     arm_periph_clk: arm_periph_clk {
>       #clock-cells = <0>;
>       compatible = "fixed-clock";
>       clock-frequency = <600000000>;
>     };
> 
> But it looks like the definitions have moved around since then?

Hi Mason,

Just recently I added support of ARM global timer as clocksource for
Vybrid SoC. This SoC doesn't use cpufreq, hence it is safe to use the
ARM global timer. The nice thing of device tree is, the patch to add
support for that did not change a single line of code:

http://thread.gmane.org/gmane.linux.kernel/1794460

--
Stefan

> 
> This device tree concept is too much to swallow in a single serving.
> Please tell me if I'm going down the correct rabbit hole, and I'll
> do some LWN readings to try to wrap my mind around the concept.
> 
> 
> Anyway, if anyone can help me out on some of these topics, I'd be
> eternally grateful.
> 
> Regards.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Russell King - ARM Linux wrote:

> Mason wrote:
>
>> Hmmm, I'm confused (again).
>>
>> CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK means "use the global timer
>> for the scheduler clock", right? In that case, are the local timers
>> unused by Linux?
>>
>> #ifdef CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
>>    sched_clock_register(gt_sched_clock_read, 64, gt_clk_rate);
>> #endif
>>
>> Is there generic code to set up local timers? (If so, where?)
>> sched_clock_register only seems to appear in machine-specific code.
>>
>> What are the pros/cons of global timer vs local timers?
>> Or is such a question irrelevant?
>> (Because they are used for different purposes?)
>
> Correct.
>
> The sched_clock itself is about providing the scheduler with a stable,
> monotonically increasing 64-bit nanosecond value which it can use to
> account for the passing of time (so it can accurately measure how long
> a thread is running for, and make a decision when to pre-empt it.)
>
> Local timers are used to set "alarms" to cause an interrupt at a certain
> point in time to do something (like, run the scheduler to switch from
> the current thread to another thread.)  It can also be used to update
> the current time of day as well.
>
> Global timers are used as a fallback when local timers are not available,
> and are less efficient - the CPU receiving the global timer interrupt
> has to broadcast the interrupt to other CPUs, and it also has to take
> account of the earliest event across all CPUs.
>
> There is a fourth "timer" which is used as a monotonically incrementing
> counter - this is called the "clocksource".  This is used to maintain
> the kernel's time-of-day.  This may be the same as the sched_clock.

Where is this sched_clock set up?

I see sched_clock_register() in kernel/time/sched_clock.c

setup_sched_clock() wraps sched_clock_register() but I only see one
user of setup_sched_clock() -- arch/arm/mach-footbridge

sched_clock_postinit() is used to set up an acceptable default,
I suppose? (i.e. jiffy_sched_clock_read)

But this still needs a time source (an actual crystal). I must be
missing something important from the big picture.

>> There is trouble in paradise. I was planning to give the global timer a try,
>> instead of the platform-specific timer, until I noticed: "The global timer
>> is clocked by PERIPHCLK." And it turns PERIPHCLK is connected to the SoC's
>> CPU_CLK (the clock that drives the CPU cores); the same clock that is managed
>> by cpufreq. I imagine it's double plus ungood for precise time-keeping to
>> have frequency changes of the clocksource input?
>
> Yes and no.  You can use that for local timers and/or a global timer, but
> you really don't want to use that for the sched_clock nor clocksource as
> these are really the fundamentals of time keeping.  I'd advise against
> using a timer derived from the CPU clock in general though, but if you
> have no other possible timers, then it has to do.

I do have several platform-specific timers available, but I was
considering using "standard" architected resources to minimize
the code needed for the port.

Also, reading Cortex-A9 MPCore Technical Reference Manual, section
5.1 Clocks, I see that, by definition, PERIPHCLK is tied to CLK
(the main clock). This means that, by definition, when using cpufreq,
CLK will change, thus PERIPHCLK will necessarily be variable too.

So the question I am trying to answer is: how do other SoCs use the
Cortex A9 global timer and local timers, along with cpufreq, and
make everything work correctly?

Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Russell King - ARM Linux wrote:

> Mason wrote:
>
>> Hmmm, I'm confused (again).
>>
>> CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK means "use the global timer
>> for the scheduler clock", right? In that case, are the local timers
>> unused by Linux?
>>
>> #ifdef CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
>>    sched_clock_register(gt_sched_clock_read, 64, gt_clk_rate);
>> #endif
>>
>> Is there generic code to set up local timers? (If so, where?)
>> sched_clock_register only seems to appear in machine-specific code.
>>
>> What are the pros/cons of global timer vs local timers?
>> Or is such a question irrelevant?
>> (Because they are used for different purposes?)
>
> Correct.
>
> The sched_clock itself is about providing the scheduler with a stable,
> monotonically increasing 64-bit nanosecond value which it can use to
> account for the passing of time (so it can accurately measure how long
> a thread is running for, and make a decision when to pre-empt it.)
>
> Local timers are used to set "alarms" to cause an interrupt at a certain
> point in time to do something (like, run the scheduler to switch from
> the current thread to another thread.)  It can also be used to update
> the current time of day as well.
>
> Global timers are used as a fallback when local timers are not available,
> and are less efficient - the CPU receiving the global timer interrupt
> has to broadcast the interrupt to other CPUs, and it also has to take
> account of the earliest event across all CPUs.
>
> There is a fourth "timer" which is used as a monotonically incrementing
> counter - this is called the "clocksource".  This is used to maintain
> the kernel's time-of-day.  This may be the same as the sched_clock.

Where is this sched_clock set up?

I see sched_clock_register() in kernel/time/sched_clock.c

setup_sched_clock() wraps sched_clock_register() but I only see one
user of setup_sched_clock() -- arch/arm/mach-footbridge

sched_clock_postinit() is used to set up an acceptable default,
I suppose? (i.e. jiffy_sched_clock_read)

But this still needs a time source (an actual crystal). I must be
missing something important from the big picture.

>> There is trouble in paradise. I was planning to give the global timer a try,
>> instead of the platform-specific timer, until I noticed: "The global timer
>> is clocked by PERIPHCLK." And it turns PERIPHCLK is connected to the SoC's
>> CPU_CLK (the clock that drives the CPU cores); the same clock that is managed
>> by cpufreq. I imagine it's double plus ungood for precise time-keeping to
>> have frequency changes of the clocksource input?
>
> Yes and no.  You can use that for local timers and/or a global timer, but
> you really don't want to use that for the sched_clock nor clocksource as
> these are really the fundamentals of time keeping.  I'd advise against
> using a timer derived from the CPU clock in general though, but if you
> have no other possible timers, then it has to do.

I do have several platform-specific timers available, but I was
considering using "standard" architected resources to minimize
the code needed for the port.

Also, reading Cortex-A9 MPCore Technical Reference Manual, section
5.1 Clocks, I see that, by definition, PERIPHCLK is tied to CLK
(the main clock). This means that, by definition, when using cpufreq,
CLK will change, thus PERIPHCLK will necessarily be variable too.

So the question I am trying to answer is: how do other SoCs use the
Cortex A9 global timer and local timers, along with cpufreq, and
make everything work correctly?

Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Stefan Agner wrote:

> Just recently I added support of ARM global timer as clocksource for
> Vybrid SoC. This SoC doesn't use cpufreq, hence it is safe to use the
> ARM global timer. The nice thing of device tree is, the patch to add
> support for that did not change a single line of code:
>
> http://thread.gmane.org/gmane.linux.kernel/1794460

Hello Stefan,

Your changes are not yet accepted in mainline, are they?
(I don't see them in 3.18.5)

Do you also use the ARM local timers in your port?
Is there generic code to handle them?

Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Stefan Agner wrote:

> Just recently I added support of ARM global timer as clocksource for
> Vybrid SoC. This SoC doesn't use cpufreq, hence it is safe to use the
> ARM global timer. The nice thing of device tree is, the patch to add
> support for that did not change a single line of code:
>
> http://thread.gmane.org/gmane.linux.kernel/1794460

Hello Stefan,

Your changes are not yet accepted in mainline, are they?
(I don't see them in 3.18.5)

Do you also use the ARM local timers in your port?
Is there generic code to handle them?

Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Stefan Agner]

On 2015-02-06 22:17, Mason wrote:
> Stefan Agner wrote:
> 
>> Just recently I added support of ARM global timer as clocksource for
>> Vybrid SoC. This SoC doesn't use cpufreq, hence it is safe to use the
>> ARM global timer. The nice thing of device tree is, the patch to add
>> support for that did not change a single line of code:
>>
>> http://thread.gmane.org/gmane.linux.kernel/1794460
> 
> Hello Stefan,
> 
> Your changes are not yet accepted in mainline, are they?
> (I don't see them in 3.18.5)

The changes have been accepted and went upstream in the 3.19 merge
window, see
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=81c4831907fb00efdc97093b09e333009a57d005

> Do you also use the ARM local timers in your port?
> Is there generic code to handle them?

It seems that there has been support for local timers once, but has been
removed. But I'm not aware of the details:
https://lkml.org/lkml/2013/2/22/49

--
Stefan

Delays, clocks, timers, hrtimers, etc

[Stefan Agner]

On 2015-02-06 22:17, Mason wrote:
> Stefan Agner wrote:
> 
>> Just recently I added support of ARM global timer as clocksource for
>> Vybrid SoC. This SoC doesn't use cpufreq, hence it is safe to use the
>> ARM global timer. The nice thing of device tree is, the patch to add
>> support for that did not change a single line of code:
>>
>> http://thread.gmane.org/gmane.linux.kernel/1794460
> 
> Hello Stefan,
> 
> Your changes are not yet accepted in mainline, are they?
> (I don't see them in 3.18.5)

The changes have been accepted and went upstream in the 3.19 merge
window, see
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=81c4831907fb00efdc97093b09e333009a57d005

> Do you also use the ARM local timers in your port?
> Is there generic code to handle them?

It seems that there has been support for local timers once, but has been
removed. But I'm not aware of the details:
https://lkml.org/lkml/2013/2/22/49

--
Stefan

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Stefan Agner wrote:

> On 2015-02-06 22:17, Mason wrote:
>
>> Do you also use the ARM local timers in your port?
>> Is there generic code to handle them?
>
> It seems that there has been support for local timers once, but has been
> removed. But I'm not aware of the details:
> https://lkml.org/lkml/2013/2/22/49

The equivalent gmane link would be:
http://thread.gmane.org/gmane.linux.kernel/1445799

The description used to say:

-config LOCAL_TIMERS
-	bool "Use local timer interrupts"
-	depends on SMP
-	default y
-	select HAVE_ARM_TWD if (!ARCH_MSM_SCORPIONMP && !EXYNOS4_MCT)
-	help
-	  Enable support for local timers on SMP platforms, rather then the
-	  legacy IPI broadcast method.  Local timers allows the system
-	  accounting to be spread across the timer interval, preventing a
-	  "thundering herd" at every timer tick.

which seems to have been replaced with HAVE_ARM_TWD

TWD stands for "Timer Watch Dog".
https://www.kernel.org/doc/Documentation/devicetree/bindings/arm/twd.txt

"ARM 11MP, Cortex-A5 and Cortex-A9 are often associated with a per-core
Timer-Watchdog (aka TWD), which provides both a per-cpu local timer
and watchdog.

The TWD is usually attached to a GIC to deliver its two per-processor
interrupts."


config HAVE_ARM_TWD
   bool
   depends on SMP
   select CLKSRC_OF if OF
   help
     This options enables support for the ARM timer and watchdog unit

One problem I see is that HAVE_ARM_TWD depends on SMP...

One of the systems I want to support is UP (single-core Cortex A9).
Does that mean I should use an SMP kernel even for that system?
Or is there a different subsystem for UP systems?

Also, reading arch/arm/kernel/smp_twd.c, I see that frequency changes
due to cpufreq are properly accounted for. (Although I imagine they do
introduce a small error in time-keeping because moving from freq_A to
freq_B is not instantaneous, so the exact time elapsed in-between is
impossible to determine.)

Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Stefan Agner wrote:

> On 2015-02-06 22:17, Mason wrote:
>
>> Do you also use the ARM local timers in your port?
>> Is there generic code to handle them?
>
> It seems that there has been support for local timers once, but has been
> removed. But I'm not aware of the details:
> https://lkml.org/lkml/2013/2/22/49

The equivalent gmane link would be:
http://thread.gmane.org/gmane.linux.kernel/1445799

The description used to say:

-config LOCAL_TIMERS
-	bool "Use local timer interrupts"
-	depends on SMP
-	default y
-	select HAVE_ARM_TWD if (!ARCH_MSM_SCORPIONMP && !EXYNOS4_MCT)
-	help
-	  Enable support for local timers on SMP platforms, rather then the
-	  legacy IPI broadcast method.  Local timers allows the system
-	  accounting to be spread across the timer interval, preventing a
-	  "thundering herd" at every timer tick.

which seems to have been replaced with HAVE_ARM_TWD

TWD stands for "Timer Watch Dog".
https://www.kernel.org/doc/Documentation/devicetree/bindings/arm/twd.txt

"ARM 11MP, Cortex-A5 and Cortex-A9 are often associated with a per-core
Timer-Watchdog (aka TWD), which provides both a per-cpu local timer
and watchdog.

The TWD is usually attached to a GIC to deliver its two per-processor
interrupts."


config HAVE_ARM_TWD
   bool
   depends on SMP
   select CLKSRC_OF if OF
   help
     This options enables support for the ARM timer and watchdog unit

One problem I see is that HAVE_ARM_TWD depends on SMP...

One of the systems I want to support is UP (single-core Cortex A9).
Does that mean I should use an SMP kernel even for that system?
Or is there a different subsystem for UP systems?

Also, reading arch/arm/kernel/smp_twd.c, I see that frequency changes
due to cpufreq are properly accounted for. (Although I imagine they do
introduce a small error in time-keeping because moving from freq_A to
freq_B is not instantaneous, so the exact time elapsed in-between is
impossible to determine.)

Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Russell King - ARM Linux]

On Fri, Feb 06, 2015 at 06:21:37PM -0800, Mason wrote:
> Also, reading arch/arm/kernel/smp_twd.c, I see that frequency changes
> due to cpufreq are properly accounted for. (Although I imagine they do
> introduce a small error in time-keeping because moving from freq_A to
> freq_B is not instantaneous, so the exact time elapsed in-between is
> impossible to determine.)

TWD is not used for time keeping as such.  As I explained in one of my
previous emails, local timers are used to signal events.

The active clocksource is used to determine the passing of time-of-day,
and sched_clock is used to account for process time by the scheduler.

An early/late/missed local timer interrupt has no impact on time keeping.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Delays, clocks, timers, hrtimers, etc

[Russell King - ARM Linux]

On Fri, Feb 06, 2015 at 06:21:37PM -0800, Mason wrote:
> Also, reading arch/arm/kernel/smp_twd.c, I see that frequency changes
> due to cpufreq are properly accounted for. (Although I imagine they do
> introduce a small error in time-keeping because moving from freq_A to
> freq_B is not instantaneous, so the exact time elapsed in-between is
> impossible to determine.)

TWD is not used for time keeping as such.  As I explained in one of my
previous emails, local timers are used to signal events.

The active clocksource is used to determine the passing of time-of-day,
and sched_clock is used to account for process time by the scheduler.

An early/late/missed local timer interrupt has no impact on time keeping.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: Delays, clocks, timers, hrtimers, etc

[Russell King - ARM Linux]

On Fri, Feb 06, 2015 at 01:03:19PM -0800, Mason wrote:
> Russell King - ARM Linux wrote:
> 
> >Mason wrote:
> >
> >>Hmmm, I'm confused (again).
> >>
> >>CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK means "use the global timer
> >>for the scheduler clock", right? In that case, are the local timers
> >>unused by Linux?
> >>
> >>#ifdef CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
> >>   sched_clock_register(gt_sched_clock_read, 64, gt_clk_rate);
> >>#endif
> >>
> >>Is there generic code to set up local timers? (If so, where?)
> >>sched_clock_register only seems to appear in machine-specific code.
> >>
> >>What are the pros/cons of global timer vs local timers?
> >>Or is such a question irrelevant?
> >>(Because they are used for different purposes?)
> >
> >Correct.
> >
> >The sched_clock itself is about providing the scheduler with a stable,
> >monotonically increasing 64-bit nanosecond value which it can use to
> >account for the passing of time (so it can accurately measure how long
> >a thread is running for, and make a decision when to pre-empt it.)
> >
> >Local timers are used to set "alarms" to cause an interrupt at a certain
> >point in time to do something (like, run the scheduler to switch from
> >the current thread to another thread.)  It can also be used to update
> >the current time of day as well.
> >
> >Global timers are used as a fallback when local timers are not available,
> >and are less efficient - the CPU receiving the global timer interrupt
> >has to broadcast the interrupt to other CPUs, and it also has to take
> >account of the earliest event across all CPUs.
> >
> >There is a fourth "timer" which is used as a monotonically incrementing
> >counter - this is called the "clocksource".  This is used to maintain
> >the kernel's time-of-day.  This may be the same as the sched_clock.
> 
> Where is this sched_clock set up?

What do you mean "set up" ?

> I see sched_clock_register() in kernel/time/sched_clock.c

Correct, and most users use sched_clock_register() rather than
setup_sched_clock().

> setup_sched_clock() wraps sched_clock_register() but I only see one
> user of setup_sched_clock() -- arch/arm/mach-footbridge

Probably hasn't been converted to use sched_clock_register() yet.

> sched_clock_postinit() is used to set up an acceptable default,
> I suppose? (i.e. jiffy_sched_clock_read)

That's to finish off the sched clock initialisation at a point where we
can do so (which should be after sched_clock_register() has been called.)

> But this still needs a time source (an actual crystal). I must be
> missing something important from the big picture.

All that sched_clock cares about is reading a value from a counter which
increments (or decrements) at a single, uniform, known rate, and the
code internal to sched_clock() converts that to a 64-bit nanosecond
value.

> 
> >>There is trouble in paradise. I was planning to give the global timer a try,
> >>instead of the platform-specific timer, until I noticed: "The global timer
> >>is clocked by PERIPHCLK." And it turns PERIPHCLK is connected to the SoC's
> >>CPU_CLK (the clock that drives the CPU cores); the same clock that is managed
> >>by cpufreq. I imagine it's double plus ungood for precise time-keeping to
> >>have frequency changes of the clocksource input?
> >
> >Yes and no.  You can use that for local timers and/or a global timer, but
> >you really don't want to use that for the sched_clock nor clocksource as
> >these are really the fundamentals of time keeping.  I'd advise against
> >using a timer derived from the CPU clock in general though, but if you
> >have no other possible timers, then it has to do.
> 
> I do have several platform-specific timers available, but I was
> considering using "standard" architected resources to minimize
> the code needed for the port.
> 
> Also, reading Cortex-A9 MPCore Technical Reference Manual, section
> 5.1 Clocks, I see that, by definition, PERIPHCLK is tied to CLK
> (the main clock). This means that, by definition, when using cpufreq,
> CLK will change, thus PERIPHCLK will necessarily be variable too.
> 
> So the question I am trying to answer is: how do other SoCs use the
> Cortex A9 global timer and local timers, along with cpufreq, and
> make everything work correctly?

You can use them for local and global timers, provided you have a stable
clocksource and sched_clock (which can both be derived from the same
counter.)

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Delays, clocks, timers, hrtimers, etc

[Russell King - ARM Linux]

On Fri, Feb 06, 2015 at 01:03:19PM -0800, Mason wrote:
> Russell King - ARM Linux wrote:
> 
> >Mason wrote:
> >
> >>Hmmm, I'm confused (again).
> >>
> >>CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK means "use the global timer
> >>for the scheduler clock", right? In that case, are the local timers
> >>unused by Linux?
> >>
> >>#ifdef CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
> >>   sched_clock_register(gt_sched_clock_read, 64, gt_clk_rate);
> >>#endif
> >>
> >>Is there generic code to set up local timers? (If so, where?)
> >>sched_clock_register only seems to appear in machine-specific code.
> >>
> >>What are the pros/cons of global timer vs local timers?
> >>Or is such a question irrelevant?
> >>(Because they are used for different purposes?)
> >
> >Correct.
> >
> >The sched_clock itself is about providing the scheduler with a stable,
> >monotonically increasing 64-bit nanosecond value which it can use to
> >account for the passing of time (so it can accurately measure how long
> >a thread is running for, and make a decision when to pre-empt it.)
> >
> >Local timers are used to set "alarms" to cause an interrupt at a certain
> >point in time to do something (like, run the scheduler to switch from
> >the current thread to another thread.)  It can also be used to update
> >the current time of day as well.
> >
> >Global timers are used as a fallback when local timers are not available,
> >and are less efficient - the CPU receiving the global timer interrupt
> >has to broadcast the interrupt to other CPUs, and it also has to take
> >account of the earliest event across all CPUs.
> >
> >There is a fourth "timer" which is used as a monotonically incrementing
> >counter - this is called the "clocksource".  This is used to maintain
> >the kernel's time-of-day.  This may be the same as the sched_clock.
> 
> Where is this sched_clock set up?

What do you mean "set up" ?

> I see sched_clock_register() in kernel/time/sched_clock.c

Correct, and most users use sched_clock_register() rather than
setup_sched_clock().

> setup_sched_clock() wraps sched_clock_register() but I only see one
> user of setup_sched_clock() -- arch/arm/mach-footbridge

Probably hasn't been converted to use sched_clock_register() yet.

> sched_clock_postinit() is used to set up an acceptable default,
> I suppose? (i.e. jiffy_sched_clock_read)

That's to finish off the sched clock initialisation at a point where we
can do so (which should be after sched_clock_register() has been called.)

> But this still needs a time source (an actual crystal). I must be
> missing something important from the big picture.

All that sched_clock cares about is reading a value from a counter which
increments (or decrements) at a single, uniform, known rate, and the
code internal to sched_clock() converts that to a 64-bit nanosecond
value.

> 
> >>There is trouble in paradise. I was planning to give the global timer a try,
> >>instead of the platform-specific timer, until I noticed: "The global timer
> >>is clocked by PERIPHCLK." And it turns PERIPHCLK is connected to the SoC's
> >>CPU_CLK (the clock that drives the CPU cores); the same clock that is managed
> >>by cpufreq. I imagine it's double plus ungood for precise time-keeping to
> >>have frequency changes of the clocksource input?
> >
> >Yes and no.  You can use that for local timers and/or a global timer, but
> >you really don't want to use that for the sched_clock nor clocksource as
> >these are really the fundamentals of time keeping.  I'd advise against
> >using a timer derived from the CPU clock in general though, but if you
> >have no other possible timers, then it has to do.
> 
> I do have several platform-specific timers available, but I was
> considering using "standard" architected resources to minimize
> the code needed for the port.
> 
> Also, reading Cortex-A9 MPCore Technical Reference Manual, section
> 5.1 Clocks, I see that, by definition, PERIPHCLK is tied to CLK
> (the main clock). This means that, by definition, when using cpufreq,
> CLK will change, thus PERIPHCLK will necessarily be variable too.
> 
> So the question I am trying to answer is: how do other SoCs use the
> Cortex A9 global timer and local timers, along with cpufreq, and
> make everything work correctly?

You can use them for local and global timers, provided you have a stable
clocksource and sched_clock (which can both be derived from the same
counter.)

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: Delays, clocks, timers, hrtimers, etc

[Michal Simek]

+ Soren

>> $ grep -rn ARM_GLOBAL_TIMER | grep -v SCHED_CLOCK
>> arch/arm/mach-vexpress/Kconfig:7:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-bcm/Kconfig:87:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-bcm/Kconfig:108:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-rockchip/Kconfig:14:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-sti/Kconfig:4:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-zynq/Kconfig:6:	select ARM_GLOBAL_TIMER if !CPU_FREQ
>>
>> There are indeed a few users; also I wonder why mach-zynq enables
>> ARM_GLOBAL_TIMER only if !CPU_FREQ. (Could it be that they have
>> the CPU CLOCK driving PERIPHCLK?)
> 
> I'm afraid I don't know.

The reason is that cpu clock input and global timer clock input are not independent
that's why changing clock for cpu also affect global timer freq which is wrong.
It is caused by incorrect hw design.
Soren can give you more detail information if you need.

Thanks,
Michal

Delays, clocks, timers, hrtimers, etc

[Michal Simek]

+ Soren

>> $ grep -rn ARM_GLOBAL_TIMER | grep -v SCHED_CLOCK
>> arch/arm/mach-vexpress/Kconfig:7:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-bcm/Kconfig:87:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-bcm/Kconfig:108:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-rockchip/Kconfig:14:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-sti/Kconfig:4:	select ARM_GLOBAL_TIMER
>> arch/arm/mach-zynq/Kconfig:6:	select ARM_GLOBAL_TIMER if !CPU_FREQ
>>
>> There are indeed a few users; also I wonder why mach-zynq enables
>> ARM_GLOBAL_TIMER only if !CPU_FREQ. (Could it be that they have
>> the CPU CLOCK driving PERIPHCLK?)
> 
> I'm afraid I don't know.

The reason is that cpu clock input and global timer clock input are not independent
that's why changing clock for cpu also affect global timer freq which is wrong.
It is caused by incorrect hw design.
Soren can give you more detail information if you need.

Thanks,
Michal

Re: Delays, clocks, timers, hrtimers, etc

[Sören Brinkmann]

On Mon, 2015-02-09 at 08:45AM +0100, Michal Simek wrote:
> + Soren
> 
> >> $ grep -rn ARM_GLOBAL_TIMER | grep -v SCHED_CLOCK
> >> arch/arm/mach-vexpress/Kconfig:7:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-bcm/Kconfig:87:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-bcm/Kconfig:108:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-rockchip/Kconfig:14:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-sti/Kconfig:4:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-zynq/Kconfig:6:	select ARM_GLOBAL_TIMER if !CPU_FREQ
> >>
> >> There are indeed a few users; also I wonder why mach-zynq enables
> >> ARM_GLOBAL_TIMER only if !CPU_FREQ. (Could it be that they have
> >> the CPU CLOCK driving PERIPHCLK?)
> > 
> > I'm afraid I don't know.
> 
> The reason is that cpu clock input and global timer clock input are not independent
> that's why changing clock for cpu also affect global timer freq which is wrong.
> It is caused by incorrect hw design.
> Soren can give you more detail information if you need.

Right, it is a (broken) attempt to avoid usage of the global timer in
case cpufreq is enabled. On Zynq, all timer's clock inputs are derived
from the CPU clock. Hence they all do change when cpufreq scales the CPU
clock. I wrote a fragile construct of clk notifiers for our cadence TTC
which compensates for those changes. But the TTC is fairly narrow and
slow, hence to guarantee it becomes the selected clocksource, I tried to
blacklist the global timer this way (which obviously fails when
something else selects the timer).

	Thanks,
	Soren

Delays, clocks, timers, hrtimers, etc

[Sören Brinkmann]

On Mon, 2015-02-09 at 08:45AM +0100, Michal Simek wrote:
> + Soren
> 
> >> $ grep -rn ARM_GLOBAL_TIMER | grep -v SCHED_CLOCK
> >> arch/arm/mach-vexpress/Kconfig:7:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-bcm/Kconfig:87:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-bcm/Kconfig:108:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-rockchip/Kconfig:14:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-sti/Kconfig:4:	select ARM_GLOBAL_TIMER
> >> arch/arm/mach-zynq/Kconfig:6:	select ARM_GLOBAL_TIMER if !CPU_FREQ
> >>
> >> There are indeed a few users; also I wonder why mach-zynq enables
> >> ARM_GLOBAL_TIMER only if !CPU_FREQ. (Could it be that they have
> >> the CPU CLOCK driving PERIPHCLK?)
> > 
> > I'm afraid I don't know.
> 
> The reason is that cpu clock input and global timer clock input are not independent
> that's why changing clock for cpu also affect global timer freq which is wrong.
> It is caused by incorrect hw design.
> Soren can give you more detail information if you need.

Right, it is a (broken) attempt to avoid usage of the global timer in
case cpufreq is enabled. On Zynq, all timer's clock inputs are derived
from the CPU clock. Hence they all do change when cpufreq scales the CPU
clock. I wrote a fragile construct of clk notifiers for our cadence TTC
which compensates for those changes. But the TTC is fairly narrow and
slow, hence to guarantee it becomes the selected clocksource, I tried to
blacklist the global timer this way (which obviously fails when
something else selects the timer).

	Thanks,
	Soren

Re: Delays, clocks, timers, hrtimers, etc

[Stephen Boyd]

On 02/06/15 18:21, Mason wrote:
>
>
> config HAVE_ARM_TWD
>   bool
>   depends on SMP
>   select CLKSRC_OF if OF
>   help
>     This options enables support for the ARM timer and watchdog unit
>
> One problem I see is that HAVE_ARM_TWD depends on SMP...
>
> One of the systems I want to support is UP (single-core Cortex A9).
> Does that mean I should use an SMP kernel even for that system?
> Or is there a different subsystem for UP systems?

I don't see any problem with the TWD dropping the dependency on SMP. The
code should work the same on a UP configuration and if that's the only
timer you have that can deliver interrupts to your processor then it
would be required. You'd still need a clocksource though, which the TWD
doesn't provide.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Delays, clocks, timers, hrtimers, etc

[Stephen Boyd]

On 02/06/15 18:21, Mason wrote:
>
>
> config HAVE_ARM_TWD
>   bool
>   depends on SMP
>   select CLKSRC_OF if OF
>   help
>     This options enables support for the ARM timer and watchdog unit
>
> One problem I see is that HAVE_ARM_TWD depends on SMP...
>
> One of the systems I want to support is UP (single-core Cortex A9).
> Does that mean I should use an SMP kernel even for that system?
> Or is there a different subsystem for UP systems?

I don't see any problem with the TWD dropping the dependency on SMP. The
code should work the same on a UP configuration and if that's the only
timer you have that can deliver interrupts to your processor then it
would be required. You'd still need a clocksource though, which the TWD
doesn't provide.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Stephen Boyd wrote:

> Mason wrote:
>
>> config HAVE_ARM_TWD
>>    bool
>>    depends on SMP
>>    select CLKSRC_OF if OF
>>    help
>>      This options enables support for the ARM timer and watchdog unit
>>
>> One problem I see is that HAVE_ARM_TWD depends on SMP...
>>
>> One of the systems I want to support is UP (single-core Cortex A9).
>> Does that mean I should use an SMP kernel even for that system?
>> Or is there a different subsystem for UP systems?
>
> I don't see any problem with the TWD dropping the dependency on SMP. The
> code should work the same on a UP configuration and if that's the only
> timer you have that can deliver interrupts to your processor then it
> would be required. You'd still need a clocksource though, which the TWD
> doesn't provide.

I want to use as much mainlined code as possible. Thus, this means
(if my reasoning is not flawed) that I should use standard ARM
"services" (can't think of a better term) for which code is already
available in the kernel.

As for clock events, IIUC, I can choose either TWD or the global
timer (with a slight advantage to TWD). Are there other options?

(I probably have some platform-specific infrastructure available,
but I'm asking about standard support.)

My platform provides a 32-bit counter, ticking at a constant 27 MHz.
Reading this counter has a latency of roughly 70 ns (it has to go
over the system memory bus). I think this is good enough for both
the clock source and sched_clock, is it not?

So the plan would be:
- clocksource and sched_clock : 27 MHz, 32-bit counter, platform
- clockevents : TWD, standard

Regards.


For my own reference
http://elinux.org/Kernel_Timer_Systems
http://thread.gmane.org/gmane.linux.kernel/1062438 (not merged)
Message-ID: <20150206191419.GA8656@n2100.arm.linux.org.uk>

Delays, clocks, timers, hrtimers, etc

[Mason]

Stephen Boyd wrote:

> Mason wrote:
>
>> config HAVE_ARM_TWD
>>    bool
>>    depends on SMP
>>    select CLKSRC_OF if OF
>>    help
>>      This options enables support for the ARM timer and watchdog unit
>>
>> One problem I see is that HAVE_ARM_TWD depends on SMP...
>>
>> One of the systems I want to support is UP (single-core Cortex A9).
>> Does that mean I should use an SMP kernel even for that system?
>> Or is there a different subsystem for UP systems?
>
> I don't see any problem with the TWD dropping the dependency on SMP. The
> code should work the same on a UP configuration and if that's the only
> timer you have that can deliver interrupts to your processor then it
> would be required. You'd still need a clocksource though, which the TWD
> doesn't provide.

I want to use as much mainlined code as possible. Thus, this means
(if my reasoning is not flawed) that I should use standard ARM
"services" (can't think of a better term) for which code is already
available in the kernel.

As for clock events, IIUC, I can choose either TWD or the global
timer (with a slight advantage to TWD). Are there other options?

(I probably have some platform-specific infrastructure available,
but I'm asking about standard support.)

My platform provides a 32-bit counter, ticking at a constant 27 MHz.
Reading this counter has a latency of roughly 70 ns (it has to go
over the system memory bus). I think this is good enough for both
the clock source and sched_clock, is it not?

So the plan would be:
- clocksource and sched_clock : 27 MHz, 32-bit counter, platform
- clockevents : TWD, standard

Regards.


For my own reference
http://elinux.org/Kernel_Timer_Systems
http://thread.gmane.org/gmane.linux.kernel/1062438 (not merged)
Message-ID: <20150206191419.GA8656@n2100.arm.linux.org.uk>

Re: Delays, clocks, timers, hrtimers, etc

[Stephen Boyd]

On 02/09/15 14:31, Mason wrote:
>
> My platform provides a 32-bit counter, ticking at a constant 27 MHz.
> Reading this counter has a latency of roughly 70 ns (it has to go
> over the system memory bus). I think this is good enough for both
> the clock source and sched_clock, is it not?
>
> So the plan would be:
> - clocksource and sched_clock : 27 MHz, 32-bit counter, platform
> - clockevents : TWD, standard

Yep, that sounds like a good plan. If your platform has the ARM global
timer (drivers/clocksource/arm_global_timer.c) then you don't need
anything besides that timer because it provides both the clocksource,
sched_clock, and clockevents. Sounds like you don't have that timer
though, so you have to write a driver for your custom platform timer and
at least hook up clocksource and sched_clock to it. If you have
interrupts with your platform timer you can skip out on TWD and also
register a clockevent in your platform timer driver.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Delays, clocks, timers, hrtimers, etc

[Stephen Boyd]

On 02/09/15 14:31, Mason wrote:
>
> My platform provides a 32-bit counter, ticking at a constant 27 MHz.
> Reading this counter has a latency of roughly 70 ns (it has to go
> over the system memory bus). I think this is good enough for both
> the clock source and sched_clock, is it not?
>
> So the plan would be:
> - clocksource and sched_clock : 27 MHz, 32-bit counter, platform
> - clockevents : TWD, standard

Yep, that sounds like a good plan. If your platform has the ARM global
timer (drivers/clocksource/arm_global_timer.c) then you don't need
anything besides that timer because it provides both the clocksource,
sched_clock, and clockevents. Sounds like you don't have that timer
though, so you have to write a driver for your custom platform timer and
at least hook up clocksource and sched_clock to it. If you have
interrupts with your platform timer you can skip out on TWD and also
register a clockevent in your platform timer driver.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Russell King - ARM Linux wrote:

> Mason wrote:
>  
>> Q1. the {n,u,m}delay function family
>>
>> [...]
>
> Timers are preferred because of the problems with the software delay loop.

Can you confirm that if one intends to use timer-based delays, one must
call register_current_timer_delay at init?

There aren't many callers of register_current_timer_delay.

   arch/arm/kernel/arch_timer.c:   register_current_timer_delay(&arch_delay_timer);
   arch/arm/mach-u300/timer.c:     register_current_timer_delay(&u300_delay_timer);
   drivers/clocksource/nomadik-mtu.c:      register_current_timer_delay(&mtu_delay_timer);

Does that mean that every other platform is using software delay-loops
instead of timer-based delay-loops?

(AFAIU, architected / generic timers are new-ish in Cortex A7 and A15.)

Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Russell King - ARM Linux wrote:

> Mason wrote:
>  
>> Q1. the {n,u,m}delay function family
>>
>> [...]
>
> Timers are preferred because of the problems with the software delay loop.

Can you confirm that if one intends to use timer-based delays, one must
call register_current_timer_delay at init?

There aren't many callers of register_current_timer_delay.

   arch/arm/kernel/arch_timer.c:   register_current_timer_delay(&arch_delay_timer);
   arch/arm/mach-u300/timer.c:     register_current_timer_delay(&u300_delay_timer);
   drivers/clocksource/nomadik-mtu.c:      register_current_timer_delay(&mtu_delay_timer);

Does that mean that every other platform is using software delay-loops
instead of timer-based delay-loops?

(AFAIU, architected / generic timers are new-ish in Cortex A7 and A15.)

Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Stephen Boyd wrote:

> Mason wrote:
>
>> My platform provides a 32-bit counter, ticking at a constant 27 MHz.
>> Reading this counter has a latency of roughly 70 ns (it has to go
>> over the system memory bus). I think this is good enough for both
>> the clock source and sched_clock, is it not?
>>
>> So the plan would be:
>> - clocksource and sched_clock : 27 MHz, 32-bit counter, platform
>> - clockevents : TWD, standard
>
> Yep, that sounds like a good plan. If your platform has the ARM global
> timer (drivers/clocksource/arm_global_timer.c) then you don't need
> anything besides that timer because it provides both the clocksource,
> sched_clock, and clockevents. Sounds like you don't have that timer

I'm using Cortex A9, so I do have the global timer (AFAIU).

However, I don't think I can safely use it as a clock source because
I'm also using cpufreq, and I don't think the arm_global_timer driver
handles CPU frequency updates, while the TWD driver does (?)

/*
  * Updates clockevent frequency when the cpu frequency changes.
  * Called on the cpu that is changing frequency with interrupts disabled.
  */

"The Interrupt Controller, global timer, private timers, and watchdogs
are clocked with PERIPHCLK." And PERIPHCLK is tied to CLK, which is
modified by cpufreq. I didn't see any code in the arm_global_timer
driver to deal with with that.

Did I overlook something fundamental?

> though, so you have to write a driver for your custom platform timer and
> at least hook up clocksource and sched_clock to it. If you have
> interrupts with your platform timer you can skip out on TWD and also
> register a clockevent in your platform timer driver.

Registering a clock source or a sched_clock seems straight-forward.
All I need to provide is a function to read the platform counter.

However, why would I skip out on TWD?
(I'm trying to minimize code needed for the port.)

Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Stephen Boyd wrote:

> Mason wrote:
>
>> My platform provides a 32-bit counter, ticking at a constant 27 MHz.
>> Reading this counter has a latency of roughly 70 ns (it has to go
>> over the system memory bus). I think this is good enough for both
>> the clock source and sched_clock, is it not?
>>
>> So the plan would be:
>> - clocksource and sched_clock : 27 MHz, 32-bit counter, platform
>> - clockevents : TWD, standard
>
> Yep, that sounds like a good plan. If your platform has the ARM global
> timer (drivers/clocksource/arm_global_timer.c) then you don't need
> anything besides that timer because it provides both the clocksource,
> sched_clock, and clockevents. Sounds like you don't have that timer

I'm using Cortex A9, so I do have the global timer (AFAIU).

However, I don't think I can safely use it as a clock source because
I'm also using cpufreq, and I don't think the arm_global_timer driver
handles CPU frequency updates, while the TWD driver does (?)

/*
  * Updates clockevent frequency when the cpu frequency changes.
  * Called on the cpu that is changing frequency with interrupts disabled.
  */

"The Interrupt Controller, global timer, private timers, and watchdogs
are clocked with PERIPHCLK." And PERIPHCLK is tied to CLK, which is
modified by cpufreq. I didn't see any code in the arm_global_timer
driver to deal with with that.

Did I overlook something fundamental?

> though, so you have to write a driver for your custom platform timer and
> at least hook up clocksource and sched_clock to it. If you have
> interrupts with your platform timer you can skip out on TWD and also
> register a clockevent in your platform timer driver.

Registering a clock source or a sched_clock seems straight-forward.
All I need to provide is a function to read the platform counter.

However, why would I skip out on TWD?
(I'm trying to minimize code needed for the port.)

Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Mason wrote:

> Stephen Boyd wrote:
>
>> Mason wrote:
>>
>>> My platform provides a 32-bit counter, ticking at a constant 27 MHz.
>>> Reading this counter has a latency of roughly 70 ns (it has to go
>>> over the system memory bus). I think this is good enough for both
>>> the clock source and sched_clock, is it not?
>>>
>>> So the plan would be:
>>> - clocksource and sched_clock : 27 MHz, 32-bit counter, platform
>>> - clockevents : TWD, standard
>>
>> Yep, that sounds like a good plan. If your platform has the ARM global
>> timer (drivers/clocksource/arm_global_timer.c) then you don't need
>> anything besides that timer because it provides both the clocksource,
>> sched_clock, and clockevents. Sounds like you don't have that timer
>
> I'm using Cortex A9, so I do have the global timer (AFAIU).
>
> However, I don't think I can safely use it as a clock source because
> I'm also using cpufreq, and I don't think the arm_global_timer driver
> handles CPU frequency updates, while the TWD driver does (?)
>
> /*
>   * Updates clockevent frequency when the cpu frequency changes.
>   * Called on the cpu that is changing frequency with interrupts disabled.
>   */
>
> "The Interrupt Controller, global timer, private timers, and watchdogs
> are clocked with PERIPHCLK." And PERIPHCLK is tied to CLK, which is
> modified by cpufreq. I didn't see any code in the arm_global_timer
> driver to deal with with that.
>
> Did I overlook something fundamental?
>
>> though, so you have to write a driver for your custom platform timer and
>> at least hook up clocksource and sched_clock to it. If you have
>> interrupts with your platform timer you can skip out on TWD and also
>> register a clockevent in your platform timer driver.
>
> Registering a clock source or a sched_clock seems straight-forward.
> All I need to provide is a function to read the platform counter.
>
> However, why would I skip out on TWD?
> (I'm trying to minimize code needed for the port.)

Please forgive me for piling on extra questions:

Cortex A9 provides the global timer, and the local timers/watchdog,
both of which can be used for clockevents, AFAICT. Are there other
choices available, or are these the only (standard) options?

Also, you wrote "I don't see any problem with the TWD dropping the
dependency on SMP." Would something as simple as this be acceptable?
(Most probably NOT; there are a lot of smp* occurrences in smp_twd.c
even the file name.) What is the rationale for the dependency?


diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index a34698d..47b02c8 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1565,7 +1565,6 @@ config HAVE_ARM_ARCH_TIMER
  
  config HAVE_ARM_TWD
         bool
-       depends on SMP
         select CLKSRC_OF if OF
         help
           This options enables support for the ARM timer and watchdog unit


Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Mason wrote:

> Stephen Boyd wrote:
>
>> Mason wrote:
>>
>>> My platform provides a 32-bit counter, ticking at a constant 27 MHz.
>>> Reading this counter has a latency of roughly 70 ns (it has to go
>>> over the system memory bus). I think this is good enough for both
>>> the clock source and sched_clock, is it not?
>>>
>>> So the plan would be:
>>> - clocksource and sched_clock : 27 MHz, 32-bit counter, platform
>>> - clockevents : TWD, standard
>>
>> Yep, that sounds like a good plan. If your platform has the ARM global
>> timer (drivers/clocksource/arm_global_timer.c) then you don't need
>> anything besides that timer because it provides both the clocksource,
>> sched_clock, and clockevents. Sounds like you don't have that timer
>
> I'm using Cortex A9, so I do have the global timer (AFAIU).
>
> However, I don't think I can safely use it as a clock source because
> I'm also using cpufreq, and I don't think the arm_global_timer driver
> handles CPU frequency updates, while the TWD driver does (?)
>
> /*
>   * Updates clockevent frequency when the cpu frequency changes.
>   * Called on the cpu that is changing frequency with interrupts disabled.
>   */
>
> "The Interrupt Controller, global timer, private timers, and watchdogs
> are clocked with PERIPHCLK." And PERIPHCLK is tied to CLK, which is
> modified by cpufreq. I didn't see any code in the arm_global_timer
> driver to deal with with that.
>
> Did I overlook something fundamental?
>
>> though, so you have to write a driver for your custom platform timer and
>> at least hook up clocksource and sched_clock to it. If you have
>> interrupts with your platform timer you can skip out on TWD and also
>> register a clockevent in your platform timer driver.
>
> Registering a clock source or a sched_clock seems straight-forward.
> All I need to provide is a function to read the platform counter.
>
> However, why would I skip out on TWD?
> (I'm trying to minimize code needed for the port.)

Please forgive me for piling on extra questions:

Cortex A9 provides the global timer, and the local timers/watchdog,
both of which can be used for clockevents, AFAICT. Are there other
choices available, or are these the only (standard) options?

Also, you wrote "I don't see any problem with the TWD dropping the
dependency on SMP." Would something as simple as this be acceptable?
(Most probably NOT; there are a lot of smp* occurrences in smp_twd.c
even the file name.) What is the rationale for the dependency?


diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index a34698d..47b02c8 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1565,7 +1565,6 @@ config HAVE_ARM_ARCH_TIMER
  
  config HAVE_ARM_TWD
         bool
-       depends on SMP
         select CLKSRC_OF if OF
         help
           This options enables support for the ARM timer and watchdog unit


Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Stephen Boyd]

On 02/11, Mason wrote:
> Mason wrote:
> 
> 
> Also, you wrote "I don't see any problem with the TWD dropping the
> dependency on SMP." Would something as simple as this be acceptable?
> (Most probably NOT; there are a lot of smp* occurrences in smp_twd.c
> even the file name.) What is the rationale for the dependency?
> 
> 
> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> index a34698d..47b02c8 100644
> --- a/arch/arm/Kconfig
> +++ b/arch/arm/Kconfig
> @@ -1565,7 +1565,6 @@ config HAVE_ARM_ARCH_TIMER
>  config HAVE_ARM_TWD
>         bool
> -       depends on SMP
>         select CLKSRC_OF if OF
>         help
>           This options enables support for the ARM timer and watchdog unit
> 

Hmm it looks like we would also need to add this to the patch. It
looks like a holdover from when the local timer APIs were around.
Back then twd_local_timer_common_register() would fail if
is_smp() was false or setup_max_cpus was 0. Now it will just
register a clockevent that may or may not be used depending on
what other clockevents are in the system and the ratings of those
other clockevents.

----8<----
diff --git a/arch/arm/kernel/smp_twd.c b/arch/arm/kernel/smp_twd.c
index 172c6a05d27f..e8f6d241881f 100644
--- a/arch/arm/kernel/smp_twd.c
+++ b/arch/arm/kernel/smp_twd.c
@@ -23,7 +23,6 @@
 #include <linux/of_irq.h>
 #include <linux/of_address.h>
 
-#include <asm/smp_plat.h>
 #include <asm/smp_twd.h>
 
 /* set up by the platform code */
@@ -388,9 +387,6 @@ static void __init twd_local_timer_of_register(struct device_node *np)
 {
 	int err;
 
-	if (!is_smp() || !setup_max_cpus)
-		return;
-
 	twd_ppi = irq_of_parse_and_map(np, 0);
 	if (!twd_ppi) {
 		err = -EINVAL;


-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Delays, clocks, timers, hrtimers, etc

[Stephen Boyd]

On 02/11, Mason wrote:
> Mason wrote:
> 
> 
> Also, you wrote "I don't see any problem with the TWD dropping the
> dependency on SMP." Would something as simple as this be acceptable?
> (Most probably NOT; there are a lot of smp* occurrences in smp_twd.c
> even the file name.) What is the rationale for the dependency?
> 
> 
> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> index a34698d..47b02c8 100644
> --- a/arch/arm/Kconfig
> +++ b/arch/arm/Kconfig
> @@ -1565,7 +1565,6 @@ config HAVE_ARM_ARCH_TIMER
>  config HAVE_ARM_TWD
>         bool
> -       depends on SMP
>         select CLKSRC_OF if OF
>         help
>           This options enables support for the ARM timer and watchdog unit
> 

Hmm it looks like we would also need to add this to the patch. It
looks like a holdover from when the local timer APIs were around.
Back then twd_local_timer_common_register() would fail if
is_smp() was false or setup_max_cpus was 0. Now it will just
register a clockevent that may or may not be used depending on
what other clockevents are in the system and the ratings of those
other clockevents.

----8<----
diff --git a/arch/arm/kernel/smp_twd.c b/arch/arm/kernel/smp_twd.c
index 172c6a05d27f..e8f6d241881f 100644
--- a/arch/arm/kernel/smp_twd.c
+++ b/arch/arm/kernel/smp_twd.c
@@ -23,7 +23,6 @@
 #include <linux/of_irq.h>
 #include <linux/of_address.h>
 
-#include <asm/smp_plat.h>
 #include <asm/smp_twd.h>
 
 /* set up by the platform code */
@@ -388,9 +387,6 @@ static void __init twd_local_timer_of_register(struct device_node *np)
 {
 	int err;
 
-	if (!is_smp() || !setup_max_cpus)
-		return;
-
 	twd_ppi = irq_of_parse_and_map(np, 0);
 	if (!twd_ppi) {
 		err = -EINVAL;


-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: Delays, clocks, timers, hrtimers, etc

[Mason]

Stephen Boyd wrote:

> Mason wrote:
>
>> Also, you wrote "I don't see any problem with the TWD dropping the
>> dependency on SMP." Would something as simple as this be acceptable?
>> (Most probably NOT; there are a lot of smp* occurrences in smp_twd.c
>> even the file name.) What is the rationale for the dependency?
>>
>>
>> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
>> index a34698d..47b02c8 100644
>> --- a/arch/arm/Kconfig
>> +++ b/arch/arm/Kconfig
>> @@ -1565,7 +1565,6 @@ config HAVE_ARM_ARCH_TIMER
>>   config HAVE_ARM_TWD
>>          bool
>> -       depends on SMP
>>          select CLKSRC_OF if OF
>>          help
>>            This options enables support for the ARM timer and watchdog unit
>>
>
> Hmm it looks like we would also need to add this to the patch. It
> looks like a holdover from when the local timer APIs were around.
> Back then twd_local_timer_common_register() would fail if
> is_smp() was false or setup_max_cpus was 0. Now it will just
> register a clockevent that may or may not be used depending on
> what other clockevents are in the system and the ratings of those
> other clockevents.
>
> ----8<----
> diff --git a/arch/arm/kernel/smp_twd.c b/arch/arm/kernel/smp_twd.c
> index 172c6a05d27f..e8f6d241881f 100644
> --- a/arch/arm/kernel/smp_twd.c
> +++ b/arch/arm/kernel/smp_twd.c
> @@ -23,7 +23,6 @@
>   #include <linux/of_irq.h>
>   #include <linux/of_address.h>
>
> -#include <asm/smp_plat.h>
>   #include <asm/smp_twd.h>
>
>   /* set up by the platform code */
> @@ -388,9 +387,6 @@ static void __init twd_local_timer_of_register(struct device_node *np)
>   {
>   	int err;
>
> -	if (!is_smp() || !setup_max_cpus)
> -		return;
> -
>   	twd_ppi = irq_of_parse_and_map(np, 0);
>   	if (!twd_ppi) {
>   		err = -EINVAL;

Looking at the difference between OF and !OF registration functions
in smp_twd.c

!OF
   if (twd_base || twd_evt) return -EBUSY;
   assign twd_ppi and twd_base
   twd_local_timer_common_register(NULL)

OF
   if (!is_smp() || !setup_max_cpus) return; // TO BE DELETED
   assign twd_ppi and twd_base
   twd_local_timer_common_register(np)

I suppose OF guarantees that the init code is called only once?
(Hence the !OF guard code would be unnecessary.)

Regards.

Delays, clocks, timers, hrtimers, etc

[Mason]

Stephen Boyd wrote:

> Mason wrote:
>
>> Also, you wrote "I don't see any problem with the TWD dropping the
>> dependency on SMP." Would something as simple as this be acceptable?
>> (Most probably NOT; there are a lot of smp* occurrences in smp_twd.c
>> even the file name.) What is the rationale for the dependency?
>>
>>
>> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
>> index a34698d..47b02c8 100644
>> --- a/arch/arm/Kconfig
>> +++ b/arch/arm/Kconfig
>> @@ -1565,7 +1565,6 @@ config HAVE_ARM_ARCH_TIMER
>>   config HAVE_ARM_TWD
>>          bool
>> -       depends on SMP
>>          select CLKSRC_OF if OF
>>          help
>>            This options enables support for the ARM timer and watchdog unit
>>
>
> Hmm it looks like we would also need to add this to the patch. It
> looks like a holdover from when the local timer APIs were around.
> Back then twd_local_timer_common_register() would fail if
> is_smp() was false or setup_max_cpus was 0. Now it will just
> register a clockevent that may or may not be used depending on
> what other clockevents are in the system and the ratings of those
> other clockevents.
>
> ----8<----
> diff --git a/arch/arm/kernel/smp_twd.c b/arch/arm/kernel/smp_twd.c
> index 172c6a05d27f..e8f6d241881f 100644
> --- a/arch/arm/kernel/smp_twd.c
> +++ b/arch/arm/kernel/smp_twd.c
> @@ -23,7 +23,6 @@
>   #include <linux/of_irq.h>
>   #include <linux/of_address.h>
>
> -#include <asm/smp_plat.h>
>   #include <asm/smp_twd.h>
>
>   /* set up by the platform code */
> @@ -388,9 +387,6 @@ static void __init twd_local_timer_of_register(struct device_node *np)
>   {
>   	int err;
>
> -	if (!is_smp() || !setup_max_cpus)
> -		return;
> -
>   	twd_ppi = irq_of_parse_and_map(np, 0);
>   	if (!twd_ppi) {
>   		err = -EINVAL;

Looking at the difference between OF and !OF registration functions
in smp_twd.c

!OF
   if (twd_base || twd_evt) return -EBUSY;
   assign twd_ppi and twd_base
   twd_local_timer_common_register(NULL)

OF
   if (!is_smp() || !setup_max_cpus) return; // TO BE DELETED
   assign twd_ppi and twd_base
   twd_local_timer_common_register(np)

I suppose OF guarantees that the init code is called only once?
(Hence the !OF guard code would be unnecessary.)

Regards.

Re: Delays, clocks, timers, hrtimers, etc

[Stephen Boyd]

On 02/11, Mason wrote:
> 
> Looking at the difference between OF and !OF registration functions
> in smp_twd.c
> 
> !OF
>   if (twd_base || twd_evt) return -EBUSY;
>   assign twd_ppi and twd_base
>   twd_local_timer_common_register(NULL)
> 
> OF
>   if (!is_smp() || !setup_max_cpus) return; // TO BE DELETED
>   assign twd_ppi and twd_base
>   twd_local_timer_common_register(np)
> 
> I suppose OF guarantees that the init code is called only once?

It's called as many times as there's a match with the
corresponding CLOCKSOURCE_OF_DECLARE. In practice that's once.

> (Hence the !OF guard code would be unnecessary.)
> 

The other difference is that OF based registration needs to use
OF functions for irqs and ioremap whereas the non-OF code relies
on the irqs being provided before hand and the ioremap done
without OF functions.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Delays, clocks, timers, hrtimers, etc

[Stephen Boyd]

On 02/11, Mason wrote:
> 
> Looking at the difference between OF and !OF registration functions
> in smp_twd.c
> 
> !OF
>   if (twd_base || twd_evt) return -EBUSY;
>   assign twd_ppi and twd_base
>   twd_local_timer_common_register(NULL)
> 
> OF
>   if (!is_smp() || !setup_max_cpus) return; // TO BE DELETED
>   assign twd_ppi and twd_base
>   twd_local_timer_common_register(np)
> 
> I suppose OF guarantees that the init code is called only once?

It's called as many times as there's a match with the
corresponding CLOCKSOURCE_OF_DECLARE. In practice that's once.

> (Hence the !OF guard code would be unnecessary.)
> 

The other difference is that OF based registration needs to use
OF functions for irqs and ioremap whereas the non-OF code relies
on the irqs being provided before hand and the ioremap done
without OF functions.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project