CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Mason]

Hello,

I'm working on a dual-core SoC (Cortex-A9 MPCore) with Linux running
as the non-secure OS. The secure OS implements some of the APIs used
in OMAP2 via SMC.

To start the second core, I used to define .smp in my DT_MACHINE_START
definition, but Arnd wrote: "There are other ways to do [.smp and .restart]
too, see CPU_METHOD_OF_DECLARE() and register_restart_handler()."

Is CPU_METHOD_OF_DECLARE() the way to go?
Is it "compatible" with executing smc to ask the secure OS to do whatever
is needed to start the second core? (I don't understand why starting cores
is secure OS responsibility.)

I've located three relevant secure OS services:
get_aux_core_boot
set_aux_core_boot0
set_aux_core_boot_addr

I'm looking for guidance. What mach-* should I take as a reference to
implement "second core startup from non-secure OS"?

An ARM engineer suggested PSCI a while back, but I can't change the
secure OS API at this time.

Regards.

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Rob Herring]

On Wed, Oct 28, 2015 at 7:36 AM, Mason <slash.tmp@free.fr> wrote:
> Hello,
>
> I'm working on a dual-core SoC (Cortex-A9 MPCore) with Linux running
> as the non-secure OS. The secure OS implements some of the APIs used
> in OMAP2 via SMC.
>
> To start the second core, I used to define .smp in my DT_MACHINE_START
> definition, but Arnd wrote: "There are other ways to do [.smp and .restart]
> too, see CPU_METHOD_OF_DECLARE() and register_restart_handler()."
>
> Is CPU_METHOD_OF_DECLARE() the way to go?

Yes.

> Is it "compatible" with executing smc to ask the secure OS to do whatever
> is needed to start the second core? (I don't understand why starting cores
> is secure OS responsibility.)

Yes, that is what PSCI does. In your case, these are not PSCI calls though?

>
> I've located three relevant secure OS services:
> get_aux_core_boot
> set_aux_core_boot0
> set_aux_core_boot_addr
>
> I'm looking for guidance. What mach-* should I take as a reference to
> implement "second core startup from non-secure OS"?

I don't think there is a mach example that matches. PSCI would be the
closest example. But doing a smc call versus writing some registers
are not all that different and there are numerous examples of the
latter.

Rob

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Måns Rullgård]

Mason <slash.tmp@free.fr> writes:

> Hello,
>
> I'm working on a dual-core SoC (Cortex-A9 MPCore) with Linux running
> as the non-secure OS. The secure OS implements some of the APIs used
> in OMAP2 via SMC.
>
> To start the second core, I used to define .smp in my DT_MACHINE_START
> definition, but Arnd wrote: "There are other ways to do [.smp and .restart]
> too, see CPU_METHOD_OF_DECLARE() and register_restart_handler()."
>
> Is CPU_METHOD_OF_DECLARE() the way to go?
> Is it "compatible" with executing smc to ask the secure OS to do whatever
> is needed to start the second core? (I don't understand why starting cores
> is secure OS responsibility.)
>
> I've located three relevant secure OS services:
> get_aux_core_boot
> set_aux_core_boot0
> set_aux_core_boot_addr
>
> I'm looking for guidance. What mach-* should I take as a reference to
> implement "second core startup from non-secure OS"?

See https://github.com/mansr/linux-tangox/commit/717dd5fab87aa90255460ba409970331cb577247

-- 
M?ns Rullg?rd
mans at mansr.com

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Mason]

On 29/10/2015 17:03, M?ns Rullg?rd wrote:

> Mason writes:
> 
>> I'm working on a dual-core SoC (Cortex-A9 MPCore) with Linux running
>> as the non-secure OS. The secure OS implements some of the APIs used
>> in OMAP2 via SMC.
>>
>> To start the second core, I used to define .smp in my DT_MACHINE_START
>> definition, but Arnd wrote: "There are other ways to do [.smp and .restart]
>> too, see CPU_METHOD_OF_DECLARE() and register_restart_handler()."
>>
>> Is CPU_METHOD_OF_DECLARE() the way to go?
>> Is it "compatible" with executing smc to ask the secure OS to do whatever
>> is needed to start the second core? (I don't understand why starting cores
>> is secure OS responsibility.)
>>
>> I've located three relevant secure OS services:
>> get_aux_core_boot
>> set_aux_core_boot0
>> set_aux_core_boot_addr
>>
>> I'm looking for guidance. What mach-* should I take as a reference to
>> implement "second core startup from non-secure OS"?
> 
> See https://github.com/mansr/linux-tangox/commit/717dd5fab87aa90255460ba409970331cb577247

I discussed cpu1 startup with the firmware author; he said the
secure OS does all the heavy lifting, Linux is only expected to
provide secondary_startup. AFAICT, we don't need to jump hoops
like mach-vexpress did.

I don't think you can define the cpus node in your smp87xx.dtsi
because smp8756 is single-core.

BTW, I see that you've tried to enable L2C. I would expect your
board would panic?

http://thread.gmane.org/gmane.linux.ports.arm.kernel/447493

Regards.

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Måns Rullgård]

Mason <slash.tmp@free.fr> writes:

> On 29/10/2015 17:03, M?ns Rullg?rd wrote:
>
>> Mason writes:
>> 
>>> I'm working on a dual-core SoC (Cortex-A9 MPCore) with Linux running
>>> as the non-secure OS. The secure OS implements some of the APIs used
>>> in OMAP2 via SMC.
>>>
>>> To start the second core, I used to define .smp in my DT_MACHINE_START
>>> definition, but Arnd wrote: "There are other ways to do [.smp and .restart]
>>> too, see CPU_METHOD_OF_DECLARE() and register_restart_handler()."
>>>
>>> Is CPU_METHOD_OF_DECLARE() the way to go?
>>> Is it "compatible" with executing smc to ask the secure OS to do whatever
>>> is needed to start the second core? (I don't understand why starting cores
>>> is secure OS responsibility.)
>>>
>>> I've located three relevant secure OS services:
>>> get_aux_core_boot
>>> set_aux_core_boot0
>>> set_aux_core_boot_addr
>>>
>>> I'm looking for guidance. What mach-* should I take as a reference to
>>> implement "second core startup from non-secure OS"?
>> 
>> See https://github.com/mansr/linux-tangox/commit/717dd5fab87aa90255460ba409970331cb577247
>
> I discussed cpu1 startup with the firmware author; he said the
> secure OS does all the heavy lifting, Linux is only expected to
> provide secondary_startup. AFAICT, we don't need to jump hoops
> like mach-vexpress did.

No major hoops really, and it's nice to have there as an extra safety
check.

> I don't think you can define the cpus node in your smp87xx.dtsi
> because smp8756 is single-core.

OK, I'll fix that.

> BTW, I see that you've tried to enable L2C. I would expect your
> board would panic?
>
> http://thread.gmane.org/gmane.linux.ports.arm.kernel/447493

I hacked around that, and everything works fine.  Nice performance boost
too.

There's also something wrong with the L2C-310 aux control register
setting.  The SMC call ID from OMAP (0x109) which is also used in some
Sigma code I found somewhere doesn't seem to do anything, so the
register is left at the value set by the secure boot code.  Perhaps you
can check with your firmware guy if there's another way of writing that
register.

-- 
M?ns Rullg?rd
mans at mansr.com

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Mason]

On 29/10/2015 19:04, M?ns Rullg?rd wrote:

> There's also something wrong with the L2C-310 aux control register
> setting.  The SMC call ID from OMAP (0x109) which is also used in some
> Sigma code I found somewhere doesn't seem to do anything, so the
> register is left at the value set by the secure boot code.  Perhaps you
> can check with your firmware guy if there's another way of writing that
> register.

IIRC, only debug firmware allows writes to L2 AUXCTRL (after filtering
some of the bits out), while production firmware ignores them completely.

IME, the smc handler should default to return ENOTSUP; that way, when
a syscall disappears due to ifdef-ery, the caller gets a meaningful
answer.

We just had an interesting internal discussion about L2 AUXCTRL.
For my education, what value would you like to write to AUXCTRL? :-)

Regards.

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Måns Rullgård]

Mason <slash.tmp@free.fr> writes:

> On 29/10/2015 19:04, M?ns Rullg?rd wrote:
>
>> There's also something wrong with the L2C-310 aux control register
>> setting.  The SMC call ID from OMAP (0x109) which is also used in some
>> Sigma code I found somewhere doesn't seem to do anything, so the
>> register is left at the value set by the secure boot code.  Perhaps you
>> can check with your firmware guy if there's another way of writing that
>> register.
>
> IIRC, only debug firmware allows writes to L2 AUXCTRL (after filtering
> some of the bits out), while production firmware ignores them completely.
>
> IME, the smc handler should default to return ENOTSUP; that way, when
> a syscall disappears due to ifdef-ery, the caller gets a meaningful
> answer.
>
> We just had an interesting internal discussion about L2 AUXCTRL.
> For my education, what value would you like to write to AUXCTRL? :-)

The best value to use depends on the workload, so it would be nice to be
able to control all the purely performance related bits.  I see no
possible benefit in restricting the non-secure kernel from writing
these.

-- 
M?ns Rullg?rd
mans at mansr.com

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Mason]

On 31/10/2015 16:41, M?ns Rullg?rd wrote:

> Mason writes:
> 
>> On 29/10/2015 19:04, M?ns Rullg?rd wrote:
>>
>>> There's also something wrong with the L2C-310 aux control register
>>> setting.  The SMC call ID from OMAP (0x109) which is also used in some
>>> Sigma code I found somewhere doesn't seem to do anything, so the
>>> register is left at the value set by the secure boot code.  Perhaps you
>>> can check with your firmware guy if there's another way of writing that
>>> register.
>>
>> IIRC, only debug firmware allows writes to L2 AUXCTRL (after filtering
>> some of the bits out), while production firmware ignores them completely.
>>
>> IME, the smc handler should default to return ENOTSUP; that way, when
>> a syscall disappears due to ifdef-ery, the caller gets a meaningful
>> answer.
>>
>> We just had an interesting internal discussion about L2 AUXCTRL.
>> For my education, what value would you like to write to AUXCTRL? :-)
> 
> The best value to use depends on the workload, so it would be nice to be
> able to control all the purely performance related bits.  I see no
> possible benefit in restricting the non-secure kernel from writing
> these.

For the record, the latest firmware uses 0x72860401.

[ 0] Full Line of Zero enable
[10] High Priority for SO and Dev Reads enable
[11] Store buffer device limitation disable
[12] Exclusive cache configuration disable
[13] Shared Attribute Invalidate disable
[16] Associativity 8-way
[17-19] Way-size = 64
[20] Event monitor bus disable
[21] Parity disable
[22] Shared attribute override disable
[23:24] Force write allocate = 0x1
[25] Cache replacement policy = 1
[26] Non-secure lock-down disable
[27] Non-secure interrupt access control disable
[28] Data prefetch enable
[29] Instruction prefetch enable
[30] Early BRESP enable

Everyone, feel free to comment on these choices ;-)

Regards.

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Russell King - ARM Linux]

On Sat, Oct 31, 2015 at 03:41:57PM +0000, M?ns Rullg?rd wrote:
> The best value to use depends on the workload, so it would be nice to be
> able to control all the purely performance related bits.  I see no
> possible benefit in restricting the non-secure kernel from writing
> these.

For the most case, the auxiliary control register settings are static.
Once the L2 cache is enabled, you can't write to the control register
until the L2 cache is disabled, and you can't sensibly do that on a
running system without taking the system down.

Those which can be tuned at runtime (the prefetch enables) are also
available in the prefetch control register, along with the prefetch
offset.

The auxiliary control register is really to do with configuring the
cache to the system its in rather than about performance.

Eg, you could have the FLZ bit set in the L2 cache auxiliary register
if its wired to a Cortex A9 CPU and the FLZ signal is wired.  That
would be perfectly reasonable, provided the FLZ bit in the Cortex
A9's control register is disabled when the L2 cache is otherwise
disabled.

(I guess I ought to quieten down the pr_err() in that case...)

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Mason]

On 02/11/2015 11:17, Russell King - ARM Linux wrote:

> Eg, you could have the FLZ bit set in the L2 cache auxiliary register
> if its wired to a Cortex A9 CPU and the FLZ signal is wired.  That
> would be perfectly reasonable, provided the FLZ bit in the Cortex
> A9's control register is disabled when the L2 cache is otherwise
> disabled.
> 
> (I guess I ought to quieten down the pr_err() in that case...)

Based on the pr_err(), I thought it was considered an error, so
I asked the firmware author to leave L2AUX.FLZ cleared, and only
set it just before the cache is enabled, and clear it after the
cache is disabled.

Do you remember why you considered it an error at the time?

Regards.

CPU_METHOD_OF_DECLARE() with Linux as non-secure OS

[Måns Rullgård]

Mason <slash.tmp@free.fr> writes:

> On 31/10/2015 16:41, M?ns Rullg?rd wrote:
>
>> Mason writes:
>> 
>>> On 29/10/2015 19:04, M?ns Rullg?rd wrote:
>>>
>>>> There's also something wrong with the L2C-310 aux control register
>>>> setting.  The SMC call ID from OMAP (0x109) which is also used in some
>>>> Sigma code I found somewhere doesn't seem to do anything, so the
>>>> register is left at the value set by the secure boot code.  Perhaps you
>>>> can check with your firmware guy if there's another way of writing that
>>>> register.
>>>
>>> IIRC, only debug firmware allows writes to L2 AUXCTRL (after filtering
>>> some of the bits out), while production firmware ignores them completely.
>>>
>>> IME, the smc handler should default to return ENOTSUP; that way, when
>>> a syscall disappears due to ifdef-ery, the caller gets a meaningful
>>> answer.
>>>
>>> We just had an interesting internal discussion about L2 AUXCTRL.
>>> For my education, what value would you like to write to AUXCTRL? :-)
>> 
>> The best value to use depends on the workload, so it would be nice to be
>> able to control all the purely performance related bits.  I see no
>> possible benefit in restricting the non-secure kernel from writing
>> these.
>
> For the record, the latest firmware uses 0x72860401.

My SMP8759 firmware sets it to 0x62460801.

> [ 0] Full Line of Zero enable

This is only takes effect if the feature is also enabled in the cp15 aux
control register, so it's safe to always set it.  It would of course be
nice to have a way of updating the cp15 aux control register...

> [10] High Priority for SO and Dev Reads enable

The best value here depends on the workload.  It would be nice for the
system integrator to have control over it.

> [11] Store buffer device limitation disable

Ditto.

> [12] Exclusive cache configuration disable

Like FLZ, this has a corresponding cp15 bit.  It is a potentially useful
feature, so again it would be nice to have control over it.

> [13] Shared Attribute Invalidate disable
> [22] Shared attribute override disable

Not sure about these.

> [16] Associativity 8-way
> [17-19] Way-size = 64

These must match the actual hardware.

> [20] Event monitor bus disable

OK to disable.

> [21] Parity disable

Must match hardware.

> [23:24] Force write allocate = 0x1

Why was this changed?  Write-allocate can be useful, and this attribute
is normally supplied by the page table.  Forcing it off should only be
done to work around broken hardware or software that can't be fixed.

> [25] Cache replacement policy = 1

Again, the best value depends on the workload.  Better to let the OS choose.

> [26] Non-secure lock-down disable

The Linux setup code has this comment:

	/*
	 * Always enable non-secure access to the lockdown registers -
	 * we write to them as part of the L2C enable sequence so they
	 * need to be accessible.
	 */

The lockdown registers are only accessed if this bit is actually set, so
leaving it off is harmless provided the cache is left unlocked by the
secure firmware.

> [27] Non-secure interrupt access control disable

Access to the interrupt control registers is useful in conjunction with
the event counters.

> [28] Data prefetch enable
> [29] Instruction prefetch enable

These are accessible also from the prefetch control register, so as long
as that is accessible, the values set here don't matter.

> [30] Early BRESP enable

Linux wants this enabled.  It is generally good for performance.

-- 
M?ns Rullg?rd
mans at mansr.com