[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Kevin Hilman]

From: Kevin Hilman <khilman@linaro.org>

Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
and 7 do not boot:

   [...]
   Exynos MCPM support installed
   CPU1: update cpu_capacity 1535
   CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
   CPU2: update cpu_capacity 1535
   CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
   CPU3: update cpu_capacity 1535
   CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
   CPU4: update cpu_capacity 1535
   CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
   CPU5: failed to come online
   CPU6: update cpu_capacity 448
   CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
   CPU7: failed to come online
   Brought up 6 CPUs
   CPU: WARNING: CPU(s) started in wrong/inconsistent modes
   (primary CPU mode 0x13)
   CPU: This may indicate a broken bootloader or firmware.

Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
again, but the warning about CPUs started in inconsistent modes
remains.  Also, not being terribly familiar with Exynos internals,
it's not at all obvious to me why this register write (done for *all*
secondaries) makes things work works for the 2 secondary CPUs that
didn't come online.  It's also not obvious whether this is the right
general fix, since it doesn't seem to be needed on other 542x or 5800
platforms.

I suspect the "right" fix is in the bootloader someplace, but not
knowing this hardware well, I'm not sure if the fix is in u-boot
proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
before u-boot.  The u-boot I'm using is from the hardkernel u-boot
repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
my own u-boot from there, but only have binaries for bl1/bl2/tz.

[1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git


Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
Cc: Andrew Bresticker <abrestic@chromium.org>
Cc: Doug Anderson <dianders@chromium.org>
Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: Kevin Hilman <khilman@linaro.org>
---
 arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
 arch/arm/mach-exynos/regs-pmu.h    | 1 +
 2 files changed, 3 insertions(+)

diff --git a/arch/arm/mach-exynos/mcpm-exynos.c b/arch/arm/mach-exynos/mcpm-exynos.c
index b0d3c2e876fb..612a770d5284 100644
--- a/arch/arm/mach-exynos/mcpm-exynos.c
+++ b/arch/arm/mach-exynos/mcpm-exynos.c
@@ -88,6 +88,8 @@ static int exynos_power_up(unsigned int cpu, unsigned int cluster)
 		cluster >= EXYNOS5420_NR_CLUSTERS)
 		return -EINVAL;
 
+	pmu_raw_writel(0x1, S5P_PMU_SPARE2);
+
 	/*
 	 * Since this is called with IRQs enabled, and no arch_spin_lock_irq
 	 * variant exists, we need to disable IRQs manually here.
diff --git a/arch/arm/mach-exynos/regs-pmu.h b/arch/arm/mach-exynos/regs-pmu.h
index b5f4406fc1b5..70d9eb5a4fcc 100644
--- a/arch/arm/mach-exynos/regs-pmu.h
+++ b/arch/arm/mach-exynos/regs-pmu.h
@@ -49,6 +49,7 @@
 #define S5P_INFORM5				0x0814
 #define S5P_INFORM6				0x0818
 #define S5P_INFORM7				0x081C
+#define S5P_PMU_SPARE2				0x0908
 #define S5P_PMU_SPARE3				0x090C
 
 #define EXYNOS_IROM_DATA2			0x0988
-- 
2.1.3

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Kevin Hilman]

From: Kevin Hilman <khilman@linaro.org>

Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
and 7 do not boot:

   [...]
   Exynos MCPM support installed
   CPU1: update cpu_capacity 1535
   CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
   CPU2: update cpu_capacity 1535
   CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
   CPU3: update cpu_capacity 1535
   CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
   CPU4: update cpu_capacity 1535
   CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
   CPU5: failed to come online
   CPU6: update cpu_capacity 448
   CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
   CPU7: failed to come online
   Brought up 6 CPUs
   CPU: WARNING: CPU(s) started in wrong/inconsistent modes
   (primary CPU mode 0x13)
   CPU: This may indicate a broken bootloader or firmware.

Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
again, but the warning about CPUs started in inconsistent modes
remains.  Also, not being terribly familiar with Exynos internals,
it's not at all obvious to me why this register write (done for *all*
secondaries) makes things work works for the 2 secondary CPUs that
didn't come online.  It's also not obvious whether this is the right
general fix, since it doesn't seem to be needed on other 542x or 5800
platforms.

I suspect the "right" fix is in the bootloader someplace, but not
knowing this hardware well, I'm not sure if the fix is in u-boot
proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
before u-boot.  The u-boot I'm using is from the hardkernel u-boot
repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
my own u-boot from there, but only have binaries for bl1/bl2/tz.

[1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git


Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
Cc: Andrew Bresticker <abrestic@chromium.org>
Cc: Doug Anderson <dianders@chromium.org>
Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: Kevin Hilman <khilman@linaro.org>
---
 arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
 arch/arm/mach-exynos/regs-pmu.h    | 1 +
 2 files changed, 3 insertions(+)

diff --git a/arch/arm/mach-exynos/mcpm-exynos.c b/arch/arm/mach-exynos/mcpm-exynos.c
index b0d3c2e876fb..612a770d5284 100644
--- a/arch/arm/mach-exynos/mcpm-exynos.c
+++ b/arch/arm/mach-exynos/mcpm-exynos.c
@@ -88,6 +88,8 @@ static int exynos_power_up(unsigned int cpu, unsigned int cluster)
 		cluster >= EXYNOS5420_NR_CLUSTERS)
 		return -EINVAL;
 
+	pmu_raw_writel(0x1, S5P_PMU_SPARE2);
+
 	/*
 	 * Since this is called with IRQs enabled, and no arch_spin_lock_irq
 	 * variant exists, we need to disable IRQs manually here.
diff --git a/arch/arm/mach-exynos/regs-pmu.h b/arch/arm/mach-exynos/regs-pmu.h
index b5f4406fc1b5..70d9eb5a4fcc 100644
--- a/arch/arm/mach-exynos/regs-pmu.h
+++ b/arch/arm/mach-exynos/regs-pmu.h
@@ -49,6 +49,7 @@
 #define S5P_INFORM5				0x0814
 #define S5P_INFORM6				0x0818
 #define S5P_INFORM7				0x081C
+#define S5P_PMU_SPARE2				0x0908
 #define S5P_PMU_SPARE3				0x090C
 
 #define EXYNOS_IROM_DATA2			0x0988
-- 
2.1.3

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Heesub Shin]

Kevin Hilman <khilman <at> kernel.org> writes:

> 
> From: Kevin Hilman <khilman <at> linaro.org>
> 
> Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
> 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
> A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
> and 7 do not boot:
> 
>    [...]
>    Exynos MCPM support installed
>    CPU1: update cpu_capacity 1535
>    CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
>    CPU2: update cpu_capacity 1535
>    CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
>    CPU3: update cpu_capacity 1535
>    CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
>    CPU4: update cpu_capacity 1535
>    CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
>    CPU5: failed to come online
>    CPU6: update cpu_capacity 448
>    CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
>    CPU7: failed to come online
>    Brought up 6 CPUs
>    CPU: WARNING: CPU(s) started in wrong/inconsistent modes
>    (primary CPU mode 0x13)
>    CPU: This may indicate a broken bootloader or firmware.
> 
> Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
> again, but the warning about CPUs started in inconsistent modes
> remains.  Also, not being terribly familiar with Exynos internals,
> it's not at all obvious to me why this register write (done for *all*
> secondaries) makes things work works for the 2 secondary CPUs that
> didn't come online.  It's also not obvious whether this is the right
> general fix, since it doesn't seem to be needed on other 542x or 5800
> platforms.

Hi,

Very interesting to see your post. I was also suffering from the same 
problem with my Odroid-XU3 board. With your patch 8 CPUs are brought up, 
but Cortex-A15 CPUs are always offline, showing low performance.

heesub@odroid:~$ cat /sys/devices/system/cpu/online
0,5-7
heesub@odroid:~$ cat /sys/devices/system/cpu/offline
1-4

Any suggestion?

Thanks a lot!!
Heesub

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Kevin Hilman]

Hello,

Heesub Shin <heesub.shin@samsung.com> writes:

>> Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
>> 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
>> A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
>> and 7 do not boot:
>> 
>>    [...]
>>    Exynos MCPM support installed
>>    CPU1: update cpu_capacity 1535
>>    CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
>>    CPU2: update cpu_capacity 1535
>>    CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
>>    CPU3: update cpu_capacity 1535
>>    CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
>>    CPU4: update cpu_capacity 1535
>>    CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
>>    CPU5: failed to come online
>>    CPU6: update cpu_capacity 448
>>    CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
>>    CPU7: failed to come online
>>    Brought up 6 CPUs
>>    CPU: WARNING: CPU(s) started in wrong/inconsistent modes
>>    (primary CPU mode 0x13)
>>    CPU: This may indicate a broken bootloader or firmware.
>> 
>> Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
>> again, but the warning about CPUs started in inconsistent modes
>> remains.  Also, not being terribly familiar with Exynos internals,
>> it's not at all obvious to me why this register write (done for *all*
>> secondaries) makes things work works for the 2 secondary CPUs that
>> didn't come online.  It's also not obvious whether this is the right
>> general fix, since it doesn't seem to be needed on other 542x or 5800
>> platforms.
>
> Very interesting to see your post. I was also suffering from the same 
> problem with my Odroid-XU3 board. With your patch 8 CPUs are brought up, 
> but Cortex-A15 CPUs are always offline, showing low performance.
>
> heesub@odroid:~$ cat /sys/devices/system/cpu/online
> 0,5-7
> heesub@odroid:~$ cat /sys/devices/system/cpu/offline
> 1-4
>
> Any suggestion?

That's probably because you have the big.LITTLE switcher enabled in your
.config (which is the default when using exynos_defconfig).

If you modify your .config and set CONFIG_BL_SWITCHER=n, you will see
all 8 cores online.

Kevin

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Krzysztof Kozłowski]

2014-11-25 15:21 GMT+09:00 Kevin Hilman <khilman@kernel.org>:
> From: Kevin Hilman <khilman@linaro.org>
>
> Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
> 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
> A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
> and 7 do not boot:
>
>    [...]
>    Exynos MCPM support installed
>    CPU1: update cpu_capacity 1535
>    CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
>    CPU2: update cpu_capacity 1535
>    CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
>    CPU3: update cpu_capacity 1535
>    CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
>    CPU4: update cpu_capacity 1535
>    CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
>    CPU5: failed to come online
>    CPU6: update cpu_capacity 448
>    CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
>    CPU7: failed to come online
>    Brought up 6 CPUs
>    CPU: WARNING: CPU(s) started in wrong/inconsistent modes
>    (primary CPU mode 0x13)
>    CPU: This may indicate a broken bootloader or firmware.
>
> Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
> again, but the warning about CPUs started in inconsistent modes
> remains.  Also, not being terribly familiar with Exynos internals,
> it's not at all obvious to me why this register write (done for *all*
> secondaries) makes things work works for the 2 secondary CPUs that
> didn't come online.  It's also not obvious whether this is the right
> general fix, since it doesn't seem to be needed on other 542x or 5800
> platforms.
>
> I suspect the "right" fix is in the bootloader someplace, but not
> knowing this hardware well, I'm not sure if the fix is in u-boot
> proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
> before u-boot.  The u-boot I'm using is from the hardkernel u-boot
> repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
> my own u-boot from there, but only have binaries for bl1/bl2/tz.
>
> [1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git
>
>
> Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
> Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
> Cc: Andrew Bresticker <abrestic@chromium.org>
> Cc: Doug Anderson <dianders@chromium.org>
> Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
> Signed-off-by: Kevin Hilman <khilman@linaro.org>
> ---
>  arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
>  arch/arm/mach-exynos/regs-pmu.h    | 1 +
>  2 files changed, 3 insertions(+)

Hi,

+Cc Marek, Bartlomiej, Kukjin Kim,


I would like to bring back this topic. Unfortunately I don't have
access to source code of BL1 (or any other firmware blob) so my
knowledge here comes mostly from experimenting and from looking at
sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
S5, Exynos5422).

It seems that some booting firmware (I would suspect BL1 because this
ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
For example vendor kernel, when booting little core, it waits till
SPARE2==1 and then executes software reset for this core.

Observations shown that BL1 for Odroid, when booting secondary little core:
1. Expects that SPARE2 register will be initialized to 1.
2. If it is, then it sets it to 0, proceeds further and little core boots.
3. If it is not, then it sets it to 1 and waits. Maybe this is a
notification to userspace - reset me please!

Unfortunately executing software reset in that time (at point 3)
stopped kernel from booting. No logs/dmesg and I was unable to turn on
early printk.

The answer why two of little cores boot is quite simple now. At
beginning the SPARE2==0 so first little core will set it to 1 and wait
till software reset. Kernel timeouts on this CPU bring up so it starts
the sequence for next little core. Now the SPARE2==1 so the core boots
fine and SPARE2 is set to 0. The last little core starts from
SPARE2==0, sets it to 1 and waits for software reset.

Since no one knows how this exactly works and we are stuck with BL1
provided as is, then IMHO the patch makes sense.

Kevin, can you refresh the patch?
It would be nice to:
1. set SPARE2 only for Odroid (of_machine_is_compatible()),
2. extend the explanation.


Best regards,
Krzysztof

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Krzysztof Kozłowski]

2014-11-25 15:21 GMT+09:00 Kevin Hilman <khilman@kernel.org>:
> From: Kevin Hilman <khilman@linaro.org>
>
> Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
> 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
> A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
> and 7 do not boot:
>
>    [...]
>    Exynos MCPM support installed
>    CPU1: update cpu_capacity 1535
>    CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
>    CPU2: update cpu_capacity 1535
>    CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
>    CPU3: update cpu_capacity 1535
>    CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
>    CPU4: update cpu_capacity 1535
>    CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
>    CPU5: failed to come online
>    CPU6: update cpu_capacity 448
>    CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
>    CPU7: failed to come online
>    Brought up 6 CPUs
>    CPU: WARNING: CPU(s) started in wrong/inconsistent modes
>    (primary CPU mode 0x13)
>    CPU: This may indicate a broken bootloader or firmware.
>
> Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
> again, but the warning about CPUs started in inconsistent modes
> remains.  Also, not being terribly familiar with Exynos internals,
> it's not at all obvious to me why this register write (done for *all*
> secondaries) makes things work works for the 2 secondary CPUs that
> didn't come online.  It's also not obvious whether this is the right
> general fix, since it doesn't seem to be needed on other 542x or 5800
> platforms.
>
> I suspect the "right" fix is in the bootloader someplace, but not
> knowing this hardware well, I'm not sure if the fix is in u-boot
> proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
> before u-boot.  The u-boot I'm using is from the hardkernel u-boot
> repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
> my own u-boot from there, but only have binaries for bl1/bl2/tz.
>
> [1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git
>
>
> Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
> Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
> Cc: Andrew Bresticker <abrestic@chromium.org>
> Cc: Doug Anderson <dianders@chromium.org>
> Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
> Signed-off-by: Kevin Hilman <khilman@linaro.org>
> ---
>  arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
>  arch/arm/mach-exynos/regs-pmu.h    | 1 +
>  2 files changed, 3 insertions(+)

Hi,

+Cc Marek, Bartlomiej, Kukjin Kim,


I would like to bring back this topic. Unfortunately I don't have
access to source code of BL1 (or any other firmware blob) so my
knowledge here comes mostly from experimenting and from looking at
sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
S5, Exynos5422).

It seems that some booting firmware (I would suspect BL1 because this
ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
For example vendor kernel, when booting little core, it waits till
SPARE2==1 and then executes software reset for this core.

Observations shown that BL1 for Odroid, when booting secondary little core:
1. Expects that SPARE2 register will be initialized to 1.
2. If it is, then it sets it to 0, proceeds further and little core boots.
3. If it is not, then it sets it to 1 and waits. Maybe this is a
notification to userspace - reset me please!

Unfortunately executing software reset in that time (at point 3)
stopped kernel from booting. No logs/dmesg and I was unable to turn on
early printk.

The answer why two of little cores boot is quite simple now. At
beginning the SPARE2==0 so first little core will set it to 1 and wait
till software reset. Kernel timeouts on this CPU bring up so it starts
the sequence for next little core. Now the SPARE2==1 so the core boots
fine and SPARE2 is set to 0. The last little core starts from
SPARE2==0, sets it to 1 and waits for software reset.

Since no one knows how this exactly works and we are stuck with BL1
provided as is, then IMHO the patch makes sense.

Kevin, can you refresh the patch?
It would be nice to:
1. set SPARE2 only for Odroid (of_machine_is_compatible()),
2. extend the explanation.


Best regards,
Krzysztof

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Bartlomiej Zolnierkiewicz]

Hi,

+ Cc Przemyslaw Marczak (he is working on fixing u-boot fox XU3)

Best regards,
--
Bartlomiej Zolnierkiewicz
Samsung R&D Institute Poland
Samsung Electronics

On Sunday, June 14, 2015 05:56:20 PM Krzysztof Kozłowski wrote:
> 2014-11-25 15:21 GMT+09:00 Kevin Hilman <khilman@kernel.org>:
> > From: Kevin Hilman <khilman@linaro.org>
> >
> > Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
> > 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
> > A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
> > and 7 do not boot:
> >
> >    [...]
> >    Exynos MCPM support installed
> >    CPU1: update cpu_capacity 1535
> >    CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
> >    CPU2: update cpu_capacity 1535
> >    CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
> >    CPU3: update cpu_capacity 1535
> >    CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
> >    CPU4: update cpu_capacity 1535
> >    CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
> >    CPU5: failed to come online
> >    CPU6: update cpu_capacity 448
> >    CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
> >    CPU7: failed to come online
> >    Brought up 6 CPUs
> >    CPU: WARNING: CPU(s) started in wrong/inconsistent modes
> >    (primary CPU mode 0x13)
> >    CPU: This may indicate a broken bootloader or firmware.
> >
> > Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
> > again, but the warning about CPUs started in inconsistent modes
> > remains.  Also, not being terribly familiar with Exynos internals,
> > it's not at all obvious to me why this register write (done for *all*
> > secondaries) makes things work works for the 2 secondary CPUs that
> > didn't come online.  It's also not obvious whether this is the right
> > general fix, since it doesn't seem to be needed on other 542x or 5800
> > platforms.
> >
> > I suspect the "right" fix is in the bootloader someplace, but not
> > knowing this hardware well, I'm not sure if the fix is in u-boot
> > proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
> > before u-boot.  The u-boot I'm using is from the hardkernel u-boot
> > repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
> > my own u-boot from there, but only have binaries for bl1/bl2/tz.
> >
> > [1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git
> >
> >
> > Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
> > Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
> > Cc: Andrew Bresticker <abrestic@chromium.org>
> > Cc: Doug Anderson <dianders@chromium.org>
> > Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
> > Signed-off-by: Kevin Hilman <khilman@linaro.org>
> > ---
> >  arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
> >  arch/arm/mach-exynos/regs-pmu.h    | 1 +
> >  2 files changed, 3 insertions(+)
> 
> Hi,
> 
> +Cc Marek, Bartlomiej, Kukjin Kim,
> 
> 
> I would like to bring back this topic. Unfortunately I don't have
> access to source code of BL1 (or any other firmware blob) so my
> knowledge here comes mostly from experimenting and from looking at
> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
> S5, Exynos5422).
> 
> It seems that some booting firmware (I would suspect BL1 because this
> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
> For example vendor kernel, when booting little core, it waits till
> SPARE2==1 and then executes software reset for this core.
> 
> Observations shown that BL1 for Odroid, when booting secondary little core:
> 1. Expects that SPARE2 register will be initialized to 1.
> 2. If it is, then it sets it to 0, proceeds further and little core boots.
> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
> notification to userspace - reset me please!
> 
> Unfortunately executing software reset in that time (at point 3)
> stopped kernel from booting. No logs/dmesg and I was unable to turn on
> early printk.
> 
> The answer why two of little cores boot is quite simple now. At
> beginning the SPARE2==0 so first little core will set it to 1 and wait
> till software reset. Kernel timeouts on this CPU bring up so it starts
> the sequence for next little core. Now the SPARE2==1 so the core boots
> fine and SPARE2 is set to 0. The last little core starts from
> SPARE2==0, sets it to 1 and waits for software reset.
> 
> Since no one knows how this exactly works and we are stuck with BL1
> provided as is, then IMHO the patch makes sense.
> 
> Kevin, can you refresh the patch?
> It would be nice to:
> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
> 2. extend the explanation.
> 
> 
> Best regards,
> Krzysztof

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Bartlomiej Zolnierkiewicz]

Hi,

+ Cc Przemyslaw Marczak (he is working on fixing u-boot fox XU3)

Best regards,
--
Bartlomiej Zolnierkiewicz
Samsung R&D Institute Poland
Samsung Electronics

On Sunday, June 14, 2015 05:56:20 PM Krzysztof Koz?owski wrote:
> 2014-11-25 15:21 GMT+09:00 Kevin Hilman <khilman@kernel.org>:
> > From: Kevin Hilman <khilman@linaro.org>
> >
> > Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
> > 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
> > A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
> > and 7 do not boot:
> >
> >    [...]
> >    Exynos MCPM support installed
> >    CPU1: update cpu_capacity 1535
> >    CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
> >    CPU2: update cpu_capacity 1535
> >    CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
> >    CPU3: update cpu_capacity 1535
> >    CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
> >    CPU4: update cpu_capacity 1535
> >    CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
> >    CPU5: failed to come online
> >    CPU6: update cpu_capacity 448
> >    CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
> >    CPU7: failed to come online
> >    Brought up 6 CPUs
> >    CPU: WARNING: CPU(s) started in wrong/inconsistent modes
> >    (primary CPU mode 0x13)
> >    CPU: This may indicate a broken bootloader or firmware.
> >
> > Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
> > again, but the warning about CPUs started in inconsistent modes
> > remains.  Also, not being terribly familiar with Exynos internals,
> > it's not at all obvious to me why this register write (done for *all*
> > secondaries) makes things work works for the 2 secondary CPUs that
> > didn't come online.  It's also not obvious whether this is the right
> > general fix, since it doesn't seem to be needed on other 542x or 5800
> > platforms.
> >
> > I suspect the "right" fix is in the bootloader someplace, but not
> > knowing this hardware well, I'm not sure if the fix is in u-boot
> > proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
> > before u-boot.  The u-boot I'm using is from the hardkernel u-boot
> > repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
> > my own u-boot from there, but only have binaries for bl1/bl2/tz.
> >
> > [1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git
> >
> >
> > Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
> > Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
> > Cc: Andrew Bresticker <abrestic@chromium.org>
> > Cc: Doug Anderson <dianders@chromium.org>
> > Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
> > Signed-off-by: Kevin Hilman <khilman@linaro.org>
> > ---
> >  arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
> >  arch/arm/mach-exynos/regs-pmu.h    | 1 +
> >  2 files changed, 3 insertions(+)
> 
> Hi,
> 
> +Cc Marek, Bartlomiej, Kukjin Kim,
> 
> 
> I would like to bring back this topic. Unfortunately I don't have
> access to source code of BL1 (or any other firmware blob) so my
> knowledge here comes mostly from experimenting and from looking at
> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
> S5, Exynos5422).
> 
> It seems that some booting firmware (I would suspect BL1 because this
> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
> For example vendor kernel, when booting little core, it waits till
> SPARE2==1 and then executes software reset for this core.
> 
> Observations shown that BL1 for Odroid, when booting secondary little core:
> 1. Expects that SPARE2 register will be initialized to 1.
> 2. If it is, then it sets it to 0, proceeds further and little core boots.
> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
> notification to userspace - reset me please!
> 
> Unfortunately executing software reset in that time (at point 3)
> stopped kernel from booting. No logs/dmesg and I was unable to turn on
> early printk.
> 
> The answer why two of little cores boot is quite simple now. At
> beginning the SPARE2==0 so first little core will set it to 1 and wait
> till software reset. Kernel timeouts on this CPU bring up so it starts
> the sequence for next little core. Now the SPARE2==1 so the core boots
> fine and SPARE2 is set to 0. The last little core starts from
> SPARE2==0, sets it to 1 and waits for software reset.
> 
> Since no one knows how this exactly works and we are stuck with BL1
> provided as is, then IMHO the patch makes sense.
> 
> Kevin, can you refresh the patch?
> It would be nice to:
> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
> 2. extend the explanation.
> 
> 
> Best regards,
> Krzysztof

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Przemyslaw Marczak]

Hello Krzysztof,

On 06/14/2015 10:56 AM, Krzysztof Kozłowski wrote:
> 2014-11-25 15:21 GMT+09:00 Kevin Hilman <khilman@kernel.org>:
>> From: Kevin Hilman <khilman@linaro.org>
>>
>> Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
>> 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
>> A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
>> and 7 do not boot:
>>
>>     [...]
>>     Exynos MCPM support installed
>>     CPU1: update cpu_capacity 1535
>>     CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
>>     CPU2: update cpu_capacity 1535
>>     CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
>>     CPU3: update cpu_capacity 1535
>>     CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
>>     CPU4: update cpu_capacity 1535
>>     CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
>>     CPU5: failed to come online
>>     CPU6: update cpu_capacity 448
>>     CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
>>     CPU7: failed to come online
>>     Brought up 6 CPUs
>>     CPU: WARNING: CPU(s) started in wrong/inconsistent modes
>>     (primary CPU mode 0x13)
>>     CPU: This may indicate a broken bootloader or firmware.
>>
>> Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
>> again, but the warning about CPUs started in inconsistent modes
>> remains.  Also, not being terribly familiar with Exynos internals,
>> it's not at all obvious to me why this register write (done for *all*
>> secondaries) makes things work works for the 2 secondary CPUs that
>> didn't come online.  It's also not obvious whether this is the right
>> general fix, since it doesn't seem to be needed on other 542x or 5800
>> platforms.
>>
>> I suspect the "right" fix is in the bootloader someplace, but not
>> knowing this hardware well, I'm not sure if the fix is in u-boot
>> proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
>> before u-boot.  The u-boot I'm using is from the hardkernel u-boot
>> repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
>> my own u-boot from there, but only have binaries for bl1/bl2/tz.
>>
>> [1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git
>>
>>
>> Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
>> Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
>> Cc: Andrew Bresticker <abrestic@chromium.org>
>> Cc: Doug Anderson <dianders@chromium.org>
>> Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
>> Signed-off-by: Kevin Hilman <khilman@linaro.org>
>> ---
>>   arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
>>   arch/arm/mach-exynos/regs-pmu.h    | 1 +
>>   2 files changed, 3 insertions(+)
>
> Hi,
>
> +Cc Marek, Bartlomiej, Kukjin Kim,
>
>
> I would like to bring back this topic. Unfortunately I don't have
> access to source code of BL1 (or any other firmware blob) so my
> knowledge here comes mostly from experimenting and from looking at
> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
> S5, Exynos5422).
>
> It seems that some booting firmware (I would suspect BL1 because this
> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
> For example vendor kernel, when booting little core, it waits till
> SPARE2==1 and then executes software reset for this core.
>
> Observations shown that BL1 for Odroid, when booting secondary little core:
> 1. Expects that SPARE2 register will be initialized to 1.
> 2. If it is, then it sets it to 0, proceeds further and little core boots.
> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
> notification to userspace - reset me please!
>
> Unfortunately executing software reset in that time (at point 3)
> stopped kernel from booting. No logs/dmesg and I was unable to turn on
> early printk.
>
> The answer why two of little cores boot is quite simple now. At
> beginning the SPARE2==0 so first little core will set it to 1 and wait
> till software reset. Kernel timeouts on this CPU bring up so it starts
> the sequence for next little core. Now the SPARE2==1 so the core boots
> fine and SPARE2 is set to 0. The last little core starts from
> SPARE2==0, sets it to 1 and waits for software reset.
>
> Since no one knows how this exactly works and we are stuck with BL1
> provided as is, then IMHO the patch makes sense.
>
> Kevin, can you refresh the patch?
> It would be nice to:
> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
> 2. extend the explanation.
>
>
> Best regards,
> Krzysztof
>

I'm trying port the hardkernel's SPL to the mainline U-Boot at present. 
The mainline SPL is implemented for E5420 and E5800. But there are few 
differences:
- different DRAM
- different clocks
- different boot core (peach-pi boots from A15)
- bl2 signature
- hdk's SPL uses smc calls
... and some more.

The BL1 keeps signature key and some part of code, but it's code is 
proprietary - but we should be able to setup the secondary cores in BL2.

When, I get the basic setup working, then I'm going to focus on the 
secondary CPU's init. I don't have the documentation for iROM code, so 
everything takes a while.

  If you looking for the lowlevel code, which is executed after wakeup, 
please check this :
https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S

The 'lowlevel_init' label is always executed on boot.

Best regards,
-- 
Przemyslaw Marczak
Samsung R&D Institute Poland
Samsung Electronics
p.marczak@samsung.com

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Przemyslaw Marczak]

Hello Krzysztof,

On 06/14/2015 10:56 AM, Krzysztof Koz?owski wrote:
> 2014-11-25 15:21 GMT+09:00 Kevin Hilman <khilman@kernel.org>:
>> From: Kevin Hilman <khilman@linaro.org>
>>
>> Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
>> 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
>> A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
>> and 7 do not boot:
>>
>>     [...]
>>     Exynos MCPM support installed
>>     CPU1: update cpu_capacity 1535
>>     CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
>>     CPU2: update cpu_capacity 1535
>>     CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
>>     CPU3: update cpu_capacity 1535
>>     CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
>>     CPU4: update cpu_capacity 1535
>>     CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
>>     CPU5: failed to come online
>>     CPU6: update cpu_capacity 448
>>     CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
>>     CPU7: failed to come online
>>     Brought up 6 CPUs
>>     CPU: WARNING: CPU(s) started in wrong/inconsistent modes
>>     (primary CPU mode 0x13)
>>     CPU: This may indicate a broken bootloader or firmware.
>>
>> Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
>> again, but the warning about CPUs started in inconsistent modes
>> remains.  Also, not being terribly familiar with Exynos internals,
>> it's not at all obvious to me why this register write (done for *all*
>> secondaries) makes things work works for the 2 secondary CPUs that
>> didn't come online.  It's also not obvious whether this is the right
>> general fix, since it doesn't seem to be needed on other 542x or 5800
>> platforms.
>>
>> I suspect the "right" fix is in the bootloader someplace, but not
>> knowing this hardware well, I'm not sure if the fix is in u-boot
>> proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
>> before u-boot.  The u-boot I'm using is from the hardkernel u-boot
>> repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
>> my own u-boot from there, but only have binaries for bl1/bl2/tz.
>>
>> [1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git
>>
>>
>> Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
>> Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
>> Cc: Andrew Bresticker <abrestic@chromium.org>
>> Cc: Doug Anderson <dianders@chromium.org>
>> Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
>> Signed-off-by: Kevin Hilman <khilman@linaro.org>
>> ---
>>   arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
>>   arch/arm/mach-exynos/regs-pmu.h    | 1 +
>>   2 files changed, 3 insertions(+)
>
> Hi,
>
> +Cc Marek, Bartlomiej, Kukjin Kim,
>
>
> I would like to bring back this topic. Unfortunately I don't have
> access to source code of BL1 (or any other firmware blob) so my
> knowledge here comes mostly from experimenting and from looking at
> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
> S5, Exynos5422).
>
> It seems that some booting firmware (I would suspect BL1 because this
> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
> For example vendor kernel, when booting little core, it waits till
> SPARE2==1 and then executes software reset for this core.
>
> Observations shown that BL1 for Odroid, when booting secondary little core:
> 1. Expects that SPARE2 register will be initialized to 1.
> 2. If it is, then it sets it to 0, proceeds further and little core boots.
> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
> notification to userspace - reset me please!
>
> Unfortunately executing software reset in that time (at point 3)
> stopped kernel from booting. No logs/dmesg and I was unable to turn on
> early printk.
>
> The answer why two of little cores boot is quite simple now. At
> beginning the SPARE2==0 so first little core will set it to 1 and wait
> till software reset. Kernel timeouts on this CPU bring up so it starts
> the sequence for next little core. Now the SPARE2==1 so the core boots
> fine and SPARE2 is set to 0. The last little core starts from
> SPARE2==0, sets it to 1 and waits for software reset.
>
> Since no one knows how this exactly works and we are stuck with BL1
> provided as is, then IMHO the patch makes sense.
>
> Kevin, can you refresh the patch?
> It would be nice to:
> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
> 2. extend the explanation.
>
>
> Best regards,
> Krzysztof
>

I'm trying port the hardkernel's SPL to the mainline U-Boot at present. 
The mainline SPL is implemented for E5420 and E5800. But there are few 
differences:
- different DRAM
- different clocks
- different boot core (peach-pi boots from A15)
- bl2 signature
- hdk's SPL uses smc calls
... and some more.

The BL1 keeps signature key and some part of code, but it's code is 
proprietary - but we should be able to setup the secondary cores in BL2.

When, I get the basic setup working, then I'm going to focus on the 
secondary CPU's init. I don't have the documentation for iROM code, so 
everything takes a while.

  If you looking for the lowlevel code, which is executed after wakeup, 
please check this :
https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S

The 'lowlevel_init' label is always executed on boot.

Best regards,
-- 
Przemyslaw Marczak
Samsung R&D Institute Poland
Samsung Electronics
p.marczak at samsung.com

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Amit Kucheria]

On Mon, Jun 15, 2015 at 3:49 PM, Przemyslaw Marczak
<p.marczak@samsung.com> wrote:
> Hello Krzysztof,
>
>
> On 06/14/2015 10:56 AM, Krzysztof Kozłowski wrote:

<snip>

> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
> mainline SPL is implemented for E5420 and E5800. But there are few
> differences:
> - different DRAM
> - different clocks
> - different boot core (peach-pi boots from A15)
> - bl2 signature
> - hdk's SPL uses smc calls
> ... and some more.

This is really good news! Would this work leave CCI control to Linux
so that we may use MCPM to manage cpu and cluster OFF?

> The BL1 keeps signature key and some part of code, but it's code is
> proprietary - but we should be able to setup the secondary cores in BL2.
>
> When, I get the basic setup working, then I'm going to focus on the
> secondary CPU's init. I don't have the documentation for iROM code, so
> everything takes a while.
>
>  If you looking for the lowlevel code, which is executed after wakeup,
> please check this :
> https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S
>
> The 'lowlevel_init' label is always executed on boot.

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Amit Kucheria]

On Mon, Jun 15, 2015 at 3:49 PM, Przemyslaw Marczak
<p.marczak@samsung.com> wrote:
> Hello Krzysztof,
>
>
> On 06/14/2015 10:56 AM, Krzysztof Koz?owski wrote:

<snip>

> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
> mainline SPL is implemented for E5420 and E5800. But there are few
> differences:
> - different DRAM
> - different clocks
> - different boot core (peach-pi boots from A15)
> - bl2 signature
> - hdk's SPL uses smc calls
> ... and some more.

This is really good news! Would this work leave CCI control to Linux
so that we may use MCPM to manage cpu and cluster OFF?

> The BL1 keeps signature key and some part of code, but it's code is
> proprietary - but we should be able to setup the secondary cores in BL2.
>
> When, I get the basic setup working, then I'm going to focus on the
> secondary CPU's init. I don't have the documentation for iROM code, so
> everything takes a while.
>
>  If you looking for the lowlevel code, which is executed after wakeup,
> please check this :
> https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S
>
> The 'lowlevel_init' label is always executed on boot.

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Krzysztof Kozłowski]

2015-06-15 19:19 GMT+09:00 Przemyslaw Marczak <p.marczak@samsung.com>:
> Hello Krzysztof,
>
>
> On 06/14/2015 10:56 AM, Krzysztof Kozłowski wrote:
>>
>>
>> Hi,
>>
>> +Cc Marek, Bartlomiej, Kukjin Kim,
>>
>>
>> I would like to bring back this topic. Unfortunately I don't have
>> access to source code of BL1 (or any other firmware blob) so my
>> knowledge here comes mostly from experimenting and from looking at
>> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
>> S5, Exynos5422).
>>
>> It seems that some booting firmware (I would suspect BL1 because this
>> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
>> For example vendor kernel, when booting little core, it waits till
>> SPARE2==1 and then executes software reset for this core.
>>
>> Observations shown that BL1 for Odroid, when booting secondary little
>> core:
>> 1. Expects that SPARE2 register will be initialized to 1.
>> 2. If it is, then it sets it to 0, proceeds further and little core boots.
>> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
>> notification to userspace - reset me please!
>>
>> Unfortunately executing software reset in that time (at point 3)
>> stopped kernel from booting. No logs/dmesg and I was unable to turn on
>> early printk.
>>
>> The answer why two of little cores boot is quite simple now. At
>> beginning the SPARE2==0 so first little core will set it to 1 and wait
>> till software reset. Kernel timeouts on this CPU bring up so it starts
>> the sequence for next little core. Now the SPARE2==1 so the core boots
>> fine and SPARE2 is set to 0. The last little core starts from
>> SPARE2==0, sets it to 1 and waits for software reset.
>>
>> Since no one knows how this exactly works and we are stuck with BL1
>> provided as is, then IMHO the patch makes sense.
>>
>> Kevin, can you refresh the patch?
>> It would be nice to:
>> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
>> 2. extend the explanation.
>>
>>
>> Best regards,
>> Krzysztof
>>
>
> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
> mainline SPL is implemented for E5420 and E5800. But there are few
> differences:
> - different DRAM
> - different clocks
> - different boot core (peach-pi boots from A15)
> - bl2 signature
> - hdk's SPL uses smc calls
> ... and some more.
>
> The BL1 keeps signature key and some part of code, but it's code is
> proprietary - but we should be able to setup the secondary cores in BL2.
>
> When, I get the basic setup working, then I'm going to focus on the
> secondary CPU's init. I don't have the documentation for iROM code, so
> everything takes a while.

Great, good luck!

>
>  If you looking for the lowlevel code, which is executed after wakeup,
> please check this :
> https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S
>
> The 'lowlevel_init' label is always executed on boot.

I already looked at it without any success. I couldn't find the reason
of this SPARE2 behaviour in that code. However I found there one small
funny fact about magic values for low power modes - kernel and u-boot
expect some of the values in different places.

Best regards,
Krzysztof

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Krzysztof Kozłowski]

2015-06-15 19:19 GMT+09:00 Przemyslaw Marczak <p.marczak@samsung.com>:
> Hello Krzysztof,
>
>
> On 06/14/2015 10:56 AM, Krzysztof Koz?owski wrote:
>>
>>
>> Hi,
>>
>> +Cc Marek, Bartlomiej, Kukjin Kim,
>>
>>
>> I would like to bring back this topic. Unfortunately I don't have
>> access to source code of BL1 (or any other firmware blob) so my
>> knowledge here comes mostly from experimenting and from looking at
>> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
>> S5, Exynos5422).
>>
>> It seems that some booting firmware (I would suspect BL1 because this
>> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
>> For example vendor kernel, when booting little core, it waits till
>> SPARE2==1 and then executes software reset for this core.
>>
>> Observations shown that BL1 for Odroid, when booting secondary little
>> core:
>> 1. Expects that SPARE2 register will be initialized to 1.
>> 2. If it is, then it sets it to 0, proceeds further and little core boots.
>> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
>> notification to userspace - reset me please!
>>
>> Unfortunately executing software reset in that time (at point 3)
>> stopped kernel from booting. No logs/dmesg and I was unable to turn on
>> early printk.
>>
>> The answer why two of little cores boot is quite simple now. At
>> beginning the SPARE2==0 so first little core will set it to 1 and wait
>> till software reset. Kernel timeouts on this CPU bring up so it starts
>> the sequence for next little core. Now the SPARE2==1 so the core boots
>> fine and SPARE2 is set to 0. The last little core starts from
>> SPARE2==0, sets it to 1 and waits for software reset.
>>
>> Since no one knows how this exactly works and we are stuck with BL1
>> provided as is, then IMHO the patch makes sense.
>>
>> Kevin, can you refresh the patch?
>> It would be nice to:
>> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
>> 2. extend the explanation.
>>
>>
>> Best regards,
>> Krzysztof
>>
>
> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
> mainline SPL is implemented for E5420 and E5800. But there are few
> differences:
> - different DRAM
> - different clocks
> - different boot core (peach-pi boots from A15)
> - bl2 signature
> - hdk's SPL uses smc calls
> ... and some more.
>
> The BL1 keeps signature key and some part of code, but it's code is
> proprietary - but we should be able to setup the secondary cores in BL2.
>
> When, I get the basic setup working, then I'm going to focus on the
> secondary CPU's init. I don't have the documentation for iROM code, so
> everything takes a while.

Great, good luck!

>
>  If you looking for the lowlevel code, which is executed after wakeup,
> please check this :
> https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S
>
> The 'lowlevel_init' label is always executed on boot.

I already looked at it without any success. I couldn't find the reason
of this SPARE2 behaviour in that code. However I found there one small
funny fact about magic values for low power modes - kernel and u-boot
expect some of the values in different places.

Best regards,
Krzysztof

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Przemyslaw Marczak]

On 06/15/2015 01:19 PM, Amit Kucheria wrote:
> On Mon, Jun 15, 2015 at 3:49 PM, Przemyslaw Marczak
> <p.marczak@samsung.com> wrote:
>> Hello Krzysztof,
>>
>>
>> On 06/14/2015 10:56 AM, Krzysztof Kozłowski wrote:
>
> <snip>
>
>> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
>> mainline SPL is implemented for E5420 and E5800. But there are few
>> differences:
>> - different DRAM
>> - different clocks
>> - different boot core (peach-pi boots from A15)
>> - bl2 signature
>> - hdk's SPL uses smc calls
>> ... and some more.
>
> This is really good news! Would this work leave CCI control to Linux
> so that we may use MCPM to manage cpu and cluster OFF?
>

Yes, I would like to get this stuff working.

>> The BL1 keeps signature key and some part of code, but it's code is
>> proprietary - but we should be able to setup the secondary cores in BL2.
>>
>> When, I get the basic setup working, then I'm going to focus on the
>> secondary CPU's init. I don't have the documentation for iROM code, so
>> everything takes a while.
>>
>>   If you looking for the lowlevel code, which is executed after wakeup,
>> please check this :
>> https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S
>>
>> The 'lowlevel_init' label is always executed on boot.
>

Best regards,
-- 
Przemyslaw Marczak
Samsung R&D Institute Poland
Samsung Electronics
p.marczak@samsung.com

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Przemyslaw Marczak]

On 06/15/2015 01:19 PM, Amit Kucheria wrote:
> On Mon, Jun 15, 2015 at 3:49 PM, Przemyslaw Marczak
> <p.marczak@samsung.com> wrote:
>> Hello Krzysztof,
>>
>>
>> On 06/14/2015 10:56 AM, Krzysztof Koz?owski wrote:
>
> <snip>
>
>> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
>> mainline SPL is implemented for E5420 and E5800. But there are few
>> differences:
>> - different DRAM
>> - different clocks
>> - different boot core (peach-pi boots from A15)
>> - bl2 signature
>> - hdk's SPL uses smc calls
>> ... and some more.
>
> This is really good news! Would this work leave CCI control to Linux
> so that we may use MCPM to manage cpu and cluster OFF?
>

Yes, I would like to get this stuff working.

>> The BL1 keeps signature key and some part of code, but it's code is
>> proprietary - but we should be able to setup the secondary cores in BL2.
>>
>> When, I get the basic setup working, then I'm going to focus on the
>> secondary CPU's init. I don't have the documentation for iROM code, so
>> everything takes a while.
>>
>>   If you looking for the lowlevel code, which is executed after wakeup,
>> please check this :
>> https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S
>>
>> The 'lowlevel_init' label is always executed on boot.
>

Best regards,
-- 
Przemyslaw Marczak
Samsung R&D Institute Poland
Samsung Electronics
p.marczak at samsung.com

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Przemyslaw Marczak]

On 06/15/2015 02:17 PM, Krzysztof Kozłowski wrote:
> 2015-06-15 19:19 GMT+09:00 Przemyslaw Marczak <p.marczak@samsung.com>:
>> Hello Krzysztof,
>>
>>
>> On 06/14/2015 10:56 AM, Krzysztof Kozłowski wrote:
>>>
>>>
>>> Hi,
>>>
>>> +Cc Marek, Bartlomiej, Kukjin Kim,
>>>
>>>
>>> I would like to bring back this topic. Unfortunately I don't have
>>> access to source code of BL1 (or any other firmware blob) so my
>>> knowledge here comes mostly from experimenting and from looking at
>>> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
>>> S5, Exynos5422).
>>>
>>> It seems that some booting firmware (I would suspect BL1 because this
>>> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
>>> For example vendor kernel, when booting little core, it waits till
>>> SPARE2==1 and then executes software reset for this core.
>>>
>>> Observations shown that BL1 for Odroid, when booting secondary little
>>> core:
>>> 1. Expects that SPARE2 register will be initialized to 1.
>>> 2. If it is, then it sets it to 0, proceeds further and little core boots.
>>> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
>>> notification to userspace - reset me please!
>>>
>>> Unfortunately executing software reset in that time (at point 3)
>>> stopped kernel from booting. No logs/dmesg and I was unable to turn on
>>> early printk.
>>>
>>> The answer why two of little cores boot is quite simple now. At
>>> beginning the SPARE2==0 so first little core will set it to 1 and wait
>>> till software reset. Kernel timeouts on this CPU bring up so it starts
>>> the sequence for next little core. Now the SPARE2==1 so the core boots
>>> fine and SPARE2 is set to 0. The last little core starts from
>>> SPARE2==0, sets it to 1 and waits for software reset.
>>>
>>> Since no one knows how this exactly works and we are stuck with BL1
>>> provided as is, then IMHO the patch makes sense.
>>>
>>> Kevin, can you refresh the patch?
>>> It would be nice to:
>>> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
>>> 2. extend the explanation.
>>>
>>>
>>> Best regards,
>>> Krzysztof
>>>
>>
>> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
>> mainline SPL is implemented for E5420 and E5800. But there are few
>> differences:
>> - different DRAM
>> - different clocks
>> - different boot core (peach-pi boots from A15)
>> - bl2 signature
>> - hdk's SPL uses smc calls
>> ... and some more.
>>
>> The BL1 keeps signature key and some part of code, but it's code is
>> proprietary - but we should be able to setup the secondary cores in BL2.
>>
>> When, I get the basic setup working, then I'm going to focus on the
>> secondary CPU's init. I don't have the documentation for iROM code, so
>> everything takes a while.
>
> Great, good luck!
>

Thanks!

>>
>>   If you looking for the lowlevel code, which is executed after wakeup,
>> please check this :
>> https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S
>>
>> The 'lowlevel_init' label is always executed on boot.
>
> I already looked at it without any success. I couldn't find the reason
> of this SPARE2 behaviour in that code. However I found there one small
> funny fact about magic values for low power modes - kernel and u-boot
> expect some of the values in different places.
>
> Best regards,
> Krzysztof
>

I checked the SPARE registers, and only SPARE0 is set by iROM code to 
0xFCBA0D10, it is defined in the kernel as: S5P_CHECK_AFTR. I didn't 
analyzed the kernel code yet, however it looks, that it doesn't use 
SPARE0 register.

I need finish the basic SPL support, and then will start synchronize 
with the kernel.

Best regards,
-- 
Przemyslaw Marczak
Samsung R&D Institute Poland
Samsung Electronics
p.marczak@samsung.com

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Przemyslaw Marczak]

On 06/15/2015 02:17 PM, Krzysztof Koz?owski wrote:
> 2015-06-15 19:19 GMT+09:00 Przemyslaw Marczak <p.marczak@samsung.com>:
>> Hello Krzysztof,
>>
>>
>> On 06/14/2015 10:56 AM, Krzysztof Koz?owski wrote:
>>>
>>>
>>> Hi,
>>>
>>> +Cc Marek, Bartlomiej, Kukjin Kim,
>>>
>>>
>>> I would like to bring back this topic. Unfortunately I don't have
>>> access to source code of BL1 (or any other firmware blob) so my
>>> knowledge here comes mostly from experimenting and from looking at
>>> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
>>> S5, Exynos5422).
>>>
>>> It seems that some booting firmware (I would suspect BL1 because this
>>> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
>>> For example vendor kernel, when booting little core, it waits till
>>> SPARE2==1 and then executes software reset for this core.
>>>
>>> Observations shown that BL1 for Odroid, when booting secondary little
>>> core:
>>> 1. Expects that SPARE2 register will be initialized to 1.
>>> 2. If it is, then it sets it to 0, proceeds further and little core boots.
>>> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
>>> notification to userspace - reset me please!
>>>
>>> Unfortunately executing software reset in that time (at point 3)
>>> stopped kernel from booting. No logs/dmesg and I was unable to turn on
>>> early printk.
>>>
>>> The answer why two of little cores boot is quite simple now. At
>>> beginning the SPARE2==0 so first little core will set it to 1 and wait
>>> till software reset. Kernel timeouts on this CPU bring up so it starts
>>> the sequence for next little core. Now the SPARE2==1 so the core boots
>>> fine and SPARE2 is set to 0. The last little core starts from
>>> SPARE2==0, sets it to 1 and waits for software reset.
>>>
>>> Since no one knows how this exactly works and we are stuck with BL1
>>> provided as is, then IMHO the patch makes sense.
>>>
>>> Kevin, can you refresh the patch?
>>> It would be nice to:
>>> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
>>> 2. extend the explanation.
>>>
>>>
>>> Best regards,
>>> Krzysztof
>>>
>>
>> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
>> mainline SPL is implemented for E5420 and E5800. But there are few
>> differences:
>> - different DRAM
>> - different clocks
>> - different boot core (peach-pi boots from A15)
>> - bl2 signature
>> - hdk's SPL uses smc calls
>> ... and some more.
>>
>> The BL1 keeps signature key and some part of code, but it's code is
>> proprietary - but we should be able to setup the secondary cores in BL2.
>>
>> When, I get the basic setup working, then I'm going to focus on the
>> secondary CPU's init. I don't have the documentation for iROM code, so
>> everything takes a while.
>
> Great, good luck!
>

Thanks!

>>
>>   If you looking for the lowlevel code, which is executed after wakeup,
>> please check this :
>> https://github.com/hardkernel/u-boot/blob/odroidxu3-v2012.07/board/samsung/smdk5422/lowlevel_init.S
>>
>> The 'lowlevel_init' label is always executed on boot.
>
> I already looked at it without any success. I couldn't find the reason
> of this SPARE2 behaviour in that code. However I found there one small
> funny fact about magic values for low power modes - kernel and u-boot
> expect some of the values in different places.
>
> Best regards,
> Krzysztof
>

I checked the SPARE registers, and only SPARE0 is set by iROM code to 
0xFCBA0D10, it is defined in the kernel as: S5P_CHECK_AFTR. I didn't 
analyzed the kernel code yet, however it looks, that it doesn't use 
SPARE0 register.

I need finish the basic SPL support, and then will start synchronize 
with the kernel.

Best regards,
-- 
Przemyslaw Marczak
Samsung R&D Institute Poland
Samsung Electronics
p.marczak at samsung.com

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Kevin Hilman]

Krzysztof Kozłowski <k.kozlowski.k@gmail.com> writes:

> 2014-11-25 15:21 GMT+09:00 Kevin Hilman <khilman@kernel.org>:
>> From: Kevin Hilman <khilman@linaro.org>
>>
>> Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
>> 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
>> A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
>> and 7 do not boot:
>>
>>    [...]
>>    Exynos MCPM support installed
>>    CPU1: update cpu_capacity 1535
>>    CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
>>    CPU2: update cpu_capacity 1535
>>    CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
>>    CPU3: update cpu_capacity 1535
>>    CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
>>    CPU4: update cpu_capacity 1535
>>    CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
>>    CPU5: failed to come online
>>    CPU6: update cpu_capacity 448
>>    CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
>>    CPU7: failed to come online
>>    Brought up 6 CPUs
>>    CPU: WARNING: CPU(s) started in wrong/inconsistent modes
>>    (primary CPU mode 0x13)
>>    CPU: This may indicate a broken bootloader or firmware.
>>
>> Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
>> again, but the warning about CPUs started in inconsistent modes
>> remains.  Also, not being terribly familiar with Exynos internals,
>> it's not at all obvious to me why this register write (done for *all*
>> secondaries) makes things work works for the 2 secondary CPUs that
>> didn't come online.  It's also not obvious whether this is the right
>> general fix, since it doesn't seem to be needed on other 542x or 5800
>> platforms.
>>
>> I suspect the "right" fix is in the bootloader someplace, but not
>> knowing this hardware well, I'm not sure if the fix is in u-boot
>> proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
>> before u-boot.  The u-boot I'm using is from the hardkernel u-boot
>> repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
>> my own u-boot from there, but only have binaries for bl1/bl2/tz.
>>
>> [1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git
>>
>>
>> Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
>> Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
>> Cc: Andrew Bresticker <abrestic@chromium.org>
>> Cc: Doug Anderson <dianders@chromium.org>
>> Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
>> Signed-off-by: Kevin Hilman <khilman@linaro.org>
>> ---
>>  arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
>>  arch/arm/mach-exynos/regs-pmu.h    | 1 +
>>  2 files changed, 3 insertions(+)
>
> Hi,
>
> +Cc Marek, Bartlomiej, Kukjin Kim,
>
>
> I would like to bring back this topic. Unfortunately I don't have
> access to source code of BL1 (or any other firmware blob) so my
> knowledge here comes mostly from experimenting and from looking at
> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
> S5, Exynos5422).
>
> It seems that some booting firmware (I would suspect BL1 because this
> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
> For example vendor kernel, when booting little core, it waits till
> SPARE2==1 and then executes software reset for this core.
>
> Observations shown that BL1 for Odroid, when booting secondary little core:
> 1. Expects that SPARE2 register will be initialized to 1.
> 2. If it is, then it sets it to 0, proceeds further and little core boots.
> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
> notification to userspace - reset me please!
>
> Unfortunately executing software reset in that time (at point 3)
> stopped kernel from booting. No logs/dmesg and I was unable to turn on
> early printk.
>
> The answer why two of little cores boot is quite simple now. At
> beginning the SPARE2==0 so first little core will set it to 1 and wait
> till software reset. Kernel timeouts on this CPU bring up so it starts
> the sequence for next little core. Now the SPARE2==1 so the core boots
> fine and SPARE2 is set to 0. The last little core starts from
> SPARE2==0, sets it to 1 and waits for software reset.
>
> Since no one knows how this exactly works and we are stuck with BL1
> provided as is, then IMHO the patch makes sense.
>
> Kevin, can you refresh the patch?
> It would be nice to:
> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
> 2. extend the explanation.

I'd rather someone else refresh this patch who actually understands
what's going on here and could write a descriptive changelog.  I have no
claims to the authorship as Abhilash is the one who suggested this fix
in the first place.

Also, please note that 8 cores booting doesn't mean all 8 cores fully
working.  This firmware is still horribly broken for low-power modes.

Even with all 8 cores enabled, the firmware also as CCI left in secure
mode which means the kernel MCPM cannot manage CCI, and thus cannot hit
any low-power states.  If CPUidle is enabled, the kernel will hang as
soon as any MCPM state is attempted.  In order to get WFI-only CPUidle
working, I've also had to disable CCI in the DTS by appending something
like this to the board DTS file:

&cci {
     status = "disabled";
};

All of this though makes me wonder how the hardkernel kernel actually
does any low-power idle.  Is it relying on firmware for this?  I have a
hard time believe that the firmware would be able to handle this in a
race-free way.

Kevin

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Kevin Hilman]

Krzysztof Koz?owski <k.kozlowski.k@gmail.com> writes:

> 2014-11-25 15:21 GMT+09:00 Kevin Hilman <khilman@kernel.org>:
>> From: Kevin Hilman <khilman@linaro.org>
>>
>> Using the current exynos_defconfig on the exynos5422-odroid-xu3, only
>> 6 of 8 CPUs come online with MCPM boot.  CPU0 is an A7, CPUs 1-4 are
>> A15s and CPU5-7 are the other A7s, but with the current code, CPUs 5
>> and 7 do not boot:
>>
>>    [...]
>>    Exynos MCPM support installed
>>    CPU1: update cpu_capacity 1535
>>    CPU1: thread -1, cpu 0, socket 0, mpidr 80000000
>>    CPU2: update cpu_capacity 1535
>>    CPU2: thread -1, cpu 1, socket 0, mpidr 80000001
>>    CPU3: update cpu_capacity 1535
>>    CPU3: thread -1, cpu 2, socket 0, mpidr 80000002
>>    CPU4: update cpu_capacity 1535
>>    CPU4: thread -1, cpu 3, socket 0, mpidr 80000003
>>    CPU5: failed to come online
>>    CPU6: update cpu_capacity 448
>>    CPU6: thread -1, cpu 2, socket 1, mpidr 80000102
>>    CPU7: failed to come online
>>    Brought up 6 CPUs
>>    CPU: WARNING: CPU(s) started in wrong/inconsistent modes
>>    (primary CPU mode 0x13)
>>    CPU: This may indicate a broken bootloader or firmware.
>>
>> Thanks to a tip from Abhilash, this patch gets all 8 CPUs booting
>> again, but the warning about CPUs started in inconsistent modes
>> remains.  Also, not being terribly familiar with Exynos internals,
>> it's not at all obvious to me why this register write (done for *all*
>> secondaries) makes things work works for the 2 secondary CPUs that
>> didn't come online.  It's also not obvious whether this is the right
>> general fix, since it doesn't seem to be needed on other 542x or 5800
>> platforms.
>>
>> I suspect the "right" fix is in the bootloader someplace, but not
>> knowing this hardware well, I'm not sure if the fix is in u-boot
>> proper, or somewhere in the binary blobs (bl1/bl2/tz) that start
>> before u-boot.  The u-boot I'm using is from the hardkernel u-boot
>> repo[1], and I'd welcome any suggestions to try.  I'm able to rebuild
>> my own u-boot from there, but only have binaries for bl1/bl2/tz.
>>
>> [1] branch "odroidxu3-v2012.07" of: https://github.com/hardkernel/u-boot.git
>>
>>
>> Cc: Mauro Ribeiro <mauro.ribeiro@hardkernel.com>
>> Cc: Abhilash Kesavan <a.kesavan@samsung.com>,
>> Cc: Andrew Bresticker <abrestic@chromium.org>
>> Cc: Doug Anderson <dianders@chromium.org>
>> Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
>> Signed-off-by: Kevin Hilman <khilman@linaro.org>
>> ---
>>  arch/arm/mach-exynos/mcpm-exynos.c | 2 ++
>>  arch/arm/mach-exynos/regs-pmu.h    | 1 +
>>  2 files changed, 3 insertions(+)
>
> Hi,
>
> +Cc Marek, Bartlomiej, Kukjin Kim,
>
>
> I would like to bring back this topic. Unfortunately I don't have
> access to source code of BL1 (or any other firmware blob) so my
> knowledge here comes mostly from experimenting and from looking at
> sources of vendor kernel for Gear 2 (Exynos3250) and SM-G900H (Galaxy
> S5, Exynos5422).
>
> It seems that some booting firmware (I would suspect BL1 because this
> ships Samsung to Hardkernel) uses SPARE2 as synchronization mechanism.
> For example vendor kernel, when booting little core, it waits till
> SPARE2==1 and then executes software reset for this core.
>
> Observations shown that BL1 for Odroid, when booting secondary little core:
> 1. Expects that SPARE2 register will be initialized to 1.
> 2. If it is, then it sets it to 0, proceeds further and little core boots.
> 3. If it is not, then it sets it to 1 and waits. Maybe this is a
> notification to userspace - reset me please!
>
> Unfortunately executing software reset in that time (at point 3)
> stopped kernel from booting. No logs/dmesg and I was unable to turn on
> early printk.
>
> The answer why two of little cores boot is quite simple now. At
> beginning the SPARE2==0 so first little core will set it to 1 and wait
> till software reset. Kernel timeouts on this CPU bring up so it starts
> the sequence for next little core. Now the SPARE2==1 so the core boots
> fine and SPARE2 is set to 0. The last little core starts from
> SPARE2==0, sets it to 1 and waits for software reset.
>
> Since no one knows how this exactly works and we are stuck with BL1
> provided as is, then IMHO the patch makes sense.
>
> Kevin, can you refresh the patch?
> It would be nice to:
> 1. set SPARE2 only for Odroid (of_machine_is_compatible()),
> 2. extend the explanation.

I'd rather someone else refresh this patch who actually understands
what's going on here and could write a descriptive changelog.  I have no
claims to the authorship as Abhilash is the one who suggested this fix
in the first place.

Also, please note that 8 cores booting doesn't mean all 8 cores fully
working.  This firmware is still horribly broken for low-power modes.

Even with all 8 cores enabled, the firmware also as CCI left in secure
mode which means the kernel MCPM cannot manage CCI, and thus cannot hit
any low-power states.  If CPUidle is enabled, the kernel will hang as
soon as any MCPM state is attempted.  In order to get WFI-only CPUidle
working, I've also had to disable CCI in the DTS by appending something
like this to the board DTS file:

&cci {
     status = "disabled";
};

All of this though makes me wonder how the hardkernel kernel actually
does any low-power idle.  Is it relying on firmware for this?  I have a
hard time believe that the firmware would be able to handle this in a
race-free way.

Kevin

Re: [RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Kevin Hilman]

Przemyslaw Marczak <p.marczak@samsung.com> writes:

> On 06/15/2015 01:19 PM, Amit Kucheria wrote:
>> On Mon, Jun 15, 2015 at 3:49 PM, Przemyslaw Marczak
>> <p.marczak@samsung.com> wrote:
>>> Hello Krzysztof,
>>>
>>>
>>> On 06/14/2015 10:56 AM, Krzysztof Kozłowski wrote:
>>
>> <snip>
>>
>>> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
>>> mainline SPL is implemented for E5420 and E5800. But there are few
>>> differences:
>>> - different DRAM
>>> - different clocks
>>> - different boot core (peach-pi boots from A15)
>>> - bl2 signature
>>> - hdk's SPL uses smc calls
>>> ... and some more.
>>
>> This is really good news! Would this work leave CCI control to Linux
>> so that we may use MCPM to manage cpu and cluster OFF?
>>
>
> Yes, I would like to get this stuff working.
>

Do you have access to BL1 sources to change this?

IIUC, what is happening is BL1 is leaving CCI in secure mode, which
means the kernel MCPM cannot manage it.  That means the kernel cannot
manage any low-power CPU or cluster states.

Does anyone know at which stage of the boot secure world is left for
normal world?  in BL1?  in BL2?

If it's in BL2, maybe the CCI issue can also be fixed by ensuring that
CCI is not left in secure mode so the kernel can properly manage CCI.

Kevin

[RFC] ARM: exynos: MCPM: [is this a] fix for secondary boot on 5422?

[Kevin Hilman]

Przemyslaw Marczak <p.marczak@samsung.com> writes:

> On 06/15/2015 01:19 PM, Amit Kucheria wrote:
>> On Mon, Jun 15, 2015 at 3:49 PM, Przemyslaw Marczak
>> <p.marczak@samsung.com> wrote:
>>> Hello Krzysztof,
>>>
>>>
>>> On 06/14/2015 10:56 AM, Krzysztof Koz?owski wrote:
>>
>> <snip>
>>
>>> I'm trying port the hardkernel's SPL to the mainline U-Boot at present. The
>>> mainline SPL is implemented for E5420 and E5800. But there are few
>>> differences:
>>> - different DRAM
>>> - different clocks
>>> - different boot core (peach-pi boots from A15)
>>> - bl2 signature
>>> - hdk's SPL uses smc calls
>>> ... and some more.
>>
>> This is really good news! Would this work leave CCI control to Linux
>> so that we may use MCPM to manage cpu and cluster OFF?
>>
>
> Yes, I would like to get this stuff working.
>

Do you have access to BL1 sources to change this?

IIUC, what is happening is BL1 is leaving CCI in secure mode, which
means the kernel MCPM cannot manage it.  That means the kernel cannot
manage any low-power CPU or cluster states.

Does anyone know at which stage of the boot secure world is left for
normal world?  in BL1?  in BL2?

If it's in BL2, maybe the CCI issue can also be fixed by ensuring that
CCI is not left in secure mode so the kernel can properly manage CCI.

Kevin