Can cores have different CPU features in a SMP system?

Can cores have different CPU features in a SMP system?

[Jun Sun]

One specific example is that if a phone has two clusters.  Cores in
one cluster has v8 crypto extension and cores in the other don't.
Will this work?

My gut feeling is no, since many apps query cpu features at run-time.
And process can migrate among cores.  If an app starts on a core with
CE and migrates to another core without CE.  Illegal instruction fault
will happen.

Just wanna confirm if this is indeed the case.  Surprisingly google
search did not yield any meaningful answers.

Cheers.

Jun

Can cores have different CPU features in a SMP system?

[Jun Sun]

I looked at the kernel source code again.  Kernel currently only
detects CPU features of CPU0 and use it for all other CPUs.   In other
words, it is assuming all cores must have the same CPU features, which
in a sense is a form of "enforcing".

As a result, it is definitely no-no to have cores with different CPU
features, which is what I suspected.  Case closed. :)

Jun

On Fri, Jul 29, 2016 at 2:30 PM, Jun Sun <jsun@junsun.net> wrote:
> One specific example is that if a phone has two clusters.  Cores in
> one cluster has v8 crypto extension and cores in the other don't.
> Will this work?
>
> My gut feeling is no, since many apps query cpu features at run-time.
> And process can migrate among cores.  If an app starts on a core with
> CE and migrates to another core without CE.  Illegal instruction fault
> will happen.
>
> Just wanna confirm if this is indeed the case.  Surprisingly google
> search did not yield any meaningful answers.
>
> Cheers.
>
> Jun

Can cores have different CPU features in a SMP system?

[Suzuki K Poulose]

On 01/08/16 20:08, Jun Sun wrote:
> I looked at the kernel source code again.  Kernel currently only
> detects CPU features of CPU0 and use it for all other CPUs.   In other
> words, it is assuming all cores must have the same CPU features, which
> in a sense is a form of "enforcing".

On arm64 kernels, the CPU features are detected based on their availability
on all the CPUs. Hence, for e.g, if one cluster doesn't have crypto, kernel
doesn't expose the ELF AUX_HWCAP  crypto bits to the application (which should be
consulted by the application before doing a CPU Feature specific operation).

>
> As a result, it is definitely no-no to have cores with different CPU
> features, which is what I suspected.  Case closed. :)

It is not recommended to have cores with different CPU features, but as
mentioned above, arm64 kernel can cope up with the situation and disallow
any features that aren't available across all the CPUs.
See arch/arm64/kernel/cpufeature.c for more information.

Cheers
Suzuki



>
> Jun
>
> On Fri, Jul 29, 2016 at 2:30 PM, Jun Sun <jsun@junsun.net> wrote:
>> One specific example is that if a phone has two clusters.  Cores in
>> one cluster has v8 crypto extension and cores in the other don't.
>> Will this work?
>>
>> My gut feeling is no, since many apps query cpu features at run-time.
>> And process can migrate among cores.  If an app starts on a core with
>> CE and migrates to another core without CE.  Illegal instruction fault
>> will happen.
>>
>> Just wanna confirm if this is indeed the case.  Surprisingly google
>> search did not yield any meaningful answers.
>>
>> Cheers.
>>
>> Jun
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>

Can cores have different CPU features in a SMP system?

[Jun Sun]

Hi, Suzuki,

I was looking at kernel 3.18 and drew the earlier conclusion.

I'm now looking at kernel v4.7-rc4.  It appears the logic is

  - cpu 0 sets up elf_hwcap (init_cpu_ftr_reg(), init_cpu_features() )
  - other cpus call verify_local_cpu_capabilities() in
secondary_start_kernel() and parks itself if it finds any missing
features.

Is this what you see? It seems to be slightly different from your description.

BTW, this is exactly the kind of enforcement I was hoping to see.  I
couldn't be happier now. :) Thanks for your pointer.

Jun

On Tue, Aug 2, 2016 at 3:09 AM, Suzuki K Poulose <Suzuki.Poulose@arm.com> wrote:
> On 01/08/16 20:08, Jun Sun wrote:
>>
>> I looked at the kernel source code again.  Kernel currently only
>> detects CPU features of CPU0 and use it for all other CPUs.   In other
>> words, it is assuming all cores must have the same CPU features, which
>> in a sense is a form of "enforcing".
>
>
> On arm64 kernels, the CPU features are detected based on their availability
> on all the CPUs. Hence, for e.g, if one cluster doesn't have crypto, kernel
> doesn't expose the ELF AUX_HWCAP  crypto bits to the application (which
> should be
> consulted by the application before doing a CPU Feature specific operation).
>
>>
>> As a result, it is definitely no-no to have cores with different CPU
>> features, which is what I suspected.  Case closed. :)
>
>
> It is not recommended to have cores with different CPU features, but as
> mentioned above, arm64 kernel can cope up with the situation and disallow
> any features that aren't available across all the CPUs.
> See arch/arm64/kernel/cpufeature.c for more information.
>
> Cheers
> Suzuki
>
>
>
>>
>> Jun
>>
>> On Fri, Jul 29, 2016 at 2:30 PM, Jun Sun <jsun@junsun.net> wrote:
>>>
>>> One specific example is that if a phone has two clusters.  Cores in
>>> one cluster has v8 crypto extension and cores in the other don't.
>>> Will this work?
>>>
>>> My gut feeling is no, since many apps query cpu features at run-time.
>>> And process can migrate among cores.  If an app starts on a core with
>>> CE and migrates to another core without CE.  Illegal instruction fault
>>> will happen.
>>>
>>> Just wanna confirm if this is indeed the case.  Surprisingly google
>>> search did not yield any meaningful answers.
>>>
>>> Cheers.
>>>
>>> Jun
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
>

Can cores have different CPU features in a SMP system?

[Suzuki K Poulose]

On 02/08/16 18:23, Jun Sun wrote:
> Hi, Suzuki,
>
> I was looking at kernel 3.18 and drew the earlier conclusion.
>
> I'm now looking at kernel v4.7-rc4.  It appears the logic is
>
>   - cpu 0 sets up elf_hwcap (init_cpu_ftr_reg(), init_cpu_features() )

On arm64 CPU 0 doesn't set up elf_hwcap. It initialises the CPU feature register
infrastructure which keeps track of the system wide safe values for the ID
registers. init_cpu_features() initialises the table with the values from
the boot CPU. This will also be used to detect if any secondary CPU has
variations in features which could affect the normal functioning of Linux (via
update_cpu_features() -> check_update_ftr_reg() ).

The elf_hwcap and the system CPU_HWCAP bits are initialised only after all
the boot time active CPUs are online, from
setup_cpu_features()-> { setup_elf_hwcaps(), setup_feature_capabilities() }


>   - other cpus call verify_local_cpu_capabilities() in

All secondary CPUs call verify_local_cpu_capabilities(). But only the CPUs
which are brought up later by the user, perform capability, hwcap verification
and parks them if they miss something.


> secondary_start_kernel() and parks itself if it finds any missing
> features.
>
> Is this what you see? It seems to be slightly different from your description.

See above

>
> BTW, this is exactly the kind of enforcement I was hoping to see.  I
> couldn't be happier now. :) Thanks for your pointer.
>

Suzuki

Can cores have different CPU features in a SMP system?

[Suzuki K Poulose]

On 03/08/16 10:00, Suzuki K Poulose wrote:
> On 02/08/16 18:23, Jun Sun wrote:
>> Hi, Suzuki,
>>
>> I was looking at kernel 3.18 and drew the earlier conclusion.
>>
>> I'm now looking at kernel v4.7-rc4.  It appears the logic is
>>
>>   - cpu 0 sets up elf_hwcap (init_cpu_ftr_reg(), init_cpu_features() )
>
> On arm64 CPU 0 doesn't set up elf_hwcap. It initialises the CPU feature register
> infrastructure which keeps track of the system wide safe values for the ID
> registers. init_cpu_features() initialises the table with the values from
> the boot CPU. This will also be used to detect if any secondary CPU has
> variations in features which could affect the normal functioning of Linux (via
> update_cpu_features() -> check_update_ftr_reg() ).
>
> The elf_hwcap and the system CPU_HWCAP bits are initialised only after all
> the boot time active CPUs are online, from
> setup_cpu_features()-> { setup_elf_hwcaps(), setup_feature_capabilities() }

And the values are set based on the system wide safe values from the CPU feature
infrastructure (using read_system_reg()).

Suzuki

Can cores have different CPU features in a SMP system?

[Jun Sun]

Maybe I'm missing something.  read_system_reg() seems to be reading
the values initialized by cpu0.  Not true?

Jun

On Wed, Aug 3, 2016 at 2:42 AM, Suzuki K Poulose <Suzuki.Poulose@arm.com> wrote:
> On 03/08/16 10:00, Suzuki K Poulose wrote:
>>
>> On 02/08/16 18:23, Jun Sun wrote:
>>>
>>> Hi, Suzuki,
>>>
>>> I was looking at kernel 3.18 and drew the earlier conclusion.
>>>
>>> I'm now looking at kernel v4.7-rc4.  It appears the logic is
>>>
>>>   - cpu 0 sets up elf_hwcap (init_cpu_ftr_reg(), init_cpu_features() )
>>
>>
>> On arm64 CPU 0 doesn't set up elf_hwcap. It initialises the CPU feature
>> register
>> infrastructure which keeps track of the system wide safe values for the ID
>> registers. init_cpu_features() initialises the table with the values from
>> the boot CPU. This will also be used to detect if any secondary CPU has
>> variations in features which could affect the normal functioning of Linux
>> (via
>> update_cpu_features() -> check_update_ftr_reg() ).
>>
>> The elf_hwcap and the system CPU_HWCAP bits are initialised only after all
>> the boot time active CPUs are online, from
>> setup_cpu_features()-> { setup_elf_hwcaps(), setup_feature_capabilities()
>> }
>
>
> And the values are set based on the system wide safe values from the CPU
> feature
> infrastructure (using read_system_reg()).
>
> Suzuki
>

Can cores have different CPU features in a SMP system?

[Suzuki K Poulose]

On 04/08/16 05:14, Jun Sun wrote:
> Maybe I'm missing something.  read_system_reg() seems to be reading
> the values initialized by cpu0.

and subsequently updated by other CPUs. See : update_cpu_ftr_reg()

Suzuki