[PATCH 0/6] An alternative series for asymmetric AArch32 systems

[PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Will Deacon]

Hi all,

I was playing around with the asymmetric AArch32 RFCv2 from Qais:

https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com

and ended up writing my own implementation this afternoon. I think it's
smaller, simpler and easier to work with. In particular:

  * I got rid of the sysctl in favour of a plain cmdline parameter
  * I don't have a new CPU capability
  * I don't have a new thread flag
  * I expose a cpumask to userspace via sysfs to identify the 32-bit CPUs

Anyway, I don't think we should merge this stuff (other than the first patch)
until we've figured out what's going on in Android, but I wanted to get
this out as something which we might be able to build on.

Cheers,

Will

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Morten Rasmussen <morten.rasmussen@arm.com>
Cc: Qais Yousef <qais.yousef@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: kernel-team@android.com

--->8

Qais Yousef (1):
  KVM: arm64: Handle Asymmetric AArch32 systems

Will Deacon (5):
  arm64: Allow mismatched 32-bit EL0 support
  KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support
  arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs
  arm64: Advertise CPUs capable of running 32-bit applcations in sysfs
  arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0

 .../ABI/testing/sysfs-devices-system-cpu      |  8 ++
 .../admin-guide/kernel-parameters.txt         |  7 ++
 arch/arm64/include/asm/cpufeature.h           |  3 +
 arch/arm64/kernel/cpufeature.c                | 80 ++++++++++++++++++-
 arch/arm64/kernel/process.c                   | 21 ++++-
 arch/arm64/kernel/signal.c                    | 26 ++++++
 arch/arm64/kvm/arm.c                          | 27 +++++++
 7 files changed, 168 insertions(+), 4 deletions(-)

-- 
2.29.0.rc2.309.g374f81d7ae-goog

[PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Will Deacon]

Hi all,

I was playing around with the asymmetric AArch32 RFCv2 from Qais:

https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com

and ended up writing my own implementation this afternoon. I think it's
smaller, simpler and easier to work with. In particular:

  * I got rid of the sysctl in favour of a plain cmdline parameter
  * I don't have a new CPU capability
  * I don't have a new thread flag
  * I expose a cpumask to userspace via sysfs to identify the 32-bit CPUs

Anyway, I don't think we should merge this stuff (other than the first patch)
until we've figured out what's going on in Android, but I wanted to get
this out as something which we might be able to build on.

Cheers,

Will

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Morten Rasmussen <morten.rasmussen@arm.com>
Cc: Qais Yousef <qais.yousef@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: kernel-team@android.com

--->8

Qais Yousef (1):
  KVM: arm64: Handle Asymmetric AArch32 systems

Will Deacon (5):
  arm64: Allow mismatched 32-bit EL0 support
  KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support
  arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs
  arm64: Advertise CPUs capable of running 32-bit applcations in sysfs
  arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0

 .../ABI/testing/sysfs-devices-system-cpu      |  8 ++
 .../admin-guide/kernel-parameters.txt         |  7 ++
 arch/arm64/include/asm/cpufeature.h           |  3 +
 arch/arm64/kernel/cpufeature.c                | 80 ++++++++++++++++++-
 arch/arm64/kernel/process.c                   | 21 ++++-
 arch/arm64/kernel/signal.c                    | 26 ++++++
 arch/arm64/kvm/arm.c                          | 27 +++++++
 7 files changed, 168 insertions(+), 4 deletions(-)

-- 
2.29.0.rc2.309.g374f81d7ae-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 1/6] KVM: arm64: Handle Asymmetric AArch32 systems

[Will Deacon]

From: Qais Yousef <qais.yousef@arm.com>

On a system without uniform support for AArch32 at EL0, it is possible
for the guest to force run AArch32 at EL0 and potentially cause an
illegal exception if running on a core without AArch32. Add an extra
check so that if we catch the guest doing that, then we prevent it from
running again by resetting vcpu->arch.target and return
ARM_EXCEPTION_IL.

We try to catch this misbehaviour as early as possible and not rely on
an illegal exception occuring to signal the problem. Attempting to run a
32bit app in the guest will produce an error from QEMU if the guest
exits while running in AArch32 EL0.

Tested on Juno by instrumenting the host to fake asym aarch32 and
instrumenting KVM to make the asymmetry visible to the guest.

Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
[will: Incorporated feedback from Marc]
Link: https://lore.kernel.org/r/20201021104611.2744565-2-qais.yousef@arm.com
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kvm/arm.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index f56122eedffc..a3b32df1afb0 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -808,6 +808,25 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 		preempt_enable();
 
+		/*
+		 * The ARMv8 architecture doesn't give the hypervisor
+		 * a mechanism to prevent a guest from dropping to AArch32 EL0
+		 * if implemented by the CPU. If we spot the guest in such
+		 * state and that we decided it wasn't supposed to do so (like
+		 * with the asymmetric AArch32 case), return to userspace with
+		 * a fatal error.
+		 */
+		if (!system_supports_32bit_el0() && vcpu_mode_is_32bit(vcpu)) {
+			/*
+			 * As we have caught the guest red-handed, decide that
+			 * it isn't fit for purpose anymore by making the vcpu
+			 * invalid. The VMM can try and fix it by issuing  a
+			 * KVM_ARM_VCPU_INIT if it really wants to.
+			 */
+			vcpu->arch.target = -1;
+			ret = ARM_EXCEPTION_IL;
+		}
+
 		ret = handle_exit(vcpu, ret);
 	}
 
-- 
2.29.0.rc2.309.g374f81d7ae-goog

[PATCH 1/6] KVM: arm64: Handle Asymmetric AArch32 systems

[Will Deacon]

From: Qais Yousef <qais.yousef@arm.com>

On a system without uniform support for AArch32 at EL0, it is possible
for the guest to force run AArch32 at EL0 and potentially cause an
illegal exception if running on a core without AArch32. Add an extra
check so that if we catch the guest doing that, then we prevent it from
running again by resetting vcpu->arch.target and return
ARM_EXCEPTION_IL.

We try to catch this misbehaviour as early as possible and not rely on
an illegal exception occuring to signal the problem. Attempting to run a
32bit app in the guest will produce an error from QEMU if the guest
exits while running in AArch32 EL0.

Tested on Juno by instrumenting the host to fake asym aarch32 and
instrumenting KVM to make the asymmetry visible to the guest.

Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
[will: Incorporated feedback from Marc]
Link: https://lore.kernel.org/r/20201021104611.2744565-2-qais.yousef@arm.com
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kvm/arm.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index f56122eedffc..a3b32df1afb0 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -808,6 +808,25 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 		preempt_enable();
 
+		/*
+		 * The ARMv8 architecture doesn't give the hypervisor
+		 * a mechanism to prevent a guest from dropping to AArch32 EL0
+		 * if implemented by the CPU. If we spot the guest in such
+		 * state and that we decided it wasn't supposed to do so (like
+		 * with the asymmetric AArch32 case), return to userspace with
+		 * a fatal error.
+		 */
+		if (!system_supports_32bit_el0() && vcpu_mode_is_32bit(vcpu)) {
+			/*
+			 * As we have caught the guest red-handed, decide that
+			 * it isn't fit for purpose anymore by making the vcpu
+			 * invalid. The VMM can try and fix it by issuing  a
+			 * KVM_ARM_VCPU_INIT if it really wants to.
+			 */
+			vcpu->arch.target = -1;
+			ret = ARM_EXCEPTION_IL;
+		}
+
 		ret = handle_exit(vcpu, ret);
 	}
 
-- 
2.29.0.rc2.309.g374f81d7ae-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

When confronted with a mixture of CPUs, some of which support 32-bit
applications and others which don't, we quite sensibly treat the system
as 64-bit only for userspace and prevent execve() of 32-bit binaries.

Unfortunately, some crazy folks have decided to build systems like this
with the intention of running 32-bit applications, so relax our
sanitisation logic to continue to advertise 32-bit support to userspace
on these systems and track the real 32-bit capable cores in a cpumask
instead. For now, the default behaviour remains but will be tied to
a command-line option in a later patch.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/cpufeature.h |  3 ++
 arch/arm64/kernel/cpufeature.c      | 54 +++++++++++++++++++++++++++--
 2 files changed, 54 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index f7e7144af174..aeab42cb917e 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -582,6 +582,9 @@ static inline bool cpu_supports_mixed_endian_el0(void)
 	return id_aa64mmfr0_mixed_endian_el0(read_cpuid(ID_AA64MMFR0_EL1));
 }
 
+const struct cpumask *system_32bit_el0_cpumask(void);
+bool system_has_mismatched_32bit_el0(void);
+
 static inline bool system_supports_32bit_el0(void)
 {
 	return cpus_have_const_cap(ARM64_HAS_32BIT_EL0);
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index dcc165b3fc04..2e2219cbd54c 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -104,6 +104,10 @@ DECLARE_BITMAP(boot_capabilities, ARM64_NPATCHABLE);
 bool arm64_use_ng_mappings = false;
 EXPORT_SYMBOL(arm64_use_ng_mappings);
 
+static bool __read_mostly __allow_mismatched_32bit_el0;
+/* Mask of CPUs supporting 32-bit EL0. Only valid if we allow a mismatch */
+static cpumask_var_t cpu_32bit_el0_mask __cpumask_var_read_mostly;
+
 /*
  * Flag to indicate if we have computed the system wide
  * capabilities based on the boot time active CPUs. This
@@ -942,8 +946,11 @@ static int update_32bit_cpu_features(int cpu, struct cpuinfo_arm64 *info,
 	 * as the register values may be UNKNOWN and we're not going to be
 	 * using them for anything.
 	 */
-	if (!id_aa64pfr0_32bit_el0(pfr0))
-		return taint;
+	if (!id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0))
+		return 0;
+
+	if (__allow_mismatched_32bit_el0)
+		cpumask_set_cpu(cpu, cpu_32bit_el0_mask);
 
 	/*
 	 * If we don't have AArch32 at EL1, then relax the strictness of
@@ -1193,6 +1200,47 @@ has_cpuid_feature(const struct arm64_cpu_capabilities *entry, int scope)
 	return feature_matches(val, entry);
 }
 
+static int __init init_32bit_el0_mask(void)
+{
+	if (!__allow_mismatched_32bit_el0)
+		return 0;
+
+	if (!alloc_cpumask_var(&cpu_32bit_el0_mask, GFP_KERNEL))
+		return -ENOMEM;
+
+	if (id_aa64pfr0_32bit_el0(per_cpu(cpu_data, 0).reg_id_aa64pfr0))
+		cpumask_set_cpu(0, cpu_32bit_el0_mask);
+
+	return 0;
+}
+early_initcall(init_32bit_el0_mask);
+
+const struct cpumask *system_32bit_el0_cpumask(void)
+{
+	if (__allow_mismatched_32bit_el0)
+		return cpu_32bit_el0_mask;
+
+	return system_supports_32bit_el0() ? cpu_present_mask : cpu_none_mask;
+}
+
+bool system_has_mismatched_32bit_el0(void)
+{
+	u64 reg;
+	unsigned int fld;
+
+	if (!__allow_mismatched_32bit_el0)
+		return false;
+
+	reg = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
+	fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64PFR0_EL0_SHIFT);
+	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
+}
+
+static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
+{
+	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
+}
+
 static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
 {
 	bool has_sre;
@@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.desc = "32-bit EL0 Support",
 		.capability = ARM64_HAS_32BIT_EL0,
 		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
-		.matches = has_cpuid_feature,
+		.matches = has_32bit_el0,
 		.sys_reg = SYS_ID_AA64PFR0_EL1,
 		.sign = FTR_UNSIGNED,
 		.field_pos = ID_AA64PFR0_EL0_SHIFT,
-- 
2.29.0.rc2.309.g374f81d7ae-goog

[PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

When confronted with a mixture of CPUs, some of which support 32-bit
applications and others which don't, we quite sensibly treat the system
as 64-bit only for userspace and prevent execve() of 32-bit binaries.

Unfortunately, some crazy folks have decided to build systems like this
with the intention of running 32-bit applications, so relax our
sanitisation logic to continue to advertise 32-bit support to userspace
on these systems and track the real 32-bit capable cores in a cpumask
instead. For now, the default behaviour remains but will be tied to
a command-line option in a later patch.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/cpufeature.h |  3 ++
 arch/arm64/kernel/cpufeature.c      | 54 +++++++++++++++++++++++++++--
 2 files changed, 54 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index f7e7144af174..aeab42cb917e 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -582,6 +582,9 @@ static inline bool cpu_supports_mixed_endian_el0(void)
 	return id_aa64mmfr0_mixed_endian_el0(read_cpuid(ID_AA64MMFR0_EL1));
 }
 
+const struct cpumask *system_32bit_el0_cpumask(void);
+bool system_has_mismatched_32bit_el0(void);
+
 static inline bool system_supports_32bit_el0(void)
 {
 	return cpus_have_const_cap(ARM64_HAS_32BIT_EL0);
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index dcc165b3fc04..2e2219cbd54c 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -104,6 +104,10 @@ DECLARE_BITMAP(boot_capabilities, ARM64_NPATCHABLE);
 bool arm64_use_ng_mappings = false;
 EXPORT_SYMBOL(arm64_use_ng_mappings);
 
+static bool __read_mostly __allow_mismatched_32bit_el0;
+/* Mask of CPUs supporting 32-bit EL0. Only valid if we allow a mismatch */
+static cpumask_var_t cpu_32bit_el0_mask __cpumask_var_read_mostly;
+
 /*
  * Flag to indicate if we have computed the system wide
  * capabilities based on the boot time active CPUs. This
@@ -942,8 +946,11 @@ static int update_32bit_cpu_features(int cpu, struct cpuinfo_arm64 *info,
 	 * as the register values may be UNKNOWN and we're not going to be
 	 * using them for anything.
 	 */
-	if (!id_aa64pfr0_32bit_el0(pfr0))
-		return taint;
+	if (!id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0))
+		return 0;
+
+	if (__allow_mismatched_32bit_el0)
+		cpumask_set_cpu(cpu, cpu_32bit_el0_mask);
 
 	/*
 	 * If we don't have AArch32 at EL1, then relax the strictness of
@@ -1193,6 +1200,47 @@ has_cpuid_feature(const struct arm64_cpu_capabilities *entry, int scope)
 	return feature_matches(val, entry);
 }
 
+static int __init init_32bit_el0_mask(void)
+{
+	if (!__allow_mismatched_32bit_el0)
+		return 0;
+
+	if (!alloc_cpumask_var(&cpu_32bit_el0_mask, GFP_KERNEL))
+		return -ENOMEM;
+
+	if (id_aa64pfr0_32bit_el0(per_cpu(cpu_data, 0).reg_id_aa64pfr0))
+		cpumask_set_cpu(0, cpu_32bit_el0_mask);
+
+	return 0;
+}
+early_initcall(init_32bit_el0_mask);
+
+const struct cpumask *system_32bit_el0_cpumask(void)
+{
+	if (__allow_mismatched_32bit_el0)
+		return cpu_32bit_el0_mask;
+
+	return system_supports_32bit_el0() ? cpu_present_mask : cpu_none_mask;
+}
+
+bool system_has_mismatched_32bit_el0(void)
+{
+	u64 reg;
+	unsigned int fld;
+
+	if (!__allow_mismatched_32bit_el0)
+		return false;
+
+	reg = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
+	fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64PFR0_EL0_SHIFT);
+	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
+}
+
+static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
+{
+	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
+}
+
 static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
 {
 	bool has_sre;
@@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.desc = "32-bit EL0 Support",
 		.capability = ARM64_HAS_32BIT_EL0,
 		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
-		.matches = has_cpuid_feature,
+		.matches = has_32bit_el0,
 		.sys_reg = SYS_ID_AA64PFR0_EL1,
 		.sign = FTR_UNSIGNED,
 		.field_pos = ID_AA64PFR0_EL0_SHIFT,
-- 
2.29.0.rc2.309.g374f81d7ae-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 3/6] KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support

[Will Deacon]

If a vCPU tries to run 32-bit code on a system with mismatched support
at EL0, then we should kill it.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kvm/arm.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index a3b32df1afb0..7150dd996e1d 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -633,6 +633,14 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
 	}
 }
 
+static bool vcpu_mode_is_bad_32bit(struct kvm_vcpu *vcpu)
+{
+	if (likely(!vcpu_mode_is_32bit(vcpu)))
+		return false;
+
+	return !system_supports_32bit_el0() || system_has_mismatched_32bit_el0();
+}
+
 /**
  * kvm_arch_vcpu_ioctl_run - the main VCPU run function to execute guest code
  * @vcpu:	The VCPU pointer
@@ -816,7 +824,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		 * with the asymmetric AArch32 case), return to userspace with
 		 * a fatal error.
 		 */
-		if (!system_supports_32bit_el0() && vcpu_mode_is_32bit(vcpu)) {
+		if (vcpu_mode_is_bad_32bit(vcpu)) {
 			/*
 			 * As we have caught the guest red-handed, decide that
 			 * it isn't fit for purpose anymore by making the vcpu
-- 
2.29.0.rc2.309.g374f81d7ae-goog

[PATCH 3/6] KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support

[Will Deacon]

If a vCPU tries to run 32-bit code on a system with mismatched support
at EL0, then we should kill it.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kvm/arm.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index a3b32df1afb0..7150dd996e1d 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -633,6 +633,14 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
 	}
 }
 
+static bool vcpu_mode_is_bad_32bit(struct kvm_vcpu *vcpu)
+{
+	if (likely(!vcpu_mode_is_32bit(vcpu)))
+		return false;
+
+	return !system_supports_32bit_el0() || system_has_mismatched_32bit_el0();
+}
+
 /**
  * kvm_arch_vcpu_ioctl_run - the main VCPU run function to execute guest code
  * @vcpu:	The VCPU pointer
@@ -816,7 +824,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		 * with the asymmetric AArch32 case), return to userspace with
 		 * a fatal error.
 		 */
-		if (!system_supports_32bit_el0() && vcpu_mode_is_32bit(vcpu)) {
+		if (vcpu_mode_is_bad_32bit(vcpu)) {
 			/*
 			 * As we have caught the guest red-handed, decide that
 			 * it isn't fit for purpose anymore by making the vcpu
-- 
2.29.0.rc2.309.g374f81d7ae-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 4/6] arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs

[Will Deacon]

Scheduling a 32-bit application on a 64-bit-only CPU is a bad idea.

Ensure that 32-bit applications always take the slow-path when returning
to userspace on a system with mismatched support at EL0, so that we can
avoid trying to run on a 64-bit-only CPU and force a SIGKILL instead.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/process.c | 21 ++++++++++++++++++++-
 arch/arm64/kernel/signal.c  | 26 ++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 4784011cecac..c45b5f9dd66b 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -542,6 +542,17 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,
 	write_sysreg(val, cntkctl_el1);
 }
 
+static void compat_thread_switch(struct task_struct *next)
+{
+	if (!is_compat_thread(task_thread_info(next)))
+		return;
+
+	if (!system_has_mismatched_32bit_el0())
+		return;
+
+	set_tsk_thread_flag(next, TIF_NOTIFY_RESUME);
+}
+
 /*
  * Thread switching.
  */
@@ -558,6 +569,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,
 	uao_thread_switch(next);
 	ssbs_thread_switch(next);
 	erratum_1418040_thread_switch(prev, next);
+	compat_thread_switch(next);
 
 	/*
 	 * Complete any pending TLB or cache maintenance on this CPU in case
@@ -620,8 +632,15 @@ unsigned long arch_align_stack(unsigned long sp)
  */
 void arch_setup_new_exec(void)
 {
-	current->mm->context.flags = is_compat_task() ? MMCF_AARCH32 : 0;
+	unsigned long mmflags = 0;
+
+	if (is_compat_task()) {
+		mmflags = MMCF_AARCH32;
+		if (system_has_mismatched_32bit_el0())
+			set_tsk_thread_flag(current, TIF_NOTIFY_RESUME);
+	}
 
+	current->mm->context.flags = mmflags;
 	ptrauth_thread_init_user(current);
 
 	if (task_spec_ssb_noexec(current)) {
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index a8184cad8890..bcb6ca2d9a7c 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -911,6 +911,19 @@ static void do_signal(struct pt_regs *regs)
 	restore_saved_sigmask();
 }
 
+static bool cpu_affinity_invalid(struct pt_regs *regs)
+{
+	if (!compat_user_mode(regs))
+		return false;
+
+	/*
+	 * We're preemptible, but a reschedule will cause us to check the
+	 * affinity again.
+	 */
+	return !cpumask_test_cpu(raw_smp_processor_id(),
+				 system_32bit_el0_cpumask());
+}
+
 asmlinkage void do_notify_resume(struct pt_regs *regs,
 				 unsigned long thread_flags)
 {
@@ -948,6 +961,19 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,
 			if (thread_flags & _TIF_NOTIFY_RESUME) {
 				tracehook_notify_resume(regs);
 				rseq_handle_notify_resume(NULL, regs);
+
+				/*
+				 * If we reschedule after checking the affinity
+				 * then we must ensure that TIF_NOTIFY_RESUME
+				 * is set so that we check the affinity again.
+				 * Since tracehook_notify_resume() clears the
+				 * flag, ensure that the compiler doesn't move
+				 * it after the affinity check.
+				 */
+				barrier();
+
+				if (cpu_affinity_invalid(regs))
+					force_sig(SIGKILL);
 			}
 
 			if (thread_flags & _TIF_FOREIGN_FPSTATE)
-- 
2.29.0.rc2.309.g374f81d7ae-goog

[PATCH 4/6] arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs

[Will Deacon]

Scheduling a 32-bit application on a 64-bit-only CPU is a bad idea.

Ensure that 32-bit applications always take the slow-path when returning
to userspace on a system with mismatched support at EL0, so that we can
avoid trying to run on a 64-bit-only CPU and force a SIGKILL instead.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/process.c | 21 ++++++++++++++++++++-
 arch/arm64/kernel/signal.c  | 26 ++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 4784011cecac..c45b5f9dd66b 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -542,6 +542,17 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,
 	write_sysreg(val, cntkctl_el1);
 }
 
+static void compat_thread_switch(struct task_struct *next)
+{
+	if (!is_compat_thread(task_thread_info(next)))
+		return;
+
+	if (!system_has_mismatched_32bit_el0())
+		return;
+
+	set_tsk_thread_flag(next, TIF_NOTIFY_RESUME);
+}
+
 /*
  * Thread switching.
  */
@@ -558,6 +569,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,
 	uao_thread_switch(next);
 	ssbs_thread_switch(next);
 	erratum_1418040_thread_switch(prev, next);
+	compat_thread_switch(next);
 
 	/*
 	 * Complete any pending TLB or cache maintenance on this CPU in case
@@ -620,8 +632,15 @@ unsigned long arch_align_stack(unsigned long sp)
  */
 void arch_setup_new_exec(void)
 {
-	current->mm->context.flags = is_compat_task() ? MMCF_AARCH32 : 0;
+	unsigned long mmflags = 0;
+
+	if (is_compat_task()) {
+		mmflags = MMCF_AARCH32;
+		if (system_has_mismatched_32bit_el0())
+			set_tsk_thread_flag(current, TIF_NOTIFY_RESUME);
+	}
 
+	current->mm->context.flags = mmflags;
 	ptrauth_thread_init_user(current);
 
 	if (task_spec_ssb_noexec(current)) {
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index a8184cad8890..bcb6ca2d9a7c 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -911,6 +911,19 @@ static void do_signal(struct pt_regs *regs)
 	restore_saved_sigmask();
 }
 
+static bool cpu_affinity_invalid(struct pt_regs *regs)
+{
+	if (!compat_user_mode(regs))
+		return false;
+
+	/*
+	 * We're preemptible, but a reschedule will cause us to check the
+	 * affinity again.
+	 */
+	return !cpumask_test_cpu(raw_smp_processor_id(),
+				 system_32bit_el0_cpumask());
+}
+
 asmlinkage void do_notify_resume(struct pt_regs *regs,
 				 unsigned long thread_flags)
 {
@@ -948,6 +961,19 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,
 			if (thread_flags & _TIF_NOTIFY_RESUME) {
 				tracehook_notify_resume(regs);
 				rseq_handle_notify_resume(NULL, regs);
+
+				/*
+				 * If we reschedule after checking the affinity
+				 * then we must ensure that TIF_NOTIFY_RESUME
+				 * is set so that we check the affinity again.
+				 * Since tracehook_notify_resume() clears the
+				 * flag, ensure that the compiler doesn't move
+				 * it after the affinity check.
+				 */
+				barrier();
+
+				if (cpu_affinity_invalid(regs))
+					force_sig(SIGKILL);
 			}
 
 			if (thread_flags & _TIF_FOREIGN_FPSTATE)
-- 
2.29.0.rc2.309.g374f81d7ae-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Will Deacon]

Since 32-bit applications will be killed if they are caught trying to
execute on a 64-bit-only CPU in a mismatched system, advertise the set
of 32-bit capable CPUs to userspace in sysfs.

Signed-off-by: Will Deacon <will@kernel.org>
---
 .../ABI/testing/sysfs-devices-system-cpu      |  8 ++++++++
 arch/arm64/kernel/cpufeature.c                | 19 +++++++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
index b555df825447..19893fb8e870 100644
--- a/Documentation/ABI/testing/sysfs-devices-system-cpu
+++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
@@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
 		'identification' directory exposes the CPU ID registers for
 		 identifying model and revision of the CPU.
 
+What:		/sys/devices/system/cpu/aarch32_el0
+Date:		October 2020
+Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
+Description:	Identifies the subset of CPUs in the system that can execute
+		AArch32 (32-bit ARM) applications. If absent, then all or none
+		of the CPUs can execute AArch32 applications and execve() will
+		behave accordingly.
+
 What:		/sys/devices/system/cpu/cpu#/cpu_capacity
 Date:		December 2016
 Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 2e2219cbd54c..9f29d4d1ef7e 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -67,6 +67,7 @@
 #include <linux/crash_dump.h>
 #include <linux/sort.h>
 #include <linux/stop_machine.h>
+#include <linux/sysfs.h>
 #include <linux/types.h>
 #include <linux/mm.h>
 #include <linux/cpu.h>
@@ -1236,6 +1237,24 @@ bool system_has_mismatched_32bit_el0(void)
 	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
 }
 
+static ssize_t aarch32_el0_show(struct kobject *kobj,
+				struct kobj_attribute *attr, char *buf)
+{
+	const struct cpumask *mask = system_32bit_el0_cpumask();
+	return sprintf(buf, "%*pbl\n", cpumask_pr_args(mask));
+}
+static const struct kobj_attribute aarch32_el0_attr = __ATTR_RO(aarch32_el0);
+
+static int __init aarch32_el0_sysfs_init(void)
+{
+	if (!__allow_mismatched_32bit_el0)
+		return 0;
+
+	return sysfs_create_file(&cpu_subsys.dev_root->kobj,
+				 &aarch32_el0_attr.attr);
+}
+device_initcall(aarch32_el0_sysfs_init);
+
 static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
 {
 	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
-- 
2.29.0.rc2.309.g374f81d7ae-goog

[PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Will Deacon]

Since 32-bit applications will be killed if they are caught trying to
execute on a 64-bit-only CPU in a mismatched system, advertise the set
of 32-bit capable CPUs to userspace in sysfs.

Signed-off-by: Will Deacon <will@kernel.org>
---
 .../ABI/testing/sysfs-devices-system-cpu      |  8 ++++++++
 arch/arm64/kernel/cpufeature.c                | 19 +++++++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
index b555df825447..19893fb8e870 100644
--- a/Documentation/ABI/testing/sysfs-devices-system-cpu
+++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
@@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
 		'identification' directory exposes the CPU ID registers for
 		 identifying model and revision of the CPU.
 
+What:		/sys/devices/system/cpu/aarch32_el0
+Date:		October 2020
+Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
+Description:	Identifies the subset of CPUs in the system that can execute
+		AArch32 (32-bit ARM) applications. If absent, then all or none
+		of the CPUs can execute AArch32 applications and execve() will
+		behave accordingly.
+
 What:		/sys/devices/system/cpu/cpu#/cpu_capacity
 Date:		December 2016
 Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 2e2219cbd54c..9f29d4d1ef7e 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -67,6 +67,7 @@
 #include <linux/crash_dump.h>
 #include <linux/sort.h>
 #include <linux/stop_machine.h>
+#include <linux/sysfs.h>
 #include <linux/types.h>
 #include <linux/mm.h>
 #include <linux/cpu.h>
@@ -1236,6 +1237,24 @@ bool system_has_mismatched_32bit_el0(void)
 	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
 }
 
+static ssize_t aarch32_el0_show(struct kobject *kobj,
+				struct kobj_attribute *attr, char *buf)
+{
+	const struct cpumask *mask = system_32bit_el0_cpumask();
+	return sprintf(buf, "%*pbl\n", cpumask_pr_args(mask));
+}
+static const struct kobj_attribute aarch32_el0_attr = __ATTR_RO(aarch32_el0);
+
+static int __init aarch32_el0_sysfs_init(void)
+{
+	if (!__allow_mismatched_32bit_el0)
+		return 0;
+
+	return sysfs_create_file(&cpu_subsys.dev_root->kobj,
+				 &aarch32_el0_attr.attr);
+}
+device_initcall(aarch32_el0_sysfs_init);
+
 static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
 {
 	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
-- 
2.29.0.rc2.309.g374f81d7ae-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 6/6] arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0

[Will Deacon]

Allow systems with mismatched 32-bit support at EL0 to run 32-bit
applications based on a new kernel parameter.

Signed-off-by: Will Deacon <will@kernel.org>
---
 Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
 arch/arm64/kernel/cpufeature.c                  | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 526d65d8573a..1d9021db4d9f 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -289,6 +289,13 @@
 			do not want to use tracing_snapshot_alloc() as it needs
 			to be done where GFP_KERNEL allocations are allowed.
 
+	allow_mismatched_32bit_el0 [ARM64]
+			Allow execve() of 32-bit applications and setting of the
+			PER_LINUX32 personality on systems where not all of the
+			CPUs support 32-bit EL0. When this parameter is present,
+			the set of CPUs supporting 32-bit EL0 is indicated by
+			/sys/devices/system/cpu/aarch32_el0.
+
 	amd_iommu=	[HW,X86-64]
 			Pass parameters to the AMD IOMMU driver in the system.
 			Possible values are:
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9f29d4d1ef7e..ccc08da443ec 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1237,6 +1237,13 @@ bool system_has_mismatched_32bit_el0(void)
 	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
 }
 
+static int __init parse_32bit_el0_param(char *str)
+{
+	__allow_mismatched_32bit_el0 = true;
+	return 0;
+}
+early_param("allow_mismatched_32bit_el0", parse_32bit_el0_param);
+
 static ssize_t aarch32_el0_show(struct kobject *kobj,
 				struct kobj_attribute *attr, char *buf)
 {
-- 
2.29.0.rc2.309.g374f81d7ae-goog

[PATCH 6/6] arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0

[Will Deacon]

Allow systems with mismatched 32-bit support at EL0 to run 32-bit
applications based on a new kernel parameter.

Signed-off-by: Will Deacon <will@kernel.org>
---
 Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
 arch/arm64/kernel/cpufeature.c                  | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 526d65d8573a..1d9021db4d9f 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -289,6 +289,13 @@
 			do not want to use tracing_snapshot_alloc() as it needs
 			to be done where GFP_KERNEL allocations are allowed.
 
+	allow_mismatched_32bit_el0 [ARM64]
+			Allow execve() of 32-bit applications and setting of the
+			PER_LINUX32 personality on systems where not all of the
+			CPUs support 32-bit EL0. When this parameter is present,
+			the set of CPUs supporting 32-bit EL0 is indicated by
+			/sys/devices/system/cpu/aarch32_el0.
+
 	amd_iommu=	[HW,X86-64]
 			Pass parameters to the AMD IOMMU driver in the system.
 			Possible values are:
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9f29d4d1ef7e..ccc08da443ec 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1237,6 +1237,13 @@ bool system_has_mismatched_32bit_el0(void)
 	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
 }
 
+static int __init parse_32bit_el0_param(char *str)
+{
+	__allow_mismatched_32bit_el0 = true;
+	return 0;
+}
+early_param("allow_mismatched_32bit_el0", parse_32bit_el0_param);
+
 static ssize_t aarch32_el0_show(struct kobject *kobj,
 				struct kobj_attribute *attr, char *buf)
 {
-- 
2.29.0.rc2.309.g374f81d7ae-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Greg Kroah-Hartman]

On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> Since 32-bit applications will be killed if they are caught trying to
> execute on a 64-bit-only CPU in a mismatched system, advertise the set
> of 32-bit capable CPUs to userspace in sysfs.
> 
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
>  .../ABI/testing/sysfs-devices-system-cpu      |  8 ++++++++
>  arch/arm64/kernel/cpufeature.c                | 19 +++++++++++++++++++
>  2 files changed, 27 insertions(+)
> 
> diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> index b555df825447..19893fb8e870 100644
> --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
>  		'identification' directory exposes the CPU ID registers for
>  		 identifying model and revision of the CPU.
>  
> +What:		/sys/devices/system/cpu/aarch32_el0
> +Date:		October 2020
> +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> +Description:	Identifies the subset of CPUs in the system that can execute
> +		AArch32 (32-bit ARM) applications. If absent, then all or none
> +		of the CPUs can execute AArch32 applications and execve() will
> +		behave accordingly.

How is this value represented?  A hint here would be nice.

> +
>  What:		/sys/devices/system/cpu/cpu#/cpu_capacity
>  Date:		December 2016
>  Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index 2e2219cbd54c..9f29d4d1ef7e 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -67,6 +67,7 @@
>  #include <linux/crash_dump.h>
>  #include <linux/sort.h>
>  #include <linux/stop_machine.h>
> +#include <linux/sysfs.h>
>  #include <linux/types.h>
>  #include <linux/mm.h>
>  #include <linux/cpu.h>
> @@ -1236,6 +1237,24 @@ bool system_has_mismatched_32bit_el0(void)
>  	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
>  }
>  
> +static ssize_t aarch32_el0_show(struct kobject *kobj,
> +				struct kobj_attribute *attr, char *buf)
> +{
> +	const struct cpumask *mask = system_32bit_el0_cpumask();
> +	return sprintf(buf, "%*pbl\n", cpumask_pr_args(mask));

sysfs_emit()?

And a blank line to make checkpatch.pl happy :)

> +}
> +static const struct kobj_attribute aarch32_el0_attr = __ATTR_RO(aarch32_el0);

DEVICE_ATTR_RO()?

> +
> +static int __init aarch32_el0_sysfs_init(void)
> +{
> +	if (!__allow_mismatched_32bit_el0)
> +		return 0;
> +
> +	return sysfs_create_file(&cpu_subsys.dev_root->kobj,
> +				 &aarch32_el0_attr.attr);

device_create_file() please, dev_root is a struct device, no need to
"thunk" down to a "raw" sysfs call.

thanks,

greg k-h

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Greg Kroah-Hartman]

On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> Since 32-bit applications will be killed if they are caught trying to
> execute on a 64-bit-only CPU in a mismatched system, advertise the set
> of 32-bit capable CPUs to userspace in sysfs.
> 
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
>  .../ABI/testing/sysfs-devices-system-cpu      |  8 ++++++++
>  arch/arm64/kernel/cpufeature.c                | 19 +++++++++++++++++++
>  2 files changed, 27 insertions(+)
> 
> diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> index b555df825447..19893fb8e870 100644
> --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
>  		'identification' directory exposes the CPU ID registers for
>  		 identifying model and revision of the CPU.
>  
> +What:		/sys/devices/system/cpu/aarch32_el0
> +Date:		October 2020
> +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> +Description:	Identifies the subset of CPUs in the system that can execute
> +		AArch32 (32-bit ARM) applications. If absent, then all or none
> +		of the CPUs can execute AArch32 applications and execve() will
> +		behave accordingly.

How is this value represented?  A hint here would be nice.

> +
>  What:		/sys/devices/system/cpu/cpu#/cpu_capacity
>  Date:		December 2016
>  Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index 2e2219cbd54c..9f29d4d1ef7e 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -67,6 +67,7 @@
>  #include <linux/crash_dump.h>
>  #include <linux/sort.h>
>  #include <linux/stop_machine.h>
> +#include <linux/sysfs.h>
>  #include <linux/types.h>
>  #include <linux/mm.h>
>  #include <linux/cpu.h>
> @@ -1236,6 +1237,24 @@ bool system_has_mismatched_32bit_el0(void)
>  	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
>  }
>  
> +static ssize_t aarch32_el0_show(struct kobject *kobj,
> +				struct kobj_attribute *attr, char *buf)
> +{
> +	const struct cpumask *mask = system_32bit_el0_cpumask();
> +	return sprintf(buf, "%*pbl\n", cpumask_pr_args(mask));

sysfs_emit()?

And a blank line to make checkpatch.pl happy :)

> +}
> +static const struct kobj_attribute aarch32_el0_attr = __ATTR_RO(aarch32_el0);

DEVICE_ATTR_RO()?

> +
> +static int __init aarch32_el0_sysfs_init(void)
> +{
> +	if (!__allow_mismatched_32bit_el0)
> +		return 0;
> +
> +	return sysfs_create_file(&cpu_subsys.dev_root->kobj,
> +				 &aarch32_el0_attr.attr);

device_create_file() please, dev_root is a struct device, no need to
"thunk" down to a "raw" sysfs call.

thanks,

greg k-h

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Will Deacon]

On Wed, Oct 28, 2020 at 09:37:46AM +0100, Greg Kroah-Hartman wrote:
> On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> > Since 32-bit applications will be killed if they are caught trying to
> > execute on a 64-bit-only CPU in a mismatched system, advertise the set
> > of 32-bit capable CPUs to userspace in sysfs.
> > 
> > Signed-off-by: Will Deacon <will@kernel.org>
> > ---
> >  .../ABI/testing/sysfs-devices-system-cpu      |  8 ++++++++
> >  arch/arm64/kernel/cpufeature.c                | 19 +++++++++++++++++++
> >  2 files changed, 27 insertions(+)
> > 
> > diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > index b555df825447..19893fb8e870 100644
> > --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> > +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
> >  		'identification' directory exposes the CPU ID registers for
> >  		 identifying model and revision of the CPU.
> >  
> > +What:		/sys/devices/system/cpu/aarch32_el0
> > +Date:		October 2020
> > +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> > +Description:	Identifies the subset of CPUs in the system that can execute
> > +		AArch32 (32-bit ARM) applications. If absent, then all or none
> > +		of the CPUs can execute AArch32 applications and execve() will
> > +		behave accordingly.
> 
> How is this value represented?  A hint here would be nice.

It's in the same format as
/sys/devices/system/cpu/{online,offline,possible,present}, so I'll just say
that (although the text for those doesn't seem to specify it either...).

> >  What:		/sys/devices/system/cpu/cpu#/cpu_capacity
> >  Date:		December 2016
> >  Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
> > diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> > index 2e2219cbd54c..9f29d4d1ef7e 100644
> > --- a/arch/arm64/kernel/cpufeature.c
> > +++ b/arch/arm64/kernel/cpufeature.c
> > @@ -67,6 +67,7 @@
> >  #include <linux/crash_dump.h>
> >  #include <linux/sort.h>
> >  #include <linux/stop_machine.h>
> > +#include <linux/sysfs.h>
> >  #include <linux/types.h>
> >  #include <linux/mm.h>
> >  #include <linux/cpu.h>
> > @@ -1236,6 +1237,24 @@ bool system_has_mismatched_32bit_el0(void)
> >  	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
> >  }
> >  
> > +static ssize_t aarch32_el0_show(struct kobject *kobj,
> > +				struct kobj_attribute *attr, char *buf)
> > +{
> > +	const struct cpumask *mask = system_32bit_el0_cpumask();
> > +	return sprintf(buf, "%*pbl\n", cpumask_pr_args(mask));
> 
> sysfs_emit()?
> 
> And a blank line to make checkpatch.pl happy :)

Hehe, yeah ok.

> > +}
> > +static const struct kobj_attribute aarch32_el0_attr = __ATTR_RO(aarch32_el0);
> 
> DEVICE_ATTR_RO()?
> 
> > +
> > +static int __init aarch32_el0_sysfs_init(void)
> > +{
> > +	if (!__allow_mismatched_32bit_el0)
> > +		return 0;
> > +
> > +	return sysfs_create_file(&cpu_subsys.dev_root->kobj,
> > +				 &aarch32_el0_attr.attr);
> 
> device_create_file() please, dev_root is a struct device, no need to
> "thunk" down to a "raw" sysfs call.

Totally missed I had a struct device in my hand, so hopefully that will tidy
things up a little bit.

Cheers,

Will

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Will Deacon]

On Wed, Oct 28, 2020 at 09:37:46AM +0100, Greg Kroah-Hartman wrote:
> On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> > Since 32-bit applications will be killed if they are caught trying to
> > execute on a 64-bit-only CPU in a mismatched system, advertise the set
> > of 32-bit capable CPUs to userspace in sysfs.
> > 
> > Signed-off-by: Will Deacon <will@kernel.org>
> > ---
> >  .../ABI/testing/sysfs-devices-system-cpu      |  8 ++++++++
> >  arch/arm64/kernel/cpufeature.c                | 19 +++++++++++++++++++
> >  2 files changed, 27 insertions(+)
> > 
> > diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > index b555df825447..19893fb8e870 100644
> > --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> > +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
> >  		'identification' directory exposes the CPU ID registers for
> >  		 identifying model and revision of the CPU.
> >  
> > +What:		/sys/devices/system/cpu/aarch32_el0
> > +Date:		October 2020
> > +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> > +Description:	Identifies the subset of CPUs in the system that can execute
> > +		AArch32 (32-bit ARM) applications. If absent, then all or none
> > +		of the CPUs can execute AArch32 applications and execve() will
> > +		behave accordingly.
> 
> How is this value represented?  A hint here would be nice.

It's in the same format as
/sys/devices/system/cpu/{online,offline,possible,present}, so I'll just say
that (although the text for those doesn't seem to specify it either...).

> >  What:		/sys/devices/system/cpu/cpu#/cpu_capacity
> >  Date:		December 2016
> >  Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
> > diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> > index 2e2219cbd54c..9f29d4d1ef7e 100644
> > --- a/arch/arm64/kernel/cpufeature.c
> > +++ b/arch/arm64/kernel/cpufeature.c
> > @@ -67,6 +67,7 @@
> >  #include <linux/crash_dump.h>
> >  #include <linux/sort.h>
> >  #include <linux/stop_machine.h>
> > +#include <linux/sysfs.h>
> >  #include <linux/types.h>
> >  #include <linux/mm.h>
> >  #include <linux/cpu.h>
> > @@ -1236,6 +1237,24 @@ bool system_has_mismatched_32bit_el0(void)
> >  	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
> >  }
> >  
> > +static ssize_t aarch32_el0_show(struct kobject *kobj,
> > +				struct kobj_attribute *attr, char *buf)
> > +{
> > +	const struct cpumask *mask = system_32bit_el0_cpumask();
> > +	return sprintf(buf, "%*pbl\n", cpumask_pr_args(mask));
> 
> sysfs_emit()?
> 
> And a blank line to make checkpatch.pl happy :)

Hehe, yeah ok.

> > +}
> > +static const struct kobj_attribute aarch32_el0_attr = __ATTR_RO(aarch32_el0);
> 
> DEVICE_ATTR_RO()?
> 
> > +
> > +static int __init aarch32_el0_sysfs_init(void)
> > +{
> > +	if (!__allow_mismatched_32bit_el0)
> > +		return 0;
> > +
> > +	return sysfs_create_file(&cpu_subsys.dev_root->kobj,
> > +				 &aarch32_el0_attr.attr);
> 
> device_create_file() please, dev_root is a struct device, no need to
> "thunk" down to a "raw" sysfs call.

Totally missed I had a struct device in my hand, so hopefully that will tidy
things up a little bit.

Cheers,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> +{
> +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> +}
> +
>  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
>  {
>  	bool has_sre;
> @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
>  		.desc = "32-bit EL0 Support",
>  		.capability = ARM64_HAS_32BIT_EL0,
>  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> -		.matches = has_cpuid_feature,
> +		.matches = has_32bit_el0,

Ah, so this one reports 32-bit EL0 support even if no CPU actually
supports 32-bit (passing the command line option on TX2 would come up
with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
information elsewhere when have at least one CPU supporting this.

-- 
Catalin

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> +{
> +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> +}
> +
>  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
>  {
>  	bool has_sre;
> @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
>  		.desc = "32-bit EL0 Support",
>  		.capability = ARM64_HAS_32BIT_EL0,
>  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> -		.matches = has_cpuid_feature,
> +		.matches = has_32bit_el0,

Ah, so this one reports 32-bit EL0 support even if no CPU actually
supports 32-bit (passing the command line option on TX2 would come up
with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
information elsewhere when have at least one CPU supporting this.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > +{
> > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > +}
> > +
> >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> >  {
> >  	bool has_sre;
> > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> >  		.desc = "32-bit EL0 Support",
> >  		.capability = ARM64_HAS_32BIT_EL0,
> >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > -		.matches = has_cpuid_feature,
> > +		.matches = has_32bit_el0,
> 
> Ah, so this one reports 32-bit EL0 support even if no CPU actually
> supports 32-bit (passing the command line option on TX2 would come up
> with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> information elsewhere when have at least one CPU supporting this.

Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
then we would have 32-bit support, so I think this is an oddity that you
get when the command line is passed. That said, I could nobble .desc and
print it from the .matches function, with a slightly different message
when the command line is passed.

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > +{
> > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > +}
> > +
> >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> >  {
> >  	bool has_sre;
> > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> >  		.desc = "32-bit EL0 Support",
> >  		.capability = ARM64_HAS_32BIT_EL0,
> >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > -		.matches = has_cpuid_feature,
> > +		.matches = has_32bit_el0,
> 
> Ah, so this one reports 32-bit EL0 support even if no CPU actually
> supports 32-bit (passing the command line option on TX2 would come up
> with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> information elsewhere when have at least one CPU supporting this.

Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
then we would have 32-bit support, so I think this is an oddity that you
get when the command line is passed. That said, I could nobble .desc and
print it from the .matches function, with a slightly different message
when the command line is passed.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> +bool system_has_mismatched_32bit_el0(void)
> +{
> +	u64 reg;
> +	unsigned int fld;
> +
> +	if (!__allow_mismatched_32bit_el0)
> +		return false;
> +
> +	reg = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> +	fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64PFR0_EL0_SHIFT);
> +	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
> +}

Same here, this reports true even if no 32-bit is available (I have yet
to go through the other patches to see how this function is used).

-- 
Catalin

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> +bool system_has_mismatched_32bit_el0(void)
> +{
> +	u64 reg;
> +	unsigned int fld;
> +
> +	if (!__allow_mismatched_32bit_el0)
> +		return false;
> +
> +	reg = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> +	fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64PFR0_EL0_SHIFT);
> +	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
> +}

Same here, this reports true even if no 32-bit is available (I have yet
to go through the other patches to see how this function is used).

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 11:18:10AM +0000, Catalin Marinas wrote:
> On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > +bool system_has_mismatched_32bit_el0(void)
> > +{
> > +	u64 reg;
> > +	unsigned int fld;
> > +
> > +	if (!__allow_mismatched_32bit_el0)
> > +		return false;
> > +
> > +	reg = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> > +	fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64PFR0_EL0_SHIFT);
> > +	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
> > +}
> 
> Same here, this reports true even if no 32-bit is available (I have yet
> to go through the other patches to see how this function is used).

Same deal, really. We need to report true to deal with late-onlining.

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 11:18:10AM +0000, Catalin Marinas wrote:
> On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > +bool system_has_mismatched_32bit_el0(void)
> > +{
> > +	u64 reg;
> > +	unsigned int fld;
> > +
> > +	if (!__allow_mismatched_32bit_el0)
> > +		return false;
> > +
> > +	reg = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> > +	fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64PFR0_EL0_SHIFT);
> > +	return fld == ID_AA64PFR0_EL0_64BIT_ONLY;
> > +}
> 
> Same here, this reports true even if no 32-bit is available (I have yet
> to go through the other patches to see how this function is used).

Same deal, really. We need to report true to deal with late-onlining.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > +{
> > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > +}
> > > +
> > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > >  {
> > >  	bool has_sre;
> > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > >  		.desc = "32-bit EL0 Support",
> > >  		.capability = ARM64_HAS_32BIT_EL0,
> > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > -		.matches = has_cpuid_feature,
> > > +		.matches = has_32bit_el0,
> > 
> > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > supports 32-bit (passing the command line option on TX2 would come up
> > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > information elsewhere when have at least one CPU supporting this.
> 
> Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> then we would have 32-bit support, so I think this is an oddity that you
> get when the command line is passed. That said, I could nobble .desc and
> print it from the .matches function, with a slightly different message
> when the command line is passed.

I think we could do a pr_info_once() in update_32bit_cpu_features().

-- 
Catalin

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > +{
> > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > +}
> > > +
> > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > >  {
> > >  	bool has_sre;
> > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > >  		.desc = "32-bit EL0 Support",
> > >  		.capability = ARM64_HAS_32BIT_EL0,
> > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > -		.matches = has_cpuid_feature,
> > > +		.matches = has_32bit_el0,
> > 
> > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > supports 32-bit (passing the command line option on TX2 would come up
> > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > information elsewhere when have at least one CPU supporting this.
> 
> Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> then we would have 32-bit support, so I think this is an oddity that you
> get when the command line is passed. That said, I could nobble .desc and
> print it from the .matches function, with a slightly different message
> when the command line is passed.

I think we could do a pr_info_once() in update_32bit_cpu_features().

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > > +{
> > > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > > +}
> > > > +
> > > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > > >  {
> > > >  	bool has_sre;
> > > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > > >  		.desc = "32-bit EL0 Support",
> > > >  		.capability = ARM64_HAS_32BIT_EL0,
> > > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > > -		.matches = has_cpuid_feature,
> > > > +		.matches = has_32bit_el0,
> > > 
> > > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > > supports 32-bit (passing the command line option on TX2 would come up
> > > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > > information elsewhere when have at least one CPU supporting this.
> > 
> > Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> > then we would have 32-bit support, so I think this is an oddity that you
> > get when the command line is passed. That said, I could nobble .desc and
> > print it from the .matches function, with a slightly different message
> > when the command line is passed.
> 
> I think we could do a pr_info_once() in update_32bit_cpu_features().

Is that called on a system with one CPU?

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > > +{
> > > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > > +}
> > > > +
> > > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > > >  {
> > > >  	bool has_sre;
> > > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > > >  		.desc = "32-bit EL0 Support",
> > > >  		.capability = ARM64_HAS_32BIT_EL0,
> > > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > > -		.matches = has_cpuid_feature,
> > > > +		.matches = has_32bit_el0,
> > > 
> > > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > > supports 32-bit (passing the command line option on TX2 would come up
> > > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > > information elsewhere when have at least one CPU supporting this.
> > 
> > Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> > then we would have 32-bit support, so I think this is an oddity that you
> > get when the command line is passed. That said, I could nobble .desc and
> > print it from the .matches function, with a slightly different message
> > when the command line is passed.
> 
> I think we could do a pr_info_once() in update_32bit_cpu_features().

Is that called on a system with one CPU?

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > +{
> > > > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > > > +}
> > > > > +
> > > > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > > > >  {
> > > > >  	bool has_sre;
> > > > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > > > >  		.desc = "32-bit EL0 Support",
> > > > >  		.capability = ARM64_HAS_32BIT_EL0,
> > > > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > > > -		.matches = has_cpuid_feature,
> > > > > +		.matches = has_32bit_el0,
> > > > 
> > > > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > > > supports 32-bit (passing the command line option on TX2 would come up
> > > > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > > > information elsewhere when have at least one CPU supporting this.
> > > 
> > > Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> > > then we would have 32-bit support, so I think this is an oddity that you
> > > get when the command line is passed. That said, I could nobble .desc and
> > > print it from the .matches function, with a slightly different message
> > > when the command line is passed.
> > 
> > I think we could do a pr_info_once() in update_32bit_cpu_features().
> 
> Is that called on a system with one CPU?

Ah, it's not.

Anyway, I see your reasoning behind the late CPUs but I don't
particularly like abusing the cpufeature support to pretend a
SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
it in other cases, I haven't checked).

Could we not instead add a new feature for asymmetric support that's
defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
for late CPUs and we keep the system_supports_32bit_el0() unchanged.

-- 
Catalin

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > +{
> > > > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > > > +}
> > > > > +
> > > > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > > > >  {
> > > > >  	bool has_sre;
> > > > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > > > >  		.desc = "32-bit EL0 Support",
> > > > >  		.capability = ARM64_HAS_32BIT_EL0,
> > > > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > > > -		.matches = has_cpuid_feature,
> > > > > +		.matches = has_32bit_el0,
> > > > 
> > > > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > > > supports 32-bit (passing the command line option on TX2 would come up
> > > > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > > > information elsewhere when have at least one CPU supporting this.
> > > 
> > > Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> > > then we would have 32-bit support, so I think this is an oddity that you
> > > get when the command line is passed. That said, I could nobble .desc and
> > > print it from the .matches function, with a slightly different message
> > > when the command line is passed.
> > 
> > I think we could do a pr_info_once() in update_32bit_cpu_features().
> 
> Is that called on a system with one CPU?

Ah, it's not.

Anyway, I see your reasoning behind the late CPUs but I don't
particularly like abusing the cpufeature support to pretend a
SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
it in other cases, I haven't checked).

Could we not instead add a new feature for asymmetric support that's
defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
for late CPUs and we keep the system_supports_32bit_el0() unchanged.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 4/6] arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs

[Catalin Marinas]

On Tue, Oct 27, 2020 at 09:51:16PM +0000, Will Deacon wrote:
> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index 4784011cecac..c45b5f9dd66b 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -542,6 +542,17 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,
>  	write_sysreg(val, cntkctl_el1);
>  }
>  
> +static void compat_thread_switch(struct task_struct *next)
> +{
> +	if (!is_compat_thread(task_thread_info(next)))
> +		return;
> +
> +	if (!system_has_mismatched_32bit_el0())
> +		return;
> +
> +	set_tsk_thread_flag(next, TIF_NOTIFY_RESUME);
> +}

I wonder whether you could use set_notify_resume() for consistency.

-- 
Catalin

Re: [PATCH 4/6] arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs

[Catalin Marinas]

On Tue, Oct 27, 2020 at 09:51:16PM +0000, Will Deacon wrote:
> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index 4784011cecac..c45b5f9dd66b 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -542,6 +542,17 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,
>  	write_sysreg(val, cntkctl_el1);
>  }
>  
> +static void compat_thread_switch(struct task_struct *next)
> +{
> +	if (!is_compat_thread(task_thread_info(next)))
> +		return;
> +
> +	if (!system_has_mismatched_32bit_el0())
> +		return;
> +
> +	set_tsk_thread_flag(next, TIF_NOTIFY_RESUME);
> +}

I wonder whether you could use set_notify_resume() for consistency.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Catalin Marinas]

On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> index b555df825447..19893fb8e870 100644
> --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
>  		'identification' directory exposes the CPU ID registers for
>  		 identifying model and revision of the CPU.
>  
> +What:		/sys/devices/system/cpu/aarch32_el0

Nitpick: should we call this aarch32_el0_present? It's not exactly
present as we populate it as CPUs come online but it's closer to this
mask than to the online one.

> +Date:		October 2020
> +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> +Description:	Identifies the subset of CPUs in the system that can execute
> +		AArch32 (32-bit ARM) applications. If absent, then all or none
> +		of the CPUs can execute AArch32 applications and execve() will
> +		behave accordingly.

What does "accordingly" mean? Normally, we'd get ENOEXEC but here the
execve() "succeeds" followed by a SIGKILL if it ends up on the wrong
CPU.

-- 
Catalin

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Catalin Marinas]

On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> index b555df825447..19893fb8e870 100644
> --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
>  		'identification' directory exposes the CPU ID registers for
>  		 identifying model and revision of the CPU.
>  
> +What:		/sys/devices/system/cpu/aarch32_el0

Nitpick: should we call this aarch32_el0_present? It's not exactly
present as we populate it as CPUs come online but it's closer to this
mask than to the online one.

> +Date:		October 2020
> +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> +Description:	Identifies the subset of CPUs in the system that can execute
> +		AArch32 (32-bit ARM) applications. If absent, then all or none
> +		of the CPUs can execute AArch32 applications and execve() will
> +		behave accordingly.

What does "accordingly" mean? Normally, we'd get ENOEXEC but here the
execve() "succeeds" followed by a SIGKILL if it ends up on the wrong
CPU.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Will Deacon]

On Wed, Oct 28, 2020 at 12:15:07PM +0000, Catalin Marinas wrote:
> On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> > diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > index b555df825447..19893fb8e870 100644
> > --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> > +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
> >  		'identification' directory exposes the CPU ID registers for
> >  		 identifying model and revision of the CPU.
> >  
> > +What:		/sys/devices/system/cpu/aarch32_el0
> 
> Nitpick: should we call this aarch32_el0_present? It's not exactly
> present as we populate it as CPUs come online but it's closer to this
> mask than to the online one.

I don't think so, because a CPU could be set in this mask but not in the
present mask, which is hugely confusing it it has "present" in the name!

> > +Date:		October 2020
> > +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> > +Description:	Identifies the subset of CPUs in the system that can execute
> > +		AArch32 (32-bit ARM) applications. If absent, then all or none
> > +		of the CPUs can execute AArch32 applications and execve() will
> > +		behave accordingly.
> 
> What does "accordingly" mean? Normally, we'd get ENOEXEC but here the
> execve() "succeeds" followed by a SIGKILL if it ends up on the wrong
> CPU.

No; if the file is absent then execve() behaves as it always has.

Will

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Will Deacon]

On Wed, Oct 28, 2020 at 12:15:07PM +0000, Catalin Marinas wrote:
> On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> > diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > index b555df825447..19893fb8e870 100644
> > --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> > +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
> >  		'identification' directory exposes the CPU ID registers for
> >  		 identifying model and revision of the CPU.
> >  
> > +What:		/sys/devices/system/cpu/aarch32_el0
> 
> Nitpick: should we call this aarch32_el0_present? It's not exactly
> present as we populate it as CPUs come online but it's closer to this
> mask than to the online one.

I don't think so, because a CPU could be set in this mask but not in the
present mask, which is hugely confusing it it has "present" in the name!

> > +Date:		October 2020
> > +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> > +Description:	Identifies the subset of CPUs in the system that can execute
> > +		AArch32 (32-bit ARM) applications. If absent, then all or none
> > +		of the CPUs can execute AArch32 applications and execve() will
> > +		behave accordingly.
> 
> What does "accordingly" mean? Normally, we'd get ENOEXEC but here the
> execve() "succeeds" followed by a SIGKILL if it ends up on the wrong
> CPU.

No; if the file is absent then execve() behaves as it always has.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 4/6] arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs

[Will Deacon]

On Wed, Oct 28, 2020 at 12:10:53PM +0000, Catalin Marinas wrote:
> On Tue, Oct 27, 2020 at 09:51:16PM +0000, Will Deacon wrote:
> > diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> > index 4784011cecac..c45b5f9dd66b 100644
> > --- a/arch/arm64/kernel/process.c
> > +++ b/arch/arm64/kernel/process.c
> > @@ -542,6 +542,17 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,
> >  	write_sysreg(val, cntkctl_el1);
> >  }
> >  
> > +static void compat_thread_switch(struct task_struct *next)
> > +{
> > +	if (!is_compat_thread(task_thread_info(next)))
> > +		return;
> > +
> > +	if (!system_has_mismatched_32bit_el0())
> > +		return;
> > +
> > +	set_tsk_thread_flag(next, TIF_NOTIFY_RESUME);
> > +}
> 
> I wonder whether you could use set_notify_resume() for consistency.

I looked at that, but it ends up kicking the task which is not needed
here and can result in a reschedule via preempt_enable().

Will

Re: [PATCH 4/6] arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs

[Will Deacon]

On Wed, Oct 28, 2020 at 12:10:53PM +0000, Catalin Marinas wrote:
> On Tue, Oct 27, 2020 at 09:51:16PM +0000, Will Deacon wrote:
> > diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> > index 4784011cecac..c45b5f9dd66b 100644
> > --- a/arch/arm64/kernel/process.c
> > +++ b/arch/arm64/kernel/process.c
> > @@ -542,6 +542,17 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,
> >  	write_sysreg(val, cntkctl_el1);
> >  }
> >  
> > +static void compat_thread_switch(struct task_struct *next)
> > +{
> > +	if (!is_compat_thread(task_thread_info(next)))
> > +		return;
> > +
> > +	if (!system_has_mismatched_32bit_el0())
> > +		return;
> > +
> > +	set_tsk_thread_flag(next, TIF_NOTIFY_RESUME);
> > +}
> 
> I wonder whether you could use set_notify_resume() for consistency.

I looked at that, but it ends up kicking the task which is not needed
here and can result in a reschedule via preempt_enable().

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > > > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > > +{
> > > > > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > > > > +}
> > > > > > +
> > > > > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > >  {
> > > > > >  	bool has_sre;
> > > > > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > > > > >  		.desc = "32-bit EL0 Support",
> > > > > >  		.capability = ARM64_HAS_32BIT_EL0,
> > > > > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > > > > -		.matches = has_cpuid_feature,
> > > > > > +		.matches = has_32bit_el0,
> > > > > 
> > > > > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > > > > supports 32-bit (passing the command line option on TX2 would come up
> > > > > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > > > > information elsewhere when have at least one CPU supporting this.
> > > > 
> > > > Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> > > > then we would have 32-bit support, so I think this is an oddity that you
> > > > get when the command line is passed. That said, I could nobble .desc and
> > > > print it from the .matches function, with a slightly different message
> > > > when the command line is passed.
> > > 
> > > I think we could do a pr_info_once() in update_32bit_cpu_features().
> > 
> > Is that called on a system with one CPU?
> 
> Ah, it's not.
> 
> Anyway, I see your reasoning behind the late CPUs but I don't
> particularly like abusing the cpufeature support to pretend a
> SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> it in other cases, I haven't checked).

Hmm, but that's exactly what this cmdline option is about. We pretend that
the system has 32-bit EL0 when normally we would say that we don't.

> Could we not instead add a new feature for asymmetric support that's
> defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> for late CPUs and we keep the system_supports_32bit_el0() unchanged.

I really don't think this gains us anything. The current users of
system_supports_32bit_el0() are:

  - The ELF loader
  - CPU feature sanitisation code
  - Personality syscall
  - KVM

and, afaict, all of these would need to check the new feature if we added
it.  I think it would also mean that at least one 32-bit capable CPU would
have to boot early in order for the new feature to be advertised, which
feels like an artificial restriction to me, particularly as you could just
offline it immediately.

That said, I have dropped the print (see below) because the whole
"Detected" part is clearly bogus.

Will

--->8

diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 813362a91995..ac6aff3a69de 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1264,7 +1264,12 @@ device_initcall(aarch32_el0_sysfs_init);
 
 static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
 {
-       return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
+       if (has_cpuid_feature(entry, scope)) {
+               pr_info("detected: 32-bit EL0 Support\n");
+               return true;
+       }
+
+       return __allow_mismatched_32bit_el0;
 }
 
 static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
@@ -1874,7 +1879,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
        },
 #endif /* CONFIG_ARM64_VHE */
        {
-               .desc = "32-bit EL0 Support",
                .capability = ARM64_HAS_32BIT_EL0,
                .type = ARM64_CPUCAP_SYSTEM_FEATURE,
                .matches = has_32bit_el0,

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > > > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > > +{
> > > > > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > > > > +}
> > > > > > +
> > > > > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > >  {
> > > > > >  	bool has_sre;
> > > > > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > > > > >  		.desc = "32-bit EL0 Support",
> > > > > >  		.capability = ARM64_HAS_32BIT_EL0,
> > > > > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > > > > -		.matches = has_cpuid_feature,
> > > > > > +		.matches = has_32bit_el0,
> > > > > 
> > > > > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > > > > supports 32-bit (passing the command line option on TX2 would come up
> > > > > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > > > > information elsewhere when have at least one CPU supporting this.
> > > > 
> > > > Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> > > > then we would have 32-bit support, so I think this is an oddity that you
> > > > get when the command line is passed. That said, I could nobble .desc and
> > > > print it from the .matches function, with a slightly different message
> > > > when the command line is passed.
> > > 
> > > I think we could do a pr_info_once() in update_32bit_cpu_features().
> > 
> > Is that called on a system with one CPU?
> 
> Ah, it's not.
> 
> Anyway, I see your reasoning behind the late CPUs but I don't
> particularly like abusing the cpufeature support to pretend a
> SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> it in other cases, I haven't checked).

Hmm, but that's exactly what this cmdline option is about. We pretend that
the system has 32-bit EL0 when normally we would say that we don't.

> Could we not instead add a new feature for asymmetric support that's
> defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> for late CPUs and we keep the system_supports_32bit_el0() unchanged.

I really don't think this gains us anything. The current users of
system_supports_32bit_el0() are:

  - The ELF loader
  - CPU feature sanitisation code
  - Personality syscall
  - KVM

and, afaict, all of these would need to check the new feature if we added
it.  I think it would also mean that at least one 32-bit capable CPU would
have to boot early in order for the new feature to be advertised, which
feels like an artificial restriction to me, particularly as you could just
offline it immediately.

That said, I have dropped the print (see below) because the whole
"Detected" part is clearly bogus.

Will

--->8

diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 813362a91995..ac6aff3a69de 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1264,7 +1264,12 @@ device_initcall(aarch32_el0_sysfs_init);
 
 static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
 {
-       return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
+       if (has_cpuid_feature(entry, scope)) {
+               pr_info("detected: 32-bit EL0 Support\n");
+               return true;
+       }
+
+       return __allow_mismatched_32bit_el0;
 }
 
 static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
@@ -1874,7 +1879,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
        },
 #endif /* CONFIG_ARM64_VHE */
        {
-               .desc = "32-bit EL0 Support",
                .capability = ARM64_HAS_32BIT_EL0,
                .type = ARM64_CPUCAP_SYSTEM_FEATURE,
                .matches = has_32bit_el0,


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Catalin Marinas]

On Wed, Oct 28, 2020 at 12:27:59PM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 12:15:07PM +0000, Catalin Marinas wrote:
> > On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> > > diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > index b555df825447..19893fb8e870 100644
> > > --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
> > >  		'identification' directory exposes the CPU ID registers for
> > >  		 identifying model and revision of the CPU.
> > >  
> > > +What:		/sys/devices/system/cpu/aarch32_el0
> > 
> > Nitpick: should we call this aarch32_el0_present? It's not exactly
> > present as we populate it as CPUs come online but it's closer to this
> > mask than to the online one.
> 
> I don't think so, because a CPU could be set in this mask but not in the
> present mask, which is hugely confusing it it has "present" in the name!

How can it end up here but not in the present mask? We populate present
early if they have a corresponding DT entry.

> > > +Date:		October 2020
> > > +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> > > +Description:	Identifies the subset of CPUs in the system that can execute
> > > +		AArch32 (32-bit ARM) applications. If absent, then all or none
> > > +		of the CPUs can execute AArch32 applications and execve() will
> > > +		behave accordingly.
> > 
> > What does "accordingly" mean? Normally, we'd get ENOEXEC but here the
> > execve() "succeeds" followed by a SIGKILL if it ends up on the wrong
> > CPU.
> 
> No; if the file is absent then execve() behaves as it always has.

Ah, I missed the "absent" part and got confused.

-- 
Catalin

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Catalin Marinas]

On Wed, Oct 28, 2020 at 12:27:59PM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 12:15:07PM +0000, Catalin Marinas wrote:
> > On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> > > diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > index b555df825447..19893fb8e870 100644
> > > --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
> > >  		'identification' directory exposes the CPU ID registers for
> > >  		 identifying model and revision of the CPU.
> > >  
> > > +What:		/sys/devices/system/cpu/aarch32_el0
> > 
> > Nitpick: should we call this aarch32_el0_present? It's not exactly
> > present as we populate it as CPUs come online but it's closer to this
> > mask than to the online one.
> 
> I don't think so, because a CPU could be set in this mask but not in the
> present mask, which is hugely confusing it it has "present" in the name!

How can it end up here but not in the present mask? We populate present
early if they have a corresponding DT entry.

> > > +Date:		October 2020
> > > +Contact:	Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org>
> > > +Description:	Identifies the subset of CPUs in the system that can execute
> > > +		AArch32 (32-bit ARM) applications. If absent, then all or none
> > > +		of the CPUs can execute AArch32 applications and execve() will
> > > +		behave accordingly.
> > 
> > What does "accordingly" mean? Normally, we'd get ENOEXEC but here the
> > execve() "succeeds" followed by a SIGKILL if it ends up on the wrong
> > CPU.
> 
> No; if the file is absent then execve() behaves as it always has.

Ah, I missed the "absent" part and got confused.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Will Deacon]

On Wed, Oct 28, 2020 at 03:14:43PM +0000, Catalin Marinas wrote:
> On Wed, Oct 28, 2020 at 12:27:59PM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 12:15:07PM +0000, Catalin Marinas wrote:
> > > On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> > > > diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > > index b555df825447..19893fb8e870 100644
> > > > --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > > +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > > @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
> > > >  		'identification' directory exposes the CPU ID registers for
> > > >  		 identifying model and revision of the CPU.
> > > >  
> > > > +What:		/sys/devices/system/cpu/aarch32_el0
> > > 
> > > Nitpick: should we call this aarch32_el0_present? It's not exactly
> > > present as we populate it as CPUs come online but it's closer to this
> > > mask than to the online one.
> > 
> > I don't think so, because a CPU could be set in this mask but not in the
> > present mask, which is hugely confusing it it has "present" in the name!
> 
> How can it end up here but not in the present mask? We populate present
> early if they have a corresponding DT entry.

I was under the impression that physical CPU hotplug with ACPI would clear
the entry in the present mask, but I can't say I have any machines that I
can test that with and it looks like it might only be implemented for x86
at the moment.

That said, it looks like cpu_die_early() also marks CPUs as not being
present and this can happen due to a late capability conflict.

Will

Re: [PATCH 5/6] arm64: Advertise CPUs capable of running 32-bit applcations in sysfs

[Will Deacon]

On Wed, Oct 28, 2020 at 03:14:43PM +0000, Catalin Marinas wrote:
> On Wed, Oct 28, 2020 at 12:27:59PM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 12:15:07PM +0000, Catalin Marinas wrote:
> > > On Tue, Oct 27, 2020 at 09:51:17PM +0000, Will Deacon wrote:
> > > > diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > > index b555df825447..19893fb8e870 100644
> > > > --- a/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > > +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
> > > > @@ -472,6 +472,14 @@ Description:	AArch64 CPU registers
> > > >  		'identification' directory exposes the CPU ID registers for
> > > >  		 identifying model and revision of the CPU.
> > > >  
> > > > +What:		/sys/devices/system/cpu/aarch32_el0
> > > 
> > > Nitpick: should we call this aarch32_el0_present? It's not exactly
> > > present as we populate it as CPUs come online but it's closer to this
> > > mask than to the online one.
> > 
> > I don't think so, because a CPU could be set in this mask but not in the
> > present mask, which is hugely confusing it it has "present" in the name!
> 
> How can it end up here but not in the present mask? We populate present
> early if they have a corresponding DT entry.

I was under the impression that physical CPU hotplug with ACPI would clear
the entry in the present mask, but I can't say I have any machines that I
can test that with and it looks like it might only be implemented for x86
at the moment.

That said, it looks like cpu_die_early() also marks CPUs as not being
present and this can happen due to a late capability conflict.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Wed, Oct 28, 2020 at 12:40:49PM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> > On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > > > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > > > > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > > > +{
> > > > > > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > > > > > +}
> > > > > > > +
> > > > > > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > > >  {
> > > > > > >  	bool has_sre;
> > > > > > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > > > > > >  		.desc = "32-bit EL0 Support",
> > > > > > >  		.capability = ARM64_HAS_32BIT_EL0,
> > > > > > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > > > > > -		.matches = has_cpuid_feature,
> > > > > > > +		.matches = has_32bit_el0,
> > > > > > 
> > > > > > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > > > > > supports 32-bit (passing the command line option on TX2 would come up
> > > > > > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > > > > > information elsewhere when have at least one CPU supporting this.
> > > > > 
> > > > > Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> > > > > then we would have 32-bit support, so I think this is an oddity that you
> > > > > get when the command line is passed. That said, I could nobble .desc and
> > > > > print it from the .matches function, with a slightly different message
> > > > > when the command line is passed.
> > > > 
> > > > I think we could do a pr_info_once() in update_32bit_cpu_features().
> > > 
> > > Is that called on a system with one CPU?
> > 
> > Ah, it's not.
> > 
> > Anyway, I see your reasoning behind the late CPUs but I don't
> > particularly like abusing the cpufeature support to pretend a
> > SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> > it in other cases, I haven't checked).
> 
> Hmm, but that's exactly what this cmdline option is about. We pretend that
> the system has 32-bit EL0 when normally we would say that we don't.

So that's more about force-enabling 32-bit irrespective of whether any
CPU supports it (not just in the mismatched/asymmetric case). Of course,
if the aarch32_el0 mask is empty, the apps would get SIGKILL'ed.

> > Could we not instead add a new feature for asymmetric support that's
> > defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> > for late CPUs and we keep the system_supports_32bit_el0() unchanged.
> 
> I really don't think this gains us anything.

It saves us having to explain to someone passing this option on a TX2
why personality(PER_LINUX32) and even execve() appear to work (well,
until SIGKILL). The lscpu tool, for example, uses personality() to
display whether the CPUs support 32-bit.

Also with PER_LINUX32, /proc/cpuinfo shows the 32-bit HWCAPs. We have
compat_elf_hwcap pre-populated with some stuff which is entirely untrue
if AArch32 is missing.

Thinking about the COMPAT_HWCAPs, do we actually populate them properly
on an asymmetric system if the boot CPU is not AArch32-capable? In my
original patch I had to defer populating boot_cpu_data with AArch32
information until a capable CPU was found. If not,
update_32bit_cpu_features() will set most 32-bit features to 0.

> The current users of system_supports_32bit_el0() are:
> 
>   - The ELF loader
>   - CPU feature sanitisation code
>   - Personality syscall

There three need a relaxed system_supports_32bit_el0(), so we could
change it to check a new relaxed feature.

>   - KVM

Here I think we need the stronger guarantee, no 32-bit allowed in
guests (the original symmetric feature check).

> and, afaict, all of these would need to check the new feature if we added
> it.  I think it would also mean that at least one 32-bit capable CPU would
> have to boot early in order for the new feature to be advertised, which
> feels like an artificial restriction to me, particularly as you could just
> offline it immediately.

How strong requirement is to allow late CPUs here? I think we'd miss the
COMPAT_HWCAPs as we no longer populate them once user-space started,
they are actually setup via smp_cpus_done() -> setup_cpu_features().

-- 
Catalin

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Wed, Oct 28, 2020 at 12:40:49PM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> > On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > > > On Tue, Oct 27, 2020 at 09:51:14PM +0000, Will Deacon wrote:
> > > > > > > +static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > > > +{
> > > > > > > +	return has_cpuid_feature(entry, scope) || __allow_mismatched_32bit_el0;
> > > > > > > +}
> > > > > > > +
> > > > > > >  static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
> > > > > > >  {
> > > > > > >  	bool has_sre;
> > > > > > > @@ -1803,7 +1851,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> > > > > > >  		.desc = "32-bit EL0 Support",
> > > > > > >  		.capability = ARM64_HAS_32BIT_EL0,
> > > > > > >  		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> > > > > > > -		.matches = has_cpuid_feature,
> > > > > > > +		.matches = has_32bit_el0,
> > > > > > 
> > > > > > Ah, so this one reports 32-bit EL0 support even if no CPU actually
> > > > > > supports 32-bit (passing the command line option on TX2 would come up
> > > > > > with 32-bit EL0 in dmesg). I'd rather hide the .desc above and print the
> > > > > > information elsewhere when have at least one CPU supporting this.
> > > > > 
> > > > > Yeah, the problem is if a CPU with 32-bit EL0 support was late-onlined,
> > > > > then we would have 32-bit support, so I think this is an oddity that you
> > > > > get when the command line is passed. That said, I could nobble .desc and
> > > > > print it from the .matches function, with a slightly different message
> > > > > when the command line is passed.
> > > > 
> > > > I think we could do a pr_info_once() in update_32bit_cpu_features().
> > > 
> > > Is that called on a system with one CPU?
> > 
> > Ah, it's not.
> > 
> > Anyway, I see your reasoning behind the late CPUs but I don't
> > particularly like abusing the cpufeature support to pretend a
> > SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> > it in other cases, I haven't checked).
> 
> Hmm, but that's exactly what this cmdline option is about. We pretend that
> the system has 32-bit EL0 when normally we would say that we don't.

So that's more about force-enabling 32-bit irrespective of whether any
CPU supports it (not just in the mismatched/asymmetric case). Of course,
if the aarch32_el0 mask is empty, the apps would get SIGKILL'ed.

> > Could we not instead add a new feature for asymmetric support that's
> > defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> > for late CPUs and we keep the system_supports_32bit_el0() unchanged.
> 
> I really don't think this gains us anything.

It saves us having to explain to someone passing this option on a TX2
why personality(PER_LINUX32) and even execve() appear to work (well,
until SIGKILL). The lscpu tool, for example, uses personality() to
display whether the CPUs support 32-bit.

Also with PER_LINUX32, /proc/cpuinfo shows the 32-bit HWCAPs. We have
compat_elf_hwcap pre-populated with some stuff which is entirely untrue
if AArch32 is missing.

Thinking about the COMPAT_HWCAPs, do we actually populate them properly
on an asymmetric system if the boot CPU is not AArch32-capable? In my
original patch I had to defer populating boot_cpu_data with AArch32
information until a capable CPU was found. If not,
update_32bit_cpu_features() will set most 32-bit features to 0.

> The current users of system_supports_32bit_el0() are:
> 
>   - The ELF loader
>   - CPU feature sanitisation code
>   - Personality syscall

There three need a relaxed system_supports_32bit_el0(), so we could
change it to check a new relaxed feature.

>   - KVM

Here I think we need the stronger guarantee, no 32-bit allowed in
guests (the original symmetric feature check).

> and, afaict, all of these would need to check the new feature if we added
> it.  I think it would also mean that at least one 32-bit capable CPU would
> have to boot early in order for the new feature to be advertised, which
> feels like an artificial restriction to me, particularly as you could just
> offline it immediately.

How strong requirement is to allow late CPUs here? I think we'd miss the
COMPAT_HWCAPs as we no longer populate them once user-space started,
they are actually setup via smp_cpus_done() -> setup_cpu_features().

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Suren Baghdasaryan]

On Tue, Oct 27, 2020 at 2:51 PM Will Deacon <will@kernel.org> wrote:
>
> Hi all,
>
> I was playing around with the asymmetric AArch32 RFCv2 from Qais:
>
> https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
>
> and ended up writing my own implementation this afternoon. I think it's
> smaller, simpler and easier to work with. In particular:
>
>   * I got rid of the sysctl in favour of a plain cmdline parameter
>   * I don't have a new CPU capability
>   * I don't have a new thread flag
>   * I expose a cpumask to userspace via sysfs to identify the 32-bit CPUs
>
> Anyway, I don't think we should merge this stuff (other than the first patch)
> until we've figured out what's going on in Android, but I wanted to get
> this out as something which we might be able to build on.

Hi Will,
Thanks for posting this series. Just to provide some more background,
on Android, 64-bit apps are forked from zygote64 process and 32-bit
ones from zygote. So normally we could handle the issues with such
asymmetric architectures using cpuset cgroup and placing zygote
process (and consequently all its children) in a separate cgroup with
affinity mask that includes only 32-bit capable cores. We would have
to take care of the affinity mask for such tasks during task
migrations, but it's still doable from userspace. However there are
64-bit apps which fork 32-bit processes and that is the case which is
unclear how to handle without help from the kernel. Still discussing
possible solutions. CC'ing more people from Android to be in the loop.
Thanks,
Suren.



>
> Cheers,
>
> Will
>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Morten Rasmussen <morten.rasmussen@arm.com>
> Cc: Qais Yousef <qais.yousef@arm.com>
> Cc: Suren Baghdasaryan <surenb@google.com>
> Cc: kernel-team@android.com
>
> --->8
>
> Qais Yousef (1):
>   KVM: arm64: Handle Asymmetric AArch32 systems
>
> Will Deacon (5):
>   arm64: Allow mismatched 32-bit EL0 support
>   KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support
>   arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs
>   arm64: Advertise CPUs capable of running 32-bit applcations in sysfs
>   arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0
>
>  .../ABI/testing/sysfs-devices-system-cpu      |  8 ++
>  .../admin-guide/kernel-parameters.txt         |  7 ++
>  arch/arm64/include/asm/cpufeature.h           |  3 +
>  arch/arm64/kernel/cpufeature.c                | 80 ++++++++++++++++++-
>  arch/arm64/kernel/process.c                   | 21 ++++-
>  arch/arm64/kernel/signal.c                    | 26 ++++++
>  arch/arm64/kvm/arm.c                          | 27 +++++++
>  7 files changed, 168 insertions(+), 4 deletions(-)
>
> --
> 2.29.0.rc2.309.g374f81d7ae-goog
>

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Suren Baghdasaryan]

On Tue, Oct 27, 2020 at 2:51 PM Will Deacon <will@kernel.org> wrote:
>
> Hi all,
>
> I was playing around with the asymmetric AArch32 RFCv2 from Qais:
>
> https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
>
> and ended up writing my own implementation this afternoon. I think it's
> smaller, simpler and easier to work with. In particular:
>
>   * I got rid of the sysctl in favour of a plain cmdline parameter
>   * I don't have a new CPU capability
>   * I don't have a new thread flag
>   * I expose a cpumask to userspace via sysfs to identify the 32-bit CPUs
>
> Anyway, I don't think we should merge this stuff (other than the first patch)
> until we've figured out what's going on in Android, but I wanted to get
> this out as something which we might be able to build on.

Hi Will,
Thanks for posting this series. Just to provide some more background,
on Android, 64-bit apps are forked from zygote64 process and 32-bit
ones from zygote. So normally we could handle the issues with such
asymmetric architectures using cpuset cgroup and placing zygote
process (and consequently all its children) in a separate cgroup with
affinity mask that includes only 32-bit capable cores. We would have
to take care of the affinity mask for such tasks during task
migrations, but it's still doable from userspace. However there are
64-bit apps which fork 32-bit processes and that is the case which is
unclear how to handle without help from the kernel. Still discussing
possible solutions. CC'ing more people from Android to be in the loop.
Thanks,
Suren.



>
> Cheers,
>
> Will
>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Morten Rasmussen <morten.rasmussen@arm.com>
> Cc: Qais Yousef <qais.yousef@arm.com>
> Cc: Suren Baghdasaryan <surenb@google.com>
> Cc: kernel-team@android.com
>
> --->8
>
> Qais Yousef (1):
>   KVM: arm64: Handle Asymmetric AArch32 systems
>
> Will Deacon (5):
>   arm64: Allow mismatched 32-bit EL0 support
>   KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support
>   arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs
>   arm64: Advertise CPUs capable of running 32-bit applcations in sysfs
>   arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0
>
>  .../ABI/testing/sysfs-devices-system-cpu      |  8 ++
>  .../admin-guide/kernel-parameters.txt         |  7 ++
>  arch/arm64/include/asm/cpufeature.h           |  3 +
>  arch/arm64/kernel/cpufeature.c                | 80 ++++++++++++++++++-
>  arch/arm64/kernel/process.c                   | 21 ++++-
>  arch/arm64/kernel/signal.c                    | 26 ++++++
>  arch/arm64/kvm/arm.c                          | 27 +++++++
>  7 files changed, 168 insertions(+), 4 deletions(-)
>
> --
> 2.29.0.rc2.309.g374f81d7ae-goog
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Will Deacon]

Hi Suren,

On Thu, Oct 29, 2020 at 11:42:04AM -0700, Suren Baghdasaryan wrote:
> On Tue, Oct 27, 2020 at 2:51 PM Will Deacon <will@kernel.org> wrote:
> > I was playing around with the asymmetric AArch32 RFCv2 from Qais:
> >
> > https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
> >
> > and ended up writing my own implementation this afternoon. I think it's
> > smaller, simpler and easier to work with. In particular:
> >
> >   * I got rid of the sysctl in favour of a plain cmdline parameter
> >   * I don't have a new CPU capability
> >   * I don't have a new thread flag
> >   * I expose a cpumask to userspace via sysfs to identify the 32-bit CPUs
> >
> > Anyway, I don't think we should merge this stuff (other than the first patch)
> > until we've figured out what's going on in Android, but I wanted to get
> > this out as something which we might be able to build on.
> 
> Thanks for posting this series. Just to provide some more background,
> on Android, 64-bit apps are forked from zygote64 process and 32-bit
> ones from zygote. So normally we could handle the issues with such
> asymmetric architectures using cpuset cgroup and placing zygote
> process (and consequently all its children) in a separate cgroup with
> affinity mask that includes only 32-bit capable cores. We would have
> to take care of the affinity mask for such tasks during task
> migrations, but it's still doable from userspace. However there are
> 64-bit apps which fork 32-bit processes and that is the case which is
> unclear how to handle without help from the kernel. Still discussing
> possible solutions. CC'ing more people from Android to be in the loop.

Thanks for joining in and adding others. Perhaps one thing we could do
on top of this series is to restrict the affinity mask when execve()ing
a 32-bit application. The major problem I have with that is that it
goes directly against the man page for sched_setaffinity(), which states:

  | A child created via fork(2) inherits its parent's CPU affinity mask.
  | The affinity mask is preserved across an execve(2)

so there's a risk of regression if applications rely on the mask being
preserved.

Will

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Will Deacon]

Hi Suren,

On Thu, Oct 29, 2020 at 11:42:04AM -0700, Suren Baghdasaryan wrote:
> On Tue, Oct 27, 2020 at 2:51 PM Will Deacon <will@kernel.org> wrote:
> > I was playing around with the asymmetric AArch32 RFCv2 from Qais:
> >
> > https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
> >
> > and ended up writing my own implementation this afternoon. I think it's
> > smaller, simpler and easier to work with. In particular:
> >
> >   * I got rid of the sysctl in favour of a plain cmdline parameter
> >   * I don't have a new CPU capability
> >   * I don't have a new thread flag
> >   * I expose a cpumask to userspace via sysfs to identify the 32-bit CPUs
> >
> > Anyway, I don't think we should merge this stuff (other than the first patch)
> > until we've figured out what's going on in Android, but I wanted to get
> > this out as something which we might be able to build on.
> 
> Thanks for posting this series. Just to provide some more background,
> on Android, 64-bit apps are forked from zygote64 process and 32-bit
> ones from zygote. So normally we could handle the issues with such
> asymmetric architectures using cpuset cgroup and placing zygote
> process (and consequently all its children) in a separate cgroup with
> affinity mask that includes only 32-bit capable cores. We would have
> to take care of the affinity mask for such tasks during task
> migrations, but it's still doable from userspace. However there are
> 64-bit apps which fork 32-bit processes and that is the case which is
> unclear how to handle without help from the kernel. Still discussing
> possible solutions. CC'ing more people from Android to be in the loop.

Thanks for joining in and adding others. Perhaps one thing we could do
on top of this series is to restrict the affinity mask when execve()ing
a 32-bit application. The major problem I have with that is that it
goes directly against the man page for sched_setaffinity(), which states:

  | A child created via fork(2) inherits its parent's CPU affinity mask.
  | The affinity mask is preserved across an execve(2)

so there's a risk of regression if applications rely on the mask being
preserved.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 06:56:21PM +0000, Catalin Marinas wrote:
> On Wed, Oct 28, 2020 at 12:40:49PM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> > > On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > > > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > Anyway, I see your reasoning behind the late CPUs but I don't
> > > particularly like abusing the cpufeature support to pretend a
> > > SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> > > it in other cases, I haven't checked).
> > 
> > Hmm, but that's exactly what this cmdline option is about. We pretend that
> > the system has 32-bit EL0 when normally we would say that we don't.
> 
> So that's more about force-enabling 32-bit irrespective of whether any
> CPU supports it (not just in the mismatched/asymmetric case). Of course,
> if the aarch32_el0 mask is empty, the apps would get SIGKILL'ed.

Yes, but given that we can't generally rule out a 32-bit CPU coming online
late, I don't think we should pretend that we're in a position to detect
a 64-bit-only system.

> > > Could we not instead add a new feature for asymmetric support that's
> > > defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> > > for late CPUs and we keep the system_supports_32bit_el0() unchanged.
> > 
> > I really don't think this gains us anything.
> 
> It saves us having to explain to someone passing this option on a TX2
> why personality(PER_LINUX32) and even execve() appear to work (well,
> until SIGKILL). The lscpu tool, for example, uses personality() to
> display whether the CPUs support 32-bit.

It really doesn't save us having to explain what this option does: it will
need to be documented regardless. If you pass this option on TX2 then, yes,
you get access to the PER_LINUX32 personality because that's what this
option does. If you don't like that, don't pass the option! It's like
passing "nosmp" and being surprised that you only have one CPU online.

> Also with PER_LINUX32, /proc/cpuinfo shows the 32-bit HWCAPs. We have
> compat_elf_hwcap pre-populated with some stuff which is entirely untrue
> if AArch32 is missing.
> 
> Thinking about the COMPAT_HWCAPs, do we actually populate them properly
> on an asymmetric system if the boot CPU is not AArch32-capable? In my
> original patch I had to defer populating boot_cpu_data with AArch32
> information until a capable CPU was found. If not,
> update_32bit_cpu_features() will set most 32-bit features to 0.

I think there are two interesting aspects to the COMPAT_HWCAPS:

 1. Both my patches and the patches from Qais seem to get this wrong, so
    the set of reported 32-bit caps is too small. I'll look at fixing this
    for v2.

 2. The 32-bit hwcaps are exposed by the PER_LINUX32 personality, and so
    have to be fixed at boot *even if the boot CPUs are all 64-bit-only*.
    This means that if the first 32-bit-capable core is onlined late, then
    it will only get the base capabilities, but I think that's fine and
    consistent with our overall handling of hwcaps (which cannot appear
    dynamically to userspace).

> > The current users of system_supports_32bit_el0() are:
> > 
> >   - The ELF loader
> >   - CPU feature sanitisation code
> >   - Personality syscall
> 
> There three need a relaxed system_supports_32bit_el0(), so we could
> change it to check a new relaxed feature.
> 
> >   - KVM
> 
> Here I think we need the stronger guarantee, no 32-bit allowed in
> guests (the original symmetric feature check).

Right, and I handle this in these patches. But the point is really that
the majority of the users of system_supports_32bit_el0() are actually
interested in the asymmetric case and it doesn't make sense to call that out
separately. If it's the naming you object to, we could rename
system_supports_32bit_el0() to something else? Adding a new cap just adds
complexity that we don't need.

> > and, afaict, all of these would need to check the new feature if we added
> > it.  I think it would also mean that at least one 32-bit capable CPU would
> > have to boot early in order for the new feature to be advertised, which
> > feels like an artificial restriction to me, particularly as you could just
> > offline it immediately.
> 
> How strong requirement is to allow late CPUs here? I think we'd miss the
> COMPAT_HWCAPs as we no longer populate them once user-space started,
> they are actually setup via smp_cpus_done() -> setup_cpu_features().

I think the requirement is to be as consistent as possible and not introduce
more new behaviours than we really have to. Allowing late onlining of cores
but fixing the (compat) hwcaps during boot is fine.

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Oct 28, 2020 at 06:56:21PM +0000, Catalin Marinas wrote:
> On Wed, Oct 28, 2020 at 12:40:49PM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> > > On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > > > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > Anyway, I see your reasoning behind the late CPUs but I don't
> > > particularly like abusing the cpufeature support to pretend a
> > > SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> > > it in other cases, I haven't checked).
> > 
> > Hmm, but that's exactly what this cmdline option is about. We pretend that
> > the system has 32-bit EL0 when normally we would say that we don't.
> 
> So that's more about force-enabling 32-bit irrespective of whether any
> CPU supports it (not just in the mismatched/asymmetric case). Of course,
> if the aarch32_el0 mask is empty, the apps would get SIGKILL'ed.

Yes, but given that we can't generally rule out a 32-bit CPU coming online
late, I don't think we should pretend that we're in a position to detect
a 64-bit-only system.

> > > Could we not instead add a new feature for asymmetric support that's
> > > defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> > > for late CPUs and we keep the system_supports_32bit_el0() unchanged.
> > 
> > I really don't think this gains us anything.
> 
> It saves us having to explain to someone passing this option on a TX2
> why personality(PER_LINUX32) and even execve() appear to work (well,
> until SIGKILL). The lscpu tool, for example, uses personality() to
> display whether the CPUs support 32-bit.

It really doesn't save us having to explain what this option does: it will
need to be documented regardless. If you pass this option on TX2 then, yes,
you get access to the PER_LINUX32 personality because that's what this
option does. If you don't like that, don't pass the option! It's like
passing "nosmp" and being surprised that you only have one CPU online.

> Also with PER_LINUX32, /proc/cpuinfo shows the 32-bit HWCAPs. We have
> compat_elf_hwcap pre-populated with some stuff which is entirely untrue
> if AArch32 is missing.
> 
> Thinking about the COMPAT_HWCAPs, do we actually populate them properly
> on an asymmetric system if the boot CPU is not AArch32-capable? In my
> original patch I had to defer populating boot_cpu_data with AArch32
> information until a capable CPU was found. If not,
> update_32bit_cpu_features() will set most 32-bit features to 0.

I think there are two interesting aspects to the COMPAT_HWCAPS:

 1. Both my patches and the patches from Qais seem to get this wrong, so
    the set of reported 32-bit caps is too small. I'll look at fixing this
    for v2.

 2. The 32-bit hwcaps are exposed by the PER_LINUX32 personality, and so
    have to be fixed at boot *even if the boot CPUs are all 64-bit-only*.
    This means that if the first 32-bit-capable core is onlined late, then
    it will only get the base capabilities, but I think that's fine and
    consistent with our overall handling of hwcaps (which cannot appear
    dynamically to userspace).

> > The current users of system_supports_32bit_el0() are:
> > 
> >   - The ELF loader
> >   - CPU feature sanitisation code
> >   - Personality syscall
> 
> There three need a relaxed system_supports_32bit_el0(), so we could
> change it to check a new relaxed feature.
> 
> >   - KVM
> 
> Here I think we need the stronger guarantee, no 32-bit allowed in
> guests (the original symmetric feature check).

Right, and I handle this in these patches. But the point is really that
the majority of the users of system_supports_32bit_el0() are actually
interested in the asymmetric case and it doesn't make sense to call that out
separately. If it's the naming you object to, we could rename
system_supports_32bit_el0() to something else? Adding a new cap just adds
complexity that we don't need.

> > and, afaict, all of these would need to check the new feature if we added
> > it.  I think it would also mean that at least one 32-bit capable CPU would
> > have to boot early in order for the new feature to be advertised, which
> > feels like an artificial restriction to me, particularly as you could just
> > offline it immediately.
> 
> How strong requirement is to allow late CPUs here? I think we'd miss the
> COMPAT_HWCAPs as we no longer populate them once user-space started,
> they are actually setup via smp_cpus_done() -> setup_cpu_features().

I think the requirement is to be as consistent as possible and not introduce
more new behaviours than we really have to. Allowing late onlining of cores
but fixing the (compat) hwcaps during boot is fine.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 06:56:21PM +0000, Catalin Marinas wrote:
> > On Wed, Oct 28, 2020 at 12:40:49PM +0000, Will Deacon wrote:
> > > On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> > > > On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > > > > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > > > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > Anyway, I see your reasoning behind the late CPUs but I don't
> > > > particularly like abusing the cpufeature support to pretend a
> > > > SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> > > > it in other cases, I haven't checked).
> > > 
> > > Hmm, but that's exactly what this cmdline option is about. We pretend that
> > > the system has 32-bit EL0 when normally we would say that we don't.
> > 
> > So that's more about force-enabling 32-bit irrespective of whether any
> > CPU supports it (not just in the mismatched/asymmetric case). Of course,
> > if the aarch32_el0 mask is empty, the apps would get SIGKILL'ed.
> 
> Yes, but given that we can't generally rule out a 32-bit CPU coming online
> late, I don't think we should pretend that we're in a position to detect
> a 64-bit-only system.

No, we can't pretend it's a 64-bit only system but the kernel could
insist that 32-bit is not available if CPUs come in late. And since we
won't add late compat HWCAPs, I think allowing late 32-bit CPUs is a
pretty useless feature (see more below; of course, we allow those late
CPUs but they'd only run 64-bit apps).

> > > > Could we not instead add a new feature for asymmetric support that's
> > > > defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> > > > for late CPUs and we keep the system_supports_32bit_el0() unchanged.
> > > 
> > > I really don't think this gains us anything.
> > 
> > It saves us having to explain to someone passing this option on a TX2
> > why personality(PER_LINUX32) and even execve() appear to work (well,
> > until SIGKILL). The lscpu tool, for example, uses personality() to
> > display whether the CPUs support 32-bit.
> 
> It really doesn't save us having to explain what this option does: it will
> need to be documented regardless. If you pass this option on TX2 then, yes,
> you get access to the PER_LINUX32 personality because that's what this
> option does. If you don't like that, don't pass the option! It's like
> passing "nosmp" and being surprised that you only have one CPU online.

I see this analogy the other way around, you pass "allow_smp" on a UP
system and the kernel pretends there are more CPUs available, only that
the app gets SIGKILL'ed if it ends up on the fake core. Well, if it's
properly documented and people read it, we could live with this.

> > Also with PER_LINUX32, /proc/cpuinfo shows the 32-bit HWCAPs. We have
> > compat_elf_hwcap pre-populated with some stuff which is entirely untrue
> > if AArch32 is missing.
> > 
> > Thinking about the COMPAT_HWCAPs, do we actually populate them properly
> > on an asymmetric system if the boot CPU is not AArch32-capable? In my
> > original patch I had to defer populating boot_cpu_data with AArch32
> > information until a capable CPU was found. If not,
> > update_32bit_cpu_features() will set most 32-bit features to 0.
> 
> I think there are two interesting aspects to the COMPAT_HWCAPS:
> 
>  1. Both my patches and the patches from Qais seem to get this wrong, so
>     the set of reported 32-bit caps is too small. I'll look at fixing this
>     for v2.

I had it addressed in my original patch and Qais' first post on lakml.
It deferred filling in the 32-bit part of boot_cpu_data until the first
encounter of a 32-bit CPU.

I suggested we drop the cpuinfo_arm64.aarch32_valid member entirely but
I think we should have made it a static variable in cpuinfo_store_cpu()
or somewhere around there and still populate the 32-bit part of
boot_cpu_data.

>  2. The 32-bit hwcaps are exposed by the PER_LINUX32 personality, and so
>     have to be fixed at boot *even if the boot CPUs are all 64-bit-only*.

Note that the kernel rejects PER_LINUX32 if the system is 64-bit only.
With this option, arguably the 64-bit personality() ABI is slightly
changed. It doesn't matter much, only for lscpu reporting 32-bit
support when it's not there.

>     This means that if the first 32-bit-capable core is onlined late, then
>     it will only get the base capabilities, but I think that's fine and
>     consistent with our overall handling of hwcaps (which cannot appear
>     dynamically to userspace).

Yes but such bare 32-bit mode is entirely useless and I don't think we
should even pretend we have 32-bit. The compat hwcaps here would be
"half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
filled in. It's missing major bits like "vfp" and "neon" which are
necessary for the general purpose 32-bit EABI.

Yes, in theory you can run an applications that don't make use of FP but
are there such apps still around? Even worse, I think not having
HWCAP_VFP or HWCAP_NEON should lead to a SIGILL if executing such
instructions but I have a suspicion they'd work just fine. So our compat
ABI in this case is no longer consistent with what we'd expose on a
symmetric system.

> > > The current users of system_supports_32bit_el0() are:
> > > 
> > >   - The ELF loader
> > >   - CPU feature sanitisation code
> > >   - Personality syscall
> > 
> > There three need a relaxed system_supports_32bit_el0(), so we could
> > change it to check a new relaxed feature.
> > 
> > >   - KVM
> > 
> > Here I think we need the stronger guarantee, no 32-bit allowed in
> > guests (the original symmetric feature check).
> 
> Right, and I handle this in these patches. But the point is really that
> the majority of the users of system_supports_32bit_el0() are actually
> interested in the asymmetric case and it doesn't make sense to call that out
> separately. If it's the naming you object to, we could rename
> system_supports_32bit_el0() to something else? Adding a new cap just adds
> complexity that we don't need.

I think we can debate the naming once we conclude on the late-only
32-bit CPUs handling. With a new feature entry, we have more control on
whether to allow late 32-bit support.

> > > and, afaict, all of these would need to check the new feature if we added
> > > it.  I think it would also mean that at least one 32-bit capable CPU would
> > > have to boot early in order for the new feature to be advertised, which
> > > feels like an artificial restriction to me, particularly as you could just
> > > offline it immediately.
> > 
> > How strong requirement is to allow late CPUs here? I think we'd miss the
> > COMPAT_HWCAPs as we no longer populate them once user-space started,
> > they are actually setup via smp_cpus_done() -> setup_cpu_features().
> 
> I think the requirement is to be as consistent as possible and not introduce
> more new behaviours than we really have to. Allowing late onlining of cores
> but fixing the (compat) hwcaps during boot is fine.

As I said above, I think we would be even more inconsistent w.r.t.
HWCAPs if we require at least one early AArch32-capable CPU, otherwise
don't expose 32-bit at all. I don't see what we gain by allowing all
32-bit CPUs to come in late, other than maybe saving an entry in the
cpufeature array.

-- 
Catalin

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> On Wed, Oct 28, 2020 at 06:56:21PM +0000, Catalin Marinas wrote:
> > On Wed, Oct 28, 2020 at 12:40:49PM +0000, Will Deacon wrote:
> > > On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> > > > On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > > > > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > > > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > Anyway, I see your reasoning behind the late CPUs but I don't
> > > > particularly like abusing the cpufeature support to pretend a
> > > > SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> > > > it in other cases, I haven't checked).
> > > 
> > > Hmm, but that's exactly what this cmdline option is about. We pretend that
> > > the system has 32-bit EL0 when normally we would say that we don't.
> > 
> > So that's more about force-enabling 32-bit irrespective of whether any
> > CPU supports it (not just in the mismatched/asymmetric case). Of course,
> > if the aarch32_el0 mask is empty, the apps would get SIGKILL'ed.
> 
> Yes, but given that we can't generally rule out a 32-bit CPU coming online
> late, I don't think we should pretend that we're in a position to detect
> a 64-bit-only system.

No, we can't pretend it's a 64-bit only system but the kernel could
insist that 32-bit is not available if CPUs come in late. And since we
won't add late compat HWCAPs, I think allowing late 32-bit CPUs is a
pretty useless feature (see more below; of course, we allow those late
CPUs but they'd only run 64-bit apps).

> > > > Could we not instead add a new feature for asymmetric support that's
> > > > defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> > > > for late CPUs and we keep the system_supports_32bit_el0() unchanged.
> > > 
> > > I really don't think this gains us anything.
> > 
> > It saves us having to explain to someone passing this option on a TX2
> > why personality(PER_LINUX32) and even execve() appear to work (well,
> > until SIGKILL). The lscpu tool, for example, uses personality() to
> > display whether the CPUs support 32-bit.
> 
> It really doesn't save us having to explain what this option does: it will
> need to be documented regardless. If you pass this option on TX2 then, yes,
> you get access to the PER_LINUX32 personality because that's what this
> option does. If you don't like that, don't pass the option! It's like
> passing "nosmp" and being surprised that you only have one CPU online.

I see this analogy the other way around, you pass "allow_smp" on a UP
system and the kernel pretends there are more CPUs available, only that
the app gets SIGKILL'ed if it ends up on the fake core. Well, if it's
properly documented and people read it, we could live with this.

> > Also with PER_LINUX32, /proc/cpuinfo shows the 32-bit HWCAPs. We have
> > compat_elf_hwcap pre-populated with some stuff which is entirely untrue
> > if AArch32 is missing.
> > 
> > Thinking about the COMPAT_HWCAPs, do we actually populate them properly
> > on an asymmetric system if the boot CPU is not AArch32-capable? In my
> > original patch I had to defer populating boot_cpu_data with AArch32
> > information until a capable CPU was found. If not,
> > update_32bit_cpu_features() will set most 32-bit features to 0.
> 
> I think there are two interesting aspects to the COMPAT_HWCAPS:
> 
>  1. Both my patches and the patches from Qais seem to get this wrong, so
>     the set of reported 32-bit caps is too small. I'll look at fixing this
>     for v2.

I had it addressed in my original patch and Qais' first post on lakml.
It deferred filling in the 32-bit part of boot_cpu_data until the first
encounter of a 32-bit CPU.

I suggested we drop the cpuinfo_arm64.aarch32_valid member entirely but
I think we should have made it a static variable in cpuinfo_store_cpu()
or somewhere around there and still populate the 32-bit part of
boot_cpu_data.

>  2. The 32-bit hwcaps are exposed by the PER_LINUX32 personality, and so
>     have to be fixed at boot *even if the boot CPUs are all 64-bit-only*.

Note that the kernel rejects PER_LINUX32 if the system is 64-bit only.
With this option, arguably the 64-bit personality() ABI is slightly
changed. It doesn't matter much, only for lscpu reporting 32-bit
support when it's not there.

>     This means that if the first 32-bit-capable core is onlined late, then
>     it will only get the base capabilities, but I think that's fine and
>     consistent with our overall handling of hwcaps (which cannot appear
>     dynamically to userspace).

Yes but such bare 32-bit mode is entirely useless and I don't think we
should even pretend we have 32-bit. The compat hwcaps here would be
"half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
filled in. It's missing major bits like "vfp" and "neon" which are
necessary for the general purpose 32-bit EABI.

Yes, in theory you can run an applications that don't make use of FP but
are there such apps still around? Even worse, I think not having
HWCAP_VFP or HWCAP_NEON should lead to a SIGILL if executing such
instructions but I have a suspicion they'd work just fine. So our compat
ABI in this case is no longer consistent with what we'd expose on a
symmetric system.

> > > The current users of system_supports_32bit_el0() are:
> > > 
> > >   - The ELF loader
> > >   - CPU feature sanitisation code
> > >   - Personality syscall
> > 
> > There three need a relaxed system_supports_32bit_el0(), so we could
> > change it to check a new relaxed feature.
> > 
> > >   - KVM
> > 
> > Here I think we need the stronger guarantee, no 32-bit allowed in
> > guests (the original symmetric feature check).
> 
> Right, and I handle this in these patches. But the point is really that
> the majority of the users of system_supports_32bit_el0() are actually
> interested in the asymmetric case and it doesn't make sense to call that out
> separately. If it's the naming you object to, we could rename
> system_supports_32bit_el0() to something else? Adding a new cap just adds
> complexity that we don't need.

I think we can debate the naming once we conclude on the late-only
32-bit CPUs handling. With a new feature entry, we have more control on
whether to allow late 32-bit support.

> > > and, afaict, all of these would need to check the new feature if we added
> > > it.  I think it would also mean that at least one 32-bit capable CPU would
> > > have to boot early in order for the new feature to be advertised, which
> > > feels like an artificial restriction to me, particularly as you could just
> > > offline it immediately.
> > 
> > How strong requirement is to allow late CPUs here? I think we'd miss the
> > COMPAT_HWCAPs as we no longer populate them once user-space started,
> > they are actually setup via smp_cpus_done() -> setup_cpu_features().
> 
> I think the requirement is to be as consistent as possible and not introduce
> more new behaviours than we really have to. Allowing late onlining of cores
> but fixing the (compat) hwcaps during boot is fine.

As I said above, I think we would be even more inconsistent w.r.t.
HWCAPs if we require at least one early AArch32-capable CPU, otherwise
don't expose 32-bit at all. I don't see what we gain by allowing all
32-bit CPUs to come in late, other than maybe saving an entry in the
cpufeature array.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 06:56:21PM +0000, Catalin Marinas wrote:
> > > On Wed, Oct 28, 2020 at 12:40:49PM +0000, Will Deacon wrote:
> > > > On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> > > > > On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > > > > > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > > > > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > > > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > > Anyway, I see your reasoning behind the late CPUs but I don't
> > > > > particularly like abusing the cpufeature support to pretend a
> > > > > SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> > > > > it in other cases, I haven't checked).
> > > > 
> > > > Hmm, but that's exactly what this cmdline option is about. We pretend that
> > > > the system has 32-bit EL0 when normally we would say that we don't.
> > > 
> > > So that's more about force-enabling 32-bit irrespective of whether any
> > > CPU supports it (not just in the mismatched/asymmetric case). Of course,
> > > if the aarch32_el0 mask is empty, the apps would get SIGKILL'ed.
> > 
> > Yes, but given that we can't generally rule out a 32-bit CPU coming online
> > late, I don't think we should pretend that we're in a position to detect
> > a 64-bit-only system.
> 
> No, we can't pretend it's a 64-bit only system but the kernel could
> insist that 32-bit is not available if CPUs come in late. And since we
> won't add late compat HWCAPs, I think allowing late 32-bit CPUs is a
> pretty useless feature (see more below; of course, we allow those late
> CPUs but they'd only run 64-bit apps).

Why should we insist on this? It doesn't gain us anything in the kernel and
I don't think we should police userspace just because we can (the presence
of the command-line option is indicative that we should really be letting
userspace get on with it).

> > > > > Could we not instead add a new feature for asymmetric support that's
> > > > > defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> > > > > for late CPUs and we keep the system_supports_32bit_el0() unchanged.
> > > > 
> > > > I really don't think this gains us anything.
> > > 
> > > It saves us having to explain to someone passing this option on a TX2
> > > why personality(PER_LINUX32) and even execve() appear to work (well,
> > > until SIGKILL). The lscpu tool, for example, uses personality() to
> > > display whether the CPUs support 32-bit.
> > 
> > It really doesn't save us having to explain what this option does: it will
> > need to be documented regardless. If you pass this option on TX2 then, yes,
> > you get access to the PER_LINUX32 personality because that's what this
> > option does. If you don't like that, don't pass the option! It's like
> > passing "nosmp" and being surprised that you only have one CPU online.
> 
> I see this analogy the other way around, you pass "allow_smp" on a UP
> system and the kernel pretends there are more CPUs available, only that
> the app gets SIGKILL'ed if it ends up on the fake core. Well, if it's
> properly documented and people read it, we could live with this.

The bottom line is that passing command-line options can make things look
different and so we need to document what they do. I don't think we're
disagreeing here.

> > > Also with PER_LINUX32, /proc/cpuinfo shows the 32-bit HWCAPs. We have
> > > compat_elf_hwcap pre-populated with some stuff which is entirely untrue
> > > if AArch32 is missing.
> > > 
> > > Thinking about the COMPAT_HWCAPs, do we actually populate them properly
> > > on an asymmetric system if the boot CPU is not AArch32-capable? In my
> > > original patch I had to defer populating boot_cpu_data with AArch32
> > > information until a capable CPU was found. If not,
> > > update_32bit_cpu_features() will set most 32-bit features to 0.
> > 
> > I think there are two interesting aspects to the COMPAT_HWCAPS:
> > 
> >  1. Both my patches and the patches from Qais seem to get this wrong, so
> >     the set of reported 32-bit caps is too small. I'll look at fixing this
> >     for v2.
> 
> I had it addressed in my original patch and Qais' first post on lakml.
> It deferred filling in the 32-bit part of boot_cpu_data until the first
> encounter of a 32-bit CPU.
> 
> I suggested we drop the cpuinfo_arm64.aarch32_valid member entirely but
> I think we should have made it a static variable in cpuinfo_store_cpu()
> or somewhere around there and still populate the 32-bit part of
> boot_cpu_data.

OK, I haven't looked into the reason for this yet.

> >  2. The 32-bit hwcaps are exposed by the PER_LINUX32 personality, and so
> >     have to be fixed at boot *even if the boot CPUs are all 64-bit-only*.
> 
> Note that the kernel rejects PER_LINUX32 if the system is 64-bit only.
> With this option, arguably the 64-bit personality() ABI is slightly
> changed. It doesn't matter much, only for lscpu reporting 32-bit
> support when it's not there.

I think this is just a consequence of enabling this option.

> >     This means that if the first 32-bit-capable core is onlined late, then
> >     it will only get the base capabilities, but I think that's fine and
> >     consistent with our overall handling of hwcaps (which cannot appear
> >     dynamically to userspace).
> 
> Yes but such bare 32-bit mode is entirely useless and I don't think we
> should even pretend we have 32-bit. The compat hwcaps here would be
> "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> filled in. It's missing major bits like "vfp" and "neon" which are
> necessary for the general purpose 32-bit EABI.

So? If we found such a CPU during boot, would we refuse to online it because
we consider it "entirely useless"? No! That said, given that it's _very_
likely for the late CPUs to support vfp and neon, we could set those caps
speculatively if the 64-bit cores have fpsimd (late onlining would be
prevented for cores lacking those). Does the architecture allow you to
implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?

> Yes, in theory you can run an applications that don't make use of FP but
> are there such apps still around? Even worse, I think not having
> HWCAP_VFP or HWCAP_NEON should lead to a SIGILL if executing such
> instructions but I have a suspicion they'd work just fine. So our compat
> ABI in this case is no longer consistent with what we'd expose on a
> symmetric system.

In general, we already have this case for other capabilities where the
architecture doesn't provide us with controls to disable instructions at
EL0, so I don't think this is something new. However, given that we can
trap fp, then I agree that this should be consistent with the caps.

> > > How strong requirement is to allow late CPUs here? I think we'd miss the
> > > COMPAT_HWCAPs as we no longer populate them once user-space started,
> > > they are actually setup via smp_cpus_done() -> setup_cpu_features().
> > 
> > I think the requirement is to be as consistent as possible and not introduce
> > more new behaviours than we really have to. Allowing late onlining of cores
> > but fixing the (compat) hwcaps during boot is fine.
> 
> As I said above, I think we would be even more inconsistent w.r.t.
> HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> don't expose 32-bit at all. I don't see what we gain by allowing all
> 32-bit CPUs to come in late, other than maybe saving an entry in the
> cpufeature array.

It's a combination of there not being a good reason to prevent the
late-onlining and not gaining anything from the additional feature (I've
already shown why it doesn't help with the vast majority of callsites).

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > On Wed, Oct 28, 2020 at 06:56:21PM +0000, Catalin Marinas wrote:
> > > On Wed, Oct 28, 2020 at 12:40:49PM +0000, Will Deacon wrote:
> > > > On Wed, Oct 28, 2020 at 11:49:46AM +0000, Catalin Marinas wrote:
> > > > > On Wed, Oct 28, 2020 at 11:23:43AM +0000, Will Deacon wrote:
> > > > > > On Wed, Oct 28, 2020 at 11:22:06AM +0000, Catalin Marinas wrote:
> > > > > > > On Wed, Oct 28, 2020 at 11:17:13AM +0000, Will Deacon wrote:
> > > > > > > > On Wed, Oct 28, 2020 at 11:12:04AM +0000, Catalin Marinas wrote:
> > > > > Anyway, I see your reasoning behind the late CPUs but I don't
> > > > > particularly like abusing the cpufeature support to pretend a
> > > > > SYSTEM_FEATURE is available before knowing any CPU has it (maybe we do
> > > > > it in other cases, I haven't checked).
> > > > 
> > > > Hmm, but that's exactly what this cmdline option is about. We pretend that
> > > > the system has 32-bit EL0 when normally we would say that we don't.
> > > 
> > > So that's more about force-enabling 32-bit irrespective of whether any
> > > CPU supports it (not just in the mismatched/asymmetric case). Of course,
> > > if the aarch32_el0 mask is empty, the apps would get SIGKILL'ed.
> > 
> > Yes, but given that we can't generally rule out a 32-bit CPU coming online
> > late, I don't think we should pretend that we're in a position to detect
> > a 64-bit-only system.
> 
> No, we can't pretend it's a 64-bit only system but the kernel could
> insist that 32-bit is not available if CPUs come in late. And since we
> won't add late compat HWCAPs, I think allowing late 32-bit CPUs is a
> pretty useless feature (see more below; of course, we allow those late
> CPUs but they'd only run 64-bit apps).

Why should we insist on this? It doesn't gain us anything in the kernel and
I don't think we should police userspace just because we can (the presence
of the command-line option is indicative that we should really be letting
userspace get on with it).

> > > > > Could we not instead add a new feature for asymmetric support that's
> > > > > defined as ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE? This would be allowed
> > > > > for late CPUs and we keep the system_supports_32bit_el0() unchanged.
> > > > 
> > > > I really don't think this gains us anything.
> > > 
> > > It saves us having to explain to someone passing this option on a TX2
> > > why personality(PER_LINUX32) and even execve() appear to work (well,
> > > until SIGKILL). The lscpu tool, for example, uses personality() to
> > > display whether the CPUs support 32-bit.
> > 
> > It really doesn't save us having to explain what this option does: it will
> > need to be documented regardless. If you pass this option on TX2 then, yes,
> > you get access to the PER_LINUX32 personality because that's what this
> > option does. If you don't like that, don't pass the option! It's like
> > passing "nosmp" and being surprised that you only have one CPU online.
> 
> I see this analogy the other way around, you pass "allow_smp" on a UP
> system and the kernel pretends there are more CPUs available, only that
> the app gets SIGKILL'ed if it ends up on the fake core. Well, if it's
> properly documented and people read it, we could live with this.

The bottom line is that passing command-line options can make things look
different and so we need to document what they do. I don't think we're
disagreeing here.

> > > Also with PER_LINUX32, /proc/cpuinfo shows the 32-bit HWCAPs. We have
> > > compat_elf_hwcap pre-populated with some stuff which is entirely untrue
> > > if AArch32 is missing.
> > > 
> > > Thinking about the COMPAT_HWCAPs, do we actually populate them properly
> > > on an asymmetric system if the boot CPU is not AArch32-capable? In my
> > > original patch I had to defer populating boot_cpu_data with AArch32
> > > information until a capable CPU was found. If not,
> > > update_32bit_cpu_features() will set most 32-bit features to 0.
> > 
> > I think there are two interesting aspects to the COMPAT_HWCAPS:
> > 
> >  1. Both my patches and the patches from Qais seem to get this wrong, so
> >     the set of reported 32-bit caps is too small. I'll look at fixing this
> >     for v2.
> 
> I had it addressed in my original patch and Qais' first post on lakml.
> It deferred filling in the 32-bit part of boot_cpu_data until the first
> encounter of a 32-bit CPU.
> 
> I suggested we drop the cpuinfo_arm64.aarch32_valid member entirely but
> I think we should have made it a static variable in cpuinfo_store_cpu()
> or somewhere around there and still populate the 32-bit part of
> boot_cpu_data.

OK, I haven't looked into the reason for this yet.

> >  2. The 32-bit hwcaps are exposed by the PER_LINUX32 personality, and so
> >     have to be fixed at boot *even if the boot CPUs are all 64-bit-only*.
> 
> Note that the kernel rejects PER_LINUX32 if the system is 64-bit only.
> With this option, arguably the 64-bit personality() ABI is slightly
> changed. It doesn't matter much, only for lscpu reporting 32-bit
> support when it's not there.

I think this is just a consequence of enabling this option.

> >     This means that if the first 32-bit-capable core is onlined late, then
> >     it will only get the base capabilities, but I think that's fine and
> >     consistent with our overall handling of hwcaps (which cannot appear
> >     dynamically to userspace).
> 
> Yes but such bare 32-bit mode is entirely useless and I don't think we
> should even pretend we have 32-bit. The compat hwcaps here would be
> "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> filled in. It's missing major bits like "vfp" and "neon" which are
> necessary for the general purpose 32-bit EABI.

So? If we found such a CPU during boot, would we refuse to online it because
we consider it "entirely useless"? No! That said, given that it's _very_
likely for the late CPUs to support vfp and neon, we could set those caps
speculatively if the 64-bit cores have fpsimd (late onlining would be
prevented for cores lacking those). Does the architecture allow you to
implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?

> Yes, in theory you can run an applications that don't make use of FP but
> are there such apps still around? Even worse, I think not having
> HWCAP_VFP or HWCAP_NEON should lead to a SIGILL if executing such
> instructions but I have a suspicion they'd work just fine. So our compat
> ABI in this case is no longer consistent with what we'd expose on a
> symmetric system.

In general, we already have this case for other capabilities where the
architecture doesn't provide us with controls to disable instructions at
EL0, so I don't think this is something new. However, given that we can
trap fp, then I agree that this should be consistent with the caps.

> > > How strong requirement is to allow late CPUs here? I think we'd miss the
> > > COMPAT_HWCAPs as we no longer populate them once user-space started,
> > > they are actually setup via smp_cpus_done() -> setup_cpu_features().
> > 
> > I think the requirement is to be as consistent as possible and not introduce
> > more new behaviours than we really have to. Allowing late onlining of cores
> > but fixing the (compat) hwcaps during boot is fine.
> 
> As I said above, I think we would be even more inconsistent w.r.t.
> HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> don't expose 32-bit at all. I don't see what we gain by allowing all
> 32-bit CPUs to come in late, other than maybe saving an entry in the
> cpufeature array.

It's a combination of there not being a good reason to prevent the
late-onlining and not gaining anything from the additional feature (I've
already shown why it doesn't help with the vast majority of callsites).

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Marc Zyngier]

On Tue, 27 Oct 2020 21:51:12 +0000, Will Deacon wrote:
> I was playing around with the asymmetric AArch32 RFCv2 from Qais:
> 
> https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
> 
> and ended up writing my own implementation this afternoon. I think it's
> smaller, simpler and easier to work with. In particular:
> 
> [...]

Applied to next, thanks!

[1/1] KVM: arm64: Handle Asymmetric AArch32 systems
      commit: 22f553842b14a1289c088a79a67fb479d3fa2a4e

Cheers,

	M.
-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Marc Zyngier]

On Tue, 27 Oct 2020 21:51:12 +0000, Will Deacon wrote:
> I was playing around with the asymmetric AArch32 RFCv2 from Qais:
> 
> https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
> 
> and ended up writing my own implementation this afternoon. I think it's
> smaller, simpler and easier to work with. In particular:
> 
> [...]

Applied to next, thanks!

[1/1] KVM: arm64: Handle Asymmetric AArch32 systems
      commit: 22f553842b14a1289c088a79a67fb479d3fa2a4e

Cheers,

	M.
-- 
Without deviation from the norm, progress is not possible.



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Will Deacon]

On Fri, Oct 30, 2020 at 04:16:48PM +0000, Marc Zyngier wrote:
> On Tue, 27 Oct 2020 21:51:12 +0000, Will Deacon wrote:
> > I was playing around with the asymmetric AArch32 RFCv2 from Qais:
> > 
> > https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
> > 
> > and ended up writing my own implementation this afternoon. I think it's
> > smaller, simpler and easier to work with. In particular:
> > 
> > [...]
> 
> Applied to next, thanks!
> 
> [1/1] KVM: arm64: Handle Asymmetric AArch32 systems
>       commit: 22f553842b14a1289c088a79a67fb479d3fa2a4e

Got to be honest, this gave me a heart attack at first! First patch is
good though, just please don't apply the rest of this stuff :)

Will

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Will Deacon]

On Fri, Oct 30, 2020 at 04:16:48PM +0000, Marc Zyngier wrote:
> On Tue, 27 Oct 2020 21:51:12 +0000, Will Deacon wrote:
> > I was playing around with the asymmetric AArch32 RFCv2 from Qais:
> > 
> > https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
> > 
> > and ended up writing my own implementation this afternoon. I think it's
> > smaller, simpler and easier to work with. In particular:
> > 
> > [...]
> 
> Applied to next, thanks!
> 
> [1/1] KVM: arm64: Handle Asymmetric AArch32 systems
>       commit: 22f553842b14a1289c088a79a67fb479d3fa2a4e

Got to be honest, this gave me a heart attack at first! First patch is
good though, just please don't apply the rest of this stuff :)

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Marc Zyngier]

On 2020-10-30 16:24, Will Deacon wrote:
> On Fri, Oct 30, 2020 at 04:16:48PM +0000, Marc Zyngier wrote:
>> On Tue, 27 Oct 2020 21:51:12 +0000, Will Deacon wrote:
>> > I was playing around with the asymmetric AArch32 RFCv2 from Qais:
>> >
>> > https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
>> >
>> > and ended up writing my own implementation this afternoon. I think it's
>> > smaller, simpler and easier to work with. In particular:
>> >
>> > [...]
>> 
>> Applied to next, thanks!
>> 
>> [1/1] KVM: arm64: Handle Asymmetric AArch32 systems
>>       commit: 22f553842b14a1289c088a79a67fb479d3fa2a4e
> 
> Got to be honest, this gave me a heart attack at first! First patch is
> good though, just please don't apply the rest of this stuff :)

The only reason I took this patch is that it actually prevent these
silly configuration from being actively harmful. As for the rest,
I have no desire to touch them!

         M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 0/6] An alternative series for asymmetric AArch32 systems

[Marc Zyngier]

On 2020-10-30 16:24, Will Deacon wrote:
> On Fri, Oct 30, 2020 at 04:16:48PM +0000, Marc Zyngier wrote:
>> On Tue, 27 Oct 2020 21:51:12 +0000, Will Deacon wrote:
>> > I was playing around with the asymmetric AArch32 RFCv2 from Qais:
>> >
>> > https://lore.kernel.org/r/20201021104611.2744565-1-qais.yousef@arm.com
>> >
>> > and ended up writing my own implementation this afternoon. I think it's
>> > smaller, simpler and easier to work with. In particular:
>> >
>> > [...]
>> 
>> Applied to next, thanks!
>> 
>> [1/1] KVM: arm64: Handle Asymmetric AArch32 systems
>>       commit: 22f553842b14a1289c088a79a67fb479d3fa2a4e
> 
> Got to be honest, this gave me a heart attack at first! First patch is
> good though, just please don't apply the rest of this stuff :)

The only reason I took this patch is that it actually prevent these
silly configuration from being actively harmful. As for the rest,
I have no desire to touch them!

         M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > >     This means that if the first 32-bit-capable core is onlined late, then
> > >     it will only get the base capabilities, but I think that's fine and
> > >     consistent with our overall handling of hwcaps (which cannot appear
> > >     dynamically to userspace).
> > 
> > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > should even pretend we have 32-bit. The compat hwcaps here would be
> > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > filled in. It's missing major bits like "vfp" and "neon" which are
> > necessary for the general purpose 32-bit EABI.
> 
> So? If we found such a CPU during boot, would we refuse to online it because
> we consider it "entirely useless"? No!

We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
CPUs since we are not updating the compat hwcaps anyway (and that's
handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
places already).

> That said, given that it's _very_
> likely for the late CPUs to support vfp and neon, we could set those caps
> speculatively if the 64-bit cores have fpsimd (late onlining would be
> prevented for cores lacking those). Does the architecture allow you to
> implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?

Probably not but I don't want to butcher the cpufeature support further
and have compat hwcaps derived from ID_AA64* regs. I find this hack even
worse and I'd rather live with the partial hwcap information (and hope
user space doesn't read hwcaps anyway ;)).

I don't see why we should change this code further when the requirement
to the mobile vendors is to simply allow a 32-bit CPU to come up early.

> > As I said above, I think we would be even more inconsistent w.r.t.
> > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > don't expose 32-bit at all. I don't see what we gain by allowing all
> > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > cpufeature array.
> 
> It's a combination of there not being a good reason to prevent the
> late-onlining and not gaining anything from the additional feature (I've
> already shown why it doesn't help with the vast majority of callsites).

I underlined above, this is not about preventing late onlining, only
preventing late 32-bit support. Late AArch32-capable CPUs will be
onlined just fine, only that if we haven't got any prior 32-bit CPU, we
no longer report the feature and the sysfs mask.

All I'm asking is something along the lines of the diff below instead of
forcing ARM64_HAS_32BIT_EL0 always on (untested):

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 42868dbd29fd..f73631aeedae 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -65,7 +65,8 @@
 #define ARM64_HAS_ARMv8_4_TTL			55
 #define ARM64_HAS_TLB_RANGE			56
 #define ARM64_MTE				57
+#define ARM64_HAS_WEAK_32BIT_EL0		58
 
-#define ARM64_NCAPS				58
+#define ARM64_NCAPS				59
 
 #endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index f7e7144af174..f8da673a9a20 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -584,7 +584,16 @@ static inline bool cpu_supports_mixed_endian_el0(void)
 
 static inline bool system_supports_32bit_el0(void)
 {
-	return cpus_have_const_cap(ARM64_HAS_32BIT_EL0);
+	return __allow_mismatched_32bit_el0 ?
+		cpus_have_const_cap(ARM64_HAS_WEAK_32BIT_EL0) :
+		cpus_have_const_cap(ARM64_HAS_32BIT_EL0)
+}
+
+static inline bool system_has_mismatched_32bit_el0(void)
+{
+	return __allow_mismatched_32bit_el0 &&
+		cpus_have_const_cap(ARM64_HAS_WEAK_32BIT_EL0) &&
+		!cpus_have_const_cap(ARM64_HAS_32BIT_EL0)
 }
 
 static inline bool system_supports_4kb_granule(void)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index dcc165b3fc04..fd7554602c5e 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1809,6 +1809,15 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.field_pos = ID_AA64PFR0_EL0_SHIFT,
 		.min_field_value = ID_AA64PFR0_EL0_32BIT_64BIT,
 	},
+	{
+		.capability = ARM64_HAS_WEAK_32BIT_EL0,
+		.type = ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE,
+		.matches = has_cpuid_feature,
+		.sys_reg = SYS_ID_AA64PFR0_EL1,
+		.sign = FTR_UNSIGNED,
+		.field_pos = ID_AA64PFR0_EL0_SHIFT,
+		.min_field_value = ID_AA64PFR0_EL0_32BIT_64BIT,
+	},
 #ifdef CONFIG_KVM
 	{
 		.desc = "32-bit EL1 Support",

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > >     This means that if the first 32-bit-capable core is onlined late, then
> > >     it will only get the base capabilities, but I think that's fine and
> > >     consistent with our overall handling of hwcaps (which cannot appear
> > >     dynamically to userspace).
> > 
> > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > should even pretend we have 32-bit. The compat hwcaps here would be
> > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > filled in. It's missing major bits like "vfp" and "neon" which are
> > necessary for the general purpose 32-bit EABI.
> 
> So? If we found such a CPU during boot, would we refuse to online it because
> we consider it "entirely useless"? No!

We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
CPUs since we are not updating the compat hwcaps anyway (and that's
handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
places already).

> That said, given that it's _very_
> likely for the late CPUs to support vfp and neon, we could set those caps
> speculatively if the 64-bit cores have fpsimd (late onlining would be
> prevented for cores lacking those). Does the architecture allow you to
> implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?

Probably not but I don't want to butcher the cpufeature support further
and have compat hwcaps derived from ID_AA64* regs. I find this hack even
worse and I'd rather live with the partial hwcap information (and hope
user space doesn't read hwcaps anyway ;)).

I don't see why we should change this code further when the requirement
to the mobile vendors is to simply allow a 32-bit CPU to come up early.

> > As I said above, I think we would be even more inconsistent w.r.t.
> > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > don't expose 32-bit at all. I don't see what we gain by allowing all
> > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > cpufeature array.
> 
> It's a combination of there not being a good reason to prevent the
> late-onlining and not gaining anything from the additional feature (I've
> already shown why it doesn't help with the vast majority of callsites).

I underlined above, this is not about preventing late onlining, only
preventing late 32-bit support. Late AArch32-capable CPUs will be
onlined just fine, only that if we haven't got any prior 32-bit CPU, we
no longer report the feature and the sysfs mask.

All I'm asking is something along the lines of the diff below instead of
forcing ARM64_HAS_32BIT_EL0 always on (untested):

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 42868dbd29fd..f73631aeedae 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -65,7 +65,8 @@
 #define ARM64_HAS_ARMv8_4_TTL			55
 #define ARM64_HAS_TLB_RANGE			56
 #define ARM64_MTE				57
+#define ARM64_HAS_WEAK_32BIT_EL0		58
 
-#define ARM64_NCAPS				58
+#define ARM64_NCAPS				59
 
 #endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index f7e7144af174..f8da673a9a20 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -584,7 +584,16 @@ static inline bool cpu_supports_mixed_endian_el0(void)
 
 static inline bool system_supports_32bit_el0(void)
 {
-	return cpus_have_const_cap(ARM64_HAS_32BIT_EL0);
+	return __allow_mismatched_32bit_el0 ?
+		cpus_have_const_cap(ARM64_HAS_WEAK_32BIT_EL0) :
+		cpus_have_const_cap(ARM64_HAS_32BIT_EL0)
+}
+
+static inline bool system_has_mismatched_32bit_el0(void)
+{
+	return __allow_mismatched_32bit_el0 &&
+		cpus_have_const_cap(ARM64_HAS_WEAK_32BIT_EL0) &&
+		!cpus_have_const_cap(ARM64_HAS_32BIT_EL0)
 }
 
 static inline bool system_supports_4kb_granule(void)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index dcc165b3fc04..fd7554602c5e 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1809,6 +1809,15 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.field_pos = ID_AA64PFR0_EL0_SHIFT,
 		.min_field_value = ID_AA64PFR0_EL0_32BIT_64BIT,
 	},
+	{
+		.capability = ARM64_HAS_WEAK_32BIT_EL0,
+		.type = ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE,
+		.matches = has_cpuid_feature,
+		.sys_reg = SYS_ID_AA64PFR0_EL1,
+		.sign = FTR_UNSIGNED,
+		.field_pos = ID_AA64PFR0_EL0_SHIFT,
+		.min_field_value = ID_AA64PFR0_EL0_32BIT_64BIT,
+	},
 #ifdef CONFIG_KVM
 	{
 		.desc = "32-bit EL1 Support",

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Mon, Nov 02, 2020 at 11:44:45AM +0000, Catalin Marinas wrote:
> On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> > On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > > >     This means that if the first 32-bit-capable core is onlined late, then
> > > >     it will only get the base capabilities, but I think that's fine and
> > > >     consistent with our overall handling of hwcaps (which cannot appear
> > > >     dynamically to userspace).
> > > 
> > > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > > should even pretend we have 32-bit. The compat hwcaps here would be
> > > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > > filled in. It's missing major bits like "vfp" and "neon" which are
> > > necessary for the general purpose 32-bit EABI.
> > 
> > So? If we found such a CPU during boot, would we refuse to online it because
> > we consider it "entirely useless"? No!
> 
> We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
> CPUs since we are not updating the compat hwcaps anyway (and that's
> handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
> places already).
> 
> > That said, given that it's _very_
> > likely for the late CPUs to support vfp and neon, we could set those caps
> > speculatively if the 64-bit cores have fpsimd (late onlining would be
> > prevented for cores lacking those). Does the architecture allow you to
> > implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?
> 
> Probably not but I don't want to butcher the cpufeature support further
> and have compat hwcaps derived from ID_AA64* regs. I find this hack even
> worse and I'd rather live with the partial hwcap information (and hope
> user space doesn't read hwcaps anyway ;)).
> 
> I don't see why we should change this code further when the requirement
> to the mobile vendors is to simply allow a 32-bit CPU to come up early.
> 
> > > As I said above, I think we would be even more inconsistent w.r.t.
> > > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > > don't expose 32-bit at all. I don't see what we gain by allowing all
> > > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > > cpufeature array.
> > 
> > It's a combination of there not being a good reason to prevent the
> > late-onlining and not gaining anything from the additional feature (I've
> > already shown why it doesn't help with the vast majority of callsites).
> 
> I underlined above, this is not about preventing late onlining, only
> preventing late 32-bit support. Late AArch32-capable CPUs will be
> onlined just fine, only that if we haven't got any prior 32-bit CPU, we
> no longer report the feature and the sysfs mask.

Ok. Then we're in agreement about not preventing late-onlining. The problem
then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
your diff, we still have an issue if you boot on the CPUs that support
32-bit and then try to online a 64-bit-only core (it will fail).

So I think we do need my changes to the existing cap, but perhaps we
could return false from system_supports_32bit_el0() until we've actually
seen a 32-bit capable core. That way you would keep the existing behaviour
on TX2, and we wouldn't get any unusual late-onlining failures.

I've hacked something together that seems to work, so I'll clean it up and
post it tomorrow. I've spotted a couple of pre-existing issues at the same
time, so I need to fix those first (WEAK_LOCAL_CPU_FEATURE doesn't set the
cap for late CPUs and failed onlining causes RCU stalls).

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Mon, Nov 02, 2020 at 11:44:45AM +0000, Catalin Marinas wrote:
> On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> > On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > > >     This means that if the first 32-bit-capable core is onlined late, then
> > > >     it will only get the base capabilities, but I think that's fine and
> > > >     consistent with our overall handling of hwcaps (which cannot appear
> > > >     dynamically to userspace).
> > > 
> > > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > > should even pretend we have 32-bit. The compat hwcaps here would be
> > > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > > filled in. It's missing major bits like "vfp" and "neon" which are
> > > necessary for the general purpose 32-bit EABI.
> > 
> > So? If we found such a CPU during boot, would we refuse to online it because
> > we consider it "entirely useless"? No!
> 
> We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
> CPUs since we are not updating the compat hwcaps anyway (and that's
> handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
> places already).
> 
> > That said, given that it's _very_
> > likely for the late CPUs to support vfp and neon, we could set those caps
> > speculatively if the 64-bit cores have fpsimd (late onlining would be
> > prevented for cores lacking those). Does the architecture allow you to
> > implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?
> 
> Probably not but I don't want to butcher the cpufeature support further
> and have compat hwcaps derived from ID_AA64* regs. I find this hack even
> worse and I'd rather live with the partial hwcap information (and hope
> user space doesn't read hwcaps anyway ;)).
> 
> I don't see why we should change this code further when the requirement
> to the mobile vendors is to simply allow a 32-bit CPU to come up early.
> 
> > > As I said above, I think we would be even more inconsistent w.r.t.
> > > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > > don't expose 32-bit at all. I don't see what we gain by allowing all
> > > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > > cpufeature array.
> > 
> > It's a combination of there not being a good reason to prevent the
> > late-onlining and not gaining anything from the additional feature (I've
> > already shown why it doesn't help with the vast majority of callsites).
> 
> I underlined above, this is not about preventing late onlining, only
> preventing late 32-bit support. Late AArch32-capable CPUs will be
> onlined just fine, only that if we haven't got any prior 32-bit CPU, we
> no longer report the feature and the sysfs mask.

Ok. Then we're in agreement about not preventing late-onlining. The problem
then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
your diff, we still have an issue if you boot on the CPUs that support
32-bit and then try to online a 64-bit-only core (it will fail).

So I think we do need my changes to the existing cap, but perhaps we
could return false from system_supports_32bit_el0() until we've actually
seen a 32-bit capable core. That way you would keep the existing behaviour
on TX2, and we wouldn't get any unusual late-onlining failures.

I've hacked something together that seems to work, so I'll clean it up and
post it tomorrow. I've spotted a couple of pre-existing issues at the same
time, so I need to fix those first (WEAK_LOCAL_CPU_FEATURE doesn't set the
cap for late CPUs and failed onlining causes RCU stalls).

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

Hi Will

On 11/05/20 21:38, Will Deacon wrote:
> On Mon, Nov 02, 2020 at 11:44:45AM +0000, Catalin Marinas wrote:
> > On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> > > On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > > > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > > > >     This means that if the first 32-bit-capable core is onlined late, then
> > > > >     it will only get the base capabilities, but I think that's fine and
> > > > >     consistent with our overall handling of hwcaps (which cannot appear
> > > > >     dynamically to userspace).
> > > > 
> > > > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > > > should even pretend we have 32-bit. The compat hwcaps here would be
> > > > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > > > filled in. It's missing major bits like "vfp" and "neon" which are
> > > > necessary for the general purpose 32-bit EABI.
> > > 
> > > So? If we found such a CPU during boot, would we refuse to online it because
> > > we consider it "entirely useless"? No!
> > 
> > We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
> > CPUs since we are not updating the compat hwcaps anyway (and that's
> > handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
> > places already).
> > 
> > > That said, given that it's _very_
> > > likely for the late CPUs to support vfp and neon, we could set those caps
> > > speculatively if the 64-bit cores have fpsimd (late onlining would be
> > > prevented for cores lacking those). Does the architecture allow you to
> > > implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?
> > 
> > Probably not but I don't want to butcher the cpufeature support further
> > and have compat hwcaps derived from ID_AA64* regs. I find this hack even
> > worse and I'd rather live with the partial hwcap information (and hope
> > user space doesn't read hwcaps anyway ;)).
> > 
> > I don't see why we should change this code further when the requirement
> > to the mobile vendors is to simply allow a 32-bit CPU to come up early.
> > 
> > > > As I said above, I think we would be even more inconsistent w.r.t.
> > > > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > > > don't expose 32-bit at all. I don't see what we gain by allowing all
> > > > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > > > cpufeature array.
> > > 
> > > It's a combination of there not being a good reason to prevent the
> > > late-onlining and not gaining anything from the additional feature (I've
> > > already shown why it doesn't help with the vast majority of callsites).
> > 
> > I underlined above, this is not about preventing late onlining, only
> > preventing late 32-bit support. Late AArch32-capable CPUs will be
> > onlined just fine, only that if we haven't got any prior 32-bit CPU, we
> > no longer report the feature and the sysfs mask.
> 
> Ok. Then we're in agreement about not preventing late-onlining. The problem
> then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
> your diff, we still have an issue if you boot on the CPUs that support
> 32-bit and then try to online a 64-bit-only core (it will fail).
> 
> So I think we do need my changes to the existing cap, but perhaps we
> could return false from system_supports_32bit_el0() until we've actually
> seen a 32-bit capable core. That way you would keep the existing behaviour
> on TX2, and we wouldn't get any unusual late-onlining failures.
> 
> I've hacked something together that seems to work, so I'll clean it up and
> post it tomorrow. I've spotted a couple of pre-existing issues at the same
> time, so I need to fix those first (WEAK_LOCAL_CPU_FEATURE doesn't set the
> cap for late CPUs and failed onlining causes RCU stalls).

FWIW I have my v3 over here in case it's of any help. It solves the problem of
HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
with 32bit features then.

	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3

I abandoned the sysctl because HWCAP are boot time discoverable and it seemed
not worth the headache to do the switch at run time for that. So except for
patch 1, the remaining patches look similar to yours.

	linux32 cat /proc/cpuinfo

return the right features if booted with the right cmdline option; where cpu0
is not 32bit capable.

Thanks

--
Qais Yousef

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

Hi Will

On 11/05/20 21:38, Will Deacon wrote:
> On Mon, Nov 02, 2020 at 11:44:45AM +0000, Catalin Marinas wrote:
> > On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> > > On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > > > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > > > >     This means that if the first 32-bit-capable core is onlined late, then
> > > > >     it will only get the base capabilities, but I think that's fine and
> > > > >     consistent with our overall handling of hwcaps (which cannot appear
> > > > >     dynamically to userspace).
> > > > 
> > > > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > > > should even pretend we have 32-bit. The compat hwcaps here would be
> > > > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > > > filled in. It's missing major bits like "vfp" and "neon" which are
> > > > necessary for the general purpose 32-bit EABI.
> > > 
> > > So? If we found such a CPU during boot, would we refuse to online it because
> > > we consider it "entirely useless"? No!
> > 
> > We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
> > CPUs since we are not updating the compat hwcaps anyway (and that's
> > handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
> > places already).
> > 
> > > That said, given that it's _very_
> > > likely for the late CPUs to support vfp and neon, we could set those caps
> > > speculatively if the 64-bit cores have fpsimd (late onlining would be
> > > prevented for cores lacking those). Does the architecture allow you to
> > > implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?
> > 
> > Probably not but I don't want to butcher the cpufeature support further
> > and have compat hwcaps derived from ID_AA64* regs. I find this hack even
> > worse and I'd rather live with the partial hwcap information (and hope
> > user space doesn't read hwcaps anyway ;)).
> > 
> > I don't see why we should change this code further when the requirement
> > to the mobile vendors is to simply allow a 32-bit CPU to come up early.
> > 
> > > > As I said above, I think we would be even more inconsistent w.r.t.
> > > > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > > > don't expose 32-bit at all. I don't see what we gain by allowing all
> > > > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > > > cpufeature array.
> > > 
> > > It's a combination of there not being a good reason to prevent the
> > > late-onlining and not gaining anything from the additional feature (I've
> > > already shown why it doesn't help with the vast majority of callsites).
> > 
> > I underlined above, this is not about preventing late onlining, only
> > preventing late 32-bit support. Late AArch32-capable CPUs will be
> > onlined just fine, only that if we haven't got any prior 32-bit CPU, we
> > no longer report the feature and the sysfs mask.
> 
> Ok. Then we're in agreement about not preventing late-onlining. The problem
> then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
> your diff, we still have an issue if you boot on the CPUs that support
> 32-bit and then try to online a 64-bit-only core (it will fail).
> 
> So I think we do need my changes to the existing cap, but perhaps we
> could return false from system_supports_32bit_el0() until we've actually
> seen a 32-bit capable core. That way you would keep the existing behaviour
> on TX2, and we wouldn't get any unusual late-onlining failures.
> 
> I've hacked something together that seems to work, so I'll clean it up and
> post it tomorrow. I've spotted a couple of pre-existing issues at the same
> time, so I need to fix those first (WEAK_LOCAL_CPU_FEATURE doesn't set the
> cap for late CPUs and failed onlining causes RCU stalls).

FWIW I have my v3 over here in case it's of any help. It solves the problem of
HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
with 32bit features then.

	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3

I abandoned the sysctl because HWCAP are boot time discoverable and it seemed
not worth the headache to do the switch at run time for that. So except for
patch 1, the remaining patches look similar to yours.

	linux32 cat /proc/cpuinfo

return the right features if booted with the right cmdline option; where cpu0
is not 32bit capable.

Thanks

--
Qais Yousef

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> Hi Will
> 
> On 11/05/20 21:38, Will Deacon wrote:
> > On Mon, Nov 02, 2020 at 11:44:45AM +0000, Catalin Marinas wrote:
> > > On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> > > > On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > > > > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > > > > >     This means that if the first 32-bit-capable core is onlined late, then
> > > > > >     it will only get the base capabilities, but I think that's fine and
> > > > > >     consistent with our overall handling of hwcaps (which cannot appear
> > > > > >     dynamically to userspace).
> > > > > 
> > > > > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > > > > should even pretend we have 32-bit. The compat hwcaps here would be
> > > > > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > > > > filled in. It's missing major bits like "vfp" and "neon" which are
> > > > > necessary for the general purpose 32-bit EABI.
> > > > 
> > > > So? If we found such a CPU during boot, would we refuse to online it because
> > > > we consider it "entirely useless"? No!
> > > 
> > > We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
> > > CPUs since we are not updating the compat hwcaps anyway (and that's
> > > handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
> > > places already).
> > > 
> > > > That said, given that it's _very_
> > > > likely for the late CPUs to support vfp and neon, we could set those caps
> > > > speculatively if the 64-bit cores have fpsimd (late onlining would be
> > > > prevented for cores lacking those). Does the architecture allow you to
> > > > implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?
> > > 
> > > Probably not but I don't want to butcher the cpufeature support further
> > > and have compat hwcaps derived from ID_AA64* regs. I find this hack even
> > > worse and I'd rather live with the partial hwcap information (and hope
> > > user space doesn't read hwcaps anyway ;)).
> > > 
> > > I don't see why we should change this code further when the requirement
> > > to the mobile vendors is to simply allow a 32-bit CPU to come up early.
> > > 
> > > > > As I said above, I think we would be even more inconsistent w.r.t.
> > > > > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > > > > don't expose 32-bit at all. I don't see what we gain by allowing all
> > > > > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > > > > cpufeature array.
> > > > 
> > > > It's a combination of there not being a good reason to prevent the
> > > > late-onlining and not gaining anything from the additional feature (I've
> > > > already shown why it doesn't help with the vast majority of callsites).
> > > 
> > > I underlined above, this is not about preventing late onlining, only
> > > preventing late 32-bit support. Late AArch32-capable CPUs will be
> > > onlined just fine, only that if we haven't got any prior 32-bit CPU, we
> > > no longer report the feature and the sysfs mask.
> > 
> > Ok. Then we're in agreement about not preventing late-onlining. The problem
> > then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
> > your diff, we still have an issue if you boot on the CPUs that support
> > 32-bit and then try to online a 64-bit-only core (it will fail).
> > 
> > So I think we do need my changes to the existing cap, but perhaps we
> > could return false from system_supports_32bit_el0() until we've actually
> > seen a 32-bit capable core. That way you would keep the existing behaviour
> > on TX2, and we wouldn't get any unusual late-onlining failures.
> > 
> > I've hacked something together that seems to work, so I'll clean it up and
> > post it tomorrow. I've spotted a couple of pre-existing issues at the same
> > time, so I need to fix those first (WEAK_LOCAL_CPU_FEATURE doesn't set the
> > cap for late CPUs and failed onlining causes RCU stalls).
> 
> FWIW I have my v3 over here in case it's of any help. It solves the problem of
> HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> with 32bit features then.
> 
> 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3

Cheers, I've done something similar. I was hoping to post it today, but I've
been side-tracked with bug fixing this morning. The main headache I ended up
with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
are 32-bit capable. What do you do in that case?

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> Hi Will
> 
> On 11/05/20 21:38, Will Deacon wrote:
> > On Mon, Nov 02, 2020 at 11:44:45AM +0000, Catalin Marinas wrote:
> > > On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> > > > On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > > > > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > > > > >     This means that if the first 32-bit-capable core is onlined late, then
> > > > > >     it will only get the base capabilities, but I think that's fine and
> > > > > >     consistent with our overall handling of hwcaps (which cannot appear
> > > > > >     dynamically to userspace).
> > > > > 
> > > > > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > > > > should even pretend we have 32-bit. The compat hwcaps here would be
> > > > > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > > > > filled in. It's missing major bits like "vfp" and "neon" which are
> > > > > necessary for the general purpose 32-bit EABI.
> > > > 
> > > > So? If we found such a CPU during boot, would we refuse to online it because
> > > > we consider it "entirely useless"? No!
> > > 
> > > We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
> > > CPUs since we are not updating the compat hwcaps anyway (and that's
> > > handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
> > > places already).
> > > 
> > > > That said, given that it's _very_
> > > > likely for the late CPUs to support vfp and neon, we could set those caps
> > > > speculatively if the 64-bit cores have fpsimd (late onlining would be
> > > > prevented for cores lacking those). Does the architecture allow you to
> > > > implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?
> > > 
> > > Probably not but I don't want to butcher the cpufeature support further
> > > and have compat hwcaps derived from ID_AA64* regs. I find this hack even
> > > worse and I'd rather live with the partial hwcap information (and hope
> > > user space doesn't read hwcaps anyway ;)).
> > > 
> > > I don't see why we should change this code further when the requirement
> > > to the mobile vendors is to simply allow a 32-bit CPU to come up early.
> > > 
> > > > > As I said above, I think we would be even more inconsistent w.r.t.
> > > > > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > > > > don't expose 32-bit at all. I don't see what we gain by allowing all
> > > > > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > > > > cpufeature array.
> > > > 
> > > > It's a combination of there not being a good reason to prevent the
> > > > late-onlining and not gaining anything from the additional feature (I've
> > > > already shown why it doesn't help with the vast majority of callsites).
> > > 
> > > I underlined above, this is not about preventing late onlining, only
> > > preventing late 32-bit support. Late AArch32-capable CPUs will be
> > > onlined just fine, only that if we haven't got any prior 32-bit CPU, we
> > > no longer report the feature and the sysfs mask.
> > 
> > Ok. Then we're in agreement about not preventing late-onlining. The problem
> > then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
> > your diff, we still have an issue if you boot on the CPUs that support
> > 32-bit and then try to online a 64-bit-only core (it will fail).
> > 
> > So I think we do need my changes to the existing cap, but perhaps we
> > could return false from system_supports_32bit_el0() until we've actually
> > seen a 32-bit capable core. That way you would keep the existing behaviour
> > on TX2, and we wouldn't get any unusual late-onlining failures.
> > 
> > I've hacked something together that seems to work, so I'll clean it up and
> > post it tomorrow. I've spotted a couple of pre-existing issues at the same
> > time, so I need to fix those first (WEAK_LOCAL_CPU_FEATURE doesn't set the
> > cap for late CPUs and failed onlining causes RCU stalls).
> 
> FWIW I have my v3 over here in case it's of any help. It solves the problem of
> HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> with 32bit features then.
> 
> 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3

Cheers, I've done something similar. I was hoping to post it today, but I've
been side-tracked with bug fixing this morning. The main headache I ended up
with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
are 32-bit capable. What do you do in that case?

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Thu, Nov 05, 2020 at 09:38:46PM +0000, Will Deacon wrote:
> Ok. Then we're in agreement about not preventing late-onlining. The problem
> then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
> your diff, we still have an issue if you boot on the CPUs that support
> 32-bit and then try to online a 64-bit-only core (it will fail).

Ah, I focussed too much on the 32-bit capable CPUs coming up late. In my
original hack, I made the capability weak based on the config option.
Here we want to make it weak based on cmdline but that structure is
const (we could remove the const though).

> So I think we do need my changes to the existing cap, but perhaps we
> could return false from system_supports_32bit_el0() until we've actually
> seen a 32-bit capable core. That way you would keep the existing behaviour
> on TX2, and we wouldn't get any unusual late-onlining failures.

If we see the first 32-bit capable core late, we may report it's
available but no proper hwcaps.

We could do a combination of a new weak feature together with your
always-on 32-bit feature when forced by the cmdline. So the system would
support 32-bit if both the system feature (with the detection override)
and the asym one are set.

However, I think it may be simpler if we made the current feature weak
(so no new one) together with a bool somewhere that tells us if we found
a CPU that doesn't have 32-bit (asym mode). system_supports_32bit_el0()
would check if the cap is set together with (!asym_found ||
asym_allowed).

-- 
Catalin

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Catalin Marinas]

On Thu, Nov 05, 2020 at 09:38:46PM +0000, Will Deacon wrote:
> Ok. Then we're in agreement about not preventing late-onlining. The problem
> then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
> your diff, we still have an issue if you boot on the CPUs that support
> 32-bit and then try to online a 64-bit-only core (it will fail).

Ah, I focussed too much on the 32-bit capable CPUs coming up late. In my
original hack, I made the capability weak based on the config option.
Here we want to make it weak based on cmdline but that structure is
const (we could remove the const though).

> So I think we do need my changes to the existing cap, but perhaps we
> could return false from system_supports_32bit_el0() until we've actually
> seen a 32-bit capable core. That way you would keep the existing behaviour
> on TX2, and we wouldn't get any unusual late-onlining failures.

If we see the first 32-bit capable core late, we may report it's
available but no proper hwcaps.

We could do a combination of a new weak feature together with your
always-on 32-bit feature when forced by the cmdline. So the system would
support 32-bit if both the system feature (with the detection override)
and the asym one are set.

However, I think it may be simpler if we made the current feature weak
(so no new one) together with a bool somewhere that tells us if we found
a CPU that doesn't have 32-bit (asym mode). system_supports_32bit_el0()
would check if the cap is set together with (!asym_found ||
asym_allowed).

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/06/20 13:00, Will Deacon wrote:
> On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > Hi Will
> > 
> > On 11/05/20 21:38, Will Deacon wrote:
> > > On Mon, Nov 02, 2020 at 11:44:45AM +0000, Catalin Marinas wrote:
> > > > On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> > > > > On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > > > > > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > > > > > >     This means that if the first 32-bit-capable core is onlined late, then
> > > > > > >     it will only get the base capabilities, but I think that's fine and
> > > > > > >     consistent with our overall handling of hwcaps (which cannot appear
> > > > > > >     dynamically to userspace).
> > > > > > 
> > > > > > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > > > > > should even pretend we have 32-bit. The compat hwcaps here would be
> > > > > > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > > > > > filled in. It's missing major bits like "vfp" and "neon" which are
> > > > > > necessary for the general purpose 32-bit EABI.
> > > > > 
> > > > > So? If we found such a CPU during boot, would we refuse to online it because
> > > > > we consider it "entirely useless"? No!
> > > > 
> > > > We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
> > > > CPUs since we are not updating the compat hwcaps anyway (and that's
> > > > handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
> > > > places already).
> > > > 
> > > > > That said, given that it's _very_
> > > > > likely for the late CPUs to support vfp and neon, we could set those caps
> > > > > speculatively if the 64-bit cores have fpsimd (late onlining would be
> > > > > prevented for cores lacking those). Does the architecture allow you to
> > > > > implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?
> > > > 
> > > > Probably not but I don't want to butcher the cpufeature support further
> > > > and have compat hwcaps derived from ID_AA64* regs. I find this hack even
> > > > worse and I'd rather live with the partial hwcap information (and hope
> > > > user space doesn't read hwcaps anyway ;)).
> > > > 
> > > > I don't see why we should change this code further when the requirement
> > > > to the mobile vendors is to simply allow a 32-bit CPU to come up early.
> > > > 
> > > > > > As I said above, I think we would be even more inconsistent w.r.t.
> > > > > > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > > > > > don't expose 32-bit at all. I don't see what we gain by allowing all
> > > > > > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > > > > > cpufeature array.
> > > > > 
> > > > > It's a combination of there not being a good reason to prevent the
> > > > > late-onlining and not gaining anything from the additional feature (I've
> > > > > already shown why it doesn't help with the vast majority of callsites).
> > > > 
> > > > I underlined above, this is not about preventing late onlining, only
> > > > preventing late 32-bit support. Late AArch32-capable CPUs will be
> > > > onlined just fine, only that if we haven't got any prior 32-bit CPU, we
> > > > no longer report the feature and the sysfs mask.
> > > 
> > > Ok. Then we're in agreement about not preventing late-onlining. The problem
> > > then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
> > > your diff, we still have an issue if you boot on the CPUs that support
> > > 32-bit and then try to online a 64-bit-only core (it will fail).
> > > 
> > > So I think we do need my changes to the existing cap, but perhaps we
> > > could return false from system_supports_32bit_el0() until we've actually
> > > seen a 32-bit capable core. That way you would keep the existing behaviour
> > > on TX2, and we wouldn't get any unusual late-onlining failures.
> > > 
> > > I've hacked something together that seems to work, so I'll clean it up and
> > > post it tomorrow. I've spotted a couple of pre-existing issues at the same
> > > time, so I need to fix those first (WEAK_LOCAL_CPU_FEATURE doesn't set the
> > > cap for late CPUs and failed onlining causes RCU stalls).
> > 
> > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > with 32bit features then.
> > 
> > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> 
> Cheers, I've done something similar. I was hoping to post it today, but I've
> been side-tracked with bug fixing this morning. The main headache I ended up
> with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> are 32-bit capable. What do you do in that case?

Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
attempt to bring the remaining 64bit-only cpus online later?

Haven't tried that tbh. What symptoms do you expect to see? I can try it out.
I'm off for the remainder of the day, but can spend few mins to run an
experiment for sure.

Thanks

--
Qais Yousef

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/06/20 13:00, Will Deacon wrote:
> On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > Hi Will
> > 
> > On 11/05/20 21:38, Will Deacon wrote:
> > > On Mon, Nov 02, 2020 at 11:44:45AM +0000, Catalin Marinas wrote:
> > > > On Fri, Oct 30, 2020 at 04:13:53PM +0000, Will Deacon wrote:
> > > > > On Fri, Oct 30, 2020 at 11:18:47AM +0000, Catalin Marinas wrote:
> > > > > > On Thu, Oct 29, 2020 at 10:20:48PM +0000, Will Deacon wrote:
> > > > > > >     This means that if the first 32-bit-capable core is onlined late, then
> > > > > > >     it will only get the base capabilities, but I think that's fine and
> > > > > > >     consistent with our overall handling of hwcaps (which cannot appear
> > > > > > >     dynamically to userspace).
> > > > > > 
> > > > > > Yes but such bare 32-bit mode is entirely useless and I don't think we
> > > > > > should even pretend we have 32-bit. The compat hwcaps here would be
> > > > > > "half thumb fastmult edsp tls idiva idivt lpae evtstrm", statically
> > > > > > filled in. It's missing major bits like "vfp" and "neon" which are
> > > > > > necessary for the general purpose 32-bit EABI.
> > > > > 
> > > > > So? If we found such a CPU during boot, would we refuse to online it because
> > > > > we consider it "entirely useless"? No!
> > > > 
> > > > We _do_ online it but as a 64-bit only CPU if there were no early 32-bit
> > > > CPUs since we are not updating the compat hwcaps anyway (and that's
> > > > handled automatically by WEAK_LOCAL_CPU_FEATURE; we do this in a few
> > > > places already).
> > > > 
> > > > > That said, given that it's _very_
> > > > > likely for the late CPUs to support vfp and neon, we could set those caps
> > > > > speculatively if the 64-bit cores have fpsimd (late onlining would be
> > > > > prevented for cores lacking those). Does the architecture allow you to
> > > > > implement both AArch64 and AArch32 at EL0, but only have fpsimd for AArch64?
> > > > 
> > > > Probably not but I don't want to butcher the cpufeature support further
> > > > and have compat hwcaps derived from ID_AA64* regs. I find this hack even
> > > > worse and I'd rather live with the partial hwcap information (and hope
> > > > user space doesn't read hwcaps anyway ;)).
> > > > 
> > > > I don't see why we should change this code further when the requirement
> > > > to the mobile vendors is to simply allow a 32-bit CPU to come up early.
> > > > 
> > > > > > As I said above, I think we would be even more inconsistent w.r.t.
> > > > > > HWCAPs if we require at least one early AArch32-capable CPU, otherwise
> > > > > > don't expose 32-bit at all. I don't see what we gain by allowing all
> > > > > > 32-bit CPUs to come in late, other than maybe saving an entry in the
> > > > > > cpufeature array.
> > > > > 
> > > > > It's a combination of there not being a good reason to prevent the
> > > > > late-onlining and not gaining anything from the additional feature (I've
> > > > > already shown why it doesn't help with the vast majority of callsites).
> > > > 
> > > > I underlined above, this is not about preventing late onlining, only
> > > > preventing late 32-bit support. Late AArch32-capable CPUs will be
> > > > onlined just fine, only that if we haven't got any prior 32-bit CPU, we
> > > > no longer report the feature and the sysfs mask.
> > > 
> > > Ok. Then we're in agreement about not preventing late-onlining. The problem
> > > then is that the existing 32-bit EL0 capability is a SYSTEM cap so even with
> > > your diff, we still have an issue if you boot on the CPUs that support
> > > 32-bit and then try to online a 64-bit-only core (it will fail).
> > > 
> > > So I think we do need my changes to the existing cap, but perhaps we
> > > could return false from system_supports_32bit_el0() until we've actually
> > > seen a 32-bit capable core. That way you would keep the existing behaviour
> > > on TX2, and we wouldn't get any unusual late-onlining failures.
> > > 
> > > I've hacked something together that seems to work, so I'll clean it up and
> > > post it tomorrow. I've spotted a couple of pre-existing issues at the same
> > > time, so I need to fix those first (WEAK_LOCAL_CPU_FEATURE doesn't set the
> > > cap for late CPUs and failed onlining causes RCU stalls).
> > 
> > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > with 32bit features then.
> > 
> > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> 
> Cheers, I've done something similar. I was hoping to post it today, but I've
> been side-tracked with bug fixing this morning. The main headache I ended up
> with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> are 32-bit capable. What do you do in that case?

Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
attempt to bring the remaining 64bit-only cpus online later?

Haven't tried that tbh. What symptoms do you expect to see? I can try it out.
I'm off for the remainder of the day, but can spend few mins to run an
experiment for sure.

Thanks

--
Qais Yousef

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> On 11/06/20 13:00, Will Deacon wrote:
> > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > with 32bit features then.
> > > 
> > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > 
> > Cheers, I've done something similar. I was hoping to post it today, but I've
> > been side-tracked with bug fixing this morning. The main headache I ended up
> > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > are 32-bit capable. What do you do in that case?
> 
> Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> attempt to bring the remaining 64bit-only cpus online later?

Right. I think we will refuse to online them. I'll post my attempt at
handling that shortly.

> Haven't tried that tbh. What symptoms do you expect to see? I can try it out.
> I'm off for the remainder of the day, but can spend few mins to run an
> experiment for sure.

No probs; I've been taking Friday afternoons off to burn holiday anyway, so
you didn't miss anything!

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> On 11/06/20 13:00, Will Deacon wrote:
> > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > with 32bit features then.
> > > 
> > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > 
> > Cheers, I've done something similar. I was hoping to post it today, but I've
> > been side-tracked with bug fixing this morning. The main headache I ended up
> > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > are 32-bit capable. What do you do in that case?
> 
> Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> attempt to bring the remaining 64bit-only cpus online later?

Right. I think we will refuse to online them. I'll post my attempt at
handling that shortly.

> Haven't tried that tbh. What symptoms do you expect to see? I can try it out.
> I'm off for the remainder of the day, but can spend few mins to run an
> experiment for sure.

No probs; I've been taking Friday afternoons off to burn holiday anyway, so
you didn't miss anything!

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/09/20 13:52, Will Deacon wrote:
> On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > On 11/06/20 13:00, Will Deacon wrote:
> > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > with 32bit features then.
> > > > 
> > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > 
> > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > are 32-bit capable. What do you do in that case?
> > 
> > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > attempt to bring the remaining 64bit-only cpus online later?
> 
> Right. I think we will refuse to online them. I'll post my attempt at
> handling that shortly.

Sorry for the delayed response.

You're right, I tried that and they refuse to come online. We missed that tbh.

Haven't thought what we should do yet. I tried your v2 and it failed similarly.

I usually have a similar hunk in my testing to check how the kernel perceives
the 32bit support when I execute a binary:

	diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
	index f447d313a9c5..a9549e55a6c8 100644
	--- a/arch/arm64/include/asm/cpufeature.h
	+++ b/arch/arm64/include/asm/cpufeature.h
	@@ -611,6 +611,9 @@ static inline bool system_supports_32bit_el0(void)
	 {
		u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);

	+       pr_err("System supports symmetric 32bit el0: %d\n", id_aa64pfr0_32bit_el0(pfr0));
	+       pr_err("System supports Asymmetric 32bit el0: %ld\n", static_branch_unlikely(&arm64_mismatched_32bit_el0));
	+
		return id_aa64pfr0_32bit_el0(pfr0) ||
		       static_branch_unlikely(&arm64_mismatched_32bit_el0);
	 }

In your v2 both conditions are true. In my series we see the system as
symmetric if we boot the 32bit capable cpus _only_.

> > Haven't tried that tbh. What symptoms do you expect to see? I can try it out.
> > I'm off for the remainder of the day, but can spend few mins to run an
> > experiment for sure.
> 
> No probs; I've been taking Friday afternoons off to burn holiday anyway, so
> you didn't miss anything!

:-)

Thanks

--
Qais Yousef

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/09/20 13:52, Will Deacon wrote:
> On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > On 11/06/20 13:00, Will Deacon wrote:
> > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > with 32bit features then.
> > > > 
> > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > 
> > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > are 32-bit capable. What do you do in that case?
> > 
> > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > attempt to bring the remaining 64bit-only cpus online later?
> 
> Right. I think we will refuse to online them. I'll post my attempt at
> handling that shortly.

Sorry for the delayed response.

You're right, I tried that and they refuse to come online. We missed that tbh.

Haven't thought what we should do yet. I tried your v2 and it failed similarly.

I usually have a similar hunk in my testing to check how the kernel perceives
the 32bit support when I execute a binary:

	diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
	index f447d313a9c5..a9549e55a6c8 100644
	--- a/arch/arm64/include/asm/cpufeature.h
	+++ b/arch/arm64/include/asm/cpufeature.h
	@@ -611,6 +611,9 @@ static inline bool system_supports_32bit_el0(void)
	 {
		u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);

	+       pr_err("System supports symmetric 32bit el0: %d\n", id_aa64pfr0_32bit_el0(pfr0));
	+       pr_err("System supports Asymmetric 32bit el0: %ld\n", static_branch_unlikely(&arm64_mismatched_32bit_el0));
	+
		return id_aa64pfr0_32bit_el0(pfr0) ||
		       static_branch_unlikely(&arm64_mismatched_32bit_el0);
	 }

In your v2 both conditions are true. In my series we see the system as
symmetric if we boot the 32bit capable cpus _only_.

> > Haven't tried that tbh. What symptoms do you expect to see? I can try it out.
> > I'm off for the remainder of the day, but can spend few mins to run an
> > experiment for sure.
> 
> No probs; I've been taking Friday afternoons off to burn holiday anyway, so
> you didn't miss anything!

:-)

Thanks

--
Qais Yousef

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> On 11/09/20 13:52, Will Deacon wrote:
> > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > On 11/06/20 13:00, Will Deacon wrote:
> > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > with 32bit features then.
> > > > > 
> > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > 
> > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > are 32-bit capable. What do you do in that case?
> > > 
> > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > attempt to bring the remaining 64bit-only cpus online later?
> > 
> > Right. I think we will refuse to online them. I'll post my attempt at
> > handling that shortly.
> 
> Sorry for the delayed response.
> 
> You're right, I tried that and they refuse to come online. We missed that tbh.
> 
> Haven't thought what we should do yet. I tried your v2 and it failed similarly.

Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
that with a late CPU. Do you know if the CPUs being integrated into these
broken designs have a mismatch at EL1 as well?

> I usually have a similar hunk in my testing to check how the kernel perceives
> the 32bit support when I execute a binary:
> 
> 	diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
> 	index f447d313a9c5..a9549e55a6c8 100644
> 	--- a/arch/arm64/include/asm/cpufeature.h
> 	+++ b/arch/arm64/include/asm/cpufeature.h
> 	@@ -611,6 +611,9 @@ static inline bool system_supports_32bit_el0(void)
> 	 {
> 		u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> 
> 	+       pr_err("System supports symmetric 32bit el0: %d\n", id_aa64pfr0_32bit_el0(pfr0));
> 	+       pr_err("System supports Asymmetric 32bit el0: %ld\n", static_branch_unlikely(&arm64_mismatched_32bit_el0));
> 	+
> 		return id_aa64pfr0_32bit_el0(pfr0) ||
> 		       static_branch_unlikely(&arm64_mismatched_32bit_el0);
> 	 }
> 
> In your v2 both conditions are true. In my series we see the system as
> symmetric if we boot the 32bit capable cpus _only_.

The "arm64_mismatched_32bit_el0" key drives both the creation of the sysfs
file and the allocation of the cpu mask. See the comment in cpufeature.c
That file should be created whenever the command-line is passed to enable
this feature, because a late CPU could come up and set bits in there. The
presence of the file can therefore inform userspace that this can happen.

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> On 11/09/20 13:52, Will Deacon wrote:
> > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > On 11/06/20 13:00, Will Deacon wrote:
> > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > with 32bit features then.
> > > > > 
> > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > 
> > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > are 32-bit capable. What do you do in that case?
> > > 
> > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > attempt to bring the remaining 64bit-only cpus online later?
> > 
> > Right. I think we will refuse to online them. I'll post my attempt at
> > handling that shortly.
> 
> Sorry for the delayed response.
> 
> You're right, I tried that and they refuse to come online. We missed that tbh.
> 
> Haven't thought what we should do yet. I tried your v2 and it failed similarly.

Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
that with a late CPU. Do you know if the CPUs being integrated into these
broken designs have a mismatch at EL1 as well?

> I usually have a similar hunk in my testing to check how the kernel perceives
> the 32bit support when I execute a binary:
> 
> 	diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
> 	index f447d313a9c5..a9549e55a6c8 100644
> 	--- a/arch/arm64/include/asm/cpufeature.h
> 	+++ b/arch/arm64/include/asm/cpufeature.h
> 	@@ -611,6 +611,9 @@ static inline bool system_supports_32bit_el0(void)
> 	 {
> 		u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> 
> 	+       pr_err("System supports symmetric 32bit el0: %d\n", id_aa64pfr0_32bit_el0(pfr0));
> 	+       pr_err("System supports Asymmetric 32bit el0: %ld\n", static_branch_unlikely(&arm64_mismatched_32bit_el0));
> 	+
> 		return id_aa64pfr0_32bit_el0(pfr0) ||
> 		       static_branch_unlikely(&arm64_mismatched_32bit_el0);
> 	 }
> 
> In your v2 both conditions are true. In my series we see the system as
> symmetric if we boot the 32bit capable cpus _only_.

The "arm64_mismatched_32bit_el0" key drives both the creation of the sysfs
file and the allocation of the cpu mask. See the comment in cpufeature.c
That file should be created whenever the command-line is passed to enable
this feature, because a late CPU could come up and set bits in there. The
presence of the file can therefore inform userspace that this can happen.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/12/20 10:24, Will Deacon wrote:
> On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > On 11/09/20 13:52, Will Deacon wrote:
> > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > with 32bit features then.
> > > > > > 
> > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > 
> > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > are 32-bit capable. What do you do in that case?
> > > > 
> > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > attempt to bring the remaining 64bit-only cpus online later?
> > > 
> > > Right. I think we will refuse to online them. I'll post my attempt at
> > > handling that shortly.
> > 
> > Sorry for the delayed response.
> > 
> > You're right, I tried that and they refuse to come online. We missed that tbh.
> > 
> > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> 
> Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> that with a late CPU. Do you know if the CPUs being integrated into these
> broken designs have a mismatch at EL1 as well?

Hmm my test could have been invalid then. We shouldn't have mismatch at EL1,
for ease of testing I used a hacked up patch to fake asymmetry on Juno. Testing
on FVP now, it takes time to boot up though..

Let me re-run this and get you the log from proper environment. Assuming it
still fails.

> > I usually have a similar hunk in my testing to check how the kernel perceives
> > the 32bit support when I execute a binary:
> > 
> > 	diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
> > 	index f447d313a9c5..a9549e55a6c8 100644
> > 	--- a/arch/arm64/include/asm/cpufeature.h
> > 	+++ b/arch/arm64/include/asm/cpufeature.h
> > 	@@ -611,6 +611,9 @@ static inline bool system_supports_32bit_el0(void)
> > 	 {
> > 		u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> > 
> > 	+       pr_err("System supports symmetric 32bit el0: %d\n", id_aa64pfr0_32bit_el0(pfr0));
> > 	+       pr_err("System supports Asymmetric 32bit el0: %ld\n", static_branch_unlikely(&arm64_mismatched_32bit_el0));
> > 	+
> > 		return id_aa64pfr0_32bit_el0(pfr0) ||
> > 		       static_branch_unlikely(&arm64_mismatched_32bit_el0);
> > 	 }
> > 
> > In your v2 both conditions are true. In my series we see the system as
> > symmetric if we boot the 32bit capable cpus _only_.
> 
> The "arm64_mismatched_32bit_el0" key drives both the creation of the sysfs
> file and the allocation of the cpu mask. See the comment in cpufeature.c
> That file should be created whenever the command-line is passed to enable
> this feature, because a late CPU could come up and set bits in there. The
> presence of the file can therefore inform userspace that this can happen.

Okay. I just didn't expect both to return true here. It's not a bug per se.
It's just slightly misleading for arm64_mismatched_32bit_el0 to be true when
the system is symmetric.

Thanks

--
Qais Yousef

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/12/20 10:24, Will Deacon wrote:
> On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > On 11/09/20 13:52, Will Deacon wrote:
> > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > with 32bit features then.
> > > > > > 
> > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > 
> > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > are 32-bit capable. What do you do in that case?
> > > > 
> > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > attempt to bring the remaining 64bit-only cpus online later?
> > > 
> > > Right. I think we will refuse to online them. I'll post my attempt at
> > > handling that shortly.
> > 
> > Sorry for the delayed response.
> > 
> > You're right, I tried that and they refuse to come online. We missed that tbh.
> > 
> > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> 
> Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> that with a late CPU. Do you know if the CPUs being integrated into these
> broken designs have a mismatch at EL1 as well?

Hmm my test could have been invalid then. We shouldn't have mismatch at EL1,
for ease of testing I used a hacked up patch to fake asymmetry on Juno. Testing
on FVP now, it takes time to boot up though..

Let me re-run this and get you the log from proper environment. Assuming it
still fails.

> > I usually have a similar hunk in my testing to check how the kernel perceives
> > the 32bit support when I execute a binary:
> > 
> > 	diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
> > 	index f447d313a9c5..a9549e55a6c8 100644
> > 	--- a/arch/arm64/include/asm/cpufeature.h
> > 	+++ b/arch/arm64/include/asm/cpufeature.h
> > 	@@ -611,6 +611,9 @@ static inline bool system_supports_32bit_el0(void)
> > 	 {
> > 		u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> > 
> > 	+       pr_err("System supports symmetric 32bit el0: %d\n", id_aa64pfr0_32bit_el0(pfr0));
> > 	+       pr_err("System supports Asymmetric 32bit el0: %ld\n", static_branch_unlikely(&arm64_mismatched_32bit_el0));
> > 	+
> > 		return id_aa64pfr0_32bit_el0(pfr0) ||
> > 		       static_branch_unlikely(&arm64_mismatched_32bit_el0);
> > 	 }
> > 
> > In your v2 both conditions are true. In my series we see the system as
> > symmetric if we boot the 32bit capable cpus _only_.
> 
> The "arm64_mismatched_32bit_el0" key drives both the creation of the sysfs
> file and the allocation of the cpu mask. See the comment in cpufeature.c
> That file should be created whenever the command-line is passed to enable
> this feature, because a late CPU could come up and set bits in there. The
> presence of the file can therefore inform userspace that this can happen.

Okay. I just didn't expect both to return true here. It's not a bug per se.
It's just slightly misleading for arm64_mismatched_32bit_el0 to be true when
the system is symmetric.

Thanks

--
Qais Yousef

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

[-- Attachment #1: Type: text/plain, Size: 2768 bytes --]

On 11/12/20 11:55, Qais Yousef wrote:
> On 11/12/20 10:24, Will Deacon wrote:
> > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > On 11/09/20 13:52, Will Deacon wrote:
> > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > with 32bit features then.
> > > > > > > 
> > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > 
> > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > are 32-bit capable. What do you do in that case?
> > > > > 
> > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > 
> > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > handling that shortly.
> > > 
> > > Sorry for the delayed response.
> > > 
> > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > 
> > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > 
> > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > that with a late CPU. Do you know if the CPUs being integrated into these
> > broken designs have a mismatch at EL1 as well?
> 
> Hmm my test could have been invalid then. We shouldn't have mismatch at EL1,
> for ease of testing I used a hacked up patch to fake asymmetry on Juno. Testing
> on FVP now, it takes time to boot up though..
> 
> Let me re-run this and get you the log from proper environment. Assuming it
> still fails.

Still fails the same on FVP. dmesg attached. There's a splat shortly after
attempting to online CPU 4.

	# cat /sys/devices/system/cpu/online
	0-3
	# cat /sys/devices/system/cpu/aarch32_el0
	0-3

Now while writing this I just realized I tell the FVP to disable aarch32
support at EL0. So this might still make the kernel thinks there's AArch32
support at EL1 - which seems is what makes your series get confused?

Anyway. No real hardware to test on and not sure if I can tell the FVP to
disable AArch32 support at EL1.

/me goes and dig

Thanks

--
Qais Yousef

[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 36330 bytes --]

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd0f0]
[    0.000000] Linux version 5.10.0-rc2-00006-g0e6c7349e720-dirty (qyousef@e107158-lin) (aarch64-linux-gnu-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #239 SMP PREEMPT Thu Nov 12 11:39:59 GMT 2020
[    0.000000] Machine model: FVP Base RevC
[    0.000000] earlycon: pl11 at MMIO 0x000000001c090000 (options '')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000018000000, size 8 MiB
[    0.000000] OF: reserved mem: initialized node vram@18000000, compatible id shared-dma-pool
[    0.000000] cma: Reserved 32 MiB at 0x00000000fe000000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x00000008ffffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x8ff7f7000-0x8ff7f8fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000]   DMA32    [mem 0x00000000c0000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000008ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000880000000-0x00000008ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000008ffffffff]
[    0.000000] On node 0 totalpages: 1048576
[    0.000000]   DMA zone: 4096 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 262144 pages, LIFO batch:63
[    0.000000]   DMA32 zone: 4096 pages used for memmap
[    0.000000]   DMA32 zone: 262144 pages, LIFO batch:63
[    0.000000]   Normal zone: 8192 pages used for memmap
[    0.000000]   Normal zone: 524288 pages, LIFO batch:63
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: Using PSCI v0.1 Function IDs from DT
[    0.000000] percpu: Embedded 51 pages/cpu s168080 r8192 d32624 u208896
[    0.000000] pcpu-alloc: s168080 r8192 d32624 u208896 alloc=51*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5 [0] 6 [0] 7 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1032192
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyAMA0 earlycon=pl011,0x1c090000 root=/dev/vda rootwait allow_mismatched_32bit_el0 maxcpus=4
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000bbfff000-0x00000000bffff000] (64MB)
[    0.000000] Memory: 3901352K/4194304K available (26752K kernel code, 4802K rwdata, 15452K rodata, 54144K init, 11359K bss, 260184K reserved, 32768K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] ftrace: allocating 82932 entries in 324 pages
[    0.000000] ftrace: allocated 324 pages with 3 groups
[    0.000000] Running RCU self tests
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU lockdep checking is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Rude variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 224 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000002f100000
[    0.000000] ITS [mem 0x2f020000-0x2f03ffff]
[    0.000000] ITS@0x000000002f020000: allocated 8192 Devices @880250000 (indirect, esz 8, psz 64K, shr 1)
[    0.000000] ITS@0x000000002f020000: allocated 8192 Virtual CPUs @880260000 (indirect, esz 8, psz 64K, shr 1)
[    0.000000] ITS@0x000000002f020000: allocated 8192 Interrupt Collections @880270000 (flat, esz 8, psz 64K, shr 1)
[    0.000000] GICv3: using LPI property table @0x0000000880280000
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000880290000
[    0.000000] random: get_random_bytes called from start_kernel+0x3a8/0x574 with crng_init=0
[    0.000000] clocksource: arm,sp804: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.000000] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.001387] Failed to initialize '/bus@8000000/motherboard-bus/iofpga-bus@300000000/timer@120000': -22
[    0.005714] arch_timer: cp15 timer(s) running at 25.16MHz (phys).
[    0.005919] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5cdd39714, max_idle_ns: 440795202620 ns
[    0.006286] sched_clock: 56 bits at 25MHz, resolution 39ns, wraps every 4398046511084ns
[    0.020121] Console: colour dummy device 80x25
[    0.022074] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.023050] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.024027] ... MAX_LOCK_DEPTH:          48
[    0.025024] ... MAX_LOCKDEP_KEYS:        8192
[    0.026631] ... CLASSHASH_SIZE:          4096
[    0.027608] ... MAX_LOCKDEP_ENTRIES:     32768
[    0.028422] ... MAX_LOCKDEP_CHAINS:      65536
[    0.029398] ... CHAINHASH_SIZE:          32768
[    0.030958]  memory used by lock dependency info: 6365 kB
[    0.032002]  memory used for stack traces: 4224 kB
[    0.032979]  per task-struct memory footprint: 1920 bytes
[    0.038769] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.33 BogoMIPS (lpj=100663)
[    0.040954] pid_max: default: 32768 minimum: 301
[    0.052020] LSM: Security Framework initializing
[    0.058532] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.059834] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.359022] rcu: Hierarchical SRCU implementation.
[    0.473083] Platform MSI: msi-controller@2f020000 domain created
[    0.482364] PCI/MSI: /interrupt-controller@2f000000/msi-controller@2f020000 domain created
[    0.515889] EFI services will not be available.
[    0.604752] smp: Bringing up secondary CPUs ...
[    0.745022] Detected PIPT I-cache on CPU1
[    0.745419] GICv3: CPU1: found redistributor 100 region 0:0x000000002f120000
[    0.746033] GICv3: CPU1: using allocated LPI pending table @0x00000008802a0000
[    0.747406] CPU1: Booted secondary processor 0x0000000100 [0x410fd0f0]
[    0.933372] Detected PIPT I-cache on CPU2
[    0.934167] GICv3: CPU2: found redistributor 200 region 0:0x000000002f140000
[    0.934565] GICv3: CPU2: using allocated LPI pending table @0x00000008802b0000
[    0.936314] CPU2: Booted secondary processor 0x0000000200 [0x410fd0f0]
[    1.111392] Detected PIPT I-cache on CPU3
[    1.112243] GICv3: CPU3: found redistributor 300 region 0:0x000000002f160000
[    1.112584] GICv3: CPU3: using allocated LPI pending table @0x00000008802c0000
[    1.113802] CPU3: Booted secondary processor 0x0000000300 [0x410fd0f0]
[    1.157328] smp: Brought up 1 node, 4 CPUs
[    1.173380] SMP: Total of 4 processors activated.
[    1.174580] CPU features: detected: 32-bit EL0 Support
[    1.175882] CPU features: detected: 32-bit EL1 Support
[    4.568064] CPU: All CPU(s) started at EL2
[    4.574484] alternatives: patching kernel code
[    4.738533] devtmpfs: initialized
[    5.517367] KASLR disabled due to lack of seed
[    5.659444] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    5.661397] futex hash table entries: 2048 (order: 6, 262144 bytes, linear)
[    5.779398] pinctrl core: initialized pinctrl subsystem
[    6.002946] DMI not present or invalid.
[    6.083214] NET: Registered protocol family 16
[    6.471294] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[    6.480340] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    6.489522] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    6.504023] audit: initializing netlink subsys (disabled)
[    6.525878] audit: type=2000 audit(4.728:1): state=initialized audit_enabled=0 res=1
[    6.707783] thermal_sys: Registered thermal governor 'step_wise'
[    6.708269] thermal_sys: Registered thermal governor 'power_allocator'
[    6.724162] cpuidle: using governor menu
[    6.744893] NET: Registered protocol family 42
[    6.784281] hw-breakpoint: found 16 breakpoint and 16 watchpoint registers.
[    6.802673] ASID allocator initialised with 32768 entries
[    6.990010] Serial: AMBA PL011 UART driver
[   10.396603] 1c090000.serial: ttyAMA0 at MMIO 0x1c090000 (irq = 21, base_baud = 0) is a PL011 rev2
[   10.404561] printk: console [ttyAMA0] enabled
[   10.406677] printk: bootconsole [pl11] disabled
[   10.552510] 1c0a0000.serial: ttyAMA1 at MMIO 0x1c0a0000 (irq = 22, base_baud = 0) is a PL011 rev2
[   10.671813] 1c0b0000.serial: ttyAMA2 at MMIO 0x1c0b0000 (irq = 23, base_baud = 0) is a PL011 rev2
[   10.801915] 1c0c0000.serial: ttyAMA3 at MMIO 0x1c0c0000 (irq = 24, base_baud = 0) is a PL011 rev2
[   16.484622] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[   16.487403] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[   16.489203] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[   16.490824] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[   17.066760] cryptd: max_cpu_qlen set to 1000
[   17.808695] raid6: neonx8   gen()    51 MB/s
[   17.892087] raid6: neonx8   xor()    35 MB/s
[   17.976501] raid6: neonx4   gen()    48 MB/s
[   18.061787] raid6: neonx4   xor()    34 MB/s
[   18.145446] raid6: neonx2   gen()    43 MB/s
[   18.228780] raid6: neonx2   xor()    28 MB/s
[   18.313689] raid6: neonx1   gen()    35 MB/s
[   18.398725] raid6: neonx1   xor()    22 MB/s
[   18.483499] raid6: int64x8  gen()    17 MB/s
[   18.567648] raid6: int64x8  xor()     9 MB/s
[   18.652244] raid6: int64x4  gen()    18 MB/s
[   18.739269] raid6: int64x4  xor()     9 MB/s
[   18.824808] raid6: int64x2  gen()    16 MB/s
[   18.911490] raid6: int64x2  xor()     9 MB/s
[   18.995965] raid6: int64x1  gen()    14 MB/s
[   19.079808] raid6: int64x1  xor()     7 MB/s
[   19.081397] raid6: using algorithm neonx8 gen() 51 MB/s
[   19.082987] raid6: .... xor() 35 MB/s, rmw enabled
[   19.084179] raid6: using neon recovery algorithm
[   19.158910] ACPI: Interpreter disabled.
[   19.375264] iommu: Default domain type: Translated 
[   19.416376] vgaarb: loaded
[   19.497660] SCSI subsystem initialized
[   19.531351] libata version 3.00 loaded.
[   19.582132] usbcore: registered new interface driver usbfs
[   19.592805] usbcore: registered new interface driver hub
[   19.613071] usbcore: registered new device driver usb
[   19.707643] mc: Linux media interface: v0.10
[   19.715597] videodev: Linux video capture interface: v2.00
[   19.737446] pps_core: LinuxPPS API ver. 1 registered
[   19.739035] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[   19.743804] PTP clock support registered
[   19.809184] EDAC MC: Ver: 3.0.0
[   19.987732] FPGA manager framework
[   20.023151] Advanced Linux Sound Architecture Driver Initialized.
[   20.154281] Bluetooth: Core ver 2.22
[   20.160691] NET: Registered protocol family 31
[   20.163023] Bluetooth: HCI device and connection manager initialized
[   20.166997] Bluetooth: HCI socket layer initialized
[   20.169209] Bluetooth: L2CAP socket layer initialized
[   20.174944] Bluetooth: SCO socket layer initialized
[   20.263954] clocksource: Switched to clocksource arch_sys_counter
[  132.044209] VFS: Disk quotas dquot_6.6.0
[  132.056835] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[  132.135911] pnp: PnP ACPI: disabled
[  134.446227] NET: Registered protocol family 2
[  134.529633] tcp_listen_portaddr_hash hash table entries: 2048 (order: 5, 163840 bytes, linear)
[  134.562217] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[  134.673187] TCP bind hash table entries: 32768 (order: 9, 2359296 bytes, linear)
[  134.931578] TCP: Hash tables configured (established 32768 bind 32768)
[  134.965433] UDP hash table entries: 2048 (order: 6, 327680 bytes, linear)
[  135.009252] UDP-Lite hash table entries: 2048 (order: 6, 327680 bytes, linear)
[  135.081481] NET: Registered protocol family 1
[  135.212503] RPC: Registered named UNIX socket transport module.
[  135.216994] RPC: Registered udp transport module.
[  135.220478] RPC: Registered tcp transport module.
[  135.224059] RPC: Registered tcp NFSv4.1 backchannel transport module.
[  135.348204] PCI: CLS 0 bytes, default 64
[ 1598.688719] hw perfevents: enabled with armv8_pmuv3 PMU driver, 9 counters available
[ 1598.705155] kvm [1]: IPA Size Limit: 40 bits
[ 1599.985716] kvm [1]: vgic-v2@2c02f000
[ 1599.992608] kvm [1]: GIC system register CPU interface enabled
[ 1600.019626] kvm [1]: vgic interrupt IRQ9
[ 1600.049294] kvm [1]: Hyp mode initialized successfully
[ 1601.384836] Initialise system trusted keyrings
[ 1601.425225] workingset: timestamp_bits=44 max_order=20 bucket_order=0
[ 1605.817544] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 1606.057839] NFS: Registering the id_resolver key type
[ 1606.065389] Key type id_resolver registered
[ 1606.069977] Key type id_legacy registered
[ 1606.120700] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 1606.168705] fuse: init (API version 7.32)
[ 1606.373793] 9p: Installing v9fs 9p2000 file system support
[ 1606.744515] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 1606.747979] NET: Registered protocol family 38
[ 1606.752861] xor: measuring software checksum speed
[ 1607.123467]    8regs           :    27 MB/sec
[ 1607.400137]    32regs          :    36 MB/sec
[ 1607.589288]    arm64_neon      :    53 MB/sec
[ 1607.592544] xor: using function: arm64_neon (53 MB/sec)
[ 1607.595798] Key type asymmetric registered
[ 1607.600518] Asymmetric key parser 'x509' registered
[ 1607.609180] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[ 1607.613419] io scheduler mq-deadline registered
[ 1607.616700] io scheduler kyber registered
[ 1609.357679] pci-host-generic 40000000.pci: host bridge /pci@40000000 ranges:
[ 1609.365329] pci-host-generic 40000000.pci:      MEM 0x0050000000..0x005fffffff -> 0x0050000000
[ 1609.377976] pci-host-generic 40000000.pci: ECAM at [mem 0x40000000-0x4fffffff] for [bus 00-01]
[ 1609.424707] pci-host-generic 40000000.pci: PCI host bridge to bus 0000:00
[ 1609.428284] pci_bus 0000:00: root bus resource [bus 00-01]
[ 1609.432257] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[ 1609.444756] pci 0000:00:00.0: [1af4:1001] type 00 class 0x018000
[ 1609.450138] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00000fff]
[ 1609.456894] pci 0000:00:00.0: reg 0x18: [mem 0x00000000-0x00000fff]
[ 1609.461684] pci 0000:00:00.0: reg 0x20: [mem 0x00000000-0x00000fff]
[ 1609.484709] pci 0000:00:00.0: PME# supported from D3hot
[ 1609.565374] pci 0000:00:02.0: [1af4:1001] type 00 class 0x018000
[ 1609.572528] pci 0000:00:02.0: reg 0x10: [mem 0x00000000-0x00000fff]
[ 1609.577569] pci 0000:00:02.0: reg 0x18: [mem 0x00000000-0x00000fff]
[ 1609.584567] pci 0000:00:02.0: reg 0x20: [mem 0x00000000-0x00000fff]
[ 1609.604750] pci 0000:00:02.0: PME# supported from D3hot
[ 1609.680759] pci 0000:00:03.0: [0abc:aced] type 00 class 0x010601
[ 1609.685804] pci 0000:00:03.0: reg 0x10: [mem 0x00000000-0x00001fff]
[ 1609.692004] pci 0000:00:03.0: reg 0x14: [mem 0x00000000-0x00001fff]
[ 1609.696384] pci 0000:00:03.0: reg 0x18: [mem 0x00000000-0x00000fff]
[ 1609.700479] pci 0000:00:03.0: reg 0x1c: [mem 0x00000000-0x00001fff]
[ 1609.705010] pci 0000:00:03.0: reg 0x20: [mem 0x00000000-0x00000fff]
[ 1609.709220] pci 0000:00:03.0: reg 0x24: [mem 0x00000000-0x00001fff]
[ 1609.724718] pci 0000:00:03.0: PME# supported from D3hot
[ 1609.848077] pci 0000:00:03.0: BAR 0: assigned [mem 0x50000000-0x50001fff]
[ 1609.851983] pci 0000:00:03.0: BAR 1: assigned [mem 0x50002000-0x50003fff]
[ 1609.856245] pci 0000:00:03.0: BAR 3: assigned [mem 0x50004000-0x50005fff]
[ 1609.860219] pci 0000:00:03.0: BAR 5: assigned [mem 0x50006000-0x50007fff]
[ 1609.864192] pci 0000:00:00.0: BAR 0: assigned [mem 0x50008000-0x50008fff]
[ 1609.868259] pci 0000:00:00.0: BAR 2: assigned [mem 0x50009000-0x50009fff]
[ 1609.872140] pci 0000:00:00.0: BAR 4: assigned [mem 0x5000a000-0x5000afff]
[ 1609.876234] pci 0000:00:02.0: BAR 0: assigned [mem 0x5000b000-0x5000bfff]
[ 1609.880087] pci 0000:00:02.0: BAR 2: assigned [mem 0x5000c000-0x5000cfff]
[ 1609.884060] pci 0000:00:02.0: BAR 4: assigned [mem 0x5000d000-0x5000dfff]
[ 1609.888034] pci 0000:00:03.0: BAR 2: assigned [mem 0x5000e000-0x5000efff]
[ 1609.892008] pci 0000:00:03.0: BAR 4: assigned [mem 0x5000f000-0x5000ffff]
[ 1610.171807] IPMI message handler: version 39.2
[ 1610.191989] ipmi device interface
[ 1610.204359] ipmi_si: IPMI System Interface driver
[ 1610.264633] ipmi_si: Unable to find any System Interface(s)
[ 1610.360103] EINJ: ACPI disabled.
[ 1613.177667] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 1613.888301] SuperH (H)SCI(F) driver initialized
[ 1613.948012] msm_serial: driver initialized
[ 1614.364740] arm-smmu-v3 2b400000.iommu: ias 48-bit, oas 48-bit (features 0x00001fef)
[ 1614.392184] arm-smmu-v3 2b400000.iommu: allocated 65536 entries for cmdq
[ 1614.505975] arm-smmu-v3 2b400000.iommu: allocated 32768 entries for evtq
[ 1616.056898] panel-simple panel: supply power not found, using dummy regulator
[ 1616.143343] ------------[ cut here ]------------
[ 1616.144952] WARNING: CPU: 1 PID: 1 at drivers/gpu/drm/panel/panel-simple.c:467 panel_simple_parse_panel_timing_node+0x1e4/0x1f8
[ 1616.146416] Modules linked in:
[ 1616.148207] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.10.0-rc2-00006-g0e6c7349e720-dirty #239
[ 1616.149701] Hardware name: FVP Base RevC (DT)
[ 1616.151135] pstate: 20000005 (nzCv daif -PAN -UAO -TCO BTYPE=--)
[ 1616.152601] pc : panel_simple_parse_panel_timing_node+0x1e4/0x1f8
[ 1616.154102] lr : panel_simple_probe+0x2b4/0x3e0
[ 1616.155375] sp : ffff80001005ba60
[ 1616.156670] x29: ffff80001005ba60 x28: 0000000000000000 
[ 1616.159188] x27: ffff8000129d04e0 x26: ffff800012ad10d8 
[ 1616.161622] x25: ffff8000161d0a38 x24: 0000000000000000 
[ 1616.164006] x23: ffff8000120d4848 x22: ffff000800388000 
[ 1616.166390] x21: ffff000800ab0010 x20: ffff000800388000 
[ 1616.168937] x19: ffff000800d3b680 x18: 0000000000001460 
[ 1616.171319] x17: 0000000000000004 x16: 0000000000000000 
[ 1616.173597] x15: ffffffffffffffff x14: ffffffff00000000 
[ 1616.176324] x13: ffffffffffffffff x12: 0000000000000020 
[ 1616.178709] x11: 0000000000000004 x10: 0101010101010101 
[ 1616.181093] x9 : ffff800010dcf4ec x8 : 00000000ffffffff 
[ 1616.183526] x7 : 00000000000000e3 x6 : 0000000000000001 
[ 1616.185968] x5 : ffff00087f805b10 x4 : 0000000000000000 
[ 1616.188245] x3 : ffff8000120d4848 x2 : ffff80001005bb18 
[ 1616.190850] x1 : 0000000000000001 x0 : ffff000800ab0010 
[ 1616.193293] Call trace:
[ 1616.194756]  panel_simple_parse_panel_timing_node+0x1e4/0x1f8
[ 1616.196221]  panel_simple_probe+0x2b4/0x3e0
[ 1616.197523]  panel_simple_platform_probe+0x3c/0x58
[ 1616.199030]  platform_drv_probe+0x5c/0xb0
[ 1616.200453]  really_probe+0xec/0x3b8
[ 1616.201918]  driver_probe_device+0x60/0xc0
[ 1616.203345]  device_driver_attach+0x7c/0x88
[ 1616.204538]  __driver_attach+0x60/0xf0
[ 1616.206149]  bus_for_each_dev+0x78/0xd0
[ 1616.207507]  driver_attach+0x2c/0x38
[ 1616.208917]  bus_add_driver+0x158/0x200
[ 1616.210381]  driver_register+0x6c/0x128
[ 1616.211746]  __platform_driver_register+0x50/0x60
[ 1616.213279]  panel_simple_init+0x2c/0x54
[ 1616.214613]  do_one_initcall+0x94/0x4b0
[ 1616.216079]  kernel_init_freeable+0x2c0/0x32c
[ 1616.217253]  kernel_init+0x1c/0x128
[ 1616.218844]  ret_from_fork+0x10/0x30
[ 1616.220223] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.10.0-rc2-00006-g0e6c7349e720-dirty #239
[ 1616.221624] Hardware name: FVP Base RevC (DT)
[ 1616.222915] Call trace:
[ 1616.224406]  dump_backtrace+0x0/0x1b0
[ 1616.225734]  show_stack+0x20/0x70
[ 1616.227187]  dump_stack+0xf8/0x168
[ 1616.228610]  __warn+0xfc/0x180
[ 1616.229968]  report_bug+0xfc/0x170
[ 1616.231377]  bug_handler+0x28/0x70
[ 1616.232750]  call_break_hook+0x70/0x88
[ 1616.234144]  brk_handler+0x24/0x68
[ 1616.235532]  do_debug_exception+0xb8/0x130
[ 1616.236911]  el1_sync_handler+0xd8/0x120
[ 1616.238377]  el1_sync+0x80/0x100
[ 1616.239721]  panel_simple_parse_panel_timing_node+0x1e4/0x1f8
[ 1616.241143]  panel_simple_probe+0x2b4/0x3e0
[ 1616.242608]  panel_simple_platform_probe+0x3c/0x58
[ 1616.244073]  platform_drv_probe+0x5c/0xb0
[ 1616.245465]  really_probe+0xec/0x3b8
[ 1616.246927]  driver_probe_device+0x60/0xc0
[ 1616.248305]  device_driver_attach+0x7c/0x88
[ 1616.249441]  __driver_attach+0x60/0xf0
[ 1616.251164]  bus_for_each_dev+0x78/0xd0
[ 1616.252536]  driver_attach+0x2c/0x38
[ 1616.253838]  bus_add_driver+0x158/0x200
[ 1616.255399]  driver_register+0x6c/0x128
[ 1616.256768]  __platform_driver_register+0x50/0x60
[ 1616.258233]  panel_simple_init+0x2c/0x54
[ 1616.259699]  do_one_initcall+0x94/0x4b0
[ 1616.261000]  kernel_init_freeable+0x2c0/0x32c
[ 1616.262464]  kernel_init+0x1c/0x128
[ 1616.263880]  ret_from_fork+0x10/0x30
[ 1616.265154] irq event stamp: 2125172
[ 1616.266533] hardirqs last  enabled at (2125171): [<ffff800011a970a0>] _raw_spin_unlock_irqrestore+0xa0/0xa8
[ 1616.268115] hardirqs last disabled at (2125172): [<ffff8000100ba094>] debug_exception_enter+0xb4/0xf0
[ 1616.269789] softirqs last  enabled at (2124972): [<ffff800010090a8c>] __do_softirq+0x604/0x684
[ 1616.271295] softirqs last disabled at (2124967): [<ffff800010130c70>] irq_exit+0x180/0x1a0
[ 1616.272884] ---[ end trace 8ab98a1b1a71c11c ]---
[ 1616.276857] panel-simple panel: Reject override mode: panel has a fixed mode
[ 1616.280434] panel-simple panel: Specify missing connector_type
[ 1616.441990] drm-clcd-pl111 1c1f0000.clcd: assigned reserved memory node vram@18000000
[ 1616.445570] drm-clcd-pl111 1c1f0000.clcd: using device-specific reserved memory
[ 1616.449151] drm-clcd-pl111 1c1f0000.clcd: no max memory bandwidth specified, assume unlimited
[ 1616.473553] drm-clcd-pl111 1c1f0000.clcd: DVI muxed to motherboard CLCD
[ 1616.483982] drm-clcd-pl111 1c1f0000.clcd: can't find the sysreg device, deferring
[ 1616.485772] drm-clcd-pl111 1c1f0000.clcd: Versatile Express init failed - -517
[ 1616.632907] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 1621.848727] loop: module loaded
[ 1632.609010] virtio_blk virtio0: [vda] 0 512-byte logical blocks (0 B/0 B)
[ 1633.432663] basic-mmio-gpio: Failed to locate of_node [id: -2]
[ 1633.572138] basic-mmio-gpio: Failed to locate of_node [id: -2]
[ 1633.705986] basic-mmio-gpio: Failed to locate of_node [id: -2]
[ 1636.908623] mpt3sas version 35.100.00.00 loaded
[ 1637.410414] ahci 0000:00:03.0: Adding to iommu group 0
[ 1637.497303] ahci 0000:00:03.0: version 3.0
[ 1637.500908] ahci 0000:00:03.0: enabling device (0000 -> 0002)
[ 1637.560154] ahci 0000:00:03.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
[ 1637.562106] ahci 0000:00:03.0: flags: 64bit ncq only 
[ 1637.809828] scsi host0: ahci
[ 1637.969769] ata1: SATA max UDMA/133 abar m8192@0x50006000 port 0x50006100 irq 41
[ 1638.312297] ata1: SATA link down (SStatus 0 SControl 300)
[ 1639.064548] libphy: Fixed MDIO Bus: probed
[ 1639.517510] tun: Universal TUN/TAP device driver, 1.6
[ 1639.727960] virtio_net: probe of virtio2 failed with error -2
[ 1639.956988] thunder_xcv, ver 1.0
[ 1639.972589] thunder_bgx, ver 1.0
[ 1639.992750] nicpf, ver 1.0
[ 1640.089310] hclge is initializing
[ 1640.108005] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[ 1640.109632] hns3: Copyright (c) 2017 Huawei Corporation.
[ 1640.124932] e1000e: Intel(R) PRO/1000 Network Driver
[ 1640.127854] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[ 1640.141762] igb: Intel(R) Gigabit Ethernet Network Driver
[ 1640.145278] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 1640.156670] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[ 1640.160438] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 1640.233493] sky2: driver version 1.30
[ 1640.410088] smc91x 1a000000.ethernet (unnamed net_device) (uninitialized): smc91x: IOADDR (____ptrval____) doesn't match configuration (300).
[ 1640.413832] smc91x.c: v1.1, sep 22 2004 by Nicolas Pitre <nico@fluxnic.net>
[ 1643.585381] smc91x 1a000000.ethernet eth0: SMC91C11xFD (rev 1) at (____ptrval____) IRQ 15
[ 1643.588897] 
[ 1643.589693] smc91x 1a000000.ethernet eth0: Ethernet addr: 00:02:f7:ef:36:98
[ 1643.969175] pegasus: v0.9.3 (2013/04/25), Pegasus/Pegasus II USB Ethernet driver
[ 1643.980699] usbcore: registered new interface driver pegasus
[ 1643.992619] usbcore: registered new interface driver rtl8150
[ 1644.005733] usbcore: registered new interface driver r8152
[ 1644.016859] usbcore: registered new interface driver lan78xx
[ 1644.029972] usbcore: registered new interface driver asix
[ 1644.041273] usbcore: registered new interface driver ax88179_178a
[ 1644.052829] usbcore: registered new interface driver cdc_ether
[ 1644.062159] usbcore: registered new interface driver dm9601
[ 1644.076838] usbcore: registered new interface driver CoreChips
[ 1644.089938] usbcore: registered new interface driver smsc75xx
[ 1644.105074] usbcore: registered new interface driver smsc95xx
[ 1644.116995] usbcore: registered new interface driver net1080
[ 1644.128121] usbcore: registered new interface driver plusb
[ 1644.137464] usbcore: registered new interface driver cdc_subset
[ 1644.148387] usbcore: registered new interface driver zaurus
[ 1644.159910] usbcore: registered new interface driver MOSCHIP usb-ethernet driver
[ 1644.173901] usbcore: registered new interface driver cdc_ncm
[ 1644.221449] VFIO - User Level meta-driver version: 0.3
[ 1644.648370] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1644.649997] ehci-pci: EHCI PCI platform driver
[ 1644.661879] ehci-platform: EHCI generic platform driver
[ 1644.685641] ehci-orion: EHCI orion driver
[ 1644.709067] ehci-exynos: EHCI Exynos driver
[ 1644.732514] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 1644.737400] ohci-pci: OHCI PCI platform driver
[ 1644.749201] ohci-platform: OHCI generic platform driver
[ 1644.772881] ohci-exynos: OHCI Exynos driver
[ 1644.868410] usbcore: registered new interface driver usb-storage
[ 1645.674265] rtc-pl031 1c170000.rtc: registered as rtc0
[ 1645.681247] rtc-pl031 1c170000.rtc: setting system clock to 2020-11-12T15:38:43 UTC (1605195523)
[ 1645.836391] i2c /dev entries driver
[ 1646.276670] usbcore: registered new interface driver uvcvideo
[ 1646.279848] USB Video Class driver (1.1.1)
[ 1646.281439] gspca_main: v2.14.0 registered
[ 1647.137302] sp805-wdt 1c0f0000.wdt: registration successful
[ 1647.452471] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com
[ 1647.457637] Bluetooth: HCI UART driver ver 2.3
[ 1647.460898] Bluetooth: HCI UART protocol H4 registered
[ 1647.468365] Bluetooth: HCI UART protocol LL registered
[ 1647.492264] Bluetooth: HCI UART protocol Broadcom registered
[ 1647.499757] Bluetooth: HCI UART protocol QCA registered
[ 1647.513117] usbcore: registered new interface driver btusb
[ 1647.732613] mmci-pl18x 1c050000.mmci: Got CD GPIO
[ 1647.740818] mmci-pl18x 1c050000.mmci: Got WP GPIO
[ 1647.864027] mmci-pl18x 1c050000.mmci: mmc0: PL180 manf 41 rev0 at 0x1c050000 irq 17,18 (pio)
[ 1648.063617] sdhci: Secure Digital Host Controller Interface driver
[ 1648.065361] sdhci: Copyright(c) Pierre Ossman
[ 1648.131963] Synopsys Designware Multimedia Card Interface Driver
[ 1648.305759] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1648.313956] input: PS/2 Generic Mouse as /devices/platform/bus@8000000/bus@8000000:motherboard-bus/bus@8000000:motherboard-bus:iofpga-bus@300000000/1c070000.kmi/serio1/input/input1
[ 1648.588933] ledtrig-cpu: registered to indicate activity on CPUs
[ 1648.909605] input: AT Raw Set 2 keyboard as /devices/platform/bus@8000000/bus@8000000:motherboard-bus/bus@8000000:motherboard-bus:iofpga-bus@300000000/1c060000.kmi/serio0/input/input2
[ 1649.000999] usbcore: registered new interface driver usbhid
[ 1649.004178] usbhid: USB HID core driver
[ 1650.760131] drop_monitor: Initializing network drop monitor service
[ 1651.172295] NET: Registered protocol family 10
[ 1651.436842] Segment Routing with IPv6
[ 1651.503998] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 1651.668506] NET: Registered protocol family 17
[ 1651.682016] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[ 1651.685592] Bluetooth: HIDP socket layer initialized
[ 1651.697362] 8021q: 802.1Q VLAN Support v1.8
[ 1651.769463] 9pnet: Installing 9P2000 support
[ 1651.876328] Key type dns_resolver registered
[ 1651.997166] registered taskstats version 1
[ 1652.001659] Loading compiled-in X.509 certificates
[ 1652.388279] Btrfs loaded, crc32c=crc32c-generic
[ 1653.089124] virtio-pci 0000:00:00.0: Adding to iommu group 1
[ 1653.170017] virtio-pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 1653.177298] virtio-pci 0000:00:00.0: virtio_pci: leaving for legacy driver
[ 1653.470155] virtio_blk virtio3: [vdb] 0 512-byte logical blocks (0 B/0 B)
[ 1654.545215] virtio-pci 0000:00:02.0: Adding to iommu group 2
[ 1654.616794] virtio-pci 0000:00:02.0: enabling device (0000 -> 0002)
[ 1654.624281] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver
[ 1654.852188] virtio_blk virtio4: [vdc] 0 512-byte logical blocks (0 B/0 B)
[ 1655.924899] drm-clcd-pl111 1c1f0000.clcd: assigned reserved memory node vram@18000000
[ 1655.929046] drm-clcd-pl111 1c1f0000.clcd: using device-specific reserved memory
[ 1655.933417] drm-clcd-pl111 1c1f0000.clcd: no max memory bandwidth specified, assume unlimited
[ 1655.965604] drm-clcd-pl111 1c1f0000.clcd: DVI muxed to motherboard CLCD
[ 1656.032523] drm-clcd-pl111 1c1f0000.clcd: initializing Versatile Express PL111
[ 1656.105078] drm-clcd-pl111 1c1f0000.clcd: found panel on endpoint 0
[ 1656.381644] [drm] Initialized pl111 1.0.0 20170317 for 1c1f0000.clcd on minor 0
[ 1657.562213] Console: switching to colour frame buffer device 128x48
[ 1658.301312] drm-clcd-pl111 1c1f0000.clcd: [drm] fb0: pl111drmfb frame buffer device
[ 1658.471780] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 1659.821384] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 1659.860171] ALSA device list:
[ 1659.871695]   No soundcards found.
[ 1659.918064] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ 1659.924544] platform regulatory.0: Falling back to sysfs fallback for: regulatory.db
[ 1659.944412] uart-pl011 1c090000.serial: no DMA platform data
[ 1665.641822] Freeing unused kernel memory: 54144K
[ 1665.656524] Run /init as init process
[ 1665.657716]   with arguments:
[ 1665.660100]     /init
[ 1665.660895]   with environment:
[ 1665.661553]     HOME=/
[ 1665.663676]     TERM=linux
[ 1720.461405] cfg80211: failed to load regulatory.db
[ 2264.157156] random: dd: uninitialized urandom read (512 bytes read)
[ 2391.928388] random: crng init done
[ 2400.436615] smc91x 1a000000.ethernet eth0: link up, 10Mbps, half-duplex, lpa 0x0000
[ 8569.024731] CPU features: CPU4: Detected conflict for capability 53 (32-bit EL1 Support), System: 1, CPU: 0
[ 8569.025525] CPU4: will not boot
[ 8574.269542] CPU4: failed to come online
[ 8574.282258] CPU4: died during early boot
[ 8599.142560] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8599.149806] rcu: 	4-O...: (0 ticks this GP) idle=002/1/0x4000000000000000 softirq=0/0 fqs=2435 
[ 8599.150508] 	(detected by 0, t=6502 jiffies, g=44965, q=100)
[ 8599.150620] Task dump for CPU 4:
[ 8599.151302] task:swapper/4       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00000028
[ 8599.152248] Call trace:
[ 8599.152736]  ret_from_fork+0x0/0x30
[ 8599.153876] 
[ 8599.153945] =============================
[ 8599.154084] [ BUG: Invalid wait context ]
[ 8599.154084] 5.10.0-rc2-00006-g0e6c7349e720-dirty #239 Tainted: G        W        
[ 8599.154084] -----------------------------
[ 8599.154084] swapper/0/0 is trying to lock:
[ 8599.154482] ffff000800ab3898 (&port_lock_key){-.-.}-{3:3}, at: pl011_console_write+0x138/0x218
[ 8599.154879] other info that might help us debug this:
[ 8599.155050] context-{2:2}
[ 8599.155177] 2 locks held by swapper/0/0:
[ 8599.155340]  #0: ffff800015efb2b0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x154/0x338
[ 8599.155861]  #1: ffff800015efb3d8 (console_owner){-.-.}-{0:0}, at: console_unlock+0x190/0x6d8
[ 8599.156945] stack backtrace:
[ 8599.156945] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W         5.10.0-rc2-00006-g0e6c7349e720-dirty #239
[ 8599.157263] Hardware name: FVP Base RevC (DT)
[ 8599.157660] Call trace:
[ 8599.157782]  dump_backtrace+0x0/0x1b0
[ 8599.157944]  show_stack+0x20/0x70
[ 8599.158455]  dump_stack+0xf8/0x168
[ 8599.158455]  __lock_acquire+0x8c8/0x1a08
[ 8599.158455]  lock_acquire+0x26c/0x500
[ 8599.158455]  _raw_spin_lock+0x64/0x88
[ 8599.158852]  pl011_console_write+0x138/0x218
[ 8599.159251]  console_unlock+0x2c4/0x6d8
[ 8599.159251]  vprintk_emit+0x15c/0x338
[ 8599.159409]  vprintk_default+0x40/0x50
[ 8599.159409]  vprintk_func+0xfc/0x2b0
[ 8599.159572]  printk+0x68/0x90
[ 8599.159647]  rcu_sched_clock_irq+0x454/0xbb8
[ 8599.159735]  update_process_times+0x68/0xa8
[ 8599.159897]  tick_sched_handle.isra.0+0x3c/0x60
[ 8599.159897]  tick_sched_timer+0x54/0xb0
[ 8599.160060]  __hrtimer_run_queues+0x368/0x6a0
[ 8599.160060]  hrtimer_interrupt+0xf0/0x250
[ 8599.160223]  arch_timer_handler_phys+0x3c/0x50
[ 8599.160223]  handle_percpu_devid_irq+0xd8/0x460
[ 8599.160442]  generic_handle_irq+0x38/0x50
[ 8599.160442]  __handle_domain_irq+0x6c/0xc8
[ 8599.160442]  gic_handle_irq+0x60/0x12c
[ 8599.160442]  el1_irq+0xc0/0x180
[ 8599.160442]  arch_cpu_idle+0x28/0x38
[ 8599.160874]  default_idle_call+0x80/0x390
[ 8599.160874]  do_idle+0x25c/0x2a8
[ 8599.161078]  cpu_startup_entry+0x2c/0x78
[ 8599.161078]  rest_init+0x1b4/0x288
[ 8599.161237]  arch_call_rest_init+0x18/0x24
[ 8599.161237]  start_kernel+0x538/0x574
[ 8677.161944] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8677.162656] rcu: 	4-O...: (0 ticks this GP) idle=002/1/0x4000000000000000 softirq=0/0 fqs=11256 
[ 8677.163479] 	(detected by 0, t=26007 jiffies, g=44965, q=226)
[ 8677.163641] Task dump for CPU 4:
[ 8677.164292] task:swapper/4       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00000028
[ 8677.165106] Call trace:
[ 8677.165594]  ret_from_fork+0x0/0x30
[ 8755.182354] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8755.183009] rcu: 	4-O...: (0 ticks this GP) idle=002/1/0x4000000000000000 softirq=0/0 fqs=19219 
[ 8755.183660] 	(detected by 0, t=45512 jiffies, g=44965, q=609)
[ 8755.183822] Task dump for CPU 4:
[ 8755.184473] task:swapper/4       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00000028
[ 8755.185450] Call trace:
[ 8755.185776]  ret_from_fork+0x0/0x30

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

[-- Attachment #1: Type: text/plain, Size: 2768 bytes --]

On 11/12/20 11:55, Qais Yousef wrote:
> On 11/12/20 10:24, Will Deacon wrote:
> > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > On 11/09/20 13:52, Will Deacon wrote:
> > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > with 32bit features then.
> > > > > > > 
> > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > 
> > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > are 32-bit capable. What do you do in that case?
> > > > > 
> > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > 
> > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > handling that shortly.
> > > 
> > > Sorry for the delayed response.
> > > 
> > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > 
> > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > 
> > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > that with a late CPU. Do you know if the CPUs being integrated into these
> > broken designs have a mismatch at EL1 as well?
> 
> Hmm my test could have been invalid then. We shouldn't have mismatch at EL1,
> for ease of testing I used a hacked up patch to fake asymmetry on Juno. Testing
> on FVP now, it takes time to boot up though..
> 
> Let me re-run this and get you the log from proper environment. Assuming it
> still fails.

Still fails the same on FVP. dmesg attached. There's a splat shortly after
attempting to online CPU 4.

	# cat /sys/devices/system/cpu/online
	0-3
	# cat /sys/devices/system/cpu/aarch32_el0
	0-3

Now while writing this I just realized I tell the FVP to disable aarch32
support at EL0. So this might still make the kernel thinks there's AArch32
support at EL1 - which seems is what makes your series get confused?

Anyway. No real hardware to test on and not sure if I can tell the FVP to
disable AArch32 support at EL1.

/me goes and dig

Thanks

--
Qais Yousef

[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 36330 bytes --]

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd0f0]
[    0.000000] Linux version 5.10.0-rc2-00006-g0e6c7349e720-dirty (qyousef@e107158-lin) (aarch64-linux-gnu-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #239 SMP PREEMPT Thu Nov 12 11:39:59 GMT 2020
[    0.000000] Machine model: FVP Base RevC
[    0.000000] earlycon: pl11 at MMIO 0x000000001c090000 (options '')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000018000000, size 8 MiB
[    0.000000] OF: reserved mem: initialized node vram@18000000, compatible id shared-dma-pool
[    0.000000] cma: Reserved 32 MiB at 0x00000000fe000000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x00000008ffffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x8ff7f7000-0x8ff7f8fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000]   DMA32    [mem 0x00000000c0000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000008ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000880000000-0x00000008ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000008ffffffff]
[    0.000000] On node 0 totalpages: 1048576
[    0.000000]   DMA zone: 4096 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 262144 pages, LIFO batch:63
[    0.000000]   DMA32 zone: 4096 pages used for memmap
[    0.000000]   DMA32 zone: 262144 pages, LIFO batch:63
[    0.000000]   Normal zone: 8192 pages used for memmap
[    0.000000]   Normal zone: 524288 pages, LIFO batch:63
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: Using PSCI v0.1 Function IDs from DT
[    0.000000] percpu: Embedded 51 pages/cpu s168080 r8192 d32624 u208896
[    0.000000] pcpu-alloc: s168080 r8192 d32624 u208896 alloc=51*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5 [0] 6 [0] 7 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1032192
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyAMA0 earlycon=pl011,0x1c090000 root=/dev/vda rootwait allow_mismatched_32bit_el0 maxcpus=4
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000bbfff000-0x00000000bffff000] (64MB)
[    0.000000] Memory: 3901352K/4194304K available (26752K kernel code, 4802K rwdata, 15452K rodata, 54144K init, 11359K bss, 260184K reserved, 32768K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] ftrace: allocating 82932 entries in 324 pages
[    0.000000] ftrace: allocated 324 pages with 3 groups
[    0.000000] Running RCU self tests
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU lockdep checking is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Rude variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 224 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000002f100000
[    0.000000] ITS [mem 0x2f020000-0x2f03ffff]
[    0.000000] ITS@0x000000002f020000: allocated 8192 Devices @880250000 (indirect, esz 8, psz 64K, shr 1)
[    0.000000] ITS@0x000000002f020000: allocated 8192 Virtual CPUs @880260000 (indirect, esz 8, psz 64K, shr 1)
[    0.000000] ITS@0x000000002f020000: allocated 8192 Interrupt Collections @880270000 (flat, esz 8, psz 64K, shr 1)
[    0.000000] GICv3: using LPI property table @0x0000000880280000
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000880290000
[    0.000000] random: get_random_bytes called from start_kernel+0x3a8/0x574 with crng_init=0
[    0.000000] clocksource: arm,sp804: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.000000] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.001387] Failed to initialize '/bus@8000000/motherboard-bus/iofpga-bus@300000000/timer@120000': -22
[    0.005714] arch_timer: cp15 timer(s) running at 25.16MHz (phys).
[    0.005919] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5cdd39714, max_idle_ns: 440795202620 ns
[    0.006286] sched_clock: 56 bits at 25MHz, resolution 39ns, wraps every 4398046511084ns
[    0.020121] Console: colour dummy device 80x25
[    0.022074] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.023050] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.024027] ... MAX_LOCK_DEPTH:          48
[    0.025024] ... MAX_LOCKDEP_KEYS:        8192
[    0.026631] ... CLASSHASH_SIZE:          4096
[    0.027608] ... MAX_LOCKDEP_ENTRIES:     32768
[    0.028422] ... MAX_LOCKDEP_CHAINS:      65536
[    0.029398] ... CHAINHASH_SIZE:          32768
[    0.030958]  memory used by lock dependency info: 6365 kB
[    0.032002]  memory used for stack traces: 4224 kB
[    0.032979]  per task-struct memory footprint: 1920 bytes
[    0.038769] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.33 BogoMIPS (lpj=100663)
[    0.040954] pid_max: default: 32768 minimum: 301
[    0.052020] LSM: Security Framework initializing
[    0.058532] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.059834] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.359022] rcu: Hierarchical SRCU implementation.
[    0.473083] Platform MSI: msi-controller@2f020000 domain created
[    0.482364] PCI/MSI: /interrupt-controller@2f000000/msi-controller@2f020000 domain created
[    0.515889] EFI services will not be available.
[    0.604752] smp: Bringing up secondary CPUs ...
[    0.745022] Detected PIPT I-cache on CPU1
[    0.745419] GICv3: CPU1: found redistributor 100 region 0:0x000000002f120000
[    0.746033] GICv3: CPU1: using allocated LPI pending table @0x00000008802a0000
[    0.747406] CPU1: Booted secondary processor 0x0000000100 [0x410fd0f0]
[    0.933372] Detected PIPT I-cache on CPU2
[    0.934167] GICv3: CPU2: found redistributor 200 region 0:0x000000002f140000
[    0.934565] GICv3: CPU2: using allocated LPI pending table @0x00000008802b0000
[    0.936314] CPU2: Booted secondary processor 0x0000000200 [0x410fd0f0]
[    1.111392] Detected PIPT I-cache on CPU3
[    1.112243] GICv3: CPU3: found redistributor 300 region 0:0x000000002f160000
[    1.112584] GICv3: CPU3: using allocated LPI pending table @0x00000008802c0000
[    1.113802] CPU3: Booted secondary processor 0x0000000300 [0x410fd0f0]
[    1.157328] smp: Brought up 1 node, 4 CPUs
[    1.173380] SMP: Total of 4 processors activated.
[    1.174580] CPU features: detected: 32-bit EL0 Support
[    1.175882] CPU features: detected: 32-bit EL1 Support
[    4.568064] CPU: All CPU(s) started at EL2
[    4.574484] alternatives: patching kernel code
[    4.738533] devtmpfs: initialized
[    5.517367] KASLR disabled due to lack of seed
[    5.659444] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    5.661397] futex hash table entries: 2048 (order: 6, 262144 bytes, linear)
[    5.779398] pinctrl core: initialized pinctrl subsystem
[    6.002946] DMI not present or invalid.
[    6.083214] NET: Registered protocol family 16
[    6.471294] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[    6.480340] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    6.489522] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    6.504023] audit: initializing netlink subsys (disabled)
[    6.525878] audit: type=2000 audit(4.728:1): state=initialized audit_enabled=0 res=1
[    6.707783] thermal_sys: Registered thermal governor 'step_wise'
[    6.708269] thermal_sys: Registered thermal governor 'power_allocator'
[    6.724162] cpuidle: using governor menu
[    6.744893] NET: Registered protocol family 42
[    6.784281] hw-breakpoint: found 16 breakpoint and 16 watchpoint registers.
[    6.802673] ASID allocator initialised with 32768 entries
[    6.990010] Serial: AMBA PL011 UART driver
[   10.396603] 1c090000.serial: ttyAMA0 at MMIO 0x1c090000 (irq = 21, base_baud = 0) is a PL011 rev2
[   10.404561] printk: console [ttyAMA0] enabled
[   10.406677] printk: bootconsole [pl11] disabled
[   10.552510] 1c0a0000.serial: ttyAMA1 at MMIO 0x1c0a0000 (irq = 22, base_baud = 0) is a PL011 rev2
[   10.671813] 1c0b0000.serial: ttyAMA2 at MMIO 0x1c0b0000 (irq = 23, base_baud = 0) is a PL011 rev2
[   10.801915] 1c0c0000.serial: ttyAMA3 at MMIO 0x1c0c0000 (irq = 24, base_baud = 0) is a PL011 rev2
[   16.484622] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[   16.487403] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[   16.489203] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[   16.490824] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[   17.066760] cryptd: max_cpu_qlen set to 1000
[   17.808695] raid6: neonx8   gen()    51 MB/s
[   17.892087] raid6: neonx8   xor()    35 MB/s
[   17.976501] raid6: neonx4   gen()    48 MB/s
[   18.061787] raid6: neonx4   xor()    34 MB/s
[   18.145446] raid6: neonx2   gen()    43 MB/s
[   18.228780] raid6: neonx2   xor()    28 MB/s
[   18.313689] raid6: neonx1   gen()    35 MB/s
[   18.398725] raid6: neonx1   xor()    22 MB/s
[   18.483499] raid6: int64x8  gen()    17 MB/s
[   18.567648] raid6: int64x8  xor()     9 MB/s
[   18.652244] raid6: int64x4  gen()    18 MB/s
[   18.739269] raid6: int64x4  xor()     9 MB/s
[   18.824808] raid6: int64x2  gen()    16 MB/s
[   18.911490] raid6: int64x2  xor()     9 MB/s
[   18.995965] raid6: int64x1  gen()    14 MB/s
[   19.079808] raid6: int64x1  xor()     7 MB/s
[   19.081397] raid6: using algorithm neonx8 gen() 51 MB/s
[   19.082987] raid6: .... xor() 35 MB/s, rmw enabled
[   19.084179] raid6: using neon recovery algorithm
[   19.158910] ACPI: Interpreter disabled.
[   19.375264] iommu: Default domain type: Translated 
[   19.416376] vgaarb: loaded
[   19.497660] SCSI subsystem initialized
[   19.531351] libata version 3.00 loaded.
[   19.582132] usbcore: registered new interface driver usbfs
[   19.592805] usbcore: registered new interface driver hub
[   19.613071] usbcore: registered new device driver usb
[   19.707643] mc: Linux media interface: v0.10
[   19.715597] videodev: Linux video capture interface: v2.00
[   19.737446] pps_core: LinuxPPS API ver. 1 registered
[   19.739035] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[   19.743804] PTP clock support registered
[   19.809184] EDAC MC: Ver: 3.0.0
[   19.987732] FPGA manager framework
[   20.023151] Advanced Linux Sound Architecture Driver Initialized.
[   20.154281] Bluetooth: Core ver 2.22
[   20.160691] NET: Registered protocol family 31
[   20.163023] Bluetooth: HCI device and connection manager initialized
[   20.166997] Bluetooth: HCI socket layer initialized
[   20.169209] Bluetooth: L2CAP socket layer initialized
[   20.174944] Bluetooth: SCO socket layer initialized
[   20.263954] clocksource: Switched to clocksource arch_sys_counter
[  132.044209] VFS: Disk quotas dquot_6.6.0
[  132.056835] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[  132.135911] pnp: PnP ACPI: disabled
[  134.446227] NET: Registered protocol family 2
[  134.529633] tcp_listen_portaddr_hash hash table entries: 2048 (order: 5, 163840 bytes, linear)
[  134.562217] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[  134.673187] TCP bind hash table entries: 32768 (order: 9, 2359296 bytes, linear)
[  134.931578] TCP: Hash tables configured (established 32768 bind 32768)
[  134.965433] UDP hash table entries: 2048 (order: 6, 327680 bytes, linear)
[  135.009252] UDP-Lite hash table entries: 2048 (order: 6, 327680 bytes, linear)
[  135.081481] NET: Registered protocol family 1
[  135.212503] RPC: Registered named UNIX socket transport module.
[  135.216994] RPC: Registered udp transport module.
[  135.220478] RPC: Registered tcp transport module.
[  135.224059] RPC: Registered tcp NFSv4.1 backchannel transport module.
[  135.348204] PCI: CLS 0 bytes, default 64
[ 1598.688719] hw perfevents: enabled with armv8_pmuv3 PMU driver, 9 counters available
[ 1598.705155] kvm [1]: IPA Size Limit: 40 bits
[ 1599.985716] kvm [1]: vgic-v2@2c02f000
[ 1599.992608] kvm [1]: GIC system register CPU interface enabled
[ 1600.019626] kvm [1]: vgic interrupt IRQ9
[ 1600.049294] kvm [1]: Hyp mode initialized successfully
[ 1601.384836] Initialise system trusted keyrings
[ 1601.425225] workingset: timestamp_bits=44 max_order=20 bucket_order=0
[ 1605.817544] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 1606.057839] NFS: Registering the id_resolver key type
[ 1606.065389] Key type id_resolver registered
[ 1606.069977] Key type id_legacy registered
[ 1606.120700] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 1606.168705] fuse: init (API version 7.32)
[ 1606.373793] 9p: Installing v9fs 9p2000 file system support
[ 1606.744515] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 1606.747979] NET: Registered protocol family 38
[ 1606.752861] xor: measuring software checksum speed
[ 1607.123467]    8regs           :    27 MB/sec
[ 1607.400137]    32regs          :    36 MB/sec
[ 1607.589288]    arm64_neon      :    53 MB/sec
[ 1607.592544] xor: using function: arm64_neon (53 MB/sec)
[ 1607.595798] Key type asymmetric registered
[ 1607.600518] Asymmetric key parser 'x509' registered
[ 1607.609180] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[ 1607.613419] io scheduler mq-deadline registered
[ 1607.616700] io scheduler kyber registered
[ 1609.357679] pci-host-generic 40000000.pci: host bridge /pci@40000000 ranges:
[ 1609.365329] pci-host-generic 40000000.pci:      MEM 0x0050000000..0x005fffffff -> 0x0050000000
[ 1609.377976] pci-host-generic 40000000.pci: ECAM at [mem 0x40000000-0x4fffffff] for [bus 00-01]
[ 1609.424707] pci-host-generic 40000000.pci: PCI host bridge to bus 0000:00
[ 1609.428284] pci_bus 0000:00: root bus resource [bus 00-01]
[ 1609.432257] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[ 1609.444756] pci 0000:00:00.0: [1af4:1001] type 00 class 0x018000
[ 1609.450138] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00000fff]
[ 1609.456894] pci 0000:00:00.0: reg 0x18: [mem 0x00000000-0x00000fff]
[ 1609.461684] pci 0000:00:00.0: reg 0x20: [mem 0x00000000-0x00000fff]
[ 1609.484709] pci 0000:00:00.0: PME# supported from D3hot
[ 1609.565374] pci 0000:00:02.0: [1af4:1001] type 00 class 0x018000
[ 1609.572528] pci 0000:00:02.0: reg 0x10: [mem 0x00000000-0x00000fff]
[ 1609.577569] pci 0000:00:02.0: reg 0x18: [mem 0x00000000-0x00000fff]
[ 1609.584567] pci 0000:00:02.0: reg 0x20: [mem 0x00000000-0x00000fff]
[ 1609.604750] pci 0000:00:02.0: PME# supported from D3hot
[ 1609.680759] pci 0000:00:03.0: [0abc:aced] type 00 class 0x010601
[ 1609.685804] pci 0000:00:03.0: reg 0x10: [mem 0x00000000-0x00001fff]
[ 1609.692004] pci 0000:00:03.0: reg 0x14: [mem 0x00000000-0x00001fff]
[ 1609.696384] pci 0000:00:03.0: reg 0x18: [mem 0x00000000-0x00000fff]
[ 1609.700479] pci 0000:00:03.0: reg 0x1c: [mem 0x00000000-0x00001fff]
[ 1609.705010] pci 0000:00:03.0: reg 0x20: [mem 0x00000000-0x00000fff]
[ 1609.709220] pci 0000:00:03.0: reg 0x24: [mem 0x00000000-0x00001fff]
[ 1609.724718] pci 0000:00:03.0: PME# supported from D3hot
[ 1609.848077] pci 0000:00:03.0: BAR 0: assigned [mem 0x50000000-0x50001fff]
[ 1609.851983] pci 0000:00:03.0: BAR 1: assigned [mem 0x50002000-0x50003fff]
[ 1609.856245] pci 0000:00:03.0: BAR 3: assigned [mem 0x50004000-0x50005fff]
[ 1609.860219] pci 0000:00:03.0: BAR 5: assigned [mem 0x50006000-0x50007fff]
[ 1609.864192] pci 0000:00:00.0: BAR 0: assigned [mem 0x50008000-0x50008fff]
[ 1609.868259] pci 0000:00:00.0: BAR 2: assigned [mem 0x50009000-0x50009fff]
[ 1609.872140] pci 0000:00:00.0: BAR 4: assigned [mem 0x5000a000-0x5000afff]
[ 1609.876234] pci 0000:00:02.0: BAR 0: assigned [mem 0x5000b000-0x5000bfff]
[ 1609.880087] pci 0000:00:02.0: BAR 2: assigned [mem 0x5000c000-0x5000cfff]
[ 1609.884060] pci 0000:00:02.0: BAR 4: assigned [mem 0x5000d000-0x5000dfff]
[ 1609.888034] pci 0000:00:03.0: BAR 2: assigned [mem 0x5000e000-0x5000efff]
[ 1609.892008] pci 0000:00:03.0: BAR 4: assigned [mem 0x5000f000-0x5000ffff]
[ 1610.171807] IPMI message handler: version 39.2
[ 1610.191989] ipmi device interface
[ 1610.204359] ipmi_si: IPMI System Interface driver
[ 1610.264633] ipmi_si: Unable to find any System Interface(s)
[ 1610.360103] EINJ: ACPI disabled.
[ 1613.177667] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 1613.888301] SuperH (H)SCI(F) driver initialized
[ 1613.948012] msm_serial: driver initialized
[ 1614.364740] arm-smmu-v3 2b400000.iommu: ias 48-bit, oas 48-bit (features 0x00001fef)
[ 1614.392184] arm-smmu-v3 2b400000.iommu: allocated 65536 entries for cmdq
[ 1614.505975] arm-smmu-v3 2b400000.iommu: allocated 32768 entries for evtq
[ 1616.056898] panel-simple panel: supply power not found, using dummy regulator
[ 1616.143343] ------------[ cut here ]------------
[ 1616.144952] WARNING: CPU: 1 PID: 1 at drivers/gpu/drm/panel/panel-simple.c:467 panel_simple_parse_panel_timing_node+0x1e4/0x1f8
[ 1616.146416] Modules linked in:
[ 1616.148207] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.10.0-rc2-00006-g0e6c7349e720-dirty #239
[ 1616.149701] Hardware name: FVP Base RevC (DT)
[ 1616.151135] pstate: 20000005 (nzCv daif -PAN -UAO -TCO BTYPE=--)
[ 1616.152601] pc : panel_simple_parse_panel_timing_node+0x1e4/0x1f8
[ 1616.154102] lr : panel_simple_probe+0x2b4/0x3e0
[ 1616.155375] sp : ffff80001005ba60
[ 1616.156670] x29: ffff80001005ba60 x28: 0000000000000000 
[ 1616.159188] x27: ffff8000129d04e0 x26: ffff800012ad10d8 
[ 1616.161622] x25: ffff8000161d0a38 x24: 0000000000000000 
[ 1616.164006] x23: ffff8000120d4848 x22: ffff000800388000 
[ 1616.166390] x21: ffff000800ab0010 x20: ffff000800388000 
[ 1616.168937] x19: ffff000800d3b680 x18: 0000000000001460 
[ 1616.171319] x17: 0000000000000004 x16: 0000000000000000 
[ 1616.173597] x15: ffffffffffffffff x14: ffffffff00000000 
[ 1616.176324] x13: ffffffffffffffff x12: 0000000000000020 
[ 1616.178709] x11: 0000000000000004 x10: 0101010101010101 
[ 1616.181093] x9 : ffff800010dcf4ec x8 : 00000000ffffffff 
[ 1616.183526] x7 : 00000000000000e3 x6 : 0000000000000001 
[ 1616.185968] x5 : ffff00087f805b10 x4 : 0000000000000000 
[ 1616.188245] x3 : ffff8000120d4848 x2 : ffff80001005bb18 
[ 1616.190850] x1 : 0000000000000001 x0 : ffff000800ab0010 
[ 1616.193293] Call trace:
[ 1616.194756]  panel_simple_parse_panel_timing_node+0x1e4/0x1f8
[ 1616.196221]  panel_simple_probe+0x2b4/0x3e0
[ 1616.197523]  panel_simple_platform_probe+0x3c/0x58
[ 1616.199030]  platform_drv_probe+0x5c/0xb0
[ 1616.200453]  really_probe+0xec/0x3b8
[ 1616.201918]  driver_probe_device+0x60/0xc0
[ 1616.203345]  device_driver_attach+0x7c/0x88
[ 1616.204538]  __driver_attach+0x60/0xf0
[ 1616.206149]  bus_for_each_dev+0x78/0xd0
[ 1616.207507]  driver_attach+0x2c/0x38
[ 1616.208917]  bus_add_driver+0x158/0x200
[ 1616.210381]  driver_register+0x6c/0x128
[ 1616.211746]  __platform_driver_register+0x50/0x60
[ 1616.213279]  panel_simple_init+0x2c/0x54
[ 1616.214613]  do_one_initcall+0x94/0x4b0
[ 1616.216079]  kernel_init_freeable+0x2c0/0x32c
[ 1616.217253]  kernel_init+0x1c/0x128
[ 1616.218844]  ret_from_fork+0x10/0x30
[ 1616.220223] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.10.0-rc2-00006-g0e6c7349e720-dirty #239
[ 1616.221624] Hardware name: FVP Base RevC (DT)
[ 1616.222915] Call trace:
[ 1616.224406]  dump_backtrace+0x0/0x1b0
[ 1616.225734]  show_stack+0x20/0x70
[ 1616.227187]  dump_stack+0xf8/0x168
[ 1616.228610]  __warn+0xfc/0x180
[ 1616.229968]  report_bug+0xfc/0x170
[ 1616.231377]  bug_handler+0x28/0x70
[ 1616.232750]  call_break_hook+0x70/0x88
[ 1616.234144]  brk_handler+0x24/0x68
[ 1616.235532]  do_debug_exception+0xb8/0x130
[ 1616.236911]  el1_sync_handler+0xd8/0x120
[ 1616.238377]  el1_sync+0x80/0x100
[ 1616.239721]  panel_simple_parse_panel_timing_node+0x1e4/0x1f8
[ 1616.241143]  panel_simple_probe+0x2b4/0x3e0
[ 1616.242608]  panel_simple_platform_probe+0x3c/0x58
[ 1616.244073]  platform_drv_probe+0x5c/0xb0
[ 1616.245465]  really_probe+0xec/0x3b8
[ 1616.246927]  driver_probe_device+0x60/0xc0
[ 1616.248305]  device_driver_attach+0x7c/0x88
[ 1616.249441]  __driver_attach+0x60/0xf0
[ 1616.251164]  bus_for_each_dev+0x78/0xd0
[ 1616.252536]  driver_attach+0x2c/0x38
[ 1616.253838]  bus_add_driver+0x158/0x200
[ 1616.255399]  driver_register+0x6c/0x128
[ 1616.256768]  __platform_driver_register+0x50/0x60
[ 1616.258233]  panel_simple_init+0x2c/0x54
[ 1616.259699]  do_one_initcall+0x94/0x4b0
[ 1616.261000]  kernel_init_freeable+0x2c0/0x32c
[ 1616.262464]  kernel_init+0x1c/0x128
[ 1616.263880]  ret_from_fork+0x10/0x30
[ 1616.265154] irq event stamp: 2125172
[ 1616.266533] hardirqs last  enabled at (2125171): [<ffff800011a970a0>] _raw_spin_unlock_irqrestore+0xa0/0xa8
[ 1616.268115] hardirqs last disabled at (2125172): [<ffff8000100ba094>] debug_exception_enter+0xb4/0xf0
[ 1616.269789] softirqs last  enabled at (2124972): [<ffff800010090a8c>] __do_softirq+0x604/0x684
[ 1616.271295] softirqs last disabled at (2124967): [<ffff800010130c70>] irq_exit+0x180/0x1a0
[ 1616.272884] ---[ end trace 8ab98a1b1a71c11c ]---
[ 1616.276857] panel-simple panel: Reject override mode: panel has a fixed mode
[ 1616.280434] panel-simple panel: Specify missing connector_type
[ 1616.441990] drm-clcd-pl111 1c1f0000.clcd: assigned reserved memory node vram@18000000
[ 1616.445570] drm-clcd-pl111 1c1f0000.clcd: using device-specific reserved memory
[ 1616.449151] drm-clcd-pl111 1c1f0000.clcd: no max memory bandwidth specified, assume unlimited
[ 1616.473553] drm-clcd-pl111 1c1f0000.clcd: DVI muxed to motherboard CLCD
[ 1616.483982] drm-clcd-pl111 1c1f0000.clcd: can't find the sysreg device, deferring
[ 1616.485772] drm-clcd-pl111 1c1f0000.clcd: Versatile Express init failed - -517
[ 1616.632907] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 1621.848727] loop: module loaded
[ 1632.609010] virtio_blk virtio0: [vda] 0 512-byte logical blocks (0 B/0 B)
[ 1633.432663] basic-mmio-gpio: Failed to locate of_node [id: -2]
[ 1633.572138] basic-mmio-gpio: Failed to locate of_node [id: -2]
[ 1633.705986] basic-mmio-gpio: Failed to locate of_node [id: -2]
[ 1636.908623] mpt3sas version 35.100.00.00 loaded
[ 1637.410414] ahci 0000:00:03.0: Adding to iommu group 0
[ 1637.497303] ahci 0000:00:03.0: version 3.0
[ 1637.500908] ahci 0000:00:03.0: enabling device (0000 -> 0002)
[ 1637.560154] ahci 0000:00:03.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
[ 1637.562106] ahci 0000:00:03.0: flags: 64bit ncq only 
[ 1637.809828] scsi host0: ahci
[ 1637.969769] ata1: SATA max UDMA/133 abar m8192@0x50006000 port 0x50006100 irq 41
[ 1638.312297] ata1: SATA link down (SStatus 0 SControl 300)
[ 1639.064548] libphy: Fixed MDIO Bus: probed
[ 1639.517510] tun: Universal TUN/TAP device driver, 1.6
[ 1639.727960] virtio_net: probe of virtio2 failed with error -2
[ 1639.956988] thunder_xcv, ver 1.0
[ 1639.972589] thunder_bgx, ver 1.0
[ 1639.992750] nicpf, ver 1.0
[ 1640.089310] hclge is initializing
[ 1640.108005] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[ 1640.109632] hns3: Copyright (c) 2017 Huawei Corporation.
[ 1640.124932] e1000e: Intel(R) PRO/1000 Network Driver
[ 1640.127854] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[ 1640.141762] igb: Intel(R) Gigabit Ethernet Network Driver
[ 1640.145278] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 1640.156670] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[ 1640.160438] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 1640.233493] sky2: driver version 1.30
[ 1640.410088] smc91x 1a000000.ethernet (unnamed net_device) (uninitialized): smc91x: IOADDR (____ptrval____) doesn't match configuration (300).
[ 1640.413832] smc91x.c: v1.1, sep 22 2004 by Nicolas Pitre <nico@fluxnic.net>
[ 1643.585381] smc91x 1a000000.ethernet eth0: SMC91C11xFD (rev 1) at (____ptrval____) IRQ 15
[ 1643.588897] 
[ 1643.589693] smc91x 1a000000.ethernet eth0: Ethernet addr: 00:02:f7:ef:36:98
[ 1643.969175] pegasus: v0.9.3 (2013/04/25), Pegasus/Pegasus II USB Ethernet driver
[ 1643.980699] usbcore: registered new interface driver pegasus
[ 1643.992619] usbcore: registered new interface driver rtl8150
[ 1644.005733] usbcore: registered new interface driver r8152
[ 1644.016859] usbcore: registered new interface driver lan78xx
[ 1644.029972] usbcore: registered new interface driver asix
[ 1644.041273] usbcore: registered new interface driver ax88179_178a
[ 1644.052829] usbcore: registered new interface driver cdc_ether
[ 1644.062159] usbcore: registered new interface driver dm9601
[ 1644.076838] usbcore: registered new interface driver CoreChips
[ 1644.089938] usbcore: registered new interface driver smsc75xx
[ 1644.105074] usbcore: registered new interface driver smsc95xx
[ 1644.116995] usbcore: registered new interface driver net1080
[ 1644.128121] usbcore: registered new interface driver plusb
[ 1644.137464] usbcore: registered new interface driver cdc_subset
[ 1644.148387] usbcore: registered new interface driver zaurus
[ 1644.159910] usbcore: registered new interface driver MOSCHIP usb-ethernet driver
[ 1644.173901] usbcore: registered new interface driver cdc_ncm
[ 1644.221449] VFIO - User Level meta-driver version: 0.3
[ 1644.648370] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1644.649997] ehci-pci: EHCI PCI platform driver
[ 1644.661879] ehci-platform: EHCI generic platform driver
[ 1644.685641] ehci-orion: EHCI orion driver
[ 1644.709067] ehci-exynos: EHCI Exynos driver
[ 1644.732514] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 1644.737400] ohci-pci: OHCI PCI platform driver
[ 1644.749201] ohci-platform: OHCI generic platform driver
[ 1644.772881] ohci-exynos: OHCI Exynos driver
[ 1644.868410] usbcore: registered new interface driver usb-storage
[ 1645.674265] rtc-pl031 1c170000.rtc: registered as rtc0
[ 1645.681247] rtc-pl031 1c170000.rtc: setting system clock to 2020-11-12T15:38:43 UTC (1605195523)
[ 1645.836391] i2c /dev entries driver
[ 1646.276670] usbcore: registered new interface driver uvcvideo
[ 1646.279848] USB Video Class driver (1.1.1)
[ 1646.281439] gspca_main: v2.14.0 registered
[ 1647.137302] sp805-wdt 1c0f0000.wdt: registration successful
[ 1647.452471] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com
[ 1647.457637] Bluetooth: HCI UART driver ver 2.3
[ 1647.460898] Bluetooth: HCI UART protocol H4 registered
[ 1647.468365] Bluetooth: HCI UART protocol LL registered
[ 1647.492264] Bluetooth: HCI UART protocol Broadcom registered
[ 1647.499757] Bluetooth: HCI UART protocol QCA registered
[ 1647.513117] usbcore: registered new interface driver btusb
[ 1647.732613] mmci-pl18x 1c050000.mmci: Got CD GPIO
[ 1647.740818] mmci-pl18x 1c050000.mmci: Got WP GPIO
[ 1647.864027] mmci-pl18x 1c050000.mmci: mmc0: PL180 manf 41 rev0 at 0x1c050000 irq 17,18 (pio)
[ 1648.063617] sdhci: Secure Digital Host Controller Interface driver
[ 1648.065361] sdhci: Copyright(c) Pierre Ossman
[ 1648.131963] Synopsys Designware Multimedia Card Interface Driver
[ 1648.305759] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1648.313956] input: PS/2 Generic Mouse as /devices/platform/bus@8000000/bus@8000000:motherboard-bus/bus@8000000:motherboard-bus:iofpga-bus@300000000/1c070000.kmi/serio1/input/input1
[ 1648.588933] ledtrig-cpu: registered to indicate activity on CPUs
[ 1648.909605] input: AT Raw Set 2 keyboard as /devices/platform/bus@8000000/bus@8000000:motherboard-bus/bus@8000000:motherboard-bus:iofpga-bus@300000000/1c060000.kmi/serio0/input/input2
[ 1649.000999] usbcore: registered new interface driver usbhid
[ 1649.004178] usbhid: USB HID core driver
[ 1650.760131] drop_monitor: Initializing network drop monitor service
[ 1651.172295] NET: Registered protocol family 10
[ 1651.436842] Segment Routing with IPv6
[ 1651.503998] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 1651.668506] NET: Registered protocol family 17
[ 1651.682016] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[ 1651.685592] Bluetooth: HIDP socket layer initialized
[ 1651.697362] 8021q: 802.1Q VLAN Support v1.8
[ 1651.769463] 9pnet: Installing 9P2000 support
[ 1651.876328] Key type dns_resolver registered
[ 1651.997166] registered taskstats version 1
[ 1652.001659] Loading compiled-in X.509 certificates
[ 1652.388279] Btrfs loaded, crc32c=crc32c-generic
[ 1653.089124] virtio-pci 0000:00:00.0: Adding to iommu group 1
[ 1653.170017] virtio-pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 1653.177298] virtio-pci 0000:00:00.0: virtio_pci: leaving for legacy driver
[ 1653.470155] virtio_blk virtio3: [vdb] 0 512-byte logical blocks (0 B/0 B)
[ 1654.545215] virtio-pci 0000:00:02.0: Adding to iommu group 2
[ 1654.616794] virtio-pci 0000:00:02.0: enabling device (0000 -> 0002)
[ 1654.624281] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver
[ 1654.852188] virtio_blk virtio4: [vdc] 0 512-byte logical blocks (0 B/0 B)
[ 1655.924899] drm-clcd-pl111 1c1f0000.clcd: assigned reserved memory node vram@18000000
[ 1655.929046] drm-clcd-pl111 1c1f0000.clcd: using device-specific reserved memory
[ 1655.933417] drm-clcd-pl111 1c1f0000.clcd: no max memory bandwidth specified, assume unlimited
[ 1655.965604] drm-clcd-pl111 1c1f0000.clcd: DVI muxed to motherboard CLCD
[ 1656.032523] drm-clcd-pl111 1c1f0000.clcd: initializing Versatile Express PL111
[ 1656.105078] drm-clcd-pl111 1c1f0000.clcd: found panel on endpoint 0
[ 1656.381644] [drm] Initialized pl111 1.0.0 20170317 for 1c1f0000.clcd on minor 0
[ 1657.562213] Console: switching to colour frame buffer device 128x48
[ 1658.301312] drm-clcd-pl111 1c1f0000.clcd: [drm] fb0: pl111drmfb frame buffer device
[ 1658.471780] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 1659.821384] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 1659.860171] ALSA device list:
[ 1659.871695]   No soundcards found.
[ 1659.918064] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ 1659.924544] platform regulatory.0: Falling back to sysfs fallback for: regulatory.db
[ 1659.944412] uart-pl011 1c090000.serial: no DMA platform data
[ 1665.641822] Freeing unused kernel memory: 54144K
[ 1665.656524] Run /init as init process
[ 1665.657716]   with arguments:
[ 1665.660100]     /init
[ 1665.660895]   with environment:
[ 1665.661553]     HOME=/
[ 1665.663676]     TERM=linux
[ 1720.461405] cfg80211: failed to load regulatory.db
[ 2264.157156] random: dd: uninitialized urandom read (512 bytes read)
[ 2391.928388] random: crng init done
[ 2400.436615] smc91x 1a000000.ethernet eth0: link up, 10Mbps, half-duplex, lpa 0x0000
[ 8569.024731] CPU features: CPU4: Detected conflict for capability 53 (32-bit EL1 Support), System: 1, CPU: 0
[ 8569.025525] CPU4: will not boot
[ 8574.269542] CPU4: failed to come online
[ 8574.282258] CPU4: died during early boot
[ 8599.142560] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8599.149806] rcu: 	4-O...: (0 ticks this GP) idle=002/1/0x4000000000000000 softirq=0/0 fqs=2435 
[ 8599.150508] 	(detected by 0, t=6502 jiffies, g=44965, q=100)
[ 8599.150620] Task dump for CPU 4:
[ 8599.151302] task:swapper/4       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00000028
[ 8599.152248] Call trace:
[ 8599.152736]  ret_from_fork+0x0/0x30
[ 8599.153876] 
[ 8599.153945] =============================
[ 8599.154084] [ BUG: Invalid wait context ]
[ 8599.154084] 5.10.0-rc2-00006-g0e6c7349e720-dirty #239 Tainted: G        W        
[ 8599.154084] -----------------------------
[ 8599.154084] swapper/0/0 is trying to lock:
[ 8599.154482] ffff000800ab3898 (&port_lock_key){-.-.}-{3:3}, at: pl011_console_write+0x138/0x218
[ 8599.154879] other info that might help us debug this:
[ 8599.155050] context-{2:2}
[ 8599.155177] 2 locks held by swapper/0/0:
[ 8599.155340]  #0: ffff800015efb2b0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x154/0x338
[ 8599.155861]  #1: ffff800015efb3d8 (console_owner){-.-.}-{0:0}, at: console_unlock+0x190/0x6d8
[ 8599.156945] stack backtrace:
[ 8599.156945] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W         5.10.0-rc2-00006-g0e6c7349e720-dirty #239
[ 8599.157263] Hardware name: FVP Base RevC (DT)
[ 8599.157660] Call trace:
[ 8599.157782]  dump_backtrace+0x0/0x1b0
[ 8599.157944]  show_stack+0x20/0x70
[ 8599.158455]  dump_stack+0xf8/0x168
[ 8599.158455]  __lock_acquire+0x8c8/0x1a08
[ 8599.158455]  lock_acquire+0x26c/0x500
[ 8599.158455]  _raw_spin_lock+0x64/0x88
[ 8599.158852]  pl011_console_write+0x138/0x218
[ 8599.159251]  console_unlock+0x2c4/0x6d8
[ 8599.159251]  vprintk_emit+0x15c/0x338
[ 8599.159409]  vprintk_default+0x40/0x50
[ 8599.159409]  vprintk_func+0xfc/0x2b0
[ 8599.159572]  printk+0x68/0x90
[ 8599.159647]  rcu_sched_clock_irq+0x454/0xbb8
[ 8599.159735]  update_process_times+0x68/0xa8
[ 8599.159897]  tick_sched_handle.isra.0+0x3c/0x60
[ 8599.159897]  tick_sched_timer+0x54/0xb0
[ 8599.160060]  __hrtimer_run_queues+0x368/0x6a0
[ 8599.160060]  hrtimer_interrupt+0xf0/0x250
[ 8599.160223]  arch_timer_handler_phys+0x3c/0x50
[ 8599.160223]  handle_percpu_devid_irq+0xd8/0x460
[ 8599.160442]  generic_handle_irq+0x38/0x50
[ 8599.160442]  __handle_domain_irq+0x6c/0xc8
[ 8599.160442]  gic_handle_irq+0x60/0x12c
[ 8599.160442]  el1_irq+0xc0/0x180
[ 8599.160442]  arch_cpu_idle+0x28/0x38
[ 8599.160874]  default_idle_call+0x80/0x390
[ 8599.160874]  do_idle+0x25c/0x2a8
[ 8599.161078]  cpu_startup_entry+0x2c/0x78
[ 8599.161078]  rest_init+0x1b4/0x288
[ 8599.161237]  arch_call_rest_init+0x18/0x24
[ 8599.161237]  start_kernel+0x538/0x574
[ 8677.161944] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8677.162656] rcu: 	4-O...: (0 ticks this GP) idle=002/1/0x4000000000000000 softirq=0/0 fqs=11256 
[ 8677.163479] 	(detected by 0, t=26007 jiffies, g=44965, q=226)
[ 8677.163641] Task dump for CPU 4:
[ 8677.164292] task:swapper/4       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00000028
[ 8677.165106] Call trace:
[ 8677.165594]  ret_from_fork+0x0/0x30
[ 8755.182354] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8755.183009] rcu: 	4-O...: (0 ticks this GP) idle=002/1/0x4000000000000000 softirq=0/0 fqs=19219 
[ 8755.183660] 	(detected by 0, t=45512 jiffies, g=44965, q=609)
[ 8755.183822] Task dump for CPU 4:
[ 8755.184473] task:swapper/4       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00000028
[ 8755.185450] Call trace:
[ 8755.185776]  ret_from_fork+0x0/0x30

[-- Attachment #3: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Marc Zyngier]

On 2020-11-12 16:49, Qais Yousef wrote:
> On 11/12/20 11:55, Qais Yousef wrote:
>> On 11/12/20 10:24, Will Deacon wrote:
>> > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
>> > > On 11/09/20 13:52, Will Deacon wrote:
>> > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
>> > > > > On 11/06/20 13:00, Will Deacon wrote:
>> > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
>> > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
>> > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
>> > > > > > > with 32bit features then.
>> > > > > > >
>> > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
>> > > > > >
>> > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
>> > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
>> > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
>> > > > > > are 32-bit capable. What do you do in that case?
>> > > > >
>> > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
>> > > > > attempt to bring the remaining 64bit-only cpus online later?
>> > > >
>> > > > Right. I think we will refuse to online them. I'll post my attempt at
>> > > > handling that shortly.
>> > >
>> > > Sorry for the delayed response.
>> > >
>> > > You're right, I tried that and they refuse to come online. We missed that tbh.
>> > >
>> > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
>> >
>> > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
>> > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
>> > that with a late CPU. Do you know if the CPUs being integrated into these
>> > broken designs have a mismatch at EL1 as well?
>> 
>> Hmm my test could have been invalid then. We shouldn't have mismatch 
>> at EL1,
>> for ease of testing I used a hacked up patch to fake asymmetry on 
>> Juno. Testing
>> on FVP now, it takes time to boot up though..
>> 
>> Let me re-run this and get you the log from proper environment. 
>> Assuming it
>> still fails.
> 
> Still fails the same on FVP. dmesg attached. There's a splat shortly 
> after
> attempting to online CPU 4.
> 
> 	# cat /sys/devices/system/cpu/online
> 	0-3
> 	# cat /sys/devices/system/cpu/aarch32_el0
> 	0-3
> 
> Now while writing this I just realized I tell the FVP to disable 
> aarch32
> support at EL0. So this might still make the kernel thinks there's 
> AArch32
> support at EL1 - which seems is what makes your series get confused?

You can't have AArch32 at EL1 and not have it at EL0.

> Anyway. No real hardware to test on and not sure if I can tell the FVP 
> to
> disable AArch32 support at EL1.
> 
> /me goes and dig

         -C cluster0.max_32bit_el=-1     # no 32bit support whatsoever
         -C cluster1.max_32bit_el=0      # 32bit support at EL0 only

         M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Marc Zyngier]

On 2020-11-12 16:49, Qais Yousef wrote:
> On 11/12/20 11:55, Qais Yousef wrote:
>> On 11/12/20 10:24, Will Deacon wrote:
>> > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
>> > > On 11/09/20 13:52, Will Deacon wrote:
>> > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
>> > > > > On 11/06/20 13:00, Will Deacon wrote:
>> > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
>> > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
>> > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
>> > > > > > > with 32bit features then.
>> > > > > > >
>> > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
>> > > > > >
>> > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
>> > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
>> > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
>> > > > > > are 32-bit capable. What do you do in that case?
>> > > > >
>> > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
>> > > > > attempt to bring the remaining 64bit-only cpus online later?
>> > > >
>> > > > Right. I think we will refuse to online them. I'll post my attempt at
>> > > > handling that shortly.
>> > >
>> > > Sorry for the delayed response.
>> > >
>> > > You're right, I tried that and they refuse to come online. We missed that tbh.
>> > >
>> > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
>> >
>> > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
>> > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
>> > that with a late CPU. Do you know if the CPUs being integrated into these
>> > broken designs have a mismatch at EL1 as well?
>> 
>> Hmm my test could have been invalid then. We shouldn't have mismatch 
>> at EL1,
>> for ease of testing I used a hacked up patch to fake asymmetry on 
>> Juno. Testing
>> on FVP now, it takes time to boot up though..
>> 
>> Let me re-run this and get you the log from proper environment. 
>> Assuming it
>> still fails.
> 
> Still fails the same on FVP. dmesg attached. There's a splat shortly 
> after
> attempting to online CPU 4.
> 
> 	# cat /sys/devices/system/cpu/online
> 	0-3
> 	# cat /sys/devices/system/cpu/aarch32_el0
> 	0-3
> 
> Now while writing this I just realized I tell the FVP to disable 
> aarch32
> support at EL0. So this might still make the kernel thinks there's 
> AArch32
> support at EL1 - which seems is what makes your series get confused?

You can't have AArch32 at EL1 and not have it at EL0.

> Anyway. No real hardware to test on and not sure if I can tell the FVP 
> to
> disable AArch32 support at EL1.
> 
> /me goes and dig

         -C cluster0.max_32bit_el=-1     # no 32bit support whatsoever
         -C cluster1.max_32bit_el=0      # 32bit support at EL0 only

         M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Thu, Nov 12, 2020 at 04:49:43PM +0000, Qais Yousef wrote:
> On 11/12/20 11:55, Qais Yousef wrote:
> > On 11/12/20 10:24, Will Deacon wrote:
> > > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > > On 11/09/20 13:52, Will Deacon wrote:
> > > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > > with 32bit features then.
> > > > > > > > 
> > > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > > 
> > > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > > are 32-bit capable. What do you do in that case?
> > > > > > 
> > > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > > 
> > > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > > handling that shortly.
> > > > 
> > > > Sorry for the delayed response.
> > > > 
> > > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > > 
> > > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > > 
> > > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > > that with a late CPU. Do you know if the CPUs being integrated into these
> > > broken designs have a mismatch at EL1 as well?
> > 
> > Hmm my test could have been invalid then. We shouldn't have mismatch at EL1,
> > for ease of testing I used a hacked up patch to fake asymmetry on Juno. Testing
> > on FVP now, it takes time to boot up though..
> > 
> > Let me re-run this and get you the log from proper environment. Assuming it
> > still fails.
> 
> Still fails the same on FVP. dmesg attached. There's a splat shortly after
> attempting to online CPU 4.
> 
> 	# cat /sys/devices/system/cpu/online
> 	0-3
> 	# cat /sys/devices/system/cpu/aarch32_el0
> 	0-3
> 
> Now while writing this I just realized I tell the FVP to disable aarch32
> support at EL0. So this might still make the kernel thinks there's AArch32
> support at EL1 - which seems is what makes your series get confused?

 | CPU features: CPU4: Detected conflict for capability 53 (32-bit EL1 Support), System: 1, CPU: 0

So we're doing the right thing here.

I've already queued fixes for the splat in for-next/fixes, as it's
unrelated.

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Thu, Nov 12, 2020 at 04:49:43PM +0000, Qais Yousef wrote:
> On 11/12/20 11:55, Qais Yousef wrote:
> > On 11/12/20 10:24, Will Deacon wrote:
> > > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > > On 11/09/20 13:52, Will Deacon wrote:
> > > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > > with 32bit features then.
> > > > > > > > 
> > > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > > 
> > > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > > are 32-bit capable. What do you do in that case?
> > > > > > 
> > > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > > 
> > > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > > handling that shortly.
> > > > 
> > > > Sorry for the delayed response.
> > > > 
> > > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > > 
> > > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > > 
> > > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > > that with a late CPU. Do you know if the CPUs being integrated into these
> > > broken designs have a mismatch at EL1 as well?
> > 
> > Hmm my test could have been invalid then. We shouldn't have mismatch at EL1,
> > for ease of testing I used a hacked up patch to fake asymmetry on Juno. Testing
> > on FVP now, it takes time to boot up though..
> > 
> > Let me re-run this and get you the log from proper environment. Assuming it
> > still fails.
> 
> Still fails the same on FVP. dmesg attached. There's a splat shortly after
> attempting to online CPU 4.
> 
> 	# cat /sys/devices/system/cpu/online
> 	0-3
> 	# cat /sys/devices/system/cpu/aarch32_el0
> 	0-3
> 
> Now while writing this I just realized I tell the FVP to disable aarch32
> support at EL0. So this might still make the kernel thinks there's AArch32
> support at EL1 - which seems is what makes your series get confused?

 | CPU features: CPU4: Detected conflict for capability 53 (32-bit EL1 Support), System: 1, CPU: 0

So we're doing the right thing here.

I've already queued fixes for the splat in for-next/fixes, as it's
unrelated.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/12/20 17:06, Marc Zyngier wrote:
> On 2020-11-12 16:49, Qais Yousef wrote:
> > On 11/12/20 11:55, Qais Yousef wrote:
> > > On 11/12/20 10:24, Will Deacon wrote:
> > > > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > > > On 11/09/20 13:52, Will Deacon wrote:
> > > > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > > > with 32bit features then.
> > > > > > > > >
> > > > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > > >
> > > > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > > > are 32-bit capable. What do you do in that case?
> > > > > > >
> > > > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > > >
> > > > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > > > handling that shortly.
> > > > >
> > > > > Sorry for the delayed response.
> > > > >
> > > > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > > >
> > > > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > > >
> > > > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > > > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > > > that with a late CPU. Do you know if the CPUs being integrated into these
> > > > broken designs have a mismatch at EL1 as well?
> > > 
> > > Hmm my test could have been invalid then. We shouldn't have mismatch
> > > at EL1,
> > > for ease of testing I used a hacked up patch to fake asymmetry on
> > > Juno. Testing
> > > on FVP now, it takes time to boot up though..
> > > 
> > > Let me re-run this and get you the log from proper environment.
> > > Assuming it
> > > still fails.
> > 
> > Still fails the same on FVP. dmesg attached. There's a splat shortly
> > after
> > attempting to online CPU 4.
> > 
> > 	# cat /sys/devices/system/cpu/online
> > 	0-3
> > 	# cat /sys/devices/system/cpu/aarch32_el0
> > 	0-3
> > 
> > Now while writing this I just realized I tell the FVP to disable aarch32
> > support at EL0. So this might still make the kernel thinks there's
> > AArch32
> > support at EL1 - which seems is what makes your series get confused?
> 
> You can't have AArch32 at EL1 and not have it at EL0.
> 
> > Anyway. No real hardware to test on and not sure if I can tell the FVP
> > to
> > disable AArch32 support at EL1.
> > 
> > /me goes and dig
> 
>         -C cluster0.max_32bit_el=-1     # no 32bit support whatsoever
>         -C cluster1.max_32bit_el=0      # 32bit support at EL0 only

Ah okay. That's the option I use. I must have misinterpreted Will's words then
'blatting 32-bit EL1'. Blame my English :-)

Thanks

--
Qais Yousef

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/12/20 17:06, Marc Zyngier wrote:
> On 2020-11-12 16:49, Qais Yousef wrote:
> > On 11/12/20 11:55, Qais Yousef wrote:
> > > On 11/12/20 10:24, Will Deacon wrote:
> > > > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > > > On 11/09/20 13:52, Will Deacon wrote:
> > > > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > > > with 32bit features then.
> > > > > > > > >
> > > > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > > >
> > > > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > > > are 32-bit capable. What do you do in that case?
> > > > > > >
> > > > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > > >
> > > > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > > > handling that shortly.
> > > > >
> > > > > Sorry for the delayed response.
> > > > >
> > > > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > > >
> > > > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > > >
> > > > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > > > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > > > that with a late CPU. Do you know if the CPUs being integrated into these
> > > > broken designs have a mismatch at EL1 as well?
> > > 
> > > Hmm my test could have been invalid then. We shouldn't have mismatch
> > > at EL1,
> > > for ease of testing I used a hacked up patch to fake asymmetry on
> > > Juno. Testing
> > > on FVP now, it takes time to boot up though..
> > > 
> > > Let me re-run this and get you the log from proper environment.
> > > Assuming it
> > > still fails.
> > 
> > Still fails the same on FVP. dmesg attached. There's a splat shortly
> > after
> > attempting to online CPU 4.
> > 
> > 	# cat /sys/devices/system/cpu/online
> > 	0-3
> > 	# cat /sys/devices/system/cpu/aarch32_el0
> > 	0-3
> > 
> > Now while writing this I just realized I tell the FVP to disable aarch32
> > support at EL0. So this might still make the kernel thinks there's
> > AArch32
> > support at EL1 - which seems is what makes your series get confused?
> 
> You can't have AArch32 at EL1 and not have it at EL0.
> 
> > Anyway. No real hardware to test on and not sure if I can tell the FVP
> > to
> > disable AArch32 support at EL1.
> > 
> > /me goes and dig
> 
>         -C cluster0.max_32bit_el=-1     # no 32bit support whatsoever
>         -C cluster1.max_32bit_el=0      # 32bit support at EL0 only

Ah okay. That's the option I use. I must have misinterpreted Will's words then
'blatting 32-bit EL1'. Blame my English :-)

Thanks

--
Qais Yousef

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Thu, Nov 12, 2020 at 05:36:45PM +0000, Qais Yousef wrote:
> On 11/12/20 17:06, Marc Zyngier wrote:
> > On 2020-11-12 16:49, Qais Yousef wrote:
> > > On 11/12/20 11:55, Qais Yousef wrote:
> > > > On 11/12/20 10:24, Will Deacon wrote:
> > > > > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > > > > On 11/09/20 13:52, Will Deacon wrote:
> > > > > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > > > > with 32bit features then.
> > > > > > > > > >
> > > > > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > > > >
> > > > > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > > > > are 32-bit capable. What do you do in that case?
> > > > > > > >
> > > > > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > > > >
> > > > > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > > > > handling that shortly.
> > > > > >
> > > > > > Sorry for the delayed response.
> > > > > >
> > > > > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > > > >
> > > > > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > > > >
> > > > > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > > > > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > > > > that with a late CPU. Do you know if the CPUs being integrated into these
> > > > > broken designs have a mismatch at EL1 as well?
> > > > 
> > > > Hmm my test could have been invalid then. We shouldn't have mismatch
> > > > at EL1,
> > > > for ease of testing I used a hacked up patch to fake asymmetry on
> > > > Juno. Testing
> > > > on FVP now, it takes time to boot up though..
> > > > 
> > > > Let me re-run this and get you the log from proper environment.
> > > > Assuming it
> > > > still fails.
> > > 
> > > Still fails the same on FVP. dmesg attached. There's a splat shortly
> > > after
> > > attempting to online CPU 4.
> > > 
> > > 	# cat /sys/devices/system/cpu/online
> > > 	0-3
> > > 	# cat /sys/devices/system/cpu/aarch32_el0
> > > 	0-3
> > > 
> > > Now while writing this I just realized I tell the FVP to disable aarch32
> > > support at EL0. So this might still make the kernel thinks there's
> > > AArch32
> > > support at EL1 - which seems is what makes your series get confused?
> > 
> > You can't have AArch32 at EL1 and not have it at EL0.
> > 
> > > Anyway. No real hardware to test on and not sure if I can tell the FVP
> > > to
> > > disable AArch32 support at EL1.
> > > 
> > > /me goes and dig
> > 
> >         -C cluster0.max_32bit_el=-1     # no 32bit support whatsoever
> >         -C cluster1.max_32bit_el=0      # 32bit support at EL0 only
> 
> Ah okay. That's the option I use. I must have misinterpreted Will's words then
> 'blatting 32-bit EL1'. Blame my English :-)

I don't think "blatting" is a word, so I'm entirely to blame here!

Will

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Will Deacon]

On Thu, Nov 12, 2020 at 05:36:45PM +0000, Qais Yousef wrote:
> On 11/12/20 17:06, Marc Zyngier wrote:
> > On 2020-11-12 16:49, Qais Yousef wrote:
> > > On 11/12/20 11:55, Qais Yousef wrote:
> > > > On 11/12/20 10:24, Will Deacon wrote:
> > > > > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > > > > On 11/09/20 13:52, Will Deacon wrote:
> > > > > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > > > > with 32bit features then.
> > > > > > > > > >
> > > > > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > > > >
> > > > > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > > > > are 32-bit capable. What do you do in that case?
> > > > > > > >
> > > > > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > > > >
> > > > > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > > > > handling that shortly.
> > > > > >
> > > > > > Sorry for the delayed response.
> > > > > >
> > > > > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > > > >
> > > > > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > > > >
> > > > > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > > > > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > > > > that with a late CPU. Do you know if the CPUs being integrated into these
> > > > > broken designs have a mismatch at EL1 as well?
> > > > 
> > > > Hmm my test could have been invalid then. We shouldn't have mismatch
> > > > at EL1,
> > > > for ease of testing I used a hacked up patch to fake asymmetry on
> > > > Juno. Testing
> > > > on FVP now, it takes time to boot up though..
> > > > 
> > > > Let me re-run this and get you the log from proper environment.
> > > > Assuming it
> > > > still fails.
> > > 
> > > Still fails the same on FVP. dmesg attached. There's a splat shortly
> > > after
> > > attempting to online CPU 4.
> > > 
> > > 	# cat /sys/devices/system/cpu/online
> > > 	0-3
> > > 	# cat /sys/devices/system/cpu/aarch32_el0
> > > 	0-3
> > > 
> > > Now while writing this I just realized I tell the FVP to disable aarch32
> > > support at EL0. So this might still make the kernel thinks there's
> > > AArch32
> > > support at EL1 - which seems is what makes your series get confused?
> > 
> > You can't have AArch32 at EL1 and not have it at EL0.
> > 
> > > Anyway. No real hardware to test on and not sure if I can tell the FVP
> > > to
> > > disable AArch32 support at EL1.
> > > 
> > > /me goes and dig
> > 
> >         -C cluster0.max_32bit_el=-1     # no 32bit support whatsoever
> >         -C cluster1.max_32bit_el=0      # 32bit support at EL0 only
> 
> Ah okay. That's the option I use. I must have misinterpreted Will's words then
> 'blatting 32-bit EL1'. Blame my English :-)

I don't think "blatting" is a word, so I'm entirely to blame here!

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/12/20 17:36, Will Deacon wrote:
> On Thu, Nov 12, 2020 at 04:49:43PM +0000, Qais Yousef wrote:
> > On 11/12/20 11:55, Qais Yousef wrote:
> > > On 11/12/20 10:24, Will Deacon wrote:
> > > > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > > > On 11/09/20 13:52, Will Deacon wrote:
> > > > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > > > with 32bit features then.
> > > > > > > > > 
> > > > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > > > 
> > > > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > > > are 32-bit capable. What do you do in that case?
> > > > > > > 
> > > > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > > > 
> > > > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > > > handling that shortly.
> > > > > 
> > > > > Sorry for the delayed response.
> > > > > 
> > > > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > > > 
> > > > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > > > 
> > > > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > > > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > > > that with a late CPU. Do you know if the CPUs being integrated into these
> > > > broken designs have a mismatch at EL1 as well?
> > > 
> > > Hmm my test could have been invalid then. We shouldn't have mismatch at EL1,
> > > for ease of testing I used a hacked up patch to fake asymmetry on Juno. Testing
> > > on FVP now, it takes time to boot up though..
> > > 
> > > Let me re-run this and get you the log from proper environment. Assuming it
> > > still fails.
> > 
> > Still fails the same on FVP. dmesg attached. There's a splat shortly after
> > attempting to online CPU 4.
> > 
> > 	# cat /sys/devices/system/cpu/online
> > 	0-3
> > 	# cat /sys/devices/system/cpu/aarch32_el0
> > 	0-3
> > 
> > Now while writing this I just realized I tell the FVP to disable aarch32
> > support at EL0. So this might still make the kernel thinks there's AArch32
> > support at EL1 - which seems is what makes your series get confused?
> 
>  | CPU features: CPU4: Detected conflict for capability 53 (32-bit EL1 Support), System: 1, CPU: 0
> 
> So we're doing the right thing here.

Okay. So the systems we test on will always fail until we have real hardware.
Makes sense.

> I've already queued fixes for the splat in for-next/fixes, as it's
> unrelated.

Great thanks!

Cheers

--
Qais Yousef

Re: [PATCH 2/6] arm64: Allow mismatched 32-bit EL0 support

[Qais Yousef]

On 11/12/20 17:36, Will Deacon wrote:
> On Thu, Nov 12, 2020 at 04:49:43PM +0000, Qais Yousef wrote:
> > On 11/12/20 11:55, Qais Yousef wrote:
> > > On 11/12/20 10:24, Will Deacon wrote:
> > > > On Wed, Nov 11, 2020 at 04:27:00PM +0000, Qais Yousef wrote:
> > > > > On 11/09/20 13:52, Will Deacon wrote:
> > > > > > On Fri, Nov 06, 2020 at 02:48:35PM +0000, Qais Yousef wrote:
> > > > > > > On 11/06/20 13:00, Will Deacon wrote:
> > > > > > > > On Fri, Nov 06, 2020 at 12:54:25PM +0000, Qais Yousef wrote:
> > > > > > > > > FWIW I have my v3 over here in case it's of any help. It solves the problem of
> > > > > > > > > HWCAP discovery when late AArch32 CPU is booted by populating boot_cpu_date
> > > > > > > > > with 32bit features then.
> > > > > > > > > 
> > > > > > > > > 	git clone https://git.gitlab.arm.com/linux-arm/linux-qy.git -b asym-aarch32-upstream-v3 origin/asym-aarch32-upstream-v3
> > > > > > > > 
> > > > > > > > Cheers, I've done something similar. I was hoping to post it today, but I've
> > > > > > > > been side-tracked with bug fixing this morning. The main headache I ended up
> > > > > > > > with was allowing late-onlining of 64-bit-only CPUs if all the boot CPUs
> > > > > > > > are 32-bit capable. What do you do in that case?
> > > > > > > 
> > > > > > > Do you mean if CPUs 0-3 were 32bit capable and we boot with maxcpus=4 then
> > > > > > > attempt to bring the remaining 64bit-only cpus online later?
> > > > > > 
> > > > > > Right. I think we will refuse to online them. I'll post my attempt at
> > > > > > handling that shortly.
> > > > > 
> > > > > Sorry for the delayed response.
> > > > > 
> > > > > You're right, I tried that and they refuse to come online. We missed that tbh.
> > > > > 
> > > > > Haven't thought what we should do yet. I tried your v2 and it failed similarly.
> > > > 
> > > > Hmm, it shouldn't do. Please could you provide the log? My hunch is that you
> > > > are blatting 32-bit EL1 support as well, and we can't handle a mismatch for
> > > > that with a late CPU. Do you know if the CPUs being integrated into these
> > > > broken designs have a mismatch at EL1 as well?
> > > 
> > > Hmm my test could have been invalid then. We shouldn't have mismatch at EL1,
> > > for ease of testing I used a hacked up patch to fake asymmetry on Juno. Testing
> > > on FVP now, it takes time to boot up though..
> > > 
> > > Let me re-run this and get you the log from proper environment. Assuming it
> > > still fails.
> > 
> > Still fails the same on FVP. dmesg attached. There's a splat shortly after
> > attempting to online CPU 4.
> > 
> > 	# cat /sys/devices/system/cpu/online
> > 	0-3
> > 	# cat /sys/devices/system/cpu/aarch32_el0
> > 	0-3
> > 
> > Now while writing this I just realized I tell the FVP to disable aarch32
> > support at EL0. So this might still make the kernel thinks there's AArch32
> > support at EL1 - which seems is what makes your series get confused?
> 
>  | CPU features: CPU4: Detected conflict for capability 53 (32-bit EL1 Support), System: 1, CPU: 0
> 
> So we're doing the right thing here.

Okay. So the systems we test on will always fail until we have real hardware.
Makes sense.

> I've already queued fixes for the splat in for-next/fixes, as it's
> unrelated.

Great thanks!

Cheers

--
Qais Yousef

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel