From: Mark Rutland <mark.rutland@arm.com> To: Will Deacon <will@kernel.org> Cc: linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas <catalin.marinas@arm.com>, Marc Zyngier <maz@kernel.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Peter Zijlstra <peterz@infradead.org>, Morten Rasmussen <morten.rasmussen@arm.com>, Qais Yousef <qais.yousef@arm.com>, Suren Baghdasaryan <surenb@google.com>, Quentin Perret <qperret@google.com>, Tejun Heo <tj@kernel.org>, Johannes Weiner <hannes@cmpxchg.org>, Ingo Molnar <mingo@redhat.com>, Juri Lelli <juri.lelli@redhat.com>, Vincent Guittot <vincent.guittot@linaro.org>, "Rafael J. Wysocki" <rjw@rjwysocki.net>, Dietmar Eggemann <dietmar.eggemann@arm.com>, Daniel Bristot de Oliveira <bristot@redhat.com>, Valentin Schneider <valentin.schneider@arm.com>, kernel-team@android.com Subject: Re: [PATCH v8 15/19] arm64: Prevent offlining first CPU with 32-bit EL0 on mismatched system Date: Fri, 4 Jun 2021 10:49:26 +0100 [thread overview] Message-ID: <20210604094926.GB64162@C02TD0UTHF1T.local> (raw) In-Reply-To: <20210603174056.GB1170@willie-the-truck> On Thu, Jun 03, 2021 at 06:40:57PM +0100, Will Deacon wrote: > On Thu, Jun 03, 2021 at 01:58:56PM +0100, Mark Rutland wrote: > > On Wed, Jun 02, 2021 at 05:47:15PM +0100, Will Deacon wrote: > > > If we want to support 32-bit applications, then when we identify a CPU > > > with mismatched 32-bit EL0 support we must ensure that we will always > > > have an active 32-bit CPU available to us from then on. This is important > > > for the scheduler, because is_cpu_allowed() will be constrained to 32-bit > > > CPUs for compat tasks and forced migration due to a hotplug event will > > > hang if no 32-bit CPUs are available. > > > > > > On detecting a mismatch, prevent offlining of either the mismatching CPU > > > if it is 32-bit capable, or find the first active 32-bit capable CPU > > > otherwise. > > > > > > Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> > > > Signed-off-by: Will Deacon <will@kernel.org> > > > --- > > > arch/arm64/kernel/cpufeature.c | 20 +++++++++++++++++++- > > > 1 file changed, 19 insertions(+), 1 deletion(-) > > > > > > diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c > > > index 4194a47de62d..b31d7a1eaed6 100644 > > > --- a/arch/arm64/kernel/cpufeature.c > > > +++ b/arch/arm64/kernel/cpufeature.c > > > @@ -2877,15 +2877,33 @@ void __init setup_cpu_features(void) > > > > > > static int enable_mismatched_32bit_el0(unsigned int cpu) > > > { > > > + static int lucky_winner = -1; > > > > This is cute, but could we please give it a meaningful name, e.g. > > `pinned_cpu` ? > > I really don't see the problem, nor why it's "cute". > > Tell you what, I'll add a comment instead: > > /* > * The first 32-bit-capable CPU we detected and so can no longer > * be offlined by userspace. -1 indicates we haven't yet onlined > * a 32-bit-capable CPU. > */ Thanks for the comment; that's helpful. However, my concern here is that when we inevitably have to discuss this with others in future, "lucky winner" is jarring (and also unclear to those where English is not their native language). For clarity, it would be really nice to use a term like "cpu", "chosen_cpu", "pinned_cpu", etc. However, you're the maintainer; choose what you think is appropriate. > > > struct cpuinfo_arm64 *info = &per_cpu(cpu_data, cpu); > > > bool cpu_32bit = id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0); > > > > > > if (cpu_32bit) { > > > cpumask_set_cpu(cpu, cpu_32bit_el0_mask); > > > static_branch_enable_cpuslocked(&arm64_mismatched_32bit_el0); > > > - setup_elf_hwcaps(compat_elf_hwcaps); > > > } > > > > > > + if (cpumask_test_cpu(0, cpu_32bit_el0_mask) == cpu_32bit) > > > + return 0; > > > + > > > + if (lucky_winner >= 0) > > > + return 0; > > > + > > > + /* > > > + * We've detected a mismatch. We need to keep one of our CPUs with > > > + * 32-bit EL0 online so that is_cpu_allowed() doesn't end up rejecting > > > + * every CPU in the system for a 32-bit task. > > > + */ > > > + lucky_winner = cpu_32bit ? cpu : cpumask_any_and(cpu_32bit_el0_mask, > > > + cpu_active_mask); > > > + get_cpu_device(lucky_winner)->offline_disabled = true; > > > + setup_elf_hwcaps(compat_elf_hwcaps); > > > + pr_info("Asymmetric 32-bit EL0 support detected on CPU %u; CPU hot-unplug disabled on CPU %u\n", > > > + cpu, lucky_winner); > > > return 0; > > > } > > > > I guess this is going to play havoc with kexec and hibernate. :/ > > The kernel can still offline the CPUs (see the whole freezer mess that I > linked to in the cover letter). What specific havoc are you thinking of? Ah. If this is just inhibiting userspace-driven offlining, that sounds fine. For kexec, I was concerned that either this would inhibit kexec, or smp_shutdown_nonboot_cpus() would fail to offline the pinned CPU, and that'd trigger a BUG(), which would be unfortunate. For hibernate, the equivalent is freeze_secondary_cpus(), which I guess is dealt with by the freezer bits you mention. Thanks, Mark.
WARNING: multiple messages have this Message-ID (diff)
From: Mark Rutland <mark.rutland@arm.com> To: Will Deacon <will@kernel.org> Cc: linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas <catalin.marinas@arm.com>, Marc Zyngier <maz@kernel.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Peter Zijlstra <peterz@infradead.org>, Morten Rasmussen <morten.rasmussen@arm.com>, Qais Yousef <qais.yousef@arm.com>, Suren Baghdasaryan <surenb@google.com>, Quentin Perret <qperret@google.com>, Tejun Heo <tj@kernel.org>, Johannes Weiner <hannes@cmpxchg.org>, Ingo Molnar <mingo@redhat.com>, Juri Lelli <juri.lelli@redhat.com>, Vincent Guittot <vincent.guittot@linaro.org>, "Rafael J. Wysocki" <rjw@rjwysocki.net>, Dietmar Eggemann <dietmar.eggemann@arm.com>, Daniel Bristot de Oliveira <bristot@redhat.com>, Valentin Schneider <valentin.schneider@arm.com>, kernel-team@android.com Subject: Re: [PATCH v8 15/19] arm64: Prevent offlining first CPU with 32-bit EL0 on mismatched system Date: Fri, 4 Jun 2021 10:49:26 +0100 [thread overview] Message-ID: <20210604094926.GB64162@C02TD0UTHF1T.local> (raw) In-Reply-To: <20210603174056.GB1170@willie-the-truck> On Thu, Jun 03, 2021 at 06:40:57PM +0100, Will Deacon wrote: > On Thu, Jun 03, 2021 at 01:58:56PM +0100, Mark Rutland wrote: > > On Wed, Jun 02, 2021 at 05:47:15PM +0100, Will Deacon wrote: > > > If we want to support 32-bit applications, then when we identify a CPU > > > with mismatched 32-bit EL0 support we must ensure that we will always > > > have an active 32-bit CPU available to us from then on. This is important > > > for the scheduler, because is_cpu_allowed() will be constrained to 32-bit > > > CPUs for compat tasks and forced migration due to a hotplug event will > > > hang if no 32-bit CPUs are available. > > > > > > On detecting a mismatch, prevent offlining of either the mismatching CPU > > > if it is 32-bit capable, or find the first active 32-bit capable CPU > > > otherwise. > > > > > > Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> > > > Signed-off-by: Will Deacon <will@kernel.org> > > > --- > > > arch/arm64/kernel/cpufeature.c | 20 +++++++++++++++++++- > > > 1 file changed, 19 insertions(+), 1 deletion(-) > > > > > > diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c > > > index 4194a47de62d..b31d7a1eaed6 100644 > > > --- a/arch/arm64/kernel/cpufeature.c > > > +++ b/arch/arm64/kernel/cpufeature.c > > > @@ -2877,15 +2877,33 @@ void __init setup_cpu_features(void) > > > > > > static int enable_mismatched_32bit_el0(unsigned int cpu) > > > { > > > + static int lucky_winner = -1; > > > > This is cute, but could we please give it a meaningful name, e.g. > > `pinned_cpu` ? > > I really don't see the problem, nor why it's "cute". > > Tell you what, I'll add a comment instead: > > /* > * The first 32-bit-capable CPU we detected and so can no longer > * be offlined by userspace. -1 indicates we haven't yet onlined > * a 32-bit-capable CPU. > */ Thanks for the comment; that's helpful. However, my concern here is that when we inevitably have to discuss this with others in future, "lucky winner" is jarring (and also unclear to those where English is not their native language). For clarity, it would be really nice to use a term like "cpu", "chosen_cpu", "pinned_cpu", etc. However, you're the maintainer; choose what you think is appropriate. > > > struct cpuinfo_arm64 *info = &per_cpu(cpu_data, cpu); > > > bool cpu_32bit = id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0); > > > > > > if (cpu_32bit) { > > > cpumask_set_cpu(cpu, cpu_32bit_el0_mask); > > > static_branch_enable_cpuslocked(&arm64_mismatched_32bit_el0); > > > - setup_elf_hwcaps(compat_elf_hwcaps); > > > } > > > > > > + if (cpumask_test_cpu(0, cpu_32bit_el0_mask) == cpu_32bit) > > > + return 0; > > > + > > > + if (lucky_winner >= 0) > > > + return 0; > > > + > > > + /* > > > + * We've detected a mismatch. We need to keep one of our CPUs with > > > + * 32-bit EL0 online so that is_cpu_allowed() doesn't end up rejecting > > > + * every CPU in the system for a 32-bit task. > > > + */ > > > + lucky_winner = cpu_32bit ? cpu : cpumask_any_and(cpu_32bit_el0_mask, > > > + cpu_active_mask); > > > + get_cpu_device(lucky_winner)->offline_disabled = true; > > > + setup_elf_hwcaps(compat_elf_hwcaps); > > > + pr_info("Asymmetric 32-bit EL0 support detected on CPU %u; CPU hot-unplug disabled on CPU %u\n", > > > + cpu, lucky_winner); > > > return 0; > > > } > > > > I guess this is going to play havoc with kexec and hibernate. :/ > > The kernel can still offline the CPUs (see the whole freezer mess that I > linked to in the cover letter). What specific havoc are you thinking of? Ah. If this is just inhibiting userspace-driven offlining, that sounds fine. For kexec, I was concerned that either this would inhibit kexec, or smp_shutdown_nonboot_cpus() would fail to offline the pinned CPU, and that'd trigger a BUG(), which would be unfortunate. For hibernate, the equivalent is freeze_secondary_cpus(), which I guess is dealt with by the freezer bits you mention. Thanks, Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-06-04 9:49 UTC|newest] Thread overview: 96+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-06-02 16:47 [PATCH v8 00/19] Add support for 32-bit tasks on asymmetric AArch32 systems Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 01/19] arm64: cpuinfo: Split AArch32 registers out into a separate struct Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-03 12:38 ` Mark Rutland 2021-06-03 12:38 ` Mark Rutland 2021-06-03 17:24 ` Will Deacon 2021-06-03 17:24 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 02/19] arm64: Allow mismatched 32-bit EL0 support Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-03 12:37 ` Mark Rutland 2021-06-03 12:37 ` Mark Rutland 2021-06-03 17:44 ` Will Deacon 2021-06-03 17:44 ` Will Deacon 2021-06-04 9:38 ` Mark Rutland 2021-06-04 9:38 ` Mark Rutland 2021-06-04 11:05 ` Will Deacon 2021-06-04 11:05 ` Will Deacon 2021-06-04 12:04 ` Mark Rutland 2021-06-04 12:04 ` Mark Rutland 2021-06-04 13:50 ` Will Deacon 2021-06-04 13:50 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 03/19] KVM: arm64: Kill 32-bit vCPUs on systems with mismatched " Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 04/19] arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 05/19] sched: Introduce task_cpu_possible_mask() to limit fallback rq selection Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-04 17:10 ` Valentin Schneider 2021-06-04 17:10 ` Valentin Schneider 2021-06-07 17:04 ` Will Deacon 2021-06-07 17:04 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 06/19] cpuset: Don't use the cpu_possible_mask as a last resort for cgroup v1 Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-04 17:11 ` Valentin Schneider 2021-06-04 17:11 ` Valentin Schneider 2021-06-07 17:20 ` Will Deacon 2021-06-07 17:20 ` Will Deacon 2021-06-10 10:20 ` Valentin Schneider 2021-06-10 10:20 ` Valentin Schneider 2021-06-02 16:47 ` [PATCH v8 07/19] cpuset: Honour task_cpu_possible_mask() in guarantee_online_cpus() Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-04 17:11 ` Valentin Schneider 2021-06-04 17:11 ` Valentin Schneider 2021-06-02 16:47 ` [PATCH v8 08/19] sched: Reject CPU affinity changes based on task_cpu_possible_mask() Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-04 17:11 ` Valentin Schneider 2021-06-04 17:11 ` Valentin Schneider 2021-06-07 22:43 ` Will Deacon 2021-06-07 22:43 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 09/19] sched: Introduce task_struct::user_cpus_ptr to track requested affinity Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-04 17:12 ` Valentin Schneider 2021-06-04 17:12 ` Valentin Schneider 2021-06-02 16:47 ` [PATCH v8 10/19] sched: Split the guts of sched_setaffinity() into a helper function Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-04 17:12 ` Valentin Schneider 2021-06-04 17:12 ` Valentin Schneider 2021-06-02 16:47 ` [PATCH v8 11/19] sched: Allow task CPU affinity to be restricted on asymmetric systems Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-04 17:12 ` Valentin Schneider 2021-06-04 17:12 ` Valentin Schneider 2021-06-07 22:52 ` Will Deacon 2021-06-07 22:52 ` Will Deacon 2021-06-10 10:20 ` Valentin Schneider 2021-06-10 10:20 ` Valentin Schneider 2021-06-02 16:47 ` [PATCH v8 12/19] sched: Introduce task_cpus_dl_admissible() to check proposed affinity Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-03 9:43 ` Daniel Bristot de Oliveira 2021-06-03 9:43 ` Daniel Bristot de Oliveira 2021-06-03 9:52 ` Will Deacon 2021-06-03 9:52 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 13/19] arm64: Implement task_cpu_possible_mask() Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 14/19] arm64: exec: Adjust affinity for compat tasks with mismatched 32-bit EL0 Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-03 9:45 ` Daniel Bristot de Oliveira 2021-06-03 9:45 ` Daniel Bristot de Oliveira 2021-06-02 16:47 ` [PATCH v8 15/19] arm64: Prevent offlining first CPU with 32-bit EL0 on mismatched system Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-03 12:58 ` Mark Rutland 2021-06-03 12:58 ` Mark Rutland 2021-06-03 17:40 ` Will Deacon 2021-06-03 17:40 ` Will Deacon 2021-06-04 9:49 ` Mark Rutland [this message] 2021-06-04 9:49 ` Mark Rutland 2021-06-04 12:14 ` Qais Yousef 2021-06-04 12:14 ` Qais Yousef 2021-06-02 16:47 ` [PATCH v8 16/19] arm64: Advertise CPUs capable of running 32-bit applications in sysfs Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 17/19] arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0 Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 18/19] arm64: Remove logic to kill 32-bit tasks on 64-bit-only cores Will Deacon 2021-06-02 16:47 ` Will Deacon 2021-06-02 16:47 ` [PATCH v8 19/19] Documentation: arm64: describe asymmetric 32-bit support Will Deacon 2021-06-02 16:47 ` Will Deacon
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210604094926.GB64162@C02TD0UTHF1T.local \ --to=mark.rutland@arm.com \ --cc=bristot@redhat.com \ --cc=catalin.marinas@arm.com \ --cc=dietmar.eggemann@arm.com \ --cc=gregkh@linuxfoundation.org \ --cc=hannes@cmpxchg.org \ --cc=juri.lelli@redhat.com \ --cc=kernel-team@android.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=maz@kernel.org \ --cc=mingo@redhat.com \ --cc=morten.rasmussen@arm.com \ --cc=peterz@infradead.org \ --cc=qais.yousef@arm.com \ --cc=qperret@google.com \ --cc=rjw@rjwysocki.net \ --cc=surenb@google.com \ --cc=tj@kernel.org \ --cc=valentin.schneider@arm.com \ --cc=vincent.guittot@linaro.org \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.