From: Sumit Garg <sumit.garg@linaro.org> To: Doug Anderson <dianders@chromium.org> Cc: Linux ARM <linux-arm-kernel@lists.infradead.org>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Jason Cooper <jason@lakedaemon.net>, Marc Zyngier <maz@kernel.org>, julien.thierry.kdev@gmail.com, Daniel Thompson <daniel.thompson@linaro.org>, Jason Wessel <jason.wessel@windriver.com>, kgdb-bugreport@lists.sourceforge.net, LKML <linux-kernel@vger.kernel.org> Subject: Re: [RFC Patch v1 4/4] arm64: kgdb: Round up cpus using IPI_CALL_NMI_FUNC Date: Mon, 27 Apr 2020 10:22:33 +0530 [thread overview] Message-ID: <CAFA6WYOorX8iRHhpOq26QCE8T1dKmXFUXVTxoWOP0OZErqJqBQ@mail.gmail.com> (raw) In-Reply-To: <CAD=FV=VSrVnx_R=Y38tQ=bw_o22zxWmm=+M+AaqmEMAtK66b-Q@mail.gmail.com> Hi Doug, Thanks for your comments. On Sat, 25 Apr 2020 at 02:17, Doug Anderson <dianders@chromium.org> wrote: > > Hi, > > On Fri, Apr 24, 2020 at 4:11 AM Sumit Garg <sumit.garg@linaro.org> wrote: > > > > arm64 platforms with GICv3 or later supports pseudo NMIs which can be > > leveraged to round up CPUs which are stuck in hard lockup state with > > interrupts disabled that wouldn't be possible with a normal IPI. > > > > So instead switch to round up CPUs using IPI_CALL_NMI_FUNC. And in > > case a particular arm64 platform doesn't supports pseudo NMIs, > > IPI_CALL_NMI_FUNC will act as a normal IPI which maintains existing > > kgdb functionality. > > > > Also, one thing to note here is that with CPUs running in NMI context, > > kernel has special handling for printk() which involves CPU specific > > buffers and defering printk() until exit from NMI context. But with kgdb > > we don't want to defer printk() especially backtrace on corresponding > > CPUs. So switch to normal printk() context instead prior to entering > > kgdb context. > > > > Signed-off-by: Sumit Garg <sumit.garg@linaro.org> > > --- > > arch/arm64/kernel/kgdb.c | 15 +++++++++++++++ > > arch/arm64/kernel/smp.c | 17 ++++++++++++++--- > > 2 files changed, 29 insertions(+), 3 deletions(-) > > > > diff --git a/arch/arm64/kernel/kgdb.c b/arch/arm64/kernel/kgdb.c > > index 4311992..0851ead 100644 > > --- a/arch/arm64/kernel/kgdb.c > > +++ b/arch/arm64/kernel/kgdb.c > > @@ -14,6 +14,7 @@ > > #include <linux/kgdb.h> > > #include <linux/kprobes.h> > > #include <linux/sched/task_stack.h> > > +#include <linux/smp.h> > > > > #include <asm/debug-monitors.h> > > #include <asm/insn.h> > > @@ -353,3 +354,17 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) > > return aarch64_insn_write((void *)bpt->bpt_addr, > > *(u32 *)bpt->saved_instr); > > } > > + > > +#ifdef CONFIG_SMP > > +void kgdb_roundup_cpus(void) > > +{ > > + struct cpumask mask; > > + > > + cpumask_copy(&mask, cpu_online_mask); > > + cpumask_clear_cpu(raw_smp_processor_id(), &mask); > > + if (cpumask_empty(&mask)) > > + return; > > + > > + arch_send_call_nmi_func_ipi_mask(&mask); > > +} > > +#endif > > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c > > index 27c8ee1..c7158f6e8 100644 > > --- a/arch/arm64/kernel/smp.c > > +++ b/arch/arm64/kernel/smp.c > > @@ -31,6 +31,7 @@ > > #include <linux/of.h> > > #include <linux/irq_work.h> > > #include <linux/kexec.h> > > +#include <linux/kgdb.h> > > #include <linux/kvm_host.h> > > > > #include <asm/alternative.h> > > @@ -976,9 +977,19 @@ void handle_IPI(int ipinr, struct pt_regs *regs) > > /* Handle it as a normal interrupt if not in NMI context */ > > if (!in_nmi()) > > irq_enter(); > > - > > - /* nop, IPI handlers for special features can be added here. */ > > - > > +#ifdef CONFIG_KGDB > > My vote would be to keep "ifdef"s out of the middle of functions. Can > you put your code in "arch/arm64/kernel/kgdb.c" and then have a dummpy > no-op function if "CONFIG_KGDB" isn't defined? > Sure. > > > + if (atomic_read(&kgdb_active) != -1) { > > + /* > > + * For kgdb to work properly, we need printk to operate > > + * in normal context. > > + */ > > + if (in_nmi()) > > + printk_nmi_exit(); > > It feels like all the printk management belongs in kgdb_nmicallback(). > ...or is there some reason that this isn't a problem for other > platforms using NMI? Maybe it's just that nobody has noticed it yet? > Initially I was skeptical of moving this printk handling in the common kgdb framework but after exploring other platforms like x86 (probably unnoticed bug), I agree with you that it belongs to kgdb_nmicallback(). So I will move it there. > > > + kgdb_nmicallback(raw_smp_processor_id(), regs); > > Why do you need to call raw_smp_processor_id()? Are you expecting a > different value than the local variable "cpu"? Ah, no. Will use the local variable "cpu" instead. > > > > + if (in_nmi()) > > + printk_nmi_enter(); > > + } > > +#endif > > if (!in_nmi()) > > irq_exit(); > > break; > > Not that I really know what I'm talking about since I really don't > know arm64 at this level very well, but I'll ask anyway and probably > look like a fool... I had a note that said: > > * Will Deacon says: > * > * the whole roundup code is sketchy and it's the only place in the kernel > * which tries to perform I-cache maintenance with irqs disabled, leading > * to this nasty hack in the arch code: > * > * https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/include/asm/cacheflush.h#n74 > > I presume that, if nothing else, the comment needs to be updated. > ...but is the situation any better (or worse?) with your new solution? I think the situation remains the same with new solution as well. As either we use IPI being a pseudo NMI or a normal IRQ to roundup CPUs, kgdb still does I-cache maintenance with irqs disabled which could lead to a deadlock trying to IPI the secondary CPUs without this nasty hack in the arch code. -Sumit > > -Doug
WARNING: multiple messages have this Message-ID (diff)
From: Sumit Garg <sumit.garg@linaro.org> To: Doug Anderson <dianders@chromium.org> Cc: Daniel Thompson <daniel.thompson@linaro.org>, Jason Cooper <jason@lakedaemon.net>, Catalin Marinas <catalin.marinas@arm.com>, Jason Wessel <jason.wessel@windriver.com>, LKML <linux-kernel@vger.kernel.org>, julien.thierry.kdev@gmail.com, Marc Zyngier <maz@kernel.org>, kgdb-bugreport@lists.sourceforge.net, Thomas Gleixner <tglx@linutronix.de>, Will Deacon <will@kernel.org>, Linux ARM <linux-arm-kernel@lists.infradead.org> Subject: Re: [RFC Patch v1 4/4] arm64: kgdb: Round up cpus using IPI_CALL_NMI_FUNC Date: Mon, 27 Apr 2020 10:22:33 +0530 [thread overview] Message-ID: <CAFA6WYOorX8iRHhpOq26QCE8T1dKmXFUXVTxoWOP0OZErqJqBQ@mail.gmail.com> (raw) In-Reply-To: <CAD=FV=VSrVnx_R=Y38tQ=bw_o22zxWmm=+M+AaqmEMAtK66b-Q@mail.gmail.com> Hi Doug, Thanks for your comments. On Sat, 25 Apr 2020 at 02:17, Doug Anderson <dianders@chromium.org> wrote: > > Hi, > > On Fri, Apr 24, 2020 at 4:11 AM Sumit Garg <sumit.garg@linaro.org> wrote: > > > > arm64 platforms with GICv3 or later supports pseudo NMIs which can be > > leveraged to round up CPUs which are stuck in hard lockup state with > > interrupts disabled that wouldn't be possible with a normal IPI. > > > > So instead switch to round up CPUs using IPI_CALL_NMI_FUNC. And in > > case a particular arm64 platform doesn't supports pseudo NMIs, > > IPI_CALL_NMI_FUNC will act as a normal IPI which maintains existing > > kgdb functionality. > > > > Also, one thing to note here is that with CPUs running in NMI context, > > kernel has special handling for printk() which involves CPU specific > > buffers and defering printk() until exit from NMI context. But with kgdb > > we don't want to defer printk() especially backtrace on corresponding > > CPUs. So switch to normal printk() context instead prior to entering > > kgdb context. > > > > Signed-off-by: Sumit Garg <sumit.garg@linaro.org> > > --- > > arch/arm64/kernel/kgdb.c | 15 +++++++++++++++ > > arch/arm64/kernel/smp.c | 17 ++++++++++++++--- > > 2 files changed, 29 insertions(+), 3 deletions(-) > > > > diff --git a/arch/arm64/kernel/kgdb.c b/arch/arm64/kernel/kgdb.c > > index 4311992..0851ead 100644 > > --- a/arch/arm64/kernel/kgdb.c > > +++ b/arch/arm64/kernel/kgdb.c > > @@ -14,6 +14,7 @@ > > #include <linux/kgdb.h> > > #include <linux/kprobes.h> > > #include <linux/sched/task_stack.h> > > +#include <linux/smp.h> > > > > #include <asm/debug-monitors.h> > > #include <asm/insn.h> > > @@ -353,3 +354,17 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) > > return aarch64_insn_write((void *)bpt->bpt_addr, > > *(u32 *)bpt->saved_instr); > > } > > + > > +#ifdef CONFIG_SMP > > +void kgdb_roundup_cpus(void) > > +{ > > + struct cpumask mask; > > + > > + cpumask_copy(&mask, cpu_online_mask); > > + cpumask_clear_cpu(raw_smp_processor_id(), &mask); > > + if (cpumask_empty(&mask)) > > + return; > > + > > + arch_send_call_nmi_func_ipi_mask(&mask); > > +} > > +#endif > > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c > > index 27c8ee1..c7158f6e8 100644 > > --- a/arch/arm64/kernel/smp.c > > +++ b/arch/arm64/kernel/smp.c > > @@ -31,6 +31,7 @@ > > #include <linux/of.h> > > #include <linux/irq_work.h> > > #include <linux/kexec.h> > > +#include <linux/kgdb.h> > > #include <linux/kvm_host.h> > > > > #include <asm/alternative.h> > > @@ -976,9 +977,19 @@ void handle_IPI(int ipinr, struct pt_regs *regs) > > /* Handle it as a normal interrupt if not in NMI context */ > > if (!in_nmi()) > > irq_enter(); > > - > > - /* nop, IPI handlers for special features can be added here. */ > > - > > +#ifdef CONFIG_KGDB > > My vote would be to keep "ifdef"s out of the middle of functions. Can > you put your code in "arch/arm64/kernel/kgdb.c" and then have a dummpy > no-op function if "CONFIG_KGDB" isn't defined? > Sure. > > > + if (atomic_read(&kgdb_active) != -1) { > > + /* > > + * For kgdb to work properly, we need printk to operate > > + * in normal context. > > + */ > > + if (in_nmi()) > > + printk_nmi_exit(); > > It feels like all the printk management belongs in kgdb_nmicallback(). > ...or is there some reason that this isn't a problem for other > platforms using NMI? Maybe it's just that nobody has noticed it yet? > Initially I was skeptical of moving this printk handling in the common kgdb framework but after exploring other platforms like x86 (probably unnoticed bug), I agree with you that it belongs to kgdb_nmicallback(). So I will move it there. > > > + kgdb_nmicallback(raw_smp_processor_id(), regs); > > Why do you need to call raw_smp_processor_id()? Are you expecting a > different value than the local variable "cpu"? Ah, no. Will use the local variable "cpu" instead. > > > > + if (in_nmi()) > > + printk_nmi_enter(); > > + } > > +#endif > > if (!in_nmi()) > > irq_exit(); > > break; > > Not that I really know what I'm talking about since I really don't > know arm64 at this level very well, but I'll ask anyway and probably > look like a fool... I had a note that said: > > * Will Deacon says: > * > * the whole roundup code is sketchy and it's the only place in the kernel > * which tries to perform I-cache maintenance with irqs disabled, leading > * to this nasty hack in the arch code: > * > * https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/include/asm/cacheflush.h#n74 > > I presume that, if nothing else, the comment needs to be updated. > ...but is the situation any better (or worse?) with your new solution? I think the situation remains the same with new solution as well. As either we use IPI being a pseudo NMI or a normal IRQ to roundup CPUs, kgdb still does I-cache maintenance with irqs disabled which could lead to a deadlock trying to IPI the secondary CPUs without this nasty hack in the arch code. -Sumit > > -Doug _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-04-27 4:52 UTC|newest] Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-04-24 11:09 [RFC Patch v1 0/4] arm64: Introduce new IPI as IPI_CALL_NMI_FUNC Sumit Garg 2020-04-24 11:09 ` Sumit Garg 2020-04-24 11:09 ` [RFC Patch v1 1/4] arm64: smp: Introduce a " Sumit Garg 2020-04-24 11:09 ` Sumit Garg 2020-04-24 11:09 ` [RFC Patch v1 2/4] irqchip/gic-v3: Add support to handle SGI as pseudo NMI Sumit Garg 2020-04-24 11:09 ` Sumit Garg 2020-04-25 10:29 ` Marc Zyngier 2020-04-25 10:29 ` Marc Zyngier 2020-04-25 14:32 ` Marc Zyngier 2020-04-25 14:32 ` Marc Zyngier 2020-04-28 14:11 ` Sumit Garg 2020-04-28 14:11 ` Sumit Garg 2020-04-29 8:23 ` Marc Zyngier 2020-04-29 8:23 ` Marc Zyngier 2020-04-30 7:20 ` Sumit Garg 2020-04-30 7:20 ` Sumit Garg 2020-04-30 9:13 ` Marc Zyngier 2020-04-30 9:13 ` Marc Zyngier 2020-04-30 12:13 ` Sumit Garg 2020-04-30 12:13 ` Sumit Garg 2020-05-01 13:03 ` Sumit Garg 2020-05-01 13:03 ` Sumit Garg 2020-05-05 4:09 ` Sumit Garg 2020-05-05 4:09 ` Sumit Garg 2020-05-05 10:08 ` Marc Zyngier 2020-05-05 10:08 ` Marc Zyngier 2020-05-05 11:33 ` Sumit Garg 2020-05-05 11:33 ` Sumit Garg 2020-05-13 10:02 ` Sumit Garg 2020-05-13 10:02 ` Sumit Garg 2020-05-19 14:51 ` Marc Zyngier 2020-05-19 14:51 ` Marc Zyngier 2020-04-24 11:09 ` [RFC Patch v1 3/4] irqchip/gic-v3: Enable arch specific IPI " Sumit Garg 2020-04-24 11:09 ` Sumit Garg 2020-04-24 11:09 ` [RFC Patch v1 4/4] arm64: kgdb: Round up cpus using IPI_CALL_NMI_FUNC Sumit Garg 2020-04-24 11:09 ` Sumit Garg 2020-04-24 20:46 ` Doug Anderson 2020-04-24 20:46 ` Doug Anderson 2020-04-27 4:52 ` Sumit Garg [this message] 2020-04-27 4:52 ` Sumit Garg 2020-04-24 20:49 ` [RFC Patch v1 0/4] arm64: Introduce new IPI as IPI_CALL_NMI_FUNC Doug Anderson 2020-04-24 20:49 ` Doug Anderson 2020-04-27 4:54 ` Sumit Garg 2020-04-27 4:54 ` Sumit Garg
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=CAFA6WYOorX8iRHhpOq26QCE8T1dKmXFUXVTxoWOP0OZErqJqBQ@mail.gmail.com \ --to=sumit.garg@linaro.org \ --cc=catalin.marinas@arm.com \ --cc=daniel.thompson@linaro.org \ --cc=dianders@chromium.org \ --cc=jason.wessel@windriver.com \ --cc=jason@lakedaemon.net \ --cc=julien.thierry.kdev@gmail.com \ --cc=kgdb-bugreport@lists.sourceforge.net \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=maz@kernel.org \ --cc=tglx@linutronix.de \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.