From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AED98C54FCB for ; Fri, 24 Apr 2020 20:47:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 82BD520857 for ; Fri, 24 Apr 2020 20:47:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Gy4e1DrY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726130AbgDXUrQ (ORCPT ); Fri, 24 Apr 2020 16:47:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726102AbgDXUrQ (ORCPT ); Fri, 24 Apr 2020 16:47:16 -0400 Received: from mail-ua1-x944.google.com (mail-ua1-x944.google.com [IPv6:2607:f8b0:4864:20::944]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BBE6C09B048 for ; Fri, 24 Apr 2020 13:47:14 -0700 (PDT) Received: by mail-ua1-x944.google.com with SMTP id c24so10876099uap.13 for ; Fri, 24 Apr 2020 13:47:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qeSr0lei+fRC9/QJxXSmPKueqlEcByGy/C+Sm2/krvg=; b=Gy4e1DrYdXOs9V95msoY1FVrKuSQ1ZhCKigCD3DdHYa4vQSZF0/wZGOzsXWdhDRQMJ ntjWHIAzoYQW/EbWleAXGSfEnaY3ZJ7a7cSARn7y8iNmYUhWJlNIe/hR5f+f+ETIthcj +7H9KyKHdOyYM6Yq87MOT6M0ttEfo85nWdehE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qeSr0lei+fRC9/QJxXSmPKueqlEcByGy/C+Sm2/krvg=; b=NIjMtvWx1Ggz0GN4+18Wq8QYQVYznLUdxM4n8VkyNlv5fOzYO9FzC9sxIYOd9cbCkt FKpZQ9p3Grn0GVdfy9T5HmoFnAeihlLapk+h3YaTE+HsEsn8VQOZQUs0Ym07lig5gU3m EVtb/0b3JS8a0CmMYCQNpI1ZlmU2Tpdh0IPzo81VVwNJfK8yNeCrtz+oAm3Ar3Gvvlax Tdii5iqom/0yi7riMceqL9cU4ZSSrR8RoEHPZz25CeaU9CnvlJItP1tatgkQIB0mxVtZ 09D6x0NLPBBBDa683PhJ24VdN/C9UmnduN02h+iVy4NtpKOw3ZSqVQI2aay+IRQOGqxz ADew== X-Gm-Message-State: AGi0PubGTjd1Hn1RbLxOykO8Q9itsmQc4y1mSOsmwVMuP2+5PN2CODgf tgmJt+j04KFnQezvY/98iE9fdom4NhI= X-Google-Smtp-Source: APiQypITkQIdsFy00Hd27mx8HtPMRNoAGkvIdyyKMcRfg3SdeHMg0YoYwTfn+EnHkeu8AEn05n3fJg== X-Received: by 2002:a67:2247:: with SMTP id i68mr9117338vsi.19.1587761233275; Fri, 24 Apr 2020 13:47:13 -0700 (PDT) Received: from mail-ua1-f49.google.com (mail-ua1-f49.google.com. [209.85.222.49]) by smtp.gmail.com with ESMTPSA id v16sm1678906vsq.18.2020.04.24.13.47.11 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Apr 2020 13:47:12 -0700 (PDT) Received: by mail-ua1-f49.google.com with SMTP id z16so10884430uae.11 for ; Fri, 24 Apr 2020 13:47:11 -0700 (PDT) X-Received: by 2002:ab0:b13:: with SMTP id b19mr8989653uak.91.1587761231452; Fri, 24 Apr 2020 13:47:11 -0700 (PDT) MIME-Version: 1.0 References: <1587726554-32018-1-git-send-email-sumit.garg@linaro.org> <1587726554-32018-5-git-send-email-sumit.garg@linaro.org> In-Reply-To: <1587726554-32018-5-git-send-email-sumit.garg@linaro.org> From: Doug Anderson Date: Fri, 24 Apr 2020 13:46:59 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC Patch v1 4/4] arm64: kgdb: Round up cpus using IPI_CALL_NMI_FUNC To: Sumit Garg Cc: Linux ARM , Catalin Marinas , Will Deacon , Thomas Gleixner , Jason Cooper , Marc Zyngier , julien.thierry.kdev@gmail.com, Daniel Thompson , Jason Wessel , kgdb-bugreport@lists.sourceforge.net, LKML Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On Fri, Apr 24, 2020 at 4:11 AM Sumit Garg wrote: > > arm64 platforms with GICv3 or later supports pseudo NMIs which can be > leveraged to round up CPUs which are stuck in hard lockup state with > interrupts disabled that wouldn't be possible with a normal IPI. > > So instead switch to round up CPUs using IPI_CALL_NMI_FUNC. And in > case a particular arm64 platform doesn't supports pseudo NMIs, > IPI_CALL_NMI_FUNC will act as a normal IPI which maintains existing > kgdb functionality. > > Also, one thing to note here is that with CPUs running in NMI context, > kernel has special handling for printk() which involves CPU specific > buffers and defering printk() until exit from NMI context. But with kgdb > we don't want to defer printk() especially backtrace on corresponding > CPUs. So switch to normal printk() context instead prior to entering > kgdb context. > > Signed-off-by: Sumit Garg > --- > arch/arm64/kernel/kgdb.c | 15 +++++++++++++++ > arch/arm64/kernel/smp.c | 17 ++++++++++++++--- > 2 files changed, 29 insertions(+), 3 deletions(-) > > diff --git a/arch/arm64/kernel/kgdb.c b/arch/arm64/kernel/kgdb.c > index 4311992..0851ead 100644 > --- a/arch/arm64/kernel/kgdb.c > +++ b/arch/arm64/kernel/kgdb.c > @@ -14,6 +14,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -353,3 +354,17 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) > return aarch64_insn_write((void *)bpt->bpt_addr, > *(u32 *)bpt->saved_instr); > } > + > +#ifdef CONFIG_SMP > +void kgdb_roundup_cpus(void) > +{ > + struct cpumask mask; > + > + cpumask_copy(&mask, cpu_online_mask); > + cpumask_clear_cpu(raw_smp_processor_id(), &mask); > + if (cpumask_empty(&mask)) > + return; > + > + arch_send_call_nmi_func_ipi_mask(&mask); > +} > +#endif > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c > index 27c8ee1..c7158f6e8 100644 > --- a/arch/arm64/kernel/smp.c > +++ b/arch/arm64/kernel/smp.c > @@ -31,6 +31,7 @@ > #include > #include > #include > +#include > #include > > #include > @@ -976,9 +977,19 @@ void handle_IPI(int ipinr, struct pt_regs *regs) > /* Handle it as a normal interrupt if not in NMI context */ > if (!in_nmi()) > irq_enter(); > - > - /* nop, IPI handlers for special features can be added here. */ > - > +#ifdef CONFIG_KGDB My vote would be to keep "ifdef"s out of the middle of functions. Can you put your code in "arch/arm64/kernel/kgdb.c" and then have a dummpy no-op function if "CONFIG_KGDB" isn't defined? > + if (atomic_read(&kgdb_active) != -1) { > + /* > + * For kgdb to work properly, we need printk to operate > + * in normal context. > + */ > + if (in_nmi()) > + printk_nmi_exit(); It feels like all the printk management belongs in kgdb_nmicallback(). ...or is there some reason that this isn't a problem for other platforms using NMI? Maybe it's just that nobody has noticed it yet? > + kgdb_nmicallback(raw_smp_processor_id(), regs); Why do you need to call raw_smp_processor_id()? Are you expecting a different value than the local variable "cpu"? > + if (in_nmi()) > + printk_nmi_enter(); > + } > +#endif > if (!in_nmi()) > irq_exit(); > break; Not that I really know what I'm talking about since I really don't know arm64 at this level very well, but I'll ask anyway and probably look like a fool... I had a note that said: * Will Deacon says: * * the whole roundup code is sketchy and it's the only place in the kernel * which tries to perform I-cache maintenance with irqs disabled, leading * to this nasty hack in the arch code: * * https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/include/asm/cacheflush.h#n74 I presume that, if nothing else, the comment needs to be updated. ...but is the situation any better (or worse?) with your new solution? -Doug From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F95AC54FD0 for ; Fri, 24 Apr 2020 20:47:25 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 61C17215A4 for ; Fri, 24 Apr 2020 20:47:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="l1jOVF72"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Gy4e1DrY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 61C17215A4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2VrSAESGNdd6+0N8AL7R9tKZCk5GBO9o95L/+DOpiOk=; b=l1jOVF72AhKpbC ihuyGoLXKp8tNiLDv0/5l54QHkXEB86F3hYrBxosxL6JqsPt/PcVxyuQR996ZeAj/4ch12Ees8y1u wGLtgeXkrsAPDoGhITMIoyMSK7vvcejhhTXO1kQlqpWA/ujlO2oOX5V3spDe+6YkEYmqSuNqmrA96 LfxtDadcqr1F2nBsGyB086IafXdwtj7Fjg1sOG9wtnIuH4C/k4YluEgXXHf5976Qh02OP7ctfE/J6 PG0kUM+471QOaW3JDt0z+LZIuR1MSVAbctIYCKqFtP/FjsYVb9jCbxSdJyyAJp7PrQ9GgdQSDiMcL LZhpK8yDXUeUDJDqaKIQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jS5EG-0007fs-Dl; Fri, 24 Apr 2020 20:47:20 +0000 Received: from mail-ua1-x942.google.com ([2607:f8b0:4864:20::942]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jS5ED-0007f0-Bm for linux-arm-kernel@lists.infradead.org; Fri, 24 Apr 2020 20:47:18 +0000 Received: by mail-ua1-x942.google.com with SMTP id v24so10878980uak.0 for ; Fri, 24 Apr 2020 13:47:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qeSr0lei+fRC9/QJxXSmPKueqlEcByGy/C+Sm2/krvg=; b=Gy4e1DrYdXOs9V95msoY1FVrKuSQ1ZhCKigCD3DdHYa4vQSZF0/wZGOzsXWdhDRQMJ ntjWHIAzoYQW/EbWleAXGSfEnaY3ZJ7a7cSARn7y8iNmYUhWJlNIe/hR5f+f+ETIthcj +7H9KyKHdOyYM6Yq87MOT6M0ttEfo85nWdehE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qeSr0lei+fRC9/QJxXSmPKueqlEcByGy/C+Sm2/krvg=; b=LBREj80w036zlYi4ACzA0E08ANnxozRUcFy5gCcCSJJOhta7qUYlDjtYkhp4940qKM glAu5C3Bl7OhU6ezH/DphjjDLWTwSapRg+ffYlRoFdWg50NbF0U46nJn17Kub0lT4ALj W5tkbaP3ywf9LgoTJ/+kbfFFElCaqHvxh6SLPd/7hyXKouP1Y2xU9z54PHVir2iP9x27 fR5PX+V3quD90ehGtO/FAU6xEOsaVAeCZ6yHi8dA177fPMb7CwV9/mkzUqKXiVh7AXU9 /FwNkBNiQtgkDLOWMAQt7rycNhSoKIqImwE845FWLBcz3p+5cG/XLZ9DvSF9wbIfUKZA QhGw== X-Gm-Message-State: AGi0PuZHv3FN+StX5ecJPg3KKCb1BOgl2zxmPaCE/Jw3BCxGXaSWiwL7 y0yLjAvo9QrsdPri7QVl/vN66Dy8oLg= X-Google-Smtp-Source: APiQypLGG9d+DYvpOiswpLWSLZSAiaOJ4hhEn2UY5BgoFdrhon6w5h2d0SUYDCbiGq/WvlIcVoRAxQ== X-Received: by 2002:a05:6102:5f4:: with SMTP id w20mr9800685vsf.61.1587761234732; Fri, 24 Apr 2020 13:47:14 -0700 (PDT) Received: from mail-ua1-f41.google.com (mail-ua1-f41.google.com. [209.85.222.41]) by smtp.gmail.com with ESMTPSA id 62sm1881599vku.10.2020.04.24.13.47.11 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Apr 2020 13:47:13 -0700 (PDT) Received: by mail-ua1-f41.google.com with SMTP id i22so10877702uak.6 for ; Fri, 24 Apr 2020 13:47:11 -0700 (PDT) X-Received: by 2002:ab0:b13:: with SMTP id b19mr8989653uak.91.1587761231452; Fri, 24 Apr 2020 13:47:11 -0700 (PDT) MIME-Version: 1.0 References: <1587726554-32018-1-git-send-email-sumit.garg@linaro.org> <1587726554-32018-5-git-send-email-sumit.garg@linaro.org> In-Reply-To: <1587726554-32018-5-git-send-email-sumit.garg@linaro.org> From: Doug Anderson Date: Fri, 24 Apr 2020 13:46:59 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC Patch v1 4/4] arm64: kgdb: Round up cpus using IPI_CALL_NMI_FUNC To: Sumit Garg X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200424_134717_428108_3C116331 X-CRM114-Status: GOOD ( 22.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Thompson , Jason Cooper , Catalin Marinas , Jason Wessel , LKML , julien.thierry.kdev@gmail.com, Marc Zyngier , kgdb-bugreport@lists.sourceforge.net, Thomas Gleixner , Will Deacon , Linux ARM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, On Fri, Apr 24, 2020 at 4:11 AM Sumit Garg wrote: > > arm64 platforms with GICv3 or later supports pseudo NMIs which can be > leveraged to round up CPUs which are stuck in hard lockup state with > interrupts disabled that wouldn't be possible with a normal IPI. > > So instead switch to round up CPUs using IPI_CALL_NMI_FUNC. And in > case a particular arm64 platform doesn't supports pseudo NMIs, > IPI_CALL_NMI_FUNC will act as a normal IPI which maintains existing > kgdb functionality. > > Also, one thing to note here is that with CPUs running in NMI context, > kernel has special handling for printk() which involves CPU specific > buffers and defering printk() until exit from NMI context. But with kgdb > we don't want to defer printk() especially backtrace on corresponding > CPUs. So switch to normal printk() context instead prior to entering > kgdb context. > > Signed-off-by: Sumit Garg > --- > arch/arm64/kernel/kgdb.c | 15 +++++++++++++++ > arch/arm64/kernel/smp.c | 17 ++++++++++++++--- > 2 files changed, 29 insertions(+), 3 deletions(-) > > diff --git a/arch/arm64/kernel/kgdb.c b/arch/arm64/kernel/kgdb.c > index 4311992..0851ead 100644 > --- a/arch/arm64/kernel/kgdb.c > +++ b/arch/arm64/kernel/kgdb.c > @@ -14,6 +14,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -353,3 +354,17 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) > return aarch64_insn_write((void *)bpt->bpt_addr, > *(u32 *)bpt->saved_instr); > } > + > +#ifdef CONFIG_SMP > +void kgdb_roundup_cpus(void) > +{ > + struct cpumask mask; > + > + cpumask_copy(&mask, cpu_online_mask); > + cpumask_clear_cpu(raw_smp_processor_id(), &mask); > + if (cpumask_empty(&mask)) > + return; > + > + arch_send_call_nmi_func_ipi_mask(&mask); > +} > +#endif > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c > index 27c8ee1..c7158f6e8 100644 > --- a/arch/arm64/kernel/smp.c > +++ b/arch/arm64/kernel/smp.c > @@ -31,6 +31,7 @@ > #include > #include > #include > +#include > #include > > #include > @@ -976,9 +977,19 @@ void handle_IPI(int ipinr, struct pt_regs *regs) > /* Handle it as a normal interrupt if not in NMI context */ > if (!in_nmi()) > irq_enter(); > - > - /* nop, IPI handlers for special features can be added here. */ > - > +#ifdef CONFIG_KGDB My vote would be to keep "ifdef"s out of the middle of functions. Can you put your code in "arch/arm64/kernel/kgdb.c" and then have a dummpy no-op function if "CONFIG_KGDB" isn't defined? > + if (atomic_read(&kgdb_active) != -1) { > + /* > + * For kgdb to work properly, we need printk to operate > + * in normal context. > + */ > + if (in_nmi()) > + printk_nmi_exit(); It feels like all the printk management belongs in kgdb_nmicallback(). ...or is there some reason that this isn't a problem for other platforms using NMI? Maybe it's just that nobody has noticed it yet? > + kgdb_nmicallback(raw_smp_processor_id(), regs); Why do you need to call raw_smp_processor_id()? Are you expecting a different value than the local variable "cpu"? > + if (in_nmi()) > + printk_nmi_enter(); > + } > +#endif > if (!in_nmi()) > irq_exit(); > break; Not that I really know what I'm talking about since I really don't know arm64 at this level very well, but I'll ask anyway and probably look like a fool... I had a note that said: * Will Deacon says: * * the whole roundup code is sketchy and it's the only place in the kernel * which tries to perform I-cache maintenance with irqs disabled, leading * to this nasty hack in the arch code: * * https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/include/asm/cacheflush.h#n74 I presume that, if nothing else, the comment needs to be updated. ...but is the situation any better (or worse?) with your new solution? -Doug _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel