From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2531AC4332F for ; Mon, 9 May 2022 16:40:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239221AbiEIQoK (ORCPT ); Mon, 9 May 2022 12:44:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239113AbiEIQoJ (ORCPT ); Mon, 9 May 2022 12:44:09 -0400 Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DA0F54022; Mon, 9 May 2022 09:40:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From: References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=48dgNwNPuCnyD3VVOBf09xe2NjnTaXdW0ldox8dhrAY=; b=eLWL69Jp3Hs5nKIC07c/DVmIhZ tbuXXOS52wa7kstrEszNfv7tW4JmReP+jwvf/WSB2LnPtAjsR79TSRqPXDYlR/UhgTYs3JxE63TxY WFAF2HLKU7kQDGOjOij/SccvoodNoUy/77Q7Rw1bHNyMkUGePOlK3tPiGlMKv0HFeJaDvBKfB5C4T 7/g7A8xShBgXvVATUWH+9aEVvFCEwhRR4Tyt7Zyibz7aNPTHMnncwalkA+46VUbkJ44ERvwAtMmTs hWWtwgYVptx9ug9t6aPZQ1zrU+TA/lel56oNayZ3d7me/ZMI8w2lxg28JbHkciwHu/jkRujv3gimI sEcDvBWQ==; Received: from [177.183.162.244] (helo=[192.168.0.5]) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_128_GCM:128) (Exim) id 1no6QL-0009Bp-09; Mon, 09 May 2022 18:39:53 +0200 Message-ID: <92e03ebf-6a64-69dc-bf16-9552b9fedad8@igalia.com> Date: Mon, 9 May 2022 13:39:21 -0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.1 Subject: Re: [PATCH 24/30] panic: Refactor the panic path Content-Language: en-US To: "d.hatayama@fujitsu.com" , "akpm@linux-foundation.org" , "bhe@redhat.com" , "pmladek@suse.com" , "kexec@lists.infradead.org" , Mark Rutland , Marc Zyngier , "mikelley@microsoft.com" , Hari Bathini Cc: "linux-kernel@vger.kernel.org" , "bcm-kernel-feedback-list@broadcom.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-alpha@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-edac@vger.kernel.org" , "linux-hyperv@vger.kernel.org" , "linux-leds@vger.kernel.org" , "linux-mips@vger.kernel.org" , "linux-parisc@vger.kernel.org" , "linux-pm@vger.kernel.org" , "linux-remoteproc@vger.kernel.org" , "linux-s390@vger.kernel.org" , "linux-tegra@vger.kernel.org" , "linux-um@lists.infradead.org" , "linux-xtensa@linux-xtensa.org" , "netdev@vger.kernel.org" , "openipmi-developer@lists.sourceforge.net" , "rcu@vger.kernel.org" , "sparclinux@vger.kernel.org" , "xen-devel@lists.xenproject.org" , "x86@kernel.org" , "kernel-dev@igalia.com" , "kernel@gpiccoli.net" , "halves@canonical.com" , "fabiomirmar@gmail.com" , "alejandro.j.jimenez@oracle.com" , "andriy.shevchenko@linux.intel.com" , "arnd@arndb.de" , "bp@alien8.de" , "corbet@lwn.net" , "dave.hansen@linux.intel.com" , "dyoung@redhat.com" , "feng.tang@intel.com" , "gregkh@linuxfoundation.org" , "hidehiro.kawai.ez@hitachi.com" , "jgross@suse.com" , "john.ogness@linutronix.de" , "keescook@chromium.org" , "luto@kernel.org" , "mhiramat@kernel.org" , "mingo@redhat.com" , "paulmck@kernel.org" , "peterz@infradead.org" , "rostedt@goodmis.org" , "senozhatsky@chromium.org" , "stern@rowland.harvard.edu" , "tglx@linutronix.de" , "vgoyal@redhat.com" , "vkuznets@redhat.com" , "will@kernel.org" References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-25-gpiccoli@igalia.com> From: "Guilherme G. Piccoli" In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-remoteproc@vger.kernel.org Hey Hatayma, thanks for your great analysis and no need for apologies! I'll comment/respond properly inline below, just noticing here that I've CCed Mark and Marc (from the ARM64 perspective), Michael (Hyper-V perspective) and Hari (PowerPC perspective), besides the usual suspects as Petr, Baoquan, etc. On 09/05/2022 12:16, d.hatayama@fujitsu.com wrote: > Sorry for the delayed response. Unfortunately, I had 10 days holidays > until yesterday... > [...] >> + We currently have 4 lists of panic notifiers; based >> + on the functionality and risk (for panic success) the >> + callbacks are added in a given list. The lists are: >> + - hypervisor/FW notification list (low risk); >> + - informational list (low/medium risk); >> + - pre_reboot list (higher risk); >> + - post_reboot list (only run late in panic and after >> + kdump, not configurable for now). >> + This parameter defines the ordering of the first 3 >> + lists with regards to kdump; the levels determine >> + which set of notifiers execute before kdump. The >> + accepted levels are: >> + 0: kdump is the first thing to run, NO list is >> + executed before kdump. >> + 1: only the hypervisor list is executed before kdump. >> + 2 (default level): the hypervisor list and (*if* > > Hmmm, why are you trying to change default setting? > > Based on the current design of kdump, it's natural to put what the > handlers for these level 1 and level 2 handlers do in > machine_crash_shutdown(), as these are necessary by default, right? > > Or have you already tried that and figured out it's difficult in some > reason and reached the current design? If so, why is that difficult? > Could you point to if there is already such discussion online? > > kdump is designed to perform as little things as possible before > transferring the execution to the 2nd kernel in order to increase > reliability. Just detour to panic() increases risks of kdump failure > in the sense of increasing the executed codes in the abnormal > situation, which is very the note in the explanation of > crash_kexec_post_notifiers. > > Also, the current implementation of crash_kexec_post_notifiers uses > the panic notifier, but this is not from the technical > reason. Ideally, it should have been implemented in the context of > crash_kexec() independently of panic(). > > That is, it looks to me that, in addition to changing design of panic > notifier, you are trying to integrate shutdown code of the crash kexec > and the panic paths. If so, this is a big design change for kdump. > I'm concerned about increase of reliability. I'd like you to discuss > them carefully. >From my understanding (specially based on both these threads [0] and [1]), 3 facts are clear and quite complex in nature: (a) Currently, the panic notifier list is a "no man's land" - it's a mess, all sort of callbacks are added there, some of them are extremely risk for breaking kdump, others are quite safe (like setting a variable). Petr's details in thread [0] are really clear and express in great way how confusing and conflicting the panic notifiers goals are. (b) In order to "address" problems in the antagonistic goals of notifiers (see point (a) above and thread [0]), we have this quirk/workaround called "crash_kexec_post_notifiers". This is useful, but (almost as for attesting how this is working as band-aid over complex and fundamental issues) see the following commits: a11589563e96 ("x86/Hyper-V: Report crash register data or kmsg before running crash kernel") 06e629c25daa ("powerpc/fadump: Fix inaccurate CPU state info in vmcore generated with panic") They hardcode such workaround, because they *need* some notifiers' callbacks. But notice they *don't need all of them*, only some important ones (that usually are good considering the risk, it's a good cost/benefit). Since we currently have an all-or-nothing behavior for the panic notifiers, both PowerPC and Hyper-V end-up bringing all of them to run before kdump due to the lack of flexibility, increasing a lot the risk of failure for kdump. (c) To add on top of all such complexity, we don't have a custom machine_crash_shutdown() handler for some architectures like ARM64, and the feeling is that's not right to add a bunch of callbacks / complexity in such architecture code, specially since we have the notifiers infrastructure in the kernel. I've recently started a discussion about that with ARM64 community, please take a look in [1]. With that said, we can use (a) + (b) + (c) to justify our argument here: the panic notifiers should be refactored! We need to try to encompass the antagonistic goals of kdump (wants to be the first thing to run, early as possible) VS. the notifiers that are necessary, even before kdump (like Hyper-V / PowerPC fadump ones, but there are more of these, the FW/hypervisors notifiers). I guarantee to you we cannot make 100% of people happy - the panic-related areas like kdump, etc are full of trade-offs. We improve something, other stuff breaks. This series attempts to clearly distribute the notifiers in 3 buckets, and introduces a level setting that tunes such buckets to run before or after the kdump in a highly flexible way, trying to make the most users happy and capable of tuning their systems. I understand that likely setting the notifiers to 0 would make kdump maintainers happy, because we'd keep the same behavior as before. But..then we make PPC / Hyper-V and some other users unsatisfied. Level 2 seems to me a good compromise. I'm willing to add a KConfig option to allow distros to hard set their defaults - that might make sense to legacy distros as older RHEL / CentOS or server-only distros. Finally, you mention about integrating the crash shutdown and the panic paths - I'm not officially doing that, but I see that they are very related and all cases I've ever seen in the last 3-4 years I've been working with kdump, it was triggered from panic, from settings "panic_on_X". I understand we have the regular kexec path (!kdump), some notifiers might make sense in both paths, and in this case we could duplicate them (one callback, 2 notifier lists) and control against double execution. I tried doing that in patch 16 of this series (Hyper-V stuff), but due to the lack of ARM64 custom crash shutdown handler, I couldn't. We can think in some alternatives, and improve these (naturally) connected/related paths, but this is not the goal of this specific series. I'm hereby trying to improve/clarify the panic (and kdump) path, not the pure kexec path. Let me know if I can clarify more specific points you have or if there are flaws in my logic - I appreciate the discussions/reviews. Cheers, Guilherme [0] "notifier/panic: Introduce panic_notifier_filter" - https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp@alley/ [1] "Should arm64 have a custom crash shutdown handler?" - https://lore.kernel.org/lkml/427a8277-49f0-4317-d6c3-4a15d7070e55@igalia.com/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E890AC433EF for ; Tue, 10 May 2022 00:03:20 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4KxytH3wpCz3dpX for ; Tue, 10 May 2022 10:03:19 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=igalia.com header.i=@igalia.com header.a=rsa-sha256 header.s=20170329 header.b=eLWL69Jp; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=igalia.com (client-ip=178.60.130.6; helo=fanzine2.igalia.com; envelope-from=gpiccoli@igalia.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=igalia.com header.i=@igalia.com header.a=rsa-sha256 header.s=20170329 header.b=eLWL69Jp; dkim-atps=neutral Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Kxn353wD0z3bd4 for ; Tue, 10 May 2022 02:40:17 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From: References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=48dgNwNPuCnyD3VVOBf09xe2NjnTaXdW0ldox8dhrAY=; b=eLWL69Jp3Hs5nKIC07c/DVmIhZ tbuXXOS52wa7kstrEszNfv7tW4JmReP+jwvf/WSB2LnPtAjsR79TSRqPXDYlR/UhgTYs3JxE63TxY WFAF2HLKU7kQDGOjOij/SccvoodNoUy/77Q7Rw1bHNyMkUGePOlK3tPiGlMKv0HFeJaDvBKfB5C4T 7/g7A8xShBgXvVATUWH+9aEVvFCEwhRR4Tyt7Zyibz7aNPTHMnncwalkA+46VUbkJ44ERvwAtMmTs hWWtwgYVptx9ug9t6aPZQ1zrU+TA/lel56oNayZ3d7me/ZMI8w2lxg28JbHkciwHu/jkRujv3gimI sEcDvBWQ==; Received: from [177.183.162.244] (helo=[192.168.0.5]) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_128_GCM:128) (Exim) id 1no6QL-0009Bp-09; Mon, 09 May 2022 18:39:53 +0200 Message-ID: <92e03ebf-6a64-69dc-bf16-9552b9fedad8@igalia.com> Date: Mon, 9 May 2022 13:39:21 -0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.1 Subject: Re: [PATCH 24/30] panic: Refactor the panic path Content-Language: en-US To: "d.hatayama@fujitsu.com" , "akpm@linux-foundation.org" , "bhe@redhat.com" , "pmladek@suse.com" , "kexec@lists.infradead.org" , Mark Rutland , Marc Zyngier , "mikelley@microsoft.com" , Hari Bathini References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-25-gpiccoli@igalia.com> From: "Guilherme G. Piccoli" In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 10 May 2022 09:58:00 +1000 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "linux-hyperv@vger.kernel.org" , "halves@canonical.com" , "gregkh@linuxfoundation.org" , "peterz@infradead.org" , "alejandro.j.jimenez@oracle.com" , "linux-remoteproc@vger.kernel.org" , "feng.tang@intel.com" , "linux-mips@vger.kernel.org" , "hidehiro.kawai.ez@hitachi.com" , "sparclinux@vger.kernel.org" , "will@kernel.org" , "tglx@linutronix.de" , "linux-leds@vger.kernel.org" , "linux-s390@vger.kernel.org" , "john.ogness@linutronix.de" , "corbet@lwn.net" , "paulmck@kernel.org" , "fabiomirmar@gmail.com" , "x86@kernel.org" , "mingo@redhat.com" , "bcm-kernel-feedback-list@broadcom.com" , "xen-devel@lists.xenproject.org" , "dyoung@redhat.com" , "vgoyal@redhat.com" , "linux-xtensa@linux-xtensa.org" , "dave.hansen@linux.intel.com" , "keescook@chromium.org" , "arnd@arndb.de" , "linux-pm@vger.kernel.org" , "linux-um@lists.infradead.org" , "rostedt@goodmis.org" , "rcu@vger.kernel.org" , "bp@alien8.de" , "luto@kernel.org" , "linux-tegra@vger.kernel.org" , "openipmi-developer@lists.sourceforge.net" , "andriy.shevchenko@linux.intel.com" , "linux-arm-kernel@lists.infradead.org" , "linux-edac@vger.kernel.org" , "jgross@suse.com" , "linux-parisc@vger.kernel.org" , "netdev@vger.kernel.org" , "kernel@gpiccoli.net" , "linux-kernel@vger.kernel.org" , "stern@rowland.harvard.edu" , "senozhatsky@chromium.org" , "mhiramat@kernel.org" , "kernel-dev@igalia.com" , "linux-alpha@vger.kernel.org" , "vkuznets@redhat.com" , "linuxppc-dev@lists.ozlabs.org" Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Hey Hatayma, thanks for your great analysis and no need for apologies! I'll comment/respond properly inline below, just noticing here that I've CCed Mark and Marc (from the ARM64 perspective), Michael (Hyper-V perspective) and Hari (PowerPC perspective), besides the usual suspects as Petr, Baoquan, etc. On 09/05/2022 12:16, d.hatayama@fujitsu.com wrote: > Sorry for the delayed response. Unfortunately, I had 10 days holidays > until yesterday... > [...] >> + We currently have 4 lists of panic notifiers; based >> + on the functionality and risk (for panic success) the >> + callbacks are added in a given list. The lists are: >> + - hypervisor/FW notification list (low risk); >> + - informational list (low/medium risk); >> + - pre_reboot list (higher risk); >> + - post_reboot list (only run late in panic and after >> + kdump, not configurable for now). >> + This parameter defines the ordering of the first 3 >> + lists with regards to kdump; the levels determine >> + which set of notifiers execute before kdump. The >> + accepted levels are: >> + 0: kdump is the first thing to run, NO list is >> + executed before kdump. >> + 1: only the hypervisor list is executed before kdump. >> + 2 (default level): the hypervisor list and (*if* > > Hmmm, why are you trying to change default setting? > > Based on the current design of kdump, it's natural to put what the > handlers for these level 1 and level 2 handlers do in > machine_crash_shutdown(), as these are necessary by default, right? > > Or have you already tried that and figured out it's difficult in some > reason and reached the current design? If so, why is that difficult? > Could you point to if there is already such discussion online? > > kdump is designed to perform as little things as possible before > transferring the execution to the 2nd kernel in order to increase > reliability. Just detour to panic() increases risks of kdump failure > in the sense of increasing the executed codes in the abnormal > situation, which is very the note in the explanation of > crash_kexec_post_notifiers. > > Also, the current implementation of crash_kexec_post_notifiers uses > the panic notifier, but this is not from the technical > reason. Ideally, it should have been implemented in the context of > crash_kexec() independently of panic(). > > That is, it looks to me that, in addition to changing design of panic > notifier, you are trying to integrate shutdown code of the crash kexec > and the panic paths. If so, this is a big design change for kdump. > I'm concerned about increase of reliability. I'd like you to discuss > them carefully. >From my understanding (specially based on both these threads [0] and [1]), 3 facts are clear and quite complex in nature: (a) Currently, the panic notifier list is a "no man's land" - it's a mess, all sort of callbacks are added there, some of them are extremely risk for breaking kdump, others are quite safe (like setting a variable). Petr's details in thread [0] are really clear and express in great way how confusing and conflicting the panic notifiers goals are. (b) In order to "address" problems in the antagonistic goals of notifiers (see point (a) above and thread [0]), we have this quirk/workaround called "crash_kexec_post_notifiers". This is useful, but (almost as for attesting how this is working as band-aid over complex and fundamental issues) see the following commits: a11589563e96 ("x86/Hyper-V: Report crash register data or kmsg before running crash kernel") 06e629c25daa ("powerpc/fadump: Fix inaccurate CPU state info in vmcore generated with panic") They hardcode such workaround, because they *need* some notifiers' callbacks. But notice they *don't need all of them*, only some important ones (that usually are good considering the risk, it's a good cost/benefit). Since we currently have an all-or-nothing behavior for the panic notifiers, both PowerPC and Hyper-V end-up bringing all of them to run before kdump due to the lack of flexibility, increasing a lot the risk of failure for kdump. (c) To add on top of all such complexity, we don't have a custom machine_crash_shutdown() handler for some architectures like ARM64, and the feeling is that's not right to add a bunch of callbacks / complexity in such architecture code, specially since we have the notifiers infrastructure in the kernel. I've recently started a discussion about that with ARM64 community, please take a look in [1]. With that said, we can use (a) + (b) + (c) to justify our argument here: the panic notifiers should be refactored! We need to try to encompass the antagonistic goals of kdump (wants to be the first thing to run, early as possible) VS. the notifiers that are necessary, even before kdump (like Hyper-V / PowerPC fadump ones, but there are more of these, the FW/hypervisors notifiers). I guarantee to you we cannot make 100% of people happy - the panic-related areas like kdump, etc are full of trade-offs. We improve something, other stuff breaks. This series attempts to clearly distribute the notifiers in 3 buckets, and introduces a level setting that tunes such buckets to run before or after the kdump in a highly flexible way, trying to make the most users happy and capable of tuning their systems. I understand that likely setting the notifiers to 0 would make kdump maintainers happy, because we'd keep the same behavior as before. But..then we make PPC / Hyper-V and some other users unsatisfied. Level 2 seems to me a good compromise. I'm willing to add a KConfig option to allow distros to hard set their defaults - that might make sense to legacy distros as older RHEL / CentOS or server-only distros. Finally, you mention about integrating the crash shutdown and the panic paths - I'm not officially doing that, but I see that they are very related and all cases I've ever seen in the last 3-4 years I've been working with kdump, it was triggered from panic, from settings "panic_on_X". I understand we have the regular kexec path (!kdump), some notifiers might make sense in both paths, and in this case we could duplicate them (one callback, 2 notifier lists) and control against double execution. I tried doing that in patch 16 of this series (Hyper-V stuff), but due to the lack of ARM64 custom crash shutdown handler, I couldn't. We can think in some alternatives, and improve these (naturally) connected/related paths, but this is not the goal of this specific series. I'm hereby trying to improve/clarify the panic (and kdump) path, not the pure kexec path. Let me know if I can clarify more specific points you have or if there are flaws in my logic - I appreciate the discussions/reviews. Cheers, Guilherme [0] "notifier/panic: Introduce panic_notifier_filter" - https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp@alley/ [1] "Should arm64 have a custom crash shutdown handler?" - https://lore.kernel.org/lkml/427a8277-49f0-4317-d6c3-4a15d7070e55@igalia.com/ From mboxrd@z Thu Jan 1 00:00:00 1970 From: Guilherme G. Piccoli Date: Mon, 9 May 2022 13:39:21 -0300 Subject: [PATCH 24/30] panic: Refactor the panic path In-Reply-To: References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-25-gpiccoli@igalia.com> Message-ID: <92e03ebf-6a64-69dc-bf16-9552b9fedad8@igalia.com> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: kexec@lists.infradead.org Hey Hatayma, thanks for your great analysis and no need for apologies! I'll comment/respond properly inline below, just noticing here that I've CCed Mark and Marc (from the ARM64 perspective), Michael (Hyper-V perspective) and Hari (PowerPC perspective), besides the usual suspects as Petr, Baoquan, etc. On 09/05/2022 12:16, d.hatayama at fujitsu.com wrote: > Sorry for the delayed response. Unfortunately, I had 10 days holidays > until yesterday... > [...] >> + We currently have 4 lists of panic notifiers; based >> + on the functionality and risk (for panic success) the >> + callbacks are added in a given list. The lists are: >> + - hypervisor/FW notification list (low risk); >> + - informational list (low/medium risk); >> + - pre_reboot list (higher risk); >> + - post_reboot list (only run late in panic and after >> + kdump, not configurable for now). >> + This parameter defines the ordering of the first 3 >> + lists with regards to kdump; the levels determine >> + which set of notifiers execute before kdump. The >> + accepted levels are: >> + 0: kdump is the first thing to run, NO list is >> + executed before kdump. >> + 1: only the hypervisor list is executed before kdump. >> + 2 (default level): the hypervisor list and (*if* > > Hmmm, why are you trying to change default setting? > > Based on the current design of kdump, it's natural to put what the > handlers for these level 1 and level 2 handlers do in > machine_crash_shutdown(), as these are necessary by default, right? > > Or have you already tried that and figured out it's difficult in some > reason and reached the current design? If so, why is that difficult? > Could you point to if there is already such discussion online? > > kdump is designed to perform as little things as possible before > transferring the execution to the 2nd kernel in order to increase > reliability. Just detour to panic() increases risks of kdump failure > in the sense of increasing the executed codes in the abnormal > situation, which is very the note in the explanation of > crash_kexec_post_notifiers. > > Also, the current implementation of crash_kexec_post_notifiers uses > the panic notifier, but this is not from the technical > reason. Ideally, it should have been implemented in the context of > crash_kexec() independently of panic(). > > That is, it looks to me that, in addition to changing design of panic > notifier, you are trying to integrate shutdown code of the crash kexec > and the panic paths. If so, this is a big design change for kdump. > I'm concerned about increase of reliability. I'd like you to discuss > them carefully. >From my understanding (specially based on both these threads [0] and [1]), 3 facts are clear and quite complex in nature: (a) Currently, the panic notifier list is a "no man's land" - it's a mess, all sort of callbacks are added there, some of them are extremely risk for breaking kdump, others are quite safe (like setting a variable). Petr's details in thread [0] are really clear and express in great way how confusing and conflicting the panic notifiers goals are. (b) In order to "address" problems in the antagonistic goals of notifiers (see point (a) above and thread [0]), we have this quirk/workaround called "crash_kexec_post_notifiers". This is useful, but (almost as for attesting how this is working as band-aid over complex and fundamental issues) see the following commits: a11589563e96 ("x86/Hyper-V: Report crash register data or kmsg before running crash kernel") 06e629c25daa ("powerpc/fadump: Fix inaccurate CPU state info in vmcore generated with panic") They hardcode such workaround, because they *need* some notifiers' callbacks. But notice they *don't need all of them*, only some important ones (that usually are good considering the risk, it's a good cost/benefit). Since we currently have an all-or-nothing behavior for the panic notifiers, both PowerPC and Hyper-V end-up bringing all of them to run before kdump due to the lack of flexibility, increasing a lot the risk of failure for kdump. (c) To add on top of all such complexity, we don't have a custom machine_crash_shutdown() handler for some architectures like ARM64, and the feeling is that's not right to add a bunch of callbacks / complexity in such architecture code, specially since we have the notifiers infrastructure in the kernel. I've recently started a discussion about that with ARM64 community, please take a look in [1]. With that said, we can use (a) + (b) + (c) to justify our argument here: the panic notifiers should be refactored! We need to try to encompass the antagonistic goals of kdump (wants to be the first thing to run, early as possible) VS. the notifiers that are necessary, even before kdump (like Hyper-V / PowerPC fadump ones, but there are more of these, the FW/hypervisors notifiers). I guarantee to you we cannot make 100% of people happy - the panic-related areas like kdump, etc are full of trade-offs. We improve something, other stuff breaks. This series attempts to clearly distribute the notifiers in 3 buckets, and introduces a level setting that tunes such buckets to run before or after the kdump in a highly flexible way, trying to make the most users happy and capable of tuning their systems. I understand that likely setting the notifiers to 0 would make kdump maintainers happy, because we'd keep the same behavior as before. But..then we make PPC / Hyper-V and some other users unsatisfied. Level 2 seems to me a good compromise. I'm willing to add a KConfig option to allow distros to hard set their defaults - that might make sense to legacy distros as older RHEL / CentOS or server-only distros. Finally, you mention about integrating the crash shutdown and the panic paths - I'm not officially doing that, but I see that they are very related and all cases I've ever seen in the last 3-4 years I've been working with kdump, it was triggered from panic, from settings "panic_on_X". I understand we have the regular kexec path (!kdump), some notifiers might make sense in both paths, and in this case we could duplicate them (one callback, 2 notifier lists) and control against double execution. I tried doing that in patch 16 of this series (Hyper-V stuff), but due to the lack of ARM64 custom crash shutdown handler, I couldn't. We can think in some alternatives, and improve these (naturally) connected/related paths, but this is not the goal of this specific series. I'm hereby trying to improve/clarify the panic (and kdump) path, not the pure kexec path. Let me know if I can clarify more specific points you have or if there are flaws in my logic - I appreciate the discussions/reviews. Cheers, Guilherme [0] "notifier/panic: Introduce panic_notifier_filter" - https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp at alley/ [1] "Should arm64 have a custom crash shutdown handler?" - https://lore.kernel.org/lkml/427a8277-49f0-4317-d6c3-4a15d7070e55 at igalia.com/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Message-ID: <92e03ebf-6a64-69dc-bf16-9552b9fedad8@igalia.com> Date: Mon, 9 May 2022 13:39:21 -0300 MIME-Version: 1.0 Subject: Re: [PATCH 24/30] panic: Refactor the panic path Content-Language: en-US References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-25-gpiccoli@igalia.com> From: "Guilherme G. Piccoli" In-Reply-To: List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: "d.hatayama@fujitsu.com" , "akpm@linux-foundation.org" , "bhe@redhat.com" , "pmladek@suse.com" , "kexec@lists.infradead.org" , Mark Rutland , Marc Zyngier , "mikelley@microsoft.com" , Hari Bathini Cc: "linux-kernel@vger.kernel.org" , "bcm-kernel-feedback-list@broadcom.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-alpha@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-edac@vger.kernel.org" , "linux-hyperv@vger.kernel.org" , "linux-leds@vger.kernel.org" , "linux-mips@vger.kernel.org" , "linux-parisc@vger.kernel.org" , "linux-pm@vger.kernel.org" , "linux-remoteproc@vger.kernel.org" , "linux-s390@vger.kernel.org" , "linux-tegra@vger.kernel.org" , "linux-um@lists.infradead.org" , "linux-xtensa@linux-xtensa.org" , "netdev@vger.kernel.org" , "openipmi-developer@lists.sourceforge.net" , "rcu@vger.kernel.org" , "sparclinux@vger.kernel.org" , "xen-devel@lists.xenproject.org" , "x86@kernel.org" , "kernel-dev@igalia.com" , "kernel@gpiccoli.net" , "halves@canonical.com" , "fabiomirmar@gmail.com" , "alejandro.j.jimenez@oracle.com" , "andriy.shevchenko@linux.intel.com" , "arnd@arndb.de" , "bp@alien8.de" , "corbet@lwn.net" , "dave.hansen@linux.intel.com" , "dyoung@redhat.com" , "feng.tang@intel.com" , "gregkh@linuxfoundation.org" , "hidehiro.kawai.ez@hitachi.com" , "jgross@suse.com" , "john.ogness@linutronix.de" , "keescook@chromium.org" , "luto@kernel.org" , "mhiramat@kernel.org" , "mingo@redhat.com" , "paulmck@kernel.org" , "peterz@infradead.org" , "rostedt@goodmis.org" , "senozhatsky@chromium.org" , "stern@rowland.harvard.edu" , "tglx@linutronix.de" , "vgoyal@redhat.com" , "vkuznets@redhat.com" , "will@kernel.org" Hey Hatayma, thanks for your great analysis and no need for apologies! I'll comment/respond properly inline below, just noticing here that I've CCed Mark and Marc (from the ARM64 perspective), Michael (Hyper-V perspective) and Hari (PowerPC perspective), besides the usual suspects as Petr, Baoquan, etc. On 09/05/2022 12:16, d.hatayama@fujitsu.com wrote: > Sorry for the delayed response. Unfortunately, I had 10 days holidays > until yesterday... > [...] >> + We currently have 4 lists of panic notifiers; based >> + on the functionality and risk (for panic success) the >> + callbacks are added in a given list. The lists are: >> + - hypervisor/FW notification list (low risk); >> + - informational list (low/medium risk); >> + - pre_reboot list (higher risk); >> + - post_reboot list (only run late in panic and after >> + kdump, not configurable for now). >> + This parameter defines the ordering of the first 3 >> + lists with regards to kdump; the levels determine >> + which set of notifiers execute before kdump. The >> + accepted levels are: >> + 0: kdump is the first thing to run, NO list is >> + executed before kdump. >> + 1: only the hypervisor list is executed before kdump. >> + 2 (default level): the hypervisor list and (*if* > > Hmmm, why are you trying to change default setting? > > Based on the current design of kdump, it's natural to put what the > handlers for these level 1 and level 2 handlers do in > machine_crash_shutdown(), as these are necessary by default, right? > > Or have you already tried that and figured out it's difficult in some > reason and reached the current design? If so, why is that difficult? > Could you point to if there is already such discussion online? > > kdump is designed to perform as little things as possible before > transferring the execution to the 2nd kernel in order to increase > reliability. Just detour to panic() increases risks of kdump failure > in the sense of increasing the executed codes in the abnormal > situation, which is very the note in the explanation of > crash_kexec_post_notifiers. > > Also, the current implementation of crash_kexec_post_notifiers uses > the panic notifier, but this is not from the technical > reason. Ideally, it should have been implemented in the context of > crash_kexec() independently of panic(). > > That is, it looks to me that, in addition to changing design of panic > notifier, you are trying to integrate shutdown code of the crash kexec > and the panic paths. If so, this is a big design change for kdump. > I'm concerned about increase of reliability. I'd like you to discuss > them carefully. >From my understanding (specially based on both these threads [0] and [1]), 3 facts are clear and quite complex in nature: (a) Currently, the panic notifier list is a "no man's land" - it's a mess, all sort of callbacks are added there, some of them are extremely risk for breaking kdump, others are quite safe (like setting a variable). Petr's details in thread [0] are really clear and express in great way how confusing and conflicting the panic notifiers goals are. (b) In order to "address" problems in the antagonistic goals of notifiers (see point (a) above and thread [0]), we have this quirk/workaround called "crash_kexec_post_notifiers". This is useful, but (almost as for attesting how this is working as band-aid over complex and fundamental issues) see the following commits: a11589563e96 ("x86/Hyper-V: Report crash register data or kmsg before running crash kernel") 06e629c25daa ("powerpc/fadump: Fix inaccurate CPU state info in vmcore generated with panic") They hardcode such workaround, because they *need* some notifiers' callbacks. But notice they *don't need all of them*, only some important ones (that usually are good considering the risk, it's a good cost/benefit). Since we currently have an all-or-nothing behavior for the panic notifiers, both PowerPC and Hyper-V end-up bringing all of them to run before kdump due to the lack of flexibility, increasing a lot the risk of failure for kdump. (c) To add on top of all such complexity, we don't have a custom machine_crash_shutdown() handler for some architectures like ARM64, and the feeling is that's not right to add a bunch of callbacks / complexity in such architecture code, specially since we have the notifiers infrastructure in the kernel. I've recently started a discussion about that with ARM64 community, please take a look in [1]. With that said, we can use (a) + (b) + (c) to justify our argument here: the panic notifiers should be refactored! We need to try to encompass the antagonistic goals of kdump (wants to be the first thing to run, early as possible) VS. the notifiers that are necessary, even before kdump (like Hyper-V / PowerPC fadump ones, but there are more of these, the FW/hypervisors notifiers). I guarantee to you we cannot make 100% of people happy - the panic-related areas like kdump, etc are full of trade-offs. We improve something, other stuff breaks. This series attempts to clearly distribute the notifiers in 3 buckets, and introduces a level setting that tunes such buckets to run before or after the kdump in a highly flexible way, trying to make the most users happy and capable of tuning their systems. I understand that likely setting the notifiers to 0 would make kdump maintainers happy, because we'd keep the same behavior as before. But..then we make PPC / Hyper-V and some other users unsatisfied. Level 2 seems to me a good compromise. I'm willing to add a KConfig option to allow distros to hard set their defaults - that might make sense to legacy distros as older RHEL / CentOS or server-only distros. Finally, you mention about integrating the crash shutdown and the panic paths - I'm not officially doing that, but I see that they are very related and all cases I've ever seen in the last 3-4 years I've been working with kdump, it was triggered from panic, from settings "panic_on_X". I understand we have the regular kexec path (!kdump), some notifiers might make sense in both paths, and in this case we could duplicate them (one callback, 2 notifier lists) and control against double execution. I tried doing that in patch 16 of this series (Hyper-V stuff), but due to the lack of ARM64 custom crash shutdown handler, I couldn't. We can think in some alternatives, and improve these (naturally) connected/related paths, but this is not the goal of this specific series. I'm hereby trying to improve/clarify the panic (and kdump) path, not the pure kexec path. Let me know if I can clarify more specific points you have or if there are flaws in my logic - I appreciate the discussions/reviews. Cheers, Guilherme [0] "notifier/panic: Introduce panic_notifier_filter" - https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp@alley/ [1] "Should arm64 have a custom crash shutdown handler?" - https://lore.kernel.org/lkml/427a8277-49f0-4317-d6c3-4a15d7070e55@igalia.com/ _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Guilherme G. Piccoli" Subject: Re: [PATCH 24/30] panic: Refactor the panic path Date: Mon, 9 May 2022 13:39:21 -0300 Message-ID: <92e03ebf-6a64-69dc-bf16-9552b9fedad8@igalia.com> References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-25-gpiccoli@igalia.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From: References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=48dgNwNPuCnyD3VVOBf09xe2NjnTaXdW0ldox8dhrAY=; b=eLWL69Jp3Hs5nKIC07c/DVmIhZ tbuXXOS52wa7kstrEszNfv7tW4JmReP+jwvf/WSB2LnPtAjsR79TSRqPXDYlR/UhgTYs3JxE63TxY WFAF2HLKU7kQDGOjOij/SccvoodNoUy/77Q7Rw1bHNyMkUGePOlK3tPiGlMKv0HFeJaDvBKfB5C4T 7/g7A8xShBgXvVATUWH+9aEVvFCEwhRR4Tyt7Zyibz7aNPTHMnncwalkA+46VUbkJ44ERvwAtMmTs hWWtwgYVptx9ug9t6aPZQ1zrU+TA/lel56oNayZ3d7me/ZMI8w2lxg28JbHkciwHu/jkRujv3gimI sEcDvBW Content-Language: en-US In-Reply-To: List-ID: Content-Type: text/plain; charset="us-ascii" To: "d.hatayama-LMvhtfratI1BDgjK7y7TUQ@public.gmane.org" , "akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org" , "bhe-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org" , "pmladek-IBi9RG/b67k@public.gmane.org" , "kexec-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , Mark Rutland , Marc Zyngier , "mikelley-0li6OtcxBFHby3iVrkZq2A@public.gmane.org" , Hari Bathini Cc: "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "bcm-kernel-feedback-list-dY08KVG/lbpWk0Htik3J/w@public.gmane.org" , "linuxppc-dev-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org" , "linux-alpha-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , "linux-edac-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-hyperv-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-leds-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-mips-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-parisc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-remoteproc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , lin Hey Hatayma, thanks for your great analysis and no need for apologies! I'll comment/respond properly inline below, just noticing here that I've CCed Mark and Marc (from the ARM64 perspective), Michael (Hyper-V perspective) and Hari (PowerPC perspective), besides the usual suspects as Petr, Baoquan, etc. On 09/05/2022 12:16, d.hatayama-LMvhtfratI1BDgjK7y7TUQ@public.gmane.org wrote: > Sorry for the delayed response. Unfortunately, I had 10 days holidays > until yesterday... > [...] >> + We currently have 4 lists of panic notifiers; based >> + on the functionality and risk (for panic success) the >> + callbacks are added in a given list. The lists are: >> + - hypervisor/FW notification list (low risk); >> + - informational list (low/medium risk); >> + - pre_reboot list (higher risk); >> + - post_reboot list (only run late in panic and after >> + kdump, not configurable for now). >> + This parameter defines the ordering of the first 3 >> + lists with regards to kdump; the levels determine >> + which set of notifiers execute before kdump. The >> + accepted levels are: >> + 0: kdump is the first thing to run, NO list is >> + executed before kdump. >> + 1: only the hypervisor list is executed before kdump. >> + 2 (default level): the hypervisor list and (*if* > > Hmmm, why are you trying to change default setting? > > Based on the current design of kdump, it's natural to put what the > handlers for these level 1 and level 2 handlers do in > machine_crash_shutdown(), as these are necessary by default, right? > > Or have you already tried that and figured out it's difficult in some > reason and reached the current design? If so, why is that difficult? > Could you point to if there is already such discussion online? > > kdump is designed to perform as little things as possible before > transferring the execution to the 2nd kernel in order to increase > reliability. Just detour to panic() increases risks of kdump failure > in the sense of increasing the executed codes in the abnormal > situation, which is very the note in the explanation of > crash_kexec_post_notifiers. > > Also, the current implementation of crash_kexec_post_notifiers uses > the panic notifier, but this is not from the technical > reason. Ideally, it should have been implemented in the context of > crash_kexec() independently of panic(). > > That is, it looks to me that, in addition to changing design of panic > notifier, you are trying to integrate shutdown code of the crash kexec > and the panic paths. If so, this is a big design change for kdump. > I'm concerned about increase of reliability. I'd like you to discuss > them carefully. >From my understanding (specially based on both these threads [0] and [1]), 3 facts are clear and quite complex in nature: (a) Currently, the panic notifier list is a "no man's land" - it's a mess, all sort of callbacks are added there, some of them are extremely risk for breaking kdump, others are quite safe (like setting a variable). Petr's details in thread [0] are really clear and express in great way how confusing and conflicting the panic notifiers goals are. (b) In order to "address" problems in the antagonistic goals of notifiers (see point (a) above and thread [0]), we have this quirk/workaround called "crash_kexec_post_notifiers". This is useful, but (almost as for attesting how this is working as band-aid over complex and fundamental issues) see the following commits: a11589563e96 ("x86/Hyper-V: Report crash register data or kmsg before running crash kernel") 06e629c25daa ("powerpc/fadump: Fix inaccurate CPU state info in vmcore generated with panic") They hardcode such workaround, because they *need* some notifiers' callbacks. But notice they *don't need all of them*, only some important ones (that usually are good considering the risk, it's a good cost/benefit). Since we currently have an all-or-nothing behavior for the panic notifiers, both PowerPC and Hyper-V end-up bringing all of them to run before kdump due to the lack of flexibility, increasing a lot the risk of failure for kdump. (c) To add on top of all such complexity, we don't have a custom machine_crash_shutdown() handler for some architectures like ARM64, and the feeling is that's not right to add a bunch of callbacks / complexity in such architecture code, specially since we have the notifiers infrastructure in the kernel. I've recently started a discussion about that with ARM64 community, please take a look in [1]. With that said, we can use (a) + (b) + (c) to justify our argument here: the panic notifiers should be refactored! We need to try to encompass the antagonistic goals of kdump (wants to be the first thing to run, early as possible) VS. the notifiers that are necessary, even before kdump (like Hyper-V / PowerPC fadump ones, but there are more of these, the FW/hypervisors notifiers). I guarantee to you we cannot make 100% of people happy - the panic-related areas like kdump, etc are full of trade-offs. We improve something, other stuff breaks. This series attempts to clearly distribute the notifiers in 3 buckets, and introduces a level setting that tunes such buckets to run before or after the kdump in a highly flexible way, trying to make the most users happy and capable of tuning their systems. I understand that likely setting the notifiers to 0 would make kdump maintainers happy, because we'd keep the same behavior as before. But..then we make PPC / Hyper-V and some other users unsatisfied. Level 2 seems to me a good compromise. I'm willing to add a KConfig option to allow distros to hard set their defaults - that might make sense to legacy distros as older RHEL / CentOS or server-only distros. Finally, you mention about integrating the crash shutdown and the panic paths - I'm not officially doing that, but I see that they are very related and all cases I've ever seen in the last 3-4 years I've been working with kdump, it was triggered from panic, from settings "panic_on_X". I understand we have the regular kexec path (!kdump), some notifiers might make sense in both paths, and in this case we could duplicate them (one callback, 2 notifier lists) and control against double execution. I tried doing that in patch 16 of this series (Hyper-V stuff), but due to the lack of ARM64 custom crash shutdown handler, I couldn't. We can think in some alternatives, and improve these (naturally) connected/related paths, but this is not the goal of this specific series. I'm hereby trying to improve/clarify the panic (and kdump) path, not the pure kexec path. Let me know if I can clarify more specific points you have or if there are flaws in my logic - I appreciate the discussions/reviews. Cheers, Guilherme [0] "notifier/panic: Introduce panic_notifier_filter" - https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp@alley/ [1] "Should arm64 have a custom crash shutdown handler?" - https://lore.kernel.org/lkml/427a8277-49f0-4317-d6c3-4a15d7070e55-wEGTBA9jqPzQT0dZR+AlfA@public.gmane.org/