From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F1C9C433EF for ; Thu, 19 May 2022 12:20:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237845AbiESMUk (ORCPT ); Thu, 19 May 2022 08:20:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232584AbiESMUg (ORCPT ); Thu, 19 May 2022 08:20:36 -0400 Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B947AB041F; Thu, 19 May 2022 05:20:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From: References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=e6xrJm1a3ay+4weLnrCkh2SRSEeyZycIIB/twweqf9Y=; b=BhOEtl3IEpq+ZutVgH7P2rHsfA mhGkC2PSuDrVyQ8IQfXtbB2pfD2fLPqKTP1L2vPLew3hLPbkforpZGHUalxvr86M8T2G/zNh0P/pw wi++cJ5XXfisCo4sQPAYOpaVMhBvNWB3svhNAgbQKxYFPy0Oeko+Z9jhUOVMogS4VQkCRHPLo2bnq jjqxb69zYzPORFRcZZlayXhHSqDz7B+6f3XjEVdBQ2ftzK2PDz+TwQa11JztdT1Ouzc0j+U1m1POY rbklHHuyIwI+fwGIyQbgwTEldlCv1PEui/a5Qzds7nWr2ZDdu+bcGybfUaDSvUQ/88wOr+qjGKQ2s 3BFZ9Urg==; Received: from 200-161-159-120.dsl.telesp.net.br ([200.161.159.120] helo=[192.168.1.60]) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_128_GCM:128) (Exim) id 1nrf8d-00BEVm-Gn; Thu, 19 May 2022 14:20:19 +0200 Message-ID: Date: Thu, 19 May 2022 09:19:31 -0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list Content-Language: en-US To: Scott Branden , Petr Mladek , Sebastian Reichel , Florian Fainelli , Desmond yan Cc: David Gow , Evan Green , Julius Werner , bcm-kernel-feedback-list@broadcom.com, linux-pm@vger.kernel.org, akpm@linux-foundation.org, bhe@redhat.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-edac@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-leds@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, linux-tegra@vger.kernel.org, linux-um@lists.infradead.org, linux-xtensa@linux-xtensa.org, netdev@vger.kernel.org, openipmi-developer@lists.sourceforge.net, rcu@vger.kernel.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org, kernel-dev@igalia.com, kernel@gpiccoli.net, halves@canonical.com, fabiomirmar@gmail.com, alejandro.j.jimenez@oracle.com, andriy.shevchenko@linux.intel.com, arnd@arndb.de, bp@alien8.de, corbet@lwn.net, d.hatayama@jp.fujitsu.com, dave.hansen@linux.intel.com, dyoung@redhat.com, feng.tang@intel.com, gregkh@linuxfoundation.org, mikelley@microsoft.com, hidehiro.kawai.ez@hitachi.com, jgross@suse.com, john.ogness@linutronix.de, keescook@chromium.org, luto@kernel.org, mhiramat@kernel.org, mingo@redhat.com, paulmck@kernel.org, peterz@infradead.org, rostedt@goodmis.org, senozhatsky@chromium.org, stern@rowland.harvard.edu, tglx@linutronix.de, vgoyal@redhat.com, vkuznets@redhat.com, will@kernel.org, Alexander Gordeev , Andrea Parri , Ard Biesheuvel , Benjamin Herrenschmidt , Brian Norris , Christian Borntraeger , Christophe JAILLET , "David S. Miller" , Dexuan Cui , Doug Berger , Haiyang Zhang , Hari Bathini , Heiko Carstens , Justin Chen , "K. Y. Srinivasan" , Lee Jones , Markus Mayer , Michael Ellerman , Mihai Carabas , Nicholas Piggin , Paul Mackerras , Pavel Machek , Shile Zhang , Stephen Hemminger , Sven Schnelle , Thomas Bogendoerfer , Tianyu Lan , Vasily Gorbik , Wang ShaoBo , Wei Liu , zhenwei pi References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-20-gpiccoli@igalia.com> <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> From: "Guilherme G. Piccoli" In-Reply-To: <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-remoteproc@vger.kernel.org On 18/05/2022 19:17, Scott Branden wrote: > Hi Guilherme, > > +Desmond > [...] >>>> I'm afraid it breaks kdump if this device is not reset beforehand - it's >>>> a doorbell write, so not high risk I think... >>>> >>>> But in case the not-reset device can be probed normally in kdump kernel, >>>> then I'm fine in moving this to the reboot list! I don't have the HW to >>>> test myself. >>> >>> Good question. Well, it if has to be called before kdump then >>> even "hypervisor" list is a wrong place because is not always >>> called before kdump. >> [...] > We register to the panic notifier so that we can kill the VK card ASAP > to stop DMAing things over to the host side. If it is not notified then > memory may not be frozen when kdump is occurring. > Notifying the card on panic is also needed to allow for any type of > reset to occur. > > So, the only thing preventing moving the notifier later is the chance > that memory is modified while kdump is occurring. Or, if DMA is > disabled before kdump already then this wouldn't be an issue and the > notification to the card (to allow for clean resets) can be done later. Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases? I'm not expert on that, but I guess whether DMA is "kept" or not depends a bit if IOMMU is used. IIRC, there was a copy of the DMAR table in kdump (at least for Intel IOMMU). Also, devices are not properly quiesced on kdump IIUC, we don't call shutdown/reset handlers, they're skip due to the crash nature - so there is a risk of devices doing bad things in the new kernel. With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters. With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what. Let me know your thoughts Scott / Desmond / Petr and all interested parties. Cheers, Guilherme From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03DD2C433EF for ; Thu, 19 May 2022 22:17:13 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4L443D4rd8z3cjC for ; Fri, 20 May 2022 08:17:12 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=igalia.com header.i=@igalia.com header.a=rsa-sha256 header.s=20170329 header.b=BhOEtl3I; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=igalia.com (client-ip=178.60.130.6; helo=fanzine2.igalia.com; envelope-from=gpiccoli@igalia.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=igalia.com header.i=@igalia.com header.a=rsa-sha256 header.s=20170329 header.b=BhOEtl3I; dkim-atps=neutral Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4L3pq259ZKz2xTX for ; Thu, 19 May 2022 22:20:43 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From: References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=e6xrJm1a3ay+4weLnrCkh2SRSEeyZycIIB/twweqf9Y=; b=BhOEtl3IEpq+ZutVgH7P2rHsfA mhGkC2PSuDrVyQ8IQfXtbB2pfD2fLPqKTP1L2vPLew3hLPbkforpZGHUalxvr86M8T2G/zNh0P/pw wi++cJ5XXfisCo4sQPAYOpaVMhBvNWB3svhNAgbQKxYFPy0Oeko+Z9jhUOVMogS4VQkCRHPLo2bnq jjqxb69zYzPORFRcZZlayXhHSqDz7B+6f3XjEVdBQ2ftzK2PDz+TwQa11JztdT1Ouzc0j+U1m1POY rbklHHuyIwI+fwGIyQbgwTEldlCv1PEui/a5Qzds7nWr2ZDdu+bcGybfUaDSvUQ/88wOr+qjGKQ2s 3BFZ9Urg==; Received: from 200-161-159-120.dsl.telesp.net.br ([200.161.159.120] helo=[192.168.1.60]) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_128_GCM:128) (Exim) id 1nrf8d-00BEVm-Gn; Thu, 19 May 2022 14:20:19 +0200 Message-ID: Date: Thu, 19 May 2022 09:19:31 -0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list Content-Language: en-US To: Scott Branden , Petr Mladek , Sebastian Reichel , Florian Fainelli , Desmond yan References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-20-gpiccoli@igalia.com> <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> From: "Guilherme G. Piccoli" In-Reply-To: <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 20 May 2022 08:15:30 +1000 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paul Mackerras , Justin Chen , Pavel Machek , Alexander Gordeev , "K. Y. Srinivasan" , Wei Liu , stern@rowland.harvard.edu, xen-devel@lists.xenproject.org, Christian Borntraeger , linux-pm@vger.kernel.org, linux-um@lists.infradead.org, Nicholas Piggin , luto@kernel.org, Mihai Carabas , tglx@linutronix.de, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, senozhatsky@chromium.org, d.hatayama@jp.fujitsu.com, Sven Schnelle , akpm@linux-foundation.org, linux-hyperv@vger.kernel.org, dave.hansen@linux.intel.com, linux-s390@vger.kernel.org, Stephen Hemminger , Vasily Gorbik , vgoyal@redhat.com, mhiramat@kernel.org, Andrea Parri , linux-xtensa@linux-xtensa.org, john.ogness@linutronix.de, Markus Mayer , hidehiro.kawai.ez@hitachi.com, linux-arm-kernel@lists.infradead.org, kernel-dev@igalia.com, fabiomirmar@gmail.com, halves@canonical.com, alejandro.j.jimenez@oracle.com, feng.tang@intel.com, zhenwei pi , will@kernel.org, Doug Berger , bhe@redhat.com, corbet@lwn.net, Dexuan Cui , Evan Green , bcm-kernel-feedback-list@broadcom.com, Tianyu Lan , keescook@chromium.org, arnd@arndb.de, Haiyang Zhang , rostedt@goodmis.org, rcu@vger.kernel.org, bp@alien8.de, openipmi-developer@lists.sourceforge.net, Thomas Bogendoerfer , linux-parisc@vger.kernel.org, linux-alpha@vger.kernel.org, Brian Norris , "David S. Miller" , peterz@infradead.org, linux-remoteproc@vger.kernel.org, mikelley@microsoft.com, sparclinux@vger.kernel.org, Lee Jones , Ard Biesheuvel , linux-leds@vger.kernel.org, x86@kernel.org, mingo@redhat.com, dyoung@redhat.com, paulmck@kernel.org, Heiko Carstens , Shile Zhang , Wang ShaoBo , Christophe JAILLET , David Gow , linux-tegra@vger.kernel.org, andriy.shevchenko@linux.intel.com, Hari Bathini , linux-edac@vger.kernel.org, jgross@suse.com, netdev@vger.kernel.org, kernel@gpiccoli.net, kexec@lists.infradead.org, linux-mips@vger.kernel.org, Julius Werner , vkuznets@redhat.com, linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On 18/05/2022 19:17, Scott Branden wrote: > Hi Guilherme, > > +Desmond > [...] >>>> I'm afraid it breaks kdump if this device is not reset beforehand - it's >>>> a doorbell write, so not high risk I think... >>>> >>>> But in case the not-reset device can be probed normally in kdump kernel, >>>> then I'm fine in moving this to the reboot list! I don't have the HW to >>>> test myself. >>> >>> Good question. Well, it if has to be called before kdump then >>> even "hypervisor" list is a wrong place because is not always >>> called before kdump. >> [...] > We register to the panic notifier so that we can kill the VK card ASAP > to stop DMAing things over to the host side. If it is not notified then > memory may not be frozen when kdump is occurring. > Notifying the card on panic is also needed to allow for any type of > reset to occur. > > So, the only thing preventing moving the notifier later is the chance > that memory is modified while kdump is occurring. Or, if DMA is > disabled before kdump already then this wouldn't be an issue and the > notification to the card (to allow for clean resets) can be done later. Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases? I'm not expert on that, but I guess whether DMA is "kept" or not depends a bit if IOMMU is used. IIRC, there was a copy of the DMAR table in kdump (at least for Intel IOMMU). Also, devices are not properly quiesced on kdump IIUC, we don't call shutdown/reset handlers, they're skip due to the crash nature - so there is a risk of devices doing bad things in the new kernel. With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters. With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what. Let me know your thoughts Scott / Desmond / Petr and all interested parties. Cheers, Guilherme From mboxrd@z Thu Jan 1 00:00:00 1970 From: Guilherme G. Piccoli Date: Thu, 19 May 2022 09:19:31 -0300 Subject: [PATCH 19/30] panic: Add the panic hypervisor notifier list In-Reply-To: <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-20-gpiccoli@igalia.com> <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> Message-ID: List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: kexec@lists.infradead.org On 18/05/2022 19:17, Scott Branden wrote: > Hi Guilherme, > > +Desmond > [...] >>>> I'm afraid it breaks kdump if this device is not reset beforehand - it's >>>> a doorbell write, so not high risk I think... >>>> >>>> But in case the not-reset device can be probed normally in kdump kernel, >>>> then I'm fine in moving this to the reboot list! I don't have the HW to >>>> test myself. >>> >>> Good question. Well, it if has to be called before kdump then >>> even "hypervisor" list is a wrong place because is not always >>> called before kdump. >> [...] > We register to the panic notifier so that we can kill the VK card ASAP > to stop DMAing things over to the host side. If it is not notified then > memory may not be frozen when kdump is occurring. > Notifying the card on panic is also needed to allow for any type of > reset to occur. > > So, the only thing preventing moving the notifier later is the chance > that memory is modified while kdump is occurring. Or, if DMA is > disabled before kdump already then this wouldn't be an issue and the > notification to the card (to allow for clean resets) can be done later. Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases? I'm not expert on that, but I guess whether DMA is "kept" or not depends a bit if IOMMU is used. IIRC, there was a copy of the DMAR table in kdump (at least for Intel IOMMU). Also, devices are not properly quiesced on kdump IIUC, we don't call shutdown/reset handlers, they're skip due to the crash nature - so there is a risk of devices doing bad things in the new kernel. With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters. With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what. Let me know your thoughts Scott / Desmond / Petr and all interested parties. Cheers, Guilherme From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Message-ID: Date: Thu, 19 May 2022 09:19:31 -0300 MIME-Version: 1.0 Subject: Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list Content-Language: en-US References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-20-gpiccoli@igalia.com> <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> From: "Guilherme G. Piccoli" In-Reply-To: <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: Scott Branden , Petr Mladek , Sebastian Reichel , Florian Fainelli , Desmond yan Cc: David Gow , Evan Green , Julius Werner , bcm-kernel-feedback-list@broadcom.com, linux-pm@vger.kernel.org, akpm@linux-foundation.org, bhe@redhat.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-edac@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-leds@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, linux-tegra@vger.kernel.org, linux-um@lists.infradead.org, linux-xtensa@linux-xtensa.org, netdev@vger.kernel.org, openipmi-developer@lists.sourceforge.net, rcu@vger.kernel.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org, kernel-dev@igalia.com, kernel@gpiccoli.net, halves@canonical.com, fabiomirmar@gmail.com, alejandro.j.jimenez@oracle.com, andriy.shevchenko@linux.intel.com, arnd@arndb.de, bp@alien8.de, corbet@lwn.net, d.hatayama@jp.fujitsu.com, dave.hansen@linux.intel.com, dyoung@redhat.com, feng.tang@intel.com, gregkh@linuxfoundation.org, mikelley@microsoft.com, hidehiro.kawai.ez@hitachi.com, jgross@suse.com, john.ogness@linutronix.de, keescook@chromium.org, luto@kernel.org, mhiramat@kernel.org, mingo@redhat.com, paulmck@kernel.org, peterz@infradead.org, rostedt@goodmis.org, senozhatsky@chromium.org, stern@rowland.harvard.edu, tglx@linutronix.de, vgoyal@redhat.com, vkuznets@redhat.com, will@kernel.org, Alexander Gordeev , Andrea Parri , Ard Biesheuvel , Benjamin Herrenschmidt , Brian Norris , Christian Borntraeger , Christophe JAILLET , "David S. Miller" , Dexuan Cui , Doug Berger , Haiyang Zhang , Hari Bathini , Heiko Carstens , Justin Chen , "K. Y. Srinivasan" , Lee Jones , Markus Mayer , Michael Ellerman , Mihai Carabas , Nicholas Piggin , Paul Mackerras , Pavel Machek , Shile Zhang , Stephen Hemminger , Sven Schnelle , Thomas Bogendoerfer , Tianyu Lan , Vasily Gorbik , Wang ShaoBo , Wei Liu , zhenwei pi On 18/05/2022 19:17, Scott Branden wrote: > Hi Guilherme, > > +Desmond > [...] >>>> I'm afraid it breaks kdump if this device is not reset beforehand - it's >>>> a doorbell write, so not high risk I think... >>>> >>>> But in case the not-reset device can be probed normally in kdump kernel, >>>> then I'm fine in moving this to the reboot list! I don't have the HW to >>>> test myself. >>> >>> Good question. Well, it if has to be called before kdump then >>> even "hypervisor" list is a wrong place because is not always >>> called before kdump. >> [...] > We register to the panic notifier so that we can kill the VK card ASAP > to stop DMAing things over to the host side. If it is not notified then > memory may not be frozen when kdump is occurring. > Notifying the card on panic is also needed to allow for any type of > reset to occur. > > So, the only thing preventing moving the notifier later is the chance > that memory is modified while kdump is occurring. Or, if DMA is > disabled before kdump already then this wouldn't be an issue and the > notification to the card (to allow for clean resets) can be done later. Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases? I'm not expert on that, but I guess whether DMA is "kept" or not depends a bit if IOMMU is used. IIRC, there was a copy of the DMAR table in kdump (at least for Intel IOMMU). Also, devices are not properly quiesced on kdump IIUC, we don't call shutdown/reset handlers, they're skip due to the crash nature - so there is a risk of devices doing bad things in the new kernel. With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters. With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what. Let me know your thoughts Scott / Desmond / Petr and all interested parties. Cheers, Guilherme _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Guilherme G. Piccoli" Subject: Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list Date: Thu, 19 May 2022 09:19:31 -0300 Message-ID: References: <20220427224924.592546-1-gpiccoli@igalia.com> <20220427224924.592546-20-gpiccoli@igalia.com> <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From: References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=e6xrJm1a3ay+4weLnrCkh2SRSEeyZycIIB/twweqf9Y=; b=BhOEtl3IEpq+ZutVgH7P2rHsfA mhGkC2PSuDrVyQ8IQfXtbB2pfD2fLPqKTP1L2vPLew3hLPbkforpZGHUalxvr86M8T2G/zNh0P/pw wi++cJ5XXfisCo4sQPAYOpaVMhBvNWB3svhNAgbQKxYFPy0Oeko+Z9jhUOVMogS4VQkCRHPLo2bnq jjqxb69zYzPORFRcZZlayXhHSqDz7B+6f3XjEVdBQ2ftzK2PDz+TwQa11JztdT1Ouzc0j+U1m1POY rbklHHuyIwI+fwGIyQbgwTEldlCv1PEui/a5Qzds7nWr2ZDdu+bcGybfUaDSvUQ/88wOr+qjGKQ2s 3BFZ9Ur Content-Language: en-US In-Reply-To: <81878a67-21f1-fee8-1add-f381bc8b05df@broadcom.com> List-ID: Content-Type: text/plain; charset="us-ascii" To: Scott Branden , Petr Mladek , Sebastian Reichel , Florian Fainelli , Desmond yan Cc: David Gow , Evan Green , Julius Werner , bcm-kernel-feedback-list@broadcom.com, linux-pm@vger.kernel.org, akpm@linux-foundation.org, bhe@redhat.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-edac@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-leds@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, linux-tegra@vger.kernel.org, linux-um@lists.infradead.org, linux-xtensa@linux-xtensa.org, netdev@vger.kernel.org, openipmi-developer@lists.sourceforge.net, rcu@vger.kernel.org, sparclinux@vger.kernel.org, xen-devel@li On 18/05/2022 19:17, Scott Branden wrote: > Hi Guilherme, > > +Desmond > [...] >>>> I'm afraid it breaks kdump if this device is not reset beforehand - it's >>>> a doorbell write, so not high risk I think... >>>> >>>> But in case the not-reset device can be probed normally in kdump kernel, >>>> then I'm fine in moving this to the reboot list! I don't have the HW to >>>> test myself. >>> >>> Good question. Well, it if has to be called before kdump then >>> even "hypervisor" list is a wrong place because is not always >>> called before kdump. >> [...] > We register to the panic notifier so that we can kill the VK card ASAP > to stop DMAing things over to the host side. If it is not notified then > memory may not be frozen when kdump is occurring. > Notifying the card on panic is also needed to allow for any type of > reset to occur. > > So, the only thing preventing moving the notifier later is the chance > that memory is modified while kdump is occurring. Or, if DMA is > disabled before kdump already then this wouldn't be an issue and the > notification to the card (to allow for clean resets) can be done later. Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases? I'm not expert on that, but I guess whether DMA is "kept" or not depends a bit if IOMMU is used. IIRC, there was a copy of the DMAR table in kdump (at least for Intel IOMMU). Also, devices are not properly quiesced on kdump IIUC, we don't call shutdown/reset handlers, they're skip due to the crash nature - so there is a risk of devices doing bad things in the new kernel. With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters. With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what. Let me know your thoughts Scott / Desmond / Petr and all interested parties. Cheers, Guilherme