linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Petr Mladek <pmladek@suse.com>
To: "Guilherme G. Piccoli" <gpiccoli@igalia.com>
Cc: Evan Green <evgreen@chromium.org>,
	David Gow <davidgow@google.com>,
	Julius Werner <jwerner@chromium.org>,
	Scott Branden <scott.branden@broadcom.com>,
	bcm-kernel-feedback-list@broadcom.com,
	Sebastian Reichel <sre@kernel.org>,
	Linux PM <linux-pm@vger.kernel.org>,
	Florian Fainelli <f.fainelli@gmail.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	bhe@redhat.com, kexec@lists.infradead.org,
	LKML <linux-kernel@vger.kernel.org>,
	linuxppc-dev@lists.ozlabs.org, linux-alpha@vger.kernel.org,
	linux-arm Mailing List <linux-arm-kernel@lists.infradead.org>,
	linux-edac@vger.kernel.org, linux-hyperv@vger.kernel.org,
	linux-leds@vger.kernel.org, linux-mips@vger.kernel.org,
	linux-parisc@vger.kernel.org, linux-remoteproc@vger.kernel.org,
	linux-s390@vger.kernel.org, linux-tegra@vger.kernel.org,
	linux-um@lists.infradead.org, linux-xtensa@linux-xtensa.org,
	netdev@vger.kernel.org, openipmi-developer@lists.sourceforge.net,
	rcu@vger.kernel.org, sparclinux@vger.kernel.org,
	xen-devel@lists.xenproject.org, x86@kernel.org,
	kernel-dev@igalia.com, kernel@gpiccoli.net, halves@canonical.com,
	fabiomirmar@gmail.com, alejandro.j.jimenez@oracle.com,
	Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
	Arnd Bergmann <arnd@arndb.de>, Borislav Petkov <bp@alien8.de>,
	Jonathan Corbet <corbet@lwn.net>,
	d.hatayama@jp.fujitsu.com, dave.hansen@linux.intel.com,
	dyoung@redhat.com, feng.tang@intel.com,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	mikelley@microsoft.com, hidehiro.kawai.ez@hitachi.com,
	jgross@suse.com, john.ogness@linutronix.de,
	Kees Cook <keescook@chromium.org>,
	luto@kernel.org, mhiramat@kernel.org, mingo@redhat.com,
	paulmck@kernel.org, peterz@infradead.org, rostedt@goodmis.org,
	senozhatsky@chromium.org, Alan Stern <stern@rowland.harvard.edu>,
	Thomas Gleixner <tglx@linutronix.de>,
	vgoyal@redhat.com, vkuznets@redhat.com,
	Will Deacon <will@kernel.org>,
	Alexander Gordeev <agordeev@linux.ibm.com>,
	Andrea Parri <parri.andrea@gmail.com>,
	Ard Biesheuvel <ardb@kernel.org>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>,
	Brian Norris <computersforpeace@gmail.com>,
	Christian Borntraeger <borntraeger@linux.ibm.com>,
	Christophe JAILLET <christophe.jaillet@wanadoo.fr>,
	"David S. Miller" <davem@davemloft.net>,
	Dexuan Cui <decui@microsoft.com>, Doug Berger <opendmb@gmail.com>,
	Haiyang Zhang <haiyangz@microsoft.com>,
	Hari Bathini <hbathini@linux.ibm.com>,
	Heiko Carstens <hca@linux.ibm.com>,
	Justin Chen <justinpopo6@gmail.com>,
	"K. Y. Srinivasan" <kys@microsoft.com>,
	Lee Jones <lee.jones@linaro.org>,
	Markus Mayer <mmayer@broadcom.com>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Mihai Carabas <mihai.carabas@oracle.com>,
	Nicholas Piggin <npiggin@gmail.com>,
	Paul Mackerras <paulus@samba.org>, Pavel Machek <pavel@ucw.cz>,
	Shile Zhang <shile.zhang@linux.alibaba.com>,
	Stephen Hemminger <sthemmin@microsoft.com>,
	Sven Schnelle <svens@linux.ibm.com>,
	Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
	Tianyu Lan <Tianyu.Lan@microsoft.com>,
	Vasily Gorbik <gor@linux.ibm.com>,
	Wang ShaoBo <bobo.shaobowang@huawei.com>,
	Wei Liu <wei.liu@kernel.org>,
	zhenwei pi <pizhenwei@bytedance.com>,
	Stephen Boyd <swboyd@chromium.org>
Subject: Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Date: Wed, 18 May 2022 09:33:57 +0200	[thread overview]
Message-ID: <YoShZVYNAdvvjb7z@alley> (raw)
In-Reply-To: <b9ec2fc8-216f-f261-8417-77b6dd95e25c@igalia.com>

On Tue 2022-05-17 13:37:58, Guilherme G. Piccoli wrote:
> On 17/05/2022 10:28, Petr Mladek wrote:
> > [...]
> >>> Disagree here. I'm looping Google maintainers, so they can comment.
> >>> (CCed Evan, David, Julius)
> >>>
> >>> This notifier is clearly a hypervisor notification mechanism. I've fixed
> >>> a locking stuff there (in previous patch), I feel it's low-risk but even
> >>> if it's mid-risk, the class of such callback remains a perfect fit with
> >>> the hypervisor list IMHO.
> >>
> >> This logs a panic to our "eventlog", a tiny logging area in SPI flash
> >> for critical and power-related events. In some cases this ends up
 > >> being the only clue we get in a Chromebook feedback report that a
> >> panic occurred, so from my perspective moving it to the front of the
> >> line seems like a good idea.
> > 
> > IMHO, this would really better fit into the pre-reboot notifier list:
> > 
> >    + the callback stores the log so it is similar to kmsg_dump()
> >      or console_flush_on_panic()
> > 
> >    + the callback should be proceed after "info" notifiers
> >      that might add some other useful information.
> > 
> > Honestly, I am not sure what exactly hypervisor callbacks do. But I
> > think that they do not try to extract the kernel log because they
> > would need to handle the internal format.
> > 
> 
> I guess the main point in your response is : "I am not sure what exactly
> hypervisor callbacks do". We need to be sure about the semantics of such
> list, and agree on that.
> 
> So, my opinion about this first list, that we call "hypervisor list",
> is: it contains callbacks that
> 
> (1) should run early, preferably before kdump (or even if kdump isn't
> set, should run ASAP);
> 
> (2) these callbacks perform some communication with an abstraction that
> runs "below" the kernel, like a firmware or hypervisor. Classic example:
> pvpanic, that communicates with VMM (usually qemu) and allow such VMM to
> snapshot the full guest memory, for example.
> 
> (3) Should be low-risk. What defines risk is the level of reliability of
> subsequent operations - if the callback have 50% of chance of "bricking"
> the system totally and prevent kdump / kmsg_dump() / reboot , this is
> high risk one for example.
> 
> Some good fits IMO: pvpanic, sstate_panic_event() [sparc], fadump in
> powerpc, etc.
> 
> So, this is a good case for the Google notifier as well - it's not
> collecting data like the dmesg (hence your second bullet seems to not
> apply here, info notifiers won't add info to be collected by gsmi). It
> is a firmware/hypervisor/whatever-gsmi-is notification mechanism, that
> tells such "lower" abstraction a panic occurred. It seems low risk and
> we want it to run ASAP, if possible.

" 
> >> This logs a panic to our "eventlog", a tiny logging area in SPI flash
> >> for critical and power-related events. In some cases this ends up

I see. I somehow assumed that it was about the kernel log because
Evans wrote:

  "This logs a panic to our "eventlog", a tiny logging area in SPI flash
   for critical and power-related events. In some cases this ends up"


Anyway, I would distinguish it the following way.

  + If the notifier is preserving kernel log then it should be ideally
    treated as kmsg_dump().

  + It the notifier is saving another debugging data then it better
    fits into the "hypervisor" notifier list.


Regarding the reliability. From my POV, any panic notifier enabled
in a generic kernel should be reliable with more than 99,9%.
Otherwise, they should not be in the notifier list at all.

An exception would be a platform-specific notifier that is
called only on some specific platform and developers maintaining
this platform agree on this.

The value "99,9%" is arbitrary. I am not sure if it is realistic
even in the other code, for example, console_flush_on_panic()
or emergency_restart(). I just want to point out that the border
should be rather high. Otherwise we would back in the situation
where people would want to disable particular notifiers.

Best Regards,
Petr

  reply	other threads:[~2022-05-18  7:34 UTC|newest]

Thread overview: 183+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-04-27 22:48 [PATCH 00/30] The panic notifiers refactor Guilherme G. Piccoli
2022-04-27 22:48 ` [PATCH 01/30] x86/crash,reboot: Avoid re-disabling VMX in all CPUs on crash/restart Guilherme G. Piccoli
2022-05-09 12:32   ` Guilherme G. Piccoli
2022-05-09 15:52   ` Sean Christopherson
2022-05-10 20:11     ` Guilherme G. Piccoli
2022-04-27 22:48 ` [PATCH 02/30] ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path Guilherme G. Piccoli
2022-04-29 16:26   ` Michael Kelley (LINUX)
2022-04-29 18:20   ` Marc Zyngier
2022-04-29 21:38     ` Guilherme G. Piccoli
2022-04-29 21:45       ` Russell King (Oracle)
2022-04-29 21:56         ` Guilherme G. Piccoli
2022-04-29 22:00         ` Marc Zyngier
2022-04-27 22:48 ` [PATCH 03/30] notifier: Add panic notifiers info and purge trailing whitespaces Guilherme G. Piccoli
2022-04-27 22:48 ` [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path Guilherme G. Piccoli
2022-05-03 18:03   ` Evan Green
2022-05-03 19:12     ` Guilherme G. Piccoli
2022-05-03 21:56       ` Evan Green
2022-05-04 12:45         ` Guilherme G. Piccoli
2022-05-10 11:38       ` Petr Mladek
2022-05-10 13:04         ` Guilherme G. Piccoli
2022-05-10 17:20         ` Steven Rostedt
2022-05-10 19:40           ` John Ogness
2022-05-11 11:13             ` Petr Mladek
2022-04-27 22:48 ` [PATCH 05/30] misc/pvpanic: " Guilherme G. Piccoli
2022-05-10 12:14   ` Petr Mladek
2022-05-10 13:00     ` Guilherme G. Piccoli
2022-05-17 10:58       ` Petr Mladek
2022-05-17 13:03         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 06/30] soc: bcm: brcmstb: Document panic notifier action and remove useless header Guilherme G. Piccoli
2022-05-02 15:38   ` Florian Fainelli
2022-05-02 15:47     ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 07/30] mips: ip22: Reword PANICED to PANICKED " Guilherme G. Piccoli
2022-05-04 20:32   ` Thomas Bogendoerfer
2022-05-04 21:26     ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 08/30] powerpc/setup: Refactor/untangle panic notifiers Guilherme G. Piccoli
2022-05-05 18:55   ` Hari Bathini
2022-05-05 19:28     ` Guilherme G. Piccoli
2022-05-09 12:50     ` Guilherme G. Piccoli
2022-05-10 13:53       ` Michael Ellerman
2022-05-10 14:10         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 09/30] coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier Guilherme G. Piccoli
2022-04-28  8:11   ` Suzuki K Poulose
2022-04-29 14:01     ` Guilherme G. Piccoli
2022-05-09 13:09     ` Guilherme G. Piccoli
2022-05-09 16:14       ` Suzuki K Poulose
2022-05-09 16:26         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 10/30] alpha: Clean-up the panic notifier code Guilherme G. Piccoli
2022-05-09 14:13   ` Guilherme G. Piccoli
2022-05-10 14:16     ` Petr Mladek
2022-05-11 20:10       ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 11/30] um: Improve panic notifiers consistency and ordering Guilherme G. Piccoli
2022-04-28  8:30   ` Johannes Berg
2022-04-29 15:46     ` Guilherme G. Piccoli
2022-05-10 14:28   ` Petr Mladek
2022-05-11 20:22     ` Guilherme G. Piccoli
2022-05-13 14:44       ` Johannes Berg
2022-05-15 22:12         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 12/30] parisc: Replace regular spinlock with spin_trylock on panic path Guilherme G. Piccoli
2022-04-28 16:55   ` Helge Deller
2022-04-29 14:34     ` Guilherme G. Piccoli
2022-05-23 20:40     ` Guilherme G. Piccoli
2022-05-23 21:31       ` Helge Deller
2022-05-23 21:55         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 13/30] s390/consoles: Improve panic notifiers reliability Guilherme G. Piccoli
2022-04-29 18:46   ` Heiko Carstens
2022-04-29 19:31     ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 14/30] panic: Properly identify the panic event to the notifiers' callbacks Guilherme G. Piccoli
2022-05-10 15:16   ` Petr Mladek
2022-05-10 16:16     ` Guilherme G. Piccoli
2022-05-17 13:11       ` Petr Mladek
2022-05-17 15:19         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 15/30] bus: brcmstb_gisb: Clean-up panic/die notifiers Guilherme G. Piccoli
2022-05-02 15:38   ` Florian Fainelli
2022-05-02 15:50     ` Guilherme G. Piccoli
2022-05-10 15:28   ` Petr Mladek
2022-05-17 15:32     ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 16/30] drivers/hv/vmbus, video/hyperv_fb: Untangle and refactor Hyper-V panic notifiers Guilherme G. Piccoli
2022-04-29 17:16   ` Michael Kelley (LINUX)
2022-04-29 22:35     ` Guilherme G. Piccoli
2022-05-03 18:13       ` Michael Kelley (LINUX)
2022-05-03 18:57         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 17/30] tracing: Improve panic/die notifiers Guilherme G. Piccoli
2022-04-29  9:22   ` Sergei Shtylyov
2022-04-29 13:23     ` Steven Rostedt
2022-04-29 13:46       ` Guilherme G. Piccoli
2022-04-29 13:56         ` Steven Rostedt
2022-04-29 14:44           ` Guilherme G. Piccoli
2022-05-11 11:45   ` Petr Mladek
2022-05-17 15:33     ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 18/30] notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set Guilherme G. Piccoli
2022-04-28  1:01   ` Xiaoming Ni
2022-04-29 19:38     ` Guilherme G. Piccoli
2022-05-10 17:29     ` Steven Rostedt
2022-05-16 16:14       ` Guilherme G. Piccoli
2022-04-29 16:27   ` Michael Kelley (LINUX)
2022-04-27 22:49 ` [PATCH 19/30] panic: Add the panic hypervisor notifier list Guilherme G. Piccoli
2022-04-29 17:30   ` Michael Kelley (LINUX)
2022-04-29 18:04     ` Guilherme G. Piccoli
2022-05-03 17:44       ` Michael Kelley (LINUX)
2022-05-03 17:56         ` Guilherme G. Piccoli
2022-05-16 14:01   ` Petr Mladek
2022-05-16 15:06     ` Guilherme G. Piccoli
2022-05-16 16:02       ` Evan Green
2022-05-17 13:28         ` Petr Mladek
2022-05-17 16:37           ` Guilherme G. Piccoli
2022-05-18  7:33             ` Petr Mladek [this message]
2022-05-18 13:24               ` Guilherme G. Piccoli
2022-05-17 13:57       ` Petr Mladek
2022-05-17 16:42         ` Guilherme G. Piccoli
2022-05-18  7:38           ` Petr Mladek
2022-05-18 13:09             ` Guilherme G. Piccoli
2022-05-18 22:17           ` Scott Branden
2022-05-19 12:19             ` Guilherme G. Piccoli
2022-05-19 19:20               ` Scott Branden
2022-05-23 14:56                 ` Guilherme G. Piccoli
2022-05-24  8:04                   ` Petr Mladek
2022-05-18  7:58         ` Petr Mladek
2022-05-18 13:16           ` Guilherme G. Piccoli
2022-05-19  7:03             ` Petr Mladek
2022-05-19 12:07               ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 20/30] panic: Add the panic informational " Guilherme G. Piccoli
2022-04-27 23:49   ` Paul E. McKenney
2022-04-28  8:14   ` Suzuki K Poulose
2022-04-29 14:50     ` Guilherme G. Piccoli
2022-05-16 14:11   ` Petr Mladek
2022-05-16 14:28     ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 21/30] panic: Introduce the panic pre-reboot " Guilherme G. Piccoli
2022-04-28 14:13   ` Alex Elder
2022-04-28 16:26   ` Corey Minyard
2022-04-29 15:18     ` Guilherme G. Piccoli
2022-04-29 16:04   ` Max Filippov
2022-04-29 19:34     ` Guilherme G. Piccoli
2022-05-16 14:33   ` Petr Mladek
2022-05-16 16:05     ` Guilherme G. Piccoli
2022-05-16 16:18       ` Luck, Tony
2022-05-16 16:33         ` Guilherme G. Piccoli
2022-05-17 14:11           ` Petr Mladek
2022-05-17 16:45             ` Guilherme G. Piccoli
2022-05-17 17:02               ` Luck, Tony
2022-05-17 18:12                 ` Guilherme G. Piccoli
2022-05-17 19:07                   ` Luck, Tony
2022-04-27 22:49 ` [PATCH 22/30] panic: Introduce the panic post-reboot " Guilherme G. Piccoli
2022-05-09 14:16   ` Guilherme G. Piccoli
2022-05-11 16:45     ` Heiko Carstens
2022-05-11 19:58       ` Guilherme G. Piccoli
2022-05-16 14:45   ` Petr Mladek
2022-05-16 16:08     ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 23/30] printk: kmsg_dump: Introduce helper to inform number of dumpers Guilherme G. Piccoli
2022-05-10 17:40   ` Steven Rostedt
2022-05-11 20:03     ` Guilherme G. Piccoli
2022-05-16 14:50       ` Petr Mladek
2022-05-16 16:09         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 24/30] panic: Refactor the panic path Guilherme G. Piccoli
2022-04-28  0:28   ` Randy Dunlap
2022-04-29 16:04     ` Guilherme G. Piccoli
2022-05-09 14:25       ` Guilherme G. Piccoli
2022-04-29 17:53   ` Michael Kelley (LINUX)
2022-04-29 20:38     ` Guilherme G. Piccoli
2022-05-03 17:31       ` Michael Kelley (LINUX)
2022-05-03 18:06         ` Guilherme G. Piccoli
2022-05-09 15:16   ` d.hatayama
2022-05-09 16:39     ` Guilherme G. Piccoli
2022-05-12 14:03   ` Petr Mladek
2022-05-15 22:47     ` Guilherme G. Piccoli
2022-05-16 10:21       ` Petr Mladek
2022-05-16 16:32         ` Guilherme G. Piccoli
2022-05-19 23:45       ` Baoquan He
2022-05-20 11:23         ` Guilherme G. Piccoli
2022-05-24  8:01           ` Petr Mladek
2022-05-24 10:18             ` Baoquan He
2022-05-24  8:32           ` Baoquan He
2022-05-24 14:44   ` Eric W. Biederman
2022-05-26 16:25     ` Guilherme G. Piccoli
2022-06-14 14:36       ` Petr Mladek
2022-06-15  9:36         ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 25/30] panic, printk: Add console flush parameter and convert panic_print to a notifier Guilherme G. Piccoli
2022-05-16 14:56   ` Petr Mladek
2022-05-16 16:11     ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 26/30] Drivers: hv: Do not force all panic notifiers to execute before kdump Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 27/30] powerpc: " Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 28/30] panic: Unexport crash_kexec_post_notifiers Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 29/30] powerpc: ps3, pseries: Avoid duplicate call to kmsg_dump() on panic Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 30/30] um: Avoid duplicate call to kmsg_dump() Guilherme G. Piccoli

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YoShZVYNAdvvjb7z@alley \
    --to=pmladek@suse.com \
    --cc=Tianyu.Lan@microsoft.com \
    --cc=agordeev@linux.ibm.com \
    --cc=akpm@linux-foundation.org \
    --cc=alejandro.j.jimenez@oracle.com \
    --cc=andriy.shevchenko@linux.intel.com \
    --cc=ardb@kernel.org \
    --cc=arnd@arndb.de \
    --cc=bcm-kernel-feedback-list@broadcom.com \
    --cc=benh@kernel.crashing.org \
    --cc=bhe@redhat.com \
    --cc=bobo.shaobowang@huawei.com \
    --cc=borntraeger@linux.ibm.com \
    --cc=bp@alien8.de \
    --cc=christophe.jaillet@wanadoo.fr \
    --cc=computersforpeace@gmail.com \
    --cc=corbet@lwn.net \
    --cc=d.hatayama@jp.fujitsu.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=davem@davemloft.net \
    --cc=davidgow@google.com \
    --cc=decui@microsoft.com \
    --cc=dyoung@redhat.com \
    --cc=evgreen@chromium.org \
    --cc=f.fainelli@gmail.com \
    --cc=fabiomirmar@gmail.com \
    --cc=feng.tang@intel.com \
    --cc=gor@linux.ibm.com \
    --cc=gpiccoli@igalia.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=haiyangz@microsoft.com \
    --cc=halves@canonical.com \
    --cc=hbathini@linux.ibm.com \
    --cc=hca@linux.ibm.com \
    --cc=hidehiro.kawai.ez@hitachi.com \
    --cc=jgross@suse.com \
    --cc=john.ogness@linutronix.de \
    --cc=justinpopo6@gmail.com \
    --cc=jwerner@chromium.org \
    --cc=keescook@chromium.org \
    --cc=kernel-dev@igalia.com \
    --cc=kernel@gpiccoli.net \
    --cc=kexec@lists.infradead.org \
    --cc=kys@microsoft.com \
    --cc=lee.jones@linaro.org \
    --cc=linux-alpha@vger.kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-edac@vger.kernel.org \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-leds@vger.kernel.org \
    --cc=linux-mips@vger.kernel.org \
    --cc=linux-parisc@vger.kernel.org \
    --cc=linux-pm@vger.kernel.org \
    --cc=linux-remoteproc@vger.kernel.org \
    --cc=linux-s390@vger.kernel.org \
    --cc=linux-tegra@vger.kernel.org \
    --cc=linux-um@lists.infradead.org \
    --cc=linux-xtensa@linux-xtensa.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=luto@kernel.org \
    --cc=mhiramat@kernel.org \
    --cc=mihai.carabas@oracle.com \
    --cc=mikelley@microsoft.com \
    --cc=mingo@redhat.com \
    --cc=mmayer@broadcom.com \
    --cc=mpe@ellerman.id.au \
    --cc=netdev@vger.kernel.org \
    --cc=npiggin@gmail.com \
    --cc=opendmb@gmail.com \
    --cc=openipmi-developer@lists.sourceforge.net \
    --cc=parri.andrea@gmail.com \
    --cc=paulmck@kernel.org \
    --cc=paulus@samba.org \
    --cc=pavel@ucw.cz \
    --cc=peterz@infradead.org \
    --cc=pizhenwei@bytedance.com \
    --cc=rcu@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=scott.branden@broadcom.com \
    --cc=senozhatsky@chromium.org \
    --cc=shile.zhang@linux.alibaba.com \
    --cc=sparclinux@vger.kernel.org \
    --cc=sre@kernel.org \
    --cc=stern@rowland.harvard.edu \
    --cc=sthemmin@microsoft.com \
    --cc=svens@linux.ibm.com \
    --cc=swboyd@chromium.org \
    --cc=tglx@linutronix.de \
    --cc=tsbogend@alpha.franken.de \
    --cc=vgoyal@redhat.com \
    --cc=vkuznets@redhat.com \
    --cc=wei.liu@kernel.org \
    --cc=will@kernel.org \
    --cc=x86@kernel.org \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).