From: "Guilherme G. Piccoli" <gpiccoli@igalia.com>
To: Petr Mladek <pmladek@suse.com>,
"michael Kelley (LINUX)" <mikelley@microsoft.com>,
Baoquan He <bhe@redhat.com>, Dave Young <dyoung@redhat.com>,
d.hatayama@jp.fujitsu.com
Cc: akpm@linux-foundation.org, kexec@lists.infradead.org,
linux-kernel@vger.kernel.org,
bcm-kernel-feedback-list@broadcom.com,
linuxppc-dev@lists.ozlabs.org, linux-alpha@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, linux-edac@vger.kernel.org,
linux-hyperv@vger.kernel.org, linux-leds@vger.kernel.org,
linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org,
linux-pm@vger.kernel.org, linux-remoteproc@vger.kernel.org,
linux-s390@vger.kernel.org, linux-tegra@vger.kernel.org,
linux-um@lists.infradead.org, linux-xtensa@linux-xtensa.org,
netdev@vger.kernel.org, openipmi-developer@lists.sourceforge.net,
rcu@vger.kernel.org, sparclinux@vger.kernel.org,
xen-devel@lists.xenproject.org, x86@kernel.org,
kernel-dev@igalia.com, kernel@gpiccoli.net, halves@canonical.com,
fabiomirmar@gmail.com, alejandro.j.jimenez@oracle.com,
andriy.shevchenko@linux.intel.com, arnd@arndb.de, bp@alien8.de,
corbet@lwn.net, dave.hansen@linux.intel.com, feng.tang@intel.com,
gregkh@linuxfoundation.org, hidehiro.kawai.ez@hitachi.com,
jgross@suse.com, john.ogness@linutronix.de,
keescook@chromium.org, luto@kernel.org, mhiramat@kernel.org,
mingo@redhat.com, paulmck@kernel.org, peterz@infradead.org,
rostedt@goodmis.org, senozhatsky@chromium.org,
stern@rowland.harvard.edu, tglx@linutronix.de, vgoyal@redhat.com,
vkuznets@redhat.com, will@kernel.org
Subject: Re: [PATCH 24/30] panic: Refactor the panic path
Date: Sun, 15 May 2022 19:47:39 -0300 [thread overview]
Message-ID: <d313eec2-96b6-04e3-35cd-981f103d010e@igalia.com> (raw)
In-Reply-To: <Yn0TnsWVxCcdB2yO@alley>
On 12/05/2022 11:03, Petr Mladek wrote:
> Hello,
>
> first, I am sorry for stepping into the discussion so late.
> I was busy with some other stuff and this patchset is far
> from trivial.
>
> Second, thanks a lot for putting so much effort into it.
> Most of the changes look pretty good, especially all
> the fixes of particular notifiers and split into
> four lists.
>
> Though this patch will need some more love. See below
> for more details.
Thanks a lot for your review Petr, it is much appreciated! No need for
apologies, there is no urgency here =)
> [...]
> This talks only about kdump. The reality is much more complicated.
> The level affect the order of:
>
> + notifiers vs. kdump
> + notifiers vs. crash_dump
> + crash_dump vs. kdump
First of all, I'd like to ask you please to clarify to me *exactly* what
are the differences between "crash_dump" and "kdump". I'm sorry if
that's a silly question, I need to be 100% sure I understand the
concepts the same way you do.
> There might theoretically many variants of the ordering of kdump,
> crash_dump, and the 4 notifier list. Some variants do not make
> much sense. You choose 5 variants and tried to select them by
> a level number.
>
> The question is if we really could easily describe the meaning this
> way. It is not only about a "level" of notifiers before kdump. It is
> also about the ordering of crash_dump vs. kdump. IMHO, "level"
> semantic does not fit there.
>
> Maybe more parameters might be easier to understand the effect.
> Anyway, we first need to agree on the chosen variants.
> I am going to discuss it more in the code, see below.
>
>
> [...]
> Here is the code using the above functions. It helps to discuss
> the design and logic.
>
> <kernel/panic.c>
> order_panic_notifiers_and_kdump();
>
> /* If no level, we should kdump ASAP. */
> if (!panic_notifiers_level)
> __crash_kexec(NULL);
>
> crash_smp_send_stop();
> panic_notifier_hypervisor_once(buf);
>
> if (panic_notifier_info_once(buf))
> kmsg_dump(KMSG_DUMP_PANIC);
>
> panic_notifier_pre_reboot_once(buf);
>
> __crash_kexec(NULL);
>
> panic_notifier_hypervisor_once(buf);
>
> if (panic_notifier_info_once(buf))
> kmsg_dump(KMSG_DUMP_PANIC);
>
> panic_notifier_pre_reboot_once(buf);
> </kernel/panic.c>
>
> I have to say that the logic is very unclear. Almost all
> functions are called twice:
>
> + __crash_kexec()
> + kmsg_dump()
> + panic_notifier_hypervisor_once()
> + panic_notifier_pre_reboot_once()
> + panic_notifier_info_once()
>
> It is pretty hard to find what functions are always called in the same
> order and where the order can be inverted.
>
> The really used code path is defined by order_panic_notifiers_and_kdump()
> that encodes "level" into "bits". The bits are then flipped in
> panic_notifier_*_once() calls that either do something or not.
> kmsg_dump() is called according to the bit flip.
>
> It is an interesting approach. I guess that you wanted to avoid too
> many if/then/else levels in panic(). But honestly, it looks like
> a black magic to me.
>
> IMHO, it is always easier to follow if/then/else logic than using
> a translation table that requires additional bit flips when
> a value is used more times.
>
> Also I guess that it is good proof that "level" abstraction does
> not fit here. Normal levels would not need this kind of magic.
Heheh OK, I appreciate your opinion, but I guess we'll need to agree in
disagree here - I'm much more fond to this kind of code than a bunch of
if/else blocks that almost give headaches. Encoding such "level" logic
in the if/else scheme is very convoluted, generates a very big code. And
the functions aren't so black magic - they map a level in bits, and the
functions _once() are called...once! Although we switch the position in
the code, so there are 2 calls, one of them is called and the other not.
But that's totally fine to change - especially if we're moving away from
the "level" logic. I see below you propose a much simpler approach - if
we follow that, definitely we won't need the "black magic" approach heheh
>
> OK, the question is how to make it better. Let's start with
> a clear picture of the problem:
>
> 1. panic() has basically two funtions:
>
> + show/store debug information (optional ways and amount)
> + do something with the system (reboot, stay hanged)
>
>
> 2. There are 4 ways how to show/store the information:
>
> + tell hypervisor to store what it is interested about
> + crash_dump
> + kmsg_dump()
> + consoles
>
> , where crash_dump and consoles are special:
>
> + crash_dump does not return. Instead it ends up with reboot.
>
> + Consoles work transparently. They just need an extra flush
> before reboot or staying hanged.
>
>
> 3. The various notifiers do things like:
>
> + tell hypervisor about the crash
> + print more information (also stop watchdogs)
> + prepare system for reboot (touch some interfaces)
> + prepare system for staying hanged (blinking)
>
> Note that it pretty nicely matches the 4 notifier lists.
>
I really appreciate the summary skill you have, to convert complex
problems in very clear and concise ideas. Thanks for that, very useful!
I agree with what was summarized above.
> Now, we need to decide about the ordering. The main area is how
> to store the debug information. Consoles are transparent so
> the quesition is about:
>
> + hypervisor
> + crash_dump
> + kmsg_dump
>
> Some people need none and some people want all. There is a
> risk that system might hung at any stage. This why people want to
> make the order configurable.
>
> But crash_dump() does not return when it succeeds. And kmsg_dump()
> users havn't complained about hypervisor problems yet. So, that
> two variants might be enough:
>
> + crash_dump (hypervisor, kmsg_dump as fallback)
> + hypervisor, kmsg_dump, crash_dump
>
> One option "panic_prefer_crash_dump" should be enough.
> And the code might look like:
>
> void panic()
> {
> [...]
> dump_stack();
> kgdb_panic(buf);
>
> < --- here starts the reworked code --- >
>
> /* crash dump is enough when enabled and preferred. */
> if (panic_prefer_crash_dump)
> __crash_kexec(NULL);
>
> /* Stop other CPUs and focus on handling the panic state. */
> if (has_kexec_crash_image)
> crash_smp_send_stop();
> else
> smp_send_stop()
>
Here we have a very important point. Why do we need 2 variants of SMP
CPU stopping functions? I disagree with that - my understanding of this
after some study in architectures is that the crash_() variant is
"stronger", should work in all cases and if not, we should fix that -
that'd be a bug.
Such variant either maps to smp_send_stop() (in various architectures,
including XEN/x86) or overrides the basic function with more proper
handling for panic() case...I don't see why we still need such
distinction, if you / others have some insight about that, I'd like to
hear =)
> /* Notify hypervisor about the system panic. */
> atomic_notifier_call_chain(&panic_hypervisor_list, 0, NULL);
>
> /*
> * No need to risk extra info when there is no kmsg dumper
> * registered.
> */
> if (!has_kmsg_dumper())
> __crash_kexec(NULL);
>
> /* Add extra info from different subsystems. */
> atomic_notifier_call_chain(&panic_info_list, 0, NULL);
>
> kmsg_dump(KMSG_DUMP_PANIC);
> __crash_kexec(NULL);
>
> /* Flush console */
> unblank_screen();
> console_unblank();
> debug_locks_off();
> console_flush_on_panic(CONSOLE_FLUSH_PENDING);
>
> if (panic_timeout > 0) {
> delay()
> }
>
> /*
> * Prepare system for eventual reboot and allow custom
> * reboot handling.
> */
> atomic_notifier_call_chain(&panic_reboot_list, 0, NULL);
You had the order of panic_reboot_list VS. consoles flushing inverted.
It might make sense, although I didn't do that in V1...
Are you OK in having a helper for console flushing, as I did in V1? It
makes code of panic() a bit less polluted / more focused I feel.
>
> if (panic_timeout != 0) {
> reboot();
> }
>
> /*
> * Prepare system for the infinite waiting, for example,
> * setup blinking.
> */
> atomic_notifier_call_chain(&panic_loop_list, 0, NULL);
>
> infinite_loop();
> }
>
>
> __crash_kexec() is there 3 times but otherwise the code looks
> quite straight forward.
>
> Note 1: I renamed the two last notifier list. The name 'post-reboot'
> did sound strange from the logical POV ;-)
>
> Note 2: We have to avoid the possibility to call "reboot" list
> before kmsg_dump(). All callbacks providing info
> have to be in the info list. It a callback combines
> info and reboot functionality then it should be split.
>
> There must be another way to calm down problematic
> info callbacks. And it has to be solved when such
> a problem is reported. Is there any known issue, please?
>
> It is possible that I have missed something important.
> But I would really like to make the logic as simple as possible.
OK, I agree with you! It's indeed simpler and if others agree, I can
happily change the logic to what you proposed. Although...currently the
"crash_kexec_post_notifiers" allows to call _all_ panic_reboot_list
callbacks _before kdump_.
We need to mention this change in the commit messages, but I really
would like to hear the opinions of heavy users of notifiers (as
Michael/Hyper-V) and the kdump interested parties (like Baoquan / Dave
Young / Hayatama). If we all agree on such approach, will change that
for V2 =)
Thanks again Petr, for the time spent in such detailed review!
Cheers,
Guilherme
next prev parent reply other threads:[~2022-05-15 22:48 UTC|newest]
Thread overview: 183+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-27 22:48 [PATCH 00/30] The panic notifiers refactor Guilherme G. Piccoli
2022-04-27 22:48 ` [PATCH 01/30] x86/crash,reboot: Avoid re-disabling VMX in all CPUs on crash/restart Guilherme G. Piccoli
2022-05-09 12:32 ` Guilherme G. Piccoli
2022-05-09 15:52 ` Sean Christopherson
2022-05-10 20:11 ` Guilherme G. Piccoli
2022-04-27 22:48 ` [PATCH 02/30] ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path Guilherme G. Piccoli
2022-04-29 16:26 ` Michael Kelley (LINUX)
2022-04-29 18:20 ` Marc Zyngier
2022-04-29 21:38 ` Guilherme G. Piccoli
2022-04-29 21:45 ` Russell King (Oracle)
2022-04-29 21:56 ` Guilherme G. Piccoli
2022-04-29 22:00 ` Marc Zyngier
2022-04-27 22:48 ` [PATCH 03/30] notifier: Add panic notifiers info and purge trailing whitespaces Guilherme G. Piccoli
2022-04-27 22:48 ` [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path Guilherme G. Piccoli
2022-05-03 18:03 ` Evan Green
2022-05-03 19:12 ` Guilherme G. Piccoli
2022-05-03 21:56 ` Evan Green
2022-05-04 12:45 ` Guilherme G. Piccoli
2022-05-10 11:38 ` Petr Mladek
2022-05-10 13:04 ` Guilherme G. Piccoli
2022-05-10 17:20 ` Steven Rostedt
2022-05-10 19:40 ` John Ogness
2022-05-11 11:13 ` Petr Mladek
2022-04-27 22:48 ` [PATCH 05/30] misc/pvpanic: " Guilherme G. Piccoli
2022-05-10 12:14 ` Petr Mladek
2022-05-10 13:00 ` Guilherme G. Piccoli
2022-05-17 10:58 ` Petr Mladek
2022-05-17 13:03 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 06/30] soc: bcm: brcmstb: Document panic notifier action and remove useless header Guilherme G. Piccoli
2022-05-02 15:38 ` Florian Fainelli
2022-05-02 15:47 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 07/30] mips: ip22: Reword PANICED to PANICKED " Guilherme G. Piccoli
2022-05-04 20:32 ` Thomas Bogendoerfer
2022-05-04 21:26 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 08/30] powerpc/setup: Refactor/untangle panic notifiers Guilherme G. Piccoli
2022-05-05 18:55 ` Hari Bathini
2022-05-05 19:28 ` Guilherme G. Piccoli
2022-05-09 12:50 ` Guilherme G. Piccoli
2022-05-10 13:53 ` Michael Ellerman
2022-05-10 14:10 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 09/30] coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier Guilherme G. Piccoli
2022-04-28 8:11 ` Suzuki K Poulose
2022-04-29 14:01 ` Guilherme G. Piccoli
2022-05-09 13:09 ` Guilherme G. Piccoli
2022-05-09 16:14 ` Suzuki K Poulose
2022-05-09 16:26 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 10/30] alpha: Clean-up the panic notifier code Guilherme G. Piccoli
2022-05-09 14:13 ` Guilherme G. Piccoli
2022-05-10 14:16 ` Petr Mladek
2022-05-11 20:10 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 11/30] um: Improve panic notifiers consistency and ordering Guilherme G. Piccoli
2022-04-28 8:30 ` Johannes Berg
2022-04-29 15:46 ` Guilherme G. Piccoli
2022-05-10 14:28 ` Petr Mladek
2022-05-11 20:22 ` Guilherme G. Piccoli
2022-05-13 14:44 ` Johannes Berg
2022-05-15 22:12 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 12/30] parisc: Replace regular spinlock with spin_trylock on panic path Guilherme G. Piccoli
2022-04-28 16:55 ` Helge Deller
2022-04-29 14:34 ` Guilherme G. Piccoli
2022-05-23 20:40 ` Guilherme G. Piccoli
2022-05-23 21:31 ` Helge Deller
2022-05-23 21:55 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 13/30] s390/consoles: Improve panic notifiers reliability Guilherme G. Piccoli
2022-04-29 18:46 ` Heiko Carstens
2022-04-29 19:31 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 14/30] panic: Properly identify the panic event to the notifiers' callbacks Guilherme G. Piccoli
2022-05-10 15:16 ` Petr Mladek
2022-05-10 16:16 ` Guilherme G. Piccoli
2022-05-17 13:11 ` Petr Mladek
2022-05-17 15:19 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 15/30] bus: brcmstb_gisb: Clean-up panic/die notifiers Guilherme G. Piccoli
2022-05-02 15:38 ` Florian Fainelli
2022-05-02 15:50 ` Guilherme G. Piccoli
2022-05-10 15:28 ` Petr Mladek
2022-05-17 15:32 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 16/30] drivers/hv/vmbus, video/hyperv_fb: Untangle and refactor Hyper-V panic notifiers Guilherme G. Piccoli
2022-04-29 17:16 ` Michael Kelley (LINUX)
2022-04-29 22:35 ` Guilherme G. Piccoli
2022-05-03 18:13 ` Michael Kelley (LINUX)
2022-05-03 18:57 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 17/30] tracing: Improve panic/die notifiers Guilherme G. Piccoli
2022-04-29 9:22 ` Sergei Shtylyov
2022-04-29 13:23 ` Steven Rostedt
2022-04-29 13:46 ` Guilherme G. Piccoli
2022-04-29 13:56 ` Steven Rostedt
2022-04-29 14:44 ` Guilherme G. Piccoli
2022-05-11 11:45 ` Petr Mladek
2022-05-17 15:33 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 18/30] notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set Guilherme G. Piccoli
2022-04-28 1:01 ` Xiaoming Ni
2022-04-29 19:38 ` Guilherme G. Piccoli
2022-05-10 17:29 ` Steven Rostedt
2022-05-16 16:14 ` Guilherme G. Piccoli
2022-04-29 16:27 ` Michael Kelley (LINUX)
2022-04-27 22:49 ` [PATCH 19/30] panic: Add the panic hypervisor notifier list Guilherme G. Piccoli
2022-04-29 17:30 ` Michael Kelley (LINUX)
2022-04-29 18:04 ` Guilherme G. Piccoli
2022-05-03 17:44 ` Michael Kelley (LINUX)
2022-05-03 17:56 ` Guilherme G. Piccoli
2022-05-16 14:01 ` Petr Mladek
2022-05-16 15:06 ` Guilherme G. Piccoli
2022-05-16 16:02 ` Evan Green
2022-05-17 13:28 ` Petr Mladek
2022-05-17 16:37 ` Guilherme G. Piccoli
2022-05-18 7:33 ` Petr Mladek
2022-05-18 13:24 ` Guilherme G. Piccoli
2022-05-17 13:57 ` Petr Mladek
2022-05-17 16:42 ` Guilherme G. Piccoli
2022-05-18 7:38 ` Petr Mladek
2022-05-18 13:09 ` Guilherme G. Piccoli
2022-05-18 22:17 ` Scott Branden
2022-05-19 12:19 ` Guilherme G. Piccoli
2022-05-19 19:20 ` Scott Branden
2022-05-23 14:56 ` Guilherme G. Piccoli
2022-05-24 8:04 ` Petr Mladek
2022-05-18 7:58 ` Petr Mladek
2022-05-18 13:16 ` Guilherme G. Piccoli
2022-05-19 7:03 ` Petr Mladek
2022-05-19 12:07 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 20/30] panic: Add the panic informational " Guilherme G. Piccoli
2022-04-27 23:49 ` Paul E. McKenney
2022-04-28 8:14 ` Suzuki K Poulose
2022-04-29 14:50 ` Guilherme G. Piccoli
2022-05-16 14:11 ` Petr Mladek
2022-05-16 14:28 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 21/30] panic: Introduce the panic pre-reboot " Guilherme G. Piccoli
2022-04-28 14:13 ` Alex Elder
2022-04-28 16:26 ` Corey Minyard
2022-04-29 15:18 ` Guilherme G. Piccoli
2022-04-29 16:04 ` Max Filippov
2022-04-29 19:34 ` Guilherme G. Piccoli
2022-05-16 14:33 ` Petr Mladek
2022-05-16 16:05 ` Guilherme G. Piccoli
2022-05-16 16:18 ` Luck, Tony
2022-05-16 16:33 ` Guilherme G. Piccoli
2022-05-17 14:11 ` Petr Mladek
2022-05-17 16:45 ` Guilherme G. Piccoli
2022-05-17 17:02 ` Luck, Tony
2022-05-17 18:12 ` Guilherme G. Piccoli
2022-05-17 19:07 ` Luck, Tony
2022-04-27 22:49 ` [PATCH 22/30] panic: Introduce the panic post-reboot " Guilherme G. Piccoli
2022-05-09 14:16 ` Guilherme G. Piccoli
2022-05-11 16:45 ` Heiko Carstens
2022-05-11 19:58 ` Guilherme G. Piccoli
2022-05-16 14:45 ` Petr Mladek
2022-05-16 16:08 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 23/30] printk: kmsg_dump: Introduce helper to inform number of dumpers Guilherme G. Piccoli
2022-05-10 17:40 ` Steven Rostedt
2022-05-11 20:03 ` Guilherme G. Piccoli
2022-05-16 14:50 ` Petr Mladek
2022-05-16 16:09 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 24/30] panic: Refactor the panic path Guilherme G. Piccoli
2022-04-28 0:28 ` Randy Dunlap
2022-04-29 16:04 ` Guilherme G. Piccoli
2022-05-09 14:25 ` Guilherme G. Piccoli
2022-04-29 17:53 ` Michael Kelley (LINUX)
2022-04-29 20:38 ` Guilherme G. Piccoli
2022-05-03 17:31 ` Michael Kelley (LINUX)
2022-05-03 18:06 ` Guilherme G. Piccoli
2022-05-09 15:16 ` d.hatayama
2022-05-09 16:39 ` Guilherme G. Piccoli
2022-05-12 14:03 ` Petr Mladek
2022-05-15 22:47 ` Guilherme G. Piccoli [this message]
2022-05-16 10:21 ` Petr Mladek
2022-05-16 16:32 ` Guilherme G. Piccoli
2022-05-19 23:45 ` Baoquan He
2022-05-20 11:23 ` Guilherme G. Piccoli
2022-05-24 8:01 ` Petr Mladek
2022-05-24 10:18 ` Baoquan He
2022-05-24 8:32 ` Baoquan He
2022-05-24 14:44 ` Eric W. Biederman
2022-05-26 16:25 ` Guilherme G. Piccoli
2022-06-14 14:36 ` Petr Mladek
2022-06-15 9:36 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 25/30] panic, printk: Add console flush parameter and convert panic_print to a notifier Guilherme G. Piccoli
2022-05-16 14:56 ` Petr Mladek
2022-05-16 16:11 ` Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 26/30] Drivers: hv: Do not force all panic notifiers to execute before kdump Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 27/30] powerpc: " Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 28/30] panic: Unexport crash_kexec_post_notifiers Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 29/30] powerpc: ps3, pseries: Avoid duplicate call to kmsg_dump() on panic Guilherme G. Piccoli
2022-04-27 22:49 ` [PATCH 30/30] um: Avoid duplicate call to kmsg_dump() Guilherme G. Piccoli
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d313eec2-96b6-04e3-35cd-981f103d010e@igalia.com \
--to=gpiccoli@igalia.com \
--cc=akpm@linux-foundation.org \
--cc=alejandro.j.jimenez@oracle.com \
--cc=andriy.shevchenko@linux.intel.com \
--cc=arnd@arndb.de \
--cc=bcm-kernel-feedback-list@broadcom.com \
--cc=bhe@redhat.com \
--cc=bp@alien8.de \
--cc=corbet@lwn.net \
--cc=d.hatayama@jp.fujitsu.com \
--cc=dave.hansen@linux.intel.com \
--cc=dyoung@redhat.com \
--cc=fabiomirmar@gmail.com \
--cc=feng.tang@intel.com \
--cc=gregkh@linuxfoundation.org \
--cc=halves@canonical.com \
--cc=hidehiro.kawai.ez@hitachi.com \
--cc=jgross@suse.com \
--cc=john.ogness@linutronix.de \
--cc=keescook@chromium.org \
--cc=kernel-dev@igalia.com \
--cc=kernel@gpiccoli.net \
--cc=kexec@lists.infradead.org \
--cc=linux-alpha@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-edac@vger.kernel.org \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-leds@vger.kernel.org \
--cc=linux-mips@vger.kernel.org \
--cc=linux-parisc@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=linux-remoteproc@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=linux-um@lists.infradead.org \
--cc=linux-xtensa@linux-xtensa.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=luto@kernel.org \
--cc=mhiramat@kernel.org \
--cc=mikelley@microsoft.com \
--cc=mingo@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=openipmi-developer@lists.sourceforge.net \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=senozhatsky@chromium.org \
--cc=sparclinux@vger.kernel.org \
--cc=stern@rowland.harvard.edu \
--cc=tglx@linutronix.de \
--cc=vgoyal@redhat.com \
--cc=vkuznets@redhat.com \
--cc=will@kernel.org \
--cc=x86@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).