All of lore.kernel.org
 help / color / mirror / Atom feed
From: Doug Anderson <dianders@chromium.org>
To: Petr Mladek <pmladek@suse.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Sumit Garg <sumit.garg@linaro.org>,
	Mark Rutland <mark.rutland@arm.com>,
	Matthias Kaehlcke <mka@chromium.org>,
	Stephane Eranian <eranian@google.com>,
	Stephen Boyd <swboyd@chromium.org>,
	ricardo.neri@intel.com, Tzung-Bi Shih <tzungbi@chromium.org>,
	Lecopzer Chen <lecopzer.chen@mediatek.com>,
	kgdb-bugreport@lists.sourceforge.net,
	Masayoshi Mizuma <msys.mizuma@gmail.com>,
	Guenter Roeck <groeck@chromium.org>,
	Pingfan Liu <kernelfans@gmail.com>,
	Andi Kleen <ak@linux.intel.com>, Ian Rogers <irogers@google.com>,
	linux-arm-kernel@lists.infradead.org,
	linux-perf-users@vger.kernel.org, ito-yuichi@fujitsu.com,
	Randy Dunlap <rdunlap@infradead.org>,
	Chen-Yu Tsai <wens@csie.org>,
	christophe.leroy@csgroup.eu, davem@davemloft.net,
	sparclinux@vger.kernel.org, mpe@ellerman.id.au,
	Will Deacon <will@kernel.org>,
	ravi.v.shankar@intel.com, npiggin@gmail.com,
	linuxppc-dev@lists.ozlabs.org, Marc Zyngier <maz@kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Daniel Thompson <daniel.thompson@linaro.org>
Subject: Re: [PATCH v4 10/17] watchdog/hardlockup: Move perf hardlockup watchdog petting to watchdog.c
Date: Fri, 19 May 2023 10:22:21 -0700	[thread overview]
Message-ID: <CAD=FV=VXn07rgC=OUtHAz5v5h2g2gcrLCc5BYr6LMWFkyNKASw@mail.gmail.com> (raw)
In-Reply-To: <ZF0NzSCRCapqDbC4@alley>

Hi,

On Thu, May 11, 2023 at 8:46 AM Petr Mladek <pmladek@suse.com> wrote:
>
> > @@ -111,6 +125,11 @@ static void watchdog_hardlockup_interrupt_count(void)
> >
> >  void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
> >  {
> > +     if (__this_cpu_read(watchdog_hardlockup_touch)) {
> > +             __this_cpu_write(watchdog_hardlockup_touch, false);
> > +             return;
> > +     }
>
> If we clear watchdog_hardlockup_touch() here then
> watchdog_hardlockup_check() won't be called yet another
> watchdog_hrtimer_sample_threshold perior.
>
> It means that any touch will cause ignoring one full period.
> The is_hardlockup() check will be done after full two periods.
>
> It is not ideal, see below.
>
> > +
> >       /*
> >        * Check for a hardlockup by making sure the CPU's timer
> >        * interrupt is incrementing. The timer interrupt should have
> > diff --git a/kernel/watchdog_perf.c b/kernel/watchdog_perf.c
> > index 9be90b2a2ea7..547917ebd5d3 100644
> > --- a/kernel/watchdog_perf.c
> > +++ b/kernel/watchdog_perf.c
> > @@ -112,11 +98,6 @@ static void watchdog_overflow_callback(struct perf_event *event,
> >       /* Ensure the watchdog never gets throttled */
> >       event->hw.interrupts = 0;
> >
> > -     if (__this_cpu_read(watchdog_nmi_touch) == true) {
> > -             __this_cpu_write(watchdog_nmi_touch, false);
> > -             return;
> > -     }
>
> The original code looks wrong. arch_touch_nmi_watchdog() caused
> skipping only one period of the perf event.
>
> I would expect that it caused restarting the period,
> something like:
>
>         if (__this_cpu_read(watchdog_nmi_touch) == true) {
>                 /*
>                  * Restart the period after which the interrupt
>                  * counter is checked.
>                  */
>                 __this_cpu_write(nmi_rearmed, 0);
>                 __this_cpu_write(last_timestamp, now);
>                 __this_cpu_write(watchdog_nmi_touch, false);
>                 return;
>         }
>
> By other words, we should restart the period in the very next perf
> event after the watchdog was touched.
>
> That said, the new code looks better than the original.
> IMHO, the original code was prone to false positives.

I had a little bit of a hard time following, but I _think_ the "tl;dr"
of all the above is that my change is fine. If I misunderstood, please
yell.


> Best Regards,
> Petr
>
> PS: It might be worth fixing this problem in a separate patch at the
>     beginning of this patchset. It might be a candidate for stable
>     backports.

Done. It's now its own patch and early in the series.

WARNING: multiple messages have this Message-ID (diff)
From: Doug Anderson <dianders@chromium.org>
To: Petr Mladek <pmladek@suse.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Sumit Garg <sumit.garg@linaro.org>,
	 Mark Rutland <mark.rutland@arm.com>,
	Matthias Kaehlcke <mka@chromium.org>,
	 Stephane Eranian <eranian@google.com>,
	Stephen Boyd <swboyd@chromium.org>,
	ricardo.neri@intel.com,  Tzung-Bi Shih <tzungbi@chromium.org>,
	Lecopzer Chen <lecopzer.chen@mediatek.com>,
	 kgdb-bugreport@lists.sourceforge.net,
	 Masayoshi Mizuma <msys.mizuma@gmail.com>,
	Guenter Roeck <groeck@chromium.org>,
	 Pingfan Liu <kernelfans@gmail.com>,
	Andi Kleen <ak@linux.intel.com>,  Ian Rogers <irogers@google.com>,
	linux-arm-kernel@lists.infradead.org,
	 linux-perf-users@vger.kernel.org, ito-yuichi@fujitsu.com,
	 Randy Dunlap <rdunlap@infradead.org>,
	Chen-Yu Tsai <wens@csie.org>,
	christophe.leroy@csgroup.eu,  davem@davemloft.net,
	sparclinux@vger.kernel.org, mpe@ellerman.id.au,
	 Will Deacon <will@kernel.org>,
	ravi.v.shankar@intel.com, npiggin@gmail.com,
	 linuxppc-dev@lists.ozlabs.org, Marc Zyngier <maz@kernel.org>,
	 Catalin Marinas <catalin.marinas@arm.com>,
	Daniel Thompson <daniel.thompson@linaro.org>
Subject: Re: [PATCH v4 10/17] watchdog/hardlockup: Move perf hardlockup watchdog petting to watchdog.c
Date: Fri, 19 May 2023 10:22:21 -0700	[thread overview]
Message-ID: <CAD=FV=VXn07rgC=OUtHAz5v5h2g2gcrLCc5BYr6LMWFkyNKASw@mail.gmail.com> (raw)
In-Reply-To: <ZF0NzSCRCapqDbC4@alley>

Hi,

On Thu, May 11, 2023 at 8:46 AM Petr Mladek <pmladek@suse.com> wrote:
>
> > @@ -111,6 +125,11 @@ static void watchdog_hardlockup_interrupt_count(void)
> >
> >  void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
> >  {
> > +     if (__this_cpu_read(watchdog_hardlockup_touch)) {
> > +             __this_cpu_write(watchdog_hardlockup_touch, false);
> > +             return;
> > +     }
>
> If we clear watchdog_hardlockup_touch() here then
> watchdog_hardlockup_check() won't be called yet another
> watchdog_hrtimer_sample_threshold perior.
>
> It means that any touch will cause ignoring one full period.
> The is_hardlockup() check will be done after full two periods.
>
> It is not ideal, see below.
>
> > +
> >       /*
> >        * Check for a hardlockup by making sure the CPU's timer
> >        * interrupt is incrementing. The timer interrupt should have
> > diff --git a/kernel/watchdog_perf.c b/kernel/watchdog_perf.c
> > index 9be90b2a2ea7..547917ebd5d3 100644
> > --- a/kernel/watchdog_perf.c
> > +++ b/kernel/watchdog_perf.c
> > @@ -112,11 +98,6 @@ static void watchdog_overflow_callback(struct perf_event *event,
> >       /* Ensure the watchdog never gets throttled */
> >       event->hw.interrupts = 0;
> >
> > -     if (__this_cpu_read(watchdog_nmi_touch) == true) {
> > -             __this_cpu_write(watchdog_nmi_touch, false);
> > -             return;
> > -     }
>
> The original code looks wrong. arch_touch_nmi_watchdog() caused
> skipping only one period of the perf event.
>
> I would expect that it caused restarting the period,
> something like:
>
>         if (__this_cpu_read(watchdog_nmi_touch) == true) {
>                 /*
>                  * Restart the period after which the interrupt
>                  * counter is checked.
>                  */
>                 __this_cpu_write(nmi_rearmed, 0);
>                 __this_cpu_write(last_timestamp, now);
>                 __this_cpu_write(watchdog_nmi_touch, false);
>                 return;
>         }
>
> By other words, we should restart the period in the very next perf
> event after the watchdog was touched.
>
> That said, the new code looks better than the original.
> IMHO, the original code was prone to false positives.

I had a little bit of a hard time following, but I _think_ the "tl;dr"
of all the above is that my change is fine. If I misunderstood, please
yell.


> Best Regards,
> Petr
>
> PS: It might be worth fixing this problem in a separate patch at the
>     beginning of this patchset. It might be a candidate for stable
>     backports.

Done. It's now its own patch and early in the series.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

WARNING: multiple messages have this Message-ID (diff)
From: Doug Anderson <dianders@chromium.org>
To: Petr Mladek <pmladek@suse.com>
Cc: Mark Rutland <mark.rutland@arm.com>,
	Ian Rogers <irogers@google.com>,
	Randy Dunlap <rdunlap@infradead.org>,
	Lecopzer Chen <lecopzer.chen@mediatek.com>,
	kgdb-bugreport@lists.sourceforge.net, ricardo.neri@intel.com,
	Stephane Eranian <eranian@google.com>,
	sparclinux@vger.kernel.org, Guenter Roeck <groeck@chromium.org>,
	Will Deacon <will@kernel.org>,
	Daniel Thompson <daniel.thompson@linaro.org>,
	Andi Kleen <ak@linux.intel.com>, Chen-Yu Tsai <wens@csie.org>,
	Matthias Kaehlcke <mka@chromium.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Masayoshi Mizuma <msys.mizuma@gmail.com>,
	ravi.v.shankar@intel.com, Tzung-Bi Shih <tzungbi@chromium.org>,
	npiggin@gmail.com, Stephen Boyd <swboyd@chromium.org>,
	Pingfan Liu <kernelfans@gmail.com>,
	linux-arm-kernel@lists.infradead.org,
	Sumit Garg <sumit.garg@linaro.org>,
	ito-yuichi@fujitsu.com, linux-perf-users@vger.kernel.org,
	Marc Zyngier <maz@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	linuxppc-dev@lists.ozlabs.org, davem@davemloft.net
Subject: Re: [PATCH v4 10/17] watchdog/hardlockup: Move perf hardlockup watchdog petting to watchdog.c
Date: Fri, 19 May 2023 10:22:21 -0700	[thread overview]
Message-ID: <CAD=FV=VXn07rgC=OUtHAz5v5h2g2gcrLCc5BYr6LMWFkyNKASw@mail.gmail.com> (raw)
In-Reply-To: <ZF0NzSCRCapqDbC4@alley>

Hi,

On Thu, May 11, 2023 at 8:46 AM Petr Mladek <pmladek@suse.com> wrote:
>
> > @@ -111,6 +125,11 @@ static void watchdog_hardlockup_interrupt_count(void)
> >
> >  void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
> >  {
> > +     if (__this_cpu_read(watchdog_hardlockup_touch)) {
> > +             __this_cpu_write(watchdog_hardlockup_touch, false);
> > +             return;
> > +     }
>
> If we clear watchdog_hardlockup_touch() here then
> watchdog_hardlockup_check() won't be called yet another
> watchdog_hrtimer_sample_threshold perior.
>
> It means that any touch will cause ignoring one full period.
> The is_hardlockup() check will be done after full two periods.
>
> It is not ideal, see below.
>
> > +
> >       /*
> >        * Check for a hardlockup by making sure the CPU's timer
> >        * interrupt is incrementing. The timer interrupt should have
> > diff --git a/kernel/watchdog_perf.c b/kernel/watchdog_perf.c
> > index 9be90b2a2ea7..547917ebd5d3 100644
> > --- a/kernel/watchdog_perf.c
> > +++ b/kernel/watchdog_perf.c
> > @@ -112,11 +98,6 @@ static void watchdog_overflow_callback(struct perf_event *event,
> >       /* Ensure the watchdog never gets throttled */
> >       event->hw.interrupts = 0;
> >
> > -     if (__this_cpu_read(watchdog_nmi_touch) == true) {
> > -             __this_cpu_write(watchdog_nmi_touch, false);
> > -             return;
> > -     }
>
> The original code looks wrong. arch_touch_nmi_watchdog() caused
> skipping only one period of the perf event.
>
> I would expect that it caused restarting the period,
> something like:
>
>         if (__this_cpu_read(watchdog_nmi_touch) == true) {
>                 /*
>                  * Restart the period after which the interrupt
>                  * counter is checked.
>                  */
>                 __this_cpu_write(nmi_rearmed, 0);
>                 __this_cpu_write(last_timestamp, now);
>                 __this_cpu_write(watchdog_nmi_touch, false);
>                 return;
>         }
>
> By other words, we should restart the period in the very next perf
> event after the watchdog was touched.
>
> That said, the new code looks better than the original.
> IMHO, the original code was prone to false positives.

I had a little bit of a hard time following, but I _think_ the "tl;dr"
of all the above is that my change is fine. If I misunderstood, please
yell.


> Best Regards,
> Petr
>
> PS: It might be worth fixing this problem in a separate patch at the
>     beginning of this patchset. It might be a candidate for stable
>     backports.

Done. It's now its own patch and early in the series.

  reply	other threads:[~2023-05-19 17:24 UTC|newest]

Thread overview: 130+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-04 22:13 [PATCH v4 00/17] watchdog/hardlockup: Add the buddy hardlockup detector Douglas Anderson
2023-05-04 22:13 ` Douglas Anderson
2023-05-04 22:13 ` Douglas Anderson
2023-05-04 22:13 ` [PATCH v4 01/17] watchdog/perf: Define dummy watchdog_update_hrtimer_threshold() on correct config Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-05  2:43   ` Nicholas Piggin
2023-05-05  2:43     ` Nicholas Piggin
2023-05-05  2:43     ` Nicholas Piggin
2023-05-11  8:39     ` Petr Mladek
2023-05-11  8:39       ` Petr Mladek
2023-05-04 22:13 ` [PATCH v4 02/17] watchdog: remove WATCHDOG_DEFAULT Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13 ` [PATCH v4 03/17] watchdog/hardlockup: change watchdog_nmi_enable() to void Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-05  2:45   ` Nicholas Piggin
2023-05-05  2:45     ` Nicholas Piggin
2023-05-05  2:45     ` Nicholas Piggin
2023-05-04 22:13 ` [PATCH v4 04/17] watchdog/perf: Ensure CPU-bound context when creating hardlockup detector event Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13 ` [PATCH v4 05/17] watchdog/hardlockup: Rename touch_nmi_watchdog() to touch_hardlockup_watchdog() Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-05  2:51   ` Nicholas Piggin
2023-05-05  2:51     ` Nicholas Piggin
2023-05-05  2:51     ` Nicholas Piggin
2023-05-05 16:37     ` Doug Anderson
2023-05-05 16:37       ` Doug Anderson
2023-05-05 16:37       ` Doug Anderson
2023-05-08  1:34       ` Nicholas Piggin
2023-05-08  1:34         ` Nicholas Piggin
2023-05-08  1:34         ` Nicholas Piggin
2023-05-08 15:56         ` Doug Anderson
2023-05-08 15:56           ` Doug Anderson
2023-05-11  9:24       ` Petr Mladek
2023-05-11  9:24         ` Petr Mladek
2023-05-04 22:13 ` [PATCH v4 06/17] watchdog/perf: Rename watchdog_hld.c to watchdog_perf.c Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-05  2:53   ` Nicholas Piggin
2023-05-05  2:53     ` Nicholas Piggin
2023-05-05  2:53     ` Nicholas Piggin
2023-05-11 10:09   ` Petr Mladek
2023-05-11 10:09     ` Petr Mladek
2023-05-04 22:13 ` [PATCH v4 07/17] watchdog/hardlockup: Move perf hardlockup checking/panic to common watchdog.c Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-05  2:58   ` Nicholas Piggin
2023-05-05  2:58     ` Nicholas Piggin
2023-05-05  2:58     ` Nicholas Piggin
2023-05-05 16:37     ` Doug Anderson
2023-05-05 16:37       ` Doug Anderson
2023-05-05 16:37       ` Doug Anderson
2023-05-11 12:03       ` Petr Mladek
2023-05-11 12:03         ` Petr Mladek
2023-05-04 22:13 ` [PATCH v4 08/17] watchdog/hardlockup: Style changes to watchdog_hardlockup_check() / ..._is_lockedup() Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-05  3:01   ` Nicholas Piggin
2023-05-05  3:01     ` Nicholas Piggin
2023-05-05  3:01     ` Nicholas Piggin
2023-05-05 16:38     ` Doug Anderson
2023-05-05 16:38       ` Doug Anderson
2023-05-05 16:38       ` Doug Anderson
2023-05-11 12:45       ` Petr Mladek
2023-05-11 12:45         ` Petr Mladek
2023-05-04 22:13 ` [PATCH v4 09/17] watchdog/hardlockup: Add a "cpu" param to watchdog_hardlockup_check() Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-11 14:14   ` Petr Mladek
2023-05-11 14:14     ` Petr Mladek
2023-05-19 17:21     ` Doug Anderson
2023-05-19 17:21       ` Doug Anderson
2023-05-19 17:21       ` Doug Anderson
2023-05-04 22:13 ` [PATCH v4 10/17] watchdog/hardlockup: Move perf hardlockup watchdog petting to watchdog.c Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-11 15:46   ` Petr Mladek
2023-05-11 15:46     ` Petr Mladek
2023-05-19 17:22     ` Doug Anderson [this message]
2023-05-19 17:22       ` Doug Anderson
2023-05-19 17:22       ` Doug Anderson
2023-05-04 22:13 ` [PATCH v4 11/17] watchdog/hardlockup: Rename some "NMI watchdog" constants/function Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-05  3:06   ` Nicholas Piggin
2023-05-05  3:06     ` Nicholas Piggin
2023-05-05  3:06     ` Nicholas Piggin
2023-05-05 16:38     ` Doug Anderson
2023-05-05 16:38       ` Doug Anderson
2023-05-05 16:38       ` Doug Anderson
2023-05-12 11:21     ` Petr Mladek
2023-05-12 11:21       ` Petr Mladek
2023-05-04 22:13 ` [PATCH v4 12/17] watchdog/hardlockup: Have the perf hardlockup use __weak functions more cleanly Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-12 11:55   ` Petr Mladek
2023-05-12 11:55     ` Petr Mladek
2023-05-04 22:13 ` [PATCH v4 13/17] watchdog/hardlockup: detect hard lockups using secondary (buddy) CPUs Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-05  2:35   ` Nicholas Piggin
2023-05-05  2:35     ` Nicholas Piggin
2023-05-05  2:35     ` Nicholas Piggin
2023-05-05 16:35     ` Doug Anderson
2023-05-05 16:35       ` Doug Anderson
2023-05-05 16:35       ` Doug Anderson
2023-05-08  1:04       ` Nicholas Piggin
2023-05-08  1:04         ` Nicholas Piggin
2023-05-08  1:04         ` Nicholas Piggin
2023-05-08 15:52         ` Doug Anderson
2023-05-08 15:52           ` Doug Anderson
2023-05-19 17:23           ` Doug Anderson
2023-05-19 17:23             ` Doug Anderson
2023-05-19 17:23             ` Doug Anderson
2023-05-04 22:13 ` [PATCH v4 14/17] watchdog/perf: Add a weak function for an arch to detect if perf can use NMIs Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13 ` [PATCH v4 15/17] watchdog/perf: Adapt the watchdog_perf interface for async model Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13 ` [PATCH v4 16/17] arm64: add hw_nmi_get_sample_period for preparation of lockup detector Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13 ` [PATCH v4 17/17] arm64: Enable perf events based hard " Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson
2023-05-04 22:13   ` Douglas Anderson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAD=FV=VXn07rgC=OUtHAz5v5h2g2gcrLCc5BYr6LMWFkyNKASw@mail.gmail.com' \
    --to=dianders@chromium.org \
    --cc=ak@linux.intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=catalin.marinas@arm.com \
    --cc=christophe.leroy@csgroup.eu \
    --cc=daniel.thompson@linaro.org \
    --cc=davem@davemloft.net \
    --cc=eranian@google.com \
    --cc=groeck@chromium.org \
    --cc=irogers@google.com \
    --cc=ito-yuichi@fujitsu.com \
    --cc=kernelfans@gmail.com \
    --cc=kgdb-bugreport@lists.sourceforge.net \
    --cc=lecopzer.chen@mediatek.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-perf-users@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=mark.rutland@arm.com \
    --cc=maz@kernel.org \
    --cc=mka@chromium.org \
    --cc=mpe@ellerman.id.au \
    --cc=msys.mizuma@gmail.com \
    --cc=npiggin@gmail.com \
    --cc=pmladek@suse.com \
    --cc=ravi.v.shankar@intel.com \
    --cc=rdunlap@infradead.org \
    --cc=ricardo.neri@intel.com \
    --cc=sparclinux@vger.kernel.org \
    --cc=sumit.garg@linaro.org \
    --cc=swboyd@chromium.org \
    --cc=tzungbi@chromium.org \
    --cc=wens@csie.org \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.