From: Ingo Molnar <mingo@kernel.org>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
Arnaldo Carvalho de Melo <acme@infradead.org>,
Jiri Olsa <jolsa@redhat.com>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Mark Rutland <mark.rutland@arm.com>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [GIT PULL] perf fixes
Date: Fri, 1 Nov 2019 21:30:48 +0100 [thread overview]
Message-ID: <20191101203048.GA6622@gmail.com> (raw)
In-Reply-To: <CAHk-=wi_VHc=Q2JsPbVmCgpKekNJwnbBiYrmvnSSW8aiAkg7nQ@mail.gmail.com>
* Linus Torvalds <torvalds@linux-foundation.org> wrote:
> On Fri, Nov 1, 2019 at 10:48 AM Ingo Molnar <mingo@kernel.org> wrote:
> >
> > Alexander Shishkin (1):
> > perf/core: Start rejecting the syscall with attr.__reserved_2 set
>
> This seems to quite possibly break existing apps. Is there any reason
> to believe that existing users have actually cleared that field?
Yes, there's good reason to believe that the field ends up being zero
within the kernel even if old user-space uses the old, smaller structure:
> It's suspect for another reason too: the commit that added that field
> just added it to the end of the structure, with the argument that
> "aux_watermark will only matter for new AUX-aware code, so the old
> code should still be fine".
>
> So by *definition* those old kinds of users would never have cleared
> that field, because that field didn't exist.
Firstly, non-existent fields are initialized to zero by default in the
perf ABI: *even if user-space was built well before that new field was
introduced in the kernel*.
This is done in perf_copy_attr():
/* Zero the full structure, so that a short copy will be nice. */
memset(attr, 0, sizeof(*attr));
The user-space structure that is passed in ('uattr' within that function)
might indeed be short and not contain the new field - but we handle this
via uattr->size, which is set by user-space - for example 'perf' sets it
in event_attr_init() in tools/perf/util/util.c:
/* to capture ABI version */
attr->size = sizeof(*attr);
Second, the kernel syscall then checks this size against the kernel's
size of attr:
- if uattr->size < kattr_size: then old ABI user-space binary is
running on new kernel, and we zero out residual fields.
- if uattr->size = kattr_size, this is the normal case of matching ABI (sizes)
- if uattr->size > kattr_size, then new ABI user-space binary is running
on an old kernel - in this case the kernel copies in a shortened
structure from the new tool and ignores the remaining fields. To not
surprise new user-space the kernel copies back the recognized size to
uattr->size and user-space can decide whether they want to run on this
kernel or not:
put_user(sizeof(*attr), &uattr->size);
This way the perf ABI is *both* backwards and forward compatible, even
across multiple ABI changes. We seemingly didn't "care" about the zeroing
of the new field in old binaries in that changelog, because it's
guaranteed by the syscall implementation.
[ Note that there *is* still potential for more subtle ABI breakage, see
further below. ]
This actually works fine in practice. I just tried out a brand new kernel
with really old perf tooling in the v3.19 kernel, from 4.5 years ago:
$ git checkout v3.19
$ cd tools/perf/
$ make install WERROR=0
$ perf version
perf version 3.19.gbfa76d49
$ perf top
This ancient version of 'perf top', 'perf record', 'perf report' works
just fine on the new kernel, despite passing in a smaller 'attr':
[ 558.408907] perf kABI size: 112
[ 558.412855] perf uABI size: 104
when running new tooling on a new kernel the two sizes match:
[ 331.598089] perf kABI size: 112
[ 331.602050] perf uABI size: 112
and everything works as usual.
> And the people involved should stop claiming this "fixes" anything,
> and should look hard at their random ABI expansions and "fixes".
>
> The original code that said "old users would not be impacted" is
> correct. But this "fix" is very very very questionable indeed, and I
> get the feeling that somebody doesn't understand what ABI is, when
> they claim that this "fixes" anything.
So if your primary worry is about old user-space that didn't know about
the new field passing in the new field with some random value, then if
the perf ABI handling logic I outlined above is implemented correctly
(and I believe it's correct), then this should not happen.
But arguably there's still a more subtle ABI breakage that this change
might trigger: if due to some bug user-space *does* initialize this
reserved field to non-zero, and the kernel ignored that non-zero field
for 4 years and new kernels suddenly start rejecting these syscalls,
that's an ABI bug and grounds for an immediate revert.
I don't expect this to happen though, due to the combination of how
user-space and kernel-space handles the extension of the perf ABI - but
if it happens then it's an insta-revert.
I'd also argue that because we didn't check this field for years, the
next ABI extension should probably not use the __reserved_2 field - which
is a good idea anyway as it has an unnatural __u16 size.
> Honestly, this all shows a worrying complete disregard for backwards
> compatibility. Calling this a "fix" is questionable, when it is much
> more likely to break some old user.
>
> I've pulled it, but I need people to be aware that this is utter
> garbage, and that if anybody ever reports it, this needs to be
> immediately reverted.
In light of the above logic I'd like to push back against this
characterization: this kind of ABI compatibility is actually one of the
most important aspects of the perf ABI, and we very much take it
seriously.
In fact we are proud of it: the perf ABI is one of the most iterated, yet
also one of the most compatible and tooling-friendly ABIs in the kernel. :-)
The reason it wasn't even mentioned in the changelog is because we tried
to design out this kind of mishap from the ABI, and all the main perf
developers are aware of that design. Will try to include a more careful
explanation next time.
Thanks,
Ingo
next prev parent reply other threads:[~2019-11-01 20:30 UTC|newest]
Thread overview: 376+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-01 17:48 [GIT PULL] perf fixes Ingo Molnar
2019-11-01 18:48 ` Linus Torvalds
2019-11-01 20:30 ` Ingo Molnar [this message]
2019-11-01 21:01 ` Ingo Molnar
2019-11-01 22:15 ` Linus Torvalds
2019-11-01 19:10 ` pr-tracker-bot
-- strict thread matches above, loose matches on Subject: below --
2023-01-06 11:57 Ingo Molnar
2023-01-06 21:19 ` pr-tracker-bot
2022-10-02 10:56 Ingo Molnar
2022-10-02 16:47 ` Linus Torvalds
2022-10-03 10:55 ` Ingo Molnar
2022-10-02 17:20 ` pr-tracker-bot
2022-08-28 14:35 Ingo Molnar
2022-08-28 18:18 ` pr-tracker-bot
2022-08-06 19:10 Ingo Molnar
2022-08-07 0:50 ` pr-tracker-bot
2021-07-11 13:26 Ingo Molnar
2021-07-11 18:22 ` pr-tracker-bot
2021-06-12 12:48 Ingo Molnar
2021-06-12 19:09 ` pr-tracker-bot
2021-03-21 10:56 Ingo Molnar
2021-03-21 18:45 ` pr-tracker-bot
2020-08-15 11:21 Ingo Molnar
2020-08-16 1:55 ` pr-tracker-bot
2020-04-25 9:19 Ingo Molnar
2020-04-25 19:30 ` pr-tracker-bot
2020-03-24 9:19 Ingo Molnar
2020-03-24 17:15 ` pr-tracker-bot
2020-03-02 7:23 Ingo Molnar
2020-03-03 23:35 ` pr-tracker-bot
2020-02-15 8:53 Ingo Molnar
2020-02-15 21:25 ` pr-tracker-bot
2020-01-18 17:58 Ingo Molnar
2020-01-18 21:05 ` pr-tracker-bot
2019-12-21 16:16 Ingo Molnar
2019-12-21 18:55 ` pr-tracker-bot
2019-12-17 11:34 Ingo Molnar
2019-12-17 19:06 ` Linus Torvalds
2019-12-18 6:58 ` Ingo Molnar
2019-12-17 19:20 ` pr-tracker-bot
2019-12-01 22:15 Ingo Molnar
2019-12-02 4:40 ` pr-tracker-bot
2019-11-16 21:33 Ingo Molnar
2019-11-17 0:35 ` pr-tracker-bot
2019-10-12 13:31 Ingo Molnar
2019-10-12 22:35 ` pr-tracker-bot
2019-07-14 12:01 Ingo Molnar
2019-07-14 18:45 ` pr-tracker-bot
2019-06-29 8:54 Ingo Molnar
2019-06-29 11:45 ` pr-tracker-bot
2019-06-02 17:39 Ingo Molnar
2019-06-02 18:15 ` pr-tracker-bot
2019-05-16 16:05 Ingo Molnar
2019-05-16 18:20 ` pr-tracker-bot
2019-05-05 12:47 Ingo Molnar
2019-05-05 22:10 ` pr-tracker-bot
2019-04-20 7:43 Ingo Molnar
2019-04-20 19:25 ` pr-tracker-bot
2019-04-12 13:06 Ingo Molnar
2019-04-13 4:05 ` pr-tracker-bot
2019-02-17 10:10 Ingo Molnar
2019-02-17 16:50 ` pr-tracker-bot
2019-02-10 9:01 Ingo Molnar
2019-02-10 18:30 ` pr-tracker-bot
2019-01-11 7:44 Ingo Molnar
2019-01-11 18:00 ` pr-tracker-bot
2018-11-30 6:25 Ingo Molnar
2018-11-30 21:00 ` pr-tracker-bot
2018-11-17 10:55 Ingo Molnar
2018-11-18 20:05 ` pr-tracker-bot
2018-10-20 8:10 Ingo Molnar
2018-10-20 13:28 ` Greg Kroah-Hartman
2018-10-11 9:12 Ingo Molnar
2018-10-11 12:32 ` Greg Kroah-Hartman
2018-10-11 8:59 Ingo Molnar
2018-10-05 9:42 Ingo Molnar
2018-10-05 9:55 ` Ingo Molnar
2018-10-05 23:30 ` Greg Kroah-Hartman
2018-09-15 13:11 Ingo Molnar
2018-07-30 17:53 Ingo Molnar
2018-07-13 19:59 Ingo Molnar
2018-06-30 8:44 Ingo Molnar
2018-06-04 9:04 Ingo Molnar
2018-03-31 10:40 Ingo Molnar
2018-03-25 8:53 Ingo Molnar
2018-02-06 21:29 Ingo Molnar
2017-12-06 22:17 Ingo Molnar
2017-11-26 12:40 Ingo Molnar
2017-11-05 14:40 Ingo Molnar
2017-11-09 8:13 ` Markus Trippelsdorf
2017-10-14 16:04 Ingo Molnar
2017-09-13 18:00 Ingo Molnar
2017-09-12 15:32 Ingo Molnar
2017-07-21 10:15 Ingo Molnar
2017-06-10 8:39 Ingo Molnar
2017-05-12 7:31 Ingo Molnar
2017-03-07 20:30 Ingo Molnar
2017-02-28 8:01 Ingo Molnar
2017-02-11 18:12 Ingo Molnar
2017-02-02 21:01 Ingo Molnar
2017-01-18 9:27 Ingo Molnar
2017-01-15 9:59 Ingo Molnar
2016-12-23 22:50 Ingo Molnar
2016-12-07 18:45 Ingo Molnar
2016-11-23 9:00 Ingo Molnar
2016-11-14 7:56 Ingo Molnar
2016-10-28 19:41 Ingo Molnar
2016-10-18 11:07 Ingo Molnar
2016-09-13 18:14 Ingo Molnar
2016-08-18 20:38 Ingo Molnar
2016-08-12 19:35 Ingo Molnar
2016-07-26 14:13 Ingo Molnar
2016-07-08 13:42 Ingo Molnar
2016-06-10 12:50 Ingo Molnar
2016-05-13 18:51 Ingo Molnar
2016-05-06 11:26 Ingo Molnar
2016-04-28 17:56 Ingo Molnar
2016-04-03 11:00 Ingo Molnar
2016-02-20 11:14 Ingo Molnar
2016-01-14 10:03 Ingo Molnar
2016-01-08 12:46 Ingo Molnar
2015-12-08 4:22 Ingo Molnar
2015-09-17 8:02 Ingo Molnar
2015-09-02 18:11 Ingo Molnar
2015-08-22 12:19 Ingo Molnar
2015-08-14 7:12 Ingo Molnar
2015-07-18 2:53 Ingo Molnar
2015-07-06 15:29 Ingo Molnar
2015-06-13 14:29 Ingo Molnar
2015-06-05 8:37 Ingo Molnar
2015-05-15 7:17 Ingo Molnar
2015-05-06 12:52 Ingo Molnar
2015-04-03 13:11 Ingo Molnar
2015-03-17 16:48 Ingo Molnar
2015-03-01 17:00 Ingo Molnar
2015-02-20 13:40 Ingo Molnar
2015-01-30 18:44 Ingo Molnar
2015-01-17 14:15 Ingo Molnar
2015-01-11 8:42 Ingo Molnar
2014-11-20 7:46 Ingo Molnar
2014-10-31 11:13 Ingo Molnar
2014-11-03 9:02 ` Paul Bolle
2014-11-03 10:04 ` Peter Zijlstra
2014-09-19 10:46 Ingo Molnar
2014-08-24 20:25 Ingo Molnar
2014-07-16 11:11 Ingo Molnar
2014-06-18 16:39 Ingo Molnar
2014-05-22 8:04 Ingo Molnar
2014-05-01 6:34 Ingo Molnar
2014-04-20 8:02 Ingo Molnar
2014-04-19 10:41 Ingo Molnar
2014-04-16 13:04 Ingo Molnar
2014-03-22 9:06 Ingo Molnar
2014-03-16 16:34 Ingo Molnar
2014-03-02 8:51 Ingo Molnar
2014-02-22 19:16 Ingo Molnar
2014-02-09 8:01 Ingo Molnar
2014-01-25 7:30 Ingo Molnar
2014-01-19 12:08 Ingo Molnar
2013-12-19 16:52 Ingo Molnar
2013-12-02 14:39 Ingo Molnar
2013-11-15 19:46 Ingo Molnar
2013-11-01 9:56 Ingo Molnar
2013-10-29 10:04 Ingo Molnar
2013-10-26 12:24 Ingo Molnar
2013-10-28 8:28 ` Markus Trippelsdorf
2013-10-28 9:02 ` ------------------------------ Markus Trippelsdorf
2013-10-28 9:34 ` Markus Trippelsdorf
2013-10-28 12:34 ` Arnaldo Carvalho de Melo
2013-10-28 12:42 ` Arnaldo Carvalho de Melo
2013-10-28 12:59 ` Markus Trippelsdorf
2013-10-29 9:50 ` Stephane Eranian
2013-10-29 10:06 ` Ingo Molnar
2013-10-29 12:47 ` Arnaldo Carvalho de Melo
2013-10-08 12:12 Ingo Molnar
2013-09-28 18:03 Ingo Molnar
2013-09-29 11:47 ` Markus Trippelsdorf
2013-09-29 21:33 ` Andi Kleen
2013-09-29 22:47 ` Markus Trippelsdorf
2013-09-30 6:27 ` Ingo Molnar
2013-09-30 18:54 ` Andi Kleen
2013-10-01 8:42 ` Ingo Molnar
2013-09-25 18:00 Ingo Molnar
2013-09-18 16:06 Ingo Molnar
2013-09-12 13:38 Ingo Molnar
2013-09-12 18:03 ` Linus Torvalds
2013-09-12 18:10 ` Linus Torvalds
2013-09-12 18:43 ` Arnaldo Carvalho de Melo
2013-09-12 19:12 ` Arnaldo Carvalho de Melo
2013-09-12 19:13 ` Linus Torvalds
2013-09-12 19:55 ` Ingo Molnar
2013-09-12 19:58 ` David Ahern
2013-09-12 20:02 ` Arnaldo Carvalho de Melo
2013-09-12 20:31 ` Ingo Molnar
2013-09-12 20:43 ` Ingo Molnar
2013-09-12 20:18 ` Ingo Molnar
2013-09-12 20:38 ` Arnaldo Carvalho de Melo
2013-09-12 20:46 ` Ingo Molnar
2013-09-12 21:09 ` David Ahern
2013-09-12 21:18 ` Ingo Molnar
2013-09-12 22:10 ` David Ahern
2013-09-13 5:09 ` Ingo Molnar
2013-09-13 9:32 ` Jean Pihet
2013-09-13 9:45 ` Ingo Molnar
2013-09-13 17:15 ` Jean Pihet
2013-09-12 18:51 ` Linus Torvalds
2013-09-12 20:33 ` Ingo Molnar
2013-09-12 20:38 ` Linus Torvalds
2013-09-12 20:49 ` Ingo Molnar
2013-09-12 20:52 ` Linus Torvalds
2013-09-12 21:01 ` Ingo Molnar
2013-09-12 20:10 ` Ingo Molnar
2013-08-13 16:51 Ingo Molnar
2013-07-10 8:52 Ingo Molnar
2013-06-26 8:52 Ingo Molnar
2013-06-20 8:58 Ingo Molnar
2013-05-05 10:10 Ingo Molnar
2013-04-21 8:16 Ingo Molnar
2013-04-14 15:20 Ingo Molnar
2013-03-21 9:56 Ingo Molnar
2013-03-11 14:28 Ingo Molnar
2013-02-26 7:02 Ingo Molnar
2013-03-14 20:32 ` Linus Torvalds
2013-03-14 21:06 ` Linus Torvalds
2013-03-14 22:09 ` Stephane Eranian
2013-03-14 22:17 ` Linus Torvalds
2013-03-14 22:19 ` Stephane Eranian
2013-03-14 22:42 ` Stephane Eranian
2013-03-14 22:53 ` Stephane Eranian
2013-03-14 23:11 ` Stephane Eranian
2013-03-15 0:24 ` Stephane Eranian
2013-03-15 1:06 ` Linus Torvalds
2013-03-15 8:01 ` Stephane Eranian
2013-03-15 10:50 ` Stephane Eranian
2013-02-04 18:20 Ingo Molnar
2012-12-01 11:11 Ingo Molnar
2012-10-26 14:44 Ingo Molnar
2012-10-23 11:02 Ingo Molnar
2012-10-20 0:56 Ingo Molnar
2012-09-21 19:08 Ingo Molnar
2012-09-13 14:39 Ingo Molnar
2012-08-23 10:59 Ingo Molnar
2012-08-20 9:08 Ingo Molnar
2012-08-21 7:59 ` Ingo Molnar
2012-08-05 17:43 Ingo Molnar
2012-08-03 16:40 Ingo Molnar
2012-07-14 7:51 Ingo Molnar
2012-06-22 13:36 Ingo Molnar
2012-06-22 18:07 ` Linus Torvalds
2012-06-22 18:38 ` Hagen Paul Pfeifer
2012-06-22 18:52 ` Linus Torvalds
2012-06-22 19:06 ` Hagen Paul Pfeifer
2012-06-22 19:54 ` Steven Rostedt
[not found] ` <86448d73-2e19-416f-8104-ce72aa5d76eb@email.android.com>
2012-06-22 23:18 ` Steven Rostedt
2012-06-23 0:51 ` Arjan van de Ven
2012-06-23 1:57 ` Steven Rostedt
2012-06-23 18:25 ` H. Peter Anvin
2012-06-22 18:50 ` Steven Rostedt
2012-06-15 18:48 Ingo Molnar
2012-06-08 9:20 Ingo Molnar
2012-05-30 15:39 Ingo Molnar
2012-05-17 8:19 Ingo Molnar
2012-04-27 6:32 Ingo Molnar
2012-04-16 17:48 Ingo Molnar
2012-04-14 10:54 Ingo Molnar
2012-04-03 22:40 Ingo Molnar
2012-03-13 16:56 Ingo Molnar
2012-03-05 9:27 Ingo Molnar
2012-03-02 10:47 Ingo Molnar
2012-02-10 12:45 Ingo Molnar
2012-02-02 10:00 Ingo Molnar
2012-01-26 18:11 Ingo Molnar
2011-12-29 21:02 Ingo Molnar
2011-12-09 6:16 Ingo Molnar
2011-12-05 19:13 Ingo Molnar
2011-11-07 18:49 Ingo Molnar
2011-11-07 19:00 ` Linus Torvalds
2011-11-07 19:50 ` Ingo Molnar
2011-08-22 17:00 Ingo Molnar
2011-08-11 8:17 Ingo Molnar
2011-07-07 18:11 Ingo Molnar
2011-06-19 8:44 Ingo Molnar
2011-06-13 9:53 Ingo Molnar
2011-06-08 13:46 Ingo Molnar
2011-05-28 16:34 Ingo Molnar
2011-05-24 2:41 Ingo Molnar
2011-05-23 13:41 Ingo Molnar
2011-05-23 22:10 ` Eric Dumazet
2011-05-23 22:19 ` Frederic Weisbecker
2011-05-23 22:22 ` Eric Dumazet
2011-05-20 17:18 Ingo Molnar
2011-05-17 22:07 Ingo Molnar
2011-05-07 18:20 Ingo Molnar
2011-04-29 18:17 Ingo Molnar
2011-04-22 13:42 Ingo Molnar
2011-04-19 15:56 Ingo Molnar
2011-04-16 10:03 Ingo Molnar
2011-04-07 17:48 Ingo Molnar
2011-04-02 10:25 Ingo Molnar
2011-03-25 13:11 Ingo Molnar
2011-03-10 7:53 Ingo Molnar
2011-02-28 17:34 Ingo Molnar
2011-02-22 16:03 Ingo Molnar
2011-02-15 16:58 Ingo Molnar
2011-02-06 11:27 Ingo Molnar
2011-02-03 15:47 Ingo Molnar
2011-01-24 13:34 Ingo Molnar
2011-01-24 19:48 ` Linus Torvalds
2011-01-24 20:07 ` Ingo Molnar
2011-01-24 20:11 ` Ingo Molnar
2011-01-24 20:17 ` Ingo Molnar
2011-01-24 20:17 ` Linus Torvalds
2011-01-24 20:27 ` Linus Torvalds
2011-01-24 20:38 ` Arnaldo Carvalho de Melo
2011-01-24 21:13 ` Linus Torvalds
2011-01-24 21:25 ` Ingo Molnar
2011-01-24 22:00 ` Arnaldo Carvalho de Melo
2011-01-25 0:16 ` Ingo Molnar
2011-01-24 20:37 ` Davidlohr Bueso
2011-01-24 20:14 ` Arnaldo Carvalho de Melo
2011-01-18 18:59 Ingo Molnar
2011-01-18 9:42 Ingo Molnar
2011-01-15 15:24 Ingo Molnar
2011-01-11 11:32 Ingo Molnar
2011-01-03 19:04 Ingo Molnar
2010-12-23 12:56 Ingo Molnar
2010-12-19 15:34 Ingo Molnar
2010-12-08 7:55 Ingo Molnar
2010-11-28 17:36 Ingo Molnar
2010-11-26 13:20 Ingo Molnar
2010-11-11 10:38 Ingo Molnar
2010-10-30 18:21 Ingo Molnar
2010-10-13 15:21 Ingo Molnar
2010-10-05 14:49 Ingo Molnar
2010-09-26 8:38 Ingo Molnar
2010-09-21 19:39 Ingo Molnar
2010-09-10 14:26 Ingo Molnar
2010-08-25 17:44 Ingo Molnar
2010-08-24 19:06 Ingo Molnar
2010-08-19 14:55 Ingo Molnar
2010-08-18 8:14 Ingo Molnar
2010-07-23 19:41 Ingo Molnar
2010-07-16 17:30 Ingo Molnar
2010-07-08 4:36 Frederic Weisbecker
2010-07-08 4:40 ` Frederic Weisbecker
2010-07-08 6:36 ` Ingo Molnar
2010-07-04 20:24 Ingo Molnar
2010-06-10 10:25 Ingo Molnar
2010-06-02 12:28 Ingo Molnar
2010-05-31 23:02 Frederic Weisbecker
2010-06-01 6:59 ` Ingo Molnar
2010-05-30 19:27 Ingo Molnar
2010-05-20 9:44 Frederic Weisbecker
2010-05-20 12:40 ` Ingo Molnar
2010-05-11 19:18 Ingo Molnar
2010-05-12 0:39 ` Linus Torvalds
2010-04-06 17:49 Ingo Molnar
2010-04-04 10:15 Ingo Molnar
2010-04-03 10:47 Frederic Weisbecker
2010-03-28 5:11 Frederic Weisbecker
2010-03-29 3:33 ` Ingo Molnar
2010-03-26 15:16 Ingo Molnar
2010-03-16 16:06 Ingo Molnar
2010-03-11 19:12 Ingo Molnar
2010-02-22 16:50 Ingo Molnar
2010-02-22 17:01 ` Linus Torvalds
2010-02-22 17:15 ` Frederic Weisbecker
2010-02-14 9:06 Ingo Molnar
2010-01-31 17:32 Ingo Molnar
2010-01-21 15:38 Ingo Molnar
2009-12-31 12:00 Ingo Molnar
2009-12-18 18:59 Ingo Molnar
2009-09-22 7:51 Ingo Molnar
2009-09-22 14:49 ` Linus Torvalds
2009-09-22 14:59 ` Ingo Molnar
2009-09-22 15:13 ` Linus Torvalds
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191101203048.GA6622@gmail.com \
--to=mingo@kernel.org \
--cc=a.p.zijlstra@chello.nl \
--cc=acme@infradead.org \
--cc=alexander.shishkin@linux.intel.com \
--cc=jolsa@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).