From: Kyle Huey <me@kylehuey.com> To: Marc Zyngier <maz@kernel.org> Cc: open list <linux-kernel@vger.kernel.org>, "moderated list:ARM PORT" <linux-arm-kernel@lists.infradead.org>, "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>, yyc1992@gmail.com, Keno Fischer <keno@juliacomputing.com>, "Robert O'Callahan" <robert@ocallahan.org>, Thomas Gleixner <tglx@linutronix.de>, Borislav Petkov <bp@alien8.de>, Suzuki K Poulose <suzuki.poulose@arm.com>, Will Deacon <will.deacon@arm.com> Subject: Re: arm64 equivalents of PR_SET_TSC/ARCH_SET_CPUID Date: Mon, 23 May 2022 12:27:24 -0700 [thread overview] Message-ID: <CAP045AroJ3vd8yn+_+F8ANnaUjFL41juQw9Uc9GUSGaKsd_TYA@mail.gmail.com> (raw) In-Reply-To: <87ilpxmvg3.wl-maz@kernel.org> On Sun, May 22, 2022 at 8:35 AM Marc Zyngier <maz@kernel.org> wrote: > > On Sat, 21 May 2022 21:07:14 +0100, > Kyle Huey <me@kylehuey.com> wrote: > > > > There is ongoing work by Yichao Yu to make rr, a userspace record and > > replay debugger[0], production quality on arm64[1]. One of the bigger > > remaining issues is the kernel's emulation of accesses to certain > > system registers[2] that reflect timing and CPU capabilities and are > > either non-deterministic or can vary from processor to processor. > > Just to make things clear: the kernel usually doesn't provide any > emulation for registers such as CNTVCT_EL0. On sane HW, userspace is > free to access it directly without any mediation (we only use the trap > for the sake of dealing with HW bugs). Right, in this case we would also want to set e.g. CNTKCTL_EL1.EL0VCTEN = 0 whenever a task that has used PR_SET_TSC is running. Then the hardware would deliver the fault, and the kernel would decline to emulate the instruction, ultimately leading to the rr supervisor seeing the signal, looking at the offending instruction, and deciding how to handle it itself. This is how RDTSC is handled on x86. prctl(PR_SET_TSC, PR_TSC_SIGSEGV) sets TIF_NOTSC on the task, and then at context switches that leads to setting and clearing CR4.TSD as appropriate. My understanding is that some of these (e.g. MIDR_EL1) actually are always emulated by the kernel. > > We > > would like to add the ability to tell the kernel to decline to emulate > > these instructions for a given task and pass that responsibility onto > > the supervising rr ptracer. There are analogous processor features and > > disabling mechanisms on x86. The RDTSC instruction is controlled by > > prctl(PR_SET_TSC) and the CPUID instruction is controlled (when the > > hardware allows) by arch_prctl(ARCH_SET_CPUID). > > > > The questions I'd like to raise are: > > > > 1. Is it appropriate to reuse PR_SET_TSC for roughly equivalent > > functionality on AArch64? (even if the AArch64 feature is not actually > > named Time Stamp Counter). > > My gut feeling is that you really don't want to hijack an existing > API, because this is fundamentally different. The Linux arm64 ABI > mandates that the counter (and the frequency register associated with > it) are accessible, and you can't make them disappear. We don't want to make them disappear per se. rr would emulate accesses to these features. On x86 rr itself runs an rdtsc during recording after the hardware traps for the tracee and notes the value it returns. rr then sticks that value into the supervised process and resumes execution. During replay we inject the same rdtsc value we noted during recording to produce deterministic behavior. > From what I understand, you are relying on the TSC being disabled in > the tracee and intercepting the signal that gets delivered when it > accesses the counter. Is that correct? Yes. > Assuming I'm right, I think it'd make a lot more sense if there was a > first class ptrace option, if only because this would mandate the > kernel to start trapping things that are not trapped today. > > It also begs the question of the fate of CNTFRQ_EL0, since you want to > be able to replay traces from one system to another (and the counter > is meaningless without the frequency). Yes, we'd want that too. > Finally, what of the VDSO, which is by far the most common user of the > counter? I can totally imagine the VDSO getting stuck if emulation is > used and the sequence counter moves synchronously with the traps > (which is why we disable the VDSO when trapping CNTVCT_EL0). prctl(PR_SET_TSC, PR_TSC_SIGSEGV) will break RDTSC-using VDSO syscalls like gettimeofday entirely on x86 so this isn't a new issue. rr disables the VDSO anyways for other reasons, so this doesn't actually apply to us. Like Keno I am curious what problem you expect if the counter is actually emulated though. > > 2. Likewise for ARCH_SET_CPUID > > We don't just emulate a single register, but a whole class of them. If > you are to present a different view for any of those, you'll need to > handle the lot (I really can't see why one would be more important > than the others). > > So SET_CPUID really is the wrong tool. I'd rather there was (again) an > API that described exactly that. I don't think I communicated accurately what this does. PR_SET_TSC and ARCH_SET_CPUID don't actually set the values returned from RDTSC and CPUID. They instead control whether the instruction is enabled or uses hardware capabilities to trap when executed. The rr supervisor then figures out what to do. x86's CPUID is also quite complicated and has many different sets of information that can be accessed based on the value of %eax,%edx when the CPUID instruction is executed. > > 3. Since arch_prctl is x86-only, does it make more sense to add > > arch_prctl to arm64 or to duplicate ARCH_SET_CPUID into the prctl > > world? (e.g. a PR_SET_CPUID that works on both x86/arm64) > > I don't think any applies here. Different architectures have different > ABI requirements, and you can't really merge the two. Because the next > thing you know, you'll ask for the same thing for PMU registers, and > try to map them onto something else. > > Overall, this would be better served by a framework for userspace > delegation of sysreg access by a ptrace'd process. Let's try to look > at it in those terms rather than casting arm64 into a seemingly > unrelated API. Explicit mechanisms to provide control over the hardware level trap features provided by AArch64 (e.g. SCTLR_EL1.UCT and CNTKCTL_EL1.EL0VCTEN) and the kernel's emulation of other sysregs (e.g. MIDR_EL1) might make more sense. I share Keno's wariness of running it through ptrace specifically (as opposed to a prctl and signal like on x86) though. - Kyle > Thanks, > > M. > > > > > - Kyle > > > > [0] https://rr-project.org/ > > [1] https://github.com/rr-debugger/rr/issues/3234 > > [2] e.g. CNTVCT_EL0 and MIDR_EL1, among others > > > > -- > Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
WARNING: multiple messages have this Message-ID (diff)
From: Kyle Huey <me@kylehuey.com> To: Marc Zyngier <maz@kernel.org> Cc: open list <linux-kernel@vger.kernel.org>, "moderated list:ARM PORT" <linux-arm-kernel@lists.infradead.org>, "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>, yyc1992@gmail.com, Keno Fischer <keno@juliacomputing.com>, "Robert O'Callahan" <robert@ocallahan.org>, Thomas Gleixner <tglx@linutronix.de>, Borislav Petkov <bp@alien8.de>, Suzuki K Poulose <suzuki.poulose@arm.com>, Will Deacon <will.deacon@arm.com> Subject: Re: arm64 equivalents of PR_SET_TSC/ARCH_SET_CPUID Date: Mon, 23 May 2022 12:27:24 -0700 [thread overview] Message-ID: <CAP045AroJ3vd8yn+_+F8ANnaUjFL41juQw9Uc9GUSGaKsd_TYA@mail.gmail.com> (raw) In-Reply-To: <87ilpxmvg3.wl-maz@kernel.org> On Sun, May 22, 2022 at 8:35 AM Marc Zyngier <maz@kernel.org> wrote: > > On Sat, 21 May 2022 21:07:14 +0100, > Kyle Huey <me@kylehuey.com> wrote: > > > > There is ongoing work by Yichao Yu to make rr, a userspace record and > > replay debugger[0], production quality on arm64[1]. One of the bigger > > remaining issues is the kernel's emulation of accesses to certain > > system registers[2] that reflect timing and CPU capabilities and are > > either non-deterministic or can vary from processor to processor. > > Just to make things clear: the kernel usually doesn't provide any > emulation for registers such as CNTVCT_EL0. On sane HW, userspace is > free to access it directly without any mediation (we only use the trap > for the sake of dealing with HW bugs). Right, in this case we would also want to set e.g. CNTKCTL_EL1.EL0VCTEN = 0 whenever a task that has used PR_SET_TSC is running. Then the hardware would deliver the fault, and the kernel would decline to emulate the instruction, ultimately leading to the rr supervisor seeing the signal, looking at the offending instruction, and deciding how to handle it itself. This is how RDTSC is handled on x86. prctl(PR_SET_TSC, PR_TSC_SIGSEGV) sets TIF_NOTSC on the task, and then at context switches that leads to setting and clearing CR4.TSD as appropriate. My understanding is that some of these (e.g. MIDR_EL1) actually are always emulated by the kernel. > > We > > would like to add the ability to tell the kernel to decline to emulate > > these instructions for a given task and pass that responsibility onto > > the supervising rr ptracer. There are analogous processor features and > > disabling mechanisms on x86. The RDTSC instruction is controlled by > > prctl(PR_SET_TSC) and the CPUID instruction is controlled (when the > > hardware allows) by arch_prctl(ARCH_SET_CPUID). > > > > The questions I'd like to raise are: > > > > 1. Is it appropriate to reuse PR_SET_TSC for roughly equivalent > > functionality on AArch64? (even if the AArch64 feature is not actually > > named Time Stamp Counter). > > My gut feeling is that you really don't want to hijack an existing > API, because this is fundamentally different. The Linux arm64 ABI > mandates that the counter (and the frequency register associated with > it) are accessible, and you can't make them disappear. We don't want to make them disappear per se. rr would emulate accesses to these features. On x86 rr itself runs an rdtsc during recording after the hardware traps for the tracee and notes the value it returns. rr then sticks that value into the supervised process and resumes execution. During replay we inject the same rdtsc value we noted during recording to produce deterministic behavior. > From what I understand, you are relying on the TSC being disabled in > the tracee and intercepting the signal that gets delivered when it > accesses the counter. Is that correct? Yes. > Assuming I'm right, I think it'd make a lot more sense if there was a > first class ptrace option, if only because this would mandate the > kernel to start trapping things that are not trapped today. > > It also begs the question of the fate of CNTFRQ_EL0, since you want to > be able to replay traces from one system to another (and the counter > is meaningless without the frequency). Yes, we'd want that too. > Finally, what of the VDSO, which is by far the most common user of the > counter? I can totally imagine the VDSO getting stuck if emulation is > used and the sequence counter moves synchronously with the traps > (which is why we disable the VDSO when trapping CNTVCT_EL0). prctl(PR_SET_TSC, PR_TSC_SIGSEGV) will break RDTSC-using VDSO syscalls like gettimeofday entirely on x86 so this isn't a new issue. rr disables the VDSO anyways for other reasons, so this doesn't actually apply to us. Like Keno I am curious what problem you expect if the counter is actually emulated though. > > 2. Likewise for ARCH_SET_CPUID > > We don't just emulate a single register, but a whole class of them. If > you are to present a different view for any of those, you'll need to > handle the lot (I really can't see why one would be more important > than the others). > > So SET_CPUID really is the wrong tool. I'd rather there was (again) an > API that described exactly that. I don't think I communicated accurately what this does. PR_SET_TSC and ARCH_SET_CPUID don't actually set the values returned from RDTSC and CPUID. They instead control whether the instruction is enabled or uses hardware capabilities to trap when executed. The rr supervisor then figures out what to do. x86's CPUID is also quite complicated and has many different sets of information that can be accessed based on the value of %eax,%edx when the CPUID instruction is executed. > > 3. Since arch_prctl is x86-only, does it make more sense to add > > arch_prctl to arm64 or to duplicate ARCH_SET_CPUID into the prctl > > world? (e.g. a PR_SET_CPUID that works on both x86/arm64) > > I don't think any applies here. Different architectures have different > ABI requirements, and you can't really merge the two. Because the next > thing you know, you'll ask for the same thing for PMU registers, and > try to map them onto something else. > > Overall, this would be better served by a framework for userspace > delegation of sysreg access by a ptrace'd process. Let's try to look > at it in those terms rather than casting arm64 into a seemingly > unrelated API. Explicit mechanisms to provide control over the hardware level trap features provided by AArch64 (e.g. SCTLR_EL1.UCT and CNTKCTL_EL1.EL0VCTEN) and the kernel's emulation of other sysregs (e.g. MIDR_EL1) might make more sense. I share Keno's wariness of running it through ptrace specifically (as opposed to a prctl and signal like on x86) though. - Kyle > Thanks, > > M. > > > > > - Kyle > > > > [0] https://rr-project.org/ > > [1] https://github.com/rr-debugger/rr/issues/3234 > > [2] e.g. CNTVCT_EL0 and MIDR_EL1, among others > > > > -- > Without deviation from the norm, progress is not possible.
next prev parent reply other threads:[~2022-05-23 19:28 UTC|newest] Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-05-21 20:07 arm64 equivalents of PR_SET_TSC/ARCH_SET_CPUID Kyle Huey 2022-05-21 20:07 ` Kyle Huey 2022-05-22 15:35 ` Marc Zyngier 2022-05-22 15:35 ` Marc Zyngier 2022-05-22 18:22 ` Keno Fischer 2022-05-22 18:22 ` Keno Fischer 2022-05-23 19:27 ` Kyle Huey [this message] 2022-05-23 19:27 ` Kyle Huey
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=CAP045AroJ3vd8yn+_+F8ANnaUjFL41juQw9Uc9GUSGaKsd_TYA@mail.gmail.com \ --to=me@kylehuey.com \ --cc=bp@alien8.de \ --cc=keno@juliacomputing.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=maz@kernel.org \ --cc=robert@ocallahan.org \ --cc=suzuki.poulose@arm.com \ --cc=tglx@linutronix.de \ --cc=will.deacon@arm.com \ --cc=x86@kernel.org \ --cc=yyc1992@gmail.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.