All of lore.kernel.org
 help / color / mirror / Atom feed
From: Alexander Graf <agraf@csgraf.de>
To: Peter Collingbourne <pcc@google.com>
Cc: Peter Maydell <peter.maydell@linaro.org>,
	Eduardo Habkost <ehabkost@redhat.com>,
	Richard Henderson <richard.henderson@linaro.org>,
	qemu-devel <qemu-devel@nongnu.org>,
	Cameron Esfahani <dirty@apple.com>,
	Roman Bolshakov <r.bolshakov@yadro.com>,
	qemu-arm@nongnu.org, Claudio Fontana <cfontana@suse.de>,
	Frank Yang <lfy@google.com>, Paolo Bonzini <pbonzini@redhat.com>
Subject: Re: [PATCH 2/8] hvf: Move common code out
Date: Tue, 1 Dec 2020 00:18:02 +0100	[thread overview]
Message-ID: <4e1d93a4-9dcc-c6b6-e060-6eea39ae2f16@csgraf.de> (raw)
In-Reply-To: <CAMn1gO5xs8RniRYm+Gnbh8S3GVah2+2Ew2V6tFL6PNuSJ7o5Hw@mail.gmail.com>


On 01.12.20 00:01, Peter Collingbourne wrote:
> On Mon, Nov 30, 2020 at 1:40 PM Alexander Graf <agraf@csgraf.de> wrote:
>> Hi Peter,
>>
>> On 30.11.20 22:08, Peter Collingbourne wrote:
>>> On Mon, Nov 30, 2020 at 12:56 PM Frank Yang <lfy@google.com> wrote:
>>>>
>>>> On Mon, Nov 30, 2020 at 12:34 PM Alexander Graf <agraf@csgraf.de> wrote:
>>>>> Hi Frank,
>>>>>
>>>>> Thanks for the update :). Your previous email nudged me into the right direction. I previously had implemented WFI through the internal timer framework which performed way worse.
>>>> Cool, glad it's helping. Also, Peter found out that the main thing keeping us from just using cntpct_el0 on the host directly and compare with cval is that if we sleep, cval is going to be much < cntpct_el0 by the sleep time. If we can get either the architecture or macos to read out the sleep time then we might be able to not have to use a poll interval either!
>>>>> Along the way, I stumbled over a few issues though. For starters, the signal mask for SIG_IPI was not set correctly, so while pselect() would exit, the signal would never get delivered to the thread! For a fix, check out
>>>>>
>>>>>     https://patchew.org/QEMU/20201130030723.78326-1-agraf@csgraf.de/20201130030723.78326-4-agraf@csgraf.de/
>>>>>
>>>> Thanks, we'll take a look :)
>>>>
>>>>> Please also have a look at my latest stab at WFI emulation. It doesn't handle WFE (that's only relevant in overcommitted scenarios). But it does handle WFI and even does something similar to hlt polling, albeit not with an adaptive threshold.
>>> Sorry I'm not subscribed to qemu-devel (I'll subscribe in a bit) so
>>> I'll reply to your patch here. You have:
>>>
>>> +                    /* Set cpu->hvf->sleeping so that we get a
>>> SIG_IPI signal. */
>>> +                    cpu->hvf->sleeping = true;
>>> +                    smp_mb();
>>> +
>>> +                    /* Bail out if we received an IRQ meanwhile */
>>> +                    if (cpu->thread_kicked || (cpu->interrupt_request &
>>> +                        (CPU_INTERRUPT_HARD | CPU_INTERRUPT_FIQ))) {
>>> +                        cpu->hvf->sleeping = false;
>>> +                        break;
>>> +                    }
>>> +
>>> +                    /* nanosleep returns on signal, so we wake up on kick. */
>>> +                    nanosleep(ts, NULL);
>>>
>>> and then send the signal conditional on whether sleeping is true, but
>>> I think this is racy. If the signal is sent after sleeping is set to
>>> true but before entering nanosleep then I think it will be ignored and
>>> we will miss the wakeup. That's why in my implementation I block IPI
>>> on the CPU thread at startup and then use pselect to atomically
>>> unblock and begin sleeping. The signal is sent unconditionally so
>>> there's no need to worry about races between actually sleeping and the
>>> "we think we're sleeping" state. It may lead to an extra wakeup but
>>> that's better than missing it entirely.
>>
>> Thanks a bunch for the comment! So the trick I was using here is to
>> modify the timespec from the kick function before sending the IPI
>> signal. That way, we know that either we are inside the sleep (where the
>> signal wakes it up) or we are outside the sleep (where timespec={} will
>> make it return immediately).
>>
>> The only race I can think of is if nanosleep does calculations based on
>> the timespec and we happen to send the signal right there and then.
> Yes that's the race I was thinking of. Admittedly it's a small window
> but it's theoretically possible and part of the reason why pselect was
> created.
>
>> The problem with blocking IPIs is basically what Frank was describing
>> earlier: How do you unset the IPI signal pending status? If the signal
>> is never delivered, how can pselect differentiate "signal from last time
>> is still pending" from "new signal because I got an IPI"?
> In this case we would take the additional wakeup which should be
> harmless since we will take the WFx exit again and put us in the
> correct state. But that's a lot better than busy looping.


I'm not sure I follow. I'm thinking of the following scenario:

   - trap into WFI handler
   - go to sleep with blocked SIG_IPI
   - SIG_IPI arrives, pselect() exits
   - signal is still pending because it's blocked
   - enter guest
   - trap into WFI handler
   - run pselect(), but it immediate exits because SIG_IPI is still pending

This was the loop I was seeing when running with SIG_IPI blocked. That's 
part of the reason why I switched to a different model.


> I reckon that you could improve things a little by unblocking the
> signal and then reblocking it before unlocking iothread (e.g. with a
> pselect with zero time interval), which would flush any pending
> signals. Since any such signal would correspond to a signal from last
> time (because we still have the iothread lock) we know that any future
> signals should correspond to new IPIs.


Yeah, I think you actually *have* to do exactly that, because otherwise 
pselect() will always return after 0ns because the signal is still pending.

And yes, I agree that that starts to sound a bit less racy now. But it 
means we can probably also just do

   - WFI handler
   - block SIG_IPI
   - set hvf->sleeping = true
   - check for pending interrupts
   - pselect()
   - unblock SIG_IPI

which means we run with SIG_IPI unmasked by default. I don't think the 
number of signal mask changes is any different with that compared to 
running with SIG_IPI always masked, right?


Alex



  reply	other threads:[~2020-11-30 23:19 UTC|newest]

Thread overview: 64+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-26 21:50 [PATCH 0/8] hvf: Implement Apple Silicon Support Alexander Graf
2020-11-26 21:50 ` [PATCH 1/8] hvf: Add hypervisor entitlement to output binaries Alexander Graf
2020-11-27  4:54   ` Paolo Bonzini
2020-11-27 19:44   ` Roman Bolshakov
2020-11-27 21:17     ` Paolo Bonzini
2020-11-27 21:51     ` Alexander Graf
2020-11-26 21:50 ` [PATCH 2/8] hvf: Move common code out Alexander Graf
2020-11-27 20:00   ` Roman Bolshakov
2020-11-27 21:55     ` Alexander Graf
2020-11-27 23:30       ` Frank Yang
2020-11-30 20:15         ` Frank Yang
2020-11-30 20:33           ` Alexander Graf
2020-11-30 20:55             ` Frank Yang
2020-11-30 21:08               ` Peter Collingbourne
2020-11-30 21:40                 ` Alexander Graf
2020-11-30 23:01                   ` Peter Collingbourne
2020-11-30 23:18                     ` Alexander Graf [this message]
2020-12-01  0:00                       ` Peter Collingbourne
2020-12-01  0:13                         ` Alexander Graf
2020-12-01  8:21                           ` [PATCH] arm/hvf: Optimize and simplify WFI handling Peter Collingbourne via
2020-12-01 11:16                             ` Alexander Graf
2020-12-01 18:59                               ` Peter Collingbourne
2020-12-01 22:03                                 ` Alexander Graf
2020-12-02  1:19                                   ` Peter Collingbourne
2020-12-02  1:53                                     ` Alexander Graf
2020-12-02  4:44                                       ` Peter Collingbourne
2020-12-03 10:12                                 ` Roman Bolshakov
2020-12-03 18:30                                   ` Peter Collingbourne
2020-12-01 16:26                             ` Alexander Graf
2020-12-01 20:03                               ` Peter Collingbourne
2020-12-01 22:09                                 ` Alexander Graf
2020-12-01 23:13                                   ` Alexander Graf
2020-12-02  0:52                                   ` Peter Collingbourne
2020-12-03  9:41                         ` [PATCH 2/8] hvf: Move common code out Roman Bolshakov
2020-12-03 18:42                           ` Peter Collingbourne
2020-12-03 22:13                             ` Alexander Graf
2020-12-03 23:04                               ` Roman Bolshakov
2020-12-01  0:37                   ` Roman Bolshakov
2020-11-30 22:10               ` Peter Maydell
2020-12-01  2:49                 ` Frank Yang
2020-11-30 22:46               ` Peter Collingbourne
2020-11-26 21:50 ` [PATCH 3/8] arm: Set PSCI to 0.2 for HVF Alexander Graf
2020-11-26 21:50 ` [PATCH 4/8] arm: Synchronize CPU on PSCI on Alexander Graf
2020-11-26 21:50 ` [PATCH 5/8] hvf: Add Apple Silicon support Alexander Graf
2020-11-26 21:50 ` [PATCH 6/8] hvf: Use OS provided vcpu kick function Alexander Graf
2020-11-26 22:18   ` Eduardo Habkost
2020-11-30  2:42     ` Alexander Graf
2020-11-30  7:45       ` Claudio Fontana
2020-11-26 21:50 ` [PATCH 7/8] arm: Add Hypervisor.framework build target Alexander Graf
2020-11-27  4:59   ` Paolo Bonzini
2020-11-26 21:50 ` [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework Alexander Graf
2020-11-26 22:14   ` Eduardo Habkost
2020-11-26 22:29     ` Peter Maydell
2020-11-27 16:26       ` Eduardo Habkost
2020-11-27 16:38         ` Peter Maydell
2020-11-27 16:47           ` Eduardo Habkost
2020-11-27 16:53             ` Peter Maydell
2020-11-27 17:17               ` Eduardo Habkost
2020-11-27 18:16                 ` Peter Maydell
2020-11-27 18:20                   ` Eduardo Habkost
2020-11-27 16:47           ` Peter Maydell
2020-11-30  2:40             ` Alexander Graf
2020-11-26 22:10 ` [PATCH 0/8] hvf: Implement Apple Silicon Support Eduardo Habkost
2020-11-27 17:48   ` Philippe Mathieu-Daudé

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4e1d93a4-9dcc-c6b6-e060-6eea39ae2f16@csgraf.de \
    --to=agraf@csgraf.de \
    --cc=cfontana@suse.de \
    --cc=dirty@apple.com \
    --cc=ehabkost@redhat.com \
    --cc=lfy@google.com \
    --cc=pbonzini@redhat.com \
    --cc=pcc@google.com \
    --cc=peter.maydell@linaro.org \
    --cc=qemu-arm@nongnu.org \
    --cc=qemu-devel@nongnu.org \
    --cc=r.bolshakov@yadro.com \
    --cc=richard.henderson@linaro.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.