From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91D32C4361A for ; Thu, 3 Dec 2020 22:15:28 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C6EEE2220F for ; Thu, 3 Dec 2020 22:15:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C6EEE2220F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=csgraf.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:45276 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kkwso-0006us-4u for qemu-devel@archiver.kernel.org; Thu, 03 Dec 2020 17:15:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:37504) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kkwrC-00063d-El; Thu, 03 Dec 2020 17:13:46 -0500 Received: from mail.csgraf.de ([188.138.100.120]:52726 helo=zulu616.server4you.de) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kkwr8-0003Z7-DF; Thu, 03 Dec 2020 17:13:45 -0500 Received: from Alexanders-Mac-mini.local (ec2-3-122-114-9.eu-central-1.compute.amazonaws.com [3.122.114.9]) by csgraf.de (Postfix) with UTF8SMTPSA id 4C0F439003D2; Thu, 3 Dec 2020 23:13:36 +0100 (CET) Subject: Re: [PATCH 2/8] hvf: Move common code out To: Peter Collingbourne , Roman Bolshakov References: <392c2465-157e-e15a-0a2c-2e3faa166d22@csgraf.de> <6975b4a3-1568-df40-8594-bfcf488ac425@csgraf.de> <4e1d93a4-9dcc-c6b6-e060-6eea39ae2f16@csgraf.de> <20201203094124.GA7201@SPB-NB-133.local> From: Alexander Graf Message-ID: <2a286418-aa8e-17b9-1be2-58b0b361dec5@csgraf.de> Date: Thu, 3 Dec 2020 23:13:35 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Thunderbird/84.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US Received-SPF: pass client-ip=188.138.100.120; envelope-from=agraf@csgraf.de; helo=zulu616.server4you.de X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Eduardo Habkost , Richard Henderson , qemu-devel , Cameron Esfahani , qemu-arm , Claudio Fontana , Frank Yang , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On 03.12.20 19:42, Peter Collingbourne wrote: > On Thu, Dec 3, 2020 at 1:41 AM Roman Bolshakov wrote: >> On Mon, Nov 30, 2020 at 04:00:11PM -0800, Peter Collingbourne wrote: >>> On Mon, Nov 30, 2020 at 3:18 PM Alexander Graf wrote: >>>> >>>> On 01.12.20 00:01, Peter Collingbourne wrote: >>>>> On Mon, Nov 30, 2020 at 1:40 PM Alexander Graf wrote: >>>>>> Hi Peter, >>>>>> >>>>>> On 30.11.20 22:08, Peter Collingbourne wrote: >>>>>>> On Mon, Nov 30, 2020 at 12:56 PM Frank Yang wrote: >>>>>>>> On Mon, Nov 30, 2020 at 12:34 PM Alexander Graf wrote: >>>>>>>>> Hi Frank, >>>>>>>>> >>>>>>>>> Thanks for the update :). Your previous email nudged me into the right direction. I previously had implemented WFI through the internal timer framework which performed way worse. >>>>>>>> Cool, glad it's helping. Also, Peter found out that the main thing keeping us from just using cntpct_el0 on the host directly and compare with cval is that if we sleep, cval is going to be much < cntpct_el0 by the sleep time. If we can get either the architecture or macos to read out the sleep time then we might be able to not have to use a poll interval either! >>>>>>>>> Along the way, I stumbled over a few issues though. For starters, the signal mask for SIG_IPI was not set correctly, so while pselect() would exit, the signal would never get delivered to the thread! For a fix, check out >>>>>>>>> >>>>>>>>> https://patchew.org/QEMU/20201130030723.78326-1-agraf@csgraf.de/20201130030723.78326-4-agraf@csgraf.de/ >>>>>>>>> >>>>>>>> Thanks, we'll take a look :) >>>>>>>> >>>>>>>>> Please also have a look at my latest stab at WFI emulation. It doesn't handle WFE (that's only relevant in overcommitted scenarios). But it does handle WFI and even does something similar to hlt polling, albeit not with an adaptive threshold. >>>>>>> Sorry I'm not subscribed to qemu-devel (I'll subscribe in a bit) so >>>>>>> I'll reply to your patch here. You have: >>>>>>> >>>>>>> + /* Set cpu->hvf->sleeping so that we get a >>>>>>> SIG_IPI signal. */ >>>>>>> + cpu->hvf->sleeping = true; >>>>>>> + smp_mb(); >>>>>>> + >>>>>>> + /* Bail out if we received an IRQ meanwhile */ >>>>>>> + if (cpu->thread_kicked || (cpu->interrupt_request & >>>>>>> + (CPU_INTERRUPT_HARD | CPU_INTERRUPT_FIQ))) { >>>>>>> + cpu->hvf->sleeping = false; >>>>>>> + break; >>>>>>> + } >>>>>>> + >>>>>>> + /* nanosleep returns on signal, so we wake up on kick. */ >>>>>>> + nanosleep(ts, NULL); >>>>>>> >>>>>>> and then send the signal conditional on whether sleeping is true, but >>>>>>> I think this is racy. If the signal is sent after sleeping is set to >>>>>>> true but before entering nanosleep then I think it will be ignored and >>>>>>> we will miss the wakeup. That's why in my implementation I block IPI >>>>>>> on the CPU thread at startup and then use pselect to atomically >>>>>>> unblock and begin sleeping. The signal is sent unconditionally so >>>>>>> there's no need to worry about races between actually sleeping and the >>>>>>> "we think we're sleeping" state. It may lead to an extra wakeup but >>>>>>> that's better than missing it entirely. >>>>>> Thanks a bunch for the comment! So the trick I was using here is to > > >> modify the timespec from the kick function before sending the IPI >>>>>> signal. That way, we know that either we are inside the sleep (where the >>>>>> signal wakes it up) or we are outside the sleep (where timespec={} will >>>>>> make it return immediately). >>>>>> >>>>>> The only race I can think of is if nanosleep does calculations based on >>>>>> the timespec and we happen to send the signal right there and then. >>>>> Yes that's the race I was thinking of. Admittedly it's a small window >>>>> but it's theoretically possible and part of the reason why pselect was >>>>> created. >>>>> >>>>>> The problem with blocking IPIs is basically what Frank was describing >>>>>> earlier: How do you unset the IPI signal pending status? If the signal >>>>>> is never delivered, how can pselect differentiate "signal from last time >>>>>> is still pending" from "new signal because I got an IPI"? >>>>> In this case we would take the additional wakeup which should be >>>>> harmless since we will take the WFx exit again and put us in the >>>>> correct state. But that's a lot better than busy looping. >>>> >>>> I'm not sure I follow. I'm thinking of the following scenario: >>>> >>>> - trap into WFI handler >>>> - go to sleep with blocked SIG_IPI >>>> - SIG_IPI arrives, pselect() exits >>>> - signal is still pending because it's blocked >>>> - enter guest >>>> - trap into WFI handler >>>> - run pselect(), but it immediate exits because SIG_IPI is still pending >>>> >>>> This was the loop I was seeing when running with SIG_IPI blocked. That's >>>> part of the reason why I switched to a different model. >>> What I observe is that when returning from a pending signal pselect >>> consumes the signal (which is also consistent with my understanding of >>> what pselect does). That means that it doesn't matter if we take a >>> second WFx exit because once we reach the pselect in the second WFx >>> exit the signal will have been consumed by the pselect in the first >>> exit and we will just wait for the next one. >>> >> Aha! Thanks for the explanation. So, the first WFI in the series of >> guest WFIs will likely wake up immediately? After a period without WFIs >> there must be a pending SIG_IPI... >> >> It shouldn't be a critical issue though because (as defined in D1.16.2) >> "the architecture permits a PE to leave the low-power state for any >> reason, it is permissible for a PE to treat WFI as a NOP, but this is >> not recommended for lowest power operation." >> >> BTW. I think a bit from the thread should go into the description of >> patch 8, because it's not trivial and it would really be helpful to keep >> in repo history. At least something like this (taken from an earlier >> reply in the thread): >> >> In this implementation IPI is blocked on the CPU thread at startup and >> pselect() is used to atomically unblock the signal and begin sleeping. >> The signal is sent unconditionally so there's no need to worry about >> races between actually sleeping and the "we think we're sleeping" >> state. It may lead to an extra wakeup but that's better than missing >> it entirely. > Okay, I'll add something like that to the next version of the patch I send out. If this is the only change, I've already added it for v4. If you want me to change it further, just let me know what to replace the patch description with. Alex