From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E588C6FA91 for ; Mon, 19 Sep 2022 09:23:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229566AbiISJXG (ORCPT ); Mon, 19 Sep 2022 05:23:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229854AbiISJXD (ORCPT ); Mon, 19 Sep 2022 05:23:03 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B14325DC; Mon, 19 Sep 2022 02:23:02 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D1C39B8077B; Mon, 19 Sep 2022 09:23:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 847DEC433D6; Mon, 19 Sep 2022 09:22:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1663579379; bh=T2rgL1BDoyjbQ5CdqCib48y34QtH3LVMM+evRnLrKCc=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=N+s9CEuUJrcus3WX5pXi2KpSY+FIS+746DMDxXKmrc6y5vOyr7PN+qGKmvP4OFCdf clTmyW7PcTepmUtpL6ZsQ7R/9Un5uc1RD5L2cQeSqU5J+TvNtxIAJ7jmfcIF4tO6TV iAwQmuIGemMmlgIBumr2yXxJ7O/Cjz4Ni68SE9x/eQhApEQRS3ibupVgsFVILee0ZV +ciKnEO9c+2SuSybMM6rMfgRonDZMtZ+9J17wC0f002jF3Zw8hAXKOLIhMarFVQb0R mD0N/of+l8AyPZrZ1D9A5Km4LrHjggwD0qCGXTUA6itlFZNcZF7FXQe5F9vNwRSX6I siYqKJDXc4Ciw== Received: from 185-176-101-241.host.sccbroadband.ie ([185.176.101.241] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oaCzR-00B3lO-5W; Mon, 19 Sep 2022 10:22:57 +0100 Date: Mon, 19 Sep 2022 10:22:55 +0100 Message-ID: <87czbrlnv4.wl-maz@kernel.org> From: Marc Zyngier To: Gavin Shan Cc: Peter Xu , kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, linux-doc@vger.kernel.org, catalin.marinas@arm.com, linux-kselftest@vger.kernel.org, bgardon@google.com, shuah@kernel.org, corbet@lwn.net, Andrew Jones , will@kernel.org, zhenyzha@redhat.com, dmatlack@google.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, oliver.upton@linux.dev, shan.gavin@gmail.com Subject: Re: [PATCH v2 1/5] KVM: x86: Introduce KVM_REQ_RING_SOFT_FULL In-Reply-To: References: <20220916045135.154505-1-gshan@redhat.com> <20220916045135.154505-2-gshan@redhat.com> <87illlkqfu.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.176.101.241 X-SA-Exim-Rcpt-To: gshan@redhat.com, peterx@redhat.com, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, linux-doc@vger.kernel.org, catalin.marinas@arm.com, linux-kselftest@vger.kernel.org, bgardon@google.com, shuah@kernel.org, corbet@lwn.net, andrew.jones@linux.dev, will@kernel.org, zhenyzha@redhat.com, dmatlack@google.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, oliver.upton@linux.dev, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 19 Sep 2022 00:58:10 +0100, Gavin Shan wrote: > > On 9/18/22 7:00 PM, Marc Zyngier wrote: > > On Fri, 16 Sep 2022 19:09:52 +0100, > > Peter Xu wrote: > >> > >> On Fri, Sep 16, 2022 at 12:51:31PM +0800, Gavin Shan wrote: > >>> This adds KVM_REQ_RING_SOFT_FULL, which is raised when the dirty > >>> ring of the specific VCPU becomes softly full in kvm_dirty_ring_push(). > >>> The VCPU is enforced to exit when the request is raised and its > >>> dirty ring is softly full on its entrance. > >>> > >>> Suggested-by: Marc Zyngier > >>> Signed-off-by: Gavin Shan > >>> --- > >>> arch/x86/kvm/x86.c | 5 +++-- > >>> include/linux/kvm_host.h | 1 + > >>> virt/kvm/dirty_ring.c | 4 ++++ > >>> 3 files changed, 8 insertions(+), 2 deletions(-) > >>> > >>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c > >>> index 43a6a7efc6ec..7f368f59f033 100644 > >>> --- a/arch/x86/kvm/x86.c > >>> +++ b/arch/x86/kvm/x86.c > >>> @@ -10265,8 +10265,9 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) > >>> bool req_immediate_exit = false; > >>> /* Forbid vmenter if vcpu dirty ring is soft-full */ > >>> - if (unlikely(vcpu->kvm->dirty_ring_size && > >>> - kvm_dirty_ring_soft_full(&vcpu->dirty_ring))) { > >>> + if (kvm_check_request(KVM_REQ_RING_SOFT_FULL, vcpu) && > >>> + kvm_dirty_ring_soft_full(&vcpu->dirty_ring)) { > >>> + kvm_make_request(KVM_REQ_RING_SOFT_FULL, vcpu); > >>> vcpu->run->exit_reason = KVM_EXIT_DIRTY_RING_FULL; > >>> trace_kvm_dirty_ring_exit(vcpu); > >>> r = 0; > >> > >> As commented previously - can we use kvm_test_request() instead? because we > >> don't want to unconditionally clear the bit. Instead of making the request > >> again, we can clear request only if !full. > > > > I have the feeling that this is a micro-optimisation that won't lead > > to much benefit in practice. You already have the cache line, just not > > in exclusive mode, and given that this is per-vcpu, you'd only see the > > cost if someone else is setting a request to this vcpu while > > evaluating the local requests. > > > > And now you need extra barriers... > > > > Also, can we please refrain from changing things without data showing > > that this actually is worse than what we had before? The point below > > makes me think that this is actually beneficial as is. > > > > I think Marc's explanation makes sense. It won't make difference in terms > of performance. We need to explicitly handle barrier when kvm_test_request() > is used. So I prefer to keep the code if Peter agrees. > > >> We can also safely move this into the block of below kvm_request_pending() > >> as Marc used to suggest. > > > > This, on the other hand, makes sure that we share the cost across all > > requests. Requests should be extremely rare anyway (and if they > > aren't, you have a whole lot of performance issues on your hands > > anyway). > > > > Yeah, We shouldn't have too much requests. I missed the comment from Marc > to move this chunk to kvm_request_pending(). I will fix it in v3. > > >> > >> To explicitly use kvm_clear_request(), we may need to be careful on the > >> memory barriers. I'm wondering whether we should have moved > >> smp_mb__after_atomic() into kvm_clear_request() because kvm_clear_request() > >> is used outside kvm_check_request() and IIUC all the call sites should > >> better have that barrier too to be safe. > >> > >> Side note: when I read the code around I also see some mis-use of clear > >> request where it can be omitted, e.g.: > >> > >> if (kvm_check_request(KVM_REQ_UNHALT, vcpu)) { > >> kvm_clear_request(KVM_REQ_UNHALT, vcpu); > >> vcpu->run->exit_reason = KVM_EXIT_IRQ_WINDOW_OPEN; > >> } > >> > >> Maybe it's a sign of bad naming, so we should renamed kvm_check_request() > >> to kvm_test_clear_request() too to show that clearing after that is not > >> needed? > > > > Yeah, this kvm_clear_request() is superfluous. But this is rather well > > documented, for once, and I don't think we should repaint it based on > > a sample of one. > > > > Yeah, I think Peter is correct that smp_mb__after_atomic() would be > part of kvm_clear_request(). Otherwise, the following two cases aren't > in same order: > > // kvm_check_request() // test and clear > kvm_test_request() kvm_test_request() > kvm_clear_request() kvm_clear_request() > smp_mb__after_atomic() [fixing Drew's email, again] Oh, I totally agree that a standalone use of kvm_clear_request() must come with a barrier. It is just that without additional data, it isn't obvious to me that there is any need for kvm_clear_request() itself to have the barrier. In a number of cases, kvm_clear_request() is used on in the context of the same vcpu, and this should be enough to ensure visibility (for example, I don't think kvm_vcpu_wfi() should require this barrier). But maybe I'm missing something. Thanks, M. -- Without deviation from the norm, progress is not possible.