From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19B2FC433F5 for ; Wed, 27 Oct 2021 15:07:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0121761090 for ; Wed, 27 Oct 2021 15:07:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242689AbhJ0PJd (ORCPT ); Wed, 27 Oct 2021 11:09:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242656AbhJ0PJ2 (ORCPT ); Wed, 27 Oct 2021 11:09:28 -0400 Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8AB62C061767 for ; Wed, 27 Oct 2021 08:07:02 -0700 (PDT) Received: by mail-pf1-x42b.google.com with SMTP id m26so3014630pff.3 for ; Wed, 27 Oct 2021 08:07:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=O5lEF3hXOuOGQ36jWeA5WrZK+395dTI4kUNN+CFJhtE=; b=GJjlg8jJfw+Bt9Zd0zP4mtqcSM31wSFBYCetU17CHbzQQW4kiFtHDgsZBVvEH8887p yw/Js4mEDuANgaxd1QymvaNpmXbEvaNRm2HNB868WPSRynugBhM6LN65l++VZht0g2X0 XFMEItNS+WJlFOOOZlwyo5ghqOI0eoYdx8EfG7dg89d25ghzdUn9Jr0jTft3rzInpzui t+nhlFj3SyV2ez0xA0qiaGWyaH3JZAU5KHIJMn2b+NEmBeF/xDfhUhjIDnEt41PMETob dNlEs4TAu5VbQ+MRxeaG1Cc4kLdotzXGkh/U5A20JnxHgI8kyVG06UiAaMfvZtmJbPiD c4pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=O5lEF3hXOuOGQ36jWeA5WrZK+395dTI4kUNN+CFJhtE=; b=vymnVc8ZJFt7kG8f0qaKNHKdWw0B1yCBYvSmz0izRS4LpKtA+wNthhPRwzsDULV6Hv qcOtr4Xw+r6f/NGjEBKHBAq3UpN8vogbUDdrQF5jDNSucZb6sAlkNLlTOFwDYUm18Naw B60HTOiIS3fMU6TUEeaeWYwlRSg8w3AY9GaUBRFDI3oNhOS3GdR9QrSDtZmQ5JCvUIQq YYAcwiHiEbmRWt9/+/uDAYr5cILlfCLkVGEqz4taMR/HYB1r2heN8g+FWICYv7ghAKcr aRWDvPC92t18fR5AXnhqGPFoJTiFgDtoMPXEGUak6SmUOKd1a44PqiVkU/4e7RAA9GN6 1JQA== X-Gm-Message-State: AOAM530n9hfL1IAPNckeN7F2q3clH4gvh4KnjRQY9T614muHOJgeYOVW chehgf5Gomr7A+v1hajFEHEqDA== X-Google-Smtp-Source: ABdhPJyFiG0oaOa31Ni2mfE2COsoLS8wTaHHbwW9HjhwNmX9PoY62hBCx/P2f3tKT436xIXd6QXjiA== X-Received: by 2002:a05:6a00:15c9:b0:44c:a998:b50d with SMTP id o9-20020a056a0015c900b0044ca998b50dmr33155514pfu.49.1635347221673; Wed, 27 Oct 2021 08:07:01 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id x15sm310904pfp.30.2021.10.27.08.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 08:07:00 -0700 (PDT) Date: Wed, 27 Oct 2021 15:06:57 +0000 From: Sean Christopherson To: Paolo Bonzini Cc: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Subject: Re: [PATCH v2 35/43] KVM: SVM: Signal AVIC doorbell iff vCPU is in guest mode Message-ID: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-36-seanjc@google.com> <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Oct 25, 2021, Paolo Bonzini wrote: > On 09/10/21 04:12, Sean Christopherson wrote: > > + */ > > + if (vcpu->mode == IN_GUEST_MODE) { > > int cpu = READ_ONCE(vcpu->cpu); > > /* > > @@ -687,8 +692,13 @@ int svm_deliver_avic_intr(struct kvm_vcpu *vcpu, int vec) > > if (cpu != get_cpu()) > > wrmsrl(SVM_AVIC_DOORBELL, kvm_cpu_get_apicid(cpu)); > > put_cpu(); > > - } else > > + } else { > > + /* > > + * Wake the vCPU if it was blocking. KVM will then detect the > > + * pending IRQ when checking if the vCPU has a wake event. > > + */ > > kvm_vcpu_wake_up(vcpu); > > + } > > Does this still need to check the "running" flag? That should be a strict > superset of vcpu->mode == IN_GUEST_MODE. No. Signalling the doorbell when "running" is set but the vCPU is not in the guest is just an expensive nop. So even if KVM were to rework its handling of "running" to set the flag immediately before VMRUN and clear it immediately after, keying off IN_GUEST_MODE and not "running" would not be wrong, just sub-optimal. I doubt KVM will ever make the "running" flag super precise, because keeping the flag set when the vCPU is loaded avoids VM-Exits on other vCPUs due to undelivered IPIs. But the flip side is that it means the flag has terrible granularity, and is arguably inaccurate when viewed from a software perspective. Anyways, if the treatment of "running" were ever changed, then this code should also be changed to essentially revert this commit since vcpu->mode would then be redundant. And IMO, it makes sense to intentionally separate KVM's delivery of interrupts from hardware's delivery of interrupts. I.e. use the same core rules as kvm_vcpu_kick() for when to send interrupts and when to wake for the AVIC. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8F40C433F5 for ; Wed, 27 Oct 2021 15:07:18 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7769D604AC for ; Wed, 27 Oct 2021 15:07:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7769D604AC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=R34XqV8qsTNi8pNZpFrZoE1GBXgUHrDH98YSQiDbMKM=; b=19+YJnV4oXm3LA R9Pj4XXA5/gc6gdLJO31lkwShM7Ei2zVp/rIkagtlPk/xy80+6AMSXVOUJJuP9Mx1VZg+RTXtmHQx ng6yRiL3lARLPUpROdXLtAC949VWMvw0ngkPZzXk3gWU3Kr9m5+GNSoYA8oe0grIDxbbrOiY8haGt WTCjikJmrlPHvQyzYIhIW77Y7ZOEV03MVgw0LDsTnC/K9rrDhBSBOvU7BCW8e/NJ3dxQ61dA6OtbQ 4zAxzUasOhrPf7QyN1UKjHQScuOL3VvWKkhzdZt9jGPinsvpgintOPeQ8m63s9xdzr8N43QJJ5YHo sOU77keKQtUeNzHvCCMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mfkWA-005EkG-CJ; Wed, 27 Oct 2021 15:07:06 +0000 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mfkW7-005Eis-7Z for linux-riscv@lists.infradead.org; Wed, 27 Oct 2021 15:07:04 +0000 Received: by mail-pf1-x42a.google.com with SMTP id 187so2980429pfc.10 for ; Wed, 27 Oct 2021 08:07:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=O5lEF3hXOuOGQ36jWeA5WrZK+395dTI4kUNN+CFJhtE=; b=GJjlg8jJfw+Bt9Zd0zP4mtqcSM31wSFBYCetU17CHbzQQW4kiFtHDgsZBVvEH8887p yw/Js4mEDuANgaxd1QymvaNpmXbEvaNRm2HNB868WPSRynugBhM6LN65l++VZht0g2X0 XFMEItNS+WJlFOOOZlwyo5ghqOI0eoYdx8EfG7dg89d25ghzdUn9Jr0jTft3rzInpzui t+nhlFj3SyV2ez0xA0qiaGWyaH3JZAU5KHIJMn2b+NEmBeF/xDfhUhjIDnEt41PMETob dNlEs4TAu5VbQ+MRxeaG1Cc4kLdotzXGkh/U5A20JnxHgI8kyVG06UiAaMfvZtmJbPiD c4pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=O5lEF3hXOuOGQ36jWeA5WrZK+395dTI4kUNN+CFJhtE=; b=BH7B9xhgxXxanWQXR5s1ZoT+AGPGaL3qQuMLHkEJU+xPj6VInqE6a15oeHt9H/+Zz9 Cri1oDrD/9Cv36uWxwUV2DF4YrhS4CA7W16gFzlYX0Sw3KYp4CP9urRgduOjSFWgH52G Hg1ckph+ZDlhMA+bHf1mCOtvyxqIsGhHvpFZqKGLV/ulrwcV19w5w6Mbid+IHmkJH3XO a6Ryg4NgYY6sxt4YAqHD54jMI300UxOWDSGs/sNnne+X0QfmP2k6Gv2tgpqkxLD0aWPY xfa9fDr5lQQEVD3BKIITUlI5RDRKBdhDKNN5YLIDxM8t+E2ny6Y2xC8VJGAIOxopY1eQ 3Kcg== X-Gm-Message-State: AOAM531G66vO8aQuGxVt3p3+11Qi1A9ZZL0FDfqyJ3r310DAy+arNQkO LUYkoOxOg6fm56c0baNtTgIM7Q== X-Google-Smtp-Source: ABdhPJyFiG0oaOa31Ni2mfE2COsoLS8wTaHHbwW9HjhwNmX9PoY62hBCx/P2f3tKT436xIXd6QXjiA== X-Received: by 2002:a05:6a00:15c9:b0:44c:a998:b50d with SMTP id o9-20020a056a0015c900b0044ca998b50dmr33155514pfu.49.1635347221673; Wed, 27 Oct 2021 08:07:01 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id x15sm310904pfp.30.2021.10.27.08.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 08:07:00 -0700 (PDT) Date: Wed, 27 Oct 2021 15:06:57 +0000 From: Sean Christopherson To: Paolo Bonzini Cc: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Subject: Re: [PATCH v2 35/43] KVM: SVM: Signal AVIC doorbell iff vCPU is in guest mode Message-ID: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-36-seanjc@google.com> <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211027_080703_303430_ADE62FC9 X-CRM114-Status: GOOD ( 18.45 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Mon, Oct 25, 2021, Paolo Bonzini wrote: > On 09/10/21 04:12, Sean Christopherson wrote: > > + */ > > + if (vcpu->mode == IN_GUEST_MODE) { > > int cpu = READ_ONCE(vcpu->cpu); > > /* > > @@ -687,8 +692,13 @@ int svm_deliver_avic_intr(struct kvm_vcpu *vcpu, int vec) > > if (cpu != get_cpu()) > > wrmsrl(SVM_AVIC_DOORBELL, kvm_cpu_get_apicid(cpu)); > > put_cpu(); > > - } else > > + } else { > > + /* > > + * Wake the vCPU if it was blocking. KVM will then detect the > > + * pending IRQ when checking if the vCPU has a wake event. > > + */ > > kvm_vcpu_wake_up(vcpu); > > + } > > Does this still need to check the "running" flag? That should be a strict > superset of vcpu->mode == IN_GUEST_MODE. No. Signalling the doorbell when "running" is set but the vCPU is not in the guest is just an expensive nop. So even if KVM were to rework its handling of "running" to set the flag immediately before VMRUN and clear it immediately after, keying off IN_GUEST_MODE and not "running" would not be wrong, just sub-optimal. I doubt KVM will ever make the "running" flag super precise, because keeping the flag set when the vCPU is loaded avoids VM-Exits on other vCPUs due to undelivered IPIs. But the flip side is that it means the flag has terrible granularity, and is arguably inaccurate when viewed from a software perspective. Anyways, if the treatment of "running" were ever changed, then this code should also be changed to essentially revert this commit since vcpu->mode would then be redundant. And IMO, it makes sense to intentionally separate KVM's delivery of interrupts from hardware's delivery of interrupts. I.e. use the same core rules as kvm_vcpu_kick() for when to send interrupts and when to wake for the AVIC. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 961F4C433EF for ; Wed, 27 Oct 2021 15:07:12 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 0FEFC60EFF for ; Wed, 27 Oct 2021 15:07:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0FEFC60EFF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 84A984B159; Wed, 27 Oct 2021 11:07:11 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VZIOo+vCnOEg; Wed, 27 Oct 2021 11:07:06 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0B7234B175; Wed, 27 Oct 2021 11:07:06 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 5DF3C4B168 for ; Wed, 27 Oct 2021 11:07:04 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dBoK0ao8Uo3z for ; Wed, 27 Oct 2021 11:07:03 -0400 (EDT) Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id E858D4B15C for ; Wed, 27 Oct 2021 11:07:02 -0400 (EDT) Received: by mail-pf1-f179.google.com with SMTP id m26so3014631pff.3 for ; Wed, 27 Oct 2021 08:07:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=O5lEF3hXOuOGQ36jWeA5WrZK+395dTI4kUNN+CFJhtE=; b=GJjlg8jJfw+Bt9Zd0zP4mtqcSM31wSFBYCetU17CHbzQQW4kiFtHDgsZBVvEH8887p yw/Js4mEDuANgaxd1QymvaNpmXbEvaNRm2HNB868WPSRynugBhM6LN65l++VZht0g2X0 XFMEItNS+WJlFOOOZlwyo5ghqOI0eoYdx8EfG7dg89d25ghzdUn9Jr0jTft3rzInpzui t+nhlFj3SyV2ez0xA0qiaGWyaH3JZAU5KHIJMn2b+NEmBeF/xDfhUhjIDnEt41PMETob dNlEs4TAu5VbQ+MRxeaG1Cc4kLdotzXGkh/U5A20JnxHgI8kyVG06UiAaMfvZtmJbPiD c4pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=O5lEF3hXOuOGQ36jWeA5WrZK+395dTI4kUNN+CFJhtE=; b=ZvzDkO5F6rAPnqssubrOz1BgZsrjcwUSgNBveHqiPwinLdb8Vkopta45AgS6YGvTug +fepe05D/UjgQQQYXmaJtJms0Xa8nnLEHyAmDLF30kNBxTAhD271h2kq3ZL3xOcnth7n Xt+ZwuTZeSQiUpkr8z33MzgLbq+iGH0Wcaup0XZHocrDwB/+AuB8N+tMwUOyVblyTijP oRU+DAclvmNKF7qtcvwnkTA/flnh/W3XBofUG/dR7XPowzCQD0cARDSCeTbmn9N9Gp+9 5YMr8xZF5j41K+i++mj6xb5fDTNC/Yv1tDBpsVRprPV02hqlnRmjoeLryUyp1Gy5suGi 06lw== X-Gm-Message-State: AOAM532XCIDlQDcFfm0cxdsxrW+iR3howX1/Z3ynbJ4GAB5x3QXNVwBK kPJMk7wZTmCavBMJe3teJYA3Cw== X-Google-Smtp-Source: ABdhPJyFiG0oaOa31Ni2mfE2COsoLS8wTaHHbwW9HjhwNmX9PoY62hBCx/P2f3tKT436xIXd6QXjiA== X-Received: by 2002:a05:6a00:15c9:b0:44c:a998:b50d with SMTP id o9-20020a056a0015c900b0044ca998b50dmr33155514pfu.49.1635347221673; Wed, 27 Oct 2021 08:07:01 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id x15sm310904pfp.30.2021.10.27.08.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 08:07:00 -0700 (PDT) Date: Wed, 27 Oct 2021 15:06:57 +0000 From: Sean Christopherson To: Paolo Bonzini Subject: Re: [PATCH v2 35/43] KVM: SVM: Signal AVIC doorbell iff vCPU is in guest mode Message-ID: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-36-seanjc@google.com> <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> Cc: Cornelia Huck , Wanpeng Li , kvm@vger.kernel.org, David Hildenbrand , linux-kernel@vger.kernel.org, Paul Mackerras , Atish Patra , linux-riscv@lists.infradead.org, Claudio Imbrenda , kvmarm@lists.cs.columbia.edu, Janosch Frank , Marc Zyngier , Joerg Roedel , Huacai Chen , Christian Borntraeger , Aleksandar Markovic , Albert Ou , kvm-ppc@vger.kernel.org, Paul Walmsley , David Matlack , linux-arm-kernel@lists.infradead.org, Jim Mattson , Anup Patel , linux-mips@vger.kernel.org, Palmer Dabbelt , kvm-riscv@lists.infradead.org, Vitaly Kuznetsov X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Mon, Oct 25, 2021, Paolo Bonzini wrote: > On 09/10/21 04:12, Sean Christopherson wrote: > > + */ > > + if (vcpu->mode == IN_GUEST_MODE) { > > int cpu = READ_ONCE(vcpu->cpu); > > /* > > @@ -687,8 +692,13 @@ int svm_deliver_avic_intr(struct kvm_vcpu *vcpu, int vec) > > if (cpu != get_cpu()) > > wrmsrl(SVM_AVIC_DOORBELL, kvm_cpu_get_apicid(cpu)); > > put_cpu(); > > - } else > > + } else { > > + /* > > + * Wake the vCPU if it was blocking. KVM will then detect the > > + * pending IRQ when checking if the vCPU has a wake event. > > + */ > > kvm_vcpu_wake_up(vcpu); > > + } > > Does this still need to check the "running" flag? That should be a strict > superset of vcpu->mode == IN_GUEST_MODE. No. Signalling the doorbell when "running" is set but the vCPU is not in the guest is just an expensive nop. So even if KVM were to rework its handling of "running" to set the flag immediately before VMRUN and clear it immediately after, keying off IN_GUEST_MODE and not "running" would not be wrong, just sub-optimal. I doubt KVM will ever make the "running" flag super precise, because keeping the flag set when the vCPU is loaded avoids VM-Exits on other vCPUs due to undelivered IPIs. But the flip side is that it means the flag has terrible granularity, and is arguably inaccurate when viewed from a software perspective. Anyways, if the treatment of "running" were ever changed, then this code should also be changed to essentially revert this commit since vcpu->mode would then be redundant. And IMO, it makes sense to intentionally separate KVM's delivery of interrupts from hardware's delivery of interrupts. I.e. use the same core rules as kvm_vcpu_kick() for when to send interrupts and when to wake for the AVIC. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B4FBC433F5 for ; Wed, 27 Oct 2021 15:08:31 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E78EF60F5A for ; Wed, 27 Oct 2021 15:08:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E78EF60F5A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=X5YH9cv0B2G8scULRsa9sgK+dPgpZOV6soGM/hsCFzc=; b=tGK+g9fnJMS9tc wnNM2nggW2ImSWJU7MnSJu4KNmKt+a2GmC0vd7ufpvSnEVxzbELq9Qe2b0DlHc66pjamtF2kFd168 4WpU3ucrBUjVAbv2eMtG+2ajeJtTKwKpeX8jWwi8IbmKEpmEXOhoUoT047WwD1W5uwPBAsoXPKag9 5kUI6kBftAiaAQhpNyFYZlTDvZm/+kDvaHm//iifFd+FhIg23RUcPwldCjT1ZDIkbsNFJrO7Tmnaz qGssDHebEOmmixzQhj86vg/tI4zx6Th0RXWmDR5ha6uRcbdq0lkW9aQeBdztWhhomoKtJ9BqaSRef fwdH37YkM4ric1y6NrRQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mfkWD-005El0-AX; Wed, 27 Oct 2021 15:07:09 +0000 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mfkW7-005Eit-94 for linux-arm-kernel@lists.infradead.org; Wed, 27 Oct 2021 15:07:05 +0000 Received: by mail-pf1-x435.google.com with SMTP id o133so2991099pfg.7 for ; Wed, 27 Oct 2021 08:07:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=O5lEF3hXOuOGQ36jWeA5WrZK+395dTI4kUNN+CFJhtE=; b=GJjlg8jJfw+Bt9Zd0zP4mtqcSM31wSFBYCetU17CHbzQQW4kiFtHDgsZBVvEH8887p yw/Js4mEDuANgaxd1QymvaNpmXbEvaNRm2HNB868WPSRynugBhM6LN65l++VZht0g2X0 XFMEItNS+WJlFOOOZlwyo5ghqOI0eoYdx8EfG7dg89d25ghzdUn9Jr0jTft3rzInpzui t+nhlFj3SyV2ez0xA0qiaGWyaH3JZAU5KHIJMn2b+NEmBeF/xDfhUhjIDnEt41PMETob dNlEs4TAu5VbQ+MRxeaG1Cc4kLdotzXGkh/U5A20JnxHgI8kyVG06UiAaMfvZtmJbPiD c4pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=O5lEF3hXOuOGQ36jWeA5WrZK+395dTI4kUNN+CFJhtE=; b=Arlag1Q2uoO92eo2b/3vX9bD4dEhhuWs7tgaMwLTVy1EhMOSZXJX1qOqzUxtO7PGwT aQCUyMrbytof3uSvx8Lx+aqBz6GA1bA2DO9eE9is6ZXrI0koBxbGPpWqzZHDE9QtQDeZ lJVmXAbrcMt9lO33Y3dKioO9B4kQVMjNmCAwKrqP5KxuppwiP427/MRJL/JLOgCUiqF+ UFOFrCyOJjNsuW+69oyg7FOkvxVGSxs81Eg62Tgau66bduJ7H7vN1YHjOfYG6ADHAC/i CmJP4yEko6dCkpA9iUg9pWkkRB/sJoafiarbDGoMaALHVg2QZsurHydkz9XXX5M/k/m4 lAGg== X-Gm-Message-State: AOAM533zdXgYp5P0hacfgV4IAPvfGMJzAJS16ThknSexjCa5yqMxirEL T3L86G2lH67IFFaHdqdOHsSkFw== X-Google-Smtp-Source: ABdhPJyFiG0oaOa31Ni2mfE2COsoLS8wTaHHbwW9HjhwNmX9PoY62hBCx/P2f3tKT436xIXd6QXjiA== X-Received: by 2002:a05:6a00:15c9:b0:44c:a998:b50d with SMTP id o9-20020a056a0015c900b0044ca998b50dmr33155514pfu.49.1635347221673; Wed, 27 Oct 2021 08:07:01 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id x15sm310904pfp.30.2021.10.27.08.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 08:07:00 -0700 (PDT) Date: Wed, 27 Oct 2021 15:06:57 +0000 From: Sean Christopherson To: Paolo Bonzini Cc: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Subject: Re: [PATCH v2 35/43] KVM: SVM: Signal AVIC doorbell iff vCPU is in guest mode Message-ID: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-36-seanjc@google.com> <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211027_080703_337632_46296C6D X-CRM114-Status: GOOD ( 19.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Oct 25, 2021, Paolo Bonzini wrote: > On 09/10/21 04:12, Sean Christopherson wrote: > > + */ > > + if (vcpu->mode == IN_GUEST_MODE) { > > int cpu = READ_ONCE(vcpu->cpu); > > /* > > @@ -687,8 +692,13 @@ int svm_deliver_avic_intr(struct kvm_vcpu *vcpu, int vec) > > if (cpu != get_cpu()) > > wrmsrl(SVM_AVIC_DOORBELL, kvm_cpu_get_apicid(cpu)); > > put_cpu(); > > - } else > > + } else { > > + /* > > + * Wake the vCPU if it was blocking. KVM will then detect the > > + * pending IRQ when checking if the vCPU has a wake event. > > + */ > > kvm_vcpu_wake_up(vcpu); > > + } > > Does this still need to check the "running" flag? That should be a strict > superset of vcpu->mode == IN_GUEST_MODE. No. Signalling the doorbell when "running" is set but the vCPU is not in the guest is just an expensive nop. So even if KVM were to rework its handling of "running" to set the flag immediately before VMRUN and clear it immediately after, keying off IN_GUEST_MODE and not "running" would not be wrong, just sub-optimal. I doubt KVM will ever make the "running" flag super precise, because keeping the flag set when the vCPU is loaded avoids VM-Exits on other vCPUs due to undelivered IPIs. But the flip side is that it means the flag has terrible granularity, and is arguably inaccurate when viewed from a software perspective. Anyways, if the treatment of "running" were ever changed, then this code should also be changed to essentially revert this commit since vcpu->mode would then be redundant. And IMO, it makes sense to intentionally separate KVM's delivery of interrupts from hardware's delivery of interrupts. I.e. use the same core rules as kvm_vcpu_kick() for when to send interrupts and when to wake for the AVIC. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Wed, 27 Oct 2021 15:06:57 +0000 Subject: Re: [PATCH v2 35/43] KVM: SVM: Signal AVIC doorbell iff vCPU is in guest mode Message-Id: List-Id: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-36-seanjc@google.com> <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> In-Reply-To: <0333be2a-76d8-657a-6c82-3bb5c9ff2e3b@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Paolo Bonzini Cc: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang On Mon, Oct 25, 2021, Paolo Bonzini wrote: > On 09/10/21 04:12, Sean Christopherson wrote: > > + */ > > + if (vcpu->mode = IN_GUEST_MODE) { > > int cpu = READ_ONCE(vcpu->cpu); > > /* > > @@ -687,8 +692,13 @@ int svm_deliver_avic_intr(struct kvm_vcpu *vcpu, int vec) > > if (cpu != get_cpu()) > > wrmsrl(SVM_AVIC_DOORBELL, kvm_cpu_get_apicid(cpu)); > > put_cpu(); > > - } else > > + } else { > > + /* > > + * Wake the vCPU if it was blocking. KVM will then detect the > > + * pending IRQ when checking if the vCPU has a wake event. > > + */ > > kvm_vcpu_wake_up(vcpu); > > + } > > Does this still need to check the "running" flag? That should be a strict > superset of vcpu->mode = IN_GUEST_MODE. No. Signalling the doorbell when "running" is set but the vCPU is not in the guest is just an expensive nop. So even if KVM were to rework its handling of "running" to set the flag immediately before VMRUN and clear it immediately after, keying off IN_GUEST_MODE and not "running" would not be wrong, just sub-optimal. I doubt KVM will ever make the "running" flag super precise, because keeping the flag set when the vCPU is loaded avoids VM-Exits on other vCPUs due to undelivered IPIs. But the flip side is that it means the flag has terrible granularity, and is arguably inaccurate when viewed from a software perspective. Anyways, if the treatment of "running" were ever changed, then this code should also be changed to essentially revert this commit since vcpu->mode would then be redundant. And IMO, it makes sense to intentionally separate KVM's delivery of interrupts from hardware's delivery of interrupts. I.e. use the same core rules as kvm_vcpu_kick() for when to send interrupts and when to wake for the AVIC.