From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F70BC433FE for ; Fri, 17 Dec 2021 13:21:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236632AbhLQNVz (ORCPT ); Fri, 17 Dec 2021 08:21:55 -0500 Received: from foss.arm.com ([217.140.110.172]:57394 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236633AbhLQNVy (ORCPT ); Fri, 17 Dec 2021 08:21:54 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3844A12FC; Fri, 17 Dec 2021 05:21:50 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.67.184]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 972613F5A1; Fri, 17 Dec 2021 05:21:48 -0800 (PST) Date: Fri, 17 Dec 2021 13:21:39 +0000 From: Mark Rutland To: Nicolas Saenz Julienne Cc: maz , Will Deacon , paulmck , linux-arm-kernel , rcu , Thomas Gleixner , frederic , kvmarm@lists.cs.columbia.edu, linux-kernel Subject: Re: Possible nohz-full/RCU issue in arm64 KVM Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Dec 17, 2021 at 12:51:57PM +0100, Nicolas Saenz Julienne wrote: > Hi All, Hi, > arm64's guest entry code does the following: > > int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) > { > [...] > > guest_enter_irqoff(); > > ret = kvm_call_hyp_ret(__kvm_vcpu_run, vcpu); > > [...] > > local_irq_enable(); > > /* > * We do local_irq_enable() before calling guest_exit() so > * that if a timer interrupt hits while running the guest we > * account that tick as being spent in the guest. We enable > * preemption after calling guest_exit() so that if we get > * preempted we make sure ticks after that is not counted as > * guest time. > */ > guest_exit(); > [...] > } > > > On a nohz-full CPU, guest_{enter,exit}() delimit an RCU extended quiescent > state (EQS). Any interrupt happening between local_irq_enable() and > guest_exit() should disable that EQS. Now, AFAICT all el0 interrupt handlers > do the right thing if trggered in this context, but el1's won't. Is it > possible to hit an el1 handler (for example __el1_irq()) there? I think you're right that the EL1 handlers can trigger here and won't exit the EQS. I'm not immediately sure what we *should* do here. What does x86 do for an IRQ taken from a guest mode? I couldn't spot any handling of that case, but I'm not familiar enough with the x86 exception model to know if I'm looking in the right place. Note that the EL0 handlers *cannot* trigger for an exception taken from a guest. We use separate vectors while running a guest (for both VHE and nVHE modes), and from the main kernel's PoV we return from kvm_call_hyp_ret(). We can ony take IRQ from EL1 *after* that returns. We *might* need to audit the KVM vector handlers to make sure they're not dependent on RCU protection (I assume they're not, but it's possible something has leaked into the VHE code). Thanks, Mark. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3F32C433F5 for ; Fri, 17 Dec 2021 13:21:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 61D474B32A; Fri, 17 Dec 2021 08:21:55 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A3zlyVEjf9v3; Fri, 17 Dec 2021 08:21:53 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 4A88E4B32B; Fri, 17 Dec 2021 08:21:53 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 38BE74B329 for ; Fri, 17 Dec 2021 08:21:52 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M284vA6wIIbP for ; Fri, 17 Dec 2021 08:21:51 -0500 (EST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mm01.cs.columbia.edu (Postfix) with ESMTP id EA1054B325 for ; Fri, 17 Dec 2021 08:21:50 -0500 (EST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3844A12FC; Fri, 17 Dec 2021 05:21:50 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.67.184]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 972613F5A1; Fri, 17 Dec 2021 05:21:48 -0800 (PST) Date: Fri, 17 Dec 2021 13:21:39 +0000 From: Mark Rutland To: Nicolas Saenz Julienne Subject: Re: Possible nohz-full/RCU issue in arm64 KVM Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: paulmck , maz , frederic , linux-kernel , rcu , Thomas Gleixner , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Fri, Dec 17, 2021 at 12:51:57PM +0100, Nicolas Saenz Julienne wrote: > Hi All, Hi, > arm64's guest entry code does the following: > > int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) > { > [...] > > guest_enter_irqoff(); > > ret = kvm_call_hyp_ret(__kvm_vcpu_run, vcpu); > > [...] > > local_irq_enable(); > > /* > * We do local_irq_enable() before calling guest_exit() so > * that if a timer interrupt hits while running the guest we > * account that tick as being spent in the guest. We enable > * preemption after calling guest_exit() so that if we get > * preempted we make sure ticks after that is not counted as > * guest time. > */ > guest_exit(); > [...] > } > > > On a nohz-full CPU, guest_{enter,exit}() delimit an RCU extended quiescent > state (EQS). Any interrupt happening between local_irq_enable() and > guest_exit() should disable that EQS. Now, AFAICT all el0 interrupt handlers > do the right thing if trggered in this context, but el1's won't. Is it > possible to hit an el1 handler (for example __el1_irq()) there? I think you're right that the EL1 handlers can trigger here and won't exit the EQS. I'm not immediately sure what we *should* do here. What does x86 do for an IRQ taken from a guest mode? I couldn't spot any handling of that case, but I'm not familiar enough with the x86 exception model to know if I'm looking in the right place. Note that the EL0 handlers *cannot* trigger for an exception taken from a guest. We use separate vectors while running a guest (for both VHE and nVHE modes), and from the main kernel's PoV we return from kvm_call_hyp_ret(). We can ony take IRQ from EL1 *after* that returns. We *might* need to audit the KVM vector handlers to make sure they're not dependent on RCU protection (I assume they're not, but it's possible something has leaked into the VHE code). Thanks, Mark. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0BBF0C433F5 for ; Fri, 17 Dec 2021 13:23:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rhr6un9slwCAemVlvAvNF2PP1kbstN9842R9NzUWCsQ=; b=cGK5xViS/NS7rg AHGr9wwcCDsLR3jGNDI+YClccfFSnaKAAkfXMlWAWf+3nO+/A3HDv+izLRz4DOuTviGKpZsXZ+gZP ede2WpkE3a6tcCL66dJrMV7gr1UNelBV/YutCVoqfeP+xuvJ2M5tgc7epkG1g/YEE92TUz1nnOM7F /kFSI7YEAuq21eRBzSHMz58S8DF5rT9xSBAHQv/9czJ3YEXkAKnBGe64Dsx8S0Fqfdhev1BCyfJlB 1S4Tq61gaX90vVVgOSGO76V4PDDzw9ryhhPO5IOgJuEaRCUfr+Y8hFnpoe1nzaL/hAgCoZyLaLR8p 7qVzZOLlabd/VQhiHgaw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1myDBL-00AMn8-CT; Fri, 17 Dec 2021 13:21:55 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1myDBH-00AMmC-9k for linux-arm-kernel@lists.infradead.org; Fri, 17 Dec 2021 13:21:52 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3844A12FC; Fri, 17 Dec 2021 05:21:50 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.67.184]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 972613F5A1; Fri, 17 Dec 2021 05:21:48 -0800 (PST) Date: Fri, 17 Dec 2021 13:21:39 +0000 From: Mark Rutland To: Nicolas Saenz Julienne Cc: maz , Will Deacon , paulmck , linux-arm-kernel , rcu , Thomas Gleixner , frederic , kvmarm@lists.cs.columbia.edu, linux-kernel Subject: Re: Possible nohz-full/RCU issue in arm64 KVM Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211217_052151_416392_2A0CE90E X-CRM114-Status: GOOD ( 17.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Dec 17, 2021 at 12:51:57PM +0100, Nicolas Saenz Julienne wrote: > Hi All, Hi, > arm64's guest entry code does the following: > > int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) > { > [...] > > guest_enter_irqoff(); > > ret = kvm_call_hyp_ret(__kvm_vcpu_run, vcpu); > > [...] > > local_irq_enable(); > > /* > * We do local_irq_enable() before calling guest_exit() so > * that if a timer interrupt hits while running the guest we > * account that tick as being spent in the guest. We enable > * preemption after calling guest_exit() so that if we get > * preempted we make sure ticks after that is not counted as > * guest time. > */ > guest_exit(); > [...] > } > > > On a nohz-full CPU, guest_{enter,exit}() delimit an RCU extended quiescent > state (EQS). Any interrupt happening between local_irq_enable() and > guest_exit() should disable that EQS. Now, AFAICT all el0 interrupt handlers > do the right thing if trggered in this context, but el1's won't. Is it > possible to hit an el1 handler (for example __el1_irq()) there? I think you're right that the EL1 handlers can trigger here and won't exit the EQS. I'm not immediately sure what we *should* do here. What does x86 do for an IRQ taken from a guest mode? I couldn't spot any handling of that case, but I'm not familiar enough with the x86 exception model to know if I'm looking in the right place. Note that the EL0 handlers *cannot* trigger for an exception taken from a guest. We use separate vectors while running a guest (for both VHE and nVHE modes), and from the main kernel's PoV we return from kvm_call_hyp_ret(). We can ony take IRQ from EL1 *after* that returns. We *might* need to audit the KVM vector handlers to make sure they're not dependent on RCU protection (I assume they're not, but it's possible something has leaked into the VHE code). Thanks, Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel