From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21A24C433EF for ; Fri, 17 Dec 2021 14:15:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237001AbhLQOPe (ORCPT ); Fri, 17 Dec 2021 09:15:34 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:26197 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234148AbhLQOPd (ORCPT ); Fri, 17 Dec 2021 09:15:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639750533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8ohkHIXbcCmEoZRJLeZUjZIF9jP1Brmddy61OPwOb80=; b=dJ4MVPCgBqQRi8XLU/hNxpIhKXluP+1ucjJps/8oI4TA152smLEppxFahhfqn7PbGLRYvm 0EyoCgLDi22vK4QxxamIxnwyRFkeBmnLvDPJnJJoHcGowZGJXJ3hAuNnjW/ueSR7QVVjnT RjbYvqaxHeF3GQ0bvSKW5Ls8YV0Lp6c= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-646-Is-ZLqUqMd68hIEGfcUTCQ-1; Fri, 17 Dec 2021 09:15:31 -0500 X-MC-Unique: Is-ZLqUqMd68hIEGfcUTCQ-1 Received: by mail-wm1-f72.google.com with SMTP id l4-20020a05600c1d0400b00332f47a0fa3so1104159wms.8 for ; Fri, 17 Dec 2021 06:15:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=8ohkHIXbcCmEoZRJLeZUjZIF9jP1Brmddy61OPwOb80=; b=wZ4z7GgjdjBoMmdjPifpVc23JKW6KjTFuXQvUgYxe7encS4uM0QT6bLTu/orH3Nppv vfbGRfmLDnQT1nKrxyp3qJswZ8kk35YECM0xoTygjSU2MHQE9nmcWcr85LNT5LDINiJ2 c+QBglhGS4BB8VTZ7lSNCUmmYEM00wqdzRr1XIyGZ9vpFq6QocWtZNwWgC3YGYN1rjfd q9myI9JNHqIbVHVgCC6MkH0D2cpxdYwbPZ2EPTZBrvDV8nTIqzVZiD3ZH7+z80ZOE/yj FLsIwGyVJPkt3lenJ73ogCBbedXLnnWWnHG0h62RkEJIg4qMdMnrwO9nuOZLFGFCFor8 fhFA== X-Gm-Message-State: AOAM530iycr4otHUtb6b+F1ffPUr58at7vdbjw9jb7dNsDjoHD3NTTPg Xz6g5tWyjc+OzfaudX9BuDXyTyHGr8wDBdi/6wu0IwoJEMK9pzdb107nUFFmWiNExLUToksoGhI kZemahZi5AKf94fxJe9CtuwqV X-Received: by 2002:a5d:6d41:: with SMTP id k1mr2755368wri.134.1639750530462; Fri, 17 Dec 2021 06:15:30 -0800 (PST) X-Google-Smtp-Source: ABdhPJy+5OK+wHJ+oNtmr4sqDbaf0TDCwcL3mgQ2Qu1ZhPUKAGIgzAQyZaO0vxFPOl7tahX2YBbx5w== X-Received: by 2002:a5d:6d41:: with SMTP id k1mr2755351wri.134.1639750530277; Fri, 17 Dec 2021 06:15:30 -0800 (PST) Received: from ?IPv6:2a0c:5a80:3511:9200:f938:102b:34c2:d5db? ([2a0c:5a80:3511:9200:f938:102b:34c2:d5db]) by smtp.gmail.com with ESMTPSA id m20sm11441371wmq.11.2021.12.17.06.15.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Dec 2021 06:15:30 -0800 (PST) Message-ID: <70f112072d9496d21901946ea82832d3ed3a8cb2.camel@redhat.com> Subject: Re: Possible nohz-full/RCU issue in arm64 KVM From: Nicolas Saenz Julienne To: Mark Rutland Cc: maz , Will Deacon , paulmck , linux-arm-kernel , rcu , Thomas Gleixner , frederic , kvmarm@lists.cs.columbia.edu, linux-kernel Date: Fri, 17 Dec 2021 15:15:29 +0100 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.42.2 (3.42.2-1.fc35) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 2021-12-17 at 13:21 +0000, Mark Rutland wrote: > On Fri, Dec 17, 2021 at 12:51:57PM +0100, Nicolas Saenz Julienne wrote: > > Hi All, > > Hi, > > > arm64's guest entry code does the following: > > > > int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) > > { > > [...] > > > > guest_enter_irqoff(); > > > > ret = kvm_call_hyp_ret(__kvm_vcpu_run, vcpu); > > > > [...] > > > > local_irq_enable(); > > > > /* > > * We do local_irq_enable() before calling guest_exit() so > > * that if a timer interrupt hits while running the guest we > > * account that tick as being spent in the guest. We enable > > * preemption after calling guest_exit() so that if we get > > * preempted we make sure ticks after that is not counted as > > * guest time. > > */ > > guest_exit(); > > [...] > > } > > > > > > On a nohz-full CPU, guest_{enter,exit}() delimit an RCU extended quiescent > > state (EQS). Any interrupt happening between local_irq_enable() and > > guest_exit() should disable that EQS. Now, AFAICT all el0 interrupt handlers > > do the right thing if trggered in this context, but el1's won't. Is it > > possible to hit an el1 handler (for example __el1_irq()) there? > > I think you're right that the EL1 handlers can trigger here and won't exit the > EQS. > > I'm not immediately sure what we *should* do here. What does x86 do for an IRQ > taken from a guest mode? I couldn't spot any handling of that case, but I'm not > familiar enough with the x86 exception model to know if I'm looking in the > right place. Well x86 has its own private KVM guest context exit function 'kvm_guest_exit_irqoff()', which allows it to do the right thing (simplifying things): local_irq_disable(); kvm_guest_enter_irqoff() // Inform CT, enter EQS __vmx_kvm_run() kvm_guest_exit_irqoff() // Inform CT, exit EQS, task still marked with PF_VCPU /* * Consume any pending interrupts, including the possible source of * VM-Exit on SVM and any ticks that occur between VM-Exit and now. * An instruction is required after local_irq_enable() to fully unblock * interrupts on processors that implement an interrupt shadow, the * stat.exits increment will do nicely. */ local_irq_enable(); ++vcpu->stat.exits; local_irq_disable(); /* * Wait until after servicing IRQs to account guest time so that any * ticks that occurred while running the guest are properly accounted * to the guest. Waiting until IRQs are enabled degrades the accuracy * of accounting via context tracking, but the loss of accuracy is * acceptable for all known use cases. */ vtime_account_guest_exit(); // current->flags &= ~PF_VCPU So I guess we should convert to x86's scheme, and maybe create another generic guest_{enter,exit}() flavor for virtualization schemes that run with interrupts disabled. > Note that the EL0 handlers *cannot* trigger for an exception taken from a > guest. We use separate vectors while running a guest (for both VHE and nVHE > modes), and from the main kernel's PoV we return from kvm_call_hyp_ret(). We > can ony take IRQ from EL1 *after* that returns. > > We *might* need to audit the KVM vector handlers to make sure they're not > dependent on RCU protection (I assume they're not, but it's possible something > has leaked into the VHE code). IIUC in the window between local_irq_enable() and guest_exit() any driver interrupt might trigger, isn't it? Regards, -- Nicolás Sáenz From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E53D3C433F5 for ; Fri, 17 Dec 2021 14:17:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:Cc:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/ouURfxFLrRajYvLymmG01ryp58cDhSFl2e+CakR20I=; b=n07OM3icGsISy/ kkzPzfLX1n82fQGmA/a0g6RbTWZ+BUX8U0Fj/kAdkf2ep82U/Wya6U3dFBmLYtaZEHeRK9Ficsa8k ee8p6zTnB0N78EJWFGXsh9RLgvRjQ4Hl2Dj6qh8/aX0I5T66/AW5vBXWLsP+aL91u0so2YhuVnicG bwa+8BHqg9+EIGtN/lwIcj3H3284kw4hRn3OrGK9/P9fWpqBQ4tYIUpFTUQP4UPK3mjmyT0Y0njNn hu2R0AEytRDylhyX1VkVW4Jxt0O3bDAM6vYE3hRMWVj20+94Y1WlZNN0RURaPVAQUPpVANYnSs47B ZEnYGl1wCVTlNp9E4+Kg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1myE1M-00AaRi-IS; Fri, 17 Dec 2021 14:15:40 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1myE1I-00AaQa-J3 for linux-arm-kernel@lists.infradead.org; Fri, 17 Dec 2021 14:15:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639750534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8ohkHIXbcCmEoZRJLeZUjZIF9jP1Brmddy61OPwOb80=; b=dwNleYIsoSLXSj6avhJOIuZEQkgXuhlorHldm8PUe+UsHlbgThgBcZro9tNS6yI1LN03JC CkpKejnTHLBG1jd7qXOzTsAotzUHkrMatuq069qdfPN6LssfXZcUgysbNTTYWzf0cmmeP0 GZmUcTN08Cv+La+hlec2fXioLPR6ZfU= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-261-5bh3G1LrOmSdxRNJfCYDSg-1; Fri, 17 Dec 2021 09:15:31 -0500 X-MC-Unique: 5bh3G1LrOmSdxRNJfCYDSg-1 Received: by mail-wm1-f69.google.com with SMTP id i15-20020a05600c354f00b0034566ac865bso1107246wmq.6 for ; Fri, 17 Dec 2021 06:15:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=8ohkHIXbcCmEoZRJLeZUjZIF9jP1Brmddy61OPwOb80=; b=Qs2ztq3OjpM9Q2EaiOn9G5/RhuRBC61icaD2eNtxHBA3+0vhn5bfX5GQobbBP/OWhZ NDzHtUL6bhaLjw7KcUXEYQglPrWc47w+RxPLyS31n9gUz/NQSt+uDJhThDXV4sbxN+pf 2qsg/aFTh7MoJ5ohpI91Rj17HsbE/qbZXjTn+XjEM2JornisijmjMYaSPx0drl2OgDh2 gJhss5DYksLIxfoYX4N4yx+GMHwxl3kboyTxuO6GtyF9ZU2ad47qC0soZwzIUWquQSjt DwDr2naNka4HRkjhwznvrjI1QwRVsYSKyj41tD5X5jwovflGTDczclI0HkcaTnHTBqY9 imbQ== X-Gm-Message-State: AOAM530fAkpqZYtTUJZkNXYPWTk5vRCTYvVryqDZq1Vr/R6m4VXCvPPu 2ECUJJ7T6X5xIFR6rv+ljEj/sIDTyUGzVsfME5WNYcxbrFphcQ7/lAuWIOEBYXJa2BEgX/LdEfD h8sy7/lB062qlUuh+lrmDzrGrSQugIPPKh5Y= X-Received: by 2002:a5d:6d41:: with SMTP id k1mr2755365wri.134.1639750530462; Fri, 17 Dec 2021 06:15:30 -0800 (PST) X-Google-Smtp-Source: ABdhPJy+5OK+wHJ+oNtmr4sqDbaf0TDCwcL3mgQ2Qu1ZhPUKAGIgzAQyZaO0vxFPOl7tahX2YBbx5w== X-Received: by 2002:a5d:6d41:: with SMTP id k1mr2755351wri.134.1639750530277; Fri, 17 Dec 2021 06:15:30 -0800 (PST) Received: from ?IPv6:2a0c:5a80:3511:9200:f938:102b:34c2:d5db? ([2a0c:5a80:3511:9200:f938:102b:34c2:d5db]) by smtp.gmail.com with ESMTPSA id m20sm11441371wmq.11.2021.12.17.06.15.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Dec 2021 06:15:30 -0800 (PST) Message-ID: <70f112072d9496d21901946ea82832d3ed3a8cb2.camel@redhat.com> Subject: Re: Possible nohz-full/RCU issue in arm64 KVM From: Nicolas Saenz Julienne To: Mark Rutland Cc: maz , Will Deacon , paulmck , linux-arm-kernel , rcu , Thomas Gleixner , frederic , kvmarm@lists.cs.columbia.edu, linux-kernel Date: Fri, 17 Dec 2021 15:15:29 +0100 In-Reply-To: References: User-Agent: Evolution 3.42.2 (3.42.2-1.fc35) MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=nsaenzju@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211217_061536_731550_EF3B5C74 X-CRM114-Status: GOOD ( 28.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gRnJpLCAyMDIxLTEyLTE3IGF0IDEzOjIxICswMDAwLCBNYXJrIFJ1dGxhbmQgd3JvdGU6Cj4g T24gRnJpLCBEZWMgMTcsIDIwMjEgYXQgMTI6NTE6NTdQTSArMDEwMCwgTmljb2xhcyBTYWVueiBK dWxpZW5uZSB3cm90ZToKPiA+IEhpIEFsbCwKPiAKPiBIaSwKPiAKPiA+IGFybTY0J3MgZ3Vlc3Qg ZW50cnkgY29kZSBkb2VzIHRoZSBmb2xsb3dpbmc6Cj4gPiAKPiA+IGludCBrdm1fYXJjaF92Y3B1 X2lvY3RsX3J1bihzdHJ1Y3Qga3ZtX3ZjcHUgKnZjcHUpCj4gPiB7Cj4gPiAJWy4uLl0KPiA+IAo+ ID4gCWd1ZXN0X2VudGVyX2lycW9mZigpOwo+ID4gCj4gPiAJcmV0ID0ga3ZtX2NhbGxfaHlwX3Jl dChfX2t2bV92Y3B1X3J1biwgdmNwdSk7Cj4gPiAKPiA+IAlbLi4uXQo+ID4gCj4gPiAJbG9jYWxf aXJxX2VuYWJsZSgpOwo+ID4gCj4gPiAJLyoKPiA+IAkgKiBXZSBkbyBsb2NhbF9pcnFfZW5hYmxl KCkgYmVmb3JlIGNhbGxpbmcgZ3Vlc3RfZXhpdCgpIHNvCj4gPiAJICogdGhhdCBpZiBhIHRpbWVy IGludGVycnVwdCBoaXRzIHdoaWxlIHJ1bm5pbmcgdGhlIGd1ZXN0IHdlCj4gPiAJICogYWNjb3Vu dCB0aGF0IHRpY2sgYXMgYmVpbmcgc3BlbnQgaW4gdGhlIGd1ZXN0LiAgV2UgZW5hYmxlCj4gPiAJ ICogcHJlZW1wdGlvbiBhZnRlciBjYWxsaW5nIGd1ZXN0X2V4aXQoKSBzbyB0aGF0IGlmIHdlIGdl dAo+ID4gCSAqIHByZWVtcHRlZCB3ZSBtYWtlIHN1cmUgdGlja3MgYWZ0ZXIgdGhhdCBpcyBub3Qg Y291bnRlZCBhcwo+ID4gCSAqIGd1ZXN0IHRpbWUuCj4gPiAJICovCj4gPiAJZ3Vlc3RfZXhpdCgp Owo+ID4gCVsuLi5dCj4gPiB9Cj4gPiAKPiA+IAo+ID4gT24gYSBub2h6LWZ1bGwgQ1BVLCBndWVz dF97ZW50ZXIsZXhpdH0oKSBkZWxpbWl0IGFuIFJDVSBleHRlbmRlZCBxdWllc2NlbnQKPiA+IHN0 YXRlIChFUVMpLiBBbnkgaW50ZXJydXB0IGhhcHBlbmluZyBiZXR3ZWVuIGxvY2FsX2lycV9lbmFi bGUoKSBhbmQKPiA+IGd1ZXN0X2V4aXQoKSBzaG91bGQgZGlzYWJsZSB0aGF0IEVRUy4gTm93LCBB RkFJQ1QgYWxsIGVsMCBpbnRlcnJ1cHQgaGFuZGxlcnMKPiA+IGRvIHRoZSByaWdodCB0aGluZyBp ZiB0cmdnZXJlZCBpbiB0aGlzIGNvbnRleHQsIGJ1dCBlbDEncyB3b24ndC4gSXMgaXQKPiA+IHBv c3NpYmxlIHRvIGhpdCBhbiBlbDEgaGFuZGxlciAoZm9yIGV4YW1wbGUgX19lbDFfaXJxKCkpIHRo ZXJlPwo+IAo+IEkgdGhpbmsgeW91J3JlIHJpZ2h0IHRoYXQgdGhlIEVMMSBoYW5kbGVycyBjYW4g dHJpZ2dlciBoZXJlIGFuZCB3b24ndCBleGl0IHRoZQo+IEVRUy4KPiAKPiBJJ20gbm90IGltbWVk aWF0ZWx5IHN1cmUgd2hhdCB3ZSAqc2hvdWxkKiBkbyBoZXJlLiBXaGF0IGRvZXMgeDg2IGRvIGZv ciBhbiBJUlEKPiB0YWtlbiBmcm9tIGEgZ3Vlc3QgbW9kZT8gSSBjb3VsZG4ndCBzcG90IGFueSBo YW5kbGluZyBvZiB0aGF0IGNhc2UsIGJ1dCBJJ20gbm90Cj4gZmFtaWxpYXIgZW5vdWdoIHdpdGgg dGhlIHg4NiBleGNlcHRpb24gbW9kZWwgdG8ga25vdyBpZiBJJ20gbG9va2luZyBpbiB0aGUKPiBy aWdodCBwbGFjZS4KCldlbGwgeDg2IGhhcyBpdHMgb3duIHByaXZhdGUgS1ZNIGd1ZXN0IGNvbnRl eHQgZXhpdCBmdW5jdGlvbgona3ZtX2d1ZXN0X2V4aXRfaXJxb2ZmKCknLCB3aGljaCBhbGxvd3Mg aXQgdG8gZG8gdGhlIHJpZ2h0IHRoaW5nIChzaW1wbGlmeWluZwp0aGluZ3MpOgoKCWxvY2FsX2ly cV9kaXNhYmxlKCk7Cglrdm1fZ3Vlc3RfZW50ZXJfaXJxb2ZmKCkgLy8gSW5mb3JtIENULCBlbnRl ciBFUVMKCV9fdm14X2t2bV9ydW4oKQoJa3ZtX2d1ZXN0X2V4aXRfaXJxb2ZmKCkgLy8gSW5mb3Jt IENULCBleGl0IEVRUywgdGFzayBzdGlsbCBtYXJrZWQgd2l0aCBQRl9WQ1BVCgoJLyoKCSAqIENv bnN1bWUgYW55IHBlbmRpbmcgaW50ZXJydXB0cywgaW5jbHVkaW5nIHRoZSBwb3NzaWJsZSBzb3Vy Y2Ugb2YKCSAqIFZNLUV4aXQgb24gU1ZNIGFuZCBhbnkgdGlja3MgdGhhdCBvY2N1ciBiZXR3ZWVu IFZNLUV4aXQgYW5kIG5vdy4KCSAqIEFuIGluc3RydWN0aW9uIGlzIHJlcXVpcmVkIGFmdGVyIGxv Y2FsX2lycV9lbmFibGUoKSB0byBmdWxseSB1bmJsb2NrCgkgKiBpbnRlcnJ1cHRzIG9uIHByb2Nl c3NvcnMgdGhhdCBpbXBsZW1lbnQgYW4gaW50ZXJydXB0IHNoYWRvdywgdGhlCgkgKiBzdGF0LmV4 aXRzIGluY3JlbWVudCB3aWxsIGRvIG5pY2VseS4KCSAqLwoJbG9jYWxfaXJxX2VuYWJsZSgpOwoJ Kyt2Y3B1LT5zdGF0LmV4aXRzOwoJbG9jYWxfaXJxX2Rpc2FibGUoKTsKCgkvKgoJICogV2FpdCB1 bnRpbCBhZnRlciBzZXJ2aWNpbmcgSVJRcyB0byBhY2NvdW50IGd1ZXN0IHRpbWUgc28gdGhhdCBh bnkKCSAqIHRpY2tzIHRoYXQgb2NjdXJyZWQgd2hpbGUgcnVubmluZyB0aGUgZ3Vlc3QgYXJlIHBy b3Blcmx5IGFjY291bnRlZAoJICogdG8gdGhlIGd1ZXN0LiAgV2FpdGluZyB1bnRpbCBJUlFzIGFy ZSBlbmFibGVkIGRlZ3JhZGVzIHRoZSBhY2N1cmFjeQoJICogb2YgYWNjb3VudGluZyB2aWEgY29u dGV4dCB0cmFja2luZywgYnV0IHRoZSBsb3NzIG9mIGFjY3VyYWN5IGlzCgkgKiBhY2NlcHRhYmxl IGZvciBhbGwga25vd24gdXNlIGNhc2VzLgoJICovCgl2dGltZV9hY2NvdW50X2d1ZXN0X2V4aXQo KTsgLy8gY3VycmVudC0+ZmxhZ3MgJj0gflBGX1ZDUFUKClNvIEkgZ3Vlc3Mgd2Ugc2hvdWxkIGNv bnZlcnQgdG8geDg2J3Mgc2NoZW1lLCBhbmQgbWF5YmUgY3JlYXRlIGFub3RoZXIgZ2VuZXJpYwpn dWVzdF97ZW50ZXIsZXhpdH0oKSBmbGF2b3IgZm9yIHZpcnR1YWxpemF0aW9uIHNjaGVtZXMgdGhh dCBydW4gd2l0aCBpbnRlcnJ1cHRzCmRpc2FibGVkLgoKPiBOb3RlIHRoYXQgdGhlIEVMMCBoYW5k bGVycyAqY2Fubm90KiB0cmlnZ2VyIGZvciBhbiBleGNlcHRpb24gdGFrZW4gZnJvbSBhCj4gZ3Vl c3QuIFdlIHVzZSBzZXBhcmF0ZSB2ZWN0b3JzIHdoaWxlIHJ1bm5pbmcgYSBndWVzdCAoZm9yIGJv dGggVkhFIGFuZCBuVkhFCj4gbW9kZXMpLCBhbmQgZnJvbSB0aGUgbWFpbiBrZXJuZWwncyBQb1Yg d2UgcmV0dXJuIGZyb20ga3ZtX2NhbGxfaHlwX3JldCgpLiBXZQo+IGNhbiBvbnkgdGFrZSBJUlEg ZnJvbSBFTDEgKmFmdGVyKiB0aGF0IHJldHVybnMuCj4gCj4gV2UgKm1pZ2h0KiBuZWVkIHRvIGF1 ZGl0IHRoZSBLVk0gdmVjdG9yIGhhbmRsZXJzIHRvIG1ha2Ugc3VyZSB0aGV5J3JlIG5vdAo+IGRl cGVuZGVudCBvbiBSQ1UgcHJvdGVjdGlvbiAoSSBhc3N1bWUgdGhleSdyZSBub3QsIGJ1dCBpdCdz IHBvc3NpYmxlIHNvbWV0aGluZwo+IGhhcyBsZWFrZWQgaW50byB0aGUgVkhFIGNvZGUpLgoKSUlV QyBpbiB0aGUgd2luZG93IGJldHdlZW4gbG9jYWxfaXJxX2VuYWJsZSgpIGFuZCBndWVzdF9leGl0 KCkgYW55IGRyaXZlcgppbnRlcnJ1cHQgbWlnaHQgdHJpZ2dlciwgaXNuJ3QgaXQ/CgpSZWdhcmRz LAoKLS0gCk5pY29sw6FzIFPDoWVuegoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1r ZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWls bWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AE77C433FE for ; Sun, 19 Dec 2021 12:12:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id BBD2A4B2C8; Sun, 19 Dec 2021 07:12:15 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@redhat.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cXia2hsHP4v6; Sun, 19 Dec 2021 07:12:14 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6821A4B2E7; Sun, 19 Dec 2021 07:12:13 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id E25514B31A for ; Fri, 17 Dec 2021 09:15:33 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7HYouwW+qMb0 for ; Fri, 17 Dec 2021 09:15:33 -0500 (EST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mm01.cs.columbia.edu (Postfix) with ESMTP id EF9C04B31D for ; Fri, 17 Dec 2021 09:15:32 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639750532; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8ohkHIXbcCmEoZRJLeZUjZIF9jP1Brmddy61OPwOb80=; b=Ug2xhOJn4AQZV3QGeubq7X5HmDWTb7nWntIeW+IHIU4R3DCxdAXmYO4YYQspWgocMtJygw kt/htu49O6vK8VC2XKACD0/np+HA12FEQ5Ew86LJXjCvkVPU7rNnAnDLEzINg0v2NmENjO gFXXMQ9Bqo5gCVypmzMusLqh/fgjPO0= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-600-gW8-27PfP8SGOjF1v4qm3g-1; Fri, 17 Dec 2021 09:15:31 -0500 X-MC-Unique: gW8-27PfP8SGOjF1v4qm3g-1 Received: by mail-wm1-f69.google.com with SMTP id n41-20020a05600c502900b003335ab97f41so1109117wmr.3 for ; Fri, 17 Dec 2021 06:15:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=8ohkHIXbcCmEoZRJLeZUjZIF9jP1Brmddy61OPwOb80=; b=Ou2c6frHPTHiFcc06Z90RuYExVxBCqvEvjEbMQBF/ICjLYaHZdcVxhZhWA4X3JkEPd STjh+Y8tbciaEZgF/oEVf7RlHANzu6KanXqmU99Jrj8+ZuIfENHmvG8JeY79714tRy3v kmkv2TQd1DOZaHCTPVDdYoI+fqBSS7kM5rSLfdF9m9lCMK6blMbwvb0tsSuQNBGUrgPk UhqiF9hAprp0hS8fSQf6o9n2qoZjncbxBFB3leGp1Q7wod+SZUXAm73eFBhfm+ac7DKN rWXUT+cG4Lj5YkR9SQGCgYrMO05PrapYNzUPS5ElJ7oTI16/C1ZMPn1xfl7HmNsDQ0Zr TFvg== X-Gm-Message-State: AOAM533iZVXu89Uc8gGERWj/+cQ93pS/JBIxCFvYGJeNJU2tZxTe4Rdk pNlUVq/O7jwBq4oo/ay089utbrdPO8YsjkuJN6E2Hebd5Jwf0TkSA/NK31/QQAVR9HEduNp4OJy vtIAxEh6MjxfPjcYLAXTkoHhL X-Received: by 2002:a5d:6d41:: with SMTP id k1mr2755370wri.134.1639750530463; Fri, 17 Dec 2021 06:15:30 -0800 (PST) X-Google-Smtp-Source: ABdhPJy+5OK+wHJ+oNtmr4sqDbaf0TDCwcL3mgQ2Qu1ZhPUKAGIgzAQyZaO0vxFPOl7tahX2YBbx5w== X-Received: by 2002:a5d:6d41:: with SMTP id k1mr2755351wri.134.1639750530277; Fri, 17 Dec 2021 06:15:30 -0800 (PST) Received: from ?IPv6:2a0c:5a80:3511:9200:f938:102b:34c2:d5db? ([2a0c:5a80:3511:9200:f938:102b:34c2:d5db]) by smtp.gmail.com with ESMTPSA id m20sm11441371wmq.11.2021.12.17.06.15.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Dec 2021 06:15:30 -0800 (PST) Message-ID: <70f112072d9496d21901946ea82832d3ed3a8cb2.camel@redhat.com> Subject: Re: Possible nohz-full/RCU issue in arm64 KVM From: Nicolas Saenz Julienne To: Mark Rutland Date: Fri, 17 Dec 2021 15:15:29 +0100 In-Reply-To: References: User-Agent: Evolution 3.42.2 (3.42.2-1.fc35) MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=nsaenzju@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Mailman-Approved-At: Sun, 19 Dec 2021 07:12:12 -0500 Cc: paulmck , maz , frederic , linux-kernel , rcu , Thomas Gleixner , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu T24gRnJpLCAyMDIxLTEyLTE3IGF0IDEzOjIxICswMDAwLCBNYXJrIFJ1dGxhbmQgd3JvdGU6Cj4g T24gRnJpLCBEZWMgMTcsIDIwMjEgYXQgMTI6NTE6NTdQTSArMDEwMCwgTmljb2xhcyBTYWVueiBK dWxpZW5uZSB3cm90ZToKPiA+IEhpIEFsbCwKPiAKPiBIaSwKPiAKPiA+IGFybTY0J3MgZ3Vlc3Qg ZW50cnkgY29kZSBkb2VzIHRoZSBmb2xsb3dpbmc6Cj4gPiAKPiA+IGludCBrdm1fYXJjaF92Y3B1 X2lvY3RsX3J1bihzdHJ1Y3Qga3ZtX3ZjcHUgKnZjcHUpCj4gPiB7Cj4gPiAJWy4uLl0KPiA+IAo+ ID4gCWd1ZXN0X2VudGVyX2lycW9mZigpOwo+ID4gCj4gPiAJcmV0ID0ga3ZtX2NhbGxfaHlwX3Jl dChfX2t2bV92Y3B1X3J1biwgdmNwdSk7Cj4gPiAKPiA+IAlbLi4uXQo+ID4gCj4gPiAJbG9jYWxf aXJxX2VuYWJsZSgpOwo+ID4gCj4gPiAJLyoKPiA+IAkgKiBXZSBkbyBsb2NhbF9pcnFfZW5hYmxl KCkgYmVmb3JlIGNhbGxpbmcgZ3Vlc3RfZXhpdCgpIHNvCj4gPiAJICogdGhhdCBpZiBhIHRpbWVy IGludGVycnVwdCBoaXRzIHdoaWxlIHJ1bm5pbmcgdGhlIGd1ZXN0IHdlCj4gPiAJICogYWNjb3Vu dCB0aGF0IHRpY2sgYXMgYmVpbmcgc3BlbnQgaW4gdGhlIGd1ZXN0LiAgV2UgZW5hYmxlCj4gPiAJ ICogcHJlZW1wdGlvbiBhZnRlciBjYWxsaW5nIGd1ZXN0X2V4aXQoKSBzbyB0aGF0IGlmIHdlIGdl dAo+ID4gCSAqIHByZWVtcHRlZCB3ZSBtYWtlIHN1cmUgdGlja3MgYWZ0ZXIgdGhhdCBpcyBub3Qg Y291bnRlZCBhcwo+ID4gCSAqIGd1ZXN0IHRpbWUuCj4gPiAJICovCj4gPiAJZ3Vlc3RfZXhpdCgp Owo+ID4gCVsuLi5dCj4gPiB9Cj4gPiAKPiA+IAo+ID4gT24gYSBub2h6LWZ1bGwgQ1BVLCBndWVz dF97ZW50ZXIsZXhpdH0oKSBkZWxpbWl0IGFuIFJDVSBleHRlbmRlZCBxdWllc2NlbnQKPiA+IHN0 YXRlIChFUVMpLiBBbnkgaW50ZXJydXB0IGhhcHBlbmluZyBiZXR3ZWVuIGxvY2FsX2lycV9lbmFi bGUoKSBhbmQKPiA+IGd1ZXN0X2V4aXQoKSBzaG91bGQgZGlzYWJsZSB0aGF0IEVRUy4gTm93LCBB RkFJQ1QgYWxsIGVsMCBpbnRlcnJ1cHQgaGFuZGxlcnMKPiA+IGRvIHRoZSByaWdodCB0aGluZyBp ZiB0cmdnZXJlZCBpbiB0aGlzIGNvbnRleHQsIGJ1dCBlbDEncyB3b24ndC4gSXMgaXQKPiA+IHBv c3NpYmxlIHRvIGhpdCBhbiBlbDEgaGFuZGxlciAoZm9yIGV4YW1wbGUgX19lbDFfaXJxKCkpIHRo ZXJlPwo+IAo+IEkgdGhpbmsgeW91J3JlIHJpZ2h0IHRoYXQgdGhlIEVMMSBoYW5kbGVycyBjYW4g dHJpZ2dlciBoZXJlIGFuZCB3b24ndCBleGl0IHRoZQo+IEVRUy4KPiAKPiBJJ20gbm90IGltbWVk aWF0ZWx5IHN1cmUgd2hhdCB3ZSAqc2hvdWxkKiBkbyBoZXJlLiBXaGF0IGRvZXMgeDg2IGRvIGZv ciBhbiBJUlEKPiB0YWtlbiBmcm9tIGEgZ3Vlc3QgbW9kZT8gSSBjb3VsZG4ndCBzcG90IGFueSBo YW5kbGluZyBvZiB0aGF0IGNhc2UsIGJ1dCBJJ20gbm90Cj4gZmFtaWxpYXIgZW5vdWdoIHdpdGgg dGhlIHg4NiBleGNlcHRpb24gbW9kZWwgdG8ga25vdyBpZiBJJ20gbG9va2luZyBpbiB0aGUKPiBy aWdodCBwbGFjZS4KCldlbGwgeDg2IGhhcyBpdHMgb3duIHByaXZhdGUgS1ZNIGd1ZXN0IGNvbnRl eHQgZXhpdCBmdW5jdGlvbgona3ZtX2d1ZXN0X2V4aXRfaXJxb2ZmKCknLCB3aGljaCBhbGxvd3Mg aXQgdG8gZG8gdGhlIHJpZ2h0IHRoaW5nIChzaW1wbGlmeWluZwp0aGluZ3MpOgoKCWxvY2FsX2ly cV9kaXNhYmxlKCk7Cglrdm1fZ3Vlc3RfZW50ZXJfaXJxb2ZmKCkgLy8gSW5mb3JtIENULCBlbnRl ciBFUVMKCV9fdm14X2t2bV9ydW4oKQoJa3ZtX2d1ZXN0X2V4aXRfaXJxb2ZmKCkgLy8gSW5mb3Jt IENULCBleGl0IEVRUywgdGFzayBzdGlsbCBtYXJrZWQgd2l0aCBQRl9WQ1BVCgoJLyoKCSAqIENv bnN1bWUgYW55IHBlbmRpbmcgaW50ZXJydXB0cywgaW5jbHVkaW5nIHRoZSBwb3NzaWJsZSBzb3Vy Y2Ugb2YKCSAqIFZNLUV4aXQgb24gU1ZNIGFuZCBhbnkgdGlja3MgdGhhdCBvY2N1ciBiZXR3ZWVu IFZNLUV4aXQgYW5kIG5vdy4KCSAqIEFuIGluc3RydWN0aW9uIGlzIHJlcXVpcmVkIGFmdGVyIGxv Y2FsX2lycV9lbmFibGUoKSB0byBmdWxseSB1bmJsb2NrCgkgKiBpbnRlcnJ1cHRzIG9uIHByb2Nl c3NvcnMgdGhhdCBpbXBsZW1lbnQgYW4gaW50ZXJydXB0IHNoYWRvdywgdGhlCgkgKiBzdGF0LmV4 aXRzIGluY3JlbWVudCB3aWxsIGRvIG5pY2VseS4KCSAqLwoJbG9jYWxfaXJxX2VuYWJsZSgpOwoJ Kyt2Y3B1LT5zdGF0LmV4aXRzOwoJbG9jYWxfaXJxX2Rpc2FibGUoKTsKCgkvKgoJICogV2FpdCB1 bnRpbCBhZnRlciBzZXJ2aWNpbmcgSVJRcyB0byBhY2NvdW50IGd1ZXN0IHRpbWUgc28gdGhhdCBh bnkKCSAqIHRpY2tzIHRoYXQgb2NjdXJyZWQgd2hpbGUgcnVubmluZyB0aGUgZ3Vlc3QgYXJlIHBy b3Blcmx5IGFjY291bnRlZAoJICogdG8gdGhlIGd1ZXN0LiAgV2FpdGluZyB1bnRpbCBJUlFzIGFy ZSBlbmFibGVkIGRlZ3JhZGVzIHRoZSBhY2N1cmFjeQoJICogb2YgYWNjb3VudGluZyB2aWEgY29u dGV4dCB0cmFja2luZywgYnV0IHRoZSBsb3NzIG9mIGFjY3VyYWN5IGlzCgkgKiBhY2NlcHRhYmxl IGZvciBhbGwga25vd24gdXNlIGNhc2VzLgoJICovCgl2dGltZV9hY2NvdW50X2d1ZXN0X2V4aXQo KTsgLy8gY3VycmVudC0+ZmxhZ3MgJj0gflBGX1ZDUFUKClNvIEkgZ3Vlc3Mgd2Ugc2hvdWxkIGNv bnZlcnQgdG8geDg2J3Mgc2NoZW1lLCBhbmQgbWF5YmUgY3JlYXRlIGFub3RoZXIgZ2VuZXJpYwpn dWVzdF97ZW50ZXIsZXhpdH0oKSBmbGF2b3IgZm9yIHZpcnR1YWxpemF0aW9uIHNjaGVtZXMgdGhh dCBydW4gd2l0aCBpbnRlcnJ1cHRzCmRpc2FibGVkLgoKPiBOb3RlIHRoYXQgdGhlIEVMMCBoYW5k bGVycyAqY2Fubm90KiB0cmlnZ2VyIGZvciBhbiBleGNlcHRpb24gdGFrZW4gZnJvbSBhCj4gZ3Vl c3QuIFdlIHVzZSBzZXBhcmF0ZSB2ZWN0b3JzIHdoaWxlIHJ1bm5pbmcgYSBndWVzdCAoZm9yIGJv dGggVkhFIGFuZCBuVkhFCj4gbW9kZXMpLCBhbmQgZnJvbSB0aGUgbWFpbiBrZXJuZWwncyBQb1Yg d2UgcmV0dXJuIGZyb20ga3ZtX2NhbGxfaHlwX3JldCgpLiBXZQo+IGNhbiBvbnkgdGFrZSBJUlEg ZnJvbSBFTDEgKmFmdGVyKiB0aGF0IHJldHVybnMuCj4gCj4gV2UgKm1pZ2h0KiBuZWVkIHRvIGF1 ZGl0IHRoZSBLVk0gdmVjdG9yIGhhbmRsZXJzIHRvIG1ha2Ugc3VyZSB0aGV5J3JlIG5vdAo+IGRl cGVuZGVudCBvbiBSQ1UgcHJvdGVjdGlvbiAoSSBhc3N1bWUgdGhleSdyZSBub3QsIGJ1dCBpdCdz IHBvc3NpYmxlIHNvbWV0aGluZwo+IGhhcyBsZWFrZWQgaW50byB0aGUgVkhFIGNvZGUpLgoKSUlV QyBpbiB0aGUgd2luZG93IGJldHdlZW4gbG9jYWxfaXJxX2VuYWJsZSgpIGFuZCBndWVzdF9leGl0 KCkgYW55IGRyaXZlcgppbnRlcnJ1cHQgbWlnaHQgdHJpZ2dlciwgaXNuJ3QgaXQ/CgpSZWdhcmRz LAoKLS0gCk5pY29sw6FzIFPDoWVuegoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX18Ka3ZtYXJtIG1haWxpbmcgbGlzdAprdm1hcm1AbGlzdHMuY3MuY29sdW1i aWEuZWR1Cmh0dHBzOi8vbGlzdHMuY3MuY29sdW1iaWEuZWR1L21haWxtYW4vbGlzdGluZm8va3Zt YXJtCg==