From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E25E8C433EF for ; Sun, 28 Nov 2021 22:17:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:Cc:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Et2Q1yLIK/8GZhcWRyzNBFqlOt1WRKOuGdeT6QEcj4k=; b=NVMuEi/Uo+lvz+ LSXJgDPVztzpKkiQZkX6uS3+CuLs50naMFWOk2HuF78buayMRxEZzveT/3BDtxLRbgxY/qBvfQKXg KbycSpM7dd5Iwc6+ZDfhmSE2aXbTDfRpj1SM7WMKMpcjmEkeNnAFhooD0N7eKhKPDUelWiBk5BUx5 u6QCuFkNRcShIDO4YqTWgvkcwv6GzAv6a9wr9YZ1HtS6AdOudyLYn93PQxkox5b0OGPr0NBl8gKwO IW2p65mQbKM4FPysv0iktOfrUjAs+6lTF7Sib5U+7aLAyCNvbsUW2CqDwMoxKAIcmhXipcqEKT5+h wlioErqIYTEHun/DGiwQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrSTd-00GkXU-E2; Sun, 28 Nov 2021 22:16:53 +0000 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrSTP-00GkWI-SS for linux-riscv@lists.infradead.org; Sun, 28 Nov 2021 22:16:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638137793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4dDgSQ/8TweS/kg7WcZSSnDUpsnRjzcTHNmiNwKkyh0=; b=bAY3cMhEZjqUutmMXPKBehrtXOIt3n3SDAI/hk/o8kttF+IruglmUCRNFI0AgopebADw4N Vm7L0Nj7h53Xke/eJYHtledgLwtvuEouFRCqFWjUZFOSplSTYUquqLbt228/2VVzM+yxmV GYJQbUpLQxZTVNiwtxBHyx4F1mSTShY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-95-8nKb2QCzNQClj-d059dZtw-1; Sun, 28 Nov 2021 17:16:29 -0500 X-MC-Unique: 8nKb2QCzNQClj-d059dZtw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 935911808322; Sun, 28 Nov 2021 22:16:24 +0000 (UTC) Received: from starship (unknown [10.40.192.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0B8E810016F2; Sun, 28 Nov 2021 22:16:02 +0000 (UTC) Message-ID: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful From: Maxim Levitsky To: Sean Christopherson , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Paolo Bonzini Cc: James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Date: Mon, 29 Nov 2021 00:16:01 +0200 In-Reply-To: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211128_141640_180364_97260254 X-CRM114-Status: GOOD ( 41.78 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gV2VkLCAyMDIxLTEwLTI3IGF0IDE2OjQwICswMzAwLCBNYXhpbSBMZXZpdHNreSB3cm90ZToK PiBPbiBGcmksIDIwMjEtMTAtMDggYXQgMTk6MTIgLTA3MDAsIFNlYW4gQ2hyaXN0b3BoZXJzb24g d3JvdGU6Cj4gPiBJbnZva2UgdGhlIGFyY2ggaG9va3MgZm9yIGJsb2NrK3VuYmxvY2sgaWYgYW5k IG9ubHkgaWYgS1ZNIGFjdHVhbGx5Cj4gPiBhdHRlbXB0cyB0byBibG9jayB0aGUgdkNQVS4gIFRo ZSBvbmx5IG5vbi1ub3AgaW1wbGVtZW50YXRpb24gaXMgb24geDg2LAo+ID4gc3BlY2lmaWNhbGx5 IFNWTSdzIEFWSUMsIGFuZCB0aGVyZSBpcyBubyBuZWVkIHRvIHB1dCB0aGUgQVZJQyBwcmlvciB0 bwo+ID4gaGFsdC1wb2xsaW5nIGFzIEtWTSB4ODYncyBrdm1fdmNwdV9oYXNfZXZlbnRzKCkgd2ls bCBzY291ciB0aGUgZnVsbCB2SVJSCj4gPiB0byBmaW5kIHBlbmRpbmcgSVJRcyByZWdhcmRsZXNz IG9mIHdoZXRoZXIgdGhlIEFWSUMgaXMgbG9hZGVkLyJydW5uaW5nIi4KPiA+IAo+ID4gVGhlIHBy aW1hcnkgbW90aXZhdGlvbiBpcyB0byBhbGxvdyBmdXR1cmUgY2xlYW51cCB0byBzcGxpdCBvdXQg ImJsb2NrIgo+ID4gZnJvbSAiaGFsdCIsIGJ1dCB0aGlzIGlzIGFsc28gbGlrZWx5IGEgc21hbGwg cGVyZm9ybWFuY2UgYm9vc3Qgb24geDg2IFNWTQo+ID4gd2hlbiBoYWx0LXBvbGxpbmcgaXMgc3Vj Y2Vzc2Z1bC4KPiA+IAo+ID4gQWRqdXN0IHRoZSBwb3N0LWJsb2NrIHBhdGggdG8gdXBkYXRlICJj dXIiIGFmdGVyIHVuYmxvY2tpbmcsIGkuZS4gaW5jbHVkZQo+ID4gQVZJQyBsb2FkIHRpbWUgaW4g aGFsdF93YWl0X25zIGFuZCBoYWx0X3dhaXRfaGlzdCwgc28gdGhhdCB0aGUgYmVoYXZpb3IKPiA+ IGlzIGNvbnNpc3RlbnQuICBNb3ZpbmcganVzdCB0aGUgcHJlLWJsb2NrIGFyY2ggaG9vayB3b3Vs ZCByZXN1bHQgaW4gb25seQo+ID4gdGhlIEFWSUMgcHV0IGxhdGVuY3kgYmVpbmcgaW5jbHVkZWQg aW4gdGhlIGhhbHRfd2FpdCBzdGF0cy4gIFRoZXJlIGlzIG5vCj4gPiBvYnZpb3VzIGV2aWRlbmNl IHRoYXQgb25lIHdheSBvciB0aGUgb3RoZXIgaXMgY29ycmVjdCwgc28ganVzdCBlbnN1cmUgS1ZN Cj4gPiBpcyBjb25zaXN0ZW50Lgo+ID4gCj4gPiBOb3RlLCB4ODYgaGFzIHR3byBzZXBhcmF0ZSBw YXRocyBmb3IgaGFuZGxpbmcgQVBJQ3Ygd2l0aCByZXNwZWN0IHRvIHZDUFUKPiA+IGJsb2NraW5n LiAgVk1YIHVzZXMgaG9va3MgaW4geDg2J3MgdmNwdV9ibG9jaygpLCB3aGlsZSBTVk0gdXNlcyB0 aGUgYXJjaAo+ID4gaG9va3MgaW4ga3ZtX3ZjcHVfYmxvY2soKS4gIFByaW9yIHRvIHRoaXMgcGF0 aCwgdGhlIHR3byBwYXRocyB3ZXJlIG1vcmUKPiA+IG9yIGxlc3MgZnVuY3Rpb25hbGx5IGlkZW50 aWNhbC4gIFRoYXQgaXMgdmVyeSBtdWNoIG5vdCB0aGUgY2FzZSBhZnRlcgo+ID4gdGhpcyBwYXRj aCwgYXMgdGhlIGhvb2tzIHVzZWQgYnkgVk1YIF9tdXN0XyBmaXJlIGJlZm9yZSBoYWx0LXBvbGxp bmcuCj4gPiB4ODYncyBlbnRpcmUgbWVzcyB3aWxsIGJlIGNsZWFuZWQgdXAgaW4gZnV0dXJlIHBh dGNoZXMuCj4gPiAKPiA+IFNpZ25lZC1vZmYtYnk6IFNlYW4gQ2hyaXN0b3BoZXJzb24gPHNlYW5q Y0Bnb29nbGUuY29tPgo+ID4gLS0tCj4gPiAgdmlydC9rdm0va3ZtX21haW4uYyB8IDcgKysrKy0t LQo+ID4gIDEgZmlsZSBjaGFuZ2VkLCA0IGluc2VydGlvbnMoKyksIDMgZGVsZXRpb25zKC0pCj4g PiAKPiA+IGRpZmYgLS1naXQgYS92aXJ0L2t2bS9rdm1fbWFpbi5jIGIvdmlydC9rdm0va3ZtX21h aW4uYwo+ID4gaW5kZXggZjkwYjNlZDA1NjI4Li4yMjdmNmJiZTA3MTYgMTAwNjQ0Cj4gPiAtLS0g YS92aXJ0L2t2bS9rdm1fbWFpbi5jCj4gPiArKysgYi92aXJ0L2t2bS9rdm1fbWFpbi5jCj4gPiBA QCAtMzIzNSw4ICszMjM1LDYgQEAgdm9pZCBrdm1fdmNwdV9ibG9jayhzdHJ1Y3Qga3ZtX3ZjcHUg KnZjcHUpCj4gPiAgCWJvb2wgd2FpdGVkID0gZmFsc2U7Cj4gPiAgCXU2NCBibG9ja19uczsKPiA+ ICAKPiA+IC0Ja3ZtX2FyY2hfdmNwdV9ibG9ja2luZyh2Y3B1KTsKPiA+IC0KPiA+ICAJc3RhcnQg PSBjdXIgPSBwb2xsX2VuZCA9IGt0aW1lX2dldCgpOwo+ID4gIAlpZiAoZG9faGFsdF9wb2xsKSB7 Cj4gPiAgCQlrdGltZV90IHN0b3AgPSBrdGltZV9hZGRfbnMoa3RpbWVfZ2V0KCksIHZjcHUtPmhh bHRfcG9sbF9ucyk7Cj4gPiBAQCAtMzI1Myw2ICszMjUxLDcgQEAgdm9pZCBrdm1fdmNwdV9ibG9j ayhzdHJ1Y3Qga3ZtX3ZjcHUgKnZjcHUpCj4gPiAgCQl9IHdoaWxlIChrdm1fdmNwdV9jYW5fcG9s bChjdXIsIHN0b3ApKTsKPiA+ICAJfQo+ID4gIAo+ID4gKwlrdm1fYXJjaF92Y3B1X2Jsb2NraW5n KHZjcHUpOwo+ID4gIAo+ID4gIAlwcmVwYXJlX3RvX3JjdXdhaXQod2FpdCk7Cj4gPiAgCWZvciAo OzspIHsKPiA+IEBAIC0zMjY1LDYgKzMyNjQsOSBAQCB2b2lkIGt2bV92Y3B1X2Jsb2NrKHN0cnVj dCBrdm1fdmNwdSAqdmNwdSkKPiA+ICAJCXNjaGVkdWxlKCk7Cj4gPiAgCX0KPiA+ICAJZmluaXNo X3JjdXdhaXQod2FpdCk7Cj4gPiArCj4gPiArCWt2bV9hcmNoX3ZjcHVfdW5ibG9ja2luZyh2Y3B1 KTsKPiA+ICsKPiA+ICAJY3VyID0ga3RpbWVfZ2V0KCk7Cj4gPiAgCWlmICh3YWl0ZWQpIHsKPiA+ ICAJCXZjcHUtPnN0YXQuZ2VuZXJpYy5oYWx0X3dhaXRfbnMgKz0KPiA+IEBAIC0zMjczLDcgKzMy NzUsNiBAQCB2b2lkIGt2bV92Y3B1X2Jsb2NrKHN0cnVjdCBrdm1fdmNwdSAqdmNwdSkKPiA+ICAJ CQkJa3RpbWVfdG9fbnMoY3VyKSAtIGt0aW1lX3RvX25zKHBvbGxfZW5kKSk7Cj4gPiAgCX0KPiA+ ICBvdXQ6Cj4gPiAtCWt2bV9hcmNoX3ZjcHVfdW5ibG9ja2luZyh2Y3B1KTsKPiA+ICAJYmxvY2tf bnMgPSBrdGltZV90b19ucyhjdXIpIC0ga3RpbWVfdG9fbnMoc3RhcnQpOwo+ID4gIAo+ID4gIAkv Kgo+IAo+IE1ha2VzIHNlbnNlLgo+IAo+IFJldmlld2VkLWJ5OiBNYXhpbSBMZXZpdHNreSA8bWxl dml0c2tAcmVkaGF0LmNvbT4KPiAKPiBCZXN0IHJlZ2FyZHMsCj4gCU1heGltIExldml0c2t5CgoK U28uLi4KCkxhc3Qgd2VlayBJIGRlY2lkZWQgdG8gc3R1ZHkgYSBiaXQgaG93IEFWSUMgYmVoYXZl cyB3aGVuIHZDUFVzIGFyZSBub3QgMTAwJSBydW5uaW5nCihha2Egbm8gY3B1X3BtPW9uKSwgdG8g bW9zdGx5IHVuZGVyc3RhbmQgdGhlaXIgc28tY2FsbGVkICdHQSBsb2cnIHRoaW5nLgogCihUaGlz IHRoaW5nIGlzIHRoYXQgd2hlbiB5b3UgdGVsbCB0aGUgSU9NTVUgdGhhdCBhIHZDUFUgaXMgbm90 IHJ1bm5pbmcsCnRoZSBJT01NVSBzdGFydHMgbG9nZ2luZyBhbGwgaW5jb21pbmcgcGFzc2VkLXRo cm91Z2ggaW50ZXJydXB0cyB0byBhIHJpbmcgYnVmZmVyLAphbmQgcmFpc2VzIGl0cyBvd24gaW50 ZXJydXB0LCB3aGljaOKAmXMgaGFuZGxlciBpcyBzdXBwb3NlZCB0byB3YWtlIHVwIHRoZSBWTSdz IHZDUFUuKQogClRoYXQgbGVkIHRvIG1lIGRpc2NvdmVyaW5nIHRoYXQgQU1EJ3MgSU9NTVUgaXMg dG90YWxseSBidXN0ZWQgYWZ0ZXIgYSBzdXNwZW5kL3Jlc3VtZSBjeWNsZSwKZml4aW5nIHdoaWNo IHRvb2sgbWUgZmV3IGRheXMgKGFuZCBtb3N0IG9mIHRoZSB0aW1lIEkgd29ycmllZCB0aGF0IGl0 J3Mgc29tZSBzb3J0IG9mIGEgQklPUyBidWcgd2hpY2ggbm9ib2R5IHdvdWxkIGZpeCwKYXMgdGhl IElPTU1VIGludGVycnVwdCBkZWxpdmVyeSB3YXMgdG90YWxseSBidXN0ZWQgYWZ0ZXIgcmVzdW1l LCBzb21ldGltZXMgZXZlbiBwb3dlciBjeWNsZSBkaWRuJ3QgaGVscAp0byByZXZpdmUgaXQgLSBw aGV3Li4uKS4gCkx1Y2tpbHkgSSBkaWQgZml4IGl0LCBhbmQgcGF0Y2hlcyBhcmUgd2FpdGluZyBm b3IgdGhlIHJldmlldyB1cHN0cmVhbS4KKGh0dHBzOi8vd3d3LnNwaW5pY3MubmV0L2xpc3RzL2tl cm5lbC9tc2c0MTUzNDg4Lmh0bWwpCiAKIApBbm90aGVyIHRoaW5nIEkgZGlzY292ZXJlZCB0aGF0 IHRoaXMgcGF0Y2ggc2VyaWVzIHRvdGFsbHkgYnJlYWtzIG15IFZNcywgd2l0aG91dCBjcHVfcG09 b24KVGhlIHdob2xlIHNlcmllcyAoSSBkaWRuJ3QgeWV0IGJpc2VjdCBpdCkgbWFrZXMgZXZlbiBt eSBmZWRvcmEzMiBWTSBiZSB2ZXJ5IGxhZ2d5LCBhbG1vc3QgdW51c2FibGUsCmFuZCBpdCBvbmx5 IGhhcyBvbmUgcGFzc2VkLXRocm91Z2ggZGV2aWNlLCBhIG5pYykuCiAKSWYgSSBhcHBseSB0aG91 Z2ggb25seSB0aGUgcGF0Y2ggc2VyaWVzIHVwIHRvIHRoaXMgcGF0Y2gsIG15IGZlZG9yYSBWTSBz ZWVtcyB0byB3b3JrIGZpbmUsIGJ1dApteSB3aW5kb3dzIFZNIHN0aWxsIGxvY2tzIHVwIGhhcmQg d2hlbiBJIHJ1biAnTGF0ZW5jeVRvcCcgaW4gaXQsIHdoaWNoIGRvZXNuJ3QgaGFwcGVuIHdpdGhv dXQgdGhpcyBwYXRjaC4KIAogClNvIGZhciB0aGUgc3ltcHRvbXMgSSBzZWUgaXMgdGhhdCBvbiBW Q1BVIDAsIElTUiBoYXMgcXVpdGUgaGlnaCBpbnRlcnJ1cHQgKDB4ZTEgbGFzdCB0aW1lIEkgc2Vl biBpdCksClRQUiBhbmQgUFBSIGFyZSAweGUwIChhbHRob3VnaCBJIGhhdmUgc2VlbiBUUFIgdG8g aGF2ZSBkaWZmZXJlbnQgdmFsdWVzKSwgYW5kIElSUiBoYXMgcGxlbnR5IG9mIGludGVycnVwdHMK d2l0aCBsb3dlciBwcmlvcml0eS4gVGhlIFZNIHNlZW1zIHRvIGJlIHN0dWNrIGluIHRoaXMgY2Fz ZS4gQXMgaWYgaXRzIEVPSSBnb3QgbG9zdCBvciBzb21ldGhpbmcgaXMgcHJldmVudGluZwp0aGUg SVJRIGhhbmRsZXIgZnJvbSBpc3N1aW5nIEVPSS4KIApMYXRlbmN5VG9wIGRvZXMgaW5zdGFsbCBz b21lIGZvcm0gb2YgYSBrZXJuZWwgZHJpdmVyIHdoaWNoIGxpa2VseSBkb2VzIG1lZGRsZSB3aXRo IGludGVycnVwdHMgKG1heWJlIGl0IHNlbmRzIGxvdHMgb2Ygc2VsZiBJUElzPykuCiAKMTAwJSBy ZXByb2R1Y2libGUgYXMgc29vbiBhcyBJIHN0YXJ0IG1vbml0b3Jpbmcgd2l0aCBMYXRlbmN5VG9w LgogCiAKV2l0aG91dCB0aGlzIHBhdGNoIGl0IHdvcmtzIChvciBpZiBkaXNhYmxpbmcgaGFsdCBw b2xsaW5nKSwKIApidXQgSSBzdGlsbCBkaWQgbWFuYWdlIHRvIGxvY2t1cCB0aGUgVk0gZmV3IHRp bWVzIHN0aWxsLCBhZnRlciBsb3Qgb2YgcmFuZG9tIGNsaWNraW5nL3N0YXJ0aW5nIHVwIHZhcmlv dXMgYXBwcyB3aGlsZSBMYXRlbmN5VG9wIHdhcyBydW5uaW5nLApldGMsIGJ1dCBpbiB0aGlzIGNh c2Ugd2hlbiBJIGR1bXAgbG9jYWwgYXBpYyB2aWEgcWVtdSdzIGhtcCBpbnRlcmZhY2UgdGhlIFZN IGluc3RhbnRseSByZXZpdmVzLCB3aGljaCBtaWdodCBiZSBlaXRoZXIgc2FtZSBidWcKd2hpY2gg Z290IGFtcGxpZmllZCBieSB0aGlzIHBhdGNoIG9yIHNvbWV0aGluZyBlbHNlLgpUaGF0IHdhcyB0 ZXN0ZWQgb24gdGhlIHB1cmUgNS4xNS4wIGtlcm5lbCB3aXRob3V0IGFueSBwYXRjaGVzLgogCkl0 IGlzIHBvc3NpYmxlIHRoYXQgdGhpcyBpcyBhIGJ1ZyBpbiBMYXRlbmN5VG9wIHRoYXQganVzdCBn b3QgZXhwb3NlZCBieSBkaWZmZXJlbnQgdGltaW5nLgogClRoZSB3aW5kb3dzIFZNIGRvZXMgaGF2 ZSBHUFUgYW5kIGZldyBVU0IgY29udHJvbGxlcnMgcGFzc2VkIHRvIGl0LCBhbmQgd2l0aG91dCB0 aGVtLCBpbiBwdXJlIFZNIG1vZGUsIGFzIEkgY2FsbCBpdCwKdGhlIExhdGVuY3lUb3Agc2VlbXMg dG8gd29yay4KIAoKVG9tb3Jyb3cgSSdsbCBnaXZlIGl0IGEgbW9yZSBmb3JtYWwgaW52ZXN0aWdh dGlvbi4KIApCZXN0IHJlZ2FyZHMsCglNYXhpbSBMZXZpdHNreQoKCgpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1yaXNjdiBtYWlsaW5nIGxpc3QK bGludXgtcmlzY3ZAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9y Zy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LXJpc2N2Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AF2ADC433F5 for ; Sun, 28 Nov 2021 22:18:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:Cc:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=r4lTStlJm9aT5PWaC+gkr7Ylb6im+12cT8B0+EbJWEM=; b=k74qL394sf2LVw edOXBQZrrM7J9OsE72xXrZMFpm/0l1Lmxz8vyd+VCw4+Sa4fcDlIZ4p/kVSRYFj4eZ8OUJ/GqfRpV tZyDCH96AaI5eMo4H6EErmXY99SK1te+476rSSUp55cAOvBKmR5/v9ri5/zlZ5TlSJ1e6C2WlxC6L Ql4Euke/Gcc+mC5MRCXNhnTJP6X2z4EmETn43dMQCpL+dB/kwry7ryRy7wZpXt51f7X+YXOL6plTm lPrAUlvq9kV0R8yjrkbSxFJOn4F1LPcDPDweZzgEYF7hVfjbgTBkM7jkPXXHS1K68T6nHylQX8E6O XFbvS2N85MCkWVZ+ivLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrSTU-00GkXE-4q; Sun, 28 Nov 2021 22:16:44 +0000 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrSTP-00GkWG-78 for linux-arm-kernel@lists.infradead.org; Sun, 28 Nov 2021 22:16:41 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638137793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4dDgSQ/8TweS/kg7WcZSSnDUpsnRjzcTHNmiNwKkyh0=; b=bAY3cMhEZjqUutmMXPKBehrtXOIt3n3SDAI/hk/o8kttF+IruglmUCRNFI0AgopebADw4N Vm7L0Nj7h53Xke/eJYHtledgLwtvuEouFRCqFWjUZFOSplSTYUquqLbt228/2VVzM+yxmV GYJQbUpLQxZTVNiwtxBHyx4F1mSTShY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-95-8nKb2QCzNQClj-d059dZtw-1; Sun, 28 Nov 2021 17:16:29 -0500 X-MC-Unique: 8nKb2QCzNQClj-d059dZtw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 935911808322; Sun, 28 Nov 2021 22:16:24 +0000 (UTC) Received: from starship (unknown [10.40.192.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0B8E810016F2; Sun, 28 Nov 2021 22:16:02 +0000 (UTC) Message-ID: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful From: Maxim Levitsky To: Sean Christopherson , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Paolo Bonzini Cc: James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Date: Mon, 29 Nov 2021 00:16:01 +0200 In-Reply-To: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211128_141639_407625_23F20335 X-CRM114-Status: GOOD ( 42.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCAyMDIxLTEwLTI3IGF0IDE2OjQwICswMzAwLCBNYXhpbSBMZXZpdHNreSB3cm90ZToK PiBPbiBGcmksIDIwMjEtMTAtMDggYXQgMTk6MTIgLTA3MDAsIFNlYW4gQ2hyaXN0b3BoZXJzb24g d3JvdGU6Cj4gPiBJbnZva2UgdGhlIGFyY2ggaG9va3MgZm9yIGJsb2NrK3VuYmxvY2sgaWYgYW5k IG9ubHkgaWYgS1ZNIGFjdHVhbGx5Cj4gPiBhdHRlbXB0cyB0byBibG9jayB0aGUgdkNQVS4gIFRo ZSBvbmx5IG5vbi1ub3AgaW1wbGVtZW50YXRpb24gaXMgb24geDg2LAo+ID4gc3BlY2lmaWNhbGx5 IFNWTSdzIEFWSUMsIGFuZCB0aGVyZSBpcyBubyBuZWVkIHRvIHB1dCB0aGUgQVZJQyBwcmlvciB0 bwo+ID4gaGFsdC1wb2xsaW5nIGFzIEtWTSB4ODYncyBrdm1fdmNwdV9oYXNfZXZlbnRzKCkgd2ls bCBzY291ciB0aGUgZnVsbCB2SVJSCj4gPiB0byBmaW5kIHBlbmRpbmcgSVJRcyByZWdhcmRsZXNz IG9mIHdoZXRoZXIgdGhlIEFWSUMgaXMgbG9hZGVkLyJydW5uaW5nIi4KPiA+IAo+ID4gVGhlIHBy aW1hcnkgbW90aXZhdGlvbiBpcyB0byBhbGxvdyBmdXR1cmUgY2xlYW51cCB0byBzcGxpdCBvdXQg ImJsb2NrIgo+ID4gZnJvbSAiaGFsdCIsIGJ1dCB0aGlzIGlzIGFsc28gbGlrZWx5IGEgc21hbGwg cGVyZm9ybWFuY2UgYm9vc3Qgb24geDg2IFNWTQo+ID4gd2hlbiBoYWx0LXBvbGxpbmcgaXMgc3Vj Y2Vzc2Z1bC4KPiA+IAo+ID4gQWRqdXN0IHRoZSBwb3N0LWJsb2NrIHBhdGggdG8gdXBkYXRlICJj dXIiIGFmdGVyIHVuYmxvY2tpbmcsIGkuZS4gaW5jbHVkZQo+ID4gQVZJQyBsb2FkIHRpbWUgaW4g aGFsdF93YWl0X25zIGFuZCBoYWx0X3dhaXRfaGlzdCwgc28gdGhhdCB0aGUgYmVoYXZpb3IKPiA+ IGlzIGNvbnNpc3RlbnQuICBNb3ZpbmcganVzdCB0aGUgcHJlLWJsb2NrIGFyY2ggaG9vayB3b3Vs ZCByZXN1bHQgaW4gb25seQo+ID4gdGhlIEFWSUMgcHV0IGxhdGVuY3kgYmVpbmcgaW5jbHVkZWQg aW4gdGhlIGhhbHRfd2FpdCBzdGF0cy4gIFRoZXJlIGlzIG5vCj4gPiBvYnZpb3VzIGV2aWRlbmNl IHRoYXQgb25lIHdheSBvciB0aGUgb3RoZXIgaXMgY29ycmVjdCwgc28ganVzdCBlbnN1cmUgS1ZN Cj4gPiBpcyBjb25zaXN0ZW50Lgo+ID4gCj4gPiBOb3RlLCB4ODYgaGFzIHR3byBzZXBhcmF0ZSBw YXRocyBmb3IgaGFuZGxpbmcgQVBJQ3Ygd2l0aCByZXNwZWN0IHRvIHZDUFUKPiA+IGJsb2NraW5n LiAgVk1YIHVzZXMgaG9va3MgaW4geDg2J3MgdmNwdV9ibG9jaygpLCB3aGlsZSBTVk0gdXNlcyB0 aGUgYXJjaAo+ID4gaG9va3MgaW4ga3ZtX3ZjcHVfYmxvY2soKS4gIFByaW9yIHRvIHRoaXMgcGF0 aCwgdGhlIHR3byBwYXRocyB3ZXJlIG1vcmUKPiA+IG9yIGxlc3MgZnVuY3Rpb25hbGx5IGlkZW50 aWNhbC4gIFRoYXQgaXMgdmVyeSBtdWNoIG5vdCB0aGUgY2FzZSBhZnRlcgo+ID4gdGhpcyBwYXRj aCwgYXMgdGhlIGhvb2tzIHVzZWQgYnkgVk1YIF9tdXN0XyBmaXJlIGJlZm9yZSBoYWx0LXBvbGxp bmcuCj4gPiB4ODYncyBlbnRpcmUgbWVzcyB3aWxsIGJlIGNsZWFuZWQgdXAgaW4gZnV0dXJlIHBh dGNoZXMuCj4gPiAKPiA+IFNpZ25lZC1vZmYtYnk6IFNlYW4gQ2hyaXN0b3BoZXJzb24gPHNlYW5q Y0Bnb29nbGUuY29tPgo+ID4gLS0tCj4gPiAgdmlydC9rdm0va3ZtX21haW4uYyB8IDcgKysrKy0t LQo+ID4gIDEgZmlsZSBjaGFuZ2VkLCA0IGluc2VydGlvbnMoKyksIDMgZGVsZXRpb25zKC0pCj4g PiAKPiA+IGRpZmYgLS1naXQgYS92aXJ0L2t2bS9rdm1fbWFpbi5jIGIvdmlydC9rdm0va3ZtX21h aW4uYwo+ID4gaW5kZXggZjkwYjNlZDA1NjI4Li4yMjdmNmJiZTA3MTYgMTAwNjQ0Cj4gPiAtLS0g YS92aXJ0L2t2bS9rdm1fbWFpbi5jCj4gPiArKysgYi92aXJ0L2t2bS9rdm1fbWFpbi5jCj4gPiBA QCAtMzIzNSw4ICszMjM1LDYgQEAgdm9pZCBrdm1fdmNwdV9ibG9jayhzdHJ1Y3Qga3ZtX3ZjcHUg KnZjcHUpCj4gPiAgCWJvb2wgd2FpdGVkID0gZmFsc2U7Cj4gPiAgCXU2NCBibG9ja19uczsKPiA+ ICAKPiA+IC0Ja3ZtX2FyY2hfdmNwdV9ibG9ja2luZyh2Y3B1KTsKPiA+IC0KPiA+ICAJc3RhcnQg PSBjdXIgPSBwb2xsX2VuZCA9IGt0aW1lX2dldCgpOwo+ID4gIAlpZiAoZG9faGFsdF9wb2xsKSB7 Cj4gPiAgCQlrdGltZV90IHN0b3AgPSBrdGltZV9hZGRfbnMoa3RpbWVfZ2V0KCksIHZjcHUtPmhh bHRfcG9sbF9ucyk7Cj4gPiBAQCAtMzI1Myw2ICszMjUxLDcgQEAgdm9pZCBrdm1fdmNwdV9ibG9j ayhzdHJ1Y3Qga3ZtX3ZjcHUgKnZjcHUpCj4gPiAgCQl9IHdoaWxlIChrdm1fdmNwdV9jYW5fcG9s bChjdXIsIHN0b3ApKTsKPiA+ICAJfQo+ID4gIAo+ID4gKwlrdm1fYXJjaF92Y3B1X2Jsb2NraW5n KHZjcHUpOwo+ID4gIAo+ID4gIAlwcmVwYXJlX3RvX3JjdXdhaXQod2FpdCk7Cj4gPiAgCWZvciAo OzspIHsKPiA+IEBAIC0zMjY1LDYgKzMyNjQsOSBAQCB2b2lkIGt2bV92Y3B1X2Jsb2NrKHN0cnVj dCBrdm1fdmNwdSAqdmNwdSkKPiA+ICAJCXNjaGVkdWxlKCk7Cj4gPiAgCX0KPiA+ICAJZmluaXNo X3JjdXdhaXQod2FpdCk7Cj4gPiArCj4gPiArCWt2bV9hcmNoX3ZjcHVfdW5ibG9ja2luZyh2Y3B1 KTsKPiA+ICsKPiA+ICAJY3VyID0ga3RpbWVfZ2V0KCk7Cj4gPiAgCWlmICh3YWl0ZWQpIHsKPiA+ ICAJCXZjcHUtPnN0YXQuZ2VuZXJpYy5oYWx0X3dhaXRfbnMgKz0KPiA+IEBAIC0zMjczLDcgKzMy NzUsNiBAQCB2b2lkIGt2bV92Y3B1X2Jsb2NrKHN0cnVjdCBrdm1fdmNwdSAqdmNwdSkKPiA+ICAJ CQkJa3RpbWVfdG9fbnMoY3VyKSAtIGt0aW1lX3RvX25zKHBvbGxfZW5kKSk7Cj4gPiAgCX0KPiA+ ICBvdXQ6Cj4gPiAtCWt2bV9hcmNoX3ZjcHVfdW5ibG9ja2luZyh2Y3B1KTsKPiA+ICAJYmxvY2tf bnMgPSBrdGltZV90b19ucyhjdXIpIC0ga3RpbWVfdG9fbnMoc3RhcnQpOwo+ID4gIAo+ID4gIAkv Kgo+IAo+IE1ha2VzIHNlbnNlLgo+IAo+IFJldmlld2VkLWJ5OiBNYXhpbSBMZXZpdHNreSA8bWxl dml0c2tAcmVkaGF0LmNvbT4KPiAKPiBCZXN0IHJlZ2FyZHMsCj4gCU1heGltIExldml0c2t5CgoK U28uLi4KCkxhc3Qgd2VlayBJIGRlY2lkZWQgdG8gc3R1ZHkgYSBiaXQgaG93IEFWSUMgYmVoYXZl cyB3aGVuIHZDUFVzIGFyZSBub3QgMTAwJSBydW5uaW5nCihha2Egbm8gY3B1X3BtPW9uKSwgdG8g bW9zdGx5IHVuZGVyc3RhbmQgdGhlaXIgc28tY2FsbGVkICdHQSBsb2cnIHRoaW5nLgogCihUaGlz IHRoaW5nIGlzIHRoYXQgd2hlbiB5b3UgdGVsbCB0aGUgSU9NTVUgdGhhdCBhIHZDUFUgaXMgbm90 IHJ1bm5pbmcsCnRoZSBJT01NVSBzdGFydHMgbG9nZ2luZyBhbGwgaW5jb21pbmcgcGFzc2VkLXRo cm91Z2ggaW50ZXJydXB0cyB0byBhIHJpbmcgYnVmZmVyLAphbmQgcmFpc2VzIGl0cyBvd24gaW50 ZXJydXB0LCB3aGljaOKAmXMgaGFuZGxlciBpcyBzdXBwb3NlZCB0byB3YWtlIHVwIHRoZSBWTSdz IHZDUFUuKQogClRoYXQgbGVkIHRvIG1lIGRpc2NvdmVyaW5nIHRoYXQgQU1EJ3MgSU9NTVUgaXMg dG90YWxseSBidXN0ZWQgYWZ0ZXIgYSBzdXNwZW5kL3Jlc3VtZSBjeWNsZSwKZml4aW5nIHdoaWNo IHRvb2sgbWUgZmV3IGRheXMgKGFuZCBtb3N0IG9mIHRoZSB0aW1lIEkgd29ycmllZCB0aGF0IGl0 J3Mgc29tZSBzb3J0IG9mIGEgQklPUyBidWcgd2hpY2ggbm9ib2R5IHdvdWxkIGZpeCwKYXMgdGhl IElPTU1VIGludGVycnVwdCBkZWxpdmVyeSB3YXMgdG90YWxseSBidXN0ZWQgYWZ0ZXIgcmVzdW1l LCBzb21ldGltZXMgZXZlbiBwb3dlciBjeWNsZSBkaWRuJ3QgaGVscAp0byByZXZpdmUgaXQgLSBw aGV3Li4uKS4gCkx1Y2tpbHkgSSBkaWQgZml4IGl0LCBhbmQgcGF0Y2hlcyBhcmUgd2FpdGluZyBm b3IgdGhlIHJldmlldyB1cHN0cmVhbS4KKGh0dHBzOi8vd3d3LnNwaW5pY3MubmV0L2xpc3RzL2tl cm5lbC9tc2c0MTUzNDg4Lmh0bWwpCiAKIApBbm90aGVyIHRoaW5nIEkgZGlzY292ZXJlZCB0aGF0 IHRoaXMgcGF0Y2ggc2VyaWVzIHRvdGFsbHkgYnJlYWtzIG15IFZNcywgd2l0aG91dCBjcHVfcG09 b24KVGhlIHdob2xlIHNlcmllcyAoSSBkaWRuJ3QgeWV0IGJpc2VjdCBpdCkgbWFrZXMgZXZlbiBt eSBmZWRvcmEzMiBWTSBiZSB2ZXJ5IGxhZ2d5LCBhbG1vc3QgdW51c2FibGUsCmFuZCBpdCBvbmx5 IGhhcyBvbmUgcGFzc2VkLXRocm91Z2ggZGV2aWNlLCBhIG5pYykuCiAKSWYgSSBhcHBseSB0aG91 Z2ggb25seSB0aGUgcGF0Y2ggc2VyaWVzIHVwIHRvIHRoaXMgcGF0Y2gsIG15IGZlZG9yYSBWTSBz ZWVtcyB0byB3b3JrIGZpbmUsIGJ1dApteSB3aW5kb3dzIFZNIHN0aWxsIGxvY2tzIHVwIGhhcmQg d2hlbiBJIHJ1biAnTGF0ZW5jeVRvcCcgaW4gaXQsIHdoaWNoIGRvZXNuJ3QgaGFwcGVuIHdpdGhv dXQgdGhpcyBwYXRjaC4KIAogClNvIGZhciB0aGUgc3ltcHRvbXMgSSBzZWUgaXMgdGhhdCBvbiBW Q1BVIDAsIElTUiBoYXMgcXVpdGUgaGlnaCBpbnRlcnJ1cHQgKDB4ZTEgbGFzdCB0aW1lIEkgc2Vl biBpdCksClRQUiBhbmQgUFBSIGFyZSAweGUwIChhbHRob3VnaCBJIGhhdmUgc2VlbiBUUFIgdG8g aGF2ZSBkaWZmZXJlbnQgdmFsdWVzKSwgYW5kIElSUiBoYXMgcGxlbnR5IG9mIGludGVycnVwdHMK d2l0aCBsb3dlciBwcmlvcml0eS4gVGhlIFZNIHNlZW1zIHRvIGJlIHN0dWNrIGluIHRoaXMgY2Fz ZS4gQXMgaWYgaXRzIEVPSSBnb3QgbG9zdCBvciBzb21ldGhpbmcgaXMgcHJldmVudGluZwp0aGUg SVJRIGhhbmRsZXIgZnJvbSBpc3N1aW5nIEVPSS4KIApMYXRlbmN5VG9wIGRvZXMgaW5zdGFsbCBz b21lIGZvcm0gb2YgYSBrZXJuZWwgZHJpdmVyIHdoaWNoIGxpa2VseSBkb2VzIG1lZGRsZSB3aXRo IGludGVycnVwdHMgKG1heWJlIGl0IHNlbmRzIGxvdHMgb2Ygc2VsZiBJUElzPykuCiAKMTAwJSBy ZXByb2R1Y2libGUgYXMgc29vbiBhcyBJIHN0YXJ0IG1vbml0b3Jpbmcgd2l0aCBMYXRlbmN5VG9w LgogCiAKV2l0aG91dCB0aGlzIHBhdGNoIGl0IHdvcmtzIChvciBpZiBkaXNhYmxpbmcgaGFsdCBw b2xsaW5nKSwKIApidXQgSSBzdGlsbCBkaWQgbWFuYWdlIHRvIGxvY2t1cCB0aGUgVk0gZmV3IHRp bWVzIHN0aWxsLCBhZnRlciBsb3Qgb2YgcmFuZG9tIGNsaWNraW5nL3N0YXJ0aW5nIHVwIHZhcmlv dXMgYXBwcyB3aGlsZSBMYXRlbmN5VG9wIHdhcyBydW5uaW5nLApldGMsIGJ1dCBpbiB0aGlzIGNh c2Ugd2hlbiBJIGR1bXAgbG9jYWwgYXBpYyB2aWEgcWVtdSdzIGhtcCBpbnRlcmZhY2UgdGhlIFZN IGluc3RhbnRseSByZXZpdmVzLCB3aGljaCBtaWdodCBiZSBlaXRoZXIgc2FtZSBidWcKd2hpY2gg Z290IGFtcGxpZmllZCBieSB0aGlzIHBhdGNoIG9yIHNvbWV0aGluZyBlbHNlLgpUaGF0IHdhcyB0 ZXN0ZWQgb24gdGhlIHB1cmUgNS4xNS4wIGtlcm5lbCB3aXRob3V0IGFueSBwYXRjaGVzLgogCkl0 IGlzIHBvc3NpYmxlIHRoYXQgdGhpcyBpcyBhIGJ1ZyBpbiBMYXRlbmN5VG9wIHRoYXQganVzdCBn b3QgZXhwb3NlZCBieSBkaWZmZXJlbnQgdGltaW5nLgogClRoZSB3aW5kb3dzIFZNIGRvZXMgaGF2 ZSBHUFUgYW5kIGZldyBVU0IgY29udHJvbGxlcnMgcGFzc2VkIHRvIGl0LCBhbmQgd2l0aG91dCB0 aGVtLCBpbiBwdXJlIFZNIG1vZGUsIGFzIEkgY2FsbCBpdCwKdGhlIExhdGVuY3lUb3Agc2VlbXMg dG8gd29yay4KIAoKVG9tb3Jyb3cgSSdsbCBnaXZlIGl0IGEgbW9yZSBmb3JtYWwgaW52ZXN0aWdh dGlvbi4KIApCZXN0IHJlZ2FyZHMsCglNYXhpbSBMZXZpdHNreQoKCgpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVsIG1haWxpbmcg bGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmlu ZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55AF8C433FE for ; Sun, 28 Nov 2021 22:18:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242169AbhK1WVv (ORCPT ); Sun, 28 Nov 2021 17:21:51 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:40620 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230122AbhK1WTu (ORCPT ); Sun, 28 Nov 2021 17:19:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638137792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4dDgSQ/8TweS/kg7WcZSSnDUpsnRjzcTHNmiNwKkyh0=; b=hBzaokjeQgrg8fhwWn0/f7oLf+E0Q7B6ZQYVORMZO9MQukQ2WlBlt4OD9xParZ/XOh8TVv g58zcdJ1vbyOtQw7J3X9qdLvWnM6XWBprWwnbc5s1XeYbkBM4xZK65Ms9q7qHNEXLg6pLO PU6lrEZNUUmGA88PElCPdqsvFx7xQlI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-95-8nKb2QCzNQClj-d059dZtw-1; Sun, 28 Nov 2021 17:16:29 -0500 X-MC-Unique: 8nKb2QCzNQClj-d059dZtw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 935911808322; Sun, 28 Nov 2021 22:16:24 +0000 (UTC) Received: from starship (unknown [10.40.192.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0B8E810016F2; Sun, 28 Nov 2021 22:16:02 +0000 (UTC) Message-ID: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful From: Maxim Levitsky To: Sean Christopherson , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Paolo Bonzini Cc: James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Date: Mon, 29 Nov 2021 00:16:01 +0200 In-Reply-To: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2021-10-27 at 16:40 +0300, Maxim Levitsky wrote: > On Fri, 2021-10-08 at 19:12 -0700, Sean Christopherson wrote: > > Invoke the arch hooks for block+unblock if and only if KVM actually > > attempts to block the vCPU. The only non-nop implementation is on x86, > > specifically SVM's AVIC, and there is no need to put the AVIC prior to > > halt-polling as KVM x86's kvm_vcpu_has_events() will scour the full vIRR > > to find pending IRQs regardless of whether the AVIC is loaded/"running". > > > > The primary motivation is to allow future cleanup to split out "block" > > from "halt", but this is also likely a small performance boost on x86 SVM > > when halt-polling is successful. > > > > Adjust the post-block path to update "cur" after unblocking, i.e. include > > AVIC load time in halt_wait_ns and halt_wait_hist, so that the behavior > > is consistent. Moving just the pre-block arch hook would result in only > > the AVIC put latency being included in the halt_wait stats. There is no > > obvious evidence that one way or the other is correct, so just ensure KVM > > is consistent. > > > > Note, x86 has two separate paths for handling APICv with respect to vCPU > > blocking. VMX uses hooks in x86's vcpu_block(), while SVM uses the arch > > hooks in kvm_vcpu_block(). Prior to this path, the two paths were more > > or less functionally identical. That is very much not the case after > > this patch, as the hooks used by VMX _must_ fire before halt-polling. > > x86's entire mess will be cleaned up in future patches. > > > > Signed-off-by: Sean Christopherson > > --- > > virt/kvm/kvm_main.c | 7 ++++--- > > 1 file changed, 4 insertions(+), 3 deletions(-) > > > > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c > > index f90b3ed05628..227f6bbe0716 100644 > > --- a/virt/kvm/kvm_main.c > > +++ b/virt/kvm/kvm_main.c > > @@ -3235,8 +3235,6 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) > > bool waited = false; > > u64 block_ns; > > > > - kvm_arch_vcpu_blocking(vcpu); > > - > > start = cur = poll_end = ktime_get(); > > if (do_halt_poll) { > > ktime_t stop = ktime_add_ns(ktime_get(), vcpu->halt_poll_ns); > > @@ -3253,6 +3251,7 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) > > } while (kvm_vcpu_can_poll(cur, stop)); > > } > > > > + kvm_arch_vcpu_blocking(vcpu); > > > > prepare_to_rcuwait(wait); > > for (;;) { > > @@ -3265,6 +3264,9 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) > > schedule(); > > } > > finish_rcuwait(wait); > > + > > + kvm_arch_vcpu_unblocking(vcpu); > > + > > cur = ktime_get(); > > if (waited) { > > vcpu->stat.generic.halt_wait_ns += > > @@ -3273,7 +3275,6 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) > > ktime_to_ns(cur) - ktime_to_ns(poll_end)); > > } > > out: > > - kvm_arch_vcpu_unblocking(vcpu); > > block_ns = ktime_to_ns(cur) - ktime_to_ns(start); > > > > /* > > Makes sense. > > Reviewed-by: Maxim Levitsky > > Best regards, > Maxim Levitsky So... Last week I decided to study a bit how AVIC behaves when vCPUs are not 100% running (aka no cpu_pm=on), to mostly understand their so-called 'GA log' thing. (This thing is that when you tell the IOMMU that a vCPU is not running, the IOMMU starts logging all incoming passed-through interrupts to a ring buffer, and raises its own interrupt, which’s handler is supposed to wake up the VM's vCPU.) That led to me discovering that AMD's IOMMU is totally busted after a suspend/resume cycle, fixing which took me few days (and most of the time I worried that it's some sort of a BIOS bug which nobody would fix, as the IOMMU interrupt delivery was totally busted after resume, sometimes even power cycle didn't help to revive it - phew...). Luckily I did fix it, and patches are waiting for the review upstream. (https://www.spinics.net/lists/kernel/msg4153488.html) Another thing I discovered that this patch series totally breaks my VMs, without cpu_pm=on The whole series (I didn't yet bisect it) makes even my fedora32 VM be very laggy, almost unusable, and it only has one passed-through device, a nic). If I apply though only the patch series up to this patch, my fedora VM seems to work fine, but my windows VM still locks up hard when I run 'LatencyTop' in it, which doesn't happen without this patch. So far the symptoms I see is that on VCPU 0, ISR has quite high interrupt (0xe1 last time I seen it), TPR and PPR are 0xe0 (although I have seen TPR to have different values), and IRR has plenty of interrupts with lower priority. The VM seems to be stuck in this case. As if its EOI got lost or something is preventing the IRQ handler from issuing EOI. LatencyTop does install some form of a kernel driver which likely does meddle with interrupts (maybe it sends lots of self IPIs?). 100% reproducible as soon as I start monitoring with LatencyTop. Without this patch it works (or if disabling halt polling), but I still did manage to lockup the VM few times still, after lot of random clicking/starting up various apps while LatencyTop was running, etc, but in this case when I dump local apic via qemu's hmp interface the VM instantly revives, which might be either same bug which got amplified by this patch or something else. That was tested on the pure 5.15.0 kernel without any patches. It is possible that this is a bug in LatencyTop that just got exposed by different timing. The windows VM does have GPU and few USB controllers passed to it, and without them, in pure VM mode, as I call it, the LatencyTop seems to work. Tomorrow I'll give it a more formal investigation. Best regards, Maxim Levitsky From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07DD5C433EF for ; Mon, 29 Nov 2021 08:41:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6F1154B1E2; Mon, 29 Nov 2021 03:41:52 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@redhat.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5Eh3E7Cq8hD1; Mon, 29 Nov 2021 03:41:50 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id AC98D4B217; Mon, 29 Nov 2021 03:41:50 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id D55514B27C for ; Sun, 28 Nov 2021 17:16:34 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YP-9RClh6IAP for ; Sun, 28 Nov 2021 17:16:33 -0500 (EST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 563714B25A for ; Sun, 28 Nov 2021 17:16:33 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638137793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4dDgSQ/8TweS/kg7WcZSSnDUpsnRjzcTHNmiNwKkyh0=; b=bAY3cMhEZjqUutmMXPKBehrtXOIt3n3SDAI/hk/o8kttF+IruglmUCRNFI0AgopebADw4N Vm7L0Nj7h53Xke/eJYHtledgLwtvuEouFRCqFWjUZFOSplSTYUquqLbt228/2VVzM+yxmV GYJQbUpLQxZTVNiwtxBHyx4F1mSTShY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-95-8nKb2QCzNQClj-d059dZtw-1; Sun, 28 Nov 2021 17:16:29 -0500 X-MC-Unique: 8nKb2QCzNQClj-d059dZtw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 935911808322; Sun, 28 Nov 2021 22:16:24 +0000 (UTC) Received: from starship (unknown [10.40.192.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0B8E810016F2; Sun, 28 Nov 2021 22:16:02 +0000 (UTC) Message-ID: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful From: Maxim Levitsky To: Sean Christopherson , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Paolo Bonzini Date: Mon, 29 Nov 2021 00:16:01 +0200 In-Reply-To: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Mailman-Approved-At: Mon, 29 Nov 2021 03:41:49 -0500 Cc: Wanpeng Li , kvm@vger.kernel.org, David Hildenbrand , linux-kernel@vger.kernel.org, Atish Patra , linux-riscv@lists.infradead.org, Claudio Imbrenda , kvmarm@lists.cs.columbia.edu, Joerg Roedel , kvm-ppc@vger.kernel.org, David Matlack , linux-arm-kernel@lists.infradead.org, Jim Mattson , Cornelia Huck , linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, Vitaly Kuznetsov X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu T24gV2VkLCAyMDIxLTEwLTI3IGF0IDE2OjQwICswMzAwLCBNYXhpbSBMZXZpdHNreSB3cm90ZToK PiBPbiBGcmksIDIwMjEtMTAtMDggYXQgMTk6MTIgLTA3MDAsIFNlYW4gQ2hyaXN0b3BoZXJzb24g d3JvdGU6Cj4gPiBJbnZva2UgdGhlIGFyY2ggaG9va3MgZm9yIGJsb2NrK3VuYmxvY2sgaWYgYW5k IG9ubHkgaWYgS1ZNIGFjdHVhbGx5Cj4gPiBhdHRlbXB0cyB0byBibG9jayB0aGUgdkNQVS4gIFRo ZSBvbmx5IG5vbi1ub3AgaW1wbGVtZW50YXRpb24gaXMgb24geDg2LAo+ID4gc3BlY2lmaWNhbGx5 IFNWTSdzIEFWSUMsIGFuZCB0aGVyZSBpcyBubyBuZWVkIHRvIHB1dCB0aGUgQVZJQyBwcmlvciB0 bwo+ID4gaGFsdC1wb2xsaW5nIGFzIEtWTSB4ODYncyBrdm1fdmNwdV9oYXNfZXZlbnRzKCkgd2ls bCBzY291ciB0aGUgZnVsbCB2SVJSCj4gPiB0byBmaW5kIHBlbmRpbmcgSVJRcyByZWdhcmRsZXNz IG9mIHdoZXRoZXIgdGhlIEFWSUMgaXMgbG9hZGVkLyJydW5uaW5nIi4KPiA+IAo+ID4gVGhlIHBy aW1hcnkgbW90aXZhdGlvbiBpcyB0byBhbGxvdyBmdXR1cmUgY2xlYW51cCB0byBzcGxpdCBvdXQg ImJsb2NrIgo+ID4gZnJvbSAiaGFsdCIsIGJ1dCB0aGlzIGlzIGFsc28gbGlrZWx5IGEgc21hbGwg cGVyZm9ybWFuY2UgYm9vc3Qgb24geDg2IFNWTQo+ID4gd2hlbiBoYWx0LXBvbGxpbmcgaXMgc3Vj Y2Vzc2Z1bC4KPiA+IAo+ID4gQWRqdXN0IHRoZSBwb3N0LWJsb2NrIHBhdGggdG8gdXBkYXRlICJj dXIiIGFmdGVyIHVuYmxvY2tpbmcsIGkuZS4gaW5jbHVkZQo+ID4gQVZJQyBsb2FkIHRpbWUgaW4g aGFsdF93YWl0X25zIGFuZCBoYWx0X3dhaXRfaGlzdCwgc28gdGhhdCB0aGUgYmVoYXZpb3IKPiA+ IGlzIGNvbnNpc3RlbnQuICBNb3ZpbmcganVzdCB0aGUgcHJlLWJsb2NrIGFyY2ggaG9vayB3b3Vs ZCByZXN1bHQgaW4gb25seQo+ID4gdGhlIEFWSUMgcHV0IGxhdGVuY3kgYmVpbmcgaW5jbHVkZWQg aW4gdGhlIGhhbHRfd2FpdCBzdGF0cy4gIFRoZXJlIGlzIG5vCj4gPiBvYnZpb3VzIGV2aWRlbmNl IHRoYXQgb25lIHdheSBvciB0aGUgb3RoZXIgaXMgY29ycmVjdCwgc28ganVzdCBlbnN1cmUgS1ZN Cj4gPiBpcyBjb25zaXN0ZW50Lgo+ID4gCj4gPiBOb3RlLCB4ODYgaGFzIHR3byBzZXBhcmF0ZSBw YXRocyBmb3IgaGFuZGxpbmcgQVBJQ3Ygd2l0aCByZXNwZWN0IHRvIHZDUFUKPiA+IGJsb2NraW5n LiAgVk1YIHVzZXMgaG9va3MgaW4geDg2J3MgdmNwdV9ibG9jaygpLCB3aGlsZSBTVk0gdXNlcyB0 aGUgYXJjaAo+ID4gaG9va3MgaW4ga3ZtX3ZjcHVfYmxvY2soKS4gIFByaW9yIHRvIHRoaXMgcGF0 aCwgdGhlIHR3byBwYXRocyB3ZXJlIG1vcmUKPiA+IG9yIGxlc3MgZnVuY3Rpb25hbGx5IGlkZW50 aWNhbC4gIFRoYXQgaXMgdmVyeSBtdWNoIG5vdCB0aGUgY2FzZSBhZnRlcgo+ID4gdGhpcyBwYXRj aCwgYXMgdGhlIGhvb2tzIHVzZWQgYnkgVk1YIF9tdXN0XyBmaXJlIGJlZm9yZSBoYWx0LXBvbGxp bmcuCj4gPiB4ODYncyBlbnRpcmUgbWVzcyB3aWxsIGJlIGNsZWFuZWQgdXAgaW4gZnV0dXJlIHBh dGNoZXMuCj4gPiAKPiA+IFNpZ25lZC1vZmYtYnk6IFNlYW4gQ2hyaXN0b3BoZXJzb24gPHNlYW5q Y0Bnb29nbGUuY29tPgo+ID4gLS0tCj4gPiAgdmlydC9rdm0va3ZtX21haW4uYyB8IDcgKysrKy0t LQo+ID4gIDEgZmlsZSBjaGFuZ2VkLCA0IGluc2VydGlvbnMoKyksIDMgZGVsZXRpb25zKC0pCj4g PiAKPiA+IGRpZmYgLS1naXQgYS92aXJ0L2t2bS9rdm1fbWFpbi5jIGIvdmlydC9rdm0va3ZtX21h aW4uYwo+ID4gaW5kZXggZjkwYjNlZDA1NjI4Li4yMjdmNmJiZTA3MTYgMTAwNjQ0Cj4gPiAtLS0g YS92aXJ0L2t2bS9rdm1fbWFpbi5jCj4gPiArKysgYi92aXJ0L2t2bS9rdm1fbWFpbi5jCj4gPiBA QCAtMzIzNSw4ICszMjM1LDYgQEAgdm9pZCBrdm1fdmNwdV9ibG9jayhzdHJ1Y3Qga3ZtX3ZjcHUg KnZjcHUpCj4gPiAgCWJvb2wgd2FpdGVkID0gZmFsc2U7Cj4gPiAgCXU2NCBibG9ja19uczsKPiA+ ICAKPiA+IC0Ja3ZtX2FyY2hfdmNwdV9ibG9ja2luZyh2Y3B1KTsKPiA+IC0KPiA+ICAJc3RhcnQg PSBjdXIgPSBwb2xsX2VuZCA9IGt0aW1lX2dldCgpOwo+ID4gIAlpZiAoZG9faGFsdF9wb2xsKSB7 Cj4gPiAgCQlrdGltZV90IHN0b3AgPSBrdGltZV9hZGRfbnMoa3RpbWVfZ2V0KCksIHZjcHUtPmhh bHRfcG9sbF9ucyk7Cj4gPiBAQCAtMzI1Myw2ICszMjUxLDcgQEAgdm9pZCBrdm1fdmNwdV9ibG9j ayhzdHJ1Y3Qga3ZtX3ZjcHUgKnZjcHUpCj4gPiAgCQl9IHdoaWxlIChrdm1fdmNwdV9jYW5fcG9s bChjdXIsIHN0b3ApKTsKPiA+ICAJfQo+ID4gIAo+ID4gKwlrdm1fYXJjaF92Y3B1X2Jsb2NraW5n KHZjcHUpOwo+ID4gIAo+ID4gIAlwcmVwYXJlX3RvX3JjdXdhaXQod2FpdCk7Cj4gPiAgCWZvciAo OzspIHsKPiA+IEBAIC0zMjY1LDYgKzMyNjQsOSBAQCB2b2lkIGt2bV92Y3B1X2Jsb2NrKHN0cnVj dCBrdm1fdmNwdSAqdmNwdSkKPiA+ICAJCXNjaGVkdWxlKCk7Cj4gPiAgCX0KPiA+ICAJZmluaXNo X3JjdXdhaXQod2FpdCk7Cj4gPiArCj4gPiArCWt2bV9hcmNoX3ZjcHVfdW5ibG9ja2luZyh2Y3B1 KTsKPiA+ICsKPiA+ICAJY3VyID0ga3RpbWVfZ2V0KCk7Cj4gPiAgCWlmICh3YWl0ZWQpIHsKPiA+ ICAJCXZjcHUtPnN0YXQuZ2VuZXJpYy5oYWx0X3dhaXRfbnMgKz0KPiA+IEBAIC0zMjczLDcgKzMy NzUsNiBAQCB2b2lkIGt2bV92Y3B1X2Jsb2NrKHN0cnVjdCBrdm1fdmNwdSAqdmNwdSkKPiA+ICAJ CQkJa3RpbWVfdG9fbnMoY3VyKSAtIGt0aW1lX3RvX25zKHBvbGxfZW5kKSk7Cj4gPiAgCX0KPiA+ ICBvdXQ6Cj4gPiAtCWt2bV9hcmNoX3ZjcHVfdW5ibG9ja2luZyh2Y3B1KTsKPiA+ICAJYmxvY2tf bnMgPSBrdGltZV90b19ucyhjdXIpIC0ga3RpbWVfdG9fbnMoc3RhcnQpOwo+ID4gIAo+ID4gIAkv Kgo+IAo+IE1ha2VzIHNlbnNlLgo+IAo+IFJldmlld2VkLWJ5OiBNYXhpbSBMZXZpdHNreSA8bWxl dml0c2tAcmVkaGF0LmNvbT4KPiAKPiBCZXN0IHJlZ2FyZHMsCj4gCU1heGltIExldml0c2t5CgoK U28uLi4KCkxhc3Qgd2VlayBJIGRlY2lkZWQgdG8gc3R1ZHkgYSBiaXQgaG93IEFWSUMgYmVoYXZl cyB3aGVuIHZDUFVzIGFyZSBub3QgMTAwJSBydW5uaW5nCihha2Egbm8gY3B1X3BtPW9uKSwgdG8g bW9zdGx5IHVuZGVyc3RhbmQgdGhlaXIgc28tY2FsbGVkICdHQSBsb2cnIHRoaW5nLgogCihUaGlz IHRoaW5nIGlzIHRoYXQgd2hlbiB5b3UgdGVsbCB0aGUgSU9NTVUgdGhhdCBhIHZDUFUgaXMgbm90 IHJ1bm5pbmcsCnRoZSBJT01NVSBzdGFydHMgbG9nZ2luZyBhbGwgaW5jb21pbmcgcGFzc2VkLXRo cm91Z2ggaW50ZXJydXB0cyB0byBhIHJpbmcgYnVmZmVyLAphbmQgcmFpc2VzIGl0cyBvd24gaW50 ZXJydXB0LCB3aGljaOKAmXMgaGFuZGxlciBpcyBzdXBwb3NlZCB0byB3YWtlIHVwIHRoZSBWTSdz IHZDUFUuKQogClRoYXQgbGVkIHRvIG1lIGRpc2NvdmVyaW5nIHRoYXQgQU1EJ3MgSU9NTVUgaXMg dG90YWxseSBidXN0ZWQgYWZ0ZXIgYSBzdXNwZW5kL3Jlc3VtZSBjeWNsZSwKZml4aW5nIHdoaWNo IHRvb2sgbWUgZmV3IGRheXMgKGFuZCBtb3N0IG9mIHRoZSB0aW1lIEkgd29ycmllZCB0aGF0IGl0 J3Mgc29tZSBzb3J0IG9mIGEgQklPUyBidWcgd2hpY2ggbm9ib2R5IHdvdWxkIGZpeCwKYXMgdGhl IElPTU1VIGludGVycnVwdCBkZWxpdmVyeSB3YXMgdG90YWxseSBidXN0ZWQgYWZ0ZXIgcmVzdW1l LCBzb21ldGltZXMgZXZlbiBwb3dlciBjeWNsZSBkaWRuJ3QgaGVscAp0byByZXZpdmUgaXQgLSBw aGV3Li4uKS4gCkx1Y2tpbHkgSSBkaWQgZml4IGl0LCBhbmQgcGF0Y2hlcyBhcmUgd2FpdGluZyBm b3IgdGhlIHJldmlldyB1cHN0cmVhbS4KKGh0dHBzOi8vd3d3LnNwaW5pY3MubmV0L2xpc3RzL2tl cm5lbC9tc2c0MTUzNDg4Lmh0bWwpCiAKIApBbm90aGVyIHRoaW5nIEkgZGlzY292ZXJlZCB0aGF0 IHRoaXMgcGF0Y2ggc2VyaWVzIHRvdGFsbHkgYnJlYWtzIG15IFZNcywgd2l0aG91dCBjcHVfcG09 b24KVGhlIHdob2xlIHNlcmllcyAoSSBkaWRuJ3QgeWV0IGJpc2VjdCBpdCkgbWFrZXMgZXZlbiBt eSBmZWRvcmEzMiBWTSBiZSB2ZXJ5IGxhZ2d5LCBhbG1vc3QgdW51c2FibGUsCmFuZCBpdCBvbmx5 IGhhcyBvbmUgcGFzc2VkLXRocm91Z2ggZGV2aWNlLCBhIG5pYykuCiAKSWYgSSBhcHBseSB0aG91 Z2ggb25seSB0aGUgcGF0Y2ggc2VyaWVzIHVwIHRvIHRoaXMgcGF0Y2gsIG15IGZlZG9yYSBWTSBz ZWVtcyB0byB3b3JrIGZpbmUsIGJ1dApteSB3aW5kb3dzIFZNIHN0aWxsIGxvY2tzIHVwIGhhcmQg d2hlbiBJIHJ1biAnTGF0ZW5jeVRvcCcgaW4gaXQsIHdoaWNoIGRvZXNuJ3QgaGFwcGVuIHdpdGhv dXQgdGhpcyBwYXRjaC4KIAogClNvIGZhciB0aGUgc3ltcHRvbXMgSSBzZWUgaXMgdGhhdCBvbiBW Q1BVIDAsIElTUiBoYXMgcXVpdGUgaGlnaCBpbnRlcnJ1cHQgKDB4ZTEgbGFzdCB0aW1lIEkgc2Vl biBpdCksClRQUiBhbmQgUFBSIGFyZSAweGUwIChhbHRob3VnaCBJIGhhdmUgc2VlbiBUUFIgdG8g aGF2ZSBkaWZmZXJlbnQgdmFsdWVzKSwgYW5kIElSUiBoYXMgcGxlbnR5IG9mIGludGVycnVwdHMK d2l0aCBsb3dlciBwcmlvcml0eS4gVGhlIFZNIHNlZW1zIHRvIGJlIHN0dWNrIGluIHRoaXMgY2Fz ZS4gQXMgaWYgaXRzIEVPSSBnb3QgbG9zdCBvciBzb21ldGhpbmcgaXMgcHJldmVudGluZwp0aGUg SVJRIGhhbmRsZXIgZnJvbSBpc3N1aW5nIEVPSS4KIApMYXRlbmN5VG9wIGRvZXMgaW5zdGFsbCBz b21lIGZvcm0gb2YgYSBrZXJuZWwgZHJpdmVyIHdoaWNoIGxpa2VseSBkb2VzIG1lZGRsZSB3aXRo IGludGVycnVwdHMgKG1heWJlIGl0IHNlbmRzIGxvdHMgb2Ygc2VsZiBJUElzPykuCiAKMTAwJSBy ZXByb2R1Y2libGUgYXMgc29vbiBhcyBJIHN0YXJ0IG1vbml0b3Jpbmcgd2l0aCBMYXRlbmN5VG9w LgogCiAKV2l0aG91dCB0aGlzIHBhdGNoIGl0IHdvcmtzIChvciBpZiBkaXNhYmxpbmcgaGFsdCBw b2xsaW5nKSwKIApidXQgSSBzdGlsbCBkaWQgbWFuYWdlIHRvIGxvY2t1cCB0aGUgVk0gZmV3IHRp bWVzIHN0aWxsLCBhZnRlciBsb3Qgb2YgcmFuZG9tIGNsaWNraW5nL3N0YXJ0aW5nIHVwIHZhcmlv dXMgYXBwcyB3aGlsZSBMYXRlbmN5VG9wIHdhcyBydW5uaW5nLApldGMsIGJ1dCBpbiB0aGlzIGNh c2Ugd2hlbiBJIGR1bXAgbG9jYWwgYXBpYyB2aWEgcWVtdSdzIGhtcCBpbnRlcmZhY2UgdGhlIFZN IGluc3RhbnRseSByZXZpdmVzLCB3aGljaCBtaWdodCBiZSBlaXRoZXIgc2FtZSBidWcKd2hpY2gg Z290IGFtcGxpZmllZCBieSB0aGlzIHBhdGNoIG9yIHNvbWV0aGluZyBlbHNlLgpUaGF0IHdhcyB0 ZXN0ZWQgb24gdGhlIHB1cmUgNS4xNS4wIGtlcm5lbCB3aXRob3V0IGFueSBwYXRjaGVzLgogCkl0 IGlzIHBvc3NpYmxlIHRoYXQgdGhpcyBpcyBhIGJ1ZyBpbiBMYXRlbmN5VG9wIHRoYXQganVzdCBn b3QgZXhwb3NlZCBieSBkaWZmZXJlbnQgdGltaW5nLgogClRoZSB3aW5kb3dzIFZNIGRvZXMgaGF2 ZSBHUFUgYW5kIGZldyBVU0IgY29udHJvbGxlcnMgcGFzc2VkIHRvIGl0LCBhbmQgd2l0aG91dCB0 aGVtLCBpbiBwdXJlIFZNIG1vZGUsIGFzIEkgY2FsbCBpdCwKdGhlIExhdGVuY3lUb3Agc2VlbXMg dG8gd29yay4KIAoKVG9tb3Jyb3cgSSdsbCBnaXZlIGl0IGEgbW9yZSBmb3JtYWwgaW52ZXN0aWdh dGlvbi4KIApCZXN0IHJlZ2FyZHMsCglNYXhpbSBMZXZpdHNreQoKCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmt2bWFybSBtYWlsaW5nIGxpc3QKa3ZtYXJt QGxpc3RzLmNzLmNvbHVtYmlhLmVkdQpodHRwczovL2xpc3RzLmNzLmNvbHVtYmlhLmVkdS9tYWls bWFuL2xpc3RpbmZvL2t2bWFybQo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: Maxim Levitsky Date: Sun, 28 Nov 2021 22:16:01 +0000 Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful Message-Id: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> List-Id: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable To: Sean Christopherson , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Paolo Bonzini Cc: James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang On Wed, 2021-10-27 at 16:40 +0300, Maxim Levitsky wrote: > On Fri, 2021-10-08 at 19:12 -0700, Sean Christopherson wrote: > > Invoke the arch hooks for block+unblock if and only if KVM actually > > attempts to block the vCPU. The only non-nop implementation is on x86, > > specifically SVM's AVIC, and there is no need to put the AVIC prior to > > halt-polling as KVM x86's kvm_vcpu_has_events() will scour the full vIRR > > to find pending IRQs regardless of whether the AVIC is loaded/"running". > >=20 > > The primary motivation is to allow future cleanup to split out "block" > > from "halt", but this is also likely a small performance boost on x86 S= VM > > when halt-polling is successful. > >=20 > > Adjust the post-block path to update "cur" after unblocking, i.e. inclu= de > > AVIC load time in halt_wait_ns and halt_wait_hist, so that the behavior > > is consistent. Moving just the pre-block arch hook would result in only > > the AVIC put latency being included in the halt_wait stats. There is no > > obvious evidence that one way or the other is correct, so just ensure K= VM > > is consistent. > >=20 > > Note, x86 has two separate paths for handling APICv with respect to vCPU > > blocking. VMX uses hooks in x86's vcpu_block(), while SVM uses the arch > > hooks in kvm_vcpu_block(). Prior to this path, the two paths were more > > or less functionally identical. That is very much not the case after > > this patch, as the hooks used by VMX _must_ fire before halt-polling. > > x86's entire mess will be cleaned up in future patches. > >=20 > > Signed-off-by: Sean Christopherson > > --- > > virt/kvm/kvm_main.c | 7 ++++--- > > 1 file changed, 4 insertions(+), 3 deletions(-) > >=20 > > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c > > index f90b3ed05628..227f6bbe0716 100644 > > --- a/virt/kvm/kvm_main.c > > +++ b/virt/kvm/kvm_main.c > > @@ -3235,8 +3235,6 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) > > bool waited =3D false; > > u64 block_ns; > > =20 > > - kvm_arch_vcpu_blocking(vcpu); > > - > > start =3D cur =3D poll_end =3D ktime_get(); > > if (do_halt_poll) { > > ktime_t stop =3D ktime_add_ns(ktime_get(), vcpu->halt_poll_ns); > > @@ -3253,6 +3251,7 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) > > } while (kvm_vcpu_can_poll(cur, stop)); > > } > > =20 > > + kvm_arch_vcpu_blocking(vcpu); > > =20 > > prepare_to_rcuwait(wait); > > for (;;) { > > @@ -3265,6 +3264,9 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) > > schedule(); > > } > > finish_rcuwait(wait); > > + > > + kvm_arch_vcpu_unblocking(vcpu); > > + > > cur =3D ktime_get(); > > if (waited) { > > vcpu->stat.generic.halt_wait_ns +> > @@ -3273,7 +3275,6 @@ void kvm_= vcpu_block(struct kvm_vcpu *vcpu) > > ktime_to_ns(cur) - ktime_to_ns(poll_end)); > > } > > out: > > - kvm_arch_vcpu_unblocking(vcpu); > > block_ns =3D ktime_to_ns(cur) - ktime_to_ns(start); > > =20 > > /* >=20 > Makes sense. >=20 > Reviewed-by: Maxim Levitsky >=20 > Best regards, > Maxim Levitsky So... Last week I decided to study a bit how AVIC behaves when vCPUs are not 100%= running (aka no cpu_pm=3Don), to mostly understand their so-called 'GA log' thing. =20 (This thing is that when you tell the IOMMU that a vCPU is not running, the IOMMU starts logging all incoming passed-through interrupts to a ring b= uffer, and raises its own interrupt, which=E2=80=99s handler is supposed to wake u= p the VM's vCPU.) =20 That led to me discovering that AMD's IOMMU is totally busted after a suspe= nd/resume cycle, fixing which took me few days (and most of the time I worried that it's som= e sort of a BIOS bug which nobody would fix, as the IOMMU interrupt delivery was totally busted after resume, sometimes = even power cycle didn't help to revive it - phew...).=20 Luckily I did fix it, and patches are waiting for the review upstream. (https://www.spinics.net/lists/kernel/msg4153488.html) =20 =20 Another thing I discovered that this patch series totally breaks my VMs, wi= thout cpu_pm=3Don The whole series (I didn't yet bisect it) makes even my fedora32 VM be very= laggy, almost unusable, and it only has one passed-through device, a nic). =20 If I apply though only the patch series up to this patch, my fedora VM seem= s to work fine, but my windows VM still locks up hard when I run 'LatencyTop' in it, which does= n't happen without this patch. =20 =20 So far the symptoms I see is that on VCPU 0, ISR has quite high interrupt (= 0xe1 last time I seen it), TPR and PPR are 0xe0 (although I have seen TPR to have different values), a= nd IRR has plenty of interrupts with lower priority. The VM seems to be stuck in this case. As if its EOI g= ot lost or something is preventing the IRQ handler from issuing EOI. =20 LatencyTop does install some form of a kernel driver which likely does medd= le with interrupts (maybe it sends lots of self IPIs?). =20 100% reproducible as soon as I start monitoring with LatencyTop. =20 =20 Without this patch it works (or if disabling halt polling), =20 but I still did manage to lockup the VM few times still, after lot of rando= m clicking/starting up various apps while LatencyTop was running, etc, but in this case when I dump local apic via qemu's hmp interface the V= M instantly revives, which might be either same bug which got amplified by this patch or something else. That was tested on the pure 5.15.0 kernel without any patches. =20 It is possible that this is a bug in LatencyTop that just got exposed by di= fferent timing. =20 The windows VM does have GPU and few USB controllers passed to it, and with= out them, in pure VM mode, as I call it, the LatencyTop seems to work. =20 Tomorrow I'll give it a more formal investigation. =20 Best regards, Maxim Levitsky