From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 795AEC00140 for ; Tue, 2 Aug 2022 19:34:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id DAC7350EE8; Tue, 2 Aug 2022 15:34:50 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@linux.dev Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aLwzpvOOtOSW; Tue, 2 Aug 2022 15:34:49 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6D050509E3; Tue, 2 Aug 2022 15:34:49 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 048AA509D1 for ; Tue, 2 Aug 2022 15:34:48 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3ShYu8p8haJY for ; Tue, 2 Aug 2022 15:34:46 -0400 (EDT) Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 69EEC509C1 for ; Tue, 2 Aug 2022 15:34:46 -0400 (EDT) Date: Tue, 2 Aug 2022 12:34:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1659468885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=OCcv/AYbwQMkwRUQMSe1O4XfWOyEUfS7V1bgTTCBT/U=; b=r7kFMl4+sawh0grH7bFq4KgIEfteNExDYsCihhAJ0WRVC8oeVhMVS40GpxT2kwnQwHiWBe YutnWi6wzRgPXNKXesXLJKm9fuvWuaEjAS7W9rV92zbPJ44MJ+FY0N/9+RiBvc/6dHW552 so22xQVEVKGY2OS/atIozlZW+XKIBrU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Alexandru Elisei Subject: Re: KVM/arm64: SPE: Translate VA to IPA on a stage 2 fault instead of pinning VM memory Message-ID: References: <20220419141012.GB6143@willie-the-truck> <20220801170055.GB26471@willie-the-truck> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Cc: maz@kernel.org, Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Hi folks, On Tue, Aug 02, 2022 at 10:49:07AM +0100, Alexandru Elisei wrote: > > > > A funkier approach might be to defer pinning of the buffer until the SPE is > > > > enabled and avoid pinning all of VM memory that way, although I can't > > > > immediately tell how flexible the architecture is in allowing you to cache > > > > the base/limit values. > > > > > > I was investigating this approach, and Mark raised a concern that I think > > > might be a showstopper. > > > > > > Let's consider this scenario: > > > > > > Initial conditions: guest at EL1, profiling disabled (PMBLIMITR_EL1.E = 0, > > > PMBSR_EL1.S = 0, PMSCR_EL1.{E0SPE,E1SPE} = {0,0}). > > > > > > 1. Guest programs the buffer and enables it (PMBLIMITR_EL1.E = 1). > > > 2. Guest programs SPE to enable profiling at **EL0** > > > (PMSCR_EL1.{E0SPE,E1SPE} = {1,0}). > > > 3. Guest changes the translation table entries for the buffer. The > > > architecture allows this. > > > > The architecture also allows MMIO accesses to use writeback addressing > > modes, but it doesn't provide a mechanism to virtualise them sensibly. > > > > So I'd prefer that we don't pin all of guest memory just to satisfy a corner > > case -- as long as the impact of a guest doing this funny sequence is > > constrained to the guest, then I think pinning only what is required is > > probably the most pragmatic approach. > > > > Is it ideal? No, of course not, and we should probably try to get the debug > > architecture extended to be properly virtualisable, but in the meantime > > having major operating systems as guests and being able to use SPE without > > pinning seems like a major design goal to me. > > > > In any case, that's just my thinking on this and I defer to Oliver and > > Marc on the ultimate decision. Thanks for chiming in Will, very much agree that pragmatism is likely the best route forward. While fun to poke at all the pitfalls of virtualizing SPE, pulling tricks in KVM probably has marginal return over a simpler approach. > Thank you for the input. > > To summarize the approaches we've discussed so far: > > 1. Pinning the entire guest memory > - Heavy handed and not ideal. > - Tried this approach in v5 of the SPE series [1], patches #2-#12. > > 2. Mapping the guest SPE buffer on demand, page by page, as a result of stage 2 > faults reported by SPE. > - Not feasible, because the entire contents of the buffer must be discarded is > PMBSR_EL1.DL is set to 1 when taking the fault. > - Requires KVM to walk the guest's stage 1 tables, because SPE reports the VA, > not the IPA. > > 3. Pinning the guest SPE buffer when profiling becomes enabled*: > - There is the corner case described above, when profiling becomes enabled as a > result of an ERET to EL0. This can happen when the buffer is enabled and > PMSCR_EL1.{E0SPE,E1SPE} = {1,0}; > - The previous buffer is unpinned when a new buffer is pinned, to avoid SPE > stage 2 faults when draining the buffer, which is performed with profiling > disabled. > - Also requires KVM to walk the guest's stage 1 tables. > > 4. Pin the entire guest SPE buffer after the first stage 2 fault reported by > SPE. > - Gets rid of the corner case at 3. > - Same approach to buffer unpinning as 3. > - Introduces a blackout window before the first record is written. > - Also requires KVM to walk the guest's stage 1 tables. > > As for the corner case at 3, I proposed either: > > a) Mandate that guest operating systems must never modify the buffer > translation entries if the buffer is enabled and > PMSCR_EL1.{E0SPE,E1SPE} = {1,0}. > > b) Pin the entire buffer as a result of the first stage 2 fault reported by SPE, > but **only** for this corner case. For all other cases, the buffer is pinned > when profiling becomes enabled, to eliminate the blackout window. Guest > operating systems can be modified to not change the translation entries for the > buffer if this blackout window is not desirable. > > Pinning as a result of the **first** stage 2 fault should work, because there > are no prior records that would have to be discarded if PMSBR_EL1.DL = 1. > > I hope I haven't missed anything. Thoughts and suggestions more than welcome. Thanks Alex for pulling together all of the context here. Unless there's any other strong opinions on the topic, it seems to me that option #4 (pin on S2 fault) is probably the best approach for the initial implementation. No amount of tricks in KVM can work around the fact that SPE has some serious issues w.r.t. virtualization. With that, we should probably document the behavior of SPE as a known erratum of KVM. If folks complain about EL1 profile blackout, eagerly pinning when profiling is enabled could layer on top quite easily by treating it as a synthetic S2 fault and triggering the implementation of #4. Having said that I don't believe it is a hard requirement for enabling some flavor of SPE for guests. Walking guest S1 in KVM doesn't sound too exciting although it'll need to be done eventually. Do you feel like this is an OK route forward, or have I missed something? -- Thanks, Oliver _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9C91C19F29 for ; Tue, 2 Aug 2022 19:36:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Hnl4k+b2IwTjU0s8fqdH7McLxtF3vH4h+lrqQnug8Hc=; b=G0NzH9Nsw5V7mr Ka2Av/j8u3GRPmczSYkFe4OiUwQlv7Xy1Ra4+JgGGbllsZpsIqpFWqu8okMZhfliCJXpkoq+Glapo K4QrODSpru/ZBjGLbDMbjc6CgFgZHXT5njFU3v3JhauEgPOnoQx2eH1m7+WAwr8PAWaodEJ3yKx5h c28lQ2cq1C++SAJyhPwwYsQ0MxSg3Wf21jH0P+gJNQ4u1ceHQVU2kwdOM0DaOenhnmbyX4vNW3mze thMqs9rYlAB3x0XSCXpysKbaHJpG0L9M4M4JG8xce1fMAyCDmDEBUV19q74uowLY+6OdXqEQdG5nt kJGWHWMh1P8AbSz0t0iA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oIxfQ-00Gvva-S6; Tue, 02 Aug 2022 19:35:00 +0000 Received: from out2.migadu.com ([188.165.223.204]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oIxfL-00Gvsx-9q for linux-arm-kernel@lists.infradead.org; Tue, 02 Aug 2022 19:34:58 +0000 Date: Tue, 2 Aug 2022 12:34:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1659468885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=OCcv/AYbwQMkwRUQMSe1O4XfWOyEUfS7V1bgTTCBT/U=; b=r7kFMl4+sawh0grH7bFq4KgIEfteNExDYsCihhAJ0WRVC8oeVhMVS40GpxT2kwnQwHiWBe YutnWi6wzRgPXNKXesXLJKm9fuvWuaEjAS7W9rV92zbPJ44MJ+FY0N/9+RiBvc/6dHW552 so22xQVEVKGY2OS/atIozlZW+XKIBrU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Alexandru Elisei Cc: Will Deacon , mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, kvmarm@lists.cs.columbia.edu Subject: Re: KVM/arm64: SPE: Translate VA to IPA on a stage 2 fault instead of pinning VM memory Message-ID: References: <20220419141012.GB6143@willie-the-truck> <20220801170055.GB26471@willie-the-truck> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220802_123456_870999_66C20D3D X-CRM114-Status: GOOD ( 40.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi folks, On Tue, Aug 02, 2022 at 10:49:07AM +0100, Alexandru Elisei wrote: > > > > A funkier approach might be to defer pinning of the buffer until the SPE is > > > > enabled and avoid pinning all of VM memory that way, although I can't > > > > immediately tell how flexible the architecture is in allowing you to cache > > > > the base/limit values. > > > > > > I was investigating this approach, and Mark raised a concern that I think > > > might be a showstopper. > > > > > > Let's consider this scenario: > > > > > > Initial conditions: guest at EL1, profiling disabled (PMBLIMITR_EL1.E = 0, > > > PMBSR_EL1.S = 0, PMSCR_EL1.{E0SPE,E1SPE} = {0,0}). > > > > > > 1. Guest programs the buffer and enables it (PMBLIMITR_EL1.E = 1). > > > 2. Guest programs SPE to enable profiling at **EL0** > > > (PMSCR_EL1.{E0SPE,E1SPE} = {1,0}). > > > 3. Guest changes the translation table entries for the buffer. The > > > architecture allows this. > > > > The architecture also allows MMIO accesses to use writeback addressing > > modes, but it doesn't provide a mechanism to virtualise them sensibly. > > > > So I'd prefer that we don't pin all of guest memory just to satisfy a corner > > case -- as long as the impact of a guest doing this funny sequence is > > constrained to the guest, then I think pinning only what is required is > > probably the most pragmatic approach. > > > > Is it ideal? No, of course not, and we should probably try to get the debug > > architecture extended to be properly virtualisable, but in the meantime > > having major operating systems as guests and being able to use SPE without > > pinning seems like a major design goal to me. > > > > In any case, that's just my thinking on this and I defer to Oliver and > > Marc on the ultimate decision. Thanks for chiming in Will, very much agree that pragmatism is likely the best route forward. While fun to poke at all the pitfalls of virtualizing SPE, pulling tricks in KVM probably has marginal return over a simpler approach. > Thank you for the input. > > To summarize the approaches we've discussed so far: > > 1. Pinning the entire guest memory > - Heavy handed and not ideal. > - Tried this approach in v5 of the SPE series [1], patches #2-#12. > > 2. Mapping the guest SPE buffer on demand, page by page, as a result of stage 2 > faults reported by SPE. > - Not feasible, because the entire contents of the buffer must be discarded is > PMBSR_EL1.DL is set to 1 when taking the fault. > - Requires KVM to walk the guest's stage 1 tables, because SPE reports the VA, > not the IPA. > > 3. Pinning the guest SPE buffer when profiling becomes enabled*: > - There is the corner case described above, when profiling becomes enabled as a > result of an ERET to EL0. This can happen when the buffer is enabled and > PMSCR_EL1.{E0SPE,E1SPE} = {1,0}; > - The previous buffer is unpinned when a new buffer is pinned, to avoid SPE > stage 2 faults when draining the buffer, which is performed with profiling > disabled. > - Also requires KVM to walk the guest's stage 1 tables. > > 4. Pin the entire guest SPE buffer after the first stage 2 fault reported by > SPE. > - Gets rid of the corner case at 3. > - Same approach to buffer unpinning as 3. > - Introduces a blackout window before the first record is written. > - Also requires KVM to walk the guest's stage 1 tables. > > As for the corner case at 3, I proposed either: > > a) Mandate that guest operating systems must never modify the buffer > translation entries if the buffer is enabled and > PMSCR_EL1.{E0SPE,E1SPE} = {1,0}. > > b) Pin the entire buffer as a result of the first stage 2 fault reported by SPE, > but **only** for this corner case. For all other cases, the buffer is pinned > when profiling becomes enabled, to eliminate the blackout window. Guest > operating systems can be modified to not change the translation entries for the > buffer if this blackout window is not desirable. > > Pinning as a result of the **first** stage 2 fault should work, because there > are no prior records that would have to be discarded if PMSBR_EL1.DL = 1. > > I hope I haven't missed anything. Thoughts and suggestions more than welcome. Thanks Alex for pulling together all of the context here. Unless there's any other strong opinions on the topic, it seems to me that option #4 (pin on S2 fault) is probably the best approach for the initial implementation. No amount of tricks in KVM can work around the fact that SPE has some serious issues w.r.t. virtualization. With that, we should probably document the behavior of SPE as a known erratum of KVM. If folks complain about EL1 profile blackout, eagerly pinning when profiling is enabled could layer on top quite easily by treating it as a synthetic S2 fault and triggering the implementation of #4. Having said that I don't believe it is a hard requirement for enabling some flavor of SPE for guests. Walking guest S1 in KVM doesn't sound too exciting although it'll need to be done eventually. Do you feel like this is an OK route forward, or have I missed something? -- Thanks, Oliver _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel