From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA75EC0650E for ; Mon, 1 Jul 2019 09:17:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 892E52184C for ; Mon, 1 Jul 2019 09:17:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728347AbfGAJRS (ORCPT ); Mon, 1 Jul 2019 05:17:18 -0400 Received: from foss.arm.com ([217.140.110.172]:58256 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728073AbfGAJRR (ORCPT ); Mon, 1 Jul 2019 05:17:17 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D1A8BCFC; Mon, 1 Jul 2019 02:17:16 -0700 (PDT) Received: from arrakis.emea.arm.com (arrakis.cambridge.arm.com [10.1.196.78]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 62CC23F718; Mon, 1 Jul 2019 02:17:14 -0700 (PDT) Date: Mon, 1 Jul 2019 10:17:12 +0100 From: Catalin Marinas To: Guo Ren Cc: Julien Grall , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, aou@eecs.berkeley.edu, gary@garyguo.net, Atish Patra , hch@infradead.org, paul.walmsley@sifive.com, rppt@linux.ibm.com, linux-riscv@lists.infradead.org, Anup Patel , Palmer Dabbelt , suzuki.poulose@arm.com, Marc Zyngier , julien.thierry@arm.com, Will Deacon , christoffer.dall@arm.com, james.morse@arm.com Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file Message-ID: <20190701091711.GA21774@arrakis.emea.arm.com> References: <20190321163623.20219-1-julien.grall@arm.com> <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190621141606.GF18954@arrakis.emea.arm.com> <20190624153820.GH29120@arrakis.emea.arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, Jun 30, 2019 at 12:29:46PM +0800, Guo Ren wrote: > On Mon, Jun 24, 2019 at 11:38 PM Catalin Marinas > wrote: > > On Mon, Jun 24, 2019 at 12:35:35AM +0800, Guo Ren wrote: > > > On Fri, Jun 21, 2019 at 10:16 PM Catalin Marinas > > > wrote: > > > > BTW, if you find the algorithm fairly straightforward ;), see this > > > > bug-fix which took a formal model to identify: a8ffaaa060b8 ("arm64: > > > > asid: Do not replace active_asids if already 0"). > > [...] > > > Btw, Is this detected by arm's aisd allocator TLA+ model ? Or a real > > > bug report ? > > > > This specific bug was found by the TLA+ model checker (at the time we > > were actually tracking down another bug with multi-threaded CPU sharing > > the TLB, bug also confirmed by the formal model). > > Could you tell me the ref-link about "another bug with multi-threaded > CPU sharing the TLB" ? > > In my concept, the multi-core asid mechanism is also applicable to > multi-thread shared TLB, but it will generate redundant tlbflush. From > the software design logic, multi-threaded is treated as multi-cores > without error, but performance is not optimized. >From the ASID reservation/allocation perspective, the mechanism is the same between multi-threaded with a shared TLB and multi-core. On arm64, a local_flush_tlb_all() on a thread invalidates the TLB for the other threads of the same core. The actual problem with multi-threaded CPUs is a lot more subtle. Digging some internal email from 1.5 years ago and pasting it below (where "current ASID algorithm" refers to the one prior to the fix and CnP - Common Not Private - means shared TLBs on a multi-threaded CPU): The current ASID roll-over algorithm allows for a small window where active_asids for a CPU (P1) is different from the actual ASID in TTBR0. This can lead to a roll-over on a different CPU (P2) allocating an ASID (for a different task) which is still hardware-active on P1. A TLBI on a CPU (or a peer CPU with CnP) does not guarantee that all the entries corresponding to a valid TTBRx are removed as they can still be speculatively loaded immediately after TLBI. While having two different page tables with the same ASID on different CPUs should be fine without CnP, it becomes problematic when CnP is enabled: P1 P2 -- -- TTBR0.BADDR = T1 TTBR0.ASID = A1 check_and_switch_context(T2,A2) asid_maps[P1] = A2 goto fastpath check_and_switch_context(T3,A0) new_context ASID roll-over allocates A1 since it is not active TLBI ALL speculate TTBR0.ASID = A1 entry TTBR0.BADDR = T3 TTBR0.ASID = A1 TTBR0.BADDR = T2 TTBR0.ASID = A2 After this, the common TLB on P1 and P2 (CnP) contains entries corresponding to the old T1 and A1. Task T3 using the same ASID A1 can hit such entries. (T1,A1) will eventually be removed from the TLB on the next context switch on P1 since tlb_flush_pending was set but this is not guaranteed to happen. The fix on arm64 (as part of 5ffdfaedfa0a - "arm64: mm: Support Common Not Private translations") was to set the reserved TTBR0 in check_and_switch_context(), preventing speculative loads into the TLB being tagged with the wrong ASID. So this is specific to the ARM CPUs behaviour w.r.t. speculative TLB loads, it may not be the case (yet) for your architecture. -- Catalin From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91B27C0650E for ; Mon, 1 Jul 2019 09:19:51 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 69E2420659 for ; Mon, 1 Jul 2019 09:19:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="OVlc+EyD" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69E2420659 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BLXS53tl6EH1ksKlvZsF2BDZMdQs2Ms3+tsBnRb0NWw=; b=OVlc+EyDSeWhpn 3WC4eVKgQqyQYR08Ll111RLqtVT21ZMRoRY+YVvvCzJ+tLYUta5qqPaTAeQr61hfnLvjXP7ydZDY4 2XFNCQ7x8DPeuDGncUA02pRP+9uz3e7jI71UV7PqAsLlkzJbYk36ZTIK3rDRYk+fSWxyJRWqplN85 k4o3HvEGtxKkDZ6Uv07vES+DWddWUUSXc1PrUOARc+3CwRNxzGnHsHsgCpsf4r4BTfoJ9QcyiegJ4 RQuj/4WJuSncRXNsutSTFfkYdwX7rSJMuzCI2LYckSQYVJUJ33dZxxdnyddhGsnCWs2V3Yckl6mV1 2cp6L1vrVUejlHyLcruQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hhsTS-00013C-Cq; Mon, 01 Jul 2019 09:19:46 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hhsR5-0007VD-Cc; Mon, 01 Jul 2019 09:17:22 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D1A8BCFC; Mon, 1 Jul 2019 02:17:16 -0700 (PDT) Received: from arrakis.emea.arm.com (arrakis.cambridge.arm.com [10.1.196.78]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 62CC23F718; Mon, 1 Jul 2019 02:17:14 -0700 (PDT) Date: Mon, 1 Jul 2019 10:17:12 +0100 From: Catalin Marinas To: Guo Ren Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file Message-ID: <20190701091711.GA21774@arrakis.emea.arm.com> References: <20190321163623.20219-1-julien.grall@arm.com> <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190621141606.GF18954@arrakis.emea.arm.com> <20190624153820.GH29120@arrakis.emea.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190701_021719_604931_A002A64C X-CRM114-Status: GOOD ( 20.74 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: julien.thierry@arm.com, aou@eecs.berkeley.edu, james.morse@arm.com, suzuki.poulose@arm.com, Marc Zyngier , Anup Patel , Will Deacon , linux-kernel@vger.kernel.org, rppt@linux.ibm.com, hch@infradead.org, Atish Patra , Julien Grall , Palmer Dabbelt , gary@garyguo.net, paul.walmsley@sifive.com, christoffer.dall@arm.com, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org On Sun, Jun 30, 2019 at 12:29:46PM +0800, Guo Ren wrote: > On Mon, Jun 24, 2019 at 11:38 PM Catalin Marinas > wrote: > > On Mon, Jun 24, 2019 at 12:35:35AM +0800, Guo Ren wrote: > > > On Fri, Jun 21, 2019 at 10:16 PM Catalin Marinas > > > wrote: > > > > BTW, if you find the algorithm fairly straightforward ;), see this > > > > bug-fix which took a formal model to identify: a8ffaaa060b8 ("arm64: > > > > asid: Do not replace active_asids if already 0"). > > [...] > > > Btw, Is this detected by arm's aisd allocator TLA+ model ? Or a real > > > bug report ? > > > > This specific bug was found by the TLA+ model checker (at the time we > > were actually tracking down another bug with multi-threaded CPU sharing > > the TLB, bug also confirmed by the formal model). > > Could you tell me the ref-link about "another bug with multi-threaded > CPU sharing the TLB" ? > > In my concept, the multi-core asid mechanism is also applicable to > multi-thread shared TLB, but it will generate redundant tlbflush. From > the software design logic, multi-threaded is treated as multi-cores > without error, but performance is not optimized. >From the ASID reservation/allocation perspective, the mechanism is the same between multi-threaded with a shared TLB and multi-core. On arm64, a local_flush_tlb_all() on a thread invalidates the TLB for the other threads of the same core. The actual problem with multi-threaded CPUs is a lot more subtle. Digging some internal email from 1.5 years ago and pasting it below (where "current ASID algorithm" refers to the one prior to the fix and CnP - Common Not Private - means shared TLBs on a multi-threaded CPU): The current ASID roll-over algorithm allows for a small window where active_asids for a CPU (P1) is different from the actual ASID in TTBR0. This can lead to a roll-over on a different CPU (P2) allocating an ASID (for a different task) which is still hardware-active on P1. A TLBI on a CPU (or a peer CPU with CnP) does not guarantee that all the entries corresponding to a valid TTBRx are removed as they can still be speculatively loaded immediately after TLBI. While having two different page tables with the same ASID on different CPUs should be fine without CnP, it becomes problematic when CnP is enabled: P1 P2 -- -- TTBR0.BADDR = T1 TTBR0.ASID = A1 check_and_switch_context(T2,A2) asid_maps[P1] = A2 goto fastpath check_and_switch_context(T3,A0) new_context ASID roll-over allocates A1 since it is not active TLBI ALL speculate TTBR0.ASID = A1 entry TTBR0.BADDR = T3 TTBR0.ASID = A1 TTBR0.BADDR = T2 TTBR0.ASID = A2 After this, the common TLB on P1 and P2 (CnP) contains entries corresponding to the old T1 and A1. Task T3 using the same ASID A1 can hit such entries. (T1,A1) will eventually be removed from the TLB on the next context switch on P1 since tlb_flush_pending was set but this is not guaranteed to happen. The fix on arm64 (as part of 5ffdfaedfa0a - "arm64: mm: Support Common Not Private translations") was to set the reserved TTBR0 in check_and_switch_context(), preventing speculative loads into the TLB being tagged with the wrong ASID. So this is specific to the ARM CPUs behaviour w.r.t. speculative TLB loads, it may not be the case (yet) for your architecture. -- Catalin _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5639AC0650E for ; Mon, 1 Jul 2019 09:17:22 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id E437B213F2 for ; Mon, 1 Jul 2019 09:17:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E437B213F2 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 5D0654A389; Mon, 1 Jul 2019 05:17:21 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sYCgHbbYO55E; Mon, 1 Jul 2019 05:17:20 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 034854A418; Mon, 1 Jul 2019 05:17:20 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id BAC8A4A379 for ; Mon, 1 Jul 2019 05:17:18 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fBlJWbId+uah for ; Mon, 1 Jul 2019 05:17:17 -0400 (EDT) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 36BEB4A36B for ; Mon, 1 Jul 2019 05:17:17 -0400 (EDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D1A8BCFC; Mon, 1 Jul 2019 02:17:16 -0700 (PDT) Received: from arrakis.emea.arm.com (arrakis.cambridge.arm.com [10.1.196.78]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 62CC23F718; Mon, 1 Jul 2019 02:17:14 -0700 (PDT) Date: Mon, 1 Jul 2019 10:17:12 +0100 From: Catalin Marinas To: Guo Ren Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file Message-ID: <20190701091711.GA21774@arrakis.emea.arm.com> References: <20190321163623.20219-1-julien.grall@arm.com> <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190621141606.GF18954@arrakis.emea.arm.com> <20190624153820.GH29120@arrakis.emea.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Cc: aou@eecs.berkeley.edu, Marc Zyngier , Anup Patel , Will Deacon , linux-kernel@vger.kernel.org, rppt@linux.ibm.com, hch@infradead.org, Atish Patra , Julien Grall , Palmer Dabbelt , gary@garyguo.net, paul.walmsley@sifive.com, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Sun, Jun 30, 2019 at 12:29:46PM +0800, Guo Ren wrote: > On Mon, Jun 24, 2019 at 11:38 PM Catalin Marinas > wrote: > > On Mon, Jun 24, 2019 at 12:35:35AM +0800, Guo Ren wrote: > > > On Fri, Jun 21, 2019 at 10:16 PM Catalin Marinas > > > wrote: > > > > BTW, if you find the algorithm fairly straightforward ;), see this > > > > bug-fix which took a formal model to identify: a8ffaaa060b8 ("arm64: > > > > asid: Do not replace active_asids if already 0"). > > [...] > > > Btw, Is this detected by arm's aisd allocator TLA+ model ? Or a real > > > bug report ? > > > > This specific bug was found by the TLA+ model checker (at the time we > > were actually tracking down another bug with multi-threaded CPU sharing > > the TLB, bug also confirmed by the formal model). > > Could you tell me the ref-link about "another bug with multi-threaded > CPU sharing the TLB" ? > > In my concept, the multi-core asid mechanism is also applicable to > multi-thread shared TLB, but it will generate redundant tlbflush. From > the software design logic, multi-threaded is treated as multi-cores > without error, but performance is not optimized. >From the ASID reservation/allocation perspective, the mechanism is the same between multi-threaded with a shared TLB and multi-core. On arm64, a local_flush_tlb_all() on a thread invalidates the TLB for the other threads of the same core. The actual problem with multi-threaded CPUs is a lot more subtle. Digging some internal email from 1.5 years ago and pasting it below (where "current ASID algorithm" refers to the one prior to the fix and CnP - Common Not Private - means shared TLBs on a multi-threaded CPU): The current ASID roll-over algorithm allows for a small window where active_asids for a CPU (P1) is different from the actual ASID in TTBR0. This can lead to a roll-over on a different CPU (P2) allocating an ASID (for a different task) which is still hardware-active on P1. A TLBI on a CPU (or a peer CPU with CnP) does not guarantee that all the entries corresponding to a valid TTBRx are removed as they can still be speculatively loaded immediately after TLBI. While having two different page tables with the same ASID on different CPUs should be fine without CnP, it becomes problematic when CnP is enabled: P1 P2 -- -- TTBR0.BADDR = T1 TTBR0.ASID = A1 check_and_switch_context(T2,A2) asid_maps[P1] = A2 goto fastpath check_and_switch_context(T3,A0) new_context ASID roll-over allocates A1 since it is not active TLBI ALL speculate TTBR0.ASID = A1 entry TTBR0.BADDR = T3 TTBR0.ASID = A1 TTBR0.BADDR = T2 TTBR0.ASID = A2 After this, the common TLB on P1 and P2 (CnP) contains entries corresponding to the old T1 and A1. Task T3 using the same ASID A1 can hit such entries. (T1,A1) will eventually be removed from the TLB on the next context switch on P1 since tlb_flush_pending was set but this is not guaranteed to happen. The fix on arm64 (as part of 5ffdfaedfa0a - "arm64: mm: Support Common Not Private translations") was to set the reserved TTBR0 in check_and_switch_context(), preventing speculative loads into the TLB being tagged with the wrong ASID. So this is specific to the ARM CPUs behaviour w.r.t. speculative TLB loads, it may not be the case (yet) for your architecture. -- Catalin _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7F42C0650F for ; Mon, 1 Jul 2019 09:19:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BC29820659 for ; Mon, 1 Jul 2019 09:19:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="IEkbDnk7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BC29820659 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0C59CvrqDXm4rtAuLnB0RAD0/QSH51TwDIX8uXrsAyg=; b=IEkbDnk7f9zzQy St1wr2Hj9iH1vR7p5nDWfOViacbcarpaNzfX8Uqo7hMYpeYERM18qYuOAzPPPnuGP6+O8niB7I7nD iNLUvXMilkJXFDz0W2EqThSBtVOyrqteCMyfFJqH8TesDLAzm1m+uJYJ+jYsxawRCHxd8wxVeVRDN 2zOrl8UTG9ImKXolLgPCX1T9plVSOqCMNFsTuUVgsabpTBJJHhMSg8c0LIZLoasazXXInVXI8aGbN TsJihpTAK2S/caVnocAnwsOBZkbnUD31esTiJdKSCfzV3nrUbyMHqN2PIqJIFUcb71w3tfb5RFCje 6F+9L0/wAnhx1r+Aq4OA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hhsTE-0000qf-5Y; Mon, 01 Jul 2019 09:19:32 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hhsR5-0007VD-Cc; Mon, 01 Jul 2019 09:17:22 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D1A8BCFC; Mon, 1 Jul 2019 02:17:16 -0700 (PDT) Received: from arrakis.emea.arm.com (arrakis.cambridge.arm.com [10.1.196.78]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 62CC23F718; Mon, 1 Jul 2019 02:17:14 -0700 (PDT) Date: Mon, 1 Jul 2019 10:17:12 +0100 From: Catalin Marinas To: Guo Ren Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file Message-ID: <20190701091711.GA21774@arrakis.emea.arm.com> References: <20190321163623.20219-1-julien.grall@arm.com> <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190621141606.GF18954@arrakis.emea.arm.com> <20190624153820.GH29120@arrakis.emea.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190701_021719_604931_A002A64C X-CRM114-Status: GOOD ( 20.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: julien.thierry@arm.com, aou@eecs.berkeley.edu, james.morse@arm.com, suzuki.poulose@arm.com, Marc Zyngier , Anup Patel , Will Deacon , linux-kernel@vger.kernel.org, rppt@linux.ibm.com, hch@infradead.org, Atish Patra , Julien Grall , Palmer Dabbelt , gary@garyguo.net, paul.walmsley@sifive.com, christoffer.dall@arm.com, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Sun, Jun 30, 2019 at 12:29:46PM +0800, Guo Ren wrote: > On Mon, Jun 24, 2019 at 11:38 PM Catalin Marinas > wrote: > > On Mon, Jun 24, 2019 at 12:35:35AM +0800, Guo Ren wrote: > > > On Fri, Jun 21, 2019 at 10:16 PM Catalin Marinas > > > wrote: > > > > BTW, if you find the algorithm fairly straightforward ;), see this > > > > bug-fix which took a formal model to identify: a8ffaaa060b8 ("arm64: > > > > asid: Do not replace active_asids if already 0"). > > [...] > > > Btw, Is this detected by arm's aisd allocator TLA+ model ? Or a real > > > bug report ? > > > > This specific bug was found by the TLA+ model checker (at the time we > > were actually tracking down another bug with multi-threaded CPU sharing > > the TLB, bug also confirmed by the formal model). > > Could you tell me the ref-link about "another bug with multi-threaded > CPU sharing the TLB" ? > > In my concept, the multi-core asid mechanism is also applicable to > multi-thread shared TLB, but it will generate redundant tlbflush. From > the software design logic, multi-threaded is treated as multi-cores > without error, but performance is not optimized. >From the ASID reservation/allocation perspective, the mechanism is the same between multi-threaded with a shared TLB and multi-core. On arm64, a local_flush_tlb_all() on a thread invalidates the TLB for the other threads of the same core. The actual problem with multi-threaded CPUs is a lot more subtle. Digging some internal email from 1.5 years ago and pasting it below (where "current ASID algorithm" refers to the one prior to the fix and CnP - Common Not Private - means shared TLBs on a multi-threaded CPU): The current ASID roll-over algorithm allows for a small window where active_asids for a CPU (P1) is different from the actual ASID in TTBR0. This can lead to a roll-over on a different CPU (P2) allocating an ASID (for a different task) which is still hardware-active on P1. A TLBI on a CPU (or a peer CPU with CnP) does not guarantee that all the entries corresponding to a valid TTBRx are removed as they can still be speculatively loaded immediately after TLBI. While having two different page tables with the same ASID on different CPUs should be fine without CnP, it becomes problematic when CnP is enabled: P1 P2 -- -- TTBR0.BADDR = T1 TTBR0.ASID = A1 check_and_switch_context(T2,A2) asid_maps[P1] = A2 goto fastpath check_and_switch_context(T3,A0) new_context ASID roll-over allocates A1 since it is not active TLBI ALL speculate TTBR0.ASID = A1 entry TTBR0.BADDR = T3 TTBR0.ASID = A1 TTBR0.BADDR = T2 TTBR0.ASID = A2 After this, the common TLB on P1 and P2 (CnP) contains entries corresponding to the old T1 and A1. Task T3 using the same ASID A1 can hit such entries. (T1,A1) will eventually be removed from the TLB on the next context switch on P1 since tlb_flush_pending was set but this is not guaranteed to happen. The fix on arm64 (as part of 5ffdfaedfa0a - "arm64: mm: Support Common Not Private translations") was to set the reserved TTBR0 in check_and_switch_context(), preventing speculative loads into the TLB being tagged with the wrong ASID. So this is specific to the ARM CPUs behaviour w.r.t. speculative TLB loads, it may not be the case (yet) for your architecture. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel