From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9772FC4CEC4 for ; Thu, 19 Sep 2019 13:07:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6508721924 for ; Thu, 19 Sep 2019 13:07:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568898451; bh=xDlkbYtekNq2g+j/zLZRiaXJuHeVfYxFX/p6ICWFSTw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:List-ID:From; b=umBWHBuXgj5EJWHiyf6gp/WkNLXEz+Ic1k5MjdoA8F5brgBKhBK39ieSU2W0UsV3w eJBHApCE8J860pcJzzT1fKkKQvoUj9uScbylVBnzOw4AL0n6o//SYm5KBNkaWo7UHV HeggrIQrMhuFiO7GDDwFxWhlSOIfcoYrEVH/ouQU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732259AbfISNHa (ORCPT ); Thu, 19 Sep 2019 09:07:30 -0400 Received: from mail.kernel.org ([198.145.29.99]:51932 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731332AbfISNHa (ORCPT ); Thu, 19 Sep 2019 09:07:30 -0400 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 643CF21D79 for ; Thu, 19 Sep 2019 13:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568898448; bh=xDlkbYtekNq2g+j/zLZRiaXJuHeVfYxFX/p6ICWFSTw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=BCHVqSuL/6xTcR4UIQNbftn+i7ALTmzjbkxcwXg9kdHIXUeW5wKOJj0jG8SGA29kT rlFF6uog75GxK0bdFaRZNI0ABOEfr0Om7lb2CRZPfxIDL3GXQYc0LdqrbSwCsSMul1 OvFGEKKeZ74rerDPYJ6ZKN6QBZCubXMaIdO0VgrU= Received: by mail-wr1-f46.google.com with SMTP id r3so3028521wrj.6 for ; Thu, 19 Sep 2019 06:07:28 -0700 (PDT) X-Gm-Message-State: APjAAAXHPl0m8kNYX8Ex3qEL/RaAJ+oHgVzkJkbhtMVKZXo+EkEuZ1Cf RlBBXutW0jESZlAMhaD+1RYwRKnm0SxaQKgiRTw= X-Google-Smtp-Source: APXvYqyNvo9nMD/j9PGGbZY5fbWhu9DS32hN7clZh4PtaLsYsLNB9DQGeR8UyO6Q34SnbEzf8UHVp/ztmfjTiqTh1Ok= X-Received: by 2002:a5d:66d2:: with SMTP id k18mr7066721wrw.7.1568898446854; Thu, 19 Sep 2019 06:07:26 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> In-Reply-To: <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> From: Guo Ren Date: Thu, 19 Sep 2019 21:07:15 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Jean-Philippe Brucker Cc: Will Deacon , Catalin Marinas , Palmer Dabbelt , christoffer.dall@arm.com, Atish Patra , Julien Grall , gary@garyguo.net, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, Mike Rapoport , aou@eecs.berkeley.edu, Arnd Bergmann , suzuki.poulose@arm.com, Marc Zyngier , Paul Walmsley , linux-arm-kernel@lists.infradead.org, Anup Patel , Linux Kernel Mailing List , iommu@lists.linux-foundation.org, james.morse@arm.com Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On Mon, Sep 16, 2019 at 8:57 PM Jean-Philippe Brucker wrote: > On 13/09/2019 09:13, Guo Ren wrote: > > Another idea is seperate remote TLB invalidate into two instructions: > > > > - sfence.vma.b.asyc > > - sfence.vma.b.barrier // wait all async TLB invalidate operations > > finished for all harts. > > It's not clear to me how this helps, but I probably don't have the whole > picture. If you have a place where it is safe to wait for the barrier to > complete, why not do the whole invalidate there? > > > (I remember who mentioned me separate them into two instructions after > > session. Anup? Is the idea right ?) Forget it, I still use irq signal in my formal proposal [1]. I also couldn't image the whole picture :P > > To solve the problem, we could define a async mode in sfence.vma.b to > > slove the problem and finished with per_cpu_irq/exception. > > The solution I had to this problem is pinning the ASID [1] used by the > IOMMU, to prevent the CPU from recycling the ASID on rollover. This way > the CPU doesn't have to wait for IOMMU invalidations to complete, when > scheduling a task that might not even have anything to do with the IOMMU. > > In the Arm SMMU, ASID and IOASID (PASID) are separate identifiers. IOASID > indexes an entry in the context descriptor table, which contains the ASID. > So with unpinned shared ASID you don't need to invalidate the ATC on > rollover, since the IOASID doesn't change, but you do need to modify the > context descriptor and invalidate cached versions of it. The terminology confused me a lot. I perfer use PASID for IOMMU and ASID is for CPU. Arm's entry of the context descriptor table contains a "IOASID" IOASID != ASID for CPU_TLB and IOMMU_TLB. When you say "since the IOASID doesn't change",Is it PASID or my IOASID ? -_*! PASID in PCI-sig was used to determine transfer address space. For intel, the entry which is indexed by PASID also contain S1/S2.PGD and DID(VMID). For arm, the entry which is indexed by PASID only contain S1.PGD and IOASID. Compare to Intel Vt-d Scalable mode, arm's design can't support PCI Virtual Function. > > Once you have pinned ASIDs, you could also declare that IOASID = ASID. I > don't remember finding an argument to strictly forbid it, even though ASID > and IOASID have different sizes on Arm (respectively 8/16 and 20 bits). ASID and IOASID are hard to keep the same between CPU system and IOMMU system. So I introduce S1/S2.PGD.PPN as a bridge between CPUs and IOMMUs. See my proposal [1] 1: https://lore.kernel.org/linux-csky/1568896556-28769-1-git-send-email-guoren@kernel.org/T/#u -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C1C3C3A5A6 for ; Thu, 19 Sep 2019 13:07:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3C3D321907 for ; Thu, 19 Sep 2019 13:07:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ASE+VOqo"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="BCHVqSuL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3C3D321907 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6MnhFVwkWfH4pXgxTsr+cclWOzAAB23NvNlVWv6fhdc=; b=ASE+VOqoMABTHG mIvGzCVWEruxCPJIpwb/KHbAScyZ59dzxwIJcSSZB+Y6ghfSUWsnvKir3IMthzV3civ4fITrpNyPt kgUDkT6/CBHuRCP4BxCiJJT3junsfG+bQ1zglqXjvOKrm2Y2IappX36p0J9K/yoeuTvuP1q8/z3Oe rFJ44Ar3ybCBt3dqomaGCjIjqaOXz2018bAZKboJFPDD+PxCvRU38vBmv+uhe3ADTetcmwJ4Jhctu Vq8OT7knCswvS1NibSdG+t3sVyneG6GC9bP54K5kCqmJnMqGAN/LP7SMuzw75N5kI0cBcjcKZr/Hm gvabHWMYDt1+oo6EbPnA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iAwA8-0006ix-TE; Thu, 19 Sep 2019 13:07:57 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.92.2 #3 (Red Hat Linux)) id 1iAw9g-0006Ud-VR; Thu, 19 Sep 2019 13:07:30 +0000 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 55F2921929; Thu, 19 Sep 2019 13:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568898448; bh=xDlkbYtekNq2g+j/zLZRiaXJuHeVfYxFX/p6ICWFSTw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=BCHVqSuL/6xTcR4UIQNbftn+i7ALTmzjbkxcwXg9kdHIXUeW5wKOJj0jG8SGA29kT rlFF6uog75GxK0bdFaRZNI0ABOEfr0Om7lb2CRZPfxIDL3GXQYc0LdqrbSwCsSMul1 OvFGEKKeZ74rerDPYJ6ZKN6QBZCubXMaIdO0VgrU= Received: by mail-wr1-f46.google.com with SMTP id a11so3067674wrx.1; Thu, 19 Sep 2019 06:07:28 -0700 (PDT) X-Gm-Message-State: APjAAAUSg9NrardyUfhUthtWEYgjYsSoLd6aVE1YtetAIAzllPezqVB0 mVQftB4qTkHQwMJoDPz45c7QnsyeUIbhvcitQ5s= X-Google-Smtp-Source: APXvYqyNvo9nMD/j9PGGbZY5fbWhu9DS32hN7clZh4PtaLsYsLNB9DQGeR8UyO6Q34SnbEzf8UHVp/ztmfjTiqTh1Ok= X-Received: by 2002:a5d:66d2:: with SMTP id k18mr7066721wrw.7.1568898446854; Thu, 19 Sep 2019 06:07:26 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> In-Reply-To: <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> From: Guo Ren Date: Thu, 19 Sep 2019 21:07:15 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Jean-Philippe Brucker X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190919_060729_059843_045FFC08 X-CRM114-Status: GOOD ( 20.19 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aou@eecs.berkeley.edu, Linux Kernel Mailing List , Arnd Bergmann , suzuki.poulose@arm.com, Marc Zyngier , Catalin Marinas , Palmer Dabbelt , christoffer.dall@arm.com, iommu@lists.linux-foundation.org, Mike Rapoport , Anup Patel , Atish Patra , Julien Grall , james.morse@arm.com, gary@garyguo.net, Paul Walmsley , linux-riscv@lists.infradead.org, Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org Hi, On Mon, Sep 16, 2019 at 8:57 PM Jean-Philippe Brucker wrote: > On 13/09/2019 09:13, Guo Ren wrote: > > Another idea is seperate remote TLB invalidate into two instructions: > > > > - sfence.vma.b.asyc > > - sfence.vma.b.barrier // wait all async TLB invalidate operations > > finished for all harts. > > It's not clear to me how this helps, but I probably don't have the whole > picture. If you have a place where it is safe to wait for the barrier to > complete, why not do the whole invalidate there? > > > (I remember who mentioned me separate them into two instructions after > > session. Anup? Is the idea right ?) Forget it, I still use irq signal in my formal proposal [1]. I also couldn't image the whole picture :P > > To solve the problem, we could define a async mode in sfence.vma.b to > > slove the problem and finished with per_cpu_irq/exception. > > The solution I had to this problem is pinning the ASID [1] used by the > IOMMU, to prevent the CPU from recycling the ASID on rollover. This way > the CPU doesn't have to wait for IOMMU invalidations to complete, when > scheduling a task that might not even have anything to do with the IOMMU. > > In the Arm SMMU, ASID and IOASID (PASID) are separate identifiers. IOASID > indexes an entry in the context descriptor table, which contains the ASID. > So with unpinned shared ASID you don't need to invalidate the ATC on > rollover, since the IOASID doesn't change, but you do need to modify the > context descriptor and invalidate cached versions of it. The terminology confused me a lot. I perfer use PASID for IOMMU and ASID is for CPU. Arm's entry of the context descriptor table contains a "IOASID" IOASID != ASID for CPU_TLB and IOMMU_TLB. When you say "since the IOASID doesn't change",Is it PASID or my IOASID ? -_*! PASID in PCI-sig was used to determine transfer address space. For intel, the entry which is indexed by PASID also contain S1/S2.PGD and DID(VMID). For arm, the entry which is indexed by PASID only contain S1.PGD and IOASID. Compare to Intel Vt-d Scalable mode, arm's design can't support PCI Virtual Function. > > Once you have pinned ASIDs, you could also declare that IOASID = ASID. I > don't remember finding an argument to strictly forbid it, even though ASID > and IOASID have different sizes on Arm (respectively 8/16 and 20 bits). ASID and IOASID are hard to keep the same between CPU system and IOMMU system. So I introduce S1/S2.PGD.PPN as a bridge between CPUs and IOMMUs. See my proposal [1] 1: https://lore.kernel.org/linux-csky/1568896556-28769-1-git-send-email-guoren@kernel.org/T/#u -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED845C3A5A6 for ; Thu, 19 Sep 2019 13:07:30 +0000 (UTC) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B16FC21924 for ; Thu, 19 Sep 2019 13:07:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="BCHVqSuL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B16FC21924 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 7025ECC9; Thu, 19 Sep 2019 13:07:30 +0000 (UTC) Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 51BBEC84 for ; Thu, 19 Sep 2019 13:07:29 +0000 (UTC) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id DEA17711 for ; Thu, 19 Sep 2019 13:07:28 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 76A6421D7F for ; Thu, 19 Sep 2019 13:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568898448; bh=xDlkbYtekNq2g+j/zLZRiaXJuHeVfYxFX/p6ICWFSTw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=BCHVqSuL/6xTcR4UIQNbftn+i7ALTmzjbkxcwXg9kdHIXUeW5wKOJj0jG8SGA29kT rlFF6uog75GxK0bdFaRZNI0ABOEfr0Om7lb2CRZPfxIDL3GXQYc0LdqrbSwCsSMul1 OvFGEKKeZ74rerDPYJ6ZKN6QBZCubXMaIdO0VgrU= Received: by mail-wr1-f46.google.com with SMTP id o18so2983072wrv.13 for ; Thu, 19 Sep 2019 06:07:28 -0700 (PDT) X-Gm-Message-State: APjAAAVyraFmBrpBfgyltr1W89MHUDVxWumzKugy6e03jk/ccTWky404 x+WLKNHyqKdHb9sQIIOfSYmbykGhf+ePcbh0G94= X-Google-Smtp-Source: APXvYqyNvo9nMD/j9PGGbZY5fbWhu9DS32hN7clZh4PtaLsYsLNB9DQGeR8UyO6Q34SnbEzf8UHVp/ztmfjTiqTh1Ok= X-Received: by 2002:a5d:66d2:: with SMTP id k18mr7066721wrw.7.1568898446854; Thu, 19 Sep 2019 06:07:26 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> In-Reply-To: <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> From: Guo Ren Date: Thu, 19 Sep 2019 21:07:15 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Jean-Philippe Brucker Cc: aou@eecs.berkeley.edu, Linux Kernel Mailing List , Arnd Bergmann , suzuki.poulose@arm.com, Marc Zyngier , Catalin Marinas , Palmer Dabbelt , christoffer.dall@arm.com, iommu@lists.linux-foundation.org, Mike Rapoport , Anup Patel , Atish Patra , Julien Grall , james.morse@arm.com, gary@garyguo.net, Paul Walmsley , linux-riscv@lists.infradead.org, Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: iommu-bounces@lists.linux-foundation.org Errors-To: iommu-bounces@lists.linux-foundation.org Hi, On Mon, Sep 16, 2019 at 8:57 PM Jean-Philippe Brucker wrote: > On 13/09/2019 09:13, Guo Ren wrote: > > Another idea is seperate remote TLB invalidate into two instructions: > > > > - sfence.vma.b.asyc > > - sfence.vma.b.barrier // wait all async TLB invalidate operations > > finished for all harts. > > It's not clear to me how this helps, but I probably don't have the whole > picture. If you have a place where it is safe to wait for the barrier to > complete, why not do the whole invalidate there? > > > (I remember who mentioned me separate them into two instructions after > > session. Anup? Is the idea right ?) Forget it, I still use irq signal in my formal proposal [1]. I also couldn't image the whole picture :P > > To solve the problem, we could define a async mode in sfence.vma.b to > > slove the problem and finished with per_cpu_irq/exception. > > The solution I had to this problem is pinning the ASID [1] used by the > IOMMU, to prevent the CPU from recycling the ASID on rollover. This way > the CPU doesn't have to wait for IOMMU invalidations to complete, when > scheduling a task that might not even have anything to do with the IOMMU. > > In the Arm SMMU, ASID and IOASID (PASID) are separate identifiers. IOASID > indexes an entry in the context descriptor table, which contains the ASID. > So with unpinned shared ASID you don't need to invalidate the ATC on > rollover, since the IOASID doesn't change, but you do need to modify the > context descriptor and invalidate cached versions of it. The terminology confused me a lot. I perfer use PASID for IOMMU and ASID is for CPU. Arm's entry of the context descriptor table contains a "IOASID" IOASID != ASID for CPU_TLB and IOMMU_TLB. When you say "since the IOASID doesn't change",Is it PASID or my IOASID ? -_*! PASID in PCI-sig was used to determine transfer address space. For intel, the entry which is indexed by PASID also contain S1/S2.PGD and DID(VMID). For arm, the entry which is indexed by PASID only contain S1.PGD and IOASID. Compare to Intel Vt-d Scalable mode, arm's design can't support PCI Virtual Function. > > Once you have pinned ASIDs, you could also declare that IOASID = ASID. I > don't remember finding an argument to strictly forbid it, even though ASID > and IOASID have different sizes on Arm (respectively 8/16 and 20 bits). ASID and IOASID are hard to keep the same between CPU system and IOMMU system. So I introduce S1/S2.PGD.PPN as a bridge between CPUs and IOMMUs. See my proposal [1] 1: https://lore.kernel.org/linux-csky/1568896556-28769-1-git-send-email-guoren@kernel.org/T/#u -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 988DAC4CEC4 for ; Thu, 19 Sep 2019 13:07:40 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 1690F21907 for ; Thu, 19 Sep 2019 13:07:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="BCHVqSuL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1690F21907 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 925CD4A647; Thu, 19 Sep 2019 09:07:39 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IXWspmPNIP+m; Thu, 19 Sep 2019 09:07:35 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id CB83A4A6BB; Thu, 19 Sep 2019 09:07:35 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0575B4A6B2 for ; Thu, 19 Sep 2019 09:07:34 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6efdeTLxFu6v for ; Thu, 19 Sep 2019 09:07:29 -0400 (EDT) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id C13854A647 for ; Thu, 19 Sep 2019 09:07:29 -0400 (EDT) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 726B821D7E for ; Thu, 19 Sep 2019 13:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568898448; bh=xDlkbYtekNq2g+j/zLZRiaXJuHeVfYxFX/p6ICWFSTw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=BCHVqSuL/6xTcR4UIQNbftn+i7ALTmzjbkxcwXg9kdHIXUeW5wKOJj0jG8SGA29kT rlFF6uog75GxK0bdFaRZNI0ABOEfr0Om7lb2CRZPfxIDL3GXQYc0LdqrbSwCsSMul1 OvFGEKKeZ74rerDPYJ6ZKN6QBZCubXMaIdO0VgrU= Received: by mail-wr1-f48.google.com with SMTP id l3so3016564wru.7 for ; Thu, 19 Sep 2019 06:07:28 -0700 (PDT) X-Gm-Message-State: APjAAAUkb0Q7ke5/VowHOrXCve/EzCEclKBYV6onjaJWuCmmPwnfcoDC TLI3eJ4p3lid7ksZMEGr8GHWG82UalD12ksJHmQ= X-Google-Smtp-Source: APXvYqyNvo9nMD/j9PGGbZY5fbWhu9DS32hN7clZh4PtaLsYsLNB9DQGeR8UyO6Q34SnbEzf8UHVp/ztmfjTiqTh1Ok= X-Received: by 2002:a5d:66d2:: with SMTP id k18mr7066721wrw.7.1568898446854; Thu, 19 Sep 2019 06:07:26 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> In-Reply-To: <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> From: Guo Ren Date: Thu, 19 Sep 2019 21:07:15 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Jean-Philippe Brucker Cc: aou@eecs.berkeley.edu, Linux Kernel Mailing List , Arnd Bergmann , Marc Zyngier , Catalin Marinas , Palmer Dabbelt , iommu@lists.linux-foundation.org, Mike Rapoport , Anup Patel , Atish Patra , Julien Grall , gary@garyguo.net, Paul Walmsley , linux-riscv@lists.infradead.org, Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Hi, On Mon, Sep 16, 2019 at 8:57 PM Jean-Philippe Brucker wrote: > On 13/09/2019 09:13, Guo Ren wrote: > > Another idea is seperate remote TLB invalidate into two instructions: > > > > - sfence.vma.b.asyc > > - sfence.vma.b.barrier // wait all async TLB invalidate operations > > finished for all harts. > > It's not clear to me how this helps, but I probably don't have the whole > picture. If you have a place where it is safe to wait for the barrier to > complete, why not do the whole invalidate there? > > > (I remember who mentioned me separate them into two instructions after > > session. Anup? Is the idea right ?) Forget it, I still use irq signal in my formal proposal [1]. I also couldn't image the whole picture :P > > To solve the problem, we could define a async mode in sfence.vma.b to > > slove the problem and finished with per_cpu_irq/exception. > > The solution I had to this problem is pinning the ASID [1] used by the > IOMMU, to prevent the CPU from recycling the ASID on rollover. This way > the CPU doesn't have to wait for IOMMU invalidations to complete, when > scheduling a task that might not even have anything to do with the IOMMU. > > In the Arm SMMU, ASID and IOASID (PASID) are separate identifiers. IOASID > indexes an entry in the context descriptor table, which contains the ASID. > So with unpinned shared ASID you don't need to invalidate the ATC on > rollover, since the IOASID doesn't change, but you do need to modify the > context descriptor and invalidate cached versions of it. The terminology confused me a lot. I perfer use PASID for IOMMU and ASID is for CPU. Arm's entry of the context descriptor table contains a "IOASID" IOASID != ASID for CPU_TLB and IOMMU_TLB. When you say "since the IOASID doesn't change",Is it PASID or my IOASID ? -_*! PASID in PCI-sig was used to determine transfer address space. For intel, the entry which is indexed by PASID also contain S1/S2.PGD and DID(VMID). For arm, the entry which is indexed by PASID only contain S1.PGD and IOASID. Compare to Intel Vt-d Scalable mode, arm's design can't support PCI Virtual Function. > > Once you have pinned ASIDs, you could also declare that IOASID = ASID. I > don't remember finding an argument to strictly forbid it, even though ASID > and IOASID have different sizes on Arm (respectively 8/16 and 20 bits). ASID and IOASID are hard to keep the same between CPU system and IOMMU system. So I introduce S1/S2.PGD.PPN as a bridge between CPUs and IOMMUs. See my proposal [1] 1: https://lore.kernel.org/linux-csky/1568896556-28769-1-git-send-email-guoren@kernel.org/T/#u -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A682C3A5A6 for ; Thu, 19 Sep 2019 13:07:40 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2D5DF2196F for ; Thu, 19 Sep 2019 13:07:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="P6rcrH7Y"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="BCHVqSuL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D5DF2196F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Vnu1Z25jDhzzHS9Ne3QNiTQl5wsaixH4OYWNa168BBA=; b=P6rcrH7YKqlSI2 OCv3vC5xbWlZQx4Syvjg9qp64ltbhrsXGZQNXPATfxnXLWr16dmOhghyh1n0QBicyLYSAWtDr9ESM Hgs5TSBMmEEAU9BBSDtKvWOAzY/OLyCaZ7HnabV/HhrWcjiLQk0hcsdZWqdogFGylMY9F5jx4hHhu Sc0NQ+LSHwUvJmZ4ddBgcyCowvr0ct5uGojUmpohgE7uVBL4/ksUO0ZGyKxz0FceK7s/8KYeYq2gG c2DYNU3rYnaHB5RgT1FTbBC/itUexGEUn7XsC4fV7xnMJbtZ3bFmpabinQbx/wFoR3jyUo9gKKZGp /3Rxq5mgNhRMVcSvjKcg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iAw9k-0006V7-Fl; Thu, 19 Sep 2019 13:07:32 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.92.2 #3 (Red Hat Linux)) id 1iAw9g-0006Ud-VR; Thu, 19 Sep 2019 13:07:30 +0000 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 55F2921929; Thu, 19 Sep 2019 13:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568898448; bh=xDlkbYtekNq2g+j/zLZRiaXJuHeVfYxFX/p6ICWFSTw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=BCHVqSuL/6xTcR4UIQNbftn+i7ALTmzjbkxcwXg9kdHIXUeW5wKOJj0jG8SGA29kT rlFF6uog75GxK0bdFaRZNI0ABOEfr0Om7lb2CRZPfxIDL3GXQYc0LdqrbSwCsSMul1 OvFGEKKeZ74rerDPYJ6ZKN6QBZCubXMaIdO0VgrU= Received: by mail-wr1-f46.google.com with SMTP id a11so3067674wrx.1; Thu, 19 Sep 2019 06:07:28 -0700 (PDT) X-Gm-Message-State: APjAAAUSg9NrardyUfhUthtWEYgjYsSoLd6aVE1YtetAIAzllPezqVB0 mVQftB4qTkHQwMJoDPz45c7QnsyeUIbhvcitQ5s= X-Google-Smtp-Source: APXvYqyNvo9nMD/j9PGGbZY5fbWhu9DS32hN7clZh4PtaLsYsLNB9DQGeR8UyO6Q34SnbEzf8UHVp/ztmfjTiqTh1Ok= X-Received: by 2002:a5d:66d2:: with SMTP id k18mr7066721wrw.7.1568898446854; Thu, 19 Sep 2019 06:07:26 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> In-Reply-To: <057a0af3-93f7-271c-170e-4b31e6894c3c@linaro.org> From: Guo Ren Date: Thu, 19 Sep 2019 21:07:15 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Jean-Philippe Brucker X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190919_060729_059843_045FFC08 X-CRM114-Status: GOOD ( 20.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aou@eecs.berkeley.edu, Linux Kernel Mailing List , Arnd Bergmann , suzuki.poulose@arm.com, Marc Zyngier , Catalin Marinas , Palmer Dabbelt , christoffer.dall@arm.com, iommu@lists.linux-foundation.org, Mike Rapoport , Anup Patel , Atish Patra , Julien Grall , james.morse@arm.com, gary@garyguo.net, Paul Walmsley , linux-riscv@lists.infradead.org, Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, On Mon, Sep 16, 2019 at 8:57 PM Jean-Philippe Brucker wrote: > On 13/09/2019 09:13, Guo Ren wrote: > > Another idea is seperate remote TLB invalidate into two instructions: > > > > - sfence.vma.b.asyc > > - sfence.vma.b.barrier // wait all async TLB invalidate operations > > finished for all harts. > > It's not clear to me how this helps, but I probably don't have the whole > picture. If you have a place where it is safe to wait for the barrier to > complete, why not do the whole invalidate there? > > > (I remember who mentioned me separate them into two instructions after > > session. Anup? Is the idea right ?) Forget it, I still use irq signal in my formal proposal [1]. I also couldn't image the whole picture :P > > To solve the problem, we could define a async mode in sfence.vma.b to > > slove the problem and finished with per_cpu_irq/exception. > > The solution I had to this problem is pinning the ASID [1] used by the > IOMMU, to prevent the CPU from recycling the ASID on rollover. This way > the CPU doesn't have to wait for IOMMU invalidations to complete, when > scheduling a task that might not even have anything to do with the IOMMU. > > In the Arm SMMU, ASID and IOASID (PASID) are separate identifiers. IOASID > indexes an entry in the context descriptor table, which contains the ASID. > So with unpinned shared ASID you don't need to invalidate the ATC on > rollover, since the IOASID doesn't change, but you do need to modify the > context descriptor and invalidate cached versions of it. The terminology confused me a lot. I perfer use PASID for IOMMU and ASID is for CPU. Arm's entry of the context descriptor table contains a "IOASID" IOASID != ASID for CPU_TLB and IOMMU_TLB. When you say "since the IOASID doesn't change",Is it PASID or my IOASID ? -_*! PASID in PCI-sig was used to determine transfer address space. For intel, the entry which is indexed by PASID also contain S1/S2.PGD and DID(VMID). For arm, the entry which is indexed by PASID only contain S1.PGD and IOASID. Compare to Intel Vt-d Scalable mode, arm's design can't support PCI Virtual Function. > > Once you have pinned ASIDs, you could also declare that IOASID = ASID. I > don't remember finding an argument to strictly forbid it, even though ASID > and IOASID have different sizes on Arm (respectively 8/16 and 20 bits). ASID and IOASID are hard to keep the same between CPU system and IOMMU system. So I introduce S1/S2.PGD.PPN as a bridge between CPUs and IOMMUs. See my proposal [1] 1: https://lore.kernel.org/linux-csky/1568896556-28769-1-git-send-email-guoren@kernel.org/T/#u -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel