From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB1E7C4CEC6 for ; Thu, 12 Sep 2019 14:59:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A28F3214DA for ; Thu, 12 Sep 2019 14:59:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568300399; bh=RK47i9JzV2o8zYa+doB7UiayI/FwXMaWv5Ml8LccsgY=; h=References:In-Reply-To:From:Date:Subject:To:Cc:List-ID:From; b=uSiP7cDA6JfovO1y4CBI79sQMJZyXRNBbVakMda38TlHSQbSroU8uy0CotNCD8tKK D0cAkm9etLfMRkwkzs2RntfNRd3IPT8zisYIKvjmaKiL6p2sCgyxtqPVQBwdYSKHQn QaXLkndpMmbnfbmpeohCHQ/BxZ2JJfuTcpazmxa4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732877AbfILO76 (ORCPT ); Thu, 12 Sep 2019 10:59:58 -0400 Received: from mail.kernel.org ([198.145.29.99]:33286 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732444AbfILO76 (ORCPT ); Thu, 12 Sep 2019 10:59:58 -0400 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E362F20856 for ; Thu, 12 Sep 2019 14:59:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568300397; bh=RK47i9JzV2o8zYa+doB7UiayI/FwXMaWv5Ml8LccsgY=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=Mp8tMCnHBMtri7nKuvpjSoNW3SBhD9Q2PQQus+HoDhqqsO2hfh5+kB+09qrWUHo06 Xg50YZM1a8zDvFIJJv0CI/EvakHiQne0lhgmu9mtHuf7+l8qkR8L3fuEBxSzlzY4/S /qPsvdajQdsWoOvQTd8Ut5MEZxBU2HYNYlaTeYNs= Received: by mail-wr1-f42.google.com with SMTP id k6so16676822wrn.11 for ; Thu, 12 Sep 2019 07:59:56 -0700 (PDT) X-Gm-Message-State: APjAAAUN8G3NAXB6G2kaGAQv/QpbQmaJRrxp06n/+Od23v1/kaqikAgw fqcxwD/3S3uVEYK6d68kHH9uSWEb7Wuax80YsaM= X-Google-Smtp-Source: APXvYqyWrVR+NI16eudQNtduL1OAxnO0WQ4SkOLjY2AVyz5N6syNQZ1qE/5QhN1PWc17kzrEkpHSXdkh4aFf0C6oOhM= X-Received: by 2002:a5d:6b49:: with SMTP id x9mr12703438wrw.80.1568300395358; Thu, 12 Sep 2019 07:59:55 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> In-Reply-To: <20190912140256.fwbutgmadpjbjnab@willie-the-truck> From: Guo Ren Date: Thu, 12 Sep 2019 15:59:43 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Will Deacon Cc: Will Deacon , julien.thierry@arm.com, aou@eecs.berkeley.edu, james.morse@arm.com, Arnd Bergmann , suzuki.poulose@arm.com, Marc Zyngier , Catalin Marinas , Anup Patel , Linux Kernel Mailing List , Mike Rapoport , Christoph Hellwig , Atish Patra , Julien Grall , Palmer Dabbelt , gary@garyguo.net, Paul Walmsley , christoffer.dall@arm.com, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Thx Will for reply. On Thu, Sep 12, 2019 at 3:03 PM Will Deacon wrote: > > On Sun, Sep 08, 2019 at 07:52:55AM +0800, Guo Ren wrote: > > On Mon, Jun 24, 2019 at 6:40 PM Will Deacon wrote: > > > > I'll keep my system use the same ASID for SMP + IOMMU :P > > > > > > You will want a separate allocator for that: > > > > > > https://lkml.kernel.org/r/20190610184714.6786-2-jean-philippe.brucker@arm.com > > > > Yes, it is hard to maintain ASID between IOMMU and CPUMMU or different > > system, because it's difficult to synchronize the IO_ASID when the CPU > > ASID is rollover. > > But we could still use hardware broadcast TLB invalidation instruction > > to uniformly manage the ASID and IO_ASID, or OTHER_ASID in our IOMMU. > > That's probably a bad idea, because you'll likely stall execution on the > CPU until the IOTLB has completed invalidation. In the case of ATS, I think > an endpoint ATC is permitted to take over a minute to respond. In reality, I > suspect the worst you'll ever see would be in the msec range, but that's > still an unacceptable period of time to hold a CPU. Just as I've said in the session that IOTLB invalidate delay is another topic, My main proposal is to introduce stage1.pgd and stage2.pgd as address space identifiers between different TLB systems based on vmid, asid. My last part of sildes will show you how to translate stage1/2.pgd to as/vmid in PCI ATS system and the method could work with SMMU-v3 and intel Vt-d. (It's regret for me there is no time to show you the whole slides.) In our light IOMMU implementation, there's no IOTLB invalidate delay problem. Becasue IOMMU is very close to CPU MMU and interconnect's delay is the same with SMP CPUs MMU (no PCI, VM supported). To solve the problem, we could define a async mode in sfence.vma.b to slove the problem and finished with per_cpu_irq/exception. -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2BAEC4CEC6 for ; Thu, 12 Sep 2019 15:00:22 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8821D2081B for ; Thu, 12 Sep 2019 15:00:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="mtijvUaD"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Mp8tMCnH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8821D2081B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nWmuPPPb8rWNjOof28pn3cDFoKJO7j19bK8l6nsLo5g=; b=mtijvUaDpnXdFh lr29mRS3F5tmfcYTzIZ7Wf6Bzta/x4XM1ACxtgkd+iO8zeyl2fynXdWzDV9zqfCe/ePIrlcp67Rfn 58jJmZ2g+dDdDbHlLqjM2QccoCYyamP9fN9NMntpUmGrw8jAN/iAp1PQxhE5w7JTY8jo1XrnRaAzM yIufSWcXNCW9qgoMK0zvz6wFktUWCrwtvEitrG/X+Jpgt4rCtXOHG7sJhdaNFk8ZgjZnE9jBf43kq DqFdp3KvDIn2sg3puhP1oWJnxphcFFFHUQFvz6TIdy/thKYSE98BkbSwSZJsCK08jA8hOejQ4F6Gb 0M4jxi3sTBj9sK3cXrvQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1i8Qa5-0004Jl-0w; Thu, 12 Sep 2019 15:00:21 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.92.2 #3 (Red Hat Linux)) id 1i8QZi-0002u3-3X; Thu, 12 Sep 2019 14:59:59 +0000 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0AF13214DE; Thu, 12 Sep 2019 14:59:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568300397; bh=RK47i9JzV2o8zYa+doB7UiayI/FwXMaWv5Ml8LccsgY=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=Mp8tMCnHBMtri7nKuvpjSoNW3SBhD9Q2PQQus+HoDhqqsO2hfh5+kB+09qrWUHo06 Xg50YZM1a8zDvFIJJv0CI/EvakHiQne0lhgmu9mtHuf7+l8qkR8L3fuEBxSzlzY4/S /qPsvdajQdsWoOvQTd8Ut5MEZxBU2HYNYlaTeYNs= Received: by mail-wr1-f47.google.com with SMTP id d17so16050749wrq.13; Thu, 12 Sep 2019 07:59:56 -0700 (PDT) X-Gm-Message-State: APjAAAXDCQC4irmItVQjtvprR3UzSkSyDMxX1pgwPKUMSD3updTpj6Fv GMuTIChyxK7cYkUrSPz1X5nByET5WcF88LOVAAk= X-Google-Smtp-Source: APXvYqyWrVR+NI16eudQNtduL1OAxnO0WQ4SkOLjY2AVyz5N6syNQZ1qE/5QhN1PWc17kzrEkpHSXdkh4aFf0C6oOhM= X-Received: by 2002:a5d:6b49:: with SMTP id x9mr12703438wrw.80.1568300395358; Thu, 12 Sep 2019 07:59:55 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> In-Reply-To: <20190912140256.fwbutgmadpjbjnab@willie-the-truck> From: Guo Ren Date: Thu, 12 Sep 2019 15:59:43 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Will Deacon X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190912_075958_194338_55C5B371 X-CRM114-Status: GOOD ( 16.00 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: julien.thierry@arm.com, Catalin Marinas , Palmer Dabbelt , Will Deacon , christoffer.dall@arm.com, Atish Patra , Julien Grall , gary@garyguo.net, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, Mike Rapoport , Christoph Hellwig , aou@eecs.berkeley.edu, Arnd Bergmann , suzuki.poulose@arm.com, Marc Zyngier , Paul Walmsley , linux-arm-kernel@lists.infradead.org, Anup Patel , Linux Kernel Mailing List , iommu@lists.linux-foundation.org, james.morse@arm.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org Thx Will for reply. On Thu, Sep 12, 2019 at 3:03 PM Will Deacon wrote: > > On Sun, Sep 08, 2019 at 07:52:55AM +0800, Guo Ren wrote: > > On Mon, Jun 24, 2019 at 6:40 PM Will Deacon wrote: > > > > I'll keep my system use the same ASID for SMP + IOMMU :P > > > > > > You will want a separate allocator for that: > > > > > > https://lkml.kernel.org/r/20190610184714.6786-2-jean-philippe.brucker@arm.com > > > > Yes, it is hard to maintain ASID between IOMMU and CPUMMU or different > > system, because it's difficult to synchronize the IO_ASID when the CPU > > ASID is rollover. > > But we could still use hardware broadcast TLB invalidation instruction > > to uniformly manage the ASID and IO_ASID, or OTHER_ASID in our IOMMU. > > That's probably a bad idea, because you'll likely stall execution on the > CPU until the IOTLB has completed invalidation. In the case of ATS, I think > an endpoint ATC is permitted to take over a minute to respond. In reality, I > suspect the worst you'll ever see would be in the msec range, but that's > still an unacceptable period of time to hold a CPU. Just as I've said in the session that IOTLB invalidate delay is another topic, My main proposal is to introduce stage1.pgd and stage2.pgd as address space identifiers between different TLB systems based on vmid, asid. My last part of sildes will show you how to translate stage1/2.pgd to as/vmid in PCI ATS system and the method could work with SMMU-v3 and intel Vt-d. (It's regret for me there is no time to show you the whole slides.) In our light IOMMU implementation, there's no IOTLB invalidate delay problem. Becasue IOMMU is very close to CPU MMU and interconnect's delay is the same with SMP CPUs MMU (no PCI, VM supported). To solve the problem, we could define a async mode in sfence.vma.b to slove the problem and finished with per_cpu_irq/exception. -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC683C4CEC5 for ; Thu, 12 Sep 2019 14:59:59 +0000 (UTC) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6D14B2081B for ; Thu, 12 Sep 2019 14:59:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Mp8tMCnH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6D14B2081B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 33862DAB; Thu, 12 Sep 2019 14:59:59 +0000 (UTC) Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id CF9BBC96 for ; Thu, 12 Sep 2019 14:59:57 +0000 (UTC) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 789DC7D2 for ; Thu, 12 Sep 2019 14:59:57 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 03750214D8 for ; Thu, 12 Sep 2019 14:59:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568300397; bh=RK47i9JzV2o8zYa+doB7UiayI/FwXMaWv5Ml8LccsgY=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=Mp8tMCnHBMtri7nKuvpjSoNW3SBhD9Q2PQQus+HoDhqqsO2hfh5+kB+09qrWUHo06 Xg50YZM1a8zDvFIJJv0CI/EvakHiQne0lhgmu9mtHuf7+l8qkR8L3fuEBxSzlzY4/S /qPsvdajQdsWoOvQTd8Ut5MEZxBU2HYNYlaTeYNs= Received: by mail-wr1-f46.google.com with SMTP id q14so28795764wrm.9 for ; Thu, 12 Sep 2019 07:59:56 -0700 (PDT) X-Gm-Message-State: APjAAAUsFzIMM30pCoPHpAaROKly0G2O8uh+eygiAcNO6i9+zpDY7Ww2 hclvEbHVm2/49IwsWya/vsuGlOtMZngjZaL5WaY= X-Google-Smtp-Source: APXvYqyWrVR+NI16eudQNtduL1OAxnO0WQ4SkOLjY2AVyz5N6syNQZ1qE/5QhN1PWc17kzrEkpHSXdkh4aFf0C6oOhM= X-Received: by 2002:a5d:6b49:: with SMTP id x9mr12703438wrw.80.1568300395358; Thu, 12 Sep 2019 07:59:55 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> In-Reply-To: <20190912140256.fwbutgmadpjbjnab@willie-the-truck> From: Guo Ren Date: Thu, 12 Sep 2019 15:59:43 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Will Deacon Cc: julien.thierry@arm.com, Catalin Marinas , Palmer Dabbelt , Will Deacon , christoffer.dall@arm.com, Atish Patra , Julien Grall , gary@garyguo.net, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, Mike Rapoport , aou@eecs.berkeley.edu, Arnd Bergmann , suzuki.poulose@arm.com, Marc Zyngier , Paul Walmsley , linux-arm-kernel@lists.infradead.org, Anup Patel , Linux Kernel Mailing List , iommu@lists.linux-foundation.org, james.morse@arm.com X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: iommu-bounces@lists.linux-foundation.org Errors-To: iommu-bounces@lists.linux-foundation.org Thx Will for reply. On Thu, Sep 12, 2019 at 3:03 PM Will Deacon wrote: > > On Sun, Sep 08, 2019 at 07:52:55AM +0800, Guo Ren wrote: > > On Mon, Jun 24, 2019 at 6:40 PM Will Deacon wrote: > > > > I'll keep my system use the same ASID for SMP + IOMMU :P > > > > > > You will want a separate allocator for that: > > > > > > https://lkml.kernel.org/r/20190610184714.6786-2-jean-philippe.brucker@arm.com > > > > Yes, it is hard to maintain ASID between IOMMU and CPUMMU or different > > system, because it's difficult to synchronize the IO_ASID when the CPU > > ASID is rollover. > > But we could still use hardware broadcast TLB invalidation instruction > > to uniformly manage the ASID and IO_ASID, or OTHER_ASID in our IOMMU. > > That's probably a bad idea, because you'll likely stall execution on the > CPU until the IOTLB has completed invalidation. In the case of ATS, I think > an endpoint ATC is permitted to take over a minute to respond. In reality, I > suspect the worst you'll ever see would be in the msec range, but that's > still an unacceptable period of time to hold a CPU. Just as I've said in the session that IOTLB invalidate delay is another topic, My main proposal is to introduce stage1.pgd and stage2.pgd as address space identifiers between different TLB systems based on vmid, asid. My last part of sildes will show you how to translate stage1/2.pgd to as/vmid in PCI ATS system and the method could work with SMMU-v3 and intel Vt-d. (It's regret for me there is no time to show you the whole slides.) In our light IOMMU implementation, there's no IOTLB invalidate delay problem. Becasue IOMMU is very close to CPU MMU and interconnect's delay is the same with SMP CPUs MMU (no PCI, VM supported). To solve the problem, we could define a async mode in sfence.vma.b to slove the problem and finished with per_cpu_irq/exception. -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 459ECC4CEC5 for ; Thu, 12 Sep 2019 15:00:03 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id BCC332081B for ; Thu, 12 Sep 2019 15:00:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Mp8tMCnH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BCC332081B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 623D54A64E; Thu, 12 Sep 2019 11:00:02 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Tcj+aFjENSIs; Thu, 12 Sep 2019 11:00:01 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 3FD5B4A650; Thu, 12 Sep 2019 11:00:01 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 9FDBD4A64E for ; Thu, 12 Sep 2019 10:59:59 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q0C+bN4zO9RJ for ; Thu, 12 Sep 2019 10:59:58 -0400 (EDT) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 574764A60D for ; Thu, 12 Sep 2019 10:59:58 -0400 (EDT) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1361C216F4 for ; Thu, 12 Sep 2019 14:59:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568300397; bh=RK47i9JzV2o8zYa+doB7UiayI/FwXMaWv5Ml8LccsgY=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=Mp8tMCnHBMtri7nKuvpjSoNW3SBhD9Q2PQQus+HoDhqqsO2hfh5+kB+09qrWUHo06 Xg50YZM1a8zDvFIJJv0CI/EvakHiQne0lhgmu9mtHuf7+l8qkR8L3fuEBxSzlzY4/S /qPsvdajQdsWoOvQTd8Ut5MEZxBU2HYNYlaTeYNs= Received: by mail-wr1-f52.google.com with SMTP id q17so24128756wrx.10 for ; Thu, 12 Sep 2019 07:59:56 -0700 (PDT) X-Gm-Message-State: APjAAAW525QrhjzjEXfUUNztEHo/yOA4iuy9y5hVR5LeD3Q8LsWd97nA fCKaZ2NYN2TNJ1K2Onf4TSIYEA0PF9N7olKQEFg= X-Google-Smtp-Source: APXvYqyWrVR+NI16eudQNtduL1OAxnO0WQ4SkOLjY2AVyz5N6syNQZ1qE/5QhN1PWc17kzrEkpHSXdkh4aFf0C6oOhM= X-Received: by 2002:a5d:6b49:: with SMTP id x9mr12703438wrw.80.1568300395358; Thu, 12 Sep 2019 07:59:55 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> In-Reply-To: <20190912140256.fwbutgmadpjbjnab@willie-the-truck> From: Guo Ren Date: Thu, 12 Sep 2019 15:59:43 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Will Deacon Cc: Catalin Marinas , Palmer Dabbelt , Will Deacon , Atish Patra , Julien Grall , gary@garyguo.net, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, Mike Rapoport , Christoph Hellwig , aou@eecs.berkeley.edu, Arnd Bergmann , Marc Zyngier , Paul Walmsley , linux-arm-kernel@lists.infradead.org, Anup Patel , Linux Kernel Mailing List , iommu@lists.linux-foundation.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Thx Will for reply. On Thu, Sep 12, 2019 at 3:03 PM Will Deacon wrote: > > On Sun, Sep 08, 2019 at 07:52:55AM +0800, Guo Ren wrote: > > On Mon, Jun 24, 2019 at 6:40 PM Will Deacon wrote: > > > > I'll keep my system use the same ASID for SMP + IOMMU :P > > > > > > You will want a separate allocator for that: > > > > > > https://lkml.kernel.org/r/20190610184714.6786-2-jean-philippe.brucker@arm.com > > > > Yes, it is hard to maintain ASID between IOMMU and CPUMMU or different > > system, because it's difficult to synchronize the IO_ASID when the CPU > > ASID is rollover. > > But we could still use hardware broadcast TLB invalidation instruction > > to uniformly manage the ASID and IO_ASID, or OTHER_ASID in our IOMMU. > > That's probably a bad idea, because you'll likely stall execution on the > CPU until the IOTLB has completed invalidation. In the case of ATS, I think > an endpoint ATC is permitted to take over a minute to respond. In reality, I > suspect the worst you'll ever see would be in the msec range, but that's > still an unacceptable period of time to hold a CPU. Just as I've said in the session that IOTLB invalidate delay is another topic, My main proposal is to introduce stage1.pgd and stage2.pgd as address space identifiers between different TLB systems based on vmid, asid. My last part of sildes will show you how to translate stage1/2.pgd to as/vmid in PCI ATS system and the method could work with SMMU-v3 and intel Vt-d. (It's regret for me there is no time to show you the whole slides.) In our light IOMMU implementation, there's no IOTLB invalidate delay problem. Becasue IOMMU is very close to CPU MMU and interconnect's delay is the same with SMP CPUs MMU (no PCI, VM supported). To solve the problem, we could define a async mode in sfence.vma.b to slove the problem and finished with per_cpu_irq/exception. -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03E3FC4CEC5 for ; Thu, 12 Sep 2019 15:00:11 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CBBAF2081B for ; Thu, 12 Sep 2019 15:00:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="TLR9QpDe"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Mp8tMCnH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CBBAF2081B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=t0hwH/ul2CKTJwVHaKoaEoOkuuWpE0XAFTr+zBJo4OA=; b=TLR9QpDeYU4tpt WeHXGralk3wfRUwX5O/eKMsPwPEC64qzotFqIpbKAkUBqyBMbHNOGKcO1hvox2hzTND7dsnKDzVvY 8ZCGcOjcUzuyOWECYoYTG6vLrSTQZRh2LbjJjKFDTHtf4HL5meLFQKH/AvTqgM8Z2Ge+KB/8D6hUp aNRDr4FuaYMdg+5DQF1w9/gx022KafBr/nQ6TwkOJiMO+viAT3lz7cfb6EaUasJ8LsGGkhNfDe0+a LE5OZaGeBnSO3BHiQp3A2Vmnu8Dtm6VPFUZjHRjtoNeyQstZH/ZnC5cqOXjQ4PLgOycrL1jgnMsrl iw/SN/qH6dwvfkhOVwRw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1i8QZl-0002uU-JA; Thu, 12 Sep 2019 15:00:01 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.92.2 #3 (Red Hat Linux)) id 1i8QZi-0002u3-3X; Thu, 12 Sep 2019 14:59:59 +0000 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0AF13214DE; Thu, 12 Sep 2019 14:59:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1568300397; bh=RK47i9JzV2o8zYa+doB7UiayI/FwXMaWv5Ml8LccsgY=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=Mp8tMCnHBMtri7nKuvpjSoNW3SBhD9Q2PQQus+HoDhqqsO2hfh5+kB+09qrWUHo06 Xg50YZM1a8zDvFIJJv0CI/EvakHiQne0lhgmu9mtHuf7+l8qkR8L3fuEBxSzlzY4/S /qPsvdajQdsWoOvQTd8Ut5MEZxBU2HYNYlaTeYNs= Received: by mail-wr1-f47.google.com with SMTP id d17so16050749wrq.13; Thu, 12 Sep 2019 07:59:56 -0700 (PDT) X-Gm-Message-State: APjAAAXDCQC4irmItVQjtvprR3UzSkSyDMxX1pgwPKUMSD3updTpj6Fv GMuTIChyxK7cYkUrSPz1X5nByET5WcF88LOVAAk= X-Google-Smtp-Source: APXvYqyWrVR+NI16eudQNtduL1OAxnO0WQ4SkOLjY2AVyz5N6syNQZ1qE/5QhN1PWc17kzrEkpHSXdkh4aFf0C6oOhM= X-Received: by 2002:a5d:6b49:: with SMTP id x9mr12703438wrw.80.1568300395358; Thu, 12 Sep 2019 07:59:55 -0700 (PDT) MIME-Version: 1.0 References: <20190321163623.20219-12-julien.grall@arm.com> <0dfe120b-066a-2ac8-13bc-3f5a29e2caa3@arm.com> <20190619091219.GB7767@fuggles.cambridge.arm.com> <20190619123939.GF7767@fuggles.cambridge.arm.com> <20190624104006.lvm32nahemaqklxc@willie-the-truck> <20190912140256.fwbutgmadpjbjnab@willie-the-truck> In-Reply-To: <20190912140256.fwbutgmadpjbjnab@willie-the-truck> From: Guo Ren Date: Thu, 12 Sep 2019 15:59:43 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH RFC 11/14] arm64: Move the ASID allocator code in a separate file To: Will Deacon X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190912_075958_194338_55C5B371 X-CRM114-Status: GOOD ( 16.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: julien.thierry@arm.com, Catalin Marinas , Palmer Dabbelt , Will Deacon , christoffer.dall@arm.com, Atish Patra , Julien Grall , gary@garyguo.net, linux-riscv@lists.infradead.org, kvmarm@lists.cs.columbia.edu, Mike Rapoport , Christoph Hellwig , aou@eecs.berkeley.edu, Arnd Bergmann , suzuki.poulose@arm.com, Marc Zyngier , Paul Walmsley , linux-arm-kernel@lists.infradead.org, Anup Patel , Linux Kernel Mailing List , iommu@lists.linux-foundation.org, james.morse@arm.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Thx Will for reply. On Thu, Sep 12, 2019 at 3:03 PM Will Deacon wrote: > > On Sun, Sep 08, 2019 at 07:52:55AM +0800, Guo Ren wrote: > > On Mon, Jun 24, 2019 at 6:40 PM Will Deacon wrote: > > > > I'll keep my system use the same ASID for SMP + IOMMU :P > > > > > > You will want a separate allocator for that: > > > > > > https://lkml.kernel.org/r/20190610184714.6786-2-jean-philippe.brucker@arm.com > > > > Yes, it is hard to maintain ASID between IOMMU and CPUMMU or different > > system, because it's difficult to synchronize the IO_ASID when the CPU > > ASID is rollover. > > But we could still use hardware broadcast TLB invalidation instruction > > to uniformly manage the ASID and IO_ASID, or OTHER_ASID in our IOMMU. > > That's probably a bad idea, because you'll likely stall execution on the > CPU until the IOTLB has completed invalidation. In the case of ATS, I think > an endpoint ATC is permitted to take over a minute to respond. In reality, I > suspect the worst you'll ever see would be in the msec range, but that's > still an unacceptable period of time to hold a CPU. Just as I've said in the session that IOTLB invalidate delay is another topic, My main proposal is to introduce stage1.pgd and stage2.pgd as address space identifiers between different TLB systems based on vmid, asid. My last part of sildes will show you how to translate stage1/2.pgd to as/vmid in PCI ATS system and the method could work with SMMU-v3 and intel Vt-d. (It's regret for me there is no time to show you the whole slides.) In our light IOMMU implementation, there's no IOTLB invalidate delay problem. Becasue IOMMU is very close to CPU MMU and interconnect's delay is the same with SMP CPUs MMU (no PCI, VM supported). To solve the problem, we could define a async mode in sfence.vma.b to slove the problem and finished with per_cpu_irq/exception. -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel