From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 435ACC433E0 for ; Thu, 7 Jan 2021 17:50:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 01514233FD for ; Thu, 7 Jan 2021 17:50:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728972AbhAGRuK (ORCPT ); Thu, 7 Jan 2021 12:50:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726319AbhAGRuI (ORCPT ); Thu, 7 Jan 2021 12:50:08 -0500 Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A2EBC0612F6 for ; Thu, 7 Jan 2021 09:48:42 -0800 (PST) Received: by mail-io1-xd32.google.com with SMTP id d9so6970353iob.6 for ; Thu, 07 Jan 2021 09:48:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6Eoqk68P1Bcb0VR3W1zmVx5ceWjyA10pZm4KhVQJdAs=; b=GM+iTlQ3BQkvH1ob8TdUxcmAsi46pi2mtWELHUgZFyyPO+EndHkxYi6Efkg3O18Tku edQt8UB0Llti6Epm5pi7gQD7Rvjj2XwfVndg2ehauNbNVLy3PAXR8NhD8ppHMDSgrEjB eTe/tAn9d2RFhHsPSUNMdpOJeoKvJtOqJ4IiU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6Eoqk68P1Bcb0VR3W1zmVx5ceWjyA10pZm4KhVQJdAs=; b=MQWRXZwGCPol6KXUW3EQ7NoiJcc6RnXe81xsK6ApPTY9X+kATuFdhT7jFBSRZj2VUC IYyMdoZAWJAFk4J/Hlpi8yGnPHDoFgQB1FGGRgpFUfTyDLOUEtoQTdjNEmpNnED6rfA7 BVClFiqM/4e3pjnQS5Q1xDuBG0DTyHbvK3h/A4ZkkG0/SB2kihGy6N58hqrgN68iWHrG jghs9DQkz2bfOg1Mm+KVxJ+ECR4IwoHeYdNUUxQuWd3+5c2WCyFY1urq9hHXk4gBNDqQ M4pj4jmLcjN03T2uhjphXi6xaYT0wqjKJx1MMYsk4tJSK2osF/rKZucc8sOB5m9vCkM9 2XyA== X-Gm-Message-State: AOAM532o6V2YrjR0Bm05H+NIulF5Yr36sG/4BqO/xDa2GQ9KW63FA5th qaCeWUf1AXk0KDn92U/3xyu0oZqglaZ/N+tA X-Google-Smtp-Source: ABdhPJxCb1ZtFsr8dUQ77+hg79uOIIhGabp2bc+zpycrwQ0Lrxmo0zgJAVNY9lnXTR3PjADMK0eO5g== X-Received: by 2002:a02:8791:: with SMTP id t17mr9010493jai.28.1610041721575; Thu, 07 Jan 2021 09:48:41 -0800 (PST) Received: from mail-il1-f172.google.com (mail-il1-f172.google.com. [209.85.166.172]) by smtp.gmail.com with ESMTPSA id h16sm4866906ile.6.2021.01.07.09.48.41 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 07 Jan 2021 09:48:41 -0800 (PST) Received: by mail-il1-f172.google.com with SMTP id e7so7548489ile.7 for ; Thu, 07 Jan 2021 09:48:41 -0800 (PST) X-Received: by 2002:a92:d592:: with SMTP id a18mr8620iln.64.1610041335740; Thu, 07 Jan 2021 09:42:15 -0800 (PST) MIME-Version: 1.0 References: <20210106034124.30560-1-tientzu@chromium.org> In-Reply-To: From: Claire Chang Date: Fri, 8 Jan 2021 01:42:04 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC PATCH v3 0/6] Restricted DMA To: Florian Fainelli Cc: Rob Herring , mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , will@kernel.org, Frank Rowand , Konrad Rzeszutek Wilk , boris.ostrovsky@oracle.com, jgross@suse.com, sstabellini@kernel.org, Christoph Hellwig , Marek Szyprowski , Robin Murphy , grant.likely@arm.com, xypron.glpk@gmx.de, Thierry Reding , mingo@kernel.org, bauerman@linux.ibm.com, peterz@infradead.org, Greg KH , Saravana Kannan , rafael.j.wysocki@intel.com, heikki.krogerus@linux.intel.com, Andy Shevchenko , rdunlap@infradead.org, dan.j.williams@intel.com, Bartosz Golaszewski , linux-devicetree , lkml , linuxppc-dev@lists.ozlabs.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , xen-devel@lists.xenproject.org, Tomasz Figa , Nicolas Boichat , Jim Quinlan Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jan 7, 2021 at 2:48 AM Florian Fainelli wrote: > > Hi, > > First of all let me say that I am glad that someone is working on a > upstream solution for this issue, would appreciate if you could CC and > Jim Quinlan on subsequent submissions. Sure! > > On 1/5/21 7:41 PM, Claire Chang wrote: > > This series implements mitigations for lack of DMA access control on > > systems without an IOMMU, which could result in the DMA accessing the > > system memory at unexpected times and/or unexpected addresses, possibly > > leading to data leakage or corruption. > > > > For example, we plan to use the PCI-e bus for Wi-Fi and that PCI-e bus is > > not behind an IOMMU. As PCI-e, by design, gives the device full access to > > system memory, a vulnerability in the Wi-Fi firmware could easily escalate > > to a full system exploit (remote wifi exploits: [1a], [1b] that shows a > > full chain of exploits; [2], [3]). > > > > To mitigate the security concerns, we introduce restricted DMA. Restricted > > DMA utilizes the existing swiotlb to bounce streaming DMA in and out of a > > specially allocated region and does memory allocation from the same region. > > The feature on its own provides a basic level of protection against the DMA > > overwriting buffer contents at unexpected times. However, to protect > > against general data leakage and system memory corruption, the system needs > > to provide a way to restrict the DMA to a predefined memory region (this is > > usually done at firmware level, e.g. in ATF on some ARM platforms). > > Can you explain how ATF gets involved and to what extent it does help, > besides enforcing a secure region from the ARM CPU's perpsective? Does > the PCIe root complex not have an IOMMU but can somehow be denied access > to a region that is marked NS=0 in the ARM CPU's MMU? If so, that is > still some sort of basic protection that the HW enforces, right? We need the ATF support for memory MPU (memory protection unit). Restricted DMA (with reserved-memory in dts) makes sure the predefined memory region is for PCIe DMA only, but we still need MPU to locks down PCIe access to that specific regions. > > On Broadcom STB SoCs we have had something similar for a while however > and while we don't have an IOMMU for the PCIe bridge, we do have a a > basic protection mechanism whereby we can configure a region in DRAM to > be PCIe read/write and CPU read/write which then gets used as the PCIe > inbound region for the PCIe EP. By default the PCIe bridge is not > allowed access to DRAM so we must call into a security agent to allow > the PCIe bridge to access the designated DRAM region. > > We have done this using a private CMA area region assigned via Device > Tree, assigned with a and requiring the PCIe EP driver to use > dma_alloc_from_contiguous() in order to allocate from this device > private CMA area. The only drawback with that approach is that it > requires knowing how much memory you need up front for buffers and DMA > descriptors that the PCIe EP will need to process. The problem is that > it requires driver modifications and that does not scale over the number > of PCIe EP drivers, some we absolutely do not control, but there is no > need to bounce buffer. Your approach scales better across PCIe EP > drivers however it does require bounce buffering which could be a > performance hit. Only the streaming DMA (map/unmap) needs bounce buffering. I also added alloc/free support in this series (https://lore.kernel.org/patchwork/patch/1360995/), so dma_direct_alloc() will try to allocate memory from the predefined memory region. As for the performance hit, it should be similar to the default swiotlb. Here are my experiment results. Both SoCs lack IOMMU for PCIe. PCIe wifi vht80 throughput - MTK SoC tcp_tx tcp_rx udp_tx udp_rx w/o Restricted DMA 244.1 134.66 312.56 350.79 w/ Restricted DMA 246.95 136.59 363.21 351.99 Rockchip SoC tcp_tx tcp_rx udp_tx udp_rx w/o Restricted DMA 237.87 133.86 288.28 361.88 w/ Restricted DMA 256.01 130.95 292.28 353.19 The CPU usage doesn't increase too much either. Although I didn't measure the CPU usage very precisely, it's ~3% with a single big core (Cortex-A72) and ~5% with a single small core (Cortex-A53). Thanks! > > Thanks! > -- > Florian From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CEB0C433DB for ; Thu, 7 Jan 2021 17:52:08 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 07724233FB for ; Thu, 7 Jan 2021 17:52:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 07724233FB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4DBYhj69gczDqgs for ; Fri, 8 Jan 2021 04:52:05 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=chromium.org (client-ip=2607:f8b0:4864:20::431; helo=mail-pf1-x431.google.com; envelope-from=tientzu@chromium.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=GM+iTlQ3; dkim-atps=neutral Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4DBYfy5qcCzDqVm for ; Fri, 8 Jan 2021 04:50:34 +1100 (AEDT) Received: by mail-pf1-x431.google.com with SMTP id s21so4295439pfu.13 for ; Thu, 07 Jan 2021 09:50:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6Eoqk68P1Bcb0VR3W1zmVx5ceWjyA10pZm4KhVQJdAs=; b=GM+iTlQ3BQkvH1ob8TdUxcmAsi46pi2mtWELHUgZFyyPO+EndHkxYi6Efkg3O18Tku edQt8UB0Llti6Epm5pi7gQD7Rvjj2XwfVndg2ehauNbNVLy3PAXR8NhD8ppHMDSgrEjB eTe/tAn9d2RFhHsPSUNMdpOJeoKvJtOqJ4IiU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6Eoqk68P1Bcb0VR3W1zmVx5ceWjyA10pZm4KhVQJdAs=; b=ShRqOd47IERluRfBzRR9vXszo4QNNfvFCY6W8GXs9RW7AzMkIsxA+xDmpl6kVZS5T5 6aM03t2bDnHcicC/Za9OuQNUCG5QI8/gF3MlLfLTsULM0MoTUmTlIACGkcEIO8/HDEz8 SGKuNg/BXJ8Q9Xd/VGHIwUu+Ayd/HlMXf/MoMCyccRP8XZ6s+H+omqLfUpsf4TKcsCvL 0Jv2v3B4GpmELWxdBRX7BJYL6c2Fw7fVV9IoEgOhw87eKG5XQNKM+5cwN3Co+BsNx3ET NJnfeJKfjSpCipE9spcjS6rEPKTY6vsRqNMTiljfysLiBQhnej4fqg7hbhWNFwsIfxhO WMwQ== X-Gm-Message-State: AOAM530WFcThzoBi+HRoQilV+U9evznhnM4sqnQMu3cXn1bG+AVFzCYB SRBHTBJ9U9BeCZodK2nwh49B6kNw+omRR20u X-Google-Smtp-Source: ABdhPJwoU+tCntWzw5rsgPuOtB9mHahRbpR6RYUAoOKdve/SQAiGp49jf8tRLRR7YZ7EH823mYrN/w== X-Received: by 2002:a63:ac19:: with SMTP id v25mr2970671pge.258.1610041830893; Thu, 07 Jan 2021 09:50:30 -0800 (PST) Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com. [209.85.215.174]) by smtp.gmail.com with ESMTPSA id 145sm6176227pfu.8.2021.01.07.09.50.30 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 07 Jan 2021 09:50:30 -0800 (PST) Received: by mail-pg1-f174.google.com with SMTP id q7so3540731pgm.5 for ; Thu, 07 Jan 2021 09:50:30 -0800 (PST) X-Received: by 2002:a92:d592:: with SMTP id a18mr8620iln.64.1610041335740; Thu, 07 Jan 2021 09:42:15 -0800 (PST) MIME-Version: 1.0 References: <20210106034124.30560-1-tientzu@chromium.org> In-Reply-To: From: Claire Chang Date: Fri, 8 Jan 2021 01:42:04 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC PATCH v3 0/6] Restricted DMA To: Florian Fainelli Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: heikki.krogerus@linux.intel.com, peterz@infradead.org, grant.likely@arm.com, paulus@samba.org, Frank Rowand , mingo@kernel.org, Marek Szyprowski , sstabellini@kernel.org, Saravana Kannan , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , rafael.j.wysocki@intel.com, Christoph Hellwig , Bartosz Golaszewski , xen-devel@lists.xenproject.org, Thierry Reding , linux-devicetree , will@kernel.org, Konrad Rzeszutek Wilk , dan.j.williams@intel.com, linuxppc-dev@lists.ozlabs.org, Rob Herring , boris.ostrovsky@oracle.com, Andy Shevchenko , jgross@suse.com, Nicolas Boichat , Greg KH , rdunlap@infradead.org, lkml , Tomasz Figa , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , Jim Quinlan , xypron.glpk@gmx.de, Robin Murphy , bauerman@linux.ibm.com Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu, Jan 7, 2021 at 2:48 AM Florian Fainelli wrote: > > Hi, > > First of all let me say that I am glad that someone is working on a > upstream solution for this issue, would appreciate if you could CC and > Jim Quinlan on subsequent submissions. Sure! > > On 1/5/21 7:41 PM, Claire Chang wrote: > > This series implements mitigations for lack of DMA access control on > > systems without an IOMMU, which could result in the DMA accessing the > > system memory at unexpected times and/or unexpected addresses, possibly > > leading to data leakage or corruption. > > > > For example, we plan to use the PCI-e bus for Wi-Fi and that PCI-e bus is > > not behind an IOMMU. As PCI-e, by design, gives the device full access to > > system memory, a vulnerability in the Wi-Fi firmware could easily escalate > > to a full system exploit (remote wifi exploits: [1a], [1b] that shows a > > full chain of exploits; [2], [3]). > > > > To mitigate the security concerns, we introduce restricted DMA. Restricted > > DMA utilizes the existing swiotlb to bounce streaming DMA in and out of a > > specially allocated region and does memory allocation from the same region. > > The feature on its own provides a basic level of protection against the DMA > > overwriting buffer contents at unexpected times. However, to protect > > against general data leakage and system memory corruption, the system needs > > to provide a way to restrict the DMA to a predefined memory region (this is > > usually done at firmware level, e.g. in ATF on some ARM platforms). > > Can you explain how ATF gets involved and to what extent it does help, > besides enforcing a secure region from the ARM CPU's perpsective? Does > the PCIe root complex not have an IOMMU but can somehow be denied access > to a region that is marked NS=0 in the ARM CPU's MMU? If so, that is > still some sort of basic protection that the HW enforces, right? We need the ATF support for memory MPU (memory protection unit). Restricted DMA (with reserved-memory in dts) makes sure the predefined memory region is for PCIe DMA only, but we still need MPU to locks down PCIe access to that specific regions. > > On Broadcom STB SoCs we have had something similar for a while however > and while we don't have an IOMMU for the PCIe bridge, we do have a a > basic protection mechanism whereby we can configure a region in DRAM to > be PCIe read/write and CPU read/write which then gets used as the PCIe > inbound region for the PCIe EP. By default the PCIe bridge is not > allowed access to DRAM so we must call into a security agent to allow > the PCIe bridge to access the designated DRAM region. > > We have done this using a private CMA area region assigned via Device > Tree, assigned with a and requiring the PCIe EP driver to use > dma_alloc_from_contiguous() in order to allocate from this device > private CMA area. The only drawback with that approach is that it > requires knowing how much memory you need up front for buffers and DMA > descriptors that the PCIe EP will need to process. The problem is that > it requires driver modifications and that does not scale over the number > of PCIe EP drivers, some we absolutely do not control, but there is no > need to bounce buffer. Your approach scales better across PCIe EP > drivers however it does require bounce buffering which could be a > performance hit. Only the streaming DMA (map/unmap) needs bounce buffering. I also added alloc/free support in this series (https://lore.kernel.org/patchwork/patch/1360995/), so dma_direct_alloc() will try to allocate memory from the predefined memory region. As for the performance hit, it should be similar to the default swiotlb. Here are my experiment results. Both SoCs lack IOMMU for PCIe. PCIe wifi vht80 throughput - MTK SoC tcp_tx tcp_rx udp_tx udp_rx w/o Restricted DMA 244.1 134.66 312.56 350.79 w/ Restricted DMA 246.95 136.59 363.21 351.99 Rockchip SoC tcp_tx tcp_rx udp_tx udp_rx w/o Restricted DMA 237.87 133.86 288.28 361.88 w/ Restricted DMA 256.01 130.95 292.28 353.19 The CPU usage doesn't increase too much either. Although I didn't measure the CPU usage very precisely, it's ~3% with a single big core (Cortex-A72) and ~5% with a single small core (Cortex-A53). Thanks! > > Thanks! > -- > Florian From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15AD8C433E0 for ; Thu, 7 Jan 2021 17:49:18 +0000 (UTC) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A3693233FB for ; Thu, 7 Jan 2021 17:49:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A3693233FB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6FAFF869E1; Thu, 7 Jan 2021 17:49:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zz2tvAvHpHBV; Thu, 7 Jan 2021 17:49:16 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id C070C860C5; Thu, 7 Jan 2021 17:49:16 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 94DB7C0891; Thu, 7 Jan 2021 17:49:16 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 21CEEC013A for ; Thu, 7 Jan 2021 17:49:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 049F320449 for ; Thu, 7 Jan 2021 17:49:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u4CdXxJO-7gw for ; Thu, 7 Jan 2021 17:49:14 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by silver.osuosl.org (Postfix) with ESMTPS id CB09720428 for ; Thu, 7 Jan 2021 17:49:13 +0000 (UTC) Received: by mail-pl1-f180.google.com with SMTP id g3so3939172plp.2 for ; Thu, 07 Jan 2021 09:49:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6Eoqk68P1Bcb0VR3W1zmVx5ceWjyA10pZm4KhVQJdAs=; b=GM+iTlQ3BQkvH1ob8TdUxcmAsi46pi2mtWELHUgZFyyPO+EndHkxYi6Efkg3O18Tku edQt8UB0Llti6Epm5pi7gQD7Rvjj2XwfVndg2ehauNbNVLy3PAXR8NhD8ppHMDSgrEjB eTe/tAn9d2RFhHsPSUNMdpOJeoKvJtOqJ4IiU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6Eoqk68P1Bcb0VR3W1zmVx5ceWjyA10pZm4KhVQJdAs=; b=Rh313uRtW04xXdKStlc9RgU+8v4ZEGBYAuv7IS8XhF/C34k9uVUDmnCN0VvRGYnZLx A/3vadW9G4ja0rAj0kW9wRJJv6GApQIeEjzGsgRUWcyCbBlRRTVldHBjHNY8ggqVuAFL bqzQhcXukQw3zsBoW3vzL6agiWiyAps69Q15g3Quq36979cFWpccy4zYDxsAJTUVhZcH onvmlrB2ZShK5qD+4s9+NhwtQilCWEMM4Xls2w8967JOeqTgO9catXSQ9keSNL5ibT2M 07zVGgdG2pM2Mepo+aTbvRxIl/FmzLXysROwfDzMlL+0RuU3GVpG7PTzH2b6+iFegRMp DBBA== X-Gm-Message-State: AOAM530qksM0lYirFR/3qSfgrTeD/ppTIaOoppBCFnFnS/x3KfBjKurO 4uKKUZLw8QH9YPfYVLiZn4QM6Rzpo7wy5TUv X-Google-Smtp-Source: ABdhPJxyE3Eo//tYe9yAN/o+5u+FozGmS7fXYr99zc1lphvuCSfQLKYrK4gpuArwx0gQY3pK5U0LKA== X-Received: by 2002:a17:902:a710:b029:dc:3817:e7c2 with SMTP id w16-20020a170902a710b02900dc3817e7c2mr40158plq.0.1610041753072; Thu, 07 Jan 2021 09:49:13 -0800 (PST) Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com. [209.85.210.172]) by smtp.gmail.com with ESMTPSA id z125sm5991419pfz.121.2021.01.07.09.49.12 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 07 Jan 2021 09:49:12 -0800 (PST) Received: by mail-pf1-f172.google.com with SMTP id m6so4313329pfm.6 for ; Thu, 07 Jan 2021 09:49:12 -0800 (PST) X-Received: by 2002:a92:d592:: with SMTP id a18mr8620iln.64.1610041335740; Thu, 07 Jan 2021 09:42:15 -0800 (PST) MIME-Version: 1.0 References: <20210106034124.30560-1-tientzu@chromium.org> In-Reply-To: From: Claire Chang Date: Fri, 8 Jan 2021 01:42:04 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC PATCH v3 0/6] Restricted DMA To: Florian Fainelli Cc: heikki.krogerus@linux.intel.com, peterz@infradead.org, benh@kernel.crashing.org, grant.likely@arm.com, paulus@samba.org, Frank Rowand , mingo@kernel.org, sstabellini@kernel.org, Saravana Kannan , mpe@ellerman.id.au, rafael.j.wysocki@intel.com, Christoph Hellwig , Bartosz Golaszewski , xen-devel@lists.xenproject.org, Thierry Reding , linux-devicetree , will@kernel.org, Konrad Rzeszutek Wilk , dan.j.williams@intel.com, linuxppc-dev@lists.ozlabs.org, Rob Herring , boris.ostrovsky@oracle.com, Andy Shevchenko , jgross@suse.com, Nicolas Boichat , Greg KH , rdunlap@infradead.org, lkml , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , Jim Quinlan , xypron.glpk@gmx.de, Robin Murphy X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Thu, Jan 7, 2021 at 2:48 AM Florian Fainelli wrote: > > Hi, > > First of all let me say that I am glad that someone is working on a > upstream solution for this issue, would appreciate if you could CC and > Jim Quinlan on subsequent submissions. Sure! > > On 1/5/21 7:41 PM, Claire Chang wrote: > > This series implements mitigations for lack of DMA access control on > > systems without an IOMMU, which could result in the DMA accessing the > > system memory at unexpected times and/or unexpected addresses, possibly > > leading to data leakage or corruption. > > > > For example, we plan to use the PCI-e bus for Wi-Fi and that PCI-e bus is > > not behind an IOMMU. As PCI-e, by design, gives the device full access to > > system memory, a vulnerability in the Wi-Fi firmware could easily escalate > > to a full system exploit (remote wifi exploits: [1a], [1b] that shows a > > full chain of exploits; [2], [3]). > > > > To mitigate the security concerns, we introduce restricted DMA. Restricted > > DMA utilizes the existing swiotlb to bounce streaming DMA in and out of a > > specially allocated region and does memory allocation from the same region. > > The feature on its own provides a basic level of protection against the DMA > > overwriting buffer contents at unexpected times. However, to protect > > against general data leakage and system memory corruption, the system needs > > to provide a way to restrict the DMA to a predefined memory region (this is > > usually done at firmware level, e.g. in ATF on some ARM platforms). > > Can you explain how ATF gets involved and to what extent it does help, > besides enforcing a secure region from the ARM CPU's perpsective? Does > the PCIe root complex not have an IOMMU but can somehow be denied access > to a region that is marked NS=0 in the ARM CPU's MMU? If so, that is > still some sort of basic protection that the HW enforces, right? We need the ATF support for memory MPU (memory protection unit). Restricted DMA (with reserved-memory in dts) makes sure the predefined memory region is for PCIe DMA only, but we still need MPU to locks down PCIe access to that specific regions. > > On Broadcom STB SoCs we have had something similar for a while however > and while we don't have an IOMMU for the PCIe bridge, we do have a a > basic protection mechanism whereby we can configure a region in DRAM to > be PCIe read/write and CPU read/write which then gets used as the PCIe > inbound region for the PCIe EP. By default the PCIe bridge is not > allowed access to DRAM so we must call into a security agent to allow > the PCIe bridge to access the designated DRAM region. > > We have done this using a private CMA area region assigned via Device > Tree, assigned with a and requiring the PCIe EP driver to use > dma_alloc_from_contiguous() in order to allocate from this device > private CMA area. The only drawback with that approach is that it > requires knowing how much memory you need up front for buffers and DMA > descriptors that the PCIe EP will need to process. The problem is that > it requires driver modifications and that does not scale over the number > of PCIe EP drivers, some we absolutely do not control, but there is no > need to bounce buffer. Your approach scales better across PCIe EP > drivers however it does require bounce buffering which could be a > performance hit. Only the streaming DMA (map/unmap) needs bounce buffering. I also added alloc/free support in this series (https://lore.kernel.org/patchwork/patch/1360995/), so dma_direct_alloc() will try to allocate memory from the predefined memory region. As for the performance hit, it should be similar to the default swiotlb. Here are my experiment results. Both SoCs lack IOMMU for PCIe. PCIe wifi vht80 throughput - MTK SoC tcp_tx tcp_rx udp_tx udp_rx w/o Restricted DMA 244.1 134.66 312.56 350.79 w/ Restricted DMA 246.95 136.59 363.21 351.99 Rockchip SoC tcp_tx tcp_rx udp_tx udp_rx w/o Restricted DMA 237.87 133.86 288.28 361.88 w/ Restricted DMA 256.01 130.95 292.28 353.19 The CPU usage doesn't increase too much either. Although I didn't measure the CPU usage very precisely, it's ~3% with a single big core (Cortex-A72) and ~5% with a single small core (Cortex-A53). Thanks! > > Thanks! > -- > Florian _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 687F5C433DB for ; Thu, 7 Jan 2021 17:50:41 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 299E3233FB for ; Thu, 7 Jan 2021 17:50:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 299E3233FB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.63023.111858 (Exim 4.92) (envelope-from ) id 1kxZQM-0002Cg-6f; Thu, 07 Jan 2021 17:50:14 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 63023.111858; Thu, 07 Jan 2021 17:50:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kxZQM-0002CZ-2f; Thu, 07 Jan 2021 17:50:14 +0000 Received: by outflank-mailman (input) for mailman id 63023; Thu, 07 Jan 2021 17:50:13 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kxZQL-0002CU-8O for xen-devel@lists.xenproject.org; Thu, 07 Jan 2021 17:50:13 +0000 Received: from mail-pg1-x52f.google.com (unknown [2607:f8b0:4864:20::52f]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 1e5280d3-a89f-45ca-9dd1-d62fa17897ba; Thu, 07 Jan 2021 17:50:11 +0000 (UTC) Received: by mail-pg1-x52f.google.com with SMTP id q7so3540106pgm.5 for ; Thu, 07 Jan 2021 09:50:11 -0800 (PST) Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com. [209.85.210.182]) by smtp.gmail.com with ESMTPSA id s29sm7235403pgn.65.2021.01.07.09.50.10 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 07 Jan 2021 09:50:10 -0800 (PST) Received: by mail-pf1-f182.google.com with SMTP id h186so4333527pfe.0 for ; Thu, 07 Jan 2021 09:50:10 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 1e5280d3-a89f-45ca-9dd1-d62fa17897ba DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6Eoqk68P1Bcb0VR3W1zmVx5ceWjyA10pZm4KhVQJdAs=; b=GM+iTlQ3BQkvH1ob8TdUxcmAsi46pi2mtWELHUgZFyyPO+EndHkxYi6Efkg3O18Tku edQt8UB0Llti6Epm5pi7gQD7Rvjj2XwfVndg2ehauNbNVLy3PAXR8NhD8ppHMDSgrEjB eTe/tAn9d2RFhHsPSUNMdpOJeoKvJtOqJ4IiU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6Eoqk68P1Bcb0VR3W1zmVx5ceWjyA10pZm4KhVQJdAs=; b=do1xqb5t9Viy9MNaaHnSb3YQoX0Yd7GjTpNG5Mv00KY/k9nTiokTLc1/cpW5Hu2Ccn KeXSRVOBUP2/J5f8quRo8cimWiDXe/g3caseWfpSp37ZJgDxbm7TEELL7pIfAeYsby0H luJaImZzo7JdJZPtl44VFyCeZ5VkTfeA+ozJQtMTs3WZytRnw+MiiUsiSUcTBbWHSZ6o Jk8JuDKKt7PbYHnl6PcwZkiz6Expnk64HJu+gGviHTn9t69a9HZTXS2els6cjTb2BbdJ MgiU+bWBCpT8x4om1f8xrNU6MOL9rMOqBGaQ/LcOTCZu/MGoRz2mm1CQmfiFoX8qpMi6 nBmg== X-Gm-Message-State: AOAM530VuH4gTFoyzqT+TfibJUSuBXxhwozICi8TpJEj3xsgDwuCu3lX m8XCWZ2uwvY8DUfvyrkrUfM59GJ6iJzqI74i X-Google-Smtp-Source: ABdhPJwpQz9sfB/RPAcfbSz7mI2OwL81VuCDiB22CkIrerXIS9VNnzuNUR2Tzzn6lsyH4GK75dQ6iA== X-Received: by 2002:a63:4e17:: with SMTP id c23mr2902404pgb.439.1610041810749; Thu, 07 Jan 2021 09:50:10 -0800 (PST) X-Received: by 2002:a92:d592:: with SMTP id a18mr8620iln.64.1610041335740; Thu, 07 Jan 2021 09:42:15 -0800 (PST) MIME-Version: 1.0 References: <20210106034124.30560-1-tientzu@chromium.org> In-Reply-To: From: Claire Chang Date: Fri, 8 Jan 2021 01:42:04 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC PATCH v3 0/6] Restricted DMA To: Florian Fainelli Cc: Rob Herring , mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , will@kernel.org, Frank Rowand , Konrad Rzeszutek Wilk , boris.ostrovsky@oracle.com, jgross@suse.com, sstabellini@kernel.org, Christoph Hellwig , Marek Szyprowski , Robin Murphy , grant.likely@arm.com, xypron.glpk@gmx.de, Thierry Reding , mingo@kernel.org, bauerman@linux.ibm.com, peterz@infradead.org, Greg KH , Saravana Kannan , rafael.j.wysocki@intel.com, heikki.krogerus@linux.intel.com, Andy Shevchenko , rdunlap@infradead.org, dan.j.williams@intel.com, Bartosz Golaszewski , linux-devicetree , lkml , linuxppc-dev@lists.ozlabs.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , xen-devel@lists.xenproject.org, Tomasz Figa , Nicolas Boichat , Jim Quinlan Content-Type: text/plain; charset="UTF-8" On Thu, Jan 7, 2021 at 2:48 AM Florian Fainelli wrote: > > Hi, > > First of all let me say that I am glad that someone is working on a > upstream solution for this issue, would appreciate if you could CC and > Jim Quinlan on subsequent submissions. Sure! > > On 1/5/21 7:41 PM, Claire Chang wrote: > > This series implements mitigations for lack of DMA access control on > > systems without an IOMMU, which could result in the DMA accessing the > > system memory at unexpected times and/or unexpected addresses, possibly > > leading to data leakage or corruption. > > > > For example, we plan to use the PCI-e bus for Wi-Fi and that PCI-e bus is > > not behind an IOMMU. As PCI-e, by design, gives the device full access to > > system memory, a vulnerability in the Wi-Fi firmware could easily escalate > > to a full system exploit (remote wifi exploits: [1a], [1b] that shows a > > full chain of exploits; [2], [3]). > > > > To mitigate the security concerns, we introduce restricted DMA. Restricted > > DMA utilizes the existing swiotlb to bounce streaming DMA in and out of a > > specially allocated region and does memory allocation from the same region. > > The feature on its own provides a basic level of protection against the DMA > > overwriting buffer contents at unexpected times. However, to protect > > against general data leakage and system memory corruption, the system needs > > to provide a way to restrict the DMA to a predefined memory region (this is > > usually done at firmware level, e.g. in ATF on some ARM platforms). > > Can you explain how ATF gets involved and to what extent it does help, > besides enforcing a secure region from the ARM CPU's perpsective? Does > the PCIe root complex not have an IOMMU but can somehow be denied access > to a region that is marked NS=0 in the ARM CPU's MMU? If so, that is > still some sort of basic protection that the HW enforces, right? We need the ATF support for memory MPU (memory protection unit). Restricted DMA (with reserved-memory in dts) makes sure the predefined memory region is for PCIe DMA only, but we still need MPU to locks down PCIe access to that specific regions. > > On Broadcom STB SoCs we have had something similar for a while however > and while we don't have an IOMMU for the PCIe bridge, we do have a a > basic protection mechanism whereby we can configure a region in DRAM to > be PCIe read/write and CPU read/write which then gets used as the PCIe > inbound region for the PCIe EP. By default the PCIe bridge is not > allowed access to DRAM so we must call into a security agent to allow > the PCIe bridge to access the designated DRAM region. > > We have done this using a private CMA area region assigned via Device > Tree, assigned with a and requiring the PCIe EP driver to use > dma_alloc_from_contiguous() in order to allocate from this device > private CMA area. The only drawback with that approach is that it > requires knowing how much memory you need up front for buffers and DMA > descriptors that the PCIe EP will need to process. The problem is that > it requires driver modifications and that does not scale over the number > of PCIe EP drivers, some we absolutely do not control, but there is no > need to bounce buffer. Your approach scales better across PCIe EP > drivers however it does require bounce buffering which could be a > performance hit. Only the streaming DMA (map/unmap) needs bounce buffering. I also added alloc/free support in this series (https://lore.kernel.org/patchwork/patch/1360995/), so dma_direct_alloc() will try to allocate memory from the predefined memory region. As for the performance hit, it should be similar to the default swiotlb. Here are my experiment results. Both SoCs lack IOMMU for PCIe. PCIe wifi vht80 throughput - MTK SoC tcp_tx tcp_rx udp_tx udp_rx w/o Restricted DMA 244.1 134.66 312.56 350.79 w/ Restricted DMA 246.95 136.59 363.21 351.99 Rockchip SoC tcp_tx tcp_rx udp_tx udp_rx w/o Restricted DMA 237.87 133.86 288.28 361.88 w/ Restricted DMA 256.01 130.95 292.28 353.19 The CPU usage doesn't increase too much either. Although I didn't measure the CPU usage very precisely, it's ~3% with a single big core (Cortex-A72) and ~5% with a single small core (Cortex-A53). Thanks! > > Thanks! > -- > Florian