From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 454AFC433EF for ; Thu, 4 Nov 2021 16:22:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 228D461220 for ; Thu, 4 Nov 2021 16:22:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231826AbhKDQZT (ORCPT ); Thu, 4 Nov 2021 12:25:19 -0400 Received: from mail.kernel.org ([198.145.29.99]:47618 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231689AbhKDQZR (ORCPT ); Thu, 4 Nov 2021 12:25:17 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id A3BAF61244 for ; Thu, 4 Nov 2021 16:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1636042959; bh=P3rnnoNdYJk+6i+LZOXAgcPbirulOacewaSkMDXsPs8=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=tpsZ8hOvhbnElC8qkOuCbY6ub2S+0VZPb5Z9ZLo6MhNuTAypmrJ2wGteYG8Vvb2ya GOUBrhoF08ZLaB+mD9/2DFF9t8kyc9vgFCtWXSZoOX1XPXykxc0kXJPP7cemZpwpsd WDraozpTmJ2+2mbUE+WENVKQ+4Cz9BtNrpBUFq8dYUgEnKM/BAE7yesEIvjf/QnLjB 6VDwYBBwNwPHq7ReGZKl6goBT9l/7u7Ixk7/b4m0+0wg6yeWJ0ySixhuI9QsuzIfwy vTXNwwfVJEl1VyuG0x+XTp0IRV5SfA8QkRoa2iHDbb8KZ0t8fTmKVjHcp0u19xdpyE vswS0lx3zynpw== Received: by mail-oi1-f173.google.com with SMTP id l15so10062113oie.8 for ; Thu, 04 Nov 2021 09:22:39 -0700 (PDT) X-Gm-Message-State: AOAM533MKIufWN5UuYco3plz7E+UTky+8G/G0WzP5SczgBgNr/5sEBNU I1mQ+fHZaLiHlEMAxp3nXqW4pOHha/3I9MjTNcs= X-Google-Smtp-Source: ABdhPJxsY85yy3v6ZutAz3NirxoHWcEYpbh3E2JXSFxjxTMgcKXQkwTDTl2c3zzTDK6ObyR4H61EG3aXz2evOvt6DgQ= X-Received: by 2002:a05:6808:b2b:: with SMTP id t11mr12272034oij.47.1636042958945; Thu, 04 Nov 2021 09:22:38 -0700 (PDT) MIME-Version: 1.0 References: <20211104023221.16391-1-walter-zh.wu@mediatek.com> <20211104085336.GA24260@lst.de> <9da413e95727a3b48ea35ec576aa1b1b57ffc9b9.camel@mediatek.com> In-Reply-To: <9da413e95727a3b48ea35ec576aa1b1b57ffc9b9.camel@mediatek.com> From: Ard Biesheuvel Date: Thu, 4 Nov 2021 17:22:27 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v2] dma-direct: improve DMA_ATTR_NO_KERNEL_MAPPING To: Walter Wu Cc: Christoph Hellwig , Marek Szyprowski , Robin Murphy , Matthias Brugger , Andrew Morton , Linux IOMMU , Linux Kernel Mailing List , Linux ARM , wsd_upstream , linux-mediatek@lists.infradead.org Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 4 Nov 2021 at 14:40, Walter Wu wrote: > > On Thu, 2021-11-04 at 13:47 +0100, Ard Biesheuvel wrote: > > On Thu, 4 Nov 2021 at 13:31, Walter Wu > > wrote: > > > > > > On Thu, 2021-11-04 at 09:57 +0100, Ard Biesheuvel wrote: > > > > On Thu, 4 Nov 2021 at 09:53, Christoph Hellwig > > > > wrote: > > > > > > > > > > On Thu, Nov 04, 2021 at 10:32:21AM +0800, Walter Wu wrote: > > > > > > diff --git a/include/linux/set_memory.h > > > > > > b/include/linux/set_memory.h > > > > > > index f36be5166c19..6c7d1683339c 100644 > > > > > > --- a/include/linux/set_memory.h > > > > > > +++ b/include/linux/set_memory.h > > > > > > @@ -7,11 +7,16 @@ > > > > > > > > > > > > #ifdef CONFIG_ARCH_HAS_SET_MEMORY > > > > > > #include > > > > > > + > > > > > > +#ifndef CONFIG_RODATA_FULL_DEFAULT_ENABLED > > > > > > > > > > This is an arm64-specific symbol, and one that only controls a > > > > > default. I don't think it is suitable to key off stubs in > > > > > common > > > > > code. > > > > > > > > > > > +static inline int set_memory_valid(unsigned long addr, int > > > > > > numpages, int enable) { return 0; } > > > > > > > > > > Pleae avoid overly long lines. > > > > > > > > > > > + if > > > > > > (IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED)) > > > > > > { > > > > > > + kaddr = (unsigned > > > > > > long)phys_to_virt(dma_to_phys(dev, *dma_handle)); > > > > > > > > > > This can just use page_address. > > > > > > > > > > > + /* page remove kernel mapping for arm64 > > > > > > */ > > > > > > + set_memory_valid(kaddr, size >> > > > > > > PAGE_SHIFT, > > > > > > 0); > > > > > > + } > > > > > > > > > > But more importantly: set_memory_valid only exists on arm64, > > > > > this > > > > > will break compile everywhere else. And this API is complete > > > > > crap. > > > > > Passing kernel virtual addresses as unsigned long just sucks, > > > > > and > > > > > passing an integer argument for valid/non-valid also is a > > > > > horrible > > > > > API. > > > > > > > > > > > > > ... and as I pointed out before, you can still pass rodata=off on > > > > arm64, and get the old behavior, in which case bad things will > > > > happen > > > > if you try to use an API that expects to operate on page mappings > > > > with > > > > a 1 GB block mapping. > > > > > > > > > > Thanks for your suggestion. > > > > > > > > > > And you still haven't explained what the actual problem is: is > > > > this > > > > about CPU speculation corrupting non-cache coherent inbound DMA? > > > > > > No corrupiton, only cpu read it, we hope to fix the behavior. > > > > > > > Fix which behavior? Please explain > > > > 1) the current behavior > We call dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING to get the > allocated buffer and the kernel mapping is exist. Our goal is this > buffer doesn't allow to be accessed by cpu. Unfortunately, we see cpu > speculation to read it. So we need to fix it and don't use no-map the > way. > > > 2) why the current behavior is problematic for you > dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING have kernel mapping, > so it still has cpu speculation read the buffer. Although we have > hardware to protect the buffer, we still hope use software to fix it. > But *why* is this a problem? You are saying that the speculative accesses are not causing corruption, so they are causing other issues that you want to address. So which issues are we talking about here? > > 3) how this patch changes the current behavior > When call dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING, then > remove the kernel mapping which belong to the buffer. > > > 4) why the new behavior fixes your problem. > If I understand correctly, want to block cpu speculation, then need > unmap the buffer at stage 1 and stage 2 page table and tlb invalidate. > This patch is to do stage 1 unmap at EL1. > > > > > There is no penalty for using too many words. > > Thanks. > Walter > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F49FC433F5 for ; Thu, 4 Nov 2021 16:22:43 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3F7E86121E for ; Thu, 4 Nov 2021 16:22:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3F7E86121E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0A3E360B5A; Thu, 4 Nov 2021 16:22:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wWaw0nx6FX_w; Thu, 4 Nov 2021 16:22:42 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id D33106073A; Thu, 4 Nov 2021 16:22:41 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A95EEC0019; Thu, 4 Nov 2021 16:22:41 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id DE085C000E for ; Thu, 4 Nov 2021 16:22:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id BEE4581A16 for ; Thu, 4 Nov 2021 16:22:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=kernel.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c-w93dIqh39f for ; Thu, 4 Nov 2021 16:22:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp1.osuosl.org (Postfix) with ESMTPS id 255FC819FC for ; Thu, 4 Nov 2021 16:22:40 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id 9BC3E61220 for ; Thu, 4 Nov 2021 16:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1636042959; bh=P3rnnoNdYJk+6i+LZOXAgcPbirulOacewaSkMDXsPs8=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=tpsZ8hOvhbnElC8qkOuCbY6ub2S+0VZPb5Z9ZLo6MhNuTAypmrJ2wGteYG8Vvb2ya GOUBrhoF08ZLaB+mD9/2DFF9t8kyc9vgFCtWXSZoOX1XPXykxc0kXJPP7cemZpwpsd WDraozpTmJ2+2mbUE+WENVKQ+4Cz9BtNrpBUFq8dYUgEnKM/BAE7yesEIvjf/QnLjB 6VDwYBBwNwPHq7ReGZKl6goBT9l/7u7Ixk7/b4m0+0wg6yeWJ0ySixhuI9QsuzIfwy vTXNwwfVJEl1VyuG0x+XTp0IRV5SfA8QkRoa2iHDbb8KZ0t8fTmKVjHcp0u19xdpyE vswS0lx3zynpw== Received: by mail-oi1-f177.google.com with SMTP id bg25so9387651oib.1 for ; Thu, 04 Nov 2021 09:22:39 -0700 (PDT) X-Gm-Message-State: AOAM533MZtAQWt/Uxr3Av2JMqcWTOx604ojSZWJXF+n2FnZ8NA0AfHhD 9VeB/eSEyDbSyV/nrAnGgmwsIYwL1Xx8Obh525E= X-Google-Smtp-Source: ABdhPJxsY85yy3v6ZutAz3NirxoHWcEYpbh3E2JXSFxjxTMgcKXQkwTDTl2c3zzTDK6ObyR4H61EG3aXz2evOvt6DgQ= X-Received: by 2002:a05:6808:b2b:: with SMTP id t11mr12272034oij.47.1636042958945; Thu, 04 Nov 2021 09:22:38 -0700 (PDT) MIME-Version: 1.0 References: <20211104023221.16391-1-walter-zh.wu@mediatek.com> <20211104085336.GA24260@lst.de> <9da413e95727a3b48ea35ec576aa1b1b57ffc9b9.camel@mediatek.com> In-Reply-To: <9da413e95727a3b48ea35ec576aa1b1b57ffc9b9.camel@mediatek.com> From: Ard Biesheuvel Date: Thu, 4 Nov 2021 17:22:27 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v2] dma-direct: improve DMA_ATTR_NO_KERNEL_MAPPING To: Walter Wu Cc: wsd_upstream , Linux Kernel Mailing List , Linux IOMMU , linux-mediatek@lists.infradead.org, Matthias Brugger , Andrew Morton , Robin Murphy , Christoph Hellwig , Linux ARM X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Thu, 4 Nov 2021 at 14:40, Walter Wu wrote: > > On Thu, 2021-11-04 at 13:47 +0100, Ard Biesheuvel wrote: > > On Thu, 4 Nov 2021 at 13:31, Walter Wu > > wrote: > > > > > > On Thu, 2021-11-04 at 09:57 +0100, Ard Biesheuvel wrote: > > > > On Thu, 4 Nov 2021 at 09:53, Christoph Hellwig > > > > wrote: > > > > > > > > > > On Thu, Nov 04, 2021 at 10:32:21AM +0800, Walter Wu wrote: > > > > > > diff --git a/include/linux/set_memory.h > > > > > > b/include/linux/set_memory.h > > > > > > index f36be5166c19..6c7d1683339c 100644 > > > > > > --- a/include/linux/set_memory.h > > > > > > +++ b/include/linux/set_memory.h > > > > > > @@ -7,11 +7,16 @@ > > > > > > > > > > > > #ifdef CONFIG_ARCH_HAS_SET_MEMORY > > > > > > #include > > > > > > + > > > > > > +#ifndef CONFIG_RODATA_FULL_DEFAULT_ENABLED > > > > > > > > > > This is an arm64-specific symbol, and one that only controls a > > > > > default. I don't think it is suitable to key off stubs in > > > > > common > > > > > code. > > > > > > > > > > > +static inline int set_memory_valid(unsigned long addr, int > > > > > > numpages, int enable) { return 0; } > > > > > > > > > > Pleae avoid overly long lines. > > > > > > > > > > > + if > > > > > > (IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED)) > > > > > > { > > > > > > + kaddr = (unsigned > > > > > > long)phys_to_virt(dma_to_phys(dev, *dma_handle)); > > > > > > > > > > This can just use page_address. > > > > > > > > > > > + /* page remove kernel mapping for arm64 > > > > > > */ > > > > > > + set_memory_valid(kaddr, size >> > > > > > > PAGE_SHIFT, > > > > > > 0); > > > > > > + } > > > > > > > > > > But more importantly: set_memory_valid only exists on arm64, > > > > > this > > > > > will break compile everywhere else. And this API is complete > > > > > crap. > > > > > Passing kernel virtual addresses as unsigned long just sucks, > > > > > and > > > > > passing an integer argument for valid/non-valid also is a > > > > > horrible > > > > > API. > > > > > > > > > > > > > ... and as I pointed out before, you can still pass rodata=off on > > > > arm64, and get the old behavior, in which case bad things will > > > > happen > > > > if you try to use an API that expects to operate on page mappings > > > > with > > > > a 1 GB block mapping. > > > > > > > > > > Thanks for your suggestion. > > > > > > > > > > And you still haven't explained what the actual problem is: is > > > > this > > > > about CPU speculation corrupting non-cache coherent inbound DMA? > > > > > > No corrupiton, only cpu read it, we hope to fix the behavior. > > > > > > > Fix which behavior? Please explain > > > > 1) the current behavior > We call dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING to get the > allocated buffer and the kernel mapping is exist. Our goal is this > buffer doesn't allow to be accessed by cpu. Unfortunately, we see cpu > speculation to read it. So we need to fix it and don't use no-map the > way. > > > 2) why the current behavior is problematic for you > dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING have kernel mapping, > so it still has cpu speculation read the buffer. Although we have > hardware to protect the buffer, we still hope use software to fix it. > But *why* is this a problem? You are saying that the speculative accesses are not causing corruption, so they are causing other issues that you want to address. So which issues are we talking about here? > > 3) how this patch changes the current behavior > When call dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING, then > remove the kernel mapping which belong to the buffer. > > > 4) why the new behavior fixes your problem. > If I understand correctly, want to block cpu speculation, then need > unmap the buffer at stage 1 and stage 2 page table and tlb invalidate. > This patch is to do stage 1 unmap at EL1. > > > > > There is no penalty for using too many words. > > Thanks. > Walter > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFAE4C4332F for ; Thu, 4 Nov 2021 16:24:19 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B3C406121E for ; Thu, 4 Nov 2021 16:24:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B3C406121E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=z3IzEYn0SjgrOXL1xR2eGqbxNjKp8xmfgXMq7Cbi+CU=; b=fnozau4zMrdBXV 3/Jooku1eX8CoUrsXGNF3mLhgxnRWK4g+n7wBvlsiRALV/ahucuACjSjAEKlySsUsMsEVgZj0eynE Lxyj5rp8ucgeG5SKFiy384gWgaZQiVDK1rNXzfML79+n0wGuEbThcImoaXwwMvEuCQngagQpKIN7t fQ056VM9pEMRnt1YFpfchd+WOn/RDhgIHDhtCfsvHTpbtOPxydGM3xYu/xnHOQUTF/WN/1QLVW65H ZajL2zGpk9+5nq70e9HGHWJvmkjBEDEkkpeOVgxws1xTWYX/z4T9G1/NArDjIVMGmMCZ/mNLmolUu QiDLeCz/0qOvRv/WWI6w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mifX6-009RxW-8Z; Thu, 04 Nov 2021 16:24:08 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mifVg-009RHn-O3; Thu, 04 Nov 2021 16:22:42 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 96C4961212; Thu, 4 Nov 2021 16:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1636042959; bh=P3rnnoNdYJk+6i+LZOXAgcPbirulOacewaSkMDXsPs8=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=tpsZ8hOvhbnElC8qkOuCbY6ub2S+0VZPb5Z9ZLo6MhNuTAypmrJ2wGteYG8Vvb2ya GOUBrhoF08ZLaB+mD9/2DFF9t8kyc9vgFCtWXSZoOX1XPXykxc0kXJPP7cemZpwpsd WDraozpTmJ2+2mbUE+WENVKQ+4Cz9BtNrpBUFq8dYUgEnKM/BAE7yesEIvjf/QnLjB 6VDwYBBwNwPHq7ReGZKl6goBT9l/7u7Ixk7/b4m0+0wg6yeWJ0ySixhuI9QsuzIfwy vTXNwwfVJEl1VyuG0x+XTp0IRV5SfA8QkRoa2iHDbb8KZ0t8fTmKVjHcp0u19xdpyE vswS0lx3zynpw== Received: by mail-oi1-f175.google.com with SMTP id o4so10052954oia.10; Thu, 04 Nov 2021 09:22:39 -0700 (PDT) X-Gm-Message-State: AOAM531CzSEYNIaiV9vMDYvTr2VYvdFRht4Z9iDTAfxHDqRuW5v4Tl27 ZMfTyrJc/QZPCKoE/Iep7UfZ5xv0A7ihaMGwGZA= X-Google-Smtp-Source: ABdhPJxsY85yy3v6ZutAz3NirxoHWcEYpbh3E2JXSFxjxTMgcKXQkwTDTl2c3zzTDK6ObyR4H61EG3aXz2evOvt6DgQ= X-Received: by 2002:a05:6808:b2b:: with SMTP id t11mr12272034oij.47.1636042958945; Thu, 04 Nov 2021 09:22:38 -0700 (PDT) MIME-Version: 1.0 References: <20211104023221.16391-1-walter-zh.wu@mediatek.com> <20211104085336.GA24260@lst.de> <9da413e95727a3b48ea35ec576aa1b1b57ffc9b9.camel@mediatek.com> In-Reply-To: <9da413e95727a3b48ea35ec576aa1b1b57ffc9b9.camel@mediatek.com> From: Ard Biesheuvel Date: Thu, 4 Nov 2021 17:22:27 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v2] dma-direct: improve DMA_ATTR_NO_KERNEL_MAPPING To: Walter Wu Cc: Christoph Hellwig , Marek Szyprowski , Robin Murphy , Matthias Brugger , Andrew Morton , Linux IOMMU , Linux Kernel Mailing List , Linux ARM , wsd_upstream , linux-mediatek@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211104_092240_863134_4C679C7B X-CRM114-Status: GOOD ( 44.21 ) X-BeenThere: linux-mediatek@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Linux-mediatek" Errors-To: linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org On Thu, 4 Nov 2021 at 14:40, Walter Wu wrote: > > On Thu, 2021-11-04 at 13:47 +0100, Ard Biesheuvel wrote: > > On Thu, 4 Nov 2021 at 13:31, Walter Wu > > wrote: > > > > > > On Thu, 2021-11-04 at 09:57 +0100, Ard Biesheuvel wrote: > > > > On Thu, 4 Nov 2021 at 09:53, Christoph Hellwig > > > > wrote: > > > > > > > > > > On Thu, Nov 04, 2021 at 10:32:21AM +0800, Walter Wu wrote: > > > > > > diff --git a/include/linux/set_memory.h > > > > > > b/include/linux/set_memory.h > > > > > > index f36be5166c19..6c7d1683339c 100644 > > > > > > --- a/include/linux/set_memory.h > > > > > > +++ b/include/linux/set_memory.h > > > > > > @@ -7,11 +7,16 @@ > > > > > > > > > > > > #ifdef CONFIG_ARCH_HAS_SET_MEMORY > > > > > > #include > > > > > > + > > > > > > +#ifndef CONFIG_RODATA_FULL_DEFAULT_ENABLED > > > > > > > > > > This is an arm64-specific symbol, and one that only controls a > > > > > default. I don't think it is suitable to key off stubs in > > > > > common > > > > > code. > > > > > > > > > > > +static inline int set_memory_valid(unsigned long addr, int > > > > > > numpages, int enable) { return 0; } > > > > > > > > > > Pleae avoid overly long lines. > > > > > > > > > > > + if > > > > > > (IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED)) > > > > > > { > > > > > > + kaddr = (unsigned > > > > > > long)phys_to_virt(dma_to_phys(dev, *dma_handle)); > > > > > > > > > > This can just use page_address. > > > > > > > > > > > + /* page remove kernel mapping for arm64 > > > > > > */ > > > > > > + set_memory_valid(kaddr, size >> > > > > > > PAGE_SHIFT, > > > > > > 0); > > > > > > + } > > > > > > > > > > But more importantly: set_memory_valid only exists on arm64, > > > > > this > > > > > will break compile everywhere else. And this API is complete > > > > > crap. > > > > > Passing kernel virtual addresses as unsigned long just sucks, > > > > > and > > > > > passing an integer argument for valid/non-valid also is a > > > > > horrible > > > > > API. > > > > > > > > > > > > > ... and as I pointed out before, you can still pass rodata=off on > > > > arm64, and get the old behavior, in which case bad things will > > > > happen > > > > if you try to use an API that expects to operate on page mappings > > > > with > > > > a 1 GB block mapping. > > > > > > > > > > Thanks for your suggestion. > > > > > > > > > > And you still haven't explained what the actual problem is: is > > > > this > > > > about CPU speculation corrupting non-cache coherent inbound DMA? > > > > > > No corrupiton, only cpu read it, we hope to fix the behavior. > > > > > > > Fix which behavior? Please explain > > > > 1) the current behavior > We call dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING to get the > allocated buffer and the kernel mapping is exist. Our goal is this > buffer doesn't allow to be accessed by cpu. Unfortunately, we see cpu > speculation to read it. So we need to fix it and don't use no-map the > way. > > > 2) why the current behavior is problematic for you > dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING have kernel mapping, > so it still has cpu speculation read the buffer. Although we have > hardware to protect the buffer, we still hope use software to fix it. > But *why* is this a problem? You are saying that the speculative accesses are not causing corruption, so they are causing other issues that you want to address. So which issues are we talking about here? > > 3) how this patch changes the current behavior > When call dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING, then > remove the kernel mapping which belong to the buffer. > > > 4) why the new behavior fixes your problem. > If I understand correctly, want to block cpu speculation, then need > unmap the buffer at stage 1 and stage 2 page table and tlb invalidate. > This patch is to do stage 1 unmap at EL1. > > > > > There is no penalty for using too many words. > > Thanks. > Walter > _______________________________________________ Linux-mediatek mailing list Linux-mediatek@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-mediatek From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB57BC433EF for ; Thu, 4 Nov 2021 16:25:14 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 78E8361213 for ; Thu, 4 Nov 2021 16:25:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 78E8361213 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YhaDfuygC/98412QM/AybBadFB/vtxLOU0InvgHvrk8=; b=qPAF9gQOC8XPpO gBJvDMZU1r5xFb9t+Yx5Gn3HkiNDMtooxhKjRO7pSSUNa8OLL0jEJo8dlJSMmbnKnvo0BlIo6lRVD XJWdNrRNLluDH0dJgOxFPoCshCrLEPcM+OBhgvBZgaQ+SfI0LvdXLdfaZOrF03M7gsqsQiySjPtUO hYNaKWvmy63m7XCi8j7pzK7YBIWMWO1XcCKjuodMHY4+tj8efCx5pN5OFCFik7uWYdU+ZhgouPwPp Alh+mGQJNKOWqaB1/FNpWaWBmn6cyjrJ0lMf26sypPOQHPeRoMZjeKLL8PyqXSQpdxUF987y6UPOH TBc0mgGtdARUaBvrydKw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mifWW-009RgJ-VK; Thu, 04 Nov 2021 16:23:33 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mifVg-009RHn-O3; Thu, 04 Nov 2021 16:22:42 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 96C4961212; Thu, 4 Nov 2021 16:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1636042959; bh=P3rnnoNdYJk+6i+LZOXAgcPbirulOacewaSkMDXsPs8=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=tpsZ8hOvhbnElC8qkOuCbY6ub2S+0VZPb5Z9ZLo6MhNuTAypmrJ2wGteYG8Vvb2ya GOUBrhoF08ZLaB+mD9/2DFF9t8kyc9vgFCtWXSZoOX1XPXykxc0kXJPP7cemZpwpsd WDraozpTmJ2+2mbUE+WENVKQ+4Cz9BtNrpBUFq8dYUgEnKM/BAE7yesEIvjf/QnLjB 6VDwYBBwNwPHq7ReGZKl6goBT9l/7u7Ixk7/b4m0+0wg6yeWJ0ySixhuI9QsuzIfwy vTXNwwfVJEl1VyuG0x+XTp0IRV5SfA8QkRoa2iHDbb8KZ0t8fTmKVjHcp0u19xdpyE vswS0lx3zynpw== Received: by mail-oi1-f175.google.com with SMTP id o4so10052954oia.10; Thu, 04 Nov 2021 09:22:39 -0700 (PDT) X-Gm-Message-State: AOAM531CzSEYNIaiV9vMDYvTr2VYvdFRht4Z9iDTAfxHDqRuW5v4Tl27 ZMfTyrJc/QZPCKoE/Iep7UfZ5xv0A7ihaMGwGZA= X-Google-Smtp-Source: ABdhPJxsY85yy3v6ZutAz3NirxoHWcEYpbh3E2JXSFxjxTMgcKXQkwTDTl2c3zzTDK6ObyR4H61EG3aXz2evOvt6DgQ= X-Received: by 2002:a05:6808:b2b:: with SMTP id t11mr12272034oij.47.1636042958945; Thu, 04 Nov 2021 09:22:38 -0700 (PDT) MIME-Version: 1.0 References: <20211104023221.16391-1-walter-zh.wu@mediatek.com> <20211104085336.GA24260@lst.de> <9da413e95727a3b48ea35ec576aa1b1b57ffc9b9.camel@mediatek.com> In-Reply-To: <9da413e95727a3b48ea35ec576aa1b1b57ffc9b9.camel@mediatek.com> From: Ard Biesheuvel Date: Thu, 4 Nov 2021 17:22:27 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v2] dma-direct: improve DMA_ATTR_NO_KERNEL_MAPPING To: Walter Wu Cc: Christoph Hellwig , Marek Szyprowski , Robin Murphy , Matthias Brugger , Andrew Morton , Linux IOMMU , Linux Kernel Mailing List , Linux ARM , wsd_upstream , linux-mediatek@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211104_092240_863134_4C679C7B X-CRM114-Status: GOOD ( 44.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 4 Nov 2021 at 14:40, Walter Wu wrote: > > On Thu, 2021-11-04 at 13:47 +0100, Ard Biesheuvel wrote: > > On Thu, 4 Nov 2021 at 13:31, Walter Wu > > wrote: > > > > > > On Thu, 2021-11-04 at 09:57 +0100, Ard Biesheuvel wrote: > > > > On Thu, 4 Nov 2021 at 09:53, Christoph Hellwig > > > > wrote: > > > > > > > > > > On Thu, Nov 04, 2021 at 10:32:21AM +0800, Walter Wu wrote: > > > > > > diff --git a/include/linux/set_memory.h > > > > > > b/include/linux/set_memory.h > > > > > > index f36be5166c19..6c7d1683339c 100644 > > > > > > --- a/include/linux/set_memory.h > > > > > > +++ b/include/linux/set_memory.h > > > > > > @@ -7,11 +7,16 @@ > > > > > > > > > > > > #ifdef CONFIG_ARCH_HAS_SET_MEMORY > > > > > > #include > > > > > > + > > > > > > +#ifndef CONFIG_RODATA_FULL_DEFAULT_ENABLED > > > > > > > > > > This is an arm64-specific symbol, and one that only controls a > > > > > default. I don't think it is suitable to key off stubs in > > > > > common > > > > > code. > > > > > > > > > > > +static inline int set_memory_valid(unsigned long addr, int > > > > > > numpages, int enable) { return 0; } > > > > > > > > > > Pleae avoid overly long lines. > > > > > > > > > > > + if > > > > > > (IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED)) > > > > > > { > > > > > > + kaddr = (unsigned > > > > > > long)phys_to_virt(dma_to_phys(dev, *dma_handle)); > > > > > > > > > > This can just use page_address. > > > > > > > > > > > + /* page remove kernel mapping for arm64 > > > > > > */ > > > > > > + set_memory_valid(kaddr, size >> > > > > > > PAGE_SHIFT, > > > > > > 0); > > > > > > + } > > > > > > > > > > But more importantly: set_memory_valid only exists on arm64, > > > > > this > > > > > will break compile everywhere else. And this API is complete > > > > > crap. > > > > > Passing kernel virtual addresses as unsigned long just sucks, > > > > > and > > > > > passing an integer argument for valid/non-valid also is a > > > > > horrible > > > > > API. > > > > > > > > > > > > > ... and as I pointed out before, you can still pass rodata=off on > > > > arm64, and get the old behavior, in which case bad things will > > > > happen > > > > if you try to use an API that expects to operate on page mappings > > > > with > > > > a 1 GB block mapping. > > > > > > > > > > Thanks for your suggestion. > > > > > > > > > > And you still haven't explained what the actual problem is: is > > > > this > > > > about CPU speculation corrupting non-cache coherent inbound DMA? > > > > > > No corrupiton, only cpu read it, we hope to fix the behavior. > > > > > > > Fix which behavior? Please explain > > > > 1) the current behavior > We call dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING to get the > allocated buffer and the kernel mapping is exist. Our goal is this > buffer doesn't allow to be accessed by cpu. Unfortunately, we see cpu > speculation to read it. So we need to fix it and don't use no-map the > way. > > > 2) why the current behavior is problematic for you > dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING have kernel mapping, > so it still has cpu speculation read the buffer. Although we have > hardware to protect the buffer, we still hope use software to fix it. > But *why* is this a problem? You are saying that the speculative accesses are not causing corruption, so they are causing other issues that you want to address. So which issues are we talking about here? > > 3) how this patch changes the current behavior > When call dma_direct_alloc() with DMA_ATTR_NO_KERNEL_MAPPING, then > remove the kernel mapping which belong to the buffer. > > > 4) why the new behavior fixes your problem. > If I understand correctly, want to block cpu speculation, then need > unmap the buffer at stage 1 and stage 2 page table and tlb invalidate. > This patch is to do stage 1 unmap at EL1. > > > > > There is no penalty for using too many words. > > Thanks. > Walter > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel