From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34C6FC433E1 for ; Wed, 19 Aug 2020 15:11:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 01B5E20882 for ; Wed, 19 Aug 2020 15:11:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CmFRV9MF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726809AbgHSPLo (ORCPT ); Wed, 19 Aug 2020 11:11:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726636AbgHSPLm (ORCPT ); Wed, 19 Aug 2020 11:11:42 -0400 Received: from mail-ej1-x643.google.com (mail-ej1-x643.google.com [IPv6:2a00:1450:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DF2DC061757 for ; Wed, 19 Aug 2020 08:11:41 -0700 (PDT) Received: by mail-ej1-x643.google.com with SMTP id c16so26700278ejx.12 for ; Wed, 19 Aug 2020 08:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=CmFRV9MF78Io9lyU3ZTHQoxCUWL8rgrVpgpqOj4nEXWpRTCd1+KGSochi2kuoVhtFt PQJJembCFDIORAph3WzrpDoRd2cIGJpwf9VAtP5a9ydpphhRkHDHcmKRPuhZGWASaGfl y6rPrnsIqK2MPfXnYUVbYzJtuDwuiigpqdGRc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=bN8er5n7rdMW8dHOSN+iqDpvTPf5/D2pn91zZ0+lE2hpYQeNcMbm9YhR9OdifeVliM 0OGFQAcypWnVIaC66YWMS+2jcsYiwklS0u5jgJfQlQxwvOqObY6xzYMC9EbPZOoeJawh ve+/ObXn59ZA2M+7y0n5p18ln0N/EHx6rya3ACln+bAK2AD3lo6n4VS5G5BkvjZAGnQe ivlLOb7xZByx2rLd134KbMfLdUaFnz3JR9oDtOjhCyYooIaX/sHnZOhbyc+SymeOi0qT Gf4qdew6+fj8UfP2bR5KxXHTIOi+c4hBUPmuSXHRGc1nV+SEJKyth7lJehRWHA9EPY5z Mznw== X-Gm-Message-State: AOAM530JDEUqMT2uas5GZl+pIW3qSEFoMUorqVtSafoYpMQADMGSygZn M4fi0MP7eB4MnIlv1lRy4FbW/LgCnLYCQQ== X-Google-Smtp-Source: ABdhPJy0ZcnyNZ+0zJw9CpzYp66QC9xVmUj2FuL+wZrXW/bdBoMcuJdGN9wNbutaRr0FwPFWtynAoA== X-Received: by 2002:a17:906:2e86:: with SMTP id o6mr26664242eji.380.1597849899983; Wed, 19 Aug 2020 08:11:39 -0700 (PDT) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com. [209.85.128.44]) by smtp.gmail.com with ESMTPSA id p23sm13942202edm.31.2020.08.19.08.11.39 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 19 Aug 2020 08:11:39 -0700 (PDT) Received: by mail-wm1-f44.google.com with SMTP id g8so2433601wmk.3 for ; Wed, 19 Aug 2020 08:11:39 -0700 (PDT) X-Received: by 2002:a1c:4d12:: with SMTP id o18mr5316483wmh.55.1597849448826; Wed, 19 Aug 2020 08:04:08 -0700 (PDT) MIME-Version: 1.0 References: <20200819065555.1802761-1-hch@lst.de> <20200819065555.1802761-20-hch@lst.de> In-Reply-To: <20200819065555.1802761-20-hch@lst.de> From: Tomasz Figa Date: Wed, 19 Aug 2020 17:03:52 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 19/28] dma-mapping: replace DMA_ATTR_NON_CONSISTENT with dma_{alloc, free}_pages To: Christoph Hellwig Cc: Mauro Carvalho Chehab , Thomas Bogendoerfer , "James E.J. Bottomley" , Joonyoung Shim , Seung-Woo Kim , Kyungmin Park , Ben Skeggs , Pawel Osciak , Marek Szyprowski , Matt Porter , "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , Tom Lendacky , alsa-devel@alsa-project.org, linux-samsung-soc , linux-ia64@vger.kernel.org, linux-scsi@vger.kernel.org, linux-parisc@vger.kernel.org, Linux Doc Mailing List , nouveau@lists.freedesktop.org, Linux Kernel Mailing List , linux-nvme@lists.infradead.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , Linux Media Mailing List Content-Type: text/plain; charset="UTF-8" Sender: linux-parisc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-parisc@vger.kernel.org Hi Christoph, On Wed, Aug 19, 2020 at 8:57 AM Christoph Hellwig wrote: > > Add a new API to allocate and free pages that are guaranteed to be > addressable by a device, but otherwise behave like pages allocated by > alloc_pages. The intended APIs to sync them for use with the device > and cpu are dma_sync_single_for_{device,cpu} that are also used for > streaming mappings. > > Switch all drivers over to this new API, but keep the usage of the > crufty dma_cache_sync API for now, which will be cleaned up on a driver > by driver basis. > > Signed-off-by: Christoph Hellwig > --- > Documentation/core-api/dma-api.rst | 68 +++++++++++------------ > Documentation/core-api/dma-attributes.rst | 8 --- > arch/alpha/kernel/pci_iommu.c | 2 + > arch/arm/mm/dma-mapping-nommu.c | 2 + > arch/arm/mm/dma-mapping.c | 4 ++ > arch/ia64/hp/common/sba_iommu.c | 2 + > arch/mips/jazz/jazzdma.c | 7 +-- > arch/powerpc/kernel/dma-iommu.c | 2 + > arch/powerpc/platforms/ps3/system-bus.c | 4 ++ > arch/powerpc/platforms/pseries/vio.c | 2 + > arch/s390/pci/pci_dma.c | 2 + > arch/x86/kernel/amd_gart_64.c | 2 + > drivers/iommu/dma-iommu.c | 2 + > drivers/iommu/intel/iommu.c | 4 ++ > drivers/net/ethernet/i825xx/lasi_82596.c | 13 ++--- > drivers/net/ethernet/seeq/sgiseeq.c | 12 ++-- > drivers/parisc/ccio-dma.c | 2 + > drivers/parisc/sba_iommu.c | 2 + > drivers/scsi/53c700.c | 8 +-- > drivers/scsi/sgiwd93.c | 12 ++-- > drivers/xen/swiotlb-xen.c | 2 + > include/linux/dma-direct.h | 5 ++ > include/linux/dma-mapping.h | 29 ++++++++-- > include/linux/dma-noncoherent.h | 3 - > kernel/dma/direct.c | 51 ++++++++++++++++- > kernel/dma/mapping.c | 43 +++++++++++++- > kernel/dma/ops_helpers.c | 35 ++++++++++++ > kernel/dma/virt.c | 2 + > sound/mips/hal2.c | 20 +++---- > 29 files changed, 254 insertions(+), 96 deletions(-) > Thanks for the patch. The general design looks quite nice, but please see my comments inline. > diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst > index 90239348b30f6f..047fcfffa0e5cf 100644 > --- a/Documentation/core-api/dma-api.rst > +++ b/Documentation/core-api/dma-api.rst > @@ -516,48 +516,53 @@ routines, e.g.::: > } > > > -Part II - Advanced dma usage > ----------------------------- > +Part II - Non-coherent DMA allocations > +-------------------------------------- > > -Warning: These pieces of the DMA API should not be used in the > -majority of cases, since they cater for unlikely corner cases that > -don't belong in usual drivers. > +These APIs allow to allocate pages that can be used like normal pages > +in the kernel direct mapping, but are guaranteed to be DMA addressable. Could we elaborate a bit more on what "like normal pages in kernel direct mapping" mean from the driver perspective? > > If you don't understand how cache line coherency works between a > processor and an I/O device, you should not be using this part of the > -API at all. > +API. > > :: > > void * > - dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > - gfp_t flag, unsigned long attrs) > + dma_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, > + enum dma_data_direction dir, gfp_t gfp) > + > +This routine allocates a region of bytes of consistent memory. It > +returns a pointer to the allocated region (in the processor's virtual address > +space) or NULL if the allocation failed. The returned memory is guanteed to > +behave like memory allocated using alloc_pages. There is one aspect that the existing dma_alloc_attrs() handles, but this new function doesn't: IOMMU support. The function will always allocate a physically-contiguous block memory, which is a costly operation and not even guaranteed to succeed, even if enough free memory is available. Modern SoCs employ IOMMUs to avoid the need to allocate physically-contiguous memory and those happen to be also the devices that could benefit from non-coherent allocations a lot. One of the tasks of the DMA API was making it possible to allocate suitable memory for a given device, without having the driver know about the SoC integration details, such as the presence of an IOMMU. Today, dma_alloc_attrs() uses the .alloc callback of the dma_ops struct and the IOMMU-aware implementations, like the dma-iommu helpers [1], would allocate discontiguous pages. Therefore, while I see the DMA-aware page allocation functionality as a useful functionality on its own for scatter-gather-capable hardware, I believe it is not a complete replacement for dma_alloc_attrs() with the DMA_ATTR_NON_CONSISTENT attribute. [1] https://elixir.bootlin.com/linux/v5.9-rc1/source/drivers/iommu/dma-iommu.c#L510 Best regards, Tomasz From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6724DC433E3 for ; Wed, 19 Aug 2020 15:04:23 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 37D962076E for ; Wed, 19 Aug 2020 15:04:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Z0745qz8"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CmFRV9MF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 37D962076E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=KwxmdJpcuq0aco+jn0+65e+051jGsrKi5oBcYrOY/1Y=; b=Z0745qz8oarCW5khTXILbQwKg nT9kQhEftOb1SXZY/RPJI2ZTnhzc/Y64SdDSTJviex7NUvo6jhq63M8j3sgoJtlAtjWXufWXXFGLx E7px319+feD5PZJvPpSKr+DgllfNzFOyxQplAWXoCpUsvIgxAnQFYlY20PIcxMM/I9IBXyTwE6wKK fW56UKi8+RX1DL7u5PWboJYgB0o4qp6dZKOxt//02TtG6f/wjvxr4WL8hfT2qDof1QakGo42nPf1g Ht4LoYj3F6YgCssmCtnkNGeGURcM6JMxkku1QhCe9O0ZUmhiyoUVlqwjr6yBCWvt1HxeVk2OKXEt4 inqyWj/XQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k8PdR-00026t-W1; Wed, 19 Aug 2020 15:04:18 +0000 Received: from mail-ej1-x642.google.com ([2a00:1450:4864:20::642]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k8PdO-000252-Sd for linux-nvme@lists.infradead.org; Wed, 19 Aug 2020 15:04:16 +0000 Received: by mail-ej1-x642.google.com with SMTP id t10so26682062ejs.8 for ; Wed, 19 Aug 2020 08:04:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=CmFRV9MF78Io9lyU3ZTHQoxCUWL8rgrVpgpqOj4nEXWpRTCd1+KGSochi2kuoVhtFt PQJJembCFDIORAph3WzrpDoRd2cIGJpwf9VAtP5a9ydpphhRkHDHcmKRPuhZGWASaGfl y6rPrnsIqK2MPfXnYUVbYzJtuDwuiigpqdGRc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=Sc4f3uK9msyHubmf/wvoFLsX0kPW/7BFKu3Dw3QeaPkDrxe15YWRxvULe3W/7ub6AM pDeBf/3tCPR/IRn1vV4K8FCMdQnW25iMOrDIinnud9Y+FMeOcMNrbewxyWvFPVaJeoEK 1Yc2G6yxxXeQczU25mjf9z+k0JP5g27MQ8Z9s4bUzHnixHq4YUL8Wwb6FSzZmLEoMqb6 CDO8HJb/7in3UF2gelMqLoDJO5lo3PHDPkRzd28m6AsOyxbby+2CyE1OMe/FsDwlc6q3 mVHhWbmgArV/IZiLVeSwsRwmmITDcChBidjQBlF3aTobERv4T3sgE1bcWT9Go1kSoIzJ FdgA== X-Gm-Message-State: AOAM532Vo56sbTVfJuj3R8+NFSbd3xUfZhef0OcIcGxVQZeS57JBiiKP nZrZObAj2XLhnmohaUaBRGKff3jKWFTcNg== X-Google-Smtp-Source: ABdhPJyMYSNbp7bLe8aicveCFbnikxu6cttgMc6aSDNLJXieip6SEmxBsuLNavAOCYFC70I8wuqTcw== X-Received: by 2002:a17:907:2067:: with SMTP id qp7mr25056233ejb.378.1597849452785; Wed, 19 Aug 2020 08:04:12 -0700 (PDT) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com. [209.85.128.44]) by smtp.gmail.com with ESMTPSA id s9sm20129092edt.36.2020.08.19.08.04.10 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 19 Aug 2020 08:04:10 -0700 (PDT) Received: by mail-wm1-f44.google.com with SMTP id g8so2410296wmk.3 for ; Wed, 19 Aug 2020 08:04:10 -0700 (PDT) X-Received: by 2002:a1c:4d12:: with SMTP id o18mr5316483wmh.55.1597849448826; Wed, 19 Aug 2020 08:04:08 -0700 (PDT) MIME-Version: 1.0 References: <20200819065555.1802761-1-hch@lst.de> <20200819065555.1802761-20-hch@lst.de> In-Reply-To: <20200819065555.1802761-20-hch@lst.de> From: Tomasz Figa Date: Wed, 19 Aug 2020 17:03:52 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 19/28] dma-mapping: replace DMA_ATTR_NON_CONSISTENT with dma_{alloc, free}_pages To: Christoph Hellwig X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200819_110415_025800_9B6619C0 X-CRM114-Status: GOOD ( 34.25 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alsa-devel@alsa-project.org, linux-ia64@vger.kernel.org, Linux Doc Mailing List , nouveau@lists.freedesktop.org, linux-nvme@lists.infradead.org, Linux Kernel Mailing List , "James E.J. Bottomley" , linux-mm@kvack.org, Marek Szyprowski , linux-samsung-soc , Joonyoung Shim , linux-scsi@vger.kernel.org, Kyungmin Park , Ben Skeggs , Matt Porter , Linux Media Mailing List , Tom Lendacky , Pawel Osciak , Mauro Carvalho Chehab , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , Thomas Bogendoerfer , linux-parisc@vger.kernel.org, netdev@vger.kernel.org, Seung-Woo Kim , linux-mips@vger.kernel.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel , " Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org Hi Christoph, On Wed, Aug 19, 2020 at 8:57 AM Christoph Hellwig wrote: > > Add a new API to allocate and free pages that are guaranteed to be > addressable by a device, but otherwise behave like pages allocated by > alloc_pages. The intended APIs to sync them for use with the device > and cpu are dma_sync_single_for_{device,cpu} that are also used for > streaming mappings. > > Switch all drivers over to this new API, but keep the usage of the > crufty dma_cache_sync API for now, which will be cleaned up on a driver > by driver basis. > > Signed-off-by: Christoph Hellwig > --- > Documentation/core-api/dma-api.rst | 68 +++++++++++------------ > Documentation/core-api/dma-attributes.rst | 8 --- > arch/alpha/kernel/pci_iommu.c | 2 + > arch/arm/mm/dma-mapping-nommu.c | 2 + > arch/arm/mm/dma-mapping.c | 4 ++ > arch/ia64/hp/common/sba_iommu.c | 2 + > arch/mips/jazz/jazzdma.c | 7 +-- > arch/powerpc/kernel/dma-iommu.c | 2 + > arch/powerpc/platforms/ps3/system-bus.c | 4 ++ > arch/powerpc/platforms/pseries/vio.c | 2 + > arch/s390/pci/pci_dma.c | 2 + > arch/x86/kernel/amd_gart_64.c | 2 + > drivers/iommu/dma-iommu.c | 2 + > drivers/iommu/intel/iommu.c | 4 ++ > drivers/net/ethernet/i825xx/lasi_82596.c | 13 ++--- > drivers/net/ethernet/seeq/sgiseeq.c | 12 ++-- > drivers/parisc/ccio-dma.c | 2 + > drivers/parisc/sba_iommu.c | 2 + > drivers/scsi/53c700.c | 8 +-- > drivers/scsi/sgiwd93.c | 12 ++-- > drivers/xen/swiotlb-xen.c | 2 + > include/linux/dma-direct.h | 5 ++ > include/linux/dma-mapping.h | 29 ++++++++-- > include/linux/dma-noncoherent.h | 3 - > kernel/dma/direct.c | 51 ++++++++++++++++- > kernel/dma/mapping.c | 43 +++++++++++++- > kernel/dma/ops_helpers.c | 35 ++++++++++++ > kernel/dma/virt.c | 2 + > sound/mips/hal2.c | 20 +++---- > 29 files changed, 254 insertions(+), 96 deletions(-) > Thanks for the patch. The general design looks quite nice, but please see my comments inline. > diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst > index 90239348b30f6f..047fcfffa0e5cf 100644 > --- a/Documentation/core-api/dma-api.rst > +++ b/Documentation/core-api/dma-api.rst > @@ -516,48 +516,53 @@ routines, e.g.::: > } > > > -Part II - Advanced dma usage > ----------------------------- > +Part II - Non-coherent DMA allocations > +-------------------------------------- > > -Warning: These pieces of the DMA API should not be used in the > -majority of cases, since they cater for unlikely corner cases that > -don't belong in usual drivers. > +These APIs allow to allocate pages that can be used like normal pages > +in the kernel direct mapping, but are guaranteed to be DMA addressable. Could we elaborate a bit more on what "like normal pages in kernel direct mapping" mean from the driver perspective? > > If you don't understand how cache line coherency works between a > processor and an I/O device, you should not be using this part of the > -API at all. > +API. > > :: > > void * > - dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > - gfp_t flag, unsigned long attrs) > + dma_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, > + enum dma_data_direction dir, gfp_t gfp) > + > +This routine allocates a region of bytes of consistent memory. It > +returns a pointer to the allocated region (in the processor's virtual address > +space) or NULL if the allocation failed. The returned memory is guanteed to > +behave like memory allocated using alloc_pages. There is one aspect that the existing dma_alloc_attrs() handles, but this new function doesn't: IOMMU support. The function will always allocate a physically-contiguous block memory, which is a costly operation and not even guaranteed to succeed, even if enough free memory is available. Modern SoCs employ IOMMUs to avoid the need to allocate physically-contiguous memory and those happen to be also the devices that could benefit from non-coherent allocations a lot. One of the tasks of the DMA API was making it possible to allocate suitable memory for a given device, without having the driver know about the SoC integration details, such as the presence of an IOMMU. Today, dma_alloc_attrs() uses the .alloc callback of the dma_ops struct and the IOMMU-aware implementations, like the dma-iommu helpers [1], would allocate discontiguous pages. Therefore, while I see the DMA-aware page allocation functionality as a useful functionality on its own for scatter-gather-capable hardware, I believe it is not a complete replacement for dma_alloc_attrs() with the DMA_ATTR_NON_CONSISTENT attribute. [1] https://elixir.bootlin.com/linux/v5.9-rc1/source/drivers/iommu/dma-iommu.c#L510 Best regards, Tomasz _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D7FEC433E1 for ; Wed, 19 Aug 2020 15:04:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B9A8B20882 for ; Wed, 19 Aug 2020 15:04:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CmFRV9MF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B9A8B20882 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 645558D004D; Wed, 19 Aug 2020 11:04:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61C4B8D0013; Wed, 19 Aug 2020 11:04:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5314D8D004D; Wed, 19 Aug 2020 11:04:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id 36ADF8D0013 for ; Wed, 19 Aug 2020 11:04:44 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 609CE441E for ; Wed, 19 Aug 2020 15:04:43 +0000 (UTC) X-FDA: 77167640046.07.lace87_3608faf27028 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 95A521803F922 for ; Wed, 19 Aug 2020 15:04:24 +0000 (UTC) X-HE-Tag: lace87_3608faf27028 X-Filterd-Recvd-Size: 9636 Received: from mail-ej1-f65.google.com (mail-ej1-f65.google.com [209.85.218.65]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Aug 2020 15:04:13 +0000 (UTC) Received: by mail-ej1-f65.google.com with SMTP id f24so26725326ejx.6 for ; Wed, 19 Aug 2020 08:04:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=CmFRV9MF78Io9lyU3ZTHQoxCUWL8rgrVpgpqOj4nEXWpRTCd1+KGSochi2kuoVhtFt PQJJembCFDIORAph3WzrpDoRd2cIGJpwf9VAtP5a9ydpphhRkHDHcmKRPuhZGWASaGfl y6rPrnsIqK2MPfXnYUVbYzJtuDwuiigpqdGRc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=R08Wowjcs00pmW6nGjPCFKtPuWVqhXhbtIftJwpXW9Q9t2gv4VK7V6P+CoInZGISuA Sy9/dX7L18r8VaG026AMCWOYEW+uWzIoUjGE5hFDWCjQRnVFvQPeX+ovVu1i637m0Q+0 b8clyoxi7+rzX6mPgLZUgnlEdWF3gpZcPyQ8jelxfka6L/5k9fhxCBPvstC+SWBnSrV0 Dl70DqzjVT0pO9FPHPWU6VfWZlKtdsDY2tUTn/Uzl5Bc8dM/5ov6dkJGjpr0/uvJc9nz igcf4f/ZenHoO1g3U1p7fiH6IOEJ8Exsnh2o4MU44gDRkVJ1CVzdzfiNKNhEPrmJ+wzi Y0Qg== X-Gm-Message-State: AOAM530M7Nmj/C8/KTQAaT7iTAgYziDkr+9KDyPb+Ui+WzOv6rnKPNR0 aknOo2gFTBpfzXLQglmvoKx77TGuugXBsA== X-Google-Smtp-Source: ABdhPJyBLA5mMiOMKp3fnqDYbV2j+t3VjK+zQ500AFKZ7SP3Nh2ZVf+rQbAE/Rn1jqCBo/UqzFfqgA== X-Received: by 2002:a17:906:841:: with SMTP id f1mr4738067ejd.158.1597849451879; Wed, 19 Aug 2020 08:04:11 -0700 (PDT) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com. [209.85.128.42]) by smtp.gmail.com with ESMTPSA id b24sm18514220edn.33.2020.08.19.08.04.10 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 19 Aug 2020 08:04:10 -0700 (PDT) Received: by mail-wm1-f42.google.com with SMTP id 3so2540266wmi.1 for ; Wed, 19 Aug 2020 08:04:10 -0700 (PDT) X-Received: by 2002:a1c:4d12:: with SMTP id o18mr5316483wmh.55.1597849448826; Wed, 19 Aug 2020 08:04:08 -0700 (PDT) MIME-Version: 1.0 References: <20200819065555.1802761-1-hch@lst.de> <20200819065555.1802761-20-hch@lst.de> In-Reply-To: <20200819065555.1802761-20-hch@lst.de> From: Tomasz Figa Date: Wed, 19 Aug 2020 17:03:52 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 19/28] dma-mapping: replace DMA_ATTR_NON_CONSISTENT with dma_{alloc, free}_pages To: Christoph Hellwig Cc: Mauro Carvalho Chehab , Thomas Bogendoerfer , "James E.J. Bottomley" , Joonyoung Shim , Seung-Woo Kim , Kyungmin Park , Ben Skeggs , Pawel Osciak , Marek Szyprowski , Matt Porter , "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , Tom Lendacky , alsa-devel@alsa-project.org, linux-samsung-soc , linux-ia64@vger.kernel.org, linux-scsi@vger.kernel.org, linux-parisc@vger.kernel.org, Linux Doc Mailing List , nouveau@lists.freedesktop.org, Linux Kernel Mailing List , linux-nvme@lists.infradead.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , Linux Media Mailing List Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 95A521803F922 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Christoph, On Wed, Aug 19, 2020 at 8:57 AM Christoph Hellwig wrote: > > Add a new API to allocate and free pages that are guaranteed to be > addressable by a device, but otherwise behave like pages allocated by > alloc_pages. The intended APIs to sync them for use with the device > and cpu are dma_sync_single_for_{device,cpu} that are also used for > streaming mappings. > > Switch all drivers over to this new API, but keep the usage of the > crufty dma_cache_sync API for now, which will be cleaned up on a driver > by driver basis. > > Signed-off-by: Christoph Hellwig > --- > Documentation/core-api/dma-api.rst | 68 +++++++++++------------ > Documentation/core-api/dma-attributes.rst | 8 --- > arch/alpha/kernel/pci_iommu.c | 2 + > arch/arm/mm/dma-mapping-nommu.c | 2 + > arch/arm/mm/dma-mapping.c | 4 ++ > arch/ia64/hp/common/sba_iommu.c | 2 + > arch/mips/jazz/jazzdma.c | 7 +-- > arch/powerpc/kernel/dma-iommu.c | 2 + > arch/powerpc/platforms/ps3/system-bus.c | 4 ++ > arch/powerpc/platforms/pseries/vio.c | 2 + > arch/s390/pci/pci_dma.c | 2 + > arch/x86/kernel/amd_gart_64.c | 2 + > drivers/iommu/dma-iommu.c | 2 + > drivers/iommu/intel/iommu.c | 4 ++ > drivers/net/ethernet/i825xx/lasi_82596.c | 13 ++--- > drivers/net/ethernet/seeq/sgiseeq.c | 12 ++-- > drivers/parisc/ccio-dma.c | 2 + > drivers/parisc/sba_iommu.c | 2 + > drivers/scsi/53c700.c | 8 +-- > drivers/scsi/sgiwd93.c | 12 ++-- > drivers/xen/swiotlb-xen.c | 2 + > include/linux/dma-direct.h | 5 ++ > include/linux/dma-mapping.h | 29 ++++++++-- > include/linux/dma-noncoherent.h | 3 - > kernel/dma/direct.c | 51 ++++++++++++++++- > kernel/dma/mapping.c | 43 +++++++++++++- > kernel/dma/ops_helpers.c | 35 ++++++++++++ > kernel/dma/virt.c | 2 + > sound/mips/hal2.c | 20 +++---- > 29 files changed, 254 insertions(+), 96 deletions(-) > Thanks for the patch. The general design looks quite nice, but please see my comments inline. > diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst > index 90239348b30f6f..047fcfffa0e5cf 100644 > --- a/Documentation/core-api/dma-api.rst > +++ b/Documentation/core-api/dma-api.rst > @@ -516,48 +516,53 @@ routines, e.g.::: > } > > > -Part II - Advanced dma usage > ----------------------------- > +Part II - Non-coherent DMA allocations > +-------------------------------------- > > -Warning: These pieces of the DMA API should not be used in the > -majority of cases, since they cater for unlikely corner cases that > -don't belong in usual drivers. > +These APIs allow to allocate pages that can be used like normal pages > +in the kernel direct mapping, but are guaranteed to be DMA addressable. Could we elaborate a bit more on what "like normal pages in kernel direct mapping" mean from the driver perspective? > > If you don't understand how cache line coherency works between a > processor and an I/O device, you should not be using this part of the > -API at all. > +API. > > :: > > void * > - dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > - gfp_t flag, unsigned long attrs) > + dma_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, > + enum dma_data_direction dir, gfp_t gfp) > + > +This routine allocates a region of bytes of consistent memory. It > +returns a pointer to the allocated region (in the processor's virtual address > +space) or NULL if the allocation failed. The returned memory is guanteed to > +behave like memory allocated using alloc_pages. There is one aspect that the existing dma_alloc_attrs() handles, but this new function doesn't: IOMMU support. The function will always allocate a physically-contiguous block memory, which is a costly operation and not even guaranteed to succeed, even if enough free memory is available. Modern SoCs employ IOMMUs to avoid the need to allocate physically-contiguous memory and those happen to be also the devices that could benefit from non-coherent allocations a lot. One of the tasks of the DMA API was making it possible to allocate suitable memory for a given device, without having the driver know about the SoC integration details, such as the presence of an IOMMU. Today, dma_alloc_attrs() uses the .alloc callback of the dma_ops struct and the IOMMU-aware implementations, like the dma-iommu helpers [1], would allocate discontiguous pages. Therefore, while I see the DMA-aware page allocation functionality as a useful functionality on its own for scatter-gather-capable hardware, I believe it is not a complete replacement for dma_alloc_attrs() with the DMA_ATTR_NON_CONSISTENT attribute. [1] https://elixir.bootlin.com/linux/v5.9-rc1/source/drivers/iommu/dma-iommu.c#L510 Best regards, Tomasz From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FDAFC433E1 for ; Fri, 21 Aug 2020 07:53:19 +0000 (UTC) Received: from alsa0.perex.cz (alsa0.perex.cz [77.48.224.243]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AF304207DF for ; Fri, 21 Aug 2020 07:53:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=alsa-project.org header.i=@alsa-project.org header.b="MRZgANVN"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CmFRV9MF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF304207DF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=alsa-devel-bounces@alsa-project.org Received: from alsa1.perex.cz (alsa1.perex.cz [207.180.221.201]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by alsa0.perex.cz (Postfix) with ESMTPS id 462B216B5; Fri, 21 Aug 2020 09:52:27 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.11.0 alsa0.perex.cz 462B216B5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=alsa-project.org; s=default; t=1597996397; bh=GazL5RuGzNsltJmK5TVSYc5vxyNguEsJt8YXpsvAPSI=; h=References:In-Reply-To:From:Date:Subject:To:Cc:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From; b=MRZgANVNykoTaPtSelWfylm3e/dfPGuwwSBG5lJRCGabK5v+tFCxYuCD/t06mZbKP Ojq+rrZ826BwjIcZFwluvfaV9xyNCo9BVAkTmgajEWkYbdpzfx7F4c6tDkOlh/cR3B +r0IwVzHehpwhFmE8ORIKNT6uJDhctSZXC76h3lo= Received: from alsa1.perex.cz (localhost.localdomain [127.0.0.1]) by alsa1.perex.cz (Postfix) with ESMTP id 5A95DF80406; Fri, 21 Aug 2020 09:36:31 +0200 (CEST) Received: by alsa1.perex.cz (Postfix, from userid 50401) id 6BDEBF80218; Wed, 19 Aug 2020 17:04:18 +0200 (CEST) Received: from mail-ej1-x644.google.com (mail-ej1-x644.google.com [IPv6:2a00:1450:4864:20::644]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by alsa1.perex.cz (Postfix) with ESMTPS id B3D7AF80114 for ; Wed, 19 Aug 2020 17:04:11 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.11.0 alsa1.perex.cz B3D7AF80114 Authentication-Results: alsa1.perex.cz; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CmFRV9MF" Received: by mail-ej1-x644.google.com with SMTP id kq25so26684548ejb.3 for ; Wed, 19 Aug 2020 08:04:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=CmFRV9MF78Io9lyU3ZTHQoxCUWL8rgrVpgpqOj4nEXWpRTCd1+KGSochi2kuoVhtFt PQJJembCFDIORAph3WzrpDoRd2cIGJpwf9VAtP5a9ydpphhRkHDHcmKRPuhZGWASaGfl y6rPrnsIqK2MPfXnYUVbYzJtuDwuiigpqdGRc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=CSGJGCubQpyo4Q4GD3zJIqqKDjh/euH4WMhe2V6FFDZsphLA3+hKJm8HNh+vTVyiIz aE4cCvh9pJbZI74u9Gt20APobyRZyB5HtTWG8mwSVqCTHDZXjgYYQt3Y8fwXPfbHM2mm E2t8qbkee/Jiac9ID10aVa9KNyW6C43nP6XbEJpOKDV2YbuL5tlRyBlQHYkSnK0rbFJF v6GPU+AwCa1v/0+86Am36zhs+srkqA8kep9Ru6ugbgwKvQx5yA+alhztPJwIflAbz9fg kPO80eMW8bNdGjE7AXSc9DNWL1MvzRy7QAUgFeSxz8TTOuoEDJHH/ruwGhLq9PdvFfRX ORyg== X-Gm-Message-State: AOAM532nCEk2H3BlXacbK5BozQhaahs254WSFda/rpflW2v8n7+ANKSk hFN5uTOtRvLE7fBmKzZdQ67Pj9BKEa0mCg== X-Google-Smtp-Source: ABdhPJx7STbHcpALOvv+6s3flDWW7G/giqrBYwSt0SGKwdw2CLVCWBfhgM6F9jXe0OkC6ExaOpy6CQ== X-Received: by 2002:a17:906:b6d5:: with SMTP id ec21mr24662234ejb.396.1597849450876; Wed, 19 Aug 2020 08:04:10 -0700 (PDT) Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com. [209.85.128.53]) by smtp.gmail.com with ESMTPSA id gh25sm18583495ejb.109.2020.08.19.08.04.09 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 19 Aug 2020 08:04:10 -0700 (PDT) Received: by mail-wm1-f53.google.com with SMTP id c80so2416555wme.0 for ; Wed, 19 Aug 2020 08:04:09 -0700 (PDT) X-Received: by 2002:a1c:4d12:: with SMTP id o18mr5316483wmh.55.1597849448826; Wed, 19 Aug 2020 08:04:08 -0700 (PDT) MIME-Version: 1.0 References: <20200819065555.1802761-1-hch@lst.de> <20200819065555.1802761-20-hch@lst.de> In-Reply-To: <20200819065555.1802761-20-hch@lst.de> From: Tomasz Figa Date: Wed, 19 Aug 2020 17:03:52 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 19/28] dma-mapping: replace DMA_ATTR_NON_CONSISTENT with dma_{alloc, free}_pages To: Christoph Hellwig Content-Type: text/plain; charset="UTF-8" X-Mailman-Approved-At: Fri, 21 Aug 2020 09:36:14 +0200 Cc: alsa-devel@alsa-project.org, linux-ia64@vger.kernel.org, Linux Doc Mailing List , nouveau@lists.freedesktop.org, linux-nvme@lists.infradead.org, Linux Kernel Mailing List , "James E.J. Bottomley" , linux-mm@kvack.org, Marek Szyprowski , linux-samsung-soc , Joonyoung Shim , linux-scsi@vger.kernel.org, Kyungmin Park , Ben Skeggs , Matt Porter , Linux Media Mailing List , Tom Lendacky , Pawel Osciak , Mauro Carvalho Chehab , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , Thomas Bogendoerfer , linux-parisc@vger.kernel.org, netdev@vger.kernel.org, Seung-Woo Kim , linux-mips@vger.kernel.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel , " X-BeenThere: alsa-devel@alsa-project.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Alsa-devel mailing list for ALSA developers - http://www.alsa-project.org" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: alsa-devel-bounces@alsa-project.org Sender: "Alsa-devel" Hi Christoph, On Wed, Aug 19, 2020 at 8:57 AM Christoph Hellwig wrote: > > Add a new API to allocate and free pages that are guaranteed to be > addressable by a device, but otherwise behave like pages allocated by > alloc_pages. The intended APIs to sync them for use with the device > and cpu are dma_sync_single_for_{device,cpu} that are also used for > streaming mappings. > > Switch all drivers over to this new API, but keep the usage of the > crufty dma_cache_sync API for now, which will be cleaned up on a driver > by driver basis. > > Signed-off-by: Christoph Hellwig > --- > Documentation/core-api/dma-api.rst | 68 +++++++++++------------ > Documentation/core-api/dma-attributes.rst | 8 --- > arch/alpha/kernel/pci_iommu.c | 2 + > arch/arm/mm/dma-mapping-nommu.c | 2 + > arch/arm/mm/dma-mapping.c | 4 ++ > arch/ia64/hp/common/sba_iommu.c | 2 + > arch/mips/jazz/jazzdma.c | 7 +-- > arch/powerpc/kernel/dma-iommu.c | 2 + > arch/powerpc/platforms/ps3/system-bus.c | 4 ++ > arch/powerpc/platforms/pseries/vio.c | 2 + > arch/s390/pci/pci_dma.c | 2 + > arch/x86/kernel/amd_gart_64.c | 2 + > drivers/iommu/dma-iommu.c | 2 + > drivers/iommu/intel/iommu.c | 4 ++ > drivers/net/ethernet/i825xx/lasi_82596.c | 13 ++--- > drivers/net/ethernet/seeq/sgiseeq.c | 12 ++-- > drivers/parisc/ccio-dma.c | 2 + > drivers/parisc/sba_iommu.c | 2 + > drivers/scsi/53c700.c | 8 +-- > drivers/scsi/sgiwd93.c | 12 ++-- > drivers/xen/swiotlb-xen.c | 2 + > include/linux/dma-direct.h | 5 ++ > include/linux/dma-mapping.h | 29 ++++++++-- > include/linux/dma-noncoherent.h | 3 - > kernel/dma/direct.c | 51 ++++++++++++++++- > kernel/dma/mapping.c | 43 +++++++++++++- > kernel/dma/ops_helpers.c | 35 ++++++++++++ > kernel/dma/virt.c | 2 + > sound/mips/hal2.c | 20 +++---- > 29 files changed, 254 insertions(+), 96 deletions(-) > Thanks for the patch. The general design looks quite nice, but please see my comments inline. > diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst > index 90239348b30f6f..047fcfffa0e5cf 100644 > --- a/Documentation/core-api/dma-api.rst > +++ b/Documentation/core-api/dma-api.rst > @@ -516,48 +516,53 @@ routines, e.g.::: > } > > > -Part II - Advanced dma usage > ----------------------------- > +Part II - Non-coherent DMA allocations > +-------------------------------------- > > -Warning: These pieces of the DMA API should not be used in the > -majority of cases, since they cater for unlikely corner cases that > -don't belong in usual drivers. > +These APIs allow to allocate pages that can be used like normal pages > +in the kernel direct mapping, but are guaranteed to be DMA addressable. Could we elaborate a bit more on what "like normal pages in kernel direct mapping" mean from the driver perspective? > > If you don't understand how cache line coherency works between a > processor and an I/O device, you should not be using this part of the > -API at all. > +API. > > :: > > void * > - dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > - gfp_t flag, unsigned long attrs) > + dma_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, > + enum dma_data_direction dir, gfp_t gfp) > + > +This routine allocates a region of bytes of consistent memory. It > +returns a pointer to the allocated region (in the processor's virtual address > +space) or NULL if the allocation failed. The returned memory is guanteed to > +behave like memory allocated using alloc_pages. There is one aspect that the existing dma_alloc_attrs() handles, but this new function doesn't: IOMMU support. The function will always allocate a physically-contiguous block memory, which is a costly operation and not even guaranteed to succeed, even if enough free memory is available. Modern SoCs employ IOMMUs to avoid the need to allocate physically-contiguous memory and those happen to be also the devices that could benefit from non-coherent allocations a lot. One of the tasks of the DMA API was making it possible to allocate suitable memory for a given device, without having the driver know about the SoC integration details, such as the presence of an IOMMU. Today, dma_alloc_attrs() uses the .alloc callback of the dma_ops struct and the IOMMU-aware implementations, like the dma-iommu helpers [1], would allocate discontiguous pages. Therefore, while I see the DMA-aware page allocation functionality as a useful functionality on its own for scatter-gather-capable hardware, I believe it is not a complete replacement for dma_alloc_attrs() with the DMA_ATTR_NON_CONSISTENT attribute. [1] https://elixir.bootlin.com/linux/v5.9-rc1/source/drivers/iommu/dma-iommu.c#L510 Best regards, Tomasz From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tomasz Figa Subject: Re: [PATCH 19/28] dma-mapping: replace DMA_ATTR_NON_CONSISTENT with dma_{alloc, free}_pages Date: Wed, 19 Aug 2020 17:03:52 +0200 Message-ID: References: <20200819065555.1802761-1-hch@lst.de> <20200819065555.1802761-20-hch@lst.de> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: In-Reply-To: <20200819065555.1802761-20-hch@lst.de> Sender: linux-ia64-owner@vger.kernel.org To: Christoph Hellwig Cc: Mauro Carvalho Chehab , Thomas Bogendoerfer , "James E.J. Bottomley" , Joonyoung Shim , Seung-Woo Kim , Kyungmin Park , Ben Skeggs , Pawel Osciak , Marek Szyprowski , Matt Porter , "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , Tom Lendacky , alsa-devel@alsa-project.org, linux-samsung-soc , linux-ia64@vger.kernel.org, linux-scsi@vger.kernel.org, linux-parisc@vger.k List-Id: nouveau.vger.kernel.org Hi Christoph, On Wed, Aug 19, 2020 at 8:57 AM Christoph Hellwig wrote: > > Add a new API to allocate and free pages that are guaranteed to be > addressable by a device, but otherwise behave like pages allocated by > alloc_pages. The intended APIs to sync them for use with the device > and cpu are dma_sync_single_for_{device,cpu} that are also used for > streaming mappings. > > Switch all drivers over to this new API, but keep the usage of the > crufty dma_cache_sync API for now, which will be cleaned up on a driver > by driver basis. > > Signed-off-by: Christoph Hellwig > --- > Documentation/core-api/dma-api.rst | 68 +++++++++++------------ > Documentation/core-api/dma-attributes.rst | 8 --- > arch/alpha/kernel/pci_iommu.c | 2 + > arch/arm/mm/dma-mapping-nommu.c | 2 + > arch/arm/mm/dma-mapping.c | 4 ++ > arch/ia64/hp/common/sba_iommu.c | 2 + > arch/mips/jazz/jazzdma.c | 7 +-- > arch/powerpc/kernel/dma-iommu.c | 2 + > arch/powerpc/platforms/ps3/system-bus.c | 4 ++ > arch/powerpc/platforms/pseries/vio.c | 2 + > arch/s390/pci/pci_dma.c | 2 + > arch/x86/kernel/amd_gart_64.c | 2 + > drivers/iommu/dma-iommu.c | 2 + > drivers/iommu/intel/iommu.c | 4 ++ > drivers/net/ethernet/i825xx/lasi_82596.c | 13 ++--- > drivers/net/ethernet/seeq/sgiseeq.c | 12 ++-- > drivers/parisc/ccio-dma.c | 2 + > drivers/parisc/sba_iommu.c | 2 + > drivers/scsi/53c700.c | 8 +-- > drivers/scsi/sgiwd93.c | 12 ++-- > drivers/xen/swiotlb-xen.c | 2 + > include/linux/dma-direct.h | 5 ++ > include/linux/dma-mapping.h | 29 ++++++++-- > include/linux/dma-noncoherent.h | 3 - > kernel/dma/direct.c | 51 ++++++++++++++++- > kernel/dma/mapping.c | 43 +++++++++++++- > kernel/dma/ops_helpers.c | 35 ++++++++++++ > kernel/dma/virt.c | 2 + > sound/mips/hal2.c | 20 +++---- > 29 files changed, 254 insertions(+), 96 deletions(-) > Thanks for the patch. The general design looks quite nice, but please see my comments inline. > diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst > index 90239348b30f6f..047fcfffa0e5cf 100644 > --- a/Documentation/core-api/dma-api.rst > +++ b/Documentation/core-api/dma-api.rst > @@ -516,48 +516,53 @@ routines, e.g.::: > } > > > -Part II - Advanced dma usage > ----------------------------- > +Part II - Non-coherent DMA allocations > +-------------------------------------- > > -Warning: These pieces of the DMA API should not be used in the > -majority of cases, since they cater for unlikely corner cases that > -don't belong in usual drivers. > +These APIs allow to allocate pages that can be used like normal pages > +in the kernel direct mapping, but are guaranteed to be DMA addressable. Could we elaborate a bit more on what "like normal pages in kernel direct mapping" mean from the driver perspective? > > If you don't understand how cache line coherency works between a > processor and an I/O device, you should not be using this part of the > -API at all. > +API. > > :: > > void * > - dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > - gfp_t flag, unsigned long attrs) > + dma_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, > + enum dma_data_direction dir, gfp_t gfp) > + > +This routine allocates a region of bytes of consistent memory. It > +returns a pointer to the allocated region (in the processor's virtual address > +space) or NULL if the allocation failed. The returned memory is guanteed to > +behave like memory allocated using alloc_pages. There is one aspect that the existing dma_alloc_attrs() handles, but this new function doesn't: IOMMU support. The function will always allocate a physically-contiguous block memory, which is a costly operation and not even guaranteed to succeed, even if enough free memory is available. Modern SoCs employ IOMMUs to avoid the need to allocate physically-contiguous memory and those happen to be also the devices that could benefit from non-coherent allocations a lot. One of the tasks of the DMA API was making it possible to allocate suitable memory for a given device, without having the driver know about the SoC integration details, such as the presence of an IOMMU. Today, dma_alloc_attrs() uses the .alloc callback of the dma_ops struct and the IOMMU-aware implementations, like the dma-iommu helpers [1], would allocate discontiguous pages. Therefore, while I see the DMA-aware page allocation functionality as a useful functionality on its own for scatter-gather-capable hardware, I believe it is not a complete replacement for dma_alloc_attrs() with the DMA_ATTR_NON_CONSISTENT attribute. [1] https://elixir.bootlin.com/linux/v5.9-rc1/source/drivers/iommu/dma-iommu.c#L510 Best regards, Tomasz From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75298C433E1 for ; Wed, 19 Aug 2020 15:04:19 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 379E72076E for ; Wed, 19 Aug 2020 15:04:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CmFRV9MF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 379E72076E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id F41A62041B; Wed, 19 Aug 2020 15:04:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L4VmJVowdudo; Wed, 19 Aug 2020 15:04:16 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by silver.osuosl.org (Postfix) with ESMTP id C9877204B3; Wed, 19 Aug 2020 15:04:16 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A1ACAC07FF; Wed, 19 Aug 2020 15:04:16 +0000 (UTC) Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id E0EFDC0051 for ; Wed, 19 Aug 2020 15:04:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id CA8C985074 for ; Wed, 19 Aug 2020 15:04:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eo2u+Aiteku2 for ; Wed, 19 Aug 2020 15:04:14 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ej1-f67.google.com (mail-ej1-f67.google.com [209.85.218.67]) by whitealder.osuosl.org (Postfix) with ESMTPS id A726984F57 for ; Wed, 19 Aug 2020 15:04:13 +0000 (UTC) Received: by mail-ej1-f67.google.com with SMTP id qc22so26697837ejb.4 for ; Wed, 19 Aug 2020 08:04:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=CmFRV9MF78Io9lyU3ZTHQoxCUWL8rgrVpgpqOj4nEXWpRTCd1+KGSochi2kuoVhtFt PQJJembCFDIORAph3WzrpDoRd2cIGJpwf9VAtP5a9ydpphhRkHDHcmKRPuhZGWASaGfl y6rPrnsIqK2MPfXnYUVbYzJtuDwuiigpqdGRc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=bH+HfAS1UAF4ay5Tm1OFw828F/+GmZYoXKHLjZ0USpGbQ8rFxORf43E7OfVQDZZrj1 Ll5CpnzYcuH9zKKHVb6UtG9/tHqoZV7aOW0xDWj6kB6W3ez0+SuTualRjA8hW8IghOGV +c94xoAo9EfIgejcSWl6EuD8F+FydMjaRqXOhwzXUkA+wSqln5sS4exqCrv9Ij6EUY9/ VLH1nwDmuDF9HqjVbo9k2f18rvjePWbi7af8Q7aJJIZHFBOsMHZDrUAmTp3xCQBQEX7f V5fXAfcN9DmeFFYJTfUJVrnGimrnKxgNphhEU/ie4ks7UeRzVmFCNn0GMliYCsJE5Nxw mn7Q== X-Gm-Message-State: AOAM533glVSihvSVexOeoj6b3ixc3mYHdv1J9iKCx0cbHLojE0GqoIcF l/bm2iyFN+13cXu/GminUm4Eaf/EyUClYw== X-Google-Smtp-Source: ABdhPJyfmLnQMxShqr2kSx31t9W9J4DB4BZWAwFCMWqZqNhk3avNVk/GC0lDmUmRG+iDedVMqXeHrw== X-Received: by 2002:a17:906:6d91:: with SMTP id h17mr24355691ejt.531.1597849451712; Wed, 19 Aug 2020 08:04:11 -0700 (PDT) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com. [209.85.128.42]) by smtp.gmail.com with ESMTPSA id j7sm18927011ejb.64.2020.08.19.08.04.09 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 19 Aug 2020 08:04:10 -0700 (PDT) Received: by mail-wm1-f42.google.com with SMTP id f18so1857832wmc.0 for ; Wed, 19 Aug 2020 08:04:09 -0700 (PDT) X-Received: by 2002:a1c:4d12:: with SMTP id o18mr5316483wmh.55.1597849448826; Wed, 19 Aug 2020 08:04:08 -0700 (PDT) MIME-Version: 1.0 References: <20200819065555.1802761-1-hch@lst.de> <20200819065555.1802761-20-hch@lst.de> In-Reply-To: <20200819065555.1802761-20-hch@lst.de> From: Tomasz Figa Date: Wed, 19 Aug 2020 17:03:52 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 19/28] dma-mapping: replace DMA_ATTR_NON_CONSISTENT with dma_{alloc, free}_pages To: Christoph Hellwig Cc: alsa-devel@alsa-project.org, linux-ia64@vger.kernel.org, Linux Doc Mailing List , nouveau@lists.freedesktop.org, linux-nvme@lists.infradead.org, Linux Kernel Mailing List , "James E.J. Bottomley" , linux-mm@kvack.org, linux-samsung-soc , Joonyoung Shim , linux-scsi@vger.kernel.org, Kyungmin Park , Ben Skeggs , Matt Porter , Linux Media Mailing List , Tom Lendacky , Pawel Osciak , Mauro Carvalho Chehab , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , Thomas Bogendoerfer , linux-parisc@vger.kernel.org, netdev@vger.kernel.org, Seung-Woo Kim , linux-mips@vger.kernel.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel , " X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" Hi Christoph, On Wed, Aug 19, 2020 at 8:57 AM Christoph Hellwig wrote: > > Add a new API to allocate and free pages that are guaranteed to be > addressable by a device, but otherwise behave like pages allocated by > alloc_pages. The intended APIs to sync them for use with the device > and cpu are dma_sync_single_for_{device,cpu} that are also used for > streaming mappings. > > Switch all drivers over to this new API, but keep the usage of the > crufty dma_cache_sync API for now, which will be cleaned up on a driver > by driver basis. > > Signed-off-by: Christoph Hellwig > --- > Documentation/core-api/dma-api.rst | 68 +++++++++++------------ > Documentation/core-api/dma-attributes.rst | 8 --- > arch/alpha/kernel/pci_iommu.c | 2 + > arch/arm/mm/dma-mapping-nommu.c | 2 + > arch/arm/mm/dma-mapping.c | 4 ++ > arch/ia64/hp/common/sba_iommu.c | 2 + > arch/mips/jazz/jazzdma.c | 7 +-- > arch/powerpc/kernel/dma-iommu.c | 2 + > arch/powerpc/platforms/ps3/system-bus.c | 4 ++ > arch/powerpc/platforms/pseries/vio.c | 2 + > arch/s390/pci/pci_dma.c | 2 + > arch/x86/kernel/amd_gart_64.c | 2 + > drivers/iommu/dma-iommu.c | 2 + > drivers/iommu/intel/iommu.c | 4 ++ > drivers/net/ethernet/i825xx/lasi_82596.c | 13 ++--- > drivers/net/ethernet/seeq/sgiseeq.c | 12 ++-- > drivers/parisc/ccio-dma.c | 2 + > drivers/parisc/sba_iommu.c | 2 + > drivers/scsi/53c700.c | 8 +-- > drivers/scsi/sgiwd93.c | 12 ++-- > drivers/xen/swiotlb-xen.c | 2 + > include/linux/dma-direct.h | 5 ++ > include/linux/dma-mapping.h | 29 ++++++++-- > include/linux/dma-noncoherent.h | 3 - > kernel/dma/direct.c | 51 ++++++++++++++++- > kernel/dma/mapping.c | 43 +++++++++++++- > kernel/dma/ops_helpers.c | 35 ++++++++++++ > kernel/dma/virt.c | 2 + > sound/mips/hal2.c | 20 +++---- > 29 files changed, 254 insertions(+), 96 deletions(-) > Thanks for the patch. The general design looks quite nice, but please see my comments inline. > diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst > index 90239348b30f6f..047fcfffa0e5cf 100644 > --- a/Documentation/core-api/dma-api.rst > +++ b/Documentation/core-api/dma-api.rst > @@ -516,48 +516,53 @@ routines, e.g.::: > } > > > -Part II - Advanced dma usage > ----------------------------- > +Part II - Non-coherent DMA allocations > +-------------------------------------- > > -Warning: These pieces of the DMA API should not be used in the > -majority of cases, since they cater for unlikely corner cases that > -don't belong in usual drivers. > +These APIs allow to allocate pages that can be used like normal pages > +in the kernel direct mapping, but are guaranteed to be DMA addressable. Could we elaborate a bit more on what "like normal pages in kernel direct mapping" mean from the driver perspective? > > If you don't understand how cache line coherency works between a > processor and an I/O device, you should not be using this part of the > -API at all. > +API. > > :: > > void * > - dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > - gfp_t flag, unsigned long attrs) > + dma_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, > + enum dma_data_direction dir, gfp_t gfp) > + > +This routine allocates a region of bytes of consistent memory. It > +returns a pointer to the allocated region (in the processor's virtual address > +space) or NULL if the allocation failed. The returned memory is guanteed to > +behave like memory allocated using alloc_pages. There is one aspect that the existing dma_alloc_attrs() handles, but this new function doesn't: IOMMU support. The function will always allocate a physically-contiguous block memory, which is a costly operation and not even guaranteed to succeed, even if enough free memory is available. Modern SoCs employ IOMMUs to avoid the need to allocate physically-contiguous memory and those happen to be also the devices that could benefit from non-coherent allocations a lot. One of the tasks of the DMA API was making it possible to allocate suitable memory for a given device, without having the driver know about the SoC integration details, such as the presence of an IOMMU. Today, dma_alloc_attrs() uses the .alloc callback of the dma_ops struct and the IOMMU-aware implementations, like the dma-iommu helpers [1], would allocate discontiguous pages. Therefore, while I see the DMA-aware page allocation functionality as a useful functionality on its own for scatter-gather-capable hardware, I believe it is not a complete replacement for dma_alloc_attrs() with the DMA_ATTR_NON_CONSISTENT attribute. [1] https://elixir.bootlin.com/linux/v5.9-rc1/source/drivers/iommu/dma-iommu.c#L510 Best regards, Tomasz _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A76D4C433E1 for ; Wed, 19 Aug 2020 15:13:44 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 776CA20882 for ; Wed, 19 Aug 2020 15:13:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="1lVqpSbb"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CmFRV9MF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 776CA20882 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=zbAERLNSa4LMvRD9SDNXHzeDq9AWS6YHwcJxzCBetqI=; b=1lVqpSbbMaTODtpmNSUY+5C4p SlzO9kR0dsgrpfw7aqSEprc45qnXs1mvO97Fq1YZ1doD4pdeqJTMqYHp2oFMiCn0PszHdHLYOmqOd EwgkLBrQpm0xDUsCexMA14NxJZmfrO+pzaP5+RF4Elup4SDVfhlmVhHyvXFO/FPM1cLIknaPwSO6h QOyT7IGZRQYRdiSh+iVhBg2ZUCxrcL5QKO/fJJAG0G/hFW2lHbi1+JfwXfKMc3VfQXrmC20TZ32aZ 7NH8N/qM/t5R38oiG87lUX4W/JsCpqyqibVWVWljOferrmwZCAc7YlxGu9eIvy3IFPs+B0Q282+S7 1A2+RQpfQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k8Pko-0003wl-5c; Wed, 19 Aug 2020 15:11:54 +0000 Received: from mail-ej1-x641.google.com ([2a00:1450:4864:20::641]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k8Pkl-0003wA-7A for linux-arm-kernel@lists.infradead.org; Wed, 19 Aug 2020 15:11:53 +0000 Received: by mail-ej1-x641.google.com with SMTP id jp10so26763477ejb.0 for ; Wed, 19 Aug 2020 08:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=CmFRV9MF78Io9lyU3ZTHQoxCUWL8rgrVpgpqOj4nEXWpRTCd1+KGSochi2kuoVhtFt PQJJembCFDIORAph3WzrpDoRd2cIGJpwf9VAtP5a9ydpphhRkHDHcmKRPuhZGWASaGfl y6rPrnsIqK2MPfXnYUVbYzJtuDwuiigpqdGRc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gblnYCzc62i56nCc5LRWss+Qf5wc+kIzyZ60hUww65Y=; b=RdCPfR/Sf6x8cja+kofLMv1uMypUyvm7JyGot4GkyJut9fL49Ojdu0u6lBTfwn080O Sc1hMDOpWUaMrasQ/rY4Z7odDreIkqiDCTE/2BRuACK6hNYwQcPRKUyzihgeON0+0wi8 DaxtD3Z/kebT47YssdMhdqAlBpp3fnJjZdFA8xT6CPF+PH2ozY1YoO0888GDE/ffJjUi 8Z2dLot5ozKeWj7unRkbTwK9NTWhdcTROtoWeVRnWN4hIHIQ6WTPJPE5BI0jyRLwwjCD 35Mj69Bl2XCillkRYKgWKBDRD7xMIhEX9Rae1p0kpgh1kcXhfCTDzC81k8zECDQJy6Mc ar+w== X-Gm-Message-State: AOAM530xw168Y0/nap/WsnIZiL8rkfgmjEN4fxTDcfkwiba/HWig00jv 817vA4wTbCPZFAPzq2Z+U6UrOEbnRrQ7Hw== X-Google-Smtp-Source: ABdhPJziSH0yZ4fn6Nei4E8CtQmXyLHAriIay1qpgqpfqZSgO2Vfs13aRW8dg+HBHM1/a1aAEwh9LQ== X-Received: by 2002:a17:906:6801:: with SMTP id k1mr25749741ejr.492.1597849909574; Wed, 19 Aug 2020 08:11:49 -0700 (PDT) Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com. [209.85.128.52]) by smtp.gmail.com with ESMTPSA id re10sm19091810ejb.68.2020.08.19.08.11.49 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 19 Aug 2020 08:11:49 -0700 (PDT) Received: by mail-wm1-f52.google.com with SMTP id f18so1872090wmc.0 for ; Wed, 19 Aug 2020 08:11:49 -0700 (PDT) X-Received: by 2002:a1c:4d12:: with SMTP id o18mr5316483wmh.55.1597849448826; Wed, 19 Aug 2020 08:04:08 -0700 (PDT) MIME-Version: 1.0 References: <20200819065555.1802761-1-hch@lst.de> <20200819065555.1802761-20-hch@lst.de> In-Reply-To: <20200819065555.1802761-20-hch@lst.de> From: Tomasz Figa Date: Wed, 19 Aug 2020 17:03:52 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 19/28] dma-mapping: replace DMA_ATTR_NON_CONSISTENT with dma_{alloc, free}_pages To: Christoph Hellwig X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200819_111151_280771_A78009EE X-CRM114-Status: GOOD ( 34.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alsa-devel@alsa-project.org, linux-ia64@vger.kernel.org, Linux Doc Mailing List , nouveau@lists.freedesktop.org, linux-nvme@lists.infradead.org, Linux Kernel Mailing List , "James E.J. Bottomley" , linux-mm@kvack.org, Marek Szyprowski , linux-samsung-soc , Joonyoung Shim , linux-scsi@vger.kernel.org, Kyungmin Park , Ben Skeggs , Matt Porter , Linux Media Mailing List , Tom Lendacky , Pawel Osciak , Mauro Carvalho Chehab , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , Thomas Bogendoerfer , linux-parisc@vger.kernel.org, netdev@vger.kernel.org, Seung-Woo Kim , linux-mips@vger.kernel.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel , " Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Christoph, On Wed, Aug 19, 2020 at 8:57 AM Christoph Hellwig wrote: > > Add a new API to allocate and free pages that are guaranteed to be > addressable by a device, but otherwise behave like pages allocated by > alloc_pages. The intended APIs to sync them for use with the device > and cpu are dma_sync_single_for_{device,cpu} that are also used for > streaming mappings. > > Switch all drivers over to this new API, but keep the usage of the > crufty dma_cache_sync API for now, which will be cleaned up on a driver > by driver basis. > > Signed-off-by: Christoph Hellwig > --- > Documentation/core-api/dma-api.rst | 68 +++++++++++------------ > Documentation/core-api/dma-attributes.rst | 8 --- > arch/alpha/kernel/pci_iommu.c | 2 + > arch/arm/mm/dma-mapping-nommu.c | 2 + > arch/arm/mm/dma-mapping.c | 4 ++ > arch/ia64/hp/common/sba_iommu.c | 2 + > arch/mips/jazz/jazzdma.c | 7 +-- > arch/powerpc/kernel/dma-iommu.c | 2 + > arch/powerpc/platforms/ps3/system-bus.c | 4 ++ > arch/powerpc/platforms/pseries/vio.c | 2 + > arch/s390/pci/pci_dma.c | 2 + > arch/x86/kernel/amd_gart_64.c | 2 + > drivers/iommu/dma-iommu.c | 2 + > drivers/iommu/intel/iommu.c | 4 ++ > drivers/net/ethernet/i825xx/lasi_82596.c | 13 ++--- > drivers/net/ethernet/seeq/sgiseeq.c | 12 ++-- > drivers/parisc/ccio-dma.c | 2 + > drivers/parisc/sba_iommu.c | 2 + > drivers/scsi/53c700.c | 8 +-- > drivers/scsi/sgiwd93.c | 12 ++-- > drivers/xen/swiotlb-xen.c | 2 + > include/linux/dma-direct.h | 5 ++ > include/linux/dma-mapping.h | 29 ++++++++-- > include/linux/dma-noncoherent.h | 3 - > kernel/dma/direct.c | 51 ++++++++++++++++- > kernel/dma/mapping.c | 43 +++++++++++++- > kernel/dma/ops_helpers.c | 35 ++++++++++++ > kernel/dma/virt.c | 2 + > sound/mips/hal2.c | 20 +++---- > 29 files changed, 254 insertions(+), 96 deletions(-) > Thanks for the patch. The general design looks quite nice, but please see my comments inline. > diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst > index 90239348b30f6f..047fcfffa0e5cf 100644 > --- a/Documentation/core-api/dma-api.rst > +++ b/Documentation/core-api/dma-api.rst > @@ -516,48 +516,53 @@ routines, e.g.::: > } > > > -Part II - Advanced dma usage > ----------------------------- > +Part II - Non-coherent DMA allocations > +-------------------------------------- > > -Warning: These pieces of the DMA API should not be used in the > -majority of cases, since they cater for unlikely corner cases that > -don't belong in usual drivers. > +These APIs allow to allocate pages that can be used like normal pages > +in the kernel direct mapping, but are guaranteed to be DMA addressable. Could we elaborate a bit more on what "like normal pages in kernel direct mapping" mean from the driver perspective? > > If you don't understand how cache line coherency works between a > processor and an I/O device, you should not be using this part of the > -API at all. > +API. > > :: > > void * > - dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > - gfp_t flag, unsigned long attrs) > + dma_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, > + enum dma_data_direction dir, gfp_t gfp) > + > +This routine allocates a region of bytes of consistent memory. It > +returns a pointer to the allocated region (in the processor's virtual address > +space) or NULL if the allocation failed. The returned memory is guanteed to > +behave like memory allocated using alloc_pages. There is one aspect that the existing dma_alloc_attrs() handles, but this new function doesn't: IOMMU support. The function will always allocate a physically-contiguous block memory, which is a costly operation and not even guaranteed to succeed, even if enough free memory is available. Modern SoCs employ IOMMUs to avoid the need to allocate physically-contiguous memory and those happen to be also the devices that could benefit from non-coherent allocations a lot. One of the tasks of the DMA API was making it possible to allocate suitable memory for a given device, without having the driver know about the SoC integration details, such as the presence of an IOMMU. Today, dma_alloc_attrs() uses the .alloc callback of the dma_ops struct and the IOMMU-aware implementations, like the dma-iommu helpers [1], would allocate discontiguous pages. Therefore, while I see the DMA-aware page allocation functionality as a useful functionality on its own for scatter-gather-capable hardware, I believe it is not a complete replacement for dma_alloc_attrs() with the DMA_ATTR_NON_CONSISTENT attribute. [1] https://elixir.bootlin.com/linux/v5.9-rc1/source/drivers/iommu/dma-iommu.c#L510 Best regards, Tomasz _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tomasz Figa Date: Wed, 19 Aug 2020 15:03:52 +0000 Subject: Re: [PATCH 19/28] dma-mapping: replace DMA_ATTR_NON_CONSISTENT with dma_{alloc, free}_pages Message-Id: List-Id: References: <20200819065555.1802761-1-hch@lst.de> <20200819065555.1802761-20-hch@lst.de> In-Reply-To: <20200819065555.1802761-20-hch@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Christoph Hellwig Cc: Mauro Carvalho Chehab , Thomas Bogendoerfer , "James E.J. Bottomley" , Joonyoung Shim , Seung-Woo Kim , Kyungmin Park , Ben Skeggs , Pawel Osciak , Marek Szyprowski , Matt Porter , "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , Tom Lendacky , alsa-devel@alsa-project.org, linux-samsung-soc , linux-ia64@vger.kernel.org, linux-scsi@vger.kernel.org, linux-parisc@vger.kernel.org, Linux Doc Mailing List , nouveau@lists.freedesktop.org, Linux Kernel Mailing List , linux-nvme@lists.infradead.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , Linux Media Mailing List Hi Christoph, On Wed, Aug 19, 2020 at 8:57 AM Christoph Hellwig wrote: > > Add a new API to allocate and free pages that are guaranteed to be > addressable by a device, but otherwise behave like pages allocated by > alloc_pages. The intended APIs to sync them for use with the device > and cpu are dma_sync_single_for_{device,cpu} that are also used for > streaming mappings. > > Switch all drivers over to this new API, but keep the usage of the > crufty dma_cache_sync API for now, which will be cleaned up on a driver > by driver basis. > > Signed-off-by: Christoph Hellwig > --- > Documentation/core-api/dma-api.rst | 68 +++++++++++------------ > Documentation/core-api/dma-attributes.rst | 8 --- > arch/alpha/kernel/pci_iommu.c | 2 + > arch/arm/mm/dma-mapping-nommu.c | 2 + > arch/arm/mm/dma-mapping.c | 4 ++ > arch/ia64/hp/common/sba_iommu.c | 2 + > arch/mips/jazz/jazzdma.c | 7 +-- > arch/powerpc/kernel/dma-iommu.c | 2 + > arch/powerpc/platforms/ps3/system-bus.c | 4 ++ > arch/powerpc/platforms/pseries/vio.c | 2 + > arch/s390/pci/pci_dma.c | 2 + > arch/x86/kernel/amd_gart_64.c | 2 + > drivers/iommu/dma-iommu.c | 2 + > drivers/iommu/intel/iommu.c | 4 ++ > drivers/net/ethernet/i825xx/lasi_82596.c | 13 ++--- > drivers/net/ethernet/seeq/sgiseeq.c | 12 ++-- > drivers/parisc/ccio-dma.c | 2 + > drivers/parisc/sba_iommu.c | 2 + > drivers/scsi/53c700.c | 8 +-- > drivers/scsi/sgiwd93.c | 12 ++-- > drivers/xen/swiotlb-xen.c | 2 + > include/linux/dma-direct.h | 5 ++ > include/linux/dma-mapping.h | 29 ++++++++-- > include/linux/dma-noncoherent.h | 3 - > kernel/dma/direct.c | 51 ++++++++++++++++- > kernel/dma/mapping.c | 43 +++++++++++++- > kernel/dma/ops_helpers.c | 35 ++++++++++++ > kernel/dma/virt.c | 2 + > sound/mips/hal2.c | 20 +++---- > 29 files changed, 254 insertions(+), 96 deletions(-) > Thanks for the patch. The general design looks quite nice, but please see my comments inline. > diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst > index 90239348b30f6f..047fcfffa0e5cf 100644 > --- a/Documentation/core-api/dma-api.rst > +++ b/Documentation/core-api/dma-api.rst > @@ -516,48 +516,53 @@ routines, e.g.::: > } > > > -Part II - Advanced dma usage > ----------------------------- > +Part II - Non-coherent DMA allocations > +-------------------------------------- > > -Warning: These pieces of the DMA API should not be used in the > -majority of cases, since they cater for unlikely corner cases that > -don't belong in usual drivers. > +These APIs allow to allocate pages that can be used like normal pages > +in the kernel direct mapping, but are guaranteed to be DMA addressable. Could we elaborate a bit more on what "like normal pages in kernel direct mapping" mean from the driver perspective? > > If you don't understand how cache line coherency works between a > processor and an I/O device, you should not be using this part of the > -API at all. > +API. > > :: > > void * > - dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > - gfp_t flag, unsigned long attrs) > + dma_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, > + enum dma_data_direction dir, gfp_t gfp) > + > +This routine allocates a region of bytes of consistent memory. It > +returns a pointer to the allocated region (in the processor's virtual address > +space) or NULL if the allocation failed. The returned memory is guanteed to > +behave like memory allocated using alloc_pages. There is one aspect that the existing dma_alloc_attrs() handles, but this new function doesn't: IOMMU support. The function will always allocate a physically-contiguous block memory, which is a costly operation and not even guaranteed to succeed, even if enough free memory is available. Modern SoCs employ IOMMUs to avoid the need to allocate physically-contiguous memory and those happen to be also the devices that could benefit from non-coherent allocations a lot. One of the tasks of the DMA API was making it possible to allocate suitable memory for a given device, without having the driver know about the SoC integration details, such as the presence of an IOMMU. Today, dma_alloc_attrs() uses the .alloc callback of the dma_ops struct and the IOMMU-aware implementations, like the dma-iommu helpers [1], would allocate discontiguous pages. Therefore, while I see the DMA-aware page allocation functionality as a useful functionality on its own for scatter-gather-capable hardware, I believe it is not a complete replacement for dma_alloc_attrs() with the DMA_ATTR_NON_CONSISTENT attribute. [1] https://elixir.bootlin.com/linux/v5.9-rc1/source/drivers/iommu/dma-iommu.c#L510 Best regards, Tomasz