From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4A7CC48BE8 for ; Wed, 16 Jun 2021 04:04:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BE048610A1 for ; Wed, 16 Jun 2021 04:04:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229642AbhFPEGq (ORCPT ); Wed, 16 Jun 2021 00:06:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229483AbhFPEGp (ORCPT ); Wed, 16 Jun 2021 00:06:45 -0400 Received: from mail-il1-x12f.google.com (mail-il1-x12f.google.com [IPv6:2607:f8b0:4864:20::12f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57F12C061574 for ; Tue, 15 Jun 2021 21:04:40 -0700 (PDT) Received: by mail-il1-x12f.google.com with SMTP id z1so1194101ils.0 for ; Tue, 15 Jun 2021 21:04:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=ZGcLRBiht0jP6J+AXrtU54w8Uc+zr833JxnoG08K94QWWjvLCllFqXgKhb1YXYtuSI H2kUoXb56/euW7L5KGDSV7c5wQp5UCbd+ujOwhBYXXWkAOagA4ahf3bI5zABwigXHQRs 2V+ulAhHs0i9h1/h3ywLUL41gakNk5ma80vrs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=BZ5wfZQ3Ojdl0y0iFwuafDwlujNYI8xPbNqB0nMqHkC+g1mjXLTxW6rczJRJpFvE+W 8pkOBDx3KGR1pPkLv6jXb68ho3qxLKkvaAhraaZm4tN5vsFI6DC11L/EA/e2p5kEgk0E 2PcdAUBvoCm88FhVkx9caFg38su6WZz49xq7AbFhaYkoRroYVd0bMkZI+Rm+GZFfJZYW AEmsx1LgUrqngjIxPVkZDgzOBpc8X8xNz/BWwOQttFcPVMc5ooQS3hr4Dl9a2KEaCoHW n3sKvKOxvEUeXmTBbCzqU9LmMqDb3r6B8GdmcVbyY/FqbeUDgy70EvJCK6uzFJzU2lF0 JuoA== X-Gm-Message-State: AOAM530akQ63hzNPVnltoasKj2OtniW+4yF0ROU97vQFG0OVKAentA8N W4YEzFglXJh/qvlqLrbRWJvN63y6CgkeKQ== X-Google-Smtp-Source: ABdhPJyHkgr6V9GMTLScXY1gnMjgfK2lhtHs5KlY3V6KC9P27Hyn67aCddo7crG3ifx8eo0E1nLfUg== X-Received: by 2002:a92:d452:: with SMTP id r18mr2186538ilm.73.1623816279455; Tue, 15 Jun 2021 21:04:39 -0700 (PDT) Received: from mail-io1-f48.google.com (mail-io1-f48.google.com. [209.85.166.48]) by smtp.gmail.com with ESMTPSA id z3sm571387ior.14.2021.06.15.21.04.38 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 15 Jun 2021 21:04:39 -0700 (PDT) Received: by mail-io1-f48.google.com with SMTP id 5so1598231ioe.1 for ; Tue, 15 Jun 2021 21:04:38 -0700 (PDT) X-Received: by 2002:a5e:8513:: with SMTP id i19mr1992773ioj.50.1623816266935; Tue, 15 Jun 2021 21:04:26 -0700 (PDT) MIME-Version: 1.0 References: <20210616035240.840463-1-tientzu@chromium.org> <20210616035240.840463-10-tientzu@chromium.org> In-Reply-To: <20210616035240.840463-10-tientzu@chromium.org> From: Claire Chang Date: Wed, 16 Jun 2021 12:04:16 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v11 09/12] swiotlb: Add restricted DMA alloc/free support To: Rob Herring , mpe@ellerman.id.au, Joerg Roedel , Will Deacon , Frank Rowand , Konrad Rzeszutek Wilk , boris.ostrovsky@oracle.com, jgross@suse.com, Christoph Hellwig , Marek Szyprowski Cc: benh@kernel.crashing.org, paulus@samba.org, "list@263.net:IOMMU DRIVERS" , sstabellini@kernel.org, Robin Murphy , grant.likely@arm.com, xypron.glpk@gmx.de, Thierry Reding , mingo@kernel.org, bauerman@linux.ibm.com, peterz@infradead.org, Greg KH , Saravana Kannan , "Rafael J . Wysocki" , heikki.krogerus@linux.intel.com, Andy Shevchenko , Randy Dunlap , Dan Williams , Bartosz Golaszewski , linux-devicetree , lkml , linuxppc-dev@lists.ozlabs.org, xen-devel@lists.xenproject.org, Nicolas Boichat , Jim Quinlan , Tomasz Figa , bskeggs@redhat.com, Bjorn Helgaas , chris@chris-wilson.co.uk, Daniel Vetter , airlied@linux.ie, dri-devel@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, jani.nikula@linux.intel.com, Jianxiong Gao , joonas.lahtinen@linux.intel.com, linux-pci@vger.kernel.org, maarten.lankhorst@linux.intel.com, matthew.auld@intel.com, rodrigo.vivi@intel.com, thomas.hellstrom@linux.intel.com Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jun 16, 2021 at 11:54 AM Claire Chang wrote: > > Add the functions, swiotlb_{alloc,free} to support the memory allocation > from restricted DMA pool. > > The restricted DMA pool is preferred if available. > > Note that since coherent allocation needs remapping, one must set up > another device coherent pool by shared-dma-pool and use > dma_alloc_from_dev_coherent instead for atomic coherent allocation. > > Signed-off-by: Claire Chang > Reviewed-by: Christoph Hellwig > --- > include/linux/swiotlb.h | 15 +++++++++++++ > kernel/dma/direct.c | 50 ++++++++++++++++++++++++++++++----------- > kernel/dma/swiotlb.c | 45 +++++++++++++++++++++++++++++++++++-- > 3 files changed, 95 insertions(+), 15 deletions(-) > > diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h > index efcd56e3a16c..2d5ec670e064 100644 > --- a/include/linux/swiotlb.h > +++ b/include/linux/swiotlb.h > @@ -156,4 +156,19 @@ static inline void swiotlb_adjust_size(unsigned long size) > extern void swiotlb_print_info(void); > extern void swiotlb_set_max_segment(unsigned int); > > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size); > +bool swiotlb_free(struct device *dev, struct page *page, size_t size); > +#else > +static inline struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + return NULL; > +} > +static inline bool swiotlb_free(struct device *dev, struct page *page, > + size_t size) > +{ > + return false; > +} > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > + > #endif /* __LINUX_SWIOTLB_H */ > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 3713461d6fe0..da0e09621230 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -75,6 +75,15 @@ static bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) > min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit); > } > > +static void __dma_direct_free_pages(struct device *dev, struct page *page, > + size_t size) > +{ > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL) && > + swiotlb_free(dev, page, size)) > + return; > + dma_free_contiguous(dev, page, size); > +} > + > static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > gfp_t gfp) > { > @@ -86,7 +95,16 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > gfp |= dma_direct_optimal_gfp_mask(dev, dev->coherent_dma_mask, > &phys_limit); > - page = dma_alloc_contiguous(dev, size, gfp); > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL)) { > + page = swiotlb_alloc(dev, size); > + if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > + __dma_direct_free_pages(dev, page, size); > + return NULL; > + } > + } > + > + if (!page) > + page = dma_alloc_contiguous(dev, size, gfp); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_free_contiguous(dev, page, size); > page = NULL; > @@ -142,7 +160,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > gfp |= __GFP_NOWARN; > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO); > if (!page) > return NULL; > @@ -155,18 +173,23 @@ void *dma_direct_alloc(struct device *dev, size_t size, > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) > return arch_dma_alloc(dev, size, dma_handle, gfp, attrs); Just noticed that after propagating swiotlb_force setting into io_tlb_default_mem->force, the memory allocation behavior for swiotlb_force will change (i.e. always skipping arch_dma_alloc and dma_direct_alloc_from_pool). > > /* > * Remapping or decrypting memory may block. If either is required and > * we can't block, allocate the memory from the atomic pools. > + * If restricted DMA (i.e., is_dev_swiotlb_force) is required, one must > + * set up another device coherent pool by shared-dma-pool and use > + * dma_alloc_from_dev_coherent instead. > */ > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > !gfpflags_allow_blocking(gfp) && > (force_dma_unencrypted(dev) || > - (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev)))) > + (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > + !dev_is_dma_coherent(dev))) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); And here. > > /* we always manually zero the memory once we are done */ > @@ -237,7 +260,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > return NULL; > } > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -247,15 +270,15 @@ void dma_direct_free(struct device *dev, size_t size, > unsigned int page_order = get_order(size); > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > /* cpu_addr is a struct page cookie, not a kernel address */ > dma_free_contiguous(dev, cpu_addr, size); > return; > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) { > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) { > arch_dma_free(dev, size, cpu_addr, dma_addr, attrs); > return; > } > @@ -273,7 +296,7 @@ void dma_direct_free(struct device *dev, size_t size, > else if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) > arch_dma_clear_uncached(cpu_addr, size); > > - dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size); > + __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); > } > > struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > @@ -283,7 +306,8 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > void *ret; > > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > - force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp)) > + force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); > > page = __dma_direct_alloc_pages(dev, size, gfp); > @@ -310,7 +334,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > *dma_handle = phys_to_dma_direct(dev, page_to_phys(page)); > return page; > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -329,7 +353,7 @@ void dma_direct_free_pages(struct device *dev, size_t size, > if (force_dma_unencrypted(dev)) > set_memory_encrypted((unsigned long)vaddr, 1 << page_order); > > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > } > > #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c > index fec4934b9926..6ad85b48f101 100644 > --- a/kernel/dma/swiotlb.c > +++ b/kernel/dma/swiotlb.c > @@ -462,8 +462,9 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, > > index = wrap = wrap_index(mem, ALIGN(mem->index, stride)); > do { > - if ((slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > - (orig_addr & iotlb_align_mask)) { > + if (orig_addr && > + (slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > + (orig_addr & iotlb_align_mask)) { > index = wrap_index(mem, index + 1); > continue; > } > @@ -702,3 +703,43 @@ static int __init swiotlb_create_default_debugfs(void) > late_initcall(swiotlb_create_default_debugfs); > > #endif > + > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; > + phys_addr_t tlb_addr; > + int index; > + > + /* > + * Skip io_tlb_default_mem since swiotlb_alloc doesn't support atomic > + * coherent allocation. Otherwise might break existing devices. > + * One must set up another device coherent pool by shared-dma-pool and > + * use dma_alloc_from_dev_coherent instead for atomic coherent > + * allocation to avoid memory remapping. > + */ > + if (!mem || mem == io_tlb_default_mem) > + return NULL; > + > + index = swiotlb_find_slots(dev, 0, size); > + if (index == -1) > + return NULL; > + > + tlb_addr = slot_addr(mem->start, index); > + > + return pfn_to_page(PFN_DOWN(tlb_addr)); > +} > + > +bool swiotlb_free(struct device *dev, struct page *page, size_t size) > +{ > + phys_addr_t tlb_addr = page_to_phys(page); > + > + if (!is_swiotlb_buffer(dev, tlb_addr)) > + return false; > + > + swiotlb_release_slots(dev, tlb_addr); > + > + return true; > +} > + > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > -- > 2.32.0.272.g935e593368-goog > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B7BFC48BE5 for ; Wed, 16 Jun 2021 04:05:24 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D017F6115C for ; Wed, 16 Jun 2021 04:05:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D017F6115C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4G4Wmz0klxz3c5J for ; Wed, 16 Jun 2021 14:05:23 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=ZGcLRBih; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=chromium.org (client-ip=2607:f8b0:4864:20::12b; helo=mail-il1-x12b.google.com; envelope-from=tientzu@chromium.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=ZGcLRBih; dkim-atps=neutral Received: from mail-il1-x12b.google.com (mail-il1-x12b.google.com [IPv6:2607:f8b0:4864:20::12b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4G4WmG2N9Yz2xZJ for ; Wed, 16 Jun 2021 14:04:43 +1000 (AEST) Received: by mail-il1-x12b.google.com with SMTP id b9so1156983ilr.2 for ; Tue, 15 Jun 2021 21:04:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=ZGcLRBiht0jP6J+AXrtU54w8Uc+zr833JxnoG08K94QWWjvLCllFqXgKhb1YXYtuSI H2kUoXb56/euW7L5KGDSV7c5wQp5UCbd+ujOwhBYXXWkAOagA4ahf3bI5zABwigXHQRs 2V+ulAhHs0i9h1/h3ywLUL41gakNk5ma80vrs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=o611fucfqeX0ZwcUy8aD3yDFzR6L5bnmtXLoZPYyIu6OZIwUehhY2SoV4rD0g15w7i XsPXbLGQVeoWxUUWh7vWiBrb9n/HZr6Vk56o/O5BqJ4K1TrY6B9xKsRaHD8ip4OlphSB szJe9woTJ1IgBvV1Np/3HlAAlAZ/l2uv0hXgIzQ9HWbeFbaA6eshtVrczbuQ1xG3mQ2L ugz7qFT3mLrvqTJT+OcWvaNGKF2yHKpi5cTFSNIvGfdxlo8KUYfZapvjA2TtJzPYZx48 bQcTBwcfdmxpC1nyNVP/nTnehXlr8vqyEd/gqzRI5rM/aREM5Gacugr9dIoJLGOfUDki uRPQ== X-Gm-Message-State: AOAM531PAlmw6Ma6oa9goFTbiTtjoTG9zbugHg5DMZqFhpc51ymHziSv 9LunWyqVWKlHqEePTnSMgPwHGGKTKJB53A== X-Google-Smtp-Source: ABdhPJyVfZT3X84UB/Sahtv/K5ax5MmdZ1v9toOg6GB8PeiIAHFK4/J7IyrQTTNL3frAxhSEPO948w== X-Received: by 2002:a05:6e02:4ae:: with SMTP id e14mr2055776ils.186.1623816279288; Tue, 15 Jun 2021 21:04:39 -0700 (PDT) Received: from mail-il1-f179.google.com (mail-il1-f179.google.com. [209.85.166.179]) by smtp.gmail.com with ESMTPSA id v3sm556458ilh.74.2021.06.15.21.04.38 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 15 Jun 2021 21:04:38 -0700 (PDT) Received: by mail-il1-f179.google.com with SMTP id z1so1194030ils.0 for ; Tue, 15 Jun 2021 21:04:38 -0700 (PDT) X-Received: by 2002:a5e:8513:: with SMTP id i19mr1992773ioj.50.1623816266935; Tue, 15 Jun 2021 21:04:26 -0700 (PDT) MIME-Version: 1.0 References: <20210616035240.840463-1-tientzu@chromium.org> <20210616035240.840463-10-tientzu@chromium.org> In-Reply-To: <20210616035240.840463-10-tientzu@chromium.org> From: Claire Chang Date: Wed, 16 Jun 2021 12:04:16 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v11 09/12] swiotlb: Add restricted DMA alloc/free support To: Rob Herring , mpe@ellerman.id.au, Joerg Roedel , Will Deacon , Frank Rowand , Konrad Rzeszutek Wilk , boris.ostrovsky@oracle.com, jgross@suse.com, Christoph Hellwig , Marek Szyprowski Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: heikki.krogerus@linux.intel.com, thomas.hellstrom@linux.intel.com, peterz@infradead.org, joonas.lahtinen@linux.intel.com, dri-devel@lists.freedesktop.org, chris@chris-wilson.co.uk, grant.likely@arm.com, paulus@samba.org, mingo@kernel.org, Jianxiong Gao , sstabellini@kernel.org, Saravana Kannan , xypron.glpk@gmx.de, "Rafael J . Wysocki" , Bartosz Golaszewski , bskeggs@redhat.com, linux-pci@vger.kernel.org, xen-devel@lists.xenproject.org, Thierry Reding , intel-gfx@lists.freedesktop.org, matthew.auld@intel.com, linux-devicetree , Daniel Vetter , airlied@linux.ie, maarten.lankhorst@linux.intel.com, linuxppc-dev@lists.ozlabs.org, jani.nikula@linux.intel.com, Nicolas Boichat , rodrigo.vivi@intel.com, Bjorn Helgaas , Dan Williams , Andy Shevchenko , Greg KH , Randy Dunlap , lkml , Tomasz Figa , "list@263.net:IOMMU DRIVERS" , Jim Quinlan , Robin Murphy , bauerman@linux.ibm.com Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Wed, Jun 16, 2021 at 11:54 AM Claire Chang wrote: > > Add the functions, swiotlb_{alloc,free} to support the memory allocation > from restricted DMA pool. > > The restricted DMA pool is preferred if available. > > Note that since coherent allocation needs remapping, one must set up > another device coherent pool by shared-dma-pool and use > dma_alloc_from_dev_coherent instead for atomic coherent allocation. > > Signed-off-by: Claire Chang > Reviewed-by: Christoph Hellwig > --- > include/linux/swiotlb.h | 15 +++++++++++++ > kernel/dma/direct.c | 50 ++++++++++++++++++++++++++++++----------- > kernel/dma/swiotlb.c | 45 +++++++++++++++++++++++++++++++++++-- > 3 files changed, 95 insertions(+), 15 deletions(-) > > diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h > index efcd56e3a16c..2d5ec670e064 100644 > --- a/include/linux/swiotlb.h > +++ b/include/linux/swiotlb.h > @@ -156,4 +156,19 @@ static inline void swiotlb_adjust_size(unsigned long size) > extern void swiotlb_print_info(void); > extern void swiotlb_set_max_segment(unsigned int); > > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size); > +bool swiotlb_free(struct device *dev, struct page *page, size_t size); > +#else > +static inline struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + return NULL; > +} > +static inline bool swiotlb_free(struct device *dev, struct page *page, > + size_t size) > +{ > + return false; > +} > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > + > #endif /* __LINUX_SWIOTLB_H */ > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 3713461d6fe0..da0e09621230 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -75,6 +75,15 @@ static bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) > min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit); > } > > +static void __dma_direct_free_pages(struct device *dev, struct page *page, > + size_t size) > +{ > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL) && > + swiotlb_free(dev, page, size)) > + return; > + dma_free_contiguous(dev, page, size); > +} > + > static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > gfp_t gfp) > { > @@ -86,7 +95,16 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > gfp |= dma_direct_optimal_gfp_mask(dev, dev->coherent_dma_mask, > &phys_limit); > - page = dma_alloc_contiguous(dev, size, gfp); > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL)) { > + page = swiotlb_alloc(dev, size); > + if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > + __dma_direct_free_pages(dev, page, size); > + return NULL; > + } > + } > + > + if (!page) > + page = dma_alloc_contiguous(dev, size, gfp); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_free_contiguous(dev, page, size); > page = NULL; > @@ -142,7 +160,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > gfp |= __GFP_NOWARN; > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO); > if (!page) > return NULL; > @@ -155,18 +173,23 @@ void *dma_direct_alloc(struct device *dev, size_t size, > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) > return arch_dma_alloc(dev, size, dma_handle, gfp, attrs); Just noticed that after propagating swiotlb_force setting into io_tlb_default_mem->force, the memory allocation behavior for swiotlb_force will change (i.e. always skipping arch_dma_alloc and dma_direct_alloc_from_pool). > > /* > * Remapping or decrypting memory may block. If either is required and > * we can't block, allocate the memory from the atomic pools. > + * If restricted DMA (i.e., is_dev_swiotlb_force) is required, one must > + * set up another device coherent pool by shared-dma-pool and use > + * dma_alloc_from_dev_coherent instead. > */ > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > !gfpflags_allow_blocking(gfp) && > (force_dma_unencrypted(dev) || > - (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev)))) > + (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > + !dev_is_dma_coherent(dev))) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); And here. > > /* we always manually zero the memory once we are done */ > @@ -237,7 +260,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > return NULL; > } > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -247,15 +270,15 @@ void dma_direct_free(struct device *dev, size_t size, > unsigned int page_order = get_order(size); > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > /* cpu_addr is a struct page cookie, not a kernel address */ > dma_free_contiguous(dev, cpu_addr, size); > return; > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) { > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) { > arch_dma_free(dev, size, cpu_addr, dma_addr, attrs); > return; > } > @@ -273,7 +296,7 @@ void dma_direct_free(struct device *dev, size_t size, > else if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) > arch_dma_clear_uncached(cpu_addr, size); > > - dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size); > + __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); > } > > struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > @@ -283,7 +306,8 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > void *ret; > > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > - force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp)) > + force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); > > page = __dma_direct_alloc_pages(dev, size, gfp); > @@ -310,7 +334,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > *dma_handle = phys_to_dma_direct(dev, page_to_phys(page)); > return page; > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -329,7 +353,7 @@ void dma_direct_free_pages(struct device *dev, size_t size, > if (force_dma_unencrypted(dev)) > set_memory_encrypted((unsigned long)vaddr, 1 << page_order); > > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > } > > #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c > index fec4934b9926..6ad85b48f101 100644 > --- a/kernel/dma/swiotlb.c > +++ b/kernel/dma/swiotlb.c > @@ -462,8 +462,9 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, > > index = wrap = wrap_index(mem, ALIGN(mem->index, stride)); > do { > - if ((slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > - (orig_addr & iotlb_align_mask)) { > + if (orig_addr && > + (slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > + (orig_addr & iotlb_align_mask)) { > index = wrap_index(mem, index + 1); > continue; > } > @@ -702,3 +703,43 @@ static int __init swiotlb_create_default_debugfs(void) > late_initcall(swiotlb_create_default_debugfs); > > #endif > + > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; > + phys_addr_t tlb_addr; > + int index; > + > + /* > + * Skip io_tlb_default_mem since swiotlb_alloc doesn't support atomic > + * coherent allocation. Otherwise might break existing devices. > + * One must set up another device coherent pool by shared-dma-pool and > + * use dma_alloc_from_dev_coherent instead for atomic coherent > + * allocation to avoid memory remapping. > + */ > + if (!mem || mem == io_tlb_default_mem) > + return NULL; > + > + index = swiotlb_find_slots(dev, 0, size); > + if (index == -1) > + return NULL; > + > + tlb_addr = slot_addr(mem->start, index); > + > + return pfn_to_page(PFN_DOWN(tlb_addr)); > +} > + > +bool swiotlb_free(struct device *dev, struct page *page, size_t size) > +{ > + phys_addr_t tlb_addr = page_to_phys(page); > + > + if (!is_swiotlb_buffer(dev, tlb_addr)) > + return false; > + > + swiotlb_release_slots(dev, tlb_addr); > + > + return true; > +} > + > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > -- > 2.32.0.272.g935e593368-goog > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23EEBC48BE5 for ; Wed, 16 Jun 2021 04:04:35 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B651061369 for ; Wed, 16 Jun 2021 04:04:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B651061369 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 8627E60719; Wed, 16 Jun 2021 04:04:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vkLx7K_23RGs; Wed, 16 Jun 2021 04:04:33 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 1FD93606C8; Wed, 16 Jun 2021 04:04:33 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E464FC000E; Wed, 16 Jun 2021 04:04:32 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9F3EAC000B for ; Wed, 16 Jun 2021 04:04:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 6E82D4065A for ; Wed, 16 Jun 2021 04:04:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=chromium.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CjvsthEv4CFa for ; Wed, 16 Jun 2021 04:04:30 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) by smtp4.osuosl.org (Postfix) with ESMTPS id DAEC74064F for ; Wed, 16 Jun 2021 04:04:29 +0000 (UTC) Received: by mail-io1-xd29.google.com with SMTP id l64so1560652ioa.7 for ; Tue, 15 Jun 2021 21:04:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=ZGcLRBiht0jP6J+AXrtU54w8Uc+zr833JxnoG08K94QWWjvLCllFqXgKhb1YXYtuSI H2kUoXb56/euW7L5KGDSV7c5wQp5UCbd+ujOwhBYXXWkAOagA4ahf3bI5zABwigXHQRs 2V+ulAhHs0i9h1/h3ywLUL41gakNk5ma80vrs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=jakKRcNMgaVBMB587cRqvRGRMWezy2pT/FC3y3UQ+a6ELim2tWkVC1JsJZFnCxV8py Ozq9CrDnOHjmJ4DKkViBtOqvE7rmqBGoRqNfVhZGqzz4x6cccAQjkYlshxANy9r4zwws prngvLfAN67mV3N5rgc3ipIViu5EJxiKz1uQOvyZqRKsKLUlxFC2qOLsWFaCM6JQtWP4 Za2P4aR9dIJJSX5gEoWOltBMv38YItpHHBwUsG8d99aXFVNFyCPBe++hDQozLHzMqMOV QLrBqxlFy1Onrvc9XSBjz6w5WodatZSLyLsY0h+Y9J9DLdiwXUypDcfuDR5zf9sZMCxt iDHg== X-Gm-Message-State: AOAM531yO7SyXoBnuKFO6aEIrbPSmBv+il1dPM9tDebNSGZGW+I8Wc0g F6VF72cKFpketXHqA3YgXe/QE6B03gy67Q== X-Google-Smtp-Source: ABdhPJzVGwqSCHm8BRsplr4VIPKt4CDrQ8FGhqh4875ib0dpgOGRU/iG0kMLldaDULjG9R6VoDjDmw== X-Received: by 2002:a6b:e90b:: with SMTP id u11mr2085047iof.35.1623816268511; Tue, 15 Jun 2021 21:04:28 -0700 (PDT) Received: from mail-io1-f47.google.com (mail-io1-f47.google.com. [209.85.166.47]) by smtp.gmail.com with ESMTPSA id u8sm558838ilg.43.2021.06.15.21.04.27 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 15 Jun 2021 21:04:28 -0700 (PDT) Received: by mail-io1-f47.google.com with SMTP id l64so1560597ioa.7 for ; Tue, 15 Jun 2021 21:04:27 -0700 (PDT) X-Received: by 2002:a5e:8513:: with SMTP id i19mr1992773ioj.50.1623816266935; Tue, 15 Jun 2021 21:04:26 -0700 (PDT) MIME-Version: 1.0 References: <20210616035240.840463-1-tientzu@chromium.org> <20210616035240.840463-10-tientzu@chromium.org> In-Reply-To: <20210616035240.840463-10-tientzu@chromium.org> From: Claire Chang Date: Wed, 16 Jun 2021 12:04:16 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v11 09/12] swiotlb: Add restricted DMA alloc/free support To: Rob Herring , mpe@ellerman.id.au, Joerg Roedel , Will Deacon , Frank Rowand , Konrad Rzeszutek Wilk , boris.ostrovsky@oracle.com, jgross@suse.com, Christoph Hellwig , Marek Szyprowski Cc: heikki.krogerus@linux.intel.com, thomas.hellstrom@linux.intel.com, peterz@infradead.org, benh@kernel.crashing.org, joonas.lahtinen@linux.intel.com, dri-devel@lists.freedesktop.org, chris@chris-wilson.co.uk, grant.likely@arm.com, paulus@samba.org, mingo@kernel.org, Jianxiong Gao , sstabellini@kernel.org, Saravana Kannan , xypron.glpk@gmx.de, "Rafael J . Wysocki" , Bartosz Golaszewski , bskeggs@redhat.com, linux-pci@vger.kernel.org, xen-devel@lists.xenproject.org, Thierry Reding , intel-gfx@lists.freedesktop.org, matthew.auld@intel.com, linux-devicetree , Daniel Vetter , airlied@linux.ie, maarten.lankhorst@linux.intel.com, linuxppc-dev@lists.ozlabs.org, jani.nikula@linux.intel.com, Nicolas Boichat , rodrigo.vivi@intel.com, Bjorn Helgaas , Dan Williams , Andy Shevchenko , Greg KH , Randy Dunlap , lkml , "list@263.net:IOMMU DRIVERS" , Jim Quinlan , Robin Murphy , bauerman@linux.ibm.com X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Wed, Jun 16, 2021 at 11:54 AM Claire Chang wrote: > > Add the functions, swiotlb_{alloc,free} to support the memory allocation > from restricted DMA pool. > > The restricted DMA pool is preferred if available. > > Note that since coherent allocation needs remapping, one must set up > another device coherent pool by shared-dma-pool and use > dma_alloc_from_dev_coherent instead for atomic coherent allocation. > > Signed-off-by: Claire Chang > Reviewed-by: Christoph Hellwig > --- > include/linux/swiotlb.h | 15 +++++++++++++ > kernel/dma/direct.c | 50 ++++++++++++++++++++++++++++++----------- > kernel/dma/swiotlb.c | 45 +++++++++++++++++++++++++++++++++++-- > 3 files changed, 95 insertions(+), 15 deletions(-) > > diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h > index efcd56e3a16c..2d5ec670e064 100644 > --- a/include/linux/swiotlb.h > +++ b/include/linux/swiotlb.h > @@ -156,4 +156,19 @@ static inline void swiotlb_adjust_size(unsigned long size) > extern void swiotlb_print_info(void); > extern void swiotlb_set_max_segment(unsigned int); > > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size); > +bool swiotlb_free(struct device *dev, struct page *page, size_t size); > +#else > +static inline struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + return NULL; > +} > +static inline bool swiotlb_free(struct device *dev, struct page *page, > + size_t size) > +{ > + return false; > +} > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > + > #endif /* __LINUX_SWIOTLB_H */ > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 3713461d6fe0..da0e09621230 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -75,6 +75,15 @@ static bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) > min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit); > } > > +static void __dma_direct_free_pages(struct device *dev, struct page *page, > + size_t size) > +{ > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL) && > + swiotlb_free(dev, page, size)) > + return; > + dma_free_contiguous(dev, page, size); > +} > + > static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > gfp_t gfp) > { > @@ -86,7 +95,16 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > gfp |= dma_direct_optimal_gfp_mask(dev, dev->coherent_dma_mask, > &phys_limit); > - page = dma_alloc_contiguous(dev, size, gfp); > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL)) { > + page = swiotlb_alloc(dev, size); > + if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > + __dma_direct_free_pages(dev, page, size); > + return NULL; > + } > + } > + > + if (!page) > + page = dma_alloc_contiguous(dev, size, gfp); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_free_contiguous(dev, page, size); > page = NULL; > @@ -142,7 +160,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > gfp |= __GFP_NOWARN; > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO); > if (!page) > return NULL; > @@ -155,18 +173,23 @@ void *dma_direct_alloc(struct device *dev, size_t size, > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) > return arch_dma_alloc(dev, size, dma_handle, gfp, attrs); Just noticed that after propagating swiotlb_force setting into io_tlb_default_mem->force, the memory allocation behavior for swiotlb_force will change (i.e. always skipping arch_dma_alloc and dma_direct_alloc_from_pool). > > /* > * Remapping or decrypting memory may block. If either is required and > * we can't block, allocate the memory from the atomic pools. > + * If restricted DMA (i.e., is_dev_swiotlb_force) is required, one must > + * set up another device coherent pool by shared-dma-pool and use > + * dma_alloc_from_dev_coherent instead. > */ > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > !gfpflags_allow_blocking(gfp) && > (force_dma_unencrypted(dev) || > - (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev)))) > + (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > + !dev_is_dma_coherent(dev))) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); And here. > > /* we always manually zero the memory once we are done */ > @@ -237,7 +260,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > return NULL; > } > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -247,15 +270,15 @@ void dma_direct_free(struct device *dev, size_t size, > unsigned int page_order = get_order(size); > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > /* cpu_addr is a struct page cookie, not a kernel address */ > dma_free_contiguous(dev, cpu_addr, size); > return; > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) { > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) { > arch_dma_free(dev, size, cpu_addr, dma_addr, attrs); > return; > } > @@ -273,7 +296,7 @@ void dma_direct_free(struct device *dev, size_t size, > else if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) > arch_dma_clear_uncached(cpu_addr, size); > > - dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size); > + __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); > } > > struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > @@ -283,7 +306,8 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > void *ret; > > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > - force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp)) > + force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); > > page = __dma_direct_alloc_pages(dev, size, gfp); > @@ -310,7 +334,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > *dma_handle = phys_to_dma_direct(dev, page_to_phys(page)); > return page; > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -329,7 +353,7 @@ void dma_direct_free_pages(struct device *dev, size_t size, > if (force_dma_unencrypted(dev)) > set_memory_encrypted((unsigned long)vaddr, 1 << page_order); > > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > } > > #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c > index fec4934b9926..6ad85b48f101 100644 > --- a/kernel/dma/swiotlb.c > +++ b/kernel/dma/swiotlb.c > @@ -462,8 +462,9 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, > > index = wrap = wrap_index(mem, ALIGN(mem->index, stride)); > do { > - if ((slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > - (orig_addr & iotlb_align_mask)) { > + if (orig_addr && > + (slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > + (orig_addr & iotlb_align_mask)) { > index = wrap_index(mem, index + 1); > continue; > } > @@ -702,3 +703,43 @@ static int __init swiotlb_create_default_debugfs(void) > late_initcall(swiotlb_create_default_debugfs); > > #endif > + > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; > + phys_addr_t tlb_addr; > + int index; > + > + /* > + * Skip io_tlb_default_mem since swiotlb_alloc doesn't support atomic > + * coherent allocation. Otherwise might break existing devices. > + * One must set up another device coherent pool by shared-dma-pool and > + * use dma_alloc_from_dev_coherent instead for atomic coherent > + * allocation to avoid memory remapping. > + */ > + if (!mem || mem == io_tlb_default_mem) > + return NULL; > + > + index = swiotlb_find_slots(dev, 0, size); > + if (index == -1) > + return NULL; > + > + tlb_addr = slot_addr(mem->start, index); > + > + return pfn_to_page(PFN_DOWN(tlb_addr)); > +} > + > +bool swiotlb_free(struct device *dev, struct page *page, size_t size) > +{ > + phys_addr_t tlb_addr = page_to_phys(page); > + > + if (!is_swiotlb_buffer(dev, tlb_addr)) > + return false; > + > + swiotlb_release_slots(dev, tlb_addr); > + > + return true; > +} > + > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > -- > 2.32.0.272.g935e593368-goog > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F37AEC48BE6 for ; Wed, 16 Jun 2021 04:04:42 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BD95061246 for ; Wed, 16 Jun 2021 04:04:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BD95061246 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 31C966E4CB; Wed, 16 Jun 2021 04:04:42 +0000 (UTC) Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3E43E6E4C9 for ; Wed, 16 Jun 2021 04:04:40 +0000 (UTC) Received: by mail-pg1-x52e.google.com with SMTP id l184so870346pgd.8 for ; Tue, 15 Jun 2021 21:04:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=ZGcLRBiht0jP6J+AXrtU54w8Uc+zr833JxnoG08K94QWWjvLCllFqXgKhb1YXYtuSI H2kUoXb56/euW7L5KGDSV7c5wQp5UCbd+ujOwhBYXXWkAOagA4ahf3bI5zABwigXHQRs 2V+ulAhHs0i9h1/h3ywLUL41gakNk5ma80vrs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=R3cCgrHrdHHiy55O49rKVvhz1GREDK8Wdvpek0YGgEOfB+esjhbr8DT6zoA0321eTY P2AJJu10duRmLZtW1zOzZT14wCjYKsNdWsQAO+DLls1oILp+WFoguuBNMGgQc0EmBKuv 7xODObCU8EVxF4X+/GYyiH5x+E0PPr9R5J/jaWAHki6Fo8W9vhl5G2fWXJKu1KcSOYj9 u0ZfFQglHO5ykNrh2WuFsAX883EWZKbDn7oWgPK+zxu4hgkdZHYVuOkI+2QmVFfcXu+h VHmAmc3k7yQ6T+X47GbOOYSClFvGuIzC2IhbXN0e4JYEQhXKm0jvhi5bVMEydV66rM5A mCIA== X-Gm-Message-State: AOAM532rKQdIJAQZvkrULfCtbKFei7XbdrWwbZcLReuhwfnj2D0m670D OBmVZOcXD5ZbW6qpNRxGsvIYHa6NvYxPVg== X-Google-Smtp-Source: ABdhPJzDZwmHymncmTwcP1gpcY7Lo5TkT73kmOBfdQjxJhDoIxaT9NIh+OY6SGzRBT7O8UHEUK/dHg== X-Received: by 2002:aa7:9885:0:b029:2ea:25ce:3ad2 with SMTP id r5-20020aa798850000b02902ea25ce3ad2mr7501885pfl.76.1623816279639; Tue, 15 Jun 2021 21:04:39 -0700 (PDT) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com. [209.85.210.173]) by smtp.gmail.com with ESMTPSA id r11sm621837pgl.34.2021.06.15.21.04.38 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 15 Jun 2021 21:04:39 -0700 (PDT) Received: by mail-pf1-f173.google.com with SMTP id y15so1128319pfl.4 for ; Tue, 15 Jun 2021 21:04:38 -0700 (PDT) X-Received: by 2002:a5e:8513:: with SMTP id i19mr1992773ioj.50.1623816266935; Tue, 15 Jun 2021 21:04:26 -0700 (PDT) MIME-Version: 1.0 References: <20210616035240.840463-1-tientzu@chromium.org> <20210616035240.840463-10-tientzu@chromium.org> In-Reply-To: <20210616035240.840463-10-tientzu@chromium.org> From: Claire Chang Date: Wed, 16 Jun 2021 12:04:16 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v11 09/12] swiotlb: Add restricted DMA alloc/free support To: Rob Herring , mpe@ellerman.id.au, Joerg Roedel , Will Deacon , Frank Rowand , Konrad Rzeszutek Wilk , boris.ostrovsky@oracle.com, jgross@suse.com, Christoph Hellwig , Marek Szyprowski Content-Type: text/plain; charset="UTF-8" X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: heikki.krogerus@linux.intel.com, thomas.hellstrom@linux.intel.com, peterz@infradead.org, dri-devel@lists.freedesktop.org, chris@chris-wilson.co.uk, grant.likely@arm.com, paulus@samba.org, mingo@kernel.org, Jianxiong Gao , sstabellini@kernel.org, Saravana Kannan , xypron.glpk@gmx.de, "Rafael J . Wysocki" , Bartosz Golaszewski , bskeggs@redhat.com, linux-pci@vger.kernel.org, xen-devel@lists.xenproject.org, Thierry Reding , intel-gfx@lists.freedesktop.org, matthew.auld@intel.com, linux-devicetree , airlied@linux.ie, Robin Murphy , Nicolas Boichat , rodrigo.vivi@intel.com, Bjorn Helgaas , Dan Williams , Andy Shevchenko , Greg KH , Randy Dunlap , lkml , Tomasz Figa , "list@263.net:IOMMU DRIVERS" , Jim Quinlan , linuxppc-dev@lists.ozlabs.org, bauerman@linux.ibm.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Wed, Jun 16, 2021 at 11:54 AM Claire Chang wrote: > > Add the functions, swiotlb_{alloc,free} to support the memory allocation > from restricted DMA pool. > > The restricted DMA pool is preferred if available. > > Note that since coherent allocation needs remapping, one must set up > another device coherent pool by shared-dma-pool and use > dma_alloc_from_dev_coherent instead for atomic coherent allocation. > > Signed-off-by: Claire Chang > Reviewed-by: Christoph Hellwig > --- > include/linux/swiotlb.h | 15 +++++++++++++ > kernel/dma/direct.c | 50 ++++++++++++++++++++++++++++++----------- > kernel/dma/swiotlb.c | 45 +++++++++++++++++++++++++++++++++++-- > 3 files changed, 95 insertions(+), 15 deletions(-) > > diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h > index efcd56e3a16c..2d5ec670e064 100644 > --- a/include/linux/swiotlb.h > +++ b/include/linux/swiotlb.h > @@ -156,4 +156,19 @@ static inline void swiotlb_adjust_size(unsigned long size) > extern void swiotlb_print_info(void); > extern void swiotlb_set_max_segment(unsigned int); > > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size); > +bool swiotlb_free(struct device *dev, struct page *page, size_t size); > +#else > +static inline struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + return NULL; > +} > +static inline bool swiotlb_free(struct device *dev, struct page *page, > + size_t size) > +{ > + return false; > +} > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > + > #endif /* __LINUX_SWIOTLB_H */ > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 3713461d6fe0..da0e09621230 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -75,6 +75,15 @@ static bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) > min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit); > } > > +static void __dma_direct_free_pages(struct device *dev, struct page *page, > + size_t size) > +{ > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL) && > + swiotlb_free(dev, page, size)) > + return; > + dma_free_contiguous(dev, page, size); > +} > + > static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > gfp_t gfp) > { > @@ -86,7 +95,16 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > gfp |= dma_direct_optimal_gfp_mask(dev, dev->coherent_dma_mask, > &phys_limit); > - page = dma_alloc_contiguous(dev, size, gfp); > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL)) { > + page = swiotlb_alloc(dev, size); > + if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > + __dma_direct_free_pages(dev, page, size); > + return NULL; > + } > + } > + > + if (!page) > + page = dma_alloc_contiguous(dev, size, gfp); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_free_contiguous(dev, page, size); > page = NULL; > @@ -142,7 +160,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > gfp |= __GFP_NOWARN; > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO); > if (!page) > return NULL; > @@ -155,18 +173,23 @@ void *dma_direct_alloc(struct device *dev, size_t size, > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) > return arch_dma_alloc(dev, size, dma_handle, gfp, attrs); Just noticed that after propagating swiotlb_force setting into io_tlb_default_mem->force, the memory allocation behavior for swiotlb_force will change (i.e. always skipping arch_dma_alloc and dma_direct_alloc_from_pool). > > /* > * Remapping or decrypting memory may block. If either is required and > * we can't block, allocate the memory from the atomic pools. > + * If restricted DMA (i.e., is_dev_swiotlb_force) is required, one must > + * set up another device coherent pool by shared-dma-pool and use > + * dma_alloc_from_dev_coherent instead. > */ > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > !gfpflags_allow_blocking(gfp) && > (force_dma_unencrypted(dev) || > - (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev)))) > + (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > + !dev_is_dma_coherent(dev))) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); And here. > > /* we always manually zero the memory once we are done */ > @@ -237,7 +260,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > return NULL; > } > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -247,15 +270,15 @@ void dma_direct_free(struct device *dev, size_t size, > unsigned int page_order = get_order(size); > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > /* cpu_addr is a struct page cookie, not a kernel address */ > dma_free_contiguous(dev, cpu_addr, size); > return; > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) { > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) { > arch_dma_free(dev, size, cpu_addr, dma_addr, attrs); > return; > } > @@ -273,7 +296,7 @@ void dma_direct_free(struct device *dev, size_t size, > else if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) > arch_dma_clear_uncached(cpu_addr, size); > > - dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size); > + __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); > } > > struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > @@ -283,7 +306,8 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > void *ret; > > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > - force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp)) > + force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); > > page = __dma_direct_alloc_pages(dev, size, gfp); > @@ -310,7 +334,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > *dma_handle = phys_to_dma_direct(dev, page_to_phys(page)); > return page; > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -329,7 +353,7 @@ void dma_direct_free_pages(struct device *dev, size_t size, > if (force_dma_unencrypted(dev)) > set_memory_encrypted((unsigned long)vaddr, 1 << page_order); > > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > } > > #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c > index fec4934b9926..6ad85b48f101 100644 > --- a/kernel/dma/swiotlb.c > +++ b/kernel/dma/swiotlb.c > @@ -462,8 +462,9 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, > > index = wrap = wrap_index(mem, ALIGN(mem->index, stride)); > do { > - if ((slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > - (orig_addr & iotlb_align_mask)) { > + if (orig_addr && > + (slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > + (orig_addr & iotlb_align_mask)) { > index = wrap_index(mem, index + 1); > continue; > } > @@ -702,3 +703,43 @@ static int __init swiotlb_create_default_debugfs(void) > late_initcall(swiotlb_create_default_debugfs); > > #endif > + > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; > + phys_addr_t tlb_addr; > + int index; > + > + /* > + * Skip io_tlb_default_mem since swiotlb_alloc doesn't support atomic > + * coherent allocation. Otherwise might break existing devices. > + * One must set up another device coherent pool by shared-dma-pool and > + * use dma_alloc_from_dev_coherent instead for atomic coherent > + * allocation to avoid memory remapping. > + */ > + if (!mem || mem == io_tlb_default_mem) > + return NULL; > + > + index = swiotlb_find_slots(dev, 0, size); > + if (index == -1) > + return NULL; > + > + tlb_addr = slot_addr(mem->start, index); > + > + return pfn_to_page(PFN_DOWN(tlb_addr)); > +} > + > +bool swiotlb_free(struct device *dev, struct page *page, size_t size) > +{ > + phys_addr_t tlb_addr = page_to_phys(page); > + > + if (!is_swiotlb_buffer(dev, tlb_addr)) > + return false; > + > + swiotlb_release_slots(dev, tlb_addr); > + > + return true; > +} > + > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > -- > 2.32.0.272.g935e593368-goog > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CF7EC48BE5 for ; Wed, 16 Jun 2021 04:04:41 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 101DE610A1 for ; Wed, 16 Jun 2021 04:04:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 101DE610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A9A836E4C9; Wed, 16 Jun 2021 04:04:40 +0000 (UTC) Received: from mail-il1-x12b.google.com (mail-il1-x12b.google.com [IPv6:2607:f8b0:4864:20::12b]) by gabe.freedesktop.org (Postfix) with ESMTPS id CFA796E4C9 for ; Wed, 16 Jun 2021 04:04:38 +0000 (UTC) Received: by mail-il1-x12b.google.com with SMTP id i17so1112713ilj.11 for ; Tue, 15 Jun 2021 21:04:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=ZGcLRBiht0jP6J+AXrtU54w8Uc+zr833JxnoG08K94QWWjvLCllFqXgKhb1YXYtuSI H2kUoXb56/euW7L5KGDSV7c5wQp5UCbd+ujOwhBYXXWkAOagA4ahf3bI5zABwigXHQRs 2V+ulAhHs0i9h1/h3ywLUL41gakNk5ma80vrs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=X5vx1U5LnSazY/NJTLpwUw5jZHBtErc8BG5xBeH1jV68NyG5Iq5pumytJERGY9DXla 4qnFJBpyxCBV9euK+0kv23McT3CMyWVeoencI1rBJnEpORltgIcaUsF7ANGCbv5Vt4+g 8IUG/+ZO96YJzi827rY3xpKHybKPbDuRDYxsP5jrP1K3mQQZxtxdDXTzGkaGhdia3tZT uzpyrivSAnxUd0e+rN4+A9yX4PLUMcaOiOR3cklHa9w2Q7gwCmlQYXeZOU5KbsyzvoSK ZBxC//iGKQ9nCB7bgK4GOVhENKyCqmUfoz5TISElQqE4rKM1NLqPUvcLPYV8nGdhfGFB jUkQ== X-Gm-Message-State: AOAM531c+otywzPj0y0G0eBPdScLxwZPye1UNrG6rhVbok7ilevPaKal dsE/KfltlX7KunIhT7sbV5MbfCDTtIZi2g== X-Google-Smtp-Source: ABdhPJxkaljxImCnCsiX51bSGFcPsHU6hP5NrLdb+6otbDHK+QuulKC3zncW1M/7+RvwkGMlxUWXLA== X-Received: by 2002:a92:3610:: with SMTP id d16mr2039347ila.16.1623816278075; Tue, 15 Jun 2021 21:04:38 -0700 (PDT) Received: from mail-io1-f54.google.com (mail-io1-f54.google.com. [209.85.166.54]) by smtp.gmail.com with ESMTPSA id y8sm579401ilg.27.2021.06.15.21.04.37 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 15 Jun 2021 21:04:37 -0700 (PDT) Received: by mail-io1-f54.google.com with SMTP id q15so1581774ioi.4 for ; Tue, 15 Jun 2021 21:04:37 -0700 (PDT) X-Received: by 2002:a5e:8513:: with SMTP id i19mr1992773ioj.50.1623816266935; Tue, 15 Jun 2021 21:04:26 -0700 (PDT) MIME-Version: 1.0 References: <20210616035240.840463-1-tientzu@chromium.org> <20210616035240.840463-10-tientzu@chromium.org> In-Reply-To: <20210616035240.840463-10-tientzu@chromium.org> From: Claire Chang Date: Wed, 16 Jun 2021 12:04:16 +0800 X-Gmail-Original-Message-ID: Message-ID: To: Rob Herring , mpe@ellerman.id.au, Joerg Roedel , Will Deacon , Frank Rowand , Konrad Rzeszutek Wilk , boris.ostrovsky@oracle.com, jgross@suse.com, Christoph Hellwig , Marek Szyprowski Subject: Re: [Intel-gfx] [PATCH v11 09/12] swiotlb: Add restricted DMA alloc/free support X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: heikki.krogerus@linux.intel.com, thomas.hellstrom@linux.intel.com, peterz@infradead.org, benh@kernel.crashing.org, dri-devel@lists.freedesktop.org, chris@chris-wilson.co.uk, grant.likely@arm.com, paulus@samba.org, mingo@kernel.org, Jianxiong Gao , sstabellini@kernel.org, Saravana Kannan , xypron.glpk@gmx.de, "Rafael J . Wysocki" , Bartosz Golaszewski , bskeggs@redhat.com, linux-pci@vger.kernel.org, xen-devel@lists.xenproject.org, Thierry Reding , intel-gfx@lists.freedesktop.org, matthew.auld@intel.com, linux-devicetree , airlied@linux.ie, Robin Murphy , Nicolas Boichat , Bjorn Helgaas , Dan Williams , Andy Shevchenko , Greg KH , Randy Dunlap , lkml , Tomasz Figa , "list@263.net:IOMMU DRIVERS" , Jim Quinlan , linuxppc-dev@lists.ozlabs.org, bauerman@linux.ibm.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" On Wed, Jun 16, 2021 at 11:54 AM Claire Chang wrote: > > Add the functions, swiotlb_{alloc,free} to support the memory allocation > from restricted DMA pool. > > The restricted DMA pool is preferred if available. > > Note that since coherent allocation needs remapping, one must set up > another device coherent pool by shared-dma-pool and use > dma_alloc_from_dev_coherent instead for atomic coherent allocation. > > Signed-off-by: Claire Chang > Reviewed-by: Christoph Hellwig > --- > include/linux/swiotlb.h | 15 +++++++++++++ > kernel/dma/direct.c | 50 ++++++++++++++++++++++++++++++----------- > kernel/dma/swiotlb.c | 45 +++++++++++++++++++++++++++++++++++-- > 3 files changed, 95 insertions(+), 15 deletions(-) > > diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h > index efcd56e3a16c..2d5ec670e064 100644 > --- a/include/linux/swiotlb.h > +++ b/include/linux/swiotlb.h > @@ -156,4 +156,19 @@ static inline void swiotlb_adjust_size(unsigned long size) > extern void swiotlb_print_info(void); > extern void swiotlb_set_max_segment(unsigned int); > > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size); > +bool swiotlb_free(struct device *dev, struct page *page, size_t size); > +#else > +static inline struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + return NULL; > +} > +static inline bool swiotlb_free(struct device *dev, struct page *page, > + size_t size) > +{ > + return false; > +} > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > + > #endif /* __LINUX_SWIOTLB_H */ > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 3713461d6fe0..da0e09621230 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -75,6 +75,15 @@ static bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) > min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit); > } > > +static void __dma_direct_free_pages(struct device *dev, struct page *page, > + size_t size) > +{ > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL) && > + swiotlb_free(dev, page, size)) > + return; > + dma_free_contiguous(dev, page, size); > +} > + > static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > gfp_t gfp) > { > @@ -86,7 +95,16 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > gfp |= dma_direct_optimal_gfp_mask(dev, dev->coherent_dma_mask, > &phys_limit); > - page = dma_alloc_contiguous(dev, size, gfp); > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL)) { > + page = swiotlb_alloc(dev, size); > + if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > + __dma_direct_free_pages(dev, page, size); > + return NULL; > + } > + } > + > + if (!page) > + page = dma_alloc_contiguous(dev, size, gfp); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_free_contiguous(dev, page, size); > page = NULL; > @@ -142,7 +160,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > gfp |= __GFP_NOWARN; > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO); > if (!page) > return NULL; > @@ -155,18 +173,23 @@ void *dma_direct_alloc(struct device *dev, size_t size, > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) > return arch_dma_alloc(dev, size, dma_handle, gfp, attrs); Just noticed that after propagating swiotlb_force setting into io_tlb_default_mem->force, the memory allocation behavior for swiotlb_force will change (i.e. always skipping arch_dma_alloc and dma_direct_alloc_from_pool). > > /* > * Remapping or decrypting memory may block. If either is required and > * we can't block, allocate the memory from the atomic pools. > + * If restricted DMA (i.e., is_dev_swiotlb_force) is required, one must > + * set up another device coherent pool by shared-dma-pool and use > + * dma_alloc_from_dev_coherent instead. > */ > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > !gfpflags_allow_blocking(gfp) && > (force_dma_unencrypted(dev) || > - (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev)))) > + (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > + !dev_is_dma_coherent(dev))) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); And here. > > /* we always manually zero the memory once we are done */ > @@ -237,7 +260,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > return NULL; > } > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -247,15 +270,15 @@ void dma_direct_free(struct device *dev, size_t size, > unsigned int page_order = get_order(size); > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > /* cpu_addr is a struct page cookie, not a kernel address */ > dma_free_contiguous(dev, cpu_addr, size); > return; > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) { > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) { > arch_dma_free(dev, size, cpu_addr, dma_addr, attrs); > return; > } > @@ -273,7 +296,7 @@ void dma_direct_free(struct device *dev, size_t size, > else if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) > arch_dma_clear_uncached(cpu_addr, size); > > - dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size); > + __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); > } > > struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > @@ -283,7 +306,8 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > void *ret; > > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > - force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp)) > + force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); > > page = __dma_direct_alloc_pages(dev, size, gfp); > @@ -310,7 +334,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > *dma_handle = phys_to_dma_direct(dev, page_to_phys(page)); > return page; > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -329,7 +353,7 @@ void dma_direct_free_pages(struct device *dev, size_t size, > if (force_dma_unencrypted(dev)) > set_memory_encrypted((unsigned long)vaddr, 1 << page_order); > > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > } > > #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c > index fec4934b9926..6ad85b48f101 100644 > --- a/kernel/dma/swiotlb.c > +++ b/kernel/dma/swiotlb.c > @@ -462,8 +462,9 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, > > index = wrap = wrap_index(mem, ALIGN(mem->index, stride)); > do { > - if ((slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > - (orig_addr & iotlb_align_mask)) { > + if (orig_addr && > + (slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > + (orig_addr & iotlb_align_mask)) { > index = wrap_index(mem, index + 1); > continue; > } > @@ -702,3 +703,43 @@ static int __init swiotlb_create_default_debugfs(void) > late_initcall(swiotlb_create_default_debugfs); > > #endif > + > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; > + phys_addr_t tlb_addr; > + int index; > + > + /* > + * Skip io_tlb_default_mem since swiotlb_alloc doesn't support atomic > + * coherent allocation. Otherwise might break existing devices. > + * One must set up another device coherent pool by shared-dma-pool and > + * use dma_alloc_from_dev_coherent instead for atomic coherent > + * allocation to avoid memory remapping. > + */ > + if (!mem || mem == io_tlb_default_mem) > + return NULL; > + > + index = swiotlb_find_slots(dev, 0, size); > + if (index == -1) > + return NULL; > + > + tlb_addr = slot_addr(mem->start, index); > + > + return pfn_to_page(PFN_DOWN(tlb_addr)); > +} > + > +bool swiotlb_free(struct device *dev, struct page *page, size_t size) > +{ > + phys_addr_t tlb_addr = page_to_phys(page); > + > + if (!is_swiotlb_buffer(dev, tlb_addr)) > + return false; > + > + swiotlb_release_slots(dev, tlb_addr); > + > + return true; > +} > + > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > -- > 2.32.0.272.g935e593368-goog > _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 638FCC48BE6 for ; Wed, 16 Jun 2021 04:10:56 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 15AAF611BE for ; Wed, 16 Jun 2021 04:10:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 15AAF611BE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.142574.263029 (Exim 4.92) (envelope-from ) id 1ltMt0-0008Fm-Gz; Wed, 16 Jun 2021 04:10:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 142574.263029; Wed, 16 Jun 2021 04:10:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ltMt0-0008Fd-Dz; Wed, 16 Jun 2021 04:10:42 +0000 Received: by outflank-mailman (input) for mailman id 142574; Wed, 16 Jun 2021 04:10:41 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ltMsy-0008FX-RZ for xen-devel@lists.xenproject.org; Wed, 16 Jun 2021 04:10:40 +0000 Received: from mail-pl1-x630.google.com (unknown [2607:f8b0:4864:20::630]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 6e203b4b-9854-4888-ab43-ca32fd1a0f39; Wed, 16 Jun 2021 04:10:39 +0000 (UTC) Received: by mail-pl1-x630.google.com with SMTP id b12so434473plg.11 for ; Tue, 15 Jun 2021 21:10:39 -0700 (PDT) Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com. [209.85.210.171]) by smtp.gmail.com with ESMTPSA id q3sm574733pfj.89.2021.06.15.21.10.37 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 15 Jun 2021 21:10:38 -0700 (PDT) Received: by mail-pf1-f171.google.com with SMTP id k6so1097700pfk.12 for ; Tue, 15 Jun 2021 21:10:37 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 6e203b4b-9854-4888-ab43-ca32fd1a0f39 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=ZGcLRBiht0jP6J+AXrtU54w8Uc+zr833JxnoG08K94QWWjvLCllFqXgKhb1YXYtuSI H2kUoXb56/euW7L5KGDSV7c5wQp5UCbd+ujOwhBYXXWkAOagA4ahf3bI5zABwigXHQRs 2V+ulAhHs0i9h1/h3ywLUL41gakNk5ma80vrs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=480iw3RHEmaThOE/pnDPB0pRHOKUr78KviuFxgV2m28=; b=SLpP25OccjLMmVjtlQs+O8iJb3Hl3l2EYLJvo4kFvzhMkrgV4LWlnnuKWOpzrJm2Ah OxqzWRLZpZKJeyIkcz9cLTgC77/zSfbrLI92Kdmu1ALIjMt806Fd/xhNkpwF/b/XOquz jJdARGNTEBEtjUOzUulAyA5fOkTJIUKOTwVtG/K6CXfBtnntepascbqXpErudEbohjBg 43HYorC36wnCR/z+6NBSu7NPa++63M1z6AkoyKZ/qkh83/S3mkYGrJvGNfHXKPfwybVF SCY6VMxxSoZ8fHVUXMs13YmDQQ6qc9cJfrY0BGoWEvhaG3UpDAsve31A/S1kPqX8btiF 3SZw== X-Gm-Message-State: AOAM530IUlGSJ90QJ+KHt+aOMNB4Vwnlp0x1ylV3Kmg1LIK8C7vkBCAe vm+nE7ktCNjDs8L48I8Ng3rOTlYEb3hQFw== X-Google-Smtp-Source: ABdhPJxmhx5jp+UKPKrepzpjcRAPAjmxKeIrHRMrkOSXJu4dgRguYWPzfrNi15Nna5jeEqMC7SMCIw== X-Received: by 2002:a17:90a:cf86:: with SMTP id i6mr548776pju.224.1623816638464; Tue, 15 Jun 2021 21:10:38 -0700 (PDT) X-Received: by 2002:a5e:8513:: with SMTP id i19mr1992773ioj.50.1623816266935; Tue, 15 Jun 2021 21:04:26 -0700 (PDT) MIME-Version: 1.0 References: <20210616035240.840463-1-tientzu@chromium.org> <20210616035240.840463-10-tientzu@chromium.org> In-Reply-To: <20210616035240.840463-10-tientzu@chromium.org> From: Claire Chang Date: Wed, 16 Jun 2021 12:04:16 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v11 09/12] swiotlb: Add restricted DMA alloc/free support To: Rob Herring , mpe@ellerman.id.au, Joerg Roedel , Will Deacon , Frank Rowand , Konrad Rzeszutek Wilk , boris.ostrovsky@oracle.com, jgross@suse.com, Christoph Hellwig , Marek Szyprowski Cc: benh@kernel.crashing.org, paulus@samba.org, "list@263.net:IOMMU DRIVERS" , sstabellini@kernel.org, Robin Murphy , grant.likely@arm.com, xypron.glpk@gmx.de, Thierry Reding , mingo@kernel.org, bauerman@linux.ibm.com, peterz@infradead.org, Greg KH , Saravana Kannan , "Rafael J . Wysocki" , heikki.krogerus@linux.intel.com, Andy Shevchenko , Randy Dunlap , Dan Williams , Bartosz Golaszewski , linux-devicetree , lkml , linuxppc-dev@lists.ozlabs.org, xen-devel@lists.xenproject.org, Nicolas Boichat , Jim Quinlan , Tomasz Figa , bskeggs@redhat.com, Bjorn Helgaas , chris@chris-wilson.co.uk, Daniel Vetter , airlied@linux.ie, dri-devel@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, jani.nikula@linux.intel.com, Jianxiong Gao , joonas.lahtinen@linux.intel.com, linux-pci@vger.kernel.org, maarten.lankhorst@linux.intel.com, matthew.auld@intel.com, rodrigo.vivi@intel.com, thomas.hellstrom@linux.intel.com Content-Type: text/plain; charset="UTF-8" On Wed, Jun 16, 2021 at 11:54 AM Claire Chang wrote: > > Add the functions, swiotlb_{alloc,free} to support the memory allocation > from restricted DMA pool. > > The restricted DMA pool is preferred if available. > > Note that since coherent allocation needs remapping, one must set up > another device coherent pool by shared-dma-pool and use > dma_alloc_from_dev_coherent instead for atomic coherent allocation. > > Signed-off-by: Claire Chang > Reviewed-by: Christoph Hellwig > --- > include/linux/swiotlb.h | 15 +++++++++++++ > kernel/dma/direct.c | 50 ++++++++++++++++++++++++++++++----------- > kernel/dma/swiotlb.c | 45 +++++++++++++++++++++++++++++++++++-- > 3 files changed, 95 insertions(+), 15 deletions(-) > > diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h > index efcd56e3a16c..2d5ec670e064 100644 > --- a/include/linux/swiotlb.h > +++ b/include/linux/swiotlb.h > @@ -156,4 +156,19 @@ static inline void swiotlb_adjust_size(unsigned long size) > extern void swiotlb_print_info(void); > extern void swiotlb_set_max_segment(unsigned int); > > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size); > +bool swiotlb_free(struct device *dev, struct page *page, size_t size); > +#else > +static inline struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + return NULL; > +} > +static inline bool swiotlb_free(struct device *dev, struct page *page, > + size_t size) > +{ > + return false; > +} > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > + > #endif /* __LINUX_SWIOTLB_H */ > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 3713461d6fe0..da0e09621230 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -75,6 +75,15 @@ static bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) > min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit); > } > > +static void __dma_direct_free_pages(struct device *dev, struct page *page, > + size_t size) > +{ > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL) && > + swiotlb_free(dev, page, size)) > + return; > + dma_free_contiguous(dev, page, size); > +} > + > static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > gfp_t gfp) > { > @@ -86,7 +95,16 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > gfp |= dma_direct_optimal_gfp_mask(dev, dev->coherent_dma_mask, > &phys_limit); > - page = dma_alloc_contiguous(dev, size, gfp); > + if (IS_ENABLED(CONFIG_DMA_RESTRICTED_POOL)) { > + page = swiotlb_alloc(dev, size); > + if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > + __dma_direct_free_pages(dev, page, size); > + return NULL; > + } > + } > + > + if (!page) > + page = dma_alloc_contiguous(dev, size, gfp); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_free_contiguous(dev, page, size); > page = NULL; > @@ -142,7 +160,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > gfp |= __GFP_NOWARN; > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO); > if (!page) > return NULL; > @@ -155,18 +173,23 @@ void *dma_direct_alloc(struct device *dev, size_t size, > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) > return arch_dma_alloc(dev, size, dma_handle, gfp, attrs); Just noticed that after propagating swiotlb_force setting into io_tlb_default_mem->force, the memory allocation behavior for swiotlb_force will change (i.e. always skipping arch_dma_alloc and dma_direct_alloc_from_pool). > > /* > * Remapping or decrypting memory may block. If either is required and > * we can't block, allocate the memory from the atomic pools. > + * If restricted DMA (i.e., is_dev_swiotlb_force) is required, one must > + * set up another device coherent pool by shared-dma-pool and use > + * dma_alloc_from_dev_coherent instead. > */ > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > !gfpflags_allow_blocking(gfp) && > (force_dma_unencrypted(dev) || > - (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev)))) > + (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > + !dev_is_dma_coherent(dev))) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); And here. > > /* we always manually zero the memory once we are done */ > @@ -237,7 +260,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, > return NULL; > } > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -247,15 +270,15 @@ void dma_direct_free(struct device *dev, size_t size, > unsigned int page_order = get_order(size); > > if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && > - !force_dma_unencrypted(dev)) { > + !force_dma_unencrypted(dev) && !is_dev_swiotlb_force(dev)) { > /* cpu_addr is a struct page cookie, not a kernel address */ > dma_free_contiguous(dev, cpu_addr, size); > return; > } > > if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED) && > - !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && > - !dev_is_dma_coherent(dev)) { > + !IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !dev_is_dma_coherent(dev) && > + !is_dev_swiotlb_force(dev)) { > arch_dma_free(dev, size, cpu_addr, dma_addr, attrs); > return; > } > @@ -273,7 +296,7 @@ void dma_direct_free(struct device *dev, size_t size, > else if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) > arch_dma_clear_uncached(cpu_addr, size); > > - dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size); > + __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); > } > > struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > @@ -283,7 +306,8 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > void *ret; > > if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && > - force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp)) > + force_dma_unencrypted(dev) && !gfpflags_allow_blocking(gfp) && > + !is_dev_swiotlb_force(dev)) > return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); > > page = __dma_direct_alloc_pages(dev, size, gfp); > @@ -310,7 +334,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, > *dma_handle = phys_to_dma_direct(dev, page_to_phys(page)); > return page; > out_free_pages: > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > return NULL; > } > > @@ -329,7 +353,7 @@ void dma_direct_free_pages(struct device *dev, size_t size, > if (force_dma_unencrypted(dev)) > set_memory_encrypted((unsigned long)vaddr, 1 << page_order); > > - dma_free_contiguous(dev, page, size); > + __dma_direct_free_pages(dev, page, size); > } > > #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c > index fec4934b9926..6ad85b48f101 100644 > --- a/kernel/dma/swiotlb.c > +++ b/kernel/dma/swiotlb.c > @@ -462,8 +462,9 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, > > index = wrap = wrap_index(mem, ALIGN(mem->index, stride)); > do { > - if ((slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > - (orig_addr & iotlb_align_mask)) { > + if (orig_addr && > + (slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != > + (orig_addr & iotlb_align_mask)) { > index = wrap_index(mem, index + 1); > continue; > } > @@ -702,3 +703,43 @@ static int __init swiotlb_create_default_debugfs(void) > late_initcall(swiotlb_create_default_debugfs); > > #endif > + > +#ifdef CONFIG_DMA_RESTRICTED_POOL > +struct page *swiotlb_alloc(struct device *dev, size_t size) > +{ > + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; > + phys_addr_t tlb_addr; > + int index; > + > + /* > + * Skip io_tlb_default_mem since swiotlb_alloc doesn't support atomic > + * coherent allocation. Otherwise might break existing devices. > + * One must set up another device coherent pool by shared-dma-pool and > + * use dma_alloc_from_dev_coherent instead for atomic coherent > + * allocation to avoid memory remapping. > + */ > + if (!mem || mem == io_tlb_default_mem) > + return NULL; > + > + index = swiotlb_find_slots(dev, 0, size); > + if (index == -1) > + return NULL; > + > + tlb_addr = slot_addr(mem->start, index); > + > + return pfn_to_page(PFN_DOWN(tlb_addr)); > +} > + > +bool swiotlb_free(struct device *dev, struct page *page, size_t size) > +{ > + phys_addr_t tlb_addr = page_to_phys(page); > + > + if (!is_swiotlb_buffer(dev, tlb_addr)) > + return false; > + > + swiotlb_release_slots(dev, tlb_addr); > + > + return true; > +} > + > +#endif /* CONFIG_DMA_RESTRICTED_POOL */ > -- > 2.32.0.272.g935e593368-goog >