From: Catalin Marinas <catalin.marinas@arm.com>
To: Arnd Bergmann <arnd@arndb.de>
Cc: Ding Tianhong <dingtianhong@huawei.com>,
"linux-arm-kernel@lists.infradead.org"
<linux-arm-kernel@lists.infradead.org>,
Will Deacon <Will.Deacon@arm.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: For the problem when using swiotlb
Date: Wed, 19 Nov 2014 11:29:10 +0000 [thread overview]
Message-ID: <20141119112910.GD7156@e104818-lin.cambridge.arm.com> (raw)
In-Reply-To: <1535751.CcvIi3DN4F@wuerfel>
On Wed, Nov 19, 2014 at 08:45:43AM +0000, Arnd Bergmann wrote:
> On Wednesday 19 November 2014 11:17:15 Ding Tianhong wrote:
> > On 2014/11/18 2:09, Catalin Marinas wrote:
> > > On Mon, Nov 17, 2014 at 12:18:42PM +0000, Arnd Bergmann wrote:
> > >> On Monday 17 November 2014 19:56:27 Ding Tianhong wrote:
> > >>> The commit 3690951fc6d42f3a0903987677d0e592c49dd8db(arm64: Use swiotlb late initialisation)
> > >>> switches the DMA mapping code to swiotlb_tlb_late_init_with_default_size(), this will occur a problem
> > >>> when I run the scsi stress tests, the message as below:
> > >>>
> > >>> sas_controller b1000000.sas: swiotlb buffer is full (sz: 65536 bytes)..
> > >>> DMA: Out of SW-IOMMU space for 65536 bytes at device b1000000.sas
> > >>>
> > >>> The reason is that the swiotlb_tlb_late_init_with_default_size() could only alloc 16M memory for DMA-mapping,
> > >>> and the param in cmdline "swiotlb=xxx" is useless because the get_free_pages() only use the buddy to assigned a
> > >>> maximum memory of 16M(The MAX_ORDER is 13 for 4k pages), obviously 16M is too small in many scenes, but
> > >>> the swiotlb_init() which could reserved a bigger memory as wished could work well for most drivers.
> > >>>
> > >>> I could not get a better way to fix this problem except to revert this patch, so could you please give me some
> > >>> advise and help me, thanks very much.
> > >>
> > >> In general, you should not need to use swiotlb for most devices, in
> > >> particular for high-performance devices like network or block.
> > >>
> > >> Please make sure that you have set up the dma-ranges properties in
> > >> your DT properly to allow 64-bit DMA if the device supports it.
> > >
> > > That's the problem indeed, the DMA API ends up using swiotlb bounce
> > > buffers because the physical address of the pages passed to (or
> > > allocated by) the driver are beyond 32-bit limit (which is the default
> > > dma mask).
> > >
> >
> > Thanks everyone, I think I found the way to fix it, need to enable DMA_CMA, to reserve a big memory
> > for CMA and set coherent mask for dev, then dma_alloc and dma_mapping will not use the swiotlb until
> > the memory out of mask or swiotlb_force is enabled.
> >
> > If I still understand uncorrectly, please inform me.
>
> Please do not use CMA to work around the problem, but fix the underlying bug
> instead.
Agree.
> The driver should call 'dma_set_mask_and_coherent()' with the appropriate
> dma mask, and check whether that succeeded. However, the code implementing
> dma_set_mask_and_coherent on arm64 also needs to be changed to look up
> the dma-ranges property (see of_dma_configure()), and check if the mask
> is possible.
dma_set_mask_and_coherent() is a generic function. I think the
of_dma_configure() should start with a coherent_dma_mask based on
dma-ranges if given rather than defaulting to DMA_BIT_MASK(32). The
comment in of_dma_configure() says that devices should set up the
supported mask but it's not always up to them but the bus they are
connected to.
Something like below, untested:
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 3b64d0bf5bba..dff34883db45 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -200,6 +200,10 @@ static void of_dma_configure(struct device *dev)
/* DMA ranges found. Calculate and set dma_pfn_offset */
dev->dma_pfn_offset = PFN_DOWN(paddr - dma_addr);
dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", dev->dma_pfn_offset);
+
+ /* Set the coherent_dma_mask based on the dma-ranges property */
+ if (size)
+ dev->coherent_dma_mask = DMA_BIT_MASK(ilog2(size));
}
/**
next prev parent reply other threads:[~2014-11-19 11:29 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-11-17 11:56 For the problem when using swiotlb Ding Tianhong
2014-11-17 12:18 ` Arnd Bergmann
2014-11-17 18:09 ` Catalin Marinas
2014-11-19 3:17 ` Ding Tianhong
2014-11-19 8:45 ` Arnd Bergmann
2014-11-19 11:29 ` Catalin Marinas [this message]
2014-11-19 12:48 ` Arnd Bergmann
2014-11-19 15:46 ` Catalin Marinas
2014-11-19 15:56 ` Arnd Bergmann
2014-11-21 11:06 ` Catalin Marinas
2014-11-21 11:26 ` Arnd Bergmann
2014-11-21 11:36 ` Catalin Marinas
2014-11-21 12:27 ` Arnd Bergmann
2014-11-20 2:57 ` Ding Tianhong
2014-11-20 7:40 ` Arnd Bergmann
2014-11-20 8:34 ` Ding Tianhong
2014-11-20 9:02 ` Arnd Bergmann
2014-11-20 9:21 ` Ding Tianhong
2014-11-21 9:35 ` Catalin Marinas
2014-11-21 10:32 ` Catalin Marinas
2014-11-21 12:48 ` Arnd Bergmann
2014-11-21 16:57 ` Catalin Marinas
2014-11-21 17:04 ` Arnd Bergmann
2014-11-21 17:51 ` Catalin Marinas
2014-11-21 18:09 ` Catalin Marinas
2014-11-24 20:12 ` Arnd Bergmann
2014-11-25 10:58 ` Catalin Marinas
2014-11-25 11:29 ` Russell King - ARM Linux
2014-11-25 12:23 ` Catalin Marinas
2014-11-27 2:36 ` Ding Tianhong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141119112910.GD7156@e104818-lin.cambridge.arm.com \
--to=catalin.marinas@arm.com \
--cc=Will.Deacon@arm.com \
--cc=arnd@arndb.de \
--cc=dingtianhong@huawei.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).