* [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-17 20:48 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-17 20:48 UTC (permalink / raw) To: Christoph Hellwig Cc: Marek Szyprowski, Robin Murphy, iommu, linux-kernel, linux-arm-kernel, Xinwei Kong hey, I'm seeing an issue on a couple of arm64 systems[*] where they spew ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are non-fatal, and bumping up cma to a large enough size (~128M) gets rid of them - but that seems suboptimal. Bisection shows that this started after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for coherent allocations"). It looks like __dma_direct_alloc_pages() is opportunistically using CMA memory but falls back to non-CMA if CMA disabled or unavailable. I've demonstrated that this fallback is indeed returning a valid pointer. So perhaps the issue is really just the warning emission. The following naive patch solves the problem for me - just silence the cma errors, since it looks like a soft error. But is there a better approach? [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 6310ad01f915b..0324aa606c173 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(gfp)) { page = dma_alloc_from_contiguous(dev, count, page_order, - gfp & __GFP_NOWARN); + true); if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { dma_release_from_contiguous(dev, page, count); page = NULL; ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-17 20:48 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-17 20:48 UTC (permalink / raw) To: Christoph Hellwig Cc: Xinwei Kong, linux-kernel, iommu, Robin Murphy, linux-arm-kernel, Marek Szyprowski hey, I'm seeing an issue on a couple of arm64 systems[*] where they spew ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are non-fatal, and bumping up cma to a large enough size (~128M) gets rid of them - but that seems suboptimal. Bisection shows that this started after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for coherent allocations"). It looks like __dma_direct_alloc_pages() is opportunistically using CMA memory but falls back to non-CMA if CMA disabled or unavailable. I've demonstrated that this fallback is indeed returning a valid pointer. So perhaps the issue is really just the warning emission. The following naive patch solves the problem for me - just silence the cma errors, since it looks like a soft error. But is there a better approach? [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 6310ad01f915b..0324aa606c173 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(gfp)) { page = dma_alloc_from_contiguous(dev, count, page_order, - gfp & __GFP_NOWARN); + true); if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { dma_release_from_contiguous(dev, page, count); page = NULL; _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-17 20:48 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-17 20:48 UTC (permalink / raw) To: Christoph Hellwig Cc: Xinwei Kong, linux-kernel, iommu, Robin Murphy, linux-arm-kernel hey, I'm seeing an issue on a couple of arm64 systems[*] where they spew ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are non-fatal, and bumping up cma to a large enough size (~128M) gets rid of them - but that seems suboptimal. Bisection shows that this started after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for coherent allocations"). It looks like __dma_direct_alloc_pages() is opportunistically using CMA memory but falls back to non-CMA if CMA disabled or unavailable. I've demonstrated that this fallback is indeed returning a valid pointer. So perhaps the issue is really just the warning emission. The following naive patch solves the problem for me - just silence the cma errors, since it looks like a soft error. But is there a better approach? [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 6310ad01f915b..0324aa606c173 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(gfp)) { page = dma_alloc_from_contiguous(dev, count, page_order, - gfp & __GFP_NOWARN); + true); if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { dma_release_from_contiguous(dev, page, count); page = NULL; _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-23 11:32 ` Robin Murphy 0 siblings, 0 replies; 13+ messages in thread From: Robin Murphy @ 2019-04-23 11:32 UTC (permalink / raw) To: dann frazier, Christoph Hellwig Cc: Marek Szyprowski, iommu, linux-kernel, linux-arm-kernel, Xinwei Kong On 17/04/2019 21:48, dann frazier wrote: > hey, > I'm seeing an issue on a couple of arm64 systems[*] where they spew > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > of them - but that seems suboptimal. Bisection shows that this started > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > coherent allocations"). It looks like __dma_direct_alloc_pages() > is opportunistically using CMA memory but falls back to non-CMA if CMA > disabled or unavailable. I've demonstrated that this fallback is > indeed returning a valid pointer. So perhaps the issue is really just > the warning emission. The CMA area being full isn't necessarily an ignorable non-problem, since it means you won't be able to allocate the kind of large buffers for which CMA was intended. The question is, is it actually filling up with allocations that deserve to be there, or is this the same as I've seen on a log from a ThunderX2 system where it's getting exhausted by thousands upon thousands of trivial single page allocations? If it's the latter (CONFIG_CMA_DEBUG should help shed some light if necessary), then that does lean towards spending a bit more effort on this idea: https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ Robin. > The following naive patch solves the problem for me - just silence the > cma errors, since it looks like a soft error. But is there a better > approach? > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 6310ad01f915b..0324aa606c173 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > /* CMA can be used only in the context which permits sleeping */ > if (gfpflags_allow_blocking(gfp)) { > page = dma_alloc_from_contiguous(dev, count, page_order, > - gfp & __GFP_NOWARN); > + true); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_release_from_contiguous(dev, page, count); > page = NULL; > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-23 11:32 ` Robin Murphy 0 siblings, 0 replies; 13+ messages in thread From: Robin Murphy @ 2019-04-23 11:32 UTC (permalink / raw) To: dann frazier, Christoph Hellwig Cc: iommu, Xinwei Kong, linux-kernel, linux-arm-kernel, Marek Szyprowski On 17/04/2019 21:48, dann frazier wrote: > hey, > I'm seeing an issue on a couple of arm64 systems[*] where they spew > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > of them - but that seems suboptimal. Bisection shows that this started > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > coherent allocations"). It looks like __dma_direct_alloc_pages() > is opportunistically using CMA memory but falls back to non-CMA if CMA > disabled or unavailable. I've demonstrated that this fallback is > indeed returning a valid pointer. So perhaps the issue is really just > the warning emission. The CMA area being full isn't necessarily an ignorable non-problem, since it means you won't be able to allocate the kind of large buffers for which CMA was intended. The question is, is it actually filling up with allocations that deserve to be there, or is this the same as I've seen on a log from a ThunderX2 system where it's getting exhausted by thousands upon thousands of trivial single page allocations? If it's the latter (CONFIG_CMA_DEBUG should help shed some light if necessary), then that does lean towards spending a bit more effort on this idea: https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ Robin. > The following naive patch solves the problem for me - just silence the > cma errors, since it looks like a soft error. But is there a better > approach? > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 6310ad01f915b..0324aa606c173 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > /* CMA can be used only in the context which permits sleeping */ > if (gfpflags_allow_blocking(gfp)) { > page = dma_alloc_from_contiguous(dev, count, page_order, > - gfp & __GFP_NOWARN); > + true); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_release_from_contiguous(dev, page, count); > page = NULL; > > > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-23 11:32 ` Robin Murphy 0 siblings, 0 replies; 13+ messages in thread From: Robin Murphy @ 2019-04-23 11:32 UTC (permalink / raw) To: dann frazier, Christoph Hellwig Cc: iommu, Xinwei Kong, linux-kernel, linux-arm-kernel On 17/04/2019 21:48, dann frazier wrote: > hey, > I'm seeing an issue on a couple of arm64 systems[*] where they spew > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > of them - but that seems suboptimal. Bisection shows that this started > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > coherent allocations"). It looks like __dma_direct_alloc_pages() > is opportunistically using CMA memory but falls back to non-CMA if CMA > disabled or unavailable. I've demonstrated that this fallback is > indeed returning a valid pointer. So perhaps the issue is really just > the warning emission. The CMA area being full isn't necessarily an ignorable non-problem, since it means you won't be able to allocate the kind of large buffers for which CMA was intended. The question is, is it actually filling up with allocations that deserve to be there, or is this the same as I've seen on a log from a ThunderX2 system where it's getting exhausted by thousands upon thousands of trivial single page allocations? If it's the latter (CONFIG_CMA_DEBUG should help shed some light if necessary), then that does lean towards spending a bit more effort on this idea: https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ Robin. > The following naive patch solves the problem for me - just silence the > cma errors, since it looks like a soft error. But is there a better > approach? > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 6310ad01f915b..0324aa606c173 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > /* CMA can be used only in the context which permits sleeping */ > if (gfpflags_allow_blocking(gfp)) { > page = dma_alloc_from_contiguous(dev, count, page_order, > - gfp & __GFP_NOWARN); > + true); > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > dma_release_from_contiguous(dev, page, count); > page = NULL; > > > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-23 18:03 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-23 18:03 UTC (permalink / raw) To: Robin Murphy Cc: Christoph Hellwig, Marek Szyprowski, iommu, linux-kernel, linux-arm-kernel, Xinwei Kong On Tue, Apr 23, 2019 at 5:32 AM Robin Murphy <robin.murphy@arm.com> wrote: > > On 17/04/2019 21:48, dann frazier wrote: > > hey, > > I'm seeing an issue on a couple of arm64 systems[*] where they spew > > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > > of them - but that seems suboptimal. Bisection shows that this started > > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > > coherent allocations"). It looks like __dma_direct_alloc_pages() > > is opportunistically using CMA memory but falls back to non-CMA if CMA > > disabled or unavailable. I've demonstrated that this fallback is > > indeed returning a valid pointer. So perhaps the issue is really just > > the warning emission. > > The CMA area being full isn't necessarily an ignorable non-problem, > since it means you won't be able to allocate the kind of large buffers > for which CMA was intended. The question is, is it actually filling up > with allocations that deserve to be there, or is this the same as I've > seen on a log from a ThunderX2 system where it's getting exhausted by > thousands upon thousands of trivial single page allocations? If it's the > latter (CONFIG_CMA_DEBUG should help shed some light if necessary), Appears so. Here's a histogram of count/size w/ a cma= large enough to avoid failures: $ dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count ([0-9]+)\,.*/\1/' | sort -n | uniq -c 2062 1 32 2 266 8 2 24 4 32 256 33 7 64 2 128 2 1024 -dann > then > that does lean towards spending a bit more effort on this idea: > > https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ > > Robin. > > > The following naive patch solves the problem for me - just silence the > > cma errors, since it looks like a soft error. But is there a better > > approach? > > > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > > index 6310ad01f915b..0324aa606c173 100644 > > --- a/kernel/dma/direct.c > > +++ b/kernel/dma/direct.c > > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > /* CMA can be used only in the context which permits sleeping */ > > if (gfpflags_allow_blocking(gfp)) { > > page = dma_alloc_from_contiguous(dev, count, page_order, > > - gfp & __GFP_NOWARN); > > + true); > > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > > dma_release_from_contiguous(dev, page, count); > > page = NULL; > > > > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-23 18:03 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-23 18:03 UTC (permalink / raw) To: Robin Murphy Cc: linux-kernel, iommu, Xinwei Kong, Christoph Hellwig, linux-arm-kernel, Marek Szyprowski On Tue, Apr 23, 2019 at 5:32 AM Robin Murphy <robin.murphy@arm.com> wrote: > > On 17/04/2019 21:48, dann frazier wrote: > > hey, > > I'm seeing an issue on a couple of arm64 systems[*] where they spew > > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > > of them - but that seems suboptimal. Bisection shows that this started > > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > > coherent allocations"). It looks like __dma_direct_alloc_pages() > > is opportunistically using CMA memory but falls back to non-CMA if CMA > > disabled or unavailable. I've demonstrated that this fallback is > > indeed returning a valid pointer. So perhaps the issue is really just > > the warning emission. > > The CMA area being full isn't necessarily an ignorable non-problem, > since it means you won't be able to allocate the kind of large buffers > for which CMA was intended. The question is, is it actually filling up > with allocations that deserve to be there, or is this the same as I've > seen on a log from a ThunderX2 system where it's getting exhausted by > thousands upon thousands of trivial single page allocations? If it's the > latter (CONFIG_CMA_DEBUG should help shed some light if necessary), Appears so. Here's a histogram of count/size w/ a cma= large enough to avoid failures: $ dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count ([0-9]+)\,.*/\1/' | sort -n | uniq -c 2062 1 32 2 266 8 2 24 4 32 256 33 7 64 2 128 2 1024 -dann > then > that does lean towards spending a bit more effort on this idea: > > https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ > > Robin. > > > The following naive patch solves the problem for me - just silence the > > cma errors, since it looks like a soft error. But is there a better > > approach? > > > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > > index 6310ad01f915b..0324aa606c173 100644 > > --- a/kernel/dma/direct.c > > +++ b/kernel/dma/direct.c > > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > /* CMA can be used only in the context which permits sleeping */ > > if (gfpflags_allow_blocking(gfp)) { > > page = dma_alloc_from_contiguous(dev, count, page_order, > > - gfp & __GFP_NOWARN); > > + true); > > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > > dma_release_from_contiguous(dev, page, count); > > page = NULL; > > > > > > > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-23 18:03 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-23 18:03 UTC (permalink / raw) To: Robin Murphy Cc: linux-kernel, iommu, Xinwei Kong, Christoph Hellwig, linux-arm-kernel On Tue, Apr 23, 2019 at 5:32 AM Robin Murphy <robin.murphy@arm.com> wrote: > > On 17/04/2019 21:48, dann frazier wrote: > > hey, > > I'm seeing an issue on a couple of arm64 systems[*] where they spew > > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > > of them - but that seems suboptimal. Bisection shows that this started > > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > > coherent allocations"). It looks like __dma_direct_alloc_pages() > > is opportunistically using CMA memory but falls back to non-CMA if CMA > > disabled or unavailable. I've demonstrated that this fallback is > > indeed returning a valid pointer. So perhaps the issue is really just > > the warning emission. > > The CMA area being full isn't necessarily an ignorable non-problem, > since it means you won't be able to allocate the kind of large buffers > for which CMA was intended. The question is, is it actually filling up > with allocations that deserve to be there, or is this the same as I've > seen on a log from a ThunderX2 system where it's getting exhausted by > thousands upon thousands of trivial single page allocations? If it's the > latter (CONFIG_CMA_DEBUG should help shed some light if necessary), Appears so. Here's a histogram of count/size w/ a cma= large enough to avoid failures: $ dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count ([0-9]+)\,.*/\1/' | sort -n | uniq -c 2062 1 32 2 266 8 2 24 4 32 256 33 7 64 2 128 2 1024 -dann > then > that does lean towards spending a bit more effort on this idea: > > https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ > > Robin. > > > The following naive patch solves the problem for me - just silence the > > cma errors, since it looks like a soft error. But is there a better > > approach? > > > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > > index 6310ad01f915b..0324aa606c173 100644 > > --- a/kernel/dma/direct.c > > +++ b/kernel/dma/direct.c > > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > /* CMA can be used only in the context which permits sleeping */ > > if (gfpflags_allow_blocking(gfp)) { > > page = dma_alloc_from_contiguous(dev, count, page_order, > > - gfp & __GFP_NOWARN); > > + true); > > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > > dma_release_from_contiguous(dev, page, count); > > page = NULL; > > > > > > > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-24 0:39 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-24 0:39 UTC (permalink / raw) To: Robin Murphy Cc: Christoph Hellwig, Marek Szyprowski, iommu, linux-kernel, linux-arm-kernel, Xinwei Kong On Tue, Apr 23, 2019 at 12:03 PM dann frazier <dann.frazier@canonical.com> wrote: > > On Tue, Apr 23, 2019 at 5:32 AM Robin Murphy <robin.murphy@arm.com> wrote: > > > > On 17/04/2019 21:48, dann frazier wrote: > > > hey, > > > I'm seeing an issue on a couple of arm64 systems[*] where they spew > > > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > > > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > > > of them - but that seems suboptimal. Bisection shows that this started > > > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > > > coherent allocations"). It looks like __dma_direct_alloc_pages() > > > is opportunistically using CMA memory but falls back to non-CMA if CMA > > > disabled or unavailable. I've demonstrated that this fallback is > > > indeed returning a valid pointer. So perhaps the issue is really just > > > the warning emission. > > > > The CMA area being full isn't necessarily an ignorable non-problem, > > since it means you won't be able to allocate the kind of large buffers > > for which CMA was intended. The question is, is it actually filling up > > with allocations that deserve to be there, or is this the same as I've > > seen on a log from a ThunderX2 system where it's getting exhausted by > > thousands upon thousands of trivial single page allocations? If it's the > > latter (CONFIG_CMA_DEBUG should help shed some light if necessary), > > Appears so. Here's a histogram of count/size w/ a cma= large enough to > avoid failures: > > $ dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count > ([0-9]+)\,.*/\1/' | sort -n | uniq -c > 2062 1 > 32 2 > 266 8 > 2 24 > 4 32 > 256 33 And IIUC, this is also a big culprit. The debugfs bitmap seems to show that the alignment of each of these leaves 31 pages unused, which adds up to 31MB! -dann > 7 64 > 2 128 > 2 1024 > > -dann > > > then > > that does lean towards spending a bit more effort on this idea: > > > > https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ > > > > Robin. > > > > > The following naive patch solves the problem for me - just silence the > > > cma errors, since it looks like a soft error. But is there a better > > > approach? > > > > > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > > > > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > > > index 6310ad01f915b..0324aa606c173 100644 > > > --- a/kernel/dma/direct.c > > > +++ b/kernel/dma/direct.c > > > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > > /* CMA can be used only in the context which permits sleeping */ > > > if (gfpflags_allow_blocking(gfp)) { > > > page = dma_alloc_from_contiguous(dev, count, page_order, > > > - gfp & __GFP_NOWARN); > > > + true); > > > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > > > dma_release_from_contiguous(dev, page, count); > > > page = NULL; > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-24 0:39 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-24 0:39 UTC (permalink / raw) To: Robin Murphy Cc: linux-kernel, iommu, Xinwei Kong, Christoph Hellwig, linux-arm-kernel, Marek Szyprowski On Tue, Apr 23, 2019 at 12:03 PM dann frazier <dann.frazier@canonical.com> wrote: > > On Tue, Apr 23, 2019 at 5:32 AM Robin Murphy <robin.murphy@arm.com> wrote: > > > > On 17/04/2019 21:48, dann frazier wrote: > > > hey, > > > I'm seeing an issue on a couple of arm64 systems[*] where they spew > > > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > > > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > > > of them - but that seems suboptimal. Bisection shows that this started > > > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > > > coherent allocations"). It looks like __dma_direct_alloc_pages() > > > is opportunistically using CMA memory but falls back to non-CMA if CMA > > > disabled or unavailable. I've demonstrated that this fallback is > > > indeed returning a valid pointer. So perhaps the issue is really just > > > the warning emission. > > > > The CMA area being full isn't necessarily an ignorable non-problem, > > since it means you won't be able to allocate the kind of large buffers > > for which CMA was intended. The question is, is it actually filling up > > with allocations that deserve to be there, or is this the same as I've > > seen on a log from a ThunderX2 system where it's getting exhausted by > > thousands upon thousands of trivial single page allocations? If it's the > > latter (CONFIG_CMA_DEBUG should help shed some light if necessary), > > Appears so. Here's a histogram of count/size w/ a cma= large enough to > avoid failures: > > $ dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count > ([0-9]+)\,.*/\1/' | sort -n | uniq -c > 2062 1 > 32 2 > 266 8 > 2 24 > 4 32 > 256 33 And IIUC, this is also a big culprit. The debugfs bitmap seems to show that the alignment of each of these leaves 31 pages unused, which adds up to 31MB! -dann > 7 64 > 2 128 > 2 1024 > > -dann > > > then > > that does lean towards spending a bit more effort on this idea: > > > > https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ > > > > Robin. > > > > > The following naive patch solves the problem for me - just silence the > > > cma errors, since it looks like a soft error. But is there a better > > > approach? > > > > > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > > > > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > > > index 6310ad01f915b..0324aa606c173 100644 > > > --- a/kernel/dma/direct.c > > > +++ b/kernel/dma/direct.c > > > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > > /* CMA can be used only in the context which permits sleeping */ > > > if (gfpflags_allow_blocking(gfp)) { > > > page = dma_alloc_from_contiguous(dev, count, page_order, > > > - gfp & __GFP_NOWARN); > > > + true); > > > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > > > dma_release_from_contiguous(dev, page, count); > > > page = NULL; > > > > > > > > > > > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-24 0:39 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-24 0:39 UTC (permalink / raw) To: Robin Murphy Cc: linux-kernel, iommu, Xinwei Kong, Christoph Hellwig, linux-arm-kernel On Tue, Apr 23, 2019 at 12:03 PM dann frazier <dann.frazier@canonical.com> wrote: > > On Tue, Apr 23, 2019 at 5:32 AM Robin Murphy <robin.murphy@arm.com> wrote: > > > > On 17/04/2019 21:48, dann frazier wrote: > > > hey, > > > I'm seeing an issue on a couple of arm64 systems[*] where they spew > > > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > > > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > > > of them - but that seems suboptimal. Bisection shows that this started > > > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > > > coherent allocations"). It looks like __dma_direct_alloc_pages() > > > is opportunistically using CMA memory but falls back to non-CMA if CMA > > > disabled or unavailable. I've demonstrated that this fallback is > > > indeed returning a valid pointer. So perhaps the issue is really just > > > the warning emission. > > > > The CMA area being full isn't necessarily an ignorable non-problem, > > since it means you won't be able to allocate the kind of large buffers > > for which CMA was intended. The question is, is it actually filling up > > with allocations that deserve to be there, or is this the same as I've > > seen on a log from a ThunderX2 system where it's getting exhausted by > > thousands upon thousands of trivial single page allocations? If it's the > > latter (CONFIG_CMA_DEBUG should help shed some light if necessary), > > Appears so. Here's a histogram of count/size w/ a cma= large enough to > avoid failures: > > $ dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count > ([0-9]+)\,.*/\1/' | sort -n | uniq -c > 2062 1 > 32 2 > 266 8 > 2 24 > 4 32 > 256 33 And IIUC, this is also a big culprit. The debugfs bitmap seems to show that the alignment of each of these leaves 31 pages unused, which adds up to 31MB! -dann > 7 64 > 2 128 > 2 1024 > > -dann > > > then > > that does lean towards spending a bit more effort on this idea: > > > > https://lore.kernel.org/lkml/20190327080821.GB20336@lst.de/ > > > > Robin. > > > > > The following naive patch solves the problem for me - just silence the > > > cma errors, since it looks like a soft error. But is there a better > > > approach? > > > > > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > > > > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > > > index 6310ad01f915b..0324aa606c173 100644 > > > --- a/kernel/dma/direct.c > > > +++ b/kernel/dma/direct.c > > > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > > /* CMA can be used only in the context which permits sleeping */ > > > if (gfpflags_allow_blocking(gfp)) { > > > page = dma_alloc_from_contiguous(dev, count, page_order, > > > - gfp & __GFP_NOWARN); > > > + true); > > > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > > > dma_release_from_contiguous(dev, page, count); > > > page = NULL; > > > > > > > > > > > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] arm64: swiotlb: cma_alloc error spew @ 2019-04-24 0:39 ` dann frazier 0 siblings, 0 replies; 13+ messages in thread From: dann frazier @ 2019-04-24 0:39 UTC (permalink / raw) To: Robin Murphy Cc: linux-kernel-u79uwXL29TY76Z2rM5mHXA, iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Xinwei Kong, Christoph Hellwig, linux-arm-kernel On Tue, Apr 23, 2019 at 12:03 PM dann frazier <dann.frazier-Z7WLFzj8eWMS+FvcfC7Uqw@public.gmane.org> wrote: > > On Tue, Apr 23, 2019 at 5:32 AM Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org> wrote: > > > > On 17/04/2019 21:48, dann frazier wrote: > > > hey, > > > I'm seeing an issue on a couple of arm64 systems[*] where they spew > > > ~10K "cma: cma_alloc: alloc failed" messages at boot. The errors are > > > non-fatal, and bumping up cma to a large enough size (~128M) gets rid > > > of them - but that seems suboptimal. Bisection shows that this started > > > after commit fafadcd16595 ("swiotlb: don't dip into swiotlb pool for > > > coherent allocations"). It looks like __dma_direct_alloc_pages() > > > is opportunistically using CMA memory but falls back to non-CMA if CMA > > > disabled or unavailable. I've demonstrated that this fallback is > > > indeed returning a valid pointer. So perhaps the issue is really just > > > the warning emission. > > > > The CMA area being full isn't necessarily an ignorable non-problem, > > since it means you won't be able to allocate the kind of large buffers > > for which CMA was intended. The question is, is it actually filling up > > with allocations that deserve to be there, or is this the same as I've > > seen on a log from a ThunderX2 system where it's getting exhausted by > > thousands upon thousands of trivial single page allocations? If it's the > > latter (CONFIG_CMA_DEBUG should help shed some light if necessary), > > Appears so. Here's a histogram of count/size w/ a cma= large enough to > avoid failures: > > $ dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count > ([0-9]+)\,.*/\1/' | sort -n | uniq -c > 2062 1 > 32 2 > 266 8 > 2 24 > 4 32 > 256 33 And IIUC, this is also a big culprit. The debugfs bitmap seems to show that the alignment of each of these leaves 31 pages unused, which adds up to 31MB! -dann > 7 64 > 2 128 > 2 1024 > > -dann > > > then > > that does lean towards spending a bit more effort on this idea: > > > > https://lore.kernel.org/lkml/20190327080821.GB20336-jcswGhMUV9g@public.gmane.org/ > > > > Robin. > > > > > The following naive patch solves the problem for me - just silence the > > > cma errors, since it looks like a soft error. But is there a better > > > approach? > > > > > > [*] APM X-Gene & HiSilicon Hi1620 w/ SMMU disabled > > > > > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > > > index 6310ad01f915b..0324aa606c173 100644 > > > --- a/kernel/dma/direct.c > > > +++ b/kernel/dma/direct.c > > > @@ -112,7 +112,7 @@ struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, > > > /* CMA can be used only in the context which permits sleeping */ > > > if (gfpflags_allow_blocking(gfp)) { > > > page = dma_alloc_from_contiguous(dev, count, page_order, > > > - gfp & __GFP_NOWARN); > > > + true); > > > if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { > > > dma_release_from_contiguous(dev, page, count); > > > page = NULL; > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2019-04-24 0:40 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-17 20:48 [RFC] arm64: swiotlb: cma_alloc error spew dann frazier 2019-04-17 20:48 ` dann frazier 2019-04-17 20:48 ` dann frazier 2019-04-23 11:32 ` Robin Murphy 2019-04-23 11:32 ` Robin Murphy 2019-04-23 11:32 ` Robin Murphy 2019-04-23 18:03 ` dann frazier 2019-04-23 18:03 ` dann frazier 2019-04-23 18:03 ` dann frazier 2019-04-24 0:39 ` dann frazier 2019-04-24 0:39 ` dann frazier 2019-04-24 0:39 ` dann frazier 2019-04-24 0:39 ` dann frazier
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.