From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756009AbaIZPMc (ORCPT ); Fri, 26 Sep 2014 11:12:32 -0400 Received: from mail-wi0-f182.google.com ([209.85.212.182]:38167 "EHLO mail-wi0-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755641AbaIZPMZ (ORCPT ); Fri, 26 Sep 2014 11:12:25 -0400 MIME-Version: 1.0 In-Reply-To: <542573EE.8070103@hurleysoftware.com> References: <54246506.50401@hurleysoftware.com> <20140925143555.1f276007@as> <5424AAD0.9010708@hurleysoftware.com> <542512AD.9070304@vmware.com> <20140926054005.5c7985c0@as> <542543D8.8020604@vmware.com> <54255EA5.3030207@redhat.com> <542573EE.8070103@hurleysoftware.com> Date: Fri, 26 Sep 2014 08:12:24 -0700 Message-ID: Subject: Re: page allocator bug in 3.16? From: Leann Ogasawara To: Peter Hurley Cc: Rik van Riel , Rob Clark , Thomas Hellstrom , Chuck Ebbert , Linus Torvalds , Hugh Dickens , Linux kernel , "dri-devel@lists.freedesktop.org" , linux-mm , Mel Gorman , Andrew Morton , Shaohua Li , Ingo Molnar , Marek Szyprowski , Kyungmin Park , Arnd Bergmann Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Sep 26, 2014 at 7:10 AM, Peter Hurley wrote: > [ +cc Leann Ogasawara, Marek Szyprowski, Kyungmin Park, Arnd Bergmann ] > > On 09/26/2014 08:40 AM, Rik van Riel wrote: >> On 09/26/2014 08:28 AM, Rob Clark wrote: >>> On Fri, Sep 26, 2014 at 6:45 AM, Thomas Hellstrom >>> wrote: >>>> On 09/26/2014 12:40 PM, Chuck Ebbert wrote: >>>>> On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom >>>>> wrote: >>>>> >>>>>> On 09/26/2014 01:52 AM, Peter Hurley wrote: >>>>>>> On 09/25/2014 03:35 PM, Chuck Ebbert wrote: >>>>>>>> There are six ttm patches queued for 3.16.4: >>>>>>>> >>>>>>>> drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch >>>>>>>> >>>>>>>> >> drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch >>>>>>>> drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch >>>>>>>> >>>>>>>> >> drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch >>>>>>>> drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch >>>>>>>> drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch >>>>>>> >>>>>>>> >> Thanks for info, Chuck. >>>>>>> >>>>>>> Unfortunately, none of these fix TTM dma allocation doing >>>>>>> CMA dma allocation, which is the root problem. >>>>>>> >>>>>>> Regards, Peter Hurley >>>>>> The problem is not really in TTM but in CMA, There was a guy >>>>>> offering to fix this in the CMA code but I guess he didn't >>>>>> probably because he didn't receive any feedback. >>>>>> >>>>> Yeah, the "solution" to this problem seems to be "don't enable >>>>> CMA on x86". Maybe it should even be disabled in the config >>>>> system. >>>> Or, as previously suggested, don't use CMA for order 0 (single >>>> page) allocations.... >>> >>> On devices that actually need CMA pools to arrange for memory to be >>> in certain ranges, I think you probably do want to have order 0 >>> pages come from the CMA pool. >>> >>> Seems like disabling CMA on x86 (where it should be unneeded) is >>> the better way, IMO >> >> CMA has its uses on x86. For example, CMA is used to allocate 1GB huge >> pages. >> >> There may also be people with devices that do not scatter-gather, and >> need a large physically contiguous buffer, though there should be >> relatively few of those on x86. >> >> I suspect it makes most sense to do DMA allocations up to PAGE_ORDER >> through the normal allocator on x86, and only invoking CMA for larger >> allocations. > > The code that uses CMA to satisfy DMA allocations on x86 is > specific to the x86 arch and was added in 2011 as a means of _testing_ > CMA in KVM: > > commit 0a2b9a6ea93650b8a00f9fd5ee8fdd25671e2df6 > Author: Marek Szyprowski > Date: Thu Dec 29 13:09:51 2011 +0100 > > X86: integrate CMA with DMA-mapping subsystem > > This patch adds support for CMA to dma-mapping subsystem for x86 > architecture that uses common pci-dma/pci-nommu implementation. This > allows to test CMA on KVM/QEMU and a lot of common x86 boxes. > > Signed-off-by: Marek Szyprowski > Signed-off-by: Kyungmin Park > CC: Michal Nazarewicz > Acked-by: Arnd Bergmann > > (no x86 maintainer acks?). > > Unfortunately, this code is enabled whenever CMA is enabled, rather > than as a separate test configuration. > > So, while enabling CMA may have other purposes on x86, using it for > x86 swiotlb and nommu dma allocations is not one of the them. > > And Ubuntu should not be enabling CONFIG_DMA_CMA for their i386 > and amd64 configurations, as this is trying to drive _all_ dma mapping > allocations through a _very_ small window (which is killing GPU > performance). Thanks for the note Peter. We do have this disabled for our upcoming Ubuntu 14.10 release. It is however still enabled in the previous 14.04 release. We have been tracking this in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1362261 but users able to reproduce performance impacts in 14.10 were unable to reproduce in 14.04 which is why we hadn't yet disabled it there. Thanks, Leann From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com [209.85.212.181]) by kanga.kvack.org (Postfix) with ESMTP id EEFD56B0038 for ; Fri, 26 Sep 2014 11:12:26 -0400 (EDT) Received: by mail-wi0-f181.google.com with SMTP id z2so11819122wiv.2 for ; Fri, 26 Sep 2014 08:12:26 -0700 (PDT) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by mx.google.com with ESMTPS id wg9si7038542wjb.70.2014.09.26.08.12.24 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 26 Sep 2014 08:12:25 -0700 (PDT) Received: by mail-wg0-f50.google.com with SMTP id l18so8974953wgh.33 for ; Fri, 26 Sep 2014 08:12:24 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <542573EE.8070103@hurleysoftware.com> References: <54246506.50401@hurleysoftware.com> <20140925143555.1f276007@as> <5424AAD0.9010708@hurleysoftware.com> <542512AD.9070304@vmware.com> <20140926054005.5c7985c0@as> <542543D8.8020604@vmware.com> <54255EA5.3030207@redhat.com> <542573EE.8070103@hurleysoftware.com> Date: Fri, 26 Sep 2014 08:12:24 -0700 Message-ID: Subject: Re: page allocator bug in 3.16? From: Leann Ogasawara Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Peter Hurley Cc: Rik van Riel , Rob Clark , Thomas Hellstrom , Chuck Ebbert , Linus Torvalds , Hugh Dickens , Linux kernel , "dri-devel@lists.freedesktop.org" , linux-mm , Mel Gorman , Andrew Morton , Shaohua Li , Ingo Molnar , Marek Szyprowski , Kyungmin Park , Arnd Bergmann On Fri, Sep 26, 2014 at 7:10 AM, Peter Hurley wrote: > [ +cc Leann Ogasawara, Marek Szyprowski, Kyungmin Park, Arnd Bergmann ] > > On 09/26/2014 08:40 AM, Rik van Riel wrote: >> On 09/26/2014 08:28 AM, Rob Clark wrote: >>> On Fri, Sep 26, 2014 at 6:45 AM, Thomas Hellstrom >>> wrote: >>>> On 09/26/2014 12:40 PM, Chuck Ebbert wrote: >>>>> On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom >>>>> wrote: >>>>> >>>>>> On 09/26/2014 01:52 AM, Peter Hurley wrote: >>>>>>> On 09/25/2014 03:35 PM, Chuck Ebbert wrote: >>>>>>>> There are six ttm patches queued for 3.16.4: >>>>>>>> >>>>>>>> drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch >>>>>>>> >>>>>>>> >> drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch >>>>>>>> drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch >>>>>>>> >>>>>>>> >> drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch >>>>>>>> drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch >>>>>>>> drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch >>>>>>> >>>>>>>> >> Thanks for info, Chuck. >>>>>>> >>>>>>> Unfortunately, none of these fix TTM dma allocation doing >>>>>>> CMA dma allocation, which is the root problem. >>>>>>> >>>>>>> Regards, Peter Hurley >>>>>> The problem is not really in TTM but in CMA, There was a guy >>>>>> offering to fix this in the CMA code but I guess he didn't >>>>>> probably because he didn't receive any feedback. >>>>>> >>>>> Yeah, the "solution" to this problem seems to be "don't enable >>>>> CMA on x86". Maybe it should even be disabled in the config >>>>> system. >>>> Or, as previously suggested, don't use CMA for order 0 (single >>>> page) allocations.... >>> >>> On devices that actually need CMA pools to arrange for memory to be >>> in certain ranges, I think you probably do want to have order 0 >>> pages come from the CMA pool. >>> >>> Seems like disabling CMA on x86 (where it should be unneeded) is >>> the better way, IMO >> >> CMA has its uses on x86. For example, CMA is used to allocate 1GB huge >> pages. >> >> There may also be people with devices that do not scatter-gather, and >> need a large physically contiguous buffer, though there should be >> relatively few of those on x86. >> >> I suspect it makes most sense to do DMA allocations up to PAGE_ORDER >> through the normal allocator on x86, and only invoking CMA for larger >> allocations. > > The code that uses CMA to satisfy DMA allocations on x86 is > specific to the x86 arch and was added in 2011 as a means of _testing_ > CMA in KVM: > > commit 0a2b9a6ea93650b8a00f9fd5ee8fdd25671e2df6 > Author: Marek Szyprowski > Date: Thu Dec 29 13:09:51 2011 +0100 > > X86: integrate CMA with DMA-mapping subsystem > > This patch adds support for CMA to dma-mapping subsystem for x86 > architecture that uses common pci-dma/pci-nommu implementation. This > allows to test CMA on KVM/QEMU and a lot of common x86 boxes. > > Signed-off-by: Marek Szyprowski > Signed-off-by: Kyungmin Park > CC: Michal Nazarewicz > Acked-by: Arnd Bergmann > > (no x86 maintainer acks?). > > Unfortunately, this code is enabled whenever CMA is enabled, rather > than as a separate test configuration. > > So, while enabling CMA may have other purposes on x86, using it for > x86 swiotlb and nommu dma allocations is not one of the them. > > And Ubuntu should not be enabling CONFIG_DMA_CMA for their i386 > and amd64 configurations, as this is trying to drive _all_ dma mapping > allocations through a _very_ small window (which is killing GPU > performance). Thanks for the note Peter. We do have this disabled for our upcoming Ubuntu 14.10 release. It is however still enabled in the previous 14.04 release. We have been tracking this in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1362261 but users able to reproduce performance impacts in 14.10 were unable to reproduce in 14.04 which is why we hadn't yet disabled it there. Thanks, Leann -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org