* [PATCH 0/3] support per-numa CMA for ARM server
@ 2020-06-03 2:42 Barry Song
2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song
` (2 more replies)
0 siblings, 3 replies; 17+ messages in thread
From: Barry Song @ 2020-06-03 2:42 UTC (permalink / raw)
To: hch, m.szyprowski, robin.murphy, catalin.marinas
Cc: linux-kernel, linuxarm, iommu, prime.zeng, linux-arm-kernel
Right now, smmu is using dma_alloc_coherent() to get memory to save queues
and tables. Typically, on ARM64 server, there is a default CMA located at
node0, which could be far away from node2, node3 etc.
Saving queues and tables remotely will increase the latency of ARM SMMU
significantly. For example, when SMMU is at node2 and the default global
CMA is at node0, after sending a CMD_SYNC in an empty command queue, we
have to wait more than 550ns for the completion of the command CMD_SYNC.
However, if we save them locally, we only need to wait for 240ns.
with per-numa CMA, smmu will get memory from local numa node to save command
queues and page tables. that means dma_unmap latency will be shrunk much.
Meanwhile, when iommu.passthrough is on, device drivers which call dma_
alloc_coherent() will also get local memory and avoid the travel between
numa nodes.
Barry Song (3):
dma-direct: provide the ability to reserve per-numa CMA
arm64: mm: reserve hugetlb CMA after numa_init
arm64: mm: reserve per-numa CMA after numa_init
arch/arm64/mm/init.c | 12 ++++++----
include/linux/dma-contiguous.h | 4 ++++
kernel/dma/Kconfig | 10 ++++++++
kernel/dma/contiguous.c | 43 +++++++++++++++++++++++++++++++++-
4 files changed, 63 insertions(+), 6 deletions(-)
--
2.23.0
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 17+ messages in thread* [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-03 2:42 [PATCH 0/3] support per-numa CMA for ARM server Barry Song @ 2020-06-03 2:42 ` Barry Song 2020-06-03 6:55 ` kbuild test robot ` (4 more replies) 2020-06-03 2:42 ` [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init Barry Song 2020-06-03 2:42 ` [PATCH 3/3] arm64: mm: reserve per-numa " Barry Song 2 siblings, 5 replies; 17+ messages in thread From: Barry Song @ 2020-06-03 2:42 UTC (permalink / raw) To: hch, m.szyprowski, robin.murphy, catalin.marinas Cc: linux-kernel, linuxarm, iommu, prime.zeng, Will Deacon, linux-arm-kernel This is useful for two scenarios: 1. ARM64 smmu will get memory from local numa node, it can save its command queues and page tables locally. Tests show it can decrease dma_unmap latency at lot. For example, without this patch, smmu on node2 will get memory from node0 by calling dma_alloc_coherent(), typically, it has to wait for more than 560ns for the completion of CMD_SYNC in an empty command queue; with this patch, it needs 240ns only. 2. when we set iommu passthrough, drivers will get memory from CMA, local memory means much less latency. Cc: Will Deacon <will@kernel.org> Cc: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> --- include/linux/dma-contiguous.h | 4 ++++ kernel/dma/Kconfig | 10 +++++++++ kernel/dma/contiguous.c | 41 +++++++++++++++++++++++++++++++++- 3 files changed, 54 insertions(+), 1 deletion(-) diff --git a/include/linux/dma-contiguous.h b/include/linux/dma-contiguous.h index 03f8e98e3bcc..278a80a40456 100644 --- a/include/linux/dma-contiguous.h +++ b/include/linux/dma-contiguous.h @@ -79,6 +79,8 @@ static inline void dma_contiguous_set_default(struct cma *cma) void dma_contiguous_reserve(phys_addr_t addr_limit); +void dma_pernuma_cma_reserve(void); + int __init dma_contiguous_reserve_area(phys_addr_t size, phys_addr_t base, phys_addr_t limit, struct cma **res_cma, bool fixed); @@ -128,6 +130,8 @@ static inline void dma_contiguous_set_default(struct cma *cma) { } static inline void dma_contiguous_reserve(phys_addr_t limit) { } +static inline void dma_pernuma_cma_reserve(void) { } + static inline int dma_contiguous_reserve_area(phys_addr_t size, phys_addr_t base, phys_addr_t limit, struct cma **res_cma, bool fixed) diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index 4c103a24e380..14f4a1fe4285 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -100,6 +100,16 @@ config DMA_CMA if DMA_CMA comment "Default contiguous memory area size:" +config CMA_PERNUMA_SIZE_MBYTES + int "Size in Mega Bytes for per-numa CMA areas" + depends on NUMA + default 16 if ARM64 + default 0 + help + Defines the size (in MiB) of the per-numa memory area for Contiguous + Memory Allocator. Every numa node will get a separate CMA with this + size. If the size of 0 is selected, per-numa CMA is disabled. + config CMA_SIZE_MBYTES int "Size in Mega Bytes" depends on !CMA_SIZE_SEL_PERCENTAGE diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c index 8bc6f2d670f9..4b10d0ca0456 100644 --- a/kernel/dma/contiguous.c +++ b/kernel/dma/contiguous.c @@ -30,7 +30,14 @@ #define CMA_SIZE_MBYTES 0 #endif +#ifdef CONFIG_CMA_PERNUMA_SIZE_MBYTES +#define CMA_SIZE_PERNUMA_MBYTES CONFIG_CMA_PERNUMA_SIZE_MBYTES +#else +#define CMA_SIZE_PERNUMA_MBYTES 0 +#endif + struct cma *dma_contiguous_default_area; +struct cma *dma_contiguous_pernuma_area[MAX_NUMNODES]; /* * Default global CMA area size can be defined in kernel's .config. @@ -44,6 +51,8 @@ struct cma *dma_contiguous_default_area; */ static const phys_addr_t size_bytes __initconst = (phys_addr_t)CMA_SIZE_MBYTES * SZ_1M; +static const phys_addr_t pernuma_size_bytes __initconst = + (phys_addr_t)CMA_SIZE_PERNUMA_MBYTES * SZ_1M; static phys_addr_t size_cmdline __initdata = -1; static phys_addr_t base_cmdline __initdata; static phys_addr_t limit_cmdline __initdata; @@ -96,6 +105,31 @@ static inline __maybe_unused phys_addr_t cma_early_percent_memory(void) #endif +void __init dma_pernuma_cma_reserve(void) +{ + int nid; + + if (!pernuma_size_bytes || nr_online_nodes <= 1) + return; + + for_each_node_state(nid, N_ONLINE) { + int ret; + + ret = cma_declare_contiguous_nid(0, pernuma_size_bytes, 0, 0, + 0, false, "pernuma", + &dma_contiguous_pernuma_area[nid], + nid); + if (ret) { + pr_warn("%s: reservation failed: err %d, node %d", __func__, + ret, nid); + continue; + } + + pr_debug("%s: reserved %llu MiB on node %d\n", __func__, + pernuma_size_bytes / SZ_1M, nid); + } +} + /** * dma_contiguous_reserve() - reserve area(s) for contiguous memory handling * @limit: End address of the reserved memory (optional, 0 for any). @@ -223,7 +257,8 @@ bool dma_release_from_contiguous(struct device *dev, struct page *pages, * * This function allocates contiguous memory buffer for specified device. It * first tries to use device specific contiguous memory area if available or - * the default global one, then tries a fallback allocation of normal pages. + * the per-numa ones and default global one, then tries a fallback allocation + * of normal pages. per-numa memory areas don't support address limit * * Note that it byapss one-page size of allocations from the global area as * the addresses within one page are always contiguous, so there is no need @@ -234,9 +269,13 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp) size_t count = size >> PAGE_SHIFT; struct page *page = NULL; struct cma *cma = NULL; + int nid = dev_to_node(dev); if (dev && dev->cma_area) cma = dev->cma_area; + else if ((nid != NUMA_NO_NODE) && dma_contiguous_pernuma_area[nid] + && !(gfp & (GFP_DMA | GFP_DMA32))) + cma = dma_contiguous_pernuma_area[nid]; else if (count > 1) cma = dma_contiguous_default_area; -- 2.23.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song @ 2020-06-03 6:55 ` kbuild test robot 2020-06-03 7:18 ` kbuild test robot ` (3 subsequent siblings) 4 siblings, 0 replies; 17+ messages in thread From: kbuild test robot @ 2020-06-03 6:55 UTC (permalink / raw) To: Barry Song, hch, m.szyprowski, robin.murphy, catalin.marinas Cc: kbuild-all, linux-kernel, linuxarm, iommu, linux-arm-kernel [-- Attachment #1: Type: text/plain, Size: 3177 bytes --] Hi Barry, I love your patch! Perhaps something to improve: [auto build test WARNING on arm64/for-next/core] [also build test WARNING on linus/master v5.7] [cannot apply to hch-configfs/for-next next-20200602] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Barry-Song/support-per-numa-CMA-for-ARM-server/20200603-104821 base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core config: i386-randconfig-r033-20200602 (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=i386 If you fix the issue, kindly add following tag as appropriate Reported-by: kbuild test robot <lkp@intel.com> All warnings (new ones prefixed by >>, old ones prefixed by <<): In file included from include/linux/kernel.h:15, from include/asm-generic/bug.h:19, from arch/x86/include/asm/bug.h:83, from include/linux/bug.h:5, from include/linux/mmdebug.h:5, from include/linux/mm.h:9, from include/linux/memblock.h:13, from kernel/dma/contiguous.c:21: kernel/dma/contiguous.c: In function 'dma_pernuma_cma_reserve': >> include/linux/kern_levels.h:5:18: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 3 has type 'unsigned int' [-Wformat=] 5 | #define KERN_SOH "001" /* ASCII Start Of Header */ | ^~~~~~ include/linux/printk.h:137:10: note: in definition of macro 'no_printk' 137 | printk(fmt, ##__VA_ARGS__); | ^~~ include/linux/kern_levels.h:15:20: note: in expansion of macro 'KERN_SOH' 15 | #define KERN_DEBUG KERN_SOH "7" /* debug-level messages */ | ^~~~~~~~ include/linux/printk.h:336:12: note: in expansion of macro 'KERN_DEBUG' 336 | no_printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__) | ^~~~~~~~~~ kernel/dma/contiguous.c:128:3: note: in expansion of macro 'pr_debug' 128 | pr_debug("%s: reserved %llu MiB on node %dn", __func__, | ^~~~~~~~ kernel/dma/contiguous.c:128:29: note: format string is defined here 128 | pr_debug("%s: reserved %llu MiB on node %dn", __func__, | ~~~^ | | | long long unsigned int | %u vim +5 include/linux/kern_levels.h 314ba3520e513a Joe Perches 2012-07-30 4 04d2c8c83d0e3a Joe Perches 2012-07-30 @5 #define KERN_SOH "\001" /* ASCII Start Of Header */ 04d2c8c83d0e3a Joe Perches 2012-07-30 6 #define KERN_SOH_ASCII '\001' 04d2c8c83d0e3a Joe Perches 2012-07-30 7 :::::: The code at line 5 was first introduced by commit :::::: 04d2c8c83d0e3ac5f78aeede51babb3236200112 printk: convert the format for KERN_<LEVEL> to a 2 byte pattern :::::: TO: Joe Perches <joe@perches.com> :::::: CC: Linus Torvalds <torvalds@linux-foundation.org> --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 33838 bytes --] [-- Attachment #3: Type: text/plain, Size: 156 bytes --] _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song 2020-06-03 6:55 ` kbuild test robot @ 2020-06-03 7:18 ` kbuild test robot 2020-06-03 7:18 ` [RFC PATCH] dma-direct: dma_contiguous_pernuma_area[] can be static kbuild test robot ` (2 subsequent siblings) 4 siblings, 0 replies; 17+ messages in thread From: kbuild test robot @ 2020-06-03 7:18 UTC (permalink / raw) To: Barry Song, hch, m.szyprowski, robin.murphy, catalin.marinas Cc: kbuild-all, linux-kernel, linuxarm, iommu, linux-arm-kernel [-- Attachment #1: Type: text/plain, Size: 1512 bytes --] Hi Barry, I love your patch! Perhaps something to improve: [auto build test WARNING on arm64/for-next/core] [also build test WARNING on linus/master v5.7] [cannot apply to next-20200602] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Barry-Song/support-per-numa-CMA-for-ARM-server/20200603-104821 base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core config: i386-randconfig-s002-20200602 (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.1-244-g0ee050a8-dirty # save the attached .config to linux build tree make W=1 C=1 ARCH=i386 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' If you fix the issue, kindly add following tag as appropriate Reported-by: kbuild test robot <lkp@intel.com> sparse warnings: (new ones prefixed by >>) >> kernel/dma/contiguous.c:40:12: sparse: sparse: symbol 'dma_contiguous_pernuma_area' was not declared. Should it be static? >> kernel/dma/contiguous.c:278:50: sparse: sparse: invalid access below 'dma_contiguous_pernuma_area' (-4 4) Please review and possibly fold the followup patch. --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 27963 bytes --] [-- Attachment #3: Type: text/plain, Size: 156 bytes --] _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] dma-direct: dma_contiguous_pernuma_area[] can be static 2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song 2020-06-03 6:55 ` kbuild test robot 2020-06-03 7:18 ` kbuild test robot @ 2020-06-03 7:18 ` kbuild test robot 2020-06-04 11:36 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Dan Carpenter 2020-06-05 13:57 ` kernel test robot 4 siblings, 0 replies; 17+ messages in thread From: kbuild test robot @ 2020-06-03 7:18 UTC (permalink / raw) To: Barry Song, hch, m.szyprowski, robin.murphy, catalin.marinas Cc: kbuild-all, linux-kernel, linuxarm, iommu, linux-arm-kernel Signed-off-by: kbuild test robot <lkp@intel.com> --- contiguous.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c index 4b10d0ca0456d..2094c8e0590ac 100644 --- a/kernel/dma/contiguous.c +++ b/kernel/dma/contiguous.c @@ -37,7 +37,7 @@ #endif struct cma *dma_contiguous_default_area; -struct cma *dma_contiguous_pernuma_area[MAX_NUMNODES]; +static struct cma *dma_contiguous_pernuma_area[MAX_NUMNODES]; /* * Default global CMA area size can be defined in kernel's .config. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song ` (2 preceding siblings ...) 2020-06-03 7:18 ` [RFC PATCH] dma-direct: dma_contiguous_pernuma_area[] can be static kbuild test robot @ 2020-06-04 11:36 ` Dan Carpenter 2020-06-05 6:04 ` Song Bao Hua (Barry Song) 2020-06-05 13:57 ` kernel test robot 4 siblings, 1 reply; 17+ messages in thread From: Dan Carpenter @ 2020-06-04 11:36 UTC (permalink / raw) To: kbuild, Barry Song, hch, m.szyprowski, robin.murphy, catalin.marinas Cc: kbuild-all, lkp, linux-kernel, linuxarm, iommu, linux-arm-kernel, Dan Carpenter [-- Attachment #1: Type: text/plain, Size: 3166 bytes --] Hi Barry, url: https://github.com/0day-ci/linux/commits/Barry-Song/support-per-numa-CMA-for-ARM-server/20200603-104821 base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core config: x86_64-randconfig-m001-20200603 (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> Reported-by: Dan Carpenter <dan.carpenter@oracle.com> smatch warnings: kernel/dma/contiguous.c:274 dma_alloc_contiguous() warn: variable dereferenced before check 'dev' (see line 272) # https://github.com/0day-ci/linux/commit/adb919e972c1cac3d8b11905d5258d23d3aac6a4 git remote add linux-review https://github.com/0day-ci/linux git remote update linux-review git checkout adb919e972c1cac3d8b11905d5258d23d3aac6a4 vim +/dev +274 kernel/dma/contiguous.c b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 267 struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp) b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 268 { 90ae409f9eb3bc kernel/dma/contiguous.c Christoph Hellwig 2019-08-20 269 size_t count = size >> PAGE_SHIFT; b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 270 struct page *page = NULL; bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 271 struct cma *cma = NULL; adb919e972c1ca kernel/dma/contiguous.c Barry Song 2020-06-03 @272 int nid = dev_to_node(dev); ^^^ Dereferenced inside function. bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 273 bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 @274 if (dev && dev->cma_area) ^^^ Too late. bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 275 cma = dev->cma_area; adb919e972c1ca kernel/dma/contiguous.c Barry Song 2020-06-03 276 else if ((nid != NUMA_NO_NODE) && dma_contiguous_pernuma_area[nid] adb919e972c1ca kernel/dma/contiguous.c Barry Song 2020-06-03 277 && !(gfp & (GFP_DMA | GFP_DMA32))) adb919e972c1ca kernel/dma/contiguous.c Barry Song 2020-06-03 278 cma = dma_contiguous_pernuma_area[nid]; bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 279 else if (count > 1) bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 280 cma = dma_contiguous_default_area; b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 281 b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 282 /* CMA can be used only in the context which permits sleeping */ b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen 2019-05-23 283 if (cma && gfpflags_allow_blocking(gfp)) { --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 32337 bytes --] [-- Attachment #3: Type: text/plain, Size: 156 bytes --] _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-04 11:36 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Dan Carpenter @ 2020-06-05 6:04 ` Song Bao Hua (Barry Song) 2020-06-05 8:57 ` Dan Carpenter 0 siblings, 1 reply; 17+ messages in thread From: Song Bao Hua (Barry Song) @ 2020-06-05 6:04 UTC (permalink / raw) To: Dan Carpenter, kbuild, hch, m.szyprowski, robin.murphy, catalin.marinas Cc: kbuild-all, lkp, linux-kernel, Linuxarm, iommu, linux-arm-kernel, Dan Carpenter > -----Original Message----- > From: Dan Carpenter [mailto:dan.carpenter@oracle.com] > Sent: Thursday, June 4, 2020 11:37 PM > To: kbuild@lists.01.org; Song Bao Hua (Barry Song) > <song.bao.hua@hisilicon.com>; hch@lst.de; m.szyprowski@samsung.com; > robin.murphy@arm.com; catalin.marinas@arm.com > Cc: lkp@intel.com; Dan Carpenter <error27@gmail.com>; > kbuild-all@lists.01.org; iommu@lists.linux-foundation.org; > linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; Linuxarm > <linuxarm@huawei.com>; Jonathan Cameron > <jonathan.cameron@huawei.com>; John Garry <john.garry@huawei.com> > Subject: Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa > CMA > > Hi Barry, > > url: > https://github.com/0day-ci/linux/commits/Barry-Song/support-per-numa-CM > A-for-ARM-server/20200603-104821 > base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git > for-next/core > config: x86_64-randconfig-m001-20200603 (attached as .config) > compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 > > If you fix the issue, kindly add following tag as appropriate > Reported-by: kernel test robot <lkp@intel.com> > Reported-by: Dan Carpenter <dan.carpenter@oracle.com> Dan, thanks! Good catch! as this patch has not been in mainline yet, is it correct to add these "reported-by" in patch v2? Barry > > smatch warnings: > kernel/dma/contiguous.c:274 dma_alloc_contiguous() warn: variable > dereferenced before check 'dev' (see line 272) > > # > https://github.com/0day-ci/linux/commit/adb919e972c1cac3d8b11905d525 > 8d23d3aac6a4 > git remote add linux-review https://github.com/0day-ci/linux git remote > update linux-review git checkout > adb919e972c1cac3d8b11905d5258d23d3aac6a4 > vim +/dev +274 kernel/dma/contiguous.c > > b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 267 struct page *dma_alloc_contiguous(struct device *dev, > size_t size, gfp_t gfp) > b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 268 { > 90ae409f9eb3bc kernel/dma/contiguous.c Christoph Hellwig > 2019-08-20 269 size_t count = size >> PAGE_SHIFT; > b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 270 struct page *page = NULL; > bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 271 struct cma *cma = NULL; > adb919e972c1ca kernel/dma/contiguous.c Barry Song > 2020-06-03 @272 int nid = dev_to_node(dev); > > ^^^ Dereferenced inside function. > > bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 273 > bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 @274 if (dev && dev->cma_area) > > ^^^ Too late. > > bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 275 cma = dev->cma_area; > adb919e972c1ca kernel/dma/contiguous.c Barry Song > 2020-06-03 276 else if ((nid != NUMA_NO_NODE) && > dma_contiguous_pernuma_area[nid] > adb919e972c1ca kernel/dma/contiguous.c Barry Song > 2020-06-03 277 && !(gfp & (GFP_DMA | GFP_DMA32))) > adb919e972c1ca kernel/dma/contiguous.c Barry Song > 2020-06-03 278 cma = dma_contiguous_pernuma_area[nid]; > bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 279 else if (count > 1) > bd2e75633c8012 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 280 cma = dma_contiguous_default_area; > b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 281 > b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 282 /* CMA can be used only in the context which permits > sleeping */ > b1d2dc009dece4 kernel/dma/contiguous.c Nicolin Chen > 2019-05-23 283 if (cma && gfpflags_allow_blocking(gfp)) { > > --- > 0-DAY CI Kernel Test Service, Intel Corporation > https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-05 6:04 ` Song Bao Hua (Barry Song) @ 2020-06-05 8:57 ` Dan Carpenter 2020-06-06 3:46 ` [kbuild-all] " Philip Li 0 siblings, 1 reply; 17+ messages in thread From: Dan Carpenter @ 2020-06-05 8:57 UTC (permalink / raw) To: Song Bao Hua (Barry Song) Cc: Dan Carpenter, lkp, catalin.marinas, kbuild, linux-kernel, Linuxarm, iommu, kbuild-all, robin.murphy, hch, linux-arm-kernel On Fri, Jun 05, 2020 at 06:04:31AM +0000, Song Bao Hua (Barry Song) wrote: > > > > -----Original Message----- > > From: Dan Carpenter [mailto:dan.carpenter@oracle.com] > > Sent: Thursday, June 4, 2020 11:37 PM > > To: kbuild@lists.01.org; Song Bao Hua (Barry Song) > > <song.bao.hua@hisilicon.com>; hch@lst.de; m.szyprowski@samsung.com; > > robin.murphy@arm.com; catalin.marinas@arm.com > > Cc: lkp@intel.com; Dan Carpenter <error27@gmail.com>; > > kbuild-all@lists.01.org; iommu@lists.linux-foundation.org; > > linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; Linuxarm > > <linuxarm@huawei.com>; Jonathan Cameron > > <jonathan.cameron@huawei.com>; John Garry <john.garry@huawei.com> > > Subject: Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa > > CMA > > > > Hi Barry, > > > > url: > > https://github.com/0day-ci/linux/commits/Barry-Song/support-per-numa-CM > > A-for-ARM-server/20200603-104821 > > base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git > > for-next/core > > config: x86_64-randconfig-m001-20200603 (attached as .config) > > compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 > > > > If you fix the issue, kindly add following tag as appropriate > > Reported-by: kernel test robot <lkp@intel.com> > > Reported-by: Dan Carpenter <dan.carpenter@oracle.com> > > Dan, thanks! Good catch! > as this patch has not been in mainline yet, is it correct to add these "reported-by" in patch v2? These are just an automated email from the zero day bot. I look over the Smatch warnings and then forward them on. regards, dan carpenter _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [kbuild-all] Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-05 8:57 ` Dan Carpenter @ 2020-06-06 3:46 ` Philip Li 2020-06-06 10:15 ` Song Bao Hua (Barry Song) 0 siblings, 1 reply; 17+ messages in thread From: Philip Li @ 2020-06-06 3:46 UTC (permalink / raw) To: Dan Carpenter Cc: Dan Carpenter, lkp, catalin.marinas, kbuild, linux-kernel, Linuxarm, iommu, kbuild-all, robin.murphy, hch, linux-arm-kernel On Fri, Jun 05, 2020 at 11:57:51AM +0300, Dan Carpenter wrote: > On Fri, Jun 05, 2020 at 06:04:31AM +0000, Song Bao Hua (Barry Song) wrote: > > > > > > > -----Original Message----- > > > From: Dan Carpenter [mailto:dan.carpenter@oracle.com] > > > Sent: Thursday, June 4, 2020 11:37 PM > > > To: kbuild@lists.01.org; Song Bao Hua (Barry Song) > > > <song.bao.hua@hisilicon.com>; hch@lst.de; m.szyprowski@samsung.com; > > > robin.murphy@arm.com; catalin.marinas@arm.com > > > Cc: lkp@intel.com; Dan Carpenter <error27@gmail.com>; > > > kbuild-all@lists.01.org; iommu@lists.linux-foundation.org; > > > linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; Linuxarm > > > <linuxarm@huawei.com>; Jonathan Cameron > > > <jonathan.cameron@huawei.com>; John Garry <john.garry@huawei.com> > > > Subject: Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa > > > CMA > > > > > > Hi Barry, > > > > > > url: > > > https://github.com/0day-ci/linux/commits/Barry-Song/support-per-numa-CM > > > A-for-ARM-server/20200603-104821 > > > base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git > > > for-next/core > > > config: x86_64-randconfig-m001-20200603 (attached as .config) > > > compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 > > > > > > If you fix the issue, kindly add following tag as appropriate > > > Reported-by: kernel test robot <lkp@intel.com> > > > Reported-by: Dan Carpenter <dan.carpenter@oracle.com> > > > > Dan, thanks! Good catch! > > as this patch has not been in mainline yet, is it correct to add these "reported-by" in patch v2? Hi Barry, we provides the suggestion here, but you may decide to add or not as appropriate for your situation. For the patch still under development, it is not that necessary to add i think. > > These are just an automated email from the zero day bot. I look over > the Smatch warnings and then forward them on. > > regards, > dan carpenter > _______________________________________________ > kbuild-all mailing list -- kbuild-all@lists.01.org > To unsubscribe send an email to kbuild-all-leave@lists.01.org _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [kbuild-all] Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-06 3:46 ` [kbuild-all] " Philip Li @ 2020-06-06 10:15 ` Song Bao Hua (Barry Song) 0 siblings, 0 replies; 17+ messages in thread From: Song Bao Hua (Barry Song) @ 2020-06-06 10:15 UTC (permalink / raw) To: Philip Li, Dan Carpenter Cc: Dan Carpenter, lkp, catalin.marinas, kbuild, linux-kernel, Linuxarm, iommu, kbuild-all, robin.murphy, hch, linux-arm-kernel > -----Original Message----- > From: Philip Li [mailto:philip.li@intel.com] > Sent: Saturday, June 6, 2020 3:47 PM > To: Dan Carpenter <dan.carpenter@oracle.com> > Cc: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>; > kbuild@lists.01.org; hch@lst.de; m.szyprowski@samsung.com; > robin.murphy@arm.com; catalin.marinas@arm.com; lkp@intel.com; Dan > Carpenter <error27@gmail.com>; kbuild-all@lists.01.org; > iommu@lists.linux-foundation.org; linux-arm-kernel@lists.infradead.org; > linux-kernel@vger.kernel.org; Linuxarm <linuxarm@huawei.com>; Jonathan > Cameron <jonathan.cameron@huawei.com>; John Garry > <john.garry@huawei.com> > Subject: Re: [kbuild-all] Re: [PATCH 1/3] dma-direct: provide the ability to > reserve per-numa CMA > > On Fri, Jun 05, 2020 at 11:57:51AM +0300, Dan Carpenter wrote: > > On Fri, Jun 05, 2020 at 06:04:31AM +0000, Song Bao Hua (Barry Song) > wrote: > > > > > > > > > > -----Original Message----- > > > > From: Dan Carpenter [mailto:dan.carpenter@oracle.com] > > > > Sent: Thursday, June 4, 2020 11:37 PM > > > > To: kbuild@lists.01.org; Song Bao Hua (Barry Song) > > > > <song.bao.hua@hisilicon.com>; hch@lst.de; > > > > m.szyprowski@samsung.com; robin.murphy@arm.com; > > > > catalin.marinas@arm.com > > > > Cc: lkp@intel.com; Dan Carpenter <error27@gmail.com>; > > > > kbuild-all@lists.01.org; iommu@lists.linux-foundation.org; > > > > linux-arm-kernel@lists.infradead.org; > > > > linux-kernel@vger.kernel.org; Linuxarm <linuxarm@huawei.com>; > > > > Jonathan Cameron <jonathan.cameron@huawei.com>; John Garry > > > > <john.garry@huawei.com> > > > > Subject: Re: [PATCH 1/3] dma-direct: provide the ability to > > > > reserve per-numa CMA > > > > > > > > Hi Barry, > > > > > > > > url: > > > > https://github.com/0day-ci/linux/commits/Barry-Song/support-per-nu > > > > ma-CM > > > > A-for-ARM-server/20200603-104821 > > > > base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git > > > > for-next/core > > > > config: x86_64-randconfig-m001-20200603 (attached as .config) > > > > compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 > > > > > > > > If you fix the issue, kindly add following tag as appropriate > > > > Reported-by: kernel test robot <lkp@intel.com> > > > > Reported-by: Dan Carpenter <dan.carpenter@oracle.com> > > > > > > Dan, thanks! Good catch! > > > as this patch has not been in mainline yet, is it correct to add these > "reported-by" in patch v2? > Hi Barry, we provides the suggestion here, but you may decide to add or not as > appropriate for your situation. For the patch still under development, it is not > that necessary to add i think. Hi Philip, Dan, Thanks for clarification. > > > > > These are just an automated email from the zero day bot. I look over > > the Smatch warnings and then forward them on. > > > > regards, > > dan carpenter Best regards Barry _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA 2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song ` (3 preceding siblings ...) 2020-06-04 11:36 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Dan Carpenter @ 2020-06-05 13:57 ` kernel test robot 4 siblings, 0 replies; 17+ messages in thread From: kernel test robot @ 2020-06-05 13:57 UTC (permalink / raw) To: Barry Song, hch, m.szyprowski, robin.murphy, catalin.marinas Cc: kbuild-all, linux-kernel, linuxarm, iommu, linux-arm-kernel [-- Attachment #1: Type: text/plain, Size: 4068 bytes --] Hi Barry, I love your patch! Perhaps something to improve: [auto build test WARNING on arm64/for-next/core] [also build test WARNING on linus/master v5.7] [cannot apply to next-20200605] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Barry-Song/support-per-numa-CMA-for-ARM-server/20200603-104821 base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core config: i386-randconfig-r016-20200605 (attached as .config) compiler: gcc-4.9 (Ubuntu 4.9.3-13ubuntu2) 4.9.3 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=i386 If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> All warnings (new ones prefixed by >>, old ones prefixed by <<): In file included from include/linux/printk.h:326:0, from include/linux/kernel.h:15, from include/asm-generic/bug.h:19, from arch/x86/include/asm/bug.h:83, from include/linux/bug.h:5, from include/linux/mmdebug.h:5, from include/linux/mm.h:9, from include/linux/memblock.h:13, from kernel/dma/contiguous.c:21: kernel/dma/contiguous.c: In function 'dma_pernuma_cma_reserve': >> include/linux/dynamic_debug.h:82:16: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 4 has type 'unsigned int' [-Wformat=] static struct _ddebug __aligned(8) ^ include/linux/dynamic_debug.h:123:2: note: in expansion of macro 'DEFINE_DYNAMIC_DEBUG_METADATA' DEFINE_DYNAMIC_DEBUG_METADATA(id, fmt); ^ include/linux/dynamic_debug.h:143:2: note: in expansion of macro '__dynamic_func_call' __dynamic_func_call(__UNIQUE_ID(ddebug), fmt, func, ##__VA_ARGS__) ^ include/linux/dynamic_debug.h:153:2: note: in expansion of macro '_dynamic_func_call' _dynamic_func_call(fmt, __dynamic_pr_debug, ^ include/linux/printk.h:330:2: note: in expansion of macro 'dynamic_pr_debug' dynamic_pr_debug(fmt, ##__VA_ARGS__) ^ kernel/dma/contiguous.c:128:3: note: in expansion of macro 'pr_debug' pr_debug("%s: reserved %llu MiB on node %dn", __func__, ^ vim +82 include/linux/dynamic_debug.h 923abb9d797ba0 Gal Pressman 2019-05-01 75 923abb9d797ba0 Gal Pressman 2019-05-01 76 extern __printf(3, 4) 923abb9d797ba0 Gal Pressman 2019-05-01 77 void __dynamic_ibdev_dbg(struct _ddebug *descriptor, 923abb9d797ba0 Gal Pressman 2019-05-01 78 const struct ib_device *ibdev, 923abb9d797ba0 Gal Pressman 2019-05-01 79 const char *fmt, ...); 923abb9d797ba0 Gal Pressman 2019-05-01 80 2bdde670beedf7 Rasmus Villemoes 2019-03-07 81 #define DEFINE_DYNAMIC_DEBUG_METADATA(name, fmt) \ c0d2af63786394 Joe Perches 2012-10-18 @82 static struct _ddebug __aligned(8) \ 07613b0b5ef857 Jason Baron 2011-10-04 83 __attribute__((section("__verbose"))) name = { \ 07613b0b5ef857 Jason Baron 2011-10-04 84 .modname = KBUILD_MODNAME, \ 07613b0b5ef857 Jason Baron 2011-10-04 85 .function = __func__, \ 07613b0b5ef857 Jason Baron 2011-10-04 86 .filename = __FILE__, \ 07613b0b5ef857 Jason Baron 2011-10-04 87 .format = (fmt), \ 07613b0b5ef857 Jason Baron 2011-10-04 88 .lineno = __LINE__, \ 07613b0b5ef857 Jason Baron 2011-10-04 89 .flags = _DPRINTK_FLAGS_DEFAULT, \ 2bdde670beedf7 Rasmus Villemoes 2019-03-07 90 _DPRINTK_KEY_INIT \ 07613b0b5ef857 Jason Baron 2011-10-04 91 } 07613b0b5ef857 Jason Baron 2011-10-04 92 :::::: The code at line 82 was first introduced by commit :::::: c0d2af637863940b1a4fb208224ca7acb905c39f dynamic_debug: Remove unnecessary __used :::::: TO: Joe Perches <joe@perches.com> :::::: CC: Greg Kroah-Hartman <gregkh@linuxfoundation.org> --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 37631 bytes --] [-- Attachment #3: Type: text/plain, Size: 156 bytes --] _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init 2020-06-03 2:42 [PATCH 0/3] support per-numa CMA for ARM server Barry Song 2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song @ 2020-06-03 2:42 ` Barry Song 2020-06-03 3:22 ` Roman Gushchin via iommu 2020-06-03 2:42 ` [PATCH 3/3] arm64: mm: reserve per-numa " Barry Song 2 siblings, 1 reply; 17+ messages in thread From: Barry Song @ 2020-06-03 2:42 UTC (permalink / raw) To: hch, m.szyprowski, robin.murphy, catalin.marinas Cc: linux-kernel, linuxarm, iommu, prime.zeng, Roman Gushchin, linux-arm-kernel hugetlb_cma_reserve() is called at the wrong place. numa_init has not been done yet. so all reserved memory will be located at node0. Cc: Roman Gushchin <guro@fb.com> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> --- arch/arm64/mm/init.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index e42727e3568e..8f0e70ebb49d 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -458,11 +458,6 @@ void __init arm64_memblock_init(void) high_memory = __va(memblock_end_of_DRAM() - 1) + 1; dma_contiguous_reserve(arm64_dma32_phys_limit); - -#ifdef CONFIG_ARM64_4K_PAGES - hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); -#endif - } void __init bootmem_init(void) @@ -478,6 +473,11 @@ void __init bootmem_init(void) min_low_pfn = min; arm64_numa_init(); + +#ifdef CONFIG_ARM64_4K_PAGES + hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); +#endif + /* * Sparsemem tries to allocate bootmem in memory_present(), so must be * done after the fixed reservations. -- 2.23.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init 2020-06-03 2:42 ` [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init Barry Song @ 2020-06-03 3:22 ` Roman Gushchin via iommu 2020-06-07 20:14 ` Matthias Brugger 0 siblings, 1 reply; 17+ messages in thread From: Roman Gushchin via iommu @ 2020-06-03 3:22 UTC (permalink / raw) To: Barry Song Cc: catalin.marinas, linux-kernel, linuxarm, iommu, prime.zeng, robin.murphy, hch, linux-arm-kernel On Wed, Jun 03, 2020 at 02:42:30PM +1200, Barry Song wrote: > hugetlb_cma_reserve() is called at the wrong place. numa_init has not been > done yet. so all reserved memory will be located at node0. > > Cc: Roman Gushchin <guro@fb.com> > Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> Acked-by: Roman Gushchin <guro@fb.com> Thanks! > --- > arch/arm64/mm/init.c | 10 +++++----- > 1 file changed, 5 insertions(+), 5 deletions(-) > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c > index e42727e3568e..8f0e70ebb49d 100644 > --- a/arch/arm64/mm/init.c > +++ b/arch/arm64/mm/init.c > @@ -458,11 +458,6 @@ void __init arm64_memblock_init(void) > high_memory = __va(memblock_end_of_DRAM() - 1) + 1; > > dma_contiguous_reserve(arm64_dma32_phys_limit); > - > -#ifdef CONFIG_ARM64_4K_PAGES > - hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); > -#endif > - > } > > void __init bootmem_init(void) > @@ -478,6 +473,11 @@ void __init bootmem_init(void) > min_low_pfn = min; > > arm64_numa_init(); > + > +#ifdef CONFIG_ARM64_4K_PAGES > + hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); > +#endif > + > /* > * Sparsemem tries to allocate bootmem in memory_present(), so must be > * done after the fixed reservations. > -- > 2.23.0 > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init 2020-06-03 3:22 ` Roman Gushchin via iommu @ 2020-06-07 20:14 ` Matthias Brugger 2020-06-08 0:50 ` Song Bao Hua (Barry Song) 0 siblings, 1 reply; 17+ messages in thread From: Matthias Brugger @ 2020-06-07 20:14 UTC (permalink / raw) To: Roman Gushchin, Barry Song Cc: catalin.marinas, linux-kernel, linuxarm, iommu, prime.zeng, robin.murphy, hch, linux-arm-kernel On 03/06/2020 05:22, Roman Gushchin wrote: > On Wed, Jun 03, 2020 at 02:42:30PM +1200, Barry Song wrote: >> hugetlb_cma_reserve() is called at the wrong place. numa_init has not been >> done yet. so all reserved memory will be located at node0. >> >> Cc: Roman Gushchin <guro@fb.com> >> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> > > Acked-by: Roman Gushchin <guro@fb.com> > When did this break or was it broken since the beginning? In any case, could you provide a "Fixes" tag for it, so that it can easily be backported to older releases. Regards, Matthias > Thanks! > >> --- >> arch/arm64/mm/init.c | 10 +++++----- >> 1 file changed, 5 insertions(+), 5 deletions(-) >> >> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c >> index e42727e3568e..8f0e70ebb49d 100644 >> --- a/arch/arm64/mm/init.c >> +++ b/arch/arm64/mm/init.c >> @@ -458,11 +458,6 @@ void __init arm64_memblock_init(void) >> high_memory = __va(memblock_end_of_DRAM() - 1) + 1; >> >> dma_contiguous_reserve(arm64_dma32_phys_limit); >> - >> -#ifdef CONFIG_ARM64_4K_PAGES >> - hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); >> -#endif >> - >> } >> >> void __init bootmem_init(void) >> @@ -478,6 +473,11 @@ void __init bootmem_init(void) >> min_low_pfn = min; >> >> arm64_numa_init(); >> + >> +#ifdef CONFIG_ARM64_4K_PAGES >> + hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); >> +#endif >> + >> /* >> * Sparsemem tries to allocate bootmem in memory_present(), so must be >> * done after the fixed reservations. >> -- >> 2.23.0 >> >> > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init 2020-06-07 20:14 ` Matthias Brugger @ 2020-06-08 0:50 ` Song Bao Hua (Barry Song) 2020-06-09 15:33 ` Matthias Brugger 0 siblings, 1 reply; 17+ messages in thread From: Song Bao Hua (Barry Song) @ 2020-06-08 0:50 UTC (permalink / raw) To: Matthias Brugger, Roman Gushchin Cc: catalin.marinas, linux-kernel, Linuxarm, iommu, Zengtao (B), robin.murphy, hch, linux-arm-kernel > -----Original Message----- > From: Matthias Brugger [mailto:matthias.bgg@gmail.com] > Sent: Monday, June 8, 2020 8:15 AM > To: Roman Gushchin <guro@fb.com>; Song Bao Hua (Barry Song) > <song.bao.hua@hisilicon.com> > Cc: catalin.marinas@arm.com; John Garry <john.garry@huawei.com>; > linux-kernel@vger.kernel.org; Linuxarm <linuxarm@huawei.com>; > iommu@lists.linux-foundation.org; Zengtao (B) <prime.zeng@hisilicon.com>; > Jonathan Cameron <jonathan.cameron@huawei.com>; > robin.murphy@arm.com; hch@lst.de; linux-arm-kernel@lists.infradead.org; > m.szyprowski@samsung.com > Subject: Re: [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init > > > > On 03/06/2020 05:22, Roman Gushchin wrote: > > On Wed, Jun 03, 2020 at 02:42:30PM +1200, Barry Song wrote: > >> hugetlb_cma_reserve() is called at the wrong place. numa_init has not been > >> done yet. so all reserved memory will be located at node0. > >> > >> Cc: Roman Gushchin <guro@fb.com> > >> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> > > > > Acked-by: Roman Gushchin <guro@fb.com> > > > > When did this break or was it broken since the beginning? > In any case, could you provide a "Fixes" tag for it, so that it can easily be > backported to older releases. I guess it was broken at the first beginning. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cf11e85fc08cc Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic hugepages using cma") Would you think it is better for me to send v2 for this patch separately with this tag and take this out of my original patch set for per-numa CMA? Please give your suggestion. Best Regards Barry > > Regards, > Matthias > > > Thanks! > > > >> --- > >> arch/arm64/mm/init.c | 10 +++++----- > >> 1 file changed, 5 insertions(+), 5 deletions(-) > >> > >> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c > >> index e42727e3568e..8f0e70ebb49d 100644 > >> --- a/arch/arm64/mm/init.c > >> +++ b/arch/arm64/mm/init.c > >> @@ -458,11 +458,6 @@ void __init arm64_memblock_init(void) > >> high_memory = __va(memblock_end_of_DRAM() - 1) + 1; > >> > >> dma_contiguous_reserve(arm64_dma32_phys_limit); > >> - > >> -#ifdef CONFIG_ARM64_4K_PAGES > >> - hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); > >> -#endif > >> - > >> } > >> > >> void __init bootmem_init(void) > >> @@ -478,6 +473,11 @@ void __init bootmem_init(void) > >> min_low_pfn = min; > >> > >> arm64_numa_init(); > >> + > >> +#ifdef CONFIG_ARM64_4K_PAGES > >> + hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); > >> +#endif > >> + > >> /* > >> * Sparsemem tries to allocate bootmem in memory_present(), so must > be > >> * done after the fixed reservations. > >> -- > >> 2.23.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init 2020-06-08 0:50 ` Song Bao Hua (Barry Song) @ 2020-06-09 15:33 ` Matthias Brugger 0 siblings, 0 replies; 17+ messages in thread From: Matthias Brugger @ 2020-06-09 15:33 UTC (permalink / raw) To: Song Bao Hua (Barry Song), Roman Gushchin Cc: catalin.marinas, linux-kernel, Linuxarm, iommu, Zengtao (B), robin.murphy, hch, linux-arm-kernel On 08/06/2020 02:50, Song Bao Hua (Barry Song) wrote: > > >> -----Original Message----- >> From: Matthias Brugger [mailto:matthias.bgg@gmail.com] >> Sent: Monday, June 8, 2020 8:15 AM >> To: Roman Gushchin <guro@fb.com>; Song Bao Hua (Barry Song) >> <song.bao.hua@hisilicon.com> >> Cc: catalin.marinas@arm.com; John Garry <john.garry@huawei.com>; >> linux-kernel@vger.kernel.org; Linuxarm <linuxarm@huawei.com>; >> iommu@lists.linux-foundation.org; Zengtao (B) <prime.zeng@hisilicon.com>; >> Jonathan Cameron <jonathan.cameron@huawei.com>; >> robin.murphy@arm.com; hch@lst.de; linux-arm-kernel@lists.infradead.org; >> m.szyprowski@samsung.com >> Subject: Re: [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init >> >> >> >> On 03/06/2020 05:22, Roman Gushchin wrote: >>> On Wed, Jun 03, 2020 at 02:42:30PM +1200, Barry Song wrote: >>>> hugetlb_cma_reserve() is called at the wrong place. numa_init has not been >>>> done yet. so all reserved memory will be located at node0. >>>> >>>> Cc: Roman Gushchin <guro@fb.com> >>>> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> >>> >>> Acked-by: Roman Gushchin <guro@fb.com> >>> >> >> When did this break or was it broken since the beginning? >> In any case, could you provide a "Fixes" tag for it, so that it can easily be >> backported to older releases. > > I guess it was broken at the first beginning. > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cf11e85fc08cc > > Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic hugepages using cma") > > Would you think it is better for me to send v2 for this patch separately with this tag and take this out of my original patch set for per-numa CMA? > Please give your suggestion. > I'm not the maintainer but I think it could help to get the patch accepted earlier while you address the rest of the series. Regards, Matthias _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 3/3] arm64: mm: reserve per-numa CMA after numa_init 2020-06-03 2:42 [PATCH 0/3] support per-numa CMA for ARM server Barry Song 2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song 2020-06-03 2:42 ` [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init Barry Song @ 2020-06-03 2:42 ` Barry Song 2 siblings, 0 replies; 17+ messages in thread From: Barry Song @ 2020-06-03 2:42 UTC (permalink / raw) To: hch, m.szyprowski, robin.murphy, catalin.marinas Cc: linux-kernel, linuxarm, iommu, prime.zeng, Will Deacon, linux-arm-kernel Right now, smmu is using dma_alloc_coherent() to get memory to save queues and tables. Typically, on ARM64 server, there is a default CMA located at node0, which could be far away from node2, node3 etc. with this patch, smmu will get memory from local numa node to save command queues and page tables. that means dma_unmap latency will be shrunk much. Meanwhile, when iommu.passthrough is on, device drivers which call dma_ alloc_coherent() will also get local memory and avoid the travel between numa nodes. Cc: Will Deacon <will@kernel.org> Cc: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> --- arch/arm64/mm/init.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 8f0e70ebb49d..204a534982b2 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -474,6 +474,8 @@ void __init bootmem_init(void) arm64_numa_init(); + dma_pernuma_cma_reserve(); + #ifdef CONFIG_ARM64_4K_PAGES hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); #endif -- 2.23.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 17+ messages in thread
end of thread, other threads:[~2020-06-09 15:33 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-06-03 2:42 [PATCH 0/3] support per-numa CMA for ARM server Barry Song 2020-06-03 2:42 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Barry Song 2020-06-03 6:55 ` kbuild test robot 2020-06-03 7:18 ` kbuild test robot 2020-06-03 7:18 ` [RFC PATCH] dma-direct: dma_contiguous_pernuma_area[] can be static kbuild test robot 2020-06-04 11:36 ` [PATCH 1/3] dma-direct: provide the ability to reserve per-numa CMA Dan Carpenter 2020-06-05 6:04 ` Song Bao Hua (Barry Song) 2020-06-05 8:57 ` Dan Carpenter 2020-06-06 3:46 ` [kbuild-all] " Philip Li 2020-06-06 10:15 ` Song Bao Hua (Barry Song) 2020-06-05 13:57 ` kernel test robot 2020-06-03 2:42 ` [PATCH 2/3] arm64: mm: reserve hugetlb CMA after numa_init Barry Song 2020-06-03 3:22 ` Roman Gushchin via iommu 2020-06-07 20:14 ` Matthias Brugger 2020-06-08 0:50 ` Song Bao Hua (Barry Song) 2020-06-09 15:33 ` Matthias Brugger 2020-06-03 2:42 ` [PATCH 3/3] arm64: mm: reserve per-numa " Barry Song
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).