linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
To: Qian Cai <cai@lca.pw>
Cc: linux-rpi-kernel@lists.infradead.org, f.fainelli@gmail.com,
	will@kernel.org, marc.zyngier@arm.com, catalin.marinas@arm.com,
	linux-kernel@vger.kernel.org, Rob Herring <robh+dt@kernel.org>,
	linux-mm@kvack.org, mbrugger@suse.com, wahrenst@gmx.net,
	phill@raspberrypi.org, Robin Murphy <robin.murphy@arm.com>,
	Christoph Hellwig <hch@lst.de>,
	linux-arm-kernel@lists.infradead.org, m.szyprowski@samsung.com
Subject: Re: [PATCH v6 3/4] arm64: use both ZONE_DMA and ZONE_DMA32
Date: Mon, 21 Oct 2019 19:01:34 +0200	[thread overview]
Message-ID: <6703f8dab4a21fe4e1049f8f224502e1733bf72c.camel@suse.de> (raw)
In-Reply-To: <5E3CD9CA-76C3-4D46-BA0B-DEBF650E8950@lca.pw>

[-- Attachment #1: Type: text/plain, Size: 12333 bytes --]

On Mon, 2019-10-21 at 10:46 -0400, Qian Cai wrote:
> > On Oct 21, 2019, at 10:34 AM, Nicolas Saenz Julienne <nsaenzjulienne@suse.de
> > > wrote:
> > 
> > On Mon, 2019-10-21 at 10:15 -0400, Qian Cai wrote:
> > > > On Sep 11, 2019, at 2:25 PM, Nicolas Saenz Julienne <
> > > > nsaenzjulienne@suse.de>
> > > > wrote:
> > > > 
> > > > So far all arm64 devices have supported 32 bit DMA masks for their
> > > > peripherals. This is not true anymore for the Raspberry Pi 4 as most of
> > > > it's peripherals can only address the first GB of memory on a total of
> > > > up to 4 GB.
> > > > 
> > > > This goes against ZONE_DMA32's intent, as it's expected for ZONE_DMA32
> > > > to be addressable with a 32 bit mask. So it was decided to re-introduce
> > > > ZONE_DMA in arm64.
> > > > 
> > > > ZONE_DMA will contain the lower 1G of memory, which is currently the
> > > > memory area addressable by any peripheral on an arm64 device.
> > > > ZONE_DMA32 will contain the rest of the 32 bit addressable memory.
> > > > 
> > > > Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
> > > > Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
> > > > 
> > > > ---
> > > 
> > > With ZONE_DMA=y, this config will fail to reserve 512M CMA on a server,
> > > 
> > > https://raw.githubusercontent.com/cailca/linux-mm/master/arm64.config
> > > 
> > > CONFIG_DMA_CMA=y
> > > CONFIG_CMA_SIZE_MBYTES=64
> > > CONFIG_CMA_SIZE_SEL_MBYTES=y
> > > CONFIG_CMA_ALIGNMENT=8
> > > CONFIG_CMA=y
> > > CONFIG_CMA_DEBUGFS=y
> > > CONFIG_CMA_AREAS=7
> > > 
> > > Is this expected?
> > 
> > Not really, just tested cma=512M on a Raspberry Pi4, and it went well. The
> > only
> > thing on my build that differs from your config is CONFIG_CMA_DEBUGFS.
> > 
> > Could you post more information on the device you're experiencing this on?
> > Also
> > some logs.
> 
> With the above config, it does not even need "cma=512M" kernel cmdline.
> 
> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
> [    0.000000] Linux version 5.4.0-rc4-next-20191021+ (clang version 8.0.1
> (Red Hat 8.0.1-1.module+el8.1.0+3866+6be7f4d8)) #1 SMP Mon Oct 21 10:03:03 EDT
> 2019
> [    0.000000] Setting debug_guardpage_minorder to 1
> [    0.000000] efi: Getting EFI parameters from FDT:
> [    0.000000] efi: EFI v2.70 by American Megatrends
> [    0.000000] efi:  ESRT=0xf935ed98  SMBIOS=0xfcc90000  SMBIOS
> 3.0=0xfcc80000  ACPI 2.0=0xfac80000  MEMRESERVE=0xfacd1018 
> [    0.000000] esrt: Reserving ESRT space from 0x00000000f935ed98 to
> 0x00000000f935edd0.
> [    0.000000] crashkernel reserved: 0x00000097db400000 - 0x00000097fb400000
> (512 MB)
> [    0.000000] cma: Reserved 512 MiB at 0x00000000a0000000
> 
> With ZONE_DMA=y, it will say,
> 
> cma: Failed to reserve 512 MiB
> 
> The machine is a ThunderX2 server.
> 
> 
https://buy.hpe.com/us/en/servers/apollo-systems/apollo-70-system/apollo-70-system/hpe-apollo-70-system/p/1010742472
> 
> # lscpu
> Architecture:        aarch64
> Byte Order:          Little Endian
> CPU(s):              256
> On-line CPU(s) list: 0-255
> Thread(s) per core:  4
> Core(s) per socket:  32
> Socket(s):           2
> NUMA node(s):        2
> Vendor ID:           Cavium
> Model:               1
> Model name:          ThunderX2 99xx
> Stepping:            0x1
> BogoMIPS:            400.00
> L1d cache:           32K
> L1i cache:           32K
> L2 cache:            256K
> L3 cache:            32768K
> NUMA node0 CPU(s):   0-127
> NUMA node1 CPU(s):   128-255
> Flags:               fp asimd aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm

Hi Qian,
I happen to have access to a very similar machine:

thunderx25:~ # lscpu 
Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              224
On-line CPU(s) list: 0-223
Thread(s) per core:  4
Core(s) per socket:  28
Socket(s):           2
NUMA node(s):        2
Vendor ID:           Cavium
Model:               1
Model name:          ThunderX2 99xx
Stepping:            0x1
CPU max MHz:         2500.0000
CPU min MHz:         1000.0000
BogoMIPS:            400.00
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            32768K
NUMA node0 CPU(s):   0-111
NUMA node1 CPU(s):   112-223
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid
asimdrdm

I tested a kernel with your configuration plus CONFIG_ZONE_DMA=y yet I'm unable
to reproduce the error. The CMA allocation is successful.

[    0.000000][    T0] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[    0.000000][    T0] Linux version 5.4.0-rc4-next-20191021 (nico@linux-9qgx) (gcc version 9.2.1 20190903 [gcc-9-branch revision 275330] (SUSE Linux)) #60 SMP Mon Oct 21 18:48:51 CEST 2019
[    0.000000][    T0] printk: debug: ignoring loglevel setting.
[    0.000000][    T0] efi: Getting EFI parameters from FDT:
[    0.000000][    T0] efi: EFI v2.70 by American Megatrends
[    0.000000][    T0] efi:  ESRT=0xf10b4198  SMBIOS=0xfcc90000  SMBIOS 3.0=0xfcc80000  ACPI 2.0=0xf9670000  MEMRESERVE=0xf1117018
[    0.000000][    T0] esrt: Reserving ESRT space from 0x00000000f10b4198 to 0x00000000f10b41d0.
[    0.000000][    T0] cma: Reserved 512 MiB at 0x00000000a0000000
[    0.000000][    T0] ACPI: Early table checksum verification disabled
[    0.000000][    T0] ACPI: RSDP 0x00000000F9670000 000024 (v02 HPE   )
[    0.000000][    T0] ACPI: XSDT 0x00000000F9670028 0000DC (v01 HPE    ServerCL 01072009 AMI  00010013)
[    0.000000][    T0] ACPI: FACP 0x00000000F9670108 000114 (v06 HPE    ServerCL 01072009 AMI  00010013)
[    0.000000][    T0] ACPI: DSDT 0x00000000F9670220 000714 (v02 HPE    ServerCL 20150406 INTL 20170831)
[    0.000000][    T0] ACPI: FIDT 0x00000000F9670938 00009C (v01 HPE    ServerCL 01072009 AMI  00010013)
[    0.000000][    T0] ACPI: DBG2 0x00000000F96709D8 000062 (v00 HPE    ServerCL 00000000 INTL 20170831)
[    0.000000][    T0] ACPI: SPMI 0x00000000F9670A40 000041 (v05 HPE    ServerCL 00000000 AMI. 00000000)
[    0.000000][    T0] ACPI: PCCT 0x00000000F9670A88 000FB0 (v01 HPE    ServerCL 00000001 INTL 20170831)
[    0.000000][    T0] ACPI: SLIT 0x00000000F9671A38 000030 (v01 HPE    ServerCL 00000001 INTL 20170831)
[    0.000000][    T0] ACPI: SPMI 0x00000000F9671A68 000041 (v04 HPE    ServerCL 00000001 INTL 20170831)
[    0.000000][    T0] ACPI: SSDT 0x00000000F9671AB0 004217 (v02 HPE    N0BXPCI  20150406 INTL 20170831)
[    0.000000][    T0] ACPI: SSDT 0x00000000F9675CC8 019654 (v02 HPE    ServerCL 20150406 INTL 20170831)
[    0.000000][    T0] ACPI: SSDT 0x00000000F968F320 0041CB (v02 HPE    N1BXPCI  20150406 INTL 20170831)
[    0.000000][    T0] ACPI: SSDT 0x00000000F96934F0 01980C (v02 HPE    ServerCL 20150406 INTL 20170831)
[    0.000000][    T0] ACPI: BERT 0x00000000F96ACD00 000030 (v01 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: GTDT 0x00000000F96ACD30 00007C (v02 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: HEST 0x00000000F96ACDB0 000308 (v01 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: APIC 0x00000000F96AD0B8 00468C (v04 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: MCFG 0x00000000F96B1748 00003C (v01 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: NFIT 0x00000000F96B1788 000028 (v01 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: PPTT 0x00000000F96B17B0 0018B4 (v01 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: SRAT 0x00000000F96B3068 0010A8 (v03 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: IORT 0x00000000F96B4110 000688 (v00 HPE    ServerCL 20150406 CAVM 00000099)
[    0.000000][    T0] ACPI: BGRT 0x00000000F96B4798 000038 (v01 HPE    ServerCL 01072009 AMI  00010013)
[    0.000000][    T0] ACPI: SPCR 0x00000000F96B47D0 000050 (v02 HPE    ServerCL 01072009 AMI. 0005000D)
[    0.000000][    T0] ACPI: WSMT 0x00000000F96B4820 000028 (v01 HPE    ServerCL 01072009 AMI  00010013)
[    0.000000][    T0] ACPI: SPCR: Unexpected SPCR Access Width.  Defaulting to byte size
[    0.000000][    T0] ACPI: SPCR: console: pl011,mmio,0x402020000,115200
[    0.000000][    T0] ACPI: SRAT: Node 0 PXM 0 [mem 0x80000000-0xfeffffff]
[    0.000000][    T0] ACPI: SRAT: Node 0 PXM 0 [mem 0x880000000-0xfffffffff]
[    0.000000][    T0] ACPI: SRAT: Node 0 PXM 0 [mem 0x8800000000-0x89fcffffff]
[    0.000000][    T0] ACPI: SRAT: Node 1 PXM 1 [mem 0x89fd000000-0x93fcffffff]
[    0.000000][    T0] NUMA: NODE_DATA [mem 0x89fcff5bc0-0x89fcffffff]
[    0.000000][    T0] NUMA: NODE_DATA [mem 0x93fc5b5bc0-0x93fc5bffff]
[    0.000000][    T0] Zone ranges:
[    0.000000][    T0]   DMA      [mem 0x00000000802f0000-0x00000000bfffffff]
[    0.000000][    T0]   DMA32    [mem 0x00000000c0000000-0x00000000ffffffff]
[    0.000000][    T0]   Normal   [mem 0x0000000100000000-0x00000093fcffffff]
[    0.000000][    T0] Movable zone start for each node
[    0.000000][    T0] Early memory node ranges
[    0.000000][    T0]   node   0: [mem 0x00000000802f0000-0x000000008030ffff]
[    0.000000][    T0]   node   0: [mem 0x0000000080310000-0x00000000bfffffff]
[    0.000000][    T0]   node   0: [mem 0x00000000c0000000-0x00000000c0cbffff]
[    0.000000][    T0]   node   0: [mem 0x00000000c0cc0000-0x00000000f104ffff]
[    0.000000][    T0]   node   0: [mem 0x00000000f1050000-0x00000000f10affff]
[    0.000000][    T0]   node   0: [mem 0x00000000f10b0000-0x00000000f96fffff]
[    0.000000][    T0]   node   0: [mem 0x00000000f9700000-0x00000000f98affff]
[    0.000000][    T0]   node   0: [mem 0x00000000f98b0000-0x00000000fa92ffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fa930000-0x00000000faa6ffff]
[    0.000000][    T0]   node   0: [mem 0x00000000faa70000-0x00000000fabbffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fabc0000-0x00000000fabdffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fabe0000-0x00000000fadeffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fadf0000-0x00000000fae4ffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fae50000-0x00000000fc8cffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fc8d0000-0x00000000fc8dffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fc8e0000-0x00000000fca9ffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fcaa0000-0x00000000fcaaffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fcab0000-0x00000000fcb3ffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fcb40000-0x00000000fd1effff]
[    0.000000][    T0]   node   0: [mem 0x00000000fd1f0000-0x00000000feceffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fecf0000-0x00000000fed1ffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fed20000-0x00000000fed2ffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fed30000-0x00000000feddffff]
[    0.000000][    T0]   node   0: [mem 0x00000000fede0000-0x00000000feffffff]
[    0.000000][    T0]   node   0: [mem 0x0000000880000000-0x0000000fffffffff]
[    0.000000][    T0]   node   0: [mem 0x0000008800000000-0x00000089fcffffff]
[    0.000000][    T0]   node   1: [mem 0x00000089fd000000-0x00000093fcffffff]
[    0.000000][    T0] Zeroed struct page in unavailable ranges: 440 pages
[    0.000000][    T0] Initmem setup node 0 [mem 0x00000000802f0000-0x00000089fcffffff]
[    0.000000][    T0] On node 0 totalpages: 654289
[    0.000000][    T0]   DMA zone: 16 pages used for memmap
[    0.000000][    T0]   DMA zone: 0 pages reserved
[    0.000000][    T0]   DMA zone: 16337 pages, LIFO batch:3
[    0.000000][    T0]   DMA32 zone: 16 pages used for memmap
[    0.000000][    T0]   DMA32 zone: 16128 pages, LIFO batch:3
[    0.000000][    T0]   Normal zone: 608 pages used for memmap
[    0.000000][    T0]   Normal zone: 621824 pages, LIFO batch:3
[    0.000000][    T0] Initmem setup node 1 [mem 0x00000089fd000000-0x00000093fcffffff]
[    0.000000][    T0] On node 1 totalpages: 655360
[    0.000000][    T0]   Normal zone: 640 pages used for memmap
[    0.000000][    T0]   Normal zone: 655360 pages, LIFO batch:3

Could you enable CMA debugging to see if anything interesting comes out of it.

Regards,
Nicolas


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  reply	other threads:[~2019-10-21 17:01 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-11 18:25 [PATCH v6 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
2019-09-11 18:25 ` [PATCH v6 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
2019-09-11 18:25 ` [PATCH v6 2/4] arm64: rename variables used to calculate ZONE_DMA32's size Nicolas Saenz Julienne
2019-09-11 18:25 ` [PATCH v6 3/4] arm64: use both ZONE_DMA and ZONE_DMA32 Nicolas Saenz Julienne
2019-10-21 14:15   ` Qian Cai
2019-10-21 14:34     ` Nicolas Saenz Julienne
2019-10-21 14:46       ` Qian Cai
2019-10-21 17:01         ` Nicolas Saenz Julienne [this message]
2019-10-21 17:25           ` Qian Cai
2019-10-21 17:55             ` Nicolas Saenz Julienne
2019-10-21 20:36               ` Qian Cai
2019-10-22 11:23                 ` Nicolas Saenz Julienne
2019-10-23  7:11                   ` Matthias Brugger
2019-10-31 15:51                   ` Catalin Marinas
2019-10-31 16:04                     ` Nicolas Saenz Julienne
2019-10-31 18:02                       ` Catalin Marinas
2019-10-31 18:11                         ` Nicolas Saenz Julienne
2019-10-31 18:13                           ` Catalin Marinas
2019-12-03  5:08   ` John Stultz
2019-12-03  5:38     ` John Stultz
2019-12-03  6:03       ` John Stultz
2019-12-03 10:12         ` Will Deacon
2019-12-03 11:19           ` Catalin Marinas
2019-09-11 18:25 ` [PATCH v6 4/4] mm: refresh ZONE_DMA and ZONE_DMA32 comments in 'enum zone_type' Nicolas Saenz Julienne
2019-10-01 16:05 ` [PATCH v6 0/4] Raspberry Pi 4 DMA addressing support Catalin Marinas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6703f8dab4a21fe4e1049f8f224502e1733bf72c.camel@suse.de \
    --to=nsaenzjulienne@suse.de \
    --cc=cai@lca.pw \
    --cc=catalin.marinas@arm.com \
    --cc=f.fainelli@gmail.com \
    --cc=hch@lst.de \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-rpi-kernel@lists.infradead.org \
    --cc=m.szyprowski@samsung.com \
    --cc=marc.zyngier@arm.com \
    --cc=mbrugger@suse.com \
    --cc=phill@raspberrypi.org \
    --cc=robh+dt@kernel.org \
    --cc=robin.murphy@arm.com \
    --cc=wahrenst@gmx.net \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).