Re: [PATCH 0/6] iomem cacheability

[Julien Grall <julien.grall@arm.com>]

Hi,

On 3/8/19 10:10 AM, Amit Tomer wrote:
>> Yes, you are right. I made a couple of quick fixes for this issue and
>> another issue raised by Julien during review (the usage of p2m_ram_t).
>> Amit, if you would like to give it a try I have a work-in-progress
>> branch with the fixes here:
> 
> We did try new branch with new fixes but we see some other crash now.
> 
> XEN) Loading kernel from boot module @ 000000007a000000
> (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
> (XEN) No bank has been allocated below 4GB.
> (XEN) BANK[0] 0x00000500000000-0x00000540000000 (1024MB)
> (XEN) BANK[1] 0x00000600000000-0x00000640000000 (1024MB)
> (XEN) Grant table range: 0x0000073fe00000-0x0000073fe40000
> (XEN) Allocating PPI 16 for event channel interrupt
> (XEN) Loading zImage from 000000007a000000 to 0000000500080000-0000000501880000
> (XEN) Loading dom0 DTB to 0x0000000508000000-0x00000005080117d0
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Scrubbing free RAM on in background
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: All
> (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
> input to Xen)
> (XEN) Freed 292kB init memory.
> [    0.000000] Booting Linux on physical CPU 0x0
> [    0.000000] Boot CPU: AArch64 Processor [411fd073]
> [    0.000000] Machine model: Renesas H3ULCB board based on r8a7795 ES2.0+
> [    0.000000] earlycon: xenboot0 at I/O port 0x0 (options '')
> [    0.000000] bootconsole [xenboot0] enabled
> [    0.000000] Xen 4.12 support found
> [    0.000000] efi: Getting EFI parameters from FDT:
> [    0.000000] efi: UEFI not found.
> [    0.000000] Reserved memory: created CMA memory pool at
> 0x0000000057000000, size 400 MiB
> [    0.000000] OF: reserved mem: initialized node linux,cma@57000000,
> compatible id shared-dma-pool
> [    0.000000] Reserved memory: created CMA memory pool at
> 0x0000000070000000, size 256 MiB
> [    0.000000] OF: reserved mem: initialized node
> linux,multimedia@70000000, compatible id shared-dma-pool
> [    0.000000] cma: dma_contiguous_reserve(limit 100000000)
> [    0.000000] NUMA: No NUMA configuration found
> [    0.000000] NUMA: Faking a node at [mem
> 0x0000000000000000-0x000000063fffffff]
> [    0.000000] NUMA: NODE_DATA [mem 0x63ff90c00-0x63ff926ff]
> [    0.000000] Zone ranges:
> [    0.000000]   DMA      [mem 0x0000000057000000-0x00000000ffffffff]
> [    0.000000]   Normal   [mem 0x0000000100000000-0x000000063fffffff]
> [    0.000000] Movable zone start for each node
> [    0.000000] Early memory node ranges
> [    0.000000]   node   0: [mem 0x0000000057000000-0x000000007fffffff]
> [    0.000000]   node   0: [mem 0x0000000500000000-0x000000053fffffff]
> [    0.000000]   node   0: [mem 0x0000000600000000-0x000000063fffffff]
> [    0.000000] Initmem setup node 0 [mem 0x0000000057000000-0x000000063fffffff]
> [    0.000000] On node 0 totalpages: 692224
> [    0.000000]   DMA zone: 2624 pages used for memmap
> [    0.000000]   DMA zone: 0 pages reserved
> [    0.000000]   DMA zone: 167936 pages, LIFO batch:31
> [    0.000000]   Normal zone: 8192 pages used for memmap
> [    0.000000]   Normal zone: 524288 pages, LIFO batch:31
> [    0.000000] bootmem alloc of 64 bytes failed!
> [    0.000000] Kernel panic - not syncing: Out of memory
> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted
> 4.14.75-ltsi-yocto-standard #3
> [    0.000000] Hardware name: Renesas H3ULCB board based on r8a7795 ES2.0+ (DT)
> [    0.000000] Call trace:
> [    0.000000] [<ffff000008089ae8>] dump_backtrace+0x0/0x3c0
> [    0.000000] [<ffff000008089ebc>] show_stack+0x14/0x20
> [    0.000000] [<ffff000008af20e8>] dump_stack+0x9c/0xbc
> [    0.000000] [<ffff0000080ce770>] panic+0x11c/0x28c
> [    0.000000] [<ffff0000090478fc>] free_bootmem_late+0x0/0x7c
> [    0.000000] [<ffff000009047d90>] __alloc_bootmem_low+0x2c/0x38
> [    0.000000] [<ffff0000090329fc>] setup_arch+0x258/0x4d8
> [    0.000000] [<ffff00000903083c>] start_kernel+0x64/0x3ac
> [    0.000000] ---[ end Kernel panic - not syncing: Out of memory

Interesting, the function __alloc_bootmem_low will try to allocate low 
memory. The limit for low memory is based on the physical DMA address 
limit (see ARCH_LOW_ADDRESS_LIMIT).

I guess you have CONFIG_ZONE_DMA32 (was renamed from CONFIG_ZONE_DMA is 
recent Linux) set. So the DMA limit will be computed by max_zone_dma_phys().

In your setup, if I am not mistaken, the limit will be 4GB. However, you 
have no available memory below 4GB because it is used for 
reserved-memory. So it makes sense for __alloc_bootmem_low to fail.

In your previous setup (i.e without reserved memory), all the memory was 
probably still above 4GB. In that case max_zone_dma_phys() will return 
an higher "lower" limit. So __alloc_bootmem_low is going to succeed.

AFAICT, this is probably a Linux bug. Now the question is whether 
__alloc_bootmem_low should be switch to alloc_bootmem or we need to fix 
the implementation of the function. Note that recent Linux (5.0 and 
onwards) have switch to memblock API, yet I think the issue is still here.

Fixing Linux is probably a good idea, but this means that upgrade of Xen 
may break Linux boot. In one hand, reserved-memory never worked with Xen 
(user have to manually disable it). So this is not a regression. On the 
other hand, a user will now need to upgrade their Linux (could be 
difficult if tie to a BSP kernel) or apply a fix (see above).

It might be possible to rework Dom0 memory allocation to limit the 
damage or even re-order the binary in memory. Amit, could you post the 
full Xen log with earlyprintk enabled?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel