Re: Dom0 kernel panic when porting xen to new arm soc

[Peng Fan <van.freenix@gmail.com>]

On 6/22/2015 10:02 PM, Julien Grall wrote:
> On 22/06/15 12:17, Peng Fan wrote:
>> I add debug log in this piece of code:
>> void __init sanity_check_meminfo(void)
>>
>> {
>>
>> phys_addr_t memblock_limit = 0;
>> int i, j, highmem = 0;
>> phys_addr_t vmalloc_limit = __pa(vmalloc_min - 1) + 1;
>>
>>
>> printk("vmalloc_min virt %x phys %x\n", vmalloc_min - 1,
>> __pa(vmalloc_min - 1));
>> printk("vmalloc_limit %x\n", vmalloc_limit);
>>
>> If use 512M for Dom0, I found vmalloc_limit is 0xf800000, vmalloc_min is
>> 0xef800000, This comes to a question, why __pa(vmalloc_min - 1) + will
>> make vmalloc_limit only 0xf800000 which is less than 128M. the pv stub
>> does some runtime fixup for virt_to_phys here.
>> Since vmalloc_limit is small, then all other memory bank in my platform
>> is recoginied as highmem, then arm_lowmem_limit is 0, then kernel panic:
>> "
>> dma_contiguous_reserve_area(size 1800000, base 00000000, limit 00000000)
>> CMA: failed to reserve 32 MiB
>> Memory policy: Data cache writealloc
>> Kernel panic - not syncing: ERROR: Failed to allocate 0x2000 bytes below
>> 0x0.
>> "
>> 0xffffffff - 0xef8000000 is about 264M. So I choose 256M as the Dom0
>> memory size. I do not have clear idea about this, current I am trying to
>> use xl to boot DomU, so just use 256M for Dom0 here.
> 
> There use to be some issue with the way how the offset between physical
> and virtual address was computed.
> 
> Do you use LPAE or short page table? Can you provide the xen log when
> DOM0 is using 512MB?

I did not enable LPAE for DOM0 kernel, use shor page table.
Following is the full log from uboot to kernel with DOM0 512M:

U-Boot 2015.04-rc4-00145-gf12a16e-dirty (Jun 19 2015 - 15:26:52)

CPU:   Freescale i.MX7D rev1.0 at 792 MHz
CPU:   Thermal invalid data, fuse: 0x1b800
CPU:   Temperature: Can't find sensor device
Reset cause: POR
Board: MX7D 12x12 LPDDR3 ARM2
I2C:   ready
DRAM:  2 GiB
PMIC: PFUZE300 DEV_ID=0x30 REV_ID=0x10
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
Display: MCIMX28LCD (800x480)
Video: 800x480x24
In:    serial
Out:   serial
Err:   serial
switch to partitions #0, OK
mmc0 is current device
Net:   FEC0
Error: FEC0 address not set.

Normal Boot
Hit any key to stop autoboot:  0
=> setenv xen_addr_r 0x80000000
=> setenv bootargs "console=dtuart
dtuart=/soc/aips-bus@30800000/serial@30860000 dom0_mem=256M"
=> fatload mmc 0:1 0x80000000 xen.image
reading xen.image
754544 bytes read in 129 ms (5.6 MiB/s)
=> setenv bootargs "console=dtuart
dtuart=/soc/aips-bus@30800000/serial@30860000 dom0_mem=512M"
=> run loadfdt
reading imx7d-12x12-lpddr3-arm2.dtb
40966 bytes read in 26 ms (1.5 MiB/s)
=> fatload mmc 0:1 0x84000000 zImage
reading zImage
6250016 bytes read in 286 ms (20.8 MiB/s)
=> setenv kernel_addr_r 0x84000000
=> setenv fdt_high 0xffffffff
=> setenv fdt_addr 0x83000000
=> fdt addr ${fdt_addr} 0x40000
=> fdt resize
=> fdt chosen
=> fdt set /chosen \#address-cells <1>
=> fdt set /chosen \#size-cells <1>
=> fdt set /chosen/module@0 compatible "xen,linux-zimage"
"xen,multiboot-module"
=> fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize}>
=> fdt set /chosen/module@0 bootargs "console=hvc0 ro
root=/dev/mmcblk0p2 clk_ignore_unused uart_from_osc loglevel=8 earlyprintk"
=> bootz ${xen_addr_r} - ${fdt_addr}
Kernel image @ 0x80000000 [ 0x000000 - 0x101700 ]
## Flattened Device Tree blob at 83000000
   Booting using the fdt blob at 0x83000000
   reserving fdt memory region: addr=83000000 size=b000
   Using Device Tree in place at 83000000, end 8300dfff

Starting kernel ...

- UART enabled -
- CPU 00000000 booting -
- Xen starting in Hyp mode -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000000080000000 - 00000000ffffffff
(XEN)
(XEN) MODULE[0]: 0000000083000000 - 000000008300b000 Device Tree
(XEN) MODULE[1]: 0000000084000000 - 00000000845f5e20 Kernel
console=hvc0 ro root=/dev/mmcblk0p2 clk_ignore_unused uart_from_osc
loglevel=8 earlyprintk
(XEN)  RESVD[0]: 0000000083000000 - 000000008300b000
(XEN)
(XEN) Command line: console=dtuart
dtuart=/soc/aips-bus@30800000/serial@30860000 dom0_mem=512M
(XEN) Placing Xen at 0x00000000ffe00000-0x0000000100000000
(XEN) Update BOOTMOD_XEN from 0000000080000000-0000000080101701 =>
00000000ffe00000-00000000fff01701
(XEN) Xen heap: 00000000fa000000-00000000fe000000 (16384 pages)
(XEN) Dom heap: 507904 pages
(XEN) Domain heap initialised
(XEN) Platform: i.MX 7Dual
(XEN) Looking for dtuart at "/soc/aips-bus@30800000/serial@30860000",
options ""
(XEN) imx-uart.c:226: xx 30860000
(XEN) imx-uart.c:98: ===============UFCR USR1 a01 2050
Xen 4.6-unstable
(XEN) Xen version 4.6-unstable (Freenix@(none))
(arm-poky-linux-gnueabi-gcc (GCC) 4.8.2) debug=y Fri Jun 19 18:30:50 CST
2015
(XEN) Latest ChangeSet: Mon Jun 15 18:25:34 2015 +0800 git:c01e139-dirty
(XEN) Processor: 410fc075: "ARM Limited", variant: 0x0, part 0xc07, rev 0x5
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00001131:00011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 02010555
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10101105 40000000 01240000 02102211
(XEN)  ISA Features: 02101110 13112111 21232041 11112131 10011142 00000000
(XEN) Set Secondary entry to 00000000ffe0004c (0020004c)
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 8000 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=0000000031001000
(XEN)         gic_cpu_addr=0000000031002000
(XEN)         gic_hyp_addr=0000000031004000
(XEN)         gic_vcpu_addr=0000000031006000
(XEN)         gic_maintenance_irq=25
(XEN) GICv2: 160 lines, 2 cpus, secure (IID 0100143b).
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) imx-uart.c:117: imx_uart_init_postirq
(XEN) Allocated console ring of 16 KiB.
(XEN) VFP implementer 0x41 architecture 2 part 0x30 variant 0x7 rev 0x5
(XEN) Brought up 1 CPUs
(XEN) P2M: 40-bit IPA
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80003558
(XEN) I/O virtualisation disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN) domain_build.c:1383: idle loop
(XEN) Loading kernel from boot module @ 0000000084000000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x000000a0000000-0x000000c0000000 (512MB)
(XEN) Loading zImage from 0000000084000000 to
00000000a7a00000-00000000a7ff5e20
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading dom0 DTB to 0x00000000a8000000-0x00000000a8009de2
(XEN) regs -> pc = 0xa7a00000
(XEN) it is 32 bit machine
(XEN) Scrubbing Free RAM on 1 nodes using 1 CPUs
(XEN) ................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 284kB init memory.
Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Linux version 3.14.38-02383-g5ccf32b-dirty (Freenix@linux-jyl1) (gcc
version 4.8.2 (GCC) ) #49 SMP PREEMPT Mon Jun 22 14:17:14 CST 2015
CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: Freescale i.MX7 LPDDR3 12x12 ARM2 Board
bootconsole [earlycon0] enabled
vmalloc_min virt ef7fffff phys f7fffff
dma_contiguous_reserve_area(size 1800000, base 00000000, limit 00000000)
CMA: failed to reserve 32 MiB
Memory policy: Data cache writealloc
Kernel panic - not syncing: ERROR: Failed to allocate 0x2000 bytes below
0x0.

> 
> Regards,
> 
Regards,
Peng.