* static-mem preventing dom0 from booting
@ 2021-11-05 1:35 Stefano Stabellini
2021-11-05 3:23 ` Penny Zheng
0 siblings, 1 reply; 9+ messages in thread
From: Stefano Stabellini @ 2021-11-05 1:35 UTC (permalink / raw)
To: Penny.Zheng; +Cc: sstabellini, xen-devel
[-- Attachment #1: Type: text/plain, Size: 10884 bytes --]
Hi Penny,
I am trying to test the static-mem feature in Xen 4.16 but I am having
some issues. I can boot a dom0less domU with static-mem configured
correctly, but when I do so, dom0 doesn't boot any longer.
In the same configuration, if I remove the static-mem related lines from
the domU node on device tree, both dom0 and domU boot successfully.
I am sure the dom0 memory range, allocated by Xen automatically, and the
domU memory range, hand-picked by me, do not clash as you can see from
the boot logs (appended).
Am I missing anything in the configuration? I am using a ZCU102 board,
the memory node on the host device tree is:
memory {
device_type = "memory";
reg = <0x00 0x00 0x00 0x7ff00000 0x08 0x00 0x00 0x80000000>;
};
I am attaching my u-boot boot.scr script with the device tree
modification to add static memory.
Many thanks for your help!
Cheers,
Stefano
(XEN) Xen version 4.16-rc (sstabellini@) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0) debug=y Thu Nov 4 18:28:30 PDT 2021
(XEN) Latest ChangeSet: Mon Nov 1 12:36:26 2021 +0000 git:9f8434d778-dirty
(XEN) Processor: 00000000410fd034: "ARM Limited", variant: 0x0, part 0xd03,rev 0x4
(XEN) 64-bit Execution:
(XEN) Processor Features: 1100000000002222 0000000000000000
(XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN) Extensions: FloatingPoint AdvancedSIMD
(XEN) Debug Features: 0000000010305106 0000000000000000
(XEN) Auxiliary Features: 0000000000000000 0000000000000000
(XEN) Memory Model Features: 0000000000001122 0000000000000000
(XEN) ISA Features: 0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN) Processor Features: 0000000000001231:0000000000011011
(XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
(XEN) Extensions: GenericTimer Security
(XEN) Debug Features: 0000000003010066
(XEN) Auxiliary Features: 0000000000000000
(XEN) Memory Model Features: 0000000010101105 0000000040000000
(XEN) 0000000001260000 0000000002102211
(XEN) ISA Features: 0000000002101110 0000000013112111 0000000021232042
(XEN) 0000000001112131 0000000000011142 0000000000011121
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 65000 KHz
(XEN) GICv2 initialization:
(XEN) gic_dist_addr=00000000f9010000
(XEN) gic_cpu_addr=00000000f9020000
(XEN) gic_hyp_addr=00000000f9040000
(XEN) gic_vcpu_addr=00000000f9060000
(XEN) gic_maintenance_irq=25
(XEN) GICv2: Adjusting CPU interface base to 0xf902f000
(XEN) GICv2: 192 lines, 4 cpus, secure (IID 00000000).
(XEN) XSM Framework v1.0.1 initialized
(XEN) Initialising XSM SILO mode
(XEN) Using scheduler: null Scheduler (null)
(XEN) Initializing null scheduler
(XEN) WARNING: This is experimental software in development.
(XEN) Use at your own risk.
(XEN) Allocated console ring of 16 KiB.
(XEN) Bringing up CPU1
(XEN) Bringing up CPU2
(XEN) Bringing up CPU3
(XEN) Brought up 4 CPUs
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x0000000080023558
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 0000000000e00000
(XEN) Loading ramdisk from boot module @ 0000000002200000
(XEN) Allocating 1:1 mappings totalling 1024MB for dom0:
(XEN) BANK[0] 0x00000020000000-0x00000060000000 (1024MB)
(XEN) Grant table range: 0x00000005600000-0x00000005640000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Extended region 0: 0x80000000->0xc0000000
(XEN) Extended region 1: 0xc8000000->0xe0000000
(XEN) Extended region 2: 0xf0000000->0xf9000000
(XEN) Extended region 3: 0xffc00000->0x600000000
(XEN) Extended region 4: 0x880000000->0x8000000000
(XEN) Extended region 5: 0x8001000000->0x10000000000
(XEN) Loading zImage from 0000000000e00000 to 0000000020000000-0000000021367a00
(XEN) Loading d0 initrd from 0000000002200000 to 0x0000000028200000-0x0000000029d60ac0
(XEN) Loading d0 DTB to 0x0000000028000000-0x00000000280096d4
(XEN) *** LOADING DOMU cpus=1 memory=80000KB ***
(XEN) Loading d1 kernel from boot module @ 0000000003e00000
(XEN) Loading ramdisk from boot module @ 0000000005200000
(XEN) Loading zImage from 0000000003e00000 to 0000000040000000-0000000041367a00
(XEN) Loading d1 initrd from 0000000005200000 to 0x0000000048200000-0x0000000048383400
(XEN) Loading d1 DTB to 0x0000000048000000-0x00000000480007f3
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) DEBUG start_xen 1054 id=0
(XEN) null.c:353: 0 <-- d0v0
(XEN) DEBUG start_xen 1054 id=1
(XEN) null.c:353: 1 <-- d1v0
(XEN) Freed 348kB init memory.
(XEN) traps.c:2013:d0v0 HSR=0x00000092000045 pc=0xffffffc0105300ec gva=0xfffffffdfda3b000 gpa=0x0000081ffff000
(XEN) d1v0 Unhandled SMC/HVC: 0x84000050
(XEN) d1v0 Unhandled SMC/HVC: 0x8600ff01
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER20
(XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
(XEN) DOM1: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
(XEN) DOM1: [ 0.000000] Linux version 5.14.0-00002-g9c4bc20097e8 (sstabellini@ubuntu-linux-20-04-desktop) (gcc (Ubuntu 9.3.0-17ubuntu1~
(XEN) DOM1: 20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #9 SMP Wed Nov 3 16:50:00 PDT 2021
(XEN) DOM1: [ 0.000000] efi: UEFI not found.
(XEN) DOM1: [ 0.000000] Zone ranges:
(XEN) DOM1: [ 0.000000] DMA32 [mem 0x0000000040000000-0x000000005fffffff]
(XEN) DOM1: [ 0.000000] Normal empty
(XEN) DOM1: [ 0.000000] Movable zone start for each node
(XEN) DOM1: [ 0.000000] Early memory node ranges
(XEN) DOM1: [ 0.000000] node 0: [mem 0x0000000040000000-0x000000005fffffff]
(XEN) DOM1: [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000005fffffff]
(XEN) DOM1: [ 0.000000] cma: Reserved 256 MiB at 0x000000004f400000
(XEN) DOM1: [ 0.000000] psci: probing for conduit method from DT.
(XEN) DOM1: [ 0.000000] psci: PSCIv1.1 detected in firmware.
(XEN) DOM1: [ 0.000000] psci: Using standard PSCI v0.2 function IDs
(XEN) DOM1: [ 0.000000] psci: Trusted OS migration not required
(XEN) DOM1: [ 0.000000] psci: SMC Calling Convention v1.1
(XEN) DOM1: [ 0.000000] percpu: Embedded 22 pages/cpu s51672 r8192 d30248 u90112
(XEN) DOM1: [ 0.000000] Detected VIPT I-cache on CPU0
(XEN) DOM1: [ 0.000000] CPU features: detected: ARM erratum 843419
(XEN) DOM1: [ 0.000000] CPU features: detected: ARM erratum 845719
(XEN) DOM1: [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 129280
(XEN) DOM1: [ 0.000000] Kernel command line: console=ttyAMA0
(XEN) DOM1: [ 0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
(XEN) DOM1: [ 0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
(XEN) DOM1: [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
(XEN) DOM1: [ 0.000000] Memory: 230280K/524288K available (13184K kernel code, 862K rwdata, 3636K rodata, 2048K init, 315K bss, 31864K
(XEN) DOM1: reserved, 262144K cma-reserved)
(XEN) DOM1: [ 0.000000] rcu: Hierarchical RCU implementation.
(XEN) DOM1: [ 0.000000] rcu: RCU event tracing is enabled.
(XEN) DOM1: [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=1.
(XEN) DOM1: [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
(XEN) DOM1: [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
(XEN) DOM1: [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
(XEN) DOM1: [ 0.000000] Root IRQ handler: gic_handle_irq
(XEN) DOM1: [ 0.000000] random: get_random_bytes called from start_kernel+0x4a8/0x728 with crng_init=0
(XEN) DOM1: [ 0.000000] arch_timer: cp15 timer(s) running at 65.00MHz (virt).
(XEN) DOM1: [ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xefdb196da, max_idle_ns: 440795204367 ns
(XEN) DOM1: [ 0.000055] sched_clock: 56 bits at 65MHz, resolution 15ns, wraps every 2199023255550ns
(XEN) DOM1: [ 0.015323] Console: colour dummy device 80x25
(XEN) DOM1: [ 0.017224] Calibrating delay loop (skipped), value calculated using timer frequency.. 130.00 BogoMIPS (lpj=260000)
(XEN) DOM1: [ 0.017335] pid_max: default: 32768 minimum: 301
(XEN) DOM1: [ 0.019932] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
(XEN) DOM1: [ 0.020022] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
(XEN) DOM1: [ 0.070463] rcu: Hierarchical SRCU implementation.
(XEN) DOM1: [ 0.073611] EFI services will not be available.
(XEN) DOM1: [ 0.074889] smp: Bringing up secondary CPUs ...
(XEN) DOM1: [ 0.075178] smp: Brought up 1 node, 1 CPU
(XEN) DOM1: [ 0.075205] SMP: Total of 1 processors activated.
(XEN) DOM1: [ 0.075253] CPU features: detected: 32-bit EL0 Support
(XEN) DOM1: [ 0.075304] CPU features: detected: CRC32 instructions
(XEN) DOM1: [ 0.076596] CPU: All CPU(s) started at EL1
(XEN) DOM1: [ 0.076717] alternatives: patching kernel code
(XEN) DOM1: [ 0.092884] devtmpfs: initialized
(XEN) DOM1: [ 0.109553] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
(XEN) DOM1: [ 0.109716] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
(XEN) DOM1: [ 0.124058] pinctrl core: initialized pinctrl subsystem
(XEN) DOM1: [ 0.143628] NET: Registered PF_NETLINK/PF_ROUTE protocol family
(XEN) DOM1: [ 0.157764] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
(XEN) DOM1: [ 0.158089] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
(XEN) DOM1: [ 0.158426] audit: initializing netlink subsys (disabled)
(XEN) DOM1: [ 0.166578] cpuidle: using governor menu
(XEN) DOM1: [ 0.167213] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
(XEN) DOM1: [ 0.168930] audit: type=2000 audit(0.140:1): state=initialized audit_enabled=0 res=1
(XEN) DOM1: [ 0.169475] ASID allocator initialised with 65536 entries
(XEN) DOM1: [ 0.171250] Serial: AMBA PL011 UART driver
(XEN) DOM1: [ 0.181920] 22000000.sbsa-uart: ttyAMA0 at MMIO 0x22000000 (irq = 13, base_baud = 0) is a SBSA
(XEN) DOM1: [ 0.533061] printk: console [ttyAMA0] enabled
[... cutting here, DOM1 boots until the end, no signs of dom0]
[-- Attachment #2: Type: text/plain, Size: 2220 bytes --]
tftpb 0xE00000 4.16-2022.1/Image-domU
tftpb 0x2200000 4.16-2022.1/xen-rootfs.cpio.gz
tftpb 0x3E00000 4.16-2022.1/Image-domU
tftpb 0x5200000 4.16-2022.1/initrd.cpio
tftpb 0x5400000 4.16-2022.1/passthrough-example-part.dtb
tftpb 0x5600000 4.16-2022.1/xen
tftpb 0x5800000 4.16-2022.1/mpsoc.dtb
fdt addr 0x5800000
fdt resize 1024
fdt set /chosen \#address-cells <0x2>
fdt set /chosen \#size-cells <0x2>
fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 dom0_mem=1G dom0_max_vcpus=1 bootscrub=0 vwfi=native sched=null loglvl=9"
fdt mknod /chosen dom0
fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module" "multiboot,module"
fdt set /chosen/dom0 reg <0x0 0xE00000 0x0 0x1367a00>
fdt set /chosen xen,dom0-bootargs "console=hvc0 earlycon=xen earlyprintk=xen clk_ignore_unused root=/dev/ram0"
fdt mknod /chosen dom0-ramdisk
fdt set /chosen/dom0-ramdisk compatible "xen,linux-initrd" "xen,multiboot-module" "multiboot,module"
fdt set /chosen/dom0-ramdisk reg <0x0 0x2200000 0x0 0x1b60ac0>
fdt mknod /chosen domU0
fdt set /chosen/domU0 compatible "xen,domain"
fdt set /chosen/domU0 \#address-cells <0x2>
fdt set /chosen/domU0 \#size-cells <0x2>
fdt set /chosen/domU0 memory <0 0x80000>
fdt set /chosen/domU0 \#xen,static-mem-address-cells <0x2>
fdt set /chosen/domU0 \#xen,static-mem-size-cells <0x1>
fdt set /chosen/domU0 xen,static-mem <0x8 0x0 0x20000000>
fdt set /chosen/domU0 cpus <1>
fdt set /chosen/domU0 vpl011 <0x1>
fdt mknod /chosen/domU0 module0x3E00000
fdt set /chosen/domU0/module0x3E00000 compatible "multiboot,kernel" "multiboot,module"
fdt set /chosen/domU0/module0x3E00000 reg <0x0 0x3E00000 0x0 0x1367a00>
fdt set /chosen/domU0/module0x3E00000 bootargs "console=ttyAMA0"
fdt mknod /chosen/domU0 module0x5200000
fdt set /chosen/domU0/module0x5200000 compatible "multiboot,ramdisk" "multiboot,module"
fdt set /chosen/domU0/module0x5200000 reg <0x0 0x5200000 0x0 0x183400>
fdt mknod /chosen/domU0 module0x5400000
fdt set /chosen/domU0/module0x5400000 compatible "multiboot,device-tree" "multiboot,module"
fdt set /chosen/domU0/module0x5400000 reg <0x0 0x5400000 0x0 0x45b>
setenv fdt_high 0xffffffffffffffff
booti 0x5600000 - 0x5800000
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: static-mem preventing dom0 from booting
2021-11-05 1:35 static-mem preventing dom0 from booting Stefano Stabellini
@ 2021-11-05 3:23 ` Penny Zheng
2021-11-05 6:35 ` Penny Zheng
0 siblings, 1 reply; 9+ messages in thread
From: Penny Zheng @ 2021-11-05 3:23 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, Wei Chen
Hi Stefano
> -----Original Message-----
> From: Stefano Stabellini <sstabellini@kernel.org>
> Sent: Friday, November 5, 2021 9:36 AM
> To: Penny Zheng <Penny.Zheng@arm.com>
> Cc: sstabellini@kernel.org; xen-devel@lists.xenproject.org
> Subject: static-mem preventing dom0 from booting
>
> Hi Penny,
>
> I am trying to test the static-mem feature in Xen 4.16 but I am having some
> issues. I can boot a dom0less domU with static-mem configured correctly, but
> when I do so, dom0 doesn't boot any longer.
>
Hmm, In our first intention, dom0less is a mandatory option for static allocation.
domU on static allocation shall boot when dom0 doesn't boot there.
We think that, nevertheless, dom0 memory range is allocated by Xen automatically,
and it leads to the unpredictability. Static allocation through device tree configuration
prefers the total static environment to avoid unpredictability.
> In the same configuration, if I remove the static-mem related lines from the
> domU node on device tree, both dom0 and domU boot successfully.
>
> I am sure the dom0 memory range, allocated by Xen automatically, and the
> domU memory range, hand-picked by me, do not clash as you can see from the
> boot logs (appended).
>
> Am I missing anything in the configuration? I am using a ZCU102 board, the
> memory node on the host device tree is:
>
Are you suggesting one scenario where dom0 with static-mem domU?
Hmmm, one quick thought, it may not be working with vpl011 emulation.
When dom0 exists, vpl011 will take dom0 as backend, which requests event
channel, xen store, etc, involved to communicate. but static-mem domU
without CONFIG_XEN shall not handle it.
> memory {
> device_type = "memory";
> reg = <0x00 0x00 0x00 0x7ff00000 0x08 0x00 0x00 0x80000000>;
> };
>
> I am attaching my u-boot boot.scr script with the device tree modification to
> add static memory.
>
> Many thanks for your help!
>
> Cheers,
>
> Stefano
>
>
> (XEN) Xen version 4.16-rc (sstabellini@) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04)
> 9.3.0) debug=y Thu Nov 4 18:28:30 PDT 2021
> (XEN) Latest ChangeSet: Mon Nov 1 12:36:26 2021 +0000 git:9f8434d778-dirty
> (XEN) Processor: 00000000410fd034: "ARM Limited", variant: 0x0, part
> 0xd03,rev 0x4
> (XEN) 64-bit Execution:
> (XEN) Processor Features: 1100000000002222 0000000000000000
> (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
> (XEN) Extensions: FloatingPoint AdvancedSIMD
> (XEN) Debug Features: 0000000010305106 0000000000000000
> (XEN) Auxiliary Features: 0000000000000000 0000000000000000
> (XEN) Memory Model Features: 0000000000001122 0000000000000000
> (XEN) ISA Features: 0000000000011120 0000000000000000
> (XEN) 32-bit Execution:
> (XEN) Processor Features: 0000000000001231:0000000000011011
> (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
> (XEN) Extensions: GenericTimer Security
> (XEN) Debug Features: 0000000003010066
> (XEN) Auxiliary Features: 0000000000000000
> (XEN) Memory Model Features: 0000000010101105 0000000040000000
> (XEN) 0000000001260000 0000000002102211
> (XEN) ISA Features: 0000000002101110 0000000013112111
> 0000000021232042
> (XEN) 0000000001112131 0000000000011142 0000000000011121
> (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 65000 KHz
> (XEN) GICv2 initialization:
> (XEN) gic_dist_addr=00000000f9010000
> (XEN) gic_cpu_addr=00000000f9020000
> (XEN) gic_hyp_addr=00000000f9040000
> (XEN) gic_vcpu_addr=00000000f9060000
> (XEN) gic_maintenance_irq=25
> (XEN) GICv2: Adjusting CPU interface base to 0xf902f000
> (XEN) GICv2: 192 lines, 4 cpus, secure (IID 00000000).
> (XEN) XSM Framework v1.0.1 initialized
> (XEN) Initialising XSM SILO mode
> (XEN) Using scheduler: null Scheduler (null)
> (XEN) Initializing null scheduler
> (XEN) WARNING: This is experimental software in development.
> (XEN) Use at your own risk.
> (XEN) Allocated console ring of 16 KiB.
> (XEN) Bringing up CPU1
> (XEN) Bringing up CPU2
> (XEN) Bringing up CPU3
> (XEN) Brought up 4 CPUs
> (XEN) I/O virtualisation enabled
> (XEN) - Dom0 mode: Relaxed
> (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
> (XEN) P2M: 3 levels with order-1 root, VTCR 0x0000000080023558
> (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Loading d0 kernel from boot module @ 0000000000e00000
> (XEN) Loading ramdisk from boot module @ 0000000002200000
> (XEN) Allocating 1:1 mappings totalling 1024MB for dom0:
> (XEN) BANK[0] 0x00000020000000-0x00000060000000 (1024MB)
> (XEN) Grant table range: 0x00000005600000-0x00000005640000
> (XEN) Allocating PPI 16 for event channel interrupt
> (XEN) Extended region 0: 0x80000000->0xc0000000
> (XEN) Extended region 1: 0xc8000000->0xe0000000
> (XEN) Extended region 2: 0xf0000000->0xf9000000
> (XEN) Extended region 3: 0xffc00000->0x600000000
> (XEN) Extended region 4: 0x880000000->0x8000000000
> (XEN) Extended region 5: 0x8001000000->0x10000000000
> (XEN) Loading zImage from 0000000000e00000 to 0000000020000000-
> 0000000021367a00
> (XEN) Loading d0 initrd from 0000000002200000 to 0x0000000028200000-
> 0x0000000029d60ac0
> (XEN) Loading d0 DTB to 0x0000000028000000-0x00000000280096d4
> (XEN) *** LOADING DOMU cpus=1 memory=80000KB ***
> (XEN) Loading d1 kernel from boot module @ 0000000003e00000
> (XEN) Loading ramdisk from boot module @ 0000000005200000
> (XEN) Loading zImage from 0000000003e00000 to 0000000040000000-
> 0000000041367a00
> (XEN) Loading d1 initrd from 0000000005200000 to 0x0000000048200000-
> 0x0000000048383400
> (XEN) Loading d1 DTB to 0x0000000048000000-0x00000000480007f3
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Std. Loglevel: Errors and warnings
> (XEN) Guest Loglevel: All
> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> (XEN) DEBUG start_xen 1054 id=0
> (XEN) null.c:353: 0 <-- d0v0
> (XEN) DEBUG start_xen 1054 id=1
> (XEN) null.c:353: 1 <-- d1v0
> (XEN) Freed 348kB init memory.
> (XEN) traps.c:2013:d0v0 HSR=0x00000092000045 pc=0xffffffc0105300ec
> gva=0xfffffffdfda3b000 gpa=0x0000081ffff000
> (XEN) d1v0 Unhandled SMC/HVC: 0x84000050
> (XEN) d1v0 Unhandled SMC/HVC: 0x8600ff01
> (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4
> (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8
> (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12
> (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16
> (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER20
> (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
> (XEN) DOM1: [ 0.000000] Booting Linux on physical CPU 0x0000000000
> [0x410fd034]
> (XEN) DOM1: [ 0.000000] Linux version 5.14.0-00002-g9c4bc20097e8
> (sstabellini@ubuntu-linux-20-04-desktop) (gcc (Ubuntu 9.3.0-17ubuntu1~
> (XEN) DOM1: 20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #9 SMP Wed
> Nov 3 16:50:00 PDT 2021
> (XEN) DOM1: [ 0.000000] efi: UEFI not found.
> (XEN) DOM1: [ 0.000000] Zone ranges:
> (XEN) DOM1: [ 0.000000] DMA32 [mem 0x0000000040000000-
> 0x000000005fffffff]
> (XEN) DOM1: [ 0.000000] Normal empty
> (XEN) DOM1: [ 0.000000] Movable zone start for each node
> (XEN) DOM1: [ 0.000000] Early memory node ranges
> (XEN) DOM1: [ 0.000000] node 0: [mem 0x0000000040000000-
> 0x000000005fffffff]
> (XEN) DOM1: [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-
> 0x000000005fffffff]
> (XEN) DOM1: [ 0.000000] cma: Reserved 256 MiB at 0x000000004f400000
> (XEN) DOM1: [ 0.000000] psci: probing for conduit method from DT.
> (XEN) DOM1: [ 0.000000] psci: PSCIv1.1 detected in firmware.
> (XEN) DOM1: [ 0.000000] psci: Using standard PSCI v0.2 function IDs
> (XEN) DOM1: [ 0.000000] psci: Trusted OS migration not required
> (XEN) DOM1: [ 0.000000] psci: SMC Calling Convention v1.1
> (XEN) DOM1: [ 0.000000] percpu: Embedded 22 pages/cpu s51672 r8192
> d30248 u90112
> (XEN) DOM1: [ 0.000000] Detected VIPT I-cache on CPU0
> (XEN) DOM1: [ 0.000000] CPU features: detected: ARM erratum 843419
> (XEN) DOM1: [ 0.000000] CPU features: detected: ARM erratum 845719
> (XEN) DOM1: [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages:
> 129280
> (XEN) DOM1: [ 0.000000] Kernel command line: console=ttyAMA0
> (XEN) DOM1: [ 0.000000] Dentry cache hash table entries: 65536 (order: 7,
> 524288 bytes, linear)
> (XEN) DOM1: [ 0.000000] Inode-cache hash table entries: 32768 (order: 6,
> 262144 bytes, linear)
> (XEN) DOM1: [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap
> free:off
> (XEN) DOM1: [ 0.000000] Memory: 230280K/524288K available (13184K
> kernel code, 862K rwdata, 3636K rodata, 2048K init, 315K bss, 31864K
> (XEN) DOM1: reserved, 262144K cma-reserved)
> (XEN) DOM1: [ 0.000000] rcu: Hierarchical RCU implementation.
> (XEN) DOM1: [ 0.000000] rcu: RCU event tracing is enabled.
> (XEN) DOM1: [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=4 to
> nr_cpu_ids=1.
> (XEN) DOM1: [ 0.000000] rcu: RCU calculated value of scheduler-enlistment
> delay is 25 jiffies.
> (XEN) DOM1: [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16,
> nr_cpu_ids=1
> (XEN) DOM1: [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
> (XEN) DOM1: [ 0.000000] Root IRQ handler: gic_handle_irq
> (XEN) DOM1: [ 0.000000] random: get_random_bytes called from
> start_kernel+0x4a8/0x728 with crng_init=0
> (XEN) DOM1: [ 0.000000] arch_timer: cp15 timer(s) running at 65.00MHz (virt).
> (XEN) DOM1: [ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff
> max_cycles: 0xefdb196da, max_idle_ns: 440795204367 ns
> (XEN) DOM1: [ 0.000055] sched_clock: 56 bits at 65MHz, resolution 15ns,
> wraps every 2199023255550ns
> (XEN) DOM1: [ 0.015323] Console: colour dummy device 80x25
> (XEN) DOM1: [ 0.017224] Calibrating delay loop (skipped), value calculated
> using timer frequency.. 130.00 BogoMIPS (lpj=260000)
> (XEN) DOM1: [ 0.017335] pid_max: default: 32768 minimum: 301
> (XEN) DOM1: [ 0.019932] Mount-cache hash table entries: 1024 (order: 1,
> 8192 bytes, linear)
> (XEN) DOM1: [ 0.020022] Mountpoint-cache hash table entries: 1024 (order:
> 1, 8192 bytes, linear)
> (XEN) DOM1: [ 0.070463] rcu: Hierarchical SRCU implementation.
> (XEN) DOM1: [ 0.073611] EFI services will not be available.
> (XEN) DOM1: [ 0.074889] smp: Bringing up secondary CPUs ...
> (XEN) DOM1: [ 0.075178] smp: Brought up 1 node, 1 CPU
> (XEN) DOM1: [ 0.075205] SMP: Total of 1 processors activated.
> (XEN) DOM1: [ 0.075253] CPU features: detected: 32-bit EL0 Support
> (XEN) DOM1: [ 0.075304] CPU features: detected: CRC32 instructions
> (XEN) DOM1: [ 0.076596] CPU: All CPU(s) started at EL1
> (XEN) DOM1: [ 0.076717] alternatives: patching kernel code
> (XEN) DOM1: [ 0.092884] devtmpfs: initialized
> (XEN) DOM1: [ 0.109553] clocksource: jiffies: mask: 0xffffffff max_cycles:
> 0xffffffff, max_idle_ns: 7645041785100000 ns
> (XEN) DOM1: [ 0.109716] futex hash table entries: 256 (order: 2, 16384 bytes,
> linear)
> (XEN) DOM1: [ 0.124058] pinctrl core: initialized pinctrl subsystem
> (XEN) DOM1: [ 0.143628] NET: Registered PF_NETLINK/PF_ROUTE protocol
> family
> (XEN) DOM1: [ 0.157764] DMA: preallocated 128 KiB GFP_KERNEL pool for
> atomic allocations
> (XEN) DOM1: [ 0.158089] DMA: preallocated 128 KiB
> GFP_KERNEL|GFP_DMA32 pool for atomic allocations
> (XEN) DOM1: [ 0.158426] audit: initializing netlink subsys (disabled)
> (XEN) DOM1: [ 0.166578] cpuidle: using governor menu
> (XEN) DOM1: [ 0.167213] hw-breakpoint: found 6 breakpoint and 4
> watchpoint registers.
> (XEN) DOM1: [ 0.168930] audit: type=2000 audit(0.140:1): state=initialized
> audit_enabled=0 res=1
> (XEN) DOM1: [ 0.169475] ASID allocator initialised with 65536 entries
> (XEN) DOM1: [ 0.171250] Serial: AMBA PL011 UART driver
> (XEN) DOM1: [ 0.181920] 22000000.sbsa-uart: ttyAMA0 at MMIO 0x22000000
> (irq = 13, base_baud = 0) is a SBSA
> (XEN) DOM1: [ 0.533061] printk: console [ttyAMA0] enabled
>
Cheers，
Penny Zheng
> [... cutting here, DOM1 boots until the end, no signs of dom0]
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: static-mem preventing dom0 from booting
2021-11-05 3:23 ` Penny Zheng
@ 2021-11-05 6:35 ` Penny Zheng
2021-11-05 15:41 ` Stefano Stabellini
0 siblings, 1 reply; 9+ messages in thread
From: Penny Zheng @ 2021-11-05 6:35 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, Wei Chen
Hi stefano
> -----Original Message-----
> From: Penny Zheng
> Sent: Friday, November 5, 2021 11:23 AM
> To: Stefano Stabellini <sstabellini@kernel.org>
> Cc: xen-devel@lists.xenproject.org; Wei Chen <Wei.Chen@arm.com>
> Subject: RE: static-mem preventing dom0 from booting
>
> Hi Stefano
>
> > -----Original Message-----
> > From: Stefano Stabellini <sstabellini@kernel.org>
> > Sent: Friday, November 5, 2021 9:36 AM
> > To: Penny Zheng <Penny.Zheng@arm.com>
> > Cc: sstabellini@kernel.org; xen-devel@lists.xenproject.org
> > Subject: static-mem preventing dom0 from booting
> >
> > Hi Penny,
> >
> > I am trying to test the static-mem feature in Xen 4.16 but I am having
> > some issues. I can boot a dom0less domU with static-mem configured
> > correctly, but when I do so, dom0 doesn't boot any longer.
> >
>
> Hmm, In our first intention, dom0less is a mandatory option for static
> allocation.
> domU on static allocation shall boot when dom0 doesn't boot there.
>
> We think that, nevertheless, dom0 memory range is allocated by Xen
> automatically, and it leads to the unpredictability. Static allocation through
> device tree configuration prefers the total static environment to avoid
> unpredictability.
>
> > In the same configuration, if I remove the static-mem related lines
> > from the domU node on device tree, both dom0 and domU boot successfully.
> >
> > I am sure the dom0 memory range, allocated by Xen automatically, and
> > the domU memory range, hand-picked by me, do not clash as you can see
> > from the boot logs (appended).
> >
> > Am I missing anything in the configuration? I am using a ZCU102 board,
> > the memory node on the host device tree is:
> >
>
> Are you suggesting one scenario where dom0 with static-mem domU?
>
> Hmmm, one quick thought, it may not be working with vpl011 emulation.
> When dom0 exists, vpl011 will take dom0 as backend, which requests event
> channel, xen store, etc, involved to communicate. but static-mem domU
> without CONFIG_XEN shall not handle it.
>
Digging on this, and my misunderstanding. Domain created through device tree
shall use xen as backend, not dom0. I guess that only the one created by xl is using
dom0 as backend.
Nevertheless, if this is your scenario, where dom0 and static-mem domU(NO CONFIG_XEN) co-exists,
I'll test for you.
> > memory {
> > device_type = "memory";
> > reg = <0x00 0x00 0x00 0x7ff00000 0x08 0x00 0x00 0x80000000>;
> > };
> >
> > I am attaching my u-boot boot.scr script with the device tree
> > modification to add static memory.
> >
> > Many thanks for your help!
> >
> > Cheers,
> >
> > Stefano
> >
> >
> > (XEN) Xen version 4.16-rc (sstabellini@) (gcc (Ubuntu
> > 9.3.0-17ubuntu1~20.04)
> > 9.3.0) debug=y Thu Nov 4 18:28:30 PDT 2021
> > (XEN) Latest ChangeSet: Mon Nov 1 12:36:26 2021 +0000
> > git:9f8434d778-dirty
> > (XEN) Processor: 00000000410fd034: "ARM Limited", variant: 0x0, part
> > 0xd03,rev 0x4
> > (XEN) 64-bit Execution:
> > (XEN) Processor Features: 1100000000002222 0000000000000000
> > (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
> > (XEN) Extensions: FloatingPoint AdvancedSIMD
> > (XEN) Debug Features: 0000000010305106 0000000000000000
> > (XEN) Auxiliary Features: 0000000000000000 0000000000000000
> > (XEN) Memory Model Features: 0000000000001122 0000000000000000
> > (XEN) ISA Features: 0000000000011120 0000000000000000
> > (XEN) 32-bit Execution:
> > (XEN) Processor Features: 0000000000001231:0000000000011011
> > (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
> > (XEN) Extensions: GenericTimer Security
> > (XEN) Debug Features: 0000000003010066
> > (XEN) Auxiliary Features: 0000000000000000
> > (XEN) Memory Model Features: 0000000010101105 0000000040000000
> > (XEN) 0000000001260000 0000000002102211
> > (XEN) ISA Features: 0000000002101110 0000000013112111
> > 0000000021232042
> > (XEN) 0000000001112131 0000000000011142 0000000000011121
> > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 65000 KHz
> > (XEN) GICv2 initialization:
> > (XEN) gic_dist_addr=00000000f9010000
> > (XEN) gic_cpu_addr=00000000f9020000
> > (XEN) gic_hyp_addr=00000000f9040000
> > (XEN) gic_vcpu_addr=00000000f9060000
> > (XEN) gic_maintenance_irq=25
> > (XEN) GICv2: Adjusting CPU interface base to 0xf902f000
> > (XEN) GICv2: 192 lines, 4 cpus, secure (IID 00000000).
> > (XEN) XSM Framework v1.0.1 initialized
> > (XEN) Initialising XSM SILO mode
> > (XEN) Using scheduler: null Scheduler (null)
> > (XEN) Initializing null scheduler
> > (XEN) WARNING: This is experimental software in development.
> > (XEN) Use at your own risk.
> > (XEN) Allocated console ring of 16 KiB.
> > (XEN) Bringing up CPU1
> > (XEN) Bringing up CPU2
> > (XEN) Bringing up CPU3
> > (XEN) Brought up 4 CPUs
> > (XEN) I/O virtualisation enabled
> > (XEN) - Dom0 mode: Relaxed
> > (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
> > (XEN) P2M: 3 levels with order-1 root, VTCR 0x0000000080023558
> > (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
> > (XEN) *** LOADING DOMAIN 0 ***
> > (XEN) Loading d0 kernel from boot module @ 0000000000e00000
> > (XEN) Loading ramdisk from boot module @ 0000000002200000
> > (XEN) Allocating 1:1 mappings totalling 1024MB for dom0:
> > (XEN) BANK[0] 0x00000020000000-0x00000060000000 (1024MB)
> > (XEN) Grant table range: 0x00000005600000-0x00000005640000
> > (XEN) Allocating PPI 16 for event channel interrupt
> > (XEN) Extended region 0: 0x80000000->0xc0000000
> > (XEN) Extended region 1: 0xc8000000->0xe0000000
> > (XEN) Extended region 2: 0xf0000000->0xf9000000
> > (XEN) Extended region 3: 0xffc00000->0x600000000
> > (XEN) Extended region 4: 0x880000000->0x8000000000
> > (XEN) Extended region 5: 0x8001000000->0x10000000000
> > (XEN) Loading zImage from 0000000000e00000 to 0000000020000000-
> > 0000000021367a00
> > (XEN) Loading d0 initrd from 0000000002200000 to 0x0000000028200000-
> > 0x0000000029d60ac0
> > (XEN) Loading d0 DTB to 0x0000000028000000-0x00000000280096d4
> > (XEN) *** LOADING DOMU cpus=1 memory=80000KB ***
> > (XEN) Loading d1 kernel from boot module @ 0000000003e00000
> > (XEN) Loading ramdisk from boot module @ 0000000005200000
> > (XEN) Loading zImage from 0000000003e00000 to 0000000040000000-
> > 0000000041367a00
> > (XEN) Loading d1 initrd from 0000000005200000 to 0x0000000048200000-
> > 0x0000000048383400
> > (XEN) Loading d1 DTB to 0x0000000048000000-0x00000000480007f3
> > (XEN) Initial low memory virq threshold set at 0x4000 pages.
> > (XEN) Std. Loglevel: Errors and warnings
> > (XEN) Guest Loglevel: All
> > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch
> > input)
> > (XEN) DEBUG start_xen 1054 id=0
> > (XEN) null.c:353: 0 <-- d0v0
> > (XEN) DEBUG start_xen 1054 id=1
> > (XEN) null.c:353: 1 <-- d1v0
> > (XEN) Freed 348kB init memory.
> > (XEN) traps.c:2013:d0v0 HSR=0x00000092000045 pc=0xffffffc0105300ec
> > gva=0xfffffffdfda3b000 gpa=0x0000081ffff000
> > (XEN) d1v0 Unhandled SMC/HVC: 0x84000050
> > (XEN) d1v0 Unhandled SMC/HVC: 0x8600ff01
> > (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4
> > (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8
> > (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to
> > ICACTIVER12
> > (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to
> > ICACTIVER16
> > (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to
> > ICACTIVER20
> > (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
> > (XEN) DOM1: [ 0.000000] Booting Linux on physical CPU 0x0000000000
> > [0x410fd034]
> > (XEN) DOM1: [ 0.000000] Linux version 5.14.0-00002-g9c4bc20097e8
> > (sstabellini@ubuntu-linux-20-04-desktop) (gcc (Ubuntu 9.3.0-17ubuntu1~
> > (XEN) DOM1: 20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #9
> > SMP Wed Nov 3 16:50:00 PDT 2021
> > (XEN) DOM1: [ 0.000000] efi: UEFI not found.
> > (XEN) DOM1: [ 0.000000] Zone ranges:
> > (XEN) DOM1: [ 0.000000] DMA32 [mem 0x0000000040000000-
> > 0x000000005fffffff]
> > (XEN) DOM1: [ 0.000000] Normal empty
> > (XEN) DOM1: [ 0.000000] Movable zone start for each node
> > (XEN) DOM1: [ 0.000000] Early memory node ranges
> > (XEN) DOM1: [ 0.000000] node 0: [mem 0x0000000040000000-
> > 0x000000005fffffff]
> > (XEN) DOM1: [ 0.000000] Initmem setup node 0 [mem
> 0x0000000040000000-
> > 0x000000005fffffff]
> > (XEN) DOM1: [ 0.000000] cma: Reserved 256 MiB at 0x000000004f400000
> > (XEN) DOM1: [ 0.000000] psci: probing for conduit method from DT.
> > (XEN) DOM1: [ 0.000000] psci: PSCIv1.1 detected in firmware.
> > (XEN) DOM1: [ 0.000000] psci: Using standard PSCI v0.2 function IDs
> > (XEN) DOM1: [ 0.000000] psci: Trusted OS migration not required
> > (XEN) DOM1: [ 0.000000] psci: SMC Calling Convention v1.1
> > (XEN) DOM1: [ 0.000000] percpu: Embedded 22 pages/cpu s51672 r8192
> > d30248 u90112
> > (XEN) DOM1: [ 0.000000] Detected VIPT I-cache on CPU0
> > (XEN) DOM1: [ 0.000000] CPU features: detected: ARM erratum 843419
> > (XEN) DOM1: [ 0.000000] CPU features: detected: ARM erratum 845719
> > (XEN) DOM1: [ 0.000000] Built 1 zonelists, mobility grouping on. Total
> pages:
> > 129280
> > (XEN) DOM1: [ 0.000000] Kernel command line: console=ttyAMA0
> > (XEN) DOM1: [ 0.000000] Dentry cache hash table entries: 65536 (order: 7,
> > 524288 bytes, linear)
> > (XEN) DOM1: [ 0.000000] Inode-cache hash table entries: 32768 (order: 6,
> > 262144 bytes, linear)
> > (XEN) DOM1: [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap
> > free:off
> > (XEN) DOM1: [ 0.000000] Memory: 230280K/524288K available (13184K
> > kernel code, 862K rwdata, 3636K rodata, 2048K init, 315K bss, 31864K
> > (XEN) DOM1: reserved, 262144K cma-reserved)
> > (XEN) DOM1: [ 0.000000] rcu: Hierarchical RCU implementation.
> > (XEN) DOM1: [ 0.000000] rcu: RCU event tracing is enabled.
> > (XEN) DOM1: [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=4 to
> > nr_cpu_ids=1.
> > (XEN) DOM1: [ 0.000000] rcu: RCU calculated value of scheduler-enlistment
> > delay is 25 jiffies.
> > (XEN) DOM1: [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16,
> > nr_cpu_ids=1
> > (XEN) DOM1: [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
> > (XEN) DOM1: [ 0.000000] Root IRQ handler: gic_handle_irq
> > (XEN) DOM1: [ 0.000000] random: get_random_bytes called from
> > start_kernel+0x4a8/0x728 with crng_init=0
> > (XEN) DOM1: [ 0.000000] arch_timer: cp15 timer(s) running at 65.00MHz
> (virt).
> > (XEN) DOM1: [ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff
> > max_cycles: 0xefdb196da, max_idle_ns: 440795204367 ns
> > (XEN) DOM1: [ 0.000055] sched_clock: 56 bits at 65MHz, resolution 15ns,
> > wraps every 2199023255550ns
> > (XEN) DOM1: [ 0.015323] Console: colour dummy device 80x25
> > (XEN) DOM1: [ 0.017224] Calibrating delay loop (skipped), value calculated
> > using timer frequency.. 130.00 BogoMIPS (lpj=260000)
> > (XEN) DOM1: [ 0.017335] pid_max: default: 32768 minimum: 301
> > (XEN) DOM1: [ 0.019932] Mount-cache hash table entries: 1024 (order: 1,
> > 8192 bytes, linear)
> > (XEN) DOM1: [ 0.020022] Mountpoint-cache hash table entries: 1024 (order:
> > 1, 8192 bytes, linear)
> > (XEN) DOM1: [ 0.070463] rcu: Hierarchical SRCU implementation.
> > (XEN) DOM1: [ 0.073611] EFI services will not be available.
> > (XEN) DOM1: [ 0.074889] smp: Bringing up secondary CPUs ...
> > (XEN) DOM1: [ 0.075178] smp: Brought up 1 node, 1 CPU
> > (XEN) DOM1: [ 0.075205] SMP: Total of 1 processors activated.
> > (XEN) DOM1: [ 0.075253] CPU features: detected: 32-bit EL0 Support
> > (XEN) DOM1: [ 0.075304] CPU features: detected: CRC32 instructions
> > (XEN) DOM1: [ 0.076596] CPU: All CPU(s) started at EL1
> > (XEN) DOM1: [ 0.076717] alternatives: patching kernel code
> > (XEN) DOM1: [ 0.092884] devtmpfs: initialized
> > (XEN) DOM1: [ 0.109553] clocksource: jiffies: mask: 0xffffffff max_cycles:
> > 0xffffffff, max_idle_ns: 7645041785100000 ns
> > (XEN) DOM1: [ 0.109716] futex hash table entries: 256 (order: 2, 16384
> bytes,
> > linear)
> > (XEN) DOM1: [ 0.124058] pinctrl core: initialized pinctrl subsystem
> > (XEN) DOM1: [ 0.143628] NET: Registered PF_NETLINK/PF_ROUTE protocol
> > family
> > (XEN) DOM1: [ 0.157764] DMA: preallocated 128 KiB GFP_KERNEL pool for
> > atomic allocations
> > (XEN) DOM1: [ 0.158089] DMA: preallocated 128 KiB
> > GFP_KERNEL|GFP_DMA32 pool for atomic allocations
> > (XEN) DOM1: [ 0.158426] audit: initializing netlink subsys (disabled)
> > (XEN) DOM1: [ 0.166578] cpuidle: using governor menu
> > (XEN) DOM1: [ 0.167213] hw-breakpoint: found 6 breakpoint and 4
> > watchpoint registers.
> > (XEN) DOM1: [ 0.168930] audit: type=2000 audit(0.140:1): state=initialized
> > audit_enabled=0 res=1
> > (XEN) DOM1: [ 0.169475] ASID allocator initialised with 65536 entries
> > (XEN) DOM1: [ 0.171250] Serial: AMBA PL011 UART driver
> > (XEN) DOM1: [ 0.181920] 22000000.sbsa-uart: ttyAMA0 at MMIO
> 0x22000000
> > (irq = 13, base_baud = 0) is a SBSA
> > (XEN) DOM1: [ 0.533061] printk: console [ttyAMA0] enabled
> >
>
> Cheers，
>
> Penny Zheng
> > [... cutting here, DOM1 boots until the end, no signs of dom0]
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: static-mem preventing dom0 from booting
2021-11-05 6:35 ` Penny Zheng
@ 2021-11-05 15:41 ` Stefano Stabellini
2021-11-05 23:03 ` Stefano Stabellini
0 siblings, 1 reply; 9+ messages in thread
From: Stefano Stabellini @ 2021-11-05 15:41 UTC (permalink / raw)
To: Penny Zheng; +Cc: Stefano Stabellini, xen-devel, Wei Chen
On Fri, 5 Nov 2021, Penny Zheng wrote:
> > -----Original Message-----
> > From: Penny Zheng
> > Sent: Friday, November 5, 2021 11:23 AM
> > To: Stefano Stabellini <sstabellini@kernel.org>
> > Cc: xen-devel@lists.xenproject.org; Wei Chen <Wei.Chen@arm.com>
> > Subject: RE: static-mem preventing dom0 from booting
> >
> > Hi Stefano
> >
> > > -----Original Message-----
> > > From: Stefano Stabellini <sstabellini@kernel.org>
> > > Sent: Friday, November 5, 2021 9:36 AM
> > > To: Penny Zheng <Penny.Zheng@arm.com>
> > > Cc: sstabellini@kernel.org; xen-devel@lists.xenproject.org
> > > Subject: static-mem preventing dom0 from booting
> > >
> > > Hi Penny,
> > >
> > > I am trying to test the static-mem feature in Xen 4.16 but I am having
> > > some issues. I can boot a dom0less domU with static-mem configured
> > > correctly, but when I do so, dom0 doesn't boot any longer.
> > >
> >
> > Hmm, In our first intention, dom0less is a mandatory option for static
> > allocation.
> > domU on static allocation shall boot when dom0 doesn't boot there.
> >
> > We think that, nevertheless, dom0 memory range is allocated by Xen
> > automatically, and it leads to the unpredictability. Static allocation through
> > device tree configuration prefers the total static environment to avoid
> > unpredictability.
> >
> > > In the same configuration, if I remove the static-mem related lines
> > > from the domU node on device tree, both dom0 and domU boot successfully.
> > >
> > > I am sure the dom0 memory range, allocated by Xen automatically, and
> > > the domU memory range, hand-picked by me, do not clash as you can see
> > > from the boot logs (appended).
> > >
> > > Am I missing anything in the configuration? I am using a ZCU102 board,
> > > the memory node on the host device tree is:
> > >
> >
> > Are you suggesting one scenario where dom0 with static-mem domU?
> >
> > Hmmm, one quick thought, it may not be working with vpl011 emulation.
> > When dom0 exists, vpl011 will take dom0 as backend, which requests event
> > channel, xen store, etc, involved to communicate. but static-mem domU
> > without CONFIG_XEN shall not handle it.
> >
>
> Digging on this, and my misunderstanding. Domain created through device tree
> shall use xen as backend, not dom0. I guess that only the one created by xl is using
> dom0 as backend.
>
> Nevertheless, if this is your scenario, where dom0 and static-mem domU(NO CONFIG_XEN) co-exists,
> I'll test for you.
The scenario is extremely simple; you can see the full device tree
configuration in the attachment to my previous email.
- dom0
- dom0less domU with static-mem
That's it! So basically it is just a normal dom0 + dom0less domU
configuration, which already works fine, where I added static-mem to the
domU and suddenly dom0 (not the domU!) stopped working.
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: static-mem preventing dom0 from booting
2021-11-05 15:41 ` Stefano Stabellini
@ 2021-11-05 23:03 ` Stefano Stabellini
2021-11-05 23:05 ` Stefano Stabellini
2021-11-08 6:32 ` Penny Zheng
0 siblings, 2 replies; 9+ messages in thread
From: Stefano Stabellini @ 2021-11-05 23:03 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Penny Zheng, xen-devel, Wei Chen, Bertrand.Marquis
On Fri, 5 Nov 2021, Stefano Stabellini wrote:
> The scenario is extremely simple; you can see the full device tree
> configuration in the attachment to my previous email.
>
> - dom0
> - dom0less domU with static-mem
>
> That's it! So basically it is just a normal dom0 + dom0less domU
> configuration, which already works fine, where I added static-mem to the
> domU and suddenly dom0 (not the domU!) stopped working.
I did some more debugging today and I found the problem. The issue is
that static-mem regions are added to the list of reserved_mem. However,
reserved_mem is automatically assigned to Dom0 by default at the bottom
of xen/arch/arm/domain_build.c:handle_node, see the second call to
make_memory_node. Really, we shouldn't give to dom0 static-mem ranges
meant for other domUs. E.g. the following change is sufficient to solve
the problem:
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 88a79247cb..dc609c4f0e 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -891,6 +891,9 @@ static int __init make_memory_node(const struct domain *d,
u64 start = mem->bank[i].start;
u64 size = mem->bank[i].size;
+ if ( mem->bank[i].xen_domain )
+ continue;
+
dt_dprintk(" Bank %d: %#"PRIx64"->%#"PRIx64"\n",
i, start, start + size);
However, maybe a better fix would be to purge reserved_mem of any
xen_domain items before calling make_memory_node.
I found one additional issue regarding is_domain_direct_mapped which
doesn't return true for static-mem domains. I think we need to add a
direct_map bool to arch_domain and set it for both dom0 and static-mem
dom0less domUs, so that we can change the implementation of
is_domain_direct_mapped to:
#define is_domain_direct_mapped(d) (d->arch.direct_map)
^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: static-mem preventing dom0 from booting
2021-11-05 23:03 ` Stefano Stabellini
@ 2021-11-05 23:05 ` Stefano Stabellini
2021-11-06 13:25 ` Julien Grall
2021-11-08 6:32 ` Penny Zheng
1 sibling, 1 reply; 9+ messages in thread
From: Stefano Stabellini @ 2021-11-05 23:05 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Penny Zheng, xen-devel, Wei Chen, julien, Bertrand.Marquis,
Volodymyr_Babchuk, iwj
+ Julien, Ian and others
On Fri, 5 Nov 2021, Stefano Stabellini wrote:
> On Fri, 5 Nov 2021, Stefano Stabellini wrote:
> > The scenario is extremely simple; you can see the full device tree
> > configuration in the attachment to my previous email.
> >
> > - dom0
> > - dom0less domU with static-mem
> >
> > That's it! So basically it is just a normal dom0 + dom0less domU
> > configuration, which already works fine, where I added static-mem to the
> > domU and suddenly dom0 (not the domU!) stopped working.
>
> I did some more debugging today and I found the problem. The issue is
> that static-mem regions are added to the list of reserved_mem. However,
> reserved_mem is automatically assigned to Dom0 by default at the bottom
> of xen/arch/arm/domain_build.c:handle_node, see the second call to
> make_memory_node. Really, we shouldn't give to dom0 static-mem ranges
> meant for other domUs. E.g. the following change is sufficient to solve
> the problem:
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 88a79247cb..dc609c4f0e 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -891,6 +891,9 @@ static int __init make_memory_node(const struct domain *d,
> u64 start = mem->bank[i].start;
> u64 size = mem->bank[i].size;
>
> + if ( mem->bank[i].xen_domain )
> + continue;
> +
> dt_dprintk(" Bank %d: %#"PRIx64"->%#"PRIx64"\n",
> i, start, start + size);
>
> However, maybe a better fix would be to purge reserved_mem of any
> xen_domain items before calling make_memory_node.
>
>
> I found one additional issue regarding is_domain_direct_mapped which
> doesn't return true for static-mem domains. I think we need to add a
> direct_map bool to arch_domain and set it for both dom0 and static-mem
> dom0less domUs, so that we can change the implementation of
> is_domain_direct_mapped to:
>
> #define is_domain_direct_mapped(d) (d->arch.direct_map)
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: static-mem preventing dom0 from booting
2021-11-05 23:05 ` Stefano Stabellini
@ 2021-11-06 13:25 ` Julien Grall
2021-11-09 0:37 ` Stefano Stabellini
0 siblings, 1 reply; 9+ messages in thread
From: Julien Grall @ 2021-11-06 13:25 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Penny Zheng, xen-devel, Wei Chen, Bertrand.Marquis,
Volodymyr_Babchuk, iwj
Hi Stefano,
On 05/11/2021 23:05, Stefano Stabellini wrote:
> On Fri, 5 Nov 2021, Stefano Stabellini wrote:
>> On Fri, 5 Nov 2021, Stefano Stabellini wrote:
>>> The scenario is extremely simple; you can see the full device tree
>>> configuration in the attachment to my previous email.
>>>
>>> - dom0
>>> - dom0less domU with static-mem
>>>
>>> That's it! So basically it is just a normal dom0 + dom0less domU
>>> configuration, which already works fine, where I added static-mem to the
>>> domU and suddenly dom0 (not the domU!) stopped working.
>>
>> I did some more debugging today and I found the problem. The issue is
>> that static-mem regions are added to the list of reserved_mem. However,
>> reserved_mem is automatically assigned to Dom0 by default at the bottom
>> of xen/arch/arm/domain_build.c:handle_node, see the second call to
>> make_memory_node. Really, we shouldn't give to dom0 static-mem ranges
>> meant for other domUs. E.g. the following change is sufficient to solve
>> the problem:
>>
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index 88a79247cb..dc609c4f0e 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -891,6 +891,9 @@ static int __init make_memory_node(const struct domain *d,
>> u64 start = mem->bank[i].start;
>> u64 size = mem->bank[i].size;
>>
>> + if ( mem->bank[i].xen_domain )
>> + continue;
>> +
>> dt_dprintk(" Bank %d: %#"PRIx64"->%#"PRIx64"\n",
>> i, start, start + size);
>>
>> However, maybe a better fix would be to purge reserved_mem of any
>> xen_domain items before calling make_memory_node.
I would rather not modify boot_info.reserved_mem because it may be used
afterwards. I think your approach is the right one.
Alternatively, we would rework make_memory_node() to create one node per
range (rather than a node with multiple ranges). This would move the
loop outside of make_memory_node(). The advantage is we have more
flexibily how on to filter ranges (in the future we may need to pass
some reserved ranges to a domain).
>>
>>
>> I found one additional issue regarding is_domain_direct_mapped which
>> doesn't return true for static-mem domains. I think we need to add a
>> direct_map bool to arch_domain and set it for both dom0 and static-mem
>> dom0less domUs, so that we can change the implementation of
>> is_domain_direct_mapped to:
>>
>> #define is_domain_direct_mapped(d) (d->arch.direct_map)
In Xen 4.16, static-mem domains are not direct mapped (i.e MFN == GFN).
Instead, the static memory is used to allocate memory for the domain at
the default regions in the guest memory layout.
If you want to direct map static-mem domains, then you would have to
apply [1] from Penny which is still under review.
Cheers,
[1] <20211015030945.2082898-1-penny.zheng@arm.com>
--
Julien Grall
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: static-mem preventing dom0 from booting
2021-11-05 23:03 ` Stefano Stabellini
2021-11-05 23:05 ` Stefano Stabellini
@ 2021-11-08 6:32 ` Penny Zheng
1 sibling, 0 replies; 9+ messages in thread
From: Penny Zheng @ 2021-11-08 6:32 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, Wei Chen, Bertrand Marquis, Julien Grall
Hi Stefano
> -----Original Message-----
> From: Stefano Stabellini <sstabellini@kernel.org>
> Sent: Saturday, November 6, 2021 7:03 AM
> To: Stefano Stabellini <sstabellini@kernel.org>
> Cc: Penny Zheng <Penny.Zheng@arm.com>; xen-devel@lists.xenproject.org;
> Wei Chen <Wei.Chen@arm.com>; Bertrand Marquis
> <Bertrand.Marquis@arm.com>
> Subject: RE: static-mem preventing dom0 from booting
>
> On Fri, 5 Nov 2021, Stefano Stabellini wrote:
> > The scenario is extremely simple; you can see the full device tree
> > configuration in the attachment to my previous email.
> >
> > - dom0
> > - dom0less domU with static-mem
> >
> > That's it! So basically it is just a normal dom0 + dom0less domU
> > configuration, which already works fine, where I added static-mem to
> > the domU and suddenly dom0 (not the domU!) stopped working.
>
Got it. Sorry, I missed the scenario you are talking about... I simply think what dom0less
means that dom0 is absent... ;/
> I did some more debugging today and I found the problem. The issue is that
> static-mem regions are added to the list of reserved_mem. However,
> reserved_mem is automatically assigned to Dom0 by default at the bottom of
> xen/arch/arm/domain_build.c:handle_node, see the second call to
> make_memory_node. Really, we shouldn't give to dom0 static-mem ranges
> meant for other domUs. E.g. the following change is sufficient to solve the
> problem:
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 88a79247cb..dc609c4f0e 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -891,6 +891,9 @@ static int __init make_memory_node(const struct
> domain *d,
> u64 start = mem->bank[i].start;
> u64 size = mem->bank[i].size;
>
> + if ( mem->bank[i].xen_domain )
> + continue;
> +
> dt_dprintk(" Bank %d: %#"PRIx64"->%#"PRIx64"\n",
> i, start, start + size);
>
> However, maybe a better fix would be to purge reserved_mem of any
> xen_domain items before calling make_memory_node.
>
>
> I found one additional issue regarding is_domain_direct_mapped which
> doesn't return true for static-mem domains. I think we need to add a
> direct_map bool to arch_domain and set it for both dom0 and static-mem
> dom0less domUs, so that we can change the implementation of
> is_domain_direct_mapped to:
>
> #define is_domain_direct_mapped(d) (d->arch.direct_map)
Yeah, I already pushed a patch serie regarding direct-map to community for review,
and it is also based on your old direct-map serie.
Today, I may push direct-map version 3 to community for review~~~ If you're free, plz take a look.
Cheers,
Penny Zheng
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: static-mem preventing dom0 from booting
2021-11-06 13:25 ` Julien Grall
@ 2021-11-09 0:37 ` Stefano Stabellini
0 siblings, 0 replies; 9+ messages in thread
From: Stefano Stabellini @ 2021-11-09 0:37 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, Penny Zheng, xen-devel, Wei Chen,
Bertrand.Marquis, Volodymyr_Babchuk, iwj
On Sat, 6 Nov 2021, Julien Grall wrote:
> Hi Stefano,
>
> On 05/11/2021 23:05, Stefano Stabellini wrote:
> > On Fri, 5 Nov 2021, Stefano Stabellini wrote:
> > > On Fri, 5 Nov 2021, Stefano Stabellini wrote:
> > > > The scenario is extremely simple; you can see the full device tree
> > > > configuration in the attachment to my previous email.
> > > >
> > > > - dom0
> > > > - dom0less domU with static-mem
> > > >
> > > > That's it! So basically it is just a normal dom0 + dom0less domU
> > > > configuration, which already works fine, where I added static-mem to the
> > > > domU and suddenly dom0 (not the domU!) stopped working.
> > >
> > > I did some more debugging today and I found the problem. The issue is
> > > that static-mem regions are added to the list of reserved_mem. However,
> > > reserved_mem is automatically assigned to Dom0 by default at the bottom
> > > of xen/arch/arm/domain_build.c:handle_node, see the second call to
> > > make_memory_node. Really, we shouldn't give to dom0 static-mem ranges
> > > meant for other domUs. E.g. the following change is sufficient to solve
> > > the problem:
> > >
> > > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > > index 88a79247cb..dc609c4f0e 100644
> > > --- a/xen/arch/arm/domain_build.c
> > > +++ b/xen/arch/arm/domain_build.c
> > > @@ -891,6 +891,9 @@ static int __init make_memory_node(const struct domain
> > > *d,
> > > u64 start = mem->bank[i].start;
> > > u64 size = mem->bank[i].size;
> > > + if ( mem->bank[i].xen_domain )
> > > + continue;
> > > +
> > > dt_dprintk(" Bank %d: %#"PRIx64"->%#"PRIx64"\n",
> > > i, start, start + size);
> > > However, maybe a better fix would be to purge reserved_mem of any
> > > xen_domain items before calling make_memory_node.
>
> I would rather not modify boot_info.reserved_mem because it may be used
> afterwards. I think your approach is the right one.
>
> Alternatively, we would rework make_memory_node() to create one node per range
> (rather than a node with multiple ranges). This would move the loop outside of
> make_memory_node(). The advantage is we have more flexibily how on to filter
> ranges (in the future we may need to pass some reserved ranges to a domain).
Thanks for the quick feedback, I'll send a proper patch. I'll follow the
first approach for now.
> > >
> > > I found one additional issue regarding is_domain_direct_mapped which
> > > doesn't return true for static-mem domains. I think we need to add a
> > > direct_map bool to arch_domain and set it for both dom0 and static-mem
> > > dom0less domUs, so that we can change the implementation of
> > > is_domain_direct_mapped to:
> > >
> > > #define is_domain_direct_mapped(d) (d->arch.direct_map)
>
> In Xen 4.16, static-mem domains are not direct mapped (i.e MFN == GFN).
> Instead, the static memory is used to allocate memory for the domain at the
> default regions in the guest memory layout.
I see, I forgot that the memory is not already mapped 1:1.
> If you want to direct map static-mem domains, then you would have to apply [1]
> from Penny which is still under review.
>
> Cheers,
>
> [1] <20211015030945.2082898-1-penny.zheng@arm.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-11-09 0:38 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-05 1:35 static-mem preventing dom0 from booting Stefano Stabellini
2021-11-05 3:23 ` Penny Zheng
2021-11-05 6:35 ` Penny Zheng
2021-11-05 15:41 ` Stefano Stabellini
2021-11-05 23:03 ` Stefano Stabellini
2021-11-05 23:05 ` Stefano Stabellini
2021-11-06 13:25 ` Julien Grall
2021-11-09 0:37 ` Stefano Stabellini
2021-11-08 6:32 ` Penny Zheng
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.