Re: [Xen-devel] dom0less + sched=null => broken in staging

[Stefano Stabellini <sstabellini@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

On Fri, 9 Aug 2019, Dario Faggioli wrote:
> On Wed, 2019-08-07 at 11:22 -0700, Stefano Stabellini wrote:
> > Hi Dario, George,
> > 
> > Dom0less with sched=null is broken on staging, it simply hangs soon
> > after Xen is finished loading things. My impression is that vcpus are
> > not actually started. I did a git bisection and it pointed to:
> > 
> > commit d545f1d6c2519a183ed631cfca7aff0baf29fde5 (refs/bisect/bad)
> > Author: Dario Faggioli <dfaggioli@suse.com>
> > Date:   Mon Aug 5 11:50:55 2019 +0100
> > 
> >     xen: sched: deal with vCPUs being or becoming online or offline
> >     
> > Any ideas?
> > 
> Ok, I've done some basic testing, and inspected the code again, and
> honestly I am not finding anything really suspicious.
> 
> Of course, I'm not really testing dom0less, and I'm not sure I can
> easily do that.
> 
> Can you help me with this, e.g., by providing some more info and, if
> possible, logs?

I am attaching the logs. Interestingly, I get a bunch of:

(XEN) *** LOADING DOMU cpus=1 memory=40000KB ***
(XEN) sched_null.c:458: Not inserting d2v0 (not online!)

Maybe we are missing a call to online the vcpus somewhere in
xen/arch/arm/domain_build.c:construct_domain?


> E.g., you say boot stops after Xen loading. Is there a bootlog that we
> can see (ideally from a debug build, and with "loglvl=all
> guest_loglvl=all")?
> 
> Does the system respond to debug-keys? If yes, the log after triggering
> the 'r' debug-key would be useful.

The system doesn't respond to debug keys. My guess is that it is too
early maybe.


> These patches are about vcpus going offline and online... does dom0less
> play with vcpu onffline/online in any way?
> 
> I've put together a debug patch (attached), focusing on what the
> mentioned commit does, but it's nothing more than a shot in the dark,
> for now...

[-- Attachment #2: Type: text/plain, Size: 8014 bytes --]

- UART enabled -
- Boot CPU booting -
- Current EL 00000008 -
- Zero BSS -
- Initialize CPU -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000000000000000 - 000000007fefffff
(XEN) RAM: 0000000800000000 - 000000087fffffff
(XEN) 
(XEN) MODULE[0]: 0000000005e00000 - 0000000005e08000 Device Tree 
(XEN) MODULE[1]: 0000000005c00000 - 0000000005d83400 Ramdisk     
(XEN) MODULE[2]: 0000000004c00000 - 0000000005bdfa00 Kernel      
(XEN) MODULE[3]: 0000000004a00000 - 0000000004b83400 Ramdisk     
(XEN) MODULE[4]: 0000000003a00000 - 00000000049dfa00 Kernel      
(XEN) MODULE[5]: 0000000002400000 - 00000000039fa954 Ramdisk     
(XEN) MODULE[6]: 0000000001000000 - 00000000022f2200 Kernel      
(XEN)  RESVD[0]: 0000000005e00000 - 0000000005e08000
(XEN) 
(XEN) CMDLINE[0000000004c00000]:domU1 console=ttyAMA0
(XEN) CMDLINE[0000000003a00000]:domU0 console=ttyAMA0
(XEN) CMDLINE[0000000001000000]:chosen console=hvc0 earlycon=xen earlyprintk=xen root=/dev/ram0
(XEN) 
(XEN) Command line: console=dtuart dtuart=serial0 dom0_mem=700M dom0_max_vcpus=1 bootscrub=0 serrors=forward vwfi=native sched=null
(XEN) PFN compression on bits 19...22
(XEN) Domain heap initialised
(XEN) Booting using Device Tree
(XEN) Platform: Xilinx ZynqMP
(XEN) Looking for dtuart at "serial0", options ""
 Xen 4.13-unstable
(XEN) Xen version 4.13-unstable (sstabellini@) (aarch64-linux-gnu-gcc (Linaro GCC 5.3-2016.05) 5.3.1 20160412) debug=y  Fri Aug  9 11:25:18 PDT 2019
(XEN) Latest ChangeSet: Fri Aug 9 13:14:40 2019 +0100 git:762b9a2d99-dirty
(XEN) build-id: 23d86e8e8792dcc96038a90d8dab9698ddc3ed57
(XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part 0xd03, rev 0x4
(XEN) 64-bit Execution:
(XEN)   Processor Features: 1100000000002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 0000000010305106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001122 0000000000000000
(XEN)   ISA Features:  0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00001231:00011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10101105 40000000 01260000 02102211
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Using SMC Calling Convention v1.1
(XEN) Using PSCI v1.1
(XEN) SMP: Allowing 4 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 50000 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=00000000f9010000
(XEN)         gic_cpu_addr=00000000f9020000
(XEN)         gic_hyp_addr=00000000f9040000
(XEN)         gic_vcpu_addr=00000000f9060000
(XEN)         gic_maintenance_irq=25
(XEN) GICv2: Adjusting CPU interface base to 0xf902f000
(XEN) GICv2: 192 lines, 4 cpus (IID 00000000).
(XEN) XSM Framework v1.0.0 initialized
(XEN) Initialising XSM SILO mode
(XEN) WARNING: hypervisor-timer IRQ26 is not level triggered.
(XEN) WARNING: virtual-timer IRQ27 is not level triggered.
(XEN) WARNING: NS-physical-timer IRQ30 is not level triggered.
(XEN) Using scheduler: null Scheduler (null)
(XEN) Initializing null scheduler
(XEN) WARNING: This is experimental software in development.
(XEN) Use at your own risk.
(XEN) Allocated console ring of 32 KiB.
(XEN) CPU0: Guest atomics will try 1 times before pausing the domain
(XEN) Bringing up CPU1
- CPU 00000001 booting -
- Current EL 00000008 -
- Initialize CPU -
- Turning on paging -
- Ready -
(XEN) WARNING: hypervisor-timer IRQ26 is not level triggered.
(XEN) WARNING: virtual-timer IRQ27 is not level triggered.
(XEN) WARNING: NS-physical-timer IRQ30 is not level triggered.
(XEN) CPU1: Guest atomics will try 1 times before pausing the domain
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
- CPU 00000002 booting -
- Current EL 00000008 -
- Initialize CPU -
- Turning on paging -
- Ready -
(XEN) WARNING: hypervisor-timer IRQ26 is not level triggered.
(XEN) WARNING: virtual-timer IRQ27 is not level triggered.
(XEN) WARNING: NS-physical-timer IRQ30 is not level triggered.
(XEN) CPU2: Guest atomics will try 1 times before pausing the domain
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
- CPU 00000003 booting -
- Current EL 00000008 -
- Initialize CPU -
- Turning on paging -
- Ready -
(XEN) WARNING: hypervisor-timer IRQ26 is not level triggered.
(XEN) WARNING: virtual-timer IRQ27 is not level triggered.
(XEN) WARNING: NS-physical-timer IRQ30 is not level triggered.
(XEN) CPU3: Guest atomics will try 1 times before pausing the domain
(XEN) CPU 3 booted.
(XEN) Brought up 4 CPUs
(XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
(XEN) smmu: /amba/smmu@fd800000: probing hardware configuration...
(XEN) smmu: /amba/smmu@fd800000: SMMUv2 with:
(XEN) smmu: /amba/smmu@fd800000:        stage 2 translation
(XEN) smmu: /amba/smmu@fd800000:        stream matching with 48 register groups, mask 0x7fff
(XEN) smmu: /amba/smmu@fd800000:        16 context banks (0 stage-2 only)
(XEN) smmu: /amba/smmu@fd800000:        Stage-2: 40-bit IPA -> 48-bit PA
(XEN) smmu: /amba/smmu@fd800000: registered 26 master devices
/amba@0/smmu0@0xFD800000: Decode error: write to 6c=0
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) alternatives: Patching with alt table 00000000002bbe60 -> 00000000002bc520
(XEN) sched_null.c:458: Not inserting d0v0 (not online!)
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 0000000001000000
(XEN) Loading ramdisk from boot module @ 0000000002400000
(XEN) Allocating 1:1 mappings totalling 700MB for dom0:
(XEN) BANK[0] 0x00000020000000-0x00000040000000 (512MB)
(XEN) BANK[1] 0x00000070000000-0x00000078000000 (128MB)
(XEN) BANK[2] 0x0000007c000000-0x0000007fc00000 (60MB)
(XEN) Grant table range: 0x00000000e00000-0x00000000e40000
(XEN) smmu: /amba/smmu@fd800000: d0: p2maddr 0x000000087ffa2000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 0000000001000000 to 0000000020080000-0000000021372200
(XEN) Loading dom0 initrd from 0000000002400000 to 0x0000000028200000-0x00000000297fa954
(XEN) Loading dom0 DTB to 0x0000000028000000-0x0000000028006d75
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) *** LOADING DOMU cpus=1 memory=40000KB ***
(XEN) sched_null.c:458: Not inserting d1v0 (not online!)
(XEN) Loading d1 kernel from boot module @ 0000000004c00000
(XEN) Loading ramdisk from boot module @ 0000000005c00000
(XEN) Allocating mappings totalling 256MB for d1:
(XEN) d1 BANK[0] 0x00000040000000-0x00000050000000 (256MB)
(XEN) d1 BANK[1] 0x00000200000000-0x00000200000000 (0MB)
(XEN) Loading zImage from 0000000004c00000 to 0000000040080000-000000004105fa00
(XEN) Loading dom0 initrd from 0000000005c00000 to 0x0000000048200000-0x0000000048383400
(XEN) Loading dom0 DTB to 0x0000000048000000-0x00000000480004bd
(XEN) *** LOADING DOMU cpus=1 memory=40000KB ***
(XEN) sched_null.c:458: Not inserting d2v0 (not online!)
(XEN) Loading d2 kernel from boot module @ 0000000003a00000
(XEN) Loading ramdisk from boot module @ 0000000004a00000
(XEN) Allocating mappings totalling 256MB for d2:
(XEN) d2 BANK[0] 0x00000040000000-0x00000050000000 (256MB)
(XEN) d2 BANK[1] 0x00000200000000-0x00000200000000 (0MB)
(XEN) Loading zImage from 0000000003a00000 to 0000000040080000-000000004105fa00
(XEN) Loading dom0 initrd from 0000000004a00000 to 0x0000000048200000-0x0000000048383400
(XEN) Loading dom0 DTB to 0x0000000048000000-0x00000000480004bd


[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel