Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM

[Roman Shaposhnik <roman@zededa.com>]

[-- Attachment #1: Type: text/plain, Size: 6741 bytes --]

On Tue, Dec 17, 2019 at 11:26 AM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > On Tue, Dec 17, 2019 at 10:30 AM Stefano Stabellini
> > <sstabellini@kernel.org> wrote:
> > >
> > > On Tue, 17 Dec 2019, Julien Grall wrote:
> > > > Hi,
> > > >
> > > > On 17/12/2019 04:39, Roman Shaposhnik wrote:
> > > > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini
> > > > > <sstabellini@kernel.org> wrote:
> > > > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote:
> > > > > > If I sum all the memory sizes together I get 0x3ddfd000 which is 990M.
> > > > > > If so, I wonder how you could boot succesfully with dom0_mem=1024M even
> > > > > > on Xen 4.12... :-?
> > > > >
> > > > > That is a very interesting observation indeed! I actually don't
> > > > > remember where that device tree came from, but I think it was from one
> > > > > of the Linaro sites.
> > > >
> > > > This is mostly likely because of:
> > > >
> > > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e
> > > > Author: Julien Grall <julien.grall@arm.com>
> > > > Date:   Wed Aug 21 22:42:31 2019 +0100
> > > >
> > > >     xen/arm: domain_build: Don't continue if unable to allocate all dom0 banks
> > > >
> > > >     Xen will only print a warning if there are memory unallocated when using
> > > >     1:1 mapping (only used by dom0). This also includes the case where no
> > > >     memory has been allocated.
> > > >
> > > >     It will bring to all sort of issues that can be hard to diagnostic for
> > > >     users (the warning can be difficult to spot or disregard).
> > > >
> > > >     If the users request 1GB of memory, then most likely they want the exact
> > > >     amount and not 512MB. So panic if all the memory has not been allocated.
> > > >
> > > >     After this change, the behavior is the same as for non-1:1 memory
> > > >     allocation (used by domU).
> > > >
> > > >     At the same time, reflow the message to have the format on a single
> > > >     line.
> > > >
> > > >     Signed-off-by: Julien Grall <julien.grall@arm.com>
> > > >     Acked-by: Stefano Stabellini <sstabellini@kernel.org>
> > >
> > > Ah! Roman, could you please post the full boot log of a successful 4.12
> > > boot?
> > >
> > > If it has a "Failed to allocate requested dom0 memory" message, then we
> > > know what the issue is.
> >
> > Aha! Our messages seems to have crossed ;-) Full log is attached and
> > yes -- that's
> > the problem indeed.
> >
> > So at least that mystery is solved. But I'm still not able to get to a
> > full 1G of memory
> > even with your update to the device tree file. Any chance you can send me the
> > device tree file that works for you?
>
> I didn't try on real hardware, I only tried on QEMU with a similar
> configuration. I went back and check the HiKey device tree I used and it
> is the same as yours (including the ramoops reserved-memory error).
>
> Apparently there are 1G and 2G variants of the HiKey, obviously both
> yours and my device tree are for the 1G variant. I try to dig through
> the docs but couldn't find the details of the 2G variant. I cannot find
> anywhere the memory range for the top 1G of memory not even on the
> LeMaker docs! :-/

Yup. That's exactly the issue on my end as well - can't seem to find an
authoritative source for that devicetree.

I did find this, though:
     https://releases.linaro.org/96boards/hikey/linaro/debian/15.11/
which looks like it has the latest (at least file timestamp-wise) devicetree.

If you look at the memory and reserved memory nodes there, they
are actually much simpler than what we've got:

        memory {
                device_type = "memory";
                reg = <0x0 0x0 0x0 0x40000000>;
        };

        reserved-memory {
                #address-cells = <0x2>;
                #size-cells = <0x2>;
                ranges;

                mcu-buf@05e00000 {
                        no-map;
                        reg = <0x0 0x5e00000 0x0 0x100000 0x0
0x740f000 0x0 0x1000>;
                };

                mbox-buf@06dff000 {
                        no-map;
                        reg = <0x0 0x6dff000 0x0 0x1000>;
                };
        };

So -- just on a whim -- I changed it to:
    reg = <0x0 0x0 0x0 0x80000000>;

Interestingly enough, Xen booted, and complained about only 192MB
unallocated this time.
So, I dropped the size of Dom0 to 640M and I got it boot and here's
what I'm seeing as
an output of xl info:
   total_memory           : 1120
   free_memory            : 390
It still nowhere close to 2G.

Then I booted the Linux kernel without Xen and it correctly identified
all 2G worth of RAM, and in fact,
when I converted /sys/firmware/devicetree/base back into dts, here's
what I've got:

        memory {
                device_type = "memory";
                reg = <0x0 0x0 0x0 0x5e00000 0x0 0x5f00000 0x0 0x1000
0x0 0x5f02000 0x0 0xefd000 0x0 0x6e00000 0x0 0x60f000 0x0 0x7410000
0x0 0x1aaf0000 0x0 0x21f00000 0x0 0x100000 0x0 0x22000000 0x0
0x1c000000>;
        };

        reserved-memory {
                ranges;
                #size-cells = <0x2>;
                #address-cells = <0x2>;

                ramoops@21f00000 {
                        ftrace-size = <0x20000>;
                        console-size = <0x20000>;
                        reg = <0x0 0x21f00000 0x0 0x100000>;
                        record-size = <0x20000>;
                        compatible = "ramoops";
                };

                linux,cma {
                        linux,cma-default;
                        reusable;
                        size = <0x0 0x8000000>;
                        compatible = "shared-dma-pool";
                };
        };

If you look at the REG -- it does now add up to 2Gb, but booting Xen
with it has exactly the
same effect as booting it with: reg = <0x0 0x0 0x0 0x80000000>;

I am attaching a full log, and I see the following in the logs:

(XEN) Allocating 1:1 mappings totalling 720MB for dom0:
(XEN) BANK[0] 0x00000008000000-0x0000001c000000 (320MB)
(XEN) BANK[1] 0x00000040000000-0x00000058000000 (384MB)
(XEN) BANK[2] 0x0000007b000000-0x0000007c000000 (16MB)

Which sort of makes sense, I guess -- but I still don't understand
where all these ranges
are coming from and how come Xen doesn't see the full 2Gb even with various
devicetrees I tried.

Any ideas here would be greatly apprecaited!

Thanks,
Roman.

P.S. Any guess at what these mean?

(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffff87355558
gva=0xffff872f2000 gpa=0x000000000f0000
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffffb734e558
gva=0xffffb72eb000 gpa=0x000000000f0000
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffff8f9d2558
gva=0xffff8f96f000 gpa=0x000000000f0000

[-- Attachment #2: xen4.log --]
[-- Type: application/octet-stream, Size: 6358 bytes --]

(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000000000000000 - 0000000005dfffff
(XEN) RAM: 0000000005f00000 - 0000000006dfefff
(XEN) RAM: 0000000006e00000 - 000000000740efff
(XEN) RAM: 0000000007410000 - 000000001db8dfff
(XEN) RAM: 00000000350f0000 - 000000003dbd2fff
(XEN) RAM: 000000003dbd3000 - 000000003dffffff
(XEN) RAM: 0000000040000000 - 000000005a653fff
(XEN) RAM: 000000007ada0000 - 000000007ada3fff
(XEN) RAM: 000000007aea8000 - 000000007afa9fff
(XEN) RAM: 000000007afaa000 - 000000007ec73fff
(XEN) RAM: 000000007ec74000 - 000000007fdddfff
(XEN) RAM: 000000007fdde000 - 000000007fea5fff
(XEN) RAM: 000000007fea6000 - 000000007ff6dfff
(XEN) RAM: 000000007ffff000 - 000000007fffffff
(XEN)
(XEN) MODULE[0]: 000000005a660000 - 000000005a79b900 Xen
(XEN) MODULE[1]: 000000005a654000 - 000000005a660000 Device Tree
(XEN) MODULE[2]: 000000005a7a7000 - 000000005b95a200 Kernel
(XEN)  RESVD[0]: 0000000021f00000 - 0000000021ffffff
(XEN)
(XEN) CMDLINE[000000005a7a7000]:chosen console=hvc0 root=PARTUUID=f71bd987-d99a-4c88-9781-cf4c26cae55e rootdelay=3
(XEN)
(XEN) Command line: console=dtuart dom0_mem=720M dom0_max_vcpus=1 dom0_vcpus_pin
(XEN) Domain heap initialised
(XEN) Booting using Device Tree
(XEN) Platform: Generic System
(XEN) Taking dtuart configuration from /chosen/stdout-path
(XEN) Looking for dtuart at "serial3", options "115200n8"
(XEN) WARNING: UART configuration is not supported
 Xen 4.13.0-rc
(XEN) Xen version 4.13.0-rc (@) (gcc (Alpine 6.4.0) 6.4.0) debug=y  Fri Dec  6 07:29:10 UTC 2019
(XEN) Latest ChangeSet:
(XEN) build-id: bf2e5dde8df8199230a1b0a25fb8f220be3714a0
(XEN) Processor: 410fd033: "ARM Limited", variant: 0x0, part 0xd03, rev 0x3
(XEN) 64-bit Execution:
(XEN)   Processor Features: 0000000000002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 0000000010305106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001122 0000000000000000
(XEN)   ISA Features:  0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00000131:00011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10101105 40000000 01260000 02102211
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Using SMC Calling Convention v1.0
(XEN) Using PSCI v1.0
(XEN) SMP: Allowing 8 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 1200 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=00000000f6801000
(XEN)         gic_cpu_addr=00000000f6802000
(XEN)         gic_hyp_addr=00000000f6804000
(XEN)         gic_vcpu_addr=00000000f6806000
(XEN)         gic_maintenance_irq=25
(XEN) GICv2: 160 lines, 8 cpus, secure (IID 0200143b).
(XEN) XSM Framework v1.0.0 initialized
(XEN) Initialising XSM SILO mode
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Allocated console ring of 64 KiB.
(XEN) CPU0: Guest atomics will try 8 times before pausing the domain
(XEN) Bringing up CPU1
(XEN) CPU1: Guest atomics will try 17 times before pausing the domain
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
(XEN) CPU2: Guest atomics will try 10 times before pausing the domain
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
(XEN) CPU3: Guest atomics will try 22 times before pausing the domain
(XEN) CPU 3 booted.
(XEN) Bringing up CPU4
(XEN) CPU4: Guest atomics will try 22 times before pausing the domain
(XEN) CPU 4 booted.
(XEN) Bringing up CPU5
(XEN) CPU5: Guest atomics will try 13 times before pausing the domain
(XEN) CPU 5 booted.
(XEN) Bringing up CPU6
(XEN) CPU6: Guest atomics will try 20 times before pausing the domain
(XEN) CPU 6 booted.
(XEN) Bringing up CPU7
(XEN) CPU7: Guest atomics will try 12 times before pausing the domain
(XEN) Brought up 8 CPUs
(XEN) CPU 7 booted.
(XEN) I/O virtualisation disabled
(XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) Adding cpu 4 to runqueue 0
(XEN) Adding cpu 5 to runqueue 0
(XEN) Adding cpu 6 to runqueue 0
(XEN) Adding cpu 7 to runqueue 0
(XEN) alternatives: Patching with alt table 00000000002cc068 -> 00000000002cc77c
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 000000005a7a7000
(XEN) Allocating 1:1 mappings totalling 720MB for dom0:
(XEN) BANK[0] 0x00000008000000-0x0000001c000000 (320MB)
(XEN) BANK[1] 0x00000040000000-0x00000058000000 (384MB)
(XEN) BANK[2] 0x0000007b000000-0x0000007c000000 (16MB)
(XEN) Grant table range: 0x0000005a660000-0x0000005a6a0000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 000000005a7a7000 to 0000000008080000-0000000009233200
(XEN) Loading d0 DTB to 0x0000000010000000-0x0000000010008b69
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 344kB init memory.
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
(XEN) d0v0 Unhandled SMC/HVC: 0xbf00ff01
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffff87355558 gva=0xffff872f2000 gpa=0x000000000f0000
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffffb734e558 gva=0xffffb72eb000 gpa=0x000000000f0000
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffff8f9d2558 gva=0xffff8f96f000 gpa=0x000000000f0000

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel