Stuck trying to boot Xen 4.3 on Arm Midway

Stuck trying to boot Xen 4.3 on Arm Midway

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 2923 bytes --]

I am trying to extract and combine the various pieces of information found in
[1] and its sub-pages and the Xen in-tree documentation in order to make xen
boot (potentially non-smp without some later changes). But since I am not
familiar enough with Arm I think I am stuck doing something wrong.

I compiled the hypervisor with debug and early printk for midway and use the
xen.bin file (I could get no output at all when trying to create a uboot image
with mkimage from the uncompressed xen.gz).
My uboot sequence looks like this:

mw.l 800000 0 10000
scsi scan
ext2load scsi 0 0x800000 xen.bin
ext2load scsi 0 0x1000000 vmlinuz
setenv kernsize $filesize
ext2load scsi 0 0x2000000 initrd.img
setenv initsize $filesize
# Tried dtuart=/soc/serial@fff36000 as well without
setenv bootargs "sync_console console=dtuart dtuart=serial"
fdt addr 0x1000
fdt resize
fdt set /chosen bootargs \"$bootargs\"
fdt mknod /chosen modules
# Tried with <1> and <2> for both as I was not sure wnether those numbers
# are related to number of modules
fdt set /chosen/modules \#address-cells <1>
fdt set /chosen/modules \#size-cells <1>
fdt mknod /chosen/modules module@0
fdt set /chosen/modules/module@0 compatible xen,linux-zimage xen,multiboot-module
fdt set /chosen/modules/module@0 reg <0x1000000 $kernsize>
fdt set /chosen/modules/module@0 bootargs "console=hvc0 debug"
fdt mknod /chosen/modules module@1
fdt set /chosen/modules/module@1 compatible "xen,linux-initrd"
"xen,multiboot-module"
fdt set /chosen/modules/module@1 reg <0x2000000 $initsize>
bootz 0x800000 - 0x1000

The memory locations are somewhat random (the one for the xen.img is used for
the kernel on normal installs). The boot produces the following:

## Flattened Device Tree blob at 00001000
   Booting using the fdt blob at 0x00001000
   reserving fdt memory region: addr=0 size=1000
   reserving fdt memory region: addr=1000 size=2000
   Using Device Tree in place at 00001000, end 00005fff

Starting kernel ...

- UART enabled -
- CPU 00000000 booting -
- Machine ID 00000000 -
- Started in Hyp mode -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
RAM: 0000000000000000 - 00000000ff7fffff
RAM: 0000000200000000 - 00000002ffffffff

MODULE[1]: 0000000001000000 - 0000000001471ae0 console=hvc0 debug
MODULE[2]: 0000000002000000 - 000000000223f08b
Placing Xen at 0x00000002ffe00000-0x0000000300000000
WARNING: Only using first bank of memory
Xen heap: 262144 pages  Dom heap: 784384 pages

After that nothing. Maybe I am doing the bootargs wrong. I tried
xen,xen-bootargs and xen,dom0-bootargs and combinations without success. Maybe
the console argument is wrong. Although the full dtb path at least shows up as
that in a booted linux in /proc. What am I doing wrong here?

Thanks,
Stefan


[1] http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Stuck trying to boot Xen 4.3 on Arm Midway

[Ian Campbell]

[-- Attachment #1: Type: text/plain, Size: 3721 bytes --]

On Mon, 2013-12-02 at 15:59 +0100, Stefan Bader wrote:
> I am trying to extract and combine the various pieces of information found in
> [1] and its sub-pages and the Xen in-tree documentation in order to make xen
> boot (potentially non-smp without some later changes). But since I am not
> familiar enough with Arm I think I am stuck doing something wrong.
> 
> I compiled the hypervisor with debug and early printk for midway and use the
> xen.bin file (I could get no output at all when trying to create a uboot image
> with mkimage from the uncompressed xen.gz).

What version are you using? xen.bin went away in July, the right file is
now just "xen".

> My uboot sequence looks like this:
> 
> mw.l 800000 0 10000
> scsi scan
> ext2load scsi 0 0x800000 xen.bin
> ext2load scsi 0 0x1000000 vmlinuz
> setenv kernsize $filesize
> ext2load scsi 0 0x2000000 initrd.img
> setenv initsize $filesize
> # Tried dtuart=/soc/serial@fff36000 as well without
> setenv bootargs "sync_console console=dtuart dtuart=serial"
> fdt addr 0x1000
> fdt resize
> fdt set /chosen bootargs \"$bootargs\"
> fdt mknod /chosen modules
> # Tried with <1> and <2> for both as I was not sure wnether those numbers
> # are related to number of modules

No, they are the number of u32s which are used for the address and size
in the reg field, which contains address then size.

So <2> for both would need "<0x0 0x100000 0x0 $kernsize>" or something.

> fdt set /chosen/modules \#address-cells <1>
> fdt set /chosen/modules \#size-cells <1>
> fdt mknod /chosen/modules module@0
> fdt set /chosen/modules/module@0 compatible xen,linux-zimage xen,multiboot-module
> fdt set /chosen/modules/module@0 reg <0x1000000 $kernsize>
> fdt set /chosen/modules/module@0 bootargs "console=hvc0 debug"
> fdt mknod /chosen/modules module@1
> fdt set /chosen/modules/module@1 compatible "xen,linux-initrd"
> "xen,multiboot-module"
> fdt set /chosen/modules/module@1 reg <0x2000000 $initsize>
> bootz 0x800000 - 0x1000
> 
> The memory locations are somewhat random (the one for the xen.img is used for
> the kernel on normal installs). The boot produces the following:
> 
> ## Flattened Device Tree blob at 00001000
>    Booting using the fdt blob at 0x00001000
>    reserving fdt memory region: addr=0 size=1000
>    reserving fdt memory region: addr=1000 size=2000
>    Using Device Tree in place at 00001000, end 00005fff
> 
> Starting kernel ...
> 
> - UART enabled -
> - CPU 00000000 booting -
> - Machine ID 00000000 -
> - Started in Hyp mode -
> - Zero BSS -
> - Setting up control registers -
> - Turning on paging -
> - Ready -
> RAM: 0000000000000000 - 00000000ff7fffff
> RAM: 0000000200000000 - 00000002ffffffff
> 
> MODULE[1]: 0000000001000000 - 0000000001471ae0 console=hvc0 debug
> MODULE[2]: 0000000002000000 - 000000000223f08b
> Placing Xen at 0x00000002ffe00000-0x0000000300000000
> WARNING: Only using first bank of memory
> Xen heap: 262144 pages  Dom heap: 784384 pages
> 
> After that nothing. Maybe I am doing the bootargs wrong. I tried
> xen,xen-bootargs and xen,dom0-bootargs and combinations without success. Maybe
> the console argument is wrong. Although the full dtb path at least shows up as
> that in a booted linux in /proc. What am I doing wrong here?

Your dtuart appears to be wrong, this is the point at which it would
normally be expected to start with the proper (rather than early)
console stuff.

The correct thing to use depends a bit on which version you are running,
but if it is not recent xen.git master or staging then that's your first
mistake ;-)

dtuart=/soc/serial@fff36000 is what I use. I've also attached my u-boot
script. (my local scripts append -arm32 to the binary name, it is the
"xen" file though)

Ian.

[-- Attachment #2: boot.xen --]
[-- Type: text/plain, Size: 1038 bytes --]

mw.l 800000 0 10000

fdt addr ${fdt_addr}
fdt resize

fdt set /soc/ethernet@fff50000 dma-coherent
fdt set /soc/ethernet@fff51000 dma-coherent

#fdt rm /cpus/cpu@1
#fdt rm /cpus/cpu@2
#fdt rm /cpus/cpu@3

# Make some trouble for allocator...
#fdt rsvmem add 0x2ffe00000 0x1000
#fdt rsvmem add 0xde00000 0x1000

setenv xen_addr_r 0x01000000
#  kernel_addr_r=0x02000000
# ramdisk_addr_r=0x04000000

tftp ${xen_addr_r} /marilith-n0/xen-arm32
setenv bootargs "console=dtuart dtuart=/soc/serial@fff36000 dom0_mem=1G noreboot conswitch=x loglvl=all guest_loglvl_all"

fdt set /chosen/bootargs "${bootargs}"

fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>

tftp ${kernel_addr_r} /marilith-n0/vmlinuz-arm
fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
fdt set /chosen/module@0 bootargs "console=hvc0 ro root=/dev/sda1 earlyprintk"

fdt print /chosen

printenv bootargs

bootz ${xen_addr_r} - ${fdt_addr}

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Stuck trying to boot Xen 4.3 on Arm Midway

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 4873 bytes --]

On 02.12.2013 16:24, Ian Campbell wrote:
> On Mon, 2013-12-02 at 15:59 +0100, Stefan Bader wrote:
>> I am trying to extract and combine the various pieces of information found in
>> [1] and its sub-pages and the Xen in-tree documentation in order to make xen
>> boot (potentially non-smp without some later changes). But since I am not
>> familiar enough with Arm I think I am stuck doing something wrong.
>>
>> I compiled the hypervisor with debug and early printk for midway and use the
>> xen.bin file (I could get no output at all when trying to create a uboot image
>> with mkimage from the uncompressed xen.gz).
> 
> What version are you using? xen.bin went away in July, the right file is
> now just "xen".

The released version of xen 4.3. At some point I might update to 4.3.1 (or
whatever stable is current then). This probably implies picking some additional
patches when I want to get it running on Midway. The xen.bin would not be
packaged right now but it had the right format to be used by bootz while xen.gz
(or xen after unpacking) would not be usable in uboot without converting and
that need some more address values which I could and likely do get wrong.

> 
>> My uboot sequence looks like this:
>>
>> mw.l 800000 0 10000
>> scsi scan
>> ext2load scsi 0 0x800000 xen.bin
>> ext2load scsi 0 0x1000000 vmlinuz
>> setenv kernsize $filesize
>> ext2load scsi 0 0x2000000 initrd.img
>> setenv initsize $filesize
>> # Tried dtuart=/soc/serial@fff36000 as well without
>> setenv bootargs "sync_console console=dtuart dtuart=serial"
>> fdt addr 0x1000
>> fdt resize
>> fdt set /chosen bootargs \"$bootargs\"
>> fdt mknod /chosen modules
>> # Tried with <1> and <2> for both as I was not sure wnether those numbers
>> # are related to number of modules
> 
> No, they are the number of u32s which are used for the address and size
> in the reg field, which contains address then size.
> 
> So <2> for both would need "<0x0 0x100000 0x0 $kernsize>" or something.
> 

Ah ok, thanks for that clarification. So 1 is ok.

>> fdt set /chosen/modules \#address-cells <1>
>> fdt set /chosen/modules \#size-cells <1>
>> fdt mknod /chosen/modules module@0
>> fdt set /chosen/modules/module@0 compatible xen,linux-zimage xen,multiboot-module
>> fdt set /chosen/modules/module@0 reg <0x1000000 $kernsize>
>> fdt set /chosen/modules/module@0 bootargs "console=hvc0 debug"
>> fdt mknod /chosen/modules module@1
>> fdt set /chosen/modules/module@1 compatible "xen,linux-initrd"
>> "xen,multiboot-module"
>> fdt set /chosen/modules/module@1 reg <0x2000000 $initsize>
>> bootz 0x800000 - 0x1000
>>
>> The memory locations are somewhat random (the one for the xen.img is used for
>> the kernel on normal installs). The boot produces the following:
>>
>> ## Flattened Device Tree blob at 00001000
>>    Booting using the fdt blob at 0x00001000
>>    reserving fdt memory region: addr=0 size=1000
>>    reserving fdt memory region: addr=1000 size=2000
>>    Using Device Tree in place at 00001000, end 00005fff
>>
>> Starting kernel ...
>>
>> - UART enabled -
>> - CPU 00000000 booting -
>> - Machine ID 00000000 -
>> - Started in Hyp mode -
>> - Zero BSS -
>> - Setting up control registers -
>> - Turning on paging -
>> - Ready -
>> RAM: 0000000000000000 - 00000000ff7fffff
>> RAM: 0000000200000000 - 00000002ffffffff
>>
>> MODULE[1]: 0000000001000000 - 0000000001471ae0 console=hvc0 debug
>> MODULE[2]: 0000000002000000 - 000000000223f08b
>> Placing Xen at 0x00000002ffe00000-0x0000000300000000
>> WARNING: Only using first bank of memory
>> Xen heap: 262144 pages  Dom heap: 784384 pages
>>
>> After that nothing. Maybe I am doing the bootargs wrong. I tried
>> xen,xen-bootargs and xen,dom0-bootargs and combinations without success. Maybe
>> the console argument is wrong. Although the full dtb path at least shows up as
>> that in a booted linux in /proc. What am I doing wrong here?
> 
> Your dtuart appears to be wrong, this is the point at which it would
> normally be expected to start with the proper (rather than early)
> console stuff.
> 
> The correct thing to use depends a bit on which version you are running,
> but if it is not recent xen.git master or staging then that's your first
> mistake ;-)

The one we (Ubuntu) shipped of course. And that's no mistake. :)

> 
> dtuart=/soc/serial@fff36000 is what I use. I've also attached my u-boot
> script. (my local scripts append -arm32 to the binary name, it is the
> "xen" file though)

Thanks, I will look through that later. One thin I noted is the dtb setup.
Examples out there often vary in having a modules sub-leaf or not. I would think
that from the early printk I am at least right to use the deeper nesting. At
least consistent with the in-source doc at that time/ release.
> 
> Ian.
> 



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Stuck trying to boot Xen 4.3 on Arm Midway

[Ian Campbell]

On Mon, 2013-12-02 at 16:43 +0100, Stefan Bader wrote:
> On 02.12.2013 16:24, Ian Campbell wrote:
> > On Mon, 2013-12-02 at 15:59 +0100, Stefan Bader wrote:
> >> I am trying to extract and combine the various pieces of information found in
> >> [1] and its sub-pages and the Xen in-tree documentation in order to make xen
> >> boot (potentially non-smp without some later changes). But since I am not
> >> familiar enough with Arm I think I am stuck doing something wrong.
> >>
> >> I compiled the hypervisor with debug and early printk for midway and use the
> >> xen.bin file (I could get no output at all when trying to create a uboot image
> >> with mkimage from the uncompressed xen.gz).
> > 
> > What version are you using? xen.bin went away in July, the right file is
> > now just "xen".
> 
> The released version of xen 4.3. At some point I might update to 4.3.1 (or
> whatever stable is current then). This probably implies picking some additional
> patches when I want to get it running on Midway.

Xen 4.3 for ARM was a tech preview. I believe at the time it ran on
Model and versatile express. Getting it running on anything else is
going to be a big job I'm afraid. I'd suggest either waiting for 4.4 or
using current tip in the meantime.

>  The xen.bin would not be
> packaged right now but it had the right format to be used by bootz while xen.gz
> (or xen after unpacking) would not be usable in uboot without converting and
> that need some more address values which I could and likely do get wrong.

In 4.3 xen.bin was the correct zImage compatible thing. I'm not sure
what you mean about the address values, but as I say if you are
expecting it to work on midway easily you are going to be disappointed.


> Thanks, I will look through that later. One thin I noted is the dtb setup.
> Examples out there often vary in having a modules sub-leaf or not. I would think
> that from the early printk I am at least right to use the deeper nesting. At
> least consistent with the in-source doc at that time/ release.

I can't remember what 4.3 did, but at least these days the depth is not
that important, it's the compatabile string and presence beneath /chosen
which matter.

Ian.

Re: Stuck trying to boot Xen 4.3 on Arm Midway

[Julien Grall]

On 12/02/2013 03:52 PM, Ian Campbell wrote:
>
>> Thanks, I will look through that later. One thin I noted is the dtb setup.
>> Examples out there often vary in having a modules sub-leaf or not. I would think
>> that from the early printk I am at least right to use the deeper nesting. At
>> least consistent with the in-source doc at that time/ release.
>
> I can't remember what 4.3 did, but at least these days the depth is not
> that important, it's the compatabile string and presence beneath /chosen
> which matter.

 From what I remember, 4.3 isn't able to boot on midway because of 
memory layout. That's why printk doesn't came up, Xen is aborting before 
the console is initialized.

-- 
Julien Grall

Re: Stuck trying to boot Xen 4.3 on Arm Midway

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 1023 bytes --]

On 02.12.2013 17:00, Julien Grall wrote:
> 
> 
> On 12/02/2013 03:52 PM, Ian Campbell wrote:
>>
>>> Thanks, I will look through that later. One thin I noted is the dtb setup.
>>> Examples out there often vary in having a modules sub-leaf or not. I would think
>>> that from the early printk I am at least right to use the deeper nesting. At
>>> least consistent with the in-source doc at that time/ release.
>>
>> I can't remember what 4.3 did, but at least these days the depth is not
>> that important, it's the compatabile string and presence beneath /chosen
>> which matter.
> 
> From what I remember, 4.3 isn't able to boot on midway because of memory layout.
> That's why printk doesn't came up, Xen is aborting before the console is
> initialized.
> 
Oh great, then I got mislead by having the earlyprintk support. I saw later
patches about adding smp support but those sounded like "only" preventing a
scary warning... I guess I can stop trying then. At least on Midway...

Thanks,
Stefan


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Stuck trying to boot Xen 4.3 on Arm Midway

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 2771 bytes --]

On 02.12.2013 16:52, Ian Campbell wrote:
> On Mon, 2013-12-02 at 16:43 +0100, Stefan Bader wrote:
>> On 02.12.2013 16:24, Ian Campbell wrote:
>>> On Mon, 2013-12-02 at 15:59 +0100, Stefan Bader wrote:
>>>> I am trying to extract and combine the various pieces of information found in
>>>> [1] and its sub-pages and the Xen in-tree documentation in order to make xen
>>>> boot (potentially non-smp without some later changes). But since I am not
>>>> familiar enough with Arm I think I am stuck doing something wrong.
>>>>
>>>> I compiled the hypervisor with debug and early printk for midway and use the
>>>> xen.bin file (I could get no output at all when trying to create a uboot image
>>>> with mkimage from the uncompressed xen.gz).
>>>
>>> What version are you using? xen.bin went away in July, the right file is
>>> now just "xen".
>>
>> The released version of xen 4.3. At some point I might update to 4.3.1 (or
>> whatever stable is current then). This probably implies picking some additional
>> patches when I want to get it running on Midway.
> 
> Xen 4.3 for ARM was a tech preview. I believe at the time it ran on
> Model and versatile express. Getting it running on anything else is
> going to be a big job I'm afraid. I'd suggest either waiting for 4.4 or
> using current tip in the meantime.
> 
>>  The xen.bin would not be
>> packaged right now but it had the right format to be used by bootz while xen.gz
>> (or xen after unpacking) would not be usable in uboot without converting and
>> that need some more address values which I could and likely do get wrong.
> 
> In 4.3 xen.bin was the correct zImage compatible thing. I'm not sure
> what you mean about the address values, but as I say if you are
> expecting it to work on midway easily you are going to be disappointed.

Not really expecting _anything_ to work easily these days. :) What I meant there
was that you have to specify base address and start offset in the mkimage call
and I used the 0x8000000 (give or take a zero I am typing from memory) that was
used in the xen build to convert from xen-bin into xen for both.

> 
> 
>> Thanks, I will look through that later. One thin I noted is the dtb setup.
>> Examples out there often vary in having a modules sub-leaf or not. I would think
>> that from the early printk I am at least right to use the deeper nesting. At
>> least consistent with the in-source doc at that time/ release.
> 
> I can't remember what 4.3 did, but at least these days the depth is not
> that important, it's the compatabile string and presence beneath /chosen
> which matter.

Ok, sounds like I can abandon all hope for 4.3 anyway. But good to know that.
Just a bit confusing.

-Stefan
> 
> Ian.
> 
> 



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Stuck trying to boot Xen 4.3 on Arm Midway

[Stefano Stabellini]

On Mon, 2 Dec 2013, Stefan Bader wrote:
> On 02.12.2013 16:52, Ian Campbell wrote:
> > On Mon, 2013-12-02 at 16:43 +0100, Stefan Bader wrote:
> >> On 02.12.2013 16:24, Ian Campbell wrote:
> >>> On Mon, 2013-12-02 at 15:59 +0100, Stefan Bader wrote:
> >>>> I am trying to extract and combine the various pieces of information found in
> >>>> [1] and its sub-pages and the Xen in-tree documentation in order to make xen
> >>>> boot (potentially non-smp without some later changes). But since I am not
> >>>> familiar enough with Arm I think I am stuck doing something wrong.
> >>>>
> >>>> I compiled the hypervisor with debug and early printk for midway and use the
> >>>> xen.bin file (I could get no output at all when trying to create a uboot image
> >>>> with mkimage from the uncompressed xen.gz).
> >>>
> >>> What version are you using? xen.bin went away in July, the right file is
> >>> now just "xen".
> >>
> >> The released version of xen 4.3. At some point I might update to 4.3.1 (or
> >> whatever stable is current then). This probably implies picking some additional
> >> patches when I want to get it running on Midway.
> > 
> > Xen 4.3 for ARM was a tech preview. I believe at the time it ran on
> > Model and versatile express. Getting it running on anything else is
> > going to be a big job I'm afraid. I'd suggest either waiting for 4.4 or
> > using current tip in the meantime.
> > 
> >>  The xen.bin would not be
> >> packaged right now but it had the right format to be used by bootz while xen.gz
> >> (or xen after unpacking) would not be usable in uboot without converting and
> >> that need some more address values which I could and likely do get wrong.
> > 
> > In 4.3 xen.bin was the correct zImage compatible thing. I'm not sure
> > what you mean about the address values, but as I say if you are
> > expecting it to work on midway easily you are going to be disappointed.
> 
> Not really expecting _anything_ to work easily these days. :) What I meant there
> was that you have to specify base address and start offset in the mkimage call
> and I used the 0x8000000 (give or take a zero I am typing from memory) that was
> used in the xen build to convert from xen-bin into xen for both.
> 
> > 
> > 
> >> Thanks, I will look through that later. One thin I noted is the dtb setup.
> >> Examples out there often vary in having a modules sub-leaf or not. I would think
> >> that from the early printk I am at least right to use the deeper nesting. At
> >> least consistent with the in-source doc at that time/ release.
> > 
> > I can't remember what 4.3 did, but at least these days the depth is not
> > that important, it's the compatabile string and presence beneath /chosen
> > which matter.
> 
> Ok, sounds like I can abandon all hope for 4.3 anyway. But good to know that.
> Just a bit confusing.

Yeah, but we are pretty close to the 4.4 release so you can try again in
a month or so :)