XEN 4.5, odroid-xu, domU

XEN 4.5, odroid-xu, domU

[Suriyan Ramasami]

Greetings!

   Thanks for the previous help wrt my attempts at getting XEN running
with dom0 on an exynos5410 based Odroid-XU.
   Currently, Things are OK with having a single CPU enabled.

   Previously, there was an issue with having to pass the Arch timer
frequency having to be passed through the DTS. I did successfully
update the Arch timer frequency in u-boot with the help of a modified
trustzone. It is still experimental, but the end result is that CNTFRQ
is read as 24000000 as required and not 0.

   Pursuing further, I am having issues starting a domU kernel. In all
the documentation I always find the dtb appended to the kernel. I am
refraining from doing so.

My domU config file:
root@odroid-desktop:~# cat vmTest.cfg
kernel = "/root/zImage.dom0"
memory = 512
name = "domuTest"
vcpus = 1
disk = [ 'phy:/dev/loop0,xvda,w' ]
extra = "console=hvc0 root=/dev/xvda debug rw"

I am using the same kernel which is running as dom0.

I create domU with: xl create vmTest.cfg
It creates is fine but then the domU kernel immediately crashes.

root@odroid-desktop:~# xl create vmTest.cfg -d
Parsing config from vmTest.cfg
{
    "domid": null,
    "config": {
        "c_info": {
            "type": "pv",
            "name": "domuTest",
            "uuid": "3cccea5b-330d-42a6-9d61-af3e3d6c588c",
            "run_hotplug_scripts": "True"
        },
        "b_info": {
            "max_vcpus": 1,
            "avail_vcpus": [
                0
            ],
            "max_memkb": 524288,
            "target_memkb": 524288,
            "shadow_memkb": 5120,
            "sched_params": {

            },
            "claim_mode": "True",
            "type.pv": {
                "kernel": "/root/zImage.dom0",
                "cmdline": "console=hvc0 root=/dev/xvda debug rw"
            }
        },
        "disks": [
            {
                "pdev_path": "/dev/loop0",
                "vdev": "xvda",
                "format": "raw",
                "readwrite": 1
            }
        ],
        "on_reboot": "restart"
    }
}

xl list
root@odroid-desktop:~# xl list
Name                                        ID   Mem VCPUs    State    Time(s)
Domain-0                                     0  1024     1     r-----     167.3
domuTest                                     3   512     1     ------      25.7

root@odroid-desktop:~# /usr/local/lib/xen/bin/xenctx 3
PC:       0000000c
CPSR:     600001d7
USR:               SP:00000000 LR:00000000
SVC: SPSR:000001d3 SP:40c579c0 LR:40c569c4
FIQ: SPSR:00000000 SP:00000000 LR:00000000
IRQ: SPSR:00000000 SP:00000000 LR:00000000
ABT: SPSR:600001d7 SP:00000000 LR:00000010
UND: SPSR:00000000 SP:00000000 LR:00000000

 r0_usr: 10000000     r1_usr: 0000001c     r2_usr: 40c569c4
 r3_usr: 00000030     r4_usr: 40c56980     r5_usr: 40008000
 r6_usr: ffffffff     r7_usr: 40c579e0     r8_usr: 40c679e0
 r9_usr: 407cda20    r10_usr: 407cd178    r11_usr: 40c569b4
r12_usr: 40c569b4

 r8_fiq: 00000000
 r9_fiq: 00000000    r10_fiq: 00000000    r11_fiq: 00000000
r12_fiq: 00000000

SCTLR: 00c5107d
TTBCR: 0000000000000000
TTBR0: 0000000040004000
TTBR1: 0000000000000000

Note that the PC is at 0x0000000c

My question, is how do I go about debugging this, as I do not get any
output nor a stack trace or anything in the XEN console.

The link: http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions#Common_DomU_Pitfalls
points at one such case where I could hit the issue. I do not have
CONFIG_DEBUG_LL set, but looks awefully similar to what is being
described. So, most probably this is the issue, as I do not see any
output as well.

Any and all help is deeply appreciated.

Thanks
- Suriyan

Re: XEN 4.5, odroid-xu, domU

[Ian Campbell]

On Sun, 2014-07-20 at 10:18 -0700, Suriyan Ramasami wrote:

>    Pursuing further, I am having issues starting a domU kernel. In all
> the documentation I always find the dtb appended to the kernel.

This was the case until about half way through the 4.4 development
cycle. From 4.4 onwards the toolstack will generate a DTB and you should
avoid appending one.

What docs are you looking at which needs updating?

>  I am
> refraining from doing so.
> 
> My domU config file:
> root@odroid-desktop:~# cat vmTest.cfg
> kernel = "/root/zImage.dom0"
> memory = 512
> name = "domuTest"
> vcpus = 1
> disk = [ 'phy:/dev/loop0,xvda,w' ]
> extra = "console=hvc0 root=/dev/xvda debug rw"
> 
> I am using the same kernel which is running as dom0.
> 
> I create domU with: xl create vmTest.cfg
> It creates is fine but then the domU kernel immediately crashes.

This usually indicates that you have CONFIG_DEBUG_LL set to something.
This works in dom0 (where the host platform's UART can be used) but not
for guests which see a purely virtualized physical address space.

> Note that the PC is at 0x0000000c

That's a prefetch abort. What happens is that when DEBUG_LL tries to
access the MMIO region it takes a data abort, to address 0x10. But there
is nothing there, so then you take a prefetch abort to 0xc. There is
nothing there either so the processor just loops taking aborts.

> The link: http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions#Common_DomU_Pitfalls
> points at one such case where I could hit the issue. I do not have
> CONFIG_DEBUG_LL set, but looks awefully similar to what is being
> described. So, most probably this is the issue, as I do not see any
> output as well.

Uh, I should clearly have read all the way to the end of your mail
before I commented :-).

If your issue isn't DEBUG_LL then I'm not sure what it could be.

You could try adding calls to xen_raw_printk to the guest kernel's
normal printk routines -- this might get you some earlier debugging
output if you have a debug=y build of Xen.

LR_svc is set to 0x40c569c4 which suggests that might be the original
faulting address, but it looks like it is pre-MMU and with all the
relocation which occurs during boot it might be tricky to find out what
it is.

Last trick I use is to sprinkle "hvc #0xffd" around the guest boot path
asm to see how far it is getting. This needs debug=y, see
traps.c:do_debug_trap for some of the magic immediates which you can
use.

I'm afraid that given the early fault my money is on DEBUG_LL. I suggest
making sure you have done a completely clean build without it.

Ian.

Re: XEN 4.5, odroid-xu, domU

[Julien Grall]

Hi Suriyan,

On 07/21/2014 12:02 PM, Ian Campbell wrote:
> On Sun, 2014-07-20 at 10:18 -0700, Suriyan Ramasami wrote:
> 
>>    Pursuing further, I am having issues starting a domU kernel. In all
>> the documentation I always find the dtb appended to the kernel.
> 
> This was the case until about half way through the 4.4 development
> cycle. From 4.4 onwards the toolstack will generate a DTB and you should
> avoid appending one.
> 
> What docs are you looking at which needs updating?
> 
>>  I am
>> refraining from doing so.
>>
>> My domU config file:
>> root@odroid-desktop:~# cat vmTest.cfg
>> kernel = "/root/zImage.dom0"
>> memory = 512
>> name = "domuTest"
>> vcpus = 1
>> disk = [ 'phy:/dev/loop0,xvda,w' ]
>> extra = "console=hvc0 root=/dev/xvda debug rw"
>>
>> I am using the same kernel which is running as dom0.
>>
>> I create domU with: xl create vmTest.cfg
>> It creates is fine but then the domU kernel immediately crashes.
> 
> This usually indicates that you have CONFIG_DEBUG_LL set to something.
> This works in dom0 (where the host platform's UART can be used) but not
> for guests which see a purely virtualized physical address space.
> 
>> Note that the PC is at 0x0000000c
> 
> That's a prefetch abort. What happens is that when DEBUG_LL tries to
> access the MMIO region it takes a data abort, to address 0x10. But there
> is nothing there, so then you take a prefetch abort to 0xc. There is
> nothing there either so the processor just loops taking aborts.
> 
>> The link: http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions#Common_DomU_Pitfalls
>> points at one such case where I could hit the issue. I do not have
>> CONFIG_DEBUG_LL set, but looks awefully similar to what is being
>> described. So, most probably this is the issue, as I do not see any
>> output as well.
> 
> Uh, I should clearly have read all the way to the end of your mail
> before I commented :-).
> 
> If your issue isn't DEBUG_LL then I'm not sure what it could be.
> 
> You could try adding calls to xen_raw_printk to the guest kernel's
> normal printk routines -- this might get you some earlier debugging
> output if you have a debug=y build of Xen.
> 
> LR_svc is set to 0x40c569c4 which suggests that might be the original
> faulting address, but it looks like it is pre-MMU and with all the
> relocation which occurs during boot it might be tricky to find out what
> it is.
> 
> Last trick I use is to sprinkle "hvc #0xffd" around the guest boot path
> asm to see how far it is getting. This needs debug=y, see
> traps.c:do_debug_trap for some of the magic immediates which you can
> use.
> 
> I'm afraid that given the early fault my money is on DEBUG_LL. I suggest
> making sure you have done a completely clean build without it.

I suspect the odroid xu kernel is not multiplatform. In this case you
can't reuse the same kernel for DOM0 and the guest.

You also need to make sure that CONFIG_ARCH_VIRT=y.

Regards,

-- 
Julien Grall

Re: XEN 4.5, odroid-xu, domU

[Ian Campbell]

On Mon, 2014-07-21 at 12:27 +0100, Julien Grall wrote:

> I suspect the odroid xu kernel is not multiplatform. In this case you
> can't reuse the same kernel for DOM0 and the guest.
> 
> You also need to make sure that CONFIG_ARCH_VIRT=y.

Are these things worth adding to
http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions#Common_DomU_Pitfalls
?

Re: XEN 4.5, odroid-xu, domU

[Julien Grall]

On 07/21/2014 12:29 PM, Ian Campbell wrote:
> On Mon, 2014-07-21 at 12:27 +0100, Julien Grall wrote:
> 
>> I suspect the odroid xu kernel is not multiplatform. In this case you
>> can't reuse the same kernel for DOM0 and the guest.
>>
>> You also need to make sure that CONFIG_ARCH_VIRT=y.
> 
> Are these things worth adding to
> http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions#Common_DomU_Pitfalls
> ?

I though we already added a paragraph about multi-platform kernel on the
wiki page but I can't find it.

-- 
Julien Grall

Re: XEN 4.5, odroid-xu, domU

[Suriyan Ramasami]

Thank you Julien and Ian for your comments and suggestions!

Ian,
    Thank you for your detailed comments and notes regarding
CONFIG_LL. Thank you for your suggestions on the debugging tricks. I
shall remember these for the next set of hurdles if and when I face
them.

Julien,
    Thanks for the hint on the kernel not being multiplatform. It
indeed wasn't. I do see some patches in the linux samsung next tree,
for including exynos in the multiplatform setting.
    Hence, I did a separate compile for domU, setting it to
multiplatform and CONFIG_ARCH_VIRT=y by the below:
=> System Type -> ARM system type -> Allow multiple platforms to be selected.
=> System Type -> Multiple platform selection -> ARMV v7 based
=> System Type -> Dummy Virtual machine.

With this domU boots up like a champ!

Maybe, I might try dabbling with samsung-linux next branch of the
linux kernel and see if dom0 and domU can use the same kernel.

Thank you so much again, you guys are just amazing!!


Regards
- Suriyan

PS: It would be great if the multiplatform bit and the
CONFIG_ARCH_VIRT hint is mentioned in the wiki for the next poor soul
who hits the same issue!

On Mon, Jul 21, 2014 at 4:33 AM, Julien Grall <julien.grall@linaro.org> wrote:
> On 07/21/2014 12:29 PM, Ian Campbell wrote:
>> On Mon, 2014-07-21 at 12:27 +0100, Julien Grall wrote:
>>
>>> I suspect the odroid xu kernel is not multiplatform. In this case you
>>> can't reuse the same kernel for DOM0 and the guest.
>>>
>>> You also need to make sure that CONFIG_ARCH_VIRT=y.
>>
>> Are these things worth adding to
>> http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions#Common_DomU_Pitfalls
>> ?
>
> I though we already added a paragraph about multi-platform kernel on the
> wiki page but I can't find it.
>
> --
> Julien Grall

Re: XEN 4.5, odroid-xu, domU

[Ian Campbell]

On Mon, 2014-07-21 at 12:02 -0700, Suriyan Ramasami wrote:
>     Hence, I did a separate compile for domU, setting it to
> multiplatform and CONFIG_ARCH_VIRT=y by the below:
> => System Type -> ARM system type -> Allow multiple platforms to be selected.
> => System Type -> Multiple platform selection -> ARMV v7 based
> => System Type -> Dummy Virtual machine.
> 
> With this domU boots up like a champ!

Excellent!

> Maybe, I might try dabbling with samsung-linux next branch of the
> linux kernel and see if dom0 and domU can use the same kernel.
> 
> Thank you so much again, you guys are just amazing!!
> 
> 
> Regards
> - Suriyan
> 
> PS: It would be great if the multiplatform bit and the
> CONFIG_ARCH_VIRT hint is mentioned in the wiki for the next poor soul
> who hits the same issue!

If you send me your wiki username I'll give you write permissions ;-)

(it's not given by default due to spammers)

Ian.