Xen on ARM: Force register dump from guest.

Xen on ARM: Force register dump from guest.

[Sander Bogaert]

Hi,

I'm working on some kind of microOS for Xen on ARM a bit like miniOS
but with less features. I'm having totally random crashes after make
use of variadic arguments ( using newlib ).

To debug this I'd really need a way to force a register dump from the
guest, is there a good way to do this? I don't think any of the
available hypercalls has that ability?

Regards,
Sander

Re: Xen on ARM: Force register dump from guest.

[Ian Campbell]

On Wed, 2013-06-05 at 16:36 +0200, Sander Bogaert wrote:
> Hi,
> 
> I'm working on some kind of microOS for Xen on ARM a bit like miniOS
> but with less features. I'm having totally random crashes after make
> use of variadic arguments ( using newlib ).
> 
> To debug this I'd really need a way to force a register dump from the
> guest, is there a good way to do this? I don't think any of the
> available hypercalls has that ability?

Sounds like you want "xenctx" which is in tools/xentrace/

Ian

Re: Xen on ARM: Force register dump from guest.

[Sander Bogaert]

On 5 June 2013 17:11, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2013-06-05 at 16:36 +0200, Sander Bogaert wrote:
>> Hi,
>>
>> I'm working on some kind of microOS for Xen on ARM a bit like miniOS
>> but with less features. I'm having totally random crashes after make
>> use of variadic arguments ( using newlib ).
>>
>> To debug this I'd really need a way to force a register dump from the
>> guest, is there a good way to do this? I don't think any of the
>> available hypercalls has that ability?
>
> Sounds like you want "xenctx" which is in tools/xentrace/

I doesn't show any output when invoked. Is it supposed to work for Xen
on ARM? It's also not really what I need, some hypercall or other
break that crashes the domain and does a show_execution_state() is
more like it. That way I can test how far in my code I get before the
crash and see the state that leads to it. It might be easiest to trap
something in Xen ( or use the SMC trap ) and run a custom Xen build
then?

My application crashes whenever I use variadic arguments, the crashes
are totally random and sometimes non-existent, if this would raise any
red flags in your brain I'd be very happy to hear some suggestions too
:p But my main concern is some way to debug as described.

Thanks!

Re: Xen on ARM: Force register dump from guest.

[Ian Campbell]

On Wed, 2013-06-05 at 20:18 +0200, Sander Bogaert wrote:
> On 5 June 2013 17:11, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Wed, 2013-06-05 at 16:36 +0200, Sander Bogaert wrote:
> >> Hi,
> >>
> >> I'm working on some kind of microOS for Xen on ARM a bit like miniOS
> >> but with less features. I'm having totally random crashes after make
> >> use of variadic arguments ( using newlib ).
> >>
> >> To debug this I'd really need a way to force a register dump from the
> >> guest, is there a good way to do this? I don't think any of the
> >> available hypercalls has that ability?
> >
> > Sounds like you want "xenctx" which is in tools/xentrace/
> 
> I doesn't show any output when invoked. Is it supposed to work for Xen
> on ARM?

Yes, although you will need a new enough tree, where new enough means
at least 66425df7ce61, from mid-April

>  It's also not really what I need, some hypercall or other
> break that crashes the domain and does a show_execution_state() is
> more like it.

Well, such a thing would be trivial to write, but checkout do_debug_trap
in xen/arch/arm/traps.c, you probably want "hvc #0xffff" in your code.

> My application crashes whenever I use variadic arguments,

crashes how, segfault or something else?

Where did your stdargs.h come from? I'd have thought the compiler
provided stuff ought to be reasonably robust assuming your application
isn't buggy.

Ian.

Re: Xen on ARM: Force register dump from guest.

[Sander Bogaert]

On 5 June 2013 20:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2013-06-05 at 20:18 +0200, Sander Bogaert wrote:
>> On 5 June 2013 17:11, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > On Wed, 2013-06-05 at 16:36 +0200, Sander Bogaert wrote:
>> >> Hi,
>> >>
>> >> I'm working on some kind of microOS for Xen on ARM a bit like miniOS
>> >> but with less features. I'm having totally random crashes after make
>> >> use of variadic arguments ( using newlib ).
>> >>
>> >> To debug this I'd really need a way to force a register dump from the
>> >> guest, is there a good way to do this? I don't think any of the
>> >> available hypercalls has that ability?
>> >
>> > Sounds like you want "xenctx" which is in tools/xentrace/
>>
>> I doesn't show any output when invoked. Is it supposed to work for Xen
>> on ARM?
>
> Yes, although you will need a new enough tree, where new enough means
> at least 66425df7ce61, from mid-April

As you may have read I'm having an issue with the latest Xen / linux
kernel at the moment. I haven't been able to find a working
combination of commits in both trees yet to get this functionallity in
and I was smart enough not to write down the hashes of the working
setup I do have :-)

>>  It's also not really what I need, some hypercall or other
>> break that crashes the domain and does a show_execution_state() is
>> more like it.
>
> Well, such a thing would be trivial to write, but checkout do_debug_trap
> in xen/arch/arm/traps.c, you probably want "hvc #0xffff" in your code.

This seems perfect! I'l try using these traps.

>> My application crashes whenever I use variadic arguments,
>
> crashes how, segfault or something else?

It's not as much an application as my "micro" OS :) The crashes are
totally random but it's always the domain that crashes. Sometimes a
translation fault, sometimes some "unexpected hypervisor trap" ending
up in SVC or even IRQ mode ( which shouldn't even be enabled on domain
boot?! ).

Maybe it's related to the segfaulting Julien mailed about because it
always seems to be on CPU 1 according to Xen's crash dump and maybe
when it just works and doesn't crash it's scheduled on CPU 0? I
haven't done enough testing to be sure this behavior is consistent.

> Where did your stdargs.h come from? I'd have thought the compiler
> provided stuff ought to be reasonably robust assuming your application
> isn't buggy.

I'm using newlib for now.

Re: Xen on ARM: Force register dump from guest.

[Tim Deegan]

At 20:18 +0200 on 05 Jun (1370463493), Sander Bogaert wrote:
> My application crashes whenever I use variadic arguments, the crashes
> are totally random and sometimes non-existent, if this would raise any
> red flags in your brain I'd be very happy to hear some suggestions too

In the early stages of the Xen ARM port we had intermittent problems
with varadics when the stack pointer got misaligned -- the toolchain's
code for laying out stack arguments assumes that the stack pointer is
properly (i.e. 8-byte, on 32-bit ARM!) aligned to start with, and can do
some confusing things if not.

Cheers,

Tim.

Re: Xen on ARM: Force register dump from guest.

[Sander Bogaert]

On 6 June 2013 11:33, Tim Deegan <tim@xen.org> wrote:
> At 20:18 +0200 on 05 Jun (1370463493), Sander Bogaert wrote:
>> My application crashes whenever I use variadic arguments, the crashes
>> are totally random and sometimes non-existent, if this would raise any
>> red flags in your brain I'd be very happy to hear some suggestions too
>
> In the early stages of the Xen ARM port we had intermittent problems
> with varadics when the stack pointer got misaligned -- the toolchain's
> code for laying out stack arguments assumes that the stack pointer is
> properly (i.e. 8-byte, on 32-bit ARM!) aligned to start with, and can do
> some confusing things if not.

Thank you for the suggestion. I did however verify the stack is
properly aligned in the linkerscript. I tried a bunch of things and
I'm getting nowhere so even your craziest ideas (if you have any) are
more than welcome!

Thanks

Re: Xen on ARM: Force register dump from guest.

[Tim Deegan]

At 16:48 +0200 on 07 Jun (1370623706), Sander Bogaert wrote:
> On 6 June 2013 11:33, Tim Deegan <tim@xen.org> wrote:
> > At 20:18 +0200 on 05 Jun (1370463493), Sander Bogaert wrote:
> >> My application crashes whenever I use variadic arguments, the crashes
> >> are totally random and sometimes non-existent, if this would raise any
> >> red flags in your brain I'd be very happy to hear some suggestions too
> >
> > In the early stages of the Xen ARM port we had intermittent problems
> > with varadics when the stack pointer got misaligned -- the toolchain's
> > code for laying out stack arguments assumes that the stack pointer is
> > properly (i.e. 8-byte, on 32-bit ARM!) aligned to start with, and can do
> > some confusing things if not.
> 
> Thank you for the suggestion. I did however verify the stack is
> properly aligned in the linkerscript.

It needs a bit more than that -- the stack pointer must be 8-byte aligned
on all (visible) function calls.  In our case, although the stack area
was properly aligned, there were some places (in asm code) where we
called C functions with the stack only 4-byte aligned.

Anyway, if the application runs fine on linux without Xen, I guess we
should be looking at the Xen code rather than yours. :)  If it's really
non-deterministic, even with the same call stacks, then maybe something
is messed up with interrupt delivery.  It might be that Xen's
return-from-interrupt code doesn't unwind non-aligned stacks properly.

Tim.

Re: Xen on ARM: Force register dump from guest.

[Tim Deegan]

At 16:41 +0100 on 07 Jun (1370623276), Tim Deegan wrote:
> At 16:48 +0200 on 07 Jun (1370623706), Sander Bogaert wrote:
> > On 6 June 2013 11:33, Tim Deegan <tim@xen.org> wrote:
> > > At 20:18 +0200 on 05 Jun (1370463493), Sander Bogaert wrote:
> > >> My application crashes whenever I use variadic arguments, the crashes
> > >> are totally random and sometimes non-existent, if this would raise any
> > >> red flags in your brain I'd be very happy to hear some suggestions too
> > >
> > > In the early stages of the Xen ARM port we had intermittent problems
> > > with varadics when the stack pointer got misaligned -- the toolchain's
> > > code for laying out stack arguments assumes that the stack pointer is
> > > properly (i.e. 8-byte, on 32-bit ARM!) aligned to start with, and can do
> > > some confusing things if not.
> > 
> > Thank you for the suggestion. I did however verify the stack is
> > properly aligned in the linkerscript.
> 
> It needs a bit more than that -- the stack pointer must be 8-byte aligned
> on all (visible) function calls.  In our case, although the stack area
> was properly aligned, there were some places (in asm code) where we
> called C functions with the stack only 4-byte aligned.
> 
> Anyway, if the application runs fine on linux without Xen

Looking back at the thread, I realise this makes no sense.  Sorry. :)

Tim.

> If it's really
> non-deterministic, even with the same call stacks, then maybe something
> is messed up with interrupt delivery.  It might be that Xen's
> return-from-interrupt code doesn't unwind non-aligned stacks properly.
> 
> Tim.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel