Debugging Xen early boot

Debugging Xen early boot

[Zytaruk, Kelly]

I am having problems getting XEN boot messages over a serial console.

I have two systems.  On one system I can see the XEN boot message over a serial console whereas on the other system I cannot.

I physically move the same PCIe serial card and hard drive from one system to the other.  One system shows me the Xen boot messages and the other doesn't.

My config in /etc/grub.d/20_linux_xen is
xen_args="dom0_max_vcpus=4 dom0_mem=2048M,max:3072M iommu=1 conring_size=16384 loglvl=all guest_loglvl=all com1=115200,8n1,pci console=com1"

I put a printk at the beginning of the Linux boot and I see the Dom0 Linux boot messages on both systems.  So I know that the serial port works on both systems but for some reason I don't see the Xen messages.

I would like to debug through __start_xen to see how the 16550 is being initialized.  Is there an easy way to debug __start_xen? 

Thanks,
Kelly


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Debugging Xen early boot

[Konrad Rzeszutek Wilk]

On Thu, May 05, 2016 at 01:24:26PM +0000, Zytaruk, Kelly wrote:
> I am having problems getting XEN boot messages over a serial console.
> 
> I have two systems.  On one system I can see the XEN boot message over a serial console whereas on the other system I cannot.
> 
> I physically move the same PCIe serial card and hard drive from one system to the other.  One system shows me the Xen boot messages and the other doesn't.
> 
> My config in /etc/grub.d/20_linux_xen is
> xen_args="dom0_max_vcpus=4 dom0_mem=2048M,max:3072M iommu=1 conring_size=16384 loglvl=all guest_loglvl=all com1=115200,8n1,pci console=com1"
> 
> I put a printk at the beginning of the Linux boot and I see the Dom0 Linux boot messages on both systems.  So I know that the serial port works on both systems but for some reason I don't see the Xen messages.

Also, are there any other Serial or Communication devices on the other
machine? Could you provide the lspci -v from both machines?
It may be that there is a built-in on the motherboard (like an AMT type
thing, or IPMI SoL?)

And if you boot Linux on it, can you do 'dmesg | grep tty' to see
what it finds in terms of serial cards?

> 
> I would like to debug through __start_xen to see how the 16550 is being initialized.  Is there an easy way to debug __start_xen? 

> 
> Thanks,
> Kelly
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Debugging Xen early boot

[Zytaruk, Kelly]

Konrad, immediate problem is solved although I don't know why.  I tried putting the serial card into a different PCIe slot and it works properly.  Must be something wrong with the slot.  I will leave that as a problem for another day.

As for the other question I am still curious as to how to debug early Xen.  How was __start_xen written and tested, what sort of tools were used to debug it and get it working right?  What do I do in situations where I absolutely can't get serial working properly?

Thanks,
Kelly

> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Thursday, May 05, 2016 10:20 AM
> To: Zytaruk, Kelly
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Debugging Xen early boot
> 
> On Thu, May 05, 2016 at 01:24:26PM +0000, Zytaruk, Kelly wrote:
> > I am having problems getting XEN boot messages over a serial console.
> >
> > I have two systems.  On one system I can see the XEN boot message over a
> serial console whereas on the other system I cannot.
> >
> > I physically move the same PCIe serial card and hard drive from one system to
> the other.  One system shows me the Xen boot messages and the other doesn't.
> >
> > My config in /etc/grub.d/20_linux_xen is
> > xen_args="dom0_max_vcpus=4 dom0_mem=2048M,max:3072M iommu=1
> conring_size=16384 loglvl=all guest_loglvl=all com1=115200,8n1,pci
> console=com1"
> >
> > I put a printk at the beginning of the Linux boot and I see the Dom0 Linux boot
> messages on both systems.  So I know that the serial port works on both
> systems but for some reason I don't see the Xen messages.
> 
> Also, are there any other Serial or Communication devices on the other
> machine? Could you provide the lspci -v from both machines?
> It may be that there is a built-in on the motherboard (like an AMT type thing, or
> IPMI SoL?)
> 
> And if you boot Linux on it, can you do 'dmesg | grep tty' to see what it finds in
> terms of serial cards?
> 
> >
> > I would like to debug through __start_xen to see how the 16550 is being
> initialized.  Is there an easy way to debug __start_xen?
> 
> >
> > Thanks,
> > Kelly
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Debugging Xen early boot

[Jan Beulich]

>>> On 06.05.16 at 14:31, <Kelly.Zytaruk@amd.com> wrote:
> As for the other question I am still curious as to how to debug early Xen.  
> How was __start_xen written and tested, what sort of tools were used to debug 
> it and get it working right?

I don't know how it was done here, but having been through the
exercise of debugging early boot code in other environments, I
can only say: Get creative. E.g. leverage whatever storage you
have that persists across at least a warm reboot. Depending on
system I have found e.g. video RAM or some of the I/O ports in
the 0x80-0x8f range usable. I also recall that on one specific
box I had to resort to using some of the CMOS non-volatile RAM
or even the RTC registers that don't change rapidly.

>  What do I do in situations where I absolutely 
> can't get serial working properly?

Get serial working properly ;-) ? Or try USB (EHCI). Or use text mode
video output.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Debugging Xen early boot

[David Vrabel]

On 06/05/16 13:31, Zytaruk, Kelly wrote:
> 
> As for the other question I am still curious as to how to debug
> early Xen. How was __start_xen written and tested, what sort of tools were
> used to debug it and get it working right? What do I do in situations
> where I absolutely can't get serial working properly?

When I was working on the kexec code in Xen I had some success with
using an ICE.  They're not cheap though and no off-the-shelf x86
motherboard has the correct header to connect an ICE...

David

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Debugging Xen early boot

[Zytaruk, Kelly]

> -----Original Message-----
> From: Xen-devel [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Jan
> Beulich
> Sent: Friday, May 06, 2016 8:54 AM
> To: Zytaruk, Kelly
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Debugging Xen early boot
> 
> >>> On 06.05.16 at 14:31, <Kelly.Zytaruk@amd.com> wrote:
> > As for the other question I am still curious as to how to debug early Xen.
> > How was __start_xen written and tested, what sort of tools were used
> > to debug it and get it working right?
> 
> I don't know how it was done here, but having been through the exercise of
> debugging early boot code in other environments, I can only say: Get creative.
> E.g. leverage whatever storage you have that persists across at least a warm
> reboot. Depending on system I have found e.g. video RAM or some of the I/O
> ports in the 0x80-0x8f range usable. I also recall that on one specific box I had to
> resort to using some of the CMOS non-volatile RAM or even the RTC registers
> that don't change rapidly.

Ouch! Sounds painful.  Looks like I had better hope that everything works fine up until the end of the init of the 16550s :(

> 
> >  What do I do in situations where I absolutely can't get serial
> > working properly?
> 
> Get serial working properly ;-) ? Or try USB (EHCI). Or use text mode video
> output.
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Debugging Xen early boot

[Zytaruk, Kelly]

> -----Original Message-----
> From: David Vrabel [mailto:david.vrabel@citrix.com]
> Sent: Friday, May 06, 2016 9:04 AM
> To: Zytaruk, Kelly; Konrad Rzeszutek Wilk
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Debugging Xen early boot
> 
> On 06/05/16 13:31, Zytaruk, Kelly wrote:
> >
> > As for the other question I am still curious as to how to debug early
> > Xen. How was __start_xen written and tested, what sort of tools were
> > used to debug it and get it working right? What do I do in situations
> > where I absolutely can't get serial working properly?
> 
> When I was working on the kexec code in Xen I had some success with using an
> ICE.  They're not cheap though and no off-the-shelf x86 motherboard has the
> correct header to connect an ICE...

Looks like my options are limited as I am on a budget :(
I think I will pass on trying to resolve the issue with the bad PCIe port and use the one that works.

> 
> David
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Debugging Xen early boot

[Konrad Rzeszutek Wilk]

On Fri, May 06, 2016 at 01:41:38PM +0000, Zytaruk, Kelly wrote:
> 
> 
> > -----Original Message-----
> > From: Xen-devel [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Jan
> > Beulich
> > Sent: Friday, May 06, 2016 8:54 AM
> > To: Zytaruk, Kelly
> > Cc: xen-devel@lists.xen.org
> > Subject: Re: [Xen-devel] Debugging Xen early boot
> > 
> > >>> On 06.05.16 at 14:31, <Kelly.Zytaruk@amd.com> wrote:
> > > As for the other question I am still curious as to how to debug early Xen.
> > > How was __start_xen written and tested, what sort of tools were used
> > > to debug it and get it working right?
> > 
> > I don't know how it was done here, but having been through the exercise of
> > debugging early boot code in other environments, I can only say: Get creative.
> > E.g. leverage whatever storage you have that persists across at least a warm
> > reboot. Depending on system I have found e.g. video RAM or some of the I/O
> > ports in the 0x80-0x8f range usable. I also recall that on one specific box I had to
> > resort to using some of the CMOS non-volatile RAM or even the RTC registers
> > that don't change rapidly.
> 
> Ouch! Sounds painful.  Looks like I had better hope that everything works fine up until the end of the init of the 16550s :(

The way I troubleshooted some of the ns16650 issues when working on the
PCIe support was to use the VGA buffer and print characters there.

Had to hack the Xen to allow that and it was mostly just printing 'A'
->'Z' to figure out where my code broke.

Other options (now that I think of this) would have been to pass in
the PCIe device to a guest and run Xen within the guest. That is nested
virtualization with Xen and the card in it. But of course if you are
doing early bringup that won't help much.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel