AW: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

AW: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Neobiker]

>> I'm worried about stability, changes in behaviour, changes in kernel /
>> parameters, problems with compiling some orig xen kernel, problems running
>
>All of those, except stability, are issues you are going to encounter with
>a new kernel...
>
>Can you be more specific about the stability? Have you seen it crash?

I think i had some issues seen during testing (core or even kernel error messages), also i saw a cpu panic (twice i think) which isn't reproducable (immediate reboot that worked afterwards). 
And i would have choosed the "old" kernel, but it didn't work on my Intel Standard System (not really new as Q35/Core2Duo is about 2 years old) whereas i think 2.6.18 should run without problems on that hardware. I did a lot of tests / configurations with compiling on different distros (debian squeeze and lenny, fc14, OpenSuse) with different kernel versions (also updated xen kernel sources during compilation sometimes which ends up in different results during compilation) and saw an inhomogone picture of xen 4 / kernels in total, so that is why i say "does not fit for a prod system now". I am missing a reproducable, homogene behaviour of xen / kernels and packages like it was with 3.x.

Unfortunately i don't have any logs because i only tested the new xen 4 features to
verify the xen wiki docu and features to be able to get a big picture of the actual xen 4 status. I am not really happy with the result: xen 4.0.1 with pvops kernel works mostly with standard features, missing some things like pvusb. 2.6.18 kernel didn't compile or run's only with also seen kernel errors (4.0.2-rc1-pre) on my testsystem. Additional having video problems (agpart) with standard drivers - actually only squeeze runs with X11 without errors.

Actually Squeeze gets the best xen 4 results with xen pakages available :-)
FC14 is missing a dom0 kernel package, so the plain xen 4 RPM isn't really a succes story yet... when this (dom0 kernel pakage) comes, we can say "XEN is back" again (if xen 4 is stable and homogene in behaviour as 3.4.2 was).

>> 2.6.18 kernel like above,  dependencies like pvops version .32 for > 4.0.1,
>> .31 for < 4.0.1, bugs in 4.0.0, less bugs in 4.0.1, missing features like
>
>PVUSB.. well we would love if somebody volunteered to do the driver.

Yes, me too ;-)

>
>> pvusb, windows in vhd didn't like the GPLPV drivers (blue screen), signed
>
>Uhh, no idea. I am actually using the Novell GPL drivers in Windows 2000
>and they seem to work fine.

i am using GPLPV without issues with phy:LVM devices (XP and Win7)

>
>> Citrix PV drivers only work with version 5.5, not 5.6, pvops kernel works on
>> my hardware with debian pvops xen 4.0.1 kernel, but xen pvops kernel
>> compiled according to wiki fc13 page has errors with agpart loading and so
>> on..... so i'm waiting for 4.0.3 ;-)
>
>Hm, the agpart loading I thought was fixed. When did you observe this behavior?

This is actual a problem with 4.0.1 (stable tree) on fc14. squeeze is working well at this time. Didn't verify 4.0.2-rc1 yet with X11 - i had to clean all the testing chaos on my discs in order not to mix up different things (which might happened though) ;-)

regards
neobiker

Re: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Pasi Kärkkäinen]

On Wed, Nov 17, 2010 at 09:52:06PM +0100, Neobiker wrote:
> >> I'm worried about stability, changes in behaviour, changes in kernel /
> >> parameters, problems with compiling some orig xen kernel, problems running
> >
> >All of those, except stability, are issues you are going to encounter with
> >a new kernel...
> >
> >Can you be more specific about the stability? Have you seen it crash?
> 
> I think i had some issues seen during testing (core or even kernel error messages), also i saw a cpu panic (twice i think) which isn't reproducable (immediate reboot that worked afterwards). 
> And i would have choosed the "old" kernel, but it didn't work on my Intel Standard System (not really new as Q35/Core2Duo is about 2 years old) whereas i think 2.6.18 should run without problems on that hardware. I did a lot of tests / configurations with compiling on different distros (debian squeeze and lenny, fc14, OpenSuse) with different kernel versions (also updated xen kernel sources during compilation sometimes which ends up in different results during compilation) and saw an inhomogone picture of xen 4 / kernels in total, so that is why i say "does not fit for a prod system now". I am missing a reproducable, homogene behaviour of xen / kernels and packages like it was with 3.x.
> 
> Unfortunately i don't have any logs because i only tested the new xen 4 features to
> verify the xen wiki docu and features to be able to get a big picture of the actual xen 4 status. I am not really happy with the result: xen 4.0.1 with pvops kernel works mostly with standard features, missing some things like pvusb. 2.6.18 kernel didn't compile or run's only with also seen kernel errors (4.0.2-rc1-pre) on my testsystem. Additional having video problems (agpart) with standard drivers - actually only squeeze runs with X11 without errors.
> 
> Actually Squeeze gets the best xen 4 results with xen pakages available :-)
> FC14 is missing a dom0 kernel package, so the plain xen 4 RPM isn't really a succes story yet... when this (dom0 kernel pakage) comes, we can say "XEN is back" again (if xen 4 is stable and homogene in behaviour as 3.4.2 was).
> 
> >> 2.6.18 kernel like above,  dependencies like pvops version .32 for > 4.0.1,
> >> .31 for < 4.0.1, bugs in 4.0.0, less bugs in 4.0.1, missing features like
> >
> >PVUSB.. well we would love if somebody volunteered to do the driver.
> 
> Yes, me too ;-)
> 
> >
> >> pvusb, windows in vhd didn't like the GPLPV drivers (blue screen), signed
> >
> >Uhh, no idea. I am actually using the Novell GPL drivers in Windows 2000
> >and they seem to work fine.
> 
> i am using GPLPV without issues with phy:LVM devices (XP and Win7)
> 
> >
> >> Citrix PV drivers only work with version 5.5, not 5.6, pvops kernel works on
> >> my hardware with debian pvops xen 4.0.1 kernel, but xen pvops kernel
> >> compiled according to wiki fc13 page has errors with agpart loading and so
> >> on..... so i'm waiting for 4.0.3 ;-)
> >
> >Hm, the agpart loading I thought was fixed. When did you observe this behavior?
> 
> This is actual a problem with 4.0.1 (stable tree) on fc14. squeeze is working well at this time. Didn't verify 4.0.2-rc1 yet with X11 - i had to clean all the testing chaos on my discs in order not to mix up different things (which might happened though) ;-)
> 

There are kernel rpms for Fedora, packaged by myoung.
Those kernel rpms are up-to-date.

http://repos.fedorapeople.org/repos/myoung/dom0-kernel/

Xen-4.0.1-6 rpms in Fedora contain some important backports (bugfixes) from xen-4.0.2.

-- Pasi

Re: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Konrad Rzeszutek Wilk]

On Fri, Nov 26, 2010 at 11:45:56AM -0800, Neobiker wrote:
> 
> Hi,
> most kernels (32-bit) except the squeeze 64-bit kernel produce the following
> error:
> 
> Linux agpgart interface v0.103
> agpgart-intel 0000:00:00.0: Intel Q35 Chipset
> agpgart-intel 0000:00:00.0: can't get memory for scratch page

That looks oddly familiar, but I can't remember where I saw it.

> agpgart-intel 0000:00:00.0: agp_backend_initialize() failed
> agpgart-intel: probe of 0000:00:00.0 failed with error -12
> [drm:drm_fill_in_dev] *ERROR* Cannot initialize the agpgart module.
> 
> Above ist actually seen FC14 / XEN 4.0.1 with latest kernel from MA Young:
> 2.6.32.26-174.xendom0.fc12.i686.PAE

You are comparing oranges and bananses here. The MA Young vs Squeeze (unless
you are using the IanC set of kernels) have a different set of patches for the GART controller.

Wht happens if you run a 2.6.32.26-174.xendom0.fc12.x86_64 kernel?

Re: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Konrad Rzeszutek Wilk]

On Tue, Nov 16, 2010 at 12:09:56PM -0800, Neobiker wrote:
> 
> 
> Konrad Rzeszutek Wilk wrote:
> > 
> > 
> >> Xen may be back to stay as a virtualisation standard if kernel 2.6.38
> >> will
> >> be able to start as dom0 (as 2.6.37 will) and to be able to drive domUs
> >> (missing in upcoming 2.6.37). I think that's really really (!) important
> >> for
> >> XEN future. And: XEN 4.0.x must be as rock stable as 3.4.2 as soon as
> >> possible... i personnally don't think about using 4.0.x for production
> >> systems at this point...
> > 
> > Uh, even 4.0.1? What are the issues you are worried about?
> > 
> > 
> I'm worried about stability, changes in behaviour, changes in kernel /
> parameters, problems with compiling some orig xen kernel, problems running

All of those, except stability, are issues you are going to encounter with
a new kernel...

Can you be more specific about the stability? Have you seen it crash?

> 2.6.18 kernel like above,  dependencies like pvops version .32 for > 4.0.1,
> .31 for < 4.0.1, bugs in 4.0.0, less bugs in 4.0.1, missing features like

PVUSB.. well we would love if somebody volunteered to do the driver.

> pvusb, windows in vhd didn't like the GPLPV drivers (blue screen), signed

Uhh, no idea. I am actually using the Novell GPL drivers in Windows 2000
and they seem to work fine.

> Citrix PV drivers only work with version 5.5, not 5.6, pvops kernel works on
> my hardware with debian pvops xen 4.0.1 kernel, but xen pvops kernel
> compiled according to wiki fc13 page has errors with agpart loading and so
> on..... so i'm waiting for 4.0.3 ;-)

Hm, the agpart loading I thought was fixed. When did you observe this behavior?

Re: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Sander Eikelenboom]

Hello Konrad,

Monday, November 15, 2010, 5:47:40 PM, you wrote:

> On Sat, Nov 13, 2010 at 12:20:58AM +0100, Sander Eikelenboom wrote:
>> Friday, November 12, 2010, 11:19:53 PM, you wrote:
>> 
>> > On Thu, Nov 11, 2010 at 09:16:35AM -0800, Neobiker wrote:
>> >> 
>> >> Hi Konrad,
>> >> 
>> >> many folks need to use the Xenlinux Kernel due to missing features in pvops
>> >> kernel...
>> >> For me (neobiker), it's at a minimum pvusb for my VDR System which uses an
>> >> usb device for DVB-S :-)
>> 
>> > You could also do PCI passthrough of your USB card to the domain..
>> 
>> I'm doing that, and it works, but there are a few pitfalls:
>>     - With onboard controllers it can sometimes be hard to tell which usb port ends up connected to which usb controller. Some motherboards seem to connect them up rather randomly,so you never know which one to passthrough, but it can differ per motherboard.

> Oh I forgot to mention. I've got the USB capture thing and I reproduced the problem you saw
> (with page_alloc failing). The issue was that I forgot to enable these two kernel optins:

> kernel.shmall = 134217728 kernel.shmmax = 134217728

> in the sysctl.conf. Once that was set it worked fine. Haven't put the xHCI controller in box yet
> thought.

With just buying some usb2 controllers instead of the usb3 ones, seemed to have "fixed" my problems for the time being. :-)
There were also some more problems with XHCI, so it just hasn't matured enough.
It's only a pity (E/U/O)hci controllers under linux don't support MSI interrupts, Greg KH seems to have reject a patch for that.
But apart from that it works fine ! (with the 2.6.37-rc1 kernel as domU but i don't use NFS) :-)

So apart from some other things i'm quite happy :-)

--
Sander

Re: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Konrad Rzeszutek Wilk]

> I know about the pci passthrough (of course), but my usb ports are all
> onboard and i have to connect the usb ports to different domains. So if i
> connect the usb controller to vdr domu, i possibly will miss the other usb
> ports for other domains... so pvusb works perfectly for my
> usb-to-multi-domain scenario.

Ok.
> 
> Are you interested in investigating the failure on the "old" kernel with
> another debug log?

Not really.. which is why I was hinting for you to try the PVOPS and PCI pass-through.

> I personally think it would be better to concentrate all developper
> capacitys in finishing the migration to the new pvops kernels and to
> complete all missing features ...  ;-) 
> 
> Btw: Congratlations for the work done untill now !!

Thank you.
> 
> Xen may be back to stay as a virtualisation standard if kernel 2.6.38 will
> be able to start as dom0 (as 2.6.37 will) and to be able to drive domUs
> (missing in upcoming 2.6.37). I think that's really really (!) important for
> XEN future. And: XEN 4.0.x must be as rock stable as 3.4.2 as soon as
> possible... i personnally don't think about using 4.0.x for production
> systems at this point...

Uh, even 4.0.1? What are the issues you are worried about?

Re: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Konrad Rzeszutek Wilk]

On Sat, Nov 13, 2010 at 12:20:58AM +0100, Sander Eikelenboom wrote:
> Friday, November 12, 2010, 11:19:53 PM, you wrote:
> 
> > On Thu, Nov 11, 2010 at 09:16:35AM -0800, Neobiker wrote:
> >> 
> >> Hi Konrad,
> >> 
> >> many folks need to use the Xenlinux Kernel due to missing features in pvops
> >> kernel...
> >> For me (neobiker), it's at a minimum pvusb for my VDR System which uses an
> >> usb device for DVB-S :-)
> 
> > You could also do PCI passthrough of your USB card to the domain..
> 
> I'm doing that, and it works, but there are a few pitfalls:
>     - With onboard controllers it can sometimes be hard to tell which usb port ends up connected to which usb controller. Some motherboards seem to connect them up rather randomly,so you never know which one to passthrough, but it can differ per motherboard.

Oh I forgot to mention. I've got the USB capture thing and I reproduced the problem you saw
(with page_alloc failing). The issue was that I forgot to enable these two kernel optins:

kernel.shmall = 134217728 kernel.shmmax = 134217728

in the sysctl.conf. Once that was set it worked fine. Haven't put the xHCI controller in box yet
thought.

Re: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Sander Eikelenboom]

Friday, November 12, 2010, 11:19:53 PM, you wrote:

> On Thu, Nov 11, 2010 at 09:16:35AM -0800, Neobiker wrote:
>> 
>> Hi Konrad,
>> 
>> many folks need to use the Xenlinux Kernel due to missing features in pvops
>> kernel...
>> For me (neobiker), it's at a minimum pvusb for my VDR System which uses an
>> usb device for DVB-S :-)

> You could also do PCI passthrough of your USB card to the domain..

I'm doing that, and it works, but there are a few pitfalls:
    - With onboard controllers it can sometimes be hard to tell which usb port ends up connected to which usb controller. Some motherboards seem to connect them up rather randomly,so you never know which one to passthrough, but it can differ per motherboard.
    - You don't have that problem when you passthrough a dedicated pci / pci-e controller card per domain.
    - With xhci (usb3) controllers you will most probably encounter problems, it hasn't matured enough yet (i'm back to usb2 again after trying for quite some time.)
    - Found some real cool usb2 controllers with a moschip 9990 chip, these pci-e cards have 4 usb ports, but the bonus is, it has 4 seperate usb controllers.
      That means every port has the full 480Mbps bandwidth, instead of having it shared for all port.Without that you can have only one video device per controller card, because that already saturates more than half the bandwidth most of the time.
    - All usb2 controllers i have had seems to work fine when passed through even without a hardware iommu.

So apart from these pitfalls it now seems to work like a charm !
Another possibility could be usbip which is in the staging tree of the linux kernel.

--

Sander

>> I also use a usb printer on my printer DomU.
>> 
>> Also, i think it is interesting, why the kernel doesn't work at this point,
>> the xen 4.0.2-rc1-pre version starts nowadays on this host - very strange
>> behaviour.

> That might be due to the fact that the drivers (2.6.`8.8) aren't up-to-date
> on your new box. You can also find out more details if pass in the Xen hypervisor
> command line "sync_console console_to_ring " and in the Linux command line:
> "loglevel=8 debug initcall_debug"

> That should show you why and exactly where it fails in the bootup. The thing
> you are seeing isn't actually the failure, it occurs later on but the output
> is buffered and it never reaches Xen hypervisor unless you use those command line
> arguments I mentioned.





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: Re: XEN boot hangs at ACPI: PCI Root Bridge [PCI0] (0000:00)

[Konrad Rzeszutek Wilk]

On Thu, Nov 11, 2010 at 09:16:35AM -0800, Neobiker wrote:
> 
> Hi Konrad,
> 
> many folks need to use the Xenlinux Kernel due to missing features in pvops
> kernel...
> For me (neobiker), it's at a minimum pvusb for my VDR System which uses an
> usb device for DVB-S :-)

You could also do PCI passthrough of your USB card to the domain..

> I also use a usb printer on my printer DomU.
> 
> Also, i think it is interesting, why the kernel doesn't work at this point,
> the xen 4.0.2-rc1-pre version starts nowadays on this host - very strange
> behaviour.

That might be due to the fact that the drivers (2.6.`8.8) aren't up-to-date
on your new box. You can also find out more details if pass in the Xen hypervisor
command line "sync_console console_to_ring " and in the Linux command line:
"loglevel=8 debug initcall_debug"

That should show you why and exactly where it fails in the bootup. The thing
you are seeing isn't actually the failure, it occurs later on but the output
is buffered and it never reaches Xen hypervisor unless you use those command line
arguments I mentioned.