Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Mark Adams <mark@campbell-lange.net>]

On Wed, Nov 24, 2010 at 03:28:43PM -0500, Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 24, 2010 at 05:59:26PM +0000, Mark Adams wrote:
> > > > > > > >>>> 
> > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > 
> > > > > > Looking briefly at the code it means that somebody enabled the MSI
> > > > > > already on the device and did not disable them. But I wonder how
> > > > > > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > > > > > or pciback.hide (for older kernels) to "hide" the devices away from the
> > > > > > Linux Dom0 kernel?
> > 
> > I've got a test setup in place now, and am trying to reproduce this.
> > I've not connected up serial as yet, but can see the following logs in
> > the qemu-dm log file when I get the "MSI is already in use" errors
> > above. Note also that this error -always- shows for the first specified
> > device in the pci= field, and not the 2nd.
> > 

In my new test setup, I have seen some strange behaviour. 1 of the HVM's
(with identical config in dom0 and domU) suddenly would not allow the
igb driver to be loaded in domU, even though the device was visible in
lspci. Shutting the machine down, removing the power cord, waiting 5
seconds then plugging it in again corrected that issue - Is this
possibly a motherboard bug? I have also disabled the SR-IOV
functionality in the BIOS incase this is causing any issues.

In addition, to try to correct the MSI issue noted above, I have changed
my pci= line to the following:

pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]

This has stopped the "already in use on device" log, and the devices
appear to show correctly in the domU. Is it safe to disable
msitranslate? as I understand it, its for allowing multifunction devices
to be seen as such in domU. Is that correct?

I haven't been able to reproduce the dropped raid issue yet, but I am
awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
this due to their very high interrupt usage (2000 per second).

In the mean time, I can see the following in the qemu-dm logs now with
the msitranslate=0 enabled. Is it anything to worry about?

pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.

> 
> Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> machine is toast. I mentioned in the previous email the key sequences - look on Google
> on how to pass in SysRQ if you are using a serial concentrator.

I will do this when I can get the machine to crash.

Best Regards,
Mark