Re: PCI Passthrough of SAS controller not working

["Pasi Kärkkäinen" <pasik@iki.fi>]

On Wed, Apr 11, 2012 at 04:31:46PM -0400, Aleksandr Tarutin wrote:
>    New findings!
>    Passthrough worked for a PV domU!
>    *And*... It worked if i pci-attach the controller to the same PVHVM!
>    Any suggestions?
>

The difference is that when you use a PV domU or pci-attach with PVHVM guest
the LSI Boot/Option ROM is not executed at all. 

PV domUs don't execute Option ROMS, because they don't have a BIOS.
Also when you pci-attach to a running HVM guest, the LSI BIOS/ROM is not executed.

However when you do a full boot of HVM guest with PCI passthru the 
LSI MPT Boot/Option ROM gets executed by Xen HVM ROMBIOS,
and that seems to fail for some reason, so I assume the LSI HBA
is left in some weird semi-initialized or broken state,
and thus the mpt2sas driver fails to work..

I picked the interesting entries from your logs:


Error on the HVM guest VNC console during boot (before grub is executed):
MPT BIOS Fault 09h encountered at adapter PCI(00h,05h,00h)

Xen dmesg:
Xen version 4.1.3-rc1-pre
(XEN) HVM1: Writing SMBIOS tables ...
(XEN) HVM1: Loading ROMBIOS ...
(XEN) HVM1: Loading PCI Option ROM ...
(XEN) HVM1:  - Manufacturer: LSI Corporation
(XEN) HVM1:  - Product name: LSI MPI Boot Support
(XEN) domctl.c:995:d0 memory_map:remove: gfn=f3000 mfn=fba00 nr_mfns=80
(XEN) HVM1: Invoking ROMBIOS ...

dunno if this is related?:
(XEN) traps.c:451:d0 Unhandled nmi fault/trap [#2] on VCPU 0 [ec=0000]


dom0 lspci:
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
        Subsystem: LSI Logic / Symbios Logic Device 3020
        Flags: fast devsel, IRQ 16
        I/O ports at e000 [size=256]
        Memory at fbac0000 (64-bit, non-prefetchable) [size=16K]
        Memory at fba80000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at fba00000 [disabled] [size=512K]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [d0] Vital Product Data
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [c0] MSI-X: Enable- Count=15 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Kernel driver in use: pciback
        Kernel modules: mpt2sas


HVM guest kernel dmesg:

mpt2sas version 09.101.00.00 loaded
scsi0 : Fusion MPT SAS Host
  alloc irq_desc for 36 on node -1
  alloc kstat_irqs on node -1
mpt2sas 0000:00:05.0: PCI INT A -> GSI 36 (level, low) -> IRQ 36
mpt2sas 0000:00:05.0: setting latency timer to 64
mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (2053376 kB)
  alloc irq_desc for 48 on node -1
  alloc kstat_irqs on node -1
mpt2sas 0000:00:05.0: irq 48 for MSI/MSI-X
mpt2sas0: PCI-MSI-X enabled: IRQ 48
mpt2sas0: iomem(0x00000000f30e0000), mapped(0xffffc900009e8000), size(16384)
mpt2sas0: ioport(0x000000000000c100), size(256)
mpt2sas0: Allocated physical memory: size(2688 kB)
mpt2sas0: Current Controller Queue Depth(1754), Max Controller Queue Depth(2015)
mpt2sas0: Scatter Gather Elements per IO(128)
mpt2sas0: _base_event_notification: timeout
mf:
        07000000 00000000 00000000 00000000 00000000 0f2f7fff fffffffc ffffffff
        ffffffff 00000000 00000000
mpt2sas0: sending diag reset !!
mpt2sas0: diag reset: SUCCESS
mpt2sas 0000:00:05.0: PCI INT A disabled
mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:7628/_scsih_probe()!



Hopefully someone has more ideas about why the LSI MPT Boot/Option ROM fails..


-- Pasi