From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pasi =?iso-8859-1?Q?K=E4rkk=E4inen?= Subject: Re: PCI Passthrough of SAS controller not working Date: Wed, 11 Apr 2012 23:47:54 +0300 Message-ID: <20120411204754.GH12984@reaktio.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Aleksandr Tarutin Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On Wed, Apr 11, 2012 at 04:31:46PM -0400, Aleksandr Tarutin wrote: > New findings! > Passthrough worked for a PV domU! > *And*... It worked if i pci-attach the controller to the same PVHVM! > Any suggestions? > The difference is that when you use a PV domU or pci-attach with PVHVM guest the LSI Boot/Option ROM is not executed at all. PV domUs don't execute Option ROMS, because they don't have a BIOS. Also when you pci-attach to a running HVM guest, the LSI BIOS/ROM is not executed. However when you do a full boot of HVM guest with PCI passthru the LSI MPT Boot/Option ROM gets executed by Xen HVM ROMBIOS, and that seems to fail for some reason, so I assume the LSI HBA is left in some weird semi-initialized or broken state, and thus the mpt2sas driver fails to work.. I picked the interesting entries from your logs: Error on the HVM guest VNC console during boot (before grub is executed): MPT BIOS Fault 09h encountered at adapter PCI(00h,05h,00h) Xen dmesg: Xen version 4.1.3-rc1-pre (XEN) HVM1: Writing SMBIOS tables ... (XEN) HVM1: Loading ROMBIOS ... (XEN) HVM1: Loading PCI Option ROM ... (XEN) HVM1: - Manufacturer: LSI Corporation (XEN) HVM1: - Product name: LSI MPI Boot Support (XEN) domctl.c:995:d0 memory_map:remove: gfn=f3000 mfn=fba00 nr_mfns=80 (XEN) HVM1: Invoking ROMBIOS ... dunno if this is related?: (XEN) traps.c:451:d0 Unhandled nmi fault/trap [#2] on VCPU 0 [ec=0000] dom0 lspci: 01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) Subsystem: LSI Logic / Symbios Logic Device 3020 Flags: fast devsel, IRQ 16 I/O ports at e000 [size=256] Memory at fbac0000 (64-bit, non-prefetchable) [size=16K] Memory at fba80000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at fba00000 [disabled] [size=512K] Capabilities: [50] Power Management version 3 Capabilities: [68] Express Endpoint, MSI 00 Capabilities: [d0] Vital Product Data Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [c0] MSI-X: Enable- Count=15 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [138] Power Budgeting Kernel driver in use: pciback Kernel modules: mpt2sas HVM guest kernel dmesg: mpt2sas version 09.101.00.00 loaded scsi0 : Fusion MPT SAS Host alloc irq_desc for 36 on node -1 alloc kstat_irqs on node -1 mpt2sas 0000:00:05.0: PCI INT A -> GSI 36 (level, low) -> IRQ 36 mpt2sas 0000:00:05.0: setting latency timer to 64 mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (2053376 kB) alloc irq_desc for 48 on node -1 alloc kstat_irqs on node -1 mpt2sas 0000:00:05.0: irq 48 for MSI/MSI-X mpt2sas0: PCI-MSI-X enabled: IRQ 48 mpt2sas0: iomem(0x00000000f30e0000), mapped(0xffffc900009e8000), size(16384) mpt2sas0: ioport(0x000000000000c100), size(256) mpt2sas0: Allocated physical memory: size(2688 kB) mpt2sas0: Current Controller Queue Depth(1754), Max Controller Queue Depth(2015) mpt2sas0: Scatter Gather Elements per IO(128) mpt2sas0: _base_event_notification: timeout mf: 07000000 00000000 00000000 00000000 00000000 0f2f7fff fffffffc ffffffff ffffffff 00000000 00000000 mpt2sas0: sending diag reset !! mpt2sas0: diag reset: SUCCESS mpt2sas 0000:00:05.0: PCI INT A disabled mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:7628/_scsih_probe()! Hopefully someone has more ideas about why the LSI MPT Boot/Option ROM fails.. -- Pasi