From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sheng Yang Subject: Re: [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device Date: Wed, 8 Apr 2009 10:26:01 +0800 Message-ID: <200904081026.02816.sheng@linux.intel.com> References: <1237261817-27955-1-git-send-email-sheng@linux.intel.com> <200904071409.10361.sheng@linux.intel.com> <1239122290.6184.86.camel@bling> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Alex Williamson , Avi Kivity , Marcelo Tosatti , Anthony Liguori To: kvm@vger.kernel.org Return-path: Received: from mga10.intel.com ([192.55.52.92]:44387 "EHLO fmsmga102.fm.intel.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752120AbZDHC0F convert rfc822-to-8bit (ORCPT ); Tue, 7 Apr 2009 22:26:05 -0400 In-Reply-To: <1239122290.6184.86.camel@bling> Content-Disposition: inline Sender: kvm-owner@vger.kernel.org List-ID: On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote: > On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote: > > On Saturday 04 April 2009 05:27:43 Alex Williamson wrote: > > > Do we need some disable logic here? If I toggle a bnx2 NIC in a = guest, > > > I get the following when it attempts to come back up: > > > > > > MSI-X entry number is zero! > > > assigned_dev_update_msix_mmio: No such device or address > > > > It seems that driver didn't fill the MMIO with any correct MSIX > > information, or the program fail to intercept it after driver set e= nable > > bit of MSIX. It's strange... (Have it got something to do with PM a= nd > > some EXP feature you mentioned?) > > My guess was that it filled in the MSIX info, but then can't find a f= ree > slot to reload the MSIX data when it tries to re-enable MSIX. I hack= ed > the bnx2 driver to not rely on PM and EXP capabilities for this test,= it > seems to work, but it's possible that I broke something. My host als= o > locks up the second time I try to export this device to a guest, mayb= e a > problem with my bnx2 hacks, MSIX not getting reset, or prototype > hardware. I'll see if I can find another MSIX capable device to expo= rt > to a guest. > > > Could you enable DEVICE_ASSSIGNMENT_DEBUG=3D1 in > > qemu/hw/device-assignment.c and post the output? > > Yup, see below. The error comes after I 'ifdown eth0; ifup eth0' in = the > guest. Note bnx2 appears to only turn on MSIX for SMP systems. Than= ks, > > Alex Seems your "ifdown/ifup" script reload the module? Oh god, I found one = bug=20 after checked the spec: System software reads this field to determine the MSI-X Table Size *N*,= which=20 is encoded as *N-1*. For example, a returned value of =E2=80=9C00000000= 011=E2=80=9D indicates=20 a table size of 4. But it seems still can't explain the problem...(OK, it may affect the g= uest in=20 a unknown way as well...) I would post a fix for it soon. > val=3D0x00000008 len=3D2 assigned_dev_pci_write_config: (4.0): addres= s=3D0052 > val=3D0x00000008 len=3D2 assigned_dev_pci_read_config: (4.0): address= =3D0006 > val=3D0x00000010 len=3D2 assigned_dev_pci_read_config: (4.0): address= =3D0034 > val=3D0x00000040 len=3D1 assigned_dev_pci_read_config: (4.0): address= =3D0040 > val=3D0x00000005 len=3D1 assigned_dev_pci_read_config: (4.0): address= =3D0041 > val=3D0x00000050 len=3D1 assigned_dev_pci_read_config: (4.0): address= =3D0050 > val=3D0x00000011 len=3D1 assigned_dev_pci_read_config: (4.0): address= =3D0052 > val=3D0x00000008 len=3D2 assigned_dev_pci_read_config: (4.0): address= =3D0054 > val=3D0x0000c000 len=3D4 msix_mmio_writel: write to MSI-X entry table= mmio > offset 0x0, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry tab= le > mmio offset 0x4, val 0x0 msix_mmio_writel: write to MSI-X entry table= mmio > offset 0x8, val 0x4191 msix_mmio_writel: write to MSI-X entry table m= mio > offset 0x10, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry ta= ble > mmio offset 0x14, val 0x0 msix_mmio_writel: write to MSI-X entry tabl= e mmio > offset 0x18, val 0x4199 msix_mmio_writel: write to MSI-X entry table = mmio > offset 0x20, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry ta= ble > mmio offset 0x24, val 0x0 msix_mmio_writel: write to MSI-X entry tabl= e mmio > offset 0x28, val 0x41a1 msix_mmio_writel: write to MSI-X entry table = mmio > offset 0x30, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry ta= ble > mmio offset 0x34, val 0x0 msix_mmio_writel: write to MSI-X entry tabl= e mmio > offset 0x38, val 0x41a9 msix_mmio_writel: write to MSI-X entry table = mmio > offset 0x40, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry ta= ble > mmio offset 0x44, val 0x0 msix_mmio_writel: write to MSI-X entry tabl= e mmio > offset 0x48, val 0x41b1 msix_mmio_writel: write to MSI-X entry table = mmio > offset 0x50, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry ta= ble > mmio offset 0x54, val 0x0 msix_mmio_writel: write to MSI-X entry tabl= e mmio > offset 0x58, val 0x41b9 msix_mmio_writel: write to MSI-X entry table = mmio > offset 0x60, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry ta= ble > mmio offset 0x64, val 0x0 msix_mmio_writel: write to MSI-X entry tabl= e mmio > offset 0x68, val 0x41c1 msix_mmio_writel: write to MSI-X entry table = mmio > offset 0x70, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry ta= ble > mmio offset 0x74, val 0x0 msix_mmio_writel: write to MSI-X entry tabl= e mmio > offset 0x78, val 0x41c9 msix_mmio_writel: write to MSI-X entry table = mmio > offset 0x80, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry ta= ble > mmio offset 0x84, val 0x0 msix_mmio_writel: write to MSI-X entry tabl= e mmio > offset 0x88, val 0x41d1 assigned_dev_pci_read_config: (4.0): address=3D= 0004 > val=3D0x00000046 len=3D2 assigned_dev_pci_write_config: (4.0): addres= s=3D0004 > val=3D0x00000446 len=3D2 assigned_dev_pci_write_config: NON BAR (4.0)= : The writing to MMIO have been intercepted, but code fail to count it?=20 Strange... Could you try this debug? diff --git a/qemu/hw/device-assignment.c b/qemu/hw/device-assignment.c index 09e54ae..ba31bed 100644 --- a/qemu/hw/device-assignment.c +++ b/qemu/hw/device-assignment.c @@ -45,7 +45,7 @@ #define IORESOURCE_DMA 0x00000800 #define IORESOURCE_PREFETCH 0x00001000 /* No side effects */ -/* #define DEVICE_ASSIGNMENT_DEBUG 1 */ +#define DEVICE_ASSIGNMENT_DEBUG 1 #ifdef DEVICE_ASSIGNMENT_DEBUG #define DEBUG(fmt, ...) \ @@ -816,9 +816,13 @@ static int assigned_dev_update_msix_mmio(PCIDevice= =20 *pci_dev) else pos =3D pci_dev->cap.start; + fprintf(stderr, "the MSIX capabilty position is 0x%lx\n", pos); + entries_max_nr =3D pci_dev->config[pos + 2]; entries_max_nr &=3D PCI_MSIX_TABSIZE; + fprintf(stderr, "the MSIX entries_max_nr is 0x%lx\n", entries_max_= nr); + /* Get the usable entry number for allocating */ for (i =3D 0; i < entries_max_nr; i++) { memcpy(&msg_ctrl, va + i * 16 + 12, 4); (Further more, you can check the content of msg_ctrl when code try to c= ount=20 them). And also please try the fix for MSI-X table size. Thanks! --=20 regards Yang, Sheng