Re: Hard and silent lock up since linux 3.14 with PCIe pass through (vfio)

[Alex Williamson <alex.williamson@redhat.com>]

On Tue, 2014-10-21 at 15:06 -0600, Alex Williamson wrote:
> Hi Andreas,
> 
> On Fri, 2014-10-17 at 03:04 +0200, Andreas Hartmann wrote:
> > Hello Alex,
> > 
> > Alex Williamson wrote:
> > > Hi Andreas,
> > [...]
> > > Sorry for the breakage.  Is it possible to run lspci on the device in a
> > > loop from the host and capture whether we're failing to restore some of
> > > the VC bits to their previous state? 
> > 
> > > Does the problem also occur if you
> > > unbind from host driver,
> > 
> > The machine is booted w/ blacklisted ath9k. Then, the device is bound to
> > vfio:
> > 
> > echo "168c 0030" > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 0000:03:00.0 > /sys/bus/pci/devices/0000:03:00.0/driver/unbind
> > echo 0000:03:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
> > 
> > afterwards the VM is started -> hang.
> > 
> > W/o starting th VM, I can bind it to vfio and unbind it from vfio w/o
> > any problem.
> > 
> > > echo 1 > reset in pci-sysfs,
> > 
> > echo 1 > /sys/bus/pci/devices/0000:03:00.0 works w/o any problem while
> > bound to vfio. Even after unbinding from vfio and rebinding to vfio
> > again ... .
> > 
> > > and re-bind to the
> > 
> > Do you mean loading ath9k in host system after unbinding from vfio? If
> > yes: Works w/o any problem. It's even possible to reset it or do a
> > ifconfig wlan0 up, ifconfig wlan0 down, rmmod ath9k, bind it to vfio
> > again and reset it, ....
> > 
> > Looks like the hang only is triggered by qemu-system_x86_64 on startup
> > the VM.

Also, this might be because QEMU since 1.7 will favor doing a bus reset
for a device over PM reset while the sysfs reset interface will only do
a bus reset if there are no other methods available and there are no
other devices on the bus.  Can you reproduce the hang using the sysfs
reset interface without QEMU if you modify the kernel like this:

--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3308,15 +3308,15 @@ static int __pci_dev_reset(struct pci_dev *dev, int prob
        if (rc != -ENOTTY)
                goto done;
 
-       rc = pci_pm_reset(dev, probe);
+       rc = pci_dev_reset_slot_function(dev, probe);
        if (rc != -ENOTTY)
                goto done;
 
-       rc = pci_dev_reset_slot_function(dev, probe);
+       rc = pci_parent_bus_reset(dev, probe);
        if (rc != -ENOTTY)
                goto done;
 
-       rc = pci_parent_bus_reset(dev, probe);
+       rc = pci_pm_reset(dev, probe);
 done:
        return rc;
 }



> > > host?  I'll also try to reproduce on my 990fx system, but I won't be
> > > able to do that until next week due to travel.  Thanks,
> 
> Could you send me the lspci -vvvxxxx for the device and parent root
> port?  Thanks,
> 
> Alex
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html