Re: [Qemu-devel] [PATCH] virtio-pci: enable bus master for old guests

[Greg Kurz <gkurz@linux.vnet.ibm.com>]

On Wed, 10 Sep 2014 13:44:49 +0530
Nikunj A Dadhania <nikunj@linux.vnet.ibm.com> wrote:
> Greg Kurz <gkurz@linux.vnet.ibm.com> writes:
> 
> > On Mon, 8 Sep 2014 19:05:02 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> >> commit cc943c36faa192cd4b32af8fe5edb31894017d35
> >>     pci: Use bus master address space for delivering MSI/MSI-X messages
> >> breaks virtio-net for rhel6.[56] x86 guests because they don't
> >> enable bus mastering for virtio PCI devices
> >> 
> >> Old guests forgot to enable bus mastering, enable it
> >> automatically on DRIVER_OK.
> >> 
> >> Note: we should either back out the original patch from
> >> stable or apply this one on top.
> >> 
> >> Cc: qemu-stable@nongnu.org
> >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >> ---
> >>  hw/virtio/virtio-pci.c | 2 ++
> >>  1 file changed, 2 insertions(+)
> >> 
> >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> >> index ddb5da1..af937d2 100644
> >> --- a/hw/virtio/virtio-pci.c
> >> +++ b/hw/virtio/virtio-pci.c
> >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
> >>          if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
> >>              !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
> >>              proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
> >> +            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
> >> +                                      true);
> >>          }
> >>          break;
> >>      case VIRTIO_MSI_CONFIG_VECTOR:
> >
> > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
> >
> > Michael,
> >
> > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately,
> > this fails for rhel6.5 ppc64 because it is never called... 
> 
> > I did some debugging: it looks like the guest kernel calls the OF
> > quisece call to flush pending DMA and disables bus master on the
> > virtio-blk device (PCI_COMMAND == 0x3).
> 
> Getting confused, above you are talking about virtio-net and here it is
> virtio-blk.
> 

I tried running rhel6.5 (old kernel that doesn't enable bus mastering on
virtio PCI devices), with a virtio-blk based disk and a virtio-net based
NIC for both x86_64 and ppc64. Results are as follow:
- x86_64: boots well but fails to activate network
- ppc64: does not boot because the virtio-blk notification doesn't
         reach the guest

> Anyways, the routines still remains same for both of them.  From SLOF
> during init we set DRIVER_OK, and after using the device during the
> quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then
> a VIRTIO_DEVICE_RESET is done.
> 

I chose to debug by attaching gdb to qemu-system-ppc64 itself. It appears
that SLOF seems to be enabling bus master during init but at some point bus
master gets disabled... unfortunately my SLOF knowledge is limited and I
don't know how exactly what's happening in the guest between SLOF and the
kernel.

> > The guest then continues to boot and hangs... It appears that waiting
> > for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since
> > we need this for MSI to work, I tried the following and it fixes the
> > issue:
> >
> > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> > index af937d2..3d72aa8 100644
> > --- a/hw/virtio/virtio-pci.c
> > +++ b/hw/virtio/virtio-pci.c
> > @@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector)
> >  {
> >      VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d);
> >  
> > -    if (msix_enabled(&proxy->pci_dev))
> > +    if (msix_enabled(&proxy->pci_dev)) {
> > +        if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
> > +            proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
> > +            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
> > +                                      true);
> > +        }
> >          msix_notify(&proxy->pci_dev, vector);
> > -    else {
> > +    } else {
> >          VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
> >          pci_set_irq(&proxy->pci_dev, vdev->isr & 1);
> >      }
> >
> > If this is acceptable, I'll make it a helper and squash it into your patch.
> >
> > Thoughts ?
> >
> > -- 
> > Gregory Kurz                                     kurzgreg@fr.ibm.com
> >                                                  gkurz@linux.vnet.ibm.com
> > Software Engineer @ IBM/Meiosys                  http://www.ibm.com
> > Tel +33 (0)562 165 496
> >
> > "Anarchy is about taking complete responsibility for yourself."
> >         Alan Moore.



-- 
Gregory Kurz                                     kurzgreg@fr.ibm.com
                                                 gkurz@linux.vnet.ibm.com
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.