[Qemu-devel] vfio in the guest: no available reset mechanism

[Qemu-devel] vfio in the guest: no available reset mechanism

[Le Tan]

Hi,
I am testing vfio in L1 with my VT-d emulation project. I assigned one
of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
in L1, it complains that:
qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
available reset mechanism.

Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
look into this and found that:
val = ahci_ctrl_readl(ctrl, HOST_CTL);
ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
should be cleared by default. So maybe L1 didn't reset the device
before assigning it to L2?
Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(

So can anyone give me some advice? About the state of PCI device or
bus-level reset?

Here is the detail of the environment and the way I did the vfio.
1. lspci in L1 said:
00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
(ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
Interface Controller [8086:2918] (rev 02)
00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
(ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
Controller [8086:2930] (rev 02)
2. Unbind 00:03.0 and do vfio:
modprobe -r vfio_iommu_type1
modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
modprobe vfio-pci
echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
3. run L2 with "-device vfio-pci,host=00:03.0"

Any help is appreciated! Thanks very much!

Regards,
Le

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Michael S. Tsirkin]

On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
> Hi,
> I am testing vfio in L1 with my VT-d emulation project. I assigned one
> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
> in L1, it complains that:
> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> available reset mechanism.
> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> available reset mechanism.
> 
> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
> look into this and found that:
> val = ahci_ctrl_readl(ctrl, HOST_CTL);
> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
> should be cleared by default. So maybe L1 didn't reset the device
> before assigning it to L2?
> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
> 
> So can anyone give me some advice? About the state of PCI device or
> bus-level reset?
> 
> Here is the detail of the environment and the way I did the vfio.
> 1. lspci in L1 said:
> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
> Interface Controller [8086:2918] (rev 02)
> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
> Controller [8086:2930] (rev 02)
> 2. Unbind 00:03.0 and do vfio:
> modprobe -r vfio_iommu_type1
> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
> modprobe vfio-pci
> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
> 3. run L2 with "-device vfio-pci,host=00:03.0"
> 
> Any help is appreciated! Thanks very much!
> 
> Regards,
> Le

Clearly, bus level reset can't work for the root bus :)

-- 
MST

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Le Tan]

Hi Michael,

2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
> On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
>> Hi,
>> I am testing vfio in L1 with my VT-d emulation project. I assigned one
>> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
>> in L1, it complains that:
>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>> available reset mechanism.
>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>> available reset mechanism.
>>
>> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
>> look into this and found that:
>> val = ahci_ctrl_readl(ctrl, HOST_CTL);
>> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
>> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
>> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
>> should be cleared by default. So maybe L1 didn't reset the device
>> before assigning it to L2?
>> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
>>
>> So can anyone give me some advice? About the state of PCI device or
>> bus-level reset?
>>
>> Here is the detail of the environment and the way I did the vfio.
>> 1. lspci in L1 said:
>> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
>> Interface Controller [8086:2918] (rev 02)
>> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
>> Controller [8086:2930] (rev 02)
>> 2. Unbind 00:03.0 and do vfio:
>> modprobe -r vfio_iommu_type1
>> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
>> modprobe vfio-pci
>> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
>> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
>> 3. run L2 with "-device vfio-pci,host=00:03.0"
>>
>> Any help is appreciated! Thanks very much!
>>
>> Regards,
>> Le
>
> Clearly, bus level reset can't work for the root bus :)

Thanks very much!
I test the vfio with a second-bus ahci controller and it didn't
complain about the lack of reset mechanism. :) And the return val of
HOST_CTL is normal now (the same as emulated ahci controller).
However, it still paused when the BIOS tried to write to the HOST_CTL.
Do you have any idea?
And we should just test vfio and legacy pci-assignment with second bus
devices, not considering the root-bus devices?
Thanks again!

Regards,
Le

> --
> MST

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Alex Williamson]

On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
> Hi Michael,
> 
> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
> > On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
> >> Hi,
> >> I am testing vfio in L1 with my VT-d emulation project. I assigned one
> >> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
> >> in L1, it complains that:
> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> >> available reset mechanism.
> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> >> available reset mechanism.
> >>
> >> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
> >> look into this and found that:
> >> val = ahci_ctrl_readl(ctrl, HOST_CTL);
> >> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
> >> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
> >> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
> >> should be cleared by default. So maybe L1 didn't reset the device
> >> before assigning it to L2?
> >> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
> >>
> >> So can anyone give me some advice? About the state of PCI device or
> >> bus-level reset?
> >>
> >> Here is the detail of the environment and the way I did the vfio.
> >> 1. lspci in L1 said:
> >> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> >> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
> >> Interface Controller [8086:2918] (rev 02)
> >> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> >> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
> >> Controller [8086:2930] (rev 02)
> >> 2. Unbind 00:03.0 and do vfio:
> >> modprobe -r vfio_iommu_type1
> >> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
> >> modprobe vfio-pci
> >> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
> >> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
> >> 3. run L2 with "-device vfio-pci,host=00:03.0"
> >>
> >> Any help is appreciated! Thanks very much!
> >>
> >> Regards,
> >> Le
> >
> > Clearly, bus level reset can't work for the root bus :)
> 
> Thanks very much!
> I test the vfio with a second-bus ahci controller and it didn't
> complain about the lack of reset mechanism. :) And the return val of
> HOST_CTL is normal now (the same as emulated ahci controller).
> However, it still paused when the BIOS tried to write to the HOST_CTL.
> Do you have any idea?
> And we should just test vfio and legacy pci-assignment with second bus
> devices, not considering the root-bus devices?

AHCI seems like a poor choice of device for this work, they typically
don't support any kind of reset and they can be troublesome even for the
L1 assignment.  You really want something with FLR support so that both
the host and L1 guest can potentially reset the device.  That said, you
may still run into bugs with a L1 guest directed FLR.  Thanks,

Alex

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Michael S. Tsirkin]

On Wed, Jul 30, 2014 at 10:16:32PM +0800, Le Tan wrote:
> Hi Michael,
> 
> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
> > On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
> >> Hi,
> >> I am testing vfio in L1 with my VT-d emulation project. I assigned one
> >> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
> >> in L1, it complains that:
> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> >> available reset mechanism.
> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> >> available reset mechanism.
> >>
> >> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
> >> look into this and found that:
> >> val = ahci_ctrl_readl(ctrl, HOST_CTL);
> >> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
> >> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
> >> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
> >> should be cleared by default. So maybe L1 didn't reset the device
> >> before assigning it to L2?
> >> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
> >>
> >> So can anyone give me some advice? About the state of PCI device or
> >> bus-level reset?
> >>
> >> Here is the detail of the environment and the way I did the vfio.
> >> 1. lspci in L1 said:
> >> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> >> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
> >> Interface Controller [8086:2918] (rev 02)
> >> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> >> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
> >> Controller [8086:2930] (rev 02)
> >> 2. Unbind 00:03.0 and do vfio:
> >> modprobe -r vfio_iommu_type1
> >> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
> >> modprobe vfio-pci
> >> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
> >> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
> >> 3. run L2 with "-device vfio-pci,host=00:03.0"
> >>
> >> Any help is appreciated! Thanks very much!
> >>
> >> Regards,
> >> Le
> >
> > Clearly, bus level reset can't work for the root bus :)
> 
> Thanks very much!
> I test the vfio with a second-bus ahci controller and it didn't
> complain about the lack of reset mechanism. :) And the return val of
> HOST_CTL is normal now (the same as emulated ahci controller).
> However, it still paused when the BIOS tried to write to the HOST_CTL.
> Do you have any idea?
> And we should just test vfio and legacy pci-assignment with second bus
> devices, not considering the root-bus devices?
> Thanks again!
> 
> Regards,
> Le

If you want it to work for root bus, you need to implement
a device/function level reset mechanism, e.g. using PM or FLR.

> > --
> > MST

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Le Tan]

Hi Alex,

2014-07-30 22:46 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
> On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
>> Hi Michael,
>>
>> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
>> > On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
>> >> Hi,
>> >> I am testing vfio in L1 with my VT-d emulation project. I assigned one
>> >> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
>> >> in L1, it complains that:
>> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>> >> available reset mechanism.
>> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>> >> available reset mechanism.
>> >>
>> >> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
>> >> look into this and found that:
>> >> val = ahci_ctrl_readl(ctrl, HOST_CTL);
>> >> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
>> >> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
>> >> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
>> >> should be cleared by default. So maybe L1 didn't reset the device
>> >> before assigning it to L2?
>> >> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
>> >>
>> >> So can anyone give me some advice? About the state of PCI device or
>> >> bus-level reset?
>> >>
>> >> Here is the detail of the environment and the way I did the vfio.
>> >> 1. lspci in L1 said:
>> >> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>> >> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
>> >> Interface Controller [8086:2918] (rev 02)
>> >> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>> >> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
>> >> Controller [8086:2930] (rev 02)
>> >> 2. Unbind 00:03.0 and do vfio:
>> >> modprobe -r vfio_iommu_type1
>> >> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
>> >> modprobe vfio-pci
>> >> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
>> >> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
>> >> 3. run L2 with "-device vfio-pci,host=00:03.0"
>> >>
>> >> Any help is appreciated! Thanks very much!
>> >>
>> >> Regards,
>> >> Le
>> >
>> > Clearly, bus level reset can't work for the root bus :)
>>
>> Thanks very much!
>> I test the vfio with a second-bus ahci controller and it didn't
>> complain about the lack of reset mechanism. :) And the return val of
>> HOST_CTL is normal now (the same as emulated ahci controller).
>> However, it still paused when the BIOS tried to write to the HOST_CTL.
>> Do you have any idea?
>> And we should just test vfio and legacy pci-assignment with second bus
>> devices, not considering the root-bus devices?
>
> AHCI seems like a poor choice of device for this work, they typically
> don't support any kind of reset and they can be troublesome even for the
> L1 assignment.  You really want something with FLR support so that both
> the host and L1 guest can potentially reset the device.  That said, you
> may still run into bugs with a L1 guest directed FLR.  Thanks,
>
> Alex
>

So what device do you think is suitable for the pci-assign test? e1000?
I just tested it with sound card ac97 and USB controller. But I don't
know how to attach them to a pcie-to-pci bridge, so maybe they weren't
reset before being assigned to L2. But it seems that they can work.
1. With the sound card, I assigned it to L2 via both vfio and legacy
pci-assign and I could hear the music played in L2 from host's
speakers. Of course, the vfio also complained about the lack of reset
mechanism.
2. With the USB controller, I used "-usb -usbdevice disk:file" to
attach a USB disk to L1. But there were 4 related devices in L1, so I
didn't know what should be assigned to L2. the info qtree was like
this:
 dev: ich9-usb-uhci3, id ""
        masterbus = "usb-bus.0"
        firstport = 4 (0x4)
        bandwidth = 1280 (0x500)
        maxframes = 128 (0x80)
        addr = 1d.2
dev: ich9-usb-uhci2, id ""
        masterbus = "usb-bus.0"
        firstport = 2 (0x2)
        bandwidth = 1280 (0x500)
        maxframes = 128 (0x80)
        addr = 1d.1
 dev: ich9-usb-uhci1, id ""
        masterbus = "usb-bus.0"
        firstport = 0 (0x0)
        bandwidth = 1280 (0x500)
        maxframes = 128 (0x80)
        addr = 1d.0
dev: ich9-usb-ehci1, id ""
        maxframes = 128 (0x80)
        addr = 1d.7
         bus: usb-bus.0
          type usb-bus
          dev: usb-storage, id ""
            drive = ""
            logical_block_size = 512 (0x200)
            physical_block_size = 512 (0x200)
            min_io_size = 0 (0x0)
            opt_io_size = 0 (0x0)
            bootindex = -1 (0xffffffffffffffff)
            discard_granularity = 4294967295 (0xffffffff)
            removable = false
            port = ""
            serial = ""
            full-path = true
            msos-desc = true
            addr 0.1, port 1, speed 480, name QEMU USB MSD, attached
            bus: scsi.0
              type SCSI
              dev: scsi-disk, id ""
                drive = "usb0"
                logical_block_size = 512 (0x200)
                physical_block_size = 512 (0x200)
                min_io_size = 0 (0x0)
                opt_io_size = 0 (0x0)
Then I tried to unbind and bind both 1d.7 and 1d.0. Then I assigned
1d.0 to L2. L2 could boot smoothly and I could mount and access the
USB disk.
So what do you think?

Thanks very much.

Le

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Alex Williamson]

On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote:
> Hi Alex,
> 
> 2014-07-30 22:46 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
> > On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
> >> Hi Michael,
> >>
> >> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
> >> > On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
> >> >> Hi,
> >> >> I am testing vfio in L1 with my VT-d emulation project. I assigned one
> >> >> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
> >> >> in L1, it complains that:
> >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> >> >> available reset mechanism.
> >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> >> >> available reset mechanism.
> >> >>
> >> >> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
> >> >> look into this and found that:
> >> >> val = ahci_ctrl_readl(ctrl, HOST_CTL);
> >> >> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
> >> >> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
> >> >> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
> >> >> should be cleared by default. So maybe L1 didn't reset the device
> >> >> before assigning it to L2?
> >> >> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
> >> >>
> >> >> So can anyone give me some advice? About the state of PCI device or
> >> >> bus-level reset?
> >> >>
> >> >> Here is the detail of the environment and the way I did the vfio.
> >> >> 1. lspci in L1 said:
> >> >> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> >> >> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
> >> >> Interface Controller [8086:2918] (rev 02)
> >> >> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> >> >> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
> >> >> Controller [8086:2930] (rev 02)
> >> >> 2. Unbind 00:03.0 and do vfio:
> >> >> modprobe -r vfio_iommu_type1
> >> >> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
> >> >> modprobe vfio-pci
> >> >> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
> >> >> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
> >> >> 3. run L2 with "-device vfio-pci,host=00:03.0"
> >> >>
> >> >> Any help is appreciated! Thanks very much!
> >> >>
> >> >> Regards,
> >> >> Le
> >> >
> >> > Clearly, bus level reset can't work for the root bus :)
> >>
> >> Thanks very much!
> >> I test the vfio with a second-bus ahci controller and it didn't
> >> complain about the lack of reset mechanism. :) And the return val of
> >> HOST_CTL is normal now (the same as emulated ahci controller).
> >> However, it still paused when the BIOS tried to write to the HOST_CTL.
> >> Do you have any idea?
> >> And we should just test vfio and legacy pci-assignment with second bus
> >> devices, not considering the root-bus devices?
> >
> > AHCI seems like a poor choice of device for this work, they typically
> > don't support any kind of reset and they can be troublesome even for the
> > L1 assignment.  You really want something with FLR support so that both
> > the host and L1 guest can potentially reset the device.  That said, you
> > may still run into bugs with a L1 guest directed FLR.  Thanks,
> >
> > Alex
> >
> 
> So what device do you think is suitable for the pci-assign test? e1000?
> I just tested it with sound card ac97 and USB controller. But I don't
> know how to attach them to a pcie-to-pci bridge, so maybe they weren't
> reset before being assigned to L2. But it seems that they can work.
> 1. With the sound card, I assigned it to L2 via both vfio and legacy
> pci-assign and I could hear the music played in L2 from host's
> speakers. Of course, the vfio also complained about the lack of reset
> mechanism.

First off, maybe I'm a little confused, are you trying to assigned
emulated devices for the L1 guest to the L2 guest or are these assigned
devices to the L1 guest that you want to re-assign to L2?  AFAIK, we
don't have any emulated devices that support reset, but it wouldn't be
that hard to add a PM capability to one that advertises a soft reset on
D3hot->D0 transition and calls the QEMU driver reset function when that
occurs.  This would provide the most flexibility.

The other choice is to create a topology where an individual device can
be reset via a bus reset, so putting a single device behind a bridge,
root port, or downstream switch port.

Otherwise, the emulated e1000 is probably a reasonable choice, network
drivers generally seem to be pretty good about accepting a device in a
running state since it might have been used as a boot device.

If you're looking at re-assigning an L1 assigned device to an L2 guest,
I'm not sure how that's supposed to work.  The VT-d emulation would need
to somehow push L2 physical mappings to the host, but the L1 guest
doesn't know the true isolation constraints of the host devices and
whether they can actually be mapped separately from other devices.
Since we try to use as few IOMMU domains as possible, you'd end up
modifying translations for all devices assigned to the L1 guest.

> 2. With the USB controller, I used "-usb -usbdevice disk:file" to
> attach a USB disk to L1. But there were 4 related devices in L1, so I
> didn't know what should be assigned to L2. the info qtree was like
> this:
>  dev: ich9-usb-uhci3, id ""
>         masterbus = "usb-bus.0"
>         firstport = 4 (0x4)
>         bandwidth = 1280 (0x500)
>         maxframes = 128 (0x80)
>         addr = 1d.2
> dev: ich9-usb-uhci2, id ""
>         masterbus = "usb-bus.0"
>         firstport = 2 (0x2)
>         bandwidth = 1280 (0x500)
>         maxframes = 128 (0x80)
>         addr = 1d.1
>  dev: ich9-usb-uhci1, id ""
>         masterbus = "usb-bus.0"
>         firstport = 0 (0x0)
>         bandwidth = 1280 (0x500)
>         maxframes = 128 (0x80)
>         addr = 1d.0
> dev: ich9-usb-ehci1, id ""
>         maxframes = 128 (0x80)
>         addr = 1d.7
>          bus: usb-bus.0
>           type usb-bus
>           dev: usb-storage, id ""
>             drive = ""
>             logical_block_size = 512 (0x200)
>             physical_block_size = 512 (0x200)
>             min_io_size = 0 (0x0)
>             opt_io_size = 0 (0x0)
>             bootindex = -1 (0xffffffffffffffff)
>             discard_granularity = 4294967295 (0xffffffff)
>             removable = false
>             port = ""
>             serial = ""
>             full-path = true
>             msos-desc = true
>             addr 0.1, port 1, speed 480, name QEMU USB MSD, attached
>             bus: scsi.0
>               type SCSI
>               dev: scsi-disk, id ""
>                 drive = "usb0"
>                 logical_block_size = 512 (0x200)
>                 physical_block_size = 512 (0x200)
>                 min_io_size = 0 (0x0)
>                 opt_io_size = 0 (0x0)
> Then I tried to unbind and bind both 1d.7 and 1d.0. Then I assigned
> 1d.0 to L2. L2 could boot smoothly and I could mount and access the
> USB disk.
> So what do you think?

Sounds like a reasonable test of L1 emulated assigned to L2.  Thanks,

Alex

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Alex Williamson]

On Fri, 2014-08-01 at 09:25 -0600, Alex Williamson wrote:
> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote:
> > Hi Alex,
> > 
> > 2014-07-30 22:46 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
> > > On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
> > >> Hi Michael,
> > >>
> > >> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
> > >> > On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
> > >> >> Hi,
> > >> >> I am testing vfio in L1 with my VT-d emulation project. I assigned one
> > >> >> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
> > >> >> in L1, it complains that:
> > >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> > >> >> available reset mechanism.
> > >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> > >> >> available reset mechanism.
> > >> >>
> > >> >> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
> > >> >> look into this and found that:
> > >> >> val = ahci_ctrl_readl(ctrl, HOST_CTL);
> > >> >> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
> > >> >> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
> > >> >> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
> > >> >> should be cleared by default. So maybe L1 didn't reset the device
> > >> >> before assigning it to L2?
> > >> >> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
> > >> >>
> > >> >> So can anyone give me some advice? About the state of PCI device or
> > >> >> bus-level reset?
> > >> >>
> > >> >> Here is the detail of the environment and the way I did the vfio.
> > >> >> 1. lspci in L1 said:
> > >> >> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> > >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> > >> >> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
> > >> >> Interface Controller [8086:2918] (rev 02)
> > >> >> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> > >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> > >> >> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
> > >> >> Controller [8086:2930] (rev 02)
> > >> >> 2. Unbind 00:03.0 and do vfio:
> > >> >> modprobe -r vfio_iommu_type1
> > >> >> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
> > >> >> modprobe vfio-pci
> > >> >> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
> > >> >> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
> > >> >> 3. run L2 with "-device vfio-pci,host=00:03.0"
> > >> >>
> > >> >> Any help is appreciated! Thanks very much!
> > >> >>
> > >> >> Regards,
> > >> >> Le
> > >> >
> > >> > Clearly, bus level reset can't work for the root bus :)
> > >>
> > >> Thanks very much!
> > >> I test the vfio with a second-bus ahci controller and it didn't
> > >> complain about the lack of reset mechanism. :) And the return val of
> > >> HOST_CTL is normal now (the same as emulated ahci controller).
> > >> However, it still paused when the BIOS tried to write to the HOST_CTL.
> > >> Do you have any idea?
> > >> And we should just test vfio and legacy pci-assignment with second bus
> > >> devices, not considering the root-bus devices?
> > >
> > > AHCI seems like a poor choice of device for this work, they typically
> > > don't support any kind of reset and they can be troublesome even for the
> > > L1 assignment.  You really want something with FLR support so that both
> > > the host and L1 guest can potentially reset the device.  That said, you
> > > may still run into bugs with a L1 guest directed FLR.  Thanks,
> > >
> > > Alex
> > >
> > 
> > So what device do you think is suitable for the pci-assign test? e1000?
> > I just tested it with sound card ac97 and USB controller. But I don't
> > know how to attach them to a pcie-to-pci bridge, so maybe they weren't
> > reset before being assigned to L2. But it seems that they can work.
> > 1. With the sound card, I assigned it to L2 via both vfio and legacy
> > pci-assign and I could hear the music played in L2 from host's
> > speakers. Of course, the vfio also complained about the lack of reset
> > mechanism.
> 
> First off, maybe I'm a little confused, are you trying to assigned
> emulated devices for the L1 guest to the L2 guest or are these assigned
> devices to the L1 guest that you want to re-assign to L2?  AFAIK, we
> don't have any emulated devices that support reset, but it wouldn't be
> that hard to add a PM capability to one that advertises a soft reset on
> D3hot->D0 transition and calls the QEMU driver reset function when that
> occurs.  This would provide the most flexibility.

Or even better might be to add an Advanced Features capability to each
device so it can do an explicit FLR.  We don't tend to trust PM reset as
being reliably on all devices.

> The other choice is to create a topology where an individual device can
> be reset via a bus reset, so putting a single device behind a bridge,
> root port, or downstream switch port.
> 
> Otherwise, the emulated e1000 is probably a reasonable choice, network
> drivers generally seem to be pretty good about accepting a device in a
> running state since it might have been used as a boot device.
> 
> If you're looking at re-assigning an L1 assigned device to an L2 guest,
> I'm not sure how that's supposed to work.  The VT-d emulation would need
> to somehow push L2 physical mappings to the host, but the L1 guest
> doesn't know the true isolation constraints of the host devices and
> whether they can actually be mapped separately from other devices.
> Since we try to use as few IOMMU domains as possible, you'd end up
> modifying translations for all devices assigned to the L1 guest.
> 
> > 2. With the USB controller, I used "-usb -usbdevice disk:file" to
> > attach a USB disk to L1. But there were 4 related devices in L1, so I
> > didn't know what should be assigned to L2. the info qtree was like
> > this:
> >  dev: ich9-usb-uhci3, id ""
> >         masterbus = "usb-bus.0"
> >         firstport = 4 (0x4)
> >         bandwidth = 1280 (0x500)
> >         maxframes = 128 (0x80)
> >         addr = 1d.2
> > dev: ich9-usb-uhci2, id ""
> >         masterbus = "usb-bus.0"
> >         firstport = 2 (0x2)
> >         bandwidth = 1280 (0x500)
> >         maxframes = 128 (0x80)
> >         addr = 1d.1
> >  dev: ich9-usb-uhci1, id ""
> >         masterbus = "usb-bus.0"
> >         firstport = 0 (0x0)
> >         bandwidth = 1280 (0x500)
> >         maxframes = 128 (0x80)
> >         addr = 1d.0
> > dev: ich9-usb-ehci1, id ""
> >         maxframes = 128 (0x80)
> >         addr = 1d.7
> >          bus: usb-bus.0
> >           type usb-bus
> >           dev: usb-storage, id ""
> >             drive = ""
> >             logical_block_size = 512 (0x200)
> >             physical_block_size = 512 (0x200)
> >             min_io_size = 0 (0x0)
> >             opt_io_size = 0 (0x0)
> >             bootindex = -1 (0xffffffffffffffff)
> >             discard_granularity = 4294967295 (0xffffffff)
> >             removable = false
> >             port = ""
> >             serial = ""
> >             full-path = true
> >             msos-desc = true
> >             addr 0.1, port 1, speed 480, name QEMU USB MSD, attached
> >             bus: scsi.0
> >               type SCSI
> >               dev: scsi-disk, id ""
> >                 drive = "usb0"
> >                 logical_block_size = 512 (0x200)
> >                 physical_block_size = 512 (0x200)
> >                 min_io_size = 0 (0x0)
> >                 opt_io_size = 0 (0x0)
> > Then I tried to unbind and bind both 1d.7 and 1d.0. Then I assigned
> > 1d.0 to L2. L2 could boot smoothly and I could mount and access the
> > USB disk.
> > So what do you think?
> 
> Sounds like a reasonable test of L1 emulated assigned to L2.  Thanks,
> 
> Alex
> 
> 
> 

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 5295 bytes --]

On 2014-08-01 18:35, Alex Williamson wrote:
> On Fri, 2014-08-01 at 09:25 -0600, Alex Williamson wrote:
>> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote:
>>> Hi Alex,
>>>
>>> 2014-07-30 22:46 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
>>>> On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
>>>>> Hi Michael,
>>>>>
>>>>> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
>>>>>> On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
>>>>>>> Hi,
>>>>>>> I am testing vfio in L1 with my VT-d emulation project. I assigned one
>>>>>>> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
>>>>>>> in L1, it complains that:
>>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>>>>>>> available reset mechanism.
>>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>>>>>>> available reset mechanism.
>>>>>>>
>>>>>>> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
>>>>>>> look into this and found that:
>>>>>>> val = ahci_ctrl_readl(ctrl, HOST_CTL);
>>>>>>> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
>>>>>>> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
>>>>>>> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
>>>>>>> should be cleared by default. So maybe L1 didn't reset the device
>>>>>>> before assigning it to L2?
>>>>>>> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
>>>>>>>
>>>>>>> So can anyone give me some advice? About the state of PCI device or
>>>>>>> bus-level reset?
>>>>>>>
>>>>>>> Here is the detail of the environment and the way I did the vfio.
>>>>>>> 1. lspci in L1 said:
>>>>>>> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>>>>>>> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
>>>>>>> Interface Controller [8086:2918] (rev 02)
>>>>>>> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>>>>>>> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
>>>>>>> Controller [8086:2930] (rev 02)
>>>>>>> 2. Unbind 00:03.0 and do vfio:
>>>>>>> modprobe -r vfio_iommu_type1
>>>>>>> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
>>>>>>> modprobe vfio-pci
>>>>>>> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
>>>>>>> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
>>>>>>> 3. run L2 with "-device vfio-pci,host=00:03.0"
>>>>>>>
>>>>>>> Any help is appreciated! Thanks very much!
>>>>>>>
>>>>>>> Regards,
>>>>>>> Le
>>>>>>
>>>>>> Clearly, bus level reset can't work for the root bus :)
>>>>>
>>>>> Thanks very much!
>>>>> I test the vfio with a second-bus ahci controller and it didn't
>>>>> complain about the lack of reset mechanism. :) And the return val of
>>>>> HOST_CTL is normal now (the same as emulated ahci controller).
>>>>> However, it still paused when the BIOS tried to write to the HOST_CTL.
>>>>> Do you have any idea?
>>>>> And we should just test vfio and legacy pci-assignment with second bus
>>>>> devices, not considering the root-bus devices?
>>>>
>>>> AHCI seems like a poor choice of device for this work, they typically
>>>> don't support any kind of reset and they can be troublesome even for the
>>>> L1 assignment.  You really want something with FLR support so that both
>>>> the host and L1 guest can potentially reset the device.  That said, you
>>>> may still run into bugs with a L1 guest directed FLR.  Thanks,
>>>>
>>>> Alex
>>>>
>>>
>>> So what device do you think is suitable for the pci-assign test? e1000?
>>> I just tested it with sound card ac97 and USB controller. But I don't
>>> know how to attach them to a pcie-to-pci bridge, so maybe they weren't
>>> reset before being assigned to L2. But it seems that they can work.
>>> 1. With the sound card, I assigned it to L2 via both vfio and legacy
>>> pci-assign and I could hear the music played in L2 from host's
>>> speakers. Of course, the vfio also complained about the lack of reset
>>> mechanism.
>>
>> First off, maybe I'm a little confused, are you trying to assigned
>> emulated devices for the L1 guest to the L2 guest or are these assigned
>> devices to the L1 guest that you want to re-assign to L2?  AFAIK, we
>> don't have any emulated devices that support reset, but it wouldn't be
>> that hard to add a PM capability to one that advertises a soft reset on
>> D3hot->D0 transition and calls the QEMU driver reset function when that
>> occurs.  This would provide the most flexibility.
> 
> Or even better might be to add an Advanced Features capability to each
> device so it can do an explicit FLR.  We don't tend to trust PM reset as
> being reliably on all devices.

But FLR is PCIe-only, isn't it? Also, it may let some of our device
models deviate from their real versions (I suppose, e.g., none of the
e1000 devices we currently emulate exposed FLR).

In any case, I suggested Le to focus on his GSoC task again (VT-d
emulation) and postpone this topic to later point (likely outside GSoC).
Only few weeks left...

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Alex Williamson]

On Fri, 2014-08-01 at 18:39 +0200, Jan Kiszka wrote:
> On 2014-08-01 18:35, Alex Williamson wrote:
> > On Fri, 2014-08-01 at 09:25 -0600, Alex Williamson wrote:
> >> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote:
> >>> Hi Alex,
> >>>
> >>> 2014-07-30 22:46 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
> >>>> On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
> >>>>> Hi Michael,
> >>>>>
> >>>>> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
> >>>>>> On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
> >>>>>>> Hi,
> >>>>>>> I am testing vfio in L1 with my VT-d emulation project. I assigned one
> >>>>>>> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
> >>>>>>> in L1, it complains that:
> >>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> >>>>>>> available reset mechanism.
> >>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> >>>>>>> available reset mechanism.
> >>>>>>>
> >>>>>>> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
> >>>>>>> look into this and found that:
> >>>>>>> val = ahci_ctrl_readl(ctrl, HOST_CTL);
> >>>>>>> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
> >>>>>>> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
> >>>>>>> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
> >>>>>>> should be cleared by default. So maybe L1 didn't reset the device
> >>>>>>> before assigning it to L2?
> >>>>>>> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
> >>>>>>>
> >>>>>>> So can anyone give me some advice? About the state of PCI device or
> >>>>>>> bus-level reset?
> >>>>>>>
> >>>>>>> Here is the detail of the environment and the way I did the vfio.
> >>>>>>> 1. lspci in L1 said:
> >>>>>>> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> >>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> >>>>>>> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
> >>>>>>> Interface Controller [8086:2918] (rev 02)
> >>>>>>> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> >>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> >>>>>>> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
> >>>>>>> Controller [8086:2930] (rev 02)
> >>>>>>> 2. Unbind 00:03.0 and do vfio:
> >>>>>>> modprobe -r vfio_iommu_type1
> >>>>>>> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
> >>>>>>> modprobe vfio-pci
> >>>>>>> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
> >>>>>>> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
> >>>>>>> 3. run L2 with "-device vfio-pci,host=00:03.0"
> >>>>>>>
> >>>>>>> Any help is appreciated! Thanks very much!
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Le
> >>>>>>
> >>>>>> Clearly, bus level reset can't work for the root bus :)
> >>>>>
> >>>>> Thanks very much!
> >>>>> I test the vfio with a second-bus ahci controller and it didn't
> >>>>> complain about the lack of reset mechanism. :) And the return val of
> >>>>> HOST_CTL is normal now (the same as emulated ahci controller).
> >>>>> However, it still paused when the BIOS tried to write to the HOST_CTL.
> >>>>> Do you have any idea?
> >>>>> And we should just test vfio and legacy pci-assignment with second bus
> >>>>> devices, not considering the root-bus devices?
> >>>>
> >>>> AHCI seems like a poor choice of device for this work, they typically
> >>>> don't support any kind of reset and they can be troublesome even for the
> >>>> L1 assignment.  You really want something with FLR support so that both
> >>>> the host and L1 guest can potentially reset the device.  That said, you
> >>>> may still run into bugs with a L1 guest directed FLR.  Thanks,
> >>>>
> >>>> Alex
> >>>>
> >>>
> >>> So what device do you think is suitable for the pci-assign test? e1000?
> >>> I just tested it with sound card ac97 and USB controller. But I don't
> >>> know how to attach them to a pcie-to-pci bridge, so maybe they weren't
> >>> reset before being assigned to L2. But it seems that they can work.
> >>> 1. With the sound card, I assigned it to L2 via both vfio and legacy
> >>> pci-assign and I could hear the music played in L2 from host's
> >>> speakers. Of course, the vfio also complained about the lack of reset
> >>> mechanism.
> >>
> >> First off, maybe I'm a little confused, are you trying to assigned
> >> emulated devices for the L1 guest to the L2 guest or are these assigned
> >> devices to the L1 guest that you want to re-assign to L2?  AFAIK, we
> >> don't have any emulated devices that support reset, but it wouldn't be
> >> that hard to add a PM capability to one that advertises a soft reset on
> >> D3hot->D0 transition and calls the QEMU driver reset function when that
> >> occurs.  This would provide the most flexibility.
> > 
> > Or even better might be to add an Advanced Features capability to each
> > device so it can do an explicit FLR.  We don't tend to trust PM reset as
> > being reliably on all devices.
> 
> But FLR is PCIe-only, isn't it?

Nope, according to the ECN it's intended for conventional PCI devices on
the PCIe root complex where a complete conversion to PCIe cannot be
justified.  Maybe it's not 100% to the letter of the capability to add
it to a random non-RC device, but it seems like a good solution here.

>  Also, it may let some of our device
> models deviate from their real versions (I suppose, e.g., none of the
> e1000 devices we currently emulate exposed FLR).

Of course, but what are the chances that the driver will care?

> In any case, I suggested Le to focus on his GSoC task again (VT-d
> emulation) and postpone this topic to later point (likely outside GSoC).
> Only few weeks left...

Ok.  Thanks,

Alex

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Le Tan]

2014-08-01 23:25 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote:
>> Hi Alex,
>>
>> 2014-07-30 22:46 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
>> > On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
>> >> Hi Michael,
>> >>
>> >> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
>> >> > On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
>> >> >> Hi,
>> >> >> I am testing vfio in L1 with my VT-d emulation project. I assigned one
>> >> >> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
>> >> >> in L1, it complains that:
>> >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>> >> >> available reset mechanism.
>> >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>> >> >> available reset mechanism.
>> >> >>
>> >> >> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
>> >> >> look into this and found that:
>> >> >> val = ahci_ctrl_readl(ctrl, HOST_CTL);
>> >> >> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
>> >> >> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
>> >> >> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
>> >> >> should be cleared by default. So maybe L1 didn't reset the device
>> >> >> before assigning it to L2?
>> >> >> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
>> >> >>
>> >> >> So can anyone give me some advice? About the state of PCI device or
>> >> >> bus-level reset?
>> >> >>
>> >> >> Here is the detail of the environment and the way I did the vfio.
>> >> >> 1. lspci in L1 said:
>> >> >> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>> >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>> >> >> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
>> >> >> Interface Controller [8086:2918] (rev 02)
>> >> >> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>> >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>> >> >> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
>> >> >> Controller [8086:2930] (rev 02)
>> >> >> 2. Unbind 00:03.0 and do vfio:
>> >> >> modprobe -r vfio_iommu_type1
>> >> >> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
>> >> >> modprobe vfio-pci
>> >> >> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
>> >> >> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
>> >> >> 3. run L2 with "-device vfio-pci,host=00:03.0"
>> >> >>
>> >> >> Any help is appreciated! Thanks very much!
>> >> >>
>> >> >> Regards,
>> >> >> Le
>> >> >
>> >> > Clearly, bus level reset can't work for the root bus :)
>> >>
>> >> Thanks very much!
>> >> I test the vfio with a second-bus ahci controller and it didn't
>> >> complain about the lack of reset mechanism. :) And the return val of
>> >> HOST_CTL is normal now (the same as emulated ahci controller).
>> >> However, it still paused when the BIOS tried to write to the HOST_CTL.
>> >> Do you have any idea?
>> >> And we should just test vfio and legacy pci-assignment with second bus
>> >> devices, not considering the root-bus devices?
>> >
>> > AHCI seems like a poor choice of device for this work, they typically
>> > don't support any kind of reset and they can be troublesome even for the
>> > L1 assignment.  You really want something with FLR support so that both
>> > the host and L1 guest can potentially reset the device.  That said, you
>> > may still run into bugs with a L1 guest directed FLR.  Thanks,
>> >
>> > Alex
>> >
>>
>> So what device do you think is suitable for the pci-assign test? e1000?
>> I just tested it with sound card ac97 and USB controller. But I don't
>> know how to attach them to a pcie-to-pci bridge, so maybe they weren't
>> reset before being assigned to L2. But it seems that they can work.
>> 1. With the sound card, I assigned it to L2 via both vfio and legacy
>> pci-assign and I could hear the music played in L2 from host's
>> speakers. Of course, the vfio also complained about the lack of reset
>> mechanism.
>
> First off, maybe I'm a little confused, are you trying to assigned
> emulated devices for the L1 guest to the L2 guest or are these assigned
> devices to the L1 guest that you want to re-assign to L2?  AFAIK, we
> don't have any emulated devices that support reset, but it wouldn't be
> that hard to add a PM capability to one that advertises a soft reset on
> D3hot->D0 transition and calls the QEMU driver reset function when that
> occurs.  This would provide the most flexibility.

I am trying to assign emulated devices for the L1 guest to the L2
guest. According to Jan's suggestions, I will first go on to add more
features to the VT-d emulation.
Thanks very much! Learn a lot from you! :)

Le

> The other choice is to create a topology where an individual device can
> be reset via a bus reset, so putting a single device behind a bridge,
> root port, or downstream switch port.
>
> Otherwise, the emulated e1000 is probably a reasonable choice, network
> drivers generally seem to be pretty good about accepting a device in a
> running state since it might have been used as a boot device.
>
> If you're looking at re-assigning an L1 assigned device to an L2 guest,
> I'm not sure how that's supposed to work.  The VT-d emulation would need
> to somehow push L2 physical mappings to the host, but the L1 guest
> doesn't know the true isolation constraints of the host devices and
> whether they can actually be mapped separately from other devices.
> Since we try to use as few IOMMU domains as possible, you'd end up
> modifying translations for all devices assigned to the L1 guest.
>
>> 2. With the USB controller, I used "-usb -usbdevice disk:file" to
>> attach a USB disk to L1. But there were 4 related devices in L1, so I
>> didn't know what should be assigned to L2. the info qtree was like
>> this:
>>  dev: ich9-usb-uhci3, id ""
>>         masterbus = "usb-bus.0"
>>         firstport = 4 (0x4)
>>         bandwidth = 1280 (0x500)
>>         maxframes = 128 (0x80)
>>         addr = 1d.2
>> dev: ich9-usb-uhci2, id ""
>>         masterbus = "usb-bus.0"
>>         firstport = 2 (0x2)
>>         bandwidth = 1280 (0x500)
>>         maxframes = 128 (0x80)
>>         addr = 1d.1
>>  dev: ich9-usb-uhci1, id ""
>>         masterbus = "usb-bus.0"
>>         firstport = 0 (0x0)
>>         bandwidth = 1280 (0x500)
>>         maxframes = 128 (0x80)
>>         addr = 1d.0
>> dev: ich9-usb-ehci1, id ""
>>         maxframes = 128 (0x80)
>>         addr = 1d.7
>>          bus: usb-bus.0
>>           type usb-bus
>>           dev: usb-storage, id ""
>>             drive = ""
>>             logical_block_size = 512 (0x200)
>>             physical_block_size = 512 (0x200)
>>             min_io_size = 0 (0x0)
>>             opt_io_size = 0 (0x0)
>>             bootindex = -1 (0xffffffffffffffff)
>>             discard_granularity = 4294967295 (0xffffffff)
>>             removable = false
>>             port = ""
>>             serial = ""
>>             full-path = true
>>             msos-desc = true
>>             addr 0.1, port 1, speed 480, name QEMU USB MSD, attached
>>             bus: scsi.0
>>               type SCSI
>>               dev: scsi-disk, id ""
>>                 drive = "usb0"
>>                 logical_block_size = 512 (0x200)
>>                 physical_block_size = 512 (0x200)
>>                 min_io_size = 0 (0x0)
>>                 opt_io_size = 0 (0x0)
>> Then I tried to unbind and bind both 1d.7 and 1d.0. Then I assigned
>> 1d.0 to L2. L2 could boot smoothly and I could mount and access the
>> USB disk.
>> So what do you think?
>
> Sounds like a reasonable test of L1 emulated assigned to L2.  Thanks,
>
> Alex
>
>

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

On 2014-08-01 19:16, Alex Williamson wrote:
>>  Also, it may let some of our device
>> models deviate from their real versions (I suppose, e.g., none of the
>> e1000 devices we currently emulate exposed FLR).
> 
> Of course, but what are the chances that the driver will care?

No drivers of GPOSes, but special or legacy OSes may do so. Keep in mind
that Intel e.g. is documenting their PCI devices with fixed config space
addresses for their capability.

Also, we completely lack PM caps so far. Adding them would already have
the value of increasing emulation accuracy. And here I think we are free
to always implement reset behavior behind D3->D0 transitions. So my
believe is that this will be the better path.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Michael S. Tsirkin]

On Sat, Aug 02, 2014 at 07:54:30AM +0200, Jan Kiszka wrote:
> On 2014-08-01 19:16, Alex Williamson wrote:
> >>  Also, it may let some of our device
> >> models deviate from their real versions (I suppose, e.g., none of the
> >> e1000 devices we currently emulate exposed FLR).
> > 
> > Of course, but what are the chances that the driver will care?
> 
> No drivers of GPOSes, but special or legacy OSes may do so. Keep in mind
> that Intel e.g. is documenting their PCI devices with fixed config space
> addresses for their capability.
> 
> Also, we completely lack PM caps so far. Adding them would already have
> the value of increasing emulation accuracy. And here I think we are free
> to always implement reset behavior behind D3->D0 transitions. So my
> believe is that this will be the better path.
> 
> Jan
> 
> 


That's true but is PM reset per-function or per-device?
I'll have to check the spec - do you happen to rememeber?