Re: vfio-pci problem

Re: vfio-pci problem

[Bjorn Helgaas]

[+cc Alex, kvm, linux-pci]

On Fri, Aug 13, 2021 at 09:43:39AM +0200, Idar Lund wrote:
> Hi,
> 
> I've been struggling with an error in linux since 5.11. Please find my bug
> report here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1945565
> 
> Then I stumbled upon this mail thread:
> https://www.spinics.net/lists/linux-pci/msg102243.html which seems related.
> 
> Is there another way to do this in 5.11+ or is this an unintentionally bug
> that got introduced in 5.11?

Hi Idar, sorry for the trouble and thanks for the report!  I cc'd some
VFIO experts who know more than I do about this.

If I understand correctly, you have a PCI XHCI controller:

  pci 0000:06:00.0: [1b73:1100] type 00 class 0x0c0330
  xhci_hcd 0000:06:00.0: xHCI Host Controller

and you want to unbind the xhci_hcd driver and bind vfio-pci instead:

  # echo '0000:06:00.0' > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
  # echo 0x1b73 0x1100 > /sys/bus/pci/drivers/vfio-pci/new_id

In v5.10 (5.10.17-200.fc33.x86_64) this worked fine, but in v5.11
(5.11.9-200.fc33.x86_64) the "new_id" write returns -EEXIST and
binding to vfio-pci fails.

The patch you pointed out appeared in v5.11 as 3853f9123c18 ("PCI:
Avoid duplicate IDs in driver dynamic IDs list") [1], and I agree it
looks suspicious.  There haven't been any significant changes to
pci-driver.c since then.

Have you added "0x1b73 0x1100" to vfio-pci/new_id previously?  I think
in v5.10, that would silently work (possibly adding duplicate entries
to the dynamic ID list) and every write to vfio-pci/new_id would make
vfio-pci try to bind to the device.

In v5.11, if you write a duplicate ID to vfio-pci/new_id, you would
get -EEXIST and no attempt to bind.  As far as I know, the dynamic ID
list is not visible in sysfs, so it might be hard to avoid writing a
duplicate.

But if the vfio-pci dynamic ID list already contains "0x1b73 0x1100",
you should be able to ask vfio-pci to bind to the device like this:

  # echo 0000:06:00.0 > /sys/bus/pci/drivers/virtio-pci/bind

I don't know if that's a solution, but would be useful to know whether
it's a workaround.

[1] https://git.kernel.org/linus/3853f9123c18

Re: vfio-pci problem

[Alex Williamson]

On Fri, 13 Aug 2021 11:53:07 -0500
Bjorn Helgaas <helgaas@kernel.org> wrote:

> [+cc Alex, kvm, linux-pci]
> 
> On Fri, Aug 13, 2021 at 09:43:39AM +0200, Idar Lund wrote:
> > Hi,
> > 
> > I've been struggling with an error in linux since 5.11. Please find my bug
> > report here:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1945565
> > 
> > Then I stumbled upon this mail thread:
> > https://www.spinics.net/lists/linux-pci/msg102243.html which seems related.
> > 
> > Is there another way to do this in 5.11+ or is this an unintentionally bug
> > that got introduced in 5.11?  
> 
> Hi Idar, sorry for the trouble and thanks for the report!  I cc'd some
> VFIO experts who know more than I do about this.
> 
> If I understand correctly, you have a PCI XHCI controller:
> 
>   pci 0000:06:00.0: [1b73:1100] type 00 class 0x0c0330
>   xhci_hcd 0000:06:00.0: xHCI Host Controller
> 
> and you want to unbind the xhci_hcd driver and bind vfio-pci instead:
> 
>   # echo '0000:06:00.0' > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
>   # echo 0x1b73 0x1100 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
> In v5.10 (5.10.17-200.fc33.x86_64) this worked fine, but in v5.11
> (5.11.9-200.fc33.x86_64) the "new_id" write returns -EEXIST and
> binding to vfio-pci fails.
> 
> The patch you pointed out appeared in v5.11 as 3853f9123c18 ("PCI:
> Avoid duplicate IDs in driver dynamic IDs list") [1], and I agree it
> looks suspicious.  There haven't been any significant changes to
> pci-driver.c since then.
> 
> Have you added "0x1b73 0x1100" to vfio-pci/new_id previously?  I think
> in v5.10, that would silently work (possibly adding duplicate entries
> to the dynamic ID list) and every write to vfio-pci/new_id would make
> vfio-pci try to bind to the device.
> 
> In v5.11, if you write a duplicate ID to vfio-pci/new_id, you would
> get -EEXIST and no attempt to bind.  As far as I know, the dynamic ID
> list is not visible in sysfs, so it might be hard to avoid writing a
> duplicate.
> 
> But if the vfio-pci dynamic ID list already contains "0x1b73 0x1100",
> you should be able to ask vfio-pci to bind to the device like this:
> 
>   # echo 0000:06:00.0 > /sys/bus/pci/drivers/virtio-pci/bind
> 
> I don't know if that's a solution, but would be useful to know whether
> it's a workaround.

[root@x1 vfio-pci]# echo 0x1b73 0x1100 > new_id 
[root@x1 vfio-pci]# echo 0x1b73 0x1100 > new_id 
bash: echo: write error: File exists
[root@x1 vfio-pci]# uname -r
5.12.15-200.fc33.x86_64

Seems like it behaves as expected now.  The new_id interface has some
inherit issues, essentially all vfio-pci dynamic binding cases should
instead be using the driver_override interface.  The driverctl utility
already makes use of this and will make your life a tiny bit easier.
Thanks,

Alex

Re: vfio-pci problem

[Idar Lund]

Hi all,

Thanks for the reply Bjorn!

Got a Mail Delivery system message back on this e-mail because it
contained HTML. Gawd dammit gmail! Sorry to all of you that are
getting this 2 times.
Also; Alex' comment got in between here. So I'll answer that one here
too (at the bottom).

Yes, you understand correctly. The issue here is that I have a PCI
XHCI controller that I want to bind to the vfio-pci instead of the
xhci_hcd. The reason is that I want to be able to pass through this
device to a virtual machine.

I'm not sure if I understand your question 'Have you added "0x1b73
0x1100" to vfio-pci/new_id previously', but I'll try to answer; no, I
don't add it during the 5.11 (and newer kernels) boot process or
anything like that, I only add it manually like the commands in the
bugzilla bug report when starting the virtual machine. And I also do
it only once. If someone is interested in the source on why I'm doing
it like this, check out
https://www.heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough/#Part_14_-_Passing_more_PCI_devices_to_guest

Unfortunately the workaround don't seem to be working:
[root@silje ~]# echo '0000:06:00.0' > /sys/bus/pci/drivers/virtio-pci/bind
-bash: echo: write error: No such device

Alex;
Yes, the patch works as expected for (errorusly) double adding to
new_id, the problem here is that this is the first time echoing to
new_id.

-Idar

On Fri, Aug 13, 2021 at 7:32 PM Alex Williamson
<alex.williamson@redhat.com> wrote:
>
> On Fri, 13 Aug 2021 11:53:07 -0500
> Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> > [+cc Alex, kvm, linux-pci]
> >
> > On Fri, Aug 13, 2021 at 09:43:39AM +0200, Idar Lund wrote:
> > > Hi,
> > >
> > > I've been struggling with an error in linux since 5.11. Please find my bug
> > > report here:
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1945565
> > >
> > > Then I stumbled upon this mail thread:
> > > https://www.spinics.net/lists/linux-pci/msg102243.html which seems related.
> > >
> > > Is there another way to do this in 5.11+ or is this an unintentionally bug
> > > that got introduced in 5.11?
> >
> > Hi Idar, sorry for the trouble and thanks for the report!  I cc'd some
> > VFIO experts who know more than I do about this.
> >
> > If I understand correctly, you have a PCI XHCI controller:
> >
> >   pci 0000:06:00.0: [1b73:1100] type 00 class 0x0c0330
> >   xhci_hcd 0000:06:00.0: xHCI Host Controller
> >
> > and you want to unbind the xhci_hcd driver and bind vfio-pci instead:
> >
> >   # echo '0000:06:00.0' > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
> >   # echo 0x1b73 0x1100 > /sys/bus/pci/drivers/vfio-pci/new_id
> >
> > In v5.10 (5.10.17-200.fc33.x86_64) this worked fine, but in v5.11
> > (5.11.9-200.fc33.x86_64) the "new_id" write returns -EEXIST and
> > binding to vfio-pci fails.
> >
> > The patch you pointed out appeared in v5.11 as 3853f9123c18 ("PCI:
> > Avoid duplicate IDs in driver dynamic IDs list") [1], and I agree it
> > looks suspicious.  There haven't been any significant changes to
> > pci-driver.c since then.
> >
> > Have you added "0x1b73 0x1100" to vfio-pci/new_id previously?  I think
> > in v5.10, that would silently work (possibly adding duplicate entries
> > to the dynamic ID list) and every write to vfio-pci/new_id would make
> > vfio-pci try to bind to the device.
> >
> > In v5.11, if you write a duplicate ID to vfio-pci/new_id, you would
> > get -EEXIST and no attempt to bind.  As far as I know, the dynamic ID
> > list is not visible in sysfs, so it might be hard to avoid writing a
> > duplicate.
> >
> > But if the vfio-pci dynamic ID list already contains "0x1b73 0x1100",
> > you should be able to ask vfio-pci to bind to the device like this:
> >
> >   # echo 0000:06:00.0 > /sys/bus/pci/drivers/virtio-pci/bind
> >
> > I don't know if that's a solution, but would be useful to know whether
> > it's a workaround.
>
> [root@x1 vfio-pci]# echo 0x1b73 0x1100 > new_id
> [root@x1 vfio-pci]# echo 0x1b73 0x1100 > new_id
> bash: echo: write error: File exists
> [root@x1 vfio-pci]# uname -r
> 5.12.15-200.fc33.x86_64
>
> Seems like it behaves as expected now.  The new_id interface has some
> inherit issues, essentially all vfio-pci dynamic binding cases should
> instead be using the driver_override interface.  The driverctl utility
> already makes use of this and will make your life a tiny bit easier.
> Thanks,
>
> Alex
>