vfio-pci rejects binding to devices having same pcie vendor id and device id

vfio-pci rejects binding to devices having same pcie vendor id and device id

[Major Saheb]

I have a linux system running in kvm, with 6 qemu emulated NVMe
drives, as expected all of them have the same PCIe Vendor ID and
Device ID(VID: 0x1b36 DID: 0x0010).
When I try to unbind them from the kernel NVMe driver and bind it to
vfio-pci one by one, I am getting "write error: File exists" when I
try to bind the 2nd(and other) drive to vfio-pci.

Kernel version

5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
x86_64 x86_64 GNU/Linux


lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme0n1 ->
../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1
lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme1n1 ->
../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1
lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme2n1 ->
../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1
lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme3n1 ->
../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1
lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme4n1 ->
../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1
lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme5n1 ->
../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1


Steps for repro
ubind nvme2 from kernel NVMe driver and bind it to vfio
$ ls -l /sys/bus/pci/drivers/vfio-pci/
total 0
lrwxrwxrwx 1 root root    0 Dec  8 13:04 0000:00:05.0 ->
../../../../devices/pci0000:00/0000:00:05.0
--w------- 1 root root 4096 Dec  8 13:07 bind
lrwxrwxrwx 1 root root    0 Dec  8 13:07 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 Dec  8 13:04 new_id
--w------- 1 root root 4096 Dec  8 13:07 remove_id
--w------- 1 root root 4096 Dec  8 11:32 uevent
--w------- 1 root root 4096 Dec  8 13:07 unbind

Unbind nvme3 from  kernel NVMe driver
Try binding to vfio-pci
# echo "0x1b36  0x0010" >  /sys/bus/pci/drivers/vfio-pci/new_id
-bash: echo: write error: File exists

Not sure but this seems interesting
https://github.com/torvalds/linux/commit/3853f9123c185eb4018f5ccd3cdda5968efb5e10#diff-625d2827bff96bb3a019fa705d99f0b89ec32f281c38a844457b3413d9172007

Can some help ?

Re: vfio-pci rejects binding to devices having same pcie vendor id and device id

[Bjorn Helgaas]

[+cc VFIO folks and Zhenzhong (author of the commit you mention)]

On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote:
> I have a linux system running in kvm, with 6 qemu emulated NVMe
> drives, as expected all of them have the same PCIe Vendor ID and
> Device ID(VID: 0x1b36 DID: 0x0010).
>
> When I try to unbind them from the kernel NVMe driver and bind it to
> vfio-pci one by one, I am getting "write error: File exists" when I
> try to bind the 2nd(and other) drive to vfio-pci.
> 
> Kernel version
> 
> 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
> x86_64 x86_64 GNU/Linux
> 
> lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1
> lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1
> lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1
> lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1
> lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1
> lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1
> 
> Steps for repro
> ubind nvme2 from kernel NVMe driver and bind it to vfio
> $ ls -l /sys/bus/pci/drivers/vfio-pci/
> lrwxrwxrwx 1 root root    0 Dec  8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0
> --w------- 1 root root 4096 Dec  8 13:07 bind
> lrwxrwxrwx 1 root root    0 Dec  8 13:07 module -> ../../../../module/vfio_pci
> --w------- 1 root root 4096 Dec  8 13:04 new_id
> --w------- 1 root root 4096 Dec  8 13:07 remove_id
> --w------- 1 root root 4096 Dec  8 11:32 uevent
> --w------- 1 root root 4096 Dec  8 13:07 unbind
> 
> Unbind nvme3 from  kernel NVMe driver
> Try binding to vfio-pci
> # echo "0x1b36  0x0010" >  /sys/bus/pci/drivers/vfio-pci/new_id
> -bash: echo: write error: File exists
> 
> Not sure but this seems interesting
> https://github.com/torvalds/linux/commit/3853f9123c185eb4018f5ccd3cdda5968efb5e10#diff-625d2827bff96bb3a019fa705d99f0b89ec32f281c38a844457b3413d9172007
> 
> Can some help ?

Re: vfio-pci rejects binding to devices having same pcie vendor id and device id

[Alex Williamson]

On Thu, 8 Dec 2022 10:50:08 -0600
Bjorn Helgaas <helgaas@kernel.org> wrote:

> [+cc VFIO folks and Zhenzhong (author of the commit you mention)]
> 
> On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote:
> > I have a linux system running in kvm, with 6 qemu emulated NVMe
> > drives, as expected all of them have the same PCIe Vendor ID and
> > Device ID(VID: 0x1b36 DID: 0x0010).
> >
> > When I try to unbind them from the kernel NVMe driver and bind it to
> > vfio-pci one by one, I am getting "write error: File exists" when I
> > try to bind the 2nd(and other) drive to vfio-pci.
> > 
> > Kernel version
> > 
> > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
> > x86_64 x86_64 GNU/Linux
> > 
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1
> > 
> > Steps for repro
> > ubind nvme2 from kernel NVMe driver and bind it to vfio
> > $ ls -l /sys/bus/pci/drivers/vfio-pci/
> > lrwxrwxrwx 1 root root    0 Dec  8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0
> > --w------- 1 root root 4096 Dec  8 13:07 bind
> > lrwxrwxrwx 1 root root    0 Dec  8 13:07 module -> ../../../../module/vfio_pci
> > --w------- 1 root root 4096 Dec  8 13:04 new_id
> > --w------- 1 root root 4096 Dec  8 13:07 remove_id
> > --w------- 1 root root 4096 Dec  8 11:32 uevent
> > --w------- 1 root root 4096 Dec  8 13:07 unbind
> > 
> > Unbind nvme3 from  kernel NVMe driver
> > Try binding to vfio-pci
> > # echo "0x1b36  0x0010" >  /sys/bus/pci/drivers/vfio-pci/new_id
> > -bash: echo: write error: File exists

Presumably you already wrote this same ID to the dynamic ID table from
the first device, so yes, it's going to rightfully complain that this
ID already exists.  The new_id interface has numerous problems, which
is why we added the driver_override interface, which is used by tools
like libvirt and driverctl in place of this old interface.

I'd recommend something like:

# driverctl --nosave set-override 0000:00:03.0 vfio-pci
# driverctl --nosave set-override 0000:00:04.0 vfio-pci
# driverctl --nosave set-override 0000:00:05.0 vfio-pci
...

Or if vfio-pci is generally the preferred driver for these devices, you
could remove the --nosave option to have them automatically bound at
boot.  You could also make use of pre-filling the vfio device table
using vfio-pci.ids=1b36:0010 on the kernel command line and making sure
the vfio-pci driver is loaded before the nvme driver.  In general, for
dynamic binding of devices, driver_override is the recommended solution.
Thanks,

Alex

Re: vfio-pci rejects binding to devices having same pcie vendor id and device id

[Major Saheb]

Thanks Alex ,
That works for me

~$ sudo driverctl --nosave set-override 0000:00:05.0 vfio-pci
~$ sudo driverctl --nosave set-override 0000:00:06.0 vfio-pci
~$ sudo driverctl --nosave set-override 0000:00:07.0 vfio-pci
admin@node-1:~$ sudo nvme list
Node                  SN                   Model
             Namespace Usage                      Format           FW
Rev
--------------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme10n1         akqvf2-0_10          QEMU NVMe Ctrl
             1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
/dev/nvme11n1         akqvf2-0_11          QEMU NVMe Ctrl
             1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
/dev/nvme5n1          akqvf2-0_5           QEMU NVMe Ctrl
             1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
/dev/nvme6n1          akqvf2-0_6           QEMU NVMe Ctrl
             1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
/dev/nvme7n1          akqvf2-0_7           QEMU NVMe Ctrl
             1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
/dev/nvme8n1          akqvf2-0_8           QEMU NVMe Ctrl
             1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
/dev/nvme9n1          akqvf2-0_9           QEMU NVMe Ctrl
             1         274.88  GB / 274.88  GB    512   B +  0 B   1.0

I came across you blogspot after I fired the mail
http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html
Some should update https://docs.kernel.org/driver-api/vfio.html in
public interest , If I knew how to do that I would do it,
Thanks again.

On Thu, Dec 8, 2022 at 10:55 PM Alex Williamson
<alex.williamson@redhat.com> wrote:
>
> On Thu, 8 Dec 2022 10:50:08 -0600
> Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> > [+cc VFIO folks and Zhenzhong (author of the commit you mention)]
> >
> > On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote:
> > > I have a linux system running in kvm, with 6 qemu emulated NVMe
> > > drives, as expected all of them have the same PCIe Vendor ID and
> > > Device ID(VID: 0x1b36 DID: 0x0010).
> > >
> > > When I try to unbind them from the kernel NVMe driver and bind it to
> > > vfio-pci one by one, I am getting "write error: File exists" when I
> > > try to bind the 2nd(and other) drive to vfio-pci.
> > >
> > > Kernel version
> > >
> > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
> > > x86_64 x86_64 GNU/Linux
> > >
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1
> > >
> > > Steps for repro
> > > ubind nvme2 from kernel NVMe driver and bind it to vfio
> > > $ ls -l /sys/bus/pci/drivers/vfio-pci/
> > > lrwxrwxrwx 1 root root    0 Dec  8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0
> > > --w------- 1 root root 4096 Dec  8 13:07 bind
> > > lrwxrwxrwx 1 root root    0 Dec  8 13:07 module -> ../../../../module/vfio_pci
> > > --w------- 1 root root 4096 Dec  8 13:04 new_id
> > > --w------- 1 root root 4096 Dec  8 13:07 remove_id
> > > --w------- 1 root root 4096 Dec  8 11:32 uevent
> > > --w------- 1 root root 4096 Dec  8 13:07 unbind
> > >
> > > Unbind nvme3 from  kernel NVMe driver
> > > Try binding to vfio-pci
> > > # echo "0x1b36  0x0010" >  /sys/bus/pci/drivers/vfio-pci/new_id
> > > -bash: echo: write error: File exists
>
> Presumably you already wrote this same ID to the dynamic ID table from
> the first device, so yes, it's going to rightfully complain that this
> ID already exists.  The new_id interface has numerous problems, which
> is why we added the driver_override interface, which is used by tools
> like libvirt and driverctl in place of this old interface.
>
> I'd recommend something like:
>
> # driverctl --nosave set-override 0000:00:03.0 vfio-pci
> # driverctl --nosave set-override 0000:00:04.0 vfio-pci
> # driverctl --nosave set-override 0000:00:05.0 vfio-pci
> ...
>
> Or if vfio-pci is generally the preferred driver for these devices, you
> could remove the --nosave option to have them automatically bound at
> boot.  You could also make use of pre-filling the vfio device table
> using vfio-pci.ids=1b36:0010 on the kernel command line and making sure
> the vfio-pci driver is loaded before the nvme driver.  In general, for
> dynamic binding of devices, driver_override is the recommended solution.
> Thanks,
>
> Alex
>

Re: vfio-pci rejects binding to devices having same pcie vendor id and device id

[Alex Williamson]

On Thu, 8 Dec 2022 23:25:07 +0530
Major Saheb <majosaheb@gmail.com> wrote:

> Thanks Alex ,
> That works for me
> 
> ~$ sudo driverctl --nosave set-override 0000:00:05.0 vfio-pci
> ~$ sudo driverctl --nosave set-override 0000:00:06.0 vfio-pci
> ~$ sudo driverctl --nosave set-override 0000:00:07.0 vfio-pci
> admin@node-1:~$ sudo nvme list
> Node                  SN                   Model
>              Namespace Usage                      Format           FW
> Rev
> --------------------- --------------------
> ---------------------------------------- ---------
> -------------------------- ---------------- --------
> /dev/nvme10n1         akqvf2-0_10          QEMU NVMe Ctrl
>              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> /dev/nvme11n1         akqvf2-0_11          QEMU NVMe Ctrl
>              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> /dev/nvme5n1          akqvf2-0_5           QEMU NVMe Ctrl
>              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> /dev/nvme6n1          akqvf2-0_6           QEMU NVMe Ctrl
>              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> /dev/nvme7n1          akqvf2-0_7           QEMU NVMe Ctrl
>              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> /dev/nvme8n1          akqvf2-0_8           QEMU NVMe Ctrl
>              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> /dev/nvme9n1          akqvf2-0_9           QEMU NVMe Ctrl
>              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> 
> I came across you blogspot after I fired the mail
> http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html
> Some should update https://docs.kernel.org/driver-api/vfio.html in
> public interest , If I knew how to do that I would do it,

Yes, that documentation is from before the driver_override method was
introduced.  There's some non vfio specific documentation of
driver_override here:

https://docs.kernel.org/admin-guide/abi-testing.html?highlight=driver_override#abi-sys-bus-pci-devices-driver-override

Otherwise, documentation updates gladly accepted.  This documentation
is part of the kernel source tree and follows the same process as
submitting code changes, outlined here:

https://docs.kernel.org/process/submitting-patches.html

The kvm@vger.kernel.org mailing list handles patches for vfio, but
please keep me in Cc if you submit something.  Thanks,

Alex

Re: vfio-pci rejects binding to devices having same pcie vendor id and device id

[Major Saheb]

Sure , I will read the process and try to submit the patch.
Thanks.

On Thu, Dec 8, 2022 at 11:51 PM Alex Williamson
<alex.williamson@redhat.com> wrote:
>
> On Thu, 8 Dec 2022 23:25:07 +0530
> Major Saheb <majosaheb@gmail.com> wrote:
>
> > Thanks Alex ,
> > That works for me
> >
> > ~$ sudo driverctl --nosave set-override 0000:00:05.0 vfio-pci
> > ~$ sudo driverctl --nosave set-override 0000:00:06.0 vfio-pci
> > ~$ sudo driverctl --nosave set-override 0000:00:07.0 vfio-pci
> > admin@node-1:~$ sudo nvme list
> > Node                  SN                   Model
> >              Namespace Usage                      Format           FW
> > Rev
> > --------------------- --------------------
> > ---------------------------------------- ---------
> > -------------------------- ---------------- --------
> > /dev/nvme10n1         akqvf2-0_10          QEMU NVMe Ctrl
> >              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> > /dev/nvme11n1         akqvf2-0_11          QEMU NVMe Ctrl
> >              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> > /dev/nvme5n1          akqvf2-0_5           QEMU NVMe Ctrl
> >              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> > /dev/nvme6n1          akqvf2-0_6           QEMU NVMe Ctrl
> >              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> > /dev/nvme7n1          akqvf2-0_7           QEMU NVMe Ctrl
> >              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> > /dev/nvme8n1          akqvf2-0_8           QEMU NVMe Ctrl
> >              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> > /dev/nvme9n1          akqvf2-0_9           QEMU NVMe Ctrl
> >              1         274.88  GB / 274.88  GB    512   B +  0 B   1.0
> >
> > I came across you blogspot after I fired the mail
> > http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html
> > Some should update https://docs.kernel.org/driver-api/vfio.html in
> > public interest , If I knew how to do that I would do it,
>
> Yes, that documentation is from before the driver_override method was
> introduced.  There's some non vfio specific documentation of
> driver_override here:
>
> https://docs.kernel.org/admin-guide/abi-testing.html?highlight=driver_override#abi-sys-bus-pci-devices-driver-override
>
> Otherwise, documentation updates gladly accepted.  This documentation
> is part of the kernel source tree and follows the same process as
> submitting code changes, outlined here:
>
> https://docs.kernel.org/process/submitting-patches.html
>
> The kvm@vger.kernel.org mailing list handles patches for vfio, but
> please keep me in Cc if you submit something.  Thanks,
>
> Alex
>

Re: vfio-pci rejects binding to devices having same pcie vendor id and device id

[Zhenzhong Duan]

On Fri, Dec 9, 2022 at 12:50 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc VFIO folks and Zhenzhong (author of the commit you mention)]
>
> On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote:
> > I have a linux system running in kvm, with 6 qemu emulated NVMe
> > drives, as expected all of them have the same PCIe Vendor ID and
> > Device ID(VID: 0x1b36 DID: 0x0010).
> >
> > When I try to unbind them from the kernel NVMe driver and bind it to
> > vfio-pci one by one, I am getting "write error: File exists" when I
> > try to bind the 2nd(and other) drive to vfio-pci.
> >
> > Kernel version
> >
> > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
> > x86_64 x86_64 GNU/Linux
> >
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1
> >
> > Steps for repro
> > ubind nvme2 from kernel NVMe driver and bind it to vfio
> > $ ls -l /sys/bus/pci/drivers/vfio-pci/
> > lrwxrwxrwx 1 root root    0 Dec  8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0
> > --w------- 1 root root 4096 Dec  8 13:07 bind
> > lrwxrwxrwx 1 root root    0 Dec  8 13:07 module -> ../../../../module/vfio_pci
> > --w------- 1 root root 4096 Dec  8 13:04 new_id
> > --w------- 1 root root 4096 Dec  8 13:07 remove_id
> > --w------- 1 root root 4096 Dec  8 11:32 uevent
> > --w------- 1 root root 4096 Dec  8 13:07 unbind
> >
> > Unbind nvme3 from  kernel NVMe driver
> > Try binding to vfio-pci
> > # echo "0x1b36  0x0010" >  /sys/bus/pci/drivers/vfio-pci/new_id
> > -bash: echo: write error: File exists

Above operation added new id to vfio-pci, it will trigger scanning of
unbind devices once by default.
To bind other devices of same id, besides what Alex suggested, I think you can
also use /sys/bus/pci/drivers/vfio-pci/bind interface.

Thanks
Zhenzhong

> >
> > Not sure but this seems interesting
> > https://github.com/torvalds/linux/commit/3853f9123c185eb4018f5ccd3cdda5968efb5e10#diff-625d2827bff96bb3a019fa705d99f0b89ec32f281c38a844457b3413d9172007
> >
> > Can some help ?

Re: vfio-pci rejects binding to devices having same pcie vendor id and device id

[Major Saheb]

Ok, thanks.

On Mon, Dec 12, 2022 at 10:14 AM Zhenzhong Duan
<zhenzhong.duan@gmail.com> wrote:
>
> On Fri, Dec 9, 2022 at 12:50 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > [+cc VFIO folks and Zhenzhong (author of the commit you mention)]
> >
> > On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote:
> > > I have a linux system running in kvm, with 6 qemu emulated NVMe
> > > drives, as expected all of them have the same PCIe Vendor ID and
> > > Device ID(VID: 0x1b36 DID: 0x0010).
> > >
> > > When I try to unbind them from the kernel NVMe driver and bind it to
> > > vfio-pci one by one, I am getting "write error: File exists" when I
> > > try to bind the 2nd(and other) drive to vfio-pci.
> > >
> > > Kernel version
> > >
> > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
> > > x86_64 x86_64 GNU/Linux
> > >
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1
> > > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1
> > >
> > > Steps for repro
> > > ubind nvme2 from kernel NVMe driver and bind it to vfio
> > > $ ls -l /sys/bus/pci/drivers/vfio-pci/
> > > lrwxrwxrwx 1 root root    0 Dec  8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0
> > > --w------- 1 root root 4096 Dec  8 13:07 bind
> > > lrwxrwxrwx 1 root root    0 Dec  8 13:07 module -> ../../../../module/vfio_pci
> > > --w------- 1 root root 4096 Dec  8 13:04 new_id
> > > --w------- 1 root root 4096 Dec  8 13:07 remove_id
> > > --w------- 1 root root 4096 Dec  8 11:32 uevent
> > > --w------- 1 root root 4096 Dec  8 13:07 unbind
> > >
> > > Unbind nvme3 from  kernel NVMe driver
> > > Try binding to vfio-pci
> > > # echo "0x1b36  0x0010" >  /sys/bus/pci/drivers/vfio-pci/new_id
> > > -bash: echo: write error: File exists
>
> Above operation added new id to vfio-pci, it will trigger scanning of
> unbind devices once by default.
> To bind other devices of same id, besides what Alex suggested, I think you can
> also use /sys/bus/pci/drivers/vfio-pci/bind interface.
>
> Thanks
> Zhenzhong
>
> > >
> > > Not sure but this seems interesting
> > > https://github.com/torvalds/linux/commit/3853f9123c185eb4018f5ccd3cdda5968efb5e10#diff-625d2827bff96bb3a019fa705d99f0b89ec32f281c38a844457b3413d9172007
> > >
> > > Can some help ?