[Bug 202055] Failed to PCI passthrough SSD with SMI SM2262 controller.

[Bug 202055] Failed to PCI passthrough SSD with SMI SM2262 controller.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202055

--- Comment #42 from Alex Williamson (alex.williamson@redhat.com) ---
It seems there's a partial workaround available since QEMU v2.12 hiding under
our noses.  That version adds support for relocating the MSI-X vector table on
vfio-pci devices, which recreates the MSI-X MMIO space elsewhere on the device.
 A side-effect of this is that the vector table and PBA are properly sized so
as not to collide.  The size of the tables remains wrong, but this only becomes
a problem if the nvme code attempts to allocate >16 vectors, which requires >15
vCPU (or host CPUs if the device is returned to host drivers after being
assigned)(nvme appears to allocate 1 admin queue, plus a queue per CPU, each
making use of an IRQ vector).  The QEMU vfio-pci device option is
x-msix-relocation= which allows specifying the bar to use for the MSI-X tables,
ex. bar0...bar5.  Since this device uses a 64bit bar0, we can either extend
that BAR or choose another, excluding bar1, which is consumed by the upper half
of bar0.  For instance, I tested with:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
...
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
function='0x0'/>
    </hostdev>
...
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
  </qemu:commandline>


(NB: "ua-" is a required prefix when specifying an alias)

A new virtual BAR appears in the guest hosting the MSI-X table and QEMU starts
normally so long as the guest doesn't exceed 15 vCPUs.

The vCPU/pCPU count limitations are obviously not ideal, but hopefully this
provides some degree of workaround for typical configurations.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202055] Failed to PCI passthrough SSD with SMI SM2262 controller.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202055

--- Comment #43 from LimeTech (tomm@lime-technology.com) ---
(In reply to Alex Williamson from comment #42)
> It seems there's a partial workaround available since QEMU v2.12 hiding
> under our noses.  That version adds support for relocating the MSI-X vector
> table on vfio-pci devices, which recreates the MSI-X MMIO space elsewhere on
> the device.  A side-effect of this is that the vector table and PBA are
> properly sized so as not to collide.  The size of the tables remains wrong,
> but this only becomes a problem if the nvme code attempts to allocate >16
> vectors, which requires >15 vCPU (or host CPUs if the device is returned to
> host drivers after being assigned)(nvme appears to allocate 1 admin queue,
> plus a queue per CPU, each making use of an IRQ vector).  The QEMU vfio-pci
> device option is x-msix-relocation= which allows specifying the bar to use
> for the MSI-X tables, ex. bar0...bar5.  Since this device uses a 64bit bar0,
> we can either extend that BAR or choose another, excluding bar1, which is
> consumed by the upper half of bar0.  For instance, I tested with:
> 
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> ...
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>       </source>
>       <alias name='ua-sm2262'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> function='0x0'/>
>     </hostdev>
> ...
>   <qemu:commandline>
>     <qemu:arg value='-set'/>
>     <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
>   </qemu:commandline>
> 
> 
> (NB: "ua-" is a required prefix when specifying an alias)
> 
> A new virtual BAR appears in the guest hosting the MSI-X table and QEMU
> starts normally so long as the guest doesn't exceed 15 vCPUs.
> 
> The vCPU/pCPU count limitations are obviously not ideal, but hopefully this
> provides some degree of workaround for typical configurations.

Hi Alex, are you saying the above coding in the VM xml is all that is necessary
(with noted vCPU/pCPU count limitations) to successfully pass-through
sm2262-based controllers, without above kernel patch?  Or is a kernel patch
also necessary (if so, which one)?

thx,
-tom

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202055] Failed to PCI passthrough SSD with SMI SM2262 controller.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202055

--- Comment #44 from Alex Williamson (alex.williamson@redhat.com) ---
(In reply to LimeTech from comment #43)
> 
> Hi Alex, are you saying the above coding in the VM xml is all that is
> necessary (with noted vCPU/pCPU count limitations) to successfully
> pass-through sm2262-based controllers, without above kernel patch?  Or is a
> kernel patch also necessary (if so, which one)?

Yes, without kernel patch.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202055] Failed to PCI passthrough SSD with SMI SM2262 controller.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202055

FCL (Felix.leclair123@hotmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |Felix.leclair123@hotmail.co
                   |                            |m

--- Comment #46 from FCL (Felix.leclair123@hotmail.com) ---
Hey gents! 

Trying to apply this to a proxmox VM, any advice on how to do so? I'm running
on top of the 5.4.x linux kernel. I'm experiencing the same bug as in topic
name, using the SMI 2263 controller. (Adata as above)

Anything I can provide in terms of debugging data? Otherwise, any advice on how
to use the above non kernel patch? also comfortable doing a kernel recompile if
thats the better solution. Will be going to a FreeBSD (FreeNAS 11.3) VM with 4
threads and 72GB of ram alongside an LSI SAS controller. 

Platform is a Dual Xeon 5670 on a HP ml350 G6 mother board. 

Also attached but on a different VM is a Radeon 5600XT, an intel USB controller
and an intel Nic to a windows VM

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202055] Failed to PCI passthrough SSD with SMI SM2262 controller.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202055

--- Comment #47 from FCL (Felix.leclair123@hotmail.com) ---
(In reply to FCL from comment #46)
> Hey gents! 
> 
> Trying to apply this to a proxmox VM, any advice on how to do so? I'm
> running on top of the 5.4.x linux kernel. I'm experiencing the same bug as
> in topic name, using the SMI 2263 controller. (Adata as above)
> 
> Anything I can provide in terms of debugging data? Otherwise, any advice on
> how to use the above non kernel patch? also comfortable doing a kernel
> recompile if thats the better solution. Will be going to a FreeBSD (FreeNAS
> 11.3) VM with 4 threads and 72GB of ram alongside an LSI SAS controller. 
> 
> Platform is a Dual Xeon 5670 on a HP ml350 G6 mother board. 
> 
> Also attached but on a different VM is a Radeon 5600XT, an intel USB
> controller and an intel Nic to a windows VM

Found the solution without the need for a kernel recompile. in my Qemu conf
file added:
args: -set device.hostpci1.x-msix-relocation=bar2

Where hostpci1 is 
hostpci1: 08:00.0

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202055] Failed to PCI passthrough SSD with SMI SM2262 controller.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202055

plantroon@plantroon.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |plantroon@plantroon.com

--- Comment #48 from plantroon@plantroon.com ---
For Intel SSD 660p series, the latest firmware is required for pass-through to
work without issues. The latest one should be from mid 2020 at the time of
writing this.

Not even any workarounds or special settings for qemu/libvirt are needed, just
a simple pass-through setup as with any other PCIe device.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.