Re: PCI pass-through problem for SN570 NVME SSD

From: Jan Beulich <jbeulich@suse.com>
To: "G.R." <firemeteor@users.sourceforge.net>
Cc: xen-devel <xen-devel@lists.xen.org>,
	"Roger Pau Monné" <roger.pau@citrix.com>
Subject: Re: PCI pass-through problem for SN570 NVME SSD
Date: Wed, 6 Jul 2022 08:33:41 +0200	[thread overview]
Message-ID: <ca4e8b79-c831-8c09-6398-b76852dfde53@suse.com> (raw)
In-Reply-To: <CAKhsbWYytY8_3DTqpq4u+SvywuDSK5dgy8dzm1T3TFdP5MJaDQ@mail.gmail.com>

On 06.07.2022 08:25, G.R. wrote:
> On Tue, Jul 5, 2022 at 7:59 PM Jan Beulich <jbeulich@suse.com> wrote:
>> Nothing useful in there. Yet independent of that I guess we need to
>> separate the issues you're seeing. Otherwise it'll be impossible to
>> know what piece of data belongs where.
> Yep, I think I'm seeing several different issues here:
> 1. The FLR related DPC / AER message seen on the 1st attempt only when
> pciback tries to seize and release the SN570
>     - Later-on pciback operations appear just fine.
> 2. MSI-X preparation failure message that shows up each time the SN570
> is seized by pciback or when it's passed to domU.
> 3. XEN tries to map BAR from two devices to the same page
> 4. The "write-back to unknown field" message in QEMU log that goes
> away with permissive=1 passthrough config.
> 5. The "irq 16: nobody cared" message shows up *sometimes* in a
> pattern that I haven't figured out  (See attached)
> 6. The FreeBSD domU sees the device but fails to use it because low
> level commands sent to it are aborted.
> 7. The device does not return to the pci-assignable-list when the domU
> it was assigned shuts-down. (See attached)
> 
> #3 appears to be a known issue that could be worked around with
> patches from the list.
> I suspect #1 may have something to do with the device itself. It's
> still not clear if it's deadly or just annoying.
> I was able to update the firmware to the latest version and confirmed
> that the new firmware didn't make any noticeable difference.
> 
> I suspect issue #2, #4, #5, #6, #7 may be related, and the
> pass-through was not completely successful...
> 
> Should I expect a debug build of XEN hypervisor to give better
> diagnose messages, without the debug patch that Roger mentioned?

Well, "expect" is perhaps too much to say, but with problems like
yours (and even more so with multiple ones) using a debug
hypervisor (or kernel, if there such a build mode existed) is imo
always a good idea. As is using as up-to-date a version as
possible.

Jan