[bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Yi Zhang]

Hello

I found this issue during the nvme removal test, I did some debug code
found it was failed during nvme "CSTS – Controller Status" read, could
anyone help check if this is one HW or SW issue?


# nvme list
Node                  SN                   Model
             Namespace Usage                      Format           FW
Rev
--------------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme0n1          S48CNC0N400972B      Samsung SSD 983 DCT 960GB
             1           4.10  kB / 960.20  GB    512   B +  0 B
EDA5302Q
# lspci -s 87:00.0 -v
87:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
DeviceName: PCIe SSD in Slot 23 Bay 1
Subsystem: Samsung Electronics Co Ltd Device a801
Physical Slot: 7
Flags: bus master, fast devsel, latency 0, IRQ 46, NUMA node 1
Memory at c8600000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at c8610000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158] Power Budgeting <?>
Capabilities: [168] Secondary PCI Express
Capabilities: [188] Latency Tolerance Reporting
Capabilities: [190] L1 PM Substates
Kernel driver in use: nvme
Kernel modules: nvme

# echo 1 >/sys/bus/pci/devices/0000\:87\:00.0/remove
# echo 1 >/sys/bus/pci/rescan
# dmesg
[  251.864254] pci 0000:87:00.0: [144d:a808] type 00 class 0x010802
[  251.864286] pci 0000:87:00.0: reg 0x10: [mem 0xc8600000-0xc8603fff 64bit]
[  251.864337] pci 0000:87:00.0: reg 0x30: [mem 0xffff0000-0xffffffff pref]
[  251.889196] pci 0000:87:00.0: BAR 6: assigned [mem
0xc8600000-0xc860ffff pref]
[  251.889206] pci 0000:87:00.0: BAR 0: assigned [mem
0xc8610000-0xc8613fff 64bit]
[  251.889777] nvme nvme0: pci function 0000:87:00.0
[  251.889888] nvme nvme0: readl(dev->bar + NVME_REG_CSTS) == -1,
return - ENODEV
[  251.898057] nvme nvme0: nvme_pci_enable: -19
[  251.902821] nvme nvme0: Removing after probe failure status: -19


-- 
Best Regards,
  Yi Zhang


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Chaitanya Kulkarni]

On 9/21/21 6:56 PM, Yi Zhang wrote:
> Hello
> 
> I found this issue during the nvme removal test, I did some debug code
> found it was failed during nvme "CSTS – Controller Status" read, could
> anyone help check if this is one HW or SW issue?
> 
> 

One easiest way to isolate this issue is to test the steps on the QEMU
NVMe ctrl and update everyone here...
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Chaitanya Kulkarni]

On 9/21/21 6:56 PM, Yi Zhang wrote:
> Hello
> 
> I found this issue during the nvme removal test, I did some debug code
> found it was failed during nvme "CSTS – Controller Status" read, could
> anyone help check if this is one HW or SW issue?
> 
> 

also if you can specify the git repo, branch and git head it will be
great.
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Yi Zhang]

On Wed, Sep 22, 2021 at 10:32 AM Chaitanya Kulkarni
<chaitanyak@nvidia.com> wrote:
>
> On 9/21/21 6:56 PM, Yi Zhang wrote:
> > Hello
> >
> > I found this issue during the nvme removal test, I did some debug code
> > found it was failed during nvme "CSTS – Controller Status" read, could
> > anyone help check if this is one HW or SW issue?
> >
> >
>
> One easiest way to isolate this issue is to test the steps on the QEMU
> NVMe ctrl and update everyone here...

Hi Chaitanya
I tried another NVMe SSD[1], and it works well, seems this is special
for "Samsung SSD 983 DCT 960GB".
And the kernel I used was upstream 5.15.0-rc2.

[1]
# nvme list
Node             SN                   Model
        Namespace Usage                      Format           FW Rev
---------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme2n1     3080A09HTAFR         Dell Express Flash CD5 960G SFF
        1         469.18  MB / 960.20  GB    512   B +  0 B   1.2.0

-- 
Best Regards,
  Yi Zhang


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Chaitanya Kulkarni]

On 9/22/2021 4:21 AM, Yi Zhang wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Wed, Sep 22, 2021 at 10:32 AM Chaitanya Kulkarni
> <chaitanyak@nvidia.com> wrote:
>>
>> On 9/21/21 6:56 PM, Yi Zhang wrote:
>>> Hello
>>>
>>> I found this issue during the nvme removal test, I did some debug code
>>> found it was failed during nvme "CSTS – Controller Status" read, could
>>> anyone help check if this is one HW or SW issue?
>>>
>>>
>>
>> One easiest way to isolate this issue is to test the steps on the QEMU
>> NVMe ctrl and update everyone here...
> 
> Hi Chaitanya
> I tried another NVMe SSD[1], and it works well, seems this is special
> for "Samsung SSD 983 DCT 960GB".
> And the kernel I used was upstream 5.15.0-rc2.
> 

Thanks for the update.

You can also setup a NVMeOF Block device backend controller with the
nvme-loop transport and see if the problem exists.


> [1]
> # nvme list
> Node             SN                   Model
>          Namespace Usage                      Format           FW Rev
> ---------------- --------------------
> ---------------------------------------- ---------
> -------------------------- ---------------- --------
> /dev/nvme2n1     3080A09HTAFR         Dell Express Flash CD5 960G SFF
>          1         469.18  MB / 960.20  GB    512   B +  0 B   1.2.0
> 
> --
> Best Regards,
>    Yi Zhang
> 
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Adam Manzanares]

On Wed, Sep 22, 2021 at 09:56:47AM +0800, Yi Zhang wrote:
> Hello
> 
> I found this issue during the nvme removal test, I did some debug code
> found it was failed during nvme "CSTS – Controller Status" read, could
> anyone help check if this is one HW or SW issue?

Hello Yi,

What is the nvme removal test? When I get access to the test I will run it on 
the HW on my current machine to see what happens. I don't have the same SSD in 
my current test system, but I will make sure I am able to run the test on the 
same HW you are having issues with. 

Thanks,
Adam

> 
> 
> # nvme list
> Node                  SN                   Model
>              Namespace Usage                      Format           FW
> Rev
> --------------------- --------------------
> ---------------------------------------- ---------
> -------------------------- ---------------- --------
> /dev/nvme0n1          S48CNC0N400972B      Samsung SSD 983 DCT 960GB
>              1           4.10  kB / 960.20  GB    512   B +  0 B
> EDA5302Q
> # lspci -s 87:00.0 -v
> 87:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
> NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
> DeviceName: PCIe SSD in Slot 23 Bay 1
> Subsystem: Samsung Electronics Co Ltd Device a801
> Physical Slot: 7
> Flags: bus master, fast devsel, latency 0, IRQ 46, NUMA node 1
> Memory at c8600000 (64-bit, non-prefetchable) [size=16K]
> Expansion ROM at c8610000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
> Capabilities: [70] Express Endpoint, MSI 00
> Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
> Capabilities: [158] Power Budgeting <?>
> Capabilities: [168] Secondary PCI Express
> Capabilities: [188] Latency Tolerance Reporting
> Capabilities: [190] L1 PM Substates
> Kernel driver in use: nvme
> Kernel modules: nvme
> 
> # echo 1 >/sys/bus/pci/devices/0000\:87\:00.0/remove
> # echo 1 >/sys/bus/pci/rescan
> # dmesg
> [  251.864254] pci 0000:87:00.0: [144d:a808] type 00 class 0x010802
> [  251.864286] pci 0000:87:00.0: reg 0x10: [mem 0xc8600000-0xc8603fff 64bit]
> [  251.864337] pci 0000:87:00.0: reg 0x30: [mem 0xffff0000-0xffffffff pref]
> [  251.889196] pci 0000:87:00.0: BAR 6: assigned [mem
> 0xc8600000-0xc860ffff pref]
> [  251.889206] pci 0000:87:00.0: BAR 0: assigned [mem
> 0xc8610000-0xc8613fff 64bit]
> [  251.889777] nvme nvme0: pci function 0000:87:00.0
> [  251.889888] nvme nvme0: readl(dev->bar + NVME_REG_CSTS) == -1,
> return - ENODEV
> [  251.898057] nvme nvme0: nvme_pci_enable: -19
> [  251.902821] nvme nvme0: Removing after probe failure status: -19
> 
> 
> -- 
> Best Regards,
>   Yi Zhang
> 
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme@lists.infradead.org
> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=91c450cb-ce5f6836-91c5db84-000babff317b-8f74ab0f1a6811c9&q=1&e=8f702a72-b512-4abf-8e2f-4b7d6fc9fa4c&u=http*3A*2F*2Flists.infradead.org*2Fmailman*2Flistinfo*2Flinux-nvme__;JSUlJSUl!!EwVzqGoTKBqv-0DWAJBm!Et5uoUkTlQwi8ttA2vHe_Ab8xGrNj_J0VV7ab_fhyVorL0YT4vQ2k_36r0eUs-0p78pn$ 
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Yi Zhang]

On Fri, Sep 24, 2021 at 12:58 AM Adam Manzanares
<a.manzanares@samsung.com> wrote:
>
> On Wed, Sep 22, 2021 at 09:56:47AM +0800, Yi Zhang wrote:
> > Hello
> >
> > I found this issue during the nvme removal test, I did some debug code
> > found it was failed during nvme "CSTS – Controller Status" read, could
> > anyone help check if this is one HW or SW issue?
>
> Hello Yi,
>
> What is the nvme removal test? When I get access to the test I will run it on
> the HW on my current machine to see what happens. I don't have the same SSD in
> my current test system, but I will make sure I am able to run the test on the
> same HW you are having issues with.
>
Hi Adam
Here is the steps, and the server I used is DELL R730xd and DELL R640,
feel free to let me know if you need more info, thanks.

# lspci | grep -i nvme
87:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
NVMe SSD Controller SM981/PM981/PM983
#echo 1 >/sys/bus/pci/devices/0000\:87\:00.0/remove
#echo 1 >/sys/bus/pci/rescan

Thanks
Yi


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Keith Busch]

On Wed, Sep 22, 2021 at 09:56:47AM +0800, Yi Zhang wrote:
> # echo 1 >/sys/bus/pci/devices/0000\:87\:00.0/remove
> # echo 1 >/sys/bus/pci/rescan
> # dmesg
> [  251.864254] pci 0000:87:00.0: [144d:a808] type 00 class 0x010802
> [  251.864286] pci 0000:87:00.0: reg 0x10: [mem 0xc8600000-0xc8603fff 64bit]
> [  251.864337] pci 0000:87:00.0: reg 0x30: [mem 0xffff0000-0xffffffff pref]
> [  251.889196] pci 0000:87:00.0: BAR 6: assigned [mem 0xc8600000-0xc860ffff pref]
> [  251.889206] pci 0000:87:00.0: BAR 0: assigned [mem 0xc8610000-0xc8613fff 64bit]
> [  251.889777] nvme nvme0: pci function 0000:87:00.0
> [  251.889888] nvme nvme0: readl(dev->bar + NVME_REG_CSTS) == -1,
> return - ENODEV

An all 1's return almost certainly means the memory read request failed. The
test your described usually means the target did not properly configure the
memory range it was assigned. Is this directly attached to a root port?

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Adam Manzanares]

On Fri, Sep 24, 2021 at 10:33:14AM +0800, Yi Zhang wrote:
> On Fri, Sep 24, 2021 at 12:58 AM Adam Manzanares
> <a.manzanares@samsung.com> wrote:
> >
> > On Wed, Sep 22, 2021 at 09:56:47AM +0800, Yi Zhang wrote:
> > > Hello
> > >
> > > I found this issue during the nvme removal test, I did some debug code
> > > found it was failed during nvme "CSTS – Controller Status" read, could
> > > anyone help check if this is one HW or SW issue?
> >
> > Hello Yi,
> >
> > What is the nvme removal test? When I get access to the test I will run it on
> > the HW on my current machine to see what happens. I don't have the same SSD in
> > my current test system, but I will make sure I am able to run the test on the
> > same HW you are having issues with.
> >
> Hi Adam
> Here is the steps, and the server I used is DELL R730xd and DELL R640,
> feel free to let me know if you need more info, thanks.
> 
> # lspci | grep -i nvme
> 87:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
> NVMe SSD Controller SM981/PM981/PM983
> #echo 1 >/sys/bus/pci/devices/0000\:87\:00.0/remove
> #echo 1 >/sys/bus/pci/rescan

This is enough info for now. I can't reproduce the problem on the platform 
I am currently using. I will try to track down the same hw and fw that you 
are using, retest, and send an update.


> 
> Thanks
> Yi
> 
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [bug report] nvme removing after probe failed with pci rescan after nvme sysfs removal

[Yi Zhang]

On Fri, Sep 24, 2021 at 11:13 AM Keith Busch <kbusch@kernel.org> wrote:
>
> On Wed, Sep 22, 2021 at 09:56:47AM +0800, Yi Zhang wrote:
> > # echo 1 >/sys/bus/pci/devices/0000\:87\:00.0/remove
> > # echo 1 >/sys/bus/pci/rescan
> > # dmesg
> > [  251.864254] pci 0000:87:00.0: [144d:a808] type 00 class 0x010802
> > [  251.864286] pci 0000:87:00.0: reg 0x10: [mem 0xc8600000-0xc8603fff 64bit]
> > [  251.864337] pci 0000:87:00.0: reg 0x30: [mem 0xffff0000-0xffffffff pref]
> > [  251.889196] pci 0000:87:00.0: BAR 6: assigned [mem 0xc8600000-0xc860ffff pref]
> > [  251.889206] pci 0000:87:00.0: BAR 0: assigned [mem 0xc8610000-0xc8613fff 64bit]
> > [  251.889777] nvme nvme0: pci function 0000:87:00.0
> > [  251.889888] nvme nvme0: readl(dev->bar + NVME_REG_CSTS) == -1,
> > return - ENODEV
>
> An all 1's return almost certainly means the memory read request failed. The
> test your described usually means the target did not properly configure the
> memory range it was assigned. Is this directly attached to a root port?
>
Hi Keith
It was connected to the PCIe slot through a PCIe extender card, I
added the full dmesg here, not sure if it helps.

https://pastebin.com/QUP0Y4sT


--
Best Regards,
  Yi Zhang


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme