incorrect sysfs reported by realpath for nvme disk when nvme-core:multipath enabled

incorrect sysfs reported by realpath for nvme disk when nvme-core:multipath enabled

[Yi Zhang]

Hello

When I use 'realpath' to get the nvme sysfs on latest 5.3.0-rc8 for NVMe SSD, seems it reported incorrectly sysfs, could anyone help check it?


[1] nvme-core multipath=Y
# lspci | grep -i non
86:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 172Xa/172Xb (rev 01)
af:00.0 Non-Volatile memory controller: Intel Corporation NVMe DC SSD [3DNAND, Beta Rock Controller]

# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     BTLE8190003X2P0IGN   Dell Express Flash NVMe P4600 2.0TB AIC  1           2.00  TB /   2.00  TB      4 KiB +  0 B   QDV1DP15
/dev/nvme1n1           S39WNA0K201139 Dell Express Flash PM1725a 1.6TB AIC     1           1.60  TB /   1.60  TB    512   B +  0 B   1.1.1 

# cat /sys/module/nvme_core/parameters/multipath 
Y

# lsblk 
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 222.5G  0 disk 
├─sda1                         8:1    0   600M  0 part /boot/efi
├─sda2                         8:2    0     1G  0 part /boot
└─sda3                         8:3    0 220.9G  0 part 
  ├─rhel_rdma--perf--07-root 253:0    0    50G  0 lvm  /
  ├─rhel_rdma--perf--07-swap 253:1    0     4G  0 lvm  [SWAP]
  └─rhel_rdma--perf--07-home 253:2    0 166.9G  0 lvm  /home
nvme0n1                      259:0    0   1.8T  0 disk 
nvme1n1                      259:2    0   1.5T  0 disk 

# realpath  /sys/dev/block/259:0
/sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme0n1   --------> should it be /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme0/nvme0n1 ?

# realpath  /sys/dev/block/259:2
/sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1               --------> No pci address returned, bug?



[2] nvme-core multipath=N
# rmmod nvme
# rmmod nvme-core
# nvme list
# modprobe nvme-core multipath=N
# cat /sys/module/nvme_core/parameters/multipath
N
[root@rdma-perf-07 ~]$ modprobe nvme
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 222.5G  0 disk 
├─sda1                         8:1    0   600M  0 part /boot/efi
├─sda2                         8:2    0     1G  0 part /boot
└─sda3                         8:3    0 220.9G  0 part 
  ├─rhel_rdma--perf--07-root 253:0    0    50G  0 lvm  /
  ├─rhel_rdma--perf--07-swap 253:1    0     4G  0 lvm  [SWAP]
  └─rhel_rdma--perf--07-home 253:2    0 166.9G  0 lvm  /home
nvme1n1                      259:0    0   1.8T  0 disk 
nvme0n1                      259:2    0   1.5T  0 disk 

[root@rdma-perf-07 ~]$ realpath /sys/dev/block/259:0
/sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme1n1

[root@rdma-perf-07 ~]$ realpath /sys/dev/block/259:2
/sys/devices/pci0000:85/0000:85:02.0/0000:86:00.0/nvme/nvme0/nvme0n1


   
Best Regards,
  Yi Zhang



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: incorrect sysfs reported by realpath for nvme disk when nvme-core:multipath enabled

[Keith Busch]

On Fri, Sep 13, 2019 at 01:40:50PM -0400, Yi Zhang wrote:
> # realpath  /sys/dev/block/259:0
> /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme0n1   --------> should it be /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme0/nvme0n1 ?

The sysfs links are correct. The controller-namespacenaming is just not
related, which is what you're observing.

This was confusing to enough people that we've since changed the naming
to something people expect to see, but that's staged in 5.4. Given the
number of people who find this confusing, this might be worth making a
stable patch.
 
> # realpath  /sys/dev/block/259:2
> /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1               --------> No pci address returned, bug?

For mulitpath capable controllers, the parent of a namespace is the
subsystem rather than a controller. The subsystem is a virtual device
that doesn't have a physical pci address.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: incorrect sysfs reported by realpath for nvme disk when nvme-core:multipath enabled

[Sagi Grimberg]

>> # realpath  /sys/dev/block/259:0
>> /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme0n1   --------> should it be /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme0/nvme0n1 ?
> 
> The sysfs links are correct. The controller-namespacenaming is just not
> related, which is what you're observing.
> 
> This was confusing to enough people that we've since changed the naming
> to something people expect to see, but that's staged in 5.4. Given the
> number of people who find this confusing, this might be worth making a
> stable patch.

I agree here.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: incorrect sysfs reported by realpath for nvme disk when nvme-core:multipath enabled

[Keith Busch]

On Fri, Sep 13, 2019 at 11:43:40AM -0700, Sagi Grimberg wrote:
> > This was confusing to enough people that we've since changed the naming
> > to something people expect to see, but that's staged in 5.4. Given the
> > number of people who find this confusing, this might be worth making a
> > stable patch.
> 
> I agree here.

Okay, I'll forward the commit to stable once merged to mainline. I don't
think we can count on the stable bot to pick this one up. We only need
this back to 4.19 as all the previous stables with this naming scheme
have be EOL'ed.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: incorrect sysfs reported by realpath for nvme disk when nvme-core:multipath enabled

[Yi Zhang]

On 9/14/19 11:19 PM, Keith Busch wrote:
> On Fri, Sep 13, 2019 at 11:43:40AM -0700, Sagi Grimberg wrote:
>>> This was confusing to enough people that we've since changed the naming
>>> to something people expect to see, but that's staged in 5.4. Given the
>>> number of people who find this confusing, this might be worth making a
>>> stable patch.
>> I agree here.

Hi Keith

Thanks for the info, I just test the nvme-5.4, and it works as expect.

[root@rdma-perf-07 ~]$ lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 222.5G  0 disk
├─sda1                         8:1    0   600M  0 part /boot/efi
├─sda2                         8:2    0     1G  0 part /boot
└─sda3                         8:3    0 220.9G  0 part
   ├─rhel_rdma--perf--07-root 253:0    0    50G  0 lvm  /
   ├─rhel_rdma--perf--07-swap 253:1    0     4G  0 lvm  [SWAP]
   └─rhel_rdma--perf--07-home 253:2    0 166.9G  0 lvm  /home
nvme1n1                      259:0    0   1.8T  0 disk
└─nvme1n1p1                  259:1    0   1.8T  0 part
nvme0n1                      259:3    0   1.5T  0 disk
[root@rdma-perf-07 ~]$ realpath /sys/dev/block/259:0
/sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme1n1
[root@rdma-perf-07 ~]$ realpath /sys/dev/block/259:3
/sys/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1

Thanks

Yi


> Okay, I'll forward the commit to stable once merged to mainline. I don't
> think we can count on the stable bot to pick this one up. We only need
> this back to 4.19 as all the previous stables with this naming scheme
> have be EOL'ed.
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme