All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: Multipath ID not equal to LUN scsi ID
@ 2017-07-11  3:49 liuqing
  2017-07-11  9:07 ` Steffen Maier
  0 siblings, 1 reply; 8+ messages in thread
From: liuqing @ 2017-07-11  3:49 UTC (permalink / raw)
  To: maier, dm-devel; +Cc: liuqing


[-- Attachment #1.1: Type: text/plain, Size: 4179 bytes --]

Hi Steffen,
Once a new arrived LUN mapped we will do rescan by "echo '- - -' > /sys/class/scsi_host/host2/scan". After rescan only the scsi_id tool give the right serial id, udevadm still got the prvious one.
I have monitor the udev event by udevadm monitor while mapping a new LUN to the host, who will reuse the original path. No add event is triggerred, only dm-X emits a change event. 
But if the original path is deleted(removed) then add event will be triggerred.

Flush the old WWID could make the WWID correct but the size is still incorrect as following. And 36005076300810eadf800000000000155 is actually 5GB. 
[root@localhost ~]# multipath -v2
Jul 11 12:36:06 | sdp: using deprecated getuid callout
Jul 11 12:36:06 | sdq: using deprecated getuid callout
create: 36005076300810eadf800000000000155 undef IBM,2145
size=8.0G features='1 queue_if_no_path' hwhandler='0' wp=undef


I built the tool using latest code and tried both attribute_id and getuid_callout. The issue exist in both configuration.


> Dear list,
> We have a FC storage and using multipathd to manager the FC paths.
> I've met an issue in this environment. The following is how to recreate the issue.
>
> =======
> 1. Map a LUN to host with LUN ID 0,
> 2. rescan fc_host, a new path will be found by multipath.
> 3. Unmap LUN 0.  path will failed as following.
> [root@localhost sys]# multipath -ll
> Jul 10 18:41:50 | sdp: couldn't get asymmetric access state
> Jul 10 18:41:50 | sdq: couldn't get asymmetric access state
> 36005076300810eadf800000000000156 dm-3 IBM,2145
> size=8.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
> |-+- policy='service-time 0' prio=0 status=enabled
> | `- 2:0:0:0 sdp 8:240 failed faulty running
> `-+- policy='service-time 0' prio=0 status=enabled
>    `- 2:0:1:0 sdq 65:0  failed faulty running
> 4. Map another LUN which have different ID_SERIAL but with the same LUN ID(0).
 
Did you "rescan" the SCSI device via sysfs to let Linux know that this
is now in fact a different device?
 
AFAIK, Linux decodes SCSI sense data for LUNs remapped on the storage
target and emits a udev event, but I'm not aware of any default udev
rule that would actually react. The kernel itself does not react other
than creating the uevent.
 
> Multipath(check_path function) will set the paths up, tur checker will successfully test the LUN.
> [root@localhost sys]# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/36005076300810eadf800000000000156
> 36005076300810eadf800000000000155
>   I think this is not right, although I do not remove the device by 'echo 1> /sys/block/sdp/device/delete'. A multipath is identified by LUN scsi ID, in this situation the multipath
> ID is not equal to LUN scsi ID. And if I add the previous LUN back, the paths will also be added to that multipath id(36005076300810eadf800000000000156).
>
> Any suggestion on how to make it create a new node in /dev with correct scsi ID? Thanks.
 
If you assume the old mapped volume would be gone, you could also flush
the mpath map with the old WWID.
 
After the SCSI device rescan it should create a new map with the new WWID.
 
>
> ===========
> [root@localhost sys]# cat /etc/multipath.conf
> defaults {
>      features "0"
>      no_path_retry           queue
>      getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
 
IIRC, using an explicit getuid_callout has been deprecated in later
releases of multipath tools and it would use builtin code to determine
the necessary information, potentially even using inquiry data cached in
the kernel (exported via sysfs).
 
If your toolchain is recent enough, would it work without this conf setting?
 
> }
> blacklist {
>      devnode "sd[a-m]"
> }
 
--
Mit freundlichen Gr??en / Kind regards
Steffen Maier
 
Linux on z Systems Development
 
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


liuqing@huayun.com

[-- Attachment #1.2: Type: text/html, Size: 12625 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Multipath ID not equal to LUN scsi ID
  2017-07-11  3:49 Multipath ID not equal to LUN scsi ID liuqing
@ 2017-07-11  9:07 ` Steffen Maier
  2017-07-11  9:40   ` Steffen Maier
  0 siblings, 1 reply; 8+ messages in thread
From: Steffen Maier @ 2017-07-11  9:07 UTC (permalink / raw)
  To: dm-devel

On 07/11/2017 05:49 AM, liuqing@huayun.com wrote:
> Once a new arrived LUN mapped we will do rescan by "echo '- - -' > /sys/class/scsi_host/host2/scan".

I was referring to a rescan of the scsi device,

https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.lhdd/lhdd_t_fcp_wrk_rescan.html

not a scan of the Scsi_Host.
The latter likely keeps the old scsi device with its old properties 
being oblivious to the volume remapping on the storage target; it's 
about discovering new scsi devices previously not being known to Linux.

> After rescan only the scsi_id tool give the right serial id, udevadm still got the prvious one.
> I have monitor the udev event by udevadm monitor while mapping a new LUN to the host, who will reuse the original path. No add event is triggerred, only dm-X emits a change event.
> But if the original path is deleted(removed) then add event will be triggerred.
> 
> Flush the old WWID could make the WWID correct but the size is still incorrect as following. And 36005076300810eadf800000000000155 is actually 5GB.

likely same reason as above

> I built the tool using latest code and tried both attribute_id and getuid_callout. The issue exist in both configuration.
> 
> 
>> Dear list,
>> We have a FC storage and using multipathd to manager the FC paths.
>> I've met an issue in this environment. The following is how to recreate the issue.
>>
>> =======
>> 1. Map a LUN to host with LUN ID 0,
>> 2. rescan fc_host, a new path will be found by multipath.
>> 3. Unmap LUN 0.  path will failed as following.
>> [root@localhost sys]# multipath -ll
>> Jul 10 18:41:50 | sdp: couldn't get asymmetric access state
>> Jul 10 18:41:50 | sdq: couldn't get asymmetric access state
>> 36005076300810eadf800000000000156 dm-3 IBM,2145
>> size=8.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
>> |-+- policy='service-time 0' prio=0 status=enabled
>> | `- 2:0:0:0 sdp 8:240 failed faulty running
>> `-+- policy='service-time 0' prio=0 status=enabled
>>     `- 2:0:1:0 sdq 65:0  failed faulty running
>> 4. Map another LUN which have different ID_SERIAL but with the same LUN ID(0).
>   
> Did you "rescan" the SCSI device via sysfs to let Linux know that this
> is now in fact a different device?
>   
> AFAIK, Linux decodes SCSI sense data for LUNs remapped on the storage
> target and emits a udev event, but I'm not aware of any default udev
> rule that would actually react. The kernel itself does not react other
> than creating the uevent.


-- 
Mit freundlichen Grüßen / Kind regards
Steffen Maier

Linux on z Systems Development

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Multipath ID not equal to LUN scsi ID
  2017-07-11  9:07 ` Steffen Maier
@ 2017-07-11  9:40   ` Steffen Maier
  0 siblings, 0 replies; 8+ messages in thread
From: Steffen Maier @ 2017-07-11  9:40 UTC (permalink / raw)
  To: dm-devel

Since Linux does not really automatically handle target volume 
remapping, you might be better of with explicitly tearing down the old 
volume in Linux before any unmap/remap on the target. This is already 
tricky enough to get right.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html


Resizing of the same volume on the target is a special case which should 
work, but even that one includes a rescan of the affected scsi devices 
(paths) as pre-req.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/online-iscsi-resizing.html#resizing-fc-logical-units
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/online-iscsi-resizing.html#idm8373264
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/DM_Multipath/online_device_resize.html


NB: None of the URLs provided are specific to a distro or device driver. 
Those were just example documentation references I could find quickly.

On 07/11/2017 11:07 AM, Steffen Maier wrote:
> On 07/11/2017 05:49 AM, liuqing@huayun.com wrote:
>> Once a new arrived LUN mapped we will do rescan by "echo '- - -' > 
>> /sys/class/scsi_host/host2/scan".
> 
> I was referring to a rescan of the scsi device,
> 
> https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.lhdd/lhdd_t_fcp_wrk_rescan.html 
> 
> 
> not a scan of the Scsi_Host.
> The latter likely keeps the old scsi device with its old properties 
> being oblivious to the volume remapping on the storage target; it's 
> about discovering new scsi devices previously not being known to Linux.
> 
>> After rescan only the scsi_id tool give the right serial id, udevadm 
>> still got the prvious one.
>> I have monitor the udev event by udevadm monitor while mapping a new 
>> LUN to the host, who will reuse the original path. No add event is 
>> triggerred, only dm-X emits a change event.
>> But if the original path is deleted(removed) then add event will be 
>> triggerred.
>>
>> Flush the old WWID could make the WWID correct but the size is still 
>> incorrect as following. And 36005076300810eadf800000000000155 is 
>> actually 5GB.
> 
> likely same reason as above
> 
>> I built the tool using latest code and tried both attribute_id and 
>> getuid_callout. The issue exist in both configuration.
>>
>>
>>> Dear list,
>>> We have a FC storage and using multipathd to manager the FC paths.
>>> I've met an issue in this environment. The following is how to 
>>> recreate the issue.
>>>
>>> =======
>>> 1. Map a LUN to host with LUN ID 0,
>>> 2. rescan fc_host, a new path will be found by multipath.
>>> 3. Unmap LUN 0.  path will failed as following.
>>> [root@localhost sys]# multipath -ll
>>> Jul 10 18:41:50 | sdp: couldn't get asymmetric access state
>>> Jul 10 18:41:50 | sdq: couldn't get asymmetric access state
>>> 36005076300810eadf800000000000156 dm-3 IBM,2145
>>> size=8.0G features='2 queue_if_no_path retain_attached_hw_handler' 
>>> hwhandler='1 alua' wp=rw
>>> |-+- policy='service-time 0' prio=0 status=enabled
>>> | `- 2:0:0:0 sdp 8:240 failed faulty running
>>> `-+- policy='service-time 0' prio=0 status=enabled
>>>     `- 2:0:1:0 sdq 65:0  failed faulty running
>>> 4. Map another LUN which have different ID_SERIAL but with the same 
>>> LUN ID(0).
>> Did you "rescan" the SCSI device via sysfs to let Linux know that this
>> is now in fact a different device?
>> AFAIK, Linux decodes SCSI sense data for LUNs remapped on the storage
>> target and emits a udev event, but I'm not aware of any default udev
>> rule that would actually react. The kernel itself does not react other
>> than creating the uevent.
> 
> 

-- 
Mit freundlichen Grüßen / Kind regards
Steffen Maier

Linux on z Systems Development

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Multipath ID not equal to LUN scsi ID
@ 2017-07-12  2:14 liuqing
  0 siblings, 0 replies; 8+ messages in thread
From: liuqing @ 2017-07-12  2:14 UTC (permalink / raw)
  To: maier, dm-devel; +Cc: liuqing


[-- Attachment #1.1: Type: text/plain, Size: 5172 bytes --]

Thanks Steffen.

Our management tool(Openstack) will do as the clean as the document(https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html)
in normal situation, but Openstack do have bugs which will lead to paths of unmapped LUN not been removed. This will cause data crush if the original paths are added back. 
Although Openstack should not leave the paths, multipath should found out different LUNs by different ID_SERIAL. Any FC administor may forget to delete the paths before adding a new LUN with old LUN ID.

Flush the path and rescan scsi bus and then do a multipath -v2 could make the new path currect. I am thinking whether we could add similar steps in multipath-tool when multipath detected a offlined path become
online(no udev event about new arrived LUN in this situation).  Not an expert of the multipath code, do you think this is reasonable?


===============================================
Since Linux does not really automatically handle target volume
remapping, you might be better of with explicitly tearing down the old
volume in Linux before any unmap/remap on the target. This is already
tricky enough to get right.
 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html
 
 
Resizing of the same volume on the target is a special case which should
work, but even that one includes a rescan of the affected scsi devices
(paths) as pre-req.
 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/online-iscsi-resizing.html#resizing-fc-logical-units
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/online-iscsi-resizing.html#idm8373264
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/DM_Multipath/online_device_resize.html
 
 
NB: None of the URLs provided are specific to a distro or device driver.
Those were just example documentation references I could find quickly.
 
On 07/11/2017 11:07 AM, Steffen Maier wrote:
> On 07/11/2017 05:49 AM, liuqing@huayun.com wrote:
>> Once a new arrived LUN mapped we will do rescan by "echo '- - -' >
>> /sys/class/scsi_host/host2/scan".
>
> I was referring to a rescan of the scsi device,
>
> https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.lhdd/lhdd_t_fcp_wrk_rescan.html
>
>
> not a scan of the Scsi_Host.
> The latter likely keeps the old scsi device with its old properties
> being oblivious to the volume remapping on the storage target; it's
> about discovering new scsi devices previously not being known to Linux.
>
>> After rescan only the scsi_id tool give the right serial id, udevadm
>> still got the prvious one.
>> I have monitor the udev event by udevadm monitor while mapping a new
>> LUN to the host, who will reuse the original path. No add event is
>> triggerred, only dm-X emits a change event.
>> But if the original path is deleted(removed) then add event will be
>> triggerred.
>>
>> Flush the old WWID could make the WWID correct but the size is still
>> incorrect as following. And 36005076300810eadf800000000000155 is
>> actually 5GB.
>
> likely same reason as above
>
>> I built the tool using latest code and tried both attribute_id and
>> getuid_callout. The issue exist in both configuration.
>>
>>
>>> Dear list,
>>> We have a FC storage and using multipathd to manager the FC paths.
>>> I've met an issue in this environment. The following is how to
>>> recreate the issue.
>>>
>>> =======
>>> 1. Map a LUN to host with LUN ID 0,
>>> 2. rescan fc_host, a new path will be found by multipath.
>>> 3. Unmap LUN 0.  path will failed as following.
>>> [root@localhost sys]# multipath -ll
>>> Jul 10 18:41:50 | sdp: couldn't get asymmetric access state
>>> Jul 10 18:41:50 | sdq: couldn't get asymmetric access state
>>> 36005076300810eadf800000000000156 dm-3 IBM,2145
>>> size=8.0G features='2 queue_if_no_path retain_attached_hw_handler'
>>> hwhandler='1 alua' wp=rw
>>> |-+- policy='service-time 0' prio=0 status=enabled
>>> | `- 2:0:0:0 sdp 8:240 failed faulty running
>>> `-+- policy='service-time 0' prio=0 status=enabled
>>>     `- 2:0:1:0 sdq 65:0  failed faulty running
>>> 4. Map another LUN which have different ID_SERIAL but with the same
>>> LUN ID(0).
>> Did you "rescan" the SCSI device via sysfs to let Linux know that this
>> is now in fact a different device?
>> AFAIK, Linux decodes SCSI sense data for LUNs remapped on the storage
>> target and emits a udev event, but I'm not aware of any default udev
>> rule that would actually react. The kernel itself does not react other
>> than creating the uevent.
>
>
 
--
Mit freundlichen Gr??en / Kind regards
Steffen Maier
 
Linux on z Systems Development
 
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


liuqing@huayun.com

[-- Attachment #1.2: Type: text/html, Size: 16943 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Multipath ID not equal to LUN scsi ID
  2017-07-10 13:55 ` Martin Wilck
@ 2017-07-11  1:32   ` liuqing
  0 siblings, 0 replies; 8+ messages in thread
From: liuqing @ 2017-07-11  1:32 UTC (permalink / raw)
  To: Martin Wilck, dm-devel; +Cc: liuqing


[-- Attachment #1.1: Type: text/plain, Size: 11736 bytes --]

Hi Martin,
Thanks for the response.
I built multipath using the latest code to test, but I also see this issue in our product environment(multipath-tools v0.4.9 (05/33, 2016)).

I use getuid_callout is due to the id_serial is not correct as following:
[root@localhost ~]# multipath -ll
36005076300810eadf800000000000156 dm-3 IBM,2145
size=8.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 2:0:0:0 sdp 8:240 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 2:0:1:0 sdq 65:0  active ready running

[root@localhost ~]# udevadm info /dev/sdp
P: /devices/pci0000:40/0000:40:02.0/0000:42:00.1/host2/rport-2:0-0/target2:0:0/2:0:0:0/block/sdp
N: sdp
S: disk/by-id/scsi-36005076300810eadf800000000000156
S: disk/by-id/wwn-0x6005076300810eadf800000000000156
S: disk/by-path/pci-0000:42:00.1-fc-0x500507680307ab7c-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-36005076300810eadf800000000000156 /dev/disk/by-id/wwn-0x6005076300810eadf800000000000156 /dev/disk/by-path/pci-0000:42:00.1-fc-0x500507680307ab7c-lun-0
E: DEVNAME=/dev/sdp
E: DEVPATH=/devices/pci0000:40/0000:40:02.0/0000:42:00.1/host2/rport-2:0-0/target2:0:0/2:0:0:0/block/sdp
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=2145
E: ID_MODEL_ENC=2145\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_PATH=pci-0000:42:00.1-fc-0x500507680307ab7c-lun-0
E: ID_PATH_TAG=pci-0000_42_00_1-fc-0x500507680307ab7c-lun-0
E: ID_REVISION=0000
E: ID_SCSI=1
E: ID_SCSI_SERIAL=00c02043ab7eXX00
E: ID_SERIAL=36005076300810eadf800000000000156
E: ID_SERIAL_SHORT=6005076300810eadf800000000000156
E: ID_TARGET_PORT=0
E: ID_TYPE=disk
E: ID_VENDOR=IBM
E: ID_VENDOR_ENC=IBM\x20\x20\x20\x20\x20
E: ID_WWN=0x6005076300810ead
E: ID_WWN_VENDOR_EXTENSION=0xf800000000000156
E: ID_WWN_WITH_EXTENSION=0x6005076300810eadf800000000000156
E: MAJOR=8
E: MINOR=240
E: MPATH_SBIN_PATH=/sbin
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=262761901185

[root@localhost ~]# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sdp
36005076300810eadf800000000000155

multipath -v3
===========
Jul 11 10:08:49 | libdevmapper version 1.02.135-RHEL7 (2016-11-16)
Jul 11 10:08:49 | DM multipath kernel driver v1.9.0
Jul 11 10:08:49 | loading /lib64/multipath/libchecktur.so checker
Jul 11 10:08:49 | loading /lib64/multipath/libprioconst.so prioritizer
Jul 11 10:08:49 | sda: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sda: device node name blacklisted
Jul 11 10:08:49 | sdb: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdb: device node name blacklisted
Jul 11 10:08:49 | sdk: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdk: device node name blacklisted
Jul 11 10:08:49 | sdl: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdl: device node name blacklisted
Jul 11 10:08:49 | sdm: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdm: device node name blacklisted
Jul 11 10:08:49 | sdc: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdc: device node name blacklisted
Jul 11 10:08:49 | sdd: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdd: device node name blacklisted
Jul 11 10:08:49 | sde: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sde: device node name blacklisted
Jul 11 10:08:49 | sdf: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdf: device node name blacklisted
Jul 11 10:08:49 | sdg: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdg: device node name blacklisted
Jul 11 10:08:49 | sdh: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdh: device node name blacklisted
Jul 11 10:08:49 | sdi: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdi: device node name blacklisted
Jul 11 10:08:49 | sdj: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdj: device node name blacklisted
Jul 11 10:08:49 | sdp: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdp: mask = 0x1f
Jul 11 10:08:49 | sdp: dev_t = 8:240
Jul 11 10:08:49 | sdp: size = 16777216
Jul 11 10:08:49 | sdp: vendor = IBM
Jul 11 10:08:49 | sdp: product = 2145
Jul 11 10:08:49 | sdp: rev = 0000
Jul 11 10:08:49 | sdp: h:b:t:l = 2:0:0:0
Jul 11 10:08:49 | SCSI target 2:0:0 -> FC rport 2:0-0
Jul 11 10:08:49 | sdp: tgt_node_name = 0x500507680303ab7c
Jul 11 10:08:49 | sdp: path state = running
Jul 11 10:08:49 | sdp: 8192 cyl, 64 heads, 32 sectors/track, start at 0
Jul 11 10:08:49 | sdp: serial = 00c02043ab7eXX00
Jul 11 10:08:49 | sdp: get_state
Jul 11 10:08:49 | sdp: detect_checker = yes (setting: multipath.conf defaults/devices section)
Jul 11 10:08:49 | sdp: path_checker = tur (setting: array autodetected)
Jul 11 10:08:49 | sdp: checker timeout = 30 ms (setting: multipath internal)
Jul 11 10:08:49 | sdp: state = up
Jul 11 10:08:49 | sdp: getuid = "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n" (setting: multipath.conf defaults/devices section)
Jul 11 10:08:49 | sdp: using deprecated getuid callout
Jul 11 10:08:49 | formatted callout = /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sdp
Jul 11 10:08:49 | sdp: uid = 36005076300810eadf800000000000155 (callout)
Jul 11 10:08:49 | sdp: detect_prio = yes (setting: multipath.conf defaults/devices section)
Jul 11 10:08:49 | loading /lib64/multipath/libprioalua.so prioritizer
Jul 11 10:08:49 | sdp: prio = alua (setting: array autodetected)
Jul 11 10:08:49 | sdp: prio args = "" (setting: array autodetected)
Jul 11 10:08:49 | sdp: reported target port group is 0
Jul 11 10:08:49 | sdp: aas = 00 [active/optimized]
Jul 11 10:08:49 | sdp: alua prio = 50
Jul 11 10:08:49 | sdq: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdq: mask = 0x1f
Jul 11 10:08:49 | sdq: dev_t = 65:0
Jul 11 10:08:49 | sdq: size = 16777216
Jul 11 10:08:49 | sdq: vendor = IBM
Jul 11 10:08:49 | sdq: product = 2145
Jul 11 10:08:49 | sdq: rev = 0000
Jul 11 10:08:49 | sdq: h:b:t:l = 2:0:1:0
Jul 11 10:08:49 | SCSI target 2:0:1 -> FC rport 2:0-1
Jul 11 10:08:49 | sdq: tgt_node_name = 0x500507680303ab7d
Jul 11 10:08:49 | sdq: path state = running
Jul 11 10:08:49 | sdq: 8192 cyl, 64 heads, 32 sectors/track, start at 0
Jul 11 10:08:49 | sdq: serial = 00c02043ab7eXX00
Jul 11 10:08:49 | sdq: get_state
Jul 11 10:08:49 | sdq: detect_checker = yes (setting: multipath.conf defaults/devices section)
Jul 11 10:08:49 | sdq: path_checker = tur (setting: array autodetected)
Jul 11 10:08:49 | sdq: checker timeout = 30 ms (setting: multipath internal)
Jul 11 10:08:49 | sdq: state = up
Jul 11 10:08:49 | sdq: getuid = "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n" (setting: multipath.conf defaults/devices section)
Jul 11 10:08:49 | sdq: using deprecated getuid callout
Jul 11 10:08:49 | formatted callout = /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sdq
Jul 11 10:08:49 | sdq: uid = 36005076300810eadf800000000000155 (callout)
Jul 11 10:08:49 | sdq: detect_prio = yes (setting: multipath.conf defaults/devices section)
Jul 11 10:08:49 | sdq: prio = alua (setting: array autodetected)
Jul 11 10:08:49 | sdq: prio args = "" (setting: array autodetected)
Jul 11 10:08:49 | sdq: reported target port group is 1
Jul 11 10:08:49 | sdq: aas = 01 [active/non-optimized]
Jul 11 10:08:49 | sdq: alua prio = 10
Jul 11 10:08:49 | dm-0: blacklisted, udev property missing
Jul 11 10:08:49 | dm-1: blacklisted, udev property missing
Jul 11 10:08:49 | dm-2: blacklisted, udev property missing
Jul 11 10:08:49 | dm-3: blacklisted, udev property missing
===== paths list =====
uuid                              hcil    dev dev_t pri dm_st chk_st vend/prod
36005076300810eadf800000000000155 2:0:0:0 sdp 8:240 50  undef undef  IBM,2145
36005076300810eadf800000000000155 2:0:1:0 sdq 65:0  10  undef undef  IBM,2145
Jul 11 10:08:49 | params = 2 queue_if_no_path retain_attached_hw_handler 1 alua 2 1 service-time 0 1 2 8:240 1 1 service-time 0 1 2 65:0 1 1
Jul 11 10:08:49 | status = 2 0 1 0 2 1 A 0 1 2 8:240 A 5 0 1 E 0 1 2 65:0 A 6 0 1
Jul 11 10:08:49 | 36005076300810eadf800000000000156: disassemble map [2 queue_if_no_path retain_attached_hw_handler 1 alua 2 1 service-time 0 1 2 8:240 1 1 service-time 0 1 2 65:0 1 1 ]
Jul 11 10:08:49 | 8:240: path wwid appears to have changed. Using map wwid.

Jul 11 10:08:49 | 65:0: path wwid appears to have changed. Using map wwid.

Jul 11 10:08:49 | 36005076300810eadf800000000000156: disassemble status [2 0 1 0 2 1 A 0 1 2 8:240 A 5 0 1 E 0 1 2 65:0 A 6 0 1 ]
Jul 11 10:08:49 | sdp: udev property ID_WWN whitelisted
Jul 11 10:08:49 | sdq: udev property ID_WWN whitelisted
Jul 11 10:08:49 | tur checker refcount 2
Jul 11 10:08:49 | alua prioritizer refcount 2
Jul 11 10:08:49 | tur checker refcount 1
Jul 11 10:08:49 | alua prioritizer refcount 1
Jul 11 10:08:49 | unloading alua prioritizer
Jul 11 10:08:49 | unloading const prioritizer
Jul 11 10:08:49 | unloading tur checker


liuqing@huayun.com
 
From: Martin Wilck
Date: 2017-07-10 21:55
To: dm-devel; liuqing
Subject: Re: [dm-devel] Multipath ID not equal to LUN scsi ID
On Mon, 2017-07-10 at 17:58 +0800, liuqing@huayun.com wrote:
> Dear list,
> We have a FC storage and using multipathd to manager the FC paths.
> I've met an issue in this environment. The following is how to
> recreate the issue.
 
Which version of multipath-tools have you been using? Could you please
run multipathd with "-v 3" and provide the logs of the procedure below?
 
Is there a specific reason why you are using the "getuid_callout"
option rather than "uid_attribute"?
 
Regards,
Martin
 
> 
> =======
> 1. Map a LUN to host with LUN ID 0,
> 2. rescan fc_host, a new path will be found by multipath.
> 3. Unmap LUN 0.  path will failed as following.
> [root@localhost sys]# multipath -ll
> Jul 10 18:41:50 | sdp: couldn't get asymmetric access state
> Jul 10 18:41:50 | sdq: couldn't get asymmetric access state
> 36005076300810eadf800000000000156 dm-3 IBM,2145
> size=8.0G features='2 queue_if_no_path retain_attached_hw_handler' hw
> handler='1 alua' wp=rw
> |-+- policy='service-time 0' prio=0 status=enabled
> | `- 2:0:0:0 sdp 8:240 failed faulty running
> `-+- policy='service-time 0' prio=0 status=enabled
>   `- 2:0:1:0 sdq 65:0  failed faulty running
> 4. Map another LUN which have different ID_SERIAL but with the same
> LUN ID(0). Multipath(check_path function) will set the paths up, tur
> checker will successfully test the LUN. 
> [root@localhost sys]# /lib/udev/scsi_id --whitelisted --replace-
> whitespace --device=/dev/mapper/36005076300810eadf800000000000156
> 36005076300810eadf800000000000155
>  I think this is not right, although I do not remove the device by
> 'echo 1> /sys/block/sdp/device/delete'. A multipath is identified by
> LUN scsi ID, in this situation the multipath
> ID is not equal to LUN scsi ID. And if I add the previous LUN back,
> the paths will also be added to that multipath
> id(36005076300810eadf800000000000156).
> 
> Any suggestion on how to make it create a new node in /dev with
> correct scsi ID? Thanks.
> 
> 
> ===========
> [root@localhost sys]# cat /etc/multipath.conf
> defaults {
>     features "0"
>     no_path_retry           queue
>     getuid_callout "/lib/udev/scsi_id --whitelisted --replace-
> whitespace --device=/dev/%n"
> }
> blacklist {
>     devnode "sd[a-m]"
> }
> 
> liuqing@huayun.com
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
 
-- 
Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
 

[-- Attachment #1.2: Type: text/html, Size: 20738 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Multipath ID not equal to LUN scsi ID
  2017-07-10  9:58 liuqing
  2017-07-10 13:02 ` Steffen Maier
@ 2017-07-10 13:55 ` Martin Wilck
  2017-07-11  1:32   ` liuqing
  1 sibling, 1 reply; 8+ messages in thread
From: Martin Wilck @ 2017-07-10 13:55 UTC (permalink / raw)
  To: dm-devel, liuqing

On Mon, 2017-07-10 at 17:58 +0800, liuqing@huayun.com wrote:
> Dear list,
> We have a FC storage and using multipathd to manager the FC paths.
> I've met an issue in this environment. The following is how to
> recreate the issue.

Which version of multipath-tools have you been using? Could you please
run multipathd with "-v 3" and provide the logs of the procedure below?

Is there a specific reason why you are using the "getuid_callout"
option rather than "uid_attribute"?

Regards,
Martin

> 
> =======
> 1. Map a LUN to host with LUN ID 0,
> 2. rescan fc_host, a new path will be found by multipath.
> 3. Unmap LUN 0.  path will failed as following.
> [root@localhost sys]# multipath -ll
> Jul 10 18:41:50 | sdp: couldn't get asymmetric access state
> Jul 10 18:41:50 | sdq: couldn't get asymmetric access state
> 36005076300810eadf800000000000156 dm-3 IBM,2145
> size=8.0G features='2 queue_if_no_path retain_attached_hw_handler' hw
> handler='1 alua' wp=rw
> |-+- policy='service-time 0' prio=0 status=enabled
> | `- 2:0:0:0 sdp 8:240 failed faulty running
> `-+- policy='service-time 0' prio=0 status=enabled
>   `- 2:0:1:0 sdq 65:0  failed faulty running
> 4. Map another LUN which have different ID_SERIAL but with the same
> LUN ID(0). Multipath(check_path function) will set the paths up, tur
> checker will successfully test the LUN. 
> [root@localhost sys]# /lib/udev/scsi_id --whitelisted --replace-
> whitespace --device=/dev/mapper/36005076300810eadf800000000000156
> 36005076300810eadf800000000000155
>  I think this is not right, although I do not remove the device by
> 'echo 1> /sys/block/sdp/device/delete'. A multipath is identified by
> LUN scsi ID, in this situation the multipath
> ID is not equal to LUN scsi ID. And if I add the previous LUN back,
> the paths will also be added to that multipath
> id(36005076300810eadf800000000000156).
> 
> Any suggestion on how to make it create a new node in /dev with
> correct scsi ID? Thanks.
> 
> 
> ===========
> [root@localhost sys]# cat /etc/multipath.conf
> defaults {
>     features "0"
>     no_path_retry           queue
>     getuid_callout "/lib/udev/scsi_id --whitelisted --replace-
> whitespace --device=/dev/%n"
> }
> blacklist {
>     devnode "sd[a-m]"
> }
> 
> liuqing@huayun.com
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

-- 
Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Multipath ID not equal to LUN scsi ID
  2017-07-10  9:58 liuqing
@ 2017-07-10 13:02 ` Steffen Maier
  2017-07-10 13:55 ` Martin Wilck
  1 sibling, 0 replies; 8+ messages in thread
From: Steffen Maier @ 2017-07-10 13:02 UTC (permalink / raw)
  To: dm-devel

On 07/10/2017 11:58 AM, liuqing@huayun.com wrote:
> Dear list,
> We have a FC storage and using multipathd to manager the FC paths.
> I've met an issue in this environment. The following is how to recreate the issue.
> 
> =======
> 1. Map a LUN to host with LUN ID 0,
> 2. rescan fc_host, a new path will be found by multipath.
> 3. Unmap LUN 0.  path will failed as following.
> [root@localhost sys]# multipath -ll
> Jul 10 18:41:50 | sdp: couldn't get asymmetric access state
> Jul 10 18:41:50 | sdq: couldn't get asymmetric access state
> 36005076300810eadf800000000000156 dm-3 IBM,2145
> size=8.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
> |-+- policy='service-time 0' prio=0 status=enabled
> | `- 2:0:0:0 sdp 8:240 failed faulty running
> `-+- policy='service-time 0' prio=0 status=enabled
>    `- 2:0:1:0 sdq 65:0  failed faulty running
> 4. Map another LUN which have different ID_SERIAL but with the same LUN ID(0).

Did you "rescan" the SCSI device via sysfs to let Linux know that this 
is now in fact a different device?

AFAIK, Linux decodes SCSI sense data for LUNs remapped on the storage 
target and emits a udev event, but I'm not aware of any default udev 
rule that would actually react. The kernel itself does not react other 
than creating the uevent.

> Multipath(check_path function) will set the paths up, tur checker will successfully test the LUN.
> [root@localhost sys]# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/36005076300810eadf800000000000156
> 36005076300810eadf800000000000155
>   I think this is not right, although I do not remove the device by 'echo 1> /sys/block/sdp/device/delete'. A multipath is identified by LUN scsi ID, in this situation the multipath
> ID is not equal to LUN scsi ID. And if I add the previous LUN back, the paths will also be added to that multipath id(36005076300810eadf800000000000156).
> 
> Any suggestion on how to make it create a new node in /dev with correct scsi ID? Thanks.

If you assume the old mapped volume would be gone, you could also flush 
the mpath map with the old WWID.

After the SCSI device rescan it should create a new map with the new WWID.

> 
> ===========
> [root@localhost sys]# cat /etc/multipath.conf
> defaults {
>      features "0"
>      no_path_retry           queue
>      getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"

IIRC, using an explicit getuid_callout has been deprecated in later 
releases of multipath tools and it would use builtin code to determine 
the necessary information, potentially even using inquiry data cached in 
the kernel (exported via sysfs).

If your toolchain is recent enough, would it work without this conf setting?

> }
> blacklist {
>      devnode "sd[a-m]"
> }

-- 
Mit freundlichen Grüßen / Kind regards
Steffen Maier

Linux on z Systems Development

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Multipath ID not equal to LUN scsi ID
@ 2017-07-10  9:58 liuqing
  2017-07-10 13:02 ` Steffen Maier
  2017-07-10 13:55 ` Martin Wilck
  0 siblings, 2 replies; 8+ messages in thread
From: liuqing @ 2017-07-10  9:58 UTC (permalink / raw)
  To: dm-devel; +Cc: liuqing


[-- Attachment #1.1: Type: text/plain, Size: 1880 bytes --]

Dear list,
We have a FC storage and using multipathd to manager the FC paths.
I've met an issue in this environment. The following is how to recreate the issue.

=======
1. Map a LUN to host with LUN ID 0,
2. rescan fc_host, a new path will be found by multipath.
3. Unmap LUN 0.  path will failed as following.
[root@localhost sys]# multipath -ll
Jul 10 18:41:50 | sdp: couldn't get asymmetric access state
Jul 10 18:41:50 | sdq: couldn't get asymmetric access state
36005076300810eadf800000000000156 dm-3 IBM,2145
size=8.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| `- 2:0:0:0 sdp 8:240 failed faulty running
`-+- policy='service-time 0' prio=0 status=enabled
  `- 2:0:1:0 sdq 65:0  failed faulty running
4. Map another LUN which have different ID_SERIAL but with the same LUN ID(0). Multipath(check_path function) will set the paths up, tur checker will successfully test the LUN. 
[root@localhost sys]# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/36005076300810eadf800000000000156
36005076300810eadf800000000000155
 I think this is not right, although I do not remove the device by 'echo 1> /sys/block/sdp/device/delete'. A multipath is identified by LUN scsi ID, in this situation the multipath
ID is not equal to LUN scsi ID. And if I add the previous LUN back, the paths will also be added to that multipath id(36005076300810eadf800000000000156).

Any suggestion on how to make it create a new node in /dev with correct scsi ID? Thanks.


===========
[root@localhost sys]# cat /etc/multipath.conf
defaults {
    features "0"
    no_path_retry           queue
    getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
}
blacklist {
    devnode "sd[a-m]"
}



liuqing@huayun.com

[-- Attachment #1.2: Type: text/html, Size: 3367 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2017-07-12  2:14 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-11  3:49 Multipath ID not equal to LUN scsi ID liuqing
2017-07-11  9:07 ` Steffen Maier
2017-07-11  9:40   ` Steffen Maier
  -- strict thread matches above, loose matches on Subject: below --
2017-07-12  2:14 liuqing
2017-07-10  9:58 liuqing
2017-07-10 13:02 ` Steffen Maier
2017-07-10 13:55 ` Martin Wilck
2017-07-11  1:32   ` liuqing

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.