ALUA - rescan device capacity on zero sized block devices

ALUA - rescan device capacity on zero sized block devices

[Thomas Wouters]

[-- Attachment #1.1: Type: text/plain, Size: 3157 bytes --]

Hi, 

We're performing some tests with open-iscsi and multipath on two 3par servers and their peer persistence feature. 
3par is a commercial storage solution that uses ALUA to allow failover. 
We have two connections from each 3par server to a linux server. 

Every 3par server has two network controllers, so on our linux server we initiate 4 iscsi connections. 
Multipath detects that two of these connections are active paths (both to the same 3par device, that is active at that point) and two are ghost paths, to the passive 3par device. 

At this moment we have four block devices, the active paths show the actual device size and the standby paths show the devices as zero sized: 

# multipath -ll 
360002ac000000000000000420001510c dm-3 3PARdata,VV 
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw 
|-+- policy='round-robin 0' prio=130 status=active 
| |- 48:0:0:123 sdc 8:32 active ready running 
| `- 50:0:0:123 sdb 8:16 active ready running 
`-+- policy='round-robin 0' prio=1 status=enabled 
|- 49:0:0:123 sdd 8:48 active ghost running 
`- 51:0:0:123 sde 8:64 active ghost running 

# cat /sys/block/sdb/size 
209715200 
# cat /sys/block/sdc/size 
209715200 
# cat /sys/block/sdd/size 
0 
# cat /sys/block/sde/size 
0 

As soon as we perform a switchover on the 3par systems, multipath detects the priority changes and switches paths but the new active paths fail. 
We believe this is because 3par doesn't allow us to read the capacity of the disk on a standby path - and we have proof of this in the logs: 

Apr 13 15:05:12 deb-3par-test kernel: [ 40.079736] sd 5:0:0:0: [sdc] READ CAPACITY failed 

Unfortunately, once we perform the switchover on 3par, the capacity of those old ghost paths, now active paths, is not re-read. The multipath device is therefore reduced to a size of 0 and the filesystem becomes unavailable. 

If we only login on the two active paths without starting multipath, perform a switchover, then login on the two new active paths and start multipath, we have four block devices with a non-zero size and we can perform switchovers at will without any issues. 

We've found some older discussions describing these issues on the scsi target-devel and dm-devel mailinglists: 
- http://permalink.gmane.org/gmane.linux.scsi.target.devel/6531 
- https://www.redhat.com/archives/dm-devel/2014-July/msg00156.html 

As far as we can conclude after reading these messages, it is correct behavior for disallowing READ CAPACITY on ghost paths. However, once the path becomes active, we do need a reread of the capacity in order for the path to be functional... 

We've created a workaround for our issue but we're not sure we're going in the right direction. 

diff --git a/multipathd/main.c b/multipathd/main.c 
index f876258..ff32681 100644 
--- a/multipathd/main.c 
+++ b/multipathd/main.c 
@@ -1235,6 +1235,11 @@ check_path (struct vectors * vecs, struct path * pp) 

pp->chkrstate = newstate; 
if (newstate != pp->state) { 
+ 
+ if (newstate == PATH_UP && pp->size != pp->mpp->size ) { 
+ sysfs_attr_set_value(pp->udev, "device/rescan", "1\n",2); 
+ } 
+ 
int oldstate = pp->state; 
pp->state = newstate; 


[-- Attachment #1.2: Type: text/html, Size: 3742 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: ALUA - rescan device capacity on zero sized block devices

[Bart Van Assche]

On 04/13/15 17:32, Thomas Wouters wrote:
> We're performing some tests with open-iscsi and multipath on two 3par
> servers and their peer persistence feature.
> 3par is a commercial storage solution that uses ALUA to allow failover.
> We have two connections from each 3par server to a linux server.
>
> Every 3par server has two network controllers, so on our linux server we
> initiate 4 iscsi connections.
> Multipath detects that two of these connections are active paths (both
> to the same 3par device, that is active at that point) and two are ghost
> paths, to the passive 3par device.
>
> At this moment we have four block devices, the active paths show the
> actual device size and the standby paths show the devices as zero sized:
>
> # multipath -ll
> 360002ac000000000000000420001510c dm-3 3PARdata,VV
> size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
> |-+- policy='round-robin 0' prio=130 status=active
> | |- 48:0:0:123 sdc 8:32 active ready running
> | `- 50:0:0:123 sdb 8:16 active ready running
> `-+- policy='round-robin 0' prio=1 status=enabled
>    |- 49:0:0:123 sdd 8:48 active ghost running
>    `- 51:0:0:123 sde 8:64 active ghost running
>
> # cat /sys/block/sdb/size
> 209715200
> # cat /sys/block/sdc/size
> 209715200
> # cat /sys/block/sdd/size
> 0
> # cat /sys/block/sde/size
> 0
>
> As soon as we perform a switchover on the 3par systems, multipath
> detects the priority changes and switches paths but the new active paths
> fail.
> We believe this is because 3par doesn't allow us to read the capacity of
> the disk on a standby path - and we have proof of this in the logs:
>
> Apr 13 15:05:12 deb-3par-test kernel: [   40.079736] sd 5:0:0:0: [sdc]
> READ CAPACITY failed
>
> Unfortunately, once we perform the switchover on 3par, the capacity of
> those old ghost paths, now active paths, is not re-read.  The multipath
> device is therefore reduced to a size of 0 and the filesystem becomes
> unavailable.
>
> If we only login on the two active paths without starting multipath,
> perform a switchover, then login on the two new active paths and start
> multipath, we have four block devices with a non-zero size and we can
> perform switchovers at will without any issues.
>
> We've found some older discussions describing these issues on the scsi
> target-devel and dm-devel mailinglists:
> - http://permalink.gmane.org/gmane.linux.scsi.target.devel/6531
> - https://www.redhat.com/archives/dm-devel/2014-July/msg00156.html
>
> As far as we can conclude after reading these messages, it is correct
> behavior for disallowing READ CAPACITY on ghost paths.  However, once
> the path becomes active, we do need a reread of the capacity in order
> for the path to be functional...
>
> We've created a workaround for our issue but we're not sure we're going
> in the right direction.
>
> diff --git a/multipathd/main.c b/multipathd/main.c
> index f876258..ff32681 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -1235,6 +1235,11 @@ check_path (struct vectors * vecs, struct path * pp)
>
> pp->chkrstate = newstate;
> if (newstate != pp->state) {
> +
> + if (newstate == PATH_UP && pp->size != pp->mpp->size ) {
> + sysfs_attr_set_value(pp->udev, "device/rescan", "1\n",2);
> + }
> +
> int oldstate = pp->state;
> pp->state = newstate;

Hello Thomas,

The above patch will trigger a rescan after every failover and failback. 
I'm afraid that will slow down failover and failback, especially if the 
number of LUNs is large. I would appreciate it if the capacity would be 
reexamined only if it is not yet known.

Thanks,

Bart.

Re: ALUA - rescan device capacity on zero sized block devices

[Thomas Wouters]

----- On Apr 13, 2015, at 7:44 PM, Bart Van Assche bart.vanassche@sandisk.com wrote:
> On 04/13/15 17:32, Thomas Wouters wrote:
>> We're performing some tests with open-iscsi and multipath on two 3par
>> servers and their peer persistence feature.
>> 3par is a commercial storage solution that uses ALUA to allow failover.
>> We have two connections from each 3par server to a linux server.
>>
>> Every 3par server has two network controllers, so on our linux server we
>> initiate 4 iscsi connections.
>> Multipath detects that two of these connections are active paths (both
>> to the same 3par device, that is active at that point) and two are ghost
>> paths, to the passive 3par device.
>>
>> At this moment we have four block devices, the active paths show the
>> actual device size and the standby paths show the devices as zero sized:
>>
>> # multipath -ll
>> 360002ac000000000000000420001510c dm-3 3PARdata,VV
>> size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
>> |-+- policy='round-robin 0' prio=130 status=active
>> | |- 48:0:0:123 sdc 8:32 active ready running
>> | `- 50:0:0:123 sdb 8:16 active ready running
>> `-+- policy='round-robin 0' prio=1 status=enabled
>>    |- 49:0:0:123 sdd 8:48 active ghost running
>>    `- 51:0:0:123 sde 8:64 active ghost running
>>
>> # cat /sys/block/sdb/size
>> 209715200
>> # cat /sys/block/sdc/size
>> 209715200
>> # cat /sys/block/sdd/size
>> 0
>> # cat /sys/block/sde/size
>> 0
>>
>> As soon as we perform a switchover on the 3par systems, multipath
>> detects the priority changes and switches paths but the new active paths
>> fail.
>> We believe this is because 3par doesn't allow us to read the capacity of
>> the disk on a standby path - and we have proof of this in the logs:
>>
>> Apr 13 15:05:12 deb-3par-test kernel: [   40.079736] sd 5:0:0:0: [sdc]
>> READ CAPACITY failed
>>
>> Unfortunately, once we perform the switchover on 3par, the capacity of
>> those old ghost paths, now active paths, is not re-read.  The multipath
>> device is therefore reduced to a size of 0 and the filesystem becomes
>> unavailable.
>>
>> If we only login on the two active paths without starting multipath,
>> perform a switchover, then login on the two new active paths and start
>> multipath, we have four block devices with a non-zero size and we can
>> perform switchovers at will without any issues.
>>
>> We've found some older discussions describing these issues on the scsi
>> target-devel and dm-devel mailinglists:
>> - http://permalink.gmane.org/gmane.linux.scsi.target.devel/6531
>> - https://www.redhat.com/archives/dm-devel/2014-July/msg00156.html
>>
>> As far as we can conclude after reading these messages, it is correct
>> behavior for disallowing READ CAPACITY on ghost paths.  However, once
>> the path becomes active, we do need a reread of the capacity in order
>> for the path to be functional...
>>
>> We've created a workaround for our issue but we're not sure we're going
>> in the right direction.
>>
>> diff --git a/multipathd/main.c b/multipathd/main.c
>> index f876258..ff32681 100644
>> --- a/multipathd/main.c
>> +++ b/multipathd/main.c
>> @@ -1235,6 +1235,11 @@ check_path (struct vectors * vecs, struct path * pp)
>>
>> pp->chkrstate = newstate;
>> if (newstate != pp->state) {
>> +
>> + if (newstate == PATH_UP && pp->size != pp->mpp->size ) {
>> + sysfs_attr_set_value(pp->udev, "device/rescan", "1\n",2);
>> + }
>> +
>> int oldstate = pp->state;
>> pp->state = newstate;
> 
> Hello Thomas,
> 
> The above patch will trigger a rescan after every failover and failback.
> I'm afraid that will slow down failover and failback, especially if the
> number of LUNs is large. I would appreciate it if the capacity would be
> reexamined only if it is not yet known.
> 
> Thanks,
> 
> Bart.

Hi Bart,

I realize this is not the best way to handle the situation.
This patch was never meant to be implemented as is but more of a clarification of how we look at the issue.

If we resize a lun on the storage servers, the new size can't be read on standby paths.
This means that if a failover occurs for any reason we could end up with a corrupt block device?

Is there a better way to rescan the capacity? Using sysfs_attr_set_value() like this doesn't look clean to me.

Would it make sense to make this a configurable setting which is used for systems that don't allow READ CAPACITY on standby paths?

Thomas

Re: ALUA - rescan device capacity on zero sized block devices

[Bart Van Assche]

On 04/14/15 09:20, Thomas Wouters wrote:
> ----- On Apr 13, 2015, at 7:44 PM, Bart Van Assche bart.vanassche@sandisk.com wrote:
>> On 04/13/15 17:32, Thomas Wouters wrote:
>>> We're performing some tests with open-iscsi and multipath on two 3par
>>> servers and their peer persistence feature.
>>> 3par is a commercial storage solution that uses ALUA to allow failover.
>>> We have two connections from each 3par server to a linux server.
>>>
>>> Every 3par server has two network controllers, so on our linux server we
>>> initiate 4 iscsi connections.
>>> Multipath detects that two of these connections are active paths (both
>>> to the same 3par device, that is active at that point) and two are ghost
>>> paths, to the passive 3par device.
>>>
>>> At this moment we have four block devices, the active paths show the
>>> actual device size and the standby paths show the devices as zero sized:
>>>
>>> # multipath -ll
>>> 360002ac000000000000000420001510c dm-3 3PARdata,VV
>>> size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
>>> |-+- policy='round-robin 0' prio=130 status=active
>>> | |- 48:0:0:123 sdc 8:32 active ready running
>>> | `- 50:0:0:123 sdb 8:16 active ready running
>>> `-+- policy='round-robin 0' prio=1 status=enabled
>>>     |- 49:0:0:123 sdd 8:48 active ghost running
>>>     `- 51:0:0:123 sde 8:64 active ghost running
>>>
>>> # cat /sys/block/sdb/size
>>> 209715200
>>> # cat /sys/block/sdc/size
>>> 209715200
>>> # cat /sys/block/sdd/size
>>> 0
>>> # cat /sys/block/sde/size
>>> 0
>>>
>>> As soon as we perform a switchover on the 3par systems, multipath
>>> detects the priority changes and switches paths but the new active paths
>>> fail.
>>> We believe this is because 3par doesn't allow us to read the capacity of
>>> the disk on a standby path - and we have proof of this in the logs:
>>>
>>> Apr 13 15:05:12 deb-3par-test kernel: [   40.079736] sd 5:0:0:0: [sdc]
>>> READ CAPACITY failed
>>>
>>> Unfortunately, once we perform the switchover on 3par, the capacity of
>>> those old ghost paths, now active paths, is not re-read.  The multipath
>>> device is therefore reduced to a size of 0 and the filesystem becomes
>>> unavailable.
>>>
>>> If we only login on the two active paths without starting multipath,
>>> perform a switchover, then login on the two new active paths and start
>>> multipath, we have four block devices with a non-zero size and we can
>>> perform switchovers at will without any issues.
>>>
>>> We've found some older discussions describing these issues on the scsi
>>> target-devel and dm-devel mailinglists:
>>> - http://permalink.gmane.org/gmane.linux.scsi.target.devel/6531
>>> - https://www.redhat.com/archives/dm-devel/2014-July/msg00156.html
>>>
>>> As far as we can conclude after reading these messages, it is correct
>>> behavior for disallowing READ CAPACITY on ghost paths.  However, once
>>> the path becomes active, we do need a reread of the capacity in order
>>> for the path to be functional...
>>>
>>> We've created a workaround for our issue but we're not sure we're going
>>> in the right direction.
>>>
>>> diff --git a/multipathd/main.c b/multipathd/main.c
>>> index f876258..ff32681 100644
>>> --- a/multipathd/main.c
>>> +++ b/multipathd/main.c
>>> @@ -1235,6 +1235,11 @@ check_path (struct vectors * vecs, struct path * pp)
>>>
>>> pp->chkrstate = newstate;
>>> if (newstate != pp->state) {
>>> +
>>> + if (newstate == PATH_UP && pp->size != pp->mpp->size ) {
>>> + sysfs_attr_set_value(pp->udev, "device/rescan", "1\n",2);
>>> + }
>>> +
>>> int oldstate = pp->state;
>>> pp->state = newstate;
>>
>> The above patch will trigger a rescan after every failover and failback.
>> I'm afraid that will slow down failover and failback, especially if the
>> number of LUNs is large. I would appreciate it if the capacity would be
>> reexamined only if it is not yet known.
>
> I realize this is not the best way to handle the situation.
> This patch was never meant to be implemented as is but more of a
 > clarification of how we look at the issue.
>
> If we resize a lun on the storage servers, the new size can't be read on
 > standby paths. This means that if a failover occurs for any reason we
 > could end up with a corrupt block device?
>
> Is there a better way to rescan the capacity? Using sysfs_attr_set_value()
 > like this doesn't look clean to me.
>
> Would it make sense to make this a configurable setting which is used for
 > systems that don't allow READ CAPACITY on standby paths?

Hello Thomas,

There exists at least one storage array model that accepts the READ 
CAPACITY command on standby paths. The solution I would prefer is that 
all storage arrays would behave this way.

Regarding LUN resizing: the SCSI specs require that a storage array 
reports CAPACITY DATA HAS CHANGED after a LUN has been resized. It 
should be possible to modify the SCSI core such that it rescans a device 
after having received this unit attention condition. The virtio_scsi 
already driver rescans a device after having received that unit 
attention condition. From drivers/scsi/virtio_scsi.c:

	/* Handle "Parameters changed", "Mode parameters changed", and
	   "Capacity data has changed".  */
	if (asc == 0x2a && (... || ascq == 0x09))
		scsi_rescan_device(&sdev->sdev_gendev);

A quote from SBC-4:

Any time the READ CAPACITY (10) parameter data (see 5.15.2) or the READ 
CAPACITY (16) parameter data (see 5.16.2) changes (e.g., when a FORMAT 
UNIT command or a MODE SELECT command causes a change to the logical 
block length or protection information, or when a vendor specific 
mechanism causes a change), then the device server shall establish a 
unit attention condition for the SCSI initiator port (see SAM-5) 
associated with each I_T nexus, except the I_T nexus on which the 
command causing the change was received with the additional sense code 
set to CAPACITY DATA HAS CHANGED.

Bart.

Re: ALUA - rescan device capacity on zero sized block devices

[Hannes Reinecke]

On 04/14/2015 09:20 AM, Thomas Wouters wrote:
> ----- On Apr 13, 2015, at 7:44 PM, Bart Van Assche bart.vanassche@sandisk.com wrote:
>> On 04/13/15 17:32, Thomas Wouters wrote:
>>> We're performing some tests with open-iscsi and multipath on two 3par
>>> servers and their peer persistence feature.
>>> 3par is a commercial storage solution that uses ALUA to allow failover.
>>> We have two connections from each 3par server to a linux server.
>>>
>>> Every 3par server has two network controllers, so on our linux server we
>>> initiate 4 iscsi connections.
>>> Multipath detects that two of these connections are active paths (both
>>> to the same 3par device, that is active at that point) and two are ghost
>>> paths, to the passive 3par device.
>>>
>>> At this moment we have four block devices, the active paths show the
>>> actual device size and the standby paths show the devices as zero sized:
>>>
>>> # multipath -ll
>>> 360002ac000000000000000420001510c dm-3 3PARdata,VV
>>> size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
>>> |-+- policy='round-robin 0' prio=130 status=active
>>> | |- 48:0:0:123 sdc 8:32 active ready running
>>> | `- 50:0:0:123 sdb 8:16 active ready running
>>> `-+- policy='round-robin 0' prio=1 status=enabled
>>>    |- 49:0:0:123 sdd 8:48 active ghost running
>>>    `- 51:0:0:123 sde 8:64 active ghost running
>>>
>>> # cat /sys/block/sdb/size
>>> 209715200
>>> # cat /sys/block/sdc/size
>>> 209715200
>>> # cat /sys/block/sdd/size
>>> 0
>>> # cat /sys/block/sde/size
>>> 0
>>>
>>> As soon as we perform a switchover on the 3par systems, multipath
>>> detects the priority changes and switches paths but the new active paths
>>> fail.
>>> We believe this is because 3par doesn't allow us to read the capacity of
>>> the disk on a standby path - and we have proof of this in the logs:
>>>
>>> Apr 13 15:05:12 deb-3par-test kernel: [   40.079736] sd 5:0:0:0: [sdc]
>>> READ CAPACITY failed
>>>
>>> Unfortunately, once we perform the switchover on 3par, the capacity of
>>> those old ghost paths, now active paths, is not re-read.  The multipath
>>> device is therefore reduced to a size of 0 and the filesystem becomes
>>> unavailable.
>>>
>>> If we only login on the two active paths without starting multipath,
>>> perform a switchover, then login on the two new active paths and start
>>> multipath, we have four block devices with a non-zero size and we can
>>> perform switchovers at will without any issues.
>>>
>>> We've found some older discussions describing these issues on the scsi
>>> target-devel and dm-devel mailinglists:
>>> - http://permalink.gmane.org/gmane.linux.scsi.target.devel/6531
>>> - https://www.redhat.com/archives/dm-devel/2014-July/msg00156.html
>>>
>>> As far as we can conclude after reading these messages, it is correct
>>> behavior for disallowing READ CAPACITY on ghost paths.  However, once
>>> the path becomes active, we do need a reread of the capacity in order
>>> for the path to be functional...
>>>
>>> We've created a workaround for our issue but we're not sure we're going
>>> in the right direction.
>>>
>>> diff --git a/multipathd/main.c b/multipathd/main.c
>>> index f876258..ff32681 100644
>>> --- a/multipathd/main.c
>>> +++ b/multipathd/main.c
>>> @@ -1235,6 +1235,11 @@ check_path (struct vectors * vecs, struct path * pp)
>>>
>>> pp->chkrstate = newstate;
>>> if (newstate != pp->state) {
>>> +
>>> + if (newstate == PATH_UP && pp->size != pp->mpp->size ) {
>>> + sysfs_attr_set_value(pp->udev, "device/rescan", "1\n",2);
>>> + }
>>> +
>>> int oldstate = pp->state;
>>> pp->state = newstate;
>>
>> Hello Thomas,
>>
>> The above patch will trigger a rescan after every failover and failback.
>> I'm afraid that will slow down failover and failback, especially if the
>> number of LUNs is large. I would appreciate it if the capacity would be
>> reexamined only if it is not yet known.
>>
>> Thanks,
>>
>> Bart.
> 
> Hi Bart,
> 
> I realize this is not the best way to handle the situation.
> This patch was never meant to be implemented as is but more of a clarification of how
> we look at the issue.
> 
> If we resize a lun on the storage servers, the new size can't be read on standby paths.
> This means that if a failover occurs for any reason we could end up with a corrupt block device?
> 
> Is there a better way to rescan the capacity? Using sysfs_attr_set_value() like this
> doesn't look clean to me.
> 
> Would it make sense to make this a configurable setting which is used for systems
> that don't allow READ CAPACITY on standby paths?
> 
Finally someone tripped over it.

We've noticed some time ago that the current multipath-tools
implementation does not follow SPC-3 as it relies on a valid
capacity when assembling paths.

I've done a patch for this (cf commit
cb4d86aac16bedaa22fdda8ee14130dfa2a98563), but this is only
a partial solution, as it will only allow to add paths with a zero
capacity to existing multipath maps.
If we come across a _new_ path with zero capacity it'll still be
skipped. Also the current 'rescan' functionality within the kernel
is very rudimentary, and would not allow a full device update.
IE if a device has a capacity of zero we currently will not
automatically update it upon switch-over.

What we can try is to listen to uevents for the device capacity
change or ALUA state changes, and retry the read capacity for those
events. Which would tie in nicely with the proposed device rescan
mechanism discussed at LSF.

Hmm.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: ALUA - rescan device capacity on zero sized block devices

[Bart Van Assche]

On 04/14/15 10:14, Hannes Reinecke wrote:
> What we can try is to listen to uevents for the device capacity
> change or ALUA state changes, and retry the read capacity for those
> events.

Any mechanism that relies on uevents would be asynchronous. We need a 
way to ensure that the capacity data is up to date before any 
application starts using that data. In this context that means before 
multipath starts queueing I/O to a path. This might be challenging when 
processing uevents asynchronously ...

Bart.

Re: ALUA - rescan device capacity on zero sized block devices

[Christophe Varoqui]

[-- Attachment #1.1: Type: text/plain, Size: 6785 bytes --]

Hi,

the 3par arrays peer persistence feature can perfectly be disabled (on a
per-initiator granularity, if I remember correctly). With this feature
disabled, the linux multipathing stack behaves correctly.

Thomas, I see the point of this array controller feature when you can not
or don't want to use the systems' multipathing layer, but you don't seem to
be in this case. Is there a reason why you are testing this combination ?

Best regards,
Christophe Varoqui

On Tue, Apr 14, 2015 at 9:45 AM, Bart Van Assche <bart.vanassche@sandisk.com
> wrote:

> On 04/14/15 09:20, Thomas Wouters wrote:
>
>> ----- On Apr 13, 2015, at 7:44 PM, Bart Van Assche
>> bart.vanassche@sandisk.com wrote:
>>
>>> On 04/13/15 17:32, Thomas Wouters wrote:
>>>
>>>> We're performing some tests with open-iscsi and multipath on two 3par
>>>> servers and their peer persistence feature.
>>>> 3par is a commercial storage solution that uses ALUA to allow failover.
>>>> We have two connections from each 3par server to a linux server.
>>>>
>>>> Every 3par server has two network controllers, so on our linux server we
>>>> initiate 4 iscsi connections.
>>>> Multipath detects that two of these connections are active paths (both
>>>> to the same 3par device, that is active at that point) and two are ghost
>>>> paths, to the passive 3par device.
>>>>
>>>> At this moment we have four block devices, the active paths show the
>>>> actual device size and the standby paths show the devices as zero sized:
>>>>
>>>> # multipath -ll
>>>> 360002ac000000000000000420001510c dm-3 3PARdata,VV
>>>> size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
>>>> |-+- policy='round-robin 0' prio=130 status=active
>>>> | |- 48:0:0:123 sdc 8:32 active ready running
>>>> | `- 50:0:0:123 sdb 8:16 active ready running
>>>> `-+- policy='round-robin 0' prio=1 status=enabled
>>>>     |- 49:0:0:123 sdd 8:48 active ghost running
>>>>     `- 51:0:0:123 sde 8:64 active ghost running
>>>>
>>>> # cat /sys/block/sdb/size
>>>> 209715200
>>>> # cat /sys/block/sdc/size
>>>> 209715200
>>>> # cat /sys/block/sdd/size
>>>> 0
>>>> # cat /sys/block/sde/size
>>>> 0
>>>>
>>>> As soon as we perform a switchover on the 3par systems, multipath
>>>> detects the priority changes and switches paths but the new active paths
>>>> fail.
>>>> We believe this is because 3par doesn't allow us to read the capacity of
>>>> the disk on a standby path - and we have proof of this in the logs:
>>>>
>>>> Apr 13 15:05:12 deb-3par-test kernel: [   40.079736] sd 5:0:0:0: [sdc]
>>>> READ CAPACITY failed
>>>>
>>>> Unfortunately, once we perform the switchover on 3par, the capacity of
>>>> those old ghost paths, now active paths, is not re-read.  The multipath
>>>> device is therefore reduced to a size of 0 and the filesystem becomes
>>>> unavailable.
>>>>
>>>> If we only login on the two active paths without starting multipath,
>>>> perform a switchover, then login on the two new active paths and start
>>>> multipath, we have four block devices with a non-zero size and we can
>>>> perform switchovers at will without any issues.
>>>>
>>>> We've found some older discussions describing these issues on the scsi
>>>> target-devel and dm-devel mailinglists:
>>>> - http://permalink.gmane.org/gmane.linux.scsi.target.devel/6531
>>>> - https://www.redhat.com/archives/dm-devel/2014-July/msg00156.html
>>>>
>>>> As far as we can conclude after reading these messages, it is correct
>>>> behavior for disallowing READ CAPACITY on ghost paths.  However, once
>>>> the path becomes active, we do need a reread of the capacity in order
>>>> for the path to be functional...
>>>>
>>>> We've created a workaround for our issue but we're not sure we're going
>>>> in the right direction.
>>>>
>>>> diff --git a/multipathd/main.c b/multipathd/main.c
>>>> index f876258..ff32681 100644
>>>> --- a/multipathd/main.c
>>>> +++ b/multipathd/main.c
>>>> @@ -1235,6 +1235,11 @@ check_path (struct vectors * vecs, struct path *
>>>> pp)
>>>>
>>>> pp->chkrstate = newstate;
>>>> if (newstate != pp->state) {
>>>> +
>>>> + if (newstate == PATH_UP && pp->size != pp->mpp->size ) {
>>>> + sysfs_attr_set_value(pp->udev, "device/rescan", "1\n",2);
>>>> + }
>>>> +
>>>> int oldstate = pp->state;
>>>> pp->state = newstate;
>>>>
>>>
>>> The above patch will trigger a rescan after every failover and failback.
>>> I'm afraid that will slow down failover and failback, especially if the
>>> number of LUNs is large. I would appreciate it if the capacity would be
>>> reexamined only if it is not yet known.
>>>
>>
>> I realize this is not the best way to handle the situation.
>> This patch was never meant to be implemented as is but more of a
>>
> > clarification of how we look at the issue.
>
>>
>> If we resize a lun on the storage servers, the new size can't be read on
>>
> > standby paths. This means that if a failover occurs for any reason we
> > could end up with a corrupt block device?
>
>>
>> Is there a better way to rescan the capacity? Using sysfs_attr_set_value()
>>
> > like this doesn't look clean to me.
>
>>
>> Would it make sense to make this a configurable setting which is used for
>>
> > systems that don't allow READ CAPACITY on standby paths?
>
> Hello Thomas,
>
> There exists at least one storage array model that accepts the READ
> CAPACITY command on standby paths. The solution I would prefer is that all
> storage arrays would behave this way.
>
> Regarding LUN resizing: the SCSI specs require that a storage array
> reports CAPACITY DATA HAS CHANGED after a LUN has been resized. It should
> be possible to modify the SCSI core such that it rescans a device after
> having received this unit attention condition. The virtio_scsi already
> driver rescans a device after having received that unit attention
> condition. From drivers/scsi/virtio_scsi.c:
>
>         /* Handle "Parameters changed", "Mode parameters changed", and
>            "Capacity data has changed".  */
>         if (asc == 0x2a && (... || ascq == 0x09))
>                 scsi_rescan_device(&sdev->sdev_gendev);
>
> A quote from SBC-4:
>
> Any time the READ CAPACITY (10) parameter data (see 5.15.2) or the READ
> CAPACITY (16) parameter data (see 5.16.2) changes (e.g., when a FORMAT UNIT
> command or a MODE SELECT command causes a change to the logical block
> length or protection information, or when a vendor specific mechanism
> causes a change), then the device server shall establish a unit attention
> condition for the SCSI initiator port (see SAM-5) associated with each I_T
> nexus, except the I_T nexus on which the command causing the change was
> received with the additional sense code set to CAPACITY DATA HAS CHANGED.
>
> Bart.
>
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 8889 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: ALUA - rescan device capacity on zero sized block devices

[Christophe Varoqui]

[-- Attachment #1.1: Type: text/plain, Size: 1511 bytes --]

About five years ago, we faced a somewhat simular issue with Symmetrix
arrays, where the replicated LU of a SRDF pair (R2) was flagged read-only
by the kernel upon discovery. Splitting the pair with a symcli command
 made the LU read-write from the array controller point of view, but the
Linux kernel would not promote it read-write dynamically.

I don't know if the Symmetrix array also use a unit attention to signal the
change to the initiators. If it does, it might be worth trying to address
both the 3par peer persistance and the Symmetrix SRDF situations.

On the other hand, if the SRDF R2 rw promotion issue has been fixed since,
the patch might give guidance about where/how to plug the 3par peer
persistance ghost path rescans.

Best regards,
Christophe Varoqui
OpenSVC

On Tue, Apr 14, 2015 at 11:49 AM, Bart Van Assche <
bart.vanassche@sandisk.com> wrote:

> On 04/14/15 10:14, Hannes Reinecke wrote:
>
>> What we can try is to listen to uevents for the device capacity
>> change or ALUA state changes, and retry the read capacity for those
>> events.
>>
>
> Any mechanism that relies on uevents would be asynchronous. We need a way
> to ensure that the capacity data is up to date before any application
> starts using that data. In this context that means before multipath starts
> queueing I/O to a path. This might be challenging when processing uevents
> asynchronously ...
>
> Bart.
>
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 2312 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: ALUA - rescan device capacity on zero sized block devices

[Hannes Reinecke]

On 04/19/2015 12:56 AM, Christophe Varoqui wrote:
> About five years ago, we faced a somewhat simular issue with
> Symmetrix arrays, where the replicated LU of a SRDF pair (R2) was
> flagged read-only by the kernel upon discovery. Splitting the pair
> with a symcli command  made the LU read-write from the array
> controller point of view, but the Linux kernel would not promote it
> read-write dynamically.
> 
> I don't know if the Symmetrix array also use a unit attention to
> signal the change to the initiators. If it does, it might be worth
> trying to address both the 3par peer persistance and the Symmetrix
> SRDF situations.
> 
> On the other hand, if the SRDF R2 rw promotion issue has been fixed
> since, the patch might give guidance about where/how to plug the
> 3par peer persistance ghost path rescans.
> 
It's not only that; if you are faced with LUNs in standby even the
kernel wouldn't detect them properly.

I'm currently debugging this issue and will have an update soon(-ish).

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] ALUA - rescan device capacity on zero sized block devices

[Ewan Milne]

On Mon, 2015-04-20 at 07:58 +0200, Hannes Reinecke wrote:
> On 04/19/2015 12:56 AM, Christophe Varoqui wrote:
> > About five years ago, we faced a somewhat simular issue with
> > Symmetrix arrays, where the replicated LU of a SRDF pair (R2) was
> > flagged read-only by the kernel upon discovery. Splitting the pair
> > with a symcli command  made the LU read-write from the array
> > controller point of view, but the Linux kernel would not promote it
> > read-write dynamically.
> > 
> > I don't know if the Symmetrix array also use a unit attention to
> > signal the change to the initiators. If it does, it might be worth
> > trying to address both the 3par peer persistance and the Symmetrix
> > SRDF situations.
> > 
> > On the other hand, if the SRDF R2 rw promotion issue has been fixed
> > since, the patch might give guidance about where/how to plug the
> > 3par peer persistance ghost path rescans.
> > 
> It's not only that; if you are faced with LUNs in standby even the
> kernel wouldn't detect them properly.
> 
> I'm currently debugging this issue and will have an update soon(-ish).

I have a patch set to have the kernel automatically rescan the device
when the ALUA state changes to an ACTIVE state, if it couldn't read
capacity when the device was initially probed.  I've had it for a while,
but I haven't had *any* response from the vendor if it actually works
with their product, so I haven't posted it to the list for review yet.

I did point out to them that the T10 spec does not *prohibit* supporting
the READ CAPACITY command in the ALUA standby state, which would avoid
the problem, and is what other vendors seem to do.  However, they then
raised the issue that if the capacity changes in the standby state then
they should be generating the capacity changed UA, etc and you can sort
of see their point of why this gets complicated.

-Ewan

> 
> Cheers,
> 
> Hannes
> -- 
> Dr. Hannes Reinecke		               zSeries & Storage
> hare@suse.de			               +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel


--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [dm-devel] ALUA - rescan device capacity on zero sized block devices

[Hannes Reinecke]

On 06/10/2015 05:02 PM, Ewan Milne wrote:
> On Mon, 2015-04-20 at 07:58 +0200, Hannes Reinecke wrote:
>> On 04/19/2015 12:56 AM, Christophe Varoqui wrote:
>>> About five years ago, we faced a somewhat simular issue with
>>> Symmetrix arrays, where the replicated LU of a SRDF pair (R2) was
>>> flagged read-only by the kernel upon discovery. Splitting the pair
>>> with a symcli command  made the LU read-write from the array
>>> controller point of view, but the Linux kernel would not promote it
>>> read-write dynamically.
>>>
>>> I don't know if the Symmetrix array also use a unit attention to
>>> signal the change to the initiators. If it does, it might be worth
>>> trying to address both the 3par peer persistance and the Symmetrix
>>> SRDF situations.
>>>
>>> On the other hand, if the SRDF R2 rw promotion issue has been fixed
>>> since, the patch might give guidance about where/how to plug the
>>> 3par peer persistance ghost path rescans.
>>>
>> It's not only that; if you are faced with LUNs in standby even the
>> kernel wouldn't detect them properly.
>>
>> I'm currently debugging this issue and will have an update soon(-ish).
> 
> I have a patch set to have the kernel automatically rescan the device
> when the ALUA state changes to an ACTIVE state, if it couldn't read
> capacity when the device was initially probed.  I've had it for a while,
> but I haven't had *any* response from the vendor if it actually works
> with their product, so I haven't posted it to the list for review yet.
> 
Please hold off that patchset.

I've posted the ALUA update patchset a while ago, and are working on
including the suggestions from hch.

Please check if that patchset fixes the issue.

Additionally, I've got some patches for lio-target which will blank
out the READ CAPACITY command when in standby; with that one has an
easy testbed for this kind of issues.

> I did point out to them that the T10 spec does not *prohibit* supporting
> the READ CAPACITY command in the ALUA standby state, which would avoid
> the problem, and is what other vendors seem to do.  However, they then
> raised the issue that if the capacity changes in the standby state then
> they should be generating the capacity changed UA, etc and you can sort
> of see their point of why this gets complicated.
> 
Which is actually not true. The capacity did _not_ change, it's just
the command which isn't supported. If the command was supported and
would have reported a size of '0' in standby _then_ it would have
been a capacity change. But that's not the case here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [dm-devel] ALUA - rescan device capacity on zero sized block devices

[Ewan Milne]

On Thu, 2015-06-11 at 07:52 +0200, Hannes Reinecke wrote: 
> On 06/10/2015 05:02 PM, Ewan Milne wrote:
> > On Mon, 2015-04-20 at 07:58 +0200, Hannes Reinecke wrote:
> >> On 04/19/2015 12:56 AM, Christophe Varoqui wrote:
> >>> About five years ago, we faced a somewhat simular issue with
> >>> Symmetrix arrays, where the replicated LU of a SRDF pair (R2) was
> >>> flagged read-only by the kernel upon discovery. Splitting the pair
> >>> with a symcli command  made the LU read-write from the array
> >>> controller point of view, but the Linux kernel would not promote it
> >>> read-write dynamically.
> >>>
> >>> I don't know if the Symmetrix array also use a unit attention to
> >>> signal the change to the initiators. If it does, it might be worth
> >>> trying to address both the 3par peer persistance and the Symmetrix
> >>> SRDF situations.
> >>>
> >>> On the other hand, if the SRDF R2 rw promotion issue has been fixed
> >>> since, the patch might give guidance about where/how to plug the
> >>> 3par peer persistance ghost path rescans.
> >>>
> >> It's not only that; if you are faced with LUNs in standby even the
> >> kernel wouldn't detect them properly.
> >>
> >> I'm currently debugging this issue and will have an update soon(-ish).
> > 
> > I have a patch set to have the kernel automatically rescan the device
> > when the ALUA state changes to an ACTIVE state, if it couldn't read
> > capacity when the device was initially probed.  I've had it for a while,
> > but I haven't had *any* response from the vendor if it actually works
> > with their product, so I haven't posted it to the list for review yet.
> > 
> Please hold off that patchset.

Sure.  It was really meant to be an RFC anyway.  I didn't want to
take up anyone's time unless it was a viable solution.

We talked a bit about having the kernel automatically update device
attributes at LSF back in March, this was a step towards that.
It implemented a notification mechanism so lower layers (e.g. ALUA
device handler) could propagate status changes up to upper layers
(e.g. sd device class).

> 
> I've posted the ALUA update patchset a while ago, and are working on
> including the suggestions from hch.
> 
> Please check if that patchset fixes the issue.

Will do, it's on my to-do list as soon as we get past a bunch of
other major stuff in the near term.

> 
> Additionally, I've got some patches for lio-target which will blank
> out the READ CAPACITY command when in standby; with that one has an
> easy testbed for this kind of issues.
> 
> > I did point out to them that the T10 spec does not *prohibit* supporting
> > the READ CAPACITY command in the ALUA standby state, which would avoid
> > the problem, and is what other vendors seem to do.  However, they then
> > raised the issue that if the capacity changes in the standby state then
> > they should be generating the capacity changed UA, etc and you can sort
> > of see their point of why this gets complicated.
> > 
> Which is actually not true. The capacity did _not_ change, it's just
> the command which isn't supported. If the command was supported and
> would have reported a size of '0' in standby _then_ it would have
> been a capacity change. But that's not the case here.

Yes, their argument was really more theoretical, in that "if we tell
you about the capacity in standby, we have to tell you when it changes
in standby" and they didn't want to implement that complexity in their
device server.

There's an interesting, somewhat-related issue I've come across with
iSCSI storage, when an event happens while the connection is not
established (i.e. link down, or logged out for some reason).  The T10
spec says that UAs are supposed to be reported on the I-T nexuses,


> 
> Cheers,
> 
> Hannes

Re: [dm-devel] ALUA - rescan device capacity on zero sized block devices

[Ewan Milne]

On Fri, 2015-06-12 at 11:17 -0400, Ewan Milne wrote:
> There's an interesting, somewhat-related issue I've come across with
> iSCSI storage, when an event happens while the connection is not
> established (i.e. link down, or logged out for some reason).  The T10
> spec says that UAs are supposed to be reported on the I-T nexuses,

... but if the I-T nexus doesn't exist when the event occurs, it
doesn't get reported when the nexus is re-established, and it does
not seem like there is any requirement to do so.