[PATCH] dm mpath: disable WRITE SAME if it fails

[PATCH] dm mpath: disable WRITE SAME if it fails

[Mike Snitzer]

Workaround the SCSI layer's problematic WRITE SAME heuristics by
disabling WRITE SAME in the DM multipath device's queue_limits if an
underlying device disabled it.

The WRITE SAME heuristics, with both the original commit 5db44863b6eb
("[SCSI] sd: Implement support for WRITE SAME") and the updated commit
66c28f971 ("[SCSI] sd: Update WRITE SAME heuristics"), default to enabling
WRITE SAME(10) even without successfully determining it is supported.
After the first failed WRITE SAME the SCSI layer will disable WRITE SAME
for the device (by setting sdkp->device->no_write_same which results in
'max_write_same_sectors' in device's queue_limits to be set to 0).

When a device is stacked ontop of such a SCSI device any changes to that
SCSI device's queue_limits do not automatically propagate up the stack.
As such, a DM multipath device will not have its WRITE SAME support
disabled.  This causes the block layer to continue to issue WRITE SAME
requests to the mpath device which causes paths to fail and (if mpath IO
isn't configured to queue when no paths are available) it will result in
actual IO errors to the upper layers.

This fix doesn't help configurations that have additional devices
stacked ontop of the mpath device (e.g. LVM created linear DM devices
ontop).  A proper fix that restacks all the queue_limits from the bottom
of the device stack up will need to be explored if SCSI will continue to
use this model of optimistically allowing op codes and then disabling
them after they fail for the first time.

Before this patch:

EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
device-mapper: multipath: XXX snitm debugging: got -EREMOTEIO (-121)
device-mapper: multipath: XXX snitm debugging: failing WRITE SAME IO with error=-121
end_request: critical target error, dev dm-6, sector 528
dm-6: WRITE SAME failed. Manually zeroing.
device-mapper: multipath: Failing path 8:112.
end_request: I/O error, dev dm-6, sector 4616
dm-6: WRITE SAME failed. Manually zeroing.
end_request: I/O error, dev dm-6, sector 4616
end_request: I/O error, dev dm-6, sector 5640
end_request: I/O error, dev dm-6, sector 6664
end_request: I/O error, dev dm-6, sector 7688
end_request: I/O error, dev dm-6, sector 524288
Buffer I/O error on device dm-6, logical block 65536
lost page write due to I/O error on dm-6
JBD2: Error -5 detected when updating journal superblock for dm-6-8.
end_request: I/O error, dev dm-6, sector 524296
Aborting journal on device dm-6-8.
end_request: I/O error, dev dm-6, sector 524288
Buffer I/O error on device dm-6, logical block 65536
lost page write due to I/O error on dm-6
JBD2: Error -5 detected when updating journal superblock for dm-6-8.

# cat /sys/block/sdh/queue/write_same_max_bytes
0
# cat /sys/block/dm-6/queue/write_same_max_bytes
33553920

After this patch:

EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
device-mapper: multipath: XXX snitm debugging: got -EREMOTEIO (-121)
device-mapper: multipath: XXX snitm debugging: WRITE SAME I/O failed with error=-121
end_request: critical target error, dev dm-6, sector 528
dm-6: WRITE SAME failed. Manually zeroing.

# cat /sys/block/sdh/queue/write_same_max_bytes
0
# cat /sys/block/dm-6/queue/write_same_max_bytes
0

It should be noted that WRITE SAME support wasn't enabled in DM
multipath until v3.10.

Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: Hannes Reinecke <hare@suse.de>
Cc: stable@vger.kernel.org # 3.10+
---
 drivers/md/dm-mpath.c         |   11 ++++++++++-
 drivers/md/dm.c               |   11 +++++++++++
 include/linux/device-mapper.h |    3 ++-
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index 0b3955f..de570a5 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -1299,8 +1299,17 @@ static int do_end_io(struct multipath *m, struct request *clone,
 	if (!error && !clone->errors)
 		return 0;	/* I/O complete */
 
-	if (noretry_error(error))
+	if (noretry_error(error)) {
+		if ((clone->cmd_flags & REQ_WRITE_SAME) &&
+		    !clone->q->limits.max_write_same_sectors) {
+			struct queue_limits *limits;
+
+			/* device doesn't really support WRITE SAME, disable it */
+			limits = dm_get_queue_limits(dm_table_get_md(m->ti->table));
+			limits->max_write_same_sectors = 0;
+		}
 		return error;
+	}
 
 	if (mpio->pgpath)
 		fail_path(mpio->pgpath);
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 033dc26..b3e26c7 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2323,6 +2323,17 @@ struct target_type *dm_get_immutable_target_type(struct mapped_device *md)
 }
 
 /*
+ * The queue_limits are only valid as long as you have a reference
+ * count on 'md'.
+ */
+struct queue_limits *dm_get_queue_limits(struct mapped_device *md)
+{
+	BUG_ON(!atomic_read(&md->holders));
+	return &md->queue->limits;
+}
+EXPORT_SYMBOL_GPL(dm_get_queue_limits);
+
+/*
  * Fully initialize a request-based queue (->elevator, ->request_fn, etc).
  */
 static int dm_init_request_based_queue(struct mapped_device *md)
diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
index 653073d..ed419c6 100644
--- a/include/linux/device-mapper.h
+++ b/include/linux/device-mapper.h
@@ -406,13 +406,14 @@ int dm_noflush_suspending(struct dm_target *ti);
 union map_info *dm_get_mapinfo(struct bio *bio);
 union map_info *dm_get_rq_mapinfo(struct request *rq);
 
+struct queue_limits *dm_get_queue_limits(struct mapped_device *md);
+
 /*
  * Geometry functions.
  */
 int dm_get_geometry(struct mapped_device *md, struct hd_geometry *geo);
 int dm_set_geometry(struct mapped_device *md, struct hd_geometry *geo);
 
-
 /*-----------------------------------------------------------------
  * Functions for manipulating device-mapper tables.
  *---------------------------------------------------------------*/

SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Mike Snitzer]

On Thu, Sep 19 2013 at 12:13pm -0400,
Mike Snitzer <snitzer@redhat.com> wrote:

> Workaround the SCSI layer's problematic WRITE SAME heuristics by
> disabling WRITE SAME in the DM multipath device's queue_limits if an
> underlying device disabled it.

...

> This fix doesn't help configurations that have additional devices
> stacked ontop of the mpath device (e.g. LVM created linear DM devices
> ontop).  A proper fix that restacks all the queue_limits from the bottom
> of the device stack up will need to be explored if SCSI will continue to
> use this model of optimistically allowing op codes and then disabling
> them after they fail for the first time.

I really don't think we can afford to keep SCSI's current heuristics for
enabling WRITE SAME, re-stating them for the benefit of others:
1) check if WRITE SAME is supported by sending REPORT SUPPORTED
   OPERATION CODES to the device
2a) if REPORT SUPPORTED OPERATION CODES shows WRITE SAME is supported,
    enable WRITE SAME
2b) if REPORT SUPPORTED OPERATION CODES shows WRITE SAME is not
    supported, disable WRITE SAME
2c) if REPORT SUPPORTED OPERATION CODES isn't supported _and_ the device
    doesn't have an ATA Information VPD page: enable WRITE SAME
    - if/when WRITE SAME does fail, disable it on the failing device

AFAIK the reason for these heuristics is: devices that do support WRITE
SAME cannot properly report as much because they don't support REPORT
SUPPORTED OPERATION CODES -- this lack of RSOC support is apparently
very common?

I can appreciate the idea behind the current heuristics but I think the
prevelence of the other side of the spectrum (SCSI devices that don't
support RSOC or WRITE SAME) was underestimated.  As such we're seeing a
fair amount of WRITE SAME error noise on these systems -- that noise is
itself considered a bug to many.

Also, please see my comment in a related Fedora 19 BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=995271#c23

As I say in that comment:
"A proper fix could be to make SCSI's default be to disable WRITE SAME
for devices that don't properly report they support it.  And possibly
have a whitelist to opt-in to enabling WRITE SAME for select targets."

I'm open to any ideas you might have.

Thanks,
Mike

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Martin K. Petersen]

>>>>> "Mike" == Mike Snitzer <snitzer@redhat.com> writes:

Mike,

Mike> AFAIK the reason for these heuristics is: devices that do support
Mike> WRITE SAME cannot properly report as much because they don't
Mike> support REPORT SUPPORTED OPERATION CODES -- this lack of RSOC
Mike> support is apparently very common?

Only a handful of the very latest and greatest devices support RSOC. The
number of devices that support WRITE SAME is orders of magnitude larger.

Last I checked I had exactly 1 out of about 100 devices in my lab that
supported RSOC.


Mike> I can appreciate the idea behind the current heuristics but I
Mike> think the prevelence of the other side of the spectrum (SCSI
Mike> devices that don't support RSOC or WRITE SAME) was underestimated.

If you by "devices" mean vintage PCI RAID controllers that don't pass
things through correctly, then yes. I don't think I have a single SCSI
drive that doesn't support WRITE SAME. And all the controllers I tested
with here worked fine.


Mike> As I say in that comment: "A proper fix could be to make SCSI's
Mike> default be to disable WRITE SAME for devices that don't properly
Mike> report they support it.  And possibly have a whitelist to opt-in
Mike> to enabling WRITE SAME for select targets."

The problem with the opt-in approach is that there are orders of
magnitude more devices that would need to get it enabled than there are
broken ones that need it disabled.

There are only a couple of handfuls of RAID controller drivers. We've
been working through the issues on these on a case by case basis.

Yes, I totally agree it sucks. And I hate that things broke for people
with Areca and 3ware. But we got those fixed. And it's way easier to
blacklist "all devices hanging off RAID driver xyz" than it is to
whitelist every SCSI drive known to man. It sucks in the short term but
is better long term.

The major headache here of course is that WRITE SAME is inherently
destructive. We can't just fire off one during discovery and see if it
works. For WRITE you can issue a command with a transfer length of 0 to
see if things work. But unfortunately for WRITE SAME a transfer length
of zero means "wipe the entire device". Yikes!

I guess we could read one sector and try to write it back using WRITE
SAME and a block count of one. But it's really icky. And I don't like
the notion of actually writing things during discovery.

As far as being able to trigger a restacking of the queue limits I think
it's inevitable. We see more and more devices that change properties
after a firmware upgrade. I think we'll just have to bite the bullet and
work on that...

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Douglas Gilbert]

On 13-09-20 06:03 PM, Martin K. Petersen wrote:
>>>>>> "Mike" == Mike Snitzer <snitzer@redhat.com> writes:
>
> Mike,
>
> Mike> AFAIK the reason for these heuristics is: devices that do support
> Mike> WRITE SAME cannot properly report as much because they don't
> Mike> support REPORT SUPPORTED OPERATION CODES -- this lack of RSOC
> Mike> support is apparently very common?
>
> Only a handful of the very latest and greatest devices support RSOC. The
> number of devices that support WRITE SAME is orders of magnitude larger.
>
> Last I checked I had exactly 1 out of about 100 devices in my lab that
> supported RSOC.
>
>
> Mike> I can appreciate the idea behind the current heuristics but I
> Mike> think the prevelence of the other side of the spectrum (SCSI
> Mike> devices that don't support RSOC or WRITE SAME) was underestimated.
>
> If you by "devices" mean vintage PCI RAID controllers that don't pass
> things through correctly, then yes. I don't think I have a single SCSI
> drive that doesn't support WRITE SAME. And all the controllers I tested
> with here worked fine.
>
>
> Mike> As I say in that comment: "A proper fix could be to make SCSI's
> Mike> default be to disable WRITE SAME for devices that don't properly
> Mike> report they support it.  And possibly have a whitelist to opt-in
> Mike> to enabling WRITE SAME for select targets."
>
> The problem with the opt-in approach is that there are orders of
> magnitude more devices that would need to get it enabled than there are
> broken ones that need it disabled.
>
> There are only a couple of handfuls of RAID controller drivers. We've
> been working through the issues on these on a case by case basis.
>
> Yes, I totally agree it sucks. And I hate that things broke for people
> with Areca and 3ware. But we got those fixed. And it's way easier to
> blacklist "all devices hanging off RAID driver xyz" than it is to
> whitelist every SCSI drive known to man. It sucks in the short term but
> is better long term.
>
> The major headache here of course is that WRITE SAME is inherently
> destructive. We can't just fire off one during discovery and see if it
> works. For WRITE you can issue a command with a transfer length of 0 to
> see if things work. But unfortunately for WRITE SAME a transfer length
> of zero means "wipe the entire device". Yikes!
>
> I guess we could read one sector and try to write it back using WRITE
> SAME and a block count of one. But it's really icky. And I don't like
> the notion of actually writing things during discovery.
>
> As far as being able to trigger a restacking of the queue limits I think
> it's inevitable. We see more and more devices that change properties
> after a firmware upgrade. I think we'll just have to bite the bullet and
> work on that...

Would a closer examination of the available VPD pages
help? For example support for the Logical Block
Provisioning and Block Limits VPD pages. Given either
of those two pages, even if the WRITE SAME specific
fields in those pages are not set, it is unlikely that
sending a WRITE SAME (when actually required rather
than at discovery) would wedge the disk/controller.

If WSNZ is set in the Block Limits VPD pages then it
should be "safe" ** to send a zero length WRITE SAME
command to a LU. And that is another good reason to
check the response of a VPD page request carefully
(e.g. the echo-ed page_code in byte 1 and a sensible
page_length in bytes 2 and 3) since crap devices often
return a standard INQUIRY response to a VPD page request.

Doug Gilbert


** there is departure from the normal "do nothing when
    transfer_length or number_of_LB fields are zero":
    In the case of the WS command with number_of_LBs=0
    and WSNZ=1 in the BLOCK LIMITS VPD page, the response
    should be ILLEGAL_REQUEST with INVALID FIELD IN CDB.
    I guess if it yielded status=GOOD in that case and
    you heard the disk clicking, you might get quite
    worried :-)

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Ewan Milne]

On Fri, 2013-09-20 at 18:03 -0400, Martin K. Petersen wrote:
...
> Only a handful of the very latest and greatest devices support RSOC. The
> number of devices that support WRITE SAME is orders of magnitude larger.
> 
> Last I checked I had exactly 1 out of about 100 devices in my lab that
> supported RSOC.
...
> The major headache here of course is that WRITE SAME is inherently
> destructive. We can't just fire off one during discovery and see if it
> works. For WRITE you can issue a command with a transfer length of 0 to
> see if things work. But unfortunately for WRITE SAME a transfer length
> of zero means "wipe the entire device". Yikes!
> 
> I guess we could read one sector and try to write it back using WRITE
> SAME and a block count of one. But it's really icky. And I don't like
> the notion of actually writing things during discovery.
...

Just out of curiosity, what do the devices that support WRITE SAME
report for the MAXIMUM WRITE SAME LENGTH field in VPD page B0?  The
spec says that this can be zero if there is no restriction, but is
there any chance that most/all of them report some nonzero value?

Expanding on Doug's thinking, perhaps there is some combination of
VPD page availability / field values that could be used to explicitly
enable WRITE SAME?  Or, have you been through that already?

-Ewan

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Hannes Reinecke]

On 09/23/2013 08:18 PM, Ewan Milne wrote:
> On Fri, 2013-09-20 at 18:03 -0400, Martin K. Petersen wrote:
> ...
>> Only a handful of the very latest and greatest devices support RSOC. The
>> number of devices that support WRITE SAME is orders of magnitude larger.
>>
>> Last I checked I had exactly 1 out of about 100 devices in my lab that
>> supported RSOC.
> ...
>> The major headache here of course is that WRITE SAME is inherently
>> destructive. We can't just fire off one during discovery and see if it
>> works. For WRITE you can issue a command with a transfer length of 0 to
>> see if things work. But unfortunately for WRITE SAME a transfer length
>> of zero means "wipe the entire device". Yikes!
>>
>> I guess we could read one sector and try to write it back using WRITE
>> SAME and a block count of one. But it's really icky. And I don't like
>> the notion of actually writing things during discovery.
> ...
> 
> Just out of curiosity, what do the devices that support WRITE SAME
> report for the MAXIMUM WRITE SAME LENGTH field in VPD page B0?  The
> spec says that this can be zero if there is no restriction, but is
> there any chance that most/all of them report some nonzero value?
> 
> Expanding on Doug's thinking, perhaps there is some combination of
> VPD page availability / field values that could be used to explicitly
> enable WRITE SAME?  Or, have you been through that already?
> 
Hehe. Won't do any good.

My drives support 'report opcodes', and report that write same is
supported:
...
 93                 16    Write same(16)
...

but no support for page 'b0'. And yes, these are real SAS drives.

Cheers,

Hannes

-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Paolo Bonzini]

Il 21/09/2013 00:03, Martin K. Petersen ha scritto:
> 
> The major headache here of course is that WRITE SAME is inherently
> destructive. We can't just fire off one during discovery and see if it
> works. For WRITE you can issue a command with a transfer length of 0 to
> see if things work. But unfortunately for WRITE SAME a transfer length
> of zero means "wipe the entire device". Yikes!

What about WRITE SAME with an out-of-range lba?  Or lba equal to the
size of the disk and #blocks equal to zero.

Paolo

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Mike Snitzer]

On Tue, Sep 24 2013 at  1:39am -0400,
Hannes Reinecke <hare@suse.de> wrote:

> On 09/23/2013 08:18 PM, Ewan Milne wrote:
> > On Fri, 2013-09-20 at 18:03 -0400, Martin K. Petersen wrote:
> > ...
> >> Only a handful of the very latest and greatest devices support RSOC. The
> >> number of devices that support WRITE SAME is orders of magnitude larger.
> >>
> >> Last I checked I had exactly 1 out of about 100 devices in my lab that
> >> supported RSOC.
> > ...
> >> The major headache here of course is that WRITE SAME is inherently
> >> destructive. We can't just fire off one during discovery and see if it
> >> works. For WRITE you can issue a command with a transfer length of 0 to
> >> see if things work. But unfortunately for WRITE SAME a transfer length
> >> of zero means "wipe the entire device". Yikes!
> >>
> >> I guess we could read one sector and try to write it back using WRITE
> >> SAME and a block count of one. But it's really icky. And I don't like
> >> the notion of actually writing things during discovery.
> > ...
> > 
> > Just out of curiosity, what do the devices that support WRITE SAME
> > report for the MAXIMUM WRITE SAME LENGTH field in VPD page B0?  The
> > spec says that this can be zero if there is no restriction, but is
> > there any chance that most/all of them report some nonzero value?
> > 
> > Expanding on Doug's thinking, perhaps there is some combination of
> > VPD page availability / field values that could be used to explicitly
> > enable WRITE SAME?  Or, have you been through that already?
> > 
> Hehe. Won't do any good.
> 
> My drives support 'report opcodes', and report that write same is
> supported:
> ...
>  93                 16    Write same(16)
> ...
> 
> but no support for page 'b0'. And yes, these are real SAS drives.

Your drive would be fine then since it does support RSOC and specifies
WRITE SAME is supported.  Pretty sure Ewan was only suggesting to resort
to looking at the VPD page if RSOC isn't supported.

So are there drives like this?:
1) don't support RSOC
2) do support WRITE SAME
3) do populate VPD page with either WRITE SAME w/ discard bit set or
UNMAP?

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[James Bottomley]

On Tue, 2013-09-24 at 11:37 +0200, Paolo Bonzini wrote:
> Il 21/09/2013 00:03, Martin K. Petersen ha scritto:
> > 
> > The major headache here of course is that WRITE SAME is inherently
> > destructive. We can't just fire off one during discovery and see if it
> > works. For WRITE you can issue a command with a transfer length of 0 to
> > see if things work. But unfortunately for WRITE SAME a transfer length
> > of zero means "wipe the entire device". Yikes!
> 
> What about WRITE SAME with an out-of-range lba?  Or lba equal to the
> size of the disk and #blocks equal to zero.

That's inviting undefined behaviour in a data destructive command ...
not something we really want to do.

James

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Martin K. Petersen]

>>>>> "Mike" == Mike Snitzer <snitzer@redhat.com> writes:

Mike> So are there drives like this?:
Mike> 1) don't support RSOC
Mike> 2) do support WRITE SAME
Mike> 3) do populate VPD page with either WRITE SAME w/ discard bit set
Mike>    or UNMAP?

Yes.

But again, the fundamental issue here is not the drives. It's the
controller firmware. I am not aware of a single SPI/SAS/FC drive that
does not support at least WRITE SAME(10).

For DIX and T10 PI I have a capabilities mask that each HBA driver fills
out that tells the sd driver what the controller can do. And this is
combined with whatever the drive reports to figure out whether integrity
protection can be enabled.

I have been contemplating doing something similar for "fancy" SCSI
commands. We could have a flag in the scsi host template that controls
whether the device supports WRITE SAME, EXTENDED COPY, etc. The
advantage being that we do the matching at discovery time instead of
once a WRITE SAME is issued.

This would also permit HBA drivers to toggle the feature on a per
instance basis. I.e. if "RAID controller firmware rev is lower than XYZ,
do not support WRITE SAME".

I'll do a PoC later today...

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Mike Snitzer]

On Tue, Sep 24 2013 at  9:49am -0400,
Martin K. Petersen <martin.petersen@oracle.com> wrote:

> >>>>> "Mike" == Mike Snitzer <snitzer@redhat.com> writes:
> 
> Mike> So are there drives like this?:
> Mike> 1) don't support RSOC
> Mike> 2) do support WRITE SAME
> Mike> 3) do populate VPD page with either WRITE SAME w/ discard bit set
> Mike>    or UNMAP?
> 
> Yes.
> 
> But again, the fundamental issue here is not the drives. It's the
> controller firmware. I am not aware of a single SPI/SAS/FC drive that
> does not support at least WRITE SAME(10).
> 
> For DIX and T10 PI I have a capabilities mask that each HBA driver fills
> out that tells the sd driver what the controller can do. And this is
> combined with whatever the drive reports to figure out whether integrity
> protection can be enabled.
> 
> I have been contemplating doing something similar for "fancy" SCSI
> commands. We could have a flag in the scsi host template that controls
> whether the device supports WRITE SAME, EXTENDED COPY, etc. The
> advantage being that we do the matching at discovery time instead of
> once a WRITE SAME is issued.
> 
> This would also permit HBA drivers to toggle the feature on a per
> instance basis. I.e. if "RAID controller firmware rev is lower than XYZ,
> do not support WRITE SAME".
> 
> I'll do a PoC later today...

Sounds promising, thanks!

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Mikulas Patocka]

On Fri, 20 Sep 2013, Mike Snitzer wrote:

> On Thu, Sep 19 2013 at 12:13pm -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
> 
> > Workaround the SCSI layer's problematic WRITE SAME heuristics by
> > disabling WRITE SAME in the DM multipath device's queue_limits if an
> > underlying device disabled it.
> 
> ...
> 
> > This fix doesn't help configurations that have additional devices
> > stacked ontop of the mpath device (e.g. LVM created linear DM devices
> > ontop).  A proper fix that restacks all the queue_limits from the bottom
> > of the device stack up will need to be explored if SCSI will continue to
> > use this model of optimistically allowing op codes and then disabling
> > them after they fail for the first time.

Propagating the limits up the tree won't work anyway because of a race 
condition. Think of this case:

1. We test queue limits and find out that the device supports WRITE_SAME
2. We build a WRITE_SAME bio to be submitted.
3. The user reconfigures a device so that it no longer supports WRITE_SAME 
(for example, he adds a snapshot or converts linear volume to a mirror).
4. We submit the bio that we built in step 2.
5. The bio fails.


I think the correct solution is to specify that WRITE SAME request may 
fail anytime and the caller is responsible for retrying with normal WRITE 
request if WRITE SAME failed.

If the caller needs to send WRITE SAME, it should try to send it and if it 
fails, set a flag that the device doesn't support WRITE SAME and not send 
WRITE SAME in the future. (we could also reset that flag after some long 
time, in case the device was reconfigured to support WRITE SAME).


> I really don't think we can afford to keep SCSI's current heuristics for
> enabling WRITE SAME, re-stating them for the benefit of others:
> 1) check if WRITE SAME is supported by sending REPORT SUPPORTED
>    OPERATION CODES to the device
> 2a) if REPORT SUPPORTED OPERATION CODES shows WRITE SAME is supported,
>     enable WRITE SAME
> 2b) if REPORT SUPPORTED OPERATION CODES shows WRITE SAME is not
>     supported, disable WRITE SAME
> 2c) if REPORT SUPPORTED OPERATION CODES isn't supported _and_ the device
>     doesn't have an ATA Information VPD page: enable WRITE SAME
>     - if/when WRITE SAME does fail, disable it on the failing device
> 
> AFAIK the reason for these heuristics is: devices that do support WRITE
> SAME cannot properly report as much because they don't support REPORT
> SUPPORTED OPERATION CODES -- this lack of RSOC support is apparently
> very common?
> 
> I can appreciate the idea behind the current heuristics but I think the
> prevelence of the other side of the spectrum (SCSI devices that don't
> support RSOC or WRITE SAME) was underestimated.  As such we're seeing a
> fair amount of WRITE SAME error noise on these systems -- that noise is
> itself considered a bug to many.
> 
> Also, please see my comment in a related Fedora 19 BZ:
> https://bugzilla.redhat.com/show_bug.cgi?id=995271#c23
> 
> As I say in that comment:
> "A proper fix could be to make SCSI's default be to disable WRITE SAME
> for devices that don't properly report they support it.  And possibly
> have a whitelist to opt-in to enabling WRITE SAME for select targets."
> 
> I'm open to any ideas you might have.
> 
> Thanks,
> Mike

What does happend if we send WRITE SAME to a disk that doesn't support it? 
If we only get "Invalid field in CDB" or similar sense code, we could just 
propagate it up and let the caller retry.

If we could get something worse (firmware crash), we must blacklist it.



Another idea: we could kill the REQ_WRITE_SAME flag entirely - if the some 
subsystem needs to clear some area of a disk, it builds a bio with all 
vector entries pointing to the zero page. Drivers that don't support WRITE 
SAME (such as IDE) will implement it doing DMA from the zero page. Drivers 
that support WRITE SAME can detect this special case and convert a normal 
WRITE request to WRITE SAME. The request is converted to WRITE SAME at the 
lowest level (in the SCSI driver), it passes as normal WRITE request 
through dm and md midlayers - thus, we magically get WRITE SAME support in 
all md and dm drivers without writing additional code for it.

Mikulas

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Jeremy Linton]

On 9/24/2013 12:39 AM, Hannes Reinecke wrote:

> My drives support 'report opcodes', and report that write same is 
> supported: ... 93                 16    Write same(16) ...
> 
> but no support for page 'b0'. And yes, these are real SAS drives.

	So the question is, how many devices get the protect bit in the std inquiry
incorrect?

	If that is mostly correct, how about std inquiry (SPC>2), protect bit, and then
VPD page 0x86 (or alternatively then do the READ CAPACITY, P_TYPE instead of
page 0x86)?

	After all, the set of valid read/write opcodes is limited by the protection
mode format, yes?






	

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Douglas Gilbert]

On 13-09-24 03:12 PM, Jeremy Linton wrote:
> On 9/24/2013 12:39 AM, Hannes Reinecke wrote:
>
>> My drives support 'report opcodes', and report that write same is
>> supported: ... 93                 16    Write same(16) ...
>>
>> but no support for page 'b0'. And yes, these are real SAS drives.
>
> 	So the question is, how many devices get the protect bit in the std inquiry
> incorrect?

To that specific question: all of them get it right because
if they don't the disk (LU) does not support PI (or at least
that is the conclusion the application client should reach).
What the sales blurb for the device may say is another thing.

More generally why are we using WRITE SAME:
   a) associated with LB provisioning?
   b) associated with PI (DIF)?
   c) as an optimized multi-block write of the same contents?

I'm not aware there is any special relationship between WS
and PI. That leaves a) and c) . I'm guessing it is more
important for a) .

> 	If that is mostly correct, how about std inquiry (SPC>2), protect bit, and then
> VPD page 0x86 (or alternatively then do the READ CAPACITY, P_TYPE instead of
> page 0x86)?
>
> 	After all, the set of valid read/write opcodes is limited by the protection
> mode format, yes?

Yes: READ(32), WRITE(32), WRITE SAME(32), VERIFY(32)
and WRITE AND VERIFY(32) are only valid for protection
type 2 (sbc3r35h 4.22.2.2). I'm sure T10 have a good
reason for that but they are keeping that reason to
themselves.

Doug Gilbert

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Martin K. Petersen]

>>>>> "Mikulas" == Mikulas Patocka <mpatocka@redhat.com> writes:

Mikulas> Propagating the limits up the tree won't work anyway because of
Mikulas> a race condition. Think of this case:

Mikulas> 1. We test queue limits and find out that the device supports
Mikulas>    WRITE_SAME
Mikulas> 2. We build a WRITE_SAME bio to be submitted.
Mikulas> 3. The user reconfigures a device so that it no longer supports
Mikulas>    WRITE_SAME
Mikulas> (for example, he adds a snapshot or converts linear volume to a
Mikulas> mirror).
Mikulas> 4. We submit the bio that we built in step 2.
Mikulas> 5. The bio fails.

The existing "fall back to writes if write same fails" mechanism isn't
going away. We absolutely need that.

The principal headache is caused by host adapters that perform SCSI-ATA
or SCSI-<internal RAID implementation> translation and which do not
correctly fail a WRITE SAME command that we send down to the disk.

Based on heuristics from querying the target we expect WRITE SAME to
succeed so we enable it. But for whatever reason the controller is not
able to handle the command. Most controllers/drivers will fail the
command correctly and we'll disable WRITE SAME and fall back to manually
zeroing. Some controllers, however, hang or fail the I/O in a way in
which we do not pick up it's a WRITE SAME problem.

The other headache is what happens if a stacking driver gets -EIO or
-EREMOTEIO on a WRITE SAME request. That's what Mike was trying to fix
with his patch. Maybe it would make sense for us to use -EINVAL or
something similar to single out the ILLEGAL REQUEST case. While
-EREMOTEIO specifically calls out for "no retry" it might be helpful to
indicate that we're dealing with an unsupported command. That way DM or
MD can reconfigure themselves accordingly.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Mike Snitzer]

On Tue, Sep 24 2013 at  4:44pm -0400,
Martin K. Petersen <martin.petersen@oracle.com> wrote:

> The other headache is what happens if a stacking driver gets -EIO or
> -EREMOTEIO on a WRITE SAME request. That's what Mike was trying to fix
> with his patch. Maybe it would make sense for us to use -EINVAL or
> something similar to single out the ILLEGAL REQUEST case. While
> -EREMOTEIO specifically calls out for "no retry" it might be helpful to
> indicate that we're dealing with an unsupported command. That way DM or
> MD can reconfigure themselves accordingly.

What I've committed as a v3.12 fix works, see:
http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-linus&id=f84cb8a46a771f36a04a02c61ea635c968ed5f6a

"Workaround the SCSI layer's problematic WRITE SAME heuristics by
disabling WRITE SAME in the DM multipath device's queue_limits if an
underlying device disabled it."

So I'm trapping requests with REQ_WRITE_SAME whose underlying queue's
queue_limits have disabled WRITE SAME.  And as the before and after
detailed in the patch header shows, it works.

I'm open to tweaking this later but will be leaving well enough alone
for v3.12 by keeping this fix.

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Bernd Schubert]

On 09/24/2013 03:49 PM, Martin K. Petersen wrote:
>>>>>> "Mike" == Mike Snitzer <snitzer@redhat.com> writes:
> 
> Mike> So are there drives like this?:
> Mike> 1) don't support RSOC
> Mike> 2) do support WRITE SAME
> Mike> 3) do populate VPD page with either WRITE SAME w/ discard bit set
> Mike>    or UNMAP?
> 
> Yes.
> 
> But again, the fundamental issue here is not the drives. It's the
> controller firmware. I am not aware of a single SPI/SAS/FC drive that
> does not support at least WRITE SAME(10).
> 
> For DIX and T10 PI I have a capabilities mask that each HBA driver fills
> out that tells the sd driver what the controller can do. And this is
> combined with whatever the drive reports to figure out whether integrity
> protection can be enabled.
> 
> I have been contemplating doing something similar for "fancy" SCSI
> commands. We could have a flag in the scsi host template that controls
> whether the device supports WRITE SAME, EXTENDED COPY, etc. The
> advantage being that we do the matching at discovery time instead of
> once a WRITE SAME is issued.
> 
> This would also permit HBA drivers to toggle the feature on a per
> instance basis. I.e. if "RAID controller firmware rev is lower than XYZ,
> do not support WRITE SAME".
> 
> I'll do a PoC later today...
> 

Martin,

I'm afraid we have another problem. I'm currently working on to get
discard working for our LSI2008 HBAs with attached sata-SSDs and the
heuristics in sd_read_write_same with based on VPD page 0x89 is not
correct for this HBA - its SATL supports write-same (although it does
"Logical block address out of range" at the end of the device, I'm going
to look into this tomorrow).

So allow LSI SATL or remove this check at all?


Thanks,
Bernd

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Douglas Gilbert]

On 13-09-25 04:52 PM, Bernd Schubert wrote:
> On 09/24/2013 03:49 PM, Martin K. Petersen wrote:
>>>>>>> "Mike" == Mike Snitzer <snitzer@redhat.com> writes:
>>
>> Mike> So are there drives like this?:
>> Mike> 1) don't support RSOC
>> Mike> 2) do support WRITE SAME
>> Mike> 3) do populate VPD page with either WRITE SAME w/ discard bit set
>> Mike>    or UNMAP?
>>
>> Yes.
>>
>> But again, the fundamental issue here is not the drives. It's the
>> controller firmware. I am not aware of a single SPI/SAS/FC drive that
>> does not support at least WRITE SAME(10).
>>
>> For DIX and T10 PI I have a capabilities mask that each HBA driver fills
>> out that tells the sd driver what the controller can do. And this is
>> combined with whatever the drive reports to figure out whether integrity
>> protection can be enabled.
>>
>> I have been contemplating doing something similar for "fancy" SCSI
>> commands. We could have a flag in the scsi host template that controls
>> whether the device supports WRITE SAME, EXTENDED COPY, etc. The
>> advantage being that we do the matching at discovery time instead of
>> once a WRITE SAME is issued.
>>
>> This would also permit HBA drivers to toggle the feature on a per
>> instance basis. I.e. if "RAID controller firmware rev is lower than XYZ,
>> do not support WRITE SAME".
>>
>> I'll do a PoC later today...
>>
>
> Martin,
>
> I'm afraid we have another problem. I'm currently working on to get
> discard working for our LSI2008 HBAs with attached sata-SSDs and the
> heuristics in sd_read_write_same with based on VPD page 0x89 is not
> correct for this HBA - its SATL supports write-same (although it does
> "Logical block address out of range" at the end of the device, I'm going
> to look into this tomorrow).
>
> So allow LSI SATL or remove this check at all?

LSI implement their SATL in firmware inside their HBAs. Given
their latest firmware release name (*Package_P17_IR_IT_Firmware*
dated 9 August 2013) for their SAS-2 family *** you have 17
versions of that firmware potentially out there in the field.
Updating that firmware is a fiddly process; I use a USB stick
with DOS on it! LSI do fix things in their SATL when flaws are
pointed out.

Generally speaking LSI's SATL is pretty good, at least compared
to another SATL I can think of. They are both moving targets,
and move independently.

Doug Gilbert


*** LSI's SAS-3 HBAs are still at "P1" which I assume is the
     first publically released version.

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Martin K. Petersen]

>>>>> "Bernd" == Bernd Schubert <bernd.schubert@fastmail.fm> writes:

Hey Bernd,

Bernd> I'm afraid we have another problem. I'm currently working on to
Bernd> get discard working for our LSI2008 HBAs with attached sata-SSDs
Bernd> and the heuristics in sd_read_write_same with based on VPD page
Bernd> 0x89 is not correct for this HBA - its SATL supports write-same

This has nothing to do with the WRITE SAME heuristics.

It's true that depending on wind and whether we might issue WRITE
SAME(10) or (16) with the UNMAP bit set to perform discard operations on
the low level device. But we use a set of different (and somewhat more
reliable) heuristics to decide which command to send down for that
purpose.

For discards to a SATA device to work you need a recent phase LSI
firmware. And you need the target mode firmware (IT). There is no
UNMAP->DSM TRIM translation in the RAID (IR) firmware.

If your SATA SSDs reports DSM TRIM support, the LSI firmware will set
LBPME=1 in READ CAPACITY(16) and the LOGICAL BLOCK PROVISIONING VPD page
will indicate a preference for the UNMAP command (LBPU=1).

Also, LSI firmware is well-behaved in general and will report ILLEGAL
REQUEST when you send down a command that can't be handled.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Douglas Gilbert]

On 13-09-25 08:44 PM, Martin K. Petersen wrote:
>>>>>> "Bernd" == Bernd Schubert <bernd.schubert@fastmail.fm> writes:
>
> Hey Bernd,
>
> Bernd> I'm afraid we have another problem. I'm currently working on to
> Bernd> get discard working for our LSI2008 HBAs with attached sata-SSDs
> Bernd> and the heuristics in sd_read_write_same with based on VPD page
> Bernd> 0x89 is not correct for this HBA - its SATL supports write-same
>
> This has nothing to do with the WRITE SAME heuristics.
>
> It's true that depending on wind and whether we might issue WRITE
> SAME(10) or (16) with the UNMAP bit set to perform discard operations on
> the low level device. But we use a set of different (and somewhat more
> reliable) heuristics to decide which command to send down for that
> purpose.
>
> For discards to a SATA device to work you need a recent phase LSI
> firmware. And you need the target mode firmware (IT). There is no
> UNMAP->DSM TRIM translation in the RAID (IR) firmware.
>
> If your SATA SSDs reports DSM TRIM support, the LSI firmware will set
> LBPME=1 in READ CAPACITY(16) and the LOGICAL BLOCK PROVISIONING VPD page
> will indicate a preference for the UNMAP command (LBPU=1).
>
> Also, LSI firmware is well-behaved in general and will report ILLEGAL
> REQUEST when you send down a command that can't be handled.

An example with a LSI 9212-4i4e running the latest firmware
(P17) connected to a SATA SSD (via an expander):

# sg_vpd /dev/sg1 -p sinq
standard INQUIRY:
   PQual=0  Device_type=0  RMB=0  version=0x06  [SPC-4]
   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=1  Resp_data_format=2
   SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  [BQue=0]
   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
   [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
   Vendor_identification: ATA
   Product_identification: INTEL SSDSA2M080
   Product_revision_level: 02M3

# sg_vpd /dev/sg1 -p bl
Block limits VPD page (SBC):
   Write same no zero (WSNZ): 0
   Maximum compare and write length: 0 blocks
   Optimal transfer length granularity: 0 blocks
   Maximum transfer length: 0 blocks
   Optimal transfer length: 0 blocks
   Maximum prefetch length: 0 blocks
   Maximum unmap LBA count: 4194303
   Maximum unmap block descriptor count: 32
   Optimal unmap granularity: 1
   Unmap granularity alignment valid: 0
   Unmap granularity alignment: 0
   Maximum write same length: 0x0 blocks

# sg_vpd /dev/sg1 -p lbpv
Logical block provisioning VPD page (SBC):
   Unmap command supported (LBPU): 1
   Write same (16) with unmap bit supported (LBWS): 1
   Write same (10) with unmap bit supported (LBWS10): 0
   Logical block provisioning read zeros (LBPRZ): 0
   Anchored LBAs supported (ANC_SUP): 1
   Threshold exponent: 0
   Descriptor present (DP): 0
   Provisioning type: 0

# sg_opcodes -n /dev/sg1
Report supported operation codes: operation not supported


Room for improvement there. It also supports a useful set
of mode pages (including some chageable fields) and two
log pages.

Doug Gilbert

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Bernd Schubert]

On 09/26/2013 07:39 AM, Douglas Gilbert wrote:
> On 13-09-25 08:44 PM, Martin K. Petersen wrote:
>>>>>>> "Bernd" == Bernd Schubert <bernd.schubert@fastmail.fm> writes:
>>
>> Hey Bernd,
>>
>> Bernd> I'm afraid we have another problem. I'm currently working on to
>> Bernd> get discard working for our LSI2008 HBAs with attached sata-SSDs
>> Bernd> and the heuristics in sd_read_write_same with based on VPD page
>> Bernd> 0x89 is not correct for this HBA - its SATL supports write-same
>>
>> This has nothing to do with the WRITE SAME heuristics.
>>
>> It's true that depending on wind and whether we might issue WRITE
>> SAME(10) or (16) with the UNMAP bit set to perform discard operations on
>> the low level device. But we use a set of different (and somewhat more
>> reliable) heuristics to decide which command to send down for that
>> purpose.
>>
>> For discards to a SATA device to work you need a recent phase LSI
>> firmware. And you need the target mode firmware (IT). There is no
>> UNMAP->DSM TRIM translation in the RAID (IR) firmware.
>>
>> If your SATA SSDs reports DSM TRIM support, the LSI firmware will set
>> LBPME=1 in READ CAPACITY(16) and the LOGICAL BLOCK PROVISIONING VPD page
>> will indicate a preference for the UNMAP command (LBPU=1).
>>
>> Also, LSI firmware is well-behaved in general and will report ILLEGAL
>> REQUEST when you send down a command that can't be handled.
>
> An example with a LSI 9212-4i4e running the latest firmware
> (P17) connected to a SATA SSD (via an expander):
>
> # sg_vpd /dev/sg1 -p sinq
> standard INQUIRY:
>    PQual=0  Device_type=0  RMB=0  version=0x06  [SPC-4]
>    [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=1  Resp_data_format=2
>    SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  [BQue=0]
>    EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>    [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
>    Vendor_identification: ATA
>    Product_identification: INTEL SSDSA2M080
>    Product_revision_level: 02M3
>
> # sg_vpd /dev/sg1 -p bl
> Block limits VPD page (SBC):
>    Write same no zero (WSNZ): 0
>    Maximum compare and write length: 0 blocks
>    Optimal transfer length granularity: 0 blocks
>    Maximum transfer length: 0 blocks
>    Optimal transfer length: 0 blocks
>    Maximum prefetch length: 0 blocks
>    Maximum unmap LBA count: 4194303
>    Maximum unmap block descriptor count: 32
>    Optimal unmap granularity: 1
>    Unmap granularity alignment valid: 0
>    Unmap granularity alignment: 0
>    Maximum write same length: 0x0 blocks
>
> # sg_vpd /dev/sg1 -p lbpv
> Logical block provisioning VPD page (SBC):
>    Unmap command supported (LBPU): 1
>    Write same (16) with unmap bit supported (LBWS): 1
>    Write same (10) with unmap bit supported (LBWS10): 0
>    Logical block provisioning read zeros (LBPRZ): 0
>    Anchored LBAs supported (ANC_SUP): 1
>    Threshold exponent: 0
>    Descriptor present (DP): 0
>    Provisioning type: 0
>
> # sg_opcodes -n /dev/sg1
> Report supported operation codes: operation not supported
>
>
> Room for improvement there. It also supports a useful set
> of mode pages (including some chageable fields) and two
> log pages.
>

Both types of systems we have in-house neither block limits vpd nor 
READ_CAP16 return anything that would indicate discard is supported. But 
UNMAP and WRITE SAME unmap(*) just work fine.

I certainly don't want to cause any more write-same trouble, but as all 
layers initially have to assume write same is supported anyway and need 
to dynamically disable it if it fails, can't we also enable discard by 
default with WRITE SAME16 unmap?
I'm going to send a PoC patch later on.

The older system I can play with for a few days has an Intel510 
(SSDSC2MH25) connected to an LSI SAS9211-8i via a sas enclosure.
Ignoring identificication string and revision level, sg_vdp output is 
almost the same here, but with the exception of

(wheezy)node02:~# sg_vpd /dev/sdb -p bl
Block limits VPD page (SBC):
[...]
   Maximum unmap LBA count: 0
   Maximum unmap block descriptor count: 0
   Optimal unmap granularity: 0
[...]


I think interesting for discard is also read cap 16:

> (wheezy)node02:~# sg_readcap --16 /dev/sda
> Read Capacity results:
>    Protection: prot_en=0, p_type=0, p_i_exponent=0
>    Logical block provisioning: lbpme=0, lbprz=0
>    Last logical block address=490350671 (0x1d3a284f), Number of logical blocks=490350672
>    Logical block length=512 bytes
>    Logical blocks per physical block exponent=0
>    Lowest aligned logical block address=0
> Hence:
>    Device size: 251059544064 bytes, 239429.0 MiB, 251.06 GB


So again no indication of discard support.


We also long ago flushed  the IT fw and just recently updated to fw 
version 17
> mpt2sas0: LSISAS2008: FWVersion(17.00.01.00), ChipRevision(0x03), BiosVersion(07.33.00.00)
> mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)


Thanks,
Bernd


PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot unmap the 
last sector:

> (wheezy)node02:~# cat /sys/block/sdb/size
> 488397168

> (wheezy)node02:~# sg_write_same --16 --unmap --verbose --lba=488397167 --num=1 /dev/sdb
> Default data-out buffer set to 512 zeros

So write same works. But then unmap fails:

> (wheezy)node02:~# sg_unmap --verbose --lba=488397167 --num=1 /dev/sdb
>     unmap cdb: 42 00 00 00 00 00 00 00 18 00
> unmap:  Fixed format, current;  Sense key: Illegal Request
>  Additional sense: Logical block address out of range
>   Info fld=0x1d1c596f [488397167]
> bad field in UNMAP cdb

All sectors before that work fine:

> (wheezy)node02:~# sg_unmap --verbose --lba=0 --num=488397167 /dev/sdb
>     unmap cdb: 42 00 00 00 00 00 00 00 18 00
> (

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Martin K. Petersen]

>>>>> "Bernd" == Bernd Schubert <bernd.schubert@fastmail.fm> writes:

Bernd,

Bernd> Both types of systems we have in-house neither block limits vpd
Bernd> nor READ_CAP16 return anything that would indicate discard is
Bernd> supported. But UNMAP and WRITE SAME unmap(*) just work fine.

I have a collection of different SSDs in a tray connected to an LSI
SAS2008 ASIC. The 510 is the only drive that does not have LBPME=1.
Chances are it's because DRAT and RZAT are not set but it could also be
that the 510 is blacklisted.


Bernd> I certainly don't want to cause any more write-same trouble, but
Bernd> as all layers initially have to assume write same is supported
Bernd> anyway and need to dynamically disable it if it fails, can't we
Bernd> also enable discard by default with WRITE SAME16 unmap?  

No thanks :)

But you can force discards on by doing a:

 # echo -n unmap > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

or

 # echo -n writesame_16 > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

Create a udev rule if you like.

In any case I wouldn't recommend using TRIM on that drive...


Bernd> PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot
Bernd> unmap the last sector:

Yes, it appears there's an off-by-one bug in the UNMAP translation.
Sumit, is this something you guys can look into?

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Bernd Schubert]

On 09/26/2013 04:42 PM, Martin K. Petersen wrote:
>>>>>> "Bernd" == Bernd Schubert <bernd.schubert@fastmail.fm> writes:
>
> Bernd,
>
> Bernd> Both types of systems we have in-house neither block limits vpd
> Bernd> nor READ_CAP16 return anything that would indicate discard is
> Bernd> supported. But UNMAP and WRITE SAME unmap(*) just work fine.
>
> I have a collection of different SSDs in a tray connected to an LSI
> SAS2008 ASIC. The 510 is the only drive that does not have LBPME=1.
> Chances are it's because DRAT and RZAT are not set but it could also be
> that the 510 is blacklisted.
>
>
> Bernd> I certainly don't want to cause any more write-same trouble, but
> Bernd> as all layers initially have to assume write same is supported
> Bernd> anyway and need to dynamically disable it if it fails, can't we
> Bernd> also enable discard by default with WRITE SAME16 unmap?
>
> No thanks :)
>
> But you can force discards on by doing a:
>
>   # echo -n unmap > /sys/class/scsi_disk/x:y:z:n/provisioning_mode
>
> or
>
>   # echo -n writesame_16 > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

D'oh, I didn't know about this parameter! THANKS a lot!

>
> Create a udev rule if you like.

Definitely, we have rules anyway.

>
> In any case I wouldn't recommend using TRIM on that drive...

Hmm yeah, the problem is that that these SSDs are used in our compute 
nodes (but we as file system group can also sometimes use it for 
benchmarking) and these SSDs slow down rather quickly without trim. The 
current procedure is to remove everything, to sg_unmap the full device 
and to re-create anything. So anything that would allow us to do it on 
the fly is very welcome. I don't think we need DRAT and RZAT, as we only 
use raid0 for these nodes.

So thanks a lot for your hint about the provisioning_mode!

>
>
> Bernd> PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot
> Bernd> unmap the last sector:
>
> Yes, it appears there's an off-by-one bug in the UNMAP translation.
> Sumit, is this something you guys can look into?
>


Thanks again,
Bernd

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Douglas Gilbert]

On 13-09-26 10:42 AM, Martin K. Petersen wrote:
>>>>>> "Bernd" == Bernd Schubert <bernd.schubert@fastmail.fm> writes:
>
> Bernd,
>
> Bernd> Both types of systems we have in-house neither block limits vpd
> Bernd> nor READ_CAP16 return anything that would indicate discard is
> Bernd> supported. But UNMAP and WRITE SAME unmap(*) just work fine.
>
> I have a collection of different SSDs in a tray connected to an LSI
> SAS2008 ASIC. The 510 is the only drive that does not have LBPME=1.
> Chances are it's because DRAT and RZAT are not set but it could also be
> that the 510 is blacklisted.
>
>
> Bernd> I certainly don't want to cause any more write-same trouble, but
> Bernd> as all layers initially have to assume write same is supported
> Bernd> anyway and need to dynamically disable it if it fails, can't we
> Bernd> also enable discard by default with WRITE SAME16 unmap?
>
> No thanks :)
>
> But you can force discards on by doing a:
>
>   # echo -n unmap > /sys/class/scsi_disk/x:y:z:n/provisioning_mode
>
> or
>
>   # echo -n writesame_16 > /sys/class/scsi_disk/x:y:z:n/provisioning_mode
>
> Create a udev rule if you like.
>
> In any case I wouldn't recommend using TRIM on that drive...
>
>
> Bernd> PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot
> Bernd> unmap the last sector:
>
> Yes, it appears there's an off-by-one bug in the UNMAP translation.
> Sumit, is this something you guys can look into?

Hi Sumit,
While you are looking at the HBA firmware could you fix
this minor annoyance:

# lsscsi -H -t
.....
[6]    mpt3sas       sas:0x500605b006d3b510

# smp_rep_general /dev/mpt3ctl -s 0x500605b006d3b510
Report general response:
   expander change count: 0
   expander route indexes: 0
   long response: 0
   number of phys: 0
   zone configuring: 0
   self configuring: 0
   externally configurable route table: 0
   initiates SSP close: 0
   enclosure logical identifier (hex): b005065010b5d306

So that is a SMP REPORT GENERAL (RG) directed at the HBA itself.
So either my code is incorrectly decoding the "enclosure logical
identifier" or ... . That field was defined in SAS 1.1 so no
excuses on that front. 0x0 would be better than a shuffled
version of the HBA's own SAS address.

Same bug/feature in all versions of SAS-2 and now SAS-3 firmware
that I have tried.

Doug Gilbert


BTW The bsg driver can send a RG to the HBA like this:
          smp_rep_general /dev/bsg/sas_host6
     So the intent is clearer and the response is the same.

RE: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

[Saxena, Sumit]

>-----Original Message-----
>From: Douglas Gilbert [mailto:dgilbert@interlog.com]
>Sent: Thursday, September 26, 2013 9:17 PM
>To: Martin K. Petersen; Bernd Schubert
>Cc: Mike Snitzer; Hannes Reinecke; emilne@redhat.com; device-mapper
>development; linux-scsi@vger.kernel.org; Saxena, Sumit
>Subject: Re: SCSI's heuristics for enabling WRITE SAME still need work
>[was: dm mpath: disable WRITE SAME if it fails]
>
>On 13-09-26 10:42 AM, Martin K. Petersen wrote:
>>>>>>> "Bernd" == Bernd Schubert <bernd.schubert@fastmail.fm> writes:
>>
>> Bernd,
>>
>> Bernd> Both types of systems we have in-house neither block limits vpd
>> Bernd> nor READ_CAP16 return anything that would indicate discard is
>> Bernd> supported. But UNMAP and WRITE SAME unmap(*) just work fine.
>>
>> I have a collection of different SSDs in a tray connected to an LSI
>> SAS2008 ASIC. The 510 is the only drive that does not have LBPME=1.
>> Chances are it's because DRAT and RZAT are not set but it could also
>> be that the 510 is blacklisted.
>>
>>
>> Bernd> I certainly don't want to cause any more write-same trouble,
>> Bernd> but as all layers initially have to assume write same is
>> Bernd> supported anyway and need to dynamically disable it if it
>> Bernd> fails, can't we also enable discard by default with WRITE
>SAME16 unmap?
>>
>> No thanks :)
>>
>> But you can force discards on by doing a:
>>
>>   # echo -n unmap > /sys/class/scsi_disk/x:y:z:n/provisioning_mode
>>
>> or
>>
>>   # echo -n writesame_16 >
>> /sys/class/scsi_disk/x:y:z:n/provisioning_mode
>>
>> Create a udev rule if you like.
>>
>> In any case I wouldn't recommend using TRIM on that drive...
>>
>>
>> Bernd> PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot
>> Bernd> unmap the last sector:
>>
>> Yes, it appears there's an off-by-one bug in the UNMAP translation.
>> Sumit, is this something you guys can look into?
>
>Hi Sumit,
>While you are looking at the HBA firmware could you fix this minor
>annoyance:
>

Thibash, Can you please look at this?

Sumit 

># lsscsi -H -t
>.....
>[6]    mpt3sas       sas:0x500605b006d3b510
>
># smp_rep_general /dev/mpt3ctl -s 0x500605b006d3b510 Report general
>response:
>   expander change count: 0
>   expander route indexes: 0
>   long response: 0
>   number of phys: 0
>   zone configuring: 0
>   self configuring: 0
>   externally configurable route table: 0
>   initiates SSP close: 0
>   enclosure logical identifier (hex): b005065010b5d306
>
>So that is a SMP REPORT GENERAL (RG) directed at the HBA itself.
>So either my code is incorrectly decoding the "enclosure logical
>identifier" or ... . That field was defined in SAS 1.1 so no excuses on
>that front. 0x0 would be better than a shuffled version of the HBA's own
>SAS address.
>
>Same bug/feature in all versions of SAS-2 and now SAS-3 firmware that I
>have tried.
>
>Doug Gilbert
>
>
>BTW The bsg driver can send a RG to the HBA like this:
>          smp_rep_general /dev/bsg/sas_host6
>     So the intent is clearer and the response is the same.
>
>
>
>