LSI SAS2008 SATA TRIM not working

LSI SAS2008 SATA TRIM not working

[Kurt Miller]

Various sources indicate that LSI's SAS2008 controllers support TRIM
when running their IT firmware (LSI [1] and this list [2]). However, I
have not been able to get it working with Dell PERC H200 or H310 cross
flashed into LSI IT firmware. Currently I'm testing with Samsung 840 EVO
SATA SSDs. I have tried various LSI IT firmware versions (P14, P16, P18)
and various Linux distributions (Ubuntu 13.10, Ubuntu 12.04, Ubuntu 14
beta, RHEL 7 beta, Fedora 19).

The SSDs report they support TRIM:

# hdparm -I /dev/sdc | grep "TRIM supported"
    *   Data Set Management TRIM supported (limit 8 blocks)

Checking for support shows TRIM support is not making it through:

# cat /sys/block/sdc/queue/discard_granularity 
0

# fstrim /srv/node/disk2p1
fstrim: /srv/node/disk2p1: FITRIM ioctl failed: Operation not supported

I am not using LVM or crypto. I've tried both ext4 and xfs. Here is an
example fstab line:

LABEL=disk2p1 /srv/node/disk2p1 xfs
noatime,nodiratime,nobarrier,logbufs=8,discard 0 0

The dmesg lines for mpt2sas look as follows:

mpt2sas version 15.100.00.00 loaded
mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (16418600
kB)
mpt2sas 0000:04:00.0: irq 79 for MSI/MSI-X
mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 79
mpt2sas0: iomem(0x00000000df2b0000), mapped(0xffffc90011be0000),
size(65536)
mpt2sas0: ioport(0x000000000000fc00), size(256)
mpt2sas0: sending message unit reset !!
mpt2sas0: message unit reset: SUCCESS
mpt2sas0: Allocated physical memory: size(8056 kB)
mpt2sas0: Current Controller Queue Depth(3579), Max Controller Queue
Depth(3712)
mpt2sas0: Scatter Gather Elements per IO(128)
mpt2sas0: LSISAS2008: FWVersion(14.00.01.00), ChipRevision(0x03),
BiosVersion(07.27.00.00)
mpt2sas0: Dell 6Gbps SAS HBA: Vendor(0x1000), Device(0x0072),
SSVID(0x1028), SSDID(0x1F1C)
mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot
Buffer,Diag Trace Buffer,Task Set Full,NCQ)
mpt2sas0: sending port enable !!
mpt2sas0: host_add: handle(0x0001), sas_addr(0x590b11c027281600),
phys(8)
mpt2sas0: port enable: SUCCESS

Is TRIM working for anyone using LSI SAS2008 controllers?

I'm out of ideas as to what could be the cause of the failure. Is it
just that mpt2sas doesn't support TRIM contrary to the links below?

Thanks,
-Kurt

[1]
http://mycusthelp.info/LSI/_cs/AnswerDetail.aspx?sSessionID=6784104245IHLKQYLMXJTXVUMPQBJOGGFDORLPZB&inc=8039&caller=~%2fFindAnswers.aspx%3ftxtCriteria%3dtrim%26sSessionid%3d6784104245IHLKQYLMXJTXVUMPQBJOGGFDORLPZB

[2] http://comments.gmane.org/gmane.linux.scsi/69511

Re: LSI SAS2008 SATA TRIM not working

[Martin K. Petersen]

>>>>> "Kurt" == Kurt Miller <kurt@intricatesoftware.com> writes:

Kurt> Is TRIM working for anyone using LSI SAS2008 controllers?

Yes. But its SAT appears to be somewhat selective about which devices it
flags as capable.

Please send me the output of:

# sg_readcap -l /dev/sdc
# sg_vpd -p bl /dev/sdc
# sg_vpd -p lbpv /dev/sdc

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: LSI SAS2008 SATA TRIM not working

[Kurt Miller]

On Fri, 2014-02-07 at 15:44 -0500, Martin K. Petersen wrote:
> >>>>> "Kurt" == Kurt Miller <kurt@intricatesoftware.com> writes:
> 
> Kurt> Is TRIM working for anyone using LSI SAS2008 controllers?
> 
> Yes. But its SAT appears to be somewhat selective about which devices it
> flags as capable.
> 
> Please send me the output of:
> 
> # sg_readcap -l /dev/sdc
> # sg_vpd -p bl /dev/sdc
> # sg_vpd -p lbpv /dev/sdc
> 

Hi Martin,

Thank you for your reply. Below is the output you requested. Note that
when the EVO's are connected to a SATA AHCI port on my desktop
motherboard, TRIM works okay. Please let me know if you need additional
information.

Regards,
-Kurt

# sg_readcap -l /dev/sdc
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last logical block address=976773167 (0x3a38602f), Number of logical
blocks=976773168
   Logical block length=512 bytes
   Logical blocks per physical block exponent=0
   Lowest aligned logical block address=0
Hence:
   Device size: 500107862016 bytes, 476940.0 MiB, 500.11 GB

# sg_vpd -p bl /dev/sdc
Block limits VPD page (SBC):
  Write same no zero (WSNZ): 0
  Maximum compare and write length: 0 blocks
  Optimal transfer length granularity: 0 blocks
  Maximum transfer length: 0 blocks
  Optimal transfer length: 0 blocks
  Maximum prefetch length: 0 blocks
  Maximum unmap LBA count: 0
  Maximum unmap block descriptor count: 0
  Optimal unmap granularity: 0
  Unmap granularity alignment valid: 0
  Unmap granularity alignment: 0
  Maximum write same length: 0x0 blocks

# sg_vpd -p lbpv /dev/sdc
Logical block provisioning VPD page (SBC):
  Unmap command supported (LBPU): 0
  Write same (16) with unmap bit supported (LBWS): 0
  Write same (10) with unmap bit supported (LBWS10): 0
  Logical block provisioning read zeros (LBPRZ): 0
  Anchored LBAs supported (ANC_SUP): 0
  Threshold exponent: 0
  Descriptor present (DP): 0
  Provisioning type: 0

Re: LSI SAS2008 SATA TRIM not working

[Martin K. Petersen]

>>>>> "Kurt" == Kurt Miller <kurt@intricatesoftware.com> writes:

Kurt> Thank you for your reply. Below is the output you requested. Note
Kurt> that when the EVO's are connected to a SATA AHCI port on my
Kurt> desktop motherboard, TRIM works okay. Please let me know if you
Kurt> need additional information.

Kurt>    provisioning: lbpme=0, lbprz=0 Last logical block
                       ^^^^^^^

This indicates the SAT on the mpt2sas board didn't enable discard
support for the drive.

Please provide the full output of hdparm -I. Maybe we can get some clues
as to why that is...

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: LSI SAS2008 SATA TRIM not working

[Kurt Miller]

On Fri, 2014-02-07 at 15:59 -0500, Martin K. Petersen wrote:
> >>>>> "Kurt" == Kurt Miller <kurt@intricatesoftware.com> writes:
> Kurt>    provisioning: lbpme=0, lbprz=0 Last logical block
>                        ^^^^^^^
> 
> This indicates the SAT on the mpt2sas board didn't enable discard
> support for the drive.
> 
> Please provide the full output of hdparm -I. Maybe we can get some clues
> as to why that is...

Thanks again.

#  hdparm -I /dev/sdc

/dev/sdc:

ATA device, with non-removable media
	Model Number:       Samsung SSD 840 EVO 500GB               
	Serial Number:      S1DHNSAD921246E     
	Firmware Revision:  EXT0BB0Q
	Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions,
SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
	Used: unknown (minor revision code 0x0039) 
	Supported: 9 8 7 6 5 
	Likely used: 9
Configuration:
	Logical		max	current
	cylinders	16383	16383
	heads		16	16
	sectors/track	63	63
	--
	CHS current addressable sectors:   16514064
	LBA    user addressable sectors:  268435455
	LBA48  user addressable sectors:  976773168
	Logical  Sector size:                   512 bytes
	Physical Sector size:                   512 bytes
	Logical Sector-0 offset:                  0 bytes
	device size with M = 1024*1024:      476940 MBytes
	device size with M = 1000*1000:      500107 MBytes (500 GB)
	cache/buffer size  = unknown
	Nominal Media Rotation Rate: Solid State Device
Capabilities:
	LBA, IORDY(can be disabled)
	Queue depth: 32
	Standby timer values: spec'd by Standard, no device specific minimum
	R/W multiple sector transfer: Max = 16	Current = 16
	DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
	     Cycle time: min=120ns recommended=120ns
	PIO: pio0 pio1 pio2 pio3 pio4 
	     Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
	Enabled	Supported:
	   *	SMART feature set
	    	Security Mode feature set
	   *	Power Management feature set
	   *	Write cache
	   *	Look-ahead
	   *	Host Protected Area feature set
	   *	WRITE_BUFFER command
	   *	READ_BUFFER command
	   *	NOP cmd
	   *	DOWNLOAD_MICROCODE
	    	SET_MAX security extension
	   *	48-bit Address feature set
	   *	Device Configuration Overlay feature set
	   *	Mandatory FLUSH_CACHE
	   *	FLUSH_CACHE_EXT
	   *	SMART error logging
	   *	SMART self-test
	   *	General Purpose Logging feature set
	   *	WRITE_{DMA|MULTIPLE}_FUA_EXT
	   *	64-bit World wide name
	    	Write-Read-Verify feature set
	   *	WRITE_UNCORRECTABLE_EXT command
	   *	{READ,WRITE}_DMA_EXT_GPL commands
	   *	Segmented DOWNLOAD_MICROCODE
	   *	Gen1 signaling speed (1.5Gb/s)
	   *	Gen2 signaling speed (3.0Gb/s)
	   *	Gen3 signaling speed (6.0Gb/s)
	   *	Native Command Queueing (NCQ)
	   *	Phy event counters
	   *	READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
	   *	DMA Setup Auto-Activate optimization
	    	Device-initiated interface power management
	   *	Asynchronous notification (eg. media change)
	   *	Software settings preservation
	   *	SMART Command Transport (SCT) feature set
	   *	SCT Write Same (AC2)
	   *	SCT Error Recovery Control (AC3)
	   *	SCT Features Control (AC4)
	   *	SCT Data Tables (AC5)
	   *	reserved 69[4]
	   *	DOWNLOAD MICROCODE DMA command
	   *	SET MAX SETPASSWORD/UNLOCK DMA commands
	   *	WRITE BUFFER DMA command
	   *	READ BUFFER DMA command
	   *	Data Set Management TRIM supported (limit 8 blocks)
Security: 
	Master password revision code = 65534
		supported
	not	enabled
	not	locked
	not	frozen
	not	expired: security count
		supported: enhanced erase
	2min for SECURITY ERASE UNIT. 8min for ENHANCED SECURITY ERASE UNIT. 
Logical Unit WWN Device Identifier: 50025388a003c374
	NAA		: 5
	IEEE OUI	: 002538
	Unique ID	: 8a003c374
Checksum: correct

Re: LSI SAS2008 SATA TRIM not working

[Martin K. Petersen]

>>>>> "Kurt" == Kurt Miller <kurt@intricatesoftware.com> writes:

Kurt> #  hdparm -I /dev/sdc

[...]

I don't see any deterministic read after trim/read zero after trim
support on your drive. And I have a sneaking suspicion that those are
two of the fields that the LSI firmware looks at when deciding whether
to support logical block provisioning (in addition to support for the
DSM TRIM command).

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: LSI SAS2008 SATA TRIM not working

[Kurt Miller]

On Fri, 2014-02-07 at 20:24 -0500, Martin K. Petersen wrote:
> >>>>> "Kurt" == Kurt Miller <kurt@intricatesoftware.com> writes:
> 
> Kurt> #  hdparm -I /dev/sdc
> 
> [...]
> 
> I don't see any deterministic read after trim/read zero after trim
> support on your drive. And I have a sneaking suspicion that those are
> two of the fields that the LSI firmware looks at when deciding whether
> to support logical block provisioning (in addition to support for the
> DSM TRIM command).
> 

I wonder how to bring this issue to LSI's attention? My cross flashed
used controllers are out of support with LSI so I don't think I can get
them to listen to me.

>From a pricing perspective the Samsung 840 Pro looks interesting and is
on LSI's supported list. Does anyone have one of these around and can
check for deterministic read after trim? (hdparm -I /dev/sdc | grep
TRIM).

Thanks,
-Kurt

Re: LSI SAS2008 SATA TRIM not working

[Kurt Miller]

I can report that the Samsung 840 Pro *does* support trim on the LSI
SAS2008. As suspected it supports deterministic read zeros after trim.

One other thing to note, in my testing the P16 LSI firmware has broken
trim support. P14 and P15 report incorrect values for
/sys/block/sdX/queue/discard_granularity. P17 and P18 appear to work
fine and report correct values for discard_granularity.

-Kurt

Re: LSI SAS - SSDs with DRAT and DZAT

[Bernd Schubert]

On 02/12/2014 03:27 AM, Kurt Miller wrote:
> I can report that the Samsung 840 Pro*does*  support trim on the LSI
> SAS2008. As suspected it supports deterministic read zeros after trim.
>
> One other thing to note, in my testing the P16 LSI firmware has broken
> trim support. P14 and P15 report incorrect values for
> /sys/block/sdX/queue/discard_granularity. P17 and P18 appear to work
> fine and report correct values for discard_granularity.

We are just looking for SSDs for our new compute cluster and would like 
to get SSDs that support trim on LSI sas controllers out of the box.
Unfortunately, for a test sample of Samsungs MZ7WD240HCFV discard is not 
enabled by default, so probably DRAT and DZAT are missing
(I can't test that right now, as 'hdparm -I' does not seem to work with 
devices attached to the controller).

A colleague just told me that Samsung 840 Pro are also not an ideal 
option, as these have set overprovisioning to 0 bytes by default and 
there only seems to be windows tool to set it to a higher value. Not a 
nice idea if you want to do that for dozens or hundreds of drives.

Any other suggestions for ~240GB SSDs supporting DRAT and DZAT or 
trim-on-lsi-sas in general? The LSI support link sent by Kurt [1] is 
also not perfect with respect to trim, as it lists Intel510, although we 
definitely know that Intel510s are disabled by default and forcing 
write-same trim causes data corruption (unmap works, though).


Thanks in advance,
Bernd


[1]
> http://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus%20Adapters%20Common%20Files/LSI_6Gb_SAS_SATA_HBA_Compatibility_List.pdf

Re: LSI SAS - SSDs with DRAT and DZAT

[Martin K. Petersen]

>>>>> "Bernd" == Bernd Schubert <bernd.schubert@itwm.fraunhofer.de> writes:

Bernd> Any other suggestions for ~240GB SSDs supporting DRAT and DZAT or
Bernd> trim-on-lsi-sas in general?

intel drives (other than the 510 which was a one-off) are generally
very well-behaved.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: LSI SAS - SSDs with DRAT and DZAT

[Kurt Miller]

On Thu, 2014-04-17 at 15:42 -0400, Martin K. Petersen wrote:
> >>>>> "Bernd" == Bernd Schubert <bernd.schubert@itwm.fraunhofer.de> writes:
> 
> Bernd> Any other suggestions for ~240GB SSDs supporting DRAT and DZAT or
> Bernd> trim-on-lsi-sas in general?
> 
> intel drives (other than the 510 which was a one-off) are generally
> very well-behaved.
> 

Both Intel S3500 Series and Samsung 840 Pro SSDs with LSI in IT mode are
working well in my development lab here, including TRIM support.

-Kurt

Re: LSI SAS - SSDs with DRAT and DZAT

[Kurt Miller]

On Thu, 2014-04-17 at 16:10 -0700, Purush Gupta wrote:
> Kurt, Just curious which LSI firmware version are you using?

currently 18.

Intel S3500 on LSI SAS 9207-8i IT:
Linux thorin 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux
[    3.077947] mpt2sas0: LSISAS2308: FWVersion(18.00.00.00),
ChipRevision(0x05), BiosVersion(07.22.01.00)
[    3.276611] scsi 1:0:1:0: Direct-Access     ATA      INTEL SSDSC2BB48
0355 PQ: 0 ANSI: 6
[    3.528271] sd 1:0:1:0: [sdb] 937703088 512-byte logical blocks: (480
GB/447 GiB)
cat /sys/block/sdb/queue/discard_granularity 
512

Samsung 840 Pro on LSI 9211-8i IT:
Linux gloin 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
[    2.948533] mpt2sas0: LSISAS2008: FWVersion(18.00.00.00),
ChipRevision(0x03), BiosVersion(07.35.00.00)
[    2.968993] scsi 2:0:1:0: Direct-Access     ATA      Samsung SSD 840
5B0Q PQ: 0 ANSI: 6
[    2.976587] sd 2:0:1:0: [sdb] 1000215216 512-byte logical blocks:
(512 GB/476 GiB)
cat /sys/block/sdb/queue/discard_granularity 
512

-Kurt

Re: LSI SAS2008 SATA TRIM not working

[Kurt Miller]

On Thu, 2014-02-06 at 18:56 -0500, Kurt Miller wrote:
> Various sources indicate that LSI's SAS2008 controllers support TRIM
> when running their IT firmware (LSI [1] and this list [2]). However, I
> have not been able to get it working with Dell PERC H200 or H310 cross
> flashed into LSI IT firmware. Currently I'm testing with Samsung 840 EVO
> SATA SSDs. I have tried various LSI IT firmware versions (P14, P16, P18)
> and various Linux distributions (Ubuntu 13.10, Ubuntu 12.04, Ubuntu 14
> beta, RHEL 7 beta, Fedora 19).
>
> Is TRIM working for anyone using LSI SAS2008 controllers?

It turns out yes, TRIM is working for LSI SAS2008 controllers. However,
the Samsung 840 EVO's are not compatible with it and are also missing
from the LSI compatibility list for the controller [1]. When I got my
hands on an SSD that is on the list, TRIM worked as expected.

[1] http://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus
%20Adapters%20Common%20Files/LSI_6Gb_SAS_SATA_HBA_Compatibility_List.pdf

Regards,
-Kurt

Re: LSI SAS2008 SATA TRIM not working

[Chang Limin]

Kurt Miller <kurt <at> intricatesoftware.com> writes:

> 
> On Thu, 2014-02-06 at 18:56 -0500, Kurt Miller wrote:
> > Various sources indicate that LSI's SAS2008 controllers support TRIM
> > when running their IT firmware (LSI [1] and this list [2]). However, I
> > have not been able to get it working with Dell PERC H200 or H310 cross
> > flashed into LSI IT firmware. Currently I'm testing with Samsung 840 EVO
> > SATA SSDs. I have tried various LSI IT firmware versions (P14, P16, P18)
> > and various Linux distributions (Ubuntu 13.10, Ubuntu 12.04, Ubuntu 14
> > beta, RHEL 7 beta, Fedora 19).
> >
> > Is TRIM working for anyone using LSI SAS2008 controllers?
> 
> It turns out yes, TRIM is working for LSI SAS2008 controllers. However,
> the Samsung 840 EVO's are not compatible with it and are also missing
> from the LSI compatibility list for the controller [1]. When I got my
> hands on an SSD that is on the list, TRIM worked as expected.
> 
> [1] http://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus
> %20Adapters%20Common%20Files/LSI_6Gb_SAS_SATA_HBA_Compatibility_List.pdf
> 
> Regards,
> -Kurt
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

I find the Hitachi HUS110 disk storage also does not support fstrim, 
actually it supports unmap. 
The reason is the lbpme in 'sg_readcap -16' return 0, but the unmap in 
'sg_vpd -p 0xb2' return 1. 
Maybe the storage is not fully compatible to the SCSI standard.

# sg_readcap -16 /dev/dm-2
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=1

# sg_vpd --page=0xb2 /dev/dm-2
Logical block provisioning VPD page (SBC):
  Unmap command supported (LBPU): 1
  Write same (16) with unmap bit supported (LBWS): 1