Does mdadm supports TRIM command to SSDs?

Does mdadm supports TRIM command to SSDs?

[Bar Ziony]

Hi,

I wasn't able to find any information about it after searching for
hours - Does mdadm supports sending the TRIM command to SSDs, if using
the right filesystem with the 'discard' flag?
If it does support it, from what kernel version it is?

Thank you,
Bar.

Re: Does mdadm supports TRIM command to SSDs?

[Igor M Podlesny]

On 11 July 2012 06:55, Bar Ziony <bartzy@gmail.com> wrote:
> Hi,
>
> I wasn't able to find any information about it after searching for
> hours - Does mdadm supports sending the TRIM command to SSDs, if using
> the right filesystem with the 'discard' flag?
> If it does support it, from what kernel version it is?

   Hi!

   https://lkml.org/lkml/2012/3/11/261

--

Re: Does mdadm supports TRIM command to SSDs?

[Igor M Podlesny]

On 11 July 2012 08:43, Igor M Podlesny <for.poige+lsr@gmail.com> wrote:
> On 11 July 2012 06:55, Bar Ziony <bartzy@gmail.com> wrote:
>> Hi,
>>
>> I wasn't able to find any information about it after searching for
>> hours - Does mdadm supports sending the TRIM command to SSDs, if using
>> the right filesystem with the 'discard' flag?
>> If it does support it, from what kernel version it is?
>
>    Hi!
>
>    https://lkml.org/lkml/2012/3/11/261
>
   But those patches don't seem to be accepted though.

--

Re: Does mdadm supports TRIM command to SSDs?

[Bar Ziony]

So does that mean that there's no TRIM support? Because as I
understood, TRIM support was added from 2.6.37. As I'm not familiar
with the code, can someone check some recent version to see if there
is TRIM support?

Thanks,
Bar.


On Wed, Jul 11, 2012 at 4:08 AM, Igor M Podlesny
<for.poige+lsr@gmail.com> wrote:
> On 11 July 2012 08:43, Igor M Podlesny <for.poige+lsr@gmail.com> wrote:
>> On 11 July 2012 06:55, Bar Ziony <bartzy@gmail.com> wrote:
>>> Hi,
>>>
>>> I wasn't able to find any information about it after searching for
>>> hours - Does mdadm supports sending the TRIM command to SSDs, if using
>>> the right filesystem with the 'discard' flag?
>>> If it does support it, from what kernel version it is?
>>
>>    Hi!
>>
>>    https://lkml.org/lkml/2012/3/11/261
>>
>    But those patches don't seem to be accepted though.
>
> --

Re: Does mdadm supports TRIM command to SSDs?

[David Brown]

On 07/11/2012 12:55 AM, Bar Ziony wrote:
> Hi,
>
> I wasn't able to find any information about it after searching for
> hours - Does mdadm supports sending the TRIM command to SSDs, if using
> the right filesystem with the 'discard' flag?
> If it does support it, from what kernel version it is?
>
> Thank you,
> Bar.

The big question here is /why/ would you want TRIM support?  In many 
circumstances it leads to slower operations, and for SSDs from the past 
couple of years it is almost entirely superseded by the SSD's own 
garbage collection.

So while there certainly has been some work done on TRIM and md raid, it 
is not a priority as it is of questionable benefit.  You might find some 
operations are a little faster for older SSDs if you use TRIM, but your 
metadata operations will be a lot slower.

Your best bet is just to make sure you only buy SSDs that are at least 
half decent, and don't worry about TRIM.

Re: Does mdadm supports TRIM command to SSDs?

[Bar Ziony]

Hi David,

Thanks for the info, it's very assuring to know that TRIM is not needed.

We bought Samsung 830 256GB SSDs, and as I understood, they're very
good, but their GC is only when the device is idling, and they will be
installed on a DB server. I don't know if they'll be idle for long
periods.

Do you perhaps know if we need to over-provision modern SSDs like the
Samsung 830's ?

Thanks!
Bar.

On Wed, Jul 11, 2012 at 2:27 PM, David Brown <david.brown@hesbynett.no> wrote:
> On 07/11/2012 12:55 AM, Bar Ziony wrote:
>>
>> Hi,
>>
>> I wasn't able to find any information about it after searching for
>> hours - Does mdadm supports sending the TRIM command to SSDs, if using
>> the right filesystem with the 'discard' flag?
>> If it does support it, from what kernel version it is?
>>
>> Thank you,
>> Bar.
>
>
> The big question here is /why/ would you want TRIM support?  In many
> circumstances it leads to slower operations, and for SSDs from the past
> couple of years it is almost entirely superseded by the SSD's own garbage
> collection.
>
> So while there certainly has been some work done on TRIM and md raid, it is
> not a priority as it is of questionable benefit.  You might find some
> operations are a little faster for older SSDs if you use TRIM, but your
> metadata operations will be a lot slower.
>
> Your best bet is just to make sure you only buy SSDs that are at least half
> decent, and don't worry about TRIM.
>

Re: Does mdadm supports TRIM command to SSDs?

[David Brown]

On 07/11/2012 01:49 PM, Bar Ziony wrote:
> Hi David,
>
> Thanks for the info, it's very assuring to know that TRIM is not needed.

There are some benchmarks that indicate improvement with TRIM, but I 
think it is overrated, and there are many cases where it makes things 
worse.  If TRIM had been implemented properly (such as being 
asynchronous and working with the command queue) it would have been much 
more useful - but unfortunately it was made as a rush job to minimise 
the ageing effects seen with early SSDs.

>
> We bought Samsung 830 256GB SSDs, and as I understood, they're very
> good, but their GC is only when the device is idling, and they will be
> installed on a DB server. I don't know if they'll be idle for long
> periods.

There is always some idle time, even on a busy database server.  If not, 
then you probably need more ram in the server!

>
> Do you perhaps know if we need to over-provision modern SSDs like the
> Samsung 830's ?

There is a fair amount of variation in the over-provisioning SSDs have - 
you'd have to read the specs for the drives in question.  And if the 
drives have compressing controllers, they will effectively have more 
over-provisioning if your data is easily compressible.  However, if you 
don't need all the space on the disks, then it's easy to add more 
over-provisioning - you simply leave part of the disk out of the 
partitioning (but make sure you do that before the disk is used, or 
after a complete wipe).

>
> Thanks!
> Bar.
>
> On Wed, Jul 11, 2012 at 2:27 PM, David Brown <david.brown@hesbynett.no> wrote:
>> On 07/11/2012 12:55 AM, Bar Ziony wrote:
>>>
>>> Hi,
>>>
>>> I wasn't able to find any information about it after searching for
>>> hours - Does mdadm supports sending the TRIM command to SSDs, if using
>>> the right filesystem with the 'discard' flag?
>>> If it does support it, from what kernel version it is?
>>>
>>> Thank you,
>>> Bar.
>>
>>
>> The big question here is /why/ would you want TRIM support?  In many
>> circumstances it leads to slower operations, and for SSDs from the past
>> couple of years it is almost entirely superseded by the SSD's own garbage
>> collection.
>>
>> So while there certainly has been some work done on TRIM and md raid, it is
>> not a priority as it is of questionable benefit.  You might find some
>> operations are a little faster for older SSDs if you use TRIM, but your
>> metadata operations will be a lot slower.
>>
>> Your best bet is just to make sure you only buy SSDs that are at least half
>> decent, and don't worry about TRIM.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Does mdadm supports TRIM command to SSDs?

[David Brown]

On 07/11/2012 01:58 PM, Scott E. Armitage wrote:
> On Wed, Jul 11, 2012 at 7:27 AM, David Brown <david.brown@hesbynett.no
> <mailto:david.brown@hesbynett.no>> wrote:
>
>     The big question here is /why/ would you want TRIM support?  In many
>     circumstances it leads to slower operations, and for SSDs from the
>     past couple of years it is almost entirely superseded by the SSD's
>     own garbage collection.
>
>
> Are you speaking from a general usage standpoint, or from a TRIM + RAID
> standpoint? For the general usage case, I was under the impression that
> SSD garbage collection was hamstrung by insufficient knowledge about
> which blocks are and are not in use. E.g. when a filesystem deletes a
> large file simply by marking it as deleted in the
> filetable/inode/whatever that particular system uses, the SSD is forced
> to carry around (and re-copy, if necessary) the data blocks until the
> filesystem tells it those blocks are no longer in use (i.e. a TRIM command).
>

That is correct - except that "hamstrung" is an exaggeration.  Blocks 
that are unused by the filesystem still have to be carried around by the 
SSD if they are not TRIM'ed, and that is extra work.  But it is not 
actually a great deal of extra work.  In particular, filesystems do a 
lot of re-use of deleted blocks - when the logical block is re-written, 
the SSD knows that the old data is no longer needed.

The main purpose of TRIM, and the reason it was introduced, was not so 
much to avoid moving around unneeded copies of old data, but to make 
sure there are free erased blocks available when they are needed.  With 
modern SSDs with over-provisioning and better garbage collection, that 
is no longer necessary - there are always free blocks, since there are 
many more physical blocks than logical ones.

If TRIM were well implemented, then it could still have been useful. 
But it is a synchronous command, and cannot be queued - this means it is 
slow, and breaks the flow of commands.  So the result is that operations 
such as "delete" can take many times longer to complete if TRIM is enabled.

Re: Does mdadm supports TRIM command to SSDs?

[Bernd Schubert]

On 07/11/2012 01:27 PM, David Brown wrote:
> On 07/11/2012 12:55 AM, Bar Ziony wrote:
>> Hi,
>>
>> I wasn't able to find any information about it after searching for
>> hours - Does mdadm supports sending the TRIM command to SSDs, if using
>> the right filesystem with the 'discard' flag?
>> If it does support it, from what kernel version it is?
>>
>> Thank you,
>> Bar.
> 
> The big question here is /why/ would you want TRIM support?  In many
> circumstances it leads to slower operations, and for SSDs from the past
> couple of years it is almost entirely superseded by the SSD's own
> garbage collection.
> 
> So while there certainly has been some work done on TRIM and md raid, it
> is not a priority as it is of questionable benefit.  You might find some
> operations are a little faster for older SSDs if you use TRIM, but your
> metadata operations will be a lot slower.
> 
> Your best bet is just to make sure you only buy SSDs that are at least
> half decent, and don't worry about TRIM.


Well, we do have not-so-old Intel 510 SSDs on our institute cluster and
these SSDs really need trim. And it gives us quite some headache for
some reason (reason is an annoying LSI SAS controller they are connected
to).


Cheers,
Bernd

Re: Does mdadm supports TRIM command to SSDs?

[Bar Ziony]

If they are connected to a regular LSI controller they won't get TRIM
commands... How do you know they need TRIM?

Thanks,
Bar.

On Wed, Jul 11, 2012 at 7:21 PM, Bernd Schubert
<bernd.schubert@fastmail.fm> wrote:
> On 07/11/2012 01:27 PM, David Brown wrote:
>> On 07/11/2012 12:55 AM, Bar Ziony wrote:
>>> Hi,
>>>
>>> I wasn't able to find any information about it after searching for
>>> hours - Does mdadm supports sending the TRIM command to SSDs, if using
>>> the right filesystem with the 'discard' flag?
>>> If it does support it, from what kernel version it is?
>>>
>>> Thank you,
>>> Bar.
>>
>> The big question here is /why/ would you want TRIM support?  In many
>> circumstances it leads to slower operations, and for SSDs from the past
>> couple of years it is almost entirely superseded by the SSD's own
>> garbage collection.
>>
>> So while there certainly has been some work done on TRIM and md raid, it
>> is not a priority as it is of questionable benefit.  You might find some
>> operations are a little faster for older SSDs if you use TRIM, but your
>> metadata operations will be a lot slower.
>>
>> Your best bet is just to make sure you only buy SSDs that are at least
>> half decent, and don't worry about TRIM.
>
>
> Well, we do have not-so-old Intel 510 SSDs on our institute cluster and
> these SSDs really need trim. And it gives us quite some headache for
> some reason (reason is an annoying LSI SAS controller they are connected
> to).
>
>
> Cheers,
> Bernd

Re: Does mdadm supports TRIM command to SSDs?

[Bernd Schubert]

On 07/11/2012 06:22 PM, Bar Ziony wrote:
> If they are connected to a regular LSI controller they won't get TRIM
> commands... How do you know they need TRIM?
> 

We discussed this with LSI, their answer from generic support was
basically nonsense, IMHO. This is a SAS Raid Controller, but after we
run into the trim issue for the first time, we are using it in non-raid
mode (flashed the corresponding fw). After we did so, we can trim with
sg_unmap, but as we don't know which blocks are in use by the
filesystem, we obviously have to trim the entire disk. And that means we
have to recreate LVM + file-system every time trimming is required.
Basic problem of those LSI controllers is that they don't announce their
trimming capabilities via scsi VPD or Read Capacity (16), which the
linux-scsi layer uses to query the disk for trim support. So the actual
problem is the LSI SATA-to-SAS translation, which doesn't seem to let
those important information through.
I already thought about writing a kernel patch to set those values vai
sysfs, but it probably will not get accepted, as setting the wrong
values would result in data corruption after trimming...

We are an HPC development group and our cluster is used sometimes for
benchmarking. Once trimming is required, write performance of those
disks goes down by about 50%. After manual sg_unmap performance is fine
again...

Cheers,
Bernd